2024-04-06T09:17:02.157 INFO:root:teuthology version: 0.0.1.dev262+ge691533 2024-04-06T09:17:02.157 DEBUG:teuthology.run:Teuthology command: teuthology --description rbd/mirror-thrash/{base/install clients/mirror cluster/{2-node openstack} conf/{disable-pool-app} msgr-failures/few objectstore/bluestore-comp-snappy policy/none rbd-mirror/four-per-cluster supported-random-distro$/{ubuntu_latest} workloads/rbd-mirror-journal-stress-workunit} --archive /home/teuthworker/archive/teuthology-2024-04-02_22:16:03-rbd-reef-distro-default-smithi/7637502 --verbose --name teuthology-2024-04-02_22:16:03-rbd-reef-distro-default-smithi --owner scheduled_teuthology@teuthology -- /home/teuthworker/archive/teuthology-2024-04-02_22:16:03-rbd-reef-distro-default-smithi/7637502/orig.config.yaml 2024-04-06T09:17:02.203 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-04-06T09:17:02.291 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/teuthology-2024-04-02_22:16:03-rbd-reef-distro-default-smithi/7637502 branch: reef description: rbd/mirror-thrash/{base/install clients/mirror cluster/{2-node openstack} conf/{disable-pool-app} msgr-failures/few objectstore/bluestore-comp-snappy policy/none rbd-mirror/four-per-cluster supported-random-distro$/{ubuntu_latest} workloads/rbd-mirror-journal-stress-workunit} email: ceph-qa@ceph.com first_in_suite: false job_id: '7637502' kernel: kdb: true sha1: distro ktype: distro last_in_suite: false machine_type: smithi meta: - desc: run two ceph clusters and install rbd-mirror - desc: configure the permissions for client.mirror - desc: 2 ceph clusters with 1 mon and 3 osds each - desc: run four rbd-mirror daemons per cluster - desc: run the rbd_mirror_stress.sh workunit to test the rbd-mirror daemon name: teuthology-2024-04-02_22:16:03-rbd-reef-distro-default-smithi no_nested_subset: false nuke-on-error: true openstack: - volumes: count: 3 size: 30 os_type: ubuntu os_version: '22.04' overrides: admin_socket: branch: reef ceph: conf: client: debug rbd: 20 debug rbd_mirror: 15 log to stderr: false rbd default features: 125 rbd mirror image policy type: none client.mirror.0: admin socket: /var/run/ceph/rbd-mirror.$cluster-$name.asok pid file: /var/run/ceph/rbd-mirror.$cluster-$name.pid client.mirror.1: admin socket: /var/run/ceph/rbd-mirror.$cluster-$name.asok pid file: /var/run/ceph/rbd-mirror.$cluster-$name.pid client.mirror.2: admin socket: /var/run/ceph/rbd-mirror.$cluster-$name.asok pid file: /var/run/ceph/rbd-mirror.$cluster-$name.pid client.mirror.3: admin socket: /var/run/ceph/rbd-mirror.$cluster-$name.asok pid file: /var/run/ceph/rbd-mirror.$cluster-$name.pid global: mon client directed command retry: 5 mon warn on pool no app: false ms inject socket failures: 5000 mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 osd: bluestore block size: 96636764160 bluestore compression algorithm: snappy bluestore compression mode: aggressive bluestore fsck on mount: true debug bluefs: 1/20 debug bluestore: 1/20 debug ms: 1 debug osd: 20 debug rocksdb: 4/10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - \(OSD_SLOW_PING_TIME sha1: 4d05f4cad464de0fcdd11a2856643e533aea34e8 ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} install: ceph: flavor: default sha1: 4d05f4cad464de0fcdd11a2856643e533aea34e8 thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: reef sha1: 052adcadd0534eddb0d73c5afbc083a84cd9dc46 owner: scheduled_teuthology@teuthology priority: 930 repo: https://git.ceph.com/ceph.git roles: - - cluster1.mon.a - cluster1.mgr.x - cluster2.mgr.x - cluster1.osd.0 - cluster1.osd.1 - cluster1.osd.2 - cluster1.client.0 - cluster2.client.0 - - cluster2.mon.a - cluster2.osd.0 - cluster2.osd.1 - cluster2.osd.2 - cluster1.client.mirror - cluster1.client.mirror.0 - cluster1.client.mirror.1 - cluster1.client.mirror.2 - cluster1.client.mirror.3 - cluster1.client.mirror.4 - cluster1.client.mirror.5 - cluster1.client.mirror.6 - cluster2.client.mirror - cluster2.client.mirror.0 - cluster2.client.mirror.1 - cluster2.client.mirror.2 - cluster2.client.mirror.3 - cluster2.client.mirror.4 - cluster2.client.mirror.5 - cluster2.client.mirror.6 seed: 9400 sha1: 4d05f4cad464de0fcdd11a2856643e533aea34e8 sleep_before_teardown: 0 subset: 12/128 suite: rbd suite_branch: reef suite_path: /home/teuthworker/src/git.ceph.com_ceph_052adcadd0534eddb0d73c5afbc083a84cd9dc46/qa suite_relpath: qa suite_repo: https://git.ceph.com/ceph.git suite_sha1: 052adcadd0534eddb0d73c5afbc083a84cd9dc46 targets: smithi006.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBPRsjjIxbbsh/IkYvxkow7Ob8LWlTfeg943sr/m1r913UrvId+M7tk4dUHrqVGT1fibXJxU9TjyR75QZn7tVsG0= smithi192.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBFwM+L3rHFCaXoKHuY/3HZtsZsr61XOLlFJZtgkQKNRHUQSe/PZxCqw1jcy0QQ7XEb3wk4LbI1nEnCQwkpC2S8M= tasks: - install: extra_packages: - rbd-mirror - ceph: cluster: cluster1 - ceph: cluster: cluster2 - exec: cluster1.client.mirror.0: - sudo ceph --cluster cluster1 auth caps client.mirror mon 'profile rbd-mirror-peer' osd 'profile rbd' - sudo ceph --cluster cluster1 auth caps client.mirror.0 mon 'profile rbd-mirror' osd 'profile rbd' - sudo ceph --cluster cluster1 auth caps client.mirror.1 mon 'profile rbd-mirror' osd 'profile rbd' - sudo ceph --cluster cluster1 auth caps client.mirror.2 mon 'profile rbd-mirror' osd 'profile rbd' - sudo ceph --cluster cluster1 auth caps client.mirror.3 mon 'profile rbd-mirror' osd 'profile rbd' - sudo ceph --cluster cluster2 auth caps client.mirror mon 'profile rbd-mirror-peer' osd 'profile rbd' - sudo ceph --cluster cluster2 auth caps client.mirror.0 mon 'profile rbd-mirror' osd 'profile rbd' - sudo ceph --cluster cluster2 auth caps client.mirror.1 mon 'profile rbd-mirror' osd 'profile rbd' - sudo ceph --cluster cluster2 auth caps client.mirror.2 mon 'profile rbd-mirror' osd 'profile rbd' - sudo ceph --cluster cluster2 auth caps client.mirror.3 mon 'profile rbd-mirror' osd 'profile rbd' - rbd-mirror: client: cluster1.client.mirror.0 thrash: true - rbd-mirror: client: cluster1.client.mirror.1 thrash: true - rbd-mirror: client: cluster1.client.mirror.2 thrash: true - rbd-mirror: client: cluster1.client.mirror.3 thrash: true - rbd-mirror: client: cluster2.client.mirror.0 thrash: true - rbd-mirror: client: cluster2.client.mirror.1 thrash: true - rbd-mirror: client: cluster2.client.mirror.2 thrash: true - rbd-mirror: client: cluster2.client.mirror.3 thrash: true - rbd-mirror-thrash: cluster: cluster1 - rbd-mirror-thrash: cluster: cluster2 - workunit: clients: cluster1.client.mirror: - rbd/rbd_mirror_stress.sh env: CEPH_ARGS: '' MIRROR_IMAGE_MODE: journal MIRROR_POOL_MODE: pool RBD_MIRROR_INSTANCES: '4' RBD_MIRROR_USE_EXISTING_CLUSTER: '1' RBD_MIRROR_USE_RBD_MIRROR: '1' timeout: 6h teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: e691533f9cbb33d85b2187bba20d7102f098636d timestamp: 2024-04-02_22:16:03 tube: smithi user: teuthology verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885 2024-04-06T09:17:02.291 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph_052adcadd0534eddb0d73c5afbc083a84cd9dc46/qa; will attempt to use it 2024-04-06T09:17:02.292 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph_052adcadd0534eddb0d73c5afbc083a84cd9dc46/qa/tasks 2024-04-06T09:17:02.292 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-04-06T09:17:02.294 INFO:teuthology.task.internal:Checking packages... 2024-04-06T09:17:02.321 INFO:teuthology.task.internal:Checking packages for os_type 'ubuntu', flavor 'default' and ceph hash '4d05f4cad464de0fcdd11a2856643e533aea34e8' 2024-04-06T09:17:02.322 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-04-06T09:17:02.322 INFO:teuthology.packaging:ref: None 2024-04-06T09:17:02.322 INFO:teuthology.packaging:tag: None 2024-04-06T09:17:02.322 INFO:teuthology.packaging:branch: reef 2024-04-06T09:17:02.322 INFO:teuthology.packaging:sha1: 4d05f4cad464de0fcdd11a2856643e533aea34e8 2024-04-06T09:17:02.322 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&ref=reef 2024-04-06T09:17:02.717 INFO:teuthology.task.internal:Found packages for ceph version 18.2.2-1169-g37f5b3f8-1jammy 2024-04-06T09:17:02.720 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-04-06T09:17:02.727 INFO:teuthology.task.internal:no buildpackages task found 2024-04-06T09:17:02.727 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-04-06T09:17:02.741 INFO:teuthology.task.internal:Saving configuration 2024-04-06T09:17:02.758 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-04-06T09:17:02.764 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-04-06T09:17:02.791 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi006.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-04-02_22:16:03-rbd-reef-distro-default-smithi/7637502', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'ubuntu', 'os_version': '22.04', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-06 09:11:24.751410', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIFvrD9oSiPA88uFNCeV2XCgt4mBfLwSO6wmbNSfY4Dwt'} 2024-04-06T09:17:02.814 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi192.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-04-02_22:16:03-rbd-reef-distro-default-smithi/7637502', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'ubuntu', 'os_version': '22.04', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-06 09:11:24.749428', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIOTfbIVRBEdTbRAbG0j/Qk/ZdUJo1Gtu1apVUHoqkUWU'} 2024-04-06T09:17:02.814 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-04-06T09:17:02.820 INFO:teuthology.task.internal:roles: ubuntu@smithi006.front.sepia.ceph.com - ['cluster1.mon.a', 'cluster1.mgr.x', 'cluster2.mgr.x', 'cluster1.osd.0', 'cluster1.osd.1', 'cluster1.osd.2', 'cluster1.client.0', 'cluster2.client.0'] 2024-04-06T09:17:02.820 INFO:teuthology.task.internal:roles: ubuntu@smithi192.front.sepia.ceph.com - ['cluster2.mon.a', 'cluster2.osd.0', 'cluster2.osd.1', 'cluster2.osd.2', 'cluster1.client.mirror', 'cluster1.client.mirror.0', 'cluster1.client.mirror.1', 'cluster1.client.mirror.2', 'cluster1.client.mirror.3', 'cluster1.client.mirror.4', 'cluster1.client.mirror.5', 'cluster1.client.mirror.6', 'cluster2.client.mirror', 'cluster2.client.mirror.0', 'cluster2.client.mirror.1', 'cluster2.client.mirror.2', 'cluster2.client.mirror.3', 'cluster2.client.mirror.4', 'cluster2.client.mirror.5', 'cluster2.client.mirror.6'] 2024-04-06T09:17:02.821 INFO:teuthology.run_tasks:Running task console_log... 2024-04-06T09:17:02.894 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f5ea4fdd040>, signals=[15]) 2024-04-06T09:17:02.895 INFO:teuthology.run_tasks:Running task internal.connect... 2024-04-06T09:17:02.901 INFO:teuthology.task.internal:Opening connections... 2024-04-06T09:17:02.901 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi006.front.sepia.ceph.com 2024-04-06T09:17:02.903 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi006.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-06T09:17:02.987 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi192.front.sepia.ceph.com 2024-04-06T09:17:02.988 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi192.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-06T09:17:03.063 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-04-06T09:17:03.069 DEBUG:teuthology.orchestra.run.smithi006:> uname -m 2024-04-06T09:17:03.074 INFO:teuthology.orchestra.run.smithi006.stdout:x86_64 2024-04-06T09:17:03.074 DEBUG:teuthology.orchestra.run.smithi006:> cat /etc/os-release 2024-04-06T09:17:03.126 INFO:teuthology.orchestra.run.smithi006.stdout:PRETTY_NAME="Ubuntu 22.04.1 LTS" 2024-04-06T09:17:03.126 INFO:teuthology.orchestra.run.smithi006.stdout:NAME="Ubuntu" 2024-04-06T09:17:03.126 INFO:teuthology.orchestra.run.smithi006.stdout:VERSION_ID="22.04" 2024-04-06T09:17:03.126 INFO:teuthology.orchestra.run.smithi006.stdout:VERSION="22.04.1 LTS (Jammy Jellyfish)" 2024-04-06T09:17:03.126 INFO:teuthology.orchestra.run.smithi006.stdout:VERSION_CODENAME=jammy 2024-04-06T09:17:03.126 INFO:teuthology.orchestra.run.smithi006.stdout:ID=ubuntu 2024-04-06T09:17:03.126 INFO:teuthology.orchestra.run.smithi006.stdout:ID_LIKE=debian 2024-04-06T09:17:03.126 INFO:teuthology.orchestra.run.smithi006.stdout:HOME_URL="https://www.ubuntu.com/" 2024-04-06T09:17:03.127 INFO:teuthology.orchestra.run.smithi006.stdout:SUPPORT_URL="https://help.ubuntu.com/" 2024-04-06T09:17:03.127 INFO:teuthology.orchestra.run.smithi006.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" 2024-04-06T09:17:03.127 INFO:teuthology.orchestra.run.smithi006.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" 2024-04-06T09:17:03.127 INFO:teuthology.orchestra.run.smithi006.stdout:UBUNTU_CODENAME=jammy 2024-04-06T09:17:03.127 INFO:teuthology.lock.ops:Updating smithi006.front.sepia.ceph.com on lock server 2024-04-06T09:17:03.161 DEBUG:teuthology.orchestra.run.smithi192:> uname -m 2024-04-06T09:17:03.166 INFO:teuthology.orchestra.run.smithi192.stdout:x86_64 2024-04-06T09:17:03.166 DEBUG:teuthology.orchestra.run.smithi192:> cat /etc/os-release 2024-04-06T09:17:03.211 INFO:teuthology.orchestra.run.smithi192.stdout:PRETTY_NAME="Ubuntu 22.04.1 LTS" 2024-04-06T09:17:03.212 INFO:teuthology.orchestra.run.smithi192.stdout:NAME="Ubuntu" 2024-04-06T09:17:03.212 INFO:teuthology.orchestra.run.smithi192.stdout:VERSION_ID="22.04" 2024-04-06T09:17:03.212 INFO:teuthology.orchestra.run.smithi192.stdout:VERSION="22.04.1 LTS (Jammy Jellyfish)" 2024-04-06T09:17:03.212 INFO:teuthology.orchestra.run.smithi192.stdout:VERSION_CODENAME=jammy 2024-04-06T09:17:03.212 INFO:teuthology.orchestra.run.smithi192.stdout:ID=ubuntu 2024-04-06T09:17:03.212 INFO:teuthology.orchestra.run.smithi192.stdout:ID_LIKE=debian 2024-04-06T09:17:03.212 INFO:teuthology.orchestra.run.smithi192.stdout:HOME_URL="https://www.ubuntu.com/" 2024-04-06T09:17:03.212 INFO:teuthology.orchestra.run.smithi192.stdout:SUPPORT_URL="https://help.ubuntu.com/" 2024-04-06T09:17:03.212 INFO:teuthology.orchestra.run.smithi192.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" 2024-04-06T09:17:03.212 INFO:teuthology.orchestra.run.smithi192.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" 2024-04-06T09:17:03.212 INFO:teuthology.orchestra.run.smithi192.stdout:UBUNTU_CODENAME=jammy 2024-04-06T09:17:03.212 INFO:teuthology.lock.ops:Updating smithi192.front.sepia.ceph.com on lock server 2024-04-06T09:17:03.252 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-04-06T09:17:03.262 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-04-06T09:17:03.268 INFO:teuthology.task.internal:Checking for old test directory... 2024-04-06T09:17:03.268 DEBUG:teuthology.orchestra.run.smithi006:> test '!' -e /home/ubuntu/cephtest 2024-04-06T09:17:03.270 DEBUG:teuthology.orchestra.run.smithi192:> test '!' -e /home/ubuntu/cephtest 2024-04-06T09:17:03.275 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-04-06T09:17:03.281 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-04-06T09:17:03.281 DEBUG:teuthology.orchestra.run.smithi006:> test -z $(ls -A /var/lib/ceph) 2024-04-06T09:17:03.316 DEBUG:teuthology.orchestra.run.smithi192:> test -z $(ls -A /var/lib/ceph) 2024-04-06T09:17:03.341 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-04-06T09:17:03.397 INFO:teuthology.run_tasks:Running task kernel... 2024-04-06T09:17:03.411 INFO:teuthology.task.kernel:normalize config orig: {'kdb': True, 'sha1': 'distro'} 2024-04-06T09:17:03.412 INFO:teuthology.task.kernel:config {'cluster1.mon.a': {'kdb': True, 'sha1': 'distro'}, 'cluster2.mon.a': {'kdb': True, 'sha1': 'distro'}}, timeout 300 2024-04-06T09:17:03.412 DEBUG:teuthology.orchestra.run.smithi006:> test -f /run/.containerenv -o -f /.dockerenv 2024-04-06T09:17:03.412 DEBUG:teuthology.orchestra.run.smithi192:> test -f /run/.containerenv -o -f /.dockerenv 2024-04-06T09:17:03.417 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T09:17:03.417 DEBUG:teuthology.orchestra.run.smithi006:> uname -r 2024-04-06T09:17:03.418 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T09:17:03.418 DEBUG:teuthology.orchestra.run.smithi192:> uname -r 2024-04-06T09:17:03.463 INFO:teuthology.orchestra.run.smithi006.stdout:5.15.0-56-generic 2024-04-06T09:17:03.463 INFO:teuthology.task.kernel:Running kernel on smithi006: 5.15.0-56-generic 2024-04-06T09:17:03.463 DEBUG:teuthology.orchestra.run.smithi006:> sudo apt-get clean 2024-04-06T09:17:03.464 INFO:teuthology.orchestra.run.smithi192.stdout:5.15.0-56-generic 2024-04-06T09:17:03.465 INFO:teuthology.task.kernel:Running kernel on smithi192: 5.15.0-56-generic 2024-04-06T09:17:03.465 DEBUG:teuthology.orchestra.run.smithi192:> sudo apt-get clean 2024-04-06T09:17:03.593 DEBUG:teuthology.orchestra.run.smithi006:> sudo apt-get update 2024-04-06T09:17:03.603 DEBUG:teuthology.orchestra.run.smithi192:> sudo apt-get update 2024-04-06T09:17:05.318 INFO:teuthology.orchestra.run.smithi006.stdout:Hit:1 http://archive.ubuntu.com/ubuntu jammy InRelease 2024-04-06T09:17:05.333 INFO:teuthology.orchestra.run.smithi006.stdout:Get:2 http://archive.ubuntu.com/ubuntu jammy-updates InRelease [119 kB] 2024-04-06T09:17:05.413 INFO:teuthology.orchestra.run.smithi006.stdout:Get:3 http://archive.ubuntu.com/ubuntu jammy-backports InRelease [109 kB] 2024-04-06T09:17:05.475 INFO:teuthology.orchestra.run.smithi192.stdout:Get:1 http://security.ubuntu.com/ubuntu jammy-security InRelease [110 kB] 2024-04-06T09:17:05.507 INFO:teuthology.orchestra.run.smithi006.stdout:Get:4 http://security.ubuntu.com/ubuntu jammy-security InRelease [110 kB] 2024-04-06T09:17:05.510 INFO:teuthology.orchestra.run.smithi192.stdout:Hit:2 http://archive.ubuntu.com/ubuntu jammy InRelease 2024-04-06T09:17:05.518 INFO:teuthology.orchestra.run.smithi192.stdout:Get:3 http://archive.ubuntu.com/ubuntu jammy-updates InRelease [119 kB] 2024-04-06T09:17:05.561 INFO:teuthology.orchestra.run.smithi006.stdout:Get:5 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 Packages [1,519 kB] 2024-04-06T09:17:05.610 INFO:teuthology.orchestra.run.smithi192.stdout:Get:4 http://archive.ubuntu.com/ubuntu jammy-backports InRelease [109 kB] 2024-04-06T09:17:05.670 INFO:teuthology.orchestra.run.smithi006.stdout:Get:6 http://archive.ubuntu.com/ubuntu jammy-updates/main i386 Packages [602 kB] 2024-04-06T09:17:05.676 INFO:teuthology.orchestra.run.smithi006.stdout:Get:7 http://archive.ubuntu.com/ubuntu jammy-updates/main Translation-en [293 kB] 2024-04-06T09:17:05.679 INFO:teuthology.orchestra.run.smithi006.stdout:Get:8 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 c-n-f Metadata [16.1 kB] 2024-04-06T09:17:05.679 INFO:teuthology.orchestra.run.smithi006.stdout:Get:9 http://archive.ubuntu.com/ubuntu jammy-updates/restricted i386 Packages [35.2 kB] 2024-04-06T09:17:05.680 INFO:teuthology.orchestra.run.smithi006.stdout:Get:10 http://archive.ubuntu.com/ubuntu jammy-updates/restricted amd64 Packages [1,648 kB] 2024-04-06T09:17:05.701 INFO:teuthology.orchestra.run.smithi006.stdout:Get:11 http://archive.ubuntu.com/ubuntu jammy-updates/restricted Translation-en [275 kB] 2024-04-06T09:17:05.704 INFO:teuthology.orchestra.run.smithi006.stdout:Get:12 http://archive.ubuntu.com/ubuntu jammy-updates/restricted amd64 c-n-f Metadata [520 B] 2024-04-06T09:17:05.705 INFO:teuthology.orchestra.run.smithi006.stdout:Get:13 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 Packages [1,060 kB] 2024-04-06T09:17:05.714 INFO:teuthology.orchestra.run.smithi006.stdout:Get:14 http://security.ubuntu.com/ubuntu jammy-security/main amd64 Packages [1,303 kB] 2024-04-06T09:17:05.716 INFO:teuthology.orchestra.run.smithi006.stdout:Get:15 http://archive.ubuntu.com/ubuntu jammy-updates/universe i386 Packages [698 kB] 2024-04-06T09:17:05.723 INFO:teuthology.orchestra.run.smithi006.stdout:Get:16 http://archive.ubuntu.com/ubuntu jammy-updates/universe Translation-en [241 kB] 2024-04-06T09:17:05.726 INFO:teuthology.orchestra.run.smithi006.stdout:Get:17 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 c-n-f Metadata [22.1 kB] 2024-04-06T09:17:05.727 INFO:teuthology.orchestra.run.smithi006.stdout:Get:18 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse i386 Packages [6,028 B] 2024-04-06T09:17:05.728 INFO:teuthology.orchestra.run.smithi006.stdout:Get:19 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse amd64 Packages [49.6 kB] 2024-04-06T09:17:05.728 INFO:teuthology.orchestra.run.smithi006.stdout:Get:20 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse Translation-en [12.0 kB] 2024-04-06T09:17:05.745 INFO:teuthology.orchestra.run.smithi192.stdout:Get:5 http://security.ubuntu.com/ubuntu jammy-security/main amd64 Packages [1,303 kB] 2024-04-06T09:17:05.773 INFO:teuthology.orchestra.run.smithi006.stdout:Get:21 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse amd64 c-n-f Metadata [472 B] 2024-04-06T09:17:05.773 INFO:teuthology.orchestra.run.smithi006.stdout:Get:22 http://archive.ubuntu.com/ubuntu jammy-backports/main i386 Packages [59.2 kB] 2024-04-06T09:17:05.774 INFO:teuthology.orchestra.run.smithi006.stdout:Get:23 http://archive.ubuntu.com/ubuntu jammy-backports/main amd64 Packages [67.1 kB] 2024-04-06T09:17:05.774 INFO:teuthology.orchestra.run.smithi006.stdout:Get:24 http://archive.ubuntu.com/ubuntu jammy-backports/main Translation-en [11.0 kB] 2024-04-06T09:17:05.775 INFO:teuthology.orchestra.run.smithi006.stdout:Get:25 http://archive.ubuntu.com/ubuntu jammy-backports/main amd64 c-n-f Metadata [388 B] 2024-04-06T09:17:05.775 INFO:teuthology.orchestra.run.smithi006.stdout:Get:26 http://archive.ubuntu.com/ubuntu jammy-backports/universe i386 Packages [17.2 kB] 2024-04-06T09:17:05.775 INFO:teuthology.orchestra.run.smithi006.stdout:Get:27 http://archive.ubuntu.com/ubuntu jammy-backports/universe amd64 Packages [28.4 kB] 2024-04-06T09:17:05.776 INFO:teuthology.orchestra.run.smithi006.stdout:Get:28 http://archive.ubuntu.com/ubuntu jammy-backports/universe Translation-en [16.2 kB] 2024-04-06T09:17:05.776 INFO:teuthology.orchestra.run.smithi006.stdout:Get:29 http://archive.ubuntu.com/ubuntu jammy-backports/universe amd64 c-n-f Metadata [644 B] 2024-04-06T09:17:05.821 INFO:teuthology.orchestra.run.smithi192.stdout:Get:6 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 Packages [1,519 kB] 2024-04-06T09:17:05.824 INFO:teuthology.orchestra.run.smithi006.stdout:Get:30 http://security.ubuntu.com/ubuntu jammy-security/main i386 Packages [436 kB] 2024-04-06T09:17:05.831 INFO:teuthology.orchestra.run.smithi006.stdout:Get:31 http://security.ubuntu.com/ubuntu jammy-security/main Translation-en [233 kB] 2024-04-06T09:17:05.834 INFO:teuthology.orchestra.run.smithi006.stdout:Get:32 http://security.ubuntu.com/ubuntu jammy-security/main amd64 c-n-f Metadata [11.4 kB] 2024-04-06T09:17:05.834 INFO:teuthology.orchestra.run.smithi006.stdout:Get:33 http://security.ubuntu.com/ubuntu jammy-security/restricted amd64 Packages [1,616 kB] 2024-04-06T09:17:05.856 INFO:teuthology.orchestra.run.smithi006.stdout:Get:34 http://security.ubuntu.com/ubuntu jammy-security/restricted i386 Packages [33.7 kB] 2024-04-06T09:17:05.857 INFO:teuthology.orchestra.run.smithi006.stdout:Get:35 http://security.ubuntu.com/ubuntu jammy-security/restricted Translation-en [271 kB] 2024-04-06T09:17:05.860 INFO:teuthology.orchestra.run.smithi006.stdout:Get:36 http://security.ubuntu.com/ubuntu jammy-security/restricted amd64 c-n-f Metadata [520 B] 2024-04-06T09:17:05.860 INFO:teuthology.orchestra.run.smithi006.stdout:Get:37 http://security.ubuntu.com/ubuntu jammy-security/universe amd64 Packages [852 kB] 2024-04-06T09:17:05.861 INFO:teuthology.orchestra.run.smithi192.stdout:Get:7 http://security.ubuntu.com/ubuntu jammy-security/main i386 Packages [436 kB] 2024-04-06T09:17:05.873 INFO:teuthology.orchestra.run.smithi192.stdout:Get:8 http://security.ubuntu.com/ubuntu jammy-security/main Translation-en [233 kB] 2024-04-06T09:17:05.875 INFO:teuthology.orchestra.run.smithi006.stdout:Get:38 http://security.ubuntu.com/ubuntu jammy-security/universe i386 Packages [599 kB] 2024-04-06T09:17:05.876 INFO:teuthology.orchestra.run.smithi192.stdout:Get:9 http://security.ubuntu.com/ubuntu jammy-security/main amd64 c-n-f Metadata [11.4 kB] 2024-04-06T09:17:05.877 INFO:teuthology.orchestra.run.smithi192.stdout:Get:10 http://security.ubuntu.com/ubuntu jammy-security/restricted amd64 Packages [1,616 kB] 2024-04-06T09:17:05.883 INFO:teuthology.orchestra.run.smithi006.stdout:Get:39 http://security.ubuntu.com/ubuntu jammy-security/universe Translation-en [163 kB] 2024-04-06T09:17:05.886 INFO:teuthology.orchestra.run.smithi006.stdout:Get:40 http://security.ubuntu.com/ubuntu jammy-security/universe amd64 c-n-f Metadata [16.8 kB] 2024-04-06T09:17:05.894 INFO:teuthology.orchestra.run.smithi006.stdout:Get:41 http://security.ubuntu.com/ubuntu jammy-security/multiverse amd64 Packages [37.1 kB] 2024-04-06T09:17:05.894 INFO:teuthology.orchestra.run.smithi006.stdout:Get:42 http://security.ubuntu.com/ubuntu jammy-security/multiverse i386 Packages [1,356 B] 2024-04-06T09:17:05.909 INFO:teuthology.orchestra.run.smithi192.stdout:Get:11 http://security.ubuntu.com/ubuntu jammy-security/restricted i386 Packages [33.7 kB] 2024-04-06T09:17:05.909 INFO:teuthology.orchestra.run.smithi192.stdout:Get:12 http://security.ubuntu.com/ubuntu jammy-security/restricted Translation-en [271 kB] 2024-04-06T09:17:05.914 INFO:teuthology.orchestra.run.smithi192.stdout:Get:13 http://security.ubuntu.com/ubuntu jammy-security/restricted amd64 c-n-f Metadata [520 B] 2024-04-06T09:17:05.914 INFO:teuthology.orchestra.run.smithi192.stdout:Get:14 http://security.ubuntu.com/ubuntu jammy-security/universe i386 Packages [599 kB] 2024-04-06T09:17:05.923 INFO:teuthology.orchestra.run.smithi192.stdout:Get:15 http://security.ubuntu.com/ubuntu jammy-security/universe amd64 Packages [852 kB] 2024-04-06T09:17:05.924 INFO:teuthology.orchestra.run.smithi006.stdout:Get:43 http://security.ubuntu.com/ubuntu jammy-security/multiverse Translation-en [7,476 B] 2024-04-06T09:17:05.924 INFO:teuthology.orchestra.run.smithi006.stdout:Get:44 http://security.ubuntu.com/ubuntu jammy-security/multiverse amd64 c-n-f Metadata [260 B] 2024-04-06T09:17:05.938 INFO:teuthology.orchestra.run.smithi192.stdout:Get:16 http://security.ubuntu.com/ubuntu jammy-security/universe Translation-en [163 kB] 2024-04-06T09:17:05.939 INFO:teuthology.orchestra.run.smithi192.stdout:Get:17 http://security.ubuntu.com/ubuntu jammy-security/universe amd64 c-n-f Metadata [16.8 kB] 2024-04-06T09:17:05.940 INFO:teuthology.orchestra.run.smithi192.stdout:Get:18 http://security.ubuntu.com/ubuntu jammy-security/multiverse amd64 Packages [37.1 kB] 2024-04-06T09:17:05.941 INFO:teuthology.orchestra.run.smithi192.stdout:Get:19 http://security.ubuntu.com/ubuntu jammy-security/multiverse i386 Packages [1,356 B] 2024-04-06T09:17:05.973 INFO:teuthology.orchestra.run.smithi192.stdout:Get:20 http://archive.ubuntu.com/ubuntu jammy-updates/main i386 Packages [602 kB] 2024-04-06T09:17:05.973 INFO:teuthology.orchestra.run.smithi192.stdout:Get:21 http://security.ubuntu.com/ubuntu jammy-security/multiverse Translation-en [7,476 B] 2024-04-06T09:17:05.973 INFO:teuthology.orchestra.run.smithi192.stdout:Get:22 http://security.ubuntu.com/ubuntu jammy-security/multiverse amd64 c-n-f Metadata [260 B] 2024-04-06T09:17:05.977 INFO:teuthology.orchestra.run.smithi192.stdout:Get:23 http://archive.ubuntu.com/ubuntu jammy-updates/main Translation-en [293 kB] 2024-04-06T09:17:05.980 INFO:teuthology.orchestra.run.smithi192.stdout:Get:24 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 c-n-f Metadata [16.1 kB] 2024-04-06T09:17:05.980 INFO:teuthology.orchestra.run.smithi192.stdout:Get:25 http://archive.ubuntu.com/ubuntu jammy-updates/restricted amd64 Packages [1,648 kB] 2024-04-06T09:17:06.006 INFO:teuthology.orchestra.run.smithi192.stdout:Get:26 http://archive.ubuntu.com/ubuntu jammy-updates/restricted i386 Packages [35.2 kB] 2024-04-06T09:17:06.006 INFO:teuthology.orchestra.run.smithi192.stdout:Get:27 http://archive.ubuntu.com/ubuntu jammy-updates/restricted Translation-en [275 kB] 2024-04-06T09:17:06.009 INFO:teuthology.orchestra.run.smithi192.stdout:Get:28 http://archive.ubuntu.com/ubuntu jammy-updates/restricted amd64 c-n-f Metadata [520 B] 2024-04-06T09:17:06.009 INFO:teuthology.orchestra.run.smithi192.stdout:Get:29 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 Packages [1,060 kB] 2024-04-06T09:17:06.022 INFO:teuthology.orchestra.run.smithi192.stdout:Get:30 http://archive.ubuntu.com/ubuntu jammy-updates/universe i386 Packages [698 kB] 2024-04-06T09:17:06.031 INFO:teuthology.orchestra.run.smithi192.stdout:Get:31 http://archive.ubuntu.com/ubuntu jammy-updates/universe Translation-en [241 kB] 2024-04-06T09:17:06.033 INFO:teuthology.orchestra.run.smithi192.stdout:Get:32 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 c-n-f Metadata [22.1 kB] 2024-04-06T09:17:06.034 INFO:teuthology.orchestra.run.smithi192.stdout:Get:33 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse amd64 Packages [49.6 kB] 2024-04-06T09:17:06.034 INFO:teuthology.orchestra.run.smithi192.stdout:Get:34 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse i386 Packages [6,028 B] 2024-04-06T09:17:06.052 INFO:teuthology.orchestra.run.smithi192.stdout:Get:35 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse Translation-en [12.0 kB] 2024-04-06T09:17:06.053 INFO:teuthology.orchestra.run.smithi192.stdout:Get:36 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse amd64 c-n-f Metadata [472 B] 2024-04-06T09:17:06.053 INFO:teuthology.orchestra.run.smithi192.stdout:Get:37 http://archive.ubuntu.com/ubuntu jammy-backports/main amd64 Packages [67.1 kB] 2024-04-06T09:17:06.054 INFO:teuthology.orchestra.run.smithi192.stdout:Get:38 http://archive.ubuntu.com/ubuntu jammy-backports/main i386 Packages [59.2 kB] 2024-04-06T09:17:06.054 INFO:teuthology.orchestra.run.smithi192.stdout:Get:39 http://archive.ubuntu.com/ubuntu jammy-backports/main Translation-en [11.0 kB] 2024-04-06T09:17:06.055 INFO:teuthology.orchestra.run.smithi192.stdout:Get:40 http://archive.ubuntu.com/ubuntu jammy-backports/main amd64 c-n-f Metadata [388 B] 2024-04-06T09:17:06.055 INFO:teuthology.orchestra.run.smithi192.stdout:Get:41 http://archive.ubuntu.com/ubuntu jammy-backports/universe amd64 Packages [28.4 kB] 2024-04-06T09:17:06.075 INFO:teuthology.orchestra.run.smithi192.stdout:Get:42 http://archive.ubuntu.com/ubuntu jammy-backports/universe i386 Packages [17.2 kB] 2024-04-06T09:17:06.076 INFO:teuthology.orchestra.run.smithi192.stdout:Get:43 http://archive.ubuntu.com/ubuntu jammy-backports/universe Translation-en [16.2 kB] 2024-04-06T09:17:06.076 INFO:teuthology.orchestra.run.smithi192.stdout:Get:44 http://archive.ubuntu.com/ubuntu jammy-backports/universe amd64 c-n-f Metadata [644 B] 2024-04-06T09:17:11.523 INFO:teuthology.orchestra.run.smithi192.stdout:Fetched 12.6 MB in 4s (3,559 kB/s) 2024-04-06T09:17:11.799 INFO:teuthology.orchestra.run.smithi006.stdout:Fetched 12.6 MB in 3s (3,653 kB/s) 2024-04-06T09:17:12.859 INFO:teuthology.orchestra.run.smithi192.stdout:Reading package lists... 2024-04-06T09:17:12.877 DEBUG:teuthology.orchestra.run.smithi192:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2024-04-06T09:17:12.979 INFO:teuthology.orchestra.run.smithi192.stdout:Reading package lists... 2024-04-06T09:17:13.177 INFO:teuthology.orchestra.run.smithi192.stdout:Building dependency tree... 2024-04-06T09:17:13.177 INFO:teuthology.orchestra.run.smithi192.stdout:Reading state information... 2024-04-06T09:17:13.249 INFO:teuthology.orchestra.run.smithi006.stdout:Reading package lists... 2024-04-06T09:17:13.266 DEBUG:teuthology.orchestra.run.smithi006:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2024-04-06T09:17:13.328 INFO:teuthology.orchestra.run.smithi006.stdout:Reading package lists... 2024-04-06T09:17:13.349 INFO:teuthology.orchestra.run.smithi192.stdout:The following packages were automatically installed and are no longer required: 2024-04-06T09:17:13.349 INFO:teuthology.orchestra.run.smithi192.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2024-04-06T09:17:13.349 INFO:teuthology.orchestra.run.smithi192.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2024-04-06T09:17:13.349 INFO:teuthology.orchestra.run.smithi192.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2024-04-06T09:17:13.350 INFO:teuthology.orchestra.run.smithi192.stdout: python2 python2-dev python2-minimal python2.7 python2.7-dev 2024-04-06T09:17:13.350 INFO:teuthology.orchestra.run.smithi192.stdout: python2.7-minimal 2024-04-06T09:17:13.350 INFO:teuthology.orchestra.run.smithi192.stdout:Use 'sudo apt autoremove' to remove them. 2024-04-06T09:17:13.351 INFO:teuthology.orchestra.run.smithi192.stdout:The following additional packages will be installed: 2024-04-06T09:17:13.351 INFO:teuthology.orchestra.run.smithi192.stdout: linux-generic linux-headers-5.15.0-101 linux-headers-5.15.0-101-generic 2024-04-06T09:17:13.351 INFO:teuthology.orchestra.run.smithi192.stdout: linux-headers-generic linux-image-5.15.0-101-generic 2024-04-06T09:17:13.351 INFO:teuthology.orchestra.run.smithi192.stdout: linux-modules-5.15.0-101-generic linux-modules-extra-5.15.0-101-generic 2024-04-06T09:17:13.353 INFO:teuthology.orchestra.run.smithi192.stdout:Suggested packages: 2024-04-06T09:17:13.353 INFO:teuthology.orchestra.run.smithi192.stdout: fdutils linux-doc | linux-source-5.15.0 linux-tools 2024-04-06T09:17:13.353 INFO:teuthology.orchestra.run.smithi192.stdout:Recommended packages: 2024-04-06T09:17:13.353 INFO:teuthology.orchestra.run.smithi192.stdout: thermald 2024-04-06T09:17:13.394 INFO:teuthology.orchestra.run.smithi192.stdout:The following NEW packages will be installed: 2024-04-06T09:17:13.394 INFO:teuthology.orchestra.run.smithi192.stdout: linux-headers-5.15.0-101 linux-headers-5.15.0-101-generic 2024-04-06T09:17:13.394 INFO:teuthology.orchestra.run.smithi192.stdout: linux-image-5.15.0-101-generic linux-modules-5.15.0-101-generic 2024-04-06T09:17:13.394 INFO:teuthology.orchestra.run.smithi192.stdout: linux-modules-extra-5.15.0-101-generic 2024-04-06T09:17:13.396 INFO:teuthology.orchestra.run.smithi192.stdout:The following packages will be upgraded: 2024-04-06T09:17:13.396 INFO:teuthology.orchestra.run.smithi192.stdout: linux-generic linux-headers-generic linux-image-generic 2024-04-06T09:17:13.530 INFO:teuthology.orchestra.run.smithi006.stdout:Building dependency tree... 2024-04-06T09:17:13.530 INFO:teuthology.orchestra.run.smithi006.stdout:Reading state information... 2024-04-06T09:17:13.575 INFO:teuthology.orchestra.run.smithi192.stdout:3 upgraded, 5 newly installed, 0 to remove and 312 not upgraded. 2024-04-06T09:17:13.575 INFO:teuthology.orchestra.run.smithi192.stdout:Need to get 113 MB of archives. 2024-04-06T09:17:13.575 INFO:teuthology.orchestra.run.smithi192.stdout:After this operation, 582 MB of additional disk space will be used. 2024-04-06T09:17:13.575 INFO:teuthology.orchestra.run.smithi192.stdout:Get:1 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-modules-5.15.0-101-generic amd64 5.15.0-101.111 [22.6 MB] 2024-04-06T09:17:13.710 INFO:teuthology.orchestra.run.smithi006.stdout:The following packages were automatically installed and are no longer required: 2024-04-06T09:17:13.711 INFO:teuthology.orchestra.run.smithi006.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2024-04-06T09:17:13.711 INFO:teuthology.orchestra.run.smithi006.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2024-04-06T09:17:13.711 INFO:teuthology.orchestra.run.smithi006.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2024-04-06T09:17:13.711 INFO:teuthology.orchestra.run.smithi006.stdout: python2 python2-dev python2-minimal python2.7 python2.7-dev 2024-04-06T09:17:13.711 INFO:teuthology.orchestra.run.smithi006.stdout: python2.7-minimal 2024-04-06T09:17:13.712 INFO:teuthology.orchestra.run.smithi006.stdout:Use 'sudo apt autoremove' to remove them. 2024-04-06T09:17:13.712 INFO:teuthology.orchestra.run.smithi006.stdout:The following additional packages will be installed: 2024-04-06T09:17:13.712 INFO:teuthology.orchestra.run.smithi006.stdout: linux-generic linux-headers-5.15.0-101 linux-headers-5.15.0-101-generic 2024-04-06T09:17:13.713 INFO:teuthology.orchestra.run.smithi006.stdout: linux-headers-generic linux-image-5.15.0-101-generic 2024-04-06T09:17:13.713 INFO:teuthology.orchestra.run.smithi006.stdout: linux-modules-5.15.0-101-generic linux-modules-extra-5.15.0-101-generic 2024-04-06T09:17:13.714 INFO:teuthology.orchestra.run.smithi006.stdout:Suggested packages: 2024-04-06T09:17:13.714 INFO:teuthology.orchestra.run.smithi006.stdout: fdutils linux-doc | linux-source-5.15.0 linux-tools 2024-04-06T09:17:13.714 INFO:teuthology.orchestra.run.smithi006.stdout:Recommended packages: 2024-04-06T09:17:13.714 INFO:teuthology.orchestra.run.smithi006.stdout: thermald 2024-04-06T09:17:13.762 INFO:teuthology.orchestra.run.smithi006.stdout:The following NEW packages will be installed: 2024-04-06T09:17:13.763 INFO:teuthology.orchestra.run.smithi006.stdout: linux-headers-5.15.0-101 linux-headers-5.15.0-101-generic 2024-04-06T09:17:13.763 INFO:teuthology.orchestra.run.smithi006.stdout: linux-image-5.15.0-101-generic linux-modules-5.15.0-101-generic 2024-04-06T09:17:13.763 INFO:teuthology.orchestra.run.smithi006.stdout: linux-modules-extra-5.15.0-101-generic 2024-04-06T09:17:13.764 INFO:teuthology.orchestra.run.smithi006.stdout:The following packages will be upgraded: 2024-04-06T09:17:13.765 INFO:teuthology.orchestra.run.smithi006.stdout: linux-generic linux-headers-generic linux-image-generic 2024-04-06T09:17:13.946 INFO:teuthology.orchestra.run.smithi006.stdout:3 upgraded, 5 newly installed, 0 to remove and 312 not upgraded. 2024-04-06T09:17:13.947 INFO:teuthology.orchestra.run.smithi006.stdout:Need to get 113 MB of archives. 2024-04-06T09:17:13.947 INFO:teuthology.orchestra.run.smithi006.stdout:After this operation, 582 MB of additional disk space will be used. 2024-04-06T09:17:13.947 INFO:teuthology.orchestra.run.smithi006.stdout:Get:1 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-modules-5.15.0-101-generic amd64 5.15.0-101.111 [22.6 MB] 2024-04-06T09:17:15.038 INFO:teuthology.orchestra.run.smithi192.stdout:Get:2 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-image-5.15.0-101-generic amd64 5.15.0-101.111 [11.5 MB] 2024-04-06T09:17:15.346 INFO:teuthology.orchestra.run.smithi006.stdout:Get:2 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-image-5.15.0-101-generic amd64 5.15.0-101.111 [11.5 MB] 2024-04-06T09:17:15.393 INFO:teuthology.orchestra.run.smithi192.stdout:Get:3 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-modules-extra-5.15.0-101-generic amd64 5.15.0-101.111 [63.9 MB] 2024-04-06T09:17:15.677 INFO:teuthology.orchestra.run.smithi006.stdout:Get:3 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-modules-extra-5.15.0-101-generic amd64 5.15.0-101.111 [63.9 MB] 2024-04-06T09:17:17.646 INFO:teuthology.orchestra.run.smithi192.stdout:Get:4 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-generic amd64 5.15.0.101.98 [1,698 B] 2024-04-06T09:17:17.646 INFO:teuthology.orchestra.run.smithi192.stdout:Get:5 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-image-generic amd64 5.15.0.101.98 [2,486 B] 2024-04-06T09:17:17.646 INFO:teuthology.orchestra.run.smithi192.stdout:Get:6 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-headers-5.15.0-101 all 5.15.0-101.111 [12.3 MB] 2024-04-06T09:17:17.803 INFO:teuthology.orchestra.run.smithi006.stdout:Get:4 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-generic amd64 5.15.0.101.98 [1,698 B] 2024-04-06T09:17:17.871 INFO:teuthology.orchestra.run.smithi006.stdout:Get:5 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-image-generic amd64 5.15.0.101.98 [2,486 B] 2024-04-06T09:17:17.871 INFO:teuthology.orchestra.run.smithi006.stdout:Get:6 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-headers-5.15.0-101 all 5.15.0-101.111 [12.3 MB] 2024-04-06T09:17:18.089 INFO:teuthology.orchestra.run.smithi192.stdout:Get:7 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-headers-5.15.0-101-generic amd64 5.15.0-101.111 [2,876 kB] 2024-04-06T09:17:18.336 INFO:teuthology.orchestra.run.smithi192.stdout:Get:8 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-headers-generic amd64 5.15.0.101.98 [2,332 B] 2024-04-06T09:17:18.707 INFO:teuthology.orchestra.run.smithi006.stdout:Get:7 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-headers-5.15.0-101-generic amd64 5.15.0-101.111 [2,876 kB] 2024-04-06T09:17:18.747 INFO:teuthology.orchestra.run.smithi192.stdout:Fetched 113 MB in 5s (22.9 MB/s) 2024-04-06T09:17:18.832 INFO:teuthology.orchestra.run.smithi006.stdout:Get:8 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-headers-generic amd64 5.15.0.101.98 [2,332 B] 2024-04-06T09:17:18.921 INFO:teuthology.orchestra.run.smithi192.stdout:Selecting previously unselected package linux-modules-5.15.0-101-generic. 2024-04-06T09:17:19.226 INFO:teuthology.orchestra.run.smithi006.stdout:Fetched 113 MB in 5s (22.4 MB/s) 2024-04-06T09:17:19.429 INFO:teuthology.orchestra.run.smithi006.stdout:Selecting previously unselected package linux-modules-5.15.0-101-generic. 2024-04-06T09:17:21.046 INFO:teuthology.orchestra.run.smithi192.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 129542 files and directories currently installed.) 2024-04-06T09:17:21.051 INFO:teuthology.orchestra.run.smithi192.stdout:Preparing to unpack .../0-linux-modules-5.15.0-101-generic_5.15.0-101.111_amd64.deb ... 2024-04-06T09:17:21.168 INFO:teuthology.orchestra.run.smithi192.stdout:Unpacking linux-modules-5.15.0-101-generic (5.15.0-101.111) ... 2024-04-06T09:17:21.457 INFO:teuthology.orchestra.run.smithi006.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 129542 files and directories currently installed.) 2024-04-06T09:17:21.461 INFO:teuthology.orchestra.run.smithi006.stdout:Preparing to unpack .../0-linux-modules-5.15.0-101-generic_5.15.0-101.111_amd64.deb ... 2024-04-06T09:17:21.610 INFO:teuthology.orchestra.run.smithi006.stdout:Unpacking linux-modules-5.15.0-101-generic (5.15.0-101.111) ... 2024-04-06T09:17:23.319 INFO:teuthology.orchestra.run.smithi192.stdout:Selecting previously unselected package linux-image-5.15.0-101-generic. 2024-04-06T09:17:23.340 INFO:teuthology.orchestra.run.smithi192.stdout:Preparing to unpack .../1-linux-image-5.15.0-101-generic_5.15.0-101.111_amd64.deb ... 2024-04-06T09:17:23.426 INFO:teuthology.orchestra.run.smithi192.stdout:Unpacking linux-image-5.15.0-101-generic (5.15.0-101.111) ... 2024-04-06T09:17:23.838 INFO:teuthology.orchestra.run.smithi192.stdout:Selecting previously unselected package linux-modules-extra-5.15.0-101-generic. 2024-04-06T09:17:23.859 INFO:teuthology.orchestra.run.smithi192.stdout:Preparing to unpack .../2-linux-modules-extra-5.15.0-101-generic_5.15.0-101.111_amd64.deb ... 2024-04-06T09:17:23.905 INFO:teuthology.orchestra.run.smithi192.stdout:Unpacking linux-modules-extra-5.15.0-101-generic (5.15.0-101.111) ... 2024-04-06T09:17:23.987 INFO:teuthology.orchestra.run.smithi006.stdout:Selecting previously unselected package linux-image-5.15.0-101-generic. 2024-04-06T09:17:24.008 INFO:teuthology.orchestra.run.smithi006.stdout:Preparing to unpack .../1-linux-image-5.15.0-101-generic_5.15.0-101.111_amd64.deb ... 2024-04-06T09:17:24.091 INFO:teuthology.orchestra.run.smithi006.stdout:Unpacking linux-image-5.15.0-101-generic (5.15.0-101.111) ... 2024-04-06T09:17:24.615 INFO:teuthology.orchestra.run.smithi006.stdout:Selecting previously unselected package linux-modules-extra-5.15.0-101-generic. 2024-04-06T09:17:24.636 INFO:teuthology.orchestra.run.smithi006.stdout:Preparing to unpack .../2-linux-modules-extra-5.15.0-101-generic_5.15.0-101.111_amd64.deb ... 2024-04-06T09:17:24.958 INFO:teuthology.orchestra.run.smithi006.stdout:Unpacking linux-modules-extra-5.15.0-101-generic (5.15.0-101.111) ... 2024-04-06T09:17:31.102 INFO:teuthology.orchestra.run.smithi192.stdout:Preparing to unpack .../3-linux-generic_5.15.0.101.98_amd64.deb ... 2024-04-06T09:17:31.229 INFO:teuthology.orchestra.run.smithi192.stdout:Unpacking linux-generic (5.15.0.101.98) over (5.15.0.56.54) ... 2024-04-06T09:17:31.570 INFO:teuthology.orchestra.run.smithi192.stdout:Preparing to unpack .../4-linux-image-generic_5.15.0.101.98_amd64.deb ... 2024-04-06T09:17:31.683 INFO:teuthology.orchestra.run.smithi192.stdout:Unpacking linux-image-generic (5.15.0.101.98) over (5.15.0.56.54) ... 2024-04-06T09:17:31.900 INFO:teuthology.orchestra.run.smithi006.stdout:Preparing to unpack .../3-linux-generic_5.15.0.101.98_amd64.deb ... 2024-04-06T09:17:31.978 INFO:teuthology.orchestra.run.smithi192.stdout:Selecting previously unselected package linux-headers-5.15.0-101. 2024-04-06T09:17:32.000 INFO:teuthology.orchestra.run.smithi192.stdout:Preparing to unpack .../5-linux-headers-5.15.0-101_5.15.0-101.111_all.deb ... 2024-04-06T09:17:32.043 INFO:teuthology.orchestra.run.smithi006.stdout:Unpacking linux-generic (5.15.0.101.98) over (5.15.0.56.54) ... 2024-04-06T09:17:32.044 INFO:teuthology.orchestra.run.smithi192.stdout:Unpacking linux-headers-5.15.0-101 (5.15.0-101.111) ... 2024-04-06T09:17:32.376 INFO:teuthology.orchestra.run.smithi006.stdout:Preparing to unpack .../4-linux-image-generic_5.15.0.101.98_amd64.deb ... 2024-04-06T09:17:32.501 INFO:teuthology.orchestra.run.smithi006.stdout:Unpacking linux-image-generic (5.15.0.101.98) over (5.15.0.56.54) ... 2024-04-06T09:17:32.816 INFO:teuthology.orchestra.run.smithi006.stdout:Selecting previously unselected package linux-headers-5.15.0-101. 2024-04-06T09:17:32.839 INFO:teuthology.orchestra.run.smithi006.stdout:Preparing to unpack .../5-linux-headers-5.15.0-101_5.15.0-101.111_all.deb ... 2024-04-06T09:17:32.874 INFO:teuthology.orchestra.run.smithi006.stdout:Unpacking linux-headers-5.15.0-101 (5.15.0-101.111) ... 2024-04-06T09:17:37.965 INFO:teuthology.orchestra.run.smithi192.stdout:Selecting previously unselected package linux-headers-5.15.0-101-generic. 2024-04-06T09:17:38.002 INFO:teuthology.orchestra.run.smithi192.stdout:Preparing to unpack .../6-linux-headers-5.15.0-101-generic_5.15.0-101.111_amd64.deb ... 2024-04-06T09:17:38.038 INFO:teuthology.orchestra.run.smithi192.stdout:Unpacking linux-headers-5.15.0-101-generic (5.15.0-101.111) ... 2024-04-06T09:17:38.805 INFO:teuthology.orchestra.run.smithi006.stdout:Selecting previously unselected package linux-headers-5.15.0-101-generic. 2024-04-06T09:17:38.843 INFO:teuthology.orchestra.run.smithi006.stdout:Preparing to unpack .../6-linux-headers-5.15.0-101-generic_5.15.0-101.111_amd64.deb ... 2024-04-06T09:17:38.913 INFO:teuthology.orchestra.run.smithi006.stdout:Unpacking linux-headers-5.15.0-101-generic (5.15.0-101.111) ... 2024-04-06T09:17:41.451 INFO:teuthology.orchestra.run.smithi192.stdout:Preparing to unpack .../7-linux-headers-generic_5.15.0.101.98_amd64.deb ... 2024-04-06T09:17:41.577 INFO:teuthology.orchestra.run.smithi192.stdout:Unpacking linux-headers-generic (5.15.0.101.98) over (5.15.0.56.54) ... 2024-04-06T09:17:41.927 INFO:teuthology.orchestra.run.smithi192.stdout:Setting up linux-headers-5.15.0-101 (5.15.0-101.111) ... 2024-04-06T09:17:42.074 INFO:teuthology.orchestra.run.smithi192.stdout:Setting up linux-headers-5.15.0-101-generic (5.15.0-101.111) ... 2024-04-06T09:17:42.183 INFO:teuthology.orchestra.run.smithi192.stdout:Setting up linux-headers-generic (5.15.0.101.98) ... 2024-04-06T09:17:42.304 INFO:teuthology.orchestra.run.smithi192.stdout:Setting up linux-image-5.15.0-101-generic (5.15.0-101.111) ... 2024-04-06T09:17:42.784 INFO:teuthology.orchestra.run.smithi006.stdout:Preparing to unpack .../7-linux-headers-generic_5.15.0.101.98_amd64.deb ... 2024-04-06T09:17:42.917 INFO:teuthology.orchestra.run.smithi006.stdout:Unpacking linux-headers-generic (5.15.0.101.98) over (5.15.0.56.54) ... 2024-04-06T09:17:43.318 INFO:teuthology.orchestra.run.smithi006.stdout:Setting up linux-headers-5.15.0-101 (5.15.0-101.111) ... 2024-04-06T09:17:43.430 INFO:teuthology.orchestra.run.smithi006.stdout:Setting up linux-headers-5.15.0-101-generic (5.15.0-101.111) ... 2024-04-06T09:17:43.585 INFO:teuthology.orchestra.run.smithi192.stdout:I: /vmlinuz.old is now a symlink to boot/vmlinuz-5.15.0-56-generic 2024-04-06T09:17:43.585 INFO:teuthology.orchestra.run.smithi192.stdout:I: /initrd.img.old is now a symlink to boot/initrd.img-5.15.0-56-generic 2024-04-06T09:17:43.585 INFO:teuthology.orchestra.run.smithi192.stdout:I: /vmlinuz is now a symlink to boot/vmlinuz-5.15.0-101-generic 2024-04-06T09:17:43.585 INFO:teuthology.orchestra.run.smithi192.stdout:I: /initrd.img is now a symlink to boot/initrd.img-5.15.0-101-generic 2024-04-06T09:17:43.589 INFO:teuthology.orchestra.run.smithi006.stdout:Setting up linux-headers-generic (5.15.0.101.98) ... 2024-04-06T09:17:43.719 INFO:teuthology.orchestra.run.smithi006.stdout:Setting up linux-image-5.15.0-101-generic (5.15.0-101.111) ... 2024-04-06T09:17:43.765 INFO:teuthology.orchestra.run.smithi192.stdout:Setting up linux-modules-extra-5.15.0-101-generic (5.15.0-101.111) ... 2024-04-06T09:17:45.267 INFO:teuthology.orchestra.run.smithi006.stdout:I: /vmlinuz.old is now a symlink to boot/vmlinuz-5.15.0-56-generic 2024-04-06T09:17:45.267 INFO:teuthology.orchestra.run.smithi006.stdout:I: /initrd.img.old is now a symlink to boot/initrd.img-5.15.0-56-generic 2024-04-06T09:17:45.267 INFO:teuthology.orchestra.run.smithi006.stdout:I: /vmlinuz is now a symlink to boot/vmlinuz-5.15.0-101-generic 2024-04-06T09:17:45.267 INFO:teuthology.orchestra.run.smithi006.stdout:I: /initrd.img is now a symlink to boot/initrd.img-5.15.0-101-generic 2024-04-06T09:17:45.764 INFO:teuthology.orchestra.run.smithi006.stdout:Setting up linux-modules-extra-5.15.0-101-generic (5.15.0-101.111) ... 2024-04-06T09:17:45.916 INFO:teuthology.orchestra.run.smithi192.stdout:Setting up linux-modules-5.15.0-101-generic (5.15.0-101.111) ... 2024-04-06T09:17:47.205 INFO:teuthology.orchestra.run.smithi006.stdout:Setting up linux-modules-5.15.0-101-generic (5.15.0-101.111) ... 2024-04-06T09:17:47.326 INFO:teuthology.orchestra.run.smithi192.stdout:Setting up linux-image-generic (5.15.0.101.98) ... 2024-04-06T09:17:47.451 INFO:teuthology.orchestra.run.smithi192.stdout:Setting up linux-generic (5.15.0.101.98) ... 2024-04-06T09:17:47.569 INFO:teuthology.orchestra.run.smithi192.stdout:Processing triggers for linux-image-5.15.0-101-generic (5.15.0-101.111) ... 2024-04-06T09:17:47.626 INFO:teuthology.orchestra.run.smithi192.stdout:/etc/kernel/postinst.d/initramfs-tools: 2024-04-06T09:17:47.626 INFO:teuthology.orchestra.run.smithi192.stdout:update-initramfs: Generating /boot/initrd.img-5.15.0-101-generic 2024-04-06T09:17:48.640 INFO:teuthology.orchestra.run.smithi006.stdout:Setting up linux-image-generic (5.15.0.101.98) ... 2024-04-06T09:17:48.766 INFO:teuthology.orchestra.run.smithi006.stdout:Setting up linux-generic (5.15.0.101.98) ... 2024-04-06T09:17:48.934 INFO:teuthology.orchestra.run.smithi006.stdout:Processing triggers for linux-image-5.15.0-101-generic (5.15.0-101.111) ... 2024-04-06T09:17:49.010 INFO:teuthology.orchestra.run.smithi006.stdout:/etc/kernel/postinst.d/initramfs-tools: 2024-04-06T09:17:49.010 INFO:teuthology.orchestra.run.smithi006.stdout:update-initramfs: Generating /boot/initrd.img-5.15.0-101-generic 2024-04-06T09:18:01.891 INFO:teuthology.orchestra.run.smithi192.stdout:/etc/kernel/postinst.d/zz-update-grub: 2024-04-06T09:18:01.891 INFO:teuthology.orchestra.run.smithi192.stdout:Sourcing file `/etc/default/grub' 2024-04-06T09:18:01.910 INFO:teuthology.orchestra.run.smithi192.stdout:Sourcing file `/etc/default/grub.d/init-select.cfg' 2024-04-06T09:18:01.924 INFO:teuthology.orchestra.run.smithi192.stdout:Generating grub configuration file ... 2024-04-06T09:18:02.162 INFO:teuthology.orchestra.run.smithi192.stdout:Found linux image: /boot/vmlinuz-5.15.0-101-generic 2024-04-06T09:18:02.176 INFO:teuthology.orchestra.run.smithi192.stdout:Found initrd image: /boot/initrd.img-5.15.0-101-generic 2024-04-06T09:18:02.487 INFO:teuthology.orchestra.run.smithi192.stdout:Found linux image: /boot/vmlinuz-5.15.0-56-generic 2024-04-06T09:18:02.492 INFO:teuthology.orchestra.run.smithi192.stdout:Found initrd image: /boot/initrd.img-5.15.0-56-generic 2024-04-06T09:18:02.552 INFO:teuthology.orchestra.run.smithi192.stdout:Found linux image: /boot/vmlinuz-5.4.0-135-generic 2024-04-06T09:18:02.557 INFO:teuthology.orchestra.run.smithi192.stdout:Found initrd image: /boot/initrd.img-5.4.0-135-generic 2024-04-06T09:18:02.714 INFO:teuthology.orchestra.run.smithi192.stdout:Warning: os-prober will not be executed to detect other bootable partitions. 2024-04-06T09:18:02.714 INFO:teuthology.orchestra.run.smithi192.stdout:Systems on them will not be added to the GRUB boot configuration. 2024-04-06T09:18:02.714 INFO:teuthology.orchestra.run.smithi192.stdout:Check GRUB_DISABLE_OS_PROBER documentation entry. 2024-04-06T09:18:02.764 INFO:teuthology.orchestra.run.smithi192.stdout:done 2024-04-06T09:18:03.239 DEBUG:teuthology.orchestra.run.smithi192:> dpkg -s linux-image-generic 2024-04-06T09:18:03.260 INFO:teuthology.orchestra.run.smithi192.stdout:Package: linux-image-generic 2024-04-06T09:18:03.260 INFO:teuthology.orchestra.run.smithi192.stdout:Status: install ok installed 2024-04-06T09:18:03.260 INFO:teuthology.orchestra.run.smithi192.stdout:Priority: optional 2024-04-06T09:18:03.260 INFO:teuthology.orchestra.run.smithi192.stdout:Section: kernel 2024-04-06T09:18:03.260 INFO:teuthology.orchestra.run.smithi192.stdout:Installed-Size: 21 2024-04-06T09:18:03.260 INFO:teuthology.orchestra.run.smithi192.stdout:Maintainer: Ubuntu Kernel Team 2024-04-06T09:18:03.260 INFO:teuthology.orchestra.run.smithi192.stdout:Architecture: amd64 2024-04-06T09:18:03.260 INFO:teuthology.orchestra.run.smithi192.stdout:Source: linux-meta 2024-04-06T09:18:03.260 INFO:teuthology.orchestra.run.smithi192.stdout:Version: 5.15.0.101.98 2024-04-06T09:18:03.260 INFO:teuthology.orchestra.run.smithi192.stdout:Provides: spl-modules (= 2.1.5-1ubuntu6~22.04.3), v4l2loopback-modules (= 0.12.7-2ubuntu2~22.04.1), virtualbox-guest-modules (= 5.15.0-101), wireguard-modules (= 1.0.0), zfs-modules (= 2.1.5-1ubuntu6~22.04.3) 2024-04-06T09:18:03.260 INFO:teuthology.orchestra.run.smithi192.stdout:Depends: linux-image-5.15.0-101-generic, linux-modules-extra-5.15.0-101-generic, linux-firmware, intel-microcode, amd64-microcode 2024-04-06T09:18:03.261 INFO:teuthology.orchestra.run.smithi192.stdout:Recommends: thermald 2024-04-06T09:18:03.261 INFO:teuthology.orchestra.run.smithi192.stdout:Description: Generic Linux kernel image 2024-04-06T09:18:03.261 INFO:teuthology.orchestra.run.smithi192.stdout: This package will always depend on the latest generic kernel image 2024-04-06T09:18:03.261 INFO:teuthology.orchestra.run.smithi192.stdout: available. 2024-04-06T09:18:03.261 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.15.0-56-generic Expected: 5.15.0-101-generic 2024-04-06T09:18:03.261 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-04-06T09:18:03.261 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-04-06T09:18:03.261 INFO:teuthology.task.kernel:Installing distro kernel on cluster2.mon.a... 2024-04-06T09:18:03.261 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi192.front.sepia.ceph.com, path=None, version=distro) 2024-04-06T09:18:03.262 DEBUG:teuthology.orchestra.run.smithi192:> sudo apt-get clean 2024-04-06T09:18:03.483 DEBUG:teuthology.orchestra.run.smithi192:> sudo apt-get update 2024-04-06T09:18:03.621 INFO:teuthology.orchestra.run.smithi192.stdout:Hit:1 http://security.ubuntu.com/ubuntu jammy-security InRelease 2024-04-06T09:18:03.724 INFO:teuthology.orchestra.run.smithi192.stdout:Hit:2 http://archive.ubuntu.com/ubuntu jammy InRelease 2024-04-06T09:18:03.804 INFO:teuthology.orchestra.run.smithi192.stdout:Hit:3 http://archive.ubuntu.com/ubuntu jammy-updates InRelease 2024-04-06T09:18:03.887 INFO:teuthology.orchestra.run.smithi192.stdout:Hit:4 http://archive.ubuntu.com/ubuntu jammy-backports InRelease 2024-04-06T09:18:04.208 INFO:teuthology.orchestra.run.smithi006.stdout:/etc/kernel/postinst.d/zz-update-grub: 2024-04-06T09:18:04.209 INFO:teuthology.orchestra.run.smithi006.stdout:Sourcing file `/etc/default/grub' 2024-04-06T09:18:04.242 INFO:teuthology.orchestra.run.smithi006.stdout:Sourcing file `/etc/default/grub.d/init-select.cfg' 2024-04-06T09:18:04.267 INFO:teuthology.orchestra.run.smithi006.stdout:Generating grub configuration file ... 2024-04-06T09:18:04.663 INFO:teuthology.orchestra.run.smithi006.stdout:Found linux image: /boot/vmlinuz-5.15.0-101-generic 2024-04-06T09:18:04.682 INFO:teuthology.orchestra.run.smithi006.stdout:Found initrd image: /boot/initrd.img-5.15.0-101-generic 2024-04-06T09:18:05.045 INFO:teuthology.orchestra.run.smithi192.stdout:Reading package lists... 2024-04-06T09:18:05.062 DEBUG:teuthology.orchestra.run.smithi192:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2024-04-06T09:18:05.115 INFO:teuthology.orchestra.run.smithi192.stdout:Reading package lists... 2024-04-06T09:18:05.149 INFO:teuthology.orchestra.run.smithi006.stdout:Found linux image: /boot/vmlinuz-5.15.0-56-generic 2024-04-06T09:18:05.155 INFO:teuthology.orchestra.run.smithi006.stdout:Found initrd image: /boot/initrd.img-5.15.0-56-generic 2024-04-06T09:18:05.227 INFO:teuthology.orchestra.run.smithi006.stdout:Found linux image: /boot/vmlinuz-5.4.0-135-generic 2024-04-06T09:18:05.233 INFO:teuthology.orchestra.run.smithi006.stdout:Found initrd image: /boot/initrd.img-5.4.0-135-generic 2024-04-06T09:18:05.291 INFO:teuthology.orchestra.run.smithi192.stdout:Building dependency tree... 2024-04-06T09:18:05.292 INFO:teuthology.orchestra.run.smithi192.stdout:Reading state information... 2024-04-06T09:18:05.387 INFO:teuthology.orchestra.run.smithi006.stdout:Warning: os-prober will not be executed to detect other bootable partitions. 2024-04-06T09:18:05.387 INFO:teuthology.orchestra.run.smithi006.stdout:Systems on them will not be added to the GRUB boot configuration. 2024-04-06T09:18:05.387 INFO:teuthology.orchestra.run.smithi006.stdout:Check GRUB_DISABLE_OS_PROBER documentation entry. 2024-04-06T09:18:05.442 INFO:teuthology.orchestra.run.smithi006.stdout:done 2024-04-06T09:18:05.444 INFO:teuthology.orchestra.run.smithi192.stdout:linux-image-generic is already the newest version (5.15.0.101.98). 2024-04-06T09:18:05.444 INFO:teuthology.orchestra.run.smithi192.stdout:The following packages were automatically installed and are no longer required: 2024-04-06T09:18:05.444 INFO:teuthology.orchestra.run.smithi192.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2024-04-06T09:18:05.444 INFO:teuthology.orchestra.run.smithi192.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2024-04-06T09:18:05.444 INFO:teuthology.orchestra.run.smithi192.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2024-04-06T09:18:05.445 INFO:teuthology.orchestra.run.smithi192.stdout: linux-image-5.4.0-135-generic linux-modules-5.4.0-135-generic python2 2024-04-06T09:18:05.445 INFO:teuthology.orchestra.run.smithi192.stdout: python2-dev python2-minimal python2.7 python2.7-dev python2.7-minimal 2024-04-06T09:18:05.445 INFO:teuthology.orchestra.run.smithi192.stdout:Use 'sudo apt autoremove' to remove them. 2024-04-06T09:18:05.483 INFO:teuthology.orchestra.run.smithi192.stdout:0 upgraded, 0 newly installed, 0 to remove and 312 not upgraded. 2024-04-06T09:18:05.485 DEBUG:teuthology.orchestra.run.smithi192:> dpkg -s linux-image-generic 2024-04-06T09:18:05.539 INFO:teuthology.orchestra.run.smithi192.stdout:Package: linux-image-generic 2024-04-06T09:18:05.539 INFO:teuthology.orchestra.run.smithi192.stdout:Status: install ok installed 2024-04-06T09:18:05.539 INFO:teuthology.orchestra.run.smithi192.stdout:Priority: optional 2024-04-06T09:18:05.539 INFO:teuthology.orchestra.run.smithi192.stdout:Section: kernel 2024-04-06T09:18:05.539 INFO:teuthology.orchestra.run.smithi192.stdout:Installed-Size: 21 2024-04-06T09:18:05.539 INFO:teuthology.orchestra.run.smithi192.stdout:Maintainer: Ubuntu Kernel Team 2024-04-06T09:18:05.539 INFO:teuthology.orchestra.run.smithi192.stdout:Architecture: amd64 2024-04-06T09:18:05.539 INFO:teuthology.orchestra.run.smithi192.stdout:Source: linux-meta 2024-04-06T09:18:05.539 INFO:teuthology.orchestra.run.smithi192.stdout:Version: 5.15.0.101.98 2024-04-06T09:18:05.539 INFO:teuthology.orchestra.run.smithi192.stdout:Provides: spl-modules (= 2.1.5-1ubuntu6~22.04.3), v4l2loopback-modules (= 0.12.7-2ubuntu2~22.04.1), virtualbox-guest-modules (= 5.15.0-101), wireguard-modules (= 1.0.0), zfs-modules (= 2.1.5-1ubuntu6~22.04.3) 2024-04-06T09:18:05.539 INFO:teuthology.orchestra.run.smithi192.stdout:Depends: linux-image-5.15.0-101-generic, linux-modules-extra-5.15.0-101-generic, linux-firmware, intel-microcode, amd64-microcode 2024-04-06T09:18:05.539 INFO:teuthology.orchestra.run.smithi192.stdout:Recommends: thermald 2024-04-06T09:18:05.540 INFO:teuthology.orchestra.run.smithi192.stdout:Description: Generic Linux kernel image 2024-04-06T09:18:05.540 INFO:teuthology.orchestra.run.smithi192.stdout: This package will always depend on the latest generic kernel image 2024-04-06T09:18:05.540 INFO:teuthology.orchestra.run.smithi192.stdout: available. 2024-04-06T09:18:05.540 DEBUG:teuthology.orchestra.run.smithi192:> mktemp 2024-04-06T09:18:05.586 INFO:teuthology.orchestra.run.smithi192.stdout:/tmp/tmp.KiL9Z0PqQR 2024-04-06T09:18:05.587 DEBUG:teuthology.orchestra.run.smithi192:> sudo cp /boot/grub/grub.cfg /tmp/tmp.KiL9Z0PqQR 2024-04-06T09:18:05.637 DEBUG:teuthology.orchestra.run.smithi192:> sudo chmod 0666 /tmp/tmp.KiL9Z0PqQR 2024-04-06T09:18:05.760 DEBUG:teuthology.orchestra.remote:smithi192:/tmp/tmp.KiL9Z0PqQR is 10KB 2024-04-06T09:18:05.771 DEBUG:teuthology.orchestra.run.smithi192:> rm -fr /tmp/tmp.KiL9Z0PqQR 2024-04-06T09:18:05.775 DEBUG:teuthology.orchestra.run.smithi192:> sudo rm -f -- /etc/grub.d/01_ceph_kernel 2024-04-06T09:18:05.827 DEBUG:teuthology.orchestra.run.smithi192:> set -ex 2024-04-06T09:18:05.827 DEBUG:teuthology.orchestra.run.smithi192:> sudo dd of=/etc/grub.d/01_ceph_kernel 2024-04-06T09:18:05.827 DEBUG:teuthology.orchestra.run.smithi192:> sudo chmod 755 /etc/grub.d/01_ceph_kernel 2024-04-06T09:18:05.891 INFO:teuthology.task.kernel:Distro Kernel Version: 5.15.0-101-generic 2024-04-06T09:18:05.891 DEBUG:teuthology.orchestra.run.smithi192:> sudo update-grub 2024-04-06T09:18:05.963 DEBUG:teuthology.orchestra.run.smithi006:> dpkg -s linux-image-generic 2024-04-06T09:18:05.984 INFO:teuthology.orchestra.run.smithi006.stdout:Package: linux-image-generic 2024-04-06T09:18:05.984 INFO:teuthology.orchestra.run.smithi006.stdout:Status: install ok installed 2024-04-06T09:18:05.985 INFO:teuthology.orchestra.run.smithi006.stdout:Priority: optional 2024-04-06T09:18:05.985 INFO:teuthology.orchestra.run.smithi006.stdout:Section: kernel 2024-04-06T09:18:05.985 INFO:teuthology.orchestra.run.smithi006.stdout:Installed-Size: 21 2024-04-06T09:18:05.985 INFO:teuthology.orchestra.run.smithi006.stdout:Maintainer: Ubuntu Kernel Team 2024-04-06T09:18:05.985 INFO:teuthology.orchestra.run.smithi006.stdout:Architecture: amd64 2024-04-06T09:18:05.985 INFO:teuthology.orchestra.run.smithi006.stdout:Source: linux-meta 2024-04-06T09:18:05.985 INFO:teuthology.orchestra.run.smithi006.stdout:Version: 5.15.0.101.98 2024-04-06T09:18:05.985 INFO:teuthology.orchestra.run.smithi006.stdout:Provides: spl-modules (= 2.1.5-1ubuntu6~22.04.3), v4l2loopback-modules (= 0.12.7-2ubuntu2~22.04.1), virtualbox-guest-modules (= 5.15.0-101), wireguard-modules (= 1.0.0), zfs-modules (= 2.1.5-1ubuntu6~22.04.3) 2024-04-06T09:18:05.985 INFO:teuthology.orchestra.run.smithi006.stdout:Depends: linux-image-5.15.0-101-generic, linux-modules-extra-5.15.0-101-generic, linux-firmware, intel-microcode, amd64-microcode 2024-04-06T09:18:05.985 INFO:teuthology.orchestra.run.smithi006.stdout:Recommends: thermald 2024-04-06T09:18:05.985 INFO:teuthology.orchestra.run.smithi006.stdout:Description: Generic Linux kernel image 2024-04-06T09:18:05.985 INFO:teuthology.orchestra.run.smithi006.stdout: This package will always depend on the latest generic kernel image 2024-04-06T09:18:05.985 INFO:teuthology.orchestra.run.smithi006.stdout: available. 2024-04-06T09:18:05.986 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.15.0-56-generic Expected: 5.15.0-101-generic 2024-04-06T09:18:05.986 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-04-06T09:18:05.986 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-04-06T09:18:05.987 INFO:teuthology.task.kernel:Installing distro kernel on cluster1.mon.a... 2024-04-06T09:18:05.987 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi006.front.sepia.ceph.com, path=None, version=distro) 2024-04-06T09:18:05.987 DEBUG:teuthology.orchestra.run.smithi006:> sudo apt-get clean 2024-04-06T09:18:06.232 DEBUG:teuthology.orchestra.run.smithi006:> sudo apt-get update 2024-04-06T09:18:06.365 INFO:teuthology.orchestra.run.smithi006.stdout:Hit:1 http://archive.ubuntu.com/ubuntu jammy InRelease 2024-04-06T09:18:06.378 INFO:teuthology.orchestra.run.smithi006.stdout:Hit:2 http://archive.ubuntu.com/ubuntu jammy-updates InRelease 2024-04-06T09:18:06.381 INFO:teuthology.orchestra.run.smithi006.stdout:Hit:3 http://security.ubuntu.com/ubuntu jammy-security InRelease 2024-04-06T09:18:06.404 INFO:teuthology.orchestra.run.smithi006.stdout:Hit:4 http://archive.ubuntu.com/ubuntu jammy-backports InRelease 2024-04-06T09:18:07.087 INFO:teuthology.orchestra.run.smithi192.stderr:Sourcing file `/etc/default/grub' 2024-04-06T09:18:07.089 INFO:teuthology.orchestra.run.smithi192.stderr:Sourcing file `/etc/default/grub.d/init-select.cfg' 2024-04-06T09:18:07.091 INFO:teuthology.orchestra.run.smithi192.stderr:Generating grub configuration file ... 2024-04-06T09:18:07.300 INFO:teuthology.orchestra.run.smithi192.stderr:Found linux image: /boot/vmlinuz-5.15.0-101-generic 2024-04-06T09:18:07.307 INFO:teuthology.orchestra.run.smithi192.stderr:Found initrd image: /boot/initrd.img-5.15.0-101-generic 2024-04-06T09:18:07.523 INFO:teuthology.orchestra.run.smithi192.stderr:Found linux image: /boot/vmlinuz-5.15.0-56-generic 2024-04-06T09:18:07.529 INFO:teuthology.orchestra.run.smithi192.stderr:Found initrd image: /boot/initrd.img-5.15.0-56-generic 2024-04-06T09:18:07.575 INFO:teuthology.orchestra.run.smithi192.stderr:Found linux image: /boot/vmlinuz-5.4.0-135-generic 2024-04-06T09:18:07.580 INFO:teuthology.orchestra.run.smithi192.stderr:Found initrd image: /boot/initrd.img-5.4.0-135-generic 2024-04-06T09:18:07.701 INFO:teuthology.orchestra.run.smithi192.stderr:Warning: os-prober will not be executed to detect other bootable partitions. 2024-04-06T09:18:07.701 INFO:teuthology.orchestra.run.smithi192.stderr:Systems on them will not be added to the GRUB boot configuration. 2024-04-06T09:18:07.701 INFO:teuthology.orchestra.run.smithi192.stderr:Check GRUB_DISABLE_OS_PROBER documentation entry. 2024-04-06T09:18:07.715 INFO:teuthology.orchestra.run.smithi192.stderr:done 2024-04-06T09:18:07.723 DEBUG:teuthology.orchestra.run.smithi192:> sudo shutdown -r now 2024-04-06T09:18:07.885 INFO:teuthology.orchestra.run.smithi006.stdout:Reading package lists... 2024-04-06T09:18:07.903 DEBUG:teuthology.orchestra.run.smithi006:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2024-04-06T09:18:07.965 INFO:teuthology.orchestra.run.smithi006.stdout:Reading package lists... 2024-04-06T09:18:08.177 INFO:teuthology.orchestra.run.smithi006.stdout:Building dependency tree... 2024-04-06T09:18:08.177 INFO:teuthology.orchestra.run.smithi006.stdout:Reading state information... 2024-04-06T09:18:08.358 INFO:teuthology.orchestra.run.smithi006.stdout:linux-image-generic is already the newest version (5.15.0.101.98). 2024-04-06T09:18:08.358 INFO:teuthology.orchestra.run.smithi006.stdout:The following packages were automatically installed and are no longer required: 2024-04-06T09:18:08.358 INFO:teuthology.orchestra.run.smithi006.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2024-04-06T09:18:08.358 INFO:teuthology.orchestra.run.smithi006.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2024-04-06T09:18:08.358 INFO:teuthology.orchestra.run.smithi006.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2024-04-06T09:18:08.358 INFO:teuthology.orchestra.run.smithi006.stdout: linux-image-5.4.0-135-generic linux-modules-5.4.0-135-generic python2 2024-04-06T09:18:08.359 INFO:teuthology.orchestra.run.smithi006.stdout: python2-dev python2-minimal python2.7 python2.7-dev python2.7-minimal 2024-04-06T09:18:08.359 INFO:teuthology.orchestra.run.smithi006.stdout:Use 'sudo apt autoremove' to remove them. 2024-04-06T09:18:08.399 INFO:teuthology.orchestra.run.smithi006.stdout:0 upgraded, 0 newly installed, 0 to remove and 312 not upgraded. 2024-04-06T09:18:08.400 DEBUG:teuthology.orchestra.run.smithi006:> dpkg -s linux-image-generic 2024-04-06T09:18:08.412 INFO:teuthology.orchestra.run.smithi006.stdout:Package: linux-image-generic 2024-04-06T09:18:08.413 INFO:teuthology.orchestra.run.smithi006.stdout:Status: install ok installed 2024-04-06T09:18:08.413 INFO:teuthology.orchestra.run.smithi006.stdout:Priority: optional 2024-04-06T09:18:08.413 INFO:teuthology.orchestra.run.smithi006.stdout:Section: kernel 2024-04-06T09:18:08.413 INFO:teuthology.orchestra.run.smithi006.stdout:Installed-Size: 21 2024-04-06T09:18:08.413 INFO:teuthology.orchestra.run.smithi006.stdout:Maintainer: Ubuntu Kernel Team 2024-04-06T09:18:08.413 INFO:teuthology.orchestra.run.smithi006.stdout:Architecture: amd64 2024-04-06T09:18:08.413 INFO:teuthology.orchestra.run.smithi006.stdout:Source: linux-meta 2024-04-06T09:18:08.413 INFO:teuthology.orchestra.run.smithi006.stdout:Version: 5.15.0.101.98 2024-04-06T09:18:08.413 INFO:teuthology.orchestra.run.smithi006.stdout:Provides: spl-modules (= 2.1.5-1ubuntu6~22.04.3), v4l2loopback-modules (= 0.12.7-2ubuntu2~22.04.1), virtualbox-guest-modules (= 5.15.0-101), wireguard-modules (= 1.0.0), zfs-modules (= 2.1.5-1ubuntu6~22.04.3) 2024-04-06T09:18:08.413 INFO:teuthology.orchestra.run.smithi006.stdout:Depends: linux-image-5.15.0-101-generic, linux-modules-extra-5.15.0-101-generic, linux-firmware, intel-microcode, amd64-microcode 2024-04-06T09:18:08.413 INFO:teuthology.orchestra.run.smithi006.stdout:Recommends: thermald 2024-04-06T09:18:08.413 INFO:teuthology.orchestra.run.smithi006.stdout:Description: Generic Linux kernel image 2024-04-06T09:18:08.414 INFO:teuthology.orchestra.run.smithi006.stdout: This package will always depend on the latest generic kernel image 2024-04-06T09:18:08.414 INFO:teuthology.orchestra.run.smithi006.stdout: available. 2024-04-06T09:18:08.414 DEBUG:teuthology.orchestra.run.smithi006:> mktemp 2024-04-06T09:18:08.462 INFO:teuthology.orchestra.run.smithi006.stdout:/tmp/tmp.TOpUEssOwv 2024-04-06T09:18:08.463 DEBUG:teuthology.orchestra.run.smithi006:> sudo cp /boot/grub/grub.cfg /tmp/tmp.TOpUEssOwv 2024-04-06T09:18:08.523 DEBUG:teuthology.orchestra.run.smithi006:> sudo chmod 0666 /tmp/tmp.TOpUEssOwv 2024-04-06T09:18:08.635 DEBUG:teuthology.orchestra.remote:smithi006:/tmp/tmp.TOpUEssOwv is 10KB 2024-04-06T09:18:08.648 DEBUG:teuthology.orchestra.run.smithi006:> rm -fr /tmp/tmp.TOpUEssOwv 2024-04-06T09:18:08.653 DEBUG:teuthology.orchestra.run.smithi006:> sudo rm -f -- /etc/grub.d/01_ceph_kernel 2024-04-06T09:18:08.707 DEBUG:teuthology.orchestra.run.smithi006:> set -ex 2024-04-06T09:18:08.707 DEBUG:teuthology.orchestra.run.smithi006:> sudo dd of=/etc/grub.d/01_ceph_kernel 2024-04-06T09:18:08.707 DEBUG:teuthology.orchestra.run.smithi006:> sudo chmod 755 /etc/grub.d/01_ceph_kernel 2024-04-06T09:18:08.919 INFO:teuthology.task.kernel:Distro Kernel Version: 5.15.0-101-generic 2024-04-06T09:18:08.919 DEBUG:teuthology.orchestra.run.smithi006:> sudo update-grub 2024-04-06T09:18:10.315 INFO:teuthology.orchestra.run.smithi006.stderr:Sourcing file `/etc/default/grub' 2024-04-06T09:18:10.316 INFO:teuthology.orchestra.run.smithi006.stderr:Sourcing file `/etc/default/grub.d/init-select.cfg' 2024-04-06T09:18:10.319 INFO:teuthology.orchestra.run.smithi006.stderr:Generating grub configuration file ... 2024-04-06T09:18:10.606 INFO:teuthology.orchestra.run.smithi006.stderr:Found linux image: /boot/vmlinuz-5.15.0-101-generic 2024-04-06T09:18:10.613 INFO:teuthology.orchestra.run.smithi006.stderr:Found initrd image: /boot/initrd.img-5.15.0-101-generic 2024-04-06T09:18:11.017 INFO:teuthology.orchestra.run.smithi006.stderr:Found linux image: /boot/vmlinuz-5.15.0-56-generic 2024-04-06T09:18:11.022 INFO:teuthology.orchestra.run.smithi006.stderr:Found initrd image: /boot/initrd.img-5.15.0-56-generic 2024-04-06T09:18:11.069 INFO:teuthology.orchestra.run.smithi006.stderr:Found linux image: /boot/vmlinuz-5.4.0-135-generic 2024-04-06T09:18:11.074 INFO:teuthology.orchestra.run.smithi006.stderr:Found initrd image: /boot/initrd.img-5.4.0-135-generic 2024-04-06T09:18:11.313 INFO:teuthology.orchestra.run.smithi006.stderr:Warning: os-prober will not be executed to detect other bootable partitions. 2024-04-06T09:18:11.313 INFO:teuthology.orchestra.run.smithi006.stderr:Systems on them will not be added to the GRUB boot configuration. 2024-04-06T09:18:11.313 INFO:teuthology.orchestra.run.smithi006.stderr:Check GRUB_DISABLE_OS_PROBER documentation entry. 2024-04-06T09:18:11.333 INFO:teuthology.orchestra.run.smithi006.stderr:done 2024-04-06T09:18:11.335 DEBUG:teuthology.orchestra.run.smithi006:> sudo shutdown -r now 2024-04-06T09:18:37.736 INFO:teuthology.task.kernel:Checking client cluster2.mon.a for new kernel version... 2024-04-06T09:18:37.737 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi192.front.sepia.ceph.com' 2024-04-06T09:18:37.738 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi192.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-06T09:18:41.386 INFO:teuthology.task.kernel:Checking client cluster1.mon.a for new kernel version... 2024-04-06T09:18:41.386 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi006.front.sepia.ceph.com' 2024-04-06T09:18:41.387 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi006.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-06T09:18:56.287 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.192 2024-04-06T09:18:59.870 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.6 2024-04-06T09:19:05.290 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi192.front.sepia.ceph.com' 2024-04-06T09:19:05.291 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi192.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-06T09:19:08.350 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.192 2024-04-06T09:19:08.872 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi006.front.sepia.ceph.com' 2024-04-06T09:19:08.873 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi006.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-06T09:19:11.934 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.6 2024-04-06T09:19:20.354 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi192.front.sepia.ceph.com' 2024-04-06T09:19:20.354 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi192.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-06T09:19:23.938 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi006.front.sepia.ceph.com' 2024-04-06T09:19:23.939 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi006.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-06T09:19:38.786 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.192 2024-04-06T09:19:53.802 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi192.front.sepia.ceph.com' 2024-04-06T09:19:53.803 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi192.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-06T09:19:54.070 DEBUG:teuthology.orchestra.run.smithi192:> true 2024-04-06T09:19:54.691 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi192.front.sepia.ceph.com' 2024-04-06T09:19:54.692 INFO:teuthology.task.kernel:Checking kernel version of cluster2.mon.a, want "5.15.0-101-generic"... 2024-04-06T09:19:54.692 DEBUG:teuthology.orchestra.run.smithi192:> uname -r 2024-04-06T09:19:54.738 INFO:teuthology.orchestra.run.smithi192.stdout:5.15.0-101-generic 2024-04-06T09:19:54.738 DEBUG:teuthology.task.kernel:current kernel version is 5.15.0-101-generic vs 5.15.0-101-generic 2024-04-06T09:19:54.738 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-04-06T09:19:54.738 DEBUG:teuthology.task.kernel:Distro of this test job: ubuntu 2024-04-06T09:19:55.739 INFO:teuthology.task.kernel:Enabling kdb on cluster2.mon.a... 2024-04-06T09:19:55.740 DEBUG:teuthology.orchestra.run.smithi192:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-04-06T09:19:55.746 DEBUG:teuthology.orchestra.run.smithi006:> true 2024-04-06T09:19:55.809 INFO:teuthology.orchestra.run.smithi192.stdout:ttyS1 2024-04-06T09:19:55.832 DEBUG:teuthology.parallel:result is None 2024-04-06T09:19:56.836 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi006.front.sepia.ceph.com' 2024-04-06T09:19:56.836 INFO:teuthology.task.kernel:Checking kernel version of cluster1.mon.a, want "5.15.0-101-generic"... 2024-04-06T09:19:56.836 DEBUG:teuthology.orchestra.run.smithi006:> uname -r 2024-04-06T09:19:56.842 INFO:teuthology.orchestra.run.smithi006.stdout:5.15.0-101-generic 2024-04-06T09:19:56.842 DEBUG:teuthology.task.kernel:current kernel version is 5.15.0-101-generic vs 5.15.0-101-generic 2024-04-06T09:19:56.842 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-04-06T09:19:56.842 DEBUG:teuthology.task.kernel:Distro of this test job: ubuntu 2024-04-06T09:19:57.843 INFO:teuthology.task.kernel:Enabling kdb on cluster1.mon.a... 2024-04-06T09:19:57.844 DEBUG:teuthology.orchestra.run.smithi006:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-04-06T09:19:57.927 INFO:teuthology.orchestra.run.smithi006.stdout:ttyS1 2024-04-06T09:19:57.953 DEBUG:teuthology.parallel:result is None 2024-04-06T09:19:57.953 INFO:teuthology.run_tasks:Running task internal.base... 2024-04-06T09:19:57.960 INFO:teuthology.task.internal:Creating test directory... 2024-04-06T09:19:57.961 DEBUG:teuthology.orchestra.run.smithi006:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-04-06T09:19:57.963 DEBUG:teuthology.orchestra.run.smithi192:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-04-06T09:19:57.970 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-04-06T09:19:57.977 INFO:teuthology.run_tasks:Running task internal.archive... 2024-04-06T09:19:57.986 INFO:teuthology.task.internal:Creating archive directory... 2024-04-06T09:19:57.986 DEBUG:teuthology.orchestra.run.smithi006:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-04-06T09:19:58.013 DEBUG:teuthology.orchestra.run.smithi192:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-04-06T09:19:58.067 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-04-06T09:19:58.074 INFO:teuthology.task.internal:Enabling coredump saving... 2024-04-06T09:19:58.075 DEBUG:teuthology.orchestra.run.smithi006:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-04-06T09:19:58.077 DEBUG:teuthology.orchestra.run.smithi192:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-04-06T09:19:58.098 INFO:teuthology.orchestra.run.smithi006.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-06T09:19:58.101 INFO:teuthology.orchestra.run.smithi192.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-06T09:19:58.106 INFO:teuthology.orchestra.run.smithi006.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-06T09:19:58.110 INFO:teuthology.orchestra.run.smithi192.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-06T09:19:58.111 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-04-06T09:19:58.118 INFO:teuthology.task.internal:Configuring sudo... 2024-04-06T09:19:58.118 DEBUG:teuthology.orchestra.run.smithi006:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-04-06T09:19:58.153 DEBUG:teuthology.orchestra.run.smithi192:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-04-06T09:19:58.174 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-04-06T09:19:58.183 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-04-06T09:19:58.183 DEBUG:teuthology.orchestra.run.smithi006:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-04-06T09:19:58.213 DEBUG:teuthology.orchestra.run.smithi192:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-04-06T09:19:58.223 DEBUG:teuthology.orchestra.run.smithi006:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-06T09:19:58.264 DEBUG:teuthology.orchestra.run.smithi006:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-06T09:19:58.312 DEBUG:teuthology.orchestra.run.smithi006:> set -ex 2024-04-06T09:19:58.312 DEBUG:teuthology.orchestra.run.smithi006:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-04-06T09:19:58.372 DEBUG:teuthology.orchestra.run.smithi192:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-06T09:19:58.378 DEBUG:teuthology.orchestra.run.smithi192:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-06T09:19:58.427 DEBUG:teuthology.orchestra.run.smithi192:> set -ex 2024-04-06T09:19:58.427 DEBUG:teuthology.orchestra.run.smithi192:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-04-06T09:19:58.490 DEBUG:teuthology.orchestra.run.smithi006:> sudo service rsyslog restart 2024-04-06T09:19:58.493 DEBUG:teuthology.orchestra.run.smithi192:> sudo service rsyslog restart 2024-04-06T09:19:58.566 INFO:teuthology.run_tasks:Running task internal.timer... 2024-04-06T09:19:58.574 INFO:teuthology.task.internal:Starting timer... 2024-04-06T09:19:58.574 INFO:teuthology.run_tasks:Running task pcp... 2024-04-06T09:19:58.584 INFO:teuthology.run_tasks:Running task selinux... 2024-04-06T09:19:58.592 DEBUG:teuthology.task.selinux:Excluding smithi006: OS 'ubuntu' does not support SELinux 2024-04-06T09:19:58.592 DEBUG:teuthology.task.selinux:Excluding smithi192: OS 'ubuntu' does not support SELinux 2024-04-06T09:19:58.593 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-04-06T09:19:58.593 DEBUG:teuthology.task.selinux:Existing SELinux modes: {} 2024-04-06T09:19:58.593 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-04-06T09:19:58.593 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-04-06T09:19:58.602 DEBUG:teuthology.repo_utils:Setting repo remote to https://git.ceph.com/ceph-cm-ansible.git 2024-04-06T09:19:58.607 INFO:teuthology.repo_utils:Fetching git.ceph.com_ceph-cm-ansible_main from origin 2024-04-06T09:19:58.717 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-04-06T09:19:58.738 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-04-06T09:19:58.739 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi006.front.sepia.ceph.com,smithi192.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-04-06T09:27:00.436 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi006.front.sepia.ceph.com'), Remote(name='ubuntu@smithi192.front.sepia.ceph.com')] 2024-04-06T09:27:00.437 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi006.front.sepia.ceph.com' 2024-04-06T09:27:00.439 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi006.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-06T09:27:00.523 DEBUG:teuthology.orchestra.run.smithi006:> true 2024-04-06T09:27:00.602 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi006.front.sepia.ceph.com' 2024-04-06T09:27:00.603 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi192.front.sepia.ceph.com' 2024-04-06T09:27:00.603 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi192.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-06T09:27:00.681 DEBUG:teuthology.orchestra.run.smithi192:> true 2024-04-06T09:27:00.758 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi192.front.sepia.ceph.com' 2024-04-06T09:27:00.758 INFO:teuthology.run_tasks:Running task clock... 2024-04-06T09:27:00.775 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-04-06T09:27:00.775 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-04-06T09:27:00.776 DEBUG:teuthology.orchestra.run.smithi006:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-06T09:27:00.779 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-04-06T09:27:00.779 DEBUG:teuthology.orchestra.run.smithi192:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-06T09:27:00.804 INFO:teuthology.orchestra.run.smithi006.stdout: 6 Apr 09:27:00 ntpd[17065]: ntpd 4.2.8p15@1.3728-o Wed Feb 16 17:13:02 UTC 2022 (1): Starting 2024-04-06T09:27:00.804 INFO:teuthology.orchestra.run.smithi006.stdout: 6 Apr 09:27:00 ntpd[17065]: Command line: ntpd -gq 2024-04-06T09:27:00.804 INFO:teuthology.orchestra.run.smithi006.stdout: 6 Apr 09:27:00 ntpd[17065]: ---------------------------------------------------- 2024-04-06T09:27:00.805 INFO:teuthology.orchestra.run.smithi006.stdout: 6 Apr 09:27:00 ntpd[17065]: ntp-4 is maintained by Network Time Foundation, 2024-04-06T09:27:00.805 INFO:teuthology.orchestra.run.smithi006.stdout: 6 Apr 09:27:00 ntpd[17065]: Inc. (NTF), a non-profit 501(c)(3) public-benefit 2024-04-06T09:27:00.805 INFO:teuthology.orchestra.run.smithi006.stdout: 6 Apr 09:27:00 ntpd[17065]: corporation. Support and training for ntp-4 are 2024-04-06T09:27:00.805 INFO:teuthology.orchestra.run.smithi006.stdout: 6 Apr 09:27:00 ntpd[17065]: available at https://www.nwtime.org/support 2024-04-06T09:27:00.805 INFO:teuthology.orchestra.run.smithi006.stdout: 6 Apr 09:27:00 ntpd[17065]: ---------------------------------------------------- 2024-04-06T09:27:00.805 INFO:teuthology.orchestra.run.smithi006.stdout: 6 Apr 09:27:00 ntpd[17065]: proto: precision = 0.046 usec (-24) 2024-04-06T09:27:00.806 INFO:teuthology.orchestra.run.smithi006.stdout: 6 Apr 09:27:00 ntpd[17065]: basedate set to 2022-02-04 2024-04-06T09:27:00.806 INFO:teuthology.orchestra.run.smithi006.stdout: 6 Apr 09:27:00 ntpd[17065]: gps base set to 2022-02-06 (week 2196) 2024-04-06T09:27:00.806 INFO:teuthology.orchestra.run.smithi006.stdout: 6 Apr 09:27:00 ntpd[17065]: restrict 0.0.0.0: KOD does nothing without LIMITED. 2024-04-06T09:27:00.806 INFO:teuthology.orchestra.run.smithi006.stderr:restrict 0.0.0.0: KOD does nothing without LIMITED. 2024-04-06T09:27:00.806 INFO:teuthology.orchestra.run.smithi006.stderr:restrict ::: KOD does nothing without LIMITED. 2024-04-06T09:27:00.807 INFO:teuthology.orchestra.run.smithi006.stdout: 6 Apr 09:27:00 ntpd[17065]: restrict ::: KOD does nothing without LIMITED. 2024-04-06T09:27:00.807 INFO:teuthology.orchestra.run.smithi006.stdout: 6 Apr 09:27:00 ntpd[17065]: Listen and drop on 0 v6wildcard [::]:123 2024-04-06T09:27:00.807 INFO:teuthology.orchestra.run.smithi006.stdout: 6 Apr 09:27:00 ntpd[17065]: Listen and drop on 1 v4wildcard 0.0.0.0:123 2024-04-06T09:27:00.807 INFO:teuthology.orchestra.run.smithi006.stdout: 6 Apr 09:27:00 ntpd[17065]: Listen normally on 2 lo 127.0.0.1:123 2024-04-06T09:27:00.808 INFO:teuthology.orchestra.run.smithi006.stdout: 6 Apr 09:27:00 ntpd[17065]: Listen normally on 3 ens1f0 172.21.15.6:123 2024-04-06T09:27:00.808 INFO:teuthology.orchestra.run.smithi006.stdout: 6 Apr 09:27:00 ntpd[17065]: Listen normally on 4 lo [::1]:123 2024-04-06T09:27:00.808 INFO:teuthology.orchestra.run.smithi006.stdout: 6 Apr 09:27:00 ntpd[17065]: Listen normally on 5 ens1f0 [fe80::ec4:7aff:febd:15e4%4]:123 2024-04-06T09:27:00.808 INFO:teuthology.orchestra.run.smithi006.stdout: 6 Apr 09:27:00 ntpd[17065]: Listening on routing socket on fd #22 for interface updates 2024-04-06T09:27:00.825 INFO:teuthology.orchestra.run.smithi192.stdout: 6 Apr 09:27:00 ntpd[17166]: ntpd 4.2.8p15@1.3728-o Wed Feb 16 17:13:02 UTC 2022 (1): Starting 2024-04-06T09:27:00.825 INFO:teuthology.orchestra.run.smithi192.stdout: 6 Apr 09:27:00 ntpd[17166]: Command line: ntpd -gq 2024-04-06T09:27:00.825 INFO:teuthology.orchestra.run.smithi192.stdout: 6 Apr 09:27:00 ntpd[17166]: ---------------------------------------------------- 2024-04-06T09:27:00.825 INFO:teuthology.orchestra.run.smithi192.stdout: 6 Apr 09:27:00 ntpd[17166]: ntp-4 is maintained by Network Time Foundation, 2024-04-06T09:27:00.825 INFO:teuthology.orchestra.run.smithi192.stdout: 6 Apr 09:27:00 ntpd[17166]: Inc. (NTF), a non-profit 501(c)(3) public-benefit 2024-04-06T09:27:00.826 INFO:teuthology.orchestra.run.smithi192.stdout: 6 Apr 09:27:00 ntpd[17166]: corporation. Support and training for ntp-4 are 2024-04-06T09:27:00.826 INFO:teuthology.orchestra.run.smithi192.stdout: 6 Apr 09:27:00 ntpd[17166]: available at https://www.nwtime.org/support 2024-04-06T09:27:00.826 INFO:teuthology.orchestra.run.smithi192.stdout: 6 Apr 09:27:00 ntpd[17166]: ---------------------------------------------------- 2024-04-06T09:27:00.826 INFO:teuthology.orchestra.run.smithi192.stdout: 6 Apr 09:27:00 ntpd[17166]: proto: precision = 0.045 usec (-24) 2024-04-06T09:27:00.827 INFO:teuthology.orchestra.run.smithi192.stdout: 6 Apr 09:27:00 ntpd[17166]: basedate set to 2022-02-04 2024-04-06T09:27:00.827 INFO:teuthology.orchestra.run.smithi192.stdout: 6 Apr 09:27:00 ntpd[17166]: gps base set to 2022-02-06 (week 2196) 2024-04-06T09:27:00.827 INFO:teuthology.orchestra.run.smithi192.stderr:restrict 0.0.0.0: KOD does nothing without LIMITED. 2024-04-06T09:27:00.827 INFO:teuthology.orchestra.run.smithi192.stdout: 6 Apr 09:27:00 ntpd[17166]: restrict 0.0.0.0: KOD does nothing without LIMITED. 2024-04-06T09:27:00.827 INFO:teuthology.orchestra.run.smithi192.stdout: 6 Apr 09:27:00 ntpd[17166]: restrict ::: KOD does nothing without LIMITED. 2024-04-06T09:27:00.828 INFO:teuthology.orchestra.run.smithi192.stderr:restrict ::: KOD does nothing without LIMITED. 2024-04-06T09:27:00.829 INFO:teuthology.orchestra.run.smithi192.stdout: 6 Apr 09:27:00 ntpd[17166]: Listen and drop on 0 v6wildcard [::]:123 2024-04-06T09:27:00.829 INFO:teuthology.orchestra.run.smithi192.stdout: 6 Apr 09:27:00 ntpd[17166]: Listen and drop on 1 v4wildcard 0.0.0.0:123 2024-04-06T09:27:00.829 INFO:teuthology.orchestra.run.smithi192.stdout: 6 Apr 09:27:00 ntpd[17166]: Listen normally on 2 lo 127.0.0.1:123 2024-04-06T09:27:00.829 INFO:teuthology.orchestra.run.smithi192.stdout: 6 Apr 09:27:00 ntpd[17166]: Listen normally on 3 enp3s0f1 172.21.15.192:123 2024-04-06T09:27:00.829 INFO:teuthology.orchestra.run.smithi192.stdout: 6 Apr 09:27:00 ntpd[17166]: Listen normally on 4 lo [::1]:123 2024-04-06T09:27:00.829 INFO:teuthology.orchestra.run.smithi192.stdout: 6 Apr 09:27:00 ntpd[17166]: Listen normally on 5 enp3s0f1 [fe80::ec4:7aff:fe88:8049%5]:123 2024-04-06T09:27:00.829 INFO:teuthology.orchestra.run.smithi192.stdout: 6 Apr 09:27:00 ntpd[17166]: Listening on routing socket on fd #22 for interface updates 2024-04-06T09:27:01.826 INFO:teuthology.orchestra.run.smithi192.stderr: 6 Apr 09:27:01 ntpd[17166]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-06T09:27:01.826 INFO:teuthology.orchestra.run.smithi192.stderr: 6 Apr 09:27:01 ntpd[17166]: can't open /var/log/ntpstats/rawstats.20240406: Permission denied 2024-04-06T09:27:01.826 INFO:teuthology.orchestra.run.smithi192.stderr: 6 Apr 09:27:01 ntpd[17166]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-06T09:27:01.827 INFO:teuthology.orchestra.run.smithi192.stderr: 6 Apr 09:27:01 ntpd[17166]: can't open /var/log/ntpstats/rawstats.20240406: Permission denied 2024-04-06T09:27:01.827 INFO:teuthology.orchestra.run.smithi192.stderr: 6 Apr 09:27:01 ntpd[17166]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-06T09:27:01.827 INFO:teuthology.orchestra.run.smithi192.stderr: 6 Apr 09:27:01 ntpd[17166]: can't open /var/log/ntpstats/rawstats.20240406: Permission denied 2024-04-06T09:27:01.827 INFO:teuthology.orchestra.run.smithi192.stderr: 6 Apr 09:27:01 ntpd[17166]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-04-06T09:27:01.827 INFO:teuthology.orchestra.run.smithi192.stderr: 6 Apr 09:27:01 ntpd[17166]: can't open /var/log/ntpstats/peerstats.20240406: Permission denied 2024-04-06T09:27:02.805 INFO:teuthology.orchestra.run.smithi006.stderr: 6 Apr 09:27:02 ntpd[17065]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-06T09:27:02.805 INFO:teuthology.orchestra.run.smithi006.stderr: 6 Apr 09:27:02 ntpd[17065]: can't open /var/log/ntpstats/rawstats.20240406: Permission denied 2024-04-06T09:27:02.806 INFO:teuthology.orchestra.run.smithi006.stderr: 6 Apr 09:27:02 ntpd[17065]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-06T09:27:02.806 INFO:teuthology.orchestra.run.smithi006.stderr: 6 Apr 09:27:02 ntpd[17065]: can't open /var/log/ntpstats/rawstats.20240406: Permission denied 2024-04-06T09:27:02.806 INFO:teuthology.orchestra.run.smithi006.stderr: 6 Apr 09:27:02 ntpd[17065]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-04-06T09:27:02.806 INFO:teuthology.orchestra.run.smithi006.stderr: 6 Apr 09:27:02 ntpd[17065]: can't open /var/log/ntpstats/peerstats.20240406: Permission denied 2024-04-06T09:27:03.826 INFO:teuthology.orchestra.run.smithi192.stderr: 6 Apr 09:27:03 ntpd[17166]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-06T09:27:03.827 INFO:teuthology.orchestra.run.smithi192.stderr: 6 Apr 09:27:03 ntpd[17166]: can't open /var/log/ntpstats/rawstats.20240406: Permission denied 2024-04-06T09:27:03.827 INFO:teuthology.orchestra.run.smithi192.stderr: 6 Apr 09:27:03 ntpd[17166]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-06T09:27:03.827 INFO:teuthology.orchestra.run.smithi192.stderr: 6 Apr 09:27:03 ntpd[17166]: can't open /var/log/ntpstats/rawstats.20240406: Permission denied 2024-04-06T09:27:03.827 INFO:teuthology.orchestra.run.smithi192.stderr: 6 Apr 09:27:03 ntpd[17166]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-06T09:27:03.827 INFO:teuthology.orchestra.run.smithi192.stderr: 6 Apr 09:27:03 ntpd[17166]: can't open /var/log/ntpstats/rawstats.20240406: Permission denied 2024-04-06T09:27:03.827 INFO:teuthology.orchestra.run.smithi192.stderr: 6 Apr 09:27:03 ntpd[17166]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-04-06T09:27:03.827 INFO:teuthology.orchestra.run.smithi192.stderr: 6 Apr 09:27:03 ntpd[17166]: can't open /var/log/ntpstats/peerstats.20240406: Permission denied 2024-04-06T09:27:03.827 INFO:teuthology.orchestra.run.smithi192.stderr: 6 Apr 09:27:03 ntpd[17166]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-06T09:27:03.827 INFO:teuthology.orchestra.run.smithi192.stderr: 6 Apr 09:27:03 ntpd[17166]: can't open /var/log/ntpstats/rawstats.20240406: Permission denied 2024-04-06T09:27:03.827 INFO:teuthology.orchestra.run.smithi192.stderr: 6 Apr 09:27:03 ntpd[17166]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-04-06T09:27:03.827 INFO:teuthology.orchestra.run.smithi192.stderr: 6 Apr 09:27:03 ntpd[17166]: can't open /var/log/ntpstats/peerstats.20240406: Permission denied 2024-04-06T09:27:04.807 INFO:teuthology.orchestra.run.smithi006.stderr: 6 Apr 09:27:04 ntpd[17065]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-06T09:27:04.807 INFO:teuthology.orchestra.run.smithi006.stderr: 6 Apr 09:27:04 ntpd[17065]: can't open /var/log/ntpstats/rawstats.20240406: Permission denied 2024-04-06T09:27:04.807 INFO:teuthology.orchestra.run.smithi006.stderr: 6 Apr 09:27:04 ntpd[17065]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-06T09:27:04.807 INFO:teuthology.orchestra.run.smithi006.stderr: 6 Apr 09:27:04 ntpd[17065]: can't open /var/log/ntpstats/rawstats.20240406: Permission denied 2024-04-06T09:27:04.807 INFO:teuthology.orchestra.run.smithi006.stderr: 6 Apr 09:27:04 ntpd[17065]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-06T09:27:04.807 INFO:teuthology.orchestra.run.smithi006.stderr: 6 Apr 09:27:04 ntpd[17065]: can't open /var/log/ntpstats/rawstats.20240406: Permission denied 2024-04-06T09:27:04.807 INFO:teuthology.orchestra.run.smithi006.stderr: 6 Apr 09:27:04 ntpd[17065]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-06T09:27:04.807 INFO:teuthology.orchestra.run.smithi006.stderr: 6 Apr 09:27:04 ntpd[17065]: can't open /var/log/ntpstats/rawstats.20240406: Permission denied 2024-04-06T09:27:04.807 INFO:teuthology.orchestra.run.smithi006.stderr: 6 Apr 09:27:04 ntpd[17065]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-06T09:27:04.807 INFO:teuthology.orchestra.run.smithi006.stderr: 6 Apr 09:27:04 ntpd[17065]: can't open /var/log/ntpstats/rawstats.20240406: Permission denied 2024-04-06T09:27:04.807 INFO:teuthology.orchestra.run.smithi006.stderr: 6 Apr 09:27:04 ntpd[17065]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-04-06T09:27:04.807 INFO:teuthology.orchestra.run.smithi006.stderr: 6 Apr 09:27:04 ntpd[17065]: can't open /var/log/ntpstats/peerstats.20240406: Permission denied 2024-04-06T09:27:04.808 INFO:teuthology.orchestra.run.smithi006.stderr: 6 Apr 09:27:04 ntpd[17065]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-06T09:27:04.808 INFO:teuthology.orchestra.run.smithi006.stderr: 6 Apr 09:27:04 ntpd[17065]: can't open /var/log/ntpstats/rawstats.20240406: Permission denied 2024-04-06T09:27:04.808 INFO:teuthology.orchestra.run.smithi006.stderr: 6 Apr 09:27:04 ntpd[17065]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-04-06T09:27:04.808 INFO:teuthology.orchestra.run.smithi006.stderr: 6 Apr 09:27:04 ntpd[17065]: can't open /var/log/ntpstats/peerstats.20240406: Permission denied 2024-04-06T09:27:04.808 INFO:teuthology.orchestra.run.smithi006.stderr: 6 Apr 09:27:04 ntpd[17065]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-06T09:27:04.808 INFO:teuthology.orchestra.run.smithi006.stderr: 6 Apr 09:27:04 ntpd[17065]: can't open /var/log/ntpstats/rawstats.20240406: Permission denied 2024-04-06T09:27:04.808 INFO:teuthology.orchestra.run.smithi006.stderr: 6 Apr 09:27:04 ntpd[17065]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-04-06T09:27:04.808 INFO:teuthology.orchestra.run.smithi006.stderr: 6 Apr 09:27:04 ntpd[17065]: can't open /var/log/ntpstats/peerstats.20240406: Permission denied 2024-04-06T09:27:05.827 INFO:teuthology.orchestra.run.smithi192.stderr: 6 Apr 09:27:05 ntpd[17166]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-06T09:27:05.827 INFO:teuthology.orchestra.run.smithi192.stderr: 6 Apr 09:27:05 ntpd[17166]: can't open /var/log/ntpstats/rawstats.20240406: Permission denied 2024-04-06T09:27:05.827 INFO:teuthology.orchestra.run.smithi192.stderr: 6 Apr 09:27:05 ntpd[17166]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-06T09:27:05.827 INFO:teuthology.orchestra.run.smithi192.stderr: 6 Apr 09:27:05 ntpd[17166]: can't open /var/log/ntpstats/rawstats.20240406: Permission denied 2024-04-06T09:27:05.827 INFO:teuthology.orchestra.run.smithi192.stderr: 6 Apr 09:27:05 ntpd[17166]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-06T09:27:05.827 INFO:teuthology.orchestra.run.smithi192.stderr: 6 Apr 09:27:05 ntpd[17166]: can't open /var/log/ntpstats/rawstats.20240406: Permission denied 2024-04-06T09:27:05.827 INFO:teuthology.orchestra.run.smithi192.stderr: 6 Apr 09:27:05 ntpd[17166]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-06T09:27:05.828 INFO:teuthology.orchestra.run.smithi192.stderr: 6 Apr 09:27:05 ntpd[17166]: can't open /var/log/ntpstats/rawstats.20240406: Permission denied 2024-04-06T09:27:05.828 INFO:teuthology.orchestra.run.smithi192.stderr: 6 Apr 09:27:05 ntpd[17166]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-04-06T09:27:05.828 INFO:teuthology.orchestra.run.smithi192.stderr: 6 Apr 09:27:05 ntpd[17166]: can't open /var/log/ntpstats/peerstats.20240406: Permission denied 2024-04-06T09:27:05.828 INFO:teuthology.orchestra.run.smithi192.stderr: 6 Apr 09:27:05 ntpd[17166]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-06T09:27:05.828 INFO:teuthology.orchestra.run.smithi192.stderr: 6 Apr 09:27:05 ntpd[17166]: can't open /var/log/ntpstats/rawstats.20240406: Permission denied 2024-04-06T09:27:05.828 INFO:teuthology.orchestra.run.smithi192.stderr: 6 Apr 09:27:05 ntpd[17166]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-04-06T09:27:05.828 INFO:teuthology.orchestra.run.smithi192.stderr: 6 Apr 09:27:05 ntpd[17166]: can't open /var/log/ntpstats/peerstats.20240406: Permission denied 2024-04-06T09:27:05.828 INFO:teuthology.orchestra.run.smithi192.stderr: 6 Apr 09:27:05 ntpd[17166]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-06T09:27:05.828 INFO:teuthology.orchestra.run.smithi192.stderr: 6 Apr 09:27:05 ntpd[17166]: can't open /var/log/ntpstats/rawstats.20240406: Permission denied 2024-04-06T09:27:05.828 INFO:teuthology.orchestra.run.smithi192.stderr: 6 Apr 09:27:05 ntpd[17166]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-04-06T09:27:05.828 INFO:teuthology.orchestra.run.smithi192.stderr: 6 Apr 09:27:05 ntpd[17166]: can't open /var/log/ntpstats/peerstats.20240406: Permission denied 2024-04-06T09:27:06.806 INFO:teuthology.orchestra.run.smithi006.stderr: 6 Apr 09:27:06 ntpd[17065]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-06T09:27:06.806 INFO:teuthology.orchestra.run.smithi006.stderr: 6 Apr 09:27:06 ntpd[17065]: can't open /var/log/ntpstats/rawstats.20240406: Permission denied 2024-04-06T09:27:06.807 INFO:teuthology.orchestra.run.smithi006.stderr: 6 Apr 09:27:06 ntpd[17065]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-06T09:27:06.807 INFO:teuthology.orchestra.run.smithi006.stderr: 6 Apr 09:27:06 ntpd[17065]: can't open /var/log/ntpstats/rawstats.20240406: Permission denied 2024-04-06T09:27:06.807 INFO:teuthology.orchestra.run.smithi006.stderr: 6 Apr 09:27:06 ntpd[17065]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-06T09:27:06.807 INFO:teuthology.orchestra.run.smithi006.stderr: 6 Apr 09:27:06 ntpd[17065]: can't open /var/log/ntpstats/rawstats.20240406: Permission denied 2024-04-06T09:27:06.807 INFO:teuthology.orchestra.run.smithi006.stderr: 6 Apr 09:27:06 ntpd[17065]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-06T09:27:06.807 INFO:teuthology.orchestra.run.smithi006.stderr: 6 Apr 09:27:06 ntpd[17065]: can't open /var/log/ntpstats/rawstats.20240406: Permission denied 2024-04-06T09:27:06.807 INFO:teuthology.orchestra.run.smithi006.stderr: 6 Apr 09:27:06 ntpd[17065]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-04-06T09:27:06.807 INFO:teuthology.orchestra.run.smithi006.stderr: 6 Apr 09:27:06 ntpd[17065]: can't open /var/log/ntpstats/peerstats.20240406: Permission denied 2024-04-06T09:27:06.807 INFO:teuthology.orchestra.run.smithi006.stderr: 6 Apr 09:27:06 ntpd[17065]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-06T09:27:06.807 INFO:teuthology.orchestra.run.smithi006.stderr: 6 Apr 09:27:06 ntpd[17065]: can't open /var/log/ntpstats/rawstats.20240406: Permission denied 2024-04-06T09:27:06.807 INFO:teuthology.orchestra.run.smithi006.stderr: 6 Apr 09:27:06 ntpd[17065]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-04-06T09:27:06.807 INFO:teuthology.orchestra.run.smithi006.stderr: 6 Apr 09:27:06 ntpd[17065]: can't open /var/log/ntpstats/peerstats.20240406: Permission denied 2024-04-06T09:27:06.807 INFO:teuthology.orchestra.run.smithi006.stderr: 6 Apr 09:27:06 ntpd[17065]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-06T09:27:06.807 INFO:teuthology.orchestra.run.smithi006.stderr: 6 Apr 09:27:06 ntpd[17065]: can't open /var/log/ntpstats/rawstats.20240406: Permission denied 2024-04-06T09:27:06.807 INFO:teuthology.orchestra.run.smithi006.stderr: 6 Apr 09:27:06 ntpd[17065]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-04-06T09:27:06.808 INFO:teuthology.orchestra.run.smithi006.stderr: 6 Apr 09:27:06 ntpd[17065]: can't open /var/log/ntpstats/peerstats.20240406: Permission denied 2024-04-06T09:27:07.827 INFO:teuthology.orchestra.run.smithi192.stderr: 6 Apr 09:27:07 ntpd[17166]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-06T09:27:07.827 INFO:teuthology.orchestra.run.smithi192.stderr: 6 Apr 09:27:07 ntpd[17166]: can't open /var/log/ntpstats/rawstats.20240406: Permission denied 2024-04-06T09:27:07.827 INFO:teuthology.orchestra.run.smithi192.stderr: 6 Apr 09:27:07 ntpd[17166]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-06T09:27:07.827 INFO:teuthology.orchestra.run.smithi192.stderr: 6 Apr 09:27:07 ntpd[17166]: can't open /var/log/ntpstats/rawstats.20240406: Permission denied 2024-04-06T09:27:07.827 INFO:teuthology.orchestra.run.smithi192.stderr: 6 Apr 09:27:07 ntpd[17166]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-06T09:27:07.827 INFO:teuthology.orchestra.run.smithi192.stderr: 6 Apr 09:27:07 ntpd[17166]: can't open /var/log/ntpstats/rawstats.20240406: Permission denied 2024-04-06T09:27:07.827 INFO:teuthology.orchestra.run.smithi192.stderr: 6 Apr 09:27:07 ntpd[17166]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-06T09:27:07.827 INFO:teuthology.orchestra.run.smithi192.stderr: 6 Apr 09:27:07 ntpd[17166]: can't open /var/log/ntpstats/rawstats.20240406: Permission denied 2024-04-06T09:27:07.827 INFO:teuthology.orchestra.run.smithi192.stderr: 6 Apr 09:27:07 ntpd[17166]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-04-06T09:27:07.828 INFO:teuthology.orchestra.run.smithi192.stderr: 6 Apr 09:27:07 ntpd[17166]: can't open /var/log/ntpstats/peerstats.20240406: Permission denied 2024-04-06T09:27:07.828 INFO:teuthology.orchestra.run.smithi192.stderr: 6 Apr 09:27:07 ntpd[17166]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-06T09:27:07.828 INFO:teuthology.orchestra.run.smithi192.stderr: 6 Apr 09:27:07 ntpd[17166]: can't open /var/log/ntpstats/rawstats.20240406: Permission denied 2024-04-06T09:27:07.828 INFO:teuthology.orchestra.run.smithi192.stderr: 6 Apr 09:27:07 ntpd[17166]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-04-06T09:27:07.828 INFO:teuthology.orchestra.run.smithi192.stderr: 6 Apr 09:27:07 ntpd[17166]: can't open /var/log/ntpstats/peerstats.20240406: Permission denied 2024-04-06T09:27:07.828 INFO:teuthology.orchestra.run.smithi192.stderr: 6 Apr 09:27:07 ntpd[17166]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-06T09:27:07.828 INFO:teuthology.orchestra.run.smithi192.stderr: 6 Apr 09:27:07 ntpd[17166]: can't open /var/log/ntpstats/rawstats.20240406: Permission denied 2024-04-06T09:27:07.828 INFO:teuthology.orchestra.run.smithi192.stderr: 6 Apr 09:27:07 ntpd[17166]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-04-06T09:27:07.828 INFO:teuthology.orchestra.run.smithi192.stderr: 6 Apr 09:27:07 ntpd[17166]: can't open /var/log/ntpstats/peerstats.20240406: Permission denied 2024-04-06T09:27:07.828 INFO:teuthology.orchestra.run.smithi192.stdout: 6 Apr 09:27:07 ntpd[17166]: ntpd: time slew -0.001288 s 2024-04-06T09:27:07.828 INFO:teuthology.orchestra.run.smithi192.stdout:ntpd: time slew -0.001288s 2024-04-06T09:27:07.829 INFO:teuthology.orchestra.run.smithi192.stderr: 6 Apr 09:27:07 ntpd[17166]: couldn't unlink /var/log/ntpstats/loopstats: Permission denied 2024-04-06T09:27:07.829 INFO:teuthology.orchestra.run.smithi192.stderr: 6 Apr 09:27:07 ntpd[17166]: can't open /var/log/ntpstats/loopstats.20240406: Permission denied 2024-04-06T09:27:07.886 INFO:teuthology.orchestra.run.smithi192.stdout: remote refid st t when poll reach delay offset jitter 2024-04-06T09:27:07.886 INFO:teuthology.orchestra.run.smithi192.stdout:============================================================================== 2024-04-06T09:27:07.886 INFO:teuthology.orchestra.run.smithi192.stdout: hv01.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-04-06T09:27:07.886 INFO:teuthology.orchestra.run.smithi192.stdout: hv02.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-04-06T09:27:07.886 INFO:teuthology.orchestra.run.smithi192.stdout: hv03.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-04-06T09:27:07.886 INFO:teuthology.orchestra.run.smithi192.stdout: hv04.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-04-06T09:27:08.805 INFO:teuthology.orchestra.run.smithi006.stderr: 6 Apr 09:27:08 ntpd[17065]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-06T09:27:08.806 INFO:teuthology.orchestra.run.smithi006.stderr: 6 Apr 09:27:08 ntpd[17065]: can't open /var/log/ntpstats/rawstats.20240406: Permission denied 2024-04-06T09:27:08.806 INFO:teuthology.orchestra.run.smithi006.stderr: 6 Apr 09:27:08 ntpd[17065]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-06T09:27:08.806 INFO:teuthology.orchestra.run.smithi006.stderr: 6 Apr 09:27:08 ntpd[17065]: can't open /var/log/ntpstats/rawstats.20240406: Permission denied 2024-04-06T09:27:08.806 INFO:teuthology.orchestra.run.smithi006.stderr: 6 Apr 09:27:08 ntpd[17065]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-06T09:27:08.806 INFO:teuthology.orchestra.run.smithi006.stderr: 6 Apr 09:27:08 ntpd[17065]: can't open /var/log/ntpstats/rawstats.20240406: Permission denied 2024-04-06T09:27:08.806 INFO:teuthology.orchestra.run.smithi006.stderr: 6 Apr 09:27:08 ntpd[17065]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-06T09:27:08.806 INFO:teuthology.orchestra.run.smithi006.stderr: 6 Apr 09:27:08 ntpd[17065]: can't open /var/log/ntpstats/rawstats.20240406: Permission denied 2024-04-06T09:27:08.806 INFO:teuthology.orchestra.run.smithi006.stderr: 6 Apr 09:27:08 ntpd[17065]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-04-06T09:27:08.806 INFO:teuthology.orchestra.run.smithi006.stderr: 6 Apr 09:27:08 ntpd[17065]: can't open /var/log/ntpstats/peerstats.20240406: Permission denied 2024-04-06T09:27:08.807 INFO:teuthology.orchestra.run.smithi006.stdout: 6 Apr 09:27:08 ntpd[17065]: ntpd: time slew -0.000721 s 2024-04-06T09:27:08.807 INFO:teuthology.orchestra.run.smithi006.stdout:ntpd: time slew -0.000721s 2024-04-06T09:27:08.807 INFO:teuthology.orchestra.run.smithi006.stderr: 6 Apr 09:27:08 ntpd[17065]: couldn't unlink /var/log/ntpstats/loopstats: Permission denied 2024-04-06T09:27:08.807 INFO:teuthology.orchestra.run.smithi006.stderr: 6 Apr 09:27:08 ntpd[17065]: can't open /var/log/ntpstats/loopstats.20240406: Permission denied 2024-04-06T09:27:08.854 INFO:teuthology.orchestra.run.smithi006.stdout: remote refid st t when poll reach delay offset jitter 2024-04-06T09:27:08.854 INFO:teuthology.orchestra.run.smithi006.stdout:============================================================================== 2024-04-06T09:27:08.854 INFO:teuthology.orchestra.run.smithi006.stdout: hv01.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-04-06T09:27:08.854 INFO:teuthology.orchestra.run.smithi006.stdout: hv02.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-04-06T09:27:08.854 INFO:teuthology.orchestra.run.smithi006.stdout: hv03.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-04-06T09:27:08.854 INFO:teuthology.orchestra.run.smithi006.stdout: hv04.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-04-06T09:27:08.854 INFO:teuthology.run_tasks:Running task install... 2024-04-06T09:27:08.864 DEBUG:teuthology.task.install:project ceph 2024-04-06T09:27:08.864 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': '4d05f4cad464de0fcdd11a2856643e533aea34e8'}} 2024-04-06T09:27:08.864 DEBUG:teuthology.task.install:config {'extra_packages': ['rbd-mirror'], 'flavor': 'default', 'sha1': '4d05f4cad464de0fcdd11a2856643e533aea34e8'} 2024-04-06T09:27:08.864 INFO:teuthology.task.install:Using flavor: default 2024-04-06T09:27:08.870 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2024-04-06T09:27:08.870 INFO:teuthology.task.install:extra packages: ['rbd-mirror'] 2024-04-06T09:27:08.871 DEBUG:teuthology.orchestra.run.smithi006:> sudo apt-key list | grep Ceph 2024-04-06T09:27:08.871 DEBUG:teuthology.orchestra.run.smithi192:> sudo apt-key list | grep Ceph 2024-04-06T09:27:08.937 INFO:teuthology.orchestra.run.smithi192.stderr:Warning: apt-key is deprecated. Manage keyring files in trusted.gpg.d instead (see apt-key(8)). 2024-04-06T09:27:08.958 INFO:teuthology.orchestra.run.smithi006.stderr:Warning: apt-key is deprecated. Manage keyring files in trusted.gpg.d instead (see apt-key(8)). 2024-04-06T09:27:08.987 INFO:teuthology.orchestra.run.smithi006.stdout:uid [ unknown] Ceph automated package build (Ceph automated package build) 2024-04-06T09:27:08.987 INFO:teuthology.orchestra.run.smithi006.stdout:uid [ unknown] Ceph.com (release key) 2024-04-06T09:27:08.988 INFO:teuthology.task.install.deb:Installing packages: ceph, cephadm, ceph-mds, ceph-mgr, ceph-common, ceph-fuse, ceph-test, ceph-volume, radosgw, python3-rados, python3-rgw, python3-cephfs, python3-rbd, libcephfs2, libcephfs-dev, librados2, librbd1, rbd-fuse, rbd-mirror on remote deb x86_64 2024-04-06T09:27:08.989 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&sha1=4d05f4cad464de0fcdd11a2856643e533aea34e8 2024-04-06T09:27:08.994 INFO:teuthology.orchestra.run.smithi192.stdout:uid [ unknown] Ceph automated package build (Ceph automated package build) 2024-04-06T09:27:08.994 INFO:teuthology.orchestra.run.smithi192.stdout:uid [ unknown] Ceph.com (release key) 2024-04-06T09:27:08.995 INFO:teuthology.task.install.deb:Installing packages: ceph, cephadm, ceph-mds, ceph-mgr, ceph-common, ceph-fuse, ceph-test, ceph-volume, radosgw, python3-rados, python3-rgw, python3-cephfs, python3-rbd, libcephfs2, libcephfs-dev, librados2, librbd1, rbd-fuse, rbd-mirror on remote deb x86_64 2024-04-06T09:27:08.995 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&sha1=4d05f4cad464de0fcdd11a2856643e533aea34e8 2024-04-06T09:27:09.134 INFO:teuthology.task.install.deb:Pulling from https://3.chacra.ceph.com/r/ceph/reef/4d05f4cad464de0fcdd11a2856643e533aea34e8/ubuntu/jammy/flavors/default/ 2024-04-06T09:27:09.134 INFO:teuthology.task.install.deb:Package version is 18.2.2-1059-g4d05f4ca-1jammy 2024-04-06T09:27:09.209 INFO:teuthology.task.install.deb:Pulling from https://3.chacra.ceph.com/r/ceph/reef/4d05f4cad464de0fcdd11a2856643e533aea34e8/ubuntu/jammy/flavors/default/ 2024-04-06T09:27:09.209 INFO:teuthology.task.install.deb:Package version is 18.2.2-1059-g4d05f4ca-1jammy 2024-04-06T09:27:09.312 DEBUG:teuthology.orchestra.run.smithi006:> set -ex 2024-04-06T09:27:09.312 DEBUG:teuthology.orchestra.run.smithi006:> sudo dd of=/etc/apt/sources.list.d/ceph.list 2024-04-06T09:27:09.327 DEBUG:teuthology.orchestra.run.smithi006:> sudo apt-get update 2024-04-06T09:27:09.389 DEBUG:teuthology.orchestra.run.smithi192:> set -ex 2024-04-06T09:27:09.389 DEBUG:teuthology.orchestra.run.smithi192:> sudo dd of=/etc/apt/sources.list.d/ceph.list 2024-04-06T09:27:09.404 DEBUG:teuthology.orchestra.run.smithi192:> sudo apt-get update 2024-04-06T09:27:09.505 INFO:teuthology.orchestra.run.smithi006.stdout:Hit:1 http://archive.ubuntu.com/ubuntu jammy InRelease 2024-04-06T09:27:09.515 INFO:teuthology.orchestra.run.smithi006.stdout:Hit:2 http://archive.ubuntu.com/ubuntu jammy-updates InRelease 2024-04-06T09:27:09.535 INFO:teuthology.orchestra.run.smithi006.stdout:Hit:3 http://archive.ubuntu.com/ubuntu jammy-backports InRelease 2024-04-06T09:27:09.578 INFO:teuthology.orchestra.run.smithi192.stdout:Hit:1 http://archive.ubuntu.com/ubuntu jammy InRelease 2024-04-06T09:27:09.578 INFO:teuthology.orchestra.run.smithi192.stdout:Hit:2 http://security.ubuntu.com/ubuntu jammy-security InRelease 2024-04-06T09:27:09.593 INFO:teuthology.orchestra.run.smithi192.stdout:Hit:3 http://archive.ubuntu.com/ubuntu jammy-updates InRelease 2024-04-06T09:27:09.615 INFO:teuthology.orchestra.run.smithi192.stdout:Hit:4 http://archive.ubuntu.com/ubuntu jammy-backports InRelease 2024-04-06T09:27:09.615 INFO:teuthology.orchestra.run.smithi006.stdout:Ign:4 https://3.chacra.ceph.com/r/ceph/reef/4d05f4cad464de0fcdd11a2856643e533aea34e8/ubuntu/jammy/flavors/default jammy InRelease 2024-04-06T09:27:09.620 INFO:teuthology.orchestra.run.smithi006.stdout:Hit:5 http://security.ubuntu.com/ubuntu jammy-security InRelease 2024-04-06T09:27:09.627 INFO:teuthology.orchestra.run.smithi192.stdout:Ign:5 https://3.chacra.ceph.com/r/ceph/reef/4d05f4cad464de0fcdd11a2856643e533aea34e8/ubuntu/jammy/flavors/default jammy InRelease 2024-04-06T09:27:09.648 INFO:teuthology.orchestra.run.smithi192.stdout:Get:6 https://3.chacra.ceph.com/r/ceph/reef/4d05f4cad464de0fcdd11a2856643e533aea34e8/ubuntu/jammy/flavors/default jammy Release [7,662 B] 2024-04-06T09:27:09.651 INFO:teuthology.orchestra.run.smithi006.stdout:Get:6 https://3.chacra.ceph.com/r/ceph/reef/4d05f4cad464de0fcdd11a2856643e533aea34e8/ubuntu/jammy/flavors/default jammy Release [7,662 B] 2024-04-06T09:27:09.670 INFO:teuthology.orchestra.run.smithi192.stdout:Ign:7 https://3.chacra.ceph.com/r/ceph/reef/4d05f4cad464de0fcdd11a2856643e533aea34e8/ubuntu/jammy/flavors/default jammy Release.gpg 2024-04-06T09:27:09.691 INFO:teuthology.orchestra.run.smithi192.stdout:Get:8 https://3.chacra.ceph.com/r/ceph/reef/4d05f4cad464de0fcdd11a2856643e533aea34e8/ubuntu/jammy/flavors/default jammy/main amd64 Packages [17.4 kB] 2024-04-06T09:27:09.695 INFO:teuthology.orchestra.run.smithi006.stdout:Ign:7 https://3.chacra.ceph.com/r/ceph/reef/4d05f4cad464de0fcdd11a2856643e533aea34e8/ubuntu/jammy/flavors/default jammy Release.gpg 2024-04-06T09:27:09.718 INFO:teuthology.orchestra.run.smithi192.stdout:Get:9 https://3.chacra.ceph.com/r/ceph/reef/4d05f4cad464de0fcdd11a2856643e533aea34e8/ubuntu/jammy/flavors/default jammy/main i386 Packages [3,408 B] 2024-04-06T09:27:09.731 INFO:teuthology.orchestra.run.smithi006.stdout:Get:8 https://3.chacra.ceph.com/r/ceph/reef/4d05f4cad464de0fcdd11a2856643e533aea34e8/ubuntu/jammy/flavors/default jammy/main i386 Packages [3,408 B] 2024-04-06T09:27:09.777 INFO:teuthology.orchestra.run.smithi006.stdout:Get:9 https://3.chacra.ceph.com/r/ceph/reef/4d05f4cad464de0fcdd11a2856643e533aea34e8/ubuntu/jammy/flavors/default jammy/main amd64 Packages [17.4 kB] 2024-04-06T09:27:09.900 INFO:teuthology.orchestra.run.smithi006.stdout:Fetched 28.5 kB in 0s (75.4 kB/s) 2024-04-06T09:27:09.913 INFO:teuthology.orchestra.run.smithi192.stdout:Fetched 28.5 kB in 0s (86.6 kB/s) 2024-04-06T09:27:10.873 INFO:teuthology.orchestra.run.smithi192.stdout:Reading package lists... 2024-04-06T09:27:10.880 INFO:teuthology.orchestra.run.smithi006.stdout:Reading package lists... 2024-04-06T09:27:10.888 DEBUG:teuthology.orchestra.run.smithi192:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y --force-yes -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" install ceph=18.2.2-1059-g4d05f4ca-1jammy cephadm=18.2.2-1059-g4d05f4ca-1jammy ceph-mds=18.2.2-1059-g4d05f4ca-1jammy ceph-mgr=18.2.2-1059-g4d05f4ca-1jammy ceph-common=18.2.2-1059-g4d05f4ca-1jammy ceph-fuse=18.2.2-1059-g4d05f4ca-1jammy ceph-test=18.2.2-1059-g4d05f4ca-1jammy ceph-volume=18.2.2-1059-g4d05f4ca-1jammy radosgw=18.2.2-1059-g4d05f4ca-1jammy python3-rados=18.2.2-1059-g4d05f4ca-1jammy python3-rgw=18.2.2-1059-g4d05f4ca-1jammy python3-cephfs=18.2.2-1059-g4d05f4ca-1jammy python3-rbd=18.2.2-1059-g4d05f4ca-1jammy libcephfs2=18.2.2-1059-g4d05f4ca-1jammy libcephfs-dev=18.2.2-1059-g4d05f4ca-1jammy librados2=18.2.2-1059-g4d05f4ca-1jammy librbd1=18.2.2-1059-g4d05f4ca-1jammy rbd-fuse=18.2.2-1059-g4d05f4ca-1jammy rbd-mirror=18.2.2-1059-g4d05f4ca-1jammy 2024-04-06T09:27:10.896 DEBUG:teuthology.orchestra.run.smithi006:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y --force-yes -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" install ceph=18.2.2-1059-g4d05f4ca-1jammy cephadm=18.2.2-1059-g4d05f4ca-1jammy ceph-mds=18.2.2-1059-g4d05f4ca-1jammy ceph-mgr=18.2.2-1059-g4d05f4ca-1jammy ceph-common=18.2.2-1059-g4d05f4ca-1jammy ceph-fuse=18.2.2-1059-g4d05f4ca-1jammy ceph-test=18.2.2-1059-g4d05f4ca-1jammy ceph-volume=18.2.2-1059-g4d05f4ca-1jammy radosgw=18.2.2-1059-g4d05f4ca-1jammy python3-rados=18.2.2-1059-g4d05f4ca-1jammy python3-rgw=18.2.2-1059-g4d05f4ca-1jammy python3-cephfs=18.2.2-1059-g4d05f4ca-1jammy python3-rbd=18.2.2-1059-g4d05f4ca-1jammy libcephfs2=18.2.2-1059-g4d05f4ca-1jammy libcephfs-dev=18.2.2-1059-g4d05f4ca-1jammy librados2=18.2.2-1059-g4d05f4ca-1jammy librbd1=18.2.2-1059-g4d05f4ca-1jammy rbd-fuse=18.2.2-1059-g4d05f4ca-1jammy rbd-mirror=18.2.2-1059-g4d05f4ca-1jammy 2024-04-06T09:27:10.947 INFO:teuthology.orchestra.run.smithi192.stdout:Reading package lists... 2024-04-06T09:27:10.959 INFO:teuthology.orchestra.run.smithi006.stdout:Reading package lists... 2024-04-06T09:27:11.145 INFO:teuthology.orchestra.run.smithi192.stdout:Building dependency tree... 2024-04-06T09:27:11.145 INFO:teuthology.orchestra.run.smithi192.stdout:Reading state information... 2024-04-06T09:27:11.158 INFO:teuthology.orchestra.run.smithi006.stdout:Building dependency tree... 2024-04-06T09:27:11.159 INFO:teuthology.orchestra.run.smithi006.stdout:Reading state information... 2024-04-06T09:27:11.333 INFO:teuthology.orchestra.run.smithi192.stdout:The following packages were automatically installed and are no longer required: 2024-04-06T09:27:11.333 INFO:teuthology.orchestra.run.smithi192.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2024-04-06T09:27:11.333 INFO:teuthology.orchestra.run.smithi192.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2024-04-06T09:27:11.333 INFO:teuthology.orchestra.run.smithi192.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2024-04-06T09:27:11.334 INFO:teuthology.orchestra.run.smithi192.stdout: linux-image-5.4.0-135-generic linux-modules-5.4.0-135-generic python2 2024-04-06T09:27:11.334 INFO:teuthology.orchestra.run.smithi192.stdout: python2-dev python2-minimal python2.7 python2.7-dev python2.7-minimal 2024-04-06T09:27:11.334 INFO:teuthology.orchestra.run.smithi192.stdout:Use 'sudo apt autoremove' to remove them. 2024-04-06T09:27:11.334 INFO:teuthology.orchestra.run.smithi192.stdout:The following additional packages will be installed: 2024-04-06T09:27:11.334 INFO:teuthology.orchestra.run.smithi192.stdout: ceph-base ceph-mgr-cephadm ceph-mgr-dashboard ceph-mgr-diskprediction-local 2024-04-06T09:27:11.335 INFO:teuthology.orchestra.run.smithi192.stdout: ceph-mgr-k8sevents ceph-mgr-modules-core ceph-mon ceph-osd jq 2024-04-06T09:27:11.335 INFO:teuthology.orchestra.run.smithi192.stdout: libdouble-conversion3 libfmt8 libjq1 liblttng-ust1 liboath0 libonig5 2024-04-06T09:27:11.335 INFO:teuthology.orchestra.run.smithi192.stdout: libpcre2-16-0 libqt5core5a libqt5dbus5 libqt5network5 libradosstriper1 2024-04-06T09:27:11.335 INFO:teuthology.orchestra.run.smithi192.stdout: librdkafka1 librgw2 libsqlite3-mod-ceph libthrift-0.16.0 nvme-cli 2024-04-06T09:27:11.335 INFO:teuthology.orchestra.run.smithi192.stdout: python-asyncssh-doc python-babel-localedata python-pastedeploy-tpl 2024-04-06T09:27:11.336 INFO:teuthology.orchestra.run.smithi192.stdout: python3-asyncssh python3-babel python3-bcrypt python3-cachetools 2024-04-06T09:27:11.336 INFO:teuthology.orchestra.run.smithi192.stdout: python3-ceph-argparse python3-ceph-common python3-certifi python3-cheroot 2024-04-06T09:27:11.336 INFO:teuthology.orchestra.run.smithi192.stdout: python3-cherrypy3 python3-google-auth python3-idna python3-iniconfig 2024-04-06T09:27:11.336 INFO:teuthology.orchestra.run.smithi192.stdout: python3-jaraco.classes python3-jaraco.collections python3-jaraco.functools 2024-04-06T09:27:11.336 INFO:teuthology.orchestra.run.smithi192.stdout: python3-jaraco.text python3-jinja2 python3-joblib python3-kubernetes 2024-04-06T09:27:11.336 INFO:teuthology.orchestra.run.smithi192.stdout: python3-logutils python3-mako python3-markupsafe python3-natsort 2024-04-06T09:27:11.336 INFO:teuthology.orchestra.run.smithi192.stdout: python3-openssl python3-paste python3-pastedeploy python3-pastescript 2024-04-06T09:27:11.336 INFO:teuthology.orchestra.run.smithi192.stdout: python3-pecan python3-pluggy python3-portend python3-prettytable 2024-04-06T09:27:11.336 INFO:teuthology.orchestra.run.smithi192.stdout: python3-psutil python3-py python3-pyasn1 python3-pyasn1-modules 2024-04-06T09:27:11.336 INFO:teuthology.orchestra.run.smithi192.stdout: python3-pygments python3-pyinotify python3-pytest python3-repoze.lru 2024-04-06T09:27:11.336 INFO:teuthology.orchestra.run.smithi192.stdout: python3-requests python3-requests-oauthlib python3-routes python3-rsa 2024-04-06T09:27:11.336 INFO:teuthology.orchestra.run.smithi192.stdout: python3-simplegeneric python3-simplejson python3-singledispatch 2024-04-06T09:27:11.336 INFO:teuthology.orchestra.run.smithi192.stdout: python3-sklearn python3-sklearn-lib python3-tempita python3-tempora 2024-04-06T09:27:11.336 INFO:teuthology.orchestra.run.smithi192.stdout: python3-threadpoolctl python3-toml python3-urllib3 python3-waitress 2024-04-06T09:27:11.336 INFO:teuthology.orchestra.run.smithi192.stdout: python3-wcwidth python3-webob python3-websocket python3-webtest 2024-04-06T09:27:11.337 INFO:teuthology.orchestra.run.smithi192.stdout: python3-werkzeug python3-zc.lockfile qttranslations5-l10n socat xmlstarlet 2024-04-06T09:27:11.337 INFO:teuthology.orchestra.run.smithi192.stdout:Suggested packages: 2024-04-06T09:27:11.337 INFO:teuthology.orchestra.run.smithi192.stdout: python3-influxdb python-jinja2-doc python3-beaker python-mako-doc 2024-04-06T09:27:11.337 INFO:teuthology.orchestra.run.smithi192.stdout: python-natsort-doc python-openssl-doc python3-openssl-dbg httpd-wsgi 2024-04-06T09:27:11.337 INFO:teuthology.orchestra.run.smithi192.stdout: libapache2-mod-python libapache2-mod-scgi libjs-mochikit python-pecan-doc 2024-04-06T09:27:11.337 INFO:teuthology.orchestra.run.smithi192.stdout: python-psutil-doc subversion python-pygments-doc python-pyinotify-doc 2024-04-06T09:27:11.337 INFO:teuthology.orchestra.run.smithi192.stdout: python3-socks python-requests-doc python3-dap python-sklearn-doc ipython3 2024-04-06T09:27:11.337 INFO:teuthology.orchestra.run.smithi192.stdout: python-waitress-doc python-webob-doc python-webtest-doc python-werkzeug-doc 2024-04-06T09:27:11.337 INFO:teuthology.orchestra.run.smithi192.stdout: python3-watchdog 2024-04-06T09:27:11.337 INFO:teuthology.orchestra.run.smithi192.stdout:Recommended packages: 2024-04-06T09:27:11.338 INFO:teuthology.orchestra.run.smithi192.stdout: btrfs-tools 2024-04-06T09:27:11.356 INFO:teuthology.orchestra.run.smithi006.stdout:The following packages were automatically installed and are no longer required: 2024-04-06T09:27:11.356 INFO:teuthology.orchestra.run.smithi006.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2024-04-06T09:27:11.356 INFO:teuthology.orchestra.run.smithi006.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2024-04-06T09:27:11.356 INFO:teuthology.orchestra.run.smithi006.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2024-04-06T09:27:11.357 INFO:teuthology.orchestra.run.smithi006.stdout: linux-image-5.4.0-135-generic linux-modules-5.4.0-135-generic python2 2024-04-06T09:27:11.357 INFO:teuthology.orchestra.run.smithi006.stdout: python2-dev python2-minimal python2.7 python2.7-dev python2.7-minimal 2024-04-06T09:27:11.357 INFO:teuthology.orchestra.run.smithi006.stdout:Use 'sudo apt autoremove' to remove them. 2024-04-06T09:27:11.358 INFO:teuthology.orchestra.run.smithi006.stdout:The following additional packages will be installed: 2024-04-06T09:27:11.358 INFO:teuthology.orchestra.run.smithi006.stdout: ceph-base ceph-mgr-cephadm ceph-mgr-dashboard ceph-mgr-diskprediction-local 2024-04-06T09:27:11.358 INFO:teuthology.orchestra.run.smithi006.stdout: ceph-mgr-k8sevents ceph-mgr-modules-core ceph-mon ceph-osd jq 2024-04-06T09:27:11.358 INFO:teuthology.orchestra.run.smithi006.stdout: libdouble-conversion3 libfmt8 libjq1 liblttng-ust1 liboath0 libonig5 2024-04-06T09:27:11.358 INFO:teuthology.orchestra.run.smithi006.stdout: libpcre2-16-0 libqt5core5a libqt5dbus5 libqt5network5 libradosstriper1 2024-04-06T09:27:11.359 INFO:teuthology.orchestra.run.smithi006.stdout: librdkafka1 librgw2 libsqlite3-mod-ceph libthrift-0.16.0 nvme-cli 2024-04-06T09:27:11.359 INFO:teuthology.orchestra.run.smithi006.stdout: python-asyncssh-doc python-babel-localedata python-pastedeploy-tpl 2024-04-06T09:27:11.359 INFO:teuthology.orchestra.run.smithi006.stdout: python3-asyncssh python3-babel python3-bcrypt python3-cachetools 2024-04-06T09:27:11.359 INFO:teuthology.orchestra.run.smithi006.stdout: python3-ceph-argparse python3-ceph-common python3-certifi python3-cheroot 2024-04-06T09:27:11.359 INFO:teuthology.orchestra.run.smithi006.stdout: python3-cherrypy3 python3-google-auth python3-idna python3-iniconfig 2024-04-06T09:27:11.359 INFO:teuthology.orchestra.run.smithi006.stdout: python3-jaraco.classes python3-jaraco.collections python3-jaraco.functools 2024-04-06T09:27:11.359 INFO:teuthology.orchestra.run.smithi006.stdout: python3-jaraco.text python3-jinja2 python3-joblib python3-kubernetes 2024-04-06T09:27:11.359 INFO:teuthology.orchestra.run.smithi006.stdout: python3-logutils python3-mako python3-markupsafe python3-natsort 2024-04-06T09:27:11.360 INFO:teuthology.orchestra.run.smithi006.stdout: python3-openssl python3-paste python3-pastedeploy python3-pastescript 2024-04-06T09:27:11.360 INFO:teuthology.orchestra.run.smithi006.stdout: python3-pecan python3-pluggy python3-portend python3-prettytable 2024-04-06T09:27:11.360 INFO:teuthology.orchestra.run.smithi006.stdout: python3-psutil python3-py python3-pyasn1 python3-pyasn1-modules 2024-04-06T09:27:11.360 INFO:teuthology.orchestra.run.smithi006.stdout: python3-pygments python3-pyinotify python3-pytest python3-repoze.lru 2024-04-06T09:27:11.360 INFO:teuthology.orchestra.run.smithi006.stdout: python3-requests python3-requests-oauthlib python3-routes python3-rsa 2024-04-06T09:27:11.360 INFO:teuthology.orchestra.run.smithi006.stdout: python3-simplegeneric python3-simplejson python3-singledispatch 2024-04-06T09:27:11.360 INFO:teuthology.orchestra.run.smithi006.stdout: python3-sklearn python3-sklearn-lib python3-tempita python3-tempora 2024-04-06T09:27:11.360 INFO:teuthology.orchestra.run.smithi006.stdout: python3-threadpoolctl python3-toml python3-urllib3 python3-waitress 2024-04-06T09:27:11.360 INFO:teuthology.orchestra.run.smithi006.stdout: python3-wcwidth python3-webob python3-websocket python3-webtest 2024-04-06T09:27:11.360 INFO:teuthology.orchestra.run.smithi006.stdout: python3-werkzeug python3-zc.lockfile qttranslations5-l10n socat xmlstarlet 2024-04-06T09:27:11.361 INFO:teuthology.orchestra.run.smithi006.stdout:Suggested packages: 2024-04-06T09:27:11.361 INFO:teuthology.orchestra.run.smithi006.stdout: python3-influxdb python-jinja2-doc python3-beaker python-mako-doc 2024-04-06T09:27:11.361 INFO:teuthology.orchestra.run.smithi006.stdout: python-natsort-doc python-openssl-doc python3-openssl-dbg httpd-wsgi 2024-04-06T09:27:11.361 INFO:teuthology.orchestra.run.smithi006.stdout: libapache2-mod-python libapache2-mod-scgi libjs-mochikit python-pecan-doc 2024-04-06T09:27:11.361 INFO:teuthology.orchestra.run.smithi006.stdout: python-psutil-doc subversion python-pygments-doc python-pyinotify-doc 2024-04-06T09:27:11.361 INFO:teuthology.orchestra.run.smithi006.stdout: python3-socks python-requests-doc python3-dap python-sklearn-doc ipython3 2024-04-06T09:27:11.361 INFO:teuthology.orchestra.run.smithi006.stdout: python-waitress-doc python-webob-doc python-webtest-doc python-werkzeug-doc 2024-04-06T09:27:11.361 INFO:teuthology.orchestra.run.smithi006.stdout: python3-watchdog 2024-04-06T09:27:11.361 INFO:teuthology.orchestra.run.smithi006.stdout:Recommended packages: 2024-04-06T09:27:11.361 INFO:teuthology.orchestra.run.smithi006.stdout: btrfs-tools 2024-04-06T09:27:11.437 INFO:teuthology.orchestra.run.smithi192.stdout:The following NEW packages will be installed: 2024-04-06T09:27:11.438 INFO:teuthology.orchestra.run.smithi192.stdout: ceph ceph-base ceph-common ceph-fuse ceph-mds ceph-mgr ceph-mgr-cephadm 2024-04-06T09:27:11.438 INFO:teuthology.orchestra.run.smithi192.stdout: ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-k8sevents 2024-04-06T09:27:11.438 INFO:teuthology.orchestra.run.smithi192.stdout: ceph-mgr-modules-core ceph-mon ceph-osd ceph-test ceph-volume cephadm jq 2024-04-06T09:27:11.438 INFO:teuthology.orchestra.run.smithi192.stdout: libcephfs-dev libcephfs2 libdouble-conversion3 libfmt8 libjq1 liblttng-ust1 2024-04-06T09:27:11.438 INFO:teuthology.orchestra.run.smithi192.stdout: liboath0 libonig5 libpcre2-16-0 libqt5core5a libqt5dbus5 libqt5network5 2024-04-06T09:27:11.438 INFO:teuthology.orchestra.run.smithi192.stdout: librados2 libradosstriper1 librbd1 librdkafka1 librgw2 libsqlite3-mod-ceph 2024-04-06T09:27:11.439 INFO:teuthology.orchestra.run.smithi192.stdout: libthrift-0.16.0 nvme-cli python-asyncssh-doc python-babel-localedata 2024-04-06T09:27:11.439 INFO:teuthology.orchestra.run.smithi192.stdout: python-pastedeploy-tpl python3-asyncssh python3-babel python3-bcrypt 2024-04-06T09:27:11.439 INFO:teuthology.orchestra.run.smithi192.stdout: python3-cachetools python3-ceph-argparse python3-ceph-common python3-cephfs 2024-04-06T09:27:11.439 INFO:teuthology.orchestra.run.smithi192.stdout: python3-certifi python3-cheroot python3-cherrypy3 python3-google-auth 2024-04-06T09:27:11.439 INFO:teuthology.orchestra.run.smithi192.stdout: python3-idna python3-iniconfig python3-jaraco.classes 2024-04-06T09:27:11.439 INFO:teuthology.orchestra.run.smithi192.stdout: python3-jaraco.collections python3-jaraco.functools python3-jaraco.text 2024-04-06T09:27:11.439 INFO:teuthology.orchestra.run.smithi192.stdout: python3-jinja2 python3-joblib python3-kubernetes python3-logutils 2024-04-06T09:27:11.439 INFO:teuthology.orchestra.run.smithi192.stdout: python3-mako python3-markupsafe python3-natsort python3-openssl 2024-04-06T09:27:11.439 INFO:teuthology.orchestra.run.smithi192.stdout: python3-paste python3-pastedeploy python3-pastescript python3-pecan 2024-04-06T09:27:11.440 INFO:teuthology.orchestra.run.smithi192.stdout: python3-pluggy python3-portend python3-prettytable python3-psutil python3-py 2024-04-06T09:27:11.440 INFO:teuthology.orchestra.run.smithi192.stdout: python3-pyasn1 python3-pyasn1-modules python3-pygments python3-pyinotify 2024-04-06T09:27:11.440 INFO:teuthology.orchestra.run.smithi192.stdout: python3-pytest python3-rados python3-rbd python3-repoze.lru python3-requests 2024-04-06T09:27:11.440 INFO:teuthology.orchestra.run.smithi192.stdout: python3-requests-oauthlib python3-rgw python3-routes python3-rsa 2024-04-06T09:27:11.440 INFO:teuthology.orchestra.run.smithi192.stdout: python3-simplegeneric python3-simplejson python3-singledispatch 2024-04-06T09:27:11.440 INFO:teuthology.orchestra.run.smithi192.stdout: python3-sklearn python3-sklearn-lib python3-tempita python3-tempora 2024-04-06T09:27:11.440 INFO:teuthology.orchestra.run.smithi192.stdout: python3-threadpoolctl python3-toml python3-urllib3 python3-waitress 2024-04-06T09:27:11.440 INFO:teuthology.orchestra.run.smithi192.stdout: python3-wcwidth python3-webob python3-websocket python3-webtest 2024-04-06T09:27:11.440 INFO:teuthology.orchestra.run.smithi192.stdout: python3-werkzeug python3-zc.lockfile qttranslations5-l10n radosgw rbd-fuse 2024-04-06T09:27:11.440 INFO:teuthology.orchestra.run.smithi192.stdout: rbd-mirror socat xmlstarlet 2024-04-06T09:27:11.469 INFO:teuthology.orchestra.run.smithi006.stdout:The following NEW packages will be installed: 2024-04-06T09:27:11.469 INFO:teuthology.orchestra.run.smithi006.stdout: ceph ceph-base ceph-common ceph-fuse ceph-mds ceph-mgr ceph-mgr-cephadm 2024-04-06T09:27:11.469 INFO:teuthology.orchestra.run.smithi006.stdout: ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-k8sevents 2024-04-06T09:27:11.469 INFO:teuthology.orchestra.run.smithi006.stdout: ceph-mgr-modules-core ceph-mon ceph-osd ceph-test ceph-volume cephadm jq 2024-04-06T09:27:11.470 INFO:teuthology.orchestra.run.smithi006.stdout: libcephfs-dev libcephfs2 libdouble-conversion3 libfmt8 libjq1 liblttng-ust1 2024-04-06T09:27:11.470 INFO:teuthology.orchestra.run.smithi006.stdout: liboath0 libonig5 libpcre2-16-0 libqt5core5a libqt5dbus5 libqt5network5 2024-04-06T09:27:11.470 INFO:teuthology.orchestra.run.smithi006.stdout: librados2 libradosstriper1 librbd1 librdkafka1 librgw2 libsqlite3-mod-ceph 2024-04-06T09:27:11.471 INFO:teuthology.orchestra.run.smithi006.stdout: libthrift-0.16.0 nvme-cli python-asyncssh-doc python-babel-localedata 2024-04-06T09:27:11.471 INFO:teuthology.orchestra.run.smithi006.stdout: python-pastedeploy-tpl python3-asyncssh python3-babel python3-bcrypt 2024-04-06T09:27:11.471 INFO:teuthology.orchestra.run.smithi006.stdout: python3-cachetools python3-ceph-argparse python3-ceph-common python3-cephfs 2024-04-06T09:27:11.471 INFO:teuthology.orchestra.run.smithi006.stdout: python3-certifi python3-cheroot python3-cherrypy3 python3-google-auth 2024-04-06T09:27:11.471 INFO:teuthology.orchestra.run.smithi006.stdout: python3-idna python3-iniconfig python3-jaraco.classes 2024-04-06T09:27:11.471 INFO:teuthology.orchestra.run.smithi006.stdout: python3-jaraco.collections python3-jaraco.functools python3-jaraco.text 2024-04-06T09:27:11.471 INFO:teuthology.orchestra.run.smithi006.stdout: python3-jinja2 python3-joblib python3-kubernetes python3-logutils 2024-04-06T09:27:11.471 INFO:teuthology.orchestra.run.smithi006.stdout: python3-mako python3-markupsafe python3-natsort python3-openssl 2024-04-06T09:27:11.471 INFO:teuthology.orchestra.run.smithi006.stdout: python3-paste python3-pastedeploy python3-pastescript python3-pecan 2024-04-06T09:27:11.471 INFO:teuthology.orchestra.run.smithi006.stdout: python3-pluggy python3-portend python3-prettytable python3-psutil python3-py 2024-04-06T09:27:11.471 INFO:teuthology.orchestra.run.smithi006.stdout: python3-pyasn1 python3-pyasn1-modules python3-pygments python3-pyinotify 2024-04-06T09:27:11.471 INFO:teuthology.orchestra.run.smithi006.stdout: python3-pytest python3-rados python3-rbd python3-repoze.lru python3-requests 2024-04-06T09:27:11.471 INFO:teuthology.orchestra.run.smithi006.stdout: python3-requests-oauthlib python3-rgw python3-routes python3-rsa 2024-04-06T09:27:11.471 INFO:teuthology.orchestra.run.smithi006.stdout: python3-simplegeneric python3-simplejson python3-singledispatch 2024-04-06T09:27:11.472 INFO:teuthology.orchestra.run.smithi006.stdout: python3-sklearn python3-sklearn-lib python3-tempita python3-tempora 2024-04-06T09:27:11.472 INFO:teuthology.orchestra.run.smithi006.stdout: python3-threadpoolctl python3-toml python3-urllib3 python3-waitress 2024-04-06T09:27:11.472 INFO:teuthology.orchestra.run.smithi006.stdout: python3-wcwidth python3-webob python3-websocket python3-webtest 2024-04-06T09:27:11.472 INFO:teuthology.orchestra.run.smithi006.stdout: python3-werkzeug python3-zc.lockfile qttranslations5-l10n radosgw rbd-fuse 2024-04-06T09:27:11.472 INFO:teuthology.orchestra.run.smithi006.stdout: rbd-mirror socat xmlstarlet 2024-04-06T09:27:11.503 INFO:teuthology.orchestra.run.smithi192.stdout:0 upgraded, 110 newly installed, 0 to remove and 299 not upgraded. 2024-04-06T09:27:11.503 INFO:teuthology.orchestra.run.smithi192.stdout:Need to get 165 MB of archives. 2024-04-06T09:27:11.503 INFO:teuthology.orchestra.run.smithi192.stdout:After this operation, 673 MB of additional disk space will be used. 2024-04-06T09:27:11.503 INFO:teuthology.orchestra.run.smithi192.stdout:Get:1 http://archive.ubuntu.com/ubuntu jammy/universe amd64 libfmt8 amd64 8.1.1+ds1-2 [60.2 kB] 2024-04-06T09:27:11.534 INFO:teuthology.orchestra.run.smithi006.stdout:0 upgraded, 110 newly installed, 0 to remove and 299 not upgraded. 2024-04-06T09:27:11.534 INFO:teuthology.orchestra.run.smithi006.stdout:Need to get 165 MB of archives. 2024-04-06T09:27:11.534 INFO:teuthology.orchestra.run.smithi006.stdout:After this operation, 673 MB of additional disk space will be used. 2024-04-06T09:27:11.534 INFO:teuthology.orchestra.run.smithi006.stdout:Get:1 http://archive.ubuntu.com/ubuntu jammy/universe amd64 libfmt8 amd64 8.1.1+ds1-2 [60.2 kB] 2024-04-06T09:27:11.572 INFO:teuthology.orchestra.run.smithi192.stdout:Get:2 http://archive.ubuntu.com/ubuntu jammy/main amd64 liblttng-ust1 amd64 2.13.1-1ubuntu1 [190 kB] 2024-04-06T09:27:11.578 INFO:teuthology.orchestra.run.smithi192.stdout:Get:3 https://3.chacra.ceph.com/r/ceph/reef/4d05f4cad464de0fcdd11a2856643e533aea34e8/ubuntu/jammy/flavors/default jammy/main amd64 librados2 amd64 18.2.2-1059-g4d05f4ca-1jammy [3,440 kB] 2024-04-06T09:27:11.601 INFO:teuthology.orchestra.run.smithi006.stdout:Get:2 http://archive.ubuntu.com/ubuntu jammy/main amd64 liblttng-ust1 amd64 2.13.1-1ubuntu1 [190 kB] 2024-04-06T09:27:11.616 INFO:teuthology.orchestra.run.smithi192.stdout:Get:4 http://archive.ubuntu.com/ubuntu jammy/universe amd64 libdouble-conversion3 amd64 3.1.7-4 [39.0 kB] 2024-04-06T09:27:11.621 INFO:teuthology.orchestra.run.smithi192.stdout:Get:5 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libpcre2-16-0 amd64 10.39-3ubuntu0.1 [203 kB] 2024-04-06T09:27:11.638 INFO:teuthology.orchestra.run.smithi192.stdout:Get:6 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5core5a amd64 5.15.3+dfsg-2ubuntu0.2 [2,006 kB] 2024-04-06T09:27:11.645 INFO:teuthology.orchestra.run.smithi006.stdout:Get:3 http://archive.ubuntu.com/ubuntu jammy/universe amd64 libdouble-conversion3 amd64 3.1.7-4 [39.0 kB] 2024-04-06T09:27:11.648 INFO:teuthology.orchestra.run.smithi006.stdout:Get:4 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libpcre2-16-0 amd64 10.39-3ubuntu0.1 [203 kB] 2024-04-06T09:27:11.667 INFO:teuthology.orchestra.run.smithi006.stdout:Get:5 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5core5a amd64 5.15.3+dfsg-2ubuntu0.2 [2,006 kB] 2024-04-06T09:27:11.703 INFO:teuthology.orchestra.run.smithi006.stdout:Get:6 https://3.chacra.ceph.com/r/ceph/reef/4d05f4cad464de0fcdd11a2856643e533aea34e8/ubuntu/jammy/flavors/default jammy/main amd64 librados2 amd64 18.2.2-1059-g4d05f4ca-1jammy [3,440 kB] 2024-04-06T09:27:11.710 INFO:teuthology.orchestra.run.smithi192.stdout:Get:7 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5dbus5 amd64 5.15.3+dfsg-2ubuntu0.2 [222 kB] 2024-04-06T09:27:11.714 INFO:teuthology.orchestra.run.smithi192.stdout:Get:8 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5network5 amd64 5.15.3+dfsg-2ubuntu0.2 [731 kB] 2024-04-06T09:27:11.727 INFO:teuthology.orchestra.run.smithi192.stdout:Get:9 http://archive.ubuntu.com/ubuntu jammy/universe amd64 libthrift-0.16.0 amd64 0.16.0-2 [267 kB] 2024-04-06T09:27:11.739 INFO:teuthology.orchestra.run.smithi192.stdout:Get:10 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-wcwidth all 0.2.5+dfsg1-1 [21.9 kB] 2024-04-06T09:27:11.739 INFO:teuthology.orchestra.run.smithi192.stdout:Get:11 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-prettytable all 2.5.0-2 [31.3 kB] 2024-04-06T09:27:11.740 INFO:teuthology.orchestra.run.smithi192.stdout:Get:12 http://archive.ubuntu.com/ubuntu jammy/universe amd64 librdkafka1 amd64 1.8.0-1build1 [633 kB] 2024-04-06T09:27:11.751 INFO:teuthology.orchestra.run.smithi192.stdout:Get:13 http://archive.ubuntu.com/ubuntu jammy/main amd64 liboath0 amd64 2.6.7-3build1 [41.1 kB] 2024-04-06T09:27:11.752 INFO:teuthology.orchestra.run.smithi192.stdout:Get:14 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.functools all 3.4.0-2 [9,030 B] 2024-04-06T09:27:11.753 INFO:teuthology.orchestra.run.smithi192.stdout:Get:15 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-cheroot all 8.5.2+ds1-1ubuntu3.1 [71.1 kB] 2024-04-06T09:27:11.759 INFO:teuthology.orchestra.run.smithi192.stdout:Get:16 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.classes all 3.2.1-3 [6,452 B] 2024-04-06T09:27:11.764 INFO:teuthology.orchestra.run.smithi006.stdout:Get:7 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5dbus5 amd64 5.15.3+dfsg-2ubuntu0.2 [222 kB] 2024-04-06T09:27:11.770 INFO:teuthology.orchestra.run.smithi006.stdout:Get:8 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5network5 amd64 5.15.3+dfsg-2ubuntu0.2 [731 kB] 2024-04-06T09:27:11.786 INFO:teuthology.orchestra.run.smithi006.stdout:Get:9 http://archive.ubuntu.com/ubuntu jammy/universe amd64 libthrift-0.16.0 amd64 0.16.0-2 [267 kB] 2024-04-06T09:27:11.790 INFO:teuthology.orchestra.run.smithi192.stdout:Get:17 https://3.chacra.ceph.com/r/ceph/reef/4d05f4cad464de0fcdd11a2856643e533aea34e8/ubuntu/jammy/flavors/default jammy/main amd64 librbd1 amd64 18.2.2-1059-g4d05f4ca-1jammy [3,105 kB] 2024-04-06T09:27:11.790 INFO:teuthology.orchestra.run.smithi192.stdout:Get:18 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.text all 3.6.0-2 [8,716 B] 2024-04-06T09:27:11.791 INFO:teuthology.orchestra.run.smithi192.stdout:Get:19 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.collections all 3.4.0-2 [11.4 kB] 2024-04-06T09:27:11.791 INFO:teuthology.orchestra.run.smithi192.stdout:Get:20 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-tempora all 4.1.2-1 [14.8 kB] 2024-04-06T09:27:11.791 INFO:teuthology.orchestra.run.smithi192.stdout:Get:21 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-portend all 3.0.0-1 [7,240 B] 2024-04-06T09:27:11.792 INFO:teuthology.orchestra.run.smithi192.stdout:Get:22 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-zc.lockfile all 2.0-1 [8,980 B] 2024-04-06T09:27:11.792 INFO:teuthology.orchestra.run.smithi192.stdout:Get:23 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-cherrypy3 all 18.6.1-4 [208 kB] 2024-04-06T09:27:11.794 INFO:teuthology.orchestra.run.smithi006.stdout:Get:10 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-wcwidth all 0.2.5+dfsg1-1 [21.9 kB] 2024-04-06T09:27:11.795 INFO:teuthology.orchestra.run.smithi006.stdout:Get:11 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-prettytable all 2.5.0-2 [31.3 kB] 2024-04-06T09:27:11.796 INFO:teuthology.orchestra.run.smithi006.stdout:Get:12 http://archive.ubuntu.com/ubuntu jammy/universe amd64 librdkafka1 amd64 1.8.0-1build1 [633 kB] 2024-04-06T09:27:11.796 INFO:teuthology.orchestra.run.smithi192.stdout:Get:24 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-natsort all 8.0.2-1 [35.3 kB] 2024-04-06T09:27:11.796 INFO:teuthology.orchestra.run.smithi192.stdout:Get:25 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-logutils all 0.3.3-8 [17.6 kB] 2024-04-06T09:27:11.797 INFO:teuthology.orchestra.run.smithi192.stdout:Get:26 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-markupsafe amd64 2.0.1-2build1 [12.7 kB] 2024-04-06T09:27:11.808 INFO:teuthology.orchestra.run.smithi006.stdout:Get:13 http://archive.ubuntu.com/ubuntu jammy/main amd64 liboath0 amd64 2.6.7-3build1 [41.1 kB] 2024-04-06T09:27:11.809 INFO:teuthology.orchestra.run.smithi006.stdout:Get:14 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.functools all 3.4.0-2 [9,030 B] 2024-04-06T09:27:11.809 INFO:teuthology.orchestra.run.smithi006.stdout:Get:15 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-cheroot all 8.5.2+ds1-1ubuntu3.1 [71.1 kB] 2024-04-06T09:27:11.811 INFO:teuthology.orchestra.run.smithi006.stdout:Get:16 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.classes all 3.2.1-3 [6,452 B] 2024-04-06T09:27:11.820 INFO:teuthology.orchestra.run.smithi192.stdout:Get:27 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-mako all 1.1.3+ds1-2ubuntu0.1 [60.5 kB] 2024-04-06T09:27:11.821 INFO:teuthology.orchestra.run.smithi192.stdout:Get:28 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-simplegeneric all 0.8.1-3 [11.3 kB] 2024-04-06T09:27:11.822 INFO:teuthology.orchestra.run.smithi192.stdout:Get:29 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-singledispatch all 3.4.0.3-3 [7,320 B] 2024-04-06T09:27:11.822 INFO:teuthology.orchestra.run.smithi192.stdout:Get:30 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-webob all 1:1.8.6-1.1 [85.2 kB] 2024-04-06T09:27:11.824 INFO:teuthology.orchestra.run.smithi192.stdout:Get:31 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-waitress all 1.4.4-1.1ubuntu1 [47.3 kB] 2024-04-06T09:27:11.824 INFO:teuthology.orchestra.run.smithi192.stdout:Get:32 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-tempita all 0.5.2-6ubuntu1 [15.1 kB] 2024-04-06T09:27:11.826 INFO:teuthology.orchestra.run.smithi006.stdout:Get:17 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.text all 3.6.0-2 [8,716 B] 2024-04-06T09:27:11.827 INFO:teuthology.orchestra.run.smithi006.stdout:Get:18 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.collections all 3.4.0-2 [11.4 kB] 2024-04-06T09:27:11.827 INFO:teuthology.orchestra.run.smithi006.stdout:Get:19 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-tempora all 4.1.2-1 [14.8 kB] 2024-04-06T09:27:11.828 INFO:teuthology.orchestra.run.smithi006.stdout:Get:20 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-portend all 3.0.0-1 [7,240 B] 2024-04-06T09:27:11.829 INFO:teuthology.orchestra.run.smithi006.stdout:Get:21 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-zc.lockfile all 2.0-1 [8,980 B] 2024-04-06T09:27:11.840 INFO:teuthology.orchestra.run.smithi192.stdout:Get:33 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-paste all 3.5.0+dfsg1-1 [456 kB] 2024-04-06T09:27:11.847 INFO:teuthology.orchestra.run.smithi192.stdout:Get:34 http://archive.ubuntu.com/ubuntu jammy/main amd64 python-pastedeploy-tpl all 2.1.1-1 [4,892 B] 2024-04-06T09:27:11.848 INFO:teuthology.orchestra.run.smithi192.stdout:Get:35 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pastedeploy all 2.1.1-1 [26.6 kB] 2024-04-06T09:27:11.848 INFO:teuthology.orchestra.run.smithi006.stdout:Get:22 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-cherrypy3 all 18.6.1-4 [208 kB] 2024-04-06T09:27:11.848 INFO:teuthology.orchestra.run.smithi192.stdout:Get:36 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-webtest all 2.0.35-1 [28.5 kB] 2024-04-06T09:27:11.852 INFO:teuthology.orchestra.run.smithi006.stdout:Get:23 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-natsort all 8.0.2-1 [35.3 kB] 2024-04-06T09:27:11.853 INFO:teuthology.orchestra.run.smithi006.stdout:Get:24 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-logutils all 0.3.3-8 [17.6 kB] 2024-04-06T09:27:11.854 INFO:teuthology.orchestra.run.smithi006.stdout:Get:25 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-markupsafe amd64 2.0.1-2build1 [12.7 kB] 2024-04-06T09:27:11.854 INFO:teuthology.orchestra.run.smithi006.stdout:Get:26 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-mako all 1.1.3+ds1-2ubuntu0.1 [60.5 kB] 2024-04-06T09:27:11.861 INFO:teuthology.orchestra.run.smithi192.stdout:Get:37 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pecan all 1.3.3-4ubuntu2 [87.3 kB] 2024-04-06T09:27:11.863 INFO:teuthology.orchestra.run.smithi192.stdout:Get:38 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-certifi all 2020.6.20-1 [150 kB] 2024-04-06T09:27:11.866 INFO:teuthology.orchestra.run.smithi192.stdout:Get:39 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-idna all 3.3-1 [49.3 kB] 2024-04-06T09:27:11.867 INFO:teuthology.orchestra.run.smithi192.stdout:Get:40 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-urllib3 all 1.26.5-1~exp1ubuntu0.1 [98.2 kB] 2024-04-06T09:27:11.869 INFO:teuthology.orchestra.run.smithi192.stdout:Get:41 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-requests all 2.25.1+dfsg-2ubuntu0.1 [48.8 kB] 2024-04-06T09:27:11.870 INFO:teuthology.orchestra.run.smithi192.stdout:Get:42 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-werkzeug all 2.0.2+dfsg1-1ubuntu0.22.04.1 [181 kB] 2024-04-06T09:27:11.871 INFO:teuthology.orchestra.run.smithi006.stdout:Get:27 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-simplegeneric all 0.8.1-3 [11.3 kB] 2024-04-06T09:27:11.872 INFO:teuthology.orchestra.run.smithi006.stdout:Get:28 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-singledispatch all 3.4.0.3-3 [7,320 B] 2024-04-06T09:27:11.872 INFO:teuthology.orchestra.run.smithi006.stdout:Get:29 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-webob all 1:1.8.6-1.1 [85.2 kB] 2024-04-06T09:27:11.874 INFO:teuthology.orchestra.run.smithi006.stdout:Get:30 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-waitress all 1.4.4-1.1ubuntu1 [47.3 kB] 2024-04-06T09:27:11.876 INFO:teuthology.orchestra.run.smithi006.stdout:Get:31 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-tempita all 0.5.2-6ubuntu1 [15.1 kB] 2024-04-06T09:27:11.881 INFO:teuthology.orchestra.run.smithi192.stdout:Get:43 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-openssl all 21.0.0-1 [45.2 kB] 2024-04-06T09:27:11.882 INFO:teuthology.orchestra.run.smithi192.stdout:Get:44 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-asyncssh all 2.5.0-1 [185 kB] 2024-04-06T09:27:11.886 INFO:teuthology.orchestra.run.smithi192.stdout:Get:45 http://archive.ubuntu.com/ubuntu jammy/main amd64 python-babel-localedata all 2.8.0+dfsg.1-7 [4,982 kB] 2024-04-06T09:27:11.890 INFO:teuthology.orchestra.run.smithi192.stdout:Get:46 https://3.chacra.ceph.com/r/ceph/reef/4d05f4cad464de0fcdd11a2856643e533aea34e8/ubuntu/jammy/flavors/default jammy/main amd64 libcephfs2 amd64 18.2.2-1059-g4d05f4ca-1jammy [679 kB] 2024-04-06T09:27:11.898 INFO:teuthology.orchestra.run.smithi006.stdout:Get:32 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-paste all 3.5.0+dfsg1-1 [456 kB] 2024-04-06T09:27:11.910 INFO:teuthology.orchestra.run.smithi006.stdout:Get:33 http://archive.ubuntu.com/ubuntu jammy/main amd64 python-pastedeploy-tpl all 2.1.1-1 [4,892 B] 2024-04-06T09:27:11.911 INFO:teuthology.orchestra.run.smithi006.stdout:Get:34 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pastedeploy all 2.1.1-1 [26.6 kB] 2024-04-06T09:27:11.911 INFO:teuthology.orchestra.run.smithi192.stdout:Get:47 https://3.chacra.ceph.com/r/ceph/reef/4d05f4cad464de0fcdd11a2856643e533aea34e8/ubuntu/jammy/flavors/default jammy/main amd64 python3-rados amd64 18.2.2-1059-g4d05f4ca-1jammy [356 kB] 2024-04-06T09:27:11.912 INFO:teuthology.orchestra.run.smithi006.stdout:Get:35 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-webtest all 2.0.35-1 [28.5 kB] 2024-04-06T09:27:11.912 INFO:teuthology.orchestra.run.smithi006.stdout:Get:36 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pecan all 1.3.3-4ubuntu2 [87.3 kB] 2024-04-06T09:27:11.925 INFO:teuthology.orchestra.run.smithi192.stdout:Get:48 https://3.chacra.ceph.com/r/ceph/reef/4d05f4cad464de0fcdd11a2856643e533aea34e8/ubuntu/jammy/flavors/default jammy/main amd64 python3-ceph-argparse all 18.2.2-1059-g4d05f4ca-1jammy [32.8 kB] 2024-04-06T09:27:11.926 INFO:teuthology.orchestra.run.smithi192.stdout:Get:49 https://3.chacra.ceph.com/r/ceph/reef/4d05f4cad464de0fcdd11a2856643e533aea34e8/ubuntu/jammy/flavors/default jammy/main amd64 python3-cephfs amd64 18.2.2-1059-g4d05f4ca-1jammy [180 kB] 2024-04-06T09:27:11.933 INFO:teuthology.orchestra.run.smithi192.stdout:Get:50 https://3.chacra.ceph.com/r/ceph/reef/4d05f4cad464de0fcdd11a2856643e533aea34e8/ubuntu/jammy/flavors/default jammy/main amd64 python3-ceph-common all 18.2.2-1059-g4d05f4ca-1jammy [63.7 kB] 2024-04-06T09:27:11.935 INFO:teuthology.orchestra.run.smithi192.stdout:Get:51 https://3.chacra.ceph.com/r/ceph/reef/4d05f4cad464de0fcdd11a2856643e533aea34e8/ubuntu/jammy/flavors/default jammy/main amd64 python3-rbd amd64 18.2.2-1059-g4d05f4ca-1jammy [330 kB] 2024-04-06T09:27:11.940 INFO:teuthology.orchestra.run.smithi006.stdout:Get:37 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-certifi all 2020.6.20-1 [150 kB] 2024-04-06T09:27:11.944 INFO:teuthology.orchestra.run.smithi006.stdout:Get:38 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-idna all 3.3-1 [49.3 kB] 2024-04-06T09:27:11.945 INFO:teuthology.orchestra.run.smithi006.stdout:Get:39 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-urllib3 all 1.26.5-1~exp1ubuntu0.1 [98.2 kB] 2024-04-06T09:27:11.946 INFO:teuthology.orchestra.run.smithi192.stdout:Get:52 https://3.chacra.ceph.com/r/ceph/reef/4d05f4cad464de0fcdd11a2856643e533aea34e8/ubuntu/jammy/flavors/default jammy/main amd64 librgw2 amd64 18.2.2-1059-g4d05f4ca-1jammy [5,890 kB] 2024-04-06T09:27:11.948 INFO:teuthology.orchestra.run.smithi006.stdout:Get:40 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-requests all 2.25.1+dfsg-2ubuntu0.1 [48.8 kB] 2024-04-06T09:27:11.949 INFO:teuthology.orchestra.run.smithi006.stdout:Get:41 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-werkzeug all 2.0.2+dfsg1-1ubuntu0.22.04.1 [181 kB] 2024-04-06T09:27:11.960 INFO:teuthology.orchestra.run.smithi192.stdout:Get:53 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-babel all 2.8.0+dfsg.1-7 [85.1 kB] 2024-04-06T09:27:11.961 INFO:teuthology.orchestra.run.smithi192.stdout:Get:54 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-jinja2 all 3.0.3-1ubuntu0.1 [108 kB] 2024-04-06T09:27:11.963 INFO:teuthology.orchestra.run.smithi192.stdout:Get:55 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-bcrypt amd64 3.2.0-1build1 [32.7 kB] 2024-04-06T09:27:11.963 INFO:teuthology.orchestra.run.smithi192.stdout:Get:56 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-pluggy all 0.13.0-7.1 [19.0 kB] 2024-04-06T09:27:11.964 INFO:teuthology.orchestra.run.smithi192.stdout:Get:57 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-py all 1.10.0-1 [71.9 kB] 2024-04-06T09:27:11.965 INFO:teuthology.orchestra.run.smithi192.stdout:Get:58 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-iniconfig all 1.1.1-2 [6,024 B] 2024-04-06T09:27:11.966 INFO:teuthology.orchestra.run.smithi192.stdout:Get:59 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-toml all 0.10.2-1 [16.5 kB] 2024-04-06T09:27:11.966 INFO:teuthology.orchestra.run.smithi192.stdout:Get:60 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-pytest all 6.2.5-1ubuntu2 [214 kB] 2024-04-06T09:27:11.969 INFO:teuthology.orchestra.run.smithi192.stdout:Get:61 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-repoze.lru all 0.7-2 [12.1 kB] 2024-04-06T09:27:11.975 INFO:teuthology.orchestra.run.smithi006.stdout:Get:42 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-openssl all 21.0.0-1 [45.2 kB] 2024-04-06T09:27:11.976 INFO:teuthology.orchestra.run.smithi006.stdout:Get:43 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-asyncssh all 2.5.0-1 [185 kB] 2024-04-06T09:27:11.981 INFO:teuthology.orchestra.run.smithi006.stdout:Get:44 http://archive.ubuntu.com/ubuntu jammy/main amd64 python-babel-localedata all 2.8.0+dfsg.1-7 [4,982 kB] 2024-04-06T09:27:12.005 INFO:teuthology.orchestra.run.smithi192.stdout:Get:62 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-routes all 2.5.1-1ubuntu1 [89.0 kB] 2024-04-06T09:27:12.006 INFO:teuthology.orchestra.run.smithi192.stdout:Get:63 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-sklearn-lib amd64 0.23.2-5ubuntu6 [2,058 kB] 2024-04-06T09:27:12.033 INFO:teuthology.orchestra.run.smithi192.stdout:Get:64 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-joblib all 0.17.0-4ubuntu1 [204 kB] 2024-04-06T09:27:12.059 INFO:teuthology.orchestra.run.smithi192.stdout:Get:65 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-threadpoolctl all 3.1.0-1 [21.3 kB] 2024-04-06T09:27:12.060 INFO:teuthology.orchestra.run.smithi192.stdout:Get:66 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-sklearn all 0.23.2-5ubuntu6 [1,829 kB] 2024-04-06T09:27:12.082 INFO:teuthology.orchestra.run.smithi192.stdout:Get:67 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-cachetools all 5.0.0-1 [9,722 B] 2024-04-06T09:27:12.083 INFO:teuthology.orchestra.run.smithi192.stdout:Get:68 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyasn1 all 0.4.8-1 [50.9 kB] 2024-04-06T09:27:12.083 INFO:teuthology.orchestra.run.smithi192.stdout:Get:69 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyasn1-modules all 0.2.1-1 [33.0 kB] 2024-04-06T09:27:12.084 INFO:teuthology.orchestra.run.smithi192.stdout:Get:70 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-rsa all 4.8-1 [28.4 kB] 2024-04-06T09:27:12.084 INFO:teuthology.orchestra.run.smithi192.stdout:Get:71 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-google-auth all 1.5.1-3 [35.7 kB] 2024-04-06T09:27:12.093 INFO:teuthology.orchestra.run.smithi006.stdout:Get:45 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-babel all 2.8.0+dfsg.1-7 [85.1 kB] 2024-04-06T09:27:12.094 INFO:teuthology.orchestra.run.smithi006.stdout:Get:46 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-jinja2 all 3.0.3-1ubuntu0.1 [108 kB] 2024-04-06T09:27:12.096 INFO:teuthology.orchestra.run.smithi006.stdout:Get:47 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-bcrypt amd64 3.2.0-1build1 [32.7 kB] 2024-04-06T09:27:12.097 INFO:teuthology.orchestra.run.smithi006.stdout:Get:48 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-pluggy all 0.13.0-7.1 [19.0 kB] 2024-04-06T09:27:12.097 INFO:teuthology.orchestra.run.smithi006.stdout:Get:49 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-py all 1.10.0-1 [71.9 kB] 2024-04-06T09:27:12.099 INFO:teuthology.orchestra.run.smithi006.stdout:Get:50 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-iniconfig all 1.1.1-2 [6,024 B] 2024-04-06T09:27:12.105 INFO:teuthology.orchestra.run.smithi192.stdout:Get:72 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-requests-oauthlib all 1.3.0+ds-0.1 [18.7 kB] 2024-04-06T09:27:12.105 INFO:teuthology.orchestra.run.smithi192.stdout:Get:73 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-websocket all 1.2.3-1 [34.7 kB] 2024-04-06T09:27:12.106 INFO:teuthology.orchestra.run.smithi192.stdout:Get:74 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-kubernetes all 12.0.1-1ubuntu1 [353 kB] 2024-04-06T09:27:12.119 INFO:teuthology.orchestra.run.smithi006.stdout:Get:51 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-toml all 0.10.2-1 [16.5 kB] 2024-04-06T09:27:12.119 INFO:teuthology.orchestra.run.smithi006.stdout:Get:52 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-pytest all 6.2.5-1ubuntu2 [214 kB] 2024-04-06T09:27:12.123 INFO:teuthology.orchestra.run.smithi006.stdout:Get:53 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-repoze.lru all 0.7-2 [12.1 kB] 2024-04-06T09:27:12.123 INFO:teuthology.orchestra.run.smithi006.stdout:Get:54 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-routes all 2.5.1-1ubuntu1 [89.0 kB] 2024-04-06T09:27:12.127 INFO:teuthology.orchestra.run.smithi192.stdout:Get:75 http://archive.ubuntu.com/ubuntu jammy/main amd64 libonig5 amd64 6.9.7.1-2build1 [172 kB] 2024-04-06T09:27:12.129 INFO:teuthology.orchestra.run.smithi192.stdout:Get:76 http://archive.ubuntu.com/ubuntu jammy/main amd64 libjq1 amd64 1.6-2.1ubuntu3 [133 kB] 2024-04-06T09:27:12.132 INFO:teuthology.orchestra.run.smithi192.stdout:Get:77 http://archive.ubuntu.com/ubuntu jammy/main amd64 jq amd64 1.6-2.1ubuntu3 [52.5 kB] 2024-04-06T09:27:12.132 INFO:teuthology.orchestra.run.smithi192.stdout:Get:78 http://archive.ubuntu.com/ubuntu jammy/main amd64 socat amd64 1.7.4.1-3ubuntu4 [349 kB] 2024-04-06T09:27:12.138 INFO:teuthology.orchestra.run.smithi192.stdout:Get:79 http://archive.ubuntu.com/ubuntu jammy/universe amd64 xmlstarlet amd64 1.6.1-2.1 [265 kB] 2024-04-06T09:27:12.141 INFO:teuthology.orchestra.run.smithi192.stdout:Get:80 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 nvme-cli amd64 1.16-3ubuntu0.1 [474 kB] 2024-04-06T09:27:12.148 INFO:teuthology.orchestra.run.smithi006.stdout:Get:55 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-sklearn-lib amd64 0.23.2-5ubuntu6 [2,058 kB] 2024-04-06T09:27:12.148 INFO:teuthology.orchestra.run.smithi192.stdout:Get:81 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python-asyncssh-doc all 2.5.0-1 [318 kB] 2024-04-06T09:27:12.149 INFO:teuthology.orchestra.run.smithi192.stdout:Get:82 https://3.chacra.ceph.com/r/ceph/reef/4d05f4cad464de0fcdd11a2856643e533aea34e8/ubuntu/jammy/flavors/default jammy/main amd64 python3-rgw amd64 18.2.2-1059-g4d05f4ca-1jammy [112 kB] 2024-04-06T09:27:12.151 INFO:teuthology.orchestra.run.smithi192.stdout:Get:83 https://3.chacra.ceph.com/r/ceph/reef/4d05f4cad464de0fcdd11a2856643e533aea34e8/ubuntu/jammy/flavors/default jammy/main amd64 libradosstriper1 amd64 18.2.2-1059-g4d05f4ca-1jammy [441 kB] 2024-04-06T09:27:12.161 INFO:teuthology.orchestra.run.smithi192.stdout:Get:84 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pastescript all 2.0.2-4 [54.6 kB] 2024-04-06T09:27:12.162 INFO:teuthology.orchestra.run.smithi192.stdout:Get:85 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-psutil amd64 5.9.0-1build1 [158 kB] 2024-04-06T09:27:12.165 INFO:teuthology.orchestra.run.smithi192.stdout:Get:86 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pygments all 2.11.2+dfsg-2 [750 kB] 2024-04-06T09:27:12.169 INFO:teuthology.orchestra.run.smithi192.stdout:Get:87 https://3.chacra.ceph.com/r/ceph/reef/4d05f4cad464de0fcdd11a2856643e533aea34e8/ubuntu/jammy/flavors/default jammy/main amd64 ceph-common amd64 18.2.2-1059-g4d05f4ca-1jammy [21.9 MB] 2024-04-06T09:27:12.186 INFO:teuthology.orchestra.run.smithi006.stdout:Get:56 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-joblib all 0.17.0-4ubuntu1 [204 kB] 2024-04-06T09:27:12.190 INFO:teuthology.orchestra.run.smithi006.stdout:Get:57 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-threadpoolctl all 3.1.0-1 [21.3 kB] 2024-04-06T09:27:12.190 INFO:teuthology.orchestra.run.smithi006.stdout:Get:58 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-sklearn all 0.23.2-5ubuntu6 [1,829 kB] 2024-04-06T09:27:12.207 INFO:teuthology.orchestra.run.smithi192.stdout:Get:88 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyinotify all 0.9.6-1.3 [24.8 kB] 2024-04-06T09:27:12.208 INFO:teuthology.orchestra.run.smithi192.stdout:Get:89 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-simplejson amd64 3.17.6-1build1 [54.7 kB] 2024-04-06T09:27:12.209 INFO:teuthology.orchestra.run.smithi192.stdout:Get:90 http://archive.ubuntu.com/ubuntu jammy/universe amd64 qttranslations5-l10n all 5.15.3-1 [1,983 kB] 2024-04-06T09:27:12.217 INFO:teuthology.orchestra.run.smithi006.stdout:Get:59 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-cachetools all 5.0.0-1 [9,722 B] 2024-04-06T09:27:12.217 INFO:teuthology.orchestra.run.smithi006.stdout:Get:60 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyasn1 all 0.4.8-1 [50.9 kB] 2024-04-06T09:27:12.225 INFO:teuthology.orchestra.run.smithi006.stdout:Get:61 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyasn1-modules all 0.2.1-1 [33.0 kB] 2024-04-06T09:27:12.226 INFO:teuthology.orchestra.run.smithi006.stdout:Get:62 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-rsa all 4.8-1 [28.4 kB] 2024-04-06T09:27:12.227 INFO:teuthology.orchestra.run.smithi006.stdout:Get:63 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-google-auth all 1.5.1-3 [35.7 kB] 2024-04-06T09:27:12.229 INFO:teuthology.orchestra.run.smithi006.stdout:Get:64 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-requests-oauthlib all 1.3.0+ds-0.1 [18.7 kB] 2024-04-06T09:27:12.238 INFO:teuthology.orchestra.run.smithi006.stdout:Get:65 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-websocket all 1.2.3-1 [34.7 kB] 2024-04-06T09:27:12.250 INFO:teuthology.orchestra.run.smithi006.stdout:Get:66 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-kubernetes all 12.0.1-1ubuntu1 [353 kB] 2024-04-06T09:27:12.255 INFO:teuthology.orchestra.run.smithi006.stdout:Get:67 http://archive.ubuntu.com/ubuntu jammy/main amd64 libonig5 amd64 6.9.7.1-2build1 [172 kB] 2024-04-06T09:27:12.257 INFO:teuthology.orchestra.run.smithi006.stdout:Get:68 http://archive.ubuntu.com/ubuntu jammy/main amd64 libjq1 amd64 1.6-2.1ubuntu3 [133 kB] 2024-04-06T09:27:12.259 INFO:teuthology.orchestra.run.smithi006.stdout:Get:69 http://archive.ubuntu.com/ubuntu jammy/main amd64 jq amd64 1.6-2.1ubuntu3 [52.5 kB] 2024-04-06T09:27:12.260 INFO:teuthology.orchestra.run.smithi006.stdout:Get:70 http://archive.ubuntu.com/ubuntu jammy/main amd64 socat amd64 1.7.4.1-3ubuntu4 [349 kB] 2024-04-06T09:27:12.264 INFO:teuthology.orchestra.run.smithi006.stdout:Get:71 http://archive.ubuntu.com/ubuntu jammy/universe amd64 xmlstarlet amd64 1.6.1-2.1 [265 kB] 2024-04-06T09:27:12.267 INFO:teuthology.orchestra.run.smithi006.stdout:Get:72 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 nvme-cli amd64 1.16-3ubuntu0.1 [474 kB] 2024-04-06T09:27:12.272 INFO:teuthology.orchestra.run.smithi006.stdout:Get:73 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python-asyncssh-doc all 2.5.0-1 [318 kB] 2024-04-06T09:27:12.276 INFO:teuthology.orchestra.run.smithi006.stdout:Get:74 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pastescript all 2.0.2-4 [54.6 kB] 2024-04-06T09:27:12.288 INFO:teuthology.orchestra.run.smithi006.stdout:Get:75 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-psutil amd64 5.9.0-1build1 [158 kB] 2024-04-06T09:27:12.289 INFO:teuthology.orchestra.run.smithi006.stdout:Get:76 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pygments all 2.11.2+dfsg-2 [750 kB] 2024-04-06T09:27:12.298 INFO:teuthology.orchestra.run.smithi006.stdout:Get:77 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyinotify all 0.9.6-1.3 [24.8 kB] 2024-04-06T09:27:12.298 INFO:teuthology.orchestra.run.smithi006.stdout:Get:78 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-simplejson amd64 3.17.6-1build1 [54.7 kB] 2024-04-06T09:27:12.299 INFO:teuthology.orchestra.run.smithi006.stdout:Get:79 http://archive.ubuntu.com/ubuntu jammy/universe amd64 qttranslations5-l10n all 5.15.3-1 [1,983 kB] 2024-04-06T09:27:13.013 INFO:teuthology.orchestra.run.smithi192.stdout:Get:91 https://3.chacra.ceph.com/r/ceph/reef/4d05f4cad464de0fcdd11a2856643e533aea34e8/ubuntu/jammy/flavors/default jammy/main amd64 ceph-base amd64 18.2.2-1059-g4d05f4ca-1jammy [5,001 kB] 2024-04-06T09:27:13.126 INFO:teuthology.orchestra.run.smithi192.stdout:Get:92 https://3.chacra.ceph.com/r/ceph/reef/4d05f4cad464de0fcdd11a2856643e533aea34e8/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-modules-core all 18.2.2-1059-g4d05f4ca-1jammy [241 kB] 2024-04-06T09:27:13.133 INFO:teuthology.orchestra.run.smithi192.stdout:Get:93 https://3.chacra.ceph.com/r/ceph/reef/4d05f4cad464de0fcdd11a2856643e533aea34e8/ubuntu/jammy/flavors/default jammy/main amd64 libsqlite3-mod-ceph amd64 18.2.2-1059-g4d05f4ca-1jammy [130 kB] 2024-04-06T09:27:13.136 INFO:teuthology.orchestra.run.smithi192.stdout:Get:94 https://3.chacra.ceph.com/r/ceph/reef/4d05f4cad464de0fcdd11a2856643e533aea34e8/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr amd64 18.2.2-1059-g4d05f4ca-1jammy [1,490 kB] 2024-04-06T09:27:13.193 INFO:teuthology.orchestra.run.smithi192.stdout:Get:95 https://3.chacra.ceph.com/r/ceph/reef/4d05f4cad464de0fcdd11a2856643e533aea34e8/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mon amd64 18.2.2-1059-g4d05f4ca-1jammy [6,079 kB] 2024-04-06T09:27:13.351 INFO:teuthology.orchestra.run.smithi006.stdout:Get:80 https://3.chacra.ceph.com/r/ceph/reef/4d05f4cad464de0fcdd11a2856643e533aea34e8/ubuntu/jammy/flavors/default jammy/main amd64 librbd1 amd64 18.2.2-1059-g4d05f4ca-1jammy [3,105 kB] 2024-04-06T09:27:13.421 INFO:teuthology.orchestra.run.smithi192.stdout:Get:96 https://3.chacra.ceph.com/r/ceph/reef/4d05f4cad464de0fcdd11a2856643e533aea34e8/ubuntu/jammy/flavors/default jammy/main amd64 ceph-osd amd64 18.2.2-1059-g4d05f4ca-1jammy [22.3 MB] 2024-04-06T09:27:13.962 INFO:teuthology.orchestra.run.smithi006.stdout:Get:81 https://3.chacra.ceph.com/r/ceph/reef/4d05f4cad464de0fcdd11a2856643e533aea34e8/ubuntu/jammy/flavors/default jammy/main amd64 libcephfs2 amd64 18.2.2-1059-g4d05f4ca-1jammy [679 kB] 2024-04-06T09:27:14.082 INFO:teuthology.orchestra.run.smithi006.stdout:Get:82 https://3.chacra.ceph.com/r/ceph/reef/4d05f4cad464de0fcdd11a2856643e533aea34e8/ubuntu/jammy/flavors/default jammy/main amd64 python3-rados amd64 18.2.2-1059-g4d05f4ca-1jammy [356 kB] 2024-04-06T09:27:14.143 INFO:teuthology.orchestra.run.smithi006.stdout:Get:83 https://3.chacra.ceph.com/r/ceph/reef/4d05f4cad464de0fcdd11a2856643e533aea34e8/ubuntu/jammy/flavors/default jammy/main amd64 python3-ceph-argparse all 18.2.2-1059-g4d05f4ca-1jammy [32.8 kB] 2024-04-06T09:27:14.147 INFO:teuthology.orchestra.run.smithi006.stdout:Get:84 https://3.chacra.ceph.com/r/ceph/reef/4d05f4cad464de0fcdd11a2856643e533aea34e8/ubuntu/jammy/flavors/default jammy/main amd64 python3-cephfs amd64 18.2.2-1059-g4d05f4ca-1jammy [180 kB] 2024-04-06T09:27:14.175 INFO:teuthology.orchestra.run.smithi006.stdout:Get:85 https://3.chacra.ceph.com/r/ceph/reef/4d05f4cad464de0fcdd11a2856643e533aea34e8/ubuntu/jammy/flavors/default jammy/main amd64 python3-ceph-common all 18.2.2-1059-g4d05f4ca-1jammy [63.7 kB] 2024-04-06T09:27:14.186 INFO:teuthology.orchestra.run.smithi006.stdout:Get:86 https://3.chacra.ceph.com/r/ceph/reef/4d05f4cad464de0fcdd11a2856643e533aea34e8/ubuntu/jammy/flavors/default jammy/main amd64 python3-rbd amd64 18.2.2-1059-g4d05f4ca-1jammy [330 kB] 2024-04-06T09:27:14.242 INFO:teuthology.orchestra.run.smithi006.stdout:Get:87 https://3.chacra.ceph.com/r/ceph/reef/4d05f4cad464de0fcdd11a2856643e533aea34e8/ubuntu/jammy/flavors/default jammy/main amd64 librgw2 amd64 18.2.2-1059-g4d05f4ca-1jammy [5,890 kB] 2024-04-06T09:27:14.396 INFO:teuthology.orchestra.run.smithi192.stdout:Get:97 https://3.chacra.ceph.com/r/ceph/reef/4d05f4cad464de0fcdd11a2856643e533aea34e8/ubuntu/jammy/flavors/default jammy/main amd64 ceph amd64 18.2.2-1059-g4d05f4ca-1jammy [14.1 kB] 2024-04-06T09:27:14.397 INFO:teuthology.orchestra.run.smithi192.stdout:Get:98 https://3.chacra.ceph.com/r/ceph/reef/4d05f4cad464de0fcdd11a2856643e533aea34e8/ubuntu/jammy/flavors/default jammy/main amd64 ceph-fuse amd64 18.2.2-1059-g4d05f4ca-1jammy [831 kB] 2024-04-06T09:27:14.413 INFO:teuthology.orchestra.run.smithi192.stdout:Get:99 https://3.chacra.ceph.com/r/ceph/reef/4d05f4cad464de0fcdd11a2856643e533aea34e8/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mds amd64 18.2.2-1059-g4d05f4ca-1jammy [2,153 kB] 2024-04-06T09:27:14.450 INFO:teuthology.orchestra.run.smithi192.stdout:Get:100 https://3.chacra.ceph.com/r/ceph/reef/4d05f4cad464de0fcdd11a2856643e533aea34e8/ubuntu/jammy/flavors/default jammy/main amd64 cephadm amd64 18.2.2-1059-g4d05f4ca-1jammy [238 kB] 2024-04-06T09:27:14.453 INFO:teuthology.orchestra.run.smithi192.stdout:Get:101 https://3.chacra.ceph.com/r/ceph/reef/4d05f4cad464de0fcdd11a2856643e533aea34e8/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-cephadm all 18.2.2-1059-g4d05f4ca-1jammy [144 kB] 2024-04-06T09:27:14.458 INFO:teuthology.orchestra.run.smithi192.stdout:Get:102 https://3.chacra.ceph.com/r/ceph/reef/4d05f4cad464de0fcdd11a2856643e533aea34e8/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-dashboard all 18.2.2-1059-g4d05f4ca-1jammy [1,932 kB] 2024-04-06T09:27:14.541 INFO:teuthology.orchestra.run.smithi192.stdout:Get:103 https://3.chacra.ceph.com/r/ceph/reef/4d05f4cad464de0fcdd11a2856643e533aea34e8/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-diskprediction-local all 18.2.2-1059-g4d05f4ca-1jammy [8,625 kB] 2024-04-06T09:27:14.922 INFO:teuthology.orchestra.run.smithi192.stdout:Get:104 https://3.chacra.ceph.com/r/ceph/reef/4d05f4cad464de0fcdd11a2856643e533aea34e8/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-k8sevents all 18.2.2-1059-g4d05f4ca-1jammy [14.3 kB] 2024-04-06T09:27:14.923 INFO:teuthology.orchestra.run.smithi192.stdout:Get:105 https://3.chacra.ceph.com/r/ceph/reef/4d05f4cad464de0fcdd11a2856643e533aea34e8/ubuntu/jammy/flavors/default jammy/main amd64 ceph-test amd64 18.2.2-1059-g4d05f4ca-1jammy [42.6 MB] 2024-04-06T09:27:15.013 INFO:teuthology.orchestra.run.smithi006.stdout:Get:88 https://3.chacra.ceph.com/r/ceph/reef/4d05f4cad464de0fcdd11a2856643e533aea34e8/ubuntu/jammy/flavors/default jammy/main amd64 python3-rgw amd64 18.2.2-1059-g4d05f4ca-1jammy [112 kB] 2024-04-06T09:27:15.025 INFO:teuthology.orchestra.run.smithi006.stdout:Get:89 https://3.chacra.ceph.com/r/ceph/reef/4d05f4cad464de0fcdd11a2856643e533aea34e8/ubuntu/jammy/flavors/default jammy/main amd64 libradosstriper1 amd64 18.2.2-1059-g4d05f4ca-1jammy [441 kB] 2024-04-06T09:27:15.078 INFO:teuthology.orchestra.run.smithi006.stdout:Get:90 https://3.chacra.ceph.com/r/ceph/reef/4d05f4cad464de0fcdd11a2856643e533aea34e8/ubuntu/jammy/flavors/default jammy/main amd64 ceph-common amd64 18.2.2-1059-g4d05f4ca-1jammy [21.9 MB] 2024-04-06T09:27:17.115 INFO:teuthology.orchestra.run.smithi192.stdout:Get:106 https://3.chacra.ceph.com/r/ceph/reef/4d05f4cad464de0fcdd11a2856643e533aea34e8/ubuntu/jammy/flavors/default jammy/main amd64 ceph-volume all 18.2.2-1059-g4d05f4ca-1jammy [121 kB] 2024-04-06T09:27:17.117 INFO:teuthology.orchestra.run.smithi192.stdout:Get:107 https://3.chacra.ceph.com/r/ceph/reef/4d05f4cad464de0fcdd11a2856643e533aea34e8/ubuntu/jammy/flavors/default jammy/main amd64 libcephfs-dev amd64 18.2.2-1059-g4d05f4ca-1jammy [38.3 kB] 2024-04-06T09:27:17.119 INFO:teuthology.orchestra.run.smithi192.stdout:Get:108 https://3.chacra.ceph.com/r/ceph/reef/4d05f4cad464de0fcdd11a2856643e533aea34e8/ubuntu/jammy/flavors/default jammy/main amd64 radosgw amd64 18.2.2-1059-g4d05f4ca-1jammy [11.6 MB] 2024-04-06T09:27:17.435 INFO:teuthology.orchestra.run.smithi006.stdout:Get:91 https://3.chacra.ceph.com/r/ceph/reef/4d05f4cad464de0fcdd11a2856643e533aea34e8/ubuntu/jammy/flavors/default jammy/main amd64 ceph-base amd64 18.2.2-1059-g4d05f4ca-1jammy [5,001 kB] 2024-04-06T09:27:17.520 INFO:teuthology.orchestra.run.smithi192.stdout:Get:109 https://3.chacra.ceph.com/r/ceph/reef/4d05f4cad464de0fcdd11a2856643e533aea34e8/ubuntu/jammy/flavors/default jammy/main amd64 rbd-fuse amd64 18.2.2-1059-g4d05f4ca-1jammy [93.1 kB] 2024-04-06T09:27:17.523 INFO:teuthology.orchestra.run.smithi192.stdout:Get:110 https://3.chacra.ceph.com/r/ceph/reef/4d05f4cad464de0fcdd11a2856643e533aea34e8/ubuntu/jammy/flavors/default jammy/main amd64 rbd-mirror amd64 18.2.2-1059-g4d05f4ca-1jammy [3,031 kB] 2024-04-06T09:27:17.762 INFO:teuthology.orchestra.run.smithi006.stdout:Get:92 https://3.chacra.ceph.com/r/ceph/reef/4d05f4cad464de0fcdd11a2856643e533aea34e8/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-modules-core all 18.2.2-1059-g4d05f4ca-1jammy [241 kB] 2024-04-06T09:27:17.769 INFO:teuthology.orchestra.run.smithi006.stdout:Get:93 https://3.chacra.ceph.com/r/ceph/reef/4d05f4cad464de0fcdd11a2856643e533aea34e8/ubuntu/jammy/flavors/default jammy/main amd64 libsqlite3-mod-ceph amd64 18.2.2-1059-g4d05f4ca-1jammy [130 kB] 2024-04-06T09:27:17.774 INFO:teuthology.orchestra.run.smithi006.stdout:Get:94 https://3.chacra.ceph.com/r/ceph/reef/4d05f4cad464de0fcdd11a2856643e533aea34e8/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr amd64 18.2.2-1059-g4d05f4ca-1jammy [1,490 kB] 2024-04-06T09:27:17.841 INFO:teuthology.orchestra.run.smithi006.stdout:Get:95 https://3.chacra.ceph.com/r/ceph/reef/4d05f4cad464de0fcdd11a2856643e533aea34e8/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mon amd64 18.2.2-1059-g4d05f4ca-1jammy [6,079 kB] 2024-04-06T09:27:18.106 INFO:teuthology.orchestra.run.smithi006.stdout:Get:96 https://3.chacra.ceph.com/r/ceph/reef/4d05f4cad464de0fcdd11a2856643e533aea34e8/ubuntu/jammy/flavors/default jammy/main amd64 ceph-osd amd64 18.2.2-1059-g4d05f4ca-1jammy [22.3 MB] 2024-04-06T09:27:18.223 INFO:teuthology.orchestra.run.smithi192.stdout:Fetched 165 MB in 6s (26.5 MB/s) 2024-04-06T09:27:18.357 INFO:teuthology.orchestra.run.smithi192.stdout:Selecting previously unselected package libfmt8:amd64. 2024-04-06T09:27:18.446 INFO:teuthology.orchestra.run.smithi192.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 165997 files and directories currently installed.) 2024-04-06T09:27:18.451 INFO:teuthology.orchestra.run.smithi192.stdout:Preparing to unpack .../000-libfmt8_8.1.1+ds1-2_amd64.deb ... 2024-04-06T09:27:18.531 INFO:teuthology.orchestra.run.smithi192.stdout:Unpacking libfmt8:amd64 (8.1.1+ds1-2) ... 2024-04-06T09:27:18.851 INFO:teuthology.orchestra.run.smithi192.stdout:Selecting previously unselected package liblttng-ust1:amd64. 2024-04-06T09:27:18.877 INFO:teuthology.orchestra.run.smithi192.stdout:Preparing to unpack .../001-liblttng-ust1_2.13.1-1ubuntu1_amd64.deb ... 2024-04-06T09:27:18.917 INFO:teuthology.orchestra.run.smithi192.stdout:Unpacking liblttng-ust1:amd64 (2.13.1-1ubuntu1) ... 2024-04-06T09:27:19.219 INFO:teuthology.orchestra.run.smithi006.stdout:Get:97 https://3.chacra.ceph.com/r/ceph/reef/4d05f4cad464de0fcdd11a2856643e533aea34e8/ubuntu/jammy/flavors/default jammy/main amd64 ceph amd64 18.2.2-1059-g4d05f4ca-1jammy [14.1 kB] 2024-04-06T09:27:19.220 INFO:teuthology.orchestra.run.smithi006.stdout:Get:98 https://3.chacra.ceph.com/r/ceph/reef/4d05f4cad464de0fcdd11a2856643e533aea34e8/ubuntu/jammy/flavors/default jammy/main amd64 ceph-fuse amd64 18.2.2-1059-g4d05f4ca-1jammy [831 kB] 2024-04-06T09:27:19.236 INFO:teuthology.orchestra.run.smithi006.stdout:Get:99 https://3.chacra.ceph.com/r/ceph/reef/4d05f4cad464de0fcdd11a2856643e533aea34e8/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mds amd64 18.2.2-1059-g4d05f4ca-1jammy [2,153 kB] 2024-04-06T09:27:19.254 INFO:teuthology.orchestra.run.smithi192.stdout:Selecting previously unselected package libdouble-conversion3:amd64. 2024-04-06T09:27:19.281 INFO:teuthology.orchestra.run.smithi192.stdout:Preparing to unpack .../002-libdouble-conversion3_3.1.7-4_amd64.deb ... 2024-04-06T09:27:19.321 INFO:teuthology.orchestra.run.smithi192.stdout:Unpacking libdouble-conversion3:amd64 (3.1.7-4) ... 2024-04-06T09:27:19.325 INFO:teuthology.orchestra.run.smithi006.stdout:Get:100 https://3.chacra.ceph.com/r/ceph/reef/4d05f4cad464de0fcdd11a2856643e533aea34e8/ubuntu/jammy/flavors/default jammy/main amd64 cephadm amd64 18.2.2-1059-g4d05f4ca-1jammy [238 kB] 2024-04-06T09:27:19.338 INFO:teuthology.orchestra.run.smithi006.stdout:Get:101 https://3.chacra.ceph.com/r/ceph/reef/4d05f4cad464de0fcdd11a2856643e533aea34e8/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-cephadm all 18.2.2-1059-g4d05f4ca-1jammy [144 kB] 2024-04-06T09:27:19.342 INFO:teuthology.orchestra.run.smithi006.stdout:Get:102 https://3.chacra.ceph.com/r/ceph/reef/4d05f4cad464de0fcdd11a2856643e533aea34e8/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-dashboard all 18.2.2-1059-g4d05f4ca-1jammy [1,932 kB] 2024-04-06T09:27:19.443 INFO:teuthology.orchestra.run.smithi006.stdout:Get:103 https://3.chacra.ceph.com/r/ceph/reef/4d05f4cad464de0fcdd11a2856643e533aea34e8/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-diskprediction-local all 18.2.2-1059-g4d05f4ca-1jammy [8,625 kB] 2024-04-06T09:27:19.607 INFO:teuthology.orchestra.run.smithi192.stdout:Selecting previously unselected package libpcre2-16-0:amd64. 2024-04-06T09:27:19.633 INFO:teuthology.orchestra.run.smithi192.stdout:Preparing to unpack .../003-libpcre2-16-0_10.39-3ubuntu0.1_amd64.deb ... 2024-04-06T09:27:19.665 INFO:teuthology.orchestra.run.smithi192.stdout:Unpacking libpcre2-16-0:amd64 (10.39-3ubuntu0.1) ... 2024-04-06T09:27:19.791 INFO:teuthology.orchestra.run.smithi006.stdout:Get:104 https://3.chacra.ceph.com/r/ceph/reef/4d05f4cad464de0fcdd11a2856643e533aea34e8/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-k8sevents all 18.2.2-1059-g4d05f4ca-1jammy [14.3 kB] 2024-04-06T09:27:19.791 INFO:teuthology.orchestra.run.smithi006.stdout:Get:105 https://3.chacra.ceph.com/r/ceph/reef/4d05f4cad464de0fcdd11a2856643e533aea34e8/ubuntu/jammy/flavors/default jammy/main amd64 ceph-test amd64 18.2.2-1059-g4d05f4ca-1jammy [42.6 MB] 2024-04-06T09:27:20.127 INFO:teuthology.orchestra.run.smithi192.stdout:Selecting previously unselected package libqt5core5a:amd64. 2024-04-06T09:27:20.154 INFO:teuthology.orchestra.run.smithi192.stdout:Preparing to unpack .../004-libqt5core5a_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2024-04-06T09:27:20.202 INFO:teuthology.orchestra.run.smithi192.stdout:Unpacking libqt5core5a:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-04-06T09:27:20.580 INFO:teuthology.orchestra.run.smithi192.stdout:Selecting previously unselected package libqt5dbus5:amd64. 2024-04-06T09:27:20.603 INFO:teuthology.orchestra.run.smithi192.stdout:Preparing to unpack .../005-libqt5dbus5_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2024-04-06T09:27:20.638 INFO:teuthology.orchestra.run.smithi192.stdout:Unpacking libqt5dbus5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-04-06T09:27:20.933 INFO:teuthology.orchestra.run.smithi192.stdout:Selecting previously unselected package libqt5network5:amd64. 2024-04-06T09:27:20.959 INFO:teuthology.orchestra.run.smithi192.stdout:Preparing to unpack .../006-libqt5network5_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2024-04-06T09:27:21.000 INFO:teuthology.orchestra.run.smithi192.stdout:Unpacking libqt5network5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-04-06T09:27:21.403 INFO:teuthology.orchestra.run.smithi192.stdout:Selecting previously unselected package libthrift-0.16.0:amd64. 2024-04-06T09:27:21.429 INFO:teuthology.orchestra.run.smithi192.stdout:Preparing to unpack .../007-libthrift-0.16.0_0.16.0-2_amd64.deb ... 2024-04-06T09:27:21.469 INFO:teuthology.orchestra.run.smithi192.stdout:Unpacking libthrift-0.16.0:amd64 (0.16.0-2) ... 2024-04-06T09:27:21.696 INFO:teuthology.orchestra.run.smithi006.stdout:Get:106 https://3.chacra.ceph.com/r/ceph/reef/4d05f4cad464de0fcdd11a2856643e533aea34e8/ubuntu/jammy/flavors/default jammy/main amd64 ceph-volume all 18.2.2-1059-g4d05f4ca-1jammy [121 kB] 2024-04-06T09:27:21.698 INFO:teuthology.orchestra.run.smithi006.stdout:Get:107 https://3.chacra.ceph.com/r/ceph/reef/4d05f4cad464de0fcdd11a2856643e533aea34e8/ubuntu/jammy/flavors/default jammy/main amd64 libcephfs-dev amd64 18.2.2-1059-g4d05f4ca-1jammy [38.3 kB] 2024-04-06T09:27:21.699 INFO:teuthology.orchestra.run.smithi006.stdout:Get:108 https://3.chacra.ceph.com/r/ceph/reef/4d05f4cad464de0fcdd11a2856643e533aea34e8/ubuntu/jammy/flavors/default jammy/main amd64 radosgw amd64 18.2.2-1059-g4d05f4ca-1jammy [11.6 MB] 2024-04-06T09:27:21.806 INFO:teuthology.orchestra.run.smithi192.stdout:Selecting previously unselected package librados2. 2024-04-06T09:27:21.832 INFO:teuthology.orchestra.run.smithi192.stdout:Preparing to unpack .../008-librados2_18.2.2-1059-g4d05f4ca-1jammy_amd64.deb ... 2024-04-06T09:27:21.872 INFO:teuthology.orchestra.run.smithi192.stdout:Unpacking librados2 (18.2.2-1059-g4d05f4ca-1jammy) ... 2024-04-06T09:27:22.070 INFO:teuthology.orchestra.run.smithi006.stdout:Get:109 https://3.chacra.ceph.com/r/ceph/reef/4d05f4cad464de0fcdd11a2856643e533aea34e8/ubuntu/jammy/flavors/default jammy/main amd64 rbd-fuse amd64 18.2.2-1059-g4d05f4ca-1jammy [93.1 kB] 2024-04-06T09:27:22.084 INFO:teuthology.orchestra.run.smithi006.stdout:Get:110 https://3.chacra.ceph.com/r/ceph/reef/4d05f4cad464de0fcdd11a2856643e533aea34e8/ubuntu/jammy/flavors/default jammy/main amd64 rbd-mirror amd64 18.2.2-1059-g4d05f4ca-1jammy [3,031 kB] 2024-04-06T09:27:22.308 INFO:teuthology.orchestra.run.smithi192.stdout:Selecting previously unselected package librbd1. 2024-04-06T09:27:22.324 INFO:teuthology.orchestra.run.smithi192.stdout:Preparing to unpack .../009-librbd1_18.2.2-1059-g4d05f4ca-1jammy_amd64.deb ... 2024-04-06T09:27:22.358 INFO:teuthology.orchestra.run.smithi192.stdout:Unpacking librbd1 (18.2.2-1059-g4d05f4ca-1jammy) ... 2024-04-06T09:27:22.777 INFO:teuthology.orchestra.run.smithi006.stdout:Fetched 165 MB in 11s (15.4 MB/s) 2024-04-06T09:27:22.786 INFO:teuthology.orchestra.run.smithi192.stdout:Selecting previously unselected package libcephfs2. 2024-04-06T09:27:22.813 INFO:teuthology.orchestra.run.smithi192.stdout:Preparing to unpack .../010-libcephfs2_18.2.2-1059-g4d05f4ca-1jammy_amd64.deb ... 2024-04-06T09:27:22.853 INFO:teuthology.orchestra.run.smithi192.stdout:Unpacking libcephfs2 (18.2.2-1059-g4d05f4ca-1jammy) ... 2024-04-06T09:27:23.088 INFO:teuthology.orchestra.run.smithi006.stdout:Selecting previously unselected package libfmt8:amd64. 2024-04-06T09:27:23.130 INFO:teuthology.orchestra.run.smithi192.stdout:Selecting previously unselected package python3-rados. 2024-04-06T09:27:23.157 INFO:teuthology.orchestra.run.smithi192.stdout:Preparing to unpack .../011-python3-rados_18.2.2-1059-g4d05f4ca-1jammy_amd64.deb ... 2024-04-06T09:27:23.194 INFO:teuthology.orchestra.run.smithi006.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 165997 files and directories currently installed.) 2024-04-06T09:27:23.197 INFO:teuthology.orchestra.run.smithi192.stdout:Unpacking python3-rados (18.2.2-1059-g4d05f4ca-1jammy) ... 2024-04-06T09:27:23.200 INFO:teuthology.orchestra.run.smithi006.stdout:Preparing to unpack .../000-libfmt8_8.1.1+ds1-2_amd64.deb ... 2024-04-06T09:27:23.287 INFO:teuthology.orchestra.run.smithi006.stdout:Unpacking libfmt8:amd64 (8.1.1+ds1-2) ... 2024-04-06T09:27:23.466 INFO:teuthology.orchestra.run.smithi192.stdout:Selecting previously unselected package python3-ceph-argparse. 2024-04-06T09:27:23.493 INFO:teuthology.orchestra.run.smithi192.stdout:Preparing to unpack .../012-python3-ceph-argparse_18.2.2-1059-g4d05f4ca-1jammy_all.deb ... 2024-04-06T09:27:23.533 INFO:teuthology.orchestra.run.smithi192.stdout:Unpacking python3-ceph-argparse (18.2.2-1059-g4d05f4ca-1jammy) ... 2024-04-06T09:27:23.631 INFO:teuthology.orchestra.run.smithi006.stdout:Selecting previously unselected package liblttng-ust1:amd64. 2024-04-06T09:27:23.658 INFO:teuthology.orchestra.run.smithi006.stdout:Preparing to unpack .../001-liblttng-ust1_2.13.1-1ubuntu1_amd64.deb ... 2024-04-06T09:27:23.698 INFO:teuthology.orchestra.run.smithi006.stdout:Unpacking liblttng-ust1:amd64 (2.13.1-1ubuntu1) ... 2024-04-06T09:27:23.785 INFO:teuthology.orchestra.run.smithi192.stdout:Selecting previously unselected package python3-cephfs. 2024-04-06T09:27:23.800 INFO:teuthology.orchestra.run.smithi192.stdout:Preparing to unpack .../013-python3-cephfs_18.2.2-1059-g4d05f4ca-1jammy_amd64.deb ... 2024-04-06T09:27:23.852 INFO:teuthology.orchestra.run.smithi192.stdout:Unpacking python3-cephfs (18.2.2-1059-g4d05f4ca-1jammy) ... 2024-04-06T09:27:24.026 INFO:teuthology.orchestra.run.smithi006.stdout:Selecting previously unselected package libdouble-conversion3:amd64. 2024-04-06T09:27:24.054 INFO:teuthology.orchestra.run.smithi006.stdout:Preparing to unpack .../002-libdouble-conversion3_3.1.7-4_amd64.deb ... 2024-04-06T09:27:24.093 INFO:teuthology.orchestra.run.smithi006.stdout:Unpacking libdouble-conversion3:amd64 (3.1.7-4) ... 2024-04-06T09:27:24.172 INFO:teuthology.orchestra.run.smithi192.stdout:Selecting previously unselected package python3-ceph-common. 2024-04-06T09:27:24.198 INFO:teuthology.orchestra.run.smithi192.stdout:Preparing to unpack .../014-python3-ceph-common_18.2.2-1059-g4d05f4ca-1jammy_all.deb ... 2024-04-06T09:27:24.238 INFO:teuthology.orchestra.run.smithi192.stdout:Unpacking python3-ceph-common (18.2.2-1059-g4d05f4ca-1jammy) ... 2024-04-06T09:27:24.429 INFO:teuthology.orchestra.run.smithi006.stdout:Selecting previously unselected package libpcre2-16-0:amd64. 2024-04-06T09:27:24.456 INFO:teuthology.orchestra.run.smithi006.stdout:Preparing to unpack .../003-libpcre2-16-0_10.39-3ubuntu0.1_amd64.deb ... 2024-04-06T09:27:24.496 INFO:teuthology.orchestra.run.smithi006.stdout:Unpacking libpcre2-16-0:amd64 (10.39-3ubuntu0.1) ... 2024-04-06T09:27:24.558 INFO:teuthology.orchestra.run.smithi192.stdout:Selecting previously unselected package python3-wcwidth. 2024-04-06T09:27:24.585 INFO:teuthology.orchestra.run.smithi192.stdout:Preparing to unpack .../015-python3-wcwidth_0.2.5+dfsg1-1_all.deb ... 2024-04-06T09:27:24.624 INFO:teuthology.orchestra.run.smithi192.stdout:Unpacking python3-wcwidth (0.2.5+dfsg1-1) ... 2024-04-06T09:27:24.903 INFO:teuthology.orchestra.run.smithi192.stdout:Selecting previously unselected package python3-prettytable. 2024-04-06T09:27:24.924 INFO:teuthology.orchestra.run.smithi006.stdout:Selecting previously unselected package libqt5core5a:amd64. 2024-04-06T09:27:24.929 INFO:teuthology.orchestra.run.smithi192.stdout:Preparing to unpack .../016-python3-prettytable_2.5.0-2_all.deb ... 2024-04-06T09:27:24.951 INFO:teuthology.orchestra.run.smithi006.stdout:Preparing to unpack .../004-libqt5core5a_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2024-04-06T09:27:24.986 INFO:teuthology.orchestra.run.smithi192.stdout:Unpacking python3-prettytable (2.5.0-2) ... 2024-04-06T09:27:24.999 INFO:teuthology.orchestra.run.smithi006.stdout:Unpacking libqt5core5a:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-04-06T09:27:25.238 INFO:teuthology.orchestra.run.smithi192.stdout:Selecting previously unselected package python3-rbd. 2024-04-06T09:27:25.265 INFO:teuthology.orchestra.run.smithi192.stdout:Preparing to unpack .../017-python3-rbd_18.2.2-1059-g4d05f4ca-1jammy_amd64.deb ... 2024-04-06T09:27:25.305 INFO:teuthology.orchestra.run.smithi192.stdout:Unpacking python3-rbd (18.2.2-1059-g4d05f4ca-1jammy) ... 2024-04-06T09:27:25.386 INFO:teuthology.orchestra.run.smithi006.stdout:Selecting previously unselected package libqt5dbus5:amd64. 2024-04-06T09:27:25.412 INFO:teuthology.orchestra.run.smithi006.stdout:Preparing to unpack .../005-libqt5dbus5_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2024-04-06T09:27:25.452 INFO:teuthology.orchestra.run.smithi006.stdout:Unpacking libqt5dbus5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-04-06T09:27:25.658 INFO:teuthology.orchestra.run.smithi192.stdout:Selecting previously unselected package librdkafka1:amd64. 2024-04-06T09:27:25.684 INFO:teuthology.orchestra.run.smithi192.stdout:Preparing to unpack .../018-librdkafka1_1.8.0-1build1_amd64.deb ... 2024-04-06T09:27:25.724 INFO:teuthology.orchestra.run.smithi192.stdout:Unpacking librdkafka1:amd64 (1.8.0-1build1) ... 2024-04-06T09:27:25.772 INFO:teuthology.orchestra.run.smithi006.stdout:Selecting previously unselected package libqt5network5:amd64. 2024-04-06T09:27:25.799 INFO:teuthology.orchestra.run.smithi006.stdout:Preparing to unpack .../006-libqt5network5_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2024-04-06T09:27:25.830 INFO:teuthology.orchestra.run.smithi006.stdout:Unpacking libqt5network5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-04-06T09:27:26.060 INFO:teuthology.orchestra.run.smithi192.stdout:Selecting previously unselected package librgw2. 2024-04-06T09:27:26.087 INFO:teuthology.orchestra.run.smithi192.stdout:Preparing to unpack .../019-librgw2_18.2.2-1059-g4d05f4ca-1jammy_amd64.deb ... 2024-04-06T09:27:26.127 INFO:teuthology.orchestra.run.smithi192.stdout:Unpacking librgw2 (18.2.2-1059-g4d05f4ca-1jammy) ... 2024-04-06T09:27:26.175 INFO:teuthology.orchestra.run.smithi006.stdout:Selecting previously unselected package libthrift-0.16.0:amd64. 2024-04-06T09:27:26.201 INFO:teuthology.orchestra.run.smithi006.stdout:Preparing to unpack .../007-libthrift-0.16.0_0.16.0-2_amd64.deb ... 2024-04-06T09:27:26.241 INFO:teuthology.orchestra.run.smithi006.stdout:Unpacking libthrift-0.16.0:amd64 (0.16.0-2) ... 2024-04-06T09:27:26.561 INFO:teuthology.orchestra.run.smithi006.stdout:Selecting previously unselected package librados2. 2024-04-06T09:27:26.588 INFO:teuthology.orchestra.run.smithi006.stdout:Preparing to unpack .../008-librados2_18.2.2-1059-g4d05f4ca-1jammy_amd64.deb ... 2024-04-06T09:27:26.612 INFO:teuthology.orchestra.run.smithi192.stdout:Selecting previously unselected package python3-rgw. 2024-04-06T09:27:26.627 INFO:teuthology.orchestra.run.smithi006.stdout:Unpacking librados2 (18.2.2-1059-g4d05f4ca-1jammy) ... 2024-04-06T09:27:26.628 INFO:teuthology.orchestra.run.smithi192.stdout:Preparing to unpack .../020-python3-rgw_18.2.2-1059-g4d05f4ca-1jammy_amd64.deb ... 2024-04-06T09:27:26.663 INFO:teuthology.orchestra.run.smithi192.stdout:Unpacking python3-rgw (18.2.2-1059-g4d05f4ca-1jammy) ... 2024-04-06T09:27:27.007 INFO:teuthology.orchestra.run.smithi192.stdout:Selecting previously unselected package liboath0:amd64. 2024-04-06T09:27:27.034 INFO:teuthology.orchestra.run.smithi192.stdout:Preparing to unpack .../021-liboath0_2.6.7-3build1_amd64.deb ... 2024-04-06T09:27:27.074 INFO:teuthology.orchestra.run.smithi192.stdout:Unpacking liboath0:amd64 (2.6.7-3build1) ... 2024-04-06T09:27:27.088 INFO:teuthology.orchestra.run.smithi006.stdout:Selecting previously unselected package librbd1. 2024-04-06T09:27:27.115 INFO:teuthology.orchestra.run.smithi006.stdout:Preparing to unpack .../009-librbd1_18.2.2-1059-g4d05f4ca-1jammy_amd64.deb ... 2024-04-06T09:27:27.155 INFO:teuthology.orchestra.run.smithi006.stdout:Unpacking librbd1 (18.2.2-1059-g4d05f4ca-1jammy) ... 2024-04-06T09:27:27.393 INFO:teuthology.orchestra.run.smithi192.stdout:Selecting previously unselected package libradosstriper1. 2024-04-06T09:27:27.419 INFO:teuthology.orchestra.run.smithi192.stdout:Preparing to unpack .../022-libradosstriper1_18.2.2-1059-g4d05f4ca-1jammy_amd64.deb ... 2024-04-06T09:27:27.460 INFO:teuthology.orchestra.run.smithi192.stdout:Unpacking libradosstriper1 (18.2.2-1059-g4d05f4ca-1jammy) ... 2024-04-06T09:27:27.533 INFO:teuthology.orchestra.run.smithi006.stdout:Selecting previously unselected package libcephfs2. 2024-04-06T09:27:27.559 INFO:teuthology.orchestra.run.smithi006.stdout:Preparing to unpack .../010-libcephfs2_18.2.2-1059-g4d05f4ca-1jammy_amd64.deb ... 2024-04-06T09:27:27.599 INFO:teuthology.orchestra.run.smithi006.stdout:Unpacking libcephfs2 (18.2.2-1059-g4d05f4ca-1jammy) ... 2024-04-06T09:27:27.813 INFO:teuthology.orchestra.run.smithi192.stdout:Selecting previously unselected package ceph-common. 2024-04-06T09:27:27.839 INFO:teuthology.orchestra.run.smithi192.stdout:Preparing to unpack .../023-ceph-common_18.2.2-1059-g4d05f4ca-1jammy_amd64.deb ... 2024-04-06T09:27:27.871 INFO:teuthology.orchestra.run.smithi192.stdout:Unpacking ceph-common (18.2.2-1059-g4d05f4ca-1jammy) ... 2024-04-06T09:27:27.885 INFO:teuthology.orchestra.run.smithi006.stdout:Selecting previously unselected package python3-rados. 2024-04-06T09:27:27.901 INFO:teuthology.orchestra.run.smithi006.stdout:Preparing to unpack .../011-python3-rados_18.2.2-1059-g4d05f4ca-1jammy_amd64.deb ... 2024-04-06T09:27:27.943 INFO:teuthology.orchestra.run.smithi006.stdout:Unpacking python3-rados (18.2.2-1059-g4d05f4ca-1jammy) ... 2024-04-06T09:27:28.396 INFO:teuthology.orchestra.run.smithi006.stdout:Selecting previously unselected package python3-ceph-argparse. 2024-04-06T09:27:28.423 INFO:teuthology.orchestra.run.smithi006.stdout:Preparing to unpack .../012-python3-ceph-argparse_18.2.2-1059-g4d05f4ca-1jammy_all.deb ... 2024-04-06T09:27:28.463 INFO:teuthology.orchestra.run.smithi006.stdout:Unpacking python3-ceph-argparse (18.2.2-1059-g4d05f4ca-1jammy) ... 2024-04-06T09:27:28.773 INFO:teuthology.orchestra.run.smithi006.stdout:Selecting previously unselected package python3-cephfs. 2024-04-06T09:27:28.800 INFO:teuthology.orchestra.run.smithi006.stdout:Preparing to unpack .../013-python3-cephfs_18.2.2-1059-g4d05f4ca-1jammy_amd64.deb ... 2024-04-06T09:27:28.840 INFO:teuthology.orchestra.run.smithi006.stdout:Unpacking python3-cephfs (18.2.2-1059-g4d05f4ca-1jammy) ... 2024-04-06T09:27:29.118 INFO:teuthology.orchestra.run.smithi006.stdout:Selecting previously unselected package python3-ceph-common. 2024-04-06T09:27:29.145 INFO:teuthology.orchestra.run.smithi006.stdout:Preparing to unpack .../014-python3-ceph-common_18.2.2-1059-g4d05f4ca-1jammy_all.deb ... 2024-04-06T09:27:29.184 INFO:teuthology.orchestra.run.smithi006.stdout:Unpacking python3-ceph-common (18.2.2-1059-g4d05f4ca-1jammy) ... 2024-04-06T09:27:29.361 INFO:teuthology.orchestra.run.smithi192.stdout:Selecting previously unselected package ceph-base. 2024-04-06T09:27:29.387 INFO:teuthology.orchestra.run.smithi192.stdout:Preparing to unpack .../024-ceph-base_18.2.2-1059-g4d05f4ca-1jammy_amd64.deb ... 2024-04-06T09:27:29.436 INFO:teuthology.orchestra.run.smithi192.stdout:Unpacking ceph-base (18.2.2-1059-g4d05f4ca-1jammy) ... 2024-04-06T09:27:29.504 INFO:teuthology.orchestra.run.smithi006.stdout:Selecting previously unselected package python3-wcwidth. 2024-04-06T09:27:29.531 INFO:teuthology.orchestra.run.smithi006.stdout:Preparing to unpack .../015-python3-wcwidth_0.2.5+dfsg1-1_all.deb ... 2024-04-06T09:27:29.570 INFO:teuthology.orchestra.run.smithi006.stdout:Unpacking python3-wcwidth (0.2.5+dfsg1-1) ... 2024-04-06T09:27:29.865 INFO:teuthology.orchestra.run.smithi006.stdout:Selecting previously unselected package python3-prettytable. 2024-04-06T09:27:29.892 INFO:teuthology.orchestra.run.smithi006.stdout:Preparing to unpack .../016-python3-prettytable_2.5.0-2_all.deb ... 2024-04-06T09:27:29.931 INFO:teuthology.orchestra.run.smithi006.stdout:Unpacking python3-prettytable (2.5.0-2) ... 2024-04-06T09:27:30.023 INFO:teuthology.orchestra.run.smithi192.stdout:Selecting previously unselected package python3-jaraco.functools. 2024-04-06T09:27:30.050 INFO:teuthology.orchestra.run.smithi192.stdout:Preparing to unpack .../025-python3-jaraco.functools_3.4.0-2_all.deb ... 2024-04-06T09:27:30.089 INFO:teuthology.orchestra.run.smithi192.stdout:Unpacking python3-jaraco.functools (3.4.0-2) ... 2024-04-06T09:27:30.192 INFO:teuthology.orchestra.run.smithi006.stdout:Selecting previously unselected package python3-rbd. 2024-04-06T09:27:30.219 INFO:teuthology.orchestra.run.smithi006.stdout:Preparing to unpack .../017-python3-rbd_18.2.2-1059-g4d05f4ca-1jammy_amd64.deb ... 2024-04-06T09:27:30.259 INFO:teuthology.orchestra.run.smithi006.stdout:Unpacking python3-rbd (18.2.2-1059-g4d05f4ca-1jammy) ... 2024-04-06T09:27:30.392 INFO:teuthology.orchestra.run.smithi192.stdout:Selecting previously unselected package python3-cheroot. 2024-04-06T09:27:30.419 INFO:teuthology.orchestra.run.smithi192.stdout:Preparing to unpack .../026-python3-cheroot_8.5.2+ds1-1ubuntu3.1_all.deb ... 2024-04-06T09:27:30.459 INFO:teuthology.orchestra.run.smithi192.stdout:Unpacking python3-cheroot (8.5.2+ds1-1ubuntu3.1) ... 2024-04-06T09:27:30.587 INFO:teuthology.orchestra.run.smithi006.stdout:Selecting previously unselected package librdkafka1:amd64. 2024-04-06T09:27:30.614 INFO:teuthology.orchestra.run.smithi006.stdout:Preparing to unpack .../018-librdkafka1_1.8.0-1build1_amd64.deb ... 2024-04-06T09:27:30.653 INFO:teuthology.orchestra.run.smithi006.stdout:Unpacking librdkafka1:amd64 (1.8.0-1build1) ... 2024-04-06T09:27:30.729 INFO:teuthology.orchestra.run.smithi192.stdout:Selecting previously unselected package python3-jaraco.classes. 2024-04-06T09:27:30.755 INFO:teuthology.orchestra.run.smithi192.stdout:Preparing to unpack .../027-python3-jaraco.classes_3.2.1-3_all.deb ... 2024-04-06T09:27:30.787 INFO:teuthology.orchestra.run.smithi192.stdout:Unpacking python3-jaraco.classes (3.2.1-3) ... 2024-04-06T09:27:30.973 INFO:teuthology.orchestra.run.smithi006.stdout:Selecting previously unselected package librgw2. 2024-04-06T09:27:31.000 INFO:teuthology.orchestra.run.smithi006.stdout:Preparing to unpack .../019-librgw2_18.2.2-1059-g4d05f4ca-1jammy_amd64.deb ... 2024-04-06T09:27:31.039 INFO:teuthology.orchestra.run.smithi006.stdout:Unpacking librgw2 (18.2.2-1059-g4d05f4ca-1jammy) ... 2024-04-06T09:27:31.065 INFO:teuthology.orchestra.run.smithi192.stdout:Selecting previously unselected package python3-jaraco.text. 2024-04-06T09:27:31.091 INFO:teuthology.orchestra.run.smithi192.stdout:Preparing to unpack .../028-python3-jaraco.text_3.6.0-2_all.deb ... 2024-04-06T09:27:31.123 INFO:teuthology.orchestra.run.smithi192.stdout:Unpacking python3-jaraco.text (3.6.0-2) ... 2024-04-06T09:27:31.434 INFO:teuthology.orchestra.run.smithi192.stdout:Selecting previously unselected package python3-jaraco.collections. 2024-04-06T09:27:31.461 INFO:teuthology.orchestra.run.smithi192.stdout:Preparing to unpack .../029-python3-jaraco.collections_3.4.0-2_all.deb ... 2024-04-06T09:27:31.501 INFO:teuthology.orchestra.run.smithi192.stdout:Unpacking python3-jaraco.collections (3.4.0-2) ... 2024-04-06T09:27:31.592 INFO:teuthology.orchestra.run.smithi006.stdout:Selecting previously unselected package python3-rgw. 2024-04-06T09:27:31.609 INFO:teuthology.orchestra.run.smithi006.stdout:Preparing to unpack .../020-python3-rgw_18.2.2-1059-g4d05f4ca-1jammy_amd64.deb ... 2024-04-06T09:27:31.650 INFO:teuthology.orchestra.run.smithi006.stdout:Unpacking python3-rgw (18.2.2-1059-g4d05f4ca-1jammy) ... 2024-04-06T09:27:31.804 INFO:teuthology.orchestra.run.smithi192.stdout:Selecting previously unselected package python3-tempora. 2024-04-06T09:27:31.830 INFO:teuthology.orchestra.run.smithi192.stdout:Preparing to unpack .../030-python3-tempora_4.1.2-1_all.deb ... 2024-04-06T09:27:31.870 INFO:teuthology.orchestra.run.smithi192.stdout:Unpacking python3-tempora (4.1.2-1) ... 2024-04-06T09:27:31.945 INFO:teuthology.orchestra.run.smithi006.stdout:Selecting previously unselected package liboath0:amd64. 2024-04-06T09:27:31.972 INFO:teuthology.orchestra.run.smithi006.stdout:Preparing to unpack .../021-liboath0_2.6.7-3build1_amd64.deb ... 2024-04-06T09:27:32.003 INFO:teuthology.orchestra.run.smithi006.stdout:Unpacking liboath0:amd64 (2.6.7-3build1) ... 2024-04-06T09:27:32.173 INFO:teuthology.orchestra.run.smithi192.stdout:Selecting previously unselected package python3-portend. 2024-04-06T09:27:32.200 INFO:teuthology.orchestra.run.smithi192.stdout:Preparing to unpack .../031-python3-portend_3.0.0-1_all.deb ... 2024-04-06T09:27:32.240 INFO:teuthology.orchestra.run.smithi192.stdout:Unpacking python3-portend (3.0.0-1) ... 2024-04-06T09:27:32.289 INFO:teuthology.orchestra.run.smithi006.stdout:Selecting previously unselected package libradosstriper1. 2024-04-06T09:27:32.316 INFO:teuthology.orchestra.run.smithi006.stdout:Preparing to unpack .../022-libradosstriper1_18.2.2-1059-g4d05f4ca-1jammy_amd64.deb ... 2024-04-06T09:27:32.356 INFO:teuthology.orchestra.run.smithi006.stdout:Unpacking libradosstriper1 (18.2.2-1059-g4d05f4ca-1jammy) ... 2024-04-06T09:27:32.542 INFO:teuthology.orchestra.run.smithi192.stdout:Selecting previously unselected package python3-zc.lockfile. 2024-04-06T09:27:32.569 INFO:teuthology.orchestra.run.smithi192.stdout:Preparing to unpack .../032-python3-zc.lockfile_2.0-1_all.deb ... 2024-04-06T09:27:32.609 INFO:teuthology.orchestra.run.smithi192.stdout:Unpacking python3-zc.lockfile (2.0-1) ... 2024-04-06T09:27:32.784 INFO:teuthology.orchestra.run.smithi006.stdout:Selecting previously unselected package ceph-common. 2024-04-06T09:27:32.811 INFO:teuthology.orchestra.run.smithi006.stdout:Preparing to unpack .../023-ceph-common_18.2.2-1059-g4d05f4ca-1jammy_amd64.deb ... 2024-04-06T09:27:32.850 INFO:teuthology.orchestra.run.smithi006.stdout:Unpacking ceph-common (18.2.2-1059-g4d05f4ca-1jammy) ... 2024-04-06T09:27:32.912 INFO:teuthology.orchestra.run.smithi192.stdout:Selecting previously unselected package python3-cherrypy3. 2024-04-06T09:27:32.939 INFO:teuthology.orchestra.run.smithi192.stdout:Preparing to unpack .../033-python3-cherrypy3_18.6.1-4_all.deb ... 2024-04-06T09:27:32.970 INFO:teuthology.orchestra.run.smithi192.stdout:Unpacking python3-cherrypy3 (18.6.1-4) ... 2024-04-06T09:27:33.291 INFO:teuthology.orchestra.run.smithi192.stdout:Selecting previously unselected package python3-natsort. 2024-04-06T09:27:33.317 INFO:teuthology.orchestra.run.smithi192.stdout:Preparing to unpack .../034-python3-natsort_8.0.2-1_all.deb ... 2024-04-06T09:27:33.357 INFO:teuthology.orchestra.run.smithi192.stdout:Unpacking python3-natsort (8.0.2-1) ... 2024-04-06T09:27:33.668 INFO:teuthology.orchestra.run.smithi192.stdout:Selecting previously unselected package python3-logutils. 2024-04-06T09:27:33.695 INFO:teuthology.orchestra.run.smithi192.stdout:Preparing to unpack .../035-python3-logutils_0.3.3-8_all.deb ... 2024-04-06T09:27:33.735 INFO:teuthology.orchestra.run.smithi192.stdout:Unpacking python3-logutils (0.3.3-8) ... 2024-04-06T09:27:33.996 INFO:teuthology.orchestra.run.smithi192.stdout:Selecting previously unselected package python3-markupsafe. 2024-04-06T09:27:34.023 INFO:teuthology.orchestra.run.smithi192.stdout:Preparing to unpack .../036-python3-markupsafe_2.0.1-2build1_amd64.deb ... 2024-04-06T09:27:34.054 INFO:teuthology.orchestra.run.smithi192.stdout:Unpacking python3-markupsafe (2.0.1-2build1) ... 2024-04-06T09:27:34.173 INFO:teuthology.orchestra.run.smithi006.stdout:Selecting previously unselected package ceph-base. 2024-04-06T09:27:34.200 INFO:teuthology.orchestra.run.smithi006.stdout:Preparing to unpack .../024-ceph-base_18.2.2-1059-g4d05f4ca-1jammy_amd64.deb ... 2024-04-06T09:27:34.249 INFO:teuthology.orchestra.run.smithi006.stdout:Unpacking ceph-base (18.2.2-1059-g4d05f4ca-1jammy) ... 2024-04-06T09:27:34.341 INFO:teuthology.orchestra.run.smithi192.stdout:Selecting previously unselected package python3-mako. 2024-04-06T09:27:34.368 INFO:teuthology.orchestra.run.smithi192.stdout:Preparing to unpack .../037-python3-mako_1.1.3+ds1-2ubuntu0.1_all.deb ... 2024-04-06T09:27:34.407 INFO:teuthology.orchestra.run.smithi192.stdout:Unpacking python3-mako (1.1.3+ds1-2ubuntu0.1) ... 2024-04-06T09:27:34.760 INFO:teuthology.orchestra.run.smithi192.stdout:Selecting previously unselected package python3-simplegeneric. 2024-04-06T09:27:34.787 INFO:teuthology.orchestra.run.smithi192.stdout:Preparing to unpack .../038-python3-simplegeneric_0.8.1-3_all.deb ... 2024-04-06T09:27:34.827 INFO:teuthology.orchestra.run.smithi192.stdout:Unpacking python3-simplegeneric (0.8.1-3) ... 2024-04-06T09:27:34.835 INFO:teuthology.orchestra.run.smithi006.stdout:Selecting previously unselected package python3-jaraco.functools. 2024-04-06T09:27:34.862 INFO:teuthology.orchestra.run.smithi006.stdout:Preparing to unpack .../025-python3-jaraco.functools_3.4.0-2_all.deb ... 2024-04-06T09:27:34.902 INFO:teuthology.orchestra.run.smithi006.stdout:Unpacking python3-jaraco.functools (3.4.0-2) ... 2024-04-06T09:27:35.138 INFO:teuthology.orchestra.run.smithi192.stdout:Selecting previously unselected package python3-singledispatch. 2024-04-06T09:27:35.165 INFO:teuthology.orchestra.run.smithi192.stdout:Preparing to unpack .../039-python3-singledispatch_3.4.0.3-3_all.deb ... 2024-04-06T09:27:35.180 INFO:teuthology.orchestra.run.smithi006.stdout:Selecting previously unselected package python3-cheroot. 2024-04-06T09:27:35.204 INFO:teuthology.orchestra.run.smithi192.stdout:Unpacking python3-singledispatch (3.4.0.3-3) ... 2024-04-06T09:27:35.206 INFO:teuthology.orchestra.run.smithi006.stdout:Preparing to unpack .../026-python3-cheroot_8.5.2+ds1-1ubuntu3.1_all.deb ... 2024-04-06T09:27:35.246 INFO:teuthology.orchestra.run.smithi006.stdout:Unpacking python3-cheroot (8.5.2+ds1-1ubuntu3.1) ... 2024-04-06T09:27:35.441 INFO:teuthology.orchestra.run.smithi192.stdout:Selecting previously unselected package python3-webob. 2024-04-06T09:27:35.456 INFO:teuthology.orchestra.run.smithi192.stdout:Preparing to unpack .../040-python3-webob_1%3a1.8.6-1.1_all.deb ... 2024-04-06T09:27:35.491 INFO:teuthology.orchestra.run.smithi192.stdout:Unpacking python3-webob (1:1.8.6-1.1) ... 2024-04-06T09:27:35.507 INFO:teuthology.orchestra.run.smithi006.stdout:Selecting previously unselected package python3-jaraco.classes. 2024-04-06T09:27:35.523 INFO:teuthology.orchestra.run.smithi006.stdout:Preparing to unpack .../027-python3-jaraco.classes_3.2.1-3_all.deb ... 2024-04-06T09:27:35.557 INFO:teuthology.orchestra.run.smithi006.stdout:Unpacking python3-jaraco.classes (3.2.1-3) ... 2024-04-06T09:27:35.785 INFO:teuthology.orchestra.run.smithi192.stdout:Selecting previously unselected package python3-waitress. 2024-04-06T09:27:35.801 INFO:teuthology.orchestra.run.smithi192.stdout:Preparing to unpack .../041-python3-waitress_1.4.4-1.1ubuntu1_all.deb ... 2024-04-06T09:27:35.845 INFO:teuthology.orchestra.run.smithi192.stdout:Unpacking python3-waitress (1.4.4-1.1ubuntu1) ... 2024-04-06T09:27:35.852 INFO:teuthology.orchestra.run.smithi006.stdout:Selecting previously unselected package python3-jaraco.text. 2024-04-06T09:27:35.879 INFO:teuthology.orchestra.run.smithi006.stdout:Preparing to unpack .../028-python3-jaraco.text_3.6.0-2_all.deb ... 2024-04-06T09:27:35.918 INFO:teuthology.orchestra.run.smithi006.stdout:Unpacking python3-jaraco.text (3.6.0-2) ... 2024-04-06T09:27:36.130 INFO:teuthology.orchestra.run.smithi192.stdout:Selecting previously unselected package python3-tempita. 2024-04-06T09:27:36.157 INFO:teuthology.orchestra.run.smithi192.stdout:Preparing to unpack .../042-python3-tempita_0.5.2-6ubuntu1_all.deb ... 2024-04-06T09:27:36.196 INFO:teuthology.orchestra.run.smithi192.stdout:Unpacking python3-tempita (0.5.2-6ubuntu1) ... 2024-04-06T09:27:36.206 INFO:teuthology.orchestra.run.smithi006.stdout:Selecting previously unselected package python3-jaraco.collections. 2024-04-06T09:27:36.231 INFO:teuthology.orchestra.run.smithi006.stdout:Preparing to unpack .../029-python3-jaraco.collections_3.4.0-2_all.deb ... 2024-04-06T09:27:36.271 INFO:teuthology.orchestra.run.smithi006.stdout:Unpacking python3-jaraco.collections (3.4.0-2) ... 2024-04-06T09:27:36.524 INFO:teuthology.orchestra.run.smithi192.stdout:Selecting previously unselected package python3-paste. 2024-04-06T09:27:36.549 INFO:teuthology.orchestra.run.smithi006.stdout:Selecting previously unselected package python3-tempora. 2024-04-06T09:27:36.551 INFO:teuthology.orchestra.run.smithi192.stdout:Preparing to unpack .../043-python3-paste_3.5.0+dfsg1-1_all.deb ... 2024-04-06T09:27:36.575 INFO:teuthology.orchestra.run.smithi006.stdout:Preparing to unpack .../030-python3-tempora_4.1.2-1_all.deb ... 2024-04-06T09:27:36.590 INFO:teuthology.orchestra.run.smithi192.stdout:Unpacking python3-paste (3.5.0+dfsg1-1) ... 2024-04-06T09:27:36.615 INFO:teuthology.orchestra.run.smithi006.stdout:Unpacking python3-tempora (4.1.2-1) ... 2024-04-06T09:27:36.902 INFO:teuthology.orchestra.run.smithi192.stdout:Selecting previously unselected package python-pastedeploy-tpl. 2024-04-06T09:27:36.910 INFO:teuthology.orchestra.run.smithi006.stdout:Selecting previously unselected package python3-portend. 2024-04-06T09:27:36.918 INFO:teuthology.orchestra.run.smithi192.stdout:Preparing to unpack .../044-python-pastedeploy-tpl_2.1.1-1_all.deb ... 2024-04-06T09:27:36.937 INFO:teuthology.orchestra.run.smithi006.stdout:Preparing to unpack .../031-python3-portend_3.0.0-1_all.deb ... 2024-04-06T09:27:36.960 INFO:teuthology.orchestra.run.smithi192.stdout:Unpacking python-pastedeploy-tpl (2.1.1-1) ... 2024-04-06T09:27:36.976 INFO:teuthology.orchestra.run.smithi006.stdout:Unpacking python3-portend (3.0.0-1) ... 2024-04-06T09:27:37.280 INFO:teuthology.orchestra.run.smithi006.stdout:Selecting previously unselected package python3-zc.lockfile. 2024-04-06T09:27:37.280 INFO:teuthology.orchestra.run.smithi192.stdout:Selecting previously unselected package python3-pastedeploy. 2024-04-06T09:27:37.306 INFO:teuthology.orchestra.run.smithi006.stdout:Preparing to unpack .../032-python3-zc.lockfile_2.0-1_all.deb ... 2024-04-06T09:27:37.307 INFO:teuthology.orchestra.run.smithi192.stdout:Preparing to unpack .../045-python3-pastedeploy_2.1.1-1_all.deb ... 2024-04-06T09:27:37.346 INFO:teuthology.orchestra.run.smithi006.stdout:Unpacking python3-zc.lockfile (2.0-1) ... 2024-04-06T09:27:37.346 INFO:teuthology.orchestra.run.smithi192.stdout:Unpacking python3-pastedeploy (2.1.1-1) ... 2024-04-06T09:27:37.649 INFO:teuthology.orchestra.run.smithi006.stdout:Selecting previously unselected package python3-cherrypy3. 2024-04-06T09:27:37.649 INFO:teuthology.orchestra.run.smithi192.stdout:Selecting previously unselected package python3-webtest. 2024-04-06T09:27:37.676 INFO:teuthology.orchestra.run.smithi006.stdout:Preparing to unpack .../033-python3-cherrypy3_18.6.1-4_all.deb ... 2024-04-06T09:27:37.676 INFO:teuthology.orchestra.run.smithi192.stdout:Preparing to unpack .../046-python3-webtest_2.0.35-1_all.deb ... 2024-04-06T09:27:37.707 INFO:teuthology.orchestra.run.smithi006.stdout:Unpacking python3-cherrypy3 (18.6.1-4) ... 2024-04-06T09:27:37.716 INFO:teuthology.orchestra.run.smithi192.stdout:Unpacking python3-webtest (2.0.35-1) ... 2024-04-06T09:27:37.986 INFO:teuthology.orchestra.run.smithi192.stdout:Selecting previously unselected package python3-pecan. 2024-04-06T09:27:38.012 INFO:teuthology.orchestra.run.smithi192.stdout:Preparing to unpack .../047-python3-pecan_1.3.3-4ubuntu2_all.deb ... 2024-04-06T09:27:38.046 INFO:teuthology.orchestra.run.smithi192.stdout:Unpacking python3-pecan (1.3.3-4ubuntu2) ... 2024-04-06T09:27:38.069 INFO:teuthology.orchestra.run.smithi006.stdout:Selecting previously unselected package python3-natsort. 2024-04-06T09:27:38.096 INFO:teuthology.orchestra.run.smithi006.stdout:Preparing to unpack .../034-python3-natsort_8.0.2-1_all.deb ... 2024-04-06T09:27:38.135 INFO:teuthology.orchestra.run.smithi006.stdout:Unpacking python3-natsort (8.0.2-1) ... 2024-04-06T09:27:38.365 INFO:teuthology.orchestra.run.smithi192.stdout:Selecting previously unselected package python3-certifi. 2024-04-06T09:27:38.393 INFO:teuthology.orchestra.run.smithi192.stdout:Preparing to unpack .../048-python3-certifi_2020.6.20-1_all.deb ... 2024-04-06T09:27:38.424 INFO:teuthology.orchestra.run.smithi192.stdout:Unpacking python3-certifi (2020.6.20-1) ... 2024-04-06T09:27:38.447 INFO:teuthology.orchestra.run.smithi006.stdout:Selecting previously unselected package python3-logutils. 2024-04-06T09:27:38.474 INFO:teuthology.orchestra.run.smithi006.stdout:Preparing to unpack .../035-python3-logutils_0.3.3-8_all.deb ... 2024-04-06T09:27:38.513 INFO:teuthology.orchestra.run.smithi006.stdout:Unpacking python3-logutils (0.3.3-8) ... 2024-04-06T09:27:38.726 INFO:teuthology.orchestra.run.smithi192.stdout:Selecting previously unselected package python3-idna. 2024-04-06T09:27:38.753 INFO:teuthology.orchestra.run.smithi192.stdout:Preparing to unpack .../049-python3-idna_3.3-1_all.deb ... 2024-04-06T09:27:38.793 INFO:teuthology.orchestra.run.smithi192.stdout:Unpacking python3-idna (3.3-1) ... 2024-04-06T09:27:38.983 INFO:teuthology.orchestra.run.smithi006.stdout:Selecting previously unselected package python3-markupsafe. 2024-04-06T09:27:39.010 INFO:teuthology.orchestra.run.smithi006.stdout:Preparing to unpack .../036-python3-markupsafe_2.0.1-2build1_amd64.deb ... 2024-04-06T09:27:39.049 INFO:teuthology.orchestra.run.smithi006.stdout:Unpacking python3-markupsafe (2.0.1-2build1) ... 2024-04-06T09:27:39.113 INFO:teuthology.orchestra.run.smithi192.stdout:Selecting previously unselected package python3-urllib3. 2024-04-06T09:27:39.139 INFO:teuthology.orchestra.run.smithi192.stdout:Preparing to unpack .../050-python3-urllib3_1.26.5-1~exp1ubuntu0.1_all.deb ... 2024-04-06T09:27:39.179 INFO:teuthology.orchestra.run.smithi192.stdout:Unpacking python3-urllib3 (1.26.5-1~exp1ubuntu0.1) ... 2024-04-06T09:27:39.302 INFO:teuthology.orchestra.run.smithi006.stdout:Selecting previously unselected package python3-mako. 2024-04-06T09:27:39.318 INFO:teuthology.orchestra.run.smithi006.stdout:Preparing to unpack .../037-python3-mako_1.1.3+ds1-2ubuntu0.1_all.deb ... 2024-04-06T09:27:39.352 INFO:teuthology.orchestra.run.smithi006.stdout:Unpacking python3-mako (1.1.3+ds1-2ubuntu0.1) ... 2024-04-06T09:27:39.482 INFO:teuthology.orchestra.run.smithi192.stdout:Selecting previously unselected package python3-requests. 2024-04-06T09:27:39.509 INFO:teuthology.orchestra.run.smithi192.stdout:Preparing to unpack .../051-python3-requests_2.25.1+dfsg-2ubuntu0.1_all.deb ... 2024-04-06T09:27:39.548 INFO:teuthology.orchestra.run.smithi192.stdout:Unpacking python3-requests (2.25.1+dfsg-2ubuntu0.1) ... 2024-04-06T09:27:39.672 INFO:teuthology.orchestra.run.smithi006.stdout:Selecting previously unselected package python3-simplegeneric. 2024-04-06T09:27:39.700 INFO:teuthology.orchestra.run.smithi006.stdout:Preparing to unpack .../038-python3-simplegeneric_0.8.1-3_all.deb ... 2024-04-06T09:27:39.739 INFO:teuthology.orchestra.run.smithi006.stdout:Unpacking python3-simplegeneric (0.8.1-3) ... 2024-04-06T09:27:39.860 INFO:teuthology.orchestra.run.smithi192.stdout:Selecting previously unselected package python3-werkzeug. 2024-04-06T09:27:39.887 INFO:teuthology.orchestra.run.smithi192.stdout:Preparing to unpack .../052-python3-werkzeug_2.0.2+dfsg1-1ubuntu0.22.04.1_all.deb ... 2024-04-06T09:27:39.926 INFO:teuthology.orchestra.run.smithi192.stdout:Unpacking python3-werkzeug (2.0.2+dfsg1-1ubuntu0.22.04.1) ... 2024-04-06T09:27:40.025 INFO:teuthology.orchestra.run.smithi006.stdout:Selecting previously unselected package python3-singledispatch. 2024-04-06T09:27:40.052 INFO:teuthology.orchestra.run.smithi006.stdout:Preparing to unpack .../039-python3-singledispatch_3.4.0.3-3_all.deb ... 2024-04-06T09:27:40.091 INFO:teuthology.orchestra.run.smithi006.stdout:Unpacking python3-singledispatch (3.4.0.3-3) ... 2024-04-06T09:27:40.221 INFO:teuthology.orchestra.run.smithi192.stdout:Selecting previously unselected package ceph-mgr-modules-core. 2024-04-06T09:27:40.237 INFO:teuthology.orchestra.run.smithi192.stdout:Preparing to unpack .../053-ceph-mgr-modules-core_18.2.2-1059-g4d05f4ca-1jammy_all.deb ... 2024-04-06T09:27:40.279 INFO:teuthology.orchestra.run.smithi192.stdout:Unpacking ceph-mgr-modules-core (18.2.2-1059-g4d05f4ca-1jammy) ... 2024-04-06T09:27:40.394 INFO:teuthology.orchestra.run.smithi006.stdout:Selecting previously unselected package python3-webob. 2024-04-06T09:27:40.421 INFO:teuthology.orchestra.run.smithi006.stdout:Preparing to unpack .../040-python3-webob_1%3a1.8.6-1.1_all.deb ... 2024-04-06T09:27:40.461 INFO:teuthology.orchestra.run.smithi006.stdout:Unpacking python3-webob (1:1.8.6-1.1) ... 2024-04-06T09:27:40.642 INFO:teuthology.orchestra.run.smithi192.stdout:Selecting previously unselected package libsqlite3-mod-ceph. 2024-04-06T09:27:40.669 INFO:teuthology.orchestra.run.smithi192.stdout:Preparing to unpack .../054-libsqlite3-mod-ceph_18.2.2-1059-g4d05f4ca-1jammy_amd64.deb ... 2024-04-06T09:27:40.709 INFO:teuthology.orchestra.run.smithi192.stdout:Unpacking libsqlite3-mod-ceph (18.2.2-1059-g4d05f4ca-1jammy) ... 2024-04-06T09:27:40.797 INFO:teuthology.orchestra.run.smithi006.stdout:Selecting previously unselected package python3-waitress. 2024-04-06T09:27:40.824 INFO:teuthology.orchestra.run.smithi006.stdout:Preparing to unpack .../041-python3-waitress_1.4.4-1.1ubuntu1_all.deb ... 2024-04-06T09:27:40.867 INFO:teuthology.orchestra.run.smithi006.stdout:Unpacking python3-waitress (1.4.4-1.1ubuntu1) ... 2024-04-06T09:27:41.020 INFO:teuthology.orchestra.run.smithi192.stdout:Selecting previously unselected package python3-openssl. 2024-04-06T09:27:41.047 INFO:teuthology.orchestra.run.smithi192.stdout:Preparing to unpack .../055-python3-openssl_21.0.0-1_all.deb ... 2024-04-06T09:27:41.086 INFO:teuthology.orchestra.run.smithi192.stdout:Unpacking python3-openssl (21.0.0-1) ... 2024-04-06T09:27:41.191 INFO:teuthology.orchestra.run.smithi006.stdout:Selecting previously unselected package python3-tempita. 2024-04-06T09:27:41.218 INFO:teuthology.orchestra.run.smithi006.stdout:Preparing to unpack .../042-python3-tempita_0.5.2-6ubuntu1_all.deb ... 2024-04-06T09:27:41.258 INFO:teuthology.orchestra.run.smithi006.stdout:Unpacking python3-tempita (0.5.2-6ubuntu1) ... 2024-04-06T09:27:41.423 INFO:teuthology.orchestra.run.smithi192.stdout:Selecting previously unselected package ceph-mgr. 2024-04-06T09:27:41.450 INFO:teuthology.orchestra.run.smithi192.stdout:Preparing to unpack .../056-ceph-mgr_18.2.2-1059-g4d05f4ca-1jammy_amd64.deb ... 2024-04-06T09:27:41.489 INFO:teuthology.orchestra.run.smithi192.stdout:Unpacking ceph-mgr (18.2.2-1059-g4d05f4ca-1jammy) ... 2024-04-06T09:27:41.519 INFO:teuthology.orchestra.run.smithi006.stdout:Selecting previously unselected package python3-paste. 2024-04-06T09:27:41.546 INFO:teuthology.orchestra.run.smithi006.stdout:Preparing to unpack .../043-python3-paste_3.5.0+dfsg1-1_all.deb ... 2024-04-06T09:27:41.585 INFO:teuthology.orchestra.run.smithi006.stdout:Unpacking python3-paste (3.5.0+dfsg1-1) ... 2024-04-06T09:27:41.834 INFO:teuthology.orchestra.run.smithi192.stdout:Selecting previously unselected package ceph-mon. 2024-04-06T09:27:41.861 INFO:teuthology.orchestra.run.smithi192.stdout:Preparing to unpack .../057-ceph-mon_18.2.2-1059-g4d05f4ca-1jammy_amd64.deb ... 2024-04-06T09:27:41.892 INFO:teuthology.orchestra.run.smithi192.stdout:Unpacking ceph-mon (18.2.2-1059-g4d05f4ca-1jammy) ... 2024-04-06T09:27:41.914 INFO:teuthology.orchestra.run.smithi006.stdout:Selecting previously unselected package python-pastedeploy-tpl. 2024-04-06T09:27:41.941 INFO:teuthology.orchestra.run.smithi006.stdout:Preparing to unpack .../044-python-pastedeploy-tpl_2.1.1-1_all.deb ... 2024-04-06T09:27:41.980 INFO:teuthology.orchestra.run.smithi006.stdout:Unpacking python-pastedeploy-tpl (2.1.1-1) ... 2024-04-06T09:27:42.266 INFO:teuthology.orchestra.run.smithi006.stdout:Selecting previously unselected package python3-pastedeploy. 2024-04-06T09:27:42.293 INFO:teuthology.orchestra.run.smithi006.stdout:Preparing to unpack .../045-python3-pastedeploy_2.1.1-1_all.deb ... 2024-04-06T09:27:42.333 INFO:teuthology.orchestra.run.smithi006.stdout:Unpacking python3-pastedeploy (2.1.1-1) ... 2024-04-06T09:27:42.496 INFO:teuthology.orchestra.run.smithi192.stdout:Selecting previously unselected package ceph-osd. 2024-04-06T09:27:42.523 INFO:teuthology.orchestra.run.smithi192.stdout:Preparing to unpack .../058-ceph-osd_18.2.2-1059-g4d05f4ca-1jammy_amd64.deb ... 2024-04-06T09:27:42.562 INFO:teuthology.orchestra.run.smithi192.stdout:Unpacking ceph-osd (18.2.2-1059-g4d05f4ca-1jammy) ... 2024-04-06T09:27:42.586 INFO:teuthology.orchestra.run.smithi006.stdout:Selecting previously unselected package python3-webtest. 2024-04-06T09:27:42.613 INFO:teuthology.orchestra.run.smithi006.stdout:Preparing to unpack .../046-python3-webtest_2.0.35-1_all.deb ... 2024-04-06T09:27:42.644 INFO:teuthology.orchestra.run.smithi006.stdout:Unpacking python3-webtest (2.0.35-1) ... 2024-04-06T09:27:42.947 INFO:teuthology.orchestra.run.smithi006.stdout:Selecting previously unselected package python3-pecan. 2024-04-06T09:27:42.974 INFO:teuthology.orchestra.run.smithi006.stdout:Preparing to unpack .../047-python3-pecan_1.3.3-4ubuntu2_all.deb ... 2024-04-06T09:27:43.013 INFO:teuthology.orchestra.run.smithi006.stdout:Unpacking python3-pecan (1.3.3-4ubuntu2) ... 2024-04-06T09:27:43.377 INFO:teuthology.orchestra.run.smithi006.stdout:Selecting previously unselected package python3-certifi. 2024-04-06T09:27:43.404 INFO:teuthology.orchestra.run.smithi006.stdout:Preparing to unpack .../048-python3-certifi_2020.6.20-1_all.deb ... 2024-04-06T09:27:43.443 INFO:teuthology.orchestra.run.smithi006.stdout:Unpacking python3-certifi (2020.6.20-1) ... 2024-04-06T09:27:43.763 INFO:teuthology.orchestra.run.smithi006.stdout:Selecting previously unselected package python3-idna. 2024-04-06T09:27:43.790 INFO:teuthology.orchestra.run.smithi006.stdout:Preparing to unpack .../049-python3-idna_3.3-1_all.deb ... 2024-04-06T09:27:43.829 INFO:teuthology.orchestra.run.smithi006.stdout:Unpacking python3-idna (3.3-1) ... 2024-04-06T09:27:43.916 INFO:teuthology.orchestra.run.smithi192.stdout:Selecting previously unselected package ceph. 2024-04-06T09:27:43.943 INFO:teuthology.orchestra.run.smithi192.stdout:Preparing to unpack .../059-ceph_18.2.2-1059-g4d05f4ca-1jammy_amd64.deb ... 2024-04-06T09:27:44.115 INFO:teuthology.orchestra.run.smithi006.stdout:Selecting previously unselected package python3-urllib3. 2024-04-06T09:27:44.142 INFO:teuthology.orchestra.run.smithi006.stdout:Preparing to unpack .../050-python3-urllib3_1.26.5-1~exp1ubuntu0.1_all.deb ... 2024-04-06T09:27:44.173 INFO:teuthology.orchestra.run.smithi006.stdout:Unpacking python3-urllib3 (1.26.5-1~exp1ubuntu0.1) ... 2024-04-06T09:27:44.341 INFO:teuthology.orchestra.run.smithi192.stdout:Unpacking ceph (18.2.2-1059-g4d05f4ca-1jammy) ... 2024-04-06T09:27:44.485 INFO:teuthology.orchestra.run.smithi006.stdout:Selecting previously unselected package python3-requests. 2024-04-06T09:27:44.501 INFO:teuthology.orchestra.run.smithi006.stdout:Preparing to unpack .../051-python3-requests_2.25.1+dfsg-2ubuntu0.1_all.deb ... 2024-04-06T09:27:44.535 INFO:teuthology.orchestra.run.smithi006.stdout:Unpacking python3-requests (2.25.1+dfsg-2ubuntu0.1) ... 2024-04-06T09:27:45.052 INFO:teuthology.orchestra.run.smithi192.stdout:Selecting previously unselected package ceph-fuse. 2024-04-06T09:27:45.079 INFO:teuthology.orchestra.run.smithi192.stdout:Preparing to unpack .../060-ceph-fuse_18.2.2-1059-g4d05f4ca-1jammy_amd64.deb ... 2024-04-06T09:27:45.110 INFO:teuthology.orchestra.run.smithi192.stdout:Unpacking ceph-fuse (18.2.2-1059-g4d05f4ca-1jammy) ... 2024-04-06T09:27:45.472 INFO:teuthology.orchestra.run.smithi192.stdout:Selecting previously unselected package ceph-mds. 2024-04-06T09:27:45.499 INFO:teuthology.orchestra.run.smithi192.stdout:Preparing to unpack .../061-ceph-mds_18.2.2-1059-g4d05f4ca-1jammy_amd64.deb ... 2024-04-06T09:27:45.539 INFO:teuthology.orchestra.run.smithi192.stdout:Unpacking ceph-mds (18.2.2-1059-g4d05f4ca-1jammy) ... 2024-04-06T09:27:45.614 INFO:teuthology.orchestra.run.smithi006.stdout:Selecting previously unselected package python3-werkzeug. 2024-04-06T09:27:45.641 INFO:teuthology.orchestra.run.smithi006.stdout:Preparing to unpack .../052-python3-werkzeug_2.0.2+dfsg1-1ubuntu0.22.04.1_all.deb ... 2024-04-06T09:27:45.680 INFO:teuthology.orchestra.run.smithi006.stdout:Unpacking python3-werkzeug (2.0.2+dfsg1-1ubuntu0.22.04.1) ... 2024-04-06T09:27:45.925 INFO:teuthology.orchestra.run.smithi192.stdout:Selecting previously unselected package cephadm. 2024-04-06T09:27:45.951 INFO:teuthology.orchestra.run.smithi192.stdout:Preparing to unpack .../062-cephadm_18.2.2-1059-g4d05f4ca-1jammy_amd64.deb ... 2024-04-06T09:27:45.958 INFO:teuthology.orchestra.run.smithi006.stdout:Selecting previously unselected package ceph-mgr-modules-core. 2024-04-06T09:27:45.974 INFO:teuthology.orchestra.run.smithi006.stdout:Preparing to unpack .../053-ceph-mgr-modules-core_18.2.2-1059-g4d05f4ca-1jammy_all.deb ... 2024-04-06T09:27:45.991 INFO:teuthology.orchestra.run.smithi192.stdout:Unpacking cephadm (18.2.2-1059-g4d05f4ca-1jammy) ... 2024-04-06T09:27:46.016 INFO:teuthology.orchestra.run.smithi006.stdout:Unpacking ceph-mgr-modules-core (18.2.2-1059-g4d05f4ca-1jammy) ... 2024-04-06T09:27:46.286 INFO:teuthology.orchestra.run.smithi192.stdout:Selecting previously unselected package python3-asyncssh. 2024-04-06T09:27:46.313 INFO:teuthology.orchestra.run.smithi192.stdout:Preparing to unpack .../063-python3-asyncssh_2.5.0-1_all.deb ... 2024-04-06T09:27:46.353 INFO:teuthology.orchestra.run.smithi192.stdout:Unpacking python3-asyncssh (2.5.0-1) ... 2024-04-06T09:27:46.354 INFO:teuthology.orchestra.run.smithi006.stdout:Selecting previously unselected package libsqlite3-mod-ceph. 2024-04-06T09:27:46.382 INFO:teuthology.orchestra.run.smithi006.stdout:Preparing to unpack .../054-libsqlite3-mod-ceph_18.2.2-1059-g4d05f4ca-1jammy_amd64.deb ... 2024-04-06T09:27:46.421 INFO:teuthology.orchestra.run.smithi006.stdout:Unpacking libsqlite3-mod-ceph (18.2.2-1059-g4d05f4ca-1jammy) ... 2024-04-06T09:27:46.697 INFO:teuthology.orchestra.run.smithi192.stdout:Selecting previously unselected package python-babel-localedata. 2024-04-06T09:27:46.724 INFO:teuthology.orchestra.run.smithi006.stdout:Selecting previously unselected package python3-openssl. 2024-04-06T09:27:46.724 INFO:teuthology.orchestra.run.smithi192.stdout:Preparing to unpack .../064-python-babel-localedata_2.8.0+dfsg.1-7_all.deb ... 2024-04-06T09:27:46.751 INFO:teuthology.orchestra.run.smithi006.stdout:Preparing to unpack .../055-python3-openssl_21.0.0-1_all.deb ... 2024-04-06T09:27:46.764 INFO:teuthology.orchestra.run.smithi192.stdout:Unpacking python-babel-localedata (2.8.0+dfsg.1-7) ... 2024-04-06T09:27:46.790 INFO:teuthology.orchestra.run.smithi006.stdout:Unpacking python3-openssl (21.0.0-1) ... 2024-04-06T09:27:47.102 INFO:teuthology.orchestra.run.smithi006.stdout:Selecting previously unselected package ceph-mgr. 2024-04-06T09:27:47.129 INFO:teuthology.orchestra.run.smithi006.stdout:Preparing to unpack .../056-ceph-mgr_18.2.2-1059-g4d05f4ca-1jammy_amd64.deb ... 2024-04-06T09:27:47.168 INFO:teuthology.orchestra.run.smithi006.stdout:Unpacking ceph-mgr (18.2.2-1059-g4d05f4ca-1jammy) ... 2024-04-06T09:27:47.547 INFO:teuthology.orchestra.run.smithi006.stdout:Selecting previously unselected package ceph-mon. 2024-04-06T09:27:47.574 INFO:teuthology.orchestra.run.smithi006.stdout:Preparing to unpack .../057-ceph-mon_18.2.2-1059-g4d05f4ca-1jammy_amd64.deb ... 2024-04-06T09:27:47.613 INFO:teuthology.orchestra.run.smithi006.stdout:Unpacking ceph-mon (18.2.2-1059-g4d05f4ca-1jammy) ... 2024-04-06T09:27:47.736 INFO:teuthology.orchestra.run.smithi192.stdout:Selecting previously unselected package python3-babel. 2024-04-06T09:27:47.762 INFO:teuthology.orchestra.run.smithi192.stdout:Preparing to unpack .../065-python3-babel_2.8.0+dfsg.1-7_all.deb ... 2024-04-06T09:27:47.802 INFO:teuthology.orchestra.run.smithi192.stdout:Unpacking python3-babel (2.8.0+dfsg.1-7) ... 2024-04-06T09:27:48.122 INFO:teuthology.orchestra.run.smithi192.stdout:Selecting previously unselected package python3-jinja2. 2024-04-06T09:27:48.149 INFO:teuthology.orchestra.run.smithi192.stdout:Preparing to unpack .../066-python3-jinja2_3.0.3-1ubuntu0.1_all.deb ... 2024-04-06T09:27:48.188 INFO:teuthology.orchestra.run.smithi192.stdout:Unpacking python3-jinja2 (3.0.3-1ubuntu0.1) ... 2024-04-06T09:27:48.224 INFO:teuthology.orchestra.run.smithi006.stdout:Selecting previously unselected package ceph-osd. 2024-04-06T09:27:48.252 INFO:teuthology.orchestra.run.smithi006.stdout:Preparing to unpack .../058-ceph-osd_18.2.2-1059-g4d05f4ca-1jammy_amd64.deb ... 2024-04-06T09:27:48.291 INFO:teuthology.orchestra.run.smithi006.stdout:Unpacking ceph-osd (18.2.2-1059-g4d05f4ca-1jammy) ... 2024-04-06T09:27:48.550 INFO:teuthology.orchestra.run.smithi192.stdout:Selecting previously unselected package ceph-mgr-cephadm. 2024-04-06T09:27:48.577 INFO:teuthology.orchestra.run.smithi192.stdout:Preparing to unpack .../067-ceph-mgr-cephadm_18.2.2-1059-g4d05f4ca-1jammy_all.deb ... 2024-04-06T09:27:48.617 INFO:teuthology.orchestra.run.smithi192.stdout:Unpacking ceph-mgr-cephadm (18.2.2-1059-g4d05f4ca-1jammy) ... 2024-04-06T09:27:48.953 INFO:teuthology.orchestra.run.smithi192.stdout:Selecting previously unselected package python3-bcrypt. 2024-04-06T09:27:48.980 INFO:teuthology.orchestra.run.smithi192.stdout:Preparing to unpack .../068-python3-bcrypt_3.2.0-1build1_amd64.deb ... 2024-04-06T09:27:49.020 INFO:teuthology.orchestra.run.smithi192.stdout:Unpacking python3-bcrypt (3.2.0-1build1) ... 2024-04-06T09:27:49.333 INFO:teuthology.orchestra.run.smithi192.stdout:Selecting previously unselected package python3-pluggy. 2024-04-06T09:27:49.358 INFO:teuthology.orchestra.run.smithi192.stdout:Preparing to unpack .../069-python3-pluggy_0.13.0-7.1_all.deb ... 2024-04-06T09:27:49.398 INFO:teuthology.orchestra.run.smithi192.stdout:Unpacking python3-pluggy (0.13.0-7.1) ... 2024-04-06T09:27:49.511 INFO:teuthology.orchestra.run.smithi006.stdout:Selecting previously unselected package ceph. 2024-04-06T09:27:49.535 INFO:teuthology.orchestra.run.smithi006.stdout:Preparing to unpack .../059-ceph_18.2.2-1059-g4d05f4ca-1jammy_amd64.deb ... 2024-04-06T09:27:49.578 INFO:teuthology.orchestra.run.smithi006.stdout:Unpacking ceph (18.2.2-1059-g4d05f4ca-1jammy) ... 2024-04-06T09:27:49.684 INFO:teuthology.orchestra.run.smithi192.stdout:Selecting previously unselected package python3-py. 2024-04-06T09:27:49.711 INFO:teuthology.orchestra.run.smithi192.stdout:Preparing to unpack .../070-python3-py_1.10.0-1_all.deb ... 2024-04-06T09:27:49.750 INFO:teuthology.orchestra.run.smithi192.stdout:Unpacking python3-py (1.10.0-1) ... 2024-04-06T09:27:49.839 INFO:teuthology.orchestra.run.smithi006.stdout:Selecting previously unselected package ceph-fuse. 2024-04-06T09:27:49.866 INFO:teuthology.orchestra.run.smithi006.stdout:Preparing to unpack .../060-ceph-fuse_18.2.2-1059-g4d05f4ca-1jammy_amd64.deb ... 2024-04-06T09:27:49.897 INFO:teuthology.orchestra.run.smithi006.stdout:Unpacking ceph-fuse (18.2.2-1059-g4d05f4ca-1jammy) ... 2024-04-06T09:27:50.071 INFO:teuthology.orchestra.run.smithi192.stdout:Selecting previously unselected package python3-iniconfig. 2024-04-06T09:27:50.098 INFO:teuthology.orchestra.run.smithi192.stdout:Preparing to unpack .../071-python3-iniconfig_1.1.1-2_all.deb ... 2024-04-06T09:27:50.129 INFO:teuthology.orchestra.run.smithi192.stdout:Unpacking python3-iniconfig (1.1.1-2) ... 2024-04-06T09:27:50.267 INFO:teuthology.orchestra.run.smithi006.stdout:Selecting previously unselected package ceph-mds. 2024-04-06T09:27:50.294 INFO:teuthology.orchestra.run.smithi006.stdout:Preparing to unpack .../061-ceph-mds_18.2.2-1059-g4d05f4ca-1jammy_amd64.deb ... 2024-04-06T09:27:50.333 INFO:teuthology.orchestra.run.smithi006.stdout:Unpacking ceph-mds (18.2.2-1059-g4d05f4ca-1jammy) ... 2024-04-06T09:27:50.415 INFO:teuthology.orchestra.run.smithi192.stdout:Selecting previously unselected package python3-toml. 2024-04-06T09:27:50.431 INFO:teuthology.orchestra.run.smithi192.stdout:Preparing to unpack .../072-python3-toml_0.10.2-1_all.deb ... 2024-04-06T09:27:50.473 INFO:teuthology.orchestra.run.smithi192.stdout:Unpacking python3-toml (0.10.2-1) ... 2024-04-06T09:27:50.720 INFO:teuthology.orchestra.run.smithi006.stdout:Selecting previously unselected package cephadm. 2024-04-06T09:27:50.747 INFO:teuthology.orchestra.run.smithi006.stdout:Preparing to unpack .../062-cephadm_18.2.2-1059-g4d05f4ca-1jammy_amd64.deb ... 2024-04-06T09:27:50.776 INFO:teuthology.orchestra.run.smithi192.stdout:Selecting previously unselected package python3-pytest. 2024-04-06T09:27:50.786 INFO:teuthology.orchestra.run.smithi006.stdout:Unpacking cephadm (18.2.2-1059-g4d05f4ca-1jammy) ... 2024-04-06T09:27:50.803 INFO:teuthology.orchestra.run.smithi192.stdout:Preparing to unpack .../073-python3-pytest_6.2.5-1ubuntu2_all.deb ... 2024-04-06T09:27:50.843 INFO:teuthology.orchestra.run.smithi192.stdout:Unpacking python3-pytest (6.2.5-1ubuntu2) ... 2024-04-06T09:27:51.064 INFO:teuthology.orchestra.run.smithi006.stdout:Selecting previously unselected package python3-asyncssh. 2024-04-06T09:27:51.092 INFO:teuthology.orchestra.run.smithi006.stdout:Preparing to unpack .../063-python3-asyncssh_2.5.0-1_all.deb ... 2024-04-06T09:27:51.122 INFO:teuthology.orchestra.run.smithi006.stdout:Unpacking python3-asyncssh (2.5.0-1) ... 2024-04-06T09:27:51.174 INFO:teuthology.orchestra.run.smithi192.stdout:Selecting previously unselected package python3-repoze.lru. 2024-04-06T09:27:51.200 INFO:teuthology.orchestra.run.smithi192.stdout:Preparing to unpack .../074-python3-repoze.lru_0.7-2_all.deb ... 2024-04-06T09:27:51.240 INFO:teuthology.orchestra.run.smithi192.stdout:Unpacking python3-repoze.lru (0.7-2) ... 2024-04-06T09:27:51.417 INFO:teuthology.orchestra.run.smithi006.stdout:Selecting previously unselected package python-babel-localedata. 2024-04-06T09:27:51.444 INFO:teuthology.orchestra.run.smithi006.stdout:Preparing to unpack .../064-python-babel-localedata_2.8.0+dfsg.1-7_all.deb ... 2024-04-06T09:27:51.483 INFO:teuthology.orchestra.run.smithi006.stdout:Unpacking python-babel-localedata (2.8.0+dfsg.1-7) ... 2024-04-06T09:27:51.543 INFO:teuthology.orchestra.run.smithi192.stdout:Selecting previously unselected package python3-routes. 2024-04-06T09:27:51.570 INFO:teuthology.orchestra.run.smithi192.stdout:Preparing to unpack .../075-python3-routes_2.5.1-1ubuntu1_all.deb ... 2024-04-06T09:27:51.610 INFO:teuthology.orchestra.run.smithi192.stdout:Unpacking python3-routes (2.5.1-1ubuntu1) ... 2024-04-06T09:27:51.955 INFO:teuthology.orchestra.run.smithi192.stdout:Selecting previously unselected package ceph-mgr-dashboard. 2024-04-06T09:27:51.982 INFO:teuthology.orchestra.run.smithi192.stdout:Preparing to unpack .../076-ceph-mgr-dashboard_18.2.2-1059-g4d05f4ca-1jammy_all.deb ... 2024-04-06T09:27:52.021 INFO:teuthology.orchestra.run.smithi192.stdout:Unpacking ceph-mgr-dashboard (18.2.2-1059-g4d05f4ca-1jammy) ... 2024-04-06T09:27:52.450 INFO:teuthology.orchestra.run.smithi192.stdout:Selecting previously unselected package python3-sklearn-lib:amd64. 2024-04-06T09:27:52.465 INFO:teuthology.orchestra.run.smithi192.stdout:Preparing to unpack .../077-python3-sklearn-lib_0.23.2-5ubuntu6_amd64.deb ... 2024-04-06T09:27:52.480 INFO:teuthology.orchestra.run.smithi006.stdout:Selecting previously unselected package python3-babel. 2024-04-06T09:27:52.508 INFO:teuthology.orchestra.run.smithi006.stdout:Preparing to unpack .../065-python3-babel_2.8.0+dfsg.1-7_all.deb ... 2024-04-06T09:27:52.508 INFO:teuthology.orchestra.run.smithi192.stdout:Unpacking python3-sklearn-lib:amd64 (0.23.2-5ubuntu6) ... 2024-04-06T09:27:52.546 INFO:teuthology.orchestra.run.smithi006.stdout:Unpacking python3-babel (2.8.0+dfsg.1-7) ... 2024-04-06T09:27:52.849 INFO:teuthology.orchestra.run.smithi006.stdout:Selecting previously unselected package python3-jinja2. 2024-04-06T09:27:52.866 INFO:teuthology.orchestra.run.smithi006.stdout:Preparing to unpack .../066-python3-jinja2_3.0.3-1ubuntu0.1_all.deb ... 2024-04-06T09:27:52.899 INFO:teuthology.orchestra.run.smithi006.stdout:Unpacking python3-jinja2 (3.0.3-1ubuntu0.1) ... 2024-04-06T09:27:53.021 INFO:teuthology.orchestra.run.smithi192.stdout:Selecting previously unselected package python3-joblib. 2024-04-06T09:27:53.047 INFO:teuthology.orchestra.run.smithi192.stdout:Preparing to unpack .../078-python3-joblib_0.17.0-4ubuntu1_all.deb ... 2024-04-06T09:27:53.087 INFO:teuthology.orchestra.run.smithi192.stdout:Unpacking python3-joblib (0.17.0-4ubuntu1) ... 2024-04-06T09:27:53.203 INFO:teuthology.orchestra.run.smithi006.stdout:Selecting previously unselected package ceph-mgr-cephadm. 2024-04-06T09:27:53.219 INFO:teuthology.orchestra.run.smithi006.stdout:Preparing to unpack .../067-ceph-mgr-cephadm_18.2.2-1059-g4d05f4ca-1jammy_all.deb ... 2024-04-06T09:27:53.252 INFO:teuthology.orchestra.run.smithi006.stdout:Unpacking ceph-mgr-cephadm (18.2.2-1059-g4d05f4ca-1jammy) ... 2024-04-06T09:27:53.458 INFO:teuthology.orchestra.run.smithi192.stdout:Selecting previously unselected package python3-threadpoolctl. 2024-04-06T09:27:53.485 INFO:teuthology.orchestra.run.smithi192.stdout:Preparing to unpack .../079-python3-threadpoolctl_3.1.0-1_all.deb ... 2024-04-06T09:27:53.524 INFO:teuthology.orchestra.run.smithi192.stdout:Unpacking python3-threadpoolctl (3.1.0-1) ... 2024-04-06T09:27:53.564 INFO:teuthology.orchestra.run.smithi006.stdout:Selecting previously unselected package python3-bcrypt. 2024-04-06T09:27:53.592 INFO:teuthology.orchestra.run.smithi006.stdout:Preparing to unpack .../068-python3-bcrypt_3.2.0-1build1_amd64.deb ... 2024-04-06T09:27:53.622 INFO:teuthology.orchestra.run.smithi006.stdout:Unpacking python3-bcrypt (3.2.0-1build1) ... 2024-04-06T09:27:53.802 INFO:teuthology.orchestra.run.smithi192.stdout:Selecting previously unselected package python3-sklearn. 2024-04-06T09:27:53.830 INFO:teuthology.orchestra.run.smithi192.stdout:Preparing to unpack .../080-python3-sklearn_0.23.2-5ubuntu6_all.deb ... 2024-04-06T09:27:53.860 INFO:teuthology.orchestra.run.smithi192.stdout:Unpacking python3-sklearn (0.23.2-5ubuntu6) ... 2024-04-06T09:27:53.892 INFO:teuthology.orchestra.run.smithi006.stdout:Selecting previously unselected package python3-pluggy. 2024-04-06T09:27:53.919 INFO:teuthology.orchestra.run.smithi006.stdout:Preparing to unpack .../069-python3-pluggy_0.13.0-7.1_all.deb ... 2024-04-06T09:27:53.958 INFO:teuthology.orchestra.run.smithi006.stdout:Unpacking python3-pluggy (0.13.0-7.1) ... 2024-04-06T09:27:54.278 INFO:teuthology.orchestra.run.smithi006.stdout:Selecting previously unselected package python3-py. 2024-04-06T09:27:54.305 INFO:teuthology.orchestra.run.smithi006.stdout:Preparing to unpack .../070-python3-py_1.10.0-1_all.deb ... 2024-04-06T09:27:54.344 INFO:teuthology.orchestra.run.smithi006.stdout:Unpacking python3-py (1.10.0-1) ... 2024-04-06T09:27:54.603 INFO:teuthology.orchestra.run.smithi192.stdout:Selecting previously unselected package ceph-mgr-diskprediction-local. 2024-04-06T09:27:54.630 INFO:teuthology.orchestra.run.smithi192.stdout:Preparing to unpack .../081-ceph-mgr-diskprediction-local_18.2.2-1059-g4d05f4ca-1jammy_all.deb ... 2024-04-06T09:27:54.669 INFO:teuthology.orchestra.run.smithi192.stdout:Unpacking ceph-mgr-diskprediction-local (18.2.2-1059-g4d05f4ca-1jammy) ... 2024-04-06T09:27:54.673 INFO:teuthology.orchestra.run.smithi006.stdout:Selecting previously unselected package python3-iniconfig. 2024-04-06T09:27:54.700 INFO:teuthology.orchestra.run.smithi006.stdout:Preparing to unpack .../071-python3-iniconfig_1.1.1-2_all.deb ... 2024-04-06T09:27:54.739 INFO:teuthology.orchestra.run.smithi006.stdout:Unpacking python3-iniconfig (1.1.1-2) ... 2024-04-06T09:27:55.042 INFO:teuthology.orchestra.run.smithi006.stdout:Selecting previously unselected package python3-toml. 2024-04-06T09:27:55.070 INFO:teuthology.orchestra.run.smithi006.stdout:Preparing to unpack .../072-python3-toml_0.10.2-1_all.deb ... 2024-04-06T09:27:55.109 INFO:teuthology.orchestra.run.smithi006.stdout:Unpacking python3-toml (0.10.2-1) ... 2024-04-06T09:27:55.395 INFO:teuthology.orchestra.run.smithi006.stdout:Selecting previously unselected package python3-pytest. 2024-04-06T09:27:55.423 INFO:teuthology.orchestra.run.smithi006.stdout:Preparing to unpack .../073-python3-pytest_6.2.5-1ubuntu2_all.deb ... 2024-04-06T09:27:55.453 INFO:teuthology.orchestra.run.smithi006.stdout:Unpacking python3-pytest (6.2.5-1ubuntu2) ... 2024-04-06T09:27:55.631 INFO:teuthology.orchestra.run.smithi192.stdout:Selecting previously unselected package python3-cachetools. 2024-04-06T09:27:55.658 INFO:teuthology.orchestra.run.smithi192.stdout:Preparing to unpack .../082-python3-cachetools_5.0.0-1_all.deb ... 2024-04-06T09:27:55.721 INFO:teuthology.orchestra.run.smithi192.stdout:Unpacking python3-cachetools (5.0.0-1) ... 2024-04-06T09:27:55.816 INFO:teuthology.orchestra.run.smithi006.stdout:Selecting previously unselected package python3-repoze.lru. 2024-04-06T09:27:55.843 INFO:teuthology.orchestra.run.smithi006.stdout:Preparing to unpack .../074-python3-repoze.lru_0.7-2_all.deb ... 2024-04-06T09:27:55.882 INFO:teuthology.orchestra.run.smithi006.stdout:Unpacking python3-repoze.lru (0.7-2) ... 2024-04-06T09:27:56.093 INFO:teuthology.orchestra.run.smithi192.stdout:Selecting previously unselected package python3-pyasn1. 2024-04-06T09:27:56.109 INFO:teuthology.orchestra.run.smithi192.stdout:Preparing to unpack .../083-python3-pyasn1_0.4.8-1_all.deb ... 2024-04-06T09:27:56.143 INFO:teuthology.orchestra.run.smithi192.stdout:Unpacking python3-pyasn1 (0.4.8-1) ... 2024-04-06T09:27:56.143 INFO:teuthology.orchestra.run.smithi006.stdout:Selecting previously unselected package python3-routes. 2024-04-06T09:27:56.159 INFO:teuthology.orchestra.run.smithi006.stdout:Preparing to unpack .../075-python3-routes_2.5.1-1ubuntu1_all.deb ... 2024-04-06T09:27:56.193 INFO:teuthology.orchestra.run.smithi006.stdout:Unpacking python3-routes (2.5.1-1ubuntu1) ... 2024-04-06T09:27:56.413 INFO:teuthology.orchestra.run.smithi192.stdout:Selecting previously unselected package python3-pyasn1-modules. 2024-04-06T09:27:56.441 INFO:teuthology.orchestra.run.smithi192.stdout:Preparing to unpack .../084-python3-pyasn1-modules_0.2.1-1_all.deb ... 2024-04-06T09:27:56.455 INFO:teuthology.orchestra.run.smithi006.stdout:Selecting previously unselected package ceph-mgr-dashboard. 2024-04-06T09:27:56.471 INFO:teuthology.orchestra.run.smithi192.stdout:Unpacking python3-pyasn1-modules (0.2.1-1) ... 2024-04-06T09:27:56.482 INFO:teuthology.orchestra.run.smithi006.stdout:Preparing to unpack .../076-ceph-mgr-dashboard_18.2.2-1059-g4d05f4ca-1jammy_all.deb ... 2024-04-06T09:27:56.513 INFO:teuthology.orchestra.run.smithi006.stdout:Unpacking ceph-mgr-dashboard (18.2.2-1059-g4d05f4ca-1jammy) ... 2024-04-06T09:27:56.767 INFO:teuthology.orchestra.run.smithi192.stdout:Selecting previously unselected package python3-rsa. 2024-04-06T09:27:56.794 INFO:teuthology.orchestra.run.smithi192.stdout:Preparing to unpack .../085-python3-rsa_4.8-1_all.deb ... 2024-04-06T09:27:56.833 INFO:teuthology.orchestra.run.smithi192.stdout:Unpacking python3-rsa (4.8-1) ... 2024-04-06T09:27:56.941 INFO:teuthology.orchestra.run.smithi006.stdout:Selecting previously unselected package python3-sklearn-lib:amd64. 2024-04-06T09:27:56.958 INFO:teuthology.orchestra.run.smithi006.stdout:Preparing to unpack .../077-python3-sklearn-lib_0.23.2-5ubuntu6_amd64.deb ... 2024-04-06T09:27:56.999 INFO:teuthology.orchestra.run.smithi006.stdout:Unpacking python3-sklearn-lib:amd64 (0.23.2-5ubuntu6) ... 2024-04-06T09:27:57.154 INFO:teuthology.orchestra.run.smithi192.stdout:Selecting previously unselected package python3-google-auth. 2024-04-06T09:27:57.181 INFO:teuthology.orchestra.run.smithi192.stdout:Preparing to unpack .../086-python3-google-auth_1.5.1-3_all.deb ... 2024-04-06T09:27:57.220 INFO:teuthology.orchestra.run.smithi192.stdout:Unpacking python3-google-auth (1.5.1-3) ... 2024-04-06T09:27:57.520 INFO:teuthology.orchestra.run.smithi006.stdout:Selecting previously unselected package python3-joblib. 2024-04-06T09:27:57.547 INFO:teuthology.orchestra.run.smithi006.stdout:Preparing to unpack .../078-python3-joblib_0.17.0-4ubuntu1_all.deb ... 2024-04-06T09:27:57.565 INFO:teuthology.orchestra.run.smithi192.stdout:Selecting previously unselected package python3-requests-oauthlib. 2024-04-06T09:27:57.586 INFO:teuthology.orchestra.run.smithi006.stdout:Unpacking python3-joblib (0.17.0-4ubuntu1) ... 2024-04-06T09:27:57.592 INFO:teuthology.orchestra.run.smithi192.stdout:Preparing to unpack .../087-python3-requests-oauthlib_1.3.0+ds-0.1_all.deb ... 2024-04-06T09:27:57.632 INFO:teuthology.orchestra.run.smithi192.stdout:Unpacking python3-requests-oauthlib (1.3.0+ds-0.1) ... 2024-04-06T09:27:57.910 INFO:teuthology.orchestra.run.smithi192.stdout:Selecting previously unselected package python3-websocket. 2024-04-06T09:27:57.937 INFO:teuthology.orchestra.run.smithi192.stdout:Preparing to unpack .../088-python3-websocket_1.2.3-1_all.deb ... 2024-04-06T09:27:57.957 INFO:teuthology.orchestra.run.smithi006.stdout:Selecting previously unselected package python3-threadpoolctl. 2024-04-06T09:27:57.968 INFO:teuthology.orchestra.run.smithi192.stdout:Unpacking python3-websocket (1.2.3-1) ... 2024-04-06T09:27:57.984 INFO:teuthology.orchestra.run.smithi006.stdout:Preparing to unpack .../079-python3-threadpoolctl_3.1.0-1_all.deb ... 2024-04-06T09:27:58.023 INFO:teuthology.orchestra.run.smithi006.stdout:Unpacking python3-threadpoolctl (3.1.0-1) ... 2024-04-06T09:27:58.221 INFO:teuthology.orchestra.run.smithi192.stdout:Selecting previously unselected package python3-kubernetes. 2024-04-06T09:27:58.248 INFO:teuthology.orchestra.run.smithi192.stdout:Preparing to unpack .../089-python3-kubernetes_12.0.1-1ubuntu1_all.deb ... 2024-04-06T09:27:58.287 INFO:teuthology.orchestra.run.smithi192.stdout:Unpacking python3-kubernetes (12.0.1-1ubuntu1) ... 2024-04-06T09:27:58.326 INFO:teuthology.orchestra.run.smithi006.stdout:Selecting previously unselected package python3-sklearn. 2024-04-06T09:27:58.353 INFO:teuthology.orchestra.run.smithi006.stdout:Preparing to unpack .../080-python3-sklearn_0.23.2-5ubuntu6_all.deb ... 2024-04-06T09:27:58.392 INFO:teuthology.orchestra.run.smithi006.stdout:Unpacking python3-sklearn (0.23.2-5ubuntu6) ... 2024-04-06T09:27:58.911 INFO:teuthology.orchestra.run.smithi192.stdout:Selecting previously unselected package ceph-mgr-k8sevents. 2024-04-06T09:27:58.939 INFO:teuthology.orchestra.run.smithi192.stdout:Preparing to unpack .../090-ceph-mgr-k8sevents_18.2.2-1059-g4d05f4ca-1jammy_all.deb ... 2024-04-06T09:27:58.977 INFO:teuthology.orchestra.run.smithi192.stdout:Unpacking ceph-mgr-k8sevents (18.2.2-1059-g4d05f4ca-1jammy) ... 2024-04-06T09:27:59.059 INFO:teuthology.orchestra.run.smithi006.stdout:Selecting previously unselected package ceph-mgr-diskprediction-local. 2024-04-06T09:27:59.087 INFO:teuthology.orchestra.run.smithi006.stdout:Preparing to unpack .../081-ceph-mgr-diskprediction-local_18.2.2-1059-g4d05f4ca-1jammy_all.deb ... 2024-04-06T09:27:59.117 INFO:teuthology.orchestra.run.smithi006.stdout:Unpacking ceph-mgr-diskprediction-local (18.2.2-1059-g4d05f4ca-1jammy) ... 2024-04-06T09:27:59.230 INFO:teuthology.orchestra.run.smithi192.stdout:Selecting previously unselected package libonig5:amd64. 2024-04-06T09:27:59.257 INFO:teuthology.orchestra.run.smithi192.stdout:Preparing to unpack .../091-libonig5_6.9.7.1-2build1_amd64.deb ... 2024-04-06T09:27:59.288 INFO:teuthology.orchestra.run.smithi192.stdout:Unpacking libonig5:amd64 (6.9.7.1-2build1) ... 2024-04-06T09:27:59.658 INFO:teuthology.orchestra.run.smithi192.stdout:Selecting previously unselected package libjq1:amd64. 2024-04-06T09:27:59.686 INFO:teuthology.orchestra.run.smithi192.stdout:Preparing to unpack .../092-libjq1_1.6-2.1ubuntu3_amd64.deb ... 2024-04-06T09:27:59.724 INFO:teuthology.orchestra.run.smithi192.stdout:Unpacking libjq1:amd64 (1.6-2.1ubuntu3) ... 2024-04-06T09:27:59.994 INFO:teuthology.orchestra.run.smithi192.stdout:Selecting previously unselected package jq. 2024-04-06T09:28:00.009 INFO:teuthology.orchestra.run.smithi192.stdout:Preparing to unpack .../093-jq_1.6-2.1ubuntu3_amd64.deb ... 2024-04-06T09:28:00.052 INFO:teuthology.orchestra.run.smithi192.stdout:Unpacking jq (1.6-2.1ubuntu3) ... 2024-04-06T09:28:00.229 INFO:teuthology.orchestra.run.smithi006.stdout:Selecting previously unselected package python3-cachetools. 2024-04-06T09:28:00.256 INFO:teuthology.orchestra.run.smithi006.stdout:Preparing to unpack .../082-python3-cachetools_5.0.0-1_all.deb ... 2024-04-06T09:28:00.272 INFO:teuthology.orchestra.run.smithi192.stdout:Selecting previously unselected package socat. 2024-04-06T09:28:00.287 INFO:teuthology.orchestra.run.smithi192.stdout:Preparing to unpack .../094-socat_1.7.4.1-3ubuntu4_amd64.deb ... 2024-04-06T09:28:00.295 INFO:teuthology.orchestra.run.smithi006.stdout:Unpacking python3-cachetools (5.0.0-1) ... 2024-04-06T09:28:00.530 INFO:teuthology.orchestra.run.smithi192.stdout:Unpacking socat (1.7.4.1-3ubuntu4) ... 2024-04-06T09:28:00.683 INFO:teuthology.orchestra.run.smithi006.stdout:Selecting previously unselected package python3-pyasn1. 2024-04-06T09:28:00.710 INFO:teuthology.orchestra.run.smithi006.stdout:Preparing to unpack .../083-python3-pyasn1_0.4.8-1_all.deb ... 2024-04-06T09:28:00.749 INFO:teuthology.orchestra.run.smithi006.stdout:Unpacking python3-pyasn1 (0.4.8-1) ... 2024-04-06T09:28:00.767 INFO:teuthology.orchestra.run.smithi192.stdout:Selecting previously unselected package xmlstarlet. 2024-04-06T09:28:00.782 INFO:teuthology.orchestra.run.smithi192.stdout:Preparing to unpack .../095-xmlstarlet_1.6.1-2.1_amd64.deb ... 2024-04-06T09:28:00.817 INFO:teuthology.orchestra.run.smithi192.stdout:Unpacking xmlstarlet (1.6.1-2.1) ... 2024-04-06T09:28:01.072 INFO:teuthology.orchestra.run.smithi006.stdout:Selecting previously unselected package python3-pyasn1-modules. 2024-04-06T09:28:01.089 INFO:teuthology.orchestra.run.smithi006.stdout:Preparing to unpack .../084-python3-pyasn1-modules_0.2.1-1_all.deb ... 2024-04-06T09:28:01.129 INFO:teuthology.orchestra.run.smithi006.stdout:Unpacking python3-pyasn1-modules (0.2.1-1) ... 2024-04-06T09:28:01.179 INFO:teuthology.orchestra.run.smithi192.stdout:Selecting previously unselected package ceph-test. 2024-04-06T09:28:01.195 INFO:teuthology.orchestra.run.smithi192.stdout:Preparing to unpack .../096-ceph-test_18.2.2-1059-g4d05f4ca-1jammy_amd64.deb ... 2024-04-06T09:28:01.237 INFO:teuthology.orchestra.run.smithi192.stdout:Unpacking ceph-test (18.2.2-1059-g4d05f4ca-1jammy) ... 2024-04-06T09:28:01.416 INFO:teuthology.orchestra.run.smithi006.stdout:Selecting previously unselected package python3-rsa. 2024-04-06T09:28:01.443 INFO:teuthology.orchestra.run.smithi006.stdout:Preparing to unpack .../085-python3-rsa_4.8-1_all.deb ... 2024-04-06T09:28:01.482 INFO:teuthology.orchestra.run.smithi006.stdout:Unpacking python3-rsa (4.8-1) ... 2024-04-06T09:28:01.803 INFO:teuthology.orchestra.run.smithi006.stdout:Selecting previously unselected package python3-google-auth. 2024-04-06T09:28:01.830 INFO:teuthology.orchestra.run.smithi006.stdout:Preparing to unpack .../086-python3-google-auth_1.5.1-3_all.deb ... 2024-04-06T09:28:01.869 INFO:teuthology.orchestra.run.smithi006.stdout:Unpacking python3-google-auth (1.5.1-3) ... 2024-04-06T09:28:02.181 INFO:teuthology.orchestra.run.smithi006.stdout:Selecting previously unselected package python3-requests-oauthlib. 2024-04-06T09:28:02.208 INFO:teuthology.orchestra.run.smithi006.stdout:Preparing to unpack .../087-python3-requests-oauthlib_1.3.0+ds-0.1_all.deb ... 2024-04-06T09:28:02.247 INFO:teuthology.orchestra.run.smithi006.stdout:Unpacking python3-requests-oauthlib (1.3.0+ds-0.1) ... 2024-04-06T09:28:02.575 INFO:teuthology.orchestra.run.smithi006.stdout:Selecting previously unselected package python3-websocket. 2024-04-06T09:28:02.600 INFO:teuthology.orchestra.run.smithi006.stdout:Preparing to unpack .../088-python3-websocket_1.2.3-1_all.deb ... 2024-04-06T09:28:02.641 INFO:teuthology.orchestra.run.smithi006.stdout:Unpacking python3-websocket (1.2.3-1) ... 2024-04-06T09:28:02.945 INFO:teuthology.orchestra.run.smithi006.stdout:Selecting previously unselected package python3-kubernetes. 2024-04-06T09:28:02.972 INFO:teuthology.orchestra.run.smithi006.stdout:Preparing to unpack .../089-python3-kubernetes_12.0.1-1ubuntu1_all.deb ... 2024-04-06T09:28:03.003 INFO:teuthology.orchestra.run.smithi006.stdout:Unpacking python3-kubernetes (12.0.1-1ubuntu1) ... 2024-04-06T09:28:03.167 INFO:teuthology.orchestra.run.smithi192.stdout:Selecting previously unselected package ceph-volume. 2024-04-06T09:28:03.194 INFO:teuthology.orchestra.run.smithi192.stdout:Preparing to unpack .../097-ceph-volume_18.2.2-1059-g4d05f4ca-1jammy_all.deb ... 2024-04-06T09:28:03.233 INFO:teuthology.orchestra.run.smithi192.stdout:Unpacking ceph-volume (18.2.2-1059-g4d05f4ca-1jammy) ... 2024-04-06T09:28:03.538 INFO:teuthology.orchestra.run.smithi192.stdout:Selecting previously unselected package libcephfs-dev. 2024-04-06T09:28:03.553 INFO:teuthology.orchestra.run.smithi192.stdout:Preparing to unpack .../098-libcephfs-dev_18.2.2-1059-g4d05f4ca-1jammy_amd64.deb ... 2024-04-06T09:28:03.587 INFO:teuthology.orchestra.run.smithi192.stdout:Unpacking libcephfs-dev (18.2.2-1059-g4d05f4ca-1jammy) ... 2024-04-06T09:28:03.683 INFO:teuthology.orchestra.run.smithi006.stdout:Selecting previously unselected package ceph-mgr-k8sevents. 2024-04-06T09:28:03.700 INFO:teuthology.orchestra.run.smithi006.stdout:Preparing to unpack .../090-ceph-mgr-k8sevents_18.2.2-1059-g4d05f4ca-1jammy_all.deb ... 2024-04-06T09:28:03.733 INFO:teuthology.orchestra.run.smithi006.stdout:Unpacking ceph-mgr-k8sevents (18.2.2-1059-g4d05f4ca-1jammy) ... 2024-04-06T09:28:03.974 INFO:teuthology.orchestra.run.smithi192.stdout:Selecting previously unselected package nvme-cli. 2024-04-06T09:28:04.002 INFO:teuthology.orchestra.run.smithi192.stdout:Preparing to unpack .../099-nvme-cli_1.16-3ubuntu0.1_amd64.deb ... 2024-04-06T09:28:04.036 INFO:teuthology.orchestra.run.smithi006.stdout:Selecting previously unselected package libonig5:amd64. 2024-04-06T09:28:04.041 INFO:teuthology.orchestra.run.smithi192.stdout:Unpacking nvme-cli (1.16-3ubuntu0.1) ... 2024-04-06T09:28:04.065 INFO:teuthology.orchestra.run.smithi006.stdout:Preparing to unpack .../091-libonig5_6.9.7.1-2build1_amd64.deb ... 2024-04-06T09:28:04.102 INFO:teuthology.orchestra.run.smithi006.stdout:Unpacking libonig5:amd64 (6.9.7.1-2build1) ... 2024-04-06T09:28:04.388 INFO:teuthology.orchestra.run.smithi192.stdout:Selecting previously unselected package python-asyncssh-doc. 2024-04-06T09:28:04.415 INFO:teuthology.orchestra.run.smithi192.stdout:Preparing to unpack .../100-python-asyncssh-doc_2.5.0-1_all.deb ... 2024-04-06T09:28:04.422 INFO:teuthology.orchestra.run.smithi006.stdout:Selecting previously unselected package libjq1:amd64. 2024-04-06T09:28:04.450 INFO:teuthology.orchestra.run.smithi006.stdout:Preparing to unpack .../092-libjq1_1.6-2.1ubuntu3_amd64.deb ... 2024-04-06T09:28:04.454 INFO:teuthology.orchestra.run.smithi192.stdout:Unpacking python-asyncssh-doc (2.5.0-1) ... 2024-04-06T09:28:04.489 INFO:teuthology.orchestra.run.smithi006.stdout:Unpacking libjq1:amd64 (1.6-2.1ubuntu3) ... 2024-04-06T09:28:04.750 INFO:teuthology.orchestra.run.smithi006.stdout:Selecting previously unselected package jq. 2024-04-06T09:28:04.777 INFO:teuthology.orchestra.run.smithi006.stdout:Preparing to unpack .../093-jq_1.6-2.1ubuntu3_amd64.deb ... 2024-04-06T09:28:04.816 INFO:teuthology.orchestra.run.smithi006.stdout:Unpacking jq (1.6-2.1ubuntu3) ... 2024-04-06T09:28:04.891 INFO:teuthology.orchestra.run.smithi192.stdout:Selecting previously unselected package python3-pastescript. 2024-04-06T09:28:04.918 INFO:teuthology.orchestra.run.smithi192.stdout:Preparing to unpack .../101-python3-pastescript_2.0.2-4_all.deb ... 2024-04-06T09:28:04.957 INFO:teuthology.orchestra.run.smithi192.stdout:Unpacking python3-pastescript (2.0.2-4) ... 2024-04-06T09:28:05.035 INFO:teuthology.orchestra.run.smithi006.stdout:Selecting previously unselected package socat. 2024-04-06T09:28:05.051 INFO:teuthology.orchestra.run.smithi006.stdout:Preparing to unpack .../094-socat_1.7.4.1-3ubuntu4_amd64.deb ... 2024-04-06T09:28:05.085 INFO:teuthology.orchestra.run.smithi006.stdout:Unpacking socat (1.7.4.1-3ubuntu4) ... 2024-04-06T09:28:05.311 INFO:teuthology.orchestra.run.smithi192.stdout:Selecting previously unselected package python3-psutil. 2024-04-06T09:28:05.338 INFO:teuthology.orchestra.run.smithi192.stdout:Preparing to unpack .../102-python3-psutil_5.9.0-1build1_amd64.deb ... 2024-04-06T09:28:05.377 INFO:teuthology.orchestra.run.smithi192.stdout:Unpacking python3-psutil (5.9.0-1build1) ... 2024-04-06T09:28:05.552 INFO:teuthology.orchestra.run.smithi006.stdout:Selecting previously unselected package xmlstarlet. 2024-04-06T09:28:05.580 INFO:teuthology.orchestra.run.smithi006.stdout:Preparing to unpack .../095-xmlstarlet_1.6.1-2.1_amd64.deb ... 2024-04-06T09:28:05.622 INFO:teuthology.orchestra.run.smithi006.stdout:Unpacking xmlstarlet (1.6.1-2.1) ... 2024-04-06T09:28:05.822 INFO:teuthology.orchestra.run.smithi192.stdout:Selecting previously unselected package python3-pygments. 2024-04-06T09:28:05.849 INFO:teuthology.orchestra.run.smithi192.stdout:Preparing to unpack .../103-python3-pygments_2.11.2+dfsg-2_all.deb ... 2024-04-06T09:28:05.888 INFO:teuthology.orchestra.run.smithi192.stdout:Unpacking python3-pygments (2.11.2+dfsg-2) ... 2024-04-06T09:28:05.976 INFO:teuthology.orchestra.run.smithi006.stdout:Selecting previously unselected package ceph-test. 2024-04-06T09:28:05.992 INFO:teuthology.orchestra.run.smithi006.stdout:Preparing to unpack .../096-ceph-test_18.2.2-1059-g4d05f4ca-1jammy_amd64.deb ... 2024-04-06T09:28:06.034 INFO:teuthology.orchestra.run.smithi006.stdout:Unpacking ceph-test (18.2.2-1059-g4d05f4ca-1jammy) ... 2024-04-06T09:28:06.360 INFO:teuthology.orchestra.run.smithi192.stdout:Selecting previously unselected package python3-pyinotify. 2024-04-06T09:28:06.387 INFO:teuthology.orchestra.run.smithi192.stdout:Preparing to unpack .../104-python3-pyinotify_0.9.6-1.3_all.deb ... 2024-04-06T09:28:06.426 INFO:teuthology.orchestra.run.smithi192.stdout:Unpacking python3-pyinotify (0.9.6-1.3) ... 2024-04-06T09:28:06.737 INFO:teuthology.orchestra.run.smithi192.stdout:Selecting previously unselected package python3-simplejson. 2024-04-06T09:28:06.764 INFO:teuthology.orchestra.run.smithi192.stdout:Preparing to unpack .../105-python3-simplejson_3.17.6-1build1_amd64.deb ... 2024-04-06T09:28:06.803 INFO:teuthology.orchestra.run.smithi192.stdout:Unpacking python3-simplejson (3.17.6-1build1) ... 2024-04-06T09:28:07.073 INFO:teuthology.orchestra.run.smithi192.stdout:Selecting previously unselected package qttranslations5-l10n. 2024-04-06T09:28:07.089 INFO:teuthology.orchestra.run.smithi192.stdout:Preparing to unpack .../106-qttranslations5-l10n_5.15.3-1_all.deb ... 2024-04-06T09:28:07.122 INFO:teuthology.orchestra.run.smithi192.stdout:Unpacking qttranslations5-l10n (5.15.3-1) ... 2024-04-06T09:28:07.844 INFO:teuthology.orchestra.run.smithi192.stdout:Selecting previously unselected package radosgw. 2024-04-06T09:28:07.872 INFO:teuthology.orchestra.run.smithi192.stdout:Preparing to unpack .../107-radosgw_18.2.2-1059-g4d05f4ca-1jammy_amd64.deb ... 2024-04-06T09:28:07.902 INFO:teuthology.orchestra.run.smithi192.stdout:Unpacking radosgw (18.2.2-1059-g4d05f4ca-1jammy) ... 2024-04-06T09:28:07.947 INFO:teuthology.orchestra.run.smithi006.stdout:Selecting previously unselected package ceph-volume. 2024-04-06T09:28:07.974 INFO:teuthology.orchestra.run.smithi006.stdout:Preparing to unpack .../097-ceph-volume_18.2.2-1059-g4d05f4ca-1jammy_all.deb ... 2024-04-06T09:28:08.021 INFO:teuthology.orchestra.run.smithi006.stdout:Unpacking ceph-volume (18.2.2-1059-g4d05f4ca-1jammy) ... 2024-04-06T09:28:08.334 INFO:teuthology.orchestra.run.smithi006.stdout:Selecting previously unselected package libcephfs-dev. 2024-04-06T09:28:08.362 INFO:teuthology.orchestra.run.smithi006.stdout:Preparing to unpack .../098-libcephfs-dev_18.2.2-1059-g4d05f4ca-1jammy_amd64.deb ... 2024-04-06T09:28:08.400 INFO:teuthology.orchestra.run.smithi006.stdout:Unpacking libcephfs-dev (18.2.2-1059-g4d05f4ca-1jammy) ... 2024-04-06T09:28:08.621 INFO:teuthology.orchestra.run.smithi192.stdout:Selecting previously unselected package rbd-fuse. 2024-04-06T09:28:08.637 INFO:teuthology.orchestra.run.smithi192.stdout:Preparing to unpack .../108-rbd-fuse_18.2.2-1059-g4d05f4ca-1jammy_amd64.deb ... 2024-04-06T09:28:08.679 INFO:teuthology.orchestra.run.smithi192.stdout:Unpacking rbd-fuse (18.2.2-1059-g4d05f4ca-1jammy) ... 2024-04-06T09:28:08.728 INFO:teuthology.orchestra.run.smithi006.stdout:Selecting previously unselected package nvme-cli. 2024-04-06T09:28:08.756 INFO:teuthology.orchestra.run.smithi006.stdout:Preparing to unpack .../099-nvme-cli_1.16-3ubuntu0.1_amd64.deb ... 2024-04-06T09:28:08.794 INFO:teuthology.orchestra.run.smithi006.stdout:Unpacking nvme-cli (1.16-3ubuntu0.1) ... 2024-04-06T09:28:09.016 INFO:teuthology.orchestra.run.smithi192.stdout:Selecting previously unselected package rbd-mirror. 2024-04-06T09:28:09.044 INFO:teuthology.orchestra.run.smithi192.stdout:Preparing to unpack .../109-rbd-mirror_18.2.2-1059-g4d05f4ca-1jammy_amd64.deb ... 2024-04-06T09:28:09.082 INFO:teuthology.orchestra.run.smithi192.stdout:Unpacking rbd-mirror (18.2.2-1059-g4d05f4ca-1jammy) ... 2024-04-06T09:28:09.099 INFO:teuthology.orchestra.run.smithi006.stdout:Selecting previously unselected package python-asyncssh-doc. 2024-04-06T09:28:09.116 INFO:teuthology.orchestra.run.smithi006.stdout:Preparing to unpack .../100-python-asyncssh-doc_2.5.0-1_all.deb ... 2024-04-06T09:28:09.149 INFO:teuthology.orchestra.run.smithi006.stdout:Unpacking python-asyncssh-doc (2.5.0-1) ... 2024-04-06T09:28:09.552 INFO:teuthology.orchestra.run.smithi006.stdout:Selecting previously unselected package python3-pastescript. 2024-04-06T09:28:09.570 INFO:teuthology.orchestra.run.smithi006.stdout:Preparing to unpack .../101-python3-pastescript_2.0.2-4_all.deb ... 2024-04-06T09:28:09.570 INFO:teuthology.orchestra.run.smithi192.stdout:Setting up python3-iniconfig (1.1.1-2) ... 2024-04-06T09:28:09.602 INFO:teuthology.orchestra.run.smithi006.stdout:Unpacking python3-pastescript (2.0.2-4) ... 2024-04-06T09:28:09.829 INFO:teuthology.orchestra.run.smithi192.stdout:Setting up libdouble-conversion3:amd64 (3.1.7-4) ... 2024-04-06T09:28:09.923 INFO:teuthology.orchestra.run.smithi006.stdout:Selecting previously unselected package python3-psutil. 2024-04-06T09:28:09.939 INFO:teuthology.orchestra.run.smithi192.stdout:Setting up nvme-cli (1.16-3ubuntu0.1) ... 2024-04-06T09:28:09.950 INFO:teuthology.orchestra.run.smithi006.stdout:Preparing to unpack .../102-python3-psutil_5.9.0-1build1_amd64.deb ... 2024-04-06T09:28:09.989 INFO:teuthology.orchestra.run.smithi006.stdout:Unpacking python3-psutil (5.9.0-1build1) ... 2024-04-06T09:28:10.176 INFO:teuthology.orchestra.run.smithi192.stdout:Created symlink /etc/systemd/system/default.target.wants/nvmefc-boot-connections.service -> /lib/systemd/system/nvmefc-boot-connections.service. 2024-04-06T09:28:10.492 INFO:teuthology.orchestra.run.smithi006.stdout:Selecting previously unselected package python3-pygments. 2024-04-06T09:28:10.520 INFO:teuthology.orchestra.run.smithi006.stdout:Preparing to unpack .../103-python3-pygments_2.11.2+dfsg-2_all.deb ... 2024-04-06T09:28:10.550 INFO:teuthology.orchestra.run.smithi006.stdout:Unpacking python3-pygments (2.11.2+dfsg-2) ... 2024-04-06T09:28:10.555 INFO:teuthology.orchestra.run.smithi192.stdout:Created symlink /etc/systemd/system/default.target.wants/nvmf-autoconnect.service -> /lib/systemd/system/nvmf-autoconnect.service. 2024-04-06T09:28:10.969 INFO:teuthology.orchestra.run.smithi192.stdout:nvmf-connect.target is a disabled or a static unit, not starting it. 2024-04-06T09:28:11.012 INFO:teuthology.orchestra.run.smithi006.stdout:Selecting previously unselected package python3-pyinotify. 2024-04-06T09:28:11.040 INFO:teuthology.orchestra.run.smithi006.stdout:Preparing to unpack .../104-python3-pyinotify_0.9.6-1.3_all.deb ... 2024-04-06T09:28:11.074 INFO:teuthology.orchestra.run.smithi192.stdout:Setting up cephadm (18.2.2-1059-g4d05f4ca-1jammy) ... 2024-04-06T09:28:11.078 INFO:teuthology.orchestra.run.smithi006.stdout:Unpacking python3-pyinotify (0.9.6-1.3) ... 2024-04-06T09:28:11.398 INFO:teuthology.orchestra.run.smithi006.stdout:Selecting previously unselected package python3-simplejson. 2024-04-06T09:28:11.426 INFO:teuthology.orchestra.run.smithi006.stdout:Preparing to unpack .../105-python3-simplejson_3.17.6-1build1_amd64.deb ... 2024-04-06T09:28:11.464 INFO:teuthology.orchestra.run.smithi006.stdout:Unpacking python3-simplejson (3.17.6-1build1) ... 2024-04-06T09:28:11.642 INFO:teuthology.orchestra.run.smithi192.stdout:Adding system user cephadm....done 2024-04-06T09:28:11.733 INFO:teuthology.orchestra.run.smithi006.stdout:Selecting previously unselected package qttranslations5-l10n. 2024-04-06T09:28:11.750 INFO:teuthology.orchestra.run.smithi006.stdout:Preparing to unpack .../106-qttranslations5-l10n_5.15.3-1_all.deb ... 2024-04-06T09:28:11.792 INFO:teuthology.orchestra.run.smithi006.stdout:Unpacking qttranslations5-l10n (5.15.3-1) ... 2024-04-06T09:28:11.798 INFO:teuthology.orchestra.run.smithi192.stdout:Setting up python3-waitress (1.4.4-1.1ubuntu1) ... 2024-04-06T09:28:12.351 INFO:teuthology.orchestra.run.smithi192.stdout:Setting up python3-jaraco.classes (3.2.1-3) ... 2024-04-06T09:28:12.487 INFO:teuthology.orchestra.run.smithi006.stdout:Selecting previously unselected package radosgw. 2024-04-06T09:28:12.515 INFO:teuthology.orchestra.run.smithi006.stdout:Preparing to unpack .../107-radosgw_18.2.2-1059-g4d05f4ca-1jammy_amd64.deb ... 2024-04-06T09:28:12.553 INFO:teuthology.orchestra.run.smithi006.stdout:Unpacking radosgw (18.2.2-1059-g4d05f4ca-1jammy) ... 2024-04-06T09:28:12.585 INFO:teuthology.orchestra.run.smithi192.stdout:Setting up python-asyncssh-doc (2.5.0-1) ... 2024-04-06T09:28:12.711 INFO:teuthology.orchestra.run.smithi192.stdout:Setting up python3-jaraco.functools (3.4.0-2) ... 2024-04-06T09:28:12.971 INFO:teuthology.orchestra.run.smithi192.stdout:Setting up python3-repoze.lru (0.7-2) ... 2024-04-06T09:28:13.222 INFO:teuthology.orchestra.run.smithi192.stdout:Setting up liboath0:amd64 (2.6.7-3build1) ... 2024-04-06T09:28:13.302 INFO:teuthology.orchestra.run.smithi006.stdout:Selecting previously unselected package rbd-fuse. 2024-04-06T09:28:13.323 INFO:teuthology.orchestra.run.smithi192.stdout:Setting up python3-py (1.10.0-1) ... 2024-04-06T09:28:13.330 INFO:teuthology.orchestra.run.smithi006.stdout:Preparing to unpack .../108-rbd-fuse_18.2.2-1059-g4d05f4ca-1jammy_amd64.deb ... 2024-04-06T09:28:13.365 INFO:teuthology.orchestra.run.smithi006.stdout:Unpacking rbd-fuse (18.2.2-1059-g4d05f4ca-1jammy) ... 2024-04-06T09:28:13.617 INFO:teuthology.orchestra.run.smithi192.stdout:Setting up python3-joblib (0.17.0-4ubuntu1) ... 2024-04-06T09:28:13.668 INFO:teuthology.orchestra.run.smithi006.stdout:Selecting previously unselected package rbd-mirror. 2024-04-06T09:28:13.696 INFO:teuthology.orchestra.run.smithi006.stdout:Preparing to unpack .../109-rbd-mirror_18.2.2-1059-g4d05f4ca-1jammy_amd64.deb ... 2024-04-06T09:28:13.734 INFO:teuthology.orchestra.run.smithi006.stdout:Unpacking rbd-mirror (18.2.2-1059-g4d05f4ca-1jammy) ... 2024-04-06T09:28:13.977 INFO:teuthology.orchestra.run.smithi192.stdout:Setting up python3-cachetools (5.0.0-1) ... 2024-04-06T09:28:14.172 INFO:teuthology.orchestra.run.smithi006.stdout:Setting up python3-iniconfig (1.1.1-2) ... 2024-04-06T09:28:14.228 INFO:teuthology.orchestra.run.smithi192.stdout:Setting up python3-openssl (21.0.0-1) ... 2024-04-06T09:28:14.439 INFO:teuthology.orchestra.run.smithi006.stdout:Setting up libdouble-conversion3:amd64 (3.1.7-4) ... 2024-04-06T09:28:14.513 INFO:teuthology.orchestra.run.smithi192.stdout:Setting up python3-bcrypt (3.2.0-1build1) ... 2024-04-06T09:28:14.540 INFO:teuthology.orchestra.run.smithi006.stdout:Setting up nvme-cli (1.16-3ubuntu0.1) ... 2024-04-06T09:28:14.764 INFO:teuthology.orchestra.run.smithi192.stdout:Setting up python3-pyinotify (0.9.6-1.3) ... 2024-04-06T09:28:14.782 INFO:teuthology.orchestra.run.smithi006.stdout:Created symlink /etc/systemd/system/default.target.wants/nvmefc-boot-connections.service -> /lib/systemd/system/nvmefc-boot-connections.service. 2024-04-06T09:28:15.015 INFO:teuthology.orchestra.run.smithi192.stdout:Setting up python3-threadpoolctl (3.1.0-1) ... 2024-04-06T09:28:15.095 INFO:teuthology.orchestra.run.smithi006.stdout:Created symlink /etc/systemd/system/default.target.wants/nvmf-autoconnect.service -> /lib/systemd/system/nvmf-autoconnect.service. 2024-04-06T09:28:15.258 INFO:teuthology.orchestra.run.smithi192.stdout:Setting up python3-ceph-argparse (18.2.2-1059-g4d05f4ca-1jammy) ... 2024-04-06T09:28:15.509 INFO:teuthology.orchestra.run.smithi192.stdout:Setting up python3-sklearn-lib:amd64 (0.23.2-5ubuntu6) ... 2024-04-06T09:28:15.538 INFO:teuthology.orchestra.run.smithi006.stdout:nvmf-connect.target is a disabled or a static unit, not starting it. 2024-04-06T09:28:15.609 INFO:teuthology.orchestra.run.smithi006.stdout:Setting up cephadm (18.2.2-1059-g4d05f4ca-1jammy) ... 2024-04-06T09:28:15.610 INFO:teuthology.orchestra.run.smithi192.stdout:Setting up python3-markupsafe (2.0.1-2build1) ... 2024-04-06T09:28:16.111 INFO:teuthology.orchestra.run.smithi192.stdout:Setting up libpcre2-16-0:amd64 (10.39-3ubuntu0.1) ... 2024-04-06T09:28:16.116 INFO:teuthology.orchestra.run.smithi006.stdout:Adding system user cephadm....done 2024-04-06T09:28:16.237 INFO:teuthology.orchestra.run.smithi192.stdout:Setting up python3-psutil (5.9.0-1build1) ... 2024-04-06T09:28:16.265 INFO:teuthology.orchestra.run.smithi006.stdout:Setting up python3-waitress (1.4.4-1.1ubuntu1) ... 2024-04-06T09:28:16.533 INFO:teuthology.orchestra.run.smithi006.stdout:Setting up python3-jaraco.classes (3.2.1-3) ... 2024-04-06T09:28:16.597 INFO:teuthology.orchestra.run.smithi192.stdout:Setting up python-babel-localedata (2.8.0+dfsg.1-7) ... 2024-04-06T09:28:16.723 INFO:teuthology.orchestra.run.smithi192.stdout:Setting up python3-natsort (8.0.2-1) ... 2024-04-06T09:28:16.792 INFO:teuthology.orchestra.run.smithi006.stdout:Setting up python-asyncssh-doc (2.5.0-1) ... 2024-04-06T09:28:16.943 INFO:teuthology.orchestra.run.smithi006.stdout:Setting up python3-jaraco.functools (3.4.0-2) ... 2024-04-06T09:28:16.999 INFO:teuthology.orchestra.run.smithi192.stdout:Setting up python3-routes (2.5.1-1ubuntu1) ... 2024-04-06T09:28:17.211 INFO:teuthology.orchestra.run.smithi006.stdout:Setting up python3-repoze.lru (0.7-2) ... 2024-04-06T09:28:17.267 INFO:teuthology.orchestra.run.smithi192.stdout:Setting up python3-simplejson (3.17.6-1build1) ... 2024-04-06T09:28:17.470 INFO:teuthology.orchestra.run.smithi006.stdout:Setting up liboath0:amd64 (2.6.7-3build1) ... 2024-04-06T09:28:17.578 INFO:teuthology.orchestra.run.smithi192.stdout:Setting up python3-pygments (2.11.2+dfsg-2) ... 2024-04-06T09:28:17.596 INFO:teuthology.orchestra.run.smithi006.stdout:Setting up python3-py (1.10.0-1) ... 2024-04-06T09:28:17.923 INFO:teuthology.orchestra.run.smithi006.stdout:Setting up python3-joblib (0.17.0-4ubuntu1) ... 2024-04-06T09:28:18.245 INFO:teuthology.orchestra.run.smithi192.stdout:Setting up python3-tempita (0.5.2-6ubuntu1) ... 2024-04-06T09:28:18.266 INFO:teuthology.orchestra.run.smithi006.stdout:Setting up python3-cachetools (5.0.0-1) ... 2024-04-06T09:28:18.504 INFO:teuthology.orchestra.run.smithi192.stdout:Setting up python-pastedeploy-tpl (2.1.1-1) ... 2024-04-06T09:28:18.543 INFO:teuthology.orchestra.run.smithi006.stdout:Setting up python3-openssl (21.0.0-1) ... 2024-04-06T09:28:18.606 INFO:teuthology.orchestra.run.smithi192.stdout:Setting up qttranslations5-l10n (5.15.3-1) ... 2024-04-06T09:28:18.707 INFO:teuthology.orchestra.run.smithi192.stdout:Setting up python3-wcwidth (0.2.5+dfsg1-1) ... 2024-04-06T09:28:18.819 INFO:teuthology.orchestra.run.smithi006.stdout:Setting up python3-bcrypt (3.2.0-1build1) ... 2024-04-06T09:28:19.050 INFO:teuthology.orchestra.run.smithi192.stdout:Setting up python3-asyncssh (2.5.0-1) ... 2024-04-06T09:28:19.078 INFO:teuthology.orchestra.run.smithi006.stdout:Setting up python3-pyinotify (0.9.6-1.3) ... 2024-04-06T09:28:19.346 INFO:teuthology.orchestra.run.smithi006.stdout:Setting up python3-threadpoolctl (3.1.0-1) ... 2024-04-06T09:28:19.442 INFO:teuthology.orchestra.run.smithi192.stdout:Setting up python3-certifi (2020.6.20-1) ... 2024-04-06T09:28:19.622 INFO:teuthology.orchestra.run.smithi006.stdout:Setting up python3-ceph-argparse (18.2.2-1059-g4d05f4ca-1jammy) ... 2024-04-06T09:28:19.677 INFO:teuthology.orchestra.run.smithi192.stdout:Setting up python3-paste (3.5.0+dfsg1-1) ... 2024-04-06T09:28:19.873 INFO:teuthology.orchestra.run.smithi006.stdout:Setting up python3-sklearn-lib:amd64 (0.23.2-5ubuntu6) ... 2024-04-06T09:28:20.045 INFO:teuthology.orchestra.run.smithi192.stdout:Setting up python3-cheroot (8.5.2+ds1-1ubuntu3.1) ... 2024-04-06T09:28:20.141 INFO:teuthology.orchestra.run.smithi006.stdout:Setting up python3-markupsafe (2.0.1-2build1) ... 2024-04-06T09:28:20.330 INFO:teuthology.orchestra.run.smithi192.stdout:Setting up python3-werkzeug (2.0.2+dfsg1-1ubuntu0.22.04.1) ... 2024-04-06T09:28:20.383 INFO:teuthology.orchestra.run.smithi006.stdout:Setting up libpcre2-16-0:amd64 (10.39-3ubuntu0.1) ... 2024-04-06T09:28:20.543 INFO:teuthology.orchestra.run.smithi006.stdout:Setting up python3-psutil (5.9.0-1build1) ... 2024-04-06T09:28:20.656 INFO:teuthology.orchestra.run.smithi192.stdout:Setting up python3-jaraco.text (3.6.0-2) ... 2024-04-06T09:28:20.891 INFO:teuthology.orchestra.run.smithi192.stdout:Setting up socat (1.7.4.1-3ubuntu4) ... 2024-04-06T09:28:20.944 INFO:teuthology.orchestra.run.smithi006.stdout:Setting up python-babel-localedata (2.8.0+dfsg.1-7) ... 2024-04-06T09:28:21.017 INFO:teuthology.orchestra.run.smithi192.stdout:Setting up python3-ceph-common (18.2.2-1059-g4d05f4ca-1jammy) ... 2024-04-06T09:28:21.078 INFO:teuthology.orchestra.run.smithi006.stdout:Setting up python3-natsort (8.0.2-1) ... 2024-04-06T09:28:21.310 INFO:teuthology.orchestra.run.smithi192.stdout:Setting up python3-idna (3.3-1) ... 2024-04-06T09:28:21.354 INFO:teuthology.orchestra.run.smithi006.stdout:Setting up python3-routes (2.5.1-1ubuntu1) ... 2024-04-06T09:28:21.622 INFO:teuthology.orchestra.run.smithi006.stdout:Setting up python3-simplejson (3.17.6-1build1) ... 2024-04-06T09:28:21.627 INFO:teuthology.orchestra.run.smithi192.stdout:Setting up python3-sklearn (0.23.2-5ubuntu6) ... 2024-04-06T09:28:21.932 INFO:teuthology.orchestra.run.smithi006.stdout:Setting up python3-pygments (2.11.2+dfsg-2) ... 2024-04-06T09:28:22.626 INFO:teuthology.orchestra.run.smithi006.stdout:Setting up python3-tempita (0.5.2-6ubuntu1) ... 2024-04-06T09:28:22.817 INFO:teuthology.orchestra.run.smithi192.stdout:Setting up libqt5core5a:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-04-06T09:28:22.893 INFO:teuthology.orchestra.run.smithi006.stdout:Setting up python-pastedeploy-tpl (2.1.1-1) ... 2024-04-06T09:28:22.952 INFO:teuthology.orchestra.run.smithi192.stdout:Setting up python3-toml (0.10.2-1) ... 2024-04-06T09:28:23.019 INFO:teuthology.orchestra.run.smithi006.stdout:Setting up qttranslations5-l10n (5.15.3-1) ... 2024-04-06T09:28:23.145 INFO:teuthology.orchestra.run.smithi006.stdout:Setting up python3-wcwidth (0.2.5+dfsg1-1) ... 2024-04-06T09:28:23.211 INFO:teuthology.orchestra.run.smithi192.stdout:Setting up librdkafka1:amd64 (1.8.0-1build1) ... 2024-04-06T09:28:23.337 INFO:teuthology.orchestra.run.smithi192.stdout:Setting up xmlstarlet (1.6.1-2.1) ... 2024-04-06T09:28:23.454 INFO:teuthology.orchestra.run.smithi006.stdout:Setting up python3-asyncssh (2.5.0-1) ... 2024-04-06T09:28:23.463 INFO:teuthology.orchestra.run.smithi192.stdout:Setting up python3-urllib3 (1.26.5-1~exp1ubuntu0.1) ... 2024-04-06T09:28:23.739 INFO:teuthology.orchestra.run.smithi192.stdout:Setting up python3-pluggy (0.13.0-7.1) ... 2024-04-06T09:28:23.864 INFO:teuthology.orchestra.run.smithi006.stdout:Setting up python3-certifi (2020.6.20-1) ... 2024-04-06T09:28:23.983 INFO:teuthology.orchestra.run.smithi192.stdout:Setting up python3-zc.lockfile (2.0-1) ... 2024-04-06T09:28:24.115 INFO:teuthology.orchestra.run.smithi006.stdout:Setting up python3-paste (3.5.0+dfsg1-1) ... 2024-04-06T09:28:24.225 INFO:teuthology.orchestra.run.smithi192.stdout:Setting up libqt5dbus5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-04-06T09:28:24.351 INFO:teuthology.orchestra.run.smithi192.stdout:Setting up libfmt8:amd64 (8.1.1+ds1-2) ... 2024-04-06T09:28:24.477 INFO:teuthology.orchestra.run.smithi192.stdout:Setting up python3-pyasn1 (0.4.8-1) ... 2024-04-06T09:28:24.500 INFO:teuthology.orchestra.run.smithi006.stdout:Setting up python3-cheroot (8.5.2+ds1-1ubuntu3.1) ... 2024-04-06T09:28:24.787 INFO:teuthology.orchestra.run.smithi192.stdout:Setting up python3-singledispatch (3.4.0.3-3) ... 2024-04-06T09:28:24.793 INFO:teuthology.orchestra.run.smithi006.stdout:Setting up python3-werkzeug (2.0.2+dfsg1-1ubuntu0.22.04.1) ... 2024-04-06T09:28:25.047 INFO:teuthology.orchestra.run.smithi192.stdout:Setting up python3-logutils (0.3.3-8) ... 2024-04-06T09:28:25.146 INFO:teuthology.orchestra.run.smithi006.stdout:Setting up python3-jaraco.text (3.6.0-2) ... 2024-04-06T09:28:25.289 INFO:teuthology.orchestra.run.smithi192.stdout:Setting up python3-tempora (4.1.2-1) ... 2024-04-06T09:28:25.403 INFO:teuthology.orchestra.run.smithi006.stdout:Setting up socat (1.7.4.1-3ubuntu4) ... 2024-04-06T09:28:25.530 INFO:teuthology.orchestra.run.smithi006.stdout:Setting up python3-ceph-common (18.2.2-1059-g4d05f4ca-1jammy) ... 2024-04-06T09:28:25.532 INFO:teuthology.orchestra.run.smithi192.stdout:Setting up python3-simplegeneric (0.8.1-3) ... 2024-04-06T09:28:25.792 INFO:teuthology.orchestra.run.smithi192.stdout:Setting up python3-prettytable (2.5.0-2) ... 2024-04-06T09:28:25.856 INFO:teuthology.orchestra.run.smithi006.stdout:Setting up python3-idna (3.3-1) ... 2024-04-06T09:28:26.059 INFO:teuthology.orchestra.run.smithi192.stdout:Setting up liblttng-ust1:amd64 (2.13.1-1ubuntu1) ... 2024-04-06T09:28:26.167 INFO:teuthology.orchestra.run.smithi006.stdout:Setting up python3-sklearn (0.23.2-5ubuntu6) ... 2024-04-06T09:28:26.185 INFO:teuthology.orchestra.run.smithi192.stdout:Setting up python3-websocket (1.2.3-1) ... 2024-04-06T09:28:26.470 INFO:teuthology.orchestra.run.smithi192.stdout:Setting up libonig5:amd64 (6.9.7.1-2build1) ... 2024-04-06T09:28:26.596 INFO:teuthology.orchestra.run.smithi192.stdout:Setting up python3-mako (1.1.3+ds1-2ubuntu0.1) ... 2024-04-06T09:28:26.880 INFO:teuthology.orchestra.run.smithi192.stdout:Setting up python3-webob (1:1.8.6-1.1) ... 2024-04-06T09:28:27.157 INFO:teuthology.orchestra.run.smithi192.stdout:Setting up python3-jaraco.collections (3.4.0-2) ... 2024-04-06T09:28:27.404 INFO:teuthology.orchestra.run.smithi006.stdout:Setting up libqt5core5a:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-04-06T09:28:27.424 INFO:teuthology.orchestra.run.smithi192.stdout:Setting up python3-babel (2.8.0+dfsg.1-7) ... 2024-04-06T09:28:27.530 INFO:teuthology.orchestra.run.smithi006.stdout:Setting up python3-toml (0.10.2-1) ... 2024-04-06T09:28:27.798 INFO:teuthology.orchestra.run.smithi192.stdout:update-alternatives: using /usr/bin/pybabel-python3 to provide /usr/bin/pybabel (pybabel) in auto mode 2024-04-06T09:28:27.823 INFO:teuthology.orchestra.run.smithi006.stdout:Setting up librdkafka1:amd64 (1.8.0-1build1) ... 2024-04-06T09:28:27.949 INFO:teuthology.orchestra.run.smithi006.stdout:Setting up xmlstarlet (1.6.1-2.1) ... 2024-04-06T09:28:28.036 INFO:teuthology.orchestra.run.smithi192.stdout:Setting up libjq1:amd64 (1.6-2.1ubuntu3) ... 2024-04-06T09:28:28.075 INFO:teuthology.orchestra.run.smithi006.stdout:Setting up python3-urllib3 (1.26.5-1~exp1ubuntu0.1) ... 2024-04-06T09:28:28.162 INFO:teuthology.orchestra.run.smithi192.stdout:Setting up python3-pytest (6.2.5-1ubuntu2) ... 2024-04-06T09:28:28.385 INFO:teuthology.orchestra.run.smithi006.stdout:Setting up python3-pluggy (0.13.0-7.1) ... 2024-04-06T09:28:28.555 INFO:teuthology.orchestra.run.smithi192.stdout:Setting up python3-pastedeploy (2.1.1-1) ... 2024-04-06T09:28:28.644 INFO:teuthology.orchestra.run.smithi006.stdout:Setting up python3-zc.lockfile (2.0-1) ... 2024-04-06T09:28:28.790 INFO:teuthology.orchestra.run.smithi192.stdout:Setting up python3-portend (3.0.0-1) ... 2024-04-06T09:28:28.878 INFO:teuthology.orchestra.run.smithi006.stdout:Setting up libqt5dbus5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-04-06T09:28:29.004 INFO:teuthology.orchestra.run.smithi006.stdout:Setting up libfmt8:amd64 (8.1.1+ds1-2) ... 2024-04-06T09:28:29.033 INFO:teuthology.orchestra.run.smithi192.stdout:Setting up libqt5network5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-04-06T09:28:29.131 INFO:teuthology.orchestra.run.smithi006.stdout:Setting up python3-pyasn1 (0.4.8-1) ... 2024-04-06T09:28:29.159 INFO:teuthology.orchestra.run.smithi192.stdout:Setting up python3-jinja2 (3.0.3-1ubuntu0.1) ... 2024-04-06T09:28:29.440 INFO:teuthology.orchestra.run.smithi006.stdout:Setting up python3-singledispatch (3.4.0.3-3) ... 2024-04-06T09:28:29.485 INFO:teuthology.orchestra.run.smithi192.stdout:Setting up python3-pyasn1-modules (0.2.1-1) ... 2024-04-06T09:28:29.683 INFO:teuthology.orchestra.run.smithi006.stdout:Setting up python3-logutils (0.3.3-8) ... 2024-04-06T09:28:29.786 INFO:teuthology.orchestra.run.smithi192.stdout:Setting up python3-requests (2.25.1+dfsg-2ubuntu0.1) ... 2024-04-06T09:28:29.950 INFO:teuthology.orchestra.run.smithi006.stdout:Setting up python3-tempora (4.1.2-1) ... 2024-04-06T09:28:30.062 INFO:teuthology.orchestra.run.smithi192.stdout:Setting up jq (1.6-2.1ubuntu3) ... 2024-04-06T09:28:30.185 INFO:teuthology.orchestra.run.smithi006.stdout:Setting up python3-simplegeneric (0.8.1-3) ... 2024-04-06T09:28:30.188 INFO:teuthology.orchestra.run.smithi192.stdout:Setting up python3-rsa (4.8-1) ... 2024-04-06T09:28:30.419 INFO:teuthology.orchestra.run.smithi006.stdout:Setting up python3-prettytable (2.5.0-2) ... 2024-04-06T09:28:30.497 INFO:teuthology.orchestra.run.smithi192.stdout:Setting up python3-webtest (2.0.35-1) ... 2024-04-06T09:28:30.695 INFO:teuthology.orchestra.run.smithi006.stdout:Setting up liblttng-ust1:amd64 (2.13.1-1ubuntu1) ... 2024-04-06T09:28:30.741 INFO:teuthology.orchestra.run.smithi192.stdout:Setting up python3-requests-oauthlib (1.3.0+ds-0.1) ... 2024-04-06T09:28:30.821 INFO:teuthology.orchestra.run.smithi006.stdout:Setting up python3-websocket (1.2.3-1) ... 2024-04-06T09:28:30.984 INFO:teuthology.orchestra.run.smithi192.stdout:Setting up python3-cherrypy3 (18.6.1-4) ... 2024-04-06T09:28:31.123 INFO:teuthology.orchestra.run.smithi006.stdout:Setting up libonig5:amd64 (6.9.7.1-2build1) ... 2024-04-06T09:28:31.382 INFO:teuthology.orchestra.run.smithi006.stdout:Setting up python3-mako (1.1.3+ds1-2ubuntu0.1) ... 2024-04-06T09:28:31.393 INFO:teuthology.orchestra.run.smithi192.stdout:Setting up python3-pastescript (2.0.2-4) ... 2024-04-06T09:28:31.675 INFO:teuthology.orchestra.run.smithi006.stdout:Setting up python3-webob (1:1.8.6-1.1) ... 2024-04-06T09:28:31.678 INFO:teuthology.orchestra.run.smithi192.stdout:Setting up python3-pecan (1.3.3-4ubuntu2) ... 2024-04-06T09:28:31.976 INFO:teuthology.orchestra.run.smithi006.stdout:Setting up python3-jaraco.collections (3.4.0-2) ... 2024-04-06T09:28:32.038 INFO:teuthology.orchestra.run.smithi192.stdout:Setting up libthrift-0.16.0:amd64 (0.16.0-2) ... 2024-04-06T09:28:32.148 INFO:teuthology.orchestra.run.smithi192.stdout:Setting up librados2 (18.2.2-1059-g4d05f4ca-1jammy) ... 2024-04-06T09:28:32.235 INFO:teuthology.orchestra.run.smithi006.stdout:Setting up python3-babel (2.8.0+dfsg.1-7) ... 2024-04-06T09:28:32.257 INFO:teuthology.orchestra.run.smithi192.stdout:Setting up librgw2 (18.2.2-1059-g4d05f4ca-1jammy) ... 2024-04-06T09:28:32.367 INFO:teuthology.orchestra.run.smithi192.stdout:Setting up libsqlite3-mod-ceph (18.2.2-1059-g4d05f4ca-1jammy) ... 2024-04-06T09:28:32.493 INFO:teuthology.orchestra.run.smithi192.stdout:Setting up libcephfs2 (18.2.2-1059-g4d05f4ca-1jammy) ... 2024-04-06T09:28:32.600 INFO:teuthology.orchestra.run.smithi006.stdout:update-alternatives: using /usr/bin/pybabel-python3 to provide /usr/bin/pybabel (pybabel) in auto mode 2024-04-06T09:28:32.619 INFO:teuthology.orchestra.run.smithi192.stdout:Setting up libradosstriper1 (18.2.2-1059-g4d05f4ca-1jammy) ... 2024-04-06T09:28:32.830 INFO:teuthology.orchestra.run.smithi192.stdout:Setting up python3-google-auth (1.5.1-3) ... 2024-04-06T09:28:32.896 INFO:teuthology.orchestra.run.smithi006.stdout:Setting up libjq1:amd64 (1.6-2.1ubuntu3) ... 2024-04-06T09:28:33.031 INFO:teuthology.orchestra.run.smithi006.stdout:Setting up python3-pytest (6.2.5-1ubuntu2) ... 2024-04-06T09:28:33.124 INFO:teuthology.orchestra.run.smithi192.stdout:Setting up librbd1 (18.2.2-1059-g4d05f4ca-1jammy) ... 2024-04-06T09:28:33.225 INFO:teuthology.orchestra.run.smithi192.stdout:Setting up ceph-mgr-modules-core (18.2.2-1059-g4d05f4ca-1jammy) ... 2024-04-06T09:28:33.351 INFO:teuthology.orchestra.run.smithi192.stdout:Setting up ceph-fuse (18.2.2-1059-g4d05f4ca-1jammy) ... 2024-04-06T09:28:33.441 INFO:teuthology.orchestra.run.smithi006.stdout:Setting up python3-pastedeploy (2.1.1-1) ... 2024-04-06T09:28:33.561 INFO:teuthology.orchestra.run.smithi192.stdout:Created symlink /etc/systemd/system/remote-fs.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2024-04-06T09:28:33.561 INFO:teuthology.orchestra.run.smithi192.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2024-04-06T09:28:33.700 INFO:teuthology.orchestra.run.smithi006.stdout:Setting up python3-portend (3.0.0-1) ... 2024-04-06T09:28:33.951 INFO:teuthology.orchestra.run.smithi006.stdout:Setting up libqt5network5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-04-06T09:28:34.036 INFO:teuthology.orchestra.run.smithi192.stdout:Setting up libcephfs-dev (18.2.2-1059-g4d05f4ca-1jammy) ... 2024-04-06T09:28:34.052 INFO:teuthology.orchestra.run.smithi006.stdout:Setting up python3-jinja2 (3.0.3-1ubuntu0.1) ... 2024-04-06T09:28:34.196 INFO:teuthology.orchestra.run.smithi192.stdout:Setting up python3-rados (18.2.2-1059-g4d05f4ca-1jammy) ... 2024-04-06T09:28:34.322 INFO:teuthology.orchestra.run.smithi192.stdout:Setting up python3-kubernetes (12.0.1-1ubuntu1) ... 2024-04-06T09:28:34.361 INFO:teuthology.orchestra.run.smithi006.stdout:Setting up python3-pyasn1-modules (0.2.1-1) ... 2024-04-06T09:28:34.679 INFO:teuthology.orchestra.run.smithi006.stdout:Setting up python3-requests (2.25.1+dfsg-2ubuntu0.1) ... 2024-04-06T09:28:35.039 INFO:teuthology.orchestra.run.smithi006.stdout:Setting up jq (1.6-2.1ubuntu3) ... 2024-04-06T09:28:35.156 INFO:teuthology.orchestra.run.smithi006.stdout:Setting up python3-rsa (4.8-1) ... 2024-04-06T09:28:35.457 INFO:teuthology.orchestra.run.smithi006.stdout:Setting up python3-webtest (2.0.35-1) ... 2024-04-06T09:28:35.481 INFO:teuthology.orchestra.run.smithi192.stdout:Setting up python3-rbd (18.2.2-1059-g4d05f4ca-1jammy) ... 2024-04-06T09:28:35.610 INFO:teuthology.orchestra.run.smithi192.stdout:Setting up rbd-fuse (18.2.2-1059-g4d05f4ca-1jammy) ... 2024-04-06T09:28:35.733 INFO:teuthology.orchestra.run.smithi006.stdout:Setting up python3-requests-oauthlib (1.3.0+ds-0.1) ... 2024-04-06T09:28:35.736 INFO:teuthology.orchestra.run.smithi192.stdout:Setting up python3-rgw (18.2.2-1059-g4d05f4ca-1jammy) ... 2024-04-06T09:28:35.862 INFO:teuthology.orchestra.run.smithi192.stdout:Setting up python3-cephfs (18.2.2-1059-g4d05f4ca-1jammy) ... 2024-04-06T09:28:35.988 INFO:teuthology.orchestra.run.smithi192.stdout:Setting up ceph-common (18.2.2-1059-g4d05f4ca-1jammy) ... 2024-04-06T09:28:35.993 INFO:teuthology.orchestra.run.smithi006.stdout:Setting up python3-cherrypy3 (18.6.1-4) ... 2024-04-06T09:28:36.386 INFO:teuthology.orchestra.run.smithi006.stdout:Setting up python3-pastescript (2.0.2-4) ... 2024-04-06T09:28:36.488 INFO:teuthology.orchestra.run.smithi192.stdout:Adding group ceph....done 2024-04-06T09:28:36.670 INFO:teuthology.orchestra.run.smithi006.stdout:Setting up python3-pecan (1.3.3-4ubuntu2) ... 2024-04-06T09:28:36.857 INFO:teuthology.orchestra.run.smithi192.stdout:Adding system user ceph....done 2024-04-06T09:28:37.026 INFO:teuthology.orchestra.run.smithi192.stdout:Setting system user ceph properties....done 2024-04-06T09:28:37.042 INFO:teuthology.orchestra.run.smithi192.stdout:chown: cannot access '/var/log/ceph/*.log*': No such file or directory 2024-04-06T09:28:37.064 INFO:teuthology.orchestra.run.smithi006.stdout:Setting up libthrift-0.16.0:amd64 (0.16.0-2) ... 2024-04-06T09:28:37.180 INFO:teuthology.orchestra.run.smithi192.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target -> /lib/systemd/system/ceph.target. 2024-04-06T09:28:37.190 INFO:teuthology.orchestra.run.smithi006.stdout:Setting up librados2 (18.2.2-1059-g4d05f4ca-1jammy) ... 2024-04-06T09:28:37.316 INFO:teuthology.orchestra.run.smithi006.stdout:Setting up librgw2 (18.2.2-1059-g4d05f4ca-1jammy) ... 2024-04-06T09:28:37.443 INFO:teuthology.orchestra.run.smithi006.stdout:Setting up libsqlite3-mod-ceph (18.2.2-1059-g4d05f4ca-1jammy) ... 2024-04-06T09:28:37.486 INFO:teuthology.orchestra.run.smithi192.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/rbdmap.service -> /lib/systemd/system/rbdmap.service. 2024-04-06T09:28:37.551 INFO:teuthology.orchestra.run.smithi006.stdout:Setting up libcephfs2 (18.2.2-1059-g4d05f4ca-1jammy) ... 2024-04-06T09:28:37.661 INFO:teuthology.orchestra.run.smithi006.stdout:Setting up libradosstriper1 (18.2.2-1059-g4d05f4ca-1jammy) ... 2024-04-06T09:28:37.856 INFO:teuthology.orchestra.run.smithi006.stdout:Setting up python3-google-auth (1.5.1-3) ... 2024-04-06T09:28:37.964 INFO:teuthology.orchestra.run.smithi192.stdout:Setting up ceph-test (18.2.2-1059-g4d05f4ca-1jammy) ... 2024-04-06T09:28:38.064 INFO:teuthology.orchestra.run.smithi192.stdout:Setting up radosgw (18.2.2-1059-g4d05f4ca-1jammy) ... 2024-04-06T09:28:38.132 INFO:teuthology.orchestra.run.smithi006.stdout:Setting up librbd1 (18.2.2-1059-g4d05f4ca-1jammy) ... 2024-04-06T09:28:38.241 INFO:teuthology.orchestra.run.smithi006.stdout:Setting up ceph-mgr-modules-core (18.2.2-1059-g4d05f4ca-1jammy) ... 2024-04-06T09:28:38.384 INFO:teuthology.orchestra.run.smithi006.stdout:Setting up ceph-fuse (18.2.2-1059-g4d05f4ca-1jammy) ... 2024-04-06T09:28:38.526 INFO:teuthology.orchestra.run.smithi192.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2024-04-06T09:28:38.526 INFO:teuthology.orchestra.run.smithi192.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2024-04-06T09:28:38.605 INFO:teuthology.orchestra.run.smithi006.stdout:Created symlink /etc/systemd/system/remote-fs.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2024-04-06T09:28:38.605 INFO:teuthology.orchestra.run.smithi006.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2024-04-06T09:28:38.992 INFO:teuthology.orchestra.run.smithi192.stdout:Setting up ceph-base (18.2.2-1059-g4d05f4ca-1jammy) ... 2024-04-06T09:28:39.119 INFO:teuthology.orchestra.run.smithi006.stdout:Setting up libcephfs-dev (18.2.2-1059-g4d05f4ca-1jammy) ... 2024-04-06T09:28:39.262 INFO:teuthology.orchestra.run.smithi006.stdout:Setting up python3-rados (18.2.2-1059-g4d05f4ca-1jammy) ... 2024-04-06T09:28:39.356 INFO:teuthology.orchestra.run.smithi192.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service -> /lib/systemd/system/ceph-crash.service. 2024-04-06T09:28:39.404 INFO:teuthology.orchestra.run.smithi006.stdout:Setting up python3-kubernetes (12.0.1-1ubuntu1) ... 2024-04-06T09:28:39.827 INFO:teuthology.orchestra.run.smithi192.stdout:Setting up ceph-mds (18.2.2-1059-g4d05f4ca-1jammy) ... 2024-04-06T09:28:40.054 INFO:teuthology.orchestra.run.smithi192.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2024-04-06T09:28:40.055 INFO:teuthology.orchestra.run.smithi192.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2024-04-06T09:28:40.528 INFO:teuthology.orchestra.run.smithi192.stdout:Setting up rbd-mirror (18.2.2-1059-g4d05f4ca-1jammy) ... 2024-04-06T09:28:40.668 INFO:teuthology.orchestra.run.smithi006.stdout:Setting up python3-rbd (18.2.2-1059-g4d05f4ca-1jammy) ... 2024-04-06T09:28:40.738 INFO:teuthology.orchestra.run.smithi192.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target -> /lib/systemd/system/ceph-rbd-mirror.target. 2024-04-06T09:28:40.738 INFO:teuthology.orchestra.run.smithi192.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target -> /lib/systemd/system/ceph-rbd-mirror.target. 2024-04-06T09:28:40.794 INFO:teuthology.orchestra.run.smithi006.stdout:Setting up rbd-fuse (18.2.2-1059-g4d05f4ca-1jammy) ... 2024-04-06T09:28:40.920 INFO:teuthology.orchestra.run.smithi006.stdout:Setting up python3-rgw (18.2.2-1059-g4d05f4ca-1jammy) ... 2024-04-06T09:28:41.046 INFO:teuthology.orchestra.run.smithi006.stdout:Setting up python3-cephfs (18.2.2-1059-g4d05f4ca-1jammy) ... 2024-04-06T09:28:41.156 INFO:teuthology.orchestra.run.smithi006.stdout:Setting up ceph-common (18.2.2-1059-g4d05f4ca-1jammy) ... 2024-04-06T09:28:41.204 INFO:teuthology.orchestra.run.smithi192.stdout:Setting up ceph-mgr (18.2.2-1059-g4d05f4ca-1jammy) ... 2024-04-06T09:28:41.419 INFO:teuthology.orchestra.run.smithi192.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2024-04-06T09:28:41.419 INFO:teuthology.orchestra.run.smithi192.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2024-04-06T09:28:41.674 INFO:teuthology.orchestra.run.smithi006.stdout:Adding group ceph....done 2024-04-06T09:28:41.872 INFO:teuthology.orchestra.run.smithi192.stdout:Setting up ceph-osd (18.2.2-1059-g4d05f4ca-1jammy) ... 2024-04-06T09:28:42.000 INFO:teuthology.orchestra.run.smithi006.stdout:Adding system user ceph....done 2024-04-06T09:28:42.144 INFO:teuthology.orchestra.run.smithi192.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2024-04-06T09:28:42.144 INFO:teuthology.orchestra.run.smithi192.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2024-04-06T09:28:42.169 INFO:teuthology.orchestra.run.smithi006.stdout:Setting system user ceph properties....done 2024-04-06T09:28:42.183 INFO:teuthology.orchestra.run.smithi006.stdout:chown: cannot access '/var/log/ceph/*.log*': No such file or directory 2024-04-06T09:28:42.328 INFO:teuthology.orchestra.run.smithi006.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target -> /lib/systemd/system/ceph.target. 2024-04-06T09:28:42.661 INFO:teuthology.orchestra.run.smithi006.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/rbdmap.service -> /lib/systemd/system/rbdmap.service. 2024-04-06T09:28:42.665 INFO:teuthology.orchestra.run.smithi192.stdout:Setting up ceph-mgr-k8sevents (18.2.2-1059-g4d05f4ca-1jammy) ... 2024-04-06T09:28:42.791 INFO:teuthology.orchestra.run.smithi192.stdout:Setting up ceph-mgr-diskprediction-local (18.2.2-1059-g4d05f4ca-1jammy) ... 2024-04-06T09:28:42.951 INFO:teuthology.orchestra.run.smithi192.stdout:Setting up ceph-mon (18.2.2-1059-g4d05f4ca-1jammy) ... 2024-04-06T09:28:43.124 INFO:teuthology.orchestra.run.smithi006.stdout:Setting up ceph-test (18.2.2-1059-g4d05f4ca-1jammy) ... 2024-04-06T09:28:43.164 INFO:teuthology.orchestra.run.smithi192.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2024-04-06T09:28:43.164 INFO:teuthology.orchestra.run.smithi192.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2024-04-06T09:28:43.241 INFO:teuthology.orchestra.run.smithi006.stdout:Setting up radosgw (18.2.2-1059-g4d05f4ca-1jammy) ... 2024-04-06T09:28:43.644 INFO:teuthology.orchestra.run.smithi192.stdout:Setting up ceph-mgr-cephadm (18.2.2-1059-g4d05f4ca-1jammy) ... 2024-04-06T09:28:43.726 INFO:teuthology.orchestra.run.smithi006.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2024-04-06T09:28:43.726 INFO:teuthology.orchestra.run.smithi006.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2024-04-06T09:28:43.795 INFO:teuthology.orchestra.run.smithi192.stdout:Setting up ceph (18.2.2-1059-g4d05f4ca-1jammy) ... 2024-04-06T09:28:44.071 INFO:teuthology.orchestra.run.smithi192.stdout:Setting up ceph-mgr-dashboard (18.2.2-1059-g4d05f4ca-1jammy) ... 2024-04-06T09:28:44.193 INFO:teuthology.orchestra.run.smithi006.stdout:Setting up ceph-base (18.2.2-1059-g4d05f4ca-1jammy) ... 2024-04-06T09:28:44.197 INFO:teuthology.orchestra.run.smithi192.stdout:Setting up ceph-volume (18.2.2-1059-g4d05f4ca-1jammy) ... 2024-04-06T09:28:44.524 INFO:teuthology.orchestra.run.smithi192.stdout:Processing triggers for man-db (2.10.2-1) ... 2024-04-06T09:28:44.562 INFO:teuthology.orchestra.run.smithi006.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service -> /lib/systemd/system/ceph-crash.service. 2024-04-06T09:28:45.020 INFO:teuthology.orchestra.run.smithi006.stdout:Setting up ceph-mds (18.2.2-1059-g4d05f4ca-1jammy) ... 2024-04-06T09:28:45.266 INFO:teuthology.orchestra.run.smithi006.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2024-04-06T09:28:45.266 INFO:teuthology.orchestra.run.smithi006.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2024-04-06T09:28:45.299 INFO:teuthology.orchestra.run.smithi192.stdout:Processing triggers for libc-bin (2.35-0ubuntu3.1) ... 2024-04-06T09:28:45.772 INFO:teuthology.orchestra.run.smithi006.stdout:Setting up rbd-mirror (18.2.2-1059-g4d05f4ca-1jammy) ... 2024-04-06T09:28:45.774 INFO:teuthology.orchestra.run.smithi192.stderr:W: --force-yes is deprecated, use one of the options starting with --allow instead. 2024-04-06T09:28:45.784 DEBUG:teuthology.parallel:result is None 2024-04-06T09:28:45.979 INFO:teuthology.orchestra.run.smithi006.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target -> /lib/systemd/system/ceph-rbd-mirror.target. 2024-04-06T09:28:45.979 INFO:teuthology.orchestra.run.smithi006.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target -> /lib/systemd/system/ceph-rbd-mirror.target. 2024-04-06T09:28:46.431 INFO:teuthology.orchestra.run.smithi006.stdout:Setting up ceph-mgr (18.2.2-1059-g4d05f4ca-1jammy) ... 2024-04-06T09:28:46.642 INFO:teuthology.orchestra.run.smithi006.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2024-04-06T09:28:46.642 INFO:teuthology.orchestra.run.smithi006.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2024-04-06T09:28:47.260 INFO:teuthology.orchestra.run.smithi006.stdout:Setting up ceph-osd (18.2.2-1059-g4d05f4ca-1jammy) ... 2024-04-06T09:28:47.552 INFO:teuthology.orchestra.run.smithi006.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2024-04-06T09:28:47.552 INFO:teuthology.orchestra.run.smithi006.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2024-04-06T09:28:48.020 INFO:teuthology.orchestra.run.smithi006.stdout:Setting up ceph-mgr-k8sevents (18.2.2-1059-g4d05f4ca-1jammy) ... 2024-04-06T09:28:48.146 INFO:teuthology.orchestra.run.smithi006.stdout:Setting up ceph-mgr-diskprediction-local (18.2.2-1059-g4d05f4ca-1jammy) ... 2024-04-06T09:28:48.297 INFO:teuthology.orchestra.run.smithi006.stdout:Setting up ceph-mon (18.2.2-1059-g4d05f4ca-1jammy) ... 2024-04-06T09:28:48.502 INFO:teuthology.orchestra.run.smithi006.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2024-04-06T09:28:48.502 INFO:teuthology.orchestra.run.smithi006.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2024-04-06T09:28:48.949 INFO:teuthology.orchestra.run.smithi006.stdout:Setting up ceph-mgr-cephadm (18.2.2-1059-g4d05f4ca-1jammy) ... 2024-04-06T09:28:49.091 INFO:teuthology.orchestra.run.smithi006.stdout:Setting up ceph (18.2.2-1059-g4d05f4ca-1jammy) ... 2024-04-06T09:28:49.434 INFO:teuthology.orchestra.run.smithi006.stdout:Setting up ceph-mgr-dashboard (18.2.2-1059-g4d05f4ca-1jammy) ... 2024-04-06T09:28:49.585 INFO:teuthology.orchestra.run.smithi006.stdout:Setting up ceph-volume (18.2.2-1059-g4d05f4ca-1jammy) ... 2024-04-06T09:28:49.954 INFO:teuthology.orchestra.run.smithi006.stdout:Processing triggers for man-db (2.10.2-1) ... 2024-04-06T09:28:51.080 INFO:teuthology.orchestra.run.smithi006.stdout:Processing triggers for libc-bin (2.35-0ubuntu3.1) ... 2024-04-06T09:28:51.546 INFO:teuthology.orchestra.run.smithi006.stderr:W: --force-yes is deprecated, use one of the options starting with --allow instead. 2024-04-06T09:28:51.551 DEBUG:teuthology.parallel:result is None 2024-04-06T09:28:51.552 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&sha1=4d05f4cad464de0fcdd11a2856643e533aea34e8 2024-04-06T09:28:51.773 DEBUG:teuthology.orchestra.run.smithi006:> dpkg-query -W -f '${Version}' ceph 2024-04-06T09:28:51.793 INFO:teuthology.orchestra.run.smithi006.stdout:18.2.2-1059-g4d05f4ca-1jammy 2024-04-06T09:28:51.793 INFO:teuthology.packaging:The installed version of ceph is 18.2.2-1059-g4d05f4ca-1jammy 2024-04-06T09:28:51.793 INFO:teuthology.task.install:The correct ceph version 18.2.2-1059-g4d05f4ca-1jammy is installed. 2024-04-06T09:28:51.795 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&sha1=4d05f4cad464de0fcdd11a2856643e533aea34e8 2024-04-06T09:28:51.934 DEBUG:teuthology.orchestra.run.smithi192:> dpkg-query -W -f '${Version}' ceph 2024-04-06T09:28:51.953 INFO:teuthology.orchestra.run.smithi192.stdout:18.2.2-1059-g4d05f4ca-1jammy 2024-04-06T09:28:51.954 INFO:teuthology.packaging:The installed version of ceph is 18.2.2-1059-g4d05f4ca-1jammy 2024-04-06T09:28:51.954 INFO:teuthology.task.install:The correct ceph version 18.2.2-1059-g4d05f4ca-1jammy is installed. 2024-04-06T09:28:51.955 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-04-06T09:28:51.956 DEBUG:teuthology.orchestra.run.smithi006:> set -ex 2024-04-06T09:28:51.956 DEBUG:teuthology.orchestra.run.smithi006:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-04-06T09:28:51.971 DEBUG:teuthology.orchestra.run.smithi192:> set -ex 2024-04-06T09:28:51.971 DEBUG:teuthology.orchestra.run.smithi192:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-04-06T09:28:52.012 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-04-06T09:28:52.013 DEBUG:teuthology.orchestra.run.smithi006:> set -ex 2024-04-06T09:28:52.013 DEBUG:teuthology.orchestra.run.smithi006:> sudo dd of=/usr/bin/daemon-helper 2024-04-06T09:28:52.031 DEBUG:teuthology.orchestra.run.smithi006:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-04-06T09:28:52.086 DEBUG:teuthology.orchestra.run.smithi192:> set -ex 2024-04-06T09:28:52.087 DEBUG:teuthology.orchestra.run.smithi192:> sudo dd of=/usr/bin/daemon-helper 2024-04-06T09:28:52.099 DEBUG:teuthology.orchestra.run.smithi192:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-04-06T09:28:52.155 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-04-06T09:28:52.155 DEBUG:teuthology.orchestra.run.smithi006:> set -ex 2024-04-06T09:28:52.155 DEBUG:teuthology.orchestra.run.smithi006:> sudo dd of=/usr/bin/adjust-ulimits 2024-04-06T09:28:52.169 DEBUG:teuthology.orchestra.run.smithi006:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-04-06T09:28:52.222 DEBUG:teuthology.orchestra.run.smithi192:> set -ex 2024-04-06T09:28:52.222 DEBUG:teuthology.orchestra.run.smithi192:> sudo dd of=/usr/bin/adjust-ulimits 2024-04-06T09:28:52.236 DEBUG:teuthology.orchestra.run.smithi192:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-04-06T09:28:52.291 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-04-06T09:28:52.291 DEBUG:teuthology.orchestra.run.smithi006:> set -ex 2024-04-06T09:28:52.292 DEBUG:teuthology.orchestra.run.smithi006:> sudo dd of=/usr/bin/stdin-killer 2024-04-06T09:28:52.305 DEBUG:teuthology.orchestra.run.smithi006:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-04-06T09:28:52.358 DEBUG:teuthology.orchestra.run.smithi192:> set -ex 2024-04-06T09:28:52.358 DEBUG:teuthology.orchestra.run.smithi192:> sudo dd of=/usr/bin/stdin-killer 2024-04-06T09:28:52.370 DEBUG:teuthology.orchestra.run.smithi192:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-04-06T09:28:52.427 INFO:teuthology.run_tasks:Running task ceph... 2024-04-06T09:28:52.490 INFO:tasks.ceph:Making ceph log dir writeable by non-root... 2024-04-06T09:28:52.490 DEBUG:teuthology.orchestra.run.smithi006:> sudo chmod 777 /var/log/ceph 2024-04-06T09:28:52.493 DEBUG:teuthology.orchestra.run.smithi192:> sudo chmod 777 /var/log/ceph 2024-04-06T09:28:52.505 INFO:tasks.ceph:Disabling ceph logrotate... 2024-04-06T09:28:52.505 DEBUG:teuthology.orchestra.run.smithi006:> sudo rm -f -- /etc/logrotate.d/ceph 2024-04-06T09:28:52.548 DEBUG:teuthology.orchestra.run.smithi192:> sudo rm -f -- /etc/logrotate.d/ceph 2024-04-06T09:28:52.560 INFO:tasks.ceph:Creating extra log directories... 2024-04-06T09:28:52.560 DEBUG:teuthology.orchestra.run.smithi006:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-04-06T09:28:52.604 DEBUG:teuthology.orchestra.run.smithi192:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-04-06T09:28:52.629 INFO:tasks.ceph:Creating ceph cluster cluster1... 2024-04-06T09:28:52.629 INFO:tasks.ceph:config {'conf': {'client': {'debug rbd': 20, 'debug rbd_mirror': 15, 'log to stderr': False, 'rbd default features': 125, 'rbd mirror image policy type': 'none'}, 'client.mirror.0': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.1': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.2': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.3': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'global': {'mon client directed command retry': 5, 'mon warn on pool no app': False, 'ms inject socket failures': 5000}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'bluestore block size': 96636764160, 'bluestore compression algorithm': 'snappy', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME'], 'cpu_profile': set(), 'cluster': 'cluster1', 'mon_bind_msgr2': True, 'mon_bind_addrvec': True} 2024-04-06T09:28:52.630 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/teuthology-2024-04-02_22:16:03-rbd-reef-distro-default-smithi/7637502', 'branch': 'reef', 'description': 'rbd/mirror-thrash/{base/install clients/mirror cluster/{2-node openstack} conf/{disable-pool-app} msgr-failures/few objectstore/bluestore-comp-snappy policy/none rbd-mirror/four-per-cluster supported-random-distro$/{ubuntu_latest} workloads/rbd-mirror-journal-stress-workunit}', 'email': 'ceph-qa@ceph.com', 'first_in_suite': False, 'job_id': '7637502', 'kernel': {'kdb': True, 'sha1': 'distro'}, 'ktype': 'distro', 'last_in_suite': False, 'machine_type': 'smithi', 'meta': [{'desc': 'run two ceph clusters and install rbd-mirror'}, {'desc': 'configure the permissions for client.mirror'}, {'desc': '2 ceph clusters with 1 mon and 3 osds each'}, {'desc': 'run four rbd-mirror daemons per cluster'}, {'desc': 'run the rbd_mirror_stress.sh workunit to test the rbd-mirror daemon'}], 'name': 'teuthology-2024-04-02_22:16:03-rbd-reef-distro-default-smithi', 'no_nested_subset': False, 'nuke-on-error': True, 'openstack': [{'volumes': {'count': 3, 'size': 30}}], 'os_type': 'ubuntu', 'os_version': '22.04', 'overrides': {'admin_socket': {'branch': 'reef'}, 'ceph': {'conf': {'client': {'debug rbd': 20, 'debug rbd_mirror': 15, 'log to stderr': False, 'rbd default features': 125, 'rbd mirror image policy type': 'none'}, 'client.mirror.0': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.1': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.2': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.3': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'global': {'mon client directed command retry': 5, 'mon warn on pool no app': False, 'ms inject socket failures': 5000}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'bluestore block size': 96636764160, 'bluestore compression algorithm': 'snappy', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME'], 'sha1': '4d05f4cad464de0fcdd11a2856643e533aea34e8'}, 'ceph-deploy': {'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {}}}, 'install': {'ceph': {'flavor': 'default', 'sha1': '4d05f4cad464de0fcdd11a2856643e533aea34e8'}}, 'thrashosds': {'bdev_inject_crash': 2, 'bdev_inject_crash_probability': 0.5}, 'workunit': {'branch': 'reef', 'sha1': '052adcadd0534eddb0d73c5afbc083a84cd9dc46'}}, 'owner': 'scheduled_teuthology@teuthology', 'priority': 930, 'repo': 'https://git.ceph.com/ceph.git', 'roles': [['cluster1.mon.a', 'cluster1.mgr.x', 'cluster2.mgr.x', 'cluster1.osd.0', 'cluster1.osd.1', 'cluster1.osd.2', 'cluster1.client.0', 'cluster2.client.0'], ['cluster2.mon.a', 'cluster2.osd.0', 'cluster2.osd.1', 'cluster2.osd.2', 'cluster1.client.mirror', 'cluster1.client.mirror.0', 'cluster1.client.mirror.1', 'cluster1.client.mirror.2', 'cluster1.client.mirror.3', 'cluster1.client.mirror.4', 'cluster1.client.mirror.5', 'cluster1.client.mirror.6', 'cluster2.client.mirror', 'cluster2.client.mirror.0', 'cluster2.client.mirror.1', 'cluster2.client.mirror.2', 'cluster2.client.mirror.3', 'cluster2.client.mirror.4', 'cluster2.client.mirror.5', 'cluster2.client.mirror.6']], 'seed': 9400, 'sha1': '4d05f4cad464de0fcdd11a2856643e533aea34e8', 'sleep_before_teardown': 0, 'subset': '12/128', 'suite': 'rbd', 'suite_branch': 'reef', 'suite_path': '/home/teuthworker/src/git.ceph.com_ceph_052adcadd0534eddb0d73c5afbc083a84cd9dc46/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://git.ceph.com/ceph.git', 'suite_sha1': '052adcadd0534eddb0d73c5afbc083a84cd9dc46', 'targets': {'smithi006.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBPRsjjIxbbsh/IkYvxkow7Ob8LWlTfeg943sr/m1r913UrvId+M7tk4dUHrqVGT1fibXJxU9TjyR75QZn7tVsG0=', 'smithi192.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBFwM+L3rHFCaXoKHuY/3HZtsZsr61XOLlFJZtgkQKNRHUQSe/PZxCqw1jcy0QQ7XEb3wk4LbI1nEnCQwkpC2S8M='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'kdb': True, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': {'extra_packages': ['rbd-mirror'], 'flavor': 'default', 'sha1': '4d05f4cad464de0fcdd11a2856643e533aea34e8'}}, {'ceph': {'cluster': 'cluster1', 'conf': {'client': {'debug rbd': 20, 'debug rbd_mirror': 15, 'log to stderr': False, 'rbd default features': 125, 'rbd mirror image policy type': 'none'}, 'client.mirror.0': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.1': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.2': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.3': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'global': {'mon client directed command retry': 5, 'mon warn on pool no app': False, 'ms inject socket failures': 5000}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'bluestore block size': 96636764160, 'bluestore compression algorithm': 'snappy', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME'], 'sha1': '4d05f4cad464de0fcdd11a2856643e533aea34e8'}}, {'ceph': {'cluster': 'cluster2'}}, {'exec': {'cluster1.client.mirror.0': ["sudo ceph --cluster cluster1 auth caps client.mirror mon 'profile rbd-mirror-peer' osd 'profile rbd'", "sudo ceph --cluster cluster1 auth caps client.mirror.0 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster1 auth caps client.mirror.1 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster1 auth caps client.mirror.2 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster1 auth caps client.mirror.3 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster2 auth caps client.mirror mon 'profile rbd-mirror-peer' osd 'profile rbd'", "sudo ceph --cluster cluster2 auth caps client.mirror.0 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster2 auth caps client.mirror.1 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster2 auth caps client.mirror.2 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster2 auth caps client.mirror.3 mon 'profile rbd-mirror' osd 'profile rbd'"]}}, {'rbd-mirror': {'client': 'cluster1.client.mirror.0', 'thrash': True}}, {'rbd-mirror': {'client': 'cluster1.client.mirror.1', 'thrash': True}}, {'rbd-mirror': {'client': 'cluster1.client.mirror.2', 'thrash': True}}, {'rbd-mirror': {'client': 'cluster1.client.mirror.3', 'thrash': True}}, {'rbd-mirror': {'client': 'cluster2.client.mirror.0', 'thrash': True}}, {'rbd-mirror': {'client': 'cluster2.client.mirror.1', 'thrash': True}}, {'rbd-mirror': {'client': 'cluster2.client.mirror.2', 'thrash': True}}, {'rbd-mirror': {'client': 'cluster2.client.mirror.3', 'thrash': True}}, {'rbd-mirror-thrash': {'cluster': 'cluster1'}}, {'rbd-mirror-thrash': {'cluster': 'cluster2'}}, {'workunit': {'clients': {'cluster1.client.mirror': ['rbd/rbd_mirror_stress.sh']}, 'env': {'CEPH_ARGS': '', 'MIRROR_IMAGE_MODE': 'journal', 'MIRROR_POOL_MODE': 'pool', 'RBD_MIRROR_INSTANCES': '4', 'RBD_MIRROR_USE_EXISTING_CLUSTER': '1', 'RBD_MIRROR_USE_RBD_MIRROR': '1'}, 'timeout': '6h'}}], 'teuthology': {'fragments_dropped': [], 'meta': {}, 'postmerge': []}, 'teuthology_branch': 'main', 'teuthology_sha1': 'e691533f9cbb33d85b2187bba20d7102f098636d', 'timestamp': '2024-04-02_22:16:03', 'tube': 'smithi', 'user': 'teuthology', 'verbose': False, 'worker_log': '/home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885'} 2024-04-06T09:28:52.630 DEBUG:teuthology.orchestra.run.smithi006:> install -d -m0755 -- /home/ubuntu/cephtest/cluster1.data 2024-04-06T09:28:52.660 DEBUG:teuthology.orchestra.run.smithi192:> install -d -m0755 -- /home/ubuntu/cephtest/cluster1.data 2024-04-06T09:28:52.665 DEBUG:teuthology.orchestra.run.smithi006:> sudo install -d -m0777 -- /var/run/ceph 2024-04-06T09:28:52.708 DEBUG:teuthology.orchestra.run.smithi192:> sudo install -d -m0777 -- /var/run/ceph 2024-04-06T09:28:52.721 DEBUG:teuthology.orchestra.run.smithi006:> set -ex 2024-04-06T09:28:52.721 DEBUG:teuthology.orchestra.run.smithi006:> dd if=/scratch_devs of=/dev/stdout 2024-04-06T09:28:52.767 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-04-06T09:28:52.767 DEBUG:teuthology.orchestra.run.smithi006:> stat /dev/vg_nvme/lv_1 2024-04-06T09:28:52.815 INFO:teuthology.orchestra.run.smithi006.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-04-06T09:28:52.815 INFO:teuthology.orchestra.run.smithi006.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-06T09:28:52.815 INFO:teuthology.orchestra.run.smithi006.stdout:Device: 5h/5d Inode: 879 Links: 1 2024-04-06T09:28:52.815 INFO:teuthology.orchestra.run.smithi006.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-06T09:28:52.815 INFO:teuthology.orchestra.run.smithi006.stdout:Access: 2024-04-06 09:23:41.790600142 +0000 2024-04-06T09:28:52.815 INFO:teuthology.orchestra.run.smithi006.stdout:Modify: 2024-04-06 09:23:41.530602453 +0000 2024-04-06T09:28:52.815 INFO:teuthology.orchestra.run.smithi006.stdout:Change: 2024-04-06 09:23:41.530602453 +0000 2024-04-06T09:28:52.816 INFO:teuthology.orchestra.run.smithi006.stdout: Birth: - 2024-04-06T09:28:52.816 DEBUG:teuthology.orchestra.run.smithi006:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-04-06T09:28:52.870 INFO:teuthology.orchestra.run.smithi006.stderr:1+0 records in 2024-04-06T09:28:52.870 INFO:teuthology.orchestra.run.smithi006.stderr:1+0 records out 2024-04-06T09:28:52.870 INFO:teuthology.orchestra.run.smithi006.stderr:512 bytes copied, 0.0004642 s, 1.1 MB/s 2024-04-06T09:28:52.871 DEBUG:teuthology.orchestra.run.smithi006:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-04-06T09:28:52.920 DEBUG:teuthology.orchestra.run.smithi006:> stat /dev/vg_nvme/lv_2 2024-04-06T09:28:52.967 INFO:teuthology.orchestra.run.smithi006.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-04-06T09:28:52.967 INFO:teuthology.orchestra.run.smithi006.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-06T09:28:52.967 INFO:teuthology.orchestra.run.smithi006.stdout:Device: 5h/5d Inode: 898 Links: 1 2024-04-06T09:28:52.967 INFO:teuthology.orchestra.run.smithi006.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-06T09:28:52.967 INFO:teuthology.orchestra.run.smithi006.stdout:Access: 2024-04-06 09:23:42.354595128 +0000 2024-04-06T09:28:52.967 INFO:teuthology.orchestra.run.smithi006.stdout:Modify: 2024-04-06 09:23:42.054597795 +0000 2024-04-06T09:28:52.967 INFO:teuthology.orchestra.run.smithi006.stdout:Change: 2024-04-06 09:23:42.054597795 +0000 2024-04-06T09:28:52.967 INFO:teuthology.orchestra.run.smithi006.stdout: Birth: - 2024-04-06T09:28:52.968 DEBUG:teuthology.orchestra.run.smithi006:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-04-06T09:28:53.021 INFO:teuthology.orchestra.run.smithi006.stderr:1+0 records in 2024-04-06T09:28:53.021 INFO:teuthology.orchestra.run.smithi006.stderr:1+0 records out 2024-04-06T09:28:53.022 INFO:teuthology.orchestra.run.smithi006.stderr:512 bytes copied, 0.000366829 s, 1.4 MB/s 2024-04-06T09:28:53.023 DEBUG:teuthology.orchestra.run.smithi006:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-04-06T09:28:53.074 DEBUG:teuthology.orchestra.run.smithi006:> stat /dev/vg_nvme/lv_3 2024-04-06T09:28:53.124 INFO:teuthology.orchestra.run.smithi006.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-04-06T09:28:53.124 INFO:teuthology.orchestra.run.smithi006.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-06T09:28:53.124 INFO:teuthology.orchestra.run.smithi006.stdout:Device: 5h/5d Inode: 917 Links: 1 2024-04-06T09:28:53.124 INFO:teuthology.orchestra.run.smithi006.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-06T09:28:53.124 INFO:teuthology.orchestra.run.smithi006.stdout:Access: 2024-04-06 09:23:42.858590649 +0000 2024-04-06T09:28:53.124 INFO:teuthology.orchestra.run.smithi006.stdout:Modify: 2024-04-06 09:23:42.606592888 +0000 2024-04-06T09:28:53.125 INFO:teuthology.orchestra.run.smithi006.stdout:Change: 2024-04-06 09:23:42.606592888 +0000 2024-04-06T09:28:53.125 INFO:teuthology.orchestra.run.smithi006.stdout: Birth: - 2024-04-06T09:28:53.125 DEBUG:teuthology.orchestra.run.smithi006:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-04-06T09:28:53.178 INFO:teuthology.orchestra.run.smithi006.stderr:1+0 records in 2024-04-06T09:28:53.178 INFO:teuthology.orchestra.run.smithi006.stderr:1+0 records out 2024-04-06T09:28:53.178 INFO:teuthology.orchestra.run.smithi006.stderr:512 bytes copied, 0.00041014 s, 1.2 MB/s 2024-04-06T09:28:53.180 DEBUG:teuthology.orchestra.run.smithi006:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-04-06T09:28:53.228 DEBUG:teuthology.orchestra.run.smithi006:> stat /dev/vg_nvme/lv_4 2024-04-06T09:28:53.275 INFO:teuthology.orchestra.run.smithi006.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-04-06T09:28:53.275 INFO:teuthology.orchestra.run.smithi006.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-06T09:28:53.275 INFO:teuthology.orchestra.run.smithi006.stdout:Device: 5h/5d Inode: 930 Links: 1 2024-04-06T09:28:53.275 INFO:teuthology.orchestra.run.smithi006.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-06T09:28:53.275 INFO:teuthology.orchestra.run.smithi006.stdout:Access: 2024-04-06 09:23:43.362586168 +0000 2024-04-06T09:28:53.275 INFO:teuthology.orchestra.run.smithi006.stdout:Modify: 2024-04-06 09:23:43.114588372 +0000 2024-04-06T09:28:53.275 INFO:teuthology.orchestra.run.smithi006.stdout:Change: 2024-04-06 09:23:43.114588372 +0000 2024-04-06T09:28:53.275 INFO:teuthology.orchestra.run.smithi006.stdout: Birth: - 2024-04-06T09:28:53.275 DEBUG:teuthology.orchestra.run.smithi006:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-04-06T09:28:53.329 INFO:teuthology.orchestra.run.smithi006.stderr:1+0 records in 2024-04-06T09:28:53.329 INFO:teuthology.orchestra.run.smithi006.stderr:1+0 records out 2024-04-06T09:28:53.329 INFO:teuthology.orchestra.run.smithi006.stderr:512 bytes copied, 0.000385744 s, 1.3 MB/s 2024-04-06T09:28:53.331 DEBUG:teuthology.orchestra.run.smithi006:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-04-06T09:28:53.380 INFO:tasks.ceph:osd dev map: {'cluster1.osd.0': '/dev/vg_nvme/lv_1', 'cluster1.osd.1': '/dev/vg_nvme/lv_2', 'cluster1.osd.2': '/dev/vg_nvme/lv_3'} 2024-04-06T09:28:53.380 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi006.front.sepia.ceph.com'): {'cluster1.osd.0': '/dev/vg_nvme/lv_1', 'cluster1.osd.1': '/dev/vg_nvme/lv_2', 'cluster1.osd.2': '/dev/vg_nvme/lv_3'}} 2024-04-06T09:28:53.380 INFO:tasks.ceph:Generating config... 2024-04-06T09:28:53.382 INFO:tasks.ceph:[client] debug rbd = 20 2024-04-06T09:28:53.382 INFO:tasks.ceph:[client] debug rbd_mirror = 15 2024-04-06T09:28:53.382 INFO:tasks.ceph:[client] log to stderr = False 2024-04-06T09:28:53.382 INFO:tasks.ceph:[client] rbd default features = 125 2024-04-06T09:28:53.382 INFO:tasks.ceph:[client] rbd mirror image policy type = none 2024-04-06T09:28:53.382 INFO:tasks.ceph:[client.mirror.0] admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-04-06T09:28:53.382 INFO:tasks.ceph:[client.mirror.0] pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-04-06T09:28:53.382 INFO:tasks.ceph:[client.mirror.1] admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-04-06T09:28:53.382 INFO:tasks.ceph:[client.mirror.1] pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-04-06T09:28:53.382 INFO:tasks.ceph:[client.mirror.2] admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-04-06T09:28:53.382 INFO:tasks.ceph:[client.mirror.2] pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-04-06T09:28:53.382 INFO:tasks.ceph:[client.mirror.3] admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-04-06T09:28:53.382 INFO:tasks.ceph:[client.mirror.3] pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-04-06T09:28:53.383 INFO:tasks.ceph:[global] mon client directed command retry = 5 2024-04-06T09:28:53.383 INFO:tasks.ceph:[global] mon warn on pool no app = False 2024-04-06T09:28:53.383 INFO:tasks.ceph:[global] ms inject socket failures = 5000 2024-04-06T09:28:53.383 INFO:tasks.ceph:[mgr] debug mgr = 20 2024-04-06T09:28:53.383 INFO:tasks.ceph:[mgr] debug ms = 1 2024-04-06T09:28:53.383 INFO:tasks.ceph:[mon] debug mon = 20 2024-04-06T09:28:53.383 INFO:tasks.ceph:[mon] debug ms = 1 2024-04-06T09:28:53.383 INFO:tasks.ceph:[mon] debug paxos = 20 2024-04-06T09:28:53.383 INFO:tasks.ceph:[osd] bluestore block size = 96636764160 2024-04-06T09:28:53.383 INFO:tasks.ceph:[osd] bluestore compression algorithm = snappy 2024-04-06T09:28:53.383 INFO:tasks.ceph:[osd] bluestore compression mode = aggressive 2024-04-06T09:28:53.383 INFO:tasks.ceph:[osd] bluestore fsck on mount = True 2024-04-06T09:28:53.383 INFO:tasks.ceph:[osd] debug bluefs = 1/20 2024-04-06T09:28:53.384 INFO:tasks.ceph:[osd] debug bluestore = 1/20 2024-04-06T09:28:53.384 INFO:tasks.ceph:[osd] debug ms = 1 2024-04-06T09:28:53.384 INFO:tasks.ceph:[osd] debug osd = 20 2024-04-06T09:28:53.384 INFO:tasks.ceph:[osd] debug rocksdb = 4/10 2024-04-06T09:28:53.384 INFO:tasks.ceph:[osd] mon osd backfillfull_ratio = 0.85 2024-04-06T09:28:53.384 INFO:tasks.ceph:[osd] mon osd full ratio = 0.9 2024-04-06T09:28:53.384 INFO:tasks.ceph:[osd] mon osd nearfull ratio = 0.8 2024-04-06T09:28:53.384 INFO:tasks.ceph:[osd] osd failsafe full ratio = 0.95 2024-04-06T09:28:53.384 INFO:tasks.ceph:[osd] osd objectstore = bluestore 2024-04-06T09:28:53.384 INFO:tasks.ceph:Setting up cluster1.mon.a... 2024-04-06T09:28:53.384 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/cluster1.keyring 2024-04-06T09:28:53.455 INFO:teuthology.orchestra.run.smithi006.stdout:creating /etc/ceph/cluster1.keyring 2024-04-06T09:28:53.462 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/cluster1.keyring 2024-04-06T09:28:53.543 DEBUG:teuthology.orchestra.run.smithi006:> sudo chmod 0644 /etc/ceph/cluster1.keyring 2024-04-06T09:28:53.597 DEBUG:tasks.ceph:Ceph mon addresses: [('cluster1.mon.a', '172.21.15.6')] 2024-04-06T09:28:53.598 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'cephx', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '172.21.15.6', 'mon client directed command retry': 5, 'mon warn on pool no app': False, 'ms inject socket failures': 5000}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': '600', 'osd mclock profile': 'high_recovery_ops', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'bluestore block size': 96636764160, 'bluestore compression algorithm': 'snappy', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false'}, 'mon': {'debug ms': 1, 'debug mon': 20, 'debug paxos': 20, 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false', 'mon_down_mkfs_grace': '2m', 'mon_warn_on_filestore_osds': 'false'}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok', 'debug rbd': 20, 'debug rbd_mirror': 15, 'log to stderr': False, 'rbd default features': 125, 'rbd mirror image policy type': 'none'}, 'mon.a': {}, 'client.mirror.0': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.1': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.2': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.3': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}} 2024-04-06T09:28:53.598 DEBUG:teuthology.orchestra.run.smithi006:> set -ex 2024-04-06T09:28:53.598 DEBUG:teuthology.orchestra.run.smithi006:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2024-04-06T09:28:53.643 DEBUG:teuthology.orchestra.run.smithi006:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --add a 172.21.15.6 --print /home/ubuntu/cephtest/cluster1.monmap 2024-04-06T09:28:53.716 INFO:teuthology.orchestra.run.smithi006.stdout:monmaptool: monmap file /home/ubuntu/cephtest/cluster1.monmap 2024-04-06T09:28:53.716 INFO:teuthology.orchestra.run.smithi006.stdout:monmaptool: generated fsid 06a46034-4ef3-4e36-9fc5-05c8000433c6 2024-04-06T09:28:53.716 INFO:teuthology.orchestra.run.smithi006.stdout:setting min_mon_release = pacific 2024-04-06T09:28:53.716 INFO:teuthology.orchestra.run.smithi006.stdout:epoch 0 2024-04-06T09:28:53.716 INFO:teuthology.orchestra.run.smithi006.stdout:fsid 06a46034-4ef3-4e36-9fc5-05c8000433c6 2024-04-06T09:28:53.716 INFO:teuthology.orchestra.run.smithi006.stdout:last_changed 2024-04-06T09:28:53.717395+0000 2024-04-06T09:28:53.716 INFO:teuthology.orchestra.run.smithi006.stdout:created 2024-04-06T09:28:53.717395+0000 2024-04-06T09:28:53.717 INFO:teuthology.orchestra.run.smithi006.stdout:min_mon_release 16 (pacific) 2024-04-06T09:28:53.717 INFO:teuthology.orchestra.run.smithi006.stdout:election_strategy: 1 2024-04-06T09:28:53.717 INFO:teuthology.orchestra.run.smithi006.stdout:0: [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] mon.a 2024-04-06T09:28:53.717 INFO:teuthology.orchestra.run.smithi006.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/cluster1.monmap (1 monitors) 2024-04-06T09:28:53.718 DEBUG:teuthology.orchestra.run.smithi006:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2024-04-06T09:28:53.767 INFO:tasks.ceph:Writing /etc/ceph/cluster1.conf for FSID 06a46034-4ef3-4e36-9fc5-05c8000433c6... 2024-04-06T09:28:53.768 DEBUG:teuthology.orchestra.run.smithi006:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/cluster1.conf && sudo chmod 0644 /etc/ceph/cluster1.conf > /dev/null 2024-04-06T09:28:53.811 DEBUG:teuthology.orchestra.run.smithi192:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/cluster1.conf && sudo chmod 0644 /etc/ceph/cluster1.conf > /dev/null 2024-04-06T09:28:53.838 INFO:teuthology.orchestra.run.smithi006.stdout:[global] 2024-04-06T09:28:53.839 INFO:teuthology.orchestra.run.smithi006.stdout: chdir = "" 2024-04-06T09:28:53.839 INFO:teuthology.orchestra.run.smithi006.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-04-06T09:28:53.839 INFO:teuthology.orchestra.run.smithi006.stdout: auth supported = cephx 2024-04-06T09:28:53.839 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-04-06T09:28:53.839 INFO:teuthology.orchestra.run.smithi006.stdout: filestore xattr use omap = true 2024-04-06T09:28:53.839 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-04-06T09:28:53.839 INFO:teuthology.orchestra.run.smithi006.stdout: mon clock drift allowed = 1.000 2024-04-06T09:28:53.839 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-04-06T09:28:53.839 INFO:teuthology.orchestra.run.smithi006.stdout: osd crush chooseleaf type = 0 2024-04-06T09:28:53.839 INFO:teuthology.orchestra.run.smithi006.stdout: auth debug = true 2024-04-06T09:28:53.839 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-04-06T09:28:53.839 INFO:teuthology.orchestra.run.smithi006.stdout: ms die on old message = true 2024-04-06T09:28:53.839 INFO:teuthology.orchestra.run.smithi006.stdout: ms die on bug = true 2024-04-06T09:28:53.840 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-04-06T09:28:53.840 INFO:teuthology.orchestra.run.smithi006.stdout: mon max pg per osd = 10000 # >= luminous 2024-04-06T09:28:53.840 INFO:teuthology.orchestra.run.smithi006.stdout: mon pg warn max object skew = 0 2024-04-06T09:28:53.840 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-04-06T09:28:53.840 INFO:teuthology.orchestra.run.smithi006.stdout: # disable pg_autoscaler by default for new pools 2024-04-06T09:28:53.840 INFO:teuthology.orchestra.run.smithi006.stdout: osd_pool_default_pg_autoscale_mode = off 2024-04-06T09:28:53.840 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-04-06T09:28:53.840 INFO:teuthology.orchestra.run.smithi006.stdout: osd pool default size = 2 2024-04-06T09:28:53.840 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-04-06T09:28:53.840 INFO:teuthology.orchestra.run.smithi006.stdout: mon osd allow primary affinity = true 2024-04-06T09:28:53.840 INFO:teuthology.orchestra.run.smithi006.stdout: mon osd allow pg remap = true 2024-04-06T09:28:53.840 INFO:teuthology.orchestra.run.smithi006.stdout: mon warn on legacy crush tunables = false 2024-04-06T09:28:53.840 INFO:teuthology.orchestra.run.smithi006.stdout: mon warn on crush straw calc version zero = false 2024-04-06T09:28:53.840 INFO:teuthology.orchestra.run.smithi006.stdout: mon warn on no sortbitwise = false 2024-04-06T09:28:53.841 INFO:teuthology.orchestra.run.smithi006.stdout: mon warn on osd down out interval zero = false 2024-04-06T09:28:53.841 INFO:teuthology.orchestra.run.smithi006.stdout: mon warn on too few osds = false 2024-04-06T09:28:53.841 INFO:teuthology.orchestra.run.smithi006.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-04-06T09:28:53.841 INFO:teuthology.orchestra.run.smithi006.stdout: mon_warn_on_pool_no_redundancy = false 2024-04-06T09:28:53.841 INFO:teuthology.orchestra.run.smithi006.stdout: mon_allow_pool_size_one = true 2024-04-06T09:28:53.841 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-04-06T09:28:53.841 INFO:teuthology.orchestra.run.smithi006.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-04-06T09:28:53.841 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-04-06T09:28:53.841 INFO:teuthology.orchestra.run.smithi006.stdout: osd default data pool replay window = 5 2024-04-06T09:28:53.841 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-04-06T09:28:53.841 INFO:teuthology.orchestra.run.smithi006.stdout: mon allow pool delete = true 2024-04-06T09:28:53.841 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-04-06T09:28:53.841 INFO:teuthology.orchestra.run.smithi006.stdout: mon cluster log file level = debug 2024-04-06T09:28:53.841 INFO:teuthology.orchestra.run.smithi006.stdout: debug asserts on shutdown = true 2024-04-06T09:28:53.842 INFO:teuthology.orchestra.run.smithi006.stdout: mon health detail to clog = false 2024-04-06T09:28:53.842 INFO:teuthology.orchestra.run.smithi006.stdout: mon host = 172.21.15.6 2024-04-06T09:28:53.842 INFO:teuthology.orchestra.run.smithi006.stdout: mon client directed command retry = 5 2024-04-06T09:28:53.842 INFO:teuthology.orchestra.run.smithi006.stdout: mon warn on pool no app = False 2024-04-06T09:28:53.842 INFO:teuthology.orchestra.run.smithi006.stdout: ms inject socket failures = 5000 2024-04-06T09:28:53.842 INFO:teuthology.orchestra.run.smithi006.stdout: fsid = 06a46034-4ef3-4e36-9fc5-05c8000433c6 2024-04-06T09:28:53.842 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-04-06T09:28:53.842 INFO:teuthology.orchestra.run.smithi006.stdout:[osd] 2024-04-06T09:28:53.842 INFO:teuthology.orchestra.run.smithi006.stdout: osd journal size = 100 2024-04-06T09:28:53.842 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-04-06T09:28:53.842 INFO:teuthology.orchestra.run.smithi006.stdout: osd scrub load threshold = 5.0 2024-04-06T09:28:53.842 INFO:teuthology.orchestra.run.smithi006.stdout: osd scrub max interval = 600 2024-04-06T09:28:53.842 INFO:teuthology.orchestra.run.smithi006.stdout: osd mclock profile = high_recovery_ops 2024-04-06T09:28:53.842 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-04-06T09:28:53.842 INFO:teuthology.orchestra.run.smithi006.stdout: osd recover clone overlap = true 2024-04-06T09:28:53.843 INFO:teuthology.orchestra.run.smithi006.stdout: osd recovery max chunk = 1048576 2024-04-06T09:28:53.843 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-04-06T09:28:53.843 INFO:teuthology.orchestra.run.smithi006.stdout: osd debug shutdown = true 2024-04-06T09:28:53.843 INFO:teuthology.orchestra.run.smithi006.stdout: osd debug op order = true 2024-04-06T09:28:53.843 INFO:teuthology.orchestra.run.smithi006.stdout: osd debug verify stray on activate = true 2024-04-06T09:28:53.843 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-04-06T09:28:53.843 INFO:teuthology.orchestra.run.smithi006.stdout: osd open classes on start = true 2024-04-06T09:28:53.843 INFO:teuthology.orchestra.run.smithi006.stdout: osd debug pg log writeout = true 2024-04-06T09:28:53.843 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-04-06T09:28:53.843 INFO:teuthology.orchestra.run.smithi006.stdout: osd deep scrub update digest min age = 30 2024-04-06T09:28:53.843 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-04-06T09:28:53.843 INFO:teuthology.orchestra.run.smithi006.stdout: osd map max advance = 10 2024-04-06T09:28:53.843 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-04-06T09:28:53.843 INFO:teuthology.orchestra.run.smithi006.stdout: journal zero on create = true 2024-04-06T09:28:53.844 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-04-06T09:28:53.844 INFO:teuthology.orchestra.run.smithi006.stdout: filestore ondisk finisher threads = 3 2024-04-06T09:28:53.844 INFO:teuthology.orchestra.run.smithi006.stdout: filestore apply finisher threads = 3 2024-04-06T09:28:53.844 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-04-06T09:28:53.844 INFO:teuthology.orchestra.run.smithi006.stdout: bdev debug aio = true 2024-04-06T09:28:53.844 INFO:teuthology.orchestra.run.smithi006.stdout: osd debug misdirected ops = true 2024-04-06T09:28:53.844 INFO:teuthology.orchestra.run.smithi006.stdout: bluestore block size = 96636764160 2024-04-06T09:28:53.844 INFO:teuthology.orchestra.run.smithi006.stdout: bluestore compression algorithm = snappy 2024-04-06T09:28:53.844 INFO:teuthology.orchestra.run.smithi006.stdout: bluestore compression mode = aggressive 2024-04-06T09:28:53.844 INFO:teuthology.orchestra.run.smithi006.stdout: bluestore fsck on mount = True 2024-04-06T09:28:53.844 INFO:teuthology.orchestra.run.smithi006.stdout: debug bluefs = 1/20 2024-04-06T09:28:53.844 INFO:teuthology.orchestra.run.smithi006.stdout: debug bluestore = 1/20 2024-04-06T09:28:53.844 INFO:teuthology.orchestra.run.smithi006.stdout: debug ms = 1 2024-04-06T09:28:53.844 INFO:teuthology.orchestra.run.smithi006.stdout: debug osd = 20 2024-04-06T09:28:53.844 INFO:teuthology.orchestra.run.smithi006.stdout: debug rocksdb = 4/10 2024-04-06T09:28:53.845 INFO:teuthology.orchestra.run.smithi006.stdout: mon osd backfillfull_ratio = 0.85 2024-04-06T09:28:53.845 INFO:teuthology.orchestra.run.smithi006.stdout: mon osd full ratio = 0.9 2024-04-06T09:28:53.845 INFO:teuthology.orchestra.run.smithi006.stdout: mon osd nearfull ratio = 0.8 2024-04-06T09:28:53.845 INFO:teuthology.orchestra.run.smithi006.stdout: osd failsafe full ratio = 0.95 2024-04-06T09:28:53.845 INFO:teuthology.orchestra.run.smithi006.stdout: osd objectstore = bluestore 2024-04-06T09:28:53.845 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-04-06T09:28:53.845 INFO:teuthology.orchestra.run.smithi006.stdout:[mgr] 2024-04-06T09:28:53.845 INFO:teuthology.orchestra.run.smithi006.stdout: debug ms = 1 2024-04-06T09:28:53.845 INFO:teuthology.orchestra.run.smithi006.stdout: debug mgr = 20 2024-04-06T09:28:53.845 INFO:teuthology.orchestra.run.smithi006.stdout: debug mon = 20 2024-04-06T09:28:53.845 INFO:teuthology.orchestra.run.smithi006.stdout: debug auth = 20 2024-04-06T09:28:53.845 INFO:teuthology.orchestra.run.smithi006.stdout: mon reweight min pgs per osd = 4 2024-04-06T09:28:53.845 INFO:teuthology.orchestra.run.smithi006.stdout: mon reweight min bytes per osd = 10 2024-04-06T09:28:53.845 INFO:teuthology.orchestra.run.smithi006.stdout: mgr/telemetry/nag = false 2024-04-06T09:28:53.845 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-04-06T09:28:53.846 INFO:teuthology.orchestra.run.smithi006.stdout:[mon] 2024-04-06T09:28:53.846 INFO:teuthology.orchestra.run.smithi006.stdout: debug ms = 1 2024-04-06T09:28:53.846 INFO:teuthology.orchestra.run.smithi006.stdout: debug mon = 20 2024-04-06T09:28:53.846 INFO:teuthology.orchestra.run.smithi006.stdout: debug paxos = 20 2024-04-06T09:28:53.846 INFO:teuthology.orchestra.run.smithi006.stdout: debug auth = 20 2024-04-06T09:28:53.846 INFO:teuthology.orchestra.run.smithi006.stdout: mon data avail warn = 5 2024-04-06T09:28:53.846 INFO:teuthology.orchestra.run.smithi006.stdout: mon mgr mkfs grace = 240 2024-04-06T09:28:53.846 INFO:teuthology.orchestra.run.smithi006.stdout: mon reweight min pgs per osd = 4 2024-04-06T09:28:53.846 INFO:teuthology.orchestra.run.smithi006.stdout: mon osd reporter subtree level = osd 2024-04-06T09:28:53.846 INFO:teuthology.orchestra.run.smithi006.stdout: mon osd prime pg temp = true 2024-04-06T09:28:53.846 INFO:teuthology.orchestra.run.smithi006.stdout: mon reweight min bytes per osd = 10 2024-04-06T09:28:53.846 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-04-06T09:28:53.846 INFO:teuthology.orchestra.run.smithi006.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-04-06T09:28:53.846 INFO:teuthology.orchestra.run.smithi006.stdout: auth mon ticket ttl = 660 # 11m 2024-04-06T09:28:53.847 INFO:teuthology.orchestra.run.smithi006.stdout: auth service ticket ttl = 240 # 4m 2024-04-06T09:28:53.847 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-04-06T09:28:53.847 INFO:teuthology.orchestra.run.smithi006.stdout: # don't complain about insecure global_id in the test suite 2024-04-06T09:28:53.847 INFO:teuthology.orchestra.run.smithi006.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-04-06T09:28:53.847 INFO:teuthology.orchestra.run.smithi006.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-04-06T09:28:53.847 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-04-06T09:28:53.847 INFO:teuthology.orchestra.run.smithi006.stdout: # 1m isn't quite enough 2024-04-06T09:28:53.847 INFO:teuthology.orchestra.run.smithi006.stdout: mon_down_mkfs_grace = 2m 2024-04-06T09:28:53.847 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-04-06T09:28:53.847 INFO:teuthology.orchestra.run.smithi006.stdout: mon_warn_on_filestore_osds = false 2024-04-06T09:28:53.847 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-04-06T09:28:53.847 INFO:teuthology.orchestra.run.smithi006.stdout:[client] 2024-04-06T09:28:53.847 INFO:teuthology.orchestra.run.smithi006.stdout: rgw cache enabled = true 2024-04-06T09:28:53.847 INFO:teuthology.orchestra.run.smithi006.stdout: rgw enable ops log = true 2024-04-06T09:28:53.847 INFO:teuthology.orchestra.run.smithi006.stdout: rgw enable usage log = true 2024-04-06T09:28:53.847 INFO:teuthology.orchestra.run.smithi006.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-04-06T09:28:53.848 INFO:teuthology.orchestra.run.smithi006.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-04-06T09:28:53.848 INFO:teuthology.orchestra.run.smithi006.stdout: debug rbd = 20 2024-04-06T09:28:53.848 INFO:teuthology.orchestra.run.smithi006.stdout: debug rbd_mirror = 15 2024-04-06T09:28:53.848 INFO:teuthology.orchestra.run.smithi006.stdout: log to stderr = False 2024-04-06T09:28:53.848 INFO:teuthology.orchestra.run.smithi006.stdout: rbd default features = 125 2024-04-06T09:28:53.848 INFO:teuthology.orchestra.run.smithi006.stdout: rbd mirror image policy type = none 2024-04-06T09:28:53.848 INFO:teuthology.orchestra.run.smithi006.stdout:[mon.a] 2024-04-06T09:28:53.848 INFO:teuthology.orchestra.run.smithi006.stdout:[client.mirror.0] 2024-04-06T09:28:53.848 INFO:teuthology.orchestra.run.smithi006.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-04-06T09:28:53.848 INFO:teuthology.orchestra.run.smithi006.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-04-06T09:28:53.848 INFO:teuthology.orchestra.run.smithi006.stdout:[client.mirror.1] 2024-04-06T09:28:53.848 INFO:teuthology.orchestra.run.smithi006.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-04-06T09:28:53.848 INFO:teuthology.orchestra.run.smithi006.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-04-06T09:28:53.848 INFO:teuthology.orchestra.run.smithi006.stdout:[client.mirror.2] 2024-04-06T09:28:53.848 INFO:teuthology.orchestra.run.smithi006.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-04-06T09:28:53.849 INFO:teuthology.orchestra.run.smithi006.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-04-06T09:28:53.849 INFO:teuthology.orchestra.run.smithi006.stdout:[client.mirror.3] 2024-04-06T09:28:53.849 INFO:teuthology.orchestra.run.smithi006.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-04-06T09:28:53.849 INFO:teuthology.orchestra.run.smithi006.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-04-06T09:28:53.849 INFO:teuthology.orchestra.run.smithi192.stdout:[global] 2024-04-06T09:28:53.849 INFO:teuthology.orchestra.run.smithi192.stdout: chdir = "" 2024-04-06T09:28:53.849 INFO:teuthology.orchestra.run.smithi192.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-04-06T09:28:53.849 INFO:teuthology.orchestra.run.smithi192.stdout: auth supported = cephx 2024-04-06T09:28:53.849 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-04-06T09:28:53.850 INFO:teuthology.orchestra.run.smithi192.stdout: filestore xattr use omap = true 2024-04-06T09:28:53.850 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-04-06T09:28:53.850 INFO:teuthology.orchestra.run.smithi192.stdout: mon clock drift allowed = 1.000 2024-04-06T09:28:53.850 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-04-06T09:28:53.850 INFO:teuthology.orchestra.run.smithi192.stdout: osd crush chooseleaf type = 0 2024-04-06T09:28:53.850 INFO:teuthology.orchestra.run.smithi192.stdout: auth debug = true 2024-04-06T09:28:53.850 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-04-06T09:28:53.850 INFO:teuthology.orchestra.run.smithi192.stdout: ms die on old message = true 2024-04-06T09:28:53.850 INFO:teuthology.orchestra.run.smithi192.stdout: ms die on bug = true 2024-04-06T09:28:53.850 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-04-06T09:28:53.850 INFO:teuthology.orchestra.run.smithi192.stdout: mon max pg per osd = 10000 # >= luminous 2024-04-06T09:28:53.850 INFO:teuthology.orchestra.run.smithi192.stdout: mon pg warn max object skew = 0 2024-04-06T09:28:53.850 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-04-06T09:28:53.850 INFO:teuthology.orchestra.run.smithi192.stdout: # disable pg_autoscaler by default for new pools 2024-04-06T09:28:53.851 INFO:teuthology.orchestra.run.smithi192.stdout: osd_pool_default_pg_autoscale_mode = off 2024-04-06T09:28:53.851 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-04-06T09:28:53.851 INFO:teuthology.orchestra.run.smithi192.stdout: osd pool default size = 2 2024-04-06T09:28:53.851 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-04-06T09:28:53.851 INFO:teuthology.orchestra.run.smithi192.stdout: mon osd allow primary affinity = true 2024-04-06T09:28:53.851 INFO:teuthology.orchestra.run.smithi192.stdout: mon osd allow pg remap = true 2024-04-06T09:28:53.851 INFO:teuthology.orchestra.run.smithi192.stdout: mon warn on legacy crush tunables = false 2024-04-06T09:28:53.851 INFO:teuthology.orchestra.run.smithi192.stdout: mon warn on crush straw calc version zero = false 2024-04-06T09:28:53.851 INFO:teuthology.orchestra.run.smithi192.stdout: mon warn on no sortbitwise = false 2024-04-06T09:28:53.851 INFO:teuthology.orchestra.run.smithi192.stdout: mon warn on osd down out interval zero = false 2024-04-06T09:28:53.851 INFO:teuthology.orchestra.run.smithi192.stdout: mon warn on too few osds = false 2024-04-06T09:28:53.851 INFO:teuthology.orchestra.run.smithi192.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-04-06T09:28:53.851 INFO:teuthology.orchestra.run.smithi192.stdout: mon_warn_on_pool_no_redundancy = false 2024-04-06T09:28:53.851 INFO:teuthology.orchestra.run.smithi192.stdout: mon_allow_pool_size_one = true 2024-04-06T09:28:53.852 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-04-06T09:28:53.852 INFO:teuthology.orchestra.run.smithi192.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-04-06T09:28:53.852 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-04-06T09:28:53.852 INFO:teuthology.orchestra.run.smithi192.stdout: osd default data pool replay window = 5 2024-04-06T09:28:53.852 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-04-06T09:28:53.852 INFO:teuthology.orchestra.run.smithi192.stdout: mon allow pool delete = true 2024-04-06T09:28:53.852 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-04-06T09:28:53.852 INFO:teuthology.orchestra.run.smithi192.stdout: mon cluster log file level = debug 2024-04-06T09:28:53.852 INFO:teuthology.orchestra.run.smithi192.stdout: debug asserts on shutdown = true 2024-04-06T09:28:53.852 INFO:teuthology.orchestra.run.smithi192.stdout: mon health detail to clog = false 2024-04-06T09:28:53.852 INFO:teuthology.orchestra.run.smithi192.stdout: mon host = 172.21.15.6 2024-04-06T09:28:53.852 INFO:teuthology.orchestra.run.smithi192.stdout: mon client directed command retry = 5 2024-04-06T09:28:53.852 INFO:teuthology.orchestra.run.smithi192.stdout: mon warn on pool no app = False 2024-04-06T09:28:53.852 INFO:teuthology.orchestra.run.smithi192.stdout: ms inject socket failures = 5000 2024-04-06T09:28:53.852 INFO:teuthology.orchestra.run.smithi192.stdout: fsid = 06a46034-4ef3-4e36-9fc5-05c8000433c6 2024-04-06T09:28:53.853 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-04-06T09:28:53.853 INFO:teuthology.orchestra.run.smithi192.stdout:[osd] 2024-04-06T09:28:53.853 INFO:teuthology.orchestra.run.smithi192.stdout: osd journal size = 100 2024-04-06T09:28:53.853 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-04-06T09:28:53.853 INFO:teuthology.orchestra.run.smithi192.stdout: osd scrub load threshold = 5.0 2024-04-06T09:28:53.853 INFO:teuthology.orchestra.run.smithi192.stdout: osd scrub max interval = 600 2024-04-06T09:28:53.853 INFO:teuthology.orchestra.run.smithi192.stdout: osd mclock profile = high_recovery_ops 2024-04-06T09:28:53.853 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-04-06T09:28:53.853 INFO:teuthology.orchestra.run.smithi192.stdout: osd recover clone overlap = true 2024-04-06T09:28:53.853 INFO:teuthology.orchestra.run.smithi192.stdout: osd recovery max chunk = 1048576 2024-04-06T09:28:53.853 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-04-06T09:28:53.853 INFO:teuthology.orchestra.run.smithi192.stdout: osd debug shutdown = true 2024-04-06T09:28:53.853 INFO:teuthology.orchestra.run.smithi192.stdout: osd debug op order = true 2024-04-06T09:28:53.853 INFO:teuthology.orchestra.run.smithi192.stdout: osd debug verify stray on activate = true 2024-04-06T09:28:53.854 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-04-06T09:28:53.854 INFO:teuthology.orchestra.run.smithi192.stdout: osd open classes on start = true 2024-04-06T09:28:53.854 INFO:teuthology.orchestra.run.smithi192.stdout: osd debug pg log writeout = true 2024-04-06T09:28:53.854 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-04-06T09:28:53.854 INFO:teuthology.orchestra.run.smithi192.stdout: osd deep scrub update digest min age = 30 2024-04-06T09:28:53.854 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-04-06T09:28:53.854 INFO:teuthology.orchestra.run.smithi192.stdout: osd map max advance = 10 2024-04-06T09:28:53.854 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-04-06T09:28:53.854 INFO:teuthology.orchestra.run.smithi192.stdout: journal zero on create = true 2024-04-06T09:28:53.854 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-04-06T09:28:53.854 INFO:teuthology.orchestra.run.smithi192.stdout: filestore ondisk finisher threads = 3 2024-04-06T09:28:53.854 INFO:teuthology.orchestra.run.smithi192.stdout: filestore apply finisher threads = 3 2024-04-06T09:28:53.854 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-04-06T09:28:53.855 INFO:teuthology.orchestra.run.smithi192.stdout: bdev debug aio = true 2024-04-06T09:28:53.855 INFO:teuthology.orchestra.run.smithi192.stdout: osd debug misdirected ops = true 2024-04-06T09:28:53.855 INFO:teuthology.orchestra.run.smithi192.stdout: bluestore block size = 96636764160 2024-04-06T09:28:53.855 INFO:teuthology.orchestra.run.smithi192.stdout: bluestore compression algorithm = snappy 2024-04-06T09:28:53.855 INFO:teuthology.orchestra.run.smithi192.stdout: bluestore compression mode = aggressive 2024-04-06T09:28:53.855 INFO:teuthology.orchestra.run.smithi192.stdout: bluestore fsck on mount = True 2024-04-06T09:28:53.855 INFO:teuthology.orchestra.run.smithi192.stdout: debug bluefs = 1/20 2024-04-06T09:28:53.855 INFO:teuthology.orchestra.run.smithi192.stdout: debug bluestore = 1/20 2024-04-06T09:28:53.855 INFO:teuthology.orchestra.run.smithi192.stdout: debug ms = 1 2024-04-06T09:28:53.855 INFO:teuthology.orchestra.run.smithi192.stdout: debug osd = 20 2024-04-06T09:28:53.855 INFO:teuthology.orchestra.run.smithi192.stdout: debug rocksdb = 4/10 2024-04-06T09:28:53.855 INFO:teuthology.orchestra.run.smithi192.stdout: mon osd backfillfull_ratio = 0.85 2024-04-06T09:28:53.855 INFO:teuthology.orchestra.run.smithi192.stdout: mon osd full ratio = 0.9 2024-04-06T09:28:53.855 INFO:teuthology.orchestra.run.smithi192.stdout: mon osd nearfull ratio = 0.8 2024-04-06T09:28:53.855 INFO:teuthology.orchestra.run.smithi192.stdout: osd failsafe full ratio = 0.95 2024-04-06T09:28:53.856 INFO:teuthology.orchestra.run.smithi192.stdout: osd objectstore = bluestore 2024-04-06T09:28:53.856 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-04-06T09:28:53.856 INFO:teuthology.orchestra.run.smithi192.stdout:[mgr] 2024-04-06T09:28:53.856 INFO:teuthology.orchestra.run.smithi192.stdout: debug ms = 1 2024-04-06T09:28:53.856 INFO:teuthology.orchestra.run.smithi192.stdout: debug mgr = 20 2024-04-06T09:28:53.856 INFO:teuthology.orchestra.run.smithi192.stdout: debug mon = 20 2024-04-06T09:28:53.856 INFO:teuthology.orchestra.run.smithi192.stdout: debug auth = 20 2024-04-06T09:28:53.856 INFO:teuthology.orchestra.run.smithi192.stdout: mon reweight min pgs per osd = 4 2024-04-06T09:28:53.856 INFO:teuthology.orchestra.run.smithi192.stdout: mon reweight min bytes per osd = 10 2024-04-06T09:28:53.856 INFO:teuthology.orchestra.run.smithi192.stdout: mgr/telemetry/nag = false 2024-04-06T09:28:53.856 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-04-06T09:28:53.856 INFO:teuthology.orchestra.run.smithi192.stdout:[mon] 2024-04-06T09:28:53.856 INFO:teuthology.orchestra.run.smithi192.stdout: debug ms = 1 2024-04-06T09:28:53.856 INFO:teuthology.orchestra.run.smithi192.stdout: debug mon = 20 2024-04-06T09:28:53.857 INFO:teuthology.orchestra.run.smithi192.stdout: debug paxos = 20 2024-04-06T09:28:53.857 INFO:teuthology.orchestra.run.smithi192.stdout: debug auth = 20 2024-04-06T09:28:53.857 INFO:teuthology.orchestra.run.smithi192.stdout: mon data avail warn = 5 2024-04-06T09:28:53.857 INFO:teuthology.orchestra.run.smithi192.stdout: mon mgr mkfs grace = 240 2024-04-06T09:28:53.857 INFO:teuthology.orchestra.run.smithi192.stdout: mon reweight min pgs per osd = 4 2024-04-06T09:28:53.857 INFO:teuthology.orchestra.run.smithi192.stdout: mon osd reporter subtree level = osd 2024-04-06T09:28:53.857 INFO:teuthology.orchestra.run.smithi192.stdout: mon osd prime pg temp = true 2024-04-06T09:28:53.857 INFO:teuthology.orchestra.run.smithi192.stdout: mon reweight min bytes per osd = 10 2024-04-06T09:28:53.857 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-04-06T09:28:53.857 INFO:teuthology.orchestra.run.smithi192.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-04-06T09:28:53.857 INFO:teuthology.orchestra.run.smithi192.stdout: auth mon ticket ttl = 660 # 11m 2024-04-06T09:28:53.857 INFO:teuthology.orchestra.run.smithi192.stdout: auth service ticket ttl = 240 # 4m 2024-04-06T09:28:53.857 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-04-06T09:28:53.857 INFO:teuthology.orchestra.run.smithi192.stdout: # don't complain about insecure global_id in the test suite 2024-04-06T09:28:53.858 INFO:teuthology.orchestra.run.smithi192.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-04-06T09:28:53.858 INFO:teuthology.orchestra.run.smithi192.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-04-06T09:28:53.858 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-04-06T09:28:53.858 INFO:teuthology.orchestra.run.smithi192.stdout: # 1m isn't quite enough 2024-04-06T09:28:53.858 INFO:teuthology.orchestra.run.smithi192.stdout: mon_down_mkfs_grace = 2m 2024-04-06T09:28:53.858 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-04-06T09:28:53.858 INFO:teuthology.orchestra.run.smithi192.stdout: mon_warn_on_filestore_osds = false 2024-04-06T09:28:53.858 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-04-06T09:28:53.858 INFO:teuthology.orchestra.run.smithi192.stdout:[client] 2024-04-06T09:28:53.858 INFO:teuthology.orchestra.run.smithi192.stdout: rgw cache enabled = true 2024-04-06T09:28:53.858 INFO:teuthology.orchestra.run.smithi192.stdout: rgw enable ops log = true 2024-04-06T09:28:53.858 INFO:teuthology.orchestra.run.smithi192.stdout: rgw enable usage log = true 2024-04-06T09:28:53.858 INFO:teuthology.orchestra.run.smithi192.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-04-06T09:28:53.858 INFO:teuthology.orchestra.run.smithi192.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-04-06T09:28:53.858 INFO:teuthology.orchestra.run.smithi192.stdout: debug rbd = 20 2024-04-06T09:28:53.859 INFO:teuthology.orchestra.run.smithi192.stdout: debug rbd_mirror = 15 2024-04-06T09:28:53.859 INFO:teuthology.orchestra.run.smithi192.stdout: log to stderr = False 2024-04-06T09:28:53.859 INFO:teuthology.orchestra.run.smithi192.stdout: rbd default features = 125 2024-04-06T09:28:53.859 INFO:teuthology.orchestra.run.smithi192.stdout: rbd mirror image policy type = none 2024-04-06T09:28:53.859 INFO:teuthology.orchestra.run.smithi192.stdout:[mon.a] 2024-04-06T09:28:53.859 INFO:teuthology.orchestra.run.smithi192.stdout:[client.mirror.0] 2024-04-06T09:28:53.859 INFO:teuthology.orchestra.run.smithi192.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-04-06T09:28:53.859 INFO:teuthology.orchestra.run.smithi192.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-04-06T09:28:53.859 INFO:teuthology.orchestra.run.smithi192.stdout:[client.mirror.1] 2024-04-06T09:28:53.859 INFO:teuthology.orchestra.run.smithi192.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-04-06T09:28:53.859 INFO:teuthology.orchestra.run.smithi192.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-04-06T09:28:53.859 INFO:teuthology.orchestra.run.smithi192.stdout:[client.mirror.2] 2024-04-06T09:28:53.859 INFO:teuthology.orchestra.run.smithi192.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-04-06T09:28:53.859 INFO:teuthology.orchestra.run.smithi192.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-04-06T09:28:53.859 INFO:teuthology.orchestra.run.smithi192.stdout:[client.mirror.3] 2024-04-06T09:28:53.860 INFO:teuthology.orchestra.run.smithi192.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-04-06T09:28:53.860 INFO:teuthology.orchestra.run.smithi192.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-04-06T09:28:53.861 INFO:tasks.ceph:Creating admin key on cluster1.mon.a... 2024-04-06T09:28:53.861 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/cluster1.keyring 2024-04-06T09:28:53.951 INFO:tasks.ceph:Copying monmap to all nodes... 2024-04-06T09:28:53.951 DEBUG:teuthology.orchestra.run.smithi006:> set -ex 2024-04-06T09:28:53.951 DEBUG:teuthology.orchestra.run.smithi006:> dd if=/etc/ceph/cluster1.keyring of=/dev/stdout 2024-04-06T09:28:53.956 DEBUG:teuthology.orchestra.run.smithi006:> set -ex 2024-04-06T09:28:53.956 DEBUG:teuthology.orchestra.run.smithi006:> dd if=/home/ubuntu/cephtest/cluster1.monmap of=/dev/stdout 2024-04-06T09:28:54.002 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi006.front.sepia.ceph.com 2024-04-06T09:28:54.003 DEBUG:teuthology.orchestra.run.smithi006:> set -ex 2024-04-06T09:28:54.003 DEBUG:teuthology.orchestra.run.smithi006:> sudo dd of=/etc/ceph/cluster1.keyring 2024-04-06T09:28:54.003 DEBUG:teuthology.orchestra.run.smithi006:> sudo chmod 0644 /etc/ceph/cluster1.keyring 2024-04-06T09:28:54.067 DEBUG:teuthology.orchestra.run.smithi006:> set -ex 2024-04-06T09:28:54.067 DEBUG:teuthology.orchestra.run.smithi006:> dd of=/home/ubuntu/cephtest/cluster1.monmap 2024-04-06T09:28:54.119 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi192.front.sepia.ceph.com 2024-04-06T09:28:54.119 DEBUG:teuthology.orchestra.run.smithi192:> set -ex 2024-04-06T09:28:54.119 DEBUG:teuthology.orchestra.run.smithi192:> sudo dd of=/etc/ceph/cluster1.keyring 2024-04-06T09:28:54.119 DEBUG:teuthology.orchestra.run.smithi192:> sudo chmod 0644 /etc/ceph/cluster1.keyring 2024-04-06T09:28:54.143 DEBUG:teuthology.orchestra.run.smithi192:> set -ex 2024-04-06T09:28:54.143 DEBUG:teuthology.orchestra.run.smithi192:> dd of=/home/ubuntu/cephtest/cluster1.monmap 2024-04-06T09:28:54.192 INFO:tasks.ceph:Setting up mon nodes... 2024-04-06T09:28:54.192 INFO:tasks.ceph:Setting up mgr nodes... 2024-04-06T09:28:54.192 DEBUG:teuthology.orchestra.run.smithi006:> sudo mkdir -p /var/lib/ceph/mgr/cluster1-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/cluster1-x/keyring 2024-04-06T09:28:54.237 INFO:teuthology.orchestra.run.smithi006.stdout:creating /var/lib/ceph/mgr/cluster1-x/keyring 2024-04-06T09:28:54.240 INFO:tasks.ceph:Setting up mds nodes... 2024-04-06T09:28:54.241 INFO:tasks.ceph_client:Setting up client nodes... 2024-04-06T09:28:54.241 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/cluster1.client.0.keyring && sudo chmod 0644 /etc/ceph/cluster1.client.0.keyring 2024-04-06T09:28:54.316 INFO:teuthology.orchestra.run.smithi006.stdout:creating /etc/ceph/cluster1.client.0.keyring 2024-04-06T09:28:54.327 DEBUG:teuthology.orchestra.run.smithi192:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror /etc/ceph/cluster1.client.mirror.keyring && sudo chmod 0644 /etc/ceph/cluster1.client.mirror.keyring 2024-04-06T09:28:54.363 INFO:teuthology.orchestra.run.smithi192.stdout:creating /etc/ceph/cluster1.client.mirror.keyring 2024-04-06T09:28:54.381 DEBUG:teuthology.orchestra.run.smithi192:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror.0 /etc/ceph/cluster1.client.mirror.0.keyring && sudo chmod 0644 /etc/ceph/cluster1.client.mirror.0.keyring 2024-04-06T09:28:54.458 INFO:teuthology.orchestra.run.smithi192.stdout:creating /etc/ceph/cluster1.client.mirror.0.keyring 2024-04-06T09:28:54.470 DEBUG:teuthology.orchestra.run.smithi192:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror.1 /etc/ceph/cluster1.client.mirror.1.keyring && sudo chmod 0644 /etc/ceph/cluster1.client.mirror.1.keyring 2024-04-06T09:28:54.540 INFO:teuthology.orchestra.run.smithi192.stdout:creating /etc/ceph/cluster1.client.mirror.1.keyring 2024-04-06T09:28:54.548 DEBUG:teuthology.orchestra.run.smithi192:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror.2 /etc/ceph/cluster1.client.mirror.2.keyring && sudo chmod 0644 /etc/ceph/cluster1.client.mirror.2.keyring 2024-04-06T09:28:54.620 INFO:teuthology.orchestra.run.smithi192.stdout:creating /etc/ceph/cluster1.client.mirror.2.keyring 2024-04-06T09:28:54.631 DEBUG:teuthology.orchestra.run.smithi192:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror.3 /etc/ceph/cluster1.client.mirror.3.keyring && sudo chmod 0644 /etc/ceph/cluster1.client.mirror.3.keyring 2024-04-06T09:28:54.707 INFO:teuthology.orchestra.run.smithi192.stdout:creating /etc/ceph/cluster1.client.mirror.3.keyring 2024-04-06T09:28:54.717 DEBUG:teuthology.orchestra.run.smithi192:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror.4 /etc/ceph/cluster1.client.mirror.4.keyring && sudo chmod 0644 /etc/ceph/cluster1.client.mirror.4.keyring 2024-04-06T09:28:54.792 INFO:teuthology.orchestra.run.smithi192.stdout:creating /etc/ceph/cluster1.client.mirror.4.keyring 2024-04-06T09:28:54.801 DEBUG:teuthology.orchestra.run.smithi192:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror.5 /etc/ceph/cluster1.client.mirror.5.keyring && sudo chmod 0644 /etc/ceph/cluster1.client.mirror.5.keyring 2024-04-06T09:28:54.873 INFO:teuthology.orchestra.run.smithi192.stdout:creating /etc/ceph/cluster1.client.mirror.5.keyring 2024-04-06T09:28:54.884 DEBUG:teuthology.orchestra.run.smithi192:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror.6 /etc/ceph/cluster1.client.mirror.6.keyring && sudo chmod 0644 /etc/ceph/cluster1.client.mirror.6.keyring 2024-04-06T09:28:54.958 INFO:teuthology.orchestra.run.smithi192.stdout:creating /etc/ceph/cluster1.client.mirror.6.keyring 2024-04-06T09:28:54.971 INFO:tasks.ceph:Running mkfs on osd nodes... 2024-04-06T09:28:54.971 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi006.front.sepia.ceph.com'): {'cluster1.osd.0': '/dev/vg_nvme/lv_1', 'cluster1.osd.1': '/dev/vg_nvme/lv_2', 'cluster1.osd.2': '/dev/vg_nvme/lv_3'}} 2024-04-06T09:28:54.971 DEBUG:teuthology.orchestra.run.smithi006:> sudo mkdir -p /var/lib/ceph/osd/cluster1-0 2024-04-06T09:28:54.986 INFO:tasks.ceph:roles_to_devs: {'cluster1.osd.0': '/dev/vg_nvme/lv_1', 'cluster1.osd.1': '/dev/vg_nvme/lv_2', 'cluster1.osd.2': '/dev/vg_nvme/lv_3'} 2024-04-06T09:28:54.986 INFO:tasks.ceph:role: cluster1.osd.0 2024-04-06T09:28:54.986 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi006.front.sepia.ceph.com 2024-04-06T09:28:54.986 DEBUG:teuthology.orchestra.run.smithi006:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-04-06T09:28:55.050 INFO:teuthology.orchestra.run.smithi006.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-04-06T09:28:55.050 INFO:teuthology.orchestra.run.smithi006.stdout: = sectsz=512 attr=2, projid32bit=1 2024-04-06T09:28:55.050 INFO:teuthology.orchestra.run.smithi006.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-04-06T09:28:55.050 INFO:teuthology.orchestra.run.smithi006.stdout: = reflink=1 bigtime=0 inobtcount=0 2024-04-06T09:28:55.051 INFO:teuthology.orchestra.run.smithi006.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-04-06T09:28:55.051 INFO:teuthology.orchestra.run.smithi006.stdout: = sunit=0 swidth=0 blks 2024-04-06T09:28:55.051 INFO:teuthology.orchestra.run.smithi006.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-04-06T09:28:55.051 INFO:teuthology.orchestra.run.smithi006.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-04-06T09:28:55.051 INFO:teuthology.orchestra.run.smithi006.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-04-06T09:28:55.051 INFO:teuthology.orchestra.run.smithi006.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-04-06T09:28:55.505 INFO:teuthology.orchestra.run.smithi006.stdout:Discarding blocks...Done. 2024-04-06T09:28:55.508 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi006.front.sepia.ceph.com -o noatime 2024-04-06T09:28:55.508 DEBUG:teuthology.orchestra.run.smithi006:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/cluster1-0 2024-04-06T09:28:55.607 DEBUG:teuthology.orchestra.run.smithi006:> sudo /sbin/restorecon /var/lib/ceph/osd/cluster1-0 2024-04-06T09:28:55.617 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: /sbin/restorecon: command not found 2024-04-06T09:28:55.618 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T09:28:55.618 DEBUG:teuthology.orchestra.run.smithi006:> sudo mkdir -p /var/lib/ceph/osd/cluster1-1 2024-04-06T09:28:55.672 INFO:tasks.ceph:roles_to_devs: {'cluster1.osd.0': '/dev/vg_nvme/lv_1', 'cluster1.osd.1': '/dev/vg_nvme/lv_2', 'cluster1.osd.2': '/dev/vg_nvme/lv_3'} 2024-04-06T09:28:55.672 INFO:tasks.ceph:role: cluster1.osd.1 2024-04-06T09:28:55.673 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi006.front.sepia.ceph.com 2024-04-06T09:28:55.673 DEBUG:teuthology.orchestra.run.smithi006:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-04-06T09:28:55.733 INFO:teuthology.orchestra.run.smithi006.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-04-06T09:28:55.733 INFO:teuthology.orchestra.run.smithi006.stdout: = sectsz=512 attr=2, projid32bit=1 2024-04-06T09:28:55.733 INFO:teuthology.orchestra.run.smithi006.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-04-06T09:28:55.733 INFO:teuthology.orchestra.run.smithi006.stdout: = reflink=1 bigtime=0 inobtcount=0 2024-04-06T09:28:55.733 INFO:teuthology.orchestra.run.smithi006.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-04-06T09:28:55.733 INFO:teuthology.orchestra.run.smithi006.stdout: = sunit=0 swidth=0 blks 2024-04-06T09:28:55.733 INFO:teuthology.orchestra.run.smithi006.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-04-06T09:28:55.734 INFO:teuthology.orchestra.run.smithi006.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-04-06T09:28:55.734 INFO:teuthology.orchestra.run.smithi006.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-04-06T09:28:55.734 INFO:teuthology.orchestra.run.smithi006.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-04-06T09:28:56.201 INFO:teuthology.orchestra.run.smithi006.stdout:Discarding blocks...Done. 2024-04-06T09:28:56.203 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi006.front.sepia.ceph.com -o noatime 2024-04-06T09:28:56.203 DEBUG:teuthology.orchestra.run.smithi006:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/cluster1-1 2024-04-06T09:28:56.343 DEBUG:teuthology.orchestra.run.smithi006:> sudo /sbin/restorecon /var/lib/ceph/osd/cluster1-1 2024-04-06T09:28:56.355 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T09:28:56.355 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: /sbin/restorecon: command not found 2024-04-06T09:28:56.356 DEBUG:teuthology.orchestra.run.smithi006:> sudo mkdir -p /var/lib/ceph/osd/cluster1-2 2024-04-06T09:28:56.415 INFO:tasks.ceph:roles_to_devs: {'cluster1.osd.0': '/dev/vg_nvme/lv_1', 'cluster1.osd.1': '/dev/vg_nvme/lv_2', 'cluster1.osd.2': '/dev/vg_nvme/lv_3'} 2024-04-06T09:28:56.415 INFO:tasks.ceph:role: cluster1.osd.2 2024-04-06T09:28:56.415 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi006.front.sepia.ceph.com 2024-04-06T09:28:56.415 DEBUG:teuthology.orchestra.run.smithi006:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-04-06T09:28:56.480 INFO:teuthology.orchestra.run.smithi006.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-04-06T09:28:56.480 INFO:teuthology.orchestra.run.smithi006.stdout: = sectsz=512 attr=2, projid32bit=1 2024-04-06T09:28:56.480 INFO:teuthology.orchestra.run.smithi006.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-04-06T09:28:56.480 INFO:teuthology.orchestra.run.smithi006.stdout: = reflink=1 bigtime=0 inobtcount=0 2024-04-06T09:28:56.480 INFO:teuthology.orchestra.run.smithi006.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-04-06T09:28:56.480 INFO:teuthology.orchestra.run.smithi006.stdout: = sunit=0 swidth=0 blks 2024-04-06T09:28:56.480 INFO:teuthology.orchestra.run.smithi006.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-04-06T09:28:56.480 INFO:teuthology.orchestra.run.smithi006.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-04-06T09:28:56.480 INFO:teuthology.orchestra.run.smithi006.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-04-06T09:28:56.481 INFO:teuthology.orchestra.run.smithi006.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-04-06T09:28:56.964 INFO:teuthology.orchestra.run.smithi006.stdout:Discarding blocks...Done. 2024-04-06T09:28:56.967 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi006.front.sepia.ceph.com -o noatime 2024-04-06T09:28:56.967 DEBUG:teuthology.orchestra.run.smithi006:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/cluster1-2 2024-04-06T09:28:57.083 DEBUG:teuthology.orchestra.run.smithi006:> sudo /sbin/restorecon /var/lib/ceph/osd/cluster1-2 2024-04-06T09:28:57.093 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: /sbin/restorecon: command not found 2024-04-06T09:28:57.093 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T09:28:57.094 DEBUG:teuthology.orchestra.run.smithi006:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster cluster1 --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/cluster1.monmap 2024-04-06T09:28:57.203 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-06T09:28:57.203+0000 7f7ec0f50640 -1 auth: error reading file: /var/lib/ceph/osd/cluster1-0/keyring: can't open /var/lib/ceph/osd/cluster1-0/keyring: (2) No such file or directory 2024-04-06T09:28:57.204 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-06T09:28:57.203+0000 7f7ec0f50640 -1 created new key in keyring /var/lib/ceph/osd/cluster1-0/keyring 2024-04-06T09:28:57.204 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-06T09:28:57.203+0000 7f7ec0f50640 -1 bluestore(/var/lib/ceph/osd/cluster1-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/cluster1-0/block: (2) No such file or directory 2024-04-06T09:28:57.204 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-06T09:28:57.203+0000 7f7ec0f50640 -1 bluestore(/var/lib/ceph/osd/cluster1-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/cluster1-0/block: (2) No such file or directory 2024-04-06T09:28:57.204 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-06T09:28:57.203+0000 7f7ec0f50640 -1 bluestore(/var/lib/ceph/osd/cluster1-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/cluster1-0/block: (2) No such file or directory 2024-04-06T09:28:57.204 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-06T09:28:57.203+0000 7f7ec0f50640 -1 bluestore(/var/lib/ceph/osd/cluster1-0) _read_fsid unparsable uuid 2024-04-06T09:29:00.193 DEBUG:teuthology.orchestra.run.smithi006:> sudo chown -R ceph:ceph /var/lib/ceph/osd/cluster1-0 2024-04-06T09:29:00.207 DEBUG:teuthology.orchestra.run.smithi006:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster cluster1 --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/cluster1.monmap 2024-04-06T09:29:00.289 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-06T09:29:00.287+0000 7fdea7fe4640 -1 auth: error reading file: /var/lib/ceph/osd/cluster1-1/keyring: can't open /var/lib/ceph/osd/cluster1-1/keyring: (2) No such file or directory 2024-04-06T09:29:00.290 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-06T09:29:00.287+0000 7fdea7fe4640 -1 created new key in keyring /var/lib/ceph/osd/cluster1-1/keyring 2024-04-06T09:29:00.290 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-06T09:29:00.287+0000 7fdea7fe4640 -1 bluestore(/var/lib/ceph/osd/cluster1-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/cluster1-1/block: (2) No such file or directory 2024-04-06T09:29:00.290 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-06T09:29:00.287+0000 7fdea7fe4640 -1 bluestore(/var/lib/ceph/osd/cluster1-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/cluster1-1/block: (2) No such file or directory 2024-04-06T09:29:00.290 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-06T09:29:00.287+0000 7fdea7fe4640 -1 bluestore(/var/lib/ceph/osd/cluster1-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/cluster1-1/block: (2) No such file or directory 2024-04-06T09:29:00.290 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-06T09:29:00.291+0000 7fdea7fe4640 -1 bluestore(/var/lib/ceph/osd/cluster1-1) _read_fsid unparsable uuid 2024-04-06T09:29:03.297 DEBUG:teuthology.orchestra.run.smithi006:> sudo chown -R ceph:ceph /var/lib/ceph/osd/cluster1-1 2024-04-06T09:29:03.355 DEBUG:teuthology.orchestra.run.smithi006:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster cluster1 --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/cluster1.monmap 2024-04-06T09:29:03.439 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-06T09:29:03.438+0000 7f0c3658f640 -1 auth: error reading file: /var/lib/ceph/osd/cluster1-2/keyring: can't open /var/lib/ceph/osd/cluster1-2/keyring: (2) No such file or directory 2024-04-06T09:29:03.439 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-06T09:29:03.438+0000 7f0c3658f640 -1 created new key in keyring /var/lib/ceph/osd/cluster1-2/keyring 2024-04-06T09:29:03.440 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-06T09:29:03.438+0000 7f0c3658f640 -1 bluestore(/var/lib/ceph/osd/cluster1-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/cluster1-2/block: (2) No such file or directory 2024-04-06T09:29:03.440 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-06T09:29:03.438+0000 7f0c3658f640 -1 bluestore(/var/lib/ceph/osd/cluster1-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/cluster1-2/block: (2) No such file or directory 2024-04-06T09:29:03.440 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-06T09:29:03.438+0000 7f0c3658f640 -1 bluestore(/var/lib/ceph/osd/cluster1-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/cluster1-2/block: (2) No such file or directory 2024-04-06T09:29:03.440 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-06T09:29:03.438+0000 7f0c3658f640 -1 bluestore(/var/lib/ceph/osd/cluster1-2) _read_fsid unparsable uuid 2024-04-06T09:29:06.428 DEBUG:teuthology.orchestra.run.smithi006:> sudo chown -R ceph:ceph /var/lib/ceph/osd/cluster1-2 2024-04-06T09:29:06.483 INFO:tasks.ceph:Reading keys from all nodes... 2024-04-06T09:29:06.483 DEBUG:teuthology.orchestra.run.smithi006:> set -ex 2024-04-06T09:29:06.484 DEBUG:teuthology.orchestra.run.smithi006:> sudo dd if=/var/lib/ceph/mgr/cluster1-x/keyring of=/dev/stdout 2024-04-06T09:29:06.542 DEBUG:teuthology.orchestra.run.smithi006:> set -ex 2024-04-06T09:29:06.543 DEBUG:teuthology.orchestra.run.smithi006:> sudo dd if=/var/lib/ceph/osd/cluster1-0/keyring of=/dev/stdout 2024-04-06T09:29:06.598 DEBUG:teuthology.orchestra.run.smithi006:> set -ex 2024-04-06T09:29:06.598 DEBUG:teuthology.orchestra.run.smithi006:> sudo dd if=/var/lib/ceph/osd/cluster1-1/keyring of=/dev/stdout 2024-04-06T09:29:06.654 DEBUG:teuthology.orchestra.run.smithi006:> set -ex 2024-04-06T09:29:06.655 DEBUG:teuthology.orchestra.run.smithi006:> sudo dd if=/var/lib/ceph/osd/cluster1-2/keyring of=/dev/stdout 2024-04-06T09:29:06.710 DEBUG:teuthology.orchestra.run.smithi006:> set -ex 2024-04-06T09:29:06.711 DEBUG:teuthology.orchestra.run.smithi006:> dd if=/etc/ceph/cluster1.client.0.keyring of=/dev/stdout 2024-04-06T09:29:06.759 DEBUG:teuthology.orchestra.run.smithi192:> set -ex 2024-04-06T09:29:06.759 DEBUG:teuthology.orchestra.run.smithi192:> dd if=/etc/ceph/cluster1.client.mirror.keyring of=/dev/stdout 2024-04-06T09:29:06.765 DEBUG:teuthology.orchestra.run.smithi192:> set -ex 2024-04-06T09:29:06.765 DEBUG:teuthology.orchestra.run.smithi192:> dd if=/etc/ceph/cluster1.client.mirror.0.keyring of=/dev/stdout 2024-04-06T09:29:06.811 DEBUG:teuthology.orchestra.run.smithi192:> set -ex 2024-04-06T09:29:06.812 DEBUG:teuthology.orchestra.run.smithi192:> dd if=/etc/ceph/cluster1.client.mirror.1.keyring of=/dev/stdout 2024-04-06T09:29:06.859 DEBUG:teuthology.orchestra.run.smithi192:> set -ex 2024-04-06T09:29:06.859 DEBUG:teuthology.orchestra.run.smithi192:> dd if=/etc/ceph/cluster1.client.mirror.2.keyring of=/dev/stdout 2024-04-06T09:29:06.907 DEBUG:teuthology.orchestra.run.smithi192:> set -ex 2024-04-06T09:29:06.907 DEBUG:teuthology.orchestra.run.smithi192:> dd if=/etc/ceph/cluster1.client.mirror.3.keyring of=/dev/stdout 2024-04-06T09:29:06.955 DEBUG:teuthology.orchestra.run.smithi192:> set -ex 2024-04-06T09:29:06.956 DEBUG:teuthology.orchestra.run.smithi192:> dd if=/etc/ceph/cluster1.client.mirror.4.keyring of=/dev/stdout 2024-04-06T09:29:07.003 DEBUG:teuthology.orchestra.run.smithi192:> set -ex 2024-04-06T09:29:07.004 DEBUG:teuthology.orchestra.run.smithi192:> dd if=/etc/ceph/cluster1.client.mirror.5.keyring of=/dev/stdout 2024-04-06T09:29:07.051 DEBUG:teuthology.orchestra.run.smithi192:> set -ex 2024-04-06T09:29:07.052 DEBUG:teuthology.orchestra.run.smithi192:> dd if=/etc/ceph/cluster1.client.mirror.6.keyring of=/dev/stdout 2024-04-06T09:29:07.099 INFO:tasks.ceph:Adding keys to all mons... 2024-04-06T09:29:07.099 DEBUG:teuthology.orchestra.run.smithi006:> sudo tee -a /etc/ceph/cluster1.keyring 2024-04-06T09:29:07.112 INFO:teuthology.orchestra.run.smithi006.stdout:[mgr.x] 2024-04-06T09:29:07.112 INFO:teuthology.orchestra.run.smithi006.stdout: key = AQDWFRFmbkY2DhAAsJqLwqbS/U2c1kd14iVFUg== 2024-04-06T09:29:07.112 INFO:teuthology.orchestra.run.smithi006.stdout:[osd.0] 2024-04-06T09:29:07.112 INFO:teuthology.orchestra.run.smithi006.stdout: key = AQDZFRFmsMkzDBAAqqNFibtIfznYtrkmwE6t7g== 2024-04-06T09:29:07.112 INFO:teuthology.orchestra.run.smithi006.stdout:[osd.1] 2024-04-06T09:29:07.112 INFO:teuthology.orchestra.run.smithi006.stdout: key = AQDcFRFmIA5UERAAo+3llcHFFmFWt+OqgTQBSw== 2024-04-06T09:29:07.112 INFO:teuthology.orchestra.run.smithi006.stdout:[osd.2] 2024-04-06T09:29:07.112 INFO:teuthology.orchestra.run.smithi006.stdout: key = AQDfFRFmv0c/GhAAGCyVSw9MvReEsJh5qcYiZw== 2024-04-06T09:29:07.112 INFO:teuthology.orchestra.run.smithi006.stdout:[client.0] 2024-04-06T09:29:07.112 INFO:teuthology.orchestra.run.smithi006.stdout: key = AQDWFRFmbJfvEhAAULz7DiKR/+RTgUExD1G8Rg== 2024-04-06T09:29:07.113 INFO:teuthology.orchestra.run.smithi006.stdout:[client.mirror] 2024-04-06T09:29:07.113 INFO:teuthology.orchestra.run.smithi006.stdout: key = AQDWFRFm1K3PFRAA74szCSZS4VbIN0ZuyE6Niw== 2024-04-06T09:29:07.113 INFO:teuthology.orchestra.run.smithi006.stdout:[client.mirror.0] 2024-04-06T09:29:07.113 INFO:teuthology.orchestra.run.smithi006.stdout: key = AQDWFRFmGz93GxAAmtGJdxfDlnBX1ExVJnD2qQ== 2024-04-06T09:29:07.113 INFO:teuthology.orchestra.run.smithi006.stdout:[client.mirror.1] 2024-04-06T09:29:07.113 INFO:teuthology.orchestra.run.smithi006.stdout: key = AQDWFRFm59NSIBAAHrivr+PrS1ds0C6FLAqh5Q== 2024-04-06T09:29:07.113 INFO:teuthology.orchestra.run.smithi006.stdout:[client.mirror.2] 2024-04-06T09:29:07.113 INFO:teuthology.orchestra.run.smithi006.stdout: key = AQDWFRFmyVwnJRAAS5NZ2r1YR39PKJRdmE7F4Q== 2024-04-06T09:29:07.113 INFO:teuthology.orchestra.run.smithi006.stdout:[client.mirror.3] 2024-04-06T09:29:07.113 INFO:teuthology.orchestra.run.smithi006.stdout: key = AQDWFRFmBCxSKhAAplpSB7EyT55WrOYThDsHow== 2024-04-06T09:29:07.113 INFO:teuthology.orchestra.run.smithi006.stdout:[client.mirror.4] 2024-04-06T09:29:07.113 INFO:teuthology.orchestra.run.smithi006.stdout: key = AQDWFRFmsmBWLxAAKCCAWjqzWs0uFxcZV/Q5SQ== 2024-04-06T09:29:07.113 INFO:teuthology.orchestra.run.smithi006.stdout:[client.mirror.5] 2024-04-06T09:29:07.113 INFO:teuthology.orchestra.run.smithi006.stdout: key = AQDWFRFmuwA3NBAADoYmuZLpM73n/QMvzXIcnw== 2024-04-06T09:29:07.114 INFO:teuthology.orchestra.run.smithi006.stdout:[client.mirror.6] 2024-04-06T09:29:07.114 INFO:teuthology.orchestra.run.smithi006.stdout: key = AQDWFRFmzf1KORAA3ryMX9CSf2QJ1axl9WjUoQ== 2024-04-06T09:29:07.114 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster1.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-04-06T09:29:07.205 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster1.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-06T09:29:07.292 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster1.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-06T09:29:07.380 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster1.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-06T09:29:07.419 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster1.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-04-06T09:29:07.505 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster1.keyring --name=client.mirror --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-04-06T09:29:07.588 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster1.keyring --name=client.mirror.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-04-06T09:29:07.671 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster1.keyring --name=client.mirror.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-04-06T09:29:07.761 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster1.keyring --name=client.mirror.2 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-04-06T09:29:07.804 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster1.keyring --name=client.mirror.3 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-04-06T09:29:07.888 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster1.keyring --name=client.mirror.4 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-04-06T09:29:07.970 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster1.keyring --name=client.mirror.5 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-04-06T09:29:08.062 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster1.keyring --name=client.mirror.6 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-04-06T09:29:08.104 INFO:tasks.ceph:Running mkfs on mon nodes... 2024-04-06T09:29:08.104 DEBUG:teuthology.orchestra.run.smithi006:> sudo mkdir -p /var/lib/ceph/mon/cluster1-a 2024-04-06T09:29:08.156 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster cluster1 --mkfs -i a --monmap /home/ubuntu/cephtest/cluster1.monmap --keyring /etc/ceph/cluster1.keyring 2024-04-06T09:29:08.244 DEBUG:teuthology.orchestra.run.smithi006:> sudo chown -R ceph:ceph /var/lib/ceph/mon/cluster1-a 2024-04-06T09:29:08.254 DEBUG:teuthology.orchestra.run.smithi006:> rm -- /home/ubuntu/cephtest/cluster1.monmap 2024-04-06T09:29:08.301 INFO:tasks.ceph:Starting mon daemons in cluster cluster1... 2024-04-06T09:29:08.302 INFO:tasks.ceph.cluster1.mon.a:Restarting daemon 2024-04-06T09:29:08.302 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster cluster1 -i a 2024-04-06T09:29:08.347 INFO:tasks.ceph.cluster1.mon.a:Started 2024-04-06T09:29:08.347 INFO:tasks.ceph:Starting mgr daemons in cluster cluster1... 2024-04-06T09:29:08.348 INFO:tasks.ceph.cluster1.mgr.x:Restarting daemon 2024-04-06T09:29:08.348 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster cluster1 -i x 2024-04-06T09:29:08.350 INFO:tasks.ceph.cluster1.mgr.x:Started 2024-04-06T09:29:08.350 INFO:tasks.ceph:Setting crush tunables to default 2024-04-06T09:29:08.350 DEBUG:teuthology.orchestra.run.smithi006:> sudo ceph --cluster cluster1 osd crush tunables default 2024-04-06T09:29:08.599 INFO:teuthology.orchestra.run.smithi006.stderr:adjusted tunables profile to default 2024-04-06T09:29:08.617 INFO:tasks.ceph:check_enable_crimson: False 2024-04-06T09:29:08.617 INFO:tasks.ceph:Starting osd daemons in cluster cluster1... 2024-04-06T09:29:08.617 DEBUG:teuthology.orchestra.run.smithi006:> set -ex 2024-04-06T09:29:08.617 DEBUG:teuthology.orchestra.run.smithi006:> sudo dd if=/var/lib/ceph/osd/cluster1-0/fsid of=/dev/stdout 2024-04-06T09:29:08.627 DEBUG:teuthology.orchestra.run.smithi006:> set -ex 2024-04-06T09:29:08.627 DEBUG:teuthology.orchestra.run.smithi006:> sudo dd if=/var/lib/ceph/osd/cluster1-1/fsid of=/dev/stdout 2024-04-06T09:29:08.679 DEBUG:teuthology.orchestra.run.smithi006:> set -ex 2024-04-06T09:29:08.679 DEBUG:teuthology.orchestra.run.smithi006:> sudo dd if=/var/lib/ceph/osd/cluster1-2/fsid of=/dev/stdout 2024-04-06T09:29:08.713 INFO:tasks.ceph.cluster1.mgr.x.smithi006.stderr:2024-04-06T09:29:08.710+0000 7f941eac9180 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-06T09:29:08.722 DEBUG:teuthology.orchestra.run.smithi006:> sudo ceph --cluster cluster1 osd new cb53ae9a-8742-4bb3-8546-e051ff9595dc 0 2024-04-06T09:29:08.971 INFO:teuthology.orchestra.run.smithi006.stdout:0 2024-04-06T09:29:08.989 DEBUG:teuthology.orchestra.run.smithi006:> sudo ceph --cluster cluster1 osd new 9da3b4ba-409a-4e1c-9589-1d70a6c058f5 1 2024-04-06T09:29:09.188 INFO:teuthology.orchestra.run.smithi006.stdout:1 2024-04-06T09:29:09.206 DEBUG:teuthology.orchestra.run.smithi006:> sudo ceph --cluster cluster1 osd new 2d732d23-05b4-4b5a-bbb2-e022b113a84c 2 2024-04-06T09:29:09.217 INFO:tasks.ceph.cluster1.mgr.x.smithi006.stderr:2024-04-06T09:29:09.214+0000 7f941eac9180 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-06T09:29:09.417 INFO:teuthology.orchestra.run.smithi006.stdout:2 2024-04-06T09:29:09.425 INFO:tasks.ceph.cluster1.mgr.x.smithi006.stderr:2024-04-06T09:29:09.422+0000 7f941eac9180 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-06T09:29:09.433 INFO:tasks.ceph.cluster1.osd.0:Restarting daemon 2024-04-06T09:29:09.433 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster cluster1 -i 0 2024-04-06T09:29:09.475 INFO:tasks.ceph.cluster1.osd.0:Started 2024-04-06T09:29:09.475 INFO:tasks.ceph.cluster1.osd.1:Restarting daemon 2024-04-06T09:29:09.475 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster cluster1 -i 1 2024-04-06T09:29:09.478 INFO:tasks.ceph.cluster1.osd.1:Started 2024-04-06T09:29:09.478 INFO:tasks.ceph.cluster1.osd.2:Restarting daemon 2024-04-06T09:29:09.478 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster cluster1 -i 2 2024-04-06T09:29:09.481 INFO:tasks.ceph.cluster1.osd.2:Started 2024-04-06T09:29:09.481 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 osd dump --format=json 2024-04-06T09:29:09.674 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-04-06T09:29:09.674 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":5,"fsid":"06a46034-4ef3-4e36-9fc5-05c8000433c6","created":"2024-04-06T09:29:08.480618+0000","modified":"2024-04-06T09:29:09.417013+0000","last_up_change":"0.000000","last_in_change":"2024-04-06T09:29:09.417013+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"cb53ae9a-8742-4bb3-8546-e051ff9595dc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"9da3b4ba-409a-4e1c-9589-1d70a6c058f5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"2d732d23-05b4-4b5a-bbb2-e022b113a84c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-06T09:29:09.691 INFO:tasks.ceph.ceph_manager.cluster1:[] 2024-04-06T09:29:09.691 INFO:tasks.ceph:Waiting for OSDs to come up 2024-04-06T09:29:09.993 DEBUG:teuthology.orchestra.run.smithi006:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster cluster1 osd dump --format=json 2024-04-06T09:29:10.046 INFO:tasks.ceph.cluster1.mgr.x.smithi006.stderr:2024-04-06T09:29:10.046+0000 7f941eac9180 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-06T09:29:10.095 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-04-06T09:29:10.094+0000 7fd1c6814640 -1 Falling back to public interface 2024-04-06T09:29:10.103 INFO:tasks.ceph.cluster1.mgr.x.smithi006.stderr:2024-04-06T09:29:10.102+0000 7f941eac9180 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-06T09:29:10.119 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:29:10.118+0000 7f0975796640 -1 Falling back to public interface 2024-04-06T09:29:10.123 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:29:10.122+0000 7f2e9536a640 -1 Falling back to public interface 2024-04-06T09:29:10.161 INFO:tasks.ceph.cluster1.mgr.x.smithi006.stderr:2024-04-06T09:29:10.158+0000 7f941eac9180 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-06T09:29:10.175 INFO:teuthology.misc.health.smithi006.stdout: 2024-04-06T09:29:10.175 INFO:teuthology.misc.health.smithi006.stdout:{"epoch":5,"fsid":"06a46034-4ef3-4e36-9fc5-05c8000433c6","created":"2024-04-06T09:29:08.480618+0000","modified":"2024-04-06T09:29:09.417013+0000","last_up_change":"0.000000","last_in_change":"2024-04-06T09:29:09.417013+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"cb53ae9a-8742-4bb3-8546-e051ff9595dc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"9da3b4ba-409a-4e1c-9589-1d70a6c058f5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"2d732d23-05b4-4b5a-bbb2-e022b113a84c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-06T09:29:10.190 DEBUG:teuthology.misc:0 of 3 OSDs are up 2024-04-06T09:29:10.242 INFO:tasks.ceph.cluster1.mgr.x.smithi006.stderr:2024-04-06T09:29:10.242+0000 7f941eac9180 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-06T09:29:10.296 INFO:tasks.ceph.cluster1.mgr.x.smithi006.stderr:2024-04-06T09:29:10.294+0000 7f941eac9180 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-06T09:29:10.377 INFO:tasks.ceph.cluster1.mgr.x.smithi006.stderr:2024-04-06T09:29:10.374+0000 7f941eac9180 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-06T09:29:10.431 INFO:tasks.ceph.cluster1.mgr.x.smithi006.stderr:2024-04-06T09:29:10.430+0000 7f941eac9180 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-06T09:29:10.587 INFO:tasks.ceph.cluster1.mgr.x.smithi006.stderr:2024-04-06T09:29:10.586+0000 7f941eac9180 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-06T09:29:10.649 INFO:tasks.ceph.cluster1.mgr.x.smithi006.stderr:2024-04-06T09:29:10.646+0000 7f941eac9180 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-06T09:29:11.247 INFO:tasks.ceph.cluster1.mgr.x.smithi006.stderr:/usr/lib/python3/dist-packages/scipy/__init__.py:67: UserWarning: NumPy was imported from a Python sub-interpreter but NumPy does not properly support sub-interpreters. This will likely work for most users but might cause hard to track down issues or subtle bugs. A common user of the rare sub-interpreter feature is wsgi which also allows single-interpreter mode. 2024-04-06T09:29:11.247 INFO:tasks.ceph.cluster1.mgr.x.smithi006.stderr:Improvements in the case of bugs are welcome, but is not on the NumPy roadmap, and full support may require significant effort to achieve. 2024-04-06T09:29:11.247 INFO:tasks.ceph.cluster1.mgr.x.smithi006.stderr: from numpy import show_config as show_numpy_config 2024-04-06T09:29:11.309 INFO:tasks.ceph.cluster1.mgr.x.smithi006.stderr:2024-04-06T09:29:11.306+0000 7f941eac9180 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-06T09:29:11.493 INFO:tasks.ceph.cluster1.mgr.x.smithi006.stderr:2024-04-06T09:29:11.490+0000 7f941eac9180 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-06T09:29:11.671 INFO:tasks.ceph.cluster1.mgr.x.smithi006.stderr:2024-04-06T09:29:11.670+0000 7f941eac9180 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-06T09:29:11.750 INFO:tasks.ceph.cluster1.mgr.x.smithi006.stderr:2024-04-06T09:29:11.750+0000 7f941eac9180 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-06T09:29:11.889 INFO:tasks.ceph.cluster1.mgr.x.smithi006.stderr:2024-04-06T09:29:11.886+0000 7f941eac9180 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-06T09:29:12.540 INFO:tasks.ceph.cluster1.mgr.x.smithi006.stderr:2024-04-06T09:29:12.538+0000 7f941eac9180 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-06T09:29:12.619 INFO:tasks.ceph.cluster1.mgr.x.smithi006.stderr:2024-04-06T09:29:12.618+0000 7f941eac9180 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-06T09:29:12.808 INFO:tasks.ceph.cluster1.mgr.x.smithi006.stderr:2024-04-06T09:29:12.806+0000 7f941eac9180 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-06T09:29:12.885 INFO:tasks.ceph.cluster1.mgr.x.smithi006.stderr:2024-04-06T09:29:12.882+0000 7f941eac9180 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-06T09:29:13.204 INFO:tasks.ceph.cluster1.mgr.x.smithi006.stderr:2024-04-06T09:29:13.202+0000 7f941eac9180 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-06T09:29:13.275 INFO:tasks.ceph.cluster1.mgr.x.smithi006.stderr:2024-04-06T09:29:13.274+0000 7f941eac9180 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-06T09:29:13.884 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-04-06T09:29:13.882+0000 7fd1c6814640 -1 osd.1 0 log_to_monitors true 2024-04-06T09:29:13.931 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:29:13.930+0000 7f2e9536a640 -1 osd.2 0 log_to_monitors true 2024-04-06T09:29:13.979 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:29:13.978+0000 7f0975796640 -1 osd.0 0 log_to_monitors true 2024-04-06T09:29:14.513 INFO:tasks.ceph.cluster1.mgr.x.smithi006.stderr:2024-04-06T09:29:14.510+0000 7f93c5f81640 -1 mgr.server handle_report got status from non-daemon mon.a 2024-04-06T09:29:16.493 DEBUG:teuthology.orchestra.run.smithi006:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster cluster1 osd dump --format=json 2024-04-06T09:29:16.679 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:29:16.678+0000 7f2e9130f640 -1 osd.2 0 waiting for initial osdmap 2024-04-06T09:29:16.684 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-04-06T09:29:16.682+0000 7fd1c27b9640 -1 osd.1 0 waiting for initial osdmap 2024-04-06T09:29:16.699 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:29:16.698+0000 7f097173b640 -1 osd.0 0 waiting for initial osdmap 2024-04-06T09:29:16.715 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:29:16.714+0000 7f2e89142640 -1 osd.2 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-06T09:29:16.719 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-04-06T09:29:16.718+0000 7fd1ba5ec640 -1 osd.1 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-06T09:29:16.739 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:29:16.738+0000 7f096956e640 -1 osd.0 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-06T09:29:16.857 INFO:teuthology.misc.health.smithi006.stdout: 2024-04-06T09:29:16.857 INFO:teuthology.misc.health.smithi006.stdout:{"epoch":7,"fsid":"06a46034-4ef3-4e36-9fc5-05c8000433c6","created":"2024-04-06T09:29:08.480618+0000","modified":"2024-04-06T09:29:15.429401+0000","last_up_change":"0.000000","last_in_change":"2024-04-06T09:29:09.417013+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"cb53ae9a-8742-4bb3-8546-e051ff9595dc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"9da3b4ba-409a-4e1c-9589-1d70a6c058f5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"2d732d23-05b4-4b5a-bbb2-e022b113a84c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-06T09:29:16.878 DEBUG:teuthology.misc:0 of 3 OSDs are up 2024-04-06T09:29:23.181 DEBUG:teuthology.orchestra.run.smithi006:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster cluster1 osd dump --format=json 2024-04-06T09:29:23.483 INFO:teuthology.misc.health.smithi006.stdout: 2024-04-06T09:29:23.483 INFO:teuthology.misc.health.smithi006.stdout:{"epoch":12,"fsid":"06a46034-4ef3-4e36-9fc5-05c8000433c6","created":"2024-04-06T09:29:08.480618+0000","modified":"2024-04-06T09:29:21.711062+0000","last_up_change":"2024-04-06T09:29:17.701553+0000","last_in_change":"2024-04-06T09:29:09.417013+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-06T09:29:19.376917+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"12","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":2.9900000095367432,"score_stable":2.9900000095367432,"optimal_score":0.67000001668930054,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"cb53ae9a-8742-4bb3-8546-e051ff9595dc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6808","nonce":23799},{"type":"v1","addr":"172.21.15.6:6809","nonce":23799}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6810","nonce":23799},{"type":"v1","addr":"172.21.15.6:6811","nonce":23799}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6814","nonce":23799},{"type":"v1","addr":"172.21.15.6:6815","nonce":23799}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6812","nonce":23799},{"type":"v1","addr":"172.21.15.6:6813","nonce":23799}]},"public_addr":"172.21.15.6:6809/23799","cluster_addr":"172.21.15.6:6811/23799","heartbeat_back_addr":"172.21.15.6:6815/23799","heartbeat_front_addr":"172.21.15.6:6813/23799","state":["exists","up"]},{"osd":1,"uuid":"9da3b4ba-409a-4e1c-9589-1d70a6c058f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":10,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6800","nonce":23798},{"type":"v1","addr":"172.21.15.6:6801","nonce":23798}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6802","nonce":23798},{"type":"v1","addr":"172.21.15.6:6803","nonce":23798}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6806","nonce":23798},{"type":"v1","addr":"172.21.15.6:6807","nonce":23798}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6804","nonce":23798},{"type":"v1","addr":"172.21.15.6:6805","nonce":23798}]},"public_addr":"172.21.15.6:6801/23798","cluster_addr":"172.21.15.6:6803/23798","heartbeat_back_addr":"172.21.15.6:6807/23798","heartbeat_front_addr":"172.21.15.6:6805/23798","state":["exists","up"]},{"osd":2,"uuid":"2d732d23-05b4-4b5a-bbb2-e022b113a84c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6816","nonce":23800},{"type":"v1","addr":"172.21.15.6:6817","nonce":23800}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6818","nonce":23800},{"type":"v1","addr":"172.21.15.6:6819","nonce":23800}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6822","nonce":23800},{"type":"v1","addr":"172.21.15.6:6823","nonce":23800}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6820","nonce":23800},{"type":"v1","addr":"172.21.15.6:6821","nonce":23800}]},"public_addr":"172.21.15.6:6817/23800","cluster_addr":"172.21.15.6:6819/23800","heartbeat_back_addr":"172.21.15.6:6823/23800","heartbeat_front_addr":"172.21.15.6:6821/23800","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T09:29:14.935316+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T09:29:14.876558+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T09:29:14.907628+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-06T09:29:23.500 DEBUG:teuthology.misc:3 of 3 OSDs are up 2024-04-06T09:29:23.500 INFO:tasks.ceph:Creating RBD pool 2024-04-06T09:29:23.500 DEBUG:teuthology.orchestra.run.smithi006:> sudo ceph --cluster cluster1 osd pool create rbd 8 2024-04-06T09:29:24.722 INFO:teuthology.orchestra.run.smithi006.stderr:pool 'rbd' created 2024-04-06T09:29:24.747 DEBUG:teuthology.orchestra.run.smithi006:> sudo ceph --cluster cluster1 osd pool application enable rbd rbd --yes-i-really-mean-it 2024-04-06T09:29:25.729 INFO:teuthology.orchestra.run.smithi006.stderr:enabled application 'rbd' on pool 'rbd' 2024-04-06T09:29:25.756 INFO:tasks.ceph:Starting mds daemons in cluster cluster1... 2024-04-06T09:29:25.757 INFO:tasks.ceph:Waiting until cluster1 daemons up and pgs clean... 2024-04-06T09:29:25.757 INFO:tasks.ceph.ceph_manager.cluster1:waiting for mgr available 2024-04-06T09:29:25.757 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 mgr dump --format=json 2024-04-06T09:29:25.757 INFO:tasks.daemonwatchdog.daemon_watchdog:watchdog starting 2024-04-06T09:29:26.096 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-04-06T09:29:26.113 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":5,"active_gid":4099,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6824","nonce":23622},{"type":"v1","addr":"172.21.15.6:6825","nonce":23622}]},"active_addr":"172.21.15.6:6825/23622","active_change":"2024-04-06T09:29:13.296397+0000","active_mgr_features":4540138322906710015,"available":true,"standbys":[],"modules":["iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.7","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.0.0","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":0,"active_clients":[{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.6:0","nonce":543394975}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.6:0","nonce":1261358482}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.6:0","nonce":1497146861}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.6:0","nonce":3013818611}]}]} 2024-04-06T09:29:26.115 INFO:tasks.ceph.ceph_manager.cluster1:mgr available! 2024-04-06T09:29:26.115 INFO:tasks.ceph.ceph_manager.cluster1:waiting for all up 2024-04-06T09:29:26.115 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 osd dump --format=json 2024-04-06T09:29:26.398 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-04-06T09:29:26.398 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":14,"fsid":"06a46034-4ef3-4e36-9fc5-05c8000433c6","created":"2024-04-06T09:29:08.480618+0000","modified":"2024-04-06T09:29:25.719844+0000","last_up_change":"2024-04-06T09:29:17.701553+0000","last_in_change":"2024-04-06T09:29:09.417013+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-06T09:29:19.376917+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"12","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":2.9900000095367432,"score_stable":2.9900000095367432,"optimal_score":0.67000001668930054,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-04-06T09:29:23.842851+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"14","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"cb53ae9a-8742-4bb3-8546-e051ff9595dc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":13,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6808","nonce":23799},{"type":"v1","addr":"172.21.15.6:6809","nonce":23799}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6810","nonce":23799},{"type":"v1","addr":"172.21.15.6:6811","nonce":23799}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6814","nonce":23799},{"type":"v1","addr":"172.21.15.6:6815","nonce":23799}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6812","nonce":23799},{"type":"v1","addr":"172.21.15.6:6813","nonce":23799}]},"public_addr":"172.21.15.6:6809/23799","cluster_addr":"172.21.15.6:6811/23799","heartbeat_back_addr":"172.21.15.6:6815/23799","heartbeat_front_addr":"172.21.15.6:6813/23799","state":["exists","up"]},{"osd":1,"uuid":"9da3b4ba-409a-4e1c-9589-1d70a6c058f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":13,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6800","nonce":23798},{"type":"v1","addr":"172.21.15.6:6801","nonce":23798}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6802","nonce":23798},{"type":"v1","addr":"172.21.15.6:6803","nonce":23798}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6806","nonce":23798},{"type":"v1","addr":"172.21.15.6:6807","nonce":23798}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6804","nonce":23798},{"type":"v1","addr":"172.21.15.6:6805","nonce":23798}]},"public_addr":"172.21.15.6:6801/23798","cluster_addr":"172.21.15.6:6803/23798","heartbeat_back_addr":"172.21.15.6:6807/23798","heartbeat_front_addr":"172.21.15.6:6805/23798","state":["exists","up"]},{"osd":2,"uuid":"2d732d23-05b4-4b5a-bbb2-e022b113a84c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":13,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6816","nonce":23800},{"type":"v1","addr":"172.21.15.6:6817","nonce":23800}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6818","nonce":23800},{"type":"v1","addr":"172.21.15.6:6819","nonce":23800}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6822","nonce":23800},{"type":"v1","addr":"172.21.15.6:6823","nonce":23800}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6820","nonce":23800},{"type":"v1","addr":"172.21.15.6:6821","nonce":23800}]},"public_addr":"172.21.15.6:6817/23800","cluster_addr":"172.21.15.6:6819/23800","heartbeat_back_addr":"172.21.15.6:6823/23800","heartbeat_front_addr":"172.21.15.6:6821/23800","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T09:29:14.935316+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T09:29:14.876558+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T09:29:14.907628+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-06T09:29:26.415 INFO:tasks.ceph.ceph_manager.cluster1:all up! 2024-04-06T09:29:26.415 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 osd dump --format=json 2024-04-06T09:29:26.684 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-04-06T09:29:26.684 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":14,"fsid":"06a46034-4ef3-4e36-9fc5-05c8000433c6","created":"2024-04-06T09:29:08.480618+0000","modified":"2024-04-06T09:29:25.719844+0000","last_up_change":"2024-04-06T09:29:17.701553+0000","last_in_change":"2024-04-06T09:29:09.417013+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-06T09:29:19.376917+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"12","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":2.9900000095367432,"score_stable":2.9900000095367432,"optimal_score":0.67000001668930054,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-04-06T09:29:23.842851+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"14","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"cb53ae9a-8742-4bb3-8546-e051ff9595dc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":13,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6808","nonce":23799},{"type":"v1","addr":"172.21.15.6:6809","nonce":23799}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6810","nonce":23799},{"type":"v1","addr":"172.21.15.6:6811","nonce":23799}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6814","nonce":23799},{"type":"v1","addr":"172.21.15.6:6815","nonce":23799}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6812","nonce":23799},{"type":"v1","addr":"172.21.15.6:6813","nonce":23799}]},"public_addr":"172.21.15.6:6809/23799","cluster_addr":"172.21.15.6:6811/23799","heartbeat_back_addr":"172.21.15.6:6815/23799","heartbeat_front_addr":"172.21.15.6:6813/23799","state":["exists","up"]},{"osd":1,"uuid":"9da3b4ba-409a-4e1c-9589-1d70a6c058f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":13,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6800","nonce":23798},{"type":"v1","addr":"172.21.15.6:6801","nonce":23798}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6802","nonce":23798},{"type":"v1","addr":"172.21.15.6:6803","nonce":23798}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6806","nonce":23798},{"type":"v1","addr":"172.21.15.6:6807","nonce":23798}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6804","nonce":23798},{"type":"v1","addr":"172.21.15.6:6805","nonce":23798}]},"public_addr":"172.21.15.6:6801/23798","cluster_addr":"172.21.15.6:6803/23798","heartbeat_back_addr":"172.21.15.6:6807/23798","heartbeat_front_addr":"172.21.15.6:6805/23798","state":["exists","up"]},{"osd":2,"uuid":"2d732d23-05b4-4b5a-bbb2-e022b113a84c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":13,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6816","nonce":23800},{"type":"v1","addr":"172.21.15.6:6817","nonce":23800}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6818","nonce":23800},{"type":"v1","addr":"172.21.15.6:6819","nonce":23800}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6822","nonce":23800},{"type":"v1","addr":"172.21.15.6:6823","nonce":23800}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6820","nonce":23800},{"type":"v1","addr":"172.21.15.6:6821","nonce":23800}]},"public_addr":"172.21.15.6:6817/23800","cluster_addr":"172.21.15.6:6819/23800","heartbeat_back_addr":"172.21.15.6:6823/23800","heartbeat_front_addr":"172.21.15.6:6821/23800","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T09:29:14.935316+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T09:29:14.876558+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T09:29:14.907628+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-06T09:29:26.701 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 tell osd.0 flush_pg_stats 2024-04-06T09:29:26.702 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 tell osd.1 flush_pg_stats 2024-04-06T09:29:26.702 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 tell osd.2 flush_pg_stats 2024-04-06T09:29:26.854 INFO:teuthology.orchestra.run.smithi006.stdout:34359738372 2024-04-06T09:29:26.854 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 osd last-stat-seq osd.0 2024-04-06T09:29:26.858 INFO:teuthology.orchestra.run.smithi006.stdout:34359738373 2024-04-06T09:29:26.858 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 osd last-stat-seq osd.2 2024-04-06T09:29:27.054 INFO:teuthology.orchestra.run.smithi006.stdout:34359738372 2024-04-06T09:29:27.054 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 osd last-stat-seq osd.1 2024-04-06T09:29:27.163 INFO:teuthology.orchestra.run.smithi006.stdout:34359738371 2024-04-06T09:29:27.181 INFO:tasks.ceph.ceph_manager.cluster1:need seq 34359738372 got 34359738371 for osd.0 2024-04-06T09:29:27.222 INFO:teuthology.orchestra.run.smithi006.stdout:34359738372 2024-04-06T09:29:27.238 INFO:tasks.ceph.ceph_manager.cluster1:need seq 34359738373 got 34359738372 for osd.2 2024-04-06T09:29:27.326 INFO:teuthology.orchestra.run.smithi006.stdout:34359738371 2024-04-06T09:29:27.343 INFO:tasks.ceph.ceph_manager.cluster1:need seq 34359738372 got 34359738371 for osd.1 2024-04-06T09:29:28.182 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 osd last-stat-seq osd.0 2024-04-06T09:29:28.239 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 osd last-stat-seq osd.2 2024-04-06T09:29:28.344 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 osd last-stat-seq osd.1 2024-04-06T09:29:28.494 INFO:teuthology.orchestra.run.smithi006.stdout:34359738372 2024-04-06T09:29:28.511 INFO:tasks.ceph.ceph_manager.cluster1:need seq 34359738372 got 34359738372 for osd.0 2024-04-06T09:29:28.511 DEBUG:teuthology.parallel:result is None 2024-04-06T09:29:28.545 INFO:teuthology.orchestra.run.smithi006.stdout:34359738373 2024-04-06T09:29:28.562 INFO:tasks.ceph.ceph_manager.cluster1:need seq 34359738373 got 34359738373 for osd.2 2024-04-06T09:29:28.562 DEBUG:teuthology.parallel:result is None 2024-04-06T09:29:28.625 INFO:teuthology.orchestra.run.smithi006.stdout:34359738372 2024-04-06T09:29:28.643 INFO:tasks.ceph.ceph_manager.cluster1:need seq 34359738372 got 34359738372 for osd.1 2024-04-06T09:29:28.644 DEBUG:teuthology.parallel:result is None 2024-04-06T09:29:28.644 INFO:tasks.ceph.ceph_manager.cluster1:waiting for clean 2024-04-06T09:29:28.644 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 pg dump --format=json 2024-04-06T09:29:28.929 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-04-06T09:29:28.930 INFO:teuthology.orchestra.run.smithi006.stderr:dumped all 2024-04-06T09:29:28.946 INFO:teuthology.orchestra.run.smithi006.stdout:{"pg_ready":true,"pg_map":{"version":18,"stamp":"2024-04-06T09:29:27.303860+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":4,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":81556,"kb_used_data":480,"kb_used_omap":4,"kb_used_meta":81019,"kb_avail":283033964,"statfs":{"total":289910292480,"available":289826779136,"internally_reserved":0,"allocated":491520,"data_stored":1281896,"data_compressed":60676,"data_compressed_allocated":81920,"data_compressed_original":1146880,"omap_allocated":4782,"internal_metadata":82963794},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"2.583434"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":9,"reported_epoch":14,"state":"active+clean","last_fresh":"2024-04-06T09:29:25.742341+0000","last_change":"2024-04-06T09:29:25.738609+0000","last_active":"2024-04-06T09:29:25.742341+0000","last_peered":"2024-04-06T09:29:25.742341+0000","last_clean":"2024-04-06T09:29:25.742341+0000","last_became_active":"2024-04-06T09:29:25.737557+0000","last_became_peered":"2024-04-06T09:29:25.737557+0000","last_unstale":"2024-04-06T09:29:25.742341+0000","last_undegraded":"2024-04-06T09:29:25.742341+0000","last_fullsized":"2024-04-06T09:29:25.742341+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T09:29:24.717518+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T09:29:24.717518+0000","last_clean_scrub_stamp":"2024-04-06T09:29:24.717518+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T15:44:35.416009+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":9,"reported_epoch":14,"state":"active+clean","last_fresh":"2024-04-06T09:29:25.742313+0000","last_change":"2024-04-06T09:29:25.738328+0000","last_active":"2024-04-06T09:29:25.742313+0000","last_peered":"2024-04-06T09:29:25.742313+0000","last_clean":"2024-04-06T09:29:25.742313+0000","last_became_active":"2024-04-06T09:29:25.737348+0000","last_became_peered":"2024-04-06T09:29:25.737348+0000","last_unstale":"2024-04-06T09:29:25.742313+0000","last_undegraded":"2024-04-06T09:29:25.742313+0000","last_fullsized":"2024-04-06T09:29:25.742313+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T09:29:24.717518+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T09:29:24.717518+0000","last_clean_scrub_stamp":"2024-04-06T09:29:24.717518+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T10:00:30.760505+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":9,"reported_epoch":14,"state":"active+clean","last_fresh":"2024-04-06T09:29:25.742369+0000","last_change":"2024-04-06T09:29:25.738960+0000","last_active":"2024-04-06T09:29:25.742369+0000","last_peered":"2024-04-06T09:29:25.742369+0000","last_clean":"2024-04-06T09:29:25.742369+0000","last_became_active":"2024-04-06T09:29:25.738591+0000","last_became_peered":"2024-04-06T09:29:25.738591+0000","last_unstale":"2024-04-06T09:29:25.742369+0000","last_undegraded":"2024-04-06T09:29:25.742369+0000","last_fullsized":"2024-04-06T09:29:25.742369+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T09:29:24.717518+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T09:29:24.717518+0000","last_clean_scrub_stamp":"2024-04-06T09:29:24.717518+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T14:16:54.527503+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":9,"reported_epoch":14,"state":"active+clean","last_fresh":"2024-04-06T09:29:25.742302+0000","last_change":"2024-04-06T09:29:25.738138+0000","last_active":"2024-04-06T09:29:25.742302+0000","last_peered":"2024-04-06T09:29:25.742302+0000","last_clean":"2024-04-06T09:29:25.742302+0000","last_became_active":"2024-04-06T09:29:25.737263+0000","last_became_peered":"2024-04-06T09:29:25.737263+0000","last_unstale":"2024-04-06T09:29:25.742302+0000","last_undegraded":"2024-04-06T09:29:25.742302+0000","last_fullsized":"2024-04-06T09:29:25.742302+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T09:29:24.717518+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T09:29:24.717518+0000","last_clean_scrub_stamp":"2024-04-06T09:29:24.717518+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T14:30:58.846520+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":9,"reported_epoch":14,"state":"active+clean","last_fresh":"2024-04-06T09:29:25.742147+0000","last_change":"2024-04-06T09:29:25.739673+0000","last_active":"2024-04-06T09:29:25.742147+0000","last_peered":"2024-04-06T09:29:25.742147+0000","last_clean":"2024-04-06T09:29:25.742147+0000","last_became_active":"2024-04-06T09:29:25.738678+0000","last_became_peered":"2024-04-06T09:29:25.738678+0000","last_unstale":"2024-04-06T09:29:25.742147+0000","last_undegraded":"2024-04-06T09:29:25.742147+0000","last_fullsized":"2024-04-06T09:29:25.742147+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T09:29:24.717518+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T09:29:24.717518+0000","last_clean_scrub_stamp":"2024-04-06T09:29:24.717518+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T12:22:31.093977+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":10,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-04-06T09:29:26.730364+0000","last_change":"2024-04-06T09:29:25.739006+0000","last_active":"2024-04-06T09:29:26.730364+0000","last_peered":"2024-04-06T09:29:26.730364+0000","last_clean":"2024-04-06T09:29:26.730364+0000","last_became_active":"2024-04-06T09:29:25.738605+0000","last_became_peered":"2024-04-06T09:29:25.738605+0000","last_unstale":"2024-04-06T09:29:26.730364+0000","last_undegraded":"2024-04-06T09:29:26.730364+0000","last_fullsized":"2024-04-06T09:29:26.730364+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T09:29:24.717518+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T09:29:24.717518+0000","last_clean_scrub_stamp":"2024-04-06T09:29:24.717518+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T20:26:25.794984+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":10,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-04-06T09:29:26.730364+0000","last_change":"2024-04-06T09:29:25.739486+0000","last_active":"2024-04-06T09:29:26.730364+0000","last_peered":"2024-04-06T09:29:26.730364+0000","last_clean":"2024-04-06T09:29:26.730364+0000","last_became_active":"2024-04-06T09:29:25.738108+0000","last_became_peered":"2024-04-06T09:29:25.738108+0000","last_unstale":"2024-04-06T09:29:26.730364+0000","last_undegraded":"2024-04-06T09:29:26.730364+0000","last_fullsized":"2024-04-06T09:29:26.730364+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T09:29:24.717518+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T09:29:24.717518+0000","last_clean_scrub_stamp":"2024-04-06T09:29:24.717518+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T18:32:48.633225+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":9,"reported_epoch":14,"state":"active+clean","last_fresh":"2024-04-06T09:29:25.742268+0000","last_change":"2024-04-06T09:29:25.737897+0000","last_active":"2024-04-06T09:29:25.742268+0000","last_peered":"2024-04-06T09:29:25.742268+0000","last_clean":"2024-04-06T09:29:25.742268+0000","last_became_active":"2024-04-06T09:29:25.737369+0000","last_became_peered":"2024-04-06T09:29:25.737369+0000","last_unstale":"2024-04-06T09:29:25.742268+0000","last_undegraded":"2024-04-06T09:29:25.742268+0000","last_fullsized":"2024-04-06T09:29:25.742268+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T09:29:24.717518+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T09:29:24.717518+0000","last_clean_scrub_stamp":"2024-04-06T09:29:24.717518+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T16:57:27.614162+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"12'97","reported_seq":139,"reported_epoch":13,"state":"active+clean","last_fresh":"2024-04-06T09:29:24.730761+0000","last_change":"2024-04-06T09:29:21.046364+0000","last_active":"2024-04-06T09:29:24.730761+0000","last_peered":"2024-04-06T09:29:24.730761+0000","last_clean":"2024-04-06T09:29:24.730761+0000","last_became_active":"2024-04-06T09:29:21.043913+0000","last_became_peered":"2024-04-06T09:29:21.043913+0000","last_unstale":"2024-04-06T09:29:24.730761+0000","last_undegraded":"2024-04-06T09:29:24.730761+0000","last_fullsized":"2024-04-06T09:29:24.730761+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T09:29:19.704883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T09:29:19.704883+0000","last_clean_scrub_stamp":"2024-04-06T09:29:19.704883+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T14:04:37.575084+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":122880,"data_stored":1180736,"data_compressed":60676,"data_compressed_allocated":81920,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738373,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27140,"kb_used_data":120,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344700,"statfs":{"total":96636764160,"available":96608972800,"internally_reserved":0,"allocated":122880,"data_stored":33720,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.64500000000000002}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.47799999999999998}]}]},{"osd":1,"up_from":8,"seq":34359738372,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27216,"kb_used_data":180,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344624,"statfs":{"total":96636764160,"available":96608894976,"internally_reserved":0,"allocated":184320,"data_stored":624088,"data_compressed":30338,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.621}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.70499999999999996}]}]},{"osd":0,"up_from":8,"seq":34359738372,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27200,"kb_used_data":180,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344640,"statfs":{"total":96636764160,"available":96608911360,"internally_reserved":0,"allocated":184320,"data_stored":624088,"data_compressed":30338,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.60999999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.71699999999999997}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":590368,"data_compressed":30338,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":590368,"data_compressed":30338,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-06T09:29:28.947 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 pg dump --format=json 2024-04-06T09:29:29.223 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-04-06T09:29:29.223 INFO:teuthology.orchestra.run.smithi006.stderr:dumped all 2024-04-06T09:29:29.239 INFO:teuthology.orchestra.run.smithi006.stdout:{"pg_ready":true,"pg_map":{"version":18,"stamp":"2024-04-06T09:29:27.303860+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":4,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":81556,"kb_used_data":480,"kb_used_omap":4,"kb_used_meta":81019,"kb_avail":283033964,"statfs":{"total":289910292480,"available":289826779136,"internally_reserved":0,"allocated":491520,"data_stored":1281896,"data_compressed":60676,"data_compressed_allocated":81920,"data_compressed_original":1146880,"omap_allocated":4782,"internal_metadata":82963794},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"2.583434"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":9,"reported_epoch":14,"state":"active+clean","last_fresh":"2024-04-06T09:29:25.742341+0000","last_change":"2024-04-06T09:29:25.738609+0000","last_active":"2024-04-06T09:29:25.742341+0000","last_peered":"2024-04-06T09:29:25.742341+0000","last_clean":"2024-04-06T09:29:25.742341+0000","last_became_active":"2024-04-06T09:29:25.737557+0000","last_became_peered":"2024-04-06T09:29:25.737557+0000","last_unstale":"2024-04-06T09:29:25.742341+0000","last_undegraded":"2024-04-06T09:29:25.742341+0000","last_fullsized":"2024-04-06T09:29:25.742341+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T09:29:24.717518+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T09:29:24.717518+0000","last_clean_scrub_stamp":"2024-04-06T09:29:24.717518+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T15:44:35.416009+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":9,"reported_epoch":14,"state":"active+clean","last_fresh":"2024-04-06T09:29:25.742313+0000","last_change":"2024-04-06T09:29:25.738328+0000","last_active":"2024-04-06T09:29:25.742313+0000","last_peered":"2024-04-06T09:29:25.742313+0000","last_clean":"2024-04-06T09:29:25.742313+0000","last_became_active":"2024-04-06T09:29:25.737348+0000","last_became_peered":"2024-04-06T09:29:25.737348+0000","last_unstale":"2024-04-06T09:29:25.742313+0000","last_undegraded":"2024-04-06T09:29:25.742313+0000","last_fullsized":"2024-04-06T09:29:25.742313+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T09:29:24.717518+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T09:29:24.717518+0000","last_clean_scrub_stamp":"2024-04-06T09:29:24.717518+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T10:00:30.760505+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":9,"reported_epoch":14,"state":"active+clean","last_fresh":"2024-04-06T09:29:25.742369+0000","last_change":"2024-04-06T09:29:25.738960+0000","last_active":"2024-04-06T09:29:25.742369+0000","last_peered":"2024-04-06T09:29:25.742369+0000","last_clean":"2024-04-06T09:29:25.742369+0000","last_became_active":"2024-04-06T09:29:25.738591+0000","last_became_peered":"2024-04-06T09:29:25.738591+0000","last_unstale":"2024-04-06T09:29:25.742369+0000","last_undegraded":"2024-04-06T09:29:25.742369+0000","last_fullsized":"2024-04-06T09:29:25.742369+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T09:29:24.717518+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T09:29:24.717518+0000","last_clean_scrub_stamp":"2024-04-06T09:29:24.717518+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T14:16:54.527503+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":9,"reported_epoch":14,"state":"active+clean","last_fresh":"2024-04-06T09:29:25.742302+0000","last_change":"2024-04-06T09:29:25.738138+0000","last_active":"2024-04-06T09:29:25.742302+0000","last_peered":"2024-04-06T09:29:25.742302+0000","last_clean":"2024-04-06T09:29:25.742302+0000","last_became_active":"2024-04-06T09:29:25.737263+0000","last_became_peered":"2024-04-06T09:29:25.737263+0000","last_unstale":"2024-04-06T09:29:25.742302+0000","last_undegraded":"2024-04-06T09:29:25.742302+0000","last_fullsized":"2024-04-06T09:29:25.742302+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T09:29:24.717518+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T09:29:24.717518+0000","last_clean_scrub_stamp":"2024-04-06T09:29:24.717518+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T14:30:58.846520+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":9,"reported_epoch":14,"state":"active+clean","last_fresh":"2024-04-06T09:29:25.742147+0000","last_change":"2024-04-06T09:29:25.739673+0000","last_active":"2024-04-06T09:29:25.742147+0000","last_peered":"2024-04-06T09:29:25.742147+0000","last_clean":"2024-04-06T09:29:25.742147+0000","last_became_active":"2024-04-06T09:29:25.738678+0000","last_became_peered":"2024-04-06T09:29:25.738678+0000","last_unstale":"2024-04-06T09:29:25.742147+0000","last_undegraded":"2024-04-06T09:29:25.742147+0000","last_fullsized":"2024-04-06T09:29:25.742147+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T09:29:24.717518+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T09:29:24.717518+0000","last_clean_scrub_stamp":"2024-04-06T09:29:24.717518+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T12:22:31.093977+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":10,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-04-06T09:29:26.730364+0000","last_change":"2024-04-06T09:29:25.739006+0000","last_active":"2024-04-06T09:29:26.730364+0000","last_peered":"2024-04-06T09:29:26.730364+0000","last_clean":"2024-04-06T09:29:26.730364+0000","last_became_active":"2024-04-06T09:29:25.738605+0000","last_became_peered":"2024-04-06T09:29:25.738605+0000","last_unstale":"2024-04-06T09:29:26.730364+0000","last_undegraded":"2024-04-06T09:29:26.730364+0000","last_fullsized":"2024-04-06T09:29:26.730364+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T09:29:24.717518+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T09:29:24.717518+0000","last_clean_scrub_stamp":"2024-04-06T09:29:24.717518+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T20:26:25.794984+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":10,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-04-06T09:29:26.730364+0000","last_change":"2024-04-06T09:29:25.739486+0000","last_active":"2024-04-06T09:29:26.730364+0000","last_peered":"2024-04-06T09:29:26.730364+0000","last_clean":"2024-04-06T09:29:26.730364+0000","last_became_active":"2024-04-06T09:29:25.738108+0000","last_became_peered":"2024-04-06T09:29:25.738108+0000","last_unstale":"2024-04-06T09:29:26.730364+0000","last_undegraded":"2024-04-06T09:29:26.730364+0000","last_fullsized":"2024-04-06T09:29:26.730364+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T09:29:24.717518+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T09:29:24.717518+0000","last_clean_scrub_stamp":"2024-04-06T09:29:24.717518+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T18:32:48.633225+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":9,"reported_epoch":14,"state":"active+clean","last_fresh":"2024-04-06T09:29:25.742268+0000","last_change":"2024-04-06T09:29:25.737897+0000","last_active":"2024-04-06T09:29:25.742268+0000","last_peered":"2024-04-06T09:29:25.742268+0000","last_clean":"2024-04-06T09:29:25.742268+0000","last_became_active":"2024-04-06T09:29:25.737369+0000","last_became_peered":"2024-04-06T09:29:25.737369+0000","last_unstale":"2024-04-06T09:29:25.742268+0000","last_undegraded":"2024-04-06T09:29:25.742268+0000","last_fullsized":"2024-04-06T09:29:25.742268+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T09:29:24.717518+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T09:29:24.717518+0000","last_clean_scrub_stamp":"2024-04-06T09:29:24.717518+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T16:57:27.614162+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"12'97","reported_seq":139,"reported_epoch":13,"state":"active+clean","last_fresh":"2024-04-06T09:29:24.730761+0000","last_change":"2024-04-06T09:29:21.046364+0000","last_active":"2024-04-06T09:29:24.730761+0000","last_peered":"2024-04-06T09:29:24.730761+0000","last_clean":"2024-04-06T09:29:24.730761+0000","last_became_active":"2024-04-06T09:29:21.043913+0000","last_became_peered":"2024-04-06T09:29:21.043913+0000","last_unstale":"2024-04-06T09:29:24.730761+0000","last_undegraded":"2024-04-06T09:29:24.730761+0000","last_fullsized":"2024-04-06T09:29:24.730761+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T09:29:19.704883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T09:29:19.704883+0000","last_clean_scrub_stamp":"2024-04-06T09:29:19.704883+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T14:04:37.575084+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":122880,"data_stored":1180736,"data_compressed":60676,"data_compressed_allocated":81920,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738373,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27140,"kb_used_data":120,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344700,"statfs":{"total":96636764160,"available":96608972800,"internally_reserved":0,"allocated":122880,"data_stored":33720,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.64500000000000002}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.47799999999999998}]}]},{"osd":1,"up_from":8,"seq":34359738372,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27216,"kb_used_data":180,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344624,"statfs":{"total":96636764160,"available":96608894976,"internally_reserved":0,"allocated":184320,"data_stored":624088,"data_compressed":30338,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.621}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.70499999999999996}]}]},{"osd":0,"up_from":8,"seq":34359738372,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27200,"kb_used_data":180,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344640,"statfs":{"total":96636764160,"available":96608911360,"internally_reserved":0,"allocated":184320,"data_stored":624088,"data_compressed":30338,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.60999999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.71699999999999997}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":590368,"data_compressed":30338,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":590368,"data_compressed":30338,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-06T09:29:29.240 INFO:tasks.ceph.ceph_manager.cluster1:clean! 2024-04-06T09:29:29.240 INFO:tasks.ceph:Waiting until ceph cluster cluster1 is healthy... 2024-04-06T09:29:29.240 INFO:tasks.ceph.ceph_manager.cluster1:wait_until_healthy 2024-04-06T09:29:29.240 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 health --format=json 2024-04-06T09:29:29.563 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-04-06T09:29:29.564 INFO:teuthology.orchestra.run.smithi006.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-04-06T09:29:29.581 INFO:tasks.ceph.ceph_manager.cluster1:wait_until_healthy done 2024-04-06T09:29:29.581 INFO:teuthology.run_tasks:Running task ceph... 2024-04-06T09:29:29.590 INFO:tasks.ceph:Creating ceph cluster cluster2... 2024-04-06T09:29:29.590 INFO:tasks.ceph:config {'conf': {'client': {'debug rbd': 20, 'debug rbd_mirror': 15, 'log to stderr': False, 'rbd default features': 125, 'rbd mirror image policy type': 'none'}, 'client.mirror.0': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.1': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.2': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.3': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'global': {'mon client directed command retry': 5, 'mon warn on pool no app': False, 'ms inject socket failures': 5000}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'bluestore block size': 96636764160, 'bluestore compression algorithm': 'snappy', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME'], 'cpu_profile': set(), 'cluster': 'cluster2', 'mon_bind_msgr2': True, 'mon_bind_addrvec': True} 2024-04-06T09:29:29.590 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/teuthology-2024-04-02_22:16:03-rbd-reef-distro-default-smithi/7637502', 'branch': 'reef', 'description': 'rbd/mirror-thrash/{base/install clients/mirror cluster/{2-node openstack} conf/{disable-pool-app} msgr-failures/few objectstore/bluestore-comp-snappy policy/none rbd-mirror/four-per-cluster supported-random-distro$/{ubuntu_latest} workloads/rbd-mirror-journal-stress-workunit}', 'email': 'ceph-qa@ceph.com', 'first_in_suite': False, 'job_id': '7637502', 'kernel': {'kdb': True, 'sha1': 'distro'}, 'ktype': 'distro', 'last_in_suite': False, 'machine_type': 'smithi', 'meta': [{'desc': 'run two ceph clusters and install rbd-mirror'}, {'desc': 'configure the permissions for client.mirror'}, {'desc': '2 ceph clusters with 1 mon and 3 osds each'}, {'desc': 'run four rbd-mirror daemons per cluster'}, {'desc': 'run the rbd_mirror_stress.sh workunit to test the rbd-mirror daemon'}], 'name': 'teuthology-2024-04-02_22:16:03-rbd-reef-distro-default-smithi', 'no_nested_subset': False, 'nuke-on-error': True, 'openstack': [{'volumes': {'count': 3, 'size': 30}}], 'os_type': 'ubuntu', 'os_version': '22.04', 'overrides': {'admin_socket': {'branch': 'reef'}, 'ceph': {'conf': {'client': {'debug rbd': 20, 'debug rbd_mirror': 15, 'log to stderr': False, 'rbd default features': 125, 'rbd mirror image policy type': 'none'}, 'client.mirror.0': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.1': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.2': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.3': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'global': {'mon client directed command retry': 5, 'mon warn on pool no app': False, 'ms inject socket failures': 5000}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'bluestore block size': 96636764160, 'bluestore compression algorithm': 'snappy', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME'], 'sha1': '4d05f4cad464de0fcdd11a2856643e533aea34e8'}, 'ceph-deploy': {'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {}}}, 'install': {'ceph': {'flavor': 'default', 'sha1': '4d05f4cad464de0fcdd11a2856643e533aea34e8'}}, 'thrashosds': {'bdev_inject_crash': 2, 'bdev_inject_crash_probability': 0.5}, 'workunit': {'branch': 'reef', 'sha1': '052adcadd0534eddb0d73c5afbc083a84cd9dc46'}}, 'owner': 'scheduled_teuthology@teuthology', 'priority': 930, 'repo': 'https://git.ceph.com/ceph.git', 'roles': [['cluster1.mon.a', 'cluster1.mgr.x', 'cluster2.mgr.x', 'cluster1.osd.0', 'cluster1.osd.1', 'cluster1.osd.2', 'cluster1.client.0', 'cluster2.client.0'], ['cluster2.mon.a', 'cluster2.osd.0', 'cluster2.osd.1', 'cluster2.osd.2', 'cluster1.client.mirror', 'cluster1.client.mirror.0', 'cluster1.client.mirror.1', 'cluster1.client.mirror.2', 'cluster1.client.mirror.3', 'cluster1.client.mirror.4', 'cluster1.client.mirror.5', 'cluster1.client.mirror.6', 'cluster2.client.mirror', 'cluster2.client.mirror.0', 'cluster2.client.mirror.1', 'cluster2.client.mirror.2', 'cluster2.client.mirror.3', 'cluster2.client.mirror.4', 'cluster2.client.mirror.5', 'cluster2.client.mirror.6']], 'seed': 9400, 'sha1': '4d05f4cad464de0fcdd11a2856643e533aea34e8', 'sleep_before_teardown': 0, 'subset': '12/128', 'suite': 'rbd', 'suite_branch': 'reef', 'suite_path': '/home/teuthworker/src/git.ceph.com_ceph_052adcadd0534eddb0d73c5afbc083a84cd9dc46/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://git.ceph.com/ceph.git', 'suite_sha1': '052adcadd0534eddb0d73c5afbc083a84cd9dc46', 'targets': {'smithi006.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBPRsjjIxbbsh/IkYvxkow7Ob8LWlTfeg943sr/m1r913UrvId+M7tk4dUHrqVGT1fibXJxU9TjyR75QZn7tVsG0=', 'smithi192.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBFwM+L3rHFCaXoKHuY/3HZtsZsr61XOLlFJZtgkQKNRHUQSe/PZxCqw1jcy0QQ7XEb3wk4LbI1nEnCQwkpC2S8M='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'kdb': True, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': {'extra_packages': ['rbd-mirror'], 'flavor': 'default', 'sha1': '4d05f4cad464de0fcdd11a2856643e533aea34e8'}}, {'ceph': {'cluster': 'cluster1', 'conf': {'client': {'debug rbd': 20, 'debug rbd_mirror': 15, 'log to stderr': False, 'rbd default features': 125, 'rbd mirror image policy type': 'none'}, 'client.mirror.0': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.1': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.2': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.3': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'global': {'mon client directed command retry': 5, 'mon warn on pool no app': False, 'ms inject socket failures': 5000}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'bluestore block size': 96636764160, 'bluestore compression algorithm': 'snappy', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME'], 'sha1': '4d05f4cad464de0fcdd11a2856643e533aea34e8'}}, {'ceph': {'cluster': 'cluster2', 'conf': {'client': {'debug rbd': 20, 'debug rbd_mirror': 15, 'log to stderr': False, 'rbd default features': 125, 'rbd mirror image policy type': 'none'}, 'client.mirror.0': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.1': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.2': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.3': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'global': {'mon client directed command retry': 5, 'mon warn on pool no app': False, 'ms inject socket failures': 5000}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'bluestore block size': 96636764160, 'bluestore compression algorithm': 'snappy', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME'], 'sha1': '4d05f4cad464de0fcdd11a2856643e533aea34e8'}}, {'exec': {'cluster1.client.mirror.0': ["sudo ceph --cluster cluster1 auth caps client.mirror mon 'profile rbd-mirror-peer' osd 'profile rbd'", "sudo ceph --cluster cluster1 auth caps client.mirror.0 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster1 auth caps client.mirror.1 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster1 auth caps client.mirror.2 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster1 auth caps client.mirror.3 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster2 auth caps client.mirror mon 'profile rbd-mirror-peer' osd 'profile rbd'", "sudo ceph --cluster cluster2 auth caps client.mirror.0 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster2 auth caps client.mirror.1 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster2 auth caps client.mirror.2 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster2 auth caps client.mirror.3 mon 'profile rbd-mirror' osd 'profile rbd'"]}}, {'rbd-mirror': {'client': 'cluster1.client.mirror.0', 'thrash': True}}, {'rbd-mirror': {'client': 'cluster1.client.mirror.1', 'thrash': True}}, {'rbd-mirror': {'client': 'cluster1.client.mirror.2', 'thrash': True}}, {'rbd-mirror': {'client': 'cluster1.client.mirror.3', 'thrash': True}}, {'rbd-mirror': {'client': 'cluster2.client.mirror.0', 'thrash': True}}, {'rbd-mirror': {'client': 'cluster2.client.mirror.1', 'thrash': True}}, {'rbd-mirror': {'client': 'cluster2.client.mirror.2', 'thrash': True}}, {'rbd-mirror': {'client': 'cluster2.client.mirror.3', 'thrash': True}}, {'rbd-mirror-thrash': {'cluster': 'cluster1'}}, {'rbd-mirror-thrash': {'cluster': 'cluster2'}}, {'workunit': {'clients': {'cluster1.client.mirror': ['rbd/rbd_mirror_stress.sh']}, 'env': {'CEPH_ARGS': '', 'MIRROR_IMAGE_MODE': 'journal', 'MIRROR_POOL_MODE': 'pool', 'RBD_MIRROR_INSTANCES': '4', 'RBD_MIRROR_USE_EXISTING_CLUSTER': '1', 'RBD_MIRROR_USE_RBD_MIRROR': '1'}, 'timeout': '6h'}}], 'teuthology': {'fragments_dropped': [], 'meta': {}, 'postmerge': []}, 'teuthology_branch': 'main', 'teuthology_sha1': 'e691533f9cbb33d85b2187bba20d7102f098636d', 'timestamp': '2024-04-02_22:16:03', 'tube': 'smithi', 'user': 'teuthology', 'verbose': False, 'worker_log': '/home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885'} 2024-04-06T09:29:29.591 DEBUG:teuthology.orchestra.run.smithi006:> install -d -m0755 -- /home/ubuntu/cephtest/cluster2.data 2024-04-06T09:29:29.593 DEBUG:teuthology.orchestra.run.smithi192:> install -d -m0755 -- /home/ubuntu/cephtest/cluster2.data 2024-04-06T09:29:29.600 DEBUG:teuthology.orchestra.run.smithi006:> sudo install -d -m0777 -- /var/run/ceph 2024-04-06T09:29:29.639 DEBUG:teuthology.orchestra.run.smithi192:> sudo install -d -m0777 -- /var/run/ceph 2024-04-06T09:29:29.656 DEBUG:teuthology.orchestra.run.smithi192:> set -ex 2024-04-06T09:29:29.656 DEBUG:teuthology.orchestra.run.smithi192:> dd if=/scratch_devs of=/dev/stdout 2024-04-06T09:29:29.703 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-04-06T09:29:29.703 DEBUG:teuthology.orchestra.run.smithi192:> stat /dev/vg_nvme/lv_1 2024-04-06T09:29:29.753 INFO:teuthology.orchestra.run.smithi192.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-04-06T09:29:29.753 INFO:teuthology.orchestra.run.smithi192.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-06T09:29:29.753 INFO:teuthology.orchestra.run.smithi192.stdout:Device: 5h/5d Inode: 881 Links: 1 2024-04-06T09:29:29.753 INFO:teuthology.orchestra.run.smithi192.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-06T09:29:29.754 INFO:teuthology.orchestra.run.smithi192.stdout:Access: 2024-04-06 09:23:39.024534835 +0000 2024-04-06T09:29:29.754 INFO:teuthology.orchestra.run.smithi192.stdout:Modify: 2024-04-06 09:23:38.760537194 +0000 2024-04-06T09:29:29.754 INFO:teuthology.orchestra.run.smithi192.stdout:Change: 2024-04-06 09:23:38.760537194 +0000 2024-04-06T09:29:29.754 INFO:teuthology.orchestra.run.smithi192.stdout: Birth: - 2024-04-06T09:29:29.754 DEBUG:teuthology.orchestra.run.smithi192:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-04-06T09:29:29.805 INFO:teuthology.orchestra.run.smithi192.stderr:1+0 records in 2024-04-06T09:29:29.805 INFO:teuthology.orchestra.run.smithi192.stderr:1+0 records out 2024-04-06T09:29:29.806 INFO:teuthology.orchestra.run.smithi192.stderr:512 bytes copied, 0.000446753 s, 1.1 MB/s 2024-04-06T09:29:29.807 DEBUG:teuthology.orchestra.run.smithi192:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-04-06T09:29:29.856 DEBUG:teuthology.orchestra.run.smithi192:> stat /dev/vg_nvme/lv_2 2024-04-06T09:29:29.903 INFO:teuthology.orchestra.run.smithi192.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-04-06T09:29:29.903 INFO:teuthology.orchestra.run.smithi192.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-06T09:29:29.903 INFO:teuthology.orchestra.run.smithi192.stdout:Device: 5h/5d Inode: 901 Links: 1 2024-04-06T09:29:29.903 INFO:teuthology.orchestra.run.smithi192.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-06T09:29:29.903 INFO:teuthology.orchestra.run.smithi192.stdout:Access: 2024-04-06 09:23:39.504530548 +0000 2024-04-06T09:29:29.903 INFO:teuthology.orchestra.run.smithi192.stdout:Modify: 2024-04-06 09:23:39.244532871 +0000 2024-04-06T09:29:29.903 INFO:teuthology.orchestra.run.smithi192.stdout:Change: 2024-04-06 09:23:39.244532871 +0000 2024-04-06T09:29:29.903 INFO:teuthology.orchestra.run.smithi192.stdout: Birth: - 2024-04-06T09:29:29.903 DEBUG:teuthology.orchestra.run.smithi192:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-04-06T09:29:29.956 INFO:teuthology.orchestra.run.smithi192.stderr:1+0 records in 2024-04-06T09:29:29.957 INFO:teuthology.orchestra.run.smithi192.stderr:1+0 records out 2024-04-06T09:29:29.957 INFO:teuthology.orchestra.run.smithi192.stderr:512 bytes copied, 0.000420702 s, 1.2 MB/s 2024-04-06T09:29:29.962 DEBUG:teuthology.orchestra.run.smithi192:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-04-06T09:29:30.008 DEBUG:teuthology.orchestra.run.smithi192:> stat /dev/vg_nvme/lv_3 2024-04-06T09:29:30.055 INFO:teuthology.orchestra.run.smithi192.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-04-06T09:29:30.055 INFO:teuthology.orchestra.run.smithi192.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-06T09:29:30.055 INFO:teuthology.orchestra.run.smithi192.stdout:Device: 5h/5d Inode: 919 Links: 1 2024-04-06T09:29:30.055 INFO:teuthology.orchestra.run.smithi192.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-06T09:29:30.055 INFO:teuthology.orchestra.run.smithi192.stdout:Access: 2024-04-06 09:23:40.020525939 +0000 2024-04-06T09:29:30.056 INFO:teuthology.orchestra.run.smithi192.stdout:Modify: 2024-04-06 09:23:39.756528297 +0000 2024-04-06T09:29:30.056 INFO:teuthology.orchestra.run.smithi192.stdout:Change: 2024-04-06 09:23:39.756528297 +0000 2024-04-06T09:29:30.056 INFO:teuthology.orchestra.run.smithi192.stdout: Birth: - 2024-04-06T09:29:30.056 DEBUG:teuthology.orchestra.run.smithi192:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-04-06T09:29:30.109 INFO:teuthology.orchestra.run.smithi192.stderr:1+0 records in 2024-04-06T09:29:30.109 INFO:teuthology.orchestra.run.smithi192.stderr:1+0 records out 2024-04-06T09:29:30.109 INFO:teuthology.orchestra.run.smithi192.stderr:512 bytes copied, 0.000329999 s, 1.6 MB/s 2024-04-06T09:29:30.110 DEBUG:teuthology.orchestra.run.smithi192:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-04-06T09:29:30.156 DEBUG:teuthology.orchestra.run.smithi192:> stat /dev/vg_nvme/lv_4 2024-04-06T09:29:30.203 INFO:teuthology.orchestra.run.smithi192.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-04-06T09:29:30.203 INFO:teuthology.orchestra.run.smithi192.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-06T09:29:30.203 INFO:teuthology.orchestra.run.smithi192.stdout:Device: 5h/5d Inode: 937 Links: 1 2024-04-06T09:29:30.203 INFO:teuthology.orchestra.run.smithi192.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-06T09:29:30.203 INFO:teuthology.orchestra.run.smithi192.stdout:Access: 2024-04-06 09:23:40.524521438 +0000 2024-04-06T09:29:30.204 INFO:teuthology.orchestra.run.smithi192.stdout:Modify: 2024-04-06 09:23:40.280523617 +0000 2024-04-06T09:29:30.204 INFO:teuthology.orchestra.run.smithi192.stdout:Change: 2024-04-06 09:23:40.280523617 +0000 2024-04-06T09:29:30.204 INFO:teuthology.orchestra.run.smithi192.stdout: Birth: - 2024-04-06T09:29:30.204 DEBUG:teuthology.orchestra.run.smithi192:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-04-06T09:29:30.257 INFO:teuthology.orchestra.run.smithi192.stderr:1+0 records in 2024-04-06T09:29:30.257 INFO:teuthology.orchestra.run.smithi192.stderr:1+0 records out 2024-04-06T09:29:30.257 INFO:teuthology.orchestra.run.smithi192.stderr:512 bytes copied, 0.000288896 s, 1.8 MB/s 2024-04-06T09:29:30.258 DEBUG:teuthology.orchestra.run.smithi192:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-04-06T09:29:30.308 INFO:tasks.ceph:osd dev map: {'cluster2.osd.0': '/dev/vg_nvme/lv_1', 'cluster2.osd.1': '/dev/vg_nvme/lv_2', 'cluster2.osd.2': '/dev/vg_nvme/lv_3'} 2024-04-06T09:29:30.308 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi192.front.sepia.ceph.com'): {'cluster2.osd.0': '/dev/vg_nvme/lv_1', 'cluster2.osd.1': '/dev/vg_nvme/lv_2', 'cluster2.osd.2': '/dev/vg_nvme/lv_3'}} 2024-04-06T09:29:30.308 INFO:tasks.ceph:Generating config... 2024-04-06T09:29:30.309 INFO:tasks.ceph:[client] debug rbd = 20 2024-04-06T09:29:30.309 INFO:tasks.ceph:[client] debug rbd_mirror = 15 2024-04-06T09:29:30.309 INFO:tasks.ceph:[client] log to stderr = False 2024-04-06T09:29:30.309 INFO:tasks.ceph:[client] rbd default features = 125 2024-04-06T09:29:30.309 INFO:tasks.ceph:[client] rbd mirror image policy type = none 2024-04-06T09:29:30.310 INFO:tasks.ceph:[client.mirror.0] admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-04-06T09:29:30.310 INFO:tasks.ceph:[client.mirror.0] pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-04-06T09:29:30.310 INFO:tasks.ceph:[client.mirror.1] admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-04-06T09:29:30.310 INFO:tasks.ceph:[client.mirror.1] pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-04-06T09:29:30.310 INFO:tasks.ceph:[client.mirror.2] admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-04-06T09:29:30.310 INFO:tasks.ceph:[client.mirror.2] pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-04-06T09:29:30.310 INFO:tasks.ceph:[client.mirror.3] admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-04-06T09:29:30.310 INFO:tasks.ceph:[client.mirror.3] pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-04-06T09:29:30.310 INFO:tasks.ceph:[global] mon client directed command retry = 5 2024-04-06T09:29:30.310 INFO:tasks.ceph:[global] mon warn on pool no app = False 2024-04-06T09:29:30.310 INFO:tasks.ceph:[global] ms inject socket failures = 5000 2024-04-06T09:29:30.310 INFO:tasks.ceph:[mgr] debug mgr = 20 2024-04-06T09:29:30.310 INFO:tasks.ceph:[mgr] debug ms = 1 2024-04-06T09:29:30.311 INFO:tasks.ceph:[mon] debug mon = 20 2024-04-06T09:29:30.311 INFO:tasks.ceph:[mon] debug ms = 1 2024-04-06T09:29:30.311 INFO:tasks.ceph:[mon] debug paxos = 20 2024-04-06T09:29:30.311 INFO:tasks.ceph:[osd] bluestore block size = 96636764160 2024-04-06T09:29:30.311 INFO:tasks.ceph:[osd] bluestore compression algorithm = snappy 2024-04-06T09:29:30.311 INFO:tasks.ceph:[osd] bluestore compression mode = aggressive 2024-04-06T09:29:30.311 INFO:tasks.ceph:[osd] bluestore fsck on mount = True 2024-04-06T09:29:30.311 INFO:tasks.ceph:[osd] debug bluefs = 1/20 2024-04-06T09:29:30.311 INFO:tasks.ceph:[osd] debug bluestore = 1/20 2024-04-06T09:29:30.311 INFO:tasks.ceph:[osd] debug ms = 1 2024-04-06T09:29:30.311 INFO:tasks.ceph:[osd] debug osd = 20 2024-04-06T09:29:30.311 INFO:tasks.ceph:[osd] debug rocksdb = 4/10 2024-04-06T09:29:30.311 INFO:tasks.ceph:[osd] mon osd backfillfull_ratio = 0.85 2024-04-06T09:29:30.311 INFO:tasks.ceph:[osd] mon osd full ratio = 0.9 2024-04-06T09:29:30.311 INFO:tasks.ceph:[osd] mon osd nearfull ratio = 0.8 2024-04-06T09:29:30.312 INFO:tasks.ceph:[osd] osd failsafe full ratio = 0.95 2024-04-06T09:29:30.312 INFO:tasks.ceph:[osd] osd objectstore = bluestore 2024-04-06T09:29:30.312 INFO:tasks.ceph:Setting up cluster2.mon.a... 2024-04-06T09:29:30.312 DEBUG:teuthology.orchestra.run.smithi192:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/cluster2.keyring 2024-04-06T09:29:30.382 INFO:teuthology.orchestra.run.smithi192.stdout:creating /etc/ceph/cluster2.keyring 2024-04-06T09:29:30.385 DEBUG:teuthology.orchestra.run.smithi192:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/cluster2.keyring 2024-04-06T09:29:30.460 DEBUG:teuthology.orchestra.run.smithi192:> sudo chmod 0644 /etc/ceph/cluster2.keyring 2024-04-06T09:29:30.513 DEBUG:tasks.ceph:Ceph mon addresses: [('cluster2.mon.a', '172.21.15.192')] 2024-04-06T09:29:30.514 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'cephx', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '172.21.15.192', 'mon client directed command retry': 5, 'mon warn on pool no app': False, 'ms inject socket failures': 5000}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': '600', 'osd mclock profile': 'high_recovery_ops', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'bluestore block size': 96636764160, 'bluestore compression algorithm': 'snappy', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false'}, 'mon': {'debug ms': 1, 'debug mon': 20, 'debug paxos': 20, 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false', 'mon_down_mkfs_grace': '2m', 'mon_warn_on_filestore_osds': 'false'}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok', 'debug rbd': 20, 'debug rbd_mirror': 15, 'log to stderr': False, 'rbd default features': 125, 'rbd mirror image policy type': 'none'}, 'mon.a': {}, 'client.mirror.0': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.1': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.2': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.3': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}} 2024-04-06T09:29:30.514 DEBUG:teuthology.orchestra.run.smithi192:> set -ex 2024-04-06T09:29:30.515 DEBUG:teuthology.orchestra.run.smithi192:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2024-04-06T09:29:30.559 DEBUG:teuthology.orchestra.run.smithi192:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --add a 172.21.15.192 --print /home/ubuntu/cephtest/cluster2.monmap 2024-04-06T09:29:30.633 INFO:teuthology.orchestra.run.smithi192.stdout:monmaptool: monmap file /home/ubuntu/cephtest/cluster2.monmap 2024-04-06T09:29:30.634 INFO:teuthology.orchestra.run.smithi192.stdout:monmaptool: generated fsid cb2d68e3-446e-4a71-a214-bfd667a21bca 2024-04-06T09:29:30.634 INFO:teuthology.orchestra.run.smithi192.stdout:setting min_mon_release = pacific 2024-04-06T09:29:30.634 INFO:teuthology.orchestra.run.smithi192.stdout:epoch 0 2024-04-06T09:29:30.634 INFO:teuthology.orchestra.run.smithi192.stdout:fsid cb2d68e3-446e-4a71-a214-bfd667a21bca 2024-04-06T09:29:30.634 INFO:teuthology.orchestra.run.smithi192.stdout:last_changed 2024-04-06T09:29:30.636650+0000 2024-04-06T09:29:30.634 INFO:teuthology.orchestra.run.smithi192.stdout:created 2024-04-06T09:29:30.636650+0000 2024-04-06T09:29:30.634 INFO:teuthology.orchestra.run.smithi192.stdout:min_mon_release 16 (pacific) 2024-04-06T09:29:30.634 INFO:teuthology.orchestra.run.smithi192.stdout:election_strategy: 1 2024-04-06T09:29:30.634 INFO:teuthology.orchestra.run.smithi192.stdout:0: [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] mon.a 2024-04-06T09:29:30.634 INFO:teuthology.orchestra.run.smithi192.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/cluster2.monmap (1 monitors) 2024-04-06T09:29:30.636 DEBUG:teuthology.orchestra.run.smithi192:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2024-04-06T09:29:30.683 INFO:tasks.ceph:Writing /etc/ceph/cluster2.conf for FSID cb2d68e3-446e-4a71-a214-bfd667a21bca... 2024-04-06T09:29:30.684 DEBUG:teuthology.orchestra.run.smithi006:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/cluster2.conf && sudo chmod 0644 /etc/ceph/cluster2.conf > /dev/null 2024-04-06T09:29:30.686 DEBUG:teuthology.orchestra.run.smithi192:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/cluster2.conf && sudo chmod 0644 /etc/ceph/cluster2.conf > /dev/null 2024-04-06T09:29:30.729 INFO:teuthology.orchestra.run.smithi006.stdout:[global] 2024-04-06T09:29:30.729 INFO:teuthology.orchestra.run.smithi006.stdout: chdir = "" 2024-04-06T09:29:30.729 INFO:teuthology.orchestra.run.smithi006.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-04-06T09:29:30.729 INFO:teuthology.orchestra.run.smithi006.stdout: auth supported = cephx 2024-04-06T09:29:30.729 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-04-06T09:29:30.729 INFO:teuthology.orchestra.run.smithi006.stdout: filestore xattr use omap = true 2024-04-06T09:29:30.730 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-04-06T09:29:30.730 INFO:teuthology.orchestra.run.smithi006.stdout: mon clock drift allowed = 1.000 2024-04-06T09:29:30.730 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-04-06T09:29:30.730 INFO:teuthology.orchestra.run.smithi006.stdout: osd crush chooseleaf type = 0 2024-04-06T09:29:30.730 INFO:teuthology.orchestra.run.smithi006.stdout: auth debug = true 2024-04-06T09:29:30.730 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-04-06T09:29:30.730 INFO:teuthology.orchestra.run.smithi006.stdout: ms die on old message = true 2024-04-06T09:29:30.730 INFO:teuthology.orchestra.run.smithi006.stdout: ms die on bug = true 2024-04-06T09:29:30.730 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-04-06T09:29:30.730 INFO:teuthology.orchestra.run.smithi006.stdout: mon max pg per osd = 10000 # >= luminous 2024-04-06T09:29:30.730 INFO:teuthology.orchestra.run.smithi006.stdout: mon pg warn max object skew = 0 2024-04-06T09:29:30.730 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-04-06T09:29:30.730 INFO:teuthology.orchestra.run.smithi006.stdout: # disable pg_autoscaler by default for new pools 2024-04-06T09:29:30.730 INFO:teuthology.orchestra.run.smithi006.stdout: osd_pool_default_pg_autoscale_mode = off 2024-04-06T09:29:30.730 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-04-06T09:29:30.731 INFO:teuthology.orchestra.run.smithi006.stdout: osd pool default size = 2 2024-04-06T09:29:30.731 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-04-06T09:29:30.731 INFO:teuthology.orchestra.run.smithi006.stdout: mon osd allow primary affinity = true 2024-04-06T09:29:30.731 INFO:teuthology.orchestra.run.smithi006.stdout: mon osd allow pg remap = true 2024-04-06T09:29:30.731 INFO:teuthology.orchestra.run.smithi006.stdout: mon warn on legacy crush tunables = false 2024-04-06T09:29:30.731 INFO:teuthology.orchestra.run.smithi006.stdout: mon warn on crush straw calc version zero = false 2024-04-06T09:29:30.731 INFO:teuthology.orchestra.run.smithi006.stdout: mon warn on no sortbitwise = false 2024-04-06T09:29:30.731 INFO:teuthology.orchestra.run.smithi006.stdout: mon warn on osd down out interval zero = false 2024-04-06T09:29:30.731 INFO:teuthology.orchestra.run.smithi006.stdout: mon warn on too few osds = false 2024-04-06T09:29:30.731 INFO:teuthology.orchestra.run.smithi006.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-04-06T09:29:30.731 INFO:teuthology.orchestra.run.smithi006.stdout: mon_warn_on_pool_no_redundancy = false 2024-04-06T09:29:30.731 INFO:teuthology.orchestra.run.smithi006.stdout: mon_allow_pool_size_one = true 2024-04-06T09:29:30.731 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-04-06T09:29:30.731 INFO:teuthology.orchestra.run.smithi006.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-04-06T09:29:30.732 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-04-06T09:29:30.732 INFO:teuthology.orchestra.run.smithi006.stdout: osd default data pool replay window = 5 2024-04-06T09:29:30.732 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-04-06T09:29:30.732 INFO:teuthology.orchestra.run.smithi006.stdout: mon allow pool delete = true 2024-04-06T09:29:30.732 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-04-06T09:29:30.732 INFO:teuthology.orchestra.run.smithi006.stdout: mon cluster log file level = debug 2024-04-06T09:29:30.732 INFO:teuthology.orchestra.run.smithi006.stdout: debug asserts on shutdown = true 2024-04-06T09:29:30.732 INFO:teuthology.orchestra.run.smithi006.stdout: mon health detail to clog = false 2024-04-06T09:29:30.732 INFO:teuthology.orchestra.run.smithi006.stdout: mon host = 172.21.15.192 2024-04-06T09:29:30.732 INFO:teuthology.orchestra.run.smithi006.stdout: mon client directed command retry = 5 2024-04-06T09:29:30.732 INFO:teuthology.orchestra.run.smithi006.stdout: mon warn on pool no app = False 2024-04-06T09:29:30.732 INFO:teuthology.orchestra.run.smithi006.stdout: ms inject socket failures = 5000 2024-04-06T09:29:30.732 INFO:teuthology.orchestra.run.smithi006.stdout: fsid = cb2d68e3-446e-4a71-a214-bfd667a21bca 2024-04-06T09:29:30.732 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-04-06T09:29:30.732 INFO:teuthology.orchestra.run.smithi006.stdout:[osd] 2024-04-06T09:29:30.732 INFO:teuthology.orchestra.run.smithi006.stdout: osd journal size = 100 2024-04-06T09:29:30.733 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-04-06T09:29:30.733 INFO:teuthology.orchestra.run.smithi006.stdout: osd scrub load threshold = 5.0 2024-04-06T09:29:30.733 INFO:teuthology.orchestra.run.smithi006.stdout: osd scrub max interval = 600 2024-04-06T09:29:30.733 INFO:teuthology.orchestra.run.smithi006.stdout: osd mclock profile = high_recovery_ops 2024-04-06T09:29:30.733 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-04-06T09:29:30.733 INFO:teuthology.orchestra.run.smithi006.stdout: osd recover clone overlap = true 2024-04-06T09:29:30.733 INFO:teuthology.orchestra.run.smithi006.stdout: osd recovery max chunk = 1048576 2024-04-06T09:29:30.733 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-04-06T09:29:30.733 INFO:teuthology.orchestra.run.smithi006.stdout: osd debug shutdown = true 2024-04-06T09:29:30.733 INFO:teuthology.orchestra.run.smithi006.stdout: osd debug op order = true 2024-04-06T09:29:30.733 INFO:teuthology.orchestra.run.smithi006.stdout: osd debug verify stray on activate = true 2024-04-06T09:29:30.733 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-04-06T09:29:30.733 INFO:teuthology.orchestra.run.smithi006.stdout: osd open classes on start = true 2024-04-06T09:29:30.733 INFO:teuthology.orchestra.run.smithi006.stdout: osd debug pg log writeout = true 2024-04-06T09:29:30.734 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-04-06T09:29:30.734 INFO:teuthology.orchestra.run.smithi006.stdout: osd deep scrub update digest min age = 30 2024-04-06T09:29:30.734 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-04-06T09:29:30.734 INFO:teuthology.orchestra.run.smithi006.stdout: osd map max advance = 10 2024-04-06T09:29:30.734 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-04-06T09:29:30.734 INFO:teuthology.orchestra.run.smithi006.stdout: journal zero on create = true 2024-04-06T09:29:30.734 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-04-06T09:29:30.734 INFO:teuthology.orchestra.run.smithi006.stdout: filestore ondisk finisher threads = 3 2024-04-06T09:29:30.734 INFO:teuthology.orchestra.run.smithi006.stdout: filestore apply finisher threads = 3 2024-04-06T09:29:30.734 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-04-06T09:29:30.734 INFO:teuthology.orchestra.run.smithi006.stdout: bdev debug aio = true 2024-04-06T09:29:30.734 INFO:teuthology.orchestra.run.smithi006.stdout: osd debug misdirected ops = true 2024-04-06T09:29:30.734 INFO:teuthology.orchestra.run.smithi006.stdout: bluestore block size = 96636764160 2024-04-06T09:29:30.734 INFO:teuthology.orchestra.run.smithi006.stdout: bluestore compression algorithm = snappy 2024-04-06T09:29:30.735 INFO:teuthology.orchestra.run.smithi006.stdout: bluestore compression mode = aggressive 2024-04-06T09:29:30.735 INFO:teuthology.orchestra.run.smithi006.stdout: bluestore fsck on mount = True 2024-04-06T09:29:30.735 INFO:teuthology.orchestra.run.smithi006.stdout: debug bluefs = 1/20 2024-04-06T09:29:30.735 INFO:teuthology.orchestra.run.smithi006.stdout: debug bluestore = 1/20 2024-04-06T09:29:30.735 INFO:teuthology.orchestra.run.smithi006.stdout: debug ms = 1 2024-04-06T09:29:30.735 INFO:teuthology.orchestra.run.smithi006.stdout: debug osd = 20 2024-04-06T09:29:30.735 INFO:teuthology.orchestra.run.smithi006.stdout: debug rocksdb = 4/10 2024-04-06T09:29:30.735 INFO:teuthology.orchestra.run.smithi006.stdout: mon osd backfillfull_ratio = 0.85 2024-04-06T09:29:30.735 INFO:teuthology.orchestra.run.smithi006.stdout: mon osd full ratio = 0.9 2024-04-06T09:29:30.735 INFO:teuthology.orchestra.run.smithi006.stdout: mon osd nearfull ratio = 0.8 2024-04-06T09:29:30.735 INFO:teuthology.orchestra.run.smithi006.stdout: osd failsafe full ratio = 0.95 2024-04-06T09:29:30.735 INFO:teuthology.orchestra.run.smithi006.stdout: osd objectstore = bluestore 2024-04-06T09:29:30.735 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-04-06T09:29:30.735 INFO:teuthology.orchestra.run.smithi006.stdout:[mgr] 2024-04-06T09:29:30.735 INFO:teuthology.orchestra.run.smithi006.stdout: debug ms = 1 2024-04-06T09:29:30.735 INFO:teuthology.orchestra.run.smithi006.stdout: debug mgr = 20 2024-04-06T09:29:30.736 INFO:teuthology.orchestra.run.smithi006.stdout: debug mon = 20 2024-04-06T09:29:30.736 INFO:teuthology.orchestra.run.smithi006.stdout: debug auth = 20 2024-04-06T09:29:30.736 INFO:teuthology.orchestra.run.smithi006.stdout: mon reweight min pgs per osd = 4 2024-04-06T09:29:30.736 INFO:teuthology.orchestra.run.smithi006.stdout: mon reweight min bytes per osd = 10 2024-04-06T09:29:30.736 INFO:teuthology.orchestra.run.smithi006.stdout: mgr/telemetry/nag = false 2024-04-06T09:29:30.736 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-04-06T09:29:30.736 INFO:teuthology.orchestra.run.smithi006.stdout:[mon] 2024-04-06T09:29:30.736 INFO:teuthology.orchestra.run.smithi006.stdout: debug ms = 1 2024-04-06T09:29:30.736 INFO:teuthology.orchestra.run.smithi006.stdout: debug mon = 20 2024-04-06T09:29:30.736 INFO:teuthology.orchestra.run.smithi006.stdout: debug paxos = 20 2024-04-06T09:29:30.736 INFO:teuthology.orchestra.run.smithi006.stdout: debug auth = 20 2024-04-06T09:29:30.736 INFO:teuthology.orchestra.run.smithi006.stdout: mon data avail warn = 5 2024-04-06T09:29:30.736 INFO:teuthology.orchestra.run.smithi006.stdout: mon mgr mkfs grace = 240 2024-04-06T09:29:30.736 INFO:teuthology.orchestra.run.smithi006.stdout: mon reweight min pgs per osd = 4 2024-04-06T09:29:30.736 INFO:teuthology.orchestra.run.smithi006.stdout: mon osd reporter subtree level = osd 2024-04-06T09:29:30.736 INFO:teuthology.orchestra.run.smithi006.stdout: mon osd prime pg temp = true 2024-04-06T09:29:30.737 INFO:teuthology.orchestra.run.smithi006.stdout: mon reweight min bytes per osd = 10 2024-04-06T09:29:30.737 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-04-06T09:29:30.737 INFO:teuthology.orchestra.run.smithi006.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-04-06T09:29:30.737 INFO:teuthology.orchestra.run.smithi006.stdout: auth mon ticket ttl = 660 # 11m 2024-04-06T09:29:30.737 INFO:teuthology.orchestra.run.smithi006.stdout: auth service ticket ttl = 240 # 4m 2024-04-06T09:29:30.737 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-04-06T09:29:30.737 INFO:teuthology.orchestra.run.smithi006.stdout: # don't complain about insecure global_id in the test suite 2024-04-06T09:29:30.737 INFO:teuthology.orchestra.run.smithi006.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-04-06T09:29:30.737 INFO:teuthology.orchestra.run.smithi006.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-04-06T09:29:30.737 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-04-06T09:29:30.737 INFO:teuthology.orchestra.run.smithi006.stdout: # 1m isn't quite enough 2024-04-06T09:29:30.737 INFO:teuthology.orchestra.run.smithi006.stdout: mon_down_mkfs_grace = 2m 2024-04-06T09:29:30.737 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-04-06T09:29:30.737 INFO:teuthology.orchestra.run.smithi006.stdout: mon_warn_on_filestore_osds = false 2024-04-06T09:29:30.737 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-04-06T09:29:30.738 INFO:teuthology.orchestra.run.smithi006.stdout:[client] 2024-04-06T09:29:30.738 INFO:teuthology.orchestra.run.smithi006.stdout: rgw cache enabled = true 2024-04-06T09:29:30.738 INFO:teuthology.orchestra.run.smithi006.stdout: rgw enable ops log = true 2024-04-06T09:29:30.738 INFO:teuthology.orchestra.run.smithi006.stdout: rgw enable usage log = true 2024-04-06T09:29:30.738 INFO:teuthology.orchestra.run.smithi006.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-04-06T09:29:30.738 INFO:teuthology.orchestra.run.smithi006.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-04-06T09:29:30.738 INFO:teuthology.orchestra.run.smithi006.stdout: debug rbd = 20 2024-04-06T09:29:30.738 INFO:teuthology.orchestra.run.smithi006.stdout: debug rbd_mirror = 15 2024-04-06T09:29:30.738 INFO:teuthology.orchestra.run.smithi006.stdout: log to stderr = False 2024-04-06T09:29:30.738 INFO:teuthology.orchestra.run.smithi006.stdout: rbd default features = 125 2024-04-06T09:29:30.738 INFO:teuthology.orchestra.run.smithi006.stdout: rbd mirror image policy type = none 2024-04-06T09:29:30.738 INFO:teuthology.orchestra.run.smithi006.stdout:[mon.a] 2024-04-06T09:29:30.738 INFO:teuthology.orchestra.run.smithi006.stdout:[client.mirror.0] 2024-04-06T09:29:30.739 INFO:teuthology.orchestra.run.smithi006.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-04-06T09:29:30.739 INFO:teuthology.orchestra.run.smithi006.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-04-06T09:29:30.739 INFO:teuthology.orchestra.run.smithi006.stdout:[client.mirror.1] 2024-04-06T09:29:30.739 INFO:teuthology.orchestra.run.smithi006.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-04-06T09:29:30.739 INFO:teuthology.orchestra.run.smithi006.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-04-06T09:29:30.739 INFO:teuthology.orchestra.run.smithi006.stdout:[client.mirror.2] 2024-04-06T09:29:30.739 INFO:teuthology.orchestra.run.smithi006.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-04-06T09:29:30.739 INFO:teuthology.orchestra.run.smithi006.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-04-06T09:29:30.739 INFO:teuthology.orchestra.run.smithi006.stdout:[client.mirror.3] 2024-04-06T09:29:30.739 INFO:teuthology.orchestra.run.smithi006.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-04-06T09:29:30.739 INFO:teuthology.orchestra.run.smithi006.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-04-06T09:29:30.756 INFO:teuthology.orchestra.run.smithi192.stdout:[global] 2024-04-06T09:29:30.756 INFO:teuthology.orchestra.run.smithi192.stdout: chdir = "" 2024-04-06T09:29:30.757 INFO:teuthology.orchestra.run.smithi192.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-04-06T09:29:30.757 INFO:teuthology.orchestra.run.smithi192.stdout: auth supported = cephx 2024-04-06T09:29:30.757 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-04-06T09:29:30.757 INFO:teuthology.orchestra.run.smithi192.stdout: filestore xattr use omap = true 2024-04-06T09:29:30.757 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-04-06T09:29:30.757 INFO:teuthology.orchestra.run.smithi192.stdout: mon clock drift allowed = 1.000 2024-04-06T09:29:30.757 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-04-06T09:29:30.757 INFO:teuthology.orchestra.run.smithi192.stdout: osd crush chooseleaf type = 0 2024-04-06T09:29:30.757 INFO:teuthology.orchestra.run.smithi192.stdout: auth debug = true 2024-04-06T09:29:30.757 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-04-06T09:29:30.757 INFO:teuthology.orchestra.run.smithi192.stdout: ms die on old message = true 2024-04-06T09:29:30.757 INFO:teuthology.orchestra.run.smithi192.stdout: ms die on bug = true 2024-04-06T09:29:30.757 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-04-06T09:29:30.757 INFO:teuthology.orchestra.run.smithi192.stdout: mon max pg per osd = 10000 # >= luminous 2024-04-06T09:29:30.757 INFO:teuthology.orchestra.run.smithi192.stdout: mon pg warn max object skew = 0 2024-04-06T09:29:30.758 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-04-06T09:29:30.758 INFO:teuthology.orchestra.run.smithi192.stdout: # disable pg_autoscaler by default for new pools 2024-04-06T09:29:30.758 INFO:teuthology.orchestra.run.smithi192.stdout: osd_pool_default_pg_autoscale_mode = off 2024-04-06T09:29:30.758 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-04-06T09:29:30.758 INFO:teuthology.orchestra.run.smithi192.stdout: osd pool default size = 2 2024-04-06T09:29:30.758 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-04-06T09:29:30.758 INFO:teuthology.orchestra.run.smithi192.stdout: mon osd allow primary affinity = true 2024-04-06T09:29:30.758 INFO:teuthology.orchestra.run.smithi192.stdout: mon osd allow pg remap = true 2024-04-06T09:29:30.758 INFO:teuthology.orchestra.run.smithi192.stdout: mon warn on legacy crush tunables = false 2024-04-06T09:29:30.758 INFO:teuthology.orchestra.run.smithi192.stdout: mon warn on crush straw calc version zero = false 2024-04-06T09:29:30.758 INFO:teuthology.orchestra.run.smithi192.stdout: mon warn on no sortbitwise = false 2024-04-06T09:29:30.758 INFO:teuthology.orchestra.run.smithi192.stdout: mon warn on osd down out interval zero = false 2024-04-06T09:29:30.759 INFO:teuthology.orchestra.run.smithi192.stdout: mon warn on too few osds = false 2024-04-06T09:29:30.759 INFO:teuthology.orchestra.run.smithi192.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-04-06T09:29:30.759 INFO:teuthology.orchestra.run.smithi192.stdout: mon_warn_on_pool_no_redundancy = false 2024-04-06T09:29:30.759 INFO:teuthology.orchestra.run.smithi192.stdout: mon_allow_pool_size_one = true 2024-04-06T09:29:30.759 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-04-06T09:29:30.759 INFO:teuthology.orchestra.run.smithi192.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-04-06T09:29:30.759 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-04-06T09:29:30.759 INFO:teuthology.orchestra.run.smithi192.stdout: osd default data pool replay window = 5 2024-04-06T09:29:30.759 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-04-06T09:29:30.759 INFO:teuthology.orchestra.run.smithi192.stdout: mon allow pool delete = true 2024-04-06T09:29:30.760 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-04-06T09:29:30.760 INFO:teuthology.orchestra.run.smithi192.stdout: mon cluster log file level = debug 2024-04-06T09:29:30.760 INFO:teuthology.orchestra.run.smithi192.stdout: debug asserts on shutdown = true 2024-04-06T09:29:30.760 INFO:teuthology.orchestra.run.smithi192.stdout: mon health detail to clog = false 2024-04-06T09:29:30.760 INFO:teuthology.orchestra.run.smithi192.stdout: mon host = 172.21.15.192 2024-04-06T09:29:30.760 INFO:teuthology.orchestra.run.smithi192.stdout: mon client directed command retry = 5 2024-04-06T09:29:30.760 INFO:teuthology.orchestra.run.smithi192.stdout: mon warn on pool no app = False 2024-04-06T09:29:30.760 INFO:teuthology.orchestra.run.smithi192.stdout: ms inject socket failures = 5000 2024-04-06T09:29:30.760 INFO:teuthology.orchestra.run.smithi192.stdout: fsid = cb2d68e3-446e-4a71-a214-bfd667a21bca 2024-04-06T09:29:30.760 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-04-06T09:29:30.760 INFO:teuthology.orchestra.run.smithi192.stdout:[osd] 2024-04-06T09:29:30.760 INFO:teuthology.orchestra.run.smithi192.stdout: osd journal size = 100 2024-04-06T09:29:30.761 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-04-06T09:29:30.761 INFO:teuthology.orchestra.run.smithi192.stdout: osd scrub load threshold = 5.0 2024-04-06T09:29:30.761 INFO:teuthology.orchestra.run.smithi192.stdout: osd scrub max interval = 600 2024-04-06T09:29:30.761 INFO:teuthology.orchestra.run.smithi192.stdout: osd mclock profile = high_recovery_ops 2024-04-06T09:29:30.761 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-04-06T09:29:30.761 INFO:teuthology.orchestra.run.smithi192.stdout: osd recover clone overlap = true 2024-04-06T09:29:30.761 INFO:teuthology.orchestra.run.smithi192.stdout: osd recovery max chunk = 1048576 2024-04-06T09:29:30.761 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-04-06T09:29:30.761 INFO:teuthology.orchestra.run.smithi192.stdout: osd debug shutdown = true 2024-04-06T09:29:30.761 INFO:teuthology.orchestra.run.smithi192.stdout: osd debug op order = true 2024-04-06T09:29:30.761 INFO:teuthology.orchestra.run.smithi192.stdout: osd debug verify stray on activate = true 2024-04-06T09:29:30.761 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-04-06T09:29:30.761 INFO:teuthology.orchestra.run.smithi192.stdout: osd open classes on start = true 2024-04-06T09:29:30.761 INFO:teuthology.orchestra.run.smithi192.stdout: osd debug pg log writeout = true 2024-04-06T09:29:30.762 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-04-06T09:29:30.762 INFO:teuthology.orchestra.run.smithi192.stdout: osd deep scrub update digest min age = 30 2024-04-06T09:29:30.762 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-04-06T09:29:30.762 INFO:teuthology.orchestra.run.smithi192.stdout: osd map max advance = 10 2024-04-06T09:29:30.762 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-04-06T09:29:30.762 INFO:teuthology.orchestra.run.smithi192.stdout: journal zero on create = true 2024-04-06T09:29:30.762 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-04-06T09:29:30.762 INFO:teuthology.orchestra.run.smithi192.stdout: filestore ondisk finisher threads = 3 2024-04-06T09:29:30.762 INFO:teuthology.orchestra.run.smithi192.stdout: filestore apply finisher threads = 3 2024-04-06T09:29:30.762 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-04-06T09:29:30.762 INFO:teuthology.orchestra.run.smithi192.stdout: bdev debug aio = true 2024-04-06T09:29:30.762 INFO:teuthology.orchestra.run.smithi192.stdout: osd debug misdirected ops = true 2024-04-06T09:29:30.762 INFO:teuthology.orchestra.run.smithi192.stdout: bluestore block size = 96636764160 2024-04-06T09:29:30.762 INFO:teuthology.orchestra.run.smithi192.stdout: bluestore compression algorithm = snappy 2024-04-06T09:29:30.763 INFO:teuthology.orchestra.run.smithi192.stdout: bluestore compression mode = aggressive 2024-04-06T09:29:30.763 INFO:teuthology.orchestra.run.smithi192.stdout: bluestore fsck on mount = True 2024-04-06T09:29:30.763 INFO:teuthology.orchestra.run.smithi192.stdout: debug bluefs = 1/20 2024-04-06T09:29:30.763 INFO:teuthology.orchestra.run.smithi192.stdout: debug bluestore = 1/20 2024-04-06T09:29:30.763 INFO:teuthology.orchestra.run.smithi192.stdout: debug ms = 1 2024-04-06T09:29:30.763 INFO:teuthology.orchestra.run.smithi192.stdout: debug osd = 20 2024-04-06T09:29:30.763 INFO:teuthology.orchestra.run.smithi192.stdout: debug rocksdb = 4/10 2024-04-06T09:29:30.763 INFO:teuthology.orchestra.run.smithi192.stdout: mon osd backfillfull_ratio = 0.85 2024-04-06T09:29:30.763 INFO:teuthology.orchestra.run.smithi192.stdout: mon osd full ratio = 0.9 2024-04-06T09:29:30.763 INFO:teuthology.orchestra.run.smithi192.stdout: mon osd nearfull ratio = 0.8 2024-04-06T09:29:30.763 INFO:teuthology.orchestra.run.smithi192.stdout: osd failsafe full ratio = 0.95 2024-04-06T09:29:30.763 INFO:teuthology.orchestra.run.smithi192.stdout: osd objectstore = bluestore 2024-04-06T09:29:30.763 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-04-06T09:29:30.763 INFO:teuthology.orchestra.run.smithi192.stdout:[mgr] 2024-04-06T09:29:30.764 INFO:teuthology.orchestra.run.smithi192.stdout: debug ms = 1 2024-04-06T09:29:30.764 INFO:teuthology.orchestra.run.smithi192.stdout: debug mgr = 20 2024-04-06T09:29:30.764 INFO:teuthology.orchestra.run.smithi192.stdout: debug mon = 20 2024-04-06T09:29:30.764 INFO:teuthology.orchestra.run.smithi192.stdout: debug auth = 20 2024-04-06T09:29:30.764 INFO:teuthology.orchestra.run.smithi192.stdout: mon reweight min pgs per osd = 4 2024-04-06T09:29:30.764 INFO:teuthology.orchestra.run.smithi192.stdout: mon reweight min bytes per osd = 10 2024-04-06T09:29:30.764 INFO:teuthology.orchestra.run.smithi192.stdout: mgr/telemetry/nag = false 2024-04-06T09:29:30.764 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-04-06T09:29:30.764 INFO:teuthology.orchestra.run.smithi192.stdout:[mon] 2024-04-06T09:29:30.764 INFO:teuthology.orchestra.run.smithi192.stdout: debug ms = 1 2024-04-06T09:29:30.764 INFO:teuthology.orchestra.run.smithi192.stdout: debug mon = 20 2024-04-06T09:29:30.764 INFO:teuthology.orchestra.run.smithi192.stdout: debug paxos = 20 2024-04-06T09:29:30.764 INFO:teuthology.orchestra.run.smithi192.stdout: debug auth = 20 2024-04-06T09:29:30.764 INFO:teuthology.orchestra.run.smithi192.stdout: mon data avail warn = 5 2024-04-06T09:29:30.764 INFO:teuthology.orchestra.run.smithi192.stdout: mon mgr mkfs grace = 240 2024-04-06T09:29:30.764 INFO:teuthology.orchestra.run.smithi192.stdout: mon reweight min pgs per osd = 4 2024-04-06T09:29:30.765 INFO:teuthology.orchestra.run.smithi192.stdout: mon osd reporter subtree level = osd 2024-04-06T09:29:30.765 INFO:teuthology.orchestra.run.smithi192.stdout: mon osd prime pg temp = true 2024-04-06T09:29:30.765 INFO:teuthology.orchestra.run.smithi192.stdout: mon reweight min bytes per osd = 10 2024-04-06T09:29:30.765 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-04-06T09:29:30.765 INFO:teuthology.orchestra.run.smithi192.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-04-06T09:29:30.765 INFO:teuthology.orchestra.run.smithi192.stdout: auth mon ticket ttl = 660 # 11m 2024-04-06T09:29:30.765 INFO:teuthology.orchestra.run.smithi192.stdout: auth service ticket ttl = 240 # 4m 2024-04-06T09:29:30.765 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-04-06T09:29:30.765 INFO:teuthology.orchestra.run.smithi192.stdout: # don't complain about insecure global_id in the test suite 2024-04-06T09:29:30.765 INFO:teuthology.orchestra.run.smithi192.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-04-06T09:29:30.765 INFO:teuthology.orchestra.run.smithi192.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-04-06T09:29:30.765 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-04-06T09:29:30.765 INFO:teuthology.orchestra.run.smithi192.stdout: # 1m isn't quite enough 2024-04-06T09:29:30.765 INFO:teuthology.orchestra.run.smithi192.stdout: mon_down_mkfs_grace = 2m 2024-04-06T09:29:30.765 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-04-06T09:29:30.765 INFO:teuthology.orchestra.run.smithi192.stdout: mon_warn_on_filestore_osds = false 2024-04-06T09:29:30.766 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-04-06T09:29:30.766 INFO:teuthology.orchestra.run.smithi192.stdout:[client] 2024-04-06T09:29:30.766 INFO:teuthology.orchestra.run.smithi192.stdout: rgw cache enabled = true 2024-04-06T09:29:30.766 INFO:teuthology.orchestra.run.smithi192.stdout: rgw enable ops log = true 2024-04-06T09:29:30.766 INFO:teuthology.orchestra.run.smithi192.stdout: rgw enable usage log = true 2024-04-06T09:29:30.766 INFO:teuthology.orchestra.run.smithi192.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-04-06T09:29:30.766 INFO:teuthology.orchestra.run.smithi192.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-04-06T09:29:30.766 INFO:teuthology.orchestra.run.smithi192.stdout: debug rbd = 20 2024-04-06T09:29:30.766 INFO:teuthology.orchestra.run.smithi192.stdout: debug rbd_mirror = 15 2024-04-06T09:29:30.766 INFO:teuthology.orchestra.run.smithi192.stdout: log to stderr = False 2024-04-06T09:29:30.766 INFO:teuthology.orchestra.run.smithi192.stdout: rbd default features = 125 2024-04-06T09:29:30.766 INFO:teuthology.orchestra.run.smithi192.stdout: rbd mirror image policy type = none 2024-04-06T09:29:30.766 INFO:teuthology.orchestra.run.smithi192.stdout:[mon.a] 2024-04-06T09:29:30.766 INFO:teuthology.orchestra.run.smithi192.stdout:[client.mirror.0] 2024-04-06T09:29:30.767 INFO:teuthology.orchestra.run.smithi192.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-04-06T09:29:30.767 INFO:teuthology.orchestra.run.smithi192.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-04-06T09:29:30.767 INFO:teuthology.orchestra.run.smithi192.stdout:[client.mirror.1] 2024-04-06T09:29:30.767 INFO:teuthology.orchestra.run.smithi192.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-04-06T09:29:30.767 INFO:teuthology.orchestra.run.smithi192.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-04-06T09:29:30.767 INFO:teuthology.orchestra.run.smithi192.stdout:[client.mirror.2] 2024-04-06T09:29:30.767 INFO:teuthology.orchestra.run.smithi192.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-04-06T09:29:30.767 INFO:teuthology.orchestra.run.smithi192.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-04-06T09:29:30.767 INFO:teuthology.orchestra.run.smithi192.stdout:[client.mirror.3] 2024-04-06T09:29:30.767 INFO:teuthology.orchestra.run.smithi192.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-04-06T09:29:30.767 INFO:teuthology.orchestra.run.smithi192.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-04-06T09:29:30.768 INFO:tasks.ceph:Creating admin key on cluster2.mon.a... 2024-04-06T09:29:30.768 DEBUG:teuthology.orchestra.run.smithi192:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/cluster2.keyring 2024-04-06T09:29:30.856 INFO:tasks.ceph:Copying monmap to all nodes... 2024-04-06T09:29:30.856 DEBUG:teuthology.orchestra.run.smithi192:> set -ex 2024-04-06T09:29:30.856 DEBUG:teuthology.orchestra.run.smithi192:> dd if=/etc/ceph/cluster2.keyring of=/dev/stdout 2024-04-06T09:29:30.861 DEBUG:teuthology.orchestra.run.smithi192:> set -ex 2024-04-06T09:29:30.861 DEBUG:teuthology.orchestra.run.smithi192:> dd if=/home/ubuntu/cephtest/cluster2.monmap of=/dev/stdout 2024-04-06T09:29:30.906 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi006.front.sepia.ceph.com 2024-04-06T09:29:30.907 DEBUG:teuthology.orchestra.run.smithi006:> set -ex 2024-04-06T09:29:30.907 DEBUG:teuthology.orchestra.run.smithi006:> sudo dd of=/etc/ceph/cluster2.keyring 2024-04-06T09:29:30.907 DEBUG:teuthology.orchestra.run.smithi006:> sudo chmod 0644 /etc/ceph/cluster2.keyring 2024-04-06T09:29:30.932 DEBUG:teuthology.orchestra.run.smithi006:> set -ex 2024-04-06T09:29:30.932 DEBUG:teuthology.orchestra.run.smithi006:> dd of=/home/ubuntu/cephtest/cluster2.monmap 2024-04-06T09:29:31.027 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi192.front.sepia.ceph.com 2024-04-06T09:29:31.027 DEBUG:teuthology.orchestra.run.smithi192:> set -ex 2024-04-06T09:29:31.027 DEBUG:teuthology.orchestra.run.smithi192:> sudo dd of=/etc/ceph/cluster2.keyring 2024-04-06T09:29:31.027 DEBUG:teuthology.orchestra.run.smithi192:> sudo chmod 0644 /etc/ceph/cluster2.keyring 2024-04-06T09:29:31.051 DEBUG:teuthology.orchestra.run.smithi192:> set -ex 2024-04-06T09:29:31.051 DEBUG:teuthology.orchestra.run.smithi192:> dd of=/home/ubuntu/cephtest/cluster2.monmap 2024-04-06T09:29:31.100 INFO:tasks.ceph:Setting up mon nodes... 2024-04-06T09:29:31.100 INFO:tasks.ceph:Setting up mgr nodes... 2024-04-06T09:29:31.101 DEBUG:teuthology.orchestra.run.smithi006:> sudo mkdir -p /var/lib/ceph/mgr/cluster2-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/cluster2-x/keyring 2024-04-06T09:29:31.146 INFO:teuthology.orchestra.run.smithi006.stdout:creating /var/lib/ceph/mgr/cluster2-x/keyring 2024-04-06T09:29:31.149 INFO:tasks.ceph:Setting up mds nodes... 2024-04-06T09:29:31.149 INFO:tasks.ceph_client:Setting up client nodes... 2024-04-06T09:29:31.149 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/cluster2.client.0.keyring && sudo chmod 0644 /etc/ceph/cluster2.client.0.keyring 2024-04-06T09:29:31.222 INFO:teuthology.orchestra.run.smithi006.stdout:creating /etc/ceph/cluster2.client.0.keyring 2024-04-06T09:29:31.234 DEBUG:teuthology.orchestra.run.smithi192:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror /etc/ceph/cluster2.client.mirror.keyring && sudo chmod 0644 /etc/ceph/cluster2.client.mirror.keyring 2024-04-06T09:29:31.270 INFO:teuthology.orchestra.run.smithi192.stdout:creating /etc/ceph/cluster2.client.mirror.keyring 2024-04-06T09:29:31.281 DEBUG:teuthology.orchestra.run.smithi192:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror.0 /etc/ceph/cluster2.client.mirror.0.keyring && sudo chmod 0644 /etc/ceph/cluster2.client.mirror.0.keyring 2024-04-06T09:29:31.356 INFO:teuthology.orchestra.run.smithi192.stdout:creating /etc/ceph/cluster2.client.mirror.0.keyring 2024-04-06T09:29:31.365 DEBUG:teuthology.orchestra.run.smithi192:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror.1 /etc/ceph/cluster2.client.mirror.1.keyring && sudo chmod 0644 /etc/ceph/cluster2.client.mirror.1.keyring 2024-04-06T09:29:31.435 INFO:teuthology.orchestra.run.smithi192.stdout:creating /etc/ceph/cluster2.client.mirror.1.keyring 2024-04-06T09:29:31.445 DEBUG:teuthology.orchestra.run.smithi192:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror.2 /etc/ceph/cluster2.client.mirror.2.keyring && sudo chmod 0644 /etc/ceph/cluster2.client.mirror.2.keyring 2024-04-06T09:29:31.514 INFO:teuthology.orchestra.run.smithi192.stdout:creating /etc/ceph/cluster2.client.mirror.2.keyring 2024-04-06T09:29:31.523 DEBUG:teuthology.orchestra.run.smithi192:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror.3 /etc/ceph/cluster2.client.mirror.3.keyring && sudo chmod 0644 /etc/ceph/cluster2.client.mirror.3.keyring 2024-04-06T09:29:31.597 INFO:teuthology.orchestra.run.smithi192.stdout:creating /etc/ceph/cluster2.client.mirror.3.keyring 2024-04-06T09:29:31.608 DEBUG:teuthology.orchestra.run.smithi192:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror.4 /etc/ceph/cluster2.client.mirror.4.keyring && sudo chmod 0644 /etc/ceph/cluster2.client.mirror.4.keyring 2024-04-06T09:29:31.683 INFO:teuthology.orchestra.run.smithi192.stdout:creating /etc/ceph/cluster2.client.mirror.4.keyring 2024-04-06T09:29:31.694 DEBUG:teuthology.orchestra.run.smithi192:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror.5 /etc/ceph/cluster2.client.mirror.5.keyring && sudo chmod 0644 /etc/ceph/cluster2.client.mirror.5.keyring 2024-04-06T09:29:31.764 INFO:teuthology.orchestra.run.smithi192.stdout:creating /etc/ceph/cluster2.client.mirror.5.keyring 2024-04-06T09:29:31.774 DEBUG:teuthology.orchestra.run.smithi192:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror.6 /etc/ceph/cluster2.client.mirror.6.keyring && sudo chmod 0644 /etc/ceph/cluster2.client.mirror.6.keyring 2024-04-06T09:29:31.849 INFO:teuthology.orchestra.run.smithi192.stdout:creating /etc/ceph/cluster2.client.mirror.6.keyring 2024-04-06T09:29:31.859 INFO:tasks.ceph:Running mkfs on osd nodes... 2024-04-06T09:29:31.860 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi006.front.sepia.ceph.com'): {'cluster1.osd.0': '/dev/vg_nvme/lv_1', 'cluster1.osd.1': '/dev/vg_nvme/lv_2', 'cluster1.osd.2': '/dev/vg_nvme/lv_3'}, Remote(name='ubuntu@smithi192.front.sepia.ceph.com'): {'cluster2.osd.0': '/dev/vg_nvme/lv_1', 'cluster2.osd.1': '/dev/vg_nvme/lv_2', 'cluster2.osd.2': '/dev/vg_nvme/lv_3'}} 2024-04-06T09:29:31.860 DEBUG:teuthology.orchestra.run.smithi192:> sudo mkdir -p /var/lib/ceph/osd/cluster2-0 2024-04-06T09:29:31.913 INFO:tasks.ceph:roles_to_devs: {'cluster2.osd.0': '/dev/vg_nvme/lv_1', 'cluster2.osd.1': '/dev/vg_nvme/lv_2', 'cluster2.osd.2': '/dev/vg_nvme/lv_3'} 2024-04-06T09:29:31.913 INFO:tasks.ceph:role: cluster2.osd.0 2024-04-06T09:29:31.913 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi192.front.sepia.ceph.com 2024-04-06T09:29:31.913 DEBUG:teuthology.orchestra.run.smithi192:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-04-06T09:29:31.973 INFO:teuthology.orchestra.run.smithi192.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-04-06T09:29:31.973 INFO:teuthology.orchestra.run.smithi192.stdout: = sectsz=512 attr=2, projid32bit=1 2024-04-06T09:29:31.973 INFO:teuthology.orchestra.run.smithi192.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-04-06T09:29:31.973 INFO:teuthology.orchestra.run.smithi192.stdout: = reflink=1 bigtime=0 inobtcount=0 2024-04-06T09:29:31.973 INFO:teuthology.orchestra.run.smithi192.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-04-06T09:29:31.974 INFO:teuthology.orchestra.run.smithi192.stdout: = sunit=0 swidth=0 blks 2024-04-06T09:29:31.974 INFO:teuthology.orchestra.run.smithi192.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-04-06T09:29:31.974 INFO:teuthology.orchestra.run.smithi192.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-04-06T09:29:31.974 INFO:teuthology.orchestra.run.smithi192.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-04-06T09:29:31.974 INFO:teuthology.orchestra.run.smithi192.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-04-06T09:29:32.429 INFO:teuthology.orchestra.run.smithi192.stdout:Discarding blocks...Done. 2024-04-06T09:29:32.431 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi192.front.sepia.ceph.com -o noatime 2024-04-06T09:29:32.431 DEBUG:teuthology.orchestra.run.smithi192:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/cluster2-0 2024-04-06T09:29:32.521 DEBUG:teuthology.orchestra.run.smithi192:> sudo /sbin/restorecon /var/lib/ceph/osd/cluster2-0 2024-04-06T09:29:32.532 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: /sbin/restorecon: command not found 2024-04-06T09:29:32.533 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T09:29:32.533 DEBUG:teuthology.orchestra.run.smithi192:> sudo mkdir -p /var/lib/ceph/osd/cluster2-1 2024-04-06T09:29:32.586 INFO:tasks.ceph:roles_to_devs: {'cluster2.osd.0': '/dev/vg_nvme/lv_1', 'cluster2.osd.1': '/dev/vg_nvme/lv_2', 'cluster2.osd.2': '/dev/vg_nvme/lv_3'} 2024-04-06T09:29:32.587 INFO:tasks.ceph:role: cluster2.osd.1 2024-04-06T09:29:32.587 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi192.front.sepia.ceph.com 2024-04-06T09:29:32.587 DEBUG:teuthology.orchestra.run.smithi192:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-04-06T09:29:32.651 INFO:teuthology.orchestra.run.smithi192.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-04-06T09:29:32.651 INFO:teuthology.orchestra.run.smithi192.stdout: = sectsz=512 attr=2, projid32bit=1 2024-04-06T09:29:32.651 INFO:teuthology.orchestra.run.smithi192.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-04-06T09:29:32.651 INFO:teuthology.orchestra.run.smithi192.stdout: = reflink=1 bigtime=0 inobtcount=0 2024-04-06T09:29:32.651 INFO:teuthology.orchestra.run.smithi192.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-04-06T09:29:32.651 INFO:teuthology.orchestra.run.smithi192.stdout: = sunit=0 swidth=0 blks 2024-04-06T09:29:32.651 INFO:teuthology.orchestra.run.smithi192.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-04-06T09:29:32.651 INFO:teuthology.orchestra.run.smithi192.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-04-06T09:29:32.651 INFO:teuthology.orchestra.run.smithi192.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-04-06T09:29:32.651 INFO:teuthology.orchestra.run.smithi192.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-04-06T09:29:33.129 INFO:teuthology.orchestra.run.smithi192.stdout:Discarding blocks...Done. 2024-04-06T09:29:33.131 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi192.front.sepia.ceph.com -o noatime 2024-04-06T09:29:33.132 DEBUG:teuthology.orchestra.run.smithi192:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/cluster2-1 2024-04-06T09:29:33.295 DEBUG:teuthology.orchestra.run.smithi192:> sudo /sbin/restorecon /var/lib/ceph/osd/cluster2-1 2024-04-06T09:29:33.308 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: /sbin/restorecon: command not found 2024-04-06T09:29:33.309 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T09:29:33.309 DEBUG:teuthology.orchestra.run.smithi192:> sudo mkdir -p /var/lib/ceph/osd/cluster2-2 2024-04-06T09:29:33.363 INFO:tasks.ceph:roles_to_devs: {'cluster2.osd.0': '/dev/vg_nvme/lv_1', 'cluster2.osd.1': '/dev/vg_nvme/lv_2', 'cluster2.osd.2': '/dev/vg_nvme/lv_3'} 2024-04-06T09:29:33.363 INFO:tasks.ceph:role: cluster2.osd.2 2024-04-06T09:29:33.363 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi192.front.sepia.ceph.com 2024-04-06T09:29:33.363 DEBUG:teuthology.orchestra.run.smithi192:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-04-06T09:29:33.427 INFO:teuthology.orchestra.run.smithi192.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-04-06T09:29:33.427 INFO:teuthology.orchestra.run.smithi192.stdout: = sectsz=512 attr=2, projid32bit=1 2024-04-06T09:29:33.427 INFO:teuthology.orchestra.run.smithi192.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-04-06T09:29:33.427 INFO:teuthology.orchestra.run.smithi192.stdout: = reflink=1 bigtime=0 inobtcount=0 2024-04-06T09:29:33.427 INFO:teuthology.orchestra.run.smithi192.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-04-06T09:29:33.427 INFO:teuthology.orchestra.run.smithi192.stdout: = sunit=0 swidth=0 blks 2024-04-06T09:29:33.427 INFO:teuthology.orchestra.run.smithi192.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-04-06T09:29:33.427 INFO:teuthology.orchestra.run.smithi192.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-04-06T09:29:33.427 INFO:teuthology.orchestra.run.smithi192.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-04-06T09:29:33.428 INFO:teuthology.orchestra.run.smithi192.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-04-06T09:29:33.917 INFO:teuthology.orchestra.run.smithi192.stdout:Discarding blocks...Done. 2024-04-06T09:29:33.919 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi192.front.sepia.ceph.com -o noatime 2024-04-06T09:29:33.919 DEBUG:teuthology.orchestra.run.smithi192:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/cluster2-2 2024-04-06T09:29:34.063 DEBUG:teuthology.orchestra.run.smithi192:> sudo /sbin/restorecon /var/lib/ceph/osd/cluster2-2 2024-04-06T09:29:34.073 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: /sbin/restorecon: command not found 2024-04-06T09:29:34.073 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T09:29:34.074 DEBUG:teuthology.orchestra.run.smithi192:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster cluster2 --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/cluster2.monmap 2024-04-06T09:29:34.195 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-06T09:29:34.194+0000 7f267e2a2640 -1 auth: error reading file: /var/lib/ceph/osd/cluster2-0/keyring: can't open /var/lib/ceph/osd/cluster2-0/keyring: (2) No such file or directory 2024-04-06T09:29:34.196 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-06T09:29:34.194+0000 7f267e2a2640 -1 created new key in keyring /var/lib/ceph/osd/cluster2-0/keyring 2024-04-06T09:29:34.196 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-06T09:29:34.194+0000 7f267e2a2640 -1 bluestore(/var/lib/ceph/osd/cluster2-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/cluster2-0/block: (2) No such file or directory 2024-04-06T09:29:34.196 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-06T09:29:34.194+0000 7f267e2a2640 -1 bluestore(/var/lib/ceph/osd/cluster2-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/cluster2-0/block: (2) No such file or directory 2024-04-06T09:29:34.196 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-06T09:29:34.194+0000 7f267e2a2640 -1 bluestore(/var/lib/ceph/osd/cluster2-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/cluster2-0/block: (2) No such file or directory 2024-04-06T09:29:34.196 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-06T09:29:34.194+0000 7f267e2a2640 -1 bluestore(/var/lib/ceph/osd/cluster2-0) _read_fsid unparsable uuid 2024-04-06T09:29:37.179 DEBUG:teuthology.orchestra.run.smithi192:> sudo chown -R ceph:ceph /var/lib/ceph/osd/cluster2-0 2024-04-06T09:29:37.193 DEBUG:teuthology.orchestra.run.smithi192:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster cluster2 --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/cluster2.monmap 2024-04-06T09:29:37.272 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-06T09:29:37.269+0000 7fcfb6095640 -1 auth: error reading file: /var/lib/ceph/osd/cluster2-1/keyring: can't open /var/lib/ceph/osd/cluster2-1/keyring: (2) No such file or directory 2024-04-06T09:29:37.272 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-06T09:29:37.269+0000 7fcfb6095640 -1 created new key in keyring /var/lib/ceph/osd/cluster2-1/keyring 2024-04-06T09:29:37.272 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-06T09:29:37.269+0000 7fcfb6095640 -1 bluestore(/var/lib/ceph/osd/cluster2-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/cluster2-1/block: (2) No such file or directory 2024-04-06T09:29:37.272 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-06T09:29:37.269+0000 7fcfb6095640 -1 bluestore(/var/lib/ceph/osd/cluster2-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/cluster2-1/block: (2) No such file or directory 2024-04-06T09:29:37.273 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-06T09:29:37.269+0000 7fcfb6095640 -1 bluestore(/var/lib/ceph/osd/cluster2-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/cluster2-1/block: (2) No such file or directory 2024-04-06T09:29:37.273 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-06T09:29:37.269+0000 7fcfb6095640 -1 bluestore(/var/lib/ceph/osd/cluster2-1) _read_fsid unparsable uuid 2024-04-06T09:29:40.251 DEBUG:teuthology.orchestra.run.smithi192:> sudo chown -R ceph:ceph /var/lib/ceph/osd/cluster2-1 2024-04-06T09:29:40.307 DEBUG:teuthology.orchestra.run.smithi192:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster cluster2 --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/cluster2.monmap 2024-04-06T09:29:40.387 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-06T09:29:40.385+0000 7f8fce11b640 -1 auth: error reading file: /var/lib/ceph/osd/cluster2-2/keyring: can't open /var/lib/ceph/osd/cluster2-2/keyring: (2) No such file or directory 2024-04-06T09:29:40.388 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-06T09:29:40.385+0000 7f8fce11b640 -1 created new key in keyring /var/lib/ceph/osd/cluster2-2/keyring 2024-04-06T09:29:40.388 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-06T09:29:40.385+0000 7f8fce11b640 -1 bluestore(/var/lib/ceph/osd/cluster2-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/cluster2-2/block: (2) No such file or directory 2024-04-06T09:29:40.388 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-06T09:29:40.385+0000 7f8fce11b640 -1 bluestore(/var/lib/ceph/osd/cluster2-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/cluster2-2/block: (2) No such file or directory 2024-04-06T09:29:40.388 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-06T09:29:40.385+0000 7f8fce11b640 -1 bluestore(/var/lib/ceph/osd/cluster2-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/cluster2-2/block: (2) No such file or directory 2024-04-06T09:29:40.389 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-06T09:29:40.385+0000 7f8fce11b640 -1 bluestore(/var/lib/ceph/osd/cluster2-2) _read_fsid unparsable uuid 2024-04-06T09:29:43.358 DEBUG:teuthology.orchestra.run.smithi192:> sudo chown -R ceph:ceph /var/lib/ceph/osd/cluster2-2 2024-04-06T09:29:43.411 INFO:tasks.ceph:Reading keys from all nodes... 2024-04-06T09:29:43.411 DEBUG:teuthology.orchestra.run.smithi006:> set -ex 2024-04-06T09:29:43.411 DEBUG:teuthology.orchestra.run.smithi006:> sudo dd if=/var/lib/ceph/mgr/cluster2-x/keyring of=/dev/stdout 2024-04-06T09:29:43.426 DEBUG:teuthology.orchestra.run.smithi192:> set -ex 2024-04-06T09:29:43.426 DEBUG:teuthology.orchestra.run.smithi192:> sudo dd if=/var/lib/ceph/osd/cluster2-0/keyring of=/dev/stdout 2024-04-06T09:29:43.465 DEBUG:teuthology.orchestra.run.smithi192:> set -ex 2024-04-06T09:29:43.466 DEBUG:teuthology.orchestra.run.smithi192:> sudo dd if=/var/lib/ceph/osd/cluster2-1/keyring of=/dev/stdout 2024-04-06T09:29:43.518 DEBUG:teuthology.orchestra.run.smithi192:> set -ex 2024-04-06T09:29:43.518 DEBUG:teuthology.orchestra.run.smithi192:> sudo dd if=/var/lib/ceph/osd/cluster2-2/keyring of=/dev/stdout 2024-04-06T09:29:43.574 DEBUG:teuthology.orchestra.run.smithi006:> set -ex 2024-04-06T09:29:43.574 DEBUG:teuthology.orchestra.run.smithi006:> dd if=/etc/ceph/cluster2.client.0.keyring of=/dev/stdout 2024-04-06T09:29:43.580 DEBUG:teuthology.orchestra.run.smithi192:> set -ex 2024-04-06T09:29:43.580 DEBUG:teuthology.orchestra.run.smithi192:> dd if=/etc/ceph/cluster2.client.mirror.keyring of=/dev/stdout 2024-04-06T09:29:43.619 DEBUG:teuthology.orchestra.run.smithi192:> set -ex 2024-04-06T09:29:43.619 DEBUG:teuthology.orchestra.run.smithi192:> dd if=/etc/ceph/cluster2.client.mirror.0.keyring of=/dev/stdout 2024-04-06T09:29:43.667 DEBUG:teuthology.orchestra.run.smithi192:> set -ex 2024-04-06T09:29:43.667 DEBUG:teuthology.orchestra.run.smithi192:> dd if=/etc/ceph/cluster2.client.mirror.1.keyring of=/dev/stdout 2024-04-06T09:29:43.715 DEBUG:teuthology.orchestra.run.smithi192:> set -ex 2024-04-06T09:29:43.715 DEBUG:teuthology.orchestra.run.smithi192:> dd if=/etc/ceph/cluster2.client.mirror.2.keyring of=/dev/stdout 2024-04-06T09:29:43.763 DEBUG:teuthology.orchestra.run.smithi192:> set -ex 2024-04-06T09:29:43.763 DEBUG:teuthology.orchestra.run.smithi192:> dd if=/etc/ceph/cluster2.client.mirror.3.keyring of=/dev/stdout 2024-04-06T09:29:43.811 DEBUG:teuthology.orchestra.run.smithi192:> set -ex 2024-04-06T09:29:43.811 DEBUG:teuthology.orchestra.run.smithi192:> dd if=/etc/ceph/cluster2.client.mirror.4.keyring of=/dev/stdout 2024-04-06T09:29:43.859 DEBUG:teuthology.orchestra.run.smithi192:> set -ex 2024-04-06T09:29:43.859 DEBUG:teuthology.orchestra.run.smithi192:> dd if=/etc/ceph/cluster2.client.mirror.5.keyring of=/dev/stdout 2024-04-06T09:29:43.907 DEBUG:teuthology.orchestra.run.smithi192:> set -ex 2024-04-06T09:29:43.907 DEBUG:teuthology.orchestra.run.smithi192:> dd if=/etc/ceph/cluster2.client.mirror.6.keyring of=/dev/stdout 2024-04-06T09:29:43.955 INFO:tasks.ceph:Adding keys to all mons... 2024-04-06T09:29:43.955 DEBUG:teuthology.orchestra.run.smithi192:> sudo tee -a /etc/ceph/cluster2.keyring 2024-04-06T09:29:44.009 INFO:teuthology.orchestra.run.smithi192.stdout:[mgr.x] 2024-04-06T09:29:44.010 INFO:teuthology.orchestra.run.smithi192.stdout: key = AQD7FRFmaELNCBAAo97ScXY3A2OQyj7cDX9UrQ== 2024-04-06T09:29:44.010 INFO:teuthology.orchestra.run.smithi192.stdout:[osd.0] 2024-04-06T09:29:44.010 INFO:teuthology.orchestra.run.smithi192.stdout: key = AQD+FRFmGlvOCxAA4iR+lca+OB6QIngPVSn9gw== 2024-04-06T09:29:44.010 INFO:teuthology.orchestra.run.smithi192.stdout:[osd.1] 2024-04-06T09:29:44.010 INFO:teuthology.orchestra.run.smithi192.stdout: key = AQABFhFmLddlEBAAsLlHFUs40QrXviR5XRfgaQ== 2024-04-06T09:29:44.010 INFO:teuthology.orchestra.run.smithi192.stdout:[osd.2] 2024-04-06T09:29:44.010 INFO:teuthology.orchestra.run.smithi192.stdout: key = AQAEFhFmObRGFxAAmcu1UjiP3WjWwjGq34a7Gw== 2024-04-06T09:29:44.010 INFO:teuthology.orchestra.run.smithi192.stdout:[client.0] 2024-04-06T09:29:44.010 INFO:teuthology.orchestra.run.smithi192.stdout: key = AQD7FRFmhsJZDRAAenxwAwoFQsgNPSeSYkN4xg== 2024-04-06T09:29:44.010 INFO:teuthology.orchestra.run.smithi192.stdout:[client.mirror] 2024-04-06T09:29:44.010 INFO:teuthology.orchestra.run.smithi192.stdout: key = AQD7FRFmycpDEBAAMQpNbZqLa7sdymDWbTQ9dA== 2024-04-06T09:29:44.011 INFO:teuthology.orchestra.run.smithi192.stdout:[client.mirror.0] 2024-04-06T09:29:44.011 INFO:teuthology.orchestra.run.smithi192.stdout: key = AQD7FRFmpDNkFRAAVdeR0u20QgUJZT0gmtaRSA== 2024-04-06T09:29:44.011 INFO:teuthology.orchestra.run.smithi192.stdout:[client.mirror.1] 2024-04-06T09:29:44.011 INFO:teuthology.orchestra.run.smithi192.stdout: key = AQD7FRFmnW8nGhAAqmVSeG9hIG+NyKpLxxRqPg== 2024-04-06T09:29:44.011 INFO:teuthology.orchestra.run.smithi192.stdout:[client.mirror.2] 2024-04-06T09:29:44.011 INFO:teuthology.orchestra.run.smithi192.stdout: key = AQD7FRFmqIzUHhAAfn64GMo9E7UydRyR5MLj5g== 2024-04-06T09:29:44.011 INFO:teuthology.orchestra.run.smithi192.stdout:[client.mirror.3] 2024-04-06T09:29:44.011 INFO:teuthology.orchestra.run.smithi192.stdout: key = AQD7FRFm8Oy/IxAAL3DVjucMI9mxWU1/RZKsZg== 2024-04-06T09:29:44.011 INFO:teuthology.orchestra.run.smithi192.stdout:[client.mirror.4] 2024-04-06T09:29:44.011 INFO:teuthology.orchestra.run.smithi192.stdout: key = AQD7FRFmwJvoKBAAB57L0Ki3/j5AuTxthwUdnQ== 2024-04-06T09:29:44.011 INFO:teuthology.orchestra.run.smithi192.stdout:[client.mirror.5] 2024-04-06T09:29:44.011 INFO:teuthology.orchestra.run.smithi192.stdout: key = AQD7FRFmLuXBLRAAQ7qLYeAWbwIunIZOtH47Mg== 2024-04-06T09:29:44.012 INFO:teuthology.orchestra.run.smithi192.stdout:[client.mirror.6] 2024-04-06T09:29:44.012 INFO:teuthology.orchestra.run.smithi192.stdout: key = AQD7FRFmukzNMhAASp11ccXEjsieTC56nWKXfg== 2024-04-06T09:29:44.013 DEBUG:teuthology.orchestra.run.smithi192:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster2.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-04-06T09:29:44.098 DEBUG:teuthology.orchestra.run.smithi192:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster2.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-06T09:29:44.187 DEBUG:teuthology.orchestra.run.smithi192:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster2.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-06T09:29:44.269 DEBUG:teuthology.orchestra.run.smithi192:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster2.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-06T09:29:44.349 DEBUG:teuthology.orchestra.run.smithi192:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster2.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-04-06T09:29:44.432 DEBUG:teuthology.orchestra.run.smithi192:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster2.keyring --name=client.mirror --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-04-06T09:29:44.520 DEBUG:teuthology.orchestra.run.smithi192:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster2.keyring --name=client.mirror.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-04-06T09:29:44.561 DEBUG:teuthology.orchestra.run.smithi192:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster2.keyring --name=client.mirror.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-04-06T09:29:44.644 DEBUG:teuthology.orchestra.run.smithi192:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster2.keyring --name=client.mirror.2 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-04-06T09:29:44.732 DEBUG:teuthology.orchestra.run.smithi192:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster2.keyring --name=client.mirror.3 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-04-06T09:29:44.777 DEBUG:teuthology.orchestra.run.smithi192:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster2.keyring --name=client.mirror.4 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-04-06T09:29:44.861 DEBUG:teuthology.orchestra.run.smithi192:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster2.keyring --name=client.mirror.5 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-04-06T09:29:44.942 DEBUG:teuthology.orchestra.run.smithi192:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster2.keyring --name=client.mirror.6 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-04-06T09:29:45.030 INFO:tasks.ceph:Running mkfs on mon nodes... 2024-04-06T09:29:45.030 DEBUG:teuthology.orchestra.run.smithi192:> sudo mkdir -p /var/lib/ceph/mon/cluster2-a 2024-04-06T09:29:45.084 DEBUG:teuthology.orchestra.run.smithi192:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster cluster2 --mkfs -i a --monmap /home/ubuntu/cephtest/cluster2.monmap --keyring /etc/ceph/cluster2.keyring 2024-04-06T09:29:45.174 DEBUG:teuthology.orchestra.run.smithi192:> sudo chown -R ceph:ceph /var/lib/ceph/mon/cluster2-a 2024-04-06T09:29:45.184 DEBUG:teuthology.orchestra.run.smithi192:> rm -- /home/ubuntu/cephtest/cluster2.monmap 2024-04-06T09:29:45.230 INFO:tasks.ceph:Starting mon daemons in cluster cluster2... 2024-04-06T09:29:45.230 INFO:tasks.ceph.cluster2.mon.a:Restarting daemon 2024-04-06T09:29:45.231 DEBUG:teuthology.orchestra.run.smithi192:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster cluster2 -i a 2024-04-06T09:29:45.276 INFO:tasks.ceph.cluster2.mon.a:Started 2024-04-06T09:29:45.276 INFO:tasks.ceph:Starting mgr daemons in cluster cluster2... 2024-04-06T09:29:45.276 INFO:tasks.ceph.cluster2.mgr.x:Restarting daemon 2024-04-06T09:29:45.276 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster cluster2 -i x 2024-04-06T09:29:45.279 INFO:tasks.ceph.cluster2.mgr.x:Started 2024-04-06T09:29:45.279 INFO:tasks.ceph:Setting crush tunables to default 2024-04-06T09:29:45.279 DEBUG:teuthology.orchestra.run.smithi192:> sudo ceph --cluster cluster2 osd crush tunables default 2024-04-06T09:29:45.518 INFO:teuthology.orchestra.run.smithi192.stderr:adjusted tunables profile to default 2024-04-06T09:29:45.536 INFO:tasks.ceph:check_enable_crimson: False 2024-04-06T09:29:45.536 INFO:tasks.ceph:Starting osd daemons in cluster cluster2... 2024-04-06T09:29:45.536 DEBUG:teuthology.orchestra.run.smithi192:> set -ex 2024-04-06T09:29:45.537 DEBUG:teuthology.orchestra.run.smithi192:> sudo dd if=/var/lib/ceph/osd/cluster2-0/fsid of=/dev/stdout 2024-04-06T09:29:45.549 DEBUG:teuthology.orchestra.run.smithi192:> set -ex 2024-04-06T09:29:45.549 DEBUG:teuthology.orchestra.run.smithi192:> sudo dd if=/var/lib/ceph/osd/cluster2-1/fsid of=/dev/stdout 2024-04-06T09:29:45.551 INFO:tasks.ceph.cluster2.mgr.x.smithi006.stderr:2024-04-06T09:29:45.550+0000 7f33b5dc8180 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-06T09:29:45.600 DEBUG:teuthology.orchestra.run.smithi192:> set -ex 2024-04-06T09:29:45.600 DEBUG:teuthology.orchestra.run.smithi192:> sudo dd if=/var/lib/ceph/osd/cluster2-2/fsid of=/dev/stdout 2024-04-06T09:29:45.657 DEBUG:teuthology.orchestra.run.smithi192:> sudo ceph --cluster cluster2 osd new 92ff8878-e8ea-4f7a-80e8-a98fd9a387c5 0 2024-04-06T09:29:45.900 INFO:teuthology.orchestra.run.smithi192.stdout:0 2024-04-06T09:29:45.917 DEBUG:teuthology.orchestra.run.smithi192:> sudo ceph --cluster cluster2 osd new 7beb4d05-1fb2-4116-b33e-559d2734ebe7 1 2024-04-06T09:29:45.956 INFO:tasks.ceph.cluster2.mgr.x.smithi006.stderr:2024-04-06T09:29:45.954+0000 7f33b5dc8180 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-06T09:29:46.091 INFO:tasks.ceph.cluster2.mgr.x.smithi006.stderr:2024-04-06T09:29:46.090+0000 7f33b5dc8180 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-06T09:29:46.120 INFO:teuthology.orchestra.run.smithi192.stdout:1 2024-04-06T09:29:46.137 DEBUG:teuthology.orchestra.run.smithi192:> sudo ceph --cluster cluster2 osd new c99a1679-ff03-419c-a26b-797120489d61 2 2024-04-06T09:29:46.342 INFO:teuthology.orchestra.run.smithi192.stdout:2 2024-04-06T09:29:46.359 INFO:tasks.ceph.cluster2.osd.0:Restarting daemon 2024-04-06T09:29:46.359 DEBUG:teuthology.orchestra.run.smithi192:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster cluster2 -i 0 2024-04-06T09:29:46.361 INFO:tasks.ceph.cluster2.osd.0:Started 2024-04-06T09:29:46.361 INFO:tasks.ceph.cluster2.osd.1:Restarting daemon 2024-04-06T09:29:46.361 DEBUG:teuthology.orchestra.run.smithi192:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster cluster2 -i 1 2024-04-06T09:29:46.364 INFO:tasks.ceph.cluster2.osd.1:Started 2024-04-06T09:29:46.364 INFO:tasks.ceph.cluster2.osd.2:Restarting daemon 2024-04-06T09:29:46.364 DEBUG:teuthology.orchestra.run.smithi192:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster cluster2 -i 2 2024-04-06T09:29:46.366 INFO:tasks.ceph.cluster2.osd.2:Started 2024-04-06T09:29:46.366 DEBUG:teuthology.orchestra.run.smithi192:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 osd dump --format=json 2024-04-06T09:29:46.526 INFO:tasks.ceph.cluster2.mgr.x.smithi006.stderr:2024-04-06T09:29:46.522+0000 7f33b5dc8180 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-06T09:29:46.545 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-04-06T09:29:46.545 INFO:teuthology.orchestra.run.smithi192.stdout:{"epoch":5,"fsid":"cb2d68e3-446e-4a71-a214-bfd667a21bca","created":"2024-04-06T09:29:45.413204+0000","modified":"2024-04-06T09:29:46.341898+0000","last_up_change":"0.000000","last_in_change":"2024-04-06T09:29:46.341898+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"92ff8878-e8ea-4f7a-80e8-a98fd9a387c5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"7beb4d05-1fb2-4116-b33e-559d2734ebe7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"c99a1679-ff03-419c-a26b-797120489d61","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-06T09:29:46.563 INFO:tasks.ceph.ceph_manager.cluster2:[] 2024-04-06T09:29:46.563 INFO:tasks.ceph:Waiting for OSDs to come up 2024-04-06T09:29:46.576 INFO:tasks.ceph.cluster2.mgr.x.smithi006.stderr:2024-04-06T09:29:46.574+0000 7f33b5dc8180 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-06T09:29:46.626 INFO:tasks.ceph.cluster2.mgr.x.smithi006.stderr:2024-04-06T09:29:46.626+0000 7f33b5dc8180 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-06T09:29:46.704 INFO:tasks.ceph.cluster2.mgr.x.smithi006.stderr:2024-04-06T09:29:46.702+0000 7f33b5dc8180 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-06T09:29:46.755 INFO:tasks.ceph.cluster2.mgr.x.smithi006.stderr:2024-04-06T09:29:46.754+0000 7f33b5dc8180 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-06T09:29:46.831 INFO:tasks.ceph.cluster2.mgr.x.smithi006.stderr:2024-04-06T09:29:46.830+0000 7f33b5dc8180 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-06T09:29:46.865 DEBUG:teuthology.orchestra.run.smithi192:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster cluster2 osd dump --format=json 2024-04-06T09:29:46.882 INFO:tasks.ceph.cluster2.mgr.x.smithi006.stderr:2024-04-06T09:29:46.882+0000 7f33b5dc8180 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-06T09:29:46.991 INFO:tasks.ceph.cluster2.osd.2.smithi192.stderr:2024-04-06T09:29:46.989+0000 7fa5450be640 -1 Falling back to public interface 2024-04-06T09:29:46.992 INFO:tasks.ceph.cluster2.osd.1.smithi192.stderr:2024-04-06T09:29:46.989+0000 7f0df0d1d640 -1 Falling back to public interface 2024-04-06T09:29:46.999 INFO:tasks.ceph.cluster2.osd.0.smithi192.stderr:2024-04-06T09:29:46.997+0000 7f78bfce3640 -1 Falling back to public interface 2024-04-06T09:29:47.026 INFO:tasks.ceph.cluster2.mgr.x.smithi006.stderr:2024-04-06T09:29:47.026+0000 7f33b5dc8180 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-06T09:29:47.054 INFO:teuthology.misc.health.smithi192.stdout: 2024-04-06T09:29:47.054 INFO:teuthology.misc.health.smithi192.stdout:{"epoch":5,"fsid":"cb2d68e3-446e-4a71-a214-bfd667a21bca","created":"2024-04-06T09:29:45.413204+0000","modified":"2024-04-06T09:29:46.341898+0000","last_up_change":"0.000000","last_in_change":"2024-04-06T09:29:46.341898+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"92ff8878-e8ea-4f7a-80e8-a98fd9a387c5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"7beb4d05-1fb2-4116-b33e-559d2734ebe7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"c99a1679-ff03-419c-a26b-797120489d61","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-06T09:29:47.070 DEBUG:teuthology.misc:0 of 3 OSDs are up 2024-04-06T09:29:47.079 INFO:tasks.ceph.cluster2.mgr.x.smithi006.stderr:2024-04-06T09:29:47.078+0000 7f33b5dc8180 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-06T09:29:47.305 INFO:tasks.ceph.cluster2.mgr.x.smithi006.stderr:/usr/lib/python3/dist-packages/scipy/__init__.py:67: UserWarning: NumPy was imported from a Python sub-interpreter but NumPy does not properly support sub-interpreters. This will likely work for most users but might cause hard to track down issues or subtle bugs. A common user of the rare sub-interpreter feature is wsgi which also allows single-interpreter mode. 2024-04-06T09:29:47.305 INFO:tasks.ceph.cluster2.mgr.x.smithi006.stderr:Improvements in the case of bugs are welcome, but is not on the NumPy roadmap, and full support may require significant effort to achieve. 2024-04-06T09:29:47.305 INFO:tasks.ceph.cluster2.mgr.x.smithi006.stderr: from numpy import show_config as show_numpy_config 2024-04-06T09:29:47.312 INFO:tasks.ceph.cluster2.mgr.x.smithi006.stderr:2024-04-06T09:29:47.310+0000 7f33b5dc8180 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-06T09:29:47.451 INFO:tasks.ceph.cluster2.mgr.x.smithi006.stderr:2024-04-06T09:29:47.450+0000 7f33b5dc8180 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-06T09:29:47.564 INFO:tasks.ceph.cluster2.mgr.x.smithi006.stderr:2024-04-06T09:29:47.562+0000 7f33b5dc8180 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-06T09:29:47.628 INFO:tasks.ceph.cluster2.mgr.x.smithi006.stderr:2024-04-06T09:29:47.626+0000 7f33b5dc8180 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-06T09:29:47.765 INFO:tasks.ceph.cluster2.mgr.x.smithi006.stderr:2024-04-06T09:29:47.762+0000 7f33b5dc8180 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-06T09:29:48.262 INFO:tasks.ceph.cluster2.mgr.x.smithi006.stderr:2024-04-06T09:29:48.262+0000 7f33b5dc8180 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-06T09:29:48.338 INFO:tasks.ceph.cluster2.mgr.x.smithi006.stderr:2024-04-06T09:29:48.338+0000 7f33b5dc8180 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-06T09:29:48.488 INFO:tasks.ceph.cluster2.mgr.x.smithi006.stderr:2024-04-06T09:29:48.486+0000 7f33b5dc8180 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-06T09:29:48.564 INFO:tasks.ceph.cluster2.mgr.x.smithi006.stderr:2024-04-06T09:29:48.562+0000 7f33b5dc8180 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-06T09:29:48.861 INFO:tasks.ceph.cluster2.mgr.x.smithi006.stderr:2024-04-06T09:29:48.854+0000 7f33b5dc8180 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-06T09:29:48.914 INFO:tasks.ceph.cluster2.mgr.x.smithi006.stderr:2024-04-06T09:29:48.914+0000 7f33b5dc8180 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-06T09:29:49.948 INFO:tasks.ceph.cluster2.mgr.x.smithi006.stderr:2024-04-06T09:29:49.946+0000 7f335d381640 -1 mgr.server handle_report got status from non-daemon mon.a 2024-04-06T09:29:50.009 INFO:tasks.ceph.cluster2.osd.1.smithi192.stderr:2024-04-06T09:29:50.005+0000 7f0df0d1d640 -1 osd.1 0 log_to_monitors true 2024-04-06T09:29:50.283 INFO:tasks.ceph.cluster2.osd.2.smithi192.stderr:2024-04-06T09:29:50.281+0000 7fa5450be640 -1 osd.2 0 log_to_monitors true 2024-04-06T09:29:50.714 INFO:tasks.ceph.cluster2.osd.0.smithi192.stderr:2024-04-06T09:29:50.709+0000 7f78bfce3640 -1 osd.0 0 log_to_monitors true 2024-04-06T09:29:53.163 INFO:tasks.ceph.cluster2.osd.1.smithi192.stderr:2024-04-06T09:29:53.161+0000 7f0ded4d9640 -1 osd.1 0 waiting for initial osdmap 2024-04-06T09:29:53.184 INFO:tasks.ceph.cluster2.osd.2.smithi192.stderr:2024-04-06T09:29:53.181+0000 7fa541063640 -1 osd.2 0 waiting for initial osdmap 2024-04-06T09:29:53.191 INFO:tasks.ceph.cluster2.osd.1.smithi192.stderr:2024-04-06T09:29:53.189+0000 7f0de4af5640 -1 osd.1 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-06T09:29:53.203 INFO:tasks.ceph.cluster2.osd.2.smithi192.stderr:2024-04-06T09:29:53.201+0000 7fa538e96640 -1 osd.2 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-06T09:29:53.205 INFO:tasks.ceph.cluster2.osd.0.smithi192.stderr:2024-04-06T09:29:53.201+0000 7f78bbc88640 -1 osd.0 0 waiting for initial osdmap 2024-04-06T09:29:53.239 INFO:tasks.ceph.cluster2.osd.0.smithi192.stderr:2024-04-06T09:29:53.237+0000 7f78b3abb640 -1 osd.0 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-06T09:29:53.372 DEBUG:teuthology.orchestra.run.smithi192:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster cluster2 osd dump --format=json 2024-04-06T09:29:53.654 INFO:teuthology.misc.health.smithi192.stdout: 2024-04-06T09:29:53.655 INFO:teuthology.misc.health.smithi192.stdout:{"epoch":7,"fsid":"cb2d68e3-446e-4a71-a214-bfd667a21bca","created":"2024-04-06T09:29:45.413204+0000","modified":"2024-04-06T09:29:52.002853+0000","last_up_change":"0.000000","last_in_change":"2024-04-06T09:29:46.341898+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"92ff8878-e8ea-4f7a-80e8-a98fd9a387c5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"7beb4d05-1fb2-4116-b33e-559d2734ebe7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"c99a1679-ff03-419c-a26b-797120489d61","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-06T09:29:53.670 DEBUG:teuthology.misc:0 of 3 OSDs are up 2024-04-06T09:29:59.974 DEBUG:teuthology.orchestra.run.smithi192:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster cluster2 osd dump --format=json 2024-04-06T09:30:00.269 INFO:teuthology.misc.health.smithi192.stdout: 2024-04-06T09:30:00.269 INFO:teuthology.misc.health.smithi192.stdout:{"epoch":11,"fsid":"cb2d68e3-446e-4a71-a214-bfd667a21bca","created":"2024-04-06T09:29:45.413204+0000","modified":"2024-04-06T09:29:57.223711+0000","last_up_change":"2024-04-06T09:29:54.209374+0000","last_in_change":"2024-04-06T09:29:46.341898+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-06T09:29:55.025846+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"11","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":2.9900000095367432,"score_stable":2.9900000095367432,"optimal_score":0.67000001668930054,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"92ff8878-e8ea-4f7a-80e8-a98fd9a387c5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6816","nonce":23962},{"type":"v1","addr":"172.21.15.192:6817","nonce":23962}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6818","nonce":23962},{"type":"v1","addr":"172.21.15.192:6819","nonce":23962}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6822","nonce":23962},{"type":"v1","addr":"172.21.15.192:6823","nonce":23962}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6820","nonce":23962},{"type":"v1","addr":"172.21.15.192:6821","nonce":23962}]},"public_addr":"172.21.15.192:6817/23962","cluster_addr":"172.21.15.192:6819/23962","heartbeat_back_addr":"172.21.15.192:6823/23962","heartbeat_front_addr":"172.21.15.192:6821/23962","state":["exists","up"]},{"osd":1,"uuid":"7beb4d05-1fb2-4116-b33e-559d2734ebe7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":9,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6802","nonce":23961},{"type":"v1","addr":"172.21.15.192:6804","nonce":23961}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6806","nonce":23961},{"type":"v1","addr":"172.21.15.192:6808","nonce":23961}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6814","nonce":23961},{"type":"v1","addr":"172.21.15.192:6815","nonce":23961}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6810","nonce":23961},{"type":"v1","addr":"172.21.15.192:6812","nonce":23961}]},"public_addr":"172.21.15.192:6804/23961","cluster_addr":"172.21.15.192:6808/23961","heartbeat_back_addr":"172.21.15.192:6815/23961","heartbeat_front_addr":"172.21.15.192:6812/23961","state":["exists","up"]},{"osd":2,"uuid":"c99a1679-ff03-419c-a26b-797120489d61","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6800","nonce":23963},{"type":"v1","addr":"172.21.15.192:6801","nonce":23963}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6803","nonce":23963},{"type":"v1","addr":"172.21.15.192:6805","nonce":23963}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6811","nonce":23963},{"type":"v1","addr":"172.21.15.192:6813","nonce":23963}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6807","nonce":23963},{"type":"v1","addr":"172.21.15.192:6809","nonce":23963}]},"public_addr":"172.21.15.192:6801/23963","cluster_addr":"172.21.15.192:6805/23963","heartbeat_back_addr":"172.21.15.192:6813/23963","heartbeat_front_addr":"172.21.15.192:6809/23963","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T09:29:51.734405+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T09:29:51.033517+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T09:29:51.288986+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-06T09:30:00.287 DEBUG:teuthology.misc:3 of 3 OSDs are up 2024-04-06T09:30:00.287 INFO:tasks.ceph:Creating RBD pool 2024-04-06T09:30:00.287 DEBUG:teuthology.orchestra.run.smithi192:> sudo ceph --cluster cluster2 osd pool create rbd 8 2024-04-06T09:30:01.364 INFO:teuthology.orchestra.run.smithi192.stderr:pool 'rbd' created 2024-04-06T09:30:01.390 DEBUG:teuthology.orchestra.run.smithi192:> sudo ceph --cluster cluster2 osd pool application enable rbd rbd --yes-i-really-mean-it 2024-04-06T09:30:02.370 INFO:teuthology.orchestra.run.smithi192.stderr:enabled application 'rbd' on pool 'rbd' 2024-04-06T09:30:02.398 INFO:tasks.ceph:Starting mds daemons in cluster cluster2... 2024-04-06T09:30:02.399 INFO:tasks.ceph:Waiting until cluster2 daemons up and pgs clean... 2024-04-06T09:30:02.399 INFO:tasks.ceph.ceph_manager.cluster2:waiting for mgr available 2024-04-06T09:30:02.399 DEBUG:teuthology.orchestra.run.smithi192:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 mgr dump --format=json 2024-04-06T09:30:02.400 INFO:tasks.daemonwatchdog.daemon_watchdog:watchdog starting 2024-04-06T09:30:02.712 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-04-06T09:30:02.728 INFO:teuthology.orchestra.run.smithi192.stdout:{"epoch":5,"active_gid":4100,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6826","nonce":27147},{"type":"v1","addr":"172.21.15.6:6827","nonce":27147}]},"active_addr":"172.21.15.6:6827/27147","active_change":"2024-04-06T09:29:48.938807+0000","active_mgr_features":4540138322906710015,"available":true,"standbys":[],"modules":["iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.7","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.0.0","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":0,"active_clients":[{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.6:0","nonce":2684786281}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.6:0","nonce":3680322413}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.6:0","nonce":1586983564}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.6:0","nonce":343501210}]}]} 2024-04-06T09:30:02.730 INFO:tasks.ceph.ceph_manager.cluster2:mgr available! 2024-04-06T09:30:02.730 INFO:tasks.ceph.ceph_manager.cluster2:waiting for all up 2024-04-06T09:30:02.730 DEBUG:teuthology.orchestra.run.smithi192:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 osd dump --format=json 2024-04-06T09:30:03.017 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-04-06T09:30:03.017 INFO:teuthology.orchestra.run.smithi192.stdout:{"epoch":13,"fsid":"cb2d68e3-446e-4a71-a214-bfd667a21bca","created":"2024-04-06T09:29:45.413204+0000","modified":"2024-04-06T09:30:02.364182+0000","last_up_change":"2024-04-06T09:29:54.209374+0000","last_in_change":"2024-04-06T09:29:46.341898+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-06T09:29:55.025846+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"11","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":2.9900000095367432,"score_stable":2.9900000095367432,"optimal_score":0.67000001668930054,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-04-06T09:30:00.619189+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"13","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"92ff8878-e8ea-4f7a-80e8-a98fd9a387c5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":12,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6816","nonce":23962},{"type":"v1","addr":"172.21.15.192:6817","nonce":23962}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6818","nonce":23962},{"type":"v1","addr":"172.21.15.192:6819","nonce":23962}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6822","nonce":23962},{"type":"v1","addr":"172.21.15.192:6823","nonce":23962}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6820","nonce":23962},{"type":"v1","addr":"172.21.15.192:6821","nonce":23962}]},"public_addr":"172.21.15.192:6817/23962","cluster_addr":"172.21.15.192:6819/23962","heartbeat_back_addr":"172.21.15.192:6823/23962","heartbeat_front_addr":"172.21.15.192:6821/23962","state":["exists","up"]},{"osd":1,"uuid":"7beb4d05-1fb2-4116-b33e-559d2734ebe7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":12,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6802","nonce":23961},{"type":"v1","addr":"172.21.15.192:6804","nonce":23961}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6806","nonce":23961},{"type":"v1","addr":"172.21.15.192:6808","nonce":23961}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6814","nonce":23961},{"type":"v1","addr":"172.21.15.192:6815","nonce":23961}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6810","nonce":23961},{"type":"v1","addr":"172.21.15.192:6812","nonce":23961}]},"public_addr":"172.21.15.192:6804/23961","cluster_addr":"172.21.15.192:6808/23961","heartbeat_back_addr":"172.21.15.192:6815/23961","heartbeat_front_addr":"172.21.15.192:6812/23961","state":["exists","up"]},{"osd":2,"uuid":"c99a1679-ff03-419c-a26b-797120489d61","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":12,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6800","nonce":23963},{"type":"v1","addr":"172.21.15.192:6801","nonce":23963}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6803","nonce":23963},{"type":"v1","addr":"172.21.15.192:6805","nonce":23963}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6811","nonce":23963},{"type":"v1","addr":"172.21.15.192:6813","nonce":23963}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6807","nonce":23963},{"type":"v1","addr":"172.21.15.192:6809","nonce":23963}]},"public_addr":"172.21.15.192:6801/23963","cluster_addr":"172.21.15.192:6805/23963","heartbeat_back_addr":"172.21.15.192:6813/23963","heartbeat_front_addr":"172.21.15.192:6809/23963","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T09:29:51.734405+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T09:29:51.033517+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T09:29:51.288986+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-06T09:30:03.035 INFO:tasks.ceph.ceph_manager.cluster2:all up! 2024-04-06T09:30:03.035 DEBUG:teuthology.orchestra.run.smithi192:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 osd dump --format=json 2024-04-06T09:30:03.315 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-04-06T09:30:03.315 INFO:teuthology.orchestra.run.smithi192.stdout:{"epoch":13,"fsid":"cb2d68e3-446e-4a71-a214-bfd667a21bca","created":"2024-04-06T09:29:45.413204+0000","modified":"2024-04-06T09:30:02.364182+0000","last_up_change":"2024-04-06T09:29:54.209374+0000","last_in_change":"2024-04-06T09:29:46.341898+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-06T09:29:55.025846+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"11","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":2.9900000095367432,"score_stable":2.9900000095367432,"optimal_score":0.67000001668930054,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-04-06T09:30:00.619189+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"13","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"92ff8878-e8ea-4f7a-80e8-a98fd9a387c5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":12,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6816","nonce":23962},{"type":"v1","addr":"172.21.15.192:6817","nonce":23962}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6818","nonce":23962},{"type":"v1","addr":"172.21.15.192:6819","nonce":23962}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6822","nonce":23962},{"type":"v1","addr":"172.21.15.192:6823","nonce":23962}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6820","nonce":23962},{"type":"v1","addr":"172.21.15.192:6821","nonce":23962}]},"public_addr":"172.21.15.192:6817/23962","cluster_addr":"172.21.15.192:6819/23962","heartbeat_back_addr":"172.21.15.192:6823/23962","heartbeat_front_addr":"172.21.15.192:6821/23962","state":["exists","up"]},{"osd":1,"uuid":"7beb4d05-1fb2-4116-b33e-559d2734ebe7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":12,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6802","nonce":23961},{"type":"v1","addr":"172.21.15.192:6804","nonce":23961}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6806","nonce":23961},{"type":"v1","addr":"172.21.15.192:6808","nonce":23961}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6814","nonce":23961},{"type":"v1","addr":"172.21.15.192:6815","nonce":23961}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6810","nonce":23961},{"type":"v1","addr":"172.21.15.192:6812","nonce":23961}]},"public_addr":"172.21.15.192:6804/23961","cluster_addr":"172.21.15.192:6808/23961","heartbeat_back_addr":"172.21.15.192:6815/23961","heartbeat_front_addr":"172.21.15.192:6812/23961","state":["exists","up"]},{"osd":2,"uuid":"c99a1679-ff03-419c-a26b-797120489d61","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":12,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6800","nonce":23963},{"type":"v1","addr":"172.21.15.192:6801","nonce":23963}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6803","nonce":23963},{"type":"v1","addr":"172.21.15.192:6805","nonce":23963}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6811","nonce":23963},{"type":"v1","addr":"172.21.15.192:6813","nonce":23963}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6807","nonce":23963},{"type":"v1","addr":"172.21.15.192:6809","nonce":23963}]},"public_addr":"172.21.15.192:6801/23963","cluster_addr":"172.21.15.192:6805/23963","heartbeat_back_addr":"172.21.15.192:6813/23963","heartbeat_front_addr":"172.21.15.192:6809/23963","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T09:29:51.734405+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T09:29:51.033517+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T09:29:51.288986+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-06T09:30:03.332 DEBUG:teuthology.orchestra.run.smithi192:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 tell osd.0 flush_pg_stats 2024-04-06T09:30:03.332 DEBUG:teuthology.orchestra.run.smithi192:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 tell osd.1 flush_pg_stats 2024-04-06T09:30:03.332 DEBUG:teuthology.orchestra.run.smithi192:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 tell osd.2 flush_pg_stats 2024-04-06T09:30:03.484 INFO:teuthology.orchestra.run.smithi192.stdout:34359738373 2024-04-06T09:30:03.485 DEBUG:teuthology.orchestra.run.smithi192:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 osd last-stat-seq osd.2 2024-04-06T09:30:03.492 INFO:teuthology.orchestra.run.smithi192.stdout:34359738373 2024-04-06T09:30:03.493 DEBUG:teuthology.orchestra.run.smithi192:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 osd last-stat-seq osd.0 2024-04-06T09:30:03.493 INFO:teuthology.orchestra.run.smithi192.stdout:34359738372 2024-04-06T09:30:03.493 DEBUG:teuthology.orchestra.run.smithi192:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 osd last-stat-seq osd.1 2024-04-06T09:30:03.771 INFO:teuthology.orchestra.run.smithi192.stdout:34359738372 2024-04-06T09:30:03.789 INFO:tasks.ceph.ceph_manager.cluster2:need seq 34359738373 got 34359738372 for osd.2 2024-04-06T09:30:03.812 INFO:teuthology.orchestra.run.smithi192.stdout:34359738372 2024-04-06T09:30:03.827 INFO:teuthology.orchestra.run.smithi192.stdout:34359738371 2024-04-06T09:30:03.828 INFO:tasks.ceph.ceph_manager.cluster2:need seq 34359738373 got 34359738372 for osd.0 2024-04-06T09:30:03.844 INFO:tasks.ceph.ceph_manager.cluster2:need seq 34359738372 got 34359738371 for osd.1 2024-04-06T09:30:04.789 DEBUG:teuthology.orchestra.run.smithi192:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 osd last-stat-seq osd.2 2024-04-06T09:30:04.830 DEBUG:teuthology.orchestra.run.smithi192:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 osd last-stat-seq osd.0 2024-04-06T09:30:04.845 DEBUG:teuthology.orchestra.run.smithi192:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 osd last-stat-seq osd.1 2024-04-06T09:30:05.102 INFO:teuthology.orchestra.run.smithi192.stdout:34359738372 2024-04-06T09:30:05.120 INFO:tasks.ceph.ceph_manager.cluster2:need seq 34359738373 got 34359738372 for osd.2 2024-04-06T09:30:05.138 INFO:teuthology.orchestra.run.smithi192.stdout:34359738372 2024-04-06T09:30:05.154 INFO:tasks.ceph.ceph_manager.cluster2:need seq 34359738373 got 34359738372 for osd.0 2024-04-06T09:30:05.164 INFO:teuthology.orchestra.run.smithi192.stdout:34359738371 2024-04-06T09:30:05.181 INFO:tasks.ceph.ceph_manager.cluster2:need seq 34359738372 got 34359738371 for osd.1 2024-04-06T09:30:06.121 DEBUG:teuthology.orchestra.run.smithi192:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 osd last-stat-seq osd.2 2024-04-06T09:30:06.154 DEBUG:teuthology.orchestra.run.smithi192:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 osd last-stat-seq osd.0 2024-04-06T09:30:06.181 DEBUG:teuthology.orchestra.run.smithi192:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 osd last-stat-seq osd.1 2024-04-06T09:30:06.423 INFO:teuthology.orchestra.run.smithi192.stdout:34359738373 2024-04-06T09:30:06.440 INFO:tasks.ceph.ceph_manager.cluster2:need seq 34359738373 got 34359738373 for osd.2 2024-04-06T09:30:06.441 DEBUG:teuthology.parallel:result is None 2024-04-06T09:30:06.467 INFO:teuthology.orchestra.run.smithi192.stdout:34359738373 2024-04-06T09:30:06.484 INFO:tasks.ceph.ceph_manager.cluster2:need seq 34359738373 got 34359738373 for osd.0 2024-04-06T09:30:06.484 DEBUG:teuthology.parallel:result is None 2024-04-06T09:30:06.509 INFO:teuthology.orchestra.run.smithi192.stdout:34359738372 2024-04-06T09:30:06.526 INFO:tasks.ceph.ceph_manager.cluster2:need seq 34359738372 got 34359738372 for osd.1 2024-04-06T09:30:06.526 DEBUG:teuthology.parallel:result is None 2024-04-06T09:30:06.526 INFO:tasks.ceph.ceph_manager.cluster2:waiting for clean 2024-04-06T09:30:06.526 DEBUG:teuthology.orchestra.run.smithi192:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2024-04-06T09:30:06.856 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-04-06T09:30:06.856 INFO:teuthology.orchestra.run.smithi192.stderr:dumped all 2024-04-06T09:30:06.874 INFO:teuthology.orchestra.run.smithi192.stdout:{"pg_ready":true,"pg_map":{"version":17,"stamp":"2024-04-06T09:30:04.946392+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":15,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":81528,"kb_used_data":456,"kb_used_omap":4,"kb_used_meta":81019,"kb_avail":283033992,"statfs":{"total":289910292480,"available":289826807808,"internally_reserved":0,"allocated":466944,"data_stored":1274693,"data_compressed":60694,"data_compressed_allocated":81920,"data_compressed_original":1146880,"omap_allocated":4782,"internal_metadata":82963794},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.000981"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":10,"reported_epoch":14,"state":"active+clean","last_fresh":"2024-04-06T09:30:03.388965+0000","last_change":"2024-04-06T09:30:02.384212+0000","last_active":"2024-04-06T09:30:03.388965+0000","last_peered":"2024-04-06T09:30:03.388965+0000","last_clean":"2024-04-06T09:30:03.388965+0000","last_became_active":"2024-04-06T09:30:02.383710+0000","last_became_peered":"2024-04-06T09:30:02.383710+0000","last_unstale":"2024-04-06T09:30:03.388965+0000","last_undegraded":"2024-04-06T09:30:03.388965+0000","last_fullsized":"2024-04-06T09:30:03.388965+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T09:30:01.359377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T09:30:01.359377+0000","last_clean_scrub_stamp":"2024-04-06T09:30:01.359377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T18:51:44.859927+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":10,"reported_epoch":14,"state":"active+clean","last_fresh":"2024-04-06T09:30:03.388895+0000","last_change":"2024-04-06T09:30:02.385797+0000","last_active":"2024-04-06T09:30:03.388895+0000","last_peered":"2024-04-06T09:30:03.388895+0000","last_clean":"2024-04-06T09:30:03.388895+0000","last_became_active":"2024-04-06T09:30:02.385501+0000","last_became_peered":"2024-04-06T09:30:02.385501+0000","last_unstale":"2024-04-06T09:30:03.388895+0000","last_undegraded":"2024-04-06T09:30:03.388895+0000","last_fullsized":"2024-04-06T09:30:03.388895+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T09:30:01.359377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T09:30:01.359377+0000","last_clean_scrub_stamp":"2024-04-06T09:30:01.359377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T15:08:40.865789+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":10,"reported_epoch":14,"state":"active+clean","last_fresh":"2024-04-06T09:30:03.388825+0000","last_change":"2024-04-06T09:30:02.383199+0000","last_active":"2024-04-06T09:30:03.388825+0000","last_peered":"2024-04-06T09:30:03.388825+0000","last_clean":"2024-04-06T09:30:03.388825+0000","last_became_active":"2024-04-06T09:30:02.382798+0000","last_became_peered":"2024-04-06T09:30:02.382798+0000","last_unstale":"2024-04-06T09:30:03.388825+0000","last_undegraded":"2024-04-06T09:30:03.388825+0000","last_fullsized":"2024-04-06T09:30:03.388825+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T09:30:01.359377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T09:30:01.359377+0000","last_clean_scrub_stamp":"2024-04-06T09:30:01.359377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T18:11:34.478290+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":10,"reported_epoch":14,"state":"active+clean","last_fresh":"2024-04-06T09:30:03.388756+0000","last_change":"2024-04-06T09:30:02.383483+0000","last_active":"2024-04-06T09:30:03.388756+0000","last_peered":"2024-04-06T09:30:03.388756+0000","last_clean":"2024-04-06T09:30:03.388756+0000","last_became_active":"2024-04-06T09:30:02.382963+0000","last_became_peered":"2024-04-06T09:30:02.382963+0000","last_unstale":"2024-04-06T09:30:03.388756+0000","last_undegraded":"2024-04-06T09:30:03.388756+0000","last_fullsized":"2024-04-06T09:30:03.388756+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T09:30:01.359377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T09:30:01.359377+0000","last_clean_scrub_stamp":"2024-04-06T09:30:01.359377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T10:19:04.998532+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":9,"reported_epoch":13,"state":"active+clean","last_fresh":"2024-04-06T09:30:02.385558+0000","last_change":"2024-04-06T09:30:02.383104+0000","last_active":"2024-04-06T09:30:02.385558+0000","last_peered":"2024-04-06T09:30:02.385558+0000","last_clean":"2024-04-06T09:30:02.385558+0000","last_became_active":"2024-04-06T09:30:02.382554+0000","last_became_peered":"2024-04-06T09:30:02.382554+0000","last_unstale":"2024-04-06T09:30:02.385558+0000","last_undegraded":"2024-04-06T09:30:02.385558+0000","last_fullsized":"2024-04-06T09:30:02.385558+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T09:30:01.359377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T09:30:01.359377+0000","last_clean_scrub_stamp":"2024-04-06T09:30:01.359377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T18:47:16.568825+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":9,"reported_epoch":13,"state":"active+clean","last_fresh":"2024-04-06T09:30:02.387827+0000","last_change":"2024-04-06T09:30:02.386812+0000","last_active":"2024-04-06T09:30:02.387827+0000","last_peered":"2024-04-06T09:30:02.387827+0000","last_clean":"2024-04-06T09:30:02.387827+0000","last_became_active":"2024-04-06T09:30:02.386256+0000","last_became_peered":"2024-04-06T09:30:02.386256+0000","last_unstale":"2024-04-06T09:30:02.387827+0000","last_undegraded":"2024-04-06T09:30:02.387827+0000","last_fullsized":"2024-04-06T09:30:02.387827+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T09:30:01.359377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T09:30:01.359377+0000","last_clean_scrub_stamp":"2024-04-06T09:30:01.359377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T21:22:07.757399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":9,"reported_epoch":13,"state":"active+clean","last_fresh":"2024-04-06T09:30:02.387815+0000","last_change":"2024-04-06T09:30:02.386748+0000","last_active":"2024-04-06T09:30:02.387815+0000","last_peered":"2024-04-06T09:30:02.387815+0000","last_clean":"2024-04-06T09:30:02.387815+0000","last_became_active":"2024-04-06T09:30:02.386318+0000","last_became_peered":"2024-04-06T09:30:02.386318+0000","last_unstale":"2024-04-06T09:30:02.387815+0000","last_undegraded":"2024-04-06T09:30:02.387815+0000","last_fullsized":"2024-04-06T09:30:02.387815+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T09:30:01.359377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T09:30:01.359377+0000","last_clean_scrub_stamp":"2024-04-06T09:30:01.359377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T10:50:42.918574+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":10,"reported_epoch":14,"state":"active+clean","last_fresh":"2024-04-06T09:30:03.388678+0000","last_change":"2024-04-06T09:30:02.382580+0000","last_active":"2024-04-06T09:30:03.388678+0000","last_peered":"2024-04-06T09:30:03.388678+0000","last_clean":"2024-04-06T09:30:03.388678+0000","last_became_active":"2024-04-06T09:30:02.382103+0000","last_became_peered":"2024-04-06T09:30:02.382103+0000","last_unstale":"2024-04-06T09:30:03.388678+0000","last_undegraded":"2024-04-06T09:30:03.388678+0000","last_fullsized":"2024-04-06T09:30:03.388678+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T09:30:01.359377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T09:30:01.359377+0000","last_clean_scrub_stamp":"2024-04-06T09:30:01.359377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T18:50:24.144774+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"10'97","reported_seq":140,"reported_epoch":13,"state":"active+clean","last_fresh":"2024-04-06T09:30:02.371155+0000","last_change":"2024-04-06T09:29:56.421931+0000","last_active":"2024-04-06T09:30:02.371155+0000","last_peered":"2024-04-06T09:30:02.371155+0000","last_clean":"2024-04-06T09:30:02.371155+0000","last_became_active":"2024-04-06T09:29:56.419754+0000","last_became_peered":"2024-04-06T09:29:56.419754+0000","last_unstale":"2024-04-06T09:30:02.371155+0000","last_undegraded":"2024-04-06T09:30:02.371155+0000","last_fullsized":"2024-04-06T09:30:02.371155+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T09:29:55.213061+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T09:29:55.213061+0000","last_clean_scrub_stamp":"2024-04-06T09:29:55.213061+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T19:28:59.898297+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":122880,"data_stored":1180736,"data_compressed":60694,"data_compressed_allocated":81920,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738373,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27136,"kb_used_data":112,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344704,"statfs":{"total":96636764160,"available":96608976896,"internally_reserved":0,"allocated":114688,"data_stored":31319,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56799999999999995}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.495}]}]},{"osd":1,"up_from":8,"seq":34359738372,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27196,"kb_used_data":172,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344644,"statfs":{"total":96636764160,"available":96608915456,"internally_reserved":0,"allocated":176128,"data_stored":621687,"data_compressed":30347,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.42699999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56799999999999995}]}]},{"osd":0,"up_from":8,"seq":34359738373,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27196,"kb_used_data":172,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344644,"statfs":{"total":96636764160,"available":96608915456,"internally_reserved":0,"allocated":176128,"data_stored":621687,"data_compressed":30347,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.40200000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.32100000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":590368,"data_compressed":30347,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":590368,"data_compressed":30347,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-06T09:30:06.875 DEBUG:teuthology.orchestra.run.smithi192:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2024-04-06T09:30:07.144 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-04-06T09:30:07.144 INFO:teuthology.orchestra.run.smithi192.stderr:dumped all 2024-04-06T09:30:07.160 INFO:teuthology.orchestra.run.smithi192.stdout:{"pg_ready":true,"pg_map":{"version":18,"stamp":"2024-04-06T09:30:06.946945+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":81544,"kb_used_data":472,"kb_used_omap":4,"kb_used_meta":81019,"kb_avail":283033976,"statfs":{"total":289910292480,"available":289826791424,"internally_reserved":0,"allocated":483328,"data_stored":1281649,"data_compressed":60694,"data_compressed_allocated":81920,"data_compressed_original":1146880,"omap_allocated":4782,"internal_metadata":82963794},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001534"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":10,"reported_epoch":14,"state":"active+clean","last_fresh":"2024-04-06T09:30:03.388965+0000","last_change":"2024-04-06T09:30:02.384212+0000","last_active":"2024-04-06T09:30:03.388965+0000","last_peered":"2024-04-06T09:30:03.388965+0000","last_clean":"2024-04-06T09:30:03.388965+0000","last_became_active":"2024-04-06T09:30:02.383710+0000","last_became_peered":"2024-04-06T09:30:02.383710+0000","last_unstale":"2024-04-06T09:30:03.388965+0000","last_undegraded":"2024-04-06T09:30:03.388965+0000","last_fullsized":"2024-04-06T09:30:03.388965+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T09:30:01.359377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T09:30:01.359377+0000","last_clean_scrub_stamp":"2024-04-06T09:30:01.359377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T18:51:44.859927+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":10,"reported_epoch":14,"state":"active+clean","last_fresh":"2024-04-06T09:30:03.388895+0000","last_change":"2024-04-06T09:30:02.385797+0000","last_active":"2024-04-06T09:30:03.388895+0000","last_peered":"2024-04-06T09:30:03.388895+0000","last_clean":"2024-04-06T09:30:03.388895+0000","last_became_active":"2024-04-06T09:30:02.385501+0000","last_became_peered":"2024-04-06T09:30:02.385501+0000","last_unstale":"2024-04-06T09:30:03.388895+0000","last_undegraded":"2024-04-06T09:30:03.388895+0000","last_fullsized":"2024-04-06T09:30:03.388895+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T09:30:01.359377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T09:30:01.359377+0000","last_clean_scrub_stamp":"2024-04-06T09:30:01.359377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T15:08:40.865789+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":10,"reported_epoch":14,"state":"active+clean","last_fresh":"2024-04-06T09:30:03.388825+0000","last_change":"2024-04-06T09:30:02.383199+0000","last_active":"2024-04-06T09:30:03.388825+0000","last_peered":"2024-04-06T09:30:03.388825+0000","last_clean":"2024-04-06T09:30:03.388825+0000","last_became_active":"2024-04-06T09:30:02.382798+0000","last_became_peered":"2024-04-06T09:30:02.382798+0000","last_unstale":"2024-04-06T09:30:03.388825+0000","last_undegraded":"2024-04-06T09:30:03.388825+0000","last_fullsized":"2024-04-06T09:30:03.388825+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T09:30:01.359377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T09:30:01.359377+0000","last_clean_scrub_stamp":"2024-04-06T09:30:01.359377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T18:11:34.478290+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":10,"reported_epoch":14,"state":"active+clean","last_fresh":"2024-04-06T09:30:03.388756+0000","last_change":"2024-04-06T09:30:02.383483+0000","last_active":"2024-04-06T09:30:03.388756+0000","last_peered":"2024-04-06T09:30:03.388756+0000","last_clean":"2024-04-06T09:30:03.388756+0000","last_became_active":"2024-04-06T09:30:02.382963+0000","last_became_peered":"2024-04-06T09:30:02.382963+0000","last_unstale":"2024-04-06T09:30:03.388756+0000","last_undegraded":"2024-04-06T09:30:03.388756+0000","last_fullsized":"2024-04-06T09:30:03.388756+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T09:30:01.359377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T09:30:01.359377+0000","last_clean_scrub_stamp":"2024-04-06T09:30:01.359377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T10:19:04.998532+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":10,"reported_epoch":13,"state":"active+clean","last_fresh":"2024-04-06T09:30:03.468197+0000","last_change":"2024-04-06T09:30:02.383104+0000","last_active":"2024-04-06T09:30:03.468197+0000","last_peered":"2024-04-06T09:30:03.468197+0000","last_clean":"2024-04-06T09:30:03.468197+0000","last_became_active":"2024-04-06T09:30:02.382554+0000","last_became_peered":"2024-04-06T09:30:02.382554+0000","last_unstale":"2024-04-06T09:30:03.468197+0000","last_undegraded":"2024-04-06T09:30:03.468197+0000","last_fullsized":"2024-04-06T09:30:03.468197+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T09:30:01.359377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T09:30:01.359377+0000","last_clean_scrub_stamp":"2024-04-06T09:30:01.359377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T18:47:16.568825+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":10,"reported_epoch":13,"state":"active+clean","last_fresh":"2024-04-06T09:30:03.468469+0000","last_change":"2024-04-06T09:30:02.386812+0000","last_active":"2024-04-06T09:30:03.468469+0000","last_peered":"2024-04-06T09:30:03.468469+0000","last_clean":"2024-04-06T09:30:03.468469+0000","last_became_active":"2024-04-06T09:30:02.386256+0000","last_became_peered":"2024-04-06T09:30:02.386256+0000","last_unstale":"2024-04-06T09:30:03.468469+0000","last_undegraded":"2024-04-06T09:30:03.468469+0000","last_fullsized":"2024-04-06T09:30:03.468469+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T09:30:01.359377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T09:30:01.359377+0000","last_clean_scrub_stamp":"2024-04-06T09:30:01.359377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T21:22:07.757399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":10,"reported_epoch":13,"state":"active+clean","last_fresh":"2024-04-06T09:30:03.468447+0000","last_change":"2024-04-06T09:30:02.386748+0000","last_active":"2024-04-06T09:30:03.468447+0000","last_peered":"2024-04-06T09:30:03.468447+0000","last_clean":"2024-04-06T09:30:03.468447+0000","last_became_active":"2024-04-06T09:30:02.386318+0000","last_became_peered":"2024-04-06T09:30:02.386318+0000","last_unstale":"2024-04-06T09:30:03.468447+0000","last_undegraded":"2024-04-06T09:30:03.468447+0000","last_fullsized":"2024-04-06T09:30:03.468447+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T09:30:01.359377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T09:30:01.359377+0000","last_clean_scrub_stamp":"2024-04-06T09:30:01.359377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T10:50:42.918574+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":10,"reported_epoch":14,"state":"active+clean","last_fresh":"2024-04-06T09:30:03.388678+0000","last_change":"2024-04-06T09:30:02.382580+0000","last_active":"2024-04-06T09:30:03.388678+0000","last_peered":"2024-04-06T09:30:03.388678+0000","last_clean":"2024-04-06T09:30:03.388678+0000","last_became_active":"2024-04-06T09:30:02.382103+0000","last_became_peered":"2024-04-06T09:30:02.382103+0000","last_unstale":"2024-04-06T09:30:03.388678+0000","last_undegraded":"2024-04-06T09:30:03.388678+0000","last_fullsized":"2024-04-06T09:30:03.388678+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T09:30:01.359377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T09:30:01.359377+0000","last_clean_scrub_stamp":"2024-04-06T09:30:01.359377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T18:50:24.144774+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"10'97","reported_seq":140,"reported_epoch":13,"state":"active+clean","last_fresh":"2024-04-06T09:30:02.371155+0000","last_change":"2024-04-06T09:29:56.421931+0000","last_active":"2024-04-06T09:30:02.371155+0000","last_peered":"2024-04-06T09:30:02.371155+0000","last_clean":"2024-04-06T09:30:02.371155+0000","last_became_active":"2024-04-06T09:29:56.419754+0000","last_became_peered":"2024-04-06T09:29:56.419754+0000","last_unstale":"2024-04-06T09:30:02.371155+0000","last_undegraded":"2024-04-06T09:30:02.371155+0000","last_fullsized":"2024-04-06T09:30:02.371155+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T09:29:55.213061+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T09:29:55.213061+0000","last_clean_scrub_stamp":"2024-04-06T09:29:55.213061+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T19:28:59.898297+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":122880,"data_stored":1180736,"data_compressed":60694,"data_compressed_allocated":81920,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738374,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27148,"kb_used_data":120,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344692,"statfs":{"total":96636764160,"available":96608964608,"internally_reserved":0,"allocated":122880,"data_stored":34797,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.47499999999999998}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.62}]}]},{"osd":1,"up_from":8,"seq":34359738373,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27200,"kb_used_data":180,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344640,"statfs":{"total":96636764160,"available":96608911360,"internally_reserved":0,"allocated":184320,"data_stored":625165,"data_compressed":30347,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.29399999999999998}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.28100000000000003}]}]},{"osd":0,"up_from":8,"seq":34359738374,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27196,"kb_used_data":172,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344644,"statfs":{"total":96636764160,"available":96608915456,"internally_reserved":0,"allocated":176128,"data_stored":621687,"data_compressed":30347,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.40200000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.32100000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":590368,"data_compressed":30347,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":590368,"data_compressed":30347,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-06T09:30:07.161 INFO:tasks.ceph.ceph_manager.cluster2:clean! 2024-04-06T09:30:07.161 INFO:tasks.ceph:Waiting until ceph cluster cluster2 is healthy... 2024-04-06T09:30:07.161 INFO:tasks.ceph.ceph_manager.cluster2:wait_until_healthy 2024-04-06T09:30:07.161 DEBUG:teuthology.orchestra.run.smithi192:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 health --format=json 2024-04-06T09:30:07.462 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-04-06T09:30:07.462 INFO:teuthology.orchestra.run.smithi192.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-04-06T09:30:07.479 INFO:tasks.ceph.ceph_manager.cluster2:wait_until_healthy done 2024-04-06T09:30:07.479 INFO:teuthology.run_tasks:Running task exec... 2024-04-06T09:30:07.489 INFO:teuthology.task.exec:Executing custom commands... 2024-04-06T09:30:07.489 INFO:teuthology.task.exec:Running commands on role cluster1.client.mirror.0 host ubuntu@smithi192.front.sepia.ceph.com 2024-04-06T09:30:07.490 DEBUG:teuthology.orchestra.run.smithi192:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo ceph --cluster cluster1 auth caps client.mirror mon '"'"'profile rbd-mirror-peer'"'"' osd '"'"'profile rbd'"'"'' 2024-04-06T09:30:07.844 INFO:teuthology.orchestra.run.smithi192.stderr:updated caps for client.mirror 2024-04-06T09:30:07.861 DEBUG:teuthology.orchestra.run.smithi192:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo ceph --cluster cluster1 auth caps client.mirror.0 mon '"'"'profile rbd-mirror'"'"' osd '"'"'profile rbd'"'"'' 2024-04-06T09:30:08.255 INFO:teuthology.orchestra.run.smithi192.stderr:updated caps for client.mirror.0 2024-04-06T09:30:08.274 DEBUG:teuthology.orchestra.run.smithi192:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo ceph --cluster cluster1 auth caps client.mirror.1 mon '"'"'profile rbd-mirror'"'"' osd '"'"'profile rbd'"'"'' 2024-04-06T09:30:08.642 INFO:teuthology.orchestra.run.smithi192.stderr:updated caps for client.mirror.1 2024-04-06T09:30:08.662 DEBUG:teuthology.orchestra.run.smithi192:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo ceph --cluster cluster1 auth caps client.mirror.2 mon '"'"'profile rbd-mirror'"'"' osd '"'"'profile rbd'"'"'' 2024-04-06T09:30:09.037 INFO:teuthology.orchestra.run.smithi192.stderr:updated caps for client.mirror.2 2024-04-06T09:30:09.057 DEBUG:teuthology.orchestra.run.smithi192:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo ceph --cluster cluster1 auth caps client.mirror.3 mon '"'"'profile rbd-mirror'"'"' osd '"'"'profile rbd'"'"'' 2024-04-06T09:30:09.402 INFO:teuthology.orchestra.run.smithi192.stderr:updated caps for client.mirror.3 2024-04-06T09:30:09.420 DEBUG:teuthology.orchestra.run.smithi192:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo ceph --cluster cluster2 auth caps client.mirror mon '"'"'profile rbd-mirror-peer'"'"' osd '"'"'profile rbd'"'"'' 2024-04-06T09:30:09.749 INFO:teuthology.orchestra.run.smithi192.stderr:updated caps for client.mirror 2024-04-06T09:30:09.767 DEBUG:teuthology.orchestra.run.smithi192:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo ceph --cluster cluster2 auth caps client.mirror.0 mon '"'"'profile rbd-mirror'"'"' osd '"'"'profile rbd'"'"'' 2024-04-06T09:30:10.120 INFO:teuthology.orchestra.run.smithi192.stderr:updated caps for client.mirror.0 2024-04-06T09:30:10.139 DEBUG:teuthology.orchestra.run.smithi192:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo ceph --cluster cluster2 auth caps client.mirror.1 mon '"'"'profile rbd-mirror'"'"' osd '"'"'profile rbd'"'"'' 2024-04-06T09:30:10.503 INFO:teuthology.orchestra.run.smithi192.stderr:updated caps for client.mirror.1 2024-04-06T09:30:10.522 DEBUG:teuthology.orchestra.run.smithi192:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo ceph --cluster cluster2 auth caps client.mirror.2 mon '"'"'profile rbd-mirror'"'"' osd '"'"'profile rbd'"'"'' 2024-04-06T09:30:10.865 INFO:teuthology.orchestra.run.smithi192.stderr:updated caps for client.mirror.2 2024-04-06T09:30:10.883 DEBUG:teuthology.orchestra.run.smithi192:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo ceph --cluster cluster2 auth caps client.mirror.3 mon '"'"'profile rbd-mirror'"'"' osd '"'"'profile rbd'"'"'' 2024-04-06T09:30:11.225 INFO:teuthology.orchestra.run.smithi192.stderr:updated caps for client.mirror.3 2024-04-06T09:30:11.243 INFO:teuthology.run_tasks:Running task rbd-mirror... 2024-04-06T09:30:11.253 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Restarting daemon 2024-04-06T09:30:11.253 DEBUG:teuthology.orchestra.run.smithi192:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.0 2024-04-06T09:30:11.257 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Started 2024-04-06T09:30:11.257 INFO:teuthology.run_tasks:Running task rbd-mirror... 2024-04-06T09:30:11.265 INFO:tasks.rbd_mirror.cluster1.client.mirror.1:Restarting daemon 2024-04-06T09:30:11.266 DEBUG:teuthology.orchestra.run.smithi192:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.1 2024-04-06T09:30:11.267 INFO:tasks.rbd_mirror.cluster1.client.mirror.1:Started 2024-04-06T09:30:11.268 INFO:teuthology.run_tasks:Running task rbd-mirror... 2024-04-06T09:30:11.275 INFO:tasks.rbd_mirror.cluster1.client.mirror.2:Restarting daemon 2024-04-06T09:30:11.275 DEBUG:teuthology.orchestra.run.smithi192:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.2 2024-04-06T09:30:11.277 INFO:tasks.rbd_mirror.cluster1.client.mirror.2:Started 2024-04-06T09:30:11.277 INFO:teuthology.run_tasks:Running task rbd-mirror... 2024-04-06T09:30:11.285 INFO:tasks.rbd_mirror.cluster1.client.mirror.3:Restarting daemon 2024-04-06T09:30:11.285 DEBUG:teuthology.orchestra.run.smithi192:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.3 2024-04-06T09:30:11.287 INFO:tasks.rbd_mirror.cluster1.client.mirror.3:Started 2024-04-06T09:30:11.287 INFO:teuthology.run_tasks:Running task rbd-mirror... 2024-04-06T09:30:11.296 INFO:tasks.rbd_mirror.cluster2.client.mirror.0:Restarting daemon 2024-04-06T09:30:11.296 DEBUG:teuthology.orchestra.run.smithi192:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.0 2024-04-06T09:30:11.298 INFO:tasks.rbd_mirror.cluster2.client.mirror.0:Started 2024-04-06T09:30:11.298 INFO:teuthology.run_tasks:Running task rbd-mirror... 2024-04-06T09:30:11.307 INFO:tasks.rbd_mirror.cluster2.client.mirror.1:Restarting daemon 2024-04-06T09:30:11.307 DEBUG:teuthology.orchestra.run.smithi192:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.1 2024-04-06T09:30:11.309 INFO:tasks.rbd_mirror.cluster2.client.mirror.1:Started 2024-04-06T09:30:11.309 INFO:teuthology.run_tasks:Running task rbd-mirror... 2024-04-06T09:30:11.318 INFO:tasks.rbd_mirror.cluster2.client.mirror.2:Restarting daemon 2024-04-06T09:30:11.318 DEBUG:teuthology.orchestra.run.smithi192:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.2 2024-04-06T09:30:11.321 INFO:tasks.rbd_mirror.cluster2.client.mirror.2:Started 2024-04-06T09:30:11.321 INFO:teuthology.run_tasks:Running task rbd-mirror... 2024-04-06T09:30:11.330 INFO:tasks.rbd_mirror.cluster2.client.mirror.3:Restarting daemon 2024-04-06T09:30:11.330 DEBUG:teuthology.orchestra.run.smithi192:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.3 2024-04-06T09:30:11.332 INFO:tasks.rbd_mirror.cluster2.client.mirror.3:Started 2024-04-06T09:30:11.332 INFO:teuthology.run_tasks:Running task rbd-mirror-thrash... 2024-04-06T09:30:11.343 INFO:tasks.rbd_mirror_thrash:rbd_mirror_thrash using random seed: 1712395811 2024-04-06T09:30:11.344 DEBUG:tasks.rbd_mirror_thrash:Yielding 2024-04-06T09:30:11.344 INFO:teuthology.run_tasks:Running task rbd-mirror-thrash... 2024-04-06T09:30:11.353 INFO:tasks.rbd_mirror_thrash:rbd_mirror_thrash using random seed: 1712395811 2024-04-06T09:30:11.353 DEBUG:tasks.rbd_mirror_thrash:Yielding 2024-04-06T09:30:11.353 INFO:teuthology.run_tasks:Running task workunit... 2024-04-06T09:30:11.365 INFO:tasks.workunit:Pulling workunits from ref 052adcadd0534eddb0d73c5afbc083a84cd9dc46 2024-04-06T09:30:11.365 INFO:tasks.workunit:Making a separate scratch dir for every client... 2024-04-06T09:30:11.365 DEBUG:teuthology.orchestra.run.smithi192:> stat -- /home/ubuntu/cephtest/mnt.cluster1.mirror 2024-04-06T09:30:11.366 INFO:tasks.rbd_mirror_thrash:starting thrash for cluster cluster1 2024-04-06T09:30:11.366 INFO:tasks.rbd_mirror_thrash:waiting for 98 secs before thrashing 2024-04-06T09:30:11.366 INFO:tasks.rbd_mirror_thrash:starting thrash for cluster cluster2 2024-04-06T09:30:11.366 INFO:tasks.rbd_mirror_thrash:waiting for 64 secs before thrashing 2024-04-06T09:30:11.375 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T09:30:11.376 INFO:teuthology.orchestra.run.smithi192.stderr:stat: cannot statx '/home/ubuntu/cephtest/mnt.cluster1.mirror': No such file or directory 2024-04-06T09:30:11.376 DEBUG:teuthology.orchestra.run.smithi192:> mkdir -- /home/ubuntu/cephtest/mnt.cluster1.mirror 2024-04-06T09:30:11.425 INFO:tasks.workunit:Created dir /home/ubuntu/cephtest/mnt.cluster1.mirror 2024-04-06T09:30:11.426 DEBUG:teuthology.orchestra.run.smithi192:> cd -- /home/ubuntu/cephtest/mnt.cluster1.mirror && mkdir -- client.mirror 2024-04-06T09:30:11.473 INFO:tasks.workunit:timeout=6h 2024-04-06T09:30:11.473 INFO:tasks.workunit:cleanup=True 2024-04-06T09:30:11.474 DEBUG:teuthology.orchestra.run.smithi192:> rm -rf /home/ubuntu/cephtest/clone.cluster1.client.mirror && git clone https://git.ceph.com/ceph.git /home/ubuntu/cephtest/clone.cluster1.client.mirror && cd /home/ubuntu/cephtest/clone.cluster1.client.mirror && git checkout 052adcadd0534eddb0d73c5afbc083a84cd9dc46 2024-04-06T09:30:11.523 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:Cloning into '/home/ubuntu/cephtest/clone.cluster1.client.mirror'... 2024-04-06T09:31:15.367 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.0 with skip (0.9878240555382581) > weight (0.25) 2024-04-06T09:31:15.368 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.1 with skip (0.3959070137801587) > weight (0.25) 2024-04-06T09:31:15.368 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.2 with skip (0.3823934608661621) > weight (0.25) 2024-04-06T09:31:15.368 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.3 with skip (0.8158628966384064) > weight (0.25) 2024-04-06T09:31:15.368 INFO:tasks.rbd_mirror_thrash:waiting for 75 secs before thrashing 2024-04-06T09:31:39.191 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:Note: switching to '052adcadd0534eddb0d73c5afbc083a84cd9dc46'. 2024-04-06T09:31:39.191 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 2024-04-06T09:31:39.191 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:You are in 'detached HEAD' state. You can look around, make experimental 2024-04-06T09:31:39.191 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:changes and commit them, and you can discard any commits you make in this 2024-04-06T09:31:39.192 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:state without impacting any branches by switching back to a branch. 2024-04-06T09:31:39.192 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 2024-04-06T09:31:39.192 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:If you want to create a new branch to retain commits you create, you may 2024-04-06T09:31:39.192 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:do so (now or later) by using -c with the switch command. Example: 2024-04-06T09:31:39.192 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 2024-04-06T09:31:39.192 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: git switch -c 2024-04-06T09:31:39.192 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 2024-04-06T09:31:39.192 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:Or undo this operation with: 2024-04-06T09:31:39.192 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 2024-04-06T09:31:39.192 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: git switch - 2024-04-06T09:31:39.192 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 2024-04-06T09:31:39.192 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:Turn off this advice by setting config variable advice.detachedHead to false 2024-04-06T09:31:39.192 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 2024-04-06T09:31:39.192 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:HEAD is now at 052adcadd05 Merge PR #56567 into reef 2024-04-06T09:31:39.198 DEBUG:teuthology.orchestra.run.smithi192:> cd -- /home/ubuntu/cephtest/clone.cluster1.client.mirror/qa/workunits && if test -e Makefile ; then make ; fi && find -executable -type f -printf '%P\0' >/home/ubuntu/cephtest/workunits.list.cluster1.client.mirror 2024-04-06T09:31:39.246 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:for d in direct_io fs ; do ( cd $d ; make all ) ; done 2024-04-06T09:31:39.248 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.cluster1.client.mirror/qa/workunits/direct_io' 2024-04-06T09:31:39.248 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:cc -Wall -Wextra -D_GNU_SOURCE direct_io_test.c -o direct_io_test 2024-04-06T09:31:40.009 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_sync_io.c -o test_sync_io 2024-04-06T09:31:40.074 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_short_dio_read.c -o test_short_dio_read 2024-04-06T09:31:40.125 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.cluster1.client.mirror/qa/workunits/direct_io' 2024-04-06T09:31:40.126 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.cluster1.client.mirror/qa/workunits/fs' 2024-04-06T09:31:40.126 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_o_trunc.c -o test_o_trunc 2024-04-06T09:31:40.169 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.cluster1.client.mirror/qa/workunits/fs' 2024-04-06T09:31:40.172 DEBUG:teuthology.orchestra.run.smithi192:> set -ex 2024-04-06T09:31:40.172 DEBUG:teuthology.orchestra.run.smithi192:> dd if=/home/ubuntu/cephtest/workunits.list.cluster1.client.mirror of=/dev/stdout 2024-04-06T09:31:40.220 INFO:tasks.workunit:Running workunits matching rbd/rbd_mirror_stress.sh on cluster1.client.mirror... 2024-04-06T09:31:40.222 INFO:tasks.workunit:Running workunit rbd/rbd_mirror_stress.sh... 2024-04-06T09:31:40.222 DEBUG:teuthology.orchestra.run.smithi192:workunit test rbd/rbd_mirror_stress.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp && cd -- /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=052adcadd0534eddb0d73c5afbc083a84cd9dc46 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster cluster1" CEPH_ID="mirror" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.cluster1.client.mirror CEPH_ROOT=/home/ubuntu/cephtest/clone.cluster1.client.mirror CEPH_MNT=/home/ubuntu/cephtest/mnt.cluster1.mirror CEPH_ARGS='' MIRROR_IMAGE_MODE=journal MIRROR_POOL_MODE=pool RBD_MIRROR_INSTANCES=4 RBD_MIRROR_USE_EXISTING_CLUSTER=1 RBD_MIRROR_USE_RBD_MIRROR=1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 6h /home/ubuntu/cephtest/clone.cluster1.client.mirror/qa/workunits/rbd/rbd_mirror_stress.sh 2024-04-06T09:31:40.272 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ IMAGE_COUNT=50 2024-04-06T09:31:40.273 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ export LOCKDEP=0 2024-04-06T09:31:40.273 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ LOCKDEP=0 2024-04-06T09:31:40.273 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ dirname /home/ubuntu/cephtest/clone.cluster1.client.mirror/qa/workunits/rbd/rbd_mirror_stress.sh 2024-04-06T09:31:40.274 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ . /home/ubuntu/cephtest/clone.cluster1.client.mirror/qa/workunits/rbd/rbd_mirror_helpers.sh 2024-04-06T09:31:40.274 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ type xmlstarlet 2024-04-06T09:31:40.275 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ XMLSTARLET=xmlstarlet 2024-04-06T09:31:40.275 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ RBD_MIRROR_INSTANCES=4 2024-04-06T09:31:40.275 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ CLUSTER1=cluster1 2024-04-06T09:31:40.275 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ CLUSTER2=cluster2 2024-04-06T09:31:40.275 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ PEER_CLUSTER_SUFFIX= 2024-04-06T09:31:40.275 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ POOL=mirror 2024-04-06T09:31:40.275 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ PARENT_POOL=mirror_parent 2024-04-06T09:31:40.275 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ NS1=ns1 2024-04-06T09:31:40.275 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ NS2=ns2 2024-04-06T09:31:40.275 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ TEMPDIR= 2024-04-06T09:31:40.275 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ CEPH_ID=mirror 2024-04-06T09:31:40.275 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ RBD_IMAGE_FEATURES=layering,exclusive-lock,journaling 2024-04-06T09:31:40.275 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ MIRROR_USER_ID_PREFIX=mirror. 2024-04-06T09:31:40.275 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ MIRROR_POOL_MODE=pool 2024-04-06T09:31:40.275 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ MIRROR_IMAGE_MODE=journal 2024-04-06T09:31:40.276 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ export 'CEPH_ARGS=--id mirror' 2024-04-06T09:31:40.276 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ CEPH_ARGS='--id mirror' 2024-04-06T09:31:40.276 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ LAST_MIRROR_INSTANCE=3 2024-04-06T09:31:40.276 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++++ dirname /home/ubuntu/cephtest/clone.cluster1.client.mirror/qa/workunits/rbd/rbd_mirror_stress.sh 2024-04-06T09:31:40.276 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ readlink -f /home/ubuntu/cephtest/clone.cluster1.client.mirror/qa/workunits/rbd/../../../src 2024-04-06T09:31:40.277 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ CEPH_ROOT=/home/ubuntu/cephtest/clone.cluster1.client.mirror/src 2024-04-06T09:31:40.277 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ CEPH_BIN=. 2024-04-06T09:31:40.277 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ CEPH_SRC=. 2024-04-06T09:31:40.277 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' -e CMakeCache.txt ']' 2024-04-06T09:31:40.279 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 -gt 0 ']' 2024-04-06T09:31:40.279 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ setup 2024-04-06T09:31:40.279 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local c 2024-04-06T09:31:40.279 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ trap 'cleanup $?' INT TERM EXIT 2024-04-06T09:31:40.279 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ setup_tempdir 2024-04-06T09:31:40.279 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n '' ']' 2024-04-06T09:31:40.279 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ mktemp -d 2024-04-06T09:31:40.280 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ TEMPDIR=/tmp/tmp.BhW6xKmROU 2024-04-06T09:31:40.280 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -z 1 ']' 2024-04-06T09:31:40.280 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ setup_pools cluster1 cluster2 2024-04-06T09:31:40.280 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:31:40.280 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local remote_cluster=cluster2 2024-04-06T09:31:40.280 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local mon_map_file 2024-04-06T09:31:40.280 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local mon_addr 2024-04-06T09:31:40.281 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local admin_key_file 2024-04-06T09:31:40.281 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local uuid 2024-04-06T09:31:40.281 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ CEPH_ARGS= 2024-04-06T09:31:40.281 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --cluster cluster1 osd pool create mirror 64 64 2024-04-06T09:31:41.597 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:pool 'mirror' already exists 2024-04-06T09:31:41.615 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ CEPH_ARGS= 2024-04-06T09:31:41.615 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --cluster cluster1 osd pool create mirror_parent 64 64 2024-04-06T09:31:42.596 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:pool 'mirror_parent' already exists 2024-04-06T09:31:42.615 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ CEPH_ARGS= 2024-04-06T09:31:42.615 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 pool init mirror 2024-04-06T09:31:45.664 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ CEPH_ARGS= 2024-04-06T09:31:45.664 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 pool init mirror_parent 2024-04-06T09:31:48.566 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n '' ']' 2024-04-06T09:31:48.566 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ CEPH_ARGS= 2024-04-06T09:31:48.566 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror pool enable --site-name cluster1 mirror pool 2024-04-06T09:31:48.647 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror pool enable mirror_parent image 2024-04-06T09:31:48.699 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 namespace create mirror/ns1 2024-04-06T09:31:48.745 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 namespace create mirror/ns2 2024-04-06T09:31:48.789 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror pool enable mirror/ns1 pool 2024-04-06T09:31:48.842 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror pool enable mirror/ns2 image 2024-04-06T09:31:48.890 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -z ']' 2024-04-06T09:31:48.890 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -z ']' 2024-04-06T09:31:48.890 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ peer_add cluster1 mirror cluster2 2024-04-06T09:31:48.890 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:31:48.890 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:31:48.890 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:31:48.890 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:31:48.890 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local client_cluster=cluster2 2024-04-06T09:31:48.891 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:31:48.891 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local remote_cluster=cluster2 2024-04-06T09:31:48.891 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local uuid_var_name 2024-04-06T09:31:48.891 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n '' ']' 2024-04-06T09:31:48.891 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local error_code 2024-04-06T09:31:48.891 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local peer_uuid 2024-04-06T09:31:48.891 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 1 2 4 8 16 32 2024-04-06T09:31:48.891 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set +e 2024-04-06T09:31:48.891 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster1 mirror pool peer add mirror cluster2 2024-04-06T09:31:48.933 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ peer_uuid=1b7e1297-4cd3-47aa-9668-0ab45677d6c6 2024-04-06T09:31:48.933 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ error_code=0 2024-04-06T09:31:48.933 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set -e 2024-04-06T09:31:48.933 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 -eq 17 ']' 2024-04-06T09:31:48.933 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 0 -eq 0 2024-04-06T09:31:48.933 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n '' ']' 2024-04-06T09:31:48.933 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:31:48.933 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ peer_add cluster1 mirror_parent cluster2 2024-04-06T09:31:48.933 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:31:48.934 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:31:48.934 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror_parent 2024-04-06T09:31:48.934 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:31:48.934 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local client_cluster=cluster2 2024-04-06T09:31:48.934 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:31:48.934 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local remote_cluster=cluster2 2024-04-06T09:31:48.934 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local uuid_var_name 2024-04-06T09:31:48.934 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n '' ']' 2024-04-06T09:31:48.934 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local error_code 2024-04-06T09:31:48.934 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local peer_uuid 2024-04-06T09:31:48.934 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 1 2 4 8 16 32 2024-04-06T09:31:48.934 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set +e 2024-04-06T09:31:48.934 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster1 mirror pool peer add mirror_parent cluster2 2024-04-06T09:31:48.977 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ peer_uuid=faff099b-d0ba-41d6-96e0-1d372b0ff8db 2024-04-06T09:31:48.977 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ error_code=0 2024-04-06T09:31:48.977 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set -e 2024-04-06T09:31:48.977 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 -eq 17 ']' 2024-04-06T09:31:48.977 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 0 -eq 0 2024-04-06T09:31:48.977 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n '' ']' 2024-04-06T09:31:48.977 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:31:48.977 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ setup_pools cluster2 cluster1 2024-04-06T09:31:48.977 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-04-06T09:31:48.977 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local remote_cluster=cluster1 2024-04-06T09:31:48.978 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local mon_map_file 2024-04-06T09:31:48.978 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local mon_addr 2024-04-06T09:31:48.978 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local admin_key_file 2024-04-06T09:31:48.978 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local uuid 2024-04-06T09:31:48.978 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ CEPH_ARGS= 2024-04-06T09:31:48.978 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --cluster cluster2 osd pool create mirror 64 64 2024-04-06T09:31:49.366 INFO:tasks.rbd_mirror_thrash:kill cluster1.client.mirror.0 2024-04-06T09:31:49.367 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Sent signal 15 2024-04-06T09:31:49.367 INFO:tasks.rbd_mirror_thrash:waiting for 8 secs before reviving daemons 2024-04-06T09:31:49.368 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi192.stderr:2024-04-06T09:31:49.366+0000 7feb4bd8f640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.0 (PID: 27472) UID: 1000 2024-04-06T09:31:49.521 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:pool 'mirror' already exists 2024-04-06T09:31:49.537 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ CEPH_ARGS= 2024-04-06T09:31:49.538 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --cluster cluster2 osd pool create mirror_parent 64 64 2024-04-06T09:31:50.561 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:pool 'mirror_parent' already exists 2024-04-06T09:31:50.579 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ CEPH_ARGS= 2024-04-06T09:31:50.580 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster2 pool init mirror 2024-04-06T09:31:53.476 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ CEPH_ARGS= 2024-04-06T09:31:53.476 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster2 pool init mirror_parent 2024-04-06T09:31:56.486 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n '' ']' 2024-04-06T09:31:56.487 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ CEPH_ARGS= 2024-04-06T09:31:56.487 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster2 mirror pool enable --site-name cluster2 mirror pool 2024-04-06T09:31:56.549 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster2 mirror pool enable mirror_parent image 2024-04-06T09:31:56.590 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster2 namespace create mirror/ns1 2024-04-06T09:31:56.626 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster2 namespace create mirror/ns2 2024-04-06T09:31:56.664 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster2 mirror pool enable mirror/ns1 pool 2024-04-06T09:31:56.707 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster2 mirror pool enable mirror/ns2 image 2024-04-06T09:31:56.747 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -z ']' 2024-04-06T09:31:56.747 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -z ']' 2024-04-06T09:31:56.747 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ peer_add cluster2 mirror cluster1 2024-04-06T09:31:56.747 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-04-06T09:31:56.748 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:31:56.748 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:31:56.748 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:31:56.748 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local client_cluster=cluster1 2024-04-06T09:31:56.748 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:31:56.748 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local remote_cluster=cluster1 2024-04-06T09:31:56.748 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local uuid_var_name 2024-04-06T09:31:56.748 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n '' ']' 2024-04-06T09:31:56.748 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local error_code 2024-04-06T09:31:56.748 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local peer_uuid 2024-04-06T09:31:56.748 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 1 2 4 8 16 32 2024-04-06T09:31:56.748 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set +e 2024-04-06T09:31:56.748 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 mirror pool peer add mirror cluster1 2024-04-06T09:31:56.786 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ peer_uuid=cbb8d01d-2fb8-40fe-8df2-48f68a8217ab 2024-04-06T09:31:56.786 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ error_code=0 2024-04-06T09:31:56.787 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set -e 2024-04-06T09:31:56.787 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 -eq 17 ']' 2024-04-06T09:31:56.787 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 0 -eq 0 2024-04-06T09:31:56.787 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n '' ']' 2024-04-06T09:31:56.787 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:31:56.787 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ peer_add cluster2 mirror_parent cluster1 2024-04-06T09:31:56.787 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-04-06T09:31:56.787 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:31:56.787 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror_parent 2024-04-06T09:31:56.787 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:31:56.787 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local client_cluster=cluster1 2024-04-06T09:31:56.787 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:31:56.787 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local remote_cluster=cluster1 2024-04-06T09:31:56.787 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local uuid_var_name 2024-04-06T09:31:56.788 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n '' ']' 2024-04-06T09:31:56.788 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local error_code 2024-04-06T09:31:56.788 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local peer_uuid 2024-04-06T09:31:56.788 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 1 2 4 8 16 32 2024-04-06T09:31:56.788 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set +e 2024-04-06T09:31:56.788 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 mirror pool peer add mirror_parent cluster1 2024-04-06T09:31:56.824 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ peer_uuid=cfc9cb33-92e1-478a-8b30-ab04223b31f1 2024-04-06T09:31:56.825 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ error_code=0 2024-04-06T09:31:56.825 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set -e 2024-04-06T09:31:56.825 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 -eq 17 ']' 2024-04-06T09:31:56.825 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 0 -eq 0 2024-04-06T09:31:56.825 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n '' ']' 2024-04-06T09:31:56.825 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:31:56.825 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n '' ']' 2024-04-06T09:31:56.825 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ start_mirrors cluster1 2024-04-06T09:31:56.825 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:31:56.825 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:31:56.933 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:31:56.933 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ start_mirror cluster1:0 2024-04-06T09:31:56.934 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:31:56.934 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:31:56.934 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:31:56.934 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:31:56.934 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:31:56.934 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:31:56.934 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:31:56.934 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:31:56.934 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:31:56.934 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:31:56.935 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:31:56.935 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:31:56.935 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:31:56.935 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 1 2024-04-06T09:31:56.935 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 2024-04-06T09:31:56.935 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:31:56.935 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ start_mirror cluster1:1 2024-04-06T09:31:56.935 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:1 2024-04-06T09:31:56.935 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:31:56.935 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-06T09:31:56.935 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:1 2024-04-06T09:31:56.935 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:31:56.936 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:31:56.936 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:31:56.936 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=1 2024-04-06T09:31:56.936 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-06T09:31:56.936 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:31:56.936 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:31:56.936 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=1 2024-04-06T09:31:56.936 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=1 2024-04-06T09:31:56.936 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 1 2024-04-06T09:31:56.936 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 2024-04-06T09:31:56.936 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:31:56.936 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ start_mirror cluster1:2 2024-04-06T09:31:56.936 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:2 2024-04-06T09:31:56.937 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:31:56.937 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:2 cluster instance 2024-04-06T09:31:56.937 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:2 2024-04-06T09:31:56.937 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:31:56.937 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:31:56.937 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:31:56.937 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=2 2024-04-06T09:31:56.937 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 2 = cluster1:2 ']' 2024-04-06T09:31:56.937 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:31:56.937 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:31:56.938 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=2 2024-04-06T09:31:56.938 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=2 2024-04-06T09:31:56.938 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 1 2024-04-06T09:31:56.938 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 2024-04-06T09:31:56.938 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:31:56.938 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ start_mirror cluster1:3 2024-04-06T09:31:56.938 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:3 2024-04-06T09:31:56.938 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:31:56.938 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:3 cluster instance 2024-04-06T09:31:56.938 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:3 2024-04-06T09:31:56.938 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:31:56.938 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:31:56.939 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:31:56.939 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=3 2024-04-06T09:31:56.939 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 3 = cluster1:3 ']' 2024-04-06T09:31:56.939 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:31:56.939 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:31:56.939 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=3 2024-04-06T09:31:56.939 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=3 2024-04-06T09:31:56.939 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 1 2024-04-06T09:31:56.939 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 2024-04-06T09:31:56.939 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ start_mirrors cluster2 2024-04-06T09:31:56.939 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-04-06T09:31:56.939 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:31:56.940 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:31:56.940 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ start_mirror cluster2:0 2024-04-06T09:31:56.940 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2:0 2024-04-06T09:31:56.940 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:31:56.940 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster2:0 cluster instance 2024-04-06T09:31:56.940 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster2:0 2024-04-06T09:31:56.940 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:31:56.940 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:31:56.940 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster2 2024-04-06T09:31:56.940 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:31:56.940 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster2:0 ']' 2024-04-06T09:31:56.940 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster2 2024-04-06T09:31:56.940 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster2 2024-04-06T09:31:56.940 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:31:56.941 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:31:56.941 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 1 2024-04-06T09:31:56.941 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 2024-04-06T09:31:56.941 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:31:56.941 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ start_mirror cluster2:1 2024-04-06T09:31:56.941 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2:1 2024-04-06T09:31:56.941 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:31:56.941 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster2:1 cluster instance 2024-04-06T09:31:56.941 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster2:1 2024-04-06T09:31:56.941 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:31:56.941 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:31:56.941 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster2 2024-04-06T09:31:56.941 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=1 2024-04-06T09:31:56.942 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 1 = cluster2:1 ']' 2024-04-06T09:31:56.942 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster2 2024-04-06T09:31:56.942 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster2 2024-04-06T09:31:56.942 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=1 2024-04-06T09:31:56.942 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=1 2024-04-06T09:31:56.942 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 1 2024-04-06T09:31:56.942 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 2024-04-06T09:31:56.942 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:31:56.942 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ start_mirror cluster2:2 2024-04-06T09:31:56.942 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2:2 2024-04-06T09:31:56.942 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:31:56.943 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster2:2 cluster instance 2024-04-06T09:31:56.943 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster2:2 2024-04-06T09:31:56.943 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:31:56.943 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:31:56.943 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster2 2024-04-06T09:31:56.943 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=2 2024-04-06T09:31:56.943 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 2 = cluster2:2 ']' 2024-04-06T09:31:56.943 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster2 2024-04-06T09:31:56.943 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster2 2024-04-06T09:31:56.943 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=2 2024-04-06T09:31:56.944 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=2 2024-04-06T09:31:56.944 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 1 2024-04-06T09:31:56.944 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 2024-04-06T09:31:56.944 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:31:56.944 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ start_mirror cluster2:3 2024-04-06T09:31:56.944 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2:3 2024-04-06T09:31:56.944 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:31:56.944 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster2:3 cluster instance 2024-04-06T09:31:56.944 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster2:3 2024-04-06T09:31:56.944 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:31:56.944 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:31:56.944 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster2 2024-04-06T09:31:56.945 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=3 2024-04-06T09:31:56.945 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 3 = cluster2:3 ']' 2024-04-06T09:31:56.945 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster2 2024-04-06T09:31:56.945 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster2 2024-04-06T09:31:56.945 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=3 2024-04-06T09:31:56.945 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=3 2024-04-06T09:31:56.945 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 1 2024-04-06T09:31:56.945 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 2024-04-06T09:31:56.945 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ testlog 'TEST: add image and test replay after client crashes' 2024-04-06T09:31:56.945 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ tee -a /tmp/tmp.BhW6xKmROU/rbd-mirror.test.log 2024-04-06T09:31:56.945 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ date '+%F %T' 2024-04-06T09:31:56.945 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ echo 2024-04-06 09:31:56 TEST: add image and test replay after client crashes 2024-04-06T09:31:56.945 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:2024-04-06 09:31:56 TEST: add image and test replay after client crashes 2024-04-06T09:31:56.946 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ image=test 2024-04-06T09:31:56.946 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ create_image_and_enable_mirror cluster2 mirror test journal 512M 2024-04-06T09:31:56.946 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-04-06T09:31:56.946 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:31:56.946 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:31:56.946 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:31:56.946 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:31:56.946 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:31:56.946 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local mode=journal 2024-04-06T09:31:56.946 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n journal ']' 2024-04-06T09:31:56.946 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:31:56.946 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ create_image cluster2 mirror test 512M 2024-04-06T09:31:56.946 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-04-06T09:31:56.946 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:31:56.947 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:31:56.947 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:31:56.947 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:31:56.947 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:31:56.947 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local size=128 2024-04-06T09:31:56.947 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n 512M ']' 2024-04-06T09:31:56.947 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ size=512M 2024-04-06T09:31:56.947 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:31:56.947 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster2 create --size 512M --image-feature layering,exclusive-lock,journaling mirror/test 2024-04-06T09:31:56.990 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' pool = image ']' 2024-04-06T09:31:56.990 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' mirror = mirror_parent ']' 2024-04-06T09:31:56.990 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_image_replay_started cluster1 mirror test 2024-04-06T09:31:56.990 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:31:56.991 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:31:56.991 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:31:56.991 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_image_replay_state cluster1 mirror test started 2024-04-06T09:31:56.991 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:31:56.991 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:31:56.991 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:31:56.991 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state=started 2024-04-06T09:31:56.991 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local s 2024-04-06T09:31:56.991 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:31:56.991 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 1 2024-04-06T09:31:57.369 INFO:tasks.rbd_mirror_thrash:waiting for cluster1.client.mirror.0 2024-04-06T09:31:57.369 INFO:teuthology.orchestra.run:waiting for 600 2024-04-06T09:31:57.369 INFO:tasks.rbd_mirror_thrash:reviving cluster1.client.mirror.0 2024-04-06T09:31:57.369 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Restarting daemon 2024-04-06T09:31:57.369 DEBUG:teuthology.orchestra.run.smithi192:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.0 2024-04-06T09:31:57.372 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Started 2024-04-06T09:31:57.372 INFO:tasks.rbd_mirror_thrash:waiting for 117 secs before thrashing 2024-04-06T09:31:57.991 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test_image_replay_state cluster1 mirror test started 2024-04-06T09:31:57.991 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:31:57.992 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:31:57.992 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:31:57.992 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local test_state=started 2024-04-06T09:31:57.992 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local status_result 2024-04-06T09:31:57.992 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local current_state=stopped 2024-04-06T09:31:57.993 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2024-04-06T09:31:57.993 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -i state 2024-04-06T09:31:57.993 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_instance=cluster1 2024-04-06T09:31:57.993 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ shift 2024-04-06T09:31:57.993 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:31:57.993 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance=cluster1 2024-04-06T09:31:57.993 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local loop_instance 2024-04-06T09:31:57.993 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:31:57.994 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sleep 0 2024-04-06T09:31:57.994 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:31:57.994 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ seq 0 3 2024-04-06T09:31:57.997 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:31:57.997 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2024-04-06T09:31:57.997 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1:0 2024-04-06T09:31:57.997 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ shift 2024-04-06T09:31:57.997 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:31:57.997 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:31:57.997 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:31:57.997 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=cluster 2024-04-06T09:31:57.997 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:31:57.997 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:31:57.997 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:31:57.997 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:31:57.998 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval cluster=cluster1 2024-04-06T09:31:57.998 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ cluster=cluster1 2024-04-06T09:31:57.998 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:31:57.998 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:31:57.998 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:31:57.998 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local local_cluster=cluster1:0 2024-04-06T09:31:57.998 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local cluster=cluster1 2024-04-06T09:31:57.998 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local instance 2024-04-06T09:31:57.998 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:31:57.998 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local val=cluster1:0 2024-04-06T09:31:57.998 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local cluster_var_name=local_cluster 2024-04-06T09:31:57.998 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local instance_var_name=instance 2024-04-06T09:31:57.998 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ cluster=cluster1 2024-04-06T09:31:57.998 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:31:57.999 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ '[' 0 = cluster1:0 ']' 2024-04-06T09:31:57.999 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ eval local_cluster=cluster1 2024-04-06T09:31:57.999 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++++ local_cluster=cluster1 2024-04-06T09:31:57.999 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ eval instance=0 2024-04-06T09:31:58.000 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++++ instance=0 2024-04-06T09:31:58.000 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:31:58.020 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:31:58.020 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:31:58.020 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:31:58.021 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2024-04-06T09:31:58.126 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-06T09:31:58.126 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:rbd mirror status : get status for rbd mirror 2024-04-06T09:31:58.126 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:admin_socket: invalid command 2024-04-06T09:31:58.130 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:31:58.130 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/test 2024-04-06T09:31:58.130 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1:1 2024-04-06T09:31:58.130 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ shift 2024-04-06T09:31:58.131 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:31:58.131 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-04-06T09:31:58.131 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:1 2024-04-06T09:31:58.131 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=cluster 2024-04-06T09:31:58.131 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:31:58.131 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:31:58.131 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=1 2024-04-06T09:31:58.131 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-06T09:31:58.131 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval cluster=cluster1 2024-04-06T09:31:58.131 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ cluster=cluster1 2024-04-06T09:31:58.131 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=1 2024-04-06T09:31:58.131 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=1 2024-04-06T09:31:58.131 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-04-06T09:31:58.132 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local local_cluster=cluster1:1 2024-04-06T09:31:58.132 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local cluster=cluster1 2024-04-06T09:31:58.132 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local instance 2024-04-06T09:31:58.132 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-06T09:31:58.132 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local val=cluster1:1 2024-04-06T09:31:58.132 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local cluster_var_name=local_cluster 2024-04-06T09:31:58.132 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local instance_var_name=instance 2024-04-06T09:31:58.132 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ cluster=cluster1 2024-04-06T09:31:58.132 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=1 2024-04-06T09:31:58.132 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ '[' 1 = cluster1:1 ']' 2024-04-06T09:31:58.132 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ eval local_cluster=cluster1 2024-04-06T09:31:58.132 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++++ local_cluster=cluster1 2024-04-06T09:31:58.132 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ eval instance=1 2024-04-06T09:31:58.132 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++++ instance=1 2024-04-06T09:31:58.132 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-06T09:31:58.153 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T09:31:58.153 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T09:31:58.153 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T09:31:58.153 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/test 2024-04-06T09:31:58.248 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-06T09:31:58.249 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:rbd mirror status : get status for rbd mirror 2024-04-06T09:31:58.249 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:admin_socket: invalid command 2024-04-06T09:31:58.252 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:31:58.253 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/test 2024-04-06T09:31:58.253 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1:2 2024-04-06T09:31:58.253 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ shift 2024-04-06T09:31:58.253 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:31:58.253 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:2 cluster instance 2024-04-06T09:31:58.253 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:2 2024-04-06T09:31:58.253 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=cluster 2024-04-06T09:31:58.253 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:31:58.253 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:31:58.253 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=2 2024-04-06T09:31:58.253 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 2 = cluster1:2 ']' 2024-04-06T09:31:58.254 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval cluster=cluster1 2024-04-06T09:31:58.254 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ cluster=cluster1 2024-04-06T09:31:58.254 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=2 2024-04-06T09:31:58.254 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=2 2024-04-06T09:31:58.254 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ daemon_asok_file cluster1:2 cluster1 2024-04-06T09:31:58.254 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local local_cluster=cluster1:2 2024-04-06T09:31:58.254 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local cluster=cluster1 2024-04-06T09:31:58.254 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local instance 2024-04-06T09:31:58.254 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2024-04-06T09:31:58.254 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local val=cluster1:2 2024-04-06T09:31:58.254 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local cluster_var_name=local_cluster 2024-04-06T09:31:58.254 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local instance_var_name=instance 2024-04-06T09:31:58.254 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ cluster=cluster1 2024-04-06T09:31:58.254 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=2 2024-04-06T09:31:58.255 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ '[' 2 = cluster1:2 ']' 2024-04-06T09:31:58.255 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ eval local_cluster=cluster1 2024-04-06T09:31:58.255 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++++ local_cluster=cluster1 2024-04-06T09:31:58.255 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ eval instance=2 2024-04-06T09:31:58.255 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++++ instance=2 2024-04-06T09:31:58.255 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-04-06T09:31:58.276 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-06T09:31:58.277 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-06T09:31:58.277 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-06T09:31:58.277 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/test 2024-04-06T09:31:58.373 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-06T09:31:58.373 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:rbd mirror status : get status for rbd mirror 2024-04-06T09:31:58.373 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:admin_socket: invalid command 2024-04-06T09:31:58.377 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:31:58.377 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/test 2024-04-06T09:31:58.377 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1:3 2024-04-06T09:31:58.377 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ shift 2024-04-06T09:31:58.377 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:31:58.377 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:3 cluster instance 2024-04-06T09:31:58.377 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:3 2024-04-06T09:31:58.377 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=cluster 2024-04-06T09:31:58.377 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:31:58.377 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:31:58.377 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=3 2024-04-06T09:31:58.377 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 3 = cluster1:3 ']' 2024-04-06T09:31:58.378 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval cluster=cluster1 2024-04-06T09:31:58.378 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ cluster=cluster1 2024-04-06T09:31:58.378 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=3 2024-04-06T09:31:58.378 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=3 2024-04-06T09:31:58.378 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ daemon_asok_file cluster1:3 cluster1 2024-04-06T09:31:58.378 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local local_cluster=cluster1:3 2024-04-06T09:31:58.378 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local cluster=cluster1 2024-04-06T09:31:58.378 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local instance 2024-04-06T09:31:58.378 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2024-04-06T09:31:58.378 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local val=cluster1:3 2024-04-06T09:31:58.378 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local cluster_var_name=local_cluster 2024-04-06T09:31:58.378 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local instance_var_name=instance 2024-04-06T09:31:58.378 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ cluster=cluster1 2024-04-06T09:31:58.379 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=3 2024-04-06T09:31:58.379 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ '[' 3 = cluster1:3 ']' 2024-04-06T09:31:58.379 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ eval local_cluster=cluster1 2024-04-06T09:31:58.379 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++++ local_cluster=cluster1 2024-04-06T09:31:58.379 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ eval instance=3 2024-04-06T09:31:58.379 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++++ instance=3 2024-04-06T09:31:58.379 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-04-06T09:31:58.402 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-06T09:31:58.402 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-06T09:31:58.402 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-06T09:31:58.402 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/test 2024-04-06T09:31:58.502 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-06T09:31:58.502 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:rbd mirror status : get status for rbd mirror 2024-04-06T09:31:58.502 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:admin_socket: invalid command 2024-04-06T09:31:58.503 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:31:58.503 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sleep 1 2024-04-06T09:31:59.505 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:31:59.506 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ seq 0 3 2024-04-06T09:31:59.508 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:31:59.508 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2024-04-06T09:31:59.508 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1:0 2024-04-06T09:31:59.508 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ shift 2024-04-06T09:31:59.508 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:31:59.508 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:31:59.508 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:31:59.508 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=cluster 2024-04-06T09:31:59.508 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:31:59.508 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:31:59.508 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:31:59.509 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:31:59.509 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval cluster=cluster1 2024-04-06T09:31:59.509 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ cluster=cluster1 2024-04-06T09:31:59.509 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:31:59.509 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:31:59.509 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:31:59.509 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local local_cluster=cluster1:0 2024-04-06T09:31:59.509 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local cluster=cluster1 2024-04-06T09:31:59.509 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local instance 2024-04-06T09:31:59.509 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:31:59.509 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local val=cluster1:0 2024-04-06T09:31:59.509 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local cluster_var_name=local_cluster 2024-04-06T09:31:59.509 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local instance_var_name=instance 2024-04-06T09:31:59.509 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ cluster=cluster1 2024-04-06T09:31:59.509 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:31:59.510 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ '[' 0 = cluster1:0 ']' 2024-04-06T09:31:59.510 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ eval local_cluster=cluster1 2024-04-06T09:31:59.510 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++++ local_cluster=cluster1 2024-04-06T09:31:59.510 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ eval instance=0 2024-04-06T09:31:59.510 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++++ instance=0 2024-04-06T09:31:59.510 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:31:59.535 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:31:59.535 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:31:59.535 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:31:59.535 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2024-04-06T09:31:59.647 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-06T09:31:59.647 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:rbd mirror status : get status for rbd mirror 2024-04-06T09:31:59.648 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:admin_socket: invalid command 2024-04-06T09:31:59.650 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:31:59.650 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/test 2024-04-06T09:31:59.650 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1:1 2024-04-06T09:31:59.650 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ shift 2024-04-06T09:31:59.650 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:31:59.650 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-04-06T09:31:59.650 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:1 2024-04-06T09:31:59.650 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=cluster 2024-04-06T09:31:59.651 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:31:59.651 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:31:59.651 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=1 2024-04-06T09:31:59.651 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-06T09:31:59.651 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval cluster=cluster1 2024-04-06T09:31:59.651 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ cluster=cluster1 2024-04-06T09:31:59.651 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=1 2024-04-06T09:31:59.653 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=1 2024-04-06T09:31:59.653 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-04-06T09:31:59.653 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local local_cluster=cluster1:1 2024-04-06T09:31:59.653 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local cluster=cluster1 2024-04-06T09:31:59.653 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local instance 2024-04-06T09:31:59.653 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-06T09:31:59.653 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local val=cluster1:1 2024-04-06T09:31:59.653 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local cluster_var_name=local_cluster 2024-04-06T09:31:59.653 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local instance_var_name=instance 2024-04-06T09:31:59.653 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ cluster=cluster1 2024-04-06T09:31:59.653 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=1 2024-04-06T09:31:59.654 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ '[' 1 = cluster1:1 ']' 2024-04-06T09:31:59.654 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ eval local_cluster=cluster1 2024-04-06T09:31:59.654 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++++ local_cluster=cluster1 2024-04-06T09:31:59.654 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ eval instance=1 2024-04-06T09:31:59.654 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++++ instance=1 2024-04-06T09:31:59.654 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-06T09:31:59.672 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T09:31:59.672 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T09:31:59.672 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T09:31:59.672 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/test 2024-04-06T09:31:59.771 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-06T09:31:59.771 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:rbd mirror status : get status for rbd mirror 2024-04-06T09:31:59.771 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:admin_socket: invalid command 2024-04-06T09:31:59.773 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:31:59.773 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/test 2024-04-06T09:31:59.773 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1:2 2024-04-06T09:31:59.773 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ shift 2024-04-06T09:31:59.773 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:31:59.773 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:2 cluster instance 2024-04-06T09:31:59.773 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:2 2024-04-06T09:31:59.773 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=cluster 2024-04-06T09:31:59.774 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:31:59.774 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:31:59.774 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=2 2024-04-06T09:31:59.774 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 2 = cluster1:2 ']' 2024-04-06T09:31:59.774 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval cluster=cluster1 2024-04-06T09:31:59.774 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ cluster=cluster1 2024-04-06T09:31:59.774 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=2 2024-04-06T09:31:59.774 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=2 2024-04-06T09:31:59.775 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ daemon_asok_file cluster1:2 cluster1 2024-04-06T09:31:59.776 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local local_cluster=cluster1:2 2024-04-06T09:31:59.776 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local cluster=cluster1 2024-04-06T09:31:59.776 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local instance 2024-04-06T09:31:59.776 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2024-04-06T09:31:59.776 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local val=cluster1:2 2024-04-06T09:31:59.776 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local cluster_var_name=local_cluster 2024-04-06T09:31:59.776 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local instance_var_name=instance 2024-04-06T09:31:59.776 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ cluster=cluster1 2024-04-06T09:31:59.776 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=2 2024-04-06T09:31:59.776 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ '[' 2 = cluster1:2 ']' 2024-04-06T09:31:59.776 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ eval local_cluster=cluster1 2024-04-06T09:31:59.776 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++++ local_cluster=cluster1 2024-04-06T09:31:59.776 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ eval instance=2 2024-04-06T09:31:59.776 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++++ instance=2 2024-04-06T09:31:59.777 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-04-06T09:31:59.800 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-06T09:31:59.800 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-06T09:31:59.800 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-06T09:31:59.800 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/test 2024-04-06T09:31:59.896 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-06T09:31:59.896 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:rbd mirror status : get status for rbd mirror 2024-04-06T09:31:59.896 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:admin_socket: invalid command 2024-04-06T09:31:59.897 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:31:59.897 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/test 2024-04-06T09:31:59.898 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1:3 2024-04-06T09:31:59.898 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ shift 2024-04-06T09:31:59.898 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:31:59.898 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:3 cluster instance 2024-04-06T09:31:59.898 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:3 2024-04-06T09:31:59.898 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=cluster 2024-04-06T09:31:59.898 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:31:59.898 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:31:59.898 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=3 2024-04-06T09:31:59.898 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 3 = cluster1:3 ']' 2024-04-06T09:31:59.898 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval cluster=cluster1 2024-04-06T09:31:59.898 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ cluster=cluster1 2024-04-06T09:31:59.898 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=3 2024-04-06T09:31:59.899 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=3 2024-04-06T09:31:59.900 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ daemon_asok_file cluster1:3 cluster1 2024-04-06T09:31:59.900 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local local_cluster=cluster1:3 2024-04-06T09:31:59.900 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local cluster=cluster1 2024-04-06T09:31:59.900 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local instance 2024-04-06T09:31:59.900 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2024-04-06T09:31:59.900 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local val=cluster1:3 2024-04-06T09:31:59.900 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local cluster_var_name=local_cluster 2024-04-06T09:31:59.900 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local instance_var_name=instance 2024-04-06T09:31:59.900 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ cluster=cluster1 2024-04-06T09:31:59.900 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=3 2024-04-06T09:31:59.900 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ '[' 3 = cluster1:3 ']' 2024-04-06T09:31:59.901 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ eval local_cluster=cluster1 2024-04-06T09:31:59.901 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++++ local_cluster=cluster1 2024-04-06T09:31:59.901 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ eval instance=3 2024-04-06T09:31:59.901 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++++ instance=3 2024-04-06T09:31:59.901 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-04-06T09:31:59.923 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-06T09:31:59.923 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-06T09:31:59.924 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-06T09:31:59.924 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/test 2024-04-06T09:32:00.034 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-06T09:32:00.034 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:rbd mirror status : get status for rbd mirror 2024-04-06T09:32:00.034 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:admin_socket: invalid command 2024-04-06T09:32:00.035 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:32:00.036 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sleep 2 2024-04-06T09:32:02.037 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:32:02.038 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ seq 0 3 2024-04-06T09:32:02.041 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:32:02.042 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2024-04-06T09:32:02.042 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1:0 2024-04-06T09:32:02.042 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ shift 2024-04-06T09:32:02.042 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:32:02.042 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:32:02.042 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:32:02.042 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=cluster 2024-04-06T09:32:02.042 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:32:02.042 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:32:02.042 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:32:02.042 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:32:02.042 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval cluster=cluster1 2024-04-06T09:32:02.042 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ cluster=cluster1 2024-04-06T09:32:02.043 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:32:02.043 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:32:02.043 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:32:02.043 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local local_cluster=cluster1:0 2024-04-06T09:32:02.043 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local cluster=cluster1 2024-04-06T09:32:02.043 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local instance 2024-04-06T09:32:02.043 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:32:02.043 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local val=cluster1:0 2024-04-06T09:32:02.043 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local cluster_var_name=local_cluster 2024-04-06T09:32:02.043 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local instance_var_name=instance 2024-04-06T09:32:02.043 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ cluster=cluster1 2024-04-06T09:32:02.043 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:32:02.043 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ '[' 0 = cluster1:0 ']' 2024-04-06T09:32:02.043 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ eval local_cluster=cluster1 2024-04-06T09:32:02.044 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++++ local_cluster=cluster1 2024-04-06T09:32:02.044 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ eval instance=0 2024-04-06T09:32:02.044 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++++ instance=0 2024-04-06T09:32:02.044 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:32:02.068 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:32:02.068 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:32:02.068 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:32:02.068 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2024-04-06T09:32:02.185 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ return 0 2024-04-06T09:32:02.185 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ status_result=' "state": "Replaying"' 2024-04-06T09:32:02.186 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ echo ' "state": "Replaying"' 2024-04-06T09:32:02.186 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep -i Replaying 2024-04-06T09:32:02.188 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: "state": "Replaying" 2024-04-06T09:32:02.188 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ current_state=started 2024-04-06T09:32:02.189 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test started = started 2024-04-06T09:32:02.189 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:32:02.189 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ clean_snap_name= 2024-04-06T09:32:02.189 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 1 10 2024-04-06T09:32:02.189 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in `seq 1 10` 2024-04-06T09:32:02.189 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ stress_write_image cluster2 mirror test 2024-04-06T09:32:02.189 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-04-06T09:32:02.189 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:32:02.189 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:32:02.190 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-04-06T09:32:02.191 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local duration=12 2024-04-06T09:32:02.192 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set +e 2024-04-06T09:32:02.192 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ timeout 12s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror test --debug-rbd=20 --debug-journaler=20 2024-04-06T09:32:04.126 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:100 IOs, 70603776 bytes 2024-04-06T09:32:05.467 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:200 IOs, 105739264 bytes 2024-04-06T09:32:06.798 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:300 IOs, 164578304 bytes 2024-04-06T09:32:07.901 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:400 IOs, 206275584 bytes 2024-04-06T09:32:08.530 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:500 IOs, 220036096 bytes 2024-04-06T09:32:09.231 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:600 IOs, 246723584 bytes 2024-04-06T09:32:10.014 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:700 IOs, 256444416 bytes 2024-04-06T09:32:11.055 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:800 IOs, 294460416 bytes 2024-04-06T09:32:13.406 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:900 IOs, 381511680 bytes 2024-04-06T09:32:14.205 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ error_code=124 2024-04-06T09:32:14.205 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set -e 2024-04-06T09:32:14.205 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 124 -eq 124 ']' 2024-04-06T09:32:14.205 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:32:14.205 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+replaying 2024-04-06T09:32:14.205 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:32:14.205 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:32:14.206 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:32:14.206 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+replaying 2024-04-06T09:32:14.206 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local description_pattern= 2024-04-06T09:32:14.206 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local service_pattern= 2024-04-06T09:32:14.206 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:32:14.206 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 1 2024-04-06T09:32:15.206 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test_status_in_pool_dir cluster1 mirror test up+replaying '' '' 2024-04-06T09:32:15.210 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:32:15.210 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:32:15.210 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:32:15.210 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+replaying 2024-04-06T09:32:15.210 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local description_pattern= 2024-04-06T09:32:15.210 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local service_pattern= 2024-04-06T09:32:15.210 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local status 2024-04-06T09:32:15.211 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ CEPH_ARGS= 2024-04-06T09:32:15.211 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster1 mirror image status mirror/test 2024-04-06T09:32:15.288 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ status='test: 2024-04-06T09:32:15.289 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: global_id: 8ad5dedc-b3e0-425d-8993-b3e2231fc5ca 2024-04-06T09:32:15.289 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: state: up+replaying 2024-04-06T09:32:15.289 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":3,"entries_per_second":0.0,"non_primary_position":{},"primary_position":{"entry_tid":3,"object_number":3,"tag_tid":1}} 2024-04-06T09:32:15.289 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: service: mirror.0 on smithi192 2024-04-06T09:32:15.289 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: last_update: 2024-04-06 09:32:01 2024-04-06T09:32:15.289 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: peer_sites: 2024-04-06T09:32:15.289 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: name: cluster2 2024-04-06T09:32:15.289 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: state: up+stopped 2024-04-06T09:32:15.289 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: description: local image is primary 2024-04-06T09:32:15.289 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: last_update: 2024-04-06 09:32:13' 2024-04-06T09:32:15.289 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^ state: .*up+replaying' 2024-04-06T09:32:15.290 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: state: up+replaying 2024-04-06T09:32:15.290 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^ description: .*' 2024-04-06T09:32:15.291 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":3,"entries_per_second":0.0,"non_primary_position":{},"primary_position":{"entry_tid":3,"object_number":3,"tag_tid":1}} 2024-04-06T09:32:15.291 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n '' ']' 2024-04-06T09:32:15.291 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ echo up+replaying 2024-04-06T09:32:15.291 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^up+' 2024-04-06T09:32:15.292 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:up+replaying 2024-04-06T09:32:15.292 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep 'service: *mirror..* on ' 2024-04-06T09:32:15.293 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: service: mirror.0 on smithi192 2024-04-06T09:32:15.293 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local last_update 2024-04-06T09:32:15.293 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-04-06T09:32:15.295 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_update='2024-04-06 09:32:01' 2024-04-06T09:32:15.295 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+replaying '2024-04-06 09:32:01' 2024-04-06T09:32:15.295 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:32:15.295 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:32:15.296 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:32:15.296 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+replaying 2024-04-06T09:32:15.296 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'prev_last_update=2024-04-06 09:32:01' 2024-04-06T09:32:15.296 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local status 2024-04-06T09:32:15.296 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ CEPH_ARGS= 2024-04-06T09:32:15.296 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-04-06T09:32:15.374 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ status='WARNINGWARNINGOK14209mirror.1smithi19218.2.2-1059-g4d05f4cafalseWARNINGnot reporting status4216mirror.2smithi19218.2.2-1059-g4d05f4cafalseWARNINGnot reporting status4218mirror.3smithi19218.2.2-1059-g4d05f4cafalseWARNINGnot reporting status42584261mirror.0smithi19218.2.2-1059-g4d05f4catrueOKtest8ad5dedc-b3e0-425d-8993-b3e2231fc5caup+replayingreplaying, {"bytes_per_second":0.0,"entries_behind_primary":3,"entries_per_second":0.0,"non_primary_position":{},"primary_position":{"entry_tid":3,"object_number":3,"tag_tid":1}}42584261mirror.0smithi1922024-04-06 09:32:01cluster2c5e3bdc3-c286-4289-8fd0-6b8628b92376up+stoppedlocal image is primary2024-04-06 09:32:13' 2024-04-06T09:32:15.374 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local last_update state 2024-04-06T09:32:15.374 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2024-04-06T09:32:15.965 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_update='2024-04-06 09:32:01' 2024-04-06T09:32:15.966 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2024-04-06T09:32:15.970 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ state=up+replaying 2024-04-06T09:32:15.971 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ echo up+replaying 2024-04-06T09:32:15.971 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep up+replaying 2024-04-06T09:32:15.972 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:up+replaying 2024-04-06T09:32:15.974 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:32:15.974 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:32:15.974 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ snap_name=snap1 2024-04-06T09:32:15.974 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ create_snap cluster2 mirror test snap1 2024-04-06T09:32:15.974 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-04-06T09:32:15.974 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:32:15.974 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:32:15.974 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local snap_name=snap1 2024-04-06T09:32:15.974 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap1 --debug-rbd=20 --debug-journaler=20 2024-04-06T09:32:30.369 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.0 with skip (0.3154867933311356) > weight (0.25) 2024-04-06T09:32:30.369 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.1 with skip (0.5362379511377586) > weight (0.25) 2024-04-06T09:32:30.370 INFO:tasks.rbd_mirror_thrash:kill cluster2.client.mirror.2 2024-04-06T09:32:30.370 INFO:tasks.rbd_mirror.cluster2.client.mirror.2:Sent signal 15 2024-04-06T09:32:30.370 INFO:tasks.rbd_mirror_thrash:waiting for 9 secs before reviving daemons 2024-04-06T09:32:30.372 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi192.stderr:2024-04-06T09:32:30.369+0000 7f62adadf640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.2 (PID: 27487) UID: 1000 2024-04-06T09:32:39.371 INFO:tasks.rbd_mirror_thrash:waiting for cluster2.client.mirror.2 2024-04-06T09:32:39.371 INFO:teuthology.orchestra.run:waiting for 600 2024-04-06T09:32:39.371 INFO:tasks.rbd_mirror_thrash:reviving cluster2.client.mirror.2 2024-04-06T09:32:39.371 INFO:tasks.rbd_mirror.cluster2.client.mirror.2:Restarting daemon 2024-04-06T09:32:39.372 DEBUG:teuthology.orchestra.run.smithi192:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.2 2024-04-06T09:32:39.375 INFO:tasks.rbd_mirror.cluster2.client.mirror.2:Started 2024-04-06T09:32:39.375 INFO:tasks.rbd_mirror_thrash:waiting for 70 secs before thrashing 2024-04-06T09:32:46.517 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_image_replay_started cluster1 mirror test 2024-04-06T09:32:46.518 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:32:46.518 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:32:46.518 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:32:46.518 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_image_replay_state cluster1 mirror test started 2024-04-06T09:32:46.518 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:32:46.518 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:32:46.518 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:32:46.518 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state=started 2024-04-06T09:32:46.518 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local s 2024-04-06T09:32:46.518 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:32:46.518 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 1 2024-04-06T09:32:47.519 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test_image_replay_state cluster1 mirror test started 2024-04-06T09:32:47.519 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:32:47.519 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:32:47.519 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:32:47.520 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local test_state=started 2024-04-06T09:32:47.520 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local status_result 2024-04-06T09:32:47.520 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local current_state=stopped 2024-04-06T09:32:47.521 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2024-04-06T09:32:47.521 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -i state 2024-04-06T09:32:47.521 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_instance=cluster1 2024-04-06T09:32:47.521 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ shift 2024-04-06T09:32:47.521 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:32:47.521 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance=cluster1 2024-04-06T09:32:47.521 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local loop_instance 2024-04-06T09:32:47.521 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:32:47.521 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sleep 0 2024-04-06T09:32:47.522 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:32:47.523 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ seq 0 3 2024-04-06T09:32:47.525 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:32:47.525 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2024-04-06T09:32:47.525 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1:0 2024-04-06T09:32:47.525 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ shift 2024-04-06T09:32:47.526 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:32:47.526 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:32:47.526 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:32:47.526 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=cluster 2024-04-06T09:32:47.526 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:32:47.526 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:32:47.526 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:32:47.526 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:32:47.526 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval cluster=cluster1 2024-04-06T09:32:47.526 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ cluster=cluster1 2024-04-06T09:32:47.526 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:32:47.526 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:32:47.526 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:32:47.526 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local local_cluster=cluster1:0 2024-04-06T09:32:47.526 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local cluster=cluster1 2024-04-06T09:32:47.527 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local instance 2024-04-06T09:32:47.527 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:32:47.527 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local val=cluster1:0 2024-04-06T09:32:47.527 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local cluster_var_name=local_cluster 2024-04-06T09:32:47.527 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local instance_var_name=instance 2024-04-06T09:32:47.528 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ cluster=cluster1 2024-04-06T09:32:47.528 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:32:47.528 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ '[' 0 = cluster1:0 ']' 2024-04-06T09:32:47.528 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ eval local_cluster=cluster1 2024-04-06T09:32:47.528 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++++ local_cluster=cluster1 2024-04-06T09:32:47.528 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ eval instance=0 2024-04-06T09:32:47.528 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++++ instance=0 2024-04-06T09:32:47.528 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:32:47.548 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:32:47.548 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:32:47.548 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:32:47.548 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2024-04-06T09:32:47.670 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ return 0 2024-04-06T09:32:47.671 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ status_result=' "state": "Replaying"' 2024-04-06T09:32:47.671 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ echo ' "state": "Replaying"' 2024-04-06T09:32:47.671 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep -i Replaying 2024-04-06T09:32:47.673 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: "state": "Replaying" 2024-04-06T09:32:47.674 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ current_state=started 2024-04-06T09:32:47.674 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test started = started 2024-04-06T09:32:47.674 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:32:47.674 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror test 2024-04-06T09:32:47.674 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local local_cluster=cluster1 2024-04-06T09:32:47.674 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-04-06T09:32:47.674 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:32:47.674 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:32:47.674 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' journal = journal ']' 2024-04-06T09:32:47.675 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_journal_replay_complete cluster1 cluster2 mirror test 2024-04-06T09:32:47.675 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local local_cluster=cluster1 2024-04-06T09:32:47.675 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-04-06T09:32:47.675 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:32:47.675 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:32:47.675 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local s master_pos mirror_pos last_mirror_pos 2024-04-06T09:32:47.675 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local master_tag master_entry mirror_tag mirror_entry 2024-04-06T09:32:47.675 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:32:47.675 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:32:47.675 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:32:47.875 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:32:47.876 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:32:47.876 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:32:47.876 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:32:47.876 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:32:47.876 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:32:47.876 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:32:47.876 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:32:47.876 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:32:47.876 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:32:47.876 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:32:47.876 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:32:47.876 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:32:47.876 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:32:47.876 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:32:47.877 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:32:47.877 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:32:47.879 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:32:47.879 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:32:47.879 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:32:47.880 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:32:47.880 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:32:47.880 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:32:47.880 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:32:47.880 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:32:47.880 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:32:47.880 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:32:47.880 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:32:47.880 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:32:47.880 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:32:47.880 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:32:47.880 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:32:47.880 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:32:47.882 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:32:47.882 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:32:47.882 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:32:47.882 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:32:47.882 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:32:47.882 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:32:47.882 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:32:47.882 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:32:47.882 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:32:47.882 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:32:47.882 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:32:47.882 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:32:47.883 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:32:47.883 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:32:47.883 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:32:47.883 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:32:47.906 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:32:47.907 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:32:47.907 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:32:47.907 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:32:48.035 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:32:48.036 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:32:48.036 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:32:48.036 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:32:48.036 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:32:48.036 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:32:48.036 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:32:48.036 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:32:48.036 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:32:48.036 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:32:48.036 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:32:48.037 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:32:48.037 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:32:48.039 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:32:48.039 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:32:48.039 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:32:48.089 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 2 2024-04-06T09:32:48.089 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 6 2024-04-06T09:32:48.089 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:32:48.089 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=25, tag_tid=3, entry_tid=1], [object_number=24, tag_tid=3, entry_tid=0], [object_number=27, tag_tid=2, entry_tid=27223], [object_number=26, tag_tid=2, entry_tid=27222]]], state=connected] 2024-04-06T09:32:48.089 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=10, tag_tid=2, entry_tid=10606], [object_number=9, tag_tid=2, entry_tid=10605], [object_number=8, tag_tid=2, entry_tid=10604], [object_number=11, tag_tid=2, entry_tid=10603]]], state=connected] 2024-04-06T09:32:48.093 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:32:48.094 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=25, tag_tid=3, entry_tid=1' 2024-04-06T09:32:48.095 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:32:48.095 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:32:48.095 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:32:48.095 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:32:48.096 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:32:48.096 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:32:48.096 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:32:48.096 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:32:48.096 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:32:48.096 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:32:48.096 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:32:48.096 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:32:48.097 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:32:48.097 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:32:48.097 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:32:48.141 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 2 2024-04-06T09:32:48.142 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 6 2024-04-06T09:32:48.142 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:32:48.142 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=25, tag_tid=3, entry_tid=1], [object_number=24, tag_tid=3, entry_tid=0], [object_number=27, tag_tid=2, entry_tid=27223], [object_number=26, tag_tid=2, entry_tid=27222]]], state=connected] 2024-04-06T09:32:48.142 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=10, tag_tid=2, entry_tid=10606], [object_number=9, tag_tid=2, entry_tid=10605], [object_number=8, tag_tid=2, entry_tid=10604], [object_number=11, tag_tid=2, entry_tid=10603]]], state=connected] 2024-04-06T09:32:48.145 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:32:48.147 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=10, tag_tid=2, entry_tid=10606' 2024-04-06T09:32:48.147 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=25, tag_tid=3, entry_tid=1' -a 'object_number=25, tag_tid=3, entry_tid=1' = 'object_number=10, tag_tid=2, entry_tid=10606' 2024-04-06T09:32:48.147 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=10, tag_tid=2, entry_tid=10606' '!=' '' 2024-04-06T09:32:48.147 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:32:48.148 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=10, tag_tid=2, entry_tid=10606' = '' 2024-04-06T09:32:48.148 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=10, tag_tid=2, entry_tid=10606' 2024-04-06T09:32:48.148 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=25, tag_tid=3, entry_tid=1' 2024-04-06T09:32:48.148 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:32:48.148 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:32:48.148 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=3 2024-04-06T09:32:48.149 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=10, tag_tid=2, entry_tid=10606' 2024-04-06T09:32:48.149 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:32:48.149 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:32:48.150 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=2 2024-04-06T09:32:48.150 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=25, tag_tid=3, entry_tid=1' 2024-04-06T09:32:48.151 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:32:48.151 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:32:48.152 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:32:48.152 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=10, tag_tid=2, entry_tid=10606' 2024-04-06T09:32:48.152 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:32:48.152 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:32:48.154 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=10606 2024-04-06T09:32:48.154 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 3 = 2 -a 1 -le 10606 2024-04-06T09:32:48.154 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:32:48.154 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:32:48.154 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:32:48.355 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:32:48.355 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:32:48.356 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:32:48.356 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:32:48.356 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:32:48.356 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:32:48.356 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:32:48.356 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:32:48.356 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:32:48.356 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:32:48.356 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:32:48.356 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:32:48.356 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:32:48.357 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:32:48.357 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:32:48.357 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:32:48.357 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:32:48.360 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:32:48.360 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:32:48.360 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:32:48.360 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:32:48.360 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:32:48.360 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:32:48.360 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:32:48.360 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:32:48.360 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:32:48.360 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:32:48.360 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:32:48.360 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:32:48.361 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:32:48.361 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:32:48.361 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:32:48.361 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:32:48.361 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:32:48.361 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:32:48.361 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:32:48.361 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:32:48.361 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:32:48.361 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:32:48.361 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:32:48.361 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:32:48.361 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:32:48.361 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:32:48.361 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:32:48.362 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:32:48.362 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:32:48.362 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:32:48.362 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:32:48.362 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:32:48.384 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:32:48.385 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:32:48.385 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:32:48.385 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:32:48.522 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:32:48.523 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:32:48.523 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:32:48.523 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:32:48.523 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:32:48.524 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:32:48.524 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:32:48.524 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:32:48.524 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:32:48.524 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:32:48.524 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:32:48.525 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:32:48.525 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:32:48.526 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:32:48.526 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:32:48.526 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:32:48.580 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 2 2024-04-06T09:32:48.580 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 6 2024-04-06T09:32:48.580 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:32:48.580 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=25, tag_tid=3, entry_tid=1], [object_number=24, tag_tid=3, entry_tid=0], [object_number=27, tag_tid=2, entry_tid=27223], [object_number=26, tag_tid=2, entry_tid=27222]]], state=connected] 2024-04-06T09:32:48.580 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=8, tag_tid=2, entry_tid=10704], [object_number=11, tag_tid=2, entry_tid=10703], [object_number=10, tag_tid=2, entry_tid=10702], [object_number=9, tag_tid=2, entry_tid=10701]]], state=connected] 2024-04-06T09:32:48.584 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:32:48.585 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=25, tag_tid=3, entry_tid=1' 2024-04-06T09:32:48.586 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:32:48.586 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:32:48.587 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:32:48.587 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:32:48.587 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:32:48.587 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:32:48.587 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:32:48.587 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:32:48.587 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:32:48.587 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:32:48.587 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:32:48.587 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:32:48.588 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:32:48.588 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:32:48.588 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:32:48.634 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 2 2024-04-06T09:32:48.634 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 6 2024-04-06T09:32:48.634 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:32:48.634 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=25, tag_tid=3, entry_tid=1], [object_number=24, tag_tid=3, entry_tid=0], [object_number=27, tag_tid=2, entry_tid=27223], [object_number=26, tag_tid=2, entry_tid=27222]]], state=connected] 2024-04-06T09:32:48.634 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=8, tag_tid=2, entry_tid=10704], [object_number=11, tag_tid=2, entry_tid=10703], [object_number=10, tag_tid=2, entry_tid=10702], [object_number=9, tag_tid=2, entry_tid=10701]]], state=connected] 2024-04-06T09:32:48.637 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:32:48.639 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=8, tag_tid=2, entry_tid=10704' 2024-04-06T09:32:48.639 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=25, tag_tid=3, entry_tid=1' -a 'object_number=25, tag_tid=3, entry_tid=1' = 'object_number=8, tag_tid=2, entry_tid=10704' 2024-04-06T09:32:48.639 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=8, tag_tid=2, entry_tid=10704' '!=' 'object_number=10, tag_tid=2, entry_tid=10606' 2024-04-06T09:32:48.639 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:32:48.639 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=8, tag_tid=2, entry_tid=10704' = 'object_number=10, tag_tid=2, entry_tid=10606' 2024-04-06T09:32:48.640 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=8, tag_tid=2, entry_tid=10704' 2024-04-06T09:32:48.640 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=25, tag_tid=3, entry_tid=1' 2024-04-06T09:32:48.640 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:32:48.640 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:32:48.641 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=3 2024-04-06T09:32:48.641 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=8, tag_tid=2, entry_tid=10704' 2024-04-06T09:32:48.641 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:32:48.642 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:32:48.642 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=2 2024-04-06T09:32:48.643 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=25, tag_tid=3, entry_tid=1' 2024-04-06T09:32:48.643 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:32:48.644 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:32:48.644 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:32:48.645 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=8, tag_tid=2, entry_tid=10704' 2024-04-06T09:32:48.645 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:32:48.645 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:32:48.646 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=10704 2024-04-06T09:32:48.646 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 3 = 2 -a 1 -le 10704 2024-04-06T09:32:48.647 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:32:48.647 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:32:48.647 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:32:48.848 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:32:48.848 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:32:48.848 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:32:48.848 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:32:48.848 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:32:48.849 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:32:48.849 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:32:48.849 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:32:48.849 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:32:48.849 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:32:48.849 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:32:48.849 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:32:48.849 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:32:48.849 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:32:48.849 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:32:48.849 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:32:48.850 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:32:48.852 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:32:48.852 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:32:48.852 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:32:48.853 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:32:48.853 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:32:48.853 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:32:48.853 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:32:48.853 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:32:48.853 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:32:48.853 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:32:48.853 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:32:48.853 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:32:48.853 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:32:48.853 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:32:48.853 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:32:48.853 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:32:48.853 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:32:48.853 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:32:48.853 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:32:48.854 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:32:48.854 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:32:48.854 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:32:48.854 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:32:48.854 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:32:48.854 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:32:48.854 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:32:48.854 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:32:48.854 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:32:48.854 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:32:48.854 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:32:48.854 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:32:48.854 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:32:48.880 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:32:48.880 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:32:48.880 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:32:48.880 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:32:49.011 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:32:49.012 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:32:49.013 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:32:49.013 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:32:49.013 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:32:49.013 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:32:49.013 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:32:49.013 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:32:49.013 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:32:49.013 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:32:49.013 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:32:49.013 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:32:49.013 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:32:49.014 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:32:49.014 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:32:49.014 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:32:49.064 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 2 2024-04-06T09:32:49.064 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 6 2024-04-06T09:32:49.064 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:32:49.064 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=25, tag_tid=3, entry_tid=1], [object_number=24, tag_tid=3, entry_tid=0], [object_number=27, tag_tid=2, entry_tid=27223], [object_number=26, tag_tid=2, entry_tid=27222]]], state=connected] 2024-04-06T09:32:49.064 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=11, tag_tid=2, entry_tid=10799], [object_number=10, tag_tid=2, entry_tid=10798], [object_number=9, tag_tid=2, entry_tid=10797], [object_number=8, tag_tid=2, entry_tid=10796]]], state=connected] 2024-04-06T09:32:49.068 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:32:49.071 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=25, tag_tid=3, entry_tid=1' 2024-04-06T09:32:49.071 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:32:49.071 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:32:49.071 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:32:49.071 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:32:49.071 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:32:49.071 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:32:49.071 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:32:49.071 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:32:49.071 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:32:49.071 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:32:49.072 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:32:49.072 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:32:49.072 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:32:49.072 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:32:49.072 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:32:49.116 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 2 2024-04-06T09:32:49.116 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 6 2024-04-06T09:32:49.116 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:32:49.117 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=25, tag_tid=3, entry_tid=1], [object_number=24, tag_tid=3, entry_tid=0], [object_number=27, tag_tid=2, entry_tid=27223], [object_number=26, tag_tid=2, entry_tid=27222]]], state=connected] 2024-04-06T09:32:49.117 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=11, tag_tid=2, entry_tid=10799], [object_number=10, tag_tid=2, entry_tid=10798], [object_number=9, tag_tid=2, entry_tid=10797], [object_number=8, tag_tid=2, entry_tid=10796]]], state=connected] 2024-04-06T09:32:49.120 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:32:49.122 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=11, tag_tid=2, entry_tid=10799' 2024-04-06T09:32:49.122 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=25, tag_tid=3, entry_tid=1' -a 'object_number=25, tag_tid=3, entry_tid=1' = 'object_number=11, tag_tid=2, entry_tid=10799' 2024-04-06T09:32:49.123 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=11, tag_tid=2, entry_tid=10799' '!=' 'object_number=8, tag_tid=2, entry_tid=10704' 2024-04-06T09:32:49.123 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:32:49.123 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=11, tag_tid=2, entry_tid=10799' = 'object_number=8, tag_tid=2, entry_tid=10704' 2024-04-06T09:32:49.123 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=11, tag_tid=2, entry_tid=10799' 2024-04-06T09:32:49.123 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=25, tag_tid=3, entry_tid=1' 2024-04-06T09:32:49.123 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:32:49.123 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:32:49.124 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=3 2024-04-06T09:32:49.124 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=11, tag_tid=2, entry_tid=10799' 2024-04-06T09:32:49.124 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:32:49.124 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:32:49.125 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=2 2024-04-06T09:32:49.125 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=25, tag_tid=3, entry_tid=1' 2024-04-06T09:32:49.126 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:32:49.126 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:32:49.127 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:32:49.127 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=11, tag_tid=2, entry_tid=10799' 2024-04-06T09:32:49.127 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:32:49.127 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:32:49.128 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=10799 2024-04-06T09:32:49.129 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 3 = 2 -a 1 -le 10799 2024-04-06T09:32:49.129 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:32:49.129 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:32:49.129 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:32:49.330 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:32:49.331 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:32:49.331 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:32:49.331 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:32:49.331 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:32:49.331 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:32:49.331 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:32:49.331 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:32:49.331 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:32:49.331 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:32:49.331 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:32:49.331 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:32:49.331 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:32:49.331 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:32:49.331 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:32:49.332 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:32:49.332 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:32:49.335 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:32:49.335 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:32:49.335 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:32:49.335 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:32:49.335 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:32:49.336 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:32:49.336 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:32:49.336 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:32:49.336 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:32:49.336 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:32:49.336 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:32:49.336 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:32:49.336 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:32:49.336 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:32:49.336 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:32:49.336 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:32:49.336 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:32:49.336 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:32:49.336 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:32:49.336 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:32:49.336 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:32:49.337 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:32:49.337 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:32:49.337 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:32:49.337 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:32:49.337 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:32:49.337 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:32:49.337 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:32:49.337 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:32:49.337 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:32:49.337 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:32:49.337 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:32:49.361 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:32:49.361 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:32:49.361 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:32:49.361 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:32:49.509 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:32:49.510 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:32:49.510 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:32:49.510 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:32:49.510 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:32:49.510 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:32:49.510 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:32:49.510 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:32:49.510 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:32:49.510 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:32:49.511 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:32:49.511 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:32:49.511 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:32:49.513 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:32:49.513 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:32:49.513 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:32:49.561 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 2 2024-04-06T09:32:49.561 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 6 2024-04-06T09:32:49.561 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:32:49.561 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=25, tag_tid=3, entry_tid=1], [object_number=24, tag_tid=3, entry_tid=0], [object_number=27, tag_tid=2, entry_tid=27223], [object_number=26, tag_tid=2, entry_tid=27222]]], state=connected] 2024-04-06T09:32:49.561 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=10, tag_tid=2, entry_tid=10890], [object_number=9, tag_tid=2, entry_tid=10889], [object_number=8, tag_tid=2, entry_tid=10888], [object_number=11, tag_tid=2, entry_tid=10887]]], state=connected] 2024-04-06T09:32:49.603 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:32:49.603 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=25, tag_tid=3, entry_tid=1' 2024-04-06T09:32:49.603 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:32:49.603 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:32:49.603 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:32:49.603 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:32:49.603 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:32:49.603 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:32:49.603 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:32:49.603 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:32:49.603 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:32:49.603 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:32:49.603 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:32:49.603 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:32:49.604 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:32:49.604 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:32:49.604 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:32:49.650 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 2 2024-04-06T09:32:49.650 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 6 2024-04-06T09:32:49.650 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:32:49.650 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=25, tag_tid=3, entry_tid=1], [object_number=24, tag_tid=3, entry_tid=0], [object_number=27, tag_tid=2, entry_tid=27223], [object_number=26, tag_tid=2, entry_tid=27222]]], state=connected] 2024-04-06T09:32:49.650 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=10, tag_tid=2, entry_tid=10890], [object_number=9, tag_tid=2, entry_tid=10889], [object_number=8, tag_tid=2, entry_tid=10888], [object_number=11, tag_tid=2, entry_tid=10887]]], state=connected] 2024-04-06T09:32:49.655 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:32:49.657 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=10, tag_tid=2, entry_tid=10890' 2024-04-06T09:32:49.658 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=25, tag_tid=3, entry_tid=1' -a 'object_number=25, tag_tid=3, entry_tid=1' = 'object_number=10, tag_tid=2, entry_tid=10890' 2024-04-06T09:32:49.658 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=10, tag_tid=2, entry_tid=10890' '!=' 'object_number=11, tag_tid=2, entry_tid=10799' 2024-04-06T09:32:49.658 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:32:49.658 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=10, tag_tid=2, entry_tid=10890' = 'object_number=11, tag_tid=2, entry_tid=10799' 2024-04-06T09:32:49.658 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=10, tag_tid=2, entry_tid=10890' 2024-04-06T09:32:49.658 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=25, tag_tid=3, entry_tid=1' 2024-04-06T09:32:49.658 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:32:49.658 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:32:49.659 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=3 2024-04-06T09:32:49.659 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=10, tag_tid=2, entry_tid=10890' 2024-04-06T09:32:49.659 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:32:49.660 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:32:49.661 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=2 2024-04-06T09:32:49.661 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=25, tag_tid=3, entry_tid=1' 2024-04-06T09:32:49.662 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:32:49.662 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:32:49.662 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:32:49.663 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=10, tag_tid=2, entry_tid=10890' 2024-04-06T09:32:49.663 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:32:49.663 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:32:49.664 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=10890 2024-04-06T09:32:49.664 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 3 = 2 -a 1 -le 10890 2024-04-06T09:32:49.664 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:32:49.664 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:32:49.664 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:32:49.866 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:32:49.866 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:32:49.866 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:32:49.866 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:32:49.866 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:32:49.866 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:32:49.866 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:32:49.867 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:32:49.867 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:32:49.867 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:32:49.867 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:32:49.867 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:32:49.867 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:32:49.867 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:32:49.867 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:32:49.867 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:32:49.868 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:32:49.870 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:32:49.870 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:32:49.870 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:32:49.870 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:32:49.870 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:32:49.871 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:32:49.871 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:32:49.871 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:32:49.871 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:32:49.871 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:32:49.871 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:32:49.871 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:32:49.871 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:32:49.871 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:32:49.871 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:32:49.871 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:32:49.872 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:32:49.873 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:32:49.873 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:32:49.873 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:32:49.873 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:32:49.873 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:32:49.873 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:32:49.873 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:32:49.873 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:32:49.873 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:32:49.873 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:32:49.873 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:32:49.873 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:32:49.873 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:32:49.873 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:32:49.874 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:32:49.897 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:32:49.897 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:32:49.897 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:32:49.898 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:32:50.032 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:32:50.033 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:32:50.033 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:32:50.033 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:32:50.033 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:32:50.033 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:32:50.033 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:32:50.033 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:32:50.033 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:32:50.034 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:32:50.034 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:32:50.034 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:32:50.034 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:32:50.036 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:32:50.036 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:32:50.037 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:32:50.082 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 2 2024-04-06T09:32:50.082 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 6 2024-04-06T09:32:50.082 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:32:50.082 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=25, tag_tid=3, entry_tid=1], [object_number=24, tag_tid=3, entry_tid=0], [object_number=27, tag_tid=2, entry_tid=27223], [object_number=26, tag_tid=2, entry_tid=27222]]], state=connected] 2024-04-06T09:32:50.082 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=9, tag_tid=2, entry_tid=10993], [object_number=8, tag_tid=2, entry_tid=10992], [object_number=11, tag_tid=2, entry_tid=10991], [object_number=10, tag_tid=2, entry_tid=10990]]], state=connected] 2024-04-06T09:32:50.114 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:32:50.115 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=25, tag_tid=3, entry_tid=1' 2024-04-06T09:32:50.117 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:32:50.117 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:32:50.117 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:32:50.117 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:32:50.117 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:32:50.117 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:32:50.117 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:32:50.117 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:32:50.117 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:32:50.117 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:32:50.117 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:32:50.117 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:32:50.118 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:32:50.118 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:32:50.118 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:32:50.164 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 2 2024-04-06T09:32:50.164 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 6 2024-04-06T09:32:50.164 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:32:50.164 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=25, tag_tid=3, entry_tid=1], [object_number=24, tag_tid=3, entry_tid=0], [object_number=27, tag_tid=2, entry_tid=27223], [object_number=26, tag_tid=2, entry_tid=27222]]], state=connected] 2024-04-06T09:32:50.165 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=9, tag_tid=2, entry_tid=10993], [object_number=8, tag_tid=2, entry_tid=10992], [object_number=11, tag_tid=2, entry_tid=10991], [object_number=10, tag_tid=2, entry_tid=10990]]], state=connected] 2024-04-06T09:32:50.168 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:32:50.170 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=9, tag_tid=2, entry_tid=10993' 2024-04-06T09:32:50.170 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=25, tag_tid=3, entry_tid=1' -a 'object_number=25, tag_tid=3, entry_tid=1' = 'object_number=9, tag_tid=2, entry_tid=10993' 2024-04-06T09:32:50.170 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=9, tag_tid=2, entry_tid=10993' '!=' 'object_number=10, tag_tid=2, entry_tid=10890' 2024-04-06T09:32:50.170 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:32:50.170 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=9, tag_tid=2, entry_tid=10993' = 'object_number=10, tag_tid=2, entry_tid=10890' 2024-04-06T09:32:50.170 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=9, tag_tid=2, entry_tid=10993' 2024-04-06T09:32:50.171 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=25, tag_tid=3, entry_tid=1' 2024-04-06T09:32:50.171 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:32:50.171 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:32:50.171 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=3 2024-04-06T09:32:50.172 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=9, tag_tid=2, entry_tid=10993' 2024-04-06T09:32:50.172 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:32:50.172 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:32:50.173 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=2 2024-04-06T09:32:50.173 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=25, tag_tid=3, entry_tid=1' 2024-04-06T09:32:50.174 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:32:50.174 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:32:50.174 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:32:50.175 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=9, tag_tid=2, entry_tid=10993' 2024-04-06T09:32:50.175 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:32:50.175 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:32:50.176 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=10993 2024-04-06T09:32:50.176 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 3 = 2 -a 1 -le 10993 2024-04-06T09:32:50.177 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:32:50.177 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:32:50.177 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:32:50.378 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:32:50.379 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:32:50.379 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:32:50.379 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:32:50.379 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:32:50.379 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:32:50.379 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:32:50.379 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:32:50.379 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:32:50.379 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:32:50.379 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:32:50.380 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:32:50.380 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:32:50.380 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:32:50.380 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:32:50.380 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:32:50.380 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:32:50.383 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:32:50.383 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:32:50.383 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:32:50.383 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:32:50.383 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:32:50.383 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:32:50.384 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:32:50.384 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:32:50.384 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:32:50.384 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:32:50.384 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:32:50.385 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:32:50.385 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:32:50.385 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:32:50.385 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:32:50.385 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:32:50.386 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:32:50.386 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:32:50.387 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:32:50.387 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:32:50.387 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:32:50.387 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:32:50.387 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:32:50.387 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:32:50.387 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:32:50.387 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:32:50.387 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:32:50.388 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:32:50.388 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:32:50.388 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:32:50.388 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:32:50.388 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:32:50.419 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:32:50.420 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:32:50.420 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:32:50.420 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:32:50.548 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:32:50.548 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:32:50.548 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:32:50.548 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:32:50.549 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:32:50.549 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:32:50.549 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:32:50.549 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:32:50.549 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:32:50.549 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:32:50.550 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:32:50.550 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:32:50.550 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:32:50.552 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:32:50.553 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:32:50.553 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:32:50.602 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 2 2024-04-06T09:32:50.603 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 6 2024-04-06T09:32:50.603 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:32:50.603 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=25, tag_tid=3, entry_tid=1], [object_number=24, tag_tid=3, entry_tid=0], [object_number=27, tag_tid=2, entry_tid=27223], [object_number=26, tag_tid=2, entry_tid=27222]]], state=connected] 2024-04-06T09:32:50.603 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=11, tag_tid=2, entry_tid=11099], [object_number=10, tag_tid=2, entry_tid=11098], [object_number=9, tag_tid=2, entry_tid=11097], [object_number=8, tag_tid=2, entry_tid=11096]]], state=connected] 2024-04-06T09:32:50.606 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:32:50.607 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=25, tag_tid=3, entry_tid=1' 2024-04-06T09:32:50.608 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:32:50.609 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:32:50.609 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:32:50.609 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:32:50.609 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:32:50.609 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:32:50.609 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:32:50.609 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:32:50.609 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:32:50.609 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:32:50.609 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:32:50.609 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:32:50.610 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:32:50.610 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:32:50.610 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:32:50.655 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 2 2024-04-06T09:32:50.655 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 6 2024-04-06T09:32:50.655 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:32:50.655 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=25, tag_tid=3, entry_tid=1], [object_number=24, tag_tid=3, entry_tid=0], [object_number=27, tag_tid=2, entry_tid=27223], [object_number=26, tag_tid=2, entry_tid=27222]]], state=connected] 2024-04-06T09:32:50.655 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=11, tag_tid=2, entry_tid=11099], [object_number=10, tag_tid=2, entry_tid=11098], [object_number=9, tag_tid=2, entry_tid=11097], [object_number=8, tag_tid=2, entry_tid=11096]]], state=connected] 2024-04-06T09:32:50.659 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:32:50.661 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=11, tag_tid=2, entry_tid=11099' 2024-04-06T09:32:50.661 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=25, tag_tid=3, entry_tid=1' -a 'object_number=25, tag_tid=3, entry_tid=1' = 'object_number=11, tag_tid=2, entry_tid=11099' 2024-04-06T09:32:50.661 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=11, tag_tid=2, entry_tid=11099' '!=' 'object_number=9, tag_tid=2, entry_tid=10993' 2024-04-06T09:32:50.661 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:32:50.661 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=11, tag_tid=2, entry_tid=11099' = 'object_number=9, tag_tid=2, entry_tid=10993' 2024-04-06T09:32:50.661 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=11, tag_tid=2, entry_tid=11099' 2024-04-06T09:32:50.662 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=25, tag_tid=3, entry_tid=1' 2024-04-06T09:32:50.662 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:32:50.662 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:32:50.662 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=3 2024-04-06T09:32:50.663 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=11, tag_tid=2, entry_tid=11099' 2024-04-06T09:32:50.663 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:32:50.663 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:32:50.663 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=2 2024-04-06T09:32:50.664 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=25, tag_tid=3, entry_tid=1' 2024-04-06T09:32:50.664 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:32:50.664 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:32:50.665 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:32:50.666 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=11, tag_tid=2, entry_tid=11099' 2024-04-06T09:32:50.666 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:32:50.666 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:32:50.667 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=11099 2024-04-06T09:32:50.667 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 3 = 2 -a 1 -le 11099 2024-04-06T09:32:50.667 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:32:50.667 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:32:50.667 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:32:50.869 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:32:50.869 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:32:50.869 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:32:50.869 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:32:50.869 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:32:50.869 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:32:50.869 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:32:50.869 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:32:50.869 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:32:50.869 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:32:50.869 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:32:50.870 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:32:50.870 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:32:50.870 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:32:50.870 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:32:50.870 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:32:50.871 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:32:50.872 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:32:50.872 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:32:50.872 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:32:50.872 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:32:50.872 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:32:50.872 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:32:50.872 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:32:50.873 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:32:50.873 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:32:50.873 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:32:50.873 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:32:50.873 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:32:50.873 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:32:50.873 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:32:50.873 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:32:50.873 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:32:50.874 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:32:50.874 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:32:50.874 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:32:50.874 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:32:50.874 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:32:50.874 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:32:50.875 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:32:50.875 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:32:50.875 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:32:50.875 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:32:50.875 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:32:50.875 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:32:50.875 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:32:50.875 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:32:50.875 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:32:50.875 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:32:50.899 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:32:50.899 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:32:50.899 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:32:50.899 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:32:51.027 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:32:51.028 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:32:51.028 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:32:51.029 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:32:51.029 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:32:51.029 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:32:51.029 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:32:51.029 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:32:51.029 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:32:51.029 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:32:51.029 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:32:51.030 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:32:51.030 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:32:51.030 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:32:51.031 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:32:51.034 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:32:51.074 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 2 2024-04-06T09:32:51.074 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 6 2024-04-06T09:32:51.074 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:32:51.075 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=25, tag_tid=3, entry_tid=1], [object_number=24, tag_tid=3, entry_tid=0], [object_number=27, tag_tid=2, entry_tid=27223], [object_number=26, tag_tid=2, entry_tid=27222]]], state=connected] 2024-04-06T09:32:51.075 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=9, tag_tid=2, entry_tid=11189], [object_number=8, tag_tid=2, entry_tid=11188], [object_number=11, tag_tid=2, entry_tid=11187], [object_number=10, tag_tid=2, entry_tid=11186]]], state=connected] 2024-04-06T09:32:51.078 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:32:51.080 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=25, tag_tid=3, entry_tid=1' 2024-04-06T09:32:51.080 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:32:51.080 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:32:51.080 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:32:51.080 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:32:51.081 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:32:51.081 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:32:51.081 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:32:51.081 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:32:51.081 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:32:51.081 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:32:51.081 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:32:51.081 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:32:51.081 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:32:51.082 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:32:51.082 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:32:51.125 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 2 2024-04-06T09:32:51.126 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 6 2024-04-06T09:32:51.126 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:32:51.126 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=25, tag_tid=3, entry_tid=1], [object_number=24, tag_tid=3, entry_tid=0], [object_number=27, tag_tid=2, entry_tid=27223], [object_number=26, tag_tid=2, entry_tid=27222]]], state=connected] 2024-04-06T09:32:51.126 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=9, tag_tid=2, entry_tid=11189], [object_number=8, tag_tid=2, entry_tid=11188], [object_number=11, tag_tid=2, entry_tid=11187], [object_number=10, tag_tid=2, entry_tid=11186]]], state=connected] 2024-04-06T09:32:51.129 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:32:51.131 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=9, tag_tid=2, entry_tid=11189' 2024-04-06T09:32:51.131 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=25, tag_tid=3, entry_tid=1' -a 'object_number=25, tag_tid=3, entry_tid=1' = 'object_number=9, tag_tid=2, entry_tid=11189' 2024-04-06T09:32:51.131 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=9, tag_tid=2, entry_tid=11189' '!=' 'object_number=11, tag_tid=2, entry_tid=11099' 2024-04-06T09:32:51.131 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:32:51.131 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=9, tag_tid=2, entry_tid=11189' = 'object_number=11, tag_tid=2, entry_tid=11099' 2024-04-06T09:32:51.131 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=9, tag_tid=2, entry_tid=11189' 2024-04-06T09:32:51.132 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=25, tag_tid=3, entry_tid=1' 2024-04-06T09:32:51.132 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:32:51.132 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:32:51.132 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=3 2024-04-06T09:32:51.133 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=9, tag_tid=2, entry_tid=11189' 2024-04-06T09:32:51.133 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:32:51.133 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:32:51.134 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=2 2024-04-06T09:32:51.135 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=25, tag_tid=3, entry_tid=1' 2024-04-06T09:32:51.135 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:32:51.135 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:32:51.135 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:32:51.136 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=9, tag_tid=2, entry_tid=11189' 2024-04-06T09:32:51.136 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:32:51.136 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:32:51.137 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=11189 2024-04-06T09:32:51.137 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 3 = 2 -a 1 -le 11189 2024-04-06T09:32:51.137 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:32:51.138 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:32:51.138 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:32:51.339 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:32:51.340 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:32:51.340 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:32:51.340 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:32:51.340 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:32:51.340 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:32:51.340 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:32:51.340 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:32:51.340 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:32:51.340 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:32:51.340 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:32:51.340 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:32:51.341 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:32:51.341 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:32:51.341 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:32:51.344 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:32:51.344 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:32:51.344 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:32:51.344 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:32:51.344 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:32:51.345 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:32:51.345 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:32:51.345 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:32:51.345 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:32:51.345 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:32:51.345 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:32:51.345 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:32:51.345 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:32:51.345 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:32:51.345 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:32:51.345 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:32:51.345 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:32:51.345 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:32:51.345 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:32:51.346 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:32:51.346 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:32:51.346 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:32:51.346 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:32:51.346 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:32:51.346 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:32:51.346 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:32:51.346 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:32:51.346 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:32:51.346 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:32:51.346 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:32:51.346 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:32:51.346 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:32:51.347 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:32:51.347 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:32:51.369 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:32:51.369 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:32:51.369 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:32:51.369 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:32:51.524 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:32:51.525 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:32:51.525 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:32:51.525 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:32:51.525 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:32:51.525 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:32:51.525 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:32:51.526 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:32:51.526 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:32:51.526 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:32:51.526 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:32:51.526 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:32:51.526 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:32:51.527 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:32:51.527 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:32:51.527 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:32:51.579 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 2 2024-04-06T09:32:51.579 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 6 2024-04-06T09:32:51.579 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:32:51.579 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=25, tag_tid=3, entry_tid=1], [object_number=24, tag_tid=3, entry_tid=0], [object_number=27, tag_tid=2, entry_tid=27223], [object_number=26, tag_tid=2, entry_tid=27222]]], state=connected] 2024-04-06T09:32:51.579 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=9, tag_tid=2, entry_tid=11297], [object_number=8, tag_tid=2, entry_tid=11296], [object_number=11, tag_tid=2, entry_tid=11295], [object_number=10, tag_tid=2, entry_tid=11294]]], state=connected] 2024-04-06T09:32:51.582 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:32:51.584 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=25, tag_tid=3, entry_tid=1' 2024-04-06T09:32:51.585 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:32:51.585 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:32:51.585 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:32:51.585 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:32:51.585 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:32:51.585 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:32:51.585 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:32:51.585 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:32:51.585 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:32:51.586 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:32:51.586 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:32:51.586 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:32:51.586 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:32:51.586 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:32:51.586 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:32:51.631 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 2 2024-04-06T09:32:51.631 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 6 2024-04-06T09:32:51.632 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:32:51.632 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=25, tag_tid=3, entry_tid=1], [object_number=24, tag_tid=3, entry_tid=0], [object_number=27, tag_tid=2, entry_tid=27223], [object_number=26, tag_tid=2, entry_tid=27222]]], state=connected] 2024-04-06T09:32:51.632 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=9, tag_tid=2, entry_tid=11297], [object_number=8, tag_tid=2, entry_tid=11296], [object_number=11, tag_tid=2, entry_tid=11295], [object_number=10, tag_tid=2, entry_tid=11294]]], state=connected] 2024-04-06T09:32:51.635 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:32:51.637 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=9, tag_tid=2, entry_tid=11297' 2024-04-06T09:32:51.637 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=25, tag_tid=3, entry_tid=1' -a 'object_number=25, tag_tid=3, entry_tid=1' = 'object_number=9, tag_tid=2, entry_tid=11297' 2024-04-06T09:32:51.637 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=9, tag_tid=2, entry_tid=11297' '!=' 'object_number=9, tag_tid=2, entry_tid=11189' 2024-04-06T09:32:51.637 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:32:51.637 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=9, tag_tid=2, entry_tid=11297' = 'object_number=9, tag_tid=2, entry_tid=11189' 2024-04-06T09:32:51.637 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=9, tag_tid=2, entry_tid=11297' 2024-04-06T09:32:51.637 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=25, tag_tid=3, entry_tid=1' 2024-04-06T09:32:51.637 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:32:51.637 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:32:51.638 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=3 2024-04-06T09:32:51.638 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=9, tag_tid=2, entry_tid=11297' 2024-04-06T09:32:51.639 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:32:51.639 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:32:51.640 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=2 2024-04-06T09:32:51.640 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=25, tag_tid=3, entry_tid=1' 2024-04-06T09:32:51.640 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:32:51.640 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:32:51.641 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:32:51.642 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=9, tag_tid=2, entry_tid=11297' 2024-04-06T09:32:51.642 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:32:51.642 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:32:51.643 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=11297 2024-04-06T09:32:51.643 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 3 = 2 -a 1 -le 11297 2024-04-06T09:32:51.643 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:32:51.643 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:32:51.643 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:32:51.845 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:32:51.845 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:32:51.845 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:32:51.845 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:32:51.845 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:32:51.845 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:32:51.845 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:32:51.846 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:32:51.846 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:32:51.846 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:32:51.846 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:32:51.846 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:32:51.846 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:32:51.846 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:32:51.846 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:32:51.846 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:32:51.847 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:32:51.850 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:32:51.850 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:32:51.850 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:32:51.850 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:32:51.850 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:32:51.850 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:32:51.850 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:32:51.850 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:32:51.850 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:32:51.850 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:32:51.850 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:32:51.851 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:32:51.851 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:32:51.851 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:32:51.851 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:32:51.851 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:32:51.852 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:32:51.852 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:32:51.852 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:32:51.852 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:32:51.852 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:32:51.852 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:32:51.852 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:32:51.852 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:32:51.852 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:32:51.852 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:32:51.852 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:32:51.852 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:32:51.852 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:32:51.852 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:32:51.852 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:32:51.852 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:32:51.875 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:32:51.876 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:32:51.876 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:32:51.876 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:32:52.008 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:32:52.009 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:32:52.009 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:32:52.009 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:32:52.009 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:32:52.009 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:32:52.009 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:32:52.009 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:32:52.010 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:32:52.010 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:32:52.010 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:32:52.010 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:32:52.010 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:32:52.012 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:32:52.012 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:32:52.012 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:32:52.057 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 2 2024-04-06T09:32:52.057 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 6 2024-04-06T09:32:52.057 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:32:52.058 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=25, tag_tid=3, entry_tid=1], [object_number=24, tag_tid=3, entry_tid=0], [object_number=27, tag_tid=2, entry_tid=27223], [object_number=26, tag_tid=2, entry_tid=27222]]], state=connected] 2024-04-06T09:32:52.058 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=8, tag_tid=2, entry_tid=11408], [object_number=11, tag_tid=2, entry_tid=11407], [object_number=10, tag_tid=2, entry_tid=11406], [object_number=9, tag_tid=2, entry_tid=11405]]], state=connected] 2024-04-06T09:32:52.061 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:32:52.062 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=25, tag_tid=3, entry_tid=1' 2024-04-06T09:32:52.063 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:32:52.063 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:32:52.063 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:32:52.063 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:32:52.064 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:32:52.064 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:32:52.064 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:32:52.064 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:32:52.064 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:32:52.064 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:32:52.064 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:32:52.064 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:32:52.065 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:32:52.065 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:32:52.065 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:32:52.109 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 2 2024-04-06T09:32:52.109 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 6 2024-04-06T09:32:52.109 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:32:52.109 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=25, tag_tid=3, entry_tid=1], [object_number=24, tag_tid=3, entry_tid=0], [object_number=27, tag_tid=2, entry_tid=27223], [object_number=26, tag_tid=2, entry_tid=27222]]], state=connected] 2024-04-06T09:32:52.109 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=8, tag_tid=2, entry_tid=11408], [object_number=11, tag_tid=2, entry_tid=11407], [object_number=10, tag_tid=2, entry_tid=11406], [object_number=9, tag_tid=2, entry_tid=11405]]], state=connected] 2024-04-06T09:32:52.113 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:32:52.115 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=8, tag_tid=2, entry_tid=11408' 2024-04-06T09:32:52.115 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=25, tag_tid=3, entry_tid=1' -a 'object_number=25, tag_tid=3, entry_tid=1' = 'object_number=8, tag_tid=2, entry_tid=11408' 2024-04-06T09:32:52.115 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=8, tag_tid=2, entry_tid=11408' '!=' 'object_number=9, tag_tid=2, entry_tid=11297' 2024-04-06T09:32:52.115 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:32:52.115 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=8, tag_tid=2, entry_tid=11408' = 'object_number=9, tag_tid=2, entry_tid=11297' 2024-04-06T09:32:52.115 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=8, tag_tid=2, entry_tid=11408' 2024-04-06T09:32:52.115 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=25, tag_tid=3, entry_tid=1' 2024-04-06T09:32:52.115 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:32:52.115 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:32:52.116 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=3 2024-04-06T09:32:52.117 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=8, tag_tid=2, entry_tid=11408' 2024-04-06T09:32:52.117 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:32:52.117 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:32:52.118 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=2 2024-04-06T09:32:52.118 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=25, tag_tid=3, entry_tid=1' 2024-04-06T09:32:52.118 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:32:52.118 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:32:52.119 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:32:52.120 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=8, tag_tid=2, entry_tid=11408' 2024-04-06T09:32:52.120 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:32:52.120 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:32:52.121 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=11408 2024-04-06T09:32:52.121 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 3 = 2 -a 1 -le 11408 2024-04-06T09:32:52.121 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:32:52.121 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:32:52.121 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:32:52.323 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:32:52.324 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:32:52.324 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:32:52.324 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:32:52.324 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:32:52.324 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:32:52.324 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:32:52.324 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:32:52.324 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:32:52.324 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:32:52.324 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:32:52.324 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:32:52.324 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:32:52.324 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:32:52.324 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:32:52.325 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:32:52.325 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:32:52.326 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:32:52.326 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:32:52.326 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:32:52.326 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:32:52.326 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:32:52.326 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:32:52.327 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:32:52.327 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:32:52.327 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:32:52.327 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:32:52.327 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:32:52.327 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:32:52.327 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:32:52.327 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:32:52.327 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:32:52.327 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:32:52.328 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:32:52.328 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:32:52.329 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:32:52.329 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:32:52.329 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:32:52.329 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:32:52.329 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:32:52.329 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:32:52.329 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:32:52.329 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:32:52.329 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:32:52.329 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:32:52.329 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:32:52.329 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:32:52.329 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:32:52.329 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:32:52.353 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:32:52.354 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:32:52.354 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:32:52.354 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:32:52.492 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:32:52.492 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:32:52.493 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:32:52.493 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:32:52.493 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:32:52.493 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:32:52.493 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:32:52.493 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:32:52.493 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:32:52.493 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:32:52.494 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:32:52.494 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:32:52.494 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:32:52.496 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:32:52.496 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:32:52.496 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:32:52.546 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 2 2024-04-06T09:32:52.547 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 6 2024-04-06T09:32:52.547 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:32:52.547 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=25, tag_tid=3, entry_tid=1], [object_number=24, tag_tid=3, entry_tid=0], [object_number=27, tag_tid=2, entry_tid=27223], [object_number=26, tag_tid=2, entry_tid=27222]]], state=connected] 2024-04-06T09:32:52.547 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=11, tag_tid=2, entry_tid=11515], [object_number=10, tag_tid=2, entry_tid=11514], [object_number=9, tag_tid=2, entry_tid=11513], [object_number=8, tag_tid=2, entry_tid=11512]]], state=connected] 2024-04-06T09:32:52.550 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:32:52.552 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=25, tag_tid=3, entry_tid=1' 2024-04-06T09:32:52.553 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:32:52.553 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:32:52.553 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:32:52.553 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:32:52.553 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:32:52.553 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:32:52.553 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:32:52.553 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:32:52.553 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:32:52.553 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:32:52.554 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:32:52.554 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:32:52.554 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:32:52.554 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:32:52.555 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:32:52.598 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 2 2024-04-06T09:32:52.599 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 6 2024-04-06T09:32:52.599 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:32:52.599 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=25, tag_tid=3, entry_tid=1], [object_number=24, tag_tid=3, entry_tid=0], [object_number=27, tag_tid=2, entry_tid=27223], [object_number=26, tag_tid=2, entry_tid=27222]]], state=connected] 2024-04-06T09:32:52.599 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=11, tag_tid=2, entry_tid=11515], [object_number=10, tag_tid=2, entry_tid=11514], [object_number=9, tag_tid=2, entry_tid=11513], [object_number=8, tag_tid=2, entry_tid=11512]]], state=connected] 2024-04-06T09:32:52.602 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:32:52.604 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=11, tag_tid=2, entry_tid=11515' 2024-04-06T09:32:52.604 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=25, tag_tid=3, entry_tid=1' -a 'object_number=25, tag_tid=3, entry_tid=1' = 'object_number=11, tag_tid=2, entry_tid=11515' 2024-04-06T09:32:52.605 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=11, tag_tid=2, entry_tid=11515' '!=' 'object_number=8, tag_tid=2, entry_tid=11408' 2024-04-06T09:32:52.605 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:32:52.605 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=11, tag_tid=2, entry_tid=11515' = 'object_number=8, tag_tid=2, entry_tid=11408' 2024-04-06T09:32:52.605 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=11, tag_tid=2, entry_tid=11515' 2024-04-06T09:32:52.605 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=25, tag_tid=3, entry_tid=1' 2024-04-06T09:32:52.605 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:32:52.605 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:32:52.605 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=3 2024-04-06T09:32:52.606 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=11, tag_tid=2, entry_tid=11515' 2024-04-06T09:32:52.606 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:32:52.606 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:32:52.608 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=2 2024-04-06T09:32:52.608 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=25, tag_tid=3, entry_tid=1' 2024-04-06T09:32:52.608 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:32:52.608 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:32:52.610 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:32:52.610 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=11, tag_tid=2, entry_tid=11515' 2024-04-06T09:32:52.610 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:32:52.610 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:32:52.611 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=11515 2024-04-06T09:32:52.611 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 3 = 2 -a 1 -le 11515 2024-04-06T09:32:52.611 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:32:52.611 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:32:52.611 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:32:52.813 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:32:52.813 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:32:52.813 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:32:52.813 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:32:52.813 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:32:52.813 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:32:52.813 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:32:52.813 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:32:52.813 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:32:52.813 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:32:52.813 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:32:52.813 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:32:52.814 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:32:52.814 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:32:52.814 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:32:52.814 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:32:52.814 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:32:52.817 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:32:52.817 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:32:52.817 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:32:52.817 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:32:52.817 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:32:52.817 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:32:52.817 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:32:52.817 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:32:52.817 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:32:52.817 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:32:52.817 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:32:52.818 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:32:52.818 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:32:52.818 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:32:52.818 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:32:52.818 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:32:52.819 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:32:52.819 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:32:52.819 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:32:52.819 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:32:52.819 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:32:52.819 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:32:52.819 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:32:52.819 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:32:52.819 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:32:52.819 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:32:52.820 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:32:52.820 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:32:52.820 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:32:52.820 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:32:52.820 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:32:52.820 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:32:52.843 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:32:52.844 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:32:52.844 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:32:52.844 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:32:52.973 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:32:52.973 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:32:52.974 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:32:52.974 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:32:52.974 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:32:52.974 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:32:52.974 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:32:52.974 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:32:52.974 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:32:52.974 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:32:52.975 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:32:52.975 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:32:52.975 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:32:52.977 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:32:52.977 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:32:52.977 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:32:53.021 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 2 2024-04-06T09:32:53.022 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 6 2024-04-06T09:32:53.022 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:32:53.022 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=25, tag_tid=3, entry_tid=1], [object_number=24, tag_tid=3, entry_tid=0], [object_number=27, tag_tid=2, entry_tid=27223], [object_number=26, tag_tid=2, entry_tid=27222]]], state=connected] 2024-04-06T09:32:53.022 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=11, tag_tid=2, entry_tid=11627], [object_number=10, tag_tid=2, entry_tid=11626], [object_number=9, tag_tid=2, entry_tid=11625], [object_number=8, tag_tid=2, entry_tid=11624]]], state=connected] 2024-04-06T09:32:53.025 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:32:53.026 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=25, tag_tid=3, entry_tid=1' 2024-04-06T09:32:53.027 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:32:53.027 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:32:53.027 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:32:53.027 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:32:53.028 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:32:53.028 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:32:53.028 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:32:53.028 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:32:53.028 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:32:53.028 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:32:53.028 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:32:53.028 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:32:53.028 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:32:53.029 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:32:53.029 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:32:53.073 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 2 2024-04-06T09:32:53.073 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 6 2024-04-06T09:32:53.073 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:32:53.074 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=25, tag_tid=3, entry_tid=1], [object_number=24, tag_tid=3, entry_tid=0], [object_number=27, tag_tid=2, entry_tid=27223], [object_number=26, tag_tid=2, entry_tid=27222]]], state=connected] 2024-04-06T09:32:53.074 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=11, tag_tid=2, entry_tid=11627], [object_number=10, tag_tid=2, entry_tid=11626], [object_number=9, tag_tid=2, entry_tid=11625], [object_number=8, tag_tid=2, entry_tid=11624]]], state=connected] 2024-04-06T09:32:53.077 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:32:53.079 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=11, tag_tid=2, entry_tid=11627' 2024-04-06T09:32:53.079 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=25, tag_tid=3, entry_tid=1' -a 'object_number=25, tag_tid=3, entry_tid=1' = 'object_number=11, tag_tid=2, entry_tid=11627' 2024-04-06T09:32:53.079 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=11, tag_tid=2, entry_tid=11627' '!=' 'object_number=11, tag_tid=2, entry_tid=11515' 2024-04-06T09:32:53.079 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:32:53.079 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=11, tag_tid=2, entry_tid=11627' = 'object_number=11, tag_tid=2, entry_tid=11515' 2024-04-06T09:32:53.079 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=11, tag_tid=2, entry_tid=11627' 2024-04-06T09:32:53.079 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=25, tag_tid=3, entry_tid=1' 2024-04-06T09:32:53.079 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:32:53.079 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:32:53.080 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=3 2024-04-06T09:32:53.080 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=11, tag_tid=2, entry_tid=11627' 2024-04-06T09:32:53.081 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:32:53.081 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:32:53.081 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=2 2024-04-06T09:32:53.082 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=25, tag_tid=3, entry_tid=1' 2024-04-06T09:32:53.082 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:32:53.082 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:32:53.083 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:32:53.084 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=11, tag_tid=2, entry_tid=11627' 2024-04-06T09:32:53.084 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:32:53.084 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:32:53.085 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=11627 2024-04-06T09:32:53.085 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 3 = 2 -a 1 -le 11627 2024-04-06T09:32:53.085 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:32:53.085 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:32:53.085 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:32:53.287 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:32:53.287 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:32:53.287 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:32:53.287 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:32:53.287 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:32:53.287 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:32:53.287 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:32:53.287 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:32:53.287 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:32:53.287 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:32:53.287 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:32:53.288 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:32:53.288 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:32:53.288 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:32:53.288 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:32:53.288 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:32:53.288 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:32:53.291 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:32:53.291 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:32:53.291 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:32:53.291 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:32:53.291 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:32:53.292 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:32:53.292 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:32:53.292 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:32:53.292 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:32:53.292 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:32:53.292 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:32:53.292 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:32:53.292 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:32:53.292 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:32:53.292 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:32:53.292 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:32:53.292 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:32:53.292 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:32:53.292 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:32:53.293 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:32:53.293 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:32:53.293 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:32:53.293 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:32:53.293 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:32:53.293 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:32:53.293 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:32:53.293 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:32:53.293 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:32:53.293 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:32:53.293 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:32:53.293 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:32:53.293 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:32:53.316 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:32:53.317 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:32:53.317 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:32:53.317 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:32:53.442 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:32:53.444 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:32:53.444 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:32:53.444 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:32:53.444 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:32:53.444 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:32:53.444 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:32:53.444 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:32:53.444 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:32:53.444 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:32:53.444 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:32:53.444 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:32:53.445 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:32:53.447 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:32:53.447 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:32:53.447 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:32:53.505 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 2 2024-04-06T09:32:53.506 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 6 2024-04-06T09:32:53.506 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:32:53.506 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=25, tag_tid=3, entry_tid=1], [object_number=24, tag_tid=3, entry_tid=0], [object_number=27, tag_tid=2, entry_tid=27223], [object_number=26, tag_tid=2, entry_tid=27222]]], state=connected] 2024-04-06T09:32:53.506 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=8, tag_tid=2, entry_tid=11720], [object_number=11, tag_tid=2, entry_tid=11719], [object_number=10, tag_tid=2, entry_tid=11718], [object_number=9, tag_tid=2, entry_tid=11717]]], state=connected] 2024-04-06T09:32:53.509 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:32:53.510 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=25, tag_tid=3, entry_tid=1' 2024-04-06T09:32:53.511 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:32:53.512 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:32:53.512 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:32:53.512 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:32:53.512 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:32:53.512 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:32:53.512 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:32:53.512 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:32:53.512 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:32:53.512 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:32:53.512 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:32:53.512 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:32:53.513 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:32:53.513 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:32:53.513 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:32:53.557 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 2 2024-04-06T09:32:53.557 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 6 2024-04-06T09:32:53.557 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:32:53.557 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=25, tag_tid=3, entry_tid=1], [object_number=24, tag_tid=3, entry_tid=0], [object_number=27, tag_tid=2, entry_tid=27223], [object_number=26, tag_tid=2, entry_tid=27222]]], state=connected] 2024-04-06T09:32:53.557 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=8, tag_tid=2, entry_tid=11720], [object_number=11, tag_tid=2, entry_tid=11719], [object_number=10, tag_tid=2, entry_tid=11718], [object_number=9, tag_tid=2, entry_tid=11717]]], state=connected] 2024-04-06T09:32:53.560 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:32:53.562 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=8, tag_tid=2, entry_tid=11720' 2024-04-06T09:32:53.563 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=25, tag_tid=3, entry_tid=1' -a 'object_number=25, tag_tid=3, entry_tid=1' = 'object_number=8, tag_tid=2, entry_tid=11720' 2024-04-06T09:32:53.563 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=8, tag_tid=2, entry_tid=11720' '!=' 'object_number=11, tag_tid=2, entry_tid=11627' 2024-04-06T09:32:53.563 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:32:53.563 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=8, tag_tid=2, entry_tid=11720' = 'object_number=11, tag_tid=2, entry_tid=11627' 2024-04-06T09:32:53.563 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=8, tag_tid=2, entry_tid=11720' 2024-04-06T09:32:53.563 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=25, tag_tid=3, entry_tid=1' 2024-04-06T09:32:53.563 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:32:53.563 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:32:53.564 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=3 2024-04-06T09:32:53.564 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=8, tag_tid=2, entry_tid=11720' 2024-04-06T09:32:53.564 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:32:53.564 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:32:53.565 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=2 2024-04-06T09:32:53.566 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=25, tag_tid=3, entry_tid=1' 2024-04-06T09:32:53.566 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:32:53.566 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:32:53.567 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:32:53.568 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=8, tag_tid=2, entry_tid=11720' 2024-04-06T09:32:53.568 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:32:53.568 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:32:53.569 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=11720 2024-04-06T09:32:53.569 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 3 = 2 -a 1 -le 11720 2024-04-06T09:32:53.569 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:32:53.569 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:32:53.569 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:32:53.771 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:32:53.771 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:32:53.771 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:32:53.771 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:32:53.771 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:32:53.771 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:32:53.771 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:32:53.771 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:32:53.771 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:32:53.771 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:32:53.772 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:32:53.772 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:32:53.772 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:32:53.772 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:32:53.772 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:32:53.772 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:32:53.773 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:32:53.775 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:32:53.775 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:32:53.775 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:32:53.775 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:32:53.775 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:32:53.775 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:32:53.775 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:32:53.775 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:32:53.775 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:32:53.776 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:32:53.776 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:32:53.776 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:32:53.776 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:32:53.776 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:32:53.776 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:32:53.776 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:32:53.777 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:32:53.777 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:32:53.777 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:32:53.777 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:32:53.777 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:32:53.777 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:32:53.777 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:32:53.777 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:32:53.777 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:32:53.778 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:32:53.778 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:32:53.778 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:32:53.778 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:32:53.778 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:32:53.778 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:32:53.778 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:32:53.803 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:32:53.803 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:32:53.803 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:32:53.803 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:32:53.941 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:32:53.942 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:32:53.942 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:32:53.942 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:32:53.942 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:32:53.942 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:32:53.942 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:32:53.942 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:32:53.942 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:32:53.942 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:32:53.943 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:32:53.943 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:32:53.943 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:32:53.944 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:32:53.945 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:32:53.945 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:32:53.994 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 2 2024-04-06T09:32:53.994 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 6 2024-04-06T09:32:53.994 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:32:53.994 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=25, tag_tid=3, entry_tid=1], [object_number=24, tag_tid=3, entry_tid=0], [object_number=27, tag_tid=2, entry_tid=27223], [object_number=26, tag_tid=2, entry_tid=27222]]], state=connected] 2024-04-06T09:32:53.995 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=11, tag_tid=2, entry_tid=11819], [object_number=10, tag_tid=2, entry_tid=11818], [object_number=9, tag_tid=2, entry_tid=11817], [object_number=8, tag_tid=2, entry_tid=11816]]], state=connected] 2024-04-06T09:32:53.998 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:32:53.999 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=25, tag_tid=3, entry_tid=1' 2024-04-06T09:32:54.000 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:32:54.000 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:32:54.000 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:32:54.000 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:32:54.000 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:32:54.001 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:32:54.001 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:32:54.001 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:32:54.001 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:32:54.001 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:32:54.001 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:32:54.001 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:32:54.001 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:32:54.002 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:32:54.002 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:32:54.046 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 2 2024-04-06T09:32:54.046 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 6 2024-04-06T09:32:54.047 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:32:54.047 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=25, tag_tid=3, entry_tid=1], [object_number=24, tag_tid=3, entry_tid=0], [object_number=27, tag_tid=2, entry_tid=27223], [object_number=26, tag_tid=2, entry_tid=27222]]], state=connected] 2024-04-06T09:32:54.047 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=11, tag_tid=2, entry_tid=11819], [object_number=10, tag_tid=2, entry_tid=11818], [object_number=9, tag_tid=2, entry_tid=11817], [object_number=8, tag_tid=2, entry_tid=11816]]], state=connected] 2024-04-06T09:32:54.050 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:32:54.052 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=11, tag_tid=2, entry_tid=11819' 2024-04-06T09:32:54.052 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=25, tag_tid=3, entry_tid=1' -a 'object_number=25, tag_tid=3, entry_tid=1' = 'object_number=11, tag_tid=2, entry_tid=11819' 2024-04-06T09:32:54.052 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=11, tag_tid=2, entry_tid=11819' '!=' 'object_number=8, tag_tid=2, entry_tid=11720' 2024-04-06T09:32:54.052 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:32:54.052 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=11, tag_tid=2, entry_tid=11819' = 'object_number=8, tag_tid=2, entry_tid=11720' 2024-04-06T09:32:54.052 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=11, tag_tid=2, entry_tid=11819' 2024-04-06T09:32:54.053 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=25, tag_tid=3, entry_tid=1' 2024-04-06T09:32:54.053 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:32:54.053 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:32:54.053 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=3 2024-04-06T09:32:54.054 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=11, tag_tid=2, entry_tid=11819' 2024-04-06T09:32:54.054 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:32:54.054 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:32:54.055 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=2 2024-04-06T09:32:54.056 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=25, tag_tid=3, entry_tid=1' 2024-04-06T09:32:54.056 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:32:54.056 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:32:54.057 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:32:54.057 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=11, tag_tid=2, entry_tid=11819' 2024-04-06T09:32:54.057 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:32:54.058 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:32:54.059 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=11819 2024-04-06T09:32:54.059 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 3 = 2 -a 1 -le 11819 2024-04-06T09:32:54.059 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:32:54.059 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:32:54.059 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:32:54.261 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:32:54.261 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:32:54.261 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:32:54.261 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:32:54.261 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:32:54.261 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:32:54.261 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:32:54.261 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:32:54.261 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:32:54.262 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:32:54.262 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:32:54.262 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:32:54.262 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:32:54.262 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:32:54.262 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:32:54.262 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:32:54.262 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:32:54.265 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:32:54.265 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:32:54.265 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:32:54.265 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:32:54.265 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:32:54.265 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:32:54.265 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:32:54.265 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:32:54.265 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:32:54.265 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:32:54.265 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:32:54.265 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:32:54.265 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:32:54.265 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:32:54.266 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:32:54.266 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:32:54.267 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:32:54.267 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:32:54.267 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:32:54.267 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:32:54.267 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:32:54.267 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:32:54.267 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:32:54.267 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:32:54.267 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:32:54.267 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:32:54.267 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:32:54.267 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:32:54.267 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:32:54.267 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:32:54.268 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:32:54.268 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:32:54.292 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:32:54.292 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:32:54.292 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:32:54.293 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:32:54.423 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:32:54.424 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:32:54.424 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:32:54.424 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:32:54.424 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:32:54.424 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:32:54.425 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:32:54.425 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:32:54.425 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:32:54.425 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:32:54.425 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:32:54.425 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:32:54.425 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:32:54.427 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:32:54.428 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:32:54.428 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:32:54.480 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 2 2024-04-06T09:32:54.480 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 6 2024-04-06T09:32:54.480 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:32:54.480 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=25, tag_tid=3, entry_tid=1], [object_number=24, tag_tid=3, entry_tid=0], [object_number=27, tag_tid=2, entry_tid=27223], [object_number=26, tag_tid=2, entry_tid=27222]]], state=connected] 2024-04-06T09:32:54.480 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=8, tag_tid=2, entry_tid=11924], [object_number=11, tag_tid=2, entry_tid=11923], [object_number=10, tag_tid=2, entry_tid=11922], [object_number=9, tag_tid=2, entry_tid=11921]]], state=connected] 2024-04-06T09:32:54.484 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:32:54.485 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=25, tag_tid=3, entry_tid=1' 2024-04-06T09:32:54.486 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:32:54.486 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:32:54.486 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:32:54.486 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:32:54.486 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:32:54.487 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:32:54.487 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:32:54.487 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:32:54.487 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:32:54.487 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:32:54.487 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:32:54.487 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:32:54.487 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:32:54.488 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:32:54.488 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:32:54.535 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 2 2024-04-06T09:32:54.535 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 6 2024-04-06T09:32:54.535 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:32:54.535 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=25, tag_tid=3, entry_tid=1], [object_number=24, tag_tid=3, entry_tid=0], [object_number=27, tag_tid=2, entry_tid=27223], [object_number=26, tag_tid=2, entry_tid=27222]]], state=connected] 2024-04-06T09:32:54.535 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=8, tag_tid=2, entry_tid=11924], [object_number=11, tag_tid=2, entry_tid=11923], [object_number=10, tag_tid=2, entry_tid=11922], [object_number=9, tag_tid=2, entry_tid=11921]]], state=connected] 2024-04-06T09:32:54.538 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:32:54.540 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=8, tag_tid=2, entry_tid=11924' 2024-04-06T09:32:54.540 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=25, tag_tid=3, entry_tid=1' -a 'object_number=25, tag_tid=3, entry_tid=1' = 'object_number=8, tag_tid=2, entry_tid=11924' 2024-04-06T09:32:54.540 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=8, tag_tid=2, entry_tid=11924' '!=' 'object_number=11, tag_tid=2, entry_tid=11819' 2024-04-06T09:32:54.540 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:32:54.540 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=8, tag_tid=2, entry_tid=11924' = 'object_number=11, tag_tid=2, entry_tid=11819' 2024-04-06T09:32:54.540 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=8, tag_tid=2, entry_tid=11924' 2024-04-06T09:32:54.541 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=25, tag_tid=3, entry_tid=1' 2024-04-06T09:32:54.541 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:32:54.541 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:32:54.541 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=3 2024-04-06T09:32:54.542 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=8, tag_tid=2, entry_tid=11924' 2024-04-06T09:32:54.542 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:32:54.542 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:32:54.543 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=2 2024-04-06T09:32:54.543 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=25, tag_tid=3, entry_tid=1' 2024-04-06T09:32:54.543 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:32:54.544 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:32:54.544 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:32:54.545 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=8, tag_tid=2, entry_tid=11924' 2024-04-06T09:32:54.545 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:32:54.545 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:32:54.546 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=11924 2024-04-06T09:32:54.547 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 3 = 2 -a 1 -le 11924 2024-04-06T09:32:54.547 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:32:54.547 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:32:54.547 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:32:54.749 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:32:54.749 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:32:54.749 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:32:54.749 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:32:54.749 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:32:54.749 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:32:54.749 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:32:54.749 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:32:54.749 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:32:54.749 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:32:54.749 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:32:54.749 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:32:54.749 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:32:54.750 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:32:54.750 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:32:54.750 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:32:54.750 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:32:54.753 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:32:54.754 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:32:54.754 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:32:54.754 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:32:54.754 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:32:54.754 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:32:54.754 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:32:54.754 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:32:54.754 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:32:54.754 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:32:54.754 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:32:54.754 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:32:54.755 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:32:54.755 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:32:54.755 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:32:54.755 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:32:54.755 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:32:54.755 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:32:54.755 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:32:54.755 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:32:54.755 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:32:54.755 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:32:54.755 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:32:54.755 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:32:54.755 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:32:54.755 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:32:54.755 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:32:54.755 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:32:54.756 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:32:54.756 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:32:54.756 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:32:54.756 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:32:54.778 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:32:54.779 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:32:54.779 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:32:54.779 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:32:54.910 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:32:54.911 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:32:54.911 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:32:54.912 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:32:54.912 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:32:54.912 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:32:54.912 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:32:54.912 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:32:54.912 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:32:54.912 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:32:54.912 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:32:54.912 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:32:54.913 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:32:54.914 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:32:54.915 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:32:54.915 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:32:54.959 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 2 2024-04-06T09:32:54.959 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 6 2024-04-06T09:32:54.959 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:32:54.959 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=25, tag_tid=3, entry_tid=1], [object_number=24, tag_tid=3, entry_tid=0], [object_number=27, tag_tid=2, entry_tid=27223], [object_number=26, tag_tid=2, entry_tid=27222]]], state=connected] 2024-04-06T09:32:54.960 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=8, tag_tid=2, entry_tid=12040], [object_number=11, tag_tid=2, entry_tid=12039], [object_number=10, tag_tid=2, entry_tid=12038], [object_number=9, tag_tid=2, entry_tid=12037]]], state=connected] 2024-04-06T09:32:54.963 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:32:54.964 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=25, tag_tid=3, entry_tid=1' 2024-04-06T09:32:54.965 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:32:54.965 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:32:54.965 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:32:54.965 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:32:54.966 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:32:54.966 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:32:54.966 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:32:54.966 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:32:54.966 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:32:54.966 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:32:54.966 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:32:54.966 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:32:54.967 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:32:54.967 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:32:54.967 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:32:55.012 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 2 2024-04-06T09:32:55.012 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 6 2024-04-06T09:32:55.012 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:32:55.012 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=25, tag_tid=3, entry_tid=1], [object_number=24, tag_tid=3, entry_tid=0], [object_number=27, tag_tid=2, entry_tid=27223], [object_number=26, tag_tid=2, entry_tid=27222]]], state=connected] 2024-04-06T09:32:55.012 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=8, tag_tid=2, entry_tid=12040], [object_number=11, tag_tid=2, entry_tid=12039], [object_number=10, tag_tid=2, entry_tid=12038], [object_number=9, tag_tid=2, entry_tid=12037]]], state=connected] 2024-04-06T09:32:55.015 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:32:55.017 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=8, tag_tid=2, entry_tid=12040' 2024-04-06T09:32:55.017 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=25, tag_tid=3, entry_tid=1' -a 'object_number=25, tag_tid=3, entry_tid=1' = 'object_number=8, tag_tid=2, entry_tid=12040' 2024-04-06T09:32:55.017 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=8, tag_tid=2, entry_tid=12040' '!=' 'object_number=8, tag_tid=2, entry_tid=11924' 2024-04-06T09:32:55.017 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:32:55.017 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=8, tag_tid=2, entry_tid=12040' = 'object_number=8, tag_tid=2, entry_tid=11924' 2024-04-06T09:32:55.018 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=8, tag_tid=2, entry_tid=12040' 2024-04-06T09:32:55.018 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=25, tag_tid=3, entry_tid=1' 2024-04-06T09:32:55.018 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:32:55.018 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:32:55.018 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=3 2024-04-06T09:32:55.019 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=8, tag_tid=2, entry_tid=12040' 2024-04-06T09:32:55.019 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:32:55.019 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:32:55.020 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=2 2024-04-06T09:32:55.021 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=25, tag_tid=3, entry_tid=1' 2024-04-06T09:32:55.021 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:32:55.021 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:32:55.023 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:32:55.023 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=8, tag_tid=2, entry_tid=12040' 2024-04-06T09:32:55.023 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:32:55.023 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:32:55.024 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=12040 2024-04-06T09:32:55.024 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 3 = 2 -a 1 -le 12040 2024-04-06T09:32:55.024 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:32:55.024 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:32:55.024 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:32:55.225 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:32:55.225 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:32:55.226 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:32:55.226 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:32:55.226 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:32:55.226 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:32:55.226 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:32:55.226 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:32:55.226 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:32:55.226 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:32:55.226 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:32:55.226 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:32:55.226 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:32:55.226 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:32:55.226 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:32:55.227 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:32:55.227 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:32:55.230 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:32:55.230 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:32:55.230 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:32:55.230 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:32:55.231 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:32:55.231 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:32:55.231 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:32:55.231 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:32:55.231 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:32:55.231 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:32:55.231 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:32:55.231 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:32:55.231 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:32:55.231 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:32:55.231 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:32:55.231 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:32:55.231 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:32:55.232 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:32:55.232 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:32:55.232 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:32:55.232 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:32:55.232 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:32:55.232 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:32:55.232 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:32:55.232 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:32:55.232 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:32:55.232 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:32:55.232 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:32:55.232 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:32:55.232 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:32:55.232 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:32:55.232 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:32:55.255 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:32:55.256 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:32:55.256 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:32:55.256 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:32:55.380 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:32:55.381 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:32:55.381 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:32:55.381 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:32:55.381 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:32:55.381 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:32:55.382 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:32:55.382 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:32:55.382 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:32:55.382 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:32:55.382 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:32:55.382 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:32:55.383 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:32:55.384 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:32:55.384 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:32:55.384 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:32:55.429 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 2 2024-04-06T09:32:55.429 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 6 2024-04-06T09:32:55.429 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:32:55.429 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=25, tag_tid=3, entry_tid=1], [object_number=24, tag_tid=3, entry_tid=0], [object_number=27, tag_tid=2, entry_tid=27223], [object_number=26, tag_tid=2, entry_tid=27222]]], state=connected] 2024-04-06T09:32:55.430 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=8, tag_tid=2, entry_tid=12152], [object_number=11, tag_tid=2, entry_tid=12151], [object_number=10, tag_tid=2, entry_tid=12150], [object_number=9, tag_tid=2, entry_tid=12149]]], state=connected] 2024-04-06T09:32:55.433 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:32:55.434 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=25, tag_tid=3, entry_tid=1' 2024-04-06T09:32:55.435 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:32:55.435 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:32:55.435 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:32:55.436 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:32:55.436 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:32:55.436 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:32:55.436 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:32:55.436 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:32:55.436 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:32:55.436 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:32:55.436 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:32:55.436 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:32:55.437 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:32:55.437 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:32:55.437 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:32:55.484 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 2 2024-04-06T09:32:55.484 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 6 2024-04-06T09:32:55.484 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:32:55.484 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=25, tag_tid=3, entry_tid=1], [object_number=24, tag_tid=3, entry_tid=0], [object_number=27, tag_tid=2, entry_tid=27223], [object_number=26, tag_tid=2, entry_tid=27222]]], state=connected] 2024-04-06T09:32:55.484 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=8, tag_tid=2, entry_tid=12152], [object_number=11, tag_tid=2, entry_tid=12151], [object_number=10, tag_tid=2, entry_tid=12150], [object_number=9, tag_tid=2, entry_tid=12149]]], state=connected] 2024-04-06T09:32:55.487 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:32:55.489 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=8, tag_tid=2, entry_tid=12152' 2024-04-06T09:32:55.490 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=25, tag_tid=3, entry_tid=1' -a 'object_number=25, tag_tid=3, entry_tid=1' = 'object_number=8, tag_tid=2, entry_tid=12152' 2024-04-06T09:32:55.490 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=8, tag_tid=2, entry_tid=12152' '!=' 'object_number=8, tag_tid=2, entry_tid=12040' 2024-04-06T09:32:55.490 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:32:55.490 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=8, tag_tid=2, entry_tid=12152' = 'object_number=8, tag_tid=2, entry_tid=12040' 2024-04-06T09:32:55.490 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=8, tag_tid=2, entry_tid=12152' 2024-04-06T09:32:55.490 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=25, tag_tid=3, entry_tid=1' 2024-04-06T09:32:55.490 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:32:55.490 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:32:55.491 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=3 2024-04-06T09:32:55.491 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=8, tag_tid=2, entry_tid=12152' 2024-04-06T09:32:55.491 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:32:55.491 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:32:55.492 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=2 2024-04-06T09:32:55.493 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=25, tag_tid=3, entry_tid=1' 2024-04-06T09:32:55.493 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:32:55.493 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:32:55.494 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:32:55.495 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=8, tag_tid=2, entry_tid=12152' 2024-04-06T09:32:55.495 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:32:55.495 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:32:55.496 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=12152 2024-04-06T09:32:55.496 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 3 = 2 -a 1 -le 12152 2024-04-06T09:32:55.496 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:32:55.496 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:32:55.497 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:32:55.698 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:32:55.699 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:32:55.699 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:32:55.699 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:32:55.699 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:32:55.699 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:32:55.699 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:32:55.699 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:32:55.699 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:32:55.699 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:32:55.699 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:32:55.699 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:32:55.700 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:32:55.700 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:32:55.700 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:32:55.700 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:32:55.700 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:32:55.703 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:32:55.703 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:32:55.703 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:32:55.703 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:32:55.703 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:32:55.704 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:32:55.704 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:32:55.704 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:32:55.704 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:32:55.704 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:32:55.704 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:32:55.704 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:32:55.704 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:32:55.704 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:32:55.704 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:32:55.704 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:32:55.704 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:32:55.704 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:32:55.704 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:32:55.704 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:32:55.705 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:32:55.705 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:32:55.705 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:32:55.705 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:32:55.705 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:32:55.705 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:32:55.705 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:32:55.705 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:32:55.705 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:32:55.705 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:32:55.705 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:32:55.705 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:32:55.729 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:32:55.729 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:32:55.729 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:32:55.729 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:32:55.860 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:32:55.860 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:32:55.860 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:32:55.860 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:32:55.860 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:32:55.860 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:32:55.860 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:32:55.860 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:32:55.860 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:32:55.861 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:32:55.861 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:32:55.861 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:32:55.861 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:32:55.864 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:32:55.864 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:32:55.864 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:32:55.914 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 2 2024-04-06T09:32:55.914 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 6 2024-04-06T09:32:55.914 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:32:55.914 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=25, tag_tid=3, entry_tid=1], [object_number=24, tag_tid=3, entry_tid=0], [object_number=27, tag_tid=2, entry_tid=27223], [object_number=26, tag_tid=2, entry_tid=27222]]], state=connected] 2024-04-06T09:32:55.914 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=8, tag_tid=2, entry_tid=12264], [object_number=11, tag_tid=2, entry_tid=12263], [object_number=10, tag_tid=2, entry_tid=12262], [object_number=9, tag_tid=2, entry_tid=12261]]], state=connected] 2024-04-06T09:32:55.918 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:32:55.919 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=25, tag_tid=3, entry_tid=1' 2024-04-06T09:32:55.920 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:32:55.920 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:32:55.920 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:32:55.920 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:32:55.920 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:32:55.921 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:32:55.921 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:32:55.921 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:32:55.921 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:32:55.921 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:32:55.921 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:32:55.921 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:32:55.921 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:32:55.922 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:32:55.922 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:32:55.967 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 2 2024-04-06T09:32:55.967 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 6 2024-04-06T09:32:55.967 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:32:55.967 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=25, tag_tid=3, entry_tid=1], [object_number=24, tag_tid=3, entry_tid=0], [object_number=27, tag_tid=2, entry_tid=27223], [object_number=26, tag_tid=2, entry_tid=27222]]], state=connected] 2024-04-06T09:32:55.967 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=8, tag_tid=2, entry_tid=12264], [object_number=11, tag_tid=2, entry_tid=12263], [object_number=10, tag_tid=2, entry_tid=12262], [object_number=9, tag_tid=2, entry_tid=12261]]], state=connected] 2024-04-06T09:32:55.970 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:32:55.972 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=8, tag_tid=2, entry_tid=12264' 2024-04-06T09:32:55.972 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=25, tag_tid=3, entry_tid=1' -a 'object_number=25, tag_tid=3, entry_tid=1' = 'object_number=8, tag_tid=2, entry_tid=12264' 2024-04-06T09:32:55.972 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=8, tag_tid=2, entry_tid=12264' '!=' 'object_number=8, tag_tid=2, entry_tid=12152' 2024-04-06T09:32:55.972 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:32:55.972 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=8, tag_tid=2, entry_tid=12264' = 'object_number=8, tag_tid=2, entry_tid=12152' 2024-04-06T09:32:55.972 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=8, tag_tid=2, entry_tid=12264' 2024-04-06T09:32:55.973 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=25, tag_tid=3, entry_tid=1' 2024-04-06T09:32:55.973 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:32:55.973 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:32:55.974 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=3 2024-04-06T09:32:55.974 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=8, tag_tid=2, entry_tid=12264' 2024-04-06T09:32:55.974 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:32:55.974 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:32:55.975 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=2 2024-04-06T09:32:55.976 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=25, tag_tid=3, entry_tid=1' 2024-04-06T09:32:55.976 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:32:55.976 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:32:55.977 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:32:55.977 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=8, tag_tid=2, entry_tid=12264' 2024-04-06T09:32:55.978 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:32:55.978 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:32:55.979 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=12264 2024-04-06T09:32:55.979 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 3 = 2 -a 1 -le 12264 2024-04-06T09:32:55.979 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:32:55.979 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:32:55.979 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:32:56.181 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:32:56.181 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:32:56.181 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:32:56.181 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:32:56.181 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:32:56.181 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:32:56.181 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:32:56.181 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:32:56.181 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:32:56.181 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:32:56.181 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:32:56.181 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:32:56.182 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:32:56.182 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:32:56.182 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:32:56.182 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:32:56.182 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:32:56.186 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:32:56.186 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:32:56.186 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:32:56.186 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:32:56.186 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:32:56.186 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:32:56.186 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:32:56.186 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:32:56.187 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:32:56.187 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:32:56.187 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:32:56.187 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:32:56.187 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:32:56.187 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:32:56.187 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:32:56.187 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:32:56.187 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:32:56.187 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:32:56.187 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:32:56.187 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:32:56.187 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:32:56.187 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:32:56.187 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:32:56.188 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:32:56.188 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:32:56.188 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:32:56.188 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:32:56.188 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:32:56.188 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:32:56.188 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:32:56.188 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:32:56.188 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:32:56.211 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:32:56.212 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:32:56.212 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:32:56.212 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:32:56.341 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:32:56.343 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:32:56.343 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:32:56.343 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:32:56.343 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:32:56.343 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:32:56.343 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:32:56.343 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:32:56.343 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:32:56.343 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:32:56.343 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:32:56.344 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:32:56.344 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:32:56.345 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:32:56.345 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:32:56.345 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:32:56.397 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 2 2024-04-06T09:32:56.397 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 6 2024-04-06T09:32:56.397 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:32:56.397 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=25, tag_tid=3, entry_tid=1], [object_number=24, tag_tid=3, entry_tid=0], [object_number=27, tag_tid=2, entry_tid=27223], [object_number=26, tag_tid=2, entry_tid=27222]]], state=connected] 2024-04-06T09:32:56.397 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=8, tag_tid=2, entry_tid=12368], [object_number=11, tag_tid=2, entry_tid=12367], [object_number=10, tag_tid=2, entry_tid=12366], [object_number=9, tag_tid=2, entry_tid=12365]]], state=connected] 2024-04-06T09:32:56.400 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:32:56.401 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=25, tag_tid=3, entry_tid=1' 2024-04-06T09:32:56.403 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:32:56.403 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:32:56.403 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:32:56.403 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:32:56.403 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:32:56.403 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:32:56.403 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:32:56.403 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:32:56.403 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:32:56.403 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:32:56.403 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:32:56.403 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:32:56.404 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:32:56.404 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:32:56.404 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:32:56.450 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 2 2024-04-06T09:32:56.450 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 6 2024-04-06T09:32:56.450 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:32:56.450 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=25, tag_tid=3, entry_tid=1], [object_number=24, tag_tid=3, entry_tid=0], [object_number=27, tag_tid=2, entry_tid=27223], [object_number=26, tag_tid=2, entry_tid=27222]]], state=connected] 2024-04-06T09:32:56.450 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=8, tag_tid=2, entry_tid=12368], [object_number=11, tag_tid=2, entry_tid=12367], [object_number=10, tag_tid=2, entry_tid=12366], [object_number=9, tag_tid=2, entry_tid=12365]]], state=connected] 2024-04-06T09:32:56.453 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:32:56.455 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=8, tag_tid=2, entry_tid=12368' 2024-04-06T09:32:56.455 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=25, tag_tid=3, entry_tid=1' -a 'object_number=25, tag_tid=3, entry_tid=1' = 'object_number=8, tag_tid=2, entry_tid=12368' 2024-04-06T09:32:56.456 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=8, tag_tid=2, entry_tid=12368' '!=' 'object_number=8, tag_tid=2, entry_tid=12264' 2024-04-06T09:32:56.456 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:32:56.456 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=8, tag_tid=2, entry_tid=12368' = 'object_number=8, tag_tid=2, entry_tid=12264' 2024-04-06T09:32:56.456 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=8, tag_tid=2, entry_tid=12368' 2024-04-06T09:32:56.456 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=25, tag_tid=3, entry_tid=1' 2024-04-06T09:32:56.456 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:32:56.456 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:32:56.456 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=3 2024-04-06T09:32:56.457 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=8, tag_tid=2, entry_tid=12368' 2024-04-06T09:32:56.457 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:32:56.457 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:32:56.458 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=2 2024-04-06T09:32:56.459 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=25, tag_tid=3, entry_tid=1' 2024-04-06T09:32:56.459 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:32:56.459 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:32:56.460 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:32:56.461 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=8, tag_tid=2, entry_tid=12368' 2024-04-06T09:32:56.461 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:32:56.461 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:32:56.462 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=12368 2024-04-06T09:32:56.462 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 3 = 2 -a 1 -le 12368 2024-04-06T09:32:56.462 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:32:56.462 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:32:56.462 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:32:56.664 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:32:56.664 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:32:56.664 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:32:56.664 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:32:56.664 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:32:56.664 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:32:56.664 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:32:56.664 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:32:56.664 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:32:56.664 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:32:56.664 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:32:56.665 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:32:56.665 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:32:56.665 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:32:56.665 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:32:56.665 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:32:56.665 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:32:56.667 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:32:56.667 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:32:56.667 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:32:56.667 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:32:56.667 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:32:56.667 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:32:56.667 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:32:56.668 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:32:56.668 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:32:56.668 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:32:56.668 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:32:56.668 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:32:56.668 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:32:56.668 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:32:56.668 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:32:56.668 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:32:56.669 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:32:56.669 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:32:56.669 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:32:56.670 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:32:56.670 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:32:56.670 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:32:56.670 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:32:56.670 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:32:56.670 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:32:56.670 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:32:56.670 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:32:56.670 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:32:56.670 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:32:56.670 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:32:56.670 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:32:56.670 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:32:56.693 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:32:56.694 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:32:56.694 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:32:56.694 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:32:56.826 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:32:56.827 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:32:56.827 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:32:56.827 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:32:56.827 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:32:56.827 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:32:56.827 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:32:56.828 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:32:56.828 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:32:56.828 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:32:56.828 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:32:56.828 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:32:56.829 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:32:56.830 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:32:56.830 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:32:56.830 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:32:56.912 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 2 2024-04-06T09:32:56.912 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 6 2024-04-06T09:32:56.913 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:32:56.913 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=25, tag_tid=3, entry_tid=1], [object_number=24, tag_tid=3, entry_tid=0], [object_number=27, tag_tid=2, entry_tid=27223], [object_number=26, tag_tid=2, entry_tid=27222]]], state=connected] 2024-04-06T09:32:56.913 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=11, tag_tid=2, entry_tid=12479], [object_number=10, tag_tid=2, entry_tid=12478], [object_number=9, tag_tid=2, entry_tid=12477], [object_number=8, tag_tid=2, entry_tid=12476]]], state=connected] 2024-04-06T09:32:56.916 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:32:56.918 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=25, tag_tid=3, entry_tid=1' 2024-04-06T09:32:56.919 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:32:56.919 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:32:56.919 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:32:56.919 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:32:56.919 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:32:56.920 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:32:56.920 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:32:56.920 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:32:56.920 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:32:56.920 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:32:56.920 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:32:56.920 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:32:56.921 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:32:56.921 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:32:56.921 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:32:56.972 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 2 2024-04-06T09:32:56.972 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 6 2024-04-06T09:32:56.972 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:32:56.972 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=25, tag_tid=3, entry_tid=1], [object_number=24, tag_tid=3, entry_tid=0], [object_number=27, tag_tid=2, entry_tid=27223], [object_number=26, tag_tid=2, entry_tid=27222]]], state=connected] 2024-04-06T09:32:56.972 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=11, tag_tid=2, entry_tid=12479], [object_number=10, tag_tid=2, entry_tid=12478], [object_number=9, tag_tid=2, entry_tid=12477], [object_number=8, tag_tid=2, entry_tid=12476]]], state=connected] 2024-04-06T09:32:56.980 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:32:56.981 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=11, tag_tid=2, entry_tid=12479' 2024-04-06T09:32:56.981 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=25, tag_tid=3, entry_tid=1' -a 'object_number=25, tag_tid=3, entry_tid=1' = 'object_number=11, tag_tid=2, entry_tid=12479' 2024-04-06T09:32:56.981 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=11, tag_tid=2, entry_tid=12479' '!=' 'object_number=8, tag_tid=2, entry_tid=12368' 2024-04-06T09:32:56.981 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:32:56.981 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=11, tag_tid=2, entry_tid=12479' = 'object_number=8, tag_tid=2, entry_tid=12368' 2024-04-06T09:32:56.981 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=11, tag_tid=2, entry_tid=12479' 2024-04-06T09:32:56.982 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=25, tag_tid=3, entry_tid=1' 2024-04-06T09:32:56.982 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:32:56.982 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:32:56.983 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=3 2024-04-06T09:32:56.983 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=11, tag_tid=2, entry_tid=12479' 2024-04-06T09:32:56.983 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:32:56.984 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:32:56.984 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=2 2024-04-06T09:32:56.985 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=25, tag_tid=3, entry_tid=1' 2024-04-06T09:32:56.985 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:32:56.985 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:32:56.986 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:32:56.987 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=11, tag_tid=2, entry_tid=12479' 2024-04-06T09:32:56.987 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:32:56.987 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:32:56.988 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=12479 2024-04-06T09:32:56.988 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 3 = 2 -a 1 -le 12479 2024-04-06T09:32:56.988 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:32:56.988 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:32:56.988 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:32:57.190 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:32:57.190 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:32:57.190 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:32:57.190 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:32:57.190 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:32:57.190 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:32:57.190 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:32:57.190 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:32:57.191 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:32:57.191 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:32:57.191 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:32:57.191 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:32:57.191 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:32:57.191 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:32:57.191 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:32:57.192 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:32:57.192 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:32:57.194 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:32:57.194 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:32:57.194 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:32:57.194 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:32:57.194 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:32:57.194 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:32:57.195 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:32:57.195 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:32:57.195 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:32:57.195 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:32:57.195 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:32:57.195 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:32:57.195 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:32:57.195 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:32:57.195 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:32:57.195 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:32:57.195 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:32:57.195 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:32:57.195 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:32:57.196 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:32:57.196 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:32:57.196 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:32:57.196 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:32:57.196 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:32:57.196 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:32:57.196 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:32:57.196 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:32:57.196 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:32:57.197 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:32:57.197 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:32:57.197 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:32:57.197 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:32:57.219 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:32:57.220 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:32:57.220 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:32:57.220 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:32:57.361 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:32:57.362 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:32:57.362 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:32:57.363 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:32:57.363 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:32:57.363 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:32:57.363 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:32:57.363 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:32:57.363 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:32:57.363 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:32:57.363 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:32:57.364 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:32:57.364 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:32:57.365 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:32:57.366 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:32:57.367 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:32:57.418 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 3 2024-04-06T09:32:57.418 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 6 2024-04-06T09:32:57.418 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:32:57.418 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=25, tag_tid=3, entry_tid=1], [object_number=24, tag_tid=3, entry_tid=0], [object_number=27, tag_tid=2, entry_tid=27223], [object_number=26, tag_tid=2, entry_tid=27222]]], state=connected] 2024-04-06T09:32:57.418 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=12, tag_tid=2, entry_tid=12544], [object_number=15, tag_tid=2, entry_tid=12543], [object_number=14, tag_tid=2, entry_tid=12542], [object_number=13, tag_tid=2, entry_tid=12541]]], state=connected] 2024-04-06T09:32:57.421 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:32:57.424 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=25, tag_tid=3, entry_tid=1' 2024-04-06T09:32:57.424 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:32:57.424 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:32:57.424 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:32:57.424 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:32:57.424 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:32:57.424 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:32:57.424 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:32:57.424 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:32:57.424 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:32:57.424 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:32:57.424 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:32:57.425 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:32:57.425 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:32:57.425 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:32:57.425 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:32:57.469 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 3 2024-04-06T09:32:57.469 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 6 2024-04-06T09:32:57.470 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:32:57.470 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=25, tag_tid=3, entry_tid=1], [object_number=24, tag_tid=3, entry_tid=0], [object_number=27, tag_tid=2, entry_tid=27223], [object_number=26, tag_tid=2, entry_tid=27222]]], state=connected] 2024-04-06T09:32:57.470 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=12, tag_tid=2, entry_tid=12544], [object_number=15, tag_tid=2, entry_tid=12543], [object_number=14, tag_tid=2, entry_tid=12542], [object_number=13, tag_tid=2, entry_tid=12541]]], state=connected] 2024-04-06T09:32:57.473 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:32:57.475 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=12, tag_tid=2, entry_tid=12544' 2024-04-06T09:32:57.475 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=25, tag_tid=3, entry_tid=1' -a 'object_number=25, tag_tid=3, entry_tid=1' = 'object_number=12, tag_tid=2, entry_tid=12544' 2024-04-06T09:32:57.475 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=12, tag_tid=2, entry_tid=12544' '!=' 'object_number=11, tag_tid=2, entry_tid=12479' 2024-04-06T09:32:57.475 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:32:57.475 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=12, tag_tid=2, entry_tid=12544' = 'object_number=11, tag_tid=2, entry_tid=12479' 2024-04-06T09:32:57.475 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=12, tag_tid=2, entry_tid=12544' 2024-04-06T09:32:57.476 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=25, tag_tid=3, entry_tid=1' 2024-04-06T09:32:57.476 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:32:57.476 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:32:57.476 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=3 2024-04-06T09:32:57.477 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=12, tag_tid=2, entry_tid=12544' 2024-04-06T09:32:57.477 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:32:57.477 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:32:57.478 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=2 2024-04-06T09:32:57.479 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=25, tag_tid=3, entry_tid=1' 2024-04-06T09:32:57.479 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:32:57.479 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:32:57.480 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:32:57.480 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=12, tag_tid=2, entry_tid=12544' 2024-04-06T09:32:57.481 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:32:57.481 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:32:57.482 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=12544 2024-04-06T09:32:57.482 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 3 = 2 -a 1 -le 12544 2024-04-06T09:32:57.482 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:32:57.482 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:32:57.482 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:32:57.684 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:32:57.684 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:32:57.684 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:32:57.684 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:32:57.684 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:32:57.684 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:32:57.684 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:32:57.684 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:32:57.684 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:32:57.685 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:32:57.685 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:32:57.685 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:32:57.685 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:32:57.685 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:32:57.685 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:32:57.685 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:32:57.685 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:32:57.688 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:32:57.688 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:32:57.688 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:32:57.688 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:32:57.688 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:32:57.688 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:32:57.689 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:32:57.689 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:32:57.689 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:32:57.689 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:32:57.689 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:32:57.689 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:32:57.689 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:32:57.689 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:32:57.689 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:32:57.689 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:32:57.689 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:32:57.689 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:32:57.689 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:32:57.689 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:32:57.690 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:32:57.690 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:32:57.690 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:32:57.690 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:32:57.690 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:32:57.690 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:32:57.690 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:32:57.690 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:32:57.690 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:32:57.690 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:32:57.690 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:32:57.690 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:32:57.714 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:32:57.715 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:32:57.715 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:32:57.715 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:32:57.842 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:32:57.843 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:32:57.843 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:32:57.843 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:32:57.843 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:32:57.843 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:32:57.843 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:32:57.843 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:32:57.843 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:32:57.843 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:32:57.844 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:32:57.844 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:32:57.844 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:32:57.845 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:32:57.845 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:32:57.845 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:32:57.894 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 3 2024-04-06T09:32:57.894 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 6 2024-04-06T09:32:57.894 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:32:57.894 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=25, tag_tid=3, entry_tid=1], [object_number=24, tag_tid=3, entry_tid=0], [object_number=27, tag_tid=2, entry_tid=27223], [object_number=26, tag_tid=2, entry_tid=27222]]], state=connected] 2024-04-06T09:32:57.895 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=13, tag_tid=2, entry_tid=12669], [object_number=12, tag_tid=2, entry_tid=12668], [object_number=15, tag_tid=2, entry_tid=12667], [object_number=14, tag_tid=2, entry_tid=12666]]], state=connected] 2024-04-06T09:32:57.898 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:32:57.899 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=25, tag_tid=3, entry_tid=1' 2024-04-06T09:32:57.900 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:32:57.900 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:32:57.900 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:32:57.900 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:32:57.900 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:32:57.900 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:32:57.900 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:32:57.900 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:32:57.901 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:32:57.901 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:32:57.901 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:32:57.901 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:32:57.901 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:32:57.902 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:32:57.902 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:32:58.156 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 3 2024-04-06T09:32:58.156 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 6 2024-04-06T09:32:58.156 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:32:58.156 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=25, tag_tid=3, entry_tid=1], [object_number=24, tag_tid=3, entry_tid=0], [object_number=27, tag_tid=2, entry_tid=27223], [object_number=26, tag_tid=2, entry_tid=27222]]], state=connected] 2024-04-06T09:32:58.156 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=13, tag_tid=2, entry_tid=12669], [object_number=12, tag_tid=2, entry_tid=12668], [object_number=15, tag_tid=2, entry_tid=12667], [object_number=14, tag_tid=2, entry_tid=12666]]], state=connected] 2024-04-06T09:32:58.160 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:32:58.162 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=13, tag_tid=2, entry_tid=12669' 2024-04-06T09:32:58.162 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=25, tag_tid=3, entry_tid=1' -a 'object_number=25, tag_tid=3, entry_tid=1' = 'object_number=13, tag_tid=2, entry_tid=12669' 2024-04-06T09:32:58.162 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=13, tag_tid=2, entry_tid=12669' '!=' 'object_number=12, tag_tid=2, entry_tid=12544' 2024-04-06T09:32:58.162 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:32:58.162 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=13, tag_tid=2, entry_tid=12669' = 'object_number=12, tag_tid=2, entry_tid=12544' 2024-04-06T09:32:58.163 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=13, tag_tid=2, entry_tid=12669' 2024-04-06T09:32:58.163 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=25, tag_tid=3, entry_tid=1' 2024-04-06T09:32:58.163 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:32:58.163 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:32:58.164 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=3 2024-04-06T09:32:58.165 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=13, tag_tid=2, entry_tid=12669' 2024-04-06T09:32:58.165 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:32:58.165 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:32:58.166 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=2 2024-04-06T09:32:58.167 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=25, tag_tid=3, entry_tid=1' 2024-04-06T09:32:58.167 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:32:58.167 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:32:58.168 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:32:58.169 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=13, tag_tid=2, entry_tid=12669' 2024-04-06T09:32:58.169 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:32:58.169 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:32:58.170 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=12669 2024-04-06T09:32:58.170 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 3 = 2 -a 1 -le 12669 2024-04-06T09:32:58.170 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:32:58.170 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:32:58.171 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:32:58.372 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:32:58.372 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:32:58.372 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:32:58.372 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:32:58.373 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:32:58.373 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:32:58.373 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:32:58.373 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:32:58.373 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:32:58.373 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:32:58.373 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:32:58.373 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:32:58.373 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:32:58.373 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:32:58.373 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:32:58.374 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:32:58.374 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:32:58.376 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:32:58.377 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:32:58.377 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:32:58.377 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:32:58.377 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:32:58.377 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:32:58.377 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:32:58.377 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:32:58.377 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:32:58.377 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:32:58.377 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:32:58.377 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:32:58.377 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:32:58.377 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:32:58.377 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:32:58.377 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:32:58.379 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:32:58.379 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:32:58.379 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:32:58.379 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:32:58.379 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:32:58.379 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:32:58.379 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:32:58.379 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:32:58.379 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:32:58.379 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:32:58.379 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:32:58.379 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:32:58.379 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:32:58.380 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:32:58.380 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:32:58.380 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:32:58.405 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:32:58.405 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:32:58.405 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:32:58.405 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:32:58.535 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:32:58.536 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:32:58.536 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:32:58.536 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:32:58.536 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:32:58.536 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:32:58.536 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:32:58.536 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:32:58.536 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:32:58.536 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:32:58.537 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:32:58.537 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:32:58.537 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:32:58.538 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:32:58.538 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:32:58.539 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:32:59.011 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 3 2024-04-06T09:32:59.011 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 6 2024-04-06T09:32:59.011 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:32:59.011 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=25, tag_tid=3, entry_tid=1], [object_number=24, tag_tid=3, entry_tid=0], [object_number=27, tag_tid=2, entry_tid=27223], [object_number=26, tag_tid=2, entry_tid=27222]]], state=connected] 2024-04-06T09:32:59.011 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=14, tag_tid=2, entry_tid=12826], [object_number=13, tag_tid=2, entry_tid=12825], [object_number=12, tag_tid=2, entry_tid=12824], [object_number=15, tag_tid=2, entry_tid=12823]]], state=connected] 2024-04-06T09:32:59.019 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:32:59.021 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=25, tag_tid=3, entry_tid=1' 2024-04-06T09:32:59.022 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:32:59.022 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:32:59.022 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:32:59.022 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:32:59.022 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:32:59.023 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:32:59.023 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:32:59.023 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:32:59.023 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:32:59.023 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:32:59.023 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:32:59.023 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:32:59.025 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:32:59.025 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:32:59.025 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:32:59.079 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 3 2024-04-06T09:32:59.080 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 6 2024-04-06T09:32:59.080 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:32:59.080 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=25, tag_tid=3, entry_tid=1], [object_number=24, tag_tid=3, entry_tid=0], [object_number=27, tag_tid=2, entry_tid=27223], [object_number=26, tag_tid=2, entry_tid=27222]]], state=connected] 2024-04-06T09:32:59.080 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=14, tag_tid=2, entry_tid=12826], [object_number=13, tag_tid=2, entry_tid=12825], [object_number=12, tag_tid=2, entry_tid=12824], [object_number=15, tag_tid=2, entry_tid=12823]]], state=connected] 2024-04-06T09:32:59.084 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:32:59.085 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=14, tag_tid=2, entry_tid=12826' 2024-04-06T09:32:59.085 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=25, tag_tid=3, entry_tid=1' -a 'object_number=25, tag_tid=3, entry_tid=1' = 'object_number=14, tag_tid=2, entry_tid=12826' 2024-04-06T09:32:59.085 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=14, tag_tid=2, entry_tid=12826' '!=' 'object_number=13, tag_tid=2, entry_tid=12669' 2024-04-06T09:32:59.086 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:32:59.086 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=14, tag_tid=2, entry_tid=12826' = 'object_number=13, tag_tid=2, entry_tid=12669' 2024-04-06T09:32:59.086 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=14, tag_tid=2, entry_tid=12826' 2024-04-06T09:32:59.086 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=25, tag_tid=3, entry_tid=1' 2024-04-06T09:32:59.086 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:32:59.087 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:32:59.088 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=3 2024-04-06T09:32:59.088 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=14, tag_tid=2, entry_tid=12826' 2024-04-06T09:32:59.088 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:32:59.088 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:32:59.089 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=2 2024-04-06T09:32:59.090 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=25, tag_tid=3, entry_tid=1' 2024-04-06T09:32:59.090 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:32:59.090 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:32:59.091 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:32:59.091 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=14, tag_tid=2, entry_tid=12826' 2024-04-06T09:32:59.091 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:32:59.092 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:32:59.093 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=12826 2024-04-06T09:32:59.093 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 3 = 2 -a 1 -le 12826 2024-04-06T09:32:59.093 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:32:59.093 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:32:59.093 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:32:59.295 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:32:59.295 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:32:59.295 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:32:59.295 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:32:59.295 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:32:59.295 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:32:59.295 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:32:59.295 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:32:59.295 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:32:59.295 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:32:59.295 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:32:59.295 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:32:59.295 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:32:59.296 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:32:59.296 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:32:59.296 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:32:59.296 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:32:59.298 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:32:59.298 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:32:59.299 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:32:59.299 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:32:59.299 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:32:59.299 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:32:59.299 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:32:59.299 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:32:59.299 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:32:59.299 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:32:59.299 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:32:59.299 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:32:59.299 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:32:59.299 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:32:59.299 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:32:59.299 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:32:59.300 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:32:59.301 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:32:59.301 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:32:59.301 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:32:59.301 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:32:59.301 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:32:59.301 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:32:59.301 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:32:59.301 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:32:59.301 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:32:59.302 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:32:59.302 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:32:59.302 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:32:59.302 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:32:59.302 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:32:59.302 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:32:59.325 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:32:59.325 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:32:59.325 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:32:59.325 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:32:59.457 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:32:59.458 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:32:59.458 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:32:59.458 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:32:59.458 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:32:59.458 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:32:59.458 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:32:59.458 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:32:59.458 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:32:59.459 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:32:59.459 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:32:59.459 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:32:59.459 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:32:59.461 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:32:59.461 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:32:59.461 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:32:59.506 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 3 2024-04-06T09:32:59.506 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 6 2024-04-06T09:32:59.506 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:32:59.506 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=25, tag_tid=3, entry_tid=1], [object_number=24, tag_tid=3, entry_tid=0], [object_number=27, tag_tid=2, entry_tid=27223], [object_number=26, tag_tid=2, entry_tid=27222]]], state=connected] 2024-04-06T09:32:59.506 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=13, tag_tid=2, entry_tid=13037], [object_number=12, tag_tid=2, entry_tid=13036], [object_number=15, tag_tid=2, entry_tid=13035], [object_number=14, tag_tid=2, entry_tid=13034]]], state=connected] 2024-04-06T09:32:59.510 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:32:59.511 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=25, tag_tid=3, entry_tid=1' 2024-04-06T09:32:59.512 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:32:59.512 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:32:59.512 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:32:59.513 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:32:59.513 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:32:59.513 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:32:59.513 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:32:59.513 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:32:59.513 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:32:59.513 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:32:59.513 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:32:59.513 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:32:59.514 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:32:59.514 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:32:59.514 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:32:59.558 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 3 2024-04-06T09:32:59.558 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 6 2024-04-06T09:32:59.558 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:32:59.558 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=25, tag_tid=3, entry_tid=1], [object_number=24, tag_tid=3, entry_tid=0], [object_number=27, tag_tid=2, entry_tid=27223], [object_number=26, tag_tid=2, entry_tid=27222]]], state=connected] 2024-04-06T09:32:59.558 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=13, tag_tid=2, entry_tid=13037], [object_number=12, tag_tid=2, entry_tid=13036], [object_number=15, tag_tid=2, entry_tid=13035], [object_number=14, tag_tid=2, entry_tid=13034]]], state=connected] 2024-04-06T09:32:59.561 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:32:59.563 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=13, tag_tid=2, entry_tid=13037' 2024-04-06T09:32:59.563 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=25, tag_tid=3, entry_tid=1' -a 'object_number=25, tag_tid=3, entry_tid=1' = 'object_number=13, tag_tid=2, entry_tid=13037' 2024-04-06T09:32:59.564 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=13, tag_tid=2, entry_tid=13037' '!=' 'object_number=14, tag_tid=2, entry_tid=12826' 2024-04-06T09:32:59.564 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:32:59.564 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=13, tag_tid=2, entry_tid=13037' = 'object_number=14, tag_tid=2, entry_tid=12826' 2024-04-06T09:32:59.564 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=13, tag_tid=2, entry_tid=13037' 2024-04-06T09:32:59.564 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=25, tag_tid=3, entry_tid=1' 2024-04-06T09:32:59.564 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:32:59.564 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:32:59.564 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=3 2024-04-06T09:32:59.565 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=13, tag_tid=2, entry_tid=13037' 2024-04-06T09:32:59.565 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:32:59.565 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:32:59.566 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=2 2024-04-06T09:32:59.567 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=25, tag_tid=3, entry_tid=1' 2024-04-06T09:32:59.567 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:32:59.567 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:32:59.568 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:32:59.568 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=13, tag_tid=2, entry_tid=13037' 2024-04-06T09:32:59.569 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:32:59.569 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:32:59.570 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=13037 2024-04-06T09:32:59.570 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 3 = 2 -a 1 -le 13037 2024-04-06T09:32:59.570 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:32:59.570 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:32:59.570 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:32:59.771 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:32:59.772 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:32:59.772 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:32:59.772 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:32:59.772 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:32:59.772 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:32:59.772 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:32:59.772 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:32:59.772 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:32:59.772 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:32:59.772 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:32:59.772 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:32:59.772 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:32:59.772 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:32:59.772 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:32:59.773 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:32:59.773 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:32:59.776 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:32:59.776 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:32:59.776 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:32:59.777 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:32:59.777 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:32:59.777 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:32:59.777 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:32:59.777 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:32:59.777 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:32:59.777 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:32:59.777 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:32:59.777 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:32:59.777 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:32:59.777 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:32:59.777 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:32:59.777 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:32:59.777 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:32:59.777 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:32:59.778 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:32:59.778 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:32:59.778 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:32:59.778 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:32:59.778 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:32:59.778 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:32:59.778 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:32:59.778 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:32:59.778 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:32:59.778 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:32:59.778 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:32:59.778 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:32:59.778 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:32:59.778 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:32:59.803 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:32:59.803 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:32:59.803 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:32:59.803 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:32:59.930 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:32:59.931 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:32:59.931 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:32:59.931 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:32:59.931 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:32:59.931 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:32:59.931 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:32:59.932 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:32:59.932 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:32:59.932 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:32:59.932 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:32:59.932 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:32:59.932 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:32:59.933 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:32:59.933 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:32:59.934 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:32:59.984 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 3 2024-04-06T09:32:59.984 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 6 2024-04-06T09:32:59.984 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:32:59.984 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=25, tag_tid=3, entry_tid=1], [object_number=24, tag_tid=3, entry_tid=0], [object_number=27, tag_tid=2, entry_tid=27223], [object_number=26, tag_tid=2, entry_tid=27222]]], state=connected] 2024-04-06T09:32:59.984 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=15, tag_tid=2, entry_tid=13143], [object_number=14, tag_tid=2, entry_tid=13142], [object_number=13, tag_tid=2, entry_tid=13141], [object_number=12, tag_tid=2, entry_tid=13140]]], state=connected] 2024-04-06T09:32:59.988 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:32:59.989 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=25, tag_tid=3, entry_tid=1' 2024-04-06T09:32:59.990 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:32:59.990 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:32:59.990 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:32:59.990 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:32:59.991 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:32:59.991 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:32:59.991 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:32:59.991 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:32:59.991 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:32:59.991 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:32:59.991 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:32:59.991 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:32:59.992 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:32:59.992 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:32:59.992 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:33:00.036 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 3 2024-04-06T09:33:00.037 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 6 2024-04-06T09:33:00.037 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:33:00.037 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=25, tag_tid=3, entry_tid=1], [object_number=24, tag_tid=3, entry_tid=0], [object_number=27, tag_tid=2, entry_tid=27223], [object_number=26, tag_tid=2, entry_tid=27222]]], state=connected] 2024-04-06T09:33:00.037 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=15, tag_tid=2, entry_tid=13143], [object_number=14, tag_tid=2, entry_tid=13142], [object_number=13, tag_tid=2, entry_tid=13141], [object_number=12, tag_tid=2, entry_tid=13140]]], state=connected] 2024-04-06T09:33:00.040 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:33:00.042 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=15, tag_tid=2, entry_tid=13143' 2024-04-06T09:33:00.042 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=25, tag_tid=3, entry_tid=1' -a 'object_number=25, tag_tid=3, entry_tid=1' = 'object_number=15, tag_tid=2, entry_tid=13143' 2024-04-06T09:33:00.042 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=15, tag_tid=2, entry_tid=13143' '!=' 'object_number=13, tag_tid=2, entry_tid=13037' 2024-04-06T09:33:00.042 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:33:00.042 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=15, tag_tid=2, entry_tid=13143' = 'object_number=13, tag_tid=2, entry_tid=13037' 2024-04-06T09:33:00.042 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=15, tag_tid=2, entry_tid=13143' 2024-04-06T09:33:00.042 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=25, tag_tid=3, entry_tid=1' 2024-04-06T09:33:00.042 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:33:00.043 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:33:00.043 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=3 2024-04-06T09:33:00.044 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=15, tag_tid=2, entry_tid=13143' 2024-04-06T09:33:00.044 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:33:00.044 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:33:00.045 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=2 2024-04-06T09:33:00.045 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=25, tag_tid=3, entry_tid=1' 2024-04-06T09:33:00.045 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:33:00.045 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:33:00.046 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:33:00.047 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=15, tag_tid=2, entry_tid=13143' 2024-04-06T09:33:00.047 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:33:00.047 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:33:00.048 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=13143 2024-04-06T09:33:00.049 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 3 = 2 -a 1 -le 13143 2024-04-06T09:33:00.049 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:33:00.049 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:33:00.049 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:33:00.250 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:33:00.250 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:33:00.250 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:33:00.250 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:33:00.250 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:33:00.251 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:33:00.251 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:33:00.251 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:33:00.251 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:33:00.251 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:33:00.251 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:33:00.251 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:33:00.251 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:33:00.251 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:33:00.251 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:33:00.252 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:33:00.252 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:33:00.254 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:33:00.255 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:33:00.255 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:33:00.255 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:33:00.255 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:33:00.255 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:33:00.255 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:33:00.255 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:33:00.255 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:33:00.255 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:33:00.255 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:33:00.255 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:33:00.255 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:33:00.255 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:33:00.255 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:33:00.256 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:33:00.256 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:33:00.256 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:33:00.256 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:33:00.256 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:33:00.256 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:33:00.256 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:33:00.256 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:33:00.256 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:33:00.256 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:33:00.256 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:33:00.256 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:33:00.256 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:33:00.256 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:33:00.257 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:33:00.257 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:33:00.257 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:33:00.281 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:33:00.281 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:33:00.281 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:33:00.282 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:33:00.410 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:33:00.411 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:33:00.411 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:33:00.412 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:33:00.412 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:33:00.412 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:33:00.412 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:33:00.412 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:33:00.412 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:33:00.412 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:33:00.412 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:33:00.413 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:33:00.413 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:33:00.413 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:33:00.414 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:33:00.414 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:33:00.467 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 3 2024-04-06T09:33:00.467 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 6 2024-04-06T09:33:00.467 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:33:00.467 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=25, tag_tid=3, entry_tid=1], [object_number=24, tag_tid=3, entry_tid=0], [object_number=27, tag_tid=2, entry_tid=27223], [object_number=26, tag_tid=2, entry_tid=27222]]], state=connected] 2024-04-06T09:33:00.467 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=15, tag_tid=2, entry_tid=13247], [object_number=14, tag_tid=2, entry_tid=13246], [object_number=13, tag_tid=2, entry_tid=13245], [object_number=12, tag_tid=2, entry_tid=13244]]], state=connected] 2024-04-06T09:33:00.471 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:33:00.472 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=25, tag_tid=3, entry_tid=1' 2024-04-06T09:33:00.474 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:33:00.474 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:33:00.474 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:33:00.474 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:33:00.474 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:33:00.474 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:33:00.474 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:33:00.474 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:33:00.474 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:33:00.474 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:33:00.474 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:33:00.474 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:33:00.478 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:33:00.478 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:33:00.478 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:33:00.521 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 3 2024-04-06T09:33:00.521 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 6 2024-04-06T09:33:00.521 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:33:00.521 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=25, tag_tid=3, entry_tid=1], [object_number=24, tag_tid=3, entry_tid=0], [object_number=27, tag_tid=2, entry_tid=27223], [object_number=26, tag_tid=2, entry_tid=27222]]], state=connected] 2024-04-06T09:33:00.521 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=15, tag_tid=2, entry_tid=13247], [object_number=14, tag_tid=2, entry_tid=13246], [object_number=13, tag_tid=2, entry_tid=13245], [object_number=12, tag_tid=2, entry_tid=13244]]], state=connected] 2024-04-06T09:33:00.524 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:33:00.527 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=15, tag_tid=2, entry_tid=13247' 2024-04-06T09:33:00.527 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=25, tag_tid=3, entry_tid=1' -a 'object_number=25, tag_tid=3, entry_tid=1' = 'object_number=15, tag_tid=2, entry_tid=13247' 2024-04-06T09:33:00.527 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=15, tag_tid=2, entry_tid=13247' '!=' 'object_number=15, tag_tid=2, entry_tid=13143' 2024-04-06T09:33:00.527 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:33:00.527 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=15, tag_tid=2, entry_tid=13247' = 'object_number=15, tag_tid=2, entry_tid=13143' 2024-04-06T09:33:00.527 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=15, tag_tid=2, entry_tid=13247' 2024-04-06T09:33:00.528 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=25, tag_tid=3, entry_tid=1' 2024-04-06T09:33:00.528 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:33:00.528 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:33:00.528 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=3 2024-04-06T09:33:00.529 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=15, tag_tid=2, entry_tid=13247' 2024-04-06T09:33:00.529 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:33:00.529 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:33:00.529 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=2 2024-04-06T09:33:00.530 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=25, tag_tid=3, entry_tid=1' 2024-04-06T09:33:00.530 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:33:00.530 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:33:00.531 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:33:00.532 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=15, tag_tid=2, entry_tid=13247' 2024-04-06T09:33:00.532 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:33:00.532 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:33:00.533 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=13247 2024-04-06T09:33:00.533 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 3 = 2 -a 1 -le 13247 2024-04-06T09:33:00.533 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:33:00.533 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:33:00.533 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:33:00.734 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:33:00.735 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:33:00.735 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:33:00.735 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:33:00.735 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:33:00.735 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:33:00.735 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:33:00.735 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:33:00.735 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:33:00.735 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:33:00.735 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:33:00.735 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:33:00.735 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:33:00.735 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:33:00.736 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:33:00.736 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:33:00.736 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:33:00.739 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:33:00.739 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:33:00.739 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:33:00.739 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:33:00.739 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:33:00.739 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:33:00.739 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:33:00.739 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:33:00.739 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:33:00.739 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:33:00.739 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:33:00.739 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:33:00.739 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:33:00.739 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:33:00.740 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:33:00.740 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:33:00.741 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:33:00.741 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:33:00.742 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:33:00.742 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:33:00.742 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:33:00.742 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:33:00.742 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:33:00.742 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:33:00.742 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:33:00.742 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:33:00.742 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:33:00.742 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:33:00.743 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:33:00.743 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:33:00.743 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:33:00.743 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:33:00.766 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:33:00.766 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:33:00.766 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:33:00.766 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:33:00.894 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:33:00.895 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:33:00.895 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:33:00.895 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:33:00.895 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:33:00.895 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:33:00.895 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:33:00.895 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:33:00.895 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:33:00.896 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:33:00.896 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:33:00.896 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:33:00.896 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:33:00.897 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:33:00.897 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:33:00.897 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:33:00.951 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 3 2024-04-06T09:33:00.952 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 6 2024-04-06T09:33:00.952 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:33:00.952 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=25, tag_tid=3, entry_tid=1], [object_number=24, tag_tid=3, entry_tid=0], [object_number=27, tag_tid=2, entry_tid=27223], [object_number=26, tag_tid=2, entry_tid=27222]]], state=connected] 2024-04-06T09:33:00.952 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=13, tag_tid=2, entry_tid=13361], [object_number=12, tag_tid=2, entry_tid=13360], [object_number=15, tag_tid=2, entry_tid=13359], [object_number=14, tag_tid=2, entry_tid=13358]]], state=connected] 2024-04-06T09:33:00.955 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:33:00.957 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=25, tag_tid=3, entry_tid=1' 2024-04-06T09:33:00.958 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:33:00.958 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:33:00.958 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:33:00.958 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:33:00.958 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:33:00.958 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:33:00.958 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:33:00.958 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:33:00.958 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:33:00.958 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:33:00.958 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:33:00.958 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:33:00.959 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:33:00.959 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:33:00.959 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:33:01.004 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 3 2024-04-06T09:33:01.005 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 6 2024-04-06T09:33:01.005 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:33:01.005 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=25, tag_tid=3, entry_tid=1], [object_number=24, tag_tid=3, entry_tid=0], [object_number=27, tag_tid=2, entry_tid=27223], [object_number=26, tag_tid=2, entry_tid=27222]]], state=connected] 2024-04-06T09:33:01.005 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=13, tag_tid=2, entry_tid=13361], [object_number=12, tag_tid=2, entry_tid=13360], [object_number=15, tag_tid=2, entry_tid=13359], [object_number=14, tag_tid=2, entry_tid=13358]]], state=connected] 2024-04-06T09:33:01.008 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:33:01.010 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=13, tag_tid=2, entry_tid=13361' 2024-04-06T09:33:01.010 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=25, tag_tid=3, entry_tid=1' -a 'object_number=25, tag_tid=3, entry_tid=1' = 'object_number=13, tag_tid=2, entry_tid=13361' 2024-04-06T09:33:01.010 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=13, tag_tid=2, entry_tid=13361' '!=' 'object_number=15, tag_tid=2, entry_tid=13247' 2024-04-06T09:33:01.011 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:33:01.011 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=13, tag_tid=2, entry_tid=13361' = 'object_number=15, tag_tid=2, entry_tid=13247' 2024-04-06T09:33:01.011 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=13, tag_tid=2, entry_tid=13361' 2024-04-06T09:33:01.011 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=25, tag_tid=3, entry_tid=1' 2024-04-06T09:33:01.011 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:33:01.011 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:33:01.012 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=3 2024-04-06T09:33:01.012 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=13, tag_tid=2, entry_tid=13361' 2024-04-06T09:33:01.012 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:33:01.012 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:33:01.014 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=2 2024-04-06T09:33:01.014 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=25, tag_tid=3, entry_tid=1' 2024-04-06T09:33:01.014 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:33:01.014 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:33:01.016 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:33:01.016 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=13, tag_tid=2, entry_tid=13361' 2024-04-06T09:33:01.016 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:33:01.016 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:33:01.017 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=13361 2024-04-06T09:33:01.017 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 3 = 2 -a 1 -le 13361 2024-04-06T09:33:01.017 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:33:01.017 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:33:01.017 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:33:01.219 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:33:01.219 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:33:01.220 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:33:01.220 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:33:01.220 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:33:01.220 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:33:01.220 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:33:01.220 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:33:01.220 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:33:01.220 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:33:01.220 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:33:01.220 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:33:01.220 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:33:01.220 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:33:01.220 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:33:01.220 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:33:01.221 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:33:01.224 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:33:01.224 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:33:01.224 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:33:01.224 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:33:01.224 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:33:01.224 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:33:01.224 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:33:01.225 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:33:01.225 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:33:01.225 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:33:01.225 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:33:01.225 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:33:01.225 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:33:01.225 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:33:01.225 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:33:01.225 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:33:01.225 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:33:01.225 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:33:01.225 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:33:01.225 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:33:01.225 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:33:01.225 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:33:01.226 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:33:01.226 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:33:01.226 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:33:01.226 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:33:01.226 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:33:01.226 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:33:01.226 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:33:01.226 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:33:01.226 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:33:01.226 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:33:01.249 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:33:01.250 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:33:01.250 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:33:01.250 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:33:01.393 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:33:01.395 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:33:01.395 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:33:01.395 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:33:01.395 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:33:01.395 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:33:01.395 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:33:01.395 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:33:01.395 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:33:01.395 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:33:01.395 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:33:01.395 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:33:01.395 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:33:01.397 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:33:01.397 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:33:01.397 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:33:01.444 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 3 2024-04-06T09:33:01.444 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 6 2024-04-06T09:33:01.444 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:33:01.444 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=25, tag_tid=3, entry_tid=1], [object_number=24, tag_tid=3, entry_tid=0], [object_number=27, tag_tid=2, entry_tid=27223], [object_number=26, tag_tid=2, entry_tid=27222]]], state=connected] 2024-04-06T09:33:01.444 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=14, tag_tid=2, entry_tid=13470], [object_number=13, tag_tid=2, entry_tid=13469], [object_number=12, tag_tid=2, entry_tid=13468], [object_number=15, tag_tid=2, entry_tid=13467]]], state=connected] 2024-04-06T09:33:01.447 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:33:01.448 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=25, tag_tid=3, entry_tid=1' 2024-04-06T09:33:01.449 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:33:01.450 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:33:01.450 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:33:01.450 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:33:01.450 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:33:01.450 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:33:01.450 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:33:01.450 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:33:01.450 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:33:01.450 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:33:01.450 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:33:01.450 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:33:01.451 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:33:01.451 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:33:01.451 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:33:01.495 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 3 2024-04-06T09:33:01.496 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 6 2024-04-06T09:33:01.496 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:33:01.496 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=25, tag_tid=3, entry_tid=1], [object_number=24, tag_tid=3, entry_tid=0], [object_number=27, tag_tid=2, entry_tid=27223], [object_number=26, tag_tid=2, entry_tid=27222]]], state=connected] 2024-04-06T09:33:01.496 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=14, tag_tid=2, entry_tid=13470], [object_number=13, tag_tid=2, entry_tid=13469], [object_number=12, tag_tid=2, entry_tid=13468], [object_number=15, tag_tid=2, entry_tid=13467]]], state=connected] 2024-04-06T09:33:01.499 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:33:01.501 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=14, tag_tid=2, entry_tid=13470' 2024-04-06T09:33:01.501 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=25, tag_tid=3, entry_tid=1' -a 'object_number=25, tag_tid=3, entry_tid=1' = 'object_number=14, tag_tid=2, entry_tid=13470' 2024-04-06T09:33:01.502 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=14, tag_tid=2, entry_tid=13470' '!=' 'object_number=13, tag_tid=2, entry_tid=13361' 2024-04-06T09:33:01.502 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:33:01.502 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=14, tag_tid=2, entry_tid=13470' = 'object_number=13, tag_tid=2, entry_tid=13361' 2024-04-06T09:33:01.502 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=14, tag_tid=2, entry_tid=13470' 2024-04-06T09:33:01.502 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=25, tag_tid=3, entry_tid=1' 2024-04-06T09:33:01.502 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:33:01.502 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:33:01.502 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=3 2024-04-06T09:33:01.503 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=14, tag_tid=2, entry_tid=13470' 2024-04-06T09:33:01.503 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:33:01.503 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:33:01.504 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=2 2024-04-06T09:33:01.505 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=25, tag_tid=3, entry_tid=1' 2024-04-06T09:33:01.505 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:33:01.505 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:33:01.506 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:33:01.506 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=14, tag_tid=2, entry_tid=13470' 2024-04-06T09:33:01.507 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:33:01.507 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:33:01.508 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=13470 2024-04-06T09:33:01.508 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 3 = 2 -a 1 -le 13470 2024-04-06T09:33:01.508 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:33:01.508 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:33:01.508 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:33:01.709 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:33:01.710 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:33:01.710 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:33:01.710 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:33:01.710 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:33:01.710 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:33:01.710 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:33:01.710 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:33:01.710 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:33:01.710 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:33:01.710 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:33:01.710 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:33:01.710 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:33:01.710 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:33:01.711 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:33:01.711 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:33:01.711 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:33:01.714 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:33:01.714 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:33:01.714 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:33:01.714 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:33:01.714 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:33:01.714 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:33:01.714 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:33:01.714 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:33:01.714 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:33:01.714 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:33:01.714 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:33:01.714 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:33:01.714 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:33:01.715 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:33:01.715 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:33:01.715 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:33:01.715 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:33:01.715 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:33:01.715 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:33:01.715 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:33:01.715 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:33:01.715 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:33:01.715 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:33:01.715 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:33:01.715 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:33:01.715 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:33:01.715 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:33:01.715 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:33:01.715 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:33:01.716 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:33:01.716 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:33:01.716 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:33:01.741 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:33:01.741 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:33:01.741 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:33:01.741 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:33:01.875 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:33:01.875 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:33:01.875 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:33:01.876 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:33:01.876 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:33:01.876 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:33:01.876 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:33:01.876 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:33:01.876 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:33:01.876 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:33:01.876 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:33:01.877 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:33:01.877 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:33:01.877 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:33:01.878 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:33:01.878 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:33:02.137 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 3 2024-04-06T09:33:02.137 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 6 2024-04-06T09:33:02.137 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:33:02.137 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=25, tag_tid=3, entry_tid=1], [object_number=24, tag_tid=3, entry_tid=0], [object_number=27, tag_tid=2, entry_tid=27223], [object_number=26, tag_tid=2, entry_tid=27222]]], state=connected] 2024-04-06T09:33:02.137 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=15, tag_tid=2, entry_tid=13527], [object_number=14, tag_tid=2, entry_tid=13526], [object_number=13, tag_tid=2, entry_tid=13525], [object_number=12, tag_tid=2, entry_tid=13524]]], state=connected] 2024-04-06T09:33:02.141 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:33:02.143 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=25, tag_tid=3, entry_tid=1' 2024-04-06T09:33:02.144 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:33:02.144 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:33:02.144 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:33:02.144 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:33:02.144 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:33:02.144 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:33:02.145 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:33:02.145 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:33:02.145 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:33:02.145 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:33:02.145 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:33:02.145 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:33:02.146 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:33:02.146 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:33:02.146 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:33:02.196 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 3 2024-04-06T09:33:02.197 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 6 2024-04-06T09:33:02.197 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:33:02.197 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=25, tag_tid=3, entry_tid=1], [object_number=24, tag_tid=3, entry_tid=0], [object_number=27, tag_tid=2, entry_tid=27223], [object_number=26, tag_tid=2, entry_tid=27222]]], state=connected] 2024-04-06T09:33:02.197 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=15, tag_tid=2, entry_tid=13527], [object_number=14, tag_tid=2, entry_tid=13526], [object_number=13, tag_tid=2, entry_tid=13525], [object_number=12, tag_tid=2, entry_tid=13524]]], state=connected] 2024-04-06T09:33:02.199 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:33:02.201 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=15, tag_tid=2, entry_tid=13527' 2024-04-06T09:33:02.201 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=25, tag_tid=3, entry_tid=1' -a 'object_number=25, tag_tid=3, entry_tid=1' = 'object_number=15, tag_tid=2, entry_tid=13527' 2024-04-06T09:33:02.201 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=15, tag_tid=2, entry_tid=13527' '!=' 'object_number=14, tag_tid=2, entry_tid=13470' 2024-04-06T09:33:02.202 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:33:02.202 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=15, tag_tid=2, entry_tid=13527' = 'object_number=14, tag_tid=2, entry_tid=13470' 2024-04-06T09:33:02.202 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=15, tag_tid=2, entry_tid=13527' 2024-04-06T09:33:02.202 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=25, tag_tid=3, entry_tid=1' 2024-04-06T09:33:02.202 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:33:02.202 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:33:02.203 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=3 2024-04-06T09:33:02.204 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=15, tag_tid=2, entry_tid=13527' 2024-04-06T09:33:02.204 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:33:02.204 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:33:02.205 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=2 2024-04-06T09:33:02.206 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=25, tag_tid=3, entry_tid=1' 2024-04-06T09:33:02.206 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:33:02.206 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:33:02.207 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:33:02.208 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=15, tag_tid=2, entry_tid=13527' 2024-04-06T09:33:02.208 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:33:02.208 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:33:02.209 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=13527 2024-04-06T09:33:02.209 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 3 = 2 -a 1 -le 13527 2024-04-06T09:33:02.209 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:33:02.209 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:33:02.209 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:33:02.411 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:33:02.411 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:33:02.411 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:33:02.411 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:33:02.412 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:33:02.412 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:33:02.412 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:33:02.412 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:33:02.412 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:33:02.412 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:33:02.412 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:33:02.412 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:33:02.412 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:33:02.412 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:33:02.412 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:33:02.413 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:33:02.413 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:33:02.414 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:33:02.414 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:33:02.414 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:33:02.414 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:33:02.414 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:33:02.414 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:33:02.414 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:33:02.414 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:33:02.415 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:33:02.415 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:33:02.415 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:33:02.415 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:33:02.415 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:33:02.415 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:33:02.415 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:33:02.415 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:33:02.416 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:33:02.416 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:33:02.416 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:33:02.417 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:33:02.417 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:33:02.417 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:33:02.417 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:33:02.417 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:33:02.417 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:33:02.417 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:33:02.417 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:33:02.417 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:33:02.417 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:33:02.417 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:33:02.417 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:33:02.417 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:33:02.443 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:33:02.443 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:33:02.443 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:33:02.443 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:33:02.574 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:33:02.575 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:33:02.575 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:33:02.576 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:33:02.576 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:33:02.576 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:33:02.576 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:33:02.576 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:33:02.576 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:33:02.576 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:33:02.576 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:33:02.577 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:33:02.577 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:33:02.577 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:33:02.578 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:33:02.578 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:33:02.631 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 3 2024-04-06T09:33:02.632 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 6 2024-04-06T09:33:02.632 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:33:02.632 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=25, tag_tid=3, entry_tid=1], [object_number=24, tag_tid=3, entry_tid=0], [object_number=27, tag_tid=2, entry_tid=27223], [object_number=26, tag_tid=2, entry_tid=27222]]], state=connected] 2024-04-06T09:33:02.632 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=13, tag_tid=2, entry_tid=13677], [object_number=12, tag_tid=2, entry_tid=13676], [object_number=15, tag_tid=2, entry_tid=13675], [object_number=14, tag_tid=2, entry_tid=13674]]], state=connected] 2024-04-06T09:33:02.635 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:33:02.637 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=25, tag_tid=3, entry_tid=1' 2024-04-06T09:33:02.638 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:33:02.638 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:33:02.638 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:33:02.638 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:33:02.638 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:33:02.638 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:33:02.638 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:33:02.638 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:33:02.638 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:33:02.638 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:33:02.639 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:33:02.639 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:33:02.639 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:33:02.639 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:33:02.639 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:33:02.685 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 3 2024-04-06T09:33:02.685 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 6 2024-04-06T09:33:02.686 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:33:02.686 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=25, tag_tid=3, entry_tid=1], [object_number=24, tag_tid=3, entry_tid=0], [object_number=27, tag_tid=2, entry_tid=27223], [object_number=26, tag_tid=2, entry_tid=27222]]], state=connected] 2024-04-06T09:33:02.686 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=13, tag_tid=2, entry_tid=13677], [object_number=12, tag_tid=2, entry_tid=13676], [object_number=15, tag_tid=2, entry_tid=13675], [object_number=14, tag_tid=2, entry_tid=13674]]], state=connected] 2024-04-06T09:33:02.688 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:33:02.690 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=13, tag_tid=2, entry_tid=13677' 2024-04-06T09:33:02.690 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=25, tag_tid=3, entry_tid=1' -a 'object_number=25, tag_tid=3, entry_tid=1' = 'object_number=13, tag_tid=2, entry_tid=13677' 2024-04-06T09:33:02.690 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=13, tag_tid=2, entry_tid=13677' '!=' 'object_number=15, tag_tid=2, entry_tid=13527' 2024-04-06T09:33:02.690 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:33:02.690 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=13, tag_tid=2, entry_tid=13677' = 'object_number=15, tag_tid=2, entry_tid=13527' 2024-04-06T09:33:02.690 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=13, tag_tid=2, entry_tid=13677' 2024-04-06T09:33:02.690 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=25, tag_tid=3, entry_tid=1' 2024-04-06T09:33:02.690 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:33:02.691 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:33:02.691 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=3 2024-04-06T09:33:02.692 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=13, tag_tid=2, entry_tid=13677' 2024-04-06T09:33:02.692 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:33:02.692 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:33:02.693 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=2 2024-04-06T09:33:02.693 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=25, tag_tid=3, entry_tid=1' 2024-04-06T09:33:02.693 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:33:02.694 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:33:02.694 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:33:02.695 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=13, tag_tid=2, entry_tid=13677' 2024-04-06T09:33:02.695 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:33:02.695 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:33:02.696 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=13677 2024-04-06T09:33:02.696 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 3 = 2 -a 1 -le 13677 2024-04-06T09:33:02.697 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:33:02.697 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:33:02.697 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:33:02.898 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:33:02.898 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:33:02.899 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:33:02.899 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:33:02.899 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:33:02.899 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:33:02.899 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:33:02.899 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:33:02.899 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:33:02.899 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:33:02.899 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:33:02.899 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:33:02.899 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:33:02.899 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:33:02.899 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:33:02.900 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:33:02.900 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:33:02.904 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:33:02.904 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:33:02.904 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:33:02.904 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:33:02.904 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:33:02.904 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:33:02.904 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:33:02.904 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:33:02.904 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:33:02.904 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:33:02.904 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:33:02.905 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:33:02.905 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:33:02.905 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:33:02.905 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:33:02.905 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:33:02.905 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:33:02.905 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:33:02.905 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:33:02.905 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:33:02.905 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:33:02.905 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:33:02.905 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:33:02.905 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:33:02.905 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:33:02.905 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:33:02.906 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:33:02.906 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:33:02.906 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:33:02.906 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:33:02.906 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:33:02.906 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:33:02.928 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:33:02.929 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:33:02.929 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:33:02.929 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:33:03.058 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:33:03.059 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:33:03.059 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:33:03.059 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:33:03.059 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:33:03.060 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:33:03.060 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:33:03.060 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:33:03.060 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:33:03.060 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:33:03.060 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:33:03.061 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:33:03.061 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:33:03.063 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:33:03.063 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:33:03.063 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:33:03.117 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 3 2024-04-06T09:33:03.117 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 6 2024-04-06T09:33:03.117 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:33:03.117 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=25, tag_tid=3, entry_tid=1], [object_number=24, tag_tid=3, entry_tid=0], [object_number=27, tag_tid=2, entry_tid=27223], [object_number=26, tag_tid=2, entry_tid=27222]]], state=connected] 2024-04-06T09:33:03.117 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=12, tag_tid=2, entry_tid=13792], [object_number=15, tag_tid=2, entry_tid=13791], [object_number=14, tag_tid=2, entry_tid=13790], [object_number=13, tag_tid=2, entry_tid=13789]]], state=connected] 2024-04-06T09:33:03.120 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:33:03.122 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=25, tag_tid=3, entry_tid=1' 2024-04-06T09:33:03.123 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:33:03.123 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:33:03.123 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:33:03.123 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:33:03.123 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:33:03.123 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:33:03.124 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:33:03.124 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:33:03.124 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:33:03.124 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:33:03.124 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:33:03.124 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:33:03.124 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:33:03.125 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:33:03.125 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:33:03.170 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 3 2024-04-06T09:33:03.170 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 6 2024-04-06T09:33:03.170 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:33:03.170 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=25, tag_tid=3, entry_tid=1], [object_number=24, tag_tid=3, entry_tid=0], [object_number=27, tag_tid=2, entry_tid=27223], [object_number=26, tag_tid=2, entry_tid=27222]]], state=connected] 2024-04-06T09:33:03.171 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=12, tag_tid=2, entry_tid=13792], [object_number=15, tag_tid=2, entry_tid=13791], [object_number=14, tag_tid=2, entry_tid=13790], [object_number=13, tag_tid=2, entry_tid=13789]]], state=connected] 2024-04-06T09:33:03.174 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:33:03.176 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=12, tag_tid=2, entry_tid=13792' 2024-04-06T09:33:03.176 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=25, tag_tid=3, entry_tid=1' -a 'object_number=25, tag_tid=3, entry_tid=1' = 'object_number=12, tag_tid=2, entry_tid=13792' 2024-04-06T09:33:03.176 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=12, tag_tid=2, entry_tid=13792' '!=' 'object_number=13, tag_tid=2, entry_tid=13677' 2024-04-06T09:33:03.176 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:33:03.176 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=12, tag_tid=2, entry_tid=13792' = 'object_number=13, tag_tid=2, entry_tid=13677' 2024-04-06T09:33:03.176 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=12, tag_tid=2, entry_tid=13792' 2024-04-06T09:33:03.176 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=25, tag_tid=3, entry_tid=1' 2024-04-06T09:33:03.176 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:33:03.176 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:33:03.177 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=3 2024-04-06T09:33:03.178 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=12, tag_tid=2, entry_tid=13792' 2024-04-06T09:33:03.178 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:33:03.178 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:33:03.179 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=2 2024-04-06T09:33:03.179 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=25, tag_tid=3, entry_tid=1' 2024-04-06T09:33:03.180 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:33:03.180 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:33:03.181 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:33:03.181 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=12, tag_tid=2, entry_tid=13792' 2024-04-06T09:33:03.181 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:33:03.181 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:33:03.183 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=13792 2024-04-06T09:33:03.183 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 3 = 2 -a 1 -le 13792 2024-04-06T09:33:03.183 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:33:03.183 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:33:03.183 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:33:03.385 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:33:03.385 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:33:03.385 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:33:03.385 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:33:03.385 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:33:03.385 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:33:03.385 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:33:03.385 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:33:03.385 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:33:03.385 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:33:03.386 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:33:03.386 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:33:03.386 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:33:03.386 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:33:03.386 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:33:03.386 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:33:03.386 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:33:03.389 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:33:03.389 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:33:03.389 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:33:03.390 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:33:03.390 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:33:03.390 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:33:03.390 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:33:03.390 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:33:03.390 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:33:03.390 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:33:03.390 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:33:03.390 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:33:03.390 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:33:03.390 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:33:03.390 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:33:03.390 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:33:03.391 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:33:03.391 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:33:03.391 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:33:03.391 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:33:03.391 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:33:03.391 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:33:03.391 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:33:03.391 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:33:03.391 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:33:03.391 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:33:03.391 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:33:03.391 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:33:03.391 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:33:03.392 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:33:03.392 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:33:03.392 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:33:03.415 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:33:03.416 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:33:03.416 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:33:03.416 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:33:03.547 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:33:03.548 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:33:03.548 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:33:03.548 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:33:03.548 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:33:03.548 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:33:03.548 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:33:03.548 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:33:03.548 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:33:03.548 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:33:03.549 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:33:03.549 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:33:03.549 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:33:03.550 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:33:03.550 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:33:03.551 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:33:03.594 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 3 2024-04-06T09:33:03.594 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 6 2024-04-06T09:33:03.594 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:33:03.595 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=25, tag_tid=3, entry_tid=1], [object_number=24, tag_tid=3, entry_tid=0], [object_number=27, tag_tid=2, entry_tid=27223], [object_number=26, tag_tid=2, entry_tid=27222]]], state=connected] 2024-04-06T09:33:03.595 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=15, tag_tid=2, entry_tid=13899], [object_number=14, tag_tid=2, entry_tid=13898], [object_number=13, tag_tid=2, entry_tid=13897], [object_number=12, tag_tid=2, entry_tid=13896]]], state=connected] 2024-04-06T09:33:03.598 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:33:03.599 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=25, tag_tid=3, entry_tid=1' 2024-04-06T09:33:03.600 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:33:03.601 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:33:03.601 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:33:03.601 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:33:03.601 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:33:03.601 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:33:03.601 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:33:03.601 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:33:03.601 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:33:03.601 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:33:03.601 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:33:03.601 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:33:03.602 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:33:03.602 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:33:03.602 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:33:03.647 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 3 2024-04-06T09:33:03.647 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 6 2024-04-06T09:33:03.647 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:33:03.647 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=25, tag_tid=3, entry_tid=1], [object_number=24, tag_tid=3, entry_tid=0], [object_number=27, tag_tid=2, entry_tid=27223], [object_number=26, tag_tid=2, entry_tid=27222]]], state=connected] 2024-04-06T09:33:03.647 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=15, tag_tid=2, entry_tid=13899], [object_number=14, tag_tid=2, entry_tid=13898], [object_number=13, tag_tid=2, entry_tid=13897], [object_number=12, tag_tid=2, entry_tid=13896]]], state=connected] 2024-04-06T09:33:03.650 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:33:03.652 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=15, tag_tid=2, entry_tid=13899' 2024-04-06T09:33:03.653 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=25, tag_tid=3, entry_tid=1' -a 'object_number=25, tag_tid=3, entry_tid=1' = 'object_number=15, tag_tid=2, entry_tid=13899' 2024-04-06T09:33:03.653 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=15, tag_tid=2, entry_tid=13899' '!=' 'object_number=12, tag_tid=2, entry_tid=13792' 2024-04-06T09:33:03.653 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:33:03.653 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=15, tag_tid=2, entry_tid=13899' = 'object_number=12, tag_tid=2, entry_tid=13792' 2024-04-06T09:33:03.653 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=15, tag_tid=2, entry_tid=13899' 2024-04-06T09:33:03.653 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=25, tag_tid=3, entry_tid=1' 2024-04-06T09:33:03.653 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:33:03.653 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:33:03.653 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=3 2024-04-06T09:33:03.654 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=15, tag_tid=2, entry_tid=13899' 2024-04-06T09:33:03.654 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:33:03.654 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:33:03.655 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=2 2024-04-06T09:33:03.656 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=25, tag_tid=3, entry_tid=1' 2024-04-06T09:33:03.656 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:33:03.656 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:33:03.657 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:33:03.658 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=15, tag_tid=2, entry_tid=13899' 2024-04-06T09:33:03.658 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:33:03.658 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:33:03.659 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=13899 2024-04-06T09:33:03.659 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 3 = 2 -a 1 -le 13899 2024-04-06T09:33:03.659 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:33:03.659 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:33:03.659 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:33:03.861 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:33:03.861 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:33:03.861 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:33:03.861 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:33:03.861 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:33:03.861 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:33:03.861 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:33:03.861 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:33:03.861 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:33:03.862 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:33:03.862 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:33:03.862 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:33:03.862 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:33:03.862 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:33:03.862 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:33:03.863 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:33:03.863 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:33:03.865 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:33:03.865 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:33:03.865 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:33:03.865 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:33:03.865 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:33:03.866 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:33:03.866 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:33:03.866 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:33:03.866 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:33:03.866 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:33:03.866 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:33:03.866 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:33:03.866 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:33:03.866 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:33:03.866 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:33:03.866 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:33:03.867 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:33:03.867 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:33:03.867 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:33:03.867 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:33:03.867 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:33:03.867 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:33:03.867 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:33:03.867 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:33:03.867 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:33:03.867 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:33:03.867 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:33:03.868 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:33:03.868 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:33:03.868 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:33:03.868 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:33:03.868 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:33:03.892 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:33:03.892 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:33:03.893 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:33:03.893 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:33:04.021 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:33:04.021 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:33:04.021 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:33:04.022 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:33:04.022 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:33:04.022 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:33:04.022 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:33:04.022 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:33:04.022 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:33:04.022 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:33:04.023 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:33:04.023 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:33:04.023 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:33:04.024 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:33:04.024 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:33:04.024 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:33:04.069 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 3 2024-04-06T09:33:04.069 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 6 2024-04-06T09:33:04.069 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:33:04.069 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=25, tag_tid=3, entry_tid=1], [object_number=24, tag_tid=3, entry_tid=0], [object_number=27, tag_tid=2, entry_tid=27223], [object_number=26, tag_tid=2, entry_tid=27222]]], state=connected] 2024-04-06T09:33:04.069 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=14, tag_tid=2, entry_tid=14014], [object_number=13, tag_tid=2, entry_tid=14013], [object_number=12, tag_tid=2, entry_tid=14012], [object_number=15, tag_tid=2, entry_tid=14011]]], state=connected] 2024-04-06T09:33:04.073 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:33:04.074 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=25, tag_tid=3, entry_tid=1' 2024-04-06T09:33:04.075 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:33:04.075 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:33:04.076 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:33:04.076 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:33:04.076 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:33:04.076 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:33:04.076 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:33:04.076 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:33:04.076 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:33:04.076 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:33:04.076 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:33:04.076 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:33:04.077 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:33:04.077 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:33:04.077 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:33:04.121 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 3 2024-04-06T09:33:04.121 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 6 2024-04-06T09:33:04.121 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:33:04.121 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=25, tag_tid=3, entry_tid=1], [object_number=24, tag_tid=3, entry_tid=0], [object_number=27, tag_tid=2, entry_tid=27223], [object_number=26, tag_tid=2, entry_tid=27222]]], state=connected] 2024-04-06T09:33:04.122 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=14, tag_tid=2, entry_tid=14014], [object_number=13, tag_tid=2, entry_tid=14013], [object_number=12, tag_tid=2, entry_tid=14012], [object_number=15, tag_tid=2, entry_tid=14011]]], state=connected] 2024-04-06T09:33:04.125 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:33:04.127 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=14, tag_tid=2, entry_tid=14014' 2024-04-06T09:33:04.127 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=25, tag_tid=3, entry_tid=1' -a 'object_number=25, tag_tid=3, entry_tid=1' = 'object_number=14, tag_tid=2, entry_tid=14014' 2024-04-06T09:33:04.127 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=14, tag_tid=2, entry_tid=14014' '!=' 'object_number=15, tag_tid=2, entry_tid=13899' 2024-04-06T09:33:04.127 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:33:04.127 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=14, tag_tid=2, entry_tid=14014' = 'object_number=15, tag_tid=2, entry_tid=13899' 2024-04-06T09:33:04.128 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=14, tag_tid=2, entry_tid=14014' 2024-04-06T09:33:04.128 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=25, tag_tid=3, entry_tid=1' 2024-04-06T09:33:04.128 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:33:04.128 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:33:04.128 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=3 2024-04-06T09:33:04.129 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=14, tag_tid=2, entry_tid=14014' 2024-04-06T09:33:04.129 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:33:04.129 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:33:04.130 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=2 2024-04-06T09:33:04.130 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=25, tag_tid=3, entry_tid=1' 2024-04-06T09:33:04.130 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:33:04.131 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:33:04.131 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:33:04.132 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=14, tag_tid=2, entry_tid=14014' 2024-04-06T09:33:04.132 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:33:04.132 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:33:04.133 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=14014 2024-04-06T09:33:04.133 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 3 = 2 -a 1 -le 14014 2024-04-06T09:33:04.134 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:33:04.134 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:33:04.134 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:33:04.335 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:33:04.335 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:33:04.335 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:33:04.335 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:33:04.335 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:33:04.335 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:33:04.335 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:33:04.336 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:33:04.336 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:33:04.336 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:33:04.336 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:33:04.336 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:33:04.336 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:33:04.336 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:33:04.336 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:33:04.337 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:33:04.337 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:33:04.339 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:33:04.339 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:33:04.339 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:33:04.339 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:33:04.339 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:33:04.339 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:33:04.339 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:33:04.339 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:33:04.339 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:33:04.339 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:33:04.339 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:33:04.339 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:33:04.339 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:33:04.340 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:33:04.340 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:33:04.340 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:33:04.340 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:33:04.340 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:33:04.340 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:33:04.340 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:33:04.340 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:33:04.340 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:33:04.340 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:33:04.340 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:33:04.340 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:33:04.340 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:33:04.340 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:33:04.340 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:33:04.341 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:33:04.341 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:33:04.341 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:33:04.341 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:33:04.367 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:33:04.367 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:33:04.367 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:33:04.367 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:33:04.497 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:33:04.498 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:33:04.499 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:33:04.499 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:33:04.499 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:33:04.499 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:33:04.499 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:33:04.499 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:33:04.499 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:33:04.499 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:33:04.499 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:33:04.500 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:33:04.500 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:33:04.501 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:33:04.501 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:33:04.501 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:33:04.545 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 3 2024-04-06T09:33:04.545 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 6 2024-04-06T09:33:04.545 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:33:04.545 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=25, tag_tid=3, entry_tid=1], [object_number=24, tag_tid=3, entry_tid=0], [object_number=27, tag_tid=2, entry_tid=27223], [object_number=26, tag_tid=2, entry_tid=27222]]], state=connected] 2024-04-06T09:33:04.545 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=12, tag_tid=2, entry_tid=14132], [object_number=15, tag_tid=2, entry_tid=14131], [object_number=14, tag_tid=2, entry_tid=14130], [object_number=13, tag_tid=2, entry_tid=14129]]], state=connected] 2024-04-06T09:33:04.548 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:33:04.550 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=25, tag_tid=3, entry_tid=1' 2024-04-06T09:33:04.551 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:33:04.551 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:33:04.551 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:33:04.551 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:33:04.551 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:33:04.551 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:33:04.551 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:33:04.551 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:33:04.552 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:33:04.552 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:33:04.552 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:33:04.552 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:33:04.552 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:33:04.552 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:33:04.553 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:33:04.597 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 3 2024-04-06T09:33:04.597 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 6 2024-04-06T09:33:04.597 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:33:04.597 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=25, tag_tid=3, entry_tid=1], [object_number=24, tag_tid=3, entry_tid=0], [object_number=27, tag_tid=2, entry_tid=27223], [object_number=26, tag_tid=2, entry_tid=27222]]], state=connected] 2024-04-06T09:33:04.597 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=12, tag_tid=2, entry_tid=14132], [object_number=15, tag_tid=2, entry_tid=14131], [object_number=14, tag_tid=2, entry_tid=14130], [object_number=13, tag_tid=2, entry_tid=14129]]], state=connected] 2024-04-06T09:33:04.600 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:33:04.602 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=12, tag_tid=2, entry_tid=14132' 2024-04-06T09:33:04.602 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=25, tag_tid=3, entry_tid=1' -a 'object_number=25, tag_tid=3, entry_tid=1' = 'object_number=12, tag_tid=2, entry_tid=14132' 2024-04-06T09:33:04.602 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=12, tag_tid=2, entry_tid=14132' '!=' 'object_number=14, tag_tid=2, entry_tid=14014' 2024-04-06T09:33:04.603 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:33:04.603 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=12, tag_tid=2, entry_tid=14132' = 'object_number=14, tag_tid=2, entry_tid=14014' 2024-04-06T09:33:04.603 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=12, tag_tid=2, entry_tid=14132' 2024-04-06T09:33:04.603 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=25, tag_tid=3, entry_tid=1' 2024-04-06T09:33:04.603 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:33:04.603 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:33:04.603 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=3 2024-04-06T09:33:04.604 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=12, tag_tid=2, entry_tid=14132' 2024-04-06T09:33:04.604 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:33:04.604 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:33:04.605 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=2 2024-04-06T09:33:04.606 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=25, tag_tid=3, entry_tid=1' 2024-04-06T09:33:04.606 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:33:04.606 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:33:04.607 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:33:04.607 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=12, tag_tid=2, entry_tid=14132' 2024-04-06T09:33:04.607 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:33:04.607 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:33:04.608 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=14132 2024-04-06T09:33:04.609 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 3 = 2 -a 1 -le 14132 2024-04-06T09:33:04.609 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:33:04.609 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:33:04.609 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:33:04.810 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:33:04.811 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:33:04.811 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:33:04.811 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:33:04.811 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:33:04.811 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:33:04.811 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:33:04.811 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:33:04.811 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:33:04.811 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:33:04.811 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:33:04.811 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:33:04.811 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:33:04.811 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:33:04.812 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:33:04.812 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:33:04.812 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:33:04.815 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:33:04.815 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:33:04.815 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:33:04.815 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:33:04.815 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:33:04.815 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:33:04.815 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:33:04.816 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:33:04.816 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:33:04.816 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:33:04.816 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:33:04.816 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:33:04.816 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:33:04.816 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:33:04.816 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:33:04.816 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:33:04.816 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:33:04.816 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:33:04.816 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:33:04.816 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:33:04.816 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:33:04.817 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:33:04.817 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:33:04.817 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:33:04.817 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:33:04.817 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:33:04.817 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:33:04.817 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:33:04.817 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:33:04.817 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:33:04.817 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:33:04.817 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:33:04.840 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:33:04.840 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:33:04.840 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:33:04.841 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:33:04.961 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:33:04.962 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:33:04.962 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:33:04.962 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:33:04.962 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:33:04.962 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:33:04.962 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:33:04.962 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:33:04.963 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:33:04.963 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:33:04.963 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:33:04.963 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:33:04.963 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:33:04.964 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:33:04.965 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:33:04.965 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:33:05.017 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 3 2024-04-06T09:33:05.017 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 6 2024-04-06T09:33:05.017 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:33:05.017 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=25, tag_tid=3, entry_tid=1], [object_number=24, tag_tid=3, entry_tid=0], [object_number=27, tag_tid=2, entry_tid=27223], [object_number=26, tag_tid=2, entry_tid=27222]]], state=connected] 2024-04-06T09:33:05.017 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=14, tag_tid=2, entry_tid=14246], [object_number=13, tag_tid=2, entry_tid=14245], [object_number=12, tag_tid=2, entry_tid=14244], [object_number=15, tag_tid=2, entry_tid=14243]]], state=connected] 2024-04-06T09:33:05.020 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:33:05.021 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=25, tag_tid=3, entry_tid=1' 2024-04-06T09:33:05.022 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:33:05.022 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:33:05.022 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:33:05.023 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:33:05.023 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:33:05.023 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:33:05.023 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:33:05.023 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:33:05.023 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:33:05.023 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:33:05.023 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:33:05.023 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:33:05.024 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:33:05.024 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:33:05.024 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:33:05.065 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 3 2024-04-06T09:33:05.065 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 6 2024-04-06T09:33:05.065 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:33:05.065 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=25, tag_tid=3, entry_tid=1], [object_number=24, tag_tid=3, entry_tid=0], [object_number=27, tag_tid=2, entry_tid=27223], [object_number=26, tag_tid=2, entry_tid=27222]]], state=connected] 2024-04-06T09:33:05.065 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=14, tag_tid=2, entry_tid=14246], [object_number=13, tag_tid=2, entry_tid=14245], [object_number=12, tag_tid=2, entry_tid=14244], [object_number=15, tag_tid=2, entry_tid=14243]]], state=connected] 2024-04-06T09:33:05.068 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:33:05.070 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=14, tag_tid=2, entry_tid=14246' 2024-04-06T09:33:05.070 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=25, tag_tid=3, entry_tid=1' -a 'object_number=25, tag_tid=3, entry_tid=1' = 'object_number=14, tag_tid=2, entry_tid=14246' 2024-04-06T09:33:05.071 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=14, tag_tid=2, entry_tid=14246' '!=' 'object_number=12, tag_tid=2, entry_tid=14132' 2024-04-06T09:33:05.071 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:33:05.071 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=14, tag_tid=2, entry_tid=14246' = 'object_number=12, tag_tid=2, entry_tid=14132' 2024-04-06T09:33:05.071 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=14, tag_tid=2, entry_tid=14246' 2024-04-06T09:33:05.071 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=25, tag_tid=3, entry_tid=1' 2024-04-06T09:33:05.071 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:33:05.071 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:33:05.072 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=3 2024-04-06T09:33:05.072 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=14, tag_tid=2, entry_tid=14246' 2024-04-06T09:33:05.072 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:33:05.072 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:33:05.073 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=2 2024-04-06T09:33:05.073 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=25, tag_tid=3, entry_tid=1' 2024-04-06T09:33:05.073 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:33:05.073 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:33:05.074 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:33:05.075 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=14, tag_tid=2, entry_tid=14246' 2024-04-06T09:33:05.075 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:33:05.075 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:33:05.076 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=14246 2024-04-06T09:33:05.076 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 3 = 2 -a 1 -le 14246 2024-04-06T09:33:05.076 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:33:05.077 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:33:05.077 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:33:05.278 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:33:05.278 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:33:05.278 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:33:05.278 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:33:05.278 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:33:05.279 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:33:05.279 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:33:05.279 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:33:05.279 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:33:05.279 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:33:05.279 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:33:05.279 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:33:05.279 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:33:05.279 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:33:05.279 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:33:05.280 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:33:05.280 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:33:05.282 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:33:05.282 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:33:05.283 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:33:05.283 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:33:05.283 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:33:05.283 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:33:05.283 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:33:05.283 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:33:05.283 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:33:05.283 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:33:05.283 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:33:05.283 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:33:05.283 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:33:05.283 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:33:05.283 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:33:05.283 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:33:05.283 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:33:05.284 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:33:05.284 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:33:05.284 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:33:05.284 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:33:05.284 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:33:05.284 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:33:05.284 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:33:05.284 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:33:05.284 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:33:05.284 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:33:05.284 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:33:05.284 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:33:05.284 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:33:05.284 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:33:05.285 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:33:05.307 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:33:05.307 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:33:05.307 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:33:05.307 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:33:05.432 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:33:05.432 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:33:05.433 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:33:05.433 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:33:05.433 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:33:05.433 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:33:05.433 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:33:05.433 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:33:05.433 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:33:05.433 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:33:05.433 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:33:05.434 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:33:05.434 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:33:05.436 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:33:05.436 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:33:05.436 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:33:05.480 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 3 2024-04-06T09:33:05.480 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 6 2024-04-06T09:33:05.480 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:33:05.480 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=25, tag_tid=3, entry_tid=1], [object_number=24, tag_tid=3, entry_tid=0], [object_number=27, tag_tid=2, entry_tid=27223], [object_number=26, tag_tid=2, entry_tid=27222]]], state=connected] 2024-04-06T09:33:05.480 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=14, tag_tid=2, entry_tid=14350], [object_number=13, tag_tid=2, entry_tid=14349], [object_number=12, tag_tid=2, entry_tid=14348], [object_number=15, tag_tid=2, entry_tid=14347]]], state=connected] 2024-04-06T09:33:05.483 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:33:05.485 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=25, tag_tid=3, entry_tid=1' 2024-04-06T09:33:05.486 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:33:05.486 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:33:05.486 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:33:05.486 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:33:05.486 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:33:05.486 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:33:05.486 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:33:05.486 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:33:05.486 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:33:05.487 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:33:05.487 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:33:05.487 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:33:05.487 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:33:05.487 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:33:05.487 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:33:05.532 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 3 2024-04-06T09:33:05.532 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 6 2024-04-06T09:33:05.532 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:33:05.532 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=25, tag_tid=3, entry_tid=1], [object_number=24, tag_tid=3, entry_tid=0], [object_number=27, tag_tid=2, entry_tid=27223], [object_number=26, tag_tid=2, entry_tid=27222]]], state=connected] 2024-04-06T09:33:05.532 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=14, tag_tid=2, entry_tid=14350], [object_number=13, tag_tid=2, entry_tid=14349], [object_number=12, tag_tid=2, entry_tid=14348], [object_number=15, tag_tid=2, entry_tid=14347]]], state=connected] 2024-04-06T09:33:05.535 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:33:05.537 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=14, tag_tid=2, entry_tid=14350' 2024-04-06T09:33:05.537 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=25, tag_tid=3, entry_tid=1' -a 'object_number=25, tag_tid=3, entry_tid=1' = 'object_number=14, tag_tid=2, entry_tid=14350' 2024-04-06T09:33:05.538 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=14, tag_tid=2, entry_tid=14350' '!=' 'object_number=14, tag_tid=2, entry_tid=14246' 2024-04-06T09:33:05.538 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:33:05.538 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=14, tag_tid=2, entry_tid=14350' = 'object_number=14, tag_tid=2, entry_tid=14246' 2024-04-06T09:33:05.538 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=14, tag_tid=2, entry_tid=14350' 2024-04-06T09:33:05.538 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=25, tag_tid=3, entry_tid=1' 2024-04-06T09:33:05.538 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:33:05.538 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:33:05.538 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=3 2024-04-06T09:33:05.539 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=14, tag_tid=2, entry_tid=14350' 2024-04-06T09:33:05.539 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:33:05.539 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:33:05.540 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=2 2024-04-06T09:33:05.541 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=25, tag_tid=3, entry_tid=1' 2024-04-06T09:33:05.541 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:33:05.541 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:33:05.541 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:33:05.542 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=14, tag_tid=2, entry_tid=14350' 2024-04-06T09:33:05.542 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:33:05.542 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:33:05.543 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=14350 2024-04-06T09:33:05.544 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 3 = 2 -a 1 -le 14350 2024-04-06T09:33:05.544 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:33:05.544 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:33:05.544 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:33:05.745 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:33:05.745 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:33:05.745 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:33:05.745 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:33:05.745 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:33:05.746 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:33:05.746 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:33:05.746 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:33:05.746 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:33:05.746 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:33:05.746 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:33:05.746 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:33:05.746 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:33:05.746 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:33:05.746 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:33:05.747 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:33:05.747 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:33:05.750 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:33:05.750 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:33:05.750 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:33:05.750 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:33:05.750 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:33:05.751 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:33:05.751 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:33:05.751 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:33:05.751 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:33:05.751 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:33:05.751 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:33:05.751 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:33:05.751 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:33:05.751 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:33:05.751 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:33:05.751 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:33:05.751 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:33:05.751 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:33:05.752 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:33:05.752 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:33:05.752 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:33:05.752 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:33:05.752 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:33:05.752 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:33:05.752 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:33:05.752 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:33:05.752 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:33:05.752 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:33:05.752 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:33:05.752 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:33:05.753 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:33:05.753 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:33:05.775 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:33:05.775 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:33:05.775 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:33:05.775 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:33:05.897 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:33:05.898 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:33:05.898 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:33:05.898 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:33:05.898 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:33:05.898 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:33:05.898 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:33:05.898 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:33:05.898 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:33:05.899 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:33:05.899 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:33:05.899 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:33:05.899 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:33:05.900 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:33:05.900 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:33:05.900 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:33:05.946 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 3 2024-04-06T09:33:05.946 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 6 2024-04-06T09:33:05.946 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:33:05.946 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=25, tag_tid=3, entry_tid=1], [object_number=24, tag_tid=3, entry_tid=0], [object_number=27, tag_tid=2, entry_tid=27223], [object_number=26, tag_tid=2, entry_tid=27222]]], state=connected] 2024-04-06T09:33:05.947 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=12, tag_tid=2, entry_tid=14456], [object_number=15, tag_tid=2, entry_tid=14455], [object_number=14, tag_tid=2, entry_tid=14454], [object_number=13, tag_tid=2, entry_tid=14453]]], state=connected] 2024-04-06T09:33:05.949 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:33:05.951 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=25, tag_tid=3, entry_tid=1' 2024-04-06T09:33:05.951 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:33:05.951 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:33:05.952 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:33:05.952 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:33:05.952 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:33:05.952 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:33:05.952 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:33:05.952 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:33:05.952 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:33:05.952 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:33:05.953 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:33:05.953 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:33:05.953 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:33:05.953 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:33:05.953 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:33:05.995 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 3 2024-04-06T09:33:05.995 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 6 2024-04-06T09:33:05.995 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:33:05.995 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=25, tag_tid=3, entry_tid=1], [object_number=24, tag_tid=3, entry_tid=0], [object_number=27, tag_tid=2, entry_tid=27223], [object_number=26, tag_tid=2, entry_tid=27222]]], state=connected] 2024-04-06T09:33:05.995 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=12, tag_tid=2, entry_tid=14456], [object_number=15, tag_tid=2, entry_tid=14455], [object_number=14, tag_tid=2, entry_tid=14454], [object_number=13, tag_tid=2, entry_tid=14453]]], state=connected] 2024-04-06T09:33:05.998 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:33:06.000 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=12, tag_tid=2, entry_tid=14456' 2024-04-06T09:33:06.000 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=25, tag_tid=3, entry_tid=1' -a 'object_number=25, tag_tid=3, entry_tid=1' = 'object_number=12, tag_tid=2, entry_tid=14456' 2024-04-06T09:33:06.000 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=12, tag_tid=2, entry_tid=14456' '!=' 'object_number=14, tag_tid=2, entry_tid=14350' 2024-04-06T09:33:06.000 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:33:06.000 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=12, tag_tid=2, entry_tid=14456' = 'object_number=14, tag_tid=2, entry_tid=14350' 2024-04-06T09:33:06.000 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=12, tag_tid=2, entry_tid=14456' 2024-04-06T09:33:06.001 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=25, tag_tid=3, entry_tid=1' 2024-04-06T09:33:06.001 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:33:06.001 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:33:06.001 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=3 2024-04-06T09:33:06.002 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=12, tag_tid=2, entry_tid=14456' 2024-04-06T09:33:06.002 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:33:06.002 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:33:06.003 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=2 2024-04-06T09:33:06.007 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=25, tag_tid=3, entry_tid=1' 2024-04-06T09:33:06.007 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:33:06.007 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:33:06.007 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:33:06.007 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=12, tag_tid=2, entry_tid=14456' 2024-04-06T09:33:06.007 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:33:06.007 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:33:06.007 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=14456 2024-04-06T09:33:06.007 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 3 = 2 -a 1 -le 14456 2024-04-06T09:33:06.007 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:33:06.007 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:33:06.007 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:33:06.208 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:33:06.208 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:33:06.208 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:33:06.208 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:33:06.208 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:33:06.208 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:33:06.208 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:33:06.208 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:33:06.208 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:33:06.208 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:33:06.208 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:33:06.209 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:33:06.209 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:33:06.209 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:33:06.209 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:33:06.209 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:33:06.209 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:33:06.212 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:33:06.212 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:33:06.212 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:33:06.212 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:33:06.212 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:33:06.212 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:33:06.212 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:33:06.212 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:33:06.212 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:33:06.213 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:33:06.213 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:33:06.213 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:33:06.213 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:33:06.213 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:33:06.213 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:33:06.213 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:33:06.213 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:33:06.213 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:33:06.213 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:33:06.213 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:33:06.213 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:33:06.213 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:33:06.213 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:33:06.214 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:33:06.214 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:33:06.214 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:33:06.214 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:33:06.214 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:33:06.214 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:33:06.214 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:33:06.214 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:33:06.214 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:33:06.237 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:33:06.237 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:33:06.237 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:33:06.237 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:33:06.362 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:33:06.363 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:33:06.363 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:33:06.363 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:33:06.363 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:33:06.363 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:33:06.363 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:33:06.363 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:33:06.363 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:33:06.363 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:33:06.364 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:33:06.364 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:33:06.364 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:33:06.367 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:33:06.367 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:33:06.367 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:33:06.422 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 3 2024-04-06T09:33:06.422 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 6 2024-04-06T09:33:06.422 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:33:06.422 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=25, tag_tid=3, entry_tid=1], [object_number=24, tag_tid=3, entry_tid=0], [object_number=27, tag_tid=2, entry_tid=27223], [object_number=26, tag_tid=2, entry_tid=27222]]], state=connected] 2024-04-06T09:33:06.423 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=15, tag_tid=2, entry_tid=14563], [object_number=14, tag_tid=2, entry_tid=14562], [object_number=13, tag_tid=2, entry_tid=14561], [object_number=12, tag_tid=2, entry_tid=14560]]], state=connected] 2024-04-06T09:33:06.426 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:33:06.427 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=25, tag_tid=3, entry_tid=1' 2024-04-06T09:33:06.428 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:33:06.428 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:33:06.428 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:33:06.428 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:33:06.429 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:33:06.429 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:33:06.429 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:33:06.429 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:33:06.429 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:33:06.429 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:33:06.429 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:33:06.429 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:33:06.430 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:33:06.430 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:33:06.430 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:33:06.475 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 3 2024-04-06T09:33:06.475 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 6 2024-04-06T09:33:06.475 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:33:06.475 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=25, tag_tid=3, entry_tid=1], [object_number=24, tag_tid=3, entry_tid=0], [object_number=27, tag_tid=2, entry_tid=27223], [object_number=26, tag_tid=2, entry_tid=27222]]], state=connected] 2024-04-06T09:33:06.475 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=15, tag_tid=2, entry_tid=14563], [object_number=14, tag_tid=2, entry_tid=14562], [object_number=13, tag_tid=2, entry_tid=14561], [object_number=12, tag_tid=2, entry_tid=14560]]], state=connected] 2024-04-06T09:33:06.478 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:33:06.481 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=15, tag_tid=2, entry_tid=14563' 2024-04-06T09:33:06.481 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=25, tag_tid=3, entry_tid=1' -a 'object_number=25, tag_tid=3, entry_tid=1' = 'object_number=15, tag_tid=2, entry_tid=14563' 2024-04-06T09:33:06.481 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=15, tag_tid=2, entry_tid=14563' '!=' 'object_number=12, tag_tid=2, entry_tid=14456' 2024-04-06T09:33:06.481 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:33:06.481 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=15, tag_tid=2, entry_tid=14563' = 'object_number=12, tag_tid=2, entry_tid=14456' 2024-04-06T09:33:06.481 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=15, tag_tid=2, entry_tid=14563' 2024-04-06T09:33:06.481 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=25, tag_tid=3, entry_tid=1' 2024-04-06T09:33:06.481 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:33:06.481 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:33:06.482 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=3 2024-04-06T09:33:06.482 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=15, tag_tid=2, entry_tid=14563' 2024-04-06T09:33:06.483 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:33:06.483 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:33:06.483 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=2 2024-04-06T09:33:06.484 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=25, tag_tid=3, entry_tid=1' 2024-04-06T09:33:06.484 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:33:06.484 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:33:06.485 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:33:06.486 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=15, tag_tid=2, entry_tid=14563' 2024-04-06T09:33:06.486 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:33:06.486 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:33:06.487 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=14563 2024-04-06T09:33:06.487 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 3 = 2 -a 1 -le 14563 2024-04-06T09:33:06.487 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:33:06.487 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:33:06.488 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:33:06.689 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:33:06.689 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:33:06.689 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:33:06.689 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:33:06.689 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:33:06.689 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:33:06.689 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:33:06.689 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:33:06.689 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:33:06.690 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:33:06.690 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:33:06.690 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:33:06.690 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:33:06.690 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:33:06.690 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:33:06.691 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:33:06.691 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:33:06.695 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:33:06.695 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:33:06.695 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:33:06.695 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:33:06.695 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:33:06.696 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:33:06.696 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:33:06.696 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:33:06.696 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:33:06.696 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:33:06.696 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:33:06.696 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:33:06.696 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:33:06.696 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:33:06.696 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:33:06.696 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:33:06.696 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:33:06.696 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:33:06.697 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:33:06.697 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:33:06.697 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:33:06.697 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:33:06.697 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:33:06.697 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:33:06.697 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:33:06.697 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:33:06.697 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:33:06.697 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:33:06.697 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:33:06.697 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:33:06.697 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:33:06.697 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:33:06.720 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:33:06.720 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:33:06.720 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:33:06.720 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:33:06.848 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:33:06.849 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:33:06.849 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:33:06.849 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:33:06.849 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:33:06.849 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:33:06.849 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:33:06.849 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:33:06.850 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:33:06.850 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:33:06.850 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:33:06.850 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:33:06.850 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:33:06.852 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:33:06.853 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:33:06.853 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:33:06.903 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 3 2024-04-06T09:33:06.903 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 6 2024-04-06T09:33:06.903 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:33:06.903 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=25, tag_tid=3, entry_tid=1], [object_number=24, tag_tid=3, entry_tid=0], [object_number=27, tag_tid=2, entry_tid=27223], [object_number=26, tag_tid=2, entry_tid=27222]]], state=connected] 2024-04-06T09:33:06.903 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=14, tag_tid=2, entry_tid=14670], [object_number=13, tag_tid=2, entry_tid=14669], [object_number=12, tag_tid=2, entry_tid=14668], [object_number=15, tag_tid=2, entry_tid=14667]]], state=connected] 2024-04-06T09:33:06.907 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:33:06.908 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=25, tag_tid=3, entry_tid=1' 2024-04-06T09:33:06.909 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:33:06.909 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:33:06.909 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:33:06.909 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:33:06.909 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:33:06.909 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:33:06.909 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:33:06.909 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:33:06.909 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:33:06.909 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:33:06.910 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:33:06.910 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:33:06.910 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:33:06.911 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:33:06.911 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:33:06.956 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 3 2024-04-06T09:33:06.956 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 6 2024-04-06T09:33:06.956 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:33:06.956 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=25, tag_tid=3, entry_tid=1], [object_number=24, tag_tid=3, entry_tid=0], [object_number=27, tag_tid=2, entry_tid=27223], [object_number=26, tag_tid=2, entry_tid=27222]]], state=connected] 2024-04-06T09:33:06.956 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=14, tag_tid=2, entry_tid=14670], [object_number=13, tag_tid=2, entry_tid=14669], [object_number=12, tag_tid=2, entry_tid=14668], [object_number=15, tag_tid=2, entry_tid=14667]]], state=connected] 2024-04-06T09:33:06.959 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:33:06.961 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=14, tag_tid=2, entry_tid=14670' 2024-04-06T09:33:06.962 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=25, tag_tid=3, entry_tid=1' -a 'object_number=25, tag_tid=3, entry_tid=1' = 'object_number=14, tag_tid=2, entry_tid=14670' 2024-04-06T09:33:06.962 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=14, tag_tid=2, entry_tid=14670' '!=' 'object_number=15, tag_tid=2, entry_tid=14563' 2024-04-06T09:33:06.962 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:33:06.962 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=14, tag_tid=2, entry_tid=14670' = 'object_number=15, tag_tid=2, entry_tid=14563' 2024-04-06T09:33:06.962 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=14, tag_tid=2, entry_tid=14670' 2024-04-06T09:33:06.962 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=25, tag_tid=3, entry_tid=1' 2024-04-06T09:33:06.962 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:33:06.962 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:33:06.963 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=3 2024-04-06T09:33:06.963 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=14, tag_tid=2, entry_tid=14670' 2024-04-06T09:33:06.963 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:33:06.964 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:33:06.964 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=2 2024-04-06T09:33:06.965 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=25, tag_tid=3, entry_tid=1' 2024-04-06T09:33:06.965 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:33:06.965 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:33:06.966 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:33:06.967 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=14, tag_tid=2, entry_tid=14670' 2024-04-06T09:33:06.967 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:33:06.967 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:33:06.968 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=14670 2024-04-06T09:33:06.968 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 3 = 2 -a 1 -le 14670 2024-04-06T09:33:06.968 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:33:06.968 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:33:06.968 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:33:07.170 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:33:07.170 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:33:07.170 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:33:07.170 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:33:07.170 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:33:07.171 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:33:07.171 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:33:07.171 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:33:07.171 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:33:07.171 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:33:07.171 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:33:07.171 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:33:07.171 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:33:07.171 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:33:07.171 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:33:07.172 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:33:07.172 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:33:07.174 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:33:07.175 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:33:07.175 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:33:07.175 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:33:07.175 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:33:07.175 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:33:07.175 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:33:07.175 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:33:07.175 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:33:07.175 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:33:07.175 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:33:07.175 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:33:07.175 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:33:07.175 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:33:07.175 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:33:07.175 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:33:07.176 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:33:07.176 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:33:07.176 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:33:07.176 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:33:07.176 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:33:07.176 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:33:07.176 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:33:07.176 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:33:07.176 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:33:07.176 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:33:07.176 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:33:07.176 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:33:07.176 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:33:07.176 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:33:07.176 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:33:07.177 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:33:07.200 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:33:07.200 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:33:07.200 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:33:07.200 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:33:07.331 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:33:07.332 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:33:07.332 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:33:07.333 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:33:07.333 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:33:07.333 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:33:07.333 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:33:07.333 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:33:07.333 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:33:07.333 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:33:07.333 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:33:07.333 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:33:07.334 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:33:07.335 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:33:07.336 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:33:07.336 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:33:07.386 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 3 2024-04-06T09:33:07.386 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 6 2024-04-06T09:33:07.386 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:33:07.386 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=25, tag_tid=3, entry_tid=1], [object_number=24, tag_tid=3, entry_tid=0], [object_number=27, tag_tid=2, entry_tid=27223], [object_number=26, tag_tid=2, entry_tid=27222]]], state=connected] 2024-04-06T09:33:07.387 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=15, tag_tid=2, entry_tid=14775], [object_number=14, tag_tid=2, entry_tid=14774], [object_number=13, tag_tid=2, entry_tid=14773], [object_number=12, tag_tid=2, entry_tid=14772]]], state=connected] 2024-04-06T09:33:07.390 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:33:07.391 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=25, tag_tid=3, entry_tid=1' 2024-04-06T09:33:07.392 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:33:07.392 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:33:07.392 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:33:07.392 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:33:07.392 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:33:07.392 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:33:07.392 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:33:07.392 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:33:07.393 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:33:07.393 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:33:07.393 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:33:07.393 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:33:07.394 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:33:07.394 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:33:07.394 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:33:07.439 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 3 2024-04-06T09:33:07.439 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 6 2024-04-06T09:33:07.439 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:33:07.439 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=25, tag_tid=3, entry_tid=1], [object_number=24, tag_tid=3, entry_tid=0], [object_number=27, tag_tid=2, entry_tid=27223], [object_number=26, tag_tid=2, entry_tid=27222]]], state=connected] 2024-04-06T09:33:07.439 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=15, tag_tid=2, entry_tid=14775], [object_number=14, tag_tid=2, entry_tid=14774], [object_number=13, tag_tid=2, entry_tid=14773], [object_number=12, tag_tid=2, entry_tid=14772]]], state=connected] 2024-04-06T09:33:07.443 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:33:07.444 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=15, tag_tid=2, entry_tid=14775' 2024-04-06T09:33:07.444 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=25, tag_tid=3, entry_tid=1' -a 'object_number=25, tag_tid=3, entry_tid=1' = 'object_number=15, tag_tid=2, entry_tid=14775' 2024-04-06T09:33:07.445 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=15, tag_tid=2, entry_tid=14775' '!=' 'object_number=14, tag_tid=2, entry_tid=14670' 2024-04-06T09:33:07.445 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:33:07.445 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=15, tag_tid=2, entry_tid=14775' = 'object_number=14, tag_tid=2, entry_tid=14670' 2024-04-06T09:33:07.445 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=15, tag_tid=2, entry_tid=14775' 2024-04-06T09:33:07.445 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=25, tag_tid=3, entry_tid=1' 2024-04-06T09:33:07.445 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:33:07.446 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:33:07.446 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=3 2024-04-06T09:33:07.447 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=15, tag_tid=2, entry_tid=14775' 2024-04-06T09:33:07.447 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:33:07.447 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:33:07.448 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=2 2024-04-06T09:33:07.448 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=25, tag_tid=3, entry_tid=1' 2024-04-06T09:33:07.448 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:33:07.448 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:33:07.449 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:33:07.450 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=15, tag_tid=2, entry_tid=14775' 2024-04-06T09:33:07.450 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:33:07.450 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:33:07.451 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=14775 2024-04-06T09:33:07.451 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 3 = 2 -a 1 -le 14775 2024-04-06T09:33:07.451 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:33:07.451 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:33:07.452 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:33:07.653 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:33:07.654 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:33:07.654 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:33:07.654 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:33:07.654 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:33:07.654 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:33:07.654 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:33:07.654 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:33:07.654 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:33:07.654 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:33:07.654 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:33:07.654 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:33:07.654 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:33:07.654 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:33:07.655 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:33:07.655 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:33:07.655 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:33:07.658 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:33:07.658 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:33:07.659 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:33:07.659 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:33:07.659 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:33:07.659 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:33:07.659 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:33:07.659 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:33:07.659 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:33:07.659 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:33:07.659 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:33:07.659 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:33:07.659 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:33:07.659 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:33:07.659 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:33:07.660 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:33:07.660 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:33:07.660 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:33:07.660 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:33:07.660 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:33:07.660 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:33:07.660 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:33:07.660 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:33:07.660 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:33:07.660 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:33:07.660 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:33:07.660 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:33:07.660 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:33:07.660 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:33:07.660 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:33:07.660 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:33:07.661 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:33:07.684 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:33:07.684 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:33:07.684 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:33:07.684 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:33:07.818 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:33:07.819 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:33:07.819 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:33:07.819 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:33:07.819 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:33:07.819 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:33:07.819 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:33:07.819 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:33:07.819 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:33:07.819 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:33:07.820 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:33:07.820 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:33:07.820 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:33:07.823 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:33:07.823 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:33:07.823 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:33:07.868 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 3 2024-04-06T09:33:07.868 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 6 2024-04-06T09:33:07.868 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:33:07.868 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=25, tag_tid=3, entry_tid=1], [object_number=24, tag_tid=3, entry_tid=0], [object_number=27, tag_tid=2, entry_tid=27223], [object_number=26, tag_tid=2, entry_tid=27222]]], state=connected] 2024-04-06T09:33:07.868 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=15, tag_tid=2, entry_tid=14879], [object_number=14, tag_tid=2, entry_tid=14878], [object_number=13, tag_tid=2, entry_tid=14877], [object_number=12, tag_tid=2, entry_tid=14876]]], state=connected] 2024-04-06T09:33:07.871 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:33:07.873 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=25, tag_tid=3, entry_tid=1' 2024-04-06T09:33:07.874 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:33:07.874 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:33:07.874 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:33:07.874 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:33:07.874 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:33:07.874 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:33:07.874 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:33:07.874 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:33:07.874 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:33:07.874 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:33:07.874 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:33:07.875 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:33:07.875 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:33:07.875 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:33:07.875 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:33:07.921 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 3 2024-04-06T09:33:07.921 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 6 2024-04-06T09:33:07.921 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:33:07.921 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=25, tag_tid=3, entry_tid=1], [object_number=24, tag_tid=3, entry_tid=0], [object_number=27, tag_tid=2, entry_tid=27223], [object_number=26, tag_tid=2, entry_tid=27222]]], state=connected] 2024-04-06T09:33:07.921 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=15, tag_tid=2, entry_tid=14879], [object_number=14, tag_tid=2, entry_tid=14878], [object_number=13, tag_tid=2, entry_tid=14877], [object_number=12, tag_tid=2, entry_tid=14876]]], state=connected] 2024-04-06T09:33:07.924 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:33:07.927 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=15, tag_tid=2, entry_tid=14879' 2024-04-06T09:33:07.927 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=25, tag_tid=3, entry_tid=1' -a 'object_number=25, tag_tid=3, entry_tid=1' = 'object_number=15, tag_tid=2, entry_tid=14879' 2024-04-06T09:33:07.927 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=15, tag_tid=2, entry_tid=14879' '!=' 'object_number=15, tag_tid=2, entry_tid=14775' 2024-04-06T09:33:07.927 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:33:07.927 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=15, tag_tid=2, entry_tid=14879' = 'object_number=15, tag_tid=2, entry_tid=14775' 2024-04-06T09:33:07.927 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=15, tag_tid=2, entry_tid=14879' 2024-04-06T09:33:07.927 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=25, tag_tid=3, entry_tid=1' 2024-04-06T09:33:07.927 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:33:07.927 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:33:07.928 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=3 2024-04-06T09:33:07.928 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=15, tag_tid=2, entry_tid=14879' 2024-04-06T09:33:07.929 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:33:07.929 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:33:07.930 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=2 2024-04-06T09:33:07.930 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=25, tag_tid=3, entry_tid=1' 2024-04-06T09:33:07.930 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:33:07.930 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:33:07.931 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:33:07.932 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=15, tag_tid=2, entry_tid=14879' 2024-04-06T09:33:07.932 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:33:07.932 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:33:07.933 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=14879 2024-04-06T09:33:07.933 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 3 = 2 -a 1 -le 14879 2024-04-06T09:33:07.933 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:33:07.933 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:33:07.933 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:33:08.135 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:33:08.135 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:33:08.135 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:33:08.135 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:33:08.135 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:33:08.135 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:33:08.136 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:33:08.136 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:33:08.136 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:33:08.136 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:33:08.136 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:33:08.136 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:33:08.136 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:33:08.136 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:33:08.136 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:33:08.137 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:33:08.137 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:33:08.139 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:33:08.139 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:33:08.139 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:33:08.139 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:33:08.139 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:33:08.139 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:33:08.139 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:33:08.139 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:33:08.139 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:33:08.139 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:33:08.139 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:33:08.139 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:33:08.140 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:33:08.140 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:33:08.140 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:33:08.140 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:33:08.140 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:33:08.140 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:33:08.140 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:33:08.140 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:33:08.140 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:33:08.140 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:33:08.140 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:33:08.140 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:33:08.140 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:33:08.140 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:33:08.141 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:33:08.141 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:33:08.141 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:33:08.141 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:33:08.141 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:33:08.141 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:33:08.166 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:33:08.167 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:33:08.167 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:33:08.167 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:33:08.295 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:33:08.297 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:33:08.297 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:33:08.297 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:33:08.297 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:33:08.297 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:33:08.297 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:33:08.297 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:33:08.297 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:33:08.297 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:33:08.298 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:33:08.298 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:33:08.298 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:33:08.300 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:33:08.300 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:33:08.301 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:33:08.351 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 3 2024-04-06T09:33:08.351 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 6 2024-04-06T09:33:08.351 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:33:08.351 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=25, tag_tid=3, entry_tid=1], [object_number=24, tag_tid=3, entry_tid=0], [object_number=27, tag_tid=2, entry_tid=27223], [object_number=26, tag_tid=2, entry_tid=27222]]], state=connected] 2024-04-06T09:33:08.351 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=14, tag_tid=2, entry_tid=14978], [object_number=13, tag_tid=2, entry_tid=14977], [object_number=12, tag_tid=2, entry_tid=14976], [object_number=15, tag_tid=2, entry_tid=14975]]], state=connected] 2024-04-06T09:33:08.354 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:33:08.355 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=25, tag_tid=3, entry_tid=1' 2024-04-06T09:33:08.357 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:33:08.357 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:33:08.357 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:33:08.357 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:33:08.357 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:33:08.357 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:33:08.357 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:33:08.357 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:33:08.357 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:33:08.357 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:33:08.357 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:33:08.357 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:33:08.358 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:33:08.358 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:33:08.358 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:33:08.407 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 3 2024-04-06T09:33:08.407 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 6 2024-04-06T09:33:08.407 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:33:08.407 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=25, tag_tid=3, entry_tid=1], [object_number=24, tag_tid=3, entry_tid=0], [object_number=27, tag_tid=2, entry_tid=27223], [object_number=26, tag_tid=2, entry_tid=27222]]], state=connected] 2024-04-06T09:33:08.407 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=14, tag_tid=2, entry_tid=14978], [object_number=13, tag_tid=2, entry_tid=14977], [object_number=12, tag_tid=2, entry_tid=14976], [object_number=15, tag_tid=2, entry_tid=14975]]], state=connected] 2024-04-06T09:33:08.411 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:33:08.413 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=14, tag_tid=2, entry_tid=14978' 2024-04-06T09:33:08.413 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=25, tag_tid=3, entry_tid=1' -a 'object_number=25, tag_tid=3, entry_tid=1' = 'object_number=14, tag_tid=2, entry_tid=14978' 2024-04-06T09:33:08.413 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=14, tag_tid=2, entry_tid=14978' '!=' 'object_number=15, tag_tid=2, entry_tid=14879' 2024-04-06T09:33:08.413 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:33:08.413 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=14, tag_tid=2, entry_tid=14978' = 'object_number=15, tag_tid=2, entry_tid=14879' 2024-04-06T09:33:08.413 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=14, tag_tid=2, entry_tid=14978' 2024-04-06T09:33:08.413 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=25, tag_tid=3, entry_tid=1' 2024-04-06T09:33:08.413 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:33:08.413 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:33:08.414 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=3 2024-04-06T09:33:08.414 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=14, tag_tid=2, entry_tid=14978' 2024-04-06T09:33:08.415 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:33:08.415 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:33:08.416 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=2 2024-04-06T09:33:08.416 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=25, tag_tid=3, entry_tid=1' 2024-04-06T09:33:08.416 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:33:08.416 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:33:08.417 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:33:08.418 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=14, tag_tid=2, entry_tid=14978' 2024-04-06T09:33:08.418 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:33:08.418 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:33:08.419 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=14978 2024-04-06T09:33:08.419 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 3 = 2 -a 1 -le 14978 2024-04-06T09:33:08.419 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:33:08.419 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:33:08.419 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:33:08.621 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:33:08.621 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:33:08.621 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:33:08.622 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:33:08.622 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:33:08.622 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:33:08.622 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:33:08.622 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:33:08.622 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:33:08.622 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:33:08.622 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:33:08.622 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:33:08.622 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:33:08.622 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:33:08.622 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:33:08.623 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:33:08.623 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:33:08.626 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:33:08.626 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:33:08.626 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:33:08.626 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:33:08.626 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:33:08.626 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:33:08.626 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:33:08.627 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:33:08.627 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:33:08.627 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:33:08.627 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:33:08.627 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:33:08.627 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:33:08.627 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:33:08.627 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:33:08.627 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:33:08.627 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:33:08.627 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:33:08.627 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:33:08.627 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:33:08.627 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:33:08.627 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:33:08.627 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:33:08.628 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:33:08.628 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:33:08.628 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:33:08.628 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:33:08.628 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:33:08.628 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:33:08.628 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:33:08.628 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:33:08.628 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:33:08.652 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:33:08.652 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:33:08.652 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:33:08.652 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:33:08.787 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:33:08.788 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:33:08.788 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:33:08.788 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:33:08.788 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:33:08.788 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:33:08.788 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:33:08.788 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:33:08.788 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:33:08.788 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:33:08.789 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:33:08.789 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:33:08.789 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:33:08.792 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:33:08.792 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:33:08.793 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:33:08.841 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 3 2024-04-06T09:33:08.842 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 6 2024-04-06T09:33:08.842 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:33:08.842 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=25, tag_tid=3, entry_tid=1], [object_number=24, tag_tid=3, entry_tid=0], [object_number=27, tag_tid=2, entry_tid=27223], [object_number=26, tag_tid=2, entry_tid=27222]]], state=connected] 2024-04-06T09:33:08.842 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=14, tag_tid=2, entry_tid=15074], [object_number=13, tag_tid=2, entry_tid=15073], [object_number=12, tag_tid=2, entry_tid=15072], [object_number=15, tag_tid=2, entry_tid=15071]]], state=connected] 2024-04-06T09:33:08.845 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:33:08.847 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=25, tag_tid=3, entry_tid=1' 2024-04-06T09:33:08.848 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:33:08.848 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:33:08.848 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:33:08.848 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:33:08.848 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:33:08.848 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:33:08.848 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:33:08.848 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:33:08.848 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:33:08.848 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:33:08.849 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:33:08.849 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:33:08.849 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:33:08.849 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:33:08.849 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:33:08.895 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 3 2024-04-06T09:33:08.895 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 6 2024-04-06T09:33:08.895 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:33:08.895 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=25, tag_tid=3, entry_tid=1], [object_number=24, tag_tid=3, entry_tid=0], [object_number=27, tag_tid=2, entry_tid=27223], [object_number=26, tag_tid=2, entry_tid=27222]]], state=connected] 2024-04-06T09:33:08.895 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=14, tag_tid=2, entry_tid=15074], [object_number=13, tag_tid=2, entry_tid=15073], [object_number=12, tag_tid=2, entry_tid=15072], [object_number=15, tag_tid=2, entry_tid=15071]]], state=connected] 2024-04-06T09:33:08.898 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:33:08.901 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=14, tag_tid=2, entry_tid=15074' 2024-04-06T09:33:08.901 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=25, tag_tid=3, entry_tid=1' -a 'object_number=25, tag_tid=3, entry_tid=1' = 'object_number=14, tag_tid=2, entry_tid=15074' 2024-04-06T09:33:08.901 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=14, tag_tid=2, entry_tid=15074' '!=' 'object_number=14, tag_tid=2, entry_tid=14978' 2024-04-06T09:33:08.901 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:33:08.901 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=14, tag_tid=2, entry_tid=15074' = 'object_number=14, tag_tid=2, entry_tid=14978' 2024-04-06T09:33:08.901 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=14, tag_tid=2, entry_tid=15074' 2024-04-06T09:33:08.901 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=25, tag_tid=3, entry_tid=1' 2024-04-06T09:33:08.901 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:33:08.901 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:33:08.902 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=3 2024-04-06T09:33:08.902 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=14, tag_tid=2, entry_tid=15074' 2024-04-06T09:33:08.903 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:33:08.903 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:33:08.903 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=2 2024-04-06T09:33:08.904 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=25, tag_tid=3, entry_tid=1' 2024-04-06T09:33:08.904 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:33:08.904 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:33:08.905 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:33:08.906 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=14, tag_tid=2, entry_tid=15074' 2024-04-06T09:33:08.906 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:33:08.906 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:33:08.907 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=15074 2024-04-06T09:33:08.907 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 3 = 2 -a 1 -le 15074 2024-04-06T09:33:08.907 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:33:08.907 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:33:08.907 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:33:09.109 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:33:09.109 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:33:09.109 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:33:09.109 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:33:09.109 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:33:09.109 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:33:09.109 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:33:09.110 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:33:09.110 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:33:09.110 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:33:09.110 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:33:09.110 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:33:09.110 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:33:09.110 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:33:09.110 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:33:09.111 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:33:09.111 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:33:09.114 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:33:09.114 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:33:09.115 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:33:09.115 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:33:09.115 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:33:09.115 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:33:09.115 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:33:09.115 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:33:09.115 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:33:09.115 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:33:09.115 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:33:09.115 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:33:09.115 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:33:09.115 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:33:09.115 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:33:09.115 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:33:09.115 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:33:09.115 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:33:09.116 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:33:09.116 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:33:09.116 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:33:09.116 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:33:09.116 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:33:09.116 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:33:09.116 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:33:09.116 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:33:09.116 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:33:09.116 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:33:09.116 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:33:09.116 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:33:09.116 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:33:09.116 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:33:09.142 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:33:09.142 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:33:09.142 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:33:09.142 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:33:09.270 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:33:09.270 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:33:09.270 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:33:09.270 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:33:09.271 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:33:09.271 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:33:09.271 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:33:09.271 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:33:09.271 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:33:09.271 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:33:09.271 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:33:09.272 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:33:09.272 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:33:09.273 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:33:09.273 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:33:09.273 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:33:09.328 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 3 2024-04-06T09:33:09.328 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 6 2024-04-06T09:33:09.328 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:33:09.328 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=25, tag_tid=3, entry_tid=1], [object_number=24, tag_tid=3, entry_tid=0], [object_number=27, tag_tid=2, entry_tid=27223], [object_number=26, tag_tid=2, entry_tid=27222]]], state=connected] 2024-04-06T09:33:09.329 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=12, tag_tid=2, entry_tid=15168], [object_number=15, tag_tid=2, entry_tid=15167], [object_number=14, tag_tid=2, entry_tid=15166], [object_number=13, tag_tid=2, entry_tid=15165]]], state=connected] 2024-04-06T09:33:09.332 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:33:09.333 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=25, tag_tid=3, entry_tid=1' 2024-04-06T09:33:09.334 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:33:09.334 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:33:09.334 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:33:09.334 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:33:09.335 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:33:09.335 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:33:09.335 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:33:09.335 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:33:09.335 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:33:09.335 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:33:09.335 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:33:09.335 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:33:09.336 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:33:09.336 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:33:09.336 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:33:09.381 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 3 2024-04-06T09:33:09.381 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 6 2024-04-06T09:33:09.381 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:33:09.381 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=25, tag_tid=3, entry_tid=1], [object_number=24, tag_tid=3, entry_tid=0], [object_number=27, tag_tid=2, entry_tid=27223], [object_number=26, tag_tid=2, entry_tid=27222]]], state=connected] 2024-04-06T09:33:09.382 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=12, tag_tid=2, entry_tid=15168], [object_number=15, tag_tid=2, entry_tid=15167], [object_number=14, tag_tid=2, entry_tid=15166], [object_number=13, tag_tid=2, entry_tid=15165]]], state=connected] 2024-04-06T09:33:09.385 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:33:09.387 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=12, tag_tid=2, entry_tid=15168' 2024-04-06T09:33:09.387 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=25, tag_tid=3, entry_tid=1' -a 'object_number=25, tag_tid=3, entry_tid=1' = 'object_number=12, tag_tid=2, entry_tid=15168' 2024-04-06T09:33:09.387 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=12, tag_tid=2, entry_tid=15168' '!=' 'object_number=14, tag_tid=2, entry_tid=15074' 2024-04-06T09:33:09.387 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:33:09.388 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=12, tag_tid=2, entry_tid=15168' = 'object_number=14, tag_tid=2, entry_tid=15074' 2024-04-06T09:33:09.388 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=12, tag_tid=2, entry_tid=15168' 2024-04-06T09:33:09.388 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=25, tag_tid=3, entry_tid=1' 2024-04-06T09:33:09.388 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:33:09.388 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:33:09.388 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=3 2024-04-06T09:33:09.389 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=12, tag_tid=2, entry_tid=15168' 2024-04-06T09:33:09.389 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:33:09.389 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:33:09.390 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=2 2024-04-06T09:33:09.390 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=25, tag_tid=3, entry_tid=1' 2024-04-06T09:33:09.391 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:33:09.391 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:33:09.391 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:33:09.392 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=12, tag_tid=2, entry_tid=15168' 2024-04-06T09:33:09.392 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:33:09.392 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:33:09.393 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=15168 2024-04-06T09:33:09.394 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 3 = 2 -a 1 -le 15168 2024-04-06T09:33:09.394 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:33:09.394 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:33:09.394 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:33:09.595 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:33:09.596 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:33:09.596 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:33:09.596 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:33:09.596 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:33:09.596 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:33:09.596 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:33:09.596 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:33:09.596 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:33:09.596 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:33:09.596 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:33:09.596 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:33:09.596 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:33:09.596 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:33:09.596 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:33:09.597 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:33:09.597 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:33:09.600 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:33:09.600 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:33:09.600 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:33:09.600 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:33:09.600 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:33:09.600 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:33:09.601 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:33:09.601 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:33:09.601 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:33:09.601 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:33:09.601 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:33:09.601 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:33:09.601 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:33:09.601 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:33:09.601 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:33:09.601 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:33:09.601 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:33:09.601 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:33:09.601 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:33:09.601 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:33:09.601 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:33:09.602 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:33:09.602 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:33:09.602 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:33:09.602 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:33:09.602 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:33:09.602 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:33:09.602 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:33:09.602 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:33:09.602 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:33:09.602 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:33:09.602 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:33:09.625 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:33:09.625 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:33:09.625 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:33:09.625 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:33:09.753 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:33:09.754 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:33:09.754 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:33:09.755 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:33:09.755 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:33:09.755 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:33:09.755 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:33:09.755 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:33:09.755 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:33:09.755 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:33:09.756 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:33:09.756 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:33:09.756 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:33:09.757 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:33:09.757 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:33:09.757 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:33:09.808 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 3 2024-04-06T09:33:09.808 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 6 2024-04-06T09:33:09.808 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:33:09.808 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=25, tag_tid=3, entry_tid=1], [object_number=24, tag_tid=3, entry_tid=0], [object_number=27, tag_tid=2, entry_tid=27223], [object_number=26, tag_tid=2, entry_tid=27222]]], state=connected] 2024-04-06T09:33:09.808 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=12, tag_tid=2, entry_tid=15264], [object_number=15, tag_tid=2, entry_tid=15263], [object_number=14, tag_tid=2, entry_tid=15262], [object_number=13, tag_tid=2, entry_tid=15261]]], state=connected] 2024-04-06T09:33:09.812 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:33:09.813 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=25, tag_tid=3, entry_tid=1' 2024-04-06T09:33:09.814 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:33:09.814 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:33:09.815 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:33:09.815 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:33:09.815 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:33:09.815 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:33:09.815 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:33:09.815 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:33:09.815 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:33:09.815 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:33:09.815 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:33:09.815 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:33:09.816 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:33:09.816 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:33:09.816 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:33:09.861 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 3 2024-04-06T09:33:09.861 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 6 2024-04-06T09:33:09.861 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:33:09.861 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=25, tag_tid=3, entry_tid=1], [object_number=24, tag_tid=3, entry_tid=0], [object_number=27, tag_tid=2, entry_tid=27223], [object_number=26, tag_tid=2, entry_tid=27222]]], state=connected] 2024-04-06T09:33:09.861 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=12, tag_tid=2, entry_tid=15264], [object_number=15, tag_tid=2, entry_tid=15263], [object_number=14, tag_tid=2, entry_tid=15262], [object_number=13, tag_tid=2, entry_tid=15261]]], state=connected] 2024-04-06T09:33:09.865 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:33:09.867 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=12, tag_tid=2, entry_tid=15264' 2024-04-06T09:33:09.867 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=25, tag_tid=3, entry_tid=1' -a 'object_number=25, tag_tid=3, entry_tid=1' = 'object_number=12, tag_tid=2, entry_tid=15264' 2024-04-06T09:33:09.867 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=12, tag_tid=2, entry_tid=15264' '!=' 'object_number=12, tag_tid=2, entry_tid=15168' 2024-04-06T09:33:09.867 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:33:09.867 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=12, tag_tid=2, entry_tid=15264' = 'object_number=12, tag_tid=2, entry_tid=15168' 2024-04-06T09:33:09.867 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=12, tag_tid=2, entry_tid=15264' 2024-04-06T09:33:09.867 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=25, tag_tid=3, entry_tid=1' 2024-04-06T09:33:09.867 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:33:09.867 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:33:09.868 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=3 2024-04-06T09:33:09.869 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=12, tag_tid=2, entry_tid=15264' 2024-04-06T09:33:09.869 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:33:09.869 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:33:09.870 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=2 2024-04-06T09:33:09.870 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=25, tag_tid=3, entry_tid=1' 2024-04-06T09:33:09.870 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:33:09.870 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:33:09.871 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:33:09.872 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=12, tag_tid=2, entry_tid=15264' 2024-04-06T09:33:09.872 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:33:09.872 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:33:09.873 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=15264 2024-04-06T09:33:09.873 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 3 = 2 -a 1 -le 15264 2024-04-06T09:33:09.873 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:33:09.873 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:33:09.873 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:33:10.075 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:33:10.075 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:33:10.075 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:33:10.075 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:33:10.076 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:33:10.076 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:33:10.076 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:33:10.076 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:33:10.076 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:33:10.076 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:33:10.076 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:33:10.076 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:33:10.076 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:33:10.076 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:33:10.076 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:33:10.077 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:33:10.077 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:33:10.080 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:33:10.080 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:33:10.080 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:33:10.080 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:33:10.080 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:33:10.080 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:33:10.080 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:33:10.081 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:33:10.081 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:33:10.081 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:33:10.081 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:33:10.081 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:33:10.081 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:33:10.081 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:33:10.081 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:33:10.081 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:33:10.081 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:33:10.081 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:33:10.081 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:33:10.081 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:33:10.081 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:33:10.082 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:33:10.082 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:33:10.082 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:33:10.082 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:33:10.082 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:33:10.082 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:33:10.082 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:33:10.082 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:33:10.082 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:33:10.082 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:33:10.082 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:33:10.105 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:33:10.106 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:33:10.106 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:33:10.106 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:33:10.234 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:33:10.236 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:33:10.236 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:33:10.236 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:33:10.236 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:33:10.236 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:33:10.236 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:33:10.236 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:33:10.236 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:33:10.237 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:33:10.237 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:33:10.237 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:33:10.237 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:33:10.238 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:33:10.238 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:33:10.239 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:33:10.289 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 3 2024-04-06T09:33:10.289 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 6 2024-04-06T09:33:10.289 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:33:10.289 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=25, tag_tid=3, entry_tid=1], [object_number=24, tag_tid=3, entry_tid=0], [object_number=27, tag_tid=2, entry_tid=27223], [object_number=26, tag_tid=2, entry_tid=27222]]], state=connected] 2024-04-06T09:33:10.289 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=13, tag_tid=2, entry_tid=15357], [object_number=12, tag_tid=2, entry_tid=15356], [object_number=15, tag_tid=2, entry_tid=15355], [object_number=14, tag_tid=2, entry_tid=15354]]], state=connected] 2024-04-06T09:33:10.292 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:33:10.294 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=25, tag_tid=3, entry_tid=1' 2024-04-06T09:33:10.295 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:33:10.295 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:33:10.295 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:33:10.295 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:33:10.295 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:33:10.295 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:33:10.295 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:33:10.295 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:33:10.295 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:33:10.296 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:33:10.296 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:33:10.296 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:33:10.296 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:33:10.296 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:33:10.297 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:33:10.342 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 3 2024-04-06T09:33:10.343 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 6 2024-04-06T09:33:10.343 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:33:10.343 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=25, tag_tid=3, entry_tid=1], [object_number=24, tag_tid=3, entry_tid=0], [object_number=27, tag_tid=2, entry_tid=27223], [object_number=26, tag_tid=2, entry_tid=27222]]], state=connected] 2024-04-06T09:33:10.343 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=13, tag_tid=2, entry_tid=15357], [object_number=12, tag_tid=2, entry_tid=15356], [object_number=15, tag_tid=2, entry_tid=15355], [object_number=14, tag_tid=2, entry_tid=15354]]], state=connected] 2024-04-06T09:33:10.345 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:33:10.348 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=13, tag_tid=2, entry_tid=15357' 2024-04-06T09:33:10.348 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=25, tag_tid=3, entry_tid=1' -a 'object_number=25, tag_tid=3, entry_tid=1' = 'object_number=13, tag_tid=2, entry_tid=15357' 2024-04-06T09:33:10.348 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=13, tag_tid=2, entry_tid=15357' '!=' 'object_number=12, tag_tid=2, entry_tid=15264' 2024-04-06T09:33:10.348 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:33:10.348 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=13, tag_tid=2, entry_tid=15357' = 'object_number=12, tag_tid=2, entry_tid=15264' 2024-04-06T09:33:10.348 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=13, tag_tid=2, entry_tid=15357' 2024-04-06T09:33:10.348 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=25, tag_tid=3, entry_tid=1' 2024-04-06T09:33:10.348 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:33:10.348 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:33:10.349 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=3 2024-04-06T09:33:10.349 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=13, tag_tid=2, entry_tid=15357' 2024-04-06T09:33:10.349 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:33:10.350 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:33:10.350 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=2 2024-04-06T09:33:10.351 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=25, tag_tid=3, entry_tid=1' 2024-04-06T09:33:10.351 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:33:10.351 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:33:10.352 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:33:10.353 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=13, tag_tid=2, entry_tid=15357' 2024-04-06T09:33:10.353 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:33:10.353 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:33:10.354 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=15357 2024-04-06T09:33:10.354 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 3 = 2 -a 1 -le 15357 2024-04-06T09:33:10.354 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:33:10.354 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:33:10.354 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:33:10.556 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:33:10.557 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:33:10.557 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:33:10.557 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:33:10.557 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:33:10.557 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:33:10.557 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:33:10.557 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:33:10.557 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:33:10.557 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:33:10.557 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:33:10.557 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:33:10.557 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:33:10.557 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:33:10.558 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:33:10.558 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:33:10.558 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:33:10.561 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:33:10.561 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:33:10.561 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:33:10.561 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:33:10.561 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:33:10.561 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:33:10.562 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:33:10.562 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:33:10.562 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:33:10.562 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:33:10.562 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:33:10.562 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:33:10.562 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:33:10.562 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:33:10.562 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:33:10.562 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:33:10.562 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:33:10.562 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:33:10.562 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:33:10.562 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:33:10.562 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:33:10.563 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:33:10.563 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:33:10.563 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:33:10.563 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:33:10.563 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:33:10.563 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:33:10.563 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:33:10.563 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:33:10.563 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:33:10.563 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:33:10.563 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:33:10.586 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:33:10.586 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:33:10.586 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:33:10.586 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:33:10.712 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:33:10.714 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:33:10.714 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:33:10.714 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:33:10.714 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:33:10.714 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:33:10.714 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:33:10.714 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:33:10.715 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:33:10.715 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:33:10.715 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:33:10.715 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:33:10.715 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:33:10.716 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:33:10.717 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:33:10.717 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:33:10.767 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 3 2024-04-06T09:33:10.767 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 6 2024-04-06T09:33:10.767 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:33:10.768 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=25, tag_tid=3, entry_tid=1], [object_number=24, tag_tid=3, entry_tid=0], [object_number=27, tag_tid=2, entry_tid=27223], [object_number=26, tag_tid=2, entry_tid=27222]]], state=connected] 2024-04-06T09:33:10.768 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=15, tag_tid=2, entry_tid=15451], [object_number=14, tag_tid=2, entry_tid=15450], [object_number=13, tag_tid=2, entry_tid=15449], [object_number=12, tag_tid=2, entry_tid=15448]]], state=connected] 2024-04-06T09:33:10.771 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:33:10.774 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=25, tag_tid=3, entry_tid=1' 2024-04-06T09:33:10.774 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:33:10.774 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:33:10.774 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:33:10.774 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:33:10.774 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:33:10.774 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:33:10.774 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:33:10.774 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:33:10.774 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:33:10.775 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:33:10.775 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:33:10.775 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:33:10.775 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:33:10.776 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:33:10.776 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:33:10.819 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 3 2024-04-06T09:33:10.820 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 6 2024-04-06T09:33:10.820 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:33:10.820 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=25, tag_tid=3, entry_tid=1], [object_number=24, tag_tid=3, entry_tid=0], [object_number=27, tag_tid=2, entry_tid=27223], [object_number=26, tag_tid=2, entry_tid=27222]]], state=connected] 2024-04-06T09:33:10.820 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=15, tag_tid=2, entry_tid=15451], [object_number=14, tag_tid=2, entry_tid=15450], [object_number=13, tag_tid=2, entry_tid=15449], [object_number=12, tag_tid=2, entry_tid=15448]]], state=connected] 2024-04-06T09:33:10.823 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:33:10.825 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=15, tag_tid=2, entry_tid=15451' 2024-04-06T09:33:10.825 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=25, tag_tid=3, entry_tid=1' -a 'object_number=25, tag_tid=3, entry_tid=1' = 'object_number=15, tag_tid=2, entry_tid=15451' 2024-04-06T09:33:10.825 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=15, tag_tid=2, entry_tid=15451' '!=' 'object_number=13, tag_tid=2, entry_tid=15357' 2024-04-06T09:33:10.825 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:33:10.826 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=15, tag_tid=2, entry_tid=15451' = 'object_number=13, tag_tid=2, entry_tid=15357' 2024-04-06T09:33:10.826 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=15, tag_tid=2, entry_tid=15451' 2024-04-06T09:33:10.826 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=25, tag_tid=3, entry_tid=1' 2024-04-06T09:33:10.826 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:33:10.826 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:33:10.826 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=3 2024-04-06T09:33:10.827 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=15, tag_tid=2, entry_tid=15451' 2024-04-06T09:33:10.827 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:33:10.827 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:33:10.828 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=2 2024-04-06T09:33:10.829 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=25, tag_tid=3, entry_tid=1' 2024-04-06T09:33:10.829 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:33:10.829 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:33:10.830 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:33:10.831 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=15, tag_tid=2, entry_tid=15451' 2024-04-06T09:33:10.831 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:33:10.831 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:33:10.832 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=15451 2024-04-06T09:33:10.832 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 3 = 2 -a 1 -le 15451 2024-04-06T09:33:10.832 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:33:10.832 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:33:10.832 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:33:11.034 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:33:11.034 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:33:11.034 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:33:11.034 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:33:11.034 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:33:11.034 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:33:11.034 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:33:11.034 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:33:11.034 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:33:11.035 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:33:11.035 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:33:11.035 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:33:11.035 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:33:11.035 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:33:11.035 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:33:11.036 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:33:11.036 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:33:11.039 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:33:11.039 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:33:11.039 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:33:11.039 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:33:11.039 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:33:11.039 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:33:11.040 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:33:11.040 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:33:11.040 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:33:11.040 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:33:11.040 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:33:11.040 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:33:11.040 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:33:11.040 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:33:11.040 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:33:11.040 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:33:11.040 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:33:11.040 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:33:11.041 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:33:11.041 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:33:11.041 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:33:11.041 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:33:11.041 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:33:11.041 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:33:11.041 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:33:11.041 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:33:11.041 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:33:11.041 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:33:11.041 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:33:11.042 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:33:11.042 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:33:11.042 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:33:11.065 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:33:11.066 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:33:11.066 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:33:11.066 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:33:11.199 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:33:11.200 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:33:11.200 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:33:11.200 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:33:11.200 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:33:11.200 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:33:11.200 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:33:11.200 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:33:11.200 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:33:11.200 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:33:11.201 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:33:11.201 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:33:11.201 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:33:11.203 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:33:11.203 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:33:11.203 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:33:11.248 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 3 2024-04-06T09:33:11.248 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 6 2024-04-06T09:33:11.248 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:33:11.248 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=25, tag_tid=3, entry_tid=1], [object_number=24, tag_tid=3, entry_tid=0], [object_number=27, tag_tid=2, entry_tid=27223], [object_number=26, tag_tid=2, entry_tid=27222]]], state=connected] 2024-04-06T09:33:11.248 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=12, tag_tid=2, entry_tid=15548], [object_number=15, tag_tid=2, entry_tid=15547], [object_number=14, tag_tid=2, entry_tid=15546], [object_number=13, tag_tid=2, entry_tid=15545]]], state=connected] 2024-04-06T09:33:11.252 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:33:11.253 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=25, tag_tid=3, entry_tid=1' 2024-04-06T09:33:11.254 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:33:11.254 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:33:11.255 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:33:11.255 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:33:11.255 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:33:11.255 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:33:11.255 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:33:11.255 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:33:11.255 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:33:11.255 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:33:11.255 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:33:11.255 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:33:11.256 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:33:11.256 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:33:11.256 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:33:11.301 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 3 2024-04-06T09:33:11.301 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 6 2024-04-06T09:33:11.302 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:33:11.302 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=25, tag_tid=3, entry_tid=1], [object_number=24, tag_tid=3, entry_tid=0], [object_number=27, tag_tid=2, entry_tid=27223], [object_number=26, tag_tid=2, entry_tid=27222]]], state=connected] 2024-04-06T09:33:11.302 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=12, tag_tid=2, entry_tid=15548], [object_number=15, tag_tid=2, entry_tid=15547], [object_number=14, tag_tid=2, entry_tid=15546], [object_number=13, tag_tid=2, entry_tid=15545]]], state=connected] 2024-04-06T09:33:11.305 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:33:11.307 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=12, tag_tid=2, entry_tid=15548' 2024-04-06T09:33:11.307 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=25, tag_tid=3, entry_tid=1' -a 'object_number=25, tag_tid=3, entry_tid=1' = 'object_number=12, tag_tid=2, entry_tid=15548' 2024-04-06T09:33:11.307 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=12, tag_tid=2, entry_tid=15548' '!=' 'object_number=15, tag_tid=2, entry_tid=15451' 2024-04-06T09:33:11.307 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:33:11.307 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=12, tag_tid=2, entry_tid=15548' = 'object_number=15, tag_tid=2, entry_tid=15451' 2024-04-06T09:33:11.307 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=12, tag_tid=2, entry_tid=15548' 2024-04-06T09:33:11.307 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=25, tag_tid=3, entry_tid=1' 2024-04-06T09:33:11.307 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:33:11.308 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:33:11.308 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=3 2024-04-06T09:33:11.309 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=12, tag_tid=2, entry_tid=15548' 2024-04-06T09:33:11.309 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:33:11.309 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:33:11.310 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=2 2024-04-06T09:33:11.310 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=25, tag_tid=3, entry_tid=1' 2024-04-06T09:33:11.310 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:33:11.310 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:33:11.311 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:33:11.312 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=12, tag_tid=2, entry_tid=15548' 2024-04-06T09:33:11.312 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:33:11.312 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:33:11.313 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=15548 2024-04-06T09:33:11.313 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 3 = 2 -a 1 -le 15548 2024-04-06T09:33:11.313 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:33:11.313 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:33:11.313 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:33:11.516 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:33:11.516 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:33:11.516 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:33:11.516 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:33:11.516 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:33:11.516 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:33:11.516 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:33:11.516 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:33:11.516 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:33:11.516 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:33:11.516 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:33:11.517 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:33:11.517 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:33:11.517 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:33:11.517 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:33:11.517 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:33:11.517 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:33:11.520 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:33:11.520 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:33:11.520 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:33:11.520 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:33:11.520 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:33:11.520 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:33:11.521 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:33:11.521 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:33:11.521 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:33:11.521 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:33:11.521 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:33:11.521 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:33:11.521 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:33:11.521 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:33:11.521 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:33:11.521 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:33:11.522 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:33:11.523 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:33:11.523 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:33:11.523 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:33:11.523 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:33:11.523 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:33:11.523 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:33:11.523 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:33:11.523 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:33:11.523 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:33:11.523 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:33:11.523 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:33:11.523 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:33:11.523 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:33:11.523 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:33:11.523 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:33:11.549 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:33:11.549 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:33:11.549 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:33:11.549 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:33:11.679 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:33:11.680 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:33:11.680 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:33:11.680 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:33:11.680 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:33:11.680 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:33:11.680 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:33:11.680 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:33:11.680 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:33:11.680 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:33:11.681 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:33:11.681 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:33:11.681 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:33:11.682 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:33:11.683 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:33:11.683 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:33:11.733 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 3 2024-04-06T09:33:11.733 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 6 2024-04-06T09:33:11.733 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:33:11.733 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=25, tag_tid=3, entry_tid=1], [object_number=24, tag_tid=3, entry_tid=0], [object_number=27, tag_tid=2, entry_tid=27223], [object_number=26, tag_tid=2, entry_tid=27222]]], state=connected] 2024-04-06T09:33:11.733 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=15, tag_tid=2, entry_tid=15643], [object_number=14, tag_tid=2, entry_tid=15642], [object_number=13, tag_tid=2, entry_tid=15641], [object_number=12, tag_tid=2, entry_tid=15640]]], state=connected] 2024-04-06T09:33:11.737 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:33:11.738 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=25, tag_tid=3, entry_tid=1' 2024-04-06T09:33:11.739 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:33:11.739 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:33:11.739 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:33:11.740 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:33:11.740 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:33:11.740 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:33:11.740 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:33:11.740 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:33:11.740 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:33:11.740 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:33:11.740 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:33:11.740 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:33:11.741 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:33:11.741 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:33:11.741 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:33:11.786 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 3 2024-04-06T09:33:11.786 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 6 2024-04-06T09:33:11.786 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:33:11.786 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=25, tag_tid=3, entry_tid=1], [object_number=24, tag_tid=3, entry_tid=0], [object_number=27, tag_tid=2, entry_tid=27223], [object_number=26, tag_tid=2, entry_tid=27222]]], state=connected] 2024-04-06T09:33:11.786 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=15, tag_tid=2, entry_tid=15643], [object_number=14, tag_tid=2, entry_tid=15642], [object_number=13, tag_tid=2, entry_tid=15641], [object_number=12, tag_tid=2, entry_tid=15640]]], state=connected] 2024-04-06T09:33:11.790 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:33:11.792 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=15, tag_tid=2, entry_tid=15643' 2024-04-06T09:33:11.792 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=25, tag_tid=3, entry_tid=1' -a 'object_number=25, tag_tid=3, entry_tid=1' = 'object_number=15, tag_tid=2, entry_tid=15643' 2024-04-06T09:33:11.792 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=15, tag_tid=2, entry_tid=15643' '!=' 'object_number=12, tag_tid=2, entry_tid=15548' 2024-04-06T09:33:11.792 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:33:11.793 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=15, tag_tid=2, entry_tid=15643' = 'object_number=12, tag_tid=2, entry_tid=15548' 2024-04-06T09:33:11.793 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=15, tag_tid=2, entry_tid=15643' 2024-04-06T09:33:11.793 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=25, tag_tid=3, entry_tid=1' 2024-04-06T09:33:11.793 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:33:11.793 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:33:11.793 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=3 2024-04-06T09:33:11.794 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=15, tag_tid=2, entry_tid=15643' 2024-04-06T09:33:11.794 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:33:11.794 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:33:11.795 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=2 2024-04-06T09:33:11.796 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=25, tag_tid=3, entry_tid=1' 2024-04-06T09:33:11.796 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:33:11.796 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:33:11.797 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:33:11.797 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=15, tag_tid=2, entry_tid=15643' 2024-04-06T09:33:11.797 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:33:11.797 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:33:11.799 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=15643 2024-04-06T09:33:11.799 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 3 = 2 -a 1 -le 15643 2024-04-06T09:33:11.799 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:33:11.799 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:33:11.799 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:33:12.000 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:33:12.001 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:33:12.001 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:33:12.001 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:33:12.001 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:33:12.001 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:33:12.001 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:33:12.001 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:33:12.001 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:33:12.001 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:33:12.001 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:33:12.001 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:33:12.001 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:33:12.001 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:33:12.002 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:33:12.002 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:33:12.002 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:33:12.005 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:33:12.005 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:33:12.005 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:33:12.005 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:33:12.005 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:33:12.005 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:33:12.005 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:33:12.005 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:33:12.005 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:33:12.005 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:33:12.005 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:33:12.005 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:33:12.005 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:33:12.005 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:33:12.006 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:33:12.006 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:33:12.006 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:33:12.006 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:33:12.006 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:33:12.006 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:33:12.006 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:33:12.006 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:33:12.006 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:33:12.006 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:33:12.006 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:33:12.006 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:33:12.006 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:33:12.006 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:33:12.007 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:33:12.007 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:33:12.007 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:33:12.007 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:33:12.031 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:33:12.032 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:33:12.032 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:33:12.032 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:33:12.167 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:33:12.168 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:33:12.168 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:33:12.168 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:33:12.168 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:33:12.168 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:33:12.168 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:33:12.168 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:33:12.168 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:33:12.169 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:33:12.169 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:33:12.169 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:33:12.169 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:33:12.171 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:33:12.172 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:33:12.172 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:33:12.221 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 3 2024-04-06T09:33:12.221 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 6 2024-04-06T09:33:12.221 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:33:12.221 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=25, tag_tid=3, entry_tid=1], [object_number=24, tag_tid=3, entry_tid=0], [object_number=27, tag_tid=2, entry_tid=27223], [object_number=26, tag_tid=2, entry_tid=27222]]], state=connected] 2024-04-06T09:33:12.221 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=14, tag_tid=2, entry_tid=15762], [object_number=13, tag_tid=2, entry_tid=15761], [object_number=12, tag_tid=2, entry_tid=15760], [object_number=15, tag_tid=2, entry_tid=15759]]], state=connected] 2024-04-06T09:33:12.225 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:33:12.226 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=25, tag_tid=3, entry_tid=1' 2024-04-06T09:33:12.227 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:33:12.227 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:33:12.228 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:33:12.228 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:33:12.228 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:33:12.228 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:33:12.228 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:33:12.228 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:33:12.228 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:33:12.228 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:33:12.228 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:33:12.228 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:33:12.229 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:33:12.229 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:33:12.229 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:33:12.274 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 3 2024-04-06T09:33:12.274 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 6 2024-04-06T09:33:12.274 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:33:12.274 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=25, tag_tid=3, entry_tid=1], [object_number=24, tag_tid=3, entry_tid=0], [object_number=27, tag_tid=2, entry_tid=27223], [object_number=26, tag_tid=2, entry_tid=27222]]], state=connected] 2024-04-06T09:33:12.274 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=14, tag_tid=2, entry_tid=15762], [object_number=13, tag_tid=2, entry_tid=15761], [object_number=12, tag_tid=2, entry_tid=15760], [object_number=15, tag_tid=2, entry_tid=15759]]], state=connected] 2024-04-06T09:33:12.278 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:33:12.279 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=14, tag_tid=2, entry_tid=15762' 2024-04-06T09:33:12.280 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=25, tag_tid=3, entry_tid=1' -a 'object_number=25, tag_tid=3, entry_tid=1' = 'object_number=14, tag_tid=2, entry_tid=15762' 2024-04-06T09:33:12.280 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=14, tag_tid=2, entry_tid=15762' '!=' 'object_number=15, tag_tid=2, entry_tid=15643' 2024-04-06T09:33:12.280 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:33:12.280 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=14, tag_tid=2, entry_tid=15762' = 'object_number=15, tag_tid=2, entry_tid=15643' 2024-04-06T09:33:12.280 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=14, tag_tid=2, entry_tid=15762' 2024-04-06T09:33:12.280 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=25, tag_tid=3, entry_tid=1' 2024-04-06T09:33:12.281 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:33:12.281 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:33:12.281 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=3 2024-04-06T09:33:12.282 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=14, tag_tid=2, entry_tid=15762' 2024-04-06T09:33:12.282 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:33:12.282 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:33:12.283 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=2 2024-04-06T09:33:12.283 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=25, tag_tid=3, entry_tid=1' 2024-04-06T09:33:12.284 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:33:12.284 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:33:12.284 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:33:12.285 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=14, tag_tid=2, entry_tid=15762' 2024-04-06T09:33:12.285 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:33:12.285 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:33:12.287 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=15762 2024-04-06T09:33:12.287 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 3 = 2 -a 1 -le 15762 2024-04-06T09:33:12.287 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:33:12.287 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:33:12.287 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:33:12.489 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:33:12.489 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:33:12.489 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:33:12.489 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:33:12.489 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:33:12.489 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:33:12.489 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:33:12.489 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:33:12.489 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:33:12.489 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:33:12.489 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:33:12.489 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:33:12.490 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:33:12.490 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:33:12.490 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:33:12.490 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:33:12.490 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:33:12.493 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:33:12.493 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:33:12.493 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:33:12.493 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:33:12.493 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:33:12.493 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:33:12.493 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:33:12.493 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:33:12.493 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:33:12.493 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:33:12.493 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:33:12.493 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:33:12.494 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:33:12.494 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:33:12.494 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:33:12.494 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:33:12.495 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:33:12.495 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:33:12.495 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:33:12.495 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:33:12.495 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:33:12.495 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:33:12.496 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:33:12.496 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:33:12.496 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:33:12.496 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:33:12.496 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:33:12.496 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:33:12.496 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:33:12.496 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:33:12.496 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:33:12.496 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:33:12.519 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:33:12.520 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:33:12.520 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:33:12.520 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:33:12.652 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:33:12.653 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:33:12.653 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:33:12.653 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:33:12.653 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:33:12.653 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:33:12.653 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:33:12.654 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:33:12.654 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:33:12.654 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:33:12.654 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:33:12.654 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:33:12.654 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:33:12.656 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:33:12.657 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:33:12.657 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:33:12.707 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 3 2024-04-06T09:33:12.708 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 6 2024-04-06T09:33:12.708 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:33:12.708 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=25, tag_tid=3, entry_tid=1], [object_number=24, tag_tid=3, entry_tid=0], [object_number=27, tag_tid=2, entry_tid=27223], [object_number=26, tag_tid=2, entry_tid=27222]]], state=connected] 2024-04-06T09:33:12.708 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=15, tag_tid=2, entry_tid=15875], [object_number=14, tag_tid=2, entry_tid=15874], [object_number=13, tag_tid=2, entry_tid=15873], [object_number=12, tag_tid=2, entry_tid=15872]]], state=connected] 2024-04-06T09:33:12.711 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:33:12.712 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=25, tag_tid=3, entry_tid=1' 2024-04-06T09:33:12.713 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:33:12.714 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:33:12.714 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:33:12.714 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:33:12.714 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:33:12.714 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:33:12.714 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:33:12.714 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:33:12.714 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:33:12.714 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:33:12.714 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:33:12.714 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:33:12.715 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:33:12.715 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:33:12.715 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:33:12.759 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 3 2024-04-06T09:33:12.759 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 6 2024-04-06T09:33:12.760 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:33:12.760 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=25, tag_tid=3, entry_tid=1], [object_number=24, tag_tid=3, entry_tid=0], [object_number=27, tag_tid=2, entry_tid=27223], [object_number=26, tag_tid=2, entry_tid=27222]]], state=connected] 2024-04-06T09:33:12.760 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=15, tag_tid=2, entry_tid=15875], [object_number=14, tag_tid=2, entry_tid=15874], [object_number=13, tag_tid=2, entry_tid=15873], [object_number=12, tag_tid=2, entry_tid=15872]]], state=connected] 2024-04-06T09:33:12.763 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:33:12.765 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=15, tag_tid=2, entry_tid=15875' 2024-04-06T09:33:12.765 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=25, tag_tid=3, entry_tid=1' -a 'object_number=25, tag_tid=3, entry_tid=1' = 'object_number=15, tag_tid=2, entry_tid=15875' 2024-04-06T09:33:12.765 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=15, tag_tid=2, entry_tid=15875' '!=' 'object_number=14, tag_tid=2, entry_tid=15762' 2024-04-06T09:33:12.765 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:33:12.765 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=15, tag_tid=2, entry_tid=15875' = 'object_number=14, tag_tid=2, entry_tid=15762' 2024-04-06T09:33:12.765 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=15, tag_tid=2, entry_tid=15875' 2024-04-06T09:33:12.766 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=25, tag_tid=3, entry_tid=1' 2024-04-06T09:33:12.766 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:33:12.766 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:33:12.766 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=3 2024-04-06T09:33:12.767 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=15, tag_tid=2, entry_tid=15875' 2024-04-06T09:33:12.767 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:33:12.767 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:33:12.768 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=2 2024-04-06T09:33:12.768 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=25, tag_tid=3, entry_tid=1' 2024-04-06T09:33:12.768 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:33:12.769 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:33:12.769 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:33:12.770 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=15, tag_tid=2, entry_tid=15875' 2024-04-06T09:33:12.770 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:33:12.770 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:33:12.771 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=15875 2024-04-06T09:33:12.771 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 3 = 2 -a 1 -le 15875 2024-04-06T09:33:12.771 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:33:12.772 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:33:12.772 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:33:12.973 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:33:12.973 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:33:12.974 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:33:12.974 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:33:12.974 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:33:12.974 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:33:12.974 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:33:12.974 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:33:12.974 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:33:12.974 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:33:12.974 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:33:12.974 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:33:12.974 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:33:12.974 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:33:12.974 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:33:12.975 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:33:12.975 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:33:12.977 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:33:12.977 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:33:12.977 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:33:12.977 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:33:12.977 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:33:12.977 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:33:12.977 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:33:12.977 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:33:12.977 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:33:12.978 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:33:12.978 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:33:12.978 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:33:12.978 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:33:12.978 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:33:12.978 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:33:12.978 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:33:12.979 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:33:12.979 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:33:12.979 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:33:12.979 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:33:12.979 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:33:12.979 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:33:12.979 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:33:12.979 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:33:12.979 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:33:12.980 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:33:12.980 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:33:12.980 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:33:12.980 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:33:12.980 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:33:12.980 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:33:12.980 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:33:13.005 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:33:13.005 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:33:13.005 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:33:13.005 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:33:13.130 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:33:13.131 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:33:13.131 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:33:13.131 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:33:13.132 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:33:13.132 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:33:13.132 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:33:13.132 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:33:13.132 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:33:13.132 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:33:13.132 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:33:13.132 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:33:13.132 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:33:13.134 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:33:13.134 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:33:13.135 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:33:13.179 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 3 2024-04-06T09:33:13.179 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 6 2024-04-06T09:33:13.180 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:33:13.180 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=25, tag_tid=3, entry_tid=1], [object_number=24, tag_tid=3, entry_tid=0], [object_number=27, tag_tid=2, entry_tid=27223], [object_number=26, tag_tid=2, entry_tid=27222]]], state=connected] 2024-04-06T09:33:13.180 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=15, tag_tid=2, entry_tid=15983], [object_number=14, tag_tid=2, entry_tid=15982], [object_number=13, tag_tid=2, entry_tid=15981], [object_number=12, tag_tid=2, entry_tid=15980]]], state=connected] 2024-04-06T09:33:13.183 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:33:13.184 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=25, tag_tid=3, entry_tid=1' 2024-04-06T09:33:13.186 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:33:13.186 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:33:13.186 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:33:13.186 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:33:13.186 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:33:13.186 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:33:13.186 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:33:13.186 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:33:13.186 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:33:13.186 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:33:13.186 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:33:13.187 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:33:13.187 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:33:13.187 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:33:13.187 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:33:13.232 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 3 2024-04-06T09:33:13.232 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 6 2024-04-06T09:33:13.232 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:33:13.232 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=25, tag_tid=3, entry_tid=1], [object_number=24, tag_tid=3, entry_tid=0], [object_number=27, tag_tid=2, entry_tid=27223], [object_number=26, tag_tid=2, entry_tid=27222]]], state=connected] 2024-04-06T09:33:13.232 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=15, tag_tid=2, entry_tid=15983], [object_number=14, tag_tid=2, entry_tid=15982], [object_number=13, tag_tid=2, entry_tid=15981], [object_number=12, tag_tid=2, entry_tid=15980]]], state=connected] 2024-04-06T09:33:13.235 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:33:13.237 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=15, tag_tid=2, entry_tid=15983' 2024-04-06T09:33:13.237 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=25, tag_tid=3, entry_tid=1' -a 'object_number=25, tag_tid=3, entry_tid=1' = 'object_number=15, tag_tid=2, entry_tid=15983' 2024-04-06T09:33:13.237 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=15, tag_tid=2, entry_tid=15983' '!=' 'object_number=15, tag_tid=2, entry_tid=15875' 2024-04-06T09:33:13.237 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:33:13.237 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=15, tag_tid=2, entry_tid=15983' = 'object_number=15, tag_tid=2, entry_tid=15875' 2024-04-06T09:33:13.238 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=15, tag_tid=2, entry_tid=15983' 2024-04-06T09:33:13.238 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=25, tag_tid=3, entry_tid=1' 2024-04-06T09:33:13.238 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:33:13.238 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:33:13.238 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=3 2024-04-06T09:33:13.239 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=15, tag_tid=2, entry_tid=15983' 2024-04-06T09:33:13.239 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:33:13.239 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:33:13.240 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=2 2024-04-06T09:33:13.241 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=25, tag_tid=3, entry_tid=1' 2024-04-06T09:33:13.241 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:33:13.241 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:33:13.242 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:33:13.242 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=15, tag_tid=2, entry_tid=15983' 2024-04-06T09:33:13.242 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:33:13.242 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:33:13.244 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=15983 2024-04-06T09:33:13.244 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 3 = 2 -a 1 -le 15983 2024-04-06T09:33:13.244 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:33:13.244 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:33:13.244 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:33:13.446 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:33:13.446 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:33:13.446 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:33:13.446 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:33:13.446 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:33:13.446 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:33:13.446 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:33:13.446 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:33:13.447 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:33:13.447 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:33:13.447 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:33:13.447 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:33:13.447 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:33:13.447 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:33:13.447 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:33:13.447 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:33:13.448 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:33:13.451 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:33:13.451 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:33:13.451 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:33:13.451 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:33:13.451 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:33:13.451 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:33:13.451 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:33:13.451 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:33:13.451 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:33:13.451 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:33:13.451 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:33:13.451 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:33:13.451 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:33:13.452 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:33:13.452 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:33:13.452 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:33:13.452 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:33:13.452 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:33:13.452 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:33:13.452 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:33:13.452 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:33:13.452 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:33:13.452 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:33:13.452 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:33:13.452 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:33:13.452 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:33:13.452 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:33:13.453 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:33:13.453 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:33:13.453 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:33:13.453 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:33:13.453 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:33:13.477 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:33:13.477 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:33:13.477 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:33:13.477 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:33:13.605 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:33:13.607 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:33:13.607 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:33:13.607 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:33:13.607 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:33:13.607 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:33:13.607 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:33:13.607 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:33:13.607 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:33:13.607 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:33:13.607 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:33:13.608 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:33:13.608 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:33:13.608 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:33:13.609 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:33:13.609 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:33:13.654 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 3 2024-04-06T09:33:13.654 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 6 2024-04-06T09:33:13.654 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:33:13.654 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=25, tag_tid=3, entry_tid=1], [object_number=24, tag_tid=3, entry_tid=0], [object_number=27, tag_tid=2, entry_tid=27223], [object_number=26, tag_tid=2, entry_tid=27222]]], state=connected] 2024-04-06T09:33:13.654 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=12, tag_tid=2, entry_tid=16088], [object_number=15, tag_tid=2, entry_tid=16087], [object_number=14, tag_tid=2, entry_tid=16086], [object_number=13, tag_tid=2, entry_tid=16085]]], state=connected] 2024-04-06T09:33:13.657 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:33:13.659 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=25, tag_tid=3, entry_tid=1' 2024-04-06T09:33:13.660 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:33:13.660 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:33:13.660 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:33:13.661 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:33:13.661 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:33:13.661 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:33:13.661 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:33:13.661 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:33:13.661 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:33:13.661 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:33:13.661 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:33:13.661 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:33:13.662 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:33:13.662 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:33:13.662 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:33:13.706 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 3 2024-04-06T09:33:13.706 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 6 2024-04-06T09:33:13.706 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:33:13.706 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=25, tag_tid=3, entry_tid=1], [object_number=24, tag_tid=3, entry_tid=0], [object_number=27, tag_tid=2, entry_tid=27223], [object_number=26, tag_tid=2, entry_tid=27222]]], state=connected] 2024-04-06T09:33:13.706 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=12, tag_tid=2, entry_tid=16088], [object_number=15, tag_tid=2, entry_tid=16087], [object_number=14, tag_tid=2, entry_tid=16086], [object_number=13, tag_tid=2, entry_tid=16085]]], state=connected] 2024-04-06T09:33:13.709 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:33:13.711 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=12, tag_tid=2, entry_tid=16088' 2024-04-06T09:33:13.711 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=25, tag_tid=3, entry_tid=1' -a 'object_number=25, tag_tid=3, entry_tid=1' = 'object_number=12, tag_tid=2, entry_tid=16088' 2024-04-06T09:33:13.711 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=12, tag_tid=2, entry_tid=16088' '!=' 'object_number=15, tag_tid=2, entry_tid=15983' 2024-04-06T09:33:13.711 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:33:13.711 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=12, tag_tid=2, entry_tid=16088' = 'object_number=15, tag_tid=2, entry_tid=15983' 2024-04-06T09:33:13.711 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=12, tag_tid=2, entry_tid=16088' 2024-04-06T09:33:13.712 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=25, tag_tid=3, entry_tid=1' 2024-04-06T09:33:13.712 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:33:13.712 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:33:13.712 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=3 2024-04-06T09:33:13.713 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=12, tag_tid=2, entry_tid=16088' 2024-04-06T09:33:13.713 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:33:13.713 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:33:13.714 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=2 2024-04-06T09:33:13.715 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=25, tag_tid=3, entry_tid=1' 2024-04-06T09:33:13.715 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:33:13.715 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:33:13.715 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:33:13.716 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=12, tag_tid=2, entry_tid=16088' 2024-04-06T09:33:13.716 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:33:13.716 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:33:13.718 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=16088 2024-04-06T09:33:13.718 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 3 = 2 -a 1 -le 16088 2024-04-06T09:33:13.718 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:33:13.718 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:33:13.718 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:33:13.920 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:33:13.920 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:33:13.920 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:33:13.920 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:33:13.920 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:33:13.920 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:33:13.921 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:33:13.921 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:33:13.921 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:33:13.921 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:33:13.921 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:33:13.921 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:33:13.921 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:33:13.921 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:33:13.921 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:33:13.924 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:33:13.924 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:33:13.924 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:33:13.924 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:33:13.924 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:33:13.924 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:33:13.924 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:33:13.924 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:33:13.924 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:33:13.924 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:33:13.924 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:33:13.924 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:33:13.925 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:33:13.925 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:33:13.925 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:33:13.925 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:33:13.925 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:33:13.925 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:33:13.925 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:33:13.925 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:33:13.925 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:33:13.925 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:33:13.925 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:33:13.925 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:33:13.925 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:33:13.925 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:33:13.926 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:33:13.926 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:33:13.926 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:33:13.926 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:33:13.926 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:33:13.926 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:33:13.926 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:33:13.926 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:33:13.951 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:33:13.951 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:33:13.951 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:33:13.951 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:33:14.077 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:33:14.079 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:33:14.079 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:33:14.079 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:33:14.079 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:33:14.079 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:33:14.079 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:33:14.079 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:33:14.079 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:33:14.079 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:33:14.080 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:33:14.080 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:33:14.080 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:33:14.081 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:33:14.081 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:33:14.081 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:33:14.133 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 3 2024-04-06T09:33:14.133 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 6 2024-04-06T09:33:14.133 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:33:14.133 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=25, tag_tid=3, entry_tid=1], [object_number=24, tag_tid=3, entry_tid=0], [object_number=27, tag_tid=2, entry_tid=27223], [object_number=26, tag_tid=2, entry_tid=27222]]], state=connected] 2024-04-06T09:33:14.133 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=12, tag_tid=2, entry_tid=16200], [object_number=15, tag_tid=2, entry_tid=16199], [object_number=14, tag_tid=2, entry_tid=16198], [object_number=13, tag_tid=2, entry_tid=16197]]], state=connected] 2024-04-06T09:33:14.136 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:33:14.138 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=25, tag_tid=3, entry_tid=1' 2024-04-06T09:33:14.139 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:33:14.139 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:33:14.139 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:33:14.139 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:33:14.139 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:33:14.139 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:33:14.140 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:33:14.140 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:33:14.140 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:33:14.140 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:33:14.140 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:33:14.140 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:33:14.141 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:33:14.141 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:33:14.141 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:33:14.185 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 3 2024-04-06T09:33:14.185 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 6 2024-04-06T09:33:14.185 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:33:14.185 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=25, tag_tid=3, entry_tid=1], [object_number=24, tag_tid=3, entry_tid=0], [object_number=27, tag_tid=2, entry_tid=27223], [object_number=26, tag_tid=2, entry_tid=27222]]], state=connected] 2024-04-06T09:33:14.185 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=12, tag_tid=2, entry_tid=16200], [object_number=15, tag_tid=2, entry_tid=16199], [object_number=14, tag_tid=2, entry_tid=16198], [object_number=13, tag_tid=2, entry_tid=16197]]], state=connected] 2024-04-06T09:33:14.189 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:33:14.191 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=12, tag_tid=2, entry_tid=16200' 2024-04-06T09:33:14.191 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=25, tag_tid=3, entry_tid=1' -a 'object_number=25, tag_tid=3, entry_tid=1' = 'object_number=12, tag_tid=2, entry_tid=16200' 2024-04-06T09:33:14.191 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=12, tag_tid=2, entry_tid=16200' '!=' 'object_number=12, tag_tid=2, entry_tid=16088' 2024-04-06T09:33:14.191 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:33:14.191 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=12, tag_tid=2, entry_tid=16200' = 'object_number=12, tag_tid=2, entry_tid=16088' 2024-04-06T09:33:14.191 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=12, tag_tid=2, entry_tid=16200' 2024-04-06T09:33:14.191 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=25, tag_tid=3, entry_tid=1' 2024-04-06T09:33:14.191 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:33:14.191 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:33:14.192 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=3 2024-04-06T09:33:14.193 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=12, tag_tid=2, entry_tid=16200' 2024-04-06T09:33:14.193 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:33:14.193 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:33:14.194 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=2 2024-04-06T09:33:14.194 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=25, tag_tid=3, entry_tid=1' 2024-04-06T09:33:14.194 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:33:14.194 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:33:14.195 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:33:14.196 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=12, tag_tid=2, entry_tid=16200' 2024-04-06T09:33:14.196 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:33:14.196 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:33:14.197 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=16200 2024-04-06T09:33:14.197 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 3 = 2 -a 1 -le 16200 2024-04-06T09:33:14.197 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:33:14.197 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:33:14.198 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:33:14.399 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:33:14.399 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:33:14.399 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:33:14.399 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:33:14.400 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:33:14.400 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:33:14.400 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:33:14.400 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:33:14.400 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:33:14.400 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:33:14.400 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:33:14.400 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:33:14.400 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:33:14.400 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:33:14.400 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:33:14.401 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:33:14.401 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:33:14.403 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:33:14.404 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:33:14.404 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:33:14.404 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:33:14.404 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:33:14.404 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:33:14.404 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:33:14.404 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:33:14.404 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:33:14.404 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:33:14.404 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:33:14.404 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:33:14.404 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:33:14.404 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:33:14.405 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:33:14.405 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:33:14.405 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:33:14.406 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:33:14.406 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:33:14.406 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:33:14.406 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:33:14.406 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:33:14.406 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:33:14.406 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:33:14.406 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:33:14.406 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:33:14.406 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:33:14.406 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:33:14.406 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:33:14.407 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:33:14.407 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:33:14.407 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:33:14.431 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:33:14.431 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:33:14.432 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:33:14.432 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:33:14.557 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:33:14.558 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:33:14.558 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:33:14.558 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:33:14.558 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:33:14.558 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:33:14.558 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:33:14.558 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:33:14.559 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:33:14.559 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:33:14.559 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:33:14.559 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:33:14.559 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:33:14.561 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:33:14.561 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:33:14.561 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:33:14.607 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 3 2024-04-06T09:33:14.607 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 6 2024-04-06T09:33:14.607 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:33:14.607 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=25, tag_tid=3, entry_tid=1], [object_number=24, tag_tid=3, entry_tid=0], [object_number=27, tag_tid=2, entry_tid=27223], [object_number=26, tag_tid=2, entry_tid=27222]]], state=connected] 2024-04-06T09:33:14.608 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=15, tag_tid=2, entry_tid=16315], [object_number=14, tag_tid=2, entry_tid=16314], [object_number=13, tag_tid=2, entry_tid=16313], [object_number=12, tag_tid=2, entry_tid=16312]]], state=connected] 2024-04-06T09:33:14.611 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:33:14.612 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=25, tag_tid=3, entry_tid=1' 2024-04-06T09:33:14.613 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:33:14.613 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:33:14.613 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:33:14.613 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:33:14.613 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:33:14.614 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:33:14.614 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:33:14.614 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:33:14.614 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:33:14.614 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:33:14.614 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:33:14.614 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:33:14.615 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:33:14.615 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:33:14.615 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:33:14.660 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 3 2024-04-06T09:33:14.660 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 6 2024-04-06T09:33:14.660 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:33:14.660 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=25, tag_tid=3, entry_tid=1], [object_number=24, tag_tid=3, entry_tid=0], [object_number=27, tag_tid=2, entry_tid=27223], [object_number=26, tag_tid=2, entry_tid=27222]]], state=connected] 2024-04-06T09:33:14.660 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=15, tag_tid=2, entry_tid=16315], [object_number=14, tag_tid=2, entry_tid=16314], [object_number=13, tag_tid=2, entry_tid=16313], [object_number=12, tag_tid=2, entry_tid=16312]]], state=connected] 2024-04-06T09:33:14.663 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:33:14.665 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=15, tag_tid=2, entry_tid=16315' 2024-04-06T09:33:14.665 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=25, tag_tid=3, entry_tid=1' -a 'object_number=25, tag_tid=3, entry_tid=1' = 'object_number=15, tag_tid=2, entry_tid=16315' 2024-04-06T09:33:14.665 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=15, tag_tid=2, entry_tid=16315' '!=' 'object_number=12, tag_tid=2, entry_tid=16200' 2024-04-06T09:33:14.665 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:33:14.665 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=15, tag_tid=2, entry_tid=16315' = 'object_number=12, tag_tid=2, entry_tid=16200' 2024-04-06T09:33:14.665 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=15, tag_tid=2, entry_tid=16315' 2024-04-06T09:33:14.666 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=25, tag_tid=3, entry_tid=1' 2024-04-06T09:33:14.666 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:33:14.666 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:33:14.667 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=3 2024-04-06T09:33:14.667 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=15, tag_tid=2, entry_tid=16315' 2024-04-06T09:33:14.667 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:33:14.667 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:33:14.668 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=2 2024-04-06T09:33:14.669 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=25, tag_tid=3, entry_tid=1' 2024-04-06T09:33:14.669 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:33:14.669 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:33:14.670 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:33:14.671 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=15, tag_tid=2, entry_tid=16315' 2024-04-06T09:33:14.671 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:33:14.671 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:33:14.672 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=16315 2024-04-06T09:33:14.672 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 3 = 2 -a 1 -le 16315 2024-04-06T09:33:14.672 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:33:14.672 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:33:14.672 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:33:14.874 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:33:14.874 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:33:14.874 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:33:14.874 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:33:14.874 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:33:14.874 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:33:14.874 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:33:14.874 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:33:14.874 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:33:14.874 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:33:14.874 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:33:14.875 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:33:14.875 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:33:14.875 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:33:14.875 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:33:14.875 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:33:14.875 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:33:14.878 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:33:14.878 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:33:14.878 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:33:14.878 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:33:14.878 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:33:14.878 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:33:14.878 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:33:14.878 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:33:14.878 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:33:14.878 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:33:14.878 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:33:14.878 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:33:14.879 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:33:14.879 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:33:14.879 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:33:14.879 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:33:14.879 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:33:14.879 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:33:14.879 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:33:14.879 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:33:14.879 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:33:14.879 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:33:14.879 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:33:14.879 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:33:14.879 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:33:14.879 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:33:14.879 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:33:14.880 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:33:14.880 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:33:14.880 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:33:14.880 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:33:14.880 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:33:14.904 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:33:14.904 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:33:14.904 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:33:14.905 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:33:15.178 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:33:15.179 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:33:15.179 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:33:15.180 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:33:15.180 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:33:15.180 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:33:15.180 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:33:15.180 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:33:15.180 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:33:15.180 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:33:15.180 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:33:15.181 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:33:15.181 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:33:15.184 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:33:15.184 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:33:15.184 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:33:15.232 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 3 2024-04-06T09:33:15.232 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 6 2024-04-06T09:33:15.232 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:33:15.232 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=25, tag_tid=3, entry_tid=1], [object_number=24, tag_tid=3, entry_tid=0], [object_number=27, tag_tid=2, entry_tid=27223], [object_number=26, tag_tid=2, entry_tid=27222]]], state=connected] 2024-04-06T09:33:15.232 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=13, tag_tid=2, entry_tid=16413], [object_number=12, tag_tid=2, entry_tid=16412], [object_number=15, tag_tid=2, entry_tid=16411], [object_number=14, tag_tid=2, entry_tid=16410]]], state=connected] 2024-04-06T09:33:15.236 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:33:15.237 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=25, tag_tid=3, entry_tid=1' 2024-04-06T09:33:15.238 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:33:15.238 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:33:15.238 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:33:15.238 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:33:15.238 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:33:15.239 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:33:15.239 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:33:15.239 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:33:15.239 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:33:15.239 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:33:15.239 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:33:15.239 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:33:15.240 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:33:15.240 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:33:15.240 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:33:15.285 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 3 2024-04-06T09:33:15.285 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 6 2024-04-06T09:33:15.285 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:33:15.285 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=25, tag_tid=3, entry_tid=1], [object_number=24, tag_tid=3, entry_tid=0], [object_number=27, tag_tid=2, entry_tid=27223], [object_number=26, tag_tid=2, entry_tid=27222]]], state=connected] 2024-04-06T09:33:15.285 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=13, tag_tid=2, entry_tid=16413], [object_number=12, tag_tid=2, entry_tid=16412], [object_number=15, tag_tid=2, entry_tid=16411], [object_number=14, tag_tid=2, entry_tid=16410]]], state=connected] 2024-04-06T09:33:15.289 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:33:15.291 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=13, tag_tid=2, entry_tid=16413' 2024-04-06T09:33:15.291 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=25, tag_tid=3, entry_tid=1' -a 'object_number=25, tag_tid=3, entry_tid=1' = 'object_number=13, tag_tid=2, entry_tid=16413' 2024-04-06T09:33:15.291 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=13, tag_tid=2, entry_tid=16413' '!=' 'object_number=15, tag_tid=2, entry_tid=16315' 2024-04-06T09:33:15.291 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:33:15.292 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=13, tag_tid=2, entry_tid=16413' = 'object_number=15, tag_tid=2, entry_tid=16315' 2024-04-06T09:33:15.292 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=13, tag_tid=2, entry_tid=16413' 2024-04-06T09:33:15.292 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=25, tag_tid=3, entry_tid=1' 2024-04-06T09:33:15.292 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:33:15.292 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:33:15.292 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=3 2024-04-06T09:33:15.293 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=13, tag_tid=2, entry_tid=16413' 2024-04-06T09:33:15.293 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:33:15.293 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:33:15.294 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=2 2024-04-06T09:33:15.295 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=25, tag_tid=3, entry_tid=1' 2024-04-06T09:33:15.295 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:33:15.295 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:33:15.296 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:33:15.297 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=13, tag_tid=2, entry_tid=16413' 2024-04-06T09:33:15.297 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:33:15.297 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:33:15.298 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=16413 2024-04-06T09:33:15.298 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 3 = 2 -a 1 -le 16413 2024-04-06T09:33:15.298 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:33:15.298 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:33:15.298 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:33:15.500 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:33:15.500 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:33:15.500 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:33:15.500 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:33:15.500 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:33:15.500 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:33:15.500 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:33:15.500 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:33:15.500 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:33:15.500 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:33:15.500 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:33:15.501 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:33:15.501 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:33:15.501 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:33:15.501 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:33:15.501 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:33:15.502 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:33:15.506 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:33:15.506 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:33:15.506 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:33:15.506 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:33:15.506 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:33:15.506 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:33:15.506 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:33:15.506 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:33:15.506 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:33:15.506 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:33:15.507 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:33:15.507 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:33:15.507 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:33:15.507 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:33:15.507 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:33:15.507 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:33:15.507 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:33:15.507 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:33:15.507 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:33:15.507 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:33:15.507 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:33:15.507 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:33:15.507 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:33:15.507 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:33:15.507 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:33:15.508 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:33:15.508 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:33:15.508 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:33:15.508 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:33:15.508 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:33:15.508 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:33:15.508 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:33:15.531 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:33:15.531 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:33:15.531 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:33:15.531 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:33:15.661 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:33:15.661 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:33:15.662 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:33:15.662 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:33:15.662 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:33:15.662 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:33:15.662 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:33:15.662 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:33:15.662 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:33:15.662 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:33:15.663 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:33:15.663 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:33:15.663 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:33:15.664 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:33:15.664 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:33:15.665 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:33:15.709 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 3 2024-04-06T09:33:15.709 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 6 2024-04-06T09:33:15.709 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:33:15.709 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=25, tag_tid=3, entry_tid=1], [object_number=24, tag_tid=3, entry_tid=0], [object_number=27, tag_tid=2, entry_tid=27223], [object_number=26, tag_tid=2, entry_tid=27222]]], state=connected] 2024-04-06T09:33:15.709 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=15, tag_tid=2, entry_tid=16519], [object_number=14, tag_tid=2, entry_tid=16518], [object_number=13, tag_tid=2, entry_tid=16517], [object_number=12, tag_tid=2, entry_tid=16516]]], state=connected] 2024-04-06T09:33:15.733 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:33:15.734 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=25, tag_tid=3, entry_tid=1' 2024-04-06T09:33:15.735 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:33:15.735 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:33:15.736 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:33:15.736 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:33:15.736 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:33:15.736 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:33:15.736 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:33:15.736 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:33:15.736 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:33:15.736 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:33:15.736 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:33:15.736 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:33:15.737 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:33:15.737 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:33:15.737 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:33:15.782 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 3 2024-04-06T09:33:15.782 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 6 2024-04-06T09:33:15.782 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:33:15.782 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=25, tag_tid=3, entry_tid=1], [object_number=24, tag_tid=3, entry_tid=0], [object_number=27, tag_tid=2, entry_tid=27223], [object_number=26, tag_tid=2, entry_tid=27222]]], state=connected] 2024-04-06T09:33:15.783 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=15, tag_tid=2, entry_tid=16519], [object_number=14, tag_tid=2, entry_tid=16518], [object_number=13, tag_tid=2, entry_tid=16517], [object_number=12, tag_tid=2, entry_tid=16516]]], state=connected] 2024-04-06T09:33:15.786 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:33:15.789 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=15, tag_tid=2, entry_tid=16519' 2024-04-06T09:33:15.789 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=25, tag_tid=3, entry_tid=1' -a 'object_number=25, tag_tid=3, entry_tid=1' = 'object_number=15, tag_tid=2, entry_tid=16519' 2024-04-06T09:33:15.789 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=15, tag_tid=2, entry_tid=16519' '!=' 'object_number=13, tag_tid=2, entry_tid=16413' 2024-04-06T09:33:15.789 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:33:15.789 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=15, tag_tid=2, entry_tid=16519' = 'object_number=13, tag_tid=2, entry_tid=16413' 2024-04-06T09:33:15.789 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=15, tag_tid=2, entry_tid=16519' 2024-04-06T09:33:15.789 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=25, tag_tid=3, entry_tid=1' 2024-04-06T09:33:15.789 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:33:15.789 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:33:15.790 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=3 2024-04-06T09:33:15.791 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=15, tag_tid=2, entry_tid=16519' 2024-04-06T09:33:15.791 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:33:15.791 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:33:15.791 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=2 2024-04-06T09:33:15.792 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=25, tag_tid=3, entry_tid=1' 2024-04-06T09:33:15.792 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:33:15.792 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:33:15.793 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:33:15.794 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=15, tag_tid=2, entry_tid=16519' 2024-04-06T09:33:15.794 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:33:15.794 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:33:15.795 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=16519 2024-04-06T09:33:15.795 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 3 = 2 -a 1 -le 16519 2024-04-06T09:33:15.795 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:33:15.795 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:33:15.795 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:33:15.997 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:33:15.997 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:33:15.997 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:33:15.997 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:33:15.997 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:33:15.997 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:33:15.997 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:33:15.998 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:33:15.998 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:33:15.998 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:33:15.998 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:33:15.998 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:33:15.998 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:33:15.998 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:33:15.998 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:33:15.999 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:33:15.999 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:33:16.002 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:33:16.002 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:33:16.002 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:33:16.002 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:33:16.002 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:33:16.002 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:33:16.002 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:33:16.003 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:33:16.003 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:33:16.003 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:33:16.003 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:33:16.003 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:33:16.003 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:33:16.003 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:33:16.003 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:33:16.003 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:33:16.003 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:33:16.003 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:33:16.003 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:33:16.003 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:33:16.003 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:33:16.004 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:33:16.004 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:33:16.004 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:33:16.004 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:33:16.004 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:33:16.004 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:33:16.004 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:33:16.004 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:33:16.004 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:33:16.004 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:33:16.004 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:33:16.028 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:33:16.028 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:33:16.028 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:33:16.028 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:33:16.158 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:33:16.159 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:33:16.160 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:33:16.160 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:33:16.160 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:33:16.160 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:33:16.160 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:33:16.160 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:33:16.160 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:33:16.160 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:33:16.161 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:33:16.161 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:33:16.161 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:33:16.163 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:33:16.164 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:33:16.164 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:33:16.214 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 3 2024-04-06T09:33:16.214 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 6 2024-04-06T09:33:16.214 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:33:16.214 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=25, tag_tid=3, entry_tid=1], [object_number=24, tag_tid=3, entry_tid=0], [object_number=27, tag_tid=2, entry_tid=27223], [object_number=26, tag_tid=2, entry_tid=27222]]], state=connected] 2024-04-06T09:33:16.214 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=12, tag_tid=2, entry_tid=16628], [object_number=15, tag_tid=2, entry_tid=16627], [object_number=14, tag_tid=2, entry_tid=16626], [object_number=13, tag_tid=2, entry_tid=16625]]], state=connected] 2024-04-06T09:33:16.217 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:33:16.219 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=25, tag_tid=3, entry_tid=1' 2024-04-06T09:33:16.220 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:33:16.220 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:33:16.220 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:33:16.220 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:33:16.220 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:33:16.220 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:33:16.221 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:33:16.221 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:33:16.221 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:33:16.221 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:33:16.221 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:33:16.221 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:33:16.221 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:33:16.222 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:33:16.222 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:33:16.266 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 3 2024-04-06T09:33:16.267 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 6 2024-04-06T09:33:16.267 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:33:16.267 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=25, tag_tid=3, entry_tid=1], [object_number=24, tag_tid=3, entry_tid=0], [object_number=27, tag_tid=2, entry_tid=27223], [object_number=26, tag_tid=2, entry_tid=27222]]], state=connected] 2024-04-06T09:33:16.267 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=12, tag_tid=2, entry_tid=16628], [object_number=15, tag_tid=2, entry_tid=16627], [object_number=14, tag_tid=2, entry_tid=16626], [object_number=13, tag_tid=2, entry_tid=16625]]], state=connected] 2024-04-06T09:33:16.270 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:33:16.272 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=12, tag_tid=2, entry_tid=16628' 2024-04-06T09:33:16.272 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=25, tag_tid=3, entry_tid=1' -a 'object_number=25, tag_tid=3, entry_tid=1' = 'object_number=12, tag_tid=2, entry_tid=16628' 2024-04-06T09:33:16.272 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=12, tag_tid=2, entry_tid=16628' '!=' 'object_number=15, tag_tid=2, entry_tid=16519' 2024-04-06T09:33:16.272 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:33:16.272 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=12, tag_tid=2, entry_tid=16628' = 'object_number=15, tag_tid=2, entry_tid=16519' 2024-04-06T09:33:16.272 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=12, tag_tid=2, entry_tid=16628' 2024-04-06T09:33:16.272 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=25, tag_tid=3, entry_tid=1' 2024-04-06T09:33:16.272 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:33:16.273 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:33:16.273 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=3 2024-04-06T09:33:16.274 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=12, tag_tid=2, entry_tid=16628' 2024-04-06T09:33:16.274 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:33:16.274 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:33:16.275 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=2 2024-04-06T09:33:16.275 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=25, tag_tid=3, entry_tid=1' 2024-04-06T09:33:16.275 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:33:16.276 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:33:16.276 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:33:16.277 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=12, tag_tid=2, entry_tid=16628' 2024-04-06T09:33:16.277 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:33:16.277 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:33:16.278 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=16628 2024-04-06T09:33:16.278 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 3 = 2 -a 1 -le 16628 2024-04-06T09:33:16.278 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:33:16.279 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:33:16.279 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:33:16.480 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:33:16.480 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:33:16.480 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:33:16.480 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:33:16.480 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:33:16.481 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:33:16.481 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:33:16.481 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:33:16.481 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:33:16.481 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:33:16.481 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:33:16.481 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:33:16.481 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:33:16.481 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:33:16.481 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:33:16.482 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:33:16.482 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:33:16.485 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:33:16.485 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:33:16.485 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:33:16.485 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:33:16.485 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:33:16.486 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:33:16.486 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:33:16.486 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:33:16.486 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:33:16.486 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:33:16.486 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:33:16.486 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:33:16.486 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:33:16.486 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:33:16.486 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:33:16.486 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:33:16.486 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:33:16.486 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:33:16.486 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:33:16.486 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:33:16.487 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:33:16.487 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:33:16.487 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:33:16.487 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:33:16.487 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:33:16.487 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:33:16.487 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:33:16.487 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:33:16.487 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:33:16.487 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:33:16.487 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:33:16.487 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:33:16.511 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:33:16.512 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:33:16.513 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:33:16.513 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:33:16.640 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:33:16.641 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:33:16.642 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:33:16.642 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:33:16.642 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:33:16.642 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:33:16.642 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:33:16.642 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:33:16.642 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:33:16.642 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:33:16.642 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:33:16.643 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:33:16.643 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:33:16.644 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:33:16.645 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:33:16.645 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:33:16.689 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 3 2024-04-06T09:33:16.689 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 6 2024-04-06T09:33:16.689 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:33:16.689 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=25, tag_tid=3, entry_tid=1], [object_number=24, tag_tid=3, entry_tid=0], [object_number=27, tag_tid=2, entry_tid=27223], [object_number=26, tag_tid=2, entry_tid=27222]]], state=connected] 2024-04-06T09:33:16.689 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=15, tag_tid=2, entry_tid=16735], [object_number=14, tag_tid=2, entry_tid=16734], [object_number=13, tag_tid=2, entry_tid=16733], [object_number=12, tag_tid=2, entry_tid=16732]]], state=connected] 2024-04-06T09:33:16.693 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:33:16.695 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=25, tag_tid=3, entry_tid=1' 2024-04-06T09:33:16.695 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:33:16.695 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:33:16.695 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:33:16.695 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:33:16.695 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:33:16.695 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:33:16.696 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:33:16.696 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:33:16.696 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:33:16.696 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:33:16.696 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:33:16.696 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:33:16.696 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:33:16.696 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:33:16.697 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:33:16.741 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 3 2024-04-06T09:33:16.742 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 6 2024-04-06T09:33:16.742 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:33:16.742 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=25, tag_tid=3, entry_tid=1], [object_number=24, tag_tid=3, entry_tid=0], [object_number=27, tag_tid=2, entry_tid=27223], [object_number=26, tag_tid=2, entry_tid=27222]]], state=connected] 2024-04-06T09:33:16.742 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=15, tag_tid=2, entry_tid=16735], [object_number=14, tag_tid=2, entry_tid=16734], [object_number=13, tag_tid=2, entry_tid=16733], [object_number=12, tag_tid=2, entry_tid=16732]]], state=connected] 2024-04-06T09:33:16.745 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:33:16.747 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=15, tag_tid=2, entry_tid=16735' 2024-04-06T09:33:16.747 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=25, tag_tid=3, entry_tid=1' -a 'object_number=25, tag_tid=3, entry_tid=1' = 'object_number=15, tag_tid=2, entry_tid=16735' 2024-04-06T09:33:16.747 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=15, tag_tid=2, entry_tid=16735' '!=' 'object_number=12, tag_tid=2, entry_tid=16628' 2024-04-06T09:33:16.747 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:33:16.747 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=15, tag_tid=2, entry_tid=16735' = 'object_number=12, tag_tid=2, entry_tid=16628' 2024-04-06T09:33:16.747 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=15, tag_tid=2, entry_tid=16735' 2024-04-06T09:33:16.747 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=25, tag_tid=3, entry_tid=1' 2024-04-06T09:33:16.748 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:33:16.748 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:33:16.748 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=3 2024-04-06T09:33:16.749 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=15, tag_tid=2, entry_tid=16735' 2024-04-06T09:33:16.749 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:33:16.749 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:33:16.750 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=2 2024-04-06T09:33:16.750 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=25, tag_tid=3, entry_tid=1' 2024-04-06T09:33:16.750 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:33:16.751 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:33:16.751 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:33:16.752 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=15, tag_tid=2, entry_tid=16735' 2024-04-06T09:33:16.752 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:33:16.752 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:33:16.753 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=16735 2024-04-06T09:33:16.754 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 3 = 2 -a 1 -le 16735 2024-04-06T09:33:16.754 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:33:16.754 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:33:16.754 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:33:16.955 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:33:16.955 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:33:16.955 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:33:16.955 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:33:16.956 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:33:16.956 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:33:16.956 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:33:16.956 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:33:16.956 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:33:16.956 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:33:16.956 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:33:16.956 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:33:16.956 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:33:16.956 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:33:16.956 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:33:16.957 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:33:16.957 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:33:16.960 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:33:16.960 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:33:16.961 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:33:16.961 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:33:16.961 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:33:16.961 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:33:16.961 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:33:16.961 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:33:16.961 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:33:16.961 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:33:16.961 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:33:16.961 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:33:16.961 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:33:16.961 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:33:16.961 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:33:16.961 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:33:16.962 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:33:16.962 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:33:16.962 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:33:16.962 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:33:16.962 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:33:16.962 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:33:16.962 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:33:16.962 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:33:16.962 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:33:16.962 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:33:16.962 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:33:16.963 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:33:16.963 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:33:16.963 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:33:16.963 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:33:16.963 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:33:16.985 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:33:16.985 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:33:16.985 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:33:16.986 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:33:17.120 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:33:17.120 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:33:17.120 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:33:17.120 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:33:17.121 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:33:17.121 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:33:17.121 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:33:17.121 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:33:17.121 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:33:17.121 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:33:17.121 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:33:17.122 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:33:17.122 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:33:17.124 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:33:17.124 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:33:17.125 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:33:17.181 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 3 2024-04-06T09:33:17.181 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 6 2024-04-06T09:33:17.181 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:33:17.181 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=25, tag_tid=3, entry_tid=1], [object_number=24, tag_tid=3, entry_tid=0], [object_number=27, tag_tid=2, entry_tid=27223], [object_number=26, tag_tid=2, entry_tid=27222]]], state=connected] 2024-04-06T09:33:17.181 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=19, tag_tid=2, entry_tid=16795], [object_number=18, tag_tid=2, entry_tid=16794], [object_number=13, tag_tid=2, entry_tid=16793], [object_number=12, tag_tid=2, entry_tid=16792]]], state=connected] 2024-04-06T09:33:17.186 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:33:17.187 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=25, tag_tid=3, entry_tid=1' 2024-04-06T09:33:17.188 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:33:17.188 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:33:17.189 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:33:17.189 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:33:17.189 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:33:17.189 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:33:17.189 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:33:17.189 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:33:17.189 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:33:17.189 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:33:17.189 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:33:17.189 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:33:17.190 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:33:17.190 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:33:17.190 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:33:17.236 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 3 2024-04-06T09:33:17.236 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 6 2024-04-06T09:33:17.236 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:33:17.236 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=25, tag_tid=3, entry_tid=1], [object_number=24, tag_tid=3, entry_tid=0], [object_number=27, tag_tid=2, entry_tid=27223], [object_number=26, tag_tid=2, entry_tid=27222]]], state=connected] 2024-04-06T09:33:17.236 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=19, tag_tid=2, entry_tid=16795], [object_number=18, tag_tid=2, entry_tid=16794], [object_number=13, tag_tid=2, entry_tid=16793], [object_number=12, tag_tid=2, entry_tid=16792]]], state=connected] 2024-04-06T09:33:17.240 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:33:17.242 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=19, tag_tid=2, entry_tid=16795' 2024-04-06T09:33:17.242 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=25, tag_tid=3, entry_tid=1' -a 'object_number=25, tag_tid=3, entry_tid=1' = 'object_number=19, tag_tid=2, entry_tid=16795' 2024-04-06T09:33:17.242 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=19, tag_tid=2, entry_tid=16795' '!=' 'object_number=15, tag_tid=2, entry_tid=16735' 2024-04-06T09:33:17.242 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:33:17.242 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=19, tag_tid=2, entry_tid=16795' = 'object_number=15, tag_tid=2, entry_tid=16735' 2024-04-06T09:33:17.242 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=19, tag_tid=2, entry_tid=16795' 2024-04-06T09:33:17.242 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=25, tag_tid=3, entry_tid=1' 2024-04-06T09:33:17.242 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:33:17.242 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:33:17.243 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=3 2024-04-06T09:33:17.243 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=19, tag_tid=2, entry_tid=16795' 2024-04-06T09:33:17.244 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:33:17.244 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:33:17.244 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=2 2024-04-06T09:33:17.245 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=25, tag_tid=3, entry_tid=1' 2024-04-06T09:33:17.245 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:33:17.245 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:33:17.246 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:33:17.247 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=19, tag_tid=2, entry_tid=16795' 2024-04-06T09:33:17.247 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:33:17.247 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:33:17.248 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=16795 2024-04-06T09:33:17.248 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 3 = 2 -a 1 -le 16795 2024-04-06T09:33:17.248 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:33:17.248 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:33:17.249 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:33:17.450 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:33:17.450 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:33:17.450 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:33:17.450 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:33:17.450 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:33:17.450 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:33:17.451 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:33:17.451 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:33:17.451 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:33:17.451 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:33:17.451 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:33:17.451 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:33:17.451 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:33:17.451 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:33:17.451 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:33:17.452 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:33:17.452 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:33:17.457 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:33:17.458 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:33:17.458 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:33:17.458 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:33:17.458 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:33:17.458 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:33:17.458 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:33:17.458 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:33:17.458 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:33:17.458 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:33:17.458 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:33:17.458 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:33:17.458 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:33:17.458 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:33:17.459 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:33:17.459 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:33:17.459 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:33:17.459 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:33:17.459 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:33:17.459 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:33:17.459 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:33:17.459 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:33:17.459 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:33:17.459 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:33:17.459 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:33:17.459 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:33:17.459 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:33:17.459 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:33:17.460 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:33:17.460 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:33:17.460 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:33:17.460 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:33:17.481 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:33:17.481 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:33:17.481 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:33:17.481 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:33:17.618 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:33:17.620 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:33:17.620 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:33:17.620 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:33:17.620 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:33:17.620 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:33:17.620 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:33:17.620 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:33:17.621 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:33:17.621 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:33:17.621 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:33:17.621 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:33:17.622 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:33:17.623 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:33:17.623 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:33:17.624 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:33:17.687 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 4 2024-04-06T09:33:17.687 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 6 2024-04-06T09:33:17.687 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:33:17.687 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=25, tag_tid=3, entry_tid=1], [object_number=24, tag_tid=3, entry_tid=0], [object_number=27, tag_tid=2, entry_tid=27223], [object_number=26, tag_tid=2, entry_tid=27222]]], state=connected] 2024-04-06T09:33:17.688 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=17, tag_tid=2, entry_tid=16893], [object_number=16, tag_tid=2, entry_tid=16892], [object_number=19, tag_tid=2, entry_tid=16891], [object_number=18, tag_tid=2, entry_tid=16890]]], state=connected] 2024-04-06T09:33:17.691 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:33:17.693 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=25, tag_tid=3, entry_tid=1' 2024-04-06T09:33:17.693 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:33:17.693 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:33:17.694 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:33:17.694 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:33:17.694 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:33:17.694 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:33:17.694 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:33:17.694 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:33:17.694 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:33:17.694 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:33:17.694 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:33:17.694 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:33:17.695 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:33:17.695 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:33:17.695 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:33:17.739 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 4 2024-04-06T09:33:17.739 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 6 2024-04-06T09:33:17.739 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:33:17.739 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=25, tag_tid=3, entry_tid=1], [object_number=24, tag_tid=3, entry_tid=0], [object_number=27, tag_tid=2, entry_tid=27223], [object_number=26, tag_tid=2, entry_tid=27222]]], state=connected] 2024-04-06T09:33:17.740 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=17, tag_tid=2, entry_tid=16893], [object_number=16, tag_tid=2, entry_tid=16892], [object_number=19, tag_tid=2, entry_tid=16891], [object_number=18, tag_tid=2, entry_tid=16890]]], state=connected] 2024-04-06T09:33:17.743 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:33:17.744 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=17, tag_tid=2, entry_tid=16893' 2024-04-06T09:33:17.744 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=25, tag_tid=3, entry_tid=1' -a 'object_number=25, tag_tid=3, entry_tid=1' = 'object_number=17, tag_tid=2, entry_tid=16893' 2024-04-06T09:33:17.745 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=17, tag_tid=2, entry_tid=16893' '!=' 'object_number=19, tag_tid=2, entry_tid=16795' 2024-04-06T09:33:17.745 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:33:17.745 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=17, tag_tid=2, entry_tid=16893' = 'object_number=19, tag_tid=2, entry_tid=16795' 2024-04-06T09:33:17.745 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=17, tag_tid=2, entry_tid=16893' 2024-04-06T09:33:17.745 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=25, tag_tid=3, entry_tid=1' 2024-04-06T09:33:17.745 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:33:17.746 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:33:17.746 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=3 2024-04-06T09:33:17.747 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=17, tag_tid=2, entry_tid=16893' 2024-04-06T09:33:17.747 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:33:17.747 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:33:17.747 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=2 2024-04-06T09:33:17.748 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=25, tag_tid=3, entry_tid=1' 2024-04-06T09:33:17.748 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:33:17.748 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:33:17.749 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:33:17.750 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=17, tag_tid=2, entry_tid=16893' 2024-04-06T09:33:17.750 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:33:17.750 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:33:17.751 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=16893 2024-04-06T09:33:17.751 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 3 = 2 -a 1 -le 16893 2024-04-06T09:33:17.751 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:33:17.751 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:33:17.751 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:33:17.953 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:33:17.953 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:33:17.953 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:33:17.953 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:33:17.953 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:33:17.954 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:33:17.954 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:33:17.954 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:33:17.954 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:33:17.954 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:33:17.954 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:33:17.954 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:33:17.954 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:33:17.954 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:33:17.954 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:33:17.955 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:33:17.955 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:33:17.957 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:33:17.957 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:33:17.957 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:33:17.958 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:33:17.958 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:33:17.958 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:33:17.958 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:33:17.958 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:33:17.958 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:33:17.958 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:33:17.958 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:33:17.958 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:33:17.958 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:33:17.958 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:33:17.958 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:33:17.958 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:33:17.959 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:33:17.959 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:33:17.959 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:33:17.959 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:33:17.959 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:33:17.959 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:33:17.959 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:33:17.959 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:33:17.959 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:33:17.959 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:33:17.959 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:33:17.960 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:33:17.960 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:33:17.960 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:33:17.960 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:33:17.960 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:33:17.983 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:33:17.984 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:33:17.984 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:33:17.984 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:33:18.117 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:33:18.119 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:33:18.119 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:33:18.119 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:33:18.119 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:33:18.119 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:33:18.119 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:33:18.119 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:33:18.119 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:33:18.120 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:33:18.120 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:33:18.120 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:33:18.120 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:33:18.121 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:33:18.122 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:33:18.122 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:33:18.167 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 4 2024-04-06T09:33:18.167 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 6 2024-04-06T09:33:18.167 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:33:18.168 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=25, tag_tid=3, entry_tid=1], [object_number=24, tag_tid=3, entry_tid=0], [object_number=27, tag_tid=2, entry_tid=27223], [object_number=26, tag_tid=2, entry_tid=27222]]], state=connected] 2024-04-06T09:33:18.168 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=16, tag_tid=2, entry_tid=17004], [object_number=19, tag_tid=2, entry_tid=17003], [object_number=18, tag_tid=2, entry_tid=17002], [object_number=17, tag_tid=2, entry_tid=17001]]], state=connected] 2024-04-06T09:33:18.171 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:33:18.172 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=25, tag_tid=3, entry_tid=1' 2024-04-06T09:33:18.174 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:33:18.174 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:33:18.174 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:33:18.174 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:33:18.174 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:33:18.174 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:33:18.174 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:33:18.174 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:33:18.174 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:33:18.174 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:33:18.174 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:33:18.175 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:33:18.175 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:33:18.175 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:33:18.175 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:33:18.220 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 4 2024-04-06T09:33:18.221 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 6 2024-04-06T09:33:18.221 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:33:18.221 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=25, tag_tid=3, entry_tid=1], [object_number=24, tag_tid=3, entry_tid=0], [object_number=27, tag_tid=2, entry_tid=27223], [object_number=26, tag_tid=2, entry_tid=27222]]], state=connected] 2024-04-06T09:33:18.221 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=16, tag_tid=2, entry_tid=17004], [object_number=19, tag_tid=2, entry_tid=17003], [object_number=18, tag_tid=2, entry_tid=17002], [object_number=17, tag_tid=2, entry_tid=17001]]], state=connected] 2024-04-06T09:33:18.224 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:33:18.226 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=16, tag_tid=2, entry_tid=17004' 2024-04-06T09:33:18.226 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=25, tag_tid=3, entry_tid=1' -a 'object_number=25, tag_tid=3, entry_tid=1' = 'object_number=16, tag_tid=2, entry_tid=17004' 2024-04-06T09:33:18.226 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=16, tag_tid=2, entry_tid=17004' '!=' 'object_number=17, tag_tid=2, entry_tid=16893' 2024-04-06T09:33:18.226 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:33:18.227 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=16, tag_tid=2, entry_tid=17004' = 'object_number=17, tag_tid=2, entry_tid=16893' 2024-04-06T09:33:18.227 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=16, tag_tid=2, entry_tid=17004' 2024-04-06T09:33:18.227 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=25, tag_tid=3, entry_tid=1' 2024-04-06T09:33:18.227 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:33:18.227 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:33:18.227 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=3 2024-04-06T09:33:18.228 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=16, tag_tid=2, entry_tid=17004' 2024-04-06T09:33:18.228 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:33:18.228 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:33:18.229 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=2 2024-04-06T09:33:18.230 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=25, tag_tid=3, entry_tid=1' 2024-04-06T09:33:18.230 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:33:18.230 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:33:18.231 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:33:18.232 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=16, tag_tid=2, entry_tid=17004' 2024-04-06T09:33:18.232 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:33:18.232 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:33:18.233 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=17004 2024-04-06T09:33:18.233 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 3 = 2 -a 1 -le 17004 2024-04-06T09:33:18.233 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:33:18.233 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:33:18.233 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:33:18.435 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:33:18.435 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:33:18.436 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:33:18.436 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:33:18.436 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:33:18.436 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:33:18.436 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:33:18.436 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:33:18.436 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:33:18.436 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:33:18.436 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:33:18.436 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:33:18.436 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:33:18.436 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:33:18.436 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:33:18.437 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:33:18.437 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:33:18.440 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:33:18.440 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:33:18.440 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:33:18.440 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:33:18.440 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:33:18.440 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:33:18.440 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:33:18.440 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:33:18.440 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:33:18.440 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:33:18.440 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:33:18.441 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:33:18.441 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:33:18.441 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:33:18.441 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:33:18.441 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:33:18.441 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:33:18.441 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:33:18.441 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:33:18.441 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:33:18.441 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:33:18.441 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:33:18.441 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:33:18.441 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:33:18.441 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:33:18.441 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:33:18.441 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:33:18.442 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:33:18.442 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:33:18.442 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:33:18.442 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:33:18.442 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:33:18.466 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:33:18.466 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:33:18.466 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:33:18.466 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:33:18.596 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:33:18.598 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:33:18.598 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:33:18.598 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:33:18.598 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:33:18.598 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:33:18.598 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:33:18.598 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:33:18.598 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:33:18.598 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:33:18.599 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:33:18.599 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:33:18.599 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:33:18.600 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:33:18.601 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:33:18.601 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:33:18.649 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 4 2024-04-06T09:33:18.649 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 6 2024-04-06T09:33:18.649 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:33:18.649 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=25, tag_tid=3, entry_tid=1], [object_number=24, tag_tid=3, entry_tid=0], [object_number=27, tag_tid=2, entry_tid=27223], [object_number=26, tag_tid=2, entry_tid=27222]]], state=connected] 2024-04-06T09:33:18.649 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=16, tag_tid=2, entry_tid=17112], [object_number=19, tag_tid=2, entry_tid=17111], [object_number=18, tag_tid=2, entry_tid=17110], [object_number=17, tag_tid=2, entry_tid=17109]]], state=connected] 2024-04-06T09:33:18.652 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:33:18.654 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=25, tag_tid=3, entry_tid=1' 2024-04-06T09:33:18.655 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:33:18.655 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:33:18.655 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:33:18.655 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:33:18.655 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:33:18.655 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:33:18.655 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:33:18.655 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:33:18.655 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:33:18.655 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:33:18.655 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:33:18.655 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:33:18.656 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:33:18.656 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:33:18.656 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:33:18.701 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 4 2024-04-06T09:33:18.701 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 6 2024-04-06T09:33:18.701 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:33:18.701 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=25, tag_tid=3, entry_tid=1], [object_number=24, tag_tid=3, entry_tid=0], [object_number=27, tag_tid=2, entry_tid=27223], [object_number=26, tag_tid=2, entry_tid=27222]]], state=connected] 2024-04-06T09:33:18.701 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=16, tag_tid=2, entry_tid=17112], [object_number=19, tag_tid=2, entry_tid=17111], [object_number=18, tag_tid=2, entry_tid=17110], [object_number=17, tag_tid=2, entry_tid=17109]]], state=connected] 2024-04-06T09:33:18.705 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:33:18.707 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=16, tag_tid=2, entry_tid=17112' 2024-04-06T09:33:18.707 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=25, tag_tid=3, entry_tid=1' -a 'object_number=25, tag_tid=3, entry_tid=1' = 'object_number=16, tag_tid=2, entry_tid=17112' 2024-04-06T09:33:18.707 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=16, tag_tid=2, entry_tid=17112' '!=' 'object_number=16, tag_tid=2, entry_tid=17004' 2024-04-06T09:33:18.707 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:33:18.707 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=16, tag_tid=2, entry_tid=17112' = 'object_number=16, tag_tid=2, entry_tid=17004' 2024-04-06T09:33:18.707 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=16, tag_tid=2, entry_tid=17112' 2024-04-06T09:33:18.707 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=25, tag_tid=3, entry_tid=1' 2024-04-06T09:33:18.708 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:33:18.708 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:33:18.708 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=3 2024-04-06T09:33:18.709 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=16, tag_tid=2, entry_tid=17112' 2024-04-06T09:33:18.709 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:33:18.709 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:33:18.710 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=2 2024-04-06T09:33:18.710 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=25, tag_tid=3, entry_tid=1' 2024-04-06T09:33:18.710 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:33:18.710 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:33:18.711 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:33:18.712 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=16, tag_tid=2, entry_tid=17112' 2024-04-06T09:33:18.712 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:33:18.712 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:33:18.713 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=17112 2024-04-06T09:33:18.713 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 3 = 2 -a 1 -le 17112 2024-04-06T09:33:18.713 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:33:18.714 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:33:18.714 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:33:18.915 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:33:18.915 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:33:18.915 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:33:18.915 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:33:18.916 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:33:18.916 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:33:18.916 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:33:18.916 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:33:18.916 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:33:18.916 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:33:18.916 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:33:18.916 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:33:18.916 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:33:18.916 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:33:18.916 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:33:18.917 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:33:18.917 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:33:18.919 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:33:18.920 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:33:18.920 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:33:18.920 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:33:18.920 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:33:18.920 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:33:18.920 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:33:18.920 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:33:18.920 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:33:18.920 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:33:18.920 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:33:18.920 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:33:18.920 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:33:18.920 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:33:18.920 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:33:18.921 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:33:18.921 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:33:18.921 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:33:18.921 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:33:18.921 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:33:18.921 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:33:18.921 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:33:18.921 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:33:18.921 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:33:18.921 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:33:18.921 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:33:18.921 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:33:18.921 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:33:18.921 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:33:18.922 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:33:18.922 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:33:18.922 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:33:18.945 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:33:18.946 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:33:18.946 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:33:18.946 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:33:19.078 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:33:19.079 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:33:19.079 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:33:19.080 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:33:19.080 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:33:19.080 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:33:19.080 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:33:19.080 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:33:19.080 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:33:19.080 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:33:19.081 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:33:19.081 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:33:19.081 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:33:19.082 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:33:19.083 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:33:19.083 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:33:19.136 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 4 2024-04-06T09:33:19.136 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 6 2024-04-06T09:33:19.136 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:33:19.136 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=25, tag_tid=3, entry_tid=1], [object_number=24, tag_tid=3, entry_tid=0], [object_number=27, tag_tid=2, entry_tid=27223], [object_number=26, tag_tid=2, entry_tid=27222]]], state=connected] 2024-04-06T09:33:19.136 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=17, tag_tid=2, entry_tid=17221], [object_number=16, tag_tid=2, entry_tid=17220], [object_number=19, tag_tid=2, entry_tid=17219], [object_number=18, tag_tid=2, entry_tid=17218]]], state=connected] 2024-04-06T09:33:19.139 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:33:19.141 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=25, tag_tid=3, entry_tid=1' 2024-04-06T09:33:19.142 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:33:19.142 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:33:19.142 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:33:19.142 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:33:19.142 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:33:19.142 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:33:19.142 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:33:19.142 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:33:19.142 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:33:19.143 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:33:19.143 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:33:19.143 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:33:19.143 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:33:19.143 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:33:19.143 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:33:19.188 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 4 2024-04-06T09:33:19.188 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 6 2024-04-06T09:33:19.188 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:33:19.188 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=25, tag_tid=3, entry_tid=1], [object_number=24, tag_tid=3, entry_tid=0], [object_number=27, tag_tid=2, entry_tid=27223], [object_number=26, tag_tid=2, entry_tid=27222]]], state=connected] 2024-04-06T09:33:19.188 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=17, tag_tid=2, entry_tid=17221], [object_number=16, tag_tid=2, entry_tid=17220], [object_number=19, tag_tid=2, entry_tid=17219], [object_number=18, tag_tid=2, entry_tid=17218]]], state=connected] 2024-04-06T09:33:19.192 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:33:19.194 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=17, tag_tid=2, entry_tid=17221' 2024-04-06T09:33:19.194 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=25, tag_tid=3, entry_tid=1' -a 'object_number=25, tag_tid=3, entry_tid=1' = 'object_number=17, tag_tid=2, entry_tid=17221' 2024-04-06T09:33:19.194 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=17, tag_tid=2, entry_tid=17221' '!=' 'object_number=16, tag_tid=2, entry_tid=17112' 2024-04-06T09:33:19.194 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:33:19.194 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=17, tag_tid=2, entry_tid=17221' = 'object_number=16, tag_tid=2, entry_tid=17112' 2024-04-06T09:33:19.194 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=17, tag_tid=2, entry_tid=17221' 2024-04-06T09:33:19.194 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=25, tag_tid=3, entry_tid=1' 2024-04-06T09:33:19.194 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:33:19.194 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:33:19.195 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=3 2024-04-06T09:33:19.195 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=17, tag_tid=2, entry_tid=17221' 2024-04-06T09:33:19.195 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:33:19.196 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:33:19.196 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=2 2024-04-06T09:33:19.197 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=25, tag_tid=3, entry_tid=1' 2024-04-06T09:33:19.197 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:33:19.197 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:33:19.198 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:33:19.199 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=17, tag_tid=2, entry_tid=17221' 2024-04-06T09:33:19.199 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:33:19.199 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:33:19.200 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=17221 2024-04-06T09:33:19.200 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 3 = 2 -a 1 -le 17221 2024-04-06T09:33:19.200 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:33:19.200 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:33:19.200 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:33:19.402 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:33:19.402 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:33:19.402 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:33:19.402 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:33:19.403 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:33:19.403 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:33:19.403 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:33:19.403 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:33:19.403 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:33:19.403 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:33:19.403 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:33:19.403 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:33:19.403 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:33:19.403 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:33:19.403 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:33:19.404 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:33:19.404 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:33:19.406 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:33:19.406 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:33:19.406 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:33:19.406 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:33:19.406 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:33:19.406 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:33:19.406 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:33:19.406 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:33:19.406 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:33:19.406 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:33:19.406 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:33:19.406 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:33:19.406 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:33:19.407 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:33:19.407 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:33:19.407 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:33:19.408 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:33:19.408 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:33:19.408 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:33:19.408 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:33:19.408 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:33:19.408 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:33:19.408 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:33:19.408 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:33:19.409 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:33:19.409 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:33:19.409 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:33:19.409 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:33:19.409 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:33:19.409 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:33:19.409 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:33:19.409 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:33:19.434 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:33:19.434 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:33:19.434 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:33:19.434 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:33:19.567 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:33:19.568 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:33:19.568 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:33:19.568 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:33:19.568 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:33:19.568 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:33:19.568 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:33:19.568 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:33:19.569 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:33:19.569 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:33:19.569 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:33:19.569 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:33:19.569 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:33:19.571 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:33:19.571 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:33:19.572 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:33:19.616 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 4 2024-04-06T09:33:19.616 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 6 2024-04-06T09:33:19.617 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:33:19.617 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=25, tag_tid=3, entry_tid=1], [object_number=24, tag_tid=3, entry_tid=0], [object_number=27, tag_tid=2, entry_tid=27223], [object_number=26, tag_tid=2, entry_tid=27222]]], state=connected] 2024-04-06T09:33:19.617 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=17, tag_tid=2, entry_tid=17337], [object_number=16, tag_tid=2, entry_tid=17336], [object_number=19, tag_tid=2, entry_tid=17335], [object_number=18, tag_tid=2, entry_tid=17334]]], state=connected] 2024-04-06T09:33:19.619 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:33:19.622 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=25, tag_tid=3, entry_tid=1' 2024-04-06T09:33:19.622 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:33:19.622 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:33:19.623 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:33:19.623 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:33:19.623 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:33:19.623 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:33:19.623 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:33:19.623 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:33:19.623 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:33:19.623 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:33:19.623 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:33:19.623 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:33:19.624 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:33:19.624 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:33:19.624 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:33:19.667 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 4 2024-04-06T09:33:19.668 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 6 2024-04-06T09:33:19.668 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:33:19.668 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=25, tag_tid=3, entry_tid=1], [object_number=24, tag_tid=3, entry_tid=0], [object_number=27, tag_tid=2, entry_tid=27223], [object_number=26, tag_tid=2, entry_tid=27222]]], state=connected] 2024-04-06T09:33:19.668 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=17, tag_tid=2, entry_tid=17337], [object_number=16, tag_tid=2, entry_tid=17336], [object_number=19, tag_tid=2, entry_tid=17335], [object_number=18, tag_tid=2, entry_tid=17334]]], state=connected] 2024-04-06T09:33:19.671 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:33:19.673 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=17, tag_tid=2, entry_tid=17337' 2024-04-06T09:33:19.673 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=25, tag_tid=3, entry_tid=1' -a 'object_number=25, tag_tid=3, entry_tid=1' = 'object_number=17, tag_tid=2, entry_tid=17337' 2024-04-06T09:33:19.673 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=17, tag_tid=2, entry_tid=17337' '!=' 'object_number=17, tag_tid=2, entry_tid=17221' 2024-04-06T09:33:19.673 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:33:19.673 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=17, tag_tid=2, entry_tid=17337' = 'object_number=17, tag_tid=2, entry_tid=17221' 2024-04-06T09:33:19.673 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=17, tag_tid=2, entry_tid=17337' 2024-04-06T09:33:19.673 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=25, tag_tid=3, entry_tid=1' 2024-04-06T09:33:19.673 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:33:19.673 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:33:19.674 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=3 2024-04-06T09:33:19.674 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=17, tag_tid=2, entry_tid=17337' 2024-04-06T09:33:19.675 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:33:19.675 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:33:19.675 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=2 2024-04-06T09:33:19.676 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=25, tag_tid=3, entry_tid=1' 2024-04-06T09:33:19.676 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:33:19.676 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:33:19.677 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:33:19.678 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=17, tag_tid=2, entry_tid=17337' 2024-04-06T09:33:19.678 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:33:19.678 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:33:19.679 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=17337 2024-04-06T09:33:19.679 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 3 = 2 -a 1 -le 17337 2024-04-06T09:33:19.679 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:33:19.679 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:33:19.679 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:33:19.881 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:33:19.881 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:33:19.881 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:33:19.881 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:33:19.881 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:33:19.881 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:33:19.881 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:33:19.881 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:33:19.882 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:33:19.882 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:33:19.882 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:33:19.882 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:33:19.882 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:33:19.882 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:33:19.882 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:33:19.882 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:33:19.883 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:33:19.887 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:33:19.887 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:33:19.887 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:33:19.887 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:33:19.887 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:33:19.887 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:33:19.888 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:33:19.888 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:33:19.888 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:33:19.888 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:33:19.888 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:33:19.888 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:33:19.888 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:33:19.888 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:33:19.888 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:33:19.889 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:33:19.889 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:33:19.889 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:33:19.889 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:33:19.889 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:33:19.889 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:33:19.889 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:33:19.889 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:33:19.890 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:33:19.890 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:33:19.890 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:33:19.891 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:33:19.891 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:33:19.891 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:33:19.891 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:33:19.891 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:33:19.891 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:33:19.912 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:33:19.912 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:33:19.912 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:33:19.912 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:33:20.041 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:33:20.042 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:33:20.042 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:33:20.042 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:33:20.042 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:33:20.042 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:33:20.042 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:33:20.042 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:33:20.042 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:33:20.042 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:33:20.043 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:33:20.043 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:33:20.043 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:33:20.044 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:33:20.045 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:33:20.045 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:33:20.090 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 4 2024-04-06T09:33:20.090 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 6 2024-04-06T09:33:20.090 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:33:20.090 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=25, tag_tid=3, entry_tid=1], [object_number=24, tag_tid=3, entry_tid=0], [object_number=27, tag_tid=2, entry_tid=27223], [object_number=26, tag_tid=2, entry_tid=27222]]], state=connected] 2024-04-06T09:33:20.091 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=16, tag_tid=2, entry_tid=17440], [object_number=19, tag_tid=2, entry_tid=17439], [object_number=18, tag_tid=2, entry_tid=17438], [object_number=17, tag_tid=2, entry_tid=17437]]], state=connected] 2024-04-06T09:33:20.094 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:33:20.095 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=25, tag_tid=3, entry_tid=1' 2024-04-06T09:33:20.096 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:33:20.096 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:33:20.096 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:33:20.096 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:33:20.097 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:33:20.097 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:33:20.097 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:33:20.097 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:33:20.097 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:33:20.097 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:33:20.097 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:33:20.097 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:33:20.098 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:33:20.098 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:33:20.098 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:33:20.142 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 4 2024-04-06T09:33:20.142 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 6 2024-04-06T09:33:20.142 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:33:20.142 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=25, tag_tid=3, entry_tid=1], [object_number=24, tag_tid=3, entry_tid=0], [object_number=27, tag_tid=2, entry_tid=27223], [object_number=26, tag_tid=2, entry_tid=27222]]], state=connected] 2024-04-06T09:33:20.142 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=16, tag_tid=2, entry_tid=17440], [object_number=19, tag_tid=2, entry_tid=17439], [object_number=18, tag_tid=2, entry_tid=17438], [object_number=17, tag_tid=2, entry_tid=17437]]], state=connected] 2024-04-06T09:33:20.145 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:33:20.147 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=16, tag_tid=2, entry_tid=17440' 2024-04-06T09:33:20.147 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=25, tag_tid=3, entry_tid=1' -a 'object_number=25, tag_tid=3, entry_tid=1' = 'object_number=16, tag_tid=2, entry_tid=17440' 2024-04-06T09:33:20.148 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=16, tag_tid=2, entry_tid=17440' '!=' 'object_number=17, tag_tid=2, entry_tid=17337' 2024-04-06T09:33:20.148 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:33:20.148 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=16, tag_tid=2, entry_tid=17440' = 'object_number=17, tag_tid=2, entry_tid=17337' 2024-04-06T09:33:20.148 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=16, tag_tid=2, entry_tid=17440' 2024-04-06T09:33:20.148 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=25, tag_tid=3, entry_tid=1' 2024-04-06T09:33:20.148 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:33:20.148 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:33:20.148 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=3 2024-04-06T09:33:20.149 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=16, tag_tid=2, entry_tid=17440' 2024-04-06T09:33:20.149 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:33:20.149 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:33:20.150 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=2 2024-04-06T09:33:20.151 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=25, tag_tid=3, entry_tid=1' 2024-04-06T09:33:20.151 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:33:20.151 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:33:20.152 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:33:20.152 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=16, tag_tid=2, entry_tid=17440' 2024-04-06T09:33:20.153 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:33:20.153 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:33:20.154 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=17440 2024-04-06T09:33:20.154 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 3 = 2 -a 1 -le 17440 2024-04-06T09:33:20.154 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:33:20.154 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:33:20.154 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:33:20.356 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:33:20.356 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:33:20.356 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:33:20.356 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:33:20.356 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:33:20.356 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:33:20.356 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:33:20.356 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:33:20.356 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:33:20.356 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:33:20.357 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:33:20.357 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:33:20.357 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:33:20.357 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:33:20.357 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:33:20.357 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:33:20.358 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:33:20.360 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:33:20.360 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:33:20.360 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:33:20.360 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:33:20.360 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:33:20.361 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:33:20.361 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:33:20.361 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:33:20.361 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:33:20.361 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:33:20.361 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:33:20.361 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:33:20.361 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:33:20.361 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:33:20.361 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:33:20.361 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:33:20.362 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:33:20.362 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:33:20.362 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:33:20.362 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:33:20.362 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:33:20.362 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:33:20.362 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:33:20.363 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:33:20.363 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:33:20.363 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:33:20.363 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:33:20.363 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:33:20.363 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:33:20.363 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:33:20.363 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:33:20.363 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:33:20.388 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:33:20.388 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:33:20.388 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:33:20.388 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:33:20.524 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:33:20.525 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:33:20.526 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:33:20.526 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:33:20.526 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:33:20.526 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:33:20.526 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:33:20.526 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:33:20.526 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:33:20.526 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:33:20.526 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:33:20.527 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:33:20.527 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:33:20.527 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:33:20.528 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:33:20.528 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:33:20.573 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 4 2024-04-06T09:33:20.573 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 6 2024-04-06T09:33:20.573 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:33:20.573 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=25, tag_tid=3, entry_tid=1], [object_number=24, tag_tid=3, entry_tid=0], [object_number=27, tag_tid=2, entry_tid=27223], [object_number=26, tag_tid=2, entry_tid=27222]]], state=connected] 2024-04-06T09:33:20.573 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=19, tag_tid=2, entry_tid=17551], [object_number=18, tag_tid=2, entry_tid=17550], [object_number=17, tag_tid=2, entry_tid=17549], [object_number=16, tag_tid=2, entry_tid=17548]]], state=connected] 2024-04-06T09:33:20.576 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:33:20.578 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=25, tag_tid=3, entry_tid=1' 2024-04-06T09:33:20.579 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:33:20.579 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:33:20.579 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:33:20.579 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:33:20.579 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:33:20.579 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:33:20.579 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:33:20.579 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:33:20.579 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:33:20.580 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:33:20.580 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:33:20.580 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:33:20.580 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:33:20.580 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:33:20.580 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:33:20.833 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 4 2024-04-06T09:33:20.834 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 6 2024-04-06T09:33:20.834 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:33:20.834 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=25, tag_tid=3, entry_tid=1], [object_number=24, tag_tid=3, entry_tid=0], [object_number=27, tag_tid=2, entry_tid=27223], [object_number=26, tag_tid=2, entry_tid=27222]]], state=connected] 2024-04-06T09:33:20.834 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=19, tag_tid=2, entry_tid=17551], [object_number=18, tag_tid=2, entry_tid=17550], [object_number=17, tag_tid=2, entry_tid=17549], [object_number=16, tag_tid=2, entry_tid=17548]]], state=connected] 2024-04-06T09:33:20.838 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:33:20.840 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=19, tag_tid=2, entry_tid=17551' 2024-04-06T09:33:20.840 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=25, tag_tid=3, entry_tid=1' -a 'object_number=25, tag_tid=3, entry_tid=1' = 'object_number=19, tag_tid=2, entry_tid=17551' 2024-04-06T09:33:20.840 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=19, tag_tid=2, entry_tid=17551' '!=' 'object_number=16, tag_tid=2, entry_tid=17440' 2024-04-06T09:33:20.840 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:33:20.840 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=19, tag_tid=2, entry_tid=17551' = 'object_number=16, tag_tid=2, entry_tid=17440' 2024-04-06T09:33:20.840 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=19, tag_tid=2, entry_tid=17551' 2024-04-06T09:33:20.841 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=25, tag_tid=3, entry_tid=1' 2024-04-06T09:33:20.841 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:33:20.841 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:33:20.841 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=3 2024-04-06T09:33:20.842 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=19, tag_tid=2, entry_tid=17551' 2024-04-06T09:33:20.842 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:33:20.842 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:33:20.843 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=2 2024-04-06T09:33:20.844 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=25, tag_tid=3, entry_tid=1' 2024-04-06T09:33:20.844 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:33:20.844 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:33:20.846 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:33:20.847 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=19, tag_tid=2, entry_tid=17551' 2024-04-06T09:33:20.847 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:33:20.847 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:33:20.848 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=17551 2024-04-06T09:33:20.848 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 3 = 2 -a 1 -le 17551 2024-04-06T09:33:20.848 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:33:20.848 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:33:20.848 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:33:21.050 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:33:21.050 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:33:21.050 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:33:21.050 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:33:21.051 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:33:21.051 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:33:21.051 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:33:21.051 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:33:21.051 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:33:21.051 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:33:21.051 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:33:21.051 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:33:21.051 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:33:21.051 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:33:21.051 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:33:21.052 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:33:21.052 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:33:21.055 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:33:21.055 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:33:21.055 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:33:21.055 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:33:21.055 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:33:21.055 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:33:21.055 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:33:21.055 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:33:21.055 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:33:21.055 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:33:21.055 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:33:21.055 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:33:21.056 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:33:21.056 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:33:21.056 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:33:21.056 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:33:21.057 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:33:21.057 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:33:21.057 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:33:21.057 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:33:21.057 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:33:21.057 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:33:21.057 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:33:21.057 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:33:21.057 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:33:21.057 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:33:21.057 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:33:21.058 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:33:21.058 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:33:21.058 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:33:21.058 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:33:21.058 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:33:21.081 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:33:21.081 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:33:21.081 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:33:21.082 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:33:21.204 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:33:21.206 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:33:21.206 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:33:21.206 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:33:21.206 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:33:21.206 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:33:21.207 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:33:21.207 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:33:21.207 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:33:21.207 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:33:21.207 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:33:21.207 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:33:21.207 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:33:21.209 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:33:21.209 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:33:21.209 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:33:21.255 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 4 2024-04-06T09:33:21.255 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 6 2024-04-06T09:33:21.255 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:33:21.255 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=25, tag_tid=3, entry_tid=1], [object_number=24, tag_tid=3, entry_tid=0], [object_number=27, tag_tid=2, entry_tid=27223], [object_number=26, tag_tid=2, entry_tid=27222]]], state=connected] 2024-04-06T09:33:21.255 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=16, tag_tid=2, entry_tid=17696], [object_number=19, tag_tid=2, entry_tid=17695], [object_number=18, tag_tid=2, entry_tid=17694], [object_number=17, tag_tid=2, entry_tid=17693]]], state=connected] 2024-04-06T09:33:21.259 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:33:21.260 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=25, tag_tid=3, entry_tid=1' 2024-04-06T09:33:21.261 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:33:21.261 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:33:21.261 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:33:21.261 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:33:21.261 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:33:21.261 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:33:21.261 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:33:21.261 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:33:21.261 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:33:21.262 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:33:21.262 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:33:21.262 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:33:21.262 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:33:21.262 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:33:21.262 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:33:21.307 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 4 2024-04-06T09:33:21.308 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 6 2024-04-06T09:33:21.308 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:33:21.308 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=25, tag_tid=3, entry_tid=1], [object_number=24, tag_tid=3, entry_tid=0], [object_number=27, tag_tid=2, entry_tid=27223], [object_number=26, tag_tid=2, entry_tid=27222]]], state=connected] 2024-04-06T09:33:21.308 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=16, tag_tid=2, entry_tid=17696], [object_number=19, tag_tid=2, entry_tid=17695], [object_number=18, tag_tid=2, entry_tid=17694], [object_number=17, tag_tid=2, entry_tid=17693]]], state=connected] 2024-04-06T09:33:21.311 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:33:21.313 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=16, tag_tid=2, entry_tid=17696' 2024-04-06T09:33:21.313 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=25, tag_tid=3, entry_tid=1' -a 'object_number=25, tag_tid=3, entry_tid=1' = 'object_number=16, tag_tid=2, entry_tid=17696' 2024-04-06T09:33:21.313 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=16, tag_tid=2, entry_tid=17696' '!=' 'object_number=19, tag_tid=2, entry_tid=17551' 2024-04-06T09:33:21.313 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:33:21.313 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=16, tag_tid=2, entry_tid=17696' = 'object_number=19, tag_tid=2, entry_tid=17551' 2024-04-06T09:33:21.313 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=16, tag_tid=2, entry_tid=17696' 2024-04-06T09:33:21.313 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=25, tag_tid=3, entry_tid=1' 2024-04-06T09:33:21.314 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:33:21.314 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:33:21.314 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=3 2024-04-06T09:33:21.315 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=16, tag_tid=2, entry_tid=17696' 2024-04-06T09:33:21.315 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:33:21.315 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:33:21.316 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=2 2024-04-06T09:33:21.316 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=25, tag_tid=3, entry_tid=1' 2024-04-06T09:33:21.317 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:33:21.317 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:33:21.317 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:33:21.318 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=16, tag_tid=2, entry_tid=17696' 2024-04-06T09:33:21.318 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:33:21.318 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:33:21.319 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=17696 2024-04-06T09:33:21.319 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 3 = 2 -a 1 -le 17696 2024-04-06T09:33:21.319 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:33:21.320 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:33:21.320 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:33:21.521 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:33:21.521 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:33:21.521 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:33:21.521 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:33:21.521 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:33:21.521 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:33:21.522 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:33:21.522 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:33:21.522 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:33:21.522 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:33:21.522 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:33:21.522 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:33:21.522 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:33:21.522 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:33:21.522 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:33:21.523 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:33:21.523 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:33:21.527 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:33:21.527 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:33:21.527 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:33:21.527 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:33:21.527 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:33:21.527 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:33:21.527 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:33:21.527 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:33:21.527 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:33:21.527 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:33:21.527 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:33:21.528 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:33:21.528 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:33:21.528 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:33:21.528 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:33:21.528 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:33:21.528 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:33:21.528 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:33:21.528 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:33:21.528 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:33:21.528 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:33:21.528 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:33:21.528 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:33:21.528 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:33:21.528 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:33:21.528 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:33:21.529 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:33:21.529 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:33:21.529 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:33:21.529 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:33:21.529 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:33:21.529 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:33:21.553 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:33:21.553 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:33:21.553 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:33:21.553 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:33:21.678 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:33:21.680 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:33:21.680 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:33:21.680 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:33:21.680 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:33:21.680 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:33:21.680 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:33:21.680 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:33:21.681 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:33:21.681 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:33:21.681 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:33:21.681 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:33:21.681 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:33:21.682 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:33:21.683 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:33:21.683 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:33:21.734 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 4 2024-04-06T09:33:21.735 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 6 2024-04-06T09:33:21.735 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:33:21.735 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=25, tag_tid=3, entry_tid=1], [object_number=24, tag_tid=3, entry_tid=0], [object_number=27, tag_tid=2, entry_tid=27223], [object_number=26, tag_tid=2, entry_tid=27222]]], state=connected] 2024-04-06T09:33:21.735 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=18, tag_tid=2, entry_tid=17794], [object_number=17, tag_tid=2, entry_tid=17793], [object_number=16, tag_tid=2, entry_tid=17792], [object_number=19, tag_tid=2, entry_tid=17791]]], state=connected] 2024-04-06T09:33:21.738 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:33:21.739 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=25, tag_tid=3, entry_tid=1' 2024-04-06T09:33:21.740 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:33:21.741 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:33:21.741 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:33:21.741 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:33:21.741 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:33:21.741 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:33:21.741 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:33:21.741 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:33:21.741 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:33:21.741 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:33:21.741 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:33:21.741 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:33:21.742 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:33:21.742 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:33:21.742 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:33:21.787 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 4 2024-04-06T09:33:21.788 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 6 2024-04-06T09:33:21.788 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:33:21.788 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=25, tag_tid=3, entry_tid=1], [object_number=24, tag_tid=3, entry_tid=0], [object_number=27, tag_tid=2, entry_tid=27223], [object_number=26, tag_tid=2, entry_tid=27222]]], state=connected] 2024-04-06T09:33:21.788 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=18, tag_tid=2, entry_tid=17794], [object_number=17, tag_tid=2, entry_tid=17793], [object_number=16, tag_tid=2, entry_tid=17792], [object_number=19, tag_tid=2, entry_tid=17791]]], state=connected] 2024-04-06T09:33:21.791 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:33:21.794 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=18, tag_tid=2, entry_tid=17794' 2024-04-06T09:33:21.794 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=25, tag_tid=3, entry_tid=1' -a 'object_number=25, tag_tid=3, entry_tid=1' = 'object_number=18, tag_tid=2, entry_tid=17794' 2024-04-06T09:33:21.794 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=18, tag_tid=2, entry_tid=17794' '!=' 'object_number=16, tag_tid=2, entry_tid=17696' 2024-04-06T09:33:21.794 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:33:21.794 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=18, tag_tid=2, entry_tid=17794' = 'object_number=16, tag_tid=2, entry_tid=17696' 2024-04-06T09:33:21.794 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=18, tag_tid=2, entry_tid=17794' 2024-04-06T09:33:21.794 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=25, tag_tid=3, entry_tid=1' 2024-04-06T09:33:21.794 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:33:21.794 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:33:21.795 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=3 2024-04-06T09:33:21.795 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=18, tag_tid=2, entry_tid=17794' 2024-04-06T09:33:21.796 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:33:21.796 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:33:21.797 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=2 2024-04-06T09:33:21.797 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=25, tag_tid=3, entry_tid=1' 2024-04-06T09:33:21.797 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:33:21.797 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:33:21.798 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:33:21.799 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=18, tag_tid=2, entry_tid=17794' 2024-04-06T09:33:21.799 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:33:21.799 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:33:21.800 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=17794 2024-04-06T09:33:21.800 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 3 = 2 -a 1 -le 17794 2024-04-06T09:33:21.800 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:33:21.800 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:33:21.801 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:33:22.002 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:33:22.002 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:33:22.003 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:33:22.003 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:33:22.003 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:33:22.003 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:33:22.003 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:33:22.003 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:33:22.003 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:33:22.003 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:33:22.003 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:33:22.003 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:33:22.003 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:33:22.003 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:33:22.003 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:33:22.004 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:33:22.004 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:33:22.005 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:33:22.006 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:33:22.006 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:33:22.006 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:33:22.006 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:33:22.006 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:33:22.006 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:33:22.006 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:33:22.006 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:33:22.006 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:33:22.006 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:33:22.006 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:33:22.006 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:33:22.006 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:33:22.006 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:33:22.007 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:33:22.008 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:33:22.008 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:33:22.008 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:33:22.008 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:33:22.008 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:33:22.008 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:33:22.008 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:33:22.008 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:33:22.009 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:33:22.009 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:33:22.009 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:33:22.009 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:33:22.009 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:33:22.009 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:33:22.009 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:33:22.009 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:33:22.034 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:33:22.034 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:33:22.035 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:33:22.035 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:33:22.179 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:33:22.180 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:33:22.180 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:33:22.180 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:33:22.180 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:33:22.180 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:33:22.180 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:33:22.180 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:33:22.180 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:33:22.180 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:33:22.181 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:33:22.181 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:33:22.181 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:33:22.183 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:33:22.184 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:33:22.184 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:33:22.234 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 4 2024-04-06T09:33:22.234 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 6 2024-04-06T09:33:22.234 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:33:22.234 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=25, tag_tid=3, entry_tid=1], [object_number=24, tag_tid=3, entry_tid=0], [object_number=27, tag_tid=2, entry_tid=27223], [object_number=26, tag_tid=2, entry_tid=27222]]], state=connected] 2024-04-06T09:33:22.234 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=18, tag_tid=2, entry_tid=17882], [object_number=17, tag_tid=2, entry_tid=17881], [object_number=16, tag_tid=2, entry_tid=17880], [object_number=19, tag_tid=2, entry_tid=17879]]], state=connected] 2024-04-06T09:33:22.238 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:33:22.239 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=25, tag_tid=3, entry_tid=1' 2024-04-06T09:33:22.240 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:33:22.240 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:33:22.240 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:33:22.240 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:33:22.240 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:33:22.241 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:33:22.241 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:33:22.241 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:33:22.241 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:33:22.241 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:33:22.241 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:33:22.241 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:33:22.242 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:33:22.242 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:33:22.242 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:33:22.288 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 4 2024-04-06T09:33:22.288 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 6 2024-04-06T09:33:22.288 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:33:22.288 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=25, tag_tid=3, entry_tid=1], [object_number=24, tag_tid=3, entry_tid=0], [object_number=27, tag_tid=2, entry_tid=27223], [object_number=26, tag_tid=2, entry_tid=27222]]], state=connected] 2024-04-06T09:33:22.288 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=18, tag_tid=2, entry_tid=17882], [object_number=17, tag_tid=2, entry_tid=17881], [object_number=16, tag_tid=2, entry_tid=17880], [object_number=19, tag_tid=2, entry_tid=17879]]], state=connected] 2024-04-06T09:33:22.291 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:33:22.293 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=18, tag_tid=2, entry_tid=17882' 2024-04-06T09:33:22.293 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=25, tag_tid=3, entry_tid=1' -a 'object_number=25, tag_tid=3, entry_tid=1' = 'object_number=18, tag_tid=2, entry_tid=17882' 2024-04-06T09:33:22.294 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=18, tag_tid=2, entry_tid=17882' '!=' 'object_number=18, tag_tid=2, entry_tid=17794' 2024-04-06T09:33:22.294 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:33:22.294 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=18, tag_tid=2, entry_tid=17882' = 'object_number=18, tag_tid=2, entry_tid=17794' 2024-04-06T09:33:22.294 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=18, tag_tid=2, entry_tid=17882' 2024-04-06T09:33:22.294 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=25, tag_tid=3, entry_tid=1' 2024-04-06T09:33:22.294 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:33:22.294 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:33:22.294 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=3 2024-04-06T09:33:22.295 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=18, tag_tid=2, entry_tid=17882' 2024-04-06T09:33:22.295 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:33:22.295 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:33:22.296 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=2 2024-04-06T09:33:22.297 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=25, tag_tid=3, entry_tid=1' 2024-04-06T09:33:22.297 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:33:22.297 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:33:22.298 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:33:22.298 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=18, tag_tid=2, entry_tid=17882' 2024-04-06T09:33:22.298 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:33:22.299 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:33:22.300 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=17882 2024-04-06T09:33:22.300 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 3 = 2 -a 1 -le 17882 2024-04-06T09:33:22.300 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:33:22.300 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:33:22.300 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:33:22.502 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:33:22.502 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:33:22.502 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:33:22.502 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:33:22.502 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:33:22.502 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:33:22.502 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:33:22.502 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:33:22.502 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:33:22.503 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:33:22.503 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:33:22.503 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:33:22.503 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:33:22.503 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:33:22.503 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:33:22.503 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:33:22.504 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:33:22.505 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:33:22.505 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:33:22.505 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:33:22.505 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:33:22.505 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:33:22.505 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:33:22.506 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:33:22.506 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:33:22.506 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:33:22.506 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:33:22.506 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:33:22.506 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:33:22.506 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:33:22.506 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:33:22.506 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:33:22.506 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:33:22.507 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:33:22.508 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:33:22.508 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:33:22.508 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:33:22.508 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:33:22.508 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:33:22.508 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:33:22.508 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:33:22.508 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:33:22.508 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:33:22.508 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:33:22.508 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:33:22.508 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:33:22.508 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:33:22.509 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:33:22.509 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:33:22.532 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:33:22.533 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:33:22.533 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:33:22.533 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:33:22.662 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:33:22.663 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:33:22.663 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:33:22.663 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:33:22.663 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:33:22.663 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:33:22.663 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:33:22.663 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:33:22.663 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:33:22.663 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:33:22.664 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:33:22.664 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:33:22.664 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:33:22.665 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:33:22.665 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:33:22.665 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:33:22.712 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 4 2024-04-06T09:33:22.712 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 6 2024-04-06T09:33:22.712 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:33:22.712 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=25, tag_tid=3, entry_tid=1], [object_number=24, tag_tid=3, entry_tid=0], [object_number=27, tag_tid=2, entry_tid=27223], [object_number=26, tag_tid=2, entry_tid=27222]]], state=connected] 2024-04-06T09:33:22.712 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=16, tag_tid=2, entry_tid=17984], [object_number=19, tag_tid=2, entry_tid=17983], [object_number=18, tag_tid=2, entry_tid=17982], [object_number=17, tag_tid=2, entry_tid=17981]]], state=connected] 2024-04-06T09:33:22.715 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:33:22.717 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=25, tag_tid=3, entry_tid=1' 2024-04-06T09:33:22.718 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:33:22.718 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:33:22.718 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:33:22.718 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:33:22.718 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:33:22.718 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:33:22.718 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:33:22.718 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:33:22.718 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:33:22.719 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:33:22.719 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:33:22.719 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:33:22.719 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:33:22.720 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:33:22.720 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:33:22.764 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 4 2024-04-06T09:33:22.764 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 6 2024-04-06T09:33:22.764 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:33:22.765 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=25, tag_tid=3, entry_tid=1], [object_number=24, tag_tid=3, entry_tid=0], [object_number=27, tag_tid=2, entry_tid=27223], [object_number=26, tag_tid=2, entry_tid=27222]]], state=connected] 2024-04-06T09:33:22.765 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=16, tag_tid=2, entry_tid=17984], [object_number=19, tag_tid=2, entry_tid=17983], [object_number=18, tag_tid=2, entry_tid=17982], [object_number=17, tag_tid=2, entry_tid=17981]]], state=connected] 2024-04-06T09:33:22.768 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:33:22.770 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=16, tag_tid=2, entry_tid=17984' 2024-04-06T09:33:22.770 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=25, tag_tid=3, entry_tid=1' -a 'object_number=25, tag_tid=3, entry_tid=1' = 'object_number=16, tag_tid=2, entry_tid=17984' 2024-04-06T09:33:22.770 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=16, tag_tid=2, entry_tid=17984' '!=' 'object_number=18, tag_tid=2, entry_tid=17882' 2024-04-06T09:33:22.770 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:33:22.770 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=16, tag_tid=2, entry_tid=17984' = 'object_number=18, tag_tid=2, entry_tid=17882' 2024-04-06T09:33:22.771 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=16, tag_tid=2, entry_tid=17984' 2024-04-06T09:33:22.771 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=25, tag_tid=3, entry_tid=1' 2024-04-06T09:33:22.771 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:33:22.771 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:33:22.771 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=3 2024-04-06T09:33:22.772 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=16, tag_tid=2, entry_tid=17984' 2024-04-06T09:33:22.772 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:33:22.772 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:33:22.773 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=2 2024-04-06T09:33:22.774 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=25, tag_tid=3, entry_tid=1' 2024-04-06T09:33:22.774 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:33:22.774 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:33:22.775 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:33:22.776 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=16, tag_tid=2, entry_tid=17984' 2024-04-06T09:33:22.776 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:33:22.776 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:33:22.777 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=17984 2024-04-06T09:33:22.777 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 3 = 2 -a 1 -le 17984 2024-04-06T09:33:22.777 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:33:22.777 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:33:22.777 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:33:22.985 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:33:22.985 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:33:22.985 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:33:22.985 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:33:22.985 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:33:22.985 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:33:22.985 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:33:22.985 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:33:22.985 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:33:22.985 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:33:22.986 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:33:22.986 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:33:22.986 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:33:22.986 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:33:22.986 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:33:22.986 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:33:22.986 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:33:22.986 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:33:22.986 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:33:22.986 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:33:22.986 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:33:22.986 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:33:22.987 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:33:22.987 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:33:22.987 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:33:22.987 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:33:22.987 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:33:22.987 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:33:22.987 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:33:22.987 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:33:22.987 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:33:22.987 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:33:22.987 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:33:22.987 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:33:22.987 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:33:22.988 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:33:22.988 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:33:22.988 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:33:22.988 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:33:22.988 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:33:22.988 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:33:22.988 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:33:22.988 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:33:22.988 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:33:22.988 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:33:22.988 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:33:22.988 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:33:22.988 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:33:22.988 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:33:23.010 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:33:23.010 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:33:23.010 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:33:23.010 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:33:23.142 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:33:23.143 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:33:23.143 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:33:23.143 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:33:23.143 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:33:23.143 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:33:23.143 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:33:23.144 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:33:23.144 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:33:23.144 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:33:23.144 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:33:23.144 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:33:23.144 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:33:23.145 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:33:23.145 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:33:23.146 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:33:23.190 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 4 2024-04-06T09:33:23.191 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 6 2024-04-06T09:33:23.191 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:33:23.191 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=25, tag_tid=3, entry_tid=1], [object_number=24, tag_tid=3, entry_tid=0], [object_number=27, tag_tid=2, entry_tid=27223], [object_number=26, tag_tid=2, entry_tid=27222]]], state=connected] 2024-04-06T09:33:23.191 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=16, tag_tid=2, entry_tid=18108], [object_number=19, tag_tid=2, entry_tid=18107], [object_number=18, tag_tid=2, entry_tid=18106], [object_number=17, tag_tid=2, entry_tid=18105]]], state=connected] 2024-04-06T09:33:23.195 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:33:23.196 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=25, tag_tid=3, entry_tid=1' 2024-04-06T09:33:23.197 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:33:23.197 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:33:23.197 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:33:23.197 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:33:23.197 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:33:23.197 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:33:23.197 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:33:23.197 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:33:23.197 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:33:23.197 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:33:23.197 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:33:23.197 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:33:23.198 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:33:23.198 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:33:23.198 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:33:23.243 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 4 2024-04-06T09:33:23.243 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 6 2024-04-06T09:33:23.243 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:33:23.243 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=25, tag_tid=3, entry_tid=1], [object_number=24, tag_tid=3, entry_tid=0], [object_number=27, tag_tid=2, entry_tid=27223], [object_number=26, tag_tid=2, entry_tid=27222]]], state=connected] 2024-04-06T09:33:23.244 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=16, tag_tid=2, entry_tid=18108], [object_number=19, tag_tid=2, entry_tid=18107], [object_number=18, tag_tid=2, entry_tid=18106], [object_number=17, tag_tid=2, entry_tid=18105]]], state=connected] 2024-04-06T09:33:23.247 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:33:23.249 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=16, tag_tid=2, entry_tid=18108' 2024-04-06T09:33:23.249 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=25, tag_tid=3, entry_tid=1' -a 'object_number=25, tag_tid=3, entry_tid=1' = 'object_number=16, tag_tid=2, entry_tid=18108' 2024-04-06T09:33:23.249 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=16, tag_tid=2, entry_tid=18108' '!=' 'object_number=16, tag_tid=2, entry_tid=17984' 2024-04-06T09:33:23.249 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:33:23.249 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=16, tag_tid=2, entry_tid=18108' = 'object_number=16, tag_tid=2, entry_tid=17984' 2024-04-06T09:33:23.249 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=16, tag_tid=2, entry_tid=18108' 2024-04-06T09:33:23.249 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=25, tag_tid=3, entry_tid=1' 2024-04-06T09:33:23.249 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:33:23.249 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:33:23.250 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=3 2024-04-06T09:33:23.251 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=16, tag_tid=2, entry_tid=18108' 2024-04-06T09:33:23.251 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:33:23.251 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:33:23.252 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=2 2024-04-06T09:33:23.252 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=25, tag_tid=3, entry_tid=1' 2024-04-06T09:33:23.252 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:33:23.252 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:33:23.253 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:33:23.254 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=16, tag_tid=2, entry_tid=18108' 2024-04-06T09:33:23.254 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:33:23.254 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:33:23.255 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=18108 2024-04-06T09:33:23.255 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 3 = 2 -a 1 -le 18108 2024-04-06T09:33:23.255 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:33:23.255 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:33:23.255 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:33:23.457 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:33:23.457 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:33:23.457 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:33:23.457 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:33:23.457 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:33:23.457 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:33:23.458 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:33:23.458 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:33:23.458 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:33:23.458 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:33:23.458 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:33:23.458 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:33:23.458 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:33:23.458 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:33:23.458 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:33:23.459 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:33:23.459 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:33:23.461 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:33:23.461 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:33:23.462 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:33:23.462 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:33:23.462 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:33:23.462 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:33:23.462 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:33:23.462 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:33:23.462 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:33:23.462 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:33:23.462 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:33:23.462 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:33:23.462 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:33:23.462 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:33:23.462 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:33:23.462 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:33:23.463 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:33:23.463 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:33:23.463 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:33:23.463 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:33:23.463 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:33:23.463 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:33:23.463 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:33:23.463 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:33:23.463 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:33:23.463 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:33:23.463 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:33:23.463 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:33:23.463 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:33:23.463 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:33:23.464 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:33:23.464 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:33:23.489 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:33:23.490 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:33:23.490 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:33:23.490 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:33:23.618 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:33:23.620 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:33:23.620 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:33:23.620 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:33:23.620 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:33:23.620 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:33:23.620 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:33:23.620 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:33:23.620 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:33:23.620 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:33:23.620 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:33:23.621 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:33:23.621 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:33:23.621 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:33:23.622 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:33:23.622 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:33:23.667 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 4 2024-04-06T09:33:23.667 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 6 2024-04-06T09:33:23.667 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:33:23.668 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=25, tag_tid=3, entry_tid=1], [object_number=24, tag_tid=3, entry_tid=0], [object_number=27, tag_tid=2, entry_tid=27223], [object_number=26, tag_tid=2, entry_tid=27222]]], state=connected] 2024-04-06T09:33:23.668 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=19, tag_tid=2, entry_tid=18223], [object_number=18, tag_tid=2, entry_tid=18222], [object_number=17, tag_tid=2, entry_tid=18221], [object_number=16, tag_tid=2, entry_tid=18220]]], state=connected] 2024-04-06T09:33:23.671 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:33:23.672 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=25, tag_tid=3, entry_tid=1' 2024-04-06T09:33:23.673 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:33:23.673 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:33:23.673 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:33:23.674 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:33:23.674 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:33:23.674 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:33:23.674 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:33:23.674 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:33:23.674 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:33:23.674 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:33:23.674 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:33:23.674 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:33:23.675 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:33:23.675 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:33:23.675 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:33:23.720 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 4 2024-04-06T09:33:23.720 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 6 2024-04-06T09:33:23.720 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:33:23.720 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=25, tag_tid=3, entry_tid=1], [object_number=24, tag_tid=3, entry_tid=0], [object_number=27, tag_tid=2, entry_tid=27223], [object_number=26, tag_tid=2, entry_tid=27222]]], state=connected] 2024-04-06T09:33:23.720 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=19, tag_tid=2, entry_tid=18223], [object_number=18, tag_tid=2, entry_tid=18222], [object_number=17, tag_tid=2, entry_tid=18221], [object_number=16, tag_tid=2, entry_tid=18220]]], state=connected] 2024-04-06T09:33:23.724 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:33:23.726 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=19, tag_tid=2, entry_tid=18223' 2024-04-06T09:33:23.726 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=25, tag_tid=3, entry_tid=1' -a 'object_number=25, tag_tid=3, entry_tid=1' = 'object_number=19, tag_tid=2, entry_tid=18223' 2024-04-06T09:33:23.726 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=19, tag_tid=2, entry_tid=18223' '!=' 'object_number=16, tag_tid=2, entry_tid=18108' 2024-04-06T09:33:23.726 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:33:23.726 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=19, tag_tid=2, entry_tid=18223' = 'object_number=16, tag_tid=2, entry_tid=18108' 2024-04-06T09:33:23.726 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=19, tag_tid=2, entry_tid=18223' 2024-04-06T09:33:23.726 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=25, tag_tid=3, entry_tid=1' 2024-04-06T09:33:23.727 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:33:23.727 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:33:23.727 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=3 2024-04-06T09:33:23.728 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=19, tag_tid=2, entry_tid=18223' 2024-04-06T09:33:23.728 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:33:23.728 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:33:23.729 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=2 2024-04-06T09:33:23.729 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=25, tag_tid=3, entry_tid=1' 2024-04-06T09:33:23.730 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:33:23.730 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:33:23.730 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:33:23.731 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=19, tag_tid=2, entry_tid=18223' 2024-04-06T09:33:23.731 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:33:23.731 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:33:23.733 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=18223 2024-04-06T09:33:23.733 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 3 = 2 -a 1 -le 18223 2024-04-06T09:33:23.733 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:33:23.733 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:33:23.733 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:33:23.934 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:33:23.935 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:33:23.935 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:33:23.935 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:33:23.935 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:33:23.935 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:33:23.935 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:33:23.935 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:33:23.935 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:33:23.935 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:33:23.935 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:33:23.935 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:33:23.936 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:33:23.936 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:33:23.936 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:33:23.936 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:33:23.936 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:33:23.938 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:33:23.938 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:33:23.938 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:33:23.938 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:33:23.938 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:33:23.939 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:33:23.939 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:33:23.939 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:33:23.939 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:33:23.939 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:33:23.939 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:33:23.939 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:33:23.939 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:33:23.939 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:33:23.939 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:33:23.939 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:33:23.941 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:33:23.941 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:33:23.941 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:33:23.942 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:33:23.942 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:33:23.942 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:33:23.942 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:33:23.942 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:33:23.942 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:33:23.942 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:33:23.942 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:33:23.942 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:33:23.942 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:33:23.942 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:33:23.942 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:33:23.942 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:33:23.965 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:33:23.966 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:33:23.966 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:33:23.966 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:33:24.091 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:33:24.092 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:33:24.092 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:33:24.092 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:33:24.093 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:33:24.093 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:33:24.093 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:33:24.093 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:33:24.093 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:33:24.093 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:33:24.093 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:33:24.094 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:33:24.094 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:33:24.096 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:33:24.096 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:33:24.096 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:33:24.146 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 4 2024-04-06T09:33:24.146 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 6 2024-04-06T09:33:24.147 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:33:24.147 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=25, tag_tid=3, entry_tid=1], [object_number=24, tag_tid=3, entry_tid=0], [object_number=27, tag_tid=2, entry_tid=27223], [object_number=26, tag_tid=2, entry_tid=27222]]], state=connected] 2024-04-06T09:33:24.147 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=18, tag_tid=2, entry_tid=18346], [object_number=17, tag_tid=2, entry_tid=18345], [object_number=16, tag_tid=2, entry_tid=18344], [object_number=19, tag_tid=2, entry_tid=18343]]], state=connected] 2024-04-06T09:33:24.150 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:33:24.151 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=25, tag_tid=3, entry_tid=1' 2024-04-06T09:33:24.152 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:33:24.152 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:33:24.153 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:33:24.153 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:33:24.153 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:33:24.153 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:33:24.153 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:33:24.153 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:33:24.153 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:33:24.153 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:33:24.153 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:33:24.153 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:33:24.154 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:33:24.154 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:33:24.154 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:33:24.199 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 4 2024-04-06T09:33:24.199 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 6 2024-04-06T09:33:24.199 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:33:24.199 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=25, tag_tid=3, entry_tid=1], [object_number=24, tag_tid=3, entry_tid=0], [object_number=27, tag_tid=2, entry_tid=27223], [object_number=26, tag_tid=2, entry_tid=27222]]], state=connected] 2024-04-06T09:33:24.199 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=18, tag_tid=2, entry_tid=18346], [object_number=17, tag_tid=2, entry_tid=18345], [object_number=16, tag_tid=2, entry_tid=18344], [object_number=19, tag_tid=2, entry_tid=18343]]], state=connected] 2024-04-06T09:33:24.202 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:33:24.204 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=18, tag_tid=2, entry_tid=18346' 2024-04-06T09:33:24.205 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=25, tag_tid=3, entry_tid=1' -a 'object_number=25, tag_tid=3, entry_tid=1' = 'object_number=18, tag_tid=2, entry_tid=18346' 2024-04-06T09:33:24.205 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=18, tag_tid=2, entry_tid=18346' '!=' 'object_number=19, tag_tid=2, entry_tid=18223' 2024-04-06T09:33:24.205 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:33:24.205 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=18, tag_tid=2, entry_tid=18346' = 'object_number=19, tag_tid=2, entry_tid=18223' 2024-04-06T09:33:24.205 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=18, tag_tid=2, entry_tid=18346' 2024-04-06T09:33:24.205 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=25, tag_tid=3, entry_tid=1' 2024-04-06T09:33:24.205 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:33:24.205 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:33:24.206 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=3 2024-04-06T09:33:24.206 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=18, tag_tid=2, entry_tid=18346' 2024-04-06T09:33:24.206 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:33:24.207 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:33:24.207 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=2 2024-04-06T09:33:24.208 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=25, tag_tid=3, entry_tid=1' 2024-04-06T09:33:24.208 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:33:24.208 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:33:24.209 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:33:24.210 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=18, tag_tid=2, entry_tid=18346' 2024-04-06T09:33:24.210 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:33:24.210 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:33:24.214 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=18346 2024-04-06T09:33:24.214 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 3 = 2 -a 1 -le 18346 2024-04-06T09:33:24.215 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:33:24.215 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:33:24.215 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:33:24.413 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:33:24.413 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:33:24.413 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:33:24.413 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:33:24.413 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:33:24.413 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:33:24.413 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:33:24.413 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:33:24.413 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:33:24.413 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:33:24.413 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:33:24.414 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:33:24.414 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:33:24.414 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:33:24.414 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:33:24.419 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:33:24.419 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:33:24.420 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:33:24.420 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:33:24.420 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:33:24.420 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:33:24.420 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:33:24.420 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:33:24.420 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:33:24.420 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:33:24.420 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:33:24.420 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:33:24.420 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:33:24.420 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:33:24.420 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:33:24.421 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:33:24.421 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:33:24.421 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:33:24.421 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:33:24.421 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:33:24.421 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:33:24.421 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:33:24.421 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:33:24.421 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:33:24.421 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:33:24.421 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:33:24.421 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:33:24.421 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:33:24.422 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:33:24.422 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:33:24.422 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:33:24.422 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:33:24.422 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:33:24.422 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:33:24.444 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:33:24.444 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:33:24.444 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:33:24.444 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:33:24.584 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:33:24.585 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:33:24.585 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:33:24.585 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:33:24.585 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:33:24.585 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:33:24.585 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:33:24.585 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:33:24.585 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:33:24.585 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:33:24.586 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:33:24.586 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:33:24.586 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:33:24.588 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:33:24.588 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:33:24.588 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:33:24.632 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 4 2024-04-06T09:33:24.632 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 6 2024-04-06T09:33:24.632 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:33:24.632 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=25, tag_tid=3, entry_tid=1], [object_number=24, tag_tid=3, entry_tid=0], [object_number=27, tag_tid=2, entry_tid=27223], [object_number=26, tag_tid=2, entry_tid=27222]]], state=connected] 2024-04-06T09:33:24.633 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=19, tag_tid=2, entry_tid=18475], [object_number=18, tag_tid=2, entry_tid=18474], [object_number=17, tag_tid=2, entry_tid=18473], [object_number=16, tag_tid=2, entry_tid=18472]]], state=connected] 2024-04-06T09:33:24.636 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:33:24.637 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=25, tag_tid=3, entry_tid=1' 2024-04-06T09:33:24.638 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:33:24.639 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:33:24.639 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:33:24.639 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:33:24.639 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:33:24.639 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:33:24.639 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:33:24.639 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:33:24.639 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:33:24.639 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:33:24.639 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:33:24.639 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:33:24.640 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:33:24.640 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:33:24.640 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:33:24.685 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 4 2024-04-06T09:33:24.685 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 6 2024-04-06T09:33:24.685 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:33:24.685 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=25, tag_tid=3, entry_tid=1], [object_number=24, tag_tid=3, entry_tid=0], [object_number=27, tag_tid=2, entry_tid=27223], [object_number=26, tag_tid=2, entry_tid=27222]]], state=connected] 2024-04-06T09:33:24.685 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=19, tag_tid=2, entry_tid=18475], [object_number=18, tag_tid=2, entry_tid=18474], [object_number=17, tag_tid=2, entry_tid=18473], [object_number=16, tag_tid=2, entry_tid=18472]]], state=connected] 2024-04-06T09:33:24.688 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:33:24.690 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=19, tag_tid=2, entry_tid=18475' 2024-04-06T09:33:24.690 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=25, tag_tid=3, entry_tid=1' -a 'object_number=25, tag_tid=3, entry_tid=1' = 'object_number=19, tag_tid=2, entry_tid=18475' 2024-04-06T09:33:24.690 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=19, tag_tid=2, entry_tid=18475' '!=' 'object_number=18, tag_tid=2, entry_tid=18346' 2024-04-06T09:33:24.690 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:33:24.691 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=19, tag_tid=2, entry_tid=18475' = 'object_number=18, tag_tid=2, entry_tid=18346' 2024-04-06T09:33:24.691 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=19, tag_tid=2, entry_tid=18475' 2024-04-06T09:33:24.691 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=25, tag_tid=3, entry_tid=1' 2024-04-06T09:33:24.691 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:33:24.691 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:33:24.691 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=3 2024-04-06T09:33:24.692 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=19, tag_tid=2, entry_tid=18475' 2024-04-06T09:33:24.692 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:33:24.692 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:33:24.693 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=2 2024-04-06T09:33:24.694 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=25, tag_tid=3, entry_tid=1' 2024-04-06T09:33:24.694 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:33:24.694 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:33:24.694 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:33:24.695 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=19, tag_tid=2, entry_tid=18475' 2024-04-06T09:33:24.695 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:33:24.696 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:33:24.696 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=18475 2024-04-06T09:33:24.697 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 3 = 2 -a 1 -le 18475 2024-04-06T09:33:24.697 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:33:24.697 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:33:24.697 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:33:24.898 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:33:24.899 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:33:24.899 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:33:24.899 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:33:24.899 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:33:24.899 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:33:24.899 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:33:24.899 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:33:24.899 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:33:24.899 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:33:24.899 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:33:24.899 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:33:24.900 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:33:24.900 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:33:24.900 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:33:24.900 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:33:24.900 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:33:24.903 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:33:24.903 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:33:24.903 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:33:24.903 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:33:24.904 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:33:24.904 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:33:24.904 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:33:24.904 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:33:24.904 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:33:24.904 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:33:24.904 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:33:24.904 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:33:24.904 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:33:24.904 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:33:24.904 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:33:24.904 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:33:24.905 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:33:24.905 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:33:24.906 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:33:24.906 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:33:24.906 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:33:24.906 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:33:24.906 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:33:24.906 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:33:24.906 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:33:24.906 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:33:24.906 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:33:24.906 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:33:24.906 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:33:24.906 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:33:24.907 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:33:24.907 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:33:24.930 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:33:24.930 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:33:24.930 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:33:24.930 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:33:25.062 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:33:25.063 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:33:25.063 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:33:25.063 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:33:25.063 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:33:25.063 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:33:25.063 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:33:25.063 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:33:25.064 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:33:25.064 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:33:25.064 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:33:25.064 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:33:25.064 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:33:25.066 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:33:25.067 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:33:25.067 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:33:25.118 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 4 2024-04-06T09:33:25.118 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 6 2024-04-06T09:33:25.118 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:33:25.118 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=25, tag_tid=3, entry_tid=1], [object_number=24, tag_tid=3, entry_tid=0], [object_number=27, tag_tid=2, entry_tid=27223], [object_number=26, tag_tid=2, entry_tid=27222]]], state=connected] 2024-04-06T09:33:25.118 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=17, tag_tid=2, entry_tid=18597], [object_number=16, tag_tid=2, entry_tid=18596], [object_number=19, tag_tid=2, entry_tid=18595], [object_number=18, tag_tid=2, entry_tid=18594]]], state=connected] 2024-04-06T09:33:25.122 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:33:25.123 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=25, tag_tid=3, entry_tid=1' 2024-04-06T09:33:25.124 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:33:25.124 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:33:25.124 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:33:25.124 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:33:25.125 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:33:25.125 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:33:25.125 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:33:25.125 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:33:25.125 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:33:25.125 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:33:25.125 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:33:25.125 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:33:25.126 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:33:25.126 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:33:25.126 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:33:25.170 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 4 2024-04-06T09:33:25.170 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 6 2024-04-06T09:33:25.170 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:33:25.170 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=25, tag_tid=3, entry_tid=1], [object_number=24, tag_tid=3, entry_tid=0], [object_number=27, tag_tid=2, entry_tid=27223], [object_number=26, tag_tid=2, entry_tid=27222]]], state=connected] 2024-04-06T09:33:25.171 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=17, tag_tid=2, entry_tid=18597], [object_number=16, tag_tid=2, entry_tid=18596], [object_number=19, tag_tid=2, entry_tid=18595], [object_number=18, tag_tid=2, entry_tid=18594]]], state=connected] 2024-04-06T09:33:25.174 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:33:25.176 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=17, tag_tid=2, entry_tid=18597' 2024-04-06T09:33:25.176 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=25, tag_tid=3, entry_tid=1' -a 'object_number=25, tag_tid=3, entry_tid=1' = 'object_number=17, tag_tid=2, entry_tid=18597' 2024-04-06T09:33:25.176 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=17, tag_tid=2, entry_tid=18597' '!=' 'object_number=19, tag_tid=2, entry_tid=18475' 2024-04-06T09:33:25.176 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:33:25.176 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=17, tag_tid=2, entry_tid=18597' = 'object_number=19, tag_tid=2, entry_tid=18475' 2024-04-06T09:33:25.176 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=17, tag_tid=2, entry_tid=18597' 2024-04-06T09:33:25.177 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=25, tag_tid=3, entry_tid=1' 2024-04-06T09:33:25.177 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:33:25.177 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:33:25.178 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=3 2024-04-06T09:33:25.178 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=17, tag_tid=2, entry_tid=18597' 2024-04-06T09:33:25.178 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:33:25.178 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:33:25.179 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=2 2024-04-06T09:33:25.180 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=25, tag_tid=3, entry_tid=1' 2024-04-06T09:33:25.180 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:33:25.180 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:33:25.181 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:33:25.181 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=17, tag_tid=2, entry_tid=18597' 2024-04-06T09:33:25.181 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:33:25.181 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:33:25.182 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=18597 2024-04-06T09:33:25.183 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 3 = 2 -a 1 -le 18597 2024-04-06T09:33:25.183 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:33:25.183 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:33:25.183 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:33:25.384 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:33:25.385 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:33:25.385 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:33:25.385 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:33:25.385 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:33:25.385 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:33:25.385 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:33:25.385 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:33:25.385 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:33:25.385 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:33:25.385 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:33:25.385 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:33:25.386 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:33:25.386 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:33:25.386 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:33:25.386 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:33:25.386 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:33:25.388 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:33:25.388 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:33:25.388 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:33:25.388 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:33:25.388 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:33:25.388 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:33:25.388 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:33:25.388 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:33:25.388 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:33:25.388 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:33:25.388 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:33:25.388 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:33:25.388 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:33:25.389 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:33:25.389 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:33:25.389 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:33:25.389 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:33:25.390 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:33:25.390 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:33:25.390 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:33:25.390 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:33:25.390 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:33:25.390 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:33:25.390 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:33:25.390 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:33:25.390 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:33:25.390 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:33:25.390 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:33:25.390 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:33:25.390 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:33:25.391 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:33:25.391 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:33:25.414 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:33:25.415 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:33:25.415 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:33:25.415 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:33:25.690 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:33:25.691 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:33:25.691 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:33:25.692 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:33:25.692 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:33:25.692 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:33:25.692 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:33:25.692 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:33:25.692 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:33:25.692 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:33:25.692 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:33:25.693 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:33:25.693 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:33:25.695 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:33:25.695 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:33:25.695 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:33:25.745 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 4 2024-04-06T09:33:25.745 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 6 2024-04-06T09:33:25.745 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:33:25.745 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=25, tag_tid=3, entry_tid=1], [object_number=24, tag_tid=3, entry_tid=0], [object_number=27, tag_tid=2, entry_tid=27223], [object_number=26, tag_tid=2, entry_tid=27222]]], state=connected] 2024-04-06T09:33:25.745 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=18, tag_tid=2, entry_tid=18702], [object_number=17, tag_tid=2, entry_tid=18701], [object_number=16, tag_tid=2, entry_tid=18700], [object_number=19, tag_tid=2, entry_tid=18699]]], state=connected] 2024-04-06T09:33:25.749 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:33:25.750 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=25, tag_tid=3, entry_tid=1' 2024-04-06T09:33:25.752 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:33:25.752 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:33:25.752 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:33:25.752 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:33:25.752 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:33:25.752 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:33:25.752 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:33:25.752 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:33:25.752 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:33:25.752 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:33:25.752 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:33:25.753 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:33:25.753 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:33:25.753 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:33:25.753 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:33:25.800 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 4 2024-04-06T09:33:25.800 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 6 2024-04-06T09:33:25.800 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:33:25.800 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=25, tag_tid=3, entry_tid=1], [object_number=24, tag_tid=3, entry_tid=0], [object_number=27, tag_tid=2, entry_tid=27223], [object_number=26, tag_tid=2, entry_tid=27222]]], state=connected] 2024-04-06T09:33:25.800 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=18, tag_tid=2, entry_tid=18702], [object_number=17, tag_tid=2, entry_tid=18701], [object_number=16, tag_tid=2, entry_tid=18700], [object_number=19, tag_tid=2, entry_tid=18699]]], state=connected] 2024-04-06T09:33:25.804 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:33:25.806 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=18, tag_tid=2, entry_tid=18702' 2024-04-06T09:33:25.806 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=25, tag_tid=3, entry_tid=1' -a 'object_number=25, tag_tid=3, entry_tid=1' = 'object_number=18, tag_tid=2, entry_tid=18702' 2024-04-06T09:33:25.806 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=18, tag_tid=2, entry_tid=18702' '!=' 'object_number=17, tag_tid=2, entry_tid=18597' 2024-04-06T09:33:25.806 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:33:25.806 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=18, tag_tid=2, entry_tid=18702' = 'object_number=17, tag_tid=2, entry_tid=18597' 2024-04-06T09:33:25.806 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=18, tag_tid=2, entry_tid=18702' 2024-04-06T09:33:25.806 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=25, tag_tid=3, entry_tid=1' 2024-04-06T09:33:25.807 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:33:25.807 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:33:25.807 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=3 2024-04-06T09:33:25.808 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=18, tag_tid=2, entry_tid=18702' 2024-04-06T09:33:25.808 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:33:25.808 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:33:25.809 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=2 2024-04-06T09:33:25.810 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=25, tag_tid=3, entry_tid=1' 2024-04-06T09:33:25.810 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:33:25.810 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:33:25.811 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:33:25.811 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=18, tag_tid=2, entry_tid=18702' 2024-04-06T09:33:25.811 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:33:25.812 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:33:25.813 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=18702 2024-04-06T09:33:25.813 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 3 = 2 -a 1 -le 18702 2024-04-06T09:33:25.813 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:33:25.813 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:33:25.813 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:33:26.014 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:33:26.015 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:33:26.015 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:33:26.015 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:33:26.015 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:33:26.015 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:33:26.015 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:33:26.015 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:33:26.015 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:33:26.015 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:33:26.015 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:33:26.015 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:33:26.015 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:33:26.015 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:33:26.015 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:33:26.016 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:33:26.016 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:33:26.019 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:33:26.019 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:33:26.020 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:33:26.020 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:33:26.020 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:33:26.020 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:33:26.020 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:33:26.020 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:33:26.020 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:33:26.020 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:33:26.020 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:33:26.020 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:33:26.020 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:33:26.020 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:33:26.020 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:33:26.020 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:33:26.020 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:33:26.021 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:33:26.021 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:33:26.021 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:33:26.021 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:33:26.021 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:33:26.021 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:33:26.021 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:33:26.021 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:33:26.021 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:33:26.021 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:33:26.021 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:33:26.021 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:33:26.021 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:33:26.021 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:33:26.021 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:33:26.044 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:33:26.045 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:33:26.045 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:33:26.045 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:33:26.171 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:33:26.172 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:33:26.172 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:33:26.172 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:33:26.172 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:33:26.172 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:33:26.172 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:33:26.172 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:33:26.172 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:33:26.172 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:33:26.173 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:33:26.173 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:33:26.173 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:33:26.175 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:33:26.175 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:33:26.175 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:33:26.220 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 4 2024-04-06T09:33:26.220 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 6 2024-04-06T09:33:26.220 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:33:26.220 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=25, tag_tid=3, entry_tid=1], [object_number=24, tag_tid=3, entry_tid=0], [object_number=27, tag_tid=2, entry_tid=27223], [object_number=26, tag_tid=2, entry_tid=27222]]], state=connected] 2024-04-06T09:33:26.220 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=16, tag_tid=2, entry_tid=18808], [object_number=19, tag_tid=2, entry_tid=18807], [object_number=18, tag_tid=2, entry_tid=18806], [object_number=17, tag_tid=2, entry_tid=18805]]], state=connected] 2024-04-06T09:33:26.223 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:33:26.224 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=25, tag_tid=3, entry_tid=1' 2024-04-06T09:33:26.226 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:33:26.226 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:33:26.226 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:33:26.226 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:33:26.226 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:33:26.226 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:33:26.226 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:33:26.226 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:33:26.226 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:33:26.226 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:33:26.226 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:33:26.226 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:33:26.227 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:33:26.227 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:33:26.227 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:33:26.271 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 4 2024-04-06T09:33:26.271 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 6 2024-04-06T09:33:26.271 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:33:26.271 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=25, tag_tid=3, entry_tid=1], [object_number=24, tag_tid=3, entry_tid=0], [object_number=27, tag_tid=2, entry_tid=27223], [object_number=26, tag_tid=2, entry_tid=27222]]], state=connected] 2024-04-06T09:33:26.272 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=16, tag_tid=2, entry_tid=18808], [object_number=19, tag_tid=2, entry_tid=18807], [object_number=18, tag_tid=2, entry_tid=18806], [object_number=17, tag_tid=2, entry_tid=18805]]], state=connected] 2024-04-06T09:33:26.274 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:33:26.276 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=16, tag_tid=2, entry_tid=18808' 2024-04-06T09:33:26.277 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=25, tag_tid=3, entry_tid=1' -a 'object_number=25, tag_tid=3, entry_tid=1' = 'object_number=16, tag_tid=2, entry_tid=18808' 2024-04-06T09:33:26.277 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=16, tag_tid=2, entry_tid=18808' '!=' 'object_number=18, tag_tid=2, entry_tid=18702' 2024-04-06T09:33:26.277 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:33:26.277 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=16, tag_tid=2, entry_tid=18808' = 'object_number=18, tag_tid=2, entry_tid=18702' 2024-04-06T09:33:26.277 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=16, tag_tid=2, entry_tid=18808' 2024-04-06T09:33:26.277 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=25, tag_tid=3, entry_tid=1' 2024-04-06T09:33:26.277 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:33:26.277 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:33:26.278 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=3 2024-04-06T09:33:26.278 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=16, tag_tid=2, entry_tid=18808' 2024-04-06T09:33:26.278 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:33:26.278 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:33:26.279 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=2 2024-04-06T09:33:26.280 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=25, tag_tid=3, entry_tid=1' 2024-04-06T09:33:26.280 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:33:26.280 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:33:26.281 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:33:26.282 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=16, tag_tid=2, entry_tid=18808' 2024-04-06T09:33:26.282 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:33:26.282 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:33:26.283 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=18808 2024-04-06T09:33:26.283 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 3 = 2 -a 1 -le 18808 2024-04-06T09:33:26.283 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:33:26.283 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:33:26.283 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:33:26.485 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:33:26.485 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:33:26.485 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:33:26.485 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:33:26.485 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:33:26.485 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:33:26.485 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:33:26.485 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:33:26.486 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:33:26.486 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:33:26.486 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:33:26.486 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:33:26.486 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:33:26.486 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:33:26.486 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:33:26.486 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:33:26.487 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:33:26.489 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:33:26.489 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:33:26.489 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:33:26.489 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:33:26.489 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:33:26.489 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:33:26.489 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:33:26.489 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:33:26.489 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:33:26.489 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:33:26.489 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:33:26.489 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:33:26.489 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:33:26.490 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:33:26.490 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:33:26.490 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:33:26.491 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:33:26.491 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:33:26.491 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:33:26.491 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:33:26.491 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:33:26.492 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:33:26.492 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:33:26.492 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:33:26.492 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:33:26.492 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:33:26.492 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:33:26.492 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:33:26.492 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:33:26.492 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:33:26.492 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:33:26.492 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:33:26.517 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:33:26.517 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:33:26.517 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:33:26.517 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:33:26.644 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:33:26.645 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:33:26.645 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:33:26.645 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:33:26.645 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:33:26.645 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:33:26.645 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:33:26.645 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:33:26.646 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:33:26.646 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:33:26.646 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:33:26.646 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:33:26.646 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:33:26.647 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:33:26.648 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:33:26.648 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:33:26.694 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 4 2024-04-06T09:33:26.695 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 6 2024-04-06T09:33:26.695 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:33:26.695 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=25, tag_tid=3, entry_tid=1], [object_number=24, tag_tid=3, entry_tid=0], [object_number=27, tag_tid=2, entry_tid=27223], [object_number=26, tag_tid=2, entry_tid=27222]]], state=connected] 2024-04-06T09:33:26.695 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=19, tag_tid=2, entry_tid=18915], [object_number=18, tag_tid=2, entry_tid=18914], [object_number=17, tag_tid=2, entry_tid=18913], [object_number=16, tag_tid=2, entry_tid=18912]]], state=connected] 2024-04-06T09:33:26.698 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:33:26.699 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=25, tag_tid=3, entry_tid=1' 2024-04-06T09:33:26.701 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:33:26.701 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:33:26.701 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:33:26.701 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:33:26.701 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:33:26.701 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:33:26.701 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:33:26.701 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:33:26.701 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:33:26.701 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:33:26.701 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:33:26.702 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:33:26.702 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:33:26.702 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:33:26.703 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:33:26.747 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 4 2024-04-06T09:33:26.747 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 6 2024-04-06T09:33:26.748 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:33:26.748 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=25, tag_tid=3, entry_tid=1], [object_number=24, tag_tid=3, entry_tid=0], [object_number=27, tag_tid=2, entry_tid=27223], [object_number=26, tag_tid=2, entry_tid=27222]]], state=connected] 2024-04-06T09:33:26.748 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=19, tag_tid=2, entry_tid=18915], [object_number=18, tag_tid=2, entry_tid=18914], [object_number=17, tag_tid=2, entry_tid=18913], [object_number=16, tag_tid=2, entry_tid=18912]]], state=connected] 2024-04-06T09:33:26.751 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:33:26.753 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=19, tag_tid=2, entry_tid=18915' 2024-04-06T09:33:26.753 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=25, tag_tid=3, entry_tid=1' -a 'object_number=25, tag_tid=3, entry_tid=1' = 'object_number=19, tag_tid=2, entry_tid=18915' 2024-04-06T09:33:26.753 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=19, tag_tid=2, entry_tid=18915' '!=' 'object_number=16, tag_tid=2, entry_tid=18808' 2024-04-06T09:33:26.753 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:33:26.754 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=19, tag_tid=2, entry_tid=18915' = 'object_number=16, tag_tid=2, entry_tid=18808' 2024-04-06T09:33:26.754 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=19, tag_tid=2, entry_tid=18915' 2024-04-06T09:33:26.754 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=25, tag_tid=3, entry_tid=1' 2024-04-06T09:33:26.754 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:33:26.754 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:33:26.754 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=3 2024-04-06T09:33:26.755 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=19, tag_tid=2, entry_tid=18915' 2024-04-06T09:33:26.755 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:33:26.755 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:33:26.756 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=2 2024-04-06T09:33:26.757 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=25, tag_tid=3, entry_tid=1' 2024-04-06T09:33:26.757 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:33:26.757 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:33:26.758 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:33:26.759 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=19, tag_tid=2, entry_tid=18915' 2024-04-06T09:33:26.759 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:33:26.759 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:33:26.760 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=18915 2024-04-06T09:33:26.760 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 3 = 2 -a 1 -le 18915 2024-04-06T09:33:26.760 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:33:26.760 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:33:26.760 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:33:26.961 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:33:26.961 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:33:26.961 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:33:26.962 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:33:26.962 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:33:26.962 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:33:26.962 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:33:26.962 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:33:26.962 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:33:26.962 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:33:26.962 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:33:26.962 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:33:26.962 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:33:26.962 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:33:26.962 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:33:26.963 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:33:26.963 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:33:26.964 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:33:26.964 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:33:26.964 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:33:26.964 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:33:26.964 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:33:26.964 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:33:26.964 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:33:26.965 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:33:26.965 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:33:26.965 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:33:26.965 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:33:26.965 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:33:26.965 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:33:26.965 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:33:26.965 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:33:26.965 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:33:26.967 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:33:26.967 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:33:26.967 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:33:26.967 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:33:26.967 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:33:26.967 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:33:26.967 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:33:26.968 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:33:26.968 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:33:26.968 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:33:26.968 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:33:26.968 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:33:26.968 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:33:26.968 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:33:26.968 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:33:26.968 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:33:26.991 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:33:26.992 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:33:26.992 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:33:26.992 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:33:27.119 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:33:27.119 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:33:27.119 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:33:27.119 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:33:27.119 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:33:27.119 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:33:27.119 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:33:27.119 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:33:27.120 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:33:27.120 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:33:27.120 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:33:27.120 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:33:27.120 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:33:27.122 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:33:27.123 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:33:27.123 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:33:27.180 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 4 2024-04-06T09:33:27.180 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 6 2024-04-06T09:33:27.180 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:33:27.180 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=25, tag_tid=3, entry_tid=1], [object_number=24, tag_tid=3, entry_tid=0], [object_number=27, tag_tid=2, entry_tid=27223], [object_number=26, tag_tid=2, entry_tid=27222]]], state=connected] 2024-04-06T09:33:27.180 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=18, tag_tid=2, entry_tid=19026], [object_number=17, tag_tid=2, entry_tid=19025], [object_number=16, tag_tid=2, entry_tid=19024], [object_number=19, tag_tid=2, entry_tid=19023]]], state=connected] 2024-04-06T09:33:27.183 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:33:27.185 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=25, tag_tid=3, entry_tid=1' 2024-04-06T09:33:27.186 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:33:27.186 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:33:27.186 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:33:27.186 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:33:27.187 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:33:27.187 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:33:27.187 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:33:27.187 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:33:27.187 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:33:27.187 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:33:27.187 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:33:27.187 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:33:27.188 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:33:27.188 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:33:27.188 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:33:27.234 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 4 2024-04-06T09:33:27.234 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 6 2024-04-06T09:33:27.234 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:33:27.235 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=25, tag_tid=3, entry_tid=1], [object_number=24, tag_tid=3, entry_tid=0], [object_number=27, tag_tid=2, entry_tid=27223], [object_number=26, tag_tid=2, entry_tid=27222]]], state=connected] 2024-04-06T09:33:27.235 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=18, tag_tid=2, entry_tid=19026], [object_number=17, tag_tid=2, entry_tid=19025], [object_number=16, tag_tid=2, entry_tid=19024], [object_number=19, tag_tid=2, entry_tid=19023]]], state=connected] 2024-04-06T09:33:27.238 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:33:27.240 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=18, tag_tid=2, entry_tid=19026' 2024-04-06T09:33:27.240 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=25, tag_tid=3, entry_tid=1' -a 'object_number=25, tag_tid=3, entry_tid=1' = 'object_number=18, tag_tid=2, entry_tid=19026' 2024-04-06T09:33:27.240 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=18, tag_tid=2, entry_tid=19026' '!=' 'object_number=19, tag_tid=2, entry_tid=18915' 2024-04-06T09:33:27.241 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:33:27.241 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=18, tag_tid=2, entry_tid=19026' = 'object_number=19, tag_tid=2, entry_tid=18915' 2024-04-06T09:33:27.241 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=18, tag_tid=2, entry_tid=19026' 2024-04-06T09:33:27.241 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=25, tag_tid=3, entry_tid=1' 2024-04-06T09:33:27.241 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:33:27.241 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:33:27.241 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=3 2024-04-06T09:33:27.242 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=18, tag_tid=2, entry_tid=19026' 2024-04-06T09:33:27.242 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:33:27.242 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:33:27.243 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=2 2024-04-06T09:33:27.244 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=25, tag_tid=3, entry_tid=1' 2024-04-06T09:33:27.244 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:33:27.244 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:33:27.245 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:33:27.246 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=18, tag_tid=2, entry_tid=19026' 2024-04-06T09:33:27.246 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:33:27.246 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:33:27.247 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=19026 2024-04-06T09:33:27.247 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 3 = 2 -a 1 -le 19026 2024-04-06T09:33:27.247 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:33:27.247 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:33:27.247 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:33:27.448 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:33:27.448 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:33:27.448 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:33:27.448 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:33:27.448 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:33:27.449 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:33:27.449 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:33:27.449 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:33:27.449 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:33:27.449 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:33:27.449 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:33:27.449 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:33:27.449 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:33:27.449 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:33:27.449 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:33:27.450 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:33:27.450 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:33:27.453 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:33:27.453 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:33:27.453 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:33:27.453 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:33:27.453 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:33:27.453 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:33:27.453 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:33:27.453 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:33:27.453 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:33:27.453 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:33:27.454 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:33:27.454 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:33:27.454 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:33:27.454 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:33:27.454 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:33:27.454 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:33:27.454 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:33:27.454 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:33:27.454 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:33:27.454 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:33:27.454 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:33:27.454 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:33:27.455 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:33:27.455 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:33:27.455 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:33:27.455 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:33:27.455 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:33:27.455 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:33:27.455 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:33:27.455 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:33:27.455 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:33:27.455 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:33:27.480 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:33:27.481 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:33:27.481 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:33:27.481 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:33:27.609 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:33:27.610 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:33:27.610 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:33:27.610 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:33:27.610 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:33:27.610 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:33:27.610 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:33:27.610 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:33:27.611 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:33:27.611 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:33:27.611 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:33:27.611 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:33:27.611 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:33:27.612 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:33:27.612 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:33:27.613 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:33:27.663 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 4 2024-04-06T09:33:27.664 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 6 2024-04-06T09:33:27.664 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:33:27.664 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=25, tag_tid=3, entry_tid=1], [object_number=24, tag_tid=3, entry_tid=0], [object_number=27, tag_tid=2, entry_tid=27223], [object_number=26, tag_tid=2, entry_tid=27222]]], state=connected] 2024-04-06T09:33:27.664 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=16, tag_tid=2, entry_tid=19136], [object_number=19, tag_tid=2, entry_tid=19135], [object_number=18, tag_tid=2, entry_tid=19134], [object_number=17, tag_tid=2, entry_tid=19133]]], state=connected] 2024-04-06T09:33:27.667 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:33:27.668 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=25, tag_tid=3, entry_tid=1' 2024-04-06T09:33:27.670 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:33:27.670 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:33:27.670 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:33:27.670 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:33:27.670 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:33:27.670 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:33:27.670 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:33:27.670 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:33:27.670 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:33:27.670 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:33:27.670 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:33:27.671 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:33:27.671 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:33:27.671 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:33:27.671 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:33:27.716 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 4 2024-04-06T09:33:27.716 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 6 2024-04-06T09:33:27.717 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:33:27.717 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=25, tag_tid=3, entry_tid=1], [object_number=24, tag_tid=3, entry_tid=0], [object_number=27, tag_tid=2, entry_tid=27223], [object_number=26, tag_tid=2, entry_tid=27222]]], state=connected] 2024-04-06T09:33:27.717 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=16, tag_tid=2, entry_tid=19136], [object_number=19, tag_tid=2, entry_tid=19135], [object_number=18, tag_tid=2, entry_tid=19134], [object_number=17, tag_tid=2, entry_tid=19133]]], state=connected] 2024-04-06T09:33:27.720 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:33:27.722 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=16, tag_tid=2, entry_tid=19136' 2024-04-06T09:33:27.722 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=25, tag_tid=3, entry_tid=1' -a 'object_number=25, tag_tid=3, entry_tid=1' = 'object_number=16, tag_tid=2, entry_tid=19136' 2024-04-06T09:33:27.722 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=16, tag_tid=2, entry_tid=19136' '!=' 'object_number=18, tag_tid=2, entry_tid=19026' 2024-04-06T09:33:27.723 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:33:27.723 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=16, tag_tid=2, entry_tid=19136' = 'object_number=18, tag_tid=2, entry_tid=19026' 2024-04-06T09:33:27.723 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=16, tag_tid=2, entry_tid=19136' 2024-04-06T09:33:27.723 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=25, tag_tid=3, entry_tid=1' 2024-04-06T09:33:27.723 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:33:27.723 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:33:27.723 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=3 2024-04-06T09:33:27.724 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=16, tag_tid=2, entry_tid=19136' 2024-04-06T09:33:27.724 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:33:27.724 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:33:27.725 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=2 2024-04-06T09:33:27.726 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=25, tag_tid=3, entry_tid=1' 2024-04-06T09:33:27.726 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:33:27.726 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:33:27.726 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:33:27.727 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=16, tag_tid=2, entry_tid=19136' 2024-04-06T09:33:27.727 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:33:27.727 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:33:27.728 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=19136 2024-04-06T09:33:27.729 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 3 = 2 -a 1 -le 19136 2024-04-06T09:33:27.729 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:33:27.729 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:33:27.729 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:33:27.931 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:33:27.931 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:33:27.931 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:33:27.931 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:33:27.931 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:33:27.931 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:33:27.931 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:33:27.931 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:33:27.931 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:33:27.931 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:33:27.932 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:33:27.932 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:33:27.932 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:33:27.932 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:33:27.932 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:33:27.932 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:33:27.932 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:33:27.936 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:33:27.936 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:33:27.936 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:33:27.936 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:33:27.936 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:33:27.936 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:33:27.936 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:33:27.936 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:33:27.936 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:33:27.936 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:33:27.937 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:33:27.937 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:33:27.937 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:33:27.937 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:33:27.937 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:33:27.937 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:33:27.937 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:33:27.937 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:33:27.937 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:33:27.937 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:33:27.937 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:33:27.937 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:33:27.937 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:33:27.937 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:33:27.937 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:33:27.938 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:33:27.938 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:33:27.938 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:33:27.938 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:33:27.938 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:33:27.938 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:33:27.938 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:33:27.960 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:33:27.961 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:33:27.961 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:33:27.961 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:33:28.082 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:33:28.083 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:33:28.084 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:33:28.084 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:33:28.084 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:33:28.084 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:33:28.084 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:33:28.084 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:33:28.084 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:33:28.084 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:33:28.084 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:33:28.085 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:33:28.085 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:33:28.087 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:33:28.088 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:33:28.088 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:33:28.137 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 4 2024-04-06T09:33:28.137 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 6 2024-04-06T09:33:28.137 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:33:28.137 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=25, tag_tid=3, entry_tid=1], [object_number=24, tag_tid=3, entry_tid=0], [object_number=27, tag_tid=2, entry_tid=27223], [object_number=26, tag_tid=2, entry_tid=27222]]], state=connected] 2024-04-06T09:33:28.137 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=19, tag_tid=2, entry_tid=19255], [object_number=18, tag_tid=2, entry_tid=19254], [object_number=17, tag_tid=2, entry_tid=19253], [object_number=16, tag_tid=2, entry_tid=19252]]], state=connected] 2024-04-06T09:33:28.140 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:33:28.142 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=25, tag_tid=3, entry_tid=1' 2024-04-06T09:33:28.143 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:33:28.143 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:33:28.143 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:33:28.143 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:33:28.143 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:33:28.143 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:33:28.143 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:33:28.144 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:33:28.144 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:33:28.144 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:33:28.144 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:33:28.144 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:33:28.145 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:33:28.145 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:33:28.145 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:33:28.190 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 4 2024-04-06T09:33:28.190 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 6 2024-04-06T09:33:28.190 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:33:28.190 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=25, tag_tid=3, entry_tid=1], [object_number=24, tag_tid=3, entry_tid=0], [object_number=27, tag_tid=2, entry_tid=27223], [object_number=26, tag_tid=2, entry_tid=27222]]], state=connected] 2024-04-06T09:33:28.190 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=19, tag_tid=2, entry_tid=19255], [object_number=18, tag_tid=2, entry_tid=19254], [object_number=17, tag_tid=2, entry_tid=19253], [object_number=16, tag_tid=2, entry_tid=19252]]], state=connected] 2024-04-06T09:33:28.193 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:33:28.196 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=19, tag_tid=2, entry_tid=19255' 2024-04-06T09:33:28.196 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=25, tag_tid=3, entry_tid=1' -a 'object_number=25, tag_tid=3, entry_tid=1' = 'object_number=19, tag_tid=2, entry_tid=19255' 2024-04-06T09:33:28.196 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=19, tag_tid=2, entry_tid=19255' '!=' 'object_number=16, tag_tid=2, entry_tid=19136' 2024-04-06T09:33:28.196 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:33:28.196 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=19, tag_tid=2, entry_tid=19255' = 'object_number=16, tag_tid=2, entry_tid=19136' 2024-04-06T09:33:28.196 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=19, tag_tid=2, entry_tid=19255' 2024-04-06T09:33:28.196 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=25, tag_tid=3, entry_tid=1' 2024-04-06T09:33:28.196 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:33:28.196 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:33:28.197 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=3 2024-04-06T09:33:28.197 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=19, tag_tid=2, entry_tid=19255' 2024-04-06T09:33:28.198 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:33:28.198 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:33:28.198 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=2 2024-04-06T09:33:28.199 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=25, tag_tid=3, entry_tid=1' 2024-04-06T09:33:28.199 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:33:28.199 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:33:28.200 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:33:28.201 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=19, tag_tid=2, entry_tid=19255' 2024-04-06T09:33:28.201 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:33:28.201 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:33:28.202 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=19255 2024-04-06T09:33:28.202 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 3 = 2 -a 1 -le 19255 2024-04-06T09:33:28.202 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:33:28.202 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:33:28.202 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:33:28.404 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:33:28.404 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:33:28.404 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:33:28.404 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:33:28.404 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:33:28.404 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:33:28.404 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:33:28.404 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:33:28.404 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:33:28.404 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:33:28.404 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:33:28.405 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:33:28.405 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:33:28.405 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:33:28.405 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:33:28.405 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:33:28.405 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:33:28.408 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:33:28.408 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:33:28.408 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:33:28.408 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:33:28.409 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:33:28.409 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:33:28.409 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:33:28.409 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:33:28.409 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:33:28.409 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:33:28.409 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:33:28.409 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:33:28.409 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:33:28.409 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:33:28.409 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:33:28.409 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:33:28.409 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:33:28.409 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:33:28.410 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:33:28.410 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:33:28.410 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:33:28.410 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:33:28.410 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:33:28.410 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:33:28.410 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:33:28.410 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:33:28.410 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:33:28.410 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:33:28.410 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:33:28.410 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:33:28.410 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:33:28.411 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:33:28.435 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:33:28.436 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:33:28.436 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:33:28.436 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:33:28.565 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:33:28.566 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:33:28.567 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:33:28.567 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:33:28.567 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:33:28.567 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:33:28.567 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:33:28.567 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:33:28.567 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:33:28.567 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:33:28.568 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:33:28.568 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:33:28.568 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:33:28.569 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:33:28.569 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:33:28.569 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:33:28.619 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 4 2024-04-06T09:33:28.619 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 6 2024-04-06T09:33:28.619 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:33:28.619 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=25, tag_tid=3, entry_tid=1], [object_number=24, tag_tid=3, entry_tid=0], [object_number=27, tag_tid=2, entry_tid=27223], [object_number=26, tag_tid=2, entry_tid=27222]]], state=connected] 2024-04-06T09:33:28.619 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=16, tag_tid=2, entry_tid=19384], [object_number=19, tag_tid=2, entry_tid=19383], [object_number=18, tag_tid=2, entry_tid=19382], [object_number=17, tag_tid=2, entry_tid=19381]]], state=connected] 2024-04-06T09:33:28.622 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:33:28.624 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=25, tag_tid=3, entry_tid=1' 2024-04-06T09:33:28.625 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:33:28.625 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:33:28.625 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:33:28.625 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:33:28.625 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:33:28.625 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:33:28.625 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:33:28.626 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:33:28.626 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:33:28.626 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:33:28.626 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:33:28.626 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:33:28.627 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:33:28.627 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:33:28.627 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:33:28.673 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 4 2024-04-06T09:33:28.673 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 6 2024-04-06T09:33:28.673 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:33:28.673 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=25, tag_tid=3, entry_tid=1], [object_number=24, tag_tid=3, entry_tid=0], [object_number=27, tag_tid=2, entry_tid=27223], [object_number=26, tag_tid=2, entry_tid=27222]]], state=connected] 2024-04-06T09:33:28.673 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=16, tag_tid=2, entry_tid=19384], [object_number=19, tag_tid=2, entry_tid=19383], [object_number=18, tag_tid=2, entry_tid=19382], [object_number=17, tag_tid=2, entry_tid=19381]]], state=connected] 2024-04-06T09:33:28.676 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:33:28.678 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=16, tag_tid=2, entry_tid=19384' 2024-04-06T09:33:28.679 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=25, tag_tid=3, entry_tid=1' -a 'object_number=25, tag_tid=3, entry_tid=1' = 'object_number=16, tag_tid=2, entry_tid=19384' 2024-04-06T09:33:28.679 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=16, tag_tid=2, entry_tid=19384' '!=' 'object_number=19, tag_tid=2, entry_tid=19255' 2024-04-06T09:33:28.679 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:33:28.679 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=16, tag_tid=2, entry_tid=19384' = 'object_number=19, tag_tid=2, entry_tid=19255' 2024-04-06T09:33:28.679 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=16, tag_tid=2, entry_tid=19384' 2024-04-06T09:33:28.679 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=25, tag_tid=3, entry_tid=1' 2024-04-06T09:33:28.679 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:33:28.679 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:33:28.680 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=3 2024-04-06T09:33:28.680 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=16, tag_tid=2, entry_tid=19384' 2024-04-06T09:33:28.680 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:33:28.681 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:33:28.682 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=2 2024-04-06T09:33:28.682 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=25, tag_tid=3, entry_tid=1' 2024-04-06T09:33:28.682 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:33:28.682 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:33:28.683 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:33:28.684 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=16, tag_tid=2, entry_tid=19384' 2024-04-06T09:33:28.684 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:33:28.684 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:33:28.685 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=19384 2024-04-06T09:33:28.685 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 3 = 2 -a 1 -le 19384 2024-04-06T09:33:28.685 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:33:28.685 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:33:28.685 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:33:28.887 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:33:28.887 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:33:28.888 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:33:28.888 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:33:28.888 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:33:28.888 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:33:28.888 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:33:28.888 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:33:28.888 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:33:28.888 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:33:28.888 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:33:28.888 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:33:28.888 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:33:28.888 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:33:28.888 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:33:28.889 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:33:28.889 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:33:28.891 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:33:28.891 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:33:28.891 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:33:28.891 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:33:28.891 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:33:28.892 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:33:28.892 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:33:28.892 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:33:28.892 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:33:28.892 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:33:28.892 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:33:28.892 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:33:28.892 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:33:28.892 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:33:28.892 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:33:28.892 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:33:28.893 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:33:28.893 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:33:28.894 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:33:28.894 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:33:28.894 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:33:28.894 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:33:28.894 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:33:28.894 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:33:28.894 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:33:28.894 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:33:28.894 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:33:28.894 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:33:28.894 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:33:28.894 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:33:28.895 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:33:28.895 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:33:28.917 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:33:28.918 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:33:28.918 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:33:28.918 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:33:29.052 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:33:29.053 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:33:29.053 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:33:29.053 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:33:29.053 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:33:29.053 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:33:29.053 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:33:29.053 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:33:29.053 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:33:29.053 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:33:29.054 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:33:29.054 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:33:29.054 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:33:29.056 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:33:29.056 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:33:29.056 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:33:29.100 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 4 2024-04-06T09:33:29.100 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 6 2024-04-06T09:33:29.100 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:33:29.100 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=25, tag_tid=3, entry_tid=1], [object_number=24, tag_tid=3, entry_tid=0], [object_number=27, tag_tid=2, entry_tid=27223], [object_number=26, tag_tid=2, entry_tid=27222]]], state=connected] 2024-04-06T09:33:29.100 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=19, tag_tid=2, entry_tid=19507], [object_number=18, tag_tid=2, entry_tid=19506], [object_number=17, tag_tid=2, entry_tid=19505], [object_number=16, tag_tid=2, entry_tid=19504]]], state=connected] 2024-04-06T09:33:29.103 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:33:29.105 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=25, tag_tid=3, entry_tid=1' 2024-04-06T09:33:29.106 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:33:29.106 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:33:29.106 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:33:29.106 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:33:29.106 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:33:29.107 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:33:29.107 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:33:29.107 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:33:29.107 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:33:29.107 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:33:29.107 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:33:29.107 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:33:29.108 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:33:29.108 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:33:29.108 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:33:29.151 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 4 2024-04-06T09:33:29.151 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 6 2024-04-06T09:33:29.151 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:33:29.151 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=25, tag_tid=3, entry_tid=1], [object_number=24, tag_tid=3, entry_tid=0], [object_number=27, tag_tid=2, entry_tid=27223], [object_number=26, tag_tid=2, entry_tid=27222]]], state=connected] 2024-04-06T09:33:29.152 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=19, tag_tid=2, entry_tid=19507], [object_number=18, tag_tid=2, entry_tid=19506], [object_number=17, tag_tid=2, entry_tid=19505], [object_number=16, tag_tid=2, entry_tid=19504]]], state=connected] 2024-04-06T09:33:29.155 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:33:29.157 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=19, tag_tid=2, entry_tid=19507' 2024-04-06T09:33:29.157 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=25, tag_tid=3, entry_tid=1' -a 'object_number=25, tag_tid=3, entry_tid=1' = 'object_number=19, tag_tid=2, entry_tid=19507' 2024-04-06T09:33:29.157 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=19, tag_tid=2, entry_tid=19507' '!=' 'object_number=16, tag_tid=2, entry_tid=19384' 2024-04-06T09:33:29.157 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:33:29.157 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=19, tag_tid=2, entry_tid=19507' = 'object_number=16, tag_tid=2, entry_tid=19384' 2024-04-06T09:33:29.157 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=19, tag_tid=2, entry_tid=19507' 2024-04-06T09:33:29.157 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=25, tag_tid=3, entry_tid=1' 2024-04-06T09:33:29.157 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:33:29.158 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:33:29.158 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=3 2024-04-06T09:33:29.159 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=19, tag_tid=2, entry_tid=19507' 2024-04-06T09:33:29.159 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:33:29.159 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:33:29.160 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=2 2024-04-06T09:33:29.160 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=25, tag_tid=3, entry_tid=1' 2024-04-06T09:33:29.160 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:33:29.161 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:33:29.161 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:33:29.162 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=19, tag_tid=2, entry_tid=19507' 2024-04-06T09:33:29.162 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:33:29.162 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:33:29.163 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=19507 2024-04-06T09:33:29.163 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 3 = 2 -a 1 -le 19507 2024-04-06T09:33:29.163 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:33:29.163 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:33:29.164 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:33:29.365 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:33:29.365 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:33:29.365 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:33:29.366 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:33:29.366 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:33:29.366 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:33:29.366 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:33:29.366 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:33:29.366 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:33:29.366 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:33:29.366 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:33:29.366 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:33:29.366 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:33:29.366 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:33:29.367 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:33:29.367 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:33:29.367 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:33:29.369 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:33:29.369 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:33:29.369 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:33:29.369 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:33:29.369 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:33:29.369 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:33:29.369 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:33:29.370 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:33:29.370 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:33:29.370 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:33:29.370 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:33:29.370 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:33:29.370 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:33:29.370 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:33:29.370 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:33:29.370 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:33:29.372 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:33:29.372 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:33:29.372 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:33:29.372 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:33:29.372 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:33:29.372 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:33:29.372 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:33:29.372 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:33:29.373 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:33:29.373 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:33:29.373 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:33:29.373 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:33:29.373 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:33:29.373 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:33:29.373 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:33:29.373 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:33:29.396 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:33:29.396 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:33:29.396 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:33:29.397 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:33:29.528 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:33:29.530 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:33:29.530 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:33:29.530 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:33:29.530 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:33:29.530 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:33:29.531 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:33:29.531 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:33:29.531 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:33:29.531 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:33:29.531 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:33:29.531 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:33:29.532 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:33:29.533 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:33:29.533 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:33:29.533 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:33:29.791 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 4 2024-04-06T09:33:29.791 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 6 2024-04-06T09:33:29.791 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:33:29.792 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=25, tag_tid=3, entry_tid=1], [object_number=24, tag_tid=3, entry_tid=0], [object_number=27, tag_tid=2, entry_tid=27223], [object_number=26, tag_tid=2, entry_tid=27222]]], state=connected] 2024-04-06T09:33:29.792 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=19, tag_tid=2, entry_tid=19623], [object_number=18, tag_tid=2, entry_tid=19622], [object_number=17, tag_tid=2, entry_tid=19621], [object_number=16, tag_tid=2, entry_tid=19620]]], state=connected] 2024-04-06T09:33:29.796 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:33:29.797 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=25, tag_tid=3, entry_tid=1' 2024-04-06T09:33:29.798 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:33:29.798 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:33:29.798 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:33:29.798 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:33:29.799 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:33:29.799 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:33:29.799 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:33:29.799 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:33:29.799 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:33:29.799 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:33:29.799 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:33:29.799 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:33:29.800 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:33:29.800 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:33:29.801 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:33:29.851 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 4 2024-04-06T09:33:29.851 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 6 2024-04-06T09:33:29.851 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:33:29.851 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=25, tag_tid=3, entry_tid=1], [object_number=24, tag_tid=3, entry_tid=0], [object_number=27, tag_tid=2, entry_tid=27223], [object_number=26, tag_tid=2, entry_tid=27222]]], state=connected] 2024-04-06T09:33:29.851 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=19, tag_tid=2, entry_tid=19623], [object_number=18, tag_tid=2, entry_tid=19622], [object_number=17, tag_tid=2, entry_tid=19621], [object_number=16, tag_tid=2, entry_tid=19620]]], state=connected] 2024-04-06T09:33:29.855 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:33:29.857 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=19, tag_tid=2, entry_tid=19623' 2024-04-06T09:33:29.857 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=25, tag_tid=3, entry_tid=1' -a 'object_number=25, tag_tid=3, entry_tid=1' = 'object_number=19, tag_tid=2, entry_tid=19623' 2024-04-06T09:33:29.857 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=19, tag_tid=2, entry_tid=19623' '!=' 'object_number=19, tag_tid=2, entry_tid=19507' 2024-04-06T09:33:29.857 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:33:29.857 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=19, tag_tid=2, entry_tid=19623' = 'object_number=19, tag_tid=2, entry_tid=19507' 2024-04-06T09:33:29.857 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=19, tag_tid=2, entry_tid=19623' 2024-04-06T09:33:29.858 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=25, tag_tid=3, entry_tid=1' 2024-04-06T09:33:29.858 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:33:29.858 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:33:29.858 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=3 2024-04-06T09:33:29.859 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=19, tag_tid=2, entry_tid=19623' 2024-04-06T09:33:29.859 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:33:29.859 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:33:29.860 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=2 2024-04-06T09:33:29.861 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=25, tag_tid=3, entry_tid=1' 2024-04-06T09:33:29.861 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:33:29.861 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:33:29.862 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:33:29.863 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=19, tag_tid=2, entry_tid=19623' 2024-04-06T09:33:29.863 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:33:29.863 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:33:29.864 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=19623 2024-04-06T09:33:29.864 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 3 = 2 -a 1 -le 19623 2024-04-06T09:33:29.864 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:33:29.864 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:33:29.864 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:33:30.066 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:33:30.066 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:33:30.066 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:33:30.067 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:33:30.067 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:33:30.067 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:33:30.067 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:33:30.067 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:33:30.067 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:33:30.067 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:33:30.067 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:33:30.067 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:33:30.067 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:33:30.067 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:33:30.067 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:33:30.068 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:33:30.068 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:33:30.070 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:33:30.070 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:33:30.070 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:33:30.071 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:33:30.071 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:33:30.071 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:33:30.071 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:33:30.071 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:33:30.071 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:33:30.071 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:33:30.071 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:33:30.071 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:33:30.071 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:33:30.071 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:33:30.071 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:33:30.072 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:33:30.073 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:33:30.073 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:33:30.073 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:33:30.073 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:33:30.073 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:33:30.073 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:33:30.073 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:33:30.073 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:33:30.073 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:33:30.073 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:33:30.073 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:33:30.074 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:33:30.074 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:33:30.074 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:33:30.074 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:33:30.074 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:33:30.097 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:33:30.098 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:33:30.098 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:33:30.098 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:33:30.232 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:33:30.233 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:33:30.233 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:33:30.233 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:33:30.233 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:33:30.233 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:33:30.234 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:33:30.234 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:33:30.234 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:33:30.234 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:33:30.234 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:33:30.235 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:33:30.235 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:33:30.236 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:33:30.236 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:33:30.236 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:33:30.286 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 4 2024-04-06T09:33:30.286 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 6 2024-04-06T09:33:30.286 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:33:30.286 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=25, tag_tid=3, entry_tid=1], [object_number=24, tag_tid=3, entry_tid=0], [object_number=27, tag_tid=2, entry_tid=27223], [object_number=26, tag_tid=2, entry_tid=27222]]], state=connected] 2024-04-06T09:33:30.286 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=18, tag_tid=2, entry_tid=19742], [object_number=17, tag_tid=2, entry_tid=19741], [object_number=16, tag_tid=2, entry_tid=19740], [object_number=19, tag_tid=2, entry_tid=19739]]], state=connected] 2024-04-06T09:33:30.291 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:33:30.291 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=25, tag_tid=3, entry_tid=1' 2024-04-06T09:33:30.292 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:33:30.292 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:33:30.292 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:33:30.292 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:33:30.293 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:33:30.293 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:33:30.293 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:33:30.293 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:33:30.293 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:33:30.293 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:33:30.293 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:33:30.293 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:33:30.294 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:33:30.294 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:33:30.294 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:33:30.339 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 4 2024-04-06T09:33:30.339 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 6 2024-04-06T09:33:30.339 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:33:30.339 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=25, tag_tid=3, entry_tid=1], [object_number=24, tag_tid=3, entry_tid=0], [object_number=27, tag_tid=2, entry_tid=27223], [object_number=26, tag_tid=2, entry_tid=27222]]], state=connected] 2024-04-06T09:33:30.339 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=18, tag_tid=2, entry_tid=19742], [object_number=17, tag_tid=2, entry_tid=19741], [object_number=16, tag_tid=2, entry_tid=19740], [object_number=19, tag_tid=2, entry_tid=19739]]], state=connected] 2024-04-06T09:33:30.343 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:33:30.345 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=18, tag_tid=2, entry_tid=19742' 2024-04-06T09:33:30.345 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=25, tag_tid=3, entry_tid=1' -a 'object_number=25, tag_tid=3, entry_tid=1' = 'object_number=18, tag_tid=2, entry_tid=19742' 2024-04-06T09:33:30.345 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=18, tag_tid=2, entry_tid=19742' '!=' 'object_number=19, tag_tid=2, entry_tid=19623' 2024-04-06T09:33:30.345 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:33:30.345 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=18, tag_tid=2, entry_tid=19742' = 'object_number=19, tag_tid=2, entry_tid=19623' 2024-04-06T09:33:30.345 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=18, tag_tid=2, entry_tid=19742' 2024-04-06T09:33:30.346 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=25, tag_tid=3, entry_tid=1' 2024-04-06T09:33:30.346 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:33:30.346 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:33:30.347 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=3 2024-04-06T09:33:30.347 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=18, tag_tid=2, entry_tid=19742' 2024-04-06T09:33:30.347 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:33:30.347 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:33:30.348 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=2 2024-04-06T09:33:30.348 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=25, tag_tid=3, entry_tid=1' 2024-04-06T09:33:30.348 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:33:30.348 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:33:30.349 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:33:30.350 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=18, tag_tid=2, entry_tid=19742' 2024-04-06T09:33:30.350 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:33:30.350 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:33:30.351 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=19742 2024-04-06T09:33:30.351 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 3 = 2 -a 1 -le 19742 2024-04-06T09:33:30.351 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:33:30.352 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:33:30.352 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:33:30.553 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:33:30.553 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:33:30.553 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:33:30.553 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:33:30.553 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:33:30.553 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:33:30.553 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:33:30.553 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:33:30.554 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:33:30.554 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:33:30.554 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:33:30.554 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:33:30.554 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:33:30.554 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:33:30.554 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:33:30.554 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:33:30.555 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:33:30.558 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:33:30.558 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:33:30.558 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:33:30.558 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:33:30.559 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:33:30.559 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:33:30.559 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:33:30.559 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:33:30.559 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:33:30.559 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:33:30.559 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:33:30.559 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:33:30.559 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:33:30.559 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:33:30.559 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:33:30.559 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:33:30.559 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:33:30.560 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:33:30.560 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:33:30.560 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:33:30.560 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:33:30.560 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:33:30.560 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:33:30.560 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:33:30.560 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:33:30.560 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:33:30.560 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:33:30.560 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:33:30.560 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:33:30.560 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:33:30.560 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:33:30.560 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:33:30.584 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:33:30.584 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:33:30.584 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:33:30.585 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:33:30.715 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:33:30.717 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:33:30.717 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:33:30.717 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:33:30.717 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:33:30.717 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:33:30.717 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:33:30.718 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:33:30.718 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:33:30.718 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:33:30.718 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:33:30.718 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:33:30.718 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:33:30.719 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:33:30.720 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:33:30.720 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:33:30.769 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 4 2024-04-06T09:33:30.769 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 6 2024-04-06T09:33:30.769 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:33:30.770 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=25, tag_tid=3, entry_tid=1], [object_number=24, tag_tid=3, entry_tid=0], [object_number=27, tag_tid=2, entry_tid=27223], [object_number=26, tag_tid=2, entry_tid=27222]]], state=connected] 2024-04-06T09:33:30.770 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=19, tag_tid=2, entry_tid=19835], [object_number=18, tag_tid=2, entry_tid=19834], [object_number=17, tag_tid=2, entry_tid=19833], [object_number=16, tag_tid=2, entry_tid=19832]]], state=connected] 2024-04-06T09:33:30.773 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:33:30.774 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=25, tag_tid=3, entry_tid=1' 2024-04-06T09:33:30.775 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:33:30.775 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:33:30.776 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:33:30.776 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:33:30.776 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:33:30.776 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:33:30.776 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:33:30.776 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:33:30.776 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:33:30.776 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:33:30.776 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:33:30.776 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:33:30.777 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:33:30.777 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:33:30.777 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:33:30.822 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 4 2024-04-06T09:33:30.822 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 6 2024-04-06T09:33:30.822 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:33:30.822 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=25, tag_tid=3, entry_tid=1], [object_number=24, tag_tid=3, entry_tid=0], [object_number=27, tag_tid=2, entry_tid=27223], [object_number=26, tag_tid=2, entry_tid=27222]]], state=connected] 2024-04-06T09:33:30.822 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=19, tag_tid=2, entry_tid=19835], [object_number=18, tag_tid=2, entry_tid=19834], [object_number=17, tag_tid=2, entry_tid=19833], [object_number=16, tag_tid=2, entry_tid=19832]]], state=connected] 2024-04-06T09:33:30.825 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:33:30.827 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=19, tag_tid=2, entry_tid=19835' 2024-04-06T09:33:30.828 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=25, tag_tid=3, entry_tid=1' -a 'object_number=25, tag_tid=3, entry_tid=1' = 'object_number=19, tag_tid=2, entry_tid=19835' 2024-04-06T09:33:30.828 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=19, tag_tid=2, entry_tid=19835' '!=' 'object_number=18, tag_tid=2, entry_tid=19742' 2024-04-06T09:33:30.828 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:33:30.828 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=19, tag_tid=2, entry_tid=19835' = 'object_number=18, tag_tid=2, entry_tid=19742' 2024-04-06T09:33:30.828 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=19, tag_tid=2, entry_tid=19835' 2024-04-06T09:33:30.828 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=25, tag_tid=3, entry_tid=1' 2024-04-06T09:33:30.828 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:33:30.828 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:33:30.829 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=3 2024-04-06T09:33:30.829 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=19, tag_tid=2, entry_tid=19835' 2024-04-06T09:33:30.829 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:33:30.830 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:33:30.830 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=2 2024-04-06T09:33:30.831 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=25, tag_tid=3, entry_tid=1' 2024-04-06T09:33:30.831 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:33:30.831 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:33:30.832 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:33:30.833 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=19, tag_tid=2, entry_tid=19835' 2024-04-06T09:33:30.833 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:33:30.833 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:33:30.834 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=19835 2024-04-06T09:33:30.834 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 3 = 2 -a 1 -le 19835 2024-04-06T09:33:30.834 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:33:30.834 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:33:30.834 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:33:31.036 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:33:31.036 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:33:31.036 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:33:31.036 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:33:31.036 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:33:31.036 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:33:31.036 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:33:31.036 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:33:31.036 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:33:31.036 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:33:31.036 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:33:31.037 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:33:31.037 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:33:31.037 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:33:31.037 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:33:31.037 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:33:31.037 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:33:31.040 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:33:31.040 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:33:31.040 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:33:31.040 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:33:31.040 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:33:31.040 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:33:31.040 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:33:31.040 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:33:31.040 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:33:31.041 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:33:31.041 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:33:31.041 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:33:31.041 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:33:31.041 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:33:31.041 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:33:31.041 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:33:31.041 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:33:31.041 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:33:31.041 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:33:31.041 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:33:31.041 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:33:31.041 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:33:31.041 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:33:31.041 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:33:31.042 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:33:31.042 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:33:31.042 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:33:31.042 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:33:31.042 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:33:31.042 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:33:31.042 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:33:31.042 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:33:31.067 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:33:31.068 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:33:31.068 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:33:31.068 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:33:31.193 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:33:31.194 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:33:31.195 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:33:31.195 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:33:31.195 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:33:31.195 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:33:31.195 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:33:31.195 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:33:31.195 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:33:31.195 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:33:31.196 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:33:31.196 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:33:31.196 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:33:31.198 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:33:31.199 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:33:31.199 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:33:31.248 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 4 2024-04-06T09:33:31.248 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 6 2024-04-06T09:33:31.248 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:33:31.248 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=25, tag_tid=3, entry_tid=1], [object_number=24, tag_tid=3, entry_tid=0], [object_number=27, tag_tid=2, entry_tid=27223], [object_number=26, tag_tid=2, entry_tid=27222]]], state=connected] 2024-04-06T09:33:31.248 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=16, tag_tid=2, entry_tid=19924], [object_number=19, tag_tid=2, entry_tid=19923], [object_number=18, tag_tid=2, entry_tid=19922], [object_number=17, tag_tid=2, entry_tid=19921]]], state=connected] 2024-04-06T09:33:31.252 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:33:31.254 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=25, tag_tid=3, entry_tid=1' 2024-04-06T09:33:31.254 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:33:31.254 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:33:31.254 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:33:31.254 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:33:31.254 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:33:31.254 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:33:31.254 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:33:31.255 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:33:31.255 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:33:31.255 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:33:31.255 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:33:31.255 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:33:31.255 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:33:31.256 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:33:31.256 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:33:31.300 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 4 2024-04-06T09:33:31.351 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 6 2024-04-06T09:33:31.352 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:33:31.352 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=25, tag_tid=3, entry_tid=1], [object_number=24, tag_tid=3, entry_tid=0], [object_number=27, tag_tid=2, entry_tid=27223], [object_number=26, tag_tid=2, entry_tid=27222]]], state=connected] 2024-04-06T09:33:31.352 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=16, tag_tid=2, entry_tid=19924], [object_number=19, tag_tid=2, entry_tid=19923], [object_number=18, tag_tid=2, entry_tid=19922], [object_number=17, tag_tid=2, entry_tid=19921]]], state=connected] 2024-04-06T09:33:31.355 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:33:31.355 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=16, tag_tid=2, entry_tid=19924' 2024-04-06T09:33:31.355 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=25, tag_tid=3, entry_tid=1' -a 'object_number=25, tag_tid=3, entry_tid=1' = 'object_number=16, tag_tid=2, entry_tid=19924' 2024-04-06T09:33:31.355 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=16, tag_tid=2, entry_tid=19924' '!=' 'object_number=19, tag_tid=2, entry_tid=19835' 2024-04-06T09:33:31.355 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:33:31.355 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=16, tag_tid=2, entry_tid=19924' = 'object_number=19, tag_tid=2, entry_tid=19835' 2024-04-06T09:33:31.355 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=16, tag_tid=2, entry_tid=19924' 2024-04-06T09:33:31.355 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=25, tag_tid=3, entry_tid=1' 2024-04-06T09:33:31.355 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:33:31.355 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:33:31.355 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=3 2024-04-06T09:33:31.355 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=16, tag_tid=2, entry_tid=19924' 2024-04-06T09:33:31.355 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:33:31.355 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:33:31.356 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=2 2024-04-06T09:33:31.356 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=25, tag_tid=3, entry_tid=1' 2024-04-06T09:33:31.356 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:33:31.356 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:33:31.356 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:33:31.356 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=16, tag_tid=2, entry_tid=19924' 2024-04-06T09:33:31.356 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:33:31.356 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:33:31.356 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=19924 2024-04-06T09:33:31.356 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 3 = 2 -a 1 -le 19924 2024-04-06T09:33:31.356 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:33:31.356 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:33:31.356 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:33:31.514 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:33:31.514 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:33:31.514 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:33:31.514 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:33:31.514 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:33:31.515 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:33:31.515 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:33:31.515 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:33:31.515 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:33:31.515 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:33:31.515 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:33:31.515 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:33:31.515 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:33:31.515 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:33:31.515 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:33:31.515 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:33:31.516 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:33:31.518 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:33:31.518 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:33:31.519 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:33:31.519 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:33:31.519 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:33:31.519 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:33:31.519 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:33:31.519 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:33:31.519 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:33:31.519 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:33:31.519 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:33:31.519 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:33:31.519 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:33:31.519 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:33:31.519 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:33:31.519 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:33:31.521 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:33:31.521 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:33:31.521 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:33:31.521 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:33:31.521 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:33:31.521 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:33:31.521 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:33:31.521 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:33:31.521 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:33:31.521 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:33:31.521 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:33:31.521 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:33:31.522 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:33:31.522 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:33:31.522 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:33:31.522 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:33:31.547 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:33:31.547 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:33:31.547 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:33:31.547 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:33:31.691 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:33:31.691 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:33:31.691 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:33:31.691 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:33:31.692 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:33:31.692 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:33:31.692 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:33:31.692 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:33:31.692 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:33:31.692 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:33:31.693 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:33:31.693 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:33:31.693 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:33:31.695 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:33:31.696 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:33:31.696 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:33:31.951 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 4 2024-04-06T09:33:31.951 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 6 2024-04-06T09:33:31.951 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:33:31.951 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=25, tag_tid=3, entry_tid=1], [object_number=24, tag_tid=3, entry_tid=0], [object_number=27, tag_tid=2, entry_tid=27223], [object_number=26, tag_tid=2, entry_tid=27222]]], state=connected] 2024-04-06T09:33:31.951 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=16, tag_tid=2, entry_tid=20020], [object_number=19, tag_tid=2, entry_tid=20019], [object_number=18, tag_tid=2, entry_tid=20018], [object_number=17, tag_tid=2, entry_tid=20017]]], state=connected] 2024-04-06T09:33:31.956 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:33:31.957 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=25, tag_tid=3, entry_tid=1' 2024-04-06T09:33:31.958 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:33:31.958 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:33:31.959 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:33:31.959 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:33:31.959 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:33:31.959 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:33:31.959 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:33:31.959 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:33:31.959 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:33:31.959 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:33:31.959 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:33:31.959 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:33:31.960 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:33:31.960 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:33:31.960 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:33:32.010 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 4 2024-04-06T09:33:32.010 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 6 2024-04-06T09:33:32.010 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:33:32.010 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=25, tag_tid=3, entry_tid=1], [object_number=24, tag_tid=3, entry_tid=0], [object_number=27, tag_tid=2, entry_tid=27223], [object_number=26, tag_tid=2, entry_tid=27222]]], state=connected] 2024-04-06T09:33:32.010 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=16, tag_tid=2, entry_tid=20020], [object_number=19, tag_tid=2, entry_tid=20019], [object_number=18, tag_tid=2, entry_tid=20018], [object_number=17, tag_tid=2, entry_tid=20017]]], state=connected] 2024-04-06T09:33:32.014 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:33:32.016 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=16, tag_tid=2, entry_tid=20020' 2024-04-06T09:33:32.016 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=25, tag_tid=3, entry_tid=1' -a 'object_number=25, tag_tid=3, entry_tid=1' = 'object_number=16, tag_tid=2, entry_tid=20020' 2024-04-06T09:33:32.016 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=16, tag_tid=2, entry_tid=20020' '!=' 'object_number=16, tag_tid=2, entry_tid=19924' 2024-04-06T09:33:32.017 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:33:32.017 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=16, tag_tid=2, entry_tid=20020' = 'object_number=16, tag_tid=2, entry_tid=19924' 2024-04-06T09:33:32.017 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=16, tag_tid=2, entry_tid=20020' 2024-04-06T09:33:32.017 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=25, tag_tid=3, entry_tid=1' 2024-04-06T09:33:32.017 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:33:32.017 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:33:32.017 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=3 2024-04-06T09:33:32.018 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=16, tag_tid=2, entry_tid=20020' 2024-04-06T09:33:32.018 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:33:32.018 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:33:32.019 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=2 2024-04-06T09:33:32.020 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=25, tag_tid=3, entry_tid=1' 2024-04-06T09:33:32.020 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:33:32.020 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:33:32.021 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:33:32.022 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=16, tag_tid=2, entry_tid=20020' 2024-04-06T09:33:32.022 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:33:32.022 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:33:32.023 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=20020 2024-04-06T09:33:32.023 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 3 = 2 -a 1 -le 20020 2024-04-06T09:33:32.023 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:33:32.023 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:33:32.023 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:33:32.225 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:33:32.225 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:33:32.225 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:33:32.225 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:33:32.226 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:33:32.226 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:33:32.226 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:33:32.226 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:33:32.226 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:33:32.226 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:33:32.226 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:33:32.226 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:33:32.226 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:33:32.226 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:33:32.227 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:33:32.227 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:33:32.227 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:33:32.230 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:33:32.231 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:33:32.231 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:33:32.231 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:33:32.231 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:33:32.231 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:33:32.231 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:33:32.231 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:33:32.231 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:33:32.231 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:33:32.232 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:33:32.232 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:33:32.232 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:33:32.232 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:33:32.232 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:33:32.232 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:33:32.232 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:33:32.232 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:33:32.232 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:33:32.232 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:33:32.232 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:33:32.233 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:33:32.233 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:33:32.233 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:33:32.233 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:33:32.233 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:33:32.233 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:33:32.233 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:33:32.233 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:33:32.233 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:33:32.233 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:33:32.233 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:33:32.255 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:33:32.256 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:33:32.256 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:33:32.256 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:33:32.383 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:33:32.385 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:33:32.385 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:33:32.385 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:33:32.385 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:33:32.385 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:33:32.385 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:33:32.385 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:33:32.385 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:33:32.385 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:33:32.386 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:33:32.386 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:33:32.386 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:33:32.387 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:33:32.387 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:33:32.387 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:33:32.439 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 4 2024-04-06T09:33:32.439 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 6 2024-04-06T09:33:32.439 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:33:32.440 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=25, tag_tid=3, entry_tid=1], [object_number=24, tag_tid=3, entry_tid=0], [object_number=27, tag_tid=2, entry_tid=27223], [object_number=26, tag_tid=2, entry_tid=27222]]], state=connected] 2024-04-06T09:33:32.440 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=17, tag_tid=2, entry_tid=20161], [object_number=16, tag_tid=2, entry_tid=20160], [object_number=19, tag_tid=2, entry_tid=20159], [object_number=18, tag_tid=2, entry_tid=20158]]], state=connected] 2024-04-06T09:33:32.443 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:33:32.444 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=25, tag_tid=3, entry_tid=1' 2024-04-06T09:33:32.445 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:33:32.446 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:33:32.446 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:33:32.446 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:33:32.446 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:33:32.446 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:33:32.446 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:33:32.446 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:33:32.446 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:33:32.446 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:33:32.446 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:33:32.446 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:33:32.447 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:33:32.447 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:33:32.447 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:33:32.492 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 4 2024-04-06T09:33:32.492 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 6 2024-04-06T09:33:32.492 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:33:32.492 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=25, tag_tid=3, entry_tid=1], [object_number=24, tag_tid=3, entry_tid=0], [object_number=27, tag_tid=2, entry_tid=27223], [object_number=26, tag_tid=2, entry_tid=27222]]], state=connected] 2024-04-06T09:33:32.492 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=17, tag_tid=2, entry_tid=20161], [object_number=16, tag_tid=2, entry_tid=20160], [object_number=19, tag_tid=2, entry_tid=20159], [object_number=18, tag_tid=2, entry_tid=20158]]], state=connected] 2024-04-06T09:33:32.495 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:33:32.498 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=17, tag_tid=2, entry_tid=20161' 2024-04-06T09:33:32.498 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=25, tag_tid=3, entry_tid=1' -a 'object_number=25, tag_tid=3, entry_tid=1' = 'object_number=17, tag_tid=2, entry_tid=20161' 2024-04-06T09:33:32.498 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=17, tag_tid=2, entry_tid=20161' '!=' 'object_number=16, tag_tid=2, entry_tid=20020' 2024-04-06T09:33:32.498 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:33:32.498 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=17, tag_tid=2, entry_tid=20161' = 'object_number=16, tag_tid=2, entry_tid=20020' 2024-04-06T09:33:32.498 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=17, tag_tid=2, entry_tid=20161' 2024-04-06T09:33:32.498 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=25, tag_tid=3, entry_tid=1' 2024-04-06T09:33:32.498 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:33:32.498 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:33:32.499 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=3 2024-04-06T09:33:32.499 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=17, tag_tid=2, entry_tid=20161' 2024-04-06T09:33:32.500 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:33:32.500 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:33:32.500 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=2 2024-04-06T09:33:32.501 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=25, tag_tid=3, entry_tid=1' 2024-04-06T09:33:32.501 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:33:32.501 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:33:32.502 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:33:32.503 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=17, tag_tid=2, entry_tid=20161' 2024-04-06T09:33:32.503 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:33:32.503 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:33:32.504 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=20161 2024-04-06T09:33:32.504 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 3 = 2 -a 1 -le 20161 2024-04-06T09:33:32.504 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:33:32.504 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:33:32.504 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:33:32.706 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:33:32.706 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:33:32.706 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:33:32.706 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:33:32.707 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:33:32.707 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:33:32.707 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:33:32.707 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:33:32.707 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:33:32.707 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:33:32.707 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:33:32.707 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:33:32.707 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:33:32.707 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:33:32.707 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:33:32.708 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:33:32.708 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:33:32.712 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:33:32.712 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:33:32.712 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:33:32.712 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:33:32.712 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:33:32.712 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:33:32.712 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:33:32.712 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:33:32.712 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:33:32.713 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:33:32.713 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:33:32.713 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:33:32.713 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:33:32.713 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:33:32.713 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:33:32.713 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:33:32.713 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:33:32.713 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:33:32.713 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:33:32.713 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:33:32.713 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:33:32.713 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:33:32.713 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:33:32.713 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:33:32.714 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:33:32.714 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:33:32.714 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:33:32.714 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:33:32.714 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:33:32.714 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:33:32.714 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:33:32.714 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:33:32.737 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:33:32.737 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:33:32.737 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:33:32.737 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:33:32.868 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:33:32.870 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:33:32.870 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:33:32.870 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:33:32.870 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:33:32.870 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:33:32.870 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:33:32.870 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:33:32.870 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:33:32.870 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:33:32.871 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:33:32.871 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:33:32.871 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:33:32.872 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:33:32.873 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:33:32.873 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:33:32.919 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 4 2024-04-06T09:33:32.919 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 6 2024-04-06T09:33:32.919 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:33:32.919 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=25, tag_tid=3, entry_tid=1], [object_number=24, tag_tid=3, entry_tid=0], [object_number=27, tag_tid=2, entry_tid=27223], [object_number=26, tag_tid=2, entry_tid=27222]]], state=connected] 2024-04-06T09:33:32.919 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=17, tag_tid=2, entry_tid=20261], [object_number=16, tag_tid=2, entry_tid=20260], [object_number=19, tag_tid=2, entry_tid=20259], [object_number=18, tag_tid=2, entry_tid=20258]]], state=connected] 2024-04-06T09:33:32.923 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:33:32.924 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=25, tag_tid=3, entry_tid=1' 2024-04-06T09:33:32.924 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:33:32.925 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:33:32.925 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:33:32.925 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:33:32.925 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:33:32.925 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:33:32.925 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:33:32.925 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:33:32.925 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:33:32.925 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:33:32.925 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:33:32.926 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:33:32.926 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:33:32.926 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:33:32.926 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:33:32.970 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 4 2024-04-06T09:33:32.970 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 6 2024-04-06T09:33:32.971 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:33:32.971 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=25, tag_tid=3, entry_tid=1], [object_number=24, tag_tid=3, entry_tid=0], [object_number=27, tag_tid=2, entry_tid=27223], [object_number=26, tag_tid=2, entry_tid=27222]]], state=connected] 2024-04-06T09:33:32.971 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=17, tag_tid=2, entry_tid=20261], [object_number=16, tag_tid=2, entry_tid=20260], [object_number=19, tag_tid=2, entry_tid=20259], [object_number=18, tag_tid=2, entry_tid=20258]]], state=connected] 2024-04-06T09:33:32.974 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:33:32.976 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=17, tag_tid=2, entry_tid=20261' 2024-04-06T09:33:32.976 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=25, tag_tid=3, entry_tid=1' -a 'object_number=25, tag_tid=3, entry_tid=1' = 'object_number=17, tag_tid=2, entry_tid=20261' 2024-04-06T09:33:32.976 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=17, tag_tid=2, entry_tid=20261' '!=' 'object_number=17, tag_tid=2, entry_tid=20161' 2024-04-06T09:33:32.976 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:33:32.976 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=17, tag_tid=2, entry_tid=20261' = 'object_number=17, tag_tid=2, entry_tid=20161' 2024-04-06T09:33:32.976 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=17, tag_tid=2, entry_tid=20261' 2024-04-06T09:33:32.977 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=25, tag_tid=3, entry_tid=1' 2024-04-06T09:33:32.977 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:33:32.977 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:33:32.977 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=3 2024-04-06T09:33:32.978 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=17, tag_tid=2, entry_tid=20261' 2024-04-06T09:33:32.978 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:33:32.978 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:33:32.979 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=2 2024-04-06T09:33:32.980 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=25, tag_tid=3, entry_tid=1' 2024-04-06T09:33:32.980 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:33:32.980 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:33:32.981 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:33:32.981 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=17, tag_tid=2, entry_tid=20261' 2024-04-06T09:33:32.981 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:33:32.981 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:33:32.983 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=20261 2024-04-06T09:33:32.983 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 3 = 2 -a 1 -le 20261 2024-04-06T09:33:32.983 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:33:32.983 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:33:32.983 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:33:33.184 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:33:33.184 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:33:33.184 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:33:33.184 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:33:33.184 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:33:33.184 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:33:33.184 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:33:33.184 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:33:33.185 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:33:33.185 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:33:33.185 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:33:33.185 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:33:33.185 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:33:33.185 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:33:33.185 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:33:33.186 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:33:33.186 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:33:33.187 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:33:33.187 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:33:33.187 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:33:33.188 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:33:33.188 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:33:33.188 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:33:33.188 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:33:33.188 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:33:33.188 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:33:33.188 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:33:33.188 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:33:33.188 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:33:33.188 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:33:33.188 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:33:33.188 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:33:33.188 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:33:33.190 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:33:33.190 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:33:33.190 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:33:33.190 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:33:33.190 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:33:33.190 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:33:33.190 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:33:33.191 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:33:33.191 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:33:33.191 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:33:33.191 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:33:33.191 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:33:33.191 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:33:33.191 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:33:33.191 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:33:33.191 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:33:33.214 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:33:33.214 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:33:33.214 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:33:33.214 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:33:33.345 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:33:33.347 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:33:33.347 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:33:33.347 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:33:33.347 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:33:33.347 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:33:33.347 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:33:33.347 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:33:33.347 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:33:33.347 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:33:33.348 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:33:33.348 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:33:33.348 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:33:33.350 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:33:33.351 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:33:33.351 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:33:33.398 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 4 2024-04-06T09:33:33.398 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 6 2024-04-06T09:33:33.398 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:33:33.398 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=25, tag_tid=3, entry_tid=1], [object_number=24, tag_tid=3, entry_tid=0], [object_number=27, tag_tid=2, entry_tid=27223], [object_number=26, tag_tid=2, entry_tid=27222]]], state=connected] 2024-04-06T09:33:33.398 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=16, tag_tid=2, entry_tid=20360], [object_number=19, tag_tid=2, entry_tid=20359], [object_number=18, tag_tid=2, entry_tid=20358], [object_number=17, tag_tid=2, entry_tid=20357]]], state=connected] 2024-04-06T09:33:33.401 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:33:33.403 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=25, tag_tid=3, entry_tid=1' 2024-04-06T09:33:33.404 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:33:33.404 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:33:33.404 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:33:33.404 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:33:33.404 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:33:33.404 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:33:33.404 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:33:33.404 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:33:33.404 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:33:33.404 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:33:33.404 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:33:33.405 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:33:33.405 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:33:33.405 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:33:33.405 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:33:33.450 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 4 2024-04-06T09:33:33.451 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 6 2024-04-06T09:33:33.451 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:33:33.451 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=25, tag_tid=3, entry_tid=1], [object_number=24, tag_tid=3, entry_tid=0], [object_number=27, tag_tid=2, entry_tid=27223], [object_number=26, tag_tid=2, entry_tid=27222]]], state=connected] 2024-04-06T09:33:33.451 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=16, tag_tid=2, entry_tid=20360], [object_number=19, tag_tid=2, entry_tid=20359], [object_number=18, tag_tid=2, entry_tid=20358], [object_number=17, tag_tid=2, entry_tid=20357]]], state=connected] 2024-04-06T09:33:33.454 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:33:33.456 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=16, tag_tid=2, entry_tid=20360' 2024-04-06T09:33:33.456 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=25, tag_tid=3, entry_tid=1' -a 'object_number=25, tag_tid=3, entry_tid=1' = 'object_number=16, tag_tid=2, entry_tid=20360' 2024-04-06T09:33:33.456 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=16, tag_tid=2, entry_tid=20360' '!=' 'object_number=17, tag_tid=2, entry_tid=20261' 2024-04-06T09:33:33.456 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:33:33.456 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=16, tag_tid=2, entry_tid=20360' = 'object_number=17, tag_tid=2, entry_tid=20261' 2024-04-06T09:33:33.456 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=16, tag_tid=2, entry_tid=20360' 2024-04-06T09:33:33.456 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=25, tag_tid=3, entry_tid=1' 2024-04-06T09:33:33.457 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:33:33.457 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:33:33.457 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=3 2024-04-06T09:33:33.458 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=16, tag_tid=2, entry_tid=20360' 2024-04-06T09:33:33.458 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:33:33.458 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:33:33.459 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=2 2024-04-06T09:33:33.460 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=25, tag_tid=3, entry_tid=1' 2024-04-06T09:33:33.460 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:33:33.460 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:33:33.461 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:33:33.461 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=16, tag_tid=2, entry_tid=20360' 2024-04-06T09:33:33.461 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:33:33.461 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:33:33.463 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=20360 2024-04-06T09:33:33.463 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 3 = 2 -a 1 -le 20360 2024-04-06T09:33:33.463 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:33:33.463 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:33:33.463 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:33:33.664 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:33:33.665 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:33:33.665 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:33:33.665 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:33:33.665 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:33:33.665 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:33:33.665 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:33:33.665 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:33:33.665 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:33:33.665 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:33:33.665 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:33:33.665 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:33:33.665 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:33:33.665 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:33:33.665 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:33:33.666 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:33:33.666 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:33:33.669 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:33:33.669 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:33:33.669 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:33:33.670 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:33:33.670 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:33:33.670 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:33:33.670 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:33:33.670 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:33:33.670 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:33:33.670 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:33:33.670 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:33:33.670 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:33:33.670 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:33:33.670 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:33:33.670 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:33:33.671 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:33:33.671 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:33:33.671 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:33:33.671 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:33:33.671 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:33:33.671 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:33:33.671 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:33:33.671 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:33:33.671 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:33:33.671 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:33:33.671 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:33:33.671 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:33:33.671 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:33:33.671 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:33:33.672 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:33:33.672 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:33:33.672 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:33:33.695 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:33:33.695 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:33:33.695 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:33:33.695 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:33:33.821 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:33:33.822 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:33:33.822 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:33:33.822 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:33:33.822 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:33:33.822 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:33:33.822 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:33:33.822 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:33:33.822 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:33:33.822 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:33:33.823 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:33:33.823 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:33:33.823 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:33:33.824 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:33:33.825 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:33:33.825 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:33:33.869 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 4 2024-04-06T09:33:33.870 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 6 2024-04-06T09:33:33.870 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:33:33.870 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=25, tag_tid=3, entry_tid=1], [object_number=24, tag_tid=3, entry_tid=0], [object_number=27, tag_tid=2, entry_tid=27223], [object_number=26, tag_tid=2, entry_tid=27222]]], state=connected] 2024-04-06T09:33:33.870 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=16, tag_tid=2, entry_tid=20476], [object_number=19, tag_tid=2, entry_tid=20475], [object_number=18, tag_tid=2, entry_tid=20474], [object_number=17, tag_tid=2, entry_tid=20473]]], state=connected] 2024-04-06T09:33:33.873 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:33:33.874 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=25, tag_tid=3, entry_tid=1' 2024-04-06T09:33:33.875 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:33:33.875 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:33:33.876 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:33:33.876 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:33:33.876 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:33:33.876 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:33:33.876 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:33:33.876 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:33:33.876 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:33:33.876 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:33:33.876 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:33:33.876 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:33:33.877 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:33:33.877 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:33:33.877 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:33:33.923 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 4 2024-04-06T09:33:33.923 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 6 2024-04-06T09:33:33.923 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:33:33.923 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=25, tag_tid=3, entry_tid=1], [object_number=24, tag_tid=3, entry_tid=0], [object_number=27, tag_tid=2, entry_tid=27223], [object_number=26, tag_tid=2, entry_tid=27222]]], state=connected] 2024-04-06T09:33:33.923 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=16, tag_tid=2, entry_tid=20476], [object_number=19, tag_tid=2, entry_tid=20475], [object_number=18, tag_tid=2, entry_tid=20474], [object_number=17, tag_tid=2, entry_tid=20473]]], state=connected] 2024-04-06T09:33:33.926 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:33:33.928 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=16, tag_tid=2, entry_tid=20476' 2024-04-06T09:33:33.928 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=25, tag_tid=3, entry_tid=1' -a 'object_number=25, tag_tid=3, entry_tid=1' = 'object_number=16, tag_tid=2, entry_tid=20476' 2024-04-06T09:33:33.928 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=16, tag_tid=2, entry_tid=20476' '!=' 'object_number=16, tag_tid=2, entry_tid=20360' 2024-04-06T09:33:33.928 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:33:33.928 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=16, tag_tid=2, entry_tid=20476' = 'object_number=16, tag_tid=2, entry_tid=20360' 2024-04-06T09:33:33.928 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=16, tag_tid=2, entry_tid=20476' 2024-04-06T09:33:33.929 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=25, tag_tid=3, entry_tid=1' 2024-04-06T09:33:33.929 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:33:33.929 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:33:33.930 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=3 2024-04-06T09:33:33.930 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=16, tag_tid=2, entry_tid=20476' 2024-04-06T09:33:33.930 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:33:33.930 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:33:33.931 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=2 2024-04-06T09:33:33.932 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=25, tag_tid=3, entry_tid=1' 2024-04-06T09:33:33.932 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:33:33.932 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:33:33.933 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:33:33.934 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=16, tag_tid=2, entry_tid=20476' 2024-04-06T09:33:33.934 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:33:33.934 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:33:33.935 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=20476 2024-04-06T09:33:33.935 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 3 = 2 -a 1 -le 20476 2024-04-06T09:33:33.935 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:33:33.935 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:33:33.935 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:33:34.137 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:33:34.137 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:33:34.137 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:33:34.137 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:33:34.137 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:33:34.137 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:33:34.137 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:33:34.137 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:33:34.138 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:33:34.138 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:33:34.138 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:33:34.138 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:33:34.138 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:33:34.138 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:33:34.138 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:33:34.138 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:33:34.139 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:33:34.141 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:33:34.142 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:33:34.142 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:33:34.142 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:33:34.142 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:33:34.142 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:33:34.142 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:33:34.142 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:33:34.142 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:33:34.142 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:33:34.142 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:33:34.142 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:33:34.143 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:33:34.143 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:33:34.143 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:33:34.143 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:33:34.144 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:33:34.144 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:33:34.144 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:33:34.144 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:33:34.144 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:33:34.144 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:33:34.145 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:33:34.145 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:33:34.145 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:33:34.145 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:33:34.145 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:33:34.145 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:33:34.145 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:33:34.145 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:33:34.145 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:33:34.145 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:33:34.169 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:33:34.169 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:33:34.169 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:33:34.169 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:33:34.312 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:33:34.313 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:33:34.313 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:33:34.313 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:33:34.313 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:33:34.313 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:33:34.313 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:33:34.314 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:33:34.314 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:33:34.314 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:33:34.314 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:33:34.314 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:33:34.315 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:33:34.316 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:33:34.317 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:33:34.317 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:33:34.363 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 4 2024-04-06T09:33:34.363 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 6 2024-04-06T09:33:34.363 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:33:34.363 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=25, tag_tid=3, entry_tid=1], [object_number=24, tag_tid=3, entry_tid=0], [object_number=27, tag_tid=2, entry_tid=27223], [object_number=26, tag_tid=2, entry_tid=27222]]], state=connected] 2024-04-06T09:33:34.363 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=16, tag_tid=2, entry_tid=20584], [object_number=19, tag_tid=2, entry_tid=20583], [object_number=18, tag_tid=2, entry_tid=20582], [object_number=17, tag_tid=2, entry_tid=20581]]], state=connected] 2024-04-06T09:33:34.366 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:33:34.368 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=25, tag_tid=3, entry_tid=1' 2024-04-06T09:33:34.369 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:33:34.369 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:33:34.369 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:33:34.369 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:33:34.369 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:33:34.369 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:33:34.369 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:33:34.370 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:33:34.370 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:33:34.370 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:33:34.370 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:33:34.370 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:33:34.370 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:33:34.371 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:33:34.371 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:33:34.417 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 4 2024-04-06T09:33:34.417 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 6 2024-04-06T09:33:34.417 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:33:34.417 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=25, tag_tid=3, entry_tid=1], [object_number=24, tag_tid=3, entry_tid=0], [object_number=27, tag_tid=2, entry_tid=27223], [object_number=26, tag_tid=2, entry_tid=27222]]], state=connected] 2024-04-06T09:33:34.417 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=16, tag_tid=2, entry_tid=20584], [object_number=19, tag_tid=2, entry_tid=20583], [object_number=18, tag_tid=2, entry_tid=20582], [object_number=17, tag_tid=2, entry_tid=20581]]], state=connected] 2024-04-06T09:33:34.420 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:33:34.423 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=16, tag_tid=2, entry_tid=20584' 2024-04-06T09:33:34.423 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=25, tag_tid=3, entry_tid=1' -a 'object_number=25, tag_tid=3, entry_tid=1' = 'object_number=16, tag_tid=2, entry_tid=20584' 2024-04-06T09:33:34.423 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=16, tag_tid=2, entry_tid=20584' '!=' 'object_number=16, tag_tid=2, entry_tid=20476' 2024-04-06T09:33:34.423 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:33:34.423 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=16, tag_tid=2, entry_tid=20584' = 'object_number=16, tag_tid=2, entry_tid=20476' 2024-04-06T09:33:34.423 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=16, tag_tid=2, entry_tid=20584' 2024-04-06T09:33:34.423 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=25, tag_tid=3, entry_tid=1' 2024-04-06T09:33:34.423 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:33:34.423 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:33:34.424 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=3 2024-04-06T09:33:34.424 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=16, tag_tid=2, entry_tid=20584' 2024-04-06T09:33:34.424 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:33:34.425 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:33:34.426 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=2 2024-04-06T09:33:34.426 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=25, tag_tid=3, entry_tid=1' 2024-04-06T09:33:34.426 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:33:34.427 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:33:34.427 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:33:34.428 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=16, tag_tid=2, entry_tid=20584' 2024-04-06T09:33:34.428 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:33:34.428 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:33:34.429 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=20584 2024-04-06T09:33:34.429 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 3 = 2 -a 1 -le 20584 2024-04-06T09:33:34.429 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:33:34.430 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:33:34.430 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:33:34.631 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:33:34.631 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:33:34.631 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:33:34.632 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:33:34.632 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:33:34.632 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:33:34.632 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:33:34.632 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:33:34.632 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:33:34.632 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:33:34.632 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:33:34.632 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:33:34.632 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:33:34.632 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:33:34.632 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:33:34.633 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:33:34.633 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:33:34.635 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:33:34.635 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:33:34.635 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:33:34.635 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:33:34.635 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:33:34.635 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:33:34.635 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:33:34.635 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:33:34.635 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:33:34.635 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:33:34.635 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:33:34.635 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:33:34.635 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:33:34.635 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:33:34.635 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:33:34.636 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:33:34.636 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:33:34.637 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:33:34.637 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:33:34.637 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:33:34.637 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:33:34.637 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:33:34.637 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:33:34.637 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:33:34.637 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:33:34.637 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:33:34.637 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:33:34.637 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:33:34.637 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:33:34.637 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:33:34.637 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:33:34.637 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:33:34.663 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:33:34.663 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:33:34.663 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:33:34.663 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:33:34.790 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:33:34.790 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:33:34.790 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:33:34.791 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:33:34.791 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:33:34.791 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:33:34.791 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:33:34.791 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:33:34.791 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:33:34.791 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:33:34.791 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:33:34.792 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:33:34.792 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:33:34.793 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:33:34.793 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:33:34.793 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:33:34.839 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 4 2024-04-06T09:33:34.839 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 6 2024-04-06T09:33:34.839 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:33:34.839 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=25, tag_tid=3, entry_tid=1], [object_number=24, tag_tid=3, entry_tid=0], [object_number=27, tag_tid=2, entry_tid=27223], [object_number=26, tag_tid=2, entry_tid=27222]]], state=connected] 2024-04-06T09:33:34.839 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=16, tag_tid=2, entry_tid=20716], [object_number=19, tag_tid=2, entry_tid=20715], [object_number=18, tag_tid=2, entry_tid=20714], [object_number=17, tag_tid=2, entry_tid=20713]]], state=connected] 2024-04-06T09:33:34.842 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:33:34.844 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=25, tag_tid=3, entry_tid=1' 2024-04-06T09:33:34.845 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:33:34.845 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:33:34.845 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:33:34.845 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:33:34.845 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:33:34.845 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:33:34.845 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:33:34.845 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:33:34.845 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:33:34.845 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:33:34.845 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:33:34.846 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:33:34.846 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:33:34.846 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:33:34.846 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:33:34.890 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 4 2024-04-06T09:33:34.890 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 6 2024-04-06T09:33:34.890 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:33:34.890 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=25, tag_tid=3, entry_tid=1], [object_number=24, tag_tid=3, entry_tid=0], [object_number=27, tag_tid=2, entry_tid=27223], [object_number=26, tag_tid=2, entry_tid=27222]]], state=connected] 2024-04-06T09:33:34.890 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=16, tag_tid=2, entry_tid=20716], [object_number=19, tag_tid=2, entry_tid=20715], [object_number=18, tag_tid=2, entry_tid=20714], [object_number=17, tag_tid=2, entry_tid=20713]]], state=connected] 2024-04-06T09:33:34.893 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:33:34.895 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=16, tag_tid=2, entry_tid=20716' 2024-04-06T09:33:34.895 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=25, tag_tid=3, entry_tid=1' -a 'object_number=25, tag_tid=3, entry_tid=1' = 'object_number=16, tag_tid=2, entry_tid=20716' 2024-04-06T09:33:34.895 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=16, tag_tid=2, entry_tid=20716' '!=' 'object_number=16, tag_tid=2, entry_tid=20584' 2024-04-06T09:33:34.895 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:33:34.895 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=16, tag_tid=2, entry_tid=20716' = 'object_number=16, tag_tid=2, entry_tid=20584' 2024-04-06T09:33:34.895 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=16, tag_tid=2, entry_tid=20716' 2024-04-06T09:33:34.896 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=25, tag_tid=3, entry_tid=1' 2024-04-06T09:33:34.896 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:33:34.896 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:33:34.897 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=3 2024-04-06T09:33:34.897 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=16, tag_tid=2, entry_tid=20716' 2024-04-06T09:33:34.897 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:33:34.898 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:33:34.898 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=2 2024-04-06T09:33:34.899 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=25, tag_tid=3, entry_tid=1' 2024-04-06T09:33:34.899 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:33:34.899 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:33:34.900 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:33:34.901 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=16, tag_tid=2, entry_tid=20716' 2024-04-06T09:33:34.901 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:33:34.901 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:33:34.902 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=20716 2024-04-06T09:33:34.902 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 3 = 2 -a 1 -le 20716 2024-04-06T09:33:34.902 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:33:34.902 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:33:34.902 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:33:35.104 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:33:35.104 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:33:35.104 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:33:35.104 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:33:35.104 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:33:35.104 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:33:35.104 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:33:35.104 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:33:35.105 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:33:35.105 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:33:35.105 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:33:35.105 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:33:35.105 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:33:35.105 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:33:35.105 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:33:35.105 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:33:35.106 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:33:35.108 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:33:35.108 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:33:35.108 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:33:35.108 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:33:35.108 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:33:35.109 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:33:35.109 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:33:35.109 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:33:35.109 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:33:35.109 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:33:35.109 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:33:35.109 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:33:35.109 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:33:35.109 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:33:35.109 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:33:35.109 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:33:35.110 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:33:35.110 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:33:35.110 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:33:35.110 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:33:35.110 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:33:35.111 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:33:35.111 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:33:35.111 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:33:35.111 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:33:35.111 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:33:35.111 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:33:35.111 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:33:35.111 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:33:35.111 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:33:35.111 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:33:35.111 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:33:35.136 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:33:35.136 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:33:35.136 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:33:35.136 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:33:35.268 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:33:35.270 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:33:35.270 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:33:35.270 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:33:35.270 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:33:35.270 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:33:35.270 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:33:35.270 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:33:35.270 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:33:35.270 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:33:35.271 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:33:35.271 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:33:35.271 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:33:35.272 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:33:35.273 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:33:35.273 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:33:35.318 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 4 2024-04-06T09:33:35.318 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 6 2024-04-06T09:33:35.318 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:33:35.319 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=25, tag_tid=3, entry_tid=1], [object_number=24, tag_tid=3, entry_tid=0], [object_number=27, tag_tid=2, entry_tid=27223], [object_number=26, tag_tid=2, entry_tid=27222]]], state=connected] 2024-04-06T09:33:35.319 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=17, tag_tid=2, entry_tid=20841], [object_number=16, tag_tid=2, entry_tid=20840], [object_number=19, tag_tid=2, entry_tid=20839], [object_number=18, tag_tid=2, entry_tid=20838]]], state=connected] 2024-04-06T09:33:35.322 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:33:35.324 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=25, tag_tid=3, entry_tid=1' 2024-04-06T09:33:35.325 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:33:35.325 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:33:35.325 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:33:35.325 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:33:35.325 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:33:35.325 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:33:35.325 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:33:35.325 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:33:35.325 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:33:35.325 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:33:35.325 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:33:35.325 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:33:35.326 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:33:35.326 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:33:35.326 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:33:35.370 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 4 2024-04-06T09:33:35.370 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 6 2024-04-06T09:33:35.370 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:33:35.370 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=25, tag_tid=3, entry_tid=1], [object_number=24, tag_tid=3, entry_tid=0], [object_number=27, tag_tid=2, entry_tid=27223], [object_number=26, tag_tid=2, entry_tid=27222]]], state=connected] 2024-04-06T09:33:35.370 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=17, tag_tid=2, entry_tid=20841], [object_number=16, tag_tid=2, entry_tid=20840], [object_number=19, tag_tid=2, entry_tid=20839], [object_number=18, tag_tid=2, entry_tid=20838]]], state=connected] 2024-04-06T09:33:35.374 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:33:35.376 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=17, tag_tid=2, entry_tid=20841' 2024-04-06T09:33:35.376 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=25, tag_tid=3, entry_tid=1' -a 'object_number=25, tag_tid=3, entry_tid=1' = 'object_number=17, tag_tid=2, entry_tid=20841' 2024-04-06T09:33:35.376 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=17, tag_tid=2, entry_tid=20841' '!=' 'object_number=16, tag_tid=2, entry_tid=20716' 2024-04-06T09:33:35.376 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:33:35.376 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=17, tag_tid=2, entry_tid=20841' = 'object_number=16, tag_tid=2, entry_tid=20716' 2024-04-06T09:33:35.376 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=17, tag_tid=2, entry_tid=20841' 2024-04-06T09:33:35.376 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=25, tag_tid=3, entry_tid=1' 2024-04-06T09:33:35.376 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:33:35.376 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:33:35.377 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=3 2024-04-06T09:33:35.377 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=17, tag_tid=2, entry_tid=20841' 2024-04-06T09:33:35.378 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:33:35.378 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:33:35.378 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=2 2024-04-06T09:33:35.379 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=25, tag_tid=3, entry_tid=1' 2024-04-06T09:33:35.379 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:33:35.379 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:33:35.380 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:33:35.381 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=17, tag_tid=2, entry_tid=20841' 2024-04-06T09:33:35.381 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:33:35.381 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:33:35.382 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=20841 2024-04-06T09:33:35.382 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 3 = 2 -a 1 -le 20841 2024-04-06T09:33:35.382 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:33:35.382 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:33:35.382 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:33:35.584 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:33:35.584 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:33:35.584 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:33:35.584 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:33:35.584 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:33:35.584 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:33:35.584 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:33:35.584 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:33:35.584 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:33:35.585 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:33:35.585 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:33:35.585 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:33:35.585 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:33:35.585 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:33:35.585 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:33:35.585 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:33:35.586 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:33:35.589 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:33:35.589 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:33:35.589 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:33:35.589 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:33:35.589 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:33:35.589 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:33:35.589 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:33:35.589 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:33:35.589 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:33:35.589 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:33:35.589 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:33:35.590 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:33:35.590 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:33:35.590 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:33:35.590 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:33:35.590 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:33:35.591 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:33:35.591 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:33:35.591 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:33:35.591 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:33:35.591 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:33:35.592 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:33:35.592 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:33:35.592 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:33:35.592 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:33:35.592 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:33:35.592 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:33:35.592 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:33:35.592 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:33:35.592 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:33:35.592 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:33:35.592 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:33:35.617 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:33:35.617 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:33:35.617 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:33:35.617 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:33:35.754 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:33:35.755 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:33:35.756 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:33:35.756 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:33:35.756 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:33:35.756 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:33:35.756 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:33:35.756 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:33:35.756 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:33:35.756 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:33:35.756 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:33:35.756 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:33:35.756 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:33:35.757 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:33:35.758 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:33:35.758 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:33:35.806 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 4 2024-04-06T09:33:35.806 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 6 2024-04-06T09:33:35.806 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:33:35.806 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=25, tag_tid=3, entry_tid=1], [object_number=24, tag_tid=3, entry_tid=0], [object_number=27, tag_tid=2, entry_tid=27223], [object_number=26, tag_tid=2, entry_tid=27222]]], state=connected] 2024-04-06T09:33:35.806 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=23, tag_tid=2, entry_tid=20927], [object_number=18, tag_tid=2, entry_tid=20926], [object_number=17, tag_tid=2, entry_tid=20925], [object_number=16, tag_tid=2, entry_tid=20924]]], state=connected] 2024-04-06T09:33:35.811 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:33:35.812 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=25, tag_tid=3, entry_tid=1' 2024-04-06T09:33:35.813 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:33:35.813 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:33:35.813 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:33:35.813 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:33:35.813 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:33:35.814 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:33:35.814 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:33:35.814 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:33:35.814 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:33:35.814 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:33:35.814 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:33:35.814 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:33:35.814 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:33:35.815 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:33:35.815 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:33:35.863 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 4 2024-04-06T09:33:35.863 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 6 2024-04-06T09:33:35.863 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:33:35.863 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=25, tag_tid=3, entry_tid=1], [object_number=24, tag_tid=3, entry_tid=0], [object_number=27, tag_tid=2, entry_tid=27223], [object_number=26, tag_tid=2, entry_tid=27222]]], state=connected] 2024-04-06T09:33:35.863 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=23, tag_tid=2, entry_tid=20927], [object_number=18, tag_tid=2, entry_tid=20926], [object_number=17, tag_tid=2, entry_tid=20925], [object_number=16, tag_tid=2, entry_tid=20924]]], state=connected] 2024-04-06T09:33:35.866 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:33:35.868 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=23, tag_tid=2, entry_tid=20927' 2024-04-06T09:33:35.868 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=25, tag_tid=3, entry_tid=1' -a 'object_number=25, tag_tid=3, entry_tid=1' = 'object_number=23, tag_tid=2, entry_tid=20927' 2024-04-06T09:33:35.868 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=23, tag_tid=2, entry_tid=20927' '!=' 'object_number=17, tag_tid=2, entry_tid=20841' 2024-04-06T09:33:35.868 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:33:35.868 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=23, tag_tid=2, entry_tid=20927' = 'object_number=17, tag_tid=2, entry_tid=20841' 2024-04-06T09:33:35.868 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=23, tag_tid=2, entry_tid=20927' 2024-04-06T09:33:35.869 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=25, tag_tid=3, entry_tid=1' 2024-04-06T09:33:35.869 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:33:35.869 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:33:35.870 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=3 2024-04-06T09:33:35.870 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=23, tag_tid=2, entry_tid=20927' 2024-04-06T09:33:35.870 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:33:35.871 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:33:35.873 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=2 2024-04-06T09:33:35.873 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=25, tag_tid=3, entry_tid=1' 2024-04-06T09:33:35.873 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:33:35.873 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:33:35.873 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:33:35.874 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=23, tag_tid=2, entry_tid=20927' 2024-04-06T09:33:35.874 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:33:35.874 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:33:35.875 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=20927 2024-04-06T09:33:35.875 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 3 = 2 -a 1 -le 20927 2024-04-06T09:33:35.875 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:33:35.875 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:33:35.875 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:33:36.078 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:33:36.078 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:33:36.078 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:33:36.078 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:33:36.078 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:33:36.078 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:33:36.078 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:33:36.078 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:33:36.078 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:33:36.079 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:33:36.079 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:33:36.079 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:33:36.079 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:33:36.079 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:33:36.079 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:33:36.079 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:33:36.081 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:33:36.081 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:33:36.081 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:33:36.081 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:33:36.081 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:33:36.081 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:33:36.081 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:33:36.082 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:33:36.082 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:33:36.082 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:33:36.082 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:33:36.082 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:33:36.082 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:33:36.082 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:33:36.082 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:33:36.082 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:33:36.082 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:33:36.082 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:33:36.082 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:33:36.082 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:33:36.082 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:33:36.082 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:33:36.083 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:33:36.083 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:33:36.083 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:33:36.083 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:33:36.083 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:33:36.083 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:33:36.083 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:33:36.083 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:33:36.083 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:33:36.083 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:33:36.083 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:33:36.107 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:33:36.107 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:33:36.108 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:33:36.108 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:33:36.245 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:33:36.245 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:33:36.245 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:33:36.246 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:33:36.246 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:33:36.246 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:33:36.246 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:33:36.246 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:33:36.246 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:33:36.246 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:33:36.247 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:33:36.247 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:33:36.248 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:33:36.248 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:33:36.249 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:33:36.249 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:33:36.309 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 5 2024-04-06T09:33:36.309 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 6 2024-04-06T09:33:36.309 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:33:36.309 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=25, tag_tid=3, entry_tid=1], [object_number=24, tag_tid=3, entry_tid=0], [object_number=27, tag_tid=2, entry_tid=27223], [object_number=26, tag_tid=2, entry_tid=27222]]], state=connected] 2024-04-06T09:33:36.309 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=22, tag_tid=2, entry_tid=21010], [object_number=21, tag_tid=2, entry_tid=21009], [object_number=20, tag_tid=2, entry_tid=21008], [object_number=23, tag_tid=2, entry_tid=21007]]], state=connected] 2024-04-06T09:33:36.312 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:33:36.314 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=25, tag_tid=3, entry_tid=1' 2024-04-06T09:33:36.315 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:33:36.315 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:33:36.315 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:33:36.315 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:33:36.315 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:33:36.315 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:33:36.315 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:33:36.315 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:33:36.315 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:33:36.315 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:33:36.315 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:33:36.315 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:33:36.316 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:33:36.316 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:33:36.316 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:33:36.361 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 5 2024-04-06T09:33:36.361 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 6 2024-04-06T09:33:36.361 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:33:36.361 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=25, tag_tid=3, entry_tid=1], [object_number=24, tag_tid=3, entry_tid=0], [object_number=27, tag_tid=2, entry_tid=27223], [object_number=26, tag_tid=2, entry_tid=27222]]], state=connected] 2024-04-06T09:33:36.361 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=22, tag_tid=2, entry_tid=21010], [object_number=21, tag_tid=2, entry_tid=21009], [object_number=20, tag_tid=2, entry_tid=21008], [object_number=23, tag_tid=2, entry_tid=21007]]], state=connected] 2024-04-06T09:33:36.364 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:33:36.366 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=22, tag_tid=2, entry_tid=21010' 2024-04-06T09:33:36.366 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=25, tag_tid=3, entry_tid=1' -a 'object_number=25, tag_tid=3, entry_tid=1' = 'object_number=22, tag_tid=2, entry_tid=21010' 2024-04-06T09:33:36.367 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=22, tag_tid=2, entry_tid=21010' '!=' 'object_number=23, tag_tid=2, entry_tid=20927' 2024-04-06T09:33:36.367 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:33:36.367 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=22, tag_tid=2, entry_tid=21010' = 'object_number=23, tag_tid=2, entry_tid=20927' 2024-04-06T09:33:36.367 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=22, tag_tid=2, entry_tid=21010' 2024-04-06T09:33:36.367 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=25, tag_tid=3, entry_tid=1' 2024-04-06T09:33:36.367 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:33:36.367 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:33:36.367 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=3 2024-04-06T09:33:36.368 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=22, tag_tid=2, entry_tid=21010' 2024-04-06T09:33:36.368 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:33:36.368 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:33:36.369 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=2 2024-04-06T09:33:36.370 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=25, tag_tid=3, entry_tid=1' 2024-04-06T09:33:36.370 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:33:36.370 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:33:36.371 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:33:36.372 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=22, tag_tid=2, entry_tid=21010' 2024-04-06T09:33:36.372 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:33:36.372 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:33:36.373 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=21010 2024-04-06T09:33:36.373 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 3 = 2 -a 1 -le 21010 2024-04-06T09:33:36.373 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:33:36.373 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:33:36.373 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:33:36.575 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:33:36.575 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:33:36.575 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:33:36.575 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:33:36.575 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:33:36.575 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:33:36.575 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:33:36.575 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:33:36.575 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:33:36.575 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:33:36.575 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:33:36.576 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:33:36.576 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:33:36.576 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:33:36.576 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:33:36.576 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:33:36.576 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:33:36.581 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:33:36.581 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:33:36.581 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:33:36.581 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:33:36.581 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:33:36.581 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:33:36.581 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:33:36.581 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:33:36.581 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:33:36.581 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:33:36.581 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:33:36.581 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:33:36.582 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:33:36.582 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:33:36.582 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:33:36.582 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:33:36.582 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:33:36.582 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:33:36.582 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:33:36.582 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:33:36.582 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:33:36.582 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:33:36.582 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:33:36.582 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:33:36.582 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:33:36.582 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:33:36.583 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:33:36.583 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:33:36.583 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:33:36.583 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:33:36.583 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:33:36.583 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:33:36.605 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:33:36.605 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:33:36.606 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:33:36.606 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:33:36.742 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:33:36.743 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:33:36.743 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:33:36.743 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:33:36.743 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:33:36.743 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:33:36.743 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:33:36.743 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:33:36.744 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:33:36.744 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:33:36.744 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:33:36.744 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:33:36.744 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:33:36.745 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:33:36.745 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:33:36.746 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:33:36.791 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 5 2024-04-06T09:33:36.791 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 6 2024-04-06T09:33:36.792 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:33:36.792 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=25, tag_tid=3, entry_tid=1], [object_number=24, tag_tid=3, entry_tid=0], [object_number=27, tag_tid=2, entry_tid=27223], [object_number=26, tag_tid=2, entry_tid=27222]]], state=connected] 2024-04-06T09:33:36.792 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=21, tag_tid=2, entry_tid=21121], [object_number=20, tag_tid=2, entry_tid=21120], [object_number=23, tag_tid=2, entry_tid=21119], [object_number=22, tag_tid=2, entry_tid=21118]]], state=connected] 2024-04-06T09:33:36.795 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:33:36.797 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=25, tag_tid=3, entry_tid=1' 2024-04-06T09:33:36.798 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:33:36.798 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:33:36.798 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:33:36.798 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:33:36.798 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:33:36.798 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:33:36.798 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:33:36.798 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:33:36.798 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:33:36.798 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:33:36.798 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:33:36.798 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:33:36.799 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:33:36.799 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:33:36.799 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:33:36.843 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 5 2024-04-06T09:33:36.843 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 6 2024-04-06T09:33:36.843 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:33:36.843 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=25, tag_tid=3, entry_tid=1], [object_number=24, tag_tid=3, entry_tid=0], [object_number=27, tag_tid=2, entry_tid=27223], [object_number=26, tag_tid=2, entry_tid=27222]]], state=connected] 2024-04-06T09:33:36.843 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=21, tag_tid=2, entry_tid=21145], [object_number=20, tag_tid=2, entry_tid=21144], [object_number=23, tag_tid=2, entry_tid=21143], [object_number=22, tag_tid=2, entry_tid=21142]]], state=connected] 2024-04-06T09:33:36.847 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:33:36.848 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=21, tag_tid=2, entry_tid=21145' 2024-04-06T09:33:36.848 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=25, tag_tid=3, entry_tid=1' -a 'object_number=25, tag_tid=3, entry_tid=1' = 'object_number=21, tag_tid=2, entry_tid=21145' 2024-04-06T09:33:36.849 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=21, tag_tid=2, entry_tid=21145' '!=' 'object_number=22, tag_tid=2, entry_tid=21010' 2024-04-06T09:33:36.849 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:33:36.849 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=21, tag_tid=2, entry_tid=21145' = 'object_number=22, tag_tid=2, entry_tid=21010' 2024-04-06T09:33:36.849 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=21, tag_tid=2, entry_tid=21145' 2024-04-06T09:33:36.849 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=25, tag_tid=3, entry_tid=1' 2024-04-06T09:33:36.849 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:33:36.849 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:33:36.850 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=3 2024-04-06T09:33:36.851 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=21, tag_tid=2, entry_tid=21145' 2024-04-06T09:33:36.851 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:33:36.851 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:33:36.852 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=2 2024-04-06T09:33:36.852 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=25, tag_tid=3, entry_tid=1' 2024-04-06T09:33:36.852 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:33:36.852 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:33:36.853 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:33:36.854 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=21, tag_tid=2, entry_tid=21145' 2024-04-06T09:33:36.854 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:33:36.854 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:33:36.855 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=21145 2024-04-06T09:33:36.855 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 3 = 2 -a 1 -le 21145 2024-04-06T09:33:36.855 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:33:36.855 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:33:36.855 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:33:37.057 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:33:37.057 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:33:37.057 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:33:37.057 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:33:37.057 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:33:37.057 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:33:37.057 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:33:37.058 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:33:37.058 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:33:37.058 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:33:37.058 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:33:37.058 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:33:37.058 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:33:37.058 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:33:37.058 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:33:37.058 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:33:37.059 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:33:37.060 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:33:37.060 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:33:37.060 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:33:37.061 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:33:37.061 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:33:37.061 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:33:37.061 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:33:37.061 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:33:37.061 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:33:37.061 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:33:37.061 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:33:37.061 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:33:37.061 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:33:37.061 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:33:37.061 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:33:37.061 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:33:37.062 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:33:37.062 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:33:37.062 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:33:37.062 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:33:37.062 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:33:37.062 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:33:37.063 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:33:37.063 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:33:37.063 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:33:37.063 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:33:37.063 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:33:37.063 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:33:37.063 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:33:37.063 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:33:37.063 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:33:37.063 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:33:37.087 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:33:37.087 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:33:37.087 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:33:37.087 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:33:37.221 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:33:37.221 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:33:37.222 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:33:37.222 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:33:37.222 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:33:37.222 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:33:37.222 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:33:37.222 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:33:37.222 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:33:37.222 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:33:37.222 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:33:37.223 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:33:37.223 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:33:37.225 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:33:37.225 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:33:37.225 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:33:37.270 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 5 2024-04-06T09:33:37.271 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 6 2024-04-06T09:33:37.271 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:33:37.271 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=25, tag_tid=3, entry_tid=1], [object_number=24, tag_tid=3, entry_tid=0], [object_number=27, tag_tid=2, entry_tid=27223], [object_number=26, tag_tid=2, entry_tid=27222]]], state=connected] 2024-04-06T09:33:37.271 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=22, tag_tid=2, entry_tid=21226], [object_number=21, tag_tid=2, entry_tid=21225], [object_number=20, tag_tid=2, entry_tid=21224], [object_number=23, tag_tid=2, entry_tid=21223]]], state=connected] 2024-04-06T09:33:37.274 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:33:37.276 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=25, tag_tid=3, entry_tid=1' 2024-04-06T09:33:37.276 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:33:37.276 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:33:37.276 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:33:37.277 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:33:37.277 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:33:37.277 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:33:37.277 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:33:37.277 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:33:37.277 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:33:37.277 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:33:37.277 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:33:37.277 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:33:37.278 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:33:37.278 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:33:37.278 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:33:37.322 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 5 2024-04-06T09:33:37.323 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 6 2024-04-06T09:33:37.323 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:33:37.323 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=25, tag_tid=3, entry_tid=1], [object_number=24, tag_tid=3, entry_tid=0], [object_number=27, tag_tid=2, entry_tid=27223], [object_number=26, tag_tid=2, entry_tid=27222]]], state=connected] 2024-04-06T09:33:37.323 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=22, tag_tid=2, entry_tid=21226], [object_number=21, tag_tid=2, entry_tid=21225], [object_number=20, tag_tid=2, entry_tid=21224], [object_number=23, tag_tid=2, entry_tid=21223]]], state=connected] 2024-04-06T09:33:37.326 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:33:37.328 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=22, tag_tid=2, entry_tid=21226' 2024-04-06T09:33:37.328 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=25, tag_tid=3, entry_tid=1' -a 'object_number=25, tag_tid=3, entry_tid=1' = 'object_number=22, tag_tid=2, entry_tid=21226' 2024-04-06T09:33:37.328 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=22, tag_tid=2, entry_tid=21226' '!=' 'object_number=21, tag_tid=2, entry_tid=21145' 2024-04-06T09:33:37.328 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:33:37.328 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=22, tag_tid=2, entry_tid=21226' = 'object_number=21, tag_tid=2, entry_tid=21145' 2024-04-06T09:33:37.328 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=22, tag_tid=2, entry_tid=21226' 2024-04-06T09:33:37.328 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=25, tag_tid=3, entry_tid=1' 2024-04-06T09:33:37.328 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:33:37.329 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:33:37.329 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=3 2024-04-06T09:33:37.330 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=22, tag_tid=2, entry_tid=21226' 2024-04-06T09:33:37.330 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:33:37.330 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:33:37.331 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=2 2024-04-06T09:33:37.331 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=25, tag_tid=3, entry_tid=1' 2024-04-06T09:33:37.332 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:33:37.332 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:33:37.332 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:33:37.333 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=22, tag_tid=2, entry_tid=21226' 2024-04-06T09:33:37.333 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:33:37.333 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:33:37.334 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=21226 2024-04-06T09:33:37.334 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 3 = 2 -a 1 -le 21226 2024-04-06T09:33:37.334 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:33:37.335 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:33:37.335 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:33:37.536 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:33:37.537 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:33:37.537 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:33:37.537 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:33:37.537 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:33:37.537 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:33:37.537 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:33:37.537 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:33:37.537 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:33:37.537 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:33:37.537 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:33:37.537 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:33:37.537 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:33:37.537 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:33:37.537 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:33:37.538 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:33:37.538 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:33:37.541 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:33:37.541 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:33:37.541 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:33:37.541 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:33:37.541 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:33:37.541 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:33:37.541 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:33:37.541 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:33:37.541 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:33:37.541 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:33:37.541 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:33:37.541 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:33:37.542 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:33:37.542 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:33:37.542 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:33:37.542 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:33:37.543 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:33:37.543 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:33:37.543 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:33:37.543 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:33:37.543 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:33:37.543 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:33:37.543 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:33:37.543 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:33:37.543 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:33:37.543 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:33:37.543 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:33:37.543 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:33:37.543 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:33:37.544 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:33:37.544 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:33:37.544 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:33:37.568 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:33:37.568 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:33:37.568 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:33:37.568 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:33:37.701 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:33:37.702 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:33:37.702 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:33:37.702 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:33:37.702 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:33:37.702 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:33:37.702 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:33:37.703 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:33:37.703 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:33:37.703 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:33:37.703 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:33:37.703 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:33:37.703 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:33:37.705 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:33:37.705 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:33:37.705 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:33:37.964 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 5 2024-04-06T09:33:37.964 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 6 2024-04-06T09:33:37.964 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:33:37.964 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=25, tag_tid=3, entry_tid=1], [object_number=24, tag_tid=3, entry_tid=0], [object_number=27, tag_tid=2, entry_tid=27223], [object_number=26, tag_tid=2, entry_tid=27222]]], state=connected] 2024-04-06T09:33:37.964 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=23, tag_tid=2, entry_tid=21315], [object_number=22, tag_tid=2, entry_tid=21314], [object_number=21, tag_tid=2, entry_tid=21313], [object_number=20, tag_tid=2, entry_tid=21312]]], state=connected] 2024-04-06T09:33:37.969 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:33:37.970 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=25, tag_tid=3, entry_tid=1' 2024-04-06T09:33:37.972 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:33:37.972 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:33:37.972 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:33:37.972 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:33:37.972 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:33:37.972 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:33:37.972 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:33:37.972 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:33:37.972 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:33:37.972 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:33:37.972 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:33:37.972 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:33:37.973 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:33:37.974 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:33:37.974 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:33:38.023 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 5 2024-04-06T09:33:38.023 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 6 2024-04-06T09:33:38.023 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:33:38.023 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=25, tag_tid=3, entry_tid=1], [object_number=24, tag_tid=3, entry_tid=0], [object_number=27, tag_tid=2, entry_tid=27223], [object_number=26, tag_tid=2, entry_tid=27222]]], state=connected] 2024-04-06T09:33:38.023 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=23, tag_tid=2, entry_tid=21315], [object_number=22, tag_tid=2, entry_tid=21314], [object_number=21, tag_tid=2, entry_tid=21313], [object_number=20, tag_tid=2, entry_tid=21312]]], state=connected] 2024-04-06T09:33:38.026 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:33:38.029 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=23, tag_tid=2, entry_tid=21315' 2024-04-06T09:33:38.029 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=25, tag_tid=3, entry_tid=1' -a 'object_number=25, tag_tid=3, entry_tid=1' = 'object_number=23, tag_tid=2, entry_tid=21315' 2024-04-06T09:33:38.029 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=23, tag_tid=2, entry_tid=21315' '!=' 'object_number=22, tag_tid=2, entry_tid=21226' 2024-04-06T09:33:38.029 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:33:38.029 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=23, tag_tid=2, entry_tid=21315' = 'object_number=22, tag_tid=2, entry_tid=21226' 2024-04-06T09:33:38.029 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=23, tag_tid=2, entry_tid=21315' 2024-04-06T09:33:38.029 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=25, tag_tid=3, entry_tid=1' 2024-04-06T09:33:38.029 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:33:38.029 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:33:38.030 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=3 2024-04-06T09:33:38.031 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=23, tag_tid=2, entry_tid=21315' 2024-04-06T09:33:38.031 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:33:38.031 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:33:38.032 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=2 2024-04-06T09:33:38.033 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=25, tag_tid=3, entry_tid=1' 2024-04-06T09:33:38.033 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:33:38.033 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:33:38.034 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:33:38.034 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=23, tag_tid=2, entry_tid=21315' 2024-04-06T09:33:38.035 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:33:38.035 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:33:38.036 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=21315 2024-04-06T09:33:38.036 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 3 = 2 -a 1 -le 21315 2024-04-06T09:33:38.036 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:33:38.036 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:33:38.036 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:33:38.237 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:33:38.237 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:33:38.238 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:33:38.238 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:33:38.238 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:33:38.238 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:33:38.238 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:33:38.238 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:33:38.238 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:33:38.238 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:33:38.238 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:33:38.238 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:33:38.238 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:33:38.238 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:33:38.238 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:33:38.239 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:33:38.239 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:33:38.242 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:33:38.242 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:33:38.242 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:33:38.242 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:33:38.242 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:33:38.242 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:33:38.242 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:33:38.242 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:33:38.242 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:33:38.242 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:33:38.242 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:33:38.242 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:33:38.242 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:33:38.243 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:33:38.243 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:33:38.243 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:33:38.243 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:33:38.243 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:33:38.243 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:33:38.243 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:33:38.243 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:33:38.243 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:33:38.243 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:33:38.243 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:33:38.243 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:33:38.243 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:33:38.243 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:33:38.243 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:33:38.243 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:33:38.243 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:33:38.244 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:33:38.244 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:33:38.268 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:33:38.269 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:33:38.269 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:33:38.269 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:33:38.399 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:33:38.400 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:33:38.401 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:33:38.401 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:33:38.401 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:33:38.401 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:33:38.401 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:33:38.401 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:33:38.401 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:33:38.401 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:33:38.401 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:33:38.401 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:33:38.401 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:33:38.403 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:33:38.403 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:33:38.404 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:33:38.454 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 5 2024-04-06T09:33:38.454 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 6 2024-04-06T09:33:38.454 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:33:38.455 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=25, tag_tid=3, entry_tid=1], [object_number=24, tag_tid=3, entry_tid=0], [object_number=27, tag_tid=2, entry_tid=27223], [object_number=26, tag_tid=2, entry_tid=27222]]], state=connected] 2024-04-06T09:33:38.455 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=21, tag_tid=2, entry_tid=21465], [object_number=20, tag_tid=2, entry_tid=21464], [object_number=23, tag_tid=2, entry_tid=21463], [object_number=22, tag_tid=2, entry_tid=21462]]], state=connected] 2024-04-06T09:33:38.458 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:33:38.459 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=25, tag_tid=3, entry_tid=1' 2024-04-06T09:33:38.461 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:33:38.461 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:33:38.461 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:33:38.461 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:33:38.461 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:33:38.461 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:33:38.461 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:33:38.461 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:33:38.461 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:33:38.461 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:33:38.461 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:33:38.461 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:33:38.462 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:33:38.462 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:33:38.462 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:33:38.507 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 5 2024-04-06T09:33:38.507 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 6 2024-04-06T09:33:38.507 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:33:38.507 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=25, tag_tid=3, entry_tid=1], [object_number=24, tag_tid=3, entry_tid=0], [object_number=27, tag_tid=2, entry_tid=27223], [object_number=26, tag_tid=2, entry_tid=27222]]], state=connected] 2024-04-06T09:33:38.507 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=21, tag_tid=2, entry_tid=21465], [object_number=20, tag_tid=2, entry_tid=21464], [object_number=23, tag_tid=2, entry_tid=21463], [object_number=22, tag_tid=2, entry_tid=21462]]], state=connected] 2024-04-06T09:33:38.510 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:33:38.512 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=21, tag_tid=2, entry_tid=21465' 2024-04-06T09:33:38.512 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=25, tag_tid=3, entry_tid=1' -a 'object_number=25, tag_tid=3, entry_tid=1' = 'object_number=21, tag_tid=2, entry_tid=21465' 2024-04-06T09:33:38.513 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=21, tag_tid=2, entry_tid=21465' '!=' 'object_number=23, tag_tid=2, entry_tid=21315' 2024-04-06T09:33:38.513 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:33:38.513 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=21, tag_tid=2, entry_tid=21465' = 'object_number=23, tag_tid=2, entry_tid=21315' 2024-04-06T09:33:38.513 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=21, tag_tid=2, entry_tid=21465' 2024-04-06T09:33:38.513 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=25, tag_tid=3, entry_tid=1' 2024-04-06T09:33:38.513 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:33:38.513 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:33:38.513 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=3 2024-04-06T09:33:38.514 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=21, tag_tid=2, entry_tid=21465' 2024-04-06T09:33:38.514 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:33:38.514 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:33:38.515 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=2 2024-04-06T09:33:38.516 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=25, tag_tid=3, entry_tid=1' 2024-04-06T09:33:38.516 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:33:38.516 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:33:38.517 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:33:38.517 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=21, tag_tid=2, entry_tid=21465' 2024-04-06T09:33:38.518 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:33:38.518 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:33:38.519 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=21465 2024-04-06T09:33:38.519 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 3 = 2 -a 1 -le 21465 2024-04-06T09:33:38.519 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:33:38.519 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:33:38.519 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:33:38.721 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:33:38.721 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:33:38.721 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:33:38.721 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:33:38.721 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:33:38.721 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:33:38.721 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:33:38.721 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:33:38.721 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:33:38.722 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:33:38.722 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:33:38.722 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:33:38.722 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:33:38.722 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:33:38.722 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:33:38.723 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:33:38.723 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:33:38.726 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:33:38.726 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:33:38.726 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:33:38.726 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:33:38.726 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:33:38.726 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:33:38.727 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:33:38.727 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:33:38.727 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:33:38.727 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:33:38.727 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:33:38.727 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:33:38.727 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:33:38.727 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:33:38.727 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:33:38.727 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:33:38.727 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:33:38.727 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:33:38.728 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:33:38.728 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:33:38.728 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:33:38.728 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:33:38.728 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:33:38.728 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:33:38.728 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:33:38.728 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:33:38.728 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:33:38.728 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:33:38.728 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:33:38.728 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:33:38.728 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:33:38.728 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:33:38.751 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:33:38.752 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:33:38.752 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:33:38.752 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:33:38.882 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:33:38.884 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:33:38.884 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:33:38.884 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:33:38.884 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:33:38.884 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:33:38.884 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:33:38.884 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:33:38.884 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:33:38.884 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:33:38.884 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:33:38.885 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:33:38.885 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:33:38.887 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:33:38.887 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:33:38.887 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:33:38.931 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 5 2024-04-06T09:33:38.931 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 6 2024-04-06T09:33:38.931 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:33:38.931 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=25, tag_tid=3, entry_tid=1], [object_number=24, tag_tid=3, entry_tid=0], [object_number=27, tag_tid=2, entry_tid=27223], [object_number=26, tag_tid=2, entry_tid=27222]]], state=connected] 2024-04-06T09:33:38.931 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=22, tag_tid=2, entry_tid=21578], [object_number=21, tag_tid=2, entry_tid=21577], [object_number=20, tag_tid=2, entry_tid=21576], [object_number=23, tag_tid=2, entry_tid=21575]]], state=connected] 2024-04-06T09:33:38.934 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:33:38.936 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=25, tag_tid=3, entry_tid=1' 2024-04-06T09:33:38.937 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:33:38.937 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:33:38.937 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:33:38.937 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:33:38.937 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:33:38.937 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:33:38.937 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:33:38.937 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:33:38.938 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:33:38.938 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:33:38.938 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:33:38.938 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:33:38.938 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:33:38.939 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:33:38.939 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:33:38.983 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 5 2024-04-06T09:33:38.983 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 6 2024-04-06T09:33:38.983 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:33:38.983 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=25, tag_tid=3, entry_tid=1], [object_number=24, tag_tid=3, entry_tid=0], [object_number=27, tag_tid=2, entry_tid=27223], [object_number=26, tag_tid=2, entry_tid=27222]]], state=connected] 2024-04-06T09:33:38.984 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=22, tag_tid=2, entry_tid=21578], [object_number=21, tag_tid=2, entry_tid=21577], [object_number=20, tag_tid=2, entry_tid=21576], [object_number=23, tag_tid=2, entry_tid=21575]]], state=connected] 2024-04-06T09:33:38.987 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:33:38.989 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=22, tag_tid=2, entry_tid=21578' 2024-04-06T09:33:38.989 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=25, tag_tid=3, entry_tid=1' -a 'object_number=25, tag_tid=3, entry_tid=1' = 'object_number=22, tag_tid=2, entry_tid=21578' 2024-04-06T09:33:38.989 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=22, tag_tid=2, entry_tid=21578' '!=' 'object_number=21, tag_tid=2, entry_tid=21465' 2024-04-06T09:33:38.989 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:33:38.989 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=22, tag_tid=2, entry_tid=21578' = 'object_number=21, tag_tid=2, entry_tid=21465' 2024-04-06T09:33:38.990 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=22, tag_tid=2, entry_tid=21578' 2024-04-06T09:33:38.990 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=25, tag_tid=3, entry_tid=1' 2024-04-06T09:33:38.990 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:33:38.990 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:33:38.990 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=3 2024-04-06T09:33:38.991 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=22, tag_tid=2, entry_tid=21578' 2024-04-06T09:33:38.991 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:33:38.991 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:33:38.992 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=2 2024-04-06T09:33:38.992 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=25, tag_tid=3, entry_tid=1' 2024-04-06T09:33:38.992 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:33:38.992 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:33:38.993 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:33:38.994 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=22, tag_tid=2, entry_tid=21578' 2024-04-06T09:33:38.994 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:33:38.994 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:33:38.995 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=21578 2024-04-06T09:33:38.995 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 3 = 2 -a 1 -le 21578 2024-04-06T09:33:38.995 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:33:38.996 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:33:38.996 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:33:39.197 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:33:39.198 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:33:39.198 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:33:39.198 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:33:39.198 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:33:39.198 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:33:39.198 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:33:39.198 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:33:39.198 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:33:39.198 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:33:39.198 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:33:39.198 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:33:39.199 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:33:39.199 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:33:39.199 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:33:39.199 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:33:39.199 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:33:39.202 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:33:39.202 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:33:39.202 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:33:39.203 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:33:39.203 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:33:39.203 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:33:39.203 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:33:39.203 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:33:39.203 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:33:39.203 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:33:39.203 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:33:39.203 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:33:39.203 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:33:39.203 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:33:39.203 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:33:39.203 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:33:39.203 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:33:39.203 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:33:39.204 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:33:39.204 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:33:39.204 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:33:39.204 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:33:39.204 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:33:39.204 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:33:39.204 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:33:39.204 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:33:39.204 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:33:39.204 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:33:39.204 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:33:39.204 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:33:39.204 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:33:39.204 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:33:39.227 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:33:39.227 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:33:39.227 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:33:39.228 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:33:39.355 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:33:39.356 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:33:39.356 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:33:39.356 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:33:39.357 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:33:39.357 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:33:39.357 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:33:39.357 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:33:39.357 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:33:39.357 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:33:39.357 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:33:39.357 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:33:39.357 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:33:39.359 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:33:39.360 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:33:39.360 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:33:39.406 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 5 2024-04-06T09:33:39.406 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 6 2024-04-06T09:33:39.406 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:33:39.406 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=25, tag_tid=3, entry_tid=1], [object_number=24, tag_tid=3, entry_tid=0], [object_number=27, tag_tid=2, entry_tid=27223], [object_number=26, tag_tid=2, entry_tid=27222]]], state=connected] 2024-04-06T09:33:39.406 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=21, tag_tid=2, entry_tid=21681], [object_number=20, tag_tid=2, entry_tid=21680], [object_number=23, tag_tid=2, entry_tid=21679], [object_number=22, tag_tid=2, entry_tid=21678]]], state=connected] 2024-04-06T09:33:39.410 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:33:39.411 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=25, tag_tid=3, entry_tid=1' 2024-04-06T09:33:39.412 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:33:39.412 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:33:39.412 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:33:39.412 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:33:39.412 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:33:39.412 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:33:39.412 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:33:39.412 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:33:39.413 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:33:39.413 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:33:39.413 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:33:39.413 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:33:39.413 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:33:39.413 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:33:39.414 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:33:39.459 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 5 2024-04-06T09:33:39.459 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 6 2024-04-06T09:33:39.460 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:33:39.460 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=25, tag_tid=3, entry_tid=1], [object_number=24, tag_tid=3, entry_tid=0], [object_number=27, tag_tid=2, entry_tid=27223], [object_number=26, tag_tid=2, entry_tid=27222]]], state=connected] 2024-04-06T09:33:39.460 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=21, tag_tid=2, entry_tid=21681], [object_number=20, tag_tid=2, entry_tid=21680], [object_number=23, tag_tid=2, entry_tid=21679], [object_number=22, tag_tid=2, entry_tid=21678]]], state=connected] 2024-04-06T09:33:39.463 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:33:39.465 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=21, tag_tid=2, entry_tid=21681' 2024-04-06T09:33:39.465 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=25, tag_tid=3, entry_tid=1' -a 'object_number=25, tag_tid=3, entry_tid=1' = 'object_number=21, tag_tid=2, entry_tid=21681' 2024-04-06T09:33:39.466 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=21, tag_tid=2, entry_tid=21681' '!=' 'object_number=22, tag_tid=2, entry_tid=21578' 2024-04-06T09:33:39.466 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:33:39.466 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=21, tag_tid=2, entry_tid=21681' = 'object_number=22, tag_tid=2, entry_tid=21578' 2024-04-06T09:33:39.466 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=21, tag_tid=2, entry_tid=21681' 2024-04-06T09:33:39.466 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=25, tag_tid=3, entry_tid=1' 2024-04-06T09:33:39.466 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:33:39.466 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:33:39.466 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=3 2024-04-06T09:33:39.467 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=21, tag_tid=2, entry_tid=21681' 2024-04-06T09:33:39.467 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:33:39.467 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:33:39.468 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=2 2024-04-06T09:33:39.469 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=25, tag_tid=3, entry_tid=1' 2024-04-06T09:33:39.469 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:33:39.469 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:33:39.471 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:33:39.471 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=21, tag_tid=2, entry_tid=21681' 2024-04-06T09:33:39.471 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:33:39.471 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:33:39.472 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=21681 2024-04-06T09:33:39.472 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 3 = 2 -a 1 -le 21681 2024-04-06T09:33:39.472 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:33:39.472 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:33:39.472 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:33:39.674 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:33:39.674 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:33:39.674 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:33:39.674 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:33:39.674 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:33:39.674 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:33:39.674 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:33:39.674 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:33:39.674 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:33:39.674 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:33:39.675 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:33:39.675 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:33:39.675 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:33:39.675 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:33:39.675 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:33:39.675 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:33:39.675 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:33:39.676 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:33:39.676 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:33:39.676 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:33:39.676 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:33:39.676 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:33:39.677 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:33:39.677 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:33:39.677 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:33:39.677 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:33:39.677 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:33:39.677 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:33:39.677 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:33:39.677 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:33:39.677 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:33:39.677 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:33:39.677 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:33:39.678 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:33:39.679 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:33:39.679 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:33:39.679 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:33:39.679 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:33:39.679 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:33:39.679 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:33:39.679 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:33:39.679 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:33:39.679 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:33:39.679 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:33:39.679 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:33:39.679 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:33:39.679 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:33:39.680 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:33:39.680 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:33:39.703 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:33:39.704 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:33:39.704 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:33:39.704 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:33:39.830 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:33:39.832 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:33:39.832 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:33:39.832 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:33:39.832 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:33:39.832 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:33:39.832 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:33:39.832 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:33:39.832 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:33:39.832 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:33:39.832 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:33:39.833 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:33:39.833 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:33:39.835 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:33:39.836 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:33:39.836 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:33:39.880 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 5 2024-04-06T09:33:39.880 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 6 2024-04-06T09:33:39.880 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:33:39.880 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=25, tag_tid=3, entry_tid=1], [object_number=24, tag_tid=3, entry_tid=0], [object_number=27, tag_tid=2, entry_tid=27223], [object_number=26, tag_tid=2, entry_tid=27222]]], state=connected] 2024-04-06T09:33:39.880 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=22, tag_tid=2, entry_tid=21782], [object_number=21, tag_tid=2, entry_tid=21781], [object_number=20, tag_tid=2, entry_tid=21780], [object_number=23, tag_tid=2, entry_tid=21779]]], state=connected] 2024-04-06T09:33:39.883 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:33:39.885 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=25, tag_tid=3, entry_tid=1' 2024-04-06T09:33:39.886 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:33:39.886 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:33:39.886 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:33:39.886 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:33:39.886 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:33:39.886 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:33:39.886 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:33:39.887 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:33:39.887 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:33:39.887 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:33:39.887 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:33:39.887 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:33:39.887 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:33:39.887 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:33:39.888 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:33:39.932 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 5 2024-04-06T09:33:39.932 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 6 2024-04-06T09:33:39.932 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:33:39.932 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=25, tag_tid=3, entry_tid=1], [object_number=24, tag_tid=3, entry_tid=0], [object_number=27, tag_tid=2, entry_tid=27223], [object_number=26, tag_tid=2, entry_tid=27222]]], state=connected] 2024-04-06T09:33:39.932 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=22, tag_tid=2, entry_tid=21782], [object_number=21, tag_tid=2, entry_tid=21781], [object_number=20, tag_tid=2, entry_tid=21780], [object_number=23, tag_tid=2, entry_tid=21779]]], state=connected] 2024-04-06T09:33:39.935 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:33:39.938 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=22, tag_tid=2, entry_tid=21782' 2024-04-06T09:33:39.938 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=25, tag_tid=3, entry_tid=1' -a 'object_number=25, tag_tid=3, entry_tid=1' = 'object_number=22, tag_tid=2, entry_tid=21782' 2024-04-06T09:33:39.938 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=22, tag_tid=2, entry_tid=21782' '!=' 'object_number=21, tag_tid=2, entry_tid=21681' 2024-04-06T09:33:39.938 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:33:39.938 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=22, tag_tid=2, entry_tid=21782' = 'object_number=21, tag_tid=2, entry_tid=21681' 2024-04-06T09:33:39.938 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=22, tag_tid=2, entry_tid=21782' 2024-04-06T09:33:39.938 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=25, tag_tid=3, entry_tid=1' 2024-04-06T09:33:39.938 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:33:39.938 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:33:39.939 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=3 2024-04-06T09:33:39.939 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=22, tag_tid=2, entry_tid=21782' 2024-04-06T09:33:39.940 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:33:39.940 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:33:39.940 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=2 2024-04-06T09:33:39.941 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=25, tag_tid=3, entry_tid=1' 2024-04-06T09:33:39.941 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:33:39.941 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:33:39.942 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:33:39.943 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=22, tag_tid=2, entry_tid=21782' 2024-04-06T09:33:39.943 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:33:39.943 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:33:39.944 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=21782 2024-04-06T09:33:39.944 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 3 = 2 -a 1 -le 21782 2024-04-06T09:33:39.944 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:33:39.944 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:33:39.944 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:33:40.146 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:33:40.146 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:33:40.146 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:33:40.146 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:33:40.146 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:33:40.146 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:33:40.146 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:33:40.146 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:33:40.146 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:33:40.146 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:33:40.146 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:33:40.147 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:33:40.147 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:33:40.147 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:33:40.147 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:33:40.147 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:33:40.147 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:33:40.149 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:33:40.149 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:33:40.150 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:33:40.150 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:33:40.150 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:33:40.150 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:33:40.150 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:33:40.150 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:33:40.150 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:33:40.150 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:33:40.150 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:33:40.150 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:33:40.150 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:33:40.150 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:33:40.150 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:33:40.150 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:33:40.152 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:33:40.152 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:33:40.152 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:33:40.152 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:33:40.152 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:33:40.152 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:33:40.152 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:33:40.152 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:33:40.152 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:33:40.152 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:33:40.152 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:33:40.152 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:33:40.153 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:33:40.153 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:33:40.153 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:33:40.153 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:33:40.177 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:33:40.177 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:33:40.177 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:33:40.177 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:33:40.300 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:33:40.302 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:33:40.302 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:33:40.302 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:33:40.302 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:33:40.302 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:33:40.302 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:33:40.302 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:33:40.302 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:33:40.303 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:33:40.303 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:33:40.303 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:33:40.303 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:33:40.304 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:33:40.305 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:33:40.305 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:33:40.355 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 5 2024-04-06T09:33:40.355 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 6 2024-04-06T09:33:40.355 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:33:40.355 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=25, tag_tid=3, entry_tid=1], [object_number=24, tag_tid=3, entry_tid=0], [object_number=27, tag_tid=2, entry_tid=27223], [object_number=26, tag_tid=2, entry_tid=27222]]], state=connected] 2024-04-06T09:33:40.356 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=23, tag_tid=2, entry_tid=21887], [object_number=22, tag_tid=2, entry_tid=21886], [object_number=21, tag_tid=2, entry_tid=21885], [object_number=20, tag_tid=2, entry_tid=21884]]], state=connected] 2024-04-06T09:33:40.359 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:33:40.361 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=25, tag_tid=3, entry_tid=1' 2024-04-06T09:33:40.362 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:33:40.362 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:33:40.362 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:33:40.362 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:33:40.362 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:33:40.362 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:33:40.362 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:33:40.362 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:33:40.362 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:33:40.362 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:33:40.362 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:33:40.362 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:33:40.363 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:33:40.363 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:33:40.363 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:33:40.408 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 5 2024-04-06T09:33:40.408 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 6 2024-04-06T09:33:40.408 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:33:40.408 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=25, tag_tid=3, entry_tid=1], [object_number=24, tag_tid=3, entry_tid=0], [object_number=27, tag_tid=2, entry_tid=27223], [object_number=26, tag_tid=2, entry_tid=27222]]], state=connected] 2024-04-06T09:33:40.408 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=23, tag_tid=2, entry_tid=21887], [object_number=22, tag_tid=2, entry_tid=21886], [object_number=21, tag_tid=2, entry_tid=21885], [object_number=20, tag_tid=2, entry_tid=21884]]], state=connected] 2024-04-06T09:33:40.412 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:33:40.414 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=23, tag_tid=2, entry_tid=21887' 2024-04-06T09:33:40.414 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=25, tag_tid=3, entry_tid=1' -a 'object_number=25, tag_tid=3, entry_tid=1' = 'object_number=23, tag_tid=2, entry_tid=21887' 2024-04-06T09:33:40.414 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=23, tag_tid=2, entry_tid=21887' '!=' 'object_number=22, tag_tid=2, entry_tid=21782' 2024-04-06T09:33:40.414 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:33:40.414 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=23, tag_tid=2, entry_tid=21887' = 'object_number=22, tag_tid=2, entry_tid=21782' 2024-04-06T09:33:40.414 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=23, tag_tid=2, entry_tid=21887' 2024-04-06T09:33:40.414 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=25, tag_tid=3, entry_tid=1' 2024-04-06T09:33:40.415 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:33:40.415 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:33:40.415 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=3 2024-04-06T09:33:40.416 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=23, tag_tid=2, entry_tid=21887' 2024-04-06T09:33:40.416 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:33:40.416 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:33:40.417 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=2 2024-04-06T09:33:40.417 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=25, tag_tid=3, entry_tid=1' 2024-04-06T09:33:40.418 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:33:40.418 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:33:40.419 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:33:40.419 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=23, tag_tid=2, entry_tid=21887' 2024-04-06T09:33:40.420 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:33:40.420 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:33:40.421 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=21887 2024-04-06T09:33:40.421 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 3 = 2 -a 1 -le 21887 2024-04-06T09:33:40.421 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:33:40.421 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:33:40.421 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:33:40.623 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:33:40.623 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:33:40.623 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:33:40.623 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:33:40.623 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:33:40.623 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:33:40.623 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:33:40.623 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:33:40.623 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:33:40.623 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:33:40.624 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:33:40.624 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:33:40.624 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:33:40.624 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:33:40.624 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:33:40.624 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:33:40.625 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:33:40.627 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:33:40.627 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:33:40.627 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:33:40.627 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:33:40.627 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:33:40.627 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:33:40.627 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:33:40.627 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:33:40.628 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:33:40.628 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:33:40.628 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:33:40.628 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:33:40.628 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:33:40.628 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:33:40.628 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:33:40.628 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:33:40.629 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:33:40.629 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:33:40.629 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:33:40.629 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:33:40.629 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:33:40.629 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:33:40.629 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:33:40.629 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:33:40.630 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:33:40.630 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:33:40.630 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:33:40.630 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:33:40.630 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:33:40.630 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:33:40.630 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:33:40.630 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:33:40.655 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:33:40.655 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:33:40.655 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:33:40.655 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:33:40.786 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:33:40.788 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:33:40.788 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:33:40.788 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:33:40.788 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:33:40.788 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:33:40.788 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:33:40.788 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:33:40.788 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:33:40.788 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:33:40.788 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:33:40.788 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:33:40.789 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:33:40.791 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:33:40.791 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:33:40.791 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:33:40.840 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 5 2024-04-06T09:33:40.840 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 6 2024-04-06T09:33:40.840 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:33:40.841 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=25, tag_tid=3, entry_tid=1], [object_number=24, tag_tid=3, entry_tid=0], [object_number=27, tag_tid=2, entry_tid=27223], [object_number=26, tag_tid=2, entry_tid=27222]]], state=connected] 2024-04-06T09:33:40.841 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=23, tag_tid=2, entry_tid=21983], [object_number=22, tag_tid=2, entry_tid=21982], [object_number=21, tag_tid=2, entry_tid=21981], [object_number=20, tag_tid=2, entry_tid=21980]]], state=connected] 2024-04-06T09:33:40.844 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:33:40.845 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=25, tag_tid=3, entry_tid=1' 2024-04-06T09:33:40.847 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:33:40.847 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:33:40.847 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:33:40.847 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:33:40.847 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:33:40.847 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:33:40.847 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:33:40.847 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:33:40.847 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:33:40.847 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:33:40.847 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:33:40.847 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:33:40.848 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:33:40.848 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:33:40.848 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:33:40.892 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 5 2024-04-06T09:33:40.892 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 6 2024-04-06T09:33:40.892 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:33:40.892 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=25, tag_tid=3, entry_tid=1], [object_number=24, tag_tid=3, entry_tid=0], [object_number=27, tag_tid=2, entry_tid=27223], [object_number=26, tag_tid=2, entry_tid=27222]]], state=connected] 2024-04-06T09:33:40.893 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=23, tag_tid=2, entry_tid=21983], [object_number=22, tag_tid=2, entry_tid=21982], [object_number=21, tag_tid=2, entry_tid=21981], [object_number=20, tag_tid=2, entry_tid=21980]]], state=connected] 2024-04-06T09:33:40.896 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:33:40.898 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=23, tag_tid=2, entry_tid=21983' 2024-04-06T09:33:40.898 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=25, tag_tid=3, entry_tid=1' -a 'object_number=25, tag_tid=3, entry_tid=1' = 'object_number=23, tag_tid=2, entry_tid=21983' 2024-04-06T09:33:40.898 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=23, tag_tid=2, entry_tid=21983' '!=' 'object_number=23, tag_tid=2, entry_tid=21887' 2024-04-06T09:33:40.898 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:33:40.898 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=23, tag_tid=2, entry_tid=21983' = 'object_number=23, tag_tid=2, entry_tid=21887' 2024-04-06T09:33:40.898 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=23, tag_tid=2, entry_tid=21983' 2024-04-06T09:33:40.898 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=25, tag_tid=3, entry_tid=1' 2024-04-06T09:33:40.899 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:33:40.899 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:33:40.899 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=3 2024-04-06T09:33:40.900 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=23, tag_tid=2, entry_tid=21983' 2024-04-06T09:33:40.900 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:33:40.900 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:33:40.901 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=2 2024-04-06T09:33:40.902 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=25, tag_tid=3, entry_tid=1' 2024-04-06T09:33:40.902 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:33:40.902 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:33:40.903 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:33:40.903 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=23, tag_tid=2, entry_tid=21983' 2024-04-06T09:33:40.903 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:33:40.904 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:33:40.905 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=21983 2024-04-06T09:33:40.905 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 3 = 2 -a 1 -le 21983 2024-04-06T09:33:40.905 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:33:40.905 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:33:40.905 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:33:41.107 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:33:41.107 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:33:41.107 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:33:41.107 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:33:41.107 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:33:41.107 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:33:41.107 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:33:41.107 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:33:41.107 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:33:41.107 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:33:41.107 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:33:41.107 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:33:41.108 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:33:41.108 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:33:41.108 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:33:41.108 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:33:41.108 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:33:41.110 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:33:41.110 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:33:41.110 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:33:41.110 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:33:41.110 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:33:41.110 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:33:41.110 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:33:41.111 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:33:41.111 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:33:41.111 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:33:41.111 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:33:41.111 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:33:41.111 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:33:41.111 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:33:41.111 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:33:41.111 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:33:41.111 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:33:41.111 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:33:41.111 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:33:41.111 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:33:41.111 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:33:41.111 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:33:41.112 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:33:41.112 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:33:41.112 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:33:41.112 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:33:41.112 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:33:41.112 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:33:41.112 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:33:41.112 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:33:41.112 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:33:41.112 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:33:41.136 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:33:41.136 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:33:41.136 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:33:41.137 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:33:41.264 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:33:41.266 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:33:41.266 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:33:41.266 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:33:41.266 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:33:41.266 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:33:41.266 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:33:41.266 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:33:41.266 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:33:41.267 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:33:41.267 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:33:41.267 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:33:41.267 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:33:41.269 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:33:41.269 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:33:41.269 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:33:41.313 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 5 2024-04-06T09:33:41.313 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 6 2024-04-06T09:33:41.313 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:33:41.313 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=25, tag_tid=3, entry_tid=1], [object_number=24, tag_tid=3, entry_tid=0], [object_number=27, tag_tid=2, entry_tid=27223], [object_number=26, tag_tid=2, entry_tid=27222]]], state=connected] 2024-04-06T09:33:41.313 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=22, tag_tid=2, entry_tid=22070], [object_number=21, tag_tid=2, entry_tid=22069], [object_number=20, tag_tid=2, entry_tid=22068], [object_number=23, tag_tid=2, entry_tid=22067]]], state=connected] 2024-04-06T09:33:41.317 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:33:41.319 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=25, tag_tid=3, entry_tid=1' 2024-04-06T09:33:41.319 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:33:41.319 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:33:41.320 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:33:41.320 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:33:41.320 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:33:41.320 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:33:41.320 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:33:41.320 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:33:41.320 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:33:41.320 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:33:41.320 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:33:41.320 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:33:41.321 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:33:41.321 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:33:41.321 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:33:41.365 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 5 2024-04-06T09:33:41.365 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 6 2024-04-06T09:33:41.365 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:33:41.365 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=25, tag_tid=3, entry_tid=1], [object_number=24, tag_tid=3, entry_tid=0], [object_number=27, tag_tid=2, entry_tid=27223], [object_number=26, tag_tid=2, entry_tid=27222]]], state=connected] 2024-04-06T09:33:41.365 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=22, tag_tid=2, entry_tid=22070], [object_number=21, tag_tid=2, entry_tid=22069], [object_number=20, tag_tid=2, entry_tid=22068], [object_number=23, tag_tid=2, entry_tid=22067]]], state=connected] 2024-04-06T09:33:41.369 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:33:41.371 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=22, tag_tid=2, entry_tid=22070' 2024-04-06T09:33:41.371 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=25, tag_tid=3, entry_tid=1' -a 'object_number=25, tag_tid=3, entry_tid=1' = 'object_number=22, tag_tid=2, entry_tid=22070' 2024-04-06T09:33:41.371 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=22, tag_tid=2, entry_tid=22070' '!=' 'object_number=23, tag_tid=2, entry_tid=21983' 2024-04-06T09:33:41.371 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:33:41.371 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=22, tag_tid=2, entry_tid=22070' = 'object_number=23, tag_tid=2, entry_tid=21983' 2024-04-06T09:33:41.371 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=22, tag_tid=2, entry_tid=22070' 2024-04-06T09:33:41.371 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=25, tag_tid=3, entry_tid=1' 2024-04-06T09:33:41.371 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:33:41.371 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:33:41.372 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=3 2024-04-06T09:33:41.373 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=22, tag_tid=2, entry_tid=22070' 2024-04-06T09:33:41.373 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:33:41.373 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:33:41.373 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=2 2024-04-06T09:33:41.374 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=25, tag_tid=3, entry_tid=1' 2024-04-06T09:33:41.374 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:33:41.374 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:33:41.375 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:33:41.376 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=22, tag_tid=2, entry_tid=22070' 2024-04-06T09:33:41.376 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:33:41.376 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:33:41.377 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=22070 2024-04-06T09:33:41.377 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 3 = 2 -a 1 -le 22070 2024-04-06T09:33:41.377 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:33:41.377 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:33:41.377 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:33:41.579 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:33:41.579 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:33:41.579 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:33:41.579 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:33:41.579 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:33:41.579 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:33:41.579 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:33:41.579 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:33:41.579 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:33:41.579 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:33:41.580 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:33:41.580 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:33:41.580 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:33:41.580 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:33:41.580 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:33:41.580 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:33:41.580 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:33:41.583 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:33:41.583 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:33:41.583 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:33:41.583 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:33:41.583 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:33:41.583 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:33:41.583 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:33:41.583 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:33:41.584 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:33:41.584 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:33:41.584 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:33:41.584 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:33:41.584 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:33:41.584 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:33:41.584 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:33:41.584 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:33:41.584 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:33:41.584 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:33:41.584 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:33:41.584 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:33:41.584 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:33:41.584 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:33:41.585 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:33:41.585 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:33:41.585 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:33:41.585 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:33:41.585 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:33:41.585 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:33:41.585 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:33:41.585 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:33:41.585 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:33:41.585 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:33:41.608 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:33:41.608 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:33:41.608 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:33:41.608 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:33:41.740 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:33:41.741 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:33:41.741 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:33:41.741 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:33:41.741 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:33:41.741 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:33:41.741 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:33:41.742 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:33:41.742 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:33:41.742 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:33:41.742 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:33:41.742 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:33:41.742 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:33:41.743 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:33:41.744 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:33:41.744 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:33:41.794 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 5 2024-04-06T09:33:41.795 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 6 2024-04-06T09:33:41.795 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:33:41.795 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=25, tag_tid=3, entry_tid=1], [object_number=24, tag_tid=3, entry_tid=0], [object_number=27, tag_tid=2, entry_tid=27223], [object_number=26, tag_tid=2, entry_tid=27222]]], state=connected] 2024-04-06T09:33:41.795 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=23, tag_tid=2, entry_tid=22159], [object_number=22, tag_tid=2, entry_tid=22158], [object_number=21, tag_tid=2, entry_tid=22157], [object_number=20, tag_tid=2, entry_tid=22156]]], state=connected] 2024-04-06T09:33:41.799 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:33:41.800 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=25, tag_tid=3, entry_tid=1' 2024-04-06T09:33:41.801 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:33:41.801 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:33:41.801 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:33:41.801 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:33:41.802 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:33:41.802 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:33:41.802 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:33:41.802 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:33:41.802 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:33:41.802 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:33:41.802 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:33:41.802 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:33:41.803 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:33:41.803 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:33:41.803 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:33:42.053 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 5 2024-04-06T09:33:42.053 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 6 2024-04-06T09:33:42.053 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:33:42.054 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=25, tag_tid=3, entry_tid=1], [object_number=24, tag_tid=3, entry_tid=0], [object_number=27, tag_tid=2, entry_tid=27223], [object_number=26, tag_tid=2, entry_tid=27222]]], state=connected] 2024-04-06T09:33:42.054 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=23, tag_tid=2, entry_tid=22159], [object_number=22, tag_tid=2, entry_tid=22158], [object_number=21, tag_tid=2, entry_tid=22157], [object_number=20, tag_tid=2, entry_tid=22156]]], state=connected] 2024-04-06T09:33:42.058 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:33:42.060 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=23, tag_tid=2, entry_tid=22159' 2024-04-06T09:33:42.060 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=25, tag_tid=3, entry_tid=1' -a 'object_number=25, tag_tid=3, entry_tid=1' = 'object_number=23, tag_tid=2, entry_tid=22159' 2024-04-06T09:33:42.060 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=23, tag_tid=2, entry_tid=22159' '!=' 'object_number=22, tag_tid=2, entry_tid=22070' 2024-04-06T09:33:42.060 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:33:42.060 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=23, tag_tid=2, entry_tid=22159' = 'object_number=22, tag_tid=2, entry_tid=22070' 2024-04-06T09:33:42.061 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=23, tag_tid=2, entry_tid=22159' 2024-04-06T09:33:42.061 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=25, tag_tid=3, entry_tid=1' 2024-04-06T09:33:42.061 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:33:42.062 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:33:42.063 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=3 2024-04-06T09:33:42.064 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=23, tag_tid=2, entry_tid=22159' 2024-04-06T09:33:42.064 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:33:42.064 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:33:42.066 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=2 2024-04-06T09:33:42.067 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=25, tag_tid=3, entry_tid=1' 2024-04-06T09:33:42.067 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:33:42.067 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:33:42.068 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:33:42.069 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=23, tag_tid=2, entry_tid=22159' 2024-04-06T09:33:42.069 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:33:42.070 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:33:42.071 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=22159 2024-04-06T09:33:42.071 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 3 = 2 -a 1 -le 22159 2024-04-06T09:33:42.071 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:33:42.072 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:33:42.072 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:33:42.273 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:33:42.273 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:33:42.273 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:33:42.273 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:33:42.273 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:33:42.274 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:33:42.274 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:33:42.274 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:33:42.274 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:33:42.274 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:33:42.274 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:33:42.274 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:33:42.274 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:33:42.274 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:33:42.274 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:33:42.275 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:33:42.275 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:33:42.278 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:33:42.278 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:33:42.278 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:33:42.278 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:33:42.278 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:33:42.278 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:33:42.278 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:33:42.278 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:33:42.278 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:33:42.278 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:33:42.279 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:33:42.279 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:33:42.279 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:33:42.279 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:33:42.279 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:33:42.279 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:33:42.279 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:33:42.279 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:33:42.279 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:33:42.279 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:33:42.279 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:33:42.279 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:33:42.279 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:33:42.279 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:33:42.279 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:33:42.279 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:33:42.279 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:33:42.280 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:33:42.280 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:33:42.280 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:33:42.280 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:33:42.280 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:33:42.304 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:33:42.304 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:33:42.304 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:33:42.304 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:33:42.435 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:33:42.436 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:33:42.436 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:33:42.436 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:33:42.436 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:33:42.436 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:33:42.436 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:33:42.436 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:33:42.436 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:33:42.436 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:33:42.437 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:33:42.437 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:33:42.437 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:33:42.438 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:33:42.438 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:33:42.438 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:33:42.490 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 5 2024-04-06T09:33:42.491 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 6 2024-04-06T09:33:42.491 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:33:42.491 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=25, tag_tid=3, entry_tid=1], [object_number=24, tag_tid=3, entry_tid=0], [object_number=27, tag_tid=2, entry_tid=27223], [object_number=26, tag_tid=2, entry_tid=27222]]], state=connected] 2024-04-06T09:33:42.491 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=20, tag_tid=2, entry_tid=22300], [object_number=23, tag_tid=2, entry_tid=22299], [object_number=22, tag_tid=2, entry_tid=22298], [object_number=21, tag_tid=2, entry_tid=22297]]], state=connected] 2024-04-06T09:33:42.495 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:33:42.496 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=25, tag_tid=3, entry_tid=1' 2024-04-06T09:33:42.497 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:33:42.497 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:33:42.497 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:33:42.497 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:33:42.498 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:33:42.498 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:33:42.498 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:33:42.498 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:33:42.498 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:33:42.498 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:33:42.498 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:33:42.498 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:33:42.499 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:33:42.499 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:33:42.499 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:33:42.544 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 5 2024-04-06T09:33:42.545 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 6 2024-04-06T09:33:42.545 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:33:42.545 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=25, tag_tid=3, entry_tid=1], [object_number=24, tag_tid=3, entry_tid=0], [object_number=27, tag_tid=2, entry_tid=27223], [object_number=26, tag_tid=2, entry_tid=27222]]], state=connected] 2024-04-06T09:33:42.545 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=20, tag_tid=2, entry_tid=22300], [object_number=23, tag_tid=2, entry_tid=22299], [object_number=22, tag_tid=2, entry_tid=22298], [object_number=21, tag_tid=2, entry_tid=22297]]], state=connected] 2024-04-06T09:33:42.548 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:33:42.550 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=20, tag_tid=2, entry_tid=22300' 2024-04-06T09:33:42.550 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=25, tag_tid=3, entry_tid=1' -a 'object_number=25, tag_tid=3, entry_tid=1' = 'object_number=20, tag_tid=2, entry_tid=22300' 2024-04-06T09:33:42.550 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=20, tag_tid=2, entry_tid=22300' '!=' 'object_number=23, tag_tid=2, entry_tid=22159' 2024-04-06T09:33:42.551 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:33:42.551 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=20, tag_tid=2, entry_tid=22300' = 'object_number=23, tag_tid=2, entry_tid=22159' 2024-04-06T09:33:42.551 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=20, tag_tid=2, entry_tid=22300' 2024-04-06T09:33:42.551 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=25, tag_tid=3, entry_tid=1' 2024-04-06T09:33:42.551 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:33:42.551 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:33:42.551 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=3 2024-04-06T09:33:42.552 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=20, tag_tid=2, entry_tid=22300' 2024-04-06T09:33:42.552 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:33:42.552 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:33:42.553 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=2 2024-04-06T09:33:42.554 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=25, tag_tid=3, entry_tid=1' 2024-04-06T09:33:42.554 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:33:42.554 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:33:42.555 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:33:42.555 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=20, tag_tid=2, entry_tid=22300' 2024-04-06T09:33:42.555 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:33:42.556 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:33:42.557 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=22300 2024-04-06T09:33:42.557 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 3 = 2 -a 1 -le 22300 2024-04-06T09:33:42.557 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:33:42.557 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:33:42.557 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:33:42.758 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:33:42.758 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:33:42.758 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:33:42.759 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:33:42.759 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:33:42.759 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:33:42.759 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:33:42.759 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:33:42.759 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:33:42.759 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:33:42.759 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:33:42.759 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:33:42.759 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:33:42.759 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:33:42.759 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:33:42.760 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:33:42.760 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:33:42.764 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:33:42.764 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:33:42.764 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:33:42.764 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:33:42.764 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:33:42.764 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:33:42.765 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:33:42.765 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:33:42.765 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:33:42.765 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:33:42.765 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:33:42.765 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:33:42.765 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:33:42.765 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:33:42.765 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:33:42.765 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:33:42.765 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:33:42.765 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:33:42.765 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:33:42.765 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:33:42.765 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:33:42.766 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:33:42.766 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:33:42.766 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:33:42.766 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:33:42.766 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:33:42.766 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:33:42.766 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:33:42.766 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:33:42.766 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:33:42.766 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:33:42.766 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:33:42.790 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:33:42.790 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:33:42.790 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:33:42.790 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:33:42.917 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:33:42.919 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:33:42.919 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:33:42.919 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:33:42.919 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:33:42.919 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:33:42.919 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:33:42.919 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:33:42.919 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:33:42.919 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:33:42.920 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:33:42.920 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:33:42.920 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:33:42.921 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:33:42.921 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:33:42.921 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:33:42.976 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 5 2024-04-06T09:33:42.976 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 6 2024-04-06T09:33:42.977 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:33:42.977 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=25, tag_tid=3, entry_tid=1], [object_number=24, tag_tid=3, entry_tid=0], [object_number=27, tag_tid=2, entry_tid=27223], [object_number=26, tag_tid=2, entry_tid=27222]]], state=connected] 2024-04-06T09:33:42.977 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=22, tag_tid=2, entry_tid=22394], [object_number=21, tag_tid=2, entry_tid=22393], [object_number=20, tag_tid=2, entry_tid=22392], [object_number=23, tag_tid=2, entry_tid=22391]]], state=connected] 2024-04-06T09:33:42.980 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:33:42.981 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=25, tag_tid=3, entry_tid=1' 2024-04-06T09:33:42.982 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:33:42.983 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:33:42.983 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:33:42.983 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:33:42.983 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:33:42.983 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:33:42.983 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:33:42.983 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:33:42.983 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:33:42.983 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:33:42.983 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:33:42.983 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:33:42.984 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:33:42.984 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:33:42.984 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:33:43.029 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 5 2024-04-06T09:33:43.029 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 6 2024-04-06T09:33:43.029 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:33:43.029 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=25, tag_tid=3, entry_tid=1], [object_number=24, tag_tid=3, entry_tid=0], [object_number=27, tag_tid=2, entry_tid=27223], [object_number=26, tag_tid=2, entry_tid=27222]]], state=connected] 2024-04-06T09:33:43.029 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=22, tag_tid=2, entry_tid=22394], [object_number=21, tag_tid=2, entry_tid=22393], [object_number=20, tag_tid=2, entry_tid=22392], [object_number=23, tag_tid=2, entry_tid=22391]]], state=connected] 2024-04-06T09:33:43.032 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:33:43.035 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=22, tag_tid=2, entry_tid=22394' 2024-04-06T09:33:43.035 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=25, tag_tid=3, entry_tid=1' -a 'object_number=25, tag_tid=3, entry_tid=1' = 'object_number=22, tag_tid=2, entry_tid=22394' 2024-04-06T09:33:43.035 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=22, tag_tid=2, entry_tid=22394' '!=' 'object_number=20, tag_tid=2, entry_tid=22300' 2024-04-06T09:33:43.035 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:33:43.035 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=22, tag_tid=2, entry_tid=22394' = 'object_number=20, tag_tid=2, entry_tid=22300' 2024-04-06T09:33:43.035 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=22, tag_tid=2, entry_tid=22394' 2024-04-06T09:33:43.035 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=25, tag_tid=3, entry_tid=1' 2024-04-06T09:33:43.035 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:33:43.035 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:33:43.036 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=3 2024-04-06T09:33:43.036 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=22, tag_tid=2, entry_tid=22394' 2024-04-06T09:33:43.037 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:33:43.037 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:33:43.037 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=2 2024-04-06T09:33:43.038 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=25, tag_tid=3, entry_tid=1' 2024-04-06T09:33:43.038 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:33:43.038 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:33:43.039 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:33:43.040 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=22, tag_tid=2, entry_tid=22394' 2024-04-06T09:33:43.040 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:33:43.040 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:33:43.041 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=22394 2024-04-06T09:33:43.041 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 3 = 2 -a 1 -le 22394 2024-04-06T09:33:43.041 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:33:43.041 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:33:43.041 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:33:43.242 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:33:43.243 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:33:43.243 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:33:43.243 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:33:43.243 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:33:43.243 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:33:43.243 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:33:43.243 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:33:43.243 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:33:43.243 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:33:43.243 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:33:43.243 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:33:43.243 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:33:43.243 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:33:43.243 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:33:43.244 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:33:43.244 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:33:43.248 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:33:43.248 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:33:43.248 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:33:43.248 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:33:43.248 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:33:43.248 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:33:43.248 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:33:43.248 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:33:43.248 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:33:43.248 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:33:43.248 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:33:43.248 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:33:43.248 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:33:43.248 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:33:43.248 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:33:43.249 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:33:43.249 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:33:43.249 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:33:43.249 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:33:43.249 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:33:43.249 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:33:43.249 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:33:43.249 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:33:43.249 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:33:43.249 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:33:43.249 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:33:43.249 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:33:43.249 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:33:43.249 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:33:43.249 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:33:43.249 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:33:43.249 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:33:43.272 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:33:43.272 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:33:43.273 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:33:43.273 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:33:43.408 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:33:43.410 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:33:43.410 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:33:43.410 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:33:43.410 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:33:43.410 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:33:43.410 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:33:43.410 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:33:43.410 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:33:43.410 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:33:43.410 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:33:43.411 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:33:43.411 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:33:43.412 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:33:43.412 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:33:43.413 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:33:43.460 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 5 2024-04-06T09:33:43.460 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 6 2024-04-06T09:33:43.460 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:33:43.460 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=25, tag_tid=3, entry_tid=1], [object_number=24, tag_tid=3, entry_tid=0], [object_number=27, tag_tid=2, entry_tid=27223], [object_number=26, tag_tid=2, entry_tid=27222]]], state=connected] 2024-04-06T09:33:43.460 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=22, tag_tid=2, entry_tid=22490], [object_number=21, tag_tid=2, entry_tid=22489], [object_number=20, tag_tid=2, entry_tid=22488], [object_number=23, tag_tid=2, entry_tid=22487]]], state=connected] 2024-04-06T09:33:43.464 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:33:43.465 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=25, tag_tid=3, entry_tid=1' 2024-04-06T09:33:43.467 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:33:43.467 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:33:43.467 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:33:43.467 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:33:43.467 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:33:43.467 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:33:43.467 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:33:43.467 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:33:43.467 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:33:43.467 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:33:43.467 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:33:43.467 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:33:43.468 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:33:43.468 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:33:43.468 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:33:43.515 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 5 2024-04-06T09:33:43.515 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 6 2024-04-06T09:33:43.515 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:33:43.515 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=25, tag_tid=3, entry_tid=1], [object_number=24, tag_tid=3, entry_tid=0], [object_number=27, tag_tid=2, entry_tid=27223], [object_number=26, tag_tid=2, entry_tid=27222]]], state=connected] 2024-04-06T09:33:43.516 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=22, tag_tid=2, entry_tid=22490], [object_number=21, tag_tid=2, entry_tid=22489], [object_number=20, tag_tid=2, entry_tid=22488], [object_number=23, tag_tid=2, entry_tid=22487]]], state=connected] 2024-04-06T09:33:43.519 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:33:43.521 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=22, tag_tid=2, entry_tid=22490' 2024-04-06T09:33:43.521 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=25, tag_tid=3, entry_tid=1' -a 'object_number=25, tag_tid=3, entry_tid=1' = 'object_number=22, tag_tid=2, entry_tid=22490' 2024-04-06T09:33:43.521 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=22, tag_tid=2, entry_tid=22490' '!=' 'object_number=22, tag_tid=2, entry_tid=22394' 2024-04-06T09:33:43.522 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:33:43.522 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=22, tag_tid=2, entry_tid=22490' = 'object_number=22, tag_tid=2, entry_tid=22394' 2024-04-06T09:33:43.522 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=22, tag_tid=2, entry_tid=22490' 2024-04-06T09:33:43.522 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=25, tag_tid=3, entry_tid=1' 2024-04-06T09:33:43.522 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:33:43.522 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:33:43.522 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=3 2024-04-06T09:33:43.523 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=22, tag_tid=2, entry_tid=22490' 2024-04-06T09:33:43.523 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:33:43.523 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:33:43.524 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=2 2024-04-06T09:33:43.525 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=25, tag_tid=3, entry_tid=1' 2024-04-06T09:33:43.525 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:33:43.525 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:33:43.526 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:33:43.526 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=22, tag_tid=2, entry_tid=22490' 2024-04-06T09:33:43.526 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:33:43.527 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:33:43.528 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=22490 2024-04-06T09:33:43.528 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 3 = 2 -a 1 -le 22490 2024-04-06T09:33:43.528 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:33:43.528 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:33:43.528 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:33:43.730 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:33:43.730 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:33:43.730 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:33:43.730 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:33:43.730 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:33:43.730 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:33:43.731 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:33:43.731 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:33:43.731 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:33:43.731 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:33:43.731 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:33:43.731 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:33:43.731 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:33:43.731 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:33:43.731 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:33:43.731 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:33:43.731 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:33:43.734 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:33:43.734 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:33:43.734 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:33:43.734 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:33:43.734 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:33:43.734 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:33:43.734 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:33:43.734 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:33:43.735 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:33:43.735 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:33:43.735 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:33:43.735 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:33:43.735 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:33:43.735 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:33:43.735 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:33:43.735 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:33:43.735 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:33:43.735 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:33:43.735 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:33:43.735 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:33:43.735 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:33:43.735 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:33:43.735 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:33:43.735 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:33:43.736 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:33:43.736 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:33:43.736 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:33:43.736 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:33:43.736 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:33:43.736 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:33:43.736 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:33:43.736 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:33:43.759 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:33:43.759 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:33:43.760 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:33:43.760 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:33:43.886 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:33:43.887 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:33:43.887 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:33:43.887 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:33:43.887 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:33:43.887 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:33:43.887 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:33:43.887 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:33:43.887 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:33:43.887 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:33:43.888 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:33:43.888 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:33:43.888 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:33:43.890 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:33:43.891 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:33:43.891 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:33:43.936 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 5 2024-04-06T09:33:43.936 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 6 2024-04-06T09:33:43.936 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:33:43.936 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=25, tag_tid=3, entry_tid=1], [object_number=24, tag_tid=3, entry_tid=0], [object_number=27, tag_tid=2, entry_tid=27223], [object_number=26, tag_tid=2, entry_tid=27222]]], state=connected] 2024-04-06T09:33:43.936 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=20, tag_tid=2, entry_tid=22596], [object_number=23, tag_tid=2, entry_tid=22595], [object_number=22, tag_tid=2, entry_tid=22594], [object_number=21, tag_tid=2, entry_tid=22593]]], state=connected] 2024-04-06T09:33:43.939 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:33:43.941 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=25, tag_tid=3, entry_tid=1' 2024-04-06T09:33:43.942 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:33:43.942 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:33:43.942 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:33:43.942 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:33:43.942 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:33:43.942 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:33:43.942 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:33:43.942 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:33:43.942 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:33:43.942 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:33:43.942 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:33:43.943 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:33:43.943 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:33:43.943 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:33:43.943 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:33:43.988 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 5 2024-04-06T09:33:43.988 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 6 2024-04-06T09:33:43.988 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:33:43.988 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=25, tag_tid=3, entry_tid=1], [object_number=24, tag_tid=3, entry_tid=0], [object_number=27, tag_tid=2, entry_tid=27223], [object_number=26, tag_tid=2, entry_tid=27222]]], state=connected] 2024-04-06T09:33:43.988 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=20, tag_tid=2, entry_tid=22596], [object_number=23, tag_tid=2, entry_tid=22595], [object_number=22, tag_tid=2, entry_tid=22594], [object_number=21, tag_tid=2, entry_tid=22593]]], state=connected] 2024-04-06T09:33:43.991 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:33:43.993 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=20, tag_tid=2, entry_tid=22596' 2024-04-06T09:33:43.993 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=25, tag_tid=3, entry_tid=1' -a 'object_number=25, tag_tid=3, entry_tid=1' = 'object_number=20, tag_tid=2, entry_tid=22596' 2024-04-06T09:33:43.994 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=20, tag_tid=2, entry_tid=22596' '!=' 'object_number=22, tag_tid=2, entry_tid=22490' 2024-04-06T09:33:43.994 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:33:43.994 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=20, tag_tid=2, entry_tid=22596' = 'object_number=22, tag_tid=2, entry_tid=22490' 2024-04-06T09:33:43.994 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=20, tag_tid=2, entry_tid=22596' 2024-04-06T09:33:43.994 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=25, tag_tid=3, entry_tid=1' 2024-04-06T09:33:43.994 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:33:43.994 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:33:43.995 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=3 2024-04-06T09:33:43.995 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=20, tag_tid=2, entry_tid=22596' 2024-04-06T09:33:43.995 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:33:43.995 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:33:43.996 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=2 2024-04-06T09:33:43.997 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=25, tag_tid=3, entry_tid=1' 2024-04-06T09:33:43.997 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:33:43.997 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:33:43.999 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:33:43.999 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=20, tag_tid=2, entry_tid=22596' 2024-04-06T09:33:43.999 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:33:43.999 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:33:44.000 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=22596 2024-04-06T09:33:44.000 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 3 = 2 -a 1 -le 22596 2024-04-06T09:33:44.000 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:33:44.000 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:33:44.000 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:33:44.201 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:33:44.202 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:33:44.202 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:33:44.202 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:33:44.202 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:33:44.202 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:33:44.202 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:33:44.202 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:33:44.202 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:33:44.202 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:33:44.202 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:33:44.202 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:33:44.202 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:33:44.203 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:33:44.203 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:33:44.203 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:33:44.203 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:33:44.206 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:33:44.206 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:33:44.206 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:33:44.206 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:33:44.206 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:33:44.206 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:33:44.206 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:33:44.206 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:33:44.206 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:33:44.206 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:33:44.207 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:33:44.207 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:33:44.207 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:33:44.207 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:33:44.207 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:33:44.207 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:33:44.207 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:33:44.207 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:33:44.207 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:33:44.207 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:33:44.207 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:33:44.207 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:33:44.207 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:33:44.207 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:33:44.208 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:33:44.208 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:33:44.208 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:33:44.208 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:33:44.208 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:33:44.208 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:33:44.208 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:33:44.208 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:33:44.232 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:33:44.232 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:33:44.232 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:33:44.232 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:33:44.362 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:33:44.364 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:33:44.364 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:33:44.364 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:33:44.364 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:33:44.364 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:33:44.364 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:33:44.364 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:33:44.364 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:33:44.364 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:33:44.364 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:33:44.365 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:33:44.365 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:33:44.366 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:33:44.367 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:33:44.367 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:33:44.411 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 5 2024-04-06T09:33:44.411 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 6 2024-04-06T09:33:44.411 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:33:44.411 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=25, tag_tid=3, entry_tid=1], [object_number=24, tag_tid=3, entry_tid=0], [object_number=27, tag_tid=2, entry_tid=27223], [object_number=26, tag_tid=2, entry_tid=27222]]], state=connected] 2024-04-06T09:33:44.411 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=22, tag_tid=2, entry_tid=22690], [object_number=21, tag_tid=2, entry_tid=22689], [object_number=20, tag_tid=2, entry_tid=22688], [object_number=23, tag_tid=2, entry_tid=22687]]], state=connected] 2024-04-06T09:33:44.415 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:33:44.416 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=25, tag_tid=3, entry_tid=1' 2024-04-06T09:33:44.417 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:33:44.417 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:33:44.417 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:33:44.417 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:33:44.417 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:33:44.417 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:33:44.417 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:33:44.418 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:33:44.418 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:33:44.418 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:33:44.418 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:33:44.418 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:33:44.418 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:33:44.419 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:33:44.419 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:33:44.463 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 5 2024-04-06T09:33:44.463 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 6 2024-04-06T09:33:44.463 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:33:44.463 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=25, tag_tid=3, entry_tid=1], [object_number=24, tag_tid=3, entry_tid=0], [object_number=27, tag_tid=2, entry_tid=27223], [object_number=26, tag_tid=2, entry_tid=27222]]], state=connected] 2024-04-06T09:33:44.463 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=22, tag_tid=2, entry_tid=22690], [object_number=21, tag_tid=2, entry_tid=22689], [object_number=20, tag_tid=2, entry_tid=22688], [object_number=23, tag_tid=2, entry_tid=22687]]], state=connected] 2024-04-06T09:33:44.466 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:33:44.468 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=22, tag_tid=2, entry_tid=22690' 2024-04-06T09:33:44.468 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=25, tag_tid=3, entry_tid=1' -a 'object_number=25, tag_tid=3, entry_tid=1' = 'object_number=22, tag_tid=2, entry_tid=22690' 2024-04-06T09:33:44.469 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=22, tag_tid=2, entry_tid=22690' '!=' 'object_number=20, tag_tid=2, entry_tid=22596' 2024-04-06T09:33:44.469 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:33:44.469 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=22, tag_tid=2, entry_tid=22690' = 'object_number=20, tag_tid=2, entry_tid=22596' 2024-04-06T09:33:44.469 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=22, tag_tid=2, entry_tid=22690' 2024-04-06T09:33:44.469 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=25, tag_tid=3, entry_tid=1' 2024-04-06T09:33:44.469 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:33:44.469 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:33:44.469 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=3 2024-04-06T09:33:44.470 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=22, tag_tid=2, entry_tid=22690' 2024-04-06T09:33:44.470 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:33:44.470 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:33:44.471 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=2 2024-04-06T09:33:44.472 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=25, tag_tid=3, entry_tid=1' 2024-04-06T09:33:44.472 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:33:44.472 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:33:44.473 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:33:44.473 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=22, tag_tid=2, entry_tid=22690' 2024-04-06T09:33:44.474 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:33:44.474 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:33:44.475 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=22690 2024-04-06T09:33:44.475 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 3 = 2 -a 1 -le 22690 2024-04-06T09:33:44.475 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:33:44.475 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:33:44.475 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:33:44.677 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:33:44.677 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:33:44.677 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:33:44.677 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:33:44.677 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:33:44.677 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:33:44.677 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:33:44.677 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:33:44.677 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:33:44.677 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:33:44.677 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:33:44.677 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:33:44.678 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:33:44.678 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:33:44.678 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:33:44.678 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:33:44.679 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:33:44.681 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:33:44.681 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:33:44.681 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:33:44.681 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:33:44.681 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:33:44.681 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:33:44.681 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:33:44.681 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:33:44.681 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:33:44.681 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:33:44.682 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:33:44.682 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:33:44.682 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:33:44.682 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:33:44.682 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:33:44.682 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:33:44.683 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:33:44.683 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:33:44.683 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:33:44.683 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:33:44.683 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:33:44.683 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:33:44.683 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:33:44.683 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:33:44.683 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:33:44.683 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:33:44.683 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:33:44.683 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:33:44.683 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:33:44.684 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:33:44.684 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:33:44.684 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:33:44.707 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:33:44.708 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:33:44.708 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:33:44.708 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:33:44.832 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:33:44.833 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:33:44.833 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:33:44.833 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:33:44.833 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:33:44.833 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:33:44.833 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:33:44.833 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:33:44.833 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:33:44.833 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:33:44.834 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:33:44.834 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:33:44.834 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:33:44.836 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:33:44.836 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:33:44.836 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:33:44.881 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 5 2024-04-06T09:33:44.881 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 6 2024-04-06T09:33:44.881 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:33:44.881 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=25, tag_tid=3, entry_tid=1], [object_number=24, tag_tid=3, entry_tid=0], [object_number=27, tag_tid=2, entry_tid=27223], [object_number=26, tag_tid=2, entry_tid=27222]]], state=connected] 2024-04-06T09:33:44.881 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=20, tag_tid=2, entry_tid=22792], [object_number=23, tag_tid=2, entry_tid=22791], [object_number=22, tag_tid=2, entry_tid=22790], [object_number=21, tag_tid=2, entry_tid=22789]]], state=connected] 2024-04-06T09:33:44.885 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:33:44.886 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=25, tag_tid=3, entry_tid=1' 2024-04-06T09:33:44.887 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:33:44.887 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:33:44.887 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:33:44.888 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:33:44.888 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:33:44.888 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:33:44.888 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:33:44.888 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:33:44.888 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:33:44.888 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:33:44.888 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:33:44.888 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:33:44.889 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:33:44.889 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:33:44.889 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:33:44.933 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 5 2024-04-06T09:33:44.933 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 6 2024-04-06T09:33:44.933 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:33:44.933 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=25, tag_tid=3, entry_tid=1], [object_number=24, tag_tid=3, entry_tid=0], [object_number=27, tag_tid=2, entry_tid=27223], [object_number=26, tag_tid=2, entry_tid=27222]]], state=connected] 2024-04-06T09:33:44.933 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=20, tag_tid=2, entry_tid=22792], [object_number=23, tag_tid=2, entry_tid=22791], [object_number=22, tag_tid=2, entry_tid=22790], [object_number=21, tag_tid=2, entry_tid=22789]]], state=connected] 2024-04-06T09:33:44.936 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:33:44.939 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=20, tag_tid=2, entry_tid=22792' 2024-04-06T09:33:44.939 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=25, tag_tid=3, entry_tid=1' -a 'object_number=25, tag_tid=3, entry_tid=1' = 'object_number=20, tag_tid=2, entry_tid=22792' 2024-04-06T09:33:44.939 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=20, tag_tid=2, entry_tid=22792' '!=' 'object_number=22, tag_tid=2, entry_tid=22690' 2024-04-06T09:33:44.939 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:33:44.939 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=20, tag_tid=2, entry_tid=22792' = 'object_number=22, tag_tid=2, entry_tid=22690' 2024-04-06T09:33:44.939 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=20, tag_tid=2, entry_tid=22792' 2024-04-06T09:33:44.939 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=25, tag_tid=3, entry_tid=1' 2024-04-06T09:33:44.939 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:33:44.939 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:33:44.940 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=3 2024-04-06T09:33:44.940 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=20, tag_tid=2, entry_tid=22792' 2024-04-06T09:33:44.940 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:33:44.941 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:33:44.941 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=2 2024-04-06T09:33:44.942 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=25, tag_tid=3, entry_tid=1' 2024-04-06T09:33:44.942 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:33:44.942 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:33:44.943 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:33:44.944 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=20, tag_tid=2, entry_tid=22792' 2024-04-06T09:33:44.944 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:33:44.944 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:33:44.945 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=22792 2024-04-06T09:33:44.945 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 3 = 2 -a 1 -le 22792 2024-04-06T09:33:44.945 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:33:44.945 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:33:44.945 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:33:45.147 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:33:45.147 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:33:45.147 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:33:45.147 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:33:45.148 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:33:45.148 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:33:45.148 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:33:45.148 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:33:45.148 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:33:45.148 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:33:45.148 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:33:45.148 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:33:45.148 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:33:45.148 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:33:45.148 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:33:45.149 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:33:45.149 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:33:45.150 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:33:45.150 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:33:45.150 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:33:45.150 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:33:45.150 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:33:45.150 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:33:45.150 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:33:45.151 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:33:45.151 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:33:45.151 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:33:45.151 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:33:45.151 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:33:45.151 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:33:45.151 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:33:45.151 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:33:45.151 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:33:45.153 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:33:45.153 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:33:45.153 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:33:45.153 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:33:45.153 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:33:45.153 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:33:45.153 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:33:45.153 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:33:45.153 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:33:45.153 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:33:45.153 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:33:45.154 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:33:45.154 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:33:45.154 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:33:45.154 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:33:45.154 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:33:45.176 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:33:45.177 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:33:45.177 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:33:45.177 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:33:45.305 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:33:45.305 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:33:45.305 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:33:45.305 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:33:45.306 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:33:45.306 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:33:45.306 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:33:45.306 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:33:45.306 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:33:45.306 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:33:45.306 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:33:45.306 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:33:45.307 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:33:45.308 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:33:45.308 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:33:45.309 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:33:45.360 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 5 2024-04-06T09:33:45.360 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 6 2024-04-06T09:33:45.360 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:33:45.360 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=25, tag_tid=3, entry_tid=1], [object_number=24, tag_tid=3, entry_tid=0], [object_number=27, tag_tid=2, entry_tid=27223], [object_number=26, tag_tid=2, entry_tid=27222]]], state=connected] 2024-04-06T09:33:45.360 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=20, tag_tid=2, entry_tid=22904], [object_number=23, tag_tid=2, entry_tid=22903], [object_number=22, tag_tid=2, entry_tid=22902], [object_number=21, tag_tid=2, entry_tid=22901]]], state=connected] 2024-04-06T09:33:45.363 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:33:45.365 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=25, tag_tid=3, entry_tid=1' 2024-04-06T09:33:45.366 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:33:45.366 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:33:45.366 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:33:45.366 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:33:45.366 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:33:45.366 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:33:45.366 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:33:45.367 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:33:45.367 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:33:45.367 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:33:45.367 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:33:45.367 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:33:45.368 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:33:45.368 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:33:45.368 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:33:45.414 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 5 2024-04-06T09:33:45.414 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 6 2024-04-06T09:33:45.414 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:33:45.414 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=25, tag_tid=3, entry_tid=1], [object_number=24, tag_tid=3, entry_tid=0], [object_number=27, tag_tid=2, entry_tid=27223], [object_number=26, tag_tid=2, entry_tid=27222]]], state=connected] 2024-04-06T09:33:45.414 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=20, tag_tid=2, entry_tid=22904], [object_number=23, tag_tid=2, entry_tid=22903], [object_number=22, tag_tid=2, entry_tid=22902], [object_number=21, tag_tid=2, entry_tid=22901]]], state=connected] 2024-04-06T09:33:45.417 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:33:45.420 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=20, tag_tid=2, entry_tid=22904' 2024-04-06T09:33:45.420 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=25, tag_tid=3, entry_tid=1' -a 'object_number=25, tag_tid=3, entry_tid=1' = 'object_number=20, tag_tid=2, entry_tid=22904' 2024-04-06T09:33:45.420 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=20, tag_tid=2, entry_tid=22904' '!=' 'object_number=20, tag_tid=2, entry_tid=22792' 2024-04-06T09:33:45.420 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:33:45.420 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=20, tag_tid=2, entry_tid=22904' = 'object_number=20, tag_tid=2, entry_tid=22792' 2024-04-06T09:33:45.420 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=20, tag_tid=2, entry_tid=22904' 2024-04-06T09:33:45.420 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=25, tag_tid=3, entry_tid=1' 2024-04-06T09:33:45.420 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:33:45.420 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:33:45.421 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=3 2024-04-06T09:33:45.421 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=20, tag_tid=2, entry_tid=22904' 2024-04-06T09:33:45.422 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:33:45.422 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:33:45.422 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=2 2024-04-06T09:33:45.423 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=25, tag_tid=3, entry_tid=1' 2024-04-06T09:33:45.423 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:33:45.423 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:33:45.424 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:33:45.425 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=20, tag_tid=2, entry_tid=22904' 2024-04-06T09:33:45.425 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:33:45.425 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:33:45.426 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=22904 2024-04-06T09:33:45.426 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 3 = 2 -a 1 -le 22904 2024-04-06T09:33:45.426 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:33:45.426 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:33:45.426 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:33:45.629 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:33:45.629 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:33:45.629 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:33:45.629 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:33:45.629 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:33:45.629 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:33:45.629 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:33:45.629 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:33:45.629 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:33:45.629 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:33:45.630 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:33:45.630 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:33:45.630 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:33:45.630 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:33:45.630 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:33:45.632 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:33:45.632 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:33:45.632 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:33:45.632 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:33:45.632 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:33:45.632 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:33:45.632 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:33:45.633 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:33:45.633 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:33:45.633 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:33:45.633 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:33:45.633 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:33:45.633 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:33:45.633 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:33:45.633 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:33:45.633 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:33:45.633 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:33:45.633 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:33:45.634 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:33:45.634 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:33:45.634 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:33:45.634 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:33:45.634 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:33:45.634 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:33:45.634 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:33:45.634 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:33:45.634 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:33:45.634 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:33:45.634 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:33:45.634 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:33:45.635 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:33:45.635 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:33:45.635 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:33:45.635 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:33:45.658 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:33:45.658 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:33:45.658 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:33:45.659 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:33:45.787 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:33:45.788 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:33:45.788 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:33:45.788 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:33:45.788 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:33:45.788 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:33:45.788 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:33:45.789 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:33:45.789 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:33:45.789 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:33:45.789 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:33:45.789 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:33:45.789 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:33:45.790 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:33:45.790 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:33:45.791 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:33:45.840 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 5 2024-04-06T09:33:45.840 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 6 2024-04-06T09:33:45.840 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:33:45.840 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=25, tag_tid=3, entry_tid=1], [object_number=24, tag_tid=3, entry_tid=0], [object_number=27, tag_tid=2, entry_tid=27223], [object_number=26, tag_tid=2, entry_tid=27222]]], state=connected] 2024-04-06T09:33:45.840 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=20, tag_tid=2, entry_tid=23024], [object_number=23, tag_tid=2, entry_tid=23023], [object_number=22, tag_tid=2, entry_tid=23022], [object_number=21, tag_tid=2, entry_tid=23021]]], state=connected] 2024-04-06T09:33:45.844 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:33:45.845 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=25, tag_tid=3, entry_tid=1' 2024-04-06T09:33:45.846 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:33:45.846 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:33:45.846 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:33:45.846 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:33:45.847 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:33:45.847 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:33:45.847 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:33:45.847 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:33:45.847 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:33:45.847 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:33:45.847 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:33:45.847 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:33:45.848 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:33:45.848 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:33:45.848 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:33:45.893 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 5 2024-04-06T09:33:45.893 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 6 2024-04-06T09:33:45.894 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:33:45.894 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=25, tag_tid=3, entry_tid=1], [object_number=24, tag_tid=3, entry_tid=0], [object_number=27, tag_tid=2, entry_tid=27223], [object_number=26, tag_tid=2, entry_tid=27222]]], state=connected] 2024-04-06T09:33:45.894 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=20, tag_tid=2, entry_tid=23024], [object_number=23, tag_tid=2, entry_tid=23023], [object_number=22, tag_tid=2, entry_tid=23022], [object_number=21, tag_tid=2, entry_tid=23021]]], state=connected] 2024-04-06T09:33:45.897 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:33:45.900 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=20, tag_tid=2, entry_tid=23024' 2024-04-06T09:33:45.900 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=25, tag_tid=3, entry_tid=1' -a 'object_number=25, tag_tid=3, entry_tid=1' = 'object_number=20, tag_tid=2, entry_tid=23024' 2024-04-06T09:33:45.900 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=20, tag_tid=2, entry_tid=23024' '!=' 'object_number=20, tag_tid=2, entry_tid=22904' 2024-04-06T09:33:45.900 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:33:45.900 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=20, tag_tid=2, entry_tid=23024' = 'object_number=20, tag_tid=2, entry_tid=22904' 2024-04-06T09:33:45.900 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=20, tag_tid=2, entry_tid=23024' 2024-04-06T09:33:45.900 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=25, tag_tid=3, entry_tid=1' 2024-04-06T09:33:45.900 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:33:45.900 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:33:45.901 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=3 2024-04-06T09:33:45.901 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=20, tag_tid=2, entry_tid=23024' 2024-04-06T09:33:45.902 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:33:45.902 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:33:45.902 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=2 2024-04-06T09:33:45.903 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=25, tag_tid=3, entry_tid=1' 2024-04-06T09:33:45.903 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:33:45.903 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:33:45.904 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:33:45.905 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=20, tag_tid=2, entry_tid=23024' 2024-04-06T09:33:45.905 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:33:45.905 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:33:45.906 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=23024 2024-04-06T09:33:45.906 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 3 = 2 -a 1 -le 23024 2024-04-06T09:33:45.906 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:33:45.906 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:33:45.906 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:33:46.108 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:33:46.109 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:33:46.109 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:33:46.109 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:33:46.109 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:33:46.109 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:33:46.109 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:33:46.109 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:33:46.109 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:33:46.109 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:33:46.109 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:33:46.109 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:33:46.109 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:33:46.109 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:33:46.109 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:33:46.110 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:33:46.110 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:33:46.113 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:33:46.113 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:33:46.113 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:33:46.113 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:33:46.113 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:33:46.113 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:33:46.113 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:33:46.113 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:33:46.113 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:33:46.114 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:33:46.114 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:33:46.114 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:33:46.114 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:33:46.114 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:33:46.114 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:33:46.114 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:33:46.114 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:33:46.114 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:33:46.114 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:33:46.114 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:33:46.114 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:33:46.114 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:33:46.114 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:33:46.115 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:33:46.115 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:33:46.115 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:33:46.115 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:33:46.115 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:33:46.115 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:33:46.115 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:33:46.115 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:33:46.115 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:33:46.139 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:33:46.139 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:33:46.139 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:33:46.139 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:33:46.270 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:33:46.271 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:33:46.271 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:33:46.271 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:33:46.271 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:33:46.271 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:33:46.271 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:33:46.272 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:33:46.272 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:33:46.272 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:33:46.272 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:33:46.272 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:33:46.272 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:33:46.274 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:33:46.275 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:33:46.275 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:33:46.325 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 5 2024-04-06T09:33:46.325 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 6 2024-04-06T09:33:46.325 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:33:46.326 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=25, tag_tid=3, entry_tid=1], [object_number=24, tag_tid=3, entry_tid=0], [object_number=27, tag_tid=2, entry_tid=27223], [object_number=26, tag_tid=2, entry_tid=27222]]], state=connected] 2024-04-06T09:33:46.326 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=20, tag_tid=2, entry_tid=23144], [object_number=23, tag_tid=2, entry_tid=23143], [object_number=22, tag_tid=2, entry_tid=23142], [object_number=21, tag_tid=2, entry_tid=23141]]], state=connected] 2024-04-06T09:33:46.329 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:33:46.331 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=25, tag_tid=3, entry_tid=1' 2024-04-06T09:33:46.331 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:33:46.332 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:33:46.332 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:33:46.332 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:33:46.332 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:33:46.332 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:33:46.332 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:33:46.332 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:33:46.332 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:33:46.332 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:33:46.332 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:33:46.332 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:33:46.333 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:33:46.333 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:33:46.333 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:33:46.378 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 5 2024-04-06T09:33:46.378 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 6 2024-04-06T09:33:46.378 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:33:46.378 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=25, tag_tid=3, entry_tid=1], [object_number=24, tag_tid=3, entry_tid=0], [object_number=27, tag_tid=2, entry_tid=27223], [object_number=26, tag_tid=2, entry_tid=27222]]], state=connected] 2024-04-06T09:33:46.378 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=20, tag_tid=2, entry_tid=23144], [object_number=23, tag_tid=2, entry_tid=23143], [object_number=22, tag_tid=2, entry_tid=23142], [object_number=21, tag_tid=2, entry_tid=23141]]], state=connected] 2024-04-06T09:33:46.382 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:33:46.383 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=20, tag_tid=2, entry_tid=23144' 2024-04-06T09:33:46.383 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=25, tag_tid=3, entry_tid=1' -a 'object_number=25, tag_tid=3, entry_tid=1' = 'object_number=20, tag_tid=2, entry_tid=23144' 2024-04-06T09:33:46.383 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=20, tag_tid=2, entry_tid=23144' '!=' 'object_number=20, tag_tid=2, entry_tid=23024' 2024-04-06T09:33:46.384 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:33:46.384 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=20, tag_tid=2, entry_tid=23144' = 'object_number=20, tag_tid=2, entry_tid=23024' 2024-04-06T09:33:46.384 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=20, tag_tid=2, entry_tid=23144' 2024-04-06T09:33:46.384 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=25, tag_tid=3, entry_tid=1' 2024-04-06T09:33:46.384 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:33:46.384 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:33:46.385 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=3 2024-04-06T09:33:46.386 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=20, tag_tid=2, entry_tid=23144' 2024-04-06T09:33:46.386 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:33:46.386 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:33:46.387 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=2 2024-04-06T09:33:46.387 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=25, tag_tid=3, entry_tid=1' 2024-04-06T09:33:46.387 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:33:46.387 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:33:46.388 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:33:46.389 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=20, tag_tid=2, entry_tid=23144' 2024-04-06T09:33:46.389 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:33:46.389 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:33:46.390 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=23144 2024-04-06T09:33:46.390 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 3 = 2 -a 1 -le 23144 2024-04-06T09:33:46.391 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:33:46.391 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:33:46.391 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:33:46.592 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:33:46.593 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:33:46.593 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:33:46.593 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:33:46.593 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:33:46.593 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:33:46.593 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:33:46.593 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:33:46.593 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:33:46.593 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:33:46.593 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:33:46.593 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:33:46.593 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:33:46.594 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:33:46.594 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:33:46.594 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:33:46.594 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:33:46.597 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:33:46.597 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:33:46.598 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:33:46.598 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:33:46.598 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:33:46.598 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:33:46.598 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:33:46.598 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:33:46.598 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:33:46.598 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:33:46.598 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:33:46.598 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:33:46.598 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:33:46.598 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:33:46.598 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:33:46.598 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:33:46.599 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:33:46.599 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:33:46.599 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:33:46.599 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:33:46.599 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:33:46.599 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:33:46.599 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:33:46.599 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:33:46.599 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:33:46.599 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:33:46.599 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:33:46.599 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:33:46.599 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:33:46.599 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:33:46.600 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:33:46.600 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:33:46.625 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:33:46.625 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:33:46.625 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:33:46.625 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:33:46.765 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:33:46.767 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:33:46.767 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:33:46.767 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:33:46.767 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:33:46.767 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:33:46.767 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:33:46.767 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:33:46.767 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:33:46.767 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:33:46.768 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:33:46.768 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:33:46.768 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:33:46.769 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:33:46.769 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:33:46.769 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:33:46.823 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 5 2024-04-06T09:33:46.823 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 6 2024-04-06T09:33:46.823 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:33:46.824 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=25, tag_tid=3, entry_tid=1], [object_number=24, tag_tid=3, entry_tid=0], [object_number=27, tag_tid=2, entry_tid=27223], [object_number=26, tag_tid=2, entry_tid=27222]]], state=connected] 2024-04-06T09:33:46.824 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=20, tag_tid=2, entry_tid=23256], [object_number=23, tag_tid=2, entry_tid=23255], [object_number=22, tag_tid=2, entry_tid=23254], [object_number=21, tag_tid=2, entry_tid=23253]]], state=connected] 2024-04-06T09:33:46.827 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:33:46.828 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=25, tag_tid=3, entry_tid=1' 2024-04-06T09:33:46.829 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:33:46.830 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:33:46.830 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:33:46.830 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:33:46.830 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:33:46.830 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:33:46.830 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:33:46.830 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:33:46.830 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:33:46.830 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:33:46.830 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:33:46.830 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:33:46.831 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:33:46.831 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:33:46.831 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:33:46.877 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 5 2024-04-06T09:33:46.877 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 6 2024-04-06T09:33:46.877 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:33:46.877 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=25, tag_tid=3, entry_tid=1], [object_number=24, tag_tid=3, entry_tid=0], [object_number=27, tag_tid=2, entry_tid=27223], [object_number=26, tag_tid=2, entry_tid=27222]]], state=connected] 2024-04-06T09:33:46.877 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=20, tag_tid=2, entry_tid=23256], [object_number=23, tag_tid=2, entry_tid=23255], [object_number=22, tag_tid=2, entry_tid=23254], [object_number=21, tag_tid=2, entry_tid=23253]]], state=connected] 2024-04-06T09:33:46.881 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:33:46.883 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=20, tag_tid=2, entry_tid=23256' 2024-04-06T09:33:46.883 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=25, tag_tid=3, entry_tid=1' -a 'object_number=25, tag_tid=3, entry_tid=1' = 'object_number=20, tag_tid=2, entry_tid=23256' 2024-04-06T09:33:46.883 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=20, tag_tid=2, entry_tid=23256' '!=' 'object_number=20, tag_tid=2, entry_tid=23144' 2024-04-06T09:33:46.883 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:33:46.883 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=20, tag_tid=2, entry_tid=23256' = 'object_number=20, tag_tid=2, entry_tid=23144' 2024-04-06T09:33:46.883 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=20, tag_tid=2, entry_tid=23256' 2024-04-06T09:33:46.884 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=25, tag_tid=3, entry_tid=1' 2024-04-06T09:33:46.884 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:33:46.884 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:33:46.884 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=3 2024-04-06T09:33:46.885 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=20, tag_tid=2, entry_tid=23256' 2024-04-06T09:33:46.885 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:33:46.885 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:33:46.886 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=2 2024-04-06T09:33:46.887 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=25, tag_tid=3, entry_tid=1' 2024-04-06T09:33:46.887 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:33:46.887 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:33:46.888 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:33:46.889 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=20, tag_tid=2, entry_tid=23256' 2024-04-06T09:33:46.889 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:33:46.889 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:33:46.890 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=23256 2024-04-06T09:33:46.890 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 3 = 2 -a 1 -le 23256 2024-04-06T09:33:46.890 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:33:46.890 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:33:46.890 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:33:47.092 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:33:47.092 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:33:47.092 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:33:47.092 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:33:47.092 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:33:47.092 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:33:47.092 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:33:47.092 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:33:47.093 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:33:47.093 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:33:47.093 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:33:47.093 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:33:47.093 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:33:47.093 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:33:47.093 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:33:47.093 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:33:47.094 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:33:47.096 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:33:47.096 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:33:47.096 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:33:47.096 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:33:47.096 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:33:47.096 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:33:47.096 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:33:47.097 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:33:47.097 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:33:47.097 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:33:47.097 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:33:47.097 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:33:47.097 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:33:47.097 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:33:47.097 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:33:47.097 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:33:47.098 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:33:47.098 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:33:47.098 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:33:47.098 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:33:47.098 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:33:47.098 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:33:47.099 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:33:47.099 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:33:47.099 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:33:47.099 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:33:47.099 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:33:47.099 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:33:47.099 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:33:47.099 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:33:47.099 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:33:47.099 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:33:47.124 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:33:47.124 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:33:47.124 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:33:47.124 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:33:47.250 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:33:47.251 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:33:47.251 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:33:47.251 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:33:47.251 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:33:47.251 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:33:47.251 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:33:47.251 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:33:47.251 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:33:47.251 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:33:47.252 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:33:47.252 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:33:47.252 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:33:47.255 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:33:47.255 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:33:47.255 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:33:47.306 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 5 2024-04-06T09:33:47.306 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 6 2024-04-06T09:33:47.306 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:33:47.306 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=25, tag_tid=3, entry_tid=1], [object_number=24, tag_tid=3, entry_tid=0], [object_number=27, tag_tid=2, entry_tid=27223], [object_number=26, tag_tid=2, entry_tid=27222]]], state=connected] 2024-04-06T09:33:47.306 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=21, tag_tid=2, entry_tid=23353], [object_number=20, tag_tid=2, entry_tid=23352], [object_number=23, tag_tid=2, entry_tid=23351], [object_number=22, tag_tid=2, entry_tid=23350]]], state=connected] 2024-04-06T09:33:47.310 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:33:47.311 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=25, tag_tid=3, entry_tid=1' 2024-04-06T09:33:47.312 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:33:47.312 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:33:47.312 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:33:47.312 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:33:47.312 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:33:47.312 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:33:47.313 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:33:47.313 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:33:47.313 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:33:47.313 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:33:47.313 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:33:47.313 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:33:47.314 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:33:47.314 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:33:47.314 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:33:47.359 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 5 2024-04-06T09:33:47.359 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 6 2024-04-06T09:33:47.359 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:33:47.359 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=25, tag_tid=3, entry_tid=1], [object_number=24, tag_tid=3, entry_tid=0], [object_number=27, tag_tid=2, entry_tid=27223], [object_number=26, tag_tid=2, entry_tid=27222]]], state=connected] 2024-04-06T09:33:47.359 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=21, tag_tid=2, entry_tid=23353], [object_number=20, tag_tid=2, entry_tid=23352], [object_number=23, tag_tid=2, entry_tid=23351], [object_number=22, tag_tid=2, entry_tid=23350]]], state=connected] 2024-04-06T09:33:47.363 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:33:47.365 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=21, tag_tid=2, entry_tid=23353' 2024-04-06T09:33:47.365 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=25, tag_tid=3, entry_tid=1' -a 'object_number=25, tag_tid=3, entry_tid=1' = 'object_number=21, tag_tid=2, entry_tid=23353' 2024-04-06T09:33:47.365 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=21, tag_tid=2, entry_tid=23353' '!=' 'object_number=20, tag_tid=2, entry_tid=23256' 2024-04-06T09:33:47.365 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:33:47.365 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=21, tag_tid=2, entry_tid=23353' = 'object_number=20, tag_tid=2, entry_tid=23256' 2024-04-06T09:33:47.365 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=21, tag_tid=2, entry_tid=23353' 2024-04-06T09:33:47.365 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=25, tag_tid=3, entry_tid=1' 2024-04-06T09:33:47.365 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:33:47.365 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:33:47.366 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=3 2024-04-06T09:33:47.367 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=21, tag_tid=2, entry_tid=23353' 2024-04-06T09:33:47.367 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:33:47.367 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:33:47.368 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=2 2024-04-06T09:33:47.368 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=25, tag_tid=3, entry_tid=1' 2024-04-06T09:33:47.368 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:33:47.369 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:33:47.369 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:33:47.370 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=21, tag_tid=2, entry_tid=23353' 2024-04-06T09:33:47.370 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:33:47.370 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:33:47.371 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=23353 2024-04-06T09:33:47.371 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 3 = 2 -a 1 -le 23353 2024-04-06T09:33:47.371 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:33:47.371 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:33:47.372 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:33:47.573 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:33:47.574 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:33:47.574 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:33:47.574 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:33:47.574 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:33:47.574 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:33:47.574 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:33:47.574 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:33:47.574 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:33:47.574 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:33:47.574 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:33:47.574 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:33:47.574 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:33:47.574 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:33:47.574 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:33:47.575 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:33:47.575 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:33:47.577 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:33:47.577 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:33:47.577 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:33:47.577 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:33:47.578 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:33:47.578 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:33:47.578 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:33:47.578 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:33:47.578 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:33:47.578 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:33:47.578 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:33:47.578 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:33:47.578 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:33:47.578 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:33:47.578 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:33:47.578 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:33:47.578 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:33:47.578 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:33:47.579 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:33:47.579 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:33:47.579 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:33:47.579 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:33:47.579 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:33:47.579 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:33:47.579 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:33:47.579 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:33:47.579 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:33:47.579 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:33:47.579 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:33:47.579 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:33:47.579 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:33:47.580 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:33:47.603 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:33:47.604 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:33:47.604 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:33:47.604 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:33:47.734 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:33:47.736 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:33:47.736 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:33:47.736 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:33:47.736 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:33:47.736 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:33:47.736 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:33:47.736 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:33:47.736 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:33:47.736 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:33:47.736 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:33:47.737 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:33:47.737 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:33:47.738 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:33:47.738 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:33:47.738 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:33:47.794 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 5 2024-04-06T09:33:47.794 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 6 2024-04-06T09:33:47.794 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:33:47.794 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=25, tag_tid=3, entry_tid=1], [object_number=24, tag_tid=3, entry_tid=0], [object_number=27, tag_tid=2, entry_tid=27223], [object_number=26, tag_tid=2, entry_tid=27222]]], state=connected] 2024-04-06T09:33:47.794 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=20, tag_tid=2, entry_tid=23448], [object_number=23, tag_tid=2, entry_tid=23447], [object_number=22, tag_tid=2, entry_tid=23446], [object_number=21, tag_tid=2, entry_tid=23445]]], state=connected] 2024-04-06T09:33:47.797 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:33:47.798 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=25, tag_tid=3, entry_tid=1' 2024-04-06T09:33:47.800 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:33:47.800 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:33:47.800 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:33:47.800 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:33:47.800 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:33:47.800 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:33:47.800 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:33:47.800 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:33:47.800 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:33:47.801 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:33:47.801 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:33:47.801 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:33:47.801 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:33:47.801 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:33:47.801 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:33:47.846 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 5 2024-04-06T09:33:47.846 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 6 2024-04-06T09:33:47.846 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:33:47.846 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=25, tag_tid=3, entry_tid=1], [object_number=24, tag_tid=3, entry_tid=0], [object_number=27, tag_tid=2, entry_tid=27223], [object_number=26, tag_tid=2, entry_tid=27222]]], state=connected] 2024-04-06T09:33:47.846 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=20, tag_tid=2, entry_tid=23448], [object_number=23, tag_tid=2, entry_tid=23447], [object_number=22, tag_tid=2, entry_tid=23446], [object_number=21, tag_tid=2, entry_tid=23445]]], state=connected] 2024-04-06T09:33:47.850 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:33:47.852 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=20, tag_tid=2, entry_tid=23448' 2024-04-06T09:33:47.852 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=25, tag_tid=3, entry_tid=1' -a 'object_number=25, tag_tid=3, entry_tid=1' = 'object_number=20, tag_tid=2, entry_tid=23448' 2024-04-06T09:33:47.852 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=20, tag_tid=2, entry_tid=23448' '!=' 'object_number=21, tag_tid=2, entry_tid=23353' 2024-04-06T09:33:47.852 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:33:47.852 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=20, tag_tid=2, entry_tid=23448' = 'object_number=21, tag_tid=2, entry_tid=23353' 2024-04-06T09:33:47.852 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=20, tag_tid=2, entry_tid=23448' 2024-04-06T09:33:47.852 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=25, tag_tid=3, entry_tid=1' 2024-04-06T09:33:47.852 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:33:47.852 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:33:47.853 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=3 2024-04-06T09:33:47.853 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=20, tag_tid=2, entry_tid=23448' 2024-04-06T09:33:47.853 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:33:47.854 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:33:47.854 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=2 2024-04-06T09:33:47.855 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=25, tag_tid=3, entry_tid=1' 2024-04-06T09:33:47.855 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:33:47.855 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:33:47.856 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:33:47.857 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=20, tag_tid=2, entry_tid=23448' 2024-04-06T09:33:47.857 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:33:47.857 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:33:47.858 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=23448 2024-04-06T09:33:47.858 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 3 = 2 -a 1 -le 23448 2024-04-06T09:33:47.858 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:33:47.858 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:33:47.858 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:33:48.060 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:33:48.061 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:33:48.061 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:33:48.061 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:33:48.061 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:33:48.061 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:33:48.061 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:33:48.061 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:33:48.061 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:33:48.061 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:33:48.061 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:33:48.061 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:33:48.061 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:33:48.062 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:33:48.062 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:33:48.062 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:33:48.065 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:33:48.065 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:33:48.065 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:33:48.065 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:33:48.065 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:33:48.065 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:33:48.065 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:33:48.065 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:33:48.065 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:33:48.065 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:33:48.065 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:33:48.065 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:33:48.065 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:33:48.065 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:33:48.066 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:33:48.066 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:33:48.066 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:33:48.066 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:33:48.066 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:33:48.066 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:33:48.066 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:33:48.066 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:33:48.066 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:33:48.066 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:33:48.066 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:33:48.066 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:33:48.066 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:33:48.067 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:33:48.067 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:33:48.067 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:33:48.067 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:33:48.067 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:33:48.067 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:33:48.091 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:33:48.091 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:33:48.091 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:33:48.091 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:33:48.219 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:33:48.221 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:33:48.221 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:33:48.221 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:33:48.221 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:33:48.221 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:33:48.221 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:33:48.221 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:33:48.221 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:33:48.221 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:33:48.221 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:33:48.222 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:33:48.222 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:33:48.223 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:33:48.223 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:33:48.224 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:33:48.274 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 5 2024-04-06T09:33:48.274 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 6 2024-04-06T09:33:48.274 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:33:48.274 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=25, tag_tid=3, entry_tid=1], [object_number=24, tag_tid=3, entry_tid=0], [object_number=27, tag_tid=2, entry_tid=27223], [object_number=26, tag_tid=2, entry_tid=27222]]], state=connected] 2024-04-06T09:33:48.274 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=20, tag_tid=2, entry_tid=23540], [object_number=23, tag_tid=2, entry_tid=23539], [object_number=22, tag_tid=2, entry_tid=23538], [object_number=21, tag_tid=2, entry_tid=23537]]], state=connected] 2024-04-06T09:33:48.277 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:33:48.280 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=25, tag_tid=3, entry_tid=1' 2024-04-06T09:33:48.280 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:33:48.280 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:33:48.280 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:33:48.280 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:33:48.280 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:33:48.280 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:33:48.280 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:33:48.280 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:33:48.280 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:33:48.280 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:33:48.280 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:33:48.280 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:33:48.281 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:33:48.281 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:33:48.281 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:33:48.326 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 5 2024-04-06T09:33:48.327 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 6 2024-04-06T09:33:48.327 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:33:48.327 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=25, tag_tid=3, entry_tid=1], [object_number=24, tag_tid=3, entry_tid=0], [object_number=27, tag_tid=2, entry_tid=27223], [object_number=26, tag_tid=2, entry_tid=27222]]], state=connected] 2024-04-06T09:33:48.327 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=20, tag_tid=2, entry_tid=23540], [object_number=23, tag_tid=2, entry_tid=23539], [object_number=22, tag_tid=2, entry_tid=23538], [object_number=21, tag_tid=2, entry_tid=23537]]], state=connected] 2024-04-06T09:33:48.330 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:33:48.332 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=20, tag_tid=2, entry_tid=23540' 2024-04-06T09:33:48.332 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=25, tag_tid=3, entry_tid=1' -a 'object_number=25, tag_tid=3, entry_tid=1' = 'object_number=20, tag_tid=2, entry_tid=23540' 2024-04-06T09:33:48.332 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=20, tag_tid=2, entry_tid=23540' '!=' 'object_number=20, tag_tid=2, entry_tid=23448' 2024-04-06T09:33:48.332 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:33:48.332 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=20, tag_tid=2, entry_tid=23540' = 'object_number=20, tag_tid=2, entry_tid=23448' 2024-04-06T09:33:48.333 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=20, tag_tid=2, entry_tid=23540' 2024-04-06T09:33:48.333 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=25, tag_tid=3, entry_tid=1' 2024-04-06T09:33:48.333 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:33:48.333 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:33:48.333 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=3 2024-04-06T09:33:48.334 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=20, tag_tid=2, entry_tid=23540' 2024-04-06T09:33:48.334 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:33:48.334 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:33:48.335 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=2 2024-04-06T09:33:48.336 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=25, tag_tid=3, entry_tid=1' 2024-04-06T09:33:48.336 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:33:48.336 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:33:48.337 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:33:48.337 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=20, tag_tid=2, entry_tid=23540' 2024-04-06T09:33:48.337 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:33:48.337 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:33:48.338 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=23540 2024-04-06T09:33:48.339 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 3 = 2 -a 1 -le 23540 2024-04-06T09:33:48.339 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:33:48.339 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:33:48.339 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:33:48.541 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:33:48.541 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:33:48.541 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:33:48.541 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:33:48.541 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:33:48.541 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:33:48.541 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:33:48.541 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:33:48.541 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:33:48.541 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:33:48.541 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:33:48.542 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:33:48.542 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:33:48.542 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:33:48.542 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:33:48.542 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:33:48.542 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:33:48.546 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:33:48.546 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:33:48.546 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:33:48.546 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:33:48.546 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:33:48.546 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:33:48.547 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:33:48.547 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:33:48.547 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:33:48.547 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:33:48.547 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:33:48.547 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:33:48.547 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:33:48.547 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:33:48.547 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:33:48.547 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:33:48.547 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:33:48.547 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:33:48.547 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:33:48.547 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:33:48.548 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:33:48.548 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:33:48.548 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:33:48.548 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:33:48.548 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:33:48.548 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:33:48.548 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:33:48.548 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:33:48.548 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:33:48.548 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:33:48.548 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:33:48.548 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:33:48.571 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:33:48.572 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:33:48.572 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:33:48.572 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:33:48.698 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:33:48.699 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:33:48.699 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:33:48.699 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:33:48.699 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:33:48.699 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:33:48.699 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:33:48.699 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:33:48.699 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:33:48.699 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:33:48.700 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:33:48.700 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:33:48.700 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:33:48.701 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:33:48.701 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:33:48.702 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:33:48.752 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 5 2024-04-06T09:33:48.752 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 6 2024-04-06T09:33:48.752 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:33:48.752 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=25, tag_tid=3, entry_tid=1], [object_number=24, tag_tid=3, entry_tid=0], [object_number=27, tag_tid=2, entry_tid=27223], [object_number=26, tag_tid=2, entry_tid=27222]]], state=connected] 2024-04-06T09:33:48.752 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=21, tag_tid=2, entry_tid=23641], [object_number=20, tag_tid=2, entry_tid=23640], [object_number=23, tag_tid=2, entry_tid=23639], [object_number=22, tag_tid=2, entry_tid=23638]]], state=connected] 2024-04-06T09:33:48.755 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:33:48.757 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=25, tag_tid=3, entry_tid=1' 2024-04-06T09:33:48.758 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:33:48.758 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:33:48.758 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:33:48.758 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:33:48.758 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:33:48.758 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:33:48.759 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:33:48.759 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:33:48.759 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:33:48.759 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:33:48.759 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:33:48.759 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:33:48.759 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:33:48.759 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:33:48.759 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:33:48.805 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 5 2024-04-06T09:33:48.805 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 6 2024-04-06T09:33:48.805 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:33:48.805 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=25, tag_tid=3, entry_tid=1], [object_number=24, tag_tid=3, entry_tid=0], [object_number=27, tag_tid=2, entry_tid=27223], [object_number=26, tag_tid=2, entry_tid=27222]]], state=connected] 2024-04-06T09:33:48.805 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=21, tag_tid=2, entry_tid=23641], [object_number=20, tag_tid=2, entry_tid=23640], [object_number=23, tag_tid=2, entry_tid=23639], [object_number=22, tag_tid=2, entry_tid=23638]]], state=connected] 2024-04-06T09:33:48.808 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:33:48.810 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=21, tag_tid=2, entry_tid=23641' 2024-04-06T09:33:48.811 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=25, tag_tid=3, entry_tid=1' -a 'object_number=25, tag_tid=3, entry_tid=1' = 'object_number=21, tag_tid=2, entry_tid=23641' 2024-04-06T09:33:48.811 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=21, tag_tid=2, entry_tid=23641' '!=' 'object_number=20, tag_tid=2, entry_tid=23540' 2024-04-06T09:33:48.811 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:33:48.811 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=21, tag_tid=2, entry_tid=23641' = 'object_number=20, tag_tid=2, entry_tid=23540' 2024-04-06T09:33:48.811 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=21, tag_tid=2, entry_tid=23641' 2024-04-06T09:33:48.811 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=25, tag_tid=3, entry_tid=1' 2024-04-06T09:33:48.811 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:33:48.811 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:33:48.812 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=3 2024-04-06T09:33:48.812 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=21, tag_tid=2, entry_tid=23641' 2024-04-06T09:33:48.812 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:33:48.812 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:33:48.813 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=2 2024-04-06T09:33:48.814 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=25, tag_tid=3, entry_tid=1' 2024-04-06T09:33:48.814 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:33:48.814 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:33:48.815 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:33:48.815 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=21, tag_tid=2, entry_tid=23641' 2024-04-06T09:33:48.815 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:33:48.815 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:33:48.817 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=23641 2024-04-06T09:33:48.817 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 3 = 2 -a 1 -le 23641 2024-04-06T09:33:48.817 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:33:48.817 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:33:48.817 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:33:49.018 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:33:49.019 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:33:49.019 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:33:49.019 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:33:49.019 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:33:49.019 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:33:49.019 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:33:49.019 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:33:49.019 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:33:49.019 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:33:49.019 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:33:49.019 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:33:49.019 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:33:49.019 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:33:49.019 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:33:49.020 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:33:49.020 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:33:49.024 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:33:49.024 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:33:49.024 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:33:49.024 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:33:49.024 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:33:49.024 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:33:49.025 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:33:49.025 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:33:49.025 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:33:49.025 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:33:49.025 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:33:49.025 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:33:49.025 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:33:49.025 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:33:49.025 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:33:49.025 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:33:49.025 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:33:49.025 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:33:49.025 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:33:49.025 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:33:49.025 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:33:49.025 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:33:49.026 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:33:49.026 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:33:49.026 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:33:49.026 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:33:49.026 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:33:49.026 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:33:49.026 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:33:49.026 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:33:49.026 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:33:49.026 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:33:49.050 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:33:49.050 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:33:49.050 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:33:49.050 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:33:49.178 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:33:49.179 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:33:49.180 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:33:49.180 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:33:49.180 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:33:49.180 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:33:49.180 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:33:49.180 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:33:49.180 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:33:49.180 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:33:49.181 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:33:49.181 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:33:49.181 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:33:49.183 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:33:49.184 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:33:49.184 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:33:49.234 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 5 2024-04-06T09:33:49.234 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 6 2024-04-06T09:33:49.234 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:33:49.234 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=25, tag_tid=3, entry_tid=1], [object_number=24, tag_tid=3, entry_tid=0], [object_number=27, tag_tid=2, entry_tid=27223], [object_number=26, tag_tid=2, entry_tid=27222]]], state=connected] 2024-04-06T09:33:49.234 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=21, tag_tid=2, entry_tid=23733], [object_number=20, tag_tid=2, entry_tid=23732], [object_number=23, tag_tid=2, entry_tid=23731], [object_number=22, tag_tid=2, entry_tid=23730]]], state=connected] 2024-04-06T09:33:49.238 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:33:49.239 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=25, tag_tid=3, entry_tid=1' 2024-04-06T09:33:49.240 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:33:49.240 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:33:49.240 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:33:49.240 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:33:49.241 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:33:49.241 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:33:49.241 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:33:49.241 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:33:49.241 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:33:49.241 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:33:49.241 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:33:49.241 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:33:49.242 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:33:49.242 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:33:49.242 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:33:49.286 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 5 2024-04-06T09:33:49.287 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 6 2024-04-06T09:33:49.287 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:33:49.287 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=25, tag_tid=3, entry_tid=1], [object_number=24, tag_tid=3, entry_tid=0], [object_number=27, tag_tid=2, entry_tid=27223], [object_number=26, tag_tid=2, entry_tid=27222]]], state=connected] 2024-04-06T09:33:49.287 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=21, tag_tid=2, entry_tid=23733], [object_number=20, tag_tid=2, entry_tid=23732], [object_number=23, tag_tid=2, entry_tid=23731], [object_number=22, tag_tid=2, entry_tid=23730]]], state=connected] 2024-04-06T09:33:49.290 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:33:49.292 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=21, tag_tid=2, entry_tid=23733' 2024-04-06T09:33:49.292 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=25, tag_tid=3, entry_tid=1' -a 'object_number=25, tag_tid=3, entry_tid=1' = 'object_number=21, tag_tid=2, entry_tid=23733' 2024-04-06T09:33:49.292 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=21, tag_tid=2, entry_tid=23733' '!=' 'object_number=21, tag_tid=2, entry_tid=23641' 2024-04-06T09:33:49.292 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:33:49.292 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=21, tag_tid=2, entry_tid=23733' = 'object_number=21, tag_tid=2, entry_tid=23641' 2024-04-06T09:33:49.292 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=21, tag_tid=2, entry_tid=23733' 2024-04-06T09:33:49.292 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=25, tag_tid=3, entry_tid=1' 2024-04-06T09:33:49.293 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:33:49.293 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:33:49.293 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=3 2024-04-06T09:33:49.294 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=21, tag_tid=2, entry_tid=23733' 2024-04-06T09:33:49.294 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:33:49.294 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:33:49.295 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=2 2024-04-06T09:33:49.295 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=25, tag_tid=3, entry_tid=1' 2024-04-06T09:33:49.295 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:33:49.296 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:33:49.296 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:33:49.297 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=21, tag_tid=2, entry_tid=23733' 2024-04-06T09:33:49.297 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:33:49.297 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:33:49.298 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=23733 2024-04-06T09:33:49.298 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 3 = 2 -a 1 -le 23733 2024-04-06T09:33:49.298 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:33:49.299 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:33:49.299 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:33:49.376 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.0 with skip (0.38659210326432336) > weight (0.25) 2024-04-06T09:33:49.376 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.1 with skip (0.5029116196339577) > weight (0.25) 2024-04-06T09:33:49.376 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.2 with skip (0.8729587326033085) > weight (0.25) 2024-04-06T09:33:49.376 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.3 with skip (0.6088471665887478) > weight (0.25) 2024-04-06T09:33:49.376 INFO:tasks.rbd_mirror_thrash:waiting for 77 secs before thrashing 2024-04-06T09:33:49.500 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:33:49.500 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:33:49.501 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:33:49.501 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:33:49.501 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:33:49.501 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:33:49.501 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:33:49.501 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:33:49.501 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:33:49.501 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:33:49.501 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:33:49.501 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:33:49.502 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:33:49.502 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:33:49.502 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:33:49.502 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:33:49.502 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:33:49.505 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:33:49.505 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:33:49.505 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:33:49.505 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:33:49.505 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:33:49.505 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:33:49.505 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:33:49.505 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:33:49.505 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:33:49.505 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:33:49.505 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:33:49.505 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:33:49.505 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:33:49.505 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:33:49.505 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:33:49.506 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:33:49.506 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:33:49.506 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:33:49.506 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:33:49.506 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:33:49.506 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:33:49.506 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:33:49.506 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:33:49.506 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:33:49.506 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:33:49.506 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:33:49.506 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:33:49.506 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:33:49.506 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:33:49.506 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:33:49.507 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:33:49.507 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:33:49.530 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:33:49.530 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:33:49.530 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:33:49.530 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:33:49.659 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:33:49.661 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:33:49.661 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:33:49.661 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:33:49.661 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:33:49.661 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:33:49.661 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:33:49.661 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:33:49.661 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:33:49.661 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:33:49.661 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:33:49.662 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:33:49.662 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:33:49.663 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:33:49.664 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:33:49.664 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:33:49.708 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 5 2024-04-06T09:33:49.708 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 6 2024-04-06T09:33:49.708 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:33:49.708 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=25, tag_tid=3, entry_tid=1], [object_number=24, tag_tid=3, entry_tid=0], [object_number=27, tag_tid=2, entry_tid=27223], [object_number=26, tag_tid=2, entry_tid=27222]]], state=connected] 2024-04-06T09:33:49.708 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=20, tag_tid=2, entry_tid=23832], [object_number=23, tag_tid=2, entry_tid=23831], [object_number=22, tag_tid=2, entry_tid=23830], [object_number=21, tag_tid=2, entry_tid=23829]]], state=connected] 2024-04-06T09:33:49.711 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:33:49.714 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=25, tag_tid=3, entry_tid=1' 2024-04-06T09:33:49.714 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:33:49.714 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:33:49.714 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:33:49.714 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:33:49.714 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:33:49.714 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:33:49.714 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:33:49.714 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:33:49.714 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:33:49.714 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:33:49.714 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:33:49.715 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:33:49.715 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:33:49.715 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:33:49.715 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:33:49.759 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 5 2024-04-06T09:33:49.759 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 6 2024-04-06T09:33:49.760 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:33:49.760 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=25, tag_tid=3, entry_tid=1], [object_number=24, tag_tid=3, entry_tid=0], [object_number=27, tag_tid=2, entry_tid=27223], [object_number=26, tag_tid=2, entry_tid=27222]]], state=connected] 2024-04-06T09:33:49.760 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=20, tag_tid=2, entry_tid=23832], [object_number=23, tag_tid=2, entry_tid=23831], [object_number=22, tag_tid=2, entry_tid=23830], [object_number=21, tag_tid=2, entry_tid=23829]]], state=connected] 2024-04-06T09:33:49.763 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:33:49.765 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=20, tag_tid=2, entry_tid=23832' 2024-04-06T09:33:49.765 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=25, tag_tid=3, entry_tid=1' -a 'object_number=25, tag_tid=3, entry_tid=1' = 'object_number=20, tag_tid=2, entry_tid=23832' 2024-04-06T09:33:49.765 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=20, tag_tid=2, entry_tid=23832' '!=' 'object_number=21, tag_tid=2, entry_tid=23733' 2024-04-06T09:33:49.765 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:33:49.766 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=20, tag_tid=2, entry_tid=23832' = 'object_number=21, tag_tid=2, entry_tid=23733' 2024-04-06T09:33:49.766 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=20, tag_tid=2, entry_tid=23832' 2024-04-06T09:33:49.766 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=25, tag_tid=3, entry_tid=1' 2024-04-06T09:33:49.766 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:33:49.766 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:33:49.766 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=3 2024-04-06T09:33:49.767 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=20, tag_tid=2, entry_tid=23832' 2024-04-06T09:33:49.767 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:33:49.767 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:33:49.768 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=2 2024-04-06T09:33:49.769 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=25, tag_tid=3, entry_tid=1' 2024-04-06T09:33:49.769 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:33:49.769 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:33:49.770 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:33:49.770 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=20, tag_tid=2, entry_tid=23832' 2024-04-06T09:33:49.770 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:33:49.771 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:33:49.772 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=23832 2024-04-06T09:33:49.772 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 3 = 2 -a 1 -le 23832 2024-04-06T09:33:49.772 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:33:49.772 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:33:49.772 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:33:49.973 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:33:49.974 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:33:49.974 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:33:49.974 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:33:49.974 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:33:49.974 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:33:49.974 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:33:49.974 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:33:49.974 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:33:49.974 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:33:49.974 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:33:49.974 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:33:49.974 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:33:49.975 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:33:49.975 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:33:49.975 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:33:49.975 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:33:49.978 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:33:49.979 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:33:49.979 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:33:49.979 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:33:49.979 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:33:49.979 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:33:49.979 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:33:49.979 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:33:49.979 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:33:49.979 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:33:49.979 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:33:49.979 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:33:49.979 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:33:49.979 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:33:49.980 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:33:49.980 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:33:49.980 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:33:49.980 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:33:49.980 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:33:49.980 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:33:49.980 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:33:49.980 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:33:49.980 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:33:49.980 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:33:49.980 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:33:49.980 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:33:49.980 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:33:49.980 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:33:49.980 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:33:49.980 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:33:49.981 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:33:49.981 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:33:50.003 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:33:50.004 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:33:50.004 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:33:50.004 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:33:50.130 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:33:50.130 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:33:50.131 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:33:50.131 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:33:50.131 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:33:50.131 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:33:50.131 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:33:50.131 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:33:50.131 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:33:50.131 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:33:50.131 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:33:50.132 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:33:50.132 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:33:50.132 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:33:50.132 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:33:50.132 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:33:50.178 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 5 2024-04-06T09:33:50.178 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 6 2024-04-06T09:33:50.178 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:33:50.178 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=25, tag_tid=3, entry_tid=1], [object_number=24, tag_tid=3, entry_tid=0], [object_number=27, tag_tid=2, entry_tid=27223], [object_number=26, tag_tid=2, entry_tid=27222]]], state=connected] 2024-04-06T09:33:50.178 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=22, tag_tid=2, entry_tid=23942], [object_number=21, tag_tid=2, entry_tid=23941], [object_number=20, tag_tid=2, entry_tid=23940], [object_number=23, tag_tid=2, entry_tid=23939]]], state=connected] 2024-04-06T09:33:50.182 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:33:50.183 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=25, tag_tid=3, entry_tid=1' 2024-04-06T09:33:50.184 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:33:50.184 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:33:50.184 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:33:50.184 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:33:50.184 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:33:50.184 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:33:50.184 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:33:50.184 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:33:50.184 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:33:50.184 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:33:50.185 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:33:50.185 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:33:50.185 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:33:50.185 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:33:50.186 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:33:50.230 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 5 2024-04-06T09:33:50.230 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 6 2024-04-06T09:33:50.230 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:33:50.230 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=25, tag_tid=3, entry_tid=1], [object_number=24, tag_tid=3, entry_tid=0], [object_number=27, tag_tid=2, entry_tid=27223], [object_number=26, tag_tid=2, entry_tid=27222]]], state=connected] 2024-04-06T09:33:50.230 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=22, tag_tid=2, entry_tid=23942], [object_number=21, tag_tid=2, entry_tid=23941], [object_number=20, tag_tid=2, entry_tid=23940], [object_number=23, tag_tid=2, entry_tid=23939]]], state=connected] 2024-04-06T09:33:50.233 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:33:50.235 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=22, tag_tid=2, entry_tid=23942' 2024-04-06T09:33:50.235 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=25, tag_tid=3, entry_tid=1' -a 'object_number=25, tag_tid=3, entry_tid=1' = 'object_number=22, tag_tid=2, entry_tid=23942' 2024-04-06T09:33:50.235 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=22, tag_tid=2, entry_tid=23942' '!=' 'object_number=20, tag_tid=2, entry_tid=23832' 2024-04-06T09:33:50.235 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:33:50.236 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=22, tag_tid=2, entry_tid=23942' = 'object_number=20, tag_tid=2, entry_tid=23832' 2024-04-06T09:33:50.236 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=22, tag_tid=2, entry_tid=23942' 2024-04-06T09:33:50.236 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=25, tag_tid=3, entry_tid=1' 2024-04-06T09:33:50.236 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:33:50.236 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:33:50.236 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=3 2024-04-06T09:33:50.237 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=22, tag_tid=2, entry_tid=23942' 2024-04-06T09:33:50.237 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:33:50.237 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:33:50.238 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=2 2024-04-06T09:33:50.239 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=25, tag_tid=3, entry_tid=1' 2024-04-06T09:33:50.239 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:33:50.239 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:33:50.240 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:33:50.240 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=22, tag_tid=2, entry_tid=23942' 2024-04-06T09:33:50.241 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:33:50.241 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:33:50.242 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=23942 2024-04-06T09:33:50.242 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 3 = 2 -a 1 -le 23942 2024-04-06T09:33:50.242 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:33:50.242 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:33:50.242 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:33:50.443 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:33:50.443 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:33:50.443 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:33:50.443 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:33:50.443 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:33:50.444 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:33:50.444 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:33:50.444 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:33:50.444 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:33:50.444 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:33:50.444 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:33:50.444 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:33:50.444 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:33:50.444 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:33:50.444 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:33:50.445 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:33:50.445 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:33:50.447 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:33:50.448 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:33:50.448 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:33:50.448 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:33:50.448 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:33:50.448 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:33:50.448 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:33:50.448 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:33:50.448 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:33:50.448 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:33:50.448 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:33:50.448 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:33:50.448 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:33:50.449 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:33:50.449 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:33:50.449 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:33:50.450 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:33:50.450 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:33:50.450 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:33:50.451 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:33:50.451 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:33:50.451 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:33:50.451 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:33:50.451 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:33:50.451 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:33:50.451 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:33:50.451 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:33:50.451 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:33:50.451 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:33:50.451 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:33:50.451 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:33:50.451 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:33:50.476 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:33:50.476 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:33:50.477 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:33:50.477 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:33:50.612 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:33:50.613 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:33:50.613 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:33:50.614 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:33:50.614 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:33:50.614 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:33:50.614 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:33:50.614 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:33:50.614 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:33:50.614 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:33:50.614 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:33:50.615 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:33:50.615 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:33:50.616 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:33:50.616 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:33:50.616 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:33:50.662 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 5 2024-04-06T09:33:50.663 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 6 2024-04-06T09:33:50.663 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:33:50.663 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=25, tag_tid=3, entry_tid=1], [object_number=24, tag_tid=3, entry_tid=0], [object_number=27, tag_tid=2, entry_tid=27223], [object_number=26, tag_tid=2, entry_tid=27222]]], state=connected] 2024-04-06T09:33:50.663 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=20, tag_tid=2, entry_tid=24064], [object_number=23, tag_tid=2, entry_tid=24063], [object_number=22, tag_tid=2, entry_tid=24062], [object_number=21, tag_tid=2, entry_tid=24061]]], state=connected] 2024-04-06T09:33:50.666 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:33:50.667 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=25, tag_tid=3, entry_tid=1' 2024-04-06T09:33:50.668 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:33:50.669 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:33:50.669 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:33:50.669 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:33:50.669 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:33:50.669 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:33:50.669 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:33:50.669 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:33:50.669 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:33:50.669 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:33:50.669 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:33:50.669 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:33:50.670 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:33:50.670 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:33:50.670 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:33:50.715 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 5 2024-04-06T09:33:50.715 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 6 2024-04-06T09:33:50.715 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:33:50.715 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=25, tag_tid=3, entry_tid=1], [object_number=24, tag_tid=3, entry_tid=0], [object_number=27, tag_tid=2, entry_tid=27223], [object_number=26, tag_tid=2, entry_tid=27222]]], state=connected] 2024-04-06T09:33:50.715 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=20, tag_tid=2, entry_tid=24064], [object_number=23, tag_tid=2, entry_tid=24063], [object_number=22, tag_tid=2, entry_tid=24062], [object_number=21, tag_tid=2, entry_tid=24061]]], state=connected] 2024-04-06T09:33:50.718 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:33:50.721 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=20, tag_tid=2, entry_tid=24064' 2024-04-06T09:33:50.721 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=25, tag_tid=3, entry_tid=1' -a 'object_number=25, tag_tid=3, entry_tid=1' = 'object_number=20, tag_tid=2, entry_tid=24064' 2024-04-06T09:33:50.721 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=20, tag_tid=2, entry_tid=24064' '!=' 'object_number=22, tag_tid=2, entry_tid=23942' 2024-04-06T09:33:50.721 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:33:50.721 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=20, tag_tid=2, entry_tid=24064' = 'object_number=22, tag_tid=2, entry_tid=23942' 2024-04-06T09:33:50.721 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=20, tag_tid=2, entry_tid=24064' 2024-04-06T09:33:50.721 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=25, tag_tid=3, entry_tid=1' 2024-04-06T09:33:50.721 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:33:50.721 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:33:50.722 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=3 2024-04-06T09:33:50.722 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=20, tag_tid=2, entry_tid=24064' 2024-04-06T09:33:50.723 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:33:50.723 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:33:50.723 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=2 2024-04-06T09:33:50.724 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=25, tag_tid=3, entry_tid=1' 2024-04-06T09:33:50.724 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:33:50.724 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:33:50.725 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:33:50.726 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=20, tag_tid=2, entry_tid=24064' 2024-04-06T09:33:50.726 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:33:50.726 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:33:50.727 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=24064 2024-04-06T09:33:50.727 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 3 = 2 -a 1 -le 24064 2024-04-06T09:33:50.727 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:33:50.727 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:33:50.727 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:33:50.928 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:33:50.929 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:33:50.929 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:33:50.929 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:33:50.929 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:33:50.929 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:33:50.929 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:33:50.929 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:33:50.929 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:33:50.929 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:33:50.929 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:33:50.929 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:33:50.929 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:33:50.929 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:33:50.930 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:33:50.930 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:33:50.930 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:33:50.933 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:33:50.933 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:33:50.933 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:33:50.933 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:33:50.933 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:33:50.933 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:33:50.933 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:33:50.934 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:33:50.934 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:33:50.934 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:33:50.934 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:33:50.934 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:33:50.934 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:33:50.934 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:33:50.934 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:33:50.934 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:33:50.934 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:33:50.934 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:33:50.934 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:33:50.934 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:33:50.934 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:33:50.934 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:33:50.934 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:33:50.935 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:33:50.935 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:33:50.935 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:33:50.935 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:33:50.935 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:33:50.935 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:33:50.935 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:33:50.935 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:33:50.935 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:33:50.958 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:33:50.958 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:33:50.959 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:33:50.959 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:33:51.081 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:33:51.083 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:33:51.083 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:33:51.083 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:33:51.083 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:33:51.083 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:33:51.083 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:33:51.084 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:33:51.084 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:33:51.084 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:33:51.084 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:33:51.084 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:33:51.084 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:33:51.085 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:33:51.085 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:33:51.086 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:33:51.129 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 5 2024-04-06T09:33:51.129 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 6 2024-04-06T09:33:51.130 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:33:51.130 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=25, tag_tid=3, entry_tid=1], [object_number=24, tag_tid=3, entry_tid=0], [object_number=27, tag_tid=2, entry_tid=27223], [object_number=26, tag_tid=2, entry_tid=27222]]], state=connected] 2024-04-06T09:33:51.130 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=21, tag_tid=2, entry_tid=24161], [object_number=20, tag_tid=2, entry_tid=24160], [object_number=23, tag_tid=2, entry_tid=24159], [object_number=22, tag_tid=2, entry_tid=24158]]], state=connected] 2024-04-06T09:33:51.133 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:33:51.134 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=25, tag_tid=3, entry_tid=1' 2024-04-06T09:33:51.135 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:33:51.136 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:33:51.136 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:33:51.136 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:33:51.136 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:33:51.136 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:33:51.136 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:33:51.136 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:33:51.136 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:33:51.136 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:33:51.136 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:33:51.136 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:33:51.137 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:33:51.137 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:33:51.137 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:33:51.181 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 5 2024-04-06T09:33:51.181 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 6 2024-04-06T09:33:51.181 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:33:51.181 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=25, tag_tid=3, entry_tid=1], [object_number=24, tag_tid=3, entry_tid=0], [object_number=27, tag_tid=2, entry_tid=27223], [object_number=26, tag_tid=2, entry_tid=27222]]], state=connected] 2024-04-06T09:33:51.181 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=21, tag_tid=2, entry_tid=24161], [object_number=20, tag_tid=2, entry_tid=24160], [object_number=23, tag_tid=2, entry_tid=24159], [object_number=22, tag_tid=2, entry_tid=24158]]], state=connected] 2024-04-06T09:33:51.184 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:33:51.187 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=21, tag_tid=2, entry_tid=24161' 2024-04-06T09:33:51.187 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=25, tag_tid=3, entry_tid=1' -a 'object_number=25, tag_tid=3, entry_tid=1' = 'object_number=21, tag_tid=2, entry_tid=24161' 2024-04-06T09:33:51.187 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=21, tag_tid=2, entry_tid=24161' '!=' 'object_number=20, tag_tid=2, entry_tid=24064' 2024-04-06T09:33:51.187 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:33:51.187 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=21, tag_tid=2, entry_tid=24161' = 'object_number=20, tag_tid=2, entry_tid=24064' 2024-04-06T09:33:51.187 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=21, tag_tid=2, entry_tid=24161' 2024-04-06T09:33:51.187 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=25, tag_tid=3, entry_tid=1' 2024-04-06T09:33:51.187 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:33:51.187 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:33:51.188 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=3 2024-04-06T09:33:51.188 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=21, tag_tid=2, entry_tid=24161' 2024-04-06T09:33:51.189 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:33:51.189 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:33:51.189 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=2 2024-04-06T09:33:51.190 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=25, tag_tid=3, entry_tid=1' 2024-04-06T09:33:51.190 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:33:51.190 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:33:51.191 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:33:51.192 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=21, tag_tid=2, entry_tid=24161' 2024-04-06T09:33:51.192 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:33:51.192 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:33:51.193 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=24161 2024-04-06T09:33:51.193 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 3 = 2 -a 1 -le 24161 2024-04-06T09:33:51.193 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:33:51.193 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:33:51.193 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:33:51.395 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:33:51.395 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:33:51.395 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:33:51.396 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:33:51.396 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:33:51.396 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:33:51.396 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:33:51.396 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:33:51.396 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:33:51.396 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:33:51.396 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:33:51.396 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:33:51.396 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:33:51.396 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:33:51.396 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:33:51.397 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:33:51.397 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:33:51.400 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:33:51.400 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:33:51.400 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:33:51.400 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:33:51.401 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:33:51.401 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:33:51.401 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:33:51.401 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:33:51.401 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:33:51.401 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:33:51.401 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:33:51.401 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:33:51.401 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:33:51.401 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:33:51.401 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:33:51.401 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:33:51.401 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:33:51.401 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:33:51.401 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:33:51.401 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:33:51.402 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:33:51.402 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:33:51.402 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:33:51.402 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:33:51.402 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:33:51.402 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:33:51.402 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:33:51.402 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:33:51.402 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:33:51.402 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:33:51.402 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:33:51.402 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:33:51.426 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:33:51.426 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:33:51.426 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:33:51.426 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:33:51.562 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:33:51.563 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:33:51.563 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:33:51.564 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:33:51.564 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:33:51.564 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:33:51.564 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:33:51.564 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:33:51.564 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:33:51.564 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:33:51.564 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:33:51.565 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:33:51.565 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:33:51.566 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:33:51.566 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:33:51.566 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:33:51.613 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 5 2024-04-06T09:33:51.614 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 6 2024-04-06T09:33:51.614 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:33:51.614 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=25, tag_tid=3, entry_tid=1], [object_number=24, tag_tid=3, entry_tid=0], [object_number=27, tag_tid=2, entry_tid=27223], [object_number=26, tag_tid=2, entry_tid=27222]]], state=connected] 2024-04-06T09:33:51.614 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=23, tag_tid=2, entry_tid=24247], [object_number=22, tag_tid=2, entry_tid=24246], [object_number=21, tag_tid=2, entry_tid=24245], [object_number=20, tag_tid=2, entry_tid=24244]]], state=connected] 2024-04-06T09:33:51.618 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:33:51.618 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=25, tag_tid=3, entry_tid=1' 2024-04-06T09:33:51.619 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:33:51.619 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:33:51.619 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:33:51.619 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:33:51.620 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:33:51.620 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:33:51.620 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:33:51.620 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:33:51.620 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:33:51.620 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:33:51.620 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:33:51.620 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:33:51.621 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:33:51.621 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:33:51.621 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:33:51.665 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 5 2024-04-06T09:33:51.665 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 6 2024-04-06T09:33:51.665 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:33:51.665 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=25, tag_tid=3, entry_tid=1], [object_number=24, tag_tid=3, entry_tid=0], [object_number=27, tag_tid=2, entry_tid=27223], [object_number=26, tag_tid=2, entry_tid=27222]]], state=connected] 2024-04-06T09:33:51.665 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=23, tag_tid=2, entry_tid=24247], [object_number=22, tag_tid=2, entry_tid=24246], [object_number=21, tag_tid=2, entry_tid=24245], [object_number=20, tag_tid=2, entry_tid=24244]]], state=connected] 2024-04-06T09:33:51.668 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:33:51.671 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=23, tag_tid=2, entry_tid=24247' 2024-04-06T09:33:51.671 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=25, tag_tid=3, entry_tid=1' -a 'object_number=25, tag_tid=3, entry_tid=1' = 'object_number=23, tag_tid=2, entry_tid=24247' 2024-04-06T09:33:51.671 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=23, tag_tid=2, entry_tid=24247' '!=' 'object_number=21, tag_tid=2, entry_tid=24161' 2024-04-06T09:33:51.671 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:33:51.671 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=23, tag_tid=2, entry_tid=24247' = 'object_number=21, tag_tid=2, entry_tid=24161' 2024-04-06T09:33:51.671 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=23, tag_tid=2, entry_tid=24247' 2024-04-06T09:33:51.671 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=25, tag_tid=3, entry_tid=1' 2024-04-06T09:33:51.671 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:33:51.671 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:33:51.672 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=3 2024-04-06T09:33:51.672 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=23, tag_tid=2, entry_tid=24247' 2024-04-06T09:33:51.672 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:33:51.673 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:33:51.673 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=2 2024-04-06T09:33:51.674 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=25, tag_tid=3, entry_tid=1' 2024-04-06T09:33:51.674 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:33:51.674 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:33:51.675 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:33:51.676 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=23, tag_tid=2, entry_tid=24247' 2024-04-06T09:33:51.676 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:33:51.676 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:33:51.677 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=24247 2024-04-06T09:33:51.677 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 3 = 2 -a 1 -le 24247 2024-04-06T09:33:51.677 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:33:51.677 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:33:51.677 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:33:51.879 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:33:51.880 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:33:51.880 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:33:51.880 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:33:51.880 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:33:51.880 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:33:51.880 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:33:51.880 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:33:51.880 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:33:51.880 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:33:51.880 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:33:51.880 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:33:51.880 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:33:51.880 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:33:51.880 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:33:51.881 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:33:51.881 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:33:51.882 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:33:51.882 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:33:51.883 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:33:51.883 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:33:51.883 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:33:51.883 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:33:51.883 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:33:51.883 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:33:51.883 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:33:51.883 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:33:51.883 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:33:51.883 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:33:51.883 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:33:51.883 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:33:51.883 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:33:51.883 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:33:51.884 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:33:51.885 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:33:51.885 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:33:51.885 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:33:51.885 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:33:51.885 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:33:51.885 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:33:51.885 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:33:51.885 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:33:51.885 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:33:51.885 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:33:51.885 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:33:51.885 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:33:51.885 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:33:51.885 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:33:51.885 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:33:51.909 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:33:51.910 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:33:51.910 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:33:51.910 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:33:52.042 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:33:52.043 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:33:52.044 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:33:52.044 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:33:52.044 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:33:52.044 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:33:52.044 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:33:52.044 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:33:52.044 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:33:52.044 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:33:52.044 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:33:52.044 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:33:52.044 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:33:52.046 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:33:52.047 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:33:52.047 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:33:52.097 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 5 2024-04-06T09:33:52.097 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 6 2024-04-06T09:33:52.097 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:33:52.097 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=25, tag_tid=3, entry_tid=1], [object_number=24, tag_tid=3, entry_tid=0], [object_number=27, tag_tid=2, entry_tid=27223], [object_number=26, tag_tid=2, entry_tid=27222]]], state=connected] 2024-04-06T09:33:52.097 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=23, tag_tid=2, entry_tid=24339], [object_number=22, tag_tid=2, entry_tid=24338], [object_number=21, tag_tid=2, entry_tid=24337], [object_number=20, tag_tid=2, entry_tid=24336]]], state=connected] 2024-04-06T09:33:52.192 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:33:52.195 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=25, tag_tid=3, entry_tid=1' 2024-04-06T09:33:52.195 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:33:52.196 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:33:52.196 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:33:52.196 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:33:52.196 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:33:52.196 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:33:52.196 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:33:52.196 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:33:52.196 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:33:52.197 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:33:52.197 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:33:52.197 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:33:52.200 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:33:52.200 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:33:52.200 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:33:52.253 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 5 2024-04-06T09:33:52.253 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 6 2024-04-06T09:33:52.253 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:33:52.253 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=25, tag_tid=3, entry_tid=1], [object_number=24, tag_tid=3, entry_tid=0], [object_number=27, tag_tid=2, entry_tid=27223], [object_number=26, tag_tid=2, entry_tid=27222]]], state=connected] 2024-04-06T09:33:52.253 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=23, tag_tid=2, entry_tid=24339], [object_number=22, tag_tid=2, entry_tid=24338], [object_number=21, tag_tid=2, entry_tid=24337], [object_number=20, tag_tid=2, entry_tid=24336]]], state=connected] 2024-04-06T09:33:52.257 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:33:52.259 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=23, tag_tid=2, entry_tid=24339' 2024-04-06T09:33:52.259 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=25, tag_tid=3, entry_tid=1' -a 'object_number=25, tag_tid=3, entry_tid=1' = 'object_number=23, tag_tid=2, entry_tid=24339' 2024-04-06T09:33:52.259 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=23, tag_tid=2, entry_tid=24339' '!=' 'object_number=23, tag_tid=2, entry_tid=24247' 2024-04-06T09:33:52.259 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:33:52.259 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=23, tag_tid=2, entry_tid=24339' = 'object_number=23, tag_tid=2, entry_tid=24247' 2024-04-06T09:33:52.259 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=23, tag_tid=2, entry_tid=24339' 2024-04-06T09:33:52.260 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=25, tag_tid=3, entry_tid=1' 2024-04-06T09:33:52.260 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:33:52.260 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:33:52.260 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=3 2024-04-06T09:33:52.261 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=23, tag_tid=2, entry_tid=24339' 2024-04-06T09:33:52.261 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:33:52.261 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:33:52.262 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=2 2024-04-06T09:33:52.263 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=25, tag_tid=3, entry_tid=1' 2024-04-06T09:33:52.263 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:33:52.263 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:33:52.264 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:33:52.265 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=23, tag_tid=2, entry_tid=24339' 2024-04-06T09:33:52.265 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:33:52.265 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:33:52.266 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=24339 2024-04-06T09:33:52.266 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 3 = 2 -a 1 -le 24339 2024-04-06T09:33:52.266 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:33:52.266 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:33:52.266 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:33:52.468 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:33:52.469 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:33:52.469 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:33:52.469 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:33:52.469 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:33:52.469 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:33:52.469 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:33:52.469 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:33:52.469 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:33:52.469 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:33:52.469 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:33:52.469 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:33:52.469 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:33:52.469 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:33:52.469 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:33:52.470 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:33:52.470 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:33:52.473 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:33:52.474 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:33:52.474 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:33:52.474 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:33:52.474 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:33:52.474 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:33:52.474 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:33:52.474 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:33:52.474 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:33:52.474 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:33:52.474 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:33:52.474 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:33:52.474 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:33:52.475 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:33:52.475 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:33:52.475 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:33:52.475 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:33:52.475 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:33:52.475 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:33:52.475 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:33:52.475 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:33:52.475 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:33:52.475 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:33:52.475 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:33:52.475 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:33:52.475 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:33:52.475 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:33:52.475 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:33:52.475 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:33:52.476 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:33:52.476 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:33:52.476 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:33:52.499 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:33:52.499 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:33:52.499 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:33:52.499 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:33:52.627 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:33:52.629 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:33:52.629 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:33:52.629 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:33:52.629 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:33:52.629 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:33:52.629 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:33:52.629 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:33:52.629 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:33:52.629 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:33:52.630 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:33:52.630 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:33:52.630 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:33:52.632 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:33:52.632 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:33:52.632 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:33:52.680 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 5 2024-04-06T09:33:52.680 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 6 2024-04-06T09:33:52.680 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:33:52.680 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=25, tag_tid=3, entry_tid=1], [object_number=24, tag_tid=3, entry_tid=0], [object_number=27, tag_tid=2, entry_tid=27223], [object_number=26, tag_tid=2, entry_tid=27222]]], state=connected] 2024-04-06T09:33:52.680 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=23, tag_tid=2, entry_tid=24459], [object_number=22, tag_tid=2, entry_tid=24458], [object_number=21, tag_tid=2, entry_tid=24457], [object_number=20, tag_tid=2, entry_tid=24456]]], state=connected] 2024-04-06T09:33:52.684 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:33:52.686 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=25, tag_tid=3, entry_tid=1' 2024-04-06T09:33:52.686 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:33:52.687 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:33:52.687 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:33:52.687 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:33:52.687 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:33:52.687 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:33:52.687 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:33:52.687 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:33:52.687 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:33:52.687 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:33:52.687 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:33:52.687 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:33:52.688 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:33:52.688 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:33:52.688 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:33:52.732 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 5 2024-04-06T09:33:52.732 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 6 2024-04-06T09:33:52.732 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:33:52.732 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=25, tag_tid=3, entry_tid=1], [object_number=24, tag_tid=3, entry_tid=0], [object_number=27, tag_tid=2, entry_tid=27223], [object_number=26, tag_tid=2, entry_tid=27222]]], state=connected] 2024-04-06T09:33:52.732 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=23, tag_tid=2, entry_tid=24459], [object_number=22, tag_tid=2, entry_tid=24458], [object_number=21, tag_tid=2, entry_tid=24457], [object_number=20, tag_tid=2, entry_tid=24456]]], state=connected] 2024-04-06T09:33:52.736 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:33:52.738 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=23, tag_tid=2, entry_tid=24459' 2024-04-06T09:33:52.738 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=25, tag_tid=3, entry_tid=1' -a 'object_number=25, tag_tid=3, entry_tid=1' = 'object_number=23, tag_tid=2, entry_tid=24459' 2024-04-06T09:33:52.738 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=23, tag_tid=2, entry_tid=24459' '!=' 'object_number=23, tag_tid=2, entry_tid=24339' 2024-04-06T09:33:52.738 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:33:52.738 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=23, tag_tid=2, entry_tid=24459' = 'object_number=23, tag_tid=2, entry_tid=24339' 2024-04-06T09:33:52.738 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=23, tag_tid=2, entry_tid=24459' 2024-04-06T09:33:52.738 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=25, tag_tid=3, entry_tid=1' 2024-04-06T09:33:52.738 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:33:52.738 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:33:52.739 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=3 2024-04-06T09:33:52.739 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=23, tag_tid=2, entry_tid=24459' 2024-04-06T09:33:52.740 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:33:52.740 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:33:52.740 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=2 2024-04-06T09:33:52.741 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=25, tag_tid=3, entry_tid=1' 2024-04-06T09:33:52.741 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:33:52.741 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:33:52.742 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:33:52.743 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=23, tag_tid=2, entry_tid=24459' 2024-04-06T09:33:52.743 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:33:52.743 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:33:52.744 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=24459 2024-04-06T09:33:52.744 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 3 = 2 -a 1 -le 24459 2024-04-06T09:33:52.744 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:33:52.744 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:33:52.744 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:33:52.946 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:33:52.946 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:33:52.946 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:33:52.946 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:33:52.946 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:33:52.946 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:33:52.946 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:33:52.946 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:33:52.946 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:33:52.947 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:33:52.947 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:33:52.947 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:33:52.947 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:33:52.947 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:33:52.947 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:33:52.947 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:33:52.948 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:33:52.951 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:33:52.951 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:33:52.952 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:33:52.952 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:33:52.952 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:33:52.952 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:33:52.952 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:33:52.952 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:33:52.952 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:33:52.952 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:33:52.952 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:33:52.952 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:33:52.952 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:33:52.952 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:33:52.952 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:33:52.952 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:33:52.952 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:33:52.953 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:33:52.953 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:33:52.953 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:33:52.953 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:33:52.953 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:33:52.953 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:33:52.953 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:33:52.953 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:33:52.953 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:33:52.953 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:33:52.953 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:33:52.953 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:33:52.953 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:33:52.953 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:33:52.954 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:33:52.976 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:33:52.977 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:33:52.977 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:33:52.977 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:33:53.103 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:33:53.104 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:33:53.104 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:33:53.104 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:33:53.104 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:33:53.105 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:33:53.105 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:33:53.105 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:33:53.105 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:33:53.105 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:33:53.105 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:33:53.105 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:33:53.105 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:33:53.107 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:33:53.107 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:33:53.107 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:33:53.160 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 5 2024-04-06T09:33:53.160 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 6 2024-04-06T09:33:53.160 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:33:53.160 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=25, tag_tid=3, entry_tid=1], [object_number=24, tag_tid=3, entry_tid=0], [object_number=27, tag_tid=2, entry_tid=27223], [object_number=26, tag_tid=2, entry_tid=27222]]], state=connected] 2024-04-06T09:33:53.160 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=21, tag_tid=2, entry_tid=24557], [object_number=20, tag_tid=2, entry_tid=24556], [object_number=23, tag_tid=2, entry_tid=24555], [object_number=22, tag_tid=2, entry_tid=24554]]], state=connected] 2024-04-06T09:33:53.163 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:33:53.165 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=25, tag_tid=3, entry_tid=1' 2024-04-06T09:33:53.166 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:33:53.166 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:33:53.166 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:33:53.166 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:33:53.166 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:33:53.167 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:33:53.167 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:33:53.167 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:33:53.167 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:33:53.167 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:33:53.167 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:33:53.167 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:33:53.168 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:33:53.168 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:33:53.168 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:33:53.212 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 5 2024-04-06T09:33:53.212 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 6 2024-04-06T09:33:53.212 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:33:53.212 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=25, tag_tid=3, entry_tid=1], [object_number=24, tag_tid=3, entry_tid=0], [object_number=27, tag_tid=2, entry_tid=27223], [object_number=26, tag_tid=2, entry_tid=27222]]], state=connected] 2024-04-06T09:33:53.212 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=21, tag_tid=2, entry_tid=24557], [object_number=20, tag_tid=2, entry_tid=24556], [object_number=23, tag_tid=2, entry_tid=24555], [object_number=22, tag_tid=2, entry_tid=24554]]], state=connected] 2024-04-06T09:33:53.216 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:33:53.218 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=21, tag_tid=2, entry_tid=24557' 2024-04-06T09:33:53.218 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=25, tag_tid=3, entry_tid=1' -a 'object_number=25, tag_tid=3, entry_tid=1' = 'object_number=21, tag_tid=2, entry_tid=24557' 2024-04-06T09:33:53.218 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=21, tag_tid=2, entry_tid=24557' '!=' 'object_number=23, tag_tid=2, entry_tid=24459' 2024-04-06T09:33:53.218 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:33:53.219 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=21, tag_tid=2, entry_tid=24557' = 'object_number=23, tag_tid=2, entry_tid=24459' 2024-04-06T09:33:53.219 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=21, tag_tid=2, entry_tid=24557' 2024-04-06T09:33:53.219 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=25, tag_tid=3, entry_tid=1' 2024-04-06T09:33:53.219 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:33:53.219 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:33:53.219 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=3 2024-04-06T09:33:53.220 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=21, tag_tid=2, entry_tid=24557' 2024-04-06T09:33:53.220 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:33:53.220 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:33:53.221 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=2 2024-04-06T09:33:53.222 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=25, tag_tid=3, entry_tid=1' 2024-04-06T09:33:53.222 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:33:53.222 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:33:53.223 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:33:53.223 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=21, tag_tid=2, entry_tid=24557' 2024-04-06T09:33:53.223 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:33:53.224 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:33:53.225 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=24557 2024-04-06T09:33:53.225 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 3 = 2 -a 1 -le 24557 2024-04-06T09:33:53.225 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:33:53.225 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:33:53.225 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:33:53.427 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:33:53.427 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:33:53.427 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:33:53.427 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:33:53.427 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:33:53.427 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:33:53.427 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:33:53.428 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:33:53.428 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:33:53.428 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:33:53.428 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:33:53.428 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:33:53.428 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:33:53.428 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:33:53.428 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:33:53.428 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:33:53.429 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:33:53.430 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:33:53.430 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:33:53.430 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:33:53.430 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:33:53.430 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:33:53.430 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:33:53.430 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:33:53.430 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:33:53.430 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:33:53.431 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:33:53.431 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:33:53.431 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:33:53.431 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:33:53.431 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:33:53.431 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:33:53.431 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:33:53.433 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:33:53.433 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:33:53.433 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:33:53.433 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:33:53.433 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:33:53.433 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:33:53.433 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:33:53.433 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:33:53.433 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:33:53.433 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:33:53.433 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:33:53.433 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:33:53.433 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:33:53.433 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:33:53.434 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:33:53.434 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:33:53.457 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:33:53.457 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:33:53.457 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:33:53.457 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:33:53.583 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:33:53.584 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:33:53.585 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:33:53.585 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:33:53.585 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:33:53.585 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:33:53.585 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:33:53.585 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:33:53.585 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:33:53.585 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:33:53.586 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:33:53.586 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:33:53.586 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:33:53.587 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:33:53.587 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:33:53.588 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:33:53.632 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 5 2024-04-06T09:33:53.632 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 6 2024-04-06T09:33:53.632 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:33:53.632 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=25, tag_tid=3, entry_tid=1], [object_number=24, tag_tid=3, entry_tid=0], [object_number=27, tag_tid=2, entry_tid=27223], [object_number=26, tag_tid=2, entry_tid=27222]]], state=connected] 2024-04-06T09:33:53.632 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=20, tag_tid=2, entry_tid=24656], [object_number=23, tag_tid=2, entry_tid=24655], [object_number=22, tag_tid=2, entry_tid=24654], [object_number=21, tag_tid=2, entry_tid=24653]]], state=connected] 2024-04-06T09:33:53.635 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:33:53.638 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=25, tag_tid=3, entry_tid=1' 2024-04-06T09:33:53.638 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:33:53.638 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:33:53.638 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:33:53.638 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:33:53.638 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:33:53.638 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:33:53.638 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:33:53.638 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:33:53.638 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:33:53.638 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:33:53.638 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:33:53.639 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:33:53.639 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:33:53.639 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:33:53.640 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:33:53.683 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 5 2024-04-06T09:33:53.683 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 6 2024-04-06T09:33:53.683 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:33:53.683 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=25, tag_tid=3, entry_tid=1], [object_number=24, tag_tid=3, entry_tid=0], [object_number=27, tag_tid=2, entry_tid=27223], [object_number=26, tag_tid=2, entry_tid=27222]]], state=connected] 2024-04-06T09:33:53.683 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=20, tag_tid=2, entry_tid=24656], [object_number=23, tag_tid=2, entry_tid=24655], [object_number=22, tag_tid=2, entry_tid=24654], [object_number=21, tag_tid=2, entry_tid=24653]]], state=connected] 2024-04-06T09:33:53.686 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:33:53.688 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=20, tag_tid=2, entry_tid=24656' 2024-04-06T09:33:53.688 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=25, tag_tid=3, entry_tid=1' -a 'object_number=25, tag_tid=3, entry_tid=1' = 'object_number=20, tag_tid=2, entry_tid=24656' 2024-04-06T09:33:53.688 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=20, tag_tid=2, entry_tid=24656' '!=' 'object_number=21, tag_tid=2, entry_tid=24557' 2024-04-06T09:33:53.688 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:33:53.688 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=20, tag_tid=2, entry_tid=24656' = 'object_number=21, tag_tid=2, entry_tid=24557' 2024-04-06T09:33:53.688 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=20, tag_tid=2, entry_tid=24656' 2024-04-06T09:33:53.689 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=25, tag_tid=3, entry_tid=1' 2024-04-06T09:33:53.689 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:33:53.689 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:33:53.690 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=3 2024-04-06T09:33:53.690 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=20, tag_tid=2, entry_tid=24656' 2024-04-06T09:33:53.690 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:33:53.691 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:33:53.691 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=2 2024-04-06T09:33:53.692 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=25, tag_tid=3, entry_tid=1' 2024-04-06T09:33:53.692 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:33:53.692 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:33:53.693 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:33:53.694 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=20, tag_tid=2, entry_tid=24656' 2024-04-06T09:33:53.694 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:33:53.694 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:33:53.695 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=24656 2024-04-06T09:33:53.695 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 3 = 2 -a 1 -le 24656 2024-04-06T09:33:53.695 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:33:53.695 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:33:53.695 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:33:53.896 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:33:53.897 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:33:53.897 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:33:53.897 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:33:53.897 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:33:53.897 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:33:53.897 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:33:53.897 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:33:53.897 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:33:53.897 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:33:53.897 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:33:53.898 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:33:53.898 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:33:53.898 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:33:53.898 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:33:53.898 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:33:53.899 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:33:53.903 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:33:53.903 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:33:53.903 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:33:53.903 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:33:53.903 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:33:53.903 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:33:53.903 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:33:53.903 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:33:53.903 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:33:53.903 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:33:53.903 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:33:53.903 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:33:53.904 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:33:53.904 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:33:53.904 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:33:53.904 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:33:53.904 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:33:53.904 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:33:53.904 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:33:53.904 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:33:53.904 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:33:53.904 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:33:53.904 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:33:53.904 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:33:53.904 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:33:53.904 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:33:53.904 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:33:53.905 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:33:53.905 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:33:53.905 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:33:53.905 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:33:53.905 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:33:53.927 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:33:53.928 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:33:53.928 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:33:53.928 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:33:54.059 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:33:54.060 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:33:54.060 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:33:54.061 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:33:54.061 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:33:54.061 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:33:54.061 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:33:54.061 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:33:54.061 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:33:54.061 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:33:54.061 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:33:54.062 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:33:54.062 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:33:54.063 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:33:54.063 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:33:54.063 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:33:54.117 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 5 2024-04-06T09:33:54.117 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 6 2024-04-06T09:33:54.117 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:33:54.117 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=25, tag_tid=3, entry_tid=1], [object_number=24, tag_tid=3, entry_tid=0], [object_number=27, tag_tid=2, entry_tid=27223], [object_number=26, tag_tid=2, entry_tid=27222]]], state=connected] 2024-04-06T09:33:54.117 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=22, tag_tid=2, entry_tid=24758], [object_number=21, tag_tid=2, entry_tid=24757], [object_number=20, tag_tid=2, entry_tid=24756], [object_number=23, tag_tid=2, entry_tid=24755]]], state=connected] 2024-04-06T09:33:54.121 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:33:54.123 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=25, tag_tid=3, entry_tid=1' 2024-04-06T09:33:54.124 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:33:54.124 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:33:54.124 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:33:54.124 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:33:54.124 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:33:54.124 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:33:54.124 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:33:54.124 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:33:54.124 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:33:54.124 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:33:54.124 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:33:54.124 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:33:54.125 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:33:54.125 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:33:54.125 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:33:54.171 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 5 2024-04-06T09:33:54.171 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 6 2024-04-06T09:33:54.171 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:33:54.171 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=25, tag_tid=3, entry_tid=1], [object_number=24, tag_tid=3, entry_tid=0], [object_number=27, tag_tid=2, entry_tid=27223], [object_number=26, tag_tid=2, entry_tid=27222]]], state=connected] 2024-04-06T09:33:54.171 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=22, tag_tid=2, entry_tid=24758], [object_number=21, tag_tid=2, entry_tid=24757], [object_number=20, tag_tid=2, entry_tid=24756], [object_number=23, tag_tid=2, entry_tid=24755]]], state=connected] 2024-04-06T09:33:54.174 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:33:54.176 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=22, tag_tid=2, entry_tid=24758' 2024-04-06T09:33:54.176 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=25, tag_tid=3, entry_tid=1' -a 'object_number=25, tag_tid=3, entry_tid=1' = 'object_number=22, tag_tid=2, entry_tid=24758' 2024-04-06T09:33:54.177 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=22, tag_tid=2, entry_tid=24758' '!=' 'object_number=20, tag_tid=2, entry_tid=24656' 2024-04-06T09:33:54.177 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:33:54.177 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=22, tag_tid=2, entry_tid=24758' = 'object_number=20, tag_tid=2, entry_tid=24656' 2024-04-06T09:33:54.177 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=22, tag_tid=2, entry_tid=24758' 2024-04-06T09:33:54.177 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=25, tag_tid=3, entry_tid=1' 2024-04-06T09:33:54.177 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:33:54.177 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:33:54.177 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=3 2024-04-06T09:33:54.178 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=22, tag_tid=2, entry_tid=24758' 2024-04-06T09:33:54.178 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:33:54.178 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:33:54.180 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=2 2024-04-06T09:33:54.180 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=25, tag_tid=3, entry_tid=1' 2024-04-06T09:33:54.180 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:33:54.180 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:33:54.181 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:33:54.182 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=22, tag_tid=2, entry_tid=24758' 2024-04-06T09:33:54.182 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:33:54.182 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:33:54.183 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=24758 2024-04-06T09:33:54.183 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 3 = 2 -a 1 -le 24758 2024-04-06T09:33:54.183 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:33:54.183 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:33:54.183 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:33:54.374 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.0 with skip (0.3783169841851982) > weight (0.25) 2024-04-06T09:33:54.374 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.1 with skip (0.41014871180019563) > weight (0.25) 2024-04-06T09:33:54.374 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.2 with skip (0.748192657259852) > weight (0.25) 2024-04-06T09:33:54.374 INFO:tasks.rbd_mirror_thrash:kill cluster1.client.mirror.3 2024-04-06T09:33:54.374 INFO:tasks.rbd_mirror.cluster1.client.mirror.3:Sent signal 15 2024-04-06T09:33:54.374 INFO:tasks.rbd_mirror_thrash:waiting for 8 secs before reviving daemons 2024-04-06T09:33:54.375 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi192.stderr:2024-04-06T09:33:54.375+0000 7f65c9fdf640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.3 (PID: 27478) UID: 1000 2024-04-06T09:33:54.384 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:33:54.385 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:33:54.385 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:33:54.385 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:33:54.385 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:33:54.385 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:33:54.385 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:33:54.385 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:33:54.385 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:33:54.385 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:33:54.385 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:33:54.385 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:33:54.385 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:33:54.385 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:33:54.385 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:33:54.386 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:33:54.386 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:33:54.388 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:33:54.388 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:33:54.388 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:33:54.388 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:33:54.389 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:33:54.389 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:33:54.389 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:33:54.389 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:33:54.389 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:33:54.389 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:33:54.389 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:33:54.389 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:33:54.389 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:33:54.389 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:33:54.389 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:33:54.389 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:33:54.390 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:33:54.390 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:33:54.390 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:33:54.391 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:33:54.391 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:33:54.391 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:33:54.391 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:33:54.391 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:33:54.391 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:33:54.391 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:33:54.391 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:33:54.391 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:33:54.391 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:33:54.391 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:33:54.391 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:33:54.391 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:33:54.417 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:33:54.417 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:33:54.417 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:33:54.417 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:33:54.547 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:33:54.549 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:33:54.549 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:33:54.549 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:33:54.549 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:33:54.550 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:33:54.550 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:33:54.550 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:33:54.550 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:33:54.550 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:33:54.550 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:33:54.550 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:33:54.550 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:33:54.551 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:33:54.551 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:33:54.552 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:33:54.601 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 5 2024-04-06T09:33:54.602 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 6 2024-04-06T09:33:54.602 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:33:54.602 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=25, tag_tid=3, entry_tid=1], [object_number=24, tag_tid=3, entry_tid=0], [object_number=27, tag_tid=2, entry_tid=27223], [object_number=26, tag_tid=2, entry_tid=27222]]], state=connected] 2024-04-06T09:33:54.602 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=23, tag_tid=2, entry_tid=24863], [object_number=22, tag_tid=2, entry_tid=24862], [object_number=21, tag_tid=2, entry_tid=24861], [object_number=20, tag_tid=2, entry_tid=24860]]], state=connected] 2024-04-06T09:33:54.605 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:33:54.607 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=25, tag_tid=3, entry_tid=1' 2024-04-06T09:33:54.607 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:33:54.607 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:33:54.607 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:33:54.607 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:33:54.607 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:33:54.607 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:33:54.607 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:33:54.608 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:33:54.608 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:33:54.608 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:33:54.608 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:33:54.608 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:33:54.608 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:33:54.609 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:33:54.609 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:33:54.653 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 5 2024-04-06T09:33:54.653 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 6 2024-04-06T09:33:54.653 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:33:54.653 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=25, tag_tid=3, entry_tid=1], [object_number=24, tag_tid=3, entry_tid=0], [object_number=27, tag_tid=2, entry_tid=27223], [object_number=26, tag_tid=2, entry_tid=27222]]], state=connected] 2024-04-06T09:33:54.653 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=23, tag_tid=2, entry_tid=24863], [object_number=22, tag_tid=2, entry_tid=24862], [object_number=21, tag_tid=2, entry_tid=24861], [object_number=20, tag_tid=2, entry_tid=24860]]], state=connected] 2024-04-06T09:33:54.656 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:33:54.658 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=23, tag_tid=2, entry_tid=24863' 2024-04-06T09:33:54.658 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=25, tag_tid=3, entry_tid=1' -a 'object_number=25, tag_tid=3, entry_tid=1' = 'object_number=23, tag_tid=2, entry_tid=24863' 2024-04-06T09:33:54.658 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=23, tag_tid=2, entry_tid=24863' '!=' 'object_number=22, tag_tid=2, entry_tid=24758' 2024-04-06T09:33:54.658 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:33:54.659 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=23, tag_tid=2, entry_tid=24863' = 'object_number=22, tag_tid=2, entry_tid=24758' 2024-04-06T09:33:54.659 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=23, tag_tid=2, entry_tid=24863' 2024-04-06T09:33:54.659 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=25, tag_tid=3, entry_tid=1' 2024-04-06T09:33:54.659 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:33:54.659 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:33:54.659 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=3 2024-04-06T09:33:54.660 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=23, tag_tid=2, entry_tid=24863' 2024-04-06T09:33:54.660 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:33:54.660 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:33:54.661 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=2 2024-04-06T09:33:54.662 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=25, tag_tid=3, entry_tid=1' 2024-04-06T09:33:54.662 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:33:54.662 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:33:54.663 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:33:54.663 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=23, tag_tid=2, entry_tid=24863' 2024-04-06T09:33:54.664 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:33:54.664 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:33:54.665 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=24863 2024-04-06T09:33:54.665 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 3 = 2 -a 1 -le 24863 2024-04-06T09:33:54.665 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:33:54.665 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:33:54.665 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:33:54.867 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:33:54.867 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:33:54.867 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:33:54.867 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:33:54.867 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:33:54.867 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:33:54.867 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:33:54.867 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:33:54.867 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:33:54.867 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:33:54.867 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:33:54.867 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:33:54.868 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:33:54.868 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:33:54.868 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:33:54.868 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:33:54.868 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:33:54.872 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:33:54.872 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:33:54.872 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:33:54.872 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:33:54.872 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:33:54.872 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:33:54.872 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:33:54.872 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:33:54.872 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:33:54.872 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:33:54.872 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:33:54.872 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:33:54.872 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:33:54.873 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:33:54.873 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:33:54.873 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:33:54.873 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:33:54.873 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:33:54.873 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:33:54.873 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:33:54.873 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:33:54.873 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:33:54.873 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:33:54.873 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:33:54.873 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:33:54.874 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:33:54.874 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:33:54.874 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:33:54.874 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:33:54.874 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:33:54.874 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:33:54.874 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:33:54.898 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:33:54.898 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:33:54.898 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:33:54.898 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:33:55.025 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:33:55.027 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:33:55.027 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:33:55.027 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:33:55.027 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:33:55.027 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:33:55.027 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:33:55.027 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:33:55.027 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:33:55.027 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:33:55.027 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:33:55.028 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:33:55.028 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:33:55.029 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:33:55.029 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:33:55.029 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:33:55.083 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 5 2024-04-06T09:33:55.083 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 6 2024-04-06T09:33:55.083 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:33:55.083 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=25, tag_tid=3, entry_tid=1], [object_number=24, tag_tid=3, entry_tid=0], [object_number=27, tag_tid=2, entry_tid=27223], [object_number=26, tag_tid=2, entry_tid=27222]]], state=connected] 2024-04-06T09:33:55.083 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=22, tag_tid=2, entry_tid=24974], [object_number=21, tag_tid=2, entry_tid=24973], [object_number=20, tag_tid=2, entry_tid=24972], [object_number=23, tag_tid=2, entry_tid=24971]]], state=connected] 2024-04-06T09:33:55.087 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:33:55.088 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=25, tag_tid=3, entry_tid=1' 2024-04-06T09:33:55.089 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:33:55.089 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:33:55.089 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:33:55.089 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:33:55.090 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:33:55.090 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:33:55.090 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:33:55.090 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:33:55.090 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:33:55.090 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:33:55.090 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:33:55.090 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:33:55.091 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:33:55.091 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:33:55.091 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:33:55.136 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 5 2024-04-06T09:33:55.136 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 6 2024-04-06T09:33:55.136 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:33:55.136 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=25, tag_tid=3, entry_tid=1], [object_number=24, tag_tid=3, entry_tid=0], [object_number=27, tag_tid=2, entry_tid=27223], [object_number=26, tag_tid=2, entry_tid=27222]]], state=connected] 2024-04-06T09:33:55.136 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=22, tag_tid=2, entry_tid=24974], [object_number=21, tag_tid=2, entry_tid=24973], [object_number=20, tag_tid=2, entry_tid=24972], [object_number=23, tag_tid=2, entry_tid=24971]]], state=connected] 2024-04-06T09:33:55.140 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:33:55.142 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=22, tag_tid=2, entry_tid=24974' 2024-04-06T09:33:55.142 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=25, tag_tid=3, entry_tid=1' -a 'object_number=25, tag_tid=3, entry_tid=1' = 'object_number=22, tag_tid=2, entry_tid=24974' 2024-04-06T09:33:55.142 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=22, tag_tid=2, entry_tid=24974' '!=' 'object_number=23, tag_tid=2, entry_tid=24863' 2024-04-06T09:33:55.142 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:33:55.143 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=22, tag_tid=2, entry_tid=24974' = 'object_number=23, tag_tid=2, entry_tid=24863' 2024-04-06T09:33:55.143 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=22, tag_tid=2, entry_tid=24974' 2024-04-06T09:33:55.143 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=25, tag_tid=3, entry_tid=1' 2024-04-06T09:33:55.143 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:33:55.143 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:33:55.143 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=3 2024-04-06T09:33:55.144 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=22, tag_tid=2, entry_tid=24974' 2024-04-06T09:33:55.144 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:33:55.144 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:33:55.145 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=2 2024-04-06T09:33:55.146 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=25, tag_tid=3, entry_tid=1' 2024-04-06T09:33:55.146 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:33:55.146 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:33:55.147 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:33:55.147 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=22, tag_tid=2, entry_tid=24974' 2024-04-06T09:33:55.148 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:33:55.148 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:33:55.148 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=24974 2024-04-06T09:33:55.149 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 3 = 2 -a 1 -le 24974 2024-04-06T09:33:55.149 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:33:55.149 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:33:55.149 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:33:55.351 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:33:55.351 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:33:55.351 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:33:55.351 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:33:55.351 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:33:55.351 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:33:55.351 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:33:55.351 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:33:55.351 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:33:55.351 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:33:55.351 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:33:55.351 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:33:55.352 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:33:55.352 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:33:55.352 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:33:55.355 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:33:55.355 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:33:55.355 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:33:55.355 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:33:55.355 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:33:55.355 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:33:55.355 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:33:55.355 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:33:55.355 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:33:55.355 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:33:55.355 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:33:55.355 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:33:55.356 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:33:55.356 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:33:55.356 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:33:55.356 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:33:55.356 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:33:55.356 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:33:55.356 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:33:55.356 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:33:55.356 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:33:55.356 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:33:55.356 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:33:55.356 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:33:55.356 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:33:55.356 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:33:55.356 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:33:55.356 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:33:55.357 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:33:55.357 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:33:55.357 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:33:55.357 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:33:55.357 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:33:55.357 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:33:55.382 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:33:55.383 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:33:55.383 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:33:55.383 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:33:55.512 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:33:55.513 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:33:55.513 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:33:55.513 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:33:55.514 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:33:55.514 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:33:55.514 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:33:55.514 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:33:55.514 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:33:55.514 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:33:55.514 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:33:55.514 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:33:55.514 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:33:55.515 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:33:55.515 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:33:55.515 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:33:55.570 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 5 2024-04-06T09:33:55.570 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 6 2024-04-06T09:33:55.571 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:33:55.571 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=25, tag_tid=3, entry_tid=1], [object_number=24, tag_tid=3, entry_tid=0], [object_number=27, tag_tid=2, entry_tid=27223], [object_number=26, tag_tid=2, entry_tid=27222]]], state=connected] 2024-04-06T09:33:55.571 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=20, tag_tid=2, entry_tid=25092], [object_number=23, tag_tid=2, entry_tid=25091], [object_number=22, tag_tid=2, entry_tid=25090], [object_number=21, tag_tid=2, entry_tid=25089]]], state=connected] 2024-04-06T09:33:55.574 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:33:55.575 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=25, tag_tid=3, entry_tid=1' 2024-04-06T09:33:55.576 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:33:55.577 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:33:55.577 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:33:55.577 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:33:55.577 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:33:55.577 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:33:55.577 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:33:55.577 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:33:55.577 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:33:55.577 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:33:55.577 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:33:55.577 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:33:55.578 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:33:55.578 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:33:55.578 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:33:55.622 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 5 2024-04-06T09:33:55.622 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 6 2024-04-06T09:33:55.622 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:33:55.622 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=25, tag_tid=3, entry_tid=1], [object_number=24, tag_tid=3, entry_tid=0], [object_number=27, tag_tid=2, entry_tid=27223], [object_number=26, tag_tid=2, entry_tid=27222]]], state=connected] 2024-04-06T09:33:55.622 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=20, tag_tid=2, entry_tid=25092], [object_number=23, tag_tid=2, entry_tid=25091], [object_number=22, tag_tid=2, entry_tid=25090], [object_number=21, tag_tid=2, entry_tid=25089]]], state=connected] 2024-04-06T09:33:55.625 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:33:55.627 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=20, tag_tid=2, entry_tid=25092' 2024-04-06T09:33:55.628 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=25, tag_tid=3, entry_tid=1' -a 'object_number=25, tag_tid=3, entry_tid=1' = 'object_number=20, tag_tid=2, entry_tid=25092' 2024-04-06T09:33:55.628 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=20, tag_tid=2, entry_tid=25092' '!=' 'object_number=22, tag_tid=2, entry_tid=24974' 2024-04-06T09:33:55.628 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:33:55.628 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=20, tag_tid=2, entry_tid=25092' = 'object_number=22, tag_tid=2, entry_tid=24974' 2024-04-06T09:33:55.628 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=20, tag_tid=2, entry_tid=25092' 2024-04-06T09:33:55.628 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=25, tag_tid=3, entry_tid=1' 2024-04-06T09:33:55.628 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:33:55.628 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:33:55.629 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=3 2024-04-06T09:33:55.629 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=20, tag_tid=2, entry_tid=25092' 2024-04-06T09:33:55.629 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:33:55.629 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:33:55.630 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=2 2024-04-06T09:33:55.631 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=25, tag_tid=3, entry_tid=1' 2024-04-06T09:33:55.631 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:33:55.631 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:33:55.632 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:33:55.633 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=20, tag_tid=2, entry_tid=25092' 2024-04-06T09:33:55.633 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:33:55.633 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:33:55.634 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=25092 2024-04-06T09:33:55.634 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 3 = 2 -a 1 -le 25092 2024-04-06T09:33:55.634 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:33:55.634 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:33:55.634 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:33:55.836 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:33:55.836 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:33:55.836 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:33:55.836 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:33:55.836 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:33:55.836 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:33:55.836 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:33:55.836 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:33:55.836 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:33:55.836 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:33:55.836 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:33:55.837 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:33:55.837 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:33:55.837 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:33:55.837 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:33:55.837 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:33:55.837 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:33:55.841 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:33:55.841 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:33:55.841 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:33:55.841 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:33:55.841 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:33:55.841 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:33:55.841 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:33:55.841 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:33:55.841 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:33:55.841 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:33:55.841 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:33:55.841 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:33:55.842 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:33:55.842 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:33:55.842 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:33:55.842 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:33:55.842 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:33:55.842 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:33:55.842 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:33:55.842 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:33:55.842 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:33:55.842 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:33:55.842 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:33:55.842 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:33:55.842 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:33:55.842 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:33:55.843 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:33:55.843 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:33:55.843 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:33:55.843 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:33:55.843 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:33:55.843 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:33:55.867 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:33:55.867 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:33:55.868 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:33:55.868 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:33:56.006 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:33:56.006 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:33:56.006 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:33:56.006 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:33:56.006 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:33:56.006 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:33:56.006 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:33:56.006 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:33:56.007 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:33:56.007 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:33:56.007 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:33:56.007 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:33:56.007 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:33:56.009 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:33:56.009 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:33:56.010 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:33:56.072 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 6 2024-04-06T09:33:56.072 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 6 2024-04-06T09:33:56.072 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:33:56.072 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=25, tag_tid=3, entry_tid=1], [object_number=24, tag_tid=3, entry_tid=0], [object_number=27, tag_tid=2, entry_tid=27223], [object_number=26, tag_tid=2, entry_tid=27222]]], state=connected] 2024-04-06T09:33:56.072 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=26, tag_tid=2, entry_tid=25182], [object_number=25, tag_tid=2, entry_tid=25181], [object_number=24, tag_tid=2, entry_tid=25180], [object_number=27, tag_tid=2, entry_tid=25179]]], state=connected] 2024-04-06T09:33:56.076 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:33:56.078 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=25, tag_tid=3, entry_tid=1' 2024-04-06T09:33:56.078 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:33:56.078 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:33:56.079 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:33:56.079 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:33:56.079 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:33:56.079 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:33:56.079 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:33:56.079 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:33:56.079 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:33:56.079 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:33:56.079 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:33:56.079 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:33:56.080 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:33:56.080 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:33:56.080 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:33:56.124 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 6 2024-04-06T09:33:56.124 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 6 2024-04-06T09:33:56.124 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:33:56.124 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=25, tag_tid=3, entry_tid=1], [object_number=24, tag_tid=3, entry_tid=0], [object_number=27, tag_tid=2, entry_tid=27223], [object_number=26, tag_tid=2, entry_tid=27222]]], state=connected] 2024-04-06T09:33:56.124 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=26, tag_tid=2, entry_tid=25182], [object_number=25, tag_tid=2, entry_tid=25181], [object_number=24, tag_tid=2, entry_tid=25180], [object_number=27, tag_tid=2, entry_tid=25179]]], state=connected] 2024-04-06T09:33:56.127 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:33:56.129 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=26, tag_tid=2, entry_tid=25182' 2024-04-06T09:33:56.130 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=25, tag_tid=3, entry_tid=1' -a 'object_number=25, tag_tid=3, entry_tid=1' = 'object_number=26, tag_tid=2, entry_tid=25182' 2024-04-06T09:33:56.130 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=26, tag_tid=2, entry_tid=25182' '!=' 'object_number=20, tag_tid=2, entry_tid=25092' 2024-04-06T09:33:56.130 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:33:56.130 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=26, tag_tid=2, entry_tid=25182' = 'object_number=20, tag_tid=2, entry_tid=25092' 2024-04-06T09:33:56.130 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=26, tag_tid=2, entry_tid=25182' 2024-04-06T09:33:56.130 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=25, tag_tid=3, entry_tid=1' 2024-04-06T09:33:56.130 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:33:56.130 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:33:56.130 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=3 2024-04-06T09:33:56.131 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=26, tag_tid=2, entry_tid=25182' 2024-04-06T09:33:56.131 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:33:56.131 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:33:56.132 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=2 2024-04-06T09:33:56.133 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=25, tag_tid=3, entry_tid=1' 2024-04-06T09:33:56.133 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:33:56.133 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:33:56.134 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:33:56.134 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=26, tag_tid=2, entry_tid=25182' 2024-04-06T09:33:56.135 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:33:56.135 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:33:56.136 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=25182 2024-04-06T09:33:56.136 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 3 = 2 -a 1 -le 25182 2024-04-06T09:33:56.136 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:33:56.136 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:33:56.136 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:33:56.338 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:33:56.338 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:33:56.338 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:33:56.338 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:33:56.338 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:33:56.339 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:33:56.339 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:33:56.339 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:33:56.339 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:33:56.339 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:33:56.339 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:33:56.339 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:33:56.339 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:33:56.339 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:33:56.339 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:33:56.340 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:33:56.340 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:33:56.341 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:33:56.341 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:33:56.341 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:33:56.341 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:33:56.341 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:33:56.341 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:33:56.341 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:33:56.341 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:33:56.341 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:33:56.341 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:33:56.342 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:33:56.342 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:33:56.342 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:33:56.342 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:33:56.342 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:33:56.342 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:33:56.344 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:33:56.344 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:33:56.344 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:33:56.344 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:33:56.344 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:33:56.344 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:33:56.344 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:33:56.344 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:33:56.344 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:33:56.344 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:33:56.344 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:33:56.345 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:33:56.345 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:33:56.345 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:33:56.345 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:33:56.345 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:33:56.367 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:33:56.368 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:33:56.368 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:33:56.368 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:33:56.495 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:33:56.496 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:33:56.496 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:33:56.497 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:33:56.497 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:33:56.497 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:33:56.497 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:33:56.497 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:33:56.497 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:33:56.497 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:33:56.497 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:33:56.498 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:33:56.498 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:33:56.498 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:33:56.499 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:33:56.499 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:33:56.555 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 6 2024-04-06T09:33:56.555 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 6 2024-04-06T09:33:56.555 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:33:56.555 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=25, tag_tid=3, entry_tid=1], [object_number=24, tag_tid=3, entry_tid=0], [object_number=27, tag_tid=2, entry_tid=27223], [object_number=26, tag_tid=2, entry_tid=27222]]], state=connected] 2024-04-06T09:33:56.556 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=27, tag_tid=2, entry_tid=25295], [object_number=26, tag_tid=2, entry_tid=25294], [object_number=25, tag_tid=2, entry_tid=25293], [object_number=24, tag_tid=2, entry_tid=25292]]], state=connected] 2024-04-06T09:33:56.559 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:33:56.561 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=25, tag_tid=3, entry_tid=1' 2024-04-06T09:33:56.562 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:33:56.562 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:33:56.562 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:33:56.562 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:33:56.562 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:33:56.562 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:33:56.562 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:33:56.562 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:33:56.562 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:33:56.563 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:33:56.563 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:33:56.563 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:33:56.563 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:33:56.563 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:33:56.564 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:33:56.608 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 6 2024-04-06T09:33:56.609 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 6 2024-04-06T09:33:56.609 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:33:56.609 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=25, tag_tid=3, entry_tid=1], [object_number=24, tag_tid=3, entry_tid=0], [object_number=27, tag_tid=2, entry_tid=27223], [object_number=26, tag_tid=2, entry_tid=27222]]], state=connected] 2024-04-06T09:33:56.609 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=27, tag_tid=2, entry_tid=25295], [object_number=26, tag_tid=2, entry_tid=25294], [object_number=25, tag_tid=2, entry_tid=25293], [object_number=24, tag_tid=2, entry_tid=25292]]], state=connected] 2024-04-06T09:33:56.612 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:33:56.614 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=27, tag_tid=2, entry_tid=25295' 2024-04-06T09:33:56.615 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=25, tag_tid=3, entry_tid=1' -a 'object_number=25, tag_tid=3, entry_tid=1' = 'object_number=27, tag_tid=2, entry_tid=25295' 2024-04-06T09:33:56.615 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=27, tag_tid=2, entry_tid=25295' '!=' 'object_number=26, tag_tid=2, entry_tid=25182' 2024-04-06T09:33:56.615 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:33:56.615 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=27, tag_tid=2, entry_tid=25295' = 'object_number=26, tag_tid=2, entry_tid=25182' 2024-04-06T09:33:56.615 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=27, tag_tid=2, entry_tid=25295' 2024-04-06T09:33:56.615 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=25, tag_tid=3, entry_tid=1' 2024-04-06T09:33:56.615 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:33:56.615 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:33:56.616 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=3 2024-04-06T09:33:56.616 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=27, tag_tid=2, entry_tid=25295' 2024-04-06T09:33:56.616 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:33:56.616 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:33:56.617 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=2 2024-04-06T09:33:56.618 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=25, tag_tid=3, entry_tid=1' 2024-04-06T09:33:56.618 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:33:56.618 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:33:56.619 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:33:56.619 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=27, tag_tid=2, entry_tid=25295' 2024-04-06T09:33:56.620 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:33:56.620 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:33:56.621 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=25295 2024-04-06T09:33:56.621 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 3 = 2 -a 1 -le 25295 2024-04-06T09:33:56.621 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:33:56.621 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:33:56.621 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:33:56.823 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:33:56.823 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:33:56.823 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:33:56.823 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:33:56.823 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:33:56.823 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:33:56.823 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:33:56.823 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:33:56.823 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:33:56.823 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:33:56.823 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:33:56.823 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:33:56.824 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:33:56.824 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:33:56.824 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:33:56.824 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:33:56.824 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:33:56.827 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:33:56.827 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:33:56.827 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:33:56.828 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:33:56.828 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:33:56.828 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:33:56.828 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:33:56.828 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:33:56.828 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:33:56.828 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:33:56.828 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:33:56.828 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:33:56.828 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:33:56.828 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:33:56.828 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:33:56.828 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:33:56.829 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:33:56.829 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:33:56.829 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:33:56.829 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:33:56.829 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:33:56.829 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:33:56.829 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:33:56.829 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:33:56.829 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:33:56.829 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:33:56.829 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:33:56.829 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:33:56.829 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:33:56.829 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:33:56.829 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:33:56.830 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:33:56.853 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:33:56.853 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:33:56.853 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:33:56.854 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:33:56.981 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:33:56.982 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:33:56.982 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:33:56.982 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:33:56.982 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:33:56.983 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:33:56.983 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:33:56.983 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:33:56.983 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:33:56.983 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:33:56.983 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:33:56.983 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:33:56.984 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:33:56.985 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:33:56.985 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:33:56.985 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:33:57.241 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 6 2024-04-06T09:33:57.242 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 6 2024-04-06T09:33:57.242 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:33:57.242 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=25, tag_tid=3, entry_tid=1], [object_number=24, tag_tid=3, entry_tid=0], [object_number=27, tag_tid=2, entry_tid=27223], [object_number=26, tag_tid=2, entry_tid=27222]]], state=connected] 2024-04-06T09:33:57.242 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=25, tag_tid=2, entry_tid=25393], [object_number=24, tag_tid=2, entry_tid=25392], [object_number=27, tag_tid=2, entry_tid=25391], [object_number=26, tag_tid=2, entry_tid=25390]]], state=connected] 2024-04-06T09:33:57.247 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:33:57.249 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=25, tag_tid=3, entry_tid=1' 2024-04-06T09:33:57.250 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:33:57.251 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:33:57.251 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:33:57.251 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:33:57.251 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:33:57.251 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:33:57.251 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:33:57.251 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:33:57.251 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:33:57.251 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:33:57.252 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:33:57.252 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:33:57.254 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:33:57.254 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:33:57.254 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:33:57.307 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 6 2024-04-06T09:33:57.308 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 6 2024-04-06T09:33:57.308 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:33:57.308 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=25, tag_tid=3, entry_tid=1], [object_number=24, tag_tid=3, entry_tid=0], [object_number=27, tag_tid=2, entry_tid=27223], [object_number=26, tag_tid=2, entry_tid=27222]]], state=connected] 2024-04-06T09:33:57.308 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=25, tag_tid=2, entry_tid=25393], [object_number=24, tag_tid=2, entry_tid=25392], [object_number=27, tag_tid=2, entry_tid=25391], [object_number=26, tag_tid=2, entry_tid=25390]]], state=connected] 2024-04-06T09:33:57.312 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:33:57.315 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=25, tag_tid=2, entry_tid=25393' 2024-04-06T09:33:57.315 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=25, tag_tid=3, entry_tid=1' -a 'object_number=25, tag_tid=3, entry_tid=1' = 'object_number=25, tag_tid=2, entry_tid=25393' 2024-04-06T09:33:57.315 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=25, tag_tid=2, entry_tid=25393' '!=' 'object_number=27, tag_tid=2, entry_tid=25295' 2024-04-06T09:33:57.315 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:33:57.315 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=25, tag_tid=2, entry_tid=25393' = 'object_number=27, tag_tid=2, entry_tid=25295' 2024-04-06T09:33:57.315 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=25, tag_tid=2, entry_tid=25393' 2024-04-06T09:33:57.316 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=25, tag_tid=3, entry_tid=1' 2024-04-06T09:33:57.316 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:33:57.316 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:33:57.317 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=3 2024-04-06T09:33:57.318 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=25, tag_tid=2, entry_tid=25393' 2024-04-06T09:33:57.318 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:33:57.318 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:33:57.318 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=2 2024-04-06T09:33:57.320 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=25, tag_tid=3, entry_tid=1' 2024-04-06T09:33:57.320 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:33:57.320 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:33:57.320 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:33:57.321 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=25, tag_tid=2, entry_tid=25393' 2024-04-06T09:33:57.321 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:33:57.321 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:33:57.322 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=25393 2024-04-06T09:33:57.322 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 3 = 2 -a 1 -le 25393 2024-04-06T09:33:57.322 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:33:57.322 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:33:57.323 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:33:57.523 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:33:57.523 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:33:57.523 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:33:57.523 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:33:57.523 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:33:57.524 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:33:57.524 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:33:57.524 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:33:57.524 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:33:57.524 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:33:57.524 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:33:57.524 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:33:57.524 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:33:57.524 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:33:57.524 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:33:57.525 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:33:57.526 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:33:57.530 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:33:57.530 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:33:57.530 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:33:57.530 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:33:57.530 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:33:57.530 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:33:57.530 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:33:57.530 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:33:57.531 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:33:57.531 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:33:57.531 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:33:57.531 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:33:57.531 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:33:57.531 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:33:57.531 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:33:57.531 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:33:57.531 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:33:57.531 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:33:57.531 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:33:57.531 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:33:57.531 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:33:57.531 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:33:57.531 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:33:57.532 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:33:57.532 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:33:57.532 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:33:57.532 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:33:57.532 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:33:57.532 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:33:57.532 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:33:57.532 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:33:57.532 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:33:57.555 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:33:57.555 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:33:57.555 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:33:57.555 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:33:57.683 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:33:57.684 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:33:57.684 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:33:57.684 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:33:57.684 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:33:57.684 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:33:57.684 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:33:57.684 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:33:57.684 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:33:57.684 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:33:57.684 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:33:57.685 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:33:57.685 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:33:57.685 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:33:57.686 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:33:57.686 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:33:57.730 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 6 2024-04-06T09:33:57.730 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 6 2024-04-06T09:33:57.730 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:33:57.730 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=25, tag_tid=3, entry_tid=1], [object_number=24, tag_tid=3, entry_tid=0], [object_number=27, tag_tid=2, entry_tid=27223], [object_number=26, tag_tid=2, entry_tid=27222]]], state=connected] 2024-04-06T09:33:57.731 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=26, tag_tid=2, entry_tid=25538], [object_number=25, tag_tid=2, entry_tid=25537], [object_number=24, tag_tid=2, entry_tid=25536], [object_number=27, tag_tid=2, entry_tid=25535]]], state=connected] 2024-04-06T09:33:57.734 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:33:57.736 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=25, tag_tid=3, entry_tid=1' 2024-04-06T09:33:57.736 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:33:57.736 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:33:57.736 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:33:57.736 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:33:57.736 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:33:57.736 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:33:57.736 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:33:57.737 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:33:57.737 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:33:57.737 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:33:57.737 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:33:57.737 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:33:57.737 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:33:57.738 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:33:57.738 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:33:57.782 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 6 2024-04-06T09:33:57.782 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 6 2024-04-06T09:33:57.782 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:33:57.782 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=25, tag_tid=3, entry_tid=1], [object_number=24, tag_tid=3, entry_tid=0], [object_number=27, tag_tid=2, entry_tid=27223], [object_number=26, tag_tid=2, entry_tid=27222]]], state=connected] 2024-04-06T09:33:57.782 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=26, tag_tid=2, entry_tid=25538], [object_number=25, tag_tid=2, entry_tid=25537], [object_number=24, tag_tid=2, entry_tid=25536], [object_number=27, tag_tid=2, entry_tid=25535]]], state=connected] 2024-04-06T09:33:57.786 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:33:57.787 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=26, tag_tid=2, entry_tid=25538' 2024-04-06T09:33:57.787 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=25, tag_tid=3, entry_tid=1' -a 'object_number=25, tag_tid=3, entry_tid=1' = 'object_number=26, tag_tid=2, entry_tid=25538' 2024-04-06T09:33:57.788 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=26, tag_tid=2, entry_tid=25538' '!=' 'object_number=25, tag_tid=2, entry_tid=25393' 2024-04-06T09:33:57.788 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:33:57.788 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=26, tag_tid=2, entry_tid=25538' = 'object_number=25, tag_tid=2, entry_tid=25393' 2024-04-06T09:33:57.788 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=26, tag_tid=2, entry_tid=25538' 2024-04-06T09:33:57.788 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=25, tag_tid=3, entry_tid=1' 2024-04-06T09:33:57.788 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:33:57.789 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:33:57.789 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=3 2024-04-06T09:33:57.789 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=26, tag_tid=2, entry_tid=25538' 2024-04-06T09:33:57.790 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:33:57.790 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:33:57.790 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=2 2024-04-06T09:33:57.791 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=25, tag_tid=3, entry_tid=1' 2024-04-06T09:33:57.791 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:33:57.791 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:33:57.792 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:33:57.793 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=26, tag_tid=2, entry_tid=25538' 2024-04-06T09:33:57.793 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:33:57.793 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:33:57.794 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=25538 2024-04-06T09:33:57.794 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 3 = 2 -a 1 -le 25538 2024-04-06T09:33:57.794 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:33:57.794 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:33:57.794 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:33:57.996 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:33:57.996 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:33:57.996 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:33:57.996 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:33:57.996 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:33:57.996 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:33:57.996 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:33:57.996 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:33:57.996 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:33:57.996 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:33:57.997 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:33:57.997 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:33:57.997 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:33:57.997 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:33:57.997 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:33:57.997 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:33:57.997 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:33:57.999 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:33:57.999 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:33:58.000 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:33:58.000 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:33:58.000 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:33:58.000 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:33:58.000 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:33:58.000 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:33:58.000 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:33:58.000 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:33:58.000 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:33:58.000 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:33:58.000 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:33:58.000 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:33:58.000 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:33:58.001 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:33:58.002 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:33:58.002 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:33:58.002 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:33:58.002 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:33:58.002 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:33:58.003 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:33:58.003 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:33:58.003 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:33:58.003 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:33:58.003 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:33:58.003 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:33:58.003 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:33:58.003 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:33:58.003 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:33:58.003 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:33:58.003 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:33:58.027 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:33:58.028 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:33:58.028 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:33:58.028 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:33:58.156 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:33:58.159 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:33:58.159 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:33:58.159 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:33:58.159 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:33:58.159 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:33:58.159 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:33:58.159 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:33:58.160 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:33:58.160 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:33:58.160 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:33:58.160 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:33:58.160 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:33:58.160 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:33:58.161 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:33:58.161 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:33:58.209 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 6 2024-04-06T09:33:58.210 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 6 2024-04-06T09:33:58.210 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:33:58.210 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=25, tag_tid=3, entry_tid=1], [object_number=24, tag_tid=3, entry_tid=0], [object_number=27, tag_tid=2, entry_tid=27223], [object_number=26, tag_tid=2, entry_tid=27222]]], state=connected] 2024-04-06T09:33:58.210 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=25, tag_tid=2, entry_tid=25637], [object_number=24, tag_tid=2, entry_tid=25636], [object_number=27, tag_tid=2, entry_tid=25635], [object_number=26, tag_tid=2, entry_tid=25634]]], state=connected] 2024-04-06T09:33:58.213 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:33:58.215 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=25, tag_tid=3, entry_tid=1' 2024-04-06T09:33:58.216 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:33:58.216 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:33:58.216 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:33:58.216 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:33:58.216 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:33:58.216 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:33:58.216 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:33:58.216 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:33:58.216 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:33:58.216 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:33:58.217 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:33:58.217 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:33:58.217 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:33:58.217 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:33:58.217 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:33:58.262 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 6 2024-04-06T09:33:58.262 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 6 2024-04-06T09:33:58.262 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:33:58.262 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=25, tag_tid=3, entry_tid=1], [object_number=24, tag_tid=3, entry_tid=0], [object_number=27, tag_tid=2, entry_tid=27223], [object_number=26, tag_tid=2, entry_tid=27222]]], state=connected] 2024-04-06T09:33:58.263 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=25, tag_tid=2, entry_tid=25637], [object_number=24, tag_tid=2, entry_tid=25636], [object_number=27, tag_tid=2, entry_tid=25635], [object_number=26, tag_tid=2, entry_tid=25634]]], state=connected] 2024-04-06T09:33:58.266 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:33:58.267 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=25, tag_tid=2, entry_tid=25637' 2024-04-06T09:33:58.268 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=25, tag_tid=3, entry_tid=1' -a 'object_number=25, tag_tid=3, entry_tid=1' = 'object_number=25, tag_tid=2, entry_tid=25637' 2024-04-06T09:33:58.268 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=25, tag_tid=2, entry_tid=25637' '!=' 'object_number=26, tag_tid=2, entry_tid=25538' 2024-04-06T09:33:58.268 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:33:58.268 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=25, tag_tid=2, entry_tid=25637' = 'object_number=26, tag_tid=2, entry_tid=25538' 2024-04-06T09:33:58.268 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=25, tag_tid=2, entry_tid=25637' 2024-04-06T09:33:58.270 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=25, tag_tid=3, entry_tid=1' 2024-04-06T09:33:58.270 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:33:58.270 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:33:58.270 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=3 2024-04-06T09:33:58.270 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=25, tag_tid=2, entry_tid=25637' 2024-04-06T09:33:58.270 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:33:58.270 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:33:58.271 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=2 2024-04-06T09:33:58.271 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=25, tag_tid=3, entry_tid=1' 2024-04-06T09:33:58.271 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:33:58.272 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:33:58.272 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:33:58.273 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=25, tag_tid=2, entry_tid=25637' 2024-04-06T09:33:58.273 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:33:58.273 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:33:58.274 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=25637 2024-04-06T09:33:58.274 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 3 = 2 -a 1 -le 25637 2024-04-06T09:33:58.274 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:33:58.274 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:33:58.274 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:33:58.476 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:33:58.476 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:33:58.476 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:33:58.476 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:33:58.476 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:33:58.477 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:33:58.477 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:33:58.477 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:33:58.477 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:33:58.477 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:33:58.477 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:33:58.477 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:33:58.477 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:33:58.477 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:33:58.477 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:33:58.477 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:33:58.478 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:33:58.480 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:33:58.480 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:33:58.480 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:33:58.480 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:33:58.480 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:33:58.480 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:33:58.480 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:33:58.481 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:33:58.481 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:33:58.481 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:33:58.481 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:33:58.481 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:33:58.481 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:33:58.481 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:33:58.481 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:33:58.481 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:33:58.482 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:33:58.483 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:33:58.483 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:33:58.483 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:33:58.483 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:33:58.483 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:33:58.483 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:33:58.483 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:33:58.483 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:33:58.483 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:33:58.483 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:33:58.483 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:33:58.483 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:33:58.483 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:33:58.483 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:33:58.484 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:33:58.506 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:33:58.507 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:33:58.507 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:33:58.507 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:33:58.628 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:33:58.629 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:33:58.629 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:33:58.629 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:33:58.629 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:33:58.630 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:33:58.630 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:33:58.630 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:33:58.630 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:33:58.630 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:33:58.631 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:33:58.631 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:33:58.631 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:33:58.632 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:33:58.633 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:33:58.633 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:33:58.678 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 6 2024-04-06T09:33:58.678 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 6 2024-04-06T09:33:58.678 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:33:58.678 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=25, tag_tid=3, entry_tid=1], [object_number=24, tag_tid=3, entry_tid=0], [object_number=27, tag_tid=2, entry_tid=27223], [object_number=26, tag_tid=2, entry_tid=27222]]], state=connected] 2024-04-06T09:33:58.678 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=26, tag_tid=2, entry_tid=25730], [object_number=25, tag_tid=2, entry_tid=25729], [object_number=24, tag_tid=2, entry_tid=25728], [object_number=27, tag_tid=2, entry_tid=25727]]], state=connected] 2024-04-06T09:33:58.682 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:33:58.684 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=25, tag_tid=3, entry_tid=1' 2024-04-06T09:33:58.684 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:33:58.684 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:33:58.684 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:33:58.684 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:33:58.684 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:33:58.684 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:33:58.684 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:33:58.684 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:33:58.684 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:33:58.684 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:33:58.685 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:33:58.685 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:33:58.685 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:33:58.686 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:33:58.686 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:33:58.730 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 6 2024-04-06T09:33:58.730 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 6 2024-04-06T09:33:58.730 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:33:58.730 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=25, tag_tid=3, entry_tid=1], [object_number=24, tag_tid=3, entry_tid=0], [object_number=27, tag_tid=2, entry_tid=27223], [object_number=26, tag_tid=2, entry_tid=27222]]], state=connected] 2024-04-06T09:33:58.730 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=26, tag_tid=2, entry_tid=25730], [object_number=25, tag_tid=2, entry_tid=25729], [object_number=24, tag_tid=2, entry_tid=25728], [object_number=27, tag_tid=2, entry_tid=25727]]], state=connected] 2024-04-06T09:33:58.734 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:33:58.736 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=26, tag_tid=2, entry_tid=25730' 2024-04-06T09:33:58.736 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=25, tag_tid=3, entry_tid=1' -a 'object_number=25, tag_tid=3, entry_tid=1' = 'object_number=26, tag_tid=2, entry_tid=25730' 2024-04-06T09:33:58.736 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=26, tag_tid=2, entry_tid=25730' '!=' 'object_number=25, tag_tid=2, entry_tid=25637' 2024-04-06T09:33:58.736 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:33:58.736 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=26, tag_tid=2, entry_tid=25730' = 'object_number=25, tag_tid=2, entry_tid=25637' 2024-04-06T09:33:58.736 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=26, tag_tid=2, entry_tid=25730' 2024-04-06T09:33:58.736 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=25, tag_tid=3, entry_tid=1' 2024-04-06T09:33:58.736 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:33:58.737 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:33:58.737 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=3 2024-04-06T09:33:58.738 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=26, tag_tid=2, entry_tid=25730' 2024-04-06T09:33:58.738 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:33:58.738 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:33:58.739 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=2 2024-04-06T09:33:58.740 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=25, tag_tid=3, entry_tid=1' 2024-04-06T09:33:58.740 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:33:58.740 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:33:58.741 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:33:58.742 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=26, tag_tid=2, entry_tid=25730' 2024-04-06T09:33:58.742 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:33:58.742 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:33:58.743 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=25730 2024-04-06T09:33:58.743 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 3 = 2 -a 1 -le 25730 2024-04-06T09:33:58.743 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:33:58.743 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:33:58.743 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:33:58.944 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:33:58.945 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:33:58.945 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:33:58.945 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:33:58.945 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:33:58.945 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:33:58.945 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:33:58.945 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:33:58.945 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:33:58.945 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:33:58.945 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:33:58.945 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:33:58.945 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:33:58.946 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:33:58.946 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:33:58.946 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:33:58.946 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:33:58.948 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:33:58.948 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:33:58.948 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:33:58.948 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:33:58.948 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:33:58.949 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:33:58.949 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:33:58.949 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:33:58.949 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:33:58.949 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:33:58.949 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:33:58.949 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:33:58.949 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:33:58.949 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:33:58.949 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:33:58.949 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:33:58.950 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:33:58.950 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:33:58.950 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:33:58.950 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:33:58.950 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:33:58.950 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:33:58.951 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:33:58.951 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:33:58.951 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:33:58.951 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:33:58.951 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:33:58.951 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:33:58.951 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:33:58.951 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:33:58.951 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:33:58.951 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:33:58.976 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:33:58.976 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:33:58.976 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:33:58.976 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:33:59.110 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:33:59.112 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:33:59.112 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:33:59.112 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:33:59.112 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:33:59.112 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:33:59.112 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:33:59.112 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:33:59.112 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:33:59.112 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:33:59.113 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:33:59.113 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:33:59.113 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:33:59.115 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:33:59.116 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:33:59.116 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:33:59.171 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 6 2024-04-06T09:33:59.171 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 6 2024-04-06T09:33:59.172 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:33:59.172 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=25, tag_tid=3, entry_tid=1], [object_number=24, tag_tid=3, entry_tid=0], [object_number=27, tag_tid=2, entry_tid=27223], [object_number=26, tag_tid=2, entry_tid=27222]]], state=connected] 2024-04-06T09:33:59.172 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=25, tag_tid=2, entry_tid=25825], [object_number=24, tag_tid=2, entry_tid=25824], [object_number=27, tag_tid=2, entry_tid=25823], [object_number=26, tag_tid=2, entry_tid=25822]]], state=connected] 2024-04-06T09:33:59.175 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:33:59.177 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=25, tag_tid=3, entry_tid=1' 2024-04-06T09:33:59.177 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:33:59.177 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:33:59.178 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:33:59.178 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:33:59.178 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:33:59.178 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:33:59.178 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:33:59.178 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:33:59.178 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:33:59.178 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:33:59.178 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:33:59.178 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:33:59.179 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:33:59.179 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:33:59.179 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:33:59.224 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 6 2024-04-06T09:33:59.224 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 6 2024-04-06T09:33:59.224 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:33:59.225 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=25, tag_tid=3, entry_tid=1], [object_number=24, tag_tid=3, entry_tid=0], [object_number=27, tag_tid=2, entry_tid=27223], [object_number=26, tag_tid=2, entry_tid=27222]]], state=connected] 2024-04-06T09:33:59.225 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=25, tag_tid=2, entry_tid=25825], [object_number=24, tag_tid=2, entry_tid=25824], [object_number=27, tag_tid=2, entry_tid=25823], [object_number=26, tag_tid=2, entry_tid=25822]]], state=connected] 2024-04-06T09:33:59.228 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:33:59.230 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=25, tag_tid=2, entry_tid=25825' 2024-04-06T09:33:59.230 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=25, tag_tid=3, entry_tid=1' -a 'object_number=25, tag_tid=3, entry_tid=1' = 'object_number=25, tag_tid=2, entry_tid=25825' 2024-04-06T09:33:59.230 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=25, tag_tid=2, entry_tid=25825' '!=' 'object_number=26, tag_tid=2, entry_tid=25730' 2024-04-06T09:33:59.231 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:33:59.231 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=25, tag_tid=2, entry_tid=25825' = 'object_number=26, tag_tid=2, entry_tid=25730' 2024-04-06T09:33:59.231 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=25, tag_tid=2, entry_tid=25825' 2024-04-06T09:33:59.231 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=25, tag_tid=3, entry_tid=1' 2024-04-06T09:33:59.231 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:33:59.231 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:33:59.231 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=3 2024-04-06T09:33:59.232 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=25, tag_tid=2, entry_tid=25825' 2024-04-06T09:33:59.232 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:33:59.232 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:33:59.233 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=2 2024-04-06T09:33:59.234 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=25, tag_tid=3, entry_tid=1' 2024-04-06T09:33:59.234 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:33:59.234 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:33:59.235 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:33:59.236 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=25, tag_tid=2, entry_tid=25825' 2024-04-06T09:33:59.236 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:33:59.236 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:33:59.237 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=25825 2024-04-06T09:33:59.237 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 3 = 2 -a 1 -le 25825 2024-04-06T09:33:59.237 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:33:59.237 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:33:59.237 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:33:59.439 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:33:59.439 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:33:59.439 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:33:59.439 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:33:59.439 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:33:59.439 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:33:59.439 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:33:59.439 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:33:59.439 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:33:59.440 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:33:59.440 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:33:59.440 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:33:59.440 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:33:59.440 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:33:59.440 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:33:59.440 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:33:59.441 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:33:59.443 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:33:59.443 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:33:59.443 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:33:59.443 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:33:59.443 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:33:59.443 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:33:59.443 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:33:59.443 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:33:59.443 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:33:59.443 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:33:59.443 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:33:59.443 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:33:59.444 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:33:59.444 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:33:59.444 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:33:59.444 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:33:59.445 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:33:59.445 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:33:59.445 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:33:59.445 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:33:59.445 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:33:59.445 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:33:59.445 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:33:59.446 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:33:59.446 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:33:59.446 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:33:59.446 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:33:59.446 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:33:59.446 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:33:59.446 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:33:59.446 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:33:59.446 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:33:59.472 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:33:59.472 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:33:59.472 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:33:59.472 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:33:59.603 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:33:59.604 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:33:59.605 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:33:59.605 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:33:59.605 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:33:59.605 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:33:59.605 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:33:59.605 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:33:59.605 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:33:59.605 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:33:59.606 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:33:59.606 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:33:59.606 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:33:59.607 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:33:59.608 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:33:59.608 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:33:59.655 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 6 2024-04-06T09:33:59.655 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 6 2024-04-06T09:33:59.655 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:33:59.655 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=25, tag_tid=3, entry_tid=1], [object_number=24, tag_tid=3, entry_tid=0], [object_number=27, tag_tid=2, entry_tid=27223], [object_number=26, tag_tid=2, entry_tid=27222]]], state=connected] 2024-04-06T09:33:59.655 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=27, tag_tid=2, entry_tid=25919], [object_number=26, tag_tid=2, entry_tid=25918], [object_number=25, tag_tid=2, entry_tid=25917], [object_number=24, tag_tid=2, entry_tid=25916]]], state=connected] 2024-04-06T09:33:59.658 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:33:59.659 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=25, tag_tid=3, entry_tid=1' 2024-04-06T09:33:59.661 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:33:59.661 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:33:59.661 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:33:59.661 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:33:59.661 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:33:59.661 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:33:59.661 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:33:59.661 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:33:59.661 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:33:59.661 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:33:59.662 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:33:59.662 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:33:59.662 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:33:59.662 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:33:59.662 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:33:59.706 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 6 2024-04-06T09:33:59.706 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 6 2024-04-06T09:33:59.706 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:33:59.707 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=25, tag_tid=3, entry_tid=1], [object_number=24, tag_tid=3, entry_tid=0], [object_number=27, tag_tid=2, entry_tid=27223], [object_number=26, tag_tid=2, entry_tid=27222]]], state=connected] 2024-04-06T09:33:59.707 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=27, tag_tid=2, entry_tid=25919], [object_number=26, tag_tid=2, entry_tid=25918], [object_number=25, tag_tid=2, entry_tid=25917], [object_number=24, tag_tid=2, entry_tid=25916]]], state=connected] 2024-04-06T09:33:59.710 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:33:59.712 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=27, tag_tid=2, entry_tid=25919' 2024-04-06T09:33:59.712 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=25, tag_tid=3, entry_tid=1' -a 'object_number=25, tag_tid=3, entry_tid=1' = 'object_number=27, tag_tid=2, entry_tid=25919' 2024-04-06T09:33:59.713 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=27, tag_tid=2, entry_tid=25919' '!=' 'object_number=25, tag_tid=2, entry_tid=25825' 2024-04-06T09:33:59.713 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:33:59.713 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=27, tag_tid=2, entry_tid=25919' = 'object_number=25, tag_tid=2, entry_tid=25825' 2024-04-06T09:33:59.713 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=27, tag_tid=2, entry_tid=25919' 2024-04-06T09:33:59.713 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=25, tag_tid=3, entry_tid=1' 2024-04-06T09:33:59.713 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:33:59.713 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:33:59.713 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=3 2024-04-06T09:33:59.714 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=27, tag_tid=2, entry_tid=25919' 2024-04-06T09:33:59.714 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:33:59.714 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:33:59.715 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=2 2024-04-06T09:33:59.716 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=25, tag_tid=3, entry_tid=1' 2024-04-06T09:33:59.716 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:33:59.716 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:33:59.717 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:33:59.718 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=27, tag_tid=2, entry_tid=25919' 2024-04-06T09:33:59.718 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:33:59.718 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:33:59.719 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=25919 2024-04-06T09:33:59.719 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 3 = 2 -a 1 -le 25919 2024-04-06T09:33:59.719 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:33:59.719 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:33:59.719 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:33:59.921 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:33:59.921 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:33:59.921 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:33:59.921 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:33:59.921 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:33:59.921 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:33:59.921 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:33:59.921 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:33:59.922 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:33:59.922 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:33:59.922 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:33:59.922 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:33:59.922 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:33:59.922 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:33:59.922 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:33:59.923 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:33:59.923 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:33:59.926 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:33:59.926 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:33:59.926 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:33:59.926 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:33:59.926 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:33:59.926 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:33:59.926 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:33:59.926 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:33:59.926 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:33:59.926 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:33:59.926 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:33:59.926 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:33:59.926 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:33:59.927 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:33:59.927 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:33:59.927 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:33:59.927 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:33:59.927 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:33:59.927 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:33:59.927 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:33:59.927 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:33:59.927 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:33:59.927 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:33:59.927 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:33:59.927 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:33:59.927 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:33:59.927 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:33:59.927 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:33:59.928 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:33:59.928 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:33:59.928 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:33:59.928 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:33:59.951 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:33:59.952 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:33:59.952 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:33:59.952 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:34:00.083 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:34:00.084 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:34:00.084 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:34:00.084 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:34:00.084 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:34:00.084 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:34:00.084 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:34:00.084 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:34:00.084 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:34:00.084 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:34:00.085 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:34:00.085 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:34:00.086 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:34:00.087 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:34:00.088 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:34:00.088 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:34:00.137 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 6 2024-04-06T09:34:00.137 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 6 2024-04-06T09:34:00.138 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:34:00.138 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=25, tag_tid=3, entry_tid=1], [object_number=24, tag_tid=3, entry_tid=0], [object_number=27, tag_tid=2, entry_tid=27223], [object_number=26, tag_tid=2, entry_tid=27222]]], state=connected] 2024-04-06T09:34:00.138 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=24, tag_tid=2, entry_tid=26012], [object_number=27, tag_tid=2, entry_tid=26011], [object_number=26, tag_tid=2, entry_tid=26010], [object_number=25, tag_tid=2, entry_tid=26009]]], state=connected] 2024-04-06T09:34:00.141 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:34:00.142 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=25, tag_tid=3, entry_tid=1' 2024-04-06T09:34:00.144 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:34:00.144 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:34:00.144 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:34:00.144 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:34:00.144 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:34:00.144 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:34:00.144 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:34:00.144 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:34:00.144 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:34:00.144 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:34:00.144 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:34:00.144 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:34:00.145 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:34:00.145 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:34:00.145 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:34:00.190 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 6 2024-04-06T09:34:00.190 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 6 2024-04-06T09:34:00.190 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:34:00.191 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=25, tag_tid=3, entry_tid=1], [object_number=24, tag_tid=3, entry_tid=0], [object_number=27, tag_tid=2, entry_tid=27223], [object_number=26, tag_tid=2, entry_tid=27222]]], state=connected] 2024-04-06T09:34:00.191 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=24, tag_tid=2, entry_tid=26012], [object_number=27, tag_tid=2, entry_tid=26011], [object_number=26, tag_tid=2, entry_tid=26010], [object_number=25, tag_tid=2, entry_tid=26009]]], state=connected] 2024-04-06T09:34:00.194 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:34:00.196 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=24, tag_tid=2, entry_tid=26012' 2024-04-06T09:34:00.196 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=25, tag_tid=3, entry_tid=1' -a 'object_number=25, tag_tid=3, entry_tid=1' = 'object_number=24, tag_tid=2, entry_tid=26012' 2024-04-06T09:34:00.196 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=24, tag_tid=2, entry_tid=26012' '!=' 'object_number=27, tag_tid=2, entry_tid=25919' 2024-04-06T09:34:00.196 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:34:00.196 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=24, tag_tid=2, entry_tid=26012' = 'object_number=27, tag_tid=2, entry_tid=25919' 2024-04-06T09:34:00.196 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=24, tag_tid=2, entry_tid=26012' 2024-04-06T09:34:00.197 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=25, tag_tid=3, entry_tid=1' 2024-04-06T09:34:00.197 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:34:00.197 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:34:00.197 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=3 2024-04-06T09:34:00.198 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=24, tag_tid=2, entry_tid=26012' 2024-04-06T09:34:00.198 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:34:00.198 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:34:00.199 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=2 2024-04-06T09:34:00.202 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=25, tag_tid=3, entry_tid=1' 2024-04-06T09:34:00.202 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:34:00.203 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:34:00.203 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:34:00.203 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=24, tag_tid=2, entry_tid=26012' 2024-04-06T09:34:00.203 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:34:00.203 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:34:00.203 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=26012 2024-04-06T09:34:00.203 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 3 = 2 -a 1 -le 26012 2024-04-06T09:34:00.203 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:34:00.203 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:34:00.203 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:34:00.404 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:34:00.404 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:34:00.405 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:34:00.405 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:34:00.405 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:34:00.405 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:34:00.405 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:34:00.405 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:34:00.405 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:34:00.405 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:34:00.405 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:34:00.405 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:34:00.405 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:34:00.405 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:34:00.405 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:34:00.406 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:34:00.407 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:34:00.409 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:34:00.409 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:34:00.409 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:34:00.409 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:34:00.409 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:34:00.409 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:34:00.409 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:34:00.409 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:34:00.409 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:34:00.409 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:34:00.409 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:34:00.409 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:34:00.410 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:34:00.410 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:34:00.410 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:34:00.410 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:34:00.411 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:34:00.411 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:34:00.411 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:34:00.411 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:34:00.411 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:34:00.411 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:34:00.411 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:34:00.411 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:34:00.411 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:34:00.411 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:34:00.411 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:34:00.411 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:34:00.411 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:34:00.411 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:34:00.412 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:34:00.412 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:34:00.435 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:34:00.436 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:34:00.436 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:34:00.436 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:34:00.570 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:34:00.572 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:34:00.572 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:34:00.572 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:34:00.572 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:34:00.572 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:34:00.572 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:34:00.572 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:34:00.572 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:34:00.573 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:34:00.573 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:34:00.573 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:34:00.573 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:34:00.574 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:34:00.575 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:34:00.575 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:34:00.619 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 6 2024-04-06T09:34:00.619 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 6 2024-04-06T09:34:00.619 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:34:00.619 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=25, tag_tid=3, entry_tid=1], [object_number=24, tag_tid=3, entry_tid=0], [object_number=27, tag_tid=2, entry_tid=27223], [object_number=26, tag_tid=2, entry_tid=27222]]], state=connected] 2024-04-06T09:34:00.619 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=27, tag_tid=2, entry_tid=26107], [object_number=26, tag_tid=2, entry_tid=26106], [object_number=25, tag_tid=2, entry_tid=26105], [object_number=24, tag_tid=2, entry_tid=26104]]], state=connected] 2024-04-06T09:34:00.622 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:34:00.623 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=25, tag_tid=3, entry_tid=1' 2024-04-06T09:34:00.625 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:34:00.625 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:34:00.625 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:34:00.625 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:34:00.625 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:34:00.625 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:34:00.625 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:34:00.625 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:34:00.625 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:34:00.625 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:34:00.625 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:34:00.625 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:34:00.626 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:34:00.626 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:34:00.626 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:34:00.671 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 6 2024-04-06T09:34:00.671 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 6 2024-04-06T09:34:00.671 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:34:00.671 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=25, tag_tid=3, entry_tid=1], [object_number=24, tag_tid=3, entry_tid=0], [object_number=27, tag_tid=2, entry_tid=27223], [object_number=26, tag_tid=2, entry_tid=27222]]], state=connected] 2024-04-06T09:34:00.671 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=27, tag_tid=2, entry_tid=26107], [object_number=26, tag_tid=2, entry_tid=26106], [object_number=25, tag_tid=2, entry_tid=26105], [object_number=24, tag_tid=2, entry_tid=26104]]], state=connected] 2024-04-06T09:34:00.674 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:34:00.676 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=27, tag_tid=2, entry_tid=26107' 2024-04-06T09:34:00.676 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=25, tag_tid=3, entry_tid=1' -a 'object_number=25, tag_tid=3, entry_tid=1' = 'object_number=27, tag_tid=2, entry_tid=26107' 2024-04-06T09:34:00.676 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=27, tag_tid=2, entry_tid=26107' '!=' 'object_number=24, tag_tid=2, entry_tid=26012' 2024-04-06T09:34:00.676 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:34:00.677 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=27, tag_tid=2, entry_tid=26107' = 'object_number=24, tag_tid=2, entry_tid=26012' 2024-04-06T09:34:00.677 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=27, tag_tid=2, entry_tid=26107' 2024-04-06T09:34:00.677 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=25, tag_tid=3, entry_tid=1' 2024-04-06T09:34:00.677 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:34:00.677 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:34:00.677 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=3 2024-04-06T09:34:00.678 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=27, tag_tid=2, entry_tid=26107' 2024-04-06T09:34:00.678 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:34:00.678 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:34:00.679 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=2 2024-04-06T09:34:00.680 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=25, tag_tid=3, entry_tid=1' 2024-04-06T09:34:00.680 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:34:00.680 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:34:00.681 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:34:00.681 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=27, tag_tid=2, entry_tid=26107' 2024-04-06T09:34:00.681 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:34:00.681 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:34:00.683 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=26107 2024-04-06T09:34:00.683 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 3 = 2 -a 1 -le 26107 2024-04-06T09:34:00.683 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:34:00.683 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:34:00.683 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:34:00.885 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:34:00.885 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:34:00.885 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:34:00.885 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:34:00.885 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:34:00.886 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:34:00.886 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:34:00.886 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:34:00.886 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:34:00.886 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:34:00.886 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:34:00.886 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:34:00.886 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:34:00.886 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:34:00.886 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:34:00.889 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:34:00.889 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:34:00.889 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:34:00.889 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:34:00.889 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:34:00.889 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:34:00.889 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:34:00.890 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:34:00.890 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:34:00.890 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:34:00.890 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:34:00.890 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:34:00.890 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:34:00.890 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:34:00.890 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:34:00.890 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:34:00.890 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:34:00.890 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:34:00.890 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:34:00.890 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:34:00.890 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:34:00.891 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:34:00.891 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:34:00.891 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:34:00.891 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:34:00.891 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:34:00.891 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:34:00.891 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:34:00.891 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:34:00.891 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:34:00.891 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:34:00.891 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:34:00.891 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:34:00.891 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:34:00.914 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:34:00.914 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:34:00.914 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:34:00.914 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:34:01.039 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:34:01.041 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:34:01.041 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:34:01.041 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:34:01.041 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:34:01.041 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:34:01.041 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:34:01.041 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:34:01.041 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:34:01.041 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:34:01.041 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:34:01.042 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:34:01.042 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:34:01.043 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:34:01.044 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:34:01.044 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:34:01.095 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 6 2024-04-06T09:34:01.095 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 6 2024-04-06T09:34:01.095 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:34:01.095 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=25, tag_tid=3, entry_tid=1], [object_number=24, tag_tid=3, entry_tid=0], [object_number=27, tag_tid=2, entry_tid=27223], [object_number=26, tag_tid=2, entry_tid=27222]]], state=connected] 2024-04-06T09:34:01.095 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=26, tag_tid=2, entry_tid=26206], [object_number=25, tag_tid=2, entry_tid=26205], [object_number=24, tag_tid=2, entry_tid=26204], [object_number=27, tag_tid=2, entry_tid=26203]]], state=connected] 2024-04-06T09:34:01.099 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:34:01.100 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=25, tag_tid=3, entry_tid=1' 2024-04-06T09:34:01.101 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:34:01.101 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:34:01.101 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:34:01.101 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:34:01.101 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:34:01.101 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:34:01.102 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:34:01.102 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:34:01.102 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:34:01.102 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:34:01.102 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:34:01.102 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:34:01.104 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:34:01.104 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:34:01.104 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:34:01.354 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 6 2024-04-06T09:34:01.354 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 6 2024-04-06T09:34:01.354 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:34:01.354 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=25, tag_tid=3, entry_tid=1], [object_number=24, tag_tid=3, entry_tid=0], [object_number=27, tag_tid=2, entry_tid=27223], [object_number=26, tag_tid=2, entry_tid=27222]]], state=connected] 2024-04-06T09:34:01.354 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=26, tag_tid=2, entry_tid=26206], [object_number=25, tag_tid=2, entry_tid=26205], [object_number=24, tag_tid=2, entry_tid=26204], [object_number=27, tag_tid=2, entry_tid=26203]]], state=connected] 2024-04-06T09:34:01.360 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:34:01.362 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=26, tag_tid=2, entry_tid=26206' 2024-04-06T09:34:01.362 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=25, tag_tid=3, entry_tid=1' -a 'object_number=25, tag_tid=3, entry_tid=1' = 'object_number=26, tag_tid=2, entry_tid=26206' 2024-04-06T09:34:01.362 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=26, tag_tid=2, entry_tid=26206' '!=' 'object_number=27, tag_tid=2, entry_tid=26107' 2024-04-06T09:34:01.362 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:34:01.362 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=26, tag_tid=2, entry_tid=26206' = 'object_number=27, tag_tid=2, entry_tid=26107' 2024-04-06T09:34:01.362 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=26, tag_tid=2, entry_tid=26206' 2024-04-06T09:34:01.363 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=25, tag_tid=3, entry_tid=1' 2024-04-06T09:34:01.363 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:34:01.363 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:34:01.364 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=3 2024-04-06T09:34:01.365 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=26, tag_tid=2, entry_tid=26206' 2024-04-06T09:34:01.365 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:34:01.365 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:34:01.367 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=2 2024-04-06T09:34:01.367 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=25, tag_tid=3, entry_tid=1' 2024-04-06T09:34:01.368 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:34:01.368 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:34:01.369 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:34:01.370 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=26, tag_tid=2, entry_tid=26206' 2024-04-06T09:34:01.370 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:34:01.371 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:34:01.372 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=26206 2024-04-06T09:34:01.372 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 3 = 2 -a 1 -le 26206 2024-04-06T09:34:01.372 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:34:01.372 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:34:01.372 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:34:01.574 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:34:01.574 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:34:01.574 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:34:01.574 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:34:01.574 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:34:01.574 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:34:01.575 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:34:01.575 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:34:01.575 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:34:01.575 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:34:01.575 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:34:01.575 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:34:01.575 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:34:01.575 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:34:01.575 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:34:01.576 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:34:01.576 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:34:01.578 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:34:01.578 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:34:01.578 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:34:01.578 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:34:01.578 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:34:01.579 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:34:01.579 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:34:01.579 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:34:01.579 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:34:01.579 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:34:01.579 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:34:01.579 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:34:01.579 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:34:01.579 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:34:01.579 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:34:01.579 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:34:01.580 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:34:01.580 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:34:01.580 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:34:01.581 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:34:01.581 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:34:01.581 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:34:01.581 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:34:01.581 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:34:01.581 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:34:01.581 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:34:01.581 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:34:01.581 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:34:01.581 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:34:01.581 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:34:01.581 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:34:01.581 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:34:01.605 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:34:01.605 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:34:01.605 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:34:01.605 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:34:01.733 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:34:01.734 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:34:01.734 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:34:01.735 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:34:01.735 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:34:01.735 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:34:01.735 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:34:01.735 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:34:01.735 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:34:01.735 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:34:01.735 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:34:01.736 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:34:01.736 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:34:01.737 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:34:01.737 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:34:01.737 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:34:01.791 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 6 2024-04-06T09:34:01.791 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 6 2024-04-06T09:34:01.791 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:34:01.792 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=25, tag_tid=3, entry_tid=1], [object_number=24, tag_tid=3, entry_tid=0], [object_number=27, tag_tid=2, entry_tid=27223], [object_number=26, tag_tid=2, entry_tid=27222]]], state=connected] 2024-04-06T09:34:01.792 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=27, tag_tid=2, entry_tid=26371], [object_number=26, tag_tid=2, entry_tid=26370], [object_number=25, tag_tid=2, entry_tid=26369], [object_number=24, tag_tid=2, entry_tid=26368]]], state=connected] 2024-04-06T09:34:01.795 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:34:01.797 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=25, tag_tid=3, entry_tid=1' 2024-04-06T09:34:01.797 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:34:01.798 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:34:01.798 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:34:01.798 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:34:01.798 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:34:01.798 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:34:01.798 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:34:01.798 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:34:01.798 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:34:01.798 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:34:01.798 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:34:01.798 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:34:01.799 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:34:01.799 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:34:01.799 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:34:01.845 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 6 2024-04-06T09:34:01.845 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 6 2024-04-06T09:34:01.845 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:34:01.845 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=25, tag_tid=3, entry_tid=1], [object_number=24, tag_tid=3, entry_tid=0], [object_number=27, tag_tid=2, entry_tid=27223], [object_number=26, tag_tid=2, entry_tid=27222]]], state=connected] 2024-04-06T09:34:01.845 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=27, tag_tid=2, entry_tid=26371], [object_number=26, tag_tid=2, entry_tid=26370], [object_number=25, tag_tid=2, entry_tid=26369], [object_number=24, tag_tid=2, entry_tid=26368]]], state=connected] 2024-04-06T09:34:01.849 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:34:01.851 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=27, tag_tid=2, entry_tid=26371' 2024-04-06T09:34:01.851 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=25, tag_tid=3, entry_tid=1' -a 'object_number=25, tag_tid=3, entry_tid=1' = 'object_number=27, tag_tid=2, entry_tid=26371' 2024-04-06T09:34:01.851 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=27, tag_tid=2, entry_tid=26371' '!=' 'object_number=26, tag_tid=2, entry_tid=26206' 2024-04-06T09:34:01.851 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:34:01.851 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=27, tag_tid=2, entry_tid=26371' = 'object_number=26, tag_tid=2, entry_tid=26206' 2024-04-06T09:34:01.851 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=27, tag_tid=2, entry_tid=26371' 2024-04-06T09:34:01.851 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=25, tag_tid=3, entry_tid=1' 2024-04-06T09:34:01.851 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:34:01.851 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:34:01.852 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=3 2024-04-06T09:34:01.853 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=27, tag_tid=2, entry_tid=26371' 2024-04-06T09:34:01.853 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:34:01.853 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:34:01.853 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=2 2024-04-06T09:34:01.854 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=25, tag_tid=3, entry_tid=1' 2024-04-06T09:34:01.854 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:34:01.854 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:34:01.855 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:34:01.856 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=27, tag_tid=2, entry_tid=26371' 2024-04-06T09:34:01.856 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:34:01.856 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:34:01.857 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=26371 2024-04-06T09:34:01.857 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 3 = 2 -a 1 -le 26371 2024-04-06T09:34:01.857 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:34:01.857 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:34:01.857 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:34:02.059 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:34:02.059 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:34:02.059 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:34:02.059 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:34:02.059 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:34:02.059 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:34:02.059 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:34:02.060 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:34:02.060 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:34:02.060 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:34:02.060 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:34:02.060 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:34:02.060 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:34:02.060 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:34:02.060 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:34:02.060 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:34:02.061 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:34:02.063 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:34:02.063 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:34:02.063 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:34:02.063 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:34:02.063 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:34:02.063 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:34:02.063 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:34:02.063 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:34:02.063 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:34:02.064 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:34:02.064 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:34:02.064 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:34:02.064 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:34:02.064 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:34:02.064 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:34:02.064 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:34:02.065 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:34:02.065 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:34:02.065 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:34:02.065 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:34:02.065 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:34:02.065 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:34:02.065 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:34:02.065 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:34:02.065 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:34:02.065 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:34:02.066 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:34:02.066 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:34:02.066 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:34:02.066 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:34:02.066 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:34:02.066 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:34:02.090 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:34:02.090 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:34:02.090 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:34:02.090 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:34:02.217 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:34:02.218 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:34:02.218 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:34:02.219 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:34:02.219 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:34:02.219 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:34:02.219 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:34:02.219 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:34:02.219 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:34:02.219 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:34:02.219 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:34:02.220 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:34:02.220 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:34:02.221 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:34:02.221 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:34:02.221 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:34:02.274 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 6 2024-04-06T09:34:02.274 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 6 2024-04-06T09:34:02.274 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:34:02.275 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=25, tag_tid=3, entry_tid=1], [object_number=24, tag_tid=3, entry_tid=0], [object_number=27, tag_tid=2, entry_tid=27223], [object_number=26, tag_tid=2, entry_tid=27222]]], state=connected] 2024-04-06T09:34:02.275 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=24, tag_tid=2, entry_tid=26488], [object_number=27, tag_tid=2, entry_tid=26487], [object_number=26, tag_tid=2, entry_tid=26486], [object_number=25, tag_tid=2, entry_tid=26485]]], state=connected] 2024-04-06T09:34:02.278 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:34:02.279 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=25, tag_tid=3, entry_tid=1' 2024-04-06T09:34:02.280 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:34:02.280 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:34:02.281 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:34:02.281 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:34:02.281 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:34:02.281 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:34:02.281 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:34:02.281 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:34:02.281 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:34:02.281 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:34:02.281 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:34:02.281 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:34:02.282 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:34:02.282 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:34:02.282 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:34:02.327 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 6 2024-04-06T09:34:02.327 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 6 2024-04-06T09:34:02.327 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:34:02.327 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=25, tag_tid=3, entry_tid=1], [object_number=24, tag_tid=3, entry_tid=0], [object_number=27, tag_tid=2, entry_tid=27223], [object_number=26, tag_tid=2, entry_tid=27222]]], state=connected] 2024-04-06T09:34:02.328 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=24, tag_tid=2, entry_tid=26488], [object_number=27, tag_tid=2, entry_tid=26487], [object_number=26, tag_tid=2, entry_tid=26486], [object_number=25, tag_tid=2, entry_tid=26485]]], state=connected] 2024-04-06T09:34:02.331 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:34:02.333 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=24, tag_tid=2, entry_tid=26488' 2024-04-06T09:34:02.333 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=25, tag_tid=3, entry_tid=1' -a 'object_number=25, tag_tid=3, entry_tid=1' = 'object_number=24, tag_tid=2, entry_tid=26488' 2024-04-06T09:34:02.333 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=24, tag_tid=2, entry_tid=26488' '!=' 'object_number=27, tag_tid=2, entry_tid=26371' 2024-04-06T09:34:02.333 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:34:02.333 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=24, tag_tid=2, entry_tid=26488' = 'object_number=27, tag_tid=2, entry_tid=26371' 2024-04-06T09:34:02.333 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=24, tag_tid=2, entry_tid=26488' 2024-04-06T09:34:02.333 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=25, tag_tid=3, entry_tid=1' 2024-04-06T09:34:02.334 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:34:02.334 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:34:02.334 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=3 2024-04-06T09:34:02.335 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=24, tag_tid=2, entry_tid=26488' 2024-04-06T09:34:02.335 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:34:02.335 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:34:02.336 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=2 2024-04-06T09:34:02.337 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=25, tag_tid=3, entry_tid=1' 2024-04-06T09:34:02.337 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:34:02.337 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:34:02.337 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:34:02.338 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=24, tag_tid=2, entry_tid=26488' 2024-04-06T09:34:02.338 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:34:02.338 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:34:02.339 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=26488 2024-04-06T09:34:02.339 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 3 = 2 -a 1 -le 26488 2024-04-06T09:34:02.340 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:34:02.340 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:34:02.340 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:34:02.375 INFO:tasks.rbd_mirror_thrash:waiting for cluster1.client.mirror.3 2024-04-06T09:34:02.375 INFO:teuthology.orchestra.run:waiting for 600 2024-04-06T09:34:02.376 INFO:tasks.rbd_mirror_thrash:reviving cluster1.client.mirror.3 2024-04-06T09:34:02.376 INFO:tasks.rbd_mirror.cluster1.client.mirror.3:Restarting daemon 2024-04-06T09:34:02.376 DEBUG:teuthology.orchestra.run.smithi192:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.3 2024-04-06T09:34:02.379 INFO:tasks.rbd_mirror.cluster1.client.mirror.3:Started 2024-04-06T09:34:02.379 INFO:tasks.rbd_mirror_thrash:waiting for 83 secs before thrashing 2024-04-06T09:34:02.542 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:34:02.542 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:34:02.542 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:34:02.542 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:34:02.542 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:34:02.542 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:34:02.542 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:34:02.542 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:34:02.542 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:34:02.542 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:34:02.543 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:34:02.543 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:34:02.543 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:34:02.543 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:34:02.543 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:34:02.543 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:34:02.543 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:34:02.546 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:34:02.546 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:34:02.546 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:34:02.546 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:34:02.546 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:34:02.546 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:34:02.546 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:34:02.546 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:34:02.546 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:34:02.546 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:34:02.546 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:34:02.546 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:34:02.547 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:34:02.547 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:34:02.547 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:34:02.547 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:34:02.547 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:34:02.547 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:34:02.547 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:34:02.547 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:34:02.547 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:34:02.547 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:34:02.547 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:34:02.547 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:34:02.547 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:34:02.547 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:34:02.547 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:34:02.548 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:34:02.548 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:34:02.548 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:34:02.548 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:34:02.548 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:34:02.566 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:34:02.566 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:34:02.566 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:34:02.566 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:34:02.692 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:34:02.693 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:34:02.693 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:34:02.693 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:34:02.693 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:34:02.693 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:34:02.693 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:34:02.693 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:34:02.693 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:34:02.693 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:34:02.693 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:34:02.694 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:34:02.694 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:34:02.695 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:34:02.695 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:34:02.695 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:34:02.739 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 6 2024-04-06T09:34:02.739 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 6 2024-04-06T09:34:02.739 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:34:02.739 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=25, tag_tid=3, entry_tid=1], [object_number=24, tag_tid=3, entry_tid=0], [object_number=27, tag_tid=2, entry_tid=27223], [object_number=26, tag_tid=2, entry_tid=27222]]], state=connected] 2024-04-06T09:34:02.739 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=24, tag_tid=2, entry_tid=26592], [object_number=27, tag_tid=2, entry_tid=26591], [object_number=26, tag_tid=2, entry_tid=26590], [object_number=25, tag_tid=2, entry_tid=26589]]], state=connected] 2024-04-06T09:34:02.742 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:34:02.744 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=25, tag_tid=3, entry_tid=1' 2024-04-06T09:34:02.745 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:34:02.745 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:34:02.745 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:34:02.745 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:34:02.745 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:34:02.745 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:34:02.745 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:34:02.745 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:34:02.745 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:34:02.745 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:34:02.745 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:34:02.746 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:34:02.746 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:34:02.746 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:34:02.746 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:34:02.791 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 6 2024-04-06T09:34:02.791 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 6 2024-04-06T09:34:02.791 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:34:02.791 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=25, tag_tid=3, entry_tid=1], [object_number=24, tag_tid=3, entry_tid=0], [object_number=27, tag_tid=2, entry_tid=27223], [object_number=26, tag_tid=2, entry_tid=27222]]], state=connected] 2024-04-06T09:34:02.791 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=24, tag_tid=2, entry_tid=26592], [object_number=27, tag_tid=2, entry_tid=26591], [object_number=26, tag_tid=2, entry_tid=26590], [object_number=25, tag_tid=2, entry_tid=26589]]], state=connected] 2024-04-06T09:34:02.794 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:34:02.796 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=24, tag_tid=2, entry_tid=26592' 2024-04-06T09:34:02.796 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=25, tag_tid=3, entry_tid=1' -a 'object_number=25, tag_tid=3, entry_tid=1' = 'object_number=24, tag_tid=2, entry_tid=26592' 2024-04-06T09:34:02.796 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=24, tag_tid=2, entry_tid=26592' '!=' 'object_number=24, tag_tid=2, entry_tid=26488' 2024-04-06T09:34:02.796 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:34:02.797 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=24, tag_tid=2, entry_tid=26592' = 'object_number=24, tag_tid=2, entry_tid=26488' 2024-04-06T09:34:02.797 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=24, tag_tid=2, entry_tid=26592' 2024-04-06T09:34:02.797 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=25, tag_tid=3, entry_tid=1' 2024-04-06T09:34:02.797 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:34:02.797 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:34:02.797 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=3 2024-04-06T09:34:02.798 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=24, tag_tid=2, entry_tid=26592' 2024-04-06T09:34:02.798 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:34:02.798 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:34:02.799 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=2 2024-04-06T09:34:02.800 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=25, tag_tid=3, entry_tid=1' 2024-04-06T09:34:02.800 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:34:02.800 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:34:02.801 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:34:02.801 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=24, tag_tid=2, entry_tid=26592' 2024-04-06T09:34:02.801 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:34:02.802 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:34:02.803 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=26592 2024-04-06T09:34:02.803 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 3 = 2 -a 1 -le 26592 2024-04-06T09:34:02.803 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:34:02.803 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:34:02.803 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:34:03.004 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:34:03.004 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:34:03.004 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:34:03.005 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:34:03.005 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:34:03.005 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:34:03.005 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:34:03.005 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:34:03.005 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:34:03.005 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:34:03.005 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:34:03.005 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:34:03.005 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:34:03.005 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:34:03.005 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:34:03.006 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:34:03.006 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:34:03.009 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:34:03.009 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:34:03.009 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:34:03.009 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:34:03.009 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:34:03.009 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:34:03.009 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:34:03.009 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:34:03.009 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:34:03.009 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:34:03.009 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:34:03.009 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:34:03.009 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:34:03.010 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:34:03.010 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:34:03.010 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:34:03.010 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:34:03.010 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:34:03.010 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:34:03.010 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:34:03.010 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:34:03.010 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:34:03.010 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:34:03.010 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:34:03.010 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:34:03.010 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:34:03.011 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:34:03.011 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:34:03.011 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:34:03.011 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:34:03.011 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:34:03.011 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:34:03.036 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:34:03.036 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:34:03.036 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:34:03.036 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:34:03.167 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:34:03.167 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:34:03.167 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:34:03.168 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:34:03.168 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:34:03.168 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:34:03.168 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:34:03.168 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:34:03.168 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:34:03.168 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:34:03.168 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:34:03.169 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:34:03.169 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:34:03.171 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:34:03.172 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:34:03.172 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:34:03.218 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 6 2024-04-06T09:34:03.218 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 6 2024-04-06T09:34:03.218 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:34:03.218 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=25, tag_tid=3, entry_tid=1], [object_number=24, tag_tid=3, entry_tid=0], [object_number=27, tag_tid=2, entry_tid=27223], [object_number=26, tag_tid=2, entry_tid=27222]]], state=connected] 2024-04-06T09:34:03.218 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=27, tag_tid=2, entry_tid=26695], [object_number=26, tag_tid=2, entry_tid=26694], [object_number=25, tag_tid=2, entry_tid=26693], [object_number=24, tag_tid=2, entry_tid=26692]]], state=connected] 2024-04-06T09:34:03.221 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:34:03.223 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=25, tag_tid=3, entry_tid=1' 2024-04-06T09:34:03.224 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:34:03.224 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:34:03.224 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:34:03.224 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:34:03.224 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:34:03.224 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:34:03.224 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:34:03.224 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:34:03.224 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:34:03.224 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:34:03.225 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:34:03.225 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:34:03.225 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:34:03.226 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:34:03.226 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:34:03.271 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 6 2024-04-06T09:34:03.271 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 6 2024-04-06T09:34:03.271 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:34:03.271 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=25, tag_tid=3, entry_tid=1], [object_number=24, tag_tid=3, entry_tid=0], [object_number=27, tag_tid=2, entry_tid=27223], [object_number=26, tag_tid=2, entry_tid=27222]]], state=connected] 2024-04-06T09:34:03.271 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=27, tag_tid=2, entry_tid=26695], [object_number=26, tag_tid=2, entry_tid=26694], [object_number=25, tag_tid=2, entry_tid=26693], [object_number=24, tag_tid=2, entry_tid=26692]]], state=connected] 2024-04-06T09:34:03.274 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:34:03.276 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=27, tag_tid=2, entry_tid=26695' 2024-04-06T09:34:03.276 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=25, tag_tid=3, entry_tid=1' -a 'object_number=25, tag_tid=3, entry_tid=1' = 'object_number=27, tag_tid=2, entry_tid=26695' 2024-04-06T09:34:03.277 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=27, tag_tid=2, entry_tid=26695' '!=' 'object_number=24, tag_tid=2, entry_tid=26592' 2024-04-06T09:34:03.277 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:34:03.277 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=27, tag_tid=2, entry_tid=26695' = 'object_number=24, tag_tid=2, entry_tid=26592' 2024-04-06T09:34:03.277 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=27, tag_tid=2, entry_tid=26695' 2024-04-06T09:34:03.277 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=25, tag_tid=3, entry_tid=1' 2024-04-06T09:34:03.277 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:34:03.277 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:34:03.277 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=3 2024-04-06T09:34:03.278 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=27, tag_tid=2, entry_tid=26695' 2024-04-06T09:34:03.278 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:34:03.278 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:34:03.279 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=2 2024-04-06T09:34:03.280 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=25, tag_tid=3, entry_tid=1' 2024-04-06T09:34:03.280 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:34:03.280 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:34:03.281 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:34:03.281 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=27, tag_tid=2, entry_tid=26695' 2024-04-06T09:34:03.281 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:34:03.282 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:34:03.282 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=26695 2024-04-06T09:34:03.283 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 3 = 2 -a 1 -le 26695 2024-04-06T09:34:03.283 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:34:03.283 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:34:03.283 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:34:03.484 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:34:03.485 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:34:03.485 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:34:03.485 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:34:03.485 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:34:03.485 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:34:03.485 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:34:03.485 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:34:03.485 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:34:03.485 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:34:03.485 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:34:03.485 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:34:03.485 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:34:03.485 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:34:03.485 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:34:03.486 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:34:03.486 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:34:03.490 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:34:03.490 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:34:03.490 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:34:03.490 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:34:03.490 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:34:03.490 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:34:03.490 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:34:03.491 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:34:03.491 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:34:03.491 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:34:03.491 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:34:03.491 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:34:03.491 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:34:03.491 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:34:03.491 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:34:03.491 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:34:03.491 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:34:03.491 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:34:03.491 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:34:03.491 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:34:03.491 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:34:03.491 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:34:03.492 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:34:03.492 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:34:03.492 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:34:03.492 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:34:03.492 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:34:03.492 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:34:03.492 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:34:03.492 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:34:03.492 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:34:03.492 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:34:03.517 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:34:03.517 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:34:03.517 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:34:03.517 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:34:03.646 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:34:03.647 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:34:03.647 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:34:03.648 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:34:03.648 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:34:03.648 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:34:03.648 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:34:03.648 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:34:03.648 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:34:03.648 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:34:03.648 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:34:03.649 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:34:03.649 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:34:03.649 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:34:03.650 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:34:03.650 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:34:03.701 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 6 2024-04-06T09:34:03.701 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 6 2024-04-06T09:34:03.701 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:34:03.701 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=25, tag_tid=3, entry_tid=1], [object_number=24, tag_tid=3, entry_tid=0], [object_number=27, tag_tid=2, entry_tid=27223], [object_number=26, tag_tid=2, entry_tid=27222]]], state=connected] 2024-04-06T09:34:03.701 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=24, tag_tid=2, entry_tid=26800], [object_number=27, tag_tid=2, entry_tid=26799], [object_number=26, tag_tid=2, entry_tid=26798], [object_number=25, tag_tid=2, entry_tid=26797]]], state=connected] 2024-04-06T09:34:03.704 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:34:03.706 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=25, tag_tid=3, entry_tid=1' 2024-04-06T09:34:03.707 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:34:03.707 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:34:03.707 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:34:03.707 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:34:03.707 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:34:03.707 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:34:03.707 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:34:03.707 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:34:03.707 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:34:03.707 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:34:03.707 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:34:03.707 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:34:03.708 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:34:03.708 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:34:03.708 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:34:03.753 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 6 2024-04-06T09:34:03.753 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 6 2024-04-06T09:34:03.753 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:34:03.753 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=25, tag_tid=3, entry_tid=1], [object_number=24, tag_tid=3, entry_tid=0], [object_number=27, tag_tid=2, entry_tid=27223], [object_number=26, tag_tid=2, entry_tid=27222]]], state=connected] 2024-04-06T09:34:03.753 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=24, tag_tid=2, entry_tid=26800], [object_number=27, tag_tid=2, entry_tid=26799], [object_number=26, tag_tid=2, entry_tid=26798], [object_number=25, tag_tid=2, entry_tid=26797]]], state=connected] 2024-04-06T09:34:03.756 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:34:03.758 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=24, tag_tid=2, entry_tid=26800' 2024-04-06T09:34:03.758 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=25, tag_tid=3, entry_tid=1' -a 'object_number=25, tag_tid=3, entry_tid=1' = 'object_number=24, tag_tid=2, entry_tid=26800' 2024-04-06T09:34:03.758 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=24, tag_tid=2, entry_tid=26800' '!=' 'object_number=27, tag_tid=2, entry_tid=26695' 2024-04-06T09:34:03.758 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:34:03.758 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=24, tag_tid=2, entry_tid=26800' = 'object_number=27, tag_tid=2, entry_tid=26695' 2024-04-06T09:34:03.758 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=24, tag_tid=2, entry_tid=26800' 2024-04-06T09:34:03.759 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=25, tag_tid=3, entry_tid=1' 2024-04-06T09:34:03.759 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:34:03.759 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:34:03.759 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=3 2024-04-06T09:34:03.760 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=24, tag_tid=2, entry_tid=26800' 2024-04-06T09:34:03.760 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:34:03.760 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:34:03.761 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=2 2024-04-06T09:34:03.762 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=25, tag_tid=3, entry_tid=1' 2024-04-06T09:34:03.762 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:34:03.762 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:34:03.763 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:34:03.763 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=24, tag_tid=2, entry_tid=26800' 2024-04-06T09:34:03.764 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:34:03.764 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:34:03.765 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=26800 2024-04-06T09:34:03.765 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 3 = 2 -a 1 -le 26800 2024-04-06T09:34:03.765 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:34:03.765 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:34:03.765 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:34:03.967 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:34:03.967 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:34:03.967 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:34:03.967 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:34:03.967 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:34:03.967 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:34:03.967 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:34:03.967 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:34:03.967 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:34:03.967 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:34:03.968 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:34:03.968 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:34:03.968 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:34:03.968 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:34:03.968 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:34:03.968 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:34:03.968 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:34:03.971 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:34:03.971 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:34:03.972 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:34:03.972 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:34:03.972 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:34:03.972 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:34:03.972 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:34:03.972 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:34:03.972 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:34:03.972 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:34:03.972 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:34:03.972 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:34:03.972 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:34:03.972 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:34:03.972 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:34:03.972 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:34:03.972 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:34:03.973 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:34:03.973 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:34:03.973 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:34:03.973 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:34:03.973 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:34:03.973 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:34:03.973 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:34:03.973 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:34:03.973 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:34:03.973 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:34:03.973 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:34:03.973 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:34:03.973 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:34:03.973 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:34:03.974 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:34:03.996 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:34:03.997 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:34:03.997 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:34:03.997 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:34:04.129 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:34:04.130 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:34:04.130 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:34:04.130 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:34:04.130 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:34:04.131 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:34:04.131 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:34:04.131 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:34:04.131 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:34:04.131 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:34:04.131 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:34:04.131 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:34:04.132 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:34:04.134 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:34:04.134 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:34:04.134 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:34:04.185 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 6 2024-04-06T09:34:04.185 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 6 2024-04-06T09:34:04.185 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:34:04.185 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=25, tag_tid=3, entry_tid=1], [object_number=24, tag_tid=3, entry_tid=0], [object_number=27, tag_tid=2, entry_tid=27223], [object_number=26, tag_tid=2, entry_tid=27222]]], state=connected] 2024-04-06T09:34:04.185 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=26, tag_tid=2, entry_tid=26906], [object_number=25, tag_tid=2, entry_tid=26905], [object_number=24, tag_tid=2, entry_tid=26904], [object_number=27, tag_tid=2, entry_tid=26903]]], state=connected] 2024-04-06T09:34:04.188 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:34:04.190 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=25, tag_tid=3, entry_tid=1' 2024-04-06T09:34:04.191 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:34:04.191 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:34:04.191 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:34:04.191 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:34:04.192 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:34:04.192 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:34:04.192 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:34:04.192 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:34:04.192 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:34:04.192 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:34:04.192 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:34:04.192 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:34:04.193 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:34:04.193 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:34:04.193 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:34:04.237 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 6 2024-04-06T09:34:04.237 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 6 2024-04-06T09:34:04.237 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:34:04.238 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=25, tag_tid=3, entry_tid=1], [object_number=24, tag_tid=3, entry_tid=0], [object_number=27, tag_tid=2, entry_tid=27223], [object_number=26, tag_tid=2, entry_tid=27222]]], state=connected] 2024-04-06T09:34:04.238 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=26, tag_tid=2, entry_tid=26906], [object_number=25, tag_tid=2, entry_tid=26905], [object_number=24, tag_tid=2, entry_tid=26904], [object_number=27, tag_tid=2, entry_tid=26903]]], state=connected] 2024-04-06T09:34:04.241 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:34:04.243 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=26, tag_tid=2, entry_tid=26906' 2024-04-06T09:34:04.243 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=25, tag_tid=3, entry_tid=1' -a 'object_number=25, tag_tid=3, entry_tid=1' = 'object_number=26, tag_tid=2, entry_tid=26906' 2024-04-06T09:34:04.243 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=26, tag_tid=2, entry_tid=26906' '!=' 'object_number=24, tag_tid=2, entry_tid=26800' 2024-04-06T09:34:04.243 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:34:04.243 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=26, tag_tid=2, entry_tid=26906' = 'object_number=24, tag_tid=2, entry_tid=26800' 2024-04-06T09:34:04.243 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=26, tag_tid=2, entry_tid=26906' 2024-04-06T09:34:04.243 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=25, tag_tid=3, entry_tid=1' 2024-04-06T09:34:04.243 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:34:04.243 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:34:04.244 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=3 2024-04-06T09:34:04.245 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=26, tag_tid=2, entry_tid=26906' 2024-04-06T09:34:04.245 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:34:04.245 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:34:04.246 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=2 2024-04-06T09:34:04.246 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=25, tag_tid=3, entry_tid=1' 2024-04-06T09:34:04.246 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:34:04.246 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:34:04.247 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:34:04.248 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=26, tag_tid=2, entry_tid=26906' 2024-04-06T09:34:04.248 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:34:04.248 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:34:04.249 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=26906 2024-04-06T09:34:04.249 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 3 = 2 -a 1 -le 26906 2024-04-06T09:34:04.249 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:34:04.249 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:34:04.250 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:34:04.451 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:34:04.452 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:34:04.452 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:34:04.452 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:34:04.452 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:34:04.452 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:34:04.452 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:34:04.452 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:34:04.452 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:34:04.452 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:34:04.452 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:34:04.452 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:34:04.452 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:34:04.453 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:34:04.453 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:34:04.456 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:34:04.456 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:34:04.456 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:34:04.456 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:34:04.456 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:34:04.456 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:34:04.457 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:34:04.457 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:34:04.457 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:34:04.457 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:34:04.457 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:34:04.457 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:34:04.457 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:34:04.457 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:34:04.457 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:34:04.457 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:34:04.457 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:34:04.457 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:34:04.457 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:34:04.457 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:34:04.458 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:34:04.458 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:34:04.458 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:34:04.458 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:34:04.458 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:34:04.458 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:34:04.458 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:34:04.458 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:34:04.458 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:34:04.458 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:34:04.458 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:34:04.458 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:34:04.458 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:34:04.458 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:34:04.481 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:34:04.481 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:34:04.481 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:34:04.481 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:34:04.618 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:34:04.619 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:34:04.620 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:34:04.620 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:34:04.620 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:34:04.620 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:34:04.620 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:34:04.620 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:34:04.620 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:34:04.620 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:34:04.620 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:34:04.621 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:34:04.621 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:34:04.622 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:34:04.622 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:34:04.622 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:34:04.672 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 6 2024-04-06T09:34:04.672 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 6 2024-04-06T09:34:04.673 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:34:04.673 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=25, tag_tid=3, entry_tid=1], [object_number=24, tag_tid=3, entry_tid=0], [object_number=27, tag_tid=2, entry_tid=27223], [object_number=26, tag_tid=2, entry_tid=27222]]], state=connected] 2024-04-06T09:34:04.673 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=25, tag_tid=2, entry_tid=27013], [object_number=24, tag_tid=2, entry_tid=27012], [object_number=27, tag_tid=2, entry_tid=27011], [object_number=26, tag_tid=2, entry_tid=27010]]], state=connected] 2024-04-06T09:34:04.676 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:34:04.677 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=25, tag_tid=3, entry_tid=1' 2024-04-06T09:34:04.679 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:34:04.679 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:34:04.679 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:34:04.679 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:34:04.679 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:34:04.679 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:34:04.679 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:34:04.679 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:34:04.679 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:34:04.679 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:34:04.679 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:34:04.679 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:34:04.680 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:34:04.680 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:34:04.680 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:34:04.725 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 6 2024-04-06T09:34:04.725 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 6 2024-04-06T09:34:04.726 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:34:04.726 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=25, tag_tid=3, entry_tid=1], [object_number=24, tag_tid=3, entry_tid=0], [object_number=27, tag_tid=2, entry_tid=27223], [object_number=26, tag_tid=2, entry_tid=27222]]], state=connected] 2024-04-06T09:34:04.726 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=25, tag_tid=2, entry_tid=27013], [object_number=24, tag_tid=2, entry_tid=27012], [object_number=27, tag_tid=2, entry_tid=27011], [object_number=26, tag_tid=2, entry_tid=27010]]], state=connected] 2024-04-06T09:34:04.729 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:34:04.731 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=25, tag_tid=2, entry_tid=27013' 2024-04-06T09:34:04.731 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=25, tag_tid=3, entry_tid=1' -a 'object_number=25, tag_tid=3, entry_tid=1' = 'object_number=25, tag_tid=2, entry_tid=27013' 2024-04-06T09:34:04.731 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=25, tag_tid=2, entry_tid=27013' '!=' 'object_number=26, tag_tid=2, entry_tid=26906' 2024-04-06T09:34:04.731 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:34:04.731 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=25, tag_tid=2, entry_tid=27013' = 'object_number=26, tag_tid=2, entry_tid=26906' 2024-04-06T09:34:04.731 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=25, tag_tid=2, entry_tid=27013' 2024-04-06T09:34:04.731 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=25, tag_tid=3, entry_tid=1' 2024-04-06T09:34:04.731 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:34:04.731 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:34:04.732 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=3 2024-04-06T09:34:04.732 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=25, tag_tid=2, entry_tid=27013' 2024-04-06T09:34:04.732 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:34:04.733 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:34:04.734 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=2 2024-04-06T09:34:04.734 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=25, tag_tid=3, entry_tid=1' 2024-04-06T09:34:04.734 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:34:04.734 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:34:04.735 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:34:04.736 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=25, tag_tid=2, entry_tid=27013' 2024-04-06T09:34:04.736 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:34:04.736 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:34:04.737 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=27013 2024-04-06T09:34:04.737 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 3 = 2 -a 1 -le 27013 2024-04-06T09:34:04.737 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:34:04.737 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:34:04.738 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:34:04.939 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:34:04.939 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:34:04.939 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:34:04.939 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:34:04.939 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:34:04.939 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:34:04.939 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:34:04.939 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:34:04.939 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:34:04.939 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:34:04.940 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:34:04.940 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:34:04.940 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:34:04.940 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:34:04.940 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:34:04.940 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:34:04.940 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:34:04.943 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:34:04.944 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:34:04.944 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:34:04.944 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:34:04.944 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:34:04.944 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:34:04.944 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:34:04.944 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:34:04.944 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:34:04.944 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:34:04.944 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:34:04.944 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:34:04.944 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:34:04.944 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:34:04.945 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:34:04.945 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:34:04.945 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:34:04.945 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:34:04.945 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:34:04.945 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:34:04.945 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:34:04.945 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:34:04.945 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:34:04.945 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:34:04.945 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:34:04.945 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:34:04.945 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:34:04.945 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:34:04.945 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:34:04.945 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:34:04.946 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:34:04.946 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:34:04.969 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:34:04.969 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:34:04.969 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:34:04.969 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:34:05.097 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:34:05.099 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:34:05.099 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:34:05.099 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:34:05.099 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:34:05.099 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:34:05.099 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:34:05.099 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:34:05.099 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:34:05.099 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:34:05.099 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:34:05.100 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:34:05.100 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:34:05.102 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:34:05.102 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:34:05.102 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:34:05.157 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 6 2024-04-06T09:34:05.157 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 6 2024-04-06T09:34:05.157 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:34:05.157 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=25, tag_tid=3, entry_tid=1], [object_number=24, tag_tid=3, entry_tid=0], [object_number=27, tag_tid=2, entry_tid=27223], [object_number=26, tag_tid=2, entry_tid=27222]]], state=connected] 2024-04-06T09:34:05.157 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=26, tag_tid=2, entry_tid=27126], [object_number=25, tag_tid=2, entry_tid=27125], [object_number=24, tag_tid=2, entry_tid=27124], [object_number=27, tag_tid=2, entry_tid=27123]]], state=connected] 2024-04-06T09:34:05.160 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:34:05.161 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=25, tag_tid=3, entry_tid=1' 2024-04-06T09:34:05.163 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:34:05.163 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:34:05.163 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:34:05.163 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:34:05.163 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:34:05.163 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:34:05.163 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:34:05.163 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:34:05.163 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:34:05.164 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:34:05.164 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:34:05.164 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:34:05.164 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:34:05.165 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:34:05.165 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:34:05.210 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 6 2024-04-06T09:34:05.210 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 6 2024-04-06T09:34:05.210 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:34:05.210 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=25, tag_tid=3, entry_tid=1], [object_number=24, tag_tid=3, entry_tid=0], [object_number=27, tag_tid=2, entry_tid=27223], [object_number=26, tag_tid=2, entry_tid=27222]]], state=connected] 2024-04-06T09:34:05.210 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=26, tag_tid=2, entry_tid=27126], [object_number=25, tag_tid=2, entry_tid=27125], [object_number=24, tag_tid=2, entry_tid=27124], [object_number=27, tag_tid=2, entry_tid=27123]]], state=connected] 2024-04-06T09:34:05.213 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:34:05.215 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=26, tag_tid=2, entry_tid=27126' 2024-04-06T09:34:05.215 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=25, tag_tid=3, entry_tid=1' -a 'object_number=25, tag_tid=3, entry_tid=1' = 'object_number=26, tag_tid=2, entry_tid=27126' 2024-04-06T09:34:05.215 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=26, tag_tid=2, entry_tid=27126' '!=' 'object_number=25, tag_tid=2, entry_tid=27013' 2024-04-06T09:34:05.215 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:34:05.215 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=26, tag_tid=2, entry_tid=27126' = 'object_number=25, tag_tid=2, entry_tid=27013' 2024-04-06T09:34:05.215 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=26, tag_tid=2, entry_tid=27126' 2024-04-06T09:34:05.215 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=25, tag_tid=3, entry_tid=1' 2024-04-06T09:34:05.215 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:34:05.215 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:34:05.216 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=3 2024-04-06T09:34:05.216 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=26, tag_tid=2, entry_tid=27126' 2024-04-06T09:34:05.217 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:34:05.217 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:34:05.217 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=2 2024-04-06T09:34:05.218 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=25, tag_tid=3, entry_tid=1' 2024-04-06T09:34:05.218 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:34:05.218 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:34:05.219 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:34:05.220 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=26, tag_tid=2, entry_tid=27126' 2024-04-06T09:34:05.220 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:34:05.220 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:34:05.221 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=27126 2024-04-06T09:34:05.221 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 3 = 2 -a 1 -le 27126 2024-04-06T09:34:05.222 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:34:05.222 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:34:05.222 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:34:05.424 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:34:05.424 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:34:05.424 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:34:05.424 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:34:05.424 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:34:05.424 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:34:05.424 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:34:05.424 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:34:05.425 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:34:05.425 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:34:05.425 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:34:05.425 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:34:05.425 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:34:05.425 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:34:05.425 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:34:05.425 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:34:05.426 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:34:05.427 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:34:05.428 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:34:05.428 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:34:05.428 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:34:05.428 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:34:05.428 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:34:05.428 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:34:05.428 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:34:05.428 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:34:05.428 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:34:05.428 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:34:05.428 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:34:05.428 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:34:05.428 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:34:05.429 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:34:05.429 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:34:05.430 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:34:05.430 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:34:05.430 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:34:05.430 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:34:05.430 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:34:05.430 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:34:05.430 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:34:05.430 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:34:05.430 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:34:05.430 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:34:05.430 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:34:05.430 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:34:05.430 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:34:05.431 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:34:05.431 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:34:05.431 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:34:05.455 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:34:05.455 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:34:05.455 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:34:05.455 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:34:05.580 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:34:05.581 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:34:05.582 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:34:05.582 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:34:05.582 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:34:05.582 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:34:05.582 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:34:05.582 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:34:05.582 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:34:05.582 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:34:05.583 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:34:05.583 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:34:05.583 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:34:05.584 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:34:05.585 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:34:05.585 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:34:05.628 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 6 2024-04-06T09:34:05.629 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 6 2024-04-06T09:34:05.629 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:34:05.629 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=25, tag_tid=3, entry_tid=1], [object_number=24, tag_tid=3, entry_tid=0], [object_number=27, tag_tid=2, entry_tid=27223], [object_number=26, tag_tid=2, entry_tid=27222]]], state=connected] 2024-04-06T09:34:05.629 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=25, tag_tid=2, entry_tid=27225], [object_number=24, tag_tid=2, entry_tid=27224], [object_number=27, tag_tid=2, entry_tid=27223], [object_number=26, tag_tid=2, entry_tid=27222]]], state=connected] 2024-04-06T09:34:05.632 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:34:05.633 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=25, tag_tid=3, entry_tid=1' 2024-04-06T09:34:05.634 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:34:05.634 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:34:05.634 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:34:05.634 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:34:05.635 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:34:05.635 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:34:05.635 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:34:05.635 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:34:05.635 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:34:05.635 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:34:05.635 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:34:05.635 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:34:05.636 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:34:05.636 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:34:05.636 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:34:05.680 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 6 2024-04-06T09:34:05.680 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 6 2024-04-06T09:34:05.681 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:34:05.681 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=25, tag_tid=3, entry_tid=1], [object_number=24, tag_tid=3, entry_tid=0], [object_number=27, tag_tid=2, entry_tid=27223], [object_number=26, tag_tid=2, entry_tid=27222]]], state=connected] 2024-04-06T09:34:05.681 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=25, tag_tid=2, entry_tid=27225], [object_number=24, tag_tid=2, entry_tid=27224], [object_number=27, tag_tid=2, entry_tid=27223], [object_number=26, tag_tid=2, entry_tid=27222]]], state=connected] 2024-04-06T09:34:05.684 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:34:05.685 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=25, tag_tid=2, entry_tid=27225' 2024-04-06T09:34:05.685 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=25, tag_tid=3, entry_tid=1' -a 'object_number=25, tag_tid=3, entry_tid=1' = 'object_number=25, tag_tid=2, entry_tid=27225' 2024-04-06T09:34:05.685 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=25, tag_tid=2, entry_tid=27225' '!=' 'object_number=26, tag_tid=2, entry_tid=27126' 2024-04-06T09:34:05.686 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:34:05.686 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=25, tag_tid=2, entry_tid=27225' = 'object_number=26, tag_tid=2, entry_tid=27126' 2024-04-06T09:34:05.686 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=25, tag_tid=2, entry_tid=27225' 2024-04-06T09:34:05.686 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=25, tag_tid=3, entry_tid=1' 2024-04-06T09:34:05.686 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:34:05.687 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:34:05.687 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=3 2024-04-06T09:34:05.688 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=25, tag_tid=2, entry_tid=27225' 2024-04-06T09:34:05.688 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:34:05.688 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:34:05.689 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=2 2024-04-06T09:34:05.689 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=25, tag_tid=3, entry_tid=1' 2024-04-06T09:34:05.689 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:34:05.689 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:34:05.690 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:34:05.691 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=25, tag_tid=2, entry_tid=27225' 2024-04-06T09:34:05.691 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:34:05.691 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:34:05.692 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=27225 2024-04-06T09:34:05.692 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 3 = 2 -a 1 -le 27225 2024-04-06T09:34:05.692 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:34:05.692 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:34:05.692 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:34:05.894 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:34:05.894 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:34:05.894 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:34:05.894 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:34:05.894 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:34:05.894 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:34:05.894 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:34:05.894 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:34:05.894 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:34:05.895 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:34:05.895 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:34:05.895 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:34:05.895 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:34:05.895 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:34:05.895 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:34:05.895 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:34:05.895 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:34:05.899 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:34:05.900 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:34:05.900 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:34:05.900 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:34:05.900 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:34:05.900 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:34:05.900 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:34:05.900 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:34:05.900 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:34:05.900 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:34:05.900 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:34:05.900 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:34:05.900 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:34:05.900 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:34:05.900 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:34:05.901 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:34:05.901 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:34:05.901 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:34:05.901 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:34:05.901 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:34:05.901 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:34:05.901 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:34:05.901 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:34:05.901 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:34:05.901 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:34:05.901 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:34:05.901 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:34:05.901 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:34:05.901 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:34:05.902 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:34:05.902 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:34:05.902 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:34:05.926 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:34:05.926 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:34:05.926 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:34:05.926 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:34:06.053 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:34:06.054 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:34:06.054 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:34:06.054 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:34:06.054 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:34:06.054 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:34:06.054 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:34:06.054 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:34:06.054 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:34:06.054 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:34:06.055 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:34:06.055 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:34:06.055 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:34:06.058 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:34:06.058 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:34:06.059 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:34:06.113 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 6 2024-04-06T09:34:06.113 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 6 2024-04-06T09:34:06.113 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:34:06.114 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=25, tag_tid=3, entry_tid=1], [object_number=24, tag_tid=3, entry_tid=0], [object_number=27, tag_tid=2, entry_tid=27223], [object_number=26, tag_tid=2, entry_tid=27222]]], state=connected] 2024-04-06T09:34:06.114 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=25, tag_tid=2, entry_tid=27225], [object_number=24, tag_tid=2, entry_tid=27224], [object_number=27, tag_tid=2, entry_tid=27223], [object_number=26, tag_tid=2, entry_tid=27222]]], state=connected] 2024-04-06T09:34:06.117 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:34:06.118 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=25, tag_tid=3, entry_tid=1' 2024-04-06T09:34:06.119 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:34:06.119 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:34:06.119 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:34:06.119 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:34:06.120 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:34:06.120 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:34:06.120 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:34:06.120 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:34:06.120 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:34:06.120 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:34:06.120 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:34:06.120 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:34:06.121 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:34:06.121 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:34:06.121 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:34:06.166 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 6 2024-04-06T09:34:06.166 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 6 2024-04-06T09:34:06.167 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:34:06.167 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=25, tag_tid=3, entry_tid=1], [object_number=24, tag_tid=3, entry_tid=0], [object_number=27, tag_tid=2, entry_tid=27223], [object_number=26, tag_tid=2, entry_tid=27222]]], state=connected] 2024-04-06T09:34:06.167 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=25, tag_tid=2, entry_tid=27225], [object_number=24, tag_tid=2, entry_tid=27224], [object_number=27, tag_tid=2, entry_tid=27223], [object_number=26, tag_tid=2, entry_tid=27222]]], state=connected] 2024-04-06T09:34:06.169 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:34:06.171 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=25, tag_tid=2, entry_tid=27225' 2024-04-06T09:34:06.171 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=25, tag_tid=3, entry_tid=1' -a 'object_number=25, tag_tid=3, entry_tid=1' = 'object_number=25, tag_tid=2, entry_tid=27225' 2024-04-06T09:34:06.171 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=25, tag_tid=2, entry_tid=27225' '!=' 'object_number=25, tag_tid=2, entry_tid=27225' 2024-04-06T09:34:06.171 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:34:06.171 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.4 2024-04-06T09:34:06.573 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:34:06.573 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:34:06.573 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:34:06.573 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:34:06.573 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:34:06.574 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:34:06.574 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:34:06.574 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:34:06.574 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:34:06.574 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:34:06.574 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:34:06.574 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:34:06.574 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:34:06.574 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:34:06.574 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:34:06.575 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:34:06.575 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:34:06.579 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:34:06.579 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:34:06.579 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:34:06.579 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:34:06.579 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:34:06.579 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:34:06.579 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:34:06.579 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:34:06.579 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:34:06.579 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:34:06.579 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:34:06.579 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:34:06.580 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:34:06.580 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:34:06.580 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:34:06.580 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:34:06.580 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:34:06.580 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:34:06.580 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:34:06.580 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:34:06.580 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:34:06.580 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:34:06.580 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:34:06.580 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:34:06.581 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:34:06.581 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:34:06.581 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:34:06.581 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:34:06.581 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:34:06.581 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:34:06.581 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:34:06.581 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:34:06.603 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:34:06.604 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:34:06.604 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:34:06.604 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:34:06.723 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:34:06.723 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:34:06.723 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:34:06.724 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:34:06.724 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:34:06.724 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:34:06.724 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:34:06.724 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:34:06.724 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:34:06.724 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:34:06.724 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:34:06.724 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:34:06.725 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:34:06.727 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:34:06.727 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:34:06.727 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:34:06.777 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 6 2024-04-06T09:34:06.777 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 6 2024-04-06T09:34:06.777 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:34:06.777 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=25, tag_tid=3, entry_tid=1], [object_number=24, tag_tid=3, entry_tid=0], [object_number=27, tag_tid=2, entry_tid=27223], [object_number=26, tag_tid=2, entry_tid=27222]]], state=connected] 2024-04-06T09:34:06.777 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=25, tag_tid=2, entry_tid=27225], [object_number=24, tag_tid=2, entry_tid=27224], [object_number=27, tag_tid=2, entry_tid=27223], [object_number=26, tag_tid=2, entry_tid=27222]]], state=connected] 2024-04-06T09:34:06.781 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:34:06.782 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=25, tag_tid=3, entry_tid=1' 2024-04-06T09:34:06.783 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:34:06.783 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:34:06.783 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:34:06.784 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:34:06.784 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:34:06.784 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:34:06.784 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:34:06.784 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:34:06.784 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:34:06.784 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:34:06.784 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:34:06.784 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:34:06.785 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:34:06.785 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:34:06.785 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:34:06.830 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 6 2024-04-06T09:34:06.830 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 6 2024-04-06T09:34:06.830 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:34:06.830 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=25, tag_tid=3, entry_tid=1], [object_number=24, tag_tid=3, entry_tid=0], [object_number=27, tag_tid=2, entry_tid=27223], [object_number=26, tag_tid=2, entry_tid=27222]]], state=connected] 2024-04-06T09:34:06.830 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=25, tag_tid=2, entry_tid=27225], [object_number=24, tag_tid=2, entry_tid=27224], [object_number=27, tag_tid=2, entry_tid=27223], [object_number=26, tag_tid=2, entry_tid=27222]]], state=connected] 2024-04-06T09:34:06.834 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:34:06.835 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=25, tag_tid=2, entry_tid=27225' 2024-04-06T09:34:06.835 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=25, tag_tid=3, entry_tid=1' -a 'object_number=25, tag_tid=3, entry_tid=1' = 'object_number=25, tag_tid=2, entry_tid=27225' 2024-04-06T09:34:06.835 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=25, tag_tid=2, entry_tid=27225' '!=' 'object_number=25, tag_tid=2, entry_tid=27225' 2024-04-06T09:34:06.835 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:34:06.835 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.8 2024-04-06T09:34:07.637 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:34:07.637 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:34:07.637 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:34:07.637 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:34:07.637 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:34:07.637 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:34:07.637 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:34:07.638 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:34:07.638 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:34:07.638 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:34:07.638 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:34:07.638 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:34:07.638 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:34:07.638 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:34:07.638 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:34:07.638 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:34:07.639 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:34:07.641 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:34:07.641 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:34:07.641 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:34:07.641 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:34:07.641 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:34:07.642 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:34:07.642 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:34:07.642 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:34:07.642 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:34:07.642 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:34:07.642 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:34:07.642 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:34:07.642 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:34:07.642 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:34:07.642 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:34:07.642 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:34:07.642 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:34:07.642 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:34:07.642 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:34:07.642 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:34:07.643 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:34:07.643 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:34:07.643 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:34:07.643 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:34:07.643 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:34:07.643 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:34:07.643 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:34:07.643 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:34:07.643 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:34:07.643 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:34:07.643 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:34:07.643 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:34:07.667 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:34:07.667 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:34:07.667 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:34:07.668 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:34:07.788 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:34:07.789 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:34:07.789 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:34:07.789 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:34:07.789 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:34:07.789 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:34:07.789 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:34:07.790 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:34:07.790 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:34:07.790 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:34:07.790 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:34:07.790 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:34:07.790 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:34:07.792 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:34:07.792 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:34:07.792 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:34:07.838 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 6 2024-04-06T09:34:07.838 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 6 2024-04-06T09:34:07.838 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:34:07.838 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=25, tag_tid=3, entry_tid=1], [object_number=24, tag_tid=3, entry_tid=0], [object_number=27, tag_tid=2, entry_tid=27223], [object_number=26, tag_tid=2, entry_tid=27222]]], state=connected] 2024-04-06T09:34:07.838 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=25, tag_tid=2, entry_tid=27225], [object_number=24, tag_tid=2, entry_tid=27224], [object_number=27, tag_tid=2, entry_tid=27223], [object_number=26, tag_tid=2, entry_tid=27222]]], state=connected] 2024-04-06T09:34:07.842 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:34:07.843 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=25, tag_tid=3, entry_tid=1' 2024-04-06T09:34:07.844 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:34:07.844 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:34:07.844 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:34:07.844 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:34:07.844 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:34:07.844 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:34:07.844 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:34:07.845 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:34:07.845 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:34:07.845 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:34:07.845 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:34:07.845 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:34:07.845 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:34:07.846 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:34:07.846 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:34:07.890 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 6 2024-04-06T09:34:07.890 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 6 2024-04-06T09:34:07.890 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:34:07.890 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=25, tag_tid=3, entry_tid=1], [object_number=24, tag_tid=3, entry_tid=0], [object_number=27, tag_tid=2, entry_tid=27223], [object_number=26, tag_tid=2, entry_tid=27222]]], state=connected] 2024-04-06T09:34:07.890 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=25, tag_tid=2, entry_tid=27225], [object_number=24, tag_tid=2, entry_tid=27224], [object_number=27, tag_tid=2, entry_tid=27223], [object_number=26, tag_tid=2, entry_tid=27222]]], state=connected] 2024-04-06T09:34:07.893 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:34:07.895 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=25, tag_tid=2, entry_tid=27225' 2024-04-06T09:34:07.895 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=25, tag_tid=3, entry_tid=1' -a 'object_number=25, tag_tid=3, entry_tid=1' = 'object_number=25, tag_tid=2, entry_tid=27225' 2024-04-06T09:34:07.895 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=25, tag_tid=2, entry_tid=27225' '!=' 'object_number=25, tag_tid=2, entry_tid=27225' 2024-04-06T09:34:07.895 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:34:07.895 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 1.6 2024-04-06T09:34:09.497 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:34:09.497 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:34:09.497 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:34:09.497 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:34:09.497 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:34:09.497 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:34:09.497 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:34:09.497 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:34:09.497 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:34:09.497 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:34:09.497 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:34:09.498 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:34:09.498 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:34:09.498 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:34:09.498 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:34:09.498 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:34:09.498 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:34:09.501 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:34:09.501 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:34:09.501 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:34:09.501 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:34:09.502 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:34:09.502 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:34:09.502 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:34:09.502 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:34:09.502 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:34:09.502 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:34:09.502 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:34:09.502 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:34:09.502 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:34:09.502 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:34:09.502 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:34:09.502 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:34:09.502 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:34:09.503 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:34:09.503 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:34:09.503 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:34:09.503 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:34:09.503 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:34:09.503 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:34:09.504 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:34:09.504 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:34:09.504 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:34:09.504 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:34:09.504 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:34:09.504 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:34:09.504 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:34:09.504 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:34:09.504 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:34:09.527 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:34:09.527 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:34:09.527 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:34:09.528 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:34:09.645 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:34:09.647 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:34:09.647 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:34:09.647 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:34:09.647 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:34:09.647 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:34:09.647 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:34:09.647 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:34:09.647 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:34:09.647 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:34:09.648 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:34:09.648 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:34:09.648 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:34:09.650 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:34:09.651 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:34:09.651 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:34:09.694 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 6 2024-04-06T09:34:09.694 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 6 2024-04-06T09:34:09.694 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:34:09.694 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=25, tag_tid=3, entry_tid=1], [object_number=24, tag_tid=3, entry_tid=0], [object_number=27, tag_tid=2, entry_tid=27223], [object_number=26, tag_tid=2, entry_tid=27222]]], state=connected] 2024-04-06T09:34:09.694 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=25, tag_tid=2, entry_tid=27225], [object_number=24, tag_tid=2, entry_tid=27224], [object_number=27, tag_tid=2, entry_tid=27223], [object_number=26, tag_tid=2, entry_tid=27222]]], state=connected] 2024-04-06T09:34:09.697 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:34:09.699 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=25, tag_tid=3, entry_tid=1' 2024-04-06T09:34:09.700 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:34:09.700 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:34:09.700 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:34:09.700 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:34:09.700 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:34:09.700 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:34:09.700 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:34:09.700 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:34:09.700 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:34:09.700 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:34:09.700 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:34:09.701 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:34:09.701 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:34:09.701 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:34:09.701 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:34:09.745 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 6 2024-04-06T09:34:09.745 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 6 2024-04-06T09:34:09.745 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:34:09.745 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=25, tag_tid=3, entry_tid=1], [object_number=24, tag_tid=3, entry_tid=0], [object_number=27, tag_tid=2, entry_tid=27223], [object_number=26, tag_tid=2, entry_tid=27222]]], state=connected] 2024-04-06T09:34:09.745 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=25, tag_tid=2, entry_tid=27225], [object_number=24, tag_tid=2, entry_tid=27224], [object_number=27, tag_tid=2, entry_tid=27223], [object_number=26, tag_tid=2, entry_tid=27222]]], state=connected] 2024-04-06T09:34:09.749 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:34:09.750 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=25, tag_tid=2, entry_tid=27225' 2024-04-06T09:34:09.750 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=25, tag_tid=3, entry_tid=1' -a 'object_number=25, tag_tid=3, entry_tid=1' = 'object_number=25, tag_tid=2, entry_tid=27225' 2024-04-06T09:34:09.750 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=25, tag_tid=2, entry_tid=27225' '!=' 'object_number=25, tag_tid=2, entry_tid=27225' 2024-04-06T09:34:09.751 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:34:09.751 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 2 2024-04-06T09:34:11.752 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:34:11.752 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:34:11.752 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:34:11.752 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:34:11.753 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:34:11.753 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:34:11.753 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:34:11.753 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:34:11.753 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:34:11.753 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:34:11.753 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:34:11.753 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:34:11.753 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:34:11.753 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:34:11.753 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:34:11.754 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:34:11.754 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:34:11.756 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:34:11.756 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:34:11.756 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:34:11.757 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:34:11.757 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:34:11.757 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:34:11.757 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:34:11.757 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:34:11.757 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:34:11.757 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:34:11.757 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:34:11.757 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:34:11.757 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:34:11.757 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:34:11.757 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:34:11.757 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:34:11.757 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:34:11.757 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:34:11.758 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:34:11.758 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:34:11.759 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:34:11.759 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:34:11.759 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:34:11.759 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:34:11.759 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:34:11.759 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:34:11.759 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:34:11.759 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:34:11.759 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:34:11.759 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:34:11.759 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:34:11.760 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:34:11.784 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:34:11.784 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:34:11.784 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:34:11.784 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:34:11.909 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:34:11.910 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:34:11.911 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:34:11.911 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:34:11.911 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:34:11.911 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:34:11.911 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:34:11.911 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:34:11.911 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:34:11.911 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:34:11.912 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:34:11.912 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:34:11.912 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:34:11.914 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:34:11.914 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:34:11.915 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:34:11.960 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 6 2024-04-06T09:34:11.960 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 6 2024-04-06T09:34:11.960 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:34:11.961 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=25, tag_tid=3, entry_tid=1], [object_number=24, tag_tid=3, entry_tid=0], [object_number=27, tag_tid=2, entry_tid=27223], [object_number=26, tag_tid=2, entry_tid=27222]]], state=connected] 2024-04-06T09:34:11.961 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=25, tag_tid=3, entry_tid=1], [object_number=24, tag_tid=3, entry_tid=0], [object_number=27, tag_tid=2, entry_tid=27223], [object_number=26, tag_tid=2, entry_tid=27222]]], state=connected] 2024-04-06T09:34:11.964 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:34:11.965 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=25, tag_tid=3, entry_tid=1' 2024-04-06T09:34:11.966 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:34:11.966 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:34:11.967 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:34:11.967 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:34:11.967 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:34:11.967 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:34:11.967 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:34:11.967 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:34:11.967 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:34:11.967 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:34:11.967 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:34:11.967 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:34:11.968 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:34:11.968 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:34:11.968 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:34:12.012 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 6 2024-04-06T09:34:12.012 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 6 2024-04-06T09:34:12.013 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:34:12.013 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=25, tag_tid=3, entry_tid=1], [object_number=24, tag_tid=3, entry_tid=0], [object_number=27, tag_tid=2, entry_tid=27223], [object_number=26, tag_tid=2, entry_tid=27222]]], state=connected] 2024-04-06T09:34:12.013 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=25, tag_tid=3, entry_tid=1], [object_number=24, tag_tid=3, entry_tid=0], [object_number=27, tag_tid=2, entry_tid=27223], [object_number=26, tag_tid=2, entry_tid=27222]]], state=connected] 2024-04-06T09:34:12.016 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:34:12.018 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=25, tag_tid=3, entry_tid=1' 2024-04-06T09:34:12.018 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=25, tag_tid=3, entry_tid=1' -a 'object_number=25, tag_tid=3, entry_tid=1' = 'object_number=25, tag_tid=3, entry_tid=1' 2024-04-06T09:34:12.018 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:34:12.018 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_snap_present cluster1 mirror test snap1 2024-04-06T09:34:12.018 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:34:12.018 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:34:12.018 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:34:12.018 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local snap_name=snap1 2024-04-06T09:34:12.019 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local s 2024-04-06T09:34:12.019 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-04-06T09:34:12.019 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 1 2024-04-06T09:34:13.019 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 info mirror/test@snap1 2024-04-06T09:34:13.291 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:rbd image 'test': 2024-04-06T09:34:13.291 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: size 512 MiB in 128 objects 2024-04-06T09:34:13.291 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: order 22 (4 MiB objects) 2024-04-06T09:34:13.291 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: snapshot_count: 1 2024-04-06T09:34:13.291 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: id: 10a56da0b207 2024-04-06T09:34:13.291 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: block_name_prefix: rbd_data.10a56da0b207 2024-04-06T09:34:13.292 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: format: 2 2024-04-06T09:34:13.292 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: features: layering, exclusive-lock, journaling 2024-04-06T09:34:13.292 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: op_features: 2024-04-06T09:34:13.292 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: flags: 2024-04-06T09:34:13.292 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: create_timestamp: Sat Apr 6 09:31:59 2024 2024-04-06T09:34:13.292 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: access_timestamp: Sat Apr 6 09:31:59 2024 2024-04-06T09:34:13.292 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: modify_timestamp: Sat Apr 6 09:34:01 2024 2024-04-06T09:34:13.292 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: protected: False 2024-04-06T09:34:13.292 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: journal: 10a56da0b207 2024-04-06T09:34:13.292 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: mirroring state: enabled 2024-04-06T09:34:13.292 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: mirroring mode: journal 2024-04-06T09:34:13.292 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: mirroring global id: 8ad5dedc-b3e0-425d-8993-b3e2231fc5ca 2024-04-06T09:34:13.292 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: mirroring primary: false 2024-04-06T09:34:13.296 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:34:13.297 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n '' ']' 2024-04-06T09:34:13.297 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ compare_image_snaps mirror test snap1 2024-04-06T09:34:13.297 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:34:13.297 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:34:13.297 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local snap_name=snap1 2024-04-06T09:34:13.297 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local ret=0 2024-04-06T09:34:13.297 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local rmt_export=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.export 2024-04-06T09:34:13.297 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loc_export=/tmp/tmp.BhW6xKmROU/cluster1-mirror-test.export 2024-04-06T09:34:13.297 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rm -f /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.export /tmp/tmp.BhW6xKmROU/cluster1-mirror-test.export 2024-04-06T09:34:13.298 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster2 -p mirror export test@snap1 /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.export 2024-04-06T09:34:13.829 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-04-06T09:34:13.840 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 -p mirror export test@snap1 /tmp/tmp.BhW6xKmROU/cluster1-mirror-test.export 2024-04-06T09:34:14.986 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-04-06T09:34:15.006 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmp /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.export /tmp/tmp.BhW6xKmROU/cluster1-mirror-test.export 2024-04-06T09:34:15.286 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rm -f /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.export /tmp/tmp.BhW6xKmROU/cluster1-mirror-test.export 2024-04-06T09:34:15.420 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:34:15.420 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ clean_snap_name=snap1-clean 2024-04-06T09:34:15.421 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ create_snap cluster2 mirror test snap1-clean 2024-04-06T09:34:15.421 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-04-06T09:34:15.421 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:34:15.421 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:34:15.421 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local snap_name=snap1-clean 2024-04-06T09:34:15.421 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap1-clean --debug-rbd=20 --debug-journaler=20 2024-04-06T09:34:16.520 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in `seq 1 10` 2024-04-06T09:34:16.520 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ stress_write_image cluster2 mirror test 2024-04-06T09:34:16.520 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-04-06T09:34:16.520 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:34:16.520 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:34:16.520 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-04-06T09:34:16.523 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local duration=13 2024-04-06T09:34:16.523 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set +e 2024-04-06T09:34:16.523 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ timeout 13s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror test --debug-rbd=20 --debug-journaler=20 2024-04-06T09:34:19.811 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:100 IOs, 98366464 bytes 2024-04-06T09:34:21.634 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:200 IOs, 169490432 bytes 2024-04-06T09:34:25.024 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:300 IOs, 240135168 bytes 2024-04-06T09:34:26.226 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:400 IOs, 272802816 bytes 2024-04-06T09:34:27.664 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:500 IOs, 313946112 bytes 2024-04-06T09:34:28.465 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:600 IOs, 342128640 bytes 2024-04-06T09:34:29.533 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ error_code=124 2024-04-06T09:34:29.533 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set -e 2024-04-06T09:34:29.534 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 124 -eq 124 ']' 2024-04-06T09:34:29.534 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:34:29.534 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+replaying 2024-04-06T09:34:29.534 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:34:29.534 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:34:29.534 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:34:29.534 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+replaying 2024-04-06T09:34:29.534 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local description_pattern= 2024-04-06T09:34:29.534 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local service_pattern= 2024-04-06T09:34:29.534 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:34:29.534 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 1 2024-04-06T09:34:30.535 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test_status_in_pool_dir cluster1 mirror test up+replaying '' '' 2024-04-06T09:34:30.535 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:34:30.535 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:34:30.535 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:34:30.535 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+replaying 2024-04-06T09:34:30.535 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local description_pattern= 2024-04-06T09:34:30.535 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local service_pattern= 2024-04-06T09:34:30.535 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local status 2024-04-06T09:34:30.535 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ CEPH_ARGS= 2024-04-06T09:34:30.536 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster1 mirror image status mirror/test 2024-04-06T09:34:30.618 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ status='test: 2024-04-06T09:34:30.618 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: global_id: 8ad5dedc-b3e0-425d-8993-b3e2231fc5ca 2024-04-06T09:34:30.618 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: state: up+replaying 2024-04-06T09:34:30.618 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: description: replaying, {"bytes_per_second":1676911.57,"entries_behind_primary":1,"entries_per_second":102.6,"non_primary_position":{"entry_tid":1,"object_number":25,"tag_tid":3},"primary_position":{"entry_tid":1,"object_number":25,"tag_tid":4},"seconds_until_synced":0} 2024-04-06T09:34:30.618 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: service: mirror.0 on smithi192 2024-04-06T09:34:30.618 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: last_update: 2024-04-06 09:34:27 2024-04-06T09:34:30.618 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: peer_sites: 2024-04-06T09:34:30.618 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: name: cluster2 2024-04-06T09:34:30.618 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: state: up+stopped 2024-04-06T09:34:30.618 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: description: local image is primary 2024-04-06T09:34:30.618 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: last_update: 2024-04-06 09:34:11' 2024-04-06T09:34:30.618 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^ state: .*up+replaying' 2024-04-06T09:34:30.619 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: state: up+replaying 2024-04-06T09:34:30.619 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^ description: .*' 2024-04-06T09:34:30.620 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: description: replaying, {"bytes_per_second":1676911.57,"entries_behind_primary":1,"entries_per_second":102.6,"non_primary_position":{"entry_tid":1,"object_number":25,"tag_tid":3},"primary_position":{"entry_tid":1,"object_number":25,"tag_tid":4},"seconds_until_synced":0} 2024-04-06T09:34:30.621 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n '' ']' 2024-04-06T09:34:30.621 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ echo up+replaying 2024-04-06T09:34:30.621 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^up+' 2024-04-06T09:34:30.621 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:up+replaying 2024-04-06T09:34:30.621 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep 'service: *mirror..* on ' 2024-04-06T09:34:30.623 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: service: mirror.0 on smithi192 2024-04-06T09:34:30.623 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local last_update 2024-04-06T09:34:30.623 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-04-06T09:34:30.625 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_update='2024-04-06 09:34:27' 2024-04-06T09:34:30.625 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+replaying '2024-04-06 09:34:27' 2024-04-06T09:34:30.625 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:34:30.625 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:34:30.625 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:34:30.626 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+replaying 2024-04-06T09:34:30.626 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'prev_last_update=2024-04-06 09:34:27' 2024-04-06T09:34:30.626 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local status 2024-04-06T09:34:30.626 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ CEPH_ARGS= 2024-04-06T09:34:30.626 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-04-06T09:34:30.697 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ status='OKOKOK142094268mirror.1smithi19218.2.2-1059-g4d05f4cafalseOK42164269mirror.2smithi19218.2.2-1059-g4d05f4cafalseOK42584261mirror.0smithi19218.2.2-1059-g4d05f4catrueOK43194322mirror.3smithi19218.2.2-1059-g4d05f4cafalseOKtest8ad5dedc-b3e0-425d-8993-b3e2231fc5caup+replayingreplaying, {"bytes_per_second":1676911.57,"entries_behind_primary":1,"entries_per_second":102.6,"non_primary_position":{"entry_tid":1,"object_number":25,"tag_tid":3},"primary_position":{"entry_tid":1,"object_number":25,"tag_tid":4},"seconds_until_synced":0}42584261mirror.0smithi1922024-04-06 09:34:27cluster2c5e3bdc3-c286-4289-8fd0-6b8628b92376up+stoppedlocal image is primary2024-04-06 09:34:11' 2024-04-06T09:34:30.697 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local last_update state 2024-04-06T09:34:30.697 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2024-04-06T09:34:30.701 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_update='2024-04-06 09:34:27' 2024-04-06T09:34:30.701 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2024-04-06T09:34:30.704 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ state=up+replaying 2024-04-06T09:34:30.704 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ echo up+replaying 2024-04-06T09:34:30.704 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep up+replaying 2024-04-06T09:34:30.705 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:up+replaying 2024-04-06T09:34:30.705 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:34:30.706 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:34:30.706 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ snap_name=snap2 2024-04-06T09:34:30.706 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ create_snap cluster2 mirror test snap2 2024-04-06T09:34:30.706 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-04-06T09:34:30.706 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:34:30.706 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:34:30.706 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local snap_name=snap2 2024-04-06T09:34:30.706 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap2 --debug-rbd=20 --debug-journaler=20 2024-04-06T09:34:59.516 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_image_replay_started cluster1 mirror test 2024-04-06T09:34:59.516 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:34:59.516 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:34:59.516 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:34:59.516 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_image_replay_state cluster1 mirror test started 2024-04-06T09:34:59.516 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:34:59.517 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:34:59.517 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:34:59.517 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state=started 2024-04-06T09:34:59.517 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local s 2024-04-06T09:34:59.517 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:34:59.517 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 1 2024-04-06T09:35:00.517 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test_image_replay_state cluster1 mirror test started 2024-04-06T09:35:00.518 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:35:00.518 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:35:00.518 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:35:00.518 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local test_state=started 2024-04-06T09:35:00.518 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local status_result 2024-04-06T09:35:00.518 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local current_state=stopped 2024-04-06T09:35:00.519 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2024-04-06T09:35:00.519 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -i state 2024-04-06T09:35:00.519 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_instance=cluster1 2024-04-06T09:35:00.519 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ shift 2024-04-06T09:35:00.519 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:35:00.519 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance=cluster1 2024-04-06T09:35:00.519 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local loop_instance 2024-04-06T09:35:00.519 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:35:00.519 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sleep 0 2024-04-06T09:35:00.520 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:35:00.521 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ seq 0 3 2024-04-06T09:35:00.523 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:35:00.523 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2024-04-06T09:35:00.523 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1:0 2024-04-06T09:35:00.523 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ shift 2024-04-06T09:35:00.523 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:35:00.524 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:35:00.524 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:35:00.524 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=cluster 2024-04-06T09:35:00.524 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:35:00.524 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:35:00.524 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:35:00.524 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:35:00.524 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval cluster=cluster1 2024-04-06T09:35:00.524 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ cluster=cluster1 2024-04-06T09:35:00.524 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:35:00.524 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:35:00.524 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:35:00.524 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local local_cluster=cluster1:0 2024-04-06T09:35:00.524 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local cluster=cluster1 2024-04-06T09:35:00.524 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local instance 2024-04-06T09:35:00.525 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:35:00.525 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local val=cluster1:0 2024-04-06T09:35:00.525 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local cluster_var_name=local_cluster 2024-04-06T09:35:00.525 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local instance_var_name=instance 2024-04-06T09:35:00.525 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ cluster=cluster1 2024-04-06T09:35:00.525 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:35:00.525 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ '[' 0 = cluster1:0 ']' 2024-04-06T09:35:00.525 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ eval local_cluster=cluster1 2024-04-06T09:35:00.526 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++++ local_cluster=cluster1 2024-04-06T09:35:00.526 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ eval instance=0 2024-04-06T09:35:00.526 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++++ instance=0 2024-04-06T09:35:00.526 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:35:00.549 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:35:00.549 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:35:00.549 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:35:00.549 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2024-04-06T09:35:00.666 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ return 0 2024-04-06T09:35:00.667 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ status_result=' "state": "Replaying"' 2024-04-06T09:35:00.667 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ echo ' "state": "Replaying"' 2024-04-06T09:35:00.667 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep -i Replaying 2024-04-06T09:35:00.669 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: "state": "Replaying" 2024-04-06T09:35:00.671 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ current_state=started 2024-04-06T09:35:00.671 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test started = started 2024-04-06T09:35:00.671 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:35:00.671 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror test 2024-04-06T09:35:00.671 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local local_cluster=cluster1 2024-04-06T09:35:00.671 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-04-06T09:35:00.671 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:35:00.671 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:35:00.671 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' journal = journal ']' 2024-04-06T09:35:00.671 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_journal_replay_complete cluster1 cluster2 mirror test 2024-04-06T09:35:00.671 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local local_cluster=cluster1 2024-04-06T09:35:00.671 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-04-06T09:35:00.671 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:35:00.671 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:35:00.672 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local s master_pos mirror_pos last_mirror_pos 2024-04-06T09:35:00.672 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local master_tag master_entry mirror_tag mirror_entry 2024-04-06T09:35:00.672 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:35:00.672 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:35:00.672 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:35:00.872 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:35:00.872 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:35:00.872 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:35:00.872 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:35:00.872 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:35:00.872 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:35:00.872 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:35:00.872 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:35:00.873 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:35:00.873 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:35:00.873 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:35:00.873 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:35:00.873 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:35:00.873 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:35:00.873 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:35:00.873 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:35:00.875 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:35:00.875 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:35:00.875 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:35:00.875 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:35:00.875 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:35:00.875 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:35:00.875 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:35:00.875 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:35:00.875 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:35:00.875 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:35:00.875 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:35:00.875 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:35:00.876 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:35:00.876 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:35:00.876 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:35:00.876 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:35:00.876 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:35:00.876 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:35:00.876 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:35:00.876 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:35:00.876 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:35:00.876 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:35:00.876 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:35:00.876 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:35:00.876 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:35:00.876 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:35:00.876 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:35:00.876 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:35:00.877 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:35:00.877 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:35:00.877 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:35:00.877 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:35:00.877 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:35:00.904 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:35:00.904 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:35:00.904 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:35:00.904 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:35:01.028 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:35:01.029 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:35:01.029 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:35:01.029 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:35:01.030 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:35:01.030 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:35:01.030 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:35:01.030 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:35:01.030 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:35:01.030 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:35:01.031 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:35:01.031 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:35:01.031 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:35:01.032 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:35:01.032 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:35:01.033 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:35:01.078 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 8 2024-04-06T09:35:01.078 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 11 2024-04-06T09:35:01.078 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:35:01.078 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=45, tag_tid=6, entry_tid=1], [object_number=44, tag_tid=6, entry_tid=0], [object_number=47, tag_tid=5, entry_tid=22415], [object_number=46, tag_tid=5, entry_tid=22414]]], state=connected] 2024-04-06T09:35:01.078 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=35, tag_tid=5, entry_tid=7583], [object_number=34, tag_tid=5, entry_tid=7582], [object_number=33, tag_tid=5, entry_tid=7581], [object_number=32, tag_tid=5, entry_tid=7580]]], state=connected] 2024-04-06T09:35:01.081 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:35:01.082 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=45, tag_tid=6, entry_tid=1' 2024-04-06T09:35:01.083 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:35:01.084 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:35:01.084 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:35:01.084 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:35:01.084 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:35:01.084 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:35:01.084 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:35:01.084 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:35:01.084 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:35:01.084 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:35:01.084 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:35:01.084 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:35:01.085 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:35:01.085 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:35:01.085 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:35:01.130 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 8 2024-04-06T09:35:01.130 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 11 2024-04-06T09:35:01.130 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:35:01.130 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=45, tag_tid=6, entry_tid=1], [object_number=44, tag_tid=6, entry_tid=0], [object_number=47, tag_tid=5, entry_tid=22415], [object_number=46, tag_tid=5, entry_tid=22414]]], state=connected] 2024-04-06T09:35:01.130 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=35, tag_tid=5, entry_tid=7583], [object_number=34, tag_tid=5, entry_tid=7582], [object_number=33, tag_tid=5, entry_tid=7581], [object_number=32, tag_tid=5, entry_tid=7580]]], state=connected] 2024-04-06T09:35:01.133 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:35:01.135 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=35, tag_tid=5, entry_tid=7583' 2024-04-06T09:35:01.136 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=45, tag_tid=6, entry_tid=1' -a 'object_number=45, tag_tid=6, entry_tid=1' = 'object_number=35, tag_tid=5, entry_tid=7583' 2024-04-06T09:35:01.136 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=35, tag_tid=5, entry_tid=7583' '!=' '' 2024-04-06T09:35:01.136 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:35:01.136 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=35, tag_tid=5, entry_tid=7583' = '' 2024-04-06T09:35:01.136 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=35, tag_tid=5, entry_tid=7583' 2024-04-06T09:35:01.136 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=45, tag_tid=6, entry_tid=1' 2024-04-06T09:35:01.136 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:35:01.136 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:35:01.137 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=6 2024-04-06T09:35:01.137 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=35, tag_tid=5, entry_tid=7583' 2024-04-06T09:35:01.138 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:35:01.138 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:35:01.138 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=5 2024-04-06T09:35:01.139 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=45, tag_tid=6, entry_tid=1' 2024-04-06T09:35:01.139 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:35:01.139 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:35:01.140 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:35:01.141 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=35, tag_tid=5, entry_tid=7583' 2024-04-06T09:35:01.141 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:35:01.141 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:35:01.142 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=7583 2024-04-06T09:35:01.142 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 6 = 5 -a 1 -le 7583 2024-04-06T09:35:01.142 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:35:01.142 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:35:01.142 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:35:01.344 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:35:01.344 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:35:01.344 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:35:01.344 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:35:01.344 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:35:01.344 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:35:01.344 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:35:01.344 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:35:01.344 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:35:01.344 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:35:01.344 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:35:01.345 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:35:01.345 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:35:01.345 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:35:01.345 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:35:01.345 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:35:01.345 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:35:01.347 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:35:01.347 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:35:01.347 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:35:01.347 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:35:01.347 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:35:01.348 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:35:01.348 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:35:01.348 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:35:01.348 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:35:01.348 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:35:01.348 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:35:01.348 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:35:01.348 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:35:01.348 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:35:01.348 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:35:01.348 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:35:01.349 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:35:01.349 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:35:01.349 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:35:01.349 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:35:01.350 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:35:01.350 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:35:01.350 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:35:01.350 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:35:01.350 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:35:01.350 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:35:01.350 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:35:01.350 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:35:01.350 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:35:01.350 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:35:01.350 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:35:01.354 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:35:01.375 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:35:01.375 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:35:01.375 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:35:01.376 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:35:01.503 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:35:01.504 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:35:01.504 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:35:01.504 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:35:01.504 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:35:01.504 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:35:01.504 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:35:01.504 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:35:01.504 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:35:01.504 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:35:01.505 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:35:01.505 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:35:01.505 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:35:01.507 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:35:01.507 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:35:01.507 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:35:01.761 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 8 2024-04-06T09:35:01.761 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 11 2024-04-06T09:35:01.761 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:35:01.761 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=45, tag_tid=6, entry_tid=1], [object_number=44, tag_tid=6, entry_tid=0], [object_number=47, tag_tid=5, entry_tid=22415], [object_number=46, tag_tid=5, entry_tid=22414]]], state=connected] 2024-04-06T09:35:01.761 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=33, tag_tid=5, entry_tid=7693], [object_number=32, tag_tid=5, entry_tid=7692], [object_number=35, tag_tid=5, entry_tid=7691], [object_number=34, tag_tid=5, entry_tid=7690]]], state=connected] 2024-04-06T09:35:01.767 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:35:01.768 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=45, tag_tid=6, entry_tid=1' 2024-04-06T09:35:01.769 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:35:01.769 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:35:01.769 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:35:01.769 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:35:01.769 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:35:01.769 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:35:01.770 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:35:01.770 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:35:01.770 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:35:01.770 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:35:01.770 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:35:01.771 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:35:01.772 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:35:01.773 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:35:01.773 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:35:01.826 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 8 2024-04-06T09:35:01.826 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 11 2024-04-06T09:35:01.826 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:35:01.827 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=45, tag_tid=6, entry_tid=1], [object_number=44, tag_tid=6, entry_tid=0], [object_number=47, tag_tid=5, entry_tid=22415], [object_number=46, tag_tid=5, entry_tid=22414]]], state=connected] 2024-04-06T09:35:01.827 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=33, tag_tid=5, entry_tid=7693], [object_number=32, tag_tid=5, entry_tid=7692], [object_number=35, tag_tid=5, entry_tid=7691], [object_number=34, tag_tid=5, entry_tid=7690]]], state=connected] 2024-04-06T09:35:01.830 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:35:01.833 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=33, tag_tid=5, entry_tid=7693' 2024-04-06T09:35:01.833 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=45, tag_tid=6, entry_tid=1' -a 'object_number=45, tag_tid=6, entry_tid=1' = 'object_number=33, tag_tid=5, entry_tid=7693' 2024-04-06T09:35:01.833 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=33, tag_tid=5, entry_tid=7693' '!=' 'object_number=35, tag_tid=5, entry_tid=7583' 2024-04-06T09:35:01.833 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:35:01.833 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=33, tag_tid=5, entry_tid=7693' = 'object_number=35, tag_tid=5, entry_tid=7583' 2024-04-06T09:35:01.833 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=33, tag_tid=5, entry_tid=7693' 2024-04-06T09:35:01.833 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=45, tag_tid=6, entry_tid=1' 2024-04-06T09:35:01.833 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:35:01.833 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:35:01.834 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=6 2024-04-06T09:35:01.835 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=33, tag_tid=5, entry_tid=7693' 2024-04-06T09:35:01.835 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:35:01.835 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:35:01.836 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=5 2024-04-06T09:35:01.836 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=45, tag_tid=6, entry_tid=1' 2024-04-06T09:35:01.836 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:35:01.836 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:35:01.837 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:35:01.838 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=33, tag_tid=5, entry_tid=7693' 2024-04-06T09:35:01.838 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:35:01.838 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:35:01.840 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=7693 2024-04-06T09:35:01.840 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 6 = 5 -a 1 -le 7693 2024-04-06T09:35:01.840 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:35:01.840 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:35:01.840 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:35:02.041 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:35:02.042 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:35:02.042 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:35:02.042 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:35:02.042 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:35:02.042 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:35:02.042 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:35:02.042 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:35:02.042 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:35:02.042 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:35:02.042 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:35:02.042 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:35:02.042 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:35:02.042 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:35:02.042 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:35:02.043 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:35:02.043 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:35:02.045 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:35:02.045 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:35:02.045 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:35:02.045 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:35:02.045 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:35:02.045 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:35:02.045 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:35:02.045 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:35:02.045 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:35:02.045 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:35:02.045 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:35:02.045 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:35:02.045 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:35:02.045 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:35:02.046 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:35:02.046 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:35:02.047 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:35:02.047 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:35:02.047 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:35:02.047 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:35:02.047 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:35:02.047 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:35:02.047 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:35:02.047 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:35:02.047 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:35:02.048 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:35:02.048 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:35:02.048 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:35:02.048 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:35:02.048 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:35:02.048 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:35:02.048 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:35:02.072 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:35:02.072 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:35:02.072 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:35:02.072 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:35:02.201 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:35:02.202 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:35:02.202 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:35:02.202 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:35:02.202 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:35:02.202 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:35:02.202 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:35:02.202 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:35:02.203 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:35:02.203 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:35:02.203 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:35:02.203 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:35:02.203 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:35:02.206 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:35:02.206 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:35:02.206 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:35:02.252 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 8 2024-04-06T09:35:02.252 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 11 2024-04-06T09:35:02.252 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:35:02.252 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=45, tag_tid=6, entry_tid=1], [object_number=44, tag_tid=6, entry_tid=0], [object_number=47, tag_tid=5, entry_tid=22415], [object_number=46, tag_tid=5, entry_tid=22414]]], state=connected] 2024-04-06T09:35:02.252 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=32, tag_tid=5, entry_tid=7808], [object_number=35, tag_tid=5, entry_tid=7807], [object_number=34, tag_tid=5, entry_tid=7806], [object_number=33, tag_tid=5, entry_tid=7805]]], state=connected] 2024-04-06T09:35:02.256 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:35:02.257 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=45, tag_tid=6, entry_tid=1' 2024-04-06T09:35:02.258 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:35:02.258 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:35:02.258 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:35:02.258 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:35:02.259 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:35:02.259 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:35:02.259 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:35:02.259 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:35:02.259 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:35:02.259 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:35:02.259 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:35:02.259 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:35:02.259 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:35:02.260 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:35:02.260 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:35:02.306 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 8 2024-04-06T09:35:02.306 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 11 2024-04-06T09:35:02.306 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:35:02.306 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=45, tag_tid=6, entry_tid=1], [object_number=44, tag_tid=6, entry_tid=0], [object_number=47, tag_tid=5, entry_tid=22415], [object_number=46, tag_tid=5, entry_tid=22414]]], state=connected] 2024-04-06T09:35:02.306 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=32, tag_tid=5, entry_tid=7808], [object_number=35, tag_tid=5, entry_tid=7807], [object_number=34, tag_tid=5, entry_tid=7806], [object_number=33, tag_tid=5, entry_tid=7805]]], state=connected] 2024-04-06T09:35:02.309 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:35:02.311 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=32, tag_tid=5, entry_tid=7808' 2024-04-06T09:35:02.312 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=45, tag_tid=6, entry_tid=1' -a 'object_number=45, tag_tid=6, entry_tid=1' = 'object_number=32, tag_tid=5, entry_tid=7808' 2024-04-06T09:35:02.312 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=32, tag_tid=5, entry_tid=7808' '!=' 'object_number=33, tag_tid=5, entry_tid=7693' 2024-04-06T09:35:02.312 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:35:02.312 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=32, tag_tid=5, entry_tid=7808' = 'object_number=33, tag_tid=5, entry_tid=7693' 2024-04-06T09:35:02.312 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=32, tag_tid=5, entry_tid=7808' 2024-04-06T09:35:02.312 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=45, tag_tid=6, entry_tid=1' 2024-04-06T09:35:02.312 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:35:02.312 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:35:02.312 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=6 2024-04-06T09:35:02.313 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=32, tag_tid=5, entry_tid=7808' 2024-04-06T09:35:02.313 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:35:02.314 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:35:02.314 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=5 2024-04-06T09:35:02.315 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=45, tag_tid=6, entry_tid=1' 2024-04-06T09:35:02.315 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:35:02.315 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:35:02.316 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:35:02.317 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=32, tag_tid=5, entry_tid=7808' 2024-04-06T09:35:02.317 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:35:02.317 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:35:02.318 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=7808 2024-04-06T09:35:02.318 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 6 = 5 -a 1 -le 7808 2024-04-06T09:35:02.318 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:35:02.318 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:35:02.318 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:35:02.520 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:35:02.520 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:35:02.520 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:35:02.520 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:35:02.520 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:35:02.520 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:35:02.520 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:35:02.520 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:35:02.520 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:35:02.520 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:35:02.520 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:35:02.521 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:35:02.521 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:35:02.521 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:35:02.521 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:35:02.521 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:35:02.522 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:35:02.524 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:35:02.524 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:35:02.524 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:35:02.524 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:35:02.524 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:35:02.525 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:35:02.525 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:35:02.525 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:35:02.525 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:35:02.525 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:35:02.525 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:35:02.525 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:35:02.525 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:35:02.525 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:35:02.525 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:35:02.525 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:35:02.525 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:35:02.525 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:35:02.525 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:35:02.525 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:35:02.525 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:35:02.526 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:35:02.526 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:35:02.526 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:35:02.526 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:35:02.526 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:35:02.526 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:35:02.526 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:35:02.526 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:35:02.526 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:35:02.526 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:35:02.526 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:35:02.551 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:35:02.551 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:35:02.551 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:35:02.551 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:35:02.685 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:35:02.685 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:35:02.685 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:35:02.685 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:35:02.685 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:35:02.685 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:35:02.686 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:35:02.686 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:35:02.686 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:35:02.686 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:35:02.686 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:35:02.687 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:35:02.687 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:35:02.688 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:35:02.688 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:35:02.688 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:35:02.734 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 8 2024-04-06T09:35:02.735 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 11 2024-04-06T09:35:02.735 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:35:02.735 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=45, tag_tid=6, entry_tid=1], [object_number=44, tag_tid=6, entry_tid=0], [object_number=47, tag_tid=5, entry_tid=22415], [object_number=46, tag_tid=5, entry_tid=22414]]], state=connected] 2024-04-06T09:35:02.735 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=35, tag_tid=5, entry_tid=7907], [object_number=34, tag_tid=5, entry_tid=7906], [object_number=33, tag_tid=5, entry_tid=7905], [object_number=32, tag_tid=5, entry_tid=7904]]], state=connected] 2024-04-06T09:35:02.738 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:35:02.740 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=45, tag_tid=6, entry_tid=1' 2024-04-06T09:35:02.741 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:35:02.741 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:35:02.741 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:35:02.741 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:35:02.741 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:35:02.741 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:35:02.741 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:35:02.741 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:35:02.741 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:35:02.741 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:35:02.741 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:35:02.741 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:35:02.742 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:35:02.742 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:35:02.742 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:35:02.787 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 8 2024-04-06T09:35:02.787 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 11 2024-04-06T09:35:02.787 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:35:02.787 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=45, tag_tid=6, entry_tid=1], [object_number=44, tag_tid=6, entry_tid=0], [object_number=47, tag_tid=5, entry_tid=22415], [object_number=46, tag_tid=5, entry_tid=22414]]], state=connected] 2024-04-06T09:35:02.787 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=35, tag_tid=5, entry_tid=7907], [object_number=34, tag_tid=5, entry_tid=7906], [object_number=33, tag_tid=5, entry_tid=7905], [object_number=32, tag_tid=5, entry_tid=7904]]], state=connected] 2024-04-06T09:35:02.791 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:35:02.793 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=35, tag_tid=5, entry_tid=7907' 2024-04-06T09:35:02.793 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=45, tag_tid=6, entry_tid=1' -a 'object_number=45, tag_tid=6, entry_tid=1' = 'object_number=35, tag_tid=5, entry_tid=7907' 2024-04-06T09:35:02.793 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=35, tag_tid=5, entry_tid=7907' '!=' 'object_number=32, tag_tid=5, entry_tid=7808' 2024-04-06T09:35:02.793 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:35:02.793 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=35, tag_tid=5, entry_tid=7907' = 'object_number=32, tag_tid=5, entry_tid=7808' 2024-04-06T09:35:02.793 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=35, tag_tid=5, entry_tid=7907' 2024-04-06T09:35:02.793 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=45, tag_tid=6, entry_tid=1' 2024-04-06T09:35:02.793 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:35:02.793 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:35:02.794 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=6 2024-04-06T09:35:02.794 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=35, tag_tid=5, entry_tid=7907' 2024-04-06T09:35:02.795 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:35:02.795 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:35:02.795 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=5 2024-04-06T09:35:02.796 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=45, tag_tid=6, entry_tid=1' 2024-04-06T09:35:02.796 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:35:02.796 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:35:02.797 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:35:02.798 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=35, tag_tid=5, entry_tid=7907' 2024-04-06T09:35:02.798 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:35:02.798 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:35:02.799 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=7907 2024-04-06T09:35:02.799 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 6 = 5 -a 1 -le 7907 2024-04-06T09:35:02.800 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:35:02.800 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:35:02.800 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:35:03.001 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:35:03.001 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:35:03.001 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:35:03.002 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:35:03.002 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:35:03.002 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:35:03.002 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:35:03.002 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:35:03.002 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:35:03.002 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:35:03.002 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:35:03.002 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:35:03.002 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:35:03.002 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:35:03.002 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:35:03.003 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:35:03.003 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:35:03.006 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:35:03.006 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:35:03.006 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:35:03.006 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:35:03.006 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:35:03.006 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:35:03.006 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:35:03.006 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:35:03.006 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:35:03.006 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:35:03.006 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:35:03.007 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:35:03.007 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:35:03.007 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:35:03.007 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:35:03.007 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:35:03.008 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:35:03.008 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:35:03.008 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:35:03.008 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:35:03.008 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:35:03.008 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:35:03.008 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:35:03.008 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:35:03.008 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:35:03.008 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:35:03.009 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:35:03.009 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:35:03.009 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:35:03.009 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:35:03.009 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:35:03.009 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:35:03.032 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:35:03.032 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:35:03.033 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:35:03.033 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:35:03.167 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:35:03.168 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:35:03.168 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:35:03.168 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:35:03.168 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:35:03.168 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:35:03.168 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:35:03.168 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:35:03.168 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:35:03.168 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:35:03.168 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:35:03.169 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:35:03.169 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:35:03.171 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:35:03.172 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:35:03.172 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:35:03.217 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 8 2024-04-06T09:35:03.217 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 11 2024-04-06T09:35:03.217 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:35:03.217 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=45, tag_tid=6, entry_tid=1], [object_number=44, tag_tid=6, entry_tid=0], [object_number=47, tag_tid=5, entry_tid=22415], [object_number=46, tag_tid=5, entry_tid=22414]]], state=connected] 2024-04-06T09:35:03.217 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=33, tag_tid=5, entry_tid=8025], [object_number=32, tag_tid=5, entry_tid=8024], [object_number=35, tag_tid=5, entry_tid=8023], [object_number=34, tag_tid=5, entry_tid=8022]]], state=connected] 2024-04-06T09:35:03.220 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:35:03.222 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=45, tag_tid=6, entry_tid=1' 2024-04-06T09:35:03.223 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:35:03.223 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:35:03.223 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:35:03.223 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:35:03.223 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:35:03.223 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:35:03.223 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:35:03.223 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:35:03.223 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:35:03.224 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:35:03.224 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:35:03.224 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:35:03.224 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:35:03.224 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:35:03.224 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:35:03.270 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 8 2024-04-06T09:35:03.270 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 11 2024-04-06T09:35:03.270 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:35:03.270 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=45, tag_tid=6, entry_tid=1], [object_number=44, tag_tid=6, entry_tid=0], [object_number=47, tag_tid=5, entry_tid=22415], [object_number=46, tag_tid=5, entry_tid=22414]]], state=connected] 2024-04-06T09:35:03.270 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=33, tag_tid=5, entry_tid=8025], [object_number=32, tag_tid=5, entry_tid=8024], [object_number=35, tag_tid=5, entry_tid=8023], [object_number=34, tag_tid=5, entry_tid=8022]]], state=connected] 2024-04-06T09:35:03.273 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:35:03.276 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=33, tag_tid=5, entry_tid=8025' 2024-04-06T09:35:03.276 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=45, tag_tid=6, entry_tid=1' -a 'object_number=45, tag_tid=6, entry_tid=1' = 'object_number=33, tag_tid=5, entry_tid=8025' 2024-04-06T09:35:03.276 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=33, tag_tid=5, entry_tid=8025' '!=' 'object_number=35, tag_tid=5, entry_tid=7907' 2024-04-06T09:35:03.276 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:35:03.276 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=33, tag_tid=5, entry_tid=8025' = 'object_number=35, tag_tid=5, entry_tid=7907' 2024-04-06T09:35:03.276 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=33, tag_tid=5, entry_tid=8025' 2024-04-06T09:35:03.276 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=45, tag_tid=6, entry_tid=1' 2024-04-06T09:35:03.276 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:35:03.276 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:35:03.277 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=6 2024-04-06T09:35:03.277 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=33, tag_tid=5, entry_tid=8025' 2024-04-06T09:35:03.277 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:35:03.277 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:35:03.278 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=5 2024-04-06T09:35:03.279 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=45, tag_tid=6, entry_tid=1' 2024-04-06T09:35:03.279 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:35:03.279 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:35:03.280 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:35:03.280 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=33, tag_tid=5, entry_tid=8025' 2024-04-06T09:35:03.280 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:35:03.281 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:35:03.282 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=8025 2024-04-06T09:35:03.282 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 6 = 5 -a 1 -le 8025 2024-04-06T09:35:03.282 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:35:03.282 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:35:03.282 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:35:03.483 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:35:03.483 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:35:03.483 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:35:03.483 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:35:03.483 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:35:03.484 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:35:03.484 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:35:03.484 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:35:03.484 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:35:03.484 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:35:03.484 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:35:03.484 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:35:03.484 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:35:03.484 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:35:03.484 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:35:03.485 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:35:03.485 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:35:03.489 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:35:03.489 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:35:03.489 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:35:03.489 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:35:03.489 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:35:03.489 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:35:03.489 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:35:03.489 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:35:03.489 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:35:03.489 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:35:03.489 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:35:03.490 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:35:03.490 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:35:03.490 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:35:03.490 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:35:03.490 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:35:03.490 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:35:03.490 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:35:03.490 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:35:03.490 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:35:03.490 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:35:03.490 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:35:03.490 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:35:03.490 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:35:03.491 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:35:03.491 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:35:03.491 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:35:03.491 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:35:03.491 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:35:03.491 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:35:03.491 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:35:03.491 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:35:03.515 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:35:03.515 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:35:03.515 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:35:03.515 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:35:03.644 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:35:03.645 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:35:03.645 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:35:03.645 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:35:03.645 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:35:03.645 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:35:03.646 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:35:03.646 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:35:03.646 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:35:03.646 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:35:03.646 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:35:03.647 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:35:03.647 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:35:03.648 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:35:03.649 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:35:03.649 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:35:03.693 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 8 2024-04-06T09:35:03.693 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 11 2024-04-06T09:35:03.693 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:35:03.693 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=45, tag_tid=6, entry_tid=1], [object_number=44, tag_tid=6, entry_tid=0], [object_number=47, tag_tid=5, entry_tid=22415], [object_number=46, tag_tid=5, entry_tid=22414]]], state=connected] 2024-04-06T09:35:03.694 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=35, tag_tid=5, entry_tid=8127], [object_number=34, tag_tid=5, entry_tid=8126], [object_number=33, tag_tid=5, entry_tid=8125], [object_number=32, tag_tid=5, entry_tid=8124]]], state=connected] 2024-04-06T09:35:03.697 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:35:03.699 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=45, tag_tid=6, entry_tid=1' 2024-04-06T09:35:03.699 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:35:03.699 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:35:03.699 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:35:03.700 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:35:03.700 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:35:03.700 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:35:03.700 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:35:03.700 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:35:03.700 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:35:03.700 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:35:03.700 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:35:03.700 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:35:03.701 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:35:03.701 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:35:03.701 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:35:03.746 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 8 2024-04-06T09:35:03.746 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 11 2024-04-06T09:35:03.746 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:35:03.746 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=45, tag_tid=6, entry_tid=1], [object_number=44, tag_tid=6, entry_tid=0], [object_number=47, tag_tid=5, entry_tid=22415], [object_number=46, tag_tid=5, entry_tid=22414]]], state=connected] 2024-04-06T09:35:03.746 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=35, tag_tid=5, entry_tid=8127], [object_number=34, tag_tid=5, entry_tid=8126], [object_number=33, tag_tid=5, entry_tid=8125], [object_number=32, tag_tid=5, entry_tid=8124]]], state=connected] 2024-04-06T09:35:03.749 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:35:03.752 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=35, tag_tid=5, entry_tid=8127' 2024-04-06T09:35:03.752 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=45, tag_tid=6, entry_tid=1' -a 'object_number=45, tag_tid=6, entry_tid=1' = 'object_number=35, tag_tid=5, entry_tid=8127' 2024-04-06T09:35:03.752 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=35, tag_tid=5, entry_tid=8127' '!=' 'object_number=33, tag_tid=5, entry_tid=8025' 2024-04-06T09:35:03.752 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:35:03.752 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=35, tag_tid=5, entry_tid=8127' = 'object_number=33, tag_tid=5, entry_tid=8025' 2024-04-06T09:35:03.752 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=35, tag_tid=5, entry_tid=8127' 2024-04-06T09:35:03.752 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=45, tag_tid=6, entry_tid=1' 2024-04-06T09:35:03.752 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:35:03.752 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:35:03.753 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=6 2024-04-06T09:35:03.753 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=35, tag_tid=5, entry_tid=8127' 2024-04-06T09:35:03.753 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:35:03.753 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:35:03.754 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=5 2024-04-06T09:35:03.755 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=45, tag_tid=6, entry_tid=1' 2024-04-06T09:35:03.755 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:35:03.755 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:35:03.756 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:35:03.756 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=35, tag_tid=5, entry_tid=8127' 2024-04-06T09:35:03.757 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:35:03.757 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:35:03.758 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=8127 2024-04-06T09:35:03.758 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 6 = 5 -a 1 -le 8127 2024-04-06T09:35:03.758 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:35:03.758 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:35:03.758 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:35:03.959 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:35:03.960 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:35:03.960 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:35:03.960 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:35:03.960 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:35:03.960 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:35:03.960 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:35:03.960 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:35:03.960 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:35:03.960 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:35:03.960 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:35:03.960 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:35:03.960 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:35:03.960 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:35:03.961 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:35:03.961 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:35:03.961 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:35:03.963 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:35:03.963 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:35:03.963 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:35:03.963 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:35:03.963 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:35:03.963 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:35:03.963 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:35:03.963 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:35:03.964 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:35:03.964 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:35:03.964 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:35:03.964 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:35:03.964 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:35:03.964 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:35:03.964 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:35:03.964 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:35:03.965 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:35:03.965 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:35:03.965 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:35:03.965 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:35:03.965 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:35:03.965 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:35:03.965 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:35:03.965 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:35:03.965 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:35:03.966 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:35:03.966 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:35:03.966 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:35:03.966 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:35:03.966 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:35:03.966 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:35:03.966 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:35:03.990 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:35:03.991 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:35:03.991 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:35:03.991 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:35:04.120 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:35:04.121 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:35:04.121 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:35:04.122 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:35:04.122 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:35:04.122 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:35:04.122 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:35:04.122 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:35:04.122 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:35:04.122 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:35:04.123 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:35:04.123 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:35:04.123 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:35:04.124 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:35:04.124 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:35:04.124 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:35:04.176 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 8 2024-04-06T09:35:04.176 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 11 2024-04-06T09:35:04.176 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:35:04.176 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=45, tag_tid=6, entry_tid=1], [object_number=44, tag_tid=6, entry_tid=0], [object_number=47, tag_tid=5, entry_tid=22415], [object_number=46, tag_tid=5, entry_tid=22414]]], state=connected] 2024-04-06T09:35:04.176 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=32, tag_tid=5, entry_tid=8224], [object_number=35, tag_tid=5, entry_tid=8223], [object_number=34, tag_tid=5, entry_tid=8222], [object_number=33, tag_tid=5, entry_tid=8221]]], state=connected] 2024-04-06T09:35:04.180 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:35:04.181 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=45, tag_tid=6, entry_tid=1' 2024-04-06T09:35:04.182 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:35:04.182 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:35:04.182 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:35:04.183 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:35:04.183 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:35:04.183 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:35:04.183 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:35:04.183 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:35:04.183 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:35:04.183 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:35:04.183 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:35:04.183 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:35:04.184 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:35:04.184 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:35:04.184 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:35:04.230 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 8 2024-04-06T09:35:04.230 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 11 2024-04-06T09:35:04.230 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:35:04.230 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=45, tag_tid=6, entry_tid=1], [object_number=44, tag_tid=6, entry_tid=0], [object_number=47, tag_tid=5, entry_tid=22415], [object_number=46, tag_tid=5, entry_tid=22414]]], state=connected] 2024-04-06T09:35:04.231 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=32, tag_tid=5, entry_tid=8224], [object_number=35, tag_tid=5, entry_tid=8223], [object_number=34, tag_tid=5, entry_tid=8222], [object_number=33, tag_tid=5, entry_tid=8221]]], state=connected] 2024-04-06T09:35:04.234 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:35:04.236 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=32, tag_tid=5, entry_tid=8224' 2024-04-06T09:35:04.236 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=45, tag_tid=6, entry_tid=1' -a 'object_number=45, tag_tid=6, entry_tid=1' = 'object_number=32, tag_tid=5, entry_tid=8224' 2024-04-06T09:35:04.236 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=32, tag_tid=5, entry_tid=8224' '!=' 'object_number=35, tag_tid=5, entry_tid=8127' 2024-04-06T09:35:04.236 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:35:04.236 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=32, tag_tid=5, entry_tid=8224' = 'object_number=35, tag_tid=5, entry_tid=8127' 2024-04-06T09:35:04.236 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=32, tag_tid=5, entry_tid=8224' 2024-04-06T09:35:04.236 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=45, tag_tid=6, entry_tid=1' 2024-04-06T09:35:04.237 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:35:04.237 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:35:04.237 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=6 2024-04-06T09:35:04.238 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=32, tag_tid=5, entry_tid=8224' 2024-04-06T09:35:04.238 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:35:04.238 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:35:04.239 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=5 2024-04-06T09:35:04.239 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=45, tag_tid=6, entry_tid=1' 2024-04-06T09:35:04.239 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:35:04.240 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:35:04.240 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:35:04.241 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=32, tag_tid=5, entry_tid=8224' 2024-04-06T09:35:04.241 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:35:04.241 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:35:04.242 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=8224 2024-04-06T09:35:04.243 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 6 = 5 -a 1 -le 8224 2024-04-06T09:35:04.243 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:35:04.243 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:35:04.243 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:35:04.444 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:35:04.445 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:35:04.445 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:35:04.445 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:35:04.445 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:35:04.445 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:35:04.445 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:35:04.445 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:35:04.445 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:35:04.445 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:35:04.445 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:35:04.445 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:35:04.446 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:35:04.446 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:35:04.446 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:35:04.446 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:35:04.446 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:35:04.449 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:35:04.449 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:35:04.449 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:35:04.449 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:35:04.449 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:35:04.449 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:35:04.449 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:35:04.449 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:35:04.449 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:35:04.449 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:35:04.450 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:35:04.450 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:35:04.450 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:35:04.450 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:35:04.450 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:35:04.450 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:35:04.450 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:35:04.450 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:35:04.450 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:35:04.450 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:35:04.450 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:35:04.451 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:35:04.451 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:35:04.451 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:35:04.451 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:35:04.451 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:35:04.451 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:35:04.451 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:35:04.451 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:35:04.451 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:35:04.451 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:35:04.452 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:35:04.477 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:35:04.477 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:35:04.477 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:35:04.478 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:35:04.606 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:35:04.607 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:35:04.607 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:35:04.607 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:35:04.607 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:35:04.607 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:35:04.607 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:35:04.607 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:35:04.607 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:35:04.608 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:35:04.608 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:35:04.608 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:35:04.608 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:35:04.611 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:35:04.611 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:35:04.611 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:35:04.655 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 8 2024-04-06T09:35:04.656 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 11 2024-04-06T09:35:04.656 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:35:04.656 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=45, tag_tid=6, entry_tid=1], [object_number=44, tag_tid=6, entry_tid=0], [object_number=47, tag_tid=5, entry_tid=22415], [object_number=46, tag_tid=5, entry_tid=22414]]], state=connected] 2024-04-06T09:35:04.656 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=34, tag_tid=5, entry_tid=8318], [object_number=33, tag_tid=5, entry_tid=8317], [object_number=32, tag_tid=5, entry_tid=8316], [object_number=35, tag_tid=5, entry_tid=8315]]], state=connected] 2024-04-06T09:35:04.659 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:35:04.661 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=45, tag_tid=6, entry_tid=1' 2024-04-06T09:35:04.661 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:35:04.661 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:35:04.661 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:35:04.662 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:35:04.662 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:35:04.662 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:35:04.662 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:35:04.662 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:35:04.662 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:35:04.662 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:35:04.662 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:35:04.662 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:35:04.663 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:35:04.663 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:35:04.663 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:35:04.707 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 8 2024-04-06T09:35:04.707 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 11 2024-04-06T09:35:04.707 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:35:04.707 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=45, tag_tid=6, entry_tid=1], [object_number=44, tag_tid=6, entry_tid=0], [object_number=47, tag_tid=5, entry_tid=22415], [object_number=46, tag_tid=5, entry_tid=22414]]], state=connected] 2024-04-06T09:35:04.707 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=34, tag_tid=5, entry_tid=8318], [object_number=33, tag_tid=5, entry_tid=8317], [object_number=32, tag_tid=5, entry_tid=8316], [object_number=35, tag_tid=5, entry_tid=8315]]], state=connected] 2024-04-06T09:35:04.711 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:35:04.712 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=34, tag_tid=5, entry_tid=8318' 2024-04-06T09:35:04.712 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=45, tag_tid=6, entry_tid=1' -a 'object_number=45, tag_tid=6, entry_tid=1' = 'object_number=34, tag_tid=5, entry_tid=8318' 2024-04-06T09:35:04.712 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=34, tag_tid=5, entry_tid=8318' '!=' 'object_number=32, tag_tid=5, entry_tid=8224' 2024-04-06T09:35:04.712 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:35:04.713 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=34, tag_tid=5, entry_tid=8318' = 'object_number=32, tag_tid=5, entry_tid=8224' 2024-04-06T09:35:04.713 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=34, tag_tid=5, entry_tid=8318' 2024-04-06T09:35:04.713 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=45, tag_tid=6, entry_tid=1' 2024-04-06T09:35:04.713 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:35:04.713 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:35:04.714 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=6 2024-04-06T09:35:04.714 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=34, tag_tid=5, entry_tid=8318' 2024-04-06T09:35:04.715 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:35:04.715 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:35:04.715 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=5 2024-04-06T09:35:04.716 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=45, tag_tid=6, entry_tid=1' 2024-04-06T09:35:04.716 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:35:04.716 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:35:04.717 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:35:04.718 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=34, tag_tid=5, entry_tid=8318' 2024-04-06T09:35:04.718 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:35:04.718 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:35:04.719 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=8318 2024-04-06T09:35:04.719 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 6 = 5 -a 1 -le 8318 2024-04-06T09:35:04.719 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:35:04.719 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:35:04.719 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:35:04.921 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:35:04.921 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:35:04.921 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:35:04.921 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:35:04.921 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:35:04.921 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:35:04.922 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:35:04.922 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:35:04.922 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:35:04.922 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:35:04.922 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:35:04.922 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:35:04.922 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:35:04.922 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:35:04.922 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:35:04.923 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:35:04.923 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:35:04.925 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:35:04.925 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:35:04.925 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:35:04.925 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:35:04.925 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:35:04.925 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:35:04.925 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:35:04.925 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:35:04.925 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:35:04.925 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:35:04.926 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:35:04.926 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:35:04.926 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:35:04.926 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:35:04.926 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:35:04.926 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:35:04.927 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:35:04.927 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:35:04.927 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:35:04.927 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:35:04.928 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:35:04.928 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:35:04.928 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:35:04.928 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:35:04.928 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:35:04.928 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:35:04.928 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:35:04.928 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:35:04.928 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:35:04.928 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:35:04.928 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:35:04.928 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:35:04.952 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:35:04.952 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:35:04.952 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:35:04.953 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:35:05.085 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:35:05.086 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:35:05.086 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:35:05.086 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:35:05.086 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:35:05.086 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:35:05.086 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:35:05.086 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:35:05.086 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:35:05.087 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:35:05.087 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:35:05.087 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:35:05.087 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:35:05.088 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:35:05.089 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:35:05.089 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:35:05.133 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 8 2024-04-06T09:35:05.133 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 11 2024-04-06T09:35:05.133 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:35:05.133 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=45, tag_tid=6, entry_tid=1], [object_number=44, tag_tid=6, entry_tid=0], [object_number=47, tag_tid=5, entry_tid=22415], [object_number=46, tag_tid=5, entry_tid=22414]]], state=connected] 2024-04-06T09:35:05.133 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=32, tag_tid=5, entry_tid=8412], [object_number=35, tag_tid=5, entry_tid=8411], [object_number=34, tag_tid=5, entry_tid=8410], [object_number=33, tag_tid=5, entry_tid=8409]]], state=connected] 2024-04-06T09:35:05.137 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:35:05.138 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=45, tag_tid=6, entry_tid=1' 2024-04-06T09:35:05.139 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:35:05.139 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:35:05.139 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:35:05.139 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:35:05.139 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:35:05.140 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:35:05.140 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:35:05.140 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:35:05.140 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:35:05.140 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:35:05.140 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:35:05.140 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:35:05.141 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:35:05.141 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:35:05.141 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:35:05.186 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 8 2024-04-06T09:35:05.186 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 11 2024-04-06T09:35:05.186 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:35:05.186 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=45, tag_tid=6, entry_tid=1], [object_number=44, tag_tid=6, entry_tid=0], [object_number=47, tag_tid=5, entry_tid=22415], [object_number=46, tag_tid=5, entry_tid=22414]]], state=connected] 2024-04-06T09:35:05.186 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=32, tag_tid=5, entry_tid=8412], [object_number=35, tag_tid=5, entry_tid=8411], [object_number=34, tag_tid=5, entry_tid=8410], [object_number=33, tag_tid=5, entry_tid=8409]]], state=connected] 2024-04-06T09:35:05.189 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:35:05.191 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=32, tag_tid=5, entry_tid=8412' 2024-04-06T09:35:05.191 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=45, tag_tid=6, entry_tid=1' -a 'object_number=45, tag_tid=6, entry_tid=1' = 'object_number=32, tag_tid=5, entry_tid=8412' 2024-04-06T09:35:05.191 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=32, tag_tid=5, entry_tid=8412' '!=' 'object_number=34, tag_tid=5, entry_tid=8318' 2024-04-06T09:35:05.191 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:35:05.191 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=32, tag_tid=5, entry_tid=8412' = 'object_number=34, tag_tid=5, entry_tid=8318' 2024-04-06T09:35:05.191 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=32, tag_tid=5, entry_tid=8412' 2024-04-06T09:35:05.192 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=45, tag_tid=6, entry_tid=1' 2024-04-06T09:35:05.192 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:35:05.192 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:35:05.192 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=6 2024-04-06T09:35:05.193 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=32, tag_tid=5, entry_tid=8412' 2024-04-06T09:35:05.193 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:35:05.193 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:35:05.194 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=5 2024-04-06T09:35:05.195 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=45, tag_tid=6, entry_tid=1' 2024-04-06T09:35:05.195 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:35:05.195 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:35:05.196 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:35:05.196 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=32, tag_tid=5, entry_tid=8412' 2024-04-06T09:35:05.196 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:35:05.196 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:35:05.198 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=8412 2024-04-06T09:35:05.198 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 6 = 5 -a 1 -le 8412 2024-04-06T09:35:05.198 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:35:05.198 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:35:05.198 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:35:05.399 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:35:05.400 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:35:05.400 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:35:05.400 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:35:05.400 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:35:05.400 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:35:05.400 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:35:05.400 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:35:05.400 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:35:05.400 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:35:05.400 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:35:05.400 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:35:05.400 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:35:05.401 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:35:05.401 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:35:05.401 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:35:05.402 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:35:05.404 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:35:05.404 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:35:05.404 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:35:05.404 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:35:05.404 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:35:05.404 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:35:05.404 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:35:05.404 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:35:05.405 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:35:05.405 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:35:05.405 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:35:05.405 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:35:05.405 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:35:05.405 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:35:05.405 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:35:05.405 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:35:05.406 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:35:05.406 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:35:05.406 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:35:05.406 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:35:05.406 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:35:05.406 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:35:05.406 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:35:05.406 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:35:05.406 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:35:05.406 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:35:05.406 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:35:05.407 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:35:05.407 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:35:05.407 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:35:05.407 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:35:05.407 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:35:05.431 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:35:05.432 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:35:05.432 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:35:05.432 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:35:05.559 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:35:05.560 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:35:05.560 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:35:05.560 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:35:05.560 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:35:05.560 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:35:05.560 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:35:05.560 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:35:05.560 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:35:05.561 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:35:05.561 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:35:05.561 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:35:05.561 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:35:05.563 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:35:05.564 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:35:05.564 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:35:05.610 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 8 2024-04-06T09:35:05.610 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 11 2024-04-06T09:35:05.611 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:35:05.611 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=45, tag_tid=6, entry_tid=1], [object_number=44, tag_tid=6, entry_tid=0], [object_number=47, tag_tid=5, entry_tid=22415], [object_number=46, tag_tid=5, entry_tid=22414]]], state=connected] 2024-04-06T09:35:05.611 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=33, tag_tid=5, entry_tid=8505], [object_number=32, tag_tid=5, entry_tid=8504], [object_number=35, tag_tid=5, entry_tid=8503], [object_number=34, tag_tid=5, entry_tid=8502]]], state=connected] 2024-04-06T09:35:05.614 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:35:05.615 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=45, tag_tid=6, entry_tid=1' 2024-04-06T09:35:05.616 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:35:05.617 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:35:05.617 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:35:05.617 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:35:05.617 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:35:05.617 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:35:05.617 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:35:05.617 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:35:05.617 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:35:05.617 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:35:05.617 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:35:05.617 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:35:05.618 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:35:05.618 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:35:05.618 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:35:05.662 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 8 2024-04-06T09:35:05.662 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 11 2024-04-06T09:35:05.662 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:35:05.662 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=45, tag_tid=6, entry_tid=1], [object_number=44, tag_tid=6, entry_tid=0], [object_number=47, tag_tid=5, entry_tid=22415], [object_number=46, tag_tid=5, entry_tid=22414]]], state=connected] 2024-04-06T09:35:05.662 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=33, tag_tid=5, entry_tid=8505], [object_number=32, tag_tid=5, entry_tid=8504], [object_number=35, tag_tid=5, entry_tid=8503], [object_number=34, tag_tid=5, entry_tid=8502]]], state=connected] 2024-04-06T09:35:05.666 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:35:05.668 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=33, tag_tid=5, entry_tid=8505' 2024-04-06T09:35:05.668 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=45, tag_tid=6, entry_tid=1' -a 'object_number=45, tag_tid=6, entry_tid=1' = 'object_number=33, tag_tid=5, entry_tid=8505' 2024-04-06T09:35:05.668 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=33, tag_tid=5, entry_tid=8505' '!=' 'object_number=32, tag_tid=5, entry_tid=8412' 2024-04-06T09:35:05.669 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:35:05.669 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=33, tag_tid=5, entry_tid=8505' = 'object_number=32, tag_tid=5, entry_tid=8412' 2024-04-06T09:35:05.669 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=33, tag_tid=5, entry_tid=8505' 2024-04-06T09:35:05.669 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=45, tag_tid=6, entry_tid=1' 2024-04-06T09:35:05.669 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:35:05.669 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:35:05.670 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=6 2024-04-06T09:35:05.670 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=33, tag_tid=5, entry_tid=8505' 2024-04-06T09:35:05.670 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:35:05.670 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:35:05.671 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=5 2024-04-06T09:35:05.671 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=45, tag_tid=6, entry_tid=1' 2024-04-06T09:35:05.671 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:35:05.672 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:35:05.672 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:35:05.673 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=33, tag_tid=5, entry_tid=8505' 2024-04-06T09:35:05.673 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:35:05.673 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:35:05.674 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=8505 2024-04-06T09:35:05.674 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 6 = 5 -a 1 -le 8505 2024-04-06T09:35:05.675 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:35:05.675 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:35:05.675 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:35:05.877 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:35:05.877 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:35:05.877 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:35:05.877 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:35:05.877 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:35:05.877 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:35:05.877 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:35:05.877 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:35:05.877 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:35:05.877 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:35:05.877 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:35:05.877 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:35:05.877 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:35:05.877 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:35:05.878 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:35:05.878 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:35:05.878 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:35:05.881 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:35:05.881 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:35:05.881 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:35:05.881 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:35:05.881 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:35:05.882 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:35:05.882 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:35:05.882 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:35:05.882 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:35:05.882 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:35:05.882 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:35:05.882 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:35:05.882 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:35:05.882 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:35:05.882 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:35:05.882 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:35:05.882 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:35:05.882 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:35:05.883 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:35:05.883 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:35:05.883 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:35:05.883 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:35:05.883 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:35:05.883 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:35:05.883 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:35:05.883 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:35:05.883 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:35:05.883 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:35:05.883 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:35:05.883 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:35:05.883 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:35:05.883 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:35:05.907 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:35:05.907 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:35:05.907 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:35:05.907 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:35:06.030 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:35:06.030 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:35:06.030 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:35:06.031 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:35:06.031 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:35:06.031 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:35:06.031 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:35:06.031 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:35:06.031 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:35:06.031 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:35:06.031 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:35:06.032 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:35:06.032 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:35:06.033 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:35:06.033 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:35:06.033 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:35:06.084 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 8 2024-04-06T09:35:06.084 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 11 2024-04-06T09:35:06.084 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:35:06.084 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=45, tag_tid=6, entry_tid=1], [object_number=44, tag_tid=6, entry_tid=0], [object_number=47, tag_tid=5, entry_tid=22415], [object_number=46, tag_tid=5, entry_tid=22414]]], state=connected] 2024-04-06T09:35:06.084 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=35, tag_tid=5, entry_tid=8603], [object_number=34, tag_tid=5, entry_tid=8602], [object_number=33, tag_tid=5, entry_tid=8601], [object_number=32, tag_tid=5, entry_tid=8600]]], state=connected] 2024-04-06T09:35:06.088 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:35:06.089 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=45, tag_tid=6, entry_tid=1' 2024-04-06T09:35:06.090 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:35:06.091 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:35:06.091 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:35:06.091 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:35:06.091 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:35:06.091 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:35:06.091 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:35:06.091 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:35:06.091 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:35:06.091 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:35:06.091 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:35:06.091 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:35:06.092 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:35:06.092 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:35:06.092 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:35:06.138 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 8 2024-04-06T09:35:06.138 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 11 2024-04-06T09:35:06.138 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:35:06.138 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=45, tag_tid=6, entry_tid=1], [object_number=44, tag_tid=6, entry_tid=0], [object_number=47, tag_tid=5, entry_tid=22415], [object_number=46, tag_tid=5, entry_tid=22414]]], state=connected] 2024-04-06T09:35:06.138 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=35, tag_tid=5, entry_tid=8603], [object_number=34, tag_tid=5, entry_tid=8602], [object_number=33, tag_tid=5, entry_tid=8601], [object_number=32, tag_tid=5, entry_tid=8600]]], state=connected] 2024-04-06T09:35:06.142 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:35:06.143 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=35, tag_tid=5, entry_tid=8603' 2024-04-06T09:35:06.144 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=45, tag_tid=6, entry_tid=1' -a 'object_number=45, tag_tid=6, entry_tid=1' = 'object_number=35, tag_tid=5, entry_tid=8603' 2024-04-06T09:35:06.144 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=35, tag_tid=5, entry_tid=8603' '!=' 'object_number=33, tag_tid=5, entry_tid=8505' 2024-04-06T09:35:06.144 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:35:06.144 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=35, tag_tid=5, entry_tid=8603' = 'object_number=33, tag_tid=5, entry_tid=8505' 2024-04-06T09:35:06.144 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=35, tag_tid=5, entry_tid=8603' 2024-04-06T09:35:06.144 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=45, tag_tid=6, entry_tid=1' 2024-04-06T09:35:06.144 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:35:06.145 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:35:06.145 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=6 2024-04-06T09:35:06.146 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=35, tag_tid=5, entry_tid=8603' 2024-04-06T09:35:06.146 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:35:06.146 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:35:06.147 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=5 2024-04-06T09:35:06.147 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=45, tag_tid=6, entry_tid=1' 2024-04-06T09:35:06.147 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:35:06.148 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:35:06.149 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:35:06.149 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=35, tag_tid=5, entry_tid=8603' 2024-04-06T09:35:06.149 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:35:06.149 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:35:06.150 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=8603 2024-04-06T09:35:06.151 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 6 = 5 -a 1 -le 8603 2024-04-06T09:35:06.151 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:35:06.151 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:35:06.151 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:35:06.352 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:35:06.353 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:35:06.353 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:35:06.353 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:35:06.353 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:35:06.353 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:35:06.353 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:35:06.353 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:35:06.353 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:35:06.353 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:35:06.353 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:35:06.353 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:35:06.353 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:35:06.353 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:35:06.353 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:35:06.354 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:35:06.354 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:35:06.356 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:35:06.356 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:35:06.357 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:35:06.357 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:35:06.357 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:35:06.357 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:35:06.357 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:35:06.357 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:35:06.357 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:35:06.357 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:35:06.357 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:35:06.357 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:35:06.357 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:35:06.357 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:35:06.357 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:35:06.357 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:35:06.357 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:35:06.358 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:35:06.358 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:35:06.358 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:35:06.358 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:35:06.358 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:35:06.358 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:35:06.358 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:35:06.358 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:35:06.358 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:35:06.358 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:35:06.358 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:35:06.358 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:35:06.358 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:35:06.358 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:35:06.359 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:35:06.377 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.0 with skip (0.25003575592496774) > weight (0.25) 2024-04-06T09:35:06.378 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.1 with skip (0.64792638893565) > weight (0.25) 2024-04-06T09:35:06.378 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.2 with skip (0.935950399884712) > weight (0.25) 2024-04-06T09:35:06.378 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.3 with skip (0.3409060744804361) > weight (0.25) 2024-04-06T09:35:06.378 INFO:tasks.rbd_mirror_thrash:waiting for 83 secs before thrashing 2024-04-06T09:35:06.383 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:35:06.384 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:35:06.384 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:35:06.384 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:35:06.518 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:35:06.518 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:35:06.519 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:35:06.519 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:35:06.519 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:35:06.519 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:35:06.519 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:35:06.519 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:35:06.519 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:35:06.519 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:35:06.520 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:35:06.520 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:35:06.520 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:35:06.522 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:35:06.523 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:35:06.523 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:35:06.573 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 8 2024-04-06T09:35:06.573 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 11 2024-04-06T09:35:06.573 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:35:06.573 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=45, tag_tid=6, entry_tid=1], [object_number=44, tag_tid=6, entry_tid=0], [object_number=47, tag_tid=5, entry_tid=22415], [object_number=46, tag_tid=5, entry_tid=22414]]], state=connected] 2024-04-06T09:35:06.574 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=34, tag_tid=5, entry_tid=8690], [object_number=33, tag_tid=5, entry_tid=8689], [object_number=32, tag_tid=5, entry_tid=8688], [object_number=35, tag_tid=5, entry_tid=8687]]], state=connected] 2024-04-06T09:35:06.577 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:35:06.578 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=45, tag_tid=6, entry_tid=1' 2024-04-06T09:35:06.579 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:35:06.579 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:35:06.579 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:35:06.579 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:35:06.579 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:35:06.580 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:35:06.580 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:35:06.580 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:35:06.580 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:35:06.580 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:35:06.580 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:35:06.580 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:35:06.581 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:35:06.581 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:35:06.581 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:35:06.626 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 8 2024-04-06T09:35:06.627 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 11 2024-04-06T09:35:06.627 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:35:06.627 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=45, tag_tid=6, entry_tid=1], [object_number=44, tag_tid=6, entry_tid=0], [object_number=47, tag_tid=5, entry_tid=22415], [object_number=46, tag_tid=5, entry_tid=22414]]], state=connected] 2024-04-06T09:35:06.627 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=34, tag_tid=5, entry_tid=8690], [object_number=33, tag_tid=5, entry_tid=8689], [object_number=32, tag_tid=5, entry_tid=8688], [object_number=35, tag_tid=5, entry_tid=8687]]], state=connected] 2024-04-06T09:35:06.630 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:35:06.632 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=34, tag_tid=5, entry_tid=8690' 2024-04-06T09:35:06.632 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=45, tag_tid=6, entry_tid=1' -a 'object_number=45, tag_tid=6, entry_tid=1' = 'object_number=34, tag_tid=5, entry_tid=8690' 2024-04-06T09:35:06.632 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=34, tag_tid=5, entry_tid=8690' '!=' 'object_number=35, tag_tid=5, entry_tid=8603' 2024-04-06T09:35:06.632 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:35:06.632 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=34, tag_tid=5, entry_tid=8690' = 'object_number=35, tag_tid=5, entry_tid=8603' 2024-04-06T09:35:06.632 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=34, tag_tid=5, entry_tid=8690' 2024-04-06T09:35:06.633 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=45, tag_tid=6, entry_tid=1' 2024-04-06T09:35:06.633 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:35:06.633 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:35:06.633 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=6 2024-04-06T09:35:06.634 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=34, tag_tid=5, entry_tid=8690' 2024-04-06T09:35:06.634 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:35:06.634 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:35:06.635 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=5 2024-04-06T09:35:06.636 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=45, tag_tid=6, entry_tid=1' 2024-04-06T09:35:06.636 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:35:06.636 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:35:06.637 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:35:06.637 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=34, tag_tid=5, entry_tid=8690' 2024-04-06T09:35:06.637 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:35:06.638 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:35:06.639 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=8690 2024-04-06T09:35:06.639 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 6 = 5 -a 1 -le 8690 2024-04-06T09:35:06.639 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:35:06.639 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:35:06.639 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:35:06.841 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:35:06.841 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:35:06.841 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:35:06.841 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:35:06.841 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:35:06.841 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:35:06.842 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:35:06.842 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:35:06.842 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:35:06.842 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:35:06.842 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:35:06.842 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:35:06.842 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:35:06.842 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:35:06.842 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:35:06.843 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:35:06.843 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:35:06.846 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:35:06.846 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:35:06.846 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:35:06.846 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:35:06.846 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:35:06.846 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:35:06.846 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:35:06.847 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:35:06.847 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:35:06.847 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:35:06.847 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:35:06.847 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:35:06.847 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:35:06.847 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:35:06.847 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:35:06.847 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:35:06.847 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:35:06.847 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:35:06.847 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:35:06.847 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:35:06.847 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:35:06.847 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:35:06.848 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:35:06.848 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:35:06.848 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:35:06.848 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:35:06.848 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:35:06.848 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:35:06.848 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:35:06.848 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:35:06.848 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:35:06.848 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:35:06.872 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:35:06.872 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:35:06.872 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:35:06.872 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:35:07.001 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:35:07.002 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:35:07.002 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:35:07.003 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:35:07.003 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:35:07.003 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:35:07.003 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:35:07.003 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:35:07.003 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:35:07.003 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:35:07.003 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:35:07.003 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:35:07.004 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:35:07.006 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:35:07.006 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:35:07.006 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:35:07.056 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 8 2024-04-06T09:35:07.056 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 11 2024-04-06T09:35:07.056 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:35:07.056 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=45, tag_tid=6, entry_tid=1], [object_number=44, tag_tid=6, entry_tid=0], [object_number=47, tag_tid=5, entry_tid=22415], [object_number=46, tag_tid=5, entry_tid=22414]]], state=connected] 2024-04-06T09:35:07.056 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=32, tag_tid=5, entry_tid=8788], [object_number=35, tag_tid=5, entry_tid=8787], [object_number=34, tag_tid=5, entry_tid=8786], [object_number=33, tag_tid=5, entry_tid=8785]]], state=connected] 2024-04-06T09:35:07.060 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:35:07.061 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=45, tag_tid=6, entry_tid=1' 2024-04-06T09:35:07.062 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:35:07.062 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:35:07.062 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:35:07.063 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:35:07.063 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:35:07.063 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:35:07.063 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:35:07.063 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:35:07.063 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:35:07.063 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:35:07.063 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:35:07.063 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:35:07.064 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:35:07.064 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:35:07.064 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:35:07.109 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 8 2024-04-06T09:35:07.109 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 11 2024-04-06T09:35:07.109 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:35:07.109 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=45, tag_tid=6, entry_tid=1], [object_number=44, tag_tid=6, entry_tid=0], [object_number=47, tag_tid=5, entry_tid=22415], [object_number=46, tag_tid=5, entry_tid=22414]]], state=connected] 2024-04-06T09:35:07.109 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=32, tag_tid=5, entry_tid=8788], [object_number=35, tag_tid=5, entry_tid=8787], [object_number=34, tag_tid=5, entry_tid=8786], [object_number=33, tag_tid=5, entry_tid=8785]]], state=connected] 2024-04-06T09:35:07.113 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:35:07.115 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=32, tag_tid=5, entry_tid=8788' 2024-04-06T09:35:07.115 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=45, tag_tid=6, entry_tid=1' -a 'object_number=45, tag_tid=6, entry_tid=1' = 'object_number=32, tag_tid=5, entry_tid=8788' 2024-04-06T09:35:07.115 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=32, tag_tid=5, entry_tid=8788' '!=' 'object_number=34, tag_tid=5, entry_tid=8690' 2024-04-06T09:35:07.115 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:35:07.115 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=32, tag_tid=5, entry_tid=8788' = 'object_number=34, tag_tid=5, entry_tid=8690' 2024-04-06T09:35:07.115 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=32, tag_tid=5, entry_tid=8788' 2024-04-06T09:35:07.115 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=45, tag_tid=6, entry_tid=1' 2024-04-06T09:35:07.115 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:35:07.115 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:35:07.116 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=6 2024-04-06T09:35:07.117 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=32, tag_tid=5, entry_tid=8788' 2024-04-06T09:35:07.117 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:35:07.117 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:35:07.118 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=5 2024-04-06T09:35:07.118 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=45, tag_tid=6, entry_tid=1' 2024-04-06T09:35:07.118 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:35:07.118 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:35:07.119 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:35:07.120 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=32, tag_tid=5, entry_tid=8788' 2024-04-06T09:35:07.120 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:35:07.120 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:35:07.121 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=8788 2024-04-06T09:35:07.122 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 6 = 5 -a 1 -le 8788 2024-04-06T09:35:07.122 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:35:07.122 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:35:07.122 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:35:07.323 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:35:07.324 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:35:07.324 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:35:07.324 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:35:07.324 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:35:07.324 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:35:07.324 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:35:07.324 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:35:07.324 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:35:07.324 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:35:07.324 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:35:07.324 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:35:07.325 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:35:07.325 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:35:07.325 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:35:07.325 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:35:07.325 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:35:07.329 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:35:07.329 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:35:07.330 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:35:07.330 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:35:07.330 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:35:07.330 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:35:07.330 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:35:07.330 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:35:07.330 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:35:07.330 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:35:07.330 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:35:07.330 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:35:07.330 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:35:07.331 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:35:07.331 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:35:07.331 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:35:07.331 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:35:07.331 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:35:07.331 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:35:07.331 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:35:07.331 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:35:07.331 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:35:07.331 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:35:07.331 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:35:07.332 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:35:07.332 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:35:07.332 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:35:07.332 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:35:07.332 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:35:07.332 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:35:07.332 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:35:07.332 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:35:07.355 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:35:07.355 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:35:07.356 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:35:07.356 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:35:07.488 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:35:07.489 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:35:07.489 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:35:07.489 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:35:07.489 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:35:07.489 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:35:07.489 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:35:07.490 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:35:07.490 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:35:07.490 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:35:07.490 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:35:07.491 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:35:07.491 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:35:07.492 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:35:07.492 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:35:07.492 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:35:07.537 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 8 2024-04-06T09:35:07.537 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 11 2024-04-06T09:35:07.537 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:35:07.537 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=45, tag_tid=6, entry_tid=1], [object_number=44, tag_tid=6, entry_tid=0], [object_number=47, tag_tid=5, entry_tid=22415], [object_number=46, tag_tid=5, entry_tid=22414]]], state=connected] 2024-04-06T09:35:07.537 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=35, tag_tid=5, entry_tid=8883], [object_number=34, tag_tid=5, entry_tid=8882], [object_number=33, tag_tid=5, entry_tid=8881], [object_number=32, tag_tid=5, entry_tid=8880]]], state=connected] 2024-04-06T09:35:07.540 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:35:07.543 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=45, tag_tid=6, entry_tid=1' 2024-04-06T09:35:07.543 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:35:07.543 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:35:07.543 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:35:07.543 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:35:07.543 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:35:07.543 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:35:07.544 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:35:07.544 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:35:07.544 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:35:07.544 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:35:07.544 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:35:07.544 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:35:07.544 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:35:07.545 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:35:07.545 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:35:07.590 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 8 2024-04-06T09:35:07.590 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 11 2024-04-06T09:35:07.590 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:35:07.590 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=45, tag_tid=6, entry_tid=1], [object_number=44, tag_tid=6, entry_tid=0], [object_number=47, tag_tid=5, entry_tid=22415], [object_number=46, tag_tid=5, entry_tid=22414]]], state=connected] 2024-04-06T09:35:07.591 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=35, tag_tid=5, entry_tid=8883], [object_number=34, tag_tid=5, entry_tid=8882], [object_number=33, tag_tid=5, entry_tid=8881], [object_number=32, tag_tid=5, entry_tid=8880]]], state=connected] 2024-04-06T09:35:07.594 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:35:07.596 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=35, tag_tid=5, entry_tid=8883' 2024-04-06T09:35:07.596 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=45, tag_tid=6, entry_tid=1' -a 'object_number=45, tag_tid=6, entry_tid=1' = 'object_number=35, tag_tid=5, entry_tid=8883' 2024-04-06T09:35:07.596 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=35, tag_tid=5, entry_tid=8883' '!=' 'object_number=32, tag_tid=5, entry_tid=8788' 2024-04-06T09:35:07.596 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:35:07.596 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=35, tag_tid=5, entry_tid=8883' = 'object_number=32, tag_tid=5, entry_tid=8788' 2024-04-06T09:35:07.596 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=35, tag_tid=5, entry_tid=8883' 2024-04-06T09:35:07.596 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=45, tag_tid=6, entry_tid=1' 2024-04-06T09:35:07.596 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:35:07.596 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:35:07.597 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=6 2024-04-06T09:35:07.597 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=35, tag_tid=5, entry_tid=8883' 2024-04-06T09:35:07.597 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:35:07.597 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:35:07.599 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=5 2024-04-06T09:35:07.599 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=45, tag_tid=6, entry_tid=1' 2024-04-06T09:35:07.599 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:35:07.600 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:35:07.600 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:35:07.601 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=35, tag_tid=5, entry_tid=8883' 2024-04-06T09:35:07.601 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:35:07.602 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:35:07.603 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=8883 2024-04-06T09:35:07.603 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 6 = 5 -a 1 -le 8883 2024-04-06T09:35:07.603 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:35:07.603 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:35:07.603 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:35:07.804 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:35:07.804 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:35:07.804 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:35:07.804 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:35:07.804 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:35:07.804 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:35:07.805 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:35:07.805 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:35:07.805 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:35:07.805 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:35:07.805 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:35:07.805 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:35:07.805 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:35:07.805 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:35:07.805 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:35:07.805 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:35:07.806 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:35:07.808 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:35:07.808 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:35:07.808 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:35:07.808 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:35:07.808 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:35:07.808 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:35:07.808 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:35:07.809 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:35:07.809 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:35:07.809 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:35:07.809 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:35:07.809 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:35:07.809 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:35:07.809 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:35:07.809 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:35:07.809 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:35:07.810 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:35:07.810 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:35:07.810 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:35:07.810 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:35:07.810 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:35:07.810 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:35:07.810 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:35:07.811 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:35:07.811 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:35:07.811 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:35:07.811 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:35:07.811 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:35:07.811 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:35:07.811 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:35:07.811 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:35:07.811 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:35:07.835 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:35:07.836 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:35:07.836 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:35:07.836 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:35:07.966 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:35:07.967 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:35:07.967 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:35:07.967 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:35:07.967 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:35:07.967 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:35:07.968 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:35:07.968 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:35:07.968 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:35:07.968 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:35:07.968 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:35:07.968 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:35:07.968 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:35:07.970 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:35:07.971 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:35:07.971 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:35:08.021 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 8 2024-04-06T09:35:08.021 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 11 2024-04-06T09:35:08.021 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:35:08.021 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=45, tag_tid=6, entry_tid=1], [object_number=44, tag_tid=6, entry_tid=0], [object_number=47, tag_tid=5, entry_tid=22415], [object_number=46, tag_tid=5, entry_tid=22414]]], state=connected] 2024-04-06T09:35:08.021 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=35, tag_tid=5, entry_tid=8975], [object_number=34, tag_tid=5, entry_tid=8974], [object_number=33, tag_tid=5, entry_tid=8973], [object_number=32, tag_tid=5, entry_tid=8972]]], state=connected] 2024-04-06T09:35:08.024 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:35:08.027 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=45, tag_tid=6, entry_tid=1' 2024-04-06T09:35:08.027 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:35:08.027 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:35:08.027 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:35:08.027 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:35:08.027 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:35:08.027 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:35:08.027 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:35:08.027 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:35:08.028 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:35:08.028 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:35:08.028 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:35:08.028 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:35:08.028 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:35:08.029 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:35:08.029 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:35:08.074 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 8 2024-04-06T09:35:08.074 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 11 2024-04-06T09:35:08.074 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:35:08.074 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=45, tag_tid=6, entry_tid=1], [object_number=44, tag_tid=6, entry_tid=0], [object_number=47, tag_tid=5, entry_tid=22415], [object_number=46, tag_tid=5, entry_tid=22414]]], state=connected] 2024-04-06T09:35:08.074 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=35, tag_tid=5, entry_tid=8975], [object_number=34, tag_tid=5, entry_tid=8974], [object_number=33, tag_tid=5, entry_tid=8973], [object_number=32, tag_tid=5, entry_tid=8972]]], state=connected] 2024-04-06T09:35:08.078 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:35:08.080 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=35, tag_tid=5, entry_tid=8975' 2024-04-06T09:35:08.080 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=45, tag_tid=6, entry_tid=1' -a 'object_number=45, tag_tid=6, entry_tid=1' = 'object_number=35, tag_tid=5, entry_tid=8975' 2024-04-06T09:35:08.080 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=35, tag_tid=5, entry_tid=8975' '!=' 'object_number=35, tag_tid=5, entry_tid=8883' 2024-04-06T09:35:08.080 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:35:08.080 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=35, tag_tid=5, entry_tid=8975' = 'object_number=35, tag_tid=5, entry_tid=8883' 2024-04-06T09:35:08.080 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=35, tag_tid=5, entry_tid=8975' 2024-04-06T09:35:08.081 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=45, tag_tid=6, entry_tid=1' 2024-04-06T09:35:08.081 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:35:08.081 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:35:08.081 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=6 2024-04-06T09:35:08.082 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=35, tag_tid=5, entry_tid=8975' 2024-04-06T09:35:08.082 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:35:08.082 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:35:08.083 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=5 2024-04-06T09:35:08.083 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=45, tag_tid=6, entry_tid=1' 2024-04-06T09:35:08.083 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:35:08.083 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:35:08.084 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:35:08.085 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=35, tag_tid=5, entry_tid=8975' 2024-04-06T09:35:08.085 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:35:08.085 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:35:08.086 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=8975 2024-04-06T09:35:08.086 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 6 = 5 -a 1 -le 8975 2024-04-06T09:35:08.087 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:35:08.087 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:35:08.087 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:35:08.288 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:35:08.288 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:35:08.288 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:35:08.288 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:35:08.288 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:35:08.289 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:35:08.289 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:35:08.289 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:35:08.289 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:35:08.289 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:35:08.289 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:35:08.289 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:35:08.289 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:35:08.289 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:35:08.289 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:35:08.290 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:35:08.290 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:35:08.292 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:35:08.292 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:35:08.292 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:35:08.292 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:35:08.292 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:35:08.292 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:35:08.293 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:35:08.293 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:35:08.293 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:35:08.293 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:35:08.293 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:35:08.293 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:35:08.293 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:35:08.293 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:35:08.293 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:35:08.293 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:35:08.294 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:35:08.294 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:35:08.294 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:35:08.294 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:35:08.294 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:35:08.294 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:35:08.294 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:35:08.294 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:35:08.294 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:35:08.295 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:35:08.295 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:35:08.295 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:35:08.295 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:35:08.295 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:35:08.295 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:35:08.295 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:35:08.319 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:35:08.319 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:35:08.319 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:35:08.319 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:35:08.452 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:35:08.453 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:35:08.453 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:35:08.453 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:35:08.453 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:35:08.453 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:35:08.453 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:35:08.453 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:35:08.453 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:35:08.453 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:35:08.454 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:35:08.454 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:35:08.454 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:35:08.456 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:35:08.456 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:35:08.456 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:35:08.501 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 8 2024-04-06T09:35:08.501 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 11 2024-04-06T09:35:08.501 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:35:08.501 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=45, tag_tid=6, entry_tid=1], [object_number=44, tag_tid=6, entry_tid=0], [object_number=47, tag_tid=5, entry_tid=22415], [object_number=46, tag_tid=5, entry_tid=22414]]], state=connected] 2024-04-06T09:35:08.502 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=32, tag_tid=5, entry_tid=9064], [object_number=35, tag_tid=5, entry_tid=9063], [object_number=34, tag_tid=5, entry_tid=9062], [object_number=33, tag_tid=5, entry_tid=9061]]], state=connected] 2024-04-06T09:35:08.505 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:35:08.506 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=45, tag_tid=6, entry_tid=1' 2024-04-06T09:35:08.507 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:35:08.507 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:35:08.507 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:35:08.507 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:35:08.507 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:35:08.508 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:35:08.508 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:35:08.508 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:35:08.508 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:35:08.508 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:35:08.508 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:35:08.508 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:35:08.509 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:35:08.509 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:35:08.509 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:35:08.553 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 8 2024-04-06T09:35:08.553 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 11 2024-04-06T09:35:08.553 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:35:08.553 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=45, tag_tid=6, entry_tid=1], [object_number=44, tag_tid=6, entry_tid=0], [object_number=47, tag_tid=5, entry_tid=22415], [object_number=46, tag_tid=5, entry_tid=22414]]], state=connected] 2024-04-06T09:35:08.553 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=32, tag_tid=5, entry_tid=9064], [object_number=35, tag_tid=5, entry_tid=9063], [object_number=34, tag_tid=5, entry_tid=9062], [object_number=33, tag_tid=5, entry_tid=9061]]], state=connected] 2024-04-06T09:35:08.556 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:35:08.558 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=32, tag_tid=5, entry_tid=9064' 2024-04-06T09:35:08.558 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=45, tag_tid=6, entry_tid=1' -a 'object_number=45, tag_tid=6, entry_tid=1' = 'object_number=32, tag_tid=5, entry_tid=9064' 2024-04-06T09:35:08.558 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=32, tag_tid=5, entry_tid=9064' '!=' 'object_number=35, tag_tid=5, entry_tid=8975' 2024-04-06T09:35:08.558 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:35:08.559 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=32, tag_tid=5, entry_tid=9064' = 'object_number=35, tag_tid=5, entry_tid=8975' 2024-04-06T09:35:08.559 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=32, tag_tid=5, entry_tid=9064' 2024-04-06T09:35:08.559 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=45, tag_tid=6, entry_tid=1' 2024-04-06T09:35:08.559 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:35:08.559 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:35:08.560 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=6 2024-04-06T09:35:08.560 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=32, tag_tid=5, entry_tid=9064' 2024-04-06T09:35:08.560 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:35:08.561 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:35:08.561 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=5 2024-04-06T09:35:08.562 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=45, tag_tid=6, entry_tid=1' 2024-04-06T09:35:08.562 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:35:08.562 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:35:08.563 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:35:08.564 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=32, tag_tid=5, entry_tid=9064' 2024-04-06T09:35:08.564 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:35:08.564 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:35:08.565 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=9064 2024-04-06T09:35:08.565 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 6 = 5 -a 1 -le 9064 2024-04-06T09:35:08.565 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:35:08.565 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:35:08.565 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:35:08.767 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:35:08.767 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:35:08.767 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:35:08.767 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:35:08.767 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:35:08.768 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:35:08.768 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:35:08.768 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:35:08.768 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:35:08.768 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:35:08.768 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:35:08.768 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:35:08.768 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:35:08.768 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:35:08.768 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:35:08.768 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:35:08.769 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:35:08.771 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:35:08.771 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:35:08.771 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:35:08.772 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:35:08.772 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:35:08.772 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:35:08.772 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:35:08.772 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:35:08.772 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:35:08.772 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:35:08.772 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:35:08.772 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:35:08.772 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:35:08.772 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:35:08.772 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:35:08.772 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:35:08.773 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:35:08.773 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:35:08.773 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:35:08.773 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:35:08.773 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:35:08.773 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:35:08.774 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:35:08.774 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:35:08.774 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:35:08.774 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:35:08.774 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:35:08.774 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:35:08.774 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:35:08.774 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:35:08.774 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:35:08.774 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:35:08.799 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:35:08.799 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:35:08.799 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:35:08.799 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:35:08.935 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:35:08.936 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:35:08.936 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:35:08.936 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:35:08.936 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:35:08.936 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:35:08.936 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:35:08.937 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:35:08.937 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:35:08.937 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:35:08.937 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:35:08.937 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:35:08.937 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:35:08.939 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:35:08.940 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:35:08.940 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:35:08.985 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 8 2024-04-06T09:35:08.985 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 11 2024-04-06T09:35:08.986 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:35:08.986 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=45, tag_tid=6, entry_tid=1], [object_number=44, tag_tid=6, entry_tid=0], [object_number=47, tag_tid=5, entry_tid=22415], [object_number=46, tag_tid=5, entry_tid=22414]]], state=connected] 2024-04-06T09:35:08.986 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=32, tag_tid=5, entry_tid=9160], [object_number=35, tag_tid=5, entry_tid=9159], [object_number=34, tag_tid=5, entry_tid=9158], [object_number=33, tag_tid=5, entry_tid=9157]]], state=connected] 2024-04-06T09:35:08.989 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:35:08.990 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=45, tag_tid=6, entry_tid=1' 2024-04-06T09:35:08.991 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:35:08.991 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:35:08.991 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:35:08.991 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:35:08.991 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:35:08.992 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:35:08.992 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:35:08.992 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:35:08.992 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:35:08.992 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:35:08.992 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:35:08.992 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:35:08.992 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:35:08.993 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:35:08.993 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:35:09.037 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 8 2024-04-06T09:35:09.037 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 11 2024-04-06T09:35:09.037 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:35:09.037 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=45, tag_tid=6, entry_tid=1], [object_number=44, tag_tid=6, entry_tid=0], [object_number=47, tag_tid=5, entry_tid=22415], [object_number=46, tag_tid=5, entry_tid=22414]]], state=connected] 2024-04-06T09:35:09.037 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=32, tag_tid=5, entry_tid=9160], [object_number=35, tag_tid=5, entry_tid=9159], [object_number=34, tag_tid=5, entry_tid=9158], [object_number=33, tag_tid=5, entry_tid=9157]]], state=connected] 2024-04-06T09:35:09.040 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:35:09.042 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=32, tag_tid=5, entry_tid=9160' 2024-04-06T09:35:09.042 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=45, tag_tid=6, entry_tid=1' -a 'object_number=45, tag_tid=6, entry_tid=1' = 'object_number=32, tag_tid=5, entry_tid=9160' 2024-04-06T09:35:09.042 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=32, tag_tid=5, entry_tid=9160' '!=' 'object_number=32, tag_tid=5, entry_tid=9064' 2024-04-06T09:35:09.042 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:35:09.042 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=32, tag_tid=5, entry_tid=9160' = 'object_number=32, tag_tid=5, entry_tid=9064' 2024-04-06T09:35:09.042 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=32, tag_tid=5, entry_tid=9160' 2024-04-06T09:35:09.043 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=45, tag_tid=6, entry_tid=1' 2024-04-06T09:35:09.043 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:35:09.043 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:35:09.043 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=6 2024-04-06T09:35:09.044 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=32, tag_tid=5, entry_tid=9160' 2024-04-06T09:35:09.044 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:35:09.044 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:35:09.045 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=5 2024-04-06T09:35:09.046 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=45, tag_tid=6, entry_tid=1' 2024-04-06T09:35:09.046 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:35:09.046 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:35:09.047 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:35:09.047 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=32, tag_tid=5, entry_tid=9160' 2024-04-06T09:35:09.048 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:35:09.048 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:35:09.049 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=9160 2024-04-06T09:35:09.049 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 6 = 5 -a 1 -le 9160 2024-04-06T09:35:09.049 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:35:09.049 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:35:09.049 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:35:09.250 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:35:09.250 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:35:09.251 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:35:09.251 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:35:09.251 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:35:09.251 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:35:09.251 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:35:09.251 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:35:09.251 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:35:09.251 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:35:09.251 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:35:09.251 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:35:09.251 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:35:09.251 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:35:09.251 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:35:09.252 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:35:09.252 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:35:09.254 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:35:09.254 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:35:09.255 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:35:09.255 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:35:09.255 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:35:09.255 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:35:09.255 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:35:09.255 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:35:09.255 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:35:09.255 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:35:09.255 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:35:09.255 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:35:09.255 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:35:09.255 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:35:09.255 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:35:09.255 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:35:09.256 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:35:09.256 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:35:09.257 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:35:09.257 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:35:09.257 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:35:09.257 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:35:09.257 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:35:09.257 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:35:09.257 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:35:09.257 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:35:09.257 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:35:09.257 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:35:09.257 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:35:09.257 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:35:09.257 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:35:09.257 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:35:09.282 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:35:09.283 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:35:09.283 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:35:09.283 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:35:09.412 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:35:09.412 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:35:09.413 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:35:09.413 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:35:09.413 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:35:09.413 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:35:09.413 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:35:09.413 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:35:09.413 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:35:09.413 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:35:09.414 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:35:09.414 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:35:09.414 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:35:09.416 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:35:09.416 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:35:09.416 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:35:09.467 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 8 2024-04-06T09:35:09.467 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 11 2024-04-06T09:35:09.467 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:35:09.467 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=45, tag_tid=6, entry_tid=1], [object_number=44, tag_tid=6, entry_tid=0], [object_number=47, tag_tid=5, entry_tid=22415], [object_number=46, tag_tid=5, entry_tid=22414]]], state=connected] 2024-04-06T09:35:09.467 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=33, tag_tid=5, entry_tid=9253], [object_number=32, tag_tid=5, entry_tid=9252], [object_number=35, tag_tid=5, entry_tid=9251], [object_number=34, tag_tid=5, entry_tid=9250]]], state=connected] 2024-04-06T09:35:09.471 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:35:09.473 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=45, tag_tid=6, entry_tid=1' 2024-04-06T09:35:09.473 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:35:09.473 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:35:09.473 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:35:09.473 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:35:09.474 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:35:09.474 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:35:09.474 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:35:09.474 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:35:09.474 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:35:09.474 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:35:09.474 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:35:09.474 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:35:09.475 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:35:09.475 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:35:09.475 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:35:09.520 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 8 2024-04-06T09:35:09.520 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 11 2024-04-06T09:35:09.520 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:35:09.520 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=45, tag_tid=6, entry_tid=1], [object_number=44, tag_tid=6, entry_tid=0], [object_number=47, tag_tid=5, entry_tid=22415], [object_number=46, tag_tid=5, entry_tid=22414]]], state=connected] 2024-04-06T09:35:09.520 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=33, tag_tid=5, entry_tid=9253], [object_number=32, tag_tid=5, entry_tid=9252], [object_number=35, tag_tid=5, entry_tid=9251], [object_number=34, tag_tid=5, entry_tid=9250]]], state=connected] 2024-04-06T09:35:09.524 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:35:09.525 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=33, tag_tid=5, entry_tid=9253' 2024-04-06T09:35:09.525 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=45, tag_tid=6, entry_tid=1' -a 'object_number=45, tag_tid=6, entry_tid=1' = 'object_number=33, tag_tid=5, entry_tid=9253' 2024-04-06T09:35:09.525 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=33, tag_tid=5, entry_tid=9253' '!=' 'object_number=32, tag_tid=5, entry_tid=9160' 2024-04-06T09:35:09.525 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:35:09.525 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=33, tag_tid=5, entry_tid=9253' = 'object_number=32, tag_tid=5, entry_tid=9160' 2024-04-06T09:35:09.525 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=33, tag_tid=5, entry_tid=9253' 2024-04-06T09:35:09.526 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=45, tag_tid=6, entry_tid=1' 2024-04-06T09:35:09.526 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:35:09.526 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:35:09.527 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=6 2024-04-06T09:35:09.528 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=33, tag_tid=5, entry_tid=9253' 2024-04-06T09:35:09.528 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:35:09.528 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:35:09.528 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=5 2024-04-06T09:35:09.529 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=45, tag_tid=6, entry_tid=1' 2024-04-06T09:35:09.529 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:35:09.529 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:35:09.530 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:35:09.531 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=33, tag_tid=5, entry_tid=9253' 2024-04-06T09:35:09.531 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:35:09.531 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:35:09.532 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=9253 2024-04-06T09:35:09.532 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 6 = 5 -a 1 -le 9253 2024-04-06T09:35:09.532 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:35:09.532 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:35:09.532 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:35:09.734 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:35:09.734 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:35:09.734 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:35:09.734 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:35:09.734 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:35:09.735 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:35:09.735 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:35:09.735 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:35:09.735 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:35:09.735 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:35:09.735 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:35:09.735 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:35:09.735 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:35:09.735 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:35:09.735 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:35:09.736 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:35:09.736 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:35:09.739 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:35:09.739 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:35:09.739 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:35:09.739 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:35:09.739 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:35:09.740 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:35:09.740 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:35:09.740 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:35:09.740 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:35:09.740 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:35:09.740 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:35:09.740 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:35:09.740 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:35:09.740 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:35:09.740 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:35:09.740 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:35:09.740 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:35:09.740 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:35:09.740 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:35:09.741 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:35:09.741 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:35:09.741 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:35:09.741 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:35:09.741 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:35:09.741 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:35:09.741 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:35:09.741 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:35:09.741 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:35:09.741 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:35:09.741 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:35:09.741 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:35:09.741 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:35:09.764 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:35:09.765 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:35:09.765 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:35:09.765 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:35:09.894 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:35:09.896 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:35:09.896 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:35:09.896 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:35:09.896 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:35:09.896 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:35:09.896 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:35:09.896 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:35:09.896 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:35:09.896 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:35:09.896 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:35:09.897 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:35:09.897 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:35:09.897 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:35:09.898 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:35:09.898 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:35:09.942 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 8 2024-04-06T09:35:09.943 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 11 2024-04-06T09:35:09.943 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:35:09.943 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=45, tag_tid=6, entry_tid=1], [object_number=44, tag_tid=6, entry_tid=0], [object_number=47, tag_tid=5, entry_tid=22415], [object_number=46, tag_tid=5, entry_tid=22414]]], state=connected] 2024-04-06T09:35:09.943 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=34, tag_tid=5, entry_tid=9346], [object_number=33, tag_tid=5, entry_tid=9345], [object_number=32, tag_tid=5, entry_tid=9344], [object_number=35, tag_tid=5, entry_tid=9343]]], state=connected] 2024-04-06T09:35:09.946 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:35:09.947 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=45, tag_tid=6, entry_tid=1' 2024-04-06T09:35:09.949 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:35:09.949 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:35:09.949 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:35:09.949 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:35:09.949 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:35:09.949 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:35:09.949 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:35:09.949 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:35:09.949 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:35:09.949 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:35:09.949 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:35:09.949 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:35:09.950 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:35:09.950 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:35:09.950 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:35:09.994 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 8 2024-04-06T09:35:09.995 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 11 2024-04-06T09:35:09.995 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:35:09.995 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=45, tag_tid=6, entry_tid=1], [object_number=44, tag_tid=6, entry_tid=0], [object_number=47, tag_tid=5, entry_tid=22415], [object_number=46, tag_tid=5, entry_tid=22414]]], state=connected] 2024-04-06T09:35:09.995 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=34, tag_tid=5, entry_tid=9346], [object_number=33, tag_tid=5, entry_tid=9345], [object_number=32, tag_tid=5, entry_tid=9344], [object_number=35, tag_tid=5, entry_tid=9343]]], state=connected] 2024-04-06T09:35:09.998 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:35:10.000 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=34, tag_tid=5, entry_tid=9346' 2024-04-06T09:35:10.000 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=45, tag_tid=6, entry_tid=1' -a 'object_number=45, tag_tid=6, entry_tid=1' = 'object_number=34, tag_tid=5, entry_tid=9346' 2024-04-06T09:35:10.001 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=34, tag_tid=5, entry_tid=9346' '!=' 'object_number=33, tag_tid=5, entry_tid=9253' 2024-04-06T09:35:10.001 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:35:10.001 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=34, tag_tid=5, entry_tid=9346' = 'object_number=33, tag_tid=5, entry_tid=9253' 2024-04-06T09:35:10.001 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=34, tag_tid=5, entry_tid=9346' 2024-04-06T09:35:10.001 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=45, tag_tid=6, entry_tid=1' 2024-04-06T09:35:10.001 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:35:10.001 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:35:10.001 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=6 2024-04-06T09:35:10.002 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=34, tag_tid=5, entry_tid=9346' 2024-04-06T09:35:10.002 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:35:10.002 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:35:10.003 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=5 2024-04-06T09:35:10.004 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=45, tag_tid=6, entry_tid=1' 2024-04-06T09:35:10.004 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:35:10.004 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:35:10.005 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:35:10.006 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=34, tag_tid=5, entry_tid=9346' 2024-04-06T09:35:10.006 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:35:10.006 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:35:10.007 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=9346 2024-04-06T09:35:10.007 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 6 = 5 -a 1 -le 9346 2024-04-06T09:35:10.007 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:35:10.007 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:35:10.007 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:35:10.209 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:35:10.209 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:35:10.209 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:35:10.209 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:35:10.209 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:35:10.209 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:35:10.209 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:35:10.209 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:35:10.209 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:35:10.210 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:35:10.210 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:35:10.210 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:35:10.210 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:35:10.210 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:35:10.210 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:35:10.210 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:35:10.211 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:35:10.212 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:35:10.212 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:35:10.212 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:35:10.212 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:35:10.213 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:35:10.213 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:35:10.213 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:35:10.213 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:35:10.213 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:35:10.213 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:35:10.213 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:35:10.213 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:35:10.213 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:35:10.213 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:35:10.213 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:35:10.213 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:35:10.215 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:35:10.215 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:35:10.215 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:35:10.215 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:35:10.215 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:35:10.215 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:35:10.215 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:35:10.215 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:35:10.215 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:35:10.215 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:35:10.215 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:35:10.215 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:35:10.215 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:35:10.215 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:35:10.216 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:35:10.216 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:35:10.239 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:35:10.239 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:35:10.239 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:35:10.239 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:35:10.370 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:35:10.371 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:35:10.371 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:35:10.371 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:35:10.371 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:35:10.371 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:35:10.371 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:35:10.371 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:35:10.371 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:35:10.372 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:35:10.372 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:35:10.372 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:35:10.372 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:35:10.375 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:35:10.375 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:35:10.375 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:35:10.421 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 8 2024-04-06T09:35:10.421 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 11 2024-04-06T09:35:10.421 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:35:10.422 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=45, tag_tid=6, entry_tid=1], [object_number=44, tag_tid=6, entry_tid=0], [object_number=47, tag_tid=5, entry_tid=22415], [object_number=46, tag_tid=5, entry_tid=22414]]], state=connected] 2024-04-06T09:35:10.422 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=35, tag_tid=5, entry_tid=9435], [object_number=34, tag_tid=5, entry_tid=9434], [object_number=33, tag_tid=5, entry_tid=9433], [object_number=32, tag_tid=5, entry_tid=9432]]], state=connected] 2024-04-06T09:35:10.425 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:35:10.426 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=45, tag_tid=6, entry_tid=1' 2024-04-06T09:35:10.427 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:35:10.427 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:35:10.427 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:35:10.428 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:35:10.428 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:35:10.428 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:35:10.428 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:35:10.428 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:35:10.428 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:35:10.428 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:35:10.428 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:35:10.428 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:35:10.429 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:35:10.429 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:35:10.429 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:35:10.474 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 8 2024-04-06T09:35:10.475 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 11 2024-04-06T09:35:10.475 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:35:10.475 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=45, tag_tid=6, entry_tid=1], [object_number=44, tag_tid=6, entry_tid=0], [object_number=47, tag_tid=5, entry_tid=22415], [object_number=46, tag_tid=5, entry_tid=22414]]], state=connected] 2024-04-06T09:35:10.475 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=35, tag_tid=5, entry_tid=9435], [object_number=34, tag_tid=5, entry_tid=9434], [object_number=33, tag_tid=5, entry_tid=9433], [object_number=32, tag_tid=5, entry_tid=9432]]], state=connected] 2024-04-06T09:35:10.478 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:35:10.480 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=35, tag_tid=5, entry_tid=9435' 2024-04-06T09:35:10.480 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=45, tag_tid=6, entry_tid=1' -a 'object_number=45, tag_tid=6, entry_tid=1' = 'object_number=35, tag_tid=5, entry_tid=9435' 2024-04-06T09:35:10.480 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=35, tag_tid=5, entry_tid=9435' '!=' 'object_number=34, tag_tid=5, entry_tid=9346' 2024-04-06T09:35:10.480 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:35:10.480 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=35, tag_tid=5, entry_tid=9435' = 'object_number=34, tag_tid=5, entry_tid=9346' 2024-04-06T09:35:10.480 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=35, tag_tid=5, entry_tid=9435' 2024-04-06T09:35:10.480 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=45, tag_tid=6, entry_tid=1' 2024-04-06T09:35:10.480 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:35:10.480 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:35:10.482 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=6 2024-04-06T09:35:10.482 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=35, tag_tid=5, entry_tid=9435' 2024-04-06T09:35:10.482 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:35:10.482 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:35:10.483 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=5 2024-04-06T09:35:10.484 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=45, tag_tid=6, entry_tid=1' 2024-04-06T09:35:10.484 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:35:10.484 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:35:10.485 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:35:10.485 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=35, tag_tid=5, entry_tid=9435' 2024-04-06T09:35:10.485 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:35:10.486 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:35:10.487 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=9435 2024-04-06T09:35:10.487 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 6 = 5 -a 1 -le 9435 2024-04-06T09:35:10.487 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:35:10.487 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:35:10.487 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:35:10.689 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:35:10.689 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:35:10.689 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:35:10.689 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:35:10.689 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:35:10.689 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:35:10.689 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:35:10.689 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:35:10.689 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:35:10.689 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:35:10.689 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:35:10.689 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:35:10.690 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:35:10.690 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:35:10.690 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:35:10.690 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:35:10.690 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:35:10.693 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:35:10.694 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:35:10.694 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:35:10.694 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:35:10.694 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:35:10.694 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:35:10.694 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:35:10.694 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:35:10.694 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:35:10.694 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:35:10.694 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:35:10.694 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:35:10.694 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:35:10.695 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:35:10.695 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:35:10.695 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:35:10.695 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:35:10.695 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:35:10.695 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:35:10.695 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:35:10.695 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:35:10.695 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:35:10.695 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:35:10.695 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:35:10.695 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:35:10.695 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:35:10.696 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:35:10.696 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:35:10.696 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:35:10.696 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:35:10.696 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:35:10.696 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:35:10.719 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:35:10.719 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:35:10.719 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:35:10.719 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:35:10.850 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:35:10.852 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:35:10.852 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:35:10.852 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:35:10.852 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:35:10.852 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:35:10.852 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:35:10.852 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:35:10.852 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:35:10.852 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:35:10.853 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:35:10.853 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:35:10.853 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:35:10.854 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:35:10.854 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:35:10.855 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:35:10.905 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 8 2024-04-06T09:35:10.905 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 11 2024-04-06T09:35:10.905 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:35:10.905 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=45, tag_tid=6, entry_tid=1], [object_number=44, tag_tid=6, entry_tid=0], [object_number=47, tag_tid=5, entry_tid=22415], [object_number=46, tag_tid=5, entry_tid=22414]]], state=connected] 2024-04-06T09:35:10.905 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=34, tag_tid=5, entry_tid=9530], [object_number=33, tag_tid=5, entry_tid=9529], [object_number=32, tag_tid=5, entry_tid=9528], [object_number=35, tag_tid=5, entry_tid=9527]]], state=connected] 2024-04-06T09:35:10.909 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:35:10.910 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=45, tag_tid=6, entry_tid=1' 2024-04-06T09:35:10.911 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:35:10.911 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:35:10.911 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:35:10.912 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:35:10.912 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:35:10.912 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:35:10.912 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:35:10.912 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:35:10.912 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:35:10.912 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:35:10.912 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:35:10.912 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:35:10.913 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:35:10.913 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:35:10.913 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:35:10.959 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 8 2024-04-06T09:35:10.959 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 11 2024-04-06T09:35:10.959 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:35:10.960 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=45, tag_tid=6, entry_tid=1], [object_number=44, tag_tid=6, entry_tid=0], [object_number=47, tag_tid=5, entry_tid=22415], [object_number=46, tag_tid=5, entry_tid=22414]]], state=connected] 2024-04-06T09:35:10.960 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=34, tag_tid=5, entry_tid=9530], [object_number=33, tag_tid=5, entry_tid=9529], [object_number=32, tag_tid=5, entry_tid=9528], [object_number=35, tag_tid=5, entry_tid=9527]]], state=connected] 2024-04-06T09:35:10.963 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:35:10.965 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=34, tag_tid=5, entry_tid=9530' 2024-04-06T09:35:10.965 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=45, tag_tid=6, entry_tid=1' -a 'object_number=45, tag_tid=6, entry_tid=1' = 'object_number=34, tag_tid=5, entry_tid=9530' 2024-04-06T09:35:10.965 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=34, tag_tid=5, entry_tid=9530' '!=' 'object_number=35, tag_tid=5, entry_tid=9435' 2024-04-06T09:35:10.965 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:35:10.965 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=34, tag_tid=5, entry_tid=9530' = 'object_number=35, tag_tid=5, entry_tid=9435' 2024-04-06T09:35:10.965 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=34, tag_tid=5, entry_tid=9530' 2024-04-06T09:35:10.965 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=45, tag_tid=6, entry_tid=1' 2024-04-06T09:35:10.965 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:35:10.965 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:35:10.966 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=6 2024-04-06T09:35:10.967 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=34, tag_tid=5, entry_tid=9530' 2024-04-06T09:35:10.967 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:35:10.967 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:35:10.968 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=5 2024-04-06T09:35:10.968 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=45, tag_tid=6, entry_tid=1' 2024-04-06T09:35:10.968 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:35:10.968 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:35:10.969 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:35:10.970 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=34, tag_tid=5, entry_tid=9530' 2024-04-06T09:35:10.970 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:35:10.970 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:35:10.971 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=9530 2024-04-06T09:35:10.971 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 6 = 5 -a 1 -le 9530 2024-04-06T09:35:10.971 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:35:10.971 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:35:10.972 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:35:11.173 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:35:11.173 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:35:11.174 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:35:11.174 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:35:11.174 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:35:11.174 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:35:11.174 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:35:11.174 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:35:11.174 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:35:11.174 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:35:11.174 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:35:11.174 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:35:11.174 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:35:11.174 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:35:11.174 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:35:11.175 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:35:11.175 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:35:11.176 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:35:11.176 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:35:11.176 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:35:11.176 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:35:11.176 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:35:11.176 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:35:11.177 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:35:11.177 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:35:11.177 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:35:11.177 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:35:11.177 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:35:11.177 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:35:11.177 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:35:11.177 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:35:11.177 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:35:11.177 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:35:11.179 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:35:11.179 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:35:11.179 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:35:11.179 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:35:11.179 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:35:11.179 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:35:11.179 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:35:11.179 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:35:11.179 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:35:11.179 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:35:11.179 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:35:11.180 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:35:11.180 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:35:11.180 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:35:11.180 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:35:11.180 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:35:11.204 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:35:11.204 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:35:11.204 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:35:11.204 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:35:11.330 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:35:11.331 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:35:11.331 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:35:11.331 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:35:11.331 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:35:11.331 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:35:11.331 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:35:11.331 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:35:11.331 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:35:11.331 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:35:11.332 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:35:11.332 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:35:11.332 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:35:11.335 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:35:11.335 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:35:11.335 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:35:11.385 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 8 2024-04-06T09:35:11.385 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 11 2024-04-06T09:35:11.385 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:35:11.385 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=45, tag_tid=6, entry_tid=1], [object_number=44, tag_tid=6, entry_tid=0], [object_number=47, tag_tid=5, entry_tid=22415], [object_number=46, tag_tid=5, entry_tid=22414]]], state=connected] 2024-04-06T09:35:11.385 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=33, tag_tid=5, entry_tid=9621], [object_number=32, tag_tid=5, entry_tid=9620], [object_number=35, tag_tid=5, entry_tid=9619], [object_number=34, tag_tid=5, entry_tid=9618]]], state=connected] 2024-04-06T09:35:11.389 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:35:11.390 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=45, tag_tid=6, entry_tid=1' 2024-04-06T09:35:11.391 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:35:11.391 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:35:11.392 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:35:11.392 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:35:11.392 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:35:11.392 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:35:11.392 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:35:11.392 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:35:11.392 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:35:11.392 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:35:11.392 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:35:11.392 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:35:11.393 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:35:11.393 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:35:11.393 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:35:11.438 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 8 2024-04-06T09:35:11.438 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 11 2024-04-06T09:35:11.439 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:35:11.439 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=45, tag_tid=6, entry_tid=1], [object_number=44, tag_tid=6, entry_tid=0], [object_number=47, tag_tid=5, entry_tid=22415], [object_number=46, tag_tid=5, entry_tid=22414]]], state=connected] 2024-04-06T09:35:11.439 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=33, tag_tid=5, entry_tid=9621], [object_number=32, tag_tid=5, entry_tid=9620], [object_number=35, tag_tid=5, entry_tid=9619], [object_number=34, tag_tid=5, entry_tid=9618]]], state=connected] 2024-04-06T09:35:11.442 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:35:11.444 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=33, tag_tid=5, entry_tid=9621' 2024-04-06T09:35:11.444 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=45, tag_tid=6, entry_tid=1' -a 'object_number=45, tag_tid=6, entry_tid=1' = 'object_number=33, tag_tid=5, entry_tid=9621' 2024-04-06T09:35:11.444 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=33, tag_tid=5, entry_tid=9621' '!=' 'object_number=34, tag_tid=5, entry_tid=9530' 2024-04-06T09:35:11.444 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:35:11.444 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=33, tag_tid=5, entry_tid=9621' = 'object_number=34, tag_tid=5, entry_tid=9530' 2024-04-06T09:35:11.444 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=33, tag_tid=5, entry_tid=9621' 2024-04-06T09:35:11.444 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=45, tag_tid=6, entry_tid=1' 2024-04-06T09:35:11.445 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:35:11.445 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:35:11.445 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=6 2024-04-06T09:35:11.446 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=33, tag_tid=5, entry_tid=9621' 2024-04-06T09:35:11.446 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:35:11.446 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:35:11.447 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=5 2024-04-06T09:35:11.448 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=45, tag_tid=6, entry_tid=1' 2024-04-06T09:35:11.448 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:35:11.448 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:35:11.449 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:35:11.449 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=33, tag_tid=5, entry_tid=9621' 2024-04-06T09:35:11.449 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:35:11.450 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:35:11.451 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=9621 2024-04-06T09:35:11.451 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 6 = 5 -a 1 -le 9621 2024-04-06T09:35:11.451 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:35:11.451 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:35:11.451 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:35:11.652 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:35:11.653 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:35:11.653 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:35:11.653 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:35:11.653 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:35:11.653 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:35:11.653 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:35:11.653 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:35:11.653 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:35:11.653 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:35:11.653 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:35:11.653 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:35:11.653 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:35:11.653 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:35:11.653 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:35:11.654 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:35:11.654 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:35:11.656 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:35:11.656 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:35:11.656 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:35:11.656 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:35:11.656 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:35:11.657 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:35:11.657 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:35:11.657 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:35:11.657 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:35:11.657 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:35:11.657 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:35:11.657 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:35:11.657 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:35:11.657 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:35:11.657 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:35:11.657 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:35:11.659 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:35:11.659 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:35:11.659 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:35:11.659 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:35:11.659 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:35:11.659 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:35:11.659 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:35:11.659 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:35:11.659 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:35:11.659 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:35:11.659 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:35:11.660 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:35:11.660 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:35:11.660 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:35:11.660 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:35:11.660 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:35:11.684 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:35:11.684 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:35:11.684 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:35:11.685 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:35:11.814 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:35:11.815 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:35:11.815 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:35:11.815 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:35:11.815 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:35:11.815 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:35:11.815 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:35:11.815 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:35:11.815 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:35:11.815 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:35:11.816 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:35:11.816 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:35:11.816 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:35:11.817 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:35:11.818 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:35:11.818 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:35:11.869 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 8 2024-04-06T09:35:11.869 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 11 2024-04-06T09:35:11.869 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:35:11.869 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=45, tag_tid=6, entry_tid=1], [object_number=44, tag_tid=6, entry_tid=0], [object_number=47, tag_tid=5, entry_tid=22415], [object_number=46, tag_tid=5, entry_tid=22414]]], state=connected] 2024-04-06T09:35:11.869 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=34, tag_tid=5, entry_tid=9714], [object_number=33, tag_tid=5, entry_tid=9713], [object_number=32, tag_tid=5, entry_tid=9712], [object_number=35, tag_tid=5, entry_tid=9711]]], state=connected] 2024-04-06T09:35:11.873 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:35:11.876 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=45, tag_tid=6, entry_tid=1' 2024-04-06T09:35:11.876 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:35:11.876 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:35:11.876 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:35:11.876 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:35:11.876 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:35:11.876 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:35:11.876 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:35:11.876 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:35:11.876 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:35:11.876 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:35:11.876 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:35:11.876 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:35:11.877 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:35:11.877 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:35:11.877 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:35:12.135 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 8 2024-04-06T09:35:12.135 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 11 2024-04-06T09:35:12.135 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:35:12.135 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=45, tag_tid=6, entry_tid=1], [object_number=44, tag_tid=6, entry_tid=0], [object_number=47, tag_tid=5, entry_tid=22415], [object_number=46, tag_tid=5, entry_tid=22414]]], state=connected] 2024-04-06T09:35:12.135 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=34, tag_tid=5, entry_tid=9714], [object_number=33, tag_tid=5, entry_tid=9713], [object_number=32, tag_tid=5, entry_tid=9712], [object_number=35, tag_tid=5, entry_tid=9711]]], state=connected] 2024-04-06T09:35:12.139 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:35:12.141 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=34, tag_tid=5, entry_tid=9714' 2024-04-06T09:35:12.141 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=45, tag_tid=6, entry_tid=1' -a 'object_number=45, tag_tid=6, entry_tid=1' = 'object_number=34, tag_tid=5, entry_tid=9714' 2024-04-06T09:35:12.141 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=34, tag_tid=5, entry_tid=9714' '!=' 'object_number=33, tag_tid=5, entry_tid=9621' 2024-04-06T09:35:12.142 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:35:12.142 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=34, tag_tid=5, entry_tid=9714' = 'object_number=33, tag_tid=5, entry_tid=9621' 2024-04-06T09:35:12.142 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=34, tag_tid=5, entry_tid=9714' 2024-04-06T09:35:12.142 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=45, tag_tid=6, entry_tid=1' 2024-04-06T09:35:12.142 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:35:12.142 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:35:12.144 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=6 2024-04-06T09:35:12.144 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=34, tag_tid=5, entry_tid=9714' 2024-04-06T09:35:12.144 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:35:12.145 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:35:12.146 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=5 2024-04-06T09:35:12.147 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=45, tag_tid=6, entry_tid=1' 2024-04-06T09:35:12.147 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:35:12.147 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:35:12.148 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:35:12.149 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=34, tag_tid=5, entry_tid=9714' 2024-04-06T09:35:12.149 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:35:12.149 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:35:12.151 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=9714 2024-04-06T09:35:12.151 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 6 = 5 -a 1 -le 9714 2024-04-06T09:35:12.151 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:35:12.151 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:35:12.151 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:35:12.352 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:35:12.352 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:35:12.352 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:35:12.352 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:35:12.352 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:35:12.353 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:35:12.353 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:35:12.353 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:35:12.353 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:35:12.353 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:35:12.353 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:35:12.353 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:35:12.353 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:35:12.353 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:35:12.353 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:35:12.354 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:35:12.354 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:35:12.356 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:35:12.356 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:35:12.356 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:35:12.356 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:35:12.356 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:35:12.356 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:35:12.356 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:35:12.356 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:35:12.356 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:35:12.356 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:35:12.356 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:35:12.356 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:35:12.356 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:35:12.357 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:35:12.357 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:35:12.357 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:35:12.357 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:35:12.357 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:35:12.358 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:35:12.358 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:35:12.358 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:35:12.358 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:35:12.358 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:35:12.358 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:35:12.358 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:35:12.358 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:35:12.358 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:35:12.358 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:35:12.358 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:35:12.358 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:35:12.358 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:35:12.358 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:35:12.383 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:35:12.383 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:35:12.383 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:35:12.383 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:35:12.518 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:35:12.520 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:35:12.520 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:35:12.520 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:35:12.520 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:35:12.520 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:35:12.520 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:35:12.520 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:35:12.520 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:35:12.520 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:35:12.520 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:35:12.521 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:35:12.521 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:35:12.522 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:35:12.523 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:35:12.523 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:35:12.775 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 8 2024-04-06T09:35:12.775 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 11 2024-04-06T09:35:12.775 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:35:12.775 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=45, tag_tid=6, entry_tid=1], [object_number=44, tag_tid=6, entry_tid=0], [object_number=47, tag_tid=5, entry_tid=22415], [object_number=46, tag_tid=5, entry_tid=22414]]], state=connected] 2024-04-06T09:35:12.775 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=34, tag_tid=5, entry_tid=9846], [object_number=33, tag_tid=5, entry_tid=9845], [object_number=32, tag_tid=5, entry_tid=9844], [object_number=35, tag_tid=5, entry_tid=9843]]], state=connected] 2024-04-06T09:35:12.779 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:35:12.781 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=45, tag_tid=6, entry_tid=1' 2024-04-06T09:35:12.782 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:35:12.782 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:35:12.782 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:35:12.782 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:35:12.782 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:35:12.782 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:35:12.782 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:35:12.782 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:35:12.782 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:35:12.783 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:35:12.783 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:35:12.783 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:35:12.784 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:35:12.784 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:35:12.784 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:35:13.045 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 8 2024-04-06T09:35:13.045 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 11 2024-04-06T09:35:13.046 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:35:13.046 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=45, tag_tid=6, entry_tid=1], [object_number=44, tag_tid=6, entry_tid=0], [object_number=47, tag_tid=5, entry_tid=22415], [object_number=46, tag_tid=5, entry_tid=22414]]], state=connected] 2024-04-06T09:35:13.046 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=34, tag_tid=5, entry_tid=9846], [object_number=33, tag_tid=5, entry_tid=9845], [object_number=32, tag_tid=5, entry_tid=9844], [object_number=35, tag_tid=5, entry_tid=9843]]], state=connected] 2024-04-06T09:35:13.051 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:35:13.054 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=34, tag_tid=5, entry_tid=9846' 2024-04-06T09:35:13.054 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=45, tag_tid=6, entry_tid=1' -a 'object_number=45, tag_tid=6, entry_tid=1' = 'object_number=34, tag_tid=5, entry_tid=9846' 2024-04-06T09:35:13.054 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=34, tag_tid=5, entry_tid=9846' '!=' 'object_number=34, tag_tid=5, entry_tid=9714' 2024-04-06T09:35:13.054 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:35:13.054 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=34, tag_tid=5, entry_tid=9846' = 'object_number=34, tag_tid=5, entry_tid=9714' 2024-04-06T09:35:13.054 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=34, tag_tid=5, entry_tid=9846' 2024-04-06T09:35:13.055 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=45, tag_tid=6, entry_tid=1' 2024-04-06T09:35:13.055 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:35:13.055 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:35:13.057 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=6 2024-04-06T09:35:13.058 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=34, tag_tid=5, entry_tid=9846' 2024-04-06T09:35:13.058 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:35:13.058 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:35:13.060 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=5 2024-04-06T09:35:13.061 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=45, tag_tid=6, entry_tid=1' 2024-04-06T09:35:13.061 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:35:13.061 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:35:13.062 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:35:13.063 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=34, tag_tid=5, entry_tid=9846' 2024-04-06T09:35:13.063 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:35:13.063 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:35:13.065 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=9846 2024-04-06T09:35:13.065 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 6 = 5 -a 1 -le 9846 2024-04-06T09:35:13.065 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:35:13.065 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:35:13.065 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:35:13.267 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:35:13.267 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:35:13.267 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:35:13.267 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:35:13.267 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:35:13.267 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:35:13.268 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:35:13.268 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:35:13.268 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:35:13.268 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:35:13.268 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:35:13.268 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:35:13.268 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:35:13.268 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:35:13.268 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:35:13.268 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:35:13.269 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:35:13.272 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:35:13.272 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:35:13.272 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:35:13.272 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:35:13.272 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:35:13.272 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:35:13.272 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:35:13.272 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:35:13.273 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:35:13.273 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:35:13.273 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:35:13.273 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:35:13.273 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:35:13.273 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:35:13.273 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:35:13.273 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:35:13.273 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:35:13.273 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:35:13.273 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:35:13.273 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:35:13.273 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:35:13.273 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:35:13.274 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:35:13.274 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:35:13.274 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:35:13.274 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:35:13.274 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:35:13.274 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:35:13.274 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:35:13.274 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:35:13.274 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:35:13.274 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:35:13.299 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:35:13.299 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:35:13.299 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:35:13.299 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:35:13.427 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:35:13.428 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:35:13.428 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:35:13.428 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:35:13.428 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:35:13.428 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:35:13.428 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:35:13.428 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:35:13.429 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:35:13.429 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:35:13.429 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:35:13.429 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:35:13.430 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:35:13.431 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:35:13.432 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:35:13.432 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:35:13.477 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 8 2024-04-06T09:35:13.477 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 11 2024-04-06T09:35:13.477 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:35:13.477 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=45, tag_tid=6, entry_tid=1], [object_number=44, tag_tid=6, entry_tid=0], [object_number=47, tag_tid=5, entry_tid=22415], [object_number=46, tag_tid=5, entry_tid=22414]]], state=connected] 2024-04-06T09:35:13.477 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=35, tag_tid=5, entry_tid=10023], [object_number=34, tag_tid=5, entry_tid=10022], [object_number=33, tag_tid=5, entry_tid=10021], [object_number=32, tag_tid=5, entry_tid=10020]]], state=connected] 2024-04-06T09:35:13.481 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:35:13.482 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=45, tag_tid=6, entry_tid=1' 2024-04-06T09:35:13.483 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:35:13.483 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:35:13.483 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:35:13.483 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:35:13.484 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:35:13.484 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:35:13.484 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:35:13.484 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:35:13.484 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:35:13.484 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:35:13.484 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:35:13.484 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:35:13.484 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:35:13.485 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:35:13.485 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:35:13.529 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 8 2024-04-06T09:35:13.530 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 11 2024-04-06T09:35:13.530 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:35:13.530 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=45, tag_tid=6, entry_tid=1], [object_number=44, tag_tid=6, entry_tid=0], [object_number=47, tag_tid=5, entry_tid=22415], [object_number=46, tag_tid=5, entry_tid=22414]]], state=connected] 2024-04-06T09:35:13.530 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=35, tag_tid=5, entry_tid=10023], [object_number=34, tag_tid=5, entry_tid=10022], [object_number=33, tag_tid=5, entry_tid=10021], [object_number=32, tag_tid=5, entry_tid=10020]]], state=connected] 2024-04-06T09:35:13.533 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:35:13.535 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=35, tag_tid=5, entry_tid=10023' 2024-04-06T09:35:13.535 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=45, tag_tid=6, entry_tid=1' -a 'object_number=45, tag_tid=6, entry_tid=1' = 'object_number=35, tag_tid=5, entry_tid=10023' 2024-04-06T09:35:13.536 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=35, tag_tid=5, entry_tid=10023' '!=' 'object_number=34, tag_tid=5, entry_tid=9846' 2024-04-06T09:35:13.536 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:35:13.536 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=35, tag_tid=5, entry_tid=10023' = 'object_number=34, tag_tid=5, entry_tid=9846' 2024-04-06T09:35:13.536 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=35, tag_tid=5, entry_tid=10023' 2024-04-06T09:35:13.536 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=45, tag_tid=6, entry_tid=1' 2024-04-06T09:35:13.536 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:35:13.536 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:35:13.536 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=6 2024-04-06T09:35:13.537 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=35, tag_tid=5, entry_tid=10023' 2024-04-06T09:35:13.537 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:35:13.537 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:35:13.538 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=5 2024-04-06T09:35:13.539 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=45, tag_tid=6, entry_tid=1' 2024-04-06T09:35:13.539 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:35:13.539 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:35:13.540 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:35:13.541 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=35, tag_tid=5, entry_tid=10023' 2024-04-06T09:35:13.541 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:35:13.541 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:35:13.542 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=10023 2024-04-06T09:35:13.542 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 6 = 5 -a 1 -le 10023 2024-04-06T09:35:13.542 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:35:13.542 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:35:13.542 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:35:13.744 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:35:13.744 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:35:13.744 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:35:13.744 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:35:13.744 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:35:13.744 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:35:13.745 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:35:13.745 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:35:13.745 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:35:13.745 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:35:13.745 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:35:13.745 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:35:13.745 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:35:13.745 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:35:13.745 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:35:13.746 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:35:13.746 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:35:13.749 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:35:13.749 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:35:13.749 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:35:13.749 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:35:13.749 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:35:13.749 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:35:13.749 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:35:13.749 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:35:13.749 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:35:13.749 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:35:13.749 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:35:13.749 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:35:13.750 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:35:13.750 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:35:13.750 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:35:13.750 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:35:13.750 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:35:13.750 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:35:13.750 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:35:13.750 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:35:13.750 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:35:13.750 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:35:13.750 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:35:13.750 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:35:13.750 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:35:13.750 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:35:13.751 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:35:13.751 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:35:13.751 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:35:13.751 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:35:13.751 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:35:13.751 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:35:13.774 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:35:13.775 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:35:13.775 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:35:13.775 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:35:13.910 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:35:13.911 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:35:13.911 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:35:13.912 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:35:13.912 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:35:13.912 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:35:13.912 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:35:13.912 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:35:13.912 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:35:13.912 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:35:13.912 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:35:13.913 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:35:13.913 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:35:13.914 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:35:13.914 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:35:13.914 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:35:14.174 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 8 2024-04-06T09:35:14.174 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 11 2024-04-06T09:35:14.174 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:35:14.174 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=45, tag_tid=6, entry_tid=1], [object_number=44, tag_tid=6, entry_tid=0], [object_number=47, tag_tid=5, entry_tid=22415], [object_number=46, tag_tid=5, entry_tid=22414]]], state=connected] 2024-04-06T09:35:14.174 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=32, tag_tid=5, entry_tid=10088], [object_number=35, tag_tid=5, entry_tid=10087], [object_number=34, tag_tid=5, entry_tid=10086], [object_number=33, tag_tid=5, entry_tid=10085]]], state=connected] 2024-04-06T09:35:14.178 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:35:14.180 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=45, tag_tid=6, entry_tid=1' 2024-04-06T09:35:14.181 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:35:14.181 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:35:14.181 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:35:14.181 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:35:14.181 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:35:14.181 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:35:14.181 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:35:14.181 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:35:14.181 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:35:14.181 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:35:14.181 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:35:14.182 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:35:14.182 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:35:14.182 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:35:14.183 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:35:14.231 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 8 2024-04-06T09:35:14.231 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 11 2024-04-06T09:35:14.231 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:35:14.231 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=45, tag_tid=6, entry_tid=1], [object_number=44, tag_tid=6, entry_tid=0], [object_number=47, tag_tid=5, entry_tid=22415], [object_number=46, tag_tid=5, entry_tid=22414]]], state=connected] 2024-04-06T09:35:14.231 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=32, tag_tid=5, entry_tid=10088], [object_number=35, tag_tid=5, entry_tid=10087], [object_number=34, tag_tid=5, entry_tid=10086], [object_number=33, tag_tid=5, entry_tid=10085]]], state=connected] 2024-04-06T09:35:14.235 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:35:14.237 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=32, tag_tid=5, entry_tid=10088' 2024-04-06T09:35:14.237 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=45, tag_tid=6, entry_tid=1' -a 'object_number=45, tag_tid=6, entry_tid=1' = 'object_number=32, tag_tid=5, entry_tid=10088' 2024-04-06T09:35:14.237 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=32, tag_tid=5, entry_tid=10088' '!=' 'object_number=35, tag_tid=5, entry_tid=10023' 2024-04-06T09:35:14.237 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:35:14.237 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=32, tag_tid=5, entry_tid=10088' = 'object_number=35, tag_tid=5, entry_tid=10023' 2024-04-06T09:35:14.237 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=32, tag_tid=5, entry_tid=10088' 2024-04-06T09:35:14.238 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=45, tag_tid=6, entry_tid=1' 2024-04-06T09:35:14.238 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:35:14.238 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:35:14.238 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=6 2024-04-06T09:35:14.239 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=32, tag_tid=5, entry_tid=10088' 2024-04-06T09:35:14.239 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:35:14.239 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:35:14.240 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=5 2024-04-06T09:35:14.241 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=45, tag_tid=6, entry_tid=1' 2024-04-06T09:35:14.241 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:35:14.241 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:35:14.242 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:35:14.243 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=32, tag_tid=5, entry_tid=10088' 2024-04-06T09:35:14.243 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:35:14.243 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:35:14.244 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=10088 2024-04-06T09:35:14.244 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 6 = 5 -a 1 -le 10088 2024-04-06T09:35:14.244 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:35:14.244 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:35:14.244 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:35:14.447 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:35:14.447 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:35:14.447 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:35:14.447 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:35:14.447 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:35:14.447 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:35:14.447 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:35:14.447 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:35:14.447 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:35:14.447 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:35:14.447 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:35:14.448 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:35:14.448 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:35:14.448 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:35:14.448 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:35:14.448 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:35:14.448 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:35:14.450 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:35:14.450 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:35:14.450 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:35:14.450 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:35:14.450 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:35:14.450 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:35:14.450 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:35:14.450 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:35:14.450 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:35:14.451 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:35:14.451 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:35:14.451 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:35:14.451 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:35:14.451 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:35:14.451 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:35:14.451 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:35:14.453 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:35:14.453 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:35:14.453 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:35:14.453 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:35:14.453 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:35:14.453 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:35:14.453 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:35:14.453 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:35:14.453 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:35:14.453 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:35:14.454 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:35:14.454 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:35:14.454 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:35:14.454 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:35:14.454 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:35:14.454 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:35:14.477 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:35:14.477 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:35:14.477 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:35:14.477 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:35:14.606 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:35:14.608 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:35:14.608 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:35:14.608 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:35:14.608 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:35:14.608 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:35:14.608 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:35:14.608 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:35:14.608 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:35:14.608 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:35:14.608 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:35:14.608 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:35:14.608 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:35:14.609 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:35:14.609 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:35:14.609 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:35:14.660 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 8 2024-04-06T09:35:14.660 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 11 2024-04-06T09:35:14.660 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:35:14.660 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=45, tag_tid=6, entry_tid=1], [object_number=44, tag_tid=6, entry_tid=0], [object_number=47, tag_tid=5, entry_tid=22415], [object_number=46, tag_tid=5, entry_tid=22414]]], state=connected] 2024-04-06T09:35:14.660 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=32, tag_tid=5, entry_tid=10204], [object_number=35, tag_tid=5, entry_tid=10203], [object_number=34, tag_tid=5, entry_tid=10202], [object_number=33, tag_tid=5, entry_tid=10201]]], state=connected] 2024-04-06T09:35:14.664 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:35:14.665 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=45, tag_tid=6, entry_tid=1' 2024-04-06T09:35:14.666 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:35:14.666 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:35:14.666 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:35:14.666 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:35:14.666 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:35:14.666 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:35:14.666 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:35:14.667 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:35:14.667 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:35:14.667 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:35:14.667 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:35:14.667 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:35:14.667 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:35:14.668 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:35:14.668 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:35:14.713 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 8 2024-04-06T09:35:14.713 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 11 2024-04-06T09:35:14.713 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:35:14.713 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=45, tag_tid=6, entry_tid=1], [object_number=44, tag_tid=6, entry_tid=0], [object_number=47, tag_tid=5, entry_tid=22415], [object_number=46, tag_tid=5, entry_tid=22414]]], state=connected] 2024-04-06T09:35:14.713 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=32, tag_tid=5, entry_tid=10204], [object_number=35, tag_tid=5, entry_tid=10203], [object_number=34, tag_tid=5, entry_tid=10202], [object_number=33, tag_tid=5, entry_tid=10201]]], state=connected] 2024-04-06T09:35:14.716 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:35:14.718 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=32, tag_tid=5, entry_tid=10204' 2024-04-06T09:35:14.719 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=45, tag_tid=6, entry_tid=1' -a 'object_number=45, tag_tid=6, entry_tid=1' = 'object_number=32, tag_tid=5, entry_tid=10204' 2024-04-06T09:35:14.719 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=32, tag_tid=5, entry_tid=10204' '!=' 'object_number=32, tag_tid=5, entry_tid=10088' 2024-04-06T09:35:14.719 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:35:14.719 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=32, tag_tid=5, entry_tid=10204' = 'object_number=32, tag_tid=5, entry_tid=10088' 2024-04-06T09:35:14.719 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=32, tag_tid=5, entry_tid=10204' 2024-04-06T09:35:14.719 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=45, tag_tid=6, entry_tid=1' 2024-04-06T09:35:14.719 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:35:14.719 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:35:14.719 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=6 2024-04-06T09:35:14.720 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=32, tag_tid=5, entry_tid=10204' 2024-04-06T09:35:14.720 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:35:14.720 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:35:14.721 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=5 2024-04-06T09:35:14.722 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=45, tag_tid=6, entry_tid=1' 2024-04-06T09:35:14.722 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:35:14.722 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:35:14.723 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:35:14.723 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=32, tag_tid=5, entry_tid=10204' 2024-04-06T09:35:14.724 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:35:14.724 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:35:14.725 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=10204 2024-04-06T09:35:14.725 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 6 = 5 -a 1 -le 10204 2024-04-06T09:35:14.725 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:35:14.725 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:35:14.725 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:35:14.927 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:35:14.927 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:35:14.927 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:35:14.928 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:35:14.928 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:35:14.928 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:35:14.928 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:35:14.928 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:35:14.928 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:35:14.928 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:35:14.928 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:35:14.928 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:35:14.928 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:35:14.928 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:35:14.928 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:35:14.929 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:35:14.929 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:35:14.932 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:35:14.932 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:35:14.932 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:35:14.933 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:35:14.933 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:35:14.933 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:35:14.933 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:35:14.933 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:35:14.933 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:35:14.933 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:35:14.933 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:35:14.933 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:35:14.933 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:35:14.933 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:35:14.933 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:35:14.933 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:35:14.933 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:35:14.933 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:35:14.934 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:35:14.934 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:35:14.934 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:35:14.934 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:35:14.934 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:35:14.934 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:35:14.934 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:35:14.934 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:35:14.934 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:35:14.934 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:35:14.934 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:35:14.934 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:35:14.934 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:35:14.934 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:35:14.958 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:35:14.958 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:35:14.958 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:35:14.958 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:35:15.085 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:35:15.086 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:35:15.086 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:35:15.086 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:35:15.086 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:35:15.086 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:35:15.086 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:35:15.086 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:35:15.086 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:35:15.086 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:35:15.087 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:35:15.087 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:35:15.087 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:35:15.088 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:35:15.089 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:35:15.089 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:35:15.144 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 8 2024-04-06T09:35:15.144 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 11 2024-04-06T09:35:15.144 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:35:15.144 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=45, tag_tid=6, entry_tid=1], [object_number=44, tag_tid=6, entry_tid=0], [object_number=47, tag_tid=5, entry_tid=22415], [object_number=46, tag_tid=5, entry_tid=22414]]], state=connected] 2024-04-06T09:35:15.144 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=39, tag_tid=5, entry_tid=10247], [object_number=38, tag_tid=5, entry_tid=10246], [object_number=33, tag_tid=5, entry_tid=10245], [object_number=32, tag_tid=5, entry_tid=10244]]], state=connected] 2024-04-06T09:35:15.148 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:35:15.149 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=45, tag_tid=6, entry_tid=1' 2024-04-06T09:35:15.150 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:35:15.150 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:35:15.151 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:35:15.151 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:35:15.151 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:35:15.151 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:35:15.151 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:35:15.151 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:35:15.151 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:35:15.151 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:35:15.151 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:35:15.151 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:35:15.152 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:35:15.152 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:35:15.152 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:35:15.197 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 8 2024-04-06T09:35:15.197 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 11 2024-04-06T09:35:15.197 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:35:15.197 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=45, tag_tid=6, entry_tid=1], [object_number=44, tag_tid=6, entry_tid=0], [object_number=47, tag_tid=5, entry_tid=22415], [object_number=46, tag_tid=5, entry_tid=22414]]], state=connected] 2024-04-06T09:35:15.197 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=39, tag_tid=5, entry_tid=10247], [object_number=38, tag_tid=5, entry_tid=10246], [object_number=33, tag_tid=5, entry_tid=10245], [object_number=32, tag_tid=5, entry_tid=10244]]], state=connected] 2024-04-06T09:35:15.201 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:35:15.202 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=39, tag_tid=5, entry_tid=10247' 2024-04-06T09:35:15.202 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=45, tag_tid=6, entry_tid=1' -a 'object_number=45, tag_tid=6, entry_tid=1' = 'object_number=39, tag_tid=5, entry_tid=10247' 2024-04-06T09:35:15.202 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=39, tag_tid=5, entry_tid=10247' '!=' 'object_number=32, tag_tid=5, entry_tid=10204' 2024-04-06T09:35:15.202 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:35:15.202 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=39, tag_tid=5, entry_tid=10247' = 'object_number=32, tag_tid=5, entry_tid=10204' 2024-04-06T09:35:15.202 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=39, tag_tid=5, entry_tid=10247' 2024-04-06T09:35:15.203 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=45, tag_tid=6, entry_tid=1' 2024-04-06T09:35:15.203 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:35:15.203 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:35:15.204 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=6 2024-04-06T09:35:15.204 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=39, tag_tid=5, entry_tid=10247' 2024-04-06T09:35:15.204 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:35:15.204 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:35:15.205 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=5 2024-04-06T09:35:15.206 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=45, tag_tid=6, entry_tid=1' 2024-04-06T09:35:15.206 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:35:15.206 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:35:15.207 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:35:15.208 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=39, tag_tid=5, entry_tid=10247' 2024-04-06T09:35:15.208 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:35:15.208 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:35:15.209 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=10247 2024-04-06T09:35:15.209 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 6 = 5 -a 1 -le 10247 2024-04-06T09:35:15.209 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:35:15.209 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:35:15.209 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:35:15.411 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:35:15.411 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:35:15.412 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:35:15.412 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:35:15.412 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:35:15.412 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:35:15.412 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:35:15.412 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:35:15.412 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:35:15.412 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:35:15.412 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:35:15.412 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:35:15.412 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:35:15.412 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:35:15.412 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:35:15.413 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:35:15.413 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:35:15.416 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:35:15.416 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:35:15.416 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:35:15.416 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:35:15.417 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:35:15.417 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:35:15.417 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:35:15.417 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:35:15.417 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:35:15.417 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:35:15.417 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:35:15.417 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:35:15.417 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:35:15.417 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:35:15.417 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:35:15.417 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:35:15.417 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:35:15.417 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:35:15.417 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:35:15.417 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:35:15.418 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:35:15.418 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:35:15.418 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:35:15.418 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:35:15.418 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:35:15.418 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:35:15.418 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:35:15.418 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:35:15.418 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:35:15.418 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:35:15.418 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:35:15.418 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:35:15.441 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:35:15.441 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:35:15.441 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:35:15.441 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:35:15.574 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:35:15.575 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:35:15.575 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:35:15.575 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:35:15.576 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:35:15.576 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:35:15.576 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:35:15.576 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:35:15.576 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:35:15.576 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:35:15.576 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:35:15.576 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:35:15.577 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:35:15.579 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:35:15.579 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:35:15.580 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:35:15.637 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 9 2024-04-06T09:35:15.637 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 11 2024-04-06T09:35:15.637 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:35:15.637 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=45, tag_tid=6, entry_tid=1], [object_number=44, tag_tid=6, entry_tid=0], [object_number=47, tag_tid=5, entry_tid=22415], [object_number=46, tag_tid=5, entry_tid=22414]]], state=connected] 2024-04-06T09:35:15.637 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=37, tag_tid=5, entry_tid=10321], [object_number=36, tag_tid=5, entry_tid=10320], [object_number=39, tag_tid=5, entry_tid=10319], [object_number=38, tag_tid=5, entry_tid=10318]]], state=connected] 2024-04-06T09:35:15.641 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:35:15.642 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=45, tag_tid=6, entry_tid=1' 2024-04-06T09:35:15.643 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:35:15.643 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:35:15.643 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:35:15.643 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:35:15.643 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:35:15.643 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:35:15.643 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:35:15.643 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:35:15.643 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:35:15.643 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:35:15.643 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:35:15.644 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:35:15.644 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:35:15.644 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:35:15.644 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:35:15.689 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 9 2024-04-06T09:35:15.689 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 11 2024-04-06T09:35:15.689 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:35:15.690 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=45, tag_tid=6, entry_tid=1], [object_number=44, tag_tid=6, entry_tid=0], [object_number=47, tag_tid=5, entry_tid=22415], [object_number=46, tag_tid=5, entry_tid=22414]]], state=connected] 2024-04-06T09:35:15.690 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=37, tag_tid=5, entry_tid=10321], [object_number=36, tag_tid=5, entry_tid=10320], [object_number=39, tag_tid=5, entry_tid=10319], [object_number=38, tag_tid=5, entry_tid=10318]]], state=connected] 2024-04-06T09:35:15.693 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:35:15.695 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=37, tag_tid=5, entry_tid=10321' 2024-04-06T09:35:15.695 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=45, tag_tid=6, entry_tid=1' -a 'object_number=45, tag_tid=6, entry_tid=1' = 'object_number=37, tag_tid=5, entry_tid=10321' 2024-04-06T09:35:15.695 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=37, tag_tid=5, entry_tid=10321' '!=' 'object_number=39, tag_tid=5, entry_tid=10247' 2024-04-06T09:35:15.695 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:35:15.695 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=37, tag_tid=5, entry_tid=10321' = 'object_number=39, tag_tid=5, entry_tid=10247' 2024-04-06T09:35:15.695 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=37, tag_tid=5, entry_tid=10321' 2024-04-06T09:35:15.695 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=45, tag_tid=6, entry_tid=1' 2024-04-06T09:35:15.695 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:35:15.695 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:35:15.696 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=6 2024-04-06T09:35:15.697 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=37, tag_tid=5, entry_tid=10321' 2024-04-06T09:35:15.697 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:35:15.697 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:35:15.698 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=5 2024-04-06T09:35:15.698 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=45, tag_tid=6, entry_tid=1' 2024-04-06T09:35:15.699 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:35:15.699 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:35:15.699 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:35:15.700 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=37, tag_tid=5, entry_tid=10321' 2024-04-06T09:35:15.700 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:35:15.701 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:35:15.701 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=10321 2024-04-06T09:35:15.702 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 6 = 5 -a 1 -le 10321 2024-04-06T09:35:15.702 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:35:15.702 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:35:15.702 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:35:15.903 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:35:15.903 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:35:15.903 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:35:15.904 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:35:15.904 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:35:15.904 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:35:15.904 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:35:15.904 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:35:15.904 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:35:15.904 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:35:15.904 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:35:15.904 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:35:15.904 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:35:15.904 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:35:15.904 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:35:15.905 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:35:15.905 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:35:15.908 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:35:15.908 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:35:15.908 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:35:15.908 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:35:15.908 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:35:15.908 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:35:15.908 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:35:15.908 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:35:15.908 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:35:15.908 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:35:15.909 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:35:15.909 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:35:15.909 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:35:15.909 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:35:15.909 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:35:15.909 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:35:15.909 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:35:15.909 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:35:15.909 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:35:15.909 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:35:15.909 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:35:15.909 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:35:15.909 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:35:15.909 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:35:15.909 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:35:15.910 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:35:15.910 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:35:15.910 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:35:15.910 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:35:15.910 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:35:15.910 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:35:15.910 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:35:15.934 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:35:15.934 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:35:15.935 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:35:15.935 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:35:16.066 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:35:16.068 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:35:16.068 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:35:16.068 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:35:16.068 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:35:16.068 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:35:16.068 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:35:16.068 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:35:16.068 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:35:16.068 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:35:16.069 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:35:16.069 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:35:16.069 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:35:16.071 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:35:16.071 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:35:16.071 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:35:16.120 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 9 2024-04-06T09:35:16.120 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 11 2024-04-06T09:35:16.121 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:35:16.121 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=45, tag_tid=6, entry_tid=1], [object_number=44, tag_tid=6, entry_tid=0], [object_number=47, tag_tid=5, entry_tid=22415], [object_number=46, tag_tid=5, entry_tid=22414]]], state=connected] 2024-04-06T09:35:16.121 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=38, tag_tid=5, entry_tid=10414], [object_number=37, tag_tid=5, entry_tid=10413], [object_number=36, tag_tid=5, entry_tid=10412], [object_number=39, tag_tid=5, entry_tid=10411]]], state=connected] 2024-04-06T09:35:16.124 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:35:16.125 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=45, tag_tid=6, entry_tid=1' 2024-04-06T09:35:16.126 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:35:16.127 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:35:16.127 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:35:16.127 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:35:16.127 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:35:16.127 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:35:16.127 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:35:16.127 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:35:16.127 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:35:16.127 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:35:16.127 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:35:16.127 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:35:16.128 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:35:16.128 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:35:16.128 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:35:16.173 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 9 2024-04-06T09:35:16.173 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 11 2024-04-06T09:35:16.173 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:35:16.173 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=45, tag_tid=6, entry_tid=1], [object_number=44, tag_tid=6, entry_tid=0], [object_number=47, tag_tid=5, entry_tid=22415], [object_number=46, tag_tid=5, entry_tid=22414]]], state=connected] 2024-04-06T09:35:16.173 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=38, tag_tid=5, entry_tid=10414], [object_number=37, tag_tid=5, entry_tid=10413], [object_number=36, tag_tid=5, entry_tid=10412], [object_number=39, tag_tid=5, entry_tid=10411]]], state=connected] 2024-04-06T09:35:16.177 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:35:16.179 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=38, tag_tid=5, entry_tid=10414' 2024-04-06T09:35:16.179 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=45, tag_tid=6, entry_tid=1' -a 'object_number=45, tag_tid=6, entry_tid=1' = 'object_number=38, tag_tid=5, entry_tid=10414' 2024-04-06T09:35:16.179 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=38, tag_tid=5, entry_tid=10414' '!=' 'object_number=37, tag_tid=5, entry_tid=10321' 2024-04-06T09:35:16.179 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:35:16.179 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=38, tag_tid=5, entry_tid=10414' = 'object_number=37, tag_tid=5, entry_tid=10321' 2024-04-06T09:35:16.179 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=38, tag_tid=5, entry_tid=10414' 2024-04-06T09:35:16.179 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=45, tag_tid=6, entry_tid=1' 2024-04-06T09:35:16.179 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:35:16.179 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:35:16.180 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=6 2024-04-06T09:35:16.180 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=38, tag_tid=5, entry_tid=10414' 2024-04-06T09:35:16.181 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:35:16.181 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:35:16.181 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=5 2024-04-06T09:35:16.182 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=45, tag_tid=6, entry_tid=1' 2024-04-06T09:35:16.182 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:35:16.182 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:35:16.183 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:35:16.184 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=38, tag_tid=5, entry_tid=10414' 2024-04-06T09:35:16.184 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:35:16.184 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:35:16.185 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=10414 2024-04-06T09:35:16.185 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 6 = 5 -a 1 -le 10414 2024-04-06T09:35:16.185 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:35:16.185 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:35:16.185 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:35:16.387 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:35:16.387 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:35:16.387 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:35:16.387 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:35:16.387 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:35:16.387 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:35:16.387 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:35:16.387 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:35:16.387 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:35:16.388 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:35:16.388 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:35:16.388 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:35:16.388 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:35:16.388 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:35:16.388 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:35:16.388 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:35:16.389 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:35:16.392 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:35:16.392 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:35:16.392 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:35:16.393 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:35:16.393 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:35:16.393 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:35:16.393 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:35:16.393 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:35:16.393 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:35:16.393 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:35:16.393 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:35:16.393 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:35:16.393 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:35:16.393 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:35:16.393 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:35:16.393 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:35:16.393 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:35:16.393 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:35:16.394 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:35:16.394 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:35:16.394 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:35:16.394 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:35:16.394 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:35:16.394 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:35:16.394 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:35:16.394 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:35:16.394 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:35:16.394 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:35:16.394 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:35:16.394 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:35:16.394 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:35:16.394 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:35:16.420 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:35:16.420 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:35:16.420 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:35:16.420 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:35:16.569 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:35:16.570 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:35:16.570 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:35:16.570 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:35:16.570 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:35:16.571 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:35:16.571 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:35:16.571 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:35:16.571 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:35:16.571 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:35:16.571 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:35:16.571 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:35:16.571 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:35:16.574 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:35:16.574 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:35:16.574 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:35:16.627 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 9 2024-04-06T09:35:16.627 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 11 2024-04-06T09:35:16.627 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:35:16.627 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=45, tag_tid=6, entry_tid=1], [object_number=44, tag_tid=6, entry_tid=0], [object_number=47, tag_tid=5, entry_tid=22415], [object_number=46, tag_tid=5, entry_tid=22414]]], state=connected] 2024-04-06T09:35:16.627 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=38, tag_tid=5, entry_tid=10502], [object_number=37, tag_tid=5, entry_tid=10501], [object_number=36, tag_tid=5, entry_tid=10500], [object_number=39, tag_tid=5, entry_tid=10499]]], state=connected] 2024-04-06T09:35:16.630 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:35:16.632 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=45, tag_tid=6, entry_tid=1' 2024-04-06T09:35:16.633 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:35:16.633 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:35:16.633 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:35:16.633 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:35:16.633 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:35:16.633 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:35:16.633 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:35:16.633 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:35:16.633 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:35:16.633 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:35:16.634 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:35:16.634 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:35:16.634 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:35:16.634 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:35:16.635 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:35:16.681 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 9 2024-04-06T09:35:16.682 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 11 2024-04-06T09:35:16.682 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:35:16.682 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=45, tag_tid=6, entry_tid=1], [object_number=44, tag_tid=6, entry_tid=0], [object_number=47, tag_tid=5, entry_tid=22415], [object_number=46, tag_tid=5, entry_tid=22414]]], state=connected] 2024-04-06T09:35:16.682 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=39, tag_tid=5, entry_tid=10527], [object_number=38, tag_tid=5, entry_tid=10526], [object_number=37, tag_tid=5, entry_tid=10525], [object_number=36, tag_tid=5, entry_tid=10524]]], state=connected] 2024-04-06T09:35:16.685 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:35:16.686 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=39, tag_tid=5, entry_tid=10527' 2024-04-06T09:35:16.687 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=45, tag_tid=6, entry_tid=1' -a 'object_number=45, tag_tid=6, entry_tid=1' = 'object_number=39, tag_tid=5, entry_tid=10527' 2024-04-06T09:35:16.687 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=39, tag_tid=5, entry_tid=10527' '!=' 'object_number=38, tag_tid=5, entry_tid=10414' 2024-04-06T09:35:16.687 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:35:16.687 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=39, tag_tid=5, entry_tid=10527' = 'object_number=38, tag_tid=5, entry_tid=10414' 2024-04-06T09:35:16.687 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=39, tag_tid=5, entry_tid=10527' 2024-04-06T09:35:16.687 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=45, tag_tid=6, entry_tid=1' 2024-04-06T09:35:16.688 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:35:16.688 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:35:16.688 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=6 2024-04-06T09:35:16.689 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=39, tag_tid=5, entry_tid=10527' 2024-04-06T09:35:16.689 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:35:16.689 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:35:16.690 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=5 2024-04-06T09:35:16.690 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=45, tag_tid=6, entry_tid=1' 2024-04-06T09:35:16.691 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:35:16.691 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:35:16.692 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:35:16.692 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=39, tag_tid=5, entry_tid=10527' 2024-04-06T09:35:16.692 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:35:16.692 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:35:16.693 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=10527 2024-04-06T09:35:16.694 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 6 = 5 -a 1 -le 10527 2024-04-06T09:35:16.694 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:35:16.694 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:35:16.694 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:35:16.895 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:35:16.896 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:35:16.896 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:35:16.896 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:35:16.896 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:35:16.896 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:35:16.896 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:35:16.896 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:35:16.896 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:35:16.896 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:35:16.896 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:35:16.896 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:35:16.896 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:35:16.896 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:35:16.896 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:35:16.897 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:35:16.898 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:35:16.900 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:35:16.900 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:35:16.900 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:35:16.900 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:35:16.900 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:35:16.900 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:35:16.900 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:35:16.900 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:35:16.900 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:35:16.900 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:35:16.901 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:35:16.901 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:35:16.901 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:35:16.901 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:35:16.901 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:35:16.901 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:35:16.902 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:35:16.902 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:35:16.902 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:35:16.902 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:35:16.902 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:35:16.902 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:35:16.903 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:35:16.903 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:35:16.903 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:35:16.903 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:35:16.903 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:35:16.903 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:35:16.903 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:35:16.903 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:35:16.903 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:35:16.903 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:35:16.927 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:35:16.927 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:35:16.927 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:35:16.927 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:35:17.061 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:35:17.062 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:35:17.062 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:35:17.062 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:35:17.062 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:35:17.062 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:35:17.062 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:35:17.062 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:35:17.062 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:35:17.062 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:35:17.063 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:35:17.063 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:35:17.063 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:35:17.066 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:35:17.066 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:35:17.066 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:35:17.112 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 9 2024-04-06T09:35:17.112 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 11 2024-04-06T09:35:17.112 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:35:17.112 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=45, tag_tid=6, entry_tid=1], [object_number=44, tag_tid=6, entry_tid=0], [object_number=47, tag_tid=5, entry_tid=22415], [object_number=46, tag_tid=5, entry_tid=22414]]], state=connected] 2024-04-06T09:35:17.112 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=38, tag_tid=5, entry_tid=10594], [object_number=37, tag_tid=5, entry_tid=10593], [object_number=36, tag_tid=5, entry_tid=10592], [object_number=39, tag_tid=5, entry_tid=10591]]], state=connected] 2024-04-06T09:35:17.116 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:35:17.117 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=45, tag_tid=6, entry_tid=1' 2024-04-06T09:35:17.118 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:35:17.118 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:35:17.118 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:35:17.118 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:35:17.118 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:35:17.118 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:35:17.119 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:35:17.119 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:35:17.119 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:35:17.119 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:35:17.119 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:35:17.119 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:35:17.119 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:35:17.120 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:35:17.120 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:35:17.164 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 9 2024-04-06T09:35:17.164 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 11 2024-04-06T09:35:17.165 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:35:17.165 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=45, tag_tid=6, entry_tid=1], [object_number=44, tag_tid=6, entry_tid=0], [object_number=47, tag_tid=5, entry_tid=22415], [object_number=46, tag_tid=5, entry_tid=22414]]], state=connected] 2024-04-06T09:35:17.165 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=38, tag_tid=5, entry_tid=10594], [object_number=37, tag_tid=5, entry_tid=10593], [object_number=36, tag_tid=5, entry_tid=10592], [object_number=39, tag_tid=5, entry_tid=10591]]], state=connected] 2024-04-06T09:35:17.168 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:35:17.169 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=38, tag_tid=5, entry_tid=10594' 2024-04-06T09:35:17.170 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=45, tag_tid=6, entry_tid=1' -a 'object_number=45, tag_tid=6, entry_tid=1' = 'object_number=38, tag_tid=5, entry_tid=10594' 2024-04-06T09:35:17.170 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=38, tag_tid=5, entry_tid=10594' '!=' 'object_number=39, tag_tid=5, entry_tid=10527' 2024-04-06T09:35:17.170 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:35:17.170 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=38, tag_tid=5, entry_tid=10594' = 'object_number=39, tag_tid=5, entry_tid=10527' 2024-04-06T09:35:17.170 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=38, tag_tid=5, entry_tid=10594' 2024-04-06T09:35:17.170 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=45, tag_tid=6, entry_tid=1' 2024-04-06T09:35:17.171 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:35:17.171 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:35:17.171 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=6 2024-04-06T09:35:17.172 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=38, tag_tid=5, entry_tid=10594' 2024-04-06T09:35:17.172 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:35:17.172 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:35:17.173 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=5 2024-04-06T09:35:17.173 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=45, tag_tid=6, entry_tid=1' 2024-04-06T09:35:17.173 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:35:17.173 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:35:17.174 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:35:17.175 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=38, tag_tid=5, entry_tid=10594' 2024-04-06T09:35:17.175 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:35:17.175 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:35:17.176 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=10594 2024-04-06T09:35:17.176 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 6 = 5 -a 1 -le 10594 2024-04-06T09:35:17.177 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:35:17.177 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:35:17.177 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:35:17.378 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:35:17.379 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:35:17.379 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:35:17.379 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:35:17.379 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:35:17.379 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:35:17.379 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:35:17.379 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:35:17.379 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:35:17.379 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:35:17.379 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:35:17.379 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:35:17.379 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:35:17.379 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:35:17.379 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:35:17.380 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:35:17.380 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:35:17.384 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:35:17.384 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:35:17.384 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:35:17.384 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:35:17.384 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:35:17.384 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:35:17.384 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:35:17.384 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:35:17.384 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:35:17.384 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:35:17.384 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:35:17.384 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:35:17.384 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:35:17.385 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:35:17.385 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:35:17.385 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:35:17.385 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:35:17.385 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:35:17.385 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:35:17.385 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:35:17.385 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:35:17.385 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:35:17.385 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:35:17.385 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:35:17.385 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:35:17.385 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:35:17.385 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:35:17.385 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:35:17.386 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:35:17.386 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:35:17.386 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:35:17.386 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:35:17.410 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:35:17.411 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:35:17.411 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:35:17.411 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:35:17.540 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:35:17.541 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:35:17.541 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:35:17.541 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:35:17.541 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:35:17.542 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:35:17.542 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:35:17.542 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:35:17.542 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:35:17.542 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:35:17.542 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:35:17.542 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:35:17.543 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:35:17.543 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:35:17.544 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:35:17.544 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:35:17.589 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 9 2024-04-06T09:35:17.589 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 11 2024-04-06T09:35:17.589 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:35:17.589 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=45, tag_tid=6, entry_tid=1], [object_number=44, tag_tid=6, entry_tid=0], [object_number=47, tag_tid=5, entry_tid=22415], [object_number=46, tag_tid=5, entry_tid=22414]]], state=connected] 2024-04-06T09:35:17.589 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=37, tag_tid=5, entry_tid=10701], [object_number=36, tag_tid=5, entry_tid=10700], [object_number=39, tag_tid=5, entry_tid=10699], [object_number=38, tag_tid=5, entry_tid=10698]]], state=connected] 2024-04-06T09:35:17.593 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:35:17.594 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=45, tag_tid=6, entry_tid=1' 2024-04-06T09:35:17.595 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:35:17.595 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:35:17.595 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:35:17.595 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:35:17.595 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:35:17.595 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:35:17.595 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:35:17.595 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:35:17.595 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:35:17.596 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:35:17.596 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:35:17.596 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:35:17.596 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:35:17.596 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:35:17.597 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:35:17.640 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 9 2024-04-06T09:35:17.641 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 11 2024-04-06T09:35:17.641 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:35:17.641 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=45, tag_tid=6, entry_tid=1], [object_number=44, tag_tid=6, entry_tid=0], [object_number=47, tag_tid=5, entry_tid=22415], [object_number=46, tag_tid=5, entry_tid=22414]]], state=connected] 2024-04-06T09:35:17.641 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=37, tag_tid=5, entry_tid=10701], [object_number=36, tag_tid=5, entry_tid=10700], [object_number=39, tag_tid=5, entry_tid=10699], [object_number=38, tag_tid=5, entry_tid=10698]]], state=connected] 2024-04-06T09:35:17.644 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:35:17.646 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=37, tag_tid=5, entry_tid=10701' 2024-04-06T09:35:17.646 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=45, tag_tid=6, entry_tid=1' -a 'object_number=45, tag_tid=6, entry_tid=1' = 'object_number=37, tag_tid=5, entry_tid=10701' 2024-04-06T09:35:17.646 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=37, tag_tid=5, entry_tid=10701' '!=' 'object_number=38, tag_tid=5, entry_tid=10594' 2024-04-06T09:35:17.646 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:35:17.646 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=37, tag_tid=5, entry_tid=10701' = 'object_number=38, tag_tid=5, entry_tid=10594' 2024-04-06T09:35:17.646 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=37, tag_tid=5, entry_tid=10701' 2024-04-06T09:35:17.646 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=45, tag_tid=6, entry_tid=1' 2024-04-06T09:35:17.647 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:35:17.647 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:35:17.647 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=6 2024-04-06T09:35:17.648 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=37, tag_tid=5, entry_tid=10701' 2024-04-06T09:35:17.648 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:35:17.648 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:35:17.649 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=5 2024-04-06T09:35:17.649 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=45, tag_tid=6, entry_tid=1' 2024-04-06T09:35:17.650 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:35:17.650 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:35:17.651 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:35:17.651 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=37, tag_tid=5, entry_tid=10701' 2024-04-06T09:35:17.651 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:35:17.651 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:35:17.652 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=10701 2024-04-06T09:35:17.653 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 6 = 5 -a 1 -le 10701 2024-04-06T09:35:17.653 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:35:17.653 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:35:17.653 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:35:17.855 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:35:17.855 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:35:17.855 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:35:17.855 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:35:17.855 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:35:17.855 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:35:17.855 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:35:17.855 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:35:17.855 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:35:17.855 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:35:17.855 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:35:17.856 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:35:17.856 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:35:17.856 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:35:17.856 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:35:17.856 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:35:17.856 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:35:17.859 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:35:17.859 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:35:17.859 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:35:17.859 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:35:17.859 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:35:17.860 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:35:17.860 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:35:17.860 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:35:17.860 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:35:17.860 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:35:17.860 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:35:17.860 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:35:17.860 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:35:17.860 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:35:17.860 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:35:17.860 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:35:17.860 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:35:17.860 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:35:17.860 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:35:17.860 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:35:17.860 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:35:17.860 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:35:17.861 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:35:17.861 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:35:17.861 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:35:17.861 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:35:17.861 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:35:17.861 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:35:17.861 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:35:17.861 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:35:17.861 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:35:17.861 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:35:17.884 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:35:17.885 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:35:17.885 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:35:17.885 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:35:18.017 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:35:18.018 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:35:18.018 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:35:18.018 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:35:18.019 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:35:18.019 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:35:18.019 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:35:18.019 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:35:18.019 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:35:18.019 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:35:18.019 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:35:18.019 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:35:18.019 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:35:18.021 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:35:18.022 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:35:18.022 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:35:18.074 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 9 2024-04-06T09:35:18.075 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 11 2024-04-06T09:35:18.075 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:35:18.075 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=45, tag_tid=6, entry_tid=1], [object_number=44, tag_tid=6, entry_tid=0], [object_number=47, tag_tid=5, entry_tid=22415], [object_number=46, tag_tid=5, entry_tid=22414]]], state=connected] 2024-04-06T09:35:18.075 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=39, tag_tid=5, entry_tid=10803], [object_number=38, tag_tid=5, entry_tid=10802], [object_number=37, tag_tid=5, entry_tid=10801], [object_number=36, tag_tid=5, entry_tid=10800]]], state=connected] 2024-04-06T09:35:18.078 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:35:18.080 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=45, tag_tid=6, entry_tid=1' 2024-04-06T09:35:18.081 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:35:18.081 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:35:18.081 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:35:18.081 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:35:18.081 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:35:18.081 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:35:18.081 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:35:18.081 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:35:18.081 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:35:18.081 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:35:18.081 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:35:18.082 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:35:18.082 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:35:18.082 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:35:18.082 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:35:18.127 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 9 2024-04-06T09:35:18.128 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 11 2024-04-06T09:35:18.128 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:35:18.128 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=45, tag_tid=6, entry_tid=1], [object_number=44, tag_tid=6, entry_tid=0], [object_number=47, tag_tid=5, entry_tid=22415], [object_number=46, tag_tid=5, entry_tid=22414]]], state=connected] 2024-04-06T09:35:18.128 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=39, tag_tid=5, entry_tid=10803], [object_number=38, tag_tid=5, entry_tid=10802], [object_number=37, tag_tid=5, entry_tid=10801], [object_number=36, tag_tid=5, entry_tid=10800]]], state=connected] 2024-04-06T09:35:18.131 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:35:18.133 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=39, tag_tid=5, entry_tid=10803' 2024-04-06T09:35:18.133 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=45, tag_tid=6, entry_tid=1' -a 'object_number=45, tag_tid=6, entry_tid=1' = 'object_number=39, tag_tid=5, entry_tid=10803' 2024-04-06T09:35:18.133 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=39, tag_tid=5, entry_tid=10803' '!=' 'object_number=37, tag_tid=5, entry_tid=10701' 2024-04-06T09:35:18.133 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:35:18.134 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=39, tag_tid=5, entry_tid=10803' = 'object_number=37, tag_tid=5, entry_tid=10701' 2024-04-06T09:35:18.134 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=39, tag_tid=5, entry_tid=10803' 2024-04-06T09:35:18.134 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=45, tag_tid=6, entry_tid=1' 2024-04-06T09:35:18.134 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:35:18.134 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:35:18.134 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=6 2024-04-06T09:35:18.135 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=39, tag_tid=5, entry_tid=10803' 2024-04-06T09:35:18.135 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:35:18.135 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:35:18.136 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=5 2024-04-06T09:35:18.137 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=45, tag_tid=6, entry_tid=1' 2024-04-06T09:35:18.137 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:35:18.137 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:35:18.138 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:35:18.138 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=39, tag_tid=5, entry_tid=10803' 2024-04-06T09:35:18.139 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:35:18.139 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:35:18.140 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=10803 2024-04-06T09:35:18.140 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 6 = 5 -a 1 -le 10803 2024-04-06T09:35:18.140 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:35:18.140 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:35:18.140 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:35:18.342 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:35:18.342 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:35:18.342 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:35:18.342 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:35:18.342 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:35:18.342 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:35:18.342 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:35:18.343 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:35:18.343 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:35:18.343 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:35:18.343 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:35:18.343 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:35:18.343 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:35:18.343 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:35:18.343 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:35:18.343 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:35:18.344 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:35:18.346 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:35:18.346 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:35:18.347 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:35:18.347 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:35:18.347 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:35:18.347 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:35:18.347 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:35:18.347 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:35:18.347 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:35:18.347 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:35:18.347 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:35:18.347 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:35:18.347 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:35:18.347 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:35:18.347 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:35:18.347 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:35:18.347 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:35:18.348 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:35:18.348 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:35:18.348 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:35:18.348 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:35:18.348 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:35:18.348 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:35:18.348 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:35:18.348 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:35:18.348 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:35:18.348 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:35:18.348 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:35:18.348 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:35:18.348 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:35:18.348 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:35:18.348 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:35:18.372 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:35:18.372 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:35:18.372 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:35:18.372 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:35:18.503 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:35:18.504 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:35:18.505 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:35:18.505 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:35:18.505 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:35:18.505 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:35:18.505 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:35:18.505 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:35:18.505 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:35:18.505 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:35:18.506 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:35:18.506 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:35:18.506 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:35:18.507 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:35:18.507 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:35:18.508 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:35:18.757 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 9 2024-04-06T09:35:18.757 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 11 2024-04-06T09:35:18.757 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:35:18.757 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=45, tag_tid=6, entry_tid=1], [object_number=44, tag_tid=6, entry_tid=0], [object_number=47, tag_tid=5, entry_tid=22415], [object_number=46, tag_tid=5, entry_tid=22414]]], state=connected] 2024-04-06T09:35:18.757 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=36, tag_tid=5, entry_tid=10908], [object_number=39, tag_tid=5, entry_tid=10907], [object_number=38, tag_tid=5, entry_tid=10906], [object_number=37, tag_tid=5, entry_tid=10905]]], state=connected] 2024-04-06T09:35:18.763 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:35:18.765 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=45, tag_tid=6, entry_tid=1' 2024-04-06T09:35:18.767 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:35:18.767 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:35:18.767 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:35:18.767 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:35:18.767 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:35:18.767 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:35:18.767 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:35:18.767 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:35:18.767 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:35:18.767 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:35:18.768 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:35:18.768 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:35:18.769 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:35:18.769 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:35:18.770 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:35:18.823 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 9 2024-04-06T09:35:18.869 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 11 2024-04-06T09:35:18.870 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:35:18.870 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=45, tag_tid=6, entry_tid=1], [object_number=44, tag_tid=6, entry_tid=0], [object_number=47, tag_tid=5, entry_tid=22415], [object_number=46, tag_tid=5, entry_tid=22414]]], state=connected] 2024-04-06T09:35:18.870 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=36, tag_tid=5, entry_tid=10908], [object_number=39, tag_tid=5, entry_tid=10907], [object_number=38, tag_tid=5, entry_tid=10906], [object_number=37, tag_tid=5, entry_tid=10905]]], state=connected] 2024-04-06T09:35:18.872 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:35:18.873 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=36, tag_tid=5, entry_tid=10908' 2024-04-06T09:35:18.873 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=45, tag_tid=6, entry_tid=1' -a 'object_number=45, tag_tid=6, entry_tid=1' = 'object_number=36, tag_tid=5, entry_tid=10908' 2024-04-06T09:35:18.873 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=36, tag_tid=5, entry_tid=10908' '!=' 'object_number=39, tag_tid=5, entry_tid=10803' 2024-04-06T09:35:18.873 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:35:18.873 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=36, tag_tid=5, entry_tid=10908' = 'object_number=39, tag_tid=5, entry_tid=10803' 2024-04-06T09:35:18.873 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=36, tag_tid=5, entry_tid=10908' 2024-04-06T09:35:18.873 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=45, tag_tid=6, entry_tid=1' 2024-04-06T09:35:18.873 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:35:18.873 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:35:18.873 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=6 2024-04-06T09:35:18.873 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=36, tag_tid=5, entry_tid=10908' 2024-04-06T09:35:18.873 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:35:18.873 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:35:18.874 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=5 2024-04-06T09:35:18.874 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=45, tag_tid=6, entry_tid=1' 2024-04-06T09:35:18.874 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:35:18.874 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:35:18.874 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:35:18.874 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=36, tag_tid=5, entry_tid=10908' 2024-04-06T09:35:18.874 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:35:18.874 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:35:18.874 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=10908 2024-04-06T09:35:18.874 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 6 = 5 -a 1 -le 10908 2024-04-06T09:35:18.874 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:35:18.874 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:35:18.874 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:35:19.039 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:35:19.039 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:35:19.039 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:35:19.039 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:35:19.039 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:35:19.039 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:35:19.040 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:35:19.040 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:35:19.040 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:35:19.040 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:35:19.040 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:35:19.040 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:35:19.040 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:35:19.040 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:35:19.040 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:35:19.041 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:35:19.041 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:35:19.043 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:35:19.043 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:35:19.043 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:35:19.043 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:35:19.043 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:35:19.043 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:35:19.044 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:35:19.044 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:35:19.044 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:35:19.044 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:35:19.044 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:35:19.044 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:35:19.044 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:35:19.044 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:35:19.044 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:35:19.044 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:35:19.044 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:35:19.044 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:35:19.044 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:35:19.044 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:35:19.044 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:35:19.045 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:35:19.045 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:35:19.045 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:35:19.045 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:35:19.045 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:35:19.045 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:35:19.045 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:35:19.045 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:35:19.045 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:35:19.045 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:35:19.045 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:35:19.069 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:35:19.069 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:35:19.069 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:35:19.069 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:35:19.202 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:35:19.203 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:35:19.203 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:35:19.203 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:35:19.204 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:35:19.204 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:35:19.204 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:35:19.204 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:35:19.204 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:35:19.204 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:35:19.204 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:35:19.205 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:35:19.205 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:35:19.207 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:35:19.207 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:35:19.207 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:35:19.252 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 9 2024-04-06T09:35:19.252 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 11 2024-04-06T09:35:19.252 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:35:19.252 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=45, tag_tid=6, entry_tid=1], [object_number=44, tag_tid=6, entry_tid=0], [object_number=47, tag_tid=5, entry_tid=22415], [object_number=46, tag_tid=5, entry_tid=22414]]], state=connected] 2024-04-06T09:35:19.252 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=36, tag_tid=5, entry_tid=11052], [object_number=39, tag_tid=5, entry_tid=11051], [object_number=38, tag_tid=5, entry_tid=11050], [object_number=37, tag_tid=5, entry_tid=11049]]], state=connected] 2024-04-06T09:35:19.255 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:35:19.256 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=45, tag_tid=6, entry_tid=1' 2024-04-06T09:35:19.258 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:35:19.258 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:35:19.258 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:35:19.258 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:35:19.258 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:35:19.258 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:35:19.258 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:35:19.258 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:35:19.258 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:35:19.258 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:35:19.258 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:35:19.258 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:35:19.259 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:35:19.259 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:35:19.259 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:35:19.304 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 9 2024-04-06T09:35:19.304 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 11 2024-04-06T09:35:19.304 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:35:19.305 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=45, tag_tid=6, entry_tid=1], [object_number=44, tag_tid=6, entry_tid=0], [object_number=47, tag_tid=5, entry_tid=22415], [object_number=46, tag_tid=5, entry_tid=22414]]], state=connected] 2024-04-06T09:35:19.305 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=36, tag_tid=5, entry_tid=11052], [object_number=39, tag_tid=5, entry_tid=11051], [object_number=38, tag_tid=5, entry_tid=11050], [object_number=37, tag_tid=5, entry_tid=11049]]], state=connected] 2024-04-06T09:35:19.308 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:35:19.310 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=36, tag_tid=5, entry_tid=11052' 2024-04-06T09:35:19.310 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=45, tag_tid=6, entry_tid=1' -a 'object_number=45, tag_tid=6, entry_tid=1' = 'object_number=36, tag_tid=5, entry_tid=11052' 2024-04-06T09:35:19.310 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=36, tag_tid=5, entry_tid=11052' '!=' 'object_number=36, tag_tid=5, entry_tid=10908' 2024-04-06T09:35:19.311 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:35:19.311 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=36, tag_tid=5, entry_tid=11052' = 'object_number=36, tag_tid=5, entry_tid=10908' 2024-04-06T09:35:19.311 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=36, tag_tid=5, entry_tid=11052' 2024-04-06T09:35:19.311 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=45, tag_tid=6, entry_tid=1' 2024-04-06T09:35:19.311 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:35:19.311 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:35:19.311 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=6 2024-04-06T09:35:19.312 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=36, tag_tid=5, entry_tid=11052' 2024-04-06T09:35:19.312 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:35:19.312 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:35:19.313 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=5 2024-04-06T09:35:19.314 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=45, tag_tid=6, entry_tid=1' 2024-04-06T09:35:19.314 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:35:19.314 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:35:19.315 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:35:19.315 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=36, tag_tid=5, entry_tid=11052' 2024-04-06T09:35:19.315 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:35:19.316 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:35:19.317 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=11052 2024-04-06T09:35:19.317 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 6 = 5 -a 1 -le 11052 2024-04-06T09:35:19.317 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:35:19.317 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:35:19.317 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:35:19.519 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:35:19.519 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:35:19.519 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:35:19.519 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:35:19.519 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:35:19.519 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:35:19.519 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:35:19.519 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:35:19.519 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:35:19.520 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:35:19.520 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:35:19.520 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:35:19.520 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:35:19.520 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:35:19.520 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:35:19.520 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:35:19.521 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:35:19.522 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:35:19.522 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:35:19.522 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:35:19.522 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:35:19.522 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:35:19.522 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:35:19.522 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:35:19.522 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:35:19.522 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:35:19.523 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:35:19.523 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:35:19.523 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:35:19.523 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:35:19.523 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:35:19.523 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:35:19.523 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:35:19.524 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:35:19.524 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:35:19.524 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:35:19.524 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:35:19.524 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:35:19.525 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:35:19.525 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:35:19.525 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:35:19.525 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:35:19.525 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:35:19.525 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:35:19.525 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:35:19.525 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:35:19.525 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:35:19.525 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:35:19.525 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:35:19.548 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:35:19.548 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:35:19.548 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:35:19.548 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:35:19.675 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:35:19.677 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:35:19.677 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:35:19.677 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:35:19.677 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:35:19.677 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:35:19.677 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:35:19.677 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:35:19.677 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:35:19.677 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:35:19.677 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:35:19.678 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:35:19.678 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:35:19.680 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:35:19.680 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:35:19.680 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:35:19.726 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 9 2024-04-06T09:35:19.726 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 11 2024-04-06T09:35:19.726 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:35:19.726 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=45, tag_tid=6, entry_tid=1], [object_number=44, tag_tid=6, entry_tid=0], [object_number=47, tag_tid=5, entry_tid=22415], [object_number=46, tag_tid=5, entry_tid=22414]]], state=connected] 2024-04-06T09:35:19.726 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=39, tag_tid=5, entry_tid=11143], [object_number=38, tag_tid=5, entry_tid=11142], [object_number=37, tag_tid=5, entry_tid=11141], [object_number=36, tag_tid=5, entry_tid=11140]]], state=connected] 2024-04-06T09:35:19.730 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:35:19.731 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=45, tag_tid=6, entry_tid=1' 2024-04-06T09:35:19.732 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:35:19.732 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:35:19.732 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:35:19.732 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:35:19.732 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:35:19.732 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:35:19.732 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:35:19.732 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:35:19.732 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:35:19.733 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:35:19.733 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:35:19.733 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:35:19.733 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:35:19.733 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:35:19.733 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:35:19.779 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 9 2024-04-06T09:35:19.779 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 11 2024-04-06T09:35:19.779 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:35:19.779 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=45, tag_tid=6, entry_tid=1], [object_number=44, tag_tid=6, entry_tid=0], [object_number=47, tag_tid=5, entry_tid=22415], [object_number=46, tag_tid=5, entry_tid=22414]]], state=connected] 2024-04-06T09:35:19.779 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=39, tag_tid=5, entry_tid=11143], [object_number=38, tag_tid=5, entry_tid=11142], [object_number=37, tag_tid=5, entry_tid=11141], [object_number=36, tag_tid=5, entry_tid=11140]]], state=connected] 2024-04-06T09:35:19.782 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:35:19.784 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=39, tag_tid=5, entry_tid=11143' 2024-04-06T09:35:19.784 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=45, tag_tid=6, entry_tid=1' -a 'object_number=45, tag_tid=6, entry_tid=1' = 'object_number=39, tag_tid=5, entry_tid=11143' 2024-04-06T09:35:19.784 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=39, tag_tid=5, entry_tid=11143' '!=' 'object_number=36, tag_tid=5, entry_tid=11052' 2024-04-06T09:35:19.784 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:35:19.784 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=39, tag_tid=5, entry_tid=11143' = 'object_number=36, tag_tid=5, entry_tid=11052' 2024-04-06T09:35:19.784 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=39, tag_tid=5, entry_tid=11143' 2024-04-06T09:35:19.785 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=45, tag_tid=6, entry_tid=1' 2024-04-06T09:35:19.785 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:35:19.785 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:35:19.785 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=6 2024-04-06T09:35:19.786 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=39, tag_tid=5, entry_tid=11143' 2024-04-06T09:35:19.786 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:35:19.786 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:35:19.787 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=5 2024-04-06T09:35:19.787 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=45, tag_tid=6, entry_tid=1' 2024-04-06T09:35:19.788 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:35:19.788 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:35:19.788 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:35:19.789 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=39, tag_tid=5, entry_tid=11143' 2024-04-06T09:35:19.789 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:35:19.789 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:35:19.791 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=11143 2024-04-06T09:35:19.791 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 6 = 5 -a 1 -le 11143 2024-04-06T09:35:19.791 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:35:19.791 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:35:19.791 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:35:19.992 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:35:19.992 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:35:19.993 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:35:19.993 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:35:19.993 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:35:19.993 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:35:19.993 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:35:19.993 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:35:19.993 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:35:19.993 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:35:19.993 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:35:19.993 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:35:19.993 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:35:19.993 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:35:19.993 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:35:19.994 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:35:19.994 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:35:19.997 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:35:19.997 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:35:19.997 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:35:19.997 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:35:19.998 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:35:19.998 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:35:19.998 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:35:19.998 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:35:19.998 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:35:19.998 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:35:19.998 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:35:19.998 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:35:19.998 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:35:19.998 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:35:19.998 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:35:19.998 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:35:19.998 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:35:19.998 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:35:19.998 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:35:19.999 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:35:19.999 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:35:19.999 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:35:19.999 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:35:19.999 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:35:19.999 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:35:19.999 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:35:19.999 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:35:19.999 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:35:19.999 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:35:19.999 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:35:19.999 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:35:20.000 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:35:20.023 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:35:20.023 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:35:20.023 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:35:20.024 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:35:20.152 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:35:20.153 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:35:20.153 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:35:20.153 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:35:20.153 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:35:20.153 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:35:20.154 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:35:20.154 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:35:20.154 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:35:20.154 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:35:20.154 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:35:20.154 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:35:20.155 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:35:20.157 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:35:20.157 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:35:20.157 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:35:20.212 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 9 2024-04-06T09:35:20.212 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 11 2024-04-06T09:35:20.212 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:35:20.212 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=45, tag_tid=6, entry_tid=1], [object_number=44, tag_tid=6, entry_tid=0], [object_number=47, tag_tid=5, entry_tid=22415], [object_number=46, tag_tid=5, entry_tid=22414]]], state=connected] 2024-04-06T09:35:20.212 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=36, tag_tid=5, entry_tid=11228], [object_number=39, tag_tid=5, entry_tid=11227], [object_number=38, tag_tid=5, entry_tid=11226], [object_number=37, tag_tid=5, entry_tid=11225]]], state=connected] 2024-04-06T09:35:20.216 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:35:20.217 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=45, tag_tid=6, entry_tid=1' 2024-04-06T09:35:20.219 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:35:20.219 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:35:20.219 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:35:20.219 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:35:20.219 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:35:20.219 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:35:20.219 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:35:20.219 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:35:20.219 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:35:20.219 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:35:20.219 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:35:20.219 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:35:20.220 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:35:20.221 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:35:20.221 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:35:20.266 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 9 2024-04-06T09:35:20.266 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 11 2024-04-06T09:35:20.266 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:35:20.266 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=45, tag_tid=6, entry_tid=1], [object_number=44, tag_tid=6, entry_tid=0], [object_number=47, tag_tid=5, entry_tid=22415], [object_number=46, tag_tid=5, entry_tid=22414]]], state=connected] 2024-04-06T09:35:20.266 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=36, tag_tid=5, entry_tid=11228], [object_number=39, tag_tid=5, entry_tid=11227], [object_number=38, tag_tid=5, entry_tid=11226], [object_number=37, tag_tid=5, entry_tid=11225]]], state=connected] 2024-04-06T09:35:20.269 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:35:20.271 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=36, tag_tid=5, entry_tid=11228' 2024-04-06T09:35:20.271 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=45, tag_tid=6, entry_tid=1' -a 'object_number=45, tag_tid=6, entry_tid=1' = 'object_number=36, tag_tid=5, entry_tid=11228' 2024-04-06T09:35:20.272 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=36, tag_tid=5, entry_tid=11228' '!=' 'object_number=39, tag_tid=5, entry_tid=11143' 2024-04-06T09:35:20.272 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:35:20.272 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=36, tag_tid=5, entry_tid=11228' = 'object_number=39, tag_tid=5, entry_tid=11143' 2024-04-06T09:35:20.272 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=36, tag_tid=5, entry_tid=11228' 2024-04-06T09:35:20.272 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=45, tag_tid=6, entry_tid=1' 2024-04-06T09:35:20.272 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:35:20.272 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:35:20.272 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=6 2024-04-06T09:35:20.273 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=36, tag_tid=5, entry_tid=11228' 2024-04-06T09:35:20.273 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:35:20.273 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:35:20.274 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=5 2024-04-06T09:35:20.275 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=45, tag_tid=6, entry_tid=1' 2024-04-06T09:35:20.275 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:35:20.275 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:35:20.276 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:35:20.276 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=36, tag_tid=5, entry_tid=11228' 2024-04-06T09:35:20.276 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:35:20.277 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:35:20.278 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=11228 2024-04-06T09:35:20.278 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 6 = 5 -a 1 -le 11228 2024-04-06T09:35:20.278 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:35:20.278 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:35:20.278 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:35:20.480 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:35:20.480 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:35:20.480 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:35:20.480 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:35:20.480 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:35:20.480 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:35:20.480 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:35:20.480 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:35:20.480 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:35:20.480 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:35:20.480 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:35:20.481 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:35:20.481 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:35:20.481 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:35:20.481 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:35:20.481 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:35:20.482 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:35:20.484 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:35:20.484 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:35:20.484 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:35:20.484 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:35:20.484 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:35:20.484 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:35:20.485 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:35:20.485 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:35:20.485 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:35:20.485 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:35:20.485 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:35:20.485 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:35:20.485 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:35:20.485 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:35:20.485 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:35:20.485 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:35:20.485 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:35:20.485 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:35:20.486 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:35:20.486 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:35:20.486 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:35:20.486 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:35:20.486 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:35:20.486 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:35:20.486 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:35:20.486 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:35:20.486 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:35:20.486 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:35:20.486 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:35:20.486 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:35:20.486 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:35:20.487 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:35:20.511 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:35:20.511 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:35:20.511 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:35:20.511 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:35:20.644 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:35:20.645 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:35:20.645 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:35:20.645 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:35:20.645 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:35:20.645 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:35:20.645 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:35:20.645 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:35:20.645 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:35:20.646 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:35:20.646 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:35:20.646 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:35:20.646 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:35:20.647 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:35:20.648 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:35:20.648 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:35:20.693 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 9 2024-04-06T09:35:20.693 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 11 2024-04-06T09:35:20.693 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:35:20.693 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=45, tag_tid=6, entry_tid=1], [object_number=44, tag_tid=6, entry_tid=0], [object_number=47, tag_tid=5, entry_tid=22415], [object_number=46, tag_tid=5, entry_tid=22414]]], state=connected] 2024-04-06T09:35:20.693 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=38, tag_tid=5, entry_tid=11326], [object_number=37, tag_tid=5, entry_tid=11325], [object_number=36, tag_tid=5, entry_tid=11324], [object_number=39, tag_tid=5, entry_tid=11323]]], state=connected] 2024-04-06T09:35:20.697 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:35:20.698 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=45, tag_tid=6, entry_tid=1' 2024-04-06T09:35:20.699 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:35:20.699 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:35:20.699 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:35:20.699 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:35:20.699 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:35:20.700 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:35:20.700 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:35:20.700 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:35:20.700 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:35:20.700 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:35:20.700 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:35:20.700 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:35:20.701 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:35:20.701 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:35:20.701 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:35:20.745 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 9 2024-04-06T09:35:20.745 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 11 2024-04-06T09:35:20.745 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:35:20.745 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=45, tag_tid=6, entry_tid=1], [object_number=44, tag_tid=6, entry_tid=0], [object_number=47, tag_tid=5, entry_tid=22415], [object_number=46, tag_tid=5, entry_tid=22414]]], state=connected] 2024-04-06T09:35:20.745 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=38, tag_tid=5, entry_tid=11326], [object_number=37, tag_tid=5, entry_tid=11325], [object_number=36, tag_tid=5, entry_tid=11324], [object_number=39, tag_tid=5, entry_tid=11323]]], state=connected] 2024-04-06T09:35:20.748 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:35:20.750 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=38, tag_tid=5, entry_tid=11326' 2024-04-06T09:35:20.751 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=45, tag_tid=6, entry_tid=1' -a 'object_number=45, tag_tid=6, entry_tid=1' = 'object_number=38, tag_tid=5, entry_tid=11326' 2024-04-06T09:35:20.751 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=38, tag_tid=5, entry_tid=11326' '!=' 'object_number=36, tag_tid=5, entry_tid=11228' 2024-04-06T09:35:20.751 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:35:20.751 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=38, tag_tid=5, entry_tid=11326' = 'object_number=36, tag_tid=5, entry_tid=11228' 2024-04-06T09:35:20.751 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=38, tag_tid=5, entry_tid=11326' 2024-04-06T09:35:20.751 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=45, tag_tid=6, entry_tid=1' 2024-04-06T09:35:20.751 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:35:20.751 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:35:20.751 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=6 2024-04-06T09:35:20.752 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=38, tag_tid=5, entry_tid=11326' 2024-04-06T09:35:20.752 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:35:20.752 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:35:20.753 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=5 2024-04-06T09:35:20.754 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=45, tag_tid=6, entry_tid=1' 2024-04-06T09:35:20.754 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:35:20.754 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:35:20.755 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:35:20.756 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=38, tag_tid=5, entry_tid=11326' 2024-04-06T09:35:20.756 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:35:20.756 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:35:20.757 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=11326 2024-04-06T09:35:20.757 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 6 = 5 -a 1 -le 11326 2024-04-06T09:35:20.757 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:35:20.757 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:35:20.757 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:35:20.959 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:35:20.959 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:35:20.959 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:35:20.959 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:35:20.959 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:35:20.960 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:35:20.960 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:35:20.960 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:35:20.960 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:35:20.960 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:35:20.960 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:35:20.960 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:35:20.960 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:35:20.960 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:35:20.960 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:35:20.961 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:35:20.961 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:35:20.964 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:35:20.964 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:35:20.964 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:35:20.964 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:35:20.964 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:35:20.964 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:35:20.964 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:35:20.964 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:35:20.964 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:35:20.964 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:35:20.965 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:35:20.965 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:35:20.965 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:35:20.965 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:35:20.965 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:35:20.965 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:35:20.965 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:35:20.965 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:35:20.965 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:35:20.965 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:35:20.965 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:35:20.965 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:35:20.965 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:35:20.965 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:35:20.965 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:35:20.965 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:35:20.966 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:35:20.966 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:35:20.966 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:35:20.966 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:35:20.966 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:35:20.966 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:35:20.989 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:35:20.990 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:35:20.990 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:35:20.990 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:35:21.120 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:35:21.121 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:35:21.121 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:35:21.121 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:35:21.121 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:35:21.121 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:35:21.121 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:35:21.121 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:35:21.122 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:35:21.122 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:35:21.122 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:35:21.122 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:35:21.122 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:35:21.123 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:35:21.124 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:35:21.124 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:35:21.173 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 9 2024-04-06T09:35:21.174 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 11 2024-04-06T09:35:21.174 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:35:21.174 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=45, tag_tid=6, entry_tid=1], [object_number=44, tag_tid=6, entry_tid=0], [object_number=47, tag_tid=5, entry_tid=22415], [object_number=46, tag_tid=5, entry_tid=22414]]], state=connected] 2024-04-06T09:35:21.174 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=36, tag_tid=5, entry_tid=11412], [object_number=39, tag_tid=5, entry_tid=11411], [object_number=38, tag_tid=5, entry_tid=11410], [object_number=37, tag_tid=5, entry_tid=11409]]], state=connected] 2024-04-06T09:35:21.178 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:35:21.179 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=45, tag_tid=6, entry_tid=1' 2024-04-06T09:35:21.180 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:35:21.180 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:35:21.180 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:35:21.180 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:35:21.180 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:35:21.180 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:35:21.180 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:35:21.180 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:35:21.180 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:35:21.180 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:35:21.181 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:35:21.181 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:35:21.181 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:35:21.182 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:35:21.182 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:35:21.228 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 9 2024-04-06T09:35:21.228 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 11 2024-04-06T09:35:21.228 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:35:21.228 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=45, tag_tid=6, entry_tid=1], [object_number=44, tag_tid=6, entry_tid=0], [object_number=47, tag_tid=5, entry_tid=22415], [object_number=46, tag_tid=5, entry_tid=22414]]], state=connected] 2024-04-06T09:35:21.228 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=36, tag_tid=5, entry_tid=11412], [object_number=39, tag_tid=5, entry_tid=11411], [object_number=38, tag_tid=5, entry_tid=11410], [object_number=37, tag_tid=5, entry_tid=11409]]], state=connected] 2024-04-06T09:35:21.231 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:35:21.233 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=36, tag_tid=5, entry_tid=11412' 2024-04-06T09:35:21.233 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=45, tag_tid=6, entry_tid=1' -a 'object_number=45, tag_tid=6, entry_tid=1' = 'object_number=36, tag_tid=5, entry_tid=11412' 2024-04-06T09:35:21.233 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=36, tag_tid=5, entry_tid=11412' '!=' 'object_number=38, tag_tid=5, entry_tid=11326' 2024-04-06T09:35:21.233 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:35:21.233 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=36, tag_tid=5, entry_tid=11412' = 'object_number=38, tag_tid=5, entry_tid=11326' 2024-04-06T09:35:21.233 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=36, tag_tid=5, entry_tid=11412' 2024-04-06T09:35:21.234 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=45, tag_tid=6, entry_tid=1' 2024-04-06T09:35:21.234 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:35:21.234 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:35:21.235 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=6 2024-04-06T09:35:21.235 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=36, tag_tid=5, entry_tid=11412' 2024-04-06T09:35:21.236 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:35:21.236 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:35:21.236 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=5 2024-04-06T09:35:21.237 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=45, tag_tid=6, entry_tid=1' 2024-04-06T09:35:21.237 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:35:21.237 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:35:21.238 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:35:21.239 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=36, tag_tid=5, entry_tid=11412' 2024-04-06T09:35:21.239 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:35:21.239 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:35:21.240 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=11412 2024-04-06T09:35:21.240 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 6 = 5 -a 1 -le 11412 2024-04-06T09:35:21.240 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:35:21.240 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:35:21.240 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:35:21.442 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:35:21.442 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:35:21.442 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:35:21.442 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:35:21.442 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:35:21.442 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:35:21.442 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:35:21.442 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:35:21.442 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:35:21.443 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:35:21.443 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:35:21.443 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:35:21.443 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:35:21.443 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:35:21.443 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:35:21.443 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:35:21.444 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:35:21.446 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:35:21.446 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:35:21.446 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:35:21.446 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:35:21.446 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:35:21.446 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:35:21.446 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:35:21.446 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:35:21.446 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:35:21.446 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:35:21.447 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:35:21.447 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:35:21.447 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:35:21.447 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:35:21.447 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:35:21.447 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:35:21.448 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:35:21.448 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:35:21.448 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:35:21.448 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:35:21.448 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:35:21.448 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:35:21.448 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:35:21.448 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:35:21.449 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:35:21.449 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:35:21.449 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:35:21.449 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:35:21.449 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:35:21.449 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:35:21.449 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:35:21.449 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:35:21.472 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:35:21.473 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:35:21.473 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:35:21.473 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:35:21.609 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:35:21.611 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:35:21.611 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:35:21.611 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:35:21.611 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:35:21.611 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:35:21.611 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:35:21.611 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:35:21.611 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:35:21.611 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:35:21.612 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:35:21.612 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:35:21.612 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:35:21.614 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:35:21.615 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:35:21.615 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:35:21.660 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 9 2024-04-06T09:35:21.661 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 11 2024-04-06T09:35:21.661 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:35:21.661 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=45, tag_tid=6, entry_tid=1], [object_number=44, tag_tid=6, entry_tid=0], [object_number=47, tag_tid=5, entry_tid=22415], [object_number=46, tag_tid=5, entry_tid=22414]]], state=connected] 2024-04-06T09:35:21.661 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=37, tag_tid=5, entry_tid=11505], [object_number=36, tag_tid=5, entry_tid=11504], [object_number=39, tag_tid=5, entry_tid=11503], [object_number=38, tag_tid=5, entry_tid=11502]]], state=connected] 2024-04-06T09:35:21.664 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:35:21.665 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=45, tag_tid=6, entry_tid=1' 2024-04-06T09:35:21.666 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:35:21.667 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:35:21.667 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:35:21.667 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:35:21.667 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:35:21.667 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:35:21.667 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:35:21.667 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:35:21.667 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:35:21.667 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:35:21.667 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:35:21.667 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:35:21.668 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:35:21.668 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:35:21.668 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:35:21.712 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 9 2024-04-06T09:35:21.712 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 11 2024-04-06T09:35:21.712 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:35:21.712 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=45, tag_tid=6, entry_tid=1], [object_number=44, tag_tid=6, entry_tid=0], [object_number=47, tag_tid=5, entry_tid=22415], [object_number=46, tag_tid=5, entry_tid=22414]]], state=connected] 2024-04-06T09:35:21.713 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=37, tag_tid=5, entry_tid=11505], [object_number=36, tag_tid=5, entry_tid=11504], [object_number=39, tag_tid=5, entry_tid=11503], [object_number=38, tag_tid=5, entry_tid=11502]]], state=connected] 2024-04-06T09:35:21.716 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:35:21.718 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=37, tag_tid=5, entry_tid=11505' 2024-04-06T09:35:21.718 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=45, tag_tid=6, entry_tid=1' -a 'object_number=45, tag_tid=6, entry_tid=1' = 'object_number=37, tag_tid=5, entry_tid=11505' 2024-04-06T09:35:21.718 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=37, tag_tid=5, entry_tid=11505' '!=' 'object_number=36, tag_tid=5, entry_tid=11412' 2024-04-06T09:35:21.719 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:35:21.719 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=37, tag_tid=5, entry_tid=11505' = 'object_number=36, tag_tid=5, entry_tid=11412' 2024-04-06T09:35:21.719 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=37, tag_tid=5, entry_tid=11505' 2024-04-06T09:35:21.719 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=45, tag_tid=6, entry_tid=1' 2024-04-06T09:35:21.719 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:35:21.719 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:35:21.719 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=6 2024-04-06T09:35:21.720 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=37, tag_tid=5, entry_tid=11505' 2024-04-06T09:35:21.720 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:35:21.720 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:35:21.722 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=5 2024-04-06T09:35:21.722 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=45, tag_tid=6, entry_tid=1' 2024-04-06T09:35:21.722 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:35:21.722 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:35:21.723 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:35:21.723 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=37, tag_tid=5, entry_tid=11505' 2024-04-06T09:35:21.723 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:35:21.724 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:35:21.725 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=11505 2024-04-06T09:35:21.725 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 6 = 5 -a 1 -le 11505 2024-04-06T09:35:21.725 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:35:21.725 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:35:21.725 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:35:21.926 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:35:21.927 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:35:21.927 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:35:21.927 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:35:21.927 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:35:21.927 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:35:21.927 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:35:21.927 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:35:21.927 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:35:21.927 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:35:21.927 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:35:21.927 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:35:21.927 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:35:21.927 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:35:21.928 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:35:21.928 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:35:21.928 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:35:21.930 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:35:21.930 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:35:21.930 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:35:21.930 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:35:21.930 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:35:21.930 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:35:21.931 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:35:21.931 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:35:21.931 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:35:21.931 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:35:21.931 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:35:21.931 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:35:21.931 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:35:21.931 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:35:21.931 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:35:21.931 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:35:21.932 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:35:21.932 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:35:21.932 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:35:21.932 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:35:21.932 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:35:21.932 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:35:21.933 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:35:21.933 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:35:21.933 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:35:21.933 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:35:21.933 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:35:21.933 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:35:21.933 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:35:21.933 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:35:21.933 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:35:21.933 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:35:21.957 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:35:21.958 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:35:21.958 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:35:21.958 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:35:22.087 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:35:22.088 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:35:22.088 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:35:22.088 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:35:22.088 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:35:22.088 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:35:22.088 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:35:22.088 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:35:22.088 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:35:22.088 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:35:22.089 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:35:22.089 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:35:22.089 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:35:22.090 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:35:22.091 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:35:22.091 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:35:22.141 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 9 2024-04-06T09:35:22.141 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 11 2024-04-06T09:35:22.141 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:35:22.141 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=45, tag_tid=6, entry_tid=1], [object_number=44, tag_tid=6, entry_tid=0], [object_number=47, tag_tid=5, entry_tid=22415], [object_number=46, tag_tid=5, entry_tid=22414]]], state=connected] 2024-04-06T09:35:22.141 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=39, tag_tid=5, entry_tid=11603], [object_number=38, tag_tid=5, entry_tid=11602], [object_number=37, tag_tid=5, entry_tid=11601], [object_number=36, tag_tid=5, entry_tid=11600]]], state=connected] 2024-04-06T09:35:22.145 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:35:22.146 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=45, tag_tid=6, entry_tid=1' 2024-04-06T09:35:22.147 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:35:22.147 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:35:22.147 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:35:22.147 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:35:22.147 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:35:22.148 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:35:22.148 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:35:22.148 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:35:22.148 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:35:22.148 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:35:22.148 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:35:22.148 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:35:22.148 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:35:22.149 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:35:22.149 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:35:22.194 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 9 2024-04-06T09:35:22.194 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 11 2024-04-06T09:35:22.194 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:35:22.195 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=45, tag_tid=6, entry_tid=1], [object_number=44, tag_tid=6, entry_tid=0], [object_number=47, tag_tid=5, entry_tid=22415], [object_number=46, tag_tid=5, entry_tid=22414]]], state=connected] 2024-04-06T09:35:22.195 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=39, tag_tid=5, entry_tid=11603], [object_number=38, tag_tid=5, entry_tid=11602], [object_number=37, tag_tid=5, entry_tid=11601], [object_number=36, tag_tid=5, entry_tid=11600]]], state=connected] 2024-04-06T09:35:22.198 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:35:22.199 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=39, tag_tid=5, entry_tid=11603' 2024-04-06T09:35:22.200 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=45, tag_tid=6, entry_tid=1' -a 'object_number=45, tag_tid=6, entry_tid=1' = 'object_number=39, tag_tid=5, entry_tid=11603' 2024-04-06T09:35:22.200 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=39, tag_tid=5, entry_tid=11603' '!=' 'object_number=37, tag_tid=5, entry_tid=11505' 2024-04-06T09:35:22.200 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:35:22.200 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=39, tag_tid=5, entry_tid=11603' = 'object_number=37, tag_tid=5, entry_tid=11505' 2024-04-06T09:35:22.200 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=39, tag_tid=5, entry_tid=11603' 2024-04-06T09:35:22.201 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=45, tag_tid=6, entry_tid=1' 2024-04-06T09:35:22.201 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:35:22.201 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:35:22.201 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=6 2024-04-06T09:35:22.202 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=39, tag_tid=5, entry_tid=11603' 2024-04-06T09:35:22.202 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:35:22.202 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:35:22.203 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=5 2024-04-06T09:35:22.203 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=45, tag_tid=6, entry_tid=1' 2024-04-06T09:35:22.203 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:35:22.203 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:35:22.204 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:35:22.205 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=39, tag_tid=5, entry_tid=11603' 2024-04-06T09:35:22.205 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:35:22.205 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:35:22.206 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=11603 2024-04-06T09:35:22.206 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 6 = 5 -a 1 -le 11603 2024-04-06T09:35:22.206 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:35:22.206 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:35:22.207 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:35:22.408 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:35:22.408 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:35:22.408 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:35:22.408 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:35:22.408 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:35:22.408 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:35:22.408 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:35:22.408 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:35:22.408 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:35:22.409 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:35:22.409 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:35:22.409 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:35:22.409 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:35:22.409 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:35:22.409 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:35:22.409 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:35:22.410 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:35:22.411 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:35:22.411 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:35:22.412 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:35:22.412 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:35:22.412 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:35:22.412 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:35:22.412 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:35:22.412 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:35:22.412 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:35:22.412 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:35:22.412 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:35:22.412 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:35:22.412 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:35:22.412 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:35:22.412 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:35:22.412 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:35:22.413 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:35:22.413 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:35:22.413 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:35:22.413 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:35:22.414 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:35:22.414 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:35:22.414 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:35:22.414 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:35:22.414 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:35:22.414 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:35:22.414 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:35:22.414 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:35:22.414 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:35:22.414 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:35:22.414 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:35:22.414 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:35:22.438 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:35:22.439 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:35:22.439 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:35:22.439 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:35:22.562 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:35:22.563 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:35:22.563 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:35:22.563 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:35:22.564 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:35:22.564 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:35:22.564 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:35:22.564 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:35:22.564 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:35:22.564 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:35:22.564 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:35:22.565 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:35:22.565 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:35:22.567 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:35:22.567 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:35:22.567 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:35:22.613 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 9 2024-04-06T09:35:22.613 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 11 2024-04-06T09:35:22.613 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:35:22.613 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=45, tag_tid=6, entry_tid=1], [object_number=44, tag_tid=6, entry_tid=0], [object_number=47, tag_tid=5, entry_tid=22415], [object_number=46, tag_tid=5, entry_tid=22414]]], state=connected] 2024-04-06T09:35:22.613 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=39, tag_tid=5, entry_tid=11695], [object_number=38, tag_tid=5, entry_tid=11694], [object_number=37, tag_tid=5, entry_tid=11693], [object_number=36, tag_tid=5, entry_tid=11692]]], state=connected] 2024-04-06T09:35:22.616 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:35:22.619 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=45, tag_tid=6, entry_tid=1' 2024-04-06T09:35:22.619 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:35:22.619 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:35:22.619 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:35:22.619 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:35:22.619 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:35:22.619 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:35:22.619 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:35:22.619 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:35:22.619 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:35:22.619 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:35:22.619 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:35:22.619 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:35:22.620 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:35:22.620 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:35:22.620 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:35:22.666 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 9 2024-04-06T09:35:22.666 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 11 2024-04-06T09:35:22.666 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:35:22.666 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=45, tag_tid=6, entry_tid=1], [object_number=44, tag_tid=6, entry_tid=0], [object_number=47, tag_tid=5, entry_tid=22415], [object_number=46, tag_tid=5, entry_tid=22414]]], state=connected] 2024-04-06T09:35:22.666 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=39, tag_tid=5, entry_tid=11695], [object_number=38, tag_tid=5, entry_tid=11694], [object_number=37, tag_tid=5, entry_tid=11693], [object_number=36, tag_tid=5, entry_tid=11692]]], state=connected] 2024-04-06T09:35:22.670 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:35:22.672 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=39, tag_tid=5, entry_tid=11695' 2024-04-06T09:35:22.672 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=45, tag_tid=6, entry_tid=1' -a 'object_number=45, tag_tid=6, entry_tid=1' = 'object_number=39, tag_tid=5, entry_tid=11695' 2024-04-06T09:35:22.672 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=39, tag_tid=5, entry_tid=11695' '!=' 'object_number=39, tag_tid=5, entry_tid=11603' 2024-04-06T09:35:22.672 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:35:22.672 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=39, tag_tid=5, entry_tid=11695' = 'object_number=39, tag_tid=5, entry_tid=11603' 2024-04-06T09:35:22.672 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=39, tag_tid=5, entry_tid=11695' 2024-04-06T09:35:22.672 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=45, tag_tid=6, entry_tid=1' 2024-04-06T09:35:22.672 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:35:22.673 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:35:22.673 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=6 2024-04-06T09:35:22.674 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=39, tag_tid=5, entry_tid=11695' 2024-04-06T09:35:22.674 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:35:22.674 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:35:22.675 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=5 2024-04-06T09:35:22.675 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=45, tag_tid=6, entry_tid=1' 2024-04-06T09:35:22.675 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:35:22.676 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:35:22.676 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:35:22.677 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=39, tag_tid=5, entry_tid=11695' 2024-04-06T09:35:22.677 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:35:22.677 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:35:22.678 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=11695 2024-04-06T09:35:22.678 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 6 = 5 -a 1 -le 11695 2024-04-06T09:35:22.679 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:35:22.679 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:35:22.679 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:35:22.881 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:35:22.881 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:35:22.881 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:35:22.881 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:35:22.881 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:35:22.881 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:35:22.881 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:35:22.881 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:35:22.881 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:35:22.881 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:35:22.881 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:35:22.881 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:35:22.882 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:35:22.882 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:35:22.882 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:35:22.882 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:35:22.882 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:35:22.883 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:35:22.883 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:35:22.883 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:35:22.884 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:35:22.884 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:35:22.884 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:35:22.884 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:35:22.884 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:35:22.884 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:35:22.884 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:35:22.884 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:35:22.884 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:35:22.884 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:35:22.884 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:35:22.884 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:35:22.884 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:35:22.886 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:35:22.886 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:35:22.886 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:35:22.886 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:35:22.886 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:35:22.886 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:35:22.886 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:35:22.886 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:35:22.886 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:35:22.886 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:35:22.886 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:35:22.886 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:35:22.887 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:35:22.887 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:35:22.887 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:35:22.887 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:35:22.912 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:35:22.913 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:35:22.913 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:35:22.913 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:35:23.045 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:35:23.045 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:35:23.046 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:35:23.046 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:35:23.046 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:35:23.046 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:35:23.046 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:35:23.046 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:35:23.046 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:35:23.046 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:35:23.047 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:35:23.047 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:35:23.047 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:35:23.048 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:35:23.048 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:35:23.048 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:35:23.099 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 9 2024-04-06T09:35:23.099 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 11 2024-04-06T09:35:23.099 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:35:23.099 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=45, tag_tid=6, entry_tid=1], [object_number=44, tag_tid=6, entry_tid=0], [object_number=47, tag_tid=5, entry_tid=22415], [object_number=46, tag_tid=5, entry_tid=22414]]], state=connected] 2024-04-06T09:35:23.100 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=38, tag_tid=5, entry_tid=11786], [object_number=37, tag_tid=5, entry_tid=11785], [object_number=36, tag_tid=5, entry_tid=11784], [object_number=39, tag_tid=5, entry_tid=11783]]], state=connected] 2024-04-06T09:35:23.103 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:35:23.104 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=45, tag_tid=6, entry_tid=1' 2024-04-06T09:35:23.105 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:35:23.105 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:35:23.105 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:35:23.105 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:35:23.105 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:35:23.105 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:35:23.106 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:35:23.106 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:35:23.106 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:35:23.106 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:35:23.106 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:35:23.106 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:35:23.107 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:35:23.107 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:35:23.107 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:35:23.152 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 9 2024-04-06T09:35:23.152 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 11 2024-04-06T09:35:23.152 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:35:23.153 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=45, tag_tid=6, entry_tid=1], [object_number=44, tag_tid=6, entry_tid=0], [object_number=47, tag_tid=5, entry_tid=22415], [object_number=46, tag_tid=5, entry_tid=22414]]], state=connected] 2024-04-06T09:35:23.153 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=38, tag_tid=5, entry_tid=11786], [object_number=37, tag_tid=5, entry_tid=11785], [object_number=36, tag_tid=5, entry_tid=11784], [object_number=39, tag_tid=5, entry_tid=11783]]], state=connected] 2024-04-06T09:35:23.156 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:35:23.158 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=38, tag_tid=5, entry_tid=11786' 2024-04-06T09:35:23.158 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=45, tag_tid=6, entry_tid=1' -a 'object_number=45, tag_tid=6, entry_tid=1' = 'object_number=38, tag_tid=5, entry_tid=11786' 2024-04-06T09:35:23.158 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=38, tag_tid=5, entry_tid=11786' '!=' 'object_number=39, tag_tid=5, entry_tid=11695' 2024-04-06T09:35:23.158 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:35:23.158 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=38, tag_tid=5, entry_tid=11786' = 'object_number=39, tag_tid=5, entry_tid=11695' 2024-04-06T09:35:23.158 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=38, tag_tid=5, entry_tid=11786' 2024-04-06T09:35:23.159 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=45, tag_tid=6, entry_tid=1' 2024-04-06T09:35:23.159 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:35:23.159 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:35:23.159 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=6 2024-04-06T09:35:23.160 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=38, tag_tid=5, entry_tid=11786' 2024-04-06T09:35:23.160 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:35:23.160 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:35:23.161 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=5 2024-04-06T09:35:23.162 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=45, tag_tid=6, entry_tid=1' 2024-04-06T09:35:23.162 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:35:23.162 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:35:23.163 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:35:23.163 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=38, tag_tid=5, entry_tid=11786' 2024-04-06T09:35:23.163 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:35:23.163 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:35:23.165 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=11786 2024-04-06T09:35:23.165 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 6 = 5 -a 1 -le 11786 2024-04-06T09:35:23.165 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:35:23.165 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:35:23.165 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:35:23.366 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:35:23.366 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:35:23.366 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:35:23.366 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:35:23.367 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:35:23.367 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:35:23.367 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:35:23.367 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:35:23.367 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:35:23.367 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:35:23.367 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:35:23.367 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:35:23.367 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:35:23.367 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:35:23.367 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:35:23.367 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:35:23.368 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:35:23.371 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:35:23.371 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:35:23.371 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:35:23.372 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:35:23.372 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:35:23.372 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:35:23.372 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:35:23.372 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:35:23.372 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:35:23.372 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:35:23.372 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:35:23.372 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:35:23.372 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:35:23.372 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:35:23.372 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:35:23.372 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:35:23.372 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:35:23.372 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:35:23.372 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:35:23.373 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:35:23.373 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:35:23.373 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:35:23.373 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:35:23.373 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:35:23.373 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:35:23.373 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:35:23.373 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:35:23.373 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:35:23.373 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:35:23.373 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:35:23.373 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:35:23.373 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:35:23.397 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:35:23.398 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:35:23.398 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:35:23.398 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:35:23.524 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:35:23.526 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:35:23.526 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:35:23.526 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:35:23.526 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:35:23.526 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:35:23.526 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:35:23.526 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:35:23.526 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:35:23.526 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:35:23.526 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:35:23.527 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:35:23.527 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:35:23.529 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:35:23.529 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:35:23.529 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:35:23.579 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 9 2024-04-06T09:35:23.579 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 11 2024-04-06T09:35:23.579 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:35:23.579 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=45, tag_tid=6, entry_tid=1], [object_number=44, tag_tid=6, entry_tid=0], [object_number=47, tag_tid=5, entry_tid=22415], [object_number=46, tag_tid=5, entry_tid=22414]]], state=connected] 2024-04-06T09:35:23.579 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=37, tag_tid=5, entry_tid=11885], [object_number=36, tag_tid=5, entry_tid=11884], [object_number=39, tag_tid=5, entry_tid=11883], [object_number=38, tag_tid=5, entry_tid=11882]]], state=connected] 2024-04-06T09:35:23.583 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:35:23.584 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=45, tag_tid=6, entry_tid=1' 2024-04-06T09:35:23.585 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:35:23.585 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:35:23.585 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:35:23.585 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:35:23.585 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:35:23.585 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:35:23.585 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:35:23.586 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:35:23.586 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:35:23.586 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:35:23.586 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:35:23.586 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:35:23.586 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:35:23.587 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:35:23.587 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:35:23.632 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 9 2024-04-06T09:35:23.632 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 11 2024-04-06T09:35:23.632 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:35:23.632 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=45, tag_tid=6, entry_tid=1], [object_number=44, tag_tid=6, entry_tid=0], [object_number=47, tag_tid=5, entry_tid=22415], [object_number=46, tag_tid=5, entry_tid=22414]]], state=connected] 2024-04-06T09:35:23.632 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=37, tag_tid=5, entry_tid=11885], [object_number=36, tag_tid=5, entry_tid=11884], [object_number=39, tag_tid=5, entry_tid=11883], [object_number=38, tag_tid=5, entry_tid=11882]]], state=connected] 2024-04-06T09:35:23.636 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:35:23.638 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=37, tag_tid=5, entry_tid=11885' 2024-04-06T09:35:23.638 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=45, tag_tid=6, entry_tid=1' -a 'object_number=45, tag_tid=6, entry_tid=1' = 'object_number=37, tag_tid=5, entry_tid=11885' 2024-04-06T09:35:23.638 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=37, tag_tid=5, entry_tid=11885' '!=' 'object_number=38, tag_tid=5, entry_tid=11786' 2024-04-06T09:35:23.638 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:35:23.638 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=37, tag_tid=5, entry_tid=11885' = 'object_number=38, tag_tid=5, entry_tid=11786' 2024-04-06T09:35:23.638 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=37, tag_tid=5, entry_tid=11885' 2024-04-06T09:35:23.639 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=45, tag_tid=6, entry_tid=1' 2024-04-06T09:35:23.639 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:35:23.639 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:35:23.639 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=6 2024-04-06T09:35:23.640 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=37, tag_tid=5, entry_tid=11885' 2024-04-06T09:35:23.640 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:35:23.640 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:35:23.641 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=5 2024-04-06T09:35:23.641 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=45, tag_tid=6, entry_tid=1' 2024-04-06T09:35:23.642 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:35:23.642 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:35:23.643 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:35:23.643 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=37, tag_tid=5, entry_tid=11885' 2024-04-06T09:35:23.643 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:35:23.643 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:35:23.644 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=11885 2024-04-06T09:35:23.644 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 6 = 5 -a 1 -le 11885 2024-04-06T09:35:23.645 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:35:23.645 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:35:23.645 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:35:23.846 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:35:23.846 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:35:23.846 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:35:23.846 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:35:23.846 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:35:23.846 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:35:23.846 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:35:23.846 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:35:23.846 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:35:23.847 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:35:23.847 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:35:23.847 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:35:23.847 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:35:23.847 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:35:23.847 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:35:23.847 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:35:23.847 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:35:23.849 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:35:23.849 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:35:23.849 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:35:23.849 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:35:23.849 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:35:23.850 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:35:23.850 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:35:23.850 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:35:23.850 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:35:23.850 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:35:23.850 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:35:23.850 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:35:23.850 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:35:23.850 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:35:23.850 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:35:23.850 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:35:23.852 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:35:23.852 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:35:23.852 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:35:23.852 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:35:23.852 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:35:23.852 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:35:23.852 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:35:23.852 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:35:23.852 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:35:23.852 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:35:23.852 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:35:23.853 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:35:23.853 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:35:23.853 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:35:23.853 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:35:23.853 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:35:23.878 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:35:23.878 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:35:23.878 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:35:23.879 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:35:24.009 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:35:24.011 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:35:24.011 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:35:24.011 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:35:24.011 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:35:24.011 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:35:24.011 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:35:24.011 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:35:24.011 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:35:24.011 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:35:24.011 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:35:24.011 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:35:24.012 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:35:24.012 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:35:24.012 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:35:24.012 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:35:24.068 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 9 2024-04-06T09:35:24.068 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 11 2024-04-06T09:35:24.068 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:35:24.068 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=45, tag_tid=6, entry_tid=1], [object_number=44, tag_tid=6, entry_tid=0], [object_number=47, tag_tid=5, entry_tid=22415], [object_number=46, tag_tid=5, entry_tid=22414]]], state=connected] 2024-04-06T09:35:24.068 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=38, tag_tid=5, entry_tid=11982], [object_number=37, tag_tid=5, entry_tid=11981], [object_number=36, tag_tid=5, entry_tid=11980], [object_number=39, tag_tid=5, entry_tid=11979]]], state=connected] 2024-04-06T09:35:24.072 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:35:24.073 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=45, tag_tid=6, entry_tid=1' 2024-04-06T09:35:24.074 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:35:24.074 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:35:24.074 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:35:24.074 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:35:24.075 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:35:24.075 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:35:24.075 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:35:24.075 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:35:24.075 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:35:24.075 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:35:24.075 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:35:24.075 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:35:24.076 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:35:24.076 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:35:24.076 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:35:24.122 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 9 2024-04-06T09:35:24.122 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 11 2024-04-06T09:35:24.122 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:35:24.122 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=45, tag_tid=6, entry_tid=1], [object_number=44, tag_tid=6, entry_tid=0], [object_number=47, tag_tid=5, entry_tid=22415], [object_number=46, tag_tid=5, entry_tid=22414]]], state=connected] 2024-04-06T09:35:24.122 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=38, tag_tid=5, entry_tid=11982], [object_number=37, tag_tid=5, entry_tid=11981], [object_number=36, tag_tid=5, entry_tid=11980], [object_number=39, tag_tid=5, entry_tid=11979]]], state=connected] 2024-04-06T09:35:24.126 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:35:24.128 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=38, tag_tid=5, entry_tid=11982' 2024-04-06T09:35:24.128 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=45, tag_tid=6, entry_tid=1' -a 'object_number=45, tag_tid=6, entry_tid=1' = 'object_number=38, tag_tid=5, entry_tid=11982' 2024-04-06T09:35:24.128 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=38, tag_tid=5, entry_tid=11982' '!=' 'object_number=37, tag_tid=5, entry_tid=11885' 2024-04-06T09:35:24.128 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:35:24.128 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=38, tag_tid=5, entry_tid=11982' = 'object_number=37, tag_tid=5, entry_tid=11885' 2024-04-06T09:35:24.128 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=38, tag_tid=5, entry_tid=11982' 2024-04-06T09:35:24.128 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=45, tag_tid=6, entry_tid=1' 2024-04-06T09:35:24.128 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:35:24.128 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:35:24.129 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=6 2024-04-06T09:35:24.130 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=38, tag_tid=5, entry_tid=11982' 2024-04-06T09:35:24.130 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:35:24.130 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:35:24.131 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=5 2024-04-06T09:35:24.131 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=45, tag_tid=6, entry_tid=1' 2024-04-06T09:35:24.131 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:35:24.131 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:35:24.133 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:35:24.133 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=38, tag_tid=5, entry_tid=11982' 2024-04-06T09:35:24.133 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:35:24.133 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:35:24.134 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=11982 2024-04-06T09:35:24.134 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 6 = 5 -a 1 -le 11982 2024-04-06T09:35:24.134 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:35:24.135 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:35:24.135 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:35:24.336 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:35:24.336 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:35:24.336 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:35:24.336 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:35:24.336 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:35:24.336 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:35:24.336 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:35:24.336 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:35:24.336 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:35:24.336 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:35:24.337 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:35:24.337 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:35:24.337 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:35:24.337 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:35:24.337 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:35:24.337 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:35:24.338 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:35:24.341 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:35:24.341 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:35:24.341 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:35:24.341 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:35:24.341 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:35:24.342 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:35:24.342 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:35:24.342 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:35:24.342 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:35:24.342 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:35:24.342 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:35:24.342 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:35:24.342 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:35:24.342 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:35:24.342 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:35:24.342 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:35:24.342 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:35:24.342 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:35:24.342 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:35:24.342 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:35:24.343 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:35:24.343 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:35:24.343 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:35:24.343 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:35:24.343 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:35:24.343 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:35:24.343 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:35:24.343 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:35:24.343 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:35:24.343 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:35:24.343 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:35:24.343 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:35:24.366 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:35:24.367 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:35:24.367 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:35:24.367 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:35:24.497 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:35:24.498 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:35:24.498 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:35:24.498 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:35:24.499 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:35:24.499 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:35:24.499 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:35:24.499 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:35:24.499 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:35:24.499 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:35:24.499 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:35:24.500 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:35:24.500 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:35:24.502 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:35:24.503 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:35:24.503 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:35:24.553 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 9 2024-04-06T09:35:24.553 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 11 2024-04-06T09:35:24.553 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:35:24.553 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=45, tag_tid=6, entry_tid=1], [object_number=44, tag_tid=6, entry_tid=0], [object_number=47, tag_tid=5, entry_tid=22415], [object_number=46, tag_tid=5, entry_tid=22414]]], state=connected] 2024-04-06T09:35:24.553 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=36, tag_tid=5, entry_tid=12088], [object_number=39, tag_tid=5, entry_tid=12087], [object_number=38, tag_tid=5, entry_tid=12086], [object_number=37, tag_tid=5, entry_tid=12085]]], state=connected] 2024-04-06T09:35:24.557 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:35:24.558 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=45, tag_tid=6, entry_tid=1' 2024-04-06T09:35:24.559 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:35:24.559 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:35:24.559 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:35:24.559 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:35:24.560 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:35:24.560 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:35:24.560 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:35:24.560 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:35:24.560 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:35:24.560 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:35:24.560 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:35:24.560 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:35:24.560 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:35:24.561 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:35:24.561 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:35:24.606 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 9 2024-04-06T09:35:24.606 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 11 2024-04-06T09:35:24.606 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:35:24.607 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=45, tag_tid=6, entry_tid=1], [object_number=44, tag_tid=6, entry_tid=0], [object_number=47, tag_tid=5, entry_tid=22415], [object_number=46, tag_tid=5, entry_tid=22414]]], state=connected] 2024-04-06T09:35:24.607 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=36, tag_tid=5, entry_tid=12088], [object_number=39, tag_tid=5, entry_tid=12087], [object_number=38, tag_tid=5, entry_tid=12086], [object_number=37, tag_tid=5, entry_tid=12085]]], state=connected] 2024-04-06T09:35:24.610 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:35:24.612 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=36, tag_tid=5, entry_tid=12088' 2024-04-06T09:35:24.612 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=45, tag_tid=6, entry_tid=1' -a 'object_number=45, tag_tid=6, entry_tid=1' = 'object_number=36, tag_tid=5, entry_tid=12088' 2024-04-06T09:35:24.612 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=36, tag_tid=5, entry_tid=12088' '!=' 'object_number=38, tag_tid=5, entry_tid=11982' 2024-04-06T09:35:24.612 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:35:24.612 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=36, tag_tid=5, entry_tid=12088' = 'object_number=38, tag_tid=5, entry_tid=11982' 2024-04-06T09:35:24.612 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=36, tag_tid=5, entry_tid=12088' 2024-04-06T09:35:24.612 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=45, tag_tid=6, entry_tid=1' 2024-04-06T09:35:24.612 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:35:24.612 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:35:24.613 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=6 2024-04-06T09:35:24.614 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=36, tag_tid=5, entry_tid=12088' 2024-04-06T09:35:24.614 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:35:24.614 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:35:24.615 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=5 2024-04-06T09:35:24.616 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=45, tag_tid=6, entry_tid=1' 2024-04-06T09:35:24.616 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:35:24.616 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:35:24.617 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:35:24.617 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=36, tag_tid=5, entry_tid=12088' 2024-04-06T09:35:24.617 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:35:24.617 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:35:24.619 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=12088 2024-04-06T09:35:24.619 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 6 = 5 -a 1 -le 12088 2024-04-06T09:35:24.619 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:35:24.619 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:35:24.619 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:35:24.820 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:35:24.821 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:35:24.821 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:35:24.821 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:35:24.821 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:35:24.821 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:35:24.821 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:35:24.821 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:35:24.821 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:35:24.821 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:35:24.821 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:35:24.821 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:35:24.821 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:35:24.821 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:35:24.821 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:35:24.822 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:35:24.822 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:35:24.825 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:35:24.825 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:35:24.825 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:35:24.825 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:35:24.825 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:35:24.825 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:35:24.825 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:35:24.825 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:35:24.826 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:35:24.826 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:35:24.826 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:35:24.826 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:35:24.826 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:35:24.826 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:35:24.826 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:35:24.826 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:35:24.826 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:35:24.826 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:35:24.826 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:35:24.826 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:35:24.826 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:35:24.826 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:35:24.826 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:35:24.827 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:35:24.827 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:35:24.827 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:35:24.827 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:35:24.827 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:35:24.827 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:35:24.827 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:35:24.827 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:35:24.827 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:35:24.851 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:35:24.851 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:35:24.852 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:35:24.852 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:35:24.984 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:35:24.985 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:35:24.985 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:35:24.985 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:35:24.985 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:35:24.985 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:35:24.986 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:35:24.986 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:35:24.986 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:35:24.986 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:35:24.986 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:35:24.986 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:35:24.987 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:35:24.990 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:35:24.990 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:35:24.990 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:35:25.044 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 9 2024-04-06T09:35:25.044 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 11 2024-04-06T09:35:25.044 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:35:25.044 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=45, tag_tid=6, entry_tid=1], [object_number=44, tag_tid=6, entry_tid=0], [object_number=47, tag_tid=5, entry_tid=22415], [object_number=46, tag_tid=5, entry_tid=22414]]], state=connected] 2024-04-06T09:35:25.044 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=37, tag_tid=5, entry_tid=12193], [object_number=36, tag_tid=5, entry_tid=12192], [object_number=39, tag_tid=5, entry_tid=12191], [object_number=38, tag_tid=5, entry_tid=12190]]], state=connected] 2024-04-06T09:35:25.048 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:35:25.049 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=45, tag_tid=6, entry_tid=1' 2024-04-06T09:35:25.051 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:35:25.051 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:35:25.051 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:35:25.051 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:35:25.051 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:35:25.051 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:35:25.051 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:35:25.051 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:35:25.051 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:35:25.051 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:35:25.051 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:35:25.051 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:35:25.052 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:35:25.052 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:35:25.052 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:35:25.100 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 9 2024-04-06T09:35:25.100 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 11 2024-04-06T09:35:25.100 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:35:25.100 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=45, tag_tid=6, entry_tid=1], [object_number=44, tag_tid=6, entry_tid=0], [object_number=47, tag_tid=5, entry_tid=22415], [object_number=46, tag_tid=5, entry_tid=22414]]], state=connected] 2024-04-06T09:35:25.100 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=37, tag_tid=5, entry_tid=12193], [object_number=36, tag_tid=5, entry_tid=12192], [object_number=39, tag_tid=5, entry_tid=12191], [object_number=38, tag_tid=5, entry_tid=12190]]], state=connected] 2024-04-06T09:35:25.103 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:35:25.105 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=37, tag_tid=5, entry_tid=12193' 2024-04-06T09:35:25.105 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=45, tag_tid=6, entry_tid=1' -a 'object_number=45, tag_tid=6, entry_tid=1' = 'object_number=37, tag_tid=5, entry_tid=12193' 2024-04-06T09:35:25.105 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=37, tag_tid=5, entry_tid=12193' '!=' 'object_number=36, tag_tid=5, entry_tid=12088' 2024-04-06T09:35:25.105 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:35:25.105 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=37, tag_tid=5, entry_tid=12193' = 'object_number=36, tag_tid=5, entry_tid=12088' 2024-04-06T09:35:25.105 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=37, tag_tid=5, entry_tid=12193' 2024-04-06T09:35:25.105 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=45, tag_tid=6, entry_tid=1' 2024-04-06T09:35:25.106 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:35:25.106 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:35:25.106 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=6 2024-04-06T09:35:25.107 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=37, tag_tid=5, entry_tid=12193' 2024-04-06T09:35:25.107 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:35:25.107 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:35:25.108 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=5 2024-04-06T09:35:25.108 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=45, tag_tid=6, entry_tid=1' 2024-04-06T09:35:25.109 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:35:25.109 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:35:25.110 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:35:25.110 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=37, tag_tid=5, entry_tid=12193' 2024-04-06T09:35:25.110 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:35:25.110 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:35:25.111 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=12193 2024-04-06T09:35:25.112 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 6 = 5 -a 1 -le 12193 2024-04-06T09:35:25.112 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:35:25.112 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:35:25.112 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:35:25.313 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:35:25.313 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:35:25.314 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:35:25.314 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:35:25.314 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:35:25.314 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:35:25.314 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:35:25.314 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:35:25.314 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:35:25.314 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:35:25.314 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:35:25.314 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:35:25.314 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:35:25.314 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:35:25.314 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:35:25.315 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:35:25.315 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:35:25.316 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:35:25.316 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:35:25.316 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:35:25.316 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:35:25.316 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:35:25.316 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:35:25.316 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:35:25.316 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:35:25.317 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:35:25.317 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:35:25.317 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:35:25.317 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:35:25.317 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:35:25.317 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:35:25.317 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:35:25.317 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:35:25.318 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:35:25.318 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:35:25.318 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:35:25.318 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:35:25.319 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:35:25.319 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:35:25.319 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:35:25.319 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:35:25.319 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:35:25.319 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:35:25.319 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:35:25.319 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:35:25.319 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:35:25.319 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:35:25.319 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:35:25.319 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:35:25.344 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:35:25.344 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:35:25.344 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:35:25.344 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:35:25.379 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.0 with skip (0.3287963145128626) > weight (0.25) 2024-04-06T09:35:25.379 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.1 with skip (0.3658788666128928) > weight (0.25) 2024-04-06T09:35:25.379 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.2 with skip (0.27212861791818244) > weight (0.25) 2024-04-06T09:35:25.380 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.3 with skip (0.2924044289854456) > weight (0.25) 2024-04-06T09:35:25.380 INFO:tasks.rbd_mirror_thrash:waiting for 92 secs before thrashing 2024-04-06T09:35:25.469 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:35:25.470 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:35:25.470 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:35:25.470 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:35:25.470 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:35:25.470 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:35:25.470 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:35:25.471 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:35:25.471 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:35:25.471 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:35:25.471 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:35:25.471 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:35:25.471 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:35:25.473 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:35:25.473 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:35:25.473 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:35:25.528 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 9 2024-04-06T09:35:25.528 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 11 2024-04-06T09:35:25.528 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:35:25.529 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=45, tag_tid=6, entry_tid=1], [object_number=44, tag_tid=6, entry_tid=0], [object_number=47, tag_tid=5, entry_tid=22415], [object_number=46, tag_tid=5, entry_tid=22414]]], state=connected] 2024-04-06T09:35:25.529 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=36, tag_tid=5, entry_tid=12292], [object_number=39, tag_tid=5, entry_tid=12291], [object_number=38, tag_tid=5, entry_tid=12290], [object_number=37, tag_tid=5, entry_tid=12289]]], state=connected] 2024-04-06T09:35:25.532 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:35:25.533 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=45, tag_tid=6, entry_tid=1' 2024-04-06T09:35:25.535 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:35:25.535 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:35:25.535 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:35:25.535 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:35:25.535 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:35:25.535 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:35:25.535 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:35:25.535 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:35:25.535 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:35:25.535 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:35:25.535 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:35:25.535 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:35:25.536 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:35:25.536 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:35:25.536 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:35:25.583 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 9 2024-04-06T09:35:25.583 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 11 2024-04-06T09:35:25.583 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:35:25.584 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=45, tag_tid=6, entry_tid=1], [object_number=44, tag_tid=6, entry_tid=0], [object_number=47, tag_tid=5, entry_tid=22415], [object_number=46, tag_tid=5, entry_tid=22414]]], state=connected] 2024-04-06T09:35:25.584 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=36, tag_tid=5, entry_tid=12292], [object_number=39, tag_tid=5, entry_tid=12291], [object_number=38, tag_tid=5, entry_tid=12290], [object_number=37, tag_tid=5, entry_tid=12289]]], state=connected] 2024-04-06T09:35:25.587 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:35:25.589 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=36, tag_tid=5, entry_tid=12292' 2024-04-06T09:35:25.589 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=45, tag_tid=6, entry_tid=1' -a 'object_number=45, tag_tid=6, entry_tid=1' = 'object_number=36, tag_tid=5, entry_tid=12292' 2024-04-06T09:35:25.589 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=36, tag_tid=5, entry_tid=12292' '!=' 'object_number=37, tag_tid=5, entry_tid=12193' 2024-04-06T09:35:25.589 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:35:25.590 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=36, tag_tid=5, entry_tid=12292' = 'object_number=37, tag_tid=5, entry_tid=12193' 2024-04-06T09:35:25.590 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=36, tag_tid=5, entry_tid=12292' 2024-04-06T09:35:25.590 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=45, tag_tid=6, entry_tid=1' 2024-04-06T09:35:25.590 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:35:25.590 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:35:25.590 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=6 2024-04-06T09:35:25.591 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=36, tag_tid=5, entry_tid=12292' 2024-04-06T09:35:25.591 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:35:25.591 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:35:25.592 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=5 2024-04-06T09:35:25.593 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=45, tag_tid=6, entry_tid=1' 2024-04-06T09:35:25.593 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:35:25.593 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:35:25.594 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:35:25.594 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=36, tag_tid=5, entry_tid=12292' 2024-04-06T09:35:25.595 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:35:25.595 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:35:25.596 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=12292 2024-04-06T09:35:25.596 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 6 = 5 -a 1 -le 12292 2024-04-06T09:35:25.596 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:35:25.596 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:35:25.596 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:35:25.798 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:35:25.798 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:35:25.798 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:35:25.798 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:35:25.798 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:35:25.798 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:35:25.798 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:35:25.798 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:35:25.799 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:35:25.799 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:35:25.799 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:35:25.799 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:35:25.799 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:35:25.799 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:35:25.799 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:35:25.799 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:35:25.800 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:35:25.800 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:35:25.800 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:35:25.801 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:35:25.801 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:35:25.801 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:35:25.801 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:35:25.801 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:35:25.801 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:35:25.801 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:35:25.801 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:35:25.801 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:35:25.801 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:35:25.801 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:35:25.801 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:35:25.801 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:35:25.801 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:35:25.803 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:35:25.803 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:35:25.803 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:35:25.803 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:35:25.803 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:35:25.803 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:35:25.803 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:35:25.803 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:35:25.803 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:35:25.803 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:35:25.803 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:35:25.803 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:35:25.804 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:35:25.804 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:35:25.804 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:35:25.804 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:35:25.827 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:35:25.827 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:35:25.827 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:35:25.828 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:35:25.962 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:35:25.963 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:35:25.963 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:35:25.963 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:35:25.963 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:35:25.963 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:35:25.963 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:35:25.963 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:35:25.963 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:35:25.963 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:35:25.963 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:35:25.964 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:35:25.964 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:35:25.965 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:35:25.965 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:35:25.965 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:35:26.010 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 9 2024-04-06T09:35:26.010 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 11 2024-04-06T09:35:26.011 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:35:26.011 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=45, tag_tid=6, entry_tid=1], [object_number=44, tag_tid=6, entry_tid=0], [object_number=47, tag_tid=5, entry_tid=22415], [object_number=46, tag_tid=5, entry_tid=22414]]], state=connected] 2024-04-06T09:35:26.011 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=37, tag_tid=5, entry_tid=12385], [object_number=36, tag_tid=5, entry_tid=12384], [object_number=39, tag_tid=5, entry_tid=12383], [object_number=38, tag_tid=5, entry_tid=12382]]], state=connected] 2024-04-06T09:35:26.040 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:35:26.042 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=45, tag_tid=6, entry_tid=1' 2024-04-06T09:35:26.043 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:35:26.043 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:35:26.043 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:35:26.043 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:35:26.043 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:35:26.043 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:35:26.043 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:35:26.043 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:35:26.043 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:35:26.044 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:35:26.044 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:35:26.044 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:35:26.044 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:35:26.045 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:35:26.045 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:35:26.090 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 9 2024-04-06T09:35:26.091 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 11 2024-04-06T09:35:26.091 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:35:26.091 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=45, tag_tid=6, entry_tid=1], [object_number=44, tag_tid=6, entry_tid=0], [object_number=47, tag_tid=5, entry_tid=22415], [object_number=46, tag_tid=5, entry_tid=22414]]], state=connected] 2024-04-06T09:35:26.091 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=37, tag_tid=5, entry_tid=12385], [object_number=36, tag_tid=5, entry_tid=12384], [object_number=39, tag_tid=5, entry_tid=12383], [object_number=38, tag_tid=5, entry_tid=12382]]], state=connected] 2024-04-06T09:35:26.094 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:35:26.096 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=37, tag_tid=5, entry_tid=12385' 2024-04-06T09:35:26.096 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=45, tag_tid=6, entry_tid=1' -a 'object_number=45, tag_tid=6, entry_tid=1' = 'object_number=37, tag_tid=5, entry_tid=12385' 2024-04-06T09:35:26.096 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=37, tag_tid=5, entry_tid=12385' '!=' 'object_number=36, tag_tid=5, entry_tid=12292' 2024-04-06T09:35:26.096 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:35:26.096 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=37, tag_tid=5, entry_tid=12385' = 'object_number=36, tag_tid=5, entry_tid=12292' 2024-04-06T09:35:26.096 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=37, tag_tid=5, entry_tid=12385' 2024-04-06T09:35:26.097 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=45, tag_tid=6, entry_tid=1' 2024-04-06T09:35:26.097 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:35:26.097 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:35:26.098 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=6 2024-04-06T09:35:26.098 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=37, tag_tid=5, entry_tid=12385' 2024-04-06T09:35:26.098 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:35:26.099 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:35:26.099 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=5 2024-04-06T09:35:26.101 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=45, tag_tid=6, entry_tid=1' 2024-04-06T09:35:26.102 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:35:26.102 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:35:26.102 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:35:26.102 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=37, tag_tid=5, entry_tid=12385' 2024-04-06T09:35:26.102 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:35:26.102 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:35:26.103 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=12385 2024-04-06T09:35:26.103 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 6 = 5 -a 1 -le 12385 2024-04-06T09:35:26.103 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:35:26.103 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:35:26.103 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:35:26.306 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:35:26.306 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:35:26.306 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:35:26.306 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:35:26.306 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:35:26.306 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:35:26.306 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:35:26.306 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:35:26.306 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:35:26.306 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:35:26.306 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:35:26.306 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:35:26.307 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:35:26.307 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:35:26.307 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:35:26.307 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:35:26.307 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:35:26.310 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:35:26.310 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:35:26.310 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:35:26.310 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:35:26.310 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:35:26.310 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:35:26.310 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:35:26.310 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:35:26.311 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:35:26.311 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:35:26.311 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:35:26.311 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:35:26.311 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:35:26.311 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:35:26.311 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:35:26.311 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:35:26.311 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:35:26.311 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:35:26.311 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:35:26.311 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:35:26.311 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:35:26.311 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:35:26.311 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:35:26.311 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:35:26.312 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:35:26.312 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:35:26.312 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:35:26.312 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:35:26.312 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:35:26.312 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:35:26.312 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:35:26.312 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:35:26.336 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:35:26.336 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:35:26.336 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:35:26.336 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:35:26.480 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:35:26.481 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:35:26.481 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:35:26.481 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:35:26.481 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:35:26.481 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:35:26.481 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:35:26.481 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:35:26.481 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:35:26.481 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:35:26.482 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:35:26.482 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:35:26.483 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:35:26.485 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:35:26.486 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:35:26.486 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:35:26.533 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 9 2024-04-06T09:35:26.533 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 11 2024-04-06T09:35:26.534 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:35:26.534 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=45, tag_tid=6, entry_tid=1], [object_number=44, tag_tid=6, entry_tid=0], [object_number=47, tag_tid=5, entry_tid=22415], [object_number=46, tag_tid=5, entry_tid=22414]]], state=connected] 2024-04-06T09:35:26.534 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=37, tag_tid=5, entry_tid=12481], [object_number=36, tag_tid=5, entry_tid=12480], [object_number=39, tag_tid=5, entry_tid=12479], [object_number=38, tag_tid=5, entry_tid=12478]]], state=connected] 2024-04-06T09:35:26.537 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:35:26.538 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=45, tag_tid=6, entry_tid=1' 2024-04-06T09:35:26.539 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:35:26.540 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:35:26.540 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:35:26.540 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:35:26.540 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:35:26.540 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:35:26.540 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:35:26.540 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:35:26.540 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:35:26.540 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:35:26.540 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:35:26.540 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:35:26.541 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:35:26.541 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:35:26.541 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:35:26.585 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 9 2024-04-06T09:35:26.585 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 11 2024-04-06T09:35:26.585 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:35:26.586 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=45, tag_tid=6, entry_tid=1], [object_number=44, tag_tid=6, entry_tid=0], [object_number=47, tag_tid=5, entry_tid=22415], [object_number=46, tag_tid=5, entry_tid=22414]]], state=connected] 2024-04-06T09:35:26.586 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=37, tag_tid=5, entry_tid=12481], [object_number=36, tag_tid=5, entry_tid=12480], [object_number=39, tag_tid=5, entry_tid=12479], [object_number=38, tag_tid=5, entry_tid=12478]]], state=connected] 2024-04-06T09:35:26.589 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:35:26.591 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=37, tag_tid=5, entry_tid=12481' 2024-04-06T09:35:26.591 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=45, tag_tid=6, entry_tid=1' -a 'object_number=45, tag_tid=6, entry_tid=1' = 'object_number=37, tag_tid=5, entry_tid=12481' 2024-04-06T09:35:26.591 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=37, tag_tid=5, entry_tid=12481' '!=' 'object_number=37, tag_tid=5, entry_tid=12385' 2024-04-06T09:35:26.591 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:35:26.591 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=37, tag_tid=5, entry_tid=12481' = 'object_number=37, tag_tid=5, entry_tid=12385' 2024-04-06T09:35:26.591 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=37, tag_tid=5, entry_tid=12481' 2024-04-06T09:35:26.592 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=45, tag_tid=6, entry_tid=1' 2024-04-06T09:35:26.592 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:35:26.592 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:35:26.592 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=6 2024-04-06T09:35:26.593 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=37, tag_tid=5, entry_tid=12481' 2024-04-06T09:35:26.593 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:35:26.593 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:35:26.594 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=5 2024-04-06T09:35:26.594 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=45, tag_tid=6, entry_tid=1' 2024-04-06T09:35:26.594 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:35:26.594 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:35:26.595 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:35:26.596 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=37, tag_tid=5, entry_tid=12481' 2024-04-06T09:35:26.596 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:35:26.596 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:35:26.597 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=12481 2024-04-06T09:35:26.597 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 6 = 5 -a 1 -le 12481 2024-04-06T09:35:26.597 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:35:26.597 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:35:26.597 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:35:26.799 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:35:26.799 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:35:26.799 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:35:26.799 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:35:26.799 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:35:26.799 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:35:26.799 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:35:26.799 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:35:26.799 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:35:26.800 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:35:26.800 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:35:26.800 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:35:26.800 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:35:26.800 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:35:26.800 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:35:26.800 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:35:26.801 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:35:26.805 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:35:26.805 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:35:26.805 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:35:26.805 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:35:26.805 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:35:26.805 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:35:26.805 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:35:26.805 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:35:26.805 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:35:26.805 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:35:26.805 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:35:26.805 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:35:26.805 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:35:26.806 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:35:26.806 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:35:26.806 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:35:26.806 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:35:26.806 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:35:26.806 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:35:26.806 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:35:26.806 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:35:26.806 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:35:26.806 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:35:26.806 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:35:26.806 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:35:26.806 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:35:26.806 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:35:26.807 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:35:26.807 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:35:26.807 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:35:26.807 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:35:26.807 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:35:26.830 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:35:26.831 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:35:26.831 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:35:26.831 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:35:26.962 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:35:26.962 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:35:26.963 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:35:26.963 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:35:26.963 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:35:26.963 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:35:26.963 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:35:26.963 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:35:26.963 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:35:26.963 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:35:26.964 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:35:26.964 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:35:26.964 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:35:26.966 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:35:26.966 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:35:26.967 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:35:27.011 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 9 2024-04-06T09:35:27.011 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 11 2024-04-06T09:35:27.011 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:35:27.011 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=45, tag_tid=6, entry_tid=1], [object_number=44, tag_tid=6, entry_tid=0], [object_number=47, tag_tid=5, entry_tid=22415], [object_number=46, tag_tid=5, entry_tid=22414]]], state=connected] 2024-04-06T09:35:27.011 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=37, tag_tid=5, entry_tid=12589], [object_number=36, tag_tid=5, entry_tid=12588], [object_number=39, tag_tid=5, entry_tid=12587], [object_number=38, tag_tid=5, entry_tid=12586]]], state=connected] 2024-04-06T09:35:27.015 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:35:27.016 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=45, tag_tid=6, entry_tid=1' 2024-04-06T09:35:27.017 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:35:27.017 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:35:27.017 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:35:27.017 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:35:27.017 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:35:27.017 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:35:27.017 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:35:27.018 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:35:27.018 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:35:27.018 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:35:27.018 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:35:27.018 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:35:27.019 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:35:27.019 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:35:27.019 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:35:27.064 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 9 2024-04-06T09:35:27.064 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 11 2024-04-06T09:35:27.064 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:35:27.064 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=45, tag_tid=6, entry_tid=1], [object_number=44, tag_tid=6, entry_tid=0], [object_number=47, tag_tid=5, entry_tid=22415], [object_number=46, tag_tid=5, entry_tid=22414]]], state=connected] 2024-04-06T09:35:27.065 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=37, tag_tid=5, entry_tid=12589], [object_number=36, tag_tid=5, entry_tid=12588], [object_number=39, tag_tid=5, entry_tid=12587], [object_number=38, tag_tid=5, entry_tid=12586]]], state=connected] 2024-04-06T09:35:27.068 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:35:27.070 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=37, tag_tid=5, entry_tid=12589' 2024-04-06T09:35:27.070 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=45, tag_tid=6, entry_tid=1' -a 'object_number=45, tag_tid=6, entry_tid=1' = 'object_number=37, tag_tid=5, entry_tid=12589' 2024-04-06T09:35:27.070 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=37, tag_tid=5, entry_tid=12589' '!=' 'object_number=37, tag_tid=5, entry_tid=12481' 2024-04-06T09:35:27.071 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:35:27.071 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=37, tag_tid=5, entry_tid=12589' = 'object_number=37, tag_tid=5, entry_tid=12481' 2024-04-06T09:35:27.071 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=37, tag_tid=5, entry_tid=12589' 2024-04-06T09:35:27.071 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=45, tag_tid=6, entry_tid=1' 2024-04-06T09:35:27.071 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:35:27.071 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:35:27.071 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=6 2024-04-06T09:35:27.072 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=37, tag_tid=5, entry_tid=12589' 2024-04-06T09:35:27.072 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:35:27.072 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:35:27.073 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=5 2024-04-06T09:35:27.074 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=45, tag_tid=6, entry_tid=1' 2024-04-06T09:35:27.074 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:35:27.074 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:35:27.075 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:35:27.075 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=37, tag_tid=5, entry_tid=12589' 2024-04-06T09:35:27.076 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:35:27.076 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:35:27.077 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=12589 2024-04-06T09:35:27.077 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 6 = 5 -a 1 -le 12589 2024-04-06T09:35:27.077 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:35:27.077 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:35:27.077 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:35:27.279 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:35:27.279 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:35:27.279 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:35:27.279 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:35:27.279 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:35:27.279 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:35:27.279 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:35:27.279 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:35:27.279 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:35:27.279 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:35:27.279 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:35:27.279 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:35:27.279 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:35:27.279 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:35:27.279 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:35:27.280 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:35:27.280 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:35:27.284 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:35:27.284 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:35:27.284 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:35:27.284 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:35:27.284 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:35:27.284 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:35:27.284 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:35:27.284 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:35:27.284 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:35:27.284 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:35:27.284 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:35:27.284 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:35:27.284 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:35:27.284 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:35:27.284 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:35:27.284 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:35:27.285 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:35:27.285 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:35:27.285 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:35:27.285 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:35:27.285 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:35:27.285 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:35:27.285 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:35:27.285 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:35:27.285 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:35:27.285 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:35:27.285 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:35:27.285 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:35:27.285 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:35:27.285 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:35:27.285 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:35:27.285 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:35:27.309 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:35:27.310 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:35:27.310 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:35:27.310 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:35:27.443 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:35:27.444 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:35:27.444 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:35:27.444 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:35:27.444 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:35:27.444 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:35:27.444 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:35:27.444 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:35:27.444 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:35:27.444 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:35:27.445 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:35:27.445 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:35:27.445 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:35:27.446 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:35:27.446 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:35:27.447 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:35:27.501 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 9 2024-04-06T09:35:27.501 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 11 2024-04-06T09:35:27.501 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:35:27.501 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=45, tag_tid=6, entry_tid=1], [object_number=44, tag_tid=6, entry_tid=0], [object_number=47, tag_tid=5, entry_tid=22415], [object_number=46, tag_tid=5, entry_tid=22414]]], state=connected] 2024-04-06T09:35:27.501 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=39, tag_tid=5, entry_tid=12679], [object_number=38, tag_tid=5, entry_tid=12678], [object_number=37, tag_tid=5, entry_tid=12677], [object_number=36, tag_tid=5, entry_tid=12676]]], state=connected] 2024-04-06T09:35:27.505 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:35:27.507 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=45, tag_tid=6, entry_tid=1' 2024-04-06T09:35:27.507 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:35:27.507 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:35:27.508 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:35:27.508 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:35:27.508 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:35:27.508 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:35:27.508 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:35:27.508 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:35:27.508 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:35:27.508 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:35:27.508 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:35:27.508 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:35:27.508 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:35:27.509 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:35:27.509 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:35:27.555 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 9 2024-04-06T09:35:27.555 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 11 2024-04-06T09:35:27.555 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:35:27.556 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=45, tag_tid=6, entry_tid=1], [object_number=44, tag_tid=6, entry_tid=0], [object_number=47, tag_tid=5, entry_tid=22415], [object_number=46, tag_tid=5, entry_tid=22414]]], state=connected] 2024-04-06T09:35:27.556 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=39, tag_tid=5, entry_tid=12679], [object_number=38, tag_tid=5, entry_tid=12678], [object_number=37, tag_tid=5, entry_tid=12677], [object_number=36, tag_tid=5, entry_tid=12676]]], state=connected] 2024-04-06T09:35:27.559 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:35:27.561 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=39, tag_tid=5, entry_tid=12679' 2024-04-06T09:35:27.561 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=45, tag_tid=6, entry_tid=1' -a 'object_number=45, tag_tid=6, entry_tid=1' = 'object_number=39, tag_tid=5, entry_tid=12679' 2024-04-06T09:35:27.562 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=39, tag_tid=5, entry_tid=12679' '!=' 'object_number=37, tag_tid=5, entry_tid=12589' 2024-04-06T09:35:27.562 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:35:27.562 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=39, tag_tid=5, entry_tid=12679' = 'object_number=37, tag_tid=5, entry_tid=12589' 2024-04-06T09:35:27.562 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=39, tag_tid=5, entry_tid=12679' 2024-04-06T09:35:27.562 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=45, tag_tid=6, entry_tid=1' 2024-04-06T09:35:27.562 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:35:27.562 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:35:27.563 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=6 2024-04-06T09:35:27.563 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=39, tag_tid=5, entry_tid=12679' 2024-04-06T09:35:27.563 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:35:27.564 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:35:27.564 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=5 2024-04-06T09:35:27.565 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=45, tag_tid=6, entry_tid=1' 2024-04-06T09:35:27.565 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:35:27.565 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:35:27.566 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:35:27.567 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=39, tag_tid=5, entry_tid=12679' 2024-04-06T09:35:27.567 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:35:27.567 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:35:27.568 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=12679 2024-04-06T09:35:27.568 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 6 = 5 -a 1 -le 12679 2024-04-06T09:35:27.568 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:35:27.568 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:35:27.568 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:35:27.769 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:35:27.769 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:35:27.770 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:35:27.770 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:35:27.770 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:35:27.770 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:35:27.770 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:35:27.770 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:35:27.770 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:35:27.770 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:35:27.770 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:35:27.770 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:35:27.770 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:35:27.770 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:35:27.770 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:35:27.771 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:35:27.771 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:35:27.773 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:35:27.773 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:35:27.773 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:35:27.773 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:35:27.773 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:35:27.773 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:35:27.773 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:35:27.773 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:35:27.773 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:35:27.773 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:35:27.773 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:35:27.773 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:35:27.773 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:35:27.773 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:35:27.774 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:35:27.774 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:35:27.775 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:35:27.775 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:35:27.775 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:35:27.775 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:35:27.775 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:35:27.775 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:35:27.775 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:35:27.775 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:35:27.775 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:35:27.776 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:35:27.776 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:35:27.776 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:35:27.776 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:35:27.776 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:35:27.776 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:35:27.776 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:35:27.799 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:35:27.800 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:35:27.800 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:35:27.800 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:35:27.933 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:35:27.934 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:35:27.934 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:35:27.934 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:35:27.934 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:35:27.934 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:35:27.934 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:35:27.935 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:35:27.935 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:35:27.935 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:35:27.935 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:35:27.935 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:35:27.935 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:35:27.937 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:35:27.937 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:35:27.937 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:35:27.990 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 9 2024-04-06T09:35:27.990 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 11 2024-04-06T09:35:27.990 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:35:27.990 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=45, tag_tid=6, entry_tid=1], [object_number=44, tag_tid=6, entry_tid=0], [object_number=47, tag_tid=5, entry_tid=22415], [object_number=46, tag_tid=5, entry_tid=22414]]], state=connected] 2024-04-06T09:35:27.990 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=38, tag_tid=5, entry_tid=12774], [object_number=37, tag_tid=5, entry_tid=12773], [object_number=36, tag_tid=5, entry_tid=12772], [object_number=39, tag_tid=5, entry_tid=12771]]], state=connected] 2024-04-06T09:35:27.994 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:35:27.995 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=45, tag_tid=6, entry_tid=1' 2024-04-06T09:35:27.996 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:35:27.996 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:35:27.996 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:35:27.996 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:35:27.996 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:35:27.996 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:35:27.996 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:35:27.997 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:35:27.997 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:35:27.997 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:35:27.997 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:35:27.997 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:35:27.997 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:35:27.998 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:35:27.998 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:35:28.043 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 9 2024-04-06T09:35:28.044 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 11 2024-04-06T09:35:28.044 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:35:28.044 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=45, tag_tid=6, entry_tid=1], [object_number=44, tag_tid=6, entry_tid=0], [object_number=47, tag_tid=5, entry_tid=22415], [object_number=46, tag_tid=5, entry_tid=22414]]], state=connected] 2024-04-06T09:35:28.044 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=38, tag_tid=5, entry_tid=12774], [object_number=37, tag_tid=5, entry_tid=12773], [object_number=36, tag_tid=5, entry_tid=12772], [object_number=39, tag_tid=5, entry_tid=12771]]], state=connected] 2024-04-06T09:35:28.047 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:35:28.049 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=38, tag_tid=5, entry_tid=12774' 2024-04-06T09:35:28.049 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=45, tag_tid=6, entry_tid=1' -a 'object_number=45, tag_tid=6, entry_tid=1' = 'object_number=38, tag_tid=5, entry_tid=12774' 2024-04-06T09:35:28.049 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=38, tag_tid=5, entry_tid=12774' '!=' 'object_number=39, tag_tid=5, entry_tid=12679' 2024-04-06T09:35:28.049 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:35:28.049 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=38, tag_tid=5, entry_tid=12774' = 'object_number=39, tag_tid=5, entry_tid=12679' 2024-04-06T09:35:28.049 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=38, tag_tid=5, entry_tid=12774' 2024-04-06T09:35:28.050 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=45, tag_tid=6, entry_tid=1' 2024-04-06T09:35:28.050 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:35:28.050 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:35:28.051 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=6 2024-04-06T09:35:28.051 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=38, tag_tid=5, entry_tid=12774' 2024-04-06T09:35:28.051 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:35:28.051 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:35:28.052 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=5 2024-04-06T09:35:28.053 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=45, tag_tid=6, entry_tid=1' 2024-04-06T09:35:28.053 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:35:28.053 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:35:28.054 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:35:28.055 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=38, tag_tid=5, entry_tid=12774' 2024-04-06T09:35:28.055 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:35:28.055 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:35:28.056 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=12774 2024-04-06T09:35:28.056 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 6 = 5 -a 1 -le 12774 2024-04-06T09:35:28.056 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:35:28.056 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:35:28.056 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:35:28.258 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:35:28.258 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:35:28.258 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:35:28.258 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:35:28.258 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:35:28.258 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:35:28.259 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:35:28.259 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:35:28.259 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:35:28.259 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:35:28.259 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:35:28.259 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:35:28.259 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:35:28.259 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:35:28.259 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:35:28.260 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:35:28.260 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:35:28.263 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:35:28.263 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:35:28.263 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:35:28.263 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:35:28.264 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:35:28.264 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:35:28.264 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:35:28.264 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:35:28.264 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:35:28.264 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:35:28.264 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:35:28.264 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:35:28.264 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:35:28.264 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:35:28.264 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:35:28.264 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:35:28.264 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:35:28.264 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:35:28.264 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:35:28.265 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:35:28.265 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:35:28.265 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:35:28.265 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:35:28.265 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:35:28.265 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:35:28.265 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:35:28.265 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:35:28.265 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:35:28.265 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:35:28.265 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:35:28.265 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:35:28.265 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:35:28.288 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:35:28.288 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:35:28.288 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:35:28.288 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:35:28.423 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:35:28.424 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:35:28.424 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:35:28.424 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:35:28.424 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:35:28.424 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:35:28.424 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:35:28.424 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:35:28.425 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:35:28.425 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:35:28.425 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:35:28.425 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:35:28.425 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:35:28.428 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:35:28.428 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:35:28.428 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:35:28.474 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 9 2024-04-06T09:35:28.474 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 11 2024-04-06T09:35:28.474 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:35:28.474 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=45, tag_tid=6, entry_tid=1], [object_number=44, tag_tid=6, entry_tid=0], [object_number=47, tag_tid=5, entry_tid=22415], [object_number=46, tag_tid=5, entry_tid=22414]]], state=connected] 2024-04-06T09:35:28.474 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=38, tag_tid=5, entry_tid=12854], [object_number=37, tag_tid=5, entry_tid=12853], [object_number=36, tag_tid=5, entry_tid=12852], [object_number=39, tag_tid=5, entry_tid=12851]]], state=connected] 2024-04-06T09:35:28.478 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:35:28.479 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=45, tag_tid=6, entry_tid=1' 2024-04-06T09:35:28.480 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:35:28.480 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:35:28.480 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:35:28.480 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:35:28.480 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:35:28.481 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:35:28.481 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:35:28.481 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:35:28.481 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:35:28.481 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:35:28.481 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:35:28.481 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:35:28.482 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:35:28.482 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:35:28.482 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:35:28.527 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 9 2024-04-06T09:35:28.527 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 11 2024-04-06T09:35:28.527 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:35:28.527 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=45, tag_tid=6, entry_tid=1], [object_number=44, tag_tid=6, entry_tid=0], [object_number=47, tag_tid=5, entry_tid=22415], [object_number=46, tag_tid=5, entry_tid=22414]]], state=connected] 2024-04-06T09:35:28.527 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=38, tag_tid=5, entry_tid=12854], [object_number=37, tag_tid=5, entry_tid=12853], [object_number=36, tag_tid=5, entry_tid=12852], [object_number=39, tag_tid=5, entry_tid=12851]]], state=connected] 2024-04-06T09:35:28.530 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:35:28.532 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=38, tag_tid=5, entry_tid=12854' 2024-04-06T09:35:28.532 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=45, tag_tid=6, entry_tid=1' -a 'object_number=45, tag_tid=6, entry_tid=1' = 'object_number=38, tag_tid=5, entry_tid=12854' 2024-04-06T09:35:28.532 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=38, tag_tid=5, entry_tid=12854' '!=' 'object_number=38, tag_tid=5, entry_tid=12774' 2024-04-06T09:35:28.532 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:35:28.532 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=38, tag_tid=5, entry_tid=12854' = 'object_number=38, tag_tid=5, entry_tid=12774' 2024-04-06T09:35:28.532 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=38, tag_tid=5, entry_tid=12854' 2024-04-06T09:35:28.533 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=45, tag_tid=6, entry_tid=1' 2024-04-06T09:35:28.533 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:35:28.533 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:35:28.534 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=6 2024-04-06T09:35:28.534 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=38, tag_tid=5, entry_tid=12854' 2024-04-06T09:35:28.534 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:35:28.534 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:35:28.535 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=5 2024-04-06T09:35:28.536 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=45, tag_tid=6, entry_tid=1' 2024-04-06T09:35:28.536 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:35:28.536 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:35:28.537 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:35:28.538 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=38, tag_tid=5, entry_tid=12854' 2024-04-06T09:35:28.538 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:35:28.538 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:35:28.539 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=12854 2024-04-06T09:35:28.539 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 6 = 5 -a 1 -le 12854 2024-04-06T09:35:28.539 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:35:28.539 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:35:28.539 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:35:28.741 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:35:28.741 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:35:28.741 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:35:28.741 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:35:28.741 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:35:28.741 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:35:28.741 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:35:28.741 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:35:28.741 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:35:28.741 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:35:28.742 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:35:28.742 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:35:28.742 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:35:28.742 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:35:28.742 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:35:28.742 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:35:28.743 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:35:28.744 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:35:28.745 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:35:28.745 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:35:28.745 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:35:28.745 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:35:28.745 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:35:28.745 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:35:28.745 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:35:28.745 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:35:28.745 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:35:28.745 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:35:28.745 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:35:28.745 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:35:28.745 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:35:28.745 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:35:28.746 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:35:28.746 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:35:28.747 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:35:28.747 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:35:28.747 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:35:28.747 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:35:28.747 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:35:28.747 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:35:28.747 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:35:28.747 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:35:28.747 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:35:28.747 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:35:28.747 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:35:28.747 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:35:28.747 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:35:28.747 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:35:28.747 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:35:28.771 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:35:28.772 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:35:28.772 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:35:28.772 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:35:28.903 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:35:28.904 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:35:28.904 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:35:28.904 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:35:28.904 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:35:28.904 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:35:28.904 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:35:28.904 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:35:28.904 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:35:28.904 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:35:28.905 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:35:28.905 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:35:28.905 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:35:28.907 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:35:28.907 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:35:28.908 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:35:28.970 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 9 2024-04-06T09:35:28.970 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 11 2024-04-06T09:35:28.970 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:35:28.970 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=45, tag_tid=6, entry_tid=1], [object_number=44, tag_tid=6, entry_tid=0], [object_number=47, tag_tid=5, entry_tid=22415], [object_number=46, tag_tid=5, entry_tid=22414]]], state=connected] 2024-04-06T09:35:28.970 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=36, tag_tid=5, entry_tid=12956], [object_number=39, tag_tid=5, entry_tid=12955], [object_number=38, tag_tid=5, entry_tid=12954], [object_number=37, tag_tid=5, entry_tid=12953]]], state=connected] 2024-04-06T09:35:28.973 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:35:28.975 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=45, tag_tid=6, entry_tid=1' 2024-04-06T09:35:28.976 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:35:28.976 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:35:28.976 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:35:28.976 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:35:28.976 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:35:28.976 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:35:28.976 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:35:28.976 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:35:28.977 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:35:28.977 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:35:28.977 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:35:28.977 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:35:28.977 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:35:28.978 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:35:28.978 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:35:29.023 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 9 2024-04-06T09:35:29.024 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 11 2024-04-06T09:35:29.024 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:35:29.024 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=45, tag_tid=6, entry_tid=1], [object_number=44, tag_tid=6, entry_tid=0], [object_number=47, tag_tid=5, entry_tid=22415], [object_number=46, tag_tid=5, entry_tid=22414]]], state=connected] 2024-04-06T09:35:29.024 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=36, tag_tid=5, entry_tid=12956], [object_number=39, tag_tid=5, entry_tid=12955], [object_number=38, tag_tid=5, entry_tid=12954], [object_number=37, tag_tid=5, entry_tid=12953]]], state=connected] 2024-04-06T09:35:29.027 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:35:29.029 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=36, tag_tid=5, entry_tid=12956' 2024-04-06T09:35:29.029 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=45, tag_tid=6, entry_tid=1' -a 'object_number=45, tag_tid=6, entry_tid=1' = 'object_number=36, tag_tid=5, entry_tid=12956' 2024-04-06T09:35:29.029 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=36, tag_tid=5, entry_tid=12956' '!=' 'object_number=38, tag_tid=5, entry_tid=12854' 2024-04-06T09:35:29.029 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:35:29.029 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=36, tag_tid=5, entry_tid=12956' = 'object_number=38, tag_tid=5, entry_tid=12854' 2024-04-06T09:35:29.029 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=36, tag_tid=5, entry_tid=12956' 2024-04-06T09:35:29.029 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=45, tag_tid=6, entry_tid=1' 2024-04-06T09:35:29.029 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:35:29.030 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:35:29.030 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=6 2024-04-06T09:35:29.031 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=36, tag_tid=5, entry_tid=12956' 2024-04-06T09:35:29.031 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:35:29.031 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:35:29.032 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=5 2024-04-06T09:35:29.032 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=45, tag_tid=6, entry_tid=1' 2024-04-06T09:35:29.032 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:35:29.033 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:35:29.033 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:35:29.034 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=36, tag_tid=5, entry_tid=12956' 2024-04-06T09:35:29.034 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:35:29.034 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:35:29.035 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=12956 2024-04-06T09:35:29.035 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 6 = 5 -a 1 -le 12956 2024-04-06T09:35:29.036 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:35:29.036 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:35:29.036 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:35:29.237 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:35:29.237 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:35:29.237 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:35:29.237 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:35:29.238 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:35:29.238 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:35:29.238 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:35:29.238 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:35:29.238 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:35:29.238 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:35:29.238 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:35:29.238 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:35:29.238 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:35:29.238 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:35:29.238 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:35:29.239 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:35:29.239 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:35:29.241 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:35:29.241 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:35:29.241 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:35:29.242 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:35:29.242 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:35:29.242 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:35:29.242 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:35:29.242 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:35:29.242 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:35:29.242 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:35:29.242 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:35:29.242 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:35:29.243 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:35:29.243 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:35:29.243 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:35:29.243 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:35:29.243 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:35:29.244 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:35:29.244 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:35:29.244 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:35:29.244 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:35:29.244 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:35:29.244 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:35:29.244 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:35:29.244 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:35:29.244 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:35:29.244 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:35:29.244 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:35:29.244 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:35:29.244 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:35:29.244 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:35:29.245 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:35:29.269 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:35:29.269 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:35:29.269 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:35:29.269 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:35:29.401 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:35:29.402 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:35:29.402 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:35:29.402 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:35:29.403 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:35:29.403 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:35:29.403 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:35:29.403 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:35:29.403 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:35:29.403 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:35:29.403 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:35:29.404 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:35:29.404 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:35:29.405 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:35:29.405 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:35:29.405 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:35:29.462 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 9 2024-04-06T09:35:29.462 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 11 2024-04-06T09:35:29.462 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:35:29.462 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=45, tag_tid=6, entry_tid=1], [object_number=44, tag_tid=6, entry_tid=0], [object_number=47, tag_tid=5, entry_tid=22415], [object_number=46, tag_tid=5, entry_tid=22414]]], state=connected] 2024-04-06T09:35:29.462 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=39, tag_tid=5, entry_tid=13063], [object_number=38, tag_tid=5, entry_tid=13062], [object_number=37, tag_tid=5, entry_tid=13061], [object_number=36, tag_tid=5, entry_tid=13060]]], state=connected] 2024-04-06T09:35:29.466 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:35:29.467 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=45, tag_tid=6, entry_tid=1' 2024-04-06T09:35:29.468 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:35:29.468 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:35:29.468 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:35:29.468 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:35:29.468 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:35:29.468 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:35:29.469 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:35:29.469 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:35:29.469 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:35:29.469 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:35:29.469 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:35:29.469 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:35:29.469 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:35:29.470 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:35:29.470 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:35:29.516 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 9 2024-04-06T09:35:29.516 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 11 2024-04-06T09:35:29.516 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:35:29.516 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=45, tag_tid=6, entry_tid=1], [object_number=44, tag_tid=6, entry_tid=0], [object_number=47, tag_tid=5, entry_tid=22415], [object_number=46, tag_tid=5, entry_tid=22414]]], state=connected] 2024-04-06T09:35:29.516 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=39, tag_tid=5, entry_tid=13063], [object_number=38, tag_tid=5, entry_tid=13062], [object_number=37, tag_tid=5, entry_tid=13061], [object_number=36, tag_tid=5, entry_tid=13060]]], state=connected] 2024-04-06T09:35:29.519 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:35:29.521 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=39, tag_tid=5, entry_tid=13063' 2024-04-06T09:35:29.521 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=45, tag_tid=6, entry_tid=1' -a 'object_number=45, tag_tid=6, entry_tid=1' = 'object_number=39, tag_tid=5, entry_tid=13063' 2024-04-06T09:35:29.521 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=39, tag_tid=5, entry_tid=13063' '!=' 'object_number=36, tag_tid=5, entry_tid=12956' 2024-04-06T09:35:29.521 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:35:29.521 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=39, tag_tid=5, entry_tid=13063' = 'object_number=36, tag_tid=5, entry_tid=12956' 2024-04-06T09:35:29.521 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=39, tag_tid=5, entry_tid=13063' 2024-04-06T09:35:29.522 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=45, tag_tid=6, entry_tid=1' 2024-04-06T09:35:29.522 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:35:29.522 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:35:29.523 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=6 2024-04-06T09:35:29.523 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=39, tag_tid=5, entry_tid=13063' 2024-04-06T09:35:29.523 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:35:29.524 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:35:29.524 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=5 2024-04-06T09:35:29.525 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=45, tag_tid=6, entry_tid=1' 2024-04-06T09:35:29.525 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:35:29.525 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:35:29.526 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:35:29.527 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=39, tag_tid=5, entry_tid=13063' 2024-04-06T09:35:29.527 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:35:29.527 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:35:29.528 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=13063 2024-04-06T09:35:29.528 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 6 = 5 -a 1 -le 13063 2024-04-06T09:35:29.528 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:35:29.529 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:35:29.529 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:35:29.730 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:35:29.730 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:35:29.730 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:35:29.730 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:35:29.730 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:35:29.731 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:35:29.731 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:35:29.731 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:35:29.731 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:35:29.731 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:35:29.731 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:35:29.731 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:35:29.731 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:35:29.731 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:35:29.731 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:35:29.732 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:35:29.732 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:35:29.734 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:35:29.734 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:35:29.734 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:35:29.734 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:35:29.734 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:35:29.734 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:35:29.734 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:35:29.734 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:35:29.734 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:35:29.735 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:35:29.735 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:35:29.735 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:35:29.735 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:35:29.735 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:35:29.735 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:35:29.735 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:35:29.736 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:35:29.736 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:35:29.736 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:35:29.736 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:35:29.736 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:35:29.736 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:35:29.736 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:35:29.737 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:35:29.737 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:35:29.737 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:35:29.737 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:35:29.737 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:35:29.737 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:35:29.737 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:35:29.737 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:35:29.737 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:35:29.761 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:35:29.761 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:35:29.761 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:35:29.761 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:35:29.890 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:35:29.890 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:35:29.891 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:35:29.891 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:35:29.891 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:35:29.891 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:35:29.891 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:35:29.891 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:35:29.891 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:35:29.891 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:35:29.892 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:35:29.892 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:35:29.892 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:35:29.894 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:35:29.895 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:35:29.895 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:35:29.939 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 9 2024-04-06T09:35:29.940 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 11 2024-04-06T09:35:29.940 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:35:29.940 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=45, tag_tid=6, entry_tid=1], [object_number=44, tag_tid=6, entry_tid=0], [object_number=47, tag_tid=5, entry_tid=22415], [object_number=46, tag_tid=5, entry_tid=22414]]], state=connected] 2024-04-06T09:35:29.940 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=39, tag_tid=5, entry_tid=13167], [object_number=38, tag_tid=5, entry_tid=13166], [object_number=37, tag_tid=5, entry_tid=13165], [object_number=36, tag_tid=5, entry_tid=13164]]], state=connected] 2024-04-06T09:35:29.943 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:35:29.944 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=45, tag_tid=6, entry_tid=1' 2024-04-06T09:35:29.946 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:35:29.946 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:35:29.946 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:35:29.946 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:35:29.946 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:35:29.946 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:35:29.946 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:35:29.946 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:35:29.946 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:35:29.946 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:35:29.946 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:35:29.946 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:35:29.947 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:35:29.947 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:35:29.947 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:35:29.992 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 9 2024-04-06T09:35:29.992 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 11 2024-04-06T09:35:29.993 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:35:29.993 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=45, tag_tid=6, entry_tid=1], [object_number=44, tag_tid=6, entry_tid=0], [object_number=47, tag_tid=5, entry_tid=22415], [object_number=46, tag_tid=5, entry_tid=22414]]], state=connected] 2024-04-06T09:35:29.993 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=39, tag_tid=5, entry_tid=13167], [object_number=38, tag_tid=5, entry_tid=13166], [object_number=37, tag_tid=5, entry_tid=13165], [object_number=36, tag_tid=5, entry_tid=13164]]], state=connected] 2024-04-06T09:35:29.996 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:35:29.998 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=39, tag_tid=5, entry_tid=13167' 2024-04-06T09:35:29.998 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=45, tag_tid=6, entry_tid=1' -a 'object_number=45, tag_tid=6, entry_tid=1' = 'object_number=39, tag_tid=5, entry_tid=13167' 2024-04-06T09:35:29.998 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=39, tag_tid=5, entry_tid=13167' '!=' 'object_number=39, tag_tid=5, entry_tid=13063' 2024-04-06T09:35:29.998 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:35:29.998 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=39, tag_tid=5, entry_tid=13167' = 'object_number=39, tag_tid=5, entry_tid=13063' 2024-04-06T09:35:29.998 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=39, tag_tid=5, entry_tid=13167' 2024-04-06T09:35:29.999 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=45, tag_tid=6, entry_tid=1' 2024-04-06T09:35:29.999 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:35:29.999 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:35:29.999 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=6 2024-04-06T09:35:30.000 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=39, tag_tid=5, entry_tid=13167' 2024-04-06T09:35:30.000 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:35:30.000 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:35:30.001 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=5 2024-04-06T09:35:30.001 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=45, tag_tid=6, entry_tid=1' 2024-04-06T09:35:30.002 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:35:30.002 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:35:30.003 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:35:30.003 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=39, tag_tid=5, entry_tid=13167' 2024-04-06T09:35:30.003 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:35:30.003 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:35:30.004 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=13167 2024-04-06T09:35:30.005 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 6 = 5 -a 1 -le 13167 2024-04-06T09:35:30.005 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:35:30.005 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:35:30.005 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:35:30.206 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:35:30.207 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:35:30.207 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:35:30.207 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:35:30.207 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:35:30.207 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:35:30.207 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:35:30.207 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:35:30.207 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:35:30.207 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:35:30.207 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:35:30.207 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:35:30.207 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:35:30.207 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:35:30.207 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:35:30.208 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:35:30.208 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:35:30.210 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:35:30.210 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:35:30.210 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:35:30.210 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:35:30.211 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:35:30.211 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:35:30.211 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:35:30.211 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:35:30.211 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:35:30.211 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:35:30.211 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:35:30.211 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:35:30.211 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:35:30.211 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:35:30.211 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:35:30.211 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:35:30.211 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:35:30.211 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:35:30.211 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:35:30.212 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:35:30.212 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:35:30.212 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:35:30.212 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:35:30.212 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:35:30.212 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:35:30.212 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:35:30.212 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:35:30.212 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:35:30.212 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:35:30.212 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:35:30.212 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:35:30.213 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:35:30.238 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:35:30.239 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:35:30.239 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:35:30.239 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:35:30.366 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:35:30.367 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:35:30.367 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:35:30.367 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:35:30.367 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:35:30.367 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:35:30.367 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:35:30.368 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:35:30.368 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:35:30.368 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:35:30.368 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:35:30.368 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:35:30.368 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:35:30.369 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:35:30.370 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:35:30.370 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:35:30.425 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 9 2024-04-06T09:35:30.425 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 11 2024-04-06T09:35:30.425 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:35:30.425 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=45, tag_tid=6, entry_tid=1], [object_number=44, tag_tid=6, entry_tid=0], [object_number=47, tag_tid=5, entry_tid=22415], [object_number=46, tag_tid=5, entry_tid=22414]]], state=connected] 2024-04-06T09:35:30.425 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=39, tag_tid=5, entry_tid=13275], [object_number=38, tag_tid=5, entry_tid=13274], [object_number=37, tag_tid=5, entry_tid=13273], [object_number=36, tag_tid=5, entry_tid=13272]]], state=connected] 2024-04-06T09:35:30.429 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:35:30.431 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=45, tag_tid=6, entry_tid=1' 2024-04-06T09:35:30.432 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:35:30.432 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:35:30.432 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:35:30.432 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:35:30.432 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:35:30.432 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:35:30.432 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:35:30.432 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:35:30.432 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:35:30.432 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:35:30.432 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:35:30.432 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:35:30.433 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:35:30.433 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:35:30.433 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:35:30.478 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 9 2024-04-06T09:35:30.479 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 11 2024-04-06T09:35:30.479 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:35:30.479 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=45, tag_tid=6, entry_tid=1], [object_number=44, tag_tid=6, entry_tid=0], [object_number=47, tag_tid=5, entry_tid=22415], [object_number=46, tag_tid=5, entry_tid=22414]]], state=connected] 2024-04-06T09:35:30.479 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=39, tag_tid=5, entry_tid=13275], [object_number=38, tag_tid=5, entry_tid=13274], [object_number=37, tag_tid=5, entry_tid=13273], [object_number=36, tag_tid=5, entry_tid=13272]]], state=connected] 2024-04-06T09:35:30.482 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:35:30.484 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=39, tag_tid=5, entry_tid=13275' 2024-04-06T09:35:30.484 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=45, tag_tid=6, entry_tid=1' -a 'object_number=45, tag_tid=6, entry_tid=1' = 'object_number=39, tag_tid=5, entry_tid=13275' 2024-04-06T09:35:30.484 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=39, tag_tid=5, entry_tid=13275' '!=' 'object_number=39, tag_tid=5, entry_tid=13167' 2024-04-06T09:35:30.484 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:35:30.484 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=39, tag_tid=5, entry_tid=13275' = 'object_number=39, tag_tid=5, entry_tid=13167' 2024-04-06T09:35:30.484 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=39, tag_tid=5, entry_tid=13275' 2024-04-06T09:35:30.485 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=45, tag_tid=6, entry_tid=1' 2024-04-06T09:35:30.485 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:35:30.485 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:35:30.485 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=6 2024-04-06T09:35:30.486 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=39, tag_tid=5, entry_tid=13275' 2024-04-06T09:35:30.486 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:35:30.486 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:35:30.487 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=5 2024-04-06T09:35:30.488 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=45, tag_tid=6, entry_tid=1' 2024-04-06T09:35:30.488 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:35:30.488 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:35:30.489 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:35:30.490 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=39, tag_tid=5, entry_tid=13275' 2024-04-06T09:35:30.490 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:35:30.490 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:35:30.491 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=13275 2024-04-06T09:35:30.491 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 6 = 5 -a 1 -le 13275 2024-04-06T09:35:30.491 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:35:30.491 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:35:30.491 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:35:30.693 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:35:30.693 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:35:30.693 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:35:30.693 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:35:30.693 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:35:30.693 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:35:30.693 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:35:30.693 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:35:30.693 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:35:30.693 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:35:30.694 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:35:30.694 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:35:30.694 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:35:30.694 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:35:30.694 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:35:30.694 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:35:30.694 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:35:30.696 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:35:30.696 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:35:30.696 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:35:30.696 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:35:30.696 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:35:30.696 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:35:30.696 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:35:30.696 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:35:30.696 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:35:30.697 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:35:30.697 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:35:30.697 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:35:30.697 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:35:30.697 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:35:30.697 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:35:30.697 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:35:30.698 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:35:30.698 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:35:30.699 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:35:30.699 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:35:30.699 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:35:30.699 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:35:30.699 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:35:30.699 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:35:30.699 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:35:30.699 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:35:30.699 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:35:30.699 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:35:30.699 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:35:30.699 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:35:30.699 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:35:30.699 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:35:30.722 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:35:30.722 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:35:30.723 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:35:30.723 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:35:30.862 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:35:30.863 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:35:30.863 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:35:30.863 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:35:30.863 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:35:30.863 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:35:30.863 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:35:30.863 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:35:30.863 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:35:30.863 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:35:30.864 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:35:30.864 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:35:30.864 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:35:30.865 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:35:30.865 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:35:30.865 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:35:30.910 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 9 2024-04-06T09:35:30.911 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 11 2024-04-06T09:35:30.911 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:35:30.911 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=45, tag_tid=6, entry_tid=1], [object_number=44, tag_tid=6, entry_tid=0], [object_number=47, tag_tid=5, entry_tid=22415], [object_number=46, tag_tid=5, entry_tid=22414]]], state=connected] 2024-04-06T09:35:30.911 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=37, tag_tid=5, entry_tid=13381], [object_number=36, tag_tid=5, entry_tid=13380], [object_number=39, tag_tid=5, entry_tid=13379], [object_number=38, tag_tid=5, entry_tid=13378]]], state=connected] 2024-04-06T09:35:30.914 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:35:30.915 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=45, tag_tid=6, entry_tid=1' 2024-04-06T09:35:30.916 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:35:30.917 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:35:30.917 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:35:30.917 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:35:30.917 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:35:30.917 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:35:30.917 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:35:30.917 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:35:30.917 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:35:30.917 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:35:30.917 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:35:30.917 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:35:30.917 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:35:30.918 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:35:30.918 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:35:30.962 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 9 2024-04-06T09:35:30.962 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 11 2024-04-06T09:35:30.962 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:35:30.963 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=45, tag_tid=6, entry_tid=1], [object_number=44, tag_tid=6, entry_tid=0], [object_number=47, tag_tid=5, entry_tid=22415], [object_number=46, tag_tid=5, entry_tid=22414]]], state=connected] 2024-04-06T09:35:30.963 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=37, tag_tid=5, entry_tid=13381], [object_number=36, tag_tid=5, entry_tid=13380], [object_number=39, tag_tid=5, entry_tid=13379], [object_number=38, tag_tid=5, entry_tid=13378]]], state=connected] 2024-04-06T09:35:30.966 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:35:30.968 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=37, tag_tid=5, entry_tid=13381' 2024-04-06T09:35:30.968 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=45, tag_tid=6, entry_tid=1' -a 'object_number=45, tag_tid=6, entry_tid=1' = 'object_number=37, tag_tid=5, entry_tid=13381' 2024-04-06T09:35:30.968 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=37, tag_tid=5, entry_tid=13381' '!=' 'object_number=39, tag_tid=5, entry_tid=13275' 2024-04-06T09:35:30.968 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:35:30.968 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=37, tag_tid=5, entry_tid=13381' = 'object_number=39, tag_tid=5, entry_tid=13275' 2024-04-06T09:35:30.968 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=37, tag_tid=5, entry_tid=13381' 2024-04-06T09:35:30.968 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=45, tag_tid=6, entry_tid=1' 2024-04-06T09:35:30.968 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:35:30.968 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:35:30.969 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=6 2024-04-06T09:35:30.970 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=37, tag_tid=5, entry_tid=13381' 2024-04-06T09:35:30.970 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:35:30.970 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:35:30.971 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=5 2024-04-06T09:35:30.971 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=45, tag_tid=6, entry_tid=1' 2024-04-06T09:35:30.972 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:35:30.972 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:35:30.972 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:35:30.973 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=37, tag_tid=5, entry_tid=13381' 2024-04-06T09:35:30.973 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:35:30.973 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:35:30.974 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=13381 2024-04-06T09:35:30.975 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 6 = 5 -a 1 -le 13381 2024-04-06T09:35:30.975 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:35:30.975 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:35:30.975 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:35:31.176 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:35:31.176 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:35:31.176 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:35:31.176 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:35:31.176 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:35:31.176 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:35:31.177 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:35:31.177 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:35:31.177 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:35:31.177 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:35:31.177 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:35:31.177 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:35:31.177 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:35:31.177 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:35:31.177 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:35:31.178 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:35:31.178 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:35:31.180 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:35:31.180 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:35:31.180 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:35:31.180 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:35:31.180 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:35:31.180 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:35:31.180 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:35:31.180 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:35:31.181 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:35:31.181 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:35:31.181 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:35:31.181 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:35:31.181 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:35:31.181 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:35:31.181 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:35:31.181 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:35:31.182 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:35:31.182 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:35:31.182 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:35:31.182 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:35:31.182 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:35:31.182 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:35:31.182 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:35:31.183 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:35:31.183 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:35:31.183 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:35:31.183 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:35:31.183 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:35:31.183 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:35:31.183 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:35:31.183 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:35:31.183 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:35:31.207 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:35:31.207 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:35:31.207 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:35:31.207 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:35:31.338 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:35:31.340 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:35:31.340 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:35:31.340 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:35:31.340 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:35:31.340 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:35:31.340 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:35:31.340 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:35:31.340 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:35:31.340 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:35:31.341 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:35:31.341 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:35:31.341 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:35:31.343 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:35:31.344 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:35:31.344 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:35:31.389 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 9 2024-04-06T09:35:31.389 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 11 2024-04-06T09:35:31.389 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:35:31.389 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=45, tag_tid=6, entry_tid=1], [object_number=44, tag_tid=6, entry_tid=0], [object_number=47, tag_tid=5, entry_tid=22415], [object_number=46, tag_tid=5, entry_tid=22414]]], state=connected] 2024-04-06T09:35:31.389 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=38, tag_tid=5, entry_tid=13482], [object_number=37, tag_tid=5, entry_tid=13481], [object_number=36, tag_tid=5, entry_tid=13480], [object_number=39, tag_tid=5, entry_tid=13479]]], state=connected] 2024-04-06T09:35:31.392 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:35:31.394 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=45, tag_tid=6, entry_tid=1' 2024-04-06T09:35:31.395 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:35:31.395 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:35:31.395 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:35:31.395 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:35:31.395 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:35:31.395 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:35:31.395 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:35:31.395 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:35:31.395 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:35:31.395 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:35:31.395 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:35:31.396 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:35:31.396 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:35:31.396 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:35:31.396 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:35:31.645 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 9 2024-04-06T09:35:31.645 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 11 2024-04-06T09:35:31.645 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:35:31.645 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=45, tag_tid=6, entry_tid=1], [object_number=44, tag_tid=6, entry_tid=0], [object_number=47, tag_tid=5, entry_tid=22415], [object_number=46, tag_tid=5, entry_tid=22414]]], state=connected] 2024-04-06T09:35:31.645 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=38, tag_tid=5, entry_tid=13482], [object_number=37, tag_tid=5, entry_tid=13481], [object_number=36, tag_tid=5, entry_tid=13480], [object_number=39, tag_tid=5, entry_tid=13479]]], state=connected] 2024-04-06T09:35:31.650 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:35:31.653 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=38, tag_tid=5, entry_tid=13482' 2024-04-06T09:35:31.653 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=45, tag_tid=6, entry_tid=1' -a 'object_number=45, tag_tid=6, entry_tid=1' = 'object_number=38, tag_tid=5, entry_tid=13482' 2024-04-06T09:35:31.654 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=38, tag_tid=5, entry_tid=13482' '!=' 'object_number=37, tag_tid=5, entry_tid=13381' 2024-04-06T09:35:31.654 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:35:31.654 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=38, tag_tid=5, entry_tid=13482' = 'object_number=37, tag_tid=5, entry_tid=13381' 2024-04-06T09:35:31.654 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=38, tag_tid=5, entry_tid=13482' 2024-04-06T09:35:31.654 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=45, tag_tid=6, entry_tid=1' 2024-04-06T09:35:31.655 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:35:31.655 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:35:31.656 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=6 2024-04-06T09:35:31.657 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=38, tag_tid=5, entry_tid=13482' 2024-04-06T09:35:31.657 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:35:31.657 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:35:31.658 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=5 2024-04-06T09:35:31.659 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=45, tag_tid=6, entry_tid=1' 2024-04-06T09:35:31.659 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:35:31.659 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:35:31.661 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:35:31.662 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=38, tag_tid=5, entry_tid=13482' 2024-04-06T09:35:31.662 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:35:31.662 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:35:31.663 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=13482 2024-04-06T09:35:31.664 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 6 = 5 -a 1 -le 13482 2024-04-06T09:35:31.664 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:35:31.664 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:35:31.664 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:35:31.865 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:35:31.866 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:35:31.866 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:35:31.866 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:35:31.866 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:35:31.866 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:35:31.866 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:35:31.866 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:35:31.866 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:35:31.866 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:35:31.866 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:35:31.866 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:35:31.866 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:35:31.866 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:35:31.866 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:35:31.867 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:35:31.867 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:35:31.869 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:35:31.869 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:35:31.870 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:35:31.870 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:35:31.870 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:35:31.870 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:35:31.870 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:35:31.870 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:35:31.870 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:35:31.870 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:35:31.870 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:35:31.870 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:35:31.870 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:35:31.870 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:35:31.870 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:35:31.871 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:35:31.872 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:35:31.872 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:35:31.872 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:35:31.872 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:35:31.872 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:35:31.872 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:35:31.872 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:35:31.872 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:35:31.872 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:35:31.872 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:35:31.873 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:35:31.873 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:35:31.873 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:35:31.873 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:35:31.873 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:35:31.873 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:35:31.897 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:35:31.898 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:35:31.898 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:35:31.898 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:35:32.025 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:35:32.027 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:35:32.027 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:35:32.027 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:35:32.027 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:35:32.027 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:35:32.027 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:35:32.027 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:35:32.027 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:35:32.027 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:35:32.027 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:35:32.027 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:35:32.027 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:35:32.029 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:35:32.030 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:35:32.030 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:35:32.074 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 9 2024-04-06T09:35:32.074 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 11 2024-04-06T09:35:32.074 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:35:32.074 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=45, tag_tid=6, entry_tid=1], [object_number=44, tag_tid=6, entry_tid=0], [object_number=47, tag_tid=5, entry_tid=22415], [object_number=46, tag_tid=5, entry_tid=22414]]], state=connected] 2024-04-06T09:35:32.074 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=38, tag_tid=5, entry_tid=13630], [object_number=37, tag_tid=5, entry_tid=13629], [object_number=36, tag_tid=5, entry_tid=13628], [object_number=39, tag_tid=5, entry_tid=13627]]], state=connected] 2024-04-06T09:35:32.078 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:35:32.079 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=45, tag_tid=6, entry_tid=1' 2024-04-06T09:35:32.080 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:35:32.080 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:35:32.080 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:35:32.080 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:35:32.081 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:35:32.081 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:35:32.081 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:35:32.081 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:35:32.081 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:35:32.081 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:35:32.081 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:35:32.081 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:35:32.082 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:35:32.082 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:35:32.082 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:35:32.126 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 9 2024-04-06T09:35:32.126 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 11 2024-04-06T09:35:32.126 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:35:32.126 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=45, tag_tid=6, entry_tid=1], [object_number=44, tag_tid=6, entry_tid=0], [object_number=47, tag_tid=5, entry_tid=22415], [object_number=46, tag_tid=5, entry_tid=22414]]], state=connected] 2024-04-06T09:35:32.126 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=38, tag_tid=5, entry_tid=13630], [object_number=37, tag_tid=5, entry_tid=13629], [object_number=36, tag_tid=5, entry_tid=13628], [object_number=39, tag_tid=5, entry_tid=13627]]], state=connected] 2024-04-06T09:35:32.129 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:35:32.132 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=38, tag_tid=5, entry_tid=13630' 2024-04-06T09:35:32.132 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=45, tag_tid=6, entry_tid=1' -a 'object_number=45, tag_tid=6, entry_tid=1' = 'object_number=38, tag_tid=5, entry_tid=13630' 2024-04-06T09:35:32.132 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=38, tag_tid=5, entry_tid=13630' '!=' 'object_number=38, tag_tid=5, entry_tid=13482' 2024-04-06T09:35:32.132 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:35:32.132 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=38, tag_tid=5, entry_tid=13630' = 'object_number=38, tag_tid=5, entry_tid=13482' 2024-04-06T09:35:32.132 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=38, tag_tid=5, entry_tid=13630' 2024-04-06T09:35:32.132 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=45, tag_tid=6, entry_tid=1' 2024-04-06T09:35:32.132 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:35:32.132 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:35:32.133 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=6 2024-04-06T09:35:32.133 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=38, tag_tid=5, entry_tid=13630' 2024-04-06T09:35:32.134 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:35:32.134 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:35:32.134 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=5 2024-04-06T09:35:32.135 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=45, tag_tid=6, entry_tid=1' 2024-04-06T09:35:32.135 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:35:32.135 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:35:32.136 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:35:32.137 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=38, tag_tid=5, entry_tid=13630' 2024-04-06T09:35:32.137 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:35:32.137 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:35:32.138 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=13630 2024-04-06T09:35:32.138 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 6 = 5 -a 1 -le 13630 2024-04-06T09:35:32.138 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:35:32.138 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:35:32.138 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:35:32.340 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:35:32.340 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:35:32.340 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:35:32.340 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:35:32.340 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:35:32.341 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:35:32.341 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:35:32.341 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:35:32.341 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:35:32.341 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:35:32.341 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:35:32.341 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:35:32.341 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:35:32.341 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:35:32.341 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:35:32.341 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:35:32.342 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:35:32.344 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:35:32.344 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:35:32.344 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:35:32.344 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:35:32.344 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:35:32.344 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:35:32.344 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:35:32.344 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:35:32.344 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:35:32.344 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:35:32.344 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:35:32.344 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:35:32.344 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:35:32.345 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:35:32.345 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:35:32.345 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:35:32.346 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:35:32.346 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:35:32.346 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:35:32.346 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:35:32.346 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:35:32.346 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:35:32.346 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:35:32.346 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:35:32.346 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:35:32.346 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:35:32.346 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:35:32.346 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:35:32.347 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:35:32.347 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:35:32.347 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:35:32.347 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:35:32.371 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:35:32.371 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:35:32.371 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:35:32.371 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:35:32.504 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:35:32.505 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:35:32.505 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:35:32.505 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:35:32.505 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:35:32.505 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:35:32.506 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:35:32.506 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:35:32.506 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:35:32.506 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:35:32.506 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:35:32.506 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:35:32.506 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:35:32.507 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:35:32.508 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:35:32.508 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:35:32.554 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 9 2024-04-06T09:35:32.554 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 11 2024-04-06T09:35:32.554 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:35:32.554 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=45, tag_tid=6, entry_tid=1], [object_number=44, tag_tid=6, entry_tid=0], [object_number=47, tag_tid=5, entry_tid=22415], [object_number=46, tag_tid=5, entry_tid=22414]]], state=connected] 2024-04-06T09:35:32.554 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=37, tag_tid=5, entry_tid=13729], [object_number=36, tag_tid=5, entry_tid=13728], [object_number=39, tag_tid=5, entry_tid=13727], [object_number=38, tag_tid=5, entry_tid=13726]]], state=connected] 2024-04-06T09:35:32.558 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:35:32.559 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=45, tag_tid=6, entry_tid=1' 2024-04-06T09:35:32.560 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:35:32.560 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:35:32.560 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:35:32.561 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:35:32.561 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:35:32.561 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:35:32.561 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:35:32.561 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:35:32.561 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:35:32.561 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:35:32.561 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:35:32.561 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:35:32.562 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:35:32.562 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:35:32.562 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:35:32.606 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 9 2024-04-06T09:35:32.606 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 11 2024-04-06T09:35:32.606 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:35:32.606 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=45, tag_tid=6, entry_tid=1], [object_number=44, tag_tid=6, entry_tid=0], [object_number=47, tag_tid=5, entry_tid=22415], [object_number=46, tag_tid=5, entry_tid=22414]]], state=connected] 2024-04-06T09:35:32.606 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=37, tag_tid=5, entry_tid=13729], [object_number=36, tag_tid=5, entry_tid=13728], [object_number=39, tag_tid=5, entry_tid=13727], [object_number=38, tag_tid=5, entry_tid=13726]]], state=connected] 2024-04-06T09:35:32.609 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:35:32.611 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=37, tag_tid=5, entry_tid=13729' 2024-04-06T09:35:32.611 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=45, tag_tid=6, entry_tid=1' -a 'object_number=45, tag_tid=6, entry_tid=1' = 'object_number=37, tag_tid=5, entry_tid=13729' 2024-04-06T09:35:32.611 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=37, tag_tid=5, entry_tid=13729' '!=' 'object_number=38, tag_tid=5, entry_tid=13630' 2024-04-06T09:35:32.611 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:35:32.612 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=37, tag_tid=5, entry_tid=13729' = 'object_number=38, tag_tid=5, entry_tid=13630' 2024-04-06T09:35:32.612 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=37, tag_tid=5, entry_tid=13729' 2024-04-06T09:35:32.612 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=45, tag_tid=6, entry_tid=1' 2024-04-06T09:35:32.612 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:35:32.612 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:35:32.613 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=6 2024-04-06T09:35:32.614 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=37, tag_tid=5, entry_tid=13729' 2024-04-06T09:35:32.614 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:35:32.614 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:35:32.615 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=5 2024-04-06T09:35:32.615 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=45, tag_tid=6, entry_tid=1' 2024-04-06T09:35:32.615 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:35:32.615 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:35:32.616 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:35:32.617 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=37, tag_tid=5, entry_tid=13729' 2024-04-06T09:35:32.617 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:35:32.617 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:35:32.618 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=13729 2024-04-06T09:35:32.618 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 6 = 5 -a 1 -le 13729 2024-04-06T09:35:32.618 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:35:32.618 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:35:32.619 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:35:32.820 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:35:32.820 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:35:32.821 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:35:32.821 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:35:32.821 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:35:32.821 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:35:32.821 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:35:32.821 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:35:32.821 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:35:32.821 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:35:32.821 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:35:32.821 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:35:32.821 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:35:32.821 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:35:32.821 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:35:32.822 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:35:32.822 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:35:32.824 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:35:32.824 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:35:32.824 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:35:32.824 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:35:32.824 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:35:32.824 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:35:32.824 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:35:32.825 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:35:32.825 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:35:32.825 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:35:32.825 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:35:32.825 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:35:32.825 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:35:32.825 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:35:32.825 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:35:32.825 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:35:32.826 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:35:32.826 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:35:32.826 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:35:32.826 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:35:32.827 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:35:32.827 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:35:32.827 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:35:32.827 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:35:32.827 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:35:32.827 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:35:32.827 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:35:32.827 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:35:32.827 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:35:32.827 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:35:32.827 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:35:32.827 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:35:32.851 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:35:32.851 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:35:32.851 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:35:32.851 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:35:32.983 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:35:32.984 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:35:32.984 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:35:32.984 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:35:32.984 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:35:32.984 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:35:32.984 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:35:32.984 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:35:32.984 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:35:32.984 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:35:32.985 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:35:32.985 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:35:32.985 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:35:32.987 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:35:32.987 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:35:32.988 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:35:33.248 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 9 2024-04-06T09:35:33.248 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 11 2024-04-06T09:35:33.249 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:35:33.249 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=45, tag_tid=6, entry_tid=1], [object_number=44, tag_tid=6, entry_tid=0], [object_number=47, tag_tid=5, entry_tid=22415], [object_number=46, tag_tid=5, entry_tid=22414]]], state=connected] 2024-04-06T09:35:33.249 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=38, tag_tid=5, entry_tid=13842], [object_number=37, tag_tid=5, entry_tid=13841], [object_number=36, tag_tid=5, entry_tid=13840], [object_number=39, tag_tid=5, entry_tid=13839]]], state=connected] 2024-04-06T09:35:33.252 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:35:33.254 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=45, tag_tid=6, entry_tid=1' 2024-04-06T09:35:33.254 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:35:33.255 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:35:33.255 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:35:33.255 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:35:33.255 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:35:33.258 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:35:33.258 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:35:33.258 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:35:33.258 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:35:33.259 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:35:33.259 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:35:33.259 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:35:33.259 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:35:33.259 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:35:33.259 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:35:33.305 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 9 2024-04-06T09:35:33.305 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 11 2024-04-06T09:35:33.305 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:35:33.305 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=45, tag_tid=6, entry_tid=1], [object_number=44, tag_tid=6, entry_tid=0], [object_number=47, tag_tid=5, entry_tid=22415], [object_number=46, tag_tid=5, entry_tid=22414]]], state=connected] 2024-04-06T09:35:33.305 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=38, tag_tid=5, entry_tid=13842], [object_number=37, tag_tid=5, entry_tid=13841], [object_number=36, tag_tid=5, entry_tid=13840], [object_number=39, tag_tid=5, entry_tid=13839]]], state=connected] 2024-04-06T09:35:33.309 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:35:33.311 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=38, tag_tid=5, entry_tid=13842' 2024-04-06T09:35:33.311 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=45, tag_tid=6, entry_tid=1' -a 'object_number=45, tag_tid=6, entry_tid=1' = 'object_number=38, tag_tid=5, entry_tid=13842' 2024-04-06T09:35:33.311 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=38, tag_tid=5, entry_tid=13842' '!=' 'object_number=37, tag_tid=5, entry_tid=13729' 2024-04-06T09:35:33.311 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:35:33.311 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=38, tag_tid=5, entry_tid=13842' = 'object_number=37, tag_tid=5, entry_tid=13729' 2024-04-06T09:35:33.312 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=38, tag_tid=5, entry_tid=13842' 2024-04-06T09:35:33.312 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=45, tag_tid=6, entry_tid=1' 2024-04-06T09:35:33.312 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:35:33.312 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:35:33.312 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=6 2024-04-06T09:35:33.313 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=38, tag_tid=5, entry_tid=13842' 2024-04-06T09:35:33.313 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:35:33.313 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:35:33.314 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=5 2024-04-06T09:35:33.315 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=45, tag_tid=6, entry_tid=1' 2024-04-06T09:35:33.315 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:35:33.315 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:35:33.316 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:35:33.316 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=38, tag_tid=5, entry_tid=13842' 2024-04-06T09:35:33.317 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:35:33.317 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:35:33.318 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=13842 2024-04-06T09:35:33.318 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 6 = 5 -a 1 -le 13842 2024-04-06T09:35:33.318 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:35:33.318 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:35:33.318 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:35:33.520 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:35:33.520 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:35:33.520 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:35:33.520 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:35:33.520 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:35:33.521 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:35:33.521 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:35:33.521 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:35:33.521 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:35:33.521 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:35:33.521 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:35:33.521 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:35:33.521 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:35:33.521 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:35:33.521 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:35:33.522 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:35:33.522 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:35:33.525 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:35:33.525 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:35:33.525 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:35:33.525 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:35:33.525 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:35:33.526 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:35:33.526 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:35:33.526 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:35:33.526 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:35:33.526 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:35:33.526 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:35:33.526 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:35:33.526 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:35:33.526 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:35:33.526 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:35:33.526 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:35:33.526 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:35:33.526 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:35:33.527 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:35:33.527 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:35:33.527 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:35:33.527 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:35:33.527 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:35:33.527 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:35:33.527 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:35:33.527 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:35:33.527 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:35:33.527 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:35:33.527 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:35:33.527 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:35:33.527 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:35:33.527 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:35:33.551 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:35:33.551 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:35:33.551 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:35:33.551 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:35:33.678 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:35:33.679 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:35:33.679 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:35:33.679 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:35:33.680 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:35:33.680 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:35:33.680 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:35:33.680 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:35:33.680 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:35:33.680 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:35:33.680 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:35:33.680 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:35:33.680 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:35:33.682 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:35:33.682 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:35:33.682 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:35:33.733 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 9 2024-04-06T09:35:33.733 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 11 2024-04-06T09:35:33.733 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:35:33.733 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=45, tag_tid=6, entry_tid=1], [object_number=44, tag_tid=6, entry_tid=0], [object_number=47, tag_tid=5, entry_tid=22415], [object_number=46, tag_tid=5, entry_tid=22414]]], state=connected] 2024-04-06T09:35:33.733 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=36, tag_tid=5, entry_tid=13980], [object_number=39, tag_tid=5, entry_tid=13979], [object_number=38, tag_tid=5, entry_tid=13978], [object_number=37, tag_tid=5, entry_tid=13977]]], state=connected] 2024-04-06T09:35:33.737 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:35:33.738 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=45, tag_tid=6, entry_tid=1' 2024-04-06T09:35:33.739 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:35:33.739 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:35:33.739 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:35:33.739 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:35:33.739 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:35:33.739 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:35:33.739 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:35:33.740 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:35:33.740 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:35:33.740 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:35:33.740 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:35:33.740 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:35:33.740 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:35:33.741 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:35:33.741 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:35:33.786 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 9 2024-04-06T09:35:33.786 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 11 2024-04-06T09:35:33.786 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:35:33.787 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=45, tag_tid=6, entry_tid=1], [object_number=44, tag_tid=6, entry_tid=0], [object_number=47, tag_tid=5, entry_tid=22415], [object_number=46, tag_tid=5, entry_tid=22414]]], state=connected] 2024-04-06T09:35:33.787 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=36, tag_tid=5, entry_tid=13980], [object_number=39, tag_tid=5, entry_tid=13979], [object_number=38, tag_tid=5, entry_tid=13978], [object_number=37, tag_tid=5, entry_tid=13977]]], state=connected] 2024-04-06T09:35:33.790 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:35:33.792 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=36, tag_tid=5, entry_tid=13980' 2024-04-06T09:35:33.792 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=45, tag_tid=6, entry_tid=1' -a 'object_number=45, tag_tid=6, entry_tid=1' = 'object_number=36, tag_tid=5, entry_tid=13980' 2024-04-06T09:35:33.792 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=36, tag_tid=5, entry_tid=13980' '!=' 'object_number=38, tag_tid=5, entry_tid=13842' 2024-04-06T09:35:33.792 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:35:33.792 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=36, tag_tid=5, entry_tid=13980' = 'object_number=38, tag_tid=5, entry_tid=13842' 2024-04-06T09:35:33.792 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=36, tag_tid=5, entry_tid=13980' 2024-04-06T09:35:33.792 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=45, tag_tid=6, entry_tid=1' 2024-04-06T09:35:33.792 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:35:33.792 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:35:33.793 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=6 2024-04-06T09:35:33.793 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=36, tag_tid=5, entry_tid=13980' 2024-04-06T09:35:33.793 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:35:33.794 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:35:33.794 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=5 2024-04-06T09:35:33.795 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=45, tag_tid=6, entry_tid=1' 2024-04-06T09:35:33.795 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:35:33.795 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:35:33.796 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:35:33.797 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=36, tag_tid=5, entry_tid=13980' 2024-04-06T09:35:33.797 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:35:33.797 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:35:33.798 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=13980 2024-04-06T09:35:33.798 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 6 = 5 -a 1 -le 13980 2024-04-06T09:35:33.798 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:35:33.798 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:35:33.798 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:35:34.000 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:35:34.000 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:35:34.000 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:35:34.000 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:35:34.000 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:35:34.000 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:35:34.000 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:35:34.000 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:35:34.000 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:35:34.000 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:35:34.000 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:35:34.000 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:35:34.000 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:35:34.001 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:35:34.001 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:35:34.001 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:35:34.001 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:35:34.005 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:35:34.005 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:35:34.006 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:35:34.006 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:35:34.006 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:35:34.006 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:35:34.006 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:35:34.006 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:35:34.006 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:35:34.006 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:35:34.006 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:35:34.006 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:35:34.006 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:35:34.006 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:35:34.006 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:35:34.007 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:35:34.007 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:35:34.007 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:35:34.007 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:35:34.007 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:35:34.007 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:35:34.007 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:35:34.007 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:35:34.007 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:35:34.007 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:35:34.007 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:35:34.007 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:35:34.007 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:35:34.007 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:35:34.007 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:35:34.007 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:35:34.008 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:35:34.030 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:35:34.030 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:35:34.030 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:35:34.030 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:35:34.161 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:35:34.163 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:35:34.163 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:35:34.163 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:35:34.163 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:35:34.163 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:35:34.163 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:35:34.163 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:35:34.164 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:35:34.164 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:35:34.164 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:35:34.164 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:35:34.164 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:35:34.166 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:35:34.166 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:35:34.167 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:35:34.217 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 9 2024-04-06T09:35:34.217 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 11 2024-04-06T09:35:34.217 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:35:34.218 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=45, tag_tid=6, entry_tid=1], [object_number=44, tag_tid=6, entry_tid=0], [object_number=47, tag_tid=5, entry_tid=22415], [object_number=46, tag_tid=5, entry_tid=22414]]], state=connected] 2024-04-06T09:35:34.218 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=39, tag_tid=5, entry_tid=14075], [object_number=38, tag_tid=5, entry_tid=14074], [object_number=37, tag_tid=5, entry_tid=14073], [object_number=36, tag_tid=5, entry_tid=14072]]], state=connected] 2024-04-06T09:35:34.221 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:35:34.223 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=45, tag_tid=6, entry_tid=1' 2024-04-06T09:35:34.223 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:35:34.223 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:35:34.224 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:35:34.224 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:35:34.224 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:35:34.224 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:35:34.224 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:35:34.224 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:35:34.224 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:35:34.224 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:35:34.224 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:35:34.224 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:35:34.225 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:35:34.225 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:35:34.225 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:35:34.271 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 9 2024-04-06T09:35:34.272 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 11 2024-04-06T09:35:34.272 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:35:34.272 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=45, tag_tid=6, entry_tid=1], [object_number=44, tag_tid=6, entry_tid=0], [object_number=47, tag_tid=5, entry_tid=22415], [object_number=46, tag_tid=5, entry_tid=22414]]], state=connected] 2024-04-06T09:35:34.272 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=39, tag_tid=5, entry_tid=14075], [object_number=38, tag_tid=5, entry_tid=14074], [object_number=37, tag_tid=5, entry_tid=14073], [object_number=36, tag_tid=5, entry_tid=14072]]], state=connected] 2024-04-06T09:35:34.275 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:35:34.277 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=39, tag_tid=5, entry_tid=14075' 2024-04-06T09:35:34.278 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=45, tag_tid=6, entry_tid=1' -a 'object_number=45, tag_tid=6, entry_tid=1' = 'object_number=39, tag_tid=5, entry_tid=14075' 2024-04-06T09:35:34.278 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=39, tag_tid=5, entry_tid=14075' '!=' 'object_number=36, tag_tid=5, entry_tid=13980' 2024-04-06T09:35:34.278 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:35:34.278 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=39, tag_tid=5, entry_tid=14075' = 'object_number=36, tag_tid=5, entry_tid=13980' 2024-04-06T09:35:34.278 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=39, tag_tid=5, entry_tid=14075' 2024-04-06T09:35:34.278 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=45, tag_tid=6, entry_tid=1' 2024-04-06T09:35:34.278 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:35:34.278 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:35:34.279 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=6 2024-04-06T09:35:34.279 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=39, tag_tid=5, entry_tid=14075' 2024-04-06T09:35:34.279 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:35:34.279 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:35:34.280 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=5 2024-04-06T09:35:34.281 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=45, tag_tid=6, entry_tid=1' 2024-04-06T09:35:34.281 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:35:34.281 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:35:34.282 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:35:34.283 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=39, tag_tid=5, entry_tid=14075' 2024-04-06T09:35:34.283 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:35:34.283 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:35:34.284 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=14075 2024-04-06T09:35:34.284 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 6 = 5 -a 1 -le 14075 2024-04-06T09:35:34.284 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:35:34.284 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:35:34.284 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:35:34.486 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:35:34.486 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:35:34.486 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:35:34.486 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:35:34.486 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:35:34.487 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:35:34.487 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:35:34.487 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:35:34.487 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:35:34.487 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:35:34.487 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:35:34.487 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:35:34.487 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:35:34.487 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:35:34.487 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:35:34.488 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:35:34.488 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:35:34.492 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:35:34.492 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:35:34.492 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:35:34.492 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:35:34.492 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:35:34.492 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:35:34.492 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:35:34.492 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:35:34.492 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:35:34.492 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:35:34.492 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:35:34.492 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:35:34.492 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:35:34.492 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:35:34.492 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:35:34.493 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:35:34.493 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:35:34.493 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:35:34.493 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:35:34.493 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:35:34.493 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:35:34.493 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:35:34.493 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:35:34.493 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:35:34.493 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:35:34.493 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:35:34.493 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:35:34.493 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:35:34.493 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:35:34.493 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:35:34.493 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:35:34.494 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:35:34.518 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:35:34.518 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:35:34.518 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:35:34.518 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:35:34.653 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:35:34.654 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:35:34.654 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:35:34.655 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:35:34.655 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:35:34.655 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:35:34.655 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:35:34.655 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:35:34.655 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:35:34.655 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:35:34.655 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:35:34.655 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:35:34.655 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:35:34.656 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:35:34.657 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:35:34.657 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:35:34.702 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 9 2024-04-06T09:35:34.702 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 11 2024-04-06T09:35:34.702 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:35:34.702 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=45, tag_tid=6, entry_tid=1], [object_number=44, tag_tid=6, entry_tid=0], [object_number=47, tag_tid=5, entry_tid=22415], [object_number=46, tag_tid=5, entry_tid=22414]]], state=connected] 2024-04-06T09:35:34.702 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=37, tag_tid=5, entry_tid=14169], [object_number=36, tag_tid=5, entry_tid=14168], [object_number=39, tag_tid=5, entry_tid=14167], [object_number=38, tag_tid=5, entry_tid=14166]]], state=connected] 2024-04-06T09:35:34.706 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:35:34.707 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=45, tag_tid=6, entry_tid=1' 2024-04-06T09:35:34.708 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:35:34.708 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:35:34.708 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:35:34.708 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:35:34.708 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:35:34.709 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:35:34.709 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:35:34.709 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:35:34.709 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:35:34.709 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:35:34.709 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:35:34.709 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:35:34.709 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:35:34.710 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:35:34.710 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:35:34.754 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 9 2024-04-06T09:35:34.754 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 11 2024-04-06T09:35:34.754 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:35:34.755 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=45, tag_tid=6, entry_tid=1], [object_number=44, tag_tid=6, entry_tid=0], [object_number=47, tag_tid=5, entry_tid=22415], [object_number=46, tag_tid=5, entry_tid=22414]]], state=connected] 2024-04-06T09:35:34.755 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=37, tag_tid=5, entry_tid=14169], [object_number=36, tag_tid=5, entry_tid=14168], [object_number=39, tag_tid=5, entry_tid=14167], [object_number=38, tag_tid=5, entry_tid=14166]]], state=connected] 2024-04-06T09:35:34.758 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:35:34.760 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=37, tag_tid=5, entry_tid=14169' 2024-04-06T09:35:34.760 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=45, tag_tid=6, entry_tid=1' -a 'object_number=45, tag_tid=6, entry_tid=1' = 'object_number=37, tag_tid=5, entry_tid=14169' 2024-04-06T09:35:34.760 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=37, tag_tid=5, entry_tid=14169' '!=' 'object_number=39, tag_tid=5, entry_tid=14075' 2024-04-06T09:35:34.760 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:35:34.760 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=37, tag_tid=5, entry_tid=14169' = 'object_number=39, tag_tid=5, entry_tid=14075' 2024-04-06T09:35:34.760 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=37, tag_tid=5, entry_tid=14169' 2024-04-06T09:35:34.760 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=45, tag_tid=6, entry_tid=1' 2024-04-06T09:35:34.760 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:35:34.760 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:35:34.761 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=6 2024-04-06T09:35:34.762 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=37, tag_tid=5, entry_tid=14169' 2024-04-06T09:35:34.762 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:35:34.762 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:35:34.763 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=5 2024-04-06T09:35:34.763 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=45, tag_tid=6, entry_tid=1' 2024-04-06T09:35:34.763 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:35:34.764 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:35:34.764 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:35:34.765 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=37, tag_tid=5, entry_tid=14169' 2024-04-06T09:35:34.765 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:35:34.765 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:35:34.766 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=14169 2024-04-06T09:35:34.766 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 6 = 5 -a 1 -le 14169 2024-04-06T09:35:34.766 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:35:34.767 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:35:34.767 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:35:34.968 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:35:34.968 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:35:34.968 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:35:34.969 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:35:34.969 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:35:34.969 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:35:34.969 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:35:34.969 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:35:34.969 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:35:34.969 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:35:34.969 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:35:34.969 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:35:34.969 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:35:34.969 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:35:34.969 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:35:34.970 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:35:34.970 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:35:34.973 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:35:34.973 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:35:34.973 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:35:34.973 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:35:34.973 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:35:34.973 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:35:34.973 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:35:34.974 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:35:34.974 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:35:34.974 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:35:34.974 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:35:34.974 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:35:34.974 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:35:34.974 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:35:34.974 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:35:34.974 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:35:34.974 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:35:34.974 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:35:34.974 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:35:34.974 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:35:34.975 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:35:34.975 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:35:34.975 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:35:34.975 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:35:34.975 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:35:34.975 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:35:34.975 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:35:34.975 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:35:34.975 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:35:34.975 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:35:34.975 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:35:34.975 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:35:35.001 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:35:35.001 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:35:35.001 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:35:35.001 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:35:35.132 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:35:35.134 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:35:35.134 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:35:35.134 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:35:35.134 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:35:35.134 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:35:35.134 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:35:35.134 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:35:35.134 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:35:35.134 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:35:35.135 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:35:35.135 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:35:35.135 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:35:35.137 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:35:35.138 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:35:35.138 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:35:35.192 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 9 2024-04-06T09:35:35.192 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 11 2024-04-06T09:35:35.193 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:35:35.193 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=45, tag_tid=6, entry_tid=1], [object_number=44, tag_tid=6, entry_tid=0], [object_number=47, tag_tid=5, entry_tid=22415], [object_number=46, tag_tid=5, entry_tid=22414]]], state=connected] 2024-04-06T09:35:35.193 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=37, tag_tid=5, entry_tid=14269], [object_number=36, tag_tid=5, entry_tid=14268], [object_number=39, tag_tid=5, entry_tid=14267], [object_number=38, tag_tid=5, entry_tid=14266]]], state=connected] 2024-04-06T09:35:35.196 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:35:35.198 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=45, tag_tid=6, entry_tid=1' 2024-04-06T09:35:35.199 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:35:35.199 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:35:35.199 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:35:35.199 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:35:35.199 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:35:35.199 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:35:35.199 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:35:35.199 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:35:35.199 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:35:35.199 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:35:35.200 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:35:35.200 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:35:35.200 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:35:35.200 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:35:35.200 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:35:35.246 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 9 2024-04-06T09:35:35.247 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 11 2024-04-06T09:35:35.247 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:35:35.247 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=45, tag_tid=6, entry_tid=1], [object_number=44, tag_tid=6, entry_tid=0], [object_number=47, tag_tid=5, entry_tid=22415], [object_number=46, tag_tid=5, entry_tid=22414]]], state=connected] 2024-04-06T09:35:35.247 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=37, tag_tid=5, entry_tid=14269], [object_number=36, tag_tid=5, entry_tid=14268], [object_number=39, tag_tid=5, entry_tid=14267], [object_number=38, tag_tid=5, entry_tid=14266]]], state=connected] 2024-04-06T09:35:35.250 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:35:35.252 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=37, tag_tid=5, entry_tid=14269' 2024-04-06T09:35:35.252 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=45, tag_tid=6, entry_tid=1' -a 'object_number=45, tag_tid=6, entry_tid=1' = 'object_number=37, tag_tid=5, entry_tid=14269' 2024-04-06T09:35:35.252 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=37, tag_tid=5, entry_tid=14269' '!=' 'object_number=37, tag_tid=5, entry_tid=14169' 2024-04-06T09:35:35.252 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:35:35.252 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=37, tag_tid=5, entry_tid=14269' = 'object_number=37, tag_tid=5, entry_tid=14169' 2024-04-06T09:35:35.252 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=37, tag_tid=5, entry_tid=14269' 2024-04-06T09:35:35.253 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=45, tag_tid=6, entry_tid=1' 2024-04-06T09:35:35.253 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:35:35.253 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:35:35.254 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=6 2024-04-06T09:35:35.254 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=37, tag_tid=5, entry_tid=14269' 2024-04-06T09:35:35.254 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:35:35.254 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:35:35.256 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=5 2024-04-06T09:35:35.256 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=45, tag_tid=6, entry_tid=1' 2024-04-06T09:35:35.256 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:35:35.256 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:35:35.257 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:35:35.258 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=37, tag_tid=5, entry_tid=14269' 2024-04-06T09:35:35.258 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:35:35.258 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:35:35.260 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=14269 2024-04-06T09:35:35.260 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 6 = 5 -a 1 -le 14269 2024-04-06T09:35:35.260 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:35:35.260 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:35:35.260 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:35:35.461 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:35:35.461 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:35:35.461 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:35:35.461 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:35:35.461 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:35:35.462 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:35:35.462 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:35:35.462 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:35:35.462 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:35:35.462 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:35:35.462 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:35:35.462 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:35:35.462 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:35:35.462 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:35:35.462 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:35:35.463 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:35:35.463 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:35:35.464 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:35:35.464 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:35:35.464 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:35:35.464 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:35:35.465 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:35:35.465 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:35:35.465 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:35:35.465 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:35:35.465 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:35:35.465 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:35:35.465 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:35:35.465 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:35:35.465 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:35:35.465 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:35:35.465 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:35:35.465 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:35:35.467 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:35:35.467 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:35:35.467 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:35:35.467 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:35:35.467 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:35:35.467 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:35:35.467 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:35:35.467 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:35:35.467 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:35:35.467 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:35:35.467 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:35:35.467 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:35:35.468 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:35:35.468 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:35:35.468 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:35:35.468 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:35:35.493 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:35:35.494 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:35:35.494 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:35:35.494 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:35:35.637 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:35:35.639 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:35:35.639 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:35:35.639 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:35:35.639 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:35:35.639 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:35:35.639 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:35:35.639 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:35:35.639 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:35:35.639 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:35:35.639 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:35:35.640 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:35:35.640 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:35:35.640 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:35:35.640 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:35:35.640 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:35:35.685 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 9 2024-04-06T09:35:35.685 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 11 2024-04-06T09:35:35.685 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:35:35.686 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=45, tag_tid=6, entry_tid=1], [object_number=44, tag_tid=6, entry_tid=0], [object_number=47, tag_tid=5, entry_tid=22415], [object_number=46, tag_tid=5, entry_tid=22414]]], state=connected] 2024-04-06T09:35:35.686 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=37, tag_tid=5, entry_tid=14369], [object_number=36, tag_tid=5, entry_tid=14368], [object_number=39, tag_tid=5, entry_tid=14367], [object_number=38, tag_tid=5, entry_tid=14366]]], state=connected] 2024-04-06T09:35:35.689 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:35:35.690 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=45, tag_tid=6, entry_tid=1' 2024-04-06T09:35:35.691 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:35:35.691 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:35:35.691 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:35:35.692 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:35:35.692 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:35:35.692 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:35:35.692 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:35:35.692 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:35:35.692 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:35:35.692 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:35:35.692 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:35:35.692 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:35:35.693 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:35:35.693 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:35:35.693 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:35:35.744 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 9 2024-04-06T09:35:35.744 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 11 2024-04-06T09:35:35.744 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:35:35.744 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=45, tag_tid=6, entry_tid=1], [object_number=44, tag_tid=6, entry_tid=0], [object_number=47, tag_tid=5, entry_tid=22415], [object_number=46, tag_tid=5, entry_tid=22414]]], state=connected] 2024-04-06T09:35:35.744 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=37, tag_tid=5, entry_tid=14369], [object_number=36, tag_tid=5, entry_tid=14368], [object_number=39, tag_tid=5, entry_tid=14367], [object_number=38, tag_tid=5, entry_tid=14366]]], state=connected] 2024-04-06T09:35:35.748 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:35:35.750 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=37, tag_tid=5, entry_tid=14369' 2024-04-06T09:35:35.750 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=45, tag_tid=6, entry_tid=1' -a 'object_number=45, tag_tid=6, entry_tid=1' = 'object_number=37, tag_tid=5, entry_tid=14369' 2024-04-06T09:35:35.750 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=37, tag_tid=5, entry_tid=14369' '!=' 'object_number=37, tag_tid=5, entry_tid=14269' 2024-04-06T09:35:35.750 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:35:35.750 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=37, tag_tid=5, entry_tid=14369' = 'object_number=37, tag_tid=5, entry_tid=14269' 2024-04-06T09:35:35.750 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=37, tag_tid=5, entry_tid=14369' 2024-04-06T09:35:35.751 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=45, tag_tid=6, entry_tid=1' 2024-04-06T09:35:35.751 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:35:35.751 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:35:35.751 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=6 2024-04-06T09:35:35.752 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=37, tag_tid=5, entry_tid=14369' 2024-04-06T09:35:35.752 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:35:35.752 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:35:35.753 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=5 2024-04-06T09:35:35.754 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=45, tag_tid=6, entry_tid=1' 2024-04-06T09:35:35.754 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:35:35.754 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:35:35.755 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:35:35.755 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=37, tag_tid=5, entry_tid=14369' 2024-04-06T09:35:35.755 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:35:35.755 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:35:35.757 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=14369 2024-04-06T09:35:35.757 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 6 = 5 -a 1 -le 14369 2024-04-06T09:35:35.757 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:35:35.757 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:35:35.757 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:35:35.958 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:35:35.958 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:35:35.959 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:35:35.959 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:35:35.959 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:35:35.959 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:35:35.959 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:35:35.959 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:35:35.959 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:35:35.959 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:35:35.959 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:35:35.959 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:35:35.959 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:35:35.959 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:35:35.959 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:35:35.960 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:35:35.960 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:35:35.963 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:35:35.963 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:35:35.964 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:35:35.964 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:35:35.964 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:35:35.964 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:35:35.964 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:35:35.964 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:35:35.964 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:35:35.964 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:35:35.964 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:35:35.964 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:35:35.965 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:35:35.965 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:35:35.965 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:35:35.965 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:35:35.965 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:35:35.965 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:35:35.965 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:35:35.965 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:35:35.965 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:35:35.965 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:35:35.965 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:35:35.965 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:35:35.965 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:35:35.965 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:35:35.966 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:35:35.966 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:35:35.966 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:35:35.966 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:35:35.966 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:35:35.966 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:35:35.987 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:35:35.988 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:35:35.988 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:35:35.988 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:35:36.122 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:35:36.123 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:35:36.123 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:35:36.123 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:35:36.124 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:35:36.124 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:35:36.124 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:35:36.124 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:35:36.124 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:35:36.124 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:35:36.124 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:35:36.124 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:35:36.124 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:35:36.125 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:35:36.126 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:35:36.126 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:35:36.196 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 9 2024-04-06T09:35:36.196 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 11 2024-04-06T09:35:36.196 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:35:36.196 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=45, tag_tid=6, entry_tid=1], [object_number=44, tag_tid=6, entry_tid=0], [object_number=47, tag_tid=5, entry_tid=22415], [object_number=46, tag_tid=5, entry_tid=22414]]], state=connected] 2024-04-06T09:35:36.196 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=41, tag_tid=5, entry_tid=14413], [object_number=40, tag_tid=5, entry_tid=14412], [object_number=43, tag_tid=5, entry_tid=14411], [object_number=42, tag_tid=5, entry_tid=14410]]], state=connected] 2024-04-06T09:35:36.200 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:35:36.201 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=45, tag_tid=6, entry_tid=1' 2024-04-06T09:35:36.203 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:35:36.203 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:35:36.203 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:35:36.203 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:35:36.203 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:35:36.203 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:35:36.203 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:35:36.203 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:35:36.203 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:35:36.203 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:35:36.203 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:35:36.203 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:35:36.204 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:35:36.204 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:35:36.204 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:35:36.250 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 10 2024-04-06T09:35:36.250 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 11 2024-04-06T09:35:36.250 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:35:36.250 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=45, tag_tid=6, entry_tid=1], [object_number=44, tag_tid=6, entry_tid=0], [object_number=47, tag_tid=5, entry_tid=22415], [object_number=46, tag_tid=5, entry_tid=22414]]], state=connected] 2024-04-06T09:35:36.250 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=41, tag_tid=5, entry_tid=14413], [object_number=40, tag_tid=5, entry_tid=14412], [object_number=43, tag_tid=5, entry_tid=14411], [object_number=42, tag_tid=5, entry_tid=14410]]], state=connected] 2024-04-06T09:35:36.253 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:35:36.255 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=41, tag_tid=5, entry_tid=14413' 2024-04-06T09:35:36.255 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=45, tag_tid=6, entry_tid=1' -a 'object_number=45, tag_tid=6, entry_tid=1' = 'object_number=41, tag_tid=5, entry_tid=14413' 2024-04-06T09:35:36.255 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=41, tag_tid=5, entry_tid=14413' '!=' 'object_number=37, tag_tid=5, entry_tid=14369' 2024-04-06T09:35:36.256 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:35:36.256 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=41, tag_tid=5, entry_tid=14413' = 'object_number=37, tag_tid=5, entry_tid=14369' 2024-04-06T09:35:36.256 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=41, tag_tid=5, entry_tid=14413' 2024-04-06T09:35:36.256 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=45, tag_tid=6, entry_tid=1' 2024-04-06T09:35:36.256 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:35:36.256 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:35:36.256 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=6 2024-04-06T09:35:36.257 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=41, tag_tid=5, entry_tid=14413' 2024-04-06T09:35:36.257 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:35:36.257 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:35:36.258 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=5 2024-04-06T09:35:36.258 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=45, tag_tid=6, entry_tid=1' 2024-04-06T09:35:36.259 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:35:36.259 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:35:36.260 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:35:36.260 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=41, tag_tid=5, entry_tid=14413' 2024-04-06T09:35:36.260 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:35:36.261 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:35:36.261 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=14413 2024-04-06T09:35:36.261 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 6 = 5 -a 1 -le 14413 2024-04-06T09:35:36.262 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:35:36.262 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:35:36.262 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:35:36.463 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:35:36.463 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:35:36.463 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:35:36.464 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:35:36.464 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:35:36.464 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:35:36.464 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:35:36.464 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:35:36.464 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:35:36.464 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:35:36.464 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:35:36.464 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:35:36.464 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:35:36.464 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:35:36.464 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:35:36.465 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:35:36.465 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:35:36.468 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:35:36.468 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:35:36.468 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:35:36.468 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:35:36.468 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:35:36.468 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:35:36.468 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:35:36.468 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:35:36.468 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:35:36.468 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:35:36.469 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:35:36.469 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:35:36.469 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:35:36.469 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:35:36.469 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:35:36.469 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:35:36.469 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:35:36.469 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:35:36.469 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:35:36.469 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:35:36.469 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:35:36.469 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:35:36.469 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:35:36.469 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:35:36.469 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:35:36.469 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:35:36.469 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:35:36.470 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:35:36.470 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:35:36.470 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:35:36.470 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:35:36.470 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:35:36.495 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:35:36.495 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:35:36.496 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:35:36.496 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:35:36.620 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:35:36.621 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:35:36.621 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:35:36.621 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:35:36.621 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:35:36.621 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:35:36.622 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:35:36.622 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:35:36.622 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:35:36.622 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:35:36.622 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:35:36.622 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:35:36.622 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:35:36.623 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:35:36.623 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:35:36.623 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:35:36.885 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 10 2024-04-06T09:35:36.885 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 11 2024-04-06T09:35:36.885 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:35:36.885 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=45, tag_tid=6, entry_tid=1], [object_number=44, tag_tid=6, entry_tid=0], [object_number=47, tag_tid=5, entry_tid=22415], [object_number=46, tag_tid=5, entry_tid=22414]]], state=connected] 2024-04-06T09:35:36.885 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=40, tag_tid=5, entry_tid=14512], [object_number=43, tag_tid=5, entry_tid=14511], [object_number=42, tag_tid=5, entry_tid=14510], [object_number=41, tag_tid=5, entry_tid=14509]]], state=connected] 2024-04-06T09:35:36.890 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:35:36.892 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=45, tag_tid=6, entry_tid=1' 2024-04-06T09:35:36.893 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:35:36.893 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:35:36.893 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:35:36.893 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:35:36.893 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:35:36.894 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:35:36.894 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:35:36.894 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:35:36.894 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:35:36.894 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:35:36.894 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:35:36.894 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:35:36.895 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:35:36.895 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:35:36.895 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:35:36.947 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 10 2024-04-06T09:35:36.947 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 11 2024-04-06T09:35:36.948 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:35:36.948 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=45, tag_tid=6, entry_tid=1], [object_number=44, tag_tid=6, entry_tid=0], [object_number=47, tag_tid=5, entry_tid=22415], [object_number=46, tag_tid=5, entry_tid=22414]]], state=connected] 2024-04-06T09:35:36.948 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=40, tag_tid=5, entry_tid=14512], [object_number=43, tag_tid=5, entry_tid=14511], [object_number=42, tag_tid=5, entry_tid=14510], [object_number=41, tag_tid=5, entry_tid=14509]]], state=connected] 2024-04-06T09:35:36.951 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:35:36.954 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=40, tag_tid=5, entry_tid=14512' 2024-04-06T09:35:36.954 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=45, tag_tid=6, entry_tid=1' -a 'object_number=45, tag_tid=6, entry_tid=1' = 'object_number=40, tag_tid=5, entry_tid=14512' 2024-04-06T09:35:36.954 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=40, tag_tid=5, entry_tid=14512' '!=' 'object_number=41, tag_tid=5, entry_tid=14413' 2024-04-06T09:35:36.954 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:35:36.954 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=40, tag_tid=5, entry_tid=14512' = 'object_number=41, tag_tid=5, entry_tid=14413' 2024-04-06T09:35:36.954 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=40, tag_tid=5, entry_tid=14512' 2024-04-06T09:35:36.954 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=45, tag_tid=6, entry_tid=1' 2024-04-06T09:35:36.954 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:35:36.954 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:35:36.955 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=6 2024-04-06T09:35:36.956 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=40, tag_tid=5, entry_tid=14512' 2024-04-06T09:35:36.956 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:35:36.956 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:35:36.957 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=5 2024-04-06T09:35:36.957 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=45, tag_tid=6, entry_tid=1' 2024-04-06T09:35:36.958 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:35:36.958 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:35:36.959 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:35:36.959 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=40, tag_tid=5, entry_tid=14512' 2024-04-06T09:35:36.959 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:35:36.959 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:35:36.961 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=14512 2024-04-06T09:35:36.961 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 6 = 5 -a 1 -le 14512 2024-04-06T09:35:36.961 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:35:36.961 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:35:36.961 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:35:37.163 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:35:37.163 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:35:37.163 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:35:37.163 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:35:37.163 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:35:37.163 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:35:37.163 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:35:37.163 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:35:37.163 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:35:37.163 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:35:37.163 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:35:37.163 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:35:37.163 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:35:37.163 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:35:37.164 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:35:37.164 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:35:37.164 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:35:37.167 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:35:37.167 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:35:37.167 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:35:37.167 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:35:37.167 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:35:37.167 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:35:37.167 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:35:37.168 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:35:37.168 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:35:37.168 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:35:37.168 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:35:37.168 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:35:37.168 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:35:37.168 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:35:37.168 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:35:37.168 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:35:37.168 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:35:37.168 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:35:37.168 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:35:37.168 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:35:37.168 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:35:37.169 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:35:37.169 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:35:37.169 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:35:37.169 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:35:37.169 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:35:37.169 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:35:37.169 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:35:37.169 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:35:37.169 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:35:37.169 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:35:37.169 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:35:37.193 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:35:37.194 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:35:37.194 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:35:37.194 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:35:37.321 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:35:37.322 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:35:37.322 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:35:37.322 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:35:37.322 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:35:37.322 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:35:37.323 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:35:37.323 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:35:37.323 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:35:37.323 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:35:37.323 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:35:37.323 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:35:37.324 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:35:37.325 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:35:37.325 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:35:37.326 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:35:37.380 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 10 2024-04-06T09:35:37.380 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 11 2024-04-06T09:35:37.380 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:35:37.380 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=45, tag_tid=6, entry_tid=1], [object_number=44, tag_tid=6, entry_tid=0], [object_number=47, tag_tid=5, entry_tid=22415], [object_number=46, tag_tid=5, entry_tid=22414]]], state=connected] 2024-04-06T09:35:37.380 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=43, tag_tid=5, entry_tid=14643], [object_number=42, tag_tid=5, entry_tid=14642], [object_number=41, tag_tid=5, entry_tid=14641], [object_number=40, tag_tid=5, entry_tid=14640]]], state=connected] 2024-04-06T09:35:37.384 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:35:37.385 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=45, tag_tid=6, entry_tid=1' 2024-04-06T09:35:37.387 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:35:37.387 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:35:37.387 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:35:37.387 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:35:37.387 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:35:37.387 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:35:37.387 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:35:37.387 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:35:37.387 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:35:37.387 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:35:37.388 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:35:37.388 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:35:37.388 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:35:37.388 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:35:37.388 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:35:37.434 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 10 2024-04-06T09:35:37.434 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 11 2024-04-06T09:35:37.434 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:35:37.434 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=45, tag_tid=6, entry_tid=1], [object_number=44, tag_tid=6, entry_tid=0], [object_number=47, tag_tid=5, entry_tid=22415], [object_number=46, tag_tid=5, entry_tid=22414]]], state=connected] 2024-04-06T09:35:37.434 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=43, tag_tid=5, entry_tid=14643], [object_number=42, tag_tid=5, entry_tid=14642], [object_number=41, tag_tid=5, entry_tid=14641], [object_number=40, tag_tid=5, entry_tid=14640]]], state=connected] 2024-04-06T09:35:37.437 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:35:37.439 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=43, tag_tid=5, entry_tid=14643' 2024-04-06T09:35:37.439 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=45, tag_tid=6, entry_tid=1' -a 'object_number=45, tag_tid=6, entry_tid=1' = 'object_number=43, tag_tid=5, entry_tid=14643' 2024-04-06T09:35:37.439 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=43, tag_tid=5, entry_tid=14643' '!=' 'object_number=40, tag_tid=5, entry_tid=14512' 2024-04-06T09:35:37.439 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:35:37.439 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=43, tag_tid=5, entry_tid=14643' = 'object_number=40, tag_tid=5, entry_tid=14512' 2024-04-06T09:35:37.439 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=43, tag_tid=5, entry_tid=14643' 2024-04-06T09:35:37.440 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=45, tag_tid=6, entry_tid=1' 2024-04-06T09:35:37.440 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:35:37.440 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:35:37.441 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=6 2024-04-06T09:35:37.441 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=43, tag_tid=5, entry_tid=14643' 2024-04-06T09:35:37.441 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:35:37.442 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:35:37.443 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=5 2024-04-06T09:35:37.443 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=45, tag_tid=6, entry_tid=1' 2024-04-06T09:35:37.443 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:35:37.443 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:35:37.444 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:35:37.445 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=43, tag_tid=5, entry_tid=14643' 2024-04-06T09:35:37.445 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:35:37.445 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:35:37.446 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=14643 2024-04-06T09:35:37.446 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 6 = 5 -a 1 -le 14643 2024-04-06T09:35:37.447 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:35:37.447 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:35:37.447 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:35:37.648 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:35:37.649 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:35:37.649 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:35:37.649 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:35:37.649 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:35:37.649 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:35:37.649 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:35:37.649 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:35:37.649 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:35:37.649 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:35:37.649 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:35:37.649 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:35:37.649 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:35:37.650 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:35:37.650 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:35:37.650 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:35:37.650 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:35:37.652 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:35:37.652 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:35:37.652 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:35:37.652 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:35:37.652 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:35:37.652 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:35:37.653 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:35:37.653 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:35:37.653 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:35:37.653 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:35:37.653 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:35:37.653 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:35:37.653 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:35:37.653 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:35:37.653 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:35:37.653 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:35:37.653 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:35:37.653 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:35:37.653 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:35:37.653 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:35:37.653 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:35:37.654 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:35:37.654 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:35:37.654 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:35:37.654 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:35:37.654 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:35:37.654 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:35:37.654 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:35:37.654 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:35:37.654 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:35:37.654 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:35:37.655 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:35:37.677 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:35:37.678 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:35:37.678 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:35:37.678 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:35:37.809 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:35:37.811 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:35:37.811 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:35:37.811 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:35:37.811 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:35:37.811 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:35:37.811 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:35:37.811 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:35:37.811 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:35:37.811 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:35:37.811 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:35:37.812 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:35:37.812 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:35:37.813 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:35:37.813 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:35:37.813 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:35:37.864 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 10 2024-04-06T09:35:37.864 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 11 2024-04-06T09:35:37.864 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:35:37.864 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=45, tag_tid=6, entry_tid=1], [object_number=44, tag_tid=6, entry_tid=0], [object_number=47, tag_tid=5, entry_tid=22415], [object_number=46, tag_tid=5, entry_tid=22414]]], state=connected] 2024-04-06T09:35:37.864 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=41, tag_tid=5, entry_tid=14741], [object_number=40, tag_tid=5, entry_tid=14740], [object_number=43, tag_tid=5, entry_tid=14739], [object_number=42, tag_tid=5, entry_tid=14738]]], state=connected] 2024-04-06T09:35:37.868 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:35:37.871 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=45, tag_tid=6, entry_tid=1' 2024-04-06T09:35:37.871 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:35:37.871 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:35:37.871 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:35:37.871 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:35:37.871 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:35:37.871 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:35:37.871 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:35:37.871 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:35:37.871 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:35:37.871 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:35:37.872 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:35:37.872 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:35:37.872 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:35:37.872 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:35:37.873 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:35:37.917 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 10 2024-04-06T09:35:37.917 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 11 2024-04-06T09:35:37.918 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:35:37.918 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=45, tag_tid=6, entry_tid=1], [object_number=44, tag_tid=6, entry_tid=0], [object_number=47, tag_tid=5, entry_tid=22415], [object_number=46, tag_tid=5, entry_tid=22414]]], state=connected] 2024-04-06T09:35:37.918 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=41, tag_tid=5, entry_tid=14741], [object_number=40, tag_tid=5, entry_tid=14740], [object_number=43, tag_tid=5, entry_tid=14739], [object_number=42, tag_tid=5, entry_tid=14738]]], state=connected] 2024-04-06T09:35:37.921 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:35:37.923 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=41, tag_tid=5, entry_tid=14741' 2024-04-06T09:35:37.923 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=45, tag_tid=6, entry_tid=1' -a 'object_number=45, tag_tid=6, entry_tid=1' = 'object_number=41, tag_tid=5, entry_tid=14741' 2024-04-06T09:35:37.923 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=41, tag_tid=5, entry_tid=14741' '!=' 'object_number=43, tag_tid=5, entry_tid=14643' 2024-04-06T09:35:37.923 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:35:37.923 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=41, tag_tid=5, entry_tid=14741' = 'object_number=43, tag_tid=5, entry_tid=14643' 2024-04-06T09:35:37.923 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=41, tag_tid=5, entry_tid=14741' 2024-04-06T09:35:37.924 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=45, tag_tid=6, entry_tid=1' 2024-04-06T09:35:37.924 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:35:37.924 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:35:37.924 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=6 2024-04-06T09:35:37.925 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=41, tag_tid=5, entry_tid=14741' 2024-04-06T09:35:37.925 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:35:37.925 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:35:37.926 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=5 2024-04-06T09:35:37.927 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=45, tag_tid=6, entry_tid=1' 2024-04-06T09:35:37.927 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:35:37.927 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:35:37.928 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:35:37.928 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=41, tag_tid=5, entry_tid=14741' 2024-04-06T09:35:37.929 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:35:37.929 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:35:37.930 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=14741 2024-04-06T09:35:37.930 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 6 = 5 -a 1 -le 14741 2024-04-06T09:35:37.930 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:35:37.930 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:35:37.930 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:35:38.132 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:35:38.132 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:35:38.132 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:35:38.132 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:35:38.132 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:35:38.132 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:35:38.132 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:35:38.132 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:35:38.132 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:35:38.133 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:35:38.133 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:35:38.133 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:35:38.133 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:35:38.133 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:35:38.133 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:35:38.133 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:35:38.133 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:35:38.136 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:35:38.136 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:35:38.136 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:35:38.136 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:35:38.136 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:35:38.136 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:35:38.136 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:35:38.136 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:35:38.136 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:35:38.137 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:35:38.137 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:35:38.137 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:35:38.137 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:35:38.137 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:35:38.137 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:35:38.137 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:35:38.137 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:35:38.137 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:35:38.137 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:35:38.137 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:35:38.137 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:35:38.137 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:35:38.137 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:35:38.137 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:35:38.138 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:35:38.138 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:35:38.138 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:35:38.138 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:35:38.138 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:35:38.138 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:35:38.138 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:35:38.138 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:35:38.161 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:35:38.162 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:35:38.162 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:35:38.162 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:35:38.296 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:35:38.297 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:35:38.298 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:35:38.298 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:35:38.298 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:35:38.298 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:35:38.298 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:35:38.298 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:35:38.298 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:35:38.298 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:35:38.299 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:35:38.299 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:35:38.299 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:35:38.300 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:35:38.300 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:35:38.301 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:35:38.353 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 10 2024-04-06T09:35:38.354 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 11 2024-04-06T09:35:38.354 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:35:38.354 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=45, tag_tid=6, entry_tid=1], [object_number=44, tag_tid=6, entry_tid=0], [object_number=47, tag_tid=5, entry_tid=22415], [object_number=46, tag_tid=5, entry_tid=22414]]], state=connected] 2024-04-06T09:35:38.354 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=43, tag_tid=5, entry_tid=14835], [object_number=42, tag_tid=5, entry_tid=14834], [object_number=41, tag_tid=5, entry_tid=14833], [object_number=40, tag_tid=5, entry_tid=14832]]], state=connected] 2024-04-06T09:35:38.357 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:35:38.360 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=45, tag_tid=6, entry_tid=1' 2024-04-06T09:35:38.360 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:35:38.360 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:35:38.360 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:35:38.360 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:35:38.360 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:35:38.360 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:35:38.360 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:35:38.361 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:35:38.361 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:35:38.361 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:35:38.361 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:35:38.361 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:35:38.361 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:35:38.362 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:35:38.362 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:35:38.408 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 10 2024-04-06T09:35:38.408 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 11 2024-04-06T09:35:38.408 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:35:38.408 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=45, tag_tid=6, entry_tid=1], [object_number=44, tag_tid=6, entry_tid=0], [object_number=47, tag_tid=5, entry_tid=22415], [object_number=46, tag_tid=5, entry_tid=22414]]], state=connected] 2024-04-06T09:35:38.408 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=43, tag_tid=5, entry_tid=14835], [object_number=42, tag_tid=5, entry_tid=14834], [object_number=41, tag_tid=5, entry_tid=14833], [object_number=40, tag_tid=5, entry_tid=14832]]], state=connected] 2024-04-06T09:35:38.411 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:35:38.413 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=43, tag_tid=5, entry_tid=14835' 2024-04-06T09:35:38.413 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=45, tag_tid=6, entry_tid=1' -a 'object_number=45, tag_tid=6, entry_tid=1' = 'object_number=43, tag_tid=5, entry_tid=14835' 2024-04-06T09:35:38.413 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=43, tag_tid=5, entry_tid=14835' '!=' 'object_number=41, tag_tid=5, entry_tid=14741' 2024-04-06T09:35:38.414 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:35:38.414 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=43, tag_tid=5, entry_tid=14835' = 'object_number=41, tag_tid=5, entry_tid=14741' 2024-04-06T09:35:38.414 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=43, tag_tid=5, entry_tid=14835' 2024-04-06T09:35:38.414 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=45, tag_tid=6, entry_tid=1' 2024-04-06T09:35:38.414 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:35:38.414 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:35:38.415 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=6 2024-04-06T09:35:38.416 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=43, tag_tid=5, entry_tid=14835' 2024-04-06T09:35:38.416 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:35:38.416 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:35:38.417 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=5 2024-04-06T09:35:38.417 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=45, tag_tid=6, entry_tid=1' 2024-04-06T09:35:38.418 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:35:38.418 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:35:38.418 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:35:38.419 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=43, tag_tid=5, entry_tid=14835' 2024-04-06T09:35:38.419 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:35:38.419 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:35:38.420 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=14835 2024-04-06T09:35:38.420 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 6 = 5 -a 1 -le 14835 2024-04-06T09:35:38.421 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:35:38.421 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:35:38.421 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:35:38.623 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:35:38.623 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:35:38.623 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:35:38.623 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:35:38.623 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:35:38.623 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:35:38.623 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:35:38.623 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:35:38.623 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:35:38.623 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:35:38.624 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:35:38.624 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:35:38.624 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:35:38.624 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:35:38.624 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:35:38.624 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:35:38.625 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:35:38.628 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:35:38.628 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:35:38.628 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:35:38.628 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:35:38.628 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:35:38.628 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:35:38.628 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:35:38.628 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:35:38.628 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:35:38.628 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:35:38.628 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:35:38.628 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:35:38.629 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:35:38.629 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:35:38.629 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:35:38.629 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:35:38.629 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:35:38.629 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:35:38.629 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:35:38.629 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:35:38.629 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:35:38.629 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:35:38.629 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:35:38.629 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:35:38.629 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:35:38.629 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:35:38.629 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:35:38.630 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:35:38.630 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:35:38.630 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:35:38.630 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:35:38.630 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:35:38.653 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:35:38.653 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:35:38.653 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:35:38.654 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:35:38.788 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:35:38.789 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:35:38.789 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:35:38.789 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:35:38.789 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:35:38.789 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:35:38.789 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:35:38.789 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:35:38.789 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:35:38.789 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:35:38.790 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:35:38.790 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:35:38.790 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:35:38.792 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:35:38.792 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:35:38.792 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:35:38.843 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 10 2024-04-06T09:35:38.843 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 11 2024-04-06T09:35:38.844 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:35:38.844 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=45, tag_tid=6, entry_tid=1], [object_number=44, tag_tid=6, entry_tid=0], [object_number=47, tag_tid=5, entry_tid=22415], [object_number=46, tag_tid=5, entry_tid=22414]]], state=connected] 2024-04-06T09:35:38.844 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=40, tag_tid=5, entry_tid=14936], [object_number=43, tag_tid=5, entry_tid=14935], [object_number=42, tag_tid=5, entry_tid=14934], [object_number=41, tag_tid=5, entry_tid=14933]]], state=connected] 2024-04-06T09:35:38.847 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:35:38.848 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=45, tag_tid=6, entry_tid=1' 2024-04-06T09:35:38.850 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:35:38.850 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:35:38.850 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:35:38.850 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:35:38.850 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:35:38.850 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:35:38.850 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:35:38.850 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:35:38.850 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:35:38.850 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:35:38.850 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:35:38.851 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:35:38.851 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:35:38.851 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:35:38.851 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:35:38.897 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 10 2024-04-06T09:35:38.897 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 11 2024-04-06T09:35:38.897 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:35:38.897 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=45, tag_tid=6, entry_tid=1], [object_number=44, tag_tid=6, entry_tid=0], [object_number=47, tag_tid=5, entry_tid=22415], [object_number=46, tag_tid=5, entry_tid=22414]]], state=connected] 2024-04-06T09:35:38.897 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=40, tag_tid=5, entry_tid=14936], [object_number=43, tag_tid=5, entry_tid=14935], [object_number=42, tag_tid=5, entry_tid=14934], [object_number=41, tag_tid=5, entry_tid=14933]]], state=connected] 2024-04-06T09:35:38.900 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:35:38.903 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=40, tag_tid=5, entry_tid=14936' 2024-04-06T09:35:38.903 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=45, tag_tid=6, entry_tid=1' -a 'object_number=45, tag_tid=6, entry_tid=1' = 'object_number=40, tag_tid=5, entry_tid=14936' 2024-04-06T09:35:38.903 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=40, tag_tid=5, entry_tid=14936' '!=' 'object_number=43, tag_tid=5, entry_tid=14835' 2024-04-06T09:35:38.903 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:35:38.903 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=40, tag_tid=5, entry_tid=14936' = 'object_number=43, tag_tid=5, entry_tid=14835' 2024-04-06T09:35:38.903 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=40, tag_tid=5, entry_tid=14936' 2024-04-06T09:35:38.903 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=45, tag_tid=6, entry_tid=1' 2024-04-06T09:35:38.903 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:35:38.903 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:35:38.904 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=6 2024-04-06T09:35:38.904 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=40, tag_tid=5, entry_tid=14936' 2024-04-06T09:35:38.905 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:35:38.905 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:35:38.905 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=5 2024-04-06T09:35:38.906 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=45, tag_tid=6, entry_tid=1' 2024-04-06T09:35:38.906 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:35:38.906 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:35:38.907 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:35:38.908 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=40, tag_tid=5, entry_tid=14936' 2024-04-06T09:35:38.908 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:35:38.908 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:35:38.909 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=14936 2024-04-06T09:35:38.909 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 6 = 5 -a 1 -le 14936 2024-04-06T09:35:38.909 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:35:38.909 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:35:38.909 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:35:39.111 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:35:39.111 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:35:39.111 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:35:39.111 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:35:39.111 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:35:39.111 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:35:39.112 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:35:39.112 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:35:39.112 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:35:39.112 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:35:39.112 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:35:39.112 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:35:39.112 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:35:39.112 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:35:39.112 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:35:39.113 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:35:39.113 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:35:39.116 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:35:39.116 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:35:39.116 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:35:39.116 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:35:39.116 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:35:39.116 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:35:39.116 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:35:39.116 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:35:39.116 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:35:39.117 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:35:39.117 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:35:39.117 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:35:39.117 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:35:39.117 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:35:39.117 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:35:39.117 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:35:39.117 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:35:39.117 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:35:39.117 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:35:39.117 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:35:39.117 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:35:39.117 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:35:39.117 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:35:39.117 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:35:39.118 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:35:39.118 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:35:39.118 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:35:39.118 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:35:39.118 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:35:39.118 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:35:39.118 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:35:39.118 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:35:39.141 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:35:39.142 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:35:39.142 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:35:39.142 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:35:39.270 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:35:39.271 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:35:39.271 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:35:39.271 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:35:39.272 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:35:39.272 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:35:39.272 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:35:39.272 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:35:39.272 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:35:39.272 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:35:39.272 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:35:39.272 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:35:39.273 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:35:39.275 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:35:39.275 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:35:39.275 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:35:39.320 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 10 2024-04-06T09:35:39.320 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 11 2024-04-06T09:35:39.321 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:35:39.321 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=45, tag_tid=6, entry_tid=1], [object_number=44, tag_tid=6, entry_tid=0], [object_number=47, tag_tid=5, entry_tid=22415], [object_number=46, tag_tid=5, entry_tid=22414]]], state=connected] 2024-04-06T09:35:39.321 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=43, tag_tid=5, entry_tid=15027], [object_number=42, tag_tid=5, entry_tid=15026], [object_number=41, tag_tid=5, entry_tid=15025], [object_number=40, tag_tid=5, entry_tid=15024]]], state=connected] 2024-04-06T09:35:39.324 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:35:39.326 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=45, tag_tid=6, entry_tid=1' 2024-04-06T09:35:39.327 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:35:39.327 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:35:39.327 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:35:39.327 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:35:39.327 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:35:39.327 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:35:39.327 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:35:39.327 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:35:39.327 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:35:39.327 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:35:39.328 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:35:39.328 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:35:39.328 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:35:39.328 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:35:39.328 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:35:39.373 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 10 2024-04-06T09:35:39.373 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 11 2024-04-06T09:35:39.373 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:35:39.373 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=45, tag_tid=6, entry_tid=1], [object_number=44, tag_tid=6, entry_tid=0], [object_number=47, tag_tid=5, entry_tid=22415], [object_number=46, tag_tid=5, entry_tid=22414]]], state=connected] 2024-04-06T09:35:39.373 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=43, tag_tid=5, entry_tid=15027], [object_number=42, tag_tid=5, entry_tid=15026], [object_number=41, tag_tid=5, entry_tid=15025], [object_number=40, tag_tid=5, entry_tid=15024]]], state=connected] 2024-04-06T09:35:39.377 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:35:39.379 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=43, tag_tid=5, entry_tid=15027' 2024-04-06T09:35:39.379 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=45, tag_tid=6, entry_tid=1' -a 'object_number=45, tag_tid=6, entry_tid=1' = 'object_number=43, tag_tid=5, entry_tid=15027' 2024-04-06T09:35:39.379 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=43, tag_tid=5, entry_tid=15027' '!=' 'object_number=40, tag_tid=5, entry_tid=14936' 2024-04-06T09:35:39.379 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:35:39.379 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=43, tag_tid=5, entry_tid=15027' = 'object_number=40, tag_tid=5, entry_tid=14936' 2024-04-06T09:35:39.379 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=43, tag_tid=5, entry_tid=15027' 2024-04-06T09:35:39.379 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=45, tag_tid=6, entry_tid=1' 2024-04-06T09:35:39.379 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:35:39.379 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:35:39.380 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=6 2024-04-06T09:35:39.380 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=43, tag_tid=5, entry_tid=15027' 2024-04-06T09:35:39.381 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:35:39.381 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:35:39.381 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=5 2024-04-06T09:35:39.382 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=45, tag_tid=6, entry_tid=1' 2024-04-06T09:35:39.382 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:35:39.382 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:35:39.383 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:35:39.384 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=43, tag_tid=5, entry_tid=15027' 2024-04-06T09:35:39.384 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:35:39.384 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:35:39.385 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=15027 2024-04-06T09:35:39.385 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 6 = 5 -a 1 -le 15027 2024-04-06T09:35:39.385 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:35:39.385 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:35:39.385 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:35:39.587 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:35:39.587 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:35:39.587 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:35:39.587 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:35:39.587 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:35:39.588 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:35:39.588 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:35:39.588 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:35:39.588 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:35:39.588 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:35:39.588 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:35:39.588 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:35:39.588 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:35:39.588 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:35:39.588 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:35:39.589 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:35:39.589 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:35:39.592 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:35:39.592 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:35:39.592 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:35:39.592 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:35:39.592 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:35:39.592 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:35:39.592 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:35:39.592 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:35:39.593 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:35:39.593 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:35:39.593 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:35:39.593 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:35:39.593 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:35:39.593 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:35:39.593 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:35:39.593 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:35:39.593 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:35:39.593 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:35:39.593 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:35:39.593 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:35:39.593 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:35:39.593 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:35:39.593 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:35:39.593 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:35:39.594 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:35:39.594 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:35:39.594 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:35:39.594 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:35:39.594 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:35:39.594 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:35:39.594 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:35:39.597 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:35:39.618 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:35:39.619 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:35:39.619 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:35:39.619 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:35:39.748 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:35:39.748 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:35:39.748 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:35:39.748 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:35:39.748 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:35:39.749 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:35:39.749 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:35:39.749 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:35:39.749 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:35:39.749 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:35:39.749 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:35:39.749 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:35:39.749 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:35:39.751 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:35:39.752 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:35:39.752 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:35:39.802 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 10 2024-04-06T09:35:39.802 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 11 2024-04-06T09:35:39.802 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:35:39.802 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=45, tag_tid=6, entry_tid=1], [object_number=44, tag_tid=6, entry_tid=0], [object_number=47, tag_tid=5, entry_tid=22415], [object_number=46, tag_tid=5, entry_tid=22414]]], state=connected] 2024-04-06T09:35:39.802 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=42, tag_tid=5, entry_tid=15118], [object_number=41, tag_tid=5, entry_tid=15117], [object_number=40, tag_tid=5, entry_tid=15116], [object_number=43, tag_tid=5, entry_tid=15115]]], state=connected] 2024-04-06T09:35:39.805 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:35:39.806 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=45, tag_tid=6, entry_tid=1' 2024-04-06T09:35:39.808 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:35:39.808 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:35:39.808 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:35:39.808 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:35:39.808 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:35:39.808 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:35:39.808 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:35:39.808 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:35:39.808 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:35:39.808 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:35:39.809 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:35:39.809 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:35:39.809 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:35:39.809 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:35:39.809 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:35:39.854 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 10 2024-04-06T09:35:39.854 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 11 2024-04-06T09:35:39.854 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:35:39.854 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=45, tag_tid=6, entry_tid=1], [object_number=44, tag_tid=6, entry_tid=0], [object_number=47, tag_tid=5, entry_tid=22415], [object_number=46, tag_tid=5, entry_tid=22414]]], state=connected] 2024-04-06T09:35:39.854 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=42, tag_tid=5, entry_tid=15118], [object_number=41, tag_tid=5, entry_tid=15117], [object_number=40, tag_tid=5, entry_tid=15116], [object_number=43, tag_tid=5, entry_tid=15115]]], state=connected] 2024-04-06T09:35:39.857 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:35:39.859 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=42, tag_tid=5, entry_tid=15118' 2024-04-06T09:35:39.859 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=45, tag_tid=6, entry_tid=1' -a 'object_number=45, tag_tid=6, entry_tid=1' = 'object_number=42, tag_tid=5, entry_tid=15118' 2024-04-06T09:35:39.859 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=42, tag_tid=5, entry_tid=15118' '!=' 'object_number=43, tag_tid=5, entry_tid=15027' 2024-04-06T09:35:39.860 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:35:39.860 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=42, tag_tid=5, entry_tid=15118' = 'object_number=43, tag_tid=5, entry_tid=15027' 2024-04-06T09:35:39.860 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=42, tag_tid=5, entry_tid=15118' 2024-04-06T09:35:39.860 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=45, tag_tid=6, entry_tid=1' 2024-04-06T09:35:39.860 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:35:39.860 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:35:39.860 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=6 2024-04-06T09:35:39.861 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=42, tag_tid=5, entry_tid=15118' 2024-04-06T09:35:39.861 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:35:39.861 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:35:39.862 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=5 2024-04-06T09:35:39.863 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=45, tag_tid=6, entry_tid=1' 2024-04-06T09:35:39.863 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:35:39.863 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:35:39.864 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:35:39.864 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=42, tag_tid=5, entry_tid=15118' 2024-04-06T09:35:39.865 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:35:39.865 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:35:39.866 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=15118 2024-04-06T09:35:39.866 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 6 = 5 -a 1 -le 15118 2024-04-06T09:35:39.866 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:35:39.866 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:35:39.866 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:35:40.068 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:35:40.068 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:35:40.068 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:35:40.069 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:35:40.069 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:35:40.069 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:35:40.069 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:35:40.069 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:35:40.069 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:35:40.069 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:35:40.069 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:35:40.069 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:35:40.069 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:35:40.069 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:35:40.069 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:35:40.070 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:35:40.070 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:35:40.073 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:35:40.073 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:35:40.073 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:35:40.073 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:35:40.073 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:35:40.073 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:35:40.073 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:35:40.073 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:35:40.073 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:35:40.073 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:35:40.073 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:35:40.073 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:35:40.074 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:35:40.074 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:35:40.074 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:35:40.074 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:35:40.074 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:35:40.074 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:35:40.074 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:35:40.074 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:35:40.074 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:35:40.074 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:35:40.074 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:35:40.074 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:35:40.074 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:35:40.075 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:35:40.075 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:35:40.075 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:35:40.075 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:35:40.075 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:35:40.075 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:35:40.075 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:35:40.100 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:35:40.100 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:35:40.100 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:35:40.100 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:35:40.226 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:35:40.228 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:35:40.228 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:35:40.228 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:35:40.228 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:35:40.228 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:35:40.228 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:35:40.228 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:35:40.228 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:35:40.228 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:35:40.229 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:35:40.229 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:35:40.229 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:35:40.231 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:35:40.231 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:35:40.231 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:35:40.276 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 10 2024-04-06T09:35:40.276 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 11 2024-04-06T09:35:40.276 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:35:40.276 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=45, tag_tid=6, entry_tid=1], [object_number=44, tag_tid=6, entry_tid=0], [object_number=47, tag_tid=5, entry_tid=22415], [object_number=46, tag_tid=5, entry_tid=22414]]], state=connected] 2024-04-06T09:35:40.276 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=43, tag_tid=5, entry_tid=15207], [object_number=42, tag_tid=5, entry_tid=15206], [object_number=41, tag_tid=5, entry_tid=15205], [object_number=40, tag_tid=5, entry_tid=15204]]], state=connected] 2024-04-06T09:35:40.280 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:35:40.281 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=45, tag_tid=6, entry_tid=1' 2024-04-06T09:35:40.282 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:35:40.282 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:35:40.282 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:35:40.282 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:35:40.282 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:35:40.282 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:35:40.282 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:35:40.283 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:35:40.283 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:35:40.283 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:35:40.283 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:35:40.283 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:35:40.283 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:35:40.283 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:35:40.284 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:35:40.329 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 10 2024-04-06T09:35:40.329 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 11 2024-04-06T09:35:40.329 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:35:40.329 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=45, tag_tid=6, entry_tid=1], [object_number=44, tag_tid=6, entry_tid=0], [object_number=47, tag_tid=5, entry_tid=22415], [object_number=46, tag_tid=5, entry_tid=22414]]], state=connected] 2024-04-06T09:35:40.329 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=43, tag_tid=5, entry_tid=15207], [object_number=42, tag_tid=5, entry_tid=15206], [object_number=41, tag_tid=5, entry_tid=15205], [object_number=40, tag_tid=5, entry_tid=15204]]], state=connected] 2024-04-06T09:35:40.333 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:35:40.335 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=43, tag_tid=5, entry_tid=15207' 2024-04-06T09:35:40.335 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=45, tag_tid=6, entry_tid=1' -a 'object_number=45, tag_tid=6, entry_tid=1' = 'object_number=43, tag_tid=5, entry_tid=15207' 2024-04-06T09:35:40.335 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=43, tag_tid=5, entry_tid=15207' '!=' 'object_number=42, tag_tid=5, entry_tid=15118' 2024-04-06T09:35:40.335 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:35:40.335 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=43, tag_tid=5, entry_tid=15207' = 'object_number=42, tag_tid=5, entry_tid=15118' 2024-04-06T09:35:40.335 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=43, tag_tid=5, entry_tid=15207' 2024-04-06T09:35:40.335 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=45, tag_tid=6, entry_tid=1' 2024-04-06T09:35:40.335 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:35:40.335 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:35:40.336 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=6 2024-04-06T09:35:40.336 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=43, tag_tid=5, entry_tid=15207' 2024-04-06T09:35:40.337 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:35:40.337 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:35:40.337 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=5 2024-04-06T09:35:40.338 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=45, tag_tid=6, entry_tid=1' 2024-04-06T09:35:40.338 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:35:40.338 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:35:40.339 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:35:40.340 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=43, tag_tid=5, entry_tid=15207' 2024-04-06T09:35:40.340 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:35:40.340 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:35:40.341 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=15207 2024-04-06T09:35:40.341 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 6 = 5 -a 1 -le 15207 2024-04-06T09:35:40.341 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:35:40.341 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:35:40.341 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:35:40.543 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:35:40.543 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:35:40.543 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:35:40.543 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:35:40.543 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:35:40.543 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:35:40.543 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:35:40.543 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:35:40.544 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:35:40.544 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:35:40.544 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:35:40.544 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:35:40.544 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:35:40.544 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:35:40.544 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:35:40.544 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:35:40.545 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:35:40.548 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:35:40.548 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:35:40.548 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:35:40.548 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:35:40.548 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:35:40.548 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:35:40.548 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:35:40.548 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:35:40.548 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:35:40.548 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:35:40.548 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:35:40.549 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:35:40.549 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:35:40.549 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:35:40.549 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:35:40.549 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:35:40.549 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:35:40.549 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:35:40.549 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:35:40.549 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:35:40.549 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:35:40.549 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:35:40.549 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:35:40.549 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:35:40.549 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:35:40.549 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:35:40.550 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:35:40.550 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:35:40.550 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:35:40.550 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:35:40.550 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:35:40.550 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:35:40.574 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:35:40.574 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:35:40.574 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:35:40.574 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:35:40.702 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:35:40.703 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:35:40.703 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:35:40.703 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:35:40.704 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:35:40.704 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:35:40.704 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:35:40.704 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:35:40.704 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:35:40.704 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:35:40.704 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:35:40.705 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:35:40.705 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:35:40.706 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:35:40.707 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:35:40.707 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:35:40.753 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 10 2024-04-06T09:35:40.753 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 11 2024-04-06T09:35:40.753 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:35:40.753 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=45, tag_tid=6, entry_tid=1], [object_number=44, tag_tid=6, entry_tid=0], [object_number=47, tag_tid=5, entry_tid=22415], [object_number=46, tag_tid=5, entry_tid=22414]]], state=connected] 2024-04-06T09:35:40.753 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=40, tag_tid=5, entry_tid=15304], [object_number=43, tag_tid=5, entry_tid=15303], [object_number=42, tag_tid=5, entry_tid=15302], [object_number=41, tag_tid=5, entry_tid=15301]]], state=connected] 2024-04-06T09:35:40.757 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:35:40.758 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=45, tag_tid=6, entry_tid=1' 2024-04-06T09:35:40.759 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:35:40.759 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:35:40.759 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:35:40.759 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:35:40.759 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:35:40.759 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:35:40.759 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:35:40.760 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:35:40.760 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:35:40.760 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:35:40.760 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:35:40.760 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:35:40.760 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:35:40.760 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:35:40.761 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:35:40.805 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 10 2024-04-06T09:35:40.805 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 11 2024-04-06T09:35:40.805 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:35:40.806 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=45, tag_tid=6, entry_tid=1], [object_number=44, tag_tid=6, entry_tid=0], [object_number=47, tag_tid=5, entry_tid=22415], [object_number=46, tag_tid=5, entry_tid=22414]]], state=connected] 2024-04-06T09:35:40.806 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=40, tag_tid=5, entry_tid=15304], [object_number=43, tag_tid=5, entry_tid=15303], [object_number=42, tag_tid=5, entry_tid=15302], [object_number=41, tag_tid=5, entry_tid=15301]]], state=connected] 2024-04-06T09:35:40.809 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:35:40.811 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=40, tag_tid=5, entry_tid=15304' 2024-04-06T09:35:40.811 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=45, tag_tid=6, entry_tid=1' -a 'object_number=45, tag_tid=6, entry_tid=1' = 'object_number=40, tag_tid=5, entry_tid=15304' 2024-04-06T09:35:40.811 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=40, tag_tid=5, entry_tid=15304' '!=' 'object_number=43, tag_tid=5, entry_tid=15207' 2024-04-06T09:35:40.811 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:35:40.811 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=40, tag_tid=5, entry_tid=15304' = 'object_number=43, tag_tid=5, entry_tid=15207' 2024-04-06T09:35:40.811 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=40, tag_tid=5, entry_tid=15304' 2024-04-06T09:35:40.811 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=45, tag_tid=6, entry_tid=1' 2024-04-06T09:35:40.811 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:35:40.812 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:35:40.813 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=6 2024-04-06T09:35:40.813 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=40, tag_tid=5, entry_tid=15304' 2024-04-06T09:35:40.813 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:35:40.813 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:35:40.814 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=5 2024-04-06T09:35:40.814 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=45, tag_tid=6, entry_tid=1' 2024-04-06T09:35:40.815 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:35:40.815 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:35:40.816 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:35:40.816 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=40, tag_tid=5, entry_tid=15304' 2024-04-06T09:35:40.816 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:35:40.816 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:35:40.817 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=15304 2024-04-06T09:35:40.818 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 6 = 5 -a 1 -le 15304 2024-04-06T09:35:40.818 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:35:40.818 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:35:40.818 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:35:41.019 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:35:41.020 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:35:41.020 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:35:41.020 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:35:41.020 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:35:41.020 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:35:41.020 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:35:41.020 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:35:41.020 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:35:41.020 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:35:41.020 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:35:41.020 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:35:41.020 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:35:41.020 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:35:41.020 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:35:41.021 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:35:41.021 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:35:41.023 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:35:41.023 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:35:41.023 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:35:41.024 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:35:41.024 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:35:41.024 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:35:41.024 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:35:41.024 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:35:41.024 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:35:41.024 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:35:41.024 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:35:41.024 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:35:41.024 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:35:41.024 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:35:41.024 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:35:41.024 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:35:41.026 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:35:41.026 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:35:41.026 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:35:41.026 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:35:41.026 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:35:41.026 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:35:41.026 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:35:41.026 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:35:41.026 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:35:41.026 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:35:41.026 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:35:41.027 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:35:41.027 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:35:41.027 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:35:41.027 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:35:41.027 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:35:41.051 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:35:41.051 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:35:41.052 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:35:41.052 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:35:41.182 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:35:41.183 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:35:41.183 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:35:41.183 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:35:41.183 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:35:41.183 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:35:41.183 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:35:41.183 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:35:41.184 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:35:41.184 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:35:41.184 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:35:41.184 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:35:41.184 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:35:41.186 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:35:41.186 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:35:41.186 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:35:41.238 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 10 2024-04-06T09:35:41.238 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 11 2024-04-06T09:35:41.238 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:35:41.238 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=45, tag_tid=6, entry_tid=1], [object_number=44, tag_tid=6, entry_tid=0], [object_number=47, tag_tid=5, entry_tid=22415], [object_number=46, tag_tid=5, entry_tid=22414]]], state=connected] 2024-04-06T09:35:41.238 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=42, tag_tid=5, entry_tid=15394], [object_number=41, tag_tid=5, entry_tid=15393], [object_number=40, tag_tid=5, entry_tid=15392], [object_number=43, tag_tid=5, entry_tid=15391]]], state=connected] 2024-04-06T09:35:41.241 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:35:41.243 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=45, tag_tid=6, entry_tid=1' 2024-04-06T09:35:41.244 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:35:41.244 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:35:41.244 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:35:41.244 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:35:41.244 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:35:41.244 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:35:41.244 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:35:41.244 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:35:41.244 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:35:41.244 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:35:41.245 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:35:41.245 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:35:41.245 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:35:41.245 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:35:41.245 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:35:41.291 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 10 2024-04-06T09:35:41.291 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 11 2024-04-06T09:35:41.291 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:35:41.291 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=45, tag_tid=6, entry_tid=1], [object_number=44, tag_tid=6, entry_tid=0], [object_number=47, tag_tid=5, entry_tid=22415], [object_number=46, tag_tid=5, entry_tid=22414]]], state=connected] 2024-04-06T09:35:41.291 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=42, tag_tid=5, entry_tid=15394], [object_number=41, tag_tid=5, entry_tid=15393], [object_number=40, tag_tid=5, entry_tid=15392], [object_number=43, tag_tid=5, entry_tid=15391]]], state=connected] 2024-04-06T09:35:41.295 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:35:41.297 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=42, tag_tid=5, entry_tid=15394' 2024-04-06T09:35:41.297 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=45, tag_tid=6, entry_tid=1' -a 'object_number=45, tag_tid=6, entry_tid=1' = 'object_number=42, tag_tid=5, entry_tid=15394' 2024-04-06T09:35:41.297 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=42, tag_tid=5, entry_tid=15394' '!=' 'object_number=40, tag_tid=5, entry_tid=15304' 2024-04-06T09:35:41.297 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:35:41.297 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=42, tag_tid=5, entry_tid=15394' = 'object_number=40, tag_tid=5, entry_tid=15304' 2024-04-06T09:35:41.297 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=42, tag_tid=5, entry_tid=15394' 2024-04-06T09:35:41.297 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=45, tag_tid=6, entry_tid=1' 2024-04-06T09:35:41.298 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:35:41.298 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:35:41.298 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=6 2024-04-06T09:35:41.299 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=42, tag_tid=5, entry_tid=15394' 2024-04-06T09:35:41.299 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:35:41.299 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:35:41.300 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=5 2024-04-06T09:35:41.301 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=45, tag_tid=6, entry_tid=1' 2024-04-06T09:35:41.301 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:35:41.301 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:35:41.302 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:35:41.302 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=42, tag_tid=5, entry_tid=15394' 2024-04-06T09:35:41.302 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:35:41.303 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:35:41.304 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=15394 2024-04-06T09:35:41.304 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 6 = 5 -a 1 -le 15394 2024-04-06T09:35:41.304 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:35:41.304 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:35:41.304 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:35:41.505 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:35:41.506 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:35:41.506 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:35:41.506 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:35:41.506 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:35:41.506 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:35:41.506 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:35:41.506 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:35:41.506 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:35:41.506 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:35:41.506 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:35:41.506 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:35:41.506 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:35:41.507 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:35:41.507 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:35:41.507 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:35:41.507 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:35:41.510 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:35:41.510 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:35:41.510 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:35:41.510 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:35:41.510 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:35:41.510 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:35:41.511 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:35:41.511 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:35:41.511 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:35:41.511 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:35:41.511 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:35:41.511 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:35:41.511 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:35:41.511 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:35:41.511 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:35:41.511 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:35:41.511 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:35:41.511 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:35:41.511 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:35:41.511 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:35:41.511 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:35:41.511 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:35:41.512 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:35:41.512 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:35:41.512 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:35:41.512 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:35:41.512 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:35:41.512 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:35:41.512 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:35:41.512 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:35:41.512 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:35:41.512 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:35:41.536 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:35:41.536 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:35:41.536 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:35:41.536 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:35:41.669 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:35:41.671 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:35:41.671 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:35:41.671 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:35:41.671 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:35:41.671 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:35:41.671 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:35:41.671 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:35:41.671 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:35:41.671 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:35:41.671 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:35:41.671 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:35:41.671 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:35:41.672 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:35:41.672 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:35:41.672 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:35:41.723 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 10 2024-04-06T09:35:41.724 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 11 2024-04-06T09:35:41.724 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:35:41.724 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=45, tag_tid=6, entry_tid=1], [object_number=44, tag_tid=6, entry_tid=0], [object_number=47, tag_tid=5, entry_tid=22415], [object_number=46, tag_tid=5, entry_tid=22414]]], state=connected] 2024-04-06T09:35:41.724 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=43, tag_tid=5, entry_tid=15499], [object_number=42, tag_tid=5, entry_tid=15498], [object_number=41, tag_tid=5, entry_tid=15497], [object_number=40, tag_tid=5, entry_tid=15496]]], state=connected] 2024-04-06T09:35:41.727 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:35:41.729 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=45, tag_tid=6, entry_tid=1' 2024-04-06T09:35:41.730 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:35:41.730 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:35:41.730 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:35:41.730 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:35:41.730 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:35:41.730 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:35:41.730 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:35:41.730 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:35:41.730 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:35:41.730 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:35:41.731 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:35:41.731 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:35:41.731 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:35:41.731 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:35:41.731 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:35:41.777 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 10 2024-04-06T09:35:41.777 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 11 2024-04-06T09:35:41.778 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:35:41.778 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=45, tag_tid=6, entry_tid=1], [object_number=44, tag_tid=6, entry_tid=0], [object_number=47, tag_tid=5, entry_tid=22415], [object_number=46, tag_tid=5, entry_tid=22414]]], state=connected] 2024-04-06T09:35:41.778 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=43, tag_tid=5, entry_tid=15499], [object_number=42, tag_tid=5, entry_tid=15498], [object_number=41, tag_tid=5, entry_tid=15497], [object_number=40, tag_tid=5, entry_tid=15496]]], state=connected] 2024-04-06T09:35:41.781 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:35:41.783 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=43, tag_tid=5, entry_tid=15499' 2024-04-06T09:35:41.783 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=45, tag_tid=6, entry_tid=1' -a 'object_number=45, tag_tid=6, entry_tid=1' = 'object_number=43, tag_tid=5, entry_tid=15499' 2024-04-06T09:35:41.783 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=43, tag_tid=5, entry_tid=15499' '!=' 'object_number=42, tag_tid=5, entry_tid=15394' 2024-04-06T09:35:41.783 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:35:41.783 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=43, tag_tid=5, entry_tid=15499' = 'object_number=42, tag_tid=5, entry_tid=15394' 2024-04-06T09:35:41.783 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=43, tag_tid=5, entry_tid=15499' 2024-04-06T09:35:41.783 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=45, tag_tid=6, entry_tid=1' 2024-04-06T09:35:41.784 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:35:41.784 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:35:41.784 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=6 2024-04-06T09:35:41.785 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=43, tag_tid=5, entry_tid=15499' 2024-04-06T09:35:41.785 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:35:41.785 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:35:41.787 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=5 2024-04-06T09:35:41.787 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=45, tag_tid=6, entry_tid=1' 2024-04-06T09:35:41.787 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:35:41.787 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:35:41.787 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:35:41.788 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=43, tag_tid=5, entry_tid=15499' 2024-04-06T09:35:41.788 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:35:41.788 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:35:41.789 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=15499 2024-04-06T09:35:41.790 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 6 = 5 -a 1 -le 15499 2024-04-06T09:35:41.790 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:35:41.790 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:35:41.790 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:35:41.991 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:35:41.992 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:35:41.992 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:35:41.992 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:35:41.992 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:35:41.992 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:35:41.992 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:35:41.992 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:35:41.992 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:35:41.992 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:35:41.992 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:35:41.992 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:35:41.992 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:35:41.992 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:35:41.992 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:35:41.993 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:35:41.993 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:35:41.996 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:35:41.996 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:35:41.996 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:35:41.996 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:35:41.996 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:35:41.997 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:35:41.997 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:35:41.997 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:35:41.997 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:35:41.997 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:35:41.997 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:35:41.997 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:35:41.997 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:35:41.997 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:35:41.997 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:35:41.997 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:35:41.997 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:35:41.997 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:35:41.997 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:35:41.997 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:35:41.998 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:35:41.998 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:35:41.998 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:35:41.998 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:35:41.998 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:35:41.998 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:35:41.998 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:35:41.998 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:35:41.998 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:35:41.998 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:35:41.998 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:35:41.998 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:35:42.024 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:35:42.024 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:35:42.024 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:35:42.024 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:35:42.163 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:35:42.164 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:35:42.164 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:35:42.164 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:35:42.165 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:35:42.165 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:35:42.165 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:35:42.165 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:35:42.165 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:35:42.165 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:35:42.165 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:35:42.166 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:35:42.166 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:35:42.167 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:35:42.168 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:35:42.168 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:35:42.213 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 10 2024-04-06T09:35:42.213 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 11 2024-04-06T09:35:42.213 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:35:42.213 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=45, tag_tid=6, entry_tid=1], [object_number=44, tag_tid=6, entry_tid=0], [object_number=47, tag_tid=5, entry_tid=22415], [object_number=46, tag_tid=5, entry_tid=22414]]], state=connected] 2024-04-06T09:35:42.213 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=42, tag_tid=5, entry_tid=15594], [object_number=41, tag_tid=5, entry_tid=15593], [object_number=40, tag_tid=5, entry_tid=15592], [object_number=43, tag_tid=5, entry_tid=15591]]], state=connected] 2024-04-06T09:35:42.217 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:35:42.219 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=45, tag_tid=6, entry_tid=1' 2024-04-06T09:35:42.219 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:35:42.220 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:35:42.220 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:35:42.220 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:35:42.220 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:35:42.220 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:35:42.220 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:35:42.220 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:35:42.220 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:35:42.220 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:35:42.220 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:35:42.220 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:35:42.221 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:35:42.221 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:35:42.221 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:35:42.265 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 10 2024-04-06T09:35:42.265 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 11 2024-04-06T09:35:42.265 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:35:42.265 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=45, tag_tid=6, entry_tid=1], [object_number=44, tag_tid=6, entry_tid=0], [object_number=47, tag_tid=5, entry_tid=22415], [object_number=46, tag_tid=5, entry_tid=22414]]], state=connected] 2024-04-06T09:35:42.265 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=42, tag_tid=5, entry_tid=15594], [object_number=41, tag_tid=5, entry_tid=15593], [object_number=40, tag_tid=5, entry_tid=15592], [object_number=43, tag_tid=5, entry_tid=15591]]], state=connected] 2024-04-06T09:35:42.269 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:35:42.271 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=42, tag_tid=5, entry_tid=15594' 2024-04-06T09:35:42.271 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=45, tag_tid=6, entry_tid=1' -a 'object_number=45, tag_tid=6, entry_tid=1' = 'object_number=42, tag_tid=5, entry_tid=15594' 2024-04-06T09:35:42.271 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=42, tag_tid=5, entry_tid=15594' '!=' 'object_number=43, tag_tid=5, entry_tid=15499' 2024-04-06T09:35:42.271 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:35:42.271 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=42, tag_tid=5, entry_tid=15594' = 'object_number=43, tag_tid=5, entry_tid=15499' 2024-04-06T09:35:42.271 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=42, tag_tid=5, entry_tid=15594' 2024-04-06T09:35:42.271 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=45, tag_tid=6, entry_tid=1' 2024-04-06T09:35:42.271 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:35:42.271 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:35:42.272 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=6 2024-04-06T09:35:42.273 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=42, tag_tid=5, entry_tid=15594' 2024-04-06T09:35:42.273 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:35:42.273 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:35:42.274 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=5 2024-04-06T09:35:42.274 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=45, tag_tid=6, entry_tid=1' 2024-04-06T09:35:42.275 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:35:42.275 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:35:42.275 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:35:42.276 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=42, tag_tid=5, entry_tid=15594' 2024-04-06T09:35:42.276 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:35:42.276 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:35:42.277 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=15594 2024-04-06T09:35:42.277 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 6 = 5 -a 1 -le 15594 2024-04-06T09:35:42.277 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:35:42.278 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:35:42.278 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:35:42.479 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:35:42.479 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:35:42.479 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:35:42.479 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:35:42.479 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:35:42.480 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:35:42.480 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:35:42.480 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:35:42.480 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:35:42.480 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:35:42.480 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:35:42.480 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:35:42.480 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:35:42.480 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:35:42.480 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:35:42.480 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:35:42.481 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:35:42.485 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:35:42.485 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:35:42.485 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:35:42.485 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:35:42.485 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:35:42.485 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:35:42.485 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:35:42.485 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:35:42.485 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:35:42.485 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:35:42.485 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:35:42.485 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:35:42.485 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:35:42.485 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:35:42.486 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:35:42.486 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:35:42.486 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:35:42.486 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:35:42.486 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:35:42.486 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:35:42.486 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:35:42.486 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:35:42.486 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:35:42.486 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:35:42.486 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:35:42.486 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:35:42.486 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:35:42.486 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:35:42.486 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:35:42.487 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:35:42.487 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:35:42.487 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:35:42.512 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:35:42.512 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:35:42.512 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:35:42.512 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:35:42.645 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:35:42.646 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:35:42.646 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:35:42.646 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:35:42.646 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:35:42.647 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:35:42.647 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:35:42.647 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:35:42.647 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:35:42.647 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:35:42.647 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:35:42.647 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:35:42.648 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:35:42.650 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:35:42.651 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:35:42.651 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:35:42.697 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 10 2024-04-06T09:35:42.697 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 11 2024-04-06T09:35:42.697 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:35:42.697 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=45, tag_tid=6, entry_tid=1], [object_number=44, tag_tid=6, entry_tid=0], [object_number=47, tag_tid=5, entry_tid=22415], [object_number=46, tag_tid=5, entry_tid=22414]]], state=connected] 2024-04-06T09:35:42.697 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=41, tag_tid=5, entry_tid=15709], [object_number=40, tag_tid=5, entry_tid=15708], [object_number=43, tag_tid=5, entry_tid=15707], [object_number=42, tag_tid=5, entry_tid=15706]]], state=connected] 2024-04-06T09:35:42.701 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:35:42.702 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=45, tag_tid=6, entry_tid=1' 2024-04-06T09:35:42.703 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:35:42.703 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:35:42.703 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:35:42.703 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:35:42.703 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:35:42.703 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:35:42.704 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:35:42.704 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:35:42.704 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:35:42.704 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:35:42.704 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:35:42.704 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:35:42.704 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:35:42.704 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:35:42.705 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:35:42.750 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 10 2024-04-06T09:35:42.750 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 11 2024-04-06T09:35:42.750 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:35:42.750 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=45, tag_tid=6, entry_tid=1], [object_number=44, tag_tid=6, entry_tid=0], [object_number=47, tag_tid=5, entry_tid=22415], [object_number=46, tag_tid=5, entry_tid=22414]]], state=connected] 2024-04-06T09:35:42.750 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=41, tag_tid=5, entry_tid=15709], [object_number=40, tag_tid=5, entry_tid=15708], [object_number=43, tag_tid=5, entry_tid=15707], [object_number=42, tag_tid=5, entry_tid=15706]]], state=connected] 2024-04-06T09:35:42.754 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:35:42.756 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=41, tag_tid=5, entry_tid=15709' 2024-04-06T09:35:42.756 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=45, tag_tid=6, entry_tid=1' -a 'object_number=45, tag_tid=6, entry_tid=1' = 'object_number=41, tag_tid=5, entry_tid=15709' 2024-04-06T09:35:42.756 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=41, tag_tid=5, entry_tid=15709' '!=' 'object_number=42, tag_tid=5, entry_tid=15594' 2024-04-06T09:35:42.756 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:35:42.756 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=41, tag_tid=5, entry_tid=15709' = 'object_number=42, tag_tid=5, entry_tid=15594' 2024-04-06T09:35:42.756 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=41, tag_tid=5, entry_tid=15709' 2024-04-06T09:35:42.756 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=45, tag_tid=6, entry_tid=1' 2024-04-06T09:35:42.756 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:35:42.756 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:35:42.757 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=6 2024-04-06T09:35:42.758 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=41, tag_tid=5, entry_tid=15709' 2024-04-06T09:35:42.758 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:35:42.758 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:35:42.759 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=5 2024-04-06T09:35:42.759 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=45, tag_tid=6, entry_tid=1' 2024-04-06T09:35:42.759 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:35:42.760 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:35:42.760 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:35:42.761 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=41, tag_tid=5, entry_tid=15709' 2024-04-06T09:35:42.761 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:35:42.761 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:35:42.762 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=15709 2024-04-06T09:35:42.762 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 6 = 5 -a 1 -le 15709 2024-04-06T09:35:42.763 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:35:42.763 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:35:42.763 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:35:42.964 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:35:42.964 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:35:42.964 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:35:42.965 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:35:42.965 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:35:42.965 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:35:42.965 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:35:42.965 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:35:42.965 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:35:42.965 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:35:42.965 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:35:42.965 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:35:42.965 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:35:42.965 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:35:42.965 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:35:42.966 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:35:42.966 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:35:42.968 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:35:42.968 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:35:42.968 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:35:42.968 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:35:42.968 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:35:42.968 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:35:42.968 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:35:42.968 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:35:42.968 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:35:42.968 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:35:42.968 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:35:42.968 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:35:42.968 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:35:42.969 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:35:42.969 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:35:42.969 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:35:42.970 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:35:42.970 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:35:42.970 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:35:42.970 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:35:42.970 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:35:42.970 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:35:42.970 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:35:42.970 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:35:42.970 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:35:42.970 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:35:42.970 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:35:42.970 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:35:42.970 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:35:42.971 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:35:42.971 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:35:42.971 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:35:42.994 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:35:42.994 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:35:42.994 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:35:42.994 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:35:43.125 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:35:43.126 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:35:43.126 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:35:43.126 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:35:43.126 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:35:43.126 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:35:43.126 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:35:43.127 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:35:43.127 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:35:43.127 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:35:43.127 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:35:43.127 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:35:43.127 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:35:43.129 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:35:43.129 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:35:43.129 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:35:43.180 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 10 2024-04-06T09:35:43.180 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 11 2024-04-06T09:35:43.180 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:35:43.180 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=45, tag_tid=6, entry_tid=1], [object_number=44, tag_tid=6, entry_tid=0], [object_number=47, tag_tid=5, entry_tid=22415], [object_number=46, tag_tid=5, entry_tid=22414]]], state=connected] 2024-04-06T09:35:43.180 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=41, tag_tid=5, entry_tid=15809], [object_number=40, tag_tid=5, entry_tid=15808], [object_number=43, tag_tid=5, entry_tid=15807], [object_number=42, tag_tid=5, entry_tid=15806]]], state=connected] 2024-04-06T09:35:43.183 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:35:43.185 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=45, tag_tid=6, entry_tid=1' 2024-04-06T09:35:43.186 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:35:43.186 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:35:43.186 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:35:43.186 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:35:43.186 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:35:43.186 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:35:43.186 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:35:43.186 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:35:43.186 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:35:43.187 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:35:43.187 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:35:43.187 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:35:43.187 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:35:43.188 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:35:43.188 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:35:43.232 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 10 2024-04-06T09:35:43.233 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 11 2024-04-06T09:35:43.233 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:35:43.233 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=45, tag_tid=6, entry_tid=1], [object_number=44, tag_tid=6, entry_tid=0], [object_number=47, tag_tid=5, entry_tid=22415], [object_number=46, tag_tid=5, entry_tid=22414]]], state=connected] 2024-04-06T09:35:43.233 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=41, tag_tid=5, entry_tid=15809], [object_number=40, tag_tid=5, entry_tid=15808], [object_number=43, tag_tid=5, entry_tid=15807], [object_number=42, tag_tid=5, entry_tid=15806]]], state=connected] 2024-04-06T09:35:43.236 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:35:43.238 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=41, tag_tid=5, entry_tid=15809' 2024-04-06T09:35:43.238 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=45, tag_tid=6, entry_tid=1' -a 'object_number=45, tag_tid=6, entry_tid=1' = 'object_number=41, tag_tid=5, entry_tid=15809' 2024-04-06T09:35:43.238 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=41, tag_tid=5, entry_tid=15809' '!=' 'object_number=41, tag_tid=5, entry_tid=15709' 2024-04-06T09:35:43.238 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:35:43.238 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=41, tag_tid=5, entry_tid=15809' = 'object_number=41, tag_tid=5, entry_tid=15709' 2024-04-06T09:35:43.239 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=41, tag_tid=5, entry_tid=15809' 2024-04-06T09:35:43.239 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=45, tag_tid=6, entry_tid=1' 2024-04-06T09:35:43.239 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:35:43.239 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:35:43.240 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=6 2024-04-06T09:35:43.241 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=41, tag_tid=5, entry_tid=15809' 2024-04-06T09:35:43.241 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:35:43.241 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:35:43.242 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=5 2024-04-06T09:35:43.242 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=45, tag_tid=6, entry_tid=1' 2024-04-06T09:35:43.243 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:35:43.243 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:35:43.243 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:35:43.244 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=41, tag_tid=5, entry_tid=15809' 2024-04-06T09:35:43.244 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:35:43.244 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:35:43.245 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=15809 2024-04-06T09:35:43.245 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 6 = 5 -a 1 -le 15809 2024-04-06T09:35:43.245 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:35:43.245 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:35:43.245 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:35:43.447 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:35:43.447 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:35:43.447 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:35:43.447 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:35:43.447 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:35:43.447 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:35:43.447 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:35:43.447 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:35:43.447 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:35:43.447 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:35:43.447 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:35:43.447 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:35:43.447 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:35:43.448 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:35:43.448 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:35:43.448 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:35:43.448 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:35:43.452 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:35:43.452 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:35:43.452 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:35:43.452 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:35:43.452 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:35:43.452 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:35:43.452 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:35:43.452 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:35:43.452 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:35:43.452 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:35:43.452 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:35:43.453 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:35:43.453 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:35:43.453 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:35:43.453 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:35:43.453 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:35:43.453 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:35:43.453 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:35:43.453 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:35:43.453 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:35:43.453 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:35:43.453 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:35:43.453 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:35:43.453 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:35:43.453 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:35:43.453 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:35:43.453 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:35:43.454 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:35:43.454 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:35:43.454 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:35:43.454 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:35:43.454 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:35:43.480 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:35:43.480 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:35:43.480 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:35:43.480 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:35:43.608 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:35:43.609 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:35:43.609 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:35:43.609 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:35:43.609 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:35:43.609 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:35:43.609 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:35:43.609 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:35:43.609 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:35:43.609 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:35:43.610 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:35:43.610 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:35:43.610 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:35:43.613 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:35:43.613 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:35:43.613 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:35:43.664 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 10 2024-04-06T09:35:43.664 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 11 2024-04-06T09:35:43.664 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:35:43.664 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=45, tag_tid=6, entry_tid=1], [object_number=44, tag_tid=6, entry_tid=0], [object_number=47, tag_tid=5, entry_tid=22415], [object_number=46, tag_tid=5, entry_tid=22414]]], state=connected] 2024-04-06T09:35:43.664 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=40, tag_tid=5, entry_tid=15904], [object_number=43, tag_tid=5, entry_tid=15903], [object_number=42, tag_tid=5, entry_tid=15902], [object_number=41, tag_tid=5, entry_tid=15901]]], state=connected] 2024-04-06T09:35:43.667 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:35:43.669 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=45, tag_tid=6, entry_tid=1' 2024-04-06T09:35:43.670 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:35:43.670 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:35:43.670 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:35:43.670 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:35:43.670 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:35:43.670 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:35:43.670 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:35:43.670 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:35:43.670 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:35:43.670 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:35:43.671 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:35:43.671 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:35:43.671 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:35:43.671 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:35:43.671 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:35:43.718 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 10 2024-04-06T09:35:43.718 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 11 2024-04-06T09:35:43.718 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:35:43.718 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=45, tag_tid=6, entry_tid=1], [object_number=44, tag_tid=6, entry_tid=0], [object_number=47, tag_tid=5, entry_tid=22415], [object_number=46, tag_tid=5, entry_tid=22414]]], state=connected] 2024-04-06T09:35:43.718 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=40, tag_tid=5, entry_tid=15904], [object_number=43, tag_tid=5, entry_tid=15903], [object_number=42, tag_tid=5, entry_tid=15902], [object_number=41, tag_tid=5, entry_tid=15901]]], state=connected] 2024-04-06T09:35:43.722 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:35:43.724 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=40, tag_tid=5, entry_tid=15904' 2024-04-06T09:35:43.724 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=45, tag_tid=6, entry_tid=1' -a 'object_number=45, tag_tid=6, entry_tid=1' = 'object_number=40, tag_tid=5, entry_tid=15904' 2024-04-06T09:35:43.724 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=40, tag_tid=5, entry_tid=15904' '!=' 'object_number=41, tag_tid=5, entry_tid=15809' 2024-04-06T09:35:43.724 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:35:43.724 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=40, tag_tid=5, entry_tid=15904' = 'object_number=41, tag_tid=5, entry_tid=15809' 2024-04-06T09:35:43.724 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=40, tag_tid=5, entry_tid=15904' 2024-04-06T09:35:43.724 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=45, tag_tid=6, entry_tid=1' 2024-04-06T09:35:43.724 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:35:43.724 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:35:43.725 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=6 2024-04-06T09:35:43.726 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=40, tag_tid=5, entry_tid=15904' 2024-04-06T09:35:43.726 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:35:43.726 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:35:43.727 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=5 2024-04-06T09:35:43.727 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=45, tag_tid=6, entry_tid=1' 2024-04-06T09:35:43.727 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:35:43.727 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:35:43.728 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:35:43.729 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=40, tag_tid=5, entry_tid=15904' 2024-04-06T09:35:43.729 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:35:43.729 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:35:43.730 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=15904 2024-04-06T09:35:43.730 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 6 = 5 -a 1 -le 15904 2024-04-06T09:35:43.730 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:35:43.730 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:35:43.730 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:35:43.932 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:35:43.932 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:35:43.932 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:35:43.933 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:35:43.933 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:35:43.933 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:35:43.933 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:35:43.933 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:35:43.933 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:35:43.933 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:35:43.933 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:35:43.933 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:35:43.933 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:35:43.933 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:35:43.933 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:35:43.934 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:35:43.934 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:35:43.936 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:35:43.937 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:35:43.937 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:35:43.937 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:35:43.937 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:35:43.937 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:35:43.937 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:35:43.937 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:35:43.937 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:35:43.937 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:35:43.937 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:35:43.937 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:35:43.937 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:35:43.937 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:35:43.938 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:35:43.938 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:35:43.938 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:35:43.938 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:35:43.938 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:35:43.938 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:35:43.938 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:35:43.938 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:35:43.938 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:35:43.938 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:35:43.938 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:35:43.938 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:35:43.938 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:35:43.938 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:35:43.938 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:35:43.939 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:35:43.939 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:35:43.939 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:35:43.962 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:35:43.962 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:35:43.962 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:35:43.962 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:35:44.089 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:35:44.094 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:35:44.094 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:35:44.094 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:35:44.094 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:35:44.094 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:35:44.094 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:35:44.094 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:35:44.094 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:35:44.094 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:35:44.094 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:35:44.094 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:35:44.095 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:35:44.095 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:35:44.095 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:35:44.095 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:35:44.143 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 10 2024-04-06T09:35:44.143 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 11 2024-04-06T09:35:44.143 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:35:44.143 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=45, tag_tid=6, entry_tid=1], [object_number=44, tag_tid=6, entry_tid=0], [object_number=47, tag_tid=5, entry_tid=22415], [object_number=46, tag_tid=5, entry_tid=22414]]], state=connected] 2024-04-06T09:35:44.143 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=43, tag_tid=5, entry_tid=16007], [object_number=42, tag_tid=5, entry_tid=16006], [object_number=41, tag_tid=5, entry_tid=16005], [object_number=40, tag_tid=5, entry_tid=16004]]], state=connected] 2024-04-06T09:35:44.147 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:35:44.148 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=45, tag_tid=6, entry_tid=1' 2024-04-06T09:35:44.150 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:35:44.150 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:35:44.150 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:35:44.150 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:35:44.150 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:35:44.150 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:35:44.150 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:35:44.150 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:35:44.150 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:35:44.150 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:35:44.150 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:35:44.150 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:35:44.151 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:35:44.151 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:35:44.151 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:35:44.196 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 10 2024-04-06T09:35:44.196 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 11 2024-04-06T09:35:44.196 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:35:44.196 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=45, tag_tid=6, entry_tid=1], [object_number=44, tag_tid=6, entry_tid=0], [object_number=47, tag_tid=5, entry_tid=22415], [object_number=46, tag_tid=5, entry_tid=22414]]], state=connected] 2024-04-06T09:35:44.196 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=43, tag_tid=5, entry_tid=16007], [object_number=42, tag_tid=5, entry_tid=16006], [object_number=41, tag_tid=5, entry_tid=16005], [object_number=40, tag_tid=5, entry_tid=16004]]], state=connected] 2024-04-06T09:35:44.200 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:35:44.202 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=43, tag_tid=5, entry_tid=16007' 2024-04-06T09:35:44.202 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=45, tag_tid=6, entry_tid=1' -a 'object_number=45, tag_tid=6, entry_tid=1' = 'object_number=43, tag_tid=5, entry_tid=16007' 2024-04-06T09:35:44.202 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=43, tag_tid=5, entry_tid=16007' '!=' 'object_number=40, tag_tid=5, entry_tid=15904' 2024-04-06T09:35:44.202 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:35:44.202 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=43, tag_tid=5, entry_tid=16007' = 'object_number=40, tag_tid=5, entry_tid=15904' 2024-04-06T09:35:44.203 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=43, tag_tid=5, entry_tid=16007' 2024-04-06T09:35:44.203 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=45, tag_tid=6, entry_tid=1' 2024-04-06T09:35:44.203 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:35:44.203 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:35:44.203 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=6 2024-04-06T09:35:44.204 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=43, tag_tid=5, entry_tid=16007' 2024-04-06T09:35:44.204 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:35:44.204 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:35:44.205 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=5 2024-04-06T09:35:44.205 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=45, tag_tid=6, entry_tid=1' 2024-04-06T09:35:44.205 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:35:44.206 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:35:44.207 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:35:44.207 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=43, tag_tid=5, entry_tid=16007' 2024-04-06T09:35:44.207 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:35:44.207 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:35:44.209 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=16007 2024-04-06T09:35:44.209 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 6 = 5 -a 1 -le 16007 2024-04-06T09:35:44.209 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:35:44.209 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:35:44.209 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:35:44.411 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:35:44.411 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:35:44.411 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:35:44.411 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:35:44.411 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:35:44.411 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:35:44.411 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:35:44.412 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:35:44.412 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:35:44.412 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:35:44.412 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:35:44.412 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:35:44.412 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:35:44.412 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:35:44.412 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:35:44.412 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:35:44.415 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:35:44.463 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:35:44.463 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:35:44.463 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:35:44.464 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:35:44.464 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:35:44.464 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:35:44.464 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:35:44.464 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:35:44.464 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:35:44.464 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:35:44.464 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:35:44.464 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:35:44.464 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:35:44.464 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:35:44.464 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:35:44.464 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:35:44.464 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:35:44.465 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:35:44.465 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:35:44.465 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:35:44.465 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:35:44.465 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:35:44.465 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:35:44.465 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:35:44.465 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:35:44.465 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:35:44.465 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:35:44.465 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:35:44.465 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:35:44.465 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:35:44.465 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:35:44.466 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:35:44.466 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:35:44.467 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:35:44.467 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:35:44.467 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:35:44.567 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:35:44.568 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:35:44.568 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:35:44.568 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:35:44.568 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:35:44.568 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:35:44.568 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:35:44.569 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:35:44.569 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:35:44.569 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:35:44.569 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:35:44.569 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:35:44.569 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:35:44.572 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:35:44.572 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:35:44.572 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:35:44.617 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 10 2024-04-06T09:35:44.617 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 11 2024-04-06T09:35:44.617 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:35:44.617 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=45, tag_tid=6, entry_tid=1], [object_number=44, tag_tid=6, entry_tid=0], [object_number=47, tag_tid=5, entry_tid=22415], [object_number=46, tag_tid=5, entry_tid=22414]]], state=connected] 2024-04-06T09:35:44.617 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=41, tag_tid=5, entry_tid=16105], [object_number=40, tag_tid=5, entry_tid=16104], [object_number=43, tag_tid=5, entry_tid=16103], [object_number=42, tag_tid=5, entry_tid=16102]]], state=connected] 2024-04-06T09:35:44.621 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:35:44.622 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=45, tag_tid=6, entry_tid=1' 2024-04-06T09:35:44.623 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:35:44.623 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:35:44.623 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:35:44.623 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:35:44.623 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:35:44.623 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:35:44.624 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:35:44.624 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:35:44.624 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:35:44.624 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:35:44.624 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:35:44.624 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:35:44.624 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:35:44.624 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:35:44.625 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:35:44.669 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 10 2024-04-06T09:35:44.669 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 11 2024-04-06T09:35:44.669 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:35:44.670 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=45, tag_tid=6, entry_tid=1], [object_number=44, tag_tid=6, entry_tid=0], [object_number=47, tag_tid=5, entry_tid=22415], [object_number=46, tag_tid=5, entry_tid=22414]]], state=connected] 2024-04-06T09:35:44.670 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=41, tag_tid=5, entry_tid=16105], [object_number=40, tag_tid=5, entry_tid=16104], [object_number=43, tag_tid=5, entry_tid=16103], [object_number=42, tag_tid=5, entry_tid=16102]]], state=connected] 2024-04-06T09:35:44.673 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:35:44.675 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=41, tag_tid=5, entry_tid=16105' 2024-04-06T09:35:44.675 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=45, tag_tid=6, entry_tid=1' -a 'object_number=45, tag_tid=6, entry_tid=1' = 'object_number=41, tag_tid=5, entry_tid=16105' 2024-04-06T09:35:44.675 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=41, tag_tid=5, entry_tid=16105' '!=' 'object_number=43, tag_tid=5, entry_tid=16007' 2024-04-06T09:35:44.676 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:35:44.676 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=41, tag_tid=5, entry_tid=16105' = 'object_number=43, tag_tid=5, entry_tid=16007' 2024-04-06T09:35:44.676 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=41, tag_tid=5, entry_tid=16105' 2024-04-06T09:35:44.676 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=45, tag_tid=6, entry_tid=1' 2024-04-06T09:35:44.676 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:35:44.676 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:35:44.676 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=6 2024-04-06T09:35:44.677 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=41, tag_tid=5, entry_tid=16105' 2024-04-06T09:35:44.677 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:35:44.677 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:35:44.678 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=5 2024-04-06T09:35:44.679 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=45, tag_tid=6, entry_tid=1' 2024-04-06T09:35:44.679 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:35:44.679 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:35:44.680 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:35:44.680 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=41, tag_tid=5, entry_tid=16105' 2024-04-06T09:35:44.680 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:35:44.681 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:35:44.681 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=16105 2024-04-06T09:35:44.682 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 6 = 5 -a 1 -le 16105 2024-04-06T09:35:44.682 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:35:44.682 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:35:44.682 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:35:44.884 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:35:44.884 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:35:44.884 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:35:44.884 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:35:44.884 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:35:44.884 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:35:44.884 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:35:44.884 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:35:44.885 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:35:44.885 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:35:44.885 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:35:44.885 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:35:44.885 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:35:44.885 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:35:44.885 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:35:44.885 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:35:44.885 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:35:44.889 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:35:44.889 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:35:44.889 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:35:44.889 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:35:44.889 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:35:44.889 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:35:44.889 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:35:44.889 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:35:44.889 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:35:44.889 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:35:44.889 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:35:44.889 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:35:44.889 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:35:44.889 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:35:44.890 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:35:44.890 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:35:44.890 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:35:44.890 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:35:44.890 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:35:44.890 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:35:44.890 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:35:44.890 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:35:44.890 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:35:44.890 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:35:44.890 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:35:44.890 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:35:44.890 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:35:44.890 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:35:44.891 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:35:44.891 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:35:44.891 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:35:44.891 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:35:44.914 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:35:44.915 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:35:44.915 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:35:44.915 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:35:45.045 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:35:45.046 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:35:45.046 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:35:45.046 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:35:45.046 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:35:45.047 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:35:45.047 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:35:45.047 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:35:45.047 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:35:45.047 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:35:45.047 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:35:45.047 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:35:45.047 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:35:45.049 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:35:45.050 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:35:45.050 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:35:45.100 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 10 2024-04-06T09:35:45.101 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 11 2024-04-06T09:35:45.101 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:35:45.101 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=45, tag_tid=6, entry_tid=1], [object_number=44, tag_tid=6, entry_tid=0], [object_number=47, tag_tid=5, entry_tid=22415], [object_number=46, tag_tid=5, entry_tid=22414]]], state=connected] 2024-04-06T09:35:45.101 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=41, tag_tid=5, entry_tid=16209], [object_number=40, tag_tid=5, entry_tid=16208], [object_number=43, tag_tid=5, entry_tid=16207], [object_number=42, tag_tid=5, entry_tid=16206]]], state=connected] 2024-04-06T09:35:45.104 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:35:45.105 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=45, tag_tid=6, entry_tid=1' 2024-04-06T09:35:45.107 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:35:45.107 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:35:45.107 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:35:45.107 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:35:45.107 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:35:45.107 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:35:45.107 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:35:45.107 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:35:45.107 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:35:45.107 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:35:45.107 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:35:45.107 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:35:45.108 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:35:45.109 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:35:45.109 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:35:45.154 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 10 2024-04-06T09:35:45.154 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 11 2024-04-06T09:35:45.154 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:35:45.154 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=45, tag_tid=6, entry_tid=1], [object_number=44, tag_tid=6, entry_tid=0], [object_number=47, tag_tid=5, entry_tid=22415], [object_number=46, tag_tid=5, entry_tid=22414]]], state=connected] 2024-04-06T09:35:45.154 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=41, tag_tid=5, entry_tid=16209], [object_number=40, tag_tid=5, entry_tid=16208], [object_number=43, tag_tid=5, entry_tid=16207], [object_number=42, tag_tid=5, entry_tid=16206]]], state=connected] 2024-04-06T09:35:45.158 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:35:45.159 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=41, tag_tid=5, entry_tid=16209' 2024-04-06T09:35:45.159 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=45, tag_tid=6, entry_tid=1' -a 'object_number=45, tag_tid=6, entry_tid=1' = 'object_number=41, tag_tid=5, entry_tid=16209' 2024-04-06T09:35:45.159 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=41, tag_tid=5, entry_tid=16209' '!=' 'object_number=41, tag_tid=5, entry_tid=16105' 2024-04-06T09:35:45.160 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:35:45.160 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=41, tag_tid=5, entry_tid=16209' = 'object_number=41, tag_tid=5, entry_tid=16105' 2024-04-06T09:35:45.160 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=41, tag_tid=5, entry_tid=16209' 2024-04-06T09:35:45.160 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=45, tag_tid=6, entry_tid=1' 2024-04-06T09:35:45.160 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:35:45.160 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:35:45.160 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=6 2024-04-06T09:35:45.161 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=41, tag_tid=5, entry_tid=16209' 2024-04-06T09:35:45.161 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:35:45.161 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:35:45.162 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=5 2024-04-06T09:35:45.163 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=45, tag_tid=6, entry_tid=1' 2024-04-06T09:35:45.163 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:35:45.163 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:35:45.164 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:35:45.165 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=41, tag_tid=5, entry_tid=16209' 2024-04-06T09:35:45.165 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:35:45.165 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:35:45.166 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=16209 2024-04-06T09:35:45.166 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 6 = 5 -a 1 -le 16209 2024-04-06T09:35:45.166 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:35:45.166 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:35:45.166 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:35:45.368 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:35:45.368 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:35:45.368 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:35:45.368 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:35:45.368 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:35:45.368 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:35:45.368 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:35:45.368 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:35:45.368 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:35:45.368 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:35:45.368 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:35:45.369 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:35:45.369 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:35:45.369 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:35:45.369 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:35:45.369 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:35:45.369 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:35:45.372 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:35:45.372 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:35:45.372 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:35:45.372 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:35:45.373 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:35:45.373 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:35:45.373 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:35:45.373 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:35:45.373 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:35:45.373 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:35:45.373 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:35:45.373 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:35:45.373 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:35:45.373 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:35:45.373 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:35:45.373 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:35:45.373 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:35:45.373 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:35:45.374 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:35:45.374 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:35:45.374 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:35:45.374 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:35:45.374 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:35:45.374 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:35:45.374 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:35:45.374 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:35:45.374 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:35:45.374 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:35:45.374 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:35:45.374 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:35:45.374 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:35:45.374 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:35:45.399 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:35:45.399 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:35:45.399 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:35:45.399 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:35:45.528 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:35:45.529 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:35:45.529 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:35:45.529 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:35:45.529 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:35:45.530 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:35:45.530 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:35:45.530 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:35:45.530 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:35:45.530 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:35:45.530 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:35:45.530 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:35:45.531 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:35:45.531 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:35:45.532 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:35:45.532 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:35:45.577 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 10 2024-04-06T09:35:45.577 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 11 2024-04-06T09:35:45.577 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:35:45.577 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=45, tag_tid=6, entry_tid=1], [object_number=44, tag_tid=6, entry_tid=0], [object_number=47, tag_tid=5, entry_tid=22415], [object_number=46, tag_tid=5, entry_tid=22414]]], state=connected] 2024-04-06T09:35:45.577 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=41, tag_tid=5, entry_tid=16309], [object_number=40, tag_tid=5, entry_tid=16308], [object_number=43, tag_tid=5, entry_tid=16307], [object_number=42, tag_tid=5, entry_tid=16306]]], state=connected] 2024-04-06T09:35:45.580 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:35:45.581 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=45, tag_tid=6, entry_tid=1' 2024-04-06T09:35:45.582 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:35:45.583 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:35:45.583 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:35:45.583 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:35:45.583 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:35:45.583 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:35:45.583 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:35:45.583 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:35:45.583 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:35:45.583 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:35:45.583 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:35:45.583 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:35:45.584 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:35:45.584 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:35:45.584 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:35:45.628 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 10 2024-04-06T09:35:45.629 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 11 2024-04-06T09:35:45.629 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:35:45.629 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=45, tag_tid=6, entry_tid=1], [object_number=44, tag_tid=6, entry_tid=0], [object_number=47, tag_tid=5, entry_tid=22415], [object_number=46, tag_tid=5, entry_tid=22414]]], state=connected] 2024-04-06T09:35:45.629 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=41, tag_tid=5, entry_tid=16309], [object_number=40, tag_tid=5, entry_tid=16308], [object_number=43, tag_tid=5, entry_tid=16307], [object_number=42, tag_tid=5, entry_tid=16306]]], state=connected] 2024-04-06T09:35:45.633 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:35:45.635 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=41, tag_tid=5, entry_tid=16309' 2024-04-06T09:35:45.635 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=45, tag_tid=6, entry_tid=1' -a 'object_number=45, tag_tid=6, entry_tid=1' = 'object_number=41, tag_tid=5, entry_tid=16309' 2024-04-06T09:35:45.635 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=41, tag_tid=5, entry_tid=16309' '!=' 'object_number=41, tag_tid=5, entry_tid=16209' 2024-04-06T09:35:45.635 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:35:45.635 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=41, tag_tid=5, entry_tid=16309' = 'object_number=41, tag_tid=5, entry_tid=16209' 2024-04-06T09:35:45.635 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=41, tag_tid=5, entry_tid=16309' 2024-04-06T09:35:45.635 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=45, tag_tid=6, entry_tid=1' 2024-04-06T09:35:45.635 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:35:45.635 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:35:45.636 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=6 2024-04-06T09:35:45.637 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=41, tag_tid=5, entry_tid=16309' 2024-04-06T09:35:45.637 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:35:45.637 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:35:45.637 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=5 2024-04-06T09:35:45.638 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=45, tag_tid=6, entry_tid=1' 2024-04-06T09:35:45.638 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:35:45.638 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:35:45.640 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:35:45.640 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=41, tag_tid=5, entry_tid=16309' 2024-04-06T09:35:45.640 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:35:45.640 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:35:45.641 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=16309 2024-04-06T09:35:45.641 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 6 = 5 -a 1 -le 16309 2024-04-06T09:35:45.641 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:35:45.641 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:35:45.641 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:35:45.843 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:35:45.843 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:35:45.843 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:35:45.843 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:35:45.843 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:35:45.843 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:35:45.843 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:35:45.843 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:35:45.843 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:35:45.843 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:35:45.843 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:35:45.843 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:35:45.843 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:35:45.844 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:35:45.844 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:35:45.844 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:35:45.844 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:35:45.847 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:35:45.847 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:35:45.847 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:35:45.847 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:35:45.848 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:35:45.848 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:35:45.848 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:35:45.848 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:35:45.848 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:35:45.848 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:35:45.848 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:35:45.848 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:35:45.848 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:35:45.848 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:35:45.848 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:35:45.848 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:35:45.848 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:35:45.848 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:35:45.848 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:35:45.848 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:35:45.849 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:35:45.849 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:35:45.849 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:35:45.849 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:35:45.849 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:35:45.849 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:35:45.849 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:35:45.849 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:35:45.849 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:35:45.849 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:35:45.849 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:35:45.849 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:35:45.872 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:35:45.873 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:35:45.873 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:35:45.873 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:35:46.002 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:35:46.003 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:35:46.003 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:35:46.004 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:35:46.004 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:35:46.004 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:35:46.004 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:35:46.004 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:35:46.004 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:35:46.004 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:35:46.004 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:35:46.004 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:35:46.005 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:35:46.007 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:35:46.007 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:35:46.007 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:35:46.052 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 10 2024-04-06T09:35:46.052 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 11 2024-04-06T09:35:46.052 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:35:46.052 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=45, tag_tid=6, entry_tid=1], [object_number=44, tag_tid=6, entry_tid=0], [object_number=47, tag_tid=5, entry_tid=22415], [object_number=46, tag_tid=5, entry_tid=22414]]], state=connected] 2024-04-06T09:35:46.052 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=42, tag_tid=5, entry_tid=16406], [object_number=41, tag_tid=5, entry_tid=16405], [object_number=40, tag_tid=5, entry_tid=16404], [object_number=43, tag_tid=5, entry_tid=16403]]], state=connected] 2024-04-06T09:35:46.056 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:35:46.057 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=45, tag_tid=6, entry_tid=1' 2024-04-06T09:35:46.058 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:35:46.058 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:35:46.058 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:35:46.058 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:35:46.058 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:35:46.058 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:35:46.058 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:35:46.058 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:35:46.058 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:35:46.058 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:35:46.059 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:35:46.059 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:35:46.059 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:35:46.059 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:35:46.059 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:35:46.104 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 10 2024-04-06T09:35:46.104 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 11 2024-04-06T09:35:46.104 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:35:46.104 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=45, tag_tid=6, entry_tid=1], [object_number=44, tag_tid=6, entry_tid=0], [object_number=47, tag_tid=5, entry_tid=22415], [object_number=46, tag_tid=5, entry_tid=22414]]], state=connected] 2024-04-06T09:35:46.104 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=42, tag_tid=5, entry_tid=16406], [object_number=41, tag_tid=5, entry_tid=16405], [object_number=40, tag_tid=5, entry_tid=16404], [object_number=43, tag_tid=5, entry_tid=16403]]], state=connected] 2024-04-06T09:35:46.108 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:35:46.110 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=42, tag_tid=5, entry_tid=16406' 2024-04-06T09:35:46.110 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=45, tag_tid=6, entry_tid=1' -a 'object_number=45, tag_tid=6, entry_tid=1' = 'object_number=42, tag_tid=5, entry_tid=16406' 2024-04-06T09:35:46.110 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=42, tag_tid=5, entry_tid=16406' '!=' 'object_number=41, tag_tid=5, entry_tid=16309' 2024-04-06T09:35:46.110 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:35:46.110 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=42, tag_tid=5, entry_tid=16406' = 'object_number=41, tag_tid=5, entry_tid=16309' 2024-04-06T09:35:46.110 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=42, tag_tid=5, entry_tid=16406' 2024-04-06T09:35:46.110 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=45, tag_tid=6, entry_tid=1' 2024-04-06T09:35:46.110 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:35:46.110 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:35:46.114 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=6 2024-04-06T09:35:46.114 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=42, tag_tid=5, entry_tid=16406' 2024-04-06T09:35:46.114 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:35:46.114 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:35:46.114 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=5 2024-04-06T09:35:46.115 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=45, tag_tid=6, entry_tid=1' 2024-04-06T09:35:46.115 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:35:46.115 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:35:46.115 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:35:46.115 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=42, tag_tid=5, entry_tid=16406' 2024-04-06T09:35:46.116 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:35:46.116 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:35:46.116 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=16406 2024-04-06T09:35:46.116 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 6 = 5 -a 1 -le 16406 2024-04-06T09:35:46.116 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:35:46.116 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:35:46.117 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:35:46.318 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:35:46.319 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:35:46.319 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:35:46.319 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:35:46.319 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:35:46.319 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:35:46.319 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:35:46.319 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:35:46.319 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:35:46.319 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:35:46.319 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:35:46.319 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:35:46.319 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:35:46.319 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:35:46.320 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:35:46.320 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:35:46.320 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:35:46.323 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:35:46.323 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:35:46.323 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:35:46.323 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:35:46.323 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:35:46.323 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:35:46.323 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:35:46.323 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:35:46.323 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:35:46.323 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:35:46.323 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:35:46.324 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:35:46.324 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:35:46.324 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:35:46.324 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:35:46.324 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:35:46.324 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:35:46.325 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:35:46.325 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:35:46.325 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:35:46.325 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:35:46.325 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:35:46.325 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:35:46.325 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:35:46.325 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:35:46.325 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:35:46.325 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:35:46.325 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:35:46.325 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:35:46.325 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:35:46.325 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:35:46.325 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:35:46.350 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:35:46.350 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:35:46.351 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:35:46.351 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:35:46.481 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:35:46.483 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:35:46.483 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:35:46.483 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:35:46.483 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:35:46.483 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:35:46.483 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:35:46.483 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:35:46.483 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:35:46.483 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:35:46.483 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:35:46.483 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:35:46.483 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:35:46.484 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:35:46.484 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:35:46.484 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:35:46.533 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 10 2024-04-06T09:35:46.533 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 11 2024-04-06T09:35:46.533 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:35:46.533 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=45, tag_tid=6, entry_tid=1], [object_number=44, tag_tid=6, entry_tid=0], [object_number=47, tag_tid=5, entry_tid=22415], [object_number=46, tag_tid=5, entry_tid=22414]]], state=connected] 2024-04-06T09:35:46.533 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=40, tag_tid=5, entry_tid=16504], [object_number=43, tag_tid=5, entry_tid=16503], [object_number=42, tag_tid=5, entry_tid=16502], [object_number=41, tag_tid=5, entry_tid=16501]]], state=connected] 2024-04-06T09:35:46.537 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:35:46.538 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=45, tag_tid=6, entry_tid=1' 2024-04-06T09:35:46.539 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:35:46.539 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:35:46.539 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:35:46.539 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:35:46.539 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:35:46.539 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:35:46.539 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:35:46.539 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:35:46.539 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:35:46.539 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:35:46.539 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:35:46.540 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:35:46.540 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:35:46.540 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:35:46.541 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:35:46.586 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 10 2024-04-06T09:35:46.586 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 11 2024-04-06T09:35:46.586 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:35:46.586 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=45, tag_tid=6, entry_tid=1], [object_number=44, tag_tid=6, entry_tid=0], [object_number=47, tag_tid=5, entry_tid=22415], [object_number=46, tag_tid=5, entry_tid=22414]]], state=connected] 2024-04-06T09:35:46.586 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=40, tag_tid=5, entry_tid=16504], [object_number=43, tag_tid=5, entry_tid=16503], [object_number=42, tag_tid=5, entry_tid=16502], [object_number=41, tag_tid=5, entry_tid=16501]]], state=connected] 2024-04-06T09:35:46.589 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:35:46.591 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=40, tag_tid=5, entry_tid=16504' 2024-04-06T09:35:46.591 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=45, tag_tid=6, entry_tid=1' -a 'object_number=45, tag_tid=6, entry_tid=1' = 'object_number=40, tag_tid=5, entry_tid=16504' 2024-04-06T09:35:46.591 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=40, tag_tid=5, entry_tid=16504' '!=' 'object_number=42, tag_tid=5, entry_tid=16406' 2024-04-06T09:35:46.591 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:35:46.591 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=40, tag_tid=5, entry_tid=16504' = 'object_number=42, tag_tid=5, entry_tid=16406' 2024-04-06T09:35:46.591 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=40, tag_tid=5, entry_tid=16504' 2024-04-06T09:35:46.591 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=45, tag_tid=6, entry_tid=1' 2024-04-06T09:35:46.591 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:35:46.592 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:35:46.592 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=6 2024-04-06T09:35:46.593 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=40, tag_tid=5, entry_tid=16504' 2024-04-06T09:35:46.593 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:35:46.593 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:35:46.594 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=5 2024-04-06T09:35:46.594 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=45, tag_tid=6, entry_tid=1' 2024-04-06T09:35:46.595 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:35:46.595 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:35:46.596 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:35:46.596 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=40, tag_tid=5, entry_tid=16504' 2024-04-06T09:35:46.596 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:35:46.596 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:35:46.597 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=16504 2024-04-06T09:35:46.597 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 6 = 5 -a 1 -le 16504 2024-04-06T09:35:46.598 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:35:46.598 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:35:46.598 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:35:46.799 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:35:46.799 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:35:46.799 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:35:46.799 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:35:46.799 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:35:46.800 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:35:46.800 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:35:46.800 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:35:46.800 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:35:46.800 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:35:46.800 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:35:46.800 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:35:46.800 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:35:46.800 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:35:46.800 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:35:46.801 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:35:46.801 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:35:46.804 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:35:46.804 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:35:46.805 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:35:46.805 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:35:46.805 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:35:46.805 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:35:46.805 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:35:46.805 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:35:46.805 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:35:46.805 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:35:46.805 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:35:46.805 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:35:46.805 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:35:46.805 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:35:46.805 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:35:46.805 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:35:46.805 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:35:46.806 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:35:46.806 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:35:46.806 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:35:46.806 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:35:46.806 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:35:46.806 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:35:46.806 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:35:46.806 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:35:46.806 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:35:46.806 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:35:46.806 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:35:46.806 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:35:46.806 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:35:46.806 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:35:46.806 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:35:46.831 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:35:46.831 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:35:46.831 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:35:46.831 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:35:46.966 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:35:46.967 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:35:46.967 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:35:46.967 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:35:46.967 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:35:46.967 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:35:46.968 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:35:46.968 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:35:46.968 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:35:46.968 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:35:46.968 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:35:46.968 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:35:46.969 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:35:46.970 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:35:46.971 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:35:46.971 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:35:47.021 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 10 2024-04-06T09:35:47.021 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 11 2024-04-06T09:35:47.022 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:35:47.022 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=45, tag_tid=6, entry_tid=1], [object_number=44, tag_tid=6, entry_tid=0], [object_number=47, tag_tid=5, entry_tid=22415], [object_number=46, tag_tid=5, entry_tid=22414]]], state=connected] 2024-04-06T09:35:47.022 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=40, tag_tid=5, entry_tid=16596], [object_number=43, tag_tid=5, entry_tid=16595], [object_number=42, tag_tid=5, entry_tid=16594], [object_number=41, tag_tid=5, entry_tid=16593]]], state=connected] 2024-04-06T09:35:47.025 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:35:47.027 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=45, tag_tid=6, entry_tid=1' 2024-04-06T09:35:47.028 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:35:47.028 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:35:47.028 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:35:47.028 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:35:47.028 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:35:47.028 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:35:47.028 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:35:47.028 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:35:47.028 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:35:47.028 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:35:47.028 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:35:47.028 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:35:47.029 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:35:47.029 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:35:47.029 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:35:47.075 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 10 2024-04-06T09:35:47.075 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 11 2024-04-06T09:35:47.075 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:35:47.076 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=45, tag_tid=6, entry_tid=1], [object_number=44, tag_tid=6, entry_tid=0], [object_number=47, tag_tid=5, entry_tid=22415], [object_number=46, tag_tid=5, entry_tid=22414]]], state=connected] 2024-04-06T09:35:47.076 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=40, tag_tid=5, entry_tid=16596], [object_number=43, tag_tid=5, entry_tid=16595], [object_number=42, tag_tid=5, entry_tid=16594], [object_number=41, tag_tid=5, entry_tid=16593]]], state=connected] 2024-04-06T09:35:47.079 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:35:47.081 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=40, tag_tid=5, entry_tid=16596' 2024-04-06T09:35:47.081 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=45, tag_tid=6, entry_tid=1' -a 'object_number=45, tag_tid=6, entry_tid=1' = 'object_number=40, tag_tid=5, entry_tid=16596' 2024-04-06T09:35:47.081 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=40, tag_tid=5, entry_tid=16596' '!=' 'object_number=40, tag_tid=5, entry_tid=16504' 2024-04-06T09:35:47.081 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:35:47.081 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=40, tag_tid=5, entry_tid=16596' = 'object_number=40, tag_tid=5, entry_tid=16504' 2024-04-06T09:35:47.081 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=40, tag_tid=5, entry_tid=16596' 2024-04-06T09:35:47.082 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=45, tag_tid=6, entry_tid=1' 2024-04-06T09:35:47.082 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:35:47.082 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:35:47.082 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=6 2024-04-06T09:35:47.083 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=40, tag_tid=5, entry_tid=16596' 2024-04-06T09:35:47.083 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:35:47.083 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:35:47.084 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=5 2024-04-06T09:35:47.085 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=45, tag_tid=6, entry_tid=1' 2024-04-06T09:35:47.085 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:35:47.085 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:35:47.086 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:35:47.086 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=40, tag_tid=5, entry_tid=16596' 2024-04-06T09:35:47.087 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:35:47.087 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:35:47.088 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=16596 2024-04-06T09:35:47.088 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 6 = 5 -a 1 -le 16596 2024-04-06T09:35:47.088 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:35:47.088 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:35:47.088 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:35:47.290 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:35:47.290 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:35:47.290 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:35:47.290 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:35:47.290 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:35:47.290 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:35:47.290 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:35:47.290 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:35:47.290 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:35:47.290 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:35:47.291 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:35:47.291 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:35:47.291 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:35:47.291 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:35:47.291 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:35:47.291 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:35:47.292 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:35:47.295 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:35:47.295 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:35:47.295 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:35:47.295 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:35:47.295 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:35:47.295 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:35:47.295 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:35:47.295 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:35:47.295 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:35:47.295 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:35:47.295 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:35:47.295 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:35:47.295 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:35:47.296 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:35:47.296 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:35:47.296 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:35:47.296 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:35:47.296 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:35:47.296 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:35:47.296 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:35:47.296 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:35:47.296 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:35:47.296 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:35:47.296 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:35:47.296 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:35:47.296 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:35:47.296 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:35:47.297 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:35:47.297 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:35:47.297 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:35:47.297 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:35:47.297 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:35:47.320 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:35:47.321 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:35:47.321 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:35:47.321 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:35:47.448 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:35:47.449 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:35:47.450 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:35:47.450 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:35:47.450 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:35:47.450 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:35:47.450 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:35:47.450 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:35:47.450 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:35:47.450 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:35:47.451 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:35:47.451 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:35:47.451 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:35:47.453 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:35:47.453 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:35:47.454 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:35:47.510 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 10 2024-04-06T09:35:47.510 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 11 2024-04-06T09:35:47.510 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:35:47.510 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=45, tag_tid=6, entry_tid=1], [object_number=44, tag_tid=6, entry_tid=0], [object_number=47, tag_tid=5, entry_tid=22415], [object_number=46, tag_tid=5, entry_tid=22414]]], state=connected] 2024-04-06T09:35:47.510 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=43, tag_tid=5, entry_tid=16699], [object_number=42, tag_tid=5, entry_tid=16698], [object_number=41, tag_tid=5, entry_tid=16697], [object_number=40, tag_tid=5, entry_tid=16696]]], state=connected] 2024-04-06T09:35:47.514 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:35:47.515 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=45, tag_tid=6, entry_tid=1' 2024-04-06T09:35:47.516 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:35:47.516 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:35:47.517 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:35:47.517 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:35:47.517 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:35:47.517 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:35:47.517 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:35:47.517 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:35:47.517 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:35:47.517 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:35:47.517 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:35:47.517 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:35:47.518 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:35:47.518 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:35:47.518 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:35:47.563 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 10 2024-04-06T09:35:47.563 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 11 2024-04-06T09:35:47.563 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:35:47.563 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=45, tag_tid=6, entry_tid=1], [object_number=44, tag_tid=6, entry_tid=0], [object_number=47, tag_tid=5, entry_tid=22415], [object_number=46, tag_tid=5, entry_tid=22414]]], state=connected] 2024-04-06T09:35:47.563 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=43, tag_tid=5, entry_tid=16699], [object_number=42, tag_tid=5, entry_tid=16698], [object_number=41, tag_tid=5, entry_tid=16697], [object_number=40, tag_tid=5, entry_tid=16696]]], state=connected] 2024-04-06T09:35:47.567 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:35:47.568 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=43, tag_tid=5, entry_tid=16699' 2024-04-06T09:35:47.569 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=45, tag_tid=6, entry_tid=1' -a 'object_number=45, tag_tid=6, entry_tid=1' = 'object_number=43, tag_tid=5, entry_tid=16699' 2024-04-06T09:35:47.569 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=43, tag_tid=5, entry_tid=16699' '!=' 'object_number=40, tag_tid=5, entry_tid=16596' 2024-04-06T09:35:47.569 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:35:47.569 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=43, tag_tid=5, entry_tid=16699' = 'object_number=40, tag_tid=5, entry_tid=16596' 2024-04-06T09:35:47.569 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=43, tag_tid=5, entry_tid=16699' 2024-04-06T09:35:47.569 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=45, tag_tid=6, entry_tid=1' 2024-04-06T09:35:47.569 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:35:47.570 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:35:47.570 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=6 2024-04-06T09:35:47.571 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=43, tag_tid=5, entry_tid=16699' 2024-04-06T09:35:47.571 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:35:47.571 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:35:47.572 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=5 2024-04-06T09:35:47.572 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=45, tag_tid=6, entry_tid=1' 2024-04-06T09:35:47.573 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:35:47.573 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:35:47.573 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:35:47.574 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=43, tag_tid=5, entry_tid=16699' 2024-04-06T09:35:47.574 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:35:47.574 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:35:47.575 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=16699 2024-04-06T09:35:47.575 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 6 = 5 -a 1 -le 16699 2024-04-06T09:35:47.576 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:35:47.576 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:35:47.576 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:35:47.777 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:35:47.777 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:35:47.777 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:35:47.778 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:35:47.778 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:35:47.778 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:35:47.778 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:35:47.778 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:35:47.778 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:35:47.778 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:35:47.778 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:35:47.778 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:35:47.778 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:35:47.778 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:35:47.778 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:35:47.779 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:35:47.779 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:35:47.782 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:35:47.782 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:35:47.782 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:35:47.782 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:35:47.782 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:35:47.782 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:35:47.782 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:35:47.782 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:35:47.782 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:35:47.782 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:35:47.782 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:35:47.782 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:35:47.783 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:35:47.783 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:35:47.783 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:35:47.783 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:35:47.783 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:35:47.783 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:35:47.783 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:35:47.783 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:35:47.783 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:35:47.783 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:35:47.783 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:35:47.783 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:35:47.783 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:35:47.783 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:35:47.783 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:35:47.783 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:35:47.783 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:35:47.784 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:35:47.784 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:35:47.784 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:35:47.807 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:35:47.807 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:35:47.808 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:35:47.808 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:35:47.935 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:35:47.936 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:35:47.936 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:35:47.936 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:35:47.936 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:35:47.936 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:35:47.936 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:35:47.936 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:35:47.936 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:35:47.936 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:35:47.937 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:35:47.937 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:35:47.937 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:35:47.938 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:35:47.939 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:35:47.939 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:35:47.989 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 10 2024-04-06T09:35:47.989 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 11 2024-04-06T09:35:47.989 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:35:47.990 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=45, tag_tid=6, entry_tid=1], [object_number=44, tag_tid=6, entry_tid=0], [object_number=47, tag_tid=5, entry_tid=22415], [object_number=46, tag_tid=5, entry_tid=22414]]], state=connected] 2024-04-06T09:35:47.990 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=40, tag_tid=5, entry_tid=16800], [object_number=43, tag_tid=5, entry_tid=16799], [object_number=42, tag_tid=5, entry_tid=16798], [object_number=41, tag_tid=5, entry_tid=16797]]], state=connected] 2024-04-06T09:35:47.993 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:35:47.994 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=45, tag_tid=6, entry_tid=1' 2024-04-06T09:35:47.996 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:35:47.996 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:35:47.996 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:35:47.996 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:35:47.996 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:35:47.996 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:35:47.996 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:35:47.996 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:35:47.996 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:35:47.996 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:35:47.996 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:35:47.996 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:35:47.997 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:35:47.997 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:35:47.997 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:35:48.041 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 10 2024-04-06T09:35:48.041 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 11 2024-04-06T09:35:48.041 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:35:48.041 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=45, tag_tid=6, entry_tid=1], [object_number=44, tag_tid=6, entry_tid=0], [object_number=47, tag_tid=5, entry_tid=22415], [object_number=46, tag_tid=5, entry_tid=22414]]], state=connected] 2024-04-06T09:35:48.041 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=40, tag_tid=5, entry_tid=16800], [object_number=43, tag_tid=5, entry_tid=16799], [object_number=42, tag_tid=5, entry_tid=16798], [object_number=41, tag_tid=5, entry_tid=16797]]], state=connected] 2024-04-06T09:35:48.045 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:35:48.047 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=40, tag_tid=5, entry_tid=16800' 2024-04-06T09:35:48.047 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=45, tag_tid=6, entry_tid=1' -a 'object_number=45, tag_tid=6, entry_tid=1' = 'object_number=40, tag_tid=5, entry_tid=16800' 2024-04-06T09:35:48.047 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=40, tag_tid=5, entry_tid=16800' '!=' 'object_number=43, tag_tid=5, entry_tid=16699' 2024-04-06T09:35:48.047 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:35:48.047 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=40, tag_tid=5, entry_tid=16800' = 'object_number=43, tag_tid=5, entry_tid=16699' 2024-04-06T09:35:48.047 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=40, tag_tid=5, entry_tid=16800' 2024-04-06T09:35:48.047 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=45, tag_tid=6, entry_tid=1' 2024-04-06T09:35:48.048 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:35:48.048 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:35:48.048 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=6 2024-04-06T09:35:48.049 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=40, tag_tid=5, entry_tid=16800' 2024-04-06T09:35:48.049 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:35:48.049 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:35:48.050 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=5 2024-04-06T09:35:48.050 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=45, tag_tid=6, entry_tid=1' 2024-04-06T09:35:48.050 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:35:48.050 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:35:48.051 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:35:48.052 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=40, tag_tid=5, entry_tid=16800' 2024-04-06T09:35:48.052 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:35:48.052 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:35:48.053 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=16800 2024-04-06T09:35:48.053 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 6 = 5 -a 1 -le 16800 2024-04-06T09:35:48.053 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:35:48.053 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:35:48.053 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:35:48.255 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:35:48.255 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:35:48.255 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:35:48.255 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:35:48.255 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:35:48.255 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:35:48.255 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:35:48.255 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:35:48.256 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:35:48.256 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:35:48.256 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:35:48.256 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:35:48.256 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:35:48.256 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:35:48.256 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:35:48.256 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:35:48.257 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:35:48.259 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:35:48.259 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:35:48.259 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:35:48.259 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:35:48.259 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:35:48.259 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:35:48.260 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:35:48.260 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:35:48.260 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:35:48.260 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:35:48.260 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:35:48.260 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:35:48.260 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:35:48.260 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:35:48.260 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:35:48.260 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:35:48.260 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:35:48.260 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:35:48.260 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:35:48.260 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:35:48.261 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:35:48.261 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:35:48.261 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:35:48.261 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:35:48.261 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:35:48.261 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:35:48.261 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:35:48.261 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:35:48.261 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:35:48.261 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:35:48.261 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:35:48.261 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:35:48.284 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:35:48.285 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:35:48.285 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:35:48.285 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:35:48.414 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:35:48.414 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:35:48.414 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:35:48.415 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:35:48.415 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:35:48.415 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:35:48.415 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:35:48.415 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:35:48.415 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:35:48.415 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:35:48.416 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:35:48.416 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:35:48.416 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:35:48.418 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:35:48.418 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:35:48.418 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:35:48.469 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 10 2024-04-06T09:35:48.469 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 11 2024-04-06T09:35:48.469 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:35:48.469 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=45, tag_tid=6, entry_tid=1], [object_number=44, tag_tid=6, entry_tid=0], [object_number=47, tag_tid=5, entry_tid=22415], [object_number=46, tag_tid=5, entry_tid=22414]]], state=connected] 2024-04-06T09:35:48.469 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=40, tag_tid=5, entry_tid=16900], [object_number=43, tag_tid=5, entry_tid=16899], [object_number=42, tag_tid=5, entry_tid=16898], [object_number=41, tag_tid=5, entry_tid=16897]]], state=connected] 2024-04-06T09:35:48.473 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:35:48.474 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=45, tag_tid=6, entry_tid=1' 2024-04-06T09:35:48.475 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:35:48.475 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:35:48.475 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:35:48.475 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:35:48.475 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:35:48.475 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:35:48.476 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:35:48.476 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:35:48.476 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:35:48.476 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:35:48.476 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:35:48.476 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:35:48.476 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:35:48.477 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:35:48.477 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:35:48.522 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 10 2024-04-06T09:35:48.522 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 11 2024-04-06T09:35:48.522 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:35:48.522 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=45, tag_tid=6, entry_tid=1], [object_number=44, tag_tid=6, entry_tid=0], [object_number=47, tag_tid=5, entry_tid=22415], [object_number=46, tag_tid=5, entry_tid=22414]]], state=connected] 2024-04-06T09:35:48.522 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=40, tag_tid=5, entry_tid=16900], [object_number=43, tag_tid=5, entry_tid=16899], [object_number=42, tag_tid=5, entry_tid=16898], [object_number=41, tag_tid=5, entry_tid=16897]]], state=connected] 2024-04-06T09:35:48.526 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:35:48.528 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=40, tag_tid=5, entry_tid=16900' 2024-04-06T09:35:48.528 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=45, tag_tid=6, entry_tid=1' -a 'object_number=45, tag_tid=6, entry_tid=1' = 'object_number=40, tag_tid=5, entry_tid=16900' 2024-04-06T09:35:48.528 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=40, tag_tid=5, entry_tid=16900' '!=' 'object_number=40, tag_tid=5, entry_tid=16800' 2024-04-06T09:35:48.528 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:35:48.528 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=40, tag_tid=5, entry_tid=16900' = 'object_number=40, tag_tid=5, entry_tid=16800' 2024-04-06T09:35:48.528 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=40, tag_tid=5, entry_tid=16900' 2024-04-06T09:35:48.528 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=45, tag_tid=6, entry_tid=1' 2024-04-06T09:35:48.528 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:35:48.528 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:35:48.529 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=6 2024-04-06T09:35:48.530 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=40, tag_tid=5, entry_tid=16900' 2024-04-06T09:35:48.530 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:35:48.530 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:35:48.531 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=5 2024-04-06T09:35:48.531 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=45, tag_tid=6, entry_tid=1' 2024-04-06T09:35:48.532 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:35:48.532 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:35:48.532 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:35:48.533 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=40, tag_tid=5, entry_tid=16900' 2024-04-06T09:35:48.533 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:35:48.533 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:35:48.534 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=16900 2024-04-06T09:35:48.534 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 6 = 5 -a 1 -le 16900 2024-04-06T09:35:48.535 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:35:48.535 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:35:48.535 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:35:48.736 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:35:48.736 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:35:48.736 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:35:48.737 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:35:48.737 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:35:48.737 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:35:48.737 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:35:48.737 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:35:48.737 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:35:48.737 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:35:48.737 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:35:48.737 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:35:48.737 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:35:48.737 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:35:48.737 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:35:48.738 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:35:48.738 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:35:48.740 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:35:48.740 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:35:48.740 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:35:48.741 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:35:48.741 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:35:48.741 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:35:48.741 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:35:48.741 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:35:48.741 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:35:48.741 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:35:48.741 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:35:48.741 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:35:48.741 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:35:48.741 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:35:48.741 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:35:48.741 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:35:48.741 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:35:48.741 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:35:48.742 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:35:48.742 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:35:48.742 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:35:48.742 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:35:48.742 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:35:48.742 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:35:48.742 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:35:48.742 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:35:48.742 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:35:48.742 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:35:48.742 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:35:48.742 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:35:48.742 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:35:48.743 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:35:48.767 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:35:48.767 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:35:48.767 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:35:48.767 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:35:48.897 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:35:48.898 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:35:48.898 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:35:48.898 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:35:48.899 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:35:48.899 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:35:48.899 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:35:48.899 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:35:48.899 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:35:48.899 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:35:48.900 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:35:48.900 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:35:48.900 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:35:48.901 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:35:48.901 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:35:48.902 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:35:48.946 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 10 2024-04-06T09:35:48.946 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 11 2024-04-06T09:35:48.946 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:35:48.947 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=45, tag_tid=6, entry_tid=1], [object_number=44, tag_tid=6, entry_tid=0], [object_number=47, tag_tid=5, entry_tid=22415], [object_number=46, tag_tid=5, entry_tid=22414]]], state=connected] 2024-04-06T09:35:48.947 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=43, tag_tid=5, entry_tid=16995], [object_number=42, tag_tid=5, entry_tid=16994], [object_number=41, tag_tid=5, entry_tid=16993], [object_number=40, tag_tid=5, entry_tid=16992]]], state=connected] 2024-04-06T09:35:48.950 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:35:48.951 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=45, tag_tid=6, entry_tid=1' 2024-04-06T09:35:48.952 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:35:48.953 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:35:48.953 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:35:48.953 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:35:48.953 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:35:48.953 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:35:48.953 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:35:48.953 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:35:48.953 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:35:48.953 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:35:48.953 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:35:48.953 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:35:48.954 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:35:48.954 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:35:48.954 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:35:48.999 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 10 2024-04-06T09:35:48.999 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 11 2024-04-06T09:35:48.999 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:35:48.999 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=45, tag_tid=6, entry_tid=1], [object_number=44, tag_tid=6, entry_tid=0], [object_number=47, tag_tid=5, entry_tid=22415], [object_number=46, tag_tid=5, entry_tid=22414]]], state=connected] 2024-04-06T09:35:48.999 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=43, tag_tid=5, entry_tid=16995], [object_number=42, tag_tid=5, entry_tid=16994], [object_number=41, tag_tid=5, entry_tid=16993], [object_number=40, tag_tid=5, entry_tid=16992]]], state=connected] 2024-04-06T09:35:49.003 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:35:49.005 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=43, tag_tid=5, entry_tid=16995' 2024-04-06T09:35:49.005 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=45, tag_tid=6, entry_tid=1' -a 'object_number=45, tag_tid=6, entry_tid=1' = 'object_number=43, tag_tid=5, entry_tid=16995' 2024-04-06T09:35:49.005 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=43, tag_tid=5, entry_tid=16995' '!=' 'object_number=40, tag_tid=5, entry_tid=16900' 2024-04-06T09:35:49.005 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:35:49.005 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=43, tag_tid=5, entry_tid=16995' = 'object_number=40, tag_tid=5, entry_tid=16900' 2024-04-06T09:35:49.005 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=43, tag_tid=5, entry_tid=16995' 2024-04-06T09:35:49.005 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=45, tag_tid=6, entry_tid=1' 2024-04-06T09:35:49.005 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:35:49.005 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:35:49.006 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=6 2024-04-06T09:35:49.007 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=43, tag_tid=5, entry_tid=16995' 2024-04-06T09:35:49.007 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:35:49.007 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:35:49.008 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=5 2024-04-06T09:35:49.008 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=45, tag_tid=6, entry_tid=1' 2024-04-06T09:35:49.008 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:35:49.008 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:35:49.009 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:35:49.010 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=43, tag_tid=5, entry_tid=16995' 2024-04-06T09:35:49.010 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:35:49.010 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:35:49.011 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=16995 2024-04-06T09:35:49.012 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 6 = 5 -a 1 -le 16995 2024-04-06T09:35:49.012 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:35:49.012 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:35:49.012 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:35:49.213 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:35:49.213 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:35:49.213 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:35:49.213 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:35:49.213 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:35:49.213 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:35:49.213 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:35:49.214 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:35:49.214 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:35:49.214 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:35:49.214 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:35:49.214 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:35:49.214 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:35:49.214 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:35:49.214 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:35:49.215 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:35:49.215 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:35:49.218 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:35:49.219 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:35:49.219 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:35:49.219 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:35:49.219 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:35:49.219 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:35:49.219 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:35:49.219 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:35:49.219 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:35:49.219 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:35:49.219 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:35:49.219 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:35:49.219 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:35:49.219 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:35:49.219 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:35:49.219 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:35:49.220 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:35:49.220 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:35:49.220 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:35:49.220 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:35:49.220 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:35:49.220 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:35:49.220 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:35:49.220 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:35:49.220 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:35:49.220 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:35:49.220 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:35:49.220 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:35:49.220 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:35:49.220 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:35:49.220 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:35:49.220 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:35:49.243 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:35:49.244 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:35:49.244 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:35:49.244 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:35:49.377 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:35:49.379 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:35:49.379 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:35:49.379 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:35:49.379 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:35:49.379 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:35:49.379 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:35:49.379 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:35:49.379 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:35:49.379 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:35:49.379 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:35:49.380 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:35:49.380 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:35:49.382 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:35:49.382 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:35:49.383 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:35:49.433 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 10 2024-04-06T09:35:49.433 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 11 2024-04-06T09:35:49.433 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:35:49.433 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=45, tag_tid=6, entry_tid=1], [object_number=44, tag_tid=6, entry_tid=0], [object_number=47, tag_tid=5, entry_tid=22415], [object_number=46, tag_tid=5, entry_tid=22414]]], state=connected] 2024-04-06T09:35:49.433 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=40, tag_tid=5, entry_tid=17080], [object_number=43, tag_tid=5, entry_tid=17079], [object_number=42, tag_tid=5, entry_tid=17078], [object_number=41, tag_tid=5, entry_tid=17077]]], state=connected] 2024-04-06T09:35:49.437 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:35:49.438 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=45, tag_tid=6, entry_tid=1' 2024-04-06T09:35:49.439 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:35:49.439 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:35:49.440 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:35:49.440 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:35:49.440 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:35:49.440 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:35:49.440 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:35:49.440 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:35:49.440 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:35:49.440 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:35:49.440 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:35:49.440 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:35:49.441 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:35:49.441 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:35:49.441 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:35:49.487 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 10 2024-04-06T09:35:49.487 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 11 2024-04-06T09:35:49.487 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:35:49.487 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=45, tag_tid=6, entry_tid=1], [object_number=44, tag_tid=6, entry_tid=0], [object_number=47, tag_tid=5, entry_tid=22415], [object_number=46, tag_tid=5, entry_tid=22414]]], state=connected] 2024-04-06T09:35:49.487 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=40, tag_tid=5, entry_tid=17080], [object_number=43, tag_tid=5, entry_tid=17079], [object_number=42, tag_tid=5, entry_tid=17078], [object_number=41, tag_tid=5, entry_tid=17077]]], state=connected] 2024-04-06T09:35:49.491 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:35:49.492 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=40, tag_tid=5, entry_tid=17080' 2024-04-06T09:35:49.492 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=45, tag_tid=6, entry_tid=1' -a 'object_number=45, tag_tid=6, entry_tid=1' = 'object_number=40, tag_tid=5, entry_tid=17080' 2024-04-06T09:35:49.492 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=40, tag_tid=5, entry_tid=17080' '!=' 'object_number=43, tag_tid=5, entry_tid=16995' 2024-04-06T09:35:49.492 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:35:49.493 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=40, tag_tid=5, entry_tid=17080' = 'object_number=43, tag_tid=5, entry_tid=16995' 2024-04-06T09:35:49.493 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=40, tag_tid=5, entry_tid=17080' 2024-04-06T09:35:49.493 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=45, tag_tid=6, entry_tid=1' 2024-04-06T09:35:49.493 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:35:49.493 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:35:49.494 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=6 2024-04-06T09:35:49.494 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=40, tag_tid=5, entry_tid=17080' 2024-04-06T09:35:49.495 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:35:49.495 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:35:49.495 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=5 2024-04-06T09:35:49.496 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=45, tag_tid=6, entry_tid=1' 2024-04-06T09:35:49.496 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:35:49.496 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:35:49.497 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:35:49.498 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=40, tag_tid=5, entry_tid=17080' 2024-04-06T09:35:49.498 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:35:49.498 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:35:49.499 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=17080 2024-04-06T09:35:49.499 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 6 = 5 -a 1 -le 17080 2024-04-06T09:35:49.499 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:35:49.499 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:35:49.499 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:35:49.701 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:35:49.701 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:35:49.701 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:35:49.701 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:35:49.701 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:35:49.701 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:35:49.701 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:35:49.701 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:35:49.701 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:35:49.702 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:35:49.702 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:35:49.702 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:35:49.702 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:35:49.702 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:35:49.702 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:35:49.702 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:35:49.703 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:35:49.705 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:35:49.705 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:35:49.705 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:35:49.705 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:35:49.705 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:35:49.705 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:35:49.705 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:35:49.705 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:35:49.705 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:35:49.705 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:35:49.705 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:35:49.705 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:35:49.705 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:35:49.705 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:35:49.706 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:35:49.706 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:35:49.707 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:35:49.707 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:35:49.707 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:35:49.707 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:35:49.707 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:35:49.707 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:35:49.707 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:35:49.707 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:35:49.707 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:35:49.707 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:35:49.707 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:35:49.707 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:35:49.707 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:35:49.708 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:35:49.708 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:35:49.708 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:35:49.731 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:35:49.731 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:35:49.731 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:35:49.731 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:35:49.862 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:35:49.863 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:35:49.863 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:35:49.863 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:35:49.863 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:35:49.863 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:35:49.864 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:35:49.864 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:35:49.864 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:35:49.864 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:35:49.864 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:35:49.864 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:35:49.864 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:35:49.866 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:35:49.866 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:35:49.867 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:35:49.917 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 10 2024-04-06T09:35:49.917 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 11 2024-04-06T09:35:49.917 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:35:49.917 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=45, tag_tid=6, entry_tid=1], [object_number=44, tag_tid=6, entry_tid=0], [object_number=47, tag_tid=5, entry_tid=22415], [object_number=46, tag_tid=5, entry_tid=22414]]], state=connected] 2024-04-06T09:35:49.917 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=41, tag_tid=5, entry_tid=17177], [object_number=40, tag_tid=5, entry_tid=17176], [object_number=43, tag_tid=5, entry_tid=17175], [object_number=42, tag_tid=5, entry_tid=17174]]], state=connected] 2024-04-06T09:35:49.921 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:35:49.922 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=45, tag_tid=6, entry_tid=1' 2024-04-06T09:35:49.923 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:35:49.923 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:35:49.923 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:35:49.923 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:35:49.924 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:35:49.924 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:35:49.924 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:35:49.924 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:35:49.924 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:35:49.924 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:35:49.924 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:35:49.924 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:35:49.925 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:35:49.925 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:35:49.925 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:35:49.970 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 10 2024-04-06T09:35:49.970 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 11 2024-04-06T09:35:49.970 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:35:49.971 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=45, tag_tid=6, entry_tid=1], [object_number=44, tag_tid=6, entry_tid=0], [object_number=47, tag_tid=5, entry_tid=22415], [object_number=46, tag_tid=5, entry_tid=22414]]], state=connected] 2024-04-06T09:35:49.971 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=41, tag_tid=5, entry_tid=17177], [object_number=40, tag_tid=5, entry_tid=17176], [object_number=43, tag_tid=5, entry_tid=17175], [object_number=42, tag_tid=5, entry_tid=17174]]], state=connected] 2024-04-06T09:35:49.974 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:35:49.976 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=41, tag_tid=5, entry_tid=17177' 2024-04-06T09:35:49.976 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=45, tag_tid=6, entry_tid=1' -a 'object_number=45, tag_tid=6, entry_tid=1' = 'object_number=41, tag_tid=5, entry_tid=17177' 2024-04-06T09:35:49.976 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=41, tag_tid=5, entry_tid=17177' '!=' 'object_number=40, tag_tid=5, entry_tid=17080' 2024-04-06T09:35:49.976 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:35:49.976 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=41, tag_tid=5, entry_tid=17177' = 'object_number=40, tag_tid=5, entry_tid=17080' 2024-04-06T09:35:49.976 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=41, tag_tid=5, entry_tid=17177' 2024-04-06T09:35:49.977 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=45, tag_tid=6, entry_tid=1' 2024-04-06T09:35:49.977 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:35:49.977 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:35:49.977 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=6 2024-04-06T09:35:49.978 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=41, tag_tid=5, entry_tid=17177' 2024-04-06T09:35:49.978 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:35:49.978 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:35:49.979 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=5 2024-04-06T09:35:49.980 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=45, tag_tid=6, entry_tid=1' 2024-04-06T09:35:49.980 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:35:49.980 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:35:49.981 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:35:49.981 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=41, tag_tid=5, entry_tid=17177' 2024-04-06T09:35:49.981 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:35:49.981 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:35:49.982 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=17177 2024-04-06T09:35:49.983 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 6 = 5 -a 1 -le 17177 2024-04-06T09:35:49.983 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:35:49.983 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:35:49.983 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:35:50.184 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:35:50.184 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:35:50.185 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:35:50.185 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:35:50.185 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:35:50.185 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:35:50.185 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:35:50.185 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:35:50.185 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:35:50.185 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:35:50.185 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:35:50.185 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:35:50.185 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:35:50.185 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:35:50.185 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:35:50.186 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:35:50.187 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:35:50.190 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:35:50.190 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:35:50.190 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:35:50.190 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:35:50.190 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:35:50.190 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:35:50.190 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:35:50.190 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:35:50.190 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:35:50.190 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:35:50.190 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:35:50.190 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:35:50.191 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:35:50.191 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:35:50.191 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:35:50.191 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:35:50.191 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:35:50.191 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:35:50.191 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:35:50.191 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:35:50.191 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:35:50.191 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:35:50.191 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:35:50.191 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:35:50.191 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:35:50.191 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:35:50.192 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:35:50.192 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:35:50.192 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:35:50.192 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:35:50.192 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:35:50.192 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:35:50.213 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:35:50.214 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:35:50.214 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:35:50.214 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:35:50.341 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:35:50.342 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:35:50.342 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:35:50.342 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:35:50.342 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:35:50.342 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:35:50.342 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:35:50.342 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:35:50.342 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:35:50.342 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:35:50.343 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:35:50.343 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:35:50.343 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:35:50.345 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:35:50.346 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:35:50.346 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:35:50.395 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 10 2024-04-06T09:35:50.395 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 11 2024-04-06T09:35:50.395 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:35:50.395 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=45, tag_tid=6, entry_tid=1], [object_number=44, tag_tid=6, entry_tid=0], [object_number=47, tag_tid=5, entry_tid=22415], [object_number=46, tag_tid=5, entry_tid=22414]]], state=connected] 2024-04-06T09:35:50.395 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=43, tag_tid=5, entry_tid=17279], [object_number=42, tag_tid=5, entry_tid=17278], [object_number=41, tag_tid=5, entry_tid=17277], [object_number=40, tag_tid=5, entry_tid=17276]]], state=connected] 2024-04-06T09:35:50.398 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:35:50.401 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=45, tag_tid=6, entry_tid=1' 2024-04-06T09:35:50.401 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:35:50.401 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:35:50.401 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:35:50.401 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:35:50.401 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:35:50.401 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:35:50.401 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:35:50.401 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:35:50.401 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:35:50.401 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:35:50.401 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:35:50.401 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:35:50.402 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:35:50.402 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:35:50.402 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:35:50.446 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 10 2024-04-06T09:35:50.446 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 11 2024-04-06T09:35:50.446 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:35:50.447 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=45, tag_tid=6, entry_tid=1], [object_number=44, tag_tid=6, entry_tid=0], [object_number=47, tag_tid=5, entry_tid=22415], [object_number=46, tag_tid=5, entry_tid=22414]]], state=connected] 2024-04-06T09:35:50.447 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=43, tag_tid=5, entry_tid=17279], [object_number=42, tag_tid=5, entry_tid=17278], [object_number=41, tag_tid=5, entry_tid=17277], [object_number=40, tag_tid=5, entry_tid=17276]]], state=connected] 2024-04-06T09:35:50.450 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:35:50.452 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=43, tag_tid=5, entry_tid=17279' 2024-04-06T09:35:50.452 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=45, tag_tid=6, entry_tid=1' -a 'object_number=45, tag_tid=6, entry_tid=1' = 'object_number=43, tag_tid=5, entry_tid=17279' 2024-04-06T09:35:50.452 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=43, tag_tid=5, entry_tid=17279' '!=' 'object_number=41, tag_tid=5, entry_tid=17177' 2024-04-06T09:35:50.452 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:35:50.452 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=43, tag_tid=5, entry_tid=17279' = 'object_number=41, tag_tid=5, entry_tid=17177' 2024-04-06T09:35:50.452 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=43, tag_tid=5, entry_tid=17279' 2024-04-06T09:35:50.452 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=45, tag_tid=6, entry_tid=1' 2024-04-06T09:35:50.452 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:35:50.452 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:35:50.453 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=6 2024-04-06T09:35:50.453 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=43, tag_tid=5, entry_tid=17279' 2024-04-06T09:35:50.453 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:35:50.454 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:35:50.454 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=5 2024-04-06T09:35:50.455 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=45, tag_tid=6, entry_tid=1' 2024-04-06T09:35:50.455 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:35:50.455 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:35:50.456 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:35:50.457 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=43, tag_tid=5, entry_tid=17279' 2024-04-06T09:35:50.457 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:35:50.457 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:35:50.458 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=17279 2024-04-06T09:35:50.458 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 6 = 5 -a 1 -le 17279 2024-04-06T09:35:50.458 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:35:50.458 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:35:50.458 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:35:50.660 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:35:50.660 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:35:50.660 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:35:50.660 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:35:50.660 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:35:50.660 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:35:50.661 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:35:50.661 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:35:50.661 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:35:50.661 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:35:50.661 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:35:50.661 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:35:50.661 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:35:50.661 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:35:50.661 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:35:50.661 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:35:50.662 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:35:50.664 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:35:50.664 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:35:50.664 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:35:50.664 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:35:50.665 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:35:50.665 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:35:50.665 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:35:50.665 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:35:50.665 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:35:50.665 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:35:50.665 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:35:50.665 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:35:50.665 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:35:50.665 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:35:50.665 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:35:50.665 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:35:50.665 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:35:50.665 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:35:50.665 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:35:50.665 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:35:50.666 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:35:50.666 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:35:50.666 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:35:50.666 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:35:50.666 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:35:50.666 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:35:50.666 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:35:50.666 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:35:50.666 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:35:50.666 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:35:50.666 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:35:50.666 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:35:50.689 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:35:50.689 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:35:50.689 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:35:50.689 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:35:50.814 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:35:50.815 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:35:50.815 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:35:50.815 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:35:50.815 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:35:50.815 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:35:50.815 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:35:50.815 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:35:50.816 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:35:50.816 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:35:50.816 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:35:50.816 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:35:50.816 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:35:50.817 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:35:50.817 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:35:50.817 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:35:50.862 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 10 2024-04-06T09:35:50.862 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 11 2024-04-06T09:35:50.862 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:35:50.862 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=45, tag_tid=6, entry_tid=1], [object_number=44, tag_tid=6, entry_tid=0], [object_number=47, tag_tid=5, entry_tid=22415], [object_number=46, tag_tid=5, entry_tid=22414]]], state=connected] 2024-04-06T09:35:50.862 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=40, tag_tid=5, entry_tid=17352], [object_number=43, tag_tid=5, entry_tid=17351], [object_number=42, tag_tid=5, entry_tid=17350], [object_number=41, tag_tid=5, entry_tid=17349]]], state=connected] 2024-04-06T09:35:50.866 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:35:50.867 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=45, tag_tid=6, entry_tid=1' 2024-04-06T09:35:50.868 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:35:50.868 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:35:50.868 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:35:50.868 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:35:50.868 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:35:50.868 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:35:50.868 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:35:50.868 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:35:50.868 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:35:50.868 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:35:50.868 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:35:50.868 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:35:50.869 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:35:50.869 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:35:50.869 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:35:50.913 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 10 2024-04-06T09:35:50.913 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 11 2024-04-06T09:35:50.913 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:35:50.913 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=45, tag_tid=6, entry_tid=1], [object_number=44, tag_tid=6, entry_tid=0], [object_number=47, tag_tid=5, entry_tid=22415], [object_number=46, tag_tid=5, entry_tid=22414]]], state=connected] 2024-04-06T09:35:50.913 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=40, tag_tid=5, entry_tid=17352], [object_number=43, tag_tid=5, entry_tid=17351], [object_number=42, tag_tid=5, entry_tid=17350], [object_number=41, tag_tid=5, entry_tid=17349]]], state=connected] 2024-04-06T09:35:50.916 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:35:50.918 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=40, tag_tid=5, entry_tid=17352' 2024-04-06T09:35:50.918 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=45, tag_tid=6, entry_tid=1' -a 'object_number=45, tag_tid=6, entry_tid=1' = 'object_number=40, tag_tid=5, entry_tid=17352' 2024-04-06T09:35:50.918 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=40, tag_tid=5, entry_tid=17352' '!=' 'object_number=43, tag_tid=5, entry_tid=17279' 2024-04-06T09:35:50.918 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:35:50.918 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=40, tag_tid=5, entry_tid=17352' = 'object_number=43, tag_tid=5, entry_tid=17279' 2024-04-06T09:35:50.918 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=40, tag_tid=5, entry_tid=17352' 2024-04-06T09:35:50.919 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=45, tag_tid=6, entry_tid=1' 2024-04-06T09:35:50.919 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:35:50.919 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:35:50.919 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=6 2024-04-06T09:35:50.920 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=40, tag_tid=5, entry_tid=17352' 2024-04-06T09:35:50.920 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:35:50.920 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:35:50.921 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=5 2024-04-06T09:35:50.922 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=45, tag_tid=6, entry_tid=1' 2024-04-06T09:35:50.922 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:35:50.922 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:35:50.923 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:35:50.923 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=40, tag_tid=5, entry_tid=17352' 2024-04-06T09:35:50.923 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:35:50.923 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:35:50.925 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=17352 2024-04-06T09:35:50.925 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 6 = 5 -a 1 -le 17352 2024-04-06T09:35:50.925 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:35:50.925 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:35:50.925 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:35:51.126 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:35:51.126 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:35:51.126 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:35:51.126 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:35:51.127 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:35:51.127 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:35:51.127 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:35:51.127 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:35:51.127 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:35:51.127 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:35:51.127 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:35:51.127 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:35:51.127 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:35:51.127 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:35:51.127 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:35:51.128 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:35:51.128 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:35:51.132 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:35:51.132 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:35:51.132 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:35:51.132 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:35:51.132 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:35:51.132 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:35:51.132 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:35:51.132 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:35:51.132 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:35:51.132 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:35:51.132 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:35:51.132 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:35:51.132 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:35:51.133 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:35:51.133 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:35:51.133 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:35:51.133 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:35:51.133 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:35:51.133 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:35:51.133 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:35:51.133 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:35:51.133 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:35:51.133 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:35:51.133 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:35:51.133 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:35:51.133 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:35:51.133 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:35:51.133 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:35:51.133 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:35:51.134 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:35:51.134 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:35:51.134 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:35:51.156 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:35:51.157 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:35:51.157 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:35:51.157 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:35:51.284 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:35:51.285 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:35:51.285 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:35:51.286 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:35:51.286 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:35:51.286 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:35:51.286 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:35:51.286 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:35:51.286 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:35:51.286 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:35:51.286 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:35:51.287 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:35:51.287 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:35:51.289 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:35:51.289 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:35:51.290 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:35:51.343 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 10 2024-04-06T09:35:51.343 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 11 2024-04-06T09:35:51.343 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:35:51.343 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=45, tag_tid=6, entry_tid=1], [object_number=44, tag_tid=6, entry_tid=0], [object_number=47, tag_tid=5, entry_tid=22415], [object_number=46, tag_tid=5, entry_tid=22414]]], state=connected] 2024-04-06T09:35:51.343 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=43, tag_tid=5, entry_tid=17431], [object_number=42, tag_tid=5, entry_tid=17430], [object_number=41, tag_tid=5, entry_tid=17429], [object_number=40, tag_tid=5, entry_tid=17428]]], state=connected] 2024-04-06T09:35:51.347 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:35:51.348 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=45, tag_tid=6, entry_tid=1' 2024-04-06T09:35:51.349 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:35:51.349 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:35:51.349 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:35:51.349 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:35:51.349 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:35:51.349 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:35:51.349 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:35:51.349 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:35:51.349 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:35:51.350 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:35:51.350 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:35:51.350 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:35:51.350 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:35:51.350 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:35:51.351 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:35:51.394 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 10 2024-04-06T09:35:51.394 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 11 2024-04-06T09:35:51.395 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:35:51.395 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=45, tag_tid=6, entry_tid=1], [object_number=44, tag_tid=6, entry_tid=0], [object_number=47, tag_tid=5, entry_tid=22415], [object_number=46, tag_tid=5, entry_tid=22414]]], state=connected] 2024-04-06T09:35:51.395 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=43, tag_tid=5, entry_tid=17431], [object_number=42, tag_tid=5, entry_tid=17430], [object_number=41, tag_tid=5, entry_tid=17429], [object_number=40, tag_tid=5, entry_tid=17428]]], state=connected] 2024-04-06T09:35:51.397 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:35:51.399 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=43, tag_tid=5, entry_tid=17431' 2024-04-06T09:35:51.400 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=45, tag_tid=6, entry_tid=1' -a 'object_number=45, tag_tid=6, entry_tid=1' = 'object_number=43, tag_tid=5, entry_tid=17431' 2024-04-06T09:35:51.400 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=43, tag_tid=5, entry_tid=17431' '!=' 'object_number=40, tag_tid=5, entry_tid=17352' 2024-04-06T09:35:51.400 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:35:51.400 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=43, tag_tid=5, entry_tid=17431' = 'object_number=40, tag_tid=5, entry_tid=17352' 2024-04-06T09:35:51.400 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=43, tag_tid=5, entry_tid=17431' 2024-04-06T09:35:51.400 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=45, tag_tid=6, entry_tid=1' 2024-04-06T09:35:51.400 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:35:51.400 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:35:51.401 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=6 2024-04-06T09:35:51.401 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=43, tag_tid=5, entry_tid=17431' 2024-04-06T09:35:51.401 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:35:51.401 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:35:51.402 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=5 2024-04-06T09:35:51.403 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=45, tag_tid=6, entry_tid=1' 2024-04-06T09:35:51.403 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:35:51.403 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:35:51.404 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:35:51.404 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=43, tag_tid=5, entry_tid=17431' 2024-04-06T09:35:51.404 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:35:51.405 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:35:51.406 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=17431 2024-04-06T09:35:51.406 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 6 = 5 -a 1 -le 17431 2024-04-06T09:35:51.406 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:35:51.406 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:35:51.406 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:35:51.607 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:35:51.607 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:35:51.607 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:35:51.607 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:35:51.608 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:35:51.608 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:35:51.608 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:35:51.608 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:35:51.608 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:35:51.608 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:35:51.608 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:35:51.608 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:35:51.608 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:35:51.608 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:35:51.608 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:35:51.609 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:35:51.609 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:35:51.612 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:35:51.612 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:35:51.612 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:35:51.612 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:35:51.612 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:35:51.612 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:35:51.612 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:35:51.613 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:35:51.613 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:35:51.613 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:35:51.613 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:35:51.613 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:35:51.613 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:35:51.613 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:35:51.613 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:35:51.613 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:35:51.613 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:35:51.613 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:35:51.613 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:35:51.613 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:35:51.613 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:35:51.613 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:35:51.614 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:35:51.614 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:35:51.614 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:35:51.614 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:35:51.614 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:35:51.614 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:35:51.614 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:35:51.614 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:35:51.614 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:35:51.614 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:35:51.637 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:35:51.637 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:35:51.637 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:35:51.637 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:35:51.765 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:35:51.766 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:35:51.766 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:35:51.766 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:35:51.766 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:35:51.766 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:35:51.766 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:35:51.766 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:35:51.767 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:35:51.767 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:35:51.767 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:35:51.767 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:35:51.767 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:35:51.769 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:35:51.769 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:35:51.769 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:35:51.824 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 10 2024-04-06T09:35:51.824 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 11 2024-04-06T09:35:51.824 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:35:51.824 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=45, tag_tid=6, entry_tid=1], [object_number=44, tag_tid=6, entry_tid=0], [object_number=47, tag_tid=5, entry_tid=22415], [object_number=46, tag_tid=5, entry_tid=22414]]], state=connected] 2024-04-06T09:35:51.824 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=43, tag_tid=5, entry_tid=17515], [object_number=42, tag_tid=5, entry_tid=17514], [object_number=41, tag_tid=5, entry_tid=17513], [object_number=40, tag_tid=5, entry_tid=17512]]], state=connected] 2024-04-06T09:35:51.828 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:35:51.829 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=45, tag_tid=6, entry_tid=1' 2024-04-06T09:35:51.830 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:35:51.830 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:35:51.830 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:35:51.830 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:35:51.830 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:35:51.830 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:35:51.830 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:35:51.831 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:35:51.831 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:35:51.831 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:35:51.831 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:35:51.831 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:35:51.831 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:35:51.832 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:35:51.832 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:35:51.877 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 10 2024-04-06T09:35:51.877 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 11 2024-04-06T09:35:51.877 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:35:51.877 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=45, tag_tid=6, entry_tid=1], [object_number=44, tag_tid=6, entry_tid=0], [object_number=47, tag_tid=5, entry_tid=22415], [object_number=46, tag_tid=5, entry_tid=22414]]], state=connected] 2024-04-06T09:35:51.877 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=43, tag_tid=5, entry_tid=17515], [object_number=42, tag_tid=5, entry_tid=17514], [object_number=41, tag_tid=5, entry_tid=17513], [object_number=40, tag_tid=5, entry_tid=17512]]], state=connected] 2024-04-06T09:35:51.880 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:35:51.882 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=43, tag_tid=5, entry_tid=17515' 2024-04-06T09:35:51.882 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=45, tag_tid=6, entry_tid=1' -a 'object_number=45, tag_tid=6, entry_tid=1' = 'object_number=43, tag_tid=5, entry_tid=17515' 2024-04-06T09:35:51.882 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=43, tag_tid=5, entry_tid=17515' '!=' 'object_number=43, tag_tid=5, entry_tid=17431' 2024-04-06T09:35:51.883 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:35:51.883 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=43, tag_tid=5, entry_tid=17515' = 'object_number=43, tag_tid=5, entry_tid=17431' 2024-04-06T09:35:51.883 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=43, tag_tid=5, entry_tid=17515' 2024-04-06T09:35:51.883 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=45, tag_tid=6, entry_tid=1' 2024-04-06T09:35:51.883 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:35:51.883 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:35:51.884 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=6 2024-04-06T09:35:51.884 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=43, tag_tid=5, entry_tid=17515' 2024-04-06T09:35:51.884 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:35:51.885 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:35:51.885 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=5 2024-04-06T09:35:51.886 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=45, tag_tid=6, entry_tid=1' 2024-04-06T09:35:51.886 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:35:51.886 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:35:51.887 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:35:51.888 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=43, tag_tid=5, entry_tid=17515' 2024-04-06T09:35:51.888 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:35:51.888 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:35:51.889 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=17515 2024-04-06T09:35:51.889 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 6 = 5 -a 1 -le 17515 2024-04-06T09:35:51.889 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:35:51.889 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:35:51.889 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:35:52.091 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:35:52.091 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:35:52.091 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:35:52.091 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:35:52.091 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:35:52.091 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:35:52.092 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:35:52.092 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:35:52.092 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:35:52.092 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:35:52.092 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:35:52.092 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:35:52.092 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:35:52.092 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:35:52.092 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:35:52.092 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:35:52.093 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:35:52.096 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:35:52.096 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:35:52.096 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:35:52.096 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:35:52.096 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:35:52.096 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:35:52.097 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:35:52.097 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:35:52.097 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:35:52.097 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:35:52.097 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:35:52.097 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:35:52.097 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:35:52.097 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:35:52.097 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:35:52.097 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:35:52.097 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:35:52.097 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:35:52.097 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:35:52.098 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:35:52.098 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:35:52.098 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:35:52.098 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:35:52.098 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:35:52.098 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:35:52.098 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:35:52.098 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:35:52.098 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:35:52.098 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:35:52.098 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:35:52.098 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:35:52.098 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:35:52.121 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:35:52.122 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:35:52.122 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:35:52.122 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:35:52.252 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:35:52.253 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:35:52.253 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:35:52.253 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:35:52.253 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:35:52.253 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:35:52.254 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:35:52.254 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:35:52.254 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:35:52.254 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:35:52.254 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:35:52.254 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:35:52.254 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:35:52.256 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:35:52.256 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:35:52.256 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:35:52.301 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 10 2024-04-06T09:35:52.301 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 11 2024-04-06T09:35:52.301 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:35:52.301 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=45, tag_tid=6, entry_tid=1], [object_number=44, tag_tid=6, entry_tid=0], [object_number=47, tag_tid=5, entry_tid=22415], [object_number=46, tag_tid=5, entry_tid=22414]]], state=connected] 2024-04-06T09:35:52.301 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=43, tag_tid=5, entry_tid=17595], [object_number=42, tag_tid=5, entry_tid=17594], [object_number=41, tag_tid=5, entry_tid=17593], [object_number=40, tag_tid=5, entry_tid=17592]]], state=connected] 2024-04-06T09:35:52.304 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:35:52.306 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=45, tag_tid=6, entry_tid=1' 2024-04-06T09:35:52.307 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:35:52.307 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:35:52.307 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:35:52.307 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:35:52.307 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:35:52.307 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:35:52.307 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:35:52.307 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:35:52.307 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:35:52.307 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:35:52.307 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:35:52.307 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:35:52.308 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:35:52.308 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:35:52.308 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:35:52.354 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 10 2024-04-06T09:35:52.354 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 11 2024-04-06T09:35:52.354 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:35:52.354 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=45, tag_tid=6, entry_tid=1], [object_number=44, tag_tid=6, entry_tid=0], [object_number=47, tag_tid=5, entry_tid=22415], [object_number=46, tag_tid=5, entry_tid=22414]]], state=connected] 2024-04-06T09:35:52.354 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=43, tag_tid=5, entry_tid=17595], [object_number=42, tag_tid=5, entry_tid=17594], [object_number=41, tag_tid=5, entry_tid=17593], [object_number=40, tag_tid=5, entry_tid=17592]]], state=connected] 2024-04-06T09:35:52.357 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:35:52.359 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=43, tag_tid=5, entry_tid=17595' 2024-04-06T09:35:52.359 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=45, tag_tid=6, entry_tid=1' -a 'object_number=45, tag_tid=6, entry_tid=1' = 'object_number=43, tag_tid=5, entry_tid=17595' 2024-04-06T09:35:52.359 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=43, tag_tid=5, entry_tid=17595' '!=' 'object_number=43, tag_tid=5, entry_tid=17515' 2024-04-06T09:35:52.360 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:35:52.360 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=43, tag_tid=5, entry_tid=17595' = 'object_number=43, tag_tid=5, entry_tid=17515' 2024-04-06T09:35:52.360 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=43, tag_tid=5, entry_tid=17595' 2024-04-06T09:35:52.360 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=45, tag_tid=6, entry_tid=1' 2024-04-06T09:35:52.360 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:35:52.360 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:35:52.360 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=6 2024-04-06T09:35:52.361 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=43, tag_tid=5, entry_tid=17595' 2024-04-06T09:35:52.361 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:35:52.361 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:35:52.362 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=5 2024-04-06T09:35:52.363 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=45, tag_tid=6, entry_tid=1' 2024-04-06T09:35:52.363 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:35:52.363 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:35:52.364 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:35:52.364 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=43, tag_tid=5, entry_tid=17595' 2024-04-06T09:35:52.365 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:35:52.365 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:35:52.366 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=17595 2024-04-06T09:35:52.366 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 6 = 5 -a 1 -le 17595 2024-04-06T09:35:52.366 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:35:52.366 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:35:52.366 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:35:52.567 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:35:52.567 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:35:52.567 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:35:52.567 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:35:52.568 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:35:52.568 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:35:52.568 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:35:52.568 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:35:52.568 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:35:52.568 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:35:52.568 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:35:52.568 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:35:52.568 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:35:52.568 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:35:52.568 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:35:52.569 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:35:52.569 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:35:52.572 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:35:52.572 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:35:52.572 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:35:52.572 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:35:52.572 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:35:52.572 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:35:52.572 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:35:52.572 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:35:52.572 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:35:52.572 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:35:52.572 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:35:52.572 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:35:52.572 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:35:52.572 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:35:52.573 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:35:52.573 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:35:52.573 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:35:52.573 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:35:52.573 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:35:52.573 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:35:52.573 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:35:52.573 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:35:52.573 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:35:52.573 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:35:52.573 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:35:52.573 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:35:52.573 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:35:52.573 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:35:52.573 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:35:52.573 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:35:52.573 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:35:52.574 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:35:52.598 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:35:52.598 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:35:52.599 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:35:52.599 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:35:52.729 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:35:52.730 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:35:52.730 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:35:52.730 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:35:52.730 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:35:52.730 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:35:52.730 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:35:52.730 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:35:52.730 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:35:52.730 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:35:52.731 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:35:52.731 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:35:52.731 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:35:52.733 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:35:52.733 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:35:52.734 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:35:52.785 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 10 2024-04-06T09:35:52.785 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 11 2024-04-06T09:35:52.786 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:35:52.786 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=45, tag_tid=6, entry_tid=1], [object_number=44, tag_tid=6, entry_tid=0], [object_number=47, tag_tid=5, entry_tid=22415], [object_number=46, tag_tid=5, entry_tid=22414]]], state=connected] 2024-04-06T09:35:52.786 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=40, tag_tid=5, entry_tid=17672], [object_number=43, tag_tid=5, entry_tid=17671], [object_number=42, tag_tid=5, entry_tid=17670], [object_number=41, tag_tid=5, entry_tid=17669]]], state=connected] 2024-04-06T09:35:52.789 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:35:52.790 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=45, tag_tid=6, entry_tid=1' 2024-04-06T09:35:52.792 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:35:52.792 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:35:52.792 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:35:52.792 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:35:52.792 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:35:52.792 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:35:52.792 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:35:52.792 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:35:52.792 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:35:52.792 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:35:52.792 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:35:52.792 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:35:52.793 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:35:52.793 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:35:52.793 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:35:53.044 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 10 2024-04-06T09:35:53.045 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 11 2024-04-06T09:35:53.045 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:35:53.045 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=45, tag_tid=6, entry_tid=1], [object_number=44, tag_tid=6, entry_tid=0], [object_number=47, tag_tid=5, entry_tid=22415], [object_number=46, tag_tid=5, entry_tid=22414]]], state=connected] 2024-04-06T09:35:53.045 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=40, tag_tid=5, entry_tid=17672], [object_number=43, tag_tid=5, entry_tid=17671], [object_number=42, tag_tid=5, entry_tid=17670], [object_number=41, tag_tid=5, entry_tid=17669]]], state=connected] 2024-04-06T09:35:53.049 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:35:53.052 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=40, tag_tid=5, entry_tid=17672' 2024-04-06T09:35:53.052 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=45, tag_tid=6, entry_tid=1' -a 'object_number=45, tag_tid=6, entry_tid=1' = 'object_number=40, tag_tid=5, entry_tid=17672' 2024-04-06T09:35:53.052 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=40, tag_tid=5, entry_tid=17672' '!=' 'object_number=43, tag_tid=5, entry_tid=17595' 2024-04-06T09:35:53.052 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:35:53.052 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=40, tag_tid=5, entry_tid=17672' = 'object_number=43, tag_tid=5, entry_tid=17595' 2024-04-06T09:35:53.052 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=40, tag_tid=5, entry_tid=17672' 2024-04-06T09:35:53.053 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=45, tag_tid=6, entry_tid=1' 2024-04-06T09:35:53.053 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:35:53.053 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:35:53.055 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=6 2024-04-06T09:35:53.057 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=40, tag_tid=5, entry_tid=17672' 2024-04-06T09:35:53.057 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:35:53.057 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:35:53.058 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=5 2024-04-06T09:35:53.059 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=45, tag_tid=6, entry_tid=1' 2024-04-06T09:35:53.059 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:35:53.060 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:35:53.061 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:35:53.062 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=40, tag_tid=5, entry_tid=17672' 2024-04-06T09:35:53.062 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:35:53.062 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:35:53.063 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=17672 2024-04-06T09:35:53.064 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 6 = 5 -a 1 -le 17672 2024-04-06T09:35:53.064 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:35:53.064 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:35:53.064 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:35:53.265 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:35:53.265 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:35:53.265 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:35:53.266 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:35:53.266 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:35:53.266 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:35:53.266 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:35:53.266 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:35:53.266 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:35:53.266 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:35:53.266 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:35:53.266 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:35:53.266 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:35:53.266 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:35:53.266 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:35:53.267 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:35:53.267 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:35:53.269 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:35:53.269 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:35:53.269 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:35:53.269 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:35:53.269 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:35:53.269 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:35:53.269 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:35:53.269 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:35:53.269 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:35:53.269 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:35:53.269 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:35:53.269 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:35:53.269 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:35:53.270 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:35:53.270 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:35:53.270 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:35:53.271 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:35:53.271 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:35:53.271 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:35:53.271 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:35:53.271 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:35:53.271 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:35:53.271 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:35:53.271 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:35:53.271 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:35:53.271 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:35:53.272 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:35:53.272 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:35:53.272 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:35:53.272 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:35:53.272 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:35:53.272 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:35:53.296 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:35:53.297 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:35:53.297 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:35:53.297 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:35:53.423 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:35:53.424 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:35:53.424 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:35:53.424 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:35:53.424 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:35:53.424 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:35:53.424 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:35:53.424 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:35:53.425 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:35:53.425 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:35:53.425 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:35:53.425 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:35:53.425 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:35:53.427 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:35:53.427 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:35:53.428 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:35:53.485 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 10 2024-04-06T09:35:53.485 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 11 2024-04-06T09:35:53.485 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:35:53.486 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=45, tag_tid=6, entry_tid=1], [object_number=44, tag_tid=6, entry_tid=0], [object_number=47, tag_tid=5, entry_tid=22415], [object_number=46, tag_tid=5, entry_tid=22414]]], state=connected] 2024-04-06T09:35:53.486 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=42, tag_tid=5, entry_tid=17798], [object_number=41, tag_tid=5, entry_tid=17797], [object_number=40, tag_tid=5, entry_tid=17796], [object_number=43, tag_tid=5, entry_tid=17795]]], state=connected] 2024-04-06T09:35:53.489 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:35:53.490 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=45, tag_tid=6, entry_tid=1' 2024-04-06T09:35:53.491 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:35:53.492 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:35:53.492 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:35:53.492 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:35:53.492 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:35:53.492 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:35:53.492 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:35:53.492 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:35:53.492 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:35:53.492 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:35:53.492 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:35:53.492 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:35:53.492 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:35:53.493 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:35:53.493 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:35:53.539 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 10 2024-04-06T09:35:53.539 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 11 2024-04-06T09:35:53.539 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:35:53.539 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=45, tag_tid=6, entry_tid=1], [object_number=44, tag_tid=6, entry_tid=0], [object_number=47, tag_tid=5, entry_tid=22415], [object_number=46, tag_tid=5, entry_tid=22414]]], state=connected] 2024-04-06T09:35:53.539 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=42, tag_tid=5, entry_tid=17798], [object_number=41, tag_tid=5, entry_tid=17797], [object_number=40, tag_tid=5, entry_tid=17796], [object_number=43, tag_tid=5, entry_tid=17795]]], state=connected] 2024-04-06T09:35:53.542 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:35:53.544 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=42, tag_tid=5, entry_tid=17798' 2024-04-06T09:35:53.544 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=45, tag_tid=6, entry_tid=1' -a 'object_number=45, tag_tid=6, entry_tid=1' = 'object_number=42, tag_tid=5, entry_tid=17798' 2024-04-06T09:35:53.544 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=42, tag_tid=5, entry_tid=17798' '!=' 'object_number=40, tag_tid=5, entry_tid=17672' 2024-04-06T09:35:53.544 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:35:53.544 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=42, tag_tid=5, entry_tid=17798' = 'object_number=40, tag_tid=5, entry_tid=17672' 2024-04-06T09:35:53.544 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=42, tag_tid=5, entry_tid=17798' 2024-04-06T09:35:53.545 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=45, tag_tid=6, entry_tid=1' 2024-04-06T09:35:53.545 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:35:53.545 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:35:53.545 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=6 2024-04-06T09:35:53.546 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=42, tag_tid=5, entry_tid=17798' 2024-04-06T09:35:53.546 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:35:53.546 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:35:53.547 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=5 2024-04-06T09:35:53.548 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=45, tag_tid=6, entry_tid=1' 2024-04-06T09:35:53.548 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:35:53.548 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:35:53.549 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:35:53.549 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=42, tag_tid=5, entry_tid=17798' 2024-04-06T09:35:53.550 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:35:53.550 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:35:53.551 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=17798 2024-04-06T09:35:53.551 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 6 = 5 -a 1 -le 17798 2024-04-06T09:35:53.551 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:35:53.551 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:35:53.551 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:35:53.753 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:35:53.753 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:35:53.753 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:35:53.753 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:35:53.753 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:35:53.753 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:35:53.753 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:35:53.753 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:35:53.753 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:35:53.753 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:35:53.753 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:35:53.753 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:35:53.753 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:35:53.753 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:35:53.754 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:35:53.754 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:35:53.755 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:35:53.757 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:35:53.757 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:35:53.757 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:35:53.757 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:35:53.757 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:35:53.757 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:35:53.758 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:35:53.758 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:35:53.758 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:35:53.758 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:35:53.758 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:35:53.758 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:35:53.758 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:35:53.758 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:35:53.758 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:35:53.758 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:35:53.759 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:35:53.759 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:35:53.759 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:35:53.760 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:35:53.760 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:35:53.760 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:35:53.760 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:35:53.760 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:35:53.760 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:35:53.760 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:35:53.760 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:35:53.760 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:35:53.760 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:35:53.760 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:35:53.760 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:35:53.760 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:35:53.784 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:35:53.785 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:35:53.785 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:35:53.785 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:35:53.925 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:35:53.926 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:35:53.926 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:35:53.926 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:35:53.926 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:35:53.926 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:35:53.926 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:35:53.926 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:35:53.927 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:35:53.927 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:35:53.927 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:35:53.927 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:35:53.927 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:35:53.929 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:35:53.929 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:35:53.929 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:35:53.983 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 10 2024-04-06T09:35:53.983 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 11 2024-04-06T09:35:53.983 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:35:53.983 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=45, tag_tid=6, entry_tid=1], [object_number=44, tag_tid=6, entry_tid=0], [object_number=47, tag_tid=5, entry_tid=22415], [object_number=46, tag_tid=5, entry_tid=22414]]], state=connected] 2024-04-06T09:35:53.983 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=42, tag_tid=5, entry_tid=17894], [object_number=41, tag_tid=5, entry_tid=17893], [object_number=40, tag_tid=5, entry_tid=17892], [object_number=43, tag_tid=5, entry_tid=17891]]], state=connected] 2024-04-06T09:35:53.987 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:35:53.988 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=45, tag_tid=6, entry_tid=1' 2024-04-06T09:35:53.989 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:35:53.989 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:35:53.989 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:35:53.990 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:35:53.990 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:35:53.990 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:35:53.990 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:35:53.990 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:35:53.990 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:35:53.990 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:35:53.990 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:35:53.990 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:35:53.991 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:35:53.991 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:35:53.991 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:35:54.037 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 10 2024-04-06T09:35:54.038 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 11 2024-04-06T09:35:54.038 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:35:54.038 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=45, tag_tid=6, entry_tid=1], [object_number=44, tag_tid=6, entry_tid=0], [object_number=47, tag_tid=5, entry_tid=22415], [object_number=46, tag_tid=5, entry_tid=22414]]], state=connected] 2024-04-06T09:35:54.038 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=42, tag_tid=5, entry_tid=17894], [object_number=41, tag_tid=5, entry_tid=17893], [object_number=40, tag_tid=5, entry_tid=17892], [object_number=43, tag_tid=5, entry_tid=17891]]], state=connected] 2024-04-06T09:35:54.041 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:35:54.043 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=42, tag_tid=5, entry_tid=17894' 2024-04-06T09:35:54.043 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=45, tag_tid=6, entry_tid=1' -a 'object_number=45, tag_tid=6, entry_tid=1' = 'object_number=42, tag_tid=5, entry_tid=17894' 2024-04-06T09:35:54.043 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=42, tag_tid=5, entry_tid=17894' '!=' 'object_number=42, tag_tid=5, entry_tid=17798' 2024-04-06T09:35:54.044 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:35:54.044 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=42, tag_tid=5, entry_tid=17894' = 'object_number=42, tag_tid=5, entry_tid=17798' 2024-04-06T09:35:54.044 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=42, tag_tid=5, entry_tid=17894' 2024-04-06T09:35:54.044 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=45, tag_tid=6, entry_tid=1' 2024-04-06T09:35:54.044 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:35:54.044 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:35:54.044 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=6 2024-04-06T09:35:54.045 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=42, tag_tid=5, entry_tid=17894' 2024-04-06T09:35:54.045 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:35:54.045 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:35:54.046 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=5 2024-04-06T09:35:54.047 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=45, tag_tid=6, entry_tid=1' 2024-04-06T09:35:54.047 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:35:54.047 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:35:54.048 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:35:54.048 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=42, tag_tid=5, entry_tid=17894' 2024-04-06T09:35:54.049 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:35:54.049 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:35:54.050 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=17894 2024-04-06T09:35:54.050 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 6 = 5 -a 1 -le 17894 2024-04-06T09:35:54.050 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:35:54.050 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:35:54.050 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:35:54.252 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:35:54.252 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:35:54.252 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:35:54.252 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:35:54.252 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:35:54.252 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:35:54.252 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:35:54.252 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:35:54.252 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:35:54.252 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:35:54.252 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:35:54.252 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:35:54.252 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:35:54.252 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:35:54.252 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:35:54.253 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:35:54.254 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:35:54.257 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:35:54.257 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:35:54.257 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:35:54.257 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:35:54.257 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:35:54.257 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:35:54.257 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:35:54.257 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:35:54.257 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:35:54.257 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:35:54.257 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:35:54.258 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:35:54.258 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:35:54.258 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:35:54.258 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:35:54.258 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:35:54.258 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:35:54.258 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:35:54.258 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:35:54.258 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:35:54.258 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:35:54.258 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:35:54.258 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:35:54.258 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:35:54.258 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:35:54.259 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:35:54.259 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:35:54.259 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:35:54.259 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:35:54.259 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:35:54.259 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:35:54.259 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:35:54.282 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:35:54.283 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:35:54.283 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:35:54.283 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:35:54.411 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:35:54.412 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:35:54.412 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:35:54.413 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:35:54.413 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:35:54.413 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:35:54.413 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:35:54.413 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:35:54.413 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:35:54.413 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:35:54.413 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:35:54.414 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:35:54.414 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:35:54.417 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:35:54.417 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:35:54.417 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:35:54.469 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 10 2024-04-06T09:35:54.469 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 11 2024-04-06T09:35:54.469 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:35:54.469 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=45, tag_tid=6, entry_tid=1], [object_number=44, tag_tid=6, entry_tid=0], [object_number=47, tag_tid=5, entry_tid=22415], [object_number=46, tag_tid=5, entry_tid=22414]]], state=connected] 2024-04-06T09:35:54.469 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=40, tag_tid=5, entry_tid=18000], [object_number=43, tag_tid=5, entry_tid=17999], [object_number=42, tag_tid=5, entry_tid=17998], [object_number=41, tag_tid=5, entry_tid=17997]]], state=connected] 2024-04-06T09:35:54.472 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:35:54.474 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=45, tag_tid=6, entry_tid=1' 2024-04-06T09:35:54.475 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:35:54.475 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:35:54.475 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:35:54.475 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:35:54.475 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:35:54.475 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:35:54.475 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:35:54.475 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:35:54.475 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:35:54.475 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:35:54.475 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:35:54.475 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:35:54.476 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:35:54.476 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:35:54.476 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:35:54.524 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 10 2024-04-06T09:35:54.524 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 11 2024-04-06T09:35:54.524 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:35:54.525 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=45, tag_tid=6, entry_tid=1], [object_number=44, tag_tid=6, entry_tid=0], [object_number=47, tag_tid=5, entry_tid=22415], [object_number=46, tag_tid=5, entry_tid=22414]]], state=connected] 2024-04-06T09:35:54.525 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=40, tag_tid=5, entry_tid=18000], [object_number=43, tag_tid=5, entry_tid=17999], [object_number=42, tag_tid=5, entry_tid=17998], [object_number=41, tag_tid=5, entry_tid=17997]]], state=connected] 2024-04-06T09:35:54.528 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:35:54.529 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=40, tag_tid=5, entry_tid=18000' 2024-04-06T09:35:54.530 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=45, tag_tid=6, entry_tid=1' -a 'object_number=45, tag_tid=6, entry_tid=1' = 'object_number=40, tag_tid=5, entry_tid=18000' 2024-04-06T09:35:54.530 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=40, tag_tid=5, entry_tid=18000' '!=' 'object_number=42, tag_tid=5, entry_tid=17894' 2024-04-06T09:35:54.530 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:35:54.530 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=40, tag_tid=5, entry_tid=18000' = 'object_number=42, tag_tid=5, entry_tid=17894' 2024-04-06T09:35:54.530 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=40, tag_tid=5, entry_tid=18000' 2024-04-06T09:35:54.531 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=45, tag_tid=6, entry_tid=1' 2024-04-06T09:35:54.531 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:35:54.531 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:35:54.531 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=6 2024-04-06T09:35:54.532 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=40, tag_tid=5, entry_tid=18000' 2024-04-06T09:35:54.532 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:35:54.532 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:35:54.533 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=5 2024-04-06T09:35:54.533 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=45, tag_tid=6, entry_tid=1' 2024-04-06T09:35:54.534 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:35:54.534 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:35:54.535 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:35:54.535 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=40, tag_tid=5, entry_tid=18000' 2024-04-06T09:35:54.535 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:35:54.535 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:35:54.536 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=18000 2024-04-06T09:35:54.537 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 6 = 5 -a 1 -le 18000 2024-04-06T09:35:54.537 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:35:54.537 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:35:54.537 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:35:54.738 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:35:54.739 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:35:54.739 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:35:54.739 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:35:54.739 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:35:54.739 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:35:54.739 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:35:54.739 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:35:54.739 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:35:54.739 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:35:54.739 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:35:54.739 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:35:54.739 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:35:54.739 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:35:54.739 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:35:54.740 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:35:54.741 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:35:54.743 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:35:54.743 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:35:54.743 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:35:54.743 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:35:54.743 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:35:54.743 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:35:54.743 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:35:54.743 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:35:54.743 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:35:54.743 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:35:54.744 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:35:54.744 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:35:54.744 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:35:54.744 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:35:54.744 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:35:54.744 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:35:54.745 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:35:54.745 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:35:54.745 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:35:54.745 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:35:54.745 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:35:54.746 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:35:54.746 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:35:54.746 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:35:54.746 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:35:54.746 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:35:54.746 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:35:54.746 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:35:54.746 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:35:54.746 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:35:54.746 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:35:54.746 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:35:54.770 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:35:54.770 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:35:54.770 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:35:54.770 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:35:54.909 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:35:54.911 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:35:54.911 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:35:54.911 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:35:54.911 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:35:54.911 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:35:54.911 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:35:54.911 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:35:54.911 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:35:54.911 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:35:54.911 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:35:54.911 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:35:54.912 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:35:54.914 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:35:54.914 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:35:54.914 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:35:54.964 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 10 2024-04-06T09:35:54.964 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 11 2024-04-06T09:35:54.964 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:35:54.964 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=45, tag_tid=6, entry_tid=1], [object_number=44, tag_tid=6, entry_tid=0], [object_number=47, tag_tid=5, entry_tid=22415], [object_number=46, tag_tid=5, entry_tid=22414]]], state=connected] 2024-04-06T09:35:54.964 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=43, tag_tid=5, entry_tid=18099], [object_number=42, tag_tid=5, entry_tid=18098], [object_number=41, tag_tid=5, entry_tid=18097], [object_number=40, tag_tid=5, entry_tid=18096]]], state=connected] 2024-04-06T09:35:54.968 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:35:54.970 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=45, tag_tid=6, entry_tid=1' 2024-04-06T09:35:54.970 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:35:54.970 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:35:54.970 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:35:54.970 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:35:54.970 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:35:54.970 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:35:54.970 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:35:54.971 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:35:54.971 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:35:54.971 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:35:54.971 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:35:54.971 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:35:54.971 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:35:54.972 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:35:54.972 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:35:55.017 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 10 2024-04-06T09:35:55.017 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 11 2024-04-06T09:35:55.017 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:35:55.017 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=45, tag_tid=6, entry_tid=1], [object_number=44, tag_tid=6, entry_tid=0], [object_number=47, tag_tid=5, entry_tid=22415], [object_number=46, tag_tid=5, entry_tid=22414]]], state=connected] 2024-04-06T09:35:55.017 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=43, tag_tid=5, entry_tid=18099], [object_number=42, tag_tid=5, entry_tid=18098], [object_number=41, tag_tid=5, entry_tid=18097], [object_number=40, tag_tid=5, entry_tid=18096]]], state=connected] 2024-04-06T09:35:55.021 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:35:55.022 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=43, tag_tid=5, entry_tid=18099' 2024-04-06T09:35:55.023 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=45, tag_tid=6, entry_tid=1' -a 'object_number=45, tag_tid=6, entry_tid=1' = 'object_number=43, tag_tid=5, entry_tid=18099' 2024-04-06T09:35:55.023 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=43, tag_tid=5, entry_tid=18099' '!=' 'object_number=40, tag_tid=5, entry_tid=18000' 2024-04-06T09:35:55.023 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:35:55.023 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=43, tag_tid=5, entry_tid=18099' = 'object_number=40, tag_tid=5, entry_tid=18000' 2024-04-06T09:35:55.023 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=43, tag_tid=5, entry_tid=18099' 2024-04-06T09:35:55.023 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=45, tag_tid=6, entry_tid=1' 2024-04-06T09:35:55.023 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:35:55.024 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:35:55.024 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=6 2024-04-06T09:35:55.025 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=43, tag_tid=5, entry_tid=18099' 2024-04-06T09:35:55.025 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:35:55.025 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:35:55.026 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=5 2024-04-06T09:35:55.026 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=45, tag_tid=6, entry_tid=1' 2024-04-06T09:35:55.027 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:35:55.027 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:35:55.028 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:35:55.028 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=43, tag_tid=5, entry_tid=18099' 2024-04-06T09:35:55.028 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:35:55.028 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:35:55.030 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=18099 2024-04-06T09:35:55.030 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 6 = 5 -a 1 -le 18099 2024-04-06T09:35:55.030 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:35:55.030 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:35:55.030 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:35:55.231 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:35:55.231 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:35:55.231 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:35:55.231 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:35:55.232 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:35:55.232 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:35:55.232 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:35:55.232 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:35:55.232 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:35:55.232 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:35:55.232 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:35:55.232 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:35:55.232 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:35:55.232 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:35:55.232 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:35:55.233 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:35:55.233 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:35:55.235 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:35:55.235 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:35:55.236 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:35:55.236 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:35:55.236 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:35:55.236 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:35:55.236 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:35:55.236 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:35:55.236 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:35:55.236 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:35:55.236 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:35:55.236 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:35:55.236 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:35:55.236 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:35:55.236 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:35:55.237 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:35:55.237 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:35:55.237 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:35:55.237 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:35:55.237 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:35:55.237 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:35:55.237 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:35:55.237 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:35:55.237 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:35:55.237 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:35:55.237 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:35:55.237 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:35:55.237 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:35:55.237 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:35:55.237 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:35:55.238 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:35:55.238 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:35:55.262 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:35:55.263 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:35:55.263 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:35:55.263 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:35:55.396 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:35:55.397 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:35:55.397 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:35:55.397 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:35:55.397 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:35:55.397 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:35:55.397 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:35:55.397 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:35:55.398 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:35:55.398 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:35:55.398 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:35:55.398 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:35:55.399 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:35:55.400 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:35:55.400 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:35:55.400 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:35:55.450 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 10 2024-04-06T09:35:55.450 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 11 2024-04-06T09:35:55.450 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:35:55.450 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=45, tag_tid=6, entry_tid=1], [object_number=44, tag_tid=6, entry_tid=0], [object_number=47, tag_tid=5, entry_tid=22415], [object_number=46, tag_tid=5, entry_tid=22414]]], state=connected] 2024-04-06T09:35:55.451 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=40, tag_tid=5, entry_tid=18196], [object_number=43, tag_tid=5, entry_tid=18195], [object_number=42, tag_tid=5, entry_tid=18194], [object_number=41, tag_tid=5, entry_tid=18193]]], state=connected] 2024-04-06T09:35:55.454 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:35:55.455 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=45, tag_tid=6, entry_tid=1' 2024-04-06T09:35:55.456 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:35:55.457 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:35:55.457 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:35:55.457 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:35:55.457 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:35:55.457 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:35:55.457 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:35:55.457 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:35:55.457 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:35:55.457 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:35:55.457 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:35:55.457 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:35:55.458 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:35:55.458 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:35:55.458 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:35:55.503 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 10 2024-04-06T09:35:55.503 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 11 2024-04-06T09:35:55.503 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:35:55.503 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=45, tag_tid=6, entry_tid=1], [object_number=44, tag_tid=6, entry_tid=0], [object_number=47, tag_tid=5, entry_tid=22415], [object_number=46, tag_tid=5, entry_tid=22414]]], state=connected] 2024-04-06T09:35:55.503 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=40, tag_tid=5, entry_tid=18196], [object_number=43, tag_tid=5, entry_tid=18195], [object_number=42, tag_tid=5, entry_tid=18194], [object_number=41, tag_tid=5, entry_tid=18193]]], state=connected] 2024-04-06T09:35:55.507 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:35:55.509 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=40, tag_tid=5, entry_tid=18196' 2024-04-06T09:35:55.509 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=45, tag_tid=6, entry_tid=1' -a 'object_number=45, tag_tid=6, entry_tid=1' = 'object_number=40, tag_tid=5, entry_tid=18196' 2024-04-06T09:35:55.509 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=40, tag_tid=5, entry_tid=18196' '!=' 'object_number=43, tag_tid=5, entry_tid=18099' 2024-04-06T09:35:55.510 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:35:55.510 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=40, tag_tid=5, entry_tid=18196' = 'object_number=43, tag_tid=5, entry_tid=18099' 2024-04-06T09:35:55.510 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=40, tag_tid=5, entry_tid=18196' 2024-04-06T09:35:55.510 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=45, tag_tid=6, entry_tid=1' 2024-04-06T09:35:55.510 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:35:55.510 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:35:55.510 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=6 2024-04-06T09:35:55.511 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=40, tag_tid=5, entry_tid=18196' 2024-04-06T09:35:55.511 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:35:55.511 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:35:55.512 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=5 2024-04-06T09:35:55.513 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=45, tag_tid=6, entry_tid=1' 2024-04-06T09:35:55.513 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:35:55.513 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:35:55.514 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:35:55.514 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=40, tag_tid=5, entry_tid=18196' 2024-04-06T09:35:55.514 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:35:55.515 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:35:55.516 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=18196 2024-04-06T09:35:55.516 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 6 = 5 -a 1 -le 18196 2024-04-06T09:35:55.516 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:35:55.516 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:35:55.516 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:35:55.717 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:35:55.718 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:35:55.718 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:35:55.718 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:35:55.718 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:35:55.718 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:35:55.718 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:35:55.718 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:35:55.718 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:35:55.718 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:35:55.718 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:35:55.718 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:35:55.718 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:35:55.718 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:35:55.718 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:35:55.719 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:35:55.719 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:35:55.722 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:35:55.722 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:35:55.722 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:35:55.722 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:35:55.722 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:35:55.722 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:35:55.722 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:35:55.723 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:35:55.723 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:35:55.723 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:35:55.723 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:35:55.723 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:35:55.723 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:35:55.723 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:35:55.723 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:35:55.723 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:35:55.723 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:35:55.723 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:35:55.723 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:35:55.723 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:35:55.723 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:35:55.723 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:35:55.723 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:35:55.723 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:35:55.724 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:35:55.724 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:35:55.724 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:35:55.724 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:35:55.724 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:35:55.724 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:35:55.724 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:35:55.724 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:35:55.749 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:35:55.749 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:35:55.749 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:35:55.749 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:35:55.885 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:35:55.886 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:35:55.886 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:35:55.886 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:35:55.886 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:35:55.886 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:35:55.886 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:35:55.887 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:35:55.887 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:35:55.887 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:35:55.887 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:35:55.887 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:35:55.887 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:35:55.889 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:35:55.889 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:35:55.889 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:35:55.940 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 10 2024-04-06T09:35:55.940 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 11 2024-04-06T09:35:55.940 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:35:55.940 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=45, tag_tid=6, entry_tid=1], [object_number=44, tag_tid=6, entry_tid=0], [object_number=47, tag_tid=5, entry_tid=22415], [object_number=46, tag_tid=5, entry_tid=22414]]], state=connected] 2024-04-06T09:35:55.940 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=41, tag_tid=5, entry_tid=18289], [object_number=40, tag_tid=5, entry_tid=18288], [object_number=43, tag_tid=5, entry_tid=18287], [object_number=42, tag_tid=5, entry_tid=18286]]], state=connected] 2024-04-06T09:35:55.943 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:35:55.946 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=45, tag_tid=6, entry_tid=1' 2024-04-06T09:35:55.946 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:35:55.946 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:35:55.946 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:35:55.946 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:35:55.946 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:35:55.946 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:35:55.946 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:35:55.946 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:35:55.946 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:35:55.946 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:35:55.947 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:35:55.949 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:35:55.950 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:35:55.950 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:35:55.950 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:35:55.993 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 10 2024-04-06T09:35:55.993 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 11 2024-04-06T09:35:55.993 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:35:55.993 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=45, tag_tid=6, entry_tid=1], [object_number=44, tag_tid=6, entry_tid=0], [object_number=47, tag_tid=5, entry_tid=22415], [object_number=46, tag_tid=5, entry_tid=22414]]], state=connected] 2024-04-06T09:35:55.993 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=41, tag_tid=5, entry_tid=18289], [object_number=40, tag_tid=5, entry_tid=18288], [object_number=43, tag_tid=5, entry_tid=18287], [object_number=42, tag_tid=5, entry_tid=18286]]], state=connected] 2024-04-06T09:35:55.997 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:35:55.998 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=41, tag_tid=5, entry_tid=18289' 2024-04-06T09:35:55.999 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=45, tag_tid=6, entry_tid=1' -a 'object_number=45, tag_tid=6, entry_tid=1' = 'object_number=41, tag_tid=5, entry_tid=18289' 2024-04-06T09:35:55.999 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=41, tag_tid=5, entry_tid=18289' '!=' 'object_number=40, tag_tid=5, entry_tid=18196' 2024-04-06T09:35:55.999 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:35:55.999 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=41, tag_tid=5, entry_tid=18289' = 'object_number=40, tag_tid=5, entry_tid=18196' 2024-04-06T09:35:55.999 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=41, tag_tid=5, entry_tid=18289' 2024-04-06T09:35:55.999 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=45, tag_tid=6, entry_tid=1' 2024-04-06T09:35:55.999 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:35:55.999 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:35:56.000 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=6 2024-04-06T09:35:56.001 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=41, tag_tid=5, entry_tid=18289' 2024-04-06T09:35:56.001 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:35:56.001 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:35:56.002 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=5 2024-04-06T09:35:56.002 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=45, tag_tid=6, entry_tid=1' 2024-04-06T09:35:56.002 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:35:56.003 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:35:56.003 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:35:56.004 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=41, tag_tid=5, entry_tid=18289' 2024-04-06T09:35:56.004 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:35:56.004 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:35:56.005 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=18289 2024-04-06T09:35:56.005 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 6 = 5 -a 1 -le 18289 2024-04-06T09:35:56.005 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:35:56.005 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:35:56.006 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:35:56.207 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:35:56.207 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:35:56.207 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:35:56.207 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:35:56.207 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:35:56.207 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:35:56.208 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:35:56.208 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:35:56.208 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:35:56.208 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:35:56.208 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:35:56.208 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:35:56.208 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:35:56.208 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:35:56.208 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:35:56.208 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:35:56.209 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:35:56.212 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:35:56.212 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:35:56.212 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:35:56.212 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:35:56.212 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:35:56.212 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:35:56.212 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:35:56.212 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:35:56.213 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:35:56.213 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:35:56.213 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:35:56.213 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:35:56.213 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:35:56.213 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:35:56.213 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:35:56.213 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:35:56.213 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:35:56.213 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:35:56.213 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:35:56.213 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:35:56.213 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:35:56.213 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:35:56.213 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:35:56.213 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:35:56.214 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:35:56.214 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:35:56.214 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:35:56.214 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:35:56.214 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:35:56.214 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:35:56.214 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:35:56.214 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:35:56.238 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:35:56.238 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:35:56.238 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:35:56.238 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:35:56.366 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:35:56.366 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:35:56.366 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:35:56.366 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:35:56.366 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:35:56.366 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:35:56.366 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:35:56.366 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:35:56.367 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:35:56.367 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:35:56.367 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:35:56.367 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:35:56.367 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:35:56.370 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:35:56.370 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:35:56.370 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:35:56.419 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 10 2024-04-06T09:35:56.419 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 11 2024-04-06T09:35:56.419 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:35:56.420 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=45, tag_tid=6, entry_tid=1], [object_number=44, tag_tid=6, entry_tid=0], [object_number=47, tag_tid=5, entry_tid=22415], [object_number=46, tag_tid=5, entry_tid=22414]]], state=connected] 2024-04-06T09:35:56.420 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=40, tag_tid=5, entry_tid=18376], [object_number=43, tag_tid=5, entry_tid=18375], [object_number=42, tag_tid=5, entry_tid=18374], [object_number=41, tag_tid=5, entry_tid=18373]]], state=connected] 2024-04-06T09:35:56.423 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:35:56.424 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=45, tag_tid=6, entry_tid=1' 2024-04-06T09:35:56.425 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:35:56.426 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:35:56.426 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:35:56.426 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:35:56.426 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:35:56.426 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:35:56.426 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:35:56.426 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:35:56.426 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:35:56.426 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:35:56.426 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:35:56.426 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:35:56.427 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:35:56.427 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:35:56.427 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:35:56.472 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 10 2024-04-06T09:35:56.472 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 11 2024-04-06T09:35:56.472 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:35:56.472 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=45, tag_tid=6, entry_tid=1], [object_number=44, tag_tid=6, entry_tid=0], [object_number=47, tag_tid=5, entry_tid=22415], [object_number=46, tag_tid=5, entry_tid=22414]]], state=connected] 2024-04-06T09:35:56.472 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=40, tag_tid=5, entry_tid=18376], [object_number=43, tag_tid=5, entry_tid=18375], [object_number=42, tag_tid=5, entry_tid=18374], [object_number=41, tag_tid=5, entry_tid=18373]]], state=connected] 2024-04-06T09:35:56.476 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:35:56.478 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=40, tag_tid=5, entry_tid=18376' 2024-04-06T09:35:56.478 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=45, tag_tid=6, entry_tid=1' -a 'object_number=45, tag_tid=6, entry_tid=1' = 'object_number=40, tag_tid=5, entry_tid=18376' 2024-04-06T09:35:56.478 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=40, tag_tid=5, entry_tid=18376' '!=' 'object_number=41, tag_tid=5, entry_tid=18289' 2024-04-06T09:35:56.478 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:35:56.478 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=40, tag_tid=5, entry_tid=18376' = 'object_number=41, tag_tid=5, entry_tid=18289' 2024-04-06T09:35:56.478 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=40, tag_tid=5, entry_tid=18376' 2024-04-06T09:35:56.478 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=45, tag_tid=6, entry_tid=1' 2024-04-06T09:35:56.478 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:35:56.478 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:35:56.479 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=6 2024-04-06T09:35:56.480 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=40, tag_tid=5, entry_tid=18376' 2024-04-06T09:35:56.480 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:35:56.480 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:35:56.481 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=5 2024-04-06T09:35:56.481 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=45, tag_tid=6, entry_tid=1' 2024-04-06T09:35:56.481 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:35:56.482 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:35:56.482 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:35:56.483 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=40, tag_tid=5, entry_tid=18376' 2024-04-06T09:35:56.483 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:35:56.483 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:35:56.484 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=18376 2024-04-06T09:35:56.484 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 6 = 5 -a 1 -le 18376 2024-04-06T09:35:56.485 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:35:56.485 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:35:56.485 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:35:56.686 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:35:56.686 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:35:56.686 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:35:56.686 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:35:56.686 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:35:56.686 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:35:56.686 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:35:56.686 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:35:56.686 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:35:56.686 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:35:56.687 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:35:56.687 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:35:56.687 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:35:56.687 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:35:56.687 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:35:56.687 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:35:56.688 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:35:56.692 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:35:56.692 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:35:56.692 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:35:56.692 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:35:56.692 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:35:56.692 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:35:56.692 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:35:56.692 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:35:56.692 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:35:56.692 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:35:56.692 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:35:56.692 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:35:56.693 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:35:56.693 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:35:56.693 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:35:56.693 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:35:56.693 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:35:56.693 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:35:56.693 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:35:56.693 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:35:56.693 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:35:56.693 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:35:56.693 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:35:56.693 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:35:56.693 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:35:56.693 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:35:56.693 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:35:56.693 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:35:56.694 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:35:56.694 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:35:56.694 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:35:56.694 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:35:56.716 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:35:56.716 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:35:56.717 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:35:56.717 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:35:56.843 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:35:56.844 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:35:56.844 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:35:56.844 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:35:56.844 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:35:56.844 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:35:56.844 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:35:56.845 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:35:56.845 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:35:56.845 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:35:56.845 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:35:56.845 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:35:56.845 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:35:56.847 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:35:56.847 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:35:56.847 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:35:56.898 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 10 2024-04-06T09:35:56.898 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 11 2024-04-06T09:35:56.898 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:35:56.898 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=45, tag_tid=6, entry_tid=1], [object_number=44, tag_tid=6, entry_tid=0], [object_number=47, tag_tid=5, entry_tid=22415], [object_number=46, tag_tid=5, entry_tid=22414]]], state=connected] 2024-04-06T09:35:56.898 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=41, tag_tid=5, entry_tid=18461], [object_number=40, tag_tid=5, entry_tid=18460], [object_number=43, tag_tid=5, entry_tid=18459], [object_number=42, tag_tid=5, entry_tid=18458]]], state=connected] 2024-04-06T09:35:56.901 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:35:56.902 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=45, tag_tid=6, entry_tid=1' 2024-04-06T09:35:56.903 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:35:56.903 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:35:56.903 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:35:56.903 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:35:56.903 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:35:56.903 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:35:56.903 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:35:56.903 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:35:56.903 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:35:56.903 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:35:56.903 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:35:56.903 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:35:56.904 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:35:56.904 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:35:56.904 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:35:57.157 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 10 2024-04-06T09:35:57.157 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 11 2024-04-06T09:35:57.157 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:35:57.157 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=45, tag_tid=6, entry_tid=1], [object_number=44, tag_tid=6, entry_tid=0], [object_number=47, tag_tid=5, entry_tid=22415], [object_number=46, tag_tid=5, entry_tid=22414]]], state=connected] 2024-04-06T09:35:57.157 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=41, tag_tid=5, entry_tid=18461], [object_number=40, tag_tid=5, entry_tid=18460], [object_number=43, tag_tid=5, entry_tid=18459], [object_number=42, tag_tid=5, entry_tid=18458]]], state=connected] 2024-04-06T09:35:57.163 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:35:57.165 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=41, tag_tid=5, entry_tid=18461' 2024-04-06T09:35:57.165 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=45, tag_tid=6, entry_tid=1' -a 'object_number=45, tag_tid=6, entry_tid=1' = 'object_number=41, tag_tid=5, entry_tid=18461' 2024-04-06T09:35:57.165 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=41, tag_tid=5, entry_tid=18461' '!=' 'object_number=40, tag_tid=5, entry_tid=18376' 2024-04-06T09:35:57.165 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:35:57.165 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=41, tag_tid=5, entry_tid=18461' = 'object_number=40, tag_tid=5, entry_tid=18376' 2024-04-06T09:35:57.165 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=41, tag_tid=5, entry_tid=18461' 2024-04-06T09:35:57.166 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=45, tag_tid=6, entry_tid=1' 2024-04-06T09:35:57.166 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:35:57.166 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:35:57.168 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=6 2024-04-06T09:35:57.169 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=41, tag_tid=5, entry_tid=18461' 2024-04-06T09:35:57.169 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:35:57.169 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:35:57.171 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=5 2024-04-06T09:35:57.171 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=45, tag_tid=6, entry_tid=1' 2024-04-06T09:35:57.171 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:35:57.171 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:35:57.173 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:35:57.174 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=41, tag_tid=5, entry_tid=18461' 2024-04-06T09:35:57.174 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:35:57.174 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:35:57.177 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=18461 2024-04-06T09:35:57.178 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 6 = 5 -a 1 -le 18461 2024-04-06T09:35:57.178 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:35:57.178 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:35:57.178 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:35:57.378 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:35:57.378 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:35:57.378 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:35:57.378 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:35:57.378 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:35:57.378 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:35:57.378 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:35:57.378 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:35:57.379 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:35:57.379 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:35:57.379 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:35:57.379 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:35:57.379 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:35:57.379 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:35:57.379 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:35:57.379 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:35:57.380 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:35:57.382 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:35:57.382 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:35:57.382 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:35:57.382 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:35:57.382 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:35:57.382 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:35:57.383 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:35:57.383 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:35:57.383 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:35:57.383 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:35:57.383 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:35:57.383 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:35:57.383 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:35:57.383 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:35:57.383 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:35:57.383 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:35:57.383 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:35:57.383 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:35:57.383 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:35:57.383 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:35:57.384 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:35:57.384 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:35:57.384 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:35:57.384 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:35:57.384 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:35:57.384 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:35:57.384 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:35:57.384 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:35:57.384 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:35:57.384 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:35:57.384 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:35:57.384 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:35:57.408 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:35:57.408 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:35:57.408 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:35:57.408 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:35:57.547 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:35:57.548 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:35:57.548 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:35:57.548 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:35:57.548 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:35:57.548 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:35:57.548 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:35:57.548 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:35:57.548 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:35:57.548 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:35:57.549 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:35:57.549 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:35:57.549 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:35:57.550 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:35:57.550 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:35:57.550 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:35:57.605 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 10 2024-04-06T09:35:57.605 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 11 2024-04-06T09:35:57.606 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:35:57.606 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=45, tag_tid=6, entry_tid=1], [object_number=44, tag_tid=6, entry_tid=0], [object_number=47, tag_tid=5, entry_tid=22415], [object_number=46, tag_tid=5, entry_tid=22414]]], state=connected] 2024-04-06T09:35:57.606 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=46, tag_tid=5, entry_tid=18574], [object_number=41, tag_tid=5, entry_tid=18573], [object_number=40, tag_tid=5, entry_tid=18572], [object_number=43, tag_tid=5, entry_tid=18571]]], state=connected] 2024-04-06T09:35:57.610 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:35:57.612 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=45, tag_tid=6, entry_tid=1' 2024-04-06T09:35:57.613 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:35:57.613 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:35:57.613 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:35:57.613 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:35:57.613 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:35:57.613 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:35:57.613 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:35:57.613 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:35:57.613 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:35:57.613 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:35:57.614 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:35:57.614 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:35:57.614 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:35:57.614 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:35:57.614 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:35:57.662 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 10 2024-04-06T09:35:57.662 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 11 2024-04-06T09:35:57.662 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:35:57.662 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=45, tag_tid=6, entry_tid=1], [object_number=44, tag_tid=6, entry_tid=0], [object_number=47, tag_tid=5, entry_tid=22415], [object_number=46, tag_tid=5, entry_tid=22414]]], state=connected] 2024-04-06T09:35:57.662 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=46, tag_tid=5, entry_tid=18574], [object_number=41, tag_tid=5, entry_tid=18573], [object_number=40, tag_tid=5, entry_tid=18572], [object_number=43, tag_tid=5, entry_tid=18571]]], state=connected] 2024-04-06T09:35:57.665 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:35:57.667 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=46, tag_tid=5, entry_tid=18574' 2024-04-06T09:35:57.668 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=45, tag_tid=6, entry_tid=1' -a 'object_number=45, tag_tid=6, entry_tid=1' = 'object_number=46, tag_tid=5, entry_tid=18574' 2024-04-06T09:35:57.668 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=46, tag_tid=5, entry_tid=18574' '!=' 'object_number=41, tag_tid=5, entry_tid=18461' 2024-04-06T09:35:57.668 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:35:57.668 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=46, tag_tid=5, entry_tid=18574' = 'object_number=41, tag_tid=5, entry_tid=18461' 2024-04-06T09:35:57.668 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=46, tag_tid=5, entry_tid=18574' 2024-04-06T09:35:57.668 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=45, tag_tid=6, entry_tid=1' 2024-04-06T09:35:57.668 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:35:57.668 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:35:57.668 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=6 2024-04-06T09:35:57.669 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=46, tag_tid=5, entry_tid=18574' 2024-04-06T09:35:57.669 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:35:57.669 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:35:57.670 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=5 2024-04-06T09:35:57.671 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=45, tag_tid=6, entry_tid=1' 2024-04-06T09:35:57.671 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:35:57.671 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:35:57.672 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:35:57.672 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=46, tag_tid=5, entry_tid=18574' 2024-04-06T09:35:57.673 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:35:57.673 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:35:57.674 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=18574 2024-04-06T09:35:57.674 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 6 = 5 -a 1 -le 18574 2024-04-06T09:35:57.674 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:35:57.674 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:35:57.674 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:35:57.876 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:35:57.876 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:35:57.876 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:35:57.876 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:35:57.876 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:35:57.876 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:35:57.876 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:35:57.876 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:35:57.877 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:35:57.877 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:35:57.877 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:35:57.877 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:35:57.877 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:35:57.877 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:35:57.877 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:35:57.877 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:35:57.878 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:35:57.879 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:35:57.880 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:35:57.880 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:35:57.880 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:35:57.880 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:35:57.880 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:35:57.880 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:35:57.880 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:35:57.880 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:35:57.880 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:35:57.880 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:35:57.880 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:35:57.880 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:35:57.880 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:35:57.881 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:35:57.881 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:35:57.881 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:35:57.882 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:35:57.882 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:35:57.882 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:35:57.882 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:35:57.882 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:35:57.882 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:35:57.882 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:35:57.882 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:35:57.882 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:35:57.882 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:35:57.882 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:35:57.882 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:35:57.882 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:35:57.882 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:35:57.883 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:35:57.906 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:35:57.906 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:35:57.906 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:35:57.907 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:35:58.046 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:35:58.046 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:35:58.046 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:35:58.046 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:35:58.046 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:35:58.046 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:35:58.046 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:35:58.047 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:35:58.047 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:35:58.047 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:35:58.047 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:35:58.047 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:35:58.048 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:35:58.049 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:35:58.049 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:35:58.049 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:35:58.319 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 11 2024-04-06T09:35:58.319 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 11 2024-04-06T09:35:58.319 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:35:58.319 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=45, tag_tid=6, entry_tid=1], [object_number=44, tag_tid=6, entry_tid=0], [object_number=47, tag_tid=5, entry_tid=22415], [object_number=46, tag_tid=5, entry_tid=22414]]], state=connected] 2024-04-06T09:35:58.319 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=46, tag_tid=5, entry_tid=18654], [object_number=45, tag_tid=5, entry_tid=18653], [object_number=44, tag_tid=5, entry_tid=18652], [object_number=47, tag_tid=5, entry_tid=18651]]], state=connected] 2024-04-06T09:35:58.324 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:35:58.325 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=45, tag_tid=6, entry_tid=1' 2024-04-06T09:35:58.327 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:35:58.327 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:35:58.327 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:35:58.327 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:35:58.327 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:35:58.327 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:35:58.327 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:35:58.327 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:35:58.327 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:35:58.327 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:35:58.327 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:35:58.328 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:35:58.329 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:35:58.329 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:35:58.329 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:35:58.379 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 11 2024-04-06T09:35:58.379 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 11 2024-04-06T09:35:58.379 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:35:58.379 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=45, tag_tid=6, entry_tid=1], [object_number=44, tag_tid=6, entry_tid=0], [object_number=47, tag_tid=5, entry_tid=22415], [object_number=46, tag_tid=5, entry_tid=22414]]], state=connected] 2024-04-06T09:35:58.379 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=46, tag_tid=5, entry_tid=18654], [object_number=45, tag_tid=5, entry_tid=18653], [object_number=44, tag_tid=5, entry_tid=18652], [object_number=47, tag_tid=5, entry_tid=18651]]], state=connected] 2024-04-06T09:35:58.383 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:35:58.385 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=46, tag_tid=5, entry_tid=18654' 2024-04-06T09:35:58.385 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=45, tag_tid=6, entry_tid=1' -a 'object_number=45, tag_tid=6, entry_tid=1' = 'object_number=46, tag_tid=5, entry_tid=18654' 2024-04-06T09:35:58.385 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=46, tag_tid=5, entry_tid=18654' '!=' 'object_number=46, tag_tid=5, entry_tid=18574' 2024-04-06T09:35:58.385 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:35:58.385 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=46, tag_tid=5, entry_tid=18654' = 'object_number=46, tag_tid=5, entry_tid=18574' 2024-04-06T09:35:58.385 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=46, tag_tid=5, entry_tid=18654' 2024-04-06T09:35:58.386 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=45, tag_tid=6, entry_tid=1' 2024-04-06T09:35:58.386 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:35:58.386 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:35:58.387 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=6 2024-04-06T09:35:58.387 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=46, tag_tid=5, entry_tid=18654' 2024-04-06T09:35:58.388 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:35:58.388 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:35:58.388 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=5 2024-04-06T09:35:58.389 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=45, tag_tid=6, entry_tid=1' 2024-04-06T09:35:58.389 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:35:58.389 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:35:58.390 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:35:58.391 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=46, tag_tid=5, entry_tid=18654' 2024-04-06T09:35:58.391 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:35:58.391 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:35:58.392 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=18654 2024-04-06T09:35:58.392 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 6 = 5 -a 1 -le 18654 2024-04-06T09:35:58.392 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:35:58.392 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:35:58.392 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:35:58.594 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:35:58.594 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:35:58.594 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:35:58.595 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:35:58.595 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:35:58.595 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:35:58.595 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:35:58.595 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:35:58.595 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:35:58.595 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:35:58.595 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:35:58.595 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:35:58.595 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:35:58.595 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:35:58.595 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:35:58.596 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:35:58.596 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:35:58.598 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:35:58.598 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:35:58.598 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:35:58.599 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:35:58.599 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:35:58.599 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:35:58.599 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:35:58.599 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:35:58.599 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:35:58.599 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:35:58.599 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:35:58.599 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:35:58.599 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:35:58.599 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:35:58.599 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:35:58.599 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:35:58.599 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:35:58.599 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:35:58.599 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:35:58.600 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:35:58.600 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:35:58.600 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:35:58.600 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:35:58.600 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:35:58.600 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:35:58.600 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:35:58.600 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:35:58.600 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:35:58.600 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:35:58.600 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:35:58.600 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:35:58.601 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:35:58.625 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:35:58.625 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:35:58.626 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:35:58.626 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:35:58.759 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:35:58.760 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:35:58.760 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:35:58.760 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:35:58.760 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:35:58.760 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:35:58.760 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:35:58.761 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:35:58.761 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:35:58.761 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:35:58.761 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:35:58.761 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:35:58.761 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:35:58.762 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:35:58.763 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:35:58.763 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:35:58.808 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 11 2024-04-06T09:35:58.808 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 11 2024-04-06T09:35:58.808 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:35:58.808 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=45, tag_tid=6, entry_tid=1], [object_number=44, tag_tid=6, entry_tid=0], [object_number=47, tag_tid=5, entry_tid=22415], [object_number=46, tag_tid=5, entry_tid=22414]]], state=connected] 2024-04-06T09:35:58.808 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=44, tag_tid=5, entry_tid=18808], [object_number=47, tag_tid=5, entry_tid=18807], [object_number=46, tag_tid=5, entry_tid=18806], [object_number=45, tag_tid=5, entry_tid=18805]]], state=connected] 2024-04-06T09:35:58.812 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:35:58.813 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=45, tag_tid=6, entry_tid=1' 2024-04-06T09:35:58.814 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:35:58.814 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:35:58.814 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:35:58.814 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:35:58.814 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:35:58.814 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:35:58.815 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:35:58.815 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:35:58.815 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:35:58.815 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:35:58.815 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:35:58.815 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:35:58.817 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:35:58.817 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:35:58.817 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:35:58.861 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 11 2024-04-06T09:35:58.861 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 11 2024-04-06T09:35:58.861 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:35:58.861 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=45, tag_tid=6, entry_tid=1], [object_number=44, tag_tid=6, entry_tid=0], [object_number=47, tag_tid=5, entry_tid=22415], [object_number=46, tag_tid=5, entry_tid=22414]]], state=connected] 2024-04-06T09:35:58.861 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=44, tag_tid=5, entry_tid=18808], [object_number=47, tag_tid=5, entry_tid=18807], [object_number=46, tag_tid=5, entry_tid=18806], [object_number=45, tag_tid=5, entry_tid=18805]]], state=connected] 2024-04-06T09:35:58.864 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:35:58.866 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=44, tag_tid=5, entry_tid=18808' 2024-04-06T09:35:58.866 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=45, tag_tid=6, entry_tid=1' -a 'object_number=45, tag_tid=6, entry_tid=1' = 'object_number=44, tag_tid=5, entry_tid=18808' 2024-04-06T09:35:58.866 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=44, tag_tid=5, entry_tid=18808' '!=' 'object_number=46, tag_tid=5, entry_tid=18654' 2024-04-06T09:35:58.867 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:35:58.867 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=44, tag_tid=5, entry_tid=18808' = 'object_number=46, tag_tid=5, entry_tid=18654' 2024-04-06T09:35:58.867 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=44, tag_tid=5, entry_tid=18808' 2024-04-06T09:35:58.867 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=45, tag_tid=6, entry_tid=1' 2024-04-06T09:35:58.867 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:35:58.867 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:35:58.867 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=6 2024-04-06T09:35:58.868 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=44, tag_tid=5, entry_tid=18808' 2024-04-06T09:35:58.868 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:35:58.868 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:35:58.869 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=5 2024-04-06T09:35:58.870 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=45, tag_tid=6, entry_tid=1' 2024-04-06T09:35:58.870 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:35:58.870 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:35:58.871 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:35:58.871 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=44, tag_tid=5, entry_tid=18808' 2024-04-06T09:35:58.871 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:35:58.872 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:35:58.873 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=18808 2024-04-06T09:35:58.873 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 6 = 5 -a 1 -le 18808 2024-04-06T09:35:58.873 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:35:58.873 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:35:58.873 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:35:59.074 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:35:59.075 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:35:59.075 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:35:59.075 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:35:59.075 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:35:59.075 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:35:59.075 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:35:59.075 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:35:59.075 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:35:59.075 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:35:59.075 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:35:59.075 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:35:59.075 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:35:59.075 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:35:59.075 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:35:59.076 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:35:59.076 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:35:59.079 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:35:59.079 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:35:59.079 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:35:59.079 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:35:59.079 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:35:59.080 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:35:59.080 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:35:59.080 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:35:59.080 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:35:59.080 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:35:59.080 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:35:59.080 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:35:59.080 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:35:59.080 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:35:59.080 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:35:59.080 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:35:59.080 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:35:59.080 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:35:59.080 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:35:59.080 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:35:59.080 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:35:59.081 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:35:59.081 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:35:59.081 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:35:59.081 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:35:59.081 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:35:59.081 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:35:59.081 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:35:59.081 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:35:59.081 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:35:59.081 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:35:59.081 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:35:59.104 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:35:59.105 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:35:59.105 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:35:59.105 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:35:59.236 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:35:59.236 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:35:59.236 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:35:59.236 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:35:59.236 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:35:59.236 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:35:59.236 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:35:59.236 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:35:59.236 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:35:59.236 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:35:59.236 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:35:59.237 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:35:59.237 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:35:59.239 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:35:59.239 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:35:59.239 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:35:59.295 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 11 2024-04-06T09:35:59.295 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 11 2024-04-06T09:35:59.295 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:35:59.295 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=45, tag_tid=6, entry_tid=1], [object_number=44, tag_tid=6, entry_tid=0], [object_number=47, tag_tid=5, entry_tid=22415], [object_number=46, tag_tid=5, entry_tid=22414]]], state=connected] 2024-04-06T09:35:59.295 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=46, tag_tid=5, entry_tid=18910], [object_number=45, tag_tid=5, entry_tid=18909], [object_number=44, tag_tid=5, entry_tid=18908], [object_number=47, tag_tid=5, entry_tid=18907]]], state=connected] 2024-04-06T09:35:59.299 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:35:59.300 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=45, tag_tid=6, entry_tid=1' 2024-04-06T09:35:59.301 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:35:59.301 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:35:59.302 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:35:59.302 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:35:59.302 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:35:59.302 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:35:59.302 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:35:59.302 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:35:59.302 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:35:59.302 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:35:59.302 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:35:59.302 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:35:59.303 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:35:59.303 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:35:59.303 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:35:59.348 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 11 2024-04-06T09:35:59.348 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 11 2024-04-06T09:35:59.349 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:35:59.349 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=45, tag_tid=6, entry_tid=1], [object_number=44, tag_tid=6, entry_tid=0], [object_number=47, tag_tid=5, entry_tid=22415], [object_number=46, tag_tid=5, entry_tid=22414]]], state=connected] 2024-04-06T09:35:59.349 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=46, tag_tid=5, entry_tid=18910], [object_number=45, tag_tid=5, entry_tid=18909], [object_number=44, tag_tid=5, entry_tid=18908], [object_number=47, tag_tid=5, entry_tid=18907]]], state=connected] 2024-04-06T09:35:59.352 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:35:59.354 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=46, tag_tid=5, entry_tid=18910' 2024-04-06T09:35:59.354 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=45, tag_tid=6, entry_tid=1' -a 'object_number=45, tag_tid=6, entry_tid=1' = 'object_number=46, tag_tid=5, entry_tid=18910' 2024-04-06T09:35:59.354 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=46, tag_tid=5, entry_tid=18910' '!=' 'object_number=44, tag_tid=5, entry_tid=18808' 2024-04-06T09:35:59.354 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:35:59.354 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=46, tag_tid=5, entry_tid=18910' = 'object_number=44, tag_tid=5, entry_tid=18808' 2024-04-06T09:35:59.354 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=46, tag_tid=5, entry_tid=18910' 2024-04-06T09:35:59.355 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=45, tag_tid=6, entry_tid=1' 2024-04-06T09:35:59.355 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:35:59.358 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:35:59.358 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=6 2024-04-06T09:35:59.358 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=46, tag_tid=5, entry_tid=18910' 2024-04-06T09:35:59.358 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:35:59.358 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:35:59.358 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=5 2024-04-06T09:35:59.358 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=45, tag_tid=6, entry_tid=1' 2024-04-06T09:35:59.359 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:35:59.359 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:35:59.359 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:35:59.360 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=46, tag_tid=5, entry_tid=18910' 2024-04-06T09:35:59.360 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:35:59.360 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:35:59.361 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=18910 2024-04-06T09:35:59.361 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 6 = 5 -a 1 -le 18910 2024-04-06T09:35:59.361 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:35:59.361 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:35:59.361 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:35:59.563 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:35:59.563 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:35:59.563 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:35:59.563 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:35:59.563 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:35:59.563 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:35:59.563 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:35:59.563 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:35:59.564 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:35:59.564 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:35:59.564 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:35:59.564 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:35:59.564 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:35:59.564 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:35:59.564 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:35:59.564 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:35:59.565 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:35:59.566 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:35:59.566 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:35:59.566 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:35:59.566 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:35:59.566 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:35:59.566 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:35:59.567 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:35:59.567 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:35:59.567 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:35:59.567 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:35:59.567 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:35:59.567 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:35:59.567 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:35:59.567 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:35:59.567 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:35:59.567 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:35:59.568 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:35:59.568 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:35:59.568 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:35:59.568 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:35:59.568 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:35:59.568 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:35:59.568 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:35:59.569 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:35:59.569 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:35:59.569 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:35:59.569 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:35:59.569 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:35:59.569 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:35:59.569 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:35:59.569 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:35:59.569 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:35:59.594 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:35:59.594 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:35:59.594 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:35:59.594 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:35:59.719 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:35:59.721 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:35:59.721 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:35:59.721 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:35:59.721 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:35:59.721 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:35:59.721 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:35:59.721 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:35:59.721 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:35:59.721 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:35:59.722 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:35:59.722 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:35:59.722 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:35:59.723 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:35:59.724 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:35:59.724 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:35:59.774 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 11 2024-04-06T09:35:59.774 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 11 2024-04-06T09:35:59.774 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:35:59.774 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=45, tag_tid=6, entry_tid=1], [object_number=44, tag_tid=6, entry_tid=0], [object_number=47, tag_tid=5, entry_tid=22415], [object_number=46, tag_tid=5, entry_tid=22414]]], state=connected] 2024-04-06T09:35:59.774 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=44, tag_tid=5, entry_tid=19004], [object_number=47, tag_tid=5, entry_tid=19003], [object_number=46, tag_tid=5, entry_tid=19002], [object_number=45, tag_tid=5, entry_tid=19001]]], state=connected] 2024-04-06T09:35:59.777 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:35:59.778 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=45, tag_tid=6, entry_tid=1' 2024-04-06T09:35:59.779 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:35:59.779 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:35:59.780 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:35:59.780 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:35:59.780 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:35:59.780 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:35:59.780 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:35:59.780 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:35:59.780 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:35:59.780 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:35:59.780 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:35:59.780 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:35:59.781 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:35:59.781 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:35:59.781 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:35:59.826 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 11 2024-04-06T09:35:59.826 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 11 2024-04-06T09:35:59.826 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:35:59.827 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=45, tag_tid=6, entry_tid=1], [object_number=44, tag_tid=6, entry_tid=0], [object_number=47, tag_tid=5, entry_tid=22415], [object_number=46, tag_tid=5, entry_tid=22414]]], state=connected] 2024-04-06T09:35:59.827 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=44, tag_tid=5, entry_tid=19004], [object_number=47, tag_tid=5, entry_tid=19003], [object_number=46, tag_tid=5, entry_tid=19002], [object_number=45, tag_tid=5, entry_tid=19001]]], state=connected] 2024-04-06T09:35:59.830 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:35:59.832 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=44, tag_tid=5, entry_tid=19004' 2024-04-06T09:35:59.832 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=45, tag_tid=6, entry_tid=1' -a 'object_number=45, tag_tid=6, entry_tid=1' = 'object_number=44, tag_tid=5, entry_tid=19004' 2024-04-06T09:35:59.832 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=44, tag_tid=5, entry_tid=19004' '!=' 'object_number=46, tag_tid=5, entry_tid=18910' 2024-04-06T09:35:59.832 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:35:59.832 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=44, tag_tid=5, entry_tid=19004' = 'object_number=46, tag_tid=5, entry_tid=18910' 2024-04-06T09:35:59.832 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=44, tag_tid=5, entry_tid=19004' 2024-04-06T09:35:59.832 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=45, tag_tid=6, entry_tid=1' 2024-04-06T09:35:59.832 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:35:59.832 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:35:59.833 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=6 2024-04-06T09:35:59.834 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=44, tag_tid=5, entry_tid=19004' 2024-04-06T09:35:59.834 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:35:59.834 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:35:59.835 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=5 2024-04-06T09:35:59.835 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=45, tag_tid=6, entry_tid=1' 2024-04-06T09:35:59.835 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:35:59.836 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:35:59.836 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:35:59.837 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=44, tag_tid=5, entry_tid=19004' 2024-04-06T09:35:59.837 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:35:59.837 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:35:59.838 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=19004 2024-04-06T09:35:59.839 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 6 = 5 -a 1 -le 19004 2024-04-06T09:35:59.839 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:35:59.839 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:35:59.839 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:36:00.040 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:36:00.041 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:36:00.041 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:36:00.041 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:36:00.041 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:36:00.041 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:36:00.041 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:36:00.041 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:36:00.041 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:36:00.041 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:36:00.041 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:36:00.041 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:36:00.042 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:36:00.042 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:36:00.042 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:36:00.042 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:36:00.042 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:36:00.047 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:36:00.047 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:36:00.047 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:36:00.047 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:36:00.047 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:36:00.048 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:36:00.048 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:36:00.048 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:36:00.048 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:36:00.048 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:36:00.048 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:36:00.048 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:36:00.048 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:36:00.048 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:36:00.048 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:36:00.048 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:36:00.048 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:36:00.048 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:36:00.048 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:36:00.049 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:36:00.049 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:36:00.049 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:36:00.049 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:36:00.049 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:36:00.049 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:36:00.049 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:36:00.049 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:36:00.049 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:36:00.049 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:36:00.049 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:36:00.049 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:36:00.049 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:36:00.069 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:36:00.070 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:36:00.070 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:36:00.070 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:36:00.199 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:36:00.200 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:36:00.200 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:36:00.200 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:36:00.200 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:36:00.200 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:36:00.201 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:36:00.201 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:36:00.201 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:36:00.201 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:36:00.201 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:36:00.201 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:36:00.202 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:36:00.203 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:36:00.203 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:36:00.203 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:36:00.247 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 11 2024-04-06T09:36:00.248 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 11 2024-04-06T09:36:00.248 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:36:00.248 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=45, tag_tid=6, entry_tid=1], [object_number=44, tag_tid=6, entry_tid=0], [object_number=47, tag_tid=5, entry_tid=22415], [object_number=46, tag_tid=5, entry_tid=22414]]], state=connected] 2024-04-06T09:36:00.248 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=44, tag_tid=5, entry_tid=19096], [object_number=47, tag_tid=5, entry_tid=19095], [object_number=46, tag_tid=5, entry_tid=19094], [object_number=45, tag_tid=5, entry_tid=19093]]], state=connected] 2024-04-06T09:36:00.251 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:36:00.253 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=45, tag_tid=6, entry_tid=1' 2024-04-06T09:36:00.254 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:36:00.254 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:36:00.254 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:36:00.254 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:36:00.254 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:36:00.254 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:36:00.254 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:36:00.254 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:36:00.254 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:36:00.254 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:36:00.254 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:36:00.254 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:36:00.255 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:36:00.255 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:36:00.255 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:36:00.299 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 11 2024-04-06T09:36:00.299 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 11 2024-04-06T09:36:00.299 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:36:00.299 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=45, tag_tid=6, entry_tid=1], [object_number=44, tag_tid=6, entry_tid=0], [object_number=47, tag_tid=5, entry_tid=22415], [object_number=46, tag_tid=5, entry_tid=22414]]], state=connected] 2024-04-06T09:36:00.299 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=44, tag_tid=5, entry_tid=19096], [object_number=47, tag_tid=5, entry_tid=19095], [object_number=46, tag_tid=5, entry_tid=19094], [object_number=45, tag_tid=5, entry_tid=19093]]], state=connected] 2024-04-06T09:36:00.303 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:36:00.305 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=44, tag_tid=5, entry_tid=19096' 2024-04-06T09:36:00.305 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=45, tag_tid=6, entry_tid=1' -a 'object_number=45, tag_tid=6, entry_tid=1' = 'object_number=44, tag_tid=5, entry_tid=19096' 2024-04-06T09:36:00.305 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=44, tag_tid=5, entry_tid=19096' '!=' 'object_number=44, tag_tid=5, entry_tid=19004' 2024-04-06T09:36:00.305 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:36:00.305 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=44, tag_tid=5, entry_tid=19096' = 'object_number=44, tag_tid=5, entry_tid=19004' 2024-04-06T09:36:00.305 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=44, tag_tid=5, entry_tid=19096' 2024-04-06T09:36:00.306 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=45, tag_tid=6, entry_tid=1' 2024-04-06T09:36:00.306 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:36:00.306 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:36:00.306 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=6 2024-04-06T09:36:00.307 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=44, tag_tid=5, entry_tid=19096' 2024-04-06T09:36:00.307 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:36:00.307 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:36:00.308 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=5 2024-04-06T09:36:00.308 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=45, tag_tid=6, entry_tid=1' 2024-04-06T09:36:00.309 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:36:00.309 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:36:00.309 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:36:00.310 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=44, tag_tid=5, entry_tid=19096' 2024-04-06T09:36:00.310 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:36:00.310 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:36:00.311 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=19096 2024-04-06T09:36:00.312 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 6 = 5 -a 1 -le 19096 2024-04-06T09:36:00.312 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:36:00.312 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:36:00.312 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:36:00.514 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:36:00.514 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:36:00.514 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:36:00.514 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:36:00.514 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:36:00.514 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:36:00.514 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:36:00.514 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:36:00.514 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:36:00.514 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:36:00.514 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:36:00.514 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:36:00.514 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:36:00.515 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:36:00.515 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:36:00.515 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:36:00.515 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:36:00.518 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:36:00.518 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:36:00.518 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:36:00.519 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:36:00.519 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:36:00.519 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:36:00.519 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:36:00.519 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:36:00.519 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:36:00.519 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:36:00.519 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:36:00.519 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:36:00.519 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:36:00.519 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:36:00.519 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:36:00.519 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:36:00.519 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:36:00.519 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:36:00.520 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:36:00.520 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:36:00.520 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:36:00.520 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:36:00.520 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:36:00.520 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:36:00.520 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:36:00.520 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:36:00.520 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:36:00.520 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:36:00.520 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:36:00.520 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:36:00.520 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:36:00.521 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:36:00.546 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:36:00.546 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:36:00.546 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:36:00.547 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:36:00.674 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:36:00.675 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:36:00.675 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:36:00.675 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:36:00.675 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:36:00.675 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:36:00.675 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:36:00.675 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:36:00.676 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:36:00.676 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:36:00.676 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:36:00.676 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:36:00.676 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:36:00.678 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:36:00.678 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:36:00.678 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:36:00.723 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 11 2024-04-06T09:36:00.723 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 11 2024-04-06T09:36:00.723 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:36:00.723 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=45, tag_tid=6, entry_tid=1], [object_number=44, tag_tid=6, entry_tid=0], [object_number=47, tag_tid=5, entry_tid=22415], [object_number=46, tag_tid=5, entry_tid=22414]]], state=connected] 2024-04-06T09:36:00.723 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=46, tag_tid=5, entry_tid=19178], [object_number=45, tag_tid=5, entry_tid=19177], [object_number=44, tag_tid=5, entry_tid=19176], [object_number=47, tag_tid=5, entry_tid=19175]]], state=connected] 2024-04-06T09:36:00.726 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:36:00.728 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=45, tag_tid=6, entry_tid=1' 2024-04-06T09:36:00.729 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:36:00.729 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:36:00.729 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:36:00.729 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:36:00.729 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:36:00.729 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:36:00.729 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:36:00.729 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:36:00.729 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:36:00.730 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:36:00.730 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:36:00.730 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:36:00.730 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:36:00.731 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:36:00.731 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:36:00.775 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 11 2024-04-06T09:36:00.775 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 11 2024-04-06T09:36:00.775 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:36:00.775 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=45, tag_tid=6, entry_tid=1], [object_number=44, tag_tid=6, entry_tid=0], [object_number=47, tag_tid=5, entry_tid=22415], [object_number=46, tag_tid=5, entry_tid=22414]]], state=connected] 2024-04-06T09:36:00.775 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=46, tag_tid=5, entry_tid=19178], [object_number=45, tag_tid=5, entry_tid=19177], [object_number=44, tag_tid=5, entry_tid=19176], [object_number=47, tag_tid=5, entry_tid=19175]]], state=connected] 2024-04-06T09:36:00.779 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:36:00.781 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=46, tag_tid=5, entry_tid=19178' 2024-04-06T09:36:00.781 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=45, tag_tid=6, entry_tid=1' -a 'object_number=45, tag_tid=6, entry_tid=1' = 'object_number=46, tag_tid=5, entry_tid=19178' 2024-04-06T09:36:00.781 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=46, tag_tid=5, entry_tid=19178' '!=' 'object_number=44, tag_tid=5, entry_tid=19096' 2024-04-06T09:36:00.781 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:36:00.781 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=46, tag_tid=5, entry_tid=19178' = 'object_number=44, tag_tid=5, entry_tid=19096' 2024-04-06T09:36:00.781 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=46, tag_tid=5, entry_tid=19178' 2024-04-06T09:36:00.781 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=45, tag_tid=6, entry_tid=1' 2024-04-06T09:36:00.782 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:36:00.782 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:36:00.782 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=6 2024-04-06T09:36:00.783 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=46, tag_tid=5, entry_tid=19178' 2024-04-06T09:36:00.783 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:36:00.783 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:36:00.784 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=5 2024-04-06T09:36:00.784 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=45, tag_tid=6, entry_tid=1' 2024-04-06T09:36:00.784 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:36:00.785 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:36:00.785 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:36:00.786 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=46, tag_tid=5, entry_tid=19178' 2024-04-06T09:36:00.786 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:36:00.786 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:36:00.787 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=19178 2024-04-06T09:36:00.787 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 6 = 5 -a 1 -le 19178 2024-04-06T09:36:00.787 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:36:00.788 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:36:00.788 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:36:00.989 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:36:00.989 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:36:00.989 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:36:00.989 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:36:00.990 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:36:00.990 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:36:00.990 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:36:00.990 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:36:00.990 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:36:00.990 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:36:00.990 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:36:00.990 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:36:00.990 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:36:00.990 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:36:00.990 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:36:00.991 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:36:00.991 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:36:00.994 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:36:00.994 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:36:00.994 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:36:00.994 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:36:00.994 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:36:00.994 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:36:00.994 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:36:00.995 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:36:00.995 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:36:00.995 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:36:00.995 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:36:00.995 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:36:00.995 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:36:00.995 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:36:00.995 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:36:00.995 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:36:00.995 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:36:00.995 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:36:00.995 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:36:00.995 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:36:00.996 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:36:00.996 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:36:00.996 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:36:00.996 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:36:00.996 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:36:00.996 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:36:00.996 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:36:00.996 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:36:00.996 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:36:00.996 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:36:00.996 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:36:00.996 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:36:01.018 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:36:01.018 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:36:01.018 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:36:01.019 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:36:01.146 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:36:01.148 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:36:01.148 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:36:01.148 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:36:01.148 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:36:01.148 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:36:01.148 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:36:01.148 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:36:01.148 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:36:01.148 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:36:01.149 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:36:01.149 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:36:01.149 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:36:01.150 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:36:01.151 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:36:01.151 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:36:01.195 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 11 2024-04-06T09:36:01.195 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 11 2024-04-06T09:36:01.195 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:36:01.195 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=45, tag_tid=6, entry_tid=1], [object_number=44, tag_tid=6, entry_tid=0], [object_number=47, tag_tid=5, entry_tid=22415], [object_number=46, tag_tid=5, entry_tid=22414]]], state=connected] 2024-04-06T09:36:01.195 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=46, tag_tid=5, entry_tid=19266], [object_number=45, tag_tid=5, entry_tid=19265], [object_number=44, tag_tid=5, entry_tid=19264], [object_number=47, tag_tid=5, entry_tid=19263]]], state=connected] 2024-04-06T09:36:01.198 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:36:01.201 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=45, tag_tid=6, entry_tid=1' 2024-04-06T09:36:01.201 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:36:01.201 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:36:01.201 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:36:01.201 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:36:01.201 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:36:01.201 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:36:01.201 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:36:01.201 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:36:01.201 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:36:01.201 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:36:01.201 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:36:01.201 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:36:01.202 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:36:01.202 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:36:01.202 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:36:01.247 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 11 2024-04-06T09:36:01.247 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 11 2024-04-06T09:36:01.247 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:36:01.247 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=45, tag_tid=6, entry_tid=1], [object_number=44, tag_tid=6, entry_tid=0], [object_number=47, tag_tid=5, entry_tid=22415], [object_number=46, tag_tid=5, entry_tid=22414]]], state=connected] 2024-04-06T09:36:01.247 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=46, tag_tid=5, entry_tid=19266], [object_number=45, tag_tid=5, entry_tid=19265], [object_number=44, tag_tid=5, entry_tid=19264], [object_number=47, tag_tid=5, entry_tid=19263]]], state=connected] 2024-04-06T09:36:01.250 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:36:01.253 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=46, tag_tid=5, entry_tid=19266' 2024-04-06T09:36:01.253 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=45, tag_tid=6, entry_tid=1' -a 'object_number=45, tag_tid=6, entry_tid=1' = 'object_number=46, tag_tid=5, entry_tid=19266' 2024-04-06T09:36:01.253 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=46, tag_tid=5, entry_tid=19266' '!=' 'object_number=46, tag_tid=5, entry_tid=19178' 2024-04-06T09:36:01.253 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:36:01.253 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=46, tag_tid=5, entry_tid=19266' = 'object_number=46, tag_tid=5, entry_tid=19178' 2024-04-06T09:36:01.253 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=46, tag_tid=5, entry_tid=19266' 2024-04-06T09:36:01.253 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=45, tag_tid=6, entry_tid=1' 2024-04-06T09:36:01.253 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:36:01.253 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:36:01.254 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=6 2024-04-06T09:36:01.254 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=46, tag_tid=5, entry_tid=19266' 2024-04-06T09:36:01.255 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:36:01.255 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:36:01.255 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=5 2024-04-06T09:36:01.256 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=45, tag_tid=6, entry_tid=1' 2024-04-06T09:36:01.256 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:36:01.256 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:36:01.258 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:36:01.258 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=46, tag_tid=5, entry_tid=19266' 2024-04-06T09:36:01.258 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:36:01.258 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:36:01.259 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=19266 2024-04-06T09:36:01.259 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 6 = 5 -a 1 -le 19266 2024-04-06T09:36:01.259 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:36:01.259 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:36:01.259 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:36:01.461 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:36:01.461 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:36:01.461 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:36:01.462 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:36:01.462 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:36:01.462 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:36:01.462 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:36:01.462 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:36:01.462 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:36:01.462 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:36:01.462 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:36:01.462 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:36:01.462 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:36:01.462 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:36:01.462 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:36:01.463 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:36:01.463 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:36:01.465 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:36:01.466 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:36:01.466 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:36:01.466 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:36:01.466 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:36:01.466 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:36:01.466 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:36:01.466 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:36:01.466 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:36:01.466 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:36:01.466 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:36:01.466 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:36:01.466 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:36:01.466 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:36:01.466 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:36:01.467 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:36:01.467 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:36:01.467 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:36:01.468 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:36:01.468 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:36:01.468 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:36:01.468 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:36:01.468 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:36:01.468 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:36:01.468 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:36:01.468 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:36:01.468 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:36:01.468 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:36:01.468 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:36:01.468 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:36:01.468 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:36:01.468 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:36:01.493 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:36:01.493 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:36:01.493 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:36:01.493 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:36:01.627 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:36:01.628 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:36:01.628 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:36:01.628 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:36:01.628 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:36:01.628 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:36:01.628 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:36:01.628 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:36:01.628 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:36:01.629 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:36:01.629 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:36:01.629 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:36:01.629 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:36:01.631 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:36:01.631 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:36:01.631 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:36:01.688 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 11 2024-04-06T09:36:01.688 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 11 2024-04-06T09:36:01.688 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:36:01.688 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=45, tag_tid=6, entry_tid=1], [object_number=44, tag_tid=6, entry_tid=0], [object_number=47, tag_tid=5, entry_tid=22415], [object_number=46, tag_tid=5, entry_tid=22414]]], state=connected] 2024-04-06T09:36:01.688 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=44, tag_tid=5, entry_tid=19368], [object_number=47, tag_tid=5, entry_tid=19367], [object_number=46, tag_tid=5, entry_tid=19366], [object_number=45, tag_tid=5, entry_tid=19365]]], state=connected] 2024-04-06T09:36:01.692 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:36:01.693 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=45, tag_tid=6, entry_tid=1' 2024-04-06T09:36:01.694 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:36:01.695 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:36:01.695 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:36:01.695 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:36:01.695 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:36:01.695 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:36:01.695 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:36:01.695 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:36:01.695 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:36:01.695 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:36:01.695 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:36:01.695 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:36:01.696 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:36:01.696 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:36:01.696 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:36:01.742 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 11 2024-04-06T09:36:01.742 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 11 2024-04-06T09:36:01.742 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:36:01.742 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=45, tag_tid=6, entry_tid=1], [object_number=44, tag_tid=6, entry_tid=0], [object_number=47, tag_tid=5, entry_tid=22415], [object_number=46, tag_tid=5, entry_tid=22414]]], state=connected] 2024-04-06T09:36:01.742 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=44, tag_tid=5, entry_tid=19368], [object_number=47, tag_tid=5, entry_tid=19367], [object_number=46, tag_tid=5, entry_tid=19366], [object_number=45, tag_tid=5, entry_tid=19365]]], state=connected] 2024-04-06T09:36:01.746 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:36:01.748 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=44, tag_tid=5, entry_tid=19368' 2024-04-06T09:36:01.748 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=45, tag_tid=6, entry_tid=1' -a 'object_number=45, tag_tid=6, entry_tid=1' = 'object_number=44, tag_tid=5, entry_tid=19368' 2024-04-06T09:36:01.748 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=44, tag_tid=5, entry_tid=19368' '!=' 'object_number=46, tag_tid=5, entry_tid=19266' 2024-04-06T09:36:01.748 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:36:01.748 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=44, tag_tid=5, entry_tid=19368' = 'object_number=46, tag_tid=5, entry_tid=19266' 2024-04-06T09:36:01.748 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=44, tag_tid=5, entry_tid=19368' 2024-04-06T09:36:01.748 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=45, tag_tid=6, entry_tid=1' 2024-04-06T09:36:01.749 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:36:01.749 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:36:01.749 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=6 2024-04-06T09:36:01.750 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=44, tag_tid=5, entry_tid=19368' 2024-04-06T09:36:01.750 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:36:01.750 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:36:01.752 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=5 2024-04-06T09:36:01.752 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=45, tag_tid=6, entry_tid=1' 2024-04-06T09:36:01.752 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:36:01.752 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:36:01.752 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:36:01.753 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=44, tag_tid=5, entry_tid=19368' 2024-04-06T09:36:01.753 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:36:01.753 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:36:01.754 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=19368 2024-04-06T09:36:01.755 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 6 = 5 -a 1 -le 19368 2024-04-06T09:36:01.755 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:36:01.755 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:36:01.755 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:36:01.956 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:36:01.956 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:36:01.956 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:36:01.956 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:36:01.956 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:36:01.956 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:36:01.956 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:36:01.956 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:36:01.957 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:36:01.957 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:36:01.957 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:36:01.957 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:36:01.957 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:36:01.957 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:36:01.957 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:36:01.957 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:36:01.958 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:36:01.960 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:36:01.960 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:36:01.960 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:36:01.960 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:36:01.960 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:36:01.960 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:36:01.960 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:36:01.961 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:36:01.961 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:36:01.961 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:36:01.961 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:36:01.961 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:36:01.961 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:36:01.961 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:36:01.961 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:36:01.961 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:36:01.961 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:36:01.961 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:36:01.961 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:36:01.961 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:36:01.961 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:36:01.961 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:36:01.962 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:36:01.962 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:36:01.962 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:36:01.962 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:36:01.962 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:36:01.962 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:36:01.962 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:36:01.962 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:36:01.962 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:36:01.962 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:36:01.974 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:36:01.974+0000 7f2e84d66640 -1 reset not still connected to 0x560d27073d40 2024-04-06T09:36:01.987 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:36:01.987 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:36:01.987 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:36:01.987 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:36:02.118 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:36:02.119 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:36:02.119 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:36:02.119 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:36:02.119 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:36:02.119 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:36:02.119 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:36:02.119 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:36:02.120 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:36:02.120 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:36:02.120 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:36:02.120 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:36:02.120 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:36:02.122 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:36:02.123 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:36:02.123 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:36:02.174 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 11 2024-04-06T09:36:02.174 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 11 2024-04-06T09:36:02.174 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:36:02.175 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=45, tag_tid=6, entry_tid=1], [object_number=44, tag_tid=6, entry_tid=0], [object_number=47, tag_tid=5, entry_tid=22415], [object_number=46, tag_tid=5, entry_tid=22414]]], state=connected] 2024-04-06T09:36:02.175 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=45, tag_tid=5, entry_tid=19461], [object_number=44, tag_tid=5, entry_tid=19460], [object_number=47, tag_tid=5, entry_tid=19459], [object_number=46, tag_tid=5, entry_tid=19458]]], state=connected] 2024-04-06T09:36:02.257 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:36:02.259 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=45, tag_tid=6, entry_tid=1' 2024-04-06T09:36:02.260 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:36:02.260 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:36:02.260 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:36:02.260 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:36:02.260 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:36:02.260 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:36:02.260 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:36:02.260 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:36:02.260 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:36:02.261 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:36:02.261 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:36:02.261 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:36:02.262 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:36:02.263 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:36:02.263 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:36:02.313 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 11 2024-04-06T09:36:02.313 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 11 2024-04-06T09:36:02.313 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:36:02.313 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=45, tag_tid=6, entry_tid=1], [object_number=44, tag_tid=6, entry_tid=0], [object_number=47, tag_tid=5, entry_tid=22415], [object_number=46, tag_tid=5, entry_tid=22414]]], state=connected] 2024-04-06T09:36:02.313 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=45, tag_tid=5, entry_tid=19461], [object_number=44, tag_tid=5, entry_tid=19460], [object_number=47, tag_tid=5, entry_tid=19459], [object_number=46, tag_tid=5, entry_tid=19458]]], state=connected] 2024-04-06T09:36:02.317 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:36:02.319 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=45, tag_tid=5, entry_tid=19461' 2024-04-06T09:36:02.319 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=45, tag_tid=6, entry_tid=1' -a 'object_number=45, tag_tid=6, entry_tid=1' = 'object_number=45, tag_tid=5, entry_tid=19461' 2024-04-06T09:36:02.319 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=45, tag_tid=5, entry_tid=19461' '!=' 'object_number=44, tag_tid=5, entry_tid=19368' 2024-04-06T09:36:02.320 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:36:02.320 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=45, tag_tid=5, entry_tid=19461' = 'object_number=44, tag_tid=5, entry_tid=19368' 2024-04-06T09:36:02.320 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=45, tag_tid=5, entry_tid=19461' 2024-04-06T09:36:02.320 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=45, tag_tid=6, entry_tid=1' 2024-04-06T09:36:02.320 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:36:02.320 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:36:02.320 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=6 2024-04-06T09:36:02.321 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=45, tag_tid=5, entry_tid=19461' 2024-04-06T09:36:02.321 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:36:02.321 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:36:02.323 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=5 2024-04-06T09:36:02.323 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=45, tag_tid=6, entry_tid=1' 2024-04-06T09:36:02.323 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:36:02.323 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:36:02.324 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:36:02.325 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=45, tag_tid=5, entry_tid=19461' 2024-04-06T09:36:02.325 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:36:02.325 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:36:02.326 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=19461 2024-04-06T09:36:02.327 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 6 = 5 -a 1 -le 19461 2024-04-06T09:36:02.327 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:36:02.327 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:36:02.327 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:36:02.528 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:36:02.529 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:36:02.529 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:36:02.529 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:36:02.529 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:36:02.529 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:36:02.529 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:36:02.529 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:36:02.529 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:36:02.529 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:36:02.529 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:36:02.529 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:36:02.529 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:36:02.530 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:36:02.530 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:36:02.530 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:36:02.530 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:36:02.532 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:36:02.532 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:36:02.532 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:36:02.532 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:36:02.532 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:36:02.532 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:36:02.532 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:36:02.532 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:36:02.533 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:36:02.533 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:36:02.533 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:36:02.533 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:36:02.533 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:36:02.533 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:36:02.533 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:36:02.533 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:36:02.534 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:36:02.534 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:36:02.535 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:36:02.535 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:36:02.535 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:36:02.535 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:36:02.535 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:36:02.535 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:36:02.535 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:36:02.535 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:36:02.535 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:36:02.535 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:36:02.535 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:36:02.535 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:36:02.535 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:36:02.535 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:36:02.560 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:36:02.561 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:36:02.561 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:36:02.561 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:36:02.687 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:36:02.688 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:36:02.688 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:36:02.689 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:36:02.689 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:36:02.689 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:36:02.689 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:36:02.689 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:36:02.689 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:36:02.689 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:36:02.690 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:36:02.690 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:36:02.690 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:36:02.691 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:36:02.691 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:36:02.692 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:36:02.738 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 11 2024-04-06T09:36:02.738 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 11 2024-04-06T09:36:02.738 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:36:02.739 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=45, tag_tid=6, entry_tid=1], [object_number=44, tag_tid=6, entry_tid=0], [object_number=47, tag_tid=5, entry_tid=22415], [object_number=46, tag_tid=5, entry_tid=22414]]], state=connected] 2024-04-06T09:36:02.739 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=44, tag_tid=5, entry_tid=19576], [object_number=47, tag_tid=5, entry_tid=19575], [object_number=46, tag_tid=5, entry_tid=19574], [object_number=45, tag_tid=5, entry_tid=19573]]], state=connected] 2024-04-06T09:36:02.742 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:36:02.743 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=45, tag_tid=6, entry_tid=1' 2024-04-06T09:36:02.744 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:36:02.744 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:36:02.744 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:36:02.745 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:36:02.745 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:36:02.745 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:36:02.745 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:36:02.745 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:36:02.745 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:36:02.745 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:36:02.745 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:36:02.745 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:36:02.746 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:36:02.746 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:36:02.746 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:36:02.790 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 11 2024-04-06T09:36:02.791 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 11 2024-04-06T09:36:02.791 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:36:02.791 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=45, tag_tid=6, entry_tid=1], [object_number=44, tag_tid=6, entry_tid=0], [object_number=47, tag_tid=5, entry_tid=22415], [object_number=46, tag_tid=5, entry_tid=22414]]], state=connected] 2024-04-06T09:36:02.791 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=44, tag_tid=5, entry_tid=19576], [object_number=47, tag_tid=5, entry_tid=19575], [object_number=46, tag_tid=5, entry_tid=19574], [object_number=45, tag_tid=5, entry_tid=19573]]], state=connected] 2024-04-06T09:36:02.794 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:36:02.796 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=44, tag_tid=5, entry_tid=19576' 2024-04-06T09:36:02.797 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=45, tag_tid=6, entry_tid=1' -a 'object_number=45, tag_tid=6, entry_tid=1' = 'object_number=44, tag_tid=5, entry_tid=19576' 2024-04-06T09:36:02.797 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=44, tag_tid=5, entry_tid=19576' '!=' 'object_number=45, tag_tid=5, entry_tid=19461' 2024-04-06T09:36:02.797 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:36:02.797 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=44, tag_tid=5, entry_tid=19576' = 'object_number=45, tag_tid=5, entry_tid=19461' 2024-04-06T09:36:02.797 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=44, tag_tid=5, entry_tid=19576' 2024-04-06T09:36:02.797 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=45, tag_tid=6, entry_tid=1' 2024-04-06T09:36:02.797 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:36:02.797 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:36:02.798 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=6 2024-04-06T09:36:02.798 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=44, tag_tid=5, entry_tid=19576' 2024-04-06T09:36:02.798 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:36:02.799 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:36:02.799 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=5 2024-04-06T09:36:02.800 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=45, tag_tid=6, entry_tid=1' 2024-04-06T09:36:02.800 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:36:02.800 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:36:02.801 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:36:02.802 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=44, tag_tid=5, entry_tid=19576' 2024-04-06T09:36:02.802 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:36:02.802 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:36:02.803 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=19576 2024-04-06T09:36:02.803 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 6 = 5 -a 1 -le 19576 2024-04-06T09:36:02.803 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:36:02.803 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:36:02.803 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:36:03.005 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:36:03.005 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:36:03.005 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:36:03.005 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:36:03.005 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:36:03.005 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:36:03.005 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:36:03.005 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:36:03.006 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:36:03.006 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:36:03.006 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:36:03.006 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:36:03.006 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:36:03.006 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:36:03.006 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:36:03.006 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:36:03.007 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:36:03.011 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:36:03.011 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:36:03.011 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:36:03.011 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:36:03.011 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:36:03.011 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:36:03.011 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:36:03.011 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:36:03.011 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:36:03.011 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:36:03.011 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:36:03.011 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:36:03.011 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:36:03.012 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:36:03.012 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:36:03.012 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:36:03.012 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:36:03.012 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:36:03.012 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:36:03.012 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:36:03.012 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:36:03.012 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:36:03.012 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:36:03.012 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:36:03.012 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:36:03.012 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:36:03.012 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:36:03.013 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:36:03.013 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:36:03.013 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:36:03.013 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:36:03.013 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:36:03.035 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:36:03.036 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:36:03.036 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:36:03.036 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:36:03.169 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:36:03.171 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:36:03.171 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:36:03.171 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:36:03.171 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:36:03.171 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:36:03.171 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:36:03.171 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:36:03.171 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:36:03.171 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:36:03.171 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:36:03.171 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:36:03.171 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:36:03.174 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:36:03.174 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:36:03.174 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:36:03.218 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 11 2024-04-06T09:36:03.218 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 11 2024-04-06T09:36:03.219 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:36:03.219 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=45, tag_tid=6, entry_tid=1], [object_number=44, tag_tid=6, entry_tid=0], [object_number=47, tag_tid=5, entry_tid=22415], [object_number=46, tag_tid=5, entry_tid=22414]]], state=connected] 2024-04-06T09:36:03.219 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=47, tag_tid=5, entry_tid=19663], [object_number=46, tag_tid=5, entry_tid=19662], [object_number=45, tag_tid=5, entry_tid=19661], [object_number=44, tag_tid=5, entry_tid=19660]]], state=connected] 2024-04-06T09:36:03.222 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:36:03.223 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=45, tag_tid=6, entry_tid=1' 2024-04-06T09:36:03.225 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:36:03.225 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:36:03.225 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:36:03.225 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:36:03.225 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:36:03.225 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:36:03.225 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:36:03.225 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:36:03.225 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:36:03.225 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:36:03.225 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:36:03.225 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:36:03.226 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:36:03.226 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:36:03.226 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:36:03.271 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 11 2024-04-06T09:36:03.271 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 11 2024-04-06T09:36:03.271 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:36:03.271 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=45, tag_tid=6, entry_tid=1], [object_number=44, tag_tid=6, entry_tid=0], [object_number=47, tag_tid=5, entry_tid=22415], [object_number=46, tag_tid=5, entry_tid=22414]]], state=connected] 2024-04-06T09:36:03.271 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=47, tag_tid=5, entry_tid=19663], [object_number=46, tag_tid=5, entry_tid=19662], [object_number=45, tag_tid=5, entry_tid=19661], [object_number=44, tag_tid=5, entry_tid=19660]]], state=connected] 2024-04-06T09:36:03.275 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:36:03.276 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=47, tag_tid=5, entry_tid=19663' 2024-04-06T09:36:03.277 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=45, tag_tid=6, entry_tid=1' -a 'object_number=45, tag_tid=6, entry_tid=1' = 'object_number=47, tag_tid=5, entry_tid=19663' 2024-04-06T09:36:03.277 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=47, tag_tid=5, entry_tid=19663' '!=' 'object_number=44, tag_tid=5, entry_tid=19576' 2024-04-06T09:36:03.277 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:36:03.277 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=47, tag_tid=5, entry_tid=19663' = 'object_number=44, tag_tid=5, entry_tid=19576' 2024-04-06T09:36:03.277 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=47, tag_tid=5, entry_tid=19663' 2024-04-06T09:36:03.277 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=45, tag_tid=6, entry_tid=1' 2024-04-06T09:36:03.277 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:36:03.278 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:36:03.278 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=6 2024-04-06T09:36:03.279 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=47, tag_tid=5, entry_tid=19663' 2024-04-06T09:36:03.279 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:36:03.279 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:36:03.280 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=5 2024-04-06T09:36:03.281 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=45, tag_tid=6, entry_tid=1' 2024-04-06T09:36:03.281 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:36:03.281 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:36:03.282 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:36:03.282 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=47, tag_tid=5, entry_tid=19663' 2024-04-06T09:36:03.282 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:36:03.283 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:36:03.283 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=19663 2024-04-06T09:36:03.284 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 6 = 5 -a 1 -le 19663 2024-04-06T09:36:03.284 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:36:03.284 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:36:03.284 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:36:03.486 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:36:03.486 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:36:03.486 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:36:03.486 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:36:03.486 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:36:03.486 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:36:03.486 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:36:03.486 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:36:03.486 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:36:03.486 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:36:03.486 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:36:03.486 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:36:03.487 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:36:03.487 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:36:03.487 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:36:03.487 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:36:03.487 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:36:03.489 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:36:03.489 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:36:03.490 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:36:03.490 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:36:03.490 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:36:03.490 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:36:03.490 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:36:03.490 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:36:03.490 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:36:03.490 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:36:03.490 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:36:03.490 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:36:03.490 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:36:03.490 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:36:03.491 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:36:03.491 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:36:03.491 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:36:03.491 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:36:03.491 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:36:03.491 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:36:03.491 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:36:03.491 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:36:03.491 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:36:03.491 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:36:03.491 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:36:03.491 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:36:03.491 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:36:03.491 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:36:03.491 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:36:03.492 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:36:03.492 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:36:03.492 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:36:03.515 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:36:03.515 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:36:03.515 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:36:03.516 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:36:03.642 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:36:03.643 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:36:03.643 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:36:03.643 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:36:03.644 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:36:03.644 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:36:03.644 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:36:03.644 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:36:03.644 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:36:03.644 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:36:03.644 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:36:03.644 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:36:03.645 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:36:03.646 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:36:03.646 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:36:03.646 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:36:03.696 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 11 2024-04-06T09:36:03.696 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 11 2024-04-06T09:36:03.696 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:36:03.696 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=45, tag_tid=6, entry_tid=1], [object_number=44, tag_tid=6, entry_tid=0], [object_number=47, tag_tid=5, entry_tid=22415], [object_number=46, tag_tid=5, entry_tid=22414]]], state=connected] 2024-04-06T09:36:03.697 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=44, tag_tid=5, entry_tid=19756], [object_number=47, tag_tid=5, entry_tid=19755], [object_number=46, tag_tid=5, entry_tid=19754], [object_number=45, tag_tid=5, entry_tid=19753]]], state=connected] 2024-04-06T09:36:03.700 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:36:03.701 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=45, tag_tid=6, entry_tid=1' 2024-04-06T09:36:03.702 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:36:03.703 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:36:03.703 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:36:03.703 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:36:03.703 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:36:03.703 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:36:03.703 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:36:03.703 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:36:03.703 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:36:03.703 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:36:03.703 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:36:03.703 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:36:03.704 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:36:03.704 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:36:03.704 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:36:03.749 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 11 2024-04-06T09:36:03.749 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 11 2024-04-06T09:36:03.749 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:36:03.749 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=45, tag_tid=6, entry_tid=1], [object_number=44, tag_tid=6, entry_tid=0], [object_number=47, tag_tid=5, entry_tid=22415], [object_number=46, tag_tid=5, entry_tid=22414]]], state=connected] 2024-04-06T09:36:03.749 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=44, tag_tid=5, entry_tid=19756], [object_number=47, tag_tid=5, entry_tid=19755], [object_number=46, tag_tid=5, entry_tid=19754], [object_number=45, tag_tid=5, entry_tid=19753]]], state=connected] 2024-04-06T09:36:03.753 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:36:03.754 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=44, tag_tid=5, entry_tid=19756' 2024-04-06T09:36:03.754 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=45, tag_tid=6, entry_tid=1' -a 'object_number=45, tag_tid=6, entry_tid=1' = 'object_number=44, tag_tid=5, entry_tid=19756' 2024-04-06T09:36:03.754 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=44, tag_tid=5, entry_tid=19756' '!=' 'object_number=47, tag_tid=5, entry_tid=19663' 2024-04-06T09:36:03.754 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:36:03.755 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=44, tag_tid=5, entry_tid=19756' = 'object_number=47, tag_tid=5, entry_tid=19663' 2024-04-06T09:36:03.755 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=44, tag_tid=5, entry_tid=19756' 2024-04-06T09:36:03.755 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=45, tag_tid=6, entry_tid=1' 2024-04-06T09:36:03.755 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:36:03.755 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:36:03.756 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=6 2024-04-06T09:36:03.756 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=44, tag_tid=5, entry_tid=19756' 2024-04-06T09:36:03.756 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:36:03.757 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:36:03.757 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=5 2024-04-06T09:36:03.758 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=45, tag_tid=6, entry_tid=1' 2024-04-06T09:36:03.758 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:36:03.758 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:36:03.759 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:36:03.760 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=44, tag_tid=5, entry_tid=19756' 2024-04-06T09:36:03.760 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:36:03.760 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:36:03.761 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=19756 2024-04-06T09:36:03.761 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 6 = 5 -a 1 -le 19756 2024-04-06T09:36:03.761 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:36:03.761 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:36:03.761 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:36:03.963 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:36:03.963 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:36:03.963 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:36:03.963 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:36:03.963 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:36:03.963 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:36:03.963 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:36:03.964 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:36:03.964 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:36:03.964 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:36:03.964 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:36:03.964 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:36:03.964 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:36:03.964 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:36:03.964 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:36:03.964 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:36:03.965 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:36:03.967 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:36:03.967 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:36:03.968 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:36:03.968 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:36:03.968 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:36:03.968 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:36:03.968 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:36:03.968 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:36:03.968 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:36:03.968 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:36:03.968 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:36:03.968 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:36:03.968 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:36:03.968 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:36:03.968 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:36:03.968 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:36:03.969 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:36:03.969 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:36:03.969 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:36:03.969 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:36:03.969 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:36:03.969 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:36:03.969 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:36:03.969 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:36:03.969 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:36:03.969 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:36:03.969 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:36:03.969 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:36:03.969 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:36:03.969 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:36:03.970 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:36:03.970 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:36:03.994 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:36:03.995 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:36:03.995 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:36:03.995 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:36:04.124 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:36:04.126 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:36:04.126 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:36:04.126 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:36:04.126 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:36:04.126 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:36:04.126 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:36:04.126 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:36:04.126 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:36:04.126 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:36:04.127 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:36:04.127 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:36:04.127 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:36:04.128 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:36:04.129 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:36:04.129 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:36:04.183 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 11 2024-04-06T09:36:04.183 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 11 2024-04-06T09:36:04.183 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:36:04.183 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=45, tag_tid=6, entry_tid=1], [object_number=44, tag_tid=6, entry_tid=0], [object_number=47, tag_tid=5, entry_tid=22415], [object_number=46, tag_tid=5, entry_tid=22414]]], state=connected] 2024-04-06T09:36:04.183 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=45, tag_tid=5, entry_tid=19853], [object_number=44, tag_tid=5, entry_tid=19852], [object_number=47, tag_tid=5, entry_tid=19851], [object_number=46, tag_tid=5, entry_tid=19850]]], state=connected] 2024-04-06T09:36:04.186 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:36:04.188 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=45, tag_tid=6, entry_tid=1' 2024-04-06T09:36:04.189 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:36:04.189 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:36:04.189 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:36:04.189 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:36:04.190 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:36:04.190 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:36:04.190 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:36:04.190 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:36:04.190 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:36:04.190 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:36:04.190 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:36:04.190 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:36:04.191 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:36:04.191 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:36:04.191 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:36:04.237 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 11 2024-04-06T09:36:04.237 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 11 2024-04-06T09:36:04.237 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:36:04.237 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=45, tag_tid=6, entry_tid=1], [object_number=44, tag_tid=6, entry_tid=0], [object_number=47, tag_tid=5, entry_tid=22415], [object_number=46, tag_tid=5, entry_tid=22414]]], state=connected] 2024-04-06T09:36:04.237 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=45, tag_tid=5, entry_tid=19853], [object_number=44, tag_tid=5, entry_tid=19852], [object_number=47, tag_tid=5, entry_tid=19851], [object_number=46, tag_tid=5, entry_tid=19850]]], state=connected] 2024-04-06T09:36:04.240 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:36:04.242 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=45, tag_tid=5, entry_tid=19853' 2024-04-06T09:36:04.243 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=45, tag_tid=6, entry_tid=1' -a 'object_number=45, tag_tid=6, entry_tid=1' = 'object_number=45, tag_tid=5, entry_tid=19853' 2024-04-06T09:36:04.243 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=45, tag_tid=5, entry_tid=19853' '!=' 'object_number=44, tag_tid=5, entry_tid=19756' 2024-04-06T09:36:04.243 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:36:04.243 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=45, tag_tid=5, entry_tid=19853' = 'object_number=44, tag_tid=5, entry_tid=19756' 2024-04-06T09:36:04.243 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=45, tag_tid=5, entry_tid=19853' 2024-04-06T09:36:04.243 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=45, tag_tid=6, entry_tid=1' 2024-04-06T09:36:04.243 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:36:04.243 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:36:04.243 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=6 2024-04-06T09:36:04.244 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=45, tag_tid=5, entry_tid=19853' 2024-04-06T09:36:04.244 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:36:04.244 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:36:04.245 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=5 2024-04-06T09:36:04.246 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=45, tag_tid=6, entry_tid=1' 2024-04-06T09:36:04.246 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:36:04.246 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:36:04.247 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:36:04.247 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=45, tag_tid=5, entry_tid=19853' 2024-04-06T09:36:04.248 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:36:04.248 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:36:04.249 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=19853 2024-04-06T09:36:04.249 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 6 = 5 -a 1 -le 19853 2024-04-06T09:36:04.249 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:36:04.249 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:36:04.249 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:36:04.451 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:36:04.451 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:36:04.451 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:36:04.451 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:36:04.451 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:36:04.451 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:36:04.451 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:36:04.451 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:36:04.451 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:36:04.452 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:36:04.452 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:36:04.452 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:36:04.452 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:36:04.452 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:36:04.452 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:36:04.452 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:36:04.452 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:36:04.456 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:36:04.456 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:36:04.456 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:36:04.456 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:36:04.456 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:36:04.456 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:36:04.456 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:36:04.457 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:36:04.457 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:36:04.457 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:36:04.457 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:36:04.457 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:36:04.457 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:36:04.457 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:36:04.457 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:36:04.457 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:36:04.457 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:36:04.457 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:36:04.457 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:36:04.457 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:36:04.457 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:36:04.457 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:36:04.457 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:36:04.458 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:36:04.458 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:36:04.458 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:36:04.458 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:36:04.458 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:36:04.458 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:36:04.458 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:36:04.458 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:36:04.458 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:36:04.482 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:36:04.483 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:36:04.483 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:36:04.483 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:36:04.615 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:36:04.616 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:36:04.616 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:36:04.616 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:36:04.616 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:36:04.616 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:36:04.616 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:36:04.616 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:36:04.616 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:36:04.617 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:36:04.617 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:36:04.617 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:36:04.617 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:36:04.619 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:36:04.619 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:36:04.620 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:36:04.665 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 11 2024-04-06T09:36:04.666 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 11 2024-04-06T09:36:04.666 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:36:04.666 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=45, tag_tid=6, entry_tid=1], [object_number=44, tag_tid=6, entry_tid=0], [object_number=47, tag_tid=5, entry_tid=22415], [object_number=46, tag_tid=5, entry_tid=22414]]], state=connected] 2024-04-06T09:36:04.666 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=46, tag_tid=5, entry_tid=19942], [object_number=45, tag_tid=5, entry_tid=19941], [object_number=44, tag_tid=5, entry_tid=19940], [object_number=47, tag_tid=5, entry_tid=19939]]], state=connected] 2024-04-06T09:36:04.669 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:36:04.670 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=45, tag_tid=6, entry_tid=1' 2024-04-06T09:36:04.671 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:36:04.672 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:36:04.672 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:36:04.672 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:36:04.672 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:36:04.672 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:36:04.672 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:36:04.672 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:36:04.672 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:36:04.672 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:36:04.672 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:36:04.672 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:36:04.673 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:36:04.673 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:36:04.673 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:36:04.717 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 11 2024-04-06T09:36:04.717 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 11 2024-04-06T09:36:04.718 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:36:04.718 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=45, tag_tid=6, entry_tid=1], [object_number=44, tag_tid=6, entry_tid=0], [object_number=47, tag_tid=5, entry_tid=22415], [object_number=46, tag_tid=5, entry_tid=22414]]], state=connected] 2024-04-06T09:36:04.718 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=46, tag_tid=5, entry_tid=19942], [object_number=45, tag_tid=5, entry_tid=19941], [object_number=44, tag_tid=5, entry_tid=19940], [object_number=47, tag_tid=5, entry_tid=19939]]], state=connected] 2024-04-06T09:36:04.721 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:36:04.723 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=46, tag_tid=5, entry_tid=19942' 2024-04-06T09:36:04.723 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=45, tag_tid=6, entry_tid=1' -a 'object_number=45, tag_tid=6, entry_tid=1' = 'object_number=46, tag_tid=5, entry_tid=19942' 2024-04-06T09:36:04.723 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=46, tag_tid=5, entry_tid=19942' '!=' 'object_number=45, tag_tid=5, entry_tid=19853' 2024-04-06T09:36:04.723 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:36:04.723 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=46, tag_tid=5, entry_tid=19942' = 'object_number=45, tag_tid=5, entry_tid=19853' 2024-04-06T09:36:04.723 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=46, tag_tid=5, entry_tid=19942' 2024-04-06T09:36:04.724 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=45, tag_tid=6, entry_tid=1' 2024-04-06T09:36:04.724 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:36:04.724 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:36:04.724 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=6 2024-04-06T09:36:04.725 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=46, tag_tid=5, entry_tid=19942' 2024-04-06T09:36:04.725 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:36:04.725 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:36:04.726 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=5 2024-04-06T09:36:04.727 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=45, tag_tid=6, entry_tid=1' 2024-04-06T09:36:04.727 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:36:04.727 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:36:04.728 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:36:04.729 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=46, tag_tid=5, entry_tid=19942' 2024-04-06T09:36:04.729 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:36:04.729 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:36:04.730 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=19942 2024-04-06T09:36:04.730 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 6 = 5 -a 1 -le 19942 2024-04-06T09:36:04.730 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:36:04.730 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:36:04.730 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:36:04.932 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:36:04.932 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:36:04.932 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:36:04.932 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:36:04.932 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:36:04.932 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:36:04.932 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:36:04.932 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:36:04.932 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:36:04.932 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:36:04.932 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:36:04.932 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:36:04.933 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:36:04.933 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:36:04.933 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:36:04.933 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:36:04.934 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:36:04.936 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:36:04.936 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:36:04.936 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:36:04.936 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:36:04.936 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:36:04.936 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:36:04.936 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:36:04.937 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:36:04.937 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:36:04.937 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:36:04.937 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:36:04.937 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:36:04.937 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:36:04.937 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:36:04.937 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:36:04.937 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:36:04.937 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:36:04.937 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:36:04.937 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:36:04.937 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:36:04.937 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:36:04.937 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:36:04.937 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:36:04.938 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:36:04.938 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:36:04.938 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:36:04.938 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:36:04.938 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:36:04.938 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:36:04.938 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:36:04.938 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:36:04.938 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:36:04.962 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:36:04.963 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:36:04.963 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:36:04.963 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:36:05.090 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:36:05.092 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:36:05.092 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:36:05.092 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:36:05.092 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:36:05.092 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:36:05.092 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:36:05.092 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:36:05.092 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:36:05.092 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:36:05.092 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:36:05.093 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:36:05.093 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:36:05.094 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:36:05.094 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:36:05.095 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:36:05.142 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 11 2024-04-06T09:36:05.142 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 11 2024-04-06T09:36:05.142 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:36:05.142 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=45, tag_tid=6, entry_tid=1], [object_number=44, tag_tid=6, entry_tid=0], [object_number=47, tag_tid=5, entry_tid=22415], [object_number=46, tag_tid=5, entry_tid=22414]]], state=connected] 2024-04-06T09:36:05.142 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=44, tag_tid=5, entry_tid=20032], [object_number=47, tag_tid=5, entry_tid=20031], [object_number=46, tag_tid=5, entry_tid=20030], [object_number=45, tag_tid=5, entry_tid=20029]]], state=connected] 2024-04-06T09:36:05.146 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:36:05.149 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=45, tag_tid=6, entry_tid=1' 2024-04-06T09:36:05.151 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:36:05.151 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:36:05.151 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:36:05.151 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:36:05.151 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:36:05.151 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:36:05.151 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:36:05.151 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:36:05.151 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:36:05.151 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:36:05.151 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:36:05.152 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:36:05.152 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:36:05.152 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:36:05.152 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:36:05.195 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 11 2024-04-06T09:36:05.195 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 11 2024-04-06T09:36:05.195 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:36:05.195 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=45, tag_tid=6, entry_tid=1], [object_number=44, tag_tid=6, entry_tid=0], [object_number=47, tag_tid=5, entry_tid=22415], [object_number=46, tag_tid=5, entry_tid=22414]]], state=connected] 2024-04-06T09:36:05.195 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=44, tag_tid=5, entry_tid=20032], [object_number=47, tag_tid=5, entry_tid=20031], [object_number=46, tag_tid=5, entry_tid=20030], [object_number=45, tag_tid=5, entry_tid=20029]]], state=connected] 2024-04-06T09:36:05.198 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:36:05.200 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=44, tag_tid=5, entry_tid=20032' 2024-04-06T09:36:05.200 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=45, tag_tid=6, entry_tid=1' -a 'object_number=45, tag_tid=6, entry_tid=1' = 'object_number=44, tag_tid=5, entry_tid=20032' 2024-04-06T09:36:05.200 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=44, tag_tid=5, entry_tid=20032' '!=' 'object_number=46, tag_tid=5, entry_tid=19942' 2024-04-06T09:36:05.200 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:36:05.201 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=44, tag_tid=5, entry_tid=20032' = 'object_number=46, tag_tid=5, entry_tid=19942' 2024-04-06T09:36:05.201 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=44, tag_tid=5, entry_tid=20032' 2024-04-06T09:36:05.201 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=45, tag_tid=6, entry_tid=1' 2024-04-06T09:36:05.201 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:36:05.201 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:36:05.201 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=6 2024-04-06T09:36:05.202 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=44, tag_tid=5, entry_tid=20032' 2024-04-06T09:36:05.202 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:36:05.202 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:36:05.203 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=5 2024-04-06T09:36:05.204 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=45, tag_tid=6, entry_tid=1' 2024-04-06T09:36:05.204 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:36:05.204 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:36:05.205 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:36:05.205 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=44, tag_tid=5, entry_tid=20032' 2024-04-06T09:36:05.205 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:36:05.205 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:36:05.207 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=20032 2024-04-06T09:36:05.207 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 6 = 5 -a 1 -le 20032 2024-04-06T09:36:05.207 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:36:05.207 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:36:05.207 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:36:05.408 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:36:05.408 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:36:05.409 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:36:05.409 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:36:05.409 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:36:05.409 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:36:05.409 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:36:05.409 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:36:05.409 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:36:05.409 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:36:05.409 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:36:05.409 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:36:05.409 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:36:05.409 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:36:05.410 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:36:05.410 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:36:05.410 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:36:05.413 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:36:05.413 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:36:05.413 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:36:05.413 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:36:05.413 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:36:05.414 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:36:05.414 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:36:05.414 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:36:05.414 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:36:05.414 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:36:05.414 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:36:05.414 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:36:05.414 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:36:05.414 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:36:05.415 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:36:05.415 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:36:05.415 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:36:05.415 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:36:05.415 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:36:05.415 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:36:05.415 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:36:05.415 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:36:05.415 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:36:05.415 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:36:05.415 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:36:05.415 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:36:05.416 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:36:05.416 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:36:05.416 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:36:05.416 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:36:05.416 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:36:05.416 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:36:05.440 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:36:05.440 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:36:05.440 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:36:05.440 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:36:05.575 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:36:05.575 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:36:05.575 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:36:05.575 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:36:05.575 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:36:05.575 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:36:05.575 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:36:05.575 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:36:05.575 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:36:05.575 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:36:05.576 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:36:05.576 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:36:05.576 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:36:05.579 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:36:05.579 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:36:05.579 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:36:05.625 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 11 2024-04-06T09:36:05.625 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 11 2024-04-06T09:36:05.625 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:36:05.625 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=45, tag_tid=6, entry_tid=1], [object_number=44, tag_tid=6, entry_tid=0], [object_number=47, tag_tid=5, entry_tid=22415], [object_number=46, tag_tid=5, entry_tid=22414]]], state=connected] 2024-04-06T09:36:05.625 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=47, tag_tid=5, entry_tid=20127], [object_number=46, tag_tid=5, entry_tid=20126], [object_number=45, tag_tid=5, entry_tid=20125], [object_number=44, tag_tid=5, entry_tid=20124]]], state=connected] 2024-04-06T09:36:05.628 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:36:05.630 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=45, tag_tid=6, entry_tid=1' 2024-04-06T09:36:05.631 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:36:05.631 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:36:05.631 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:36:05.631 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:36:05.631 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:36:05.631 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:36:05.631 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:36:05.631 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:36:05.631 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:36:05.632 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:36:05.632 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:36:05.632 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:36:05.632 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:36:05.633 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:36:05.633 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:36:05.677 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 11 2024-04-06T09:36:05.677 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 11 2024-04-06T09:36:05.677 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:36:05.677 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=45, tag_tid=6, entry_tid=1], [object_number=44, tag_tid=6, entry_tid=0], [object_number=47, tag_tid=5, entry_tid=22415], [object_number=46, tag_tid=5, entry_tid=22414]]], state=connected] 2024-04-06T09:36:05.677 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=47, tag_tid=5, entry_tid=20127], [object_number=46, tag_tid=5, entry_tid=20126], [object_number=45, tag_tid=5, entry_tid=20125], [object_number=44, tag_tid=5, entry_tid=20124]]], state=connected] 2024-04-06T09:36:05.681 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:36:05.683 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=47, tag_tid=5, entry_tid=20127' 2024-04-06T09:36:05.683 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=45, tag_tid=6, entry_tid=1' -a 'object_number=45, tag_tid=6, entry_tid=1' = 'object_number=47, tag_tid=5, entry_tid=20127' 2024-04-06T09:36:05.683 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=47, tag_tid=5, entry_tid=20127' '!=' 'object_number=44, tag_tid=5, entry_tid=20032' 2024-04-06T09:36:05.683 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:36:05.684 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=47, tag_tid=5, entry_tid=20127' = 'object_number=44, tag_tid=5, entry_tid=20032' 2024-04-06T09:36:05.684 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=47, tag_tid=5, entry_tid=20127' 2024-04-06T09:36:05.684 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=45, tag_tid=6, entry_tid=1' 2024-04-06T09:36:05.684 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:36:05.684 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:36:05.684 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=6 2024-04-06T09:36:05.685 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=47, tag_tid=5, entry_tid=20127' 2024-04-06T09:36:05.685 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:36:05.685 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:36:05.687 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=5 2024-04-06T09:36:05.687 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=45, tag_tid=6, entry_tid=1' 2024-04-06T09:36:05.687 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:36:05.687 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:36:05.688 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:36:05.689 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=47, tag_tid=5, entry_tid=20127' 2024-04-06T09:36:05.689 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:36:05.689 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:36:05.690 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=20127 2024-04-06T09:36:05.690 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 6 = 5 -a 1 -le 20127 2024-04-06T09:36:05.690 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:36:05.690 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:36:05.690 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:36:05.891 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:36:05.892 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:36:05.892 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:36:05.892 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:36:05.892 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:36:05.892 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:36:05.892 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:36:05.892 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:36:05.892 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:36:05.892 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:36:05.892 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:36:05.892 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:36:05.892 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:36:05.892 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:36:05.892 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:36:05.893 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:36:05.893 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:36:05.897 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:36:05.897 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:36:05.897 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:36:05.897 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:36:05.897 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:36:05.897 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:36:05.897 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:36:05.897 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:36:05.897 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:36:05.897 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:36:05.897 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:36:05.897 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:36:05.897 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:36:05.897 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:36:05.897 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:36:05.898 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:36:05.898 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:36:05.898 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:36:05.898 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:36:05.898 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:36:05.898 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:36:05.898 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:36:05.898 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:36:05.898 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:36:05.898 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:36:05.898 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:36:05.899 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:36:05.899 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:36:05.899 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:36:05.899 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:36:05.899 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:36:05.899 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:36:05.923 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:36:05.924 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:36:05.924 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:36:05.924 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:36:06.056 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:36:06.057 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:36:06.057 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:36:06.057 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:36:06.057 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:36:06.057 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:36:06.057 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:36:06.057 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:36:06.057 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:36:06.057 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:36:06.058 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:36:06.058 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:36:06.058 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:36:06.061 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:36:06.061 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:36:06.061 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:36:06.105 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 11 2024-04-06T09:36:06.105 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 11 2024-04-06T09:36:06.105 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:36:06.105 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=45, tag_tid=6, entry_tid=1], [object_number=44, tag_tid=6, entry_tid=0], [object_number=47, tag_tid=5, entry_tid=22415], [object_number=46, tag_tid=5, entry_tid=22414]]], state=connected] 2024-04-06T09:36:06.105 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=46, tag_tid=5, entry_tid=20210], [object_number=45, tag_tid=5, entry_tid=20209], [object_number=44, tag_tid=5, entry_tid=20208], [object_number=47, tag_tid=5, entry_tid=20207]]], state=connected] 2024-04-06T09:36:06.109 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:36:06.110 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=45, tag_tid=6, entry_tid=1' 2024-04-06T09:36:06.111 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:36:06.111 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:36:06.111 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:36:06.111 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:36:06.111 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:36:06.111 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:36:06.111 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:36:06.111 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:36:06.111 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:36:06.112 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:36:06.112 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:36:06.112 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:36:06.112 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:36:06.112 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:36:06.112 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:36:06.158 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 11 2024-04-06T09:36:06.158 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 11 2024-04-06T09:36:06.158 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:36:06.158 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=45, tag_tid=6, entry_tid=1], [object_number=44, tag_tid=6, entry_tid=0], [object_number=47, tag_tid=5, entry_tid=22415], [object_number=46, tag_tid=5, entry_tid=22414]]], state=connected] 2024-04-06T09:36:06.158 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=46, tag_tid=5, entry_tid=20210], [object_number=45, tag_tid=5, entry_tid=20209], [object_number=44, tag_tid=5, entry_tid=20208], [object_number=47, tag_tid=5, entry_tid=20207]]], state=connected] 2024-04-06T09:36:06.161 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:36:06.163 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=46, tag_tid=5, entry_tid=20210' 2024-04-06T09:36:06.163 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=45, tag_tid=6, entry_tid=1' -a 'object_number=45, tag_tid=6, entry_tid=1' = 'object_number=46, tag_tid=5, entry_tid=20210' 2024-04-06T09:36:06.163 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=46, tag_tid=5, entry_tid=20210' '!=' 'object_number=47, tag_tid=5, entry_tid=20127' 2024-04-06T09:36:06.163 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:36:06.163 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=46, tag_tid=5, entry_tid=20210' = 'object_number=47, tag_tid=5, entry_tid=20127' 2024-04-06T09:36:06.163 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=46, tag_tid=5, entry_tid=20210' 2024-04-06T09:36:06.164 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=45, tag_tid=6, entry_tid=1' 2024-04-06T09:36:06.164 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:36:06.164 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:36:06.165 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=6 2024-04-06T09:36:06.165 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=46, tag_tid=5, entry_tid=20210' 2024-04-06T09:36:06.165 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:36:06.165 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:36:06.166 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=5 2024-04-06T09:36:06.167 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=45, tag_tid=6, entry_tid=1' 2024-04-06T09:36:06.167 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:36:06.167 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:36:06.168 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:36:06.169 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=46, tag_tid=5, entry_tid=20210' 2024-04-06T09:36:06.169 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:36:06.169 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:36:06.170 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=20210 2024-04-06T09:36:06.170 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 6 = 5 -a 1 -le 20210 2024-04-06T09:36:06.170 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:36:06.171 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:36:06.171 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:36:06.372 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:36:06.372 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:36:06.372 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:36:06.372 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:36:06.372 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:36:06.373 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:36:06.373 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:36:06.373 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:36:06.373 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:36:06.373 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:36:06.373 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:36:06.373 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:36:06.373 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:36:06.373 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:36:06.373 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:36:06.374 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:36:06.374 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:36:06.376 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:36:06.377 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:36:06.377 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:36:06.377 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:36:06.377 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:36:06.377 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:36:06.377 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:36:06.377 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:36:06.377 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:36:06.377 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:36:06.377 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:36:06.377 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:36:06.377 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:36:06.377 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:36:06.377 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:36:06.377 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:36:06.378 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:36:06.378 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:36:06.378 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:36:06.378 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:36:06.378 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:36:06.378 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:36:06.378 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:36:06.378 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:36:06.378 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:36:06.378 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:36:06.378 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:36:06.378 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:36:06.378 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:36:06.378 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:36:06.379 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:36:06.379 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:36:06.402 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:36:06.403 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:36:06.403 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:36:06.403 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:36:06.539 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:36:06.540 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:36:06.540 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:36:06.541 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:36:06.541 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:36:06.541 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:36:06.541 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:36:06.541 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:36:06.541 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:36:06.541 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:36:06.541 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:36:06.541 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:36:06.541 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:36:06.543 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:36:06.543 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:36:06.544 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:36:06.588 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 11 2024-04-06T09:36:06.588 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 11 2024-04-06T09:36:06.588 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:36:06.588 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=45, tag_tid=6, entry_tid=1], [object_number=44, tag_tid=6, entry_tid=0], [object_number=47, tag_tid=5, entry_tid=22415], [object_number=46, tag_tid=5, entry_tid=22414]]], state=connected] 2024-04-06T09:36:06.588 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=46, tag_tid=5, entry_tid=20298], [object_number=45, tag_tid=5, entry_tid=20297], [object_number=44, tag_tid=5, entry_tid=20296], [object_number=47, tag_tid=5, entry_tid=20295]]], state=connected] 2024-04-06T09:36:06.592 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:36:06.594 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=45, tag_tid=6, entry_tid=1' 2024-04-06T09:36:06.594 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:36:06.594 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:36:06.594 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:36:06.594 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:36:06.594 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:36:06.594 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:36:06.594 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:36:06.594 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:36:06.594 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:36:06.594 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:36:06.594 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:36:06.594 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:36:06.595 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:36:06.595 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:36:06.595 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:36:06.640 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 11 2024-04-06T09:36:06.640 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 11 2024-04-06T09:36:06.640 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:36:06.640 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=45, tag_tid=6, entry_tid=1], [object_number=44, tag_tid=6, entry_tid=0], [object_number=47, tag_tid=5, entry_tid=22415], [object_number=46, tag_tid=5, entry_tid=22414]]], state=connected] 2024-04-06T09:36:06.640 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=46, tag_tid=5, entry_tid=20298], [object_number=45, tag_tid=5, entry_tid=20297], [object_number=44, tag_tid=5, entry_tid=20296], [object_number=47, tag_tid=5, entry_tid=20295]]], state=connected] 2024-04-06T09:36:06.643 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:36:06.645 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=46, tag_tid=5, entry_tid=20298' 2024-04-06T09:36:06.645 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=45, tag_tid=6, entry_tid=1' -a 'object_number=45, tag_tid=6, entry_tid=1' = 'object_number=46, tag_tid=5, entry_tid=20298' 2024-04-06T09:36:06.646 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=46, tag_tid=5, entry_tid=20298' '!=' 'object_number=46, tag_tid=5, entry_tid=20210' 2024-04-06T09:36:06.646 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:36:06.646 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=46, tag_tid=5, entry_tid=20298' = 'object_number=46, tag_tid=5, entry_tid=20210' 2024-04-06T09:36:06.646 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=46, tag_tid=5, entry_tid=20298' 2024-04-06T09:36:06.646 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=45, tag_tid=6, entry_tid=1' 2024-04-06T09:36:06.646 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:36:06.646 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:36:06.647 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=6 2024-04-06T09:36:06.648 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=46, tag_tid=5, entry_tid=20298' 2024-04-06T09:36:06.648 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:36:06.648 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:36:06.649 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=5 2024-04-06T09:36:06.649 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=45, tag_tid=6, entry_tid=1' 2024-04-06T09:36:06.650 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:36:06.650 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:36:06.651 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:36:06.651 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=46, tag_tid=5, entry_tid=20298' 2024-04-06T09:36:06.651 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:36:06.651 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:36:06.652 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=20298 2024-04-06T09:36:06.653 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 6 = 5 -a 1 -le 20298 2024-04-06T09:36:06.653 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:36:06.653 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:36:06.653 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:36:06.855 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:36:06.855 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:36:06.855 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:36:06.855 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:36:06.855 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:36:06.855 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:36:06.855 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:36:06.855 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:36:06.855 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:36:06.855 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:36:06.855 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:36:06.855 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:36:06.856 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:36:06.856 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:36:06.856 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:36:06.856 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:36:06.856 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:36:06.859 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:36:06.859 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:36:06.859 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:36:06.859 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:36:06.859 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:36:06.859 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:36:06.859 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:36:06.859 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:36:06.860 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:36:06.860 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:36:06.860 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:36:06.860 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:36:06.860 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:36:06.860 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:36:06.860 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:36:06.860 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:36:06.861 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:36:06.861 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:36:06.861 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:36:06.861 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:36:06.861 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:36:06.861 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:36:06.861 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:36:06.861 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:36:06.861 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:36:06.861 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:36:06.861 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:36:06.861 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:36:06.861 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:36:06.862 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:36:06.862 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:36:06.862 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:36:06.886 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:36:06.887 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:36:06.887 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:36:06.887 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:36:07.017 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:36:07.019 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:36:07.019 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:36:07.019 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:36:07.019 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:36:07.019 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:36:07.019 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:36:07.019 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:36:07.019 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:36:07.019 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:36:07.019 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:36:07.020 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:36:07.020 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:36:07.022 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:36:07.022 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:36:07.022 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:36:07.072 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 11 2024-04-06T09:36:07.072 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 11 2024-04-06T09:36:07.072 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:36:07.072 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=45, tag_tid=6, entry_tid=1], [object_number=44, tag_tid=6, entry_tid=0], [object_number=47, tag_tid=5, entry_tid=22415], [object_number=46, tag_tid=5, entry_tid=22414]]], state=connected] 2024-04-06T09:36:07.072 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=46, tag_tid=5, entry_tid=20402], [object_number=45, tag_tid=5, entry_tid=20401], [object_number=44, tag_tid=5, entry_tid=20400], [object_number=47, tag_tid=5, entry_tid=20399]]], state=connected] 2024-04-06T09:36:07.076 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:36:07.077 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=45, tag_tid=6, entry_tid=1' 2024-04-06T09:36:07.078 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:36:07.078 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:36:07.078 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:36:07.079 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:36:07.079 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:36:07.079 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:36:07.079 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:36:07.079 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:36:07.079 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:36:07.079 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:36:07.079 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:36:07.079 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:36:07.080 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:36:07.080 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:36:07.080 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:36:07.125 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 11 2024-04-06T09:36:07.125 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 11 2024-04-06T09:36:07.125 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:36:07.125 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=45, tag_tid=6, entry_tid=1], [object_number=44, tag_tid=6, entry_tid=0], [object_number=47, tag_tid=5, entry_tid=22415], [object_number=46, tag_tid=5, entry_tid=22414]]], state=connected] 2024-04-06T09:36:07.125 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=46, tag_tid=5, entry_tid=20402], [object_number=45, tag_tid=5, entry_tid=20401], [object_number=44, tag_tid=5, entry_tid=20400], [object_number=47, tag_tid=5, entry_tid=20399]]], state=connected] 2024-04-06T09:36:07.129 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:36:07.130 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=46, tag_tid=5, entry_tid=20402' 2024-04-06T09:36:07.130 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=45, tag_tid=6, entry_tid=1' -a 'object_number=45, tag_tid=6, entry_tid=1' = 'object_number=46, tag_tid=5, entry_tid=20402' 2024-04-06T09:36:07.130 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=46, tag_tid=5, entry_tid=20402' '!=' 'object_number=46, tag_tid=5, entry_tid=20298' 2024-04-06T09:36:07.131 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:36:07.131 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=46, tag_tid=5, entry_tid=20402' = 'object_number=46, tag_tid=5, entry_tid=20298' 2024-04-06T09:36:07.131 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=46, tag_tid=5, entry_tid=20402' 2024-04-06T09:36:07.131 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=45, tag_tid=6, entry_tid=1' 2024-04-06T09:36:07.131 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:36:07.131 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:36:07.132 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=6 2024-04-06T09:36:07.132 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=46, tag_tid=5, entry_tid=20402' 2024-04-06T09:36:07.133 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:36:07.133 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:36:07.134 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=5 2024-04-06T09:36:07.134 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=45, tag_tid=6, entry_tid=1' 2024-04-06T09:36:07.134 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:36:07.134 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:36:07.135 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:36:07.136 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=46, tag_tid=5, entry_tid=20402' 2024-04-06T09:36:07.136 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:36:07.136 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:36:07.137 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=20402 2024-04-06T09:36:07.137 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 6 = 5 -a 1 -le 20402 2024-04-06T09:36:07.137 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:36:07.138 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:36:07.138 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:36:07.339 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:36:07.339 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:36:07.339 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:36:07.339 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:36:07.340 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:36:07.340 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:36:07.340 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:36:07.340 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:36:07.340 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:36:07.340 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:36:07.340 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:36:07.340 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:36:07.340 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:36:07.340 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:36:07.340 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:36:07.341 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:36:07.341 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:36:07.344 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:36:07.344 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:36:07.344 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:36:07.344 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:36:07.344 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:36:07.344 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:36:07.344 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:36:07.344 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:36:07.344 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:36:07.344 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:36:07.344 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:36:07.344 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:36:07.344 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:36:07.345 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:36:07.345 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:36:07.345 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:36:07.345 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:36:07.345 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:36:07.345 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:36:07.345 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:36:07.345 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:36:07.345 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:36:07.345 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:36:07.345 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:36:07.345 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:36:07.345 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:36:07.345 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:36:07.345 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:36:07.346 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:36:07.346 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:36:07.346 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:36:07.346 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:36:07.369 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:36:07.369 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:36:07.369 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:36:07.369 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:36:07.502 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:36:07.503 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:36:07.503 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:36:07.503 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:36:07.503 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:36:07.503 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:36:07.503 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:36:07.503 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:36:07.504 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:36:07.504 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:36:07.504 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:36:07.504 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:36:07.505 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:36:07.506 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:36:07.507 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:36:07.507 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:36:07.557 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 11 2024-04-06T09:36:07.557 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 11 2024-04-06T09:36:07.557 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:36:07.557 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=45, tag_tid=6, entry_tid=1], [object_number=44, tag_tid=6, entry_tid=0], [object_number=47, tag_tid=5, entry_tid=22415], [object_number=46, tag_tid=5, entry_tid=22414]]], state=connected] 2024-04-06T09:36:07.557 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=44, tag_tid=5, entry_tid=20492], [object_number=47, tag_tid=5, entry_tid=20491], [object_number=46, tag_tid=5, entry_tid=20490], [object_number=45, tag_tid=5, entry_tid=20489]]], state=connected] 2024-04-06T09:36:07.561 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:36:07.563 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=45, tag_tid=6, entry_tid=1' 2024-04-06T09:36:07.563 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:36:07.563 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:36:07.563 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:36:07.563 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:36:07.563 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:36:07.563 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:36:07.563 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:36:07.563 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:36:07.564 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:36:07.564 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:36:07.564 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:36:07.564 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:36:07.564 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:36:07.565 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:36:07.565 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:36:07.611 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 11 2024-04-06T09:36:07.611 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 11 2024-04-06T09:36:07.611 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:36:07.611 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=45, tag_tid=6, entry_tid=1], [object_number=44, tag_tid=6, entry_tid=0], [object_number=47, tag_tid=5, entry_tid=22415], [object_number=46, tag_tid=5, entry_tid=22414]]], state=connected] 2024-04-06T09:36:07.612 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=44, tag_tid=5, entry_tid=20492], [object_number=47, tag_tid=5, entry_tid=20491], [object_number=46, tag_tid=5, entry_tid=20490], [object_number=45, tag_tid=5, entry_tid=20489]]], state=connected] 2024-04-06T09:36:07.615 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:36:07.617 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=44, tag_tid=5, entry_tid=20492' 2024-04-06T09:36:07.617 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=45, tag_tid=6, entry_tid=1' -a 'object_number=45, tag_tid=6, entry_tid=1' = 'object_number=44, tag_tid=5, entry_tid=20492' 2024-04-06T09:36:07.617 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=44, tag_tid=5, entry_tid=20492' '!=' 'object_number=46, tag_tid=5, entry_tid=20402' 2024-04-06T09:36:07.617 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:36:07.617 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=44, tag_tid=5, entry_tid=20492' = 'object_number=46, tag_tid=5, entry_tid=20402' 2024-04-06T09:36:07.617 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=44, tag_tid=5, entry_tid=20492' 2024-04-06T09:36:07.617 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=45, tag_tid=6, entry_tid=1' 2024-04-06T09:36:07.617 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:36:07.618 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:36:07.618 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=6 2024-04-06T09:36:07.619 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=44, tag_tid=5, entry_tid=20492' 2024-04-06T09:36:07.619 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:36:07.619 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:36:07.620 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=5 2024-04-06T09:36:07.621 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=45, tag_tid=6, entry_tid=1' 2024-04-06T09:36:07.621 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:36:07.621 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:36:07.622 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:36:07.622 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=44, tag_tid=5, entry_tid=20492' 2024-04-06T09:36:07.622 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:36:07.622 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:36:07.624 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=20492 2024-04-06T09:36:07.624 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 6 = 5 -a 1 -le 20492 2024-04-06T09:36:07.624 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:36:07.624 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:36:07.624 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:36:07.826 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:36:07.826 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:36:07.826 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:36:07.826 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:36:07.826 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:36:07.826 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:36:07.826 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:36:07.826 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:36:07.826 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:36:07.826 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:36:07.826 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:36:07.827 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:36:07.827 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:36:07.827 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:36:07.827 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:36:07.827 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:36:07.827 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:36:07.830 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:36:07.830 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:36:07.830 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:36:07.830 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:36:07.831 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:36:07.831 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:36:07.831 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:36:07.831 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:36:07.831 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:36:07.831 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:36:07.831 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:36:07.831 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:36:07.831 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:36:07.831 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:36:07.831 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:36:07.831 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:36:07.831 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:36:07.831 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:36:07.832 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:36:07.832 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:36:07.832 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:36:07.832 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:36:07.832 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:36:07.832 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:36:07.832 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:36:07.832 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:36:07.832 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:36:07.832 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:36:07.832 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:36:07.832 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:36:07.832 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:36:07.833 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:36:07.856 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:36:07.857 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:36:07.857 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:36:07.857 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:36:07.984 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:36:07.985 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:36:07.985 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:36:07.985 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:36:07.985 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:36:07.985 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:36:07.985 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:36:07.985 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:36:07.985 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:36:07.986 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:36:07.986 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:36:07.986 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:36:07.986 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:36:07.989 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:36:07.989 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:36:07.989 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:36:08.039 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 11 2024-04-06T09:36:08.039 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 11 2024-04-06T09:36:08.039 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:36:08.039 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=45, tag_tid=6, entry_tid=1], [object_number=44, tag_tid=6, entry_tid=0], [object_number=47, tag_tid=5, entry_tid=22415], [object_number=46, tag_tid=5, entry_tid=22414]]], state=connected] 2024-04-06T09:36:08.039 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=47, tag_tid=5, entry_tid=20591], [object_number=46, tag_tid=5, entry_tid=20590], [object_number=45, tag_tid=5, entry_tid=20589], [object_number=44, tag_tid=5, entry_tid=20588]]], state=connected] 2024-04-06T09:36:08.042 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:36:08.044 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=45, tag_tid=6, entry_tid=1' 2024-04-06T09:36:08.045 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:36:08.045 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:36:08.045 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:36:08.045 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:36:08.045 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:36:08.045 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:36:08.045 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:36:08.045 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:36:08.045 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:36:08.045 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:36:08.046 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:36:08.046 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:36:08.046 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:36:08.047 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:36:08.047 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:36:08.301 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 11 2024-04-06T09:36:08.302 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 11 2024-04-06T09:36:08.302 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:36:08.302 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=45, tag_tid=6, entry_tid=1], [object_number=44, tag_tid=6, entry_tid=0], [object_number=47, tag_tid=5, entry_tid=22415], [object_number=46, tag_tid=5, entry_tid=22414]]], state=connected] 2024-04-06T09:36:08.302 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=47, tag_tid=5, entry_tid=20591], [object_number=46, tag_tid=5, entry_tid=20590], [object_number=45, tag_tid=5, entry_tid=20589], [object_number=44, tag_tid=5, entry_tid=20588]]], state=connected] 2024-04-06T09:36:08.305 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:36:08.307 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=47, tag_tid=5, entry_tid=20591' 2024-04-06T09:36:08.308 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=45, tag_tid=6, entry_tid=1' -a 'object_number=45, tag_tid=6, entry_tid=1' = 'object_number=47, tag_tid=5, entry_tid=20591' 2024-04-06T09:36:08.308 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=47, tag_tid=5, entry_tid=20591' '!=' 'object_number=44, tag_tid=5, entry_tid=20492' 2024-04-06T09:36:08.308 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:36:08.308 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=47, tag_tid=5, entry_tid=20591' = 'object_number=44, tag_tid=5, entry_tid=20492' 2024-04-06T09:36:08.308 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=47, tag_tid=5, entry_tid=20591' 2024-04-06T09:36:08.308 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=45, tag_tid=6, entry_tid=1' 2024-04-06T09:36:08.309 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:36:08.309 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:36:08.309 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=6 2024-04-06T09:36:08.310 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=47, tag_tid=5, entry_tid=20591' 2024-04-06T09:36:08.310 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:36:08.310 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:36:08.311 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=5 2024-04-06T09:36:08.312 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=45, tag_tid=6, entry_tid=1' 2024-04-06T09:36:08.312 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:36:08.312 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:36:08.313 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:36:08.314 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=47, tag_tid=5, entry_tid=20591' 2024-04-06T09:36:08.314 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:36:08.314 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:36:08.315 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=20591 2024-04-06T09:36:08.315 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 6 = 5 -a 1 -le 20591 2024-04-06T09:36:08.316 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:36:08.316 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:36:08.316 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:36:08.517 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:36:08.517 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:36:08.517 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:36:08.518 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:36:08.518 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:36:08.518 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:36:08.518 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:36:08.518 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:36:08.518 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:36:08.518 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:36:08.518 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:36:08.518 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:36:08.518 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:36:08.518 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:36:08.518 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:36:08.519 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:36:08.519 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:36:08.522 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:36:08.522 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:36:08.522 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:36:08.523 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:36:08.523 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:36:08.523 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:36:08.523 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:36:08.523 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:36:08.523 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:36:08.523 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:36:08.523 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:36:08.523 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:36:08.523 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:36:08.523 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:36:08.523 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:36:08.523 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:36:08.523 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:36:08.523 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:36:08.524 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:36:08.524 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:36:08.524 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:36:08.524 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:36:08.524 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:36:08.524 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:36:08.524 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:36:08.524 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:36:08.524 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:36:08.524 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:36:08.524 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:36:08.524 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:36:08.524 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:36:08.524 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:36:08.547 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:36:08.548 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:36:08.548 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:36:08.548 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:36:08.675 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:36:08.677 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:36:08.677 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:36:08.677 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:36:08.677 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:36:08.677 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:36:08.677 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:36:08.677 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:36:08.677 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:36:08.677 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:36:08.678 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:36:08.678 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:36:08.678 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:36:08.680 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:36:08.680 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:36:08.680 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:36:08.730 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 11 2024-04-06T09:36:08.730 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 11 2024-04-06T09:36:08.730 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:36:08.730 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=45, tag_tid=6, entry_tid=1], [object_number=44, tag_tid=6, entry_tid=0], [object_number=47, tag_tid=5, entry_tid=22415], [object_number=46, tag_tid=5, entry_tid=22414]]], state=connected] 2024-04-06T09:36:08.730 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=46, tag_tid=5, entry_tid=20722], [object_number=45, tag_tid=5, entry_tid=20721], [object_number=44, tag_tid=5, entry_tid=20720], [object_number=47, tag_tid=5, entry_tid=20719]]], state=connected] 2024-04-06T09:36:08.734 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:36:08.736 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=45, tag_tid=6, entry_tid=1' 2024-04-06T09:36:08.736 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:36:08.736 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:36:08.736 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:36:08.736 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:36:08.736 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:36:08.736 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:36:08.736 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:36:08.736 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:36:08.737 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:36:08.737 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:36:08.737 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:36:08.737 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:36:08.737 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:36:08.737 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:36:08.738 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:36:08.783 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 11 2024-04-06T09:36:08.783 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 11 2024-04-06T09:36:08.783 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:36:08.783 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=45, tag_tid=6, entry_tid=1], [object_number=44, tag_tid=6, entry_tid=0], [object_number=47, tag_tid=5, entry_tid=22415], [object_number=46, tag_tid=5, entry_tid=22414]]], state=connected] 2024-04-06T09:36:08.783 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=46, tag_tid=5, entry_tid=20722], [object_number=45, tag_tid=5, entry_tid=20721], [object_number=44, tag_tid=5, entry_tid=20720], [object_number=47, tag_tid=5, entry_tid=20719]]], state=connected] 2024-04-06T09:36:08.786 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:36:08.788 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=46, tag_tid=5, entry_tid=20722' 2024-04-06T09:36:08.788 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=45, tag_tid=6, entry_tid=1' -a 'object_number=45, tag_tid=6, entry_tid=1' = 'object_number=46, tag_tid=5, entry_tid=20722' 2024-04-06T09:36:08.789 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=46, tag_tid=5, entry_tid=20722' '!=' 'object_number=47, tag_tid=5, entry_tid=20591' 2024-04-06T09:36:08.789 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:36:08.789 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=46, tag_tid=5, entry_tid=20722' = 'object_number=47, tag_tid=5, entry_tid=20591' 2024-04-06T09:36:08.789 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=46, tag_tid=5, entry_tid=20722' 2024-04-06T09:36:08.789 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=45, tag_tid=6, entry_tid=1' 2024-04-06T09:36:08.789 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:36:08.789 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:36:08.789 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=6 2024-04-06T09:36:08.790 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=46, tag_tid=5, entry_tid=20722' 2024-04-06T09:36:08.790 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:36:08.790 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:36:08.791 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=5 2024-04-06T09:36:08.792 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=45, tag_tid=6, entry_tid=1' 2024-04-06T09:36:08.792 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:36:08.792 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:36:08.793 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:36:08.793 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=46, tag_tid=5, entry_tid=20722' 2024-04-06T09:36:08.793 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:36:08.794 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:36:08.795 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=20722 2024-04-06T09:36:08.795 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 6 = 5 -a 1 -le 20722 2024-04-06T09:36:08.795 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:36:08.795 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:36:08.795 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:36:08.996 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:36:08.996 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:36:08.997 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:36:08.997 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:36:08.997 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:36:08.997 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:36:08.997 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:36:08.997 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:36:08.997 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:36:08.997 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:36:08.997 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:36:08.997 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:36:08.997 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:36:08.997 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:36:08.997 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:36:08.998 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:36:08.998 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:36:09.000 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:36:09.000 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:36:09.000 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:36:09.000 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:36:09.000 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:36:09.000 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:36:09.000 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:36:09.000 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:36:09.000 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:36:09.000 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:36:09.000 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:36:09.001 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:36:09.001 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:36:09.001 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:36:09.001 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:36:09.001 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:36:09.002 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:36:09.002 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:36:09.002 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:36:09.002 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:36:09.002 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:36:09.002 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:36:09.002 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:36:09.002 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:36:09.003 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:36:09.003 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:36:09.003 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:36:09.003 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:36:09.003 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:36:09.003 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:36:09.003 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:36:09.003 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:36:09.027 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:36:09.027 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:36:09.027 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:36:09.027 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:36:09.154 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:36:09.155 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:36:09.155 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:36:09.156 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:36:09.156 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:36:09.156 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:36:09.156 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:36:09.156 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:36:09.156 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:36:09.156 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:36:09.157 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:36:09.157 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:36:09.157 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:36:09.158 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:36:09.159 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:36:09.159 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:36:09.205 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 11 2024-04-06T09:36:09.205 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 11 2024-04-06T09:36:09.205 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:36:09.205 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=45, tag_tid=6, entry_tid=1], [object_number=44, tag_tid=6, entry_tid=0], [object_number=47, tag_tid=5, entry_tid=22415], [object_number=46, tag_tid=5, entry_tid=22414]]], state=connected] 2024-04-06T09:36:09.205 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=44, tag_tid=5, entry_tid=20824], [object_number=47, tag_tid=5, entry_tid=20823], [object_number=46, tag_tid=5, entry_tid=20822], [object_number=45, tag_tid=5, entry_tid=20821]]], state=connected] 2024-04-06T09:36:09.209 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:36:09.211 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=45, tag_tid=6, entry_tid=1' 2024-04-06T09:36:09.211 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:36:09.211 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:36:09.211 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:36:09.212 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:36:09.212 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:36:09.212 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:36:09.212 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:36:09.212 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:36:09.212 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:36:09.212 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:36:09.212 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:36:09.212 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:36:09.213 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:36:09.213 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:36:09.213 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:36:09.258 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 11 2024-04-06T09:36:09.258 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 11 2024-04-06T09:36:09.258 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:36:09.258 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=45, tag_tid=6, entry_tid=1], [object_number=44, tag_tid=6, entry_tid=0], [object_number=47, tag_tid=5, entry_tid=22415], [object_number=46, tag_tid=5, entry_tid=22414]]], state=connected] 2024-04-06T09:36:09.258 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=44, tag_tid=5, entry_tid=20824], [object_number=47, tag_tid=5, entry_tid=20823], [object_number=46, tag_tid=5, entry_tid=20822], [object_number=45, tag_tid=5, entry_tid=20821]]], state=connected] 2024-04-06T09:36:09.262 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:36:09.263 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=44, tag_tid=5, entry_tid=20824' 2024-04-06T09:36:09.264 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=45, tag_tid=6, entry_tid=1' -a 'object_number=45, tag_tid=6, entry_tid=1' = 'object_number=44, tag_tid=5, entry_tid=20824' 2024-04-06T09:36:09.264 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=44, tag_tid=5, entry_tid=20824' '!=' 'object_number=46, tag_tid=5, entry_tid=20722' 2024-04-06T09:36:09.264 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:36:09.264 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=44, tag_tid=5, entry_tid=20824' = 'object_number=46, tag_tid=5, entry_tid=20722' 2024-04-06T09:36:09.264 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=44, tag_tid=5, entry_tid=20824' 2024-04-06T09:36:09.264 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=45, tag_tid=6, entry_tid=1' 2024-04-06T09:36:09.264 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:36:09.264 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:36:09.265 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=6 2024-04-06T09:36:09.265 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=44, tag_tid=5, entry_tid=20824' 2024-04-06T09:36:09.266 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:36:09.266 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:36:09.267 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=5 2024-04-06T09:36:09.267 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=45, tag_tid=6, entry_tid=1' 2024-04-06T09:36:09.267 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:36:09.267 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:36:09.268 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:36:09.269 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=44, tag_tid=5, entry_tid=20824' 2024-04-06T09:36:09.269 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:36:09.269 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:36:09.270 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=20824 2024-04-06T09:36:09.270 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 6 = 5 -a 1 -le 20824 2024-04-06T09:36:09.270 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:36:09.270 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:36:09.271 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:36:09.472 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:36:09.472 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:36:09.472 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:36:09.472 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:36:09.472 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:36:09.472 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:36:09.472 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:36:09.472 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:36:09.472 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:36:09.472 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:36:09.473 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:36:09.473 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:36:09.473 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:36:09.473 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:36:09.473 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:36:09.473 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:36:09.473 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:36:09.477 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:36:09.477 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:36:09.477 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:36:09.477 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:36:09.477 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:36:09.477 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:36:09.477 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:36:09.477 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:36:09.477 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:36:09.477 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:36:09.478 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:36:09.478 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:36:09.478 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:36:09.478 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:36:09.478 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:36:09.478 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:36:09.478 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:36:09.478 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:36:09.478 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:36:09.478 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:36:09.478 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:36:09.478 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:36:09.478 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:36:09.478 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:36:09.478 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:36:09.479 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:36:09.479 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:36:09.479 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:36:09.479 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:36:09.479 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:36:09.479 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:36:09.479 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:36:09.501 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:36:09.502 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:36:09.502 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:36:09.502 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:36:09.627 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:36:09.628 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:36:09.628 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:36:09.629 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:36:09.629 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:36:09.629 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:36:09.629 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:36:09.629 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:36:09.629 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:36:09.629 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:36:09.629 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:36:09.630 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:36:09.630 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:36:09.631 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:36:09.631 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:36:09.631 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:36:09.677 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 11 2024-04-06T09:36:09.677 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 11 2024-04-06T09:36:09.677 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:36:09.677 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=45, tag_tid=6, entry_tid=1], [object_number=44, tag_tid=6, entry_tid=0], [object_number=47, tag_tid=5, entry_tid=22415], [object_number=46, tag_tid=5, entry_tid=22414]]], state=connected] 2024-04-06T09:36:09.677 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=44, tag_tid=5, entry_tid=20932], [object_number=47, tag_tid=5, entry_tid=20931], [object_number=46, tag_tid=5, entry_tid=20930], [object_number=45, tag_tid=5, entry_tid=20929]]], state=connected] 2024-04-06T09:36:09.680 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:36:09.682 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=45, tag_tid=6, entry_tid=1' 2024-04-06T09:36:09.683 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:36:09.683 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:36:09.683 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:36:09.683 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:36:09.683 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:36:09.683 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:36:09.683 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:36:09.683 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:36:09.683 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:36:09.683 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:36:09.683 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:36:09.684 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:36:09.684 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:36:09.684 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:36:09.684 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:36:09.729 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 11 2024-04-06T09:36:09.729 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 11 2024-04-06T09:36:09.729 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:36:09.730 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=45, tag_tid=6, entry_tid=1], [object_number=44, tag_tid=6, entry_tid=0], [object_number=47, tag_tid=5, entry_tid=22415], [object_number=46, tag_tid=5, entry_tid=22414]]], state=connected] 2024-04-06T09:36:09.730 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=44, tag_tid=5, entry_tid=20932], [object_number=47, tag_tid=5, entry_tid=20931], [object_number=46, tag_tid=5, entry_tid=20930], [object_number=45, tag_tid=5, entry_tid=20929]]], state=connected] 2024-04-06T09:36:09.733 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:36:09.735 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=44, tag_tid=5, entry_tid=20932' 2024-04-06T09:36:09.735 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=45, tag_tid=6, entry_tid=1' -a 'object_number=45, tag_tid=6, entry_tid=1' = 'object_number=44, tag_tid=5, entry_tid=20932' 2024-04-06T09:36:09.735 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=44, tag_tid=5, entry_tid=20932' '!=' 'object_number=44, tag_tid=5, entry_tid=20824' 2024-04-06T09:36:09.735 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:36:09.735 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=44, tag_tid=5, entry_tid=20932' = 'object_number=44, tag_tid=5, entry_tid=20824' 2024-04-06T09:36:09.735 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=44, tag_tid=5, entry_tid=20932' 2024-04-06T09:36:09.736 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=45, tag_tid=6, entry_tid=1' 2024-04-06T09:36:09.736 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:36:09.736 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:36:09.736 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=6 2024-04-06T09:36:09.737 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=44, tag_tid=5, entry_tid=20932' 2024-04-06T09:36:09.737 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:36:09.737 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:36:09.738 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=5 2024-04-06T09:36:09.739 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=45, tag_tid=6, entry_tid=1' 2024-04-06T09:36:09.739 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:36:09.739 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:36:09.740 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:36:09.740 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=44, tag_tid=5, entry_tid=20932' 2024-04-06T09:36:09.741 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:36:09.741 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:36:09.742 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=20932 2024-04-06T09:36:09.742 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 6 = 5 -a 1 -le 20932 2024-04-06T09:36:09.742 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:36:09.742 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:36:09.742 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:36:09.943 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:36:09.943 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:36:09.944 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:36:09.944 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:36:09.944 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:36:09.944 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:36:09.944 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:36:09.944 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:36:09.944 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:36:09.944 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:36:09.944 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:36:09.944 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:36:09.944 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:36:09.944 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:36:09.945 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:36:09.945 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:36:09.945 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:36:09.948 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:36:09.948 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:36:09.948 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:36:09.948 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:36:09.949 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:36:09.949 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:36:09.949 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:36:09.949 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:36:09.949 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:36:09.949 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:36:09.949 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:36:09.949 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:36:09.949 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:36:09.949 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:36:09.949 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:36:09.949 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:36:09.949 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:36:09.949 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:36:09.949 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:36:09.950 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:36:09.950 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:36:09.950 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:36:09.950 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:36:09.950 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:36:09.950 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:36:09.950 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:36:09.950 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:36:09.950 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:36:09.950 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:36:09.950 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:36:09.950 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:36:09.950 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:36:09.976 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:36:09.976 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:36:09.976 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:36:09.976 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:36:10.105 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:36:10.106 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:36:10.106 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:36:10.106 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:36:10.106 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:36:10.106 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:36:10.107 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:36:10.107 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:36:10.107 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:36:10.107 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:36:10.107 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:36:10.107 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:36:10.107 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:36:10.109 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:36:10.109 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:36:10.109 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:36:10.372 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 11 2024-04-06T09:36:10.372 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 11 2024-04-06T09:36:10.372 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:36:10.372 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=45, tag_tid=6, entry_tid=1], [object_number=44, tag_tid=6, entry_tid=0], [object_number=47, tag_tid=5, entry_tid=22415], [object_number=46, tag_tid=5, entry_tid=22414]]], state=connected] 2024-04-06T09:36:10.372 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=45, tag_tid=5, entry_tid=21041], [object_number=44, tag_tid=5, entry_tid=21040], [object_number=47, tag_tid=5, entry_tid=21039], [object_number=46, tag_tid=5, entry_tid=21038]]], state=connected] 2024-04-06T09:36:10.376 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:36:10.377 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=45, tag_tid=6, entry_tid=1' 2024-04-06T09:36:10.378 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:36:10.378 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:36:10.378 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:36:10.378 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:36:10.378 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:36:10.379 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:36:10.379 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:36:10.379 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:36:10.379 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:36:10.379 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:36:10.379 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:36:10.379 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:36:10.380 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:36:10.380 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:36:10.380 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:36:10.429 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 11 2024-04-06T09:36:10.429 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 11 2024-04-06T09:36:10.429 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:36:10.429 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=45, tag_tid=6, entry_tid=1], [object_number=44, tag_tid=6, entry_tid=0], [object_number=47, tag_tid=5, entry_tid=22415], [object_number=46, tag_tid=5, entry_tid=22414]]], state=connected] 2024-04-06T09:36:10.429 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=45, tag_tid=5, entry_tid=21041], [object_number=44, tag_tid=5, entry_tid=21040], [object_number=47, tag_tid=5, entry_tid=21039], [object_number=46, tag_tid=5, entry_tid=21038]]], state=connected] 2024-04-06T09:36:10.433 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:36:10.435 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=45, tag_tid=5, entry_tid=21041' 2024-04-06T09:36:10.435 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=45, tag_tid=6, entry_tid=1' -a 'object_number=45, tag_tid=6, entry_tid=1' = 'object_number=45, tag_tid=5, entry_tid=21041' 2024-04-06T09:36:10.435 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=45, tag_tid=5, entry_tid=21041' '!=' 'object_number=44, tag_tid=5, entry_tid=20932' 2024-04-06T09:36:10.435 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:36:10.435 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=45, tag_tid=5, entry_tid=21041' = 'object_number=44, tag_tid=5, entry_tid=20932' 2024-04-06T09:36:10.435 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=45, tag_tid=5, entry_tid=21041' 2024-04-06T09:36:10.436 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=45, tag_tid=6, entry_tid=1' 2024-04-06T09:36:10.436 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:36:10.436 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:36:10.437 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=6 2024-04-06T09:36:10.437 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=45, tag_tid=5, entry_tid=21041' 2024-04-06T09:36:10.437 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:36:10.437 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:36:10.438 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=5 2024-04-06T09:36:10.439 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=45, tag_tid=6, entry_tid=1' 2024-04-06T09:36:10.439 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:36:10.439 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:36:10.440 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:36:10.441 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=45, tag_tid=5, entry_tid=21041' 2024-04-06T09:36:10.441 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:36:10.441 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:36:10.442 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=21041 2024-04-06T09:36:10.442 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 6 = 5 -a 1 -le 21041 2024-04-06T09:36:10.442 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:36:10.443 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:36:10.443 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:36:10.644 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:36:10.644 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:36:10.644 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:36:10.644 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:36:10.644 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:36:10.645 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:36:10.645 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:36:10.645 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:36:10.645 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:36:10.645 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:36:10.645 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:36:10.645 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:36:10.645 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:36:10.645 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:36:10.645 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:36:10.646 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:36:10.646 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:36:10.648 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:36:10.648 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:36:10.648 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:36:10.649 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:36:10.649 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:36:10.649 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:36:10.649 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:36:10.649 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:36:10.649 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:36:10.649 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:36:10.649 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:36:10.649 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:36:10.649 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:36:10.649 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:36:10.649 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:36:10.649 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:36:10.650 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:36:10.650 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:36:10.650 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:36:10.650 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:36:10.650 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:36:10.650 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:36:10.650 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:36:10.650 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:36:10.650 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:36:10.650 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:36:10.650 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:36:10.650 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:36:10.650 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:36:10.651 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:36:10.651 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:36:10.651 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:36:10.674 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:36:10.674 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:36:10.674 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:36:10.675 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:36:10.806 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:36:10.806 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:36:10.806 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:36:10.806 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:36:10.806 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:36:10.806 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:36:10.807 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:36:10.807 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:36:10.807 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:36:10.807 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:36:10.807 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:36:10.807 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:36:10.807 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:36:10.811 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:36:10.811 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:36:10.811 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:36:10.865 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 11 2024-04-06T09:36:10.865 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 11 2024-04-06T09:36:10.865 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:36:10.865 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=45, tag_tid=6, entry_tid=1], [object_number=44, tag_tid=6, entry_tid=0], [object_number=47, tag_tid=5, entry_tid=22415], [object_number=46, tag_tid=5, entry_tid=22414]]], state=connected] 2024-04-06T09:36:10.865 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=44, tag_tid=5, entry_tid=21164], [object_number=47, tag_tid=5, entry_tid=21163], [object_number=46, tag_tid=5, entry_tid=21162], [object_number=45, tag_tid=5, entry_tid=21161]]], state=connected] 2024-04-06T09:36:10.869 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:36:10.870 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=45, tag_tid=6, entry_tid=1' 2024-04-06T09:36:10.871 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:36:10.871 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:36:10.871 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:36:10.871 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:36:10.872 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:36:10.872 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:36:10.872 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:36:10.872 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:36:10.872 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:36:10.872 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:36:10.872 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:36:10.872 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:36:10.874 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:36:10.874 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:36:10.874 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:36:10.920 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 11 2024-04-06T09:36:10.920 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 11 2024-04-06T09:36:10.920 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:36:10.920 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=45, tag_tid=6, entry_tid=1], [object_number=44, tag_tid=6, entry_tid=0], [object_number=47, tag_tid=5, entry_tid=22415], [object_number=46, tag_tid=5, entry_tid=22414]]], state=connected] 2024-04-06T09:36:10.920 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=44, tag_tid=5, entry_tid=21164], [object_number=47, tag_tid=5, entry_tid=21163], [object_number=46, tag_tid=5, entry_tid=21162], [object_number=45, tag_tid=5, entry_tid=21161]]], state=connected] 2024-04-06T09:36:10.924 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:36:10.926 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=44, tag_tid=5, entry_tid=21164' 2024-04-06T09:36:10.926 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=45, tag_tid=6, entry_tid=1' -a 'object_number=45, tag_tid=6, entry_tid=1' = 'object_number=44, tag_tid=5, entry_tid=21164' 2024-04-06T09:36:10.926 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=44, tag_tid=5, entry_tid=21164' '!=' 'object_number=45, tag_tid=5, entry_tid=21041' 2024-04-06T09:36:10.926 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:36:10.926 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=44, tag_tid=5, entry_tid=21164' = 'object_number=45, tag_tid=5, entry_tid=21041' 2024-04-06T09:36:10.926 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=44, tag_tid=5, entry_tid=21164' 2024-04-06T09:36:10.926 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=45, tag_tid=6, entry_tid=1' 2024-04-06T09:36:10.926 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:36:10.927 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:36:10.927 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=6 2024-04-06T09:36:10.927 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=44, tag_tid=5, entry_tid=21164' 2024-04-06T09:36:10.928 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:36:10.928 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:36:10.929 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=5 2024-04-06T09:36:10.929 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=45, tag_tid=6, entry_tid=1' 2024-04-06T09:36:10.929 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:36:10.929 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:36:10.930 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:36:10.931 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=44, tag_tid=5, entry_tid=21164' 2024-04-06T09:36:10.931 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:36:10.931 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:36:10.933 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=21164 2024-04-06T09:36:10.933 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 6 = 5 -a 1 -le 21164 2024-04-06T09:36:10.933 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:36:10.933 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:36:10.933 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:36:11.134 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:36:11.134 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:36:11.134 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:36:11.134 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:36:11.134 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:36:11.135 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:36:11.135 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:36:11.135 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:36:11.135 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:36:11.135 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:36:11.135 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:36:11.135 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:36:11.135 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:36:11.135 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:36:11.135 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:36:11.136 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:36:11.137 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:36:11.139 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:36:11.139 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:36:11.139 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:36:11.139 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:36:11.139 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:36:11.139 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:36:11.139 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:36:11.139 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:36:11.139 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:36:11.140 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:36:11.140 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:36:11.140 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:36:11.140 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:36:11.140 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:36:11.140 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:36:11.140 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:36:11.141 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:36:11.141 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:36:11.141 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:36:11.141 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:36:11.141 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:36:11.141 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:36:11.141 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:36:11.141 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:36:11.142 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:36:11.142 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:36:11.142 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:36:11.142 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:36:11.142 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:36:11.142 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:36:11.142 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:36:11.142 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:36:11.166 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:36:11.166 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:36:11.166 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:36:11.166 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:36:11.299 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:36:11.301 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:36:11.301 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:36:11.301 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:36:11.301 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:36:11.301 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:36:11.301 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:36:11.301 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:36:11.301 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:36:11.301 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:36:11.302 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:36:11.302 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:36:11.302 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:36:11.303 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:36:11.303 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:36:11.304 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:36:11.361 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 11 2024-04-06T09:36:11.361 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 11 2024-04-06T09:36:11.361 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:36:11.361 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=45, tag_tid=6, entry_tid=1], [object_number=44, tag_tid=6, entry_tid=0], [object_number=47, tag_tid=5, entry_tid=22415], [object_number=46, tag_tid=5, entry_tid=22414]]], state=connected] 2024-04-06T09:36:11.361 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=46, tag_tid=5, entry_tid=21254], [object_number=45, tag_tid=5, entry_tid=21253], [object_number=44, tag_tid=5, entry_tid=21252], [object_number=47, tag_tid=5, entry_tid=21251]]], state=connected] 2024-04-06T09:36:11.365 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:36:11.366 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=45, tag_tid=6, entry_tid=1' 2024-04-06T09:36:11.367 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:36:11.367 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:36:11.367 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:36:11.368 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:36:11.368 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:36:11.368 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:36:11.368 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:36:11.368 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:36:11.368 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:36:11.368 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:36:11.368 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:36:11.368 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:36:11.368 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:36:11.369 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:36:11.369 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:36:11.415 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 11 2024-04-06T09:36:11.415 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 11 2024-04-06T09:36:11.415 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:36:11.415 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=45, tag_tid=6, entry_tid=1], [object_number=44, tag_tid=6, entry_tid=0], [object_number=47, tag_tid=5, entry_tid=22415], [object_number=46, tag_tid=5, entry_tid=22414]]], state=connected] 2024-04-06T09:36:11.415 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=46, tag_tid=5, entry_tid=21254], [object_number=45, tag_tid=5, entry_tid=21253], [object_number=44, tag_tid=5, entry_tid=21252], [object_number=47, tag_tid=5, entry_tid=21251]]], state=connected] 2024-04-06T09:36:11.419 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:36:11.421 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=46, tag_tid=5, entry_tid=21254' 2024-04-06T09:36:11.422 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=45, tag_tid=6, entry_tid=1' -a 'object_number=45, tag_tid=6, entry_tid=1' = 'object_number=46, tag_tid=5, entry_tid=21254' 2024-04-06T09:36:11.422 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=46, tag_tid=5, entry_tid=21254' '!=' 'object_number=44, tag_tid=5, entry_tid=21164' 2024-04-06T09:36:11.422 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:36:11.422 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=46, tag_tid=5, entry_tid=21254' = 'object_number=44, tag_tid=5, entry_tid=21164' 2024-04-06T09:36:11.422 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=46, tag_tid=5, entry_tid=21254' 2024-04-06T09:36:11.422 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=45, tag_tid=6, entry_tid=1' 2024-04-06T09:36:11.422 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:36:11.422 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:36:11.422 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=6 2024-04-06T09:36:11.423 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=46, tag_tid=5, entry_tid=21254' 2024-04-06T09:36:11.423 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:36:11.423 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:36:11.424 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=5 2024-04-06T09:36:11.425 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=45, tag_tid=6, entry_tid=1' 2024-04-06T09:36:11.425 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:36:11.425 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:36:11.426 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:36:11.426 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=46, tag_tid=5, entry_tid=21254' 2024-04-06T09:36:11.427 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:36:11.427 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:36:11.428 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=21254 2024-04-06T09:36:11.428 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 6 = 5 -a 1 -le 21254 2024-04-06T09:36:11.428 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:36:11.428 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:36:11.428 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:36:11.630 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:36:11.630 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:36:11.630 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:36:11.630 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:36:11.630 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:36:11.630 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:36:11.630 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:36:11.630 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:36:11.631 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:36:11.631 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:36:11.631 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:36:11.631 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:36:11.631 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:36:11.631 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:36:11.631 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:36:11.631 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:36:11.631 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:36:11.635 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:36:11.635 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:36:11.635 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:36:11.635 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:36:11.635 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:36:11.635 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:36:11.635 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:36:11.635 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:36:11.635 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:36:11.636 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:36:11.636 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:36:11.636 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:36:11.636 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:36:11.636 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:36:11.636 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:36:11.636 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:36:11.636 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:36:11.636 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:36:11.636 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:36:11.636 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:36:11.636 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:36:11.636 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:36:11.636 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:36:11.636 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:36:11.637 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:36:11.637 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:36:11.637 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:36:11.637 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:36:11.637 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:36:11.637 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:36:11.637 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:36:11.637 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:36:11.661 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:36:11.662 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:36:11.662 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:36:11.662 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:36:11.794 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:36:11.796 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:36:11.796 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:36:11.796 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:36:11.796 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:36:11.796 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:36:11.796 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:36:11.796 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:36:11.797 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:36:11.797 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:36:11.797 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:36:11.797 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:36:11.797 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:36:11.799 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:36:11.799 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:36:11.799 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:36:11.845 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 11 2024-04-06T09:36:11.845 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 11 2024-04-06T09:36:11.845 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:36:11.845 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=45, tag_tid=6, entry_tid=1], [object_number=44, tag_tid=6, entry_tid=0], [object_number=47, tag_tid=5, entry_tid=22415], [object_number=46, tag_tid=5, entry_tid=22414]]], state=connected] 2024-04-06T09:36:11.846 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=46, tag_tid=5, entry_tid=21346], [object_number=45, tag_tid=5, entry_tid=21345], [object_number=44, tag_tid=5, entry_tid=21344], [object_number=47, tag_tid=5, entry_tid=21343]]], state=connected] 2024-04-06T09:36:11.849 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:36:11.850 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=45, tag_tid=6, entry_tid=1' 2024-04-06T09:36:11.851 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:36:11.851 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:36:11.851 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:36:11.852 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:36:11.852 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:36:11.852 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:36:11.852 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:36:11.852 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:36:11.852 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:36:11.852 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:36:11.852 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:36:11.852 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:36:11.853 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:36:11.853 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:36:11.853 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:36:11.899 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 11 2024-04-06T09:36:11.899 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 11 2024-04-06T09:36:11.899 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:36:11.899 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=45, tag_tid=6, entry_tid=1], [object_number=44, tag_tid=6, entry_tid=0], [object_number=47, tag_tid=5, entry_tid=22415], [object_number=46, tag_tid=5, entry_tid=22414]]], state=connected] 2024-04-06T09:36:11.899 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=46, tag_tid=5, entry_tid=21346], [object_number=45, tag_tid=5, entry_tid=21345], [object_number=44, tag_tid=5, entry_tid=21344], [object_number=47, tag_tid=5, entry_tid=21343]]], state=connected] 2024-04-06T09:36:11.902 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:36:11.905 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=46, tag_tid=5, entry_tid=21346' 2024-04-06T09:36:11.905 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=45, tag_tid=6, entry_tid=1' -a 'object_number=45, tag_tid=6, entry_tid=1' = 'object_number=46, tag_tid=5, entry_tid=21346' 2024-04-06T09:36:11.905 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=46, tag_tid=5, entry_tid=21346' '!=' 'object_number=46, tag_tid=5, entry_tid=21254' 2024-04-06T09:36:11.905 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:36:11.905 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=46, tag_tid=5, entry_tid=21346' = 'object_number=46, tag_tid=5, entry_tid=21254' 2024-04-06T09:36:11.905 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=46, tag_tid=5, entry_tid=21346' 2024-04-06T09:36:11.905 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=45, tag_tid=6, entry_tid=1' 2024-04-06T09:36:11.905 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:36:11.905 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:36:11.906 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=6 2024-04-06T09:36:11.906 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=46, tag_tid=5, entry_tid=21346' 2024-04-06T09:36:11.907 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:36:11.907 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:36:11.908 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=5 2024-04-06T09:36:11.908 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=45, tag_tid=6, entry_tid=1' 2024-04-06T09:36:11.908 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:36:11.908 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:36:11.909 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:36:11.910 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=46, tag_tid=5, entry_tid=21346' 2024-04-06T09:36:11.910 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:36:11.910 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:36:11.911 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=21346 2024-04-06T09:36:11.911 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 6 = 5 -a 1 -le 21346 2024-04-06T09:36:11.911 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:36:11.911 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:36:11.911 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:36:12.113 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:36:12.113 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:36:12.113 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:36:12.113 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:36:12.113 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:36:12.113 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:36:12.113 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:36:12.114 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:36:12.114 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:36:12.114 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:36:12.114 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:36:12.114 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:36:12.114 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:36:12.114 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:36:12.114 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:36:12.115 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:36:12.115 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:36:12.118 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:36:12.118 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:36:12.118 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:36:12.118 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:36:12.118 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:36:12.119 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:36:12.119 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:36:12.119 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:36:12.119 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:36:12.119 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:36:12.119 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:36:12.119 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:36:12.119 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:36:12.119 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:36:12.119 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:36:12.119 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:36:12.119 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:36:12.119 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:36:12.119 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:36:12.120 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:36:12.120 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:36:12.120 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:36:12.120 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:36:12.120 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:36:12.120 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:36:12.120 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:36:12.120 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:36:12.120 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:36:12.120 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:36:12.120 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:36:12.120 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:36:12.120 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:36:12.143 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:36:12.144 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:36:12.144 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:36:12.144 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:36:12.275 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:36:12.276 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:36:12.276 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:36:12.276 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:36:12.277 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:36:12.277 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:36:12.277 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:36:12.277 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:36:12.277 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:36:12.277 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:36:12.277 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:36:12.277 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:36:12.277 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:36:12.278 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:36:12.279 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:36:12.279 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:36:12.330 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 11 2024-04-06T09:36:12.330 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 11 2024-04-06T09:36:12.331 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:36:12.331 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=45, tag_tid=6, entry_tid=1], [object_number=44, tag_tid=6, entry_tid=0], [object_number=47, tag_tid=5, entry_tid=22415], [object_number=46, tag_tid=5, entry_tid=22414]]], state=connected] 2024-04-06T09:36:12.331 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=47, tag_tid=5, entry_tid=21443], [object_number=46, tag_tid=5, entry_tid=21442], [object_number=45, tag_tid=5, entry_tid=21441], [object_number=44, tag_tid=5, entry_tid=21440]]], state=connected] 2024-04-06T09:36:12.334 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:36:12.337 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=45, tag_tid=6, entry_tid=1' 2024-04-06T09:36:12.337 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:36:12.337 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:36:12.337 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:36:12.337 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:36:12.337 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:36:12.337 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:36:12.337 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:36:12.338 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:36:12.338 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:36:12.338 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:36:12.338 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:36:12.338 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:36:12.338 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:36:12.338 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:36:12.338 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:36:12.384 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 11 2024-04-06T09:36:12.384 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 11 2024-04-06T09:36:12.384 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:36:12.384 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=45, tag_tid=6, entry_tid=1], [object_number=44, tag_tid=6, entry_tid=0], [object_number=47, tag_tid=5, entry_tid=22415], [object_number=46, tag_tid=5, entry_tid=22414]]], state=connected] 2024-04-06T09:36:12.384 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=47, tag_tid=5, entry_tid=21443], [object_number=46, tag_tid=5, entry_tid=21442], [object_number=45, tag_tid=5, entry_tid=21441], [object_number=44, tag_tid=5, entry_tid=21440]]], state=connected] 2024-04-06T09:36:12.387 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:36:12.390 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=47, tag_tid=5, entry_tid=21443' 2024-04-06T09:36:12.390 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=45, tag_tid=6, entry_tid=1' -a 'object_number=45, tag_tid=6, entry_tid=1' = 'object_number=47, tag_tid=5, entry_tid=21443' 2024-04-06T09:36:12.390 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=47, tag_tid=5, entry_tid=21443' '!=' 'object_number=46, tag_tid=5, entry_tid=21346' 2024-04-06T09:36:12.390 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:36:12.390 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=47, tag_tid=5, entry_tid=21443' = 'object_number=46, tag_tid=5, entry_tid=21346' 2024-04-06T09:36:12.390 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=47, tag_tid=5, entry_tid=21443' 2024-04-06T09:36:12.390 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=45, tag_tid=6, entry_tid=1' 2024-04-06T09:36:12.390 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:36:12.390 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:36:12.391 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=6 2024-04-06T09:36:12.391 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=47, tag_tid=5, entry_tid=21443' 2024-04-06T09:36:12.391 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:36:12.392 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:36:12.392 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=5 2024-04-06T09:36:12.393 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=45, tag_tid=6, entry_tid=1' 2024-04-06T09:36:12.393 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:36:12.393 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:36:12.394 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:36:12.395 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=47, tag_tid=5, entry_tid=21443' 2024-04-06T09:36:12.395 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:36:12.395 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:36:12.396 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=21443 2024-04-06T09:36:12.396 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 6 = 5 -a 1 -le 21443 2024-04-06T09:36:12.396 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:36:12.396 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:36:12.396 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:36:12.598 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:36:12.598 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:36:12.598 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:36:12.598 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:36:12.598 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:36:12.598 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:36:12.598 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:36:12.598 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:36:12.598 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:36:12.599 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:36:12.599 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:36:12.599 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:36:12.599 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:36:12.599 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:36:12.599 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:36:12.599 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:36:12.600 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:36:12.602 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:36:12.603 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:36:12.603 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:36:12.603 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:36:12.603 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:36:12.603 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:36:12.603 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:36:12.603 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:36:12.603 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:36:12.603 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:36:12.603 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:36:12.603 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:36:12.603 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:36:12.603 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:36:12.604 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:36:12.604 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:36:12.604 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:36:12.604 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:36:12.604 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:36:12.604 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:36:12.604 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:36:12.604 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:36:12.604 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:36:12.604 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:36:12.604 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:36:12.604 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:36:12.604 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:36:12.604 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:36:12.604 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:36:12.604 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:36:12.605 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:36:12.605 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:36:12.629 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:36:12.629 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:36:12.629 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:36:12.629 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:36:12.757 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:36:12.758 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:36:12.758 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:36:12.758 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:36:12.758 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:36:12.759 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:36:12.759 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:36:12.759 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:36:12.759 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:36:12.759 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:36:12.759 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:36:12.759 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:36:12.759 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:36:12.762 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:36:12.762 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:36:12.762 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:36:12.808 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 11 2024-04-06T09:36:12.808 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 11 2024-04-06T09:36:12.809 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:36:12.809 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=45, tag_tid=6, entry_tid=1], [object_number=44, tag_tid=6, entry_tid=0], [object_number=47, tag_tid=5, entry_tid=22415], [object_number=46, tag_tid=5, entry_tid=22414]]], state=connected] 2024-04-06T09:36:12.809 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=44, tag_tid=5, entry_tid=21540], [object_number=47, tag_tid=5, entry_tid=21539], [object_number=46, tag_tid=5, entry_tid=21538], [object_number=45, tag_tid=5, entry_tid=21537]]], state=connected] 2024-04-06T09:36:12.812 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:36:12.813 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=45, tag_tid=6, entry_tid=1' 2024-04-06T09:36:12.814 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:36:12.815 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:36:12.815 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:36:12.815 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:36:12.815 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:36:12.815 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:36:12.815 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:36:12.815 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:36:12.815 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:36:12.815 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:36:12.815 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:36:12.815 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:36:12.816 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:36:12.816 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:36:12.816 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:36:12.861 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 11 2024-04-06T09:36:12.861 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 11 2024-04-06T09:36:12.861 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:36:12.861 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=45, tag_tid=6, entry_tid=1], [object_number=44, tag_tid=6, entry_tid=0], [object_number=47, tag_tid=5, entry_tid=22415], [object_number=46, tag_tid=5, entry_tid=22414]]], state=connected] 2024-04-06T09:36:12.861 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=44, tag_tid=5, entry_tid=21540], [object_number=47, tag_tid=5, entry_tid=21539], [object_number=46, tag_tid=5, entry_tid=21538], [object_number=45, tag_tid=5, entry_tid=21537]]], state=connected] 2024-04-06T09:36:12.865 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:36:12.866 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=44, tag_tid=5, entry_tid=21540' 2024-04-06T09:36:12.866 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=45, tag_tid=6, entry_tid=1' -a 'object_number=45, tag_tid=6, entry_tid=1' = 'object_number=44, tag_tid=5, entry_tid=21540' 2024-04-06T09:36:12.867 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=44, tag_tid=5, entry_tid=21540' '!=' 'object_number=47, tag_tid=5, entry_tid=21443' 2024-04-06T09:36:12.867 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:36:12.867 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=44, tag_tid=5, entry_tid=21540' = 'object_number=47, tag_tid=5, entry_tid=21443' 2024-04-06T09:36:12.867 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=44, tag_tid=5, entry_tid=21540' 2024-04-06T09:36:12.867 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=45, tag_tid=6, entry_tid=1' 2024-04-06T09:36:12.867 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:36:12.867 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:36:12.868 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=6 2024-04-06T09:36:12.868 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=44, tag_tid=5, entry_tid=21540' 2024-04-06T09:36:12.869 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:36:12.869 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:36:12.870 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=5 2024-04-06T09:36:12.870 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=45, tag_tid=6, entry_tid=1' 2024-04-06T09:36:12.870 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:36:12.870 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:36:12.871 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:36:12.872 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=44, tag_tid=5, entry_tid=21540' 2024-04-06T09:36:12.872 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:36:12.872 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:36:12.873 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=21540 2024-04-06T09:36:12.873 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 6 = 5 -a 1 -le 21540 2024-04-06T09:36:12.873 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:36:12.873 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:36:12.873 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:36:13.075 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:36:13.075 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:36:13.075 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:36:13.075 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:36:13.075 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:36:13.075 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:36:13.075 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:36:13.075 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:36:13.075 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:36:13.075 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:36:13.075 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:36:13.076 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:36:13.076 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:36:13.076 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:36:13.076 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:36:13.076 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:36:13.077 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:36:13.079 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:36:13.079 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:36:13.079 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:36:13.079 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:36:13.079 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:36:13.079 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:36:13.080 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:36:13.080 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:36:13.080 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:36:13.080 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:36:13.080 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:36:13.080 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:36:13.080 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:36:13.080 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:36:13.080 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:36:13.080 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:36:13.081 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:36:13.081 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:36:13.081 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:36:13.081 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:36:13.081 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:36:13.081 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:36:13.081 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:36:13.081 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:36:13.081 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:36:13.082 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:36:13.082 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:36:13.082 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:36:13.082 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:36:13.082 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:36:13.082 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:36:13.082 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:36:13.107 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:36:13.107 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:36:13.107 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:36:13.107 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:36:13.233 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:36:13.234 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:36:13.234 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:36:13.234 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:36:13.234 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:36:13.234 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:36:13.234 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:36:13.234 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:36:13.234 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:36:13.235 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:36:13.235 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:36:13.235 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:36:13.235 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:36:13.237 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:36:13.237 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:36:13.237 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:36:13.282 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 11 2024-04-06T09:36:13.282 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 11 2024-04-06T09:36:13.282 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:36:13.282 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=45, tag_tid=6, entry_tid=1], [object_number=44, tag_tid=6, entry_tid=0], [object_number=47, tag_tid=5, entry_tid=22415], [object_number=46, tag_tid=5, entry_tid=22414]]], state=connected] 2024-04-06T09:36:13.282 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=46, tag_tid=5, entry_tid=21642], [object_number=45, tag_tid=5, entry_tid=21641], [object_number=44, tag_tid=5, entry_tid=21640], [object_number=47, tag_tid=5, entry_tid=21639]]], state=connected] 2024-04-06T09:36:13.286 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:36:13.287 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=45, tag_tid=6, entry_tid=1' 2024-04-06T09:36:13.288 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:36:13.288 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:36:13.288 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:36:13.288 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:36:13.288 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:36:13.288 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:36:13.288 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:36:13.288 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:36:13.288 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:36:13.288 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:36:13.288 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:36:13.289 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:36:13.289 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:36:13.289 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:36:13.289 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:36:13.334 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 11 2024-04-06T09:36:13.334 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 11 2024-04-06T09:36:13.334 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:36:13.334 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=45, tag_tid=6, entry_tid=1], [object_number=44, tag_tid=6, entry_tid=0], [object_number=47, tag_tid=5, entry_tid=22415], [object_number=46, tag_tid=5, entry_tid=22414]]], state=connected] 2024-04-06T09:36:13.334 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=46, tag_tid=5, entry_tid=21642], [object_number=45, tag_tid=5, entry_tid=21641], [object_number=44, tag_tid=5, entry_tid=21640], [object_number=47, tag_tid=5, entry_tid=21639]]], state=connected] 2024-04-06T09:36:13.338 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:36:13.339 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=46, tag_tid=5, entry_tid=21642' 2024-04-06T09:36:13.339 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=45, tag_tid=6, entry_tid=1' -a 'object_number=45, tag_tid=6, entry_tid=1' = 'object_number=46, tag_tid=5, entry_tid=21642' 2024-04-06T09:36:13.339 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=46, tag_tid=5, entry_tid=21642' '!=' 'object_number=44, tag_tid=5, entry_tid=21540' 2024-04-06T09:36:13.340 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:36:13.340 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=46, tag_tid=5, entry_tid=21642' = 'object_number=44, tag_tid=5, entry_tid=21540' 2024-04-06T09:36:13.340 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=46, tag_tid=5, entry_tid=21642' 2024-04-06T09:36:13.340 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=45, tag_tid=6, entry_tid=1' 2024-04-06T09:36:13.340 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:36:13.340 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:36:13.341 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=6 2024-04-06T09:36:13.342 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=46, tag_tid=5, entry_tid=21642' 2024-04-06T09:36:13.342 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:36:13.342 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:36:13.342 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=5 2024-04-06T09:36:13.343 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=45, tag_tid=6, entry_tid=1' 2024-04-06T09:36:13.343 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:36:13.343 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:36:13.344 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:36:13.345 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=46, tag_tid=5, entry_tid=21642' 2024-04-06T09:36:13.345 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:36:13.345 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:36:13.346 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=21642 2024-04-06T09:36:13.346 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 6 = 5 -a 1 -le 21642 2024-04-06T09:36:13.347 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:36:13.347 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:36:13.347 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:36:13.548 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:36:13.548 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:36:13.548 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:36:13.549 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:36:13.549 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:36:13.549 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:36:13.549 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:36:13.549 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:36:13.549 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:36:13.549 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:36:13.549 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:36:13.549 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:36:13.549 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:36:13.549 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:36:13.549 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:36:13.550 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:36:13.550 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:36:13.552 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:36:13.552 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:36:13.552 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:36:13.552 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:36:13.552 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:36:13.552 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:36:13.552 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:36:13.552 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:36:13.552 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:36:13.552 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:36:13.552 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:36:13.552 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:36:13.552 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:36:13.553 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:36:13.553 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:36:13.553 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:36:13.553 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:36:13.553 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:36:13.554 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:36:13.554 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:36:13.554 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:36:13.554 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:36:13.554 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:36:13.554 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:36:13.554 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:36:13.554 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:36:13.554 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:36:13.554 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:36:13.554 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:36:13.554 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:36:13.554 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:36:13.555 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:36:13.579 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:36:13.579 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:36:13.579 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:36:13.579 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:36:13.709 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:36:13.711 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:36:13.711 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:36:13.711 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:36:13.711 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:36:13.711 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:36:13.711 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:36:13.711 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:36:13.711 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:36:13.711 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:36:13.711 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:36:13.711 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:36:13.711 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:36:13.713 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:36:13.714 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:36:13.714 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:36:13.761 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 11 2024-04-06T09:36:13.761 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 11 2024-04-06T09:36:13.761 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:36:13.761 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=45, tag_tid=6, entry_tid=1], [object_number=44, tag_tid=6, entry_tid=0], [object_number=47, tag_tid=5, entry_tid=22415], [object_number=46, tag_tid=5, entry_tid=22414]]], state=connected] 2024-04-06T09:36:13.761 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=47, tag_tid=5, entry_tid=21739], [object_number=46, tag_tid=5, entry_tid=21738], [object_number=45, tag_tid=5, entry_tid=21737], [object_number=44, tag_tid=5, entry_tid=21736]]], state=connected] 2024-04-06T09:36:13.764 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:36:13.765 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=45, tag_tid=6, entry_tid=1' 2024-04-06T09:36:13.766 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:36:13.767 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:36:13.767 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:36:13.767 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:36:13.767 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:36:13.767 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:36:13.767 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:36:13.767 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:36:13.767 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:36:13.767 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:36:13.767 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:36:13.767 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:36:13.768 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:36:13.768 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:36:13.768 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:36:13.814 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 11 2024-04-06T09:36:13.814 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 11 2024-04-06T09:36:13.814 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:36:13.814 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=45, tag_tid=6, entry_tid=1], [object_number=44, tag_tid=6, entry_tid=0], [object_number=47, tag_tid=5, entry_tid=22415], [object_number=46, tag_tid=5, entry_tid=22414]]], state=connected] 2024-04-06T09:36:13.814 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=47, tag_tid=5, entry_tid=21739], [object_number=46, tag_tid=5, entry_tid=21738], [object_number=45, tag_tid=5, entry_tid=21737], [object_number=44, tag_tid=5, entry_tid=21736]]], state=connected] 2024-04-06T09:36:13.817 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:36:13.820 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=47, tag_tid=5, entry_tid=21739' 2024-04-06T09:36:13.820 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=45, tag_tid=6, entry_tid=1' -a 'object_number=45, tag_tid=6, entry_tid=1' = 'object_number=47, tag_tid=5, entry_tid=21739' 2024-04-06T09:36:13.820 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=47, tag_tid=5, entry_tid=21739' '!=' 'object_number=46, tag_tid=5, entry_tid=21642' 2024-04-06T09:36:13.820 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:36:13.820 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=47, tag_tid=5, entry_tid=21739' = 'object_number=46, tag_tid=5, entry_tid=21642' 2024-04-06T09:36:13.820 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=47, tag_tid=5, entry_tid=21739' 2024-04-06T09:36:13.820 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=45, tag_tid=6, entry_tid=1' 2024-04-06T09:36:13.820 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:36:13.820 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:36:13.821 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=6 2024-04-06T09:36:13.821 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=47, tag_tid=5, entry_tid=21739' 2024-04-06T09:36:13.822 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:36:13.822 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:36:13.823 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=5 2024-04-06T09:36:13.823 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=45, tag_tid=6, entry_tid=1' 2024-04-06T09:36:13.823 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:36:13.823 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:36:13.824 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:36:13.825 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=47, tag_tid=5, entry_tid=21739' 2024-04-06T09:36:13.825 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:36:13.825 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:36:13.826 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=21739 2024-04-06T09:36:13.826 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 6 = 5 -a 1 -le 21739 2024-04-06T09:36:13.826 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:36:13.826 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:36:13.826 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:36:14.028 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:36:14.028 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:36:14.028 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:36:14.028 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:36:14.028 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:36:14.028 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:36:14.028 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:36:14.028 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:36:14.028 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:36:14.028 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:36:14.029 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:36:14.029 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:36:14.029 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:36:14.029 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:36:14.029 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:36:14.029 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:36:14.030 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:36:14.032 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:36:14.033 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:36:14.033 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:36:14.033 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:36:14.033 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:36:14.033 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:36:14.033 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:36:14.033 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:36:14.033 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:36:14.033 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:36:14.033 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:36:14.033 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:36:14.033 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:36:14.033 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:36:14.033 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:36:14.033 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:36:14.034 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:36:14.034 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:36:14.034 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:36:14.035 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:36:14.035 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:36:14.035 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:36:14.035 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:36:14.035 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:36:14.035 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:36:14.035 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:36:14.035 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:36:14.035 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:36:14.035 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:36:14.035 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:36:14.035 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:36:14.035 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:36:14.059 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:36:14.060 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:36:14.060 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:36:14.060 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:36:14.194 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:36:14.196 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:36:14.196 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:36:14.196 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:36:14.196 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:36:14.196 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:36:14.196 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:36:14.196 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:36:14.196 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:36:14.196 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:36:14.197 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:36:14.197 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:36:14.197 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:36:14.198 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:36:14.199 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:36:14.199 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:36:14.243 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 11 2024-04-06T09:36:14.243 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 11 2024-04-06T09:36:14.243 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:36:14.243 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=45, tag_tid=6, entry_tid=1], [object_number=44, tag_tid=6, entry_tid=0], [object_number=47, tag_tid=5, entry_tid=22415], [object_number=46, tag_tid=5, entry_tid=22414]]], state=connected] 2024-04-06T09:36:14.244 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=44, tag_tid=5, entry_tid=21836], [object_number=47, tag_tid=5, entry_tid=21835], [object_number=46, tag_tid=5, entry_tid=21834], [object_number=45, tag_tid=5, entry_tid=21833]]], state=connected] 2024-04-06T09:36:14.247 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:36:14.248 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=45, tag_tid=6, entry_tid=1' 2024-04-06T09:36:14.249 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:36:14.249 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:36:14.249 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:36:14.249 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:36:14.249 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:36:14.249 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:36:14.250 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:36:14.250 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:36:14.250 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:36:14.250 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:36:14.250 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:36:14.250 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:36:14.251 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:36:14.251 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:36:14.251 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:36:14.296 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 11 2024-04-06T09:36:14.296 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 11 2024-04-06T09:36:14.296 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:36:14.296 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=45, tag_tid=6, entry_tid=1], [object_number=44, tag_tid=6, entry_tid=0], [object_number=47, tag_tid=5, entry_tid=22415], [object_number=46, tag_tid=5, entry_tid=22414]]], state=connected] 2024-04-06T09:36:14.296 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=44, tag_tid=5, entry_tid=21836], [object_number=47, tag_tid=5, entry_tid=21835], [object_number=46, tag_tid=5, entry_tid=21834], [object_number=45, tag_tid=5, entry_tid=21833]]], state=connected] 2024-04-06T09:36:14.299 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:36:14.302 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=44, tag_tid=5, entry_tid=21836' 2024-04-06T09:36:14.302 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=45, tag_tid=6, entry_tid=1' -a 'object_number=45, tag_tid=6, entry_tid=1' = 'object_number=44, tag_tid=5, entry_tid=21836' 2024-04-06T09:36:14.302 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=44, tag_tid=5, entry_tid=21836' '!=' 'object_number=47, tag_tid=5, entry_tid=21739' 2024-04-06T09:36:14.303 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:36:14.303 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=44, tag_tid=5, entry_tid=21836' = 'object_number=47, tag_tid=5, entry_tid=21739' 2024-04-06T09:36:14.303 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=44, tag_tid=5, entry_tid=21836' 2024-04-06T09:36:14.303 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=45, tag_tid=6, entry_tid=1' 2024-04-06T09:36:14.303 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:36:14.303 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:36:14.304 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=6 2024-04-06T09:36:14.305 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=44, tag_tid=5, entry_tid=21836' 2024-04-06T09:36:14.305 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:36:14.305 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:36:14.305 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=5 2024-04-06T09:36:14.306 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=45, tag_tid=6, entry_tid=1' 2024-04-06T09:36:14.306 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:36:14.306 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:36:14.307 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:36:14.307 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=44, tag_tid=5, entry_tid=21836' 2024-04-06T09:36:14.307 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:36:14.307 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:36:14.308 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=21836 2024-04-06T09:36:14.308 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 6 = 5 -a 1 -le 21836 2024-04-06T09:36:14.308 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:36:14.308 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:36:14.309 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:36:14.510 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:36:14.511 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:36:14.511 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:36:14.511 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:36:14.511 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:36:14.511 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:36:14.511 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:36:14.511 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:36:14.511 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:36:14.511 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:36:14.511 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:36:14.511 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:36:14.511 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:36:14.511 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:36:14.511 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:36:14.512 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:36:14.512 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:36:14.515 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:36:14.515 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:36:14.515 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:36:14.515 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:36:14.515 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:36:14.515 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:36:14.516 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:36:14.516 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:36:14.516 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:36:14.516 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:36:14.516 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:36:14.516 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:36:14.516 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:36:14.516 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:36:14.516 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:36:14.516 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:36:14.516 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:36:14.516 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:36:14.516 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:36:14.516 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:36:14.516 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:36:14.517 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:36:14.517 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:36:14.517 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:36:14.517 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:36:14.517 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:36:14.517 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:36:14.517 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:36:14.517 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:36:14.517 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:36:14.517 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:36:14.517 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:36:14.541 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:36:14.541 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:36:14.541 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:36:14.541 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:36:14.666 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:36:14.668 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:36:14.668 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:36:14.668 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:36:14.668 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:36:14.668 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:36:14.668 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:36:14.668 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:36:14.668 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:36:14.668 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:36:14.669 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:36:14.669 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:36:14.669 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:36:14.671 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:36:14.671 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:36:14.671 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:36:14.716 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 11 2024-04-06T09:36:14.716 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 11 2024-04-06T09:36:14.716 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:36:14.716 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=45, tag_tid=6, entry_tid=1], [object_number=44, tag_tid=6, entry_tid=0], [object_number=47, tag_tid=5, entry_tid=22415], [object_number=46, tag_tid=5, entry_tid=22414]]], state=connected] 2024-04-06T09:36:14.716 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=45, tag_tid=5, entry_tid=21937], [object_number=44, tag_tid=5, entry_tid=21936], [object_number=47, tag_tid=5, entry_tid=21935], [object_number=46, tag_tid=5, entry_tid=21934]]], state=connected] 2024-04-06T09:36:14.719 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:36:14.722 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=45, tag_tid=6, entry_tid=1' 2024-04-06T09:36:14.722 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:36:14.722 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:36:14.722 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:36:14.722 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:36:14.722 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:36:14.722 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:36:14.722 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:36:14.722 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:36:14.722 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:36:14.722 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:36:14.722 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:36:14.722 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:36:14.723 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:36:14.723 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:36:14.723 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:36:14.767 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 11 2024-04-06T09:36:14.768 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 11 2024-04-06T09:36:14.768 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:36:14.768 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=45, tag_tid=6, entry_tid=1], [object_number=44, tag_tid=6, entry_tid=0], [object_number=47, tag_tid=5, entry_tid=22415], [object_number=46, tag_tid=5, entry_tid=22414]]], state=connected] 2024-04-06T09:36:14.768 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=45, tag_tid=5, entry_tid=21937], [object_number=44, tag_tid=5, entry_tid=21936], [object_number=47, tag_tid=5, entry_tid=21935], [object_number=46, tag_tid=5, entry_tid=21934]]], state=connected] 2024-04-06T09:36:14.771 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:36:14.773 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=45, tag_tid=5, entry_tid=21937' 2024-04-06T09:36:14.773 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=45, tag_tid=6, entry_tid=1' -a 'object_number=45, tag_tid=6, entry_tid=1' = 'object_number=45, tag_tid=5, entry_tid=21937' 2024-04-06T09:36:14.773 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=45, tag_tid=5, entry_tid=21937' '!=' 'object_number=44, tag_tid=5, entry_tid=21836' 2024-04-06T09:36:14.773 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:36:14.773 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=45, tag_tid=5, entry_tid=21937' = 'object_number=44, tag_tid=5, entry_tid=21836' 2024-04-06T09:36:14.774 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=45, tag_tid=5, entry_tid=21937' 2024-04-06T09:36:14.774 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=45, tag_tid=6, entry_tid=1' 2024-04-06T09:36:14.774 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:36:14.774 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:36:14.774 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=6 2024-04-06T09:36:14.775 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=45, tag_tid=5, entry_tid=21937' 2024-04-06T09:36:14.775 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:36:14.775 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:36:14.776 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=5 2024-04-06T09:36:14.777 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=45, tag_tid=6, entry_tid=1' 2024-04-06T09:36:14.777 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:36:14.777 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:36:14.778 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:36:14.778 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=45, tag_tid=5, entry_tid=21937' 2024-04-06T09:36:14.778 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:36:14.779 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:36:14.780 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=21937 2024-04-06T09:36:14.780 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 6 = 5 -a 1 -le 21937 2024-04-06T09:36:14.780 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:36:14.780 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:36:14.780 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:36:14.981 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:36:14.982 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:36:14.982 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:36:14.982 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:36:14.982 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:36:14.982 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:36:14.982 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:36:14.982 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:36:14.982 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:36:14.982 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:36:14.982 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:36:14.982 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:36:14.983 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:36:14.983 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:36:14.983 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:36:14.983 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:36:14.984 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:36:14.986 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:36:14.986 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:36:14.986 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:36:14.986 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:36:14.986 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:36:14.987 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:36:14.987 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:36:14.987 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:36:14.987 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:36:14.987 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:36:14.987 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:36:14.987 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:36:14.987 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:36:14.987 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:36:14.987 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:36:14.987 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:36:14.988 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:36:14.988 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:36:14.988 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:36:14.988 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:36:14.988 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:36:14.988 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:36:14.989 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:36:14.989 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:36:14.989 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:36:14.989 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:36:14.989 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:36:14.989 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:36:14.989 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:36:14.989 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:36:14.989 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:36:14.989 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:36:15.013 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:36:15.014 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:36:15.014 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:36:15.014 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:36:15.140 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:36:15.141 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:36:15.141 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:36:15.141 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:36:15.142 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:36:15.142 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:36:15.142 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:36:15.142 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:36:15.142 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:36:15.142 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:36:15.143 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:36:15.143 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:36:15.143 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:36:15.143 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:36:15.144 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:36:15.144 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:36:15.195 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 11 2024-04-06T09:36:15.195 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 11 2024-04-06T09:36:15.195 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:36:15.196 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=45, tag_tid=6, entry_tid=1], [object_number=44, tag_tid=6, entry_tid=0], [object_number=47, tag_tid=5, entry_tid=22415], [object_number=46, tag_tid=5, entry_tid=22414]]], state=connected] 2024-04-06T09:36:15.196 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=46, tag_tid=5, entry_tid=22042], [object_number=45, tag_tid=5, entry_tid=22041], [object_number=44, tag_tid=5, entry_tid=22040], [object_number=47, tag_tid=5, entry_tid=22039]]], state=connected] 2024-04-06T09:36:15.199 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:36:15.202 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=45, tag_tid=6, entry_tid=1' 2024-04-06T09:36:15.202 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:36:15.202 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:36:15.202 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:36:15.202 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:36:15.202 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:36:15.202 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:36:15.202 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:36:15.202 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:36:15.202 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:36:15.203 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:36:15.203 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:36:15.203 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:36:15.203 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:36:15.204 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:36:15.204 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:36:15.249 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 11 2024-04-06T09:36:15.249 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 11 2024-04-06T09:36:15.249 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:36:15.249 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=45, tag_tid=6, entry_tid=1], [object_number=44, tag_tid=6, entry_tid=0], [object_number=47, tag_tid=5, entry_tid=22415], [object_number=46, tag_tid=5, entry_tid=22414]]], state=connected] 2024-04-06T09:36:15.250 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=46, tag_tid=5, entry_tid=22042], [object_number=45, tag_tid=5, entry_tid=22041], [object_number=44, tag_tid=5, entry_tid=22040], [object_number=47, tag_tid=5, entry_tid=22039]]], state=connected] 2024-04-06T09:36:15.253 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:36:15.255 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=46, tag_tid=5, entry_tid=22042' 2024-04-06T09:36:15.255 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=45, tag_tid=6, entry_tid=1' -a 'object_number=45, tag_tid=6, entry_tid=1' = 'object_number=46, tag_tid=5, entry_tid=22042' 2024-04-06T09:36:15.255 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=46, tag_tid=5, entry_tid=22042' '!=' 'object_number=45, tag_tid=5, entry_tid=21937' 2024-04-06T09:36:15.255 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:36:15.255 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=46, tag_tid=5, entry_tid=22042' = 'object_number=45, tag_tid=5, entry_tid=21937' 2024-04-06T09:36:15.255 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=46, tag_tid=5, entry_tid=22042' 2024-04-06T09:36:15.255 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=45, tag_tid=6, entry_tid=1' 2024-04-06T09:36:15.255 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:36:15.255 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:36:15.256 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=6 2024-04-06T09:36:15.257 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=46, tag_tid=5, entry_tid=22042' 2024-04-06T09:36:15.257 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:36:15.257 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:36:15.258 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=5 2024-04-06T09:36:15.258 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=45, tag_tid=6, entry_tid=1' 2024-04-06T09:36:15.258 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:36:15.259 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:36:15.259 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:36:15.260 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=46, tag_tid=5, entry_tid=22042' 2024-04-06T09:36:15.260 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:36:15.260 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:36:15.261 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=22042 2024-04-06T09:36:15.261 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 6 = 5 -a 1 -le 22042 2024-04-06T09:36:15.262 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:36:15.262 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:36:15.262 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:36:15.463 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:36:15.463 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:36:15.464 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:36:15.464 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:36:15.464 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:36:15.464 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:36:15.464 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:36:15.464 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:36:15.464 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:36:15.464 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:36:15.464 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:36:15.464 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:36:15.464 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:36:15.464 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:36:15.464 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:36:15.465 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:36:15.465 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:36:15.467 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:36:15.467 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:36:15.467 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:36:15.467 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:36:15.467 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:36:15.468 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:36:15.468 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:36:15.468 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:36:15.468 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:36:15.468 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:36:15.468 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:36:15.468 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:36:15.468 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:36:15.468 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:36:15.468 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:36:15.468 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:36:15.469 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:36:15.469 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:36:15.469 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:36:15.470 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:36:15.470 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:36:15.470 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:36:15.470 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:36:15.470 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:36:15.470 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:36:15.470 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:36:15.470 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:36:15.470 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:36:15.470 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:36:15.470 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:36:15.470 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:36:15.470 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:36:15.496 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:36:15.497 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:36:15.497 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:36:15.497 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:36:15.623 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:36:15.624 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:36:15.625 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:36:15.625 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:36:15.625 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:36:15.625 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:36:15.625 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:36:15.625 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:36:15.625 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:36:15.625 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:36:15.625 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:36:15.626 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:36:15.626 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:36:15.626 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:36:15.627 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:36:15.627 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:36:15.677 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 11 2024-04-06T09:36:15.677 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 11 2024-04-06T09:36:15.677 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:36:15.677 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=45, tag_tid=6, entry_tid=1], [object_number=44, tag_tid=6, entry_tid=0], [object_number=47, tag_tid=5, entry_tid=22415], [object_number=46, tag_tid=5, entry_tid=22414]]], state=connected] 2024-04-06T09:36:15.677 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=47, tag_tid=5, entry_tid=22143], [object_number=46, tag_tid=5, entry_tid=22142], [object_number=45, tag_tid=5, entry_tid=22141], [object_number=44, tag_tid=5, entry_tid=22140]]], state=connected] 2024-04-06T09:36:15.680 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:36:15.682 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=45, tag_tid=6, entry_tid=1' 2024-04-06T09:36:15.683 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:36:15.683 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:36:15.683 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:36:15.683 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:36:15.683 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:36:15.683 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:36:15.683 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:36:15.684 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:36:15.684 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:36:15.684 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:36:15.684 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:36:15.684 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:36:15.684 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:36:15.684 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:36:15.685 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:36:15.730 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 11 2024-04-06T09:36:15.730 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 11 2024-04-06T09:36:15.730 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:36:15.730 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=45, tag_tid=6, entry_tid=1], [object_number=44, tag_tid=6, entry_tid=0], [object_number=47, tag_tid=5, entry_tid=22415], [object_number=46, tag_tid=5, entry_tid=22414]]], state=connected] 2024-04-06T09:36:15.730 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=47, tag_tid=5, entry_tid=22143], [object_number=46, tag_tid=5, entry_tid=22142], [object_number=45, tag_tid=5, entry_tid=22141], [object_number=44, tag_tid=5, entry_tid=22140]]], state=connected] 2024-04-06T09:36:15.733 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:36:15.736 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=47, tag_tid=5, entry_tid=22143' 2024-04-06T09:36:15.736 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=45, tag_tid=6, entry_tid=1' -a 'object_number=45, tag_tid=6, entry_tid=1' = 'object_number=47, tag_tid=5, entry_tid=22143' 2024-04-06T09:36:15.736 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=47, tag_tid=5, entry_tid=22143' '!=' 'object_number=46, tag_tid=5, entry_tid=22042' 2024-04-06T09:36:15.736 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:36:15.736 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=47, tag_tid=5, entry_tid=22143' = 'object_number=46, tag_tid=5, entry_tid=22042' 2024-04-06T09:36:15.736 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=47, tag_tid=5, entry_tid=22143' 2024-04-06T09:36:15.736 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=45, tag_tid=6, entry_tid=1' 2024-04-06T09:36:15.736 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:36:15.736 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:36:15.736 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=6 2024-04-06T09:36:15.737 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=47, tag_tid=5, entry_tid=22143' 2024-04-06T09:36:15.737 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:36:15.737 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:36:15.738 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=5 2024-04-06T09:36:15.739 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=45, tag_tid=6, entry_tid=1' 2024-04-06T09:36:15.739 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:36:15.739 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:36:15.740 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:36:15.740 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=47, tag_tid=5, entry_tid=22143' 2024-04-06T09:36:15.741 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:36:15.741 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:36:15.742 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=22143 2024-04-06T09:36:15.742 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 6 = 5 -a 1 -le 22143 2024-04-06T09:36:15.742 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:36:15.742 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:36:15.742 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:36:15.944 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:36:15.944 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:36:15.944 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:36:15.944 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:36:15.944 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:36:15.944 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:36:15.944 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:36:15.944 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:36:15.944 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:36:15.944 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:36:15.945 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:36:15.945 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:36:15.945 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:36:15.945 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:36:15.945 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:36:15.945 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:36:15.945 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:36:15.947 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:36:15.947 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:36:15.947 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:36:15.948 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:36:15.948 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:36:15.948 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:36:15.948 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:36:15.948 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:36:15.948 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:36:15.948 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:36:15.948 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:36:15.948 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:36:15.948 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:36:15.948 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:36:15.948 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:36:15.948 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:36:15.950 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:36:15.950 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:36:15.950 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:36:15.950 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:36:15.950 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:36:15.950 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:36:15.950 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:36:15.950 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:36:15.950 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:36:15.950 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:36:15.950 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:36:15.950 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:36:15.950 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:36:15.951 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:36:15.951 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:36:15.951 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:36:15.975 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:36:15.975 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:36:15.976 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:36:15.976 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:36:16.112 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:36:16.112 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:36:16.112 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:36:16.112 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:36:16.112 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:36:16.112 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:36:16.112 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:36:16.112 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:36:16.112 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:36:16.112 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:36:16.113 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:36:16.113 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:36:16.113 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:36:16.114 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:36:16.115 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:36:16.115 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:36:16.162 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 11 2024-04-06T09:36:16.162 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 11 2024-04-06T09:36:16.162 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:36:16.162 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=45, tag_tid=6, entry_tid=1], [object_number=44, tag_tid=6, entry_tid=0], [object_number=47, tag_tid=5, entry_tid=22415], [object_number=46, tag_tid=5, entry_tid=22414]]], state=connected] 2024-04-06T09:36:16.162 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=46, tag_tid=5, entry_tid=22242], [object_number=45, tag_tid=5, entry_tid=22241], [object_number=44, tag_tid=5, entry_tid=22240], [object_number=47, tag_tid=5, entry_tid=22239]]], state=connected] 2024-04-06T09:36:16.165 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:36:16.166 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=45, tag_tid=6, entry_tid=1' 2024-04-06T09:36:16.167 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:36:16.168 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:36:16.168 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:36:16.168 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:36:16.168 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:36:16.168 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:36:16.168 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:36:16.168 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:36:16.168 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:36:16.168 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:36:16.168 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:36:16.168 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:36:16.169 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:36:16.169 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:36:16.169 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:36:16.214 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 11 2024-04-06T09:36:16.215 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 11 2024-04-06T09:36:16.215 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:36:16.215 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=45, tag_tid=6, entry_tid=1], [object_number=44, tag_tid=6, entry_tid=0], [object_number=47, tag_tid=5, entry_tid=22415], [object_number=46, tag_tid=5, entry_tid=22414]]], state=connected] 2024-04-06T09:36:16.215 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=46, tag_tid=5, entry_tid=22242], [object_number=45, tag_tid=5, entry_tid=22241], [object_number=44, tag_tid=5, entry_tid=22240], [object_number=47, tag_tid=5, entry_tid=22239]]], state=connected] 2024-04-06T09:36:16.218 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:36:16.220 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=46, tag_tid=5, entry_tid=22242' 2024-04-06T09:36:16.220 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=45, tag_tid=6, entry_tid=1' -a 'object_number=45, tag_tid=6, entry_tid=1' = 'object_number=46, tag_tid=5, entry_tid=22242' 2024-04-06T09:36:16.220 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=46, tag_tid=5, entry_tid=22242' '!=' 'object_number=47, tag_tid=5, entry_tid=22143' 2024-04-06T09:36:16.220 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:36:16.220 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=46, tag_tid=5, entry_tid=22242' = 'object_number=47, tag_tid=5, entry_tid=22143' 2024-04-06T09:36:16.221 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=46, tag_tid=5, entry_tid=22242' 2024-04-06T09:36:16.221 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=45, tag_tid=6, entry_tid=1' 2024-04-06T09:36:16.221 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:36:16.221 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:36:16.221 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=6 2024-04-06T09:36:16.222 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=46, tag_tid=5, entry_tid=22242' 2024-04-06T09:36:16.222 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:36:16.222 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:36:16.223 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=5 2024-04-06T09:36:16.224 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=45, tag_tid=6, entry_tid=1' 2024-04-06T09:36:16.224 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:36:16.224 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:36:16.224 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:36:16.225 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=46, tag_tid=5, entry_tid=22242' 2024-04-06T09:36:16.225 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:36:16.225 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:36:16.227 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=22242 2024-04-06T09:36:16.227 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 6 = 5 -a 1 -le 22242 2024-04-06T09:36:16.227 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:36:16.227 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:36:16.227 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:36:16.428 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:36:16.429 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:36:16.429 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:36:16.429 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:36:16.429 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:36:16.429 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:36:16.429 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:36:16.429 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:36:16.429 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:36:16.429 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:36:16.429 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:36:16.429 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:36:16.429 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:36:16.429 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:36:16.430 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:36:16.430 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:36:16.430 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:36:16.433 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:36:16.433 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:36:16.433 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:36:16.433 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:36:16.433 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:36:16.433 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:36:16.433 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:36:16.434 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:36:16.434 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:36:16.434 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:36:16.434 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:36:16.434 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:36:16.434 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:36:16.434 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:36:16.434 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:36:16.434 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:36:16.434 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:36:16.434 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:36:16.434 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:36:16.434 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:36:16.434 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:36:16.434 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:36:16.435 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:36:16.435 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:36:16.435 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:36:16.435 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:36:16.435 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:36:16.435 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:36:16.435 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:36:16.435 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:36:16.435 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:36:16.435 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:36:16.459 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:36:16.459 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:36:16.459 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:36:16.459 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:36:16.586 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:36:16.587 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:36:16.587 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:36:16.587 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:36:16.587 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:36:16.587 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:36:16.587 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:36:16.587 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:36:16.587 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:36:16.587 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:36:16.588 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:36:16.588 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:36:16.588 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:36:16.590 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:36:16.590 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:36:16.590 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:36:16.645 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 11 2024-04-06T09:36:16.645 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 11 2024-04-06T09:36:16.645 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:36:16.645 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=45, tag_tid=6, entry_tid=1], [object_number=44, tag_tid=6, entry_tid=0], [object_number=47, tag_tid=5, entry_tid=22415], [object_number=46, tag_tid=5, entry_tid=22414]]], state=connected] 2024-04-06T09:36:16.645 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=47, tag_tid=5, entry_tid=22351], [object_number=46, tag_tid=5, entry_tid=22350], [object_number=45, tag_tid=5, entry_tid=22349], [object_number=44, tag_tid=5, entry_tid=22348]]], state=connected] 2024-04-06T09:36:16.648 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:36:16.650 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=45, tag_tid=6, entry_tid=1' 2024-04-06T09:36:16.651 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:36:16.651 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:36:16.651 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:36:16.651 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:36:16.651 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:36:16.651 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:36:16.651 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:36:16.651 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:36:16.651 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:36:16.651 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:36:16.652 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:36:16.652 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:36:16.652 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:36:16.652 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:36:16.652 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:36:16.698 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 11 2024-04-06T09:36:16.699 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 11 2024-04-06T09:36:16.699 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:36:16.699 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=45, tag_tid=6, entry_tid=1], [object_number=44, tag_tid=6, entry_tid=0], [object_number=47, tag_tid=5, entry_tid=22415], [object_number=46, tag_tid=5, entry_tid=22414]]], state=connected] 2024-04-06T09:36:16.699 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=47, tag_tid=5, entry_tid=22351], [object_number=46, tag_tid=5, entry_tid=22350], [object_number=45, tag_tid=5, entry_tid=22349], [object_number=44, tag_tid=5, entry_tid=22348]]], state=connected] 2024-04-06T09:36:16.702 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:36:16.704 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=47, tag_tid=5, entry_tid=22351' 2024-04-06T09:36:16.704 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=45, tag_tid=6, entry_tid=1' -a 'object_number=45, tag_tid=6, entry_tid=1' = 'object_number=47, tag_tid=5, entry_tid=22351' 2024-04-06T09:36:16.704 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=47, tag_tid=5, entry_tid=22351' '!=' 'object_number=46, tag_tid=5, entry_tid=22242' 2024-04-06T09:36:16.705 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:36:16.705 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=47, tag_tid=5, entry_tid=22351' = 'object_number=46, tag_tid=5, entry_tid=22242' 2024-04-06T09:36:16.705 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=47, tag_tid=5, entry_tid=22351' 2024-04-06T09:36:16.705 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=45, tag_tid=6, entry_tid=1' 2024-04-06T09:36:16.705 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:36:16.705 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:36:16.706 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=6 2024-04-06T09:36:16.706 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=47, tag_tid=5, entry_tid=22351' 2024-04-06T09:36:16.706 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:36:16.706 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:36:16.707 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=5 2024-04-06T09:36:16.708 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=45, tag_tid=6, entry_tid=1' 2024-04-06T09:36:16.708 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:36:16.708 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:36:16.709 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:36:16.709 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=47, tag_tid=5, entry_tid=22351' 2024-04-06T09:36:16.710 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:36:16.710 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:36:16.711 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=22351 2024-04-06T09:36:16.711 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 6 = 5 -a 1 -le 22351 2024-04-06T09:36:16.711 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:36:16.711 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:36:16.711 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:36:16.912 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:36:16.912 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:36:16.912 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:36:16.912 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:36:16.913 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:36:16.913 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:36:16.913 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:36:16.913 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:36:16.913 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:36:16.913 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:36:16.913 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:36:16.913 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:36:16.913 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:36:16.913 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:36:16.913 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:36:16.914 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:36:16.914 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:36:16.918 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:36:16.918 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:36:16.918 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:36:16.918 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:36:16.919 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:36:16.919 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:36:16.919 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:36:16.919 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:36:16.919 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:36:16.919 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:36:16.919 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:36:16.919 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:36:16.919 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:36:16.919 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:36:16.919 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:36:16.919 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:36:16.919 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:36:16.919 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:36:16.919 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:36:16.919 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:36:16.919 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:36:16.920 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:36:16.920 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:36:16.920 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:36:16.920 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:36:16.920 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:36:16.920 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:36:16.920 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:36:16.920 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:36:16.920 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:36:16.920 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:36:16.920 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:36:16.944 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:36:16.944 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:36:16.944 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:36:16.944 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:36:17.064 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:36:17.065 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:36:17.065 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:36:17.065 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:36:17.065 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:36:17.065 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:36:17.066 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:36:17.066 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:36:17.066 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:36:17.066 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:36:17.066 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:36:17.066 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:36:17.067 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:36:17.069 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:36:17.070 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:36:17.070 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:36:17.120 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 11 2024-04-06T09:36:17.120 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 11 2024-04-06T09:36:17.121 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:36:17.121 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=45, tag_tid=6, entry_tid=1], [object_number=44, tag_tid=6, entry_tid=0], [object_number=47, tag_tid=5, entry_tid=22415], [object_number=46, tag_tid=5, entry_tid=22414]]], state=connected] 2024-04-06T09:36:17.121 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=47, tag_tid=5, entry_tid=22415], [object_number=46, tag_tid=5, entry_tid=22414], [object_number=45, tag_tid=5, entry_tid=22413], [object_number=44, tag_tid=5, entry_tid=22412]]], state=connected] 2024-04-06T09:36:17.124 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:36:17.125 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=45, tag_tid=6, entry_tid=1' 2024-04-06T09:36:17.126 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:36:17.126 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:36:17.126 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:36:17.126 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:36:17.127 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:36:17.127 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:36:17.127 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:36:17.127 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:36:17.127 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:36:17.127 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:36:17.127 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:36:17.127 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:36:17.128 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:36:17.128 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:36:17.128 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:36:17.173 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 11 2024-04-06T09:36:17.173 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 11 2024-04-06T09:36:17.173 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:36:17.173 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=45, tag_tid=6, entry_tid=1], [object_number=44, tag_tid=6, entry_tid=0], [object_number=47, tag_tid=5, entry_tid=22415], [object_number=46, tag_tid=5, entry_tid=22414]]], state=connected] 2024-04-06T09:36:17.173 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=47, tag_tid=5, entry_tid=22415], [object_number=46, tag_tid=5, entry_tid=22414], [object_number=45, tag_tid=5, entry_tid=22413], [object_number=44, tag_tid=5, entry_tid=22412]]], state=connected] 2024-04-06T09:36:17.176 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:36:17.179 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=47, tag_tid=5, entry_tid=22415' 2024-04-06T09:36:17.179 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=45, tag_tid=6, entry_tid=1' -a 'object_number=45, tag_tid=6, entry_tid=1' = 'object_number=47, tag_tid=5, entry_tid=22415' 2024-04-06T09:36:17.179 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=47, tag_tid=5, entry_tid=22415' '!=' 'object_number=47, tag_tid=5, entry_tid=22351' 2024-04-06T09:36:17.179 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:36:17.179 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=47, tag_tid=5, entry_tid=22415' = 'object_number=47, tag_tid=5, entry_tid=22351' 2024-04-06T09:36:17.179 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=47, tag_tid=5, entry_tid=22415' 2024-04-06T09:36:17.179 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=45, tag_tid=6, entry_tid=1' 2024-04-06T09:36:17.179 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:36:17.179 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:36:17.180 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=6 2024-04-06T09:36:17.180 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=47, tag_tid=5, entry_tid=22415' 2024-04-06T09:36:17.180 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:36:17.181 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:36:17.181 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=5 2024-04-06T09:36:17.182 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=45, tag_tid=6, entry_tid=1' 2024-04-06T09:36:17.182 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:36:17.182 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:36:17.183 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:36:17.184 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=47, tag_tid=5, entry_tid=22415' 2024-04-06T09:36:17.184 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:36:17.184 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:36:17.185 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=22415 2024-04-06T09:36:17.185 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 6 = 5 -a 1 -le 22415 2024-04-06T09:36:17.185 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:36:17.185 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:36:17.185 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:36:17.387 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:36:17.387 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:36:17.387 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:36:17.387 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:36:17.387 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:36:17.387 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:36:17.387 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:36:17.387 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:36:17.387 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:36:17.387 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:36:17.387 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:36:17.388 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:36:17.388 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:36:17.388 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:36:17.388 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:36:17.388 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:36:17.388 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:36:17.392 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:36:17.392 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:36:17.392 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:36:17.392 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:36:17.392 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:36:17.392 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:36:17.392 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:36:17.393 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:36:17.393 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:36:17.393 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:36:17.393 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:36:17.393 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:36:17.393 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:36:17.393 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:36:17.393 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:36:17.393 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:36:17.393 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:36:17.393 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:36:17.393 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:36:17.393 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:36:17.393 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:36:17.393 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:36:17.394 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:36:17.394 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:36:17.394 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:36:17.394 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:36:17.394 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:36:17.394 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:36:17.394 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:36:17.394 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:36:17.394 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:36:17.394 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:36:17.419 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:36:17.419 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:36:17.419 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:36:17.419 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:36:17.539 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:36:17.540 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:36:17.541 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:36:17.541 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:36:17.541 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:36:17.541 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:36:17.541 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:36:17.541 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:36:17.541 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:36:17.541 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:36:17.541 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:36:17.541 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:36:17.541 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:36:17.543 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:36:17.543 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:36:17.543 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:36:17.799 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 11 2024-04-06T09:36:17.799 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 11 2024-04-06T09:36:17.799 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:36:17.799 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=45, tag_tid=6, entry_tid=1], [object_number=44, tag_tid=6, entry_tid=0], [object_number=47, tag_tid=5, entry_tid=22415], [object_number=46, tag_tid=5, entry_tid=22414]]], state=connected] 2024-04-06T09:36:17.799 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=47, tag_tid=5, entry_tid=22415], [object_number=46, tag_tid=5, entry_tid=22414], [object_number=45, tag_tid=5, entry_tid=22413], [object_number=44, tag_tid=5, entry_tid=22412]]], state=connected] 2024-04-06T09:36:17.804 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:36:17.806 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=45, tag_tid=6, entry_tid=1' 2024-04-06T09:36:17.808 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:36:17.808 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:36:17.808 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:36:17.808 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:36:17.808 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:36:17.808 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:36:17.808 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:36:17.808 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:36:17.808 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:36:17.808 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:36:17.809 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:36:17.809 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:36:17.811 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:36:17.811 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:36:17.811 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:36:17.866 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 11 2024-04-06T09:36:17.866 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 11 2024-04-06T09:36:17.866 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:36:17.866 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=45, tag_tid=6, entry_tid=1], [object_number=44, tag_tid=6, entry_tid=0], [object_number=47, tag_tid=5, entry_tid=22415], [object_number=46, tag_tid=5, entry_tid=22414]]], state=connected] 2024-04-06T09:36:17.866 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=47, tag_tid=5, entry_tid=22415], [object_number=46, tag_tid=5, entry_tid=22414], [object_number=45, tag_tid=5, entry_tid=22413], [object_number=44, tag_tid=5, entry_tid=22412]]], state=connected] 2024-04-06T09:36:17.871 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:36:17.872 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=47, tag_tid=5, entry_tid=22415' 2024-04-06T09:36:17.872 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=45, tag_tid=6, entry_tid=1' -a 'object_number=45, tag_tid=6, entry_tid=1' = 'object_number=47, tag_tid=5, entry_tid=22415' 2024-04-06T09:36:17.872 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=47, tag_tid=5, entry_tid=22415' '!=' 'object_number=47, tag_tid=5, entry_tid=22415' 2024-04-06T09:36:17.873 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:36:17.873 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.4 2024-04-06T09:36:18.275 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:36:18.275 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:36:18.275 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:36:18.275 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:36:18.275 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:36:18.275 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:36:18.275 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:36:18.275 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:36:18.275 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:36:18.275 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:36:18.275 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:36:18.275 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:36:18.275 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:36:18.276 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:36:18.276 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:36:18.276 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:36:18.276 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:36:18.280 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:36:18.280 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:36:18.280 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:36:18.280 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:36:18.280 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:36:18.280 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:36:18.280 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:36:18.280 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:36:18.280 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:36:18.280 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:36:18.280 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:36:18.280 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:36:18.281 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:36:18.281 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:36:18.281 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:36:18.281 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:36:18.281 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:36:18.281 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:36:18.281 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:36:18.281 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:36:18.281 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:36:18.281 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:36:18.281 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:36:18.281 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:36:18.281 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:36:18.281 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:36:18.281 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:36:18.281 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:36:18.282 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:36:18.282 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:36:18.282 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:36:18.282 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:36:18.304 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:36:18.304 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:36:18.304 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:36:18.305 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:36:18.430 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:36:18.431 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:36:18.431 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:36:18.432 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:36:18.432 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:36:18.432 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:36:18.432 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:36:18.432 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:36:18.432 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:36:18.432 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:36:18.432 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:36:18.433 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:36:18.433 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:36:18.434 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:36:18.434 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:36:18.434 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:36:18.484 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 11 2024-04-06T09:36:18.484 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 11 2024-04-06T09:36:18.484 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:36:18.484 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=45, tag_tid=6, entry_tid=1], [object_number=44, tag_tid=6, entry_tid=0], [object_number=47, tag_tid=5, entry_tid=22415], [object_number=46, tag_tid=5, entry_tid=22414]]], state=connected] 2024-04-06T09:36:18.484 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=45, tag_tid=6, entry_tid=1], [object_number=44, tag_tid=6, entry_tid=0], [object_number=47, tag_tid=5, entry_tid=22415], [object_number=46, tag_tid=5, entry_tid=22414]]], state=connected] 2024-04-06T09:36:18.487 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:36:18.489 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=45, tag_tid=6, entry_tid=1' 2024-04-06T09:36:18.490 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:36:18.490 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:36:18.490 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:36:18.490 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:36:18.490 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:36:18.490 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:36:18.490 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:36:18.490 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:36:18.490 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:36:18.491 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:36:18.491 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:36:18.491 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:36:18.491 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:36:18.491 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:36:18.491 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:36:18.537 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 11 2024-04-06T09:36:18.537 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 11 2024-04-06T09:36:18.537 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:36:18.538 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=45, tag_tid=6, entry_tid=1], [object_number=44, tag_tid=6, entry_tid=0], [object_number=47, tag_tid=5, entry_tid=22415], [object_number=46, tag_tid=5, entry_tid=22414]]], state=connected] 2024-04-06T09:36:18.538 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=45, tag_tid=6, entry_tid=1], [object_number=44, tag_tid=6, entry_tid=0], [object_number=47, tag_tid=5, entry_tid=22415], [object_number=46, tag_tid=5, entry_tid=22414]]], state=connected] 2024-04-06T09:36:18.541 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:36:18.543 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=45, tag_tid=6, entry_tid=1' 2024-04-06T09:36:18.543 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=45, tag_tid=6, entry_tid=1' -a 'object_number=45, tag_tid=6, entry_tid=1' = 'object_number=45, tag_tid=6, entry_tid=1' 2024-04-06T09:36:18.543 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:36:18.543 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_snap_present cluster1 mirror test snap2 2024-04-06T09:36:18.543 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:36:18.543 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:36:18.543 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:36:18.543 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local snap_name=snap2 2024-04-06T09:36:18.543 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local s 2024-04-06T09:36:18.543 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-04-06T09:36:18.544 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 1 2024-04-06T09:36:19.544 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 info mirror/test@snap2 2024-04-06T09:36:19.609 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:rbd image 'test': 2024-04-06T09:36:19.609 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: size 512 MiB in 128 objects 2024-04-06T09:36:19.609 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: order 22 (4 MiB objects) 2024-04-06T09:36:19.610 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: snapshot_count: 3 2024-04-06T09:36:19.610 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: id: 10a56da0b207 2024-04-06T09:36:19.610 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: block_name_prefix: rbd_data.10a56da0b207 2024-04-06T09:36:19.610 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: format: 2 2024-04-06T09:36:19.610 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: features: layering, exclusive-lock, journaling 2024-04-06T09:36:19.610 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: op_features: 2024-04-06T09:36:19.610 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: flags: 2024-04-06T09:36:19.610 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: create_timestamp: Sat Apr 6 09:31:59 2024 2024-04-06T09:36:19.610 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: access_timestamp: Sat Apr 6 09:34:13 2024 2024-04-06T09:36:19.610 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: modify_timestamp: Sat Apr 6 09:36:03 2024 2024-04-06T09:36:19.610 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: protected: False 2024-04-06T09:36:19.610 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: journal: 10a56da0b207 2024-04-06T09:36:19.610 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: mirroring state: enabled 2024-04-06T09:36:19.610 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: mirroring mode: journal 2024-04-06T09:36:19.610 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: mirroring global id: 8ad5dedc-b3e0-425d-8993-b3e2231fc5ca 2024-04-06T09:36:19.610 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: mirroring primary: false 2024-04-06T09:36:19.614 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:36:19.614 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n snap1-clean ']' 2024-04-06T09:36:19.614 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ compare_image_snaps mirror test snap1-clean 2024-04-06T09:36:19.615 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:36:19.615 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:36:19.615 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local snap_name=snap1-clean 2024-04-06T09:36:19.615 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local ret=0 2024-04-06T09:36:19.615 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local rmt_export=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.export 2024-04-06T09:36:19.615 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loc_export=/tmp/tmp.BhW6xKmROU/cluster1-mirror-test.export 2024-04-06T09:36:19.615 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rm -f /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.export /tmp/tmp.BhW6xKmROU/cluster1-mirror-test.export 2024-04-06T09:36:19.615 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster2 -p mirror export test@snap1-clean /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.export 2024-04-06T09:36:20.139 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-04-06T09:36:20.149 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 -p mirror export test@snap1-clean /tmp/tmp.BhW6xKmROU/cluster1-mirror-test.export 2024-04-06T09:36:21.277 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-04-06T09:36:21.302 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmp /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.export /tmp/tmp.BhW6xKmROU/cluster1-mirror-test.export 2024-04-06T09:36:21.581 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rm -f /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.export /tmp/tmp.BhW6xKmROU/cluster1-mirror-test.export 2024-04-06T09:36:21.718 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:36:21.718 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ compare_image_snaps mirror test snap2 2024-04-06T09:36:21.718 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:36:21.718 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:36:21.718 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local snap_name=snap2 2024-04-06T09:36:21.719 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local ret=0 2024-04-06T09:36:21.719 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local rmt_export=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.export 2024-04-06T09:36:21.719 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loc_export=/tmp/tmp.BhW6xKmROU/cluster1-mirror-test.export 2024-04-06T09:36:21.719 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rm -f /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.export /tmp/tmp.BhW6xKmROU/cluster1-mirror-test.export 2024-04-06T09:36:21.719 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster2 -p mirror export test@snap2 /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.export 2024-04-06T09:36:22.275 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-04-06T09:36:22.287 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 -p mirror export test@snap2 /tmp/tmp.BhW6xKmROU/cluster1-mirror-test.export 2024-04-06T09:36:23.646 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-04-06T09:36:23.667 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmp /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.export /tmp/tmp.BhW6xKmROU/cluster1-mirror-test.export 2024-04-06T09:36:23.947 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rm -f /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.export /tmp/tmp.BhW6xKmROU/cluster1-mirror-test.export 2024-04-06T09:36:24.083 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:36:24.083 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ clean_snap_name=snap2-clean 2024-04-06T09:36:24.083 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ create_snap cluster2 mirror test snap2-clean 2024-04-06T09:36:24.083 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-04-06T09:36:24.083 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:36:24.084 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:36:24.084 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local snap_name=snap2-clean 2024-04-06T09:36:24.084 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap2-clean --debug-rbd=20 --debug-journaler=20 2024-04-06T09:36:24.489 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in `seq 1 10` 2024-04-06T09:36:24.490 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ stress_write_image cluster2 mirror test 2024-04-06T09:36:24.490 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-04-06T09:36:24.490 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:36:24.490 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:36:24.490 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-04-06T09:36:24.492 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local duration=9 2024-04-06T09:36:24.493 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set +e 2024-04-06T09:36:24.493 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ timeout 9s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror test --debug-rbd=20 --debug-journaler=20 2024-04-06T09:36:26.254 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:100 IOs, 51373056 bytes 2024-04-06T09:36:28.118 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:200 IOs, 87299072 bytes 2024-04-06T09:36:29.089 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:300 IOs, 107499520 bytes 2024-04-06T09:36:29.379 INFO:tasks.rbd_mirror_thrash:kill cluster2.client.mirror.0 2024-04-06T09:36:29.379 INFO:tasks.rbd_mirror.cluster2.client.mirror.0:Sent signal 15 2024-04-06T09:36:29.379 INFO:tasks.rbd_mirror_thrash:waiting for 4 secs before reviving daemons 2024-04-06T09:36:29.380 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi192.stderr:2024-04-06T09:36:29.378+0000 7fbfca79a640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.0 (PID: 27482) UID: 1000 2024-04-06T09:36:31.444 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:400 IOs, 155421696 bytes 2024-04-06T09:36:32.803 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:500 IOs, 184172544 bytes 2024-04-06T09:36:33.380 INFO:tasks.rbd_mirror_thrash:waiting for cluster2.client.mirror.0 2024-04-06T09:36:33.380 INFO:teuthology.orchestra.run:waiting for 600 2024-04-06T09:36:33.381 INFO:tasks.rbd_mirror_thrash:reviving cluster2.client.mirror.0 2024-04-06T09:36:33.381 INFO:tasks.rbd_mirror.cluster2.client.mirror.0:Restarting daemon 2024-04-06T09:36:33.381 DEBUG:teuthology.orchestra.run.smithi192:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.0 2024-04-06T09:36:33.384 INFO:tasks.rbd_mirror.cluster2.client.mirror.0:Started 2024-04-06T09:36:33.384 INFO:tasks.rbd_mirror_thrash:waiting for 115 secs before thrashing 2024-04-06T09:36:33.505 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ error_code=124 2024-04-06T09:36:33.505 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set -e 2024-04-06T09:36:33.506 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 124 -eq 124 ']' 2024-04-06T09:36:33.506 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:36:33.506 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+replaying 2024-04-06T09:36:33.506 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:36:33.506 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:36:33.506 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:36:33.506 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+replaying 2024-04-06T09:36:33.506 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local description_pattern= 2024-04-06T09:36:33.506 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local service_pattern= 2024-04-06T09:36:33.506 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:36:33.506 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 1 2024-04-06T09:36:34.507 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test_status_in_pool_dir cluster1 mirror test up+replaying '' '' 2024-04-06T09:36:34.507 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:36:34.507 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:36:34.507 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:36:34.507 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+replaying 2024-04-06T09:36:34.507 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local description_pattern= 2024-04-06T09:36:34.507 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local service_pattern= 2024-04-06T09:36:34.507 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local status 2024-04-06T09:36:34.508 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ CEPH_ARGS= 2024-04-06T09:36:34.508 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster1 mirror image status mirror/test 2024-04-06T09:36:34.592 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ status='test: 2024-04-06T09:36:34.592 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: global_id: 8ad5dedc-b3e0-425d-8993-b3e2231fc5ca 2024-04-06T09:36:34.592 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: state: up+replaying 2024-04-06T09:36:34.592 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: description: replaying, {"bytes_per_second":2656367.7,"entries_behind_primary":0,"entries_per_second":167.67,"non_primary_position":{"entry_tid":1,"object_number":45,"tag_tid":6},"primary_position":{"entry_tid":1,"object_number":45,"tag_tid":6}} 2024-04-06T09:36:34.592 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: service: mirror.0 on smithi192 2024-04-06T09:36:34.592 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: last_update: 2024-04-06 09:36:27 2024-04-06T09:36:34.593 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: peer_sites: 2024-04-06T09:36:34.593 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: name: cluster2 2024-04-06T09:36:34.593 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: state: up+stopped 2024-04-06T09:36:34.593 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: description: local image is primary 2024-04-06T09:36:34.593 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: last_update: 2024-04-06 09:36:34' 2024-04-06T09:36:34.593 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^ state: .*up+replaying' 2024-04-06T09:36:34.593 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: state: up+replaying 2024-04-06T09:36:34.594 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^ description: .*' 2024-04-06T09:36:34.595 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: description: replaying, {"bytes_per_second":2656367.7,"entries_behind_primary":0,"entries_per_second":167.67,"non_primary_position":{"entry_tid":1,"object_number":45,"tag_tid":6},"primary_position":{"entry_tid":1,"object_number":45,"tag_tid":6}} 2024-04-06T09:36:34.595 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n '' ']' 2024-04-06T09:36:34.595 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ echo up+replaying 2024-04-06T09:36:34.595 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^up+' 2024-04-06T09:36:34.596 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:up+replaying 2024-04-06T09:36:34.596 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep 'service: *mirror..* on ' 2024-04-06T09:36:34.597 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: service: mirror.0 on smithi192 2024-04-06T09:36:34.597 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local last_update 2024-04-06T09:36:34.598 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-04-06T09:36:34.600 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_update='2024-04-06 09:36:27' 2024-04-06T09:36:34.600 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+replaying '2024-04-06 09:36:27' 2024-04-06T09:36:34.600 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:36:34.600 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:36:34.600 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:36:34.600 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+replaying 2024-04-06T09:36:34.600 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'prev_last_update=2024-04-06 09:36:27' 2024-04-06T09:36:34.601 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local status 2024-04-06T09:36:34.601 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ CEPH_ARGS= 2024-04-06T09:36:34.601 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-04-06T09:36:34.684 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ status='OKOKOK142094268mirror.1smithi19218.2.2-1059-g4d05f4cafalseOK42164269mirror.2smithi19218.2.2-1059-g4d05f4cafalseOK42584261mirror.0smithi19218.2.2-1059-g4d05f4catrueOK43194322mirror.3smithi19218.2.2-1059-g4d05f4cafalseOKtest8ad5dedc-b3e0-425d-8993-b3e2231fc5caup+replayingreplaying, {"bytes_per_second":2656367.7,"entries_behind_primary":0,"entries_per_second":167.67,"non_primary_position":{"entry_tid":1,"object_number":45,"tag_tid":6},"primary_position":{"entry_tid":1,"object_number":45,"tag_tid":6}}42584261mirror.0smithi1922024-04-06 09:36:27cluster2c5e3bdc3-c286-4289-8fd0-6b8628b92376up+stoppedlocal image is primary2024-04-06 09:36:34' 2024-04-06T09:36:34.685 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local last_update state 2024-04-06T09:36:34.685 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2024-04-06T09:36:34.687 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_update='2024-04-06 09:36:27' 2024-04-06T09:36:34.687 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2024-04-06T09:36:34.690 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ state=up+replaying 2024-04-06T09:36:34.690 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ echo up+replaying 2024-04-06T09:36:34.690 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep up+replaying 2024-04-06T09:36:34.691 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:up+replaying 2024-04-06T09:36:34.692 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:36:34.692 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:36:34.692 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ snap_name=snap3 2024-04-06T09:36:34.692 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ create_snap cluster2 mirror test snap3 2024-04-06T09:36:34.692 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-04-06T09:36:34.692 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:36:34.692 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:36:34.692 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local snap_name=snap3 2024-04-06T09:36:34.692 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap3 --debug-rbd=20 --debug-journaler=20 2024-04-06T09:36:57.381 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.0 with skip (0.34511797126032295) > weight (0.25) 2024-04-06T09:36:57.381 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.1 with skip (0.7532968847156589) > weight (0.25) 2024-04-06T09:36:57.381 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.2 with skip (0.6961464044020822) > weight (0.25) 2024-04-06T09:36:57.381 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.3 with skip (0.944481071107694) > weight (0.25) 2024-04-06T09:36:57.381 INFO:tasks.rbd_mirror_thrash:waiting for 88 secs before thrashing 2024-04-06T09:37:02.583 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_image_replay_started cluster1 mirror test 2024-04-06T09:37:02.583 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:37:02.583 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:37:02.583 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:37:02.583 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_image_replay_state cluster1 mirror test started 2024-04-06T09:37:02.583 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:37:02.583 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:37:02.583 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:37:02.584 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state=started 2024-04-06T09:37:02.584 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local s 2024-04-06T09:37:02.584 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:37:02.584 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 1 2024-04-06T09:37:03.585 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test_image_replay_state cluster1 mirror test started 2024-04-06T09:37:03.585 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:37:03.585 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:37:03.585 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:37:03.585 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local test_state=started 2024-04-06T09:37:03.585 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local status_result 2024-04-06T09:37:03.585 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local current_state=stopped 2024-04-06T09:37:03.586 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2024-04-06T09:37:03.586 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -i state 2024-04-06T09:37:03.586 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_instance=cluster1 2024-04-06T09:37:03.586 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ shift 2024-04-06T09:37:03.586 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:37:03.587 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance=cluster1 2024-04-06T09:37:03.587 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local loop_instance 2024-04-06T09:37:03.587 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:37:03.587 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sleep 0 2024-04-06T09:37:03.588 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:37:03.588 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ seq 0 3 2024-04-06T09:37:03.593 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:37:03.593 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2024-04-06T09:37:03.593 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1:0 2024-04-06T09:37:03.593 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ shift 2024-04-06T09:37:03.593 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:37:03.593 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:37:03.593 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:37:03.593 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=cluster 2024-04-06T09:37:03.594 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:37:03.594 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:37:03.594 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:37:03.594 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:37:03.594 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval cluster=cluster1 2024-04-06T09:37:03.594 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ cluster=cluster1 2024-04-06T09:37:03.594 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:37:03.595 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:37:03.595 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:37:03.595 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local local_cluster=cluster1:0 2024-04-06T09:37:03.595 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local cluster=cluster1 2024-04-06T09:37:03.595 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local instance 2024-04-06T09:37:03.595 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:37:03.595 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local val=cluster1:0 2024-04-06T09:37:03.595 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local cluster_var_name=local_cluster 2024-04-06T09:37:03.595 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local instance_var_name=instance 2024-04-06T09:37:03.596 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ cluster=cluster1 2024-04-06T09:37:03.596 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:37:03.596 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ '[' 0 = cluster1:0 ']' 2024-04-06T09:37:03.596 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ eval local_cluster=cluster1 2024-04-06T09:37:03.596 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++++ local_cluster=cluster1 2024-04-06T09:37:03.596 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ eval instance=0 2024-04-06T09:37:03.596 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++++ instance=0 2024-04-06T09:37:03.596 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:37:03.613 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:37:03.614 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:37:03.614 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:37:03.614 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2024-04-06T09:37:03.736 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ return 0 2024-04-06T09:37:03.737 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ status_result=' "state": "Replaying"' 2024-04-06T09:37:03.737 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ echo ' "state": "Replaying"' 2024-04-06T09:37:03.737 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep -i Replaying 2024-04-06T09:37:03.738 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: "state": "Replaying" 2024-04-06T09:37:03.740 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ current_state=started 2024-04-06T09:37:03.740 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test started = started 2024-04-06T09:37:03.740 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:37:03.740 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror test 2024-04-06T09:37:03.740 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local local_cluster=cluster1 2024-04-06T09:37:03.740 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-04-06T09:37:03.740 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:37:03.740 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:37:03.740 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' journal = journal ']' 2024-04-06T09:37:03.740 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_journal_replay_complete cluster1 cluster2 mirror test 2024-04-06T09:37:03.741 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local local_cluster=cluster1 2024-04-06T09:37:03.741 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-04-06T09:37:03.741 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:37:03.741 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:37:03.741 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local s master_pos mirror_pos last_mirror_pos 2024-04-06T09:37:03.741 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local master_tag master_entry mirror_tag mirror_entry 2024-04-06T09:37:03.741 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:37:03.741 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:37:03.741 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:37:03.941 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:37:03.942 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:37:03.942 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:37:03.942 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:37:03.942 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:37:03.942 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:37:03.942 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:37:03.942 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:37:03.942 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:37:03.942 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:37:03.942 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:37:03.942 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:37:03.943 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:37:03.943 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:37:03.943 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:37:03.943 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:37:03.943 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:37:03.945 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:37:03.945 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:37:03.945 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:37:03.945 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:37:03.946 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:37:03.946 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:37:03.946 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:37:03.946 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:37:03.946 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:37:03.946 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:37:03.946 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:37:03.946 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:37:03.946 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:37:03.946 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:37:03.946 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:37:03.946 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:37:03.947 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:37:03.947 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:37:03.948 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:37:03.948 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:37:03.948 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:37:03.948 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:37:03.948 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:37:03.948 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:37:03.948 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:37:03.948 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:37:03.948 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:37:03.948 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:37:03.948 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:37:03.948 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:37:03.948 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:37:03.948 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:37:03.973 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:37:03.973 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:37:03.973 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:37:03.973 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:37:04.102 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:37:04.103 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:37:04.104 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:37:04.104 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:37:04.104 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:37:04.104 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:37:04.104 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:37:04.104 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:37:04.104 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:37:04.104 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:37:04.104 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:37:04.105 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:37:04.105 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:37:04.105 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:37:04.106 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:37:04.106 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:37:04.158 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 13 2024-04-06T09:37:04.158 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 14 2024-04-06T09:37:04.159 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:37:04.159 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=57, tag_tid=9, entry_tid=1], [object_number=56, tag_tid=9, entry_tid=0], [object_number=59, tag_tid=8, entry_tid=11939], [object_number=58, tag_tid=8, entry_tid=11938]]], state=connected] 2024-04-06T09:37:04.159 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=54, tag_tid=8, entry_tid=6310], [object_number=53, tag_tid=8, entry_tid=6309], [object_number=52, tag_tid=8, entry_tid=6308], [object_number=55, tag_tid=8, entry_tid=6307]]], state=connected] 2024-04-06T09:37:04.162 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:37:04.164 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=57, tag_tid=9, entry_tid=1' 2024-04-06T09:37:04.165 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:37:04.165 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:37:04.165 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:37:04.165 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:37:04.165 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:37:04.165 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:37:04.165 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:37:04.165 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:37:04.165 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:37:04.165 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:37:04.165 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:37:04.166 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:37:04.166 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:37:04.166 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:37:04.166 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:37:04.212 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 13 2024-04-06T09:37:04.212 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 14 2024-04-06T09:37:04.213 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:37:04.213 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=57, tag_tid=9, entry_tid=1], [object_number=56, tag_tid=9, entry_tid=0], [object_number=59, tag_tid=8, entry_tid=11939], [object_number=58, tag_tid=8, entry_tid=11938]]], state=connected] 2024-04-06T09:37:04.213 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=54, tag_tid=8, entry_tid=6310], [object_number=53, tag_tid=8, entry_tid=6309], [object_number=52, tag_tid=8, entry_tid=6308], [object_number=55, tag_tid=8, entry_tid=6307]]], state=connected] 2024-04-06T09:37:04.216 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:37:04.217 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=54, tag_tid=8, entry_tid=6310' 2024-04-06T09:37:04.218 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=57, tag_tid=9, entry_tid=1' -a 'object_number=57, tag_tid=9, entry_tid=1' = 'object_number=54, tag_tid=8, entry_tid=6310' 2024-04-06T09:37:04.218 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=54, tag_tid=8, entry_tid=6310' '!=' '' 2024-04-06T09:37:04.218 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:37:04.218 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=54, tag_tid=8, entry_tid=6310' = '' 2024-04-06T09:37:04.218 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=54, tag_tid=8, entry_tid=6310' 2024-04-06T09:37:04.219 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=57, tag_tid=9, entry_tid=1' 2024-04-06T09:37:04.219 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:37:04.219 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:37:04.219 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=9 2024-04-06T09:37:04.220 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=54, tag_tid=8, entry_tid=6310' 2024-04-06T09:37:04.220 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:37:04.220 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:37:04.221 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=8 2024-04-06T09:37:04.221 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=57, tag_tid=9, entry_tid=1' 2024-04-06T09:37:04.222 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:37:04.222 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:37:04.222 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:37:04.223 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=54, tag_tid=8, entry_tid=6310' 2024-04-06T09:37:04.223 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:37:04.223 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:37:04.225 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=6310 2024-04-06T09:37:04.225 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 9 = 8 -a 1 -le 6310 2024-04-06T09:37:04.225 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:37:04.225 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:37:04.225 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:37:04.426 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:37:04.427 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:37:04.427 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:37:04.427 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:37:04.427 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:37:04.427 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:37:04.427 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:37:04.427 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:37:04.427 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:37:04.427 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:37:04.427 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:37:04.427 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:37:04.427 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:37:04.427 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:37:04.428 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:37:04.428 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:37:04.428 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:37:04.430 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:37:04.430 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:37:04.430 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:37:04.430 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:37:04.430 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:37:04.430 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:37:04.430 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:37:04.430 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:37:04.430 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:37:04.430 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:37:04.430 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:37:04.430 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:37:04.431 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:37:04.431 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:37:04.431 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:37:04.431 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:37:04.432 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:37:04.432 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:37:04.432 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:37:04.432 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:37:04.432 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:37:04.432 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:37:04.432 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:37:04.433 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:37:04.433 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:37:04.433 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:37:04.433 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:37:04.433 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:37:04.433 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:37:04.433 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:37:04.433 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:37:04.433 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:37:04.457 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:37:04.458 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:37:04.458 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:37:04.458 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:37:04.585 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:37:04.586 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:37:04.586 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:37:04.586 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:37:04.586 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:37:04.586 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:37:04.586 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:37:04.587 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:37:04.587 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:37:04.587 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:37:04.587 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:37:04.587 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:37:04.587 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:37:04.588 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:37:04.588 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:37:04.588 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:37:04.639 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 13 2024-04-06T09:37:04.639 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 14 2024-04-06T09:37:04.640 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:37:04.640 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=57, tag_tid=9, entry_tid=1], [object_number=56, tag_tid=9, entry_tid=0], [object_number=59, tag_tid=8, entry_tid=11939], [object_number=58, tag_tid=8, entry_tid=11938]]], state=connected] 2024-04-06T09:37:04.640 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=53, tag_tid=8, entry_tid=6397], [object_number=52, tag_tid=8, entry_tid=6396], [object_number=55, tag_tid=8, entry_tid=6395], [object_number=54, tag_tid=8, entry_tid=6394]]], state=connected] 2024-04-06T09:37:04.643 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:37:04.644 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=57, tag_tid=9, entry_tid=1' 2024-04-06T09:37:04.646 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:37:04.646 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:37:04.646 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:37:04.646 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:37:04.646 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:37:04.646 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:37:04.646 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:37:04.646 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:37:04.646 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:37:04.646 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:37:04.646 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:37:04.646 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:37:04.647 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:37:04.647 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:37:04.647 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:37:04.692 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 13 2024-04-06T09:37:04.692 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 14 2024-04-06T09:37:04.693 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:37:04.693 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=57, tag_tid=9, entry_tid=1], [object_number=56, tag_tid=9, entry_tid=0], [object_number=59, tag_tid=8, entry_tid=11939], [object_number=58, tag_tid=8, entry_tid=11938]]], state=connected] 2024-04-06T09:37:04.693 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=53, tag_tid=8, entry_tid=6397], [object_number=52, tag_tid=8, entry_tid=6396], [object_number=55, tag_tid=8, entry_tid=6395], [object_number=54, tag_tid=8, entry_tid=6394]]], state=connected] 2024-04-06T09:37:04.697 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:37:04.699 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=53, tag_tid=8, entry_tid=6397' 2024-04-06T09:37:04.699 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=57, tag_tid=9, entry_tid=1' -a 'object_number=57, tag_tid=9, entry_tid=1' = 'object_number=53, tag_tid=8, entry_tid=6397' 2024-04-06T09:37:04.699 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=53, tag_tid=8, entry_tid=6397' '!=' 'object_number=54, tag_tid=8, entry_tid=6310' 2024-04-06T09:37:04.699 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:37:04.699 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=53, tag_tid=8, entry_tid=6397' = 'object_number=54, tag_tid=8, entry_tid=6310' 2024-04-06T09:37:04.699 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=53, tag_tid=8, entry_tid=6397' 2024-04-06T09:37:04.699 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=57, tag_tid=9, entry_tid=1' 2024-04-06T09:37:04.699 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:37:04.699 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:37:04.700 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=9 2024-04-06T09:37:04.700 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=53, tag_tid=8, entry_tid=6397' 2024-04-06T09:37:04.700 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:37:04.701 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:37:04.701 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=8 2024-04-06T09:37:04.702 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=57, tag_tid=9, entry_tid=1' 2024-04-06T09:37:04.702 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:37:04.702 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:37:04.703 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:37:04.704 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=53, tag_tid=8, entry_tid=6397' 2024-04-06T09:37:04.704 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:37:04.704 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:37:04.705 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=6397 2024-04-06T09:37:04.705 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 9 = 8 -a 1 -le 6397 2024-04-06T09:37:04.705 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:37:04.705 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:37:04.705 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:37:04.907 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:37:04.907 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:37:04.907 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:37:04.907 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:37:04.907 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:37:04.907 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:37:04.907 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:37:04.908 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:37:04.908 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:37:04.908 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:37:04.908 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:37:04.908 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:37:04.908 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:37:04.908 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:37:04.908 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:37:04.909 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:37:04.909 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:37:04.912 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:37:04.912 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:37:04.912 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:37:04.913 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:37:04.913 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:37:04.913 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:37:04.913 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:37:04.913 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:37:04.913 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:37:04.913 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:37:04.913 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:37:04.913 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:37:04.913 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:37:04.913 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:37:04.913 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:37:04.913 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:37:04.913 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:37:04.913 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:37:04.914 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:37:04.914 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:37:04.914 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:37:04.914 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:37:04.914 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:37:04.914 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:37:04.914 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:37:04.914 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:37:04.914 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:37:04.914 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:37:04.914 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:37:04.914 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:37:04.914 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:37:04.914 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:37:04.937 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:37:04.937 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:37:04.937 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:37:04.938 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:37:05.064 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:37:05.065 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:37:05.065 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:37:05.065 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:37:05.065 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:37:05.065 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:37:05.065 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:37:05.065 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:37:05.066 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:37:05.066 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:37:05.066 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:37:05.066 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:37:05.066 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:37:05.067 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:37:05.068 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:37:05.068 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:37:05.121 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 13 2024-04-06T09:37:05.122 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 14 2024-04-06T09:37:05.122 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:37:05.122 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=57, tag_tid=9, entry_tid=1], [object_number=56, tag_tid=9, entry_tid=0], [object_number=59, tag_tid=8, entry_tid=11939], [object_number=58, tag_tid=8, entry_tid=11938]]], state=connected] 2024-04-06T09:37:05.122 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=52, tag_tid=8, entry_tid=6480], [object_number=55, tag_tid=8, entry_tid=6479], [object_number=54, tag_tid=8, entry_tid=6478], [object_number=53, tag_tid=8, entry_tid=6477]]], state=connected] 2024-04-06T09:37:05.125 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:37:05.126 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=57, tag_tid=9, entry_tid=1' 2024-04-06T09:37:05.128 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:37:05.128 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:37:05.128 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:37:05.128 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:37:05.128 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:37:05.128 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:37:05.128 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:37:05.128 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:37:05.128 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:37:05.128 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:37:05.128 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:37:05.128 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:37:05.129 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:37:05.129 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:37:05.129 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:37:05.176 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 13 2024-04-06T09:37:05.176 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 14 2024-04-06T09:37:05.176 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:37:05.176 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=57, tag_tid=9, entry_tid=1], [object_number=56, tag_tid=9, entry_tid=0], [object_number=59, tag_tid=8, entry_tid=11939], [object_number=58, tag_tid=8, entry_tid=11938]]], state=connected] 2024-04-06T09:37:05.176 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=52, tag_tid=8, entry_tid=6480], [object_number=55, tag_tid=8, entry_tid=6479], [object_number=54, tag_tid=8, entry_tid=6478], [object_number=53, tag_tid=8, entry_tid=6477]]], state=connected] 2024-04-06T09:37:05.179 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:37:05.181 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=52, tag_tid=8, entry_tid=6480' 2024-04-06T09:37:05.181 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=57, tag_tid=9, entry_tid=1' -a 'object_number=57, tag_tid=9, entry_tid=1' = 'object_number=52, tag_tid=8, entry_tid=6480' 2024-04-06T09:37:05.182 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=52, tag_tid=8, entry_tid=6480' '!=' 'object_number=53, tag_tid=8, entry_tid=6397' 2024-04-06T09:37:05.182 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:37:05.182 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=52, tag_tid=8, entry_tid=6480' = 'object_number=53, tag_tid=8, entry_tid=6397' 2024-04-06T09:37:05.182 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=52, tag_tid=8, entry_tid=6480' 2024-04-06T09:37:05.182 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=57, tag_tid=9, entry_tid=1' 2024-04-06T09:37:05.182 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:37:05.182 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:37:05.182 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=9 2024-04-06T09:37:05.184 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=52, tag_tid=8, entry_tid=6480' 2024-04-06T09:37:05.184 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:37:05.184 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:37:05.184 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=8 2024-04-06T09:37:05.185 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=57, tag_tid=9, entry_tid=1' 2024-04-06T09:37:05.185 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:37:05.185 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:37:05.186 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:37:05.187 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=52, tag_tid=8, entry_tid=6480' 2024-04-06T09:37:05.187 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:37:05.187 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:37:05.188 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=6480 2024-04-06T09:37:05.188 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 9 = 8 -a 1 -le 6480 2024-04-06T09:37:05.188 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:37:05.188 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:37:05.188 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:37:05.390 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:37:05.390 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:37:05.390 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:37:05.390 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:37:05.390 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:37:05.390 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:37:05.390 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:37:05.390 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:37:05.391 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:37:05.391 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:37:05.391 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:37:05.391 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:37:05.391 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:37:05.391 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:37:05.391 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:37:05.391 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:37:05.392 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:37:05.394 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:37:05.394 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:37:05.394 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:37:05.394 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:37:05.395 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:37:05.395 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:37:05.395 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:37:05.395 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:37:05.395 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:37:05.395 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:37:05.395 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:37:05.395 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:37:05.395 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:37:05.395 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:37:05.395 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:37:05.395 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:37:05.396 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:37:05.397 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:37:05.397 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:37:05.397 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:37:05.397 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:37:05.397 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:37:05.397 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:37:05.397 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:37:05.397 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:37:05.397 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:37:05.397 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:37:05.397 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:37:05.397 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:37:05.397 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:37:05.397 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:37:05.398 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:37:05.421 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:37:05.422 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:37:05.422 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:37:05.422 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:37:05.549 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:37:05.550 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:37:05.550 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:37:05.551 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:37:05.551 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:37:05.551 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:37:05.551 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:37:05.551 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:37:05.551 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:37:05.551 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:37:05.552 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:37:05.552 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:37:05.552 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:37:05.554 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:37:05.554 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:37:05.555 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:37:05.607 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 13 2024-04-06T09:37:05.607 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 14 2024-04-06T09:37:05.607 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:37:05.607 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=57, tag_tid=9, entry_tid=1], [object_number=56, tag_tid=9, entry_tid=0], [object_number=59, tag_tid=8, entry_tid=11939], [object_number=58, tag_tid=8, entry_tid=11938]]], state=connected] 2024-04-06T09:37:05.607 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=55, tag_tid=8, entry_tid=6555], [object_number=54, tag_tid=8, entry_tid=6554], [object_number=53, tag_tid=8, entry_tid=6553], [object_number=52, tag_tid=8, entry_tid=6552]]], state=connected] 2024-04-06T09:37:05.610 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:37:05.612 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=57, tag_tid=9, entry_tid=1' 2024-04-06T09:37:05.613 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:37:05.613 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:37:05.613 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:37:05.613 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:37:05.613 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:37:05.613 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:37:05.613 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:37:05.613 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:37:05.613 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:37:05.613 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:37:05.614 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:37:05.614 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:37:05.614 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:37:05.614 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:37:05.614 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:37:05.660 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 13 2024-04-06T09:37:05.660 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 14 2024-04-06T09:37:05.660 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:37:05.660 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=57, tag_tid=9, entry_tid=1], [object_number=56, tag_tid=9, entry_tid=0], [object_number=59, tag_tid=8, entry_tid=11939], [object_number=58, tag_tid=8, entry_tid=11938]]], state=connected] 2024-04-06T09:37:05.660 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=55, tag_tid=8, entry_tid=6555], [object_number=54, tag_tid=8, entry_tid=6554], [object_number=53, tag_tid=8, entry_tid=6553], [object_number=52, tag_tid=8, entry_tid=6552]]], state=connected] 2024-04-06T09:37:05.664 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:37:05.666 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=55, tag_tid=8, entry_tid=6555' 2024-04-06T09:37:05.666 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=57, tag_tid=9, entry_tid=1' -a 'object_number=57, tag_tid=9, entry_tid=1' = 'object_number=55, tag_tid=8, entry_tid=6555' 2024-04-06T09:37:05.666 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=55, tag_tid=8, entry_tid=6555' '!=' 'object_number=52, tag_tid=8, entry_tid=6480' 2024-04-06T09:37:05.666 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:37:05.666 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=55, tag_tid=8, entry_tid=6555' = 'object_number=52, tag_tid=8, entry_tid=6480' 2024-04-06T09:37:05.666 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=55, tag_tid=8, entry_tid=6555' 2024-04-06T09:37:05.666 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=57, tag_tid=9, entry_tid=1' 2024-04-06T09:37:05.667 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:37:05.667 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:37:05.667 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=9 2024-04-06T09:37:05.668 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=55, tag_tid=8, entry_tid=6555' 2024-04-06T09:37:05.668 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:37:05.668 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:37:05.669 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=8 2024-04-06T09:37:05.670 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=57, tag_tid=9, entry_tid=1' 2024-04-06T09:37:05.670 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:37:05.670 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:37:05.671 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:37:05.671 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=55, tag_tid=8, entry_tid=6555' 2024-04-06T09:37:05.671 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:37:05.671 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:37:05.673 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=6555 2024-04-06T09:37:05.673 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 9 = 8 -a 1 -le 6555 2024-04-06T09:37:05.673 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:37:05.673 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:37:05.673 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:37:05.874 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:37:05.874 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:37:05.874 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:37:05.875 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:37:05.875 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:37:05.875 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:37:05.875 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:37:05.875 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:37:05.875 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:37:05.875 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:37:05.875 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:37:05.875 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:37:05.875 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:37:05.875 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:37:05.875 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:37:05.876 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:37:05.876 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:37:05.879 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:37:05.879 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:37:05.879 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:37:05.879 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:37:05.879 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:37:05.879 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:37:05.880 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:37:05.880 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:37:05.880 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:37:05.880 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:37:05.880 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:37:05.880 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:37:05.880 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:37:05.880 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:37:05.880 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:37:05.880 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:37:05.880 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:37:05.880 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:37:05.880 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:37:05.880 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:37:05.880 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:37:05.881 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:37:05.881 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:37:05.881 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:37:05.881 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:37:05.881 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:37:05.881 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:37:05.881 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:37:05.881 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:37:05.881 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:37:05.881 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:37:05.881 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:37:05.905 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:37:05.905 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:37:05.906 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:37:05.906 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:37:06.032 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:37:06.033 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:37:06.033 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:37:06.033 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:37:06.033 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:37:06.033 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:37:06.033 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:37:06.034 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:37:06.034 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:37:06.034 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:37:06.034 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:37:06.034 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:37:06.034 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:37:06.036 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:37:06.037 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:37:06.037 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:37:06.082 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 13 2024-04-06T09:37:06.083 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 14 2024-04-06T09:37:06.083 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:37:06.083 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=57, tag_tid=9, entry_tid=1], [object_number=56, tag_tid=9, entry_tid=0], [object_number=59, tag_tid=8, entry_tid=11939], [object_number=58, tag_tid=8, entry_tid=11938]]], state=connected] 2024-04-06T09:37:06.083 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=55, tag_tid=8, entry_tid=6627], [object_number=54, tag_tid=8, entry_tid=6626], [object_number=53, tag_tid=8, entry_tid=6625], [object_number=52, tag_tid=8, entry_tid=6624]]], state=connected] 2024-04-06T09:37:06.086 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:37:06.089 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=57, tag_tid=9, entry_tid=1' 2024-04-06T09:37:06.089 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:37:06.089 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:37:06.089 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:37:06.089 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:37:06.089 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:37:06.089 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:37:06.089 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:37:06.089 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:37:06.090 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:37:06.090 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:37:06.090 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:37:06.090 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:37:06.091 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:37:06.091 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:37:06.091 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:37:06.135 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 13 2024-04-06T09:37:06.135 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 14 2024-04-06T09:37:06.135 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:37:06.135 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=57, tag_tid=9, entry_tid=1], [object_number=56, tag_tid=9, entry_tid=0], [object_number=59, tag_tid=8, entry_tid=11939], [object_number=58, tag_tid=8, entry_tid=11938]]], state=connected] 2024-04-06T09:37:06.135 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=55, tag_tid=8, entry_tid=6627], [object_number=54, tag_tid=8, entry_tid=6626], [object_number=53, tag_tid=8, entry_tid=6625], [object_number=52, tag_tid=8, entry_tid=6624]]], state=connected] 2024-04-06T09:37:06.139 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:37:06.141 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=55, tag_tid=8, entry_tid=6627' 2024-04-06T09:37:06.141 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=57, tag_tid=9, entry_tid=1' -a 'object_number=57, tag_tid=9, entry_tid=1' = 'object_number=55, tag_tid=8, entry_tid=6627' 2024-04-06T09:37:06.141 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=55, tag_tid=8, entry_tid=6627' '!=' 'object_number=55, tag_tid=8, entry_tid=6555' 2024-04-06T09:37:06.141 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:37:06.141 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=55, tag_tid=8, entry_tid=6627' = 'object_number=55, tag_tid=8, entry_tid=6555' 2024-04-06T09:37:06.141 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=55, tag_tid=8, entry_tid=6627' 2024-04-06T09:37:06.141 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=57, tag_tid=9, entry_tid=1' 2024-04-06T09:37:06.141 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:37:06.141 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:37:06.142 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=9 2024-04-06T09:37:06.143 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=55, tag_tid=8, entry_tid=6627' 2024-04-06T09:37:06.143 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:37:06.143 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:37:06.144 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=8 2024-04-06T09:37:06.144 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=57, tag_tid=9, entry_tid=1' 2024-04-06T09:37:06.144 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:37:06.144 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:37:06.145 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:37:06.146 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=55, tag_tid=8, entry_tid=6627' 2024-04-06T09:37:06.146 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:37:06.146 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:37:06.148 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=6627 2024-04-06T09:37:06.148 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 9 = 8 -a 1 -le 6627 2024-04-06T09:37:06.148 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:37:06.148 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:37:06.148 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:37:06.350 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:37:06.442 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:37:06.442 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:37:06.442 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:37:06.442 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:37:06.442 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:37:06.442 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:37:06.442 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:37:06.442 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:37:06.442 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:37:06.443 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:37:06.443 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:37:06.443 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:37:06.443 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:37:06.443 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:37:06.446 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:37:06.446 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:37:06.446 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:37:06.446 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:37:06.447 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:37:06.447 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:37:06.447 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:37:06.447 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:37:06.447 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:37:06.447 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:37:06.447 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:37:06.447 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:37:06.447 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:37:06.447 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:37:06.447 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:37:06.447 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:37:06.447 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:37:06.447 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:37:06.447 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:37:06.448 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:37:06.448 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:37:06.448 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:37:06.448 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:37:06.448 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:37:06.448 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:37:06.448 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:37:06.448 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:37:06.448 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:37:06.448 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:37:06.448 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:37:06.448 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:37:06.448 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:37:06.448 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:37:06.449 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:37:06.449 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:37:06.449 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:37:06.449 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:37:06.449 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:37:06.510 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:37:06.511 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:37:06.511 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:37:06.511 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:37:06.511 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:37:06.511 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:37:06.511 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:37:06.511 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:37:06.511 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:37:06.512 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:37:06.512 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:37:06.512 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:37:06.512 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:37:06.514 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:37:06.515 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:37:06.515 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:37:06.567 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 13 2024-04-06T09:37:06.568 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 14 2024-04-06T09:37:06.568 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:37:06.568 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=57, tag_tid=9, entry_tid=1], [object_number=56, tag_tid=9, entry_tid=0], [object_number=59, tag_tid=8, entry_tid=11939], [object_number=58, tag_tid=8, entry_tid=11938]]], state=connected] 2024-04-06T09:37:06.568 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=52, tag_tid=8, entry_tid=6716], [object_number=55, tag_tid=8, entry_tid=6715], [object_number=54, tag_tid=8, entry_tid=6714], [object_number=53, tag_tid=8, entry_tid=6713]]], state=connected] 2024-04-06T09:37:06.571 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:37:06.572 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=57, tag_tid=9, entry_tid=1' 2024-04-06T09:37:06.573 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:37:06.573 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:37:06.573 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:37:06.574 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:37:06.574 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:37:06.574 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:37:06.574 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:37:06.574 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:37:06.574 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:37:06.574 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:37:06.574 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:37:06.574 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:37:06.575 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:37:06.575 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:37:06.575 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:37:06.620 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 13 2024-04-06T09:37:06.621 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 14 2024-04-06T09:37:06.621 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:37:06.621 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=57, tag_tid=9, entry_tid=1], [object_number=56, tag_tid=9, entry_tid=0], [object_number=59, tag_tid=8, entry_tid=11939], [object_number=58, tag_tid=8, entry_tid=11938]]], state=connected] 2024-04-06T09:37:06.621 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=52, tag_tid=8, entry_tid=6716], [object_number=55, tag_tid=8, entry_tid=6715], [object_number=54, tag_tid=8, entry_tid=6714], [object_number=53, tag_tid=8, entry_tid=6713]]], state=connected] 2024-04-06T09:37:06.624 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:37:06.626 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=52, tag_tid=8, entry_tid=6716' 2024-04-06T09:37:06.626 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=57, tag_tid=9, entry_tid=1' -a 'object_number=57, tag_tid=9, entry_tid=1' = 'object_number=52, tag_tid=8, entry_tid=6716' 2024-04-06T09:37:06.626 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=52, tag_tid=8, entry_tid=6716' '!=' 'object_number=55, tag_tid=8, entry_tid=6627' 2024-04-06T09:37:06.626 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:37:06.626 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=52, tag_tid=8, entry_tid=6716' = 'object_number=55, tag_tid=8, entry_tid=6627' 2024-04-06T09:37:06.626 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=52, tag_tid=8, entry_tid=6716' 2024-04-06T09:37:06.627 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=57, tag_tid=9, entry_tid=1' 2024-04-06T09:37:06.627 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:37:06.627 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:37:06.627 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=9 2024-04-06T09:37:06.628 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=52, tag_tid=8, entry_tid=6716' 2024-04-06T09:37:06.628 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:37:06.628 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:37:06.629 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=8 2024-04-06T09:37:06.630 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=57, tag_tid=9, entry_tid=1' 2024-04-06T09:37:06.630 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:37:06.630 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:37:06.631 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:37:06.631 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=52, tag_tid=8, entry_tid=6716' 2024-04-06T09:37:06.631 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:37:06.631 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:37:06.633 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=6716 2024-04-06T09:37:06.633 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 9 = 8 -a 1 -le 6716 2024-04-06T09:37:06.633 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:37:06.633 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:37:06.633 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:37:06.836 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:37:06.836 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:37:06.836 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:37:06.836 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:37:06.836 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:37:06.836 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:37:06.837 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:37:06.837 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:37:06.837 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:37:06.837 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:37:06.837 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:37:06.837 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:37:06.838 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:37:06.838 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:37:06.838 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:37:06.842 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:37:06.842 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:37:06.843 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:37:06.843 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:37:06.843 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:37:06.843 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:37:06.843 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:37:06.843 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:37:06.843 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:37:06.844 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:37:06.844 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:37:06.844 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:37:06.844 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:37:06.844 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:37:06.844 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:37:06.845 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:37:06.845 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:37:06.845 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:37:06.845 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:37:06.845 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:37:06.845 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:37:06.845 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:37:06.846 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:37:06.846 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:37:06.846 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:37:06.846 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:37:06.846 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:37:06.847 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:37:06.847 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:37:06.847 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:37:06.847 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:37:06.847 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:37:06.847 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:37:06.847 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:37:06.866 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:37:06.867 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:37:06.867 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:37:06.867 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:37:06.993 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:37:06.995 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:37:06.995 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:37:06.995 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:37:06.995 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:37:06.995 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:37:06.995 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:37:06.995 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:37:06.995 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:37:06.995 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:37:06.995 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:37:06.996 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:37:06.996 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:37:06.998 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:37:06.998 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:37:06.998 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:37:07.043 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 13 2024-04-06T09:37:07.044 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 14 2024-04-06T09:37:07.044 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:37:07.044 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=57, tag_tid=9, entry_tid=1], [object_number=56, tag_tid=9, entry_tid=0], [object_number=59, tag_tid=8, entry_tid=11939], [object_number=58, tag_tid=8, entry_tid=11938]]], state=connected] 2024-04-06T09:37:07.044 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=53, tag_tid=8, entry_tid=6793], [object_number=52, tag_tid=8, entry_tid=6792], [object_number=55, tag_tid=8, entry_tid=6791], [object_number=54, tag_tid=8, entry_tid=6790]]], state=connected] 2024-04-06T09:37:07.047 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:37:07.048 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=57, tag_tid=9, entry_tid=1' 2024-04-06T09:37:07.049 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:37:07.049 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:37:07.050 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:37:07.050 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:37:07.050 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:37:07.050 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:37:07.050 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:37:07.050 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:37:07.050 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:37:07.050 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:37:07.050 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:37:07.050 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:37:07.051 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:37:07.051 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:37:07.051 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:37:07.096 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 13 2024-04-06T09:37:07.096 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 14 2024-04-06T09:37:07.097 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:37:07.097 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=57, tag_tid=9, entry_tid=1], [object_number=56, tag_tid=9, entry_tid=0], [object_number=59, tag_tid=8, entry_tid=11939], [object_number=58, tag_tid=8, entry_tid=11938]]], state=connected] 2024-04-06T09:37:07.097 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=53, tag_tid=8, entry_tid=6793], [object_number=52, tag_tid=8, entry_tid=6792], [object_number=55, tag_tid=8, entry_tid=6791], [object_number=54, tag_tid=8, entry_tid=6790]]], state=connected] 2024-04-06T09:37:07.099 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:37:07.102 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=53, tag_tid=8, entry_tid=6793' 2024-04-06T09:37:07.102 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=57, tag_tid=9, entry_tid=1' -a 'object_number=57, tag_tid=9, entry_tid=1' = 'object_number=53, tag_tid=8, entry_tid=6793' 2024-04-06T09:37:07.102 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=53, tag_tid=8, entry_tid=6793' '!=' 'object_number=52, tag_tid=8, entry_tid=6716' 2024-04-06T09:37:07.102 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:37:07.102 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=53, tag_tid=8, entry_tid=6793' = 'object_number=52, tag_tid=8, entry_tid=6716' 2024-04-06T09:37:07.102 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=53, tag_tid=8, entry_tid=6793' 2024-04-06T09:37:07.102 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=57, tag_tid=9, entry_tid=1' 2024-04-06T09:37:07.102 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:37:07.102 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:37:07.103 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=9 2024-04-06T09:37:07.103 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=53, tag_tid=8, entry_tid=6793' 2024-04-06T09:37:07.103 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:37:07.103 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:37:07.104 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=8 2024-04-06T09:37:07.105 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=57, tag_tid=9, entry_tid=1' 2024-04-06T09:37:07.105 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:37:07.105 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:37:07.106 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:37:07.107 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=53, tag_tid=8, entry_tid=6793' 2024-04-06T09:37:07.107 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:37:07.107 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:37:07.108 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=6793 2024-04-06T09:37:07.108 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 9 = 8 -a 1 -le 6793 2024-04-06T09:37:07.108 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:37:07.109 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:37:07.109 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:37:07.310 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:37:07.311 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:37:07.311 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:37:07.311 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:37:07.311 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:37:07.311 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:37:07.311 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:37:07.311 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:37:07.311 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:37:07.311 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:37:07.311 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:37:07.311 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:37:07.311 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:37:07.311 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:37:07.311 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:37:07.311 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:37:07.312 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:37:07.314 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:37:07.314 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:37:07.315 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:37:07.315 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:37:07.315 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:37:07.315 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:37:07.315 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:37:07.315 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:37:07.315 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:37:07.315 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:37:07.315 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:37:07.315 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:37:07.315 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:37:07.315 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:37:07.315 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:37:07.315 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:37:07.315 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:37:07.316 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:37:07.316 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:37:07.316 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:37:07.316 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:37:07.316 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:37:07.316 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:37:07.316 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:37:07.316 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:37:07.316 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:37:07.316 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:37:07.316 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:37:07.316 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:37:07.316 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:37:07.316 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:37:07.316 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:37:07.340 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:37:07.340 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:37:07.340 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:37:07.340 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:37:07.470 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:37:07.471 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:37:07.472 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:37:07.472 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:37:07.472 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:37:07.472 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:37:07.472 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:37:07.472 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:37:07.472 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:37:07.472 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:37:07.473 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:37:07.473 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:37:07.473 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:37:07.474 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:37:07.475 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:37:07.475 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:37:07.520 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 13 2024-04-06T09:37:07.520 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 14 2024-04-06T09:37:07.520 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:37:07.520 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=57, tag_tid=9, entry_tid=1], [object_number=56, tag_tid=9, entry_tid=0], [object_number=59, tag_tid=8, entry_tid=11939], [object_number=58, tag_tid=8, entry_tid=11938]]], state=connected] 2024-04-06T09:37:07.520 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=55, tag_tid=8, entry_tid=6883], [object_number=54, tag_tid=8, entry_tid=6882], [object_number=53, tag_tid=8, entry_tid=6881], [object_number=52, tag_tid=8, entry_tid=6880]]], state=connected] 2024-04-06T09:37:07.523 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:37:07.524 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=57, tag_tid=9, entry_tid=1' 2024-04-06T09:37:07.526 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:37:07.526 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:37:07.526 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:37:07.526 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:37:07.526 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:37:07.526 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:37:07.526 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:37:07.526 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:37:07.526 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:37:07.526 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:37:07.527 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:37:07.527 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:37:07.527 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:37:07.527 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:37:07.527 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:37:07.572 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 13 2024-04-06T09:37:07.572 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 14 2024-04-06T09:37:07.573 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:37:07.573 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=57, tag_tid=9, entry_tid=1], [object_number=56, tag_tid=9, entry_tid=0], [object_number=59, tag_tid=8, entry_tid=11939], [object_number=58, tag_tid=8, entry_tid=11938]]], state=connected] 2024-04-06T09:37:07.573 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=55, tag_tid=8, entry_tid=6883], [object_number=54, tag_tid=8, entry_tid=6882], [object_number=53, tag_tid=8, entry_tid=6881], [object_number=52, tag_tid=8, entry_tid=6880]]], state=connected] 2024-04-06T09:37:07.576 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:37:07.578 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=55, tag_tid=8, entry_tid=6883' 2024-04-06T09:37:07.578 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=57, tag_tid=9, entry_tid=1' -a 'object_number=57, tag_tid=9, entry_tid=1' = 'object_number=55, tag_tid=8, entry_tid=6883' 2024-04-06T09:37:07.578 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=55, tag_tid=8, entry_tid=6883' '!=' 'object_number=53, tag_tid=8, entry_tid=6793' 2024-04-06T09:37:07.578 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:37:07.578 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=55, tag_tid=8, entry_tid=6883' = 'object_number=53, tag_tid=8, entry_tid=6793' 2024-04-06T09:37:07.578 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=55, tag_tid=8, entry_tid=6883' 2024-04-06T09:37:07.578 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=57, tag_tid=9, entry_tid=1' 2024-04-06T09:37:07.578 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:37:07.578 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:37:07.579 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=9 2024-04-06T09:37:07.580 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=55, tag_tid=8, entry_tid=6883' 2024-04-06T09:37:07.580 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:37:07.580 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:37:07.581 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=8 2024-04-06T09:37:07.581 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=57, tag_tid=9, entry_tid=1' 2024-04-06T09:37:07.581 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:37:07.581 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:37:07.582 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:37:07.583 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=55, tag_tid=8, entry_tid=6883' 2024-04-06T09:37:07.583 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:37:07.583 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:37:07.584 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=6883 2024-04-06T09:37:07.584 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 9 = 8 -a 1 -le 6883 2024-04-06T09:37:07.584 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:37:07.585 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:37:07.585 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:37:07.786 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:37:07.786 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:37:07.787 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:37:07.787 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:37:07.787 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:37:07.787 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:37:07.787 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:37:07.787 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:37:07.787 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:37:07.787 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:37:07.787 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:37:07.787 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:37:07.787 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:37:07.787 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:37:07.787 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:37:07.788 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:37:07.788 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:37:07.791 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:37:07.791 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:37:07.791 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:37:07.791 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:37:07.792 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:37:07.792 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:37:07.792 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:37:07.792 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:37:07.792 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:37:07.792 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:37:07.792 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:37:07.792 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:37:07.792 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:37:07.792 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:37:07.792 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:37:07.792 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:37:07.792 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:37:07.792 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:37:07.793 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:37:07.793 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:37:07.793 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:37:07.793 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:37:07.793 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:37:07.793 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:37:07.793 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:37:07.793 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:37:07.793 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:37:07.793 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:37:07.793 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:37:07.793 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:37:07.793 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:37:07.794 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:37:07.818 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:37:07.818 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:37:07.818 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:37:07.818 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:37:07.953 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:37:07.954 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:37:07.954 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:37:07.954 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:37:07.954 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:37:07.954 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:37:07.954 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:37:07.955 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:37:07.955 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:37:07.955 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:37:07.955 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:37:07.955 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:37:07.955 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:37:07.957 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:37:07.958 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:37:07.958 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:37:08.004 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 13 2024-04-06T09:37:08.004 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 14 2024-04-06T09:37:08.004 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:37:08.004 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=57, tag_tid=9, entry_tid=1], [object_number=56, tag_tid=9, entry_tid=0], [object_number=59, tag_tid=8, entry_tid=11939], [object_number=58, tag_tid=8, entry_tid=11938]]], state=connected] 2024-04-06T09:37:08.004 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=53, tag_tid=8, entry_tid=6977], [object_number=52, tag_tid=8, entry_tid=6976], [object_number=55, tag_tid=8, entry_tid=6975], [object_number=54, tag_tid=8, entry_tid=6974]]], state=connected] 2024-04-06T09:37:08.030 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:37:08.031 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=57, tag_tid=9, entry_tid=1' 2024-04-06T09:37:08.032 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:37:08.032 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:37:08.033 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:37:08.033 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:37:08.033 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:37:08.033 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:37:08.033 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:37:08.033 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:37:08.033 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:37:08.033 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:37:08.033 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:37:08.033 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:37:08.034 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:37:08.034 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:37:08.034 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:37:08.079 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 13 2024-04-06T09:37:08.079 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 14 2024-04-06T09:37:08.079 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:37:08.079 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=57, tag_tid=9, entry_tid=1], [object_number=56, tag_tid=9, entry_tid=0], [object_number=59, tag_tid=8, entry_tid=11939], [object_number=58, tag_tid=8, entry_tid=11938]]], state=connected] 2024-04-06T09:37:08.079 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=53, tag_tid=8, entry_tid=6977], [object_number=52, tag_tid=8, entry_tid=6976], [object_number=55, tag_tid=8, entry_tid=6975], [object_number=54, tag_tid=8, entry_tid=6974]]], state=connected] 2024-04-06T09:37:08.101 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:37:08.103 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=53, tag_tid=8, entry_tid=6977' 2024-04-06T09:37:08.103 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=57, tag_tid=9, entry_tid=1' -a 'object_number=57, tag_tid=9, entry_tid=1' = 'object_number=53, tag_tid=8, entry_tid=6977' 2024-04-06T09:37:08.103 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=53, tag_tid=8, entry_tid=6977' '!=' 'object_number=55, tag_tid=8, entry_tid=6883' 2024-04-06T09:37:08.103 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:37:08.103 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=53, tag_tid=8, entry_tid=6977' = 'object_number=55, tag_tid=8, entry_tid=6883' 2024-04-06T09:37:08.103 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=53, tag_tid=8, entry_tid=6977' 2024-04-06T09:37:08.104 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=57, tag_tid=9, entry_tid=1' 2024-04-06T09:37:08.104 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:37:08.104 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:37:08.104 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=9 2024-04-06T09:37:08.105 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=53, tag_tid=8, entry_tid=6977' 2024-04-06T09:37:08.105 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:37:08.105 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:37:08.106 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=8 2024-04-06T09:37:08.107 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=57, tag_tid=9, entry_tid=1' 2024-04-06T09:37:08.107 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:37:08.107 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:37:08.108 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:37:08.109 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=53, tag_tid=8, entry_tid=6977' 2024-04-06T09:37:08.109 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:37:08.109 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:37:08.110 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=6977 2024-04-06T09:37:08.110 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 9 = 8 -a 1 -le 6977 2024-04-06T09:37:08.110 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:37:08.110 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:37:08.110 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:37:08.312 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:37:08.312 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:37:08.312 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:37:08.312 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:37:08.312 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:37:08.312 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:37:08.313 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:37:08.313 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:37:08.313 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:37:08.313 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:37:08.313 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:37:08.313 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:37:08.313 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:37:08.313 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:37:08.313 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:37:08.313 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:37:08.314 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:37:08.316 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:37:08.316 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:37:08.316 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:37:08.317 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:37:08.317 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:37:08.317 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:37:08.317 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:37:08.317 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:37:08.317 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:37:08.317 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:37:08.317 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:37:08.317 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:37:08.317 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:37:08.317 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:37:08.317 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:37:08.317 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:37:08.318 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:37:08.318 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:37:08.318 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:37:08.318 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:37:08.319 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:37:08.319 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:37:08.319 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:37:08.319 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:37:08.319 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:37:08.319 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:37:08.319 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:37:08.319 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:37:08.319 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:37:08.319 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:37:08.319 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:37:08.319 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:37:08.343 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:37:08.343 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:37:08.343 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:37:08.343 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:37:08.472 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:37:08.473 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:37:08.473 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:37:08.473 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:37:08.473 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:37:08.473 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:37:08.473 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:37:08.473 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:37:08.473 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:37:08.473 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:37:08.474 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:37:08.474 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:37:08.474 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:37:08.475 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:37:08.475 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:37:08.476 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:37:08.526 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 13 2024-04-06T09:37:08.526 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 14 2024-04-06T09:37:08.526 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:37:08.526 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=57, tag_tid=9, entry_tid=1], [object_number=56, tag_tid=9, entry_tid=0], [object_number=59, tag_tid=8, entry_tid=11939], [object_number=58, tag_tid=8, entry_tid=11938]]], state=connected] 2024-04-06T09:37:08.526 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=54, tag_tid=8, entry_tid=7078], [object_number=53, tag_tid=8, entry_tid=7077], [object_number=52, tag_tid=8, entry_tid=7076], [object_number=55, tag_tid=8, entry_tid=7075]]], state=connected] 2024-04-06T09:37:08.529 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:37:08.531 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=57, tag_tid=9, entry_tid=1' 2024-04-06T09:37:08.532 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:37:08.532 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:37:08.532 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:37:08.532 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:37:08.532 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:37:08.532 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:37:08.532 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:37:08.532 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:37:08.532 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:37:08.532 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:37:08.532 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:37:08.532 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:37:08.533 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:37:08.533 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:37:08.533 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:37:08.579 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 13 2024-04-06T09:37:08.579 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 14 2024-04-06T09:37:08.579 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:37:08.579 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=57, tag_tid=9, entry_tid=1], [object_number=56, tag_tid=9, entry_tid=0], [object_number=59, tag_tid=8, entry_tid=11939], [object_number=58, tag_tid=8, entry_tid=11938]]], state=connected] 2024-04-06T09:37:08.579 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=54, tag_tid=8, entry_tid=7078], [object_number=53, tag_tid=8, entry_tid=7077], [object_number=52, tag_tid=8, entry_tid=7076], [object_number=55, tag_tid=8, entry_tid=7075]]], state=connected] 2024-04-06T09:37:08.583 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:37:08.584 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=54, tag_tid=8, entry_tid=7078' 2024-04-06T09:37:08.584 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=57, tag_tid=9, entry_tid=1' -a 'object_number=57, tag_tid=9, entry_tid=1' = 'object_number=54, tag_tid=8, entry_tid=7078' 2024-04-06T09:37:08.585 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=54, tag_tid=8, entry_tid=7078' '!=' 'object_number=53, tag_tid=8, entry_tid=6977' 2024-04-06T09:37:08.585 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:37:08.585 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=54, tag_tid=8, entry_tid=7078' = 'object_number=53, tag_tid=8, entry_tid=6977' 2024-04-06T09:37:08.585 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=54, tag_tid=8, entry_tid=7078' 2024-04-06T09:37:08.585 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=57, tag_tid=9, entry_tid=1' 2024-04-06T09:37:08.585 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:37:08.586 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:37:08.586 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=9 2024-04-06T09:37:08.587 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=54, tag_tid=8, entry_tid=7078' 2024-04-06T09:37:08.587 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:37:08.587 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:37:08.588 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=8 2024-04-06T09:37:08.588 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=57, tag_tid=9, entry_tid=1' 2024-04-06T09:37:08.589 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:37:08.589 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:37:08.589 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:37:08.590 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=54, tag_tid=8, entry_tid=7078' 2024-04-06T09:37:08.590 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:37:08.590 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:37:08.591 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=7078 2024-04-06T09:37:08.591 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 9 = 8 -a 1 -le 7078 2024-04-06T09:37:08.591 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:37:08.592 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:37:08.592 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:37:08.793 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:37:08.794 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:37:08.794 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:37:08.794 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:37:08.794 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:37:08.794 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:37:08.794 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:37:08.794 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:37:08.794 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:37:08.794 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:37:08.794 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:37:08.794 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:37:08.794 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:37:08.794 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:37:08.794 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:37:08.796 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:37:08.796 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:37:08.796 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:37:08.796 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:37:08.796 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:37:08.796 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:37:08.796 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:37:08.796 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:37:08.796 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:37:08.796 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:37:08.796 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:37:08.796 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:37:08.796 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:37:08.796 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:37:08.797 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:37:08.797 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:37:08.797 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:37:08.797 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:37:08.798 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:37:08.798 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:37:08.798 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:37:08.798 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:37:08.799 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:37:08.799 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:37:08.799 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:37:08.799 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:37:08.799 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:37:08.799 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:37:08.799 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:37:08.799 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:37:08.799 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:37:08.799 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:37:08.799 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:37:08.799 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:37:08.824 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:37:08.825 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:37:08.825 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:37:08.825 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:37:08.957 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:37:08.958 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:37:08.958 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:37:08.958 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:37:08.959 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:37:08.959 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:37:08.959 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:37:08.959 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:37:08.959 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:37:08.959 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:37:08.959 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:37:08.959 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:37:08.959 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:37:08.961 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:37:08.961 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:37:08.961 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:37:09.006 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 13 2024-04-06T09:37:09.006 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 14 2024-04-06T09:37:09.006 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:37:09.006 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=57, tag_tid=9, entry_tid=1], [object_number=56, tag_tid=9, entry_tid=0], [object_number=59, tag_tid=8, entry_tid=11939], [object_number=58, tag_tid=8, entry_tid=11938]]], state=connected] 2024-04-06T09:37:09.006 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=55, tag_tid=8, entry_tid=7175], [object_number=54, tag_tid=8, entry_tid=7174], [object_number=53, tag_tid=8, entry_tid=7173], [object_number=52, tag_tid=8, entry_tid=7172]]], state=connected] 2024-04-06T09:37:09.010 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:37:09.011 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=57, tag_tid=9, entry_tid=1' 2024-04-06T09:37:09.012 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:37:09.012 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:37:09.012 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:37:09.012 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:37:09.012 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:37:09.012 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:37:09.012 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:37:09.012 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:37:09.012 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:37:09.013 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:37:09.013 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:37:09.013 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:37:09.013 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:37:09.014 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:37:09.014 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:37:09.059 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 13 2024-04-06T09:37:09.059 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 14 2024-04-06T09:37:09.059 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:37:09.059 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=57, tag_tid=9, entry_tid=1], [object_number=56, tag_tid=9, entry_tid=0], [object_number=59, tag_tid=8, entry_tid=11939], [object_number=58, tag_tid=8, entry_tid=11938]]], state=connected] 2024-04-06T09:37:09.059 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=55, tag_tid=8, entry_tid=7175], [object_number=54, tag_tid=8, entry_tid=7174], [object_number=53, tag_tid=8, entry_tid=7173], [object_number=52, tag_tid=8, entry_tid=7172]]], state=connected] 2024-04-06T09:37:09.062 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:37:09.065 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=55, tag_tid=8, entry_tid=7175' 2024-04-06T09:37:09.065 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=57, tag_tid=9, entry_tid=1' -a 'object_number=57, tag_tid=9, entry_tid=1' = 'object_number=55, tag_tid=8, entry_tid=7175' 2024-04-06T09:37:09.065 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=55, tag_tid=8, entry_tid=7175' '!=' 'object_number=54, tag_tid=8, entry_tid=7078' 2024-04-06T09:37:09.065 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:37:09.065 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=55, tag_tid=8, entry_tid=7175' = 'object_number=54, tag_tid=8, entry_tid=7078' 2024-04-06T09:37:09.065 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=55, tag_tid=8, entry_tid=7175' 2024-04-06T09:37:09.065 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=57, tag_tid=9, entry_tid=1' 2024-04-06T09:37:09.065 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:37:09.065 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:37:09.066 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=9 2024-04-06T09:37:09.066 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=55, tag_tid=8, entry_tid=7175' 2024-04-06T09:37:09.066 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:37:09.066 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:37:09.067 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=8 2024-04-06T09:37:09.068 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=57, tag_tid=9, entry_tid=1' 2024-04-06T09:37:09.068 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:37:09.068 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:37:09.069 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:37:09.070 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=55, tag_tid=8, entry_tid=7175' 2024-04-06T09:37:09.070 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:37:09.070 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:37:09.071 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=7175 2024-04-06T09:37:09.071 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 9 = 8 -a 1 -le 7175 2024-04-06T09:37:09.071 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:37:09.071 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:37:09.071 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:37:09.273 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:37:09.273 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:37:09.273 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:37:09.273 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:37:09.273 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:37:09.274 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:37:09.274 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:37:09.274 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:37:09.274 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:37:09.274 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:37:09.274 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:37:09.274 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:37:09.274 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:37:09.274 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:37:09.274 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:37:09.275 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:37:09.275 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:37:09.278 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:37:09.278 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:37:09.279 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:37:09.279 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:37:09.279 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:37:09.279 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:37:09.279 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:37:09.279 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:37:09.279 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:37:09.279 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:37:09.279 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:37:09.279 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:37:09.279 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:37:09.279 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:37:09.279 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:37:09.279 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:37:09.279 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:37:09.279 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:37:09.280 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:37:09.280 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:37:09.280 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:37:09.280 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:37:09.280 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:37:09.280 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:37:09.280 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:37:09.280 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:37:09.280 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:37:09.280 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:37:09.280 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:37:09.280 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:37:09.280 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:37:09.280 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:37:09.303 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:37:09.303 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:37:09.303 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:37:09.303 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:37:09.433 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:37:09.434 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:37:09.434 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:37:09.434 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:37:09.434 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:37:09.434 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:37:09.434 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:37:09.434 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:37:09.434 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:37:09.435 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:37:09.435 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:37:09.435 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:37:09.435 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:37:09.437 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:37:09.437 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:37:09.437 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:37:09.482 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 13 2024-04-06T09:37:09.482 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 14 2024-04-06T09:37:09.483 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:37:09.483 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=57, tag_tid=9, entry_tid=1], [object_number=56, tag_tid=9, entry_tid=0], [object_number=59, tag_tid=8, entry_tid=11939], [object_number=58, tag_tid=8, entry_tid=11938]]], state=connected] 2024-04-06T09:37:09.483 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=53, tag_tid=8, entry_tid=7269], [object_number=52, tag_tid=8, entry_tid=7268], [object_number=55, tag_tid=8, entry_tid=7267], [object_number=54, tag_tid=8, entry_tid=7266]]], state=connected] 2024-04-06T09:37:09.486 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:37:09.487 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=57, tag_tid=9, entry_tid=1' 2024-04-06T09:37:09.488 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:37:09.489 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:37:09.489 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:37:09.489 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:37:09.489 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:37:09.489 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:37:09.489 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:37:09.489 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:37:09.489 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:37:09.489 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:37:09.489 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:37:09.489 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:37:09.490 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:37:09.490 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:37:09.490 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:37:09.535 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 13 2024-04-06T09:37:09.535 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 14 2024-04-06T09:37:09.535 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:37:09.535 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=57, tag_tid=9, entry_tid=1], [object_number=56, tag_tid=9, entry_tid=0], [object_number=59, tag_tid=8, entry_tid=11939], [object_number=58, tag_tid=8, entry_tid=11938]]], state=connected] 2024-04-06T09:37:09.535 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=53, tag_tid=8, entry_tid=7269], [object_number=52, tag_tid=8, entry_tid=7268], [object_number=55, tag_tid=8, entry_tid=7267], [object_number=54, tag_tid=8, entry_tid=7266]]], state=connected] 2024-04-06T09:37:09.538 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:37:09.540 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=53, tag_tid=8, entry_tid=7269' 2024-04-06T09:37:09.540 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=57, tag_tid=9, entry_tid=1' -a 'object_number=57, tag_tid=9, entry_tid=1' = 'object_number=53, tag_tid=8, entry_tid=7269' 2024-04-06T09:37:09.540 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=53, tag_tid=8, entry_tid=7269' '!=' 'object_number=55, tag_tid=8, entry_tid=7175' 2024-04-06T09:37:09.541 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:37:09.541 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=53, tag_tid=8, entry_tid=7269' = 'object_number=55, tag_tid=8, entry_tid=7175' 2024-04-06T09:37:09.541 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=53, tag_tid=8, entry_tid=7269' 2024-04-06T09:37:09.541 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=57, tag_tid=9, entry_tid=1' 2024-04-06T09:37:09.541 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:37:09.541 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:37:09.541 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=9 2024-04-06T09:37:09.542 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=53, tag_tid=8, entry_tid=7269' 2024-04-06T09:37:09.542 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:37:09.542 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:37:09.543 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=8 2024-04-06T09:37:09.544 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=57, tag_tid=9, entry_tid=1' 2024-04-06T09:37:09.544 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:37:09.544 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:37:09.545 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:37:09.545 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=53, tag_tid=8, entry_tid=7269' 2024-04-06T09:37:09.545 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:37:09.546 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:37:09.547 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=7269 2024-04-06T09:37:09.547 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 9 = 8 -a 1 -le 7269 2024-04-06T09:37:09.547 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:37:09.547 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:37:09.547 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:37:09.749 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:37:09.749 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:37:09.749 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:37:09.749 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:37:09.749 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:37:09.749 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:37:09.749 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:37:09.749 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:37:09.750 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:37:09.750 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:37:09.750 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:37:09.750 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:37:09.750 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:37:09.750 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:37:09.750 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:37:09.750 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:37:09.751 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:37:09.753 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:37:09.753 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:37:09.753 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:37:09.754 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:37:09.754 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:37:09.754 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:37:09.754 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:37:09.754 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:37:09.754 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:37:09.754 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:37:09.754 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:37:09.754 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:37:09.754 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:37:09.754 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:37:09.754 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:37:09.754 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:37:09.754 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:37:09.754 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:37:09.755 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:37:09.755 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:37:09.755 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:37:09.755 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:37:09.755 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:37:09.755 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:37:09.755 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:37:09.755 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:37:09.755 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:37:09.755 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:37:09.755 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:37:09.755 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:37:09.755 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:37:09.756 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:37:09.779 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:37:09.779 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:37:09.779 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:37:09.779 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:37:09.909 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:37:09.911 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:37:09.911 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:37:09.911 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:37:09.911 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:37:09.912 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:37:09.912 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:37:09.912 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:37:09.912 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:37:09.912 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:37:09.912 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:37:09.912 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:37:09.912 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:37:09.913 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:37:09.913 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:37:09.913 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:37:09.965 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 13 2024-04-06T09:37:09.965 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 14 2024-04-06T09:37:09.965 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:37:09.965 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=57, tag_tid=9, entry_tid=1], [object_number=56, tag_tid=9, entry_tid=0], [object_number=59, tag_tid=8, entry_tid=11939], [object_number=58, tag_tid=8, entry_tid=11938]]], state=connected] 2024-04-06T09:37:09.965 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=55, tag_tid=8, entry_tid=7363], [object_number=54, tag_tid=8, entry_tid=7362], [object_number=53, tag_tid=8, entry_tid=7361], [object_number=52, tag_tid=8, entry_tid=7360]]], state=connected] 2024-04-06T09:37:09.969 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:37:09.971 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=57, tag_tid=9, entry_tid=1' 2024-04-06T09:37:09.971 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:37:09.972 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:37:09.972 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:37:09.972 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:37:09.972 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:37:09.972 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:37:09.972 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:37:09.972 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:37:09.972 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:37:09.972 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:37:09.972 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:37:09.972 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:37:09.973 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:37:09.973 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:37:09.973 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:37:10.227 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 13 2024-04-06T09:37:10.227 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 14 2024-04-06T09:37:10.227 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:37:10.227 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=57, tag_tid=9, entry_tid=1], [object_number=56, tag_tid=9, entry_tid=0], [object_number=59, tag_tid=8, entry_tid=11939], [object_number=58, tag_tid=8, entry_tid=11938]]], state=connected] 2024-04-06T09:37:10.227 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=55, tag_tid=8, entry_tid=7363], [object_number=54, tag_tid=8, entry_tid=7362], [object_number=53, tag_tid=8, entry_tid=7361], [object_number=52, tag_tid=8, entry_tid=7360]]], state=connected] 2024-04-06T09:37:10.233 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:37:10.235 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=55, tag_tid=8, entry_tid=7363' 2024-04-06T09:37:10.235 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=57, tag_tid=9, entry_tid=1' -a 'object_number=57, tag_tid=9, entry_tid=1' = 'object_number=55, tag_tid=8, entry_tid=7363' 2024-04-06T09:37:10.235 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=55, tag_tid=8, entry_tid=7363' '!=' 'object_number=53, tag_tid=8, entry_tid=7269' 2024-04-06T09:37:10.235 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:37:10.236 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=55, tag_tid=8, entry_tid=7363' = 'object_number=53, tag_tid=8, entry_tid=7269' 2024-04-06T09:37:10.236 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=55, tag_tid=8, entry_tid=7363' 2024-04-06T09:37:10.236 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=57, tag_tid=9, entry_tid=1' 2024-04-06T09:37:10.237 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:37:10.237 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:37:10.238 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=9 2024-04-06T09:37:10.239 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=55, tag_tid=8, entry_tid=7363' 2024-04-06T09:37:10.240 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:37:10.240 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:37:10.241 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=8 2024-04-06T09:37:10.242 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=57, tag_tid=9, entry_tid=1' 2024-04-06T09:37:10.242 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:37:10.242 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:37:10.243 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:37:10.244 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=55, tag_tid=8, entry_tid=7363' 2024-04-06T09:37:10.245 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:37:10.245 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:37:10.246 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=7363 2024-04-06T09:37:10.246 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 9 = 8 -a 1 -le 7363 2024-04-06T09:37:10.247 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:37:10.247 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:37:10.247 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:37:10.449 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:37:10.449 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:37:10.449 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:37:10.449 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:37:10.449 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:37:10.449 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:37:10.449 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:37:10.449 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:37:10.449 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:37:10.449 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:37:10.449 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:37:10.449 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:37:10.450 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:37:10.450 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:37:10.450 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:37:10.450 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:37:10.450 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:37:10.453 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:37:10.453 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:37:10.453 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:37:10.453 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:37:10.453 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:37:10.454 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:37:10.454 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:37:10.454 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:37:10.454 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:37:10.454 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:37:10.454 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:37:10.454 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:37:10.454 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:37:10.454 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:37:10.454 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:37:10.454 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:37:10.454 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:37:10.454 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:37:10.454 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:37:10.455 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:37:10.455 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:37:10.455 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:37:10.455 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:37:10.455 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:37:10.455 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:37:10.455 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:37:10.455 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:37:10.455 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:37:10.455 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:37:10.455 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:37:10.455 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:37:10.455 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:37:10.479 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:37:10.480 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:37:10.480 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:37:10.480 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:37:10.612 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:37:10.613 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:37:10.613 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:37:10.613 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:37:10.613 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:37:10.614 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:37:10.614 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:37:10.614 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:37:10.614 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:37:10.614 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:37:10.614 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:37:10.615 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:37:10.615 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:37:10.616 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:37:10.617 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:37:10.617 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:37:10.669 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 13 2024-04-06T09:37:10.669 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 14 2024-04-06T09:37:10.669 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:37:10.669 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=57, tag_tid=9, entry_tid=1], [object_number=56, tag_tid=9, entry_tid=0], [object_number=59, tag_tid=8, entry_tid=11939], [object_number=58, tag_tid=8, entry_tid=11938]]], state=connected] 2024-04-06T09:37:10.669 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=54, tag_tid=8, entry_tid=7502], [object_number=53, tag_tid=8, entry_tid=7501], [object_number=52, tag_tid=8, entry_tid=7500], [object_number=55, tag_tid=8, entry_tid=7499]]], state=connected] 2024-04-06T09:37:10.673 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:37:10.674 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=57, tag_tid=9, entry_tid=1' 2024-04-06T09:37:10.675 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:37:10.675 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:37:10.675 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:37:10.675 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:37:10.675 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:37:10.676 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:37:10.676 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:37:10.676 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:37:10.676 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:37:10.676 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:37:10.676 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:37:10.676 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:37:10.676 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:37:10.677 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:37:10.677 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:37:10.722 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 13 2024-04-06T09:37:10.722 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 14 2024-04-06T09:37:10.722 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:37:10.722 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=57, tag_tid=9, entry_tid=1], [object_number=56, tag_tid=9, entry_tid=0], [object_number=59, tag_tid=8, entry_tid=11939], [object_number=58, tag_tid=8, entry_tid=11938]]], state=connected] 2024-04-06T09:37:10.722 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=54, tag_tid=8, entry_tid=7502], [object_number=53, tag_tid=8, entry_tid=7501], [object_number=52, tag_tid=8, entry_tid=7500], [object_number=55, tag_tid=8, entry_tid=7499]]], state=connected] 2024-04-06T09:37:10.726 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:37:10.728 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=54, tag_tid=8, entry_tid=7502' 2024-04-06T09:37:10.728 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=57, tag_tid=9, entry_tid=1' -a 'object_number=57, tag_tid=9, entry_tid=1' = 'object_number=54, tag_tid=8, entry_tid=7502' 2024-04-06T09:37:10.728 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=54, tag_tid=8, entry_tid=7502' '!=' 'object_number=55, tag_tid=8, entry_tid=7363' 2024-04-06T09:37:10.728 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:37:10.728 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=54, tag_tid=8, entry_tid=7502' = 'object_number=55, tag_tid=8, entry_tid=7363' 2024-04-06T09:37:10.728 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=54, tag_tid=8, entry_tid=7502' 2024-04-06T09:37:10.728 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=57, tag_tid=9, entry_tid=1' 2024-04-06T09:37:10.728 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:37:10.728 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:37:10.729 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=9 2024-04-06T09:37:10.730 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=54, tag_tid=8, entry_tid=7502' 2024-04-06T09:37:10.730 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:37:10.730 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:37:10.731 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=8 2024-04-06T09:37:10.731 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=57, tag_tid=9, entry_tid=1' 2024-04-06T09:37:10.732 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:37:10.732 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:37:10.732 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:37:10.733 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=54, tag_tid=8, entry_tid=7502' 2024-04-06T09:37:10.733 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:37:10.733 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:37:10.734 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=7502 2024-04-06T09:37:10.735 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 9 = 8 -a 1 -le 7502 2024-04-06T09:37:10.735 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:37:10.735 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:37:10.735 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:37:10.936 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:37:10.936 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:37:10.936 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:37:10.937 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:37:10.937 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:37:10.937 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:37:10.937 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:37:10.937 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:37:10.937 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:37:10.937 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:37:10.937 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:37:10.937 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:37:10.937 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:37:10.937 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:37:10.937 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:37:10.938 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:37:10.938 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:37:10.942 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:37:10.942 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:37:10.942 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:37:10.942 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:37:10.942 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:37:10.942 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:37:10.942 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:37:10.942 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:37:10.942 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:37:10.942 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:37:10.943 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:37:10.943 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:37:10.943 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:37:10.943 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:37:10.943 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:37:10.943 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:37:10.943 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:37:10.943 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:37:10.943 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:37:10.943 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:37:10.943 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:37:10.943 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:37:10.943 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:37:10.943 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:37:10.943 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:37:10.944 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:37:10.944 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:37:10.944 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:37:10.944 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:37:10.944 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:37:10.944 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:37:10.944 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:37:10.966 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:37:10.967 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:37:10.967 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:37:10.967 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:37:11.091 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:37:11.093 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:37:11.093 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:37:11.093 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:37:11.093 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:37:11.093 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:37:11.093 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:37:11.093 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:37:11.093 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:37:11.093 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:37:11.093 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:37:11.094 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:37:11.094 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:37:11.095 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:37:11.095 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:37:11.096 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:37:11.146 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 13 2024-04-06T09:37:11.146 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 14 2024-04-06T09:37:11.146 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:37:11.146 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=57, tag_tid=9, entry_tid=1], [object_number=56, tag_tid=9, entry_tid=0], [object_number=59, tag_tid=8, entry_tid=11939], [object_number=58, tag_tid=8, entry_tid=11938]]], state=connected] 2024-04-06T09:37:11.147 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=52, tag_tid=8, entry_tid=7592], [object_number=55, tag_tid=8, entry_tid=7591], [object_number=54, tag_tid=8, entry_tid=7590], [object_number=53, tag_tid=8, entry_tid=7589]]], state=connected] 2024-04-06T09:37:11.150 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:37:11.153 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=57, tag_tid=9, entry_tid=1' 2024-04-06T09:37:11.153 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:37:11.153 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:37:11.153 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:37:11.153 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:37:11.153 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:37:11.153 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:37:11.153 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:37:11.153 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:37:11.153 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:37:11.153 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:37:11.153 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:37:11.153 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:37:11.154 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:37:11.154 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:37:11.154 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:37:11.199 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 13 2024-04-06T09:37:11.199 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 14 2024-04-06T09:37:11.199 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:37:11.199 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=57, tag_tid=9, entry_tid=1], [object_number=56, tag_tid=9, entry_tid=0], [object_number=59, tag_tid=8, entry_tid=11939], [object_number=58, tag_tid=8, entry_tid=11938]]], state=connected] 2024-04-06T09:37:11.199 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=52, tag_tid=8, entry_tid=7592], [object_number=55, tag_tid=8, entry_tid=7591], [object_number=54, tag_tid=8, entry_tid=7590], [object_number=53, tag_tid=8, entry_tid=7589]]], state=connected] 2024-04-06T09:37:11.203 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:37:11.205 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=52, tag_tid=8, entry_tid=7592' 2024-04-06T09:37:11.205 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=57, tag_tid=9, entry_tid=1' -a 'object_number=57, tag_tid=9, entry_tid=1' = 'object_number=52, tag_tid=8, entry_tid=7592' 2024-04-06T09:37:11.205 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=52, tag_tid=8, entry_tid=7592' '!=' 'object_number=54, tag_tid=8, entry_tid=7502' 2024-04-06T09:37:11.206 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:37:11.206 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=52, tag_tid=8, entry_tid=7592' = 'object_number=54, tag_tid=8, entry_tid=7502' 2024-04-06T09:37:11.206 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=52, tag_tid=8, entry_tid=7592' 2024-04-06T09:37:11.206 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=57, tag_tid=9, entry_tid=1' 2024-04-06T09:37:11.206 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:37:11.206 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:37:11.206 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=9 2024-04-06T09:37:11.207 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=52, tag_tid=8, entry_tid=7592' 2024-04-06T09:37:11.207 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:37:11.207 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:37:11.208 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=8 2024-04-06T09:37:11.209 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=57, tag_tid=9, entry_tid=1' 2024-04-06T09:37:11.209 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:37:11.209 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:37:11.210 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:37:11.211 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=52, tag_tid=8, entry_tid=7592' 2024-04-06T09:37:11.211 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:37:11.211 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:37:11.212 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=7592 2024-04-06T09:37:11.212 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 9 = 8 -a 1 -le 7592 2024-04-06T09:37:11.212 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:37:11.212 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:37:11.213 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:37:11.414 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:37:11.414 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:37:11.415 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:37:11.415 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:37:11.415 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:37:11.415 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:37:11.415 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:37:11.415 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:37:11.415 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:37:11.415 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:37:11.415 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:37:11.415 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:37:11.415 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:37:11.415 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:37:11.415 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:37:11.416 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:37:11.416 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:37:11.419 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:37:11.419 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:37:11.419 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:37:11.419 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:37:11.419 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:37:11.419 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:37:11.419 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:37:11.420 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:37:11.420 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:37:11.420 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:37:11.420 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:37:11.420 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:37:11.420 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:37:11.420 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:37:11.420 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:37:11.420 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:37:11.420 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:37:11.420 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:37:11.420 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:37:11.420 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:37:11.420 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:37:11.421 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:37:11.421 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:37:11.421 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:37:11.421 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:37:11.421 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:37:11.421 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:37:11.421 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:37:11.421 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:37:11.421 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:37:11.421 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:37:11.421 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:37:11.446 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:37:11.446 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:37:11.446 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:37:11.446 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:37:11.588 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:37:11.590 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:37:11.590 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:37:11.590 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:37:11.590 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:37:11.590 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:37:11.590 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:37:11.590 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:37:11.590 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:37:11.590 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:37:11.590 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:37:11.591 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:37:11.591 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:37:11.592 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:37:11.592 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:37:11.592 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:37:11.643 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 13 2024-04-06T09:37:11.643 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 14 2024-04-06T09:37:11.643 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:37:11.643 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=57, tag_tid=9, entry_tid=1], [object_number=56, tag_tid=9, entry_tid=0], [object_number=59, tag_tid=8, entry_tid=11939], [object_number=58, tag_tid=8, entry_tid=11938]]], state=connected] 2024-04-06T09:37:11.644 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=53, tag_tid=8, entry_tid=7689], [object_number=52, tag_tid=8, entry_tid=7688], [object_number=55, tag_tid=8, entry_tid=7687], [object_number=54, tag_tid=8, entry_tid=7686]]], state=connected] 2024-04-06T09:37:11.646 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:37:11.648 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=57, tag_tid=9, entry_tid=1' 2024-04-06T09:37:11.649 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:37:11.649 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:37:11.649 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:37:11.649 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:37:11.649 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:37:11.649 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:37:11.649 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:37:11.649 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:37:11.649 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:37:11.650 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:37:11.650 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:37:11.650 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:37:11.650 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:37:11.650 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:37:11.650 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:37:11.697 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 13 2024-04-06T09:37:11.697 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 14 2024-04-06T09:37:11.697 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:37:11.697 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=57, tag_tid=9, entry_tid=1], [object_number=56, tag_tid=9, entry_tid=0], [object_number=59, tag_tid=8, entry_tid=11939], [object_number=58, tag_tid=8, entry_tid=11938]]], state=connected] 2024-04-06T09:37:11.697 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=53, tag_tid=8, entry_tid=7689], [object_number=52, tag_tid=8, entry_tid=7688], [object_number=55, tag_tid=8, entry_tid=7687], [object_number=54, tag_tid=8, entry_tid=7686]]], state=connected] 2024-04-06T09:37:11.700 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:37:11.703 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=53, tag_tid=8, entry_tid=7689' 2024-04-06T09:37:11.703 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=57, tag_tid=9, entry_tid=1' -a 'object_number=57, tag_tid=9, entry_tid=1' = 'object_number=53, tag_tid=8, entry_tid=7689' 2024-04-06T09:37:11.703 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=53, tag_tid=8, entry_tid=7689' '!=' 'object_number=52, tag_tid=8, entry_tid=7592' 2024-04-06T09:37:11.703 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:37:11.703 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=53, tag_tid=8, entry_tid=7689' = 'object_number=52, tag_tid=8, entry_tid=7592' 2024-04-06T09:37:11.703 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=53, tag_tid=8, entry_tid=7689' 2024-04-06T09:37:11.703 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=57, tag_tid=9, entry_tid=1' 2024-04-06T09:37:11.703 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:37:11.704 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:37:11.704 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=9 2024-04-06T09:37:11.705 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=53, tag_tid=8, entry_tid=7689' 2024-04-06T09:37:11.705 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:37:11.705 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:37:11.706 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=8 2024-04-06T09:37:11.706 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=57, tag_tid=9, entry_tid=1' 2024-04-06T09:37:11.707 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:37:11.707 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:37:11.708 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:37:11.708 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=53, tag_tid=8, entry_tid=7689' 2024-04-06T09:37:11.708 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:37:11.708 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:37:11.710 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=7689 2024-04-06T09:37:11.710 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 9 = 8 -a 1 -le 7689 2024-04-06T09:37:11.710 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:37:11.710 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:37:11.710 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:37:11.912 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:37:11.912 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:37:11.912 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:37:11.912 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:37:11.912 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:37:11.912 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:37:11.912 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:37:11.912 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:37:11.912 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:37:11.913 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:37:11.913 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:37:11.913 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:37:11.913 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:37:11.913 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:37:11.913 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:37:11.913 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:37:11.913 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:37:11.916 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:37:11.916 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:37:11.916 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:37:11.916 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:37:11.916 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:37:11.916 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:37:11.916 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:37:11.916 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:37:11.916 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:37:11.917 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:37:11.917 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:37:11.917 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:37:11.917 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:37:11.917 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:37:11.917 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:37:11.917 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:37:11.917 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:37:11.917 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:37:11.917 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:37:11.917 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:37:11.917 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:37:11.917 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:37:11.917 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:37:11.918 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:37:11.918 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:37:11.918 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:37:11.918 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:37:11.918 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:37:11.918 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:37:11.918 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:37:11.918 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:37:11.918 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:37:11.942 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:37:11.942 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:37:11.942 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:37:11.942 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:37:12.072 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:37:12.073 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:37:12.073 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:37:12.073 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:37:12.073 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:37:12.073 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:37:12.074 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:37:12.074 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:37:12.074 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:37:12.074 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:37:12.074 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:37:12.075 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:37:12.075 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:37:12.077 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:37:12.078 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:37:12.078 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:37:12.132 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 13 2024-04-06T09:37:12.132 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 14 2024-04-06T09:37:12.133 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:37:12.133 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=57, tag_tid=9, entry_tid=1], [object_number=56, tag_tid=9, entry_tid=0], [object_number=59, tag_tid=8, entry_tid=11939], [object_number=58, tag_tid=8, entry_tid=11938]]], state=connected] 2024-04-06T09:37:12.133 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=52, tag_tid=8, entry_tid=7788], [object_number=55, tag_tid=8, entry_tid=7787], [object_number=54, tag_tid=8, entry_tid=7786], [object_number=53, tag_tid=8, entry_tid=7785]]], state=connected] 2024-04-06T09:37:12.136 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:37:12.138 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=57, tag_tid=9, entry_tid=1' 2024-04-06T09:37:12.139 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:37:12.139 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:37:12.139 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:37:12.139 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:37:12.139 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:37:12.139 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:37:12.139 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:37:12.139 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:37:12.140 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:37:12.140 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:37:12.140 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:37:12.140 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:37:12.140 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:37:12.141 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:37:12.141 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:37:12.186 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 13 2024-04-06T09:37:12.186 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 14 2024-04-06T09:37:12.186 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:37:12.186 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=57, tag_tid=9, entry_tid=1], [object_number=56, tag_tid=9, entry_tid=0], [object_number=59, tag_tid=8, entry_tid=11939], [object_number=58, tag_tid=8, entry_tid=11938]]], state=connected] 2024-04-06T09:37:12.186 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=52, tag_tid=8, entry_tid=7788], [object_number=55, tag_tid=8, entry_tid=7787], [object_number=54, tag_tid=8, entry_tid=7786], [object_number=53, tag_tid=8, entry_tid=7785]]], state=connected] 2024-04-06T09:37:12.190 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:37:12.192 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=52, tag_tid=8, entry_tid=7788' 2024-04-06T09:37:12.192 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=57, tag_tid=9, entry_tid=1' -a 'object_number=57, tag_tid=9, entry_tid=1' = 'object_number=52, tag_tid=8, entry_tid=7788' 2024-04-06T09:37:12.192 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=52, tag_tid=8, entry_tid=7788' '!=' 'object_number=53, tag_tid=8, entry_tid=7689' 2024-04-06T09:37:12.192 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:37:12.192 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=52, tag_tid=8, entry_tid=7788' = 'object_number=53, tag_tid=8, entry_tid=7689' 2024-04-06T09:37:12.192 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=52, tag_tid=8, entry_tid=7788' 2024-04-06T09:37:12.192 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=57, tag_tid=9, entry_tid=1' 2024-04-06T09:37:12.192 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:37:12.193 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:37:12.193 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=9 2024-04-06T09:37:12.194 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=52, tag_tid=8, entry_tid=7788' 2024-04-06T09:37:12.194 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:37:12.194 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:37:12.195 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=8 2024-04-06T09:37:12.195 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=57, tag_tid=9, entry_tid=1' 2024-04-06T09:37:12.195 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:37:12.196 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:37:12.196 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:37:12.197 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=52, tag_tid=8, entry_tid=7788' 2024-04-06T09:37:12.197 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:37:12.197 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:37:12.198 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=7788 2024-04-06T09:37:12.198 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 9 = 8 -a 1 -le 7788 2024-04-06T09:37:12.199 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:37:12.199 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:37:12.199 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:37:12.400 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:37:12.401 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:37:12.401 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:37:12.401 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:37:12.401 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:37:12.401 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:37:12.401 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:37:12.401 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:37:12.401 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:37:12.401 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:37:12.401 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:37:12.401 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:37:12.401 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:37:12.401 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:37:12.402 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:37:12.402 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:37:12.402 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:37:12.405 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:37:12.406 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:37:12.406 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:37:12.406 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:37:12.406 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:37:12.406 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:37:12.406 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:37:12.406 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:37:12.406 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:37:12.406 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:37:12.406 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:37:12.406 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:37:12.407 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:37:12.407 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:37:12.407 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:37:12.407 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:37:12.407 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:37:12.407 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:37:12.407 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:37:12.407 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:37:12.407 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:37:12.407 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:37:12.407 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:37:12.407 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:37:12.407 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:37:12.407 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:37:12.407 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:37:12.408 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:37:12.408 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:37:12.408 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:37:12.408 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:37:12.408 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:37:12.431 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:37:12.431 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:37:12.431 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:37:12.431 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:37:12.563 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:37:12.564 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:37:12.564 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:37:12.564 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:37:12.564 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:37:12.564 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:37:12.565 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:37:12.565 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:37:12.565 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:37:12.565 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:37:12.566 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:37:12.566 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:37:12.566 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:37:12.567 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:37:12.567 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:37:12.568 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:37:12.614 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 13 2024-04-06T09:37:12.614 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 14 2024-04-06T09:37:12.614 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:37:12.614 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=57, tag_tid=9, entry_tid=1], [object_number=56, tag_tid=9, entry_tid=0], [object_number=59, tag_tid=8, entry_tid=11939], [object_number=58, tag_tid=8, entry_tid=11938]]], state=connected] 2024-04-06T09:37:12.614 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=52, tag_tid=8, entry_tid=7860], [object_number=55, tag_tid=8, entry_tid=7859], [object_number=54, tag_tid=8, entry_tid=7858], [object_number=53, tag_tid=8, entry_tid=7857]]], state=connected] 2024-04-06T09:37:12.618 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:37:12.619 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=57, tag_tid=9, entry_tid=1' 2024-04-06T09:37:12.620 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:37:12.620 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:37:12.621 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:37:12.621 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:37:12.621 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:37:12.621 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:37:12.621 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:37:12.621 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:37:12.621 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:37:12.621 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:37:12.621 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:37:12.621 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:37:12.622 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:37:12.622 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:37:12.622 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:37:12.668 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 13 2024-04-06T09:37:12.669 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 14 2024-04-06T09:37:12.669 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:37:12.669 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=57, tag_tid=9, entry_tid=1], [object_number=56, tag_tid=9, entry_tid=0], [object_number=59, tag_tid=8, entry_tid=11939], [object_number=58, tag_tid=8, entry_tid=11938]]], state=connected] 2024-04-06T09:37:12.669 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=52, tag_tid=8, entry_tid=7860], [object_number=55, tag_tid=8, entry_tid=7859], [object_number=54, tag_tid=8, entry_tid=7858], [object_number=53, tag_tid=8, entry_tid=7857]]], state=connected] 2024-04-06T09:37:12.672 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:37:12.674 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=52, tag_tid=8, entry_tid=7860' 2024-04-06T09:37:12.674 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=57, tag_tid=9, entry_tid=1' -a 'object_number=57, tag_tid=9, entry_tid=1' = 'object_number=52, tag_tid=8, entry_tid=7860' 2024-04-06T09:37:12.674 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=52, tag_tid=8, entry_tid=7860' '!=' 'object_number=52, tag_tid=8, entry_tid=7788' 2024-04-06T09:37:12.674 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:37:12.675 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=52, tag_tid=8, entry_tid=7860' = 'object_number=52, tag_tid=8, entry_tid=7788' 2024-04-06T09:37:12.675 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=52, tag_tid=8, entry_tid=7860' 2024-04-06T09:37:12.675 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=57, tag_tid=9, entry_tid=1' 2024-04-06T09:37:12.675 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:37:12.675 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:37:12.675 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=9 2024-04-06T09:37:12.676 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=52, tag_tid=8, entry_tid=7860' 2024-04-06T09:37:12.676 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:37:12.676 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:37:12.677 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=8 2024-04-06T09:37:12.677 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=57, tag_tid=9, entry_tid=1' 2024-04-06T09:37:12.678 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:37:12.678 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:37:12.678 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:37:12.679 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=52, tag_tid=8, entry_tid=7860' 2024-04-06T09:37:12.679 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:37:12.679 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:37:12.681 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=7860 2024-04-06T09:37:12.681 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 9 = 8 -a 1 -le 7860 2024-04-06T09:37:12.681 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:37:12.681 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:37:12.681 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:37:12.883 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:37:12.883 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:37:12.883 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:37:12.883 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:37:12.883 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:37:12.883 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:37:12.883 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:37:12.883 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:37:12.883 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:37:12.883 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:37:12.883 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:37:12.884 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:37:12.884 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:37:12.884 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:37:12.884 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:37:12.884 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:37:12.884 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:37:12.886 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:37:12.886 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:37:12.886 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:37:12.886 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:37:12.886 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:37:12.886 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:37:12.886 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:37:12.886 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:37:12.886 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:37:12.886 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:37:12.886 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:37:12.886 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:37:12.886 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:37:12.887 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:37:12.887 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:37:12.887 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:37:12.887 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:37:12.887 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:37:12.887 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:37:12.888 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:37:12.888 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:37:12.888 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:37:12.888 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:37:12.888 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:37:12.888 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:37:12.888 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:37:12.888 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:37:12.888 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:37:12.888 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:37:12.888 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:37:12.888 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:37:12.888 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:37:12.912 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:37:12.913 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:37:12.913 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:37:12.913 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:37:13.044 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:37:13.045 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:37:13.045 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:37:13.045 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:37:13.045 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:37:13.045 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:37:13.046 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:37:13.046 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:37:13.046 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:37:13.046 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:37:13.046 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:37:13.046 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:37:13.047 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:37:13.048 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:37:13.048 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:37:13.048 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:37:13.105 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 13 2024-04-06T09:37:13.105 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 14 2024-04-06T09:37:13.105 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:37:13.106 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=57, tag_tid=9, entry_tid=1], [object_number=56, tag_tid=9, entry_tid=0], [object_number=59, tag_tid=8, entry_tid=11939], [object_number=58, tag_tid=8, entry_tid=11938]]], state=connected] 2024-04-06T09:37:13.106 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=52, tag_tid=8, entry_tid=7944], [object_number=55, tag_tid=8, entry_tid=7943], [object_number=54, tag_tid=8, entry_tid=7942], [object_number=53, tag_tid=8, entry_tid=7941]]], state=connected] 2024-04-06T09:37:13.109 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:37:13.110 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=57, tag_tid=9, entry_tid=1' 2024-04-06T09:37:13.112 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:37:13.112 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:37:13.112 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:37:13.112 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:37:13.112 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:37:13.112 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:37:13.112 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:37:13.112 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:37:13.112 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:37:13.113 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:37:13.113 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:37:13.113 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:37:13.113 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:37:13.113 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:37:13.113 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:37:13.160 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 13 2024-04-06T09:37:13.160 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 14 2024-04-06T09:37:13.160 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:37:13.160 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=57, tag_tid=9, entry_tid=1], [object_number=56, tag_tid=9, entry_tid=0], [object_number=59, tag_tid=8, entry_tid=11939], [object_number=58, tag_tid=8, entry_tid=11938]]], state=connected] 2024-04-06T09:37:13.160 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=52, tag_tid=8, entry_tid=7944], [object_number=55, tag_tid=8, entry_tid=7943], [object_number=54, tag_tid=8, entry_tid=7942], [object_number=53, tag_tid=8, entry_tid=7941]]], state=connected] 2024-04-06T09:37:13.164 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:37:13.166 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=52, tag_tid=8, entry_tid=7944' 2024-04-06T09:37:13.166 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=57, tag_tid=9, entry_tid=1' -a 'object_number=57, tag_tid=9, entry_tid=1' = 'object_number=52, tag_tid=8, entry_tid=7944' 2024-04-06T09:37:13.166 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=52, tag_tid=8, entry_tid=7944' '!=' 'object_number=52, tag_tid=8, entry_tid=7860' 2024-04-06T09:37:13.166 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:37:13.166 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=52, tag_tid=8, entry_tid=7944' = 'object_number=52, tag_tid=8, entry_tid=7860' 2024-04-06T09:37:13.166 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=52, tag_tid=8, entry_tid=7944' 2024-04-06T09:37:13.166 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=57, tag_tid=9, entry_tid=1' 2024-04-06T09:37:13.167 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:37:13.167 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:37:13.167 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=9 2024-04-06T09:37:13.168 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=52, tag_tid=8, entry_tid=7944' 2024-04-06T09:37:13.168 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:37:13.168 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:37:13.169 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=8 2024-04-06T09:37:13.169 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=57, tag_tid=9, entry_tid=1' 2024-04-06T09:37:13.170 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:37:13.170 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:37:13.171 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:37:13.171 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=52, tag_tid=8, entry_tid=7944' 2024-04-06T09:37:13.171 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:37:13.171 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:37:13.173 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=7944 2024-04-06T09:37:13.173 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 9 = 8 -a 1 -le 7944 2024-04-06T09:37:13.173 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:37:13.173 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:37:13.173 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:37:13.375 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:37:13.375 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:37:13.375 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:37:13.375 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:37:13.375 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:37:13.375 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:37:13.375 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:37:13.375 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:37:13.375 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:37:13.375 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:37:13.375 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:37:13.375 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:37:13.375 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:37:13.375 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:37:13.376 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:37:13.376 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:37:13.376 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:37:13.379 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:37:13.379 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:37:13.379 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:37:13.379 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:37:13.379 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:37:13.379 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:37:13.380 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:37:13.380 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:37:13.380 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:37:13.380 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:37:13.380 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:37:13.380 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:37:13.380 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:37:13.380 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:37:13.380 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:37:13.380 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:37:13.380 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:37:13.380 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:37:13.380 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:37:13.380 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:37:13.381 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:37:13.381 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:37:13.381 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:37:13.381 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:37:13.381 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:37:13.381 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:37:13.381 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:37:13.381 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:37:13.381 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:37:13.381 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:37:13.381 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:37:13.382 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:37:13.406 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:37:13.406 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:37:13.406 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:37:13.407 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:37:13.537 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:37:13.538 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:37:13.539 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:37:13.539 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:37:13.539 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:37:13.539 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:37:13.539 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:37:13.539 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:37:13.539 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:37:13.539 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:37:13.540 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:37:13.540 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:37:13.540 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:37:13.541 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:37:13.542 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:37:13.542 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:37:13.588 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 13 2024-04-06T09:37:13.588 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 14 2024-04-06T09:37:13.589 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:37:13.589 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=57, tag_tid=9, entry_tid=1], [object_number=56, tag_tid=9, entry_tid=0], [object_number=59, tag_tid=8, entry_tid=11939], [object_number=58, tag_tid=8, entry_tid=11938]]], state=connected] 2024-04-06T09:37:13.589 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=53, tag_tid=8, entry_tid=8033], [object_number=52, tag_tid=8, entry_tid=8032], [object_number=55, tag_tid=8, entry_tid=8031], [object_number=54, tag_tid=8, entry_tid=8030]]], state=connected] 2024-04-06T09:37:13.592 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:37:13.593 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=57, tag_tid=9, entry_tid=1' 2024-04-06T09:37:13.595 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:37:13.595 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:37:13.595 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:37:13.595 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:37:13.595 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:37:13.595 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:37:13.595 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:37:13.595 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:37:13.595 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:37:13.595 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:37:13.595 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:37:13.596 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:37:13.596 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:37:13.596 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:37:13.596 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:37:13.641 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 13 2024-04-06T09:37:13.641 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 14 2024-04-06T09:37:13.642 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:37:13.642 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=57, tag_tid=9, entry_tid=1], [object_number=56, tag_tid=9, entry_tid=0], [object_number=59, tag_tid=8, entry_tid=11939], [object_number=58, tag_tid=8, entry_tid=11938]]], state=connected] 2024-04-06T09:37:13.642 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=53, tag_tid=8, entry_tid=8033], [object_number=52, tag_tid=8, entry_tid=8032], [object_number=55, tag_tid=8, entry_tid=8031], [object_number=54, tag_tid=8, entry_tid=8030]]], state=connected] 2024-04-06T09:37:13.645 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:37:13.647 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=53, tag_tid=8, entry_tid=8033' 2024-04-06T09:37:13.647 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=57, tag_tid=9, entry_tid=1' -a 'object_number=57, tag_tid=9, entry_tid=1' = 'object_number=53, tag_tid=8, entry_tid=8033' 2024-04-06T09:37:13.647 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=53, tag_tid=8, entry_tid=8033' '!=' 'object_number=52, tag_tid=8, entry_tid=7944' 2024-04-06T09:37:13.647 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:37:13.647 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=53, tag_tid=8, entry_tid=8033' = 'object_number=52, tag_tid=8, entry_tid=7944' 2024-04-06T09:37:13.647 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=53, tag_tid=8, entry_tid=8033' 2024-04-06T09:37:13.648 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=57, tag_tid=9, entry_tid=1' 2024-04-06T09:37:13.648 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:37:13.648 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:37:13.648 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=9 2024-04-06T09:37:13.649 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=53, tag_tid=8, entry_tid=8033' 2024-04-06T09:37:13.649 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:37:13.649 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:37:13.650 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=8 2024-04-06T09:37:13.651 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=57, tag_tid=9, entry_tid=1' 2024-04-06T09:37:13.651 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:37:13.651 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:37:13.651 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:37:13.652 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=53, tag_tid=8, entry_tid=8033' 2024-04-06T09:37:13.652 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:37:13.652 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:37:13.653 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=8033 2024-04-06T09:37:13.653 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 9 = 8 -a 1 -le 8033 2024-04-06T09:37:13.654 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:37:13.654 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:37:13.654 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:37:13.855 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:37:13.856 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:37:13.856 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:37:13.856 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:37:13.856 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:37:13.856 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:37:13.856 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:37:13.856 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:37:13.856 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:37:13.856 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:37:13.856 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:37:13.856 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:37:13.857 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:37:13.857 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:37:13.857 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:37:13.857 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:37:13.857 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:37:13.859 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:37:13.859 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:37:13.859 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:37:13.860 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:37:13.860 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:37:13.860 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:37:13.860 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:37:13.860 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:37:13.860 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:37:13.860 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:37:13.860 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:37:13.860 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:37:13.860 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:37:13.860 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:37:13.860 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:37:13.861 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:37:13.861 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:37:13.861 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:37:13.861 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:37:13.861 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:37:13.861 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:37:13.861 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:37:13.861 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:37:13.861 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:37:13.861 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:37:13.861 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:37:13.861 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:37:13.861 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:37:13.861 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:37:13.861 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:37:13.862 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:37:13.862 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:37:13.886 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:37:13.886 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:37:13.886 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:37:13.886 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:37:14.023 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:37:14.024 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:37:14.024 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:37:14.024 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:37:14.024 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:37:14.024 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:37:14.024 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:37:14.024 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:37:14.025 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:37:14.025 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:37:14.025 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:37:14.025 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:37:14.025 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:37:14.026 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:37:14.027 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:37:14.027 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:37:14.074 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 13 2024-04-06T09:37:14.074 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 14 2024-04-06T09:37:14.074 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:37:14.074 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=57, tag_tid=9, entry_tid=1], [object_number=56, tag_tid=9, entry_tid=0], [object_number=59, tag_tid=8, entry_tid=11939], [object_number=58, tag_tid=8, entry_tid=11938]]], state=connected] 2024-04-06T09:37:14.074 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=53, tag_tid=8, entry_tid=8129], [object_number=52, tag_tid=8, entry_tid=8128], [object_number=55, tag_tid=8, entry_tid=8127], [object_number=54, tag_tid=8, entry_tid=8126]]], state=connected] 2024-04-06T09:37:14.077 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:37:14.079 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=57, tag_tid=9, entry_tid=1' 2024-04-06T09:37:14.080 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:37:14.080 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:37:14.080 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:37:14.080 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:37:14.080 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:37:14.080 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:37:14.081 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:37:14.081 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:37:14.081 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:37:14.081 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:37:14.081 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:37:14.081 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:37:14.081 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:37:14.082 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:37:14.082 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:37:14.127 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 13 2024-04-06T09:37:14.128 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 14 2024-04-06T09:37:14.128 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:37:14.128 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=57, tag_tid=9, entry_tid=1], [object_number=56, tag_tid=9, entry_tid=0], [object_number=59, tag_tid=8, entry_tid=11939], [object_number=58, tag_tid=8, entry_tid=11938]]], state=connected] 2024-04-06T09:37:14.128 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=53, tag_tid=8, entry_tid=8129], [object_number=52, tag_tid=8, entry_tid=8128], [object_number=55, tag_tid=8, entry_tid=8127], [object_number=54, tag_tid=8, entry_tid=8126]]], state=connected] 2024-04-06T09:37:14.131 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:37:14.133 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=53, tag_tid=8, entry_tid=8129' 2024-04-06T09:37:14.133 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=57, tag_tid=9, entry_tid=1' -a 'object_number=57, tag_tid=9, entry_tid=1' = 'object_number=53, tag_tid=8, entry_tid=8129' 2024-04-06T09:37:14.133 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=53, tag_tid=8, entry_tid=8129' '!=' 'object_number=53, tag_tid=8, entry_tid=8033' 2024-04-06T09:37:14.133 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:37:14.133 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=53, tag_tid=8, entry_tid=8129' = 'object_number=53, tag_tid=8, entry_tid=8033' 2024-04-06T09:37:14.134 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=53, tag_tid=8, entry_tid=8129' 2024-04-06T09:37:14.134 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=57, tag_tid=9, entry_tid=1' 2024-04-06T09:37:14.134 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:37:14.134 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:37:14.134 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=9 2024-04-06T09:37:14.135 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=53, tag_tid=8, entry_tid=8129' 2024-04-06T09:37:14.135 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:37:14.135 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:37:14.136 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=8 2024-04-06T09:37:14.137 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=57, tag_tid=9, entry_tid=1' 2024-04-06T09:37:14.137 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:37:14.137 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:37:14.138 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:37:14.138 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=53, tag_tid=8, entry_tid=8129' 2024-04-06T09:37:14.139 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:37:14.139 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:37:14.140 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=8129 2024-04-06T09:37:14.140 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 9 = 8 -a 1 -le 8129 2024-04-06T09:37:14.140 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:37:14.140 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:37:14.140 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:37:14.341 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:37:14.342 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:37:14.342 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:37:14.342 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:37:14.342 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:37:14.342 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:37:14.342 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:37:14.342 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:37:14.342 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:37:14.342 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:37:14.342 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:37:14.342 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:37:14.342 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:37:14.343 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:37:14.343 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:37:14.343 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:37:14.343 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:37:14.347 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:37:14.347 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:37:14.347 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:37:14.347 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:37:14.348 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:37:14.348 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:37:14.348 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:37:14.348 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:37:14.348 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:37:14.348 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:37:14.348 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:37:14.348 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:37:14.348 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:37:14.348 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:37:14.348 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:37:14.348 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:37:14.348 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:37:14.348 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:37:14.348 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:37:14.349 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:37:14.349 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:37:14.349 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:37:14.349 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:37:14.349 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:37:14.349 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:37:14.349 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:37:14.349 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:37:14.349 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:37:14.349 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:37:14.349 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:37:14.349 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:37:14.349 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:37:14.373 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:37:14.373 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:37:14.374 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:37:14.374 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:37:14.503 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:37:14.504 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:37:14.504 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:37:14.504 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:37:14.504 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:37:14.504 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:37:14.505 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:37:14.505 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:37:14.505 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:37:14.505 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:37:14.505 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:37:14.505 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:37:14.506 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:37:14.508 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:37:14.509 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:37:14.509 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:37:14.564 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 13 2024-04-06T09:37:14.564 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 14 2024-04-06T09:37:14.564 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:37:14.564 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=57, tag_tid=9, entry_tid=1], [object_number=56, tag_tid=9, entry_tid=0], [object_number=59, tag_tid=8, entry_tid=11939], [object_number=58, tag_tid=8, entry_tid=11938]]], state=connected] 2024-04-06T09:37:14.564 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=54, tag_tid=8, entry_tid=8226], [object_number=53, tag_tid=8, entry_tid=8225], [object_number=52, tag_tid=8, entry_tid=8224], [object_number=55, tag_tid=8, entry_tid=8223]]], state=connected] 2024-04-06T09:37:14.567 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:37:14.569 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=57, tag_tid=9, entry_tid=1' 2024-04-06T09:37:14.570 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:37:14.570 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:37:14.570 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:37:14.570 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:37:14.570 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:37:14.570 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:37:14.570 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:37:14.571 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:37:14.571 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:37:14.571 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:37:14.571 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:37:14.571 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:37:14.571 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:37:14.571 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:37:14.572 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:37:14.617 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 13 2024-04-06T09:37:14.617 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 14 2024-04-06T09:37:14.617 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:37:14.617 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=57, tag_tid=9, entry_tid=1], [object_number=56, tag_tid=9, entry_tid=0], [object_number=59, tag_tid=8, entry_tid=11939], [object_number=58, tag_tid=8, entry_tid=11938]]], state=connected] 2024-04-06T09:37:14.618 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=54, tag_tid=8, entry_tid=8226], [object_number=53, tag_tid=8, entry_tid=8225], [object_number=52, tag_tid=8, entry_tid=8224], [object_number=55, tag_tid=8, entry_tid=8223]]], state=connected] 2024-04-06T09:37:14.621 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:37:14.623 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=54, tag_tid=8, entry_tid=8226' 2024-04-06T09:37:14.623 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=57, tag_tid=9, entry_tid=1' -a 'object_number=57, tag_tid=9, entry_tid=1' = 'object_number=54, tag_tid=8, entry_tid=8226' 2024-04-06T09:37:14.623 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=54, tag_tid=8, entry_tid=8226' '!=' 'object_number=53, tag_tid=8, entry_tid=8129' 2024-04-06T09:37:14.623 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:37:14.623 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=54, tag_tid=8, entry_tid=8226' = 'object_number=53, tag_tid=8, entry_tid=8129' 2024-04-06T09:37:14.623 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=54, tag_tid=8, entry_tid=8226' 2024-04-06T09:37:14.623 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=57, tag_tid=9, entry_tid=1' 2024-04-06T09:37:14.623 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:37:14.623 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:37:14.624 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=9 2024-04-06T09:37:14.625 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=54, tag_tid=8, entry_tid=8226' 2024-04-06T09:37:14.625 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:37:14.625 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:37:14.626 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=8 2024-04-06T09:37:14.626 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=57, tag_tid=9, entry_tid=1' 2024-04-06T09:37:14.626 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:37:14.627 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:37:14.627 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:37:14.628 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=54, tag_tid=8, entry_tid=8226' 2024-04-06T09:37:14.628 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:37:14.628 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:37:14.629 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=8226 2024-04-06T09:37:14.629 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 9 = 8 -a 1 -le 8226 2024-04-06T09:37:14.630 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:37:14.630 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:37:14.630 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:37:14.831 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:37:14.831 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:37:14.831 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:37:14.832 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:37:14.832 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:37:14.832 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:37:14.832 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:37:14.832 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:37:14.832 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:37:14.832 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:37:14.832 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:37:14.832 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:37:14.832 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:37:14.832 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:37:14.832 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:37:14.833 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:37:14.833 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:37:14.835 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:37:14.835 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:37:14.835 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:37:14.835 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:37:14.835 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:37:14.835 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:37:14.835 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:37:14.836 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:37:14.836 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:37:14.836 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:37:14.836 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:37:14.836 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:37:14.836 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:37:14.836 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:37:14.836 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:37:14.836 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:37:14.838 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:37:14.838 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:37:14.838 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:37:14.838 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:37:14.838 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:37:14.838 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:37:14.838 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:37:14.838 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:37:14.838 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:37:14.838 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:37:14.839 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:37:14.839 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:37:14.839 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:37:14.839 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:37:14.839 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:37:14.839 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:37:14.862 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:37:14.863 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:37:14.863 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:37:14.863 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:37:14.998 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:37:14.999 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:37:14.999 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:37:14.999 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:37:14.999 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:37:14.999 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:37:14.999 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:37:14.999 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:37:14.999 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:37:15.000 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:37:15.000 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:37:15.000 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:37:15.000 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:37:15.002 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:37:15.003 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:37:15.003 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:37:15.049 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 13 2024-04-06T09:37:15.049 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 14 2024-04-06T09:37:15.049 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:37:15.049 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=57, tag_tid=9, entry_tid=1], [object_number=56, tag_tid=9, entry_tid=0], [object_number=59, tag_tid=8, entry_tid=11939], [object_number=58, tag_tid=8, entry_tid=11938]]], state=connected] 2024-04-06T09:37:15.049 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=54, tag_tid=8, entry_tid=8322], [object_number=53, tag_tid=8, entry_tid=8321], [object_number=52, tag_tid=8, entry_tid=8320], [object_number=55, tag_tid=8, entry_tid=8319]]], state=connected] 2024-04-06T09:37:15.052 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:37:15.054 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=57, tag_tid=9, entry_tid=1' 2024-04-06T09:37:15.055 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:37:15.055 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:37:15.055 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:37:15.055 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:37:15.055 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:37:15.055 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:37:15.055 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:37:15.055 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:37:15.055 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:37:15.055 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:37:15.055 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:37:15.055 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:37:15.056 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:37:15.056 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:37:15.056 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:37:15.101 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 13 2024-04-06T09:37:15.101 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 14 2024-04-06T09:37:15.101 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:37:15.101 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=57, tag_tid=9, entry_tid=1], [object_number=56, tag_tid=9, entry_tid=0], [object_number=59, tag_tid=8, entry_tid=11939], [object_number=58, tag_tid=8, entry_tid=11938]]], state=connected] 2024-04-06T09:37:15.101 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=54, tag_tid=8, entry_tid=8322], [object_number=53, tag_tid=8, entry_tid=8321], [object_number=52, tag_tid=8, entry_tid=8320], [object_number=55, tag_tid=8, entry_tid=8319]]], state=connected] 2024-04-06T09:37:15.104 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:37:15.106 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=54, tag_tid=8, entry_tid=8322' 2024-04-06T09:37:15.106 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=57, tag_tid=9, entry_tid=1' -a 'object_number=57, tag_tid=9, entry_tid=1' = 'object_number=54, tag_tid=8, entry_tid=8322' 2024-04-06T09:37:15.106 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=54, tag_tid=8, entry_tid=8322' '!=' 'object_number=54, tag_tid=8, entry_tid=8226' 2024-04-06T09:37:15.106 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:37:15.107 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=54, tag_tid=8, entry_tid=8322' = 'object_number=54, tag_tid=8, entry_tid=8226' 2024-04-06T09:37:15.107 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=54, tag_tid=8, entry_tid=8322' 2024-04-06T09:37:15.107 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:37:15.107 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=57, tag_tid=9, entry_tid=1' 2024-04-06T09:37:15.107 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:37:15.108 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=9 2024-04-06T09:37:15.108 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=54, tag_tid=8, entry_tid=8322' 2024-04-06T09:37:15.108 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:37:15.108 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:37:15.109 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=8 2024-04-06T09:37:15.110 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=57, tag_tid=9, entry_tid=1' 2024-04-06T09:37:15.110 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:37:15.110 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:37:15.111 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:37:15.112 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=54, tag_tid=8, entry_tid=8322' 2024-04-06T09:37:15.112 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:37:15.112 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:37:15.113 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=8322 2024-04-06T09:37:15.113 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 9 = 8 -a 1 -le 8322 2024-04-06T09:37:15.113 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:37:15.113 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:37:15.113 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:37:15.315 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:37:15.315 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:37:15.315 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:37:15.315 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:37:15.315 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:37:15.315 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:37:15.315 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:37:15.315 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:37:15.315 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:37:15.315 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:37:15.315 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:37:15.316 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:37:15.316 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:37:15.316 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:37:15.316 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:37:15.317 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:37:15.317 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:37:15.320 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:37:15.320 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:37:15.320 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:37:15.321 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:37:15.321 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:37:15.321 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:37:15.321 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:37:15.321 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:37:15.321 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:37:15.321 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:37:15.321 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:37:15.321 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:37:15.321 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:37:15.321 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:37:15.321 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:37:15.321 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:37:15.321 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:37:15.322 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:37:15.322 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:37:15.322 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:37:15.322 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:37:15.322 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:37:15.322 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:37:15.322 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:37:15.322 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:37:15.322 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:37:15.322 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:37:15.322 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:37:15.322 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:37:15.322 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:37:15.322 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:37:15.323 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:37:15.344 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:37:15.345 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:37:15.345 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:37:15.345 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:37:15.470 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:37:15.472 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:37:15.472 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:37:15.472 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:37:15.472 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:37:15.472 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:37:15.472 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:37:15.472 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:37:15.473 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:37:15.473 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:37:15.473 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:37:15.473 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:37:15.473 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:37:15.474 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:37:15.475 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:37:15.475 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:37:15.519 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 13 2024-04-06T09:37:15.519 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 14 2024-04-06T09:37:15.519 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:37:15.520 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=57, tag_tid=9, entry_tid=1], [object_number=56, tag_tid=9, entry_tid=0], [object_number=59, tag_tid=8, entry_tid=11939], [object_number=58, tag_tid=8, entry_tid=11938]]], state=connected] 2024-04-06T09:37:15.520 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=54, tag_tid=8, entry_tid=8414], [object_number=53, tag_tid=8, entry_tid=8413], [object_number=52, tag_tid=8, entry_tid=8412], [object_number=55, tag_tid=8, entry_tid=8411]]], state=connected] 2024-04-06T09:37:15.523 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:37:15.524 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=57, tag_tid=9, entry_tid=1' 2024-04-06T09:37:15.525 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:37:15.526 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:37:15.526 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:37:15.526 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:37:15.526 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:37:15.526 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:37:15.526 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:37:15.526 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:37:15.526 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:37:15.526 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:37:15.526 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:37:15.526 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:37:15.527 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:37:15.527 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:37:15.527 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:37:15.572 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 13 2024-04-06T09:37:15.572 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 14 2024-04-06T09:37:15.572 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:37:15.572 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=57, tag_tid=9, entry_tid=1], [object_number=56, tag_tid=9, entry_tid=0], [object_number=59, tag_tid=8, entry_tid=11939], [object_number=58, tag_tid=8, entry_tid=11938]]], state=connected] 2024-04-06T09:37:15.573 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=54, tag_tid=8, entry_tid=8414], [object_number=53, tag_tid=8, entry_tid=8413], [object_number=52, tag_tid=8, entry_tid=8412], [object_number=55, tag_tid=8, entry_tid=8411]]], state=connected] 2024-04-06T09:37:15.576 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:37:15.578 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=54, tag_tid=8, entry_tid=8414' 2024-04-06T09:37:15.578 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=57, tag_tid=9, entry_tid=1' -a 'object_number=57, tag_tid=9, entry_tid=1' = 'object_number=54, tag_tid=8, entry_tid=8414' 2024-04-06T09:37:15.578 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=54, tag_tid=8, entry_tid=8414' '!=' 'object_number=54, tag_tid=8, entry_tid=8322' 2024-04-06T09:37:15.578 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:37:15.578 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=54, tag_tid=8, entry_tid=8414' = 'object_number=54, tag_tid=8, entry_tid=8322' 2024-04-06T09:37:15.578 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=54, tag_tid=8, entry_tid=8414' 2024-04-06T09:37:15.579 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=57, tag_tid=9, entry_tid=1' 2024-04-06T09:37:15.579 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:37:15.579 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:37:15.579 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=9 2024-04-06T09:37:15.580 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=54, tag_tid=8, entry_tid=8414' 2024-04-06T09:37:15.580 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:37:15.580 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:37:15.581 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=8 2024-04-06T09:37:15.582 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=57, tag_tid=9, entry_tid=1' 2024-04-06T09:37:15.582 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:37:15.582 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:37:15.583 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:37:15.583 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=54, tag_tid=8, entry_tid=8414' 2024-04-06T09:37:15.583 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:37:15.583 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:37:15.585 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=8414 2024-04-06T09:37:15.585 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 9 = 8 -a 1 -le 8414 2024-04-06T09:37:15.585 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:37:15.585 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:37:15.585 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:37:15.786 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:37:15.787 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:37:15.787 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:37:15.787 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:37:15.787 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:37:15.787 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:37:15.787 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:37:15.787 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:37:15.787 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:37:15.787 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:37:15.787 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:37:15.787 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:37:15.787 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:37:15.787 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:37:15.787 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:37:15.788 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:37:15.788 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:37:15.792 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:37:15.792 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:37:15.792 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:37:15.792 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:37:15.792 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:37:15.792 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:37:15.792 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:37:15.792 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:37:15.793 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:37:15.793 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:37:15.793 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:37:15.793 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:37:15.793 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:37:15.793 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:37:15.793 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:37:15.793 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:37:15.793 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:37:15.793 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:37:15.793 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:37:15.793 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:37:15.793 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:37:15.793 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:37:15.793 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:37:15.794 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:37:15.794 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:37:15.794 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:37:15.794 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:37:15.794 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:37:15.794 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:37:15.794 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:37:15.794 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:37:15.794 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:37:15.817 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:37:15.817 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:37:15.817 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:37:15.817 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:37:15.944 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:37:15.945 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:37:15.945 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:37:15.945 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:37:15.945 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:37:15.946 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:37:15.946 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:37:15.946 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:37:15.946 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:37:15.946 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:37:15.946 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:37:15.946 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:37:15.946 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:37:15.947 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:37:15.947 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:37:15.948 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:37:15.993 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 13 2024-04-06T09:37:15.993 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 14 2024-04-06T09:37:15.993 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:37:15.993 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=57, tag_tid=9, entry_tid=1], [object_number=56, tag_tid=9, entry_tid=0], [object_number=59, tag_tid=8, entry_tid=11939], [object_number=58, tag_tid=8, entry_tid=11938]]], state=connected] 2024-04-06T09:37:15.993 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=52, tag_tid=8, entry_tid=8496], [object_number=55, tag_tid=8, entry_tid=8495], [object_number=54, tag_tid=8, entry_tid=8494], [object_number=53, tag_tid=8, entry_tid=8493]]], state=connected] 2024-04-06T09:37:15.996 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:37:15.998 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=57, tag_tid=9, entry_tid=1' 2024-04-06T09:37:15.999 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:37:15.999 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:37:15.999 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:37:15.999 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:37:15.999 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:37:15.999 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:37:15.999 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:37:15.999 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:37:15.999 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:37:16.000 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:37:16.000 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:37:16.000 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:37:16.000 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:37:16.000 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:37:16.000 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:37:16.046 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 13 2024-04-06T09:37:16.046 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 14 2024-04-06T09:37:16.046 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:37:16.047 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=57, tag_tid=9, entry_tid=1], [object_number=56, tag_tid=9, entry_tid=0], [object_number=59, tag_tid=8, entry_tid=11939], [object_number=58, tag_tid=8, entry_tid=11938]]], state=connected] 2024-04-06T09:37:16.047 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=52, tag_tid=8, entry_tid=8496], [object_number=55, tag_tid=8, entry_tid=8495], [object_number=54, tag_tid=8, entry_tid=8494], [object_number=53, tag_tid=8, entry_tid=8493]]], state=connected] 2024-04-06T09:37:16.050 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:37:16.052 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=52, tag_tid=8, entry_tid=8496' 2024-04-06T09:37:16.052 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=57, tag_tid=9, entry_tid=1' -a 'object_number=57, tag_tid=9, entry_tid=1' = 'object_number=52, tag_tid=8, entry_tid=8496' 2024-04-06T09:37:16.052 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=52, tag_tid=8, entry_tid=8496' '!=' 'object_number=54, tag_tid=8, entry_tid=8414' 2024-04-06T09:37:16.052 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:37:16.052 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=52, tag_tid=8, entry_tid=8496' = 'object_number=54, tag_tid=8, entry_tid=8414' 2024-04-06T09:37:16.053 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=52, tag_tid=8, entry_tid=8496' 2024-04-06T09:37:16.053 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=57, tag_tid=9, entry_tid=1' 2024-04-06T09:37:16.053 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:37:16.053 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:37:16.054 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=9 2024-04-06T09:37:16.054 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=52, tag_tid=8, entry_tid=8496' 2024-04-06T09:37:16.054 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:37:16.054 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:37:16.055 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=8 2024-04-06T09:37:16.056 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=57, tag_tid=9, entry_tid=1' 2024-04-06T09:37:16.056 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:37:16.056 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:37:16.057 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:37:16.058 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=52, tag_tid=8, entry_tid=8496' 2024-04-06T09:37:16.058 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:37:16.058 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:37:16.059 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=8496 2024-04-06T09:37:16.059 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 9 = 8 -a 1 -le 8496 2024-04-06T09:37:16.059 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:37:16.059 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:37:16.059 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:37:16.261 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:37:16.261 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:37:16.261 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:37:16.261 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:37:16.261 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:37:16.261 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:37:16.261 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:37:16.262 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:37:16.262 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:37:16.262 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:37:16.262 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:37:16.262 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:37:16.262 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:37:16.262 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:37:16.262 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:37:16.263 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:37:16.263 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:37:16.265 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:37:16.265 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:37:16.265 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:37:16.265 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:37:16.266 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:37:16.266 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:37:16.266 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:37:16.266 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:37:16.266 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:37:16.266 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:37:16.266 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:37:16.266 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:37:16.266 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:37:16.266 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:37:16.266 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:37:16.266 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:37:16.267 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:37:16.268 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:37:16.268 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:37:16.268 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:37:16.268 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:37:16.268 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:37:16.268 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:37:16.268 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:37:16.268 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:37:16.268 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:37:16.268 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:37:16.268 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:37:16.268 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:37:16.268 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:37:16.268 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:37:16.269 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:37:16.293 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:37:16.293 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:37:16.293 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:37:16.294 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:37:16.420 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:37:16.421 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:37:16.421 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:37:16.421 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:37:16.421 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:37:16.422 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:37:16.422 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:37:16.422 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:37:16.422 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:37:16.422 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:37:16.422 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:37:16.422 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:37:16.423 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:37:16.424 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:37:16.424 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:37:16.424 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:37:16.479 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 13 2024-04-06T09:37:16.479 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 14 2024-04-06T09:37:16.479 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:37:16.479 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=57, tag_tid=9, entry_tid=1], [object_number=56, tag_tid=9, entry_tid=0], [object_number=59, tag_tid=8, entry_tid=11939], [object_number=58, tag_tid=8, entry_tid=11938]]], state=connected] 2024-04-06T09:37:16.480 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=55, tag_tid=8, entry_tid=8595], [object_number=54, tag_tid=8, entry_tid=8594], [object_number=53, tag_tid=8, entry_tid=8593], [object_number=52, tag_tid=8, entry_tid=8592]]], state=connected] 2024-04-06T09:37:16.483 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:37:16.484 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=57, tag_tid=9, entry_tid=1' 2024-04-06T09:37:16.485 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:37:16.485 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:37:16.485 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:37:16.485 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:37:16.485 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:37:16.486 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:37:16.486 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:37:16.486 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:37:16.486 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:37:16.486 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:37:16.486 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:37:16.486 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:37:16.486 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:37:16.487 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:37:16.487 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:37:16.533 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 13 2024-04-06T09:37:16.533 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 14 2024-04-06T09:37:16.533 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:37:16.533 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=57, tag_tid=9, entry_tid=1], [object_number=56, tag_tid=9, entry_tid=0], [object_number=59, tag_tid=8, entry_tid=11939], [object_number=58, tag_tid=8, entry_tid=11938]]], state=connected] 2024-04-06T09:37:16.533 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=55, tag_tid=8, entry_tid=8595], [object_number=54, tag_tid=8, entry_tid=8594], [object_number=53, tag_tid=8, entry_tid=8593], [object_number=52, tag_tid=8, entry_tid=8592]]], state=connected] 2024-04-06T09:37:16.538 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:37:16.538 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=55, tag_tid=8, entry_tid=8595' 2024-04-06T09:37:16.538 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=57, tag_tid=9, entry_tid=1' -a 'object_number=57, tag_tid=9, entry_tid=1' = 'object_number=55, tag_tid=8, entry_tid=8595' 2024-04-06T09:37:16.539 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=55, tag_tid=8, entry_tid=8595' '!=' 'object_number=52, tag_tid=8, entry_tid=8496' 2024-04-06T09:37:16.539 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:37:16.539 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=55, tag_tid=8, entry_tid=8595' = 'object_number=52, tag_tid=8, entry_tid=8496' 2024-04-06T09:37:16.539 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=55, tag_tid=8, entry_tid=8595' 2024-04-06T09:37:16.539 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=57, tag_tid=9, entry_tid=1' 2024-04-06T09:37:16.539 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:37:16.539 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:37:16.539 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=9 2024-04-06T09:37:16.540 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=55, tag_tid=8, entry_tid=8595' 2024-04-06T09:37:16.540 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:37:16.540 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:37:16.541 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=8 2024-04-06T09:37:16.542 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=57, tag_tid=9, entry_tid=1' 2024-04-06T09:37:16.542 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:37:16.542 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:37:16.543 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:37:16.543 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=55, tag_tid=8, entry_tid=8595' 2024-04-06T09:37:16.544 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:37:16.544 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:37:16.545 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=8595 2024-04-06T09:37:16.545 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 9 = 8 -a 1 -le 8595 2024-04-06T09:37:16.545 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:37:16.545 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:37:16.545 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:37:16.747 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:37:16.747 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:37:16.747 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:37:16.747 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:37:16.747 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:37:16.747 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:37:16.747 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:37:16.747 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:37:16.747 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:37:16.747 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:37:16.747 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:37:16.747 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:37:16.747 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:37:16.748 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:37:16.748 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:37:16.748 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:37:16.749 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:37:16.752 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:37:16.752 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:37:16.752 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:37:16.752 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:37:16.752 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:37:16.752 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:37:16.752 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:37:16.752 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:37:16.752 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:37:16.752 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:37:16.752 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:37:16.752 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:37:16.752 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:37:16.752 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:37:16.753 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:37:16.753 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:37:16.753 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:37:16.753 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:37:16.753 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:37:16.753 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:37:16.753 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:37:16.753 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:37:16.753 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:37:16.753 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:37:16.753 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:37:16.753 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:37:16.753 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:37:16.753 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:37:16.753 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:37:16.754 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:37:16.754 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:37:16.754 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:37:16.779 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:37:16.779 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:37:16.779 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:37:16.779 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:37:16.907 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:37:16.908 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:37:16.909 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:37:16.909 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:37:16.909 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:37:16.909 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:37:16.909 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:37:16.909 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:37:16.909 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:37:16.909 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:37:16.910 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:37:16.910 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:37:16.910 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:37:16.911 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:37:16.911 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:37:16.911 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:37:16.964 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 13 2024-04-06T09:37:16.964 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 14 2024-04-06T09:37:16.964 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:37:16.964 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=57, tag_tid=9, entry_tid=1], [object_number=56, tag_tid=9, entry_tid=0], [object_number=59, tag_tid=8, entry_tid=11939], [object_number=58, tag_tid=8, entry_tid=11938]]], state=connected] 2024-04-06T09:37:16.964 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=53, tag_tid=8, entry_tid=8713], [object_number=52, tag_tid=8, entry_tid=8712], [object_number=55, tag_tid=8, entry_tid=8711], [object_number=54, tag_tid=8, entry_tid=8710]]], state=connected] 2024-04-06T09:37:16.967 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:37:16.969 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=57, tag_tid=9, entry_tid=1' 2024-04-06T09:37:16.970 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:37:16.970 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:37:16.970 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:37:16.971 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:37:16.971 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:37:16.971 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:37:16.971 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:37:16.971 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:37:16.971 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:37:16.971 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:37:16.971 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:37:16.971 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:37:16.972 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:37:16.972 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:37:16.972 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:37:17.021 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 13 2024-04-06T09:37:17.021 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 14 2024-04-06T09:37:17.021 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:37:17.021 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=57, tag_tid=9, entry_tid=1], [object_number=56, tag_tid=9, entry_tid=0], [object_number=59, tag_tid=8, entry_tid=11939], [object_number=58, tag_tid=8, entry_tid=11938]]], state=connected] 2024-04-06T09:37:17.022 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=53, tag_tid=8, entry_tid=8713], [object_number=52, tag_tid=8, entry_tid=8712], [object_number=55, tag_tid=8, entry_tid=8711], [object_number=54, tag_tid=8, entry_tid=8710]]], state=connected] 2024-04-06T09:37:17.025 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:37:17.027 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=53, tag_tid=8, entry_tid=8713' 2024-04-06T09:37:17.028 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=57, tag_tid=9, entry_tid=1' -a 'object_number=57, tag_tid=9, entry_tid=1' = 'object_number=53, tag_tid=8, entry_tid=8713' 2024-04-06T09:37:17.028 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=53, tag_tid=8, entry_tid=8713' '!=' 'object_number=55, tag_tid=8, entry_tid=8595' 2024-04-06T09:37:17.028 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:37:17.028 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=53, tag_tid=8, entry_tid=8713' = 'object_number=55, tag_tid=8, entry_tid=8595' 2024-04-06T09:37:17.028 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=53, tag_tid=8, entry_tid=8713' 2024-04-06T09:37:17.028 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=57, tag_tid=9, entry_tid=1' 2024-04-06T09:37:17.028 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:37:17.028 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:37:17.028 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=9 2024-04-06T09:37:17.029 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=53, tag_tid=8, entry_tid=8713' 2024-04-06T09:37:17.029 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:37:17.029 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:37:17.030 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=8 2024-04-06T09:37:17.031 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=57, tag_tid=9, entry_tid=1' 2024-04-06T09:37:17.031 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:37:17.031 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:37:17.032 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:37:17.033 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=53, tag_tid=8, entry_tid=8713' 2024-04-06T09:37:17.033 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:37:17.033 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:37:17.034 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=8713 2024-04-06T09:37:17.034 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 9 = 8 -a 1 -le 8713 2024-04-06T09:37:17.034 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:37:17.034 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:37:17.034 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:37:17.236 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:37:17.236 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:37:17.236 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:37:17.236 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:37:17.237 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:37:17.237 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:37:17.237 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:37:17.237 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:37:17.237 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:37:17.237 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:37:17.237 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:37:17.237 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:37:17.237 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:37:17.237 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:37:17.237 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:37:17.238 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:37:17.238 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:37:17.241 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:37:17.241 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:37:17.242 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:37:17.242 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:37:17.242 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:37:17.242 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:37:17.242 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:37:17.242 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:37:17.242 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:37:17.242 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:37:17.242 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:37:17.242 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:37:17.242 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:37:17.242 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:37:17.242 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:37:17.243 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:37:17.243 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:37:17.243 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:37:17.243 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:37:17.243 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:37:17.243 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:37:17.243 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:37:17.243 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:37:17.243 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:37:17.243 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:37:17.243 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:37:17.243 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:37:17.243 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:37:17.243 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:37:17.243 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:37:17.243 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:37:17.244 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:37:17.265 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:37:17.265 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:37:17.265 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:37:17.265 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:37:17.405 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:37:17.406 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:37:17.406 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:37:17.406 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:37:17.406 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:37:17.406 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:37:17.406 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:37:17.406 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:37:17.406 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:37:17.406 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:37:17.407 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:37:17.407 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:37:17.407 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:37:17.410 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:37:17.410 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:37:17.411 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:37:17.475 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 13 2024-04-06T09:37:17.475 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 14 2024-04-06T09:37:17.475 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:37:17.475 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=57, tag_tid=9, entry_tid=1], [object_number=56, tag_tid=9, entry_tid=0], [object_number=59, tag_tid=8, entry_tid=11939], [object_number=58, tag_tid=8, entry_tid=11938]]], state=connected] 2024-04-06T09:37:17.475 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=57, tag_tid=8, entry_tid=8777], [object_number=56, tag_tid=8, entry_tid=8776], [object_number=59, tag_tid=8, entry_tid=8775], [object_number=58, tag_tid=8, entry_tid=8774]]], state=connected] 2024-04-06T09:37:17.480 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:37:17.482 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=57, tag_tid=9, entry_tid=1' 2024-04-06T09:37:17.483 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:37:17.483 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:37:17.483 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:37:17.483 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:37:17.483 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:37:17.483 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:37:17.484 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:37:17.484 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:37:17.484 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:37:17.484 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:37:17.484 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:37:17.484 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:37:17.485 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:37:17.486 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:37:17.486 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:37:17.535 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 14 2024-04-06T09:37:17.535 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 14 2024-04-06T09:37:17.535 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:37:17.535 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=57, tag_tid=9, entry_tid=1], [object_number=56, tag_tid=9, entry_tid=0], [object_number=59, tag_tid=8, entry_tid=11939], [object_number=58, tag_tid=8, entry_tid=11938]]], state=connected] 2024-04-06T09:37:17.535 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=57, tag_tid=8, entry_tid=8777], [object_number=56, tag_tid=8, entry_tid=8776], [object_number=59, tag_tid=8, entry_tid=8775], [object_number=58, tag_tid=8, entry_tid=8774]]], state=connected] 2024-04-06T09:37:17.538 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:37:17.540 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=57, tag_tid=8, entry_tid=8777' 2024-04-06T09:37:17.540 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=57, tag_tid=9, entry_tid=1' -a 'object_number=57, tag_tid=9, entry_tid=1' = 'object_number=57, tag_tid=8, entry_tid=8777' 2024-04-06T09:37:17.540 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=57, tag_tid=8, entry_tid=8777' '!=' 'object_number=53, tag_tid=8, entry_tid=8713' 2024-04-06T09:37:17.540 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:37:17.541 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=57, tag_tid=8, entry_tid=8777' = 'object_number=53, tag_tid=8, entry_tid=8713' 2024-04-06T09:37:17.541 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=57, tag_tid=8, entry_tid=8777' 2024-04-06T09:37:17.541 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=57, tag_tid=9, entry_tid=1' 2024-04-06T09:37:17.541 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:37:17.541 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:37:17.541 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=9 2024-04-06T09:37:17.542 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=57, tag_tid=8, entry_tid=8777' 2024-04-06T09:37:17.542 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:37:17.542 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:37:17.543 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=8 2024-04-06T09:37:17.544 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=57, tag_tid=9, entry_tid=1' 2024-04-06T09:37:17.544 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:37:17.544 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:37:17.545 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:37:17.546 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=57, tag_tid=8, entry_tid=8777' 2024-04-06T09:37:17.546 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:37:17.546 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:37:17.547 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=8777 2024-04-06T09:37:17.547 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 9 = 8 -a 1 -le 8777 2024-04-06T09:37:17.547 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:37:17.547 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:37:17.547 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:37:17.749 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:37:17.749 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:37:17.749 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:37:17.749 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:37:17.749 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:37:17.749 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:37:17.749 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:37:17.749 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:37:17.749 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:37:17.750 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:37:17.750 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:37:17.750 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:37:17.750 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:37:17.750 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:37:17.750 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:37:17.750 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:37:17.750 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:37:17.754 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:37:17.754 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:37:17.754 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:37:17.754 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:37:17.754 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:37:17.755 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:37:17.755 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:37:17.755 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:37:17.755 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:37:17.755 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:37:17.755 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:37:17.755 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:37:17.755 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:37:17.755 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:37:17.755 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:37:17.755 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:37:17.755 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:37:17.755 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:37:17.755 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:37:17.756 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:37:17.756 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:37:17.756 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:37:17.756 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:37:17.756 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:37:17.756 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:37:17.756 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:37:17.756 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:37:17.756 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:37:17.756 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:37:17.756 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:37:17.756 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:37:17.756 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:37:17.779 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:37:17.780 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:37:17.780 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:37:17.780 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:37:17.907 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:37:17.908 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:37:17.908 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:37:17.908 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:37:17.909 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:37:17.909 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:37:17.909 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:37:17.909 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:37:17.909 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:37:17.909 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:37:17.909 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:37:17.909 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:37:17.910 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:37:17.910 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:37:17.911 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:37:17.911 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:37:17.965 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 14 2024-04-06T09:37:17.965 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 14 2024-04-06T09:37:17.965 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:37:17.965 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=57, tag_tid=9, entry_tid=1], [object_number=56, tag_tid=9, entry_tid=0], [object_number=59, tag_tid=8, entry_tid=11939], [object_number=58, tag_tid=8, entry_tid=11938]]], state=connected] 2024-04-06T09:37:17.965 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=59, tag_tid=8, entry_tid=8879], [object_number=58, tag_tid=8, entry_tid=8878], [object_number=57, tag_tid=8, entry_tid=8877], [object_number=56, tag_tid=8, entry_tid=8876]]], state=connected] 2024-04-06T09:37:17.968 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:37:17.970 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=57, tag_tid=9, entry_tid=1' 2024-04-06T09:37:17.971 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:37:17.971 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:37:17.971 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:37:17.971 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:37:17.971 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:37:17.971 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:37:17.971 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:37:17.971 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:37:17.972 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:37:17.972 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:37:17.972 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:37:17.972 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:37:17.972 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:37:17.972 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:37:17.972 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:37:18.018 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 14 2024-04-06T09:37:18.018 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 14 2024-04-06T09:37:18.018 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:37:18.018 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=57, tag_tid=9, entry_tid=1], [object_number=56, tag_tid=9, entry_tid=0], [object_number=59, tag_tid=8, entry_tid=11939], [object_number=58, tag_tid=8, entry_tid=11938]]], state=connected] 2024-04-06T09:37:18.018 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=59, tag_tid=8, entry_tid=8879], [object_number=58, tag_tid=8, entry_tid=8878], [object_number=57, tag_tid=8, entry_tid=8877], [object_number=56, tag_tid=8, entry_tid=8876]]], state=connected] 2024-04-06T09:37:18.021 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:37:18.023 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=59, tag_tid=8, entry_tid=8879' 2024-04-06T09:37:18.023 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=57, tag_tid=9, entry_tid=1' -a 'object_number=57, tag_tid=9, entry_tid=1' = 'object_number=59, tag_tid=8, entry_tid=8879' 2024-04-06T09:37:18.023 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=59, tag_tid=8, entry_tid=8879' '!=' 'object_number=57, tag_tid=8, entry_tid=8777' 2024-04-06T09:37:18.023 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:37:18.023 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=59, tag_tid=8, entry_tid=8879' = 'object_number=57, tag_tid=8, entry_tid=8777' 2024-04-06T09:37:18.023 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=59, tag_tid=8, entry_tid=8879' 2024-04-06T09:37:18.024 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=57, tag_tid=9, entry_tid=1' 2024-04-06T09:37:18.024 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:37:18.024 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:37:18.024 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=9 2024-04-06T09:37:18.025 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=59, tag_tid=8, entry_tid=8879' 2024-04-06T09:37:18.025 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:37:18.025 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:37:18.026 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=8 2024-04-06T09:37:18.027 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=57, tag_tid=9, entry_tid=1' 2024-04-06T09:37:18.027 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:37:18.027 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:37:18.028 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:37:18.028 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=59, tag_tid=8, entry_tid=8879' 2024-04-06T09:37:18.028 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:37:18.028 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:37:18.030 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=8879 2024-04-06T09:37:18.030 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 9 = 8 -a 1 -le 8879 2024-04-06T09:37:18.030 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:37:18.030 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:37:18.030 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:37:18.232 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:37:18.232 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:37:18.232 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:37:18.232 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:37:18.232 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:37:18.232 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:37:18.232 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:37:18.233 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:37:18.233 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:37:18.233 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:37:18.233 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:37:18.233 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:37:18.233 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:37:18.233 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:37:18.233 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:37:18.233 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:37:18.233 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:37:18.235 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:37:18.235 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:37:18.236 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:37:18.236 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:37:18.236 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:37:18.236 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:37:18.236 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:37:18.236 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:37:18.236 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:37:18.236 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:37:18.236 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:37:18.236 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:37:18.236 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:37:18.236 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:37:18.236 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:37:18.236 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:37:18.236 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:37:18.237 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:37:18.237 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:37:18.237 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:37:18.237 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:37:18.237 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:37:18.237 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:37:18.237 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:37:18.237 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:37:18.237 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:37:18.237 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:37:18.237 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:37:18.237 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:37:18.237 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:37:18.237 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:37:18.238 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:37:18.258 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:37:18.258 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:37:18.258 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:37:18.258 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:37:18.392 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:37:18.393 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:37:18.393 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:37:18.393 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:37:18.393 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:37:18.393 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:37:18.393 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:37:18.393 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:37:18.393 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:37:18.394 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:37:18.394 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:37:18.394 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:37:18.394 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:37:18.396 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:37:18.397 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:37:18.397 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:37:18.443 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 14 2024-04-06T09:37:18.443 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 14 2024-04-06T09:37:18.443 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:37:18.444 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=57, tag_tid=9, entry_tid=1], [object_number=56, tag_tid=9, entry_tid=0], [object_number=59, tag_tid=8, entry_tid=11939], [object_number=58, tag_tid=8, entry_tid=11938]]], state=connected] 2024-04-06T09:37:18.444 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=56, tag_tid=8, entry_tid=8976], [object_number=59, tag_tid=8, entry_tid=8975], [object_number=58, tag_tid=8, entry_tid=8974], [object_number=57, tag_tid=8, entry_tid=8973]]], state=connected] 2024-04-06T09:37:18.447 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:37:18.448 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=57, tag_tid=9, entry_tid=1' 2024-04-06T09:37:18.449 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:37:18.450 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:37:18.450 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:37:18.450 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:37:18.450 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:37:18.450 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:37:18.450 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:37:18.450 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:37:18.450 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:37:18.450 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:37:18.450 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:37:18.450 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:37:18.451 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:37:18.451 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:37:18.451 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:37:18.498 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 14 2024-04-06T09:37:18.498 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 14 2024-04-06T09:37:18.498 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:37:18.498 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=57, tag_tid=9, entry_tid=1], [object_number=56, tag_tid=9, entry_tid=0], [object_number=59, tag_tid=8, entry_tid=11939], [object_number=58, tag_tid=8, entry_tid=11938]]], state=connected] 2024-04-06T09:37:18.498 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=56, tag_tid=8, entry_tid=8976], [object_number=59, tag_tid=8, entry_tid=8975], [object_number=58, tag_tid=8, entry_tid=8974], [object_number=57, tag_tid=8, entry_tid=8973]]], state=connected] 2024-04-06T09:37:18.502 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:37:18.504 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=56, tag_tid=8, entry_tid=8976' 2024-04-06T09:37:18.504 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=57, tag_tid=9, entry_tid=1' -a 'object_number=57, tag_tid=9, entry_tid=1' = 'object_number=56, tag_tid=8, entry_tid=8976' 2024-04-06T09:37:18.504 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=56, tag_tid=8, entry_tid=8976' '!=' 'object_number=59, tag_tid=8, entry_tid=8879' 2024-04-06T09:37:18.505 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:37:18.505 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=56, tag_tid=8, entry_tid=8976' = 'object_number=59, tag_tid=8, entry_tid=8879' 2024-04-06T09:37:18.505 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=56, tag_tid=8, entry_tid=8976' 2024-04-06T09:37:18.505 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=57, tag_tid=9, entry_tid=1' 2024-04-06T09:37:18.505 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:37:18.505 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:37:18.505 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=9 2024-04-06T09:37:18.506 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=56, tag_tid=8, entry_tid=8976' 2024-04-06T09:37:18.506 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:37:18.506 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:37:18.507 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=8 2024-04-06T09:37:18.508 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=57, tag_tid=9, entry_tid=1' 2024-04-06T09:37:18.508 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:37:18.508 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:37:18.509 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:37:18.509 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=56, tag_tid=8, entry_tid=8976' 2024-04-06T09:37:18.510 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:37:18.510 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:37:18.511 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=8976 2024-04-06T09:37:18.511 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 9 = 8 -a 1 -le 8976 2024-04-06T09:37:18.511 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:37:18.511 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:37:18.511 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:37:18.713 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:37:18.713 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:37:18.713 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:37:18.713 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:37:18.713 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:37:18.713 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:37:18.713 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:37:18.713 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:37:18.713 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:37:18.713 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:37:18.714 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:37:18.714 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:37:18.714 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:37:18.714 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:37:18.714 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:37:18.716 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:37:18.717 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:37:18.717 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:37:18.717 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:37:18.717 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:37:18.717 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:37:18.717 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:37:18.717 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:37:18.717 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:37:18.717 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:37:18.717 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:37:18.717 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:37:18.717 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:37:18.717 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:37:18.717 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:37:18.717 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:37:18.718 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:37:18.718 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:37:18.718 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:37:18.718 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:37:18.718 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:37:18.718 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:37:18.718 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:37:18.718 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:37:18.718 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:37:18.718 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:37:18.718 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:37:18.718 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:37:18.718 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:37:18.718 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:37:18.719 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:37:18.719 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:37:18.719 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:37:18.719 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:37:18.744 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:37:18.744 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:37:18.744 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:37:18.745 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:37:18.881 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:37:18.882 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:37:18.882 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:37:18.882 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:37:18.882 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:37:18.882 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:37:18.882 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:37:18.882 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:37:18.882 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:37:18.882 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:37:18.883 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:37:18.883 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:37:18.883 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:37:18.884 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:37:18.885 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:37:18.885 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:37:18.935 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 14 2024-04-06T09:37:18.935 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 14 2024-04-06T09:37:18.935 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:37:18.935 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=57, tag_tid=9, entry_tid=1], [object_number=56, tag_tid=9, entry_tid=0], [object_number=59, tag_tid=8, entry_tid=11939], [object_number=58, tag_tid=8, entry_tid=11938]]], state=connected] 2024-04-06T09:37:18.935 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=57, tag_tid=8, entry_tid=9077], [object_number=56, tag_tid=8, entry_tid=9076], [object_number=59, tag_tid=8, entry_tid=9075], [object_number=58, tag_tid=8, entry_tid=9074]]], state=connected] 2024-04-06T09:37:18.938 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:37:18.940 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=57, tag_tid=9, entry_tid=1' 2024-04-06T09:37:18.941 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:37:18.941 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:37:18.941 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:37:18.941 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:37:18.941 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:37:18.941 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:37:18.941 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:37:18.941 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:37:18.941 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:37:18.941 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:37:18.941 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:37:18.942 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:37:18.942 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:37:18.942 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:37:18.942 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:37:18.988 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 14 2024-04-06T09:37:18.988 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 14 2024-04-06T09:37:18.988 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:37:18.988 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=57, tag_tid=9, entry_tid=1], [object_number=56, tag_tid=9, entry_tid=0], [object_number=59, tag_tid=8, entry_tid=11939], [object_number=58, tag_tid=8, entry_tid=11938]]], state=connected] 2024-04-06T09:37:18.988 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=57, tag_tid=8, entry_tid=9077], [object_number=56, tag_tid=8, entry_tid=9076], [object_number=59, tag_tid=8, entry_tid=9075], [object_number=58, tag_tid=8, entry_tid=9074]]], state=connected] 2024-04-06T09:37:18.992 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:37:18.993 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=57, tag_tid=8, entry_tid=9077' 2024-04-06T09:37:18.993 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=57, tag_tid=9, entry_tid=1' -a 'object_number=57, tag_tid=9, entry_tid=1' = 'object_number=57, tag_tid=8, entry_tid=9077' 2024-04-06T09:37:18.993 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=57, tag_tid=8, entry_tid=9077' '!=' 'object_number=56, tag_tid=8, entry_tid=8976' 2024-04-06T09:37:18.994 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:37:18.994 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=57, tag_tid=8, entry_tid=9077' = 'object_number=56, tag_tid=8, entry_tid=8976' 2024-04-06T09:37:18.994 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=57, tag_tid=8, entry_tid=9077' 2024-04-06T09:37:18.994 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=57, tag_tid=9, entry_tid=1' 2024-04-06T09:37:18.994 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:37:18.994 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:37:18.995 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=9 2024-04-06T09:37:18.995 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=57, tag_tid=8, entry_tid=9077' 2024-04-06T09:37:18.996 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:37:18.996 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:37:18.996 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=8 2024-04-06T09:37:18.997 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=57, tag_tid=9, entry_tid=1' 2024-04-06T09:37:18.997 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:37:18.997 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:37:18.998 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:37:18.999 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=57, tag_tid=8, entry_tid=9077' 2024-04-06T09:37:18.999 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:37:18.999 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:37:19.000 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=9077 2024-04-06T09:37:19.000 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 9 = 8 -a 1 -le 9077 2024-04-06T09:37:19.000 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:37:19.000 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:37:19.000 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:37:19.202 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:37:19.202 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:37:19.202 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:37:19.202 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:37:19.202 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:37:19.202 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:37:19.202 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:37:19.203 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:37:19.203 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:37:19.203 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:37:19.203 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:37:19.203 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:37:19.203 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:37:19.203 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:37:19.203 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:37:19.203 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:37:19.204 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:37:19.207 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:37:19.207 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:37:19.207 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:37:19.207 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:37:19.207 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:37:19.207 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:37:19.207 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:37:19.207 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:37:19.207 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:37:19.207 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:37:19.208 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:37:19.208 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:37:19.208 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:37:19.208 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:37:19.208 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:37:19.208 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:37:19.208 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:37:19.208 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:37:19.208 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:37:19.208 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:37:19.208 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:37:19.208 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:37:19.208 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:37:19.208 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:37:19.208 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:37:19.209 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:37:19.209 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:37:19.209 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:37:19.209 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:37:19.209 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:37:19.209 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:37:19.209 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:37:19.235 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:37:19.235 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:37:19.235 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:37:19.235 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:37:19.364 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:37:19.365 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:37:19.365 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:37:19.365 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:37:19.365 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:37:19.365 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:37:19.365 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:37:19.365 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:37:19.365 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:37:19.365 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:37:19.366 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:37:19.366 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:37:19.366 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:37:19.368 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:37:19.368 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:37:19.368 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:37:19.413 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 14 2024-04-06T09:37:19.414 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 14 2024-04-06T09:37:19.414 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:37:19.414 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=57, tag_tid=9, entry_tid=1], [object_number=56, tag_tid=9, entry_tid=0], [object_number=59, tag_tid=8, entry_tid=11939], [object_number=58, tag_tid=8, entry_tid=11938]]], state=connected] 2024-04-06T09:37:19.414 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=59, tag_tid=8, entry_tid=9195], [object_number=58, tag_tid=8, entry_tid=9194], [object_number=57, tag_tid=8, entry_tid=9193], [object_number=56, tag_tid=8, entry_tid=9192]]], state=connected] 2024-04-06T09:37:19.417 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:37:19.419 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=57, tag_tid=9, entry_tid=1' 2024-04-06T09:37:19.420 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:37:19.420 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:37:19.420 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:37:19.420 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:37:19.420 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:37:19.420 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:37:19.420 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:37:19.420 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:37:19.420 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:37:19.420 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:37:19.420 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:37:19.420 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:37:19.421 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:37:19.421 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:37:19.421 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:37:19.467 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 14 2024-04-06T09:37:19.467 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 14 2024-04-06T09:37:19.467 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:37:19.467 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=57, tag_tid=9, entry_tid=1], [object_number=56, tag_tid=9, entry_tid=0], [object_number=59, tag_tid=8, entry_tid=11939], [object_number=58, tag_tid=8, entry_tid=11938]]], state=connected] 2024-04-06T09:37:19.467 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=59, tag_tid=8, entry_tid=9195], [object_number=58, tag_tid=8, entry_tid=9194], [object_number=57, tag_tid=8, entry_tid=9193], [object_number=56, tag_tid=8, entry_tid=9192]]], state=connected] 2024-04-06T09:37:19.471 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:37:19.473 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=59, tag_tid=8, entry_tid=9195' 2024-04-06T09:37:19.473 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=57, tag_tid=9, entry_tid=1' -a 'object_number=57, tag_tid=9, entry_tid=1' = 'object_number=59, tag_tid=8, entry_tid=9195' 2024-04-06T09:37:19.473 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=59, tag_tid=8, entry_tid=9195' '!=' 'object_number=57, tag_tid=8, entry_tid=9077' 2024-04-06T09:37:19.473 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:37:19.473 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=59, tag_tid=8, entry_tid=9195' = 'object_number=57, tag_tid=8, entry_tid=9077' 2024-04-06T09:37:19.473 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=59, tag_tid=8, entry_tid=9195' 2024-04-06T09:37:19.474 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=57, tag_tid=9, entry_tid=1' 2024-04-06T09:37:19.474 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:37:19.474 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:37:19.475 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=9 2024-04-06T09:37:19.475 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=59, tag_tid=8, entry_tid=9195' 2024-04-06T09:37:19.475 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:37:19.475 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:37:19.476 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=8 2024-04-06T09:37:19.476 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=57, tag_tid=9, entry_tid=1' 2024-04-06T09:37:19.476 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:37:19.477 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:37:19.477 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:37:19.478 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=59, tag_tid=8, entry_tid=9195' 2024-04-06T09:37:19.478 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:37:19.478 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:37:19.479 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=9195 2024-04-06T09:37:19.479 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 9 = 8 -a 1 -le 9195 2024-04-06T09:37:19.479 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:37:19.480 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:37:19.480 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:37:19.682 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:37:19.683 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:37:19.683 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:37:19.683 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:37:19.683 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:37:19.683 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:37:19.683 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:37:19.683 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:37:19.683 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:37:19.683 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:37:19.683 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:37:19.683 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:37:19.683 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:37:19.683 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:37:19.684 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:37:19.684 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:37:19.686 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:37:19.687 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:37:19.687 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:37:19.687 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:37:19.687 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:37:19.687 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:37:19.687 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:37:19.687 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:37:19.687 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:37:19.687 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:37:19.687 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:37:19.687 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:37:19.687 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:37:19.687 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:37:19.687 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:37:19.687 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:37:19.688 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:37:19.688 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:37:19.688 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:37:19.688 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:37:19.688 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:37:19.688 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:37:19.688 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:37:19.688 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:37:19.688 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:37:19.688 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:37:19.688 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:37:19.688 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:37:19.688 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:37:19.688 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:37:19.688 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:37:19.689 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:37:19.689 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:37:19.713 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:37:19.713 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:37:19.713 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:37:19.714 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:37:19.841 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:37:19.843 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:37:19.843 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:37:19.843 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:37:19.843 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:37:19.843 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:37:19.843 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:37:19.843 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:37:19.843 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:37:19.843 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:37:19.844 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:37:19.844 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:37:19.844 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:37:19.846 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:37:19.846 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:37:19.846 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:37:19.891 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 14 2024-04-06T09:37:19.891 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 14 2024-04-06T09:37:19.891 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:37:19.891 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=57, tag_tid=9, entry_tid=1], [object_number=56, tag_tid=9, entry_tid=0], [object_number=59, tag_tid=8, entry_tid=11939], [object_number=58, tag_tid=8, entry_tid=11938]]], state=connected] 2024-04-06T09:37:19.892 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=59, tag_tid=8, entry_tid=9299], [object_number=58, tag_tid=8, entry_tid=9298], [object_number=57, tag_tid=8, entry_tid=9297], [object_number=56, tag_tid=8, entry_tid=9296]]], state=connected] 2024-04-06T09:37:19.895 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:37:19.896 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=57, tag_tid=9, entry_tid=1' 2024-04-06T09:37:19.897 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:37:19.897 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:37:19.897 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:37:19.897 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:37:19.897 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:37:19.898 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:37:19.898 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:37:19.898 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:37:19.898 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:37:19.898 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:37:19.898 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:37:19.898 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:37:19.899 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:37:19.899 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:37:19.899 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:37:19.944 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 14 2024-04-06T09:37:19.944 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 14 2024-04-06T09:37:19.944 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:37:19.945 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=57, tag_tid=9, entry_tid=1], [object_number=56, tag_tid=9, entry_tid=0], [object_number=59, tag_tid=8, entry_tid=11939], [object_number=58, tag_tid=8, entry_tid=11938]]], state=connected] 2024-04-06T09:37:19.945 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=59, tag_tid=8, entry_tid=9299], [object_number=58, tag_tid=8, entry_tid=9298], [object_number=57, tag_tid=8, entry_tid=9297], [object_number=56, tag_tid=8, entry_tid=9296]]], state=connected] 2024-04-06T09:37:19.948 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:37:19.950 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=59, tag_tid=8, entry_tid=9299' 2024-04-06T09:37:19.950 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=57, tag_tid=9, entry_tid=1' -a 'object_number=57, tag_tid=9, entry_tid=1' = 'object_number=59, tag_tid=8, entry_tid=9299' 2024-04-06T09:37:19.950 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=59, tag_tid=8, entry_tid=9299' '!=' 'object_number=59, tag_tid=8, entry_tid=9195' 2024-04-06T09:37:19.950 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:37:19.950 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=59, tag_tid=8, entry_tid=9299' = 'object_number=59, tag_tid=8, entry_tid=9195' 2024-04-06T09:37:19.950 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=59, tag_tid=8, entry_tid=9299' 2024-04-06T09:37:19.951 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=57, tag_tid=9, entry_tid=1' 2024-04-06T09:37:19.951 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:37:19.951 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:37:19.951 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=9 2024-04-06T09:37:19.952 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=59, tag_tid=8, entry_tid=9299' 2024-04-06T09:37:19.952 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:37:19.952 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:37:19.953 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=8 2024-04-06T09:37:19.954 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=57, tag_tid=9, entry_tid=1' 2024-04-06T09:37:19.954 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:37:19.954 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:37:19.955 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:37:19.956 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=59, tag_tid=8, entry_tid=9299' 2024-04-06T09:37:19.956 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:37:19.956 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:37:19.957 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=9299 2024-04-06T09:37:19.957 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 9 = 8 -a 1 -le 9299 2024-04-06T09:37:19.957 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:37:19.957 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:37:19.957 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:37:20.159 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:37:20.159 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:37:20.159 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:37:20.159 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:37:20.159 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:37:20.159 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:37:20.159 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:37:20.159 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:37:20.159 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:37:20.159 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:37:20.160 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:37:20.160 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:37:20.160 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:37:20.160 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:37:20.160 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:37:20.160 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:37:20.160 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:37:20.163 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:37:20.164 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:37:20.164 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:37:20.164 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:37:20.164 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:37:20.164 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:37:20.164 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:37:20.164 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:37:20.164 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:37:20.164 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:37:20.164 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:37:20.165 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:37:20.165 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:37:20.165 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:37:20.165 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:37:20.165 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:37:20.165 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:37:20.165 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:37:20.165 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:37:20.165 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:37:20.165 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:37:20.165 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:37:20.165 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:37:20.165 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:37:20.165 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:37:20.166 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:37:20.166 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:37:20.166 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:37:20.166 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:37:20.166 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:37:20.166 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:37:20.166 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:37:20.190 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:37:20.190 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:37:20.191 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:37:20.191 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:37:20.318 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:37:20.319 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:37:20.319 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:37:20.319 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:37:20.320 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:37:20.320 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:37:20.320 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:37:20.320 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:37:20.320 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:37:20.320 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:37:20.320 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:37:20.320 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:37:20.321 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:37:20.323 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:37:20.323 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:37:20.324 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:37:20.371 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 14 2024-04-06T09:37:20.371 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 14 2024-04-06T09:37:20.371 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:37:20.371 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=57, tag_tid=9, entry_tid=1], [object_number=56, tag_tid=9, entry_tid=0], [object_number=59, tag_tid=8, entry_tid=11939], [object_number=58, tag_tid=8, entry_tid=11938]]], state=connected] 2024-04-06T09:37:20.371 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=56, tag_tid=8, entry_tid=9408], [object_number=59, tag_tid=8, entry_tid=9407], [object_number=58, tag_tid=8, entry_tid=9406], [object_number=57, tag_tid=8, entry_tid=9405]]], state=connected] 2024-04-06T09:37:20.375 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:37:20.376 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=57, tag_tid=9, entry_tid=1' 2024-04-06T09:37:20.377 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:37:20.377 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:37:20.378 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:37:20.378 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:37:20.378 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:37:20.378 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:37:20.378 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:37:20.378 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:37:20.378 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:37:20.378 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:37:20.378 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:37:20.378 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:37:20.379 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:37:20.379 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:37:20.379 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:37:20.426 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 14 2024-04-06T09:37:20.426 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 14 2024-04-06T09:37:20.426 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:37:20.426 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=57, tag_tid=9, entry_tid=1], [object_number=56, tag_tid=9, entry_tid=0], [object_number=59, tag_tid=8, entry_tid=11939], [object_number=58, tag_tid=8, entry_tid=11938]]], state=connected] 2024-04-06T09:37:20.426 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=56, tag_tid=8, entry_tid=9408], [object_number=59, tag_tid=8, entry_tid=9407], [object_number=58, tag_tid=8, entry_tid=9406], [object_number=57, tag_tid=8, entry_tid=9405]]], state=connected] 2024-04-06T09:37:20.429 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:37:20.431 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=56, tag_tid=8, entry_tid=9408' 2024-04-06T09:37:20.432 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=57, tag_tid=9, entry_tid=1' -a 'object_number=57, tag_tid=9, entry_tid=1' = 'object_number=56, tag_tid=8, entry_tid=9408' 2024-04-06T09:37:20.432 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=56, tag_tid=8, entry_tid=9408' '!=' 'object_number=59, tag_tid=8, entry_tid=9299' 2024-04-06T09:37:20.432 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:37:20.432 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=56, tag_tid=8, entry_tid=9408' = 'object_number=59, tag_tid=8, entry_tid=9299' 2024-04-06T09:37:20.432 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=56, tag_tid=8, entry_tid=9408' 2024-04-06T09:37:20.432 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=57, tag_tid=9, entry_tid=1' 2024-04-06T09:37:20.432 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:37:20.432 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:37:20.433 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=9 2024-04-06T09:37:20.433 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=56, tag_tid=8, entry_tid=9408' 2024-04-06T09:37:20.433 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:37:20.433 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:37:20.434 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=8 2024-04-06T09:37:20.435 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=57, tag_tid=9, entry_tid=1' 2024-04-06T09:37:20.435 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:37:20.435 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:37:20.436 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:37:20.437 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=56, tag_tid=8, entry_tid=9408' 2024-04-06T09:37:20.437 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:37:20.437 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:37:20.438 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=9408 2024-04-06T09:37:20.438 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 9 = 8 -a 1 -le 9408 2024-04-06T09:37:20.438 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:37:20.438 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:37:20.438 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:37:20.640 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:37:20.640 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:37:20.640 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:37:20.640 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:37:20.640 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:37:20.640 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:37:20.640 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:37:20.641 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:37:20.641 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:37:20.641 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:37:20.641 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:37:20.641 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:37:20.641 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:37:20.641 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:37:20.641 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:37:20.641 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:37:20.642 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:37:20.645 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:37:20.645 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:37:20.645 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:37:20.645 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:37:20.645 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:37:20.645 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:37:20.646 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:37:20.646 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:37:20.646 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:37:20.646 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:37:20.646 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:37:20.646 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:37:20.646 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:37:20.646 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:37:20.646 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:37:20.646 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:37:20.646 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:37:20.646 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:37:20.646 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:37:20.646 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:37:20.646 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:37:20.647 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:37:20.647 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:37:20.647 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:37:20.647 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:37:20.647 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:37:20.647 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:37:20.647 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:37:20.647 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:37:20.647 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:37:20.647 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:37:20.647 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:37:20.671 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:37:20.671 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:37:20.671 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:37:20.671 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:37:20.796 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:37:20.797 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:37:20.797 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:37:20.797 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:37:20.798 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:37:20.798 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:37:20.798 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:37:20.798 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:37:20.798 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:37:20.798 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:37:20.799 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:37:20.799 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:37:20.799 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:37:20.800 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:37:20.800 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:37:20.801 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:37:20.851 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 14 2024-04-06T09:37:20.852 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 14 2024-04-06T09:37:20.852 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:37:20.852 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=57, tag_tid=9, entry_tid=1], [object_number=56, tag_tid=9, entry_tid=0], [object_number=59, tag_tid=8, entry_tid=11939], [object_number=58, tag_tid=8, entry_tid=11938]]], state=connected] 2024-04-06T09:37:20.852 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=58, tag_tid=8, entry_tid=9514], [object_number=57, tag_tid=8, entry_tid=9513], [object_number=56, tag_tid=8, entry_tid=9512], [object_number=59, tag_tid=8, entry_tid=9511]]], state=connected] 2024-04-06T09:37:20.855 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:37:20.856 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=57, tag_tid=9, entry_tid=1' 2024-04-06T09:37:20.858 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:37:20.858 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:37:20.858 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:37:20.858 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:37:20.858 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:37:20.858 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:37:20.858 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:37:20.858 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:37:20.858 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:37:20.858 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:37:20.858 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:37:20.858 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:37:20.859 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:37:20.859 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:37:20.859 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:37:20.904 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 14 2024-04-06T09:37:20.905 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 14 2024-04-06T09:37:20.905 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:37:20.905 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=57, tag_tid=9, entry_tid=1], [object_number=56, tag_tid=9, entry_tid=0], [object_number=59, tag_tid=8, entry_tid=11939], [object_number=58, tag_tid=8, entry_tid=11938]]], state=connected] 2024-04-06T09:37:20.905 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=58, tag_tid=8, entry_tid=9514], [object_number=57, tag_tid=8, entry_tid=9513], [object_number=56, tag_tid=8, entry_tid=9512], [object_number=59, tag_tid=8, entry_tid=9511]]], state=connected] 2024-04-06T09:37:20.908 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:37:20.910 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=58, tag_tid=8, entry_tid=9514' 2024-04-06T09:37:20.910 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=57, tag_tid=9, entry_tid=1' -a 'object_number=57, tag_tid=9, entry_tid=1' = 'object_number=58, tag_tid=8, entry_tid=9514' 2024-04-06T09:37:20.910 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=58, tag_tid=8, entry_tid=9514' '!=' 'object_number=56, tag_tid=8, entry_tid=9408' 2024-04-06T09:37:20.910 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:37:20.910 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=58, tag_tid=8, entry_tid=9514' = 'object_number=56, tag_tid=8, entry_tid=9408' 2024-04-06T09:37:20.910 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=58, tag_tid=8, entry_tid=9514' 2024-04-06T09:37:20.911 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=57, tag_tid=9, entry_tid=1' 2024-04-06T09:37:20.911 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:37:20.911 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:37:20.911 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=9 2024-04-06T09:37:20.912 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=58, tag_tid=8, entry_tid=9514' 2024-04-06T09:37:20.912 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:37:20.912 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:37:20.913 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=8 2024-04-06T09:37:20.914 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=57, tag_tid=9, entry_tid=1' 2024-04-06T09:37:20.914 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:37:20.914 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:37:20.915 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:37:20.916 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=58, tag_tid=8, entry_tid=9514' 2024-04-06T09:37:20.916 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:37:20.916 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:37:20.917 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=9514 2024-04-06T09:37:20.917 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 9 = 8 -a 1 -le 9514 2024-04-06T09:37:20.917 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:37:20.917 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:37:20.917 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:37:21.119 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:37:21.119 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:37:21.120 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:37:21.120 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:37:21.120 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:37:21.120 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:37:21.120 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:37:21.120 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:37:21.120 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:37:21.120 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:37:21.120 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:37:21.120 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:37:21.120 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:37:21.120 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:37:21.120 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:37:21.120 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:37:21.121 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:37:21.124 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:37:21.124 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:37:21.124 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:37:21.124 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:37:21.125 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:37:21.125 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:37:21.125 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:37:21.125 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:37:21.125 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:37:21.125 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:37:21.125 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:37:21.125 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:37:21.125 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:37:21.125 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:37:21.125 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:37:21.125 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:37:21.125 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:37:21.125 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:37:21.125 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:37:21.125 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:37:21.126 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:37:21.126 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:37:21.126 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:37:21.126 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:37:21.126 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:37:21.126 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:37:21.126 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:37:21.126 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:37:21.126 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:37:21.126 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:37:21.126 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:37:21.127 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:37:21.148 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:37:21.148 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:37:21.148 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:37:21.149 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:37:21.283 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:37:21.284 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:37:21.284 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:37:21.284 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:37:21.284 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:37:21.284 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:37:21.284 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:37:21.284 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:37:21.284 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:37:21.284 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:37:21.285 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:37:21.285 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:37:21.285 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:37:21.287 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:37:21.287 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:37:21.287 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:37:21.331 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 14 2024-04-06T09:37:21.332 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 14 2024-04-06T09:37:21.332 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:37:21.332 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=57, tag_tid=9, entry_tid=1], [object_number=56, tag_tid=9, entry_tid=0], [object_number=59, tag_tid=8, entry_tid=11939], [object_number=58, tag_tid=8, entry_tid=11938]]], state=connected] 2024-04-06T09:37:21.332 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=56, tag_tid=8, entry_tid=9608], [object_number=59, tag_tid=8, entry_tid=9607], [object_number=58, tag_tid=8, entry_tid=9606], [object_number=57, tag_tid=8, entry_tid=9605]]], state=connected] 2024-04-06T09:37:21.335 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:37:21.336 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=57, tag_tid=9, entry_tid=1' 2024-04-06T09:37:21.337 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:37:21.337 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:37:21.337 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:37:21.338 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:37:21.338 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:37:21.338 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:37:21.338 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:37:21.338 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:37:21.338 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:37:21.338 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:37:21.338 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:37:21.338 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:37:21.339 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:37:21.339 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:37:21.339 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:37:21.384 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 14 2024-04-06T09:37:21.384 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 14 2024-04-06T09:37:21.384 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:37:21.384 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=57, tag_tid=9, entry_tid=1], [object_number=56, tag_tid=9, entry_tid=0], [object_number=59, tag_tid=8, entry_tid=11939], [object_number=58, tag_tid=8, entry_tid=11938]]], state=connected] 2024-04-06T09:37:21.384 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=56, tag_tid=8, entry_tid=9608], [object_number=59, tag_tid=8, entry_tid=9607], [object_number=58, tag_tid=8, entry_tid=9606], [object_number=57, tag_tid=8, entry_tid=9605]]], state=connected] 2024-04-06T09:37:21.388 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:37:21.389 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=56, tag_tid=8, entry_tid=9608' 2024-04-06T09:37:21.389 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=57, tag_tid=9, entry_tid=1' -a 'object_number=57, tag_tid=9, entry_tid=1' = 'object_number=56, tag_tid=8, entry_tid=9608' 2024-04-06T09:37:21.389 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=56, tag_tid=8, entry_tid=9608' '!=' 'object_number=58, tag_tid=8, entry_tid=9514' 2024-04-06T09:37:21.390 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:37:21.390 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=56, tag_tid=8, entry_tid=9608' = 'object_number=58, tag_tid=8, entry_tid=9514' 2024-04-06T09:37:21.390 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=56, tag_tid=8, entry_tid=9608' 2024-04-06T09:37:21.390 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=57, tag_tid=9, entry_tid=1' 2024-04-06T09:37:21.391 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:37:21.391 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:37:21.391 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=9 2024-04-06T09:37:21.392 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=56, tag_tid=8, entry_tid=9608' 2024-04-06T09:37:21.392 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:37:21.392 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:37:21.392 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=8 2024-04-06T09:37:21.393 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=57, tag_tid=9, entry_tid=1' 2024-04-06T09:37:21.393 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:37:21.393 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:37:21.394 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:37:21.395 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=56, tag_tid=8, entry_tid=9608' 2024-04-06T09:37:21.395 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:37:21.395 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:37:21.396 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=9608 2024-04-06T09:37:21.396 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 9 = 8 -a 1 -le 9608 2024-04-06T09:37:21.396 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:37:21.396 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:37:21.396 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:37:21.598 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:37:21.598 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:37:21.598 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:37:21.599 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:37:21.599 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:37:21.599 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:37:21.599 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:37:21.599 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:37:21.599 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:37:21.599 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:37:21.599 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:37:21.599 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:37:21.599 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:37:21.599 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:37:21.599 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:37:21.600 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:37:21.600 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:37:21.603 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:37:21.603 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:37:21.603 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:37:21.603 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:37:21.603 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:37:21.603 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:37:21.603 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:37:21.604 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:37:21.604 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:37:21.604 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:37:21.604 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:37:21.604 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:37:21.604 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:37:21.604 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:37:21.604 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:37:21.604 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:37:21.604 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:37:21.604 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:37:21.604 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:37:21.604 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:37:21.604 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:37:21.604 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:37:21.605 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:37:21.605 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:37:21.605 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:37:21.605 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:37:21.605 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:37:21.605 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:37:21.605 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:37:21.605 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:37:21.605 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:37:21.605 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:37:21.629 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:37:21.630 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:37:21.630 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:37:21.630 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:37:21.768 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:37:21.769 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:37:21.769 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:37:21.769 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:37:21.769 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:37:21.769 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:37:21.770 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:37:21.770 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:37:21.770 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:37:21.770 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:37:21.770 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:37:21.770 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:37:21.770 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:37:21.771 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:37:21.771 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:37:21.771 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:37:21.826 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 14 2024-04-06T09:37:21.826 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 14 2024-04-06T09:37:21.826 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:37:21.826 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=57, tag_tid=9, entry_tid=1], [object_number=56, tag_tid=9, entry_tid=0], [object_number=59, tag_tid=8, entry_tid=11939], [object_number=58, tag_tid=8, entry_tid=11938]]], state=connected] 2024-04-06T09:37:21.826 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=56, tag_tid=8, entry_tid=9716], [object_number=59, tag_tid=8, entry_tid=9715], [object_number=58, tag_tid=8, entry_tid=9714], [object_number=57, tag_tid=8, entry_tid=9713]]], state=connected] 2024-04-06T09:37:21.829 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:37:21.831 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=57, tag_tid=9, entry_tid=1' 2024-04-06T09:37:21.832 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:37:21.832 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:37:21.832 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:37:21.832 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:37:21.832 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:37:21.832 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:37:21.832 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:37:21.832 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:37:21.832 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:37:21.832 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:37:21.832 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:37:21.833 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:37:21.833 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:37:21.833 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:37:21.833 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:37:21.879 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 14 2024-04-06T09:37:21.879 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 14 2024-04-06T09:37:21.880 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:37:21.880 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=57, tag_tid=9, entry_tid=1], [object_number=56, tag_tid=9, entry_tid=0], [object_number=59, tag_tid=8, entry_tid=11939], [object_number=58, tag_tid=8, entry_tid=11938]]], state=connected] 2024-04-06T09:37:21.880 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=56, tag_tid=8, entry_tid=9716], [object_number=59, tag_tid=8, entry_tid=9715], [object_number=58, tag_tid=8, entry_tid=9714], [object_number=57, tag_tid=8, entry_tid=9713]]], state=connected] 2024-04-06T09:37:21.883 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:37:21.885 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=56, tag_tid=8, entry_tid=9716' 2024-04-06T09:37:21.885 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=57, tag_tid=9, entry_tid=1' -a 'object_number=57, tag_tid=9, entry_tid=1' = 'object_number=56, tag_tid=8, entry_tid=9716' 2024-04-06T09:37:21.885 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=56, tag_tid=8, entry_tid=9716' '!=' 'object_number=56, tag_tid=8, entry_tid=9608' 2024-04-06T09:37:21.885 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:37:21.885 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=56, tag_tid=8, entry_tid=9716' = 'object_number=56, tag_tid=8, entry_tid=9608' 2024-04-06T09:37:21.885 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=56, tag_tid=8, entry_tid=9716' 2024-04-06T09:37:21.886 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=57, tag_tid=9, entry_tid=1' 2024-04-06T09:37:21.886 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:37:21.886 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:37:21.886 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=9 2024-04-06T09:37:21.887 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=56, tag_tid=8, entry_tid=9716' 2024-04-06T09:37:21.887 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:37:21.887 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:37:21.888 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=8 2024-04-06T09:37:21.888 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=57, tag_tid=9, entry_tid=1' 2024-04-06T09:37:21.889 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:37:21.889 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:37:21.890 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:37:21.890 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=56, tag_tid=8, entry_tid=9716' 2024-04-06T09:37:21.890 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:37:21.890 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:37:21.892 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=9716 2024-04-06T09:37:21.892 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 9 = 8 -a 1 -le 9716 2024-04-06T09:37:21.892 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:37:21.892 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:37:21.892 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:37:22.093 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:37:22.093 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:37:22.093 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:37:22.094 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:37:22.094 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:37:22.094 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:37:22.094 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:37:22.094 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:37:22.094 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:37:22.094 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:37:22.094 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:37:22.094 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:37:22.094 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:37:22.094 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:37:22.094 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:37:22.095 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:37:22.095 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:37:22.098 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:37:22.098 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:37:22.098 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:37:22.098 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:37:22.098 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:37:22.098 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:37:22.098 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:37:22.098 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:37:22.098 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:37:22.098 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:37:22.099 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:37:22.099 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:37:22.099 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:37:22.099 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:37:22.099 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:37:22.099 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:37:22.100 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:37:22.100 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:37:22.100 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:37:22.100 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:37:22.100 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:37:22.100 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:37:22.100 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:37:22.100 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:37:22.100 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:37:22.100 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:37:22.101 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:37:22.101 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:37:22.101 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:37:22.101 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:37:22.101 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:37:22.101 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:37:22.124 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:37:22.125 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:37:22.125 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:37:22.125 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:37:22.255 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:37:22.257 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:37:22.257 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:37:22.257 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:37:22.257 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:37:22.257 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:37:22.257 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:37:22.257 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:37:22.257 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:37:22.257 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:37:22.258 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:37:22.258 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:37:22.258 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:37:22.260 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:37:22.261 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:37:22.261 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:37:22.306 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 14 2024-04-06T09:37:22.307 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 14 2024-04-06T09:37:22.307 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:37:22.307 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=57, tag_tid=9, entry_tid=1], [object_number=56, tag_tid=9, entry_tid=0], [object_number=59, tag_tid=8, entry_tid=11939], [object_number=58, tag_tid=8, entry_tid=11938]]], state=connected] 2024-04-06T09:37:22.307 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=57, tag_tid=8, entry_tid=9809], [object_number=56, tag_tid=8, entry_tid=9808], [object_number=59, tag_tid=8, entry_tid=9807], [object_number=58, tag_tid=8, entry_tid=9806]]], state=connected] 2024-04-06T09:37:22.310 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:37:22.311 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=57, tag_tid=9, entry_tid=1' 2024-04-06T09:37:22.312 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:37:22.312 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:37:22.312 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:37:22.312 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:37:22.313 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:37:22.313 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:37:22.313 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:37:22.313 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:37:22.313 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:37:22.313 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:37:22.313 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:37:22.313 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:37:22.314 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:37:22.314 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:37:22.314 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:37:22.359 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 14 2024-04-06T09:37:22.359 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 14 2024-04-06T09:37:22.359 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:37:22.360 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=57, tag_tid=9, entry_tid=1], [object_number=56, tag_tid=9, entry_tid=0], [object_number=59, tag_tid=8, entry_tid=11939], [object_number=58, tag_tid=8, entry_tid=11938]]], state=connected] 2024-04-06T09:37:22.360 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=57, tag_tid=8, entry_tid=9809], [object_number=56, tag_tid=8, entry_tid=9808], [object_number=59, tag_tid=8, entry_tid=9807], [object_number=58, tag_tid=8, entry_tid=9806]]], state=connected] 2024-04-06T09:37:22.362 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:37:22.364 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=57, tag_tid=8, entry_tid=9809' 2024-04-06T09:37:22.365 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=57, tag_tid=9, entry_tid=1' -a 'object_number=57, tag_tid=9, entry_tid=1' = 'object_number=57, tag_tid=8, entry_tid=9809' 2024-04-06T09:37:22.365 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=57, tag_tid=8, entry_tid=9809' '!=' 'object_number=56, tag_tid=8, entry_tid=9716' 2024-04-06T09:37:22.365 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:37:22.365 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=57, tag_tid=8, entry_tid=9809' = 'object_number=56, tag_tid=8, entry_tid=9716' 2024-04-06T09:37:22.365 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=57, tag_tid=8, entry_tid=9809' 2024-04-06T09:37:22.365 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=57, tag_tid=9, entry_tid=1' 2024-04-06T09:37:22.365 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:37:22.365 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:37:22.365 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=9 2024-04-06T09:37:22.366 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=57, tag_tid=8, entry_tid=9809' 2024-04-06T09:37:22.366 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:37:22.366 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:37:22.367 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=8 2024-04-06T09:37:22.368 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=57, tag_tid=9, entry_tid=1' 2024-04-06T09:37:22.368 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:37:22.368 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:37:22.369 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:37:22.370 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=57, tag_tid=8, entry_tid=9809' 2024-04-06T09:37:22.370 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:37:22.370 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:37:22.371 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=9809 2024-04-06T09:37:22.371 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 9 = 8 -a 1 -le 9809 2024-04-06T09:37:22.371 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:37:22.371 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:37:22.371 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:37:22.573 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:37:22.573 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:37:22.573 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:37:22.573 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:37:22.573 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:37:22.573 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:37:22.573 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:37:22.573 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:37:22.574 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:37:22.574 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:37:22.574 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:37:22.574 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:37:22.574 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:37:22.574 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:37:22.574 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:37:22.575 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:37:22.576 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:37:22.576 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:37:22.576 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:37:22.576 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:37:22.576 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:37:22.576 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:37:22.576 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:37:22.576 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:37:22.576 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:37:22.576 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:37:22.576 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:37:22.576 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:37:22.577 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:37:22.577 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:37:22.577 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:37:22.577 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:37:22.577 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:37:22.578 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:37:22.578 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:37:22.579 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:37:22.579 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:37:22.579 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:37:22.579 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:37:22.579 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:37:22.579 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:37:22.579 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:37:22.579 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:37:22.579 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:37:22.579 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:37:22.579 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:37:22.579 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:37:22.579 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:37:22.579 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:37:22.603 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:37:22.603 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:37:22.603 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:37:22.603 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:37:22.728 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:37:22.729 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:37:22.730 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:37:22.730 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:37:22.730 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:37:22.730 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:37:22.730 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:37:22.730 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:37:22.730 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:37:22.730 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:37:22.731 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:37:22.731 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:37:22.731 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:37:22.732 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:37:22.732 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:37:22.732 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:37:22.782 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 14 2024-04-06T09:37:22.782 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 14 2024-04-06T09:37:22.783 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:37:22.783 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=57, tag_tid=9, entry_tid=1], [object_number=56, tag_tid=9, entry_tid=0], [object_number=59, tag_tid=8, entry_tid=11939], [object_number=58, tag_tid=8, entry_tid=11938]]], state=connected] 2024-04-06T09:37:22.783 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=57, tag_tid=8, entry_tid=9889], [object_number=56, tag_tid=8, entry_tid=9888], [object_number=59, tag_tid=8, entry_tid=9887], [object_number=58, tag_tid=8, entry_tid=9886]]], state=connected] 2024-04-06T09:37:22.786 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:37:22.788 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=57, tag_tid=9, entry_tid=1' 2024-04-06T09:37:22.789 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:37:22.789 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:37:22.789 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:37:22.789 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:37:22.789 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:37:22.789 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:37:22.789 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:37:22.789 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:37:22.789 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:37:22.790 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:37:22.790 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:37:22.790 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:37:22.790 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:37:22.790 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:37:22.792 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:37:22.837 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 14 2024-04-06T09:37:22.837 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 14 2024-04-06T09:37:22.837 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:37:22.837 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=57, tag_tid=9, entry_tid=1], [object_number=56, tag_tid=9, entry_tid=0], [object_number=59, tag_tid=8, entry_tid=11939], [object_number=58, tag_tid=8, entry_tid=11938]]], state=connected] 2024-04-06T09:37:22.837 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=57, tag_tid=8, entry_tid=9889], [object_number=56, tag_tid=8, entry_tid=9888], [object_number=59, tag_tid=8, entry_tid=9887], [object_number=58, tag_tid=8, entry_tid=9886]]], state=connected] 2024-04-06T09:37:22.840 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:37:22.843 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=57, tag_tid=8, entry_tid=9889' 2024-04-06T09:37:22.843 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=57, tag_tid=9, entry_tid=1' -a 'object_number=57, tag_tid=9, entry_tid=1' = 'object_number=57, tag_tid=8, entry_tid=9889' 2024-04-06T09:37:22.843 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=57, tag_tid=8, entry_tid=9889' '!=' 'object_number=57, tag_tid=8, entry_tid=9809' 2024-04-06T09:37:22.843 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:37:22.843 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=57, tag_tid=8, entry_tid=9889' = 'object_number=57, tag_tid=8, entry_tid=9809' 2024-04-06T09:37:22.843 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=57, tag_tid=8, entry_tid=9889' 2024-04-06T09:37:22.843 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=57, tag_tid=9, entry_tid=1' 2024-04-06T09:37:22.843 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:37:22.843 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:37:22.844 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=9 2024-04-06T09:37:22.844 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=57, tag_tid=8, entry_tid=9889' 2024-04-06T09:37:22.845 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:37:22.845 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:37:22.845 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=8 2024-04-06T09:37:22.846 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=57, tag_tid=9, entry_tid=1' 2024-04-06T09:37:22.846 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:37:22.846 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:37:22.847 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:37:22.848 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=57, tag_tid=8, entry_tid=9889' 2024-04-06T09:37:22.848 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:37:22.848 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:37:22.849 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=9889 2024-04-06T09:37:22.849 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 9 = 8 -a 1 -le 9889 2024-04-06T09:37:22.849 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:37:22.849 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:37:22.849 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:37:23.051 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:37:23.051 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:37:23.051 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:37:23.051 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:37:23.051 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:37:23.052 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:37:23.052 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:37:23.052 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:37:23.052 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:37:23.052 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:37:23.052 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:37:23.052 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:37:23.052 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:37:23.052 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:37:23.052 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:37:23.053 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:37:23.053 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:37:23.055 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:37:23.055 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:37:23.055 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:37:23.055 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:37:23.055 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:37:23.055 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:37:23.055 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:37:23.055 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:37:23.055 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:37:23.055 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:37:23.055 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:37:23.055 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:37:23.056 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:37:23.056 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:37:23.056 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:37:23.056 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:37:23.056 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:37:23.056 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:37:23.056 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:37:23.056 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:37:23.056 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:37:23.056 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:37:23.056 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:37:23.056 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:37:23.056 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:37:23.056 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:37:23.056 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:37:23.057 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:37:23.057 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:37:23.057 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:37:23.057 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:37:23.057 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:37:23.081 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:37:23.081 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:37:23.081 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:37:23.081 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:37:23.207 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:37:23.208 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:37:23.208 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:37:23.208 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:37:23.208 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:37:23.208 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:37:23.208 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:37:23.208 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:37:23.208 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:37:23.208 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:37:23.209 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:37:23.209 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:37:23.209 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:37:23.210 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:37:23.211 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:37:23.211 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:37:23.262 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 14 2024-04-06T09:37:23.262 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 14 2024-04-06T09:37:23.262 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:37:23.262 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=57, tag_tid=9, entry_tid=1], [object_number=56, tag_tid=9, entry_tid=0], [object_number=59, tag_tid=8, entry_tid=11939], [object_number=58, tag_tid=8, entry_tid=11938]]], state=connected] 2024-04-06T09:37:23.262 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=56, tag_tid=8, entry_tid=9984], [object_number=59, tag_tid=8, entry_tid=9983], [object_number=58, tag_tid=8, entry_tid=9982], [object_number=57, tag_tid=8, entry_tid=9981]]], state=connected] 2024-04-06T09:37:23.265 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:37:23.268 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=57, tag_tid=9, entry_tid=1' 2024-04-06T09:37:23.268 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:37:23.268 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:37:23.268 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:37:23.268 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:37:23.268 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:37:23.268 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:37:23.268 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:37:23.268 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:37:23.268 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:37:23.268 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:37:23.268 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:37:23.268 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:37:23.269 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:37:23.269 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:37:23.269 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:37:23.315 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 14 2024-04-06T09:37:23.315 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 14 2024-04-06T09:37:23.315 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:37:23.315 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=57, tag_tid=9, entry_tid=1], [object_number=56, tag_tid=9, entry_tid=0], [object_number=59, tag_tid=8, entry_tid=11939], [object_number=58, tag_tid=8, entry_tid=11938]]], state=connected] 2024-04-06T09:37:23.315 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=56, tag_tid=8, entry_tid=9984], [object_number=59, tag_tid=8, entry_tid=9983], [object_number=58, tag_tid=8, entry_tid=9982], [object_number=57, tag_tid=8, entry_tid=9981]]], state=connected] 2024-04-06T09:37:23.318 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:37:23.320 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=56, tag_tid=8, entry_tid=9984' 2024-04-06T09:37:23.320 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=57, tag_tid=9, entry_tid=1' -a 'object_number=57, tag_tid=9, entry_tid=1' = 'object_number=56, tag_tid=8, entry_tid=9984' 2024-04-06T09:37:23.321 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=56, tag_tid=8, entry_tid=9984' '!=' 'object_number=57, tag_tid=8, entry_tid=9889' 2024-04-06T09:37:23.321 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:37:23.321 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=56, tag_tid=8, entry_tid=9984' = 'object_number=57, tag_tid=8, entry_tid=9889' 2024-04-06T09:37:23.321 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=56, tag_tid=8, entry_tid=9984' 2024-04-06T09:37:23.321 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=57, tag_tid=9, entry_tid=1' 2024-04-06T09:37:23.321 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:37:23.321 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:37:23.321 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=9 2024-04-06T09:37:23.322 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=56, tag_tid=8, entry_tid=9984' 2024-04-06T09:37:23.322 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:37:23.322 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:37:23.323 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=8 2024-04-06T09:37:23.324 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=57, tag_tid=9, entry_tid=1' 2024-04-06T09:37:23.324 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:37:23.324 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:37:23.325 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:37:23.326 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=56, tag_tid=8, entry_tid=9984' 2024-04-06T09:37:23.326 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:37:23.326 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:37:23.327 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=9984 2024-04-06T09:37:23.327 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 9 = 8 -a 1 -le 9984 2024-04-06T09:37:23.327 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:37:23.327 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:37:23.327 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:37:23.529 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:37:23.529 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:37:23.529 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:37:23.529 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:37:23.529 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:37:23.529 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:37:23.529 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:37:23.529 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:37:23.529 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:37:23.529 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:37:23.529 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:37:23.529 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:37:23.529 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:37:23.529 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:37:23.530 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:37:23.530 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:37:23.530 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:37:23.533 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:37:23.533 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:37:23.533 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:37:23.533 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:37:23.533 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:37:23.533 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:37:23.533 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:37:23.533 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:37:23.533 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:37:23.533 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:37:23.533 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:37:23.533 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:37:23.533 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:37:23.533 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:37:23.534 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:37:23.534 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:37:23.535 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:37:23.535 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:37:23.535 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:37:23.535 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:37:23.535 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:37:23.535 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:37:23.535 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:37:23.535 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:37:23.535 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:37:23.535 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:37:23.535 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:37:23.535 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:37:23.535 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:37:23.535 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:37:23.536 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:37:23.536 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:37:23.560 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:37:23.560 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:37:23.560 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:37:23.560 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:37:23.689 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:37:23.689 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:37:23.689 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:37:23.689 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:37:23.690 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:37:23.690 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:37:23.690 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:37:23.690 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:37:23.690 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:37:23.690 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:37:23.691 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:37:23.691 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:37:23.691 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:37:23.692 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:37:23.692 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:37:23.692 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:37:23.742 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 14 2024-04-06T09:37:23.742 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 14 2024-04-06T09:37:23.742 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:37:23.743 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=57, tag_tid=9, entry_tid=1], [object_number=56, tag_tid=9, entry_tid=0], [object_number=59, tag_tid=8, entry_tid=11939], [object_number=58, tag_tid=8, entry_tid=11938]]], state=connected] 2024-04-06T09:37:23.743 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=59, tag_tid=8, entry_tid=10079], [object_number=58, tag_tid=8, entry_tid=10078], [object_number=57, tag_tid=8, entry_tid=10077], [object_number=56, tag_tid=8, entry_tid=10076]]], state=connected] 2024-04-06T09:37:23.746 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:37:23.747 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=57, tag_tid=9, entry_tid=1' 2024-04-06T09:37:23.749 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:37:23.749 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:37:23.749 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:37:23.749 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:37:23.749 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:37:23.749 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:37:23.749 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:37:23.749 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:37:23.749 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:37:23.749 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:37:23.749 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:37:23.749 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:37:23.750 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:37:23.750 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:37:23.750 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:37:23.796 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 14 2024-04-06T09:37:23.796 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 14 2024-04-06T09:37:23.796 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:37:23.796 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=57, tag_tid=9, entry_tid=1], [object_number=56, tag_tid=9, entry_tid=0], [object_number=59, tag_tid=8, entry_tid=11939], [object_number=58, tag_tid=8, entry_tid=11938]]], state=connected] 2024-04-06T09:37:23.796 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=59, tag_tid=8, entry_tid=10079], [object_number=58, tag_tid=8, entry_tid=10078], [object_number=57, tag_tid=8, entry_tid=10077], [object_number=56, tag_tid=8, entry_tid=10076]]], state=connected] 2024-04-06T09:37:23.799 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:37:23.801 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=59, tag_tid=8, entry_tid=10079' 2024-04-06T09:37:23.801 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=57, tag_tid=9, entry_tid=1' -a 'object_number=57, tag_tid=9, entry_tid=1' = 'object_number=59, tag_tid=8, entry_tid=10079' 2024-04-06T09:37:23.801 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=59, tag_tid=8, entry_tid=10079' '!=' 'object_number=56, tag_tid=8, entry_tid=9984' 2024-04-06T09:37:23.801 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:37:23.801 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=59, tag_tid=8, entry_tid=10079' = 'object_number=56, tag_tid=8, entry_tid=9984' 2024-04-06T09:37:23.801 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=59, tag_tid=8, entry_tid=10079' 2024-04-06T09:37:23.802 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=57, tag_tid=9, entry_tid=1' 2024-04-06T09:37:23.802 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:37:23.802 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:37:23.803 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=9 2024-04-06T09:37:23.803 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:37:23.803 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=59, tag_tid=8, entry_tid=10079' 2024-04-06T09:37:23.804 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:37:23.804 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=8 2024-04-06T09:37:23.805 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=57, tag_tid=9, entry_tid=1' 2024-04-06T09:37:23.805 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:37:23.805 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:37:23.806 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:37:23.807 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=59, tag_tid=8, entry_tid=10079' 2024-04-06T09:37:23.807 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:37:23.807 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:37:23.808 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=10079 2024-04-06T09:37:23.808 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 9 = 8 -a 1 -le 10079 2024-04-06T09:37:23.808 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:37:23.808 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:37:23.808 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:37:24.010 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:37:24.010 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:37:24.010 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:37:24.010 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:37:24.010 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:37:24.010 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:37:24.010 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:37:24.011 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:37:24.011 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:37:24.011 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:37:24.011 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:37:24.011 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:37:24.011 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:37:24.011 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:37:24.011 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:37:24.011 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:37:24.012 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:37:24.016 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:37:24.016 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:37:24.016 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:37:24.016 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:37:24.016 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:37:24.016 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:37:24.016 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:37:24.016 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:37:24.016 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:37:24.016 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:37:24.016 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:37:24.016 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:37:24.017 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:37:24.017 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:37:24.017 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:37:24.017 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:37:24.017 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:37:24.017 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:37:24.017 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:37:24.017 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:37:24.017 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:37:24.017 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:37:24.017 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:37:24.017 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:37:24.017 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:37:24.017 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:37:24.017 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:37:24.017 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:37:24.018 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:37:24.018 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:37:24.018 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:37:24.018 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:37:24.041 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:37:24.041 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:37:24.041 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:37:24.041 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:37:24.169 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:37:24.170 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:37:24.170 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:37:24.170 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:37:24.170 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:37:24.170 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:37:24.170 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:37:24.170 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:37:24.171 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:37:24.171 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:37:24.171 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:37:24.171 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:37:24.171 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:37:24.173 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:37:24.173 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:37:24.173 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:37:24.224 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 14 2024-04-06T09:37:24.224 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 14 2024-04-06T09:37:24.224 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:37:24.224 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=57, tag_tid=9, entry_tid=1], [object_number=56, tag_tid=9, entry_tid=0], [object_number=59, tag_tid=8, entry_tid=11939], [object_number=58, tag_tid=8, entry_tid=11938]]], state=connected] 2024-04-06T09:37:24.224 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=57, tag_tid=8, entry_tid=10185], [object_number=56, tag_tid=8, entry_tid=10184], [object_number=59, tag_tid=8, entry_tid=10183], [object_number=58, tag_tid=8, entry_tid=10182]]], state=connected] 2024-04-06T09:37:24.227 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:37:24.229 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=57, tag_tid=9, entry_tid=1' 2024-04-06T09:37:24.230 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:37:24.230 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:37:24.230 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:37:24.230 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:37:24.230 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:37:24.230 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:37:24.230 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:37:24.230 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:37:24.231 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:37:24.231 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:37:24.231 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:37:24.231 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:37:24.231 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:37:24.231 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:37:24.231 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:37:24.278 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 14 2024-04-06T09:37:24.278 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 14 2024-04-06T09:37:24.278 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:37:24.278 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=57, tag_tid=9, entry_tid=1], [object_number=56, tag_tid=9, entry_tid=0], [object_number=59, tag_tid=8, entry_tid=11939], [object_number=58, tag_tid=8, entry_tid=11938]]], state=connected] 2024-04-06T09:37:24.278 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=57, tag_tid=8, entry_tid=10185], [object_number=56, tag_tid=8, entry_tid=10184], [object_number=59, tag_tid=8, entry_tid=10183], [object_number=58, tag_tid=8, entry_tid=10182]]], state=connected] 2024-04-06T09:37:24.282 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:37:24.283 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=57, tag_tid=8, entry_tid=10185' 2024-04-06T09:37:24.284 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=57, tag_tid=9, entry_tid=1' -a 'object_number=57, tag_tid=9, entry_tid=1' = 'object_number=57, tag_tid=8, entry_tid=10185' 2024-04-06T09:37:24.284 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=57, tag_tid=8, entry_tid=10185' '!=' 'object_number=59, tag_tid=8, entry_tid=10079' 2024-04-06T09:37:24.284 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:37:24.284 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=57, tag_tid=8, entry_tid=10185' = 'object_number=59, tag_tid=8, entry_tid=10079' 2024-04-06T09:37:24.284 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=57, tag_tid=8, entry_tid=10185' 2024-04-06T09:37:24.284 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=57, tag_tid=9, entry_tid=1' 2024-04-06T09:37:24.285 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:37:24.285 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:37:24.285 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=9 2024-04-06T09:37:24.286 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=57, tag_tid=8, entry_tid=10185' 2024-04-06T09:37:24.286 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:37:24.286 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:37:24.287 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=8 2024-04-06T09:37:24.288 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=57, tag_tid=9, entry_tid=1' 2024-04-06T09:37:24.288 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:37:24.288 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:37:24.288 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:37:24.289 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=57, tag_tid=8, entry_tid=10185' 2024-04-06T09:37:24.289 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:37:24.289 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:37:24.291 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=10185 2024-04-06T09:37:24.291 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 9 = 8 -a 1 -le 10185 2024-04-06T09:37:24.291 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:37:24.291 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:37:24.291 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:37:24.494 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:37:24.495 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:37:24.495 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:37:24.495 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:37:24.495 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:37:24.495 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:37:24.496 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:37:24.496 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:37:24.496 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:37:24.496 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:37:24.496 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:37:24.496 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:37:24.496 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:37:24.497 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:37:24.497 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:37:24.497 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:37:24.497 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:37:24.503 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:37:24.503 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:37:24.503 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:37:24.503 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:37:24.504 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:37:24.504 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:37:24.504 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:37:24.504 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:37:24.504 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:37:24.504 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:37:24.505 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:37:24.505 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:37:24.505 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:37:24.505 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:37:24.505 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:37:24.505 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:37:24.505 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:37:24.506 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:37:24.506 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:37:24.506 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:37:24.506 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:37:24.506 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:37:24.507 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:37:24.507 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:37:24.507 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:37:24.507 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:37:24.507 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:37:24.507 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:37:24.507 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:37:24.508 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:37:24.508 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:37:24.508 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:37:24.523 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:37:24.523 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:37:24.523 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:37:24.523 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:37:24.648 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:37:24.648 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:37:24.648 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:37:24.649 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:37:24.649 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:37:24.649 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:37:24.649 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:37:24.649 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:37:24.649 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:37:24.649 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:37:24.650 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:37:24.650 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:37:24.650 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:37:24.651 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:37:24.651 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:37:24.652 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:37:24.696 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 14 2024-04-06T09:37:24.696 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 14 2024-04-06T09:37:24.697 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:37:24.697 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=57, tag_tid=9, entry_tid=1], [object_number=56, tag_tid=9, entry_tid=0], [object_number=59, tag_tid=8, entry_tid=11939], [object_number=58, tag_tid=8, entry_tid=11938]]], state=connected] 2024-04-06T09:37:24.697 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=59, tag_tid=8, entry_tid=10291], [object_number=58, tag_tid=8, entry_tid=10290], [object_number=57, tag_tid=8, entry_tid=10289], [object_number=56, tag_tid=8, entry_tid=10288]]], state=connected] 2024-04-06T09:37:24.700 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:37:24.702 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=57, tag_tid=9, entry_tid=1' 2024-04-06T09:37:24.703 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:37:24.703 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:37:24.703 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:37:24.703 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:37:24.703 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:37:24.703 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:37:24.703 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:37:24.703 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:37:24.704 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:37:24.704 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:37:24.704 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:37:24.704 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:37:24.704 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:37:24.705 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:37:24.705 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:37:24.749 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 14 2024-04-06T09:37:24.749 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 14 2024-04-06T09:37:24.749 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:37:24.749 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=57, tag_tid=9, entry_tid=1], [object_number=56, tag_tid=9, entry_tid=0], [object_number=59, tag_tid=8, entry_tid=11939], [object_number=58, tag_tid=8, entry_tid=11938]]], state=connected] 2024-04-06T09:37:24.750 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=59, tag_tid=8, entry_tid=10291], [object_number=58, tag_tid=8, entry_tid=10290], [object_number=57, tag_tid=8, entry_tid=10289], [object_number=56, tag_tid=8, entry_tid=10288]]], state=connected] 2024-04-06T09:37:24.753 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:37:24.754 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=59, tag_tid=8, entry_tid=10291' 2024-04-06T09:37:24.754 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=57, tag_tid=9, entry_tid=1' -a 'object_number=57, tag_tid=9, entry_tid=1' = 'object_number=59, tag_tid=8, entry_tid=10291' 2024-04-06T09:37:24.754 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=59, tag_tid=8, entry_tid=10291' '!=' 'object_number=57, tag_tid=8, entry_tid=10185' 2024-04-06T09:37:24.755 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:37:24.755 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=59, tag_tid=8, entry_tid=10291' = 'object_number=57, tag_tid=8, entry_tid=10185' 2024-04-06T09:37:24.755 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=59, tag_tid=8, entry_tid=10291' 2024-04-06T09:37:24.755 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=57, tag_tid=9, entry_tid=1' 2024-04-06T09:37:24.755 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:37:24.755 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:37:24.756 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=9 2024-04-06T09:37:24.756 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=59, tag_tid=8, entry_tid=10291' 2024-04-06T09:37:24.757 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:37:24.757 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:37:24.757 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=8 2024-04-06T09:37:24.758 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=57, tag_tid=9, entry_tid=1' 2024-04-06T09:37:24.758 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:37:24.758 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:37:24.759 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:37:24.760 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=59, tag_tid=8, entry_tid=10291' 2024-04-06T09:37:24.760 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:37:24.760 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:37:24.761 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=10291 2024-04-06T09:37:24.761 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 9 = 8 -a 1 -le 10291 2024-04-06T09:37:24.761 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:37:24.761 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:37:24.762 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:37:24.963 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:37:24.963 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:37:24.963 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:37:24.964 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:37:24.964 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:37:24.964 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:37:24.964 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:37:24.964 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:37:24.964 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:37:24.964 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:37:24.964 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:37:24.964 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:37:24.964 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:37:24.964 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:37:24.964 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:37:24.965 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:37:24.965 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:37:24.967 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:37:24.967 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:37:24.967 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:37:24.967 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:37:24.967 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:37:24.967 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:37:24.967 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:37:24.967 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:37:24.968 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:37:24.968 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:37:24.968 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:37:24.968 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:37:24.968 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:37:24.968 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:37:24.968 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:37:24.968 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:37:24.969 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:37:24.969 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:37:24.969 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:37:24.969 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:37:24.969 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:37:24.969 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:37:24.969 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:37:24.969 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:37:24.969 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:37:24.969 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:37:24.970 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:37:24.970 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:37:24.970 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:37:24.970 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:37:24.970 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:37:24.970 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:37:24.994 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:37:24.994 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:37:24.995 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:37:24.995 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:37:25.121 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:37:25.124 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:37:25.124 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:37:25.124 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:37:25.124 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:37:25.124 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:37:25.124 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:37:25.125 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:37:25.125 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:37:25.125 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:37:25.125 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:37:25.125 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:37:25.125 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:37:25.126 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:37:25.127 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:37:25.127 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:37:25.171 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 14 2024-04-06T09:37:25.171 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 14 2024-04-06T09:37:25.171 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:37:25.171 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=57, tag_tid=9, entry_tid=1], [object_number=56, tag_tid=9, entry_tid=0], [object_number=59, tag_tid=8, entry_tid=11939], [object_number=58, tag_tid=8, entry_tid=11938]]], state=connected] 2024-04-06T09:37:25.172 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=58, tag_tid=8, entry_tid=10394], [object_number=57, tag_tid=8, entry_tid=10393], [object_number=56, tag_tid=8, entry_tid=10392], [object_number=59, tag_tid=8, entry_tid=10391]]], state=connected] 2024-04-06T09:37:25.175 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:37:25.176 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=57, tag_tid=9, entry_tid=1' 2024-04-06T09:37:25.177 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:37:25.177 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:37:25.177 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:37:25.177 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:37:25.177 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:37:25.177 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:37:25.177 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:37:25.178 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:37:25.178 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:37:25.178 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:37:25.178 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:37:25.178 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:37:25.178 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:37:25.179 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:37:25.179 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:37:25.224 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 14 2024-04-06T09:37:25.224 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 14 2024-04-06T09:37:25.224 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:37:25.224 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=57, tag_tid=9, entry_tid=1], [object_number=56, tag_tid=9, entry_tid=0], [object_number=59, tag_tid=8, entry_tid=11939], [object_number=58, tag_tid=8, entry_tid=11938]]], state=connected] 2024-04-06T09:37:25.224 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=58, tag_tid=8, entry_tid=10394], [object_number=57, tag_tid=8, entry_tid=10393], [object_number=56, tag_tid=8, entry_tid=10392], [object_number=59, tag_tid=8, entry_tid=10391]]], state=connected] 2024-04-06T09:37:25.228 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:37:25.230 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=58, tag_tid=8, entry_tid=10394' 2024-04-06T09:37:25.230 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=57, tag_tid=9, entry_tid=1' -a 'object_number=57, tag_tid=9, entry_tid=1' = 'object_number=58, tag_tid=8, entry_tid=10394' 2024-04-06T09:37:25.230 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=58, tag_tid=8, entry_tid=10394' '!=' 'object_number=59, tag_tid=8, entry_tid=10291' 2024-04-06T09:37:25.230 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:37:25.230 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=58, tag_tid=8, entry_tid=10394' = 'object_number=59, tag_tid=8, entry_tid=10291' 2024-04-06T09:37:25.230 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=58, tag_tid=8, entry_tid=10394' 2024-04-06T09:37:25.230 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=57, tag_tid=9, entry_tid=1' 2024-04-06T09:37:25.230 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:37:25.230 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:37:25.231 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=9 2024-04-06T09:37:25.232 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=58, tag_tid=8, entry_tid=10394' 2024-04-06T09:37:25.232 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:37:25.232 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:37:25.232 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=8 2024-04-06T09:37:25.233 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=57, tag_tid=9, entry_tid=1' 2024-04-06T09:37:25.233 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:37:25.233 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:37:25.234 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:37:25.235 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=58, tag_tid=8, entry_tid=10394' 2024-04-06T09:37:25.235 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:37:25.235 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:37:25.236 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=10394 2024-04-06T09:37:25.236 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 9 = 8 -a 1 -le 10394 2024-04-06T09:37:25.237 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:37:25.237 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:37:25.237 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:37:25.438 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:37:25.438 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:37:25.438 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:37:25.438 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:37:25.438 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:37:25.438 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:37:25.438 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:37:25.439 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:37:25.439 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:37:25.439 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:37:25.439 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:37:25.439 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:37:25.439 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:37:25.439 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:37:25.439 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:37:25.439 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:37:25.440 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:37:25.444 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:37:25.444 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:37:25.444 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:37:25.444 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:37:25.444 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:37:25.444 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:37:25.444 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:37:25.444 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:37:25.444 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:37:25.444 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:37:25.444 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:37:25.444 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:37:25.444 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:37:25.444 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:37:25.445 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:37:25.445 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:37:25.445 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:37:25.445 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:37:25.445 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:37:25.445 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:37:25.445 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:37:25.445 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:37:25.445 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:37:25.445 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:37:25.445 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:37:25.445 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:37:25.445 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:37:25.445 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:37:25.445 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:37:25.445 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:37:25.446 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:37:25.446 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:37:25.468 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:37:25.469 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:37:25.469 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:37:25.469 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:37:25.595 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:37:25.596 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:37:25.596 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:37:25.596 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:37:25.596 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:37:25.597 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:37:25.597 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:37:25.597 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:37:25.597 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:37:25.597 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:37:25.597 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:37:25.598 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:37:25.598 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:37:25.598 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:37:25.599 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:37:25.599 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:37:25.650 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 14 2024-04-06T09:37:25.651 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 14 2024-04-06T09:37:25.651 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:37:25.651 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=57, tag_tid=9, entry_tid=1], [object_number=56, tag_tid=9, entry_tid=0], [object_number=59, tag_tid=8, entry_tid=11939], [object_number=58, tag_tid=8, entry_tid=11938]]], state=connected] 2024-04-06T09:37:25.651 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=56, tag_tid=8, entry_tid=10508], [object_number=59, tag_tid=8, entry_tid=10507], [object_number=58, tag_tid=8, entry_tid=10506], [object_number=57, tag_tid=8, entry_tid=10505]]], state=connected] 2024-04-06T09:37:25.654 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:37:25.655 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=57, tag_tid=9, entry_tid=1' 2024-04-06T09:37:25.656 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:37:25.656 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:37:25.656 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:37:25.657 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:37:25.657 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:37:25.657 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:37:25.657 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:37:25.657 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:37:25.657 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:37:25.657 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:37:25.657 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:37:25.657 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:37:25.658 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:37:25.658 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:37:25.658 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:37:25.703 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 14 2024-04-06T09:37:25.704 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 14 2024-04-06T09:37:25.704 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:37:25.704 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=57, tag_tid=9, entry_tid=1], [object_number=56, tag_tid=9, entry_tid=0], [object_number=59, tag_tid=8, entry_tid=11939], [object_number=58, tag_tid=8, entry_tid=11938]]], state=connected] 2024-04-06T09:37:25.704 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=56, tag_tid=8, entry_tid=10508], [object_number=59, tag_tid=8, entry_tid=10507], [object_number=58, tag_tid=8, entry_tid=10506], [object_number=57, tag_tid=8, entry_tid=10505]]], state=connected] 2024-04-06T09:37:25.707 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:37:25.708 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=56, tag_tid=8, entry_tid=10508' 2024-04-06T09:37:25.708 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=57, tag_tid=9, entry_tid=1' -a 'object_number=57, tag_tid=9, entry_tid=1' = 'object_number=56, tag_tid=8, entry_tid=10508' 2024-04-06T09:37:25.709 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=56, tag_tid=8, entry_tid=10508' '!=' 'object_number=58, tag_tid=8, entry_tid=10394' 2024-04-06T09:37:25.709 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:37:25.709 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=56, tag_tid=8, entry_tid=10508' = 'object_number=58, tag_tid=8, entry_tid=10394' 2024-04-06T09:37:25.709 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=56, tag_tid=8, entry_tid=10508' 2024-04-06T09:37:25.709 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=57, tag_tid=9, entry_tid=1' 2024-04-06T09:37:25.710 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:37:25.710 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:37:25.711 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=9 2024-04-06T09:37:25.711 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=56, tag_tid=8, entry_tid=10508' 2024-04-06T09:37:25.711 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:37:25.711 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:37:25.712 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=8 2024-04-06T09:37:25.712 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=57, tag_tid=9, entry_tid=1' 2024-04-06T09:37:25.712 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:37:25.712 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:37:25.713 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:37:25.714 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=56, tag_tid=8, entry_tid=10508' 2024-04-06T09:37:25.714 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:37:25.715 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:37:25.716 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=10508 2024-04-06T09:37:25.716 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 9 = 8 -a 1 -le 10508 2024-04-06T09:37:25.716 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:37:25.716 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:37:25.716 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:37:25.917 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:37:25.917 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:37:25.918 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:37:25.918 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:37:25.918 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:37:25.918 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:37:25.918 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:37:25.918 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:37:25.918 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:37:25.918 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:37:25.918 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:37:25.918 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:37:25.918 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:37:25.918 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:37:25.918 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:37:25.921 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:37:25.921 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:37:25.921 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:37:25.921 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:37:25.921 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:37:25.921 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:37:25.921 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:37:25.921 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:37:25.921 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:37:25.921 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:37:25.921 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:37:25.921 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:37:25.921 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:37:25.921 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:37:25.922 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:37:25.922 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:37:25.922 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:37:25.922 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:37:25.922 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:37:25.922 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:37:25.922 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:37:25.922 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:37:25.922 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:37:25.922 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:37:25.922 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:37:25.922 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:37:25.922 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:37:25.922 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:37:25.923 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:37:25.923 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:37:25.923 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:37:25.923 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:37:25.923 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:37:25.923 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:37:25.947 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:37:25.947 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:37:25.947 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:37:25.947 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:37:26.077 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:37:26.079 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:37:26.079 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:37:26.079 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:37:26.079 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:37:26.079 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:37:26.079 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:37:26.079 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:37:26.079 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:37:26.079 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:37:26.079 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:37:26.080 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:37:26.080 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:37:26.081 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:37:26.081 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:37:26.081 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:37:26.131 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 14 2024-04-06T09:37:26.131 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 14 2024-04-06T09:37:26.131 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:37:26.131 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=57, tag_tid=9, entry_tid=1], [object_number=56, tag_tid=9, entry_tid=0], [object_number=59, tag_tid=8, entry_tid=11939], [object_number=58, tag_tid=8, entry_tid=11938]]], state=connected] 2024-04-06T09:37:26.131 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=59, tag_tid=8, entry_tid=10611], [object_number=58, tag_tid=8, entry_tid=10610], [object_number=57, tag_tid=8, entry_tid=10609], [object_number=56, tag_tid=8, entry_tid=10608]]], state=connected] 2024-04-06T09:37:26.134 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:37:26.136 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=57, tag_tid=9, entry_tid=1' 2024-04-06T09:37:26.137 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:37:26.137 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:37:26.137 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:37:26.137 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:37:26.137 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:37:26.138 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:37:26.138 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:37:26.138 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:37:26.138 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:37:26.138 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:37:26.138 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:37:26.138 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:37:26.139 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:37:26.139 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:37:26.139 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:37:26.185 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 14 2024-04-06T09:37:26.185 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 14 2024-04-06T09:37:26.185 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:37:26.185 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=57, tag_tid=9, entry_tid=1], [object_number=56, tag_tid=9, entry_tid=0], [object_number=59, tag_tid=8, entry_tid=11939], [object_number=58, tag_tid=8, entry_tid=11938]]], state=connected] 2024-04-06T09:37:26.185 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=59, tag_tid=8, entry_tid=10611], [object_number=58, tag_tid=8, entry_tid=10610], [object_number=57, tag_tid=8, entry_tid=10609], [object_number=56, tag_tid=8, entry_tid=10608]]], state=connected] 2024-04-06T09:37:26.189 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:37:26.191 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=59, tag_tid=8, entry_tid=10611' 2024-04-06T09:37:26.191 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=57, tag_tid=9, entry_tid=1' -a 'object_number=57, tag_tid=9, entry_tid=1' = 'object_number=59, tag_tid=8, entry_tid=10611' 2024-04-06T09:37:26.191 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=59, tag_tid=8, entry_tid=10611' '!=' 'object_number=56, tag_tid=8, entry_tid=10508' 2024-04-06T09:37:26.191 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:37:26.191 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=59, tag_tid=8, entry_tid=10611' = 'object_number=56, tag_tid=8, entry_tid=10508' 2024-04-06T09:37:26.191 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=59, tag_tid=8, entry_tid=10611' 2024-04-06T09:37:26.191 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=57, tag_tid=9, entry_tid=1' 2024-04-06T09:37:26.191 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:37:26.191 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:37:26.192 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=9 2024-04-06T09:37:26.193 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=59, tag_tid=8, entry_tid=10611' 2024-04-06T09:37:26.193 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:37:26.193 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:37:26.194 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=8 2024-04-06T09:37:26.194 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=57, tag_tid=9, entry_tid=1' 2024-04-06T09:37:26.194 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:37:26.194 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:37:26.195 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:37:26.196 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=59, tag_tid=8, entry_tid=10611' 2024-04-06T09:37:26.196 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:37:26.196 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:37:26.198 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=10611 2024-04-06T09:37:26.198 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 9 = 8 -a 1 -le 10611 2024-04-06T09:37:26.198 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:37:26.198 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:37:26.198 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:37:26.399 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:37:26.399 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:37:26.399 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:37:26.399 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:37:26.399 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:37:26.399 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:37:26.399 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:37:26.399 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:37:26.399 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:37:26.399 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:37:26.399 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:37:26.399 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:37:26.400 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:37:26.400 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:37:26.400 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:37:26.400 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:37:26.401 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:37:26.404 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:37:26.404 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:37:26.404 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:37:26.404 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:37:26.404 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:37:26.404 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:37:26.405 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:37:26.405 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:37:26.405 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:37:26.405 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:37:26.405 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:37:26.405 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:37:26.405 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:37:26.405 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:37:26.405 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:37:26.405 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:37:26.405 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:37:26.405 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:37:26.405 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:37:26.405 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:37:26.405 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:37:26.405 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:37:26.406 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:37:26.406 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:37:26.406 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:37:26.406 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:37:26.406 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:37:26.406 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:37:26.406 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:37:26.406 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:37:26.406 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:37:26.406 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:37:26.429 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:37:26.429 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:37:26.429 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:37:26.429 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:37:26.561 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:37:26.562 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:37:26.563 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:37:26.563 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:37:26.563 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:37:26.563 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:37:26.563 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:37:26.563 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:37:26.563 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:37:26.563 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:37:26.563 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:37:26.564 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:37:26.564 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:37:26.566 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:37:26.566 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:37:26.567 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:37:26.612 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 14 2024-04-06T09:37:26.612 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 14 2024-04-06T09:37:26.612 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:37:26.612 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=57, tag_tid=9, entry_tid=1], [object_number=56, tag_tid=9, entry_tid=0], [object_number=59, tag_tid=8, entry_tid=11939], [object_number=58, tag_tid=8, entry_tid=11938]]], state=connected] 2024-04-06T09:37:26.612 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=57, tag_tid=8, entry_tid=10709], [object_number=56, tag_tid=8, entry_tid=10708], [object_number=59, tag_tid=8, entry_tid=10707], [object_number=58, tag_tid=8, entry_tid=10706]]], state=connected] 2024-04-06T09:37:26.615 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:37:26.617 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=57, tag_tid=9, entry_tid=1' 2024-04-06T09:37:26.618 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:37:26.618 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:37:26.618 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:37:26.618 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:37:26.618 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:37:26.618 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:37:26.618 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:37:26.618 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:37:26.618 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:37:26.618 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:37:26.618 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:37:26.618 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:37:26.619 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:37:26.619 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:37:26.619 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:37:26.664 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 14 2024-04-06T09:37:26.664 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 14 2024-04-06T09:37:26.665 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:37:26.665 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=57, tag_tid=9, entry_tid=1], [object_number=56, tag_tid=9, entry_tid=0], [object_number=59, tag_tid=8, entry_tid=11939], [object_number=58, tag_tid=8, entry_tid=11938]]], state=connected] 2024-04-06T09:37:26.665 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=57, tag_tid=8, entry_tid=10709], [object_number=56, tag_tid=8, entry_tid=10708], [object_number=59, tag_tid=8, entry_tid=10707], [object_number=58, tag_tid=8, entry_tid=10706]]], state=connected] 2024-04-06T09:37:26.668 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:37:26.670 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=57, tag_tid=8, entry_tid=10709' 2024-04-06T09:37:26.670 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=57, tag_tid=9, entry_tid=1' -a 'object_number=57, tag_tid=9, entry_tid=1' = 'object_number=57, tag_tid=8, entry_tid=10709' 2024-04-06T09:37:26.670 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=57, tag_tid=8, entry_tid=10709' '!=' 'object_number=59, tag_tid=8, entry_tid=10611' 2024-04-06T09:37:26.670 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:37:26.670 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=57, tag_tid=8, entry_tid=10709' = 'object_number=59, tag_tid=8, entry_tid=10611' 2024-04-06T09:37:26.670 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=57, tag_tid=8, entry_tid=10709' 2024-04-06T09:37:26.670 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=57, tag_tid=9, entry_tid=1' 2024-04-06T09:37:26.671 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:37:26.671 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:37:26.671 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=9 2024-04-06T09:37:26.672 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=57, tag_tid=8, entry_tid=10709' 2024-04-06T09:37:26.672 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:37:26.672 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:37:26.673 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=8 2024-04-06T09:37:26.674 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=57, tag_tid=9, entry_tid=1' 2024-04-06T09:37:26.674 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:37:26.674 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:37:26.675 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:37:26.675 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=57, tag_tid=8, entry_tid=10709' 2024-04-06T09:37:26.675 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:37:26.675 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:37:26.676 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=10709 2024-04-06T09:37:26.677 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 9 = 8 -a 1 -le 10709 2024-04-06T09:37:26.677 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:37:26.677 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:37:26.677 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:37:26.879 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:37:26.879 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:37:26.879 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:37:26.879 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:37:26.879 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:37:26.879 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:37:26.880 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:37:26.880 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:37:26.880 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:37:26.880 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:37:26.880 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:37:26.881 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:37:26.881 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:37:26.881 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:37:26.881 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:37:26.885 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:37:26.885 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:37:26.885 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:37:26.885 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:37:26.886 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:37:26.886 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:37:26.886 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:37:26.886 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:37:26.886 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:37:26.886 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:37:26.887 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:37:26.887 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:37:26.887 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:37:26.887 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:37:26.887 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:37:26.887 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:37:26.887 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:37:26.888 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:37:26.888 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:37:26.888 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:37:26.888 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:37:26.888 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:37:26.888 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:37:26.888 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:37:26.889 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:37:26.889 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:37:26.889 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:37:26.889 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:37:26.889 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:37:26.889 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:37:26.890 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:37:26.890 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:37:26.890 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:37:26.890 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:37:26.909 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:37:26.910 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:37:26.911 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:37:26.911 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:37:27.037 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:37:27.039 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:37:27.039 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:37:27.039 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:37:27.039 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:37:27.039 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:37:27.039 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:37:27.039 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:37:27.039 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:37:27.039 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:37:27.040 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:37:27.040 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:37:27.040 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:37:27.042 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:37:27.042 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:37:27.042 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:37:27.088 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 14 2024-04-06T09:37:27.089 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 14 2024-04-06T09:37:27.089 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:37:27.089 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=57, tag_tid=9, entry_tid=1], [object_number=56, tag_tid=9, entry_tid=0], [object_number=59, tag_tid=8, entry_tid=11939], [object_number=58, tag_tid=8, entry_tid=11938]]], state=connected] 2024-04-06T09:37:27.089 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=58, tag_tid=8, entry_tid=10806], [object_number=57, tag_tid=8, entry_tid=10805], [object_number=56, tag_tid=8, entry_tid=10804], [object_number=59, tag_tid=8, entry_tid=10803]]], state=connected] 2024-04-06T09:37:27.092 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:37:27.093 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=57, tag_tid=9, entry_tid=1' 2024-04-06T09:37:27.094 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:37:27.095 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:37:27.095 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:37:27.095 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:37:27.095 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:37:27.095 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:37:27.095 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:37:27.095 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:37:27.095 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:37:27.095 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:37:27.095 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:37:27.095 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:37:27.096 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:37:27.096 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:37:27.096 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:37:27.141 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 14 2024-04-06T09:37:27.141 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 14 2024-04-06T09:37:27.142 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:37:27.142 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=57, tag_tid=9, entry_tid=1], [object_number=56, tag_tid=9, entry_tid=0], [object_number=59, tag_tid=8, entry_tid=11939], [object_number=58, tag_tid=8, entry_tid=11938]]], state=connected] 2024-04-06T09:37:27.142 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=58, tag_tid=8, entry_tid=10806], [object_number=57, tag_tid=8, entry_tid=10805], [object_number=56, tag_tid=8, entry_tid=10804], [object_number=59, tag_tid=8, entry_tid=10803]]], state=connected] 2024-04-06T09:37:27.145 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:37:27.147 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=58, tag_tid=8, entry_tid=10806' 2024-04-06T09:37:27.147 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=57, tag_tid=9, entry_tid=1' -a 'object_number=57, tag_tid=9, entry_tid=1' = 'object_number=58, tag_tid=8, entry_tid=10806' 2024-04-06T09:37:27.147 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=58, tag_tid=8, entry_tid=10806' '!=' 'object_number=57, tag_tid=8, entry_tid=10709' 2024-04-06T09:37:27.147 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:37:27.147 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=58, tag_tid=8, entry_tid=10806' = 'object_number=57, tag_tid=8, entry_tid=10709' 2024-04-06T09:37:27.147 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=58, tag_tid=8, entry_tid=10806' 2024-04-06T09:37:27.147 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=57, tag_tid=9, entry_tid=1' 2024-04-06T09:37:27.148 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:37:27.148 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:37:27.148 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=9 2024-04-06T09:37:27.149 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=58, tag_tid=8, entry_tid=10806' 2024-04-06T09:37:27.149 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:37:27.149 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:37:27.151 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=8 2024-04-06T09:37:27.151 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=57, tag_tid=9, entry_tid=1' 2024-04-06T09:37:27.151 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:37:27.151 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:37:27.152 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:37:27.152 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=58, tag_tid=8, entry_tid=10806' 2024-04-06T09:37:27.152 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:37:27.152 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:37:27.154 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=10806 2024-04-06T09:37:27.154 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 9 = 8 -a 1 -le 10806 2024-04-06T09:37:27.154 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:37:27.154 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:37:27.154 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:37:27.356 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:37:27.356 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:37:27.356 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:37:27.356 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:37:27.356 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:37:27.356 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:37:27.356 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:37:27.356 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:37:27.356 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:37:27.357 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:37:27.357 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:37:27.357 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:37:27.357 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:37:27.357 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:37:27.357 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:37:27.358 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:37:27.358 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:37:27.361 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:37:27.361 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:37:27.361 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:37:27.361 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:37:27.361 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:37:27.361 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:37:27.361 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:37:27.361 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:37:27.361 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:37:27.361 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:37:27.361 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:37:27.361 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:37:27.362 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:37:27.362 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:37:27.362 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:37:27.362 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:37:27.362 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:37:27.362 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:37:27.362 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:37:27.362 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:37:27.362 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:37:27.362 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:37:27.362 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:37:27.362 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:37:27.362 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:37:27.362 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:37:27.363 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:37:27.363 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:37:27.363 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:37:27.363 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:37:27.363 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:37:27.363 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:37:27.387 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:37:27.387 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:37:27.387 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:37:27.387 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:37:27.519 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:37:27.520 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:37:27.521 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:37:27.521 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:37:27.521 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:37:27.521 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:37:27.521 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:37:27.521 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:37:27.521 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:37:27.521 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:37:27.522 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:37:27.522 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:37:27.522 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:37:27.523 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:37:27.523 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:37:27.523 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:37:27.569 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 14 2024-04-06T09:37:27.569 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 14 2024-04-06T09:37:27.569 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:37:27.569 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=57, tag_tid=9, entry_tid=1], [object_number=56, tag_tid=9, entry_tid=0], [object_number=59, tag_tid=8, entry_tid=11939], [object_number=58, tag_tid=8, entry_tid=11938]]], state=connected] 2024-04-06T09:37:27.569 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=58, tag_tid=8, entry_tid=10898], [object_number=57, tag_tid=8, entry_tid=10897], [object_number=56, tag_tid=8, entry_tid=10896], [object_number=59, tag_tid=8, entry_tid=10895]]], state=connected] 2024-04-06T09:37:27.572 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:37:27.573 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=57, tag_tid=9, entry_tid=1' 2024-04-06T09:37:27.575 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:37:27.575 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:37:27.575 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:37:27.575 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:37:27.575 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:37:27.575 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:37:27.575 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:37:27.575 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:37:27.575 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:37:27.575 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:37:27.575 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:37:27.575 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:37:27.576 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:37:27.576 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:37:27.576 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:37:27.621 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 14 2024-04-06T09:37:27.621 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 14 2024-04-06T09:37:27.621 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:37:27.621 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=57, tag_tid=9, entry_tid=1], [object_number=56, tag_tid=9, entry_tid=0], [object_number=59, tag_tid=8, entry_tid=11939], [object_number=58, tag_tid=8, entry_tid=11938]]], state=connected] 2024-04-06T09:37:27.621 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=58, tag_tid=8, entry_tid=10898], [object_number=57, tag_tid=8, entry_tid=10897], [object_number=56, tag_tid=8, entry_tid=10896], [object_number=59, tag_tid=8, entry_tid=10895]]], state=connected] 2024-04-06T09:37:27.624 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:37:27.627 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=58, tag_tid=8, entry_tid=10898' 2024-04-06T09:37:27.627 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=57, tag_tid=9, entry_tid=1' -a 'object_number=57, tag_tid=9, entry_tid=1' = 'object_number=58, tag_tid=8, entry_tid=10898' 2024-04-06T09:37:27.627 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=58, tag_tid=8, entry_tid=10898' '!=' 'object_number=58, tag_tid=8, entry_tid=10806' 2024-04-06T09:37:27.627 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:37:27.627 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=58, tag_tid=8, entry_tid=10898' = 'object_number=58, tag_tid=8, entry_tid=10806' 2024-04-06T09:37:27.627 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=58, tag_tid=8, entry_tid=10898' 2024-04-06T09:37:27.627 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=57, tag_tid=9, entry_tid=1' 2024-04-06T09:37:27.627 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:37:27.627 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:37:27.628 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=9 2024-04-06T09:37:27.628 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=58, tag_tid=8, entry_tid=10898' 2024-04-06T09:37:27.628 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:37:27.628 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:37:27.629 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=8 2024-04-06T09:37:27.630 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=57, tag_tid=9, entry_tid=1' 2024-04-06T09:37:27.630 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:37:27.630 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:37:27.631 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:37:27.632 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=58, tag_tid=8, entry_tid=10898' 2024-04-06T09:37:27.632 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:37:27.632 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:37:27.633 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=10898 2024-04-06T09:37:27.633 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 9 = 8 -a 1 -le 10898 2024-04-06T09:37:27.633 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:37:27.633 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:37:27.633 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:37:27.835 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:37:27.835 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:37:27.835 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:37:27.835 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:37:27.835 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:37:27.836 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:37:27.836 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:37:27.836 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:37:27.836 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:37:27.836 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:37:27.836 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:37:27.836 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:37:27.836 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:37:27.836 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:37:27.836 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:37:27.837 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:37:27.837 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:37:27.840 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:37:27.840 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:37:27.840 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:37:27.840 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:37:27.840 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:37:27.840 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:37:27.840 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:37:27.840 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:37:27.840 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:37:27.841 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:37:27.841 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:37:27.841 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:37:27.841 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:37:27.841 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:37:27.841 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:37:27.841 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:37:27.841 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:37:27.841 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:37:27.841 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:37:27.841 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:37:27.841 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:37:27.841 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:37:27.841 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:37:27.841 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:37:27.842 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:37:27.842 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:37:27.842 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:37:27.842 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:37:27.842 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:37:27.842 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:37:27.842 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:37:27.842 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:37:27.864 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:37:27.865 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:37:27.865 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:37:27.865 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:37:27.989 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:37:27.990 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:37:27.990 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:37:27.990 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:37:27.990 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:37:27.990 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:37:27.990 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:37:27.990 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:37:27.990 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:37:27.990 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:37:27.991 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:37:27.991 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:37:27.991 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:37:27.993 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:37:27.994 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:37:27.994 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:37:28.044 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 14 2024-04-06T09:37:28.044 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 14 2024-04-06T09:37:28.045 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:37:28.045 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=57, tag_tid=9, entry_tid=1], [object_number=56, tag_tid=9, entry_tid=0], [object_number=59, tag_tid=8, entry_tid=11939], [object_number=58, tag_tid=8, entry_tid=11938]]], state=connected] 2024-04-06T09:37:28.045 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=57, tag_tid=8, entry_tid=10997], [object_number=56, tag_tid=8, entry_tid=10996], [object_number=59, tag_tid=8, entry_tid=10995], [object_number=58, tag_tid=8, entry_tid=10994]]], state=connected] 2024-04-06T09:37:28.048 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:37:28.049 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=57, tag_tid=9, entry_tid=1' 2024-04-06T09:37:28.051 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:37:28.051 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:37:28.051 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:37:28.051 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:37:28.051 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:37:28.051 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:37:28.051 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:37:28.051 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:37:28.051 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:37:28.051 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:37:28.051 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:37:28.051 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:37:28.052 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:37:28.052 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:37:28.052 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:37:28.098 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 14 2024-04-06T09:37:28.098 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 14 2024-04-06T09:37:28.098 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:37:28.098 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=57, tag_tid=9, entry_tid=1], [object_number=56, tag_tid=9, entry_tid=0], [object_number=59, tag_tid=8, entry_tid=11939], [object_number=58, tag_tid=8, entry_tid=11938]]], state=connected] 2024-04-06T09:37:28.098 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=57, tag_tid=8, entry_tid=10997], [object_number=56, tag_tid=8, entry_tid=10996], [object_number=59, tag_tid=8, entry_tid=10995], [object_number=58, tag_tid=8, entry_tid=10994]]], state=connected] 2024-04-06T09:37:28.101 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:37:28.103 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=57, tag_tid=8, entry_tid=10997' 2024-04-06T09:37:28.104 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=57, tag_tid=9, entry_tid=1' -a 'object_number=57, tag_tid=9, entry_tid=1' = 'object_number=57, tag_tid=8, entry_tid=10997' 2024-04-06T09:37:28.104 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=57, tag_tid=8, entry_tid=10997' '!=' 'object_number=58, tag_tid=8, entry_tid=10898' 2024-04-06T09:37:28.104 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:37:28.104 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=57, tag_tid=8, entry_tid=10997' = 'object_number=58, tag_tid=8, entry_tid=10898' 2024-04-06T09:37:28.104 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=57, tag_tid=8, entry_tid=10997' 2024-04-06T09:37:28.104 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=57, tag_tid=9, entry_tid=1' 2024-04-06T09:37:28.104 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:37:28.104 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:37:28.104 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=9 2024-04-06T09:37:28.105 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:37:28.105 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:37:28.105 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=57, tag_tid=8, entry_tid=10997' 2024-04-06T09:37:28.106 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=8 2024-04-06T09:37:28.107 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=57, tag_tid=9, entry_tid=1' 2024-04-06T09:37:28.107 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:37:28.107 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:37:28.108 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:37:28.109 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=57, tag_tid=8, entry_tid=10997' 2024-04-06T09:37:28.109 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:37:28.109 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:37:28.110 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=10997 2024-04-06T09:37:28.110 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 9 = 8 -a 1 -le 10997 2024-04-06T09:37:28.110 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:37:28.110 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:37:28.110 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:37:28.312 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:37:28.312 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:37:28.312 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:37:28.312 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:37:28.312 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:37:28.312 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:37:28.312 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:37:28.312 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:37:28.312 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:37:28.312 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:37:28.312 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:37:28.313 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:37:28.313 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:37:28.313 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:37:28.313 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:37:28.313 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:37:28.313 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:37:28.318 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:37:28.318 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:37:28.318 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:37:28.318 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:37:28.318 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:37:28.318 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:37:28.318 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:37:28.318 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:37:28.318 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:37:28.318 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:37:28.318 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:37:28.318 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:37:28.319 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:37:28.319 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:37:28.319 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:37:28.319 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:37:28.319 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:37:28.319 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:37:28.319 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:37:28.319 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:37:28.319 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:37:28.319 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:37:28.319 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:37:28.319 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:37:28.319 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:37:28.319 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:37:28.319 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:37:28.320 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:37:28.320 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:37:28.320 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:37:28.320 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:37:28.320 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:37:28.342 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:37:28.343 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:37:28.343 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:37:28.343 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:37:28.479 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:37:28.480 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:37:28.480 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:37:28.480 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:37:28.480 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:37:28.480 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:37:28.480 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:37:28.481 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:37:28.481 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:37:28.481 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:37:28.481 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:37:28.481 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:37:28.482 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:37:28.483 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:37:28.484 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:37:28.484 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:37:28.530 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 14 2024-04-06T09:37:28.530 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 14 2024-04-06T09:37:28.530 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:37:28.530 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=57, tag_tid=9, entry_tid=1], [object_number=56, tag_tid=9, entry_tid=0], [object_number=59, tag_tid=8, entry_tid=11939], [object_number=58, tag_tid=8, entry_tid=11938]]], state=connected] 2024-04-06T09:37:28.530 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=56, tag_tid=8, entry_tid=11112], [object_number=59, tag_tid=8, entry_tid=11111], [object_number=58, tag_tid=8, entry_tid=11110], [object_number=57, tag_tid=8, entry_tid=11109]]], state=connected] 2024-04-06T09:37:28.533 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:37:28.535 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=57, tag_tid=9, entry_tid=1' 2024-04-06T09:37:28.536 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:37:28.536 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:37:28.536 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:37:28.536 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:37:28.536 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:37:28.536 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:37:28.537 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:37:28.537 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:37:28.537 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:37:28.537 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:37:28.537 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:37:28.537 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:37:28.537 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:37:28.538 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:37:28.538 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:37:28.584 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 14 2024-04-06T09:37:28.584 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 14 2024-04-06T09:37:28.584 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:37:28.584 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=57, tag_tid=9, entry_tid=1], [object_number=56, tag_tid=9, entry_tid=0], [object_number=59, tag_tid=8, entry_tid=11939], [object_number=58, tag_tid=8, entry_tid=11938]]], state=connected] 2024-04-06T09:37:28.585 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=56, tag_tid=8, entry_tid=11112], [object_number=59, tag_tid=8, entry_tid=11111], [object_number=58, tag_tid=8, entry_tid=11110], [object_number=57, tag_tid=8, entry_tid=11109]]], state=connected] 2024-04-06T09:37:28.588 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:37:28.590 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=56, tag_tid=8, entry_tid=11112' 2024-04-06T09:37:28.590 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=57, tag_tid=9, entry_tid=1' -a 'object_number=57, tag_tid=9, entry_tid=1' = 'object_number=56, tag_tid=8, entry_tid=11112' 2024-04-06T09:37:28.590 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=56, tag_tid=8, entry_tid=11112' '!=' 'object_number=57, tag_tid=8, entry_tid=10997' 2024-04-06T09:37:28.590 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:37:28.590 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=56, tag_tid=8, entry_tid=11112' = 'object_number=57, tag_tid=8, entry_tid=10997' 2024-04-06T09:37:28.590 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=56, tag_tid=8, entry_tid=11112' 2024-04-06T09:37:28.591 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=57, tag_tid=9, entry_tid=1' 2024-04-06T09:37:28.591 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:37:28.591 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:37:28.591 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=9 2024-04-06T09:37:28.592 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=56, tag_tid=8, entry_tid=11112' 2024-04-06T09:37:28.592 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:37:28.592 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:37:28.593 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=8 2024-04-06T09:37:28.594 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=57, tag_tid=9, entry_tid=1' 2024-04-06T09:37:28.594 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:37:28.594 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:37:28.595 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:37:28.595 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=56, tag_tid=8, entry_tid=11112' 2024-04-06T09:37:28.595 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:37:28.595 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:37:28.597 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=11112 2024-04-06T09:37:28.597 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 9 = 8 -a 1 -le 11112 2024-04-06T09:37:28.597 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:37:28.597 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:37:28.597 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:37:28.798 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:37:28.799 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:37:28.799 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:37:28.799 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:37:28.799 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:37:28.799 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:37:28.799 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:37:28.799 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:37:28.799 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:37:28.799 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:37:28.799 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:37:28.799 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:37:28.799 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:37:28.800 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:37:28.800 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:37:28.800 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:37:28.800 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:37:28.803 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:37:28.803 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:37:28.803 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:37:28.804 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:37:28.804 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:37:28.804 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:37:28.804 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:37:28.804 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:37:28.804 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:37:28.804 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:37:28.804 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:37:28.804 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:37:28.804 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:37:28.804 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:37:28.804 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:37:28.804 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:37:28.804 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:37:28.805 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:37:28.805 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:37:28.805 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:37:28.805 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:37:28.805 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:37:28.805 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:37:28.805 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:37:28.805 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:37:28.805 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:37:28.805 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:37:28.805 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:37:28.805 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:37:28.805 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:37:28.805 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:37:28.806 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:37:28.829 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:37:28.830 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:37:28.830 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:37:28.830 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:37:28.962 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:37:28.963 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:37:28.963 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:37:28.963 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:37:28.963 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:37:28.963 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:37:28.963 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:37:28.964 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:37:28.964 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:37:28.964 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:37:28.964 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:37:28.965 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:37:28.965 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:37:28.966 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:37:28.967 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:37:28.967 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:37:29.012 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 14 2024-04-06T09:37:29.012 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 14 2024-04-06T09:37:29.012 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:37:29.013 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=57, tag_tid=9, entry_tid=1], [object_number=56, tag_tid=9, entry_tid=0], [object_number=59, tag_tid=8, entry_tid=11939], [object_number=58, tag_tid=8, entry_tid=11938]]], state=connected] 2024-04-06T09:37:29.013 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=57, tag_tid=8, entry_tid=11221], [object_number=56, tag_tid=8, entry_tid=11220], [object_number=59, tag_tid=8, entry_tid=11219], [object_number=58, tag_tid=8, entry_tid=11218]]], state=connected] 2024-04-06T09:37:29.016 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:37:29.017 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=57, tag_tid=9, entry_tid=1' 2024-04-06T09:37:29.018 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:37:29.018 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:37:29.018 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:37:29.019 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:37:29.019 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:37:29.019 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:37:29.019 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:37:29.019 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:37:29.019 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:37:29.019 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:37:29.019 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:37:29.019 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:37:29.020 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:37:29.020 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:37:29.020 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:37:29.065 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 14 2024-04-06T09:37:29.066 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 14 2024-04-06T09:37:29.066 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:37:29.066 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=57, tag_tid=9, entry_tid=1], [object_number=56, tag_tid=9, entry_tid=0], [object_number=59, tag_tid=8, entry_tid=11939], [object_number=58, tag_tid=8, entry_tid=11938]]], state=connected] 2024-04-06T09:37:29.066 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=57, tag_tid=8, entry_tid=11221], [object_number=56, tag_tid=8, entry_tid=11220], [object_number=59, tag_tid=8, entry_tid=11219], [object_number=58, tag_tid=8, entry_tid=11218]]], state=connected] 2024-04-06T09:37:29.069 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:37:29.071 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=57, tag_tid=8, entry_tid=11221' 2024-04-06T09:37:29.071 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=57, tag_tid=9, entry_tid=1' -a 'object_number=57, tag_tid=9, entry_tid=1' = 'object_number=57, tag_tid=8, entry_tid=11221' 2024-04-06T09:37:29.071 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=57, tag_tid=8, entry_tid=11221' '!=' 'object_number=56, tag_tid=8, entry_tid=11112' 2024-04-06T09:37:29.071 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:37:29.071 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=57, tag_tid=8, entry_tid=11221' = 'object_number=56, tag_tid=8, entry_tid=11112' 2024-04-06T09:37:29.071 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=57, tag_tid=8, entry_tid=11221' 2024-04-06T09:37:29.072 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=57, tag_tid=9, entry_tid=1' 2024-04-06T09:37:29.072 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:37:29.072 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:37:29.072 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=9 2024-04-06T09:37:29.073 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=57, tag_tid=8, entry_tid=11221' 2024-04-06T09:37:29.073 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:37:29.073 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:37:29.074 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=8 2024-04-06T09:37:29.075 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=57, tag_tid=9, entry_tid=1' 2024-04-06T09:37:29.075 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:37:29.075 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:37:29.076 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:37:29.077 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=57, tag_tid=8, entry_tid=11221' 2024-04-06T09:37:29.077 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:37:29.077 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:37:29.078 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=11221 2024-04-06T09:37:29.078 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 9 = 8 -a 1 -le 11221 2024-04-06T09:37:29.078 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:37:29.078 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:37:29.078 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:37:29.280 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:37:29.280 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:37:29.280 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:37:29.280 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:37:29.280 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:37:29.280 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:37:29.280 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:37:29.280 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:37:29.280 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:37:29.280 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:37:29.281 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:37:29.281 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:37:29.281 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:37:29.281 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:37:29.281 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:37:29.281 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:37:29.282 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:37:29.285 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:37:29.285 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:37:29.285 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:37:29.286 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:37:29.286 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:37:29.286 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:37:29.286 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:37:29.286 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:37:29.286 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:37:29.286 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:37:29.286 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:37:29.286 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:37:29.286 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:37:29.286 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:37:29.286 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:37:29.287 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:37:29.287 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:37:29.287 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:37:29.287 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:37:29.287 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:37:29.287 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:37:29.287 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:37:29.287 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:37:29.287 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:37:29.287 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:37:29.287 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:37:29.287 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:37:29.287 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:37:29.287 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:37:29.287 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:37:29.287 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:37:29.288 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:37:29.311 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:37:29.311 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:37:29.312 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:37:29.312 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:37:29.458 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:37:29.459 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:37:29.459 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:37:29.459 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:37:29.459 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:37:29.459 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:37:29.459 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:37:29.459 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:37:29.459 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:37:29.459 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:37:29.460 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:37:29.460 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:37:29.460 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:37:29.463 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:37:29.463 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:37:29.463 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:37:29.509 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 14 2024-04-06T09:37:29.509 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 14 2024-04-06T09:37:29.509 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:37:29.509 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=57, tag_tid=9, entry_tid=1], [object_number=56, tag_tid=9, entry_tid=0], [object_number=59, tag_tid=8, entry_tid=11939], [object_number=58, tag_tid=8, entry_tid=11938]]], state=connected] 2024-04-06T09:37:29.509 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=58, tag_tid=8, entry_tid=11310], [object_number=57, tag_tid=8, entry_tid=11309], [object_number=56, tag_tid=8, entry_tid=11308], [object_number=59, tag_tid=8, entry_tid=11307]]], state=connected] 2024-04-06T09:37:29.512 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:37:29.514 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=57, tag_tid=9, entry_tid=1' 2024-04-06T09:37:29.515 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:37:29.515 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:37:29.515 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:37:29.515 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:37:29.515 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:37:29.515 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:37:29.515 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:37:29.515 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:37:29.516 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:37:29.516 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:37:29.516 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:37:29.516 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:37:29.516 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:37:29.516 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:37:29.516 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:37:29.562 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 14 2024-04-06T09:37:29.562 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 14 2024-04-06T09:37:29.562 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:37:29.563 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=57, tag_tid=9, entry_tid=1], [object_number=56, tag_tid=9, entry_tid=0], [object_number=59, tag_tid=8, entry_tid=11939], [object_number=58, tag_tid=8, entry_tid=11938]]], state=connected] 2024-04-06T09:37:29.563 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=58, tag_tid=8, entry_tid=11310], [object_number=57, tag_tid=8, entry_tid=11309], [object_number=56, tag_tid=8, entry_tid=11308], [object_number=59, tag_tid=8, entry_tid=11307]]], state=connected] 2024-04-06T09:37:29.566 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:37:29.568 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=58, tag_tid=8, entry_tid=11310' 2024-04-06T09:37:29.568 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=57, tag_tid=9, entry_tid=1' -a 'object_number=57, tag_tid=9, entry_tid=1' = 'object_number=58, tag_tid=8, entry_tid=11310' 2024-04-06T09:37:29.569 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=58, tag_tid=8, entry_tid=11310' '!=' 'object_number=57, tag_tid=8, entry_tid=11221' 2024-04-06T09:37:29.569 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:37:29.569 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=58, tag_tid=8, entry_tid=11310' = 'object_number=57, tag_tid=8, entry_tid=11221' 2024-04-06T09:37:29.569 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=58, tag_tid=8, entry_tid=11310' 2024-04-06T09:37:29.569 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=57, tag_tid=9, entry_tid=1' 2024-04-06T09:37:29.569 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:37:29.569 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:37:29.569 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=9 2024-04-06T09:37:29.570 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=58, tag_tid=8, entry_tid=11310' 2024-04-06T09:37:29.570 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:37:29.570 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:37:29.571 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=8 2024-04-06T09:37:29.572 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=57, tag_tid=9, entry_tid=1' 2024-04-06T09:37:29.572 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:37:29.572 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:37:29.573 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:37:29.574 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=58, tag_tid=8, entry_tid=11310' 2024-04-06T09:37:29.574 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:37:29.574 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:37:29.575 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=11310 2024-04-06T09:37:29.575 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 9 = 8 -a 1 -le 11310 2024-04-06T09:37:29.575 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:37:29.575 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:37:29.575 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:37:29.777 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:37:29.777 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:37:29.777 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:37:29.777 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:37:29.777 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:37:29.777 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:37:29.777 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:37:29.778 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:37:29.778 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:37:29.778 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:37:29.778 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:37:29.778 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:37:29.778 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:37:29.778 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:37:29.778 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:37:29.779 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:37:29.779 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:37:29.781 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:37:29.781 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:37:29.781 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:37:29.781 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:37:29.781 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:37:29.781 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:37:29.781 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:37:29.782 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:37:29.782 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:37:29.782 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:37:29.782 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:37:29.782 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:37:29.782 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:37:29.782 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:37:29.782 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:37:29.782 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:37:29.783 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:37:29.783 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:37:29.783 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:37:29.783 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:37:29.783 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:37:29.784 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:37:29.784 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:37:29.784 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:37:29.784 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:37:29.784 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:37:29.784 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:37:29.784 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:37:29.784 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:37:29.784 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:37:29.784 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:37:29.784 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:37:29.808 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:37:29.809 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:37:29.809 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:37:29.809 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:37:29.933 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:37:29.935 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:37:29.935 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:37:29.935 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:37:29.935 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:37:29.935 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:37:29.935 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:37:29.935 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:37:29.935 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:37:29.935 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:37:29.935 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:37:29.936 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:37:29.936 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:37:29.938 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:37:29.938 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:37:29.938 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:37:29.989 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 14 2024-04-06T09:37:29.990 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 14 2024-04-06T09:37:29.990 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:37:29.990 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=57, tag_tid=9, entry_tid=1], [object_number=56, tag_tid=9, entry_tid=0], [object_number=59, tag_tid=8, entry_tid=11939], [object_number=58, tag_tid=8, entry_tid=11938]]], state=connected] 2024-04-06T09:37:29.990 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=56, tag_tid=8, entry_tid=11404], [object_number=59, tag_tid=8, entry_tid=11403], [object_number=58, tag_tid=8, entry_tid=11402], [object_number=57, tag_tid=8, entry_tid=11401]]], state=connected] 2024-04-06T09:37:29.993 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:37:29.995 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=57, tag_tid=9, entry_tid=1' 2024-04-06T09:37:29.996 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:37:29.996 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:37:29.996 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:37:29.996 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:37:29.996 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:37:29.996 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:37:29.996 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:37:29.996 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:37:29.996 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:37:29.996 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:37:29.996 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:37:29.996 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:37:29.997 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:37:29.997 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:37:29.997 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:37:30.044 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 14 2024-04-06T09:37:30.044 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 14 2024-04-06T09:37:30.044 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:37:30.044 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=57, tag_tid=9, entry_tid=1], [object_number=56, tag_tid=9, entry_tid=0], [object_number=59, tag_tid=8, entry_tid=11939], [object_number=58, tag_tid=8, entry_tid=11938]]], state=connected] 2024-04-06T09:37:30.044 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=56, tag_tid=8, entry_tid=11404], [object_number=59, tag_tid=8, entry_tid=11403], [object_number=58, tag_tid=8, entry_tid=11402], [object_number=57, tag_tid=8, entry_tid=11401]]], state=connected] 2024-04-06T09:37:30.047 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:37:30.049 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=56, tag_tid=8, entry_tid=11404' 2024-04-06T09:37:30.049 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=57, tag_tid=9, entry_tid=1' -a 'object_number=57, tag_tid=9, entry_tid=1' = 'object_number=56, tag_tid=8, entry_tid=11404' 2024-04-06T09:37:30.049 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=56, tag_tid=8, entry_tid=11404' '!=' 'object_number=58, tag_tid=8, entry_tid=11310' 2024-04-06T09:37:30.049 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:37:30.049 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=56, tag_tid=8, entry_tid=11404' = 'object_number=58, tag_tid=8, entry_tid=11310' 2024-04-06T09:37:30.049 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=56, tag_tid=8, entry_tid=11404' 2024-04-06T09:37:30.050 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=57, tag_tid=9, entry_tid=1' 2024-04-06T09:37:30.050 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:37:30.050 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:37:30.051 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=9 2024-04-06T09:37:30.051 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=56, tag_tid=8, entry_tid=11404' 2024-04-06T09:37:30.051 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:37:30.052 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:37:30.052 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=8 2024-04-06T09:37:30.053 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=57, tag_tid=9, entry_tid=1' 2024-04-06T09:37:30.053 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:37:30.053 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:37:30.055 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:37:30.055 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=56, tag_tid=8, entry_tid=11404' 2024-04-06T09:37:30.055 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:37:30.055 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:37:30.056 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=11404 2024-04-06T09:37:30.056 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 9 = 8 -a 1 -le 11404 2024-04-06T09:37:30.056 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:37:30.056 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:37:30.056 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:37:30.248 INFO:tasks.ceph.cluster2.osd.2.smithi192.stderr:2024-04-06T09:37:30.249+0000 7fa534aba640 -1 reset not still connected to 0x559338f41e10 2024-04-06T09:37:30.258 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:37:30.258 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:37:30.258 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:37:30.258 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:37:30.258 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:37:30.258 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:37:30.258 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:37:30.258 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:37:30.258 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:37:30.258 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:37:30.258 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:37:30.259 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:37:30.259 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:37:30.259 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:37:30.259 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:37:30.259 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:37:30.260 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:37:30.262 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:37:30.262 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:37:30.262 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:37:30.263 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:37:30.263 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:37:30.263 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:37:30.263 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:37:30.263 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:37:30.263 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:37:30.263 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:37:30.263 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:37:30.263 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:37:30.263 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:37:30.263 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:37:30.263 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:37:30.263 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:37:30.264 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:37:30.264 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:37:30.264 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:37:30.264 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:37:30.264 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:37:30.265 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:37:30.265 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:37:30.265 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:37:30.265 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:37:30.265 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:37:30.265 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:37:30.265 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:37:30.265 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:37:30.265 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:37:30.265 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:37:30.265 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:37:30.289 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:37:30.290 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:37:30.290 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:37:30.290 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:37:30.424 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:37:30.425 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:37:30.425 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:37:30.425 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:37:30.425 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:37:30.425 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:37:30.426 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:37:30.426 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:37:30.426 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:37:30.426 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:37:30.426 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:37:30.426 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:37:30.426 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:37:30.429 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:37:30.429 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:37:30.429 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:37:30.474 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 14 2024-04-06T09:37:30.474 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 14 2024-04-06T09:37:30.474 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:37:30.474 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=57, tag_tid=9, entry_tid=1], [object_number=56, tag_tid=9, entry_tid=0], [object_number=59, tag_tid=8, entry_tid=11939], [object_number=58, tag_tid=8, entry_tid=11938]]], state=connected] 2024-04-06T09:37:30.474 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=56, tag_tid=8, entry_tid=11500], [object_number=59, tag_tid=8, entry_tid=11499], [object_number=58, tag_tid=8, entry_tid=11498], [object_number=57, tag_tid=8, entry_tid=11497]]], state=connected] 2024-04-06T09:37:30.478 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:37:30.479 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=57, tag_tid=9, entry_tid=1' 2024-04-06T09:37:30.480 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:37:30.481 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:37:30.481 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:37:30.481 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:37:30.481 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:37:30.481 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:37:30.481 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:37:30.481 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:37:30.481 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:37:30.481 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:37:30.481 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:37:30.481 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:37:30.482 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:37:30.482 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:37:30.482 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:37:30.528 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 14 2024-04-06T09:37:30.528 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 14 2024-04-06T09:37:30.528 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:37:30.529 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=57, tag_tid=9, entry_tid=1], [object_number=56, tag_tid=9, entry_tid=0], [object_number=59, tag_tid=8, entry_tid=11939], [object_number=58, tag_tid=8, entry_tid=11938]]], state=connected] 2024-04-06T09:37:30.529 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=56, tag_tid=8, entry_tid=11500], [object_number=59, tag_tid=8, entry_tid=11499], [object_number=58, tag_tid=8, entry_tid=11498], [object_number=57, tag_tid=8, entry_tid=11497]]], state=connected] 2024-04-06T09:37:30.532 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:37:30.534 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=56, tag_tid=8, entry_tid=11500' 2024-04-06T09:37:30.534 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=57, tag_tid=9, entry_tid=1' -a 'object_number=57, tag_tid=9, entry_tid=1' = 'object_number=56, tag_tid=8, entry_tid=11500' 2024-04-06T09:37:30.534 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=56, tag_tid=8, entry_tid=11500' '!=' 'object_number=56, tag_tid=8, entry_tid=11404' 2024-04-06T09:37:30.534 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:37:30.534 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=56, tag_tid=8, entry_tid=11500' = 'object_number=56, tag_tid=8, entry_tid=11404' 2024-04-06T09:37:30.534 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=56, tag_tid=8, entry_tid=11500' 2024-04-06T09:37:30.535 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=57, tag_tid=9, entry_tid=1' 2024-04-06T09:37:30.535 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:37:30.535 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:37:30.535 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=9 2024-04-06T09:37:30.536 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=56, tag_tid=8, entry_tid=11500' 2024-04-06T09:37:30.536 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:37:30.536 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:37:30.537 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=8 2024-04-06T09:37:30.538 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=57, tag_tid=9, entry_tid=1' 2024-04-06T09:37:30.538 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:37:30.538 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:37:30.539 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:37:30.539 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=56, tag_tid=8, entry_tid=11500' 2024-04-06T09:37:30.539 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:37:30.540 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:37:30.541 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=11500 2024-04-06T09:37:30.541 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 9 = 8 -a 1 -le 11500 2024-04-06T09:37:30.541 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:37:30.541 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:37:30.541 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:37:30.742 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:37:30.743 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:37:30.743 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:37:30.743 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:37:30.743 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:37:30.743 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:37:30.743 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:37:30.743 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:37:30.743 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:37:30.743 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:37:30.743 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:37:30.743 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:37:30.743 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:37:30.743 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:37:30.743 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:37:30.744 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:37:30.744 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:37:30.746 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:37:30.746 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:37:30.746 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:37:30.746 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:37:30.747 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:37:30.747 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:37:30.747 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:37:30.747 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:37:30.747 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:37:30.747 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:37:30.747 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:37:30.747 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:37:30.747 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:37:30.747 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:37:30.747 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:37:30.747 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:37:30.748 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:37:30.749 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:37:30.749 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:37:30.749 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:37:30.749 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:37:30.749 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:37:30.749 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:37:30.749 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:37:30.749 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:37:30.749 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:37:30.749 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:37:30.749 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:37:30.749 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:37:30.749 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:37:30.749 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:37:30.750 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:37:30.774 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:37:30.774 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:37:30.774 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:37:30.774 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:37:30.907 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:37:30.908 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:37:30.909 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:37:30.909 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:37:30.909 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:37:30.909 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:37:30.909 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:37:30.909 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:37:30.909 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:37:30.909 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:37:30.910 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:37:30.910 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:37:30.910 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:37:30.911 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:37:30.911 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:37:30.911 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:37:30.956 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 14 2024-04-06T09:37:30.956 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 14 2024-04-06T09:37:30.956 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:37:30.956 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=57, tag_tid=9, entry_tid=1], [object_number=56, tag_tid=9, entry_tid=0], [object_number=59, tag_tid=8, entry_tid=11939], [object_number=58, tag_tid=8, entry_tid=11938]]], state=connected] 2024-04-06T09:37:30.956 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=59, tag_tid=8, entry_tid=11591], [object_number=58, tag_tid=8, entry_tid=11590], [object_number=57, tag_tid=8, entry_tid=11589], [object_number=56, tag_tid=8, entry_tid=11588]]], state=connected] 2024-04-06T09:37:30.959 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:37:30.961 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=57, tag_tid=9, entry_tid=1' 2024-04-06T09:37:30.962 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:37:30.962 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:37:30.962 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:37:30.962 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:37:30.962 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:37:30.962 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:37:30.962 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:37:30.962 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:37:30.962 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:37:30.962 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:37:30.963 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:37:30.963 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:37:30.963 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:37:30.963 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:37:30.964 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:37:31.217 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 14 2024-04-06T09:37:31.217 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 14 2024-04-06T09:37:31.217 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:37:31.217 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=57, tag_tid=9, entry_tid=1], [object_number=56, tag_tid=9, entry_tid=0], [object_number=59, tag_tid=8, entry_tid=11939], [object_number=58, tag_tid=8, entry_tid=11938]]], state=connected] 2024-04-06T09:37:31.217 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=59, tag_tid=8, entry_tid=11591], [object_number=58, tag_tid=8, entry_tid=11590], [object_number=57, tag_tid=8, entry_tid=11589], [object_number=56, tag_tid=8, entry_tid=11588]]], state=connected] 2024-04-06T09:37:31.223 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:37:31.225 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=59, tag_tid=8, entry_tid=11591' 2024-04-06T09:37:31.225 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=57, tag_tid=9, entry_tid=1' -a 'object_number=57, tag_tid=9, entry_tid=1' = 'object_number=59, tag_tid=8, entry_tid=11591' 2024-04-06T09:37:31.225 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=59, tag_tid=8, entry_tid=11591' '!=' 'object_number=56, tag_tid=8, entry_tid=11500' 2024-04-06T09:37:31.225 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:37:31.225 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=59, tag_tid=8, entry_tid=11591' = 'object_number=56, tag_tid=8, entry_tid=11500' 2024-04-06T09:37:31.225 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=59, tag_tid=8, entry_tid=11591' 2024-04-06T09:37:31.226 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=57, tag_tid=9, entry_tid=1' 2024-04-06T09:37:31.226 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:37:31.226 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:37:31.227 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=9 2024-04-06T09:37:31.228 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=59, tag_tid=8, entry_tid=11591' 2024-04-06T09:37:31.228 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:37:31.228 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:37:31.229 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=8 2024-04-06T09:37:31.230 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=57, tag_tid=9, entry_tid=1' 2024-04-06T09:37:31.230 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:37:31.230 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:37:31.231 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:37:31.232 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=59, tag_tid=8, entry_tid=11591' 2024-04-06T09:37:31.232 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:37:31.232 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:37:31.234 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=11591 2024-04-06T09:37:31.234 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 9 = 8 -a 1 -le 11591 2024-04-06T09:37:31.234 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:37:31.234 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:37:31.234 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:37:31.435 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:37:31.436 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:37:31.436 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:37:31.436 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:37:31.436 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:37:31.436 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:37:31.436 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:37:31.436 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:37:31.436 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:37:31.436 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:37:31.436 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:37:31.436 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:37:31.436 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:37:31.436 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:37:31.437 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:37:31.437 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:37:31.437 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:37:31.440 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:37:31.440 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:37:31.440 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:37:31.440 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:37:31.440 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:37:31.440 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:37:31.440 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:37:31.440 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:37:31.441 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:37:31.441 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:37:31.441 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:37:31.441 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:37:31.441 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:37:31.441 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:37:31.441 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:37:31.441 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:37:31.441 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:37:31.441 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:37:31.441 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:37:31.441 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:37:31.441 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:37:31.441 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:37:31.442 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:37:31.442 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:37:31.442 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:37:31.442 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:37:31.442 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:37:31.442 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:37:31.442 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:37:31.442 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:37:31.442 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:37:31.443 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:37:31.468 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:37:31.468 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:37:31.468 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:37:31.468 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:37:31.594 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:37:31.595 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:37:31.595 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:37:31.595 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:37:31.596 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:37:31.596 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:37:31.596 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:37:31.596 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:37:31.596 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:37:31.596 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:37:31.596 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:37:31.597 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:37:31.597 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:37:31.598 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:37:31.598 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:37:31.598 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:37:31.644 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 14 2024-04-06T09:37:31.644 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 14 2024-04-06T09:37:31.644 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:37:31.644 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=57, tag_tid=9, entry_tid=1], [object_number=56, tag_tid=9, entry_tid=0], [object_number=59, tag_tid=8, entry_tid=11939], [object_number=58, tag_tid=8, entry_tid=11938]]], state=connected] 2024-04-06T09:37:31.644 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=59, tag_tid=8, entry_tid=11703], [object_number=58, tag_tid=8, entry_tid=11702], [object_number=57, tag_tid=8, entry_tid=11701], [object_number=56, tag_tid=8, entry_tid=11700]]], state=connected] 2024-04-06T09:37:31.647 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:37:31.649 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=57, tag_tid=9, entry_tid=1' 2024-04-06T09:37:31.650 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:37:31.650 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:37:31.650 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:37:31.650 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:37:31.650 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:37:31.650 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:37:31.650 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:37:31.650 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:37:31.650 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:37:31.650 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:37:31.651 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:37:31.651 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:37:31.651 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:37:31.651 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:37:31.651 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:37:31.698 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 14 2024-04-06T09:37:31.698 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 14 2024-04-06T09:37:31.698 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:37:31.698 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=57, tag_tid=9, entry_tid=1], [object_number=56, tag_tid=9, entry_tid=0], [object_number=59, tag_tid=8, entry_tid=11939], [object_number=58, tag_tid=8, entry_tid=11938]]], state=connected] 2024-04-06T09:37:31.698 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=59, tag_tid=8, entry_tid=11703], [object_number=58, tag_tid=8, entry_tid=11702], [object_number=57, tag_tid=8, entry_tid=11701], [object_number=56, tag_tid=8, entry_tid=11700]]], state=connected] 2024-04-06T09:37:31.701 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:37:31.703 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=59, tag_tid=8, entry_tid=11703' 2024-04-06T09:37:31.704 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=57, tag_tid=9, entry_tid=1' -a 'object_number=57, tag_tid=9, entry_tid=1' = 'object_number=59, tag_tid=8, entry_tid=11703' 2024-04-06T09:37:31.704 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=59, tag_tid=8, entry_tid=11703' '!=' 'object_number=59, tag_tid=8, entry_tid=11591' 2024-04-06T09:37:31.704 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:37:31.704 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=59, tag_tid=8, entry_tid=11703' = 'object_number=59, tag_tid=8, entry_tid=11591' 2024-04-06T09:37:31.704 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=59, tag_tid=8, entry_tid=11703' 2024-04-06T09:37:31.704 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=57, tag_tid=9, entry_tid=1' 2024-04-06T09:37:31.704 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:37:31.704 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:37:31.704 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=9 2024-04-06T09:37:31.705 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=59, tag_tid=8, entry_tid=11703' 2024-04-06T09:37:31.705 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:37:31.705 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:37:31.706 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=8 2024-04-06T09:37:31.707 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=57, tag_tid=9, entry_tid=1' 2024-04-06T09:37:31.707 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:37:31.707 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:37:31.708 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:37:31.708 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=59, tag_tid=8, entry_tid=11703' 2024-04-06T09:37:31.709 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:37:31.709 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:37:31.710 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=11703 2024-04-06T09:37:31.710 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 9 = 8 -a 1 -le 11703 2024-04-06T09:37:31.710 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:37:31.710 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:37:31.710 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:37:31.912 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:37:31.912 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:37:31.912 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:37:31.912 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:37:31.912 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:37:31.912 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:37:31.912 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:37:31.912 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:37:31.912 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:37:31.912 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:37:31.913 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:37:31.913 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:37:31.913 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:37:31.913 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:37:31.913 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:37:31.913 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:37:31.913 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:37:31.917 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:37:31.918 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:37:31.918 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:37:31.918 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:37:31.918 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:37:31.918 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:37:31.918 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:37:31.918 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:37:31.918 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:37:31.918 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:37:31.918 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:37:31.918 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:37:31.918 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:37:31.919 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:37:31.919 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:37:31.919 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:37:31.919 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:37:31.919 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:37:31.919 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:37:31.919 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:37:31.919 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:37:31.919 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:37:31.919 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:37:31.919 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:37:31.919 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:37:31.919 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:37:31.919 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:37:31.919 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:37:31.919 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:37:31.920 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:37:31.920 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:37:31.920 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:37:31.942 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:37:31.943 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:37:31.943 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:37:31.943 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:37:32.069 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:37:32.070 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:37:32.070 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:37:32.070 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:37:32.070 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:37:32.071 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:37:32.071 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:37:32.071 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:37:32.071 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:37:32.071 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:37:32.071 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:37:32.071 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:37:32.071 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:37:32.072 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:37:32.073 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:37:32.073 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:37:32.129 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 14 2024-04-06T09:37:32.129 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 14 2024-04-06T09:37:32.129 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:37:32.129 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=57, tag_tid=9, entry_tid=1], [object_number=56, tag_tid=9, entry_tid=0], [object_number=59, tag_tid=8, entry_tid=11939], [object_number=58, tag_tid=8, entry_tid=11938]]], state=connected] 2024-04-06T09:37:32.129 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=59, tag_tid=8, entry_tid=11775], [object_number=58, tag_tid=8, entry_tid=11774], [object_number=57, tag_tid=8, entry_tid=11773], [object_number=56, tag_tid=8, entry_tid=11772]]], state=connected] 2024-04-06T09:37:32.132 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:37:32.134 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=57, tag_tid=9, entry_tid=1' 2024-04-06T09:37:32.135 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:37:32.135 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:37:32.135 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:37:32.135 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:37:32.135 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:37:32.135 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:37:32.135 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:37:32.136 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:37:32.136 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:37:32.136 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:37:32.136 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:37:32.136 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:37:32.136 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:37:32.136 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:37:32.137 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:37:32.182 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 14 2024-04-06T09:37:32.182 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 14 2024-04-06T09:37:32.182 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:37:32.182 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=57, tag_tid=9, entry_tid=1], [object_number=56, tag_tid=9, entry_tid=0], [object_number=59, tag_tid=8, entry_tid=11939], [object_number=58, tag_tid=8, entry_tid=11938]]], state=connected] 2024-04-06T09:37:32.182 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=59, tag_tid=8, entry_tid=11775], [object_number=58, tag_tid=8, entry_tid=11774], [object_number=57, tag_tid=8, entry_tid=11773], [object_number=56, tag_tid=8, entry_tid=11772]]], state=connected] 2024-04-06T09:37:32.186 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:37:32.187 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=59, tag_tid=8, entry_tid=11775' 2024-04-06T09:37:32.187 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=57, tag_tid=9, entry_tid=1' -a 'object_number=57, tag_tid=9, entry_tid=1' = 'object_number=59, tag_tid=8, entry_tid=11775' 2024-04-06T09:37:32.187 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=59, tag_tid=8, entry_tid=11775' '!=' 'object_number=59, tag_tid=8, entry_tid=11703' 2024-04-06T09:37:32.187 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:37:32.188 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=59, tag_tid=8, entry_tid=11775' = 'object_number=59, tag_tid=8, entry_tid=11703' 2024-04-06T09:37:32.188 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=59, tag_tid=8, entry_tid=11775' 2024-04-06T09:37:32.188 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=57, tag_tid=9, entry_tid=1' 2024-04-06T09:37:32.188 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:37:32.188 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:37:32.189 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=9 2024-04-06T09:37:32.189 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=59, tag_tid=8, entry_tid=11775' 2024-04-06T09:37:32.190 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:37:32.190 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:37:32.190 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=8 2024-04-06T09:37:32.191 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=57, tag_tid=9, entry_tid=1' 2024-04-06T09:37:32.191 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:37:32.191 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:37:32.192 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:37:32.193 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=59, tag_tid=8, entry_tid=11775' 2024-04-06T09:37:32.193 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:37:32.193 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:37:32.194 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=11775 2024-04-06T09:37:32.194 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 9 = 8 -a 1 -le 11775 2024-04-06T09:37:32.194 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:37:32.194 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:37:32.194 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:37:32.396 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:37:32.396 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:37:32.396 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:37:32.396 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:37:32.396 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:37:32.396 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:37:32.396 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:37:32.397 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:37:32.397 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:37:32.397 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:37:32.397 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:37:32.397 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:37:32.397 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:37:32.397 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:37:32.397 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:37:32.397 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:37:32.398 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:37:32.400 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:37:32.400 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:37:32.401 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:37:32.401 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:37:32.401 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:37:32.401 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:37:32.401 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:37:32.401 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:37:32.401 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:37:32.401 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:37:32.401 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:37:32.401 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:37:32.401 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:37:32.401 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:37:32.401 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:37:32.401 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:37:32.402 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:37:32.402 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:37:32.402 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:37:32.403 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:37:32.403 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:37:32.403 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:37:32.403 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:37:32.403 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:37:32.403 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:37:32.403 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:37:32.403 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:37:32.403 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:37:32.403 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:37:32.403 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:37:32.403 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:37:32.403 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:37:32.427 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:37:32.427 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:37:32.427 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:37:32.427 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:37:32.564 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:37:32.565 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:37:32.565 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:37:32.565 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:37:32.565 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:37:32.565 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:37:32.565 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:37:32.565 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:37:32.565 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:37:32.565 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:37:32.566 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:37:32.566 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:37:32.566 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:37:32.569 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:37:32.569 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:37:32.569 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:37:32.616 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 14 2024-04-06T09:37:32.616 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 14 2024-04-06T09:37:32.616 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:37:32.617 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=57, tag_tid=9, entry_tid=1], [object_number=56, tag_tid=9, entry_tid=0], [object_number=59, tag_tid=8, entry_tid=11939], [object_number=58, tag_tid=8, entry_tid=11938]]], state=connected] 2024-04-06T09:37:32.617 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=59, tag_tid=8, entry_tid=11851], [object_number=58, tag_tid=8, entry_tid=11850], [object_number=57, tag_tid=8, entry_tid=11849], [object_number=56, tag_tid=8, entry_tid=11848]]], state=connected] 2024-04-06T09:37:32.620 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:37:32.621 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=57, tag_tid=9, entry_tid=1' 2024-04-06T09:37:32.622 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:37:32.622 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:37:32.622 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:37:32.622 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:37:32.622 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:37:32.623 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:37:32.623 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:37:32.623 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:37:32.623 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:37:32.623 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:37:32.623 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:37:32.623 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:37:32.624 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:37:32.624 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:37:32.624 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:37:32.669 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 14 2024-04-06T09:37:32.670 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 14 2024-04-06T09:37:32.670 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:37:32.670 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=57, tag_tid=9, entry_tid=1], [object_number=56, tag_tid=9, entry_tid=0], [object_number=59, tag_tid=8, entry_tid=11939], [object_number=58, tag_tid=8, entry_tid=11938]]], state=connected] 2024-04-06T09:37:32.670 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=59, tag_tid=8, entry_tid=11851], [object_number=58, tag_tid=8, entry_tid=11850], [object_number=57, tag_tid=8, entry_tid=11849], [object_number=56, tag_tid=8, entry_tid=11848]]], state=connected] 2024-04-06T09:37:32.673 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:37:32.675 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=59, tag_tid=8, entry_tid=11851' 2024-04-06T09:37:32.675 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=57, tag_tid=9, entry_tid=1' -a 'object_number=57, tag_tid=9, entry_tid=1' = 'object_number=59, tag_tid=8, entry_tid=11851' 2024-04-06T09:37:32.676 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=59, tag_tid=8, entry_tid=11851' '!=' 'object_number=59, tag_tid=8, entry_tid=11775' 2024-04-06T09:37:32.676 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:37:32.676 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=59, tag_tid=8, entry_tid=11851' = 'object_number=59, tag_tid=8, entry_tid=11775' 2024-04-06T09:37:32.676 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=59, tag_tid=8, entry_tid=11851' 2024-04-06T09:37:32.676 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=57, tag_tid=9, entry_tid=1' 2024-04-06T09:37:32.676 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:37:32.676 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:37:32.676 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=9 2024-04-06T09:37:32.677 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=59, tag_tid=8, entry_tid=11851' 2024-04-06T09:37:32.677 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:37:32.677 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:37:32.678 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=8 2024-04-06T09:37:32.679 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=57, tag_tid=9, entry_tid=1' 2024-04-06T09:37:32.679 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:37:32.679 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:37:32.680 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:37:32.680 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=59, tag_tid=8, entry_tid=11851' 2024-04-06T09:37:32.680 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:37:32.680 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:37:32.681 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=11851 2024-04-06T09:37:32.682 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 9 = 8 -a 1 -le 11851 2024-04-06T09:37:32.682 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:37:32.682 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:37:32.682 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:37:32.884 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:37:32.884 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:37:32.884 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:37:32.884 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:37:32.884 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:37:32.884 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:37:32.884 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:37:32.885 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:37:32.885 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:37:32.885 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:37:32.885 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:37:32.885 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:37:32.885 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:37:32.885 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:37:32.885 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:37:32.885 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:37:32.886 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:37:32.886 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:37:32.886 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:37:32.887 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:37:32.887 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:37:32.887 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:37:32.887 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:37:32.887 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:37:32.887 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:37:32.887 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:37:32.887 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:37:32.887 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:37:32.887 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:37:32.887 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:37:32.888 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:37:32.888 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:37:32.888 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:37:32.889 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:37:32.889 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:37:32.889 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:37:32.889 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:37:32.889 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:37:32.889 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:37:32.889 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:37:32.889 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:37:32.890 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:37:32.890 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:37:32.890 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:37:32.890 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:37:32.890 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:37:32.890 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:37:32.890 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:37:32.890 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:37:32.914 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:37:32.914 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:37:32.914 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:37:32.914 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:37:33.044 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:37:33.045 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:37:33.045 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:37:33.045 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:37:33.046 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:37:33.046 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:37:33.046 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:37:33.046 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:37:33.046 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:37:33.046 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:37:33.046 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:37:33.047 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:37:33.047 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:37:33.049 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:37:33.050 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:37:33.050 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:37:33.102 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 14 2024-04-06T09:37:33.102 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 14 2024-04-06T09:37:33.102 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:37:33.102 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=57, tag_tid=9, entry_tid=1], [object_number=56, tag_tid=9, entry_tid=0], [object_number=59, tag_tid=8, entry_tid=11939], [object_number=58, tag_tid=8, entry_tid=11938]]], state=connected] 2024-04-06T09:37:33.102 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=58, tag_tid=8, entry_tid=11930], [object_number=57, tag_tid=8, entry_tid=11929], [object_number=56, tag_tid=8, entry_tid=11928], [object_number=59, tag_tid=8, entry_tid=11927]]], state=connected] 2024-04-06T09:37:33.105 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:37:33.106 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=57, tag_tid=9, entry_tid=1' 2024-04-06T09:37:33.107 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:37:33.107 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:37:33.107 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:37:33.108 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:37:33.108 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:37:33.108 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:37:33.108 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:37:33.108 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:37:33.108 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:37:33.108 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:37:33.108 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:37:33.108 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:37:33.109 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:37:33.109 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:37:33.109 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:37:33.155 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 14 2024-04-06T09:37:33.155 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 14 2024-04-06T09:37:33.155 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:37:33.155 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=57, tag_tid=9, entry_tid=1], [object_number=56, tag_tid=9, entry_tid=0], [object_number=59, tag_tid=8, entry_tid=11939], [object_number=58, tag_tid=8, entry_tid=11938]]], state=connected] 2024-04-06T09:37:33.155 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=58, tag_tid=8, entry_tid=11930], [object_number=57, tag_tid=8, entry_tid=11929], [object_number=56, tag_tid=8, entry_tid=11928], [object_number=59, tag_tid=8, entry_tid=11927]]], state=connected] 2024-04-06T09:37:33.158 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:37:33.160 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=58, tag_tid=8, entry_tid=11930' 2024-04-06T09:37:33.160 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=57, tag_tid=9, entry_tid=1' -a 'object_number=57, tag_tid=9, entry_tid=1' = 'object_number=58, tag_tid=8, entry_tid=11930' 2024-04-06T09:37:33.161 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=58, tag_tid=8, entry_tid=11930' '!=' 'object_number=59, tag_tid=8, entry_tid=11851' 2024-04-06T09:37:33.161 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:37:33.161 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=58, tag_tid=8, entry_tid=11930' = 'object_number=59, tag_tid=8, entry_tid=11851' 2024-04-06T09:37:33.161 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=58, tag_tid=8, entry_tid=11930' 2024-04-06T09:37:33.161 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=57, tag_tid=9, entry_tid=1' 2024-04-06T09:37:33.161 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:37:33.161 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:37:33.161 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=9 2024-04-06T09:37:33.162 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=58, tag_tid=8, entry_tid=11930' 2024-04-06T09:37:33.162 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:37:33.162 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:37:33.163 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=8 2024-04-06T09:37:33.164 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=57, tag_tid=9, entry_tid=1' 2024-04-06T09:37:33.164 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:37:33.164 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:37:33.165 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:37:33.166 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=58, tag_tid=8, entry_tid=11930' 2024-04-06T09:37:33.166 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:37:33.166 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:37:33.167 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=11930 2024-04-06T09:37:33.167 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 9 = 8 -a 1 -le 11930 2024-04-06T09:37:33.167 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:37:33.167 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:37:33.167 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:37:33.369 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:37:33.369 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:37:33.369 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:37:33.369 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:37:33.369 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:37:33.369 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:37:33.369 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:37:33.369 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:37:33.370 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:37:33.370 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:37:33.370 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:37:33.370 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:37:33.370 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:37:33.370 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:37:33.370 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:37:33.371 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:37:33.371 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:37:33.374 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:37:33.374 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:37:33.374 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:37:33.374 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:37:33.374 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:37:33.374 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:37:33.374 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:37:33.374 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:37:33.374 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:37:33.375 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:37:33.375 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:37:33.375 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:37:33.375 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:37:33.375 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:37:33.375 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:37:33.375 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:37:33.375 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:37:33.375 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:37:33.375 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:37:33.375 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:37:33.375 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:37:33.375 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:37:33.375 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:37:33.375 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:37:33.376 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:37:33.376 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:37:33.376 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:37:33.376 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:37:33.376 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:37:33.376 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:37:33.376 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:37:33.376 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:37:33.399 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:37:33.400 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:37:33.400 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:37:33.400 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:37:33.526 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:37:33.526 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:37:33.526 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:37:33.526 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:37:33.527 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:37:33.527 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:37:33.527 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:37:33.527 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:37:33.527 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:37:33.527 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:37:33.527 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:37:33.528 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:37:33.528 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:37:33.530 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:37:33.530 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:37:33.530 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:37:33.587 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 14 2024-04-06T09:37:33.587 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 14 2024-04-06T09:37:33.587 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:37:33.587 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=57, tag_tid=9, entry_tid=1], [object_number=56, tag_tid=9, entry_tid=0], [object_number=59, tag_tid=8, entry_tid=11939], [object_number=58, tag_tid=8, entry_tid=11938]]], state=connected] 2024-04-06T09:37:33.587 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=59, tag_tid=8, entry_tid=11939], [object_number=58, tag_tid=8, entry_tid=11938], [object_number=57, tag_tid=8, entry_tid=11937], [object_number=56, tag_tid=8, entry_tid=11936]]], state=connected] 2024-04-06T09:37:33.591 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:37:33.593 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=57, tag_tid=9, entry_tid=1' 2024-04-06T09:37:33.593 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:37:33.593 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:37:33.593 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:37:33.593 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:37:33.593 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:37:33.593 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:37:33.594 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:37:33.594 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:37:33.594 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:37:33.594 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:37:33.594 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:37:33.594 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:37:33.594 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:37:33.595 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:37:33.595 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:37:33.642 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 14 2024-04-06T09:37:33.642 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 14 2024-04-06T09:37:33.642 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:37:33.642 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=57, tag_tid=9, entry_tid=1], [object_number=56, tag_tid=9, entry_tid=0], [object_number=59, tag_tid=8, entry_tid=11939], [object_number=58, tag_tid=8, entry_tid=11938]]], state=connected] 2024-04-06T09:37:33.642 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=59, tag_tid=8, entry_tid=11939], [object_number=58, tag_tid=8, entry_tid=11938], [object_number=57, tag_tid=8, entry_tid=11937], [object_number=56, tag_tid=8, entry_tid=11936]]], state=connected] 2024-04-06T09:37:33.646 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:37:33.647 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=59, tag_tid=8, entry_tid=11939' 2024-04-06T09:37:33.647 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=57, tag_tid=9, entry_tid=1' -a 'object_number=57, tag_tid=9, entry_tid=1' = 'object_number=59, tag_tid=8, entry_tid=11939' 2024-04-06T09:37:33.647 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=59, tag_tid=8, entry_tid=11939' '!=' 'object_number=58, tag_tid=8, entry_tid=11930' 2024-04-06T09:37:33.647 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:37:33.647 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=59, tag_tid=8, entry_tid=11939' = 'object_number=58, tag_tid=8, entry_tid=11930' 2024-04-06T09:37:33.648 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=59, tag_tid=8, entry_tid=11939' 2024-04-06T09:37:33.648 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=57, tag_tid=9, entry_tid=1' 2024-04-06T09:37:33.648 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:37:33.648 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:37:33.649 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=9 2024-04-06T09:37:33.650 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=59, tag_tid=8, entry_tid=11939' 2024-04-06T09:37:33.650 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:37:33.650 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:37:33.651 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=8 2024-04-06T09:37:33.651 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=57, tag_tid=9, entry_tid=1' 2024-04-06T09:37:33.651 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:37:33.651 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:37:33.652 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:37:33.653 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=59, tag_tid=8, entry_tid=11939' 2024-04-06T09:37:33.653 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:37:33.653 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:37:33.654 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=11939 2024-04-06T09:37:33.655 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 9 = 8 -a 1 -le 11939 2024-04-06T09:37:33.655 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:37:33.655 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:37:33.655 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:37:33.856 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:37:33.856 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:37:33.857 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:37:33.857 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:37:33.857 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:37:33.857 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:37:33.857 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:37:33.857 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:37:33.857 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:37:33.857 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:37:33.857 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:37:33.857 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:37:33.857 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:37:33.857 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:37:33.857 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:37:33.858 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:37:33.858 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:37:33.861 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:37:33.861 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:37:33.861 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:37:33.861 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:37:33.861 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:37:33.861 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:37:33.861 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:37:33.861 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:37:33.861 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:37:33.861 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:37:33.861 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:37:33.861 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:37:33.861 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:37:33.862 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:37:33.862 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:37:33.862 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:37:33.862 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:37:33.862 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:37:33.862 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:37:33.863 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:37:33.863 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:37:33.863 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:37:33.863 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:37:33.863 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:37:33.863 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:37:33.863 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:37:33.863 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:37:33.863 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:37:33.863 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:37:33.863 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:37:33.863 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:37:33.863 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:37:33.888 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:37:33.888 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:37:33.889 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:37:33.889 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:37:34.011 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:37:34.011 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:37:34.011 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:37:34.011 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:37:34.012 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:37:34.012 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:37:34.012 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:37:34.012 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:37:34.012 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:37:34.012 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:37:34.012 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:37:34.013 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:37:34.013 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:37:34.015 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:37:34.016 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:37:34.016 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:37:34.062 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 14 2024-04-06T09:37:34.062 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 14 2024-04-06T09:37:34.062 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:37:34.062 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=57, tag_tid=9, entry_tid=1], [object_number=56, tag_tid=9, entry_tid=0], [object_number=59, tag_tid=8, entry_tid=11939], [object_number=58, tag_tid=8, entry_tid=11938]]], state=connected] 2024-04-06T09:37:34.062 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=57, tag_tid=9, entry_tid=1], [object_number=56, tag_tid=9, entry_tid=0], [object_number=59, tag_tid=8, entry_tid=11939], [object_number=58, tag_tid=8, entry_tid=11938]]], state=connected] 2024-04-06T09:37:34.065 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:37:34.066 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=57, tag_tid=9, entry_tid=1' 2024-04-06T09:37:34.068 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:37:34.068 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:37:34.068 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:37:34.068 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:37:34.068 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:37:34.068 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:37:34.068 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:37:34.068 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:37:34.068 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:37:34.068 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:37:34.068 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:37:34.068 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:37:34.069 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:37:34.069 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:37:34.069 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:37:34.115 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 14 2024-04-06T09:37:34.115 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 14 2024-04-06T09:37:34.115 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:37:34.116 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=57, tag_tid=9, entry_tid=1], [object_number=56, tag_tid=9, entry_tid=0], [object_number=59, tag_tid=8, entry_tid=11939], [object_number=58, tag_tid=8, entry_tid=11938]]], state=connected] 2024-04-06T09:37:34.116 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=57, tag_tid=9, entry_tid=1], [object_number=56, tag_tid=9, entry_tid=0], [object_number=59, tag_tid=8, entry_tid=11939], [object_number=58, tag_tid=8, entry_tid=11938]]], state=connected] 2024-04-06T09:37:34.119 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:37:34.121 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=57, tag_tid=9, entry_tid=1' 2024-04-06T09:37:34.121 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=57, tag_tid=9, entry_tid=1' -a 'object_number=57, tag_tid=9, entry_tid=1' = 'object_number=57, tag_tid=9, entry_tid=1' 2024-04-06T09:37:34.121 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:37:34.121 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_snap_present cluster1 mirror test snap3 2024-04-06T09:37:34.121 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:37:34.121 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:37:34.121 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:37:34.121 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local snap_name=snap3 2024-04-06T09:37:34.121 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local s 2024-04-06T09:37:34.122 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-04-06T09:37:34.122 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 1 2024-04-06T09:37:35.122 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 info mirror/test@snap3 2024-04-06T09:37:35.189 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:rbd image 'test': 2024-04-06T09:37:35.189 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: size 512 MiB in 128 objects 2024-04-06T09:37:35.189 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: order 22 (4 MiB objects) 2024-04-06T09:37:35.189 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: snapshot_count: 5 2024-04-06T09:37:35.189 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: id: 10a56da0b207 2024-04-06T09:37:35.189 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: block_name_prefix: rbd_data.10a56da0b207 2024-04-06T09:37:35.189 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: format: 2 2024-04-06T09:37:35.190 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: features: layering, exclusive-lock, journaling 2024-04-06T09:37:35.190 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: op_features: 2024-04-06T09:37:35.190 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: flags: 2024-04-06T09:37:35.190 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: create_timestamp: Sat Apr 6 09:31:59 2024 2024-04-06T09:37:35.190 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: access_timestamp: Sat Apr 6 09:36:20 2024 2024-04-06T09:37:35.190 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: modify_timestamp: Sat Apr 6 09:37:04 2024 2024-04-06T09:37:35.190 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: protected: False 2024-04-06T09:37:35.190 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: journal: 10a56da0b207 2024-04-06T09:37:35.190 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: mirroring state: enabled 2024-04-06T09:37:35.190 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: mirroring mode: journal 2024-04-06T09:37:35.190 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: mirroring global id: 8ad5dedc-b3e0-425d-8993-b3e2231fc5ca 2024-04-06T09:37:35.190 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: mirroring primary: false 2024-04-06T09:37:35.193 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:37:35.194 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n snap2-clean ']' 2024-04-06T09:37:35.194 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ compare_image_snaps mirror test snap2-clean 2024-04-06T09:37:35.194 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:37:35.194 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:37:35.194 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local snap_name=snap2-clean 2024-04-06T09:37:35.194 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local ret=0 2024-04-06T09:37:35.194 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local rmt_export=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.export 2024-04-06T09:37:35.194 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loc_export=/tmp/tmp.BhW6xKmROU/cluster1-mirror-test.export 2024-04-06T09:37:35.194 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rm -f /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.export /tmp/tmp.BhW6xKmROU/cluster1-mirror-test.export 2024-04-06T09:37:35.194 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster2 -p mirror export test@snap2-clean /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.export 2024-04-06T09:37:35.735 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-04-06T09:37:35.748 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 -p mirror export test@snap2-clean /tmp/tmp.BhW6xKmROU/cluster1-mirror-test.export 2024-04-06T09:37:36.931 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-04-06T09:37:36.953 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmp /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.export /tmp/tmp.BhW6xKmROU/cluster1-mirror-test.export 2024-04-06T09:37:37.238 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rm -f /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.export /tmp/tmp.BhW6xKmROU/cluster1-mirror-test.export 2024-04-06T09:37:37.374 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:37:37.374 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ compare_image_snaps mirror test snap3 2024-04-06T09:37:37.374 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:37:37.374 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:37:37.374 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local snap_name=snap3 2024-04-06T09:37:37.374 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local ret=0 2024-04-06T09:37:37.375 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local rmt_export=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.export 2024-04-06T09:37:37.375 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loc_export=/tmp/tmp.BhW6xKmROU/cluster1-mirror-test.export 2024-04-06T09:37:37.375 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rm -f /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.export /tmp/tmp.BhW6xKmROU/cluster1-mirror-test.export 2024-04-06T09:37:37.375 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster2 -p mirror export test@snap3 /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.export 2024-04-06T09:37:37.949 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-04-06T09:37:37.960 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 -p mirror export test@snap3 /tmp/tmp.BhW6xKmROU/cluster1-mirror-test.export 2024-04-06T09:37:39.396 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-04-06T09:37:39.420 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmp /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.export /tmp/tmp.BhW6xKmROU/cluster1-mirror-test.export 2024-04-06T09:37:39.699 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rm -f /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.export /tmp/tmp.BhW6xKmROU/cluster1-mirror-test.export 2024-04-06T09:37:39.837 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:37:39.837 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ clean_snap_name=snap3-clean 2024-04-06T09:37:39.837 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ create_snap cluster2 mirror test snap3-clean 2024-04-06T09:37:39.837 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-04-06T09:37:39.837 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:37:39.837 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:37:39.837 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local snap_name=snap3-clean 2024-04-06T09:37:39.837 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap3-clean --debug-rbd=20 --debug-journaler=20 2024-04-06T09:37:40.529 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in `seq 1 10` 2024-04-06T09:37:40.529 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ stress_write_image cluster2 mirror test 2024-04-06T09:37:40.529 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-04-06T09:37:40.529 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:37:40.529 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:37:40.530 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-04-06T09:37:40.533 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local duration=12 2024-04-06T09:37:40.533 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set +e 2024-04-06T09:37:40.533 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ timeout 12s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror test --debug-rbd=20 --debug-journaler=20 2024-04-06T09:37:43.953 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:100 IOs, 85513216 bytes 2024-04-06T09:37:48.054 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:200 IOs, 189825024 bytes 2024-04-06T09:37:49.142 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:300 IOs, 218662912 bytes 2024-04-06T09:37:50.807 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:400 IOs, 244069376 bytes 2024-04-06T09:37:52.548 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ error_code=124 2024-04-06T09:37:52.548 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set -e 2024-04-06T09:37:52.548 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 124 -eq 124 ']' 2024-04-06T09:37:52.548 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:37:52.549 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+replaying 2024-04-06T09:37:52.549 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:37:52.549 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:37:52.549 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:37:52.550 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+replaying 2024-04-06T09:37:52.550 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local description_pattern= 2024-04-06T09:37:52.550 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local service_pattern= 2024-04-06T09:37:52.550 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:37:52.550 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 1 2024-04-06T09:37:53.548 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test_status_in_pool_dir cluster1 mirror test up+replaying '' '' 2024-04-06T09:37:53.549 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:37:53.549 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:37:53.549 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:37:53.549 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+replaying 2024-04-06T09:37:53.549 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local description_pattern= 2024-04-06T09:37:53.549 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local service_pattern= 2024-04-06T09:37:53.549 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local status 2024-04-06T09:37:53.549 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ CEPH_ARGS= 2024-04-06T09:37:53.550 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster1 mirror image status mirror/test 2024-04-06T09:37:53.635 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ status='test: 2024-04-06T09:37:53.635 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: global_id: 8ad5dedc-b3e0-425d-8993-b3e2231fc5ca 2024-04-06T09:37:53.635 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: state: up+replaying 2024-04-06T09:37:53.635 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: description: replaying, {"bytes_per_second":3050522.0,"entries_behind_primary":1042,"entries_per_second":191.87,"non_primary_position":{"entry_tid":10898,"object_number":58,"tag_tid":8},"primary_position":{"entry_tid":1,"object_number":57,"tag_tid":9},"seconds_until_synced":5} 2024-04-06T09:37:53.635 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: service: mirror.0 on smithi192 2024-04-06T09:37:53.635 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: last_update: 2024-04-06 09:37:27 2024-04-06T09:37:53.635 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: peer_sites: 2024-04-06T09:37:53.635 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: name: cluster2 2024-04-06T09:37:53.636 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: state: up+stopped 2024-04-06T09:37:53.636 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: description: local image is primary 2024-04-06T09:37:53.636 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: last_update: 2024-04-06 09:37:41' 2024-04-06T09:37:53.636 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^ state: .*up+replaying' 2024-04-06T09:37:53.636 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: state: up+replaying 2024-04-06T09:37:53.637 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^ description: .*' 2024-04-06T09:37:53.637 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: description: replaying, {"bytes_per_second":3050522.0,"entries_behind_primary":1042,"entries_per_second":191.87,"non_primary_position":{"entry_tid":10898,"object_number":58,"tag_tid":8},"primary_position":{"entry_tid":1,"object_number":57,"tag_tid":9},"seconds_until_synced":5} 2024-04-06T09:37:53.637 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n '' ']' 2024-04-06T09:37:53.638 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ echo up+replaying 2024-04-06T09:37:53.638 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^up+' 2024-04-06T09:37:53.638 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:up+replaying 2024-04-06T09:37:53.638 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep 'service: *mirror..* on ' 2024-04-06T09:37:53.640 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: service: mirror.0 on smithi192 2024-04-06T09:37:53.640 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local last_update 2024-04-06T09:37:53.640 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-04-06T09:37:53.642 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_update='2024-04-06 09:37:27' 2024-04-06T09:37:53.643 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+replaying '2024-04-06 09:37:27' 2024-04-06T09:37:53.643 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:37:53.643 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:37:53.643 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:37:53.643 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+replaying 2024-04-06T09:37:53.643 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'prev_last_update=2024-04-06 09:37:27' 2024-04-06T09:37:53.643 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local status 2024-04-06T09:37:53.643 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ CEPH_ARGS= 2024-04-06T09:37:53.643 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-04-06T09:37:53.725 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ status='OKOKOK142094268mirror.1smithi19218.2.2-1059-g4d05f4cafalseOK42164269mirror.2smithi19218.2.2-1059-g4d05f4cafalseOK42584261mirror.0smithi19218.2.2-1059-g4d05f4catrueOK43194322mirror.3smithi19218.2.2-1059-g4d05f4cafalseOKtest8ad5dedc-b3e0-425d-8993-b3e2231fc5caup+replayingreplaying, {"bytes_per_second":3050522.0,"entries_behind_primary":1042,"entries_per_second":191.87,"non_primary_position":{"entry_tid":10898,"object_number":58,"tag_tid":8},"primary_position":{"entry_tid":1,"object_number":57,"tag_tid":9},"seconds_until_synced":5}42584261mirror.0smithi1922024-04-06 09:37:27cluster2c5e3bdc3-c286-4289-8fd0-6b8628b92376up+stoppedlocal image is primary2024-04-06 09:37:41' 2024-04-06T09:37:53.725 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local last_update state 2024-04-06T09:37:53.725 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2024-04-06T09:37:53.728 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_update='2024-04-06 09:37:27' 2024-04-06T09:37:53.728 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2024-04-06T09:37:53.731 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ state=up+replaying 2024-04-06T09:37:53.731 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ echo up+replaying 2024-04-06T09:37:53.731 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep up+replaying 2024-04-06T09:37:53.732 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:up+replaying 2024-04-06T09:37:53.732 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:37:53.732 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:37:53.733 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ snap_name=snap4 2024-04-06T09:37:53.733 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ create_snap cluster2 mirror test snap4 2024-04-06T09:37:53.733 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-04-06T09:37:53.733 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:37:53.733 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:37:53.733 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local snap_name=snap4 2024-04-06T09:37:53.733 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap4 --debug-rbd=20 --debug-journaler=20 2024-04-06T09:38:22.514 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_image_replay_started cluster1 mirror test 2024-04-06T09:38:22.514 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:38:22.514 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:38:22.514 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:38:22.514 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_image_replay_state cluster1 mirror test started 2024-04-06T09:38:22.514 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:38:22.514 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:38:22.514 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:38:22.514 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state=started 2024-04-06T09:38:22.514 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local s 2024-04-06T09:38:22.515 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:38:22.515 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 1 2024-04-06T09:38:23.515 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test_image_replay_state cluster1 mirror test started 2024-04-06T09:38:23.515 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:38:23.515 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:38:23.515 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:38:23.515 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local test_state=started 2024-04-06T09:38:23.515 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local status_result 2024-04-06T09:38:23.515 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local current_state=stopped 2024-04-06T09:38:23.516 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2024-04-06T09:38:23.516 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_instance=cluster1 2024-04-06T09:38:23.516 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ shift 2024-04-06T09:38:23.516 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -i state 2024-04-06T09:38:23.517 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:38:23.517 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance=cluster1 2024-04-06T09:38:23.517 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local loop_instance 2024-04-06T09:38:23.517 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:38:23.517 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sleep 0 2024-04-06T09:38:23.517 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:38:23.518 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ seq 0 3 2024-04-06T09:38:23.520 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:38:23.520 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2024-04-06T09:38:23.520 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1:0 2024-04-06T09:38:23.521 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ shift 2024-04-06T09:38:23.521 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:38:23.521 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:38:23.521 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:38:23.521 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=cluster 2024-04-06T09:38:23.521 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:38:23.521 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:38:23.521 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:38:23.521 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:38:23.521 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval cluster=cluster1 2024-04-06T09:38:23.521 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ cluster=cluster1 2024-04-06T09:38:23.521 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:38:23.521 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:38:23.522 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:38:23.522 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local local_cluster=cluster1:0 2024-04-06T09:38:23.522 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local cluster=cluster1 2024-04-06T09:38:23.522 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local instance 2024-04-06T09:38:23.522 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:38:23.522 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local val=cluster1:0 2024-04-06T09:38:23.522 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local cluster_var_name=local_cluster 2024-04-06T09:38:23.522 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local instance_var_name=instance 2024-04-06T09:38:23.522 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ cluster=cluster1 2024-04-06T09:38:23.523 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:38:23.523 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ '[' 0 = cluster1:0 ']' 2024-04-06T09:38:23.523 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ eval local_cluster=cluster1 2024-04-06T09:38:23.523 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++++ local_cluster=cluster1 2024-04-06T09:38:23.523 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ eval instance=0 2024-04-06T09:38:23.523 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++++ instance=0 2024-04-06T09:38:23.523 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:38:23.545 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:38:23.545 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:38:23.546 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:38:23.546 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2024-04-06T09:38:23.666 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ return 0 2024-04-06T09:38:23.667 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ status_result=' "state": "Replaying"' 2024-04-06T09:38:23.667 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ echo ' "state": "Replaying"' 2024-04-06T09:38:23.667 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep -i Replaying 2024-04-06T09:38:23.669 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: "state": "Replaying" 2024-04-06T09:38:23.670 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ current_state=started 2024-04-06T09:38:23.670 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test started = started 2024-04-06T09:38:23.670 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:38:23.671 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror test 2024-04-06T09:38:23.671 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local local_cluster=cluster1 2024-04-06T09:38:23.671 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-04-06T09:38:23.671 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:38:23.671 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:38:23.671 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' journal = journal ']' 2024-04-06T09:38:23.671 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_journal_replay_complete cluster1 cluster2 mirror test 2024-04-06T09:38:23.671 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local local_cluster=cluster1 2024-04-06T09:38:23.671 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-04-06T09:38:23.671 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:38:23.671 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:38:23.671 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local s master_pos mirror_pos last_mirror_pos 2024-04-06T09:38:23.671 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local master_tag master_entry mirror_tag mirror_entry 2024-04-06T09:38:23.671 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:38:23.672 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:38:23.672 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:38:23.872 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:38:23.873 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:38:23.873 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:38:23.873 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:38:23.873 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:38:23.873 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:38:23.873 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:38:23.873 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:38:23.873 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:38:23.873 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:38:23.874 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:38:23.874 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:38:23.874 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:38:23.874 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:38:23.874 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:38:23.876 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:38:23.876 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:38:23.876 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:38:23.876 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:38:23.877 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:38:23.877 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:38:23.877 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:38:23.877 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:38:23.877 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:38:23.877 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:38:23.877 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:38:23.877 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:38:23.877 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:38:23.877 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:38:23.877 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:38:23.877 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:38:23.878 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:38:23.878 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:38:23.878 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:38:23.878 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:38:23.878 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:38:23.878 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:38:23.878 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:38:23.878 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:38:23.878 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:38:23.878 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:38:23.878 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:38:23.878 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:38:23.879 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:38:23.879 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:38:23.879 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:38:23.879 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:38:23.879 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:38:23.879 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:38:23.902 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:38:23.903 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:38:23.903 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:38:23.903 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:38:24.033 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:38:24.034 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:38:24.034 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:38:24.034 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:38:24.035 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:38:24.035 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:38:24.035 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:38:24.035 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:38:24.035 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:38:24.035 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:38:24.035 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:38:24.036 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:38:24.036 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:38:24.037 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:38:24.038 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:38:24.038 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:38:24.083 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 16 2024-04-06T09:38:24.083 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 19 2024-04-06T09:38:24.083 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:38:24.084 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=77, tag_tid=12, entry_tid=1], [object_number=76, tag_tid=12, entry_tid=0], [object_number=79, tag_tid=11, entry_tid=17175], [object_number=78, tag_tid=11, entry_tid=17174]]], state=connected] 2024-04-06T09:38:24.084 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=67, tag_tid=11, entry_tid=6599], [object_number=66, tag_tid=11, entry_tid=6598], [object_number=65, tag_tid=11, entry_tid=6597], [object_number=64, tag_tid=11, entry_tid=6596]]], state=connected] 2024-04-06T09:38:24.087 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:38:24.089 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=77, tag_tid=12, entry_tid=1' 2024-04-06T09:38:24.089 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:38:24.089 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:38:24.089 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:38:24.089 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:38:24.089 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:38:24.089 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:38:24.089 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:38:24.089 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:38:24.089 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:38:24.090 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:38:24.090 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:38:24.090 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:38:24.090 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:38:24.091 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:38:24.091 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:38:24.136 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 16 2024-04-06T09:38:24.136 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 19 2024-04-06T09:38:24.136 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:38:24.137 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=77, tag_tid=12, entry_tid=1], [object_number=76, tag_tid=12, entry_tid=0], [object_number=79, tag_tid=11, entry_tid=17175], [object_number=78, tag_tid=11, entry_tid=17174]]], state=connected] 2024-04-06T09:38:24.137 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=67, tag_tid=11, entry_tid=6599], [object_number=66, tag_tid=11, entry_tid=6598], [object_number=65, tag_tid=11, entry_tid=6597], [object_number=64, tag_tid=11, entry_tid=6596]]], state=connected] 2024-04-06T09:38:24.139 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:38:24.141 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=67, tag_tid=11, entry_tid=6599' 2024-04-06T09:38:24.141 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=77, tag_tid=12, entry_tid=1' -a 'object_number=77, tag_tid=12, entry_tid=1' = 'object_number=67, tag_tid=11, entry_tid=6599' 2024-04-06T09:38:24.142 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=67, tag_tid=11, entry_tid=6599' '!=' '' 2024-04-06T09:38:24.142 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:38:24.142 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=67, tag_tid=11, entry_tid=6599' = '' 2024-04-06T09:38:24.142 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=67, tag_tid=11, entry_tid=6599' 2024-04-06T09:38:24.142 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=77, tag_tid=12, entry_tid=1' 2024-04-06T09:38:24.142 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:38:24.142 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:38:24.143 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=12 2024-04-06T09:38:24.143 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=67, tag_tid=11, entry_tid=6599' 2024-04-06T09:38:24.143 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:38:24.143 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:38:24.144 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=11 2024-04-06T09:38:24.145 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=77, tag_tid=12, entry_tid=1' 2024-04-06T09:38:24.145 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:38:24.145 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:38:24.147 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:38:24.147 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=67, tag_tid=11, entry_tid=6599' 2024-04-06T09:38:24.147 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:38:24.147 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:38:24.148 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=6599 2024-04-06T09:38:24.148 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 12 = 11 -a 1 -le 6599 2024-04-06T09:38:24.148 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:38:24.148 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:38:24.148 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:38:24.350 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:38:24.350 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:38:24.350 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:38:24.350 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:38:24.350 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:38:24.350 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:38:24.350 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:38:24.351 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:38:24.351 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:38:24.351 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:38:24.351 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:38:24.351 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:38:24.351 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:38:24.351 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:38:24.351 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:38:24.351 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:38:24.352 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:38:24.355 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:38:24.355 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:38:24.355 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:38:24.356 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:38:24.356 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:38:24.356 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:38:24.356 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:38:24.356 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:38:24.356 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:38:24.356 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:38:24.356 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:38:24.356 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:38:24.356 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:38:24.356 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:38:24.356 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:38:24.356 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:38:24.357 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:38:24.357 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:38:24.357 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:38:24.357 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:38:24.357 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:38:24.357 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:38:24.357 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:38:24.357 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:38:24.357 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:38:24.357 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:38:24.357 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:38:24.357 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:38:24.357 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:38:24.357 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:38:24.358 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:38:24.358 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:38:24.382 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:38:24.382 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:38:24.382 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:38:24.382 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:38:24.513 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:38:24.515 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:38:24.515 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:38:24.515 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:38:24.515 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:38:24.515 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:38:24.515 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:38:24.515 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:38:24.515 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:38:24.515 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:38:24.516 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:38:24.516 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:38:24.516 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:38:24.517 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:38:24.518 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:38:24.518 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:38:24.569 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 16 2024-04-06T09:38:24.569 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 19 2024-04-06T09:38:24.569 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:38:24.569 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=77, tag_tid=12, entry_tid=1], [object_number=76, tag_tid=12, entry_tid=0], [object_number=79, tag_tid=11, entry_tid=17175], [object_number=78, tag_tid=11, entry_tid=17174]]], state=connected] 2024-04-06T09:38:24.570 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=64, tag_tid=11, entry_tid=6680], [object_number=67, tag_tid=11, entry_tid=6679], [object_number=66, tag_tid=11, entry_tid=6678], [object_number=65, tag_tid=11, entry_tid=6677]]], state=connected] 2024-04-06T09:38:24.573 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:38:24.574 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=77, tag_tid=12, entry_tid=1' 2024-04-06T09:38:24.576 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:38:24.576 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:38:24.576 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:38:24.576 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:38:24.576 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:38:24.576 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:38:24.576 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:38:24.577 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:38:24.577 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:38:24.577 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:38:24.577 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:38:24.577 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:38:24.578 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:38:24.578 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:38:24.578 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:38:24.623 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 16 2024-04-06T09:38:24.623 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 19 2024-04-06T09:38:24.624 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:38:24.624 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=77, tag_tid=12, entry_tid=1], [object_number=76, tag_tid=12, entry_tid=0], [object_number=79, tag_tid=11, entry_tid=17175], [object_number=78, tag_tid=11, entry_tid=17174]]], state=connected] 2024-04-06T09:38:24.624 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=64, tag_tid=11, entry_tid=6680], [object_number=67, tag_tid=11, entry_tid=6679], [object_number=66, tag_tid=11, entry_tid=6678], [object_number=65, tag_tid=11, entry_tid=6677]]], state=connected] 2024-04-06T09:38:24.627 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:38:24.629 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=64, tag_tid=11, entry_tid=6680' 2024-04-06T09:38:24.629 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=77, tag_tid=12, entry_tid=1' -a 'object_number=77, tag_tid=12, entry_tid=1' = 'object_number=64, tag_tid=11, entry_tid=6680' 2024-04-06T09:38:24.629 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=64, tag_tid=11, entry_tid=6680' '!=' 'object_number=67, tag_tid=11, entry_tid=6599' 2024-04-06T09:38:24.629 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:38:24.630 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=64, tag_tid=11, entry_tid=6680' = 'object_number=67, tag_tid=11, entry_tid=6599' 2024-04-06T09:38:24.630 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=64, tag_tid=11, entry_tid=6680' 2024-04-06T09:38:24.630 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=77, tag_tid=12, entry_tid=1' 2024-04-06T09:38:24.630 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:38:24.630 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:38:24.631 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=12 2024-04-06T09:38:24.631 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=64, tag_tid=11, entry_tid=6680' 2024-04-06T09:38:24.631 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:38:24.631 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:38:24.632 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=11 2024-04-06T09:38:24.632 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=77, tag_tid=12, entry_tid=1' 2024-04-06T09:38:24.632 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:38:24.633 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:38:24.633 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:38:24.634 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=64, tag_tid=11, entry_tid=6680' 2024-04-06T09:38:24.634 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:38:24.634 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:38:24.635 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=6680 2024-04-06T09:38:24.635 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 12 = 11 -a 1 -le 6680 2024-04-06T09:38:24.635 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:38:24.636 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:38:24.636 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:38:24.837 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:38:24.837 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:38:24.837 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:38:24.838 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:38:24.838 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:38:24.838 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:38:24.838 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:38:24.838 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:38:24.838 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:38:24.838 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:38:24.838 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:38:24.838 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:38:24.838 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:38:24.838 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:38:24.838 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:38:24.839 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:38:24.839 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:38:24.841 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:38:24.841 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:38:24.841 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:38:24.841 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:38:24.841 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:38:24.841 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:38:24.841 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:38:24.841 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:38:24.842 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:38:24.842 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:38:24.842 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:38:24.842 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:38:24.842 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:38:24.842 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:38:24.842 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:38:24.842 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:38:24.843 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:38:24.843 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:38:24.843 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:38:24.843 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:38:24.843 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:38:24.843 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:38:24.843 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:38:24.844 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:38:24.844 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:38:24.844 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:38:24.844 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:38:24.844 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:38:24.844 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:38:24.844 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:38:24.844 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:38:24.844 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:38:24.869 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:38:24.869 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:38:24.869 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:38:24.870 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:38:24.990 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:38:24.991 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:38:24.992 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:38:24.992 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:38:24.992 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:38:24.992 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:38:24.992 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:38:24.992 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:38:24.992 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:38:24.992 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:38:24.993 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:38:24.993 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:38:24.993 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:38:24.995 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:38:24.996 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:38:24.996 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:38:25.052 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 16 2024-04-06T09:38:25.052 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 19 2024-04-06T09:38:25.052 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:38:25.052 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=77, tag_tid=12, entry_tid=1], [object_number=76, tag_tid=12, entry_tid=0], [object_number=79, tag_tid=11, entry_tid=17175], [object_number=78, tag_tid=11, entry_tid=17174]]], state=connected] 2024-04-06T09:38:25.052 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=66, tag_tid=11, entry_tid=6766], [object_number=65, tag_tid=11, entry_tid=6765], [object_number=64, tag_tid=11, entry_tid=6764], [object_number=67, tag_tid=11, entry_tid=6763]]], state=connected] 2024-04-06T09:38:25.055 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:38:25.056 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=77, tag_tid=12, entry_tid=1' 2024-04-06T09:38:25.057 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:38:25.057 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:38:25.057 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:38:25.057 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:38:25.058 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:38:25.058 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:38:25.058 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:38:25.058 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:38:25.058 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:38:25.058 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:38:25.058 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:38:25.058 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:38:25.059 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:38:25.059 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:38:25.059 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:38:25.106 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 16 2024-04-06T09:38:25.106 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 19 2024-04-06T09:38:25.106 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:38:25.106 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=77, tag_tid=12, entry_tid=1], [object_number=76, tag_tid=12, entry_tid=0], [object_number=79, tag_tid=11, entry_tid=17175], [object_number=78, tag_tid=11, entry_tid=17174]]], state=connected] 2024-04-06T09:38:25.106 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=66, tag_tid=11, entry_tid=6766], [object_number=65, tag_tid=11, entry_tid=6765], [object_number=64, tag_tid=11, entry_tid=6764], [object_number=67, tag_tid=11, entry_tid=6763]]], state=connected] 2024-04-06T09:38:25.109 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:38:25.111 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=66, tag_tid=11, entry_tid=6766' 2024-04-06T09:38:25.111 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=77, tag_tid=12, entry_tid=1' -a 'object_number=77, tag_tid=12, entry_tid=1' = 'object_number=66, tag_tid=11, entry_tid=6766' 2024-04-06T09:38:25.111 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=66, tag_tid=11, entry_tid=6766' '!=' 'object_number=64, tag_tid=11, entry_tid=6680' 2024-04-06T09:38:25.111 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:38:25.112 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=66, tag_tid=11, entry_tid=6766' = 'object_number=64, tag_tid=11, entry_tid=6680' 2024-04-06T09:38:25.112 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=66, tag_tid=11, entry_tid=6766' 2024-04-06T09:38:25.112 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=77, tag_tid=12, entry_tid=1' 2024-04-06T09:38:25.112 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:38:25.112 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:38:25.113 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=12 2024-04-06T09:38:25.113 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=66, tag_tid=11, entry_tid=6766' 2024-04-06T09:38:25.114 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:38:25.114 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:38:25.114 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=11 2024-04-06T09:38:25.115 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=77, tag_tid=12, entry_tid=1' 2024-04-06T09:38:25.115 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:38:25.115 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:38:25.116 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:38:25.117 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=66, tag_tid=11, entry_tid=6766' 2024-04-06T09:38:25.117 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:38:25.117 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:38:25.118 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=6766 2024-04-06T09:38:25.119 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 12 = 11 -a 1 -le 6766 2024-04-06T09:38:25.119 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:38:25.119 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:38:25.119 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:38:25.321 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:38:25.321 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:38:25.321 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:38:25.321 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:38:25.321 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:38:25.321 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:38:25.321 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:38:25.321 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:38:25.321 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:38:25.322 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:38:25.322 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:38:25.322 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:38:25.322 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:38:25.322 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:38:25.322 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:38:25.322 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:38:25.323 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:38:25.325 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:38:25.325 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:38:25.325 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:38:25.326 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:38:25.326 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:38:25.326 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:38:25.326 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:38:25.326 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:38:25.326 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:38:25.326 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:38:25.326 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:38:25.326 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:38:25.326 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:38:25.326 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:38:25.326 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:38:25.326 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:38:25.327 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:38:25.327 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:38:25.327 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:38:25.327 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:38:25.327 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:38:25.327 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:38:25.327 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:38:25.327 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:38:25.327 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:38:25.327 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:38:25.327 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:38:25.327 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:38:25.327 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:38:25.327 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:38:25.328 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:38:25.328 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:38:25.350 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:38:25.351 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:38:25.351 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:38:25.351 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:38:25.382 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.0 with skip (0.6442411922215063) > weight (0.25) 2024-04-06T09:38:25.382 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.1 with skip (0.7690444165427074) > weight (0.25) 2024-04-06T09:38:25.382 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.2 with skip (0.7157093829107845) > weight (0.25) 2024-04-06T09:38:25.382 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.3 with skip (0.8555563230857292) > weight (0.25) 2024-04-06T09:38:25.382 INFO:tasks.rbd_mirror_thrash:waiting for 67 secs before thrashing 2024-04-06T09:38:25.479 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:38:25.480 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:38:25.480 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:38:25.480 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:38:25.481 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:38:25.481 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:38:25.481 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:38:25.481 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:38:25.481 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:38:25.481 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:38:25.481 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:38:25.481 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:38:25.481 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:38:25.482 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:38:25.482 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:38:25.482 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:38:25.528 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 16 2024-04-06T09:38:25.528 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 19 2024-04-06T09:38:25.528 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:38:25.528 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=77, tag_tid=12, entry_tid=1], [object_number=76, tag_tid=12, entry_tid=0], [object_number=79, tag_tid=11, entry_tid=17175], [object_number=78, tag_tid=11, entry_tid=17174]]], state=connected] 2024-04-06T09:38:25.528 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=64, tag_tid=11, entry_tid=6868], [object_number=67, tag_tid=11, entry_tid=6867], [object_number=66, tag_tid=11, entry_tid=6866], [object_number=65, tag_tid=11, entry_tid=6865]]], state=connected] 2024-04-06T09:38:25.531 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:38:25.533 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=77, tag_tid=12, entry_tid=1' 2024-04-06T09:38:25.534 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:38:25.534 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:38:25.534 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:38:25.534 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:38:25.535 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:38:25.535 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:38:25.535 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:38:25.535 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:38:25.535 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:38:25.535 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:38:25.535 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:38:25.535 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:38:25.536 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:38:25.536 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:38:25.536 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:38:25.583 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 16 2024-04-06T09:38:25.583 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 19 2024-04-06T09:38:25.583 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:38:25.583 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=77, tag_tid=12, entry_tid=1], [object_number=76, tag_tid=12, entry_tid=0], [object_number=79, tag_tid=11, entry_tid=17175], [object_number=78, tag_tid=11, entry_tid=17174]]], state=connected] 2024-04-06T09:38:25.583 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=64, tag_tid=11, entry_tid=6868], [object_number=67, tag_tid=11, entry_tid=6867], [object_number=66, tag_tid=11, entry_tid=6866], [object_number=65, tag_tid=11, entry_tid=6865]]], state=connected] 2024-04-06T09:38:25.586 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:38:25.588 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=64, tag_tid=11, entry_tid=6868' 2024-04-06T09:38:25.588 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=77, tag_tid=12, entry_tid=1' -a 'object_number=77, tag_tid=12, entry_tid=1' = 'object_number=64, tag_tid=11, entry_tid=6868' 2024-04-06T09:38:25.588 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=64, tag_tid=11, entry_tid=6868' '!=' 'object_number=66, tag_tid=11, entry_tid=6766' 2024-04-06T09:38:25.588 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:38:25.588 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=64, tag_tid=11, entry_tid=6868' = 'object_number=66, tag_tid=11, entry_tid=6766' 2024-04-06T09:38:25.588 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=64, tag_tid=11, entry_tid=6868' 2024-04-06T09:38:25.589 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=77, tag_tid=12, entry_tid=1' 2024-04-06T09:38:25.589 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:38:25.589 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:38:25.589 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=12 2024-04-06T09:38:25.590 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=64, tag_tid=11, entry_tid=6868' 2024-04-06T09:38:25.590 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:38:25.590 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:38:25.591 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=11 2024-04-06T09:38:25.592 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=77, tag_tid=12, entry_tid=1' 2024-04-06T09:38:25.592 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:38:25.592 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:38:25.593 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:38:25.593 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=64, tag_tid=11, entry_tid=6868' 2024-04-06T09:38:25.593 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:38:25.593 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:38:25.594 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=6868 2024-04-06T09:38:25.595 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 12 = 11 -a 1 -le 6868 2024-04-06T09:38:25.595 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:38:25.595 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:38:25.595 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:38:25.796 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:38:25.797 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:38:25.797 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:38:25.797 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:38:25.797 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:38:25.797 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:38:25.797 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:38:25.797 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:38:25.797 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:38:25.797 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:38:25.797 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:38:25.797 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:38:25.798 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:38:25.798 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:38:25.798 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:38:25.798 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:38:25.798 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:38:25.800 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:38:25.800 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:38:25.800 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:38:25.800 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:38:25.800 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:38:25.800 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:38:25.800 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:38:25.800 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:38:25.800 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:38:25.800 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:38:25.801 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:38:25.801 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:38:25.801 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:38:25.801 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:38:25.801 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:38:25.801 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:38:25.802 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:38:25.802 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:38:25.802 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:38:25.802 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:38:25.802 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:38:25.802 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:38:25.802 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:38:25.802 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:38:25.802 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:38:25.802 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:38:25.802 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:38:25.803 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:38:25.803 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:38:25.803 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:38:25.803 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:38:25.803 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:38:25.828 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:38:25.828 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:38:25.828 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:38:25.828 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:38:25.951 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:38:25.952 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:38:25.952 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:38:25.952 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:38:25.952 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:38:25.952 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:38:25.952 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:38:25.952 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:38:25.953 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:38:25.953 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:38:25.953 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:38:25.953 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:38:25.953 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:38:25.956 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:38:25.956 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:38:25.956 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:38:26.008 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 16 2024-04-06T09:38:26.008 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 19 2024-04-06T09:38:26.008 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:38:26.008 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=77, tag_tid=12, entry_tid=1], [object_number=76, tag_tid=12, entry_tid=0], [object_number=79, tag_tid=11, entry_tid=17175], [object_number=78, tag_tid=11, entry_tid=17174]]], state=connected] 2024-04-06T09:38:26.008 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=65, tag_tid=11, entry_tid=6961], [object_number=64, tag_tid=11, entry_tid=6960], [object_number=67, tag_tid=11, entry_tid=6959], [object_number=66, tag_tid=11, entry_tid=6958]]], state=connected] 2024-04-06T09:38:26.012 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:38:26.013 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=77, tag_tid=12, entry_tid=1' 2024-04-06T09:38:26.014 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:38:26.014 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:38:26.014 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:38:26.014 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:38:26.014 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:38:26.015 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:38:26.015 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:38:26.015 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:38:26.015 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:38:26.015 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:38:26.015 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:38:26.015 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:38:26.015 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:38:26.016 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:38:26.016 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:38:26.063 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 16 2024-04-06T09:38:26.063 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 19 2024-04-06T09:38:26.063 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:38:26.063 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=77, tag_tid=12, entry_tid=1], [object_number=76, tag_tid=12, entry_tid=0], [object_number=79, tag_tid=11, entry_tid=17175], [object_number=78, tag_tid=11, entry_tid=17174]]], state=connected] 2024-04-06T09:38:26.063 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=65, tag_tid=11, entry_tid=6961], [object_number=64, tag_tid=11, entry_tid=6960], [object_number=67, tag_tid=11, entry_tid=6959], [object_number=66, tag_tid=11, entry_tid=6958]]], state=connected] 2024-04-06T09:38:26.066 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:38:26.068 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=65, tag_tid=11, entry_tid=6961' 2024-04-06T09:38:26.068 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=77, tag_tid=12, entry_tid=1' -a 'object_number=77, tag_tid=12, entry_tid=1' = 'object_number=65, tag_tid=11, entry_tid=6961' 2024-04-06T09:38:26.068 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=65, tag_tid=11, entry_tid=6961' '!=' 'object_number=64, tag_tid=11, entry_tid=6868' 2024-04-06T09:38:26.068 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:38:26.068 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=65, tag_tid=11, entry_tid=6961' = 'object_number=64, tag_tid=11, entry_tid=6868' 2024-04-06T09:38:26.069 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=65, tag_tid=11, entry_tid=6961' 2024-04-06T09:38:26.069 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=77, tag_tid=12, entry_tid=1' 2024-04-06T09:38:26.069 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:38:26.069 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:38:26.070 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=12 2024-04-06T09:38:26.070 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=65, tag_tid=11, entry_tid=6961' 2024-04-06T09:38:26.071 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:38:26.071 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:38:26.072 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=11 2024-04-06T09:38:26.072 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=77, tag_tid=12, entry_tid=1' 2024-04-06T09:38:26.072 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:38:26.072 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:38:26.073 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:38:26.074 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=65, tag_tid=11, entry_tid=6961' 2024-04-06T09:38:26.074 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:38:26.074 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:38:26.075 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=6961 2024-04-06T09:38:26.076 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 12 = 11 -a 1 -le 6961 2024-04-06T09:38:26.076 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:38:26.076 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:38:26.076 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:38:26.277 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:38:26.278 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:38:26.278 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:38:26.278 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:38:26.278 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:38:26.278 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:38:26.278 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:38:26.278 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:38:26.278 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:38:26.278 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:38:26.278 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:38:26.278 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:38:26.278 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:38:26.279 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:38:26.279 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:38:26.279 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:38:26.279 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:38:26.283 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:38:26.283 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:38:26.283 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:38:26.283 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:38:26.283 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:38:26.283 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:38:26.283 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:38:26.283 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:38:26.283 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:38:26.283 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:38:26.283 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:38:26.283 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:38:26.283 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:38:26.284 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:38:26.284 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:38:26.284 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:38:26.284 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:38:26.284 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:38:26.284 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:38:26.284 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:38:26.284 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:38:26.284 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:38:26.284 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:38:26.284 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:38:26.284 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:38:26.284 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:38:26.284 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:38:26.285 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:38:26.285 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:38:26.285 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:38:26.285 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:38:26.285 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:38:26.310 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:38:26.310 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:38:26.310 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:38:26.310 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:38:26.439 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:38:26.440 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:38:26.440 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:38:26.440 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:38:26.441 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:38:26.441 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:38:26.441 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:38:26.441 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:38:26.441 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:38:26.441 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:38:26.441 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:38:26.441 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:38:26.442 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:38:26.443 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:38:26.443 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:38:26.444 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:38:26.497 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 16 2024-04-06T09:38:26.497 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 19 2024-04-06T09:38:26.498 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:38:26.498 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=77, tag_tid=12, entry_tid=1], [object_number=76, tag_tid=12, entry_tid=0], [object_number=79, tag_tid=11, entry_tid=17175], [object_number=78, tag_tid=11, entry_tid=17174]]], state=connected] 2024-04-06T09:38:26.498 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=64, tag_tid=11, entry_tid=7072], [object_number=67, tag_tid=11, entry_tid=7071], [object_number=66, tag_tid=11, entry_tid=7070], [object_number=65, tag_tid=11, entry_tid=7069]]], state=connected] 2024-04-06T09:38:26.501 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:38:26.502 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=77, tag_tid=12, entry_tid=1' 2024-04-06T09:38:26.504 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:38:26.504 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:38:26.504 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:38:26.504 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:38:26.504 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:38:26.505 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:38:26.506 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:38:26.506 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:38:26.506 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:38:26.506 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:38:26.506 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:38:26.507 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:38:26.507 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:38:26.507 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:38:26.508 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:38:26.553 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 16 2024-04-06T09:38:26.553 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 19 2024-04-06T09:38:26.553 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:38:26.553 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=77, tag_tid=12, entry_tid=1], [object_number=76, tag_tid=12, entry_tid=0], [object_number=79, tag_tid=11, entry_tid=17175], [object_number=78, tag_tid=11, entry_tid=17174]]], state=connected] 2024-04-06T09:38:26.553 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=64, tag_tid=11, entry_tid=7072], [object_number=67, tag_tid=11, entry_tid=7071], [object_number=66, tag_tid=11, entry_tid=7070], [object_number=65, tag_tid=11, entry_tid=7069]]], state=connected] 2024-04-06T09:38:26.556 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:38:26.558 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=64, tag_tid=11, entry_tid=7072' 2024-04-06T09:38:26.559 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=77, tag_tid=12, entry_tid=1' -a 'object_number=77, tag_tid=12, entry_tid=1' = 'object_number=64, tag_tid=11, entry_tid=7072' 2024-04-06T09:38:26.559 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=64, tag_tid=11, entry_tid=7072' '!=' 'object_number=65, tag_tid=11, entry_tid=6961' 2024-04-06T09:38:26.559 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:38:26.559 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=64, tag_tid=11, entry_tid=7072' = 'object_number=65, tag_tid=11, entry_tid=6961' 2024-04-06T09:38:26.559 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=64, tag_tid=11, entry_tid=7072' 2024-04-06T09:38:26.559 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=77, tag_tid=12, entry_tid=1' 2024-04-06T09:38:26.559 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:38:26.559 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:38:26.559 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=12 2024-04-06T09:38:26.560 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=64, tag_tid=11, entry_tid=7072' 2024-04-06T09:38:26.560 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:38:26.560 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:38:26.562 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=11 2024-04-06T09:38:26.562 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=77, tag_tid=12, entry_tid=1' 2024-04-06T09:38:26.562 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:38:26.562 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:38:26.563 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:38:26.563 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=64, tag_tid=11, entry_tid=7072' 2024-04-06T09:38:26.564 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:38:26.564 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:38:26.565 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=7072 2024-04-06T09:38:26.565 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 12 = 11 -a 1 -le 7072 2024-04-06T09:38:26.565 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:38:26.565 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:38:26.565 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:38:26.767 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:38:26.767 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:38:26.768 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:38:26.768 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:38:26.768 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:38:26.768 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:38:26.768 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:38:26.768 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:38:26.768 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:38:26.769 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:38:26.769 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:38:26.769 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:38:26.769 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:38:26.769 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:38:26.769 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:38:26.774 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:38:26.774 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:38:26.774 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:38:26.775 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:38:26.775 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:38:26.775 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:38:26.775 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:38:26.775 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:38:26.775 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:38:26.776 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:38:26.776 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:38:26.776 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:38:26.776 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:38:26.776 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:38:26.776 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:38:26.776 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:38:26.777 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:38:26.777 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:38:26.777 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:38:26.777 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:38:26.777 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:38:26.777 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:38:26.777 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:38:26.777 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:38:26.778 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:38:26.778 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:38:26.778 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:38:26.778 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:38:26.778 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:38:26.778 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:38:26.778 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:38:26.778 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:38:26.778 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:38:26.778 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:38:26.797 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:38:26.797 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:38:26.798 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:38:26.798 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:38:26.926 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:38:26.928 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:38:26.928 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:38:26.928 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:38:26.928 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:38:26.928 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:38:26.928 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:38:26.928 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:38:26.928 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:38:26.928 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:38:26.929 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:38:26.929 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:38:26.929 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:38:26.930 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:38:26.930 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:38:26.931 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:38:26.982 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 16 2024-04-06T09:38:26.982 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 19 2024-04-06T09:38:26.982 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:38:26.982 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=77, tag_tid=12, entry_tid=1], [object_number=76, tag_tid=12, entry_tid=0], [object_number=79, tag_tid=11, entry_tid=17175], [object_number=78, tag_tid=11, entry_tid=17174]]], state=connected] 2024-04-06T09:38:26.982 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=65, tag_tid=11, entry_tid=7173], [object_number=64, tag_tid=11, entry_tid=7172], [object_number=67, tag_tid=11, entry_tid=7171], [object_number=66, tag_tid=11, entry_tid=7170]]], state=connected] 2024-04-06T09:38:26.985 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:38:26.987 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=77, tag_tid=12, entry_tid=1' 2024-04-06T09:38:26.988 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:38:26.988 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:38:26.988 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:38:26.988 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:38:26.988 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:38:26.988 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:38:26.989 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:38:26.989 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:38:26.989 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:38:26.989 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:38:26.989 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:38:26.989 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:38:26.989 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:38:26.990 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:38:26.990 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:38:27.036 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 16 2024-04-06T09:38:27.036 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 19 2024-04-06T09:38:27.036 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:38:27.036 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=77, tag_tid=12, entry_tid=1], [object_number=76, tag_tid=12, entry_tid=0], [object_number=79, tag_tid=11, entry_tid=17175], [object_number=78, tag_tid=11, entry_tid=17174]]], state=connected] 2024-04-06T09:38:27.036 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=65, tag_tid=11, entry_tid=7173], [object_number=64, tag_tid=11, entry_tid=7172], [object_number=67, tag_tid=11, entry_tid=7171], [object_number=66, tag_tid=11, entry_tid=7170]]], state=connected] 2024-04-06T09:38:27.040 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:38:27.043 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=65, tag_tid=11, entry_tid=7173' 2024-04-06T09:38:27.043 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=77, tag_tid=12, entry_tid=1' -a 'object_number=77, tag_tid=12, entry_tid=1' = 'object_number=65, tag_tid=11, entry_tid=7173' 2024-04-06T09:38:27.043 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=65, tag_tid=11, entry_tid=7173' '!=' 'object_number=64, tag_tid=11, entry_tid=7072' 2024-04-06T09:38:27.043 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:38:27.043 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=65, tag_tid=11, entry_tid=7173' = 'object_number=64, tag_tid=11, entry_tid=7072' 2024-04-06T09:38:27.043 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=65, tag_tid=11, entry_tid=7173' 2024-04-06T09:38:27.043 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=77, tag_tid=12, entry_tid=1' 2024-04-06T09:38:27.044 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:38:27.044 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:38:27.046 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=12 2024-04-06T09:38:27.046 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=65, tag_tid=11, entry_tid=7173' 2024-04-06T09:38:27.046 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:38:27.046 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:38:27.047 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=11 2024-04-06T09:38:27.047 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=77, tag_tid=12, entry_tid=1' 2024-04-06T09:38:27.047 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:38:27.047 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:38:27.049 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:38:27.049 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=65, tag_tid=11, entry_tid=7173' 2024-04-06T09:38:27.049 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:38:27.050 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:38:27.050 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=7173 2024-04-06T09:38:27.050 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 12 = 11 -a 1 -le 7173 2024-04-06T09:38:27.050 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:38:27.050 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:38:27.051 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:38:27.251 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:38:27.251 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:38:27.251 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:38:27.251 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:38:27.251 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:38:27.251 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:38:27.251 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:38:27.251 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:38:27.251 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:38:27.251 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:38:27.251 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:38:27.251 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:38:27.252 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:38:27.252 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:38:27.252 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:38:27.252 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:38:27.252 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:38:27.255 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:38:27.255 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:38:27.255 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:38:27.255 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:38:27.255 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:38:27.255 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:38:27.256 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:38:27.256 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:38:27.256 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:38:27.256 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:38:27.256 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:38:27.256 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:38:27.256 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:38:27.256 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:38:27.256 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:38:27.256 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:38:27.256 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:38:27.256 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:38:27.256 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:38:27.256 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:38:27.256 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:38:27.257 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:38:27.257 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:38:27.257 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:38:27.257 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:38:27.257 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:38:27.257 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:38:27.257 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:38:27.257 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:38:27.257 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:38:27.257 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:38:27.257 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:38:27.280 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:38:27.281 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:38:27.281 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:38:27.281 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:38:27.414 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:38:27.416 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:38:27.416 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:38:27.416 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:38:27.416 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:38:27.416 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:38:27.416 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:38:27.416 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:38:27.417 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:38:27.417 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:38:27.417 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:38:27.417 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:38:27.417 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:38:27.418 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:38:27.418 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:38:27.418 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:38:27.475 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 16 2024-04-06T09:38:27.475 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 19 2024-04-06T09:38:27.475 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:38:27.475 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=77, tag_tid=12, entry_tid=1], [object_number=76, tag_tid=12, entry_tid=0], [object_number=79, tag_tid=11, entry_tid=17175], [object_number=78, tag_tid=11, entry_tid=17174]]], state=connected] 2024-04-06T09:38:27.475 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=66, tag_tid=11, entry_tid=7274], [object_number=65, tag_tid=11, entry_tid=7273], [object_number=64, tag_tid=11, entry_tid=7272], [object_number=67, tag_tid=11, entry_tid=7271]]], state=connected] 2024-04-06T09:38:27.479 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:38:27.480 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=77, tag_tid=12, entry_tid=1' 2024-04-06T09:38:27.481 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:38:27.481 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:38:27.481 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:38:27.481 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:38:27.482 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:38:27.482 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:38:27.482 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:38:27.482 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:38:27.482 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:38:27.482 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:38:27.482 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:38:27.482 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:38:27.483 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:38:27.483 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:38:27.483 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:38:27.530 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 16 2024-04-06T09:38:27.531 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 19 2024-04-06T09:38:27.531 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:38:27.531 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=77, tag_tid=12, entry_tid=1], [object_number=76, tag_tid=12, entry_tid=0], [object_number=79, tag_tid=11, entry_tid=17175], [object_number=78, tag_tid=11, entry_tid=17174]]], state=connected] 2024-04-06T09:38:27.531 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=66, tag_tid=11, entry_tid=7274], [object_number=65, tag_tid=11, entry_tid=7273], [object_number=64, tag_tid=11, entry_tid=7272], [object_number=67, tag_tid=11, entry_tid=7271]]], state=connected] 2024-04-06T09:38:27.534 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:38:27.535 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=66, tag_tid=11, entry_tid=7274' 2024-04-06T09:38:27.536 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=77, tag_tid=12, entry_tid=1' -a 'object_number=77, tag_tid=12, entry_tid=1' = 'object_number=66, tag_tid=11, entry_tid=7274' 2024-04-06T09:38:27.536 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=66, tag_tid=11, entry_tid=7274' '!=' 'object_number=65, tag_tid=11, entry_tid=7173' 2024-04-06T09:38:27.536 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:38:27.536 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=66, tag_tid=11, entry_tid=7274' = 'object_number=65, tag_tid=11, entry_tid=7173' 2024-04-06T09:38:27.536 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=66, tag_tid=11, entry_tid=7274' 2024-04-06T09:38:27.536 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=77, tag_tid=12, entry_tid=1' 2024-04-06T09:38:27.536 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:38:27.537 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:38:27.537 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=12 2024-04-06T09:38:27.538 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=66, tag_tid=11, entry_tid=7274' 2024-04-06T09:38:27.538 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:38:27.538 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:38:27.539 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=11 2024-04-06T09:38:27.539 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=77, tag_tid=12, entry_tid=1' 2024-04-06T09:38:27.539 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:38:27.540 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:38:27.540 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:38:27.541 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=66, tag_tid=11, entry_tid=7274' 2024-04-06T09:38:27.541 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:38:27.541 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:38:27.542 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=7274 2024-04-06T09:38:27.543 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 12 = 11 -a 1 -le 7274 2024-04-06T09:38:27.543 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:38:27.543 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:38:27.543 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:38:27.745 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:38:27.745 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:38:27.745 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:38:27.745 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:38:27.745 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:38:27.745 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:38:27.745 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:38:27.745 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:38:27.745 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:38:27.745 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:38:27.745 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:38:27.745 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:38:27.746 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:38:27.746 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:38:27.746 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:38:27.747 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:38:27.747 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:38:27.747 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:38:27.747 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:38:27.747 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:38:27.747 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:38:27.748 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:38:27.748 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:38:27.748 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:38:27.748 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:38:27.748 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:38:27.748 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:38:27.748 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:38:27.748 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:38:27.748 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:38:27.748 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:38:27.748 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:38:27.748 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:38:27.750 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:38:27.750 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:38:27.750 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:38:27.750 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:38:27.750 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:38:27.750 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:38:27.750 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:38:27.750 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:38:27.750 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:38:27.751 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:38:27.751 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:38:27.751 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:38:27.751 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:38:27.751 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:38:27.751 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:38:27.751 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:38:27.776 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:38:27.776 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:38:27.776 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:38:27.776 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:38:27.909 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:38:27.911 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:38:27.911 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:38:27.911 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:38:27.911 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:38:27.911 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:38:27.911 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:38:27.911 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:38:27.911 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:38:27.911 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:38:27.911 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:38:27.912 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:38:27.912 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:38:27.913 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:38:27.914 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:38:27.914 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:38:27.962 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 16 2024-04-06T09:38:27.963 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 19 2024-04-06T09:38:27.963 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:38:27.963 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=77, tag_tid=12, entry_tid=1], [object_number=76, tag_tid=12, entry_tid=0], [object_number=79, tag_tid=11, entry_tid=17175], [object_number=78, tag_tid=11, entry_tid=17174]]], state=connected] 2024-04-06T09:38:27.963 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=67, tag_tid=11, entry_tid=7363], [object_number=66, tag_tid=11, entry_tid=7362], [object_number=65, tag_tid=11, entry_tid=7361], [object_number=64, tag_tid=11, entry_tid=7360]]], state=connected] 2024-04-06T09:38:27.966 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:38:27.967 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=77, tag_tid=12, entry_tid=1' 2024-04-06T09:38:27.968 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:38:27.969 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:38:27.969 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:38:27.969 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:38:27.969 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:38:27.969 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:38:27.969 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:38:27.969 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:38:27.969 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:38:27.969 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:38:27.969 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:38:27.969 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:38:27.970 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:38:27.970 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:38:27.970 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:38:28.017 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 16 2024-04-06T09:38:28.017 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 19 2024-04-06T09:38:28.017 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:38:28.018 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=77, tag_tid=12, entry_tid=1], [object_number=76, tag_tid=12, entry_tid=0], [object_number=79, tag_tid=11, entry_tid=17175], [object_number=78, tag_tid=11, entry_tid=17174]]], state=connected] 2024-04-06T09:38:28.018 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=67, tag_tid=11, entry_tid=7363], [object_number=66, tag_tid=11, entry_tid=7362], [object_number=65, tag_tid=11, entry_tid=7361], [object_number=64, tag_tid=11, entry_tid=7360]]], state=connected] 2024-04-06T09:38:28.021 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:38:28.023 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=67, tag_tid=11, entry_tid=7363' 2024-04-06T09:38:28.023 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=77, tag_tid=12, entry_tid=1' -a 'object_number=77, tag_tid=12, entry_tid=1' = 'object_number=67, tag_tid=11, entry_tid=7363' 2024-04-06T09:38:28.023 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=67, tag_tid=11, entry_tid=7363' '!=' 'object_number=66, tag_tid=11, entry_tid=7274' 2024-04-06T09:38:28.023 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:38:28.023 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=67, tag_tid=11, entry_tid=7363' = 'object_number=66, tag_tid=11, entry_tid=7274' 2024-04-06T09:38:28.023 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=67, tag_tid=11, entry_tid=7363' 2024-04-06T09:38:28.024 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=77, tag_tid=12, entry_tid=1' 2024-04-06T09:38:28.024 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:38:28.024 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:38:28.024 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=12 2024-04-06T09:38:28.025 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=67, tag_tid=11, entry_tid=7363' 2024-04-06T09:38:28.025 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:38:28.025 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:38:28.026 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=11 2024-04-06T09:38:28.026 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=77, tag_tid=12, entry_tid=1' 2024-04-06T09:38:28.027 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:38:28.027 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:38:28.027 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:38:28.028 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=67, tag_tid=11, entry_tid=7363' 2024-04-06T09:38:28.028 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:38:28.028 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:38:28.029 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=7363 2024-04-06T09:38:28.030 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 12 = 11 -a 1 -le 7363 2024-04-06T09:38:28.030 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:38:28.030 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:38:28.030 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:38:28.231 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:38:28.232 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:38:28.232 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:38:28.232 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:38:28.232 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:38:28.232 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:38:28.232 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:38:28.232 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:38:28.232 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:38:28.232 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:38:28.232 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:38:28.232 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:38:28.232 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:38:28.232 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:38:28.232 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:38:28.233 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:38:28.233 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:38:28.236 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:38:28.236 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:38:28.236 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:38:28.236 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:38:28.236 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:38:28.236 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:38:28.236 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:38:28.236 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:38:28.236 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:38:28.236 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:38:28.236 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:38:28.236 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:38:28.237 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:38:28.237 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:38:28.237 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:38:28.237 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:38:28.238 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:38:28.238 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:38:28.238 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:38:28.238 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:38:28.238 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:38:28.238 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:38:28.238 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:38:28.238 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:38:28.238 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:38:28.238 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:38:28.238 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:38:28.239 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:38:28.239 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:38:28.239 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:38:28.239 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:38:28.239 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:38:28.262 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:38:28.262 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:38:28.262 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:38:28.263 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:38:28.385 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.0 with skip (0.25915622886596523) > weight (0.25) 2024-04-06T09:38:28.385 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.1 with skip (0.944002685360105) > weight (0.25) 2024-04-06T09:38:28.385 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.2 with skip (0.9176084820216468) > weight (0.25) 2024-04-06T09:38:28.385 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.3 with skip (0.7366428646929183) > weight (0.25) 2024-04-06T09:38:28.385 INFO:tasks.rbd_mirror_thrash:waiting for 108 secs before thrashing 2024-04-06T09:38:28.400 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:38:28.401 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:38:28.402 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:38:28.402 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:38:28.402 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:38:28.402 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:38:28.402 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:38:28.402 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:38:28.402 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:38:28.402 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:38:28.403 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:38:28.403 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:38:28.403 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:38:28.405 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:38:28.405 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:38:28.406 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:38:28.456 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 16 2024-04-06T09:38:28.457 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 19 2024-04-06T09:38:28.457 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:38:28.457 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=77, tag_tid=12, entry_tid=1], [object_number=76, tag_tid=12, entry_tid=0], [object_number=79, tag_tid=11, entry_tid=17175], [object_number=78, tag_tid=11, entry_tid=17174]]], state=connected] 2024-04-06T09:38:28.457 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=65, tag_tid=11, entry_tid=7445], [object_number=64, tag_tid=11, entry_tid=7444], [object_number=67, tag_tid=11, entry_tid=7443], [object_number=66, tag_tid=11, entry_tid=7442]]], state=connected] 2024-04-06T09:38:28.460 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:38:28.461 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=77, tag_tid=12, entry_tid=1' 2024-04-06T09:38:28.463 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:38:28.463 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:38:28.463 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:38:28.463 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:38:28.463 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:38:28.463 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:38:28.463 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:38:28.463 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:38:28.463 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:38:28.463 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:38:28.463 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:38:28.463 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:38:28.464 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:38:28.464 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:38:28.464 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:38:28.510 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 16 2024-04-06T09:38:28.511 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 19 2024-04-06T09:38:28.511 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:38:28.511 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=77, tag_tid=12, entry_tid=1], [object_number=76, tag_tid=12, entry_tid=0], [object_number=79, tag_tid=11, entry_tid=17175], [object_number=78, tag_tid=11, entry_tid=17174]]], state=connected] 2024-04-06T09:38:28.511 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=65, tag_tid=11, entry_tid=7445], [object_number=64, tag_tid=11, entry_tid=7444], [object_number=67, tag_tid=11, entry_tid=7443], [object_number=66, tag_tid=11, entry_tid=7442]]], state=connected] 2024-04-06T09:38:28.514 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:38:28.516 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=65, tag_tid=11, entry_tid=7445' 2024-04-06T09:38:28.516 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=77, tag_tid=12, entry_tid=1' -a 'object_number=77, tag_tid=12, entry_tid=1' = 'object_number=65, tag_tid=11, entry_tid=7445' 2024-04-06T09:38:28.516 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=65, tag_tid=11, entry_tid=7445' '!=' 'object_number=67, tag_tid=11, entry_tid=7363' 2024-04-06T09:38:28.517 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:38:28.517 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=65, tag_tid=11, entry_tid=7445' = 'object_number=67, tag_tid=11, entry_tid=7363' 2024-04-06T09:38:28.517 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=65, tag_tid=11, entry_tid=7445' 2024-04-06T09:38:28.517 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=77, tag_tid=12, entry_tid=1' 2024-04-06T09:38:28.517 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:38:28.517 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:38:28.517 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=12 2024-04-06T09:38:28.518 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=65, tag_tid=11, entry_tid=7445' 2024-04-06T09:38:28.518 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:38:28.518 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:38:28.519 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=11 2024-04-06T09:38:28.520 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=77, tag_tid=12, entry_tid=1' 2024-04-06T09:38:28.520 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:38:28.520 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:38:28.521 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:38:28.522 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=65, tag_tid=11, entry_tid=7445' 2024-04-06T09:38:28.522 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:38:28.522 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:38:28.523 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=7445 2024-04-06T09:38:28.523 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 12 = 11 -a 1 -le 7445 2024-04-06T09:38:28.523 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:38:28.523 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:38:28.523 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:38:28.726 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:38:28.726 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:38:28.726 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:38:28.727 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:38:28.727 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:38:28.727 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:38:28.727 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:38:28.727 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:38:28.727 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:38:28.728 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:38:28.728 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:38:28.728 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:38:28.728 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:38:28.728 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:38:28.728 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:38:28.735 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:38:28.735 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:38:28.735 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:38:28.735 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:38:28.736 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:38:28.736 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:38:28.736 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:38:28.736 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:38:28.736 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:38:28.736 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:38:28.737 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:38:28.737 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:38:28.737 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:38:28.737 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:38:28.737 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:38:28.737 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:38:28.738 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:38:28.738 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:38:28.738 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:38:28.738 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:38:28.738 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:38:28.738 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:38:28.739 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:38:28.739 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:38:28.739 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:38:28.739 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:38:28.739 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:38:28.739 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:38:28.740 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:38:28.740 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:38:28.740 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:38:28.740 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:38:28.740 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:38:28.740 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:38:28.755 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:38:28.755 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:38:28.755 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:38:28.755 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:38:28.889 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:38:28.890 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:38:28.890 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:38:28.890 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:38:28.890 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:38:28.890 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:38:28.890 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:38:28.890 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:38:28.890 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:38:28.890 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:38:28.891 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:38:28.891 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:38:28.891 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:38:28.893 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:38:28.893 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:38:28.893 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:38:28.942 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 16 2024-04-06T09:38:28.942 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 19 2024-04-06T09:38:28.942 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:38:28.942 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=77, tag_tid=12, entry_tid=1], [object_number=76, tag_tid=12, entry_tid=0], [object_number=79, tag_tid=11, entry_tid=17175], [object_number=78, tag_tid=11, entry_tid=17174]]], state=connected] 2024-04-06T09:38:28.942 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=67, tag_tid=11, entry_tid=7539], [object_number=66, tag_tid=11, entry_tid=7538], [object_number=65, tag_tid=11, entry_tid=7537], [object_number=64, tag_tid=11, entry_tid=7536]]], state=connected] 2024-04-06T09:38:28.945 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:38:28.946 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=77, tag_tid=12, entry_tid=1' 2024-04-06T09:38:28.947 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:38:28.948 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:38:28.948 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:38:28.948 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:38:28.948 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:38:28.948 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:38:28.948 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:38:28.948 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:38:28.948 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:38:28.948 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:38:28.948 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:38:28.948 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:38:28.949 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:38:28.949 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:38:28.949 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:38:28.995 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 16 2024-04-06T09:38:28.995 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 19 2024-04-06T09:38:28.995 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:38:28.995 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=77, tag_tid=12, entry_tid=1], [object_number=76, tag_tid=12, entry_tid=0], [object_number=79, tag_tid=11, entry_tid=17175], [object_number=78, tag_tid=11, entry_tid=17174]]], state=connected] 2024-04-06T09:38:28.995 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=67, tag_tid=11, entry_tid=7539], [object_number=66, tag_tid=11, entry_tid=7538], [object_number=65, tag_tid=11, entry_tid=7537], [object_number=64, tag_tid=11, entry_tid=7536]]], state=connected] 2024-04-06T09:38:28.998 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:38:29.000 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=67, tag_tid=11, entry_tid=7539' 2024-04-06T09:38:29.000 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=77, tag_tid=12, entry_tid=1' -a 'object_number=77, tag_tid=12, entry_tid=1' = 'object_number=67, tag_tid=11, entry_tid=7539' 2024-04-06T09:38:29.000 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=67, tag_tid=11, entry_tid=7539' '!=' 'object_number=65, tag_tid=11, entry_tid=7445' 2024-04-06T09:38:29.000 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:38:29.000 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=67, tag_tid=11, entry_tid=7539' = 'object_number=65, tag_tid=11, entry_tid=7445' 2024-04-06T09:38:29.001 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=67, tag_tid=11, entry_tid=7539' 2024-04-06T09:38:29.001 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=77, tag_tid=12, entry_tid=1' 2024-04-06T09:38:29.001 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:38:29.001 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:38:29.002 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=12 2024-04-06T09:38:29.002 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=67, tag_tid=11, entry_tid=7539' 2024-04-06T09:38:29.003 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:38:29.003 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:38:29.003 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=11 2024-04-06T09:38:29.004 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=77, tag_tid=12, entry_tid=1' 2024-04-06T09:38:29.004 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:38:29.004 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:38:29.005 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:38:29.006 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=67, tag_tid=11, entry_tid=7539' 2024-04-06T09:38:29.006 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:38:29.006 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:38:29.007 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=7539 2024-04-06T09:38:29.007 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 12 = 11 -a 1 -le 7539 2024-04-06T09:38:29.007 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:38:29.008 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:38:29.008 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:38:29.209 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:38:29.209 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:38:29.209 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:38:29.209 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:38:29.210 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:38:29.210 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:38:29.210 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:38:29.210 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:38:29.210 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:38:29.210 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:38:29.210 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:38:29.210 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:38:29.210 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:38:29.210 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:38:29.210 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:38:29.211 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:38:29.211 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:38:29.213 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:38:29.214 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:38:29.214 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:38:29.214 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:38:29.214 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:38:29.214 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:38:29.214 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:38:29.214 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:38:29.214 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:38:29.214 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:38:29.214 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:38:29.214 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:38:29.214 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:38:29.214 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:38:29.215 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:38:29.215 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:38:29.215 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:38:29.215 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:38:29.216 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:38:29.216 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:38:29.216 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:38:29.216 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:38:29.216 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:38:29.216 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:38:29.216 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:38:29.216 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:38:29.216 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:38:29.216 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:38:29.216 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:38:29.216 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:38:29.216 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:38:29.216 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:38:29.241 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:38:29.242 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:38:29.242 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:38:29.242 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:38:29.371 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:38:29.372 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:38:29.372 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:38:29.372 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:38:29.372 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:38:29.373 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:38:29.373 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:38:29.373 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:38:29.373 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:38:29.373 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:38:29.373 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:38:29.373 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:38:29.373 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:38:29.375 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:38:29.375 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:38:29.376 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:38:29.422 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 16 2024-04-06T09:38:29.422 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 19 2024-04-06T09:38:29.422 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:38:29.422 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=77, tag_tid=12, entry_tid=1], [object_number=76, tag_tid=12, entry_tid=0], [object_number=79, tag_tid=11, entry_tid=17175], [object_number=78, tag_tid=11, entry_tid=17174]]], state=connected] 2024-04-06T09:38:29.422 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=66, tag_tid=11, entry_tid=7638], [object_number=65, tag_tid=11, entry_tid=7637], [object_number=64, tag_tid=11, entry_tid=7636], [object_number=67, tag_tid=11, entry_tid=7635]]], state=connected] 2024-04-06T09:38:29.426 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:38:29.427 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=77, tag_tid=12, entry_tid=1' 2024-04-06T09:38:29.428 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:38:29.428 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:38:29.428 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:38:29.428 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:38:29.428 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:38:29.428 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:38:29.428 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:38:29.429 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:38:29.429 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:38:29.429 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:38:29.429 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:38:29.429 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:38:29.429 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:38:29.430 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:38:29.430 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:38:29.475 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 16 2024-04-06T09:38:29.475 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 19 2024-04-06T09:38:29.476 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:38:29.476 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=77, tag_tid=12, entry_tid=1], [object_number=76, tag_tid=12, entry_tid=0], [object_number=79, tag_tid=11, entry_tid=17175], [object_number=78, tag_tid=11, entry_tid=17174]]], state=connected] 2024-04-06T09:38:29.476 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=66, tag_tid=11, entry_tid=7638], [object_number=65, tag_tid=11, entry_tid=7637], [object_number=64, tag_tid=11, entry_tid=7636], [object_number=67, tag_tid=11, entry_tid=7635]]], state=connected] 2024-04-06T09:38:29.479 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:38:29.481 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=66, tag_tid=11, entry_tid=7638' 2024-04-06T09:38:29.481 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=77, tag_tid=12, entry_tid=1' -a 'object_number=77, tag_tid=12, entry_tid=1' = 'object_number=66, tag_tid=11, entry_tid=7638' 2024-04-06T09:38:29.481 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=66, tag_tid=11, entry_tid=7638' '!=' 'object_number=67, tag_tid=11, entry_tid=7539' 2024-04-06T09:38:29.481 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:38:29.481 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=66, tag_tid=11, entry_tid=7638' = 'object_number=67, tag_tid=11, entry_tid=7539' 2024-04-06T09:38:29.481 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=66, tag_tid=11, entry_tid=7638' 2024-04-06T09:38:29.481 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=77, tag_tid=12, entry_tid=1' 2024-04-06T09:38:29.482 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:38:29.482 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:38:29.482 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=12 2024-04-06T09:38:29.483 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=66, tag_tid=11, entry_tid=7638' 2024-04-06T09:38:29.483 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:38:29.483 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:38:29.484 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=11 2024-04-06T09:38:29.484 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=77, tag_tid=12, entry_tid=1' 2024-04-06T09:38:29.484 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:38:29.485 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:38:29.485 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:38:29.486 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=66, tag_tid=11, entry_tid=7638' 2024-04-06T09:38:29.486 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:38:29.486 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:38:29.487 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=7638 2024-04-06T09:38:29.487 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 12 = 11 -a 1 -le 7638 2024-04-06T09:38:29.488 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:38:29.488 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:38:29.488 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:38:29.689 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:38:29.690 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:38:29.690 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:38:29.690 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:38:29.690 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:38:29.690 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:38:29.690 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:38:29.690 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:38:29.690 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:38:29.690 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:38:29.690 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:38:29.690 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:38:29.690 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:38:29.690 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:38:29.691 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:38:29.691 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:38:29.691 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:38:29.695 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:38:29.695 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:38:29.695 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:38:29.695 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:38:29.695 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:38:29.695 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:38:29.696 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:38:29.696 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:38:29.696 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:38:29.696 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:38:29.696 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:38:29.696 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:38:29.696 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:38:29.696 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:38:29.696 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:38:29.696 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:38:29.696 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:38:29.696 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:38:29.696 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:38:29.696 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:38:29.696 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:38:29.697 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:38:29.697 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:38:29.697 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:38:29.697 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:38:29.697 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:38:29.697 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:38:29.697 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:38:29.697 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:38:29.697 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:38:29.697 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:38:29.697 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:38:29.720 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:38:29.721 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:38:29.721 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:38:29.721 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:38:29.853 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:38:29.854 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:38:29.854 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:38:29.854 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:38:29.854 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:38:29.855 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:38:29.855 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:38:29.855 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:38:29.855 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:38:29.855 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:38:29.855 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:38:29.856 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:38:29.856 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:38:29.857 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:38:29.857 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:38:29.857 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:38:29.903 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 16 2024-04-06T09:38:29.903 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 19 2024-04-06T09:38:29.903 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:38:29.903 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=77, tag_tid=12, entry_tid=1], [object_number=76, tag_tid=12, entry_tid=0], [object_number=79, tag_tid=11, entry_tid=17175], [object_number=78, tag_tid=11, entry_tid=17174]]], state=connected] 2024-04-06T09:38:29.903 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=64, tag_tid=11, entry_tid=7728], [object_number=67, tag_tid=11, entry_tid=7727], [object_number=66, tag_tid=11, entry_tid=7726], [object_number=65, tag_tid=11, entry_tid=7725]]], state=connected] 2024-04-06T09:38:29.907 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:38:29.908 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=77, tag_tid=12, entry_tid=1' 2024-04-06T09:38:29.909 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:38:29.909 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:38:29.909 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:38:29.909 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:38:29.909 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:38:29.909 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:38:29.909 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:38:29.909 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:38:29.909 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:38:29.909 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:38:29.910 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:38:29.911 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:38:29.911 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:38:29.911 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:38:29.911 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:38:29.956 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 16 2024-04-06T09:38:29.956 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 19 2024-04-06T09:38:29.956 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:38:29.956 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=77, tag_tid=12, entry_tid=1], [object_number=76, tag_tid=12, entry_tid=0], [object_number=79, tag_tid=11, entry_tid=17175], [object_number=78, tag_tid=11, entry_tid=17174]]], state=connected] 2024-04-06T09:38:29.956 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=64, tag_tid=11, entry_tid=7728], [object_number=67, tag_tid=11, entry_tid=7727], [object_number=66, tag_tid=11, entry_tid=7726], [object_number=65, tag_tid=11, entry_tid=7725]]], state=connected] 2024-04-06T09:38:29.959 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:38:29.962 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=64, tag_tid=11, entry_tid=7728' 2024-04-06T09:38:29.962 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=77, tag_tid=12, entry_tid=1' -a 'object_number=77, tag_tid=12, entry_tid=1' = 'object_number=64, tag_tid=11, entry_tid=7728' 2024-04-06T09:38:29.962 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=64, tag_tid=11, entry_tid=7728' '!=' 'object_number=66, tag_tid=11, entry_tid=7638' 2024-04-06T09:38:29.962 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:38:29.962 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=64, tag_tid=11, entry_tid=7728' = 'object_number=66, tag_tid=11, entry_tid=7638' 2024-04-06T09:38:29.962 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=64, tag_tid=11, entry_tid=7728' 2024-04-06T09:38:29.962 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=77, tag_tid=12, entry_tid=1' 2024-04-06T09:38:29.962 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:38:29.962 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:38:29.963 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=12 2024-04-06T09:38:29.963 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=64, tag_tid=11, entry_tid=7728' 2024-04-06T09:38:29.963 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:38:29.964 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:38:29.964 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=11 2024-04-06T09:38:29.965 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=77, tag_tid=12, entry_tid=1' 2024-04-06T09:38:29.965 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:38:29.965 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:38:29.966 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:38:29.967 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=64, tag_tid=11, entry_tid=7728' 2024-04-06T09:38:29.967 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:38:29.967 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:38:29.968 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=7728 2024-04-06T09:38:29.968 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 12 = 11 -a 1 -le 7728 2024-04-06T09:38:29.968 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:38:29.968 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:38:29.968 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:38:30.170 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:38:30.170 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:38:30.170 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:38:30.170 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:38:30.170 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:38:30.170 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:38:30.171 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:38:30.171 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:38:30.171 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:38:30.171 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:38:30.171 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:38:30.171 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:38:30.171 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:38:30.171 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:38:30.171 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:38:30.171 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:38:30.172 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:38:30.174 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:38:30.174 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:38:30.174 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:38:30.174 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:38:30.175 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:38:30.175 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:38:30.175 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:38:30.175 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:38:30.175 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:38:30.175 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:38:30.175 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:38:30.175 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:38:30.175 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:38:30.175 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:38:30.175 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:38:30.175 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:38:30.176 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:38:30.176 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:38:30.176 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:38:30.176 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:38:30.176 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:38:30.176 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:38:30.177 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:38:30.177 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:38:30.177 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:38:30.177 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:38:30.177 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:38:30.177 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:38:30.177 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:38:30.177 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:38:30.177 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:38:30.177 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:38:30.200 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:38:30.201 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:38:30.201 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:38:30.201 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:38:30.331 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:38:30.332 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:38:30.332 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:38:30.332 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:38:30.333 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:38:30.333 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:38:30.333 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:38:30.333 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:38:30.333 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:38:30.333 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:38:30.334 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:38:30.334 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:38:30.334 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:38:30.336 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:38:30.337 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:38:30.337 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:38:30.389 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 16 2024-04-06T09:38:30.389 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 19 2024-04-06T09:38:30.389 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:38:30.389 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=77, tag_tid=12, entry_tid=1], [object_number=76, tag_tid=12, entry_tid=0], [object_number=79, tag_tid=11, entry_tid=17175], [object_number=78, tag_tid=11, entry_tid=17174]]], state=connected] 2024-04-06T09:38:30.389 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=64, tag_tid=11, entry_tid=7816], [object_number=67, tag_tid=11, entry_tid=7815], [object_number=66, tag_tid=11, entry_tid=7814], [object_number=65, tag_tid=11, entry_tid=7813]]], state=connected] 2024-04-06T09:38:30.391 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:38:30.394 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=77, tag_tid=12, entry_tid=1' 2024-04-06T09:38:30.395 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:38:30.395 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:38:30.395 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:38:30.395 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:38:30.395 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:38:30.395 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:38:30.395 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:38:30.395 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:38:30.395 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:38:30.396 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:38:30.396 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:38:30.396 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:38:30.396 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:38:30.396 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:38:30.397 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:38:30.443 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 16 2024-04-06T09:38:30.443 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 19 2024-04-06T09:38:30.443 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:38:30.443 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=77, tag_tid=12, entry_tid=1], [object_number=76, tag_tid=12, entry_tid=0], [object_number=79, tag_tid=11, entry_tid=17175], [object_number=78, tag_tid=11, entry_tid=17174]]], state=connected] 2024-04-06T09:38:30.443 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=64, tag_tid=11, entry_tid=7816], [object_number=67, tag_tid=11, entry_tid=7815], [object_number=66, tag_tid=11, entry_tid=7814], [object_number=65, tag_tid=11, entry_tid=7813]]], state=connected] 2024-04-06T09:38:30.446 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:38:30.448 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=64, tag_tid=11, entry_tid=7816' 2024-04-06T09:38:30.448 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=77, tag_tid=12, entry_tid=1' -a 'object_number=77, tag_tid=12, entry_tid=1' = 'object_number=64, tag_tid=11, entry_tid=7816' 2024-04-06T09:38:30.448 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=64, tag_tid=11, entry_tid=7816' '!=' 'object_number=64, tag_tid=11, entry_tid=7728' 2024-04-06T09:38:30.448 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:38:30.448 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=64, tag_tid=11, entry_tid=7816' = 'object_number=64, tag_tid=11, entry_tid=7728' 2024-04-06T09:38:30.449 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=64, tag_tid=11, entry_tid=7816' 2024-04-06T09:38:30.449 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=77, tag_tid=12, entry_tid=1' 2024-04-06T09:38:30.449 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:38:30.449 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:38:30.450 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=12 2024-04-06T09:38:30.450 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=64, tag_tid=11, entry_tid=7816' 2024-04-06T09:38:30.450 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:38:30.451 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:38:30.451 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=11 2024-04-06T09:38:30.452 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=77, tag_tid=12, entry_tid=1' 2024-04-06T09:38:30.452 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:38:30.452 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:38:30.453 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:38:30.454 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=64, tag_tid=11, entry_tid=7816' 2024-04-06T09:38:30.454 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:38:30.454 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:38:30.455 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=7816 2024-04-06T09:38:30.455 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 12 = 11 -a 1 -le 7816 2024-04-06T09:38:30.455 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:38:30.455 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:38:30.455 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:38:30.657 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:38:30.657 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:38:30.657 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:38:30.658 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:38:30.658 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:38:30.658 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:38:30.658 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:38:30.658 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:38:30.658 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:38:30.658 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:38:30.658 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:38:30.658 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:38:30.658 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:38:30.658 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:38:30.658 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:38:30.659 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:38:30.660 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:38:30.662 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:38:30.662 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:38:30.662 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:38:30.662 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:38:30.662 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:38:30.662 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:38:30.662 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:38:30.662 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:38:30.662 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:38:30.662 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:38:30.662 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:38:30.663 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:38:30.663 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:38:30.663 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:38:30.663 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:38:30.663 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:38:30.664 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:38:30.664 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:38:30.664 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:38:30.664 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:38:30.664 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:38:30.664 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:38:30.664 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:38:30.664 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:38:30.665 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:38:30.665 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:38:30.665 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:38:30.665 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:38:30.665 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:38:30.665 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:38:30.665 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:38:30.665 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:38:30.689 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:38:30.690 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:38:30.690 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:38:30.690 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:38:30.821 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:38:30.822 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:38:30.822 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:38:30.822 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:38:30.823 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:38:30.823 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:38:30.823 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:38:30.823 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:38:30.823 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:38:30.823 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:38:30.824 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:38:30.824 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:38:30.824 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:38:30.825 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:38:30.825 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:38:30.826 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:38:30.870 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 16 2024-04-06T09:38:30.870 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 19 2024-04-06T09:38:30.871 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:38:30.871 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=77, tag_tid=12, entry_tid=1], [object_number=76, tag_tid=12, entry_tid=0], [object_number=79, tag_tid=11, entry_tid=17175], [object_number=78, tag_tid=11, entry_tid=17174]]], state=connected] 2024-04-06T09:38:30.871 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=67, tag_tid=11, entry_tid=7911], [object_number=66, tag_tid=11, entry_tid=7910], [object_number=65, tag_tid=11, entry_tid=7909], [object_number=64, tag_tid=11, entry_tid=7908]]], state=connected] 2024-04-06T09:38:30.874 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:38:30.875 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=77, tag_tid=12, entry_tid=1' 2024-04-06T09:38:30.876 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:38:30.877 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:38:30.877 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:38:30.877 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:38:30.877 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:38:30.877 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:38:30.877 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:38:30.877 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:38:30.877 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:38:30.877 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:38:30.878 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:38:30.878 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:38:30.878 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:38:30.879 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:38:30.879 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:38:30.922 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 16 2024-04-06T09:38:30.923 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 19 2024-04-06T09:38:30.923 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:38:30.923 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=77, tag_tid=12, entry_tid=1], [object_number=76, tag_tid=12, entry_tid=0], [object_number=79, tag_tid=11, entry_tid=17175], [object_number=78, tag_tid=11, entry_tid=17174]]], state=connected] 2024-04-06T09:38:30.923 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=67, tag_tid=11, entry_tid=7911], [object_number=66, tag_tid=11, entry_tid=7910], [object_number=65, tag_tid=11, entry_tid=7909], [object_number=64, tag_tid=11, entry_tid=7908]]], state=connected] 2024-04-06T09:38:30.926 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:38:30.928 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=67, tag_tid=11, entry_tid=7911' 2024-04-06T09:38:30.929 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=77, tag_tid=12, entry_tid=1' -a 'object_number=77, tag_tid=12, entry_tid=1' = 'object_number=67, tag_tid=11, entry_tid=7911' 2024-04-06T09:38:30.929 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=67, tag_tid=11, entry_tid=7911' '!=' 'object_number=64, tag_tid=11, entry_tid=7816' 2024-04-06T09:38:30.929 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:38:30.929 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=67, tag_tid=11, entry_tid=7911' = 'object_number=64, tag_tid=11, entry_tid=7816' 2024-04-06T09:38:30.929 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=67, tag_tid=11, entry_tid=7911' 2024-04-06T09:38:30.929 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=77, tag_tid=12, entry_tid=1' 2024-04-06T09:38:30.929 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:38:30.929 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:38:30.930 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=12 2024-04-06T09:38:30.930 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=67, tag_tid=11, entry_tid=7911' 2024-04-06T09:38:30.931 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:38:30.931 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:38:30.932 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=11 2024-04-06T09:38:30.932 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=77, tag_tid=12, entry_tid=1' 2024-04-06T09:38:30.932 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:38:30.932 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:38:30.933 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:38:30.934 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=67, tag_tid=11, entry_tid=7911' 2024-04-06T09:38:30.934 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:38:30.934 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:38:30.935 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=7911 2024-04-06T09:38:30.935 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 12 = 11 -a 1 -le 7911 2024-04-06T09:38:30.935 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:38:30.935 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:38:30.936 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:38:31.137 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:38:31.137 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:38:31.137 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:38:31.137 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:38:31.137 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:38:31.138 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:38:31.138 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:38:31.138 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:38:31.138 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:38:31.138 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:38:31.138 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:38:31.138 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:38:31.138 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:38:31.138 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:38:31.138 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:38:31.139 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:38:31.139 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:38:31.142 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:38:31.142 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:38:31.142 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:38:31.142 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:38:31.142 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:38:31.142 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:38:31.142 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:38:31.142 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:38:31.142 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:38:31.142 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:38:31.142 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:38:31.143 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:38:31.143 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:38:31.143 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:38:31.143 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:38:31.143 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:38:31.144 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:38:31.144 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:38:31.144 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:38:31.144 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:38:31.144 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:38:31.144 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:38:31.144 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:38:31.144 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:38:31.144 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:38:31.144 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:38:31.144 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:38:31.145 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:38:31.145 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:38:31.145 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:38:31.145 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:38:31.145 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:38:31.168 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:38:31.168 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:38:31.169 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:38:31.169 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:38:31.298 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:38:31.300 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:38:31.300 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:38:31.300 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:38:31.300 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:38:31.300 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:38:31.300 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:38:31.300 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:38:31.300 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:38:31.300 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:38:31.301 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:38:31.301 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:38:31.301 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:38:31.302 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:38:31.303 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:38:31.303 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:38:31.358 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 16 2024-04-06T09:38:31.358 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 19 2024-04-06T09:38:31.358 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:38:31.359 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=77, tag_tid=12, entry_tid=1], [object_number=76, tag_tid=12, entry_tid=0], [object_number=79, tag_tid=11, entry_tid=17175], [object_number=78, tag_tid=11, entry_tid=17174]]], state=connected] 2024-04-06T09:38:31.359 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=67, tag_tid=11, entry_tid=7999], [object_number=66, tag_tid=11, entry_tid=7998], [object_number=65, tag_tid=11, entry_tid=7997], [object_number=64, tag_tid=11, entry_tid=7996]]], state=connected] 2024-04-06T09:38:31.362 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:38:31.363 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=77, tag_tid=12, entry_tid=1' 2024-04-06T09:38:31.365 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:38:31.365 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:38:31.365 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:38:31.365 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:38:31.365 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:38:31.365 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:38:31.365 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:38:31.365 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:38:31.366 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:38:31.366 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:38:31.366 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:38:31.366 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:38:31.366 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:38:31.367 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:38:31.367 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:38:31.412 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 16 2024-04-06T09:38:31.412 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 19 2024-04-06T09:38:31.412 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:38:31.412 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=77, tag_tid=12, entry_tid=1], [object_number=76, tag_tid=12, entry_tid=0], [object_number=79, tag_tid=11, entry_tid=17175], [object_number=78, tag_tid=11, entry_tid=17174]]], state=connected] 2024-04-06T09:38:31.413 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=67, tag_tid=11, entry_tid=7999], [object_number=66, tag_tid=11, entry_tid=7998], [object_number=65, tag_tid=11, entry_tid=7997], [object_number=64, tag_tid=11, entry_tid=7996]]], state=connected] 2024-04-06T09:38:31.416 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:38:31.418 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=67, tag_tid=11, entry_tid=7999' 2024-04-06T09:38:31.418 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=77, tag_tid=12, entry_tid=1' -a 'object_number=77, tag_tid=12, entry_tid=1' = 'object_number=67, tag_tid=11, entry_tid=7999' 2024-04-06T09:38:31.418 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=67, tag_tid=11, entry_tid=7999' '!=' 'object_number=67, tag_tid=11, entry_tid=7911' 2024-04-06T09:38:31.418 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:38:31.419 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=67, tag_tid=11, entry_tid=7999' = 'object_number=67, tag_tid=11, entry_tid=7911' 2024-04-06T09:38:31.419 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=67, tag_tid=11, entry_tid=7999' 2024-04-06T09:38:31.419 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=77, tag_tid=12, entry_tid=1' 2024-04-06T09:38:31.419 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:38:31.419 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:38:31.420 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=12 2024-04-06T09:38:31.420 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=67, tag_tid=11, entry_tid=7999' 2024-04-06T09:38:31.420 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:38:31.420 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:38:31.421 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=11 2024-04-06T09:38:31.422 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=77, tag_tid=12, entry_tid=1' 2024-04-06T09:38:31.422 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:38:31.422 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:38:31.423 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:38:31.423 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=67, tag_tid=11, entry_tid=7999' 2024-04-06T09:38:31.423 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:38:31.423 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:38:31.424 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=7999 2024-04-06T09:38:31.425 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 12 = 11 -a 1 -le 7999 2024-04-06T09:38:31.425 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:38:31.425 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:38:31.425 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:38:31.627 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:38:31.627 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:38:31.627 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:38:31.627 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:38:31.627 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:38:31.627 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:38:31.628 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:38:31.628 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:38:31.628 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:38:31.628 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:38:31.628 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:38:31.628 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:38:31.628 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:38:31.628 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:38:31.628 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:38:31.629 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:38:31.629 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:38:31.631 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:38:31.631 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:38:31.631 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:38:31.631 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:38:31.631 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:38:31.631 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:38:31.631 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:38:31.631 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:38:31.631 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:38:31.631 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:38:31.632 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:38:31.632 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:38:31.632 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:38:31.632 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:38:31.632 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:38:31.632 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:38:31.633 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:38:31.634 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:38:31.634 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:38:31.634 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:38:31.634 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:38:31.634 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:38:31.634 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:38:31.634 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:38:31.634 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:38:31.634 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:38:31.634 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:38:31.634 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:38:31.635 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:38:31.635 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:38:31.635 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:38:31.635 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:38:31.658 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:38:31.658 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:38:31.659 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:38:31.659 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:38:31.790 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:38:31.791 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:38:31.791 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:38:31.791 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:38:31.792 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:38:31.792 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:38:31.792 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:38:31.792 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:38:31.792 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:38:31.792 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:38:31.792 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:38:31.793 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:38:31.793 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:38:31.794 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:38:31.794 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:38:31.794 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:38:31.845 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 16 2024-04-06T09:38:31.845 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 19 2024-04-06T09:38:31.845 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:38:31.845 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=77, tag_tid=12, entry_tid=1], [object_number=76, tag_tid=12, entry_tid=0], [object_number=79, tag_tid=11, entry_tid=17175], [object_number=78, tag_tid=11, entry_tid=17174]]], state=connected] 2024-04-06T09:38:31.845 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=67, tag_tid=11, entry_tid=8087], [object_number=66, tag_tid=11, entry_tid=8086], [object_number=65, tag_tid=11, entry_tid=8085], [object_number=64, tag_tid=11, entry_tid=8084]]], state=connected] 2024-04-06T09:38:31.848 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:38:31.850 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=77, tag_tid=12, entry_tid=1' 2024-04-06T09:38:31.851 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:38:31.851 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:38:31.851 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:38:31.851 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:38:31.851 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:38:31.851 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:38:31.852 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:38:31.852 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:38:31.852 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:38:31.852 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:38:31.852 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:38:31.852 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:38:31.852 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:38:31.853 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:38:31.853 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:38:31.898 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 16 2024-04-06T09:38:31.898 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 19 2024-04-06T09:38:31.899 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:38:31.899 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=77, tag_tid=12, entry_tid=1], [object_number=76, tag_tid=12, entry_tid=0], [object_number=79, tag_tid=11, entry_tid=17175], [object_number=78, tag_tid=11, entry_tid=17174]]], state=connected] 2024-04-06T09:38:31.899 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=67, tag_tid=11, entry_tid=8087], [object_number=66, tag_tid=11, entry_tid=8086], [object_number=65, tag_tid=11, entry_tid=8085], [object_number=64, tag_tid=11, entry_tid=8084]]], state=connected] 2024-04-06T09:38:31.902 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:38:31.904 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=67, tag_tid=11, entry_tid=8087' 2024-04-06T09:38:31.904 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=77, tag_tid=12, entry_tid=1' -a 'object_number=77, tag_tid=12, entry_tid=1' = 'object_number=67, tag_tid=11, entry_tid=8087' 2024-04-06T09:38:31.904 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=67, tag_tid=11, entry_tid=8087' '!=' 'object_number=67, tag_tid=11, entry_tid=7999' 2024-04-06T09:38:31.904 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:38:31.904 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=67, tag_tid=11, entry_tid=8087' = 'object_number=67, tag_tid=11, entry_tid=7999' 2024-04-06T09:38:31.904 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=67, tag_tid=11, entry_tid=8087' 2024-04-06T09:38:31.904 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=77, tag_tid=12, entry_tid=1' 2024-04-06T09:38:31.904 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:38:31.904 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:38:31.905 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=12 2024-04-06T09:38:31.906 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=67, tag_tid=11, entry_tid=8087' 2024-04-06T09:38:31.906 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:38:31.906 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:38:31.907 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=11 2024-04-06T09:38:31.907 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=77, tag_tid=12, entry_tid=1' 2024-04-06T09:38:31.907 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:38:31.908 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:38:31.908 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:38:31.909 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=67, tag_tid=11, entry_tid=8087' 2024-04-06T09:38:31.909 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:38:31.909 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:38:31.910 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=8087 2024-04-06T09:38:31.910 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 12 = 11 -a 1 -le 8087 2024-04-06T09:38:31.911 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:38:31.911 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:38:31.911 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:38:32.112 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:38:32.112 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:38:32.113 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:38:32.113 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:38:32.113 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:38:32.113 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:38:32.113 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:38:32.113 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:38:32.113 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:38:32.113 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:38:32.113 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:38:32.113 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:38:32.113 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:38:32.113 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:38:32.113 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:38:32.114 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:38:32.114 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:38:32.117 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:38:32.117 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:38:32.117 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:38:32.117 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:38:32.117 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:38:32.117 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:38:32.117 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:38:32.117 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:38:32.117 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:38:32.117 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:38:32.117 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:38:32.118 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:38:32.118 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:38:32.118 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:38:32.118 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:38:32.118 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:38:32.119 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:38:32.119 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:38:32.119 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:38:32.119 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:38:32.119 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:38:32.119 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:38:32.120 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:38:32.120 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:38:32.120 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:38:32.120 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:38:32.120 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:38:32.120 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:38:32.120 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:38:32.120 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:38:32.120 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:38:32.120 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:38:32.144 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:38:32.145 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:38:32.145 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:38:32.145 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:38:32.276 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:38:32.277 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:38:32.278 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:38:32.278 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:38:32.278 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:38:32.278 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:38:32.278 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:38:32.278 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:38:32.278 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:38:32.278 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:38:32.279 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:38:32.279 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:38:32.279 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:38:32.281 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:38:32.281 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:38:32.281 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:38:32.333 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 16 2024-04-06T09:38:32.333 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 19 2024-04-06T09:38:32.333 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:38:32.333 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=77, tag_tid=12, entry_tid=1], [object_number=76, tag_tid=12, entry_tid=0], [object_number=79, tag_tid=11, entry_tid=17175], [object_number=78, tag_tid=11, entry_tid=17174]]], state=connected] 2024-04-06T09:38:32.333 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=64, tag_tid=11, entry_tid=8172], [object_number=67, tag_tid=11, entry_tid=8171], [object_number=66, tag_tid=11, entry_tid=8170], [object_number=65, tag_tid=11, entry_tid=8169]]], state=connected] 2024-04-06T09:38:32.337 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:38:32.338 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=77, tag_tid=12, entry_tid=1' 2024-04-06T09:38:32.339 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:38:32.339 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:38:32.339 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:38:32.339 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:38:32.340 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:38:32.340 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:38:32.340 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:38:32.340 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:38:32.340 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:38:32.340 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:38:32.341 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:38:32.341 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:38:32.342 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:38:32.342 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:38:32.342 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:38:32.388 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 16 2024-04-06T09:38:32.388 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 19 2024-04-06T09:38:32.388 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:38:32.388 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=77, tag_tid=12, entry_tid=1], [object_number=76, tag_tid=12, entry_tid=0], [object_number=79, tag_tid=11, entry_tid=17175], [object_number=78, tag_tid=11, entry_tid=17174]]], state=connected] 2024-04-06T09:38:32.388 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=64, tag_tid=11, entry_tid=8172], [object_number=67, tag_tid=11, entry_tid=8171], [object_number=66, tag_tid=11, entry_tid=8170], [object_number=65, tag_tid=11, entry_tid=8169]]], state=connected] 2024-04-06T09:38:32.391 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:38:32.393 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=64, tag_tid=11, entry_tid=8172' 2024-04-06T09:38:32.393 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=77, tag_tid=12, entry_tid=1' -a 'object_number=77, tag_tid=12, entry_tid=1' = 'object_number=64, tag_tid=11, entry_tid=8172' 2024-04-06T09:38:32.393 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=64, tag_tid=11, entry_tid=8172' '!=' 'object_number=67, tag_tid=11, entry_tid=8087' 2024-04-06T09:38:32.393 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:38:32.393 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=64, tag_tid=11, entry_tid=8172' = 'object_number=67, tag_tid=11, entry_tid=8087' 2024-04-06T09:38:32.393 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=64, tag_tid=11, entry_tid=8172' 2024-04-06T09:38:32.394 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=77, tag_tid=12, entry_tid=1' 2024-04-06T09:38:32.394 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:38:32.394 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:38:32.395 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=12 2024-04-06T09:38:32.395 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=64, tag_tid=11, entry_tid=8172' 2024-04-06T09:38:32.395 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:38:32.395 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:38:32.396 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=11 2024-04-06T09:38:32.397 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=77, tag_tid=12, entry_tid=1' 2024-04-06T09:38:32.397 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:38:32.397 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:38:32.398 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:38:32.399 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=64, tag_tid=11, entry_tid=8172' 2024-04-06T09:38:32.399 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:38:32.399 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:38:32.400 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=8172 2024-04-06T09:38:32.400 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 12 = 11 -a 1 -le 8172 2024-04-06T09:38:32.400 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:38:32.400 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:38:32.400 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:38:32.603 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:38:32.603 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:38:32.603 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:38:32.603 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:38:32.603 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:38:32.603 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:38:32.603 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:38:32.604 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:38:32.604 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:38:32.604 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:38:32.604 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:38:32.604 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:38:32.604 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:38:32.604 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:38:32.604 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:38:32.605 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:38:32.605 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:38:32.606 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:38:32.606 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:38:32.606 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:38:32.606 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:38:32.606 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:38:32.607 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:38:32.607 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:38:32.607 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:38:32.607 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:38:32.607 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:38:32.607 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:38:32.607 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:38:32.607 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:38:32.607 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:38:32.607 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:38:32.607 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:38:32.608 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:38:32.608 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:38:32.609 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:38:32.609 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:38:32.609 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:38:32.609 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:38:32.609 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:38:32.609 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:38:32.609 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:38:32.609 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:38:32.609 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:38:32.609 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:38:32.609 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:38:32.609 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:38:32.609 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:38:32.609 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:38:32.633 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:38:32.634 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:38:32.634 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:38:32.634 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:38:32.762 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:38:32.764 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:38:32.764 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:38:32.764 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:38:32.764 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:38:32.764 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:38:32.764 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:38:32.764 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:38:32.765 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:38:32.765 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:38:32.765 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:38:32.765 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:38:32.765 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:38:32.767 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:38:32.768 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:38:32.768 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:38:32.814 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 16 2024-04-06T09:38:32.815 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 19 2024-04-06T09:38:32.815 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:38:32.815 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=77, tag_tid=12, entry_tid=1], [object_number=76, tag_tid=12, entry_tid=0], [object_number=79, tag_tid=11, entry_tid=17175], [object_number=78, tag_tid=11, entry_tid=17174]]], state=connected] 2024-04-06T09:38:32.815 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=67, tag_tid=11, entry_tid=8251], [object_number=66, tag_tid=11, entry_tid=8250], [object_number=65, tag_tid=11, entry_tid=8249], [object_number=64, tag_tid=11, entry_tid=8248]]], state=connected] 2024-04-06T09:38:32.818 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:38:32.821 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=77, tag_tid=12, entry_tid=1' 2024-04-06T09:38:32.821 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:38:32.821 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:38:32.821 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:38:32.821 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:38:32.821 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:38:32.821 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:38:32.821 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:38:32.821 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:38:32.821 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:38:32.821 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:38:32.821 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:38:32.822 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:38:32.822 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:38:32.822 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:38:32.823 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:38:32.868 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 16 2024-04-06T09:38:32.868 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 19 2024-04-06T09:38:32.868 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:38:32.868 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=77, tag_tid=12, entry_tid=1], [object_number=76, tag_tid=12, entry_tid=0], [object_number=79, tag_tid=11, entry_tid=17175], [object_number=78, tag_tid=11, entry_tid=17174]]], state=connected] 2024-04-06T09:38:32.868 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=67, tag_tid=11, entry_tid=8251], [object_number=66, tag_tid=11, entry_tid=8250], [object_number=65, tag_tid=11, entry_tid=8249], [object_number=64, tag_tid=11, entry_tid=8248]]], state=connected] 2024-04-06T09:38:32.872 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:38:32.874 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=67, tag_tid=11, entry_tid=8251' 2024-04-06T09:38:32.874 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=77, tag_tid=12, entry_tid=1' -a 'object_number=77, tag_tid=12, entry_tid=1' = 'object_number=67, tag_tid=11, entry_tid=8251' 2024-04-06T09:38:32.874 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=67, tag_tid=11, entry_tid=8251' '!=' 'object_number=64, tag_tid=11, entry_tid=8172' 2024-04-06T09:38:32.874 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:38:32.875 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=67, tag_tid=11, entry_tid=8251' = 'object_number=64, tag_tid=11, entry_tid=8172' 2024-04-06T09:38:32.875 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=67, tag_tid=11, entry_tid=8251' 2024-04-06T09:38:32.875 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=77, tag_tid=12, entry_tid=1' 2024-04-06T09:38:32.875 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:38:32.875 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:38:32.875 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=12 2024-04-06T09:38:32.876 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=67, tag_tid=11, entry_tid=8251' 2024-04-06T09:38:32.876 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:38:32.876 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:38:32.877 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=11 2024-04-06T09:38:32.877 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=77, tag_tid=12, entry_tid=1' 2024-04-06T09:38:32.878 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:38:32.878 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:38:32.878 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:38:32.879 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=67, tag_tid=11, entry_tid=8251' 2024-04-06T09:38:32.879 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:38:32.879 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:38:32.880 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=8251 2024-04-06T09:38:32.880 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 12 = 11 -a 1 -le 8251 2024-04-06T09:38:32.881 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:38:32.881 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:38:32.881 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:38:33.082 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:38:33.083 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:38:33.083 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:38:33.083 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:38:33.083 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:38:33.083 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:38:33.083 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:38:33.083 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:38:33.084 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:38:33.084 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:38:33.084 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:38:33.084 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:38:33.084 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:38:33.084 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:38:33.084 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:38:33.088 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:38:33.088 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:38:33.088 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:38:33.088 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:38:33.088 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:38:33.088 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:38:33.088 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:38:33.088 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:38:33.089 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:38:33.089 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:38:33.089 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:38:33.089 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:38:33.089 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:38:33.089 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:38:33.089 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:38:33.089 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:38:33.089 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:38:33.089 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:38:33.090 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:38:33.090 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:38:33.090 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:38:33.090 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:38:33.090 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:38:33.090 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:38:33.090 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:38:33.090 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:38:33.090 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:38:33.090 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:38:33.090 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:38:33.090 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:38:33.090 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:38:33.090 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:38:33.090 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:38:33.091 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:38:33.111 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:38:33.112 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:38:33.112 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:38:33.112 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:38:33.236 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:38:33.238 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:38:33.238 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:38:33.238 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:38:33.238 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:38:33.238 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:38:33.238 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:38:33.238 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:38:33.238 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:38:33.238 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:38:33.238 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:38:33.239 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:38:33.239 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:38:33.241 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:38:33.241 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:38:33.241 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:38:33.288 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 16 2024-04-06T09:38:33.288 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 19 2024-04-06T09:38:33.288 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:38:33.288 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=77, tag_tid=12, entry_tid=1], [object_number=76, tag_tid=12, entry_tid=0], [object_number=79, tag_tid=11, entry_tid=17175], [object_number=78, tag_tid=11, entry_tid=17174]]], state=connected] 2024-04-06T09:38:33.289 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=66, tag_tid=11, entry_tid=8338], [object_number=65, tag_tid=11, entry_tid=8337], [object_number=64, tag_tid=11, entry_tid=8336], [object_number=67, tag_tid=11, entry_tid=8335]]], state=connected] 2024-04-06T09:38:33.292 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:38:33.294 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=77, tag_tid=12, entry_tid=1' 2024-04-06T09:38:33.294 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:38:33.294 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:38:33.294 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:38:33.294 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:38:33.294 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:38:33.294 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:38:33.295 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:38:33.295 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:38:33.295 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:38:33.295 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:38:33.296 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:38:33.296 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:38:33.296 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:38:33.296 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:38:33.296 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:38:33.342 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 16 2024-04-06T09:38:33.342 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 19 2024-04-06T09:38:33.343 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:38:33.343 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=77, tag_tid=12, entry_tid=1], [object_number=76, tag_tid=12, entry_tid=0], [object_number=79, tag_tid=11, entry_tid=17175], [object_number=78, tag_tid=11, entry_tid=17174]]], state=connected] 2024-04-06T09:38:33.343 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=66, tag_tid=11, entry_tid=8338], [object_number=65, tag_tid=11, entry_tid=8337], [object_number=64, tag_tid=11, entry_tid=8336], [object_number=67, tag_tid=11, entry_tid=8335]]], state=connected] 2024-04-06T09:38:33.346 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:38:33.348 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=66, tag_tid=11, entry_tid=8338' 2024-04-06T09:38:33.348 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=77, tag_tid=12, entry_tid=1' -a 'object_number=77, tag_tid=12, entry_tid=1' = 'object_number=66, tag_tid=11, entry_tid=8338' 2024-04-06T09:38:33.348 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=66, tag_tid=11, entry_tid=8338' '!=' 'object_number=67, tag_tid=11, entry_tid=8251' 2024-04-06T09:38:33.348 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:38:33.348 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=66, tag_tid=11, entry_tid=8338' = 'object_number=67, tag_tid=11, entry_tid=8251' 2024-04-06T09:38:33.349 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=66, tag_tid=11, entry_tid=8338' 2024-04-06T09:38:33.349 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=77, tag_tid=12, entry_tid=1' 2024-04-06T09:38:33.349 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:38:33.349 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:38:33.349 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=12 2024-04-06T09:38:33.350 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=66, tag_tid=11, entry_tid=8338' 2024-04-06T09:38:33.350 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:38:33.350 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:38:33.351 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=11 2024-04-06T09:38:33.352 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=77, tag_tid=12, entry_tid=1' 2024-04-06T09:38:33.352 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:38:33.352 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:38:33.353 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:38:33.353 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=66, tag_tid=11, entry_tid=8338' 2024-04-06T09:38:33.354 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:38:33.354 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:38:33.355 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=8338 2024-04-06T09:38:33.355 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 12 = 11 -a 1 -le 8338 2024-04-06T09:38:33.355 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:38:33.355 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:38:33.355 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:38:33.556 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:38:33.557 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:38:33.557 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:38:33.557 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:38:33.557 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:38:33.557 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:38:33.557 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:38:33.557 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:38:33.557 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:38:33.557 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:38:33.557 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:38:33.557 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:38:33.557 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:38:33.558 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:38:33.558 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:38:33.558 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:38:33.558 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:38:33.560 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:38:33.560 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:38:33.560 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:38:33.561 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:38:33.561 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:38:33.561 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:38:33.561 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:38:33.561 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:38:33.561 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:38:33.561 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:38:33.561 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:38:33.561 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:38:33.561 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:38:33.561 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:38:33.561 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:38:33.561 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:38:33.562 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:38:33.562 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:38:33.562 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:38:33.562 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:38:33.562 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:38:33.562 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:38:33.563 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:38:33.563 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:38:33.563 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:38:33.563 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:38:33.563 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:38:33.563 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:38:33.563 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:38:33.563 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:38:33.563 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:38:33.563 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:38:33.587 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:38:33.588 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:38:33.588 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:38:33.588 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:38:33.724 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:38:33.726 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:38:33.726 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:38:33.726 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:38:33.726 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:38:33.726 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:38:33.726 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:38:33.726 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:38:33.726 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:38:33.726 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:38:33.727 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:38:33.727 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:38:33.727 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:38:33.729 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:38:33.729 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:38:33.730 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:38:33.781 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 16 2024-04-06T09:38:33.781 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 19 2024-04-06T09:38:33.781 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:38:33.782 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=77, tag_tid=12, entry_tid=1], [object_number=76, tag_tid=12, entry_tid=0], [object_number=79, tag_tid=11, entry_tid=17175], [object_number=78, tag_tid=11, entry_tid=17174]]], state=connected] 2024-04-06T09:38:33.782 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=67, tag_tid=11, entry_tid=8435], [object_number=66, tag_tid=11, entry_tid=8434], [object_number=65, tag_tid=11, entry_tid=8433], [object_number=64, tag_tid=11, entry_tid=8432]]], state=connected] 2024-04-06T09:38:33.785 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:38:33.786 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=77, tag_tid=12, entry_tid=1' 2024-04-06T09:38:33.787 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:38:33.787 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:38:33.787 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:38:33.787 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:38:33.788 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:38:33.788 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:38:33.788 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:38:33.788 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:38:33.788 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:38:33.788 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:38:33.788 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:38:33.788 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:38:33.789 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:38:33.789 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:38:33.789 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:38:33.834 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 16 2024-04-06T09:38:33.834 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 19 2024-04-06T09:38:33.834 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:38:33.834 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=77, tag_tid=12, entry_tid=1], [object_number=76, tag_tid=12, entry_tid=0], [object_number=79, tag_tid=11, entry_tid=17175], [object_number=78, tag_tid=11, entry_tid=17174]]], state=connected] 2024-04-06T09:38:33.834 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=67, tag_tid=11, entry_tid=8435], [object_number=66, tag_tid=11, entry_tid=8434], [object_number=65, tag_tid=11, entry_tid=8433], [object_number=64, tag_tid=11, entry_tid=8432]]], state=connected] 2024-04-06T09:38:33.838 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:38:33.840 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=67, tag_tid=11, entry_tid=8435' 2024-04-06T09:38:33.840 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=77, tag_tid=12, entry_tid=1' -a 'object_number=77, tag_tid=12, entry_tid=1' = 'object_number=67, tag_tid=11, entry_tid=8435' 2024-04-06T09:38:33.840 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=67, tag_tid=11, entry_tid=8435' '!=' 'object_number=66, tag_tid=11, entry_tid=8338' 2024-04-06T09:38:33.840 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:38:33.840 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=67, tag_tid=11, entry_tid=8435' = 'object_number=66, tag_tid=11, entry_tid=8338' 2024-04-06T09:38:33.840 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=67, tag_tid=11, entry_tid=8435' 2024-04-06T09:38:33.840 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=77, tag_tid=12, entry_tid=1' 2024-04-06T09:38:33.841 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:38:33.841 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:38:33.841 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=12 2024-04-06T09:38:33.842 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=67, tag_tid=11, entry_tid=8435' 2024-04-06T09:38:33.842 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:38:33.842 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:38:33.843 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=11 2024-04-06T09:38:33.843 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=77, tag_tid=12, entry_tid=1' 2024-04-06T09:38:33.844 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:38:33.844 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:38:33.845 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:38:33.845 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=67, tag_tid=11, entry_tid=8435' 2024-04-06T09:38:33.845 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:38:33.845 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:38:33.846 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=8435 2024-04-06T09:38:33.847 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 12 = 11 -a 1 -le 8435 2024-04-06T09:38:33.847 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:38:33.847 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:38:33.847 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:38:34.049 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:38:34.049 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:38:34.049 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:38:34.049 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:38:34.049 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:38:34.049 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:38:34.049 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:38:34.049 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:38:34.049 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:38:34.049 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:38:34.049 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:38:34.050 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:38:34.050 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:38:34.050 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:38:34.050 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:38:34.050 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:38:34.050 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:38:34.053 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:38:34.054 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:38:34.054 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:38:34.054 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:38:34.054 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:38:34.054 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:38:34.054 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:38:34.054 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:38:34.054 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:38:34.054 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:38:34.054 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:38:34.054 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:38:34.054 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:38:34.054 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:38:34.054 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:38:34.055 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:38:34.055 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:38:34.055 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:38:34.055 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:38:34.055 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:38:34.055 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:38:34.055 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:38:34.055 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:38:34.055 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:38:34.055 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:38:34.055 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:38:34.055 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:38:34.055 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:38:34.055 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:38:34.056 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:38:34.056 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:38:34.056 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:38:34.080 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:38:34.080 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:38:34.080 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:38:34.080 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:38:34.206 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:38:34.207 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:38:34.207 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:38:34.208 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:38:34.208 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:38:34.208 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:38:34.208 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:38:34.208 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:38:34.208 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:38:34.208 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:38:34.208 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:38:34.209 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:38:34.209 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:38:34.209 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:38:34.210 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:38:34.210 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:38:34.261 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 16 2024-04-06T09:38:34.261 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 19 2024-04-06T09:38:34.261 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:38:34.261 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=77, tag_tid=12, entry_tid=1], [object_number=76, tag_tid=12, entry_tid=0], [object_number=79, tag_tid=11, entry_tid=17175], [object_number=78, tag_tid=11, entry_tid=17174]]], state=connected] 2024-04-06T09:38:34.261 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=67, tag_tid=11, entry_tid=8539], [object_number=66, tag_tid=11, entry_tid=8538], [object_number=65, tag_tid=11, entry_tid=8537], [object_number=64, tag_tid=11, entry_tid=8536]]], state=connected] 2024-04-06T09:38:34.264 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:38:34.266 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=77, tag_tid=12, entry_tid=1' 2024-04-06T09:38:34.267 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:38:34.267 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:38:34.267 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:38:34.267 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:38:34.267 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:38:34.267 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:38:34.267 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:38:34.267 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:38:34.267 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:38:34.268 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:38:34.268 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:38:34.268 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:38:34.268 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:38:34.268 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:38:34.269 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:38:34.316 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 16 2024-04-06T09:38:34.316 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 19 2024-04-06T09:38:34.316 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:38:34.316 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=77, tag_tid=12, entry_tid=1], [object_number=76, tag_tid=12, entry_tid=0], [object_number=79, tag_tid=11, entry_tid=17175], [object_number=78, tag_tid=11, entry_tid=17174]]], state=connected] 2024-04-06T09:38:34.316 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=67, tag_tid=11, entry_tid=8539], [object_number=66, tag_tid=11, entry_tid=8538], [object_number=65, tag_tid=11, entry_tid=8537], [object_number=64, tag_tid=11, entry_tid=8536]]], state=connected] 2024-04-06T09:38:34.319 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:38:34.322 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=67, tag_tid=11, entry_tid=8539' 2024-04-06T09:38:34.322 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=77, tag_tid=12, entry_tid=1' -a 'object_number=77, tag_tid=12, entry_tid=1' = 'object_number=67, tag_tid=11, entry_tid=8539' 2024-04-06T09:38:34.322 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=67, tag_tid=11, entry_tid=8539' '!=' 'object_number=67, tag_tid=11, entry_tid=8435' 2024-04-06T09:38:34.322 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:38:34.322 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=67, tag_tid=11, entry_tid=8539' = 'object_number=67, tag_tid=11, entry_tid=8435' 2024-04-06T09:38:34.322 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=67, tag_tid=11, entry_tid=8539' 2024-04-06T09:38:34.322 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=77, tag_tid=12, entry_tid=1' 2024-04-06T09:38:34.322 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:38:34.322 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:38:34.323 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=12 2024-04-06T09:38:34.323 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=67, tag_tid=11, entry_tid=8539' 2024-04-06T09:38:34.324 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:38:34.324 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:38:34.324 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=11 2024-04-06T09:38:34.325 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=77, tag_tid=12, entry_tid=1' 2024-04-06T09:38:34.325 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:38:34.325 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:38:34.326 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:38:34.327 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=67, tag_tid=11, entry_tid=8539' 2024-04-06T09:38:34.327 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:38:34.327 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:38:34.328 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=8539 2024-04-06T09:38:34.328 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 12 = 11 -a 1 -le 8539 2024-04-06T09:38:34.328 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:38:34.328 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:38:34.328 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:38:34.530 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:38:34.530 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:38:34.530 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:38:34.530 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:38:34.530 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:38:34.530 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:38:34.530 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:38:34.530 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:38:34.530 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:38:34.530 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:38:34.531 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:38:34.531 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:38:34.531 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:38:34.531 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:38:34.531 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:38:34.531 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:38:34.531 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:38:34.534 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:38:34.534 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:38:34.534 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:38:34.534 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:38:34.535 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:38:34.535 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:38:34.535 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:38:34.535 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:38:34.535 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:38:34.535 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:38:34.535 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:38:34.535 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:38:34.535 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:38:34.535 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:38:34.535 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:38:34.535 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:38:34.535 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:38:34.536 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:38:34.536 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:38:34.536 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:38:34.536 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:38:34.536 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:38:34.536 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:38:34.536 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:38:34.536 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:38:34.536 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:38:34.536 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:38:34.536 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:38:34.536 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:38:34.536 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:38:34.536 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:38:34.537 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:38:34.561 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:38:34.561 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:38:34.561 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:38:34.561 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:38:34.687 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:38:34.689 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:38:34.689 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:38:34.689 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:38:34.689 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:38:34.689 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:38:34.689 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:38:34.689 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:38:34.689 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:38:34.689 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:38:34.689 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:38:34.689 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:38:34.690 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:38:34.690 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:38:34.691 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:38:34.691 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:38:34.746 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 16 2024-04-06T09:38:34.747 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 19 2024-04-06T09:38:34.747 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:38:34.747 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=77, tag_tid=12, entry_tid=1], [object_number=76, tag_tid=12, entry_tid=0], [object_number=79, tag_tid=11, entry_tid=17175], [object_number=78, tag_tid=11, entry_tid=17174]]], state=connected] 2024-04-06T09:38:34.747 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=64, tag_tid=11, entry_tid=8624], [object_number=67, tag_tid=11, entry_tid=8623], [object_number=66, tag_tid=11, entry_tid=8622], [object_number=65, tag_tid=11, entry_tid=8621]]], state=connected] 2024-04-06T09:38:34.750 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:38:34.751 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=77, tag_tid=12, entry_tid=1' 2024-04-06T09:38:34.753 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:38:34.753 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:38:34.753 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:38:34.753 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:38:34.753 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:38:34.753 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:38:34.753 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:38:34.753 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:38:34.753 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:38:34.753 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:38:34.753 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:38:34.753 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:38:34.754 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:38:34.754 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:38:34.754 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:38:34.804 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 16 2024-04-06T09:38:34.804 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 19 2024-04-06T09:38:34.804 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:38:34.804 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=77, tag_tid=12, entry_tid=1], [object_number=76, tag_tid=12, entry_tid=0], [object_number=79, tag_tid=11, entry_tid=17175], [object_number=78, tag_tid=11, entry_tid=17174]]], state=connected] 2024-04-06T09:38:34.804 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=64, tag_tid=11, entry_tid=8624], [object_number=67, tag_tid=11, entry_tid=8623], [object_number=66, tag_tid=11, entry_tid=8622], [object_number=65, tag_tid=11, entry_tid=8621]]], state=connected] 2024-04-06T09:38:34.807 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:38:34.809 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=64, tag_tid=11, entry_tid=8624' 2024-04-06T09:38:34.809 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=77, tag_tid=12, entry_tid=1' -a 'object_number=77, tag_tid=12, entry_tid=1' = 'object_number=64, tag_tid=11, entry_tid=8624' 2024-04-06T09:38:34.809 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=64, tag_tid=11, entry_tid=8624' '!=' 'object_number=67, tag_tid=11, entry_tid=8539' 2024-04-06T09:38:34.809 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:38:34.809 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=64, tag_tid=11, entry_tid=8624' = 'object_number=67, tag_tid=11, entry_tid=8539' 2024-04-06T09:38:34.809 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=64, tag_tid=11, entry_tid=8624' 2024-04-06T09:38:34.810 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=77, tag_tid=12, entry_tid=1' 2024-04-06T09:38:34.810 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:38:34.810 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:38:34.811 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=12 2024-04-06T09:38:34.811 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=64, tag_tid=11, entry_tid=8624' 2024-04-06T09:38:34.811 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:38:34.811 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:38:34.812 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=11 2024-04-06T09:38:34.813 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=77, tag_tid=12, entry_tid=1' 2024-04-06T09:38:34.813 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:38:34.813 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:38:34.815 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:38:34.815 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=64, tag_tid=11, entry_tid=8624' 2024-04-06T09:38:34.815 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:38:34.815 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:38:34.816 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=8624 2024-04-06T09:38:34.816 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 12 = 11 -a 1 -le 8624 2024-04-06T09:38:34.817 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:38:34.817 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:38:34.817 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:38:35.018 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:38:35.018 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:38:35.019 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:38:35.019 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:38:35.019 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:38:35.019 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:38:35.019 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:38:35.019 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:38:35.019 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:38:35.019 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:38:35.019 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:38:35.019 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:38:35.019 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:38:35.019 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:38:35.019 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:38:35.020 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:38:35.020 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:38:35.022 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:38:35.022 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:38:35.022 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:38:35.022 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:38:35.023 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:38:35.023 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:38:35.023 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:38:35.023 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:38:35.023 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:38:35.023 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:38:35.023 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:38:35.023 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:38:35.023 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:38:35.023 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:38:35.024 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:38:35.024 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:38:35.024 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:38:35.024 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:38:35.024 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:38:35.024 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:38:35.024 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:38:35.024 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:38:35.024 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:38:35.024 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:38:35.024 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:38:35.025 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:38:35.025 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:38:35.025 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:38:35.025 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:38:35.025 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:38:35.025 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:38:35.025 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:38:35.048 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:38:35.049 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:38:35.049 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:38:35.049 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:38:35.181 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:38:35.182 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:38:35.182 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:38:35.182 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:38:35.183 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:38:35.183 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:38:35.183 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:38:35.183 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:38:35.183 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:38:35.183 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:38:35.184 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:38:35.184 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:38:35.184 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:38:35.185 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:38:35.185 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:38:35.185 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:38:35.241 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 16 2024-04-06T09:38:35.241 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 19 2024-04-06T09:38:35.241 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:38:35.241 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=77, tag_tid=12, entry_tid=1], [object_number=76, tag_tid=12, entry_tid=0], [object_number=79, tag_tid=11, entry_tid=17175], [object_number=78, tag_tid=11, entry_tid=17174]]], state=connected] 2024-04-06T09:38:35.241 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=66, tag_tid=11, entry_tid=8694], [object_number=65, tag_tid=11, entry_tid=8693], [object_number=64, tag_tid=11, entry_tid=8692], [object_number=67, tag_tid=11, entry_tid=8691]]], state=connected] 2024-04-06T09:38:35.245 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:38:35.246 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=77, tag_tid=12, entry_tid=1' 2024-04-06T09:38:35.247 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:38:35.247 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:38:35.247 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:38:35.248 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:38:35.248 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:38:35.248 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:38:35.248 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:38:35.248 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:38:35.248 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:38:35.248 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:38:35.248 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:38:35.248 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:38:35.249 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:38:35.249 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:38:35.250 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:38:35.302 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 16 2024-04-06T09:38:35.302 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 19 2024-04-06T09:38:35.302 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:38:35.302 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=77, tag_tid=12, entry_tid=1], [object_number=76, tag_tid=12, entry_tid=0], [object_number=79, tag_tid=11, entry_tid=17175], [object_number=78, tag_tid=11, entry_tid=17174]]], state=connected] 2024-04-06T09:38:35.302 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=66, tag_tid=11, entry_tid=8694], [object_number=65, tag_tid=11, entry_tid=8693], [object_number=64, tag_tid=11, entry_tid=8692], [object_number=67, tag_tid=11, entry_tid=8691]]], state=connected] 2024-04-06T09:38:35.304 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:38:35.306 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=66, tag_tid=11, entry_tid=8694' 2024-04-06T09:38:35.307 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=77, tag_tid=12, entry_tid=1' -a 'object_number=77, tag_tid=12, entry_tid=1' = 'object_number=66, tag_tid=11, entry_tid=8694' 2024-04-06T09:38:35.307 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=66, tag_tid=11, entry_tid=8694' '!=' 'object_number=64, tag_tid=11, entry_tid=8624' 2024-04-06T09:38:35.307 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:38:35.307 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=66, tag_tid=11, entry_tid=8694' = 'object_number=64, tag_tid=11, entry_tid=8624' 2024-04-06T09:38:35.307 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=66, tag_tid=11, entry_tid=8694' 2024-04-06T09:38:35.307 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=77, tag_tid=12, entry_tid=1' 2024-04-06T09:38:35.307 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:38:35.307 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:38:35.308 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=12 2024-04-06T09:38:35.308 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=66, tag_tid=11, entry_tid=8694' 2024-04-06T09:38:35.308 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:38:35.308 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:38:35.309 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=11 2024-04-06T09:38:35.310 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=77, tag_tid=12, entry_tid=1' 2024-04-06T09:38:35.310 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:38:35.310 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:38:35.311 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:38:35.312 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=66, tag_tid=11, entry_tid=8694' 2024-04-06T09:38:35.312 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:38:35.312 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:38:35.314 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=8694 2024-04-06T09:38:35.314 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 12 = 11 -a 1 -le 8694 2024-04-06T09:38:35.314 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:38:35.314 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:38:35.314 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:38:35.515 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:38:35.515 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:38:35.516 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:38:35.516 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:38:35.516 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:38:35.516 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:38:35.516 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:38:35.516 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:38:35.516 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:38:35.516 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:38:35.516 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:38:35.516 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:38:35.516 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:38:35.516 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:38:35.516 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:38:35.517 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:38:35.517 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:38:35.519 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:38:35.519 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:38:35.519 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:38:35.519 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:38:35.519 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:38:35.519 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:38:35.519 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:38:35.519 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:38:35.520 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:38:35.520 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:38:35.520 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:38:35.520 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:38:35.520 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:38:35.520 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:38:35.520 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:38:35.520 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:38:35.521 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:38:35.521 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:38:35.521 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:38:35.521 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:38:35.521 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:38:35.521 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:38:35.521 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:38:35.521 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:38:35.522 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:38:35.522 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:38:35.522 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:38:35.522 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:38:35.522 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:38:35.522 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:38:35.522 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:38:35.522 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:38:35.545 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:38:35.545 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:38:35.545 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:38:35.545 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:38:35.678 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:38:35.679 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:38:35.679 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:38:35.680 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:38:35.680 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:38:35.680 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:38:35.680 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:38:35.680 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:38:35.680 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:38:35.680 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:38:35.680 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:38:35.681 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:38:35.681 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:38:35.682 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:38:35.682 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:38:35.683 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:38:35.746 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 17 2024-04-06T09:38:35.746 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 19 2024-04-06T09:38:35.746 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:38:35.746 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=77, tag_tid=12, entry_tid=1], [object_number=76, tag_tid=12, entry_tid=0], [object_number=79, tag_tid=11, entry_tid=17175], [object_number=78, tag_tid=11, entry_tid=17174]]], state=connected] 2024-04-06T09:38:35.746 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=69, tag_tid=11, entry_tid=8741], [object_number=68, tag_tid=11, entry_tid=8740], [object_number=71, tag_tid=11, entry_tid=8739], [object_number=70, tag_tid=11, entry_tid=8738]]], state=connected] 2024-04-06T09:38:35.749 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:38:35.750 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=77, tag_tid=12, entry_tid=1' 2024-04-06T09:38:35.751 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:38:35.752 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:38:35.752 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:38:35.752 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:38:35.752 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:38:35.752 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:38:35.752 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:38:35.752 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:38:35.752 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:38:35.752 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:38:35.752 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:38:35.752 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:38:35.753 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:38:35.753 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:38:35.753 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:38:35.798 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 17 2024-04-06T09:38:35.798 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 19 2024-04-06T09:38:35.798 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:38:35.798 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=77, tag_tid=12, entry_tid=1], [object_number=76, tag_tid=12, entry_tid=0], [object_number=79, tag_tid=11, entry_tid=17175], [object_number=78, tag_tid=11, entry_tid=17174]]], state=connected] 2024-04-06T09:38:35.799 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=69, tag_tid=11, entry_tid=8741], [object_number=68, tag_tid=11, entry_tid=8740], [object_number=71, tag_tid=11, entry_tid=8739], [object_number=70, tag_tid=11, entry_tid=8738]]], state=connected] 2024-04-06T09:38:35.802 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:38:35.804 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=69, tag_tid=11, entry_tid=8741' 2024-04-06T09:38:35.804 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=77, tag_tid=12, entry_tid=1' -a 'object_number=77, tag_tid=12, entry_tid=1' = 'object_number=69, tag_tid=11, entry_tid=8741' 2024-04-06T09:38:35.804 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=69, tag_tid=11, entry_tid=8741' '!=' 'object_number=66, tag_tid=11, entry_tid=8694' 2024-04-06T09:38:35.804 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:38:35.804 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=69, tag_tid=11, entry_tid=8741' = 'object_number=66, tag_tid=11, entry_tid=8694' 2024-04-06T09:38:35.804 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=69, tag_tid=11, entry_tid=8741' 2024-04-06T09:38:35.804 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=77, tag_tid=12, entry_tid=1' 2024-04-06T09:38:35.804 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:38:35.804 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:38:35.805 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=12 2024-04-06T09:38:35.805 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=69, tag_tid=11, entry_tid=8741' 2024-04-06T09:38:35.805 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:38:35.806 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:38:35.806 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=11 2024-04-06T09:38:35.807 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=77, tag_tid=12, entry_tid=1' 2024-04-06T09:38:35.807 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:38:35.807 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:38:35.808 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:38:35.809 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=69, tag_tid=11, entry_tid=8741' 2024-04-06T09:38:35.809 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:38:35.809 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:38:35.810 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=8741 2024-04-06T09:38:35.810 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 12 = 11 -a 1 -le 8741 2024-04-06T09:38:35.810 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:38:35.810 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:38:35.810 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:38:36.012 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:38:36.012 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:38:36.012 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:38:36.012 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:38:36.013 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:38:36.013 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:38:36.013 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:38:36.013 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:38:36.013 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:38:36.013 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:38:36.013 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:38:36.013 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:38:36.013 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:38:36.013 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:38:36.013 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:38:36.015 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:38:36.015 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:38:36.015 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:38:36.015 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:38:36.015 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:38:36.015 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:38:36.015 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:38:36.015 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:38:36.015 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:38:36.015 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:38:36.015 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:38:36.016 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:38:36.016 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:38:36.016 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:38:36.016 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:38:36.016 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:38:36.016 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:38:36.016 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:38:36.017 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:38:36.017 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:38:36.017 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:38:36.017 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:38:36.017 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:38:36.018 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:38:36.018 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:38:36.018 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:38:36.018 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:38:36.018 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:38:36.018 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:38:36.018 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:38:36.018 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:38:36.018 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:38:36.018 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:38:36.018 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:38:36.041 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:38:36.042 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:38:36.042 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:38:36.042 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:38:36.172 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:38:36.172 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:38:36.173 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:38:36.173 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:38:36.173 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:38:36.173 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:38:36.173 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:38:36.173 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:38:36.173 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:38:36.173 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:38:36.174 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:38:36.174 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:38:36.174 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:38:36.175 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:38:36.175 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:38:36.175 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:38:36.226 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 17 2024-04-06T09:38:36.226 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 19 2024-04-06T09:38:36.226 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:38:36.226 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=77, tag_tid=12, entry_tid=1], [object_number=76, tag_tid=12, entry_tid=0], [object_number=79, tag_tid=11, entry_tid=17175], [object_number=78, tag_tid=11, entry_tid=17174]]], state=connected] 2024-04-06T09:38:36.226 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=71, tag_tid=11, entry_tid=8851], [object_number=70, tag_tid=11, entry_tid=8850], [object_number=69, tag_tid=11, entry_tid=8849], [object_number=68, tag_tid=11, entry_tid=8848]]], state=connected] 2024-04-06T09:38:36.230 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:38:36.231 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=77, tag_tid=12, entry_tid=1' 2024-04-06T09:38:36.232 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:38:36.232 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:38:36.232 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:38:36.232 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:38:36.232 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:38:36.232 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:38:36.232 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:38:36.232 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:38:36.233 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:38:36.233 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:38:36.233 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:38:36.233 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:38:36.233 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:38:36.234 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:38:36.234 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:38:36.279 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 17 2024-04-06T09:38:36.279 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 19 2024-04-06T09:38:36.280 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:38:36.280 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=77, tag_tid=12, entry_tid=1], [object_number=76, tag_tid=12, entry_tid=0], [object_number=79, tag_tid=11, entry_tid=17175], [object_number=78, tag_tid=11, entry_tid=17174]]], state=connected] 2024-04-06T09:38:36.280 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=71, tag_tid=11, entry_tid=8851], [object_number=70, tag_tid=11, entry_tid=8850], [object_number=69, tag_tid=11, entry_tid=8849], [object_number=68, tag_tid=11, entry_tid=8848]]], state=connected] 2024-04-06T09:38:36.283 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:38:36.285 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=71, tag_tid=11, entry_tid=8851' 2024-04-06T09:38:36.285 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=77, tag_tid=12, entry_tid=1' -a 'object_number=77, tag_tid=12, entry_tid=1' = 'object_number=71, tag_tid=11, entry_tid=8851' 2024-04-06T09:38:36.285 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=71, tag_tid=11, entry_tid=8851' '!=' 'object_number=69, tag_tid=11, entry_tid=8741' 2024-04-06T09:38:36.285 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:38:36.285 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=71, tag_tid=11, entry_tid=8851' = 'object_number=69, tag_tid=11, entry_tid=8741' 2024-04-06T09:38:36.285 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=71, tag_tid=11, entry_tid=8851' 2024-04-06T09:38:36.285 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=77, tag_tid=12, entry_tid=1' 2024-04-06T09:38:36.285 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:38:36.286 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:38:36.286 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=12 2024-04-06T09:38:36.287 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=71, tag_tid=11, entry_tid=8851' 2024-04-06T09:38:36.287 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:38:36.287 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:38:36.288 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=11 2024-04-06T09:38:36.289 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=77, tag_tid=12, entry_tid=1' 2024-04-06T09:38:36.289 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:38:36.289 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:38:36.289 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:38:36.290 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=71, tag_tid=11, entry_tid=8851' 2024-04-06T09:38:36.290 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:38:36.290 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:38:36.291 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=8851 2024-04-06T09:38:36.291 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 12 = 11 -a 1 -le 8851 2024-04-06T09:38:36.292 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:38:36.292 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:38:36.292 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:38:36.493 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:38:36.493 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:38:36.493 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:38:36.494 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:38:36.494 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:38:36.494 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:38:36.494 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:38:36.494 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:38:36.494 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:38:36.494 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:38:36.494 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:38:36.494 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:38:36.494 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:38:36.494 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:38:36.494 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:38:36.495 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:38:36.495 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:38:36.498 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:38:36.498 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:38:36.498 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:38:36.498 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:38:36.498 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:38:36.498 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:38:36.498 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:38:36.498 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:38:36.498 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:38:36.498 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:38:36.498 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:38:36.498 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:38:36.499 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:38:36.499 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:38:36.499 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:38:36.499 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:38:36.499 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:38:36.499 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:38:36.499 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:38:36.499 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:38:36.499 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:38:36.499 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:38:36.499 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:38:36.499 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:38:36.499 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:38:36.499 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:38:36.499 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:38:36.500 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:38:36.500 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:38:36.500 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:38:36.500 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:38:36.500 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:38:36.525 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:38:36.525 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:38:36.525 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:38:36.526 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:38:36.655 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:38:36.657 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:38:36.657 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:38:36.657 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:38:36.657 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:38:36.657 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:38:36.657 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:38:36.657 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:38:36.657 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:38:36.657 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:38:36.658 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:38:36.658 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:38:36.658 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:38:36.660 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:38:36.661 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:38:36.661 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:38:36.708 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 17 2024-04-06T09:38:36.708 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 19 2024-04-06T09:38:36.708 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:38:36.708 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=77, tag_tid=12, entry_tid=1], [object_number=76, tag_tid=12, entry_tid=0], [object_number=79, tag_tid=11, entry_tid=17175], [object_number=78, tag_tid=11, entry_tid=17174]]], state=connected] 2024-04-06T09:38:36.708 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=68, tag_tid=11, entry_tid=8960], [object_number=71, tag_tid=11, entry_tid=8959], [object_number=70, tag_tid=11, entry_tid=8958], [object_number=69, tag_tid=11, entry_tid=8957]]], state=connected] 2024-04-06T09:38:36.711 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:38:36.713 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=77, tag_tid=12, entry_tid=1' 2024-04-06T09:38:36.714 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:38:36.714 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:38:36.714 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:38:36.714 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:38:36.714 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:38:36.714 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:38:36.714 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:38:36.714 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:38:36.714 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:38:36.714 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:38:36.715 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:38:36.715 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:38:36.715 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:38:36.715 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:38:36.715 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:38:36.761 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 17 2024-04-06T09:38:36.761 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 19 2024-04-06T09:38:36.761 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:38:36.761 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=77, tag_tid=12, entry_tid=1], [object_number=76, tag_tid=12, entry_tid=0], [object_number=79, tag_tid=11, entry_tid=17175], [object_number=78, tag_tid=11, entry_tid=17174]]], state=connected] 2024-04-06T09:38:36.761 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=68, tag_tid=11, entry_tid=8960], [object_number=71, tag_tid=11, entry_tid=8959], [object_number=70, tag_tid=11, entry_tid=8958], [object_number=69, tag_tid=11, entry_tid=8957]]], state=connected] 2024-04-06T09:38:36.765 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:38:36.767 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=68, tag_tid=11, entry_tid=8960' 2024-04-06T09:38:36.767 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=77, tag_tid=12, entry_tid=1' -a 'object_number=77, tag_tid=12, entry_tid=1' = 'object_number=68, tag_tid=11, entry_tid=8960' 2024-04-06T09:38:36.767 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=68, tag_tid=11, entry_tid=8960' '!=' 'object_number=71, tag_tid=11, entry_tid=8851' 2024-04-06T09:38:36.767 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:38:36.767 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=68, tag_tid=11, entry_tid=8960' = 'object_number=71, tag_tid=11, entry_tid=8851' 2024-04-06T09:38:36.767 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=68, tag_tid=11, entry_tid=8960' 2024-04-06T09:38:36.767 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=77, tag_tid=12, entry_tid=1' 2024-04-06T09:38:36.767 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:38:36.767 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:38:36.768 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=12 2024-04-06T09:38:36.769 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:38:36.769 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:38:36.769 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=68, tag_tid=11, entry_tid=8960' 2024-04-06T09:38:36.770 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=11 2024-04-06T09:38:36.771 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=77, tag_tid=12, entry_tid=1' 2024-04-06T09:38:36.771 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:38:36.771 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:38:36.772 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:38:36.773 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:38:36.773 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=68, tag_tid=11, entry_tid=8960' 2024-04-06T09:38:36.773 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:38:36.774 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=8960 2024-04-06T09:38:36.774 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 12 = 11 -a 1 -le 8960 2024-04-06T09:38:36.774 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:38:36.774 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:38:36.774 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:38:36.976 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:38:36.976 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:38:36.976 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:38:36.976 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:38:36.976 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:38:36.976 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:38:36.977 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:38:36.977 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:38:36.977 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:38:36.977 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:38:36.977 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:38:36.977 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:38:36.977 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:38:36.977 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:38:36.977 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:38:36.978 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:38:36.978 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:38:36.980 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:38:36.980 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:38:36.981 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:38:36.981 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:38:36.981 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:38:36.981 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:38:36.981 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:38:36.981 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:38:36.981 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:38:36.981 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:38:36.981 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:38:36.981 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:38:36.981 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:38:36.982 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:38:36.982 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:38:36.982 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:38:36.982 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:38:36.982 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:38:36.982 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:38:36.982 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:38:36.982 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:38:36.982 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:38:36.982 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:38:36.982 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:38:36.982 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:38:36.983 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:38:36.983 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:38:36.983 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:38:36.983 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:38:36.983 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:38:36.983 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:38:36.983 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:38:37.006 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:38:37.006 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:38:37.007 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:38:37.007 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:38:37.137 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:38:37.139 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:38:37.139 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:38:37.139 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:38:37.139 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:38:37.139 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:38:37.139 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:38:37.139 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:38:37.139 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:38:37.139 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:38:37.139 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:38:37.140 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:38:37.140 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:38:37.141 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:38:37.141 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:38:37.141 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:38:37.191 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 17 2024-04-06T09:38:37.192 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 19 2024-04-06T09:38:37.192 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:38:37.192 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=77, tag_tid=12, entry_tid=1], [object_number=76, tag_tid=12, entry_tid=0], [object_number=79, tag_tid=11, entry_tid=17175], [object_number=78, tag_tid=11, entry_tid=17174]]], state=connected] 2024-04-06T09:38:37.192 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=69, tag_tid=11, entry_tid=9077], [object_number=68, tag_tid=11, entry_tid=9076], [object_number=71, tag_tid=11, entry_tid=9075], [object_number=70, tag_tid=11, entry_tid=9074]]], state=connected] 2024-04-06T09:38:37.195 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:38:37.196 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=77, tag_tid=12, entry_tid=1' 2024-04-06T09:38:37.198 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:38:37.198 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:38:37.198 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:38:37.198 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:38:37.199 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:38:37.199 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:38:37.199 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:38:37.199 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:38:37.199 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:38:37.199 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:38:37.199 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:38:37.199 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:38:37.200 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:38:37.200 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:38:37.200 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:38:37.245 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 17 2024-04-06T09:38:37.245 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 19 2024-04-06T09:38:37.245 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:38:37.245 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=77, tag_tid=12, entry_tid=1], [object_number=76, tag_tid=12, entry_tid=0], [object_number=79, tag_tid=11, entry_tid=17175], [object_number=78, tag_tid=11, entry_tid=17174]]], state=connected] 2024-04-06T09:38:37.245 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=69, tag_tid=11, entry_tid=9077], [object_number=68, tag_tid=11, entry_tid=9076], [object_number=71, tag_tid=11, entry_tid=9075], [object_number=70, tag_tid=11, entry_tid=9074]]], state=connected] 2024-04-06T09:38:37.248 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:38:37.251 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=69, tag_tid=11, entry_tid=9077' 2024-04-06T09:38:37.251 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=77, tag_tid=12, entry_tid=1' -a 'object_number=77, tag_tid=12, entry_tid=1' = 'object_number=69, tag_tid=11, entry_tid=9077' 2024-04-06T09:38:37.251 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=69, tag_tid=11, entry_tid=9077' '!=' 'object_number=68, tag_tid=11, entry_tid=8960' 2024-04-06T09:38:37.251 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:38:37.251 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=69, tag_tid=11, entry_tid=9077' = 'object_number=68, tag_tid=11, entry_tid=8960' 2024-04-06T09:38:37.251 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=69, tag_tid=11, entry_tid=9077' 2024-04-06T09:38:37.251 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=77, tag_tid=12, entry_tid=1' 2024-04-06T09:38:37.251 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:38:37.251 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:38:37.252 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=12 2024-04-06T09:38:37.253 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=69, tag_tid=11, entry_tid=9077' 2024-04-06T09:38:37.253 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:38:37.253 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:38:37.253 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=11 2024-04-06T09:38:37.254 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=77, tag_tid=12, entry_tid=1' 2024-04-06T09:38:37.254 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:38:37.254 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:38:37.255 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:38:37.256 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=69, tag_tid=11, entry_tid=9077' 2024-04-06T09:38:37.256 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:38:37.256 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:38:37.257 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=9077 2024-04-06T09:38:37.257 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 12 = 11 -a 1 -le 9077 2024-04-06T09:38:37.257 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:38:37.258 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:38:37.258 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:38:37.459 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:38:37.459 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:38:37.459 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:38:37.459 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:38:37.459 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:38:37.460 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:38:37.460 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:38:37.460 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:38:37.460 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:38:37.460 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:38:37.460 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:38:37.460 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:38:37.460 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:38:37.460 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:38:37.460 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:38:37.461 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:38:37.461 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:38:37.464 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:38:37.464 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:38:37.464 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:38:37.464 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:38:37.464 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:38:37.464 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:38:37.464 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:38:37.464 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:38:37.464 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:38:37.464 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:38:37.464 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:38:37.464 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:38:37.465 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:38:37.465 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:38:37.465 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:38:37.465 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:38:37.466 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:38:37.466 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:38:37.466 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:38:37.466 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:38:37.466 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:38:37.466 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:38:37.466 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:38:37.466 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:38:37.467 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:38:37.467 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:38:37.467 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:38:37.467 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:38:37.467 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:38:37.467 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:38:37.467 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:38:37.467 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:38:37.491 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:38:37.491 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:38:37.491 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:38:37.491 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:38:37.623 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:38:37.624 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:38:37.624 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:38:37.624 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:38:37.624 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:38:37.624 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:38:37.624 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:38:37.624 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:38:37.624 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:38:37.624 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:38:37.625 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:38:37.625 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:38:37.625 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:38:37.627 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:38:37.627 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:38:37.627 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:38:37.674 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 17 2024-04-06T09:38:37.674 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 19 2024-04-06T09:38:37.674 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:38:37.674 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=77, tag_tid=12, entry_tid=1], [object_number=76, tag_tid=12, entry_tid=0], [object_number=79, tag_tid=11, entry_tid=17175], [object_number=78, tag_tid=11, entry_tid=17174]]], state=connected] 2024-04-06T09:38:37.675 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=70, tag_tid=11, entry_tid=9186], [object_number=69, tag_tid=11, entry_tid=9185], [object_number=68, tag_tid=11, entry_tid=9184], [object_number=71, tag_tid=11, entry_tid=9183]]], state=connected] 2024-04-06T09:38:37.677 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:38:37.678 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=77, tag_tid=12, entry_tid=1' 2024-04-06T09:38:37.680 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:38:37.680 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:38:37.680 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:38:37.680 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:38:37.680 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:38:37.680 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:38:37.680 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:38:37.680 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:38:37.680 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:38:37.681 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:38:37.681 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:38:37.681 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:38:37.681 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:38:37.681 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:38:37.681 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:38:37.726 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 17 2024-04-06T09:38:37.727 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 19 2024-04-06T09:38:37.727 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:38:37.727 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=77, tag_tid=12, entry_tid=1], [object_number=76, tag_tid=12, entry_tid=0], [object_number=79, tag_tid=11, entry_tid=17175], [object_number=78, tag_tid=11, entry_tid=17174]]], state=connected] 2024-04-06T09:38:37.727 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=70, tag_tid=11, entry_tid=9186], [object_number=69, tag_tid=11, entry_tid=9185], [object_number=68, tag_tid=11, entry_tid=9184], [object_number=71, tag_tid=11, entry_tid=9183]]], state=connected] 2024-04-06T09:38:37.730 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:38:37.732 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=70, tag_tid=11, entry_tid=9186' 2024-04-06T09:38:37.732 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=77, tag_tid=12, entry_tid=1' -a 'object_number=77, tag_tid=12, entry_tid=1' = 'object_number=70, tag_tid=11, entry_tid=9186' 2024-04-06T09:38:37.732 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=70, tag_tid=11, entry_tid=9186' '!=' 'object_number=69, tag_tid=11, entry_tid=9077' 2024-04-06T09:38:37.732 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:38:37.732 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=70, tag_tid=11, entry_tid=9186' = 'object_number=69, tag_tid=11, entry_tid=9077' 2024-04-06T09:38:37.732 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=70, tag_tid=11, entry_tid=9186' 2024-04-06T09:38:37.732 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=77, tag_tid=12, entry_tid=1' 2024-04-06T09:38:37.732 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:38:37.733 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:38:37.733 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=12 2024-04-06T09:38:37.734 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=70, tag_tid=11, entry_tid=9186' 2024-04-06T09:38:37.734 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:38:37.734 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:38:37.735 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=11 2024-04-06T09:38:37.735 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=77, tag_tid=12, entry_tid=1' 2024-04-06T09:38:37.735 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:38:37.736 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:38:37.736 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:38:37.737 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=70, tag_tid=11, entry_tid=9186' 2024-04-06T09:38:37.737 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:38:37.737 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:38:37.738 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=9186 2024-04-06T09:38:37.738 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 12 = 11 -a 1 -le 9186 2024-04-06T09:38:37.739 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:38:37.739 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:38:37.739 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:38:37.940 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:38:37.940 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:38:37.940 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:38:37.941 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:38:37.941 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:38:37.941 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:38:37.941 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:38:37.941 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:38:37.941 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:38:37.941 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:38:37.941 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:38:37.941 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:38:37.941 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:38:37.941 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:38:37.941 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:38:37.942 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:38:37.942 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:38:37.943 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:38:37.943 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:38:37.943 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:38:37.943 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:38:37.943 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:38:37.944 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:38:37.944 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:38:37.944 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:38:37.944 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:38:37.944 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:38:37.944 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:38:37.944 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:38:37.944 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:38:37.944 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:38:37.944 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:38:37.944 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:38:37.945 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:38:37.946 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:38:37.946 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:38:37.946 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:38:37.946 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:38:37.946 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:38:37.946 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:38:37.946 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:38:37.946 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:38:37.946 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:38:37.946 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:38:37.946 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:38:37.946 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:38:37.946 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:38:37.947 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:38:37.947 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:38:37.970 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:38:37.971 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:38:37.971 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:38:37.971 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:38:38.101 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:38:38.102 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:38:38.102 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:38:38.102 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:38:38.102 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:38:38.102 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:38:38.102 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:38:38.102 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:38:38.103 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:38:38.103 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:38:38.103 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:38:38.103 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:38:38.103 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:38:38.105 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:38:38.105 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:38:38.105 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:38:38.151 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 17 2024-04-06T09:38:38.151 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 19 2024-04-06T09:38:38.151 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:38:38.151 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=77, tag_tid=12, entry_tid=1], [object_number=76, tag_tid=12, entry_tid=0], [object_number=79, tag_tid=11, entry_tid=17175], [object_number=78, tag_tid=11, entry_tid=17174]]], state=connected] 2024-04-06T09:38:38.151 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=71, tag_tid=11, entry_tid=9283], [object_number=70, tag_tid=11, entry_tid=9282], [object_number=69, tag_tid=11, entry_tid=9281], [object_number=68, tag_tid=11, entry_tid=9280]]], state=connected] 2024-04-06T09:38:38.154 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:38:38.156 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=77, tag_tid=12, entry_tid=1' 2024-04-06T09:38:38.157 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:38:38.157 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:38:38.157 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:38:38.157 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:38:38.157 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:38:38.157 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:38:38.157 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:38:38.157 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:38:38.157 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:38:38.157 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:38:38.157 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:38:38.158 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:38:38.161 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:38:38.161 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:38:38.162 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:38:38.204 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 17 2024-04-06T09:38:38.204 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 19 2024-04-06T09:38:38.204 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:38:38.204 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=77, tag_tid=12, entry_tid=1], [object_number=76, tag_tid=12, entry_tid=0], [object_number=79, tag_tid=11, entry_tid=17175], [object_number=78, tag_tid=11, entry_tid=17174]]], state=connected] 2024-04-06T09:38:38.205 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=71, tag_tid=11, entry_tid=9283], [object_number=70, tag_tid=11, entry_tid=9282], [object_number=69, tag_tid=11, entry_tid=9281], [object_number=68, tag_tid=11, entry_tid=9280]]], state=connected] 2024-04-06T09:38:38.208 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:38:38.210 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=71, tag_tid=11, entry_tid=9283' 2024-04-06T09:38:38.210 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=77, tag_tid=12, entry_tid=1' -a 'object_number=77, tag_tid=12, entry_tid=1' = 'object_number=71, tag_tid=11, entry_tid=9283' 2024-04-06T09:38:38.210 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=71, tag_tid=11, entry_tid=9283' '!=' 'object_number=70, tag_tid=11, entry_tid=9186' 2024-04-06T09:38:38.210 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:38:38.211 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=71, tag_tid=11, entry_tid=9283' = 'object_number=70, tag_tid=11, entry_tid=9186' 2024-04-06T09:38:38.211 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=71, tag_tid=11, entry_tid=9283' 2024-04-06T09:38:38.211 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=77, tag_tid=12, entry_tid=1' 2024-04-06T09:38:38.211 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:38:38.211 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:38:38.212 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=12 2024-04-06T09:38:38.212 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=71, tag_tid=11, entry_tid=9283' 2024-04-06T09:38:38.212 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:38:38.212 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:38:38.213 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=11 2024-04-06T09:38:38.213 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=77, tag_tid=12, entry_tid=1' 2024-04-06T09:38:38.214 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:38:38.214 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:38:38.214 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:38:38.215 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=71, tag_tid=11, entry_tid=9283' 2024-04-06T09:38:38.215 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:38:38.215 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:38:38.216 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=9283 2024-04-06T09:38:38.216 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 12 = 11 -a 1 -le 9283 2024-04-06T09:38:38.216 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:38:38.217 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:38:38.217 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:38:38.418 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:38:38.419 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:38:38.419 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:38:38.419 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:38:38.419 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:38:38.419 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:38:38.419 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:38:38.419 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:38:38.419 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:38:38.419 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:38:38.419 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:38:38.419 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:38:38.419 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:38:38.419 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:38:38.420 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:38:38.420 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:38:38.420 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:38:38.422 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:38:38.422 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:38:38.423 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:38:38.423 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:38:38.423 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:38:38.423 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:38:38.423 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:38:38.423 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:38:38.423 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:38:38.423 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:38:38.423 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:38:38.423 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:38:38.423 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:38:38.423 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:38:38.423 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:38:38.423 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:38:38.424 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:38:38.424 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:38:38.424 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:38:38.424 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:38:38.424 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:38:38.424 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:38:38.424 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:38:38.424 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:38:38.424 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:38:38.424 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:38:38.424 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:38:38.424 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:38:38.424 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:38:38.424 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:38:38.424 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:38:38.425 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:38:38.448 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:38:38.448 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:38:38.448 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:38:38.448 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:38:38.582 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:38:38.582 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:38:38.583 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:38:38.583 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:38:38.583 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:38:38.583 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:38:38.583 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:38:38.583 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:38:38.583 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:38:38.583 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:38:38.584 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:38:38.584 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:38:38.584 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:38:38.586 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:38:38.586 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:38:38.586 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:38:38.644 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 17 2024-04-06T09:38:38.644 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 19 2024-04-06T09:38:38.644 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:38:38.644 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=77, tag_tid=12, entry_tid=1], [object_number=76, tag_tid=12, entry_tid=0], [object_number=79, tag_tid=11, entry_tid=17175], [object_number=78, tag_tid=11, entry_tid=17174]]], state=connected] 2024-04-06T09:38:38.644 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=68, tag_tid=11, entry_tid=9368], [object_number=71, tag_tid=11, entry_tid=9367], [object_number=70, tag_tid=11, entry_tid=9366], [object_number=69, tag_tid=11, entry_tid=9365]]], state=connected] 2024-04-06T09:38:38.648 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:38:38.650 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=77, tag_tid=12, entry_tid=1' 2024-04-06T09:38:38.651 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:38:38.651 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:38:38.651 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:38:38.651 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:38:38.651 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:38:38.651 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:38:38.651 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:38:38.651 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:38:38.651 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:38:38.651 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:38:38.651 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:38:38.652 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:38:38.652 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:38:38.653 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:38:38.653 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:38:38.698 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 17 2024-04-06T09:38:38.698 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 19 2024-04-06T09:38:38.699 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:38:38.699 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=77, tag_tid=12, entry_tid=1], [object_number=76, tag_tid=12, entry_tid=0], [object_number=79, tag_tid=11, entry_tid=17175], [object_number=78, tag_tid=11, entry_tid=17174]]], state=connected] 2024-04-06T09:38:38.699 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=68, tag_tid=11, entry_tid=9368], [object_number=71, tag_tid=11, entry_tid=9367], [object_number=70, tag_tid=11, entry_tid=9366], [object_number=69, tag_tid=11, entry_tid=9365]]], state=connected] 2024-04-06T09:38:38.702 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:38:38.704 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=68, tag_tid=11, entry_tid=9368' 2024-04-06T09:38:38.704 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=77, tag_tid=12, entry_tid=1' -a 'object_number=77, tag_tid=12, entry_tid=1' = 'object_number=68, tag_tid=11, entry_tid=9368' 2024-04-06T09:38:38.704 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=68, tag_tid=11, entry_tid=9368' '!=' 'object_number=71, tag_tid=11, entry_tid=9283' 2024-04-06T09:38:38.705 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:38:38.705 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=68, tag_tid=11, entry_tid=9368' = 'object_number=71, tag_tid=11, entry_tid=9283' 2024-04-06T09:38:38.705 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=68, tag_tid=11, entry_tid=9368' 2024-04-06T09:38:38.705 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=77, tag_tid=12, entry_tid=1' 2024-04-06T09:38:38.705 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:38:38.705 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:38:38.705 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=12 2024-04-06T09:38:38.706 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=68, tag_tid=11, entry_tid=9368' 2024-04-06T09:38:38.707 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:38:38.707 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:38:38.707 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=11 2024-04-06T09:38:38.708 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=77, tag_tid=12, entry_tid=1' 2024-04-06T09:38:38.708 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:38:38.708 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:38:38.709 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:38:38.710 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=68, tag_tid=11, entry_tid=9368' 2024-04-06T09:38:38.710 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:38:38.710 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:38:38.711 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=9368 2024-04-06T09:38:38.711 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 12 = 11 -a 1 -le 9368 2024-04-06T09:38:38.711 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:38:38.711 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:38:38.711 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:38:38.913 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:38:38.914 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:38:38.914 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:38:38.914 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:38:38.914 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:38:38.914 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:38:38.914 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:38:38.914 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:38:38.914 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:38:38.914 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:38:38.914 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:38:38.914 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:38:38.914 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:38:38.914 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:38:38.915 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:38:38.915 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:38:38.915 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:38:38.919 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:38:38.919 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:38:38.919 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:38:38.919 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:38:38.919 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:38:38.919 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:38:38.919 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:38:38.919 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:38:38.919 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:38:38.919 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:38:38.919 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:38:38.919 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:38:38.920 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:38:38.920 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:38:38.920 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:38:38.920 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:38:38.920 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:38:38.920 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:38:38.920 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:38:38.920 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:38:38.920 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:38:38.920 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:38:38.920 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:38:38.920 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:38:38.920 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:38:38.920 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:38:38.920 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:38:38.921 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:38:38.921 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:38:38.921 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:38:38.921 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:38:38.921 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:38:38.945 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:38:38.945 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:38:38.946 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:38:38.946 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:38:39.079 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:38:39.081 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:38:39.081 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:38:39.081 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:38:39.081 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:38:39.081 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:38:39.081 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:38:39.081 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:38:39.081 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:38:39.081 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:38:39.081 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:38:39.082 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:38:39.082 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:38:39.084 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:38:39.084 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:38:39.084 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:38:39.142 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 17 2024-04-06T09:38:39.142 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 19 2024-04-06T09:38:39.142 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:38:39.142 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=77, tag_tid=12, entry_tid=1], [object_number=76, tag_tid=12, entry_tid=0], [object_number=79, tag_tid=11, entry_tid=17175], [object_number=78, tag_tid=11, entry_tid=17174]]], state=connected] 2024-04-06T09:38:39.142 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=69, tag_tid=11, entry_tid=9465], [object_number=68, tag_tid=11, entry_tid=9464], [object_number=71, tag_tid=11, entry_tid=9463], [object_number=70, tag_tid=11, entry_tid=9462]]], state=connected] 2024-04-06T09:38:39.146 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:38:39.148 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=77, tag_tid=12, entry_tid=1' 2024-04-06T09:38:39.149 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:38:39.149 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:38:39.149 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:38:39.149 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:38:39.149 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:38:39.149 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:38:39.149 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:38:39.149 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:38:39.149 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:38:39.149 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:38:39.149 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:38:39.149 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:38:39.150 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:38:39.150 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:38:39.150 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:38:39.196 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 17 2024-04-06T09:38:39.196 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 19 2024-04-06T09:38:39.196 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:38:39.196 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=77, tag_tid=12, entry_tid=1], [object_number=76, tag_tid=12, entry_tid=0], [object_number=79, tag_tid=11, entry_tid=17175], [object_number=78, tag_tid=11, entry_tid=17174]]], state=connected] 2024-04-06T09:38:39.196 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=69, tag_tid=11, entry_tid=9465], [object_number=68, tag_tid=11, entry_tid=9464], [object_number=71, tag_tid=11, entry_tid=9463], [object_number=70, tag_tid=11, entry_tid=9462]]], state=connected] 2024-04-06T09:38:39.199 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:38:39.201 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=69, tag_tid=11, entry_tid=9465' 2024-04-06T09:38:39.201 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=77, tag_tid=12, entry_tid=1' -a 'object_number=77, tag_tid=12, entry_tid=1' = 'object_number=69, tag_tid=11, entry_tid=9465' 2024-04-06T09:38:39.201 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=69, tag_tid=11, entry_tid=9465' '!=' 'object_number=68, tag_tid=11, entry_tid=9368' 2024-04-06T09:38:39.201 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:38:39.201 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=69, tag_tid=11, entry_tid=9465' = 'object_number=68, tag_tid=11, entry_tid=9368' 2024-04-06T09:38:39.201 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=69, tag_tid=11, entry_tid=9465' 2024-04-06T09:38:39.202 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=77, tag_tid=12, entry_tid=1' 2024-04-06T09:38:39.202 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:38:39.202 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:38:39.202 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=12 2024-04-06T09:38:39.203 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=69, tag_tid=11, entry_tid=9465' 2024-04-06T09:38:39.203 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:38:39.203 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:38:39.204 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=11 2024-04-06T09:38:39.205 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=77, tag_tid=12, entry_tid=1' 2024-04-06T09:38:39.205 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:38:39.205 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:38:39.206 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:38:39.206 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=69, tag_tid=11, entry_tid=9465' 2024-04-06T09:38:39.207 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:38:39.207 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:38:39.208 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=9465 2024-04-06T09:38:39.208 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 12 = 11 -a 1 -le 9465 2024-04-06T09:38:39.208 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:38:39.208 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:38:39.208 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:38:39.409 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:38:39.410 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:38:39.410 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:38:39.410 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:38:39.410 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:38:39.410 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:38:39.410 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:38:39.410 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:38:39.410 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:38:39.410 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:38:39.410 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:38:39.411 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:38:39.411 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:38:39.411 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:38:39.411 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:38:39.411 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:38:39.412 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:38:39.414 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:38:39.414 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:38:39.414 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:38:39.414 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:38:39.414 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:38:39.414 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:38:39.414 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:38:39.414 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:38:39.414 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:38:39.415 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:38:39.415 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:38:39.415 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:38:39.415 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:38:39.415 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:38:39.415 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:38:39.415 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:38:39.416 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:38:39.416 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:38:39.416 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:38:39.417 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:38:39.417 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:38:39.417 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:38:39.417 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:38:39.417 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:38:39.417 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:38:39.417 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:38:39.417 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:38:39.417 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:38:39.417 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:38:39.417 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:38:39.417 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:38:39.418 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:38:39.441 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:38:39.441 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:38:39.441 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:38:39.442 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:38:39.575 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:38:39.576 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:38:39.576 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:38:39.576 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:38:39.576 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:38:39.576 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:38:39.576 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:38:39.576 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:38:39.576 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:38:39.577 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:38:39.577 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:38:39.577 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:38:39.577 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:38:39.579 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:38:39.580 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:38:39.580 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:38:39.625 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 17 2024-04-06T09:38:39.625 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 19 2024-04-06T09:38:39.625 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:38:39.625 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=77, tag_tid=12, entry_tid=1], [object_number=76, tag_tid=12, entry_tid=0], [object_number=79, tag_tid=11, entry_tid=17175], [object_number=78, tag_tid=11, entry_tid=17174]]], state=connected] 2024-04-06T09:38:39.625 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=71, tag_tid=11, entry_tid=9555], [object_number=70, tag_tid=11, entry_tid=9554], [object_number=69, tag_tid=11, entry_tid=9553], [object_number=68, tag_tid=11, entry_tid=9552]]], state=connected] 2024-04-06T09:38:39.628 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:38:39.629 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=77, tag_tid=12, entry_tid=1' 2024-04-06T09:38:39.631 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:38:39.631 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:38:39.631 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:38:39.631 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:38:39.631 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:38:39.631 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:38:39.631 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:38:39.631 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:38:39.631 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:38:39.631 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:38:39.631 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:38:39.631 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:38:39.632 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:38:39.632 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:38:39.632 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:38:39.677 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 17 2024-04-06T09:38:39.678 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 19 2024-04-06T09:38:39.678 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:38:39.678 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=77, tag_tid=12, entry_tid=1], [object_number=76, tag_tid=12, entry_tid=0], [object_number=79, tag_tid=11, entry_tid=17175], [object_number=78, tag_tid=11, entry_tid=17174]]], state=connected] 2024-04-06T09:38:39.678 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=71, tag_tid=11, entry_tid=9555], [object_number=70, tag_tid=11, entry_tid=9554], [object_number=69, tag_tid=11, entry_tid=9553], [object_number=68, tag_tid=11, entry_tid=9552]]], state=connected] 2024-04-06T09:38:39.681 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:38:39.683 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=71, tag_tid=11, entry_tid=9555' 2024-04-06T09:38:39.683 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=77, tag_tid=12, entry_tid=1' -a 'object_number=77, tag_tid=12, entry_tid=1' = 'object_number=71, tag_tid=11, entry_tid=9555' 2024-04-06T09:38:39.683 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=71, tag_tid=11, entry_tid=9555' '!=' 'object_number=69, tag_tid=11, entry_tid=9465' 2024-04-06T09:38:39.683 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:38:39.683 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=71, tag_tid=11, entry_tid=9555' = 'object_number=69, tag_tid=11, entry_tid=9465' 2024-04-06T09:38:39.683 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=71, tag_tid=11, entry_tid=9555' 2024-04-06T09:38:39.684 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=77, tag_tid=12, entry_tid=1' 2024-04-06T09:38:39.684 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:38:39.684 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:38:39.684 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=12 2024-04-06T09:38:39.685 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=71, tag_tid=11, entry_tid=9555' 2024-04-06T09:38:39.685 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:38:39.685 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:38:39.686 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=11 2024-04-06T09:38:39.687 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=77, tag_tid=12, entry_tid=1' 2024-04-06T09:38:39.687 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:38:39.687 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:38:39.688 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:38:39.688 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=71, tag_tid=11, entry_tid=9555' 2024-04-06T09:38:39.688 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:38:39.688 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:38:39.690 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=9555 2024-04-06T09:38:39.690 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 12 = 11 -a 1 -le 9555 2024-04-06T09:38:39.690 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:38:39.690 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:38:39.690 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:38:39.892 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:38:39.892 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:38:39.892 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:38:39.892 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:38:39.892 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:38:39.892 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:38:39.892 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:38:39.892 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:38:39.892 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:38:39.892 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:38:39.893 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:38:39.893 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:38:39.893 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:38:39.893 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:38:39.893 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:38:39.893 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:38:39.894 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:38:39.896 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:38:39.896 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:38:39.896 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:38:39.896 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:38:39.896 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:38:39.896 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:38:39.896 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:38:39.897 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:38:39.897 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:38:39.897 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:38:39.897 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:38:39.897 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:38:39.897 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:38:39.897 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:38:39.897 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:38:39.897 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:38:39.898 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:38:39.898 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:38:39.898 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:38:39.898 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:38:39.898 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:38:39.898 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:38:39.898 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:38:39.898 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:38:39.899 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:38:39.899 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:38:39.899 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:38:39.899 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:38:39.899 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:38:39.899 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:38:39.899 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:38:39.899 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:38:39.923 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:38:39.924 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:38:39.924 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:38:39.924 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:38:40.056 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:38:40.057 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:38:40.057 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:38:40.057 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:38:40.057 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:38:40.057 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:38:40.057 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:38:40.057 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:38:40.057 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:38:40.057 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:38:40.058 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:38:40.058 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:38:40.058 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:38:40.060 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:38:40.061 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:38:40.061 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:38:40.112 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 17 2024-04-06T09:38:40.112 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 19 2024-04-06T09:38:40.112 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:38:40.112 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=77, tag_tid=12, entry_tid=1], [object_number=76, tag_tid=12, entry_tid=0], [object_number=79, tag_tid=11, entry_tid=17175], [object_number=78, tag_tid=11, entry_tid=17174]]], state=connected] 2024-04-06T09:38:40.112 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=71, tag_tid=11, entry_tid=9647], [object_number=70, tag_tid=11, entry_tid=9646], [object_number=69, tag_tid=11, entry_tid=9645], [object_number=68, tag_tid=11, entry_tid=9644]]], state=connected] 2024-04-06T09:38:40.115 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:38:40.117 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=77, tag_tid=12, entry_tid=1' 2024-04-06T09:38:40.118 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:38:40.118 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:38:40.118 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:38:40.118 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:38:40.118 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:38:40.118 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:38:40.118 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:38:40.118 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:38:40.118 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:38:40.119 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:38:40.119 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:38:40.119 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:38:40.119 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:38:40.119 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:38:40.119 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:38:40.165 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 17 2024-04-06T09:38:40.165 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 19 2024-04-06T09:38:40.165 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:38:40.165 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=77, tag_tid=12, entry_tid=1], [object_number=76, tag_tid=12, entry_tid=0], [object_number=79, tag_tid=11, entry_tid=17175], [object_number=78, tag_tid=11, entry_tid=17174]]], state=connected] 2024-04-06T09:38:40.165 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=71, tag_tid=11, entry_tid=9647], [object_number=70, tag_tid=11, entry_tid=9646], [object_number=69, tag_tid=11, entry_tid=9645], [object_number=68, tag_tid=11, entry_tid=9644]]], state=connected] 2024-04-06T09:38:40.168 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:38:40.170 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=71, tag_tid=11, entry_tid=9647' 2024-04-06T09:38:40.170 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=77, tag_tid=12, entry_tid=1' -a 'object_number=77, tag_tid=12, entry_tid=1' = 'object_number=71, tag_tid=11, entry_tid=9647' 2024-04-06T09:38:40.170 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=71, tag_tid=11, entry_tid=9647' '!=' 'object_number=71, tag_tid=11, entry_tid=9555' 2024-04-06T09:38:40.170 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:38:40.170 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=71, tag_tid=11, entry_tid=9647' = 'object_number=71, tag_tid=11, entry_tid=9555' 2024-04-06T09:38:40.170 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=71, tag_tid=11, entry_tid=9647' 2024-04-06T09:38:40.171 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=77, tag_tid=12, entry_tid=1' 2024-04-06T09:38:40.171 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:38:40.171 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:38:40.171 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=12 2024-04-06T09:38:40.172 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=71, tag_tid=11, entry_tid=9647' 2024-04-06T09:38:40.172 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:38:40.172 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:38:40.173 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=11 2024-04-06T09:38:40.174 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=77, tag_tid=12, entry_tid=1' 2024-04-06T09:38:40.174 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:38:40.174 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:38:40.175 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:38:40.175 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=71, tag_tid=11, entry_tid=9647' 2024-04-06T09:38:40.176 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:38:40.176 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:38:40.177 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=9647 2024-04-06T09:38:40.177 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 12 = 11 -a 1 -le 9647 2024-04-06T09:38:40.177 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:38:40.177 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:38:40.177 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:38:40.379 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:38:40.379 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:38:40.379 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:38:40.379 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:38:40.379 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:38:40.379 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:38:40.379 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:38:40.379 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:38:40.379 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:38:40.379 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:38:40.379 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:38:40.379 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:38:40.380 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:38:40.380 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:38:40.380 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:38:40.380 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:38:40.381 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:38:40.383 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:38:40.383 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:38:40.383 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:38:40.383 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:38:40.383 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:38:40.383 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:38:40.384 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:38:40.384 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:38:40.384 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:38:40.384 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:38:40.384 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:38:40.384 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:38:40.384 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:38:40.384 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:38:40.384 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:38:40.384 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:38:40.385 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:38:40.385 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:38:40.385 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:38:40.385 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:38:40.385 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:38:40.385 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:38:40.385 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:38:40.385 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:38:40.385 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:38:40.386 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:38:40.386 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:38:40.386 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:38:40.386 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:38:40.386 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:38:40.386 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:38:40.386 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:38:40.411 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:38:40.411 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:38:40.412 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:38:40.412 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:38:40.543 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:38:40.543 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:38:40.543 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:38:40.544 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:38:40.544 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:38:40.544 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:38:40.544 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:38:40.544 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:38:40.544 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:38:40.544 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:38:40.545 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:38:40.545 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:38:40.545 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:38:40.548 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:38:40.548 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:38:40.548 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:38:40.603 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 17 2024-04-06T09:38:40.603 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 19 2024-04-06T09:38:40.603 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:38:40.603 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=77, tag_tid=12, entry_tid=1], [object_number=76, tag_tid=12, entry_tid=0], [object_number=79, tag_tid=11, entry_tid=17175], [object_number=78, tag_tid=11, entry_tid=17174]]], state=connected] 2024-04-06T09:38:40.603 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=69, tag_tid=11, entry_tid=9737], [object_number=68, tag_tid=11, entry_tid=9736], [object_number=71, tag_tid=11, entry_tid=9735], [object_number=70, tag_tid=11, entry_tid=9734]]], state=connected] 2024-04-06T09:38:40.606 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:38:40.608 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=77, tag_tid=12, entry_tid=1' 2024-04-06T09:38:40.609 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:38:40.609 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:38:40.609 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:38:40.609 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:38:40.609 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:38:40.609 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:38:40.609 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:38:40.610 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:38:40.610 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:38:40.610 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:38:40.610 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:38:40.610 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:38:40.610 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:38:40.611 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:38:40.611 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:38:40.658 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 17 2024-04-06T09:38:40.658 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 19 2024-04-06T09:38:40.658 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:38:40.659 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=77, tag_tid=12, entry_tid=1], [object_number=76, tag_tid=12, entry_tid=0], [object_number=79, tag_tid=11, entry_tid=17175], [object_number=78, tag_tid=11, entry_tid=17174]]], state=connected] 2024-04-06T09:38:40.659 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=69, tag_tid=11, entry_tid=9737], [object_number=68, tag_tid=11, entry_tid=9736], [object_number=71, tag_tid=11, entry_tid=9735], [object_number=70, tag_tid=11, entry_tid=9734]]], state=connected] 2024-04-06T09:38:40.662 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:38:40.664 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=69, tag_tid=11, entry_tid=9737' 2024-04-06T09:38:40.664 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=77, tag_tid=12, entry_tid=1' -a 'object_number=77, tag_tid=12, entry_tid=1' = 'object_number=69, tag_tid=11, entry_tid=9737' 2024-04-06T09:38:40.664 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=69, tag_tid=11, entry_tid=9737' '!=' 'object_number=71, tag_tid=11, entry_tid=9647' 2024-04-06T09:38:40.664 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:38:40.664 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=69, tag_tid=11, entry_tid=9737' = 'object_number=71, tag_tid=11, entry_tid=9647' 2024-04-06T09:38:40.664 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=69, tag_tid=11, entry_tid=9737' 2024-04-06T09:38:40.664 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=77, tag_tid=12, entry_tid=1' 2024-04-06T09:38:40.664 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:38:40.664 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:38:40.665 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=12 2024-04-06T09:38:40.666 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=69, tag_tid=11, entry_tid=9737' 2024-04-06T09:38:40.666 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:38:40.666 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:38:40.667 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=11 2024-04-06T09:38:40.667 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=77, tag_tid=12, entry_tid=1' 2024-04-06T09:38:40.668 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:38:40.668 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:38:40.669 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:38:40.669 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=69, tag_tid=11, entry_tid=9737' 2024-04-06T09:38:40.669 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:38:40.669 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:38:40.671 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=9737 2024-04-06T09:38:40.671 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 12 = 11 -a 1 -le 9737 2024-04-06T09:38:40.671 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:38:40.671 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:38:40.671 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:38:40.872 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:38:40.872 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:38:40.872 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:38:40.872 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:38:40.872 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:38:40.873 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:38:40.873 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:38:40.873 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:38:40.873 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:38:40.873 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:38:40.873 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:38:40.873 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:38:40.873 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:38:40.873 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:38:40.873 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:38:40.874 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:38:40.874 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:38:40.877 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:38:40.877 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:38:40.877 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:38:40.877 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:38:40.878 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:38:40.878 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:38:40.878 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:38:40.878 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:38:40.878 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:38:40.878 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:38:40.878 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:38:40.878 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:38:40.878 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:38:40.878 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:38:40.878 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:38:40.878 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:38:40.878 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:38:40.878 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:38:40.879 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:38:40.879 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:38:40.879 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:38:40.879 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:38:40.879 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:38:40.879 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:38:40.879 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:38:40.879 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:38:40.879 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:38:40.879 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:38:40.879 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:38:40.879 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:38:40.879 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:38:40.879 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:38:40.904 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:38:40.904 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:38:40.904 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:38:40.904 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:38:41.055 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:38:41.056 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:38:41.056 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:38:41.056 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:38:41.056 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:38:41.057 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:38:41.057 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:38:41.057 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:38:41.057 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:38:41.057 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:38:41.057 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:38:41.057 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:38:41.057 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:38:41.060 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:38:41.060 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:38:41.060 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:38:41.111 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 17 2024-04-06T09:38:41.111 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 19 2024-04-06T09:38:41.111 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:38:41.111 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=77, tag_tid=12, entry_tid=1], [object_number=76, tag_tid=12, entry_tid=0], [object_number=79, tag_tid=11, entry_tid=17175], [object_number=78, tag_tid=11, entry_tid=17174]]], state=connected] 2024-04-06T09:38:41.112 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=69, tag_tid=11, entry_tid=9821], [object_number=68, tag_tid=11, entry_tid=9820], [object_number=71, tag_tid=11, entry_tid=9819], [object_number=70, tag_tid=11, entry_tid=9818]]], state=connected] 2024-04-06T09:38:41.115 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:38:41.116 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=77, tag_tid=12, entry_tid=1' 2024-04-06T09:38:41.118 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:38:41.118 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:38:41.118 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:38:41.118 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:38:41.118 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:38:41.118 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:38:41.118 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:38:41.118 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:38:41.118 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:38:41.118 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:38:41.118 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:38:41.119 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:38:41.119 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:38:41.119 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:38:41.119 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:38:41.165 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 17 2024-04-06T09:38:41.165 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 19 2024-04-06T09:38:41.165 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:38:41.165 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=77, tag_tid=12, entry_tid=1], [object_number=76, tag_tid=12, entry_tid=0], [object_number=79, tag_tid=11, entry_tid=17175], [object_number=78, tag_tid=11, entry_tid=17174]]], state=connected] 2024-04-06T09:38:41.166 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=69, tag_tid=11, entry_tid=9821], [object_number=68, tag_tid=11, entry_tid=9820], [object_number=71, tag_tid=11, entry_tid=9819], [object_number=70, tag_tid=11, entry_tid=9818]]], state=connected] 2024-04-06T09:38:41.169 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:38:41.171 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=69, tag_tid=11, entry_tid=9821' 2024-04-06T09:38:41.171 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=77, tag_tid=12, entry_tid=1' -a 'object_number=77, tag_tid=12, entry_tid=1' = 'object_number=69, tag_tid=11, entry_tid=9821' 2024-04-06T09:38:41.171 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=69, tag_tid=11, entry_tid=9821' '!=' 'object_number=69, tag_tid=11, entry_tid=9737' 2024-04-06T09:38:41.171 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:38:41.171 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=69, tag_tid=11, entry_tid=9821' = 'object_number=69, tag_tid=11, entry_tid=9737' 2024-04-06T09:38:41.171 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=69, tag_tid=11, entry_tid=9821' 2024-04-06T09:38:41.171 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=77, tag_tid=12, entry_tid=1' 2024-04-06T09:38:41.171 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:38:41.171 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:38:41.172 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=12 2024-04-06T09:38:41.172 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=69, tag_tid=11, entry_tid=9821' 2024-04-06T09:38:41.173 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:38:41.173 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:38:41.173 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=11 2024-04-06T09:38:41.174 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=77, tag_tid=12, entry_tid=1' 2024-04-06T09:38:41.174 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:38:41.174 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:38:41.175 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:38:41.176 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=69, tag_tid=11, entry_tid=9821' 2024-04-06T09:38:41.176 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:38:41.176 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:38:41.177 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=9821 2024-04-06T09:38:41.177 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 12 = 11 -a 1 -le 9821 2024-04-06T09:38:41.177 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:38:41.178 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:38:41.178 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:38:41.379 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:38:41.379 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:38:41.379 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:38:41.380 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:38:41.380 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:38:41.380 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:38:41.380 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:38:41.380 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:38:41.380 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:38:41.380 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:38:41.380 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:38:41.380 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:38:41.380 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:38:41.380 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:38:41.380 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:38:41.380 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:38:41.382 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:38:41.382 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:38:41.382 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:38:41.382 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:38:41.382 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:38:41.382 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:38:41.382 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:38:41.382 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:38:41.382 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:38:41.382 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:38:41.382 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:38:41.383 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:38:41.383 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:38:41.383 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:38:41.383 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:38:41.383 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:38:41.383 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:38:41.384 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:38:41.384 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:38:41.384 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:38:41.384 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:38:41.384 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:38:41.385 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:38:41.385 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:38:41.385 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:38:41.385 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:38:41.385 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:38:41.385 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:38:41.385 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:38:41.385 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:38:41.385 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:38:41.385 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:38:41.385 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:38:41.409 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:38:41.409 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:38:41.409 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:38:41.409 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:38:41.546 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:38:41.547 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:38:41.547 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:38:41.547 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:38:41.547 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:38:41.548 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:38:41.548 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:38:41.548 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:38:41.548 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:38:41.548 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:38:41.549 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:38:41.549 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:38:41.549 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:38:41.551 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:38:41.551 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:38:41.551 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:38:41.598 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 17 2024-04-06T09:38:41.598 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 19 2024-04-06T09:38:41.598 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:38:41.598 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=77, tag_tid=12, entry_tid=1], [object_number=76, tag_tid=12, entry_tid=0], [object_number=79, tag_tid=11, entry_tid=17175], [object_number=78, tag_tid=11, entry_tid=17174]]], state=connected] 2024-04-06T09:38:41.598 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=69, tag_tid=11, entry_tid=9901], [object_number=68, tag_tid=11, entry_tid=9900], [object_number=71, tag_tid=11, entry_tid=9899], [object_number=70, tag_tid=11, entry_tid=9898]]], state=connected] 2024-04-06T09:38:41.602 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:38:41.603 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=77, tag_tid=12, entry_tid=1' 2024-04-06T09:38:41.604 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:38:41.605 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:38:41.605 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:38:41.605 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:38:41.605 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:38:41.605 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:38:41.605 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:38:41.605 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:38:41.605 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:38:41.605 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:38:41.605 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:38:41.605 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:38:41.606 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:38:41.606 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:38:41.606 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:38:41.863 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 17 2024-04-06T09:38:41.863 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 19 2024-04-06T09:38:41.863 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:38:41.863 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=77, tag_tid=12, entry_tid=1], [object_number=76, tag_tid=12, entry_tid=0], [object_number=79, tag_tid=11, entry_tid=17175], [object_number=78, tag_tid=11, entry_tid=17174]]], state=connected] 2024-04-06T09:38:41.863 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=69, tag_tid=11, entry_tid=9901], [object_number=68, tag_tid=11, entry_tid=9900], [object_number=71, tag_tid=11, entry_tid=9899], [object_number=70, tag_tid=11, entry_tid=9898]]], state=connected] 2024-04-06T09:38:41.867 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:38:41.869 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=69, tag_tid=11, entry_tid=9901' 2024-04-06T09:38:41.869 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=77, tag_tid=12, entry_tid=1' -a 'object_number=77, tag_tid=12, entry_tid=1' = 'object_number=69, tag_tid=11, entry_tid=9901' 2024-04-06T09:38:41.869 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=69, tag_tid=11, entry_tid=9901' '!=' 'object_number=69, tag_tid=11, entry_tid=9821' 2024-04-06T09:38:41.869 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:38:41.869 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=69, tag_tid=11, entry_tid=9901' = 'object_number=69, tag_tid=11, entry_tid=9821' 2024-04-06T09:38:41.870 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=69, tag_tid=11, entry_tid=9901' 2024-04-06T09:38:41.870 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=77, tag_tid=12, entry_tid=1' 2024-04-06T09:38:41.870 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:38:41.870 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:38:41.871 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=12 2024-04-06T09:38:41.871 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=69, tag_tid=11, entry_tid=9901' 2024-04-06T09:38:41.872 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:38:41.872 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:38:41.873 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=11 2024-04-06T09:38:41.874 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=77, tag_tid=12, entry_tid=1' 2024-04-06T09:38:41.874 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:38:41.875 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:38:41.876 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:38:41.876 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=69, tag_tid=11, entry_tid=9901' 2024-04-06T09:38:41.876 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:38:41.876 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:38:41.877 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=9901 2024-04-06T09:38:41.877 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 12 = 11 -a 1 -le 9901 2024-04-06T09:38:41.877 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:38:41.877 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:38:41.877 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:38:42.079 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:38:42.079 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:38:42.079 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:38:42.079 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:38:42.079 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:38:42.079 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:38:42.079 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:38:42.080 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:38:42.080 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:38:42.080 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:38:42.080 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:38:42.080 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:38:42.080 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:38:42.080 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:38:42.080 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:38:42.080 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:38:42.081 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:38:42.082 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:38:42.082 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:38:42.082 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:38:42.082 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:38:42.082 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:38:42.083 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:38:42.083 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:38:42.083 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:38:42.083 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:38:42.083 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:38:42.083 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:38:42.083 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:38:42.083 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:38:42.083 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:38:42.083 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:38:42.083 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:38:42.085 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:38:42.085 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:38:42.085 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:38:42.085 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:38:42.085 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:38:42.085 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:38:42.085 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:38:42.085 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:38:42.085 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:38:42.085 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:38:42.086 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:38:42.086 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:38:42.086 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:38:42.086 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:38:42.086 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:38:42.086 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:38:42.109 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:38:42.109 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:38:42.109 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:38:42.109 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:38:42.242 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:38:42.244 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:38:42.244 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:38:42.244 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:38:42.244 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:38:42.244 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:38:42.244 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:38:42.244 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:38:42.244 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:38:42.244 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:38:42.245 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:38:42.245 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:38:42.245 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:38:42.247 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:38:42.248 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:38:42.248 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:38:42.503 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 17 2024-04-06T09:38:42.503 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 19 2024-04-06T09:38:42.503 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:38:42.503 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=77, tag_tid=12, entry_tid=1], [object_number=76, tag_tid=12, entry_tid=0], [object_number=79, tag_tid=11, entry_tid=17175], [object_number=78, tag_tid=11, entry_tid=17174]]], state=connected] 2024-04-06T09:38:42.503 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=69, tag_tid=11, entry_tid=10049], [object_number=68, tag_tid=11, entry_tid=10048], [object_number=71, tag_tid=11, entry_tid=10047], [object_number=70, tag_tid=11, entry_tid=10046]]], state=connected] 2024-04-06T09:38:42.509 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:38:42.511 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=77, tag_tid=12, entry_tid=1' 2024-04-06T09:38:42.513 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:38:42.513 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:38:42.513 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:38:42.513 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:38:42.513 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:38:42.513 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:38:42.513 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:38:42.514 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:38:42.514 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:38:42.514 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:38:42.514 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:38:42.514 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:38:42.515 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:38:42.515 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:38:42.515 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:38:42.572 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 17 2024-04-06T09:38:42.572 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 19 2024-04-06T09:38:42.572 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:38:42.572 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=77, tag_tid=12, entry_tid=1], [object_number=76, tag_tid=12, entry_tid=0], [object_number=79, tag_tid=11, entry_tid=17175], [object_number=78, tag_tid=11, entry_tid=17174]]], state=connected] 2024-04-06T09:38:42.572 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=69, tag_tid=11, entry_tid=10049], [object_number=68, tag_tid=11, entry_tid=10048], [object_number=71, tag_tid=11, entry_tid=10047], [object_number=70, tag_tid=11, entry_tid=10046]]], state=connected] 2024-04-06T09:38:42.575 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:38:42.578 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=69, tag_tid=11, entry_tid=10049' 2024-04-06T09:38:42.578 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=77, tag_tid=12, entry_tid=1' -a 'object_number=77, tag_tid=12, entry_tid=1' = 'object_number=69, tag_tid=11, entry_tid=10049' 2024-04-06T09:38:42.578 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=69, tag_tid=11, entry_tid=10049' '!=' 'object_number=69, tag_tid=11, entry_tid=9901' 2024-04-06T09:38:42.578 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:38:42.578 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=69, tag_tid=11, entry_tid=10049' = 'object_number=69, tag_tid=11, entry_tid=9901' 2024-04-06T09:38:42.578 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=69, tag_tid=11, entry_tid=10049' 2024-04-06T09:38:42.578 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=77, tag_tid=12, entry_tid=1' 2024-04-06T09:38:42.578 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:38:42.578 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:38:42.579 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=12 2024-04-06T09:38:42.580 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=69, tag_tid=11, entry_tid=10049' 2024-04-06T09:38:42.580 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:38:42.580 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:38:42.580 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=11 2024-04-06T09:38:42.581 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=77, tag_tid=12, entry_tid=1' 2024-04-06T09:38:42.581 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:38:42.581 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:38:42.582 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:38:42.583 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=69, tag_tid=11, entry_tid=10049' 2024-04-06T09:38:42.583 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:38:42.583 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:38:42.584 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=10049 2024-04-06T09:38:42.585 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 12 = 11 -a 1 -le 10049 2024-04-06T09:38:42.585 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:38:42.585 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:38:42.585 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:38:42.786 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:38:42.787 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:38:42.787 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:38:42.787 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:38:42.787 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:38:42.787 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:38:42.787 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:38:42.787 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:38:42.787 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:38:42.787 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:38:42.787 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:38:42.788 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:38:42.788 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:38:42.788 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:38:42.788 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:38:42.788 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:38:42.788 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:38:42.791 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:38:42.791 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:38:42.791 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:38:42.791 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:38:42.791 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:38:42.791 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:38:42.791 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:38:42.791 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:38:42.791 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:38:42.791 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:38:42.791 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:38:42.791 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:38:42.792 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:38:42.792 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:38:42.792 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:38:42.792 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:38:42.792 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:38:42.792 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:38:42.792 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:38:42.792 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:38:42.792 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:38:42.792 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:38:42.792 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:38:42.792 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:38:42.792 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:38:42.792 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:38:42.792 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:38:42.793 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:38:42.793 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:38:42.793 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:38:42.793 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:38:42.793 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:38:42.817 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:38:42.817 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:38:42.817 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:38:42.817 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:38:42.946 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:38:42.947 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:38:42.947 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:38:42.947 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:38:42.947 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:38:42.947 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:38:42.948 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:38:42.948 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:38:42.948 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:38:42.948 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:38:42.948 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:38:42.948 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:38:42.948 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:38:42.949 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:38:42.950 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:38:42.950 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:38:42.996 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 17 2024-04-06T09:38:42.996 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 19 2024-04-06T09:38:42.996 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:38:42.996 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=77, tag_tid=12, entry_tid=1], [object_number=76, tag_tid=12, entry_tid=0], [object_number=79, tag_tid=11, entry_tid=17175], [object_number=78, tag_tid=11, entry_tid=17174]]], state=connected] 2024-04-06T09:38:42.997 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=70, tag_tid=11, entry_tid=10202], [object_number=69, tag_tid=11, entry_tid=10201], [object_number=68, tag_tid=11, entry_tid=10200], [object_number=71, tag_tid=11, entry_tid=10199]]], state=connected] 2024-04-06T09:38:43.000 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:38:43.001 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=77, tag_tid=12, entry_tid=1' 2024-04-06T09:38:43.002 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:38:43.002 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:38:43.002 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:38:43.002 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:38:43.003 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:38:43.003 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:38:43.003 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:38:43.003 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:38:43.003 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:38:43.003 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:38:43.003 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:38:43.003 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:38:43.003 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:38:43.004 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:38:43.004 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:38:43.049 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 17 2024-04-06T09:38:43.049 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 19 2024-04-06T09:38:43.049 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:38:43.050 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=77, tag_tid=12, entry_tid=1], [object_number=76, tag_tid=12, entry_tid=0], [object_number=79, tag_tid=11, entry_tid=17175], [object_number=78, tag_tid=11, entry_tid=17174]]], state=connected] 2024-04-06T09:38:43.050 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=70, tag_tid=11, entry_tid=10202], [object_number=69, tag_tid=11, entry_tid=10201], [object_number=68, tag_tid=11, entry_tid=10200], [object_number=71, tag_tid=11, entry_tid=10199]]], state=connected] 2024-04-06T09:38:43.053 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:38:43.055 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=70, tag_tid=11, entry_tid=10202' 2024-04-06T09:38:43.055 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=77, tag_tid=12, entry_tid=1' -a 'object_number=77, tag_tid=12, entry_tid=1' = 'object_number=70, tag_tid=11, entry_tid=10202' 2024-04-06T09:38:43.055 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=70, tag_tid=11, entry_tid=10202' '!=' 'object_number=69, tag_tid=11, entry_tid=10049' 2024-04-06T09:38:43.055 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:38:43.055 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=70, tag_tid=11, entry_tid=10202' = 'object_number=69, tag_tid=11, entry_tid=10049' 2024-04-06T09:38:43.055 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=70, tag_tid=11, entry_tid=10202' 2024-04-06T09:38:43.055 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=77, tag_tid=12, entry_tid=1' 2024-04-06T09:38:43.055 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:38:43.055 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:38:43.056 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=12 2024-04-06T09:38:43.057 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=70, tag_tid=11, entry_tid=10202' 2024-04-06T09:38:43.057 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:38:43.057 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:38:43.058 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=11 2024-04-06T09:38:43.059 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=77, tag_tid=12, entry_tid=1' 2024-04-06T09:38:43.059 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:38:43.059 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:38:43.060 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:38:43.060 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=70, tag_tid=11, entry_tid=10202' 2024-04-06T09:38:43.060 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:38:43.060 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:38:43.061 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=10202 2024-04-06T09:38:43.061 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 12 = 11 -a 1 -le 10202 2024-04-06T09:38:43.061 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:38:43.061 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:38:43.061 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:38:43.263 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:38:43.263 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:38:43.264 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:38:43.264 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:38:43.264 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:38:43.264 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:38:43.264 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:38:43.264 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:38:43.264 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:38:43.264 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:38:43.264 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:38:43.264 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:38:43.264 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:38:43.264 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:38:43.264 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:38:43.265 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:38:43.265 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:38:43.268 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:38:43.269 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:38:43.269 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:38:43.269 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:38:43.269 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:38:43.269 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:38:43.269 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:38:43.269 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:38:43.269 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:38:43.269 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:38:43.269 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:38:43.269 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:38:43.269 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:38:43.269 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:38:43.269 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:38:43.270 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:38:43.270 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:38:43.270 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:38:43.270 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:38:43.270 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:38:43.270 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:38:43.270 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:38:43.270 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:38:43.270 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:38:43.270 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:38:43.270 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:38:43.270 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:38:43.270 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:38:43.270 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:38:43.271 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:38:43.271 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:38:43.271 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:38:43.293 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:38:43.294 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:38:43.294 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:38:43.294 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:38:43.419 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:38:43.420 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:38:43.421 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:38:43.421 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:38:43.421 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:38:43.421 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:38:43.421 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:38:43.421 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:38:43.421 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:38:43.421 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:38:43.422 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:38:43.422 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:38:43.422 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:38:43.423 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:38:43.424 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:38:43.424 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:38:43.474 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 17 2024-04-06T09:38:43.475 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 19 2024-04-06T09:38:43.475 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:38:43.475 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=77, tag_tid=12, entry_tid=1], [object_number=76, tag_tid=12, entry_tid=0], [object_number=79, tag_tid=11, entry_tid=17175], [object_number=78, tag_tid=11, entry_tid=17174]]], state=connected] 2024-04-06T09:38:43.475 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=69, tag_tid=11, entry_tid=10301], [object_number=68, tag_tid=11, entry_tid=10300], [object_number=71, tag_tid=11, entry_tid=10299], [object_number=70, tag_tid=11, entry_tid=10298]]], state=connected] 2024-04-06T09:38:43.478 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:38:43.479 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=77, tag_tid=12, entry_tid=1' 2024-04-06T09:38:43.480 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:38:43.480 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:38:43.481 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:38:43.481 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:38:43.481 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:38:43.481 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:38:43.481 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:38:43.481 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:38:43.481 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:38:43.481 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:38:43.481 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:38:43.481 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:38:43.482 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:38:43.482 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:38:43.482 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:38:43.529 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 17 2024-04-06T09:38:43.529 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 19 2024-04-06T09:38:43.529 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:38:43.529 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=77, tag_tid=12, entry_tid=1], [object_number=76, tag_tid=12, entry_tid=0], [object_number=79, tag_tid=11, entry_tid=17175], [object_number=78, tag_tid=11, entry_tid=17174]]], state=connected] 2024-04-06T09:38:43.529 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=69, tag_tid=11, entry_tid=10301], [object_number=68, tag_tid=11, entry_tid=10300], [object_number=71, tag_tid=11, entry_tid=10299], [object_number=70, tag_tid=11, entry_tid=10298]]], state=connected] 2024-04-06T09:38:43.532 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:38:43.534 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=69, tag_tid=11, entry_tid=10301' 2024-04-06T09:38:43.534 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=77, tag_tid=12, entry_tid=1' -a 'object_number=77, tag_tid=12, entry_tid=1' = 'object_number=69, tag_tid=11, entry_tid=10301' 2024-04-06T09:38:43.534 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=69, tag_tid=11, entry_tid=10301' '!=' 'object_number=70, tag_tid=11, entry_tid=10202' 2024-04-06T09:38:43.534 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:38:43.535 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=69, tag_tid=11, entry_tid=10301' = 'object_number=70, tag_tid=11, entry_tid=10202' 2024-04-06T09:38:43.535 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=69, tag_tid=11, entry_tid=10301' 2024-04-06T09:38:43.535 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=77, tag_tid=12, entry_tid=1' 2024-04-06T09:38:43.535 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:38:43.535 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:38:43.536 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=12 2024-04-06T09:38:43.536 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=69, tag_tid=11, entry_tid=10301' 2024-04-06T09:38:43.536 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:38:43.537 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:38:43.537 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=11 2024-04-06T09:38:43.538 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=77, tag_tid=12, entry_tid=1' 2024-04-06T09:38:43.538 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:38:43.538 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:38:43.539 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:38:43.540 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=69, tag_tid=11, entry_tid=10301' 2024-04-06T09:38:43.540 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:38:43.540 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:38:43.541 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=10301 2024-04-06T09:38:43.541 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 12 = 11 -a 1 -le 10301 2024-04-06T09:38:43.542 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:38:43.542 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:38:43.542 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:38:43.743 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:38:43.744 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:38:43.744 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:38:43.744 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:38:43.744 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:38:43.744 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:38:43.744 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:38:43.744 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:38:43.744 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:38:43.744 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:38:43.744 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:38:43.744 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:38:43.744 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:38:43.744 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:38:43.744 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:38:43.745 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:38:43.745 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:38:43.748 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:38:43.748 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:38:43.749 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:38:43.749 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:38:43.749 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:38:43.749 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:38:43.749 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:38:43.749 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:38:43.749 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:38:43.749 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:38:43.749 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:38:43.749 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:38:43.749 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:38:43.749 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:38:43.749 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:38:43.749 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:38:43.749 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:38:43.750 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:38:43.750 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:38:43.750 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:38:43.750 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:38:43.750 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:38:43.750 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:38:43.750 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:38:43.750 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:38:43.750 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:38:43.751 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:38:43.751 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:38:43.751 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:38:43.751 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:38:43.751 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:38:43.751 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:38:43.774 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:38:43.774 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:38:43.774 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:38:43.774 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:38:43.904 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:38:43.906 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:38:43.906 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:38:43.906 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:38:43.906 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:38:43.906 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:38:43.906 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:38:43.906 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:38:43.906 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:38:43.906 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:38:43.907 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:38:43.907 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:38:43.907 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:38:43.908 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:38:43.908 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:38:43.908 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:38:43.960 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 17 2024-04-06T09:38:43.960 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 19 2024-04-06T09:38:43.960 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:38:43.960 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=77, tag_tid=12, entry_tid=1], [object_number=76, tag_tid=12, entry_tid=0], [object_number=79, tag_tid=11, entry_tid=17175], [object_number=78, tag_tid=11, entry_tid=17174]]], state=connected] 2024-04-06T09:38:43.961 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=69, tag_tid=11, entry_tid=10401], [object_number=68, tag_tid=11, entry_tid=10400], [object_number=71, tag_tid=11, entry_tid=10399], [object_number=70, tag_tid=11, entry_tid=10398]]], state=connected] 2024-04-06T09:38:43.964 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:38:43.965 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=77, tag_tid=12, entry_tid=1' 2024-04-06T09:38:43.966 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:38:43.966 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:38:43.966 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:38:43.967 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:38:43.967 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:38:43.967 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:38:43.967 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:38:43.967 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:38:43.967 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:38:43.967 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:38:43.967 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:38:43.967 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:38:43.968 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:38:43.968 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:38:43.968 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:38:44.013 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 17 2024-04-06T09:38:44.014 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 19 2024-04-06T09:38:44.014 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:38:44.014 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=77, tag_tid=12, entry_tid=1], [object_number=76, tag_tid=12, entry_tid=0], [object_number=79, tag_tid=11, entry_tid=17175], [object_number=78, tag_tid=11, entry_tid=17174]]], state=connected] 2024-04-06T09:38:44.014 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=69, tag_tid=11, entry_tid=10401], [object_number=68, tag_tid=11, entry_tid=10400], [object_number=71, tag_tid=11, entry_tid=10399], [object_number=70, tag_tid=11, entry_tid=10398]]], state=connected] 2024-04-06T09:38:44.017 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:38:44.019 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=69, tag_tid=11, entry_tid=10401' 2024-04-06T09:38:44.019 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=77, tag_tid=12, entry_tid=1' -a 'object_number=77, tag_tid=12, entry_tid=1' = 'object_number=69, tag_tid=11, entry_tid=10401' 2024-04-06T09:38:44.019 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=69, tag_tid=11, entry_tid=10401' '!=' 'object_number=69, tag_tid=11, entry_tid=10301' 2024-04-06T09:38:44.019 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:38:44.020 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=69, tag_tid=11, entry_tid=10401' = 'object_number=69, tag_tid=11, entry_tid=10301' 2024-04-06T09:38:44.020 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=69, tag_tid=11, entry_tid=10401' 2024-04-06T09:38:44.020 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=77, tag_tid=12, entry_tid=1' 2024-04-06T09:38:44.020 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:38:44.020 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:38:44.020 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=12 2024-04-06T09:38:44.021 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=69, tag_tid=11, entry_tid=10401' 2024-04-06T09:38:44.021 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:38:44.021 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:38:44.022 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=11 2024-04-06T09:38:44.023 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=77, tag_tid=12, entry_tid=1' 2024-04-06T09:38:44.023 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:38:44.023 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:38:44.024 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:38:44.024 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=69, tag_tid=11, entry_tid=10401' 2024-04-06T09:38:44.024 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:38:44.025 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:38:44.026 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=10401 2024-04-06T09:38:44.026 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 12 = 11 -a 1 -le 10401 2024-04-06T09:38:44.026 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:38:44.026 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:38:44.026 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:38:44.228 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:38:44.228 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:38:44.228 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:38:44.228 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:38:44.228 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:38:44.228 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:38:44.228 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:38:44.228 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:38:44.228 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:38:44.228 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:38:44.228 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:38:44.229 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:38:44.229 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:38:44.229 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:38:44.229 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:38:44.229 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:38:44.229 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:38:44.230 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:38:44.231 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:38:44.231 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:38:44.231 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:38:44.231 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:38:44.231 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:38:44.231 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:38:44.231 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:38:44.231 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:38:44.231 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:38:44.231 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:38:44.231 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:38:44.231 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:38:44.231 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:38:44.232 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:38:44.232 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:38:44.233 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:38:44.233 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:38:44.233 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:38:44.234 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:38:44.234 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:38:44.234 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:38:44.234 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:38:44.234 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:38:44.234 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:38:44.234 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:38:44.234 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:38:44.234 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:38:44.234 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:38:44.234 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:38:44.234 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:38:44.234 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:38:44.258 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:38:44.259 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:38:44.259 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:38:44.259 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:38:44.388 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:38:44.389 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:38:44.389 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:38:44.389 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:38:44.389 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:38:44.389 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:38:44.389 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:38:44.389 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:38:44.390 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:38:44.390 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:38:44.390 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:38:44.390 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:38:44.390 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:38:44.393 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:38:44.393 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:38:44.393 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:38:44.450 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 17 2024-04-06T09:38:44.451 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 19 2024-04-06T09:38:44.451 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:38:44.451 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=77, tag_tid=12, entry_tid=1], [object_number=76, tag_tid=12, entry_tid=0], [object_number=79, tag_tid=11, entry_tid=17175], [object_number=78, tag_tid=11, entry_tid=17174]]], state=connected] 2024-04-06T09:38:44.451 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=70, tag_tid=11, entry_tid=10502], [object_number=69, tag_tid=11, entry_tid=10501], [object_number=68, tag_tid=11, entry_tid=10500], [object_number=71, tag_tid=11, entry_tid=10499]]], state=connected] 2024-04-06T09:38:44.454 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:38:44.456 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=77, tag_tid=12, entry_tid=1' 2024-04-06T09:38:44.457 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:38:44.457 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:38:44.457 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:38:44.457 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:38:44.457 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:38:44.457 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:38:44.457 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:38:44.457 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:38:44.457 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:38:44.457 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:38:44.457 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:38:44.458 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:38:44.458 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:38:44.459 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:38:44.459 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:38:44.506 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 17 2024-04-06T09:38:44.506 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 19 2024-04-06T09:38:44.506 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:38:44.506 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=77, tag_tid=12, entry_tid=1], [object_number=76, tag_tid=12, entry_tid=0], [object_number=79, tag_tid=11, entry_tid=17175], [object_number=78, tag_tid=11, entry_tid=17174]]], state=connected] 2024-04-06T09:38:44.506 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=70, tag_tid=11, entry_tid=10502], [object_number=69, tag_tid=11, entry_tid=10501], [object_number=68, tag_tid=11, entry_tid=10500], [object_number=71, tag_tid=11, entry_tid=10499]]], state=connected] 2024-04-06T09:38:44.510 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:38:44.511 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=70, tag_tid=11, entry_tid=10502' 2024-04-06T09:38:44.512 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=77, tag_tid=12, entry_tid=1' -a 'object_number=77, tag_tid=12, entry_tid=1' = 'object_number=70, tag_tid=11, entry_tid=10502' 2024-04-06T09:38:44.512 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=70, tag_tid=11, entry_tid=10502' '!=' 'object_number=69, tag_tid=11, entry_tid=10401' 2024-04-06T09:38:44.512 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:38:44.512 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=70, tag_tid=11, entry_tid=10502' = 'object_number=69, tag_tid=11, entry_tid=10401' 2024-04-06T09:38:44.512 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=70, tag_tid=11, entry_tid=10502' 2024-04-06T09:38:44.512 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=77, tag_tid=12, entry_tid=1' 2024-04-06T09:38:44.512 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:38:44.512 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:38:44.513 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=12 2024-04-06T09:38:44.513 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=70, tag_tid=11, entry_tid=10502' 2024-04-06T09:38:44.514 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:38:44.514 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:38:44.514 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=11 2024-04-06T09:38:44.515 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=77, tag_tid=12, entry_tid=1' 2024-04-06T09:38:44.515 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:38:44.515 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:38:44.516 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:38:44.517 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=70, tag_tid=11, entry_tid=10502' 2024-04-06T09:38:44.517 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:38:44.517 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:38:44.518 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=10502 2024-04-06T09:38:44.518 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 12 = 11 -a 1 -le 10502 2024-04-06T09:38:44.518 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:38:44.518 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:38:44.519 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:38:44.720 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:38:44.720 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:38:44.720 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:38:44.721 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:38:44.721 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:38:44.721 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:38:44.721 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:38:44.721 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:38:44.721 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:38:44.721 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:38:44.721 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:38:44.721 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:38:44.721 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:38:44.721 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:38:44.721 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:38:44.722 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:38:44.722 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:38:44.725 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:38:44.725 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:38:44.725 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:38:44.725 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:38:44.725 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:38:44.725 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:38:44.725 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:38:44.726 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:38:44.726 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:38:44.726 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:38:44.726 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:38:44.726 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:38:44.726 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:38:44.726 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:38:44.726 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:38:44.726 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:38:44.726 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:38:44.726 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:38:44.726 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:38:44.726 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:38:44.727 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:38:44.727 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:38:44.727 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:38:44.727 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:38:44.727 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:38:44.727 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:38:44.727 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:38:44.727 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:38:44.727 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:38:44.727 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:38:44.727 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:38:44.727 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:38:44.749 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:38:44.750 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:38:44.750 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:38:44.750 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:38:44.882 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:38:44.883 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:38:44.883 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:38:44.884 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:38:44.884 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:38:44.884 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:38:44.884 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:38:44.884 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:38:44.884 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:38:44.884 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:38:44.884 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:38:44.885 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:38:44.885 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:38:44.887 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:38:44.887 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:38:44.888 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:38:44.942 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 17 2024-04-06T09:38:44.943 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 19 2024-04-06T09:38:44.943 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:38:44.943 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=77, tag_tid=12, entry_tid=1], [object_number=76, tag_tid=12, entry_tid=0], [object_number=79, tag_tid=11, entry_tid=17175], [object_number=78, tag_tid=11, entry_tid=17174]]], state=connected] 2024-04-06T09:38:44.943 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=70, tag_tid=11, entry_tid=10602], [object_number=69, tag_tid=11, entry_tid=10601], [object_number=68, tag_tid=11, entry_tid=10600], [object_number=71, tag_tid=11, entry_tid=10599]]], state=connected] 2024-04-06T09:38:44.945 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:38:44.947 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=77, tag_tid=12, entry_tid=1' 2024-04-06T09:38:44.948 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:38:44.948 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:38:44.948 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:38:44.948 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:38:44.948 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:38:44.948 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:38:44.948 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:38:44.948 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:38:44.948 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:38:44.949 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:38:44.949 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:38:44.949 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:38:44.949 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:38:44.950 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:38:44.950 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:38:44.996 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 17 2024-04-06T09:38:44.996 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 19 2024-04-06T09:38:44.996 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:38:44.996 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=77, tag_tid=12, entry_tid=1], [object_number=76, tag_tid=12, entry_tid=0], [object_number=79, tag_tid=11, entry_tid=17175], [object_number=78, tag_tid=11, entry_tid=17174]]], state=connected] 2024-04-06T09:38:44.996 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=70, tag_tid=11, entry_tid=10602], [object_number=69, tag_tid=11, entry_tid=10601], [object_number=68, tag_tid=11, entry_tid=10600], [object_number=71, tag_tid=11, entry_tid=10599]]], state=connected] 2024-04-06T09:38:45.000 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:38:45.002 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=70, tag_tid=11, entry_tid=10602' 2024-04-06T09:38:45.002 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=77, tag_tid=12, entry_tid=1' -a 'object_number=77, tag_tid=12, entry_tid=1' = 'object_number=70, tag_tid=11, entry_tid=10602' 2024-04-06T09:38:45.003 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=70, tag_tid=11, entry_tid=10602' '!=' 'object_number=70, tag_tid=11, entry_tid=10502' 2024-04-06T09:38:45.003 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:38:45.003 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=70, tag_tid=11, entry_tid=10602' = 'object_number=70, tag_tid=11, entry_tid=10502' 2024-04-06T09:38:45.003 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=70, tag_tid=11, entry_tid=10602' 2024-04-06T09:38:45.003 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=77, tag_tid=12, entry_tid=1' 2024-04-06T09:38:45.003 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:38:45.003 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:38:45.004 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=12 2024-04-06T09:38:45.004 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=70, tag_tid=11, entry_tid=10602' 2024-04-06T09:38:45.004 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:38:45.004 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:38:45.005 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=11 2024-04-06T09:38:45.006 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=77, tag_tid=12, entry_tid=1' 2024-04-06T09:38:45.006 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:38:45.006 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:38:45.007 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:38:45.007 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=70, tag_tid=11, entry_tid=10602' 2024-04-06T09:38:45.008 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:38:45.008 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:38:45.009 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=10602 2024-04-06T09:38:45.009 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 12 = 11 -a 1 -le 10602 2024-04-06T09:38:45.009 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:38:45.009 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:38:45.009 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:38:45.211 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:38:45.211 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:38:45.211 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:38:45.211 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:38:45.211 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:38:45.211 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:38:45.211 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:38:45.212 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:38:45.212 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:38:45.212 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:38:45.212 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:38:45.212 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:38:45.212 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:38:45.212 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:38:45.212 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:38:45.212 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:38:45.213 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:38:45.215 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:38:45.216 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:38:45.216 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:38:45.216 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:38:45.216 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:38:45.216 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:38:45.216 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:38:45.216 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:38:45.216 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:38:45.216 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:38:45.216 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:38:45.216 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:38:45.216 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:38:45.216 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:38:45.216 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:38:45.217 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:38:45.218 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:38:45.218 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:38:45.218 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:38:45.218 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:38:45.218 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:38:45.218 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:38:45.218 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:38:45.218 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:38:45.218 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:38:45.218 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:38:45.218 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:38:45.218 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:38:45.218 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:38:45.219 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:38:45.219 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:38:45.219 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:38:45.242 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:38:45.242 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:38:45.242 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:38:45.242 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:38:45.375 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:38:45.376 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:38:45.376 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:38:45.376 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:38:45.376 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:38:45.376 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:38:45.377 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:38:45.377 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:38:45.377 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:38:45.377 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:38:45.377 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:38:45.377 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:38:45.377 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:38:45.379 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:38:45.380 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:38:45.380 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:38:45.426 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 17 2024-04-06T09:38:45.426 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 19 2024-04-06T09:38:45.426 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:38:45.426 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=77, tag_tid=12, entry_tid=1], [object_number=76, tag_tid=12, entry_tid=0], [object_number=79, tag_tid=11, entry_tid=17175], [object_number=78, tag_tid=11, entry_tid=17174]]], state=connected] 2024-04-06T09:38:45.426 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=69, tag_tid=11, entry_tid=10697], [object_number=68, tag_tid=11, entry_tid=10696], [object_number=71, tag_tid=11, entry_tid=10695], [object_number=70, tag_tid=11, entry_tid=10694]]], state=connected] 2024-04-06T09:38:45.429 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:38:45.431 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=77, tag_tid=12, entry_tid=1' 2024-04-06T09:38:45.432 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:38:45.432 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:38:45.432 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:38:45.432 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:38:45.432 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:38:45.432 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:38:45.433 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:38:45.433 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:38:45.433 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:38:45.433 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:38:45.433 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:38:45.433 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:38:45.433 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:38:45.434 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:38:45.434 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:38:45.479 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 17 2024-04-06T09:38:45.479 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 19 2024-04-06T09:38:45.479 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:38:45.479 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=77, tag_tid=12, entry_tid=1], [object_number=76, tag_tid=12, entry_tid=0], [object_number=79, tag_tid=11, entry_tid=17175], [object_number=78, tag_tid=11, entry_tid=17174]]], state=connected] 2024-04-06T09:38:45.479 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=69, tag_tid=11, entry_tid=10697], [object_number=68, tag_tid=11, entry_tid=10696], [object_number=71, tag_tid=11, entry_tid=10695], [object_number=70, tag_tid=11, entry_tid=10694]]], state=connected] 2024-04-06T09:38:45.483 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:38:45.484 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=69, tag_tid=11, entry_tid=10697' 2024-04-06T09:38:45.484 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=77, tag_tid=12, entry_tid=1' -a 'object_number=77, tag_tid=12, entry_tid=1' = 'object_number=69, tag_tid=11, entry_tid=10697' 2024-04-06T09:38:45.485 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=69, tag_tid=11, entry_tid=10697' '!=' 'object_number=70, tag_tid=11, entry_tid=10602' 2024-04-06T09:38:45.485 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:38:45.485 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=69, tag_tid=11, entry_tid=10697' = 'object_number=70, tag_tid=11, entry_tid=10602' 2024-04-06T09:38:45.485 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=69, tag_tid=11, entry_tid=10697' 2024-04-06T09:38:45.485 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=77, tag_tid=12, entry_tid=1' 2024-04-06T09:38:45.485 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:38:45.485 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:38:45.486 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=12 2024-04-06T09:38:45.486 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=69, tag_tid=11, entry_tid=10697' 2024-04-06T09:38:45.487 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:38:45.487 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:38:45.487 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=11 2024-04-06T09:38:45.488 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=77, tag_tid=12, entry_tid=1' 2024-04-06T09:38:45.488 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:38:45.488 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:38:45.489 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:38:45.490 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=69, tag_tid=11, entry_tid=10697' 2024-04-06T09:38:45.490 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:38:45.490 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:38:45.491 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=10697 2024-04-06T09:38:45.491 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 12 = 11 -a 1 -le 10697 2024-04-06T09:38:45.491 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:38:45.491 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:38:45.491 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:38:45.693 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:38:45.693 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:38:45.694 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:38:45.694 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:38:45.694 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:38:45.694 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:38:45.694 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:38:45.694 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:38:45.694 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:38:45.694 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:38:45.694 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:38:45.694 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:38:45.694 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:38:45.694 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:38:45.695 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:38:45.695 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:38:45.696 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:38:45.697 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:38:45.697 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:38:45.697 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:38:45.697 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:38:45.697 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:38:45.697 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:38:45.697 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:38:45.697 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:38:45.697 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:38:45.697 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:38:45.697 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:38:45.697 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:38:45.697 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:38:45.698 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:38:45.698 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:38:45.698 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:38:45.698 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:38:45.698 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:38:45.698 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:38:45.698 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:38:45.698 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:38:45.698 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:38:45.698 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:38:45.698 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:38:45.698 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:38:45.698 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:38:45.698 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:38:45.698 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:38:45.699 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:38:45.699 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:38:45.699 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:38:45.699 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:38:45.723 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:38:45.723 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:38:45.723 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:38:45.723 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:38:45.854 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:38:45.855 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:38:45.855 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:38:45.855 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:38:45.856 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:38:45.856 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:38:45.856 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:38:45.856 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:38:45.856 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:38:45.856 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:38:45.856 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:38:45.856 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:38:45.857 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:38:45.857 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:38:45.858 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:38:45.858 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:38:45.913 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 17 2024-04-06T09:38:45.913 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 19 2024-04-06T09:38:45.913 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:38:45.913 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=77, tag_tid=12, entry_tid=1], [object_number=76, tag_tid=12, entry_tid=0], [object_number=79, tag_tid=11, entry_tid=17175], [object_number=78, tag_tid=11, entry_tid=17174]]], state=connected] 2024-04-06T09:38:45.913 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=71, tag_tid=11, entry_tid=10795], [object_number=70, tag_tid=11, entry_tid=10794], [object_number=69, tag_tid=11, entry_tid=10793], [object_number=68, tag_tid=11, entry_tid=10792]]], state=connected] 2024-04-06T09:38:45.917 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:38:45.918 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=77, tag_tid=12, entry_tid=1' 2024-04-06T09:38:45.919 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:38:45.919 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:38:45.919 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:38:45.919 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:38:45.919 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:38:45.919 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:38:45.920 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:38:45.920 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:38:45.920 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:38:45.920 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:38:45.920 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:38:45.920 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:38:45.920 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:38:45.921 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:38:45.921 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:38:45.966 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 17 2024-04-06T09:38:45.967 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 19 2024-04-06T09:38:45.967 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:38:45.967 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=77, tag_tid=12, entry_tid=1], [object_number=76, tag_tid=12, entry_tid=0], [object_number=79, tag_tid=11, entry_tid=17175], [object_number=78, tag_tid=11, entry_tid=17174]]], state=connected] 2024-04-06T09:38:45.967 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=71, tag_tid=11, entry_tid=10795], [object_number=70, tag_tid=11, entry_tid=10794], [object_number=69, tag_tid=11, entry_tid=10793], [object_number=68, tag_tid=11, entry_tid=10792]]], state=connected] 2024-04-06T09:38:45.970 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:38:45.972 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=71, tag_tid=11, entry_tid=10795' 2024-04-06T09:38:45.972 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=77, tag_tid=12, entry_tid=1' -a 'object_number=77, tag_tid=12, entry_tid=1' = 'object_number=71, tag_tid=11, entry_tid=10795' 2024-04-06T09:38:45.972 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=71, tag_tid=11, entry_tid=10795' '!=' 'object_number=69, tag_tid=11, entry_tid=10697' 2024-04-06T09:38:45.972 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:38:45.973 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=71, tag_tid=11, entry_tid=10795' = 'object_number=69, tag_tid=11, entry_tid=10697' 2024-04-06T09:38:45.973 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=71, tag_tid=11, entry_tid=10795' 2024-04-06T09:38:45.973 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=77, tag_tid=12, entry_tid=1' 2024-04-06T09:38:45.973 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:38:45.973 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:38:45.973 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=12 2024-04-06T09:38:45.974 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=71, tag_tid=11, entry_tid=10795' 2024-04-06T09:38:45.974 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:38:45.974 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:38:45.975 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=11 2024-04-06T09:38:45.976 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=77, tag_tid=12, entry_tid=1' 2024-04-06T09:38:45.976 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:38:45.976 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:38:45.977 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:38:45.977 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=71, tag_tid=11, entry_tid=10795' 2024-04-06T09:38:45.977 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:38:45.978 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:38:45.979 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=10795 2024-04-06T09:38:45.979 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 12 = 11 -a 1 -le 10795 2024-04-06T09:38:45.979 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:38:45.979 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:38:45.979 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:38:46.181 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:38:46.181 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:38:46.181 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:38:46.181 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:38:46.181 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:38:46.182 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:38:46.182 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:38:46.182 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:38:46.182 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:38:46.182 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:38:46.182 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:38:46.182 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:38:46.182 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:38:46.182 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:38:46.182 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:38:46.183 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:38:46.183 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:38:46.186 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:38:46.186 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:38:46.186 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:38:46.186 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:38:46.186 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:38:46.186 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:38:46.186 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:38:46.186 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:38:46.186 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:38:46.187 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:38:46.187 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:38:46.187 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:38:46.187 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:38:46.187 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:38:46.187 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:38:46.187 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:38:46.187 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:38:46.187 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:38:46.188 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:38:46.188 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:38:46.188 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:38:46.188 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:38:46.188 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:38:46.188 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:38:46.188 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:38:46.188 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:38:46.188 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:38:46.188 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:38:46.188 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:38:46.188 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:38:46.188 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:38:46.188 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:38:46.212 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:38:46.212 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:38:46.212 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:38:46.212 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:38:46.338 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:38:46.339 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:38:46.339 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:38:46.339 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:38:46.340 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:38:46.340 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:38:46.340 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:38:46.340 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:38:46.340 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:38:46.340 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:38:46.340 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:38:46.341 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:38:46.341 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:38:46.342 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:38:46.342 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:38:46.342 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:38:46.395 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 17 2024-04-06T09:38:46.395 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 19 2024-04-06T09:38:46.395 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:38:46.395 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=77, tag_tid=12, entry_tid=1], [object_number=76, tag_tid=12, entry_tid=0], [object_number=79, tag_tid=11, entry_tid=17175], [object_number=78, tag_tid=11, entry_tid=17174]]], state=connected] 2024-04-06T09:38:46.395 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=71, tag_tid=11, entry_tid=10895], [object_number=70, tag_tid=11, entry_tid=10894], [object_number=69, tag_tid=11, entry_tid=10893], [object_number=68, tag_tid=11, entry_tid=10892]]], state=connected] 2024-04-06T09:38:46.398 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:38:46.400 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=77, tag_tid=12, entry_tid=1' 2024-04-06T09:38:46.401 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:38:46.401 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:38:46.401 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:38:46.401 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:38:46.402 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:38:46.402 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:38:46.402 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:38:46.402 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:38:46.402 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:38:46.402 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:38:46.402 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:38:46.402 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:38:46.403 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:38:46.403 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:38:46.403 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:38:46.449 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 17 2024-04-06T09:38:46.449 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 19 2024-04-06T09:38:46.450 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:38:46.450 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=77, tag_tid=12, entry_tid=1], [object_number=76, tag_tid=12, entry_tid=0], [object_number=79, tag_tid=11, entry_tid=17175], [object_number=78, tag_tid=11, entry_tid=17174]]], state=connected] 2024-04-06T09:38:46.450 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=71, tag_tid=11, entry_tid=10895], [object_number=70, tag_tid=11, entry_tid=10894], [object_number=69, tag_tid=11, entry_tid=10893], [object_number=68, tag_tid=11, entry_tid=10892]]], state=connected] 2024-04-06T09:38:46.453 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:38:46.455 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=71, tag_tid=11, entry_tid=10895' 2024-04-06T09:38:46.455 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=77, tag_tid=12, entry_tid=1' -a 'object_number=77, tag_tid=12, entry_tid=1' = 'object_number=71, tag_tid=11, entry_tid=10895' 2024-04-06T09:38:46.455 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=71, tag_tid=11, entry_tid=10895' '!=' 'object_number=71, tag_tid=11, entry_tid=10795' 2024-04-06T09:38:46.455 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:38:46.455 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=71, tag_tid=11, entry_tid=10895' = 'object_number=71, tag_tid=11, entry_tid=10795' 2024-04-06T09:38:46.455 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=71, tag_tid=11, entry_tid=10895' 2024-04-06T09:38:46.455 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=77, tag_tid=12, entry_tid=1' 2024-04-06T09:38:46.456 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:38:46.456 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:38:46.456 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=12 2024-04-06T09:38:46.457 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=71, tag_tid=11, entry_tid=10895' 2024-04-06T09:38:46.457 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:38:46.457 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:38:46.458 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=11 2024-04-06T09:38:46.458 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=77, tag_tid=12, entry_tid=1' 2024-04-06T09:38:46.459 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:38:46.459 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:38:46.459 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:38:46.460 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=71, tag_tid=11, entry_tid=10895' 2024-04-06T09:38:46.460 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:38:46.460 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:38:46.461 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=10895 2024-04-06T09:38:46.462 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 12 = 11 -a 1 -le 10895 2024-04-06T09:38:46.462 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:38:46.462 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:38:46.462 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:38:46.663 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:38:46.664 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:38:46.664 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:38:46.664 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:38:46.664 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:38:46.664 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:38:46.664 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:38:46.664 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:38:46.664 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:38:46.664 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:38:46.664 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:38:46.664 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:38:46.665 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:38:46.665 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:38:46.665 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:38:46.665 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:38:46.665 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:38:46.668 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:38:46.668 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:38:46.668 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:38:46.668 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:38:46.668 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:38:46.668 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:38:46.668 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:38:46.668 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:38:46.668 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:38:46.668 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:38:46.669 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:38:46.669 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:38:46.669 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:38:46.669 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:38:46.669 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:38:46.669 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:38:46.669 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:38:46.669 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:38:46.669 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:38:46.669 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:38:46.669 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:38:46.669 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:38:46.669 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:38:46.669 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:38:46.669 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:38:46.669 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:38:46.670 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:38:46.670 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:38:46.670 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:38:46.670 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:38:46.670 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:38:46.670 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:38:46.692 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:38:46.693 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:38:46.693 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:38:46.693 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:38:46.826 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:38:46.828 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:38:46.828 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:38:46.828 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:38:46.828 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:38:46.828 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:38:46.828 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:38:46.828 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:38:46.828 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:38:46.828 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:38:46.828 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:38:46.829 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:38:46.829 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:38:46.830 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:38:46.830 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:38:46.831 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:38:46.881 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 17 2024-04-06T09:38:46.881 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 19 2024-04-06T09:38:46.881 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:38:46.881 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=77, tag_tid=12, entry_tid=1], [object_number=76, tag_tid=12, entry_tid=0], [object_number=79, tag_tid=11, entry_tid=17175], [object_number=78, tag_tid=11, entry_tid=17174]]], state=connected] 2024-04-06T09:38:46.881 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=70, tag_tid=11, entry_tid=10982], [object_number=69, tag_tid=11, entry_tid=10981], [object_number=68, tag_tid=11, entry_tid=10980], [object_number=71, tag_tid=11, entry_tid=10979]]], state=connected] 2024-04-06T09:38:46.885 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:38:46.886 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=77, tag_tid=12, entry_tid=1' 2024-04-06T09:38:46.887 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:38:46.887 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:38:46.887 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:38:46.887 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:38:46.888 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:38:46.888 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:38:46.888 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:38:46.888 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:38:46.888 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:38:46.888 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:38:46.888 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:38:46.888 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:38:46.888 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:38:46.889 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:38:46.889 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:38:46.936 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 17 2024-04-06T09:38:46.936 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 19 2024-04-06T09:38:46.936 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:38:46.936 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=77, tag_tid=12, entry_tid=1], [object_number=76, tag_tid=12, entry_tid=0], [object_number=79, tag_tid=11, entry_tid=17175], [object_number=78, tag_tid=11, entry_tid=17174]]], state=connected] 2024-04-06T09:38:46.936 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=70, tag_tid=11, entry_tid=10982], [object_number=69, tag_tid=11, entry_tid=10981], [object_number=68, tag_tid=11, entry_tid=10980], [object_number=71, tag_tid=11, entry_tid=10979]]], state=connected] 2024-04-06T09:38:46.939 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:38:46.941 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=70, tag_tid=11, entry_tid=10982' 2024-04-06T09:38:46.941 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=77, tag_tid=12, entry_tid=1' -a 'object_number=77, tag_tid=12, entry_tid=1' = 'object_number=70, tag_tid=11, entry_tid=10982' 2024-04-06T09:38:46.941 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=70, tag_tid=11, entry_tid=10982' '!=' 'object_number=71, tag_tid=11, entry_tid=10895' 2024-04-06T09:38:46.941 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:38:46.942 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=70, tag_tid=11, entry_tid=10982' = 'object_number=71, tag_tid=11, entry_tid=10895' 2024-04-06T09:38:46.942 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=70, tag_tid=11, entry_tid=10982' 2024-04-06T09:38:46.942 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=77, tag_tid=12, entry_tid=1' 2024-04-06T09:38:46.942 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:38:46.942 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:38:46.942 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=12 2024-04-06T09:38:46.943 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=70, tag_tid=11, entry_tid=10982' 2024-04-06T09:38:46.943 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:38:46.943 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:38:46.944 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=11 2024-04-06T09:38:46.945 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=77, tag_tid=12, entry_tid=1' 2024-04-06T09:38:46.945 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:38:46.945 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:38:46.946 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:38:46.947 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=70, tag_tid=11, entry_tid=10982' 2024-04-06T09:38:46.947 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:38:46.947 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:38:46.948 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=10982 2024-04-06T09:38:46.948 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 12 = 11 -a 1 -le 10982 2024-04-06T09:38:46.948 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:38:46.948 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:38:46.948 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:38:47.150 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:38:47.150 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:38:47.150 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:38:47.150 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:38:47.150 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:38:47.151 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:38:47.151 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:38:47.151 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:38:47.151 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:38:47.151 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:38:47.151 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:38:47.151 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:38:47.151 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:38:47.151 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:38:47.151 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:38:47.152 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:38:47.152 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:38:47.153 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:38:47.153 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:38:47.153 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:38:47.154 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:38:47.154 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:38:47.154 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:38:47.154 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:38:47.154 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:38:47.154 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:38:47.154 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:38:47.154 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:38:47.154 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:38:47.154 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:38:47.154 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:38:47.154 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:38:47.154 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:38:47.156 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:38:47.156 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:38:47.156 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:38:47.156 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:38:47.156 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:38:47.156 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:38:47.156 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:38:47.156 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:38:47.156 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:38:47.156 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:38:47.156 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:38:47.156 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:38:47.156 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:38:47.156 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:38:47.156 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:38:47.157 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:38:47.180 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:38:47.181 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:38:47.181 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:38:47.181 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:38:47.306 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:38:47.307 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:38:47.307 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:38:47.307 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:38:47.307 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:38:47.308 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:38:47.308 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:38:47.308 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:38:47.308 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:38:47.308 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:38:47.308 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:38:47.308 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:38:47.308 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:38:47.309 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:38:47.310 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:38:47.310 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:38:47.357 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 17 2024-04-06T09:38:47.357 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 19 2024-04-06T09:38:47.357 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:38:47.357 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=77, tag_tid=12, entry_tid=1], [object_number=76, tag_tid=12, entry_tid=0], [object_number=79, tag_tid=11, entry_tid=17175], [object_number=78, tag_tid=11, entry_tid=17174]]], state=connected] 2024-04-06T09:38:47.357 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=70, tag_tid=11, entry_tid=11074], [object_number=69, tag_tid=11, entry_tid=11073], [object_number=68, tag_tid=11, entry_tid=11072], [object_number=71, tag_tid=11, entry_tid=11071]]], state=connected] 2024-04-06T09:38:47.360 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:38:47.362 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=77, tag_tid=12, entry_tid=1' 2024-04-06T09:38:47.363 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:38:47.363 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:38:47.363 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:38:47.363 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:38:47.363 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:38:47.363 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:38:47.363 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:38:47.363 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:38:47.363 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:38:47.363 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:38:47.366 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:38:47.366 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:38:47.366 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:38:47.366 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:38:47.366 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:38:47.409 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 17 2024-04-06T09:38:47.410 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 19 2024-04-06T09:38:47.410 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:38:47.410 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=77, tag_tid=12, entry_tid=1], [object_number=76, tag_tid=12, entry_tid=0], [object_number=79, tag_tid=11, entry_tid=17175], [object_number=78, tag_tid=11, entry_tid=17174]]], state=connected] 2024-04-06T09:38:47.410 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=70, tag_tid=11, entry_tid=11074], [object_number=69, tag_tid=11, entry_tid=11073], [object_number=68, tag_tid=11, entry_tid=11072], [object_number=71, tag_tid=11, entry_tid=11071]]], state=connected] 2024-04-06T09:38:47.413 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:38:47.415 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=70, tag_tid=11, entry_tid=11074' 2024-04-06T09:38:47.415 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=77, tag_tid=12, entry_tid=1' -a 'object_number=77, tag_tid=12, entry_tid=1' = 'object_number=70, tag_tid=11, entry_tid=11074' 2024-04-06T09:38:47.415 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=70, tag_tid=11, entry_tid=11074' '!=' 'object_number=70, tag_tid=11, entry_tid=10982' 2024-04-06T09:38:47.415 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:38:47.415 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=70, tag_tid=11, entry_tid=11074' = 'object_number=70, tag_tid=11, entry_tid=10982' 2024-04-06T09:38:47.415 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=70, tag_tid=11, entry_tid=11074' 2024-04-06T09:38:47.416 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=77, tag_tid=12, entry_tid=1' 2024-04-06T09:38:47.416 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:38:47.416 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:38:47.416 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=12 2024-04-06T09:38:47.417 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=70, tag_tid=11, entry_tid=11074' 2024-04-06T09:38:47.417 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:38:47.417 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:38:47.418 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=11 2024-04-06T09:38:47.419 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=77, tag_tid=12, entry_tid=1' 2024-04-06T09:38:47.419 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:38:47.419 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:38:47.420 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:38:47.420 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=70, tag_tid=11, entry_tid=11074' 2024-04-06T09:38:47.420 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:38:47.421 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:38:47.422 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=11074 2024-04-06T09:38:47.422 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 12 = 11 -a 1 -le 11074 2024-04-06T09:38:47.422 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:38:47.422 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:38:47.422 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:38:47.624 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:38:47.624 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:38:47.624 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:38:47.624 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:38:47.624 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:38:47.625 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:38:47.625 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:38:47.625 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:38:47.625 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:38:47.625 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:38:47.625 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:38:47.625 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:38:47.625 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:38:47.625 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:38:47.625 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:38:47.626 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:38:47.626 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:38:47.629 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:38:47.629 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:38:47.629 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:38:47.629 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:38:47.630 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:38:47.630 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:38:47.630 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:38:47.630 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:38:47.630 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:38:47.630 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:38:47.630 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:38:47.630 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:38:47.630 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:38:47.630 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:38:47.630 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:38:47.630 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:38:47.631 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:38:47.631 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:38:47.631 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:38:47.631 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:38:47.631 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:38:47.631 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:38:47.631 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:38:47.631 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:38:47.631 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:38:47.631 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:38:47.631 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:38:47.631 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:38:47.632 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:38:47.632 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:38:47.632 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:38:47.632 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:38:47.655 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:38:47.655 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:38:47.656 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:38:47.656 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:38:47.792 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:38:47.793 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:38:47.793 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:38:47.793 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:38:47.793 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:38:47.793 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:38:47.793 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:38:47.794 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:38:47.794 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:38:47.794 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:38:47.794 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:38:47.795 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:38:47.795 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:38:47.797 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:38:47.797 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:38:47.797 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:38:47.854 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 17 2024-04-06T09:38:47.854 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 19 2024-04-06T09:38:47.854 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:38:47.854 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=77, tag_tid=12, entry_tid=1], [object_number=76, tag_tid=12, entry_tid=0], [object_number=79, tag_tid=11, entry_tid=17175], [object_number=78, tag_tid=11, entry_tid=17174]]], state=connected] 2024-04-06T09:38:47.854 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=68, tag_tid=11, entry_tid=11164], [object_number=71, tag_tid=11, entry_tid=11163], [object_number=70, tag_tid=11, entry_tid=11162], [object_number=69, tag_tid=11, entry_tid=11161]]], state=connected] 2024-04-06T09:38:47.857 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:38:47.859 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=77, tag_tid=12, entry_tid=1' 2024-04-06T09:38:47.860 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:38:47.860 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:38:47.860 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:38:47.860 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:38:47.860 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:38:47.860 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:38:47.860 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:38:47.860 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:38:47.860 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:38:47.861 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:38:47.861 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:38:47.861 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:38:47.861 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:38:47.861 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:38:47.861 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:38:47.908 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 17 2024-04-06T09:38:47.908 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 19 2024-04-06T09:38:47.908 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:38:47.908 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=77, tag_tid=12, entry_tid=1], [object_number=76, tag_tid=12, entry_tid=0], [object_number=79, tag_tid=11, entry_tid=17175], [object_number=78, tag_tid=11, entry_tid=17174]]], state=connected] 2024-04-06T09:38:47.909 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=68, tag_tid=11, entry_tid=11164], [object_number=71, tag_tid=11, entry_tid=11163], [object_number=70, tag_tid=11, entry_tid=11162], [object_number=69, tag_tid=11, entry_tid=11161]]], state=connected] 2024-04-06T09:38:47.912 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:38:47.914 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=68, tag_tid=11, entry_tid=11164' 2024-04-06T09:38:47.914 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=77, tag_tid=12, entry_tid=1' -a 'object_number=77, tag_tid=12, entry_tid=1' = 'object_number=68, tag_tid=11, entry_tid=11164' 2024-04-06T09:38:47.914 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=68, tag_tid=11, entry_tid=11164' '!=' 'object_number=70, tag_tid=11, entry_tid=11074' 2024-04-06T09:38:47.914 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:38:47.914 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=68, tag_tid=11, entry_tid=11164' = 'object_number=70, tag_tid=11, entry_tid=11074' 2024-04-06T09:38:47.914 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=68, tag_tid=11, entry_tid=11164' 2024-04-06T09:38:47.914 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=77, tag_tid=12, entry_tid=1' 2024-04-06T09:38:47.914 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:38:47.914 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:38:47.915 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=12 2024-04-06T09:38:47.916 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=68, tag_tid=11, entry_tid=11164' 2024-04-06T09:38:47.916 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:38:47.916 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:38:47.917 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=11 2024-04-06T09:38:47.917 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=77, tag_tid=12, entry_tid=1' 2024-04-06T09:38:47.918 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:38:47.918 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:38:47.919 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:38:47.919 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=68, tag_tid=11, entry_tid=11164' 2024-04-06T09:38:47.919 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:38:47.919 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:38:47.921 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=11164 2024-04-06T09:38:47.921 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 12 = 11 -a 1 -le 11164 2024-04-06T09:38:47.921 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:38:47.921 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:38:47.921 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:38:48.123 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:38:48.123 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:38:48.123 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:38:48.123 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:38:48.123 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:38:48.123 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:38:48.124 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:38:48.124 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:38:48.124 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:38:48.124 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:38:48.124 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:38:48.124 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:38:48.124 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:38:48.124 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:38:48.124 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:38:48.125 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:38:48.125 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:38:48.127 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:38:48.127 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:38:48.128 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:38:48.128 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:38:48.128 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:38:48.128 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:38:48.128 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:38:48.128 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:38:48.128 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:38:48.128 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:38:48.128 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:38:48.128 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:38:48.128 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:38:48.128 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:38:48.128 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:38:48.128 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:38:48.129 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:38:48.129 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:38:48.129 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:38:48.129 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:38:48.129 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:38:48.129 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:38:48.129 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:38:48.129 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:38:48.129 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:38:48.129 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:38:48.129 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:38:48.129 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:38:48.129 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:38:48.129 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:38:48.130 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:38:48.130 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:38:48.153 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:38:48.154 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:38:48.154 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:38:48.154 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:38:48.284 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:38:48.285 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:38:48.285 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:38:48.285 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:38:48.285 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:38:48.285 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:38:48.286 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:38:48.286 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:38:48.286 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:38:48.286 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:38:48.286 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:38:48.286 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:38:48.286 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:38:48.288 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:38:48.288 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:38:48.288 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:38:48.339 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 17 2024-04-06T09:38:48.339 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 19 2024-04-06T09:38:48.340 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:38:48.340 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=77, tag_tid=12, entry_tid=1], [object_number=76, tag_tid=12, entry_tid=0], [object_number=79, tag_tid=11, entry_tid=17175], [object_number=78, tag_tid=11, entry_tid=17174]]], state=connected] 2024-04-06T09:38:48.340 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=69, tag_tid=11, entry_tid=11249], [object_number=68, tag_tid=11, entry_tid=11248], [object_number=71, tag_tid=11, entry_tid=11247], [object_number=70, tag_tid=11, entry_tid=11246]]], state=connected] 2024-04-06T09:38:48.343 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:38:48.344 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=77, tag_tid=12, entry_tid=1' 2024-04-06T09:38:48.345 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:38:48.346 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:38:48.346 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:38:48.346 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:38:48.346 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:38:48.346 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:38:48.346 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:38:48.346 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:38:48.346 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:38:48.346 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:38:48.346 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:38:48.346 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:38:48.347 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:38:48.347 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:38:48.347 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:38:48.393 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 17 2024-04-06T09:38:48.393 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 19 2024-04-06T09:38:48.394 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:38:48.394 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=77, tag_tid=12, entry_tid=1], [object_number=76, tag_tid=12, entry_tid=0], [object_number=79, tag_tid=11, entry_tid=17175], [object_number=78, tag_tid=11, entry_tid=17174]]], state=connected] 2024-04-06T09:38:48.394 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=69, tag_tid=11, entry_tid=11249], [object_number=68, tag_tid=11, entry_tid=11248], [object_number=71, tag_tid=11, entry_tid=11247], [object_number=70, tag_tid=11, entry_tid=11246]]], state=connected] 2024-04-06T09:38:48.397 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:38:48.399 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=69, tag_tid=11, entry_tid=11249' 2024-04-06T09:38:48.399 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=77, tag_tid=12, entry_tid=1' -a 'object_number=77, tag_tid=12, entry_tid=1' = 'object_number=69, tag_tid=11, entry_tid=11249' 2024-04-06T09:38:48.399 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=69, tag_tid=11, entry_tid=11249' '!=' 'object_number=68, tag_tid=11, entry_tid=11164' 2024-04-06T09:38:48.399 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:38:48.399 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=69, tag_tid=11, entry_tid=11249' = 'object_number=68, tag_tid=11, entry_tid=11164' 2024-04-06T09:38:48.400 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=69, tag_tid=11, entry_tid=11249' 2024-04-06T09:38:48.400 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=77, tag_tid=12, entry_tid=1' 2024-04-06T09:38:48.400 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:38:48.400 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:38:48.401 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=12 2024-04-06T09:38:48.401 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=69, tag_tid=11, entry_tid=11249' 2024-04-06T09:38:48.402 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:38:48.402 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:38:48.402 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=11 2024-04-06T09:38:48.403 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=77, tag_tid=12, entry_tid=1' 2024-04-06T09:38:48.403 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:38:48.403 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:38:48.404 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:38:48.405 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=69, tag_tid=11, entry_tid=11249' 2024-04-06T09:38:48.405 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:38:48.405 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:38:48.406 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=11249 2024-04-06T09:38:48.406 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 12 = 11 -a 1 -le 11249 2024-04-06T09:38:48.406 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:38:48.406 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:38:48.407 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:38:48.608 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:38:48.608 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:38:48.608 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:38:48.608 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:38:48.608 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:38:48.609 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:38:48.609 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:38:48.609 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:38:48.609 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:38:48.609 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:38:48.609 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:38:48.609 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:38:48.609 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:38:48.609 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:38:48.609 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:38:48.610 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:38:48.610 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:38:48.611 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:38:48.611 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:38:48.611 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:38:48.612 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:38:48.612 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:38:48.612 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:38:48.612 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:38:48.612 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:38:48.612 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:38:48.612 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:38:48.612 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:38:48.612 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:38:48.612 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:38:48.612 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:38:48.612 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:38:48.612 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:38:48.613 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:38:48.614 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:38:48.614 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:38:48.614 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:38:48.614 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:38:48.614 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:38:48.614 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:38:48.614 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:38:48.614 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:38:48.614 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:38:48.614 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:38:48.614 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:38:48.614 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:38:48.614 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:38:48.615 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:38:48.615 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:38:48.639 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:38:48.640 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:38:48.640 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:38:48.640 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:38:48.772 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:38:48.774 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:38:48.774 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:38:48.774 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:38:48.774 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:38:48.774 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:38:48.774 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:38:48.775 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:38:48.775 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:38:48.775 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:38:48.775 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:38:48.775 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:38:48.775 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:38:48.776 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:38:48.777 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:38:48.777 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:38:48.829 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 17 2024-04-06T09:38:48.829 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 19 2024-04-06T09:38:48.829 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:38:48.829 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=77, tag_tid=12, entry_tid=1], [object_number=76, tag_tid=12, entry_tid=0], [object_number=79, tag_tid=11, entry_tid=17175], [object_number=78, tag_tid=11, entry_tid=17174]]], state=connected] 2024-04-06T09:38:48.829 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=68, tag_tid=11, entry_tid=11344], [object_number=71, tag_tid=11, entry_tid=11343], [object_number=70, tag_tid=11, entry_tid=11342], [object_number=69, tag_tid=11, entry_tid=11341]]], state=connected] 2024-04-06T09:38:48.833 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:38:48.834 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=77, tag_tid=12, entry_tid=1' 2024-04-06T09:38:48.835 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:38:48.835 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:38:48.835 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:38:48.835 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:38:48.835 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:38:48.835 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:38:48.836 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:38:48.836 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:38:48.836 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:38:48.836 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:38:48.836 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:38:48.836 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:38:48.836 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:38:48.837 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:38:48.837 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:38:49.096 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 17 2024-04-06T09:38:49.096 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 19 2024-04-06T09:38:49.097 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:38:49.097 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=77, tag_tid=12, entry_tid=1], [object_number=76, tag_tid=12, entry_tid=0], [object_number=79, tag_tid=11, entry_tid=17175], [object_number=78, tag_tid=11, entry_tid=17174]]], state=connected] 2024-04-06T09:38:49.097 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=68, tag_tid=11, entry_tid=11344], [object_number=71, tag_tid=11, entry_tid=11343], [object_number=70, tag_tid=11, entry_tid=11342], [object_number=69, tag_tid=11, entry_tid=11341]]], state=connected] 2024-04-06T09:38:49.102 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:38:49.109 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=68, tag_tid=11, entry_tid=11344' 2024-04-06T09:38:49.110 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=77, tag_tid=12, entry_tid=1' -a 'object_number=77, tag_tid=12, entry_tid=1' = 'object_number=68, tag_tid=11, entry_tid=11344' 2024-04-06T09:38:49.110 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=68, tag_tid=11, entry_tid=11344' '!=' 'object_number=69, tag_tid=11, entry_tid=11249' 2024-04-06T09:38:49.110 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:38:49.110 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=68, tag_tid=11, entry_tid=11344' = 'object_number=69, tag_tid=11, entry_tid=11249' 2024-04-06T09:38:49.110 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=68, tag_tid=11, entry_tid=11344' 2024-04-06T09:38:49.113 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=77, tag_tid=12, entry_tid=1' 2024-04-06T09:38:49.113 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:38:49.113 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:38:49.113 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=12 2024-04-06T09:38:49.113 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=68, tag_tid=11, entry_tid=11344' 2024-04-06T09:38:49.114 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:38:49.114 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:38:49.114 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=11 2024-04-06T09:38:49.114 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=77, tag_tid=12, entry_tid=1' 2024-04-06T09:38:49.114 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:38:49.114 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:38:49.114 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:38:49.114 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=68, tag_tid=11, entry_tid=11344' 2024-04-06T09:38:49.114 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:38:49.114 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:38:49.115 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=11344 2024-04-06T09:38:49.115 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 12 = 11 -a 1 -le 11344 2024-04-06T09:38:49.115 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:38:49.115 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:38:49.115 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:38:49.315 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:38:49.315 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:38:49.315 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:38:49.315 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:38:49.315 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:38:49.315 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:38:49.315 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:38:49.316 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:38:49.316 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:38:49.316 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:38:49.316 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:38:49.316 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:38:49.316 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:38:49.316 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:38:49.316 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:38:49.320 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:38:49.320 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:38:49.320 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:38:49.320 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:38:49.320 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:38:49.320 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:38:49.320 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:38:49.320 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:38:49.320 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:38:49.321 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:38:49.321 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:38:49.321 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:38:49.321 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:38:49.321 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:38:49.321 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:38:49.321 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:38:49.321 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:38:49.321 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:38:49.321 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:38:49.321 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:38:49.321 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:38:49.321 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:38:49.321 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:38:49.321 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:38:49.322 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:38:49.322 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:38:49.322 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:38:49.322 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:38:49.322 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:38:49.322 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:38:49.322 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:38:49.322 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:38:49.322 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:38:49.322 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:38:49.345 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:38:49.346 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:38:49.346 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:38:49.346 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:38:49.476 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:38:49.477 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:38:49.478 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:38:49.478 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:38:49.478 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:38:49.478 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:38:49.478 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:38:49.478 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:38:49.478 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:38:49.478 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:38:49.478 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:38:49.479 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:38:49.479 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:38:49.479 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:38:49.480 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:38:49.480 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:38:49.531 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 17 2024-04-06T09:38:49.531 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 19 2024-04-06T09:38:49.531 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:38:49.531 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=77, tag_tid=12, entry_tid=1], [object_number=76, tag_tid=12, entry_tid=0], [object_number=79, tag_tid=11, entry_tid=17175], [object_number=78, tag_tid=11, entry_tid=17174]]], state=connected] 2024-04-06T09:38:49.531 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=69, tag_tid=11, entry_tid=11493], [object_number=68, tag_tid=11, entry_tid=11492], [object_number=71, tag_tid=11, entry_tid=11491], [object_number=70, tag_tid=11, entry_tid=11490]]], state=connected] 2024-04-06T09:38:49.534 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:38:49.535 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=77, tag_tid=12, entry_tid=1' 2024-04-06T09:38:49.537 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:38:49.537 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:38:49.537 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:38:49.537 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:38:49.537 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:38:49.537 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:38:49.537 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:38:49.537 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:38:49.537 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:38:49.537 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:38:49.537 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:38:49.537 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:38:49.538 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:38:49.538 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:38:49.538 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:38:49.585 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 17 2024-04-06T09:38:49.585 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 19 2024-04-06T09:38:49.585 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:38:49.585 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=77, tag_tid=12, entry_tid=1], [object_number=76, tag_tid=12, entry_tid=0], [object_number=79, tag_tid=11, entry_tid=17175], [object_number=78, tag_tid=11, entry_tid=17174]]], state=connected] 2024-04-06T09:38:49.585 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=69, tag_tid=11, entry_tid=11493], [object_number=68, tag_tid=11, entry_tid=11492], [object_number=71, tag_tid=11, entry_tid=11491], [object_number=70, tag_tid=11, entry_tid=11490]]], state=connected] 2024-04-06T09:38:49.588 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:38:49.590 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=69, tag_tid=11, entry_tid=11493' 2024-04-06T09:38:49.591 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=77, tag_tid=12, entry_tid=1' -a 'object_number=77, tag_tid=12, entry_tid=1' = 'object_number=69, tag_tid=11, entry_tid=11493' 2024-04-06T09:38:49.591 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=69, tag_tid=11, entry_tid=11493' '!=' 'object_number=68, tag_tid=11, entry_tid=11344' 2024-04-06T09:38:49.591 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:38:49.591 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=69, tag_tid=11, entry_tid=11493' = 'object_number=68, tag_tid=11, entry_tid=11344' 2024-04-06T09:38:49.591 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=69, tag_tid=11, entry_tid=11493' 2024-04-06T09:38:49.591 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=77, tag_tid=12, entry_tid=1' 2024-04-06T09:38:49.591 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:38:49.591 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:38:49.592 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=12 2024-04-06T09:38:49.592 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=69, tag_tid=11, entry_tid=11493' 2024-04-06T09:38:49.593 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:38:49.593 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:38:49.593 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=11 2024-04-06T09:38:49.594 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=77, tag_tid=12, entry_tid=1' 2024-04-06T09:38:49.594 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:38:49.594 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:38:49.595 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:38:49.596 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=69, tag_tid=11, entry_tid=11493' 2024-04-06T09:38:49.596 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:38:49.596 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:38:49.597 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=11493 2024-04-06T09:38:49.597 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 12 = 11 -a 1 -le 11493 2024-04-06T09:38:49.597 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:38:49.597 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:38:49.597 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:38:49.799 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:38:49.799 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:38:49.799 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:38:49.799 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:38:49.799 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:38:49.799 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:38:49.799 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:38:49.799 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:38:49.799 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:38:49.800 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:38:49.800 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:38:49.800 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:38:49.800 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:38:49.800 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:38:49.800 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:38:49.801 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:38:49.801 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:38:49.801 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:38:49.801 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:38:49.801 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:38:49.802 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:38:49.802 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:38:49.802 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:38:49.802 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:38:49.802 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:38:49.802 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:38:49.802 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:38:49.802 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:38:49.802 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:38:49.802 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:38:49.802 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:38:49.802 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:38:49.802 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:38:49.804 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:38:49.804 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:38:49.804 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:38:49.804 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:38:49.804 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:38:49.804 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:38:49.804 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:38:49.804 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:38:49.804 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:38:49.804 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:38:49.804 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:38:49.804 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:38:49.804 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:38:49.804 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:38:49.805 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:38:49.805 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:38:49.831 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:38:49.831 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:38:49.832 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:38:49.832 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:38:49.964 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:38:49.966 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:38:49.966 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:38:49.966 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:38:49.966 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:38:49.966 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:38:49.966 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:38:49.966 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:38:49.966 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:38:49.966 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:38:49.966 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:38:49.967 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:38:49.967 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:38:49.969 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:38:49.969 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:38:49.969 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:38:50.021 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 17 2024-04-06T09:38:50.021 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 19 2024-04-06T09:38:50.021 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:38:50.021 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=77, tag_tid=12, entry_tid=1], [object_number=76, tag_tid=12, entry_tid=0], [object_number=79, tag_tid=11, entry_tid=17175], [object_number=78, tag_tid=11, entry_tid=17174]]], state=connected] 2024-04-06T09:38:50.021 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=69, tag_tid=11, entry_tid=11597], [object_number=68, tag_tid=11, entry_tid=11596], [object_number=71, tag_tid=11, entry_tid=11595], [object_number=70, tag_tid=11, entry_tid=11594]]], state=connected] 2024-04-06T09:38:50.025 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:38:50.026 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=77, tag_tid=12, entry_tid=1' 2024-04-06T09:38:50.027 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:38:50.027 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:38:50.027 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:38:50.027 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:38:50.027 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:38:50.027 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:38:50.028 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:38:50.028 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:38:50.028 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:38:50.028 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:38:50.028 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:38:50.028 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:38:50.028 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:38:50.029 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:38:50.029 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:38:50.074 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 17 2024-04-06T09:38:50.075 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 19 2024-04-06T09:38:50.075 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:38:50.075 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=77, tag_tid=12, entry_tid=1], [object_number=76, tag_tid=12, entry_tid=0], [object_number=79, tag_tid=11, entry_tid=17175], [object_number=78, tag_tid=11, entry_tid=17174]]], state=connected] 2024-04-06T09:38:50.075 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=69, tag_tid=11, entry_tid=11597], [object_number=68, tag_tid=11, entry_tid=11596], [object_number=71, tag_tid=11, entry_tid=11595], [object_number=70, tag_tid=11, entry_tid=11594]]], state=connected] 2024-04-06T09:38:50.080 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:38:50.081 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=69, tag_tid=11, entry_tid=11597' 2024-04-06T09:38:50.082 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=77, tag_tid=12, entry_tid=1' -a 'object_number=77, tag_tid=12, entry_tid=1' = 'object_number=69, tag_tid=11, entry_tid=11597' 2024-04-06T09:38:50.082 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=69, tag_tid=11, entry_tid=11597' '!=' 'object_number=69, tag_tid=11, entry_tid=11493' 2024-04-06T09:38:50.082 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:38:50.082 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=69, tag_tid=11, entry_tid=11597' = 'object_number=69, tag_tid=11, entry_tid=11493' 2024-04-06T09:38:50.082 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=69, tag_tid=11, entry_tid=11597' 2024-04-06T09:38:50.083 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=77, tag_tid=12, entry_tid=1' 2024-04-06T09:38:50.083 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:38:50.083 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:38:50.083 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=12 2024-04-06T09:38:50.084 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=69, tag_tid=11, entry_tid=11597' 2024-04-06T09:38:50.084 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:38:50.084 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:38:50.085 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=11 2024-04-06T09:38:50.085 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=77, tag_tid=12, entry_tid=1' 2024-04-06T09:38:50.085 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:38:50.086 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:38:50.086 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:38:50.087 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=69, tag_tid=11, entry_tid=11597' 2024-04-06T09:38:50.087 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:38:50.087 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:38:50.088 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=11597 2024-04-06T09:38:50.088 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 12 = 11 -a 1 -le 11597 2024-04-06T09:38:50.088 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:38:50.088 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:38:50.089 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:38:50.290 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:38:50.290 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:38:50.290 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:38:50.290 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:38:50.291 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:38:50.291 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:38:50.291 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:38:50.291 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:38:50.291 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:38:50.291 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:38:50.291 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:38:50.291 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:38:50.291 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:38:50.291 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:38:50.291 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:38:50.292 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:38:50.292 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:38:50.294 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:38:50.294 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:38:50.294 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:38:50.294 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:38:50.294 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:38:50.294 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:38:50.294 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:38:50.294 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:38:50.295 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:38:50.295 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:38:50.295 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:38:50.295 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:38:50.295 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:38:50.295 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:38:50.295 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:38:50.295 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:38:50.296 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:38:50.296 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:38:50.297 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:38:50.297 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:38:50.297 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:38:50.297 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:38:50.297 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:38:50.297 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:38:50.297 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:38:50.297 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:38:50.297 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:38:50.297 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:38:50.297 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:38:50.297 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:38:50.297 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:38:50.297 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:38:50.321 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:38:50.321 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:38:50.321 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:38:50.321 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:38:50.449 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:38:50.451 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:38:50.451 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:38:50.451 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:38:50.451 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:38:50.451 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:38:50.451 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:38:50.451 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:38:50.451 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:38:50.451 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:38:50.452 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:38:50.452 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:38:50.452 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:38:50.453 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:38:50.453 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:38:50.453 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:38:50.499 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 17 2024-04-06T09:38:50.499 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 19 2024-04-06T09:38:50.499 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:38:50.499 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=77, tag_tid=12, entry_tid=1], [object_number=76, tag_tid=12, entry_tid=0], [object_number=79, tag_tid=11, entry_tid=17175], [object_number=78, tag_tid=11, entry_tid=17174]]], state=connected] 2024-04-06T09:38:50.499 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=69, tag_tid=11, entry_tid=11701], [object_number=68, tag_tid=11, entry_tid=11700], [object_number=71, tag_tid=11, entry_tid=11699], [object_number=70, tag_tid=11, entry_tid=11698]]], state=connected] 2024-04-06T09:38:50.502 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:38:50.504 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=77, tag_tid=12, entry_tid=1' 2024-04-06T09:38:50.505 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:38:50.505 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:38:50.505 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:38:50.505 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:38:50.505 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:38:50.505 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:38:50.505 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:38:50.505 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:38:50.505 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:38:50.506 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:38:50.506 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:38:50.506 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:38:50.506 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:38:50.506 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:38:50.507 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:38:50.552 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 17 2024-04-06T09:38:50.553 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 19 2024-04-06T09:38:50.553 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:38:50.553 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=77, tag_tid=12, entry_tid=1], [object_number=76, tag_tid=12, entry_tid=0], [object_number=79, tag_tid=11, entry_tid=17175], [object_number=78, tag_tid=11, entry_tid=17174]]], state=connected] 2024-04-06T09:38:50.553 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=69, tag_tid=11, entry_tid=11701], [object_number=68, tag_tid=11, entry_tid=11700], [object_number=71, tag_tid=11, entry_tid=11699], [object_number=70, tag_tid=11, entry_tid=11698]]], state=connected] 2024-04-06T09:38:50.556 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:38:50.559 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=69, tag_tid=11, entry_tid=11701' 2024-04-06T09:38:50.559 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=77, tag_tid=12, entry_tid=1' -a 'object_number=77, tag_tid=12, entry_tid=1' = 'object_number=69, tag_tid=11, entry_tid=11701' 2024-04-06T09:38:50.559 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=69, tag_tid=11, entry_tid=11701' '!=' 'object_number=69, tag_tid=11, entry_tid=11597' 2024-04-06T09:38:50.559 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:38:50.559 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=69, tag_tid=11, entry_tid=11701' = 'object_number=69, tag_tid=11, entry_tid=11597' 2024-04-06T09:38:50.559 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=69, tag_tid=11, entry_tid=11701' 2024-04-06T09:38:50.559 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=77, tag_tid=12, entry_tid=1' 2024-04-06T09:38:50.559 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:38:50.559 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:38:50.560 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=12 2024-04-06T09:38:50.560 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=69, tag_tid=11, entry_tid=11701' 2024-04-06T09:38:50.561 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:38:50.561 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:38:50.561 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=11 2024-04-06T09:38:50.562 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=77, tag_tid=12, entry_tid=1' 2024-04-06T09:38:50.562 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:38:50.562 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:38:50.563 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:38:50.564 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=69, tag_tid=11, entry_tid=11701' 2024-04-06T09:38:50.564 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:38:50.564 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:38:50.565 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=11701 2024-04-06T09:38:50.565 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 12 = 11 -a 1 -le 11701 2024-04-06T09:38:50.565 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:38:50.565 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:38:50.565 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:38:50.767 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:38:50.767 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:38:50.767 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:38:50.767 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:38:50.767 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:38:50.767 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:38:50.767 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:38:50.768 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:38:50.768 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:38:50.768 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:38:50.768 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:38:50.768 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:38:50.768 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:38:50.768 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:38:50.768 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:38:50.768 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:38:50.769 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:38:50.772 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:38:50.772 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:38:50.772 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:38:50.772 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:38:50.772 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:38:50.772 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:38:50.772 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:38:50.772 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:38:50.772 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:38:50.772 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:38:50.773 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:38:50.773 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:38:50.773 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:38:50.773 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:38:50.773 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:38:50.773 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:38:50.773 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:38:50.773 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:38:50.773 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:38:50.773 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:38:50.773 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:38:50.773 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:38:50.773 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:38:50.773 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:38:50.774 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:38:50.774 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:38:50.774 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:38:50.774 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:38:50.774 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:38:50.774 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:38:50.774 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:38:50.774 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:38:50.796 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:38:50.797 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:38:50.797 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:38:50.797 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:38:50.929 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:38:50.929 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:38:50.930 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:38:50.930 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:38:50.930 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:38:50.930 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:38:50.930 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:38:50.930 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:38:50.930 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:38:50.930 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:38:50.931 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:38:50.931 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:38:50.931 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:38:50.932 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:38:50.933 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:38:50.933 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:38:50.979 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 17 2024-04-06T09:38:50.979 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 19 2024-04-06T09:38:50.979 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:38:50.979 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=77, tag_tid=12, entry_tid=1], [object_number=76, tag_tid=12, entry_tid=0], [object_number=79, tag_tid=11, entry_tid=17175], [object_number=78, tag_tid=11, entry_tid=17174]]], state=connected] 2024-04-06T09:38:50.979 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=68, tag_tid=11, entry_tid=11792], [object_number=71, tag_tid=11, entry_tid=11791], [object_number=70, tag_tid=11, entry_tid=11790], [object_number=69, tag_tid=11, entry_tid=11789]]], state=connected] 2024-04-06T09:38:50.982 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:38:50.983 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=77, tag_tid=12, entry_tid=1' 2024-04-06T09:38:50.984 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:38:50.985 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:38:50.985 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:38:50.985 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:38:50.985 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:38:50.985 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:38:50.985 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:38:50.985 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:38:50.985 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:38:50.985 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:38:50.985 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:38:50.985 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:38:50.986 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:38:50.986 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:38:50.986 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:38:51.032 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 17 2024-04-06T09:38:51.032 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 19 2024-04-06T09:38:51.032 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:38:51.033 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=77, tag_tid=12, entry_tid=1], [object_number=76, tag_tid=12, entry_tid=0], [object_number=79, tag_tid=11, entry_tid=17175], [object_number=78, tag_tid=11, entry_tid=17174]]], state=connected] 2024-04-06T09:38:51.033 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=68, tag_tid=11, entry_tid=11792], [object_number=71, tag_tid=11, entry_tid=11791], [object_number=70, tag_tid=11, entry_tid=11790], [object_number=69, tag_tid=11, entry_tid=11789]]], state=connected] 2024-04-06T09:38:51.036 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:38:51.037 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=68, tag_tid=11, entry_tid=11792' 2024-04-06T09:38:51.038 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=77, tag_tid=12, entry_tid=1' -a 'object_number=77, tag_tid=12, entry_tid=1' = 'object_number=68, tag_tid=11, entry_tid=11792' 2024-04-06T09:38:51.038 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=68, tag_tid=11, entry_tid=11792' '!=' 'object_number=69, tag_tid=11, entry_tid=11701' 2024-04-06T09:38:51.038 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:38:51.038 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=68, tag_tid=11, entry_tid=11792' = 'object_number=69, tag_tid=11, entry_tid=11701' 2024-04-06T09:38:51.038 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=68, tag_tid=11, entry_tid=11792' 2024-04-06T09:38:51.038 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=77, tag_tid=12, entry_tid=1' 2024-04-06T09:38:51.038 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:38:51.038 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:38:51.039 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=12 2024-04-06T09:38:51.039 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=68, tag_tid=11, entry_tid=11792' 2024-04-06T09:38:51.040 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:38:51.040 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:38:51.040 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=11 2024-04-06T09:38:51.041 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=77, tag_tid=12, entry_tid=1' 2024-04-06T09:38:51.041 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:38:51.041 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:38:51.042 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:38:51.043 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=68, tag_tid=11, entry_tid=11792' 2024-04-06T09:38:51.043 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:38:51.043 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:38:51.044 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=11792 2024-04-06T09:38:51.044 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 12 = 11 -a 1 -le 11792 2024-04-06T09:38:51.044 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:38:51.045 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:38:51.045 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:38:51.246 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:38:51.247 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:38:51.247 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:38:51.247 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:38:51.247 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:38:51.247 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:38:51.247 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:38:51.247 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:38:51.247 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:38:51.247 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:38:51.247 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:38:51.247 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:38:51.247 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:38:51.247 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:38:51.247 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:38:51.248 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:38:51.248 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:38:51.250 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:38:51.251 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:38:51.251 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:38:51.251 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:38:51.251 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:38:51.251 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:38:51.251 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:38:51.251 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:38:51.251 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:38:51.251 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:38:51.251 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:38:51.251 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:38:51.251 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:38:51.251 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:38:51.252 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:38:51.252 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:38:51.252 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:38:51.252 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:38:51.252 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:38:51.252 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:38:51.252 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:38:51.252 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:38:51.252 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:38:51.252 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:38:51.252 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:38:51.252 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:38:51.252 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:38:51.252 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:38:51.252 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:38:51.253 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:38:51.253 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:38:51.253 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:38:51.277 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:38:51.277 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:38:51.277 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:38:51.277 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:38:51.404 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:38:51.405 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:38:51.406 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:38:51.406 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:38:51.406 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:38:51.406 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:38:51.406 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:38:51.406 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:38:51.406 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:38:51.406 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:38:51.407 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:38:51.407 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:38:51.407 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:38:51.408 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:38:51.408 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:38:51.408 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:38:51.454 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 17 2024-04-06T09:38:51.454 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 19 2024-04-06T09:38:51.454 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:38:51.454 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=77, tag_tid=12, entry_tid=1], [object_number=76, tag_tid=12, entry_tid=0], [object_number=79, tag_tid=11, entry_tid=17175], [object_number=78, tag_tid=11, entry_tid=17174]]], state=connected] 2024-04-06T09:38:51.454 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=69, tag_tid=11, entry_tid=11893], [object_number=68, tag_tid=11, entry_tid=11892], [object_number=71, tag_tid=11, entry_tid=11891], [object_number=70, tag_tid=11, entry_tid=11890]]], state=connected] 2024-04-06T09:38:51.457 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:38:51.459 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=77, tag_tid=12, entry_tid=1' 2024-04-06T09:38:51.460 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:38:51.460 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:38:51.460 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:38:51.460 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:38:51.460 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:38:51.460 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:38:51.460 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:38:51.460 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:38:51.460 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:38:51.460 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:38:51.460 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:38:51.460 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:38:51.461 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:38:51.461 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:38:51.461 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:38:51.509 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 17 2024-04-06T09:38:51.509 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 19 2024-04-06T09:38:51.509 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:38:51.509 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=77, tag_tid=12, entry_tid=1], [object_number=76, tag_tid=12, entry_tid=0], [object_number=79, tag_tid=11, entry_tid=17175], [object_number=78, tag_tid=11, entry_tid=17174]]], state=connected] 2024-04-06T09:38:51.509 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=69, tag_tid=11, entry_tid=11893], [object_number=68, tag_tid=11, entry_tid=11892], [object_number=71, tag_tid=11, entry_tid=11891], [object_number=70, tag_tid=11, entry_tid=11890]]], state=connected] 2024-04-06T09:38:51.512 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:38:51.514 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=69, tag_tid=11, entry_tid=11893' 2024-04-06T09:38:51.514 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=77, tag_tid=12, entry_tid=1' -a 'object_number=77, tag_tid=12, entry_tid=1' = 'object_number=69, tag_tid=11, entry_tid=11893' 2024-04-06T09:38:51.514 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=69, tag_tid=11, entry_tid=11893' '!=' 'object_number=68, tag_tid=11, entry_tid=11792' 2024-04-06T09:38:51.514 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:38:51.514 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=69, tag_tid=11, entry_tid=11893' = 'object_number=68, tag_tid=11, entry_tid=11792' 2024-04-06T09:38:51.514 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=69, tag_tid=11, entry_tid=11893' 2024-04-06T09:38:51.515 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=77, tag_tid=12, entry_tid=1' 2024-04-06T09:38:51.515 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:38:51.515 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:38:51.515 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=12 2024-04-06T09:38:51.516 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=69, tag_tid=11, entry_tid=11893' 2024-04-06T09:38:51.516 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:38:51.516 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:38:51.517 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=11 2024-04-06T09:38:51.518 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=77, tag_tid=12, entry_tid=1' 2024-04-06T09:38:51.518 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:38:51.518 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:38:51.519 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:38:51.519 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=69, tag_tid=11, entry_tid=11893' 2024-04-06T09:38:51.520 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:38:51.520 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:38:51.521 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=11893 2024-04-06T09:38:51.521 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 12 = 11 -a 1 -le 11893 2024-04-06T09:38:51.521 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:38:51.521 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:38:51.521 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:38:51.723 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:38:51.723 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:38:51.723 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:38:51.723 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:38:51.723 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:38:51.723 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:38:51.723 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:38:51.723 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:38:51.723 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:38:51.723 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:38:51.724 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:38:51.724 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:38:51.724 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:38:51.724 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:38:51.724 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:38:51.724 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:38:51.724 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:38:51.726 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:38:51.726 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:38:51.726 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:38:51.726 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:38:51.726 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:38:51.726 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:38:51.726 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:38:51.726 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:38:51.726 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:38:51.727 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:38:51.727 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:38:51.727 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:38:51.727 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:38:51.727 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:38:51.727 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:38:51.727 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:38:51.728 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:38:51.728 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:38:51.728 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:38:51.728 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:38:51.728 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:38:51.728 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:38:51.728 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:38:51.728 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:38:51.728 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:38:51.728 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:38:51.728 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:38:51.729 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:38:51.729 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:38:51.729 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:38:51.729 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:38:51.729 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:38:51.753 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:38:51.753 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:38:51.753 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:38:51.753 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:38:51.895 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:38:51.897 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:38:51.897 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:38:51.897 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:38:51.897 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:38:51.897 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:38:51.897 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:38:51.897 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:38:51.897 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:38:51.897 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:38:51.897 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:38:51.898 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:38:51.898 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:38:51.900 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:38:51.900 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:38:51.900 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:38:51.953 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 17 2024-04-06T09:38:51.953 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 19 2024-04-06T09:38:51.953 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:38:51.953 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=77, tag_tid=12, entry_tid=1], [object_number=76, tag_tid=12, entry_tid=0], [object_number=79, tag_tid=11, entry_tid=17175], [object_number=78, tag_tid=11, entry_tid=17174]]], state=connected] 2024-04-06T09:38:51.953 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=71, tag_tid=11, entry_tid=11987], [object_number=70, tag_tid=11, entry_tid=11986], [object_number=69, tag_tid=11, entry_tid=11985], [object_number=68, tag_tid=11, entry_tid=11984]]], state=connected] 2024-04-06T09:38:51.957 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:38:51.958 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=77, tag_tid=12, entry_tid=1' 2024-04-06T09:38:51.959 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:38:51.960 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:38:51.960 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:38:51.960 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:38:51.960 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:38:51.960 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:38:51.960 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:38:51.960 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:38:51.960 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:38:51.960 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:38:51.960 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:38:51.960 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:38:51.961 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:38:51.961 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:38:51.961 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:38:52.007 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 17 2024-04-06T09:38:52.007 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 19 2024-04-06T09:38:52.007 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:38:52.007 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=77, tag_tid=12, entry_tid=1], [object_number=76, tag_tid=12, entry_tid=0], [object_number=79, tag_tid=11, entry_tid=17175], [object_number=78, tag_tid=11, entry_tid=17174]]], state=connected] 2024-04-06T09:38:52.008 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=71, tag_tid=11, entry_tid=11987], [object_number=70, tag_tid=11, entry_tid=11986], [object_number=69, tag_tid=11, entry_tid=11985], [object_number=68, tag_tid=11, entry_tid=11984]]], state=connected] 2024-04-06T09:38:52.011 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:38:52.012 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=71, tag_tid=11, entry_tid=11987' 2024-04-06T09:38:52.013 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=77, tag_tid=12, entry_tid=1' -a 'object_number=77, tag_tid=12, entry_tid=1' = 'object_number=71, tag_tid=11, entry_tid=11987' 2024-04-06T09:38:52.013 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=71, tag_tid=11, entry_tid=11987' '!=' 'object_number=69, tag_tid=11, entry_tid=11893' 2024-04-06T09:38:52.013 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:38:52.013 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=71, tag_tid=11, entry_tid=11987' = 'object_number=69, tag_tid=11, entry_tid=11893' 2024-04-06T09:38:52.013 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=71, tag_tid=11, entry_tid=11987' 2024-04-06T09:38:52.013 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=77, tag_tid=12, entry_tid=1' 2024-04-06T09:38:52.014 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:38:52.014 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:38:52.014 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=12 2024-04-06T09:38:52.015 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=71, tag_tid=11, entry_tid=11987' 2024-04-06T09:38:52.015 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:38:52.015 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:38:52.016 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=11 2024-04-06T09:38:52.016 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=77, tag_tid=12, entry_tid=1' 2024-04-06T09:38:52.016 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:38:52.017 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:38:52.017 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:38:52.018 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=71, tag_tid=11, entry_tid=11987' 2024-04-06T09:38:52.018 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:38:52.018 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:38:52.019 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=11987 2024-04-06T09:38:52.019 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 12 = 11 -a 1 -le 11987 2024-04-06T09:38:52.020 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:38:52.020 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:38:52.020 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:38:52.221 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:38:52.222 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:38:52.222 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:38:52.222 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:38:52.222 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:38:52.222 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:38:52.222 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:38:52.222 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:38:52.222 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:38:52.222 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:38:52.222 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:38:52.222 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:38:52.222 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:38:52.223 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:38:52.223 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:38:52.223 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:38:52.223 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:38:52.226 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:38:52.226 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:38:52.226 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:38:52.226 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:38:52.226 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:38:52.226 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:38:52.226 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:38:52.226 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:38:52.226 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:38:52.226 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:38:52.226 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:38:52.227 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:38:52.227 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:38:52.227 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:38:52.227 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:38:52.227 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:38:52.228 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:38:52.228 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:38:52.228 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:38:52.228 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:38:52.228 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:38:52.228 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:38:52.228 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:38:52.228 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:38:52.228 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:38:52.228 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:38:52.228 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:38:52.228 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:38:52.228 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:38:52.229 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:38:52.229 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:38:52.229 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:38:52.253 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:38:52.253 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:38:52.254 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:38:52.254 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:38:52.389 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:38:52.390 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:38:52.390 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:38:52.391 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:38:52.391 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:38:52.391 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:38:52.391 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:38:52.391 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:38:52.391 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:38:52.391 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:38:52.391 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:38:52.392 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:38:52.392 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:38:52.392 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:38:52.393 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:38:52.393 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:38:52.438 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 17 2024-04-06T09:38:52.438 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 19 2024-04-06T09:38:52.438 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:38:52.438 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=77, tag_tid=12, entry_tid=1], [object_number=76, tag_tid=12, entry_tid=0], [object_number=79, tag_tid=11, entry_tid=17175], [object_number=78, tag_tid=11, entry_tid=17174]]], state=connected] 2024-04-06T09:38:52.438 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=69, tag_tid=11, entry_tid=12073], [object_number=68, tag_tid=11, entry_tid=12072], [object_number=71, tag_tid=11, entry_tid=12071], [object_number=70, tag_tid=11, entry_tid=12070]]], state=connected] 2024-04-06T09:38:52.441 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:38:52.442 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=77, tag_tid=12, entry_tid=1' 2024-04-06T09:38:52.444 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:38:52.444 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:38:52.444 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:38:52.444 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:38:52.444 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:38:52.444 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:38:52.444 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:38:52.444 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:38:52.444 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:38:52.444 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:38:52.444 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:38:52.444 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:38:52.445 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:38:52.445 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:38:52.445 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:38:52.496 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 17 2024-04-06T09:38:52.497 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 19 2024-04-06T09:38:52.497 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:38:52.497 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=77, tag_tid=12, entry_tid=1], [object_number=76, tag_tid=12, entry_tid=0], [object_number=79, tag_tid=11, entry_tid=17175], [object_number=78, tag_tid=11, entry_tid=17174]]], state=connected] 2024-04-06T09:38:52.497 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=69, tag_tid=11, entry_tid=12073], [object_number=68, tag_tid=11, entry_tid=12072], [object_number=71, tag_tid=11, entry_tid=12071], [object_number=70, tag_tid=11, entry_tid=12070]]], state=connected] 2024-04-06T09:38:52.500 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:38:52.502 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=69, tag_tid=11, entry_tid=12073' 2024-04-06T09:38:52.502 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=77, tag_tid=12, entry_tid=1' -a 'object_number=77, tag_tid=12, entry_tid=1' = 'object_number=69, tag_tid=11, entry_tid=12073' 2024-04-06T09:38:52.503 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=69, tag_tid=11, entry_tid=12073' '!=' 'object_number=71, tag_tid=11, entry_tid=11987' 2024-04-06T09:38:52.503 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:38:52.503 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=69, tag_tid=11, entry_tid=12073' = 'object_number=71, tag_tid=11, entry_tid=11987' 2024-04-06T09:38:52.503 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=69, tag_tid=11, entry_tid=12073' 2024-04-06T09:38:52.503 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=77, tag_tid=12, entry_tid=1' 2024-04-06T09:38:52.503 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:38:52.503 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:38:52.504 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=12 2024-04-06T09:38:52.504 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=69, tag_tid=11, entry_tid=12073' 2024-04-06T09:38:52.504 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:38:52.504 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:38:52.505 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=11 2024-04-06T09:38:52.505 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=77, tag_tid=12, entry_tid=1' 2024-04-06T09:38:52.506 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:38:52.506 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:38:52.506 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:38:52.507 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=69, tag_tid=11, entry_tid=12073' 2024-04-06T09:38:52.507 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:38:52.507 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:38:52.508 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=12073 2024-04-06T09:38:52.508 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 12 = 11 -a 1 -le 12073 2024-04-06T09:38:52.509 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:38:52.509 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:38:52.509 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:38:52.710 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:38:52.710 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:38:52.711 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:38:52.711 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:38:52.711 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:38:52.711 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:38:52.711 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:38:52.711 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:38:52.711 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:38:52.711 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:38:52.711 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:38:52.711 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:38:52.711 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:38:52.711 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:38:52.711 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:38:52.712 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:38:52.712 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:38:52.715 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:38:52.715 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:38:52.715 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:38:52.715 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:38:52.715 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:38:52.715 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:38:52.715 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:38:52.715 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:38:52.715 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:38:52.716 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:38:52.716 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:38:52.716 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:38:52.716 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:38:52.716 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:38:52.716 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:38:52.716 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:38:52.716 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:38:52.716 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:38:52.716 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:38:52.716 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:38:52.716 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:38:52.716 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:38:52.716 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:38:52.716 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:38:52.717 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:38:52.717 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:38:52.717 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:38:52.717 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:38:52.717 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:38:52.717 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:38:52.717 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:38:52.717 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:38:52.740 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:38:52.740 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:38:52.740 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:38:52.740 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:38:52.872 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:38:52.873 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:38:52.873 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:38:52.873 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:38:52.873 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:38:52.873 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:38:52.873 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:38:52.873 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:38:52.873 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:38:52.874 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:38:52.874 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:38:52.874 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:38:52.875 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:38:52.877 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:38:52.877 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:38:52.877 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:38:52.933 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 17 2024-04-06T09:38:52.933 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 19 2024-04-06T09:38:52.934 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:38:52.934 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=77, tag_tid=12, entry_tid=1], [object_number=76, tag_tid=12, entry_tid=0], [object_number=79, tag_tid=11, entry_tid=17175], [object_number=78, tag_tid=11, entry_tid=17174]]], state=connected] 2024-04-06T09:38:52.934 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=69, tag_tid=11, entry_tid=12157], [object_number=68, tag_tid=11, entry_tid=12156], [object_number=71, tag_tid=11, entry_tid=12155], [object_number=70, tag_tid=11, entry_tid=12154]]], state=connected] 2024-04-06T09:38:52.937 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:38:52.938 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=77, tag_tid=12, entry_tid=1' 2024-04-06T09:38:52.939 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:38:52.939 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:38:52.940 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:38:52.940 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:38:52.940 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:38:52.940 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:38:52.940 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:38:52.940 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:38:52.940 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:38:52.940 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:38:52.940 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:38:52.940 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:38:52.941 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:38:52.941 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:38:52.941 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:38:52.988 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 17 2024-04-06T09:38:52.988 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 19 2024-04-06T09:38:52.988 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:38:52.988 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=77, tag_tid=12, entry_tid=1], [object_number=76, tag_tid=12, entry_tid=0], [object_number=79, tag_tid=11, entry_tid=17175], [object_number=78, tag_tid=11, entry_tid=17174]]], state=connected] 2024-04-06T09:38:52.988 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=69, tag_tid=11, entry_tid=12157], [object_number=68, tag_tid=11, entry_tid=12156], [object_number=71, tag_tid=11, entry_tid=12155], [object_number=70, tag_tid=11, entry_tid=12154]]], state=connected] 2024-04-06T09:38:52.991 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:38:52.993 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=69, tag_tid=11, entry_tid=12157' 2024-04-06T09:38:52.993 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=77, tag_tid=12, entry_tid=1' -a 'object_number=77, tag_tid=12, entry_tid=1' = 'object_number=69, tag_tid=11, entry_tid=12157' 2024-04-06T09:38:52.994 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=69, tag_tid=11, entry_tid=12157' '!=' 'object_number=69, tag_tid=11, entry_tid=12073' 2024-04-06T09:38:52.994 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:38:52.994 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=69, tag_tid=11, entry_tid=12157' = 'object_number=69, tag_tid=11, entry_tid=12073' 2024-04-06T09:38:52.994 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=69, tag_tid=11, entry_tid=12157' 2024-04-06T09:38:52.994 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=77, tag_tid=12, entry_tid=1' 2024-04-06T09:38:52.994 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:38:52.994 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:38:52.995 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=12 2024-04-06T09:38:52.995 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=69, tag_tid=11, entry_tid=12157' 2024-04-06T09:38:52.995 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:38:52.995 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:38:52.996 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=11 2024-04-06T09:38:52.997 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=77, tag_tid=12, entry_tid=1' 2024-04-06T09:38:52.997 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:38:52.997 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:38:52.998 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:38:52.999 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=69, tag_tid=11, entry_tid=12157' 2024-04-06T09:38:52.999 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:38:52.999 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:38:53.000 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=12157 2024-04-06T09:38:53.000 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 12 = 11 -a 1 -le 12157 2024-04-06T09:38:53.000 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:38:53.000 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:38:53.000 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:38:53.202 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:38:53.202 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:38:53.202 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:38:53.202 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:38:53.202 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:38:53.202 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:38:53.203 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:38:53.203 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:38:53.203 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:38:53.203 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:38:53.203 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:38:53.203 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:38:53.203 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:38:53.203 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:38:53.203 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:38:53.204 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:38:53.204 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:38:53.205 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:38:53.205 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:38:53.205 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:38:53.205 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:38:53.205 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:38:53.205 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:38:53.205 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:38:53.205 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:38:53.205 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:38:53.205 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:38:53.206 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:38:53.206 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:38:53.206 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:38:53.206 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:38:53.206 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:38:53.206 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:38:53.207 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:38:53.207 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:38:53.207 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:38:53.207 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:38:53.207 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:38:53.207 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:38:53.207 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:38:53.207 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:38:53.207 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:38:53.208 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:38:53.208 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:38:53.208 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:38:53.208 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:38:53.208 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:38:53.208 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:38:53.208 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:38:53.233 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:38:53.233 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:38:53.233 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:38:53.233 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:38:53.366 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:38:53.368 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:38:53.368 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:38:53.368 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:38:53.368 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:38:53.368 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:38:53.368 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:38:53.368 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:38:53.368 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:38:53.368 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:38:53.368 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:38:53.369 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:38:53.369 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:38:53.371 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:38:53.372 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:38:53.372 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:38:53.418 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 17 2024-04-06T09:38:53.419 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 19 2024-04-06T09:38:53.419 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:38:53.419 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=77, tag_tid=12, entry_tid=1], [object_number=76, tag_tid=12, entry_tid=0], [object_number=79, tag_tid=11, entry_tid=17175], [object_number=78, tag_tid=11, entry_tid=17174]]], state=connected] 2024-04-06T09:38:53.419 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=70, tag_tid=11, entry_tid=12242], [object_number=69, tag_tid=11, entry_tid=12241], [object_number=68, tag_tid=11, entry_tid=12240], [object_number=71, tag_tid=11, entry_tid=12239]]], state=connected] 2024-04-06T09:38:53.422 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:38:53.423 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=77, tag_tid=12, entry_tid=1' 2024-04-06T09:38:53.424 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:38:53.424 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:38:53.424 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:38:53.425 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:38:53.425 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:38:53.425 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:38:53.425 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:38:53.425 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:38:53.425 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:38:53.425 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:38:53.425 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:38:53.425 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:38:53.426 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:38:53.426 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:38:53.426 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:38:53.472 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 17 2024-04-06T09:38:53.472 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 19 2024-04-06T09:38:53.472 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:38:53.472 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=77, tag_tid=12, entry_tid=1], [object_number=76, tag_tid=12, entry_tid=0], [object_number=79, tag_tid=11, entry_tid=17175], [object_number=78, tag_tid=11, entry_tid=17174]]], state=connected] 2024-04-06T09:38:53.472 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=70, tag_tid=11, entry_tid=12242], [object_number=69, tag_tid=11, entry_tid=12241], [object_number=68, tag_tid=11, entry_tid=12240], [object_number=71, tag_tid=11, entry_tid=12239]]], state=connected] 2024-04-06T09:38:53.476 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:38:53.478 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=70, tag_tid=11, entry_tid=12242' 2024-04-06T09:38:53.478 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=77, tag_tid=12, entry_tid=1' -a 'object_number=77, tag_tid=12, entry_tid=1' = 'object_number=70, tag_tid=11, entry_tid=12242' 2024-04-06T09:38:53.478 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=70, tag_tid=11, entry_tid=12242' '!=' 'object_number=69, tag_tid=11, entry_tid=12157' 2024-04-06T09:38:53.478 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:38:53.478 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=70, tag_tid=11, entry_tid=12242' = 'object_number=69, tag_tid=11, entry_tid=12157' 2024-04-06T09:38:53.478 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=70, tag_tid=11, entry_tid=12242' 2024-04-06T09:38:53.478 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=77, tag_tid=12, entry_tid=1' 2024-04-06T09:38:53.478 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:38:53.478 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:38:53.479 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=12 2024-04-06T09:38:53.480 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=70, tag_tid=11, entry_tid=12242' 2024-04-06T09:38:53.480 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:38:53.480 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:38:53.481 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=11 2024-04-06T09:38:53.481 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=77, tag_tid=12, entry_tid=1' 2024-04-06T09:38:53.481 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:38:53.482 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:38:53.482 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:38:53.483 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=70, tag_tid=11, entry_tid=12242' 2024-04-06T09:38:53.483 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:38:53.483 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:38:53.485 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=12242 2024-04-06T09:38:53.485 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 12 = 11 -a 1 -le 12242 2024-04-06T09:38:53.485 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:38:53.485 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:38:53.485 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:38:53.687 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:38:53.687 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:38:53.687 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:38:53.687 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:38:53.687 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:38:53.687 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:38:53.687 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:38:53.688 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:38:53.688 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:38:53.688 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:38:53.688 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:38:53.688 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:38:53.688 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:38:53.688 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:38:53.688 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:38:53.688 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:38:53.689 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:38:53.691 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:38:53.691 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:38:53.691 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:38:53.692 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:38:53.692 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:38:53.692 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:38:53.692 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:38:53.692 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:38:53.692 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:38:53.692 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:38:53.692 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:38:53.692 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:38:53.692 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:38:53.692 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:38:53.692 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:38:53.692 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:38:53.694 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:38:53.694 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:38:53.694 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:38:53.694 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:38:53.694 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:38:53.694 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:38:53.694 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:38:53.694 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:38:53.694 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:38:53.695 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:38:53.695 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:38:53.695 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:38:53.695 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:38:53.695 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:38:53.695 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:38:53.695 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:38:53.719 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:38:53.719 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:38:53.719 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:38:53.719 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:38:53.867 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:38:53.868 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:38:53.868 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:38:53.868 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:38:53.868 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:38:53.868 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:38:53.869 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:38:53.869 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:38:53.869 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:38:53.869 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:38:53.869 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:38:53.869 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:38:53.870 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:38:53.872 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:38:53.873 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:38:53.873 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:38:53.919 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 17 2024-04-06T09:38:53.919 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 19 2024-04-06T09:38:53.920 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:38:53.920 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=77, tag_tid=12, entry_tid=1], [object_number=76, tag_tid=12, entry_tid=0], [object_number=79, tag_tid=11, entry_tid=17175], [object_number=78, tag_tid=11, entry_tid=17174]]], state=connected] 2024-04-06T09:38:53.920 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=70, tag_tid=11, entry_tid=12338], [object_number=69, tag_tid=11, entry_tid=12337], [object_number=68, tag_tid=11, entry_tid=12336], [object_number=71, tag_tid=11, entry_tid=12335]]], state=connected] 2024-04-06T09:38:53.923 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:38:53.924 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=77, tag_tid=12, entry_tid=1' 2024-04-06T09:38:53.925 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:38:53.925 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:38:53.925 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:38:53.925 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:38:53.925 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:38:53.926 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:38:53.926 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:38:53.926 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:38:53.926 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:38:53.926 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:38:53.926 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:38:53.926 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:38:53.927 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:38:53.927 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:38:53.927 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:38:53.972 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 17 2024-04-06T09:38:53.973 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 19 2024-04-06T09:38:53.973 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:38:53.973 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=77, tag_tid=12, entry_tid=1], [object_number=76, tag_tid=12, entry_tid=0], [object_number=79, tag_tid=11, entry_tid=17175], [object_number=78, tag_tid=11, entry_tid=17174]]], state=connected] 2024-04-06T09:38:53.973 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=70, tag_tid=11, entry_tid=12338], [object_number=69, tag_tid=11, entry_tid=12337], [object_number=68, tag_tid=11, entry_tid=12336], [object_number=71, tag_tid=11, entry_tid=12335]]], state=connected] 2024-04-06T09:38:53.976 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:38:53.978 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=70, tag_tid=11, entry_tid=12338' 2024-04-06T09:38:53.978 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=77, tag_tid=12, entry_tid=1' -a 'object_number=77, tag_tid=12, entry_tid=1' = 'object_number=70, tag_tid=11, entry_tid=12338' 2024-04-06T09:38:53.978 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=70, tag_tid=11, entry_tid=12338' '!=' 'object_number=70, tag_tid=11, entry_tid=12242' 2024-04-06T09:38:53.979 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:38:53.979 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=70, tag_tid=11, entry_tid=12338' = 'object_number=70, tag_tid=11, entry_tid=12242' 2024-04-06T09:38:53.979 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=70, tag_tid=11, entry_tid=12338' 2024-04-06T09:38:53.979 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=77, tag_tid=12, entry_tid=1' 2024-04-06T09:38:53.979 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:38:53.979 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:38:53.979 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=12 2024-04-06T09:38:53.980 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=70, tag_tid=11, entry_tid=12338' 2024-04-06T09:38:53.980 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:38:53.980 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:38:53.981 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=11 2024-04-06T09:38:53.982 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=77, tag_tid=12, entry_tid=1' 2024-04-06T09:38:53.982 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:38:53.982 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:38:53.983 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:38:53.983 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=70, tag_tid=11, entry_tid=12338' 2024-04-06T09:38:53.983 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:38:53.983 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:38:53.984 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=12338 2024-04-06T09:38:53.985 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 12 = 11 -a 1 -le 12338 2024-04-06T09:38:53.985 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:38:53.985 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:38:53.985 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:38:54.186 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:38:54.187 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:38:54.187 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:38:54.187 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:38:54.187 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:38:54.187 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:38:54.187 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:38:54.187 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:38:54.187 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:38:54.187 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:38:54.187 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:38:54.187 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:38:54.187 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:38:54.187 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:38:54.187 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:38:54.188 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:38:54.188 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:38:54.192 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:38:54.192 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:38:54.192 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:38:54.192 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:38:54.192 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:38:54.192 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:38:54.192 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:38:54.192 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:38:54.192 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:38:54.192 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:38:54.192 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:38:54.192 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:38:54.193 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:38:54.193 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:38:54.193 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:38:54.193 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:38:54.193 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:38:54.193 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:38:54.193 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:38:54.193 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:38:54.193 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:38:54.193 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:38:54.193 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:38:54.193 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:38:54.193 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:38:54.193 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:38:54.194 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:38:54.194 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:38:54.194 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:38:54.194 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:38:54.194 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:38:54.194 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:38:54.216 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:38:54.217 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:38:54.217 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:38:54.217 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:38:54.347 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:38:54.348 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:38:54.348 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:38:54.348 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:38:54.348 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:38:54.349 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:38:54.349 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:38:54.349 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:38:54.349 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:38:54.349 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:38:54.349 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:38:54.350 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:38:54.350 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:38:54.352 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:38:54.352 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:38:54.352 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:38:54.408 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 17 2024-04-06T09:38:54.408 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 19 2024-04-06T09:38:54.408 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:38:54.408 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=77, tag_tid=12, entry_tid=1], [object_number=76, tag_tid=12, entry_tid=0], [object_number=79, tag_tid=11, entry_tid=17175], [object_number=78, tag_tid=11, entry_tid=17174]]], state=connected] 2024-04-06T09:38:54.408 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=69, tag_tid=11, entry_tid=12437], [object_number=68, tag_tid=11, entry_tid=12436], [object_number=71, tag_tid=11, entry_tid=12435], [object_number=70, tag_tid=11, entry_tid=12434]]], state=connected] 2024-04-06T09:38:54.412 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:38:54.413 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=77, tag_tid=12, entry_tid=1' 2024-04-06T09:38:54.414 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:38:54.414 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:38:54.415 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:38:54.415 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:38:54.415 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:38:54.415 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:38:54.415 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:38:54.415 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:38:54.415 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:38:54.415 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:38:54.415 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:38:54.415 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:38:54.416 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:38:54.416 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:38:54.416 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:38:54.463 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 17 2024-04-06T09:38:54.463 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 19 2024-04-06T09:38:54.463 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:38:54.463 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=77, tag_tid=12, entry_tid=1], [object_number=76, tag_tid=12, entry_tid=0], [object_number=79, tag_tid=11, entry_tid=17175], [object_number=78, tag_tid=11, entry_tid=17174]]], state=connected] 2024-04-06T09:38:54.463 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=69, tag_tid=11, entry_tid=12437], [object_number=68, tag_tid=11, entry_tid=12436], [object_number=71, tag_tid=11, entry_tid=12435], [object_number=70, tag_tid=11, entry_tid=12434]]], state=connected] 2024-04-06T09:38:54.467 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:38:54.469 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=69, tag_tid=11, entry_tid=12437' 2024-04-06T09:38:54.469 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=77, tag_tid=12, entry_tid=1' -a 'object_number=77, tag_tid=12, entry_tid=1' = 'object_number=69, tag_tid=11, entry_tid=12437' 2024-04-06T09:38:54.469 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=69, tag_tid=11, entry_tid=12437' '!=' 'object_number=70, tag_tid=11, entry_tid=12338' 2024-04-06T09:38:54.469 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:38:54.469 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=69, tag_tid=11, entry_tid=12437' = 'object_number=70, tag_tid=11, entry_tid=12338' 2024-04-06T09:38:54.469 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=69, tag_tid=11, entry_tid=12437' 2024-04-06T09:38:54.470 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=77, tag_tid=12, entry_tid=1' 2024-04-06T09:38:54.470 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:38:54.470 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:38:54.470 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=12 2024-04-06T09:38:54.471 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=69, tag_tid=11, entry_tid=12437' 2024-04-06T09:38:54.471 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:38:54.471 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:38:54.472 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=11 2024-04-06T09:38:54.473 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=77, tag_tid=12, entry_tid=1' 2024-04-06T09:38:54.473 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:38:54.473 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:38:54.474 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:38:54.474 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=69, tag_tid=11, entry_tid=12437' 2024-04-06T09:38:54.475 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:38:54.475 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:38:54.476 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=12437 2024-04-06T09:38:54.476 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 12 = 11 -a 1 -le 12437 2024-04-06T09:38:54.476 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:38:54.476 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:38:54.476 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:38:54.677 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:38:54.678 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:38:54.678 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:38:54.678 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:38:54.678 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:38:54.678 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:38:54.678 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:38:54.678 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:38:54.678 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:38:54.678 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:38:54.678 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:38:54.678 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:38:54.678 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:38:54.678 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:38:54.679 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:38:54.679 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:38:54.679 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:38:54.681 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:38:54.681 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:38:54.681 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:38:54.682 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:38:54.682 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:38:54.682 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:38:54.682 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:38:54.682 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:38:54.682 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:38:54.682 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:38:54.682 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:38:54.682 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:38:54.682 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:38:54.682 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:38:54.682 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:38:54.682 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:38:54.684 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:38:54.684 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:38:54.684 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:38:54.684 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:38:54.684 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:38:54.684 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:38:54.684 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:38:54.684 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:38:54.684 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:38:54.684 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:38:54.685 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:38:54.685 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:38:54.685 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:38:54.685 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:38:54.685 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:38:54.685 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:38:54.708 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:38:54.709 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:38:54.709 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:38:54.709 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:38:54.842 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:38:54.843 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:38:54.843 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:38:54.843 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:38:54.843 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:38:54.843 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:38:54.843 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:38:54.843 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:38:54.843 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:38:54.844 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:38:54.844 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:38:54.844 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:38:54.844 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:38:54.846 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:38:54.846 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:38:54.846 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:38:54.891 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 17 2024-04-06T09:38:54.892 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 19 2024-04-06T09:38:54.892 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:38:54.892 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=77, tag_tid=12, entry_tid=1], [object_number=76, tag_tid=12, entry_tid=0], [object_number=79, tag_tid=11, entry_tid=17175], [object_number=78, tag_tid=11, entry_tid=17174]]], state=connected] 2024-04-06T09:38:54.892 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=68, tag_tid=11, entry_tid=12540], [object_number=71, tag_tid=11, entry_tid=12539], [object_number=70, tag_tid=11, entry_tid=12538], [object_number=69, tag_tid=11, entry_tid=12537]]], state=connected] 2024-04-06T09:38:54.895 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:38:54.897 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=77, tag_tid=12, entry_tid=1' 2024-04-06T09:38:54.898 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:38:54.898 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:38:54.898 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:38:54.898 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:38:54.898 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:38:54.898 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:38:54.898 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:38:54.898 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:38:54.899 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:38:54.899 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:38:54.899 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:38:54.899 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:38:54.899 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:38:54.899 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:38:54.899 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:38:54.945 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 17 2024-04-06T09:38:54.945 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 19 2024-04-06T09:38:54.945 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:38:54.945 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=77, tag_tid=12, entry_tid=1], [object_number=76, tag_tid=12, entry_tid=0], [object_number=79, tag_tid=11, entry_tid=17175], [object_number=78, tag_tid=11, entry_tid=17174]]], state=connected] 2024-04-06T09:38:54.945 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=68, tag_tid=11, entry_tid=12540], [object_number=71, tag_tid=11, entry_tid=12539], [object_number=70, tag_tid=11, entry_tid=12538], [object_number=69, tag_tid=11, entry_tid=12537]]], state=connected] 2024-04-06T09:38:54.948 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:38:54.950 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=68, tag_tid=11, entry_tid=12540' 2024-04-06T09:38:54.951 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=77, tag_tid=12, entry_tid=1' -a 'object_number=77, tag_tid=12, entry_tid=1' = 'object_number=68, tag_tid=11, entry_tid=12540' 2024-04-06T09:38:54.951 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=68, tag_tid=11, entry_tid=12540' '!=' 'object_number=69, tag_tid=11, entry_tid=12437' 2024-04-06T09:38:54.951 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:38:54.951 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=68, tag_tid=11, entry_tid=12540' = 'object_number=69, tag_tid=11, entry_tid=12437' 2024-04-06T09:38:54.951 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=68, tag_tid=11, entry_tid=12540' 2024-04-06T09:38:54.951 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=77, tag_tid=12, entry_tid=1' 2024-04-06T09:38:54.951 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:38:54.951 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:38:54.951 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=12 2024-04-06T09:38:54.952 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=68, tag_tid=11, entry_tid=12540' 2024-04-06T09:38:54.953 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:38:54.953 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:38:54.953 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=11 2024-04-06T09:38:54.954 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=77, tag_tid=12, entry_tid=1' 2024-04-06T09:38:54.954 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:38:54.954 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:38:54.955 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:38:54.956 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=68, tag_tid=11, entry_tid=12540' 2024-04-06T09:38:54.956 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:38:54.956 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:38:54.957 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=12540 2024-04-06T09:38:54.957 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 12 = 11 -a 1 -le 12540 2024-04-06T09:38:54.957 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:38:54.957 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:38:54.957 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:38:55.159 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:38:55.159 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:38:55.159 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:38:55.159 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:38:55.159 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:38:55.160 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:38:55.160 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:38:55.160 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:38:55.160 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:38:55.160 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:38:55.160 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:38:55.160 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:38:55.160 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:38:55.160 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:38:55.160 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:38:55.161 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:38:55.161 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:38:55.162 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:38:55.162 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:38:55.162 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:38:55.162 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:38:55.162 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:38:55.162 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:38:55.163 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:38:55.163 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:38:55.163 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:38:55.163 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:38:55.163 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:38:55.163 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:38:55.163 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:38:55.163 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:38:55.163 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:38:55.163 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:38:55.164 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:38:55.164 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:38:55.164 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:38:55.164 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:38:55.165 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:38:55.165 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:38:55.165 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:38:55.165 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:38:55.165 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:38:55.165 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:38:55.165 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:38:55.165 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:38:55.165 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:38:55.165 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:38:55.165 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:38:55.165 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:38:55.189 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:38:55.190 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:38:55.190 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:38:55.190 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:38:55.318 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:38:55.319 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:38:55.319 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:38:55.319 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:38:55.319 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:38:55.319 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:38:55.319 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:38:55.319 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:38:55.319 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:38:55.320 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:38:55.320 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:38:55.320 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:38:55.320 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:38:55.322 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:38:55.322 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:38:55.322 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:38:55.370 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 17 2024-04-06T09:38:55.370 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 19 2024-04-06T09:38:55.370 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:38:55.370 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=77, tag_tid=12, entry_tid=1], [object_number=76, tag_tid=12, entry_tid=0], [object_number=79, tag_tid=11, entry_tid=17175], [object_number=78, tag_tid=11, entry_tid=17174]]], state=connected] 2024-04-06T09:38:55.370 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=70, tag_tid=11, entry_tid=12650], [object_number=69, tag_tid=11, entry_tid=12649], [object_number=68, tag_tid=11, entry_tid=12648], [object_number=71, tag_tid=11, entry_tid=12647]]], state=connected] 2024-04-06T09:38:55.377 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:38:55.378 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=77, tag_tid=12, entry_tid=1' 2024-04-06T09:38:55.379 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:38:55.379 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:38:55.379 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:38:55.379 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:38:55.379 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:38:55.379 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:38:55.379 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:38:55.379 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:38:55.379 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:38:55.379 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:38:55.379 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:38:55.379 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:38:55.380 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:38:55.380 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:38:55.380 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:38:55.427 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 17 2024-04-06T09:38:55.427 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 19 2024-04-06T09:38:55.427 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:38:55.427 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=77, tag_tid=12, entry_tid=1], [object_number=76, tag_tid=12, entry_tid=0], [object_number=79, tag_tid=11, entry_tid=17175], [object_number=78, tag_tid=11, entry_tid=17174]]], state=connected] 2024-04-06T09:38:55.427 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=70, tag_tid=11, entry_tid=12650], [object_number=69, tag_tid=11, entry_tid=12649], [object_number=68, tag_tid=11, entry_tid=12648], [object_number=71, tag_tid=11, entry_tid=12647]]], state=connected] 2024-04-06T09:38:55.431 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:38:55.433 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=70, tag_tid=11, entry_tid=12650' 2024-04-06T09:38:55.433 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=77, tag_tid=12, entry_tid=1' -a 'object_number=77, tag_tid=12, entry_tid=1' = 'object_number=70, tag_tid=11, entry_tid=12650' 2024-04-06T09:38:55.433 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=70, tag_tid=11, entry_tid=12650' '!=' 'object_number=68, tag_tid=11, entry_tid=12540' 2024-04-06T09:38:55.433 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:38:55.433 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=70, tag_tid=11, entry_tid=12650' = 'object_number=68, tag_tid=11, entry_tid=12540' 2024-04-06T09:38:55.433 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=70, tag_tid=11, entry_tid=12650' 2024-04-06T09:38:55.433 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=77, tag_tid=12, entry_tid=1' 2024-04-06T09:38:55.433 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:38:55.433 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:38:55.434 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=12 2024-04-06T09:38:55.435 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=70, tag_tid=11, entry_tid=12650' 2024-04-06T09:38:55.435 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:38:55.435 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:38:55.436 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=11 2024-04-06T09:38:55.436 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=77, tag_tid=12, entry_tid=1' 2024-04-06T09:38:55.436 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:38:55.436 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:38:55.437 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:38:55.438 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=70, tag_tid=11, entry_tid=12650' 2024-04-06T09:38:55.438 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:38:55.438 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:38:55.439 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=12650 2024-04-06T09:38:55.439 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 12 = 11 -a 1 -le 12650 2024-04-06T09:38:55.440 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:38:55.440 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:38:55.440 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:38:55.641 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:38:55.641 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:38:55.641 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:38:55.642 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:38:55.642 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:38:55.642 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:38:55.642 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:38:55.642 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:38:55.642 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:38:55.642 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:38:55.642 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:38:55.642 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:38:55.642 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:38:55.642 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:38:55.642 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:38:55.643 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:38:55.643 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:38:55.646 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:38:55.646 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:38:55.646 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:38:55.646 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:38:55.647 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:38:55.647 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:38:55.647 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:38:55.647 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:38:55.647 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:38:55.647 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:38:55.647 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:38:55.647 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:38:55.647 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:38:55.647 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:38:55.647 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:38:55.647 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:38:55.647 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:38:55.647 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:38:55.647 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:38:55.648 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:38:55.648 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:38:55.648 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:38:55.648 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:38:55.648 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:38:55.648 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:38:55.648 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:38:55.648 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:38:55.648 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:38:55.648 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:38:55.648 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:38:55.648 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:38:55.648 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:38:55.672 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:38:55.672 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:38:55.672 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:38:55.672 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:38:55.798 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:38:55.799 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:38:55.799 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:38:55.799 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:38:55.799 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:38:55.800 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:38:55.800 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:38:55.800 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:38:55.800 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:38:55.800 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:38:55.800 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:38:55.800 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:38:55.800 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:38:55.802 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:38:55.802 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:38:55.802 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:38:55.853 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 17 2024-04-06T09:38:55.853 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 19 2024-04-06T09:38:55.853 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:38:55.854 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=77, tag_tid=12, entry_tid=1], [object_number=76, tag_tid=12, entry_tid=0], [object_number=79, tag_tid=11, entry_tid=17175], [object_number=78, tag_tid=11, entry_tid=17174]]], state=connected] 2024-04-06T09:38:55.854 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=71, tag_tid=11, entry_tid=12759], [object_number=70, tag_tid=11, entry_tid=12758], [object_number=69, tag_tid=11, entry_tid=12757], [object_number=68, tag_tid=11, entry_tid=12756]]], state=connected] 2024-04-06T09:38:55.857 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:38:55.858 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=77, tag_tid=12, entry_tid=1' 2024-04-06T09:38:55.860 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:38:55.860 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:38:55.860 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:38:55.860 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:38:55.860 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:38:55.860 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:38:55.860 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:38:55.860 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:38:55.860 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:38:55.860 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:38:55.860 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:38:55.860 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:38:55.861 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:38:55.861 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:38:55.861 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:38:55.908 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 17 2024-04-06T09:38:55.908 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 19 2024-04-06T09:38:55.908 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:38:55.908 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=77, tag_tid=12, entry_tid=1], [object_number=76, tag_tid=12, entry_tid=0], [object_number=79, tag_tid=11, entry_tid=17175], [object_number=78, tag_tid=11, entry_tid=17174]]], state=connected] 2024-04-06T09:38:55.908 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=71, tag_tid=11, entry_tid=12759], [object_number=70, tag_tid=11, entry_tid=12758], [object_number=69, tag_tid=11, entry_tid=12757], [object_number=68, tag_tid=11, entry_tid=12756]]], state=connected] 2024-04-06T09:38:55.912 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:38:55.913 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=71, tag_tid=11, entry_tid=12759' 2024-04-06T09:38:55.913 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=77, tag_tid=12, entry_tid=1' -a 'object_number=77, tag_tid=12, entry_tid=1' = 'object_number=71, tag_tid=11, entry_tid=12759' 2024-04-06T09:38:55.913 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=71, tag_tid=11, entry_tid=12759' '!=' 'object_number=70, tag_tid=11, entry_tid=12650' 2024-04-06T09:38:55.913 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:38:55.913 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=71, tag_tid=11, entry_tid=12759' = 'object_number=70, tag_tid=11, entry_tid=12650' 2024-04-06T09:38:55.914 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=71, tag_tid=11, entry_tid=12759' 2024-04-06T09:38:55.914 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=77, tag_tid=12, entry_tid=1' 2024-04-06T09:38:55.914 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:38:55.914 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:38:55.915 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=12 2024-04-06T09:38:55.915 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=71, tag_tid=11, entry_tid=12759' 2024-04-06T09:38:55.915 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:38:55.916 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:38:55.916 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=11 2024-04-06T09:38:55.917 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=77, tag_tid=12, entry_tid=1' 2024-04-06T09:38:55.917 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:38:55.917 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:38:55.918 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:38:55.919 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=71, tag_tid=11, entry_tid=12759' 2024-04-06T09:38:55.919 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:38:55.919 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:38:55.920 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=12759 2024-04-06T09:38:55.920 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 12 = 11 -a 1 -le 12759 2024-04-06T09:38:55.921 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:38:55.921 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:38:55.921 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:38:56.122 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:38:56.122 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:38:56.122 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:38:56.122 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:38:56.122 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:38:56.123 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:38:56.123 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:38:56.123 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:38:56.123 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:38:56.123 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:38:56.123 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:38:56.123 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:38:56.123 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:38:56.123 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:38:56.123 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:38:56.124 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:38:56.124 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:38:56.127 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:38:56.127 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:38:56.127 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:38:56.127 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:38:56.127 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:38:56.127 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:38:56.127 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:38:56.127 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:38:56.128 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:38:56.128 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:38:56.128 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:38:56.128 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:38:56.128 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:38:56.128 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:38:56.128 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:38:56.128 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:38:56.128 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:38:56.128 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:38:56.128 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:38:56.128 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:38:56.128 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:38:56.128 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:38:56.128 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:38:56.129 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:38:56.129 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:38:56.129 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:38:56.129 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:38:56.129 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:38:56.129 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:38:56.129 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:38:56.129 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:38:56.129 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:38:56.153 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:38:56.154 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:38:56.154 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:38:56.154 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:38:56.281 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:38:56.283 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:38:56.283 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:38:56.283 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:38:56.283 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:38:56.283 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:38:56.283 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:38:56.283 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:38:56.283 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:38:56.283 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:38:56.284 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:38:56.284 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:38:56.284 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:38:56.284 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:38:56.284 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:38:56.284 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:38:56.347 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 17 2024-04-06T09:38:56.347 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 19 2024-04-06T09:38:56.347 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:38:56.348 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=77, tag_tid=12, entry_tid=1], [object_number=76, tag_tid=12, entry_tid=0], [object_number=79, tag_tid=11, entry_tid=17175], [object_number=78, tag_tid=11, entry_tid=17174]]], state=connected] 2024-04-06T09:38:56.348 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=70, tag_tid=11, entry_tid=12850], [object_number=69, tag_tid=11, entry_tid=12849], [object_number=68, tag_tid=11, entry_tid=12848], [object_number=71, tag_tid=11, entry_tid=12847]]], state=connected] 2024-04-06T09:38:56.351 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:38:56.352 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=77, tag_tid=12, entry_tid=1' 2024-04-06T09:38:56.353 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:38:56.353 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:38:56.354 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:38:56.354 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:38:56.354 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:38:56.354 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:38:56.354 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:38:56.354 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:38:56.354 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:38:56.354 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:38:56.354 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:38:56.354 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:38:56.355 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:38:56.355 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:38:56.355 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:38:56.404 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 17 2024-04-06T09:38:56.404 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 19 2024-04-06T09:38:56.404 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:38:56.405 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=77, tag_tid=12, entry_tid=1], [object_number=76, tag_tid=12, entry_tid=0], [object_number=79, tag_tid=11, entry_tid=17175], [object_number=78, tag_tid=11, entry_tid=17174]]], state=connected] 2024-04-06T09:38:56.405 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=70, tag_tid=11, entry_tid=12850], [object_number=69, tag_tid=11, entry_tid=12849], [object_number=68, tag_tid=11, entry_tid=12848], [object_number=71, tag_tid=11, entry_tid=12847]]], state=connected] 2024-04-06T09:38:56.409 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:38:56.411 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=70, tag_tid=11, entry_tid=12850' 2024-04-06T09:38:56.411 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=77, tag_tid=12, entry_tid=1' -a 'object_number=77, tag_tid=12, entry_tid=1' = 'object_number=70, tag_tid=11, entry_tid=12850' 2024-04-06T09:38:56.411 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=70, tag_tid=11, entry_tid=12850' '!=' 'object_number=71, tag_tid=11, entry_tid=12759' 2024-04-06T09:38:56.411 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:38:56.411 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=70, tag_tid=11, entry_tid=12850' = 'object_number=71, tag_tid=11, entry_tid=12759' 2024-04-06T09:38:56.411 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=70, tag_tid=11, entry_tid=12850' 2024-04-06T09:38:56.412 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=77, tag_tid=12, entry_tid=1' 2024-04-06T09:38:56.412 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:38:56.412 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:38:56.412 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=12 2024-04-06T09:38:56.413 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=70, tag_tid=11, entry_tid=12850' 2024-04-06T09:38:56.413 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:38:56.413 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:38:56.414 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=11 2024-04-06T09:38:56.415 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=77, tag_tid=12, entry_tid=1' 2024-04-06T09:38:56.415 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:38:56.415 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:38:56.416 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:38:56.416 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=70, tag_tid=11, entry_tid=12850' 2024-04-06T09:38:56.416 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:38:56.417 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:38:56.418 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=12850 2024-04-06T09:38:56.418 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 12 = 11 -a 1 -le 12850 2024-04-06T09:38:56.418 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:38:56.418 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:38:56.418 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:38:56.620 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:38:56.620 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:38:56.620 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:38:56.620 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:38:56.620 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:38:56.620 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:38:56.621 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:38:56.621 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:38:56.621 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:38:56.621 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:38:56.621 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:38:56.621 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:38:56.621 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:38:56.621 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:38:56.621 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:38:56.624 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:38:56.624 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:38:56.624 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:38:56.624 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:38:56.624 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:38:56.624 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:38:56.624 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:38:56.624 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:38:56.624 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:38:56.624 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:38:56.624 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:38:56.624 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:38:56.624 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:38:56.624 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:38:56.624 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:38:56.625 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:38:56.625 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:38:56.625 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:38:56.625 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:38:56.625 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:38:56.625 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:38:56.625 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:38:56.625 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:38:56.625 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:38:56.625 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:38:56.625 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:38:56.625 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:38:56.625 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:38:56.625 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:38:56.626 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:38:56.626 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:38:56.626 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:38:56.626 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:38:56.626 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:38:56.651 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:38:56.651 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:38:56.651 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:38:56.651 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:38:56.785 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:38:56.786 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:38:56.786 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:38:56.786 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:38:56.786 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:38:56.787 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:38:56.787 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:38:56.787 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:38:56.787 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:38:56.787 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:38:56.787 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:38:56.787 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:38:56.788 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:38:56.789 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:38:56.790 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:38:56.790 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:38:56.846 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 18 2024-04-06T09:38:56.846 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 19 2024-04-06T09:38:56.846 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:38:56.847 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=77, tag_tid=12, entry_tid=1], [object_number=76, tag_tid=12, entry_tid=0], [object_number=79, tag_tid=11, entry_tid=17175], [object_number=78, tag_tid=11, entry_tid=17174]]], state=connected] 2024-04-06T09:38:56.847 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=73, tag_tid=11, entry_tid=12909], [object_number=72, tag_tid=11, entry_tid=12908], [object_number=75, tag_tid=11, entry_tid=12907], [object_number=74, tag_tid=11, entry_tid=12906]]], state=connected] 2024-04-06T09:38:56.850 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:38:56.851 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=77, tag_tid=12, entry_tid=1' 2024-04-06T09:38:56.852 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:38:56.852 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:38:56.852 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:38:56.852 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:38:56.852 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:38:56.852 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:38:56.852 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:38:56.852 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:38:56.853 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:38:56.853 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:38:56.853 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:38:56.853 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:38:56.853 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:38:56.853 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:38:56.853 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:38:56.897 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 18 2024-04-06T09:38:56.897 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 19 2024-04-06T09:38:56.897 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:38:56.897 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=77, tag_tid=12, entry_tid=1], [object_number=76, tag_tid=12, entry_tid=0], [object_number=79, tag_tid=11, entry_tid=17175], [object_number=78, tag_tid=11, entry_tid=17174]]], state=connected] 2024-04-06T09:38:56.897 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=73, tag_tid=11, entry_tid=12909], [object_number=72, tag_tid=11, entry_tid=12908], [object_number=75, tag_tid=11, entry_tid=12907], [object_number=74, tag_tid=11, entry_tid=12906]]], state=connected] 2024-04-06T09:38:56.900 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:38:56.902 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=73, tag_tid=11, entry_tid=12909' 2024-04-06T09:38:56.902 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=77, tag_tid=12, entry_tid=1' -a 'object_number=77, tag_tid=12, entry_tid=1' = 'object_number=73, tag_tid=11, entry_tid=12909' 2024-04-06T09:38:56.902 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=73, tag_tid=11, entry_tid=12909' '!=' 'object_number=70, tag_tid=11, entry_tid=12850' 2024-04-06T09:38:56.902 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:38:56.902 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=73, tag_tid=11, entry_tid=12909' = 'object_number=70, tag_tid=11, entry_tid=12850' 2024-04-06T09:38:56.902 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=73, tag_tid=11, entry_tid=12909' 2024-04-06T09:38:56.903 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=77, tag_tid=12, entry_tid=1' 2024-04-06T09:38:56.903 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:38:56.903 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:38:56.903 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=12 2024-04-06T09:38:56.904 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=73, tag_tid=11, entry_tid=12909' 2024-04-06T09:38:56.904 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:38:56.904 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:38:56.905 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=11 2024-04-06T09:38:56.905 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=77, tag_tid=12, entry_tid=1' 2024-04-06T09:38:56.905 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:38:56.905 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:38:56.906 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:38:56.907 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=73, tag_tid=11, entry_tid=12909' 2024-04-06T09:38:56.907 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:38:56.907 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:38:56.908 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=12909 2024-04-06T09:38:56.908 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 12 = 11 -a 1 -le 12909 2024-04-06T09:38:56.908 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:38:56.908 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:38:56.908 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:38:57.110 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:38:57.110 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:38:57.110 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:38:57.110 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:38:57.110 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:38:57.110 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:38:57.110 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:38:57.110 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:38:57.110 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:38:57.111 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:38:57.111 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:38:57.111 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:38:57.111 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:38:57.111 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:38:57.111 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:38:57.111 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:38:57.112 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:38:57.114 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:38:57.115 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:38:57.115 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:38:57.115 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:38:57.115 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:38:57.115 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:38:57.115 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:38:57.115 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:38:57.115 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:38:57.115 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:38:57.115 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:38:57.115 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:38:57.115 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:38:57.115 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:38:57.115 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:38:57.115 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:38:57.115 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:38:57.116 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:38:57.116 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:38:57.116 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:38:57.116 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:38:57.116 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:38:57.116 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:38:57.116 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:38:57.116 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:38:57.116 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:38:57.116 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:38:57.116 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:38:57.116 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:38:57.116 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:38:57.116 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:38:57.116 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:38:57.140 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:38:57.140 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:38:57.140 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:38:57.140 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:38:57.273 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:38:57.273 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:38:57.273 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:38:57.274 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:38:57.274 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:38:57.274 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:38:57.274 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:38:57.274 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:38:57.274 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:38:57.274 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:38:57.275 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:38:57.275 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:38:57.275 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:38:57.278 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:38:57.278 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:38:57.278 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:38:57.323 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 18 2024-04-06T09:38:57.324 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 19 2024-04-06T09:38:57.324 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:38:57.324 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=77, tag_tid=12, entry_tid=1], [object_number=76, tag_tid=12, entry_tid=0], [object_number=79, tag_tid=11, entry_tid=17175], [object_number=78, tag_tid=11, entry_tid=17174]]], state=connected] 2024-04-06T09:38:57.324 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=74, tag_tid=11, entry_tid=13014], [object_number=73, tag_tid=11, entry_tid=13013], [object_number=72, tag_tid=11, entry_tid=13012], [object_number=75, tag_tid=11, entry_tid=13011]]], state=connected] 2024-04-06T09:38:57.327 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:38:57.328 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=77, tag_tid=12, entry_tid=1' 2024-04-06T09:38:57.329 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:38:57.329 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:38:57.329 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:38:57.329 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:38:57.329 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:38:57.329 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:38:57.329 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:38:57.330 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:38:57.330 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:38:57.330 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:38:57.330 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:38:57.330 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:38:57.330 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:38:57.331 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:38:57.331 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:38:57.374 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 18 2024-04-06T09:38:57.374 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 19 2024-04-06T09:38:57.374 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:38:57.375 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=77, tag_tid=12, entry_tid=1], [object_number=76, tag_tid=12, entry_tid=0], [object_number=79, tag_tid=11, entry_tid=17175], [object_number=78, tag_tid=11, entry_tid=17174]]], state=connected] 2024-04-06T09:38:57.375 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=74, tag_tid=11, entry_tid=13014], [object_number=73, tag_tid=11, entry_tid=13013], [object_number=72, tag_tid=11, entry_tid=13012], [object_number=75, tag_tid=11, entry_tid=13011]]], state=connected] 2024-04-06T09:38:57.378 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:38:57.380 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=74, tag_tid=11, entry_tid=13014' 2024-04-06T09:38:57.380 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=77, tag_tid=12, entry_tid=1' -a 'object_number=77, tag_tid=12, entry_tid=1' = 'object_number=74, tag_tid=11, entry_tid=13014' 2024-04-06T09:38:57.380 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=74, tag_tid=11, entry_tid=13014' '!=' 'object_number=73, tag_tid=11, entry_tid=12909' 2024-04-06T09:38:57.380 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:38:57.380 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=74, tag_tid=11, entry_tid=13014' = 'object_number=73, tag_tid=11, entry_tid=12909' 2024-04-06T09:38:57.380 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=74, tag_tid=11, entry_tid=13014' 2024-04-06T09:38:57.380 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=77, tag_tid=12, entry_tid=1' 2024-04-06T09:38:57.380 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:38:57.381 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:38:57.381 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=12 2024-04-06T09:38:57.381 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=74, tag_tid=11, entry_tid=13014' 2024-04-06T09:38:57.381 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:38:57.381 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:38:57.382 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=11 2024-04-06T09:38:57.383 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=77, tag_tid=12, entry_tid=1' 2024-04-06T09:38:57.383 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:38:57.383 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:38:57.384 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:38:57.384 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=74, tag_tid=11, entry_tid=13014' 2024-04-06T09:38:57.384 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:38:57.384 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:38:57.386 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=13014 2024-04-06T09:38:57.386 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 12 = 11 -a 1 -le 13014 2024-04-06T09:38:57.386 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:38:57.386 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:38:57.386 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:38:57.587 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:38:57.587 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:38:57.587 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:38:57.587 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:38:57.588 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:38:57.588 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:38:57.588 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:38:57.588 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:38:57.588 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:38:57.588 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:38:57.588 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:38:57.588 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:38:57.588 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:38:57.588 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:38:57.588 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:38:57.588 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:38:57.589 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:38:57.592 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:38:57.592 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:38:57.592 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:38:57.592 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:38:57.592 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:38:57.592 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:38:57.592 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:38:57.592 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:38:57.592 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:38:57.592 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:38:57.592 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:38:57.592 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:38:57.592 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:38:57.593 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:38:57.593 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:38:57.593 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:38:57.594 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:38:57.594 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:38:57.594 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:38:57.594 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:38:57.594 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:38:57.594 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:38:57.594 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:38:57.594 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:38:57.594 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:38:57.594 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:38:57.594 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:38:57.595 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:38:57.595 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:38:57.595 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:38:57.595 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:38:57.595 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:38:57.617 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:38:57.617 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:38:57.617 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:38:57.617 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:38:57.748 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:38:57.749 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:38:57.749 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:38:57.749 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:38:57.750 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:38:57.750 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:38:57.750 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:38:57.750 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:38:57.750 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:38:57.750 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:38:57.750 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:38:57.751 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:38:57.751 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:38:57.752 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:38:57.753 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:38:57.753 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:38:57.804 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 18 2024-04-06T09:38:57.804 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 19 2024-04-06T09:38:57.804 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:38:57.804 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=77, tag_tid=12, entry_tid=1], [object_number=76, tag_tid=12, entry_tid=0], [object_number=79, tag_tid=11, entry_tid=17175], [object_number=78, tag_tid=11, entry_tid=17174]]], state=connected] 2024-04-06T09:38:57.804 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=73, tag_tid=11, entry_tid=13105], [object_number=72, tag_tid=11, entry_tid=13104], [object_number=75, tag_tid=11, entry_tid=13103], [object_number=74, tag_tid=11, entry_tid=13102]]], state=connected] 2024-04-06T09:38:57.807 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:38:57.808 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=77, tag_tid=12, entry_tid=1' 2024-04-06T09:38:57.810 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:38:57.810 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:38:57.810 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:38:57.810 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:38:57.810 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:38:57.810 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:38:57.810 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:38:57.810 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:38:57.810 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:38:57.810 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:38:57.810 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:38:57.810 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:38:57.811 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:38:57.811 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:38:57.811 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:38:57.855 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 18 2024-04-06T09:38:57.855 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 19 2024-04-06T09:38:57.855 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:38:57.855 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=77, tag_tid=12, entry_tid=1], [object_number=76, tag_tid=12, entry_tid=0], [object_number=79, tag_tid=11, entry_tid=17175], [object_number=78, tag_tid=11, entry_tid=17174]]], state=connected] 2024-04-06T09:38:57.855 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=73, tag_tid=11, entry_tid=13105], [object_number=72, tag_tid=11, entry_tid=13104], [object_number=75, tag_tid=11, entry_tid=13103], [object_number=74, tag_tid=11, entry_tid=13102]]], state=connected] 2024-04-06T09:38:57.859 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:38:57.860 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=73, tag_tid=11, entry_tid=13105' 2024-04-06T09:38:57.860 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=77, tag_tid=12, entry_tid=1' -a 'object_number=77, tag_tid=12, entry_tid=1' = 'object_number=73, tag_tid=11, entry_tid=13105' 2024-04-06T09:38:57.860 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=73, tag_tid=11, entry_tid=13105' '!=' 'object_number=74, tag_tid=11, entry_tid=13014' 2024-04-06T09:38:57.860 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:38:57.860 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=73, tag_tid=11, entry_tid=13105' = 'object_number=74, tag_tid=11, entry_tid=13014' 2024-04-06T09:38:57.860 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=73, tag_tid=11, entry_tid=13105' 2024-04-06T09:38:57.861 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=77, tag_tid=12, entry_tid=1' 2024-04-06T09:38:57.861 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:38:57.861 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:38:57.861 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=12 2024-04-06T09:38:57.862 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=73, tag_tid=11, entry_tid=13105' 2024-04-06T09:38:57.862 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:38:57.862 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:38:57.863 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=11 2024-04-06T09:38:57.864 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=77, tag_tid=12, entry_tid=1' 2024-04-06T09:38:57.864 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:38:57.864 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:38:57.865 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:38:57.865 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=73, tag_tid=11, entry_tid=13105' 2024-04-06T09:38:57.865 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:38:57.865 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:38:57.867 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=13105 2024-04-06T09:38:57.867 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 12 = 11 -a 1 -le 13105 2024-04-06T09:38:57.867 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:38:57.867 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:38:57.867 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:38:58.068 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:38:58.068 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:38:58.068 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:38:58.069 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:38:58.069 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:38:58.069 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:38:58.069 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:38:58.069 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:38:58.069 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:38:58.069 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:38:58.069 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:38:58.069 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:38:58.069 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:38:58.069 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:38:58.069 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:38:58.070 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:38:58.070 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:38:58.072 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:38:58.072 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:38:58.072 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:38:58.072 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:38:58.072 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:38:58.072 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:38:58.072 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:38:58.072 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:38:58.072 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:38:58.072 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:38:58.072 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:38:58.072 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:38:58.073 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:38:58.073 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:38:58.073 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:38:58.073 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:38:58.074 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:38:58.074 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:38:58.074 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:38:58.074 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:38:58.074 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:38:58.075 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:38:58.075 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:38:58.075 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:38:58.075 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:38:58.075 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:38:58.075 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:38:58.075 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:38:58.075 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:38:58.075 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:38:58.075 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:38:58.075 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:38:58.097 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:38:58.097 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:38:58.097 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:38:58.097 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:38:58.229 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:38:58.230 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:38:58.230 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:38:58.230 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:38:58.230 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:38:58.230 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:38:58.230 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:38:58.230 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:38:58.230 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:38:58.230 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:38:58.230 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:38:58.231 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:38:58.231 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:38:58.231 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:38:58.232 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:38:58.232 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:38:58.277 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 18 2024-04-06T09:38:58.277 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 19 2024-04-06T09:38:58.277 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:38:58.278 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=77, tag_tid=12, entry_tid=1], [object_number=76, tag_tid=12, entry_tid=0], [object_number=79, tag_tid=11, entry_tid=17175], [object_number=78, tag_tid=11, entry_tid=17174]]], state=connected] 2024-04-06T09:38:58.278 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=75, tag_tid=11, entry_tid=13195], [object_number=74, tag_tid=11, entry_tid=13194], [object_number=73, tag_tid=11, entry_tid=13193], [object_number=72, tag_tid=11, entry_tid=13192]]], state=connected] 2024-04-06T09:38:58.281 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:38:58.282 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=77, tag_tid=12, entry_tid=1' 2024-04-06T09:38:58.284 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:38:58.284 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:38:58.284 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:38:58.284 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:38:58.284 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:38:58.284 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:38:58.284 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:38:58.284 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:38:58.284 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:38:58.284 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:38:58.284 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:38:58.285 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:38:58.285 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:38:58.285 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:38:58.285 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:38:58.330 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 18 2024-04-06T09:38:58.331 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 19 2024-04-06T09:38:58.331 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:38:58.331 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=77, tag_tid=12, entry_tid=1], [object_number=76, tag_tid=12, entry_tid=0], [object_number=79, tag_tid=11, entry_tid=17175], [object_number=78, tag_tid=11, entry_tid=17174]]], state=connected] 2024-04-06T09:38:58.331 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=75, tag_tid=11, entry_tid=13195], [object_number=74, tag_tid=11, entry_tid=13194], [object_number=73, tag_tid=11, entry_tid=13193], [object_number=72, tag_tid=11, entry_tid=13192]]], state=connected] 2024-04-06T09:38:58.334 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:38:58.335 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=75, tag_tid=11, entry_tid=13195' 2024-04-06T09:38:58.336 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=77, tag_tid=12, entry_tid=1' -a 'object_number=77, tag_tid=12, entry_tid=1' = 'object_number=75, tag_tid=11, entry_tid=13195' 2024-04-06T09:38:58.336 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=75, tag_tid=11, entry_tid=13195' '!=' 'object_number=73, tag_tid=11, entry_tid=13105' 2024-04-06T09:38:58.336 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:38:58.336 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=75, tag_tid=11, entry_tid=13195' = 'object_number=73, tag_tid=11, entry_tid=13105' 2024-04-06T09:38:58.336 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=75, tag_tid=11, entry_tid=13195' 2024-04-06T09:38:58.336 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=77, tag_tid=12, entry_tid=1' 2024-04-06T09:38:58.337 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:38:58.337 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:38:58.337 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=12 2024-04-06T09:38:58.338 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=75, tag_tid=11, entry_tid=13195' 2024-04-06T09:38:58.338 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:38:58.338 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:38:58.339 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=11 2024-04-06T09:38:58.339 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=77, tag_tid=12, entry_tid=1' 2024-04-06T09:38:58.339 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:38:58.340 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:38:58.340 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:38:58.341 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=75, tag_tid=11, entry_tid=13195' 2024-04-06T09:38:58.341 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:38:58.341 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:38:58.342 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=13195 2024-04-06T09:38:58.343 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 12 = 11 -a 1 -le 13195 2024-04-06T09:38:58.343 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:38:58.343 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:38:58.343 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:38:58.544 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:38:58.545 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:38:58.545 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:38:58.545 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:38:58.545 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:38:58.545 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:38:58.545 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:38:58.545 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:38:58.545 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:38:58.545 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:38:58.545 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:38:58.545 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:38:58.545 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:38:58.545 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:38:58.546 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:38:58.546 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:38:58.546 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:38:58.549 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:38:58.549 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:38:58.550 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:38:58.550 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:38:58.550 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:38:58.550 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:38:58.550 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:38:58.550 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:38:58.550 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:38:58.550 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:38:58.550 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:38:58.550 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:38:58.550 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:38:58.550 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:38:58.550 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:38:58.550 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:38:58.551 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:38:58.551 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:38:58.551 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:38:58.551 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:38:58.551 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:38:58.551 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:38:58.551 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:38:58.551 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:38:58.551 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:38:58.551 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:38:58.551 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:38:58.551 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:38:58.551 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:38:58.552 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:38:58.552 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:38:58.552 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:38:58.576 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:38:58.576 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:38:58.577 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:38:58.577 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:38:58.705 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:38:58.707 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:38:58.707 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:38:58.707 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:38:58.707 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:38:58.707 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:38:58.707 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:38:58.707 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:38:58.708 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:38:58.708 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:38:58.708 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:38:58.708 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:38:58.708 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:38:58.709 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:38:58.710 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:38:58.710 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:38:58.755 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 18 2024-04-06T09:38:58.755 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 19 2024-04-06T09:38:58.755 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:38:58.755 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=77, tag_tid=12, entry_tid=1], [object_number=76, tag_tid=12, entry_tid=0], [object_number=79, tag_tid=11, entry_tid=17175], [object_number=78, tag_tid=11, entry_tid=17174]]], state=connected] 2024-04-06T09:38:58.755 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=75, tag_tid=11, entry_tid=13271], [object_number=74, tag_tid=11, entry_tid=13270], [object_number=73, tag_tid=11, entry_tid=13269], [object_number=72, tag_tid=11, entry_tid=13268]]], state=connected] 2024-04-06T09:38:58.759 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:38:58.760 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=77, tag_tid=12, entry_tid=1' 2024-04-06T09:38:58.761 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:38:58.761 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:38:58.761 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:38:58.762 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:38:58.762 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:38:58.762 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:38:58.762 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:38:58.762 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:38:58.762 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:38:58.762 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:38:58.762 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:38:58.762 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:38:58.763 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:38:58.763 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:38:58.763 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:38:58.808 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 18 2024-04-06T09:38:58.808 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 19 2024-04-06T09:38:58.808 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:38:58.809 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=77, tag_tid=12, entry_tid=1], [object_number=76, tag_tid=12, entry_tid=0], [object_number=79, tag_tid=11, entry_tid=17175], [object_number=78, tag_tid=11, entry_tid=17174]]], state=connected] 2024-04-06T09:38:58.809 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=75, tag_tid=11, entry_tid=13271], [object_number=74, tag_tid=11, entry_tid=13270], [object_number=73, tag_tid=11, entry_tid=13269], [object_number=72, tag_tid=11, entry_tid=13268]]], state=connected] 2024-04-06T09:38:58.812 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:38:58.813 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=75, tag_tid=11, entry_tid=13271' 2024-04-06T09:38:58.813 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=77, tag_tid=12, entry_tid=1' -a 'object_number=77, tag_tid=12, entry_tid=1' = 'object_number=75, tag_tid=11, entry_tid=13271' 2024-04-06T09:38:58.814 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=75, tag_tid=11, entry_tid=13271' '!=' 'object_number=75, tag_tid=11, entry_tid=13195' 2024-04-06T09:38:58.814 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:38:58.814 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=75, tag_tid=11, entry_tid=13271' = 'object_number=75, tag_tid=11, entry_tid=13195' 2024-04-06T09:38:58.814 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=75, tag_tid=11, entry_tid=13271' 2024-04-06T09:38:58.814 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=77, tag_tid=12, entry_tid=1' 2024-04-06T09:38:58.814 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:38:58.815 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:38:58.815 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=12 2024-04-06T09:38:58.816 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=75, tag_tid=11, entry_tid=13271' 2024-04-06T09:38:58.816 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:38:58.816 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:38:58.817 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=11 2024-04-06T09:38:58.817 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=77, tag_tid=12, entry_tid=1' 2024-04-06T09:38:58.817 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:38:58.818 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:38:58.818 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:38:58.819 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=75, tag_tid=11, entry_tid=13271' 2024-04-06T09:38:58.819 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:38:58.819 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:38:58.820 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=13271 2024-04-06T09:38:58.820 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 12 = 11 -a 1 -le 13271 2024-04-06T09:38:58.820 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:38:58.821 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:38:58.821 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:38:59.022 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:38:59.022 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:38:59.023 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:38:59.023 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:38:59.023 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:38:59.023 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:38:59.023 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:38:59.023 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:38:59.023 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:38:59.023 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:38:59.023 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:38:59.023 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:38:59.023 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:38:59.023 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:38:59.023 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:38:59.024 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:38:59.024 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:38:59.027 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:38:59.027 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:38:59.027 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:38:59.027 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:38:59.027 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:38:59.027 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:38:59.027 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:38:59.027 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:38:59.027 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:38:59.027 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:38:59.027 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:38:59.027 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:38:59.028 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:38:59.028 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:38:59.028 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:38:59.028 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:38:59.028 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:38:59.028 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:38:59.028 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:38:59.028 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:38:59.028 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:38:59.028 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:38:59.028 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:38:59.028 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:38:59.028 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:38:59.028 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:38:59.028 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:38:59.028 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:38:59.029 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:38:59.029 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:38:59.029 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:38:59.029 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:38:59.053 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:38:59.054 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:38:59.054 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:38:59.054 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:38:59.188 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:38:59.189 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:38:59.189 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:38:59.189 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:38:59.189 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:38:59.189 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:38:59.189 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:38:59.189 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:38:59.189 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:38:59.190 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:38:59.190 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:38:59.190 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:38:59.190 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:38:59.192 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:38:59.192 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:38:59.192 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:38:59.242 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 18 2024-04-06T09:38:59.242 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 19 2024-04-06T09:38:59.243 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:38:59.243 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=77, tag_tid=12, entry_tid=1], [object_number=76, tag_tid=12, entry_tid=0], [object_number=79, tag_tid=11, entry_tid=17175], [object_number=78, tag_tid=11, entry_tid=17174]]], state=connected] 2024-04-06T09:38:59.243 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=73, tag_tid=11, entry_tid=13357], [object_number=72, tag_tid=11, entry_tid=13356], [object_number=75, tag_tid=11, entry_tid=13355], [object_number=74, tag_tid=11, entry_tid=13354]]], state=connected] 2024-04-06T09:38:59.247 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:38:59.247 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=77, tag_tid=12, entry_tid=1' 2024-04-06T09:38:59.249 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:38:59.249 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:38:59.249 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:38:59.249 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:38:59.249 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:38:59.249 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:38:59.249 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:38:59.249 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:38:59.249 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:38:59.249 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:38:59.250 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:38:59.250 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:38:59.250 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:38:59.250 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:38:59.251 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:38:59.297 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 18 2024-04-06T09:38:59.297 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 19 2024-04-06T09:38:59.297 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:38:59.297 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=77, tag_tid=12, entry_tid=1], [object_number=76, tag_tid=12, entry_tid=0], [object_number=79, tag_tid=11, entry_tid=17175], [object_number=78, tag_tid=11, entry_tid=17174]]], state=connected] 2024-04-06T09:38:59.297 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=73, tag_tid=11, entry_tid=13357], [object_number=72, tag_tid=11, entry_tid=13356], [object_number=75, tag_tid=11, entry_tid=13355], [object_number=74, tag_tid=11, entry_tid=13354]]], state=connected] 2024-04-06T09:38:59.300 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:38:59.302 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=73, tag_tid=11, entry_tid=13357' 2024-04-06T09:38:59.302 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=77, tag_tid=12, entry_tid=1' -a 'object_number=77, tag_tid=12, entry_tid=1' = 'object_number=73, tag_tid=11, entry_tid=13357' 2024-04-06T09:38:59.303 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=73, tag_tid=11, entry_tid=13357' '!=' 'object_number=75, tag_tid=11, entry_tid=13271' 2024-04-06T09:38:59.303 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:38:59.303 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=73, tag_tid=11, entry_tid=13357' = 'object_number=75, tag_tid=11, entry_tid=13271' 2024-04-06T09:38:59.303 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=73, tag_tid=11, entry_tid=13357' 2024-04-06T09:38:59.303 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=77, tag_tid=12, entry_tid=1' 2024-04-06T09:38:59.303 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:38:59.303 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:38:59.304 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=12 2024-04-06T09:38:59.304 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=73, tag_tid=11, entry_tid=13357' 2024-04-06T09:38:59.304 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:38:59.305 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:38:59.305 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=11 2024-04-06T09:38:59.306 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=77, tag_tid=12, entry_tid=1' 2024-04-06T09:38:59.306 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:38:59.306 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:38:59.307 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:38:59.308 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=73, tag_tid=11, entry_tid=13357' 2024-04-06T09:38:59.308 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:38:59.308 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:38:59.309 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=13357 2024-04-06T09:38:59.309 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 12 = 11 -a 1 -le 13357 2024-04-06T09:38:59.309 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:38:59.309 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:38:59.309 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:38:59.511 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:38:59.511 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:38:59.511 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:38:59.511 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:38:59.511 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:38:59.511 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:38:59.511 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:38:59.511 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:38:59.512 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:38:59.512 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:38:59.512 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:38:59.512 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:38:59.512 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:38:59.512 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:38:59.512 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:38:59.512 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:38:59.513 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:38:59.516 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:38:59.516 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:38:59.516 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:38:59.517 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:38:59.517 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:38:59.517 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:38:59.517 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:38:59.517 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:38:59.517 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:38:59.517 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:38:59.517 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:38:59.517 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:38:59.517 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:38:59.517 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:38:59.517 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:38:59.517 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:38:59.517 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:38:59.518 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:38:59.518 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:38:59.518 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:38:59.518 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:38:59.518 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:38:59.518 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:38:59.518 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:38:59.518 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:38:59.518 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:38:59.518 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:38:59.518 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:38:59.518 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:38:59.518 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:38:59.518 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:38:59.518 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:38:59.542 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:38:59.542 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:38:59.542 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:38:59.542 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:38:59.683 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:38:59.684 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:38:59.684 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:38:59.684 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:38:59.684 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:38:59.684 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:38:59.685 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:38:59.685 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:38:59.685 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:38:59.685 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:38:59.685 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:38:59.685 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:38:59.685 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:38:59.688 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:38:59.688 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:38:59.688 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:38:59.735 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 18 2024-04-06T09:38:59.735 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 19 2024-04-06T09:38:59.735 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:38:59.735 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=77, tag_tid=12, entry_tid=1], [object_number=76, tag_tid=12, entry_tid=0], [object_number=79, tag_tid=11, entry_tid=17175], [object_number=78, tag_tid=11, entry_tid=17174]]], state=connected] 2024-04-06T09:38:59.735 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=72, tag_tid=11, entry_tid=13436], [object_number=75, tag_tid=11, entry_tid=13435], [object_number=74, tag_tid=11, entry_tid=13434], [object_number=73, tag_tid=11, entry_tid=13433]]], state=connected] 2024-04-06T09:38:59.738 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:38:59.740 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=77, tag_tid=12, entry_tid=1' 2024-04-06T09:38:59.741 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:38:59.741 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:38:59.741 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:38:59.741 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:38:59.741 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:38:59.741 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:38:59.741 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:38:59.741 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:38:59.741 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:38:59.741 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:38:59.742 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:38:59.742 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:38:59.742 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:38:59.742 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:38:59.742 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:38:59.789 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 18 2024-04-06T09:38:59.789 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 19 2024-04-06T09:38:59.789 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:38:59.789 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=77, tag_tid=12, entry_tid=1], [object_number=76, tag_tid=12, entry_tid=0], [object_number=79, tag_tid=11, entry_tid=17175], [object_number=78, tag_tid=11, entry_tid=17174]]], state=connected] 2024-04-06T09:38:59.789 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=72, tag_tid=11, entry_tid=13436], [object_number=75, tag_tid=11, entry_tid=13435], [object_number=74, tag_tid=11, entry_tid=13434], [object_number=73, tag_tid=11, entry_tid=13433]]], state=connected] 2024-04-06T09:38:59.792 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:38:59.794 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=72, tag_tid=11, entry_tid=13436' 2024-04-06T09:38:59.794 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=77, tag_tid=12, entry_tid=1' -a 'object_number=77, tag_tid=12, entry_tid=1' = 'object_number=72, tag_tid=11, entry_tid=13436' 2024-04-06T09:38:59.794 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=72, tag_tid=11, entry_tid=13436' '!=' 'object_number=73, tag_tid=11, entry_tid=13357' 2024-04-06T09:38:59.794 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:38:59.794 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=72, tag_tid=11, entry_tid=13436' = 'object_number=73, tag_tid=11, entry_tid=13357' 2024-04-06T09:38:59.794 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=72, tag_tid=11, entry_tid=13436' 2024-04-06T09:38:59.795 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=77, tag_tid=12, entry_tid=1' 2024-04-06T09:38:59.795 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:38:59.795 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:38:59.795 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=12 2024-04-06T09:38:59.796 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=72, tag_tid=11, entry_tid=13436' 2024-04-06T09:38:59.796 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:38:59.796 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:38:59.797 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=11 2024-04-06T09:38:59.798 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=77, tag_tid=12, entry_tid=1' 2024-04-06T09:38:59.798 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:38:59.798 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:38:59.799 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:38:59.800 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=72, tag_tid=11, entry_tid=13436' 2024-04-06T09:38:59.800 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:38:59.800 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:38:59.801 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=13436 2024-04-06T09:38:59.801 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 12 = 11 -a 1 -le 13436 2024-04-06T09:38:59.801 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:38:59.801 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:38:59.801 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:39:00.003 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:39:00.003 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:39:00.003 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:39:00.003 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:39:00.003 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:39:00.003 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:39:00.004 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:39:00.004 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:39:00.004 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:39:00.004 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:39:00.004 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:39:00.004 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:39:00.004 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:39:00.004 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:39:00.004 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:39:00.005 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:39:00.005 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:39:00.008 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:39:00.008 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:39:00.008 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:39:00.008 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:39:00.008 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:39:00.008 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:39:00.008 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:39:00.008 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:39:00.008 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:39:00.008 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:39:00.008 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:39:00.009 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:39:00.009 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:39:00.009 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:39:00.009 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:39:00.009 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:39:00.009 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:39:00.009 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:39:00.009 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:39:00.009 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:39:00.009 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:39:00.009 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:39:00.009 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:39:00.009 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:39:00.010 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:39:00.010 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:39:00.010 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:39:00.010 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:39:00.010 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:39:00.010 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:39:00.010 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:39:00.011 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:39:00.034 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:39:00.034 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:39:00.034 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:39:00.034 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:39:00.164 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:39:00.165 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:39:00.165 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:39:00.166 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:39:00.166 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:39:00.166 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:39:00.166 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:39:00.166 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:39:00.166 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:39:00.166 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:39:00.166 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:39:00.167 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:39:00.167 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:39:00.168 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:39:00.168 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:39:00.169 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:39:00.220 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 18 2024-04-06T09:39:00.221 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 19 2024-04-06T09:39:00.221 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:39:00.221 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=77, tag_tid=12, entry_tid=1], [object_number=76, tag_tid=12, entry_tid=0], [object_number=79, tag_tid=11, entry_tid=17175], [object_number=78, tag_tid=11, entry_tid=17174]]], state=connected] 2024-04-06T09:39:00.221 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=73, tag_tid=11, entry_tid=13521], [object_number=72, tag_tid=11, entry_tid=13520], [object_number=75, tag_tid=11, entry_tid=13519], [object_number=74, tag_tid=11, entry_tid=13518]]], state=connected] 2024-04-06T09:39:00.224 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:39:00.226 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=77, tag_tid=12, entry_tid=1' 2024-04-06T09:39:00.227 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:39:00.227 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:39:00.227 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:39:00.227 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:39:00.227 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:39:00.227 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:39:00.227 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:39:00.227 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:39:00.227 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:39:00.227 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:39:00.228 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:39:00.228 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:39:00.228 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:39:00.228 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:39:00.228 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:39:00.275 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 18 2024-04-06T09:39:00.275 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 19 2024-04-06T09:39:00.275 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:39:00.275 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=77, tag_tid=12, entry_tid=1], [object_number=76, tag_tid=12, entry_tid=0], [object_number=79, tag_tid=11, entry_tid=17175], [object_number=78, tag_tid=11, entry_tid=17174]]], state=connected] 2024-04-06T09:39:00.276 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=73, tag_tid=11, entry_tid=13521], [object_number=72, tag_tid=11, entry_tid=13520], [object_number=75, tag_tid=11, entry_tid=13519], [object_number=74, tag_tid=11, entry_tid=13518]]], state=connected] 2024-04-06T09:39:00.279 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:39:00.283 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=73, tag_tid=11, entry_tid=13521' 2024-04-06T09:39:00.283 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=77, tag_tid=12, entry_tid=1' -a 'object_number=77, tag_tid=12, entry_tid=1' = 'object_number=73, tag_tid=11, entry_tid=13521' 2024-04-06T09:39:00.283 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=73, tag_tid=11, entry_tid=13521' '!=' 'object_number=72, tag_tid=11, entry_tid=13436' 2024-04-06T09:39:00.283 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:39:00.283 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=73, tag_tid=11, entry_tid=13521' = 'object_number=72, tag_tid=11, entry_tid=13436' 2024-04-06T09:39:00.283 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=73, tag_tid=11, entry_tid=13521' 2024-04-06T09:39:00.283 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=77, tag_tid=12, entry_tid=1' 2024-04-06T09:39:00.283 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:39:00.283 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:39:00.283 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=12 2024-04-06T09:39:00.283 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=73, tag_tid=11, entry_tid=13521' 2024-04-06T09:39:00.284 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:39:00.284 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:39:00.284 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=11 2024-04-06T09:39:00.284 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=77, tag_tid=12, entry_tid=1' 2024-04-06T09:39:00.284 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:39:00.284 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:39:00.285 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:39:00.286 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=73, tag_tid=11, entry_tid=13521' 2024-04-06T09:39:00.286 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:39:00.286 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:39:00.287 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=13521 2024-04-06T09:39:00.288 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 12 = 11 -a 1 -le 13521 2024-04-06T09:39:00.288 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:39:00.288 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:39:00.288 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:39:00.489 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:39:00.490 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:39:00.490 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:39:00.490 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:39:00.490 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:39:00.490 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:39:00.490 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:39:00.490 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:39:00.490 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:39:00.490 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:39:00.490 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:39:00.491 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:39:00.491 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:39:00.491 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:39:00.491 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:39:00.494 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:39:00.494 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:39:00.494 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:39:00.494 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:39:00.494 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:39:00.494 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:39:00.494 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:39:00.495 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:39:00.495 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:39:00.495 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:39:00.495 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:39:00.495 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:39:00.495 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:39:00.495 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:39:00.495 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:39:00.495 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:39:00.495 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:39:00.495 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:39:00.495 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:39:00.496 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:39:00.496 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:39:00.496 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:39:00.496 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:39:00.496 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:39:00.496 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:39:00.496 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:39:00.496 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:39:00.496 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:39:00.496 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:39:00.496 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:39:00.496 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:39:00.497 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:39:00.497 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:39:00.497 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:39:00.520 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:39:00.521 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:39:00.521 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:39:00.521 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:39:00.651 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:39:00.652 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:39:00.652 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:39:00.653 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:39:00.653 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:39:00.653 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:39:00.653 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:39:00.653 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:39:00.653 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:39:00.653 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:39:00.654 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:39:00.654 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:39:00.654 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:39:00.656 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:39:00.657 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:39:00.657 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:39:00.709 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 18 2024-04-06T09:39:00.709 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 19 2024-04-06T09:39:00.709 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:39:00.709 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=77, tag_tid=12, entry_tid=1], [object_number=76, tag_tid=12, entry_tid=0], [object_number=79, tag_tid=11, entry_tid=17175], [object_number=78, tag_tid=11, entry_tid=17174]]], state=connected] 2024-04-06T09:39:00.709 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=72, tag_tid=11, entry_tid=13612], [object_number=75, tag_tid=11, entry_tid=13611], [object_number=74, tag_tid=11, entry_tid=13610], [object_number=73, tag_tid=11, entry_tid=13609]]], state=connected] 2024-04-06T09:39:00.747 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:39:00.749 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=77, tag_tid=12, entry_tid=1' 2024-04-06T09:39:00.749 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:39:00.749 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:39:00.749 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:39:00.750 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:39:00.750 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:39:00.750 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:39:00.750 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:39:00.750 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:39:00.750 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:39:00.750 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:39:00.750 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:39:00.751 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:39:00.751 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:39:00.751 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:39:00.751 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:39:00.805 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 18 2024-04-06T09:39:00.805 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 19 2024-04-06T09:39:00.805 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:39:00.805 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=77, tag_tid=12, entry_tid=1], [object_number=76, tag_tid=12, entry_tid=0], [object_number=79, tag_tid=11, entry_tid=17175], [object_number=78, tag_tid=11, entry_tid=17174]]], state=connected] 2024-04-06T09:39:00.805 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=72, tag_tid=11, entry_tid=13612], [object_number=75, tag_tid=11, entry_tid=13611], [object_number=74, tag_tid=11, entry_tid=13610], [object_number=73, tag_tid=11, entry_tid=13609]]], state=connected] 2024-04-06T09:39:00.809 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:39:00.811 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=72, tag_tid=11, entry_tid=13612' 2024-04-06T09:39:00.811 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=77, tag_tid=12, entry_tid=1' -a 'object_number=77, tag_tid=12, entry_tid=1' = 'object_number=72, tag_tid=11, entry_tid=13612' 2024-04-06T09:39:00.811 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=72, tag_tid=11, entry_tid=13612' '!=' 'object_number=73, tag_tid=11, entry_tid=13521' 2024-04-06T09:39:00.811 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:39:00.811 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=72, tag_tid=11, entry_tid=13612' = 'object_number=73, tag_tid=11, entry_tid=13521' 2024-04-06T09:39:00.811 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=72, tag_tid=11, entry_tid=13612' 2024-04-06T09:39:00.811 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=77, tag_tid=12, entry_tid=1' 2024-04-06T09:39:00.811 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:39:00.811 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:39:00.812 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=12 2024-04-06T09:39:00.813 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=72, tag_tid=11, entry_tid=13612' 2024-04-06T09:39:00.813 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:39:00.813 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:39:00.813 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=11 2024-04-06T09:39:00.814 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=77, tag_tid=12, entry_tid=1' 2024-04-06T09:39:00.814 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:39:00.814 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:39:00.815 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:39:00.816 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=72, tag_tid=11, entry_tid=13612' 2024-04-06T09:39:00.816 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:39:00.816 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:39:00.817 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=13612 2024-04-06T09:39:00.817 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 12 = 11 -a 1 -le 13612 2024-04-06T09:39:00.817 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:39:00.817 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:39:00.818 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:39:01.019 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:39:01.019 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:39:01.020 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:39:01.020 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:39:01.020 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:39:01.020 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:39:01.020 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:39:01.020 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:39:01.020 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:39:01.020 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:39:01.020 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:39:01.020 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:39:01.020 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:39:01.020 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:39:01.020 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:39:01.021 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:39:01.021 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:39:01.024 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:39:01.024 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:39:01.024 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:39:01.024 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:39:01.024 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:39:01.024 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:39:01.024 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:39:01.024 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:39:01.024 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:39:01.024 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:39:01.024 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:39:01.024 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:39:01.025 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:39:01.025 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:39:01.025 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:39:01.025 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:39:01.026 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:39:01.026 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:39:01.026 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:39:01.026 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:39:01.026 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:39:01.026 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:39:01.026 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:39:01.026 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:39:01.026 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:39:01.026 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:39:01.026 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:39:01.026 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:39:01.026 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:39:01.027 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:39:01.027 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:39:01.027 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:39:01.050 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:39:01.050 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:39:01.050 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:39:01.050 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:39:01.191 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:39:01.193 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:39:01.193 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:39:01.193 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:39:01.193 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:39:01.194 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:39:01.194 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:39:01.194 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:39:01.194 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:39:01.194 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:39:01.194 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:39:01.195 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:39:01.195 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:39:01.196 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:39:01.196 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:39:01.196 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:39:01.240 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 18 2024-04-06T09:39:01.240 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 19 2024-04-06T09:39:01.240 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:39:01.241 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=77, tag_tid=12, entry_tid=1], [object_number=76, tag_tid=12, entry_tid=0], [object_number=79, tag_tid=11, entry_tid=17175], [object_number=78, tag_tid=11, entry_tid=17174]]], state=connected] 2024-04-06T09:39:01.241 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=72, tag_tid=11, entry_tid=13708], [object_number=75, tag_tid=11, entry_tid=13707], [object_number=74, tag_tid=11, entry_tid=13706], [object_number=73, tag_tid=11, entry_tid=13705]]], state=connected] 2024-04-06T09:39:01.244 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:39:01.245 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=77, tag_tid=12, entry_tid=1' 2024-04-06T09:39:01.246 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:39:01.246 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:39:01.246 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:39:01.246 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:39:01.246 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:39:01.247 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:39:01.247 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:39:01.247 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:39:01.247 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:39:01.247 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:39:01.247 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:39:01.247 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:39:01.247 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:39:01.248 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:39:01.248 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:39:01.293 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 18 2024-04-06T09:39:01.293 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 19 2024-04-06T09:39:01.293 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:39:01.293 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=77, tag_tid=12, entry_tid=1], [object_number=76, tag_tid=12, entry_tid=0], [object_number=79, tag_tid=11, entry_tid=17175], [object_number=78, tag_tid=11, entry_tid=17174]]], state=connected] 2024-04-06T09:39:01.293 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=72, tag_tid=11, entry_tid=13708], [object_number=75, tag_tid=11, entry_tid=13707], [object_number=74, tag_tid=11, entry_tid=13706], [object_number=73, tag_tid=11, entry_tid=13705]]], state=connected] 2024-04-06T09:39:01.296 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:39:01.299 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=72, tag_tid=11, entry_tid=13708' 2024-04-06T09:39:01.299 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=77, tag_tid=12, entry_tid=1' -a 'object_number=77, tag_tid=12, entry_tid=1' = 'object_number=72, tag_tid=11, entry_tid=13708' 2024-04-06T09:39:01.299 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=72, tag_tid=11, entry_tid=13708' '!=' 'object_number=72, tag_tid=11, entry_tid=13612' 2024-04-06T09:39:01.299 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:39:01.299 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=72, tag_tid=11, entry_tid=13708' = 'object_number=72, tag_tid=11, entry_tid=13612' 2024-04-06T09:39:01.299 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=72, tag_tid=11, entry_tid=13708' 2024-04-06T09:39:01.299 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=77, tag_tid=12, entry_tid=1' 2024-04-06T09:39:01.299 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:39:01.299 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:39:01.300 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=12 2024-04-06T09:39:01.300 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=72, tag_tid=11, entry_tid=13708' 2024-04-06T09:39:01.300 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:39:01.300 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:39:01.301 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=11 2024-04-06T09:39:01.302 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=77, tag_tid=12, entry_tid=1' 2024-04-06T09:39:01.302 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:39:01.302 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:39:01.303 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:39:01.304 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=72, tag_tid=11, entry_tid=13708' 2024-04-06T09:39:01.304 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:39:01.304 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:39:01.305 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=13708 2024-04-06T09:39:01.305 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 12 = 11 -a 1 -le 13708 2024-04-06T09:39:01.305 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:39:01.305 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:39:01.305 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:39:01.507 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:39:01.507 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:39:01.507 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:39:01.507 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:39:01.507 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:39:01.507 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:39:01.507 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:39:01.507 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:39:01.507 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:39:01.507 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:39:01.507 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:39:01.507 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:39:01.507 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:39:01.507 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:39:01.508 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:39:01.508 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:39:01.508 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:39:01.512 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:39:01.512 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:39:01.512 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:39:01.512 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:39:01.512 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:39:01.512 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:39:01.512 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:39:01.512 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:39:01.512 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:39:01.512 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:39:01.513 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:39:01.513 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:39:01.513 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:39:01.513 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:39:01.513 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:39:01.513 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:39:01.513 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:39:01.513 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:39:01.513 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:39:01.513 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:39:01.513 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:39:01.513 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:39:01.513 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:39:01.513 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:39:01.513 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:39:01.513 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:39:01.514 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:39:01.514 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:39:01.514 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:39:01.514 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:39:01.514 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:39:01.514 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:39:01.536 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:39:01.537 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:39:01.537 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:39:01.537 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:39:01.663 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:39:01.664 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:39:01.664 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:39:01.664 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:39:01.665 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:39:01.665 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:39:01.665 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:39:01.665 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:39:01.665 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:39:01.665 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:39:01.665 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:39:01.665 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:39:01.665 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:39:01.670 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:39:01.670 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:39:01.670 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:39:01.714 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 18 2024-04-06T09:39:01.714 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 19 2024-04-06T09:39:01.714 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:39:01.714 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=77, tag_tid=12, entry_tid=1], [object_number=76, tag_tid=12, entry_tid=0], [object_number=79, tag_tid=11, entry_tid=17175], [object_number=78, tag_tid=11, entry_tid=17174]]], state=connected] 2024-04-06T09:39:01.714 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=73, tag_tid=11, entry_tid=13793], [object_number=72, tag_tid=11, entry_tid=13792], [object_number=75, tag_tid=11, entry_tid=13791], [object_number=74, tag_tid=11, entry_tid=13790]]], state=connected] 2024-04-06T09:39:01.718 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:39:01.719 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=77, tag_tid=12, entry_tid=1' 2024-04-06T09:39:01.720 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:39:01.720 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:39:01.720 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:39:01.720 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:39:01.720 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:39:01.720 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:39:01.720 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:39:01.720 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:39:01.721 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:39:01.721 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:39:01.721 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:39:01.721 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:39:01.721 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:39:01.722 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:39:01.722 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:39:01.767 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 18 2024-04-06T09:39:01.767 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 19 2024-04-06T09:39:01.768 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:39:01.768 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=77, tag_tid=12, entry_tid=1], [object_number=76, tag_tid=12, entry_tid=0], [object_number=79, tag_tid=11, entry_tid=17175], [object_number=78, tag_tid=11, entry_tid=17174]]], state=connected] 2024-04-06T09:39:01.768 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=73, tag_tid=11, entry_tid=13793], [object_number=72, tag_tid=11, entry_tid=13792], [object_number=75, tag_tid=11, entry_tid=13791], [object_number=74, tag_tid=11, entry_tid=13790]]], state=connected] 2024-04-06T09:39:01.771 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:39:01.773 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=73, tag_tid=11, entry_tid=13793' 2024-04-06T09:39:01.773 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=77, tag_tid=12, entry_tid=1' -a 'object_number=77, tag_tid=12, entry_tid=1' = 'object_number=73, tag_tid=11, entry_tid=13793' 2024-04-06T09:39:01.773 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=73, tag_tid=11, entry_tid=13793' '!=' 'object_number=72, tag_tid=11, entry_tid=13708' 2024-04-06T09:39:01.773 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:39:01.773 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=73, tag_tid=11, entry_tid=13793' = 'object_number=72, tag_tid=11, entry_tid=13708' 2024-04-06T09:39:01.773 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=73, tag_tid=11, entry_tid=13793' 2024-04-06T09:39:01.773 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=77, tag_tid=12, entry_tid=1' 2024-04-06T09:39:01.774 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:39:01.774 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:39:01.774 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=12 2024-04-06T09:39:01.775 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=73, tag_tid=11, entry_tid=13793' 2024-04-06T09:39:01.775 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:39:01.775 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:39:01.776 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=11 2024-04-06T09:39:01.776 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=77, tag_tid=12, entry_tid=1' 2024-04-06T09:39:01.777 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:39:01.777 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:39:01.777 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:39:01.778 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=73, tag_tid=11, entry_tid=13793' 2024-04-06T09:39:01.778 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:39:01.778 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:39:01.779 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=13793 2024-04-06T09:39:01.779 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 12 = 11 -a 1 -le 13793 2024-04-06T09:39:01.780 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:39:01.780 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:39:01.780 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:39:01.981 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:39:01.982 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:39:01.982 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:39:01.982 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:39:01.982 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:39:01.982 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:39:01.982 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:39:01.982 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:39:01.982 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:39:01.982 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:39:01.982 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:39:01.982 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:39:01.983 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:39:01.983 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:39:01.983 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:39:01.983 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:39:01.983 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:39:01.987 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:39:01.987 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:39:01.987 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:39:01.987 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:39:01.987 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:39:01.987 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:39:01.987 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:39:01.987 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:39:01.987 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:39:01.987 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:39:01.988 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:39:01.988 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:39:01.988 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:39:01.988 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:39:01.988 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:39:01.988 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:39:01.988 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:39:01.988 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:39:01.988 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:39:01.988 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:39:01.988 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:39:01.988 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:39:01.988 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:39:01.988 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:39:01.988 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:39:01.989 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:39:01.989 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:39:01.989 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:39:01.989 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:39:01.989 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:39:01.989 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:39:01.989 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:39:02.013 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:39:02.013 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:39:02.013 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:39:02.013 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:39:02.146 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:39:02.147 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:39:02.147 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:39:02.147 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:39:02.147 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:39:02.148 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:39:02.148 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:39:02.148 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:39:02.148 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:39:02.148 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:39:02.148 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:39:02.149 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:39:02.149 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:39:02.151 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:39:02.151 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:39:02.151 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:39:02.202 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 18 2024-04-06T09:39:02.202 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 19 2024-04-06T09:39:02.202 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:39:02.202 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=77, tag_tid=12, entry_tid=1], [object_number=76, tag_tid=12, entry_tid=0], [object_number=79, tag_tid=11, entry_tid=17175], [object_number=78, tag_tid=11, entry_tid=17174]]], state=connected] 2024-04-06T09:39:02.203 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=72, tag_tid=11, entry_tid=13884], [object_number=75, tag_tid=11, entry_tid=13883], [object_number=74, tag_tid=11, entry_tid=13882], [object_number=73, tag_tid=11, entry_tid=13881]]], state=connected] 2024-04-06T09:39:02.206 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:39:02.207 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=77, tag_tid=12, entry_tid=1' 2024-04-06T09:39:02.208 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:39:02.208 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:39:02.208 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:39:02.208 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:39:02.208 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:39:02.208 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:39:02.208 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:39:02.208 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:39:02.208 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:39:02.208 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:39:02.209 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:39:02.209 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:39:02.210 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:39:02.210 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:39:02.210 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:39:02.257 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 18 2024-04-06T09:39:02.257 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 19 2024-04-06T09:39:02.257 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:39:02.257 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=77, tag_tid=12, entry_tid=1], [object_number=76, tag_tid=12, entry_tid=0], [object_number=79, tag_tid=11, entry_tid=17175], [object_number=78, tag_tid=11, entry_tid=17174]]], state=connected] 2024-04-06T09:39:02.257 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=72, tag_tid=11, entry_tid=13884], [object_number=75, tag_tid=11, entry_tid=13883], [object_number=74, tag_tid=11, entry_tid=13882], [object_number=73, tag_tid=11, entry_tid=13881]]], state=connected] 2024-04-06T09:39:02.260 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:39:02.262 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=72, tag_tid=11, entry_tid=13884' 2024-04-06T09:39:02.262 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=77, tag_tid=12, entry_tid=1' -a 'object_number=77, tag_tid=12, entry_tid=1' = 'object_number=72, tag_tid=11, entry_tid=13884' 2024-04-06T09:39:02.262 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=72, tag_tid=11, entry_tid=13884' '!=' 'object_number=73, tag_tid=11, entry_tid=13793' 2024-04-06T09:39:02.262 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:39:02.262 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=72, tag_tid=11, entry_tid=13884' = 'object_number=73, tag_tid=11, entry_tid=13793' 2024-04-06T09:39:02.262 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=72, tag_tid=11, entry_tid=13884' 2024-04-06T09:39:02.263 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=77, tag_tid=12, entry_tid=1' 2024-04-06T09:39:02.263 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:39:02.263 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:39:02.264 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=12 2024-04-06T09:39:02.264 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=72, tag_tid=11, entry_tid=13884' 2024-04-06T09:39:02.264 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:39:02.264 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:39:02.265 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=11 2024-04-06T09:39:02.266 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=77, tag_tid=12, entry_tid=1' 2024-04-06T09:39:02.266 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:39:02.266 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:39:02.267 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:39:02.267 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=72, tag_tid=11, entry_tid=13884' 2024-04-06T09:39:02.268 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:39:02.268 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:39:02.269 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=13884 2024-04-06T09:39:02.269 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 12 = 11 -a 1 -le 13884 2024-04-06T09:39:02.269 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:39:02.269 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:39:02.269 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:39:02.471 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:39:02.471 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:39:02.471 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:39:02.471 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:39:02.472 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:39:02.472 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:39:02.472 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:39:02.472 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:39:02.472 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:39:02.472 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:39:02.472 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:39:02.472 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:39:02.472 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:39:02.472 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:39:02.472 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:39:02.473 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:39:02.473 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:39:02.476 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:39:02.476 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:39:02.476 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:39:02.476 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:39:02.477 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:39:02.477 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:39:02.477 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:39:02.477 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:39:02.477 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:39:02.477 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:39:02.477 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:39:02.477 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:39:02.477 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:39:02.477 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:39:02.477 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:39:02.477 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:39:02.477 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:39:02.477 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:39:02.477 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:39:02.478 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:39:02.478 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:39:02.478 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:39:02.478 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:39:02.478 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:39:02.478 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:39:02.478 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:39:02.478 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:39:02.478 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:39:02.478 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:39:02.478 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:39:02.478 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:39:02.478 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:39:02.503 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:39:02.504 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:39:02.504 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:39:02.504 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:39:02.630 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:39:02.631 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:39:02.631 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:39:02.631 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:39:02.631 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:39:02.632 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:39:02.632 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:39:02.632 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:39:02.632 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:39:02.632 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:39:02.632 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:39:02.632 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:39:02.632 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:39:02.636 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:39:02.636 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:39:02.636 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:39:02.683 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 18 2024-04-06T09:39:02.683 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 19 2024-04-06T09:39:02.684 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:39:02.684 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=77, tag_tid=12, entry_tid=1], [object_number=76, tag_tid=12, entry_tid=0], [object_number=79, tag_tid=11, entry_tid=17175], [object_number=78, tag_tid=11, entry_tid=17174]]], state=connected] 2024-04-06T09:39:02.684 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=74, tag_tid=11, entry_tid=13978], [object_number=73, tag_tid=11, entry_tid=13977], [object_number=72, tag_tid=11, entry_tid=13976], [object_number=75, tag_tid=11, entry_tid=13975]]], state=connected] 2024-04-06T09:39:02.688 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:39:02.689 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=77, tag_tid=12, entry_tid=1' 2024-04-06T09:39:02.690 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:39:02.690 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:39:02.690 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:39:02.690 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:39:02.691 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:39:02.691 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:39:02.691 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:39:02.691 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:39:02.691 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:39:02.691 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:39:02.691 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:39:02.691 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:39:02.692 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:39:02.692 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:39:02.692 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:39:02.737 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 18 2024-04-06T09:39:02.737 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 19 2024-04-06T09:39:02.737 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:39:02.737 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=77, tag_tid=12, entry_tid=1], [object_number=76, tag_tid=12, entry_tid=0], [object_number=79, tag_tid=11, entry_tid=17175], [object_number=78, tag_tid=11, entry_tid=17174]]], state=connected] 2024-04-06T09:39:02.737 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=74, tag_tid=11, entry_tid=13978], [object_number=73, tag_tid=11, entry_tid=13977], [object_number=72, tag_tid=11, entry_tid=13976], [object_number=75, tag_tid=11, entry_tid=13975]]], state=connected] 2024-04-06T09:39:02.741 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:39:02.743 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=74, tag_tid=11, entry_tid=13978' 2024-04-06T09:39:02.743 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=77, tag_tid=12, entry_tid=1' -a 'object_number=77, tag_tid=12, entry_tid=1' = 'object_number=74, tag_tid=11, entry_tid=13978' 2024-04-06T09:39:02.743 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=74, tag_tid=11, entry_tid=13978' '!=' 'object_number=72, tag_tid=11, entry_tid=13884' 2024-04-06T09:39:02.743 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:39:02.743 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=74, tag_tid=11, entry_tid=13978' = 'object_number=72, tag_tid=11, entry_tid=13884' 2024-04-06T09:39:02.744 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=74, tag_tid=11, entry_tid=13978' 2024-04-06T09:39:02.744 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=77, tag_tid=12, entry_tid=1' 2024-04-06T09:39:02.744 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:39:02.744 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:39:02.744 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=12 2024-04-06T09:39:02.745 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=74, tag_tid=11, entry_tid=13978' 2024-04-06T09:39:02.745 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:39:02.745 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:39:02.746 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=11 2024-04-06T09:39:02.747 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=77, tag_tid=12, entry_tid=1' 2024-04-06T09:39:02.747 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:39:02.747 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:39:02.748 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:39:02.749 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=74, tag_tid=11, entry_tid=13978' 2024-04-06T09:39:02.749 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:39:02.749 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:39:02.750 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=13978 2024-04-06T09:39:02.750 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 12 = 11 -a 1 -le 13978 2024-04-06T09:39:02.750 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:39:02.750 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:39:02.750 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:39:02.952 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:39:02.952 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:39:02.952 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:39:02.952 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:39:02.952 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:39:02.952 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:39:02.952 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:39:02.952 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:39:02.953 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:39:02.953 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:39:02.953 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:39:02.953 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:39:02.953 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:39:02.953 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:39:02.953 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:39:02.953 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:39:02.954 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:39:02.957 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:39:02.957 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:39:02.957 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:39:02.957 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:39:02.958 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:39:02.958 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:39:02.958 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:39:02.958 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:39:02.958 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:39:02.958 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:39:02.958 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:39:02.958 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:39:02.958 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:39:02.958 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:39:02.958 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:39:02.958 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:39:02.958 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:39:02.958 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:39:02.959 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:39:02.959 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:39:02.959 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:39:02.959 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:39:02.959 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:39:02.959 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:39:02.959 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:39:02.959 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:39:02.959 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:39:02.959 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:39:02.959 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:39:02.959 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:39:02.959 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:39:02.959 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:39:02.984 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:39:02.984 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:39:02.984 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:39:02.984 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:39:03.117 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:39:03.118 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:39:03.118 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:39:03.118 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:39:03.118 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:39:03.118 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:39:03.118 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:39:03.118 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:39:03.118 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:39:03.118 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:39:03.119 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:39:03.119 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:39:03.119 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:39:03.121 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:39:03.121 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:39:03.121 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:39:03.168 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 18 2024-04-06T09:39:03.168 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 19 2024-04-06T09:39:03.168 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:39:03.168 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=77, tag_tid=12, entry_tid=1], [object_number=76, tag_tid=12, entry_tid=0], [object_number=79, tag_tid=11, entry_tid=17175], [object_number=78, tag_tid=11, entry_tid=17174]]], state=connected] 2024-04-06T09:39:03.168 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=74, tag_tid=11, entry_tid=14082], [object_number=73, tag_tid=11, entry_tid=14081], [object_number=72, tag_tid=11, entry_tid=14080], [object_number=75, tag_tid=11, entry_tid=14079]]], state=connected] 2024-04-06T09:39:03.170 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:39:03.172 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=77, tag_tid=12, entry_tid=1' 2024-04-06T09:39:03.173 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:39:03.173 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:39:03.173 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:39:03.174 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:39:03.174 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:39:03.174 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:39:03.174 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:39:03.174 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:39:03.174 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:39:03.174 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:39:03.174 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:39:03.174 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:39:03.175 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:39:03.175 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:39:03.175 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:39:03.221 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 18 2024-04-06T09:39:03.221 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 19 2024-04-06T09:39:03.221 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:39:03.221 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=77, tag_tid=12, entry_tid=1], [object_number=76, tag_tid=12, entry_tid=0], [object_number=79, tag_tid=11, entry_tid=17175], [object_number=78, tag_tid=11, entry_tid=17174]]], state=connected] 2024-04-06T09:39:03.221 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=74, tag_tid=11, entry_tid=14082], [object_number=73, tag_tid=11, entry_tid=14081], [object_number=72, tag_tid=11, entry_tid=14080], [object_number=75, tag_tid=11, entry_tid=14079]]], state=connected] 2024-04-06T09:39:03.224 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:39:03.226 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=74, tag_tid=11, entry_tid=14082' 2024-04-06T09:39:03.226 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=77, tag_tid=12, entry_tid=1' -a 'object_number=77, tag_tid=12, entry_tid=1' = 'object_number=74, tag_tid=11, entry_tid=14082' 2024-04-06T09:39:03.226 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=74, tag_tid=11, entry_tid=14082' '!=' 'object_number=74, tag_tid=11, entry_tid=13978' 2024-04-06T09:39:03.227 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:39:03.227 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=74, tag_tid=11, entry_tid=14082' = 'object_number=74, tag_tid=11, entry_tid=13978' 2024-04-06T09:39:03.227 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=74, tag_tid=11, entry_tid=14082' 2024-04-06T09:39:03.227 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=77, tag_tid=12, entry_tid=1' 2024-04-06T09:39:03.227 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:39:03.227 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:39:03.228 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=12 2024-04-06T09:39:03.228 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=74, tag_tid=11, entry_tid=14082' 2024-04-06T09:39:03.228 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:39:03.228 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:39:03.229 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=11 2024-04-06T09:39:03.230 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=77, tag_tid=12, entry_tid=1' 2024-04-06T09:39:03.230 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:39:03.230 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:39:03.231 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:39:03.231 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=74, tag_tid=11, entry_tid=14082' 2024-04-06T09:39:03.231 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:39:03.232 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:39:03.232 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=14082 2024-04-06T09:39:03.233 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 12 = 11 -a 1 -le 14082 2024-04-06T09:39:03.233 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:39:03.233 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:39:03.233 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:39:03.434 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:39:03.435 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:39:03.435 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:39:03.435 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:39:03.435 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:39:03.435 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:39:03.435 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:39:03.435 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:39:03.435 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:39:03.435 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:39:03.435 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:39:03.435 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:39:03.436 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:39:03.436 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:39:03.436 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:39:03.436 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:39:03.436 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:39:03.439 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:39:03.439 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:39:03.439 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:39:03.439 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:39:03.439 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:39:03.439 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:39:03.439 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:39:03.440 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:39:03.440 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:39:03.440 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:39:03.440 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:39:03.440 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:39:03.440 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:39:03.440 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:39:03.440 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:39:03.440 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:39:03.440 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:39:03.440 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:39:03.440 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:39:03.440 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:39:03.440 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:39:03.440 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:39:03.441 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:39:03.441 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:39:03.441 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:39:03.441 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:39:03.441 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:39:03.441 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:39:03.441 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:39:03.441 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:39:03.441 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:39:03.441 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:39:03.465 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:39:03.465 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:39:03.465 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:39:03.465 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:39:03.596 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:39:03.597 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:39:03.598 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:39:03.598 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:39:03.598 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:39:03.598 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:39:03.598 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:39:03.598 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:39:03.598 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:39:03.598 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:39:03.599 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:39:03.599 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:39:03.599 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:39:03.600 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:39:03.600 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:39:03.601 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:39:03.653 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 18 2024-04-06T09:39:03.653 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 19 2024-04-06T09:39:03.653 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:39:03.653 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=77, tag_tid=12, entry_tid=1], [object_number=76, tag_tid=12, entry_tid=0], [object_number=79, tag_tid=11, entry_tid=17175], [object_number=78, tag_tid=11, entry_tid=17174]]], state=connected] 2024-04-06T09:39:03.653 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=74, tag_tid=11, entry_tid=14162], [object_number=73, tag_tid=11, entry_tid=14161], [object_number=72, tag_tid=11, entry_tid=14160], [object_number=75, tag_tid=11, entry_tid=14159]]], state=connected] 2024-04-06T09:39:03.657 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:39:03.658 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=77, tag_tid=12, entry_tid=1' 2024-04-06T09:39:03.659 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:39:03.659 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:39:03.660 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:39:03.660 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:39:03.660 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:39:03.660 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:39:03.660 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:39:03.660 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:39:03.660 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:39:03.660 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:39:03.660 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:39:03.660 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:39:03.661 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:39:03.661 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:39:03.661 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:39:03.707 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 18 2024-04-06T09:39:03.707 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 19 2024-04-06T09:39:03.707 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:39:03.707 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=77, tag_tid=12, entry_tid=1], [object_number=76, tag_tid=12, entry_tid=0], [object_number=79, tag_tid=11, entry_tid=17175], [object_number=78, tag_tid=11, entry_tid=17174]]], state=connected] 2024-04-06T09:39:03.707 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=74, tag_tid=11, entry_tid=14162], [object_number=73, tag_tid=11, entry_tid=14161], [object_number=72, tag_tid=11, entry_tid=14160], [object_number=75, tag_tid=11, entry_tid=14159]]], state=connected] 2024-04-06T09:39:03.710 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:39:03.712 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=74, tag_tid=11, entry_tid=14162' 2024-04-06T09:39:03.712 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=77, tag_tid=12, entry_tid=1' -a 'object_number=77, tag_tid=12, entry_tid=1' = 'object_number=74, tag_tid=11, entry_tid=14162' 2024-04-06T09:39:03.712 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=74, tag_tid=11, entry_tid=14162' '!=' 'object_number=74, tag_tid=11, entry_tid=14082' 2024-04-06T09:39:03.712 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:39:03.712 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=74, tag_tid=11, entry_tid=14162' = 'object_number=74, tag_tid=11, entry_tid=14082' 2024-04-06T09:39:03.712 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=74, tag_tid=11, entry_tid=14162' 2024-04-06T09:39:03.713 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=77, tag_tid=12, entry_tid=1' 2024-04-06T09:39:03.713 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:39:03.713 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:39:03.713 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=12 2024-04-06T09:39:03.714 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=74, tag_tid=11, entry_tid=14162' 2024-04-06T09:39:03.714 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:39:03.714 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:39:03.715 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=11 2024-04-06T09:39:03.716 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=77, tag_tid=12, entry_tid=1' 2024-04-06T09:39:03.716 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:39:03.716 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:39:03.717 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:39:03.718 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=74, tag_tid=11, entry_tid=14162' 2024-04-06T09:39:03.718 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:39:03.718 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:39:03.719 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=14162 2024-04-06T09:39:03.719 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 12 = 11 -a 1 -le 14162 2024-04-06T09:39:03.719 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:39:03.719 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:39:03.720 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:39:03.921 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:39:03.921 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:39:03.921 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:39:03.921 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:39:03.921 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:39:03.921 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:39:03.921 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:39:03.921 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:39:03.922 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:39:03.922 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:39:03.922 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:39:03.922 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:39:03.922 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:39:03.922 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:39:03.922 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:39:03.923 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:39:03.923 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:39:03.926 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:39:03.926 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:39:03.926 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:39:03.927 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:39:03.927 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:39:03.927 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:39:03.927 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:39:03.927 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:39:03.927 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:39:03.927 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:39:03.927 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:39:03.927 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:39:03.927 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:39:03.927 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:39:03.927 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:39:03.927 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:39:03.927 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:39:03.928 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:39:03.928 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:39:03.928 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:39:03.928 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:39:03.928 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:39:03.928 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:39:03.928 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:39:03.928 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:39:03.928 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:39:03.928 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:39:03.928 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:39:03.928 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:39:03.928 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:39:03.928 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:39:03.929 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:39:03.952 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:39:03.953 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:39:03.953 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:39:03.953 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:39:04.083 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:39:04.084 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:39:04.084 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:39:04.084 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:39:04.085 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:39:04.085 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:39:04.085 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:39:04.085 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:39:04.085 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:39:04.085 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:39:04.085 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:39:04.085 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:39:04.085 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:39:04.088 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:39:04.088 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:39:04.088 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:39:04.139 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 18 2024-04-06T09:39:04.140 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 19 2024-04-06T09:39:04.140 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:39:04.140 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=77, tag_tid=12, entry_tid=1], [object_number=76, tag_tid=12, entry_tid=0], [object_number=79, tag_tid=11, entry_tid=17175], [object_number=78, tag_tid=11, entry_tid=17174]]], state=connected] 2024-04-06T09:39:04.140 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=74, tag_tid=11, entry_tid=14262], [object_number=73, tag_tid=11, entry_tid=14261], [object_number=72, tag_tid=11, entry_tid=14260], [object_number=75, tag_tid=11, entry_tid=14259]]], state=connected] 2024-04-06T09:39:04.143 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:39:04.144 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=77, tag_tid=12, entry_tid=1' 2024-04-06T09:39:04.145 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:39:04.145 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:39:04.145 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:39:04.145 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:39:04.145 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:39:04.145 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:39:04.146 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:39:04.146 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:39:04.146 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:39:04.146 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:39:04.146 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:39:04.146 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:39:04.146 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:39:04.147 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:39:04.147 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:39:04.192 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 18 2024-04-06T09:39:04.192 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 19 2024-04-06T09:39:04.192 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:39:04.193 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=77, tag_tid=12, entry_tid=1], [object_number=76, tag_tid=12, entry_tid=0], [object_number=79, tag_tid=11, entry_tid=17175], [object_number=78, tag_tid=11, entry_tid=17174]]], state=connected] 2024-04-06T09:39:04.193 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=74, tag_tid=11, entry_tid=14262], [object_number=73, tag_tid=11, entry_tid=14261], [object_number=72, tag_tid=11, entry_tid=14260], [object_number=75, tag_tid=11, entry_tid=14259]]], state=connected] 2024-04-06T09:39:04.196 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:39:04.199 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=74, tag_tid=11, entry_tid=14262' 2024-04-06T09:39:04.199 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=77, tag_tid=12, entry_tid=1' -a 'object_number=77, tag_tid=12, entry_tid=1' = 'object_number=74, tag_tid=11, entry_tid=14262' 2024-04-06T09:39:04.199 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=74, tag_tid=11, entry_tid=14262' '!=' 'object_number=74, tag_tid=11, entry_tid=14162' 2024-04-06T09:39:04.199 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:39:04.199 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=74, tag_tid=11, entry_tid=14262' = 'object_number=74, tag_tid=11, entry_tid=14162' 2024-04-06T09:39:04.199 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=74, tag_tid=11, entry_tid=14262' 2024-04-06T09:39:04.199 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=77, tag_tid=12, entry_tid=1' 2024-04-06T09:39:04.199 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:39:04.199 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:39:04.200 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=12 2024-04-06T09:39:04.200 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=74, tag_tid=11, entry_tid=14262' 2024-04-06T09:39:04.200 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:39:04.200 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:39:04.201 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=11 2024-04-06T09:39:04.202 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=77, tag_tid=12, entry_tid=1' 2024-04-06T09:39:04.202 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:39:04.202 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:39:04.203 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:39:04.204 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=74, tag_tid=11, entry_tid=14262' 2024-04-06T09:39:04.204 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:39:04.204 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:39:04.205 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=14262 2024-04-06T09:39:04.205 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 12 = 11 -a 1 -le 14262 2024-04-06T09:39:04.205 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:39:04.205 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:39:04.205 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:39:04.407 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:39:04.407 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:39:04.407 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:39:04.407 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:39:04.407 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:39:04.408 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:39:04.408 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:39:04.408 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:39:04.408 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:39:04.408 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:39:04.408 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:39:04.408 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:39:04.408 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:39:04.408 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:39:04.408 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:39:04.409 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:39:04.409 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:39:04.410 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:39:04.410 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:39:04.410 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:39:04.411 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:39:04.411 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:39:04.411 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:39:04.411 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:39:04.411 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:39:04.411 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:39:04.411 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:39:04.411 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:39:04.411 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:39:04.411 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:39:04.411 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:39:04.411 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:39:04.411 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:39:04.412 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:39:04.413 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:39:04.413 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:39:04.413 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:39:04.413 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:39:04.413 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:39:04.413 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:39:04.413 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:39:04.413 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:39:04.413 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:39:04.413 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:39:04.413 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:39:04.413 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:39:04.413 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:39:04.413 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:39:04.413 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:39:04.437 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:39:04.438 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:39:04.438 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:39:04.438 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:39:04.577 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:39:04.579 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:39:04.579 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:39:04.579 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:39:04.579 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:39:04.579 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:39:04.579 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:39:04.579 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:39:04.579 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:39:04.579 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:39:04.580 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:39:04.580 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:39:04.580 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:39:04.581 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:39:04.582 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:39:04.582 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:39:04.627 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 18 2024-04-06T09:39:04.627 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 19 2024-04-06T09:39:04.627 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:39:04.627 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=77, tag_tid=12, entry_tid=1], [object_number=76, tag_tid=12, entry_tid=0], [object_number=79, tag_tid=11, entry_tid=17175], [object_number=78, tag_tid=11, entry_tid=17174]]], state=connected] 2024-04-06T09:39:04.627 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=75, tag_tid=11, entry_tid=14355], [object_number=74, tag_tid=11, entry_tid=14354], [object_number=73, tag_tid=11, entry_tid=14353], [object_number=72, tag_tid=11, entry_tid=14352]]], state=connected] 2024-04-06T09:39:04.630 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:39:04.631 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=77, tag_tid=12, entry_tid=1' 2024-04-06T09:39:04.633 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:39:04.633 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:39:04.633 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:39:04.633 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:39:04.633 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:39:04.633 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:39:04.633 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:39:04.633 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:39:04.633 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:39:04.633 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:39:04.633 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:39:04.633 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:39:04.634 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:39:04.634 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:39:04.634 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:39:04.679 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 18 2024-04-06T09:39:04.679 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 19 2024-04-06T09:39:04.679 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:39:04.679 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=77, tag_tid=12, entry_tid=1], [object_number=76, tag_tid=12, entry_tid=0], [object_number=79, tag_tid=11, entry_tid=17175], [object_number=78, tag_tid=11, entry_tid=17174]]], state=connected] 2024-04-06T09:39:04.679 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=75, tag_tid=11, entry_tid=14355], [object_number=74, tag_tid=11, entry_tid=14354], [object_number=73, tag_tid=11, entry_tid=14353], [object_number=72, tag_tid=11, entry_tid=14352]]], state=connected] 2024-04-06T09:39:04.682 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:39:04.684 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=75, tag_tid=11, entry_tid=14355' 2024-04-06T09:39:04.684 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=77, tag_tid=12, entry_tid=1' -a 'object_number=77, tag_tid=12, entry_tid=1' = 'object_number=75, tag_tid=11, entry_tid=14355' 2024-04-06T09:39:04.684 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=75, tag_tid=11, entry_tid=14355' '!=' 'object_number=74, tag_tid=11, entry_tid=14262' 2024-04-06T09:39:04.684 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:39:04.684 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=75, tag_tid=11, entry_tid=14355' = 'object_number=74, tag_tid=11, entry_tid=14262' 2024-04-06T09:39:04.684 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=75, tag_tid=11, entry_tid=14355' 2024-04-06T09:39:04.685 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=77, tag_tid=12, entry_tid=1' 2024-04-06T09:39:04.685 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:39:04.685 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:39:04.686 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=12 2024-04-06T09:39:04.686 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=75, tag_tid=11, entry_tid=14355' 2024-04-06T09:39:04.686 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:39:04.687 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:39:04.687 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=11 2024-04-06T09:39:04.688 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=77, tag_tid=12, entry_tid=1' 2024-04-06T09:39:04.688 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:39:04.688 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:39:04.689 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:39:04.690 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=75, tag_tid=11, entry_tid=14355' 2024-04-06T09:39:04.690 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:39:04.690 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:39:04.691 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=14355 2024-04-06T09:39:04.691 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 12 = 11 -a 1 -le 14355 2024-04-06T09:39:04.691 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:39:04.691 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:39:04.691 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:39:04.893 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:39:04.893 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:39:04.893 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:39:04.893 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:39:04.893 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:39:04.893 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:39:04.893 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:39:04.893 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:39:04.894 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:39:04.894 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:39:04.894 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:39:04.894 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:39:04.894 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:39:04.894 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:39:04.894 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:39:04.894 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:39:04.895 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:39:04.897 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:39:04.897 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:39:04.898 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:39:04.898 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:39:04.898 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:39:04.898 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:39:04.898 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:39:04.898 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:39:04.898 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:39:04.898 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:39:04.898 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:39:04.898 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:39:04.898 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:39:04.898 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:39:04.898 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:39:04.898 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:39:04.899 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:39:04.899 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:39:04.900 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:39:04.900 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:39:04.900 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:39:04.900 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:39:04.900 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:39:04.900 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:39:04.900 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:39:04.900 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:39:04.900 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:39:04.900 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:39:04.900 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:39:04.900 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:39:04.900 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:39:04.900 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:39:04.923 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:39:04.924 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:39:04.924 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:39:04.924 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:39:05.050 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:39:05.052 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:39:05.052 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:39:05.052 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:39:05.052 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:39:05.052 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:39:05.052 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:39:05.052 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:39:05.052 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:39:05.052 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:39:05.052 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:39:05.053 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:39:05.053 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:39:05.055 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:39:05.056 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:39:05.056 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:39:05.107 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 18 2024-04-06T09:39:05.107 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 19 2024-04-06T09:39:05.107 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:39:05.107 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=77, tag_tid=12, entry_tid=1], [object_number=76, tag_tid=12, entry_tid=0], [object_number=79, tag_tid=11, entry_tid=17175], [object_number=78, tag_tid=11, entry_tid=17174]]], state=connected] 2024-04-06T09:39:05.108 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=73, tag_tid=11, entry_tid=14449], [object_number=72, tag_tid=11, entry_tid=14448], [object_number=75, tag_tid=11, entry_tid=14447], [object_number=74, tag_tid=11, entry_tid=14446]]], state=connected] 2024-04-06T09:39:05.111 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:39:05.112 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=77, tag_tid=12, entry_tid=1' 2024-04-06T09:39:05.113 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:39:05.113 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:39:05.113 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:39:05.113 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:39:05.114 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:39:05.114 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:39:05.114 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:39:05.114 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:39:05.114 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:39:05.114 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:39:05.114 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:39:05.114 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:39:05.115 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:39:05.115 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:39:05.115 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:39:05.371 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 18 2024-04-06T09:39:05.371 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 19 2024-04-06T09:39:05.371 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:39:05.371 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=77, tag_tid=12, entry_tid=1], [object_number=76, tag_tid=12, entry_tid=0], [object_number=79, tag_tid=11, entry_tid=17175], [object_number=78, tag_tid=11, entry_tid=17174]]], state=connected] 2024-04-06T09:39:05.371 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=73, tag_tid=11, entry_tid=14449], [object_number=72, tag_tid=11, entry_tid=14448], [object_number=75, tag_tid=11, entry_tid=14447], [object_number=74, tag_tid=11, entry_tid=14446]]], state=connected] 2024-04-06T09:39:05.376 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:39:05.378 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=73, tag_tid=11, entry_tid=14449' 2024-04-06T09:39:05.378 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=77, tag_tid=12, entry_tid=1' -a 'object_number=77, tag_tid=12, entry_tid=1' = 'object_number=73, tag_tid=11, entry_tid=14449' 2024-04-06T09:39:05.378 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=73, tag_tid=11, entry_tid=14449' '!=' 'object_number=75, tag_tid=11, entry_tid=14355' 2024-04-06T09:39:05.379 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:39:05.379 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=73, tag_tid=11, entry_tid=14449' = 'object_number=75, tag_tid=11, entry_tid=14355' 2024-04-06T09:39:05.379 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=73, tag_tid=11, entry_tid=14449' 2024-04-06T09:39:05.380 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=77, tag_tid=12, entry_tid=1' 2024-04-06T09:39:05.380 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:39:05.380 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:39:05.381 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=12 2024-04-06T09:39:05.382 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=73, tag_tid=11, entry_tid=14449' 2024-04-06T09:39:05.382 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:39:05.382 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:39:05.383 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=11 2024-04-06T09:39:05.383 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=77, tag_tid=12, entry_tid=1' 2024-04-06T09:39:05.384 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:39:05.384 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:39:05.384 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:39:05.385 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=73, tag_tid=11, entry_tid=14449' 2024-04-06T09:39:05.386 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:39:05.386 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:39:05.390 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=14449 2024-04-06T09:39:05.390 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 12 = 11 -a 1 -le 14449 2024-04-06T09:39:05.390 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:39:05.391 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:39:05.391 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:39:05.589 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:39:05.589 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:39:05.589 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:39:05.589 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:39:05.589 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:39:05.589 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:39:05.589 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:39:05.590 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:39:05.590 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:39:05.590 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:39:05.590 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:39:05.590 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:39:05.590 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:39:05.590 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:39:05.590 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:39:05.590 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:39:05.592 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:39:05.593 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:39:05.593 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:39:05.593 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:39:05.593 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:39:05.593 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:39:05.594 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:39:05.594 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:39:05.594 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:39:05.594 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:39:05.594 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:39:05.594 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:39:05.594 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:39:05.594 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:39:05.594 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:39:05.594 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:39:05.594 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:39:05.598 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:39:05.598 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:39:05.598 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:39:05.598 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:39:05.599 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:39:05.599 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:39:05.599 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:39:05.599 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:39:05.599 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:39:05.599 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:39:05.599 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:39:05.599 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:39:05.599 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:39:05.599 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:39:05.599 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:39:05.599 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:39:05.621 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:39:05.621 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:39:05.621 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:39:05.621 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:39:05.750 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:39:05.751 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:39:05.751 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:39:05.751 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:39:05.752 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:39:05.752 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:39:05.752 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:39:05.752 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:39:05.752 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:39:05.752 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:39:05.752 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:39:05.753 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:39:05.753 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:39:05.756 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:39:05.756 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:39:05.756 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:39:05.808 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 18 2024-04-06T09:39:05.808 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 19 2024-04-06T09:39:05.808 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:39:05.808 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=77, tag_tid=12, entry_tid=1], [object_number=76, tag_tid=12, entry_tid=0], [object_number=79, tag_tid=11, entry_tid=17175], [object_number=78, tag_tid=11, entry_tid=17174]]], state=connected] 2024-04-06T09:39:05.808 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=73, tag_tid=11, entry_tid=14581], [object_number=72, tag_tid=11, entry_tid=14580], [object_number=75, tag_tid=11, entry_tid=14579], [object_number=74, tag_tid=11, entry_tid=14578]]], state=connected] 2024-04-06T09:39:05.811 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:39:05.814 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=77, tag_tid=12, entry_tid=1' 2024-04-06T09:39:05.814 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:39:05.814 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:39:05.814 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:39:05.814 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:39:05.814 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:39:05.814 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:39:05.814 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:39:05.814 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:39:05.815 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:39:05.815 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:39:05.815 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:39:05.815 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:39:05.815 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:39:05.815 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:39:05.815 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:39:05.862 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 18 2024-04-06T09:39:05.862 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 19 2024-04-06T09:39:05.862 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:39:05.862 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=77, tag_tid=12, entry_tid=1], [object_number=76, tag_tid=12, entry_tid=0], [object_number=79, tag_tid=11, entry_tid=17175], [object_number=78, tag_tid=11, entry_tid=17174]]], state=connected] 2024-04-06T09:39:05.862 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=73, tag_tid=11, entry_tid=14581], [object_number=72, tag_tid=11, entry_tid=14580], [object_number=75, tag_tid=11, entry_tid=14579], [object_number=74, tag_tid=11, entry_tid=14578]]], state=connected] 2024-04-06T09:39:05.865 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:39:05.867 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=73, tag_tid=11, entry_tid=14581' 2024-04-06T09:39:05.867 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=77, tag_tid=12, entry_tid=1' -a 'object_number=77, tag_tid=12, entry_tid=1' = 'object_number=73, tag_tid=11, entry_tid=14581' 2024-04-06T09:39:05.867 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=73, tag_tid=11, entry_tid=14581' '!=' 'object_number=73, tag_tid=11, entry_tid=14449' 2024-04-06T09:39:05.867 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:39:05.868 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=73, tag_tid=11, entry_tid=14581' = 'object_number=73, tag_tid=11, entry_tid=14449' 2024-04-06T09:39:05.868 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=73, tag_tid=11, entry_tid=14581' 2024-04-06T09:39:05.868 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=77, tag_tid=12, entry_tid=1' 2024-04-06T09:39:05.868 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:39:05.868 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:39:05.869 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=12 2024-04-06T09:39:05.869 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=73, tag_tid=11, entry_tid=14581' 2024-04-06T09:39:05.869 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:39:05.870 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:39:05.870 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=11 2024-04-06T09:39:05.871 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=77, tag_tid=12, entry_tid=1' 2024-04-06T09:39:05.871 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:39:05.871 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:39:05.872 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:39:05.873 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=73, tag_tid=11, entry_tid=14581' 2024-04-06T09:39:05.873 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:39:05.873 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:39:05.874 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=14581 2024-04-06T09:39:05.874 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 12 = 11 -a 1 -le 14581 2024-04-06T09:39:05.874 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:39:05.875 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:39:05.875 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:39:06.078 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:39:06.078 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:39:06.078 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:39:06.078 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:39:06.078 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:39:06.079 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:39:06.079 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:39:06.079 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:39:06.079 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:39:06.079 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:39:06.079 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:39:06.079 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:39:06.080 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:39:06.080 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:39:06.080 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:39:06.080 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:39:06.080 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:39:06.084 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:39:06.084 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:39:06.084 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:39:06.084 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:39:06.085 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:39:06.085 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:39:06.085 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:39:06.085 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:39:06.085 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:39:06.085 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:39:06.085 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:39:06.085 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:39:06.086 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:39:06.086 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:39:06.086 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:39:06.086 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:39:06.086 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:39:06.086 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:39:06.086 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:39:06.087 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:39:06.087 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:39:06.087 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:39:06.087 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:39:06.087 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:39:06.087 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:39:06.087 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:39:06.087 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:39:06.087 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:39:06.088 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:39:06.088 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:39:06.088 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:39:06.088 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:39:06.108 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:39:06.108 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:39:06.108 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:39:06.108 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:39:06.243 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:39:06.244 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:39:06.244 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:39:06.244 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:39:06.244 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:39:06.244 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:39:06.244 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:39:06.244 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:39:06.244 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:39:06.245 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:39:06.245 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:39:06.245 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:39:06.245 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:39:06.248 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:39:06.249 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:39:06.249 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:39:06.294 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 18 2024-04-06T09:39:06.294 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 19 2024-04-06T09:39:06.294 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:39:06.295 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=77, tag_tid=12, entry_tid=1], [object_number=76, tag_tid=12, entry_tid=0], [object_number=79, tag_tid=11, entry_tid=17175], [object_number=78, tag_tid=11, entry_tid=17174]]], state=connected] 2024-04-06T09:39:06.295 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=73, tag_tid=11, entry_tid=14669], [object_number=72, tag_tid=11, entry_tid=14668], [object_number=75, tag_tid=11, entry_tid=14667], [object_number=74, tag_tid=11, entry_tid=14666]]], state=connected] 2024-04-06T09:39:06.298 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:39:06.299 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=77, tag_tid=12, entry_tid=1' 2024-04-06T09:39:06.300 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:39:06.300 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:39:06.300 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:39:06.300 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:39:06.300 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:39:06.300 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:39:06.300 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:39:06.301 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:39:06.301 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:39:06.301 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:39:06.301 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:39:06.301 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:39:06.301 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:39:06.302 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:39:06.302 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:39:06.347 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 18 2024-04-06T09:39:06.347 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 19 2024-04-06T09:39:06.347 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:39:06.347 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=77, tag_tid=12, entry_tid=1], [object_number=76, tag_tid=12, entry_tid=0], [object_number=79, tag_tid=11, entry_tid=17175], [object_number=78, tag_tid=11, entry_tid=17174]]], state=connected] 2024-04-06T09:39:06.348 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=73, tag_tid=11, entry_tid=14669], [object_number=72, tag_tid=11, entry_tid=14668], [object_number=75, tag_tid=11, entry_tid=14667], [object_number=74, tag_tid=11, entry_tid=14666]]], state=connected] 2024-04-06T09:39:06.351 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:39:06.352 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=73, tag_tid=11, entry_tid=14669' 2024-04-06T09:39:06.353 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=77, tag_tid=12, entry_tid=1' -a 'object_number=77, tag_tid=12, entry_tid=1' = 'object_number=73, tag_tid=11, entry_tid=14669' 2024-04-06T09:39:06.353 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=73, tag_tid=11, entry_tid=14669' '!=' 'object_number=73, tag_tid=11, entry_tid=14581' 2024-04-06T09:39:06.353 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:39:06.353 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=73, tag_tid=11, entry_tid=14669' = 'object_number=73, tag_tid=11, entry_tid=14581' 2024-04-06T09:39:06.353 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=73, tag_tid=11, entry_tid=14669' 2024-04-06T09:39:06.353 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=77, tag_tid=12, entry_tid=1' 2024-04-06T09:39:06.354 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:39:06.354 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:39:06.354 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=12 2024-04-06T09:39:06.355 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=73, tag_tid=11, entry_tid=14669' 2024-04-06T09:39:06.355 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:39:06.355 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:39:06.356 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=11 2024-04-06T09:39:06.356 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=77, tag_tid=12, entry_tid=1' 2024-04-06T09:39:06.356 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:39:06.357 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:39:06.357 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:39:06.358 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=73, tag_tid=11, entry_tid=14669' 2024-04-06T09:39:06.358 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:39:06.358 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:39:06.359 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=14669 2024-04-06T09:39:06.359 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 12 = 11 -a 1 -le 14669 2024-04-06T09:39:06.360 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:39:06.360 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:39:06.360 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:39:06.561 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:39:06.561 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:39:06.562 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:39:06.562 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:39:06.562 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:39:06.562 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:39:06.562 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:39:06.562 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:39:06.562 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:39:06.562 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:39:06.562 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:39:06.562 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:39:06.562 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:39:06.562 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:39:06.562 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:39:06.563 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:39:06.563 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:39:06.566 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:39:06.567 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:39:06.567 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:39:06.567 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:39:06.567 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:39:06.567 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:39:06.567 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:39:06.567 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:39:06.567 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:39:06.567 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:39:06.567 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:39:06.567 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:39:06.567 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:39:06.568 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:39:06.568 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:39:06.568 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:39:06.568 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:39:06.568 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:39:06.568 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:39:06.568 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:39:06.568 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:39:06.568 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:39:06.568 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:39:06.568 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:39:06.569 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:39:06.569 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:39:06.569 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:39:06.569 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:39:06.569 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:39:06.569 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:39:06.569 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:39:06.569 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:39:06.592 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:39:06.592 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:39:06.593 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:39:06.593 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:39:06.724 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:39:06.725 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:39:06.725 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:39:06.726 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:39:06.726 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:39:06.726 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:39:06.726 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:39:06.726 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:39:06.726 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:39:06.726 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:39:06.727 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:39:06.727 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:39:06.727 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:39:06.728 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:39:06.728 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:39:06.728 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:39:06.781 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 18 2024-04-06T09:39:06.781 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 19 2024-04-06T09:39:06.781 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:39:06.781 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=77, tag_tid=12, entry_tid=1], [object_number=76, tag_tid=12, entry_tid=0], [object_number=79, tag_tid=11, entry_tid=17175], [object_number=78, tag_tid=11, entry_tid=17174]]], state=connected] 2024-04-06T09:39:06.781 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=75, tag_tid=11, entry_tid=14751], [object_number=74, tag_tid=11, entry_tid=14750], [object_number=73, tag_tid=11, entry_tid=14749], [object_number=72, tag_tid=11, entry_tid=14748]]], state=connected] 2024-04-06T09:39:06.785 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:39:06.786 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=77, tag_tid=12, entry_tid=1' 2024-04-06T09:39:06.787 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:39:06.787 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:39:06.787 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:39:06.787 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:39:06.788 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:39:06.788 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:39:06.788 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:39:06.788 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:39:06.788 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:39:06.788 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:39:06.788 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:39:06.788 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:39:06.789 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:39:06.789 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:39:06.789 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:39:06.836 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 18 2024-04-06T09:39:06.836 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 19 2024-04-06T09:39:06.836 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:39:06.836 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=77, tag_tid=12, entry_tid=1], [object_number=76, tag_tid=12, entry_tid=0], [object_number=79, tag_tid=11, entry_tid=17175], [object_number=78, tag_tid=11, entry_tid=17174]]], state=connected] 2024-04-06T09:39:06.836 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=75, tag_tid=11, entry_tid=14751], [object_number=74, tag_tid=11, entry_tid=14750], [object_number=73, tag_tid=11, entry_tid=14749], [object_number=72, tag_tid=11, entry_tid=14748]]], state=connected] 2024-04-06T09:39:06.839 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:39:06.841 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=75, tag_tid=11, entry_tid=14751' 2024-04-06T09:39:06.841 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=77, tag_tid=12, entry_tid=1' -a 'object_number=77, tag_tid=12, entry_tid=1' = 'object_number=75, tag_tid=11, entry_tid=14751' 2024-04-06T09:39:06.841 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=75, tag_tid=11, entry_tid=14751' '!=' 'object_number=73, tag_tid=11, entry_tid=14669' 2024-04-06T09:39:06.841 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:39:06.841 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=75, tag_tid=11, entry_tid=14751' = 'object_number=73, tag_tid=11, entry_tid=14669' 2024-04-06T09:39:06.841 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=75, tag_tid=11, entry_tid=14751' 2024-04-06T09:39:06.842 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=77, tag_tid=12, entry_tid=1' 2024-04-06T09:39:06.842 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:39:06.842 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:39:06.843 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=12 2024-04-06T09:39:06.844 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=75, tag_tid=11, entry_tid=14751' 2024-04-06T09:39:06.844 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:39:06.844 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:39:06.844 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=11 2024-04-06T09:39:06.845 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=77, tag_tid=12, entry_tid=1' 2024-04-06T09:39:06.845 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:39:06.845 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:39:06.846 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:39:06.847 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=75, tag_tid=11, entry_tid=14751' 2024-04-06T09:39:06.847 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:39:06.847 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:39:06.848 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=14751 2024-04-06T09:39:06.848 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 12 = 11 -a 1 -le 14751 2024-04-06T09:39:06.848 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:39:06.848 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:39:06.848 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:39:07.050 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:39:07.050 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:39:07.051 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:39:07.051 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:39:07.051 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:39:07.051 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:39:07.051 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:39:07.051 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:39:07.051 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:39:07.051 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:39:07.051 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:39:07.051 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:39:07.051 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:39:07.051 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:39:07.051 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:39:07.052 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:39:07.052 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:39:07.055 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:39:07.055 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:39:07.055 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:39:07.055 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:39:07.055 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:39:07.056 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:39:07.056 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:39:07.056 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:39:07.056 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:39:07.056 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:39:07.056 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:39:07.056 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:39:07.056 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:39:07.056 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:39:07.056 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:39:07.056 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:39:07.056 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:39:07.056 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:39:07.056 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:39:07.056 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:39:07.057 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:39:07.057 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:39:07.057 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:39:07.057 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:39:07.057 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:39:07.057 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:39:07.057 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:39:07.057 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:39:07.057 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:39:07.057 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:39:07.057 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:39:07.057 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:39:07.080 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:39:07.080 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:39:07.080 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:39:07.080 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:39:07.208 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:39:07.209 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:39:07.210 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:39:07.210 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:39:07.210 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:39:07.210 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:39:07.210 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:39:07.211 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:39:07.211 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:39:07.211 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:39:07.212 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:39:07.212 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:39:07.212 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:39:07.214 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:39:07.214 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:39:07.214 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:39:07.260 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 18 2024-04-06T09:39:07.260 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 19 2024-04-06T09:39:07.260 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:39:07.260 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=77, tag_tid=12, entry_tid=1], [object_number=76, tag_tid=12, entry_tid=0], [object_number=79, tag_tid=11, entry_tid=17175], [object_number=78, tag_tid=11, entry_tid=17174]]], state=connected] 2024-04-06T09:39:07.260 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=74, tag_tid=11, entry_tid=14834], [object_number=73, tag_tid=11, entry_tid=14833], [object_number=72, tag_tid=11, entry_tid=14832], [object_number=75, tag_tid=11, entry_tid=14831]]], state=connected] 2024-04-06T09:39:07.263 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:39:07.264 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=77, tag_tid=12, entry_tid=1' 2024-04-06T09:39:07.266 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:39:07.266 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:39:07.266 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:39:07.266 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:39:07.266 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:39:07.266 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:39:07.266 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:39:07.266 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:39:07.266 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:39:07.266 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:39:07.266 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:39:07.267 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:39:07.267 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:39:07.267 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:39:07.267 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:39:07.313 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 18 2024-04-06T09:39:07.313 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 19 2024-04-06T09:39:07.313 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:39:07.313 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=77, tag_tid=12, entry_tid=1], [object_number=76, tag_tid=12, entry_tid=0], [object_number=79, tag_tid=11, entry_tid=17175], [object_number=78, tag_tid=11, entry_tid=17174]]], state=connected] 2024-04-06T09:39:07.313 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=74, tag_tid=11, entry_tid=14834], [object_number=73, tag_tid=11, entry_tid=14833], [object_number=72, tag_tid=11, entry_tid=14832], [object_number=75, tag_tid=11, entry_tid=14831]]], state=connected] 2024-04-06T09:39:07.317 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:39:07.319 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=74, tag_tid=11, entry_tid=14834' 2024-04-06T09:39:07.320 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=77, tag_tid=12, entry_tid=1' -a 'object_number=77, tag_tid=12, entry_tid=1' = 'object_number=74, tag_tid=11, entry_tid=14834' 2024-04-06T09:39:07.320 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=74, tag_tid=11, entry_tid=14834' '!=' 'object_number=75, tag_tid=11, entry_tid=14751' 2024-04-06T09:39:07.320 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:39:07.320 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=74, tag_tid=11, entry_tid=14834' = 'object_number=75, tag_tid=11, entry_tid=14751' 2024-04-06T09:39:07.320 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=74, tag_tid=11, entry_tid=14834' 2024-04-06T09:39:07.320 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=77, tag_tid=12, entry_tid=1' 2024-04-06T09:39:07.321 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:39:07.321 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:39:07.323 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=12 2024-04-06T09:39:07.323 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=74, tag_tid=11, entry_tid=14834' 2024-04-06T09:39:07.323 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:39:07.323 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:39:07.323 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=11 2024-04-06T09:39:07.323 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=77, tag_tid=12, entry_tid=1' 2024-04-06T09:39:07.324 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:39:07.324 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:39:07.325 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:39:07.326 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=74, tag_tid=11, entry_tid=14834' 2024-04-06T09:39:07.326 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:39:07.326 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:39:07.326 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=14834 2024-04-06T09:39:07.326 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 12 = 11 -a 1 -le 14834 2024-04-06T09:39:07.326 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:39:07.326 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:39:07.327 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:39:07.527 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:39:07.527 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:39:07.528 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:39:07.528 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:39:07.528 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:39:07.528 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:39:07.528 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:39:07.528 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:39:07.528 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:39:07.528 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:39:07.528 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:39:07.528 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:39:07.528 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:39:07.528 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:39:07.528 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:39:07.529 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:39:07.529 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:39:07.532 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:39:07.532 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:39:07.532 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:39:07.532 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:39:07.532 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:39:07.532 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:39:07.532 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:39:07.532 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:39:07.532 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:39:07.532 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:39:07.532 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:39:07.533 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:39:07.533 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:39:07.533 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:39:07.533 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:39:07.533 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:39:07.534 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:39:07.534 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:39:07.534 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:39:07.534 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:39:07.534 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:39:07.534 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:39:07.534 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:39:07.534 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:39:07.534 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:39:07.534 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:39:07.534 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:39:07.534 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:39:07.535 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:39:07.535 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:39:07.535 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:39:07.535 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:39:07.559 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:39:07.559 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:39:07.560 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:39:07.560 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:39:07.691 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:39:07.692 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:39:07.692 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:39:07.692 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:39:07.692 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:39:07.692 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:39:07.692 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:39:07.692 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:39:07.692 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:39:07.693 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:39:07.693 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:39:07.693 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:39:07.693 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:39:07.694 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:39:07.695 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:39:07.695 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:39:07.748 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 18 2024-04-06T09:39:07.748 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 19 2024-04-06T09:39:07.748 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:39:07.748 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=77, tag_tid=12, entry_tid=1], [object_number=76, tag_tid=12, entry_tid=0], [object_number=79, tag_tid=11, entry_tid=17175], [object_number=78, tag_tid=11, entry_tid=17174]]], state=connected] 2024-04-06T09:39:07.748 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=75, tag_tid=11, entry_tid=14919], [object_number=74, tag_tid=11, entry_tid=14918], [object_number=73, tag_tid=11, entry_tid=14917], [object_number=72, tag_tid=11, entry_tid=14916]]], state=connected] 2024-04-06T09:39:07.751 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:39:07.752 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=77, tag_tid=12, entry_tid=1' 2024-04-06T09:39:07.753 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:39:07.753 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:39:07.754 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:39:07.754 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:39:07.754 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:39:07.754 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:39:07.754 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:39:07.754 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:39:07.754 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:39:07.754 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:39:07.754 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:39:07.754 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:39:07.755 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:39:07.755 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:39:07.755 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:39:07.799 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 18 2024-04-06T09:39:07.799 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 19 2024-04-06T09:39:07.799 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:39:07.799 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=77, tag_tid=12, entry_tid=1], [object_number=76, tag_tid=12, entry_tid=0], [object_number=79, tag_tid=11, entry_tid=17175], [object_number=78, tag_tid=11, entry_tid=17174]]], state=connected] 2024-04-06T09:39:07.799 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=75, tag_tid=11, entry_tid=14919], [object_number=74, tag_tid=11, entry_tid=14918], [object_number=73, tag_tid=11, entry_tid=14917], [object_number=72, tag_tid=11, entry_tid=14916]]], state=connected] 2024-04-06T09:39:07.802 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:39:07.804 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=75, tag_tid=11, entry_tid=14919' 2024-04-06T09:39:07.804 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=77, tag_tid=12, entry_tid=1' -a 'object_number=77, tag_tid=12, entry_tid=1' = 'object_number=75, tag_tid=11, entry_tid=14919' 2024-04-06T09:39:07.804 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=75, tag_tid=11, entry_tid=14919' '!=' 'object_number=74, tag_tid=11, entry_tid=14834' 2024-04-06T09:39:07.804 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:39:07.804 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=75, tag_tid=11, entry_tid=14919' = 'object_number=74, tag_tid=11, entry_tid=14834' 2024-04-06T09:39:07.804 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=75, tag_tid=11, entry_tid=14919' 2024-04-06T09:39:07.804 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=77, tag_tid=12, entry_tid=1' 2024-04-06T09:39:07.804 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:39:07.804 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:39:07.805 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=12 2024-04-06T09:39:07.805 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=75, tag_tid=11, entry_tid=14919' 2024-04-06T09:39:07.805 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:39:07.806 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:39:07.806 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=11 2024-04-06T09:39:07.807 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=77, tag_tid=12, entry_tid=1' 2024-04-06T09:39:07.807 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:39:07.807 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:39:07.808 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:39:07.809 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=75, tag_tid=11, entry_tid=14919' 2024-04-06T09:39:07.809 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:39:07.809 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:39:07.810 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=14919 2024-04-06T09:39:07.810 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 12 = 11 -a 1 -le 14919 2024-04-06T09:39:07.810 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:39:07.810 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:39:07.810 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:39:08.011 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:39:08.012 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:39:08.012 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:39:08.012 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:39:08.012 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:39:08.012 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:39:08.012 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:39:08.012 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:39:08.012 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:39:08.012 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:39:08.012 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:39:08.012 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:39:08.012 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:39:08.013 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:39:08.013 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:39:08.013 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:39:08.013 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:39:08.017 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:39:08.017 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:39:08.017 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:39:08.017 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:39:08.017 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:39:08.017 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:39:08.017 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:39:08.017 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:39:08.017 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:39:08.017 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:39:08.018 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:39:08.018 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:39:08.018 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:39:08.018 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:39:08.018 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:39:08.018 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:39:08.018 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:39:08.018 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:39:08.018 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:39:08.018 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:39:08.018 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:39:08.018 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:39:08.018 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:39:08.018 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:39:08.019 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:39:08.019 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:39:08.019 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:39:08.019 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:39:08.019 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:39:08.019 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:39:08.019 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:39:08.019 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:39:08.041 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:39:08.041 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:39:08.041 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:39:08.041 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:39:08.167 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:39:08.168 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:39:08.168 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:39:08.169 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:39:08.169 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:39:08.169 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:39:08.169 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:39:08.169 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:39:08.169 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:39:08.169 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:39:08.170 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:39:08.170 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:39:08.170 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:39:08.171 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:39:08.171 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:39:08.171 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:39:08.227 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 18 2024-04-06T09:39:08.227 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 19 2024-04-06T09:39:08.227 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:39:08.227 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=77, tag_tid=12, entry_tid=1], [object_number=76, tag_tid=12, entry_tid=0], [object_number=79, tag_tid=11, entry_tid=17175], [object_number=78, tag_tid=11, entry_tid=17174]]], state=connected] 2024-04-06T09:39:08.227 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=75, tag_tid=11, entry_tid=15007], [object_number=74, tag_tid=11, entry_tid=15006], [object_number=73, tag_tid=11, entry_tid=15005], [object_number=72, tag_tid=11, entry_tid=15004]]], state=connected] 2024-04-06T09:39:08.230 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:39:08.232 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=77, tag_tid=12, entry_tid=1' 2024-04-06T09:39:08.233 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:39:08.233 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:39:08.233 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:39:08.233 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:39:08.233 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:39:08.233 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:39:08.233 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:39:08.233 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:39:08.233 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:39:08.233 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:39:08.233 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:39:08.234 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:39:08.234 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:39:08.234 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:39:08.234 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:39:08.279 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 18 2024-04-06T09:39:08.279 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 19 2024-04-06T09:39:08.279 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:39:08.279 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=77, tag_tid=12, entry_tid=1], [object_number=76, tag_tid=12, entry_tid=0], [object_number=79, tag_tid=11, entry_tid=17175], [object_number=78, tag_tid=11, entry_tid=17174]]], state=connected] 2024-04-06T09:39:08.279 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=75, tag_tid=11, entry_tid=15007], [object_number=74, tag_tid=11, entry_tid=15006], [object_number=73, tag_tid=11, entry_tid=15005], [object_number=72, tag_tid=11, entry_tid=15004]]], state=connected] 2024-04-06T09:39:08.282 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:39:08.284 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=75, tag_tid=11, entry_tid=15007' 2024-04-06T09:39:08.285 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=77, tag_tid=12, entry_tid=1' -a 'object_number=77, tag_tid=12, entry_tid=1' = 'object_number=75, tag_tid=11, entry_tid=15007' 2024-04-06T09:39:08.285 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=75, tag_tid=11, entry_tid=15007' '!=' 'object_number=75, tag_tid=11, entry_tid=14919' 2024-04-06T09:39:08.285 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:39:08.285 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=75, tag_tid=11, entry_tid=15007' = 'object_number=75, tag_tid=11, entry_tid=14919' 2024-04-06T09:39:08.285 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=75, tag_tid=11, entry_tid=15007' 2024-04-06T09:39:08.285 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=77, tag_tid=12, entry_tid=1' 2024-04-06T09:39:08.285 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:39:08.285 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:39:08.285 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=12 2024-04-06T09:39:08.286 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=75, tag_tid=11, entry_tid=15007' 2024-04-06T09:39:08.286 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:39:08.286 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:39:08.287 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=11 2024-04-06T09:39:08.288 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=77, tag_tid=12, entry_tid=1' 2024-04-06T09:39:08.288 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:39:08.288 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:39:08.289 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:39:08.289 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=75, tag_tid=11, entry_tid=15007' 2024-04-06T09:39:08.289 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:39:08.289 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:39:08.290 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=15007 2024-04-06T09:39:08.291 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 12 = 11 -a 1 -le 15007 2024-04-06T09:39:08.291 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:39:08.291 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:39:08.291 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:39:08.493 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:39:08.493 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:39:08.493 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:39:08.493 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:39:08.493 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:39:08.493 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:39:08.493 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:39:08.493 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:39:08.493 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:39:08.493 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:39:08.493 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:39:08.494 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:39:08.494 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:39:08.494 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:39:08.494 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:39:08.494 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:39:08.494 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:39:08.497 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:39:08.498 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:39:08.498 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:39:08.498 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:39:08.498 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:39:08.498 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:39:08.498 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:39:08.498 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:39:08.498 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:39:08.498 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:39:08.498 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:39:08.498 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:39:08.498 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:39:08.498 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:39:08.498 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:39:08.499 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:39:08.499 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:39:08.499 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:39:08.499 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:39:08.499 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:39:08.499 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:39:08.499 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:39:08.499 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:39:08.499 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:39:08.499 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:39:08.499 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:39:08.499 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:39:08.499 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:39:08.499 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:39:08.499 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:39:08.500 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:39:08.500 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:39:08.522 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:39:08.522 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:39:08.522 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:39:08.522 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:39:08.650 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:39:08.651 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:39:08.651 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:39:08.651 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:39:08.651 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:39:08.651 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:39:08.651 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:39:08.651 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:39:08.651 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:39:08.651 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:39:08.652 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:39:08.652 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:39:08.652 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:39:08.654 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:39:08.654 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:39:08.654 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:39:08.706 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 18 2024-04-06T09:39:08.706 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 19 2024-04-06T09:39:08.706 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:39:08.706 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=77, tag_tid=12, entry_tid=1], [object_number=76, tag_tid=12, entry_tid=0], [object_number=79, tag_tid=11, entry_tid=17175], [object_number=78, tag_tid=11, entry_tid=17174]]], state=connected] 2024-04-06T09:39:08.706 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=72, tag_tid=11, entry_tid=15088], [object_number=75, tag_tid=11, entry_tid=15087], [object_number=74, tag_tid=11, entry_tid=15086], [object_number=73, tag_tid=11, entry_tid=15085]]], state=connected] 2024-04-06T09:39:08.709 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:39:08.711 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=77, tag_tid=12, entry_tid=1' 2024-04-06T09:39:08.712 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:39:08.712 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:39:08.712 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:39:08.712 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:39:08.712 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:39:08.712 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:39:08.712 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:39:08.712 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:39:08.712 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:39:08.712 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:39:08.712 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:39:08.712 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:39:08.713 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:39:08.713 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:39:08.713 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:39:08.759 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 18 2024-04-06T09:39:08.759 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 19 2024-04-06T09:39:08.759 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:39:08.759 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=77, tag_tid=12, entry_tid=1], [object_number=76, tag_tid=12, entry_tid=0], [object_number=79, tag_tid=11, entry_tid=17175], [object_number=78, tag_tid=11, entry_tid=17174]]], state=connected] 2024-04-06T09:39:08.759 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=72, tag_tid=11, entry_tid=15088], [object_number=75, tag_tid=11, entry_tid=15087], [object_number=74, tag_tid=11, entry_tid=15086], [object_number=73, tag_tid=11, entry_tid=15085]]], state=connected] 2024-04-06T09:39:08.763 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:39:08.764 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=72, tag_tid=11, entry_tid=15088' 2024-04-06T09:39:08.764 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=77, tag_tid=12, entry_tid=1' -a 'object_number=77, tag_tid=12, entry_tid=1' = 'object_number=72, tag_tid=11, entry_tid=15088' 2024-04-06T09:39:08.764 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=72, tag_tid=11, entry_tid=15088' '!=' 'object_number=75, tag_tid=11, entry_tid=15007' 2024-04-06T09:39:08.765 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:39:08.765 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=72, tag_tid=11, entry_tid=15088' = 'object_number=75, tag_tid=11, entry_tid=15007' 2024-04-06T09:39:08.765 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=72, tag_tid=11, entry_tid=15088' 2024-04-06T09:39:08.765 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=77, tag_tid=12, entry_tid=1' 2024-04-06T09:39:08.765 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:39:08.765 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:39:08.766 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=12 2024-04-06T09:39:08.767 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=72, tag_tid=11, entry_tid=15088' 2024-04-06T09:39:08.767 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:39:08.767 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:39:08.768 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=11 2024-04-06T09:39:08.768 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=77, tag_tid=12, entry_tid=1' 2024-04-06T09:39:08.768 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:39:08.768 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:39:08.769 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:39:08.770 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=72, tag_tid=11, entry_tid=15088' 2024-04-06T09:39:08.770 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:39:08.770 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:39:08.771 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=15088 2024-04-06T09:39:08.771 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 12 = 11 -a 1 -le 15088 2024-04-06T09:39:08.771 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:39:08.771 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:39:08.772 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:39:08.973 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:39:08.973 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:39:08.973 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:39:08.973 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:39:08.973 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:39:08.973 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:39:08.973 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:39:08.974 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:39:08.974 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:39:08.974 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:39:08.974 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:39:08.974 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:39:08.974 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:39:08.974 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:39:08.974 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:39:08.974 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:39:08.975 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:39:08.979 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:39:08.979 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:39:08.979 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:39:08.979 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:39:08.979 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:39:08.979 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:39:08.979 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:39:08.979 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:39:08.979 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:39:08.979 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:39:08.979 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:39:08.979 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:39:08.980 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:39:08.980 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:39:08.980 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:39:08.980 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:39:08.980 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:39:08.980 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:39:08.980 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:39:08.980 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:39:08.980 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:39:08.980 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:39:08.980 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:39:08.980 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:39:08.980 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:39:08.980 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:39:08.980 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:39:08.980 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:39:08.981 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:39:08.981 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:39:08.981 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:39:08.981 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:39:09.005 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:39:09.005 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:39:09.005 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:39:09.005 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:39:09.140 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:39:09.141 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:39:09.141 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:39:09.141 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:39:09.141 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:39:09.141 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:39:09.141 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:39:09.141 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:39:09.142 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:39:09.142 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:39:09.142 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:39:09.142 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:39:09.142 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:39:09.144 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:39:09.145 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:39:09.145 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:39:09.195 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 18 2024-04-06T09:39:09.196 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 19 2024-04-06T09:39:09.196 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:39:09.196 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=77, tag_tid=12, entry_tid=1], [object_number=76, tag_tid=12, entry_tid=0], [object_number=79, tag_tid=11, entry_tid=17175], [object_number=78, tag_tid=11, entry_tid=17174]]], state=connected] 2024-04-06T09:39:09.196 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=73, tag_tid=11, entry_tid=15173], [object_number=72, tag_tid=11, entry_tid=15172], [object_number=75, tag_tid=11, entry_tid=15171], [object_number=74, tag_tid=11, entry_tid=15170]]], state=connected] 2024-04-06T09:39:09.199 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:39:09.201 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=77, tag_tid=12, entry_tid=1' 2024-04-06T09:39:09.202 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:39:09.202 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:39:09.202 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:39:09.202 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:39:09.202 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:39:09.202 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:39:09.202 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:39:09.202 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:39:09.203 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:39:09.203 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:39:09.203 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:39:09.203 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:39:09.203 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:39:09.203 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:39:09.204 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:39:09.250 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 18 2024-04-06T09:39:09.250 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 19 2024-04-06T09:39:09.250 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:39:09.250 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=77, tag_tid=12, entry_tid=1], [object_number=76, tag_tid=12, entry_tid=0], [object_number=79, tag_tid=11, entry_tid=17175], [object_number=78, tag_tid=11, entry_tid=17174]]], state=connected] 2024-04-06T09:39:09.250 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=73, tag_tid=11, entry_tid=15173], [object_number=72, tag_tid=11, entry_tid=15172], [object_number=75, tag_tid=11, entry_tid=15171], [object_number=74, tag_tid=11, entry_tid=15170]]], state=connected] 2024-04-06T09:39:09.253 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:39:09.256 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=73, tag_tid=11, entry_tid=15173' 2024-04-06T09:39:09.256 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=77, tag_tid=12, entry_tid=1' -a 'object_number=77, tag_tid=12, entry_tid=1' = 'object_number=73, tag_tid=11, entry_tid=15173' 2024-04-06T09:39:09.256 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=73, tag_tid=11, entry_tid=15173' '!=' 'object_number=72, tag_tid=11, entry_tid=15088' 2024-04-06T09:39:09.256 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:39:09.256 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=73, tag_tid=11, entry_tid=15173' = 'object_number=72, tag_tid=11, entry_tid=15088' 2024-04-06T09:39:09.256 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=73, tag_tid=11, entry_tid=15173' 2024-04-06T09:39:09.256 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=77, tag_tid=12, entry_tid=1' 2024-04-06T09:39:09.256 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:39:09.256 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:39:09.257 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=12 2024-04-06T09:39:09.257 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=73, tag_tid=11, entry_tid=15173' 2024-04-06T09:39:09.258 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:39:09.258 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:39:09.258 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=11 2024-04-06T09:39:09.259 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=77, tag_tid=12, entry_tid=1' 2024-04-06T09:39:09.259 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:39:09.259 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:39:09.260 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:39:09.261 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=73, tag_tid=11, entry_tid=15173' 2024-04-06T09:39:09.261 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:39:09.261 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:39:09.262 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=15173 2024-04-06T09:39:09.262 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 12 = 11 -a 1 -le 15173 2024-04-06T09:39:09.262 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:39:09.263 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:39:09.263 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:39:09.464 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:39:09.464 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:39:09.464 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:39:09.464 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:39:09.464 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:39:09.464 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:39:09.464 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:39:09.465 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:39:09.465 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:39:09.465 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:39:09.465 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:39:09.465 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:39:09.465 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:39:09.465 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:39:09.465 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:39:09.465 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:39:09.466 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:39:09.468 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:39:09.468 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:39:09.468 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:39:09.469 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:39:09.469 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:39:09.469 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:39:09.469 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:39:09.469 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:39:09.469 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:39:09.469 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:39:09.469 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:39:09.469 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:39:09.469 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:39:09.469 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:39:09.469 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:39:09.469 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:39:09.470 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:39:09.471 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:39:09.471 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:39:09.471 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:39:09.471 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:39:09.471 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:39:09.471 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:39:09.471 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:39:09.471 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:39:09.471 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:39:09.471 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:39:09.471 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:39:09.471 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:39:09.471 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:39:09.471 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:39:09.471 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:39:09.495 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:39:09.495 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:39:09.495 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:39:09.495 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:39:09.625 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:39:09.627 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:39:09.627 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:39:09.627 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:39:09.627 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:39:09.627 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:39:09.627 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:39:09.627 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:39:09.627 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:39:09.627 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:39:09.627 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:39:09.628 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:39:09.628 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:39:09.629 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:39:09.629 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:39:09.629 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:39:09.675 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 18 2024-04-06T09:39:09.675 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 19 2024-04-06T09:39:09.675 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:39:09.675 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=77, tag_tid=12, entry_tid=1], [object_number=76, tag_tid=12, entry_tid=0], [object_number=79, tag_tid=11, entry_tid=17175], [object_number=78, tag_tid=11, entry_tid=17174]]], state=connected] 2024-04-06T09:39:09.675 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=74, tag_tid=11, entry_tid=15262], [object_number=73, tag_tid=11, entry_tid=15261], [object_number=72, tag_tid=11, entry_tid=15260], [object_number=75, tag_tid=11, entry_tid=15259]]], state=connected] 2024-04-06T09:39:09.678 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:39:09.680 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=77, tag_tid=12, entry_tid=1' 2024-04-06T09:39:09.680 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:39:09.681 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:39:09.681 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:39:09.681 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:39:09.681 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:39:09.681 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:39:09.681 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:39:09.681 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:39:09.681 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:39:09.681 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:39:09.681 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:39:09.681 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:39:09.682 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:39:09.682 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:39:09.682 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:39:09.728 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 18 2024-04-06T09:39:09.728 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 19 2024-04-06T09:39:09.729 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:39:09.729 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=77, tag_tid=12, entry_tid=1], [object_number=76, tag_tid=12, entry_tid=0], [object_number=79, tag_tid=11, entry_tid=17175], [object_number=78, tag_tid=11, entry_tid=17174]]], state=connected] 2024-04-06T09:39:09.729 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=74, tag_tid=11, entry_tid=15262], [object_number=73, tag_tid=11, entry_tid=15261], [object_number=72, tag_tid=11, entry_tid=15260], [object_number=75, tag_tid=11, entry_tid=15259]]], state=connected] 2024-04-06T09:39:09.732 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:39:09.734 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=74, tag_tid=11, entry_tid=15262' 2024-04-06T09:39:09.734 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=77, tag_tid=12, entry_tid=1' -a 'object_number=77, tag_tid=12, entry_tid=1' = 'object_number=74, tag_tid=11, entry_tid=15262' 2024-04-06T09:39:09.734 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=74, tag_tid=11, entry_tid=15262' '!=' 'object_number=73, tag_tid=11, entry_tid=15173' 2024-04-06T09:39:09.734 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:39:09.734 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=74, tag_tid=11, entry_tid=15262' = 'object_number=73, tag_tid=11, entry_tid=15173' 2024-04-06T09:39:09.734 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=74, tag_tid=11, entry_tid=15262' 2024-04-06T09:39:09.734 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=77, tag_tid=12, entry_tid=1' 2024-04-06T09:39:09.735 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:39:09.735 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:39:09.735 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=12 2024-04-06T09:39:09.736 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=74, tag_tid=11, entry_tid=15262' 2024-04-06T09:39:09.736 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:39:09.736 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:39:09.737 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=11 2024-04-06T09:39:09.737 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=77, tag_tid=12, entry_tid=1' 2024-04-06T09:39:09.738 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:39:09.738 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:39:09.738 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:39:09.739 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=74, tag_tid=11, entry_tid=15262' 2024-04-06T09:39:09.739 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:39:09.739 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:39:09.740 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=15262 2024-04-06T09:39:09.741 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 12 = 11 -a 1 -le 15262 2024-04-06T09:39:09.741 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:39:09.741 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:39:09.741 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:39:09.942 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:39:09.942 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:39:09.943 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:39:09.943 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:39:09.943 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:39:09.943 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:39:09.943 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:39:09.943 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:39:09.943 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:39:09.943 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:39:09.943 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:39:09.943 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:39:09.943 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:39:09.943 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:39:09.943 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:39:09.944 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:39:09.944 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:39:09.948 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:39:09.948 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:39:09.948 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:39:09.948 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:39:09.948 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:39:09.948 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:39:09.948 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:39:09.948 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:39:09.948 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:39:09.949 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:39:09.949 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:39:09.949 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:39:09.949 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:39:09.949 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:39:09.949 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:39:09.949 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:39:09.949 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:39:09.949 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:39:09.949 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:39:09.949 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:39:09.949 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:39:09.949 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:39:09.949 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:39:09.949 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:39:09.949 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:39:09.950 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:39:09.950 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:39:09.950 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:39:09.950 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:39:09.950 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:39:09.950 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:39:09.950 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:39:09.974 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:39:09.974 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:39:09.974 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:39:09.974 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:39:10.106 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:39:10.107 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:39:10.107 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:39:10.108 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:39:10.108 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:39:10.108 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:39:10.108 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:39:10.108 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:39:10.108 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:39:10.108 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:39:10.108 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:39:10.108 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:39:10.109 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:39:10.110 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:39:10.110 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:39:10.111 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:39:10.167 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 18 2024-04-06T09:39:10.167 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 19 2024-04-06T09:39:10.167 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:39:10.167 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=77, tag_tid=12, entry_tid=1], [object_number=76, tag_tid=12, entry_tid=0], [object_number=79, tag_tid=11, entry_tid=17175], [object_number=78, tag_tid=11, entry_tid=17174]]], state=connected] 2024-04-06T09:39:10.167 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=73, tag_tid=11, entry_tid=15357], [object_number=72, tag_tid=11, entry_tid=15356], [object_number=75, tag_tid=11, entry_tid=15355], [object_number=74, tag_tid=11, entry_tid=15354]]], state=connected] 2024-04-06T09:39:10.170 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:39:10.172 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=77, tag_tid=12, entry_tid=1' 2024-04-06T09:39:10.173 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:39:10.173 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:39:10.173 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:39:10.173 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:39:10.174 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:39:10.174 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:39:10.174 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:39:10.174 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:39:10.174 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:39:10.174 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:39:10.174 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:39:10.174 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:39:10.175 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:39:10.175 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:39:10.175 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:39:10.222 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 18 2024-04-06T09:39:10.222 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 19 2024-04-06T09:39:10.222 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:39:10.222 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=77, tag_tid=12, entry_tid=1], [object_number=76, tag_tid=12, entry_tid=0], [object_number=79, tag_tid=11, entry_tid=17175], [object_number=78, tag_tid=11, entry_tid=17174]]], state=connected] 2024-04-06T09:39:10.222 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=73, tag_tid=11, entry_tid=15357], [object_number=72, tag_tid=11, entry_tid=15356], [object_number=75, tag_tid=11, entry_tid=15355], [object_number=74, tag_tid=11, entry_tid=15354]]], state=connected] 2024-04-06T09:39:10.225 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:39:10.227 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=73, tag_tid=11, entry_tid=15357' 2024-04-06T09:39:10.228 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=77, tag_tid=12, entry_tid=1' -a 'object_number=77, tag_tid=12, entry_tid=1' = 'object_number=73, tag_tid=11, entry_tid=15357' 2024-04-06T09:39:10.228 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=73, tag_tid=11, entry_tid=15357' '!=' 'object_number=74, tag_tid=11, entry_tid=15262' 2024-04-06T09:39:10.228 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:39:10.228 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=73, tag_tid=11, entry_tid=15357' = 'object_number=74, tag_tid=11, entry_tid=15262' 2024-04-06T09:39:10.228 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=73, tag_tid=11, entry_tid=15357' 2024-04-06T09:39:10.228 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=77, tag_tid=12, entry_tid=1' 2024-04-06T09:39:10.228 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:39:10.228 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:39:10.229 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=12 2024-04-06T09:39:10.229 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=73, tag_tid=11, entry_tid=15357' 2024-04-06T09:39:10.229 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:39:10.230 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:39:10.230 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=11 2024-04-06T09:39:10.231 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=77, tag_tid=12, entry_tid=1' 2024-04-06T09:39:10.231 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:39:10.231 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:39:10.232 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:39:10.233 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=73, tag_tid=11, entry_tid=15357' 2024-04-06T09:39:10.233 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:39:10.233 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:39:10.234 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=15357 2024-04-06T09:39:10.234 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 12 = 11 -a 1 -le 15357 2024-04-06T09:39:10.234 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:39:10.234 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:39:10.234 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:39:10.436 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:39:10.437 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:39:10.437 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:39:10.437 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:39:10.437 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:39:10.437 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:39:10.437 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:39:10.437 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:39:10.437 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:39:10.437 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:39:10.437 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:39:10.437 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:39:10.437 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:39:10.437 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:39:10.437 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:39:10.438 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:39:10.438 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:39:10.440 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:39:10.440 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:39:10.440 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:39:10.441 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:39:10.441 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:39:10.441 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:39:10.441 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:39:10.441 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:39:10.441 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:39:10.441 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:39:10.441 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:39:10.441 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:39:10.441 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:39:10.441 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:39:10.441 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:39:10.441 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:39:10.442 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:39:10.442 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:39:10.442 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:39:10.442 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:39:10.442 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:39:10.442 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:39:10.442 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:39:10.442 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:39:10.442 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:39:10.442 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:39:10.442 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:39:10.442 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:39:10.442 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:39:10.443 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:39:10.443 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:39:10.443 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:39:10.467 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:39:10.467 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:39:10.467 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:39:10.467 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:39:10.597 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:39:10.598 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:39:10.598 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:39:10.598 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:39:10.599 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:39:10.599 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:39:10.599 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:39:10.599 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:39:10.599 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:39:10.599 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:39:10.599 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:39:10.599 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:39:10.600 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:39:10.601 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:39:10.601 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:39:10.601 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:39:10.653 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 18 2024-04-06T09:39:10.653 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 19 2024-04-06T09:39:10.653 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:39:10.653 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=77, tag_tid=12, entry_tid=1], [object_number=76, tag_tid=12, entry_tid=0], [object_number=79, tag_tid=11, entry_tid=17175], [object_number=78, tag_tid=11, entry_tid=17174]]], state=connected] 2024-04-06T09:39:10.654 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=74, tag_tid=11, entry_tid=15470], [object_number=73, tag_tid=11, entry_tid=15469], [object_number=72, tag_tid=11, entry_tid=15468], [object_number=75, tag_tid=11, entry_tid=15467]]], state=connected] 2024-04-06T09:39:10.657 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:39:10.658 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=77, tag_tid=12, entry_tid=1' 2024-04-06T09:39:10.659 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:39:10.659 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:39:10.659 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:39:10.660 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:39:10.660 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:39:10.660 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:39:10.660 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:39:10.660 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:39:10.660 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:39:10.660 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:39:10.660 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:39:10.660 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:39:10.661 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:39:10.661 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:39:10.661 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:39:10.706 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 18 2024-04-06T09:39:10.706 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 19 2024-04-06T09:39:10.706 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:39:10.706 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=77, tag_tid=12, entry_tid=1], [object_number=76, tag_tid=12, entry_tid=0], [object_number=79, tag_tid=11, entry_tid=17175], [object_number=78, tag_tid=11, entry_tid=17174]]], state=connected] 2024-04-06T09:39:10.707 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=74, tag_tid=11, entry_tid=15470], [object_number=73, tag_tid=11, entry_tid=15469], [object_number=72, tag_tid=11, entry_tid=15468], [object_number=75, tag_tid=11, entry_tid=15467]]], state=connected] 2024-04-06T09:39:10.709 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:39:10.711 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=74, tag_tid=11, entry_tid=15470' 2024-04-06T09:39:10.711 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=77, tag_tid=12, entry_tid=1' -a 'object_number=77, tag_tid=12, entry_tid=1' = 'object_number=74, tag_tid=11, entry_tid=15470' 2024-04-06T09:39:10.711 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=74, tag_tid=11, entry_tid=15470' '!=' 'object_number=73, tag_tid=11, entry_tid=15357' 2024-04-06T09:39:10.711 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:39:10.712 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=74, tag_tid=11, entry_tid=15470' = 'object_number=73, tag_tid=11, entry_tid=15357' 2024-04-06T09:39:10.712 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=74, tag_tid=11, entry_tid=15470' 2024-04-06T09:39:10.712 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=77, tag_tid=12, entry_tid=1' 2024-04-06T09:39:10.712 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:39:10.712 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:39:10.713 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=12 2024-04-06T09:39:10.713 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=74, tag_tid=11, entry_tid=15470' 2024-04-06T09:39:10.714 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:39:10.714 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:39:10.715 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=11 2024-04-06T09:39:10.715 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=77, tag_tid=12, entry_tid=1' 2024-04-06T09:39:10.715 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:39:10.716 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:39:10.716 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:39:10.717 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=74, tag_tid=11, entry_tid=15470' 2024-04-06T09:39:10.717 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:39:10.717 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:39:10.718 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=15470 2024-04-06T09:39:10.718 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 12 = 11 -a 1 -le 15470 2024-04-06T09:39:10.718 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:39:10.718 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:39:10.719 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:39:10.920 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:39:10.921 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:39:10.921 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:39:10.921 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:39:10.921 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:39:10.921 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:39:10.921 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:39:10.921 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:39:10.921 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:39:10.921 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:39:10.921 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:39:10.921 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:39:10.921 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:39:10.921 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:39:10.922 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:39:10.922 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:39:10.922 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:39:10.925 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:39:10.925 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:39:10.925 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:39:10.925 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:39:10.925 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:39:10.925 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:39:10.925 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:39:10.925 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:39:10.925 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:39:10.926 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:39:10.926 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:39:10.926 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:39:10.926 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:39:10.926 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:39:10.926 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:39:10.926 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:39:10.926 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:39:10.926 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:39:10.926 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:39:10.926 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:39:10.926 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:39:10.926 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:39:10.926 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:39:10.927 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:39:10.927 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:39:10.927 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:39:10.927 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:39:10.927 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:39:10.927 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:39:10.927 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:39:10.927 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:39:10.927 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:39:10.950 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:39:10.951 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:39:10.951 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:39:10.951 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:39:11.081 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:39:11.083 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:39:11.083 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:39:11.083 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:39:11.083 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:39:11.083 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:39:11.083 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:39:11.083 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:39:11.083 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:39:11.083 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:39:11.083 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:39:11.084 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:39:11.084 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:39:11.085 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:39:11.085 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:39:11.086 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:39:11.131 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 18 2024-04-06T09:39:11.131 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 19 2024-04-06T09:39:11.131 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:39:11.131 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=77, tag_tid=12, entry_tid=1], [object_number=76, tag_tid=12, entry_tid=0], [object_number=79, tag_tid=11, entry_tid=17175], [object_number=78, tag_tid=11, entry_tid=17174]]], state=connected] 2024-04-06T09:39:11.131 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=75, tag_tid=11, entry_tid=15591], [object_number=74, tag_tid=11, entry_tid=15590], [object_number=73, tag_tid=11, entry_tid=15589], [object_number=72, tag_tid=11, entry_tid=15588]]], state=connected] 2024-04-06T09:39:11.135 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:39:11.137 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=77, tag_tid=12, entry_tid=1' 2024-04-06T09:39:11.137 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:39:11.137 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:39:11.138 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:39:11.138 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:39:11.138 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:39:11.138 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:39:11.138 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:39:11.138 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:39:11.138 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:39:11.138 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:39:11.138 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:39:11.138 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:39:11.139 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:39:11.139 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:39:11.139 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:39:11.184 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 18 2024-04-06T09:39:11.184 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 19 2024-04-06T09:39:11.184 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:39:11.184 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=77, tag_tid=12, entry_tid=1], [object_number=76, tag_tid=12, entry_tid=0], [object_number=79, tag_tid=11, entry_tid=17175], [object_number=78, tag_tid=11, entry_tid=17174]]], state=connected] 2024-04-06T09:39:11.184 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=75, tag_tid=11, entry_tid=15591], [object_number=74, tag_tid=11, entry_tid=15590], [object_number=73, tag_tid=11, entry_tid=15589], [object_number=72, tag_tid=11, entry_tid=15588]]], state=connected] 2024-04-06T09:39:11.187 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:39:11.190 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=75, tag_tid=11, entry_tid=15591' 2024-04-06T09:39:11.190 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=77, tag_tid=12, entry_tid=1' -a 'object_number=77, tag_tid=12, entry_tid=1' = 'object_number=75, tag_tid=11, entry_tid=15591' 2024-04-06T09:39:11.190 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=75, tag_tid=11, entry_tid=15591' '!=' 'object_number=74, tag_tid=11, entry_tid=15470' 2024-04-06T09:39:11.190 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:39:11.190 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=75, tag_tid=11, entry_tid=15591' = 'object_number=74, tag_tid=11, entry_tid=15470' 2024-04-06T09:39:11.190 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=75, tag_tid=11, entry_tid=15591' 2024-04-06T09:39:11.190 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=77, tag_tid=12, entry_tid=1' 2024-04-06T09:39:11.190 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:39:11.191 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:39:11.191 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=12 2024-04-06T09:39:11.191 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=75, tag_tid=11, entry_tid=15591' 2024-04-06T09:39:11.191 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:39:11.192 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:39:11.192 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=11 2024-04-06T09:39:11.193 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=77, tag_tid=12, entry_tid=1' 2024-04-06T09:39:11.193 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:39:11.193 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:39:11.194 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:39:11.195 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=75, tag_tid=11, entry_tid=15591' 2024-04-06T09:39:11.195 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:39:11.195 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:39:11.196 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=15591 2024-04-06T09:39:11.196 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 12 = 11 -a 1 -le 15591 2024-04-06T09:39:11.196 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:39:11.196 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:39:11.196 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:39:11.397 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:39:11.397 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:39:11.398 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:39:11.398 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:39:11.398 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:39:11.398 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:39:11.398 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:39:11.398 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:39:11.398 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:39:11.398 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:39:11.398 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:39:11.398 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:39:11.398 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:39:11.398 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:39:11.398 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:39:11.399 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:39:11.399 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:39:11.401 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:39:11.402 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:39:11.402 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:39:11.402 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:39:11.402 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:39:11.402 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:39:11.402 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:39:11.402 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:39:11.402 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:39:11.402 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:39:11.402 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:39:11.402 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:39:11.402 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:39:11.402 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:39:11.403 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:39:11.403 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:39:11.403 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:39:11.403 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:39:11.403 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:39:11.403 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:39:11.403 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:39:11.403 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:39:11.403 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:39:11.403 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:39:11.403 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:39:11.403 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:39:11.403 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:39:11.403 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:39:11.403 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:39:11.404 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:39:11.404 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:39:11.404 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:39:11.429 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:39:11.429 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:39:11.430 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:39:11.430 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:39:11.559 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:39:11.560 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:39:11.560 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:39:11.560 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:39:11.560 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:39:11.560 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:39:11.560 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:39:11.560 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:39:11.560 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:39:11.560 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:39:11.561 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:39:11.561 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:39:11.561 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:39:11.563 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:39:11.564 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:39:11.564 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:39:11.618 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 18 2024-04-06T09:39:11.618 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 19 2024-04-06T09:39:11.618 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:39:11.618 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=77, tag_tid=12, entry_tid=1], [object_number=76, tag_tid=12, entry_tid=0], [object_number=79, tag_tid=11, entry_tid=17175], [object_number=78, tag_tid=11, entry_tid=17174]]], state=connected] 2024-04-06T09:39:11.619 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=75, tag_tid=11, entry_tid=15699], [object_number=74, tag_tid=11, entry_tid=15698], [object_number=73, tag_tid=11, entry_tid=15697], [object_number=72, tag_tid=11, entry_tid=15696]]], state=connected] 2024-04-06T09:39:11.622 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:39:11.623 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=77, tag_tid=12, entry_tid=1' 2024-04-06T09:39:11.624 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:39:11.624 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:39:11.624 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:39:11.625 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:39:11.625 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:39:11.625 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:39:11.625 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:39:11.625 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:39:11.625 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:39:11.625 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:39:11.625 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:39:11.625 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:39:11.626 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:39:11.626 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:39:11.626 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:39:11.672 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 18 2024-04-06T09:39:11.672 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 19 2024-04-06T09:39:11.672 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:39:11.672 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=77, tag_tid=12, entry_tid=1], [object_number=76, tag_tid=12, entry_tid=0], [object_number=79, tag_tid=11, entry_tid=17175], [object_number=78, tag_tid=11, entry_tid=17174]]], state=connected] 2024-04-06T09:39:11.672 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=75, tag_tid=11, entry_tid=15699], [object_number=74, tag_tid=11, entry_tid=15698], [object_number=73, tag_tid=11, entry_tid=15697], [object_number=72, tag_tid=11, entry_tid=15696]]], state=connected] 2024-04-06T09:39:11.675 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:39:11.677 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=75, tag_tid=11, entry_tid=15699' 2024-04-06T09:39:11.677 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=77, tag_tid=12, entry_tid=1' -a 'object_number=77, tag_tid=12, entry_tid=1' = 'object_number=75, tag_tid=11, entry_tid=15699' 2024-04-06T09:39:11.677 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=75, tag_tid=11, entry_tid=15699' '!=' 'object_number=75, tag_tid=11, entry_tid=15591' 2024-04-06T09:39:11.677 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:39:11.677 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=75, tag_tid=11, entry_tid=15699' = 'object_number=75, tag_tid=11, entry_tid=15591' 2024-04-06T09:39:11.677 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=75, tag_tid=11, entry_tid=15699' 2024-04-06T09:39:11.678 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=77, tag_tid=12, entry_tid=1' 2024-04-06T09:39:11.678 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:39:11.678 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:39:11.678 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=12 2024-04-06T09:39:11.679 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=75, tag_tid=11, entry_tid=15699' 2024-04-06T09:39:11.679 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:39:11.679 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:39:11.680 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=11 2024-04-06T09:39:11.681 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=77, tag_tid=12, entry_tid=1' 2024-04-06T09:39:11.681 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:39:11.681 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:39:11.682 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:39:11.683 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=75, tag_tid=11, entry_tid=15699' 2024-04-06T09:39:11.683 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:39:11.683 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:39:11.684 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=15699 2024-04-06T09:39:11.684 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 12 = 11 -a 1 -le 15699 2024-04-06T09:39:11.684 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:39:11.684 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:39:11.684 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:39:11.886 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:39:11.886 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:39:11.886 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:39:11.886 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:39:11.886 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:39:11.887 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:39:11.887 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:39:11.887 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:39:11.887 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:39:11.887 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:39:11.887 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:39:11.887 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:39:11.887 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:39:11.887 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:39:11.887 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:39:11.888 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:39:11.888 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:39:11.891 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:39:11.891 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:39:11.891 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:39:11.892 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:39:11.892 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:39:11.892 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:39:11.892 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:39:11.892 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:39:11.892 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:39:11.892 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:39:11.892 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:39:11.892 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:39:11.892 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:39:11.892 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:39:11.892 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:39:11.893 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:39:11.893 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:39:11.893 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:39:11.893 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:39:11.893 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:39:11.893 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:39:11.893 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:39:11.893 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:39:11.893 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:39:11.893 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:39:11.893 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:39:11.893 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:39:11.893 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:39:11.893 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:39:11.893 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:39:11.894 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:39:11.894 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:39:11.917 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:39:11.917 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:39:11.917 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:39:11.918 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:39:12.045 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:39:12.046 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:39:12.047 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:39:12.047 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:39:12.047 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:39:12.047 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:39:12.047 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:39:12.047 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:39:12.047 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:39:12.047 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:39:12.047 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:39:12.048 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:39:12.048 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:39:12.048 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:39:12.049 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:39:12.049 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:39:12.102 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 18 2024-04-06T09:39:12.102 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 19 2024-04-06T09:39:12.102 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:39:12.102 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=77, tag_tid=12, entry_tid=1], [object_number=76, tag_tid=12, entry_tid=0], [object_number=79, tag_tid=11, entry_tid=17175], [object_number=78, tag_tid=11, entry_tid=17174]]], state=connected] 2024-04-06T09:39:12.102 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=75, tag_tid=11, entry_tid=15803], [object_number=74, tag_tid=11, entry_tid=15802], [object_number=73, tag_tid=11, entry_tid=15801], [object_number=72, tag_tid=11, entry_tid=15800]]], state=connected] 2024-04-06T09:39:12.106 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:39:12.107 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=77, tag_tid=12, entry_tid=1' 2024-04-06T09:39:12.108 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:39:12.108 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:39:12.108 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:39:12.108 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:39:12.109 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:39:12.109 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:39:12.109 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:39:12.109 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:39:12.109 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:39:12.109 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:39:12.109 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:39:12.109 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:39:12.110 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:39:12.110 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:39:12.110 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:39:12.156 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 18 2024-04-06T09:39:12.156 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 19 2024-04-06T09:39:12.156 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:39:12.156 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=77, tag_tid=12, entry_tid=1], [object_number=76, tag_tid=12, entry_tid=0], [object_number=79, tag_tid=11, entry_tid=17175], [object_number=78, tag_tid=11, entry_tid=17174]]], state=connected] 2024-04-06T09:39:12.156 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=75, tag_tid=11, entry_tid=15803], [object_number=74, tag_tid=11, entry_tid=15802], [object_number=73, tag_tid=11, entry_tid=15801], [object_number=72, tag_tid=11, entry_tid=15800]]], state=connected] 2024-04-06T09:39:12.159 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:39:12.162 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=75, tag_tid=11, entry_tid=15803' 2024-04-06T09:39:12.162 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=77, tag_tid=12, entry_tid=1' -a 'object_number=77, tag_tid=12, entry_tid=1' = 'object_number=75, tag_tid=11, entry_tid=15803' 2024-04-06T09:39:12.162 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=75, tag_tid=11, entry_tid=15803' '!=' 'object_number=75, tag_tid=11, entry_tid=15699' 2024-04-06T09:39:12.162 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:39:12.162 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=75, tag_tid=11, entry_tid=15803' = 'object_number=75, tag_tid=11, entry_tid=15699' 2024-04-06T09:39:12.162 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=75, tag_tid=11, entry_tid=15803' 2024-04-06T09:39:12.162 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=77, tag_tid=12, entry_tid=1' 2024-04-06T09:39:12.162 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:39:12.162 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:39:12.163 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=12 2024-04-06T09:39:12.163 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=75, tag_tid=11, entry_tid=15803' 2024-04-06T09:39:12.164 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:39:12.164 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:39:12.164 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=11 2024-04-06T09:39:12.165 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=77, tag_tid=12, entry_tid=1' 2024-04-06T09:39:12.165 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:39:12.165 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:39:12.166 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:39:12.167 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=75, tag_tid=11, entry_tid=15803' 2024-04-06T09:39:12.167 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:39:12.167 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:39:12.168 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=15803 2024-04-06T09:39:12.168 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 12 = 11 -a 1 -le 15803 2024-04-06T09:39:12.168 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:39:12.168 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:39:12.168 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:39:12.370 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:39:12.370 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:39:12.370 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:39:12.371 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:39:12.371 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:39:12.371 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:39:12.371 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:39:12.371 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:39:12.372 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:39:12.372 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:39:12.372 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:39:12.372 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:39:12.372 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:39:12.372 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:39:12.372 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:39:12.373 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:39:12.373 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:39:12.374 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:39:12.374 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:39:12.374 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:39:12.374 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:39:12.374 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:39:12.374 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:39:12.374 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:39:12.375 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:39:12.375 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:39:12.375 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:39:12.375 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:39:12.375 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:39:12.375 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:39:12.375 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:39:12.375 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:39:12.375 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:39:12.376 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:39:12.377 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:39:12.377 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:39:12.377 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:39:12.377 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:39:12.377 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:39:12.377 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:39:12.377 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:39:12.377 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:39:12.377 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:39:12.377 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:39:12.377 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:39:12.377 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:39:12.377 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:39:12.378 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:39:12.378 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:39:12.402 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:39:12.402 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:39:12.402 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:39:12.402 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:39:12.529 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:39:12.530 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:39:12.530 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:39:12.530 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:39:12.531 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:39:12.531 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:39:12.531 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:39:12.531 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:39:12.531 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:39:12.531 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:39:12.531 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:39:12.532 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:39:12.532 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:39:12.533 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:39:12.533 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:39:12.533 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:39:12.578 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 18 2024-04-06T09:39:12.578 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 19 2024-04-06T09:39:12.578 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:39:12.578 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=77, tag_tid=12, entry_tid=1], [object_number=76, tag_tid=12, entry_tid=0], [object_number=79, tag_tid=11, entry_tid=17175], [object_number=78, tag_tid=11, entry_tid=17174]]], state=connected] 2024-04-06T09:39:12.578 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=73, tag_tid=11, entry_tid=15889], [object_number=72, tag_tid=11, entry_tid=15888], [object_number=75, tag_tid=11, entry_tid=15887], [object_number=74, tag_tid=11, entry_tid=15886]]], state=connected] 2024-04-06T09:39:12.582 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:39:12.583 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=77, tag_tid=12, entry_tid=1' 2024-04-06T09:39:12.584 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:39:12.584 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:39:12.584 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:39:12.584 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:39:12.585 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:39:12.585 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:39:12.585 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:39:12.585 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:39:12.585 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:39:12.585 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:39:12.585 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:39:12.585 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:39:12.586 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:39:12.586 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:39:12.586 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:39:12.632 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 18 2024-04-06T09:39:12.632 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 19 2024-04-06T09:39:12.632 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:39:12.632 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=77, tag_tid=12, entry_tid=1], [object_number=76, tag_tid=12, entry_tid=0], [object_number=79, tag_tid=11, entry_tid=17175], [object_number=78, tag_tid=11, entry_tid=17174]]], state=connected] 2024-04-06T09:39:12.633 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=73, tag_tid=11, entry_tid=15889], [object_number=72, tag_tid=11, entry_tid=15888], [object_number=75, tag_tid=11, entry_tid=15887], [object_number=74, tag_tid=11, entry_tid=15886]]], state=connected] 2024-04-06T09:39:12.636 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:39:12.637 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=73, tag_tid=11, entry_tid=15889' 2024-04-06T09:39:12.637 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=77, tag_tid=12, entry_tid=1' -a 'object_number=77, tag_tid=12, entry_tid=1' = 'object_number=73, tag_tid=11, entry_tid=15889' 2024-04-06T09:39:12.637 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=73, tag_tid=11, entry_tid=15889' '!=' 'object_number=75, tag_tid=11, entry_tid=15803' 2024-04-06T09:39:12.638 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:39:12.638 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=73, tag_tid=11, entry_tid=15889' = 'object_number=75, tag_tid=11, entry_tid=15803' 2024-04-06T09:39:12.638 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=73, tag_tid=11, entry_tid=15889' 2024-04-06T09:39:12.638 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=77, tag_tid=12, entry_tid=1' 2024-04-06T09:39:12.639 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:39:12.639 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:39:12.639 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=12 2024-04-06T09:39:12.640 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=73, tag_tid=11, entry_tid=15889' 2024-04-06T09:39:12.640 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:39:12.640 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:39:12.641 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=11 2024-04-06T09:39:12.642 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=77, tag_tid=12, entry_tid=1' 2024-04-06T09:39:12.642 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:39:12.642 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:39:12.642 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:39:12.643 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=73, tag_tid=11, entry_tid=15889' 2024-04-06T09:39:12.643 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:39:12.643 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:39:12.644 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=15889 2024-04-06T09:39:12.644 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 12 = 11 -a 1 -le 15889 2024-04-06T09:39:12.644 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:39:12.644 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:39:12.645 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:39:12.846 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:39:12.846 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:39:12.846 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:39:12.846 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:39:12.847 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:39:12.847 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:39:12.847 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:39:12.847 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:39:12.847 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:39:12.847 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:39:12.847 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:39:12.847 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:39:12.847 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:39:12.847 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:39:12.847 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:39:12.848 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:39:12.848 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:39:12.850 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:39:12.850 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:39:12.850 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:39:12.850 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:39:12.850 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:39:12.850 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:39:12.850 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:39:12.850 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:39:12.850 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:39:12.850 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:39:12.850 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:39:12.850 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:39:12.850 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:39:12.851 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:39:12.851 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:39:12.851 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:39:12.852 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:39:12.852 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:39:12.852 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:39:12.852 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:39:12.852 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:39:12.852 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:39:12.852 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:39:12.852 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:39:12.853 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:39:12.853 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:39:12.853 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:39:12.853 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:39:12.853 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:39:12.853 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:39:12.853 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:39:12.853 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:39:12.876 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:39:12.877 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:39:12.877 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:39:12.877 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:39:13.010 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:39:13.011 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:39:13.011 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:39:13.011 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:39:13.011 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:39:13.011 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:39:13.011 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:39:13.011 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:39:13.012 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:39:13.012 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:39:13.012 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:39:13.012 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:39:13.012 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:39:13.015 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:39:13.015 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:39:13.016 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:39:13.067 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 18 2024-04-06T09:39:13.067 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 19 2024-04-06T09:39:13.067 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:39:13.067 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=77, tag_tid=12, entry_tid=1], [object_number=76, tag_tid=12, entry_tid=0], [object_number=79, tag_tid=11, entry_tid=17175], [object_number=78, tag_tid=11, entry_tid=17174]]], state=connected] 2024-04-06T09:39:13.068 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=74, tag_tid=11, entry_tid=15994], [object_number=73, tag_tid=11, entry_tid=15993], [object_number=72, tag_tid=11, entry_tid=15992], [object_number=75, tag_tid=11, entry_tid=15991]]], state=connected] 2024-04-06T09:39:13.071 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:39:13.072 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=77, tag_tid=12, entry_tid=1' 2024-04-06T09:39:13.073 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:39:13.074 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:39:13.074 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:39:13.074 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:39:13.074 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:39:13.074 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:39:13.074 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:39:13.074 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:39:13.074 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:39:13.074 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:39:13.074 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:39:13.074 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:39:13.075 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:39:13.075 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:39:13.075 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:39:13.121 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 18 2024-04-06T09:39:13.121 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 19 2024-04-06T09:39:13.121 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:39:13.121 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=77, tag_tid=12, entry_tid=1], [object_number=76, tag_tid=12, entry_tid=0], [object_number=79, tag_tid=11, entry_tid=17175], [object_number=78, tag_tid=11, entry_tid=17174]]], state=connected] 2024-04-06T09:39:13.121 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=74, tag_tid=11, entry_tid=15994], [object_number=73, tag_tid=11, entry_tid=15993], [object_number=72, tag_tid=11, entry_tid=15992], [object_number=75, tag_tid=11, entry_tid=15991]]], state=connected] 2024-04-06T09:39:13.125 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:39:13.127 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=74, tag_tid=11, entry_tid=15994' 2024-04-06T09:39:13.127 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=77, tag_tid=12, entry_tid=1' -a 'object_number=77, tag_tid=12, entry_tid=1' = 'object_number=74, tag_tid=11, entry_tid=15994' 2024-04-06T09:39:13.127 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=74, tag_tid=11, entry_tid=15994' '!=' 'object_number=73, tag_tid=11, entry_tid=15889' 2024-04-06T09:39:13.127 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:39:13.127 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=74, tag_tid=11, entry_tid=15994' = 'object_number=73, tag_tid=11, entry_tid=15889' 2024-04-06T09:39:13.127 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=74, tag_tid=11, entry_tid=15994' 2024-04-06T09:39:13.127 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=77, tag_tid=12, entry_tid=1' 2024-04-06T09:39:13.127 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:39:13.127 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:39:13.128 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=12 2024-04-06T09:39:13.129 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=74, tag_tid=11, entry_tid=15994' 2024-04-06T09:39:13.129 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:39:13.129 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:39:13.130 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=11 2024-04-06T09:39:13.130 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=77, tag_tid=12, entry_tid=1' 2024-04-06T09:39:13.130 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:39:13.130 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:39:13.131 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:39:13.132 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=74, tag_tid=11, entry_tid=15994' 2024-04-06T09:39:13.132 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:39:13.132 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:39:13.133 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=15994 2024-04-06T09:39:13.133 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 12 = 11 -a 1 -le 15994 2024-04-06T09:39:13.134 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:39:13.134 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:39:13.134 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:39:13.335 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:39:13.335 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:39:13.335 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:39:13.335 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:39:13.335 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:39:13.335 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:39:13.335 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:39:13.336 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:39:13.336 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:39:13.336 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:39:13.336 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:39:13.336 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:39:13.336 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:39:13.336 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:39:13.336 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:39:13.336 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:39:13.337 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:39:13.339 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:39:13.339 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:39:13.340 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:39:13.340 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:39:13.340 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:39:13.340 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:39:13.340 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:39:13.340 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:39:13.340 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:39:13.340 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:39:13.340 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:39:13.340 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:39:13.340 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:39:13.340 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:39:13.340 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:39:13.340 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:39:13.341 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:39:13.341 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:39:13.342 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:39:13.342 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:39:13.342 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:39:13.342 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:39:13.342 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:39:13.342 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:39:13.342 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:39:13.342 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:39:13.342 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:39:13.342 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:39:13.342 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:39:13.342 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:39:13.342 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:39:13.343 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:39:13.366 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:39:13.367 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:39:13.367 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:39:13.367 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:39:13.505 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:39:13.506 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:39:13.506 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:39:13.506 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:39:13.506 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:39:13.506 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:39:13.506 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:39:13.506 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:39:13.507 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:39:13.507 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:39:13.507 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:39:13.507 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:39:13.507 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:39:13.510 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:39:13.510 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:39:13.510 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:39:13.555 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 18 2024-04-06T09:39:13.555 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 19 2024-04-06T09:39:13.555 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:39:13.555 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=77, tag_tid=12, entry_tid=1], [object_number=76, tag_tid=12, entry_tid=0], [object_number=79, tag_tid=11, entry_tid=17175], [object_number=78, tag_tid=11, entry_tid=17174]]], state=connected] 2024-04-06T09:39:13.555 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=74, tag_tid=11, entry_tid=16114], [object_number=73, tag_tid=11, entry_tid=16113], [object_number=72, tag_tid=11, entry_tid=16112], [object_number=75, tag_tid=11, entry_tid=16111]]], state=connected] 2024-04-06T09:39:13.559 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:39:13.560 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=77, tag_tid=12, entry_tid=1' 2024-04-06T09:39:13.561 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:39:13.561 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:39:13.561 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:39:13.561 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:39:13.561 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:39:13.561 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:39:13.561 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:39:13.561 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:39:13.562 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:39:13.562 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:39:13.562 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:39:13.562 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:39:13.562 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:39:13.562 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:39:13.563 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:39:13.608 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 18 2024-04-06T09:39:13.608 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 19 2024-04-06T09:39:13.608 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:39:13.608 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=77, tag_tid=12, entry_tid=1], [object_number=76, tag_tid=12, entry_tid=0], [object_number=79, tag_tid=11, entry_tid=17175], [object_number=78, tag_tid=11, entry_tid=17174]]], state=connected] 2024-04-06T09:39:13.608 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=74, tag_tid=11, entry_tid=16114], [object_number=73, tag_tid=11, entry_tid=16113], [object_number=72, tag_tid=11, entry_tid=16112], [object_number=75, tag_tid=11, entry_tid=16111]]], state=connected] 2024-04-06T09:39:13.611 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:39:13.613 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=74, tag_tid=11, entry_tid=16114' 2024-04-06T09:39:13.614 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=77, tag_tid=12, entry_tid=1' -a 'object_number=77, tag_tid=12, entry_tid=1' = 'object_number=74, tag_tid=11, entry_tid=16114' 2024-04-06T09:39:13.614 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=74, tag_tid=11, entry_tid=16114' '!=' 'object_number=74, tag_tid=11, entry_tid=15994' 2024-04-06T09:39:13.614 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:39:13.614 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=74, tag_tid=11, entry_tid=16114' = 'object_number=74, tag_tid=11, entry_tid=15994' 2024-04-06T09:39:13.614 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=74, tag_tid=11, entry_tid=16114' 2024-04-06T09:39:13.614 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=77, tag_tid=12, entry_tid=1' 2024-04-06T09:39:13.614 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:39:13.614 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:39:13.615 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=12 2024-04-06T09:39:13.615 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=74, tag_tid=11, entry_tid=16114' 2024-04-06T09:39:13.615 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:39:13.616 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:39:13.616 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=11 2024-04-06T09:39:13.617 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=77, tag_tid=12, entry_tid=1' 2024-04-06T09:39:13.617 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:39:13.618 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:39:13.618 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:39:13.619 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=74, tag_tid=11, entry_tid=16114' 2024-04-06T09:39:13.619 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:39:13.619 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:39:13.620 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=16114 2024-04-06T09:39:13.620 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 12 = 11 -a 1 -le 16114 2024-04-06T09:39:13.620 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:39:13.620 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:39:13.620 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:39:13.822 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:39:13.822 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:39:13.822 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:39:13.822 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:39:13.822 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:39:13.822 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:39:13.822 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:39:13.823 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:39:13.823 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:39:13.823 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:39:13.823 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:39:13.823 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:39:13.823 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:39:13.823 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:39:13.823 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:39:13.823 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:39:13.824 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:39:13.827 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:39:13.827 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:39:13.827 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:39:13.827 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:39:13.827 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:39:13.827 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:39:13.827 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:39:13.827 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:39:13.827 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:39:13.827 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:39:13.827 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:39:13.828 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:39:13.828 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:39:13.828 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:39:13.828 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:39:13.828 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:39:13.828 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:39:13.828 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:39:13.828 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:39:13.828 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:39:13.828 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:39:13.828 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:39:13.829 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:39:13.829 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:39:13.829 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:39:13.829 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:39:13.829 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:39:13.829 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:39:13.829 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:39:13.829 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:39:13.829 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:39:13.829 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:39:13.853 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:39:13.853 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:39:13.854 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:39:13.854 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:39:13.980 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:39:13.981 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:39:13.981 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:39:13.981 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:39:13.981 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:39:13.981 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:39:13.981 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:39:13.981 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:39:13.982 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:39:13.982 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:39:13.982 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:39:13.982 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:39:13.983 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:39:13.984 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:39:13.985 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:39:13.985 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:39:14.032 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 18 2024-04-06T09:39:14.032 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 19 2024-04-06T09:39:14.032 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:39:14.032 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=77, tag_tid=12, entry_tid=1], [object_number=76, tag_tid=12, entry_tid=0], [object_number=79, tag_tid=11, entry_tid=17175], [object_number=78, tag_tid=11, entry_tid=17174]]], state=connected] 2024-04-06T09:39:14.032 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=72, tag_tid=11, entry_tid=16220], [object_number=75, tag_tid=11, entry_tid=16219], [object_number=74, tag_tid=11, entry_tid=16218], [object_number=73, tag_tid=11, entry_tid=16217]]], state=connected] 2024-04-06T09:39:14.036 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:39:14.037 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=77, tag_tid=12, entry_tid=1' 2024-04-06T09:39:14.038 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:39:14.038 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:39:14.038 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:39:14.038 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:39:14.038 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:39:14.038 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:39:14.039 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:39:14.039 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:39:14.039 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:39:14.039 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:39:14.039 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:39:14.039 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:39:14.039 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:39:14.040 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:39:14.040 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:39:14.085 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 18 2024-04-06T09:39:14.085 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 19 2024-04-06T09:39:14.085 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:39:14.085 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=77, tag_tid=12, entry_tid=1], [object_number=76, tag_tid=12, entry_tid=0], [object_number=79, tag_tid=11, entry_tid=17175], [object_number=78, tag_tid=11, entry_tid=17174]]], state=connected] 2024-04-06T09:39:14.085 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=72, tag_tid=11, entry_tid=16220], [object_number=75, tag_tid=11, entry_tid=16219], [object_number=74, tag_tid=11, entry_tid=16218], [object_number=73, tag_tid=11, entry_tid=16217]]], state=connected] 2024-04-06T09:39:14.089 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:39:14.091 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=72, tag_tid=11, entry_tid=16220' 2024-04-06T09:39:14.091 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=77, tag_tid=12, entry_tid=1' -a 'object_number=77, tag_tid=12, entry_tid=1' = 'object_number=72, tag_tid=11, entry_tid=16220' 2024-04-06T09:39:14.091 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=72, tag_tid=11, entry_tid=16220' '!=' 'object_number=74, tag_tid=11, entry_tid=16114' 2024-04-06T09:39:14.091 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:39:14.091 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=72, tag_tid=11, entry_tid=16220' = 'object_number=74, tag_tid=11, entry_tid=16114' 2024-04-06T09:39:14.091 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=72, tag_tid=11, entry_tid=16220' 2024-04-06T09:39:14.091 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=77, tag_tid=12, entry_tid=1' 2024-04-06T09:39:14.091 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:39:14.091 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:39:14.092 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=12 2024-04-06T09:39:14.092 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=72, tag_tid=11, entry_tid=16220' 2024-04-06T09:39:14.093 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:39:14.093 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:39:14.093 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=11 2024-04-06T09:39:14.094 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=77, tag_tid=12, entry_tid=1' 2024-04-06T09:39:14.094 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:39:14.094 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:39:14.095 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:39:14.096 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=72, tag_tid=11, entry_tid=16220' 2024-04-06T09:39:14.096 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:39:14.096 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:39:14.097 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=16220 2024-04-06T09:39:14.097 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 12 = 11 -a 1 -le 16220 2024-04-06T09:39:14.097 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:39:14.097 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:39:14.098 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:39:14.299 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:39:14.299 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:39:14.299 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:39:14.299 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:39:14.299 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:39:14.300 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:39:14.300 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:39:14.300 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:39:14.300 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:39:14.300 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:39:14.300 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:39:14.300 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:39:14.300 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:39:14.300 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:39:14.300 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:39:14.301 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:39:14.301 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:39:14.303 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:39:14.303 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:39:14.303 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:39:14.303 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:39:14.304 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:39:14.304 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:39:14.304 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:39:14.304 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:39:14.304 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:39:14.304 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:39:14.304 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:39:14.304 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:39:14.304 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:39:14.304 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:39:14.304 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:39:14.304 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:39:14.304 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:39:14.305 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:39:14.305 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:39:14.305 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:39:14.305 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:39:14.305 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:39:14.305 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:39:14.305 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:39:14.305 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:39:14.305 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:39:14.305 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:39:14.305 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:39:14.305 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:39:14.305 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:39:14.305 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:39:14.306 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:39:14.330 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:39:14.330 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:39:14.330 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:39:14.330 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:39:14.455 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:39:14.456 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:39:14.456 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:39:14.456 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:39:14.456 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:39:14.457 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:39:14.457 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:39:14.457 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:39:14.457 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:39:14.457 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:39:14.457 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:39:14.457 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:39:14.458 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:39:14.459 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:39:14.459 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:39:14.459 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:39:14.504 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 18 2024-04-06T09:39:14.504 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 19 2024-04-06T09:39:14.505 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:39:14.505 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=77, tag_tid=12, entry_tid=1], [object_number=76, tag_tid=12, entry_tid=0], [object_number=79, tag_tid=11, entry_tid=17175], [object_number=78, tag_tid=11, entry_tid=17174]]], state=connected] 2024-04-06T09:39:14.505 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=73, tag_tid=11, entry_tid=16321], [object_number=72, tag_tid=11, entry_tid=16320], [object_number=75, tag_tid=11, entry_tid=16319], [object_number=74, tag_tid=11, entry_tid=16318]]], state=connected] 2024-04-06T09:39:14.508 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:39:14.509 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=77, tag_tid=12, entry_tid=1' 2024-04-06T09:39:14.511 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:39:14.511 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:39:14.511 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:39:14.511 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:39:14.511 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:39:14.511 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:39:14.511 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:39:14.511 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:39:14.511 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:39:14.511 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:39:14.512 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:39:14.512 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:39:14.512 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:39:14.512 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:39:14.513 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:39:14.770 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 18 2024-04-06T09:39:14.770 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 19 2024-04-06T09:39:14.770 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:39:14.770 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=77, tag_tid=12, entry_tid=1], [object_number=76, tag_tid=12, entry_tid=0], [object_number=79, tag_tid=11, entry_tid=17175], [object_number=78, tag_tid=11, entry_tid=17174]]], state=connected] 2024-04-06T09:39:14.770 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=73, tag_tid=11, entry_tid=16321], [object_number=72, tag_tid=11, entry_tid=16320], [object_number=75, tag_tid=11, entry_tid=16319], [object_number=74, tag_tid=11, entry_tid=16318]]], state=connected] 2024-04-06T09:39:14.775 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:39:14.777 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=73, tag_tid=11, entry_tid=16321' 2024-04-06T09:39:14.777 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=77, tag_tid=12, entry_tid=1' -a 'object_number=77, tag_tid=12, entry_tid=1' = 'object_number=73, tag_tid=11, entry_tid=16321' 2024-04-06T09:39:14.777 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=73, tag_tid=11, entry_tid=16321' '!=' 'object_number=72, tag_tid=11, entry_tid=16220' 2024-04-06T09:39:14.777 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:39:14.778 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=73, tag_tid=11, entry_tid=16321' = 'object_number=72, tag_tid=11, entry_tid=16220' 2024-04-06T09:39:14.778 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=73, tag_tid=11, entry_tid=16321' 2024-04-06T09:39:14.778 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:39:14.778 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=77, tag_tid=12, entry_tid=1' 2024-04-06T09:39:14.778 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:39:14.779 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=12 2024-04-06T09:39:14.780 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=73, tag_tid=11, entry_tid=16321' 2024-04-06T09:39:14.780 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:39:14.781 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:39:14.782 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=11 2024-04-06T09:39:14.783 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=77, tag_tid=12, entry_tid=1' 2024-04-06T09:39:14.783 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:39:14.783 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:39:14.784 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:39:14.785 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=73, tag_tid=11, entry_tid=16321' 2024-04-06T09:39:14.785 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:39:14.786 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:39:14.787 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=16321 2024-04-06T09:39:14.787 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 12 = 11 -a 1 -le 16321 2024-04-06T09:39:14.787 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:39:14.787 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:39:14.787 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:39:14.989 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:39:14.989 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:39:14.989 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:39:14.989 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:39:14.989 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:39:14.989 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:39:14.989 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:39:14.989 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:39:14.989 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:39:14.989 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:39:14.989 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:39:14.990 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:39:14.990 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:39:14.990 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:39:14.990 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:39:14.990 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:39:14.990 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:39:14.992 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:39:14.993 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:39:14.993 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:39:14.993 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:39:14.993 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:39:14.993 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:39:14.993 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:39:14.993 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:39:14.993 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:39:14.994 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:39:14.994 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:39:14.994 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:39:14.994 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:39:14.994 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:39:14.994 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:39:14.994 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:39:14.994 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:39:14.994 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:39:14.995 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:39:14.995 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:39:14.995 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:39:14.995 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:39:14.995 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:39:14.995 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:39:14.995 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:39:14.995 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:39:14.995 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:39:14.995 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:39:14.995 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:39:14.995 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:39:14.996 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:39:14.996 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:39:15.020 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:39:15.020 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:39:15.020 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:39:15.021 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:39:15.150 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:39:15.151 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:39:15.151 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:39:15.151 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:39:15.151 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:39:15.151 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:39:15.151 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:39:15.151 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:39:15.151 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:39:15.151 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:39:15.152 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:39:15.152 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:39:15.152 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:39:15.154 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:39:15.154 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:39:15.154 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:39:15.207 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 18 2024-04-06T09:39:15.207 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 19 2024-04-06T09:39:15.208 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:39:15.208 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=77, tag_tid=12, entry_tid=1], [object_number=76, tag_tid=12, entry_tid=0], [object_number=79, tag_tid=11, entry_tid=17175], [object_number=78, tag_tid=11, entry_tid=17174]]], state=connected] 2024-04-06T09:39:15.208 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=72, tag_tid=11, entry_tid=16476], [object_number=75, tag_tid=11, entry_tid=16475], [object_number=74, tag_tid=11, entry_tid=16474], [object_number=73, tag_tid=11, entry_tid=16473]]], state=connected] 2024-04-06T09:39:15.211 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:39:15.212 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=77, tag_tid=12, entry_tid=1' 2024-04-06T09:39:15.213 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:39:15.213 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:39:15.213 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:39:15.213 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:39:15.213 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:39:15.213 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:39:15.214 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:39:15.214 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:39:15.214 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:39:15.214 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:39:15.214 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:39:15.214 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:39:15.214 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:39:15.215 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:39:15.215 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:39:15.261 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 18 2024-04-06T09:39:15.261 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 19 2024-04-06T09:39:15.261 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:39:15.261 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=77, tag_tid=12, entry_tid=1], [object_number=76, tag_tid=12, entry_tid=0], [object_number=79, tag_tid=11, entry_tid=17175], [object_number=78, tag_tid=11, entry_tid=17174]]], state=connected] 2024-04-06T09:39:15.261 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=72, tag_tid=11, entry_tid=16476], [object_number=75, tag_tid=11, entry_tid=16475], [object_number=74, tag_tid=11, entry_tid=16474], [object_number=73, tag_tid=11, entry_tid=16473]]], state=connected] 2024-04-06T09:39:15.264 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:39:15.266 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=72, tag_tid=11, entry_tid=16476' 2024-04-06T09:39:15.266 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=77, tag_tid=12, entry_tid=1' -a 'object_number=77, tag_tid=12, entry_tid=1' = 'object_number=72, tag_tid=11, entry_tid=16476' 2024-04-06T09:39:15.266 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=72, tag_tid=11, entry_tid=16476' '!=' 'object_number=73, tag_tid=11, entry_tid=16321' 2024-04-06T09:39:15.266 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:39:15.266 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=72, tag_tid=11, entry_tid=16476' = 'object_number=73, tag_tid=11, entry_tid=16321' 2024-04-06T09:39:15.266 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=72, tag_tid=11, entry_tid=16476' 2024-04-06T09:39:15.267 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=77, tag_tid=12, entry_tid=1' 2024-04-06T09:39:15.267 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:39:15.267 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:39:15.267 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=12 2024-04-06T09:39:15.268 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=72, tag_tid=11, entry_tid=16476' 2024-04-06T09:39:15.268 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:39:15.268 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:39:15.269 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=11 2024-04-06T09:39:15.270 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=77, tag_tid=12, entry_tid=1' 2024-04-06T09:39:15.270 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:39:15.270 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:39:15.270 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:39:15.271 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=72, tag_tid=11, entry_tid=16476' 2024-04-06T09:39:15.271 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:39:15.271 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:39:15.272 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=16476 2024-04-06T09:39:15.272 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 12 = 11 -a 1 -le 16476 2024-04-06T09:39:15.273 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:39:15.273 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:39:15.273 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:39:15.475 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:39:15.475 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:39:15.475 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:39:15.475 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:39:15.476 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:39:15.476 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:39:15.476 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:39:15.476 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:39:15.476 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:39:15.476 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:39:15.476 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:39:15.476 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:39:15.476 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:39:15.476 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:39:15.476 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:39:15.479 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:39:15.479 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:39:15.479 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:39:15.479 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:39:15.479 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:39:15.479 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:39:15.479 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:39:15.480 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:39:15.480 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:39:15.480 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:39:15.480 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:39:15.480 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:39:15.480 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:39:15.480 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:39:15.480 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:39:15.480 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:39:15.480 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:39:15.480 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:39:15.480 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:39:15.480 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:39:15.480 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:39:15.481 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:39:15.481 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:39:15.481 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:39:15.481 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:39:15.481 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:39:15.481 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:39:15.481 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:39:15.481 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:39:15.481 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:39:15.481 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:39:15.481 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:39:15.481 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:39:15.481 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:39:15.505 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:39:15.505 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:39:15.506 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:39:15.506 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:39:15.633 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:39:15.634 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:39:15.635 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:39:15.635 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:39:15.635 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:39:15.635 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:39:15.635 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:39:15.635 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:39:15.635 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:39:15.635 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:39:15.636 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:39:15.636 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:39:15.636 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:39:15.637 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:39:15.637 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:39:15.637 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:39:15.689 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 18 2024-04-06T09:39:15.690 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 19 2024-04-06T09:39:15.690 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:39:15.690 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=77, tag_tid=12, entry_tid=1], [object_number=76, tag_tid=12, entry_tid=0], [object_number=79, tag_tid=11, entry_tid=17175], [object_number=78, tag_tid=11, entry_tid=17174]]], state=connected] 2024-04-06T09:39:15.690 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=75, tag_tid=11, entry_tid=16587], [object_number=74, tag_tid=11, entry_tid=16586], [object_number=73, tag_tid=11, entry_tid=16585], [object_number=72, tag_tid=11, entry_tid=16584]]], state=connected] 2024-04-06T09:39:15.693 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:39:15.695 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=77, tag_tid=12, entry_tid=1' 2024-04-06T09:39:15.696 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:39:15.696 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:39:15.696 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:39:15.696 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:39:15.696 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:39:15.696 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:39:15.696 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:39:15.696 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:39:15.696 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:39:15.696 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:39:15.697 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:39:15.697 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:39:15.697 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:39:15.697 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:39:15.697 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:39:15.743 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 18 2024-04-06T09:39:15.743 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 19 2024-04-06T09:39:15.744 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:39:15.744 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=77, tag_tid=12, entry_tid=1], [object_number=76, tag_tid=12, entry_tid=0], [object_number=79, tag_tid=11, entry_tid=17175], [object_number=78, tag_tid=11, entry_tid=17174]]], state=connected] 2024-04-06T09:39:15.744 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=75, tag_tid=11, entry_tid=16587], [object_number=74, tag_tid=11, entry_tid=16586], [object_number=73, tag_tid=11, entry_tid=16585], [object_number=72, tag_tid=11, entry_tid=16584]]], state=connected] 2024-04-06T09:39:15.747 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:39:15.748 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=75, tag_tid=11, entry_tid=16587' 2024-04-06T09:39:15.749 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=77, tag_tid=12, entry_tid=1' -a 'object_number=77, tag_tid=12, entry_tid=1' = 'object_number=75, tag_tid=11, entry_tid=16587' 2024-04-06T09:39:15.749 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=75, tag_tid=11, entry_tid=16587' '!=' 'object_number=72, tag_tid=11, entry_tid=16476' 2024-04-06T09:39:15.749 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:39:15.749 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=75, tag_tid=11, entry_tid=16587' = 'object_number=72, tag_tid=11, entry_tid=16476' 2024-04-06T09:39:15.749 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=75, tag_tid=11, entry_tid=16587' 2024-04-06T09:39:15.749 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=77, tag_tid=12, entry_tid=1' 2024-04-06T09:39:15.750 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:39:15.750 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:39:15.750 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=12 2024-04-06T09:39:15.751 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=75, tag_tid=11, entry_tid=16587' 2024-04-06T09:39:15.751 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:39:15.751 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:39:15.752 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=11 2024-04-06T09:39:15.752 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=77, tag_tid=12, entry_tid=1' 2024-04-06T09:39:15.752 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:39:15.752 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:39:15.753 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:39:15.754 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=75, tag_tid=11, entry_tid=16587' 2024-04-06T09:39:15.754 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:39:15.754 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:39:15.755 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=16587 2024-04-06T09:39:15.755 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 12 = 11 -a 1 -le 16587 2024-04-06T09:39:15.755 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:39:15.756 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:39:15.756 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:39:15.957 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:39:15.957 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:39:15.957 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:39:15.958 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:39:15.958 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:39:15.958 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:39:15.958 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:39:15.958 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:39:15.958 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:39:15.958 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:39:15.958 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:39:15.958 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:39:15.958 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:39:15.958 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:39:15.958 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:39:15.959 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:39:15.959 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:39:15.962 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:39:15.962 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:39:15.962 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:39:15.962 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:39:15.962 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:39:15.962 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:39:15.962 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:39:15.962 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:39:15.962 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:39:15.962 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:39:15.963 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:39:15.963 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:39:15.963 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:39:15.963 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:39:15.963 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:39:15.963 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:39:15.963 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:39:15.963 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:39:15.963 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:39:15.963 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:39:15.963 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:39:15.963 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:39:15.963 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:39:15.963 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:39:15.964 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:39:15.964 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:39:15.964 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:39:15.964 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:39:15.964 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:39:15.964 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:39:15.964 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:39:15.964 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:39:15.988 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:39:15.988 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:39:15.989 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:39:15.989 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:39:16.129 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:39:16.130 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:39:16.130 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:39:16.130 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:39:16.130 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:39:16.130 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:39:16.130 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:39:16.130 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:39:16.130 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:39:16.130 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:39:16.131 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:39:16.131 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:39:16.131 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:39:16.132 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:39:16.132 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:39:16.133 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:39:16.178 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 18 2024-04-06T09:39:16.178 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 19 2024-04-06T09:39:16.178 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:39:16.178 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=77, tag_tid=12, entry_tid=1], [object_number=76, tag_tid=12, entry_tid=0], [object_number=79, tag_tid=11, entry_tid=17175], [object_number=78, tag_tid=11, entry_tid=17174]]], state=connected] 2024-04-06T09:39:16.178 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=72, tag_tid=11, entry_tid=16684], [object_number=75, tag_tid=11, entry_tid=16683], [object_number=74, tag_tid=11, entry_tid=16682], [object_number=73, tag_tid=11, entry_tid=16681]]], state=connected] 2024-04-06T09:39:16.181 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:39:16.182 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=77, tag_tid=12, entry_tid=1' 2024-04-06T09:39:16.183 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:39:16.184 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:39:16.184 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:39:16.184 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:39:16.184 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:39:16.184 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:39:16.184 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:39:16.184 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:39:16.184 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:39:16.184 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:39:16.184 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:39:16.184 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:39:16.185 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:39:16.185 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:39:16.185 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:39:16.230 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 18 2024-04-06T09:39:16.231 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 19 2024-04-06T09:39:16.231 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:39:16.231 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=77, tag_tid=12, entry_tid=1], [object_number=76, tag_tid=12, entry_tid=0], [object_number=79, tag_tid=11, entry_tid=17175], [object_number=78, tag_tid=11, entry_tid=17174]]], state=connected] 2024-04-06T09:39:16.231 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=72, tag_tid=11, entry_tid=16684], [object_number=75, tag_tid=11, entry_tid=16683], [object_number=74, tag_tid=11, entry_tid=16682], [object_number=73, tag_tid=11, entry_tid=16681]]], state=connected] 2024-04-06T09:39:16.234 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:39:16.236 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=72, tag_tid=11, entry_tid=16684' 2024-04-06T09:39:16.236 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=77, tag_tid=12, entry_tid=1' -a 'object_number=77, tag_tid=12, entry_tid=1' = 'object_number=72, tag_tid=11, entry_tid=16684' 2024-04-06T09:39:16.236 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=72, tag_tid=11, entry_tid=16684' '!=' 'object_number=75, tag_tid=11, entry_tid=16587' 2024-04-06T09:39:16.236 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:39:16.236 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=72, tag_tid=11, entry_tid=16684' = 'object_number=75, tag_tid=11, entry_tid=16587' 2024-04-06T09:39:16.236 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=72, tag_tid=11, entry_tid=16684' 2024-04-06T09:39:16.237 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=77, tag_tid=12, entry_tid=1' 2024-04-06T09:39:16.237 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:39:16.237 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:39:16.237 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=12 2024-04-06T09:39:16.238 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=72, tag_tid=11, entry_tid=16684' 2024-04-06T09:39:16.238 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:39:16.238 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:39:16.239 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=11 2024-04-06T09:39:16.240 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=77, tag_tid=12, entry_tid=1' 2024-04-06T09:39:16.240 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:39:16.240 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:39:16.241 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:39:16.242 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=72, tag_tid=11, entry_tid=16684' 2024-04-06T09:39:16.242 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:39:16.242 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:39:16.243 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=16684 2024-04-06T09:39:16.243 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 12 = 11 -a 1 -le 16684 2024-04-06T09:39:16.243 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:39:16.243 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:39:16.243 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:39:16.444 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:39:16.445 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:39:16.445 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:39:16.445 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:39:16.445 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:39:16.445 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:39:16.445 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:39:16.445 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:39:16.445 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:39:16.445 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:39:16.445 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:39:16.445 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:39:16.445 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:39:16.446 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:39:16.446 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:39:16.446 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:39:16.446 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:39:16.449 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:39:16.449 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:39:16.449 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:39:16.449 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:39:16.449 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:39:16.449 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:39:16.449 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:39:16.450 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:39:16.450 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:39:16.450 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:39:16.450 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:39:16.450 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:39:16.450 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:39:16.450 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:39:16.450 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:39:16.450 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:39:16.450 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:39:16.450 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:39:16.450 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:39:16.450 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:39:16.450 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:39:16.450 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:39:16.451 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:39:16.451 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:39:16.451 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:39:16.451 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:39:16.451 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:39:16.451 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:39:16.451 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:39:16.451 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:39:16.451 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:39:16.451 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:39:16.474 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:39:16.474 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:39:16.475 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:39:16.475 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:39:16.605 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:39:16.606 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:39:16.607 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:39:16.607 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:39:16.607 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:39:16.607 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:39:16.607 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:39:16.607 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:39:16.607 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:39:16.607 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:39:16.608 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:39:16.608 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:39:16.608 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:39:16.609 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:39:16.609 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:39:16.609 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:39:16.655 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 18 2024-04-06T09:39:16.655 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 19 2024-04-06T09:39:16.655 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:39:16.656 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=77, tag_tid=12, entry_tid=1], [object_number=76, tag_tid=12, entry_tid=0], [object_number=79, tag_tid=11, entry_tid=17175], [object_number=78, tag_tid=11, entry_tid=17174]]], state=connected] 2024-04-06T09:39:16.656 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=72, tag_tid=11, entry_tid=16784], [object_number=75, tag_tid=11, entry_tid=16783], [object_number=74, tag_tid=11, entry_tid=16782], [object_number=73, tag_tid=11, entry_tid=16781]]], state=connected] 2024-04-06T09:39:16.659 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:39:16.660 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=77, tag_tid=12, entry_tid=1' 2024-04-06T09:39:16.661 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:39:16.661 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:39:16.661 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:39:16.662 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:39:16.662 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:39:16.662 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:39:16.662 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:39:16.662 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:39:16.662 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:39:16.662 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:39:16.662 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:39:16.662 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:39:16.663 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:39:16.663 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:39:16.663 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:39:16.708 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 18 2024-04-06T09:39:16.708 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 19 2024-04-06T09:39:16.708 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:39:16.708 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=77, tag_tid=12, entry_tid=1], [object_number=76, tag_tid=12, entry_tid=0], [object_number=79, tag_tid=11, entry_tid=17175], [object_number=78, tag_tid=11, entry_tid=17174]]], state=connected] 2024-04-06T09:39:16.709 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=72, tag_tid=11, entry_tid=16784], [object_number=75, tag_tid=11, entry_tid=16783], [object_number=74, tag_tid=11, entry_tid=16782], [object_number=73, tag_tid=11, entry_tid=16781]]], state=connected] 2024-04-06T09:39:16.711 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:39:16.713 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=72, tag_tid=11, entry_tid=16784' 2024-04-06T09:39:16.714 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=77, tag_tid=12, entry_tid=1' -a 'object_number=77, tag_tid=12, entry_tid=1' = 'object_number=72, tag_tid=11, entry_tid=16784' 2024-04-06T09:39:16.714 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=72, tag_tid=11, entry_tid=16784' '!=' 'object_number=72, tag_tid=11, entry_tid=16684' 2024-04-06T09:39:16.714 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:39:16.714 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=72, tag_tid=11, entry_tid=16784' = 'object_number=72, tag_tid=11, entry_tid=16684' 2024-04-06T09:39:16.714 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=72, tag_tid=11, entry_tid=16784' 2024-04-06T09:39:16.714 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=77, tag_tid=12, entry_tid=1' 2024-04-06T09:39:16.714 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:39:16.714 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:39:16.715 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=12 2024-04-06T09:39:16.715 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=72, tag_tid=11, entry_tid=16784' 2024-04-06T09:39:16.715 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:39:16.715 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:39:16.716 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=11 2024-04-06T09:39:16.717 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=77, tag_tid=12, entry_tid=1' 2024-04-06T09:39:16.717 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:39:16.717 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:39:16.718 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:39:16.719 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=72, tag_tid=11, entry_tid=16784' 2024-04-06T09:39:16.719 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:39:16.719 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:39:16.720 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=16784 2024-04-06T09:39:16.720 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 12 = 11 -a 1 -le 16784 2024-04-06T09:39:16.720 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:39:16.720 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:39:16.720 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:39:16.922 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:39:16.922 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:39:16.922 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:39:16.922 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:39:16.922 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:39:16.923 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:39:16.923 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:39:16.923 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:39:16.923 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:39:16.923 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:39:16.923 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:39:16.923 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:39:16.923 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:39:16.923 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:39:16.923 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:39:16.923 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:39:16.924 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:39:16.927 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:39:16.927 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:39:16.927 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:39:16.927 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:39:16.927 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:39:16.927 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:39:16.927 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:39:16.927 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:39:16.927 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:39:16.927 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:39:16.927 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:39:16.927 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:39:16.928 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:39:16.928 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:39:16.928 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:39:16.928 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:39:16.928 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:39:16.928 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:39:16.928 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:39:16.928 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:39:16.928 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:39:16.928 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:39:16.928 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:39:16.928 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:39:16.928 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:39:16.928 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:39:16.928 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:39:16.928 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:39:16.929 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:39:16.929 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:39:16.929 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:39:16.929 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:39:16.952 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:39:16.952 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:39:16.952 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:39:16.952 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:39:17.087 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:39:17.088 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:39:17.088 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:39:17.088 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:39:17.089 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:39:17.089 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:39:17.089 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:39:17.089 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:39:17.089 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:39:17.089 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:39:17.089 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:39:17.090 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:39:17.090 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:39:17.091 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:39:17.092 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:39:17.092 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:39:17.143 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 18 2024-04-06T09:39:17.143 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 19 2024-04-06T09:39:17.144 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:39:17.144 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=77, tag_tid=12, entry_tid=1], [object_number=76, tag_tid=12, entry_tid=0], [object_number=79, tag_tid=11, entry_tid=17175], [object_number=78, tag_tid=11, entry_tid=17174]]], state=connected] 2024-04-06T09:39:17.144 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=72, tag_tid=11, entry_tid=16880], [object_number=75, tag_tid=11, entry_tid=16879], [object_number=74, tag_tid=11, entry_tid=16878], [object_number=73, tag_tid=11, entry_tid=16877]]], state=connected] 2024-04-06T09:39:17.147 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:39:17.148 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=77, tag_tid=12, entry_tid=1' 2024-04-06T09:39:17.149 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:39:17.149 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:39:17.149 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:39:17.150 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:39:17.150 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:39:17.150 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:39:17.150 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:39:17.150 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:39:17.150 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:39:17.150 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:39:17.150 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:39:17.150 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:39:17.150 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:39:17.151 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:39:17.151 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:39:17.197 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 18 2024-04-06T09:39:17.197 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 19 2024-04-06T09:39:17.197 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:39:17.197 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=77, tag_tid=12, entry_tid=1], [object_number=76, tag_tid=12, entry_tid=0], [object_number=79, tag_tid=11, entry_tid=17175], [object_number=78, tag_tid=11, entry_tid=17174]]], state=connected] 2024-04-06T09:39:17.198 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=72, tag_tid=11, entry_tid=16880], [object_number=75, tag_tid=11, entry_tid=16879], [object_number=74, tag_tid=11, entry_tid=16878], [object_number=73, tag_tid=11, entry_tid=16877]]], state=connected] 2024-04-06T09:39:17.201 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:39:17.203 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=72, tag_tid=11, entry_tid=16880' 2024-04-06T09:39:17.203 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=77, tag_tid=12, entry_tid=1' -a 'object_number=77, tag_tid=12, entry_tid=1' = 'object_number=72, tag_tid=11, entry_tid=16880' 2024-04-06T09:39:17.203 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=72, tag_tid=11, entry_tid=16880' '!=' 'object_number=72, tag_tid=11, entry_tid=16784' 2024-04-06T09:39:17.203 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:39:17.203 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=72, tag_tid=11, entry_tid=16880' = 'object_number=72, tag_tid=11, entry_tid=16784' 2024-04-06T09:39:17.203 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=72, tag_tid=11, entry_tid=16880' 2024-04-06T09:39:17.203 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=77, tag_tid=12, entry_tid=1' 2024-04-06T09:39:17.203 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:39:17.203 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:39:17.204 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=12 2024-04-06T09:39:17.205 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=72, tag_tid=11, entry_tid=16880' 2024-04-06T09:39:17.205 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:39:17.205 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:39:17.206 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=11 2024-04-06T09:39:17.206 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=77, tag_tid=12, entry_tid=1' 2024-04-06T09:39:17.206 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:39:17.207 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:39:17.207 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:39:17.208 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=72, tag_tid=11, entry_tid=16880' 2024-04-06T09:39:17.208 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:39:17.208 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:39:17.209 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=16880 2024-04-06T09:39:17.209 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 12 = 11 -a 1 -le 16880 2024-04-06T09:39:17.210 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:39:17.210 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:39:17.210 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:39:17.412 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:39:17.412 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:39:17.412 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:39:17.413 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:39:17.413 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:39:17.413 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:39:17.413 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:39:17.413 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:39:17.413 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:39:17.413 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:39:17.413 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:39:17.413 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:39:17.414 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:39:17.414 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:39:17.414 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:39:17.417 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:39:17.418 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:39:17.418 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:39:17.418 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:39:17.418 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:39:17.418 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:39:17.418 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:39:17.418 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:39:17.419 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:39:17.419 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:39:17.419 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:39:17.419 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:39:17.419 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:39:17.419 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:39:17.419 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:39:17.419 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:39:17.419 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:39:17.420 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:39:17.420 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:39:17.420 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:39:17.420 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:39:17.420 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:39:17.420 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:39:17.420 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:39:17.420 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:39:17.420 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:39:17.420 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:39:17.420 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:39:17.421 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:39:17.421 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:39:17.421 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:39:17.421 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:39:17.421 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:39:17.421 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:39:17.443 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:39:17.443 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:39:17.443 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:39:17.443 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:39:17.579 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:39:17.580 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:39:17.580 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:39:17.580 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:39:17.580 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:39:17.580 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:39:17.580 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:39:17.581 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:39:17.581 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:39:17.581 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:39:17.581 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:39:17.581 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:39:17.582 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:39:17.583 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:39:17.584 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:39:17.584 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:39:17.635 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 18 2024-04-06T09:39:17.635 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 19 2024-04-06T09:39:17.636 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:39:17.636 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=77, tag_tid=12, entry_tid=1], [object_number=76, tag_tid=12, entry_tid=0], [object_number=79, tag_tid=11, entry_tid=17175], [object_number=78, tag_tid=11, entry_tid=17174]]], state=connected] 2024-04-06T09:39:17.636 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=74, tag_tid=11, entry_tid=16962], [object_number=73, tag_tid=11, entry_tid=16961], [object_number=72, tag_tid=11, entry_tid=16960], [object_number=75, tag_tid=11, entry_tid=16959]]], state=connected] 2024-04-06T09:39:17.638 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:39:17.640 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=77, tag_tid=12, entry_tid=1' 2024-04-06T09:39:17.641 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:39:17.641 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:39:17.641 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:39:17.641 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:39:17.641 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:39:17.641 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:39:17.641 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:39:17.642 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:39:17.642 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:39:17.642 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:39:17.642 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:39:17.642 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:39:17.643 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:39:17.643 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:39:17.643 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:39:17.690 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 18 2024-04-06T09:39:17.690 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 19 2024-04-06T09:39:17.690 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:39:17.690 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=77, tag_tid=12, entry_tid=1], [object_number=76, tag_tid=12, entry_tid=0], [object_number=79, tag_tid=11, entry_tid=17175], [object_number=78, tag_tid=11, entry_tid=17174]]], state=connected] 2024-04-06T09:39:17.690 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=74, tag_tid=11, entry_tid=16962], [object_number=73, tag_tid=11, entry_tid=16961], [object_number=72, tag_tid=11, entry_tid=16960], [object_number=75, tag_tid=11, entry_tid=16959]]], state=connected] 2024-04-06T09:39:17.692 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:39:17.693 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=74, tag_tid=11, entry_tid=16962' 2024-04-06T09:39:17.694 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=77, tag_tid=12, entry_tid=1' -a 'object_number=77, tag_tid=12, entry_tid=1' = 'object_number=74, tag_tid=11, entry_tid=16962' 2024-04-06T09:39:17.694 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=74, tag_tid=11, entry_tid=16962' '!=' 'object_number=72, tag_tid=11, entry_tid=16880' 2024-04-06T09:39:17.694 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:39:17.694 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=74, tag_tid=11, entry_tid=16962' = 'object_number=72, tag_tid=11, entry_tid=16880' 2024-04-06T09:39:17.694 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=74, tag_tid=11, entry_tid=16962' 2024-04-06T09:39:17.694 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=77, tag_tid=12, entry_tid=1' 2024-04-06T09:39:17.695 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:39:17.695 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:39:17.695 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=12 2024-04-06T09:39:17.696 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=74, tag_tid=11, entry_tid=16962' 2024-04-06T09:39:17.696 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:39:17.696 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:39:17.697 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=11 2024-04-06T09:39:17.697 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=77, tag_tid=12, entry_tid=1' 2024-04-06T09:39:17.698 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:39:17.698 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:39:17.698 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:39:17.699 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=74, tag_tid=11, entry_tid=16962' 2024-04-06T09:39:17.699 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:39:17.699 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:39:17.700 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=16962 2024-04-06T09:39:17.701 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 12 = 11 -a 1 -le 16962 2024-04-06T09:39:17.701 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:39:17.701 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:39:17.701 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:39:17.902 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:39:17.903 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:39:17.903 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:39:17.903 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:39:17.903 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:39:17.903 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:39:17.903 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:39:17.903 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:39:17.903 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:39:17.903 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:39:17.903 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:39:17.903 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:39:17.903 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:39:17.903 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:39:17.903 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:39:17.904 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:39:17.904 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:39:17.906 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:39:17.906 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:39:17.907 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:39:17.907 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:39:17.907 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:39:17.907 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:39:17.907 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:39:17.907 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:39:17.907 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:39:17.907 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:39:17.907 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:39:17.907 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:39:17.907 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:39:17.907 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:39:17.907 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:39:17.907 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:39:17.907 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:39:17.908 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:39:17.908 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:39:17.908 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:39:17.908 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:39:17.908 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:39:17.908 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:39:17.908 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:39:17.908 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:39:17.908 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:39:17.908 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:39:17.909 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:39:17.909 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:39:17.909 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:39:17.909 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:39:17.909 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:39:17.933 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:39:17.933 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:39:17.934 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:39:17.934 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:39:18.062 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:39:18.063 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:39:18.063 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:39:18.063 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:39:18.063 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:39:18.063 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:39:18.063 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:39:18.063 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:39:18.064 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:39:18.064 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:39:18.064 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:39:18.064 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:39:18.064 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:39:18.066 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:39:18.066 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:39:18.066 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:39:18.122 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 18 2024-04-06T09:39:18.122 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 19 2024-04-06T09:39:18.122 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:39:18.122 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=77, tag_tid=12, entry_tid=1], [object_number=76, tag_tid=12, entry_tid=0], [object_number=79, tag_tid=11, entry_tid=17175], [object_number=78, tag_tid=11, entry_tid=17174]]], state=connected] 2024-04-06T09:39:18.123 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=72, tag_tid=11, entry_tid=17044], [object_number=75, tag_tid=11, entry_tid=17043], [object_number=74, tag_tid=11, entry_tid=17042], [object_number=73, tag_tid=11, entry_tid=17041]]], state=connected] 2024-04-06T09:39:18.126 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:39:18.127 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=77, tag_tid=12, entry_tid=1' 2024-04-06T09:39:18.128 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:39:18.128 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:39:18.128 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:39:18.128 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:39:18.129 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:39:18.130 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:39:18.130 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:39:18.130 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:39:18.130 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:39:18.130 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:39:18.130 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:39:18.130 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:39:18.131 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:39:18.131 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:39:18.131 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:39:18.177 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 18 2024-04-06T09:39:18.177 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 19 2024-04-06T09:39:18.178 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:39:18.178 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=77, tag_tid=12, entry_tid=1], [object_number=76, tag_tid=12, entry_tid=0], [object_number=79, tag_tid=11, entry_tid=17175], [object_number=78, tag_tid=11, entry_tid=17174]]], state=connected] 2024-04-06T09:39:18.178 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=72, tag_tid=11, entry_tid=17044], [object_number=75, tag_tid=11, entry_tid=17043], [object_number=74, tag_tid=11, entry_tid=17042], [object_number=73, tag_tid=11, entry_tid=17041]]], state=connected] 2024-04-06T09:39:18.181 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:39:18.183 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=72, tag_tid=11, entry_tid=17044' 2024-04-06T09:39:18.183 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=77, tag_tid=12, entry_tid=1' -a 'object_number=77, tag_tid=12, entry_tid=1' = 'object_number=72, tag_tid=11, entry_tid=17044' 2024-04-06T09:39:18.183 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=72, tag_tid=11, entry_tid=17044' '!=' 'object_number=74, tag_tid=11, entry_tid=16962' 2024-04-06T09:39:18.183 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:39:18.183 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=72, tag_tid=11, entry_tid=17044' = 'object_number=74, tag_tid=11, entry_tid=16962' 2024-04-06T09:39:18.183 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=72, tag_tid=11, entry_tid=17044' 2024-04-06T09:39:18.183 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=77, tag_tid=12, entry_tid=1' 2024-04-06T09:39:18.184 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:39:18.184 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:39:18.184 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=12 2024-04-06T09:39:18.185 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=72, tag_tid=11, entry_tid=17044' 2024-04-06T09:39:18.185 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:39:18.185 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:39:18.186 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=11 2024-04-06T09:39:18.186 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=77, tag_tid=12, entry_tid=1' 2024-04-06T09:39:18.186 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:39:18.187 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:39:18.187 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:39:18.188 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=72, tag_tid=11, entry_tid=17044' 2024-04-06T09:39:18.188 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:39:18.188 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:39:18.189 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=17044 2024-04-06T09:39:18.189 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 12 = 11 -a 1 -le 17044 2024-04-06T09:39:18.190 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:39:18.190 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:39:18.190 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:39:18.392 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:39:18.392 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:39:18.392 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:39:18.392 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:39:18.392 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:39:18.392 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:39:18.393 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:39:18.393 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:39:18.393 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:39:18.393 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:39:18.393 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:39:18.393 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:39:18.393 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:39:18.393 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:39:18.393 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:39:18.394 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:39:18.394 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:39:18.397 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:39:18.397 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:39:18.397 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:39:18.397 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:39:18.397 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:39:18.397 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:39:18.397 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:39:18.397 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:39:18.398 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:39:18.398 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:39:18.398 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:39:18.398 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:39:18.398 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:39:18.398 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:39:18.398 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:39:18.398 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:39:18.398 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:39:18.399 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:39:18.399 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:39:18.399 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:39:18.399 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:39:18.399 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:39:18.399 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:39:18.399 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:39:18.399 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:39:18.399 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:39:18.399 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:39:18.400 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:39:18.400 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:39:18.400 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:39:18.400 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:39:18.400 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:39:18.422 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:39:18.422 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:39:18.423 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:39:18.423 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:39:18.569 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:39:18.570 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:39:18.570 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:39:18.570 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:39:18.570 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:39:18.570 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:39:18.570 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:39:18.570 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:39:18.570 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:39:18.571 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:39:18.571 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:39:18.571 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:39:18.571 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:39:18.573 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:39:18.573 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:39:18.573 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:39:18.852 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 19 2024-04-06T09:39:18.852 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 19 2024-04-06T09:39:18.852 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:39:18.852 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=77, tag_tid=12, entry_tid=1], [object_number=76, tag_tid=12, entry_tid=0], [object_number=79, tag_tid=11, entry_tid=17175], [object_number=78, tag_tid=11, entry_tid=17174]]], state=connected] 2024-04-06T09:39:18.853 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=78, tag_tid=11, entry_tid=17130], [object_number=77, tag_tid=11, entry_tid=17129], [object_number=76, tag_tid=11, entry_tid=17128], [object_number=79, tag_tid=11, entry_tid=17127]]], state=connected] 2024-04-06T09:39:18.857 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:39:18.859 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=77, tag_tid=12, entry_tid=1' 2024-04-06T09:39:18.860 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:39:18.860 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:39:18.861 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:39:18.861 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:39:18.861 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:39:18.861 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:39:18.861 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:39:18.861 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:39:18.861 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:39:18.861 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:39:18.861 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:39:18.861 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:39:18.863 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:39:18.863 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:39:18.863 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:39:18.917 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 19 2024-04-06T09:39:18.917 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 19 2024-04-06T09:39:18.917 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:39:18.917 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=77, tag_tid=12, entry_tid=1], [object_number=76, tag_tid=12, entry_tid=0], [object_number=79, tag_tid=11, entry_tid=17175], [object_number=78, tag_tid=11, entry_tid=17174]]], state=connected] 2024-04-06T09:39:18.917 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=78, tag_tid=11, entry_tid=17130], [object_number=77, tag_tid=11, entry_tid=17129], [object_number=76, tag_tid=11, entry_tid=17128], [object_number=79, tag_tid=11, entry_tid=17127]]], state=connected] 2024-04-06T09:39:18.921 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:39:18.923 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=78, tag_tid=11, entry_tid=17130' 2024-04-06T09:39:18.923 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=77, tag_tid=12, entry_tid=1' -a 'object_number=77, tag_tid=12, entry_tid=1' = 'object_number=78, tag_tid=11, entry_tid=17130' 2024-04-06T09:39:18.923 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=78, tag_tid=11, entry_tid=17130' '!=' 'object_number=72, tag_tid=11, entry_tid=17044' 2024-04-06T09:39:18.923 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:39:18.923 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=78, tag_tid=11, entry_tid=17130' = 'object_number=72, tag_tid=11, entry_tid=17044' 2024-04-06T09:39:18.923 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=78, tag_tid=11, entry_tid=17130' 2024-04-06T09:39:18.924 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=77, tag_tid=12, entry_tid=1' 2024-04-06T09:39:18.924 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:39:18.924 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:39:18.924 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=12 2024-04-06T09:39:18.925 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=78, tag_tid=11, entry_tid=17130' 2024-04-06T09:39:18.925 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:39:18.925 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:39:18.926 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=11 2024-04-06T09:39:18.927 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=77, tag_tid=12, entry_tid=1' 2024-04-06T09:39:18.927 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:39:18.927 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:39:18.928 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:39:18.929 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=78, tag_tid=11, entry_tid=17130' 2024-04-06T09:39:18.929 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:39:18.929 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:39:18.931 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=17130 2024-04-06T09:39:18.931 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 12 = 11 -a 1 -le 17130 2024-04-06T09:39:18.931 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:39:18.931 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:39:18.931 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:39:19.132 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:39:19.133 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:39:19.133 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:39:19.133 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:39:19.133 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:39:19.133 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:39:19.133 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:39:19.133 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:39:19.133 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:39:19.133 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:39:19.134 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:39:19.134 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:39:19.134 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:39:19.134 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:39:19.134 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:39:19.134 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:39:19.135 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:39:19.137 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:39:19.137 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:39:19.138 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:39:19.138 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:39:19.138 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:39:19.138 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:39:19.138 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:39:19.138 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:39:19.138 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:39:19.138 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:39:19.138 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:39:19.139 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:39:19.139 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:39:19.139 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:39:19.139 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:39:19.139 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:39:19.139 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:39:19.139 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:39:19.139 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:39:19.139 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:39:19.139 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:39:19.139 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:39:19.139 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:39:19.139 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:39:19.140 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:39:19.140 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:39:19.140 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:39:19.140 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:39:19.140 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:39:19.140 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:39:19.140 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:39:19.140 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:39:19.164 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:39:19.164 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:39:19.164 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:39:19.164 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:39:19.294 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:39:19.295 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:39:19.295 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:39:19.295 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:39:19.296 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:39:19.296 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:39:19.296 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:39:19.296 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:39:19.296 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:39:19.296 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:39:19.296 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:39:19.297 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:39:19.297 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:39:19.297 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:39:19.298 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:39:19.298 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:39:19.352 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 19 2024-04-06T09:39:19.353 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 19 2024-04-06T09:39:19.353 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:39:19.353 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=77, tag_tid=12, entry_tid=1], [object_number=76, tag_tid=12, entry_tid=0], [object_number=79, tag_tid=11, entry_tid=17175], [object_number=78, tag_tid=11, entry_tid=17174]]], state=connected] 2024-04-06T09:39:19.353 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=79, tag_tid=11, entry_tid=17175], [object_number=78, tag_tid=11, entry_tid=17174], [object_number=77, tag_tid=11, entry_tid=17173], [object_number=76, tag_tid=11, entry_tid=17172]]], state=connected] 2024-04-06T09:39:19.356 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:39:19.357 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=77, tag_tid=12, entry_tid=1' 2024-04-06T09:39:19.359 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:39:19.359 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:39:19.359 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:39:19.359 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:39:19.359 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:39:19.359 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:39:19.359 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:39:19.359 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:39:19.359 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:39:19.360 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:39:19.360 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:39:19.360 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:39:19.361 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:39:19.361 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:39:19.361 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:39:19.406 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 19 2024-04-06T09:39:19.406 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 19 2024-04-06T09:39:19.406 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:39:19.406 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=77, tag_tid=12, entry_tid=1], [object_number=76, tag_tid=12, entry_tid=0], [object_number=79, tag_tid=11, entry_tid=17175], [object_number=78, tag_tid=11, entry_tid=17174]]], state=connected] 2024-04-06T09:39:19.406 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=79, tag_tid=11, entry_tid=17175], [object_number=78, tag_tid=11, entry_tid=17174], [object_number=77, tag_tid=11, entry_tid=17173], [object_number=76, tag_tid=11, entry_tid=17172]]], state=connected] 2024-04-06T09:39:19.409 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:39:19.411 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=79, tag_tid=11, entry_tid=17175' 2024-04-06T09:39:19.411 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=77, tag_tid=12, entry_tid=1' -a 'object_number=77, tag_tid=12, entry_tid=1' = 'object_number=79, tag_tid=11, entry_tid=17175' 2024-04-06T09:39:19.411 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=79, tag_tid=11, entry_tid=17175' '!=' 'object_number=78, tag_tid=11, entry_tid=17130' 2024-04-06T09:39:19.411 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:39:19.411 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=79, tag_tid=11, entry_tid=17175' = 'object_number=78, tag_tid=11, entry_tid=17130' 2024-04-06T09:39:19.412 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=79, tag_tid=11, entry_tid=17175' 2024-04-06T09:39:19.412 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=77, tag_tid=12, entry_tid=1' 2024-04-06T09:39:19.412 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:39:19.412 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:39:19.412 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=12 2024-04-06T09:39:19.413 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=79, tag_tid=11, entry_tid=17175' 2024-04-06T09:39:19.413 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:39:19.413 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:39:19.414 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=11 2024-04-06T09:39:19.415 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=77, tag_tid=12, entry_tid=1' 2024-04-06T09:39:19.415 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:39:19.415 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:39:19.416 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:39:19.417 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=79, tag_tid=11, entry_tid=17175' 2024-04-06T09:39:19.417 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:39:19.417 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:39:19.418 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=17175 2024-04-06T09:39:19.418 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 12 = 11 -a 1 -le 17175 2024-04-06T09:39:19.418 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:39:19.418 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:39:19.418 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:39:19.620 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:39:19.620 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:39:19.620 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:39:19.620 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:39:19.620 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:39:19.620 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:39:19.621 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:39:19.621 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:39:19.621 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:39:19.621 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:39:19.621 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:39:19.621 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:39:19.621 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:39:19.621 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:39:19.621 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:39:19.622 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:39:19.622 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:39:19.625 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:39:19.625 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:39:19.625 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:39:19.625 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:39:19.625 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:39:19.625 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:39:19.625 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:39:19.625 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:39:19.625 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:39:19.626 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:39:19.626 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:39:19.626 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:39:19.626 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:39:19.626 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:39:19.626 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:39:19.626 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:39:19.626 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:39:19.626 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:39:19.626 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:39:19.626 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:39:19.626 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:39:19.626 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:39:19.626 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:39:19.626 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:39:19.627 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:39:19.627 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:39:19.627 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:39:19.627 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:39:19.627 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:39:19.627 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:39:19.627 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:39:19.627 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:39:19.650 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:39:19.650 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:39:19.650 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:39:19.650 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:39:19.776 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:39:19.779 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:39:19.779 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:39:19.779 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:39:19.780 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:39:19.780 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:39:19.781 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:39:19.781 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:39:19.781 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:39:19.781 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:39:19.781 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:39:19.781 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:39:19.782 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:39:19.784 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:39:19.784 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:39:19.784 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:39:19.832 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 19 2024-04-06T09:39:19.832 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 19 2024-04-06T09:39:19.832 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:39:19.832 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=77, tag_tid=12, entry_tid=1], [object_number=76, tag_tid=12, entry_tid=0], [object_number=79, tag_tid=11, entry_tid=17175], [object_number=78, tag_tid=11, entry_tid=17174]]], state=connected] 2024-04-06T09:39:19.832 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=77, tag_tid=12, entry_tid=1], [object_number=76, tag_tid=12, entry_tid=0], [object_number=79, tag_tid=11, entry_tid=17175], [object_number=78, tag_tid=11, entry_tid=17174]]], state=connected] 2024-04-06T09:39:19.835 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:39:19.837 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=77, tag_tid=12, entry_tid=1' 2024-04-06T09:39:19.838 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:39:19.838 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:39:19.838 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:39:19.838 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:39:19.838 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:39:19.838 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:39:19.838 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:39:19.838 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:39:19.839 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:39:19.839 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:39:19.839 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:39:19.839 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:39:19.839 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:39:19.839 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:39:19.840 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:39:19.885 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 19 2024-04-06T09:39:19.885 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 19 2024-04-06T09:39:19.885 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:39:19.885 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=77, tag_tid=12, entry_tid=1], [object_number=76, tag_tid=12, entry_tid=0], [object_number=79, tag_tid=11, entry_tid=17175], [object_number=78, tag_tid=11, entry_tid=17174]]], state=connected] 2024-04-06T09:39:19.885 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=77, tag_tid=12, entry_tid=1], [object_number=76, tag_tid=12, entry_tid=0], [object_number=79, tag_tid=11, entry_tid=17175], [object_number=78, tag_tid=11, entry_tid=17174]]], state=connected] 2024-04-06T09:39:19.888 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:39:19.890 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=77, tag_tid=12, entry_tid=1' 2024-04-06T09:39:19.890 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=77, tag_tid=12, entry_tid=1' -a 'object_number=77, tag_tid=12, entry_tid=1' = 'object_number=77, tag_tid=12, entry_tid=1' 2024-04-06T09:39:19.890 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:39:19.891 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_snap_present cluster1 mirror test snap4 2024-04-06T09:39:19.891 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:39:19.891 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:39:19.891 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:39:19.891 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local snap_name=snap4 2024-04-06T09:39:19.891 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local s 2024-04-06T09:39:19.891 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-04-06T09:39:19.891 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 1 2024-04-06T09:39:20.891 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 info mirror/test@snap4 2024-04-06T09:39:20.961 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:rbd image 'test': 2024-04-06T09:39:20.961 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: size 512 MiB in 128 objects 2024-04-06T09:39:20.961 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: order 22 (4 MiB objects) 2024-04-06T09:39:20.961 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: snapshot_count: 7 2024-04-06T09:39:20.962 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: id: 10a56da0b207 2024-04-06T09:39:20.962 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: block_name_prefix: rbd_data.10a56da0b207 2024-04-06T09:39:20.962 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: format: 2 2024-04-06T09:39:20.962 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: features: layering, exclusive-lock, journaling 2024-04-06T09:39:20.962 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: op_features: 2024-04-06T09:39:20.962 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: flags: 2024-04-06T09:39:20.962 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: create_timestamp: Sat Apr 6 09:31:59 2024 2024-04-06T09:39:20.962 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: access_timestamp: Sat Apr 6 09:37:35 2024 2024-04-06T09:39:20.962 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: modify_timestamp: Sat Apr 6 09:39:06 2024 2024-04-06T09:39:20.962 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: protected: False 2024-04-06T09:39:20.962 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: journal: 10a56da0b207 2024-04-06T09:39:20.962 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: mirroring state: enabled 2024-04-06T09:39:20.962 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: mirroring mode: journal 2024-04-06T09:39:20.962 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: mirroring global id: 8ad5dedc-b3e0-425d-8993-b3e2231fc5ca 2024-04-06T09:39:20.963 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: mirroring primary: false 2024-04-06T09:39:20.966 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:39:20.966 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n snap3-clean ']' 2024-04-06T09:39:20.966 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ compare_image_snaps mirror test snap3-clean 2024-04-06T09:39:20.967 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:39:20.967 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:39:20.967 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local snap_name=snap3-clean 2024-04-06T09:39:20.967 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local ret=0 2024-04-06T09:39:20.967 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local rmt_export=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.export 2024-04-06T09:39:20.967 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loc_export=/tmp/tmp.BhW6xKmROU/cluster1-mirror-test.export 2024-04-06T09:39:20.967 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rm -f /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.export /tmp/tmp.BhW6xKmROU/cluster1-mirror-test.export 2024-04-06T09:39:20.967 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster2 -p mirror export test@snap3-clean /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.export 2024-04-06T09:39:21.544 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-04-06T09:39:21.560 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 -p mirror export test@snap3-clean /tmp/tmp.BhW6xKmROU/cluster1-mirror-test.export 2024-04-06T09:39:23.081 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-04-06T09:39:23.103 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmp /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.export /tmp/tmp.BhW6xKmROU/cluster1-mirror-test.export 2024-04-06T09:39:23.396 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rm -f /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.export /tmp/tmp.BhW6xKmROU/cluster1-mirror-test.export 2024-04-06T09:39:23.533 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:39:23.533 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ compare_image_snaps mirror test snap4 2024-04-06T09:39:23.533 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:39:23.533 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:39:23.534 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local snap_name=snap4 2024-04-06T09:39:23.534 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local ret=0 2024-04-06T09:39:23.534 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local rmt_export=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.export 2024-04-06T09:39:23.534 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loc_export=/tmp/tmp.BhW6xKmROU/cluster1-mirror-test.export 2024-04-06T09:39:23.534 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rm -f /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.export /tmp/tmp.BhW6xKmROU/cluster1-mirror-test.export 2024-04-06T09:39:23.534 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster2 -p mirror export test@snap4 /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.export 2024-04-06T09:39:24.462 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-04-06T09:39:24.481 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 -p mirror export test@snap4 /tmp/tmp.BhW6xKmROU/cluster1-mirror-test.export 2024-04-06T09:39:26.062 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-04-06T09:39:26.087 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmp /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.export /tmp/tmp.BhW6xKmROU/cluster1-mirror-test.export 2024-04-06T09:39:26.373 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rm -f /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.export /tmp/tmp.BhW6xKmROU/cluster1-mirror-test.export 2024-04-06T09:39:26.513 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:39:26.514 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ clean_snap_name=snap4-clean 2024-04-06T09:39:26.514 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ create_snap cluster2 mirror test snap4-clean 2024-04-06T09:39:26.514 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-04-06T09:39:26.514 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:39:26.514 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:39:26.514 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local snap_name=snap4-clean 2024-04-06T09:39:26.514 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap4-clean --debug-rbd=20 --debug-journaler=20 2024-04-06T09:39:27.580 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in `seq 1 10` 2024-04-06T09:39:27.581 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ stress_write_image cluster2 mirror test 2024-04-06T09:39:27.581 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-04-06T09:39:27.581 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:39:27.581 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:39:27.581 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-04-06T09:39:27.584 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local duration=12 2024-04-06T09:39:27.584 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set +e 2024-04-06T09:39:27.584 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ timeout 12s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror test --debug-rbd=20 --debug-journaler=20 2024-04-06T09:39:29.074 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:100 IOs, 30950400 bytes 2024-04-06T09:39:30.629 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:200 IOs, 61890560 bytes 2024-04-06T09:39:31.054 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:300 IOs, 86055936 bytes 2024-04-06T09:39:31.257 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:400 IOs, 86883328 bytes 2024-04-06T09:39:32.314 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:500 IOs, 103688192 bytes 2024-04-06T09:39:32.383 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.0 with skip (0.6728223841798828) > weight (0.25) 2024-04-06T09:39:32.383 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.1 with skip (0.8181495636531616) > weight (0.25) 2024-04-06T09:39:32.383 INFO:tasks.rbd_mirror_thrash:kill cluster1.client.mirror.2 2024-04-06T09:39:32.384 INFO:tasks.rbd_mirror.cluster1.client.mirror.2:Sent signal 15 2024-04-06T09:39:32.384 INFO:tasks.rbd_mirror_thrash:waiting for 3 secs before reviving daemons 2024-04-06T09:39:32.385 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi192.stderr:2024-04-06T09:39:32.388+0000 7efc5c24a640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.2 (PID: 27476) UID: 1000 2024-04-06T09:39:33.179 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:600 IOs, 136961024 bytes 2024-04-06T09:39:35.385 INFO:tasks.rbd_mirror_thrash:waiting for cluster1.client.mirror.2 2024-04-06T09:39:35.385 INFO:teuthology.orchestra.run:waiting for 600 2024-04-06T09:39:35.385 INFO:tasks.rbd_mirror_thrash:reviving cluster1.client.mirror.2 2024-04-06T09:39:35.385 INFO:tasks.rbd_mirror.cluster1.client.mirror.2:Restarting daemon 2024-04-06T09:39:35.386 DEBUG:teuthology.orchestra.run.smithi192:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.2 2024-04-06T09:39:35.388 INFO:tasks.rbd_mirror.cluster1.client.mirror.2:Started 2024-04-06T09:39:35.388 INFO:tasks.rbd_mirror_thrash:waiting for 100 secs before thrashing 2024-04-06T09:39:35.783 INFO:tasks.ceph.cluster2.osd.2.smithi192.stderr:2024-04-06T09:39:35.784+0000 7fa534aba640 -1 reset not still connected to 0x55933640ec30 2024-04-06T09:39:35.783 INFO:tasks.ceph.cluster2.osd.2.smithi192.stderr:2024-04-06T09:39:35.784+0000 7fa534aba640 -1 reset not still connected to 0x559338f41ba0 2024-04-06T09:39:35.783 INFO:tasks.ceph.cluster2.osd.2.smithi192.stderr:2024-04-06T09:39:35.784+0000 7fa534aba640 -1 reset not still connected to 0x559338f41e10 2024-04-06T09:39:35.896 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:700 IOs, 175037440 bytes 2024-04-06T09:39:37.222 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:800 IOs, 230829056 bytes 2024-04-06T09:39:39.592 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ error_code=124 2024-04-06T09:39:39.592 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set -e 2024-04-06T09:39:39.592 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 124 -eq 124 ']' 2024-04-06T09:39:39.592 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:39:39.592 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+replaying 2024-04-06T09:39:39.592 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:39:39.592 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:39:39.592 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:39:39.593 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+replaying 2024-04-06T09:39:39.593 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local description_pattern= 2024-04-06T09:39:39.593 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local service_pattern= 2024-04-06T09:39:39.593 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:39:39.593 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 1 2024-04-06T09:39:40.593 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test_status_in_pool_dir cluster1 mirror test up+replaying '' '' 2024-04-06T09:39:40.593 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:39:40.593 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:39:40.593 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:39:40.594 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+replaying 2024-04-06T09:39:40.594 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local description_pattern= 2024-04-06T09:39:40.594 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local service_pattern= 2024-04-06T09:39:40.594 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local status 2024-04-06T09:39:40.594 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ CEPH_ARGS= 2024-04-06T09:39:40.594 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster1 mirror image status mirror/test 2024-04-06T09:39:40.680 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ status='test: 2024-04-06T09:39:40.680 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: global_id: 8ad5dedc-b3e0-425d-8993-b3e2231fc5ca 2024-04-06T09:39:40.680 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: state: up+replaying 2024-04-06T09:39:40.680 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: description: replaying, {"bytes_per_second":2855629.43,"entries_behind_primary":0,"entries_per_second":178.93,"non_primary_position":{"entry_tid":1,"object_number":77,"tag_tid":12},"primary_position":{"entry_tid":1,"object_number":77,"tag_tid":12}} 2024-04-06T09:39:40.681 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: service: mirror.0 on smithi192 2024-04-06T09:39:40.681 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: last_update: 2024-04-06 09:39:27 2024-04-06T09:39:40.681 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: peer_sites: 2024-04-06T09:39:40.681 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: name: cluster2 2024-04-06T09:39:40.681 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: state: up+stopped 2024-04-06T09:39:40.681 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: description: local image is primary 2024-04-06T09:39:40.681 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: last_update: 2024-04-06 09:39:11' 2024-04-06T09:39:40.681 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^ state: .*up+replaying' 2024-04-06T09:39:40.682 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: state: up+replaying 2024-04-06T09:39:40.682 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^ description: .*' 2024-04-06T09:39:40.682 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: description: replaying, {"bytes_per_second":2855629.43,"entries_behind_primary":0,"entries_per_second":178.93,"non_primary_position":{"entry_tid":1,"object_number":77,"tag_tid":12},"primary_position":{"entry_tid":1,"object_number":77,"tag_tid":12}} 2024-04-06T09:39:40.683 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n '' ']' 2024-04-06T09:39:40.683 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ echo up+replaying 2024-04-06T09:39:40.683 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^up+' 2024-04-06T09:39:40.684 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:up+replaying 2024-04-06T09:39:40.684 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep 'service: *mirror..* on ' 2024-04-06T09:39:40.686 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: service: mirror.0 on smithi192 2024-04-06T09:39:40.686 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local last_update 2024-04-06T09:39:40.687 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-04-06T09:39:40.689 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_update='2024-04-06 09:39:27' 2024-04-06T09:39:40.689 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+replaying '2024-04-06 09:39:27' 2024-04-06T09:39:40.689 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:39:40.689 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:39:40.689 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:39:40.689 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+replaying 2024-04-06T09:39:40.689 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'prev_last_update=2024-04-06 09:39:27' 2024-04-06T09:39:40.689 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local status 2024-04-06T09:39:40.689 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ CEPH_ARGS= 2024-04-06T09:39:40.690 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-04-06T09:39:40.768 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ status='OKOKOK142094268mirror.1smithi19218.2.2-1059-g4d05f4cafalseOK42584261mirror.0smithi19218.2.2-1059-g4d05f4catrueOK43194322mirror.3smithi19218.2.2-1059-g4d05f4cafalseOK43814384mirror.2smithi19218.2.2-1059-g4d05f4cafalseOKtest8ad5dedc-b3e0-425d-8993-b3e2231fc5caup+replayingreplaying, {"bytes_per_second":2855629.43,"entries_behind_primary":0,"entries_per_second":178.93,"non_primary_position":{"entry_tid":1,"object_number":77,"tag_tid":12},"primary_position":{"entry_tid":1,"object_number":77,"tag_tid":12}}42584261mirror.0smithi1922024-04-06 09:39:27cluster2c5e3bdc3-c286-4289-8fd0-6b8628b92376up+stoppedlocal image is primary2024-04-06 09:39:11' 2024-04-06T09:39:40.768 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local last_update state 2024-04-06T09:39:40.769 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2024-04-06T09:39:40.771 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_update='2024-04-06 09:39:27' 2024-04-06T09:39:40.772 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2024-04-06T09:39:40.774 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ state=up+replaying 2024-04-06T09:39:40.775 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ echo up+replaying 2024-04-06T09:39:40.775 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep up+replaying 2024-04-06T09:39:40.776 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:up+replaying 2024-04-06T09:39:40.776 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:39:40.777 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:39:40.777 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ snap_name=snap5 2024-04-06T09:39:40.777 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ create_snap cluster2 mirror test snap5 2024-04-06T09:39:40.777 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-04-06T09:39:40.777 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:39:40.777 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:39:40.777 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local snap_name=snap5 2024-04-06T09:39:40.777 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap5 --debug-rbd=20 --debug-journaler=20 2024-04-06T09:39:50.062 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:39:50.060+0000 7f2e84d66640 -1 reset not still connected to 0x560d27073d40 2024-04-06T09:40:07.933 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:40:07.932+0000 7f2e84d66640 -1 reset not still connected to 0x560d27073d40 2024-04-06T09:40:11.547 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_image_replay_started cluster1 mirror test 2024-04-06T09:40:11.547 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:40:11.547 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:40:11.547 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:40:11.547 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_image_replay_state cluster1 mirror test started 2024-04-06T09:40:11.547 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:40:11.547 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:40:11.547 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:40:11.547 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state=started 2024-04-06T09:40:11.548 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local s 2024-04-06T09:40:11.548 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:40:11.548 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 1 2024-04-06T09:40:12.548 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test_image_replay_state cluster1 mirror test started 2024-04-06T09:40:12.548 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:40:12.548 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:40:12.548 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:40:12.549 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local test_state=started 2024-04-06T09:40:12.549 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local status_result 2024-04-06T09:40:12.549 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local current_state=stopped 2024-04-06T09:40:12.550 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2024-04-06T09:40:12.550 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -i state 2024-04-06T09:40:12.550 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_instance=cluster1 2024-04-06T09:40:12.550 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ shift 2024-04-06T09:40:12.550 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:40:12.550 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance=cluster1 2024-04-06T09:40:12.550 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local loop_instance 2024-04-06T09:40:12.550 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:40:12.550 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sleep 0 2024-04-06T09:40:12.551 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:40:12.551 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ seq 0 3 2024-04-06T09:40:12.553 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:40:12.553 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2024-04-06T09:40:12.553 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1:0 2024-04-06T09:40:12.553 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ shift 2024-04-06T09:40:12.554 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:40:12.554 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:40:12.554 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:40:12.554 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=cluster 2024-04-06T09:40:12.554 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:40:12.554 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:40:12.554 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:40:12.554 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:40:12.554 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval cluster=cluster1 2024-04-06T09:40:12.554 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ cluster=cluster1 2024-04-06T09:40:12.554 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:40:12.554 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:40:12.554 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:40:12.555 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local local_cluster=cluster1:0 2024-04-06T09:40:12.556 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local cluster=cluster1 2024-04-06T09:40:12.556 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local instance 2024-04-06T09:40:12.556 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:40:12.556 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local val=cluster1:0 2024-04-06T09:40:12.556 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local cluster_var_name=local_cluster 2024-04-06T09:40:12.556 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local instance_var_name=instance 2024-04-06T09:40:12.556 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ cluster=cluster1 2024-04-06T09:40:12.557 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:40:12.557 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ '[' 0 = cluster1:0 ']' 2024-04-06T09:40:12.557 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ eval local_cluster=cluster1 2024-04-06T09:40:12.557 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++++ local_cluster=cluster1 2024-04-06T09:40:12.557 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ eval instance=0 2024-04-06T09:40:12.557 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++++ instance=0 2024-04-06T09:40:12.557 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:40:12.576 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:40:12.576 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:40:12.577 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:40:12.577 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2024-04-06T09:40:12.696 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ return 0 2024-04-06T09:40:12.697 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ status_result=' "state": "Replaying"' 2024-04-06T09:40:12.697 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep -i Replaying 2024-04-06T09:40:12.697 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ echo ' "state": "Replaying"' 2024-04-06T09:40:12.698 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: "state": "Replaying" 2024-04-06T09:40:12.699 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ current_state=started 2024-04-06T09:40:12.699 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test started = started 2024-04-06T09:40:12.699 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:40:12.700 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror test 2024-04-06T09:40:12.700 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local local_cluster=cluster1 2024-04-06T09:40:12.700 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-04-06T09:40:12.700 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:40:12.700 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:40:12.700 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' journal = journal ']' 2024-04-06T09:40:12.700 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_journal_replay_complete cluster1 cluster2 mirror test 2024-04-06T09:40:12.700 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local local_cluster=cluster1 2024-04-06T09:40:12.700 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-04-06T09:40:12.700 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:40:12.700 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:40:12.700 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local s master_pos mirror_pos last_mirror_pos 2024-04-06T09:40:12.700 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local master_tag master_entry mirror_tag mirror_entry 2024-04-06T09:40:12.700 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:40:12.701 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:40:12.701 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:40:12.901 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:40:12.901 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:40:12.901 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:40:12.901 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:40:12.901 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:40:12.901 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:40:12.901 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:40:12.901 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:40:12.902 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:40:12.902 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:40:12.902 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:40:12.902 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:40:12.902 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:40:12.902 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:40:12.902 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:40:12.902 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:40:12.903 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:40:12.904 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:40:12.904 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:40:12.904 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:40:12.904 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:40:12.905 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:40:12.905 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:40:12.905 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:40:12.905 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:40:12.905 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:40:12.905 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:40:12.905 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:40:12.905 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:40:12.905 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:40:12.905 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:40:12.905 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:40:12.905 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:40:12.907 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:40:12.907 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:40:12.907 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:40:12.907 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:40:12.907 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:40:12.907 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:40:12.907 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:40:12.907 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:40:12.907 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:40:12.907 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:40:12.907 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:40:12.907 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:40:12.907 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:40:12.908 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:40:12.908 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:40:12.908 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:40:12.932 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:40:12.933 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:40:12.933 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:40:12.933 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:40:13.065 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:40:13.066 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:40:13.066 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:40:13.066 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:40:13.066 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:40:13.066 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:40:13.066 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:40:13.066 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:40:13.066 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:40:13.066 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:40:13.067 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:40:13.067 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:40:13.067 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:40:13.068 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:40:13.068 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:40:13.069 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:40:13.124 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 20 2024-04-06T09:40:13.124 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 23 2024-04-06T09:40:13.124 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:40:13.124 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=93, tag_tid=15, entry_tid=1], [object_number=92, tag_tid=15, entry_tid=0], [object_number=95, tag_tid=14, entry_tid=19039], [object_number=94, tag_tid=14, entry_tid=19038]]], state=connected] 2024-04-06T09:40:13.124 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=83, tag_tid=14, entry_tid=7239], [object_number=82, tag_tid=14, entry_tid=7238], [object_number=81, tag_tid=14, entry_tid=7237], [object_number=80, tag_tid=14, entry_tid=7236]]], state=connected] 2024-04-06T09:40:13.127 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:40:13.128 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=93, tag_tid=15, entry_tid=1' 2024-04-06T09:40:13.129 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:40:13.129 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:40:13.129 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:40:13.130 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:40:13.130 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:40:13.130 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:40:13.130 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:40:13.130 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:40:13.130 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:40:13.130 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:40:13.130 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:40:13.130 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:40:13.131 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:40:13.131 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:40:13.131 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:40:13.180 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 20 2024-04-06T09:40:13.180 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 23 2024-04-06T09:40:13.180 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:40:13.180 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=93, tag_tid=15, entry_tid=1], [object_number=92, tag_tid=15, entry_tid=0], [object_number=95, tag_tid=14, entry_tid=19039], [object_number=94, tag_tid=14, entry_tid=19038]]], state=connected] 2024-04-06T09:40:13.180 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=83, tag_tid=14, entry_tid=7239], [object_number=82, tag_tid=14, entry_tid=7238], [object_number=81, tag_tid=14, entry_tid=7237], [object_number=80, tag_tid=14, entry_tid=7236]]], state=connected] 2024-04-06T09:40:13.184 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:40:13.186 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=83, tag_tid=14, entry_tid=7239' 2024-04-06T09:40:13.186 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=93, tag_tid=15, entry_tid=1' -a 'object_number=93, tag_tid=15, entry_tid=1' = 'object_number=83, tag_tid=14, entry_tid=7239' 2024-04-06T09:40:13.186 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=83, tag_tid=14, entry_tid=7239' '!=' '' 2024-04-06T09:40:13.186 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:40:13.186 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=83, tag_tid=14, entry_tid=7239' = '' 2024-04-06T09:40:13.186 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=83, tag_tid=14, entry_tid=7239' 2024-04-06T09:40:13.186 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=93, tag_tid=15, entry_tid=1' 2024-04-06T09:40:13.186 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:40:13.186 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:40:13.187 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=15 2024-04-06T09:40:13.188 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=83, tag_tid=14, entry_tid=7239' 2024-04-06T09:40:13.188 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:40:13.188 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:40:13.189 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=14 2024-04-06T09:40:13.189 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=93, tag_tid=15, entry_tid=1' 2024-04-06T09:40:13.190 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:40:13.190 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:40:13.190 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:40:13.191 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=83, tag_tid=14, entry_tid=7239' 2024-04-06T09:40:13.191 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:40:13.191 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:40:13.192 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=7239 2024-04-06T09:40:13.193 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 15 = 14 -a 1 -le 7239 2024-04-06T09:40:13.193 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:40:13.193 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:40:13.193 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:40:13.394 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:40:13.395 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:40:13.395 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:40:13.395 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:40:13.395 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:40:13.395 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:40:13.395 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:40:13.395 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:40:13.395 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:40:13.395 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:40:13.395 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:40:13.395 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:40:13.395 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:40:13.395 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:40:13.396 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:40:13.396 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:40:13.396 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:40:13.398 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:40:13.399 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:40:13.399 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:40:13.399 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:40:13.399 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:40:13.399 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:40:13.399 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:40:13.399 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:40:13.399 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:40:13.399 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:40:13.399 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:40:13.399 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:40:13.399 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:40:13.399 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:40:13.400 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:40:13.400 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:40:13.401 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:40:13.401 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:40:13.401 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:40:13.401 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:40:13.401 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:40:13.401 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:40:13.401 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:40:13.401 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:40:13.401 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:40:13.401 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:40:13.401 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:40:13.401 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:40:13.402 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:40:13.402 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:40:13.402 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:40:13.402 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:40:13.426 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:40:13.427 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:40:13.427 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:40:13.427 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:40:13.562 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:40:13.563 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:40:13.563 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:40:13.563 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:40:13.563 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:40:13.564 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:40:13.564 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:40:13.564 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:40:13.564 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:40:13.564 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:40:13.564 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:40:13.565 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:40:13.565 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:40:13.567 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:40:13.567 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:40:13.568 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:40:13.621 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 20 2024-04-06T09:40:13.621 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 23 2024-04-06T09:40:13.621 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:40:13.621 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=93, tag_tid=15, entry_tid=1], [object_number=92, tag_tid=15, entry_tid=0], [object_number=95, tag_tid=14, entry_tid=19039], [object_number=94, tag_tid=14, entry_tid=19038]]], state=connected] 2024-04-06T09:40:13.621 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=82, tag_tid=14, entry_tid=7334], [object_number=81, tag_tid=14, entry_tid=7333], [object_number=80, tag_tid=14, entry_tid=7332], [object_number=83, tag_tid=14, entry_tid=7331]]], state=connected] 2024-04-06T09:40:13.625 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:40:13.626 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=93, tag_tid=15, entry_tid=1' 2024-04-06T09:40:13.627 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:40:13.627 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:40:13.627 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:40:13.627 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:40:13.627 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:40:13.627 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:40:13.628 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:40:13.628 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:40:13.628 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:40:13.628 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:40:13.628 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:40:13.628 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:40:13.628 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:40:13.629 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:40:13.629 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:40:13.676 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 20 2024-04-06T09:40:13.677 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 23 2024-04-06T09:40:13.677 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:40:13.677 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=93, tag_tid=15, entry_tid=1], [object_number=92, tag_tid=15, entry_tid=0], [object_number=95, tag_tid=14, entry_tid=19039], [object_number=94, tag_tid=14, entry_tid=19038]]], state=connected] 2024-04-06T09:40:13.677 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=82, tag_tid=14, entry_tid=7334], [object_number=81, tag_tid=14, entry_tid=7333], [object_number=80, tag_tid=14, entry_tid=7332], [object_number=83, tag_tid=14, entry_tid=7331]]], state=connected] 2024-04-06T09:40:13.680 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:40:13.683 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=82, tag_tid=14, entry_tid=7334' 2024-04-06T09:40:13.683 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=93, tag_tid=15, entry_tid=1' -a 'object_number=93, tag_tid=15, entry_tid=1' = 'object_number=82, tag_tid=14, entry_tid=7334' 2024-04-06T09:40:13.683 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=82, tag_tid=14, entry_tid=7334' '!=' 'object_number=83, tag_tid=14, entry_tid=7239' 2024-04-06T09:40:13.683 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:40:13.683 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=82, tag_tid=14, entry_tid=7334' = 'object_number=83, tag_tid=14, entry_tid=7239' 2024-04-06T09:40:13.683 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=82, tag_tid=14, entry_tid=7334' 2024-04-06T09:40:13.683 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=93, tag_tid=15, entry_tid=1' 2024-04-06T09:40:13.683 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:40:13.683 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:40:13.684 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=15 2024-04-06T09:40:13.684 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=82, tag_tid=14, entry_tid=7334' 2024-04-06T09:40:13.685 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:40:13.685 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:40:13.685 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=14 2024-04-06T09:40:13.686 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=93, tag_tid=15, entry_tid=1' 2024-04-06T09:40:13.686 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:40:13.686 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:40:13.687 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:40:13.688 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=82, tag_tid=14, entry_tid=7334' 2024-04-06T09:40:13.688 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:40:13.688 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:40:13.689 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=7334 2024-04-06T09:40:13.690 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 15 = 14 -a 1 -le 7334 2024-04-06T09:40:13.690 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:40:13.690 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:40:13.690 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:40:13.891 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:40:13.891 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:40:13.891 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:40:13.891 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:40:13.891 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:40:13.891 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:40:13.892 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:40:13.892 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:40:13.892 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:40:13.892 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:40:13.892 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:40:13.892 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:40:13.892 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:40:13.892 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:40:13.892 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:40:13.892 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:40:13.893 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:40:13.897 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:40:13.897 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:40:13.897 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:40:13.897 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:40:13.897 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:40:13.897 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:40:13.898 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:40:13.898 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:40:13.898 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:40:13.898 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:40:13.898 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:40:13.898 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:40:13.898 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:40:13.898 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:40:13.898 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:40:13.898 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:40:13.898 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:40:13.898 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:40:13.898 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:40:13.898 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:40:13.899 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:40:13.899 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:40:13.899 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:40:13.899 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:40:13.899 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:40:13.899 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:40:13.899 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:40:13.899 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:40:13.899 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:40:13.899 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:40:13.899 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:40:13.899 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:40:13.922 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:40:13.922 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:40:13.922 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:40:13.923 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:40:14.049 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:40:14.051 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:40:14.051 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:40:14.051 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:40:14.051 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:40:14.051 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:40:14.051 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:40:14.051 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:40:14.052 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:40:14.052 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:40:14.052 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:40:14.052 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:40:14.052 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:40:14.054 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:40:14.054 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:40:14.054 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:40:14.105 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 20 2024-04-06T09:40:14.105 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 23 2024-04-06T09:40:14.105 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:40:14.105 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=93, tag_tid=15, entry_tid=1], [object_number=92, tag_tid=15, entry_tid=0], [object_number=95, tag_tid=14, entry_tid=19039], [object_number=94, tag_tid=14, entry_tid=19038]]], state=connected] 2024-04-06T09:40:14.105 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=82, tag_tid=14, entry_tid=7442], [object_number=81, tag_tid=14, entry_tid=7441], [object_number=80, tag_tid=14, entry_tid=7440], [object_number=83, tag_tid=14, entry_tid=7439]]], state=connected] 2024-04-06T09:40:14.109 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:40:14.110 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=93, tag_tid=15, entry_tid=1' 2024-04-06T09:40:14.111 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:40:14.111 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:40:14.111 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:40:14.111 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:40:14.111 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:40:14.111 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:40:14.112 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:40:14.112 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:40:14.112 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:40:14.112 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:40:14.112 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:40:14.112 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:40:14.113 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:40:14.113 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:40:14.113 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:40:14.159 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 20 2024-04-06T09:40:14.159 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 23 2024-04-06T09:40:14.160 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:40:14.160 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=93, tag_tid=15, entry_tid=1], [object_number=92, tag_tid=15, entry_tid=0], [object_number=95, tag_tid=14, entry_tid=19039], [object_number=94, tag_tid=14, entry_tid=19038]]], state=connected] 2024-04-06T09:40:14.160 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=82, tag_tid=14, entry_tid=7442], [object_number=81, tag_tid=14, entry_tid=7441], [object_number=80, tag_tid=14, entry_tid=7440], [object_number=83, tag_tid=14, entry_tid=7439]]], state=connected] 2024-04-06T09:40:14.163 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:40:14.165 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=82, tag_tid=14, entry_tid=7442' 2024-04-06T09:40:14.165 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=93, tag_tid=15, entry_tid=1' -a 'object_number=93, tag_tid=15, entry_tid=1' = 'object_number=82, tag_tid=14, entry_tid=7442' 2024-04-06T09:40:14.165 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=82, tag_tid=14, entry_tid=7442' '!=' 'object_number=82, tag_tid=14, entry_tid=7334' 2024-04-06T09:40:14.165 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:40:14.165 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=82, tag_tid=14, entry_tid=7442' = 'object_number=82, tag_tid=14, entry_tid=7334' 2024-04-06T09:40:14.165 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=82, tag_tid=14, entry_tid=7442' 2024-04-06T09:40:14.165 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=93, tag_tid=15, entry_tid=1' 2024-04-06T09:40:14.166 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:40:14.166 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:40:14.166 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=15 2024-04-06T09:40:14.171 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=82, tag_tid=14, entry_tid=7442' 2024-04-06T09:40:14.172 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:40:14.172 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:40:14.172 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=14 2024-04-06T09:40:14.172 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=93, tag_tid=15, entry_tid=1' 2024-04-06T09:40:14.172 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:40:14.172 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:40:14.172 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:40:14.172 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=82, tag_tid=14, entry_tid=7442' 2024-04-06T09:40:14.172 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:40:14.172 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:40:14.172 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=7442 2024-04-06T09:40:14.172 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 15 = 14 -a 1 -le 7442 2024-04-06T09:40:14.172 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:40:14.173 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:40:14.173 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:40:14.373 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:40:14.374 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:40:14.374 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:40:14.374 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:40:14.374 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:40:14.374 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:40:14.374 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:40:14.374 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:40:14.374 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:40:14.374 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:40:14.374 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:40:14.374 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:40:14.374 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:40:14.374 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:40:14.375 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:40:14.375 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:40:14.375 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:40:14.378 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:40:14.378 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:40:14.378 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:40:14.378 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:40:14.378 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:40:14.378 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:40:14.378 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:40:14.378 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:40:14.378 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:40:14.379 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:40:14.379 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:40:14.379 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:40:14.379 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:40:14.379 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:40:14.379 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:40:14.379 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:40:14.379 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:40:14.379 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:40:14.379 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:40:14.379 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:40:14.379 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:40:14.379 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:40:14.379 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:40:14.380 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:40:14.380 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:40:14.380 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:40:14.380 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:40:14.380 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:40:14.380 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:40:14.380 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:40:14.380 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:40:14.380 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:40:14.407 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:40:14.407 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:40:14.407 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:40:14.408 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:40:14.542 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:40:14.542 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:40:14.542 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:40:14.542 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:40:14.543 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:40:14.543 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:40:14.543 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:40:14.543 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:40:14.543 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:40:14.543 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:40:14.543 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:40:14.543 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:40:14.544 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:40:14.546 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:40:14.547 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:40:14.547 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:40:14.598 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 20 2024-04-06T09:40:14.599 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 23 2024-04-06T09:40:14.599 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:40:14.599 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=93, tag_tid=15, entry_tid=1], [object_number=92, tag_tid=15, entry_tid=0], [object_number=95, tag_tid=14, entry_tid=19039], [object_number=94, tag_tid=14, entry_tid=19038]]], state=connected] 2024-04-06T09:40:14.599 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=83, tag_tid=14, entry_tid=7559], [object_number=82, tag_tid=14, entry_tid=7558], [object_number=81, tag_tid=14, entry_tid=7557], [object_number=80, tag_tid=14, entry_tid=7556]]], state=connected] 2024-04-06T09:40:14.602 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:40:14.603 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=93, tag_tid=15, entry_tid=1' 2024-04-06T09:40:14.605 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:40:14.605 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:40:14.605 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:40:14.605 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:40:14.605 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:40:14.605 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:40:14.605 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:40:14.605 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:40:14.605 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:40:14.605 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:40:14.605 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:40:14.605 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:40:14.606 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:40:14.606 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:40:14.606 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:40:14.654 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 20 2024-04-06T09:40:14.654 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 23 2024-04-06T09:40:14.654 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:40:14.654 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=93, tag_tid=15, entry_tid=1], [object_number=92, tag_tid=15, entry_tid=0], [object_number=95, tag_tid=14, entry_tid=19039], [object_number=94, tag_tid=14, entry_tid=19038]]], state=connected] 2024-04-06T09:40:14.654 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=83, tag_tid=14, entry_tid=7559], [object_number=82, tag_tid=14, entry_tid=7558], [object_number=81, tag_tid=14, entry_tid=7557], [object_number=80, tag_tid=14, entry_tid=7556]]], state=connected] 2024-04-06T09:40:14.657 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:40:14.660 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=83, tag_tid=14, entry_tid=7559' 2024-04-06T09:40:14.660 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=93, tag_tid=15, entry_tid=1' -a 'object_number=93, tag_tid=15, entry_tid=1' = 'object_number=83, tag_tid=14, entry_tid=7559' 2024-04-06T09:40:14.660 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=83, tag_tid=14, entry_tid=7559' '!=' 'object_number=82, tag_tid=14, entry_tid=7442' 2024-04-06T09:40:14.660 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:40:14.660 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=83, tag_tid=14, entry_tid=7559' = 'object_number=82, tag_tid=14, entry_tid=7442' 2024-04-06T09:40:14.660 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=83, tag_tid=14, entry_tid=7559' 2024-04-06T09:40:14.660 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=93, tag_tid=15, entry_tid=1' 2024-04-06T09:40:14.660 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:40:14.660 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:40:14.661 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=15 2024-04-06T09:40:14.661 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=83, tag_tid=14, entry_tid=7559' 2024-04-06T09:40:14.662 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:40:14.662 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:40:14.662 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=14 2024-04-06T09:40:14.663 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=93, tag_tid=15, entry_tid=1' 2024-04-06T09:40:14.663 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:40:14.663 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:40:14.664 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:40:14.665 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=83, tag_tid=14, entry_tid=7559' 2024-04-06T09:40:14.665 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:40:14.665 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:40:14.666 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=7559 2024-04-06T09:40:14.666 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 15 = 14 -a 1 -le 7559 2024-04-06T09:40:14.666 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:40:14.666 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:40:14.667 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:40:14.868 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:40:14.868 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:40:14.868 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:40:14.868 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:40:14.869 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:40:14.869 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:40:14.869 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:40:14.869 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:40:14.869 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:40:14.869 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:40:14.869 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:40:14.869 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:40:14.869 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:40:14.869 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:40:14.869 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:40:14.870 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:40:14.870 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:40:14.873 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:40:14.873 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:40:14.873 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:40:14.873 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:40:14.873 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:40:14.873 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:40:14.873 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:40:14.873 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:40:14.873 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:40:14.873 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:40:14.873 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:40:14.873 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:40:14.873 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:40:14.873 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:40:14.874 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:40:14.874 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:40:14.874 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:40:14.874 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:40:14.874 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:40:14.874 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:40:14.874 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:40:14.874 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:40:14.874 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:40:14.874 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:40:14.874 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:40:14.874 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:40:14.874 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:40:14.874 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:40:14.875 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:40:14.875 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:40:14.875 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:40:14.875 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:40:14.900 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:40:14.900 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:40:14.901 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:40:14.901 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:40:15.033 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:40:15.034 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:40:15.034 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:40:15.034 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:40:15.034 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:40:15.034 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:40:15.034 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:40:15.034 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:40:15.034 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:40:15.035 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:40:15.035 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:40:15.036 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:40:15.036 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:40:15.036 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:40:15.037 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:40:15.037 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:40:15.082 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 20 2024-04-06T09:40:15.083 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 23 2024-04-06T09:40:15.083 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:40:15.083 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=93, tag_tid=15, entry_tid=1], [object_number=92, tag_tid=15, entry_tid=0], [object_number=95, tag_tid=14, entry_tid=19039], [object_number=94, tag_tid=14, entry_tid=19038]]], state=connected] 2024-04-06T09:40:15.083 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=83, tag_tid=14, entry_tid=7679], [object_number=82, tag_tid=14, entry_tid=7678], [object_number=81, tag_tid=14, entry_tid=7677], [object_number=80, tag_tid=14, entry_tid=7676]]], state=connected] 2024-04-06T09:40:15.086 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:40:15.089 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=93, tag_tid=15, entry_tid=1' 2024-04-06T09:40:15.089 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:40:15.089 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:40:15.089 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:40:15.089 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:40:15.089 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:40:15.090 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:40:15.090 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:40:15.090 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:40:15.091 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:40:15.091 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:40:15.091 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:40:15.091 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:40:15.092 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:40:15.092 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:40:15.092 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:40:15.135 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 20 2024-04-06T09:40:15.135 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 23 2024-04-06T09:40:15.135 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:40:15.135 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=93, tag_tid=15, entry_tid=1], [object_number=92, tag_tid=15, entry_tid=0], [object_number=95, tag_tid=14, entry_tid=19039], [object_number=94, tag_tid=14, entry_tid=19038]]], state=connected] 2024-04-06T09:40:15.135 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=83, tag_tid=14, entry_tid=7679], [object_number=82, tag_tid=14, entry_tid=7678], [object_number=81, tag_tid=14, entry_tid=7677], [object_number=80, tag_tid=14, entry_tid=7676]]], state=connected] 2024-04-06T09:40:15.138 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:40:15.140 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=83, tag_tid=14, entry_tid=7679' 2024-04-06T09:40:15.140 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=93, tag_tid=15, entry_tid=1' -a 'object_number=93, tag_tid=15, entry_tid=1' = 'object_number=83, tag_tid=14, entry_tid=7679' 2024-04-06T09:40:15.140 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=83, tag_tid=14, entry_tid=7679' '!=' 'object_number=83, tag_tid=14, entry_tid=7559' 2024-04-06T09:40:15.140 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:40:15.140 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=83, tag_tid=14, entry_tid=7679' = 'object_number=83, tag_tid=14, entry_tid=7559' 2024-04-06T09:40:15.141 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=83, tag_tid=14, entry_tid=7679' 2024-04-06T09:40:15.141 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=93, tag_tid=15, entry_tid=1' 2024-04-06T09:40:15.141 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:40:15.141 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:40:15.141 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=15 2024-04-06T09:40:15.142 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=83, tag_tid=14, entry_tid=7679' 2024-04-06T09:40:15.142 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:40:15.143 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:40:15.143 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=14 2024-04-06T09:40:15.144 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=93, tag_tid=15, entry_tid=1' 2024-04-06T09:40:15.145 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:40:15.145 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:40:15.145 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:40:15.146 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=83, tag_tid=14, entry_tid=7679' 2024-04-06T09:40:15.146 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:40:15.146 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:40:15.147 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=7679 2024-04-06T09:40:15.147 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 15 = 14 -a 1 -le 7679 2024-04-06T09:40:15.147 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:40:15.148 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:40:15.148 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:40:15.349 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:40:15.349 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:40:15.349 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:40:15.349 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:40:15.349 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:40:15.349 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:40:15.349 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:40:15.349 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:40:15.349 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:40:15.349 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:40:15.350 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:40:15.350 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:40:15.350 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:40:15.350 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:40:15.350 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:40:15.350 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:40:15.351 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:40:15.353 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:40:15.353 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:40:15.353 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:40:15.353 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:40:15.353 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:40:15.353 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:40:15.353 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:40:15.354 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:40:15.354 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:40:15.354 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:40:15.354 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:40:15.354 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:40:15.354 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:40:15.354 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:40:15.354 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:40:15.354 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:40:15.354 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:40:15.354 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:40:15.354 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:40:15.354 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:40:15.354 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:40:15.355 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:40:15.355 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:40:15.355 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:40:15.355 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:40:15.355 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:40:15.355 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:40:15.355 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:40:15.355 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:40:15.355 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:40:15.355 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:40:15.356 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:40:15.379 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:40:15.380 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:40:15.380 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:40:15.380 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:40:15.516 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:40:15.518 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:40:15.518 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:40:15.518 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:40:15.518 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:40:15.518 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:40:15.518 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:40:15.518 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:40:15.518 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:40:15.518 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:40:15.519 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:40:15.519 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:40:15.519 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:40:15.521 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:40:15.521 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:40:15.521 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:40:15.572 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 20 2024-04-06T09:40:15.572 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 23 2024-04-06T09:40:15.572 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:40:15.572 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=93, tag_tid=15, entry_tid=1], [object_number=92, tag_tid=15, entry_tid=0], [object_number=95, tag_tid=14, entry_tid=19039], [object_number=94, tag_tid=14, entry_tid=19038]]], state=connected] 2024-04-06T09:40:15.572 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=83, tag_tid=14, entry_tid=7771], [object_number=82, tag_tid=14, entry_tid=7770], [object_number=81, tag_tid=14, entry_tid=7769], [object_number=80, tag_tid=14, entry_tid=7768]]], state=connected] 2024-04-06T09:40:15.576 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:40:15.577 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=93, tag_tid=15, entry_tid=1' 2024-04-06T09:40:15.578 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:40:15.578 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:40:15.578 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:40:15.578 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:40:15.578 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:40:15.579 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:40:15.579 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:40:15.579 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:40:15.579 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:40:15.579 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:40:15.579 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:40:15.579 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:40:15.580 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:40:15.580 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:40:15.580 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:40:15.836 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 20 2024-04-06T09:40:15.836 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 23 2024-04-06T09:40:15.836 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:40:15.836 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=93, tag_tid=15, entry_tid=1], [object_number=92, tag_tid=15, entry_tid=0], [object_number=95, tag_tid=14, entry_tid=19039], [object_number=94, tag_tid=14, entry_tid=19038]]], state=connected] 2024-04-06T09:40:15.836 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=83, tag_tid=14, entry_tid=7771], [object_number=82, tag_tid=14, entry_tid=7770], [object_number=81, tag_tid=14, entry_tid=7769], [object_number=80, tag_tid=14, entry_tid=7768]]], state=connected] 2024-04-06T09:40:15.841 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:40:15.843 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=83, tag_tid=14, entry_tid=7771' 2024-04-06T09:40:15.843 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=93, tag_tid=15, entry_tid=1' -a 'object_number=93, tag_tid=15, entry_tid=1' = 'object_number=83, tag_tid=14, entry_tid=7771' 2024-04-06T09:40:15.843 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=83, tag_tid=14, entry_tid=7771' '!=' 'object_number=83, tag_tid=14, entry_tid=7679' 2024-04-06T09:40:15.843 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:40:15.843 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=83, tag_tid=14, entry_tid=7771' = 'object_number=83, tag_tid=14, entry_tid=7679' 2024-04-06T09:40:15.843 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=83, tag_tid=14, entry_tid=7771' 2024-04-06T09:40:15.844 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=93, tag_tid=15, entry_tid=1' 2024-04-06T09:40:15.844 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:40:15.844 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:40:15.846 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=15 2024-04-06T09:40:15.847 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=83, tag_tid=14, entry_tid=7771' 2024-04-06T09:40:15.847 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:40:15.847 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:40:15.848 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=14 2024-04-06T09:40:15.849 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=93, tag_tid=15, entry_tid=1' 2024-04-06T09:40:15.849 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:40:15.849 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:40:15.850 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:40:15.851 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=83, tag_tid=14, entry_tid=7771' 2024-04-06T09:40:15.851 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:40:15.852 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:40:15.853 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=7771 2024-04-06T09:40:15.853 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 15 = 14 -a 1 -le 7771 2024-04-06T09:40:15.853 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:40:15.853 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:40:15.853 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:40:16.055 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:40:16.055 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:40:16.055 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:40:16.055 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:40:16.055 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:40:16.055 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:40:16.055 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:40:16.056 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:40:16.056 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:40:16.056 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:40:16.056 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:40:16.056 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:40:16.056 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:40:16.056 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:40:16.056 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:40:16.056 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:40:16.057 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:40:16.059 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:40:16.059 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:40:16.059 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:40:16.059 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:40:16.059 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:40:16.059 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:40:16.059 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:40:16.059 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:40:16.060 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:40:16.060 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:40:16.060 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:40:16.060 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:40:16.060 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:40:16.060 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:40:16.060 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:40:16.060 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:40:16.061 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:40:16.061 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:40:16.061 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:40:16.061 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:40:16.061 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:40:16.061 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:40:16.061 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:40:16.061 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:40:16.061 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:40:16.061 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:40:16.062 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:40:16.062 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:40:16.062 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:40:16.062 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:40:16.062 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:40:16.062 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:40:16.087 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:40:16.087 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:40:16.087 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:40:16.087 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:40:16.216 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:40:16.217 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:40:16.217 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:40:16.217 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:40:16.217 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:40:16.217 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:40:16.217 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:40:16.217 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:40:16.217 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:40:16.217 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:40:16.218 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:40:16.218 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:40:16.218 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:40:16.220 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:40:16.220 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:40:16.220 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:40:16.266 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 20 2024-04-06T09:40:16.266 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 23 2024-04-06T09:40:16.266 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:40:16.266 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=93, tag_tid=15, entry_tid=1], [object_number=92, tag_tid=15, entry_tid=0], [object_number=95, tag_tid=14, entry_tid=19039], [object_number=94, tag_tid=14, entry_tid=19038]]], state=connected] 2024-04-06T09:40:16.266 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=83, tag_tid=14, entry_tid=7903], [object_number=82, tag_tid=14, entry_tid=7902], [object_number=81, tag_tid=14, entry_tid=7901], [object_number=80, tag_tid=14, entry_tid=7900]]], state=connected] 2024-04-06T09:40:16.269 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:40:16.271 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=93, tag_tid=15, entry_tid=1' 2024-04-06T09:40:16.272 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:40:16.272 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:40:16.272 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:40:16.272 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:40:16.272 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:40:16.272 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:40:16.272 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:40:16.272 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:40:16.272 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:40:16.273 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:40:16.273 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:40:16.273 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:40:16.273 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:40:16.274 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:40:16.274 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:40:16.386 INFO:tasks.rbd_mirror_thrash:kill cluster2.client.mirror.0 2024-04-06T09:40:16.386 INFO:tasks.rbd_mirror.cluster2.client.mirror.0:Sent signal 15 2024-04-06T09:40:16.386 INFO:tasks.rbd_mirror_thrash:waiting for 3 secs before reviving daemons 2024-04-06T09:40:16.387 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi192.stderr:2024-04-06T09:40:16.391+0000 7f7424d81640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.0 (PID: 58905) UID: 1000 2024-04-06T09:40:16.530 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 20 2024-04-06T09:40:16.530 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 23 2024-04-06T09:40:16.530 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:40:16.530 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=93, tag_tid=15, entry_tid=1], [object_number=92, tag_tid=15, entry_tid=0], [object_number=95, tag_tid=14, entry_tid=19039], [object_number=94, tag_tid=14, entry_tid=19038]]], state=connected] 2024-04-06T09:40:16.531 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=83, tag_tid=14, entry_tid=7903], [object_number=82, tag_tid=14, entry_tid=7902], [object_number=81, tag_tid=14, entry_tid=7901], [object_number=80, tag_tid=14, entry_tid=7900]]], state=connected] 2024-04-06T09:40:16.535 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:40:16.537 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=83, tag_tid=14, entry_tid=7903' 2024-04-06T09:40:16.537 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=93, tag_tid=15, entry_tid=1' -a 'object_number=93, tag_tid=15, entry_tid=1' = 'object_number=83, tag_tid=14, entry_tid=7903' 2024-04-06T09:40:16.537 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=83, tag_tid=14, entry_tid=7903' '!=' 'object_number=83, tag_tid=14, entry_tid=7771' 2024-04-06T09:40:16.537 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:40:16.537 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=83, tag_tid=14, entry_tid=7903' = 'object_number=83, tag_tid=14, entry_tid=7771' 2024-04-06T09:40:16.537 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=83, tag_tid=14, entry_tid=7903' 2024-04-06T09:40:16.538 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=93, tag_tid=15, entry_tid=1' 2024-04-06T09:40:16.538 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:40:16.538 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:40:16.539 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=15 2024-04-06T09:40:16.539 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=83, tag_tid=14, entry_tid=7903' 2024-04-06T09:40:16.540 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:40:16.540 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:40:16.541 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=14 2024-04-06T09:40:16.541 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=93, tag_tid=15, entry_tid=1' 2024-04-06T09:40:16.542 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:40:16.542 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:40:16.543 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:40:16.544 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=83, tag_tid=14, entry_tid=7903' 2024-04-06T09:40:16.544 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:40:16.544 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:40:16.545 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=7903 2024-04-06T09:40:16.545 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 15 = 14 -a 1 -le 7903 2024-04-06T09:40:16.546 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:40:16.546 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:40:16.546 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:40:16.748 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:40:16.748 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:40:16.748 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:40:16.748 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:40:16.748 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:40:16.748 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:40:16.748 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:40:16.748 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:40:16.748 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:40:16.749 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:40:16.749 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:40:16.749 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:40:16.749 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:40:16.749 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:40:16.749 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:40:16.749 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:40:16.750 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:40:16.752 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:40:16.752 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:40:16.752 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:40:16.752 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:40:16.752 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:40:16.752 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:40:16.752 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:40:16.752 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:40:16.752 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:40:16.752 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:40:16.752 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:40:16.752 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:40:16.752 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:40:16.752 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:40:16.753 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:40:16.753 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:40:16.753 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:40:16.753 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:40:16.753 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:40:16.753 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:40:16.753 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:40:16.753 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:40:16.753 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:40:16.753 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:40:16.753 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:40:16.753 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:40:16.753 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:40:16.753 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:40:16.753 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:40:16.753 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:40:16.754 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:40:16.754 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:40:16.775 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:40:16.772+0000 7f2e84d66640 -1 reset not still connected to 0x560d27073930 2024-04-06T09:40:16.775 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:40:16.772+0000 7f2e84d66640 -1 reset not still connected to 0x560d27073ba0 2024-04-06T09:40:16.775 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:40:16.772+0000 7f2e84d66640 -1 reset not still connected to 0x560d27073d40 2024-04-06T09:40:16.779 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:40:16.779 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:40:16.779 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:40:16.780 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:40:16.911 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:40:16.912 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:40:16.912 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:40:16.912 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:40:16.912 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:40:16.912 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:40:16.912 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:40:16.913 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:40:16.913 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:40:16.913 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:40:16.913 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:40:16.913 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:40:16.914 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:40:16.916 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:40:16.916 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:40:16.917 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:40:16.964 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 20 2024-04-06T09:40:16.964 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 23 2024-04-06T09:40:16.964 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:40:16.964 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=93, tag_tid=15, entry_tid=1], [object_number=92, tag_tid=15, entry_tid=0], [object_number=95, tag_tid=14, entry_tid=19039], [object_number=94, tag_tid=14, entry_tid=19038]]], state=connected] 2024-04-06T09:40:16.964 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=80, tag_tid=14, entry_tid=8060], [object_number=83, tag_tid=14, entry_tid=8059], [object_number=82, tag_tid=14, entry_tid=8058], [object_number=81, tag_tid=14, entry_tid=8057]]], state=connected] 2024-04-06T09:40:16.967 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:40:16.969 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=93, tag_tid=15, entry_tid=1' 2024-04-06T09:40:16.970 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:40:16.970 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:40:16.970 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:40:16.970 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:40:16.970 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:40:16.970 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:40:16.970 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:40:16.970 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:40:16.971 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:40:16.971 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:40:16.971 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:40:16.971 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:40:16.971 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:40:16.972 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:40:16.972 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:40:17.017 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 20 2024-04-06T09:40:17.017 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 23 2024-04-06T09:40:17.017 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:40:17.017 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=93, tag_tid=15, entry_tid=1], [object_number=92, tag_tid=15, entry_tid=0], [object_number=95, tag_tid=14, entry_tid=19039], [object_number=94, tag_tid=14, entry_tid=19038]]], state=connected] 2024-04-06T09:40:17.017 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=80, tag_tid=14, entry_tid=8060], [object_number=83, tag_tid=14, entry_tid=8059], [object_number=82, tag_tid=14, entry_tid=8058], [object_number=81, tag_tid=14, entry_tid=8057]]], state=connected] 2024-04-06T09:40:17.020 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:40:17.025 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=80, tag_tid=14, entry_tid=8060' 2024-04-06T09:40:17.026 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=93, tag_tid=15, entry_tid=1' -a 'object_number=93, tag_tid=15, entry_tid=1' = 'object_number=80, tag_tid=14, entry_tid=8060' 2024-04-06T09:40:17.026 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=80, tag_tid=14, entry_tid=8060' '!=' 'object_number=83, tag_tid=14, entry_tid=7903' 2024-04-06T09:40:17.026 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:40:17.026 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=80, tag_tid=14, entry_tid=8060' = 'object_number=83, tag_tid=14, entry_tid=7903' 2024-04-06T09:40:17.026 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=80, tag_tid=14, entry_tid=8060' 2024-04-06T09:40:17.026 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=93, tag_tid=15, entry_tid=1' 2024-04-06T09:40:17.026 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:40:17.026 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:40:17.028 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=15 2024-04-06T09:40:17.028 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=80, tag_tid=14, entry_tid=8060' 2024-04-06T09:40:17.028 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:40:17.028 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:40:17.028 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=14 2024-04-06T09:40:17.028 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=93, tag_tid=15, entry_tid=1' 2024-04-06T09:40:17.028 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:40:17.028 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:40:17.028 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:40:17.028 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=80, tag_tid=14, entry_tid=8060' 2024-04-06T09:40:17.028 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:40:17.028 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:40:17.029 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=8060 2024-04-06T09:40:17.029 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 15 = 14 -a 1 -le 8060 2024-04-06T09:40:17.029 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:40:17.029 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:40:17.029 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:40:17.231 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:40:17.232 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:40:17.232 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:40:17.232 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:40:17.232 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:40:17.232 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:40:17.232 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:40:17.232 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:40:17.232 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:40:17.232 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:40:17.232 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:40:17.232 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:40:17.233 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:40:17.233 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:40:17.233 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:40:17.233 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:40:17.233 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:40:17.236 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:40:17.236 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:40:17.236 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:40:17.236 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:40:17.236 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:40:17.236 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:40:17.236 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:40:17.236 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:40:17.236 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:40:17.236 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:40:17.237 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:40:17.237 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:40:17.237 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:40:17.237 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:40:17.237 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:40:17.237 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:40:17.237 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:40:17.237 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:40:17.237 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:40:17.237 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:40:17.237 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:40:17.237 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:40:17.237 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:40:17.237 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:40:17.237 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:40:17.238 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:40:17.238 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:40:17.238 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:40:17.238 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:40:17.238 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:40:17.238 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:40:17.238 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:40:17.261 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:40:17.262 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:40:17.262 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:40:17.262 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:40:17.389 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:40:17.390 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:40:17.390 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:40:17.391 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:40:17.391 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:40:17.391 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:40:17.391 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:40:17.391 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:40:17.391 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:40:17.391 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:40:17.391 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:40:17.391 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:40:17.391 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:40:17.393 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:40:17.393 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:40:17.393 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:40:17.438 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 20 2024-04-06T09:40:17.439 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 23 2024-04-06T09:40:17.439 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:40:17.439 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=93, tag_tid=15, entry_tid=1], [object_number=92, tag_tid=15, entry_tid=0], [object_number=95, tag_tid=14, entry_tid=19039], [object_number=94, tag_tid=14, entry_tid=19038]]], state=connected] 2024-04-06T09:40:17.439 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=80, tag_tid=14, entry_tid=8164], [object_number=83, tag_tid=14, entry_tid=8163], [object_number=82, tag_tid=14, entry_tid=8162], [object_number=81, tag_tid=14, entry_tid=8161]]], state=connected] 2024-04-06T09:40:17.442 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:40:17.443 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=93, tag_tid=15, entry_tid=1' 2024-04-06T09:40:17.444 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:40:17.445 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:40:17.445 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:40:17.445 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:40:17.445 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:40:17.445 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:40:17.445 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:40:17.445 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:40:17.445 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:40:17.445 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:40:17.445 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:40:17.445 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:40:17.446 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:40:17.446 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:40:17.446 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:40:17.491 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 20 2024-04-06T09:40:17.491 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 23 2024-04-06T09:40:17.491 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:40:17.491 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=93, tag_tid=15, entry_tid=1], [object_number=92, tag_tid=15, entry_tid=0], [object_number=95, tag_tid=14, entry_tid=19039], [object_number=94, tag_tid=14, entry_tid=19038]]], state=connected] 2024-04-06T09:40:17.492 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=80, tag_tid=14, entry_tid=8164], [object_number=83, tag_tid=14, entry_tid=8163], [object_number=82, tag_tid=14, entry_tid=8162], [object_number=81, tag_tid=14, entry_tid=8161]]], state=connected] 2024-04-06T09:40:17.495 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:40:17.497 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=80, tag_tid=14, entry_tid=8164' 2024-04-06T09:40:17.497 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=93, tag_tid=15, entry_tid=1' -a 'object_number=93, tag_tid=15, entry_tid=1' = 'object_number=80, tag_tid=14, entry_tid=8164' 2024-04-06T09:40:17.497 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=80, tag_tid=14, entry_tid=8164' '!=' 'object_number=80, tag_tid=14, entry_tid=8060' 2024-04-06T09:40:17.497 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:40:17.497 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=80, tag_tid=14, entry_tid=8164' = 'object_number=80, tag_tid=14, entry_tid=8060' 2024-04-06T09:40:17.497 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=80, tag_tid=14, entry_tid=8164' 2024-04-06T09:40:17.497 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=93, tag_tid=15, entry_tid=1' 2024-04-06T09:40:17.497 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:40:17.497 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:40:17.498 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=15 2024-04-06T09:40:17.499 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=80, tag_tid=14, entry_tid=8164' 2024-04-06T09:40:17.499 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:40:17.499 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:40:17.500 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=14 2024-04-06T09:40:17.500 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=93, tag_tid=15, entry_tid=1' 2024-04-06T09:40:17.501 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:40:17.501 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:40:17.502 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:40:17.502 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=80, tag_tid=14, entry_tid=8164' 2024-04-06T09:40:17.502 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:40:17.502 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:40:17.504 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=8164 2024-04-06T09:40:17.504 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 15 = 14 -a 1 -le 8164 2024-04-06T09:40:17.504 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:40:17.504 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:40:17.504 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:40:17.705 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:40:17.705 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:40:17.705 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:40:17.706 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:40:17.706 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:40:17.706 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:40:17.706 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:40:17.706 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:40:17.706 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:40:17.706 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:40:17.706 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:40:17.706 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:40:17.706 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:40:17.706 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:40:17.706 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:40:17.707 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:40:17.707 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:40:17.709 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:40:17.709 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:40:17.709 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:40:17.709 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:40:17.709 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:40:17.709 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:40:17.710 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:40:17.710 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:40:17.710 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:40:17.710 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:40:17.710 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:40:17.710 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:40:17.710 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:40:17.710 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:40:17.710 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:40:17.710 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:40:17.711 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:40:17.711 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:40:17.712 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:40:17.712 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:40:17.712 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:40:17.712 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:40:17.712 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:40:17.712 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:40:17.712 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:40:17.712 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:40:17.712 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:40:17.712 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:40:17.712 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:40:17.712 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:40:17.712 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:40:17.712 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:40:17.735 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:40:17.736 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:40:17.736 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:40:17.736 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:40:17.864 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:40:17.865 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:40:17.865 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:40:17.865 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:40:17.866 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:40:17.866 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:40:17.866 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:40:17.866 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:40:17.866 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:40:17.866 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:40:17.866 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:40:17.867 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:40:17.867 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:40:17.868 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:40:17.868 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:40:17.868 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:40:17.920 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 20 2024-04-06T09:40:17.920 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 23 2024-04-06T09:40:17.920 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:40:17.920 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=93, tag_tid=15, entry_tid=1], [object_number=92, tag_tid=15, entry_tid=0], [object_number=95, tag_tid=14, entry_tid=19039], [object_number=94, tag_tid=14, entry_tid=19038]]], state=connected] 2024-04-06T09:40:17.921 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=80, tag_tid=14, entry_tid=8256], [object_number=83, tag_tid=14, entry_tid=8255], [object_number=82, tag_tid=14, entry_tid=8254], [object_number=81, tag_tid=14, entry_tid=8253]]], state=connected] 2024-04-06T09:40:17.924 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:40:17.925 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=93, tag_tid=15, entry_tid=1' 2024-04-06T09:40:17.926 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:40:17.927 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:40:17.927 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:40:17.927 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:40:17.927 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:40:17.927 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:40:17.927 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:40:17.927 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:40:17.927 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:40:17.927 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:40:17.927 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:40:17.927 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:40:17.928 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:40:17.928 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:40:17.928 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:40:17.976 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 20 2024-04-06T09:40:17.976 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 23 2024-04-06T09:40:17.976 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:40:17.976 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=93, tag_tid=15, entry_tid=1], [object_number=92, tag_tid=15, entry_tid=0], [object_number=95, tag_tid=14, entry_tid=19039], [object_number=94, tag_tid=14, entry_tid=19038]]], state=connected] 2024-04-06T09:40:17.976 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=80, tag_tid=14, entry_tid=8256], [object_number=83, tag_tid=14, entry_tid=8255], [object_number=82, tag_tid=14, entry_tid=8254], [object_number=81, tag_tid=14, entry_tid=8253]]], state=connected] 2024-04-06T09:40:17.979 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:40:17.981 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=80, tag_tid=14, entry_tid=8256' 2024-04-06T09:40:17.981 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=93, tag_tid=15, entry_tid=1' -a 'object_number=93, tag_tid=15, entry_tid=1' = 'object_number=80, tag_tid=14, entry_tid=8256' 2024-04-06T09:40:17.981 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=80, tag_tid=14, entry_tid=8256' '!=' 'object_number=80, tag_tid=14, entry_tid=8164' 2024-04-06T09:40:17.981 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:40:17.981 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=80, tag_tid=14, entry_tid=8256' = 'object_number=80, tag_tid=14, entry_tid=8164' 2024-04-06T09:40:17.981 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=80, tag_tid=14, entry_tid=8256' 2024-04-06T09:40:17.982 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=93, tag_tid=15, entry_tid=1' 2024-04-06T09:40:17.982 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:40:17.982 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:40:17.983 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=15 2024-04-06T09:40:17.983 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=80, tag_tid=14, entry_tid=8256' 2024-04-06T09:40:17.983 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:40:17.984 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:40:17.984 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=14 2024-04-06T09:40:17.985 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=93, tag_tid=15, entry_tid=1' 2024-04-06T09:40:17.985 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:40:17.986 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:40:17.986 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:40:17.987 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=80, tag_tid=14, entry_tid=8256' 2024-04-06T09:40:17.987 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:40:17.987 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:40:17.988 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=8256 2024-04-06T09:40:17.989 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 15 = 14 -a 1 -le 8256 2024-04-06T09:40:17.989 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:40:17.989 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:40:17.989 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:40:18.191 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:40:18.191 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:40:18.191 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:40:18.191 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:40:18.191 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:40:18.191 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:40:18.191 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:40:18.191 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:40:18.191 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:40:18.191 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:40:18.192 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:40:18.192 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:40:18.192 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:40:18.192 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:40:18.192 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:40:18.192 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:40:18.192 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:40:18.194 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:40:18.194 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:40:18.194 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:40:18.194 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:40:18.195 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:40:18.195 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:40:18.195 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:40:18.195 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:40:18.195 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:40:18.195 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:40:18.195 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:40:18.195 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:40:18.195 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:40:18.195 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:40:18.195 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:40:18.195 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:40:18.195 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:40:18.195 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:40:18.195 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:40:18.196 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:40:18.196 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:40:18.196 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:40:18.196 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:40:18.196 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:40:18.196 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:40:18.196 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:40:18.196 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:40:18.196 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:40:18.196 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:40:18.196 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:40:18.196 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:40:18.197 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:40:18.220 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:40:18.220 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:40:18.221 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:40:18.221 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:40:18.351 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:40:18.352 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:40:18.352 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:40:18.352 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:40:18.353 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:40:18.353 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:40:18.353 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:40:18.353 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:40:18.353 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:40:18.353 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:40:18.353 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:40:18.354 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:40:18.354 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:40:18.355 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:40:18.356 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:40:18.356 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:40:18.407 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 20 2024-04-06T09:40:18.408 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 23 2024-04-06T09:40:18.408 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:40:18.408 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=93, tag_tid=15, entry_tid=1], [object_number=92, tag_tid=15, entry_tid=0], [object_number=95, tag_tid=14, entry_tid=19039], [object_number=94, tag_tid=14, entry_tid=19038]]], state=connected] 2024-04-06T09:40:18.408 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=83, tag_tid=14, entry_tid=8347], [object_number=82, tag_tid=14, entry_tid=8346], [object_number=81, tag_tid=14, entry_tid=8345], [object_number=80, tag_tid=14, entry_tid=8344]]], state=connected] 2024-04-06T09:40:18.411 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:40:18.412 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=93, tag_tid=15, entry_tid=1' 2024-04-06T09:40:18.414 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:40:18.414 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:40:18.414 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:40:18.414 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:40:18.414 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:40:18.414 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:40:18.414 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:40:18.414 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:40:18.414 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:40:18.414 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:40:18.414 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:40:18.414 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:40:18.415 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:40:18.415 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:40:18.415 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:40:18.462 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 20 2024-04-06T09:40:18.462 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 23 2024-04-06T09:40:18.462 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:40:18.462 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=93, tag_tid=15, entry_tid=1], [object_number=92, tag_tid=15, entry_tid=0], [object_number=95, tag_tid=14, entry_tid=19039], [object_number=94, tag_tid=14, entry_tid=19038]]], state=connected] 2024-04-06T09:40:18.462 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=83, tag_tid=14, entry_tid=8347], [object_number=82, tag_tid=14, entry_tid=8346], [object_number=81, tag_tid=14, entry_tid=8345], [object_number=80, tag_tid=14, entry_tid=8344]]], state=connected] 2024-04-06T09:40:18.466 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:40:18.468 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=83, tag_tid=14, entry_tid=8347' 2024-04-06T09:40:18.468 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=93, tag_tid=15, entry_tid=1' -a 'object_number=93, tag_tid=15, entry_tid=1' = 'object_number=83, tag_tid=14, entry_tid=8347' 2024-04-06T09:40:18.468 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=83, tag_tid=14, entry_tid=8347' '!=' 'object_number=80, tag_tid=14, entry_tid=8256' 2024-04-06T09:40:18.468 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:40:18.468 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=83, tag_tid=14, entry_tid=8347' = 'object_number=80, tag_tid=14, entry_tid=8256' 2024-04-06T09:40:18.468 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=83, tag_tid=14, entry_tid=8347' 2024-04-06T09:40:18.468 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=93, tag_tid=15, entry_tid=1' 2024-04-06T09:40:18.468 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:40:18.468 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:40:18.469 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=15 2024-04-06T09:40:18.470 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=83, tag_tid=14, entry_tid=8347' 2024-04-06T09:40:18.470 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:40:18.470 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:40:18.471 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=14 2024-04-06T09:40:18.471 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=93, tag_tid=15, entry_tid=1' 2024-04-06T09:40:18.471 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:40:18.471 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:40:18.472 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:40:18.473 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=83, tag_tid=14, entry_tid=8347' 2024-04-06T09:40:18.473 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:40:18.473 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:40:18.474 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=8347 2024-04-06T09:40:18.474 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 15 = 14 -a 1 -le 8347 2024-04-06T09:40:18.475 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:40:18.475 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:40:18.475 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:40:18.676 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:40:18.677 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:40:18.677 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:40:18.677 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:40:18.677 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:40:18.677 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:40:18.677 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:40:18.677 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:40:18.677 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:40:18.677 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:40:18.677 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:40:18.677 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:40:18.677 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:40:18.677 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:40:18.678 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:40:18.679 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:40:18.679 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:40:18.679 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:40:18.679 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:40:18.679 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:40:18.679 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:40:18.679 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:40:18.679 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:40:18.679 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:40:18.680 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:40:18.680 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:40:18.680 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:40:18.680 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:40:18.680 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:40:18.680 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:40:18.680 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:40:18.680 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:40:18.680 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:40:18.681 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:40:18.681 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:40:18.682 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:40:18.682 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:40:18.682 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:40:18.682 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:40:18.682 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:40:18.682 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:40:18.682 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:40:18.682 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:40:18.682 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:40:18.682 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:40:18.682 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:40:18.682 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:40:18.682 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:40:18.682 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:40:18.705 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:40:18.706 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:40:18.706 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:40:18.707 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:40:18.833 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:40:18.835 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:40:18.835 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:40:18.835 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:40:18.835 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:40:18.835 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:40:18.835 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:40:18.835 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:40:18.835 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:40:18.835 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:40:18.836 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:40:18.836 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:40:18.836 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:40:18.837 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:40:18.837 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:40:18.837 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:40:18.888 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 20 2024-04-06T09:40:18.889 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 23 2024-04-06T09:40:18.889 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:40:18.889 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=93, tag_tid=15, entry_tid=1], [object_number=92, tag_tid=15, entry_tid=0], [object_number=95, tag_tid=14, entry_tid=19039], [object_number=94, tag_tid=14, entry_tid=19038]]], state=connected] 2024-04-06T09:40:18.889 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=85, tag_tid=14, entry_tid=8413], [object_number=80, tag_tid=14, entry_tid=8412], [object_number=83, tag_tid=14, entry_tid=8411], [object_number=82, tag_tid=14, entry_tid=8410]]], state=connected] 2024-04-06T09:40:18.893 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:40:18.894 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=93, tag_tid=15, entry_tid=1' 2024-04-06T09:40:18.895 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:40:18.895 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:40:18.896 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:40:18.896 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:40:18.896 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:40:18.896 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:40:18.896 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:40:18.896 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:40:18.896 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:40:18.896 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:40:18.896 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:40:18.896 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:40:18.897 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:40:18.897 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:40:18.897 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:40:18.945 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 20 2024-04-06T09:40:18.945 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 23 2024-04-06T09:40:18.945 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:40:18.945 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=93, tag_tid=15, entry_tid=1], [object_number=92, tag_tid=15, entry_tid=0], [object_number=95, tag_tid=14, entry_tid=19039], [object_number=94, tag_tid=14, entry_tid=19038]]], state=connected] 2024-04-06T09:40:18.945 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=85, tag_tid=14, entry_tid=8413], [object_number=80, tag_tid=14, entry_tid=8412], [object_number=83, tag_tid=14, entry_tid=8411], [object_number=82, tag_tid=14, entry_tid=8410]]], state=connected] 2024-04-06T09:40:18.948 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:40:18.950 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=85, tag_tid=14, entry_tid=8413' 2024-04-06T09:40:18.950 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=93, tag_tid=15, entry_tid=1' -a 'object_number=93, tag_tid=15, entry_tid=1' = 'object_number=85, tag_tid=14, entry_tid=8413' 2024-04-06T09:40:18.950 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=85, tag_tid=14, entry_tid=8413' '!=' 'object_number=83, tag_tid=14, entry_tid=8347' 2024-04-06T09:40:18.950 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:40:18.951 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=85, tag_tid=14, entry_tid=8413' = 'object_number=83, tag_tid=14, entry_tid=8347' 2024-04-06T09:40:18.951 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=85, tag_tid=14, entry_tid=8413' 2024-04-06T09:40:18.951 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=93, tag_tid=15, entry_tid=1' 2024-04-06T09:40:18.951 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:40:18.951 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:40:18.952 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=15 2024-04-06T09:40:18.952 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=85, tag_tid=14, entry_tid=8413' 2024-04-06T09:40:18.953 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:40:18.953 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:40:18.954 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=14 2024-04-06T09:40:18.954 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=93, tag_tid=15, entry_tid=1' 2024-04-06T09:40:18.954 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:40:18.955 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:40:18.955 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:40:18.956 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=85, tag_tid=14, entry_tid=8413' 2024-04-06T09:40:18.956 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:40:18.956 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:40:18.958 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=8413 2024-04-06T09:40:18.958 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 15 = 14 -a 1 -le 8413 2024-04-06T09:40:18.958 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:40:18.958 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:40:18.958 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:40:19.159 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:40:19.160 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:40:19.160 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:40:19.160 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:40:19.160 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:40:19.160 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:40:19.160 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:40:19.160 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:40:19.160 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:40:19.160 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:40:19.160 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:40:19.160 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:40:19.160 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:40:19.160 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:40:19.161 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:40:19.161 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:40:19.161 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:40:19.165 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:40:19.165 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:40:19.165 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:40:19.165 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:40:19.165 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:40:19.165 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:40:19.165 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:40:19.165 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:40:19.165 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:40:19.165 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:40:19.165 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:40:19.166 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:40:19.166 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:40:19.166 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:40:19.166 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:40:19.166 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:40:19.166 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:40:19.166 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:40:19.166 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:40:19.166 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:40:19.166 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:40:19.166 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:40:19.166 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:40:19.166 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:40:19.166 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:40:19.167 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:40:19.167 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:40:19.167 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:40:19.167 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:40:19.167 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:40:19.167 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:40:19.167 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:40:19.191 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:40:19.192 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:40:19.192 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:40:19.192 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:40:19.327 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:40:19.329 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:40:19.329 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:40:19.329 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:40:19.329 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:40:19.329 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:40:19.329 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:40:19.329 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:40:19.329 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:40:19.329 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:40:19.330 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:40:19.330 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:40:19.330 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:40:19.331 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:40:19.332 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:40:19.332 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:40:19.387 INFO:tasks.rbd_mirror_thrash:waiting for cluster2.client.mirror.0 2024-04-06T09:40:19.388 INFO:teuthology.orchestra.run:waiting for 600 2024-04-06T09:40:19.388 INFO:tasks.rbd_mirror_thrash:reviving cluster2.client.mirror.0 2024-04-06T09:40:19.388 INFO:tasks.rbd_mirror.cluster2.client.mirror.0:Restarting daemon 2024-04-06T09:40:19.388 DEBUG:teuthology.orchestra.run.smithi192:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.0 2024-04-06T09:40:19.390 INFO:tasks.rbd_mirror.cluster2.client.mirror.0:Started 2024-04-06T09:40:19.391 INFO:tasks.rbd_mirror_thrash:waiting for 108 secs before thrashing 2024-04-06T09:40:19.391 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 21 2024-04-06T09:40:19.392 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 23 2024-04-06T09:40:19.392 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:40:19.392 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=93, tag_tid=15, entry_tid=1], [object_number=92, tag_tid=15, entry_tid=0], [object_number=95, tag_tid=14, entry_tid=19039], [object_number=94, tag_tid=14, entry_tid=19038]]], state=connected] 2024-04-06T09:40:19.392 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=86, tag_tid=14, entry_tid=8466], [object_number=85, tag_tid=14, entry_tid=8465], [object_number=84, tag_tid=14, entry_tid=8464], [object_number=87, tag_tid=14, entry_tid=8463]]], state=connected] 2024-04-06T09:40:19.395 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:40:19.396 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=93, tag_tid=15, entry_tid=1' 2024-04-06T09:40:19.398 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:40:19.398 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:40:19.398 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:40:19.398 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:40:19.398 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:40:19.398 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:40:19.398 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:40:19.398 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:40:19.398 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:40:19.399 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:40:19.399 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:40:19.399 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:40:19.399 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:40:19.399 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:40:19.399 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:40:19.448 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 21 2024-04-06T09:40:19.448 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 23 2024-04-06T09:40:19.448 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:40:19.448 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=93, tag_tid=15, entry_tid=1], [object_number=92, tag_tid=15, entry_tid=0], [object_number=95, tag_tid=14, entry_tid=19039], [object_number=94, tag_tid=14, entry_tid=19038]]], state=connected] 2024-04-06T09:40:19.448 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=86, tag_tid=14, entry_tid=8466], [object_number=85, tag_tid=14, entry_tid=8465], [object_number=84, tag_tid=14, entry_tid=8464], [object_number=87, tag_tid=14, entry_tid=8463]]], state=connected] 2024-04-06T09:40:19.452 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:40:19.454 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=86, tag_tid=14, entry_tid=8466' 2024-04-06T09:40:19.454 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=93, tag_tid=15, entry_tid=1' -a 'object_number=93, tag_tid=15, entry_tid=1' = 'object_number=86, tag_tid=14, entry_tid=8466' 2024-04-06T09:40:19.454 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=86, tag_tid=14, entry_tid=8466' '!=' 'object_number=85, tag_tid=14, entry_tid=8413' 2024-04-06T09:40:19.454 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:40:19.454 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=86, tag_tid=14, entry_tid=8466' = 'object_number=85, tag_tid=14, entry_tid=8413' 2024-04-06T09:40:19.455 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=86, tag_tid=14, entry_tid=8466' 2024-04-06T09:40:19.455 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=93, tag_tid=15, entry_tid=1' 2024-04-06T09:40:19.455 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:40:19.455 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:40:19.455 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=15 2024-04-06T09:40:19.456 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=86, tag_tid=14, entry_tid=8466' 2024-04-06T09:40:19.456 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:40:19.456 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:40:19.457 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=14 2024-04-06T09:40:19.457 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=93, tag_tid=15, entry_tid=1' 2024-04-06T09:40:19.457 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:40:19.458 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:40:19.458 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:40:19.459 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=86, tag_tid=14, entry_tid=8466' 2024-04-06T09:40:19.459 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:40:19.459 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:40:19.460 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=8466 2024-04-06T09:40:19.460 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 15 = 14 -a 1 -le 8466 2024-04-06T09:40:19.461 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:40:19.461 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:40:19.461 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:40:19.662 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:40:19.662 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:40:19.662 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:40:19.663 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:40:19.663 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:40:19.663 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:40:19.663 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:40:19.663 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:40:19.663 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:40:19.663 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:40:19.663 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:40:19.663 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:40:19.663 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:40:19.663 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:40:19.663 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:40:19.664 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:40:19.664 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:40:19.666 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:40:19.666 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:40:19.666 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:40:19.666 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:40:19.666 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:40:19.666 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:40:19.666 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:40:19.666 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:40:19.667 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:40:19.667 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:40:19.667 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:40:19.667 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:40:19.667 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:40:19.667 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:40:19.667 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:40:19.667 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:40:19.667 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:40:19.667 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:40:19.667 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:40:19.667 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:40:19.667 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:40:19.667 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:40:19.667 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:40:19.668 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:40:19.668 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:40:19.668 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:40:19.668 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:40:19.668 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:40:19.668 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:40:19.668 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:40:19.668 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:40:19.668 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:40:19.687 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:40:19.687 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:40:19.687 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:40:19.687 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:40:19.814 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:40:19.815 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:40:19.815 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:40:19.816 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:40:19.816 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:40:19.816 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:40:19.816 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:40:19.816 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:40:19.816 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:40:19.816 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:40:19.816 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:40:19.817 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:40:19.817 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:40:19.819 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:40:19.819 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:40:19.819 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:40:19.867 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 21 2024-04-06T09:40:19.867 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 23 2024-04-06T09:40:19.867 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:40:19.867 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=93, tag_tid=15, entry_tid=1], [object_number=92, tag_tid=15, entry_tid=0], [object_number=95, tag_tid=14, entry_tid=19039], [object_number=94, tag_tid=14, entry_tid=19038]]], state=connected] 2024-04-06T09:40:19.868 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=84, tag_tid=14, entry_tid=8552], [object_number=87, tag_tid=14, entry_tid=8551], [object_number=86, tag_tid=14, entry_tid=8550], [object_number=85, tag_tid=14, entry_tid=8549]]], state=connected] 2024-04-06T09:40:19.871 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:40:19.872 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=93, tag_tid=15, entry_tid=1' 2024-04-06T09:40:19.873 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:40:19.873 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:40:19.874 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:40:19.874 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:40:19.874 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:40:19.874 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:40:19.874 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:40:19.874 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:40:19.874 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:40:19.874 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:40:19.874 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:40:19.874 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:40:19.875 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:40:19.875 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:40:19.875 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:40:20.126 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 21 2024-04-06T09:40:20.126 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 23 2024-04-06T09:40:20.126 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:40:20.126 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=93, tag_tid=15, entry_tid=1], [object_number=92, tag_tid=15, entry_tid=0], [object_number=95, tag_tid=14, entry_tid=19039], [object_number=94, tag_tid=14, entry_tid=19038]]], state=connected] 2024-04-06T09:40:20.126 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=84, tag_tid=14, entry_tid=8552], [object_number=87, tag_tid=14, entry_tid=8551], [object_number=86, tag_tid=14, entry_tid=8550], [object_number=85, tag_tid=14, entry_tid=8549]]], state=connected] 2024-04-06T09:40:20.132 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:40:20.135 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=84, tag_tid=14, entry_tid=8552' 2024-04-06T09:40:20.135 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=93, tag_tid=15, entry_tid=1' -a 'object_number=93, tag_tid=15, entry_tid=1' = 'object_number=84, tag_tid=14, entry_tid=8552' 2024-04-06T09:40:20.135 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=84, tag_tid=14, entry_tid=8552' '!=' 'object_number=86, tag_tid=14, entry_tid=8466' 2024-04-06T09:40:20.135 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:40:20.135 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=84, tag_tid=14, entry_tid=8552' = 'object_number=86, tag_tid=14, entry_tid=8466' 2024-04-06T09:40:20.135 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=84, tag_tid=14, entry_tid=8552' 2024-04-06T09:40:20.135 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=93, tag_tid=15, entry_tid=1' 2024-04-06T09:40:20.136 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:40:20.136 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:40:20.138 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=15 2024-04-06T09:40:20.139 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=84, tag_tid=14, entry_tid=8552' 2024-04-06T09:40:20.139 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:40:20.139 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:40:20.140 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=14 2024-04-06T09:40:20.141 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=93, tag_tid=15, entry_tid=1' 2024-04-06T09:40:20.141 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:40:20.142 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:40:20.143 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:40:20.144 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=84, tag_tid=14, entry_tid=8552' 2024-04-06T09:40:20.144 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:40:20.144 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:40:20.146 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=8552 2024-04-06T09:40:20.146 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 15 = 14 -a 1 -le 8552 2024-04-06T09:40:20.146 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:40:20.146 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:40:20.147 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:40:20.348 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:40:20.348 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:40:20.348 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:40:20.348 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:40:20.348 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:40:20.348 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:40:20.348 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:40:20.349 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:40:20.349 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:40:20.349 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:40:20.349 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:40:20.349 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:40:20.349 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:40:20.349 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:40:20.349 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:40:20.349 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:40:20.350 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:40:20.352 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:40:20.352 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:40:20.352 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:40:20.352 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:40:20.352 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:40:20.353 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:40:20.353 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:40:20.353 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:40:20.353 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:40:20.353 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:40:20.353 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:40:20.353 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:40:20.353 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:40:20.353 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:40:20.353 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:40:20.353 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:40:20.355 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:40:20.355 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:40:20.355 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:40:20.355 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:40:20.355 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:40:20.355 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:40:20.355 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:40:20.355 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:40:20.355 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:40:20.355 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:40:20.355 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:40:20.355 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:40:20.355 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:40:20.355 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:40:20.356 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:40:20.356 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:40:20.379 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:40:20.379 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:40:20.379 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:40:20.379 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:40:20.511 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:40:20.512 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:40:20.512 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:40:20.512 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:40:20.512 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:40:20.512 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:40:20.513 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:40:20.513 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:40:20.513 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:40:20.513 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:40:20.513 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:40:20.513 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:40:20.513 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:40:20.515 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:40:20.516 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:40:20.516 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:40:20.568 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 21 2024-04-06T09:40:20.568 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 23 2024-04-06T09:40:20.568 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:40:20.568 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=93, tag_tid=15, entry_tid=1], [object_number=92, tag_tid=15, entry_tid=0], [object_number=95, tag_tid=14, entry_tid=19039], [object_number=94, tag_tid=14, entry_tid=19038]]], state=connected] 2024-04-06T09:40:20.568 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=84, tag_tid=14, entry_tid=8664], [object_number=87, tag_tid=14, entry_tid=8663], [object_number=86, tag_tid=14, entry_tid=8662], [object_number=85, tag_tid=14, entry_tid=8661]]], state=connected] 2024-04-06T09:40:20.571 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:40:20.574 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=93, tag_tid=15, entry_tid=1' 2024-04-06T09:40:20.574 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:40:20.574 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:40:20.574 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:40:20.574 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:40:20.574 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:40:20.574 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:40:20.574 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:40:20.574 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:40:20.574 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:40:20.574 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:40:20.574 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:40:20.574 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:40:20.575 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:40:20.575 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:40:20.575 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:40:20.620 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 21 2024-04-06T09:40:20.620 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 23 2024-04-06T09:40:20.620 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:40:20.620 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=93, tag_tid=15, entry_tid=1], [object_number=92, tag_tid=15, entry_tid=0], [object_number=95, tag_tid=14, entry_tid=19039], [object_number=94, tag_tid=14, entry_tid=19038]]], state=connected] 2024-04-06T09:40:20.620 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=84, tag_tid=14, entry_tid=8664], [object_number=87, tag_tid=14, entry_tid=8663], [object_number=86, tag_tid=14, entry_tid=8662], [object_number=85, tag_tid=14, entry_tid=8661]]], state=connected] 2024-04-06T09:40:20.623 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:40:20.626 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=84, tag_tid=14, entry_tid=8664' 2024-04-06T09:40:20.626 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=93, tag_tid=15, entry_tid=1' -a 'object_number=93, tag_tid=15, entry_tid=1' = 'object_number=84, tag_tid=14, entry_tid=8664' 2024-04-06T09:40:20.626 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=84, tag_tid=14, entry_tid=8664' '!=' 'object_number=84, tag_tid=14, entry_tid=8552' 2024-04-06T09:40:20.626 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:40:20.626 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=84, tag_tid=14, entry_tid=8664' = 'object_number=84, tag_tid=14, entry_tid=8552' 2024-04-06T09:40:20.626 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=84, tag_tid=14, entry_tid=8664' 2024-04-06T09:40:20.626 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=93, tag_tid=15, entry_tid=1' 2024-04-06T09:40:20.626 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:40:20.626 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:40:20.627 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=15 2024-04-06T09:40:20.628 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=84, tag_tid=14, entry_tid=8664' 2024-04-06T09:40:20.628 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:40:20.628 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:40:20.628 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=14 2024-04-06T09:40:20.629 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=93, tag_tid=15, entry_tid=1' 2024-04-06T09:40:20.629 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:40:20.629 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:40:20.630 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:40:20.630 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=84, tag_tid=14, entry_tid=8664' 2024-04-06T09:40:20.630 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:40:20.631 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:40:20.632 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=8664 2024-04-06T09:40:20.632 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 15 = 14 -a 1 -le 8664 2024-04-06T09:40:20.632 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:40:20.632 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:40:20.632 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:40:20.834 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:40:20.834 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:40:20.834 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:40:20.834 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:40:20.834 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:40:20.834 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:40:20.835 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:40:20.835 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:40:20.835 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:40:20.835 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:40:20.835 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:40:20.835 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:40:20.835 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:40:20.835 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:40:20.835 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:40:20.836 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:40:20.836 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:40:20.838 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:40:20.838 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:40:20.838 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:40:20.838 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:40:20.838 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:40:20.838 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:40:20.839 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:40:20.839 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:40:20.839 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:40:20.839 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:40:20.839 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:40:20.839 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:40:20.839 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:40:20.839 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:40:20.839 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:40:20.839 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:40:20.840 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:40:20.840 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:40:20.840 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:40:20.840 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:40:20.840 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:40:20.840 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:40:20.840 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:40:20.841 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:40:20.841 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:40:20.841 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:40:20.841 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:40:20.841 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:40:20.841 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:40:20.841 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:40:20.841 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:40:20.841 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:40:20.864 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:40:20.865 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:40:20.865 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:40:20.865 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:40:20.995 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:40:20.996 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:40:20.996 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:40:20.996 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:40:20.996 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:40:20.996 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:40:20.996 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:40:20.996 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:40:20.996 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:40:20.997 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:40:20.997 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:40:20.997 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:40:20.997 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:40:20.999 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:40:21.000 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:40:21.000 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:40:21.052 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 21 2024-04-06T09:40:21.052 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 23 2024-04-06T09:40:21.053 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:40:21.053 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=93, tag_tid=15, entry_tid=1], [object_number=92, tag_tid=15, entry_tid=0], [object_number=95, tag_tid=14, entry_tid=19039], [object_number=94, tag_tid=14, entry_tid=19038]]], state=connected] 2024-04-06T09:40:21.053 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=86, tag_tid=14, entry_tid=8738], [object_number=85, tag_tid=14, entry_tid=8737], [object_number=84, tag_tid=14, entry_tid=8736], [object_number=87, tag_tid=14, entry_tid=8735]]], state=connected] 2024-04-06T09:40:21.056 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:40:21.057 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=93, tag_tid=15, entry_tid=1' 2024-04-06T09:40:21.059 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:40:21.059 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:40:21.059 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:40:21.059 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:40:21.059 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:40:21.059 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:40:21.059 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:40:21.060 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:40:21.060 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:40:21.060 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:40:21.060 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:40:21.060 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:40:21.061 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:40:21.061 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:40:21.061 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:40:21.089 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:40:21.084+0000 7f2e84d66640 -1 reset not still connected to 0x560d2655fee0 2024-04-06T09:40:21.089 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:40:21.084+0000 7f2e84d66640 -1 reset not still connected to 0x560d27073930 2024-04-06T09:40:21.091 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:40:21.088+0000 7f2e84d66640 -1 reset not still connected to 0x560d27073ba0 2024-04-06T09:40:21.091 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:40:21.088+0000 7f2e84d66640 -1 reset not still connected to 0x560d27073d40 2024-04-06T09:40:21.111 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 21 2024-04-06T09:40:21.111 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 23 2024-04-06T09:40:21.111 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:40:21.111 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=93, tag_tid=15, entry_tid=1], [object_number=92, tag_tid=15, entry_tid=0], [object_number=95, tag_tid=14, entry_tid=19039], [object_number=94, tag_tid=14, entry_tid=19038]]], state=connected] 2024-04-06T09:40:21.111 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=86, tag_tid=14, entry_tid=8738], [object_number=85, tag_tid=14, entry_tid=8737], [object_number=84, tag_tid=14, entry_tid=8736], [object_number=87, tag_tid=14, entry_tid=8735]]], state=connected] 2024-04-06T09:40:21.111 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:40:21.112 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=86, tag_tid=14, entry_tid=8738' 2024-04-06T09:40:21.112 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=93, tag_tid=15, entry_tid=1' -a 'object_number=93, tag_tid=15, entry_tid=1' = 'object_number=86, tag_tid=14, entry_tid=8738' 2024-04-06T09:40:21.113 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=86, tag_tid=14, entry_tid=8738' '!=' 'object_number=84, tag_tid=14, entry_tid=8664' 2024-04-06T09:40:21.113 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:40:21.113 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=86, tag_tid=14, entry_tid=8738' = 'object_number=84, tag_tid=14, entry_tid=8664' 2024-04-06T09:40:21.113 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=86, tag_tid=14, entry_tid=8738' 2024-04-06T09:40:21.113 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=93, tag_tid=15, entry_tid=1' 2024-04-06T09:40:21.113 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:40:21.113 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:40:21.113 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=15 2024-04-06T09:40:21.114 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=86, tag_tid=14, entry_tid=8738' 2024-04-06T09:40:21.114 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:40:21.114 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:40:21.115 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=14 2024-04-06T09:40:21.116 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=93, tag_tid=15, entry_tid=1' 2024-04-06T09:40:21.116 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:40:21.116 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:40:21.117 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:40:21.118 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=86, tag_tid=14, entry_tid=8738' 2024-04-06T09:40:21.118 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:40:21.118 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:40:21.119 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=8738 2024-04-06T09:40:21.119 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 15 = 14 -a 1 -le 8738 2024-04-06T09:40:21.119 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:40:21.119 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:40:21.119 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:40:21.321 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:40:21.321 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:40:21.321 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:40:21.321 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:40:21.321 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:40:21.321 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:40:21.321 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:40:21.321 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:40:21.322 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:40:21.322 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:40:21.322 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:40:21.322 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:40:21.322 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:40:21.322 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:40:21.322 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:40:21.322 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:40:21.323 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:40:21.325 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:40:21.325 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:40:21.325 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:40:21.325 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:40:21.325 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:40:21.325 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:40:21.325 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:40:21.325 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:40:21.325 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:40:21.326 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:40:21.326 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:40:21.326 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:40:21.326 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:40:21.326 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:40:21.326 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:40:21.326 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:40:21.327 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:40:21.327 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:40:21.327 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:40:21.327 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:40:21.327 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:40:21.327 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:40:21.327 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:40:21.327 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:40:21.328 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:40:21.328 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:40:21.328 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:40:21.328 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:40:21.328 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:40:21.328 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:40:21.328 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:40:21.328 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:40:21.351 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:40:21.352 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:40:21.352 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:40:21.352 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:40:21.482 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:40:21.483 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:40:21.483 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:40:21.483 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:40:21.483 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:40:21.483 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:40:21.483 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:40:21.483 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:40:21.483 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:40:21.484 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:40:21.484 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:40:21.484 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:40:21.484 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:40:21.486 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:40:21.487 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:40:21.487 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:40:21.538 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 21 2024-04-06T09:40:21.538 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 23 2024-04-06T09:40:21.539 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:40:21.539 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=93, tag_tid=15, entry_tid=1], [object_number=92, tag_tid=15, entry_tid=0], [object_number=95, tag_tid=14, entry_tid=19039], [object_number=94, tag_tid=14, entry_tid=19038]]], state=connected] 2024-04-06T09:40:21.539 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=86, tag_tid=14, entry_tid=8826], [object_number=85, tag_tid=14, entry_tid=8825], [object_number=84, tag_tid=14, entry_tid=8824], [object_number=87, tag_tid=14, entry_tid=8823]]], state=connected] 2024-04-06T09:40:21.542 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:40:21.543 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=93, tag_tid=15, entry_tid=1' 2024-04-06T09:40:21.545 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:40:21.545 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:40:21.545 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:40:21.545 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:40:21.545 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:40:21.545 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:40:21.545 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:40:21.545 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:40:21.545 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:40:21.545 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:40:21.545 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:40:21.545 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:40:21.547 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:40:21.547 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:40:21.547 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:40:21.593 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 21 2024-04-06T09:40:21.593 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 23 2024-04-06T09:40:21.593 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:40:21.593 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=93, tag_tid=15, entry_tid=1], [object_number=92, tag_tid=15, entry_tid=0], [object_number=95, tag_tid=14, entry_tid=19039], [object_number=94, tag_tid=14, entry_tid=19038]]], state=connected] 2024-04-06T09:40:21.594 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=86, tag_tid=14, entry_tid=8826], [object_number=85, tag_tid=14, entry_tid=8825], [object_number=84, tag_tid=14, entry_tid=8824], [object_number=87, tag_tid=14, entry_tid=8823]]], state=connected] 2024-04-06T09:40:21.596 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:40:21.599 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=86, tag_tid=14, entry_tid=8826' 2024-04-06T09:40:21.599 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=93, tag_tid=15, entry_tid=1' -a 'object_number=93, tag_tid=15, entry_tid=1' = 'object_number=86, tag_tid=14, entry_tid=8826' 2024-04-06T09:40:21.599 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=86, tag_tid=14, entry_tid=8826' '!=' 'object_number=86, tag_tid=14, entry_tid=8738' 2024-04-06T09:40:21.599 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:40:21.599 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=86, tag_tid=14, entry_tid=8826' = 'object_number=86, tag_tid=14, entry_tid=8738' 2024-04-06T09:40:21.599 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=86, tag_tid=14, entry_tid=8826' 2024-04-06T09:40:21.599 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=93, tag_tid=15, entry_tid=1' 2024-04-06T09:40:21.599 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:40:21.599 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:40:21.600 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=15 2024-04-06T09:40:21.600 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=86, tag_tid=14, entry_tid=8826' 2024-04-06T09:40:21.600 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:40:21.601 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:40:21.601 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=14 2024-04-06T09:40:21.602 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=93, tag_tid=15, entry_tid=1' 2024-04-06T09:40:21.602 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:40:21.602 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:40:21.603 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:40:21.604 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=86, tag_tid=14, entry_tid=8826' 2024-04-06T09:40:21.604 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:40:21.604 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:40:21.605 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=8826 2024-04-06T09:40:21.605 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 15 = 14 -a 1 -le 8826 2024-04-06T09:40:21.606 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:40:21.606 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:40:21.606 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:40:21.807 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:40:21.808 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:40:21.808 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:40:21.808 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:40:21.808 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:40:21.808 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:40:21.808 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:40:21.808 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:40:21.808 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:40:21.808 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:40:21.808 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:40:21.809 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:40:21.809 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:40:21.809 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:40:21.809 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:40:21.809 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:40:21.810 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:40:21.811 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:40:21.811 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:40:21.811 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:40:21.812 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:40:21.812 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:40:21.812 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:40:21.812 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:40:21.812 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:40:21.812 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:40:21.812 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:40:21.812 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:40:21.812 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:40:21.812 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:40:21.812 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:40:21.812 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:40:21.813 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:40:21.813 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:40:21.814 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:40:21.814 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:40:21.814 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:40:21.814 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:40:21.814 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:40:21.814 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:40:21.814 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:40:21.814 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:40:21.814 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:40:21.814 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:40:21.814 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:40:21.814 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:40:21.815 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:40:21.815 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:40:21.815 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:40:21.838 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:40:21.839 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:40:21.839 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:40:21.839 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:40:21.976 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:40:21.977 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:40:21.977 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:40:21.978 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:40:21.978 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:40:21.978 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:40:21.978 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:40:21.978 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:40:21.978 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:40:21.978 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:40:21.979 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:40:21.979 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:40:21.979 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:40:21.980 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:40:21.980 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:40:21.981 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:40:22.027 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 21 2024-04-06T09:40:22.027 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 23 2024-04-06T09:40:22.027 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:40:22.027 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=93, tag_tid=15, entry_tid=1], [object_number=92, tag_tid=15, entry_tid=0], [object_number=95, tag_tid=14, entry_tid=19039], [object_number=94, tag_tid=14, entry_tid=19038]]], state=connected] 2024-04-06T09:40:22.027 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=86, tag_tid=14, entry_tid=8914], [object_number=85, tag_tid=14, entry_tid=8913], [object_number=84, tag_tid=14, entry_tid=8912], [object_number=87, tag_tid=14, entry_tid=8911]]], state=connected] 2024-04-06T09:40:22.030 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:40:22.032 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=93, tag_tid=15, entry_tid=1' 2024-04-06T09:40:22.033 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:40:22.033 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:40:22.033 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:40:22.033 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:40:22.033 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:40:22.033 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:40:22.033 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:40:22.033 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:40:22.033 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:40:22.033 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:40:22.033 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:40:22.033 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:40:22.034 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:40:22.034 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:40:22.034 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:40:22.079 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 21 2024-04-06T09:40:22.079 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 23 2024-04-06T09:40:22.080 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:40:22.080 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=93, tag_tid=15, entry_tid=1], [object_number=92, tag_tid=15, entry_tid=0], [object_number=95, tag_tid=14, entry_tid=19039], [object_number=94, tag_tid=14, entry_tid=19038]]], state=connected] 2024-04-06T09:40:22.080 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=86, tag_tid=14, entry_tid=8914], [object_number=85, tag_tid=14, entry_tid=8913], [object_number=84, tag_tid=14, entry_tid=8912], [object_number=87, tag_tid=14, entry_tid=8911]]], state=connected] 2024-04-06T09:40:22.083 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:40:22.085 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=86, tag_tid=14, entry_tid=8914' 2024-04-06T09:40:22.085 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=93, tag_tid=15, entry_tid=1' -a 'object_number=93, tag_tid=15, entry_tid=1' = 'object_number=86, tag_tid=14, entry_tid=8914' 2024-04-06T09:40:22.086 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=86, tag_tid=14, entry_tid=8914' '!=' 'object_number=86, tag_tid=14, entry_tid=8826' 2024-04-06T09:40:22.086 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:40:22.086 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=86, tag_tid=14, entry_tid=8914' = 'object_number=86, tag_tid=14, entry_tid=8826' 2024-04-06T09:40:22.086 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=86, tag_tid=14, entry_tid=8914' 2024-04-06T09:40:22.086 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=93, tag_tid=15, entry_tid=1' 2024-04-06T09:40:22.086 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:40:22.086 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:40:22.086 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=15 2024-04-06T09:40:22.087 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=86, tag_tid=14, entry_tid=8914' 2024-04-06T09:40:22.087 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:40:22.087 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:40:22.088 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=14 2024-04-06T09:40:22.089 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=93, tag_tid=15, entry_tid=1' 2024-04-06T09:40:22.089 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:40:22.089 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:40:22.090 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:40:22.091 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=86, tag_tid=14, entry_tid=8914' 2024-04-06T09:40:22.091 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:40:22.091 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:40:22.092 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=8914 2024-04-06T09:40:22.092 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 15 = 14 -a 1 -le 8914 2024-04-06T09:40:22.092 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:40:22.092 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:40:22.092 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:40:22.294 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:40:22.294 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:40:22.295 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:40:22.295 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:40:22.295 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:40:22.295 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:40:22.295 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:40:22.295 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:40:22.295 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:40:22.295 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:40:22.295 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:40:22.295 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:40:22.295 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:40:22.295 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:40:22.295 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:40:22.296 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:40:22.296 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:40:22.299 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:40:22.299 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:40:22.299 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:40:22.299 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:40:22.299 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:40:22.299 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:40:22.299 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:40:22.299 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:40:22.299 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:40:22.299 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:40:22.299 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:40:22.299 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:40:22.300 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:40:22.300 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:40:22.300 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:40:22.300 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:40:22.300 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:40:22.300 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:40:22.300 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:40:22.300 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:40:22.300 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:40:22.300 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:40:22.300 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:40:22.300 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:40:22.300 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:40:22.300 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:40:22.300 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:40:22.301 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:40:22.301 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:40:22.301 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:40:22.301 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:40:22.301 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:40:22.324 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:40:22.324 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:40:22.324 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:40:22.324 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:40:22.459 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:40:22.460 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:40:22.460 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:40:22.460 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:40:22.460 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:40:22.460 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:40:22.460 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:40:22.461 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:40:22.461 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:40:22.461 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:40:22.461 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:40:22.461 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:40:22.461 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:40:22.463 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:40:22.463 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:40:22.463 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:40:22.512 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 21 2024-04-06T09:40:22.512 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 23 2024-04-06T09:40:22.512 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:40:22.512 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=93, tag_tid=15, entry_tid=1], [object_number=92, tag_tid=15, entry_tid=0], [object_number=95, tag_tid=14, entry_tid=19039], [object_number=94, tag_tid=14, entry_tid=19038]]], state=connected] 2024-04-06T09:40:22.512 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=84, tag_tid=14, entry_tid=9000], [object_number=87, tag_tid=14, entry_tid=8999], [object_number=86, tag_tid=14, entry_tid=8998], [object_number=85, tag_tid=14, entry_tid=8997]]], state=connected] 2024-04-06T09:40:22.516 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:40:22.517 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=93, tag_tid=15, entry_tid=1' 2024-04-06T09:40:22.519 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:40:22.519 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:40:22.519 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:40:22.519 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:40:22.519 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:40:22.519 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:40:22.519 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:40:22.519 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:40:22.520 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:40:22.520 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:40:22.520 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:40:22.520 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:40:22.521 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:40:22.521 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:40:22.521 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:40:22.565 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 21 2024-04-06T09:40:22.565 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 23 2024-04-06T09:40:22.565 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:40:22.565 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=93, tag_tid=15, entry_tid=1], [object_number=92, tag_tid=15, entry_tid=0], [object_number=95, tag_tid=14, entry_tid=19039], [object_number=94, tag_tid=14, entry_tid=19038]]], state=connected] 2024-04-06T09:40:22.565 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=84, tag_tid=14, entry_tid=9000], [object_number=87, tag_tid=14, entry_tid=8999], [object_number=86, tag_tid=14, entry_tid=8998], [object_number=85, tag_tid=14, entry_tid=8997]]], state=connected] 2024-04-06T09:40:22.568 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:40:22.571 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=84, tag_tid=14, entry_tid=9000' 2024-04-06T09:40:22.571 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=93, tag_tid=15, entry_tid=1' -a 'object_number=93, tag_tid=15, entry_tid=1' = 'object_number=84, tag_tid=14, entry_tid=9000' 2024-04-06T09:40:22.571 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=84, tag_tid=14, entry_tid=9000' '!=' 'object_number=86, tag_tid=14, entry_tid=8914' 2024-04-06T09:40:22.571 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:40:22.571 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=84, tag_tid=14, entry_tid=9000' = 'object_number=86, tag_tid=14, entry_tid=8914' 2024-04-06T09:40:22.571 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=84, tag_tid=14, entry_tid=9000' 2024-04-06T09:40:22.571 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=93, tag_tid=15, entry_tid=1' 2024-04-06T09:40:22.571 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:40:22.571 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:40:22.572 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=15 2024-04-06T09:40:22.572 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=84, tag_tid=14, entry_tid=9000' 2024-04-06T09:40:22.573 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:40:22.573 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:40:22.574 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=14 2024-04-06T09:40:22.574 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=93, tag_tid=15, entry_tid=1' 2024-04-06T09:40:22.574 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:40:22.574 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:40:22.575 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:40:22.576 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=84, tag_tid=14, entry_tid=9000' 2024-04-06T09:40:22.576 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:40:22.576 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:40:22.577 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=9000 2024-04-06T09:40:22.577 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 15 = 14 -a 1 -le 9000 2024-04-06T09:40:22.577 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:40:22.577 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:40:22.577 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:40:22.779 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:40:22.779 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:40:22.779 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:40:22.779 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:40:22.779 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:40:22.779 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:40:22.779 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:40:22.779 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:40:22.779 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:40:22.779 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:40:22.780 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:40:22.780 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:40:22.780 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:40:22.780 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:40:22.780 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:40:22.780 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:40:22.780 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:40:22.784 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:40:22.784 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:40:22.785 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:40:22.785 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:40:22.785 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:40:22.785 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:40:22.785 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:40:22.785 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:40:22.785 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:40:22.785 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:40:22.785 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:40:22.785 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:40:22.785 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:40:22.785 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:40:22.785 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:40:22.786 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:40:22.786 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:40:22.786 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:40:22.786 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:40:22.786 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:40:22.786 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:40:22.786 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:40:22.786 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:40:22.786 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:40:22.786 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:40:22.786 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:40:22.786 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:40:22.786 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:40:22.786 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:40:22.786 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:40:22.786 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:40:22.786 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:40:22.811 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:40:22.811 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:40:22.811 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:40:22.811 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:40:22.944 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:40:22.945 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:40:22.946 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:40:22.946 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:40:22.946 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:40:22.946 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:40:22.946 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:40:22.946 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:40:22.946 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:40:22.946 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:40:22.947 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:40:22.947 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:40:22.947 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:40:22.948 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:40:22.949 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:40:22.949 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:40:22.995 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 21 2024-04-06T09:40:22.995 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 23 2024-04-06T09:40:22.995 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:40:22.995 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=93, tag_tid=15, entry_tid=1], [object_number=92, tag_tid=15, entry_tid=0], [object_number=95, tag_tid=14, entry_tid=19039], [object_number=94, tag_tid=14, entry_tid=19038]]], state=connected] 2024-04-06T09:40:22.996 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=87, tag_tid=14, entry_tid=9091], [object_number=86, tag_tid=14, entry_tid=9090], [object_number=85, tag_tid=14, entry_tid=9089], [object_number=84, tag_tid=14, entry_tid=9088]]], state=connected] 2024-04-06T09:40:22.998 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:40:23.000 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=93, tag_tid=15, entry_tid=1' 2024-04-06T09:40:23.001 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:40:23.001 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:40:23.001 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:40:23.001 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:40:23.001 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:40:23.001 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:40:23.001 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:40:23.001 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:40:23.002 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:40:23.002 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:40:23.002 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:40:23.002 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:40:23.002 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:40:23.002 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:40:23.003 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:40:23.049 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 21 2024-04-06T09:40:23.049 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 23 2024-04-06T09:40:23.049 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:40:23.049 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=93, tag_tid=15, entry_tid=1], [object_number=92, tag_tid=15, entry_tid=0], [object_number=95, tag_tid=14, entry_tid=19039], [object_number=94, tag_tid=14, entry_tid=19038]]], state=connected] 2024-04-06T09:40:23.049 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=87, tag_tid=14, entry_tid=9091], [object_number=86, tag_tid=14, entry_tid=9090], [object_number=85, tag_tid=14, entry_tid=9089], [object_number=84, tag_tid=14, entry_tid=9088]]], state=connected] 2024-04-06T09:40:23.053 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:40:23.055 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=87, tag_tid=14, entry_tid=9091' 2024-04-06T09:40:23.055 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=93, tag_tid=15, entry_tid=1' -a 'object_number=93, tag_tid=15, entry_tid=1' = 'object_number=87, tag_tid=14, entry_tid=9091' 2024-04-06T09:40:23.055 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=87, tag_tid=14, entry_tid=9091' '!=' 'object_number=84, tag_tid=14, entry_tid=9000' 2024-04-06T09:40:23.055 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:40:23.055 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=87, tag_tid=14, entry_tid=9091' = 'object_number=84, tag_tid=14, entry_tid=9000' 2024-04-06T09:40:23.055 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=87, tag_tid=14, entry_tid=9091' 2024-04-06T09:40:23.055 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=93, tag_tid=15, entry_tid=1' 2024-04-06T09:40:23.055 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:40:23.056 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:40:23.056 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=15 2024-04-06T09:40:23.057 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=87, tag_tid=14, entry_tid=9091' 2024-04-06T09:40:23.057 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:40:23.057 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:40:23.058 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=14 2024-04-06T09:40:23.058 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=93, tag_tid=15, entry_tid=1' 2024-04-06T09:40:23.058 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:40:23.059 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:40:23.059 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:40:23.060 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=87, tag_tid=14, entry_tid=9091' 2024-04-06T09:40:23.060 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:40:23.060 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:40:23.061 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=9091 2024-04-06T09:40:23.062 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 15 = 14 -a 1 -le 9091 2024-04-06T09:40:23.062 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:40:23.062 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:40:23.062 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:40:23.264 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:40:23.311 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:40:23.311 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:40:23.311 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:40:23.311 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:40:23.311 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:40:23.311 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:40:23.311 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:40:23.311 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:40:23.311 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:40:23.311 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:40:23.311 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:40:23.312 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:40:23.312 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:40:23.312 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:40:23.312 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:40:23.321 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:40:23.321 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:40:23.321 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:40:23.321 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:40:23.321 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:40:23.321 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:40:23.321 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:40:23.321 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:40:23.321 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:40:23.322 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:40:23.322 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:40:23.322 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:40:23.322 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:40:23.322 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:40:23.322 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:40:23.322 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:40:23.322 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:40:23.323 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:40:23.323 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:40:23.323 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:40:23.323 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:40:23.323 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:40:23.323 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:40:23.323 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:40:23.324 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:40:23.324 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:40:23.324 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:40:23.324 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:40:23.324 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:40:23.324 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:40:23.324 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:40:23.324 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:40:23.325 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:40:23.325 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:40:23.325 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:40:23.325 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:40:23.325 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:40:23.424 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:40:23.425 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:40:23.425 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:40:23.425 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:40:23.425 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:40:23.425 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:40:23.425 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:40:23.425 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:40:23.425 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:40:23.426 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:40:23.427 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:40:23.427 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:40:23.427 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:40:23.428 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:40:23.428 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:40:23.428 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:40:23.476 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 21 2024-04-06T09:40:23.476 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 23 2024-04-06T09:40:23.476 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:40:23.476 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=93, tag_tid=15, entry_tid=1], [object_number=92, tag_tid=15, entry_tid=0], [object_number=95, tag_tid=14, entry_tid=19039], [object_number=94, tag_tid=14, entry_tid=19038]]], state=connected] 2024-04-06T09:40:23.476 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=86, tag_tid=14, entry_tid=9182], [object_number=85, tag_tid=14, entry_tid=9181], [object_number=84, tag_tid=14, entry_tid=9180], [object_number=87, tag_tid=14, entry_tid=9179]]], state=connected] 2024-04-06T09:40:23.480 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:40:23.481 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=93, tag_tid=15, entry_tid=1' 2024-04-06T09:40:23.482 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:40:23.482 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:40:23.482 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:40:23.482 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:40:23.482 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:40:23.482 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:40:23.482 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:40:23.482 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:40:23.482 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:40:23.483 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:40:23.483 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:40:23.483 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:40:23.483 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:40:23.484 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:40:23.484 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:40:23.734 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 21 2024-04-06T09:40:23.734 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 23 2024-04-06T09:40:23.734 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:40:23.734 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=93, tag_tid=15, entry_tid=1], [object_number=92, tag_tid=15, entry_tid=0], [object_number=95, tag_tid=14, entry_tid=19039], [object_number=94, tag_tid=14, entry_tid=19038]]], state=connected] 2024-04-06T09:40:23.734 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=86, tag_tid=14, entry_tid=9182], [object_number=85, tag_tid=14, entry_tid=9181], [object_number=84, tag_tid=14, entry_tid=9180], [object_number=87, tag_tid=14, entry_tid=9179]]], state=connected] 2024-04-06T09:40:23.740 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:40:23.743 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=86, tag_tid=14, entry_tid=9182' 2024-04-06T09:40:23.743 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=93, tag_tid=15, entry_tid=1' -a 'object_number=93, tag_tid=15, entry_tid=1' = 'object_number=86, tag_tid=14, entry_tid=9182' 2024-04-06T09:40:23.743 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=86, tag_tid=14, entry_tid=9182' '!=' 'object_number=87, tag_tid=14, entry_tid=9091' 2024-04-06T09:40:23.743 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:40:23.743 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=86, tag_tid=14, entry_tid=9182' = 'object_number=87, tag_tid=14, entry_tid=9091' 2024-04-06T09:40:23.743 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=86, tag_tid=14, entry_tid=9182' 2024-04-06T09:40:23.744 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=93, tag_tid=15, entry_tid=1' 2024-04-06T09:40:23.744 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:40:23.745 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:40:23.747 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=15 2024-04-06T09:40:23.748 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=86, tag_tid=14, entry_tid=9182' 2024-04-06T09:40:23.748 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:40:23.748 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:40:23.750 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=14 2024-04-06T09:40:23.751 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=93, tag_tid=15, entry_tid=1' 2024-04-06T09:40:23.751 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:40:23.751 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:40:23.752 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:40:23.753 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=86, tag_tid=14, entry_tid=9182' 2024-04-06T09:40:23.754 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:40:23.754 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:40:23.755 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=9182 2024-04-06T09:40:23.755 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 15 = 14 -a 1 -le 9182 2024-04-06T09:40:23.756 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:40:23.756 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:40:23.756 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:40:23.958 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:40:23.958 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:40:23.958 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:40:23.958 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:40:23.958 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:40:23.958 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:40:23.958 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:40:23.958 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:40:23.958 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:40:23.958 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:40:23.958 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:40:23.959 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:40:23.959 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:40:23.959 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:40:23.959 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:40:23.959 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:40:23.960 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:40:23.961 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:40:23.961 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:40:23.961 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:40:23.962 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:40:23.962 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:40:23.962 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:40:23.962 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:40:23.962 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:40:23.962 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:40:23.962 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:40:23.962 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:40:23.962 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:40:23.962 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:40:23.962 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:40:23.962 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:40:23.963 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:40:23.963 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:40:23.963 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:40:23.963 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:40:23.963 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:40:23.963 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:40:23.963 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:40:23.963 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:40:23.963 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:40:23.963 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:40:23.963 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:40:23.963 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:40:23.963 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:40:23.963 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:40:23.963 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:40:23.964 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:40:23.964 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:40:23.987 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:40:23.987 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:40:23.988 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:40:23.988 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:40:24.113 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:40:24.114 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:40:24.114 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:40:24.115 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:40:24.115 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:40:24.115 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:40:24.115 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:40:24.115 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:40:24.115 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:40:24.115 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:40:24.115 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:40:24.115 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:40:24.116 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:40:24.116 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:40:24.117 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:40:24.117 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:40:24.164 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 21 2024-04-06T09:40:24.164 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 23 2024-04-06T09:40:24.164 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:40:24.164 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=93, tag_tid=15, entry_tid=1], [object_number=92, tag_tid=15, entry_tid=0], [object_number=95, tag_tid=14, entry_tid=19039], [object_number=94, tag_tid=14, entry_tid=19038]]], state=connected] 2024-04-06T09:40:24.164 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=85, tag_tid=14, entry_tid=9269], [object_number=84, tag_tid=14, entry_tid=9268], [object_number=87, tag_tid=14, entry_tid=9267], [object_number=86, tag_tid=14, entry_tid=9266]]], state=connected] 2024-04-06T09:40:24.167 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:40:24.169 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=93, tag_tid=15, entry_tid=1' 2024-04-06T09:40:24.170 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:40:24.170 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:40:24.170 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:40:24.170 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:40:24.170 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:40:24.170 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:40:24.170 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:40:24.170 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:40:24.170 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:40:24.170 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:40:24.171 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:40:24.171 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:40:24.171 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:40:24.171 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:40:24.171 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:40:24.217 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 21 2024-04-06T09:40:24.218 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 23 2024-04-06T09:40:24.218 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:40:24.218 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=93, tag_tid=15, entry_tid=1], [object_number=92, tag_tid=15, entry_tid=0], [object_number=95, tag_tid=14, entry_tid=19039], [object_number=94, tag_tid=14, entry_tid=19038]]], state=connected] 2024-04-06T09:40:24.218 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=85, tag_tid=14, entry_tid=9269], [object_number=84, tag_tid=14, entry_tid=9268], [object_number=87, tag_tid=14, entry_tid=9267], [object_number=86, tag_tid=14, entry_tid=9266]]], state=connected] 2024-04-06T09:40:24.221 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:40:24.223 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=85, tag_tid=14, entry_tid=9269' 2024-04-06T09:40:24.223 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=93, tag_tid=15, entry_tid=1' -a 'object_number=93, tag_tid=15, entry_tid=1' = 'object_number=85, tag_tid=14, entry_tid=9269' 2024-04-06T09:40:24.223 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=85, tag_tid=14, entry_tid=9269' '!=' 'object_number=86, tag_tid=14, entry_tid=9182' 2024-04-06T09:40:24.223 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:40:24.224 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=85, tag_tid=14, entry_tid=9269' = 'object_number=86, tag_tid=14, entry_tid=9182' 2024-04-06T09:40:24.224 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=85, tag_tid=14, entry_tid=9269' 2024-04-06T09:40:24.224 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=93, tag_tid=15, entry_tid=1' 2024-04-06T09:40:24.224 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:40:24.224 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:40:24.225 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=15 2024-04-06T09:40:24.225 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=85, tag_tid=14, entry_tid=9269' 2024-04-06T09:40:24.225 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:40:24.225 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:40:24.226 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=14 2024-04-06T09:40:24.226 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=93, tag_tid=15, entry_tid=1' 2024-04-06T09:40:24.227 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:40:24.227 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:40:24.227 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:40:24.228 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=85, tag_tid=14, entry_tid=9269' 2024-04-06T09:40:24.228 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:40:24.228 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:40:24.230 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=9269 2024-04-06T09:40:24.230 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 15 = 14 -a 1 -le 9269 2024-04-06T09:40:24.230 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:40:24.230 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:40:24.230 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:40:24.431 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:40:24.431 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:40:24.431 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:40:24.432 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:40:24.432 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:40:24.432 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:40:24.432 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:40:24.432 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:40:24.432 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:40:24.432 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:40:24.432 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:40:24.432 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:40:24.432 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:40:24.432 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:40:24.432 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:40:24.433 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:40:24.433 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:40:24.437 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:40:24.437 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:40:24.437 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:40:24.437 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:40:24.437 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:40:24.437 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:40:24.437 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:40:24.437 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:40:24.437 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:40:24.438 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:40:24.438 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:40:24.438 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:40:24.438 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:40:24.438 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:40:24.438 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:40:24.438 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:40:24.438 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:40:24.438 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:40:24.438 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:40:24.438 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:40:24.438 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:40:24.438 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:40:24.438 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:40:24.439 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:40:24.439 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:40:24.439 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:40:24.439 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:40:24.439 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:40:24.439 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:40:24.439 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:40:24.439 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:40:24.439 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:40:24.461 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:40:24.462 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:40:24.462 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:40:24.462 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:40:24.590 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:40:24.591 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:40:24.591 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:40:24.592 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:40:24.592 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:40:24.592 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:40:24.592 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:40:24.592 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:40:24.592 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:40:24.592 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:40:24.592 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:40:24.593 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:40:24.593 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:40:24.595 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:40:24.595 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:40:24.595 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:40:24.641 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 21 2024-04-06T09:40:24.641 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 23 2024-04-06T09:40:24.641 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:40:24.641 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=93, tag_tid=15, entry_tid=1], [object_number=92, tag_tid=15, entry_tid=0], [object_number=95, tag_tid=14, entry_tid=19039], [object_number=94, tag_tid=14, entry_tid=19038]]], state=connected] 2024-04-06T09:40:24.641 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=87, tag_tid=14, entry_tid=9355], [object_number=86, tag_tid=14, entry_tid=9354], [object_number=85, tag_tid=14, entry_tid=9353], [object_number=84, tag_tid=14, entry_tid=9352]]], state=connected] 2024-04-06T09:40:24.644 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:40:24.646 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=93, tag_tid=15, entry_tid=1' 2024-04-06T09:40:24.647 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:40:24.647 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:40:24.647 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:40:24.647 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:40:24.647 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:40:24.647 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:40:24.648 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:40:24.648 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:40:24.648 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:40:24.648 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:40:24.648 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:40:24.648 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:40:24.649 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:40:24.649 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:40:24.649 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:40:24.694 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 21 2024-04-06T09:40:24.694 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 23 2024-04-06T09:40:24.694 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:40:24.694 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=93, tag_tid=15, entry_tid=1], [object_number=92, tag_tid=15, entry_tid=0], [object_number=95, tag_tid=14, entry_tid=19039], [object_number=94, tag_tid=14, entry_tid=19038]]], state=connected] 2024-04-06T09:40:24.695 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=87, tag_tid=14, entry_tid=9355], [object_number=86, tag_tid=14, entry_tid=9354], [object_number=85, tag_tid=14, entry_tid=9353], [object_number=84, tag_tid=14, entry_tid=9352]]], state=connected] 2024-04-06T09:40:24.698 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:40:24.700 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=87, tag_tid=14, entry_tid=9355' 2024-04-06T09:40:24.700 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=93, tag_tid=15, entry_tid=1' -a 'object_number=93, tag_tid=15, entry_tid=1' = 'object_number=87, tag_tid=14, entry_tid=9355' 2024-04-06T09:40:24.700 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=87, tag_tid=14, entry_tid=9355' '!=' 'object_number=85, tag_tid=14, entry_tid=9269' 2024-04-06T09:40:24.700 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:40:24.700 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=87, tag_tid=14, entry_tid=9355' = 'object_number=85, tag_tid=14, entry_tid=9269' 2024-04-06T09:40:24.701 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=87, tag_tid=14, entry_tid=9355' 2024-04-06T09:40:24.701 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=93, tag_tid=15, entry_tid=1' 2024-04-06T09:40:24.701 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:40:24.701 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:40:24.701 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=15 2024-04-06T09:40:24.702 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=87, tag_tid=14, entry_tid=9355' 2024-04-06T09:40:24.702 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:40:24.702 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:40:24.703 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=14 2024-04-06T09:40:24.705 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=93, tag_tid=15, entry_tid=1' 2024-04-06T09:40:24.705 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:40:24.706 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:40:24.707 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:40:24.707 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=87, tag_tid=14, entry_tid=9355' 2024-04-06T09:40:24.707 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:40:24.707 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:40:24.707 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=9355 2024-04-06T09:40:24.707 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 15 = 14 -a 1 -le 9355 2024-04-06T09:40:24.707 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:40:24.707 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:40:24.707 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:40:24.909 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:40:24.909 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:40:24.909 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:40:24.909 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:40:24.909 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:40:24.909 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:40:24.909 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:40:24.909 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:40:24.909 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:40:24.909 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:40:24.909 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:40:24.909 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:40:24.910 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:40:24.910 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:40:24.910 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:40:24.910 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:40:24.910 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:40:24.914 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:40:24.914 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:40:24.914 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:40:24.914 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:40:24.914 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:40:24.914 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:40:24.914 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:40:24.914 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:40:24.914 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:40:24.915 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:40:24.915 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:40:24.915 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:40:24.915 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:40:24.915 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:40:24.915 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:40:24.915 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:40:24.915 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:40:24.915 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:40:24.915 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:40:24.915 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:40:24.915 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:40:24.915 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:40:24.915 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:40:24.915 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:40:24.916 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:40:24.916 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:40:24.916 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:40:24.916 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:40:24.916 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:40:24.916 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:40:24.916 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:40:24.916 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:40:24.938 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:40:24.939 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:40:24.939 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:40:24.939 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:40:25.068 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:40:25.069 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:40:25.069 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:40:25.069 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:40:25.069 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:40:25.069 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:40:25.070 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:40:25.070 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:40:25.070 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:40:25.070 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:40:25.070 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:40:25.070 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:40:25.071 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:40:25.074 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:40:25.074 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:40:25.074 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:40:25.120 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 21 2024-04-06T09:40:25.120 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 23 2024-04-06T09:40:25.120 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:40:25.120 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=93, tag_tid=15, entry_tid=1], [object_number=92, tag_tid=15, entry_tid=0], [object_number=95, tag_tid=14, entry_tid=19039], [object_number=94, tag_tid=14, entry_tid=19038]]], state=connected] 2024-04-06T09:40:25.121 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=84, tag_tid=14, entry_tid=9448], [object_number=87, tag_tid=14, entry_tid=9447], [object_number=86, tag_tid=14, entry_tid=9446], [object_number=85, tag_tid=14, entry_tid=9445]]], state=connected] 2024-04-06T09:40:25.123 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:40:25.125 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=93, tag_tid=15, entry_tid=1' 2024-04-06T09:40:25.126 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:40:25.126 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:40:25.126 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:40:25.126 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:40:25.126 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:40:25.126 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:40:25.126 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:40:25.127 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:40:25.127 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:40:25.127 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:40:25.127 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:40:25.127 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:40:25.127 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:40:25.128 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:40:25.128 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:40:25.173 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 21 2024-04-06T09:40:25.173 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 23 2024-04-06T09:40:25.174 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:40:25.174 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=93, tag_tid=15, entry_tid=1], [object_number=92, tag_tid=15, entry_tid=0], [object_number=95, tag_tid=14, entry_tid=19039], [object_number=94, tag_tid=14, entry_tid=19038]]], state=connected] 2024-04-06T09:40:25.174 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=84, tag_tid=14, entry_tid=9448], [object_number=87, tag_tid=14, entry_tid=9447], [object_number=86, tag_tid=14, entry_tid=9446], [object_number=85, tag_tid=14, entry_tid=9445]]], state=connected] 2024-04-06T09:40:25.177 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:40:25.179 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=84, tag_tid=14, entry_tid=9448' 2024-04-06T09:40:25.179 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=93, tag_tid=15, entry_tid=1' -a 'object_number=93, tag_tid=15, entry_tid=1' = 'object_number=84, tag_tid=14, entry_tid=9448' 2024-04-06T09:40:25.179 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=84, tag_tid=14, entry_tid=9448' '!=' 'object_number=87, tag_tid=14, entry_tid=9355' 2024-04-06T09:40:25.179 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:40:25.179 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=84, tag_tid=14, entry_tid=9448' = 'object_number=87, tag_tid=14, entry_tid=9355' 2024-04-06T09:40:25.179 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=84, tag_tid=14, entry_tid=9448' 2024-04-06T09:40:25.179 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=93, tag_tid=15, entry_tid=1' 2024-04-06T09:40:25.179 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:40:25.179 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:40:25.180 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=15 2024-04-06T09:40:25.181 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=84, tag_tid=14, entry_tid=9448' 2024-04-06T09:40:25.181 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:40:25.181 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:40:25.182 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=14 2024-04-06T09:40:25.182 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=93, tag_tid=15, entry_tid=1' 2024-04-06T09:40:25.182 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:40:25.183 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:40:25.183 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:40:25.184 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=84, tag_tid=14, entry_tid=9448' 2024-04-06T09:40:25.184 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:40:25.184 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:40:25.185 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=9448 2024-04-06T09:40:25.186 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 15 = 14 -a 1 -le 9448 2024-04-06T09:40:25.186 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:40:25.186 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:40:25.186 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:40:25.387 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:40:25.387 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:40:25.387 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:40:25.387 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:40:25.388 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:40:25.388 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:40:25.388 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:40:25.388 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:40:25.388 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:40:25.388 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:40:25.388 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:40:25.388 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:40:25.388 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:40:25.388 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:40:25.388 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:40:25.389 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:40:25.389 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:40:25.391 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:40:25.391 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:40:25.391 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:40:25.391 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:40:25.392 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:40:25.392 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:40:25.392 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:40:25.392 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:40:25.392 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:40:25.392 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:40:25.392 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:40:25.392 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:40:25.392 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:40:25.392 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:40:25.392 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:40:25.392 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:40:25.392 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:40:25.392 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:40:25.392 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:40:25.393 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:40:25.393 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:40:25.393 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:40:25.393 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:40:25.393 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:40:25.393 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:40:25.393 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:40:25.393 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:40:25.393 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:40:25.393 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:40:25.393 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:40:25.393 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:40:25.394 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:40:25.416 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:40:25.417 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:40:25.417 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:40:25.417 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:40:25.548 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:40:25.549 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:40:25.549 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:40:25.549 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:40:25.549 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:40:25.549 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:40:25.549 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:40:25.549 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:40:25.550 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:40:25.550 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:40:25.550 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:40:25.550 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:40:25.550 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:40:25.552 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:40:25.553 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:40:25.553 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:40:25.604 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 21 2024-04-06T09:40:25.604 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 23 2024-04-06T09:40:25.604 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:40:25.605 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=93, tag_tid=15, entry_tid=1], [object_number=92, tag_tid=15, entry_tid=0], [object_number=95, tag_tid=14, entry_tid=19039], [object_number=94, tag_tid=14, entry_tid=19038]]], state=connected] 2024-04-06T09:40:25.605 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=86, tag_tid=14, entry_tid=9542], [object_number=85, tag_tid=14, entry_tid=9541], [object_number=84, tag_tid=14, entry_tid=9540], [object_number=87, tag_tid=14, entry_tid=9539]]], state=connected] 2024-04-06T09:40:25.608 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:40:25.609 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=93, tag_tid=15, entry_tid=1' 2024-04-06T09:40:25.610 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:40:25.610 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:40:25.610 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:40:25.611 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:40:25.611 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:40:25.611 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:40:25.611 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:40:25.611 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:40:25.611 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:40:25.611 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:40:25.611 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:40:25.611 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:40:25.612 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:40:25.612 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:40:25.612 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:40:25.659 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 21 2024-04-06T09:40:25.659 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 23 2024-04-06T09:40:25.660 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:40:25.660 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=93, tag_tid=15, entry_tid=1], [object_number=92, tag_tid=15, entry_tid=0], [object_number=95, tag_tid=14, entry_tid=19039], [object_number=94, tag_tid=14, entry_tid=19038]]], state=connected] 2024-04-06T09:40:25.660 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=86, tag_tid=14, entry_tid=9542], [object_number=85, tag_tid=14, entry_tid=9541], [object_number=84, tag_tid=14, entry_tid=9540], [object_number=87, tag_tid=14, entry_tid=9539]]], state=connected] 2024-04-06T09:40:25.663 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:40:25.665 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=86, tag_tid=14, entry_tid=9542' 2024-04-06T09:40:25.665 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=93, tag_tid=15, entry_tid=1' -a 'object_number=93, tag_tid=15, entry_tid=1' = 'object_number=86, tag_tid=14, entry_tid=9542' 2024-04-06T09:40:25.665 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=86, tag_tid=14, entry_tid=9542' '!=' 'object_number=84, tag_tid=14, entry_tid=9448' 2024-04-06T09:40:25.665 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:40:25.665 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=86, tag_tid=14, entry_tid=9542' = 'object_number=84, tag_tid=14, entry_tid=9448' 2024-04-06T09:40:25.665 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=86, tag_tid=14, entry_tid=9542' 2024-04-06T09:40:25.666 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=93, tag_tid=15, entry_tid=1' 2024-04-06T09:40:25.666 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:40:25.666 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:40:25.666 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=15 2024-04-06T09:40:25.667 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=86, tag_tid=14, entry_tid=9542' 2024-04-06T09:40:25.667 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:40:25.667 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:40:25.668 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=14 2024-04-06T09:40:25.669 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=93, tag_tid=15, entry_tid=1' 2024-04-06T09:40:25.669 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:40:25.669 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:40:25.670 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:40:25.670 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=86, tag_tid=14, entry_tid=9542' 2024-04-06T09:40:25.670 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:40:25.670 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:40:25.672 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=9542 2024-04-06T09:40:25.672 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 15 = 14 -a 1 -le 9542 2024-04-06T09:40:25.672 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:40:25.672 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:40:25.672 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:40:25.874 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:40:25.874 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:40:25.874 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:40:25.874 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:40:25.874 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:40:25.874 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:40:25.874 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:40:25.874 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:40:25.874 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:40:25.874 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:40:25.875 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:40:25.875 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:40:25.875 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:40:25.875 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:40:25.875 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:40:25.875 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:40:25.875 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:40:25.877 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:40:25.878 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:40:25.878 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:40:25.878 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:40:25.878 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:40:25.878 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:40:25.878 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:40:25.878 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:40:25.878 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:40:25.878 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:40:25.878 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:40:25.878 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:40:25.878 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:40:25.879 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:40:25.879 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:40:25.879 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:40:25.879 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:40:25.879 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:40:25.879 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:40:25.879 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:40:25.879 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:40:25.879 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:40:25.879 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:40:25.879 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:40:25.879 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:40:25.879 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:40:25.879 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:40:25.880 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:40:25.880 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:40:25.880 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:40:25.880 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:40:25.880 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:40:25.904 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:40:25.904 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:40:25.904 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:40:25.904 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:40:26.034 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:40:26.035 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:40:26.035 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:40:26.035 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:40:26.035 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:40:26.035 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:40:26.035 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:40:26.035 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:40:26.036 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:40:26.036 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:40:26.036 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:40:26.036 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:40:26.036 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:40:26.038 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:40:26.039 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:40:26.039 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:40:26.092 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 21 2024-04-06T09:40:26.093 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 23 2024-04-06T09:40:26.093 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:40:26.093 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=93, tag_tid=15, entry_tid=1], [object_number=92, tag_tid=15, entry_tid=0], [object_number=95, tag_tid=14, entry_tid=19039], [object_number=94, tag_tid=14, entry_tid=19038]]], state=connected] 2024-04-06T09:40:26.093 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=85, tag_tid=14, entry_tid=9641], [object_number=84, tag_tid=14, entry_tid=9640], [object_number=87, tag_tid=14, entry_tid=9639], [object_number=86, tag_tid=14, entry_tid=9638]]], state=connected] 2024-04-06T09:40:26.096 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:40:26.097 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=93, tag_tid=15, entry_tid=1' 2024-04-06T09:40:26.099 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:40:26.099 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:40:26.099 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:40:26.099 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:40:26.099 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:40:26.099 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:40:26.099 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:40:26.099 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:40:26.099 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:40:26.099 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:40:26.099 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:40:26.099 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:40:26.100 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:40:26.100 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:40:26.100 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:40:26.146 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 21 2024-04-06T09:40:26.146 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 23 2024-04-06T09:40:26.147 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:40:26.147 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=93, tag_tid=15, entry_tid=1], [object_number=92, tag_tid=15, entry_tid=0], [object_number=95, tag_tid=14, entry_tid=19039], [object_number=94, tag_tid=14, entry_tid=19038]]], state=connected] 2024-04-06T09:40:26.147 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=85, tag_tid=14, entry_tid=9641], [object_number=84, tag_tid=14, entry_tid=9640], [object_number=87, tag_tid=14, entry_tid=9639], [object_number=86, tag_tid=14, entry_tid=9638]]], state=connected] 2024-04-06T09:40:26.150 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:40:26.152 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=85, tag_tid=14, entry_tid=9641' 2024-04-06T09:40:26.152 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=93, tag_tid=15, entry_tid=1' -a 'object_number=93, tag_tid=15, entry_tid=1' = 'object_number=85, tag_tid=14, entry_tid=9641' 2024-04-06T09:40:26.152 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=85, tag_tid=14, entry_tid=9641' '!=' 'object_number=86, tag_tid=14, entry_tid=9542' 2024-04-06T09:40:26.152 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:40:26.152 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=85, tag_tid=14, entry_tid=9641' = 'object_number=86, tag_tid=14, entry_tid=9542' 2024-04-06T09:40:26.152 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=85, tag_tid=14, entry_tid=9641' 2024-04-06T09:40:26.152 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=93, tag_tid=15, entry_tid=1' 2024-04-06T09:40:26.153 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:40:26.153 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:40:26.153 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=15 2024-04-06T09:40:26.154 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=85, tag_tid=14, entry_tid=9641' 2024-04-06T09:40:26.154 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:40:26.154 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:40:26.155 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=14 2024-04-06T09:40:26.156 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=93, tag_tid=15, entry_tid=1' 2024-04-06T09:40:26.156 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:40:26.156 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:40:26.157 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:40:26.157 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=85, tag_tid=14, entry_tid=9641' 2024-04-06T09:40:26.157 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:40:26.157 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:40:26.159 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=9641 2024-04-06T09:40:26.159 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 15 = 14 -a 1 -le 9641 2024-04-06T09:40:26.159 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:40:26.159 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:40:26.159 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:40:26.360 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:40:26.361 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:40:26.361 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:40:26.361 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:40:26.361 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:40:26.361 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:40:26.361 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:40:26.361 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:40:26.361 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:40:26.361 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:40:26.361 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:40:26.361 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:40:26.361 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:40:26.361 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:40:26.361 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:40:26.362 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:40:26.362 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:40:26.364 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:40:26.365 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:40:26.365 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:40:26.365 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:40:26.365 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:40:26.365 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:40:26.365 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:40:26.365 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:40:26.365 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:40:26.365 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:40:26.365 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:40:26.365 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:40:26.365 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:40:26.365 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:40:26.365 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:40:26.366 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:40:26.367 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:40:26.367 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:40:26.367 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:40:26.367 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:40:26.367 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:40:26.367 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:40:26.368 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:40:26.368 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:40:26.368 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:40:26.368 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:40:26.368 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:40:26.368 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:40:26.368 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:40:26.368 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:40:26.368 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:40:26.368 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:40:26.391 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:40:26.392 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:40:26.392 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:40:26.392 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:40:26.519 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:40:26.520 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:40:26.521 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:40:26.521 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:40:26.521 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:40:26.521 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:40:26.521 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:40:26.521 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:40:26.521 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:40:26.521 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:40:26.522 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:40:26.522 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:40:26.522 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:40:26.523 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:40:26.523 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:40:26.523 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:40:26.576 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 21 2024-04-06T09:40:26.576 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 23 2024-04-06T09:40:26.576 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:40:26.576 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=93, tag_tid=15, entry_tid=1], [object_number=92, tag_tid=15, entry_tid=0], [object_number=95, tag_tid=14, entry_tid=19039], [object_number=94, tag_tid=14, entry_tid=19038]]], state=connected] 2024-04-06T09:40:26.576 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=84, tag_tid=14, entry_tid=9744], [object_number=87, tag_tid=14, entry_tid=9743], [object_number=86, tag_tid=14, entry_tid=9742], [object_number=85, tag_tid=14, entry_tid=9741]]], state=connected] 2024-04-06T09:40:26.579 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:40:26.581 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=93, tag_tid=15, entry_tid=1' 2024-04-06T09:40:26.582 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:40:26.582 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:40:26.582 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:40:26.582 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:40:26.582 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:40:26.582 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:40:26.582 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:40:26.582 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:40:26.582 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:40:26.583 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:40:26.583 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:40:26.583 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:40:26.583 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:40:26.583 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:40:26.583 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:40:26.630 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 21 2024-04-06T09:40:26.630 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 23 2024-04-06T09:40:26.630 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:40:26.631 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=93, tag_tid=15, entry_tid=1], [object_number=92, tag_tid=15, entry_tid=0], [object_number=95, tag_tid=14, entry_tid=19039], [object_number=94, tag_tid=14, entry_tid=19038]]], state=connected] 2024-04-06T09:40:26.631 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=84, tag_tid=14, entry_tid=9744], [object_number=87, tag_tid=14, entry_tid=9743], [object_number=86, tag_tid=14, entry_tid=9742], [object_number=85, tag_tid=14, entry_tid=9741]]], state=connected] 2024-04-06T09:40:26.634 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:40:26.636 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=84, tag_tid=14, entry_tid=9744' 2024-04-06T09:40:26.636 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=93, tag_tid=15, entry_tid=1' -a 'object_number=93, tag_tid=15, entry_tid=1' = 'object_number=84, tag_tid=14, entry_tid=9744' 2024-04-06T09:40:26.637 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=84, tag_tid=14, entry_tid=9744' '!=' 'object_number=85, tag_tid=14, entry_tid=9641' 2024-04-06T09:40:26.637 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:40:26.637 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=84, tag_tid=14, entry_tid=9744' = 'object_number=85, tag_tid=14, entry_tid=9641' 2024-04-06T09:40:26.637 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=84, tag_tid=14, entry_tid=9744' 2024-04-06T09:40:26.637 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=93, tag_tid=15, entry_tid=1' 2024-04-06T09:40:26.637 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:40:26.638 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:40:26.640 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=15 2024-04-06T09:40:26.640 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=84, tag_tid=14, entry_tid=9744' 2024-04-06T09:40:26.640 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:40:26.640 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:40:26.640 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=14 2024-04-06T09:40:26.641 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=93, tag_tid=15, entry_tid=1' 2024-04-06T09:40:26.641 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:40:26.641 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:40:26.643 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:40:26.643 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=84, tag_tid=14, entry_tid=9744' 2024-04-06T09:40:26.643 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:40:26.643 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:40:26.643 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=9744 2024-04-06T09:40:26.644 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 15 = 14 -a 1 -le 9744 2024-04-06T09:40:26.644 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:40:26.644 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:40:26.644 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:40:26.844 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:40:26.844 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:40:26.844 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:40:26.844 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:40:26.844 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:40:26.844 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:40:26.844 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:40:26.845 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:40:26.845 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:40:26.845 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:40:26.845 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:40:26.845 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:40:26.845 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:40:26.845 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:40:26.845 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:40:26.845 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:40:26.846 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:40:26.848 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:40:26.849 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:40:26.849 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:40:26.849 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:40:26.849 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:40:26.849 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:40:26.849 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:40:26.849 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:40:26.849 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:40:26.849 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:40:26.849 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:40:26.849 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:40:26.849 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:40:26.849 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:40:26.849 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:40:26.849 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:40:26.850 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:40:26.850 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:40:26.850 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:40:26.850 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:40:26.850 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:40:26.850 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:40:26.850 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:40:26.850 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:40:26.850 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:40:26.850 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:40:26.850 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:40:26.850 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:40:26.850 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:40:26.850 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:40:26.850 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:40:26.851 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:40:26.876 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:40:26.877 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:40:26.877 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:40:26.877 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:40:27.005 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:40:27.007 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:40:27.007 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:40:27.007 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:40:27.007 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:40:27.007 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:40:27.007 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:40:27.007 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:40:27.007 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:40:27.007 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:40:27.007 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:40:27.008 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:40:27.008 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:40:27.009 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:40:27.009 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:40:27.009 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:40:27.055 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 21 2024-04-06T09:40:27.055 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 23 2024-04-06T09:40:27.055 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:40:27.055 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=93, tag_tid=15, entry_tid=1], [object_number=92, tag_tid=15, entry_tid=0], [object_number=95, tag_tid=14, entry_tid=19039], [object_number=94, tag_tid=14, entry_tid=19038]]], state=connected] 2024-04-06T09:40:27.055 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=87, tag_tid=14, entry_tid=9835], [object_number=86, tag_tid=14, entry_tid=9834], [object_number=85, tag_tid=14, entry_tid=9833], [object_number=84, tag_tid=14, entry_tid=9832]]], state=connected] 2024-04-06T09:40:27.059 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:40:27.061 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=93, tag_tid=15, entry_tid=1' 2024-04-06T09:40:27.062 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:40:27.062 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:40:27.062 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:40:27.062 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:40:27.062 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:40:27.062 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:40:27.062 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:40:27.062 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:40:27.062 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:40:27.062 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:40:27.062 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:40:27.062 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:40:27.063 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:40:27.063 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:40:27.063 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:40:27.108 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 21 2024-04-06T09:40:27.108 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 23 2024-04-06T09:40:27.108 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:40:27.109 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=93, tag_tid=15, entry_tid=1], [object_number=92, tag_tid=15, entry_tid=0], [object_number=95, tag_tid=14, entry_tid=19039], [object_number=94, tag_tid=14, entry_tid=19038]]], state=connected] 2024-04-06T09:40:27.109 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=87, tag_tid=14, entry_tid=9835], [object_number=86, tag_tid=14, entry_tid=9834], [object_number=85, tag_tid=14, entry_tid=9833], [object_number=84, tag_tid=14, entry_tid=9832]]], state=connected] 2024-04-06T09:40:27.112 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:40:27.113 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=87, tag_tid=14, entry_tid=9835' 2024-04-06T09:40:27.113 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=93, tag_tid=15, entry_tid=1' -a 'object_number=93, tag_tid=15, entry_tid=1' = 'object_number=87, tag_tid=14, entry_tid=9835' 2024-04-06T09:40:27.114 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=87, tag_tid=14, entry_tid=9835' '!=' 'object_number=84, tag_tid=14, entry_tid=9744' 2024-04-06T09:40:27.114 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:40:27.114 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=87, tag_tid=14, entry_tid=9835' = 'object_number=84, tag_tid=14, entry_tid=9744' 2024-04-06T09:40:27.114 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=87, tag_tid=14, entry_tid=9835' 2024-04-06T09:40:27.114 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=93, tag_tid=15, entry_tid=1' 2024-04-06T09:40:27.115 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:40:27.115 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:40:27.115 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=15 2024-04-06T09:40:27.116 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=87, tag_tid=14, entry_tid=9835' 2024-04-06T09:40:27.116 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:40:27.116 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:40:27.117 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=14 2024-04-06T09:40:27.118 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=93, tag_tid=15, entry_tid=1' 2024-04-06T09:40:27.118 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:40:27.118 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:40:27.118 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:40:27.119 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=87, tag_tid=14, entry_tid=9835' 2024-04-06T09:40:27.119 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:40:27.119 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:40:27.120 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=9835 2024-04-06T09:40:27.121 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 15 = 14 -a 1 -le 9835 2024-04-06T09:40:27.121 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:40:27.121 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:40:27.121 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:40:27.323 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:40:27.323 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:40:27.323 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:40:27.323 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:40:27.323 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:40:27.323 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:40:27.323 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:40:27.323 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:40:27.323 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:40:27.324 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:40:27.324 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:40:27.324 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:40:27.324 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:40:27.324 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:40:27.324 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:40:27.324 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:40:27.324 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:40:27.328 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:40:27.328 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:40:27.328 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:40:27.328 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:40:27.328 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:40:27.328 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:40:27.328 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:40:27.328 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:40:27.328 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:40:27.329 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:40:27.329 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:40:27.329 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:40:27.329 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:40:27.329 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:40:27.329 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:40:27.329 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:40:27.329 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:40:27.329 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:40:27.329 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:40:27.329 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:40:27.329 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:40:27.329 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:40:27.330 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:40:27.330 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:40:27.330 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:40:27.330 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:40:27.330 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:40:27.330 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:40:27.330 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:40:27.330 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:40:27.330 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:40:27.330 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:40:27.353 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:40:27.353 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:40:27.353 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:40:27.353 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:40:27.482 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:40:27.484 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:40:27.484 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:40:27.484 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:40:27.484 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:40:27.484 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:40:27.484 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:40:27.484 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:40:27.484 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:40:27.484 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:40:27.485 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:40:27.485 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:40:27.485 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:40:27.486 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:40:27.486 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:40:27.487 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:40:27.534 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 21 2024-04-06T09:40:27.534 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 23 2024-04-06T09:40:27.534 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:40:27.534 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=93, tag_tid=15, entry_tid=1], [object_number=92, tag_tid=15, entry_tid=0], [object_number=95, tag_tid=14, entry_tid=19039], [object_number=94, tag_tid=14, entry_tid=19038]]], state=connected] 2024-04-06T09:40:27.534 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=85, tag_tid=14, entry_tid=9925], [object_number=84, tag_tid=14, entry_tid=9924], [object_number=87, tag_tid=14, entry_tid=9923], [object_number=86, tag_tid=14, entry_tid=9922]]], state=connected] 2024-04-06T09:40:27.537 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:40:27.539 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=93, tag_tid=15, entry_tid=1' 2024-04-06T09:40:27.540 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:40:27.540 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:40:27.540 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:40:27.540 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:40:27.540 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:40:27.540 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:40:27.540 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:40:27.540 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:40:27.540 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:40:27.540 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:40:27.540 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:40:27.541 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:40:27.541 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:40:27.541 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:40:27.541 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:40:27.588 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 21 2024-04-06T09:40:27.588 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 23 2024-04-06T09:40:27.588 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:40:27.588 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=93, tag_tid=15, entry_tid=1], [object_number=92, tag_tid=15, entry_tid=0], [object_number=95, tag_tid=14, entry_tid=19039], [object_number=94, tag_tid=14, entry_tid=19038]]], state=connected] 2024-04-06T09:40:27.588 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=85, tag_tid=14, entry_tid=9925], [object_number=84, tag_tid=14, entry_tid=9924], [object_number=87, tag_tid=14, entry_tid=9923], [object_number=86, tag_tid=14, entry_tid=9922]]], state=connected] 2024-04-06T09:40:27.591 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:40:27.593 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=85, tag_tid=14, entry_tid=9925' 2024-04-06T09:40:27.593 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=93, tag_tid=15, entry_tid=1' -a 'object_number=93, tag_tid=15, entry_tid=1' = 'object_number=85, tag_tid=14, entry_tid=9925' 2024-04-06T09:40:27.593 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=85, tag_tid=14, entry_tid=9925' '!=' 'object_number=87, tag_tid=14, entry_tid=9835' 2024-04-06T09:40:27.593 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:40:27.594 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=85, tag_tid=14, entry_tid=9925' = 'object_number=87, tag_tid=14, entry_tid=9835' 2024-04-06T09:40:27.594 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=85, tag_tid=14, entry_tid=9925' 2024-04-06T09:40:27.594 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=93, tag_tid=15, entry_tid=1' 2024-04-06T09:40:27.594 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:40:27.594 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:40:27.594 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=15 2024-04-06T09:40:27.595 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=85, tag_tid=14, entry_tid=9925' 2024-04-06T09:40:27.595 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:40:27.595 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:40:27.596 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=14 2024-04-06T09:40:27.597 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=93, tag_tid=15, entry_tid=1' 2024-04-06T09:40:27.597 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:40:27.597 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:40:27.598 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:40:27.598 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=85, tag_tid=14, entry_tid=9925' 2024-04-06T09:40:27.599 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:40:27.599 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:40:27.600 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=9925 2024-04-06T09:40:27.600 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 15 = 14 -a 1 -le 9925 2024-04-06T09:40:27.600 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:40:27.600 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:40:27.600 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:40:27.803 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:40:27.803 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:40:27.803 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:40:27.803 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:40:27.803 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:40:27.803 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:40:27.803 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:40:27.804 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:40:27.804 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:40:27.804 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:40:27.804 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:40:27.804 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:40:27.804 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:40:27.804 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:40:27.804 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:40:27.805 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:40:27.805 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:40:27.807 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:40:27.807 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:40:27.808 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:40:27.808 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:40:27.808 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:40:27.808 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:40:27.808 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:40:27.808 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:40:27.808 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:40:27.808 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:40:27.808 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:40:27.808 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:40:27.808 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:40:27.808 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:40:27.808 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:40:27.809 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:40:27.809 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:40:27.809 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:40:27.809 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:40:27.809 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:40:27.809 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:40:27.809 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:40:27.809 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:40:27.809 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:40:27.810 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:40:27.810 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:40:27.810 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:40:27.810 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:40:27.810 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:40:27.810 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:40:27.810 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:40:27.810 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:40:27.833 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:40:27.833 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:40:27.833 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:40:27.833 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:40:27.966 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:40:27.968 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:40:27.968 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:40:27.968 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:40:27.968 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:40:27.968 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:40:27.968 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:40:27.968 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:40:27.968 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:40:27.968 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:40:27.968 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:40:27.969 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:40:27.969 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:40:27.969 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:40:27.969 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:40:27.969 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:40:28.015 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 21 2024-04-06T09:40:28.016 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 23 2024-04-06T09:40:28.016 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:40:28.016 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=93, tag_tid=15, entry_tid=1], [object_number=92, tag_tid=15, entry_tid=0], [object_number=95, tag_tid=14, entry_tid=19039], [object_number=94, tag_tid=14, entry_tid=19038]]], state=connected] 2024-04-06T09:40:28.016 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=87, tag_tid=14, entry_tid=10027], [object_number=86, tag_tid=14, entry_tid=10026], [object_number=85, tag_tid=14, entry_tid=10025], [object_number=84, tag_tid=14, entry_tid=10024]]], state=connected] 2024-04-06T09:40:28.019 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:40:28.020 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=93, tag_tid=15, entry_tid=1' 2024-04-06T09:40:28.022 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:40:28.022 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:40:28.022 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:40:28.022 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:40:28.022 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:40:28.022 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:40:28.022 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:40:28.022 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:40:28.022 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:40:28.023 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:40:28.023 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:40:28.023 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:40:28.023 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:40:28.023 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:40:28.023 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:40:28.069 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 21 2024-04-06T09:40:28.069 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 23 2024-04-06T09:40:28.069 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:40:28.069 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=93, tag_tid=15, entry_tid=1], [object_number=92, tag_tid=15, entry_tid=0], [object_number=95, tag_tid=14, entry_tid=19039], [object_number=94, tag_tid=14, entry_tid=19038]]], state=connected] 2024-04-06T09:40:28.069 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=87, tag_tid=14, entry_tid=10027], [object_number=86, tag_tid=14, entry_tid=10026], [object_number=85, tag_tid=14, entry_tid=10025], [object_number=84, tag_tid=14, entry_tid=10024]]], state=connected] 2024-04-06T09:40:28.072 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:40:28.074 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=87, tag_tid=14, entry_tid=10027' 2024-04-06T09:40:28.075 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=93, tag_tid=15, entry_tid=1' -a 'object_number=93, tag_tid=15, entry_tid=1' = 'object_number=87, tag_tid=14, entry_tid=10027' 2024-04-06T09:40:28.075 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=87, tag_tid=14, entry_tid=10027' '!=' 'object_number=85, tag_tid=14, entry_tid=9925' 2024-04-06T09:40:28.075 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:40:28.075 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=87, tag_tid=14, entry_tid=10027' = 'object_number=85, tag_tid=14, entry_tid=9925' 2024-04-06T09:40:28.075 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=87, tag_tid=14, entry_tid=10027' 2024-04-06T09:40:28.075 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=93, tag_tid=15, entry_tid=1' 2024-04-06T09:40:28.075 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:40:28.075 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:40:28.076 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=15 2024-04-06T09:40:28.076 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=87, tag_tid=14, entry_tid=10027' 2024-04-06T09:40:28.076 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:40:28.076 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:40:28.077 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=14 2024-04-06T09:40:28.078 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=93, tag_tid=15, entry_tid=1' 2024-04-06T09:40:28.078 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:40:28.078 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:40:28.079 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:40:28.080 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=87, tag_tid=14, entry_tid=10027' 2024-04-06T09:40:28.080 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:40:28.080 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:40:28.081 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=10027 2024-04-06T09:40:28.081 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 15 = 14 -a 1 -le 10027 2024-04-06T09:40:28.081 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:40:28.081 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:40:28.081 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:40:28.283 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:40:28.283 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:40:28.283 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:40:28.283 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:40:28.283 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:40:28.283 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:40:28.284 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:40:28.284 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:40:28.284 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:40:28.284 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:40:28.284 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:40:28.284 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:40:28.284 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:40:28.284 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:40:28.284 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:40:28.285 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:40:28.285 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:40:28.288 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:40:28.288 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:40:28.288 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:40:28.288 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:40:28.289 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:40:28.289 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:40:28.289 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:40:28.289 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:40:28.289 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:40:28.289 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:40:28.289 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:40:28.289 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:40:28.289 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:40:28.289 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:40:28.289 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:40:28.289 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:40:28.289 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:40:28.289 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:40:28.290 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:40:28.290 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:40:28.290 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:40:28.290 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:40:28.290 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:40:28.290 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:40:28.290 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:40:28.290 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:40:28.290 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:40:28.290 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:40:28.290 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:40:28.290 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:40:28.290 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:40:28.290 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:40:28.314 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:40:28.314 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:40:28.314 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:40:28.314 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:40:28.449 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:40:28.450 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:40:28.450 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:40:28.450 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:40:28.450 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:40:28.450 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:40:28.451 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:40:28.451 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:40:28.451 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:40:28.451 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:40:28.451 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:40:28.451 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:40:28.451 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:40:28.453 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:40:28.453 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:40:28.453 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:40:28.505 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 21 2024-04-06T09:40:28.505 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 23 2024-04-06T09:40:28.505 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:40:28.506 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=93, tag_tid=15, entry_tid=1], [object_number=92, tag_tid=15, entry_tid=0], [object_number=95, tag_tid=14, entry_tid=19039], [object_number=94, tag_tid=14, entry_tid=19038]]], state=connected] 2024-04-06T09:40:28.506 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=85, tag_tid=14, entry_tid=10149], [object_number=84, tag_tid=14, entry_tid=10148], [object_number=87, tag_tid=14, entry_tid=10147], [object_number=86, tag_tid=14, entry_tid=10146]]], state=connected] 2024-04-06T09:40:28.508 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:40:28.510 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=93, tag_tid=15, entry_tid=1' 2024-04-06T09:40:28.511 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:40:28.511 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:40:28.511 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:40:28.511 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:40:28.511 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:40:28.512 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:40:28.512 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:40:28.512 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:40:28.512 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:40:28.512 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:40:28.512 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:40:28.512 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:40:28.513 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:40:28.513 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:40:28.513 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:40:28.559 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 21 2024-04-06T09:40:28.559 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 23 2024-04-06T09:40:28.559 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:40:28.559 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=93, tag_tid=15, entry_tid=1], [object_number=92, tag_tid=15, entry_tid=0], [object_number=95, tag_tid=14, entry_tid=19039], [object_number=94, tag_tid=14, entry_tid=19038]]], state=connected] 2024-04-06T09:40:28.559 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=85, tag_tid=14, entry_tid=10149], [object_number=84, tag_tid=14, entry_tid=10148], [object_number=87, tag_tid=14, entry_tid=10147], [object_number=86, tag_tid=14, entry_tid=10146]]], state=connected] 2024-04-06T09:40:28.562 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:40:28.564 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=85, tag_tid=14, entry_tid=10149' 2024-04-06T09:40:28.564 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=93, tag_tid=15, entry_tid=1' -a 'object_number=93, tag_tid=15, entry_tid=1' = 'object_number=85, tag_tid=14, entry_tid=10149' 2024-04-06T09:40:28.564 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=85, tag_tid=14, entry_tid=10149' '!=' 'object_number=87, tag_tid=14, entry_tid=10027' 2024-04-06T09:40:28.564 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:40:28.564 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=85, tag_tid=14, entry_tid=10149' = 'object_number=87, tag_tid=14, entry_tid=10027' 2024-04-06T09:40:28.564 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=85, tag_tid=14, entry_tid=10149' 2024-04-06T09:40:28.565 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=93, tag_tid=15, entry_tid=1' 2024-04-06T09:40:28.565 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:40:28.565 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:40:28.566 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=15 2024-04-06T09:40:28.566 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=85, tag_tid=14, entry_tid=10149' 2024-04-06T09:40:28.566 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:40:28.566 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:40:28.567 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=14 2024-04-06T09:40:28.568 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=93, tag_tid=15, entry_tid=1' 2024-04-06T09:40:28.568 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:40:28.568 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:40:28.569 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:40:28.569 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=85, tag_tid=14, entry_tid=10149' 2024-04-06T09:40:28.570 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:40:28.570 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:40:28.571 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=10149 2024-04-06T09:40:28.571 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 15 = 14 -a 1 -le 10149 2024-04-06T09:40:28.572 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:40:28.572 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:40:28.572 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:40:28.773 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:40:28.773 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:40:28.773 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:40:28.773 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:40:28.773 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:40:28.773 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:40:28.773 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:40:28.773 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:40:28.773 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:40:28.773 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:40:28.773 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:40:28.774 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:40:28.774 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:40:28.774 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:40:28.774 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:40:28.774 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:40:28.774 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:40:28.778 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:40:28.778 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:40:28.778 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:40:28.778 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:40:28.778 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:40:28.778 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:40:28.778 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:40:28.778 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:40:28.778 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:40:28.779 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:40:28.779 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:40:28.779 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:40:28.779 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:40:28.779 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:40:28.779 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:40:28.779 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:40:28.779 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:40:28.779 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:40:28.779 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:40:28.779 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:40:28.779 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:40:28.779 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:40:28.779 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:40:28.779 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:40:28.780 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:40:28.780 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:40:28.780 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:40:28.780 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:40:28.780 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:40:28.780 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:40:28.780 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:40:28.780 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:40:28.805 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:40:28.805 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:40:28.805 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:40:28.805 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:40:28.949 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:40:28.950 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:40:28.950 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:40:28.951 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:40:28.951 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:40:28.951 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:40:28.951 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:40:28.951 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:40:28.951 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:40:28.951 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:40:28.951 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:40:28.951 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:40:28.951 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:40:28.952 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:40:28.953 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:40:28.953 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:40:29.004 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 21 2024-04-06T09:40:29.004 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 23 2024-04-06T09:40:29.004 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:40:29.004 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=93, tag_tid=15, entry_tid=1], [object_number=92, tag_tid=15, entry_tid=0], [object_number=95, tag_tid=14, entry_tid=19039], [object_number=94, tag_tid=14, entry_tid=19038]]], state=connected] 2024-04-06T09:40:29.004 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=85, tag_tid=14, entry_tid=10245], [object_number=84, tag_tid=14, entry_tid=10244], [object_number=87, tag_tid=14, entry_tid=10243], [object_number=86, tag_tid=14, entry_tid=10242]]], state=connected] 2024-04-06T09:40:29.007 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:40:29.009 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=93, tag_tid=15, entry_tid=1' 2024-04-06T09:40:29.010 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:40:29.010 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:40:29.010 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:40:29.010 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:40:29.010 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:40:29.010 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:40:29.010 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:40:29.010 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:40:29.011 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:40:29.011 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:40:29.011 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:40:29.011 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:40:29.011 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:40:29.011 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:40:29.012 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:40:29.058 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 21 2024-04-06T09:40:29.059 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 23 2024-04-06T09:40:29.059 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:40:29.059 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=93, tag_tid=15, entry_tid=1], [object_number=92, tag_tid=15, entry_tid=0], [object_number=95, tag_tid=14, entry_tid=19039], [object_number=94, tag_tid=14, entry_tid=19038]]], state=connected] 2024-04-06T09:40:29.059 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=85, tag_tid=14, entry_tid=10245], [object_number=84, tag_tid=14, entry_tid=10244], [object_number=87, tag_tid=14, entry_tid=10243], [object_number=86, tag_tid=14, entry_tid=10242]]], state=connected] 2024-04-06T09:40:29.062 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:40:29.064 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=85, tag_tid=14, entry_tid=10245' 2024-04-06T09:40:29.064 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=93, tag_tid=15, entry_tid=1' -a 'object_number=93, tag_tid=15, entry_tid=1' = 'object_number=85, tag_tid=14, entry_tid=10245' 2024-04-06T09:40:29.064 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=85, tag_tid=14, entry_tid=10245' '!=' 'object_number=85, tag_tid=14, entry_tid=10149' 2024-04-06T09:40:29.064 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:40:29.064 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=85, tag_tid=14, entry_tid=10245' = 'object_number=85, tag_tid=14, entry_tid=10149' 2024-04-06T09:40:29.064 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=85, tag_tid=14, entry_tid=10245' 2024-04-06T09:40:29.065 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=93, tag_tid=15, entry_tid=1' 2024-04-06T09:40:29.065 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:40:29.065 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:40:29.065 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=15 2024-04-06T09:40:29.066 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=85, tag_tid=14, entry_tid=10245' 2024-04-06T09:40:29.066 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:40:29.066 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:40:29.067 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=14 2024-04-06T09:40:29.068 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=93, tag_tid=15, entry_tid=1' 2024-04-06T09:40:29.068 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:40:29.068 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:40:29.069 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:40:29.069 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=85, tag_tid=14, entry_tid=10245' 2024-04-06T09:40:29.069 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:40:29.069 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:40:29.071 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=10245 2024-04-06T09:40:29.071 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 15 = 14 -a 1 -le 10245 2024-04-06T09:40:29.071 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:40:29.071 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:40:29.071 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:40:29.272 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:40:29.272 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:40:29.272 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:40:29.273 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:40:29.273 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:40:29.273 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:40:29.273 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:40:29.273 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:40:29.273 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:40:29.273 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:40:29.273 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:40:29.273 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:40:29.273 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:40:29.273 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:40:29.273 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:40:29.274 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:40:29.274 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:40:29.277 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:40:29.277 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:40:29.277 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:40:29.277 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:40:29.277 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:40:29.277 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:40:29.277 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:40:29.277 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:40:29.278 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:40:29.278 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:40:29.278 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:40:29.278 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:40:29.278 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:40:29.278 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:40:29.278 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:40:29.278 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:40:29.278 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:40:29.278 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:40:29.278 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:40:29.278 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:40:29.278 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:40:29.278 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:40:29.279 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:40:29.279 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:40:29.279 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:40:29.279 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:40:29.279 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:40:29.279 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:40:29.279 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:40:29.279 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:40:29.279 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:40:29.279 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:40:29.303 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:40:29.304 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:40:29.304 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:40:29.304 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:40:29.434 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:40:29.435 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:40:29.435 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:40:29.435 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:40:29.435 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:40:29.435 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:40:29.435 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:40:29.435 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:40:29.435 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:40:29.435 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:40:29.436 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:40:29.436 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:40:29.436 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:40:29.438 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:40:29.439 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:40:29.439 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:40:29.490 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 21 2024-04-06T09:40:29.491 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 23 2024-04-06T09:40:29.491 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:40:29.491 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=93, tag_tid=15, entry_tid=1], [object_number=92, tag_tid=15, entry_tid=0], [object_number=95, tag_tid=14, entry_tid=19039], [object_number=94, tag_tid=14, entry_tid=19038]]], state=connected] 2024-04-06T09:40:29.491 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=86, tag_tid=14, entry_tid=10338], [object_number=85, tag_tid=14, entry_tid=10337], [object_number=84, tag_tid=14, entry_tid=10336], [object_number=87, tag_tid=14, entry_tid=10335]]], state=connected] 2024-04-06T09:40:29.494 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:40:29.495 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=93, tag_tid=15, entry_tid=1' 2024-04-06T09:40:29.496 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:40:29.496 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:40:29.496 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:40:29.497 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:40:29.497 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:40:29.497 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:40:29.497 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:40:29.497 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:40:29.497 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:40:29.497 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:40:29.497 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:40:29.497 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:40:29.497 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:40:29.498 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:40:29.498 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:40:29.544 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 21 2024-04-06T09:40:29.544 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 23 2024-04-06T09:40:29.544 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:40:29.544 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=93, tag_tid=15, entry_tid=1], [object_number=92, tag_tid=15, entry_tid=0], [object_number=95, tag_tid=14, entry_tid=19039], [object_number=94, tag_tid=14, entry_tid=19038]]], state=connected] 2024-04-06T09:40:29.544 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=86, tag_tid=14, entry_tid=10338], [object_number=85, tag_tid=14, entry_tid=10337], [object_number=84, tag_tid=14, entry_tid=10336], [object_number=87, tag_tid=14, entry_tid=10335]]], state=connected] 2024-04-06T09:40:29.548 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:40:29.550 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=86, tag_tid=14, entry_tid=10338' 2024-04-06T09:40:29.550 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=93, tag_tid=15, entry_tid=1' -a 'object_number=93, tag_tid=15, entry_tid=1' = 'object_number=86, tag_tid=14, entry_tid=10338' 2024-04-06T09:40:29.550 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=86, tag_tid=14, entry_tid=10338' '!=' 'object_number=85, tag_tid=14, entry_tid=10245' 2024-04-06T09:40:29.551 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:40:29.551 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=86, tag_tid=14, entry_tid=10338' = 'object_number=85, tag_tid=14, entry_tid=10245' 2024-04-06T09:40:29.551 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=86, tag_tid=14, entry_tid=10338' 2024-04-06T09:40:29.551 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=93, tag_tid=15, entry_tid=1' 2024-04-06T09:40:29.551 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:40:29.551 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:40:29.552 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=15 2024-04-06T09:40:29.552 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=86, tag_tid=14, entry_tid=10338' 2024-04-06T09:40:29.552 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:40:29.553 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:40:29.554 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=14 2024-04-06T09:40:29.554 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=93, tag_tid=15, entry_tid=1' 2024-04-06T09:40:29.554 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:40:29.554 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:40:29.555 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:40:29.555 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=86, tag_tid=14, entry_tid=10338' 2024-04-06T09:40:29.555 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:40:29.556 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:40:29.557 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=10338 2024-04-06T09:40:29.557 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 15 = 14 -a 1 -le 10338 2024-04-06T09:40:29.557 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:40:29.557 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:40:29.557 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:40:29.759 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:40:29.759 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:40:29.759 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:40:29.759 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:40:29.759 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:40:29.759 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:40:29.759 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:40:29.760 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:40:29.760 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:40:29.760 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:40:29.760 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:40:29.760 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:40:29.760 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:40:29.760 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:40:29.761 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:40:29.765 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:40:29.766 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:40:29.766 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:40:29.766 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:40:29.766 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:40:29.766 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:40:29.766 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:40:29.767 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:40:29.767 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:40:29.767 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:40:29.767 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:40:29.767 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:40:29.767 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:40:29.767 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:40:29.767 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:40:29.768 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:40:29.768 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:40:29.768 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:40:29.768 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:40:29.768 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:40:29.768 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:40:29.768 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:40:29.768 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:40:29.769 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:40:29.769 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:40:29.769 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:40:29.769 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:40:29.769 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:40:29.769 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:40:29.769 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:40:29.769 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:40:29.770 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:40:29.770 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:40:29.770 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:40:29.788 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:40:29.789 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:40:29.789 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:40:29.789 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:40:29.918 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:40:29.919 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:40:29.919 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:40:29.919 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:40:29.920 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:40:29.920 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:40:29.920 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:40:29.920 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:40:29.920 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:40:29.920 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:40:29.920 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:40:29.920 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:40:29.921 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:40:29.921 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:40:29.922 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:40:29.922 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:40:29.976 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 21 2024-04-06T09:40:29.976 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 23 2024-04-06T09:40:29.976 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:40:29.976 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=93, tag_tid=15, entry_tid=1], [object_number=92, tag_tid=15, entry_tid=0], [object_number=95, tag_tid=14, entry_tid=19039], [object_number=94, tag_tid=14, entry_tid=19038]]], state=connected] 2024-04-06T09:40:29.977 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=86, tag_tid=14, entry_tid=10430], [object_number=85, tag_tid=14, entry_tid=10429], [object_number=84, tag_tid=14, entry_tid=10428], [object_number=87, tag_tid=14, entry_tid=10427]]], state=connected] 2024-04-06T09:40:29.980 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:40:29.981 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=93, tag_tid=15, entry_tid=1' 2024-04-06T09:40:29.982 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:40:29.982 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:40:29.982 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:40:29.983 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:40:29.983 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:40:29.983 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:40:29.983 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:40:29.983 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:40:29.983 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:40:29.983 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:40:29.983 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:40:29.983 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:40:29.984 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:40:29.984 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:40:29.984 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:40:30.033 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 21 2024-04-06T09:40:30.034 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 23 2024-04-06T09:40:30.034 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:40:30.034 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=93, tag_tid=15, entry_tid=1], [object_number=92, tag_tid=15, entry_tid=0], [object_number=95, tag_tid=14, entry_tid=19039], [object_number=94, tag_tid=14, entry_tid=19038]]], state=connected] 2024-04-06T09:40:30.034 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=86, tag_tid=14, entry_tid=10430], [object_number=85, tag_tid=14, entry_tid=10429], [object_number=84, tag_tid=14, entry_tid=10428], [object_number=87, tag_tid=14, entry_tid=10427]]], state=connected] 2024-04-06T09:40:30.037 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:40:30.039 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=86, tag_tid=14, entry_tid=10430' 2024-04-06T09:40:30.039 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=93, tag_tid=15, entry_tid=1' -a 'object_number=93, tag_tid=15, entry_tid=1' = 'object_number=86, tag_tid=14, entry_tid=10430' 2024-04-06T09:40:30.039 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=86, tag_tid=14, entry_tid=10430' '!=' 'object_number=86, tag_tid=14, entry_tid=10338' 2024-04-06T09:40:30.039 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:40:30.039 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=86, tag_tid=14, entry_tid=10430' = 'object_number=86, tag_tid=14, entry_tid=10338' 2024-04-06T09:40:30.039 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=86, tag_tid=14, entry_tid=10430' 2024-04-06T09:40:30.039 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=93, tag_tid=15, entry_tid=1' 2024-04-06T09:40:30.040 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:40:30.040 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:40:30.040 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=15 2024-04-06T09:40:30.041 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=86, tag_tid=14, entry_tid=10430' 2024-04-06T09:40:30.041 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:40:30.041 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:40:30.042 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=14 2024-04-06T09:40:30.043 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=93, tag_tid=15, entry_tid=1' 2024-04-06T09:40:30.043 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:40:30.043 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:40:30.044 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:40:30.044 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=86, tag_tid=14, entry_tid=10430' 2024-04-06T09:40:30.044 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:40:30.044 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:40:30.046 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=10430 2024-04-06T09:40:30.046 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 15 = 14 -a 1 -le 10430 2024-04-06T09:40:30.046 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:40:30.046 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:40:30.046 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:40:30.247 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:40:30.247 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:40:30.248 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:40:30.248 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:40:30.248 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:40:30.248 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:40:30.248 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:40:30.248 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:40:30.248 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:40:30.248 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:40:30.248 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:40:30.248 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:40:30.248 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:40:30.248 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:40:30.248 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:40:30.249 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:40:30.249 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:40:30.253 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:40:30.253 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:40:30.253 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:40:30.253 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:40:30.254 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:40:30.254 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:40:30.254 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:40:30.254 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:40:30.254 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:40:30.254 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:40:30.254 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:40:30.254 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:40:30.254 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:40:30.254 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:40:30.254 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:40:30.254 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:40:30.254 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:40:30.255 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:40:30.255 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:40:30.255 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:40:30.255 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:40:30.255 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:40:30.255 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:40:30.255 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:40:30.255 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:40:30.255 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:40:30.255 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:40:30.255 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:40:30.255 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:40:30.255 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:40:30.256 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:40:30.256 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:40:30.278 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:40:30.278 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:40:30.278 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:40:30.279 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:40:30.408 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:40:30.410 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:40:30.410 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:40:30.410 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:40:30.410 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:40:30.410 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:40:30.410 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:40:30.410 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:40:30.410 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:40:30.411 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:40:30.411 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:40:30.411 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:40:30.411 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:40:30.412 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:40:30.413 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:40:30.413 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:40:30.458 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 21 2024-04-06T09:40:30.458 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 23 2024-04-06T09:40:30.458 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:40:30.459 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=93, tag_tid=15, entry_tid=1], [object_number=92, tag_tid=15, entry_tid=0], [object_number=95, tag_tid=14, entry_tid=19039], [object_number=94, tag_tid=14, entry_tid=19038]]], state=connected] 2024-04-06T09:40:30.459 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=85, tag_tid=14, entry_tid=10517], [object_number=84, tag_tid=14, entry_tid=10516], [object_number=87, tag_tid=14, entry_tid=10515], [object_number=86, tag_tid=14, entry_tid=10514]]], state=connected] 2024-04-06T09:40:30.461 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:40:30.463 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=93, tag_tid=15, entry_tid=1' 2024-04-06T09:40:30.465 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:40:30.465 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:40:30.465 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:40:30.465 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:40:30.465 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:40:30.465 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:40:30.465 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:40:30.465 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:40:30.465 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:40:30.466 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:40:30.466 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:40:30.466 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:40:30.466 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:40:30.466 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:40:30.467 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:40:30.515 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 21 2024-04-06T09:40:30.515 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 23 2024-04-06T09:40:30.515 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:40:30.515 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=93, tag_tid=15, entry_tid=1], [object_number=92, tag_tid=15, entry_tid=0], [object_number=95, tag_tid=14, entry_tid=19039], [object_number=94, tag_tid=14, entry_tid=19038]]], state=connected] 2024-04-06T09:40:30.515 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=85, tag_tid=14, entry_tid=10517], [object_number=84, tag_tid=14, entry_tid=10516], [object_number=87, tag_tid=14, entry_tid=10515], [object_number=86, tag_tid=14, entry_tid=10514]]], state=connected] 2024-04-06T09:40:30.518 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:40:30.520 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=85, tag_tid=14, entry_tid=10517' 2024-04-06T09:40:30.520 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=93, tag_tid=15, entry_tid=1' -a 'object_number=93, tag_tid=15, entry_tid=1' = 'object_number=85, tag_tid=14, entry_tid=10517' 2024-04-06T09:40:30.520 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=85, tag_tid=14, entry_tid=10517' '!=' 'object_number=86, tag_tid=14, entry_tid=10430' 2024-04-06T09:40:30.520 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:40:30.520 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=85, tag_tid=14, entry_tid=10517' = 'object_number=86, tag_tid=14, entry_tid=10430' 2024-04-06T09:40:30.521 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=85, tag_tid=14, entry_tid=10517' 2024-04-06T09:40:30.521 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=93, tag_tid=15, entry_tid=1' 2024-04-06T09:40:30.521 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:40:30.521 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:40:30.522 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=15 2024-04-06T09:40:30.522 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=85, tag_tid=14, entry_tid=10517' 2024-04-06T09:40:30.522 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:40:30.523 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:40:30.523 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=14 2024-04-06T09:40:30.524 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=93, tag_tid=15, entry_tid=1' 2024-04-06T09:40:30.524 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:40:30.524 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:40:30.525 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:40:30.526 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=85, tag_tid=14, entry_tid=10517' 2024-04-06T09:40:30.526 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:40:30.526 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:40:30.527 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=10517 2024-04-06T09:40:30.527 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 15 = 14 -a 1 -le 10517 2024-04-06T09:40:30.527 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:40:30.527 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:40:30.527 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:40:30.729 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:40:30.729 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:40:30.729 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:40:30.729 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:40:30.729 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:40:30.729 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:40:30.729 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:40:30.730 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:40:30.730 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:40:30.730 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:40:30.730 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:40:30.730 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:40:30.730 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:40:30.730 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:40:30.730 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:40:30.730 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:40:30.734 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:40:30.735 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:40:30.735 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:40:30.735 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:40:30.735 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:40:30.735 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:40:30.735 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:40:30.735 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:40:30.735 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:40:30.735 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:40:30.735 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:40:30.735 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:40:30.735 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:40:30.735 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:40:30.736 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:40:30.736 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:40:30.736 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:40:30.736 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:40:30.736 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:40:30.736 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:40:30.736 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:40:30.736 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:40:30.736 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:40:30.736 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:40:30.736 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:40:30.736 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:40:30.736 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:40:30.736 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:40:30.736 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:40:30.737 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:40:30.737 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:40:30.737 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:40:30.737 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:40:30.759 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:40:30.760 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:40:30.760 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:40:30.760 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:40:30.889 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:40:30.890 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:40:30.890 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:40:30.890 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:40:30.890 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:40:30.890 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:40:30.890 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:40:30.890 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:40:30.890 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:40:30.891 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:40:30.891 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:40:30.891 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:40:30.891 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:40:30.894 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:40:30.894 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:40:30.894 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:40:30.940 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 21 2024-04-06T09:40:30.941 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 23 2024-04-06T09:40:30.941 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:40:30.941 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=93, tag_tid=15, entry_tid=1], [object_number=92, tag_tid=15, entry_tid=0], [object_number=95, tag_tid=14, entry_tid=19039], [object_number=94, tag_tid=14, entry_tid=19038]]], state=connected] 2024-04-06T09:40:30.941 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=84, tag_tid=14, entry_tid=10608], [object_number=87, tag_tid=14, entry_tid=10607], [object_number=86, tag_tid=14, entry_tid=10606], [object_number=85, tag_tid=14, entry_tid=10605]]], state=connected] 2024-04-06T09:40:30.944 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:40:30.945 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=93, tag_tid=15, entry_tid=1' 2024-04-06T09:40:30.946 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:40:30.946 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:40:30.947 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:40:30.947 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:40:30.947 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:40:30.947 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:40:30.947 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:40:30.947 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:40:30.947 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:40:30.947 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:40:30.947 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:40:30.947 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:40:30.948 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:40:30.948 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:40:30.948 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:40:30.994 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 21 2024-04-06T09:40:30.995 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 23 2024-04-06T09:40:30.995 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:40:30.995 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=93, tag_tid=15, entry_tid=1], [object_number=92, tag_tid=15, entry_tid=0], [object_number=95, tag_tid=14, entry_tid=19039], [object_number=94, tag_tid=14, entry_tid=19038]]], state=connected] 2024-04-06T09:40:30.995 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=84, tag_tid=14, entry_tid=10608], [object_number=87, tag_tid=14, entry_tid=10607], [object_number=86, tag_tid=14, entry_tid=10606], [object_number=85, tag_tid=14, entry_tid=10605]]], state=connected] 2024-04-06T09:40:30.998 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:40:31.000 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=84, tag_tid=14, entry_tid=10608' 2024-04-06T09:40:31.000 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=93, tag_tid=15, entry_tid=1' -a 'object_number=93, tag_tid=15, entry_tid=1' = 'object_number=84, tag_tid=14, entry_tid=10608' 2024-04-06T09:40:31.000 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=84, tag_tid=14, entry_tid=10608' '!=' 'object_number=85, tag_tid=14, entry_tid=10517' 2024-04-06T09:40:31.000 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:40:31.001 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=84, tag_tid=14, entry_tid=10608' = 'object_number=85, tag_tid=14, entry_tid=10517' 2024-04-06T09:40:31.001 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=84, tag_tid=14, entry_tid=10608' 2024-04-06T09:40:31.001 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=93, tag_tid=15, entry_tid=1' 2024-04-06T09:40:31.001 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:40:31.001 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:40:31.001 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=15 2024-04-06T09:40:31.002 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=84, tag_tid=14, entry_tid=10608' 2024-04-06T09:40:31.002 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:40:31.002 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:40:31.003 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=14 2024-04-06T09:40:31.004 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=93, tag_tid=15, entry_tid=1' 2024-04-06T09:40:31.004 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:40:31.004 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:40:31.005 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:40:31.005 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=84, tag_tid=14, entry_tid=10608' 2024-04-06T09:40:31.006 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:40:31.006 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:40:31.007 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=10608 2024-04-06T09:40:31.007 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 15 = 14 -a 1 -le 10608 2024-04-06T09:40:31.007 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:40:31.007 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:40:31.007 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:40:31.209 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:40:31.209 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:40:31.209 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:40:31.209 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:40:31.209 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:40:31.209 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:40:31.209 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:40:31.209 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:40:31.209 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:40:31.209 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:40:31.209 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:40:31.209 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:40:31.210 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:40:31.210 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:40:31.210 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:40:31.210 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:40:31.210 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:40:31.214 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:40:31.214 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:40:31.214 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:40:31.214 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:40:31.214 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:40:31.214 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:40:31.215 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:40:31.215 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:40:31.215 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:40:31.215 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:40:31.215 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:40:31.215 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:40:31.215 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:40:31.215 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:40:31.215 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:40:31.215 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:40:31.215 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:40:31.215 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:40:31.215 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:40:31.215 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:40:31.215 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:40:31.216 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:40:31.216 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:40:31.216 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:40:31.216 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:40:31.216 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:40:31.216 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:40:31.216 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:40:31.216 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:40:31.216 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:40:31.216 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:40:31.216 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:40:31.241 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:40:31.241 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:40:31.241 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:40:31.241 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:40:31.420 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:40:31.423 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:40:31.423 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:40:31.423 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:40:31.424 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:40:31.424 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:40:31.424 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:40:31.424 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:40:31.425 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:40:31.425 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:40:31.425 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:40:31.425 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:40:31.426 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:40:31.427 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:40:31.427 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:40:31.427 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:40:31.472 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 21 2024-04-06T09:40:31.473 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 23 2024-04-06T09:40:31.473 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:40:31.473 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=93, tag_tid=15, entry_tid=1], [object_number=92, tag_tid=15, entry_tid=0], [object_number=95, tag_tid=14, entry_tid=19039], [object_number=94, tag_tid=14, entry_tid=19038]]], state=connected] 2024-04-06T09:40:31.473 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=85, tag_tid=14, entry_tid=10685], [object_number=84, tag_tid=14, entry_tid=10684], [object_number=87, tag_tid=14, entry_tid=10683], [object_number=86, tag_tid=14, entry_tid=10682]]], state=connected] 2024-04-06T09:40:31.476 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:40:31.480 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=93, tag_tid=15, entry_tid=1' 2024-04-06T09:40:31.480 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:40:31.481 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:40:31.481 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:40:31.481 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:40:31.481 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:40:31.481 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:40:31.482 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:40:31.482 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:40:31.482 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:40:31.482 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:40:31.482 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:40:31.483 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:40:31.483 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:40:31.483 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:40:31.483 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:40:31.529 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 21 2024-04-06T09:40:31.529 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 23 2024-04-06T09:40:31.529 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:40:31.529 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=93, tag_tid=15, entry_tid=1], [object_number=92, tag_tid=15, entry_tid=0], [object_number=95, tag_tid=14, entry_tid=19039], [object_number=94, tag_tid=14, entry_tid=19038]]], state=connected] 2024-04-06T09:40:31.529 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=85, tag_tid=14, entry_tid=10685], [object_number=84, tag_tid=14, entry_tid=10684], [object_number=87, tag_tid=14, entry_tid=10683], [object_number=86, tag_tid=14, entry_tid=10682]]], state=connected] 2024-04-06T09:40:31.532 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:40:31.535 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=85, tag_tid=14, entry_tid=10685' 2024-04-06T09:40:31.535 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=93, tag_tid=15, entry_tid=1' -a 'object_number=93, tag_tid=15, entry_tid=1' = 'object_number=85, tag_tid=14, entry_tid=10685' 2024-04-06T09:40:31.535 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=85, tag_tid=14, entry_tid=10685' '!=' 'object_number=84, tag_tid=14, entry_tid=10608' 2024-04-06T09:40:31.535 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:40:31.535 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=85, tag_tid=14, entry_tid=10685' = 'object_number=84, tag_tid=14, entry_tid=10608' 2024-04-06T09:40:31.535 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=85, tag_tid=14, entry_tid=10685' 2024-04-06T09:40:31.536 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=93, tag_tid=15, entry_tid=1' 2024-04-06T09:40:31.536 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:40:31.536 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:40:31.537 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=15 2024-04-06T09:40:31.537 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=85, tag_tid=14, entry_tid=10685' 2024-04-06T09:40:31.537 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:40:31.538 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:40:31.538 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=14 2024-04-06T09:40:31.540 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=93, tag_tid=15, entry_tid=1' 2024-04-06T09:40:31.540 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:40:31.541 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:40:31.541 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:40:31.541 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=85, tag_tid=14, entry_tid=10685' 2024-04-06T09:40:31.541 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:40:31.541 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:40:31.543 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=10685 2024-04-06T09:40:31.543 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 15 = 14 -a 1 -le 10685 2024-04-06T09:40:31.543 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:40:31.544 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:40:31.544 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:40:31.744 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:40:31.744 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:40:31.744 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:40:31.744 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:40:31.744 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:40:31.745 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:40:31.745 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:40:31.745 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:40:31.745 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:40:31.745 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:40:31.745 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:40:31.745 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:40:31.746 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:40:31.746 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:40:31.746 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:40:31.746 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:40:31.750 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:40:31.750 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:40:31.750 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:40:31.750 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:40:31.750 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:40:31.751 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:40:31.751 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:40:31.751 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:40:31.751 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:40:31.751 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:40:31.751 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:40:31.751 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:40:31.753 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:40:31.753 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:40:31.753 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:40:31.753 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:40:31.753 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:40:31.753 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:40:31.754 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:40:31.754 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:40:31.754 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:40:31.754 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:40:31.754 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:40:31.754 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:40:31.755 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:40:31.755 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:40:31.755 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:40:31.755 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:40:31.755 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:40:31.755 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:40:31.755 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:40:31.755 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:40:31.756 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:40:31.771 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:40:31.772 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:40:31.772 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:40:31.773 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:40:31.904 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:40:31.905 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:40:31.905 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:40:31.905 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:40:31.905 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:40:31.905 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:40:31.905 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:40:31.905 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:40:31.905 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:40:31.905 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:40:31.906 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:40:31.906 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:40:31.906 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:40:31.908 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:40:31.908 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:40:31.908 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:40:31.953 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 21 2024-04-06T09:40:31.953 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 23 2024-04-06T09:40:31.953 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:40:31.953 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=93, tag_tid=15, entry_tid=1], [object_number=92, tag_tid=15, entry_tid=0], [object_number=95, tag_tid=14, entry_tid=19039], [object_number=94, tag_tid=14, entry_tid=19038]]], state=connected] 2024-04-06T09:40:31.953 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=86, tag_tid=14, entry_tid=10766], [object_number=85, tag_tid=14, entry_tid=10765], [object_number=84, tag_tid=14, entry_tid=10764], [object_number=87, tag_tid=14, entry_tid=10763]]], state=connected] 2024-04-06T09:40:31.957 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:40:31.958 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=93, tag_tid=15, entry_tid=1' 2024-04-06T09:40:31.959 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:40:31.959 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:40:31.959 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:40:31.959 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:40:31.960 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:40:31.960 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:40:31.960 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:40:31.960 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:40:31.960 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:40:31.960 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:40:31.960 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:40:31.960 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:40:31.961 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:40:31.961 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:40:31.961 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:40:32.006 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 21 2024-04-06T09:40:32.006 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 23 2024-04-06T09:40:32.006 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:40:32.006 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=93, tag_tid=15, entry_tid=1], [object_number=92, tag_tid=15, entry_tid=0], [object_number=95, tag_tid=14, entry_tid=19039], [object_number=94, tag_tid=14, entry_tid=19038]]], state=connected] 2024-04-06T09:40:32.006 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=86, tag_tid=14, entry_tid=10766], [object_number=85, tag_tid=14, entry_tid=10765], [object_number=84, tag_tid=14, entry_tid=10764], [object_number=87, tag_tid=14, entry_tid=10763]]], state=connected] 2024-04-06T09:40:32.010 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:40:32.012 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=86, tag_tid=14, entry_tid=10766' 2024-04-06T09:40:32.012 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=93, tag_tid=15, entry_tid=1' -a 'object_number=93, tag_tid=15, entry_tid=1' = 'object_number=86, tag_tid=14, entry_tid=10766' 2024-04-06T09:40:32.012 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=86, tag_tid=14, entry_tid=10766' '!=' 'object_number=85, tag_tid=14, entry_tid=10685' 2024-04-06T09:40:32.012 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:40:32.012 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=86, tag_tid=14, entry_tid=10766' = 'object_number=85, tag_tid=14, entry_tid=10685' 2024-04-06T09:40:32.012 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=86, tag_tid=14, entry_tid=10766' 2024-04-06T09:40:32.012 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=93, tag_tid=15, entry_tid=1' 2024-04-06T09:40:32.012 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:40:32.012 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:40:32.013 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=15 2024-04-06T09:40:32.014 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=86, tag_tid=14, entry_tid=10766' 2024-04-06T09:40:32.014 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:40:32.014 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:40:32.015 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=14 2024-04-06T09:40:32.015 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=93, tag_tid=15, entry_tid=1' 2024-04-06T09:40:32.016 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:40:32.016 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:40:32.016 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:40:32.017 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=86, tag_tid=14, entry_tid=10766' 2024-04-06T09:40:32.018 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:40:32.018 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:40:32.018 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=10766 2024-04-06T09:40:32.019 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 15 = 14 -a 1 -le 10766 2024-04-06T09:40:32.019 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:40:32.019 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:40:32.019 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:40:32.220 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:40:32.221 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:40:32.221 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:40:32.221 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:40:32.221 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:40:32.221 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:40:32.221 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:40:32.221 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:40:32.221 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:40:32.221 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:40:32.221 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:40:32.221 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:40:32.221 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:40:32.221 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:40:32.222 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:40:32.222 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:40:32.222 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:40:32.224 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:40:32.224 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:40:32.225 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:40:32.225 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:40:32.225 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:40:32.225 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:40:32.225 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:40:32.225 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:40:32.225 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:40:32.225 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:40:32.225 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:40:32.225 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:40:32.225 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:40:32.225 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:40:32.225 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:40:32.225 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:40:32.227 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:40:32.228 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:40:32.228 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:40:32.228 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:40:32.228 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:40:32.228 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:40:32.228 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:40:32.228 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:40:32.228 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:40:32.228 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:40:32.228 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:40:32.228 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:40:32.228 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:40:32.228 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:40:32.228 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:40:32.229 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:40:32.251 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:40:32.252 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:40:32.252 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:40:32.252 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:40:32.376 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:40:32.377 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:40:32.377 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:40:32.377 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:40:32.377 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:40:32.378 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:40:32.378 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:40:32.378 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:40:32.378 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:40:32.378 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:40:32.378 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:40:32.378 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:40:32.378 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:40:32.380 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:40:32.380 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:40:32.380 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:40:32.433 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 21 2024-04-06T09:40:32.433 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 23 2024-04-06T09:40:32.433 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:40:32.434 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=93, tag_tid=15, entry_tid=1], [object_number=92, tag_tid=15, entry_tid=0], [object_number=95, tag_tid=14, entry_tid=19039], [object_number=94, tag_tid=14, entry_tid=19038]]], state=connected] 2024-04-06T09:40:32.434 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=87, tag_tid=14, entry_tid=10851], [object_number=86, tag_tid=14, entry_tid=10850], [object_number=85, tag_tid=14, entry_tid=10849], [object_number=84, tag_tid=14, entry_tid=10848]]], state=connected] 2024-04-06T09:40:32.437 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:40:32.439 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=93, tag_tid=15, entry_tid=1' 2024-04-06T09:40:32.440 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:40:32.440 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:40:32.440 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:40:32.440 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:40:32.440 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:40:32.440 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:40:32.440 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:40:32.440 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:40:32.440 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:40:32.440 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:40:32.440 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:40:32.441 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:40:32.441 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:40:32.441 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:40:32.442 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:40:32.488 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 21 2024-04-06T09:40:32.488 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 23 2024-04-06T09:40:32.488 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:40:32.488 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=93, tag_tid=15, entry_tid=1], [object_number=92, tag_tid=15, entry_tid=0], [object_number=95, tag_tid=14, entry_tid=19039], [object_number=94, tag_tid=14, entry_tid=19038]]], state=connected] 2024-04-06T09:40:32.489 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=87, tag_tid=14, entry_tid=10851], [object_number=86, tag_tid=14, entry_tid=10850], [object_number=85, tag_tid=14, entry_tid=10849], [object_number=84, tag_tid=14, entry_tid=10848]]], state=connected] 2024-04-06T09:40:32.492 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:40:32.494 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=87, tag_tid=14, entry_tid=10851' 2024-04-06T09:40:32.494 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=93, tag_tid=15, entry_tid=1' -a 'object_number=93, tag_tid=15, entry_tid=1' = 'object_number=87, tag_tid=14, entry_tid=10851' 2024-04-06T09:40:32.494 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=87, tag_tid=14, entry_tid=10851' '!=' 'object_number=86, tag_tid=14, entry_tid=10766' 2024-04-06T09:40:32.494 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:40:32.494 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=87, tag_tid=14, entry_tid=10851' = 'object_number=86, tag_tid=14, entry_tid=10766' 2024-04-06T09:40:32.494 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=87, tag_tid=14, entry_tid=10851' 2024-04-06T09:40:32.494 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=93, tag_tid=15, entry_tid=1' 2024-04-06T09:40:32.495 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:40:32.495 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:40:32.495 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=15 2024-04-06T09:40:32.496 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=87, tag_tid=14, entry_tid=10851' 2024-04-06T09:40:32.496 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:40:32.496 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:40:32.497 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=14 2024-04-06T09:40:32.497 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=93, tag_tid=15, entry_tid=1' 2024-04-06T09:40:32.497 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:40:32.498 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:40:32.498 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:40:32.499 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=87, tag_tid=14, entry_tid=10851' 2024-04-06T09:40:32.499 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:40:32.499 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:40:32.500 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=10851 2024-04-06T09:40:32.500 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 15 = 14 -a 1 -le 10851 2024-04-06T09:40:32.500 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:40:32.501 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:40:32.501 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:40:32.704 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:40:32.704 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:40:32.705 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:40:32.705 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:40:32.705 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:40:32.705 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:40:32.705 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:40:32.705 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:40:32.706 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:40:32.706 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:40:32.706 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:40:32.706 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:40:32.706 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:40:32.707 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:40:32.707 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:40:32.707 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:40:32.712 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:40:32.712 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:40:32.712 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:40:32.712 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:40:32.713 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:40:32.713 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:40:32.713 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:40:32.713 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:40:32.713 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:40:32.713 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:40:32.714 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:40:32.714 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:40:32.714 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:40:32.714 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:40:32.714 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:40:32.714 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:40:32.715 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:40:32.715 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:40:32.715 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:40:32.715 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:40:32.715 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:40:32.715 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:40:32.715 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:40:32.715 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:40:32.716 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:40:32.716 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:40:32.716 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:40:32.716 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:40:32.716 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:40:32.716 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:40:32.716 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:40:32.717 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:40:32.717 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:40:32.733 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:40:32.733 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:40:32.734 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:40:32.734 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:40:32.872 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:40:32.872 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:40:32.873 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:40:32.873 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:40:32.873 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:40:32.873 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:40:32.873 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:40:32.873 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:40:32.873 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:40:32.873 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:40:32.874 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:40:32.874 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:40:32.874 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:40:32.876 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:40:32.876 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:40:32.876 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:40:32.929 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 21 2024-04-06T09:40:32.929 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 23 2024-04-06T09:40:32.930 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:40:32.930 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=93, tag_tid=15, entry_tid=1], [object_number=92, tag_tid=15, entry_tid=0], [object_number=95, tag_tid=14, entry_tid=19039], [object_number=94, tag_tid=14, entry_tid=19038]]], state=connected] 2024-04-06T09:40:32.930 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=86, tag_tid=14, entry_tid=10942], [object_number=85, tag_tid=14, entry_tid=10941], [object_number=84, tag_tid=14, entry_tid=10940], [object_number=87, tag_tid=14, entry_tid=10939]]], state=connected] 2024-04-06T09:40:32.933 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:40:32.934 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=93, tag_tid=15, entry_tid=1' 2024-04-06T09:40:32.935 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:40:32.936 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:40:32.936 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:40:32.936 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:40:32.936 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:40:32.936 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:40:32.936 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:40:32.936 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:40:32.936 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:40:32.936 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:40:32.936 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:40:32.936 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:40:32.937 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:40:32.937 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:40:32.937 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:40:32.984 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 21 2024-04-06T09:40:32.985 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 23 2024-04-06T09:40:32.985 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:40:32.985 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=93, tag_tid=15, entry_tid=1], [object_number=92, tag_tid=15, entry_tid=0], [object_number=95, tag_tid=14, entry_tid=19039], [object_number=94, tag_tid=14, entry_tid=19038]]], state=connected] 2024-04-06T09:40:32.985 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=86, tag_tid=14, entry_tid=10942], [object_number=85, tag_tid=14, entry_tid=10941], [object_number=84, tag_tid=14, entry_tid=10940], [object_number=87, tag_tid=14, entry_tid=10939]]], state=connected] 2024-04-06T09:40:32.988 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:40:32.990 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=86, tag_tid=14, entry_tid=10942' 2024-04-06T09:40:32.990 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=93, tag_tid=15, entry_tid=1' -a 'object_number=93, tag_tid=15, entry_tid=1' = 'object_number=86, tag_tid=14, entry_tid=10942' 2024-04-06T09:40:32.990 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=86, tag_tid=14, entry_tid=10942' '!=' 'object_number=87, tag_tid=14, entry_tid=10851' 2024-04-06T09:40:32.990 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:40:32.990 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=86, tag_tid=14, entry_tid=10942' = 'object_number=87, tag_tid=14, entry_tid=10851' 2024-04-06T09:40:32.991 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=86, tag_tid=14, entry_tid=10942' 2024-04-06T09:40:32.991 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=93, tag_tid=15, entry_tid=1' 2024-04-06T09:40:32.991 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:40:32.991 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:40:32.991 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=15 2024-04-06T09:40:32.992 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=86, tag_tid=14, entry_tid=10942' 2024-04-06T09:40:32.992 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:40:32.992 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:40:32.993 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=14 2024-04-06T09:40:32.994 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=93, tag_tid=15, entry_tid=1' 2024-04-06T09:40:32.994 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:40:32.994 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:40:32.995 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:40:32.996 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=86, tag_tid=14, entry_tid=10942' 2024-04-06T09:40:32.996 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:40:32.996 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:40:32.997 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=10942 2024-04-06T09:40:32.997 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 15 = 14 -a 1 -le 10942 2024-04-06T09:40:32.997 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:40:32.997 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:40:32.997 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:40:33.199 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:40:33.199 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:40:33.199 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:40:33.199 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:40:33.199 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:40:33.199 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:40:33.199 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:40:33.199 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:40:33.199 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:40:33.199 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:40:33.200 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:40:33.200 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:40:33.200 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:40:33.200 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:40:33.200 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:40:33.200 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:40:33.200 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:40:33.203 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:40:33.203 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:40:33.203 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:40:33.203 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:40:33.204 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:40:33.204 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:40:33.204 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:40:33.204 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:40:33.204 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:40:33.204 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:40:33.204 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:40:33.204 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:40:33.205 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:40:33.205 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:40:33.205 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:40:33.205 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:40:33.205 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:40:33.205 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:40:33.205 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:40:33.205 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:40:33.206 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:40:33.206 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:40:33.206 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:40:33.206 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:40:33.206 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:40:33.206 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:40:33.206 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:40:33.207 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:40:33.207 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:40:33.207 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:40:33.207 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:40:33.207 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:40:33.228 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:40:33.229 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:40:33.229 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:40:33.229 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:40:33.357 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:40:33.359 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:40:33.360 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:40:33.360 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:40:33.360 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:40:33.360 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:40:33.360 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:40:33.360 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:40:33.360 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:40:33.360 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:40:33.361 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:40:33.361 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:40:33.361 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:40:33.361 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:40:33.361 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:40:33.361 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:40:33.414 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 21 2024-04-06T09:40:33.414 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 23 2024-04-06T09:40:33.415 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:40:33.415 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=93, tag_tid=15, entry_tid=1], [object_number=92, tag_tid=15, entry_tid=0], [object_number=95, tag_tid=14, entry_tid=19039], [object_number=94, tag_tid=14, entry_tid=19038]]], state=connected] 2024-04-06T09:40:33.415 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=84, tag_tid=14, entry_tid=11032], [object_number=87, tag_tid=14, entry_tid=11031], [object_number=86, tag_tid=14, entry_tid=11030], [object_number=85, tag_tid=14, entry_tid=11029]]], state=connected] 2024-04-06T09:40:33.418 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:40:33.419 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=93, tag_tid=15, entry_tid=1' 2024-04-06T09:40:33.420 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:40:33.420 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:40:33.421 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:40:33.421 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:40:33.421 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:40:33.421 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:40:33.421 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:40:33.421 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:40:33.421 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:40:33.421 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:40:33.421 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:40:33.422 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:40:33.422 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:40:33.422 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:40:33.423 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:40:33.469 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 21 2024-04-06T09:40:33.469 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 23 2024-04-06T09:40:33.469 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:40:33.469 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=93, tag_tid=15, entry_tid=1], [object_number=92, tag_tid=15, entry_tid=0], [object_number=95, tag_tid=14, entry_tid=19039], [object_number=94, tag_tid=14, entry_tid=19038]]], state=connected] 2024-04-06T09:40:33.469 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=84, tag_tid=14, entry_tid=11032], [object_number=87, tag_tid=14, entry_tid=11031], [object_number=86, tag_tid=14, entry_tid=11030], [object_number=85, tag_tid=14, entry_tid=11029]]], state=connected] 2024-04-06T09:40:33.472 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:40:33.475 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=84, tag_tid=14, entry_tid=11032' 2024-04-06T09:40:33.475 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=93, tag_tid=15, entry_tid=1' -a 'object_number=93, tag_tid=15, entry_tid=1' = 'object_number=84, tag_tid=14, entry_tid=11032' 2024-04-06T09:40:33.475 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=84, tag_tid=14, entry_tid=11032' '!=' 'object_number=86, tag_tid=14, entry_tid=10942' 2024-04-06T09:40:33.475 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:40:33.475 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=84, tag_tid=14, entry_tid=11032' = 'object_number=86, tag_tid=14, entry_tid=10942' 2024-04-06T09:40:33.475 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=84, tag_tid=14, entry_tid=11032' 2024-04-06T09:40:33.475 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=93, tag_tid=15, entry_tid=1' 2024-04-06T09:40:33.475 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:40:33.475 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:40:33.476 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=15 2024-04-06T09:40:33.477 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=84, tag_tid=14, entry_tid=11032' 2024-04-06T09:40:33.477 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:40:33.477 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:40:33.477 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=14 2024-04-06T09:40:33.478 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=93, tag_tid=15, entry_tid=1' 2024-04-06T09:40:33.478 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:40:33.478 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:40:33.479 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:40:33.480 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=84, tag_tid=14, entry_tid=11032' 2024-04-06T09:40:33.480 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:40:33.480 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:40:33.481 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=11032 2024-04-06T09:40:33.481 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 15 = 14 -a 1 -le 11032 2024-04-06T09:40:33.481 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:40:33.481 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:40:33.481 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:40:33.683 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:40:33.683 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:40:33.683 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:40:33.683 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:40:33.684 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:40:33.684 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:40:33.684 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:40:33.684 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:40:33.684 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:40:33.684 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:40:33.684 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:40:33.684 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:40:33.684 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:40:33.685 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:40:33.685 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:40:33.685 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:40:33.686 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:40:33.688 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:40:33.688 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:40:33.688 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:40:33.688 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:40:33.688 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:40:33.688 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:40:33.688 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:40:33.688 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:40:33.688 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:40:33.688 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:40:33.688 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:40:33.689 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:40:33.689 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:40:33.689 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:40:33.689 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:40:33.689 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:40:33.690 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:40:33.690 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:40:33.690 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:40:33.690 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:40:33.690 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:40:33.691 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:40:33.691 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:40:33.691 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:40:33.691 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:40:33.691 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:40:33.691 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:40:33.691 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:40:33.691 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:40:33.691 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:40:33.691 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:40:33.691 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:40:33.714 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:40:33.718 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:40:33.718 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:40:33.718 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:40:33.850 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:40:33.855 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:40:33.855 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:40:33.855 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:40:33.855 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:40:33.855 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:40:33.855 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:40:33.855 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:40:33.856 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:40:33.856 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:40:33.856 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:40:33.856 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:40:33.856 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:40:33.858 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:40:33.858 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:40:33.858 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:40:33.919 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 21 2024-04-06T09:40:33.919 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 23 2024-04-06T09:40:33.920 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:40:33.920 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=93, tag_tid=15, entry_tid=1], [object_number=92, tag_tid=15, entry_tid=0], [object_number=95, tag_tid=14, entry_tid=19039], [object_number=94, tag_tid=14, entry_tid=19038]]], state=connected] 2024-04-06T09:40:33.920 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=84, tag_tid=14, entry_tid=11120], [object_number=87, tag_tid=14, entry_tid=11119], [object_number=86, tag_tid=14, entry_tid=11118], [object_number=85, tag_tid=14, entry_tid=11117]]], state=connected] 2024-04-06T09:40:33.923 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:40:33.924 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=93, tag_tid=15, entry_tid=1' 2024-04-06T09:40:33.924 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:40:33.924 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:40:33.924 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:40:33.924 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:40:33.924 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:40:33.925 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:40:33.925 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:40:33.925 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:40:33.925 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:40:33.925 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:40:33.925 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:40:33.925 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:40:33.925 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:40:33.925 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:40:33.925 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:40:34.166 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 21 2024-04-06T09:40:34.166 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 23 2024-04-06T09:40:34.166 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:40:34.166 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=93, tag_tid=15, entry_tid=1], [object_number=92, tag_tid=15, entry_tid=0], [object_number=95, tag_tid=14, entry_tid=19039], [object_number=94, tag_tid=14, entry_tid=19038]]], state=connected] 2024-04-06T09:40:34.166 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=84, tag_tid=14, entry_tid=11120], [object_number=87, tag_tid=14, entry_tid=11119], [object_number=86, tag_tid=14, entry_tid=11118], [object_number=85, tag_tid=14, entry_tid=11117]]], state=connected] 2024-04-06T09:40:34.170 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:40:34.172 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=84, tag_tid=14, entry_tid=11120' 2024-04-06T09:40:34.172 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=93, tag_tid=15, entry_tid=1' -a 'object_number=93, tag_tid=15, entry_tid=1' = 'object_number=84, tag_tid=14, entry_tid=11120' 2024-04-06T09:40:34.172 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=84, tag_tid=14, entry_tid=11120' '!=' 'object_number=84, tag_tid=14, entry_tid=11032' 2024-04-06T09:40:34.172 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:40:34.172 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=84, tag_tid=14, entry_tid=11120' = 'object_number=84, tag_tid=14, entry_tid=11032' 2024-04-06T09:40:34.172 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=84, tag_tid=14, entry_tid=11120' 2024-04-06T09:40:34.173 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=93, tag_tid=15, entry_tid=1' 2024-04-06T09:40:34.173 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:40:34.173 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:40:34.173 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=15 2024-04-06T09:40:34.174 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=84, tag_tid=14, entry_tid=11120' 2024-04-06T09:40:34.174 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:40:34.174 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:40:34.175 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=14 2024-04-06T09:40:34.176 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=93, tag_tid=15, entry_tid=1' 2024-04-06T09:40:34.177 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:40:34.177 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:40:34.178 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:40:34.178 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=84, tag_tid=14, entry_tid=11120' 2024-04-06T09:40:34.179 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:40:34.179 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:40:34.180 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=11120 2024-04-06T09:40:34.180 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 15 = 14 -a 1 -le 11120 2024-04-06T09:40:34.180 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:40:34.180 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:40:34.180 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:40:34.382 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:40:34.382 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:40:34.382 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:40:34.382 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:40:34.383 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:40:34.383 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:40:34.383 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:40:34.383 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:40:34.383 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:40:34.383 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:40:34.383 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:40:34.383 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:40:34.383 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:40:34.383 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:40:34.383 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:40:34.384 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:40:34.384 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:40:34.386 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:40:34.386 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:40:34.386 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:40:34.386 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:40:34.387 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:40:34.387 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:40:34.387 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:40:34.387 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:40:34.387 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:40:34.387 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:40:34.387 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:40:34.387 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:40:34.387 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:40:34.387 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:40:34.387 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:40:34.387 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:40:34.388 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:40:34.388 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:40:34.388 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:40:34.389 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:40:34.389 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:40:34.389 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:40:34.389 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:40:34.389 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:40:34.389 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:40:34.389 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:40:34.389 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:40:34.389 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:40:34.389 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:40:34.389 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:40:34.389 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:40:34.389 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:40:34.413 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:40:34.413 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:40:34.413 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:40:34.413 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:40:34.540 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:40:34.542 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:40:34.542 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:40:34.542 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:40:34.542 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:40:34.543 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:40:34.543 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:40:34.543 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:40:34.543 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:40:34.543 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:40:34.543 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:40:34.544 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:40:34.544 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:40:34.544 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:40:34.545 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:40:34.545 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:40:34.597 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 21 2024-04-06T09:40:34.597 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 23 2024-04-06T09:40:34.597 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:40:34.597 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=93, tag_tid=15, entry_tid=1], [object_number=92, tag_tid=15, entry_tid=0], [object_number=95, tag_tid=14, entry_tid=19039], [object_number=94, tag_tid=14, entry_tid=19038]]], state=connected] 2024-04-06T09:40:34.597 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=86, tag_tid=14, entry_tid=11254], [object_number=85, tag_tid=14, entry_tid=11253], [object_number=84, tag_tid=14, entry_tid=11252], [object_number=87, tag_tid=14, entry_tid=11251]]], state=connected] 2024-04-06T09:40:34.601 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:40:34.603 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=93, tag_tid=15, entry_tid=1' 2024-04-06T09:40:34.603 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:40:34.603 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:40:34.603 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:40:34.603 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:40:34.603 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:40:34.603 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:40:34.604 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:40:34.604 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:40:34.604 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:40:34.604 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:40:34.604 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:40:34.604 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:40:34.605 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:40:34.605 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:40:34.605 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:40:34.652 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 21 2024-04-06T09:40:34.652 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 23 2024-04-06T09:40:34.652 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:40:34.652 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=93, tag_tid=15, entry_tid=1], [object_number=92, tag_tid=15, entry_tid=0], [object_number=95, tag_tid=14, entry_tid=19039], [object_number=94, tag_tid=14, entry_tid=19038]]], state=connected] 2024-04-06T09:40:34.652 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=86, tag_tid=14, entry_tid=11254], [object_number=85, tag_tid=14, entry_tid=11253], [object_number=84, tag_tid=14, entry_tid=11252], [object_number=87, tag_tid=14, entry_tid=11251]]], state=connected] 2024-04-06T09:40:34.655 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:40:34.658 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=86, tag_tid=14, entry_tid=11254' 2024-04-06T09:40:34.658 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=93, tag_tid=15, entry_tid=1' -a 'object_number=93, tag_tid=15, entry_tid=1' = 'object_number=86, tag_tid=14, entry_tid=11254' 2024-04-06T09:40:34.658 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=86, tag_tid=14, entry_tid=11254' '!=' 'object_number=84, tag_tid=14, entry_tid=11120' 2024-04-06T09:40:34.658 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:40:34.658 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=86, tag_tid=14, entry_tid=11254' = 'object_number=84, tag_tid=14, entry_tid=11120' 2024-04-06T09:40:34.658 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=86, tag_tid=14, entry_tid=11254' 2024-04-06T09:40:34.658 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=93, tag_tid=15, entry_tid=1' 2024-04-06T09:40:34.658 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:40:34.658 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:40:34.659 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=15 2024-04-06T09:40:34.659 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=86, tag_tid=14, entry_tid=11254' 2024-04-06T09:40:34.659 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:40:34.659 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:40:34.660 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=14 2024-04-06T09:40:34.661 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=93, tag_tid=15, entry_tid=1' 2024-04-06T09:40:34.661 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:40:34.661 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:40:34.662 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:40:34.662 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=86, tag_tid=14, entry_tid=11254' 2024-04-06T09:40:34.662 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:40:34.663 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:40:34.664 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=11254 2024-04-06T09:40:34.664 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 15 = 14 -a 1 -le 11254 2024-04-06T09:40:34.664 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:40:34.664 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:40:34.664 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:40:34.865 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:40:34.866 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:40:34.866 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:40:34.866 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:40:34.866 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:40:34.866 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:40:34.866 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:40:34.866 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:40:34.866 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:40:34.866 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:40:34.866 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:40:34.866 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:40:34.866 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:40:34.867 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:40:34.867 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:40:34.867 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:40:34.867 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:40:34.869 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:40:34.869 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:40:34.869 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:40:34.869 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:40:34.869 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:40:34.869 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:40:34.870 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:40:34.870 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:40:34.870 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:40:34.870 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:40:34.870 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:40:34.870 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:40:34.870 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:40:34.870 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:40:34.870 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:40:34.870 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:40:34.871 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:40:34.871 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:40:34.872 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:40:34.872 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:40:34.872 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:40:34.872 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:40:34.872 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:40:34.872 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:40:34.872 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:40:34.872 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:40:34.872 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:40:34.872 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:40:34.872 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:40:34.872 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:40:34.872 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:40:34.872 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:40:34.896 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:40:34.897 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:40:34.897 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:40:34.897 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:40:35.028 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:40:35.028 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:40:35.028 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:40:35.028 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:40:35.028 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:40:35.028 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:40:35.028 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:40:35.028 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:40:35.028 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:40:35.028 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:40:35.028 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:40:35.028 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:40:35.028 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:40:35.030 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:40:35.030 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:40:35.031 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:40:35.077 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 21 2024-04-06T09:40:35.077 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 23 2024-04-06T09:40:35.077 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:40:35.077 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=93, tag_tid=15, entry_tid=1], [object_number=92, tag_tid=15, entry_tid=0], [object_number=95, tag_tid=14, entry_tid=19039], [object_number=94, tag_tid=14, entry_tid=19038]]], state=connected] 2024-04-06T09:40:35.077 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=85, tag_tid=14, entry_tid=11349], [object_number=84, tag_tid=14, entry_tid=11348], [object_number=87, tag_tid=14, entry_tid=11347], [object_number=86, tag_tid=14, entry_tid=11346]]], state=connected] 2024-04-06T09:40:35.080 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:40:35.082 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=93, tag_tid=15, entry_tid=1' 2024-04-06T09:40:35.083 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:40:35.083 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:40:35.083 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:40:35.083 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:40:35.083 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:40:35.083 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:40:35.083 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:40:35.083 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:40:35.083 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:40:35.083 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:40:35.084 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:40:35.084 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:40:35.084 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:40:35.084 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:40:35.084 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:40:35.130 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 21 2024-04-06T09:40:35.131 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 23 2024-04-06T09:40:35.131 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:40:35.131 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=93, tag_tid=15, entry_tid=1], [object_number=92, tag_tid=15, entry_tid=0], [object_number=95, tag_tid=14, entry_tid=19039], [object_number=94, tag_tid=14, entry_tid=19038]]], state=connected] 2024-04-06T09:40:35.131 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=85, tag_tid=14, entry_tid=11349], [object_number=84, tag_tid=14, entry_tid=11348], [object_number=87, tag_tid=14, entry_tid=11347], [object_number=86, tag_tid=14, entry_tid=11346]]], state=connected] 2024-04-06T09:40:35.134 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:40:35.136 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=85, tag_tid=14, entry_tid=11349' 2024-04-06T09:40:35.136 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=93, tag_tid=15, entry_tid=1' -a 'object_number=93, tag_tid=15, entry_tid=1' = 'object_number=85, tag_tid=14, entry_tid=11349' 2024-04-06T09:40:35.136 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=85, tag_tid=14, entry_tid=11349' '!=' 'object_number=86, tag_tid=14, entry_tid=11254' 2024-04-06T09:40:35.136 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:40:35.136 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=85, tag_tid=14, entry_tid=11349' = 'object_number=86, tag_tid=14, entry_tid=11254' 2024-04-06T09:40:35.136 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=85, tag_tid=14, entry_tid=11349' 2024-04-06T09:40:35.137 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=93, tag_tid=15, entry_tid=1' 2024-04-06T09:40:35.137 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:40:35.137 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:40:35.137 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=15 2024-04-06T09:40:35.138 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=85, tag_tid=14, entry_tid=11349' 2024-04-06T09:40:35.138 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:40:35.138 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:40:35.139 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=14 2024-04-06T09:40:35.139 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=93, tag_tid=15, entry_tid=1' 2024-04-06T09:40:35.139 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:40:35.140 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:40:35.141 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:40:35.142 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=85, tag_tid=14, entry_tid=11349' 2024-04-06T09:40:35.142 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:40:35.142 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:40:35.143 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=11349 2024-04-06T09:40:35.144 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 15 = 14 -a 1 -le 11349 2024-04-06T09:40:35.144 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:40:35.144 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:40:35.144 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:40:35.345 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:40:35.345 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:40:35.345 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:40:35.345 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:40:35.345 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:40:35.346 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:40:35.346 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:40:35.346 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:40:35.346 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:40:35.346 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:40:35.346 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:40:35.346 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:40:35.347 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:40:35.347 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:40:35.347 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:40:35.350 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:40:35.350 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:40:35.350 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:40:35.351 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:40:35.351 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:40:35.351 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:40:35.351 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:40:35.351 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:40:35.352 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:40:35.352 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:40:35.352 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:40:35.352 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:40:35.352 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:40:35.353 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:40:35.353 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:40:35.353 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:40:35.353 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:40:35.353 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:40:35.353 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:40:35.353 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:40:35.354 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:40:35.354 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:40:35.354 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:40:35.354 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:40:35.354 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:40:35.354 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:40:35.355 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:40:35.355 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:40:35.355 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:40:35.355 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:40:35.355 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:40:35.355 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:40:35.356 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:40:35.356 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:40:35.375 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:40:35.375 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:40:35.375 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:40:35.375 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:40:35.508 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:40:35.509 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:40:35.509 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:40:35.509 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:40:35.510 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:40:35.510 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:40:35.510 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:40:35.510 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:40:35.510 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:40:35.510 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:40:35.511 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:40:35.511 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:40:35.511 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:40:35.513 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:40:35.513 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:40:35.513 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:40:35.559 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 21 2024-04-06T09:40:35.559 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 23 2024-04-06T09:40:35.559 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:40:35.559 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=93, tag_tid=15, entry_tid=1], [object_number=92, tag_tid=15, entry_tid=0], [object_number=95, tag_tid=14, entry_tid=19039], [object_number=94, tag_tid=14, entry_tid=19038]]], state=connected] 2024-04-06T09:40:35.559 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=85, tag_tid=14, entry_tid=11433], [object_number=84, tag_tid=14, entry_tid=11432], [object_number=87, tag_tid=14, entry_tid=11431], [object_number=86, tag_tid=14, entry_tid=11430]]], state=connected] 2024-04-06T09:40:35.562 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:40:35.563 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=93, tag_tid=15, entry_tid=1' 2024-04-06T09:40:35.565 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:40:35.566 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:40:35.566 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:40:35.566 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:40:35.566 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:40:35.566 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:40:35.566 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:40:35.567 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:40:35.567 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:40:35.567 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:40:35.567 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:40:35.567 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:40:35.568 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:40:35.569 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:40:35.569 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:40:35.612 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 21 2024-04-06T09:40:35.612 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 23 2024-04-06T09:40:35.612 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:40:35.613 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=93, tag_tid=15, entry_tid=1], [object_number=92, tag_tid=15, entry_tid=0], [object_number=95, tag_tid=14, entry_tid=19039], [object_number=94, tag_tid=14, entry_tid=19038]]], state=connected] 2024-04-06T09:40:35.613 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=85, tag_tid=14, entry_tid=11433], [object_number=84, tag_tid=14, entry_tid=11432], [object_number=87, tag_tid=14, entry_tid=11431], [object_number=86, tag_tid=14, entry_tid=11430]]], state=connected] 2024-04-06T09:40:35.616 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:40:35.618 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=85, tag_tid=14, entry_tid=11433' 2024-04-06T09:40:35.618 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=93, tag_tid=15, entry_tid=1' -a 'object_number=93, tag_tid=15, entry_tid=1' = 'object_number=85, tag_tid=14, entry_tid=11433' 2024-04-06T09:40:35.618 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=85, tag_tid=14, entry_tid=11433' '!=' 'object_number=85, tag_tid=14, entry_tid=11349' 2024-04-06T09:40:35.618 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:40:35.618 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=85, tag_tid=14, entry_tid=11433' = 'object_number=85, tag_tid=14, entry_tid=11349' 2024-04-06T09:40:35.619 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=85, tag_tid=14, entry_tid=11433' 2024-04-06T09:40:35.619 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=93, tag_tid=15, entry_tid=1' 2024-04-06T09:40:35.619 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:40:35.619 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:40:35.620 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=15 2024-04-06T09:40:35.620 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=85, tag_tid=14, entry_tid=11433' 2024-04-06T09:40:35.620 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:40:35.620 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:40:35.621 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=14 2024-04-06T09:40:35.622 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=93, tag_tid=15, entry_tid=1' 2024-04-06T09:40:35.622 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:40:35.622 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:40:35.623 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:40:35.623 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=85, tag_tid=14, entry_tid=11433' 2024-04-06T09:40:35.623 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:40:35.623 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:40:35.624 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=11433 2024-04-06T09:40:35.625 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 15 = 14 -a 1 -le 11433 2024-04-06T09:40:35.625 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:40:35.625 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:40:35.625 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:40:35.826 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:40:35.827 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:40:35.827 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:40:35.827 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:40:35.827 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:40:35.827 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:40:35.827 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:40:35.827 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:40:35.827 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:40:35.827 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:40:35.827 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:40:35.827 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:40:35.827 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:40:35.827 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:40:35.828 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:40:35.828 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:40:35.828 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:40:35.831 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:40:35.831 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:40:35.831 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:40:35.831 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:40:35.831 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:40:35.831 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:40:35.831 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:40:35.831 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:40:35.831 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:40:35.831 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:40:35.831 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:40:35.832 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:40:35.832 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:40:35.832 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:40:35.832 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:40:35.832 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:40:35.833 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:40:35.833 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:40:35.833 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:40:35.833 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:40:35.833 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:40:35.833 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:40:35.833 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:40:35.833 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:40:35.833 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:40:35.834 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:40:35.834 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:40:35.834 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:40:35.834 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:40:35.834 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:40:35.834 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:40:35.834 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:40:35.858 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:40:35.858 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:40:35.858 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:40:35.858 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:40:35.993 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:40:35.995 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:40:35.995 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:40:35.995 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:40:35.995 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:40:35.995 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:40:35.996 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:40:35.996 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:40:35.996 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:40:35.996 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:40:35.996 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:40:35.997 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:40:35.997 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:40:35.998 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:40:35.998 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:40:35.999 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:40:36.054 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 21 2024-04-06T09:40:36.055 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 23 2024-04-06T09:40:36.055 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:40:36.055 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=93, tag_tid=15, entry_tid=1], [object_number=92, tag_tid=15, entry_tid=0], [object_number=95, tag_tid=14, entry_tid=19039], [object_number=94, tag_tid=14, entry_tid=19038]]], state=connected] 2024-04-06T09:40:36.055 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=87, tag_tid=14, entry_tid=11523], [object_number=86, tag_tid=14, entry_tid=11522], [object_number=85, tag_tid=14, entry_tid=11521], [object_number=84, tag_tid=14, entry_tid=11520]]], state=connected] 2024-04-06T09:40:36.058 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:40:36.059 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=93, tag_tid=15, entry_tid=1' 2024-04-06T09:40:36.061 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:40:36.061 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:40:36.061 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:40:36.061 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:40:36.061 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:40:36.061 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:40:36.061 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:40:36.061 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:40:36.061 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:40:36.061 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:40:36.061 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:40:36.061 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:40:36.062 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:40:36.062 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:40:36.062 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:40:36.108 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 21 2024-04-06T09:40:36.108 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 23 2024-04-06T09:40:36.109 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:40:36.109 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=93, tag_tid=15, entry_tid=1], [object_number=92, tag_tid=15, entry_tid=0], [object_number=95, tag_tid=14, entry_tid=19039], [object_number=94, tag_tid=14, entry_tid=19038]]], state=connected] 2024-04-06T09:40:36.109 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=87, tag_tid=14, entry_tid=11523], [object_number=86, tag_tid=14, entry_tid=11522], [object_number=85, tag_tid=14, entry_tid=11521], [object_number=84, tag_tid=14, entry_tid=11520]]], state=connected] 2024-04-06T09:40:36.112 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:40:36.114 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=87, tag_tid=14, entry_tid=11523' 2024-04-06T09:40:36.114 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=93, tag_tid=15, entry_tid=1' -a 'object_number=93, tag_tid=15, entry_tid=1' = 'object_number=87, tag_tid=14, entry_tid=11523' 2024-04-06T09:40:36.114 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=87, tag_tid=14, entry_tid=11523' '!=' 'object_number=85, tag_tid=14, entry_tid=11433' 2024-04-06T09:40:36.115 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:40:36.115 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=87, tag_tid=14, entry_tid=11523' = 'object_number=85, tag_tid=14, entry_tid=11433' 2024-04-06T09:40:36.115 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=87, tag_tid=14, entry_tid=11523' 2024-04-06T09:40:36.115 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=93, tag_tid=15, entry_tid=1' 2024-04-06T09:40:36.115 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:40:36.115 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:40:36.115 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=15 2024-04-06T09:40:36.116 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=87, tag_tid=14, entry_tid=11523' 2024-04-06T09:40:36.116 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:40:36.116 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:40:36.117 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=14 2024-04-06T09:40:36.118 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=93, tag_tid=15, entry_tid=1' 2024-04-06T09:40:36.118 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:40:36.118 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:40:36.119 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:40:36.119 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=87, tag_tid=14, entry_tid=11523' 2024-04-06T09:40:36.119 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:40:36.119 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:40:36.121 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=11523 2024-04-06T09:40:36.121 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 15 = 14 -a 1 -le 11523 2024-04-06T09:40:36.121 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:40:36.121 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:40:36.121 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:40:36.323 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:40:36.323 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:40:36.323 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:40:36.323 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:40:36.323 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:40:36.323 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:40:36.324 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:40:36.324 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:40:36.324 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:40:36.324 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:40:36.324 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:40:36.324 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:40:36.324 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:40:36.324 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:40:36.324 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:40:36.325 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:40:36.325 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:40:36.328 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:40:36.328 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:40:36.328 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:40:36.328 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:40:36.328 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:40:36.328 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:40:36.328 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:40:36.328 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:40:36.328 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:40:36.328 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:40:36.328 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:40:36.328 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:40:36.328 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:40:36.329 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:40:36.329 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:40:36.329 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:40:36.329 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:40:36.329 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:40:36.329 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:40:36.329 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:40:36.329 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:40:36.329 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:40:36.329 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:40:36.329 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:40:36.329 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:40:36.329 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:40:36.330 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:40:36.330 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:40:36.330 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:40:36.330 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:40:36.330 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:40:36.330 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:40:36.352 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:40:36.353 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:40:36.353 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:40:36.353 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:40:36.481 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:40:36.482 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:40:36.482 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:40:36.482 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:40:36.482 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:40:36.483 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:40:36.483 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:40:36.483 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:40:36.483 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:40:36.483 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:40:36.483 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:40:36.483 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:40:36.484 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:40:36.485 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:40:36.485 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:40:36.485 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:40:36.537 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 21 2024-04-06T09:40:36.537 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 23 2024-04-06T09:40:36.537 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:40:36.537 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=93, tag_tid=15, entry_tid=1], [object_number=92, tag_tid=15, entry_tid=0], [object_number=95, tag_tid=14, entry_tid=19039], [object_number=94, tag_tid=14, entry_tid=19038]]], state=connected] 2024-04-06T09:40:36.537 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=85, tag_tid=14, entry_tid=11613], [object_number=84, tag_tid=14, entry_tid=11612], [object_number=87, tag_tid=14, entry_tid=11611], [object_number=86, tag_tid=14, entry_tid=11610]]], state=connected] 2024-04-06T09:40:36.541 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:40:36.542 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=93, tag_tid=15, entry_tid=1' 2024-04-06T09:40:36.543 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:40:36.543 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:40:36.543 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:40:36.543 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:40:36.543 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:40:36.543 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:40:36.543 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:40:36.544 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:40:36.544 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:40:36.544 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:40:36.544 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:40:36.544 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:40:36.544 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:40:36.545 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:40:36.545 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:40:36.593 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 21 2024-04-06T09:40:36.593 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 23 2024-04-06T09:40:36.593 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:40:36.593 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=93, tag_tid=15, entry_tid=1], [object_number=92, tag_tid=15, entry_tid=0], [object_number=95, tag_tid=14, entry_tid=19039], [object_number=94, tag_tid=14, entry_tid=19038]]], state=connected] 2024-04-06T09:40:36.593 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=85, tag_tid=14, entry_tid=11613], [object_number=84, tag_tid=14, entry_tid=11612], [object_number=87, tag_tid=14, entry_tid=11611], [object_number=86, tag_tid=14, entry_tid=11610]]], state=connected] 2024-04-06T09:40:36.596 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:40:36.598 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=85, tag_tid=14, entry_tid=11613' 2024-04-06T09:40:36.598 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=93, tag_tid=15, entry_tid=1' -a 'object_number=93, tag_tid=15, entry_tid=1' = 'object_number=85, tag_tid=14, entry_tid=11613' 2024-04-06T09:40:36.598 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=85, tag_tid=14, entry_tid=11613' '!=' 'object_number=87, tag_tid=14, entry_tid=11523' 2024-04-06T09:40:36.598 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:40:36.599 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=85, tag_tid=14, entry_tid=11613' = 'object_number=87, tag_tid=14, entry_tid=11523' 2024-04-06T09:40:36.599 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=85, tag_tid=14, entry_tid=11613' 2024-04-06T09:40:36.599 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=93, tag_tid=15, entry_tid=1' 2024-04-06T09:40:36.599 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:40:36.599 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:40:36.599 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=15 2024-04-06T09:40:36.600 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=85, tag_tid=14, entry_tid=11613' 2024-04-06T09:40:36.600 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:40:36.600 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:40:36.601 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=14 2024-04-06T09:40:36.602 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=93, tag_tid=15, entry_tid=1' 2024-04-06T09:40:36.602 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:40:36.602 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:40:36.603 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:40:36.603 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=85, tag_tid=14, entry_tid=11613' 2024-04-06T09:40:36.604 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:40:36.604 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:40:36.605 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=11613 2024-04-06T09:40:36.605 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 15 = 14 -a 1 -le 11613 2024-04-06T09:40:36.605 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:40:36.605 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:40:36.605 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:40:36.808 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:40:36.808 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:40:36.808 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:40:36.808 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:40:36.808 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:40:36.808 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:40:36.808 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:40:36.808 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:40:36.808 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:40:36.808 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:40:36.809 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:40:36.809 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:40:36.809 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:40:36.809 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:40:36.809 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:40:36.809 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:40:36.809 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:40:36.811 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:40:36.811 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:40:36.811 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:40:36.811 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:40:36.812 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:40:36.812 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:40:36.812 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:40:36.812 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:40:36.812 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:40:36.812 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:40:36.812 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:40:36.812 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:40:36.812 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:40:36.812 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:40:36.812 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:40:36.812 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:40:36.813 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:40:36.813 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:40:36.813 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:40:36.813 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:40:36.813 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:40:36.814 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:40:36.814 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:40:36.814 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:40:36.814 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:40:36.814 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:40:36.814 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:40:36.814 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:40:36.814 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:40:36.814 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:40:36.814 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:40:36.814 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:40:36.838 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:40:36.839 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:40:36.839 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:40:36.839 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:40:36.977 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:40:36.978 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:40:36.978 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:40:36.978 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:40:36.978 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:40:36.978 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:40:36.978 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:40:36.979 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:40:36.979 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:40:36.979 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:40:36.980 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:40:36.980 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:40:36.980 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:40:36.982 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:40:36.982 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:40:36.982 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:40:37.032 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 21 2024-04-06T09:40:37.032 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 23 2024-04-06T09:40:37.032 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:40:37.032 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=93, tag_tid=15, entry_tid=1], [object_number=92, tag_tid=15, entry_tid=0], [object_number=95, tag_tid=14, entry_tid=19039], [object_number=94, tag_tid=14, entry_tid=19038]]], state=connected] 2024-04-06T09:40:37.033 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=86, tag_tid=14, entry_tid=11698], [object_number=85, tag_tid=14, entry_tid=11697], [object_number=84, tag_tid=14, entry_tid=11696], [object_number=87, tag_tid=14, entry_tid=11695]]], state=connected] 2024-04-06T09:40:37.036 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:40:37.037 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=93, tag_tid=15, entry_tid=1' 2024-04-06T09:40:37.039 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:40:37.039 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:40:37.039 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:40:37.039 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:40:37.039 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:40:37.040 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:40:37.040 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:40:37.040 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:40:37.040 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:40:37.040 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:40:37.040 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:40:37.040 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:40:37.041 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:40:37.041 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:40:37.041 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:40:37.087 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 21 2024-04-06T09:40:37.088 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 23 2024-04-06T09:40:37.088 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:40:37.088 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=93, tag_tid=15, entry_tid=1], [object_number=92, tag_tid=15, entry_tid=0], [object_number=95, tag_tid=14, entry_tid=19039], [object_number=94, tag_tid=14, entry_tid=19038]]], state=connected] 2024-04-06T09:40:37.088 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=86, tag_tid=14, entry_tid=11698], [object_number=85, tag_tid=14, entry_tid=11697], [object_number=84, tag_tid=14, entry_tid=11696], [object_number=87, tag_tid=14, entry_tid=11695]]], state=connected] 2024-04-06T09:40:37.091 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:40:37.093 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=86, tag_tid=14, entry_tid=11698' 2024-04-06T09:40:37.093 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=93, tag_tid=15, entry_tid=1' -a 'object_number=93, tag_tid=15, entry_tid=1' = 'object_number=86, tag_tid=14, entry_tid=11698' 2024-04-06T09:40:37.093 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=86, tag_tid=14, entry_tid=11698' '!=' 'object_number=85, tag_tid=14, entry_tid=11613' 2024-04-06T09:40:37.093 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:40:37.093 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=86, tag_tid=14, entry_tid=11698' = 'object_number=85, tag_tid=14, entry_tid=11613' 2024-04-06T09:40:37.093 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=86, tag_tid=14, entry_tid=11698' 2024-04-06T09:40:37.094 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=93, tag_tid=15, entry_tid=1' 2024-04-06T09:40:37.094 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:40:37.094 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:40:37.094 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=15 2024-04-06T09:40:37.095 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=86, tag_tid=14, entry_tid=11698' 2024-04-06T09:40:37.095 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:40:37.095 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:40:37.096 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=14 2024-04-06T09:40:37.097 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=93, tag_tid=15, entry_tid=1' 2024-04-06T09:40:37.097 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:40:37.097 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:40:37.098 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:40:37.099 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=86, tag_tid=14, entry_tid=11698' 2024-04-06T09:40:37.099 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:40:37.099 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:40:37.100 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=11698 2024-04-06T09:40:37.100 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 15 = 14 -a 1 -le 11698 2024-04-06T09:40:37.100 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:40:37.101 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:40:37.101 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:40:37.302 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:40:37.302 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:40:37.302 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:40:37.302 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:40:37.303 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:40:37.303 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:40:37.303 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:40:37.303 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:40:37.303 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:40:37.303 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:40:37.303 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:40:37.303 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:40:37.303 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:40:37.303 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:40:37.303 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:40:37.304 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:40:37.304 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:40:37.308 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:40:37.308 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:40:37.308 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:40:37.308 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:40:37.308 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:40:37.308 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:40:37.308 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:40:37.308 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:40:37.308 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:40:37.309 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:40:37.309 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:40:37.309 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:40:37.309 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:40:37.309 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:40:37.309 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:40:37.309 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:40:37.309 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:40:37.309 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:40:37.309 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:40:37.309 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:40:37.310 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:40:37.310 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:40:37.310 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:40:37.310 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:40:37.310 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:40:37.310 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:40:37.310 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:40:37.310 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:40:37.310 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:40:37.310 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:40:37.310 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:40:37.311 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:40:37.333 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:40:37.334 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:40:37.334 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:40:37.334 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:40:37.470 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:40:37.471 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:40:37.471 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:40:37.471 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:40:37.471 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:40:37.471 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:40:37.472 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:40:37.472 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:40:37.472 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:40:37.472 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:40:37.473 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:40:37.473 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:40:37.473 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:40:37.475 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:40:37.476 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:40:37.476 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:40:37.528 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 21 2024-04-06T09:40:37.528 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 23 2024-04-06T09:40:37.528 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:40:37.528 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=93, tag_tid=15, entry_tid=1], [object_number=92, tag_tid=15, entry_tid=0], [object_number=95, tag_tid=14, entry_tid=19039], [object_number=94, tag_tid=14, entry_tid=19038]]], state=connected] 2024-04-06T09:40:37.528 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=84, tag_tid=14, entry_tid=11784], [object_number=87, tag_tid=14, entry_tid=11783], [object_number=86, tag_tid=14, entry_tid=11782], [object_number=85, tag_tid=14, entry_tid=11781]]], state=connected] 2024-04-06T09:40:37.564 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:40:37.566 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=93, tag_tid=15, entry_tid=1' 2024-04-06T09:40:37.568 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:40:37.568 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:40:37.568 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:40:37.569 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:40:37.569 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:40:37.569 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:40:37.569 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:40:37.569 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:40:37.569 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:40:37.569 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:40:37.570 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:40:37.570 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:40:37.571 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:40:37.571 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:40:37.571 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:40:37.622 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 21 2024-04-06T09:40:37.622 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 23 2024-04-06T09:40:37.623 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:40:37.623 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=93, tag_tid=15, entry_tid=1], [object_number=92, tag_tid=15, entry_tid=0], [object_number=95, tag_tid=14, entry_tid=19039], [object_number=94, tag_tid=14, entry_tid=19038]]], state=connected] 2024-04-06T09:40:37.623 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=84, tag_tid=14, entry_tid=11784], [object_number=87, tag_tid=14, entry_tid=11783], [object_number=86, tag_tid=14, entry_tid=11782], [object_number=85, tag_tid=14, entry_tid=11781]]], state=connected] 2024-04-06T09:40:37.630 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:40:37.632 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=84, tag_tid=14, entry_tid=11784' 2024-04-06T09:40:37.632 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=93, tag_tid=15, entry_tid=1' -a 'object_number=93, tag_tid=15, entry_tid=1' = 'object_number=84, tag_tid=14, entry_tid=11784' 2024-04-06T09:40:37.633 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=84, tag_tid=14, entry_tid=11784' '!=' 'object_number=86, tag_tid=14, entry_tid=11698' 2024-04-06T09:40:37.633 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:40:37.633 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=84, tag_tid=14, entry_tid=11784' = 'object_number=86, tag_tid=14, entry_tid=11698' 2024-04-06T09:40:37.633 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=84, tag_tid=14, entry_tid=11784' 2024-04-06T09:40:37.633 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=93, tag_tid=15, entry_tid=1' 2024-04-06T09:40:37.633 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:40:37.633 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:40:37.634 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=15 2024-04-06T09:40:37.635 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=84, tag_tid=14, entry_tid=11784' 2024-04-06T09:40:37.635 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:40:37.635 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:40:37.635 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=14 2024-04-06T09:40:37.636 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=93, tag_tid=15, entry_tid=1' 2024-04-06T09:40:37.636 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:40:37.636 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:40:37.637 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:40:37.638 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=84, tag_tid=14, entry_tid=11784' 2024-04-06T09:40:37.638 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:40:37.638 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:40:37.639 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=11784 2024-04-06T09:40:37.639 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 15 = 14 -a 1 -le 11784 2024-04-06T09:40:37.639 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:40:37.639 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:40:37.639 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:40:37.841 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:40:37.841 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:40:37.841 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:40:37.841 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:40:37.842 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:40:37.842 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:40:37.842 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:40:37.842 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:40:37.842 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:40:37.842 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:40:37.842 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:40:37.842 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:40:37.842 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:40:37.842 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:40:37.842 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:40:37.843 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:40:37.843 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:40:37.845 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:40:37.845 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:40:37.845 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:40:37.845 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:40:37.846 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:40:37.846 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:40:37.846 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:40:37.846 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:40:37.846 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:40:37.846 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:40:37.846 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:40:37.846 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:40:37.846 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:40:37.847 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:40:37.847 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:40:37.847 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:40:37.848 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:40:37.848 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:40:37.848 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:40:37.848 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:40:37.848 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:40:37.848 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:40:37.848 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:40:37.848 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:40:37.848 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:40:37.849 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:40:37.849 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:40:37.849 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:40:37.849 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:40:37.849 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:40:37.849 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:40:37.849 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:40:37.873 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:40:37.873 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:40:37.873 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:40:37.873 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:40:38.004 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:40:38.005 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:40:38.005 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:40:38.005 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:40:38.005 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:40:38.006 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:40:38.006 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:40:38.006 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:40:38.006 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:40:38.006 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:40:38.006 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:40:38.007 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:40:38.007 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:40:38.008 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:40:38.008 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:40:38.009 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:40:38.058 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 21 2024-04-06T09:40:38.059 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 23 2024-04-06T09:40:38.059 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:40:38.059 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=93, tag_tid=15, entry_tid=1], [object_number=92, tag_tid=15, entry_tid=0], [object_number=95, tag_tid=14, entry_tid=19039], [object_number=94, tag_tid=14, entry_tid=19038]]], state=connected] 2024-04-06T09:40:38.059 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=85, tag_tid=14, entry_tid=11869], [object_number=84, tag_tid=14, entry_tid=11868], [object_number=87, tag_tid=14, entry_tid=11867], [object_number=86, tag_tid=14, entry_tid=11866]]], state=connected] 2024-04-06T09:40:38.062 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:40:38.065 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=93, tag_tid=15, entry_tid=1' 2024-04-06T09:40:38.065 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:40:38.065 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:40:38.065 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:40:38.065 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:40:38.065 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:40:38.065 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:40:38.065 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:40:38.065 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:40:38.065 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:40:38.065 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:40:38.066 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:40:38.066 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:40:38.066 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:40:38.067 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:40:38.067 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:40:38.154 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 21 2024-04-06T09:40:38.154 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 23 2024-04-06T09:40:38.154 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:40:38.154 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=93, tag_tid=15, entry_tid=1], [object_number=92, tag_tid=15, entry_tid=0], [object_number=95, tag_tid=14, entry_tid=19039], [object_number=94, tag_tid=14, entry_tid=19038]]], state=connected] 2024-04-06T09:40:38.154 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=85, tag_tid=14, entry_tid=11869], [object_number=84, tag_tid=14, entry_tid=11868], [object_number=87, tag_tid=14, entry_tid=11867], [object_number=86, tag_tid=14, entry_tid=11866]]], state=connected] 2024-04-06T09:40:38.154 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:40:38.156 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=85, tag_tid=14, entry_tid=11869' 2024-04-06T09:40:38.156 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=93, tag_tid=15, entry_tid=1' -a 'object_number=93, tag_tid=15, entry_tid=1' = 'object_number=85, tag_tid=14, entry_tid=11869' 2024-04-06T09:40:38.156 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=85, tag_tid=14, entry_tid=11869' '!=' 'object_number=84, tag_tid=14, entry_tid=11784' 2024-04-06T09:40:38.156 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:40:38.156 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=85, tag_tid=14, entry_tid=11869' = 'object_number=84, tag_tid=14, entry_tid=11784' 2024-04-06T09:40:38.156 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=85, tag_tid=14, entry_tid=11869' 2024-04-06T09:40:38.157 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=93, tag_tid=15, entry_tid=1' 2024-04-06T09:40:38.157 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:40:38.157 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:40:38.159 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=15 2024-04-06T09:40:38.160 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=85, tag_tid=14, entry_tid=11869' 2024-04-06T09:40:38.160 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:40:38.160 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:40:38.161 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=14 2024-04-06T09:40:38.162 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=93, tag_tid=15, entry_tid=1' 2024-04-06T09:40:38.162 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:40:38.162 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:40:38.163 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:40:38.164 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=85, tag_tid=14, entry_tid=11869' 2024-04-06T09:40:38.164 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:40:38.164 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:40:38.165 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=11869 2024-04-06T09:40:38.165 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 15 = 14 -a 1 -le 11869 2024-04-06T09:40:38.165 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:40:38.165 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:40:38.166 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:40:38.368 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:40:38.368 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:40:38.368 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:40:38.368 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:40:38.368 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:40:38.368 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:40:38.368 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:40:38.368 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:40:38.368 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:40:38.368 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:40:38.368 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:40:38.369 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:40:38.369 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:40:38.369 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:40:38.369 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:40:38.369 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:40:38.369 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:40:38.372 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:40:38.372 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:40:38.372 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:40:38.372 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:40:38.373 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:40:38.373 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:40:38.373 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:40:38.373 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:40:38.373 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:40:38.373 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:40:38.373 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:40:38.373 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:40:38.373 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:40:38.373 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:40:38.373 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:40:38.373 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:40:38.373 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:40:38.373 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:40:38.373 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:40:38.374 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:40:38.374 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:40:38.374 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:40:38.374 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:40:38.374 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:40:38.374 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:40:38.374 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:40:38.374 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:40:38.374 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:40:38.374 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:40:38.374 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:40:38.374 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:40:38.374 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:40:38.397 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:40:38.397 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:40:38.398 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:40:38.398 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:40:38.534 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:40:38.536 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:40:38.536 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:40:38.536 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:40:38.536 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:40:38.536 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:40:38.536 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:40:38.536 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:40:38.536 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:40:38.536 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:40:38.536 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:40:38.536 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:40:38.536 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:40:38.537 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:40:38.537 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:40:38.537 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:40:38.589 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 21 2024-04-06T09:40:38.589 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 23 2024-04-06T09:40:38.589 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:40:38.589 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=93, tag_tid=15, entry_tid=1], [object_number=92, tag_tid=15, entry_tid=0], [object_number=95, tag_tid=14, entry_tid=19039], [object_number=94, tag_tid=14, entry_tid=19038]]], state=connected] 2024-04-06T09:40:38.589 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=86, tag_tid=14, entry_tid=11962], [object_number=85, tag_tid=14, entry_tid=11961], [object_number=84, tag_tid=14, entry_tid=11960], [object_number=87, tag_tid=14, entry_tid=11959]]], state=connected] 2024-04-06T09:40:38.592 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:40:38.594 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=93, tag_tid=15, entry_tid=1' 2024-04-06T09:40:38.596 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:40:38.596 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:40:38.596 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:40:38.596 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:40:38.596 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:40:38.596 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:40:38.596 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:40:38.597 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:40:38.597 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:40:38.597 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:40:38.597 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:40:38.597 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:40:38.597 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:40:38.597 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:40:38.597 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:40:38.643 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 21 2024-04-06T09:40:38.644 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 23 2024-04-06T09:40:38.644 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:40:38.644 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=93, tag_tid=15, entry_tid=1], [object_number=92, tag_tid=15, entry_tid=0], [object_number=95, tag_tid=14, entry_tid=19039], [object_number=94, tag_tid=14, entry_tid=19038]]], state=connected] 2024-04-06T09:40:38.644 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=86, tag_tid=14, entry_tid=11962], [object_number=85, tag_tid=14, entry_tid=11961], [object_number=84, tag_tid=14, entry_tid=11960], [object_number=87, tag_tid=14, entry_tid=11959]]], state=connected] 2024-04-06T09:40:38.647 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:40:38.651 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=86, tag_tid=14, entry_tid=11962' 2024-04-06T09:40:38.651 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=93, tag_tid=15, entry_tid=1' -a 'object_number=93, tag_tid=15, entry_tid=1' = 'object_number=86, tag_tid=14, entry_tid=11962' 2024-04-06T09:40:38.651 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=86, tag_tid=14, entry_tid=11962' '!=' 'object_number=85, tag_tid=14, entry_tid=11869' 2024-04-06T09:40:38.651 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:40:38.651 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=86, tag_tid=14, entry_tid=11962' = 'object_number=85, tag_tid=14, entry_tid=11869' 2024-04-06T09:40:38.652 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=86, tag_tid=14, entry_tid=11962' 2024-04-06T09:40:38.652 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=93, tag_tid=15, entry_tid=1' 2024-04-06T09:40:38.652 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:40:38.652 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:40:38.652 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=15 2024-04-06T09:40:38.656 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=86, tag_tid=14, entry_tid=11962' 2024-04-06T09:40:38.656 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:40:38.657 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:40:38.657 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=14 2024-04-06T09:40:38.657 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=93, tag_tid=15, entry_tid=1' 2024-04-06T09:40:38.657 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:40:38.658 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:40:38.658 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:40:38.658 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=86, tag_tid=14, entry_tid=11962' 2024-04-06T09:40:38.658 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:40:38.658 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:40:38.659 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=11962 2024-04-06T09:40:38.659 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 15 = 14 -a 1 -le 11962 2024-04-06T09:40:38.659 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:40:38.659 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:40:38.659 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:40:38.859 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:40:38.860 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:40:38.860 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:40:38.860 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:40:38.860 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:40:38.860 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:40:38.860 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:40:38.861 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:40:38.861 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:40:38.861 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:40:38.861 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:40:38.861 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:40:38.861 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:40:38.861 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:40:38.862 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:40:38.862 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:40:38.862 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:40:38.868 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:40:38.868 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:40:38.868 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:40:38.868 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:40:38.868 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:40:38.868 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:40:38.869 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:40:38.869 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:40:38.869 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:40:38.869 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:40:38.869 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:40:38.869 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:40:38.870 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:40:38.870 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:40:38.870 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:40:38.870 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:40:38.870 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:40:38.870 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:40:38.871 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:40:38.871 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:40:38.871 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:40:38.871 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:40:38.871 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:40:38.871 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:40:38.872 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:40:38.872 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:40:38.872 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:40:38.872 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:40:38.872 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:40:38.872 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:40:38.873 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:40:38.873 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:40:38.889 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:40:38.889 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:40:38.889 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:40:38.889 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:40:39.018 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:40:39.021 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:40:39.022 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:40:39.022 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:40:39.022 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:40:39.022 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:40:39.022 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:40:39.022 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:40:39.023 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:40:39.023 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:40:39.023 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:40:39.023 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:40:39.023 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:40:39.023 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:40:39.024 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:40:39.024 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:40:39.074 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 21 2024-04-06T09:40:39.075 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 23 2024-04-06T09:40:39.075 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:40:39.075 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=93, tag_tid=15, entry_tid=1], [object_number=92, tag_tid=15, entry_tid=0], [object_number=95, tag_tid=14, entry_tid=19039], [object_number=94, tag_tid=14, entry_tid=19038]]], state=connected] 2024-04-06T09:40:39.075 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=87, tag_tid=14, entry_tid=12047], [object_number=86, tag_tid=14, entry_tid=12046], [object_number=85, tag_tid=14, entry_tid=12045], [object_number=84, tag_tid=14, entry_tid=12044]]], state=connected] 2024-04-06T09:40:39.076 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:40:39.080 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=93, tag_tid=15, entry_tid=1' 2024-04-06T09:40:39.080 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:40:39.080 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:40:39.080 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:40:39.081 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:40:39.081 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:40:39.081 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:40:39.081 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:40:39.081 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:40:39.081 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:40:39.082 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:40:39.082 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:40:39.082 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:40:39.082 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:40:39.083 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:40:39.083 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:40:39.128 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 21 2024-04-06T09:40:39.128 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 23 2024-04-06T09:40:39.128 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:40:39.129 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=93, tag_tid=15, entry_tid=1], [object_number=92, tag_tid=15, entry_tid=0], [object_number=95, tag_tid=14, entry_tid=19039], [object_number=94, tag_tid=14, entry_tid=19038]]], state=connected] 2024-04-06T09:40:39.129 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=87, tag_tid=14, entry_tid=12047], [object_number=86, tag_tid=14, entry_tid=12046], [object_number=85, tag_tid=14, entry_tid=12045], [object_number=84, tag_tid=14, entry_tid=12044]]], state=connected] 2024-04-06T09:40:39.132 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:40:39.135 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=87, tag_tid=14, entry_tid=12047' 2024-04-06T09:40:39.135 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=93, tag_tid=15, entry_tid=1' -a 'object_number=93, tag_tid=15, entry_tid=1' = 'object_number=87, tag_tid=14, entry_tid=12047' 2024-04-06T09:40:39.135 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=87, tag_tid=14, entry_tid=12047' '!=' 'object_number=86, tag_tid=14, entry_tid=11962' 2024-04-06T09:40:39.136 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:40:39.136 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=87, tag_tid=14, entry_tid=12047' = 'object_number=86, tag_tid=14, entry_tid=11962' 2024-04-06T09:40:39.136 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=87, tag_tid=14, entry_tid=12047' 2024-04-06T09:40:39.136 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=93, tag_tid=15, entry_tid=1' 2024-04-06T09:40:39.137 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:40:39.137 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:40:39.137 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=15 2024-04-06T09:40:39.142 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=87, tag_tid=14, entry_tid=12047' 2024-04-06T09:40:39.142 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:40:39.142 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:40:39.142 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=14 2024-04-06T09:40:39.143 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=93, tag_tid=15, entry_tid=1' 2024-04-06T09:40:39.143 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:40:39.143 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:40:39.143 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:40:39.143 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=87, tag_tid=14, entry_tid=12047' 2024-04-06T09:40:39.143 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:40:39.143 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:40:39.144 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=12047 2024-04-06T09:40:39.144 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 15 = 14 -a 1 -le 12047 2024-04-06T09:40:39.144 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:40:39.144 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:40:39.144 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:40:39.343 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:40:39.343 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:40:39.343 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:40:39.343 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:40:39.344 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:40:39.344 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:40:39.344 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:40:39.344 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:40:39.344 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:40:39.344 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:40:39.345 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:40:39.345 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:40:39.345 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:40:39.345 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:40:39.345 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:40:39.349 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:40:39.349 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:40:39.349 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:40:39.349 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:40:39.349 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:40:39.349 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:40:39.350 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:40:39.350 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:40:39.350 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:40:39.350 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:40:39.350 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:40:39.350 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:40:39.351 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:40:39.351 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:40:39.351 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:40:39.351 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:40:39.351 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:40:39.351 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:40:39.351 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:40:39.352 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:40:39.352 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:40:39.352 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:40:39.352 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:40:39.352 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:40:39.352 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:40:39.352 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:40:39.353 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:40:39.353 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:40:39.353 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:40:39.353 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:40:39.353 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:40:39.353 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:40:39.354 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:40:39.354 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:40:39.373 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:40:39.374 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:40:39.374 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:40:39.375 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:40:39.503 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:40:39.506 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:40:39.506 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:40:39.507 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:40:39.507 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:40:39.507 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:40:39.507 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:40:39.507 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:40:39.508 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:40:39.508 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:40:39.508 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:40:39.508 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:40:39.508 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:40:39.510 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:40:39.510 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:40:39.510 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:40:39.558 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 21 2024-04-06T09:40:39.558 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 23 2024-04-06T09:40:39.559 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:40:39.559 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=93, tag_tid=15, entry_tid=1], [object_number=92, tag_tid=15, entry_tid=0], [object_number=95, tag_tid=14, entry_tid=19039], [object_number=94, tag_tid=14, entry_tid=19038]]], state=connected] 2024-04-06T09:40:39.559 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=87, tag_tid=14, entry_tid=12127], [object_number=86, tag_tid=14, entry_tid=12126], [object_number=85, tag_tid=14, entry_tid=12125], [object_number=84, tag_tid=14, entry_tid=12124]]], state=connected] 2024-04-06T09:40:39.562 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:40:39.563 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=93, tag_tid=15, entry_tid=1' 2024-04-06T09:40:39.565 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:40:39.565 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:40:39.565 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:40:39.565 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:40:39.565 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:40:39.566 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:40:39.566 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:40:39.566 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:40:39.566 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:40:39.566 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:40:39.566 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:40:39.567 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:40:39.567 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:40:39.568 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:40:39.568 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:40:39.612 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 21 2024-04-06T09:40:39.613 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 23 2024-04-06T09:40:39.613 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:40:39.613 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=93, tag_tid=15, entry_tid=1], [object_number=92, tag_tid=15, entry_tid=0], [object_number=95, tag_tid=14, entry_tid=19039], [object_number=94, tag_tid=14, entry_tid=19038]]], state=connected] 2024-04-06T09:40:39.613 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=87, tag_tid=14, entry_tid=12127], [object_number=86, tag_tid=14, entry_tid=12126], [object_number=85, tag_tid=14, entry_tid=12125], [object_number=84, tag_tid=14, entry_tid=12124]]], state=connected] 2024-04-06T09:40:39.616 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:40:39.619 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=87, tag_tid=14, entry_tid=12127' 2024-04-06T09:40:39.619 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=93, tag_tid=15, entry_tid=1' -a 'object_number=93, tag_tid=15, entry_tid=1' = 'object_number=87, tag_tid=14, entry_tid=12127' 2024-04-06T09:40:39.620 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=87, tag_tid=14, entry_tid=12127' '!=' 'object_number=87, tag_tid=14, entry_tid=12047' 2024-04-06T09:40:39.620 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:40:39.620 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=87, tag_tid=14, entry_tid=12127' = 'object_number=87, tag_tid=14, entry_tid=12047' 2024-04-06T09:40:39.620 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=87, tag_tid=14, entry_tid=12127' 2024-04-06T09:40:39.620 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=93, tag_tid=15, entry_tid=1' 2024-04-06T09:40:39.620 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:40:39.620 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:40:39.621 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=15 2024-04-06T09:40:39.623 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=87, tag_tid=14, entry_tid=12127' 2024-04-06T09:40:39.623 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:40:39.623 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:40:39.623 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=14 2024-04-06T09:40:39.623 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=93, tag_tid=15, entry_tid=1' 2024-04-06T09:40:39.623 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:40:39.624 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:40:39.624 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:40:39.625 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=87, tag_tid=14, entry_tid=12127' 2024-04-06T09:40:39.625 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:40:39.626 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:40:39.626 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=12127 2024-04-06T09:40:39.626 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 15 = 14 -a 1 -le 12127 2024-04-06T09:40:39.627 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:40:39.627 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:40:39.627 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:40:39.826 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:40:39.827 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:40:39.827 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:40:39.827 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:40:39.827 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:40:39.827 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:40:39.827 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:40:39.827 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:40:39.827 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:40:39.827 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:40:39.827 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:40:39.827 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:40:39.828 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:40:39.828 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:40:39.828 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:40:39.828 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:40:39.828 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:40:39.832 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:40:39.832 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:40:39.832 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:40:39.832 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:40:39.832 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:40:39.832 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:40:39.832 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:40:39.832 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:40:39.832 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:40:39.832 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:40:39.833 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:40:39.833 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:40:39.833 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:40:39.833 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:40:39.833 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:40:39.833 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:40:39.833 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:40:39.833 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:40:39.833 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:40:39.833 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:40:39.833 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:40:39.833 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:40:39.833 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:40:39.833 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:40:39.834 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:40:39.834 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:40:39.834 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:40:39.834 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:40:39.834 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:40:39.834 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:40:39.834 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:40:39.834 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:40:39.857 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:40:39.857 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:40:39.857 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:40:39.857 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:40:39.988 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:40:39.989 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:40:39.989 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:40:39.989 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:40:39.989 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:40:39.990 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:40:39.990 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:40:39.990 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:40:39.990 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:40:39.990 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:40:39.990 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:40:39.990 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:40:39.990 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:40:39.992 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:40:39.992 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:40:39.992 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:40:40.044 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 21 2024-04-06T09:40:40.044 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 23 2024-04-06T09:40:40.044 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:40:40.044 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=93, tag_tid=15, entry_tid=1], [object_number=92, tag_tid=15, entry_tid=0], [object_number=95, tag_tid=14, entry_tid=19039], [object_number=94, tag_tid=14, entry_tid=19038]]], state=connected] 2024-04-06T09:40:40.044 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=84, tag_tid=14, entry_tid=12208], [object_number=87, tag_tid=14, entry_tid=12207], [object_number=86, tag_tid=14, entry_tid=12206], [object_number=85, tag_tid=14, entry_tid=12205]]], state=connected] 2024-04-06T09:40:40.047 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:40:40.049 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=93, tag_tid=15, entry_tid=1' 2024-04-06T09:40:40.050 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:40:40.050 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:40:40.050 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:40:40.050 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:40:40.050 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:40:40.050 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:40:40.050 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:40:40.050 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:40:40.050 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:40:40.050 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:40:40.051 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:40:40.051 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:40:40.051 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:40:40.051 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:40:40.051 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:40:40.097 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 21 2024-04-06T09:40:40.097 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 23 2024-04-06T09:40:40.097 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:40:40.097 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=93, tag_tid=15, entry_tid=1], [object_number=92, tag_tid=15, entry_tid=0], [object_number=95, tag_tid=14, entry_tid=19039], [object_number=94, tag_tid=14, entry_tid=19038]]], state=connected] 2024-04-06T09:40:40.098 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=84, tag_tid=14, entry_tid=12208], [object_number=87, tag_tid=14, entry_tid=12207], [object_number=86, tag_tid=14, entry_tid=12206], [object_number=85, tag_tid=14, entry_tid=12205]]], state=connected] 2024-04-06T09:40:40.100 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:40:40.103 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=84, tag_tid=14, entry_tid=12208' 2024-04-06T09:40:40.103 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=93, tag_tid=15, entry_tid=1' -a 'object_number=93, tag_tid=15, entry_tid=1' = 'object_number=84, tag_tid=14, entry_tid=12208' 2024-04-06T09:40:40.103 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=84, tag_tid=14, entry_tid=12208' '!=' 'object_number=87, tag_tid=14, entry_tid=12127' 2024-04-06T09:40:40.103 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:40:40.103 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=84, tag_tid=14, entry_tid=12208' = 'object_number=87, tag_tid=14, entry_tid=12127' 2024-04-06T09:40:40.103 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=84, tag_tid=14, entry_tid=12208' 2024-04-06T09:40:40.103 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=93, tag_tid=15, entry_tid=1' 2024-04-06T09:40:40.103 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:40:40.103 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:40:40.104 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=15 2024-04-06T09:40:40.104 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=84, tag_tid=14, entry_tid=12208' 2024-04-06T09:40:40.105 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:40:40.105 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:40:40.105 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=14 2024-04-06T09:40:40.106 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=93, tag_tid=15, entry_tid=1' 2024-04-06T09:40:40.106 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:40:40.106 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:40:40.107 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:40:40.108 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=84, tag_tid=14, entry_tid=12208' 2024-04-06T09:40:40.108 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:40:40.108 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:40:40.109 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=12208 2024-04-06T09:40:40.109 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 15 = 14 -a 1 -le 12208 2024-04-06T09:40:40.109 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:40:40.109 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:40:40.110 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:40:40.313 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:40:40.313 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:40:40.313 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:40:40.313 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:40:40.314 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:40:40.314 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:40:40.314 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:40:40.314 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:40:40.314 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:40:40.314 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:40:40.314 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:40:40.314 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:40:40.315 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:40:40.315 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:40:40.315 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:40:40.315 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:40:40.315 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:40:40.319 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:40:40.319 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:40:40.319 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:40:40.319 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:40:40.319 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:40:40.320 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:40:40.320 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:40:40.320 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:40:40.320 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:40:40.320 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:40:40.320 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:40:40.320 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:40:40.320 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:40:40.321 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:40:40.321 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:40:40.321 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:40:40.321 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:40:40.321 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:40:40.321 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:40:40.321 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:40:40.321 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:40:40.321 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:40:40.322 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:40:40.322 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:40:40.322 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:40:40.322 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:40:40.322 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:40:40.322 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:40:40.322 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:40:40.322 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:40:40.322 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:40:40.323 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:40:40.341 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:40:40.342 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:40:40.342 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:40:40.342 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:40:40.475 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:40:40.476 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:40:40.476 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:40:40.476 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:40:40.476 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:40:40.476 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:40:40.476 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:40:40.477 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:40:40.477 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:40:40.477 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:40:40.477 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:40:40.477 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:40:40.477 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:40:40.480 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:40:40.480 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:40:40.480 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:40:40.527 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 21 2024-04-06T09:40:40.527 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 23 2024-04-06T09:40:40.527 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:40:40.527 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=93, tag_tid=15, entry_tid=1], [object_number=92, tag_tid=15, entry_tid=0], [object_number=95, tag_tid=14, entry_tid=19039], [object_number=94, tag_tid=14, entry_tid=19038]]], state=connected] 2024-04-06T09:40:40.527 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=85, tag_tid=14, entry_tid=12285], [object_number=84, tag_tid=14, entry_tid=12284], [object_number=87, tag_tid=14, entry_tid=12283], [object_number=86, tag_tid=14, entry_tid=12282]]], state=connected] 2024-04-06T09:40:40.531 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:40:40.532 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=93, tag_tid=15, entry_tid=1' 2024-04-06T09:40:40.533 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:40:40.533 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:40:40.533 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:40:40.533 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:40:40.533 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:40:40.533 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:40:40.534 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:40:40.534 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:40:40.534 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:40:40.534 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:40:40.534 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:40:40.534 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:40:40.534 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:40:40.535 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:40:40.535 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:40:40.580 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 21 2024-04-06T09:40:40.580 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 23 2024-04-06T09:40:40.581 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:40:40.581 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=93, tag_tid=15, entry_tid=1], [object_number=92, tag_tid=15, entry_tid=0], [object_number=95, tag_tid=14, entry_tid=19039], [object_number=94, tag_tid=14, entry_tid=19038]]], state=connected] 2024-04-06T09:40:40.581 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=85, tag_tid=14, entry_tid=12285], [object_number=84, tag_tid=14, entry_tid=12284], [object_number=87, tag_tid=14, entry_tid=12283], [object_number=86, tag_tid=14, entry_tid=12282]]], state=connected] 2024-04-06T09:40:40.584 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:40:40.586 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=85, tag_tid=14, entry_tid=12285' 2024-04-06T09:40:40.586 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=93, tag_tid=15, entry_tid=1' -a 'object_number=93, tag_tid=15, entry_tid=1' = 'object_number=85, tag_tid=14, entry_tid=12285' 2024-04-06T09:40:40.586 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=85, tag_tid=14, entry_tid=12285' '!=' 'object_number=84, tag_tid=14, entry_tid=12208' 2024-04-06T09:40:40.586 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:40:40.586 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=85, tag_tid=14, entry_tid=12285' = 'object_number=84, tag_tid=14, entry_tid=12208' 2024-04-06T09:40:40.586 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=85, tag_tid=14, entry_tid=12285' 2024-04-06T09:40:40.586 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=93, tag_tid=15, entry_tid=1' 2024-04-06T09:40:40.586 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:40:40.587 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:40:40.587 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=15 2024-04-06T09:40:40.588 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=85, tag_tid=14, entry_tid=12285' 2024-04-06T09:40:40.588 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:40:40.588 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:40:40.589 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=14 2024-04-06T09:40:40.589 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=93, tag_tid=15, entry_tid=1' 2024-04-06T09:40:40.589 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:40:40.590 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:40:40.590 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:40:40.591 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=85, tag_tid=14, entry_tid=12285' 2024-04-06T09:40:40.591 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:40:40.591 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:40:40.592 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=12285 2024-04-06T09:40:40.592 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 15 = 14 -a 1 -le 12285 2024-04-06T09:40:40.593 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:40:40.593 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:40:40.593 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:40:40.794 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:40:40.794 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:40:40.795 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:40:40.795 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:40:40.795 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:40:40.795 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:40:40.795 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:40:40.795 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:40:40.795 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:40:40.795 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:40:40.795 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:40:40.795 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:40:40.795 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:40:40.795 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:40:40.795 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:40:40.796 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:40:40.796 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:40:40.799 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:40:40.799 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:40:40.799 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:40:40.799 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:40:40.800 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:40:40.800 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:40:40.800 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:40:40.800 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:40:40.800 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:40:40.800 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:40:40.800 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:40:40.800 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:40:40.800 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:40:40.800 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:40:40.800 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:40:40.800 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:40:40.800 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:40:40.800 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:40:40.800 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:40:40.801 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:40:40.801 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:40:40.801 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:40:40.801 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:40:40.801 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:40:40.801 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:40:40.801 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:40:40.801 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:40:40.801 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:40:40.801 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:40:40.801 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:40:40.801 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:40:40.801 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:40:40.824 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:40:40.825 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:40:40.825 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:40:40.825 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:40:40.947 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:40:40.949 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:40:40.949 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:40:40.949 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:40:40.949 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:40:40.949 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:40:40.949 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:40:40.949 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:40:40.949 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:40:40.949 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:40:40.950 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:40:40.950 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:40:40.950 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:40:40.951 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:40:40.952 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:40:40.952 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:40:41.005 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 21 2024-04-06T09:40:41.005 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 23 2024-04-06T09:40:41.005 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:40:41.005 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=93, tag_tid=15, entry_tid=1], [object_number=92, tag_tid=15, entry_tid=0], [object_number=95, tag_tid=14, entry_tid=19039], [object_number=94, tag_tid=14, entry_tid=19038]]], state=connected] 2024-04-06T09:40:41.005 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=86, tag_tid=14, entry_tid=12366], [object_number=85, tag_tid=14, entry_tid=12365], [object_number=84, tag_tid=14, entry_tid=12364], [object_number=87, tag_tid=14, entry_tid=12363]]], state=connected] 2024-04-06T09:40:41.008 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:40:41.010 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=93, tag_tid=15, entry_tid=1' 2024-04-06T09:40:41.011 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:40:41.011 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:40:41.011 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:40:41.011 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:40:41.011 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:40:41.011 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:40:41.011 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:40:41.011 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:40:41.011 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:40:41.011 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:40:41.011 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:40:41.012 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:40:41.012 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:40:41.012 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:40:41.012 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:40:41.060 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 21 2024-04-06T09:40:41.060 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 23 2024-04-06T09:40:41.060 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:40:41.060 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=93, tag_tid=15, entry_tid=1], [object_number=92, tag_tid=15, entry_tid=0], [object_number=95, tag_tid=14, entry_tid=19039], [object_number=94, tag_tid=14, entry_tid=19038]]], state=connected] 2024-04-06T09:40:41.060 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=86, tag_tid=14, entry_tid=12366], [object_number=85, tag_tid=14, entry_tid=12365], [object_number=84, tag_tid=14, entry_tid=12364], [object_number=87, tag_tid=14, entry_tid=12363]]], state=connected] 2024-04-06T09:40:41.063 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:40:41.065 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=86, tag_tid=14, entry_tid=12366' 2024-04-06T09:40:41.065 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=93, tag_tid=15, entry_tid=1' -a 'object_number=93, tag_tid=15, entry_tid=1' = 'object_number=86, tag_tid=14, entry_tid=12366' 2024-04-06T09:40:41.065 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=86, tag_tid=14, entry_tid=12366' '!=' 'object_number=85, tag_tid=14, entry_tid=12285' 2024-04-06T09:40:41.065 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:40:41.066 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=86, tag_tid=14, entry_tid=12366' = 'object_number=85, tag_tid=14, entry_tid=12285' 2024-04-06T09:40:41.066 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=86, tag_tid=14, entry_tid=12366' 2024-04-06T09:40:41.066 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=93, tag_tid=15, entry_tid=1' 2024-04-06T09:40:41.066 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:40:41.066 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:40:41.066 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=15 2024-04-06T09:40:41.067 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=86, tag_tid=14, entry_tid=12366' 2024-04-06T09:40:41.067 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:40:41.067 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:40:41.068 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=14 2024-04-06T09:40:41.069 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=93, tag_tid=15, entry_tid=1' 2024-04-06T09:40:41.069 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:40:41.069 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:40:41.070 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:40:41.070 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=86, tag_tid=14, entry_tid=12366' 2024-04-06T09:40:41.071 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:40:41.071 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:40:41.072 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=12366 2024-04-06T09:40:41.072 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 15 = 14 -a 1 -le 12366 2024-04-06T09:40:41.072 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:40:41.072 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:40:41.072 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:40:41.274 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:40:41.274 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:40:41.274 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:40:41.274 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:40:41.274 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:40:41.275 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:40:41.275 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:40:41.275 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:40:41.275 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:40:41.275 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:40:41.275 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:40:41.275 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:40:41.275 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:40:41.275 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:40:41.275 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:40:41.276 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:40:41.276 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:40:41.278 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:40:41.278 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:40:41.278 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:40:41.278 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:40:41.279 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:40:41.279 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:40:41.279 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:40:41.279 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:40:41.279 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:40:41.279 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:40:41.279 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:40:41.279 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:40:41.279 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:40:41.279 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:40:41.279 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:40:41.279 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:40:41.279 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:40:41.279 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:40:41.279 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:40:41.279 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:40:41.280 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:40:41.280 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:40:41.280 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:40:41.280 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:40:41.280 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:40:41.280 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:40:41.280 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:40:41.280 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:40:41.280 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:40:41.280 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:40:41.280 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:40:41.280 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:40:41.305 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:40:41.305 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:40:41.305 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:40:41.305 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:40:41.434 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:40:41.436 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:40:41.436 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:40:41.436 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:40:41.436 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:40:41.436 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:40:41.436 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:40:41.436 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:40:41.436 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:40:41.436 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:40:41.436 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:40:41.437 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:40:41.437 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:40:41.439 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:40:41.440 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:40:41.440 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:40:41.491 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 21 2024-04-06T09:40:41.491 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 23 2024-04-06T09:40:41.491 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:40:41.491 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=93, tag_tid=15, entry_tid=1], [object_number=92, tag_tid=15, entry_tid=0], [object_number=95, tag_tid=14, entry_tid=19039], [object_number=94, tag_tid=14, entry_tid=19038]]], state=connected] 2024-04-06T09:40:41.491 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=85, tag_tid=14, entry_tid=12449], [object_number=84, tag_tid=14, entry_tid=12448], [object_number=87, tag_tid=14, entry_tid=12447], [object_number=86, tag_tid=14, entry_tid=12446]]], state=connected] 2024-04-06T09:40:41.495 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:40:41.496 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=93, tag_tid=15, entry_tid=1' 2024-04-06T09:40:41.497 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:40:41.497 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:40:41.497 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:40:41.497 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:40:41.497 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:40:41.497 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:40:41.497 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:40:41.497 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:40:41.497 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:40:41.497 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:40:41.498 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:40:41.498 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:40:41.498 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:40:41.499 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:40:41.499 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:40:41.546 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 21 2024-04-06T09:40:41.546 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 23 2024-04-06T09:40:41.546 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:40:41.546 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=93, tag_tid=15, entry_tid=1], [object_number=92, tag_tid=15, entry_tid=0], [object_number=95, tag_tid=14, entry_tid=19039], [object_number=94, tag_tid=14, entry_tid=19038]]], state=connected] 2024-04-06T09:40:41.546 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=85, tag_tid=14, entry_tid=12449], [object_number=84, tag_tid=14, entry_tid=12448], [object_number=87, tag_tid=14, entry_tid=12447], [object_number=86, tag_tid=14, entry_tid=12446]]], state=connected] 2024-04-06T09:40:41.549 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:40:41.551 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=85, tag_tid=14, entry_tid=12449' 2024-04-06T09:40:41.551 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=93, tag_tid=15, entry_tid=1' -a 'object_number=93, tag_tid=15, entry_tid=1' = 'object_number=85, tag_tid=14, entry_tid=12449' 2024-04-06T09:40:41.551 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=85, tag_tid=14, entry_tid=12449' '!=' 'object_number=86, tag_tid=14, entry_tid=12366' 2024-04-06T09:40:41.551 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:40:41.551 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=85, tag_tid=14, entry_tid=12449' = 'object_number=86, tag_tid=14, entry_tid=12366' 2024-04-06T09:40:41.552 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=85, tag_tid=14, entry_tid=12449' 2024-04-06T09:40:41.552 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=93, tag_tid=15, entry_tid=1' 2024-04-06T09:40:41.552 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:40:41.552 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:40:41.552 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=15 2024-04-06T09:40:41.553 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=85, tag_tid=14, entry_tid=12449' 2024-04-06T09:40:41.553 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:40:41.553 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:40:41.554 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=14 2024-04-06T09:40:41.555 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=93, tag_tid=15, entry_tid=1' 2024-04-06T09:40:41.555 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:40:41.555 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:40:41.556 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:40:41.557 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=85, tag_tid=14, entry_tid=12449' 2024-04-06T09:40:41.557 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:40:41.557 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:40:41.558 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=12449 2024-04-06T09:40:41.558 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 15 = 14 -a 1 -le 12449 2024-04-06T09:40:41.558 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:40:41.558 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:40:41.558 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:40:41.760 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:40:41.760 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:40:41.760 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:40:41.761 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:40:41.761 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:40:41.761 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:40:41.761 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:40:41.761 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:40:41.761 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:40:41.761 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:40:41.761 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:40:41.761 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:40:41.761 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:40:41.761 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:40:41.761 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:40:41.762 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:40:41.762 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:40:41.765 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:40:41.765 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:40:41.766 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:40:41.766 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:40:41.766 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:40:41.766 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:40:41.766 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:40:41.766 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:40:41.766 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:40:41.766 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:40:41.766 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:40:41.766 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:40:41.766 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:40:41.766 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:40:41.767 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:40:41.767 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:40:41.767 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:40:41.767 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:40:41.767 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:40:41.767 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:40:41.767 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:40:41.767 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:40:41.767 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:40:41.767 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:40:41.767 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:40:41.767 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:40:41.767 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:40:41.767 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:40:41.768 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:40:41.768 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:40:41.768 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:40:41.768 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:40:41.793 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:40:41.794 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:40:41.794 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:40:41.794 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:40:41.923 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:40:41.924 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:40:41.924 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:40:41.924 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:40:41.924 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:40:41.924 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:40:41.924 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:40:41.924 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:40:41.924 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:40:41.924 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:40:41.925 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:40:41.925 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:40:41.925 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:40:41.927 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:40:41.927 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:40:41.927 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:40:41.979 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 21 2024-04-06T09:40:41.979 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 23 2024-04-06T09:40:41.979 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:40:41.979 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=93, tag_tid=15, entry_tid=1], [object_number=92, tag_tid=15, entry_tid=0], [object_number=95, tag_tid=14, entry_tid=19039], [object_number=94, tag_tid=14, entry_tid=19038]]], state=connected] 2024-04-06T09:40:41.980 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=84, tag_tid=14, entry_tid=12532], [object_number=87, tag_tid=14, entry_tid=12531], [object_number=86, tag_tid=14, entry_tid=12530], [object_number=85, tag_tid=14, entry_tid=12529]]], state=connected] 2024-04-06T09:40:41.983 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:40:41.984 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=93, tag_tid=15, entry_tid=1' 2024-04-06T09:40:41.985 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:40:41.985 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:40:41.985 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:40:41.985 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:40:41.985 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:40:41.986 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:40:41.986 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:40:41.986 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:40:41.986 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:40:41.986 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:40:41.986 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:40:41.986 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:40:41.987 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:40:41.987 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:40:41.987 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:40:42.032 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 21 2024-04-06T09:40:42.033 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 23 2024-04-06T09:40:42.033 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:40:42.033 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=93, tag_tid=15, entry_tid=1], [object_number=92, tag_tid=15, entry_tid=0], [object_number=95, tag_tid=14, entry_tid=19039], [object_number=94, tag_tid=14, entry_tid=19038]]], state=connected] 2024-04-06T09:40:42.033 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=84, tag_tid=14, entry_tid=12532], [object_number=87, tag_tid=14, entry_tid=12531], [object_number=86, tag_tid=14, entry_tid=12530], [object_number=85, tag_tid=14, entry_tid=12529]]], state=connected] 2024-04-06T09:40:42.036 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:40:42.038 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=84, tag_tid=14, entry_tid=12532' 2024-04-06T09:40:42.038 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=93, tag_tid=15, entry_tid=1' -a 'object_number=93, tag_tid=15, entry_tid=1' = 'object_number=84, tag_tid=14, entry_tid=12532' 2024-04-06T09:40:42.038 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=84, tag_tid=14, entry_tid=12532' '!=' 'object_number=85, tag_tid=14, entry_tid=12449' 2024-04-06T09:40:42.039 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:40:42.039 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=84, tag_tid=14, entry_tid=12532' = 'object_number=85, tag_tid=14, entry_tid=12449' 2024-04-06T09:40:42.039 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=84, tag_tid=14, entry_tid=12532' 2024-04-06T09:40:42.039 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=93, tag_tid=15, entry_tid=1' 2024-04-06T09:40:42.039 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:40:42.039 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:40:42.039 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=15 2024-04-06T09:40:42.040 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=84, tag_tid=14, entry_tid=12532' 2024-04-06T09:40:42.040 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:40:42.040 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:40:42.041 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=14 2024-04-06T09:40:42.042 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=93, tag_tid=15, entry_tid=1' 2024-04-06T09:40:42.042 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:40:42.042 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:40:42.043 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:40:42.043 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=84, tag_tid=14, entry_tid=12532' 2024-04-06T09:40:42.043 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:40:42.043 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:40:42.045 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=12532 2024-04-06T09:40:42.045 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 15 = 14 -a 1 -le 12532 2024-04-06T09:40:42.045 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:40:42.045 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:40:42.045 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:40:42.247 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:40:42.247 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:40:42.247 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:40:42.247 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:40:42.247 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:40:42.247 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:40:42.247 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:40:42.247 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:40:42.247 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:40:42.247 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:40:42.247 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:40:42.248 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:40:42.248 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:40:42.248 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:40:42.248 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:40:42.248 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:40:42.248 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:40:42.252 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:40:42.252 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:40:42.252 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:40:42.252 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:40:42.252 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:40:42.252 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:40:42.252 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:40:42.252 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:40:42.253 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:40:42.253 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:40:42.253 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:40:42.253 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:40:42.253 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:40:42.253 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:40:42.253 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:40:42.253 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:40:42.253 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:40:42.253 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:40:42.253 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:40:42.253 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:40:42.253 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:40:42.253 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:40:42.253 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:40:42.254 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:40:42.254 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:40:42.254 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:40:42.254 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:40:42.254 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:40:42.254 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:40:42.254 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:40:42.254 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:40:42.254 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:40:42.278 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:40:42.278 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:40:42.278 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:40:42.278 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:40:42.409 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:40:42.409 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:40:42.409 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:40:42.410 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:40:42.410 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:40:42.410 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:40:42.410 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:40:42.410 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:40:42.410 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:40:42.410 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:40:42.411 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:40:42.411 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:40:42.411 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:40:42.411 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:40:42.412 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:40:42.412 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:40:42.471 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 21 2024-04-06T09:40:42.471 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 23 2024-04-06T09:40:42.471 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:40:42.472 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=93, tag_tid=15, entry_tid=1], [object_number=92, tag_tid=15, entry_tid=0], [object_number=95, tag_tid=14, entry_tid=19039], [object_number=94, tag_tid=14, entry_tid=19038]]], state=connected] 2024-04-06T09:40:42.472 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=84, tag_tid=14, entry_tid=12608], [object_number=87, tag_tid=14, entry_tid=12607], [object_number=86, tag_tid=14, entry_tid=12606], [object_number=85, tag_tid=14, entry_tid=12605]]], state=connected] 2024-04-06T09:40:42.475 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:40:42.476 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=93, tag_tid=15, entry_tid=1' 2024-04-06T09:40:42.478 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:40:42.478 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:40:42.478 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:40:42.478 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:40:42.478 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:40:42.478 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:40:42.478 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:40:42.478 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:40:42.478 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:40:42.478 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:40:42.478 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:40:42.479 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:40:42.479 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:40:42.479 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:40:42.479 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:40:42.532 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 21 2024-04-06T09:40:42.532 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 23 2024-04-06T09:40:42.532 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:40:42.532 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=93, tag_tid=15, entry_tid=1], [object_number=92, tag_tid=15, entry_tid=0], [object_number=95, tag_tid=14, entry_tid=19039], [object_number=94, tag_tid=14, entry_tid=19038]]], state=connected] 2024-04-06T09:40:42.532 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=84, tag_tid=14, entry_tid=12608], [object_number=87, tag_tid=14, entry_tid=12607], [object_number=86, tag_tid=14, entry_tid=12606], [object_number=85, tag_tid=14, entry_tid=12605]]], state=connected] 2024-04-06T09:40:42.536 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:40:42.538 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=84, tag_tid=14, entry_tid=12608' 2024-04-06T09:40:42.538 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=93, tag_tid=15, entry_tid=1' -a 'object_number=93, tag_tid=15, entry_tid=1' = 'object_number=84, tag_tid=14, entry_tid=12608' 2024-04-06T09:40:42.538 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=84, tag_tid=14, entry_tid=12608' '!=' 'object_number=84, tag_tid=14, entry_tid=12532' 2024-04-06T09:40:42.538 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:40:42.538 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=84, tag_tid=14, entry_tid=12608' = 'object_number=84, tag_tid=14, entry_tid=12532' 2024-04-06T09:40:42.538 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=84, tag_tid=14, entry_tid=12608' 2024-04-06T09:40:42.539 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=93, tag_tid=15, entry_tid=1' 2024-04-06T09:40:42.539 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:40:42.539 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:40:42.540 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=15 2024-04-06T09:40:42.540 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=84, tag_tid=14, entry_tid=12608' 2024-04-06T09:40:42.540 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:40:42.541 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:40:42.541 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=14 2024-04-06T09:40:42.542 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=93, tag_tid=15, entry_tid=1' 2024-04-06T09:40:42.542 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:40:42.542 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:40:42.543 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:40:42.544 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=84, tag_tid=14, entry_tid=12608' 2024-04-06T09:40:42.544 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:40:42.544 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:40:42.545 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=12608 2024-04-06T09:40:42.545 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 15 = 14 -a 1 -le 12608 2024-04-06T09:40:42.545 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:40:42.545 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:40:42.545 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:40:42.747 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:40:42.748 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:40:42.748 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:40:42.748 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:40:42.748 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:40:42.748 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:40:42.748 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:40:42.748 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:40:42.748 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:40:42.748 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:40:42.748 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:40:42.748 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:40:42.748 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:40:42.748 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:40:42.748 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:40:42.749 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:40:42.749 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:40:42.751 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:40:42.751 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:40:42.751 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:40:42.751 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:40:42.751 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:40:42.751 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:40:42.751 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:40:42.751 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:40:42.751 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:40:42.752 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:40:42.752 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:40:42.752 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:40:42.752 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:40:42.752 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:40:42.752 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:40:42.752 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:40:42.753 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:40:42.753 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:40:42.753 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:40:42.753 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:40:42.753 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:40:42.753 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:40:42.753 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:40:42.754 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:40:42.754 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:40:42.754 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:40:42.754 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:40:42.754 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:40:42.754 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:40:42.754 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:40:42.754 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:40:42.754 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:40:42.778 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:40:42.778 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:40:42.778 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:40:42.779 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:40:42.919 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:40:42.920 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:40:42.920 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:40:42.920 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:40:42.920 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:40:42.920 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:40:42.920 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:40:42.920 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:40:42.920 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:40:42.920 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:40:42.920 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:40:42.921 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:40:42.922 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:40:42.923 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:40:42.923 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:40:42.923 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:40:42.994 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 22 2024-04-06T09:40:42.994 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 23 2024-04-06T09:40:42.994 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:40:42.995 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=93, tag_tid=15, entry_tid=1], [object_number=92, tag_tid=15, entry_tid=0], [object_number=95, tag_tid=14, entry_tid=19039], [object_number=94, tag_tid=14, entry_tid=19038]]], state=connected] 2024-04-06T09:40:42.995 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=88, tag_tid=14, entry_tid=12676], [object_number=91, tag_tid=14, entry_tid=12675], [object_number=90, tag_tid=14, entry_tid=12674], [object_number=89, tag_tid=14, entry_tid=12673]]], state=connected] 2024-04-06T09:40:42.998 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:40:42.999 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=93, tag_tid=15, entry_tid=1' 2024-04-06T09:40:43.000 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:40:43.000 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:40:43.000 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:40:43.000 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:40:43.001 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:40:43.001 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:40:43.001 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:40:43.001 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:40:43.001 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:40:43.001 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:40:43.001 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:40:43.001 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:40:43.002 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:40:43.002 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:40:43.002 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:40:43.048 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 22 2024-04-06T09:40:43.048 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 23 2024-04-06T09:40:43.048 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:40:43.048 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=93, tag_tid=15, entry_tid=1], [object_number=92, tag_tid=15, entry_tid=0], [object_number=95, tag_tid=14, entry_tid=19039], [object_number=94, tag_tid=14, entry_tid=19038]]], state=connected] 2024-04-06T09:40:43.048 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=88, tag_tid=14, entry_tid=12676], [object_number=91, tag_tid=14, entry_tid=12675], [object_number=90, tag_tid=14, entry_tid=12674], [object_number=89, tag_tid=14, entry_tid=12673]]], state=connected] 2024-04-06T09:40:43.051 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:40:43.053 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=88, tag_tid=14, entry_tid=12676' 2024-04-06T09:40:43.053 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=93, tag_tid=15, entry_tid=1' -a 'object_number=93, tag_tid=15, entry_tid=1' = 'object_number=88, tag_tid=14, entry_tid=12676' 2024-04-06T09:40:43.054 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=88, tag_tid=14, entry_tid=12676' '!=' 'object_number=84, tag_tid=14, entry_tid=12608' 2024-04-06T09:40:43.054 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:40:43.054 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=88, tag_tid=14, entry_tid=12676' = 'object_number=84, tag_tid=14, entry_tid=12608' 2024-04-06T09:40:43.054 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=88, tag_tid=14, entry_tid=12676' 2024-04-06T09:40:43.054 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=93, tag_tid=15, entry_tid=1' 2024-04-06T09:40:43.054 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:40:43.054 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:40:43.054 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=15 2024-04-06T09:40:43.055 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=88, tag_tid=14, entry_tid=12676' 2024-04-06T09:40:43.055 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:40:43.055 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:40:43.056 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=14 2024-04-06T09:40:43.057 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=93, tag_tid=15, entry_tid=1' 2024-04-06T09:40:43.057 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:40:43.057 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:40:43.058 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:40:43.059 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=88, tag_tid=14, entry_tid=12676' 2024-04-06T09:40:43.059 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:40:43.059 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:40:43.060 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=12676 2024-04-06T09:40:43.060 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 15 = 14 -a 1 -le 12676 2024-04-06T09:40:43.060 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:40:43.060 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:40:43.060 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:40:43.262 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:40:43.262 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:40:43.262 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:40:43.262 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:40:43.263 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:40:43.263 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:40:43.263 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:40:43.263 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:40:43.263 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:40:43.263 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:40:43.263 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:40:43.263 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:40:43.263 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:40:43.263 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:40:43.263 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:40:43.264 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:40:43.264 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:40:43.267 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:40:43.267 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:40:43.267 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:40:43.267 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:40:43.267 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:40:43.267 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:40:43.267 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:40:43.267 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:40:43.268 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:40:43.268 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:40:43.268 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:40:43.268 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:40:43.268 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:40:43.268 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:40:43.268 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:40:43.268 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:40:43.268 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:40:43.268 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:40:43.268 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:40:43.268 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:40:43.268 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:40:43.268 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:40:43.269 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:40:43.269 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:40:43.269 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:40:43.269 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:40:43.269 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:40:43.269 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:40:43.269 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:40:43.269 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:40:43.269 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:40:43.269 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:40:43.292 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:40:43.292 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:40:43.292 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:40:43.292 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:40:43.429 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:40:43.429 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:40:43.429 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:40:43.430 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:40:43.430 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:40:43.430 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:40:43.430 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:40:43.430 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:40:43.430 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:40:43.430 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:40:43.431 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:40:43.431 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:40:43.431 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:40:43.433 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:40:43.433 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:40:43.433 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:40:43.485 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 22 2024-04-06T09:40:43.485 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 23 2024-04-06T09:40:43.485 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:40:43.485 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=93, tag_tid=15, entry_tid=1], [object_number=92, tag_tid=15, entry_tid=0], [object_number=95, tag_tid=14, entry_tid=19039], [object_number=94, tag_tid=14, entry_tid=19038]]], state=connected] 2024-04-06T09:40:43.485 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=90, tag_tid=14, entry_tid=12774], [object_number=89, tag_tid=14, entry_tid=12773], [object_number=88, tag_tid=14, entry_tid=12772], [object_number=91, tag_tid=14, entry_tid=12771]]], state=connected] 2024-04-06T09:40:43.488 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:40:43.490 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=93, tag_tid=15, entry_tid=1' 2024-04-06T09:40:43.491 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:40:43.491 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:40:43.491 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:40:43.491 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:40:43.491 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:40:43.492 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:40:43.492 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:40:43.492 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:40:43.492 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:40:43.492 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:40:43.492 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:40:43.492 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:40:43.493 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:40:43.493 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:40:43.493 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:40:43.538 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 22 2024-04-06T09:40:43.538 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 23 2024-04-06T09:40:43.538 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:40:43.538 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=93, tag_tid=15, entry_tid=1], [object_number=92, tag_tid=15, entry_tid=0], [object_number=95, tag_tid=14, entry_tid=19039], [object_number=94, tag_tid=14, entry_tid=19038]]], state=connected] 2024-04-06T09:40:43.539 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=90, tag_tid=14, entry_tid=12774], [object_number=89, tag_tid=14, entry_tid=12773], [object_number=88, tag_tid=14, entry_tid=12772], [object_number=91, tag_tid=14, entry_tid=12771]]], state=connected] 2024-04-06T09:40:43.542 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:40:43.544 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=90, tag_tid=14, entry_tid=12774' 2024-04-06T09:40:43.544 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=93, tag_tid=15, entry_tid=1' -a 'object_number=93, tag_tid=15, entry_tid=1' = 'object_number=90, tag_tid=14, entry_tid=12774' 2024-04-06T09:40:43.544 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=90, tag_tid=14, entry_tid=12774' '!=' 'object_number=88, tag_tid=14, entry_tid=12676' 2024-04-06T09:40:43.544 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:40:43.544 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=90, tag_tid=14, entry_tid=12774' = 'object_number=88, tag_tid=14, entry_tid=12676' 2024-04-06T09:40:43.544 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=90, tag_tid=14, entry_tid=12774' 2024-04-06T09:40:43.544 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=93, tag_tid=15, entry_tid=1' 2024-04-06T09:40:43.545 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:40:43.545 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:40:43.545 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=15 2024-04-06T09:40:43.546 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=90, tag_tid=14, entry_tid=12774' 2024-04-06T09:40:43.546 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:40:43.546 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:40:43.547 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=14 2024-04-06T09:40:43.547 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=93, tag_tid=15, entry_tid=1' 2024-04-06T09:40:43.547 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:40:43.547 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:40:43.548 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:40:43.549 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=90, tag_tid=14, entry_tid=12774' 2024-04-06T09:40:43.549 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:40:43.549 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:40:43.550 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=12774 2024-04-06T09:40:43.550 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 15 = 14 -a 1 -le 12774 2024-04-06T09:40:43.551 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:40:43.551 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:40:43.551 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:40:43.752 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:40:43.752 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:40:43.752 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:40:43.752 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:40:43.753 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:40:43.753 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:40:43.753 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:40:43.753 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:40:43.753 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:40:43.753 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:40:43.753 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:40:43.753 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:40:43.753 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:40:43.753 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:40:43.753 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:40:43.754 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:40:43.754 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:40:43.756 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:40:43.756 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:40:43.756 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:40:43.756 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:40:43.757 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:40:43.757 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:40:43.757 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:40:43.757 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:40:43.757 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:40:43.757 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:40:43.757 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:40:43.757 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:40:43.757 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:40:43.757 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:40:43.757 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:40:43.757 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:40:43.758 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:40:43.758 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:40:43.758 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:40:43.758 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:40:43.759 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:40:43.759 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:40:43.759 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:40:43.759 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:40:43.759 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:40:43.759 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:40:43.759 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:40:43.759 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:40:43.759 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:40:43.759 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:40:43.759 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:40:43.759 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:40:43.783 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:40:43.784 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:40:43.784 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:40:43.784 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:40:43.918 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:40:43.919 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:40:43.919 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:40:43.919 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:40:43.919 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:40:43.919 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:40:43.920 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:40:43.920 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:40:43.920 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:40:43.920 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:40:43.920 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:40:43.920 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:40:43.920 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:40:43.924 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:40:43.924 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:40:43.924 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:40:44.175 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 22 2024-04-06T09:40:44.175 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 23 2024-04-06T09:40:44.175 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:40:44.175 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=93, tag_tid=15, entry_tid=1], [object_number=92, tag_tid=15, entry_tid=0], [object_number=95, tag_tid=14, entry_tid=19039], [object_number=94, tag_tid=14, entry_tid=19038]]], state=connected] 2024-04-06T09:40:44.175 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=90, tag_tid=14, entry_tid=12866], [object_number=89, tag_tid=14, entry_tid=12865], [object_number=88, tag_tid=14, entry_tid=12864], [object_number=91, tag_tid=14, entry_tid=12863]]], state=connected] 2024-04-06T09:40:44.179 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:40:44.181 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=93, tag_tid=15, entry_tid=1' 2024-04-06T09:40:44.182 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:40:44.182 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:40:44.182 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:40:44.182 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:40:44.182 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:40:44.182 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:40:44.182 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:40:44.182 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:40:44.182 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:40:44.182 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:40:44.183 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:40:44.183 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:40:44.184 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:40:44.184 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:40:44.184 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:40:44.242 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 22 2024-04-06T09:40:44.242 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 23 2024-04-06T09:40:44.242 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:40:44.242 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=93, tag_tid=15, entry_tid=1], [object_number=92, tag_tid=15, entry_tid=0], [object_number=95, tag_tid=14, entry_tid=19039], [object_number=94, tag_tid=14, entry_tid=19038]]], state=connected] 2024-04-06T09:40:44.242 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=90, tag_tid=14, entry_tid=12866], [object_number=89, tag_tid=14, entry_tid=12865], [object_number=88, tag_tid=14, entry_tid=12864], [object_number=91, tag_tid=14, entry_tid=12863]]], state=connected] 2024-04-06T09:40:44.245 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:40:44.247 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=90, tag_tid=14, entry_tid=12866' 2024-04-06T09:40:44.247 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=93, tag_tid=15, entry_tid=1' -a 'object_number=93, tag_tid=15, entry_tid=1' = 'object_number=90, tag_tid=14, entry_tid=12866' 2024-04-06T09:40:44.247 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=90, tag_tid=14, entry_tid=12866' '!=' 'object_number=90, tag_tid=14, entry_tid=12774' 2024-04-06T09:40:44.248 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:40:44.248 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=90, tag_tid=14, entry_tid=12866' = 'object_number=90, tag_tid=14, entry_tid=12774' 2024-04-06T09:40:44.248 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=90, tag_tid=14, entry_tid=12866' 2024-04-06T09:40:44.248 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=93, tag_tid=15, entry_tid=1' 2024-04-06T09:40:44.248 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:40:44.248 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:40:44.249 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=15 2024-04-06T09:40:44.250 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=90, tag_tid=14, entry_tid=12866' 2024-04-06T09:40:44.250 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:40:44.250 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:40:44.251 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=14 2024-04-06T09:40:44.252 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=93, tag_tid=15, entry_tid=1' 2024-04-06T09:40:44.252 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:40:44.252 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:40:44.253 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:40:44.253 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=90, tag_tid=14, entry_tid=12866' 2024-04-06T09:40:44.254 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:40:44.254 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:40:44.255 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=12866 2024-04-06T09:40:44.256 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 15 = 14 -a 1 -le 12866 2024-04-06T09:40:44.256 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:40:44.256 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:40:44.256 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:40:44.457 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:40:44.457 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:40:44.457 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:40:44.457 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:40:44.457 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:40:44.457 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:40:44.457 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:40:44.457 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:40:44.457 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:40:44.457 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:40:44.457 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:40:44.458 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:40:44.458 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:40:44.458 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:40:44.458 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:40:44.458 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:40:44.459 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:40:44.461 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:40:44.461 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:40:44.461 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:40:44.461 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:40:44.461 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:40:44.461 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:40:44.461 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:40:44.461 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:40:44.461 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:40:44.461 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:40:44.461 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:40:44.461 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:40:44.461 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:40:44.462 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:40:44.462 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:40:44.462 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:40:44.463 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:40:44.463 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:40:44.463 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:40:44.463 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:40:44.463 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:40:44.463 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:40:44.463 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:40:44.463 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:40:44.463 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:40:44.463 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:40:44.463 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:40:44.463 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:40:44.463 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:40:44.464 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:40:44.464 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:40:44.464 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:40:44.489 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:40:44.490 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:40:44.490 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:40:44.490 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:40:44.617 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:40:44.619 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:40:44.619 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:40:44.619 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:40:44.619 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:40:44.619 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:40:44.619 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:40:44.619 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:40:44.619 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:40:44.619 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:40:44.619 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:40:44.620 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:40:44.620 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:40:44.622 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:40:44.622 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:40:44.622 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:40:44.668 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 22 2024-04-06T09:40:44.668 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 23 2024-04-06T09:40:44.668 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:40:44.669 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=93, tag_tid=15, entry_tid=1], [object_number=92, tag_tid=15, entry_tid=0], [object_number=95, tag_tid=14, entry_tid=19039], [object_number=94, tag_tid=14, entry_tid=19038]]], state=connected] 2024-04-06T09:40:44.669 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=89, tag_tid=14, entry_tid=13005], [object_number=88, tag_tid=14, entry_tid=13004], [object_number=91, tag_tid=14, entry_tid=13003], [object_number=90, tag_tid=14, entry_tid=13002]]], state=connected] 2024-04-06T09:40:44.672 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:40:44.673 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=93, tag_tid=15, entry_tid=1' 2024-04-06T09:40:44.674 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:40:44.674 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:40:44.674 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:40:44.675 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:40:44.675 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:40:44.675 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:40:44.675 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:40:44.675 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:40:44.675 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:40:44.675 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:40:44.675 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:40:44.675 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:40:44.676 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:40:44.676 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:40:44.676 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:40:44.723 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 22 2024-04-06T09:40:44.723 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 23 2024-04-06T09:40:44.723 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:40:44.724 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=93, tag_tid=15, entry_tid=1], [object_number=92, tag_tid=15, entry_tid=0], [object_number=95, tag_tid=14, entry_tid=19039], [object_number=94, tag_tid=14, entry_tid=19038]]], state=connected] 2024-04-06T09:40:44.724 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=89, tag_tid=14, entry_tid=13005], [object_number=88, tag_tid=14, entry_tid=13004], [object_number=91, tag_tid=14, entry_tid=13003], [object_number=90, tag_tid=14, entry_tid=13002]]], state=connected] 2024-04-06T09:40:44.727 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:40:44.728 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=89, tag_tid=14, entry_tid=13005' 2024-04-06T09:40:44.728 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=93, tag_tid=15, entry_tid=1' -a 'object_number=93, tag_tid=15, entry_tid=1' = 'object_number=89, tag_tid=14, entry_tid=13005' 2024-04-06T09:40:44.729 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=89, tag_tid=14, entry_tid=13005' '!=' 'object_number=90, tag_tid=14, entry_tid=12866' 2024-04-06T09:40:44.729 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:40:44.729 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=89, tag_tid=14, entry_tid=13005' = 'object_number=90, tag_tid=14, entry_tid=12866' 2024-04-06T09:40:44.729 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=89, tag_tid=14, entry_tid=13005' 2024-04-06T09:40:44.729 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=93, tag_tid=15, entry_tid=1' 2024-04-06T09:40:44.729 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:40:44.730 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:40:44.730 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=15 2024-04-06T09:40:44.731 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=89, tag_tid=14, entry_tid=13005' 2024-04-06T09:40:44.731 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:40:44.731 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:40:44.732 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=14 2024-04-06T09:40:44.732 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=93, tag_tid=15, entry_tid=1' 2024-04-06T09:40:44.732 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:40:44.732 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:40:44.733 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:40:44.734 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=89, tag_tid=14, entry_tid=13005' 2024-04-06T09:40:44.734 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:40:44.734 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:40:44.736 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=13005 2024-04-06T09:40:44.736 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 15 = 14 -a 1 -le 13005 2024-04-06T09:40:44.736 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:40:44.736 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:40:44.736 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:40:44.938 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:40:44.938 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:40:44.938 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:40:44.938 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:40:44.938 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:40:44.938 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:40:44.938 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:40:44.938 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:40:44.938 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:40:44.938 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:40:44.939 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:40:44.939 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:40:44.939 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:40:44.939 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:40:44.939 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:40:44.939 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:40:44.939 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:40:44.941 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:40:44.941 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:40:44.941 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:40:44.941 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:40:44.941 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:40:44.941 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:40:44.941 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:40:44.942 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:40:44.942 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:40:44.942 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:40:44.942 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:40:44.942 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:40:44.942 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:40:44.942 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:40:44.942 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:40:44.942 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:40:44.943 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:40:44.944 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:40:44.944 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:40:44.944 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:40:44.944 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:40:44.944 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:40:44.944 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:40:44.944 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:40:44.944 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:40:44.944 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:40:44.944 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:40:44.945 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:40:44.945 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:40:44.945 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:40:44.945 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:40:44.945 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:40:44.969 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:40:44.969 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:40:44.969 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:40:44.969 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:40:45.105 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:40:45.106 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:40:45.106 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:40:45.106 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:40:45.106 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:40:45.106 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:40:45.106 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:40:45.106 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:40:45.106 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:40:45.107 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:40:45.107 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:40:45.107 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:40:45.107 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:40:45.108 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:40:45.108 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:40:45.108 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:40:45.163 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 22 2024-04-06T09:40:45.163 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 23 2024-04-06T09:40:45.163 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:40:45.163 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=93, tag_tid=15, entry_tid=1], [object_number=92, tag_tid=15, entry_tid=0], [object_number=95, tag_tid=14, entry_tid=19039], [object_number=94, tag_tid=14, entry_tid=19038]]], state=connected] 2024-04-06T09:40:45.164 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=88, tag_tid=14, entry_tid=13104], [object_number=91, tag_tid=14, entry_tid=13103], [object_number=90, tag_tid=14, entry_tid=13102], [object_number=89, tag_tid=14, entry_tid=13101]]], state=connected] 2024-04-06T09:40:45.167 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:40:45.168 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=93, tag_tid=15, entry_tid=1' 2024-04-06T09:40:45.169 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:40:45.169 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:40:45.169 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:40:45.169 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:40:45.169 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:40:45.169 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:40:45.169 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:40:45.170 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:40:45.170 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:40:45.170 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:40:45.170 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:40:45.170 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:40:45.170 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:40:45.171 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:40:45.171 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:40:45.218 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 22 2024-04-06T09:40:45.218 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 23 2024-04-06T09:40:45.218 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:40:45.218 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=93, tag_tid=15, entry_tid=1], [object_number=92, tag_tid=15, entry_tid=0], [object_number=95, tag_tid=14, entry_tid=19039], [object_number=94, tag_tid=14, entry_tid=19038]]], state=connected] 2024-04-06T09:40:45.218 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=88, tag_tid=14, entry_tid=13104], [object_number=91, tag_tid=14, entry_tid=13103], [object_number=90, tag_tid=14, entry_tid=13102], [object_number=89, tag_tid=14, entry_tid=13101]]], state=connected] 2024-04-06T09:40:45.222 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:40:45.223 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=88, tag_tid=14, entry_tid=13104' 2024-04-06T09:40:45.223 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=93, tag_tid=15, entry_tid=1' -a 'object_number=93, tag_tid=15, entry_tid=1' = 'object_number=88, tag_tid=14, entry_tid=13104' 2024-04-06T09:40:45.223 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=88, tag_tid=14, entry_tid=13104' '!=' 'object_number=89, tag_tid=14, entry_tid=13005' 2024-04-06T09:40:45.224 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:40:45.224 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=88, tag_tid=14, entry_tid=13104' = 'object_number=89, tag_tid=14, entry_tid=13005' 2024-04-06T09:40:45.224 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=88, tag_tid=14, entry_tid=13104' 2024-04-06T09:40:45.224 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=93, tag_tid=15, entry_tid=1' 2024-04-06T09:40:45.224 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:40:45.225 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:40:45.225 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=15 2024-04-06T09:40:45.226 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=88, tag_tid=14, entry_tid=13104' 2024-04-06T09:40:45.226 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:40:45.226 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:40:45.226 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=14 2024-04-06T09:40:45.227 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=93, tag_tid=15, entry_tid=1' 2024-04-06T09:40:45.227 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:40:45.227 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:40:45.228 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:40:45.229 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=88, tag_tid=14, entry_tid=13104' 2024-04-06T09:40:45.229 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:40:45.229 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:40:45.231 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=13104 2024-04-06T09:40:45.231 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 15 = 14 -a 1 -le 13104 2024-04-06T09:40:45.231 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:40:45.231 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:40:45.231 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:40:45.432 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:40:45.432 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:40:45.432 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:40:45.432 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:40:45.433 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:40:45.433 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:40:45.433 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:40:45.433 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:40:45.433 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:40:45.433 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:40:45.433 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:40:45.433 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:40:45.433 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:40:45.433 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:40:45.433 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:40:45.434 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:40:45.434 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:40:45.436 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:40:45.436 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:40:45.437 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:40:45.437 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:40:45.437 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:40:45.437 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:40:45.437 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:40:45.437 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:40:45.437 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:40:45.437 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:40:45.437 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:40:45.437 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:40:45.437 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:40:45.437 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:40:45.437 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:40:45.437 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:40:45.438 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:40:45.438 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:40:45.438 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:40:45.438 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:40:45.438 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:40:45.439 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:40:45.439 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:40:45.439 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:40:45.439 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:40:45.439 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:40:45.439 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:40:45.439 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:40:45.439 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:40:45.439 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:40:45.439 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:40:45.439 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:40:45.467 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:40:45.467 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:40:45.467 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:40:45.467 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:40:45.596 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:40:45.597 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:40:45.597 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:40:45.598 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:40:45.598 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:40:45.598 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:40:45.598 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:40:45.598 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:40:45.598 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:40:45.598 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:40:45.599 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:40:45.599 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:40:45.599 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:40:45.600 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:40:45.600 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:40:45.600 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:40:45.646 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 22 2024-04-06T09:40:45.647 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 23 2024-04-06T09:40:45.647 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:40:45.647 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=93, tag_tid=15, entry_tid=1], [object_number=92, tag_tid=15, entry_tid=0], [object_number=95, tag_tid=14, entry_tid=19039], [object_number=94, tag_tid=14, entry_tid=19038]]], state=connected] 2024-04-06T09:40:45.647 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=89, tag_tid=14, entry_tid=13205], [object_number=88, tag_tid=14, entry_tid=13204], [object_number=91, tag_tid=14, entry_tid=13203], [object_number=90, tag_tid=14, entry_tid=13202]]], state=connected] 2024-04-06T09:40:45.650 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:40:45.651 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=93, tag_tid=15, entry_tid=1' 2024-04-06T09:40:45.653 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:40:45.653 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:40:45.653 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:40:45.653 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:40:45.653 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:40:45.653 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:40:45.653 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:40:45.653 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:40:45.653 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:40:45.654 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:40:45.654 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:40:45.654 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:40:45.654 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:40:45.654 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:40:45.655 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:40:45.699 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 22 2024-04-06T09:40:45.700 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 23 2024-04-06T09:40:45.700 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:40:45.700 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=93, tag_tid=15, entry_tid=1], [object_number=92, tag_tid=15, entry_tid=0], [object_number=95, tag_tid=14, entry_tid=19039], [object_number=94, tag_tid=14, entry_tid=19038]]], state=connected] 2024-04-06T09:40:45.700 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=89, tag_tid=14, entry_tid=13205], [object_number=88, tag_tid=14, entry_tid=13204], [object_number=91, tag_tid=14, entry_tid=13203], [object_number=90, tag_tid=14, entry_tid=13202]]], state=connected] 2024-04-06T09:40:45.703 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:40:45.705 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=89, tag_tid=14, entry_tid=13205' 2024-04-06T09:40:45.705 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=93, tag_tid=15, entry_tid=1' -a 'object_number=93, tag_tid=15, entry_tid=1' = 'object_number=89, tag_tid=14, entry_tid=13205' 2024-04-06T09:40:45.705 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=89, tag_tid=14, entry_tid=13205' '!=' 'object_number=88, tag_tid=14, entry_tid=13104' 2024-04-06T09:40:45.705 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:40:45.705 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=89, tag_tid=14, entry_tid=13205' = 'object_number=88, tag_tid=14, entry_tid=13104' 2024-04-06T09:40:45.706 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=89, tag_tid=14, entry_tid=13205' 2024-04-06T09:40:45.706 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=93, tag_tid=15, entry_tid=1' 2024-04-06T09:40:45.706 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:40:45.706 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:40:45.706 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=15 2024-04-06T09:40:45.707 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=89, tag_tid=14, entry_tid=13205' 2024-04-06T09:40:45.707 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:40:45.707 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:40:45.708 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=14 2024-04-06T09:40:45.708 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=93, tag_tid=15, entry_tid=1' 2024-04-06T09:40:45.709 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:40:45.709 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:40:45.709 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:40:45.710 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=89, tag_tid=14, entry_tid=13205' 2024-04-06T09:40:45.710 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:40:45.710 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:40:45.711 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=13205 2024-04-06T09:40:45.712 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 15 = 14 -a 1 -le 13205 2024-04-06T09:40:45.712 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:40:45.712 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:40:45.712 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:40:45.913 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:40:45.914 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:40:45.914 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:40:45.914 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:40:45.914 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:40:45.914 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:40:45.914 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:40:45.914 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:40:45.914 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:40:45.914 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:40:45.914 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:40:45.914 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:40:45.915 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:40:45.915 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:40:45.915 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:40:45.915 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:40:45.915 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:40:45.918 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:40:45.918 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:40:45.918 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:40:45.918 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:40:45.918 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:40:45.918 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:40:45.918 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:40:45.918 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:40:45.918 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:40:45.919 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:40:45.919 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:40:45.919 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:40:45.919 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:40:45.919 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:40:45.919 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:40:45.919 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:40:45.920 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:40:45.921 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:40:45.921 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:40:45.921 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:40:45.921 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:40:45.921 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:40:45.921 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:40:45.921 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:40:45.921 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:40:45.921 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:40:45.922 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:40:45.922 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:40:45.922 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:40:45.922 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:40:45.922 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:40:45.922 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:40:45.946 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:40:45.947 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:40:45.947 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:40:45.947 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:40:46.079 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:40:46.081 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:40:46.081 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:40:46.081 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:40:46.081 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:40:46.081 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:40:46.081 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:40:46.081 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:40:46.081 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:40:46.081 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:40:46.082 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:40:46.082 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:40:46.082 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:40:46.083 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:40:46.084 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:40:46.084 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:40:46.129 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 22 2024-04-06T09:40:46.130 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 23 2024-04-06T09:40:46.130 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:40:46.130 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=93, tag_tid=15, entry_tid=1], [object_number=92, tag_tid=15, entry_tid=0], [object_number=95, tag_tid=14, entry_tid=19039], [object_number=94, tag_tid=14, entry_tid=19038]]], state=connected] 2024-04-06T09:40:46.130 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=88, tag_tid=14, entry_tid=13304], [object_number=91, tag_tid=14, entry_tid=13303], [object_number=90, tag_tid=14, entry_tid=13302], [object_number=89, tag_tid=14, entry_tid=13301]]], state=connected] 2024-04-06T09:40:46.133 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:40:46.134 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=93, tag_tid=15, entry_tid=1' 2024-04-06T09:40:46.135 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:40:46.135 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:40:46.135 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:40:46.135 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:40:46.135 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:40:46.136 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:40:46.136 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:40:46.136 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:40:46.136 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:40:46.136 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:40:46.136 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:40:46.136 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:40:46.137 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:40:46.137 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:40:46.137 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:40:46.182 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 22 2024-04-06T09:40:46.182 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 23 2024-04-06T09:40:46.182 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:40:46.182 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=93, tag_tid=15, entry_tid=1], [object_number=92, tag_tid=15, entry_tid=0], [object_number=95, tag_tid=14, entry_tid=19039], [object_number=94, tag_tid=14, entry_tid=19038]]], state=connected] 2024-04-06T09:40:46.182 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=88, tag_tid=14, entry_tid=13304], [object_number=91, tag_tid=14, entry_tid=13303], [object_number=90, tag_tid=14, entry_tid=13302], [object_number=89, tag_tid=14, entry_tid=13301]]], state=connected] 2024-04-06T09:40:46.185 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:40:46.187 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=88, tag_tid=14, entry_tid=13304' 2024-04-06T09:40:46.187 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=93, tag_tid=15, entry_tid=1' -a 'object_number=93, tag_tid=15, entry_tid=1' = 'object_number=88, tag_tid=14, entry_tid=13304' 2024-04-06T09:40:46.187 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=88, tag_tid=14, entry_tid=13304' '!=' 'object_number=89, tag_tid=14, entry_tid=13205' 2024-04-06T09:40:46.187 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:40:46.188 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=88, tag_tid=14, entry_tid=13304' = 'object_number=89, tag_tid=14, entry_tid=13205' 2024-04-06T09:40:46.188 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=88, tag_tid=14, entry_tid=13304' 2024-04-06T09:40:46.188 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=93, tag_tid=15, entry_tid=1' 2024-04-06T09:40:46.188 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:40:46.188 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:40:46.189 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=15 2024-04-06T09:40:46.190 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=88, tag_tid=14, entry_tid=13304' 2024-04-06T09:40:46.190 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:40:46.190 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:40:46.191 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=14 2024-04-06T09:40:46.191 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=93, tag_tid=15, entry_tid=1' 2024-04-06T09:40:46.191 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:40:46.191 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:40:46.192 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:40:46.193 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=88, tag_tid=14, entry_tid=13304' 2024-04-06T09:40:46.193 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:40:46.193 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:40:46.194 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=13304 2024-04-06T09:40:46.194 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 15 = 14 -a 1 -le 13304 2024-04-06T09:40:46.194 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:40:46.195 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:40:46.195 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:40:46.396 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:40:46.396 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:40:46.396 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:40:46.397 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:40:46.397 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:40:46.397 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:40:46.397 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:40:46.397 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:40:46.397 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:40:46.397 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:40:46.397 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:40:46.397 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:40:46.397 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:40:46.397 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:40:46.397 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:40:46.399 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:40:46.399 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:40:46.399 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:40:46.399 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:40:46.399 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:40:46.399 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:40:46.399 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:40:46.400 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:40:46.400 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:40:46.400 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:40:46.400 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:40:46.400 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:40:46.400 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:40:46.400 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:40:46.400 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:40:46.400 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:40:46.400 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:40:46.400 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:40:46.401 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:40:46.401 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:40:46.402 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:40:46.402 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:40:46.402 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:40:46.402 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:40:46.402 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:40:46.402 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:40:46.402 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:40:46.402 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:40:46.402 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:40:46.402 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:40:46.402 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:40:46.402 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:40:46.403 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:40:46.403 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:40:46.421 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:40:46.422 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:40:46.422 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:40:46.422 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:40:46.553 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:40:46.554 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:40:46.554 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:40:46.554 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:40:46.554 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:40:46.554 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:40:46.554 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:40:46.554 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:40:46.554 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:40:46.554 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:40:46.555 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:40:46.555 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:40:46.555 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:40:46.556 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:40:46.556 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:40:46.557 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:40:46.604 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 22 2024-04-06T09:40:46.604 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 23 2024-04-06T09:40:46.604 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:40:46.604 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=93, tag_tid=15, entry_tid=1], [object_number=92, tag_tid=15, entry_tid=0], [object_number=95, tag_tid=14, entry_tid=19039], [object_number=94, tag_tid=14, entry_tid=19038]]], state=connected] 2024-04-06T09:40:46.604 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=89, tag_tid=14, entry_tid=13401], [object_number=88, tag_tid=14, entry_tid=13400], [object_number=91, tag_tid=14, entry_tid=13399], [object_number=90, tag_tid=14, entry_tid=13398]]], state=connected] 2024-04-06T09:40:46.607 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:40:46.609 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=93, tag_tid=15, entry_tid=1' 2024-04-06T09:40:46.610 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:40:46.610 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:40:46.610 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:40:46.610 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:40:46.610 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:40:46.610 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:40:46.610 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:40:46.610 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:40:46.610 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:40:46.610 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:40:46.611 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:40:46.611 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:40:46.611 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:40:46.611 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:40:46.611 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:40:46.657 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 22 2024-04-06T09:40:46.657 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 23 2024-04-06T09:40:46.657 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:40:46.657 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=93, tag_tid=15, entry_tid=1], [object_number=92, tag_tid=15, entry_tid=0], [object_number=95, tag_tid=14, entry_tid=19039], [object_number=94, tag_tid=14, entry_tid=19038]]], state=connected] 2024-04-06T09:40:46.658 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=89, tag_tid=14, entry_tid=13401], [object_number=88, tag_tid=14, entry_tid=13400], [object_number=91, tag_tid=14, entry_tid=13399], [object_number=90, tag_tid=14, entry_tid=13398]]], state=connected] 2024-04-06T09:40:46.661 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:40:46.663 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=89, tag_tid=14, entry_tid=13401' 2024-04-06T09:40:46.663 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=93, tag_tid=15, entry_tid=1' -a 'object_number=93, tag_tid=15, entry_tid=1' = 'object_number=89, tag_tid=14, entry_tid=13401' 2024-04-06T09:40:46.663 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=89, tag_tid=14, entry_tid=13401' '!=' 'object_number=88, tag_tid=14, entry_tid=13304' 2024-04-06T09:40:46.663 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:40:46.664 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=89, tag_tid=14, entry_tid=13401' = 'object_number=88, tag_tid=14, entry_tid=13304' 2024-04-06T09:40:46.664 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=89, tag_tid=14, entry_tid=13401' 2024-04-06T09:40:46.664 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=93, tag_tid=15, entry_tid=1' 2024-04-06T09:40:46.664 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:40:46.664 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:40:46.665 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=15 2024-04-06T09:40:46.665 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=89, tag_tid=14, entry_tid=13401' 2024-04-06T09:40:46.665 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:40:46.665 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:40:46.666 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=14 2024-04-06T09:40:46.666 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=93, tag_tid=15, entry_tid=1' 2024-04-06T09:40:46.666 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:40:46.666 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:40:46.667 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:40:46.668 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=89, tag_tid=14, entry_tid=13401' 2024-04-06T09:40:46.668 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:40:46.668 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:40:46.670 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=13401 2024-04-06T09:40:46.670 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 15 = 14 -a 1 -le 13401 2024-04-06T09:40:46.670 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:40:46.670 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:40:46.670 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:40:46.871 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:40:46.872 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:40:46.872 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:40:46.872 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:40:46.872 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:40:46.872 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:40:46.872 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:40:46.872 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:40:46.872 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:40:46.872 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:40:46.872 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:40:46.873 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:40:46.873 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:40:46.873 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:40:46.873 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:40:46.873 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:40:46.874 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:40:46.876 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:40:46.876 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:40:46.876 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:40:46.876 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:40:46.877 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:40:46.877 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:40:46.877 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:40:46.877 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:40:46.877 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:40:46.877 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:40:46.877 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:40:46.877 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:40:46.877 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:40:46.877 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:40:46.877 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:40:46.877 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:40:46.877 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:40:46.877 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:40:46.878 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:40:46.878 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:40:46.878 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:40:46.878 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:40:46.878 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:40:46.878 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:40:46.878 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:40:46.878 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:40:46.878 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:40:46.878 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:40:46.878 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:40:46.878 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:40:46.878 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:40:46.879 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:40:46.902 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:40:46.902 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:40:46.902 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:40:46.902 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:40:47.035 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:40:47.036 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:40:47.037 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:40:47.037 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:40:47.037 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:40:47.037 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:40:47.037 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:40:47.037 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:40:47.037 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:40:47.037 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:40:47.037 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:40:47.038 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:40:47.038 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:40:47.040 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:40:47.040 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:40:47.040 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:40:47.092 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 22 2024-04-06T09:40:47.092 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 23 2024-04-06T09:40:47.092 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:40:47.092 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=93, tag_tid=15, entry_tid=1], [object_number=92, tag_tid=15, entry_tid=0], [object_number=95, tag_tid=14, entry_tid=19039], [object_number=94, tag_tid=14, entry_tid=19038]]], state=connected] 2024-04-06T09:40:47.092 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=91, tag_tid=14, entry_tid=13499], [object_number=90, tag_tid=14, entry_tid=13498], [object_number=89, tag_tid=14, entry_tid=13497], [object_number=88, tag_tid=14, entry_tid=13496]]], state=connected] 2024-04-06T09:40:47.096 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:40:47.098 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=93, tag_tid=15, entry_tid=1' 2024-04-06T09:40:47.098 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:40:47.099 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:40:47.099 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:40:47.099 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:40:47.099 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:40:47.099 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:40:47.099 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:40:47.099 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:40:47.099 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:40:47.099 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:40:47.099 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:40:47.099 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:40:47.100 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:40:47.100 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:40:47.100 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:40:47.147 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 22 2024-04-06T09:40:47.147 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 23 2024-04-06T09:40:47.147 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:40:47.147 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=93, tag_tid=15, entry_tid=1], [object_number=92, tag_tid=15, entry_tid=0], [object_number=95, tag_tid=14, entry_tid=19039], [object_number=94, tag_tid=14, entry_tid=19038]]], state=connected] 2024-04-06T09:40:47.147 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=91, tag_tid=14, entry_tid=13499], [object_number=90, tag_tid=14, entry_tid=13498], [object_number=89, tag_tid=14, entry_tid=13497], [object_number=88, tag_tid=14, entry_tid=13496]]], state=connected] 2024-04-06T09:40:47.150 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:40:47.152 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=91, tag_tid=14, entry_tid=13499' 2024-04-06T09:40:47.152 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=93, tag_tid=15, entry_tid=1' -a 'object_number=93, tag_tid=15, entry_tid=1' = 'object_number=91, tag_tid=14, entry_tid=13499' 2024-04-06T09:40:47.152 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=91, tag_tid=14, entry_tid=13499' '!=' 'object_number=89, tag_tid=14, entry_tid=13401' 2024-04-06T09:40:47.152 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:40:47.152 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=91, tag_tid=14, entry_tid=13499' = 'object_number=89, tag_tid=14, entry_tid=13401' 2024-04-06T09:40:47.152 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=91, tag_tid=14, entry_tid=13499' 2024-04-06T09:40:47.153 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=93, tag_tid=15, entry_tid=1' 2024-04-06T09:40:47.153 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:40:47.153 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:40:47.153 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=15 2024-04-06T09:40:47.154 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=91, tag_tid=14, entry_tid=13499' 2024-04-06T09:40:47.154 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:40:47.154 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:40:47.156 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=14 2024-04-06T09:40:47.156 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=93, tag_tid=15, entry_tid=1' 2024-04-06T09:40:47.156 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:40:47.156 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:40:47.157 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:40:47.157 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=91, tag_tid=14, entry_tid=13499' 2024-04-06T09:40:47.157 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:40:47.157 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:40:47.159 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=13499 2024-04-06T09:40:47.159 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 15 = 14 -a 1 -le 13499 2024-04-06T09:40:47.159 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:40:47.159 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:40:47.159 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:40:47.361 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:40:47.361 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:40:47.361 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:40:47.361 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:40:47.361 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:40:47.361 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:40:47.361 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:40:47.361 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:40:47.361 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:40:47.362 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:40:47.362 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:40:47.362 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:40:47.362 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:40:47.362 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:40:47.362 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:40:47.365 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:40:47.365 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:40:47.365 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:40:47.365 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:40:47.365 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:40:47.365 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:40:47.365 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:40:47.365 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:40:47.365 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:40:47.365 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:40:47.365 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:40:47.365 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:40:47.365 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:40:47.365 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:40:47.366 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:40:47.366 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:40:47.366 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:40:47.366 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:40:47.366 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:40:47.366 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:40:47.366 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:40:47.366 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:40:47.366 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:40:47.366 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:40:47.367 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:40:47.367 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:40:47.367 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:40:47.367 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:40:47.367 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:40:47.367 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:40:47.367 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:40:47.367 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:40:47.367 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:40:47.367 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:40:47.390 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:40:47.391 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:40:47.391 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:40:47.391 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:40:47.521 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:40:47.522 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:40:47.522 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:40:47.522 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:40:47.522 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:40:47.522 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:40:47.522 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:40:47.522 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:40:47.522 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:40:47.523 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:40:47.523 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:40:47.523 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:40:47.523 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:40:47.524 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:40:47.524 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:40:47.524 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:40:47.570 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 22 2024-04-06T09:40:47.570 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 23 2024-04-06T09:40:47.570 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:40:47.570 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=93, tag_tid=15, entry_tid=1], [object_number=92, tag_tid=15, entry_tid=0], [object_number=95, tag_tid=14, entry_tid=19039], [object_number=94, tag_tid=14, entry_tid=19038]]], state=connected] 2024-04-06T09:40:47.571 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=90, tag_tid=14, entry_tid=13586], [object_number=89, tag_tid=14, entry_tid=13585], [object_number=88, tag_tid=14, entry_tid=13584], [object_number=91, tag_tid=14, entry_tid=13583]]], state=connected] 2024-04-06T09:40:47.574 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:40:47.575 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=93, tag_tid=15, entry_tid=1' 2024-04-06T09:40:47.576 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:40:47.576 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:40:47.576 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:40:47.576 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:40:47.576 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:40:47.576 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:40:47.576 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:40:47.576 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:40:47.577 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:40:47.577 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:40:47.577 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:40:47.577 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:40:47.577 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:40:47.577 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:40:47.577 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:40:47.623 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 22 2024-04-06T09:40:47.623 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 23 2024-04-06T09:40:47.623 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:40:47.624 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=93, tag_tid=15, entry_tid=1], [object_number=92, tag_tid=15, entry_tid=0], [object_number=95, tag_tid=14, entry_tid=19039], [object_number=94, tag_tid=14, entry_tid=19038]]], state=connected] 2024-04-06T09:40:47.624 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=90, tag_tid=14, entry_tid=13586], [object_number=89, tag_tid=14, entry_tid=13585], [object_number=88, tag_tid=14, entry_tid=13584], [object_number=91, tag_tid=14, entry_tid=13583]]], state=connected] 2024-04-06T09:40:47.627 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:40:47.629 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=90, tag_tid=14, entry_tid=13586' 2024-04-06T09:40:47.629 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=93, tag_tid=15, entry_tid=1' -a 'object_number=93, tag_tid=15, entry_tid=1' = 'object_number=90, tag_tid=14, entry_tid=13586' 2024-04-06T09:40:47.629 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=90, tag_tid=14, entry_tid=13586' '!=' 'object_number=91, tag_tid=14, entry_tid=13499' 2024-04-06T09:40:47.629 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:40:47.629 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=90, tag_tid=14, entry_tid=13586' = 'object_number=91, tag_tid=14, entry_tid=13499' 2024-04-06T09:40:47.629 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=90, tag_tid=14, entry_tid=13586' 2024-04-06T09:40:47.629 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=93, tag_tid=15, entry_tid=1' 2024-04-06T09:40:47.630 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:40:47.630 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:40:47.630 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=15 2024-04-06T09:40:47.631 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=90, tag_tid=14, entry_tid=13586' 2024-04-06T09:40:47.631 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:40:47.631 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:40:47.632 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=14 2024-04-06T09:40:47.633 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=93, tag_tid=15, entry_tid=1' 2024-04-06T09:40:47.633 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:40:47.633 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:40:47.634 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:40:47.634 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=90, tag_tid=14, entry_tid=13586' 2024-04-06T09:40:47.634 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:40:47.635 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:40:47.636 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=13586 2024-04-06T09:40:47.636 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 15 = 14 -a 1 -le 13586 2024-04-06T09:40:47.636 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:40:47.636 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:40:47.636 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:40:47.838 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:40:47.838 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:40:47.838 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:40:47.838 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:40:47.838 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:40:47.838 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:40:47.838 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:40:47.838 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:40:47.838 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:40:47.838 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:40:47.838 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:40:47.839 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:40:47.839 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:40:47.839 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:40:47.839 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:40:47.839 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:40:47.840 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:40:47.842 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:40:47.842 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:40:47.842 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:40:47.843 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:40:47.843 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:40:47.843 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:40:47.843 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:40:47.843 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:40:47.843 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:40:47.843 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:40:47.843 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:40:47.843 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:40:47.843 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:40:47.843 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:40:47.843 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:40:47.843 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:40:47.844 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:40:47.844 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:40:47.844 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:40:47.845 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:40:47.845 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:40:47.845 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:40:47.845 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:40:47.845 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:40:47.845 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:40:47.845 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:40:47.845 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:40:47.845 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:40:47.845 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:40:47.845 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:40:47.845 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:40:47.845 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:40:47.869 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:40:47.869 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:40:47.869 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:40:47.869 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:40:48.004 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:40:48.005 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:40:48.006 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:40:48.006 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:40:48.006 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:40:48.006 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:40:48.006 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:40:48.006 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:40:48.006 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:40:48.006 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:40:48.007 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:40:48.007 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:40:48.007 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:40:48.009 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:40:48.009 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:40:48.009 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:40:48.055 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 22 2024-04-06T09:40:48.055 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 23 2024-04-06T09:40:48.055 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:40:48.055 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=93, tag_tid=15, entry_tid=1], [object_number=92, tag_tid=15, entry_tid=0], [object_number=95, tag_tid=14, entry_tid=19039], [object_number=94, tag_tid=14, entry_tid=19038]]], state=connected] 2024-04-06T09:40:48.055 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=90, tag_tid=14, entry_tid=13666], [object_number=89, tag_tid=14, entry_tid=13665], [object_number=88, tag_tid=14, entry_tid=13664], [object_number=91, tag_tid=14, entry_tid=13663]]], state=connected] 2024-04-06T09:40:48.059 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:40:48.060 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=93, tag_tid=15, entry_tid=1' 2024-04-06T09:40:48.062 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:40:48.062 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:40:48.062 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:40:48.062 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:40:48.062 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:40:48.062 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:40:48.062 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:40:48.062 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:40:48.062 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:40:48.062 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:40:48.062 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:40:48.063 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:40:48.063 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:40:48.063 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:40:48.063 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:40:48.109 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 22 2024-04-06T09:40:48.109 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 23 2024-04-06T09:40:48.109 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:40:48.109 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=93, tag_tid=15, entry_tid=1], [object_number=92, tag_tid=15, entry_tid=0], [object_number=95, tag_tid=14, entry_tid=19039], [object_number=94, tag_tid=14, entry_tid=19038]]], state=connected] 2024-04-06T09:40:48.110 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=90, tag_tid=14, entry_tid=13666], [object_number=89, tag_tid=14, entry_tid=13665], [object_number=88, tag_tid=14, entry_tid=13664], [object_number=91, tag_tid=14, entry_tid=13663]]], state=connected] 2024-04-06T09:40:48.113 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:40:48.115 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=90, tag_tid=14, entry_tid=13666' 2024-04-06T09:40:48.115 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=93, tag_tid=15, entry_tid=1' -a 'object_number=93, tag_tid=15, entry_tid=1' = 'object_number=90, tag_tid=14, entry_tid=13666' 2024-04-06T09:40:48.115 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=90, tag_tid=14, entry_tid=13666' '!=' 'object_number=90, tag_tid=14, entry_tid=13586' 2024-04-06T09:40:48.115 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:40:48.115 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=90, tag_tid=14, entry_tid=13666' = 'object_number=90, tag_tid=14, entry_tid=13586' 2024-04-06T09:40:48.116 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=90, tag_tid=14, entry_tid=13666' 2024-04-06T09:40:48.116 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=93, tag_tid=15, entry_tid=1' 2024-04-06T09:40:48.116 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:40:48.116 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:40:48.119 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=15 2024-04-06T09:40:48.119 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=90, tag_tid=14, entry_tid=13666' 2024-04-06T09:40:48.119 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:40:48.119 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:40:48.120 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=14 2024-04-06T09:40:48.120 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=93, tag_tid=15, entry_tid=1' 2024-04-06T09:40:48.120 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:40:48.120 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:40:48.120 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:40:48.120 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=90, tag_tid=14, entry_tid=13666' 2024-04-06T09:40:48.121 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:40:48.121 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:40:48.122 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=13666 2024-04-06T09:40:48.122 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 15 = 14 -a 1 -le 13666 2024-04-06T09:40:48.122 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:40:48.122 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:40:48.122 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:40:48.324 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:40:48.324 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:40:48.324 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:40:48.324 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:40:48.324 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:40:48.324 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:40:48.324 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:40:48.324 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:40:48.324 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:40:48.324 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:40:48.324 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:40:48.324 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:40:48.325 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:40:48.325 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:40:48.325 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:40:48.325 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:40:48.325 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:40:48.327 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:40:48.327 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:40:48.327 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:40:48.327 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:40:48.327 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:40:48.327 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:40:48.327 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:40:48.328 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:40:48.328 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:40:48.328 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:40:48.328 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:40:48.328 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:40:48.328 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:40:48.328 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:40:48.328 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:40:48.328 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:40:48.329 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:40:48.329 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:40:48.330 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:40:48.330 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:40:48.330 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:40:48.330 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:40:48.330 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:40:48.330 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:40:48.330 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:40:48.330 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:40:48.330 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:40:48.330 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:40:48.330 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:40:48.330 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:40:48.330 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:40:48.330 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:40:48.336 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:40:48.331+0000 7f2e84d66640 -1 reset not still connected to 0x560d2655fee0 2024-04-06T09:40:48.336 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:40:48.331+0000 7f2e84d66640 -1 reset not still connected to 0x560d27073930 2024-04-06T09:40:48.337 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:40:48.335+0000 7f2e84d66640 -1 reset not still connected to 0x560d27073ba0 2024-04-06T09:40:48.337 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:40:48.335+0000 7f2e84d66640 -1 reset not still connected to 0x560d27073d40 2024-04-06T09:40:48.355 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:40:48.355 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:40:48.355 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:40:48.355 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:40:48.478 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:40:48.480 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:40:48.480 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:40:48.480 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:40:48.480 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:40:48.480 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:40:48.480 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:40:48.480 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:40:48.480 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:40:48.480 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:40:48.480 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:40:48.480 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:40:48.480 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:40:48.482 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:40:48.482 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:40:48.482 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:40:48.531 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 22 2024-04-06T09:40:48.531 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 23 2024-04-06T09:40:48.531 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:40:48.531 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=93, tag_tid=15, entry_tid=1], [object_number=92, tag_tid=15, entry_tid=0], [object_number=95, tag_tid=14, entry_tid=19039], [object_number=94, tag_tid=14, entry_tid=19038]]], state=connected] 2024-04-06T09:40:48.531 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=90, tag_tid=14, entry_tid=13746], [object_number=89, tag_tid=14, entry_tid=13745], [object_number=88, tag_tid=14, entry_tid=13744], [object_number=91, tag_tid=14, entry_tid=13743]]], state=connected] 2024-04-06T09:40:48.534 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:40:48.535 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=93, tag_tid=15, entry_tid=1' 2024-04-06T09:40:48.536 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:40:48.536 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:40:48.537 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:40:48.537 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:40:48.537 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:40:48.537 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:40:48.537 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:40:48.537 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:40:48.537 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:40:48.537 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:40:48.537 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:40:48.537 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:40:48.537 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:40:48.538 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:40:48.538 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:40:48.583 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 22 2024-04-06T09:40:48.583 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 23 2024-04-06T09:40:48.583 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:40:48.583 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=93, tag_tid=15, entry_tid=1], [object_number=92, tag_tid=15, entry_tid=0], [object_number=95, tag_tid=14, entry_tid=19039], [object_number=94, tag_tid=14, entry_tid=19038]]], state=connected] 2024-04-06T09:40:48.583 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=90, tag_tid=14, entry_tid=13746], [object_number=89, tag_tid=14, entry_tid=13745], [object_number=88, tag_tid=14, entry_tid=13744], [object_number=91, tag_tid=14, entry_tid=13743]]], state=connected] 2024-04-06T09:40:48.587 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:40:48.589 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=90, tag_tid=14, entry_tid=13746' 2024-04-06T09:40:48.589 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=93, tag_tid=15, entry_tid=1' -a 'object_number=93, tag_tid=15, entry_tid=1' = 'object_number=90, tag_tid=14, entry_tid=13746' 2024-04-06T09:40:48.589 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=90, tag_tid=14, entry_tid=13746' '!=' 'object_number=90, tag_tid=14, entry_tid=13666' 2024-04-06T09:40:48.589 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:40:48.589 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=90, tag_tid=14, entry_tid=13746' = 'object_number=90, tag_tid=14, entry_tid=13666' 2024-04-06T09:40:48.589 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=90, tag_tid=14, entry_tid=13746' 2024-04-06T09:40:48.589 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=93, tag_tid=15, entry_tid=1' 2024-04-06T09:40:48.590 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:40:48.590 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:40:48.590 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=15 2024-04-06T09:40:48.591 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=90, tag_tid=14, entry_tid=13746' 2024-04-06T09:40:48.591 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:40:48.591 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:40:48.592 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=14 2024-04-06T09:40:48.593 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=93, tag_tid=15, entry_tid=1' 2024-04-06T09:40:48.593 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:40:48.593 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:40:48.594 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:40:48.594 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=90, tag_tid=14, entry_tid=13746' 2024-04-06T09:40:48.594 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:40:48.595 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:40:48.596 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=13746 2024-04-06T09:40:48.596 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 15 = 14 -a 1 -le 13746 2024-04-06T09:40:48.596 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:40:48.596 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:40:48.596 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:40:48.797 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:40:48.798 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:40:48.798 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:40:48.798 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:40:48.798 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:40:48.798 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:40:48.798 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:40:48.798 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:40:48.798 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:40:48.798 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:40:48.798 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:40:48.798 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:40:48.798 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:40:48.798 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:40:48.799 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:40:48.799 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:40:48.800 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:40:48.802 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:40:48.802 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:40:48.802 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:40:48.802 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:40:48.802 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:40:48.802 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:40:48.802 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:40:48.802 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:40:48.802 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:40:48.803 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:40:48.803 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:40:48.803 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:40:48.803 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:40:48.803 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:40:48.803 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:40:48.803 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:40:48.804 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:40:48.804 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:40:48.804 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:40:48.804 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:40:48.804 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:40:48.805 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:40:48.805 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:40:48.805 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:40:48.805 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:40:48.805 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:40:48.805 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:40:48.805 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:40:48.805 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:40:48.805 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:40:48.805 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:40:48.805 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:40:48.830 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:40:48.830 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:40:48.830 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:40:48.830 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:40:48.960 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:40:48.961 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:40:48.962 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:40:48.962 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:40:48.962 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:40:48.962 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:40:48.962 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:40:48.962 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:40:48.962 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:40:48.962 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:40:48.963 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:40:48.963 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:40:48.963 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:40:48.964 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:40:48.965 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:40:48.965 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:40:49.016 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 22 2024-04-06T09:40:49.016 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 23 2024-04-06T09:40:49.016 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:40:49.016 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=93, tag_tid=15, entry_tid=1], [object_number=92, tag_tid=15, entry_tid=0], [object_number=95, tag_tid=14, entry_tid=19039], [object_number=94, tag_tid=14, entry_tid=19038]]], state=connected] 2024-04-06T09:40:49.016 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=90, tag_tid=14, entry_tid=13822], [object_number=89, tag_tid=14, entry_tid=13821], [object_number=88, tag_tid=14, entry_tid=13820], [object_number=91, tag_tid=14, entry_tid=13819]]], state=connected] 2024-04-06T09:40:49.020 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:40:49.021 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=93, tag_tid=15, entry_tid=1' 2024-04-06T09:40:49.022 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:40:49.022 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:40:49.022 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:40:49.022 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:40:49.022 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:40:49.022 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:40:49.022 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:40:49.023 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:40:49.023 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:40:49.023 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:40:49.023 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:40:49.023 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:40:49.024 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:40:49.024 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:40:49.024 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:40:49.071 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 22 2024-04-06T09:40:49.071 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 23 2024-04-06T09:40:49.071 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:40:49.071 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=93, tag_tid=15, entry_tid=1], [object_number=92, tag_tid=15, entry_tid=0], [object_number=95, tag_tid=14, entry_tid=19039], [object_number=94, tag_tid=14, entry_tid=19038]]], state=connected] 2024-04-06T09:40:49.071 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=90, tag_tid=14, entry_tid=13822], [object_number=89, tag_tid=14, entry_tid=13821], [object_number=88, tag_tid=14, entry_tid=13820], [object_number=91, tag_tid=14, entry_tid=13819]]], state=connected] 2024-04-06T09:40:49.074 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:40:49.076 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=90, tag_tid=14, entry_tid=13822' 2024-04-06T09:40:49.077 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=93, tag_tid=15, entry_tid=1' -a 'object_number=93, tag_tid=15, entry_tid=1' = 'object_number=90, tag_tid=14, entry_tid=13822' 2024-04-06T09:40:49.077 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=90, tag_tid=14, entry_tid=13822' '!=' 'object_number=90, tag_tid=14, entry_tid=13746' 2024-04-06T09:40:49.077 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:40:49.077 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=90, tag_tid=14, entry_tid=13822' = 'object_number=90, tag_tid=14, entry_tid=13746' 2024-04-06T09:40:49.077 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=90, tag_tid=14, entry_tid=13822' 2024-04-06T09:40:49.077 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=93, tag_tid=15, entry_tid=1' 2024-04-06T09:40:49.077 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:40:49.077 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:40:49.077 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=15 2024-04-06T09:40:49.078 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=90, tag_tid=14, entry_tid=13822' 2024-04-06T09:40:49.078 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:40:49.078 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:40:49.079 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=14 2024-04-06T09:40:49.080 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=93, tag_tid=15, entry_tid=1' 2024-04-06T09:40:49.080 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:40:49.080 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:40:49.081 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:40:49.082 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=90, tag_tid=14, entry_tid=13822' 2024-04-06T09:40:49.082 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:40:49.082 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:40:49.083 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=13822 2024-04-06T09:40:49.083 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 15 = 14 -a 1 -le 13822 2024-04-06T09:40:49.083 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:40:49.083 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:40:49.083 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:40:49.286 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:40:49.286 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:40:49.286 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:40:49.286 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:40:49.286 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:40:49.286 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:40:49.287 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:40:49.287 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:40:49.287 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:40:49.287 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:40:49.287 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:40:49.287 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:40:49.287 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:40:49.287 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:40:49.287 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:40:49.291 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:40:49.291 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:40:49.291 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:40:49.292 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:40:49.292 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:40:49.292 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:40:49.292 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:40:49.292 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:40:49.292 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:40:49.292 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:40:49.292 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:40:49.292 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:40:49.292 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:40:49.292 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:40:49.292 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:40:49.292 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:40:49.293 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:40:49.293 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:40:49.293 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:40:49.293 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:40:49.293 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:40:49.293 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:40:49.293 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:40:49.293 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:40:49.293 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:40:49.293 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:40:49.294 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:40:49.294 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:40:49.294 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:40:49.294 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:40:49.294 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:40:49.294 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:40:49.294 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:40:49.294 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:40:49.316 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:40:49.316 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:40:49.316 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:40:49.316 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:40:49.446 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:40:49.447 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:40:49.447 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:40:49.447 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:40:49.447 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:40:49.447 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:40:49.447 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:40:49.447 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:40:49.447 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:40:49.447 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:40:49.448 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:40:49.448 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:40:49.448 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:40:49.450 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:40:49.451 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:40:49.451 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:40:49.502 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 22 2024-04-06T09:40:49.502 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 23 2024-04-06T09:40:49.502 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:40:49.503 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=93, tag_tid=15, entry_tid=1], [object_number=92, tag_tid=15, entry_tid=0], [object_number=95, tag_tid=14, entry_tid=19039], [object_number=94, tag_tid=14, entry_tid=19038]]], state=connected] 2024-04-06T09:40:49.503 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=91, tag_tid=14, entry_tid=13907], [object_number=90, tag_tid=14, entry_tid=13906], [object_number=89, tag_tid=14, entry_tid=13905], [object_number=88, tag_tid=14, entry_tid=13904]]], state=connected] 2024-04-06T09:40:49.506 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:40:49.507 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=93, tag_tid=15, entry_tid=1' 2024-04-06T09:40:49.509 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:40:49.509 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:40:49.509 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:40:49.509 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:40:49.509 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:40:49.509 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:40:49.509 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:40:49.509 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:40:49.509 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:40:49.509 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:40:49.510 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:40:49.510 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:40:49.510 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:40:49.510 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:40:49.510 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:40:49.557 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 22 2024-04-06T09:40:49.557 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 23 2024-04-06T09:40:49.557 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:40:49.558 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=93, tag_tid=15, entry_tid=1], [object_number=92, tag_tid=15, entry_tid=0], [object_number=95, tag_tid=14, entry_tid=19039], [object_number=94, tag_tid=14, entry_tid=19038]]], state=connected] 2024-04-06T09:40:49.558 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=91, tag_tid=14, entry_tid=13907], [object_number=90, tag_tid=14, entry_tid=13906], [object_number=89, tag_tid=14, entry_tid=13905], [object_number=88, tag_tid=14, entry_tid=13904]]], state=connected] 2024-04-06T09:40:49.561 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:40:49.562 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=91, tag_tid=14, entry_tid=13907' 2024-04-06T09:40:49.563 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=93, tag_tid=15, entry_tid=1' -a 'object_number=93, tag_tid=15, entry_tid=1' = 'object_number=91, tag_tid=14, entry_tid=13907' 2024-04-06T09:40:49.563 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=91, tag_tid=14, entry_tid=13907' '!=' 'object_number=90, tag_tid=14, entry_tid=13822' 2024-04-06T09:40:49.563 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:40:49.563 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=91, tag_tid=14, entry_tid=13907' = 'object_number=90, tag_tid=14, entry_tid=13822' 2024-04-06T09:40:49.563 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=91, tag_tid=14, entry_tid=13907' 2024-04-06T09:40:49.564 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=93, tag_tid=15, entry_tid=1' 2024-04-06T09:40:49.564 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:40:49.564 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:40:49.564 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=15 2024-04-06T09:40:49.565 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=91, tag_tid=14, entry_tid=13907' 2024-04-06T09:40:49.565 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:40:49.565 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:40:49.566 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=14 2024-04-06T09:40:49.567 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=93, tag_tid=15, entry_tid=1' 2024-04-06T09:40:49.567 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:40:49.567 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:40:49.568 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:40:49.568 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=91, tag_tid=14, entry_tid=13907' 2024-04-06T09:40:49.568 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:40:49.568 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:40:49.570 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=13907 2024-04-06T09:40:49.570 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 15 = 14 -a 1 -le 13907 2024-04-06T09:40:49.570 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:40:49.570 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:40:49.570 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:40:49.772 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:40:49.772 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:40:49.772 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:40:49.772 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:40:49.772 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:40:49.772 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:40:49.772 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:40:49.772 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:40:49.772 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:40:49.772 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:40:49.772 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:40:49.772 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:40:49.773 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:40:49.773 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:40:49.773 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:40:49.773 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:40:49.774 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:40:49.776 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:40:49.776 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:40:49.776 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:40:49.776 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:40:49.776 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:40:49.776 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:40:49.776 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:40:49.776 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:40:49.776 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:40:49.777 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:40:49.777 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:40:49.777 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:40:49.777 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:40:49.777 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:40:49.777 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:40:49.777 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:40:49.778 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:40:49.778 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:40:49.778 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:40:49.778 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:40:49.779 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:40:49.779 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:40:49.779 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:40:49.779 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:40:49.779 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:40:49.779 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:40:49.779 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:40:49.779 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:40:49.779 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:40:49.779 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:40:49.779 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:40:49.779 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:40:49.804 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:40:49.804 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:40:49.804 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:40:49.804 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:40:49.935 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:40:49.936 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:40:49.936 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:40:49.936 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:40:49.936 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:40:49.936 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:40:49.936 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:40:49.936 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:40:49.936 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:40:49.936 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:40:49.937 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:40:49.937 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:40:49.937 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:40:49.939 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:40:49.940 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:40:49.940 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:40:49.991 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 22 2024-04-06T09:40:49.991 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 23 2024-04-06T09:40:49.991 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:40:49.991 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=93, tag_tid=15, entry_tid=1], [object_number=92, tag_tid=15, entry_tid=0], [object_number=95, tag_tid=14, entry_tid=19039], [object_number=94, tag_tid=14, entry_tid=19038]]], state=connected] 2024-04-06T09:40:49.991 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=91, tag_tid=14, entry_tid=13991], [object_number=90, tag_tid=14, entry_tid=13990], [object_number=89, tag_tid=14, entry_tid=13989], [object_number=88, tag_tid=14, entry_tid=13988]]], state=connected] 2024-04-06T09:40:49.995 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:40:49.996 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=93, tag_tid=15, entry_tid=1' 2024-04-06T09:40:49.997 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:40:49.997 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:40:49.997 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:40:49.998 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:40:49.998 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:40:49.998 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:40:49.998 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:40:49.998 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:40:49.998 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:40:49.998 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:40:49.998 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:40:49.998 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:40:49.998 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:40:49.999 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:40:49.999 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:40:50.048 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 22 2024-04-06T09:40:50.048 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 23 2024-04-06T09:40:50.048 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:40:50.049 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=93, tag_tid=15, entry_tid=1], [object_number=92, tag_tid=15, entry_tid=0], [object_number=95, tag_tid=14, entry_tid=19039], [object_number=94, tag_tid=14, entry_tid=19038]]], state=connected] 2024-04-06T09:40:50.049 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=91, tag_tid=14, entry_tid=13991], [object_number=90, tag_tid=14, entry_tid=13990], [object_number=89, tag_tid=14, entry_tid=13989], [object_number=88, tag_tid=14, entry_tid=13988]]], state=connected] 2024-04-06T09:40:50.052 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:40:50.054 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=91, tag_tid=14, entry_tid=13991' 2024-04-06T09:40:50.055 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=93, tag_tid=15, entry_tid=1' -a 'object_number=93, tag_tid=15, entry_tid=1' = 'object_number=91, tag_tid=14, entry_tid=13991' 2024-04-06T09:40:50.055 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=91, tag_tid=14, entry_tid=13991' '!=' 'object_number=91, tag_tid=14, entry_tid=13907' 2024-04-06T09:40:50.055 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:40:50.055 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=91, tag_tid=14, entry_tid=13991' = 'object_number=91, tag_tid=14, entry_tid=13907' 2024-04-06T09:40:50.055 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=91, tag_tid=14, entry_tid=13991' 2024-04-06T09:40:50.055 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=93, tag_tid=15, entry_tid=1' 2024-04-06T09:40:50.055 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:40:50.055 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:40:50.056 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=15 2024-04-06T09:40:50.056 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=91, tag_tid=14, entry_tid=13991' 2024-04-06T09:40:50.056 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:40:50.056 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:40:50.057 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=14 2024-04-06T09:40:50.058 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=93, tag_tid=15, entry_tid=1' 2024-04-06T09:40:50.058 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:40:50.058 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:40:50.059 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:40:50.059 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=91, tag_tid=14, entry_tid=13991' 2024-04-06T09:40:50.059 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:40:50.059 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:40:50.061 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=13991 2024-04-06T09:40:50.061 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 15 = 14 -a 1 -le 13991 2024-04-06T09:40:50.061 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:40:50.061 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:40:50.061 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:40:50.262 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:40:50.262 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:40:50.262 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:40:50.263 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:40:50.263 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:40:50.263 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:40:50.263 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:40:50.263 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:40:50.263 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:40:50.263 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:40:50.263 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:40:50.263 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:40:50.263 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:40:50.263 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:40:50.263 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:40:50.264 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:40:50.264 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:40:50.266 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:40:50.266 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:40:50.267 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:40:50.267 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:40:50.267 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:40:50.267 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:40:50.267 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:40:50.267 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:40:50.267 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:40:50.267 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:40:50.267 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:40:50.267 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:40:50.267 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:40:50.267 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:40:50.267 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:40:50.267 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:40:50.269 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:40:50.269 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:40:50.269 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:40:50.269 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:40:50.269 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:40:50.269 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:40:50.269 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:40:50.269 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:40:50.269 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:40:50.269 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:40:50.270 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:40:50.270 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:40:50.270 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:40:50.270 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:40:50.270 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:40:50.270 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:40:50.293 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:40:50.294 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:40:50.294 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:40:50.294 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:40:50.427 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:40:50.429 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:40:50.429 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:40:50.429 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:40:50.429 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:40:50.429 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:40:50.429 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:40:50.429 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:40:50.429 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:40:50.429 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:40:50.430 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:40:50.430 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:40:50.430 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:40:50.431 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:40:50.432 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:40:50.432 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:40:50.478 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 22 2024-04-06T09:40:50.478 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 23 2024-04-06T09:40:50.478 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:40:50.478 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=93, tag_tid=15, entry_tid=1], [object_number=92, tag_tid=15, entry_tid=0], [object_number=95, tag_tid=14, entry_tid=19039], [object_number=94, tag_tid=14, entry_tid=19038]]], state=connected] 2024-04-06T09:40:50.478 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=89, tag_tid=14, entry_tid=14081], [object_number=88, tag_tid=14, entry_tid=14080], [object_number=91, tag_tid=14, entry_tid=14079], [object_number=90, tag_tid=14, entry_tid=14078]]], state=connected] 2024-04-06T09:40:50.481 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:40:50.483 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=93, tag_tid=15, entry_tid=1' 2024-04-06T09:40:50.484 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:40:50.484 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:40:50.484 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:40:50.484 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:40:50.484 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:40:50.484 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:40:50.484 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:40:50.484 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:40:50.485 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:40:50.485 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:40:50.485 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:40:50.485 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:40:50.485 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:40:50.486 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:40:50.486 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:40:50.532 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 22 2024-04-06T09:40:50.532 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 23 2024-04-06T09:40:50.532 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:40:50.532 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=93, tag_tid=15, entry_tid=1], [object_number=92, tag_tid=15, entry_tid=0], [object_number=95, tag_tid=14, entry_tid=19039], [object_number=94, tag_tid=14, entry_tid=19038]]], state=connected] 2024-04-06T09:40:50.532 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=89, tag_tid=14, entry_tid=14081], [object_number=88, tag_tid=14, entry_tid=14080], [object_number=91, tag_tid=14, entry_tid=14079], [object_number=90, tag_tid=14, entry_tid=14078]]], state=connected] 2024-04-06T09:40:50.535 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:40:50.538 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=89, tag_tid=14, entry_tid=14081' 2024-04-06T09:40:50.538 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=93, tag_tid=15, entry_tid=1' -a 'object_number=93, tag_tid=15, entry_tid=1' = 'object_number=89, tag_tid=14, entry_tid=14081' 2024-04-06T09:40:50.538 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=89, tag_tid=14, entry_tid=14081' '!=' 'object_number=91, tag_tid=14, entry_tid=13991' 2024-04-06T09:40:50.538 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:40:50.538 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=89, tag_tid=14, entry_tid=14081' = 'object_number=91, tag_tid=14, entry_tid=13991' 2024-04-06T09:40:50.538 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=89, tag_tid=14, entry_tid=14081' 2024-04-06T09:40:50.538 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=93, tag_tid=15, entry_tid=1' 2024-04-06T09:40:50.538 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:40:50.538 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:40:50.539 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=15 2024-04-06T09:40:50.540 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=89, tag_tid=14, entry_tid=14081' 2024-04-06T09:40:50.540 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:40:50.540 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:40:50.541 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=14 2024-04-06T09:40:50.541 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=93, tag_tid=15, entry_tid=1' 2024-04-06T09:40:50.541 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:40:50.541 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:40:50.542 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:40:50.543 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=89, tag_tid=14, entry_tid=14081' 2024-04-06T09:40:50.543 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:40:50.543 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:40:50.544 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=14081 2024-04-06T09:40:50.544 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 15 = 14 -a 1 -le 14081 2024-04-06T09:40:50.544 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:40:50.545 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:40:50.545 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:40:50.746 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:40:50.746 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:40:50.746 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:40:50.746 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:40:50.746 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:40:50.746 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:40:50.746 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:40:50.747 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:40:50.747 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:40:50.747 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:40:50.747 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:40:50.747 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:40:50.747 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:40:50.747 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:40:50.747 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:40:50.748 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:40:50.748 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:40:50.751 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:40:50.751 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:40:50.752 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:40:50.752 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:40:50.752 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:40:50.752 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:40:50.752 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:40:50.752 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:40:50.752 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:40:50.752 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:40:50.752 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:40:50.752 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:40:50.752 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:40:50.752 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:40:50.752 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:40:50.752 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:40:50.752 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:40:50.752 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:40:50.753 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:40:50.753 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:40:50.753 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:40:50.753 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:40:50.753 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:40:50.753 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:40:50.753 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:40:50.753 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:40:50.753 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:40:50.753 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:40:50.753 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:40:50.753 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:40:50.753 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:40:50.753 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:40:50.777 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:40:50.777 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:40:50.777 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:40:50.777 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:40:50.916 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:40:50.917 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:40:50.917 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:40:50.917 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:40:50.917 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:40:50.917 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:40:50.917 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:40:50.917 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:40:50.918 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:40:50.918 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:40:50.918 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:40:50.918 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:40:50.918 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:40:50.919 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:40:50.920 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:40:50.920 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:40:50.973 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 22 2024-04-06T09:40:50.974 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 23 2024-04-06T09:40:50.974 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:40:50.974 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=93, tag_tid=15, entry_tid=1], [object_number=92, tag_tid=15, entry_tid=0], [object_number=95, tag_tid=14, entry_tid=19039], [object_number=94, tag_tid=14, entry_tid=19038]]], state=connected] 2024-04-06T09:40:50.974 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=88, tag_tid=14, entry_tid=14164], [object_number=91, tag_tid=14, entry_tid=14163], [object_number=90, tag_tid=14, entry_tid=14162], [object_number=89, tag_tid=14, entry_tid=14161]]], state=connected] 2024-04-06T09:40:50.977 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:40:50.978 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=93, tag_tid=15, entry_tid=1' 2024-04-06T09:40:50.979 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:40:50.980 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:40:50.980 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:40:50.980 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:40:50.980 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:40:50.980 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:40:50.980 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:40:50.980 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:40:50.980 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:40:50.980 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:40:50.980 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:40:50.980 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:40:50.981 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:40:50.981 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:40:50.981 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:40:51.028 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 22 2024-04-06T09:40:51.029 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 23 2024-04-06T09:40:51.029 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:40:51.029 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=93, tag_tid=15, entry_tid=1], [object_number=92, tag_tid=15, entry_tid=0], [object_number=95, tag_tid=14, entry_tid=19039], [object_number=94, tag_tid=14, entry_tid=19038]]], state=connected] 2024-04-06T09:40:51.029 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=88, tag_tid=14, entry_tid=14164], [object_number=91, tag_tid=14, entry_tid=14163], [object_number=90, tag_tid=14, entry_tid=14162], [object_number=89, tag_tid=14, entry_tid=14161]]], state=connected] 2024-04-06T09:40:51.032 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:40:51.034 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=88, tag_tid=14, entry_tid=14164' 2024-04-06T09:40:51.034 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=93, tag_tid=15, entry_tid=1' -a 'object_number=93, tag_tid=15, entry_tid=1' = 'object_number=88, tag_tid=14, entry_tid=14164' 2024-04-06T09:40:51.034 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=88, tag_tid=14, entry_tid=14164' '!=' 'object_number=89, tag_tid=14, entry_tid=14081' 2024-04-06T09:40:51.034 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:40:51.035 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=88, tag_tid=14, entry_tid=14164' = 'object_number=89, tag_tid=14, entry_tid=14081' 2024-04-06T09:40:51.035 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=88, tag_tid=14, entry_tid=14164' 2024-04-06T09:40:51.035 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=93, tag_tid=15, entry_tid=1' 2024-04-06T09:40:51.035 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:40:51.035 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:40:51.035 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=15 2024-04-06T09:40:51.036 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=88, tag_tid=14, entry_tid=14164' 2024-04-06T09:40:51.036 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:40:51.036 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:40:51.037 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=14 2024-04-06T09:40:51.038 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=93, tag_tid=15, entry_tid=1' 2024-04-06T09:40:51.038 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:40:51.038 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:40:51.039 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:40:51.040 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=88, tag_tid=14, entry_tid=14164' 2024-04-06T09:40:51.040 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:40:51.040 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:40:51.041 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=14164 2024-04-06T09:40:51.041 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 15 = 14 -a 1 -le 14164 2024-04-06T09:40:51.041 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:40:51.041 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:40:51.041 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:40:51.243 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:40:51.243 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:40:51.243 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:40:51.243 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:40:51.244 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:40:51.244 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:40:51.244 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:40:51.244 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:40:51.244 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:40:51.244 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:40:51.244 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:40:51.244 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:40:51.244 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:40:51.244 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:40:51.244 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:40:51.245 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:40:51.245 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:40:51.248 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:40:51.248 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:40:51.248 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:40:51.248 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:40:51.249 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:40:51.249 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:40:51.249 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:40:51.249 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:40:51.249 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:40:51.249 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:40:51.249 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:40:51.249 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:40:51.249 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:40:51.249 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:40:51.249 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:40:51.249 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:40:51.249 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:40:51.250 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:40:51.250 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:40:51.250 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:40:51.250 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:40:51.250 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:40:51.250 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:40:51.250 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:40:51.250 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:40:51.250 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:40:51.250 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:40:51.250 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:40:51.251 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:40:51.251 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:40:51.251 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:40:51.251 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:40:51.274 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:40:51.274 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:40:51.274 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:40:51.274 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:40:51.406 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:40:51.407 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:40:51.407 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:40:51.407 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:40:51.407 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:40:51.407 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:40:51.407 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:40:51.407 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:40:51.407 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:40:51.407 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:40:51.408 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:40:51.408 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:40:51.408 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:40:51.410 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:40:51.411 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:40:51.411 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:40:51.468 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 22 2024-04-06T09:40:51.469 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 23 2024-04-06T09:40:51.469 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:40:51.469 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=93, tag_tid=15, entry_tid=1], [object_number=92, tag_tid=15, entry_tid=0], [object_number=95, tag_tid=14, entry_tid=19039], [object_number=94, tag_tid=14, entry_tid=19038]]], state=connected] 2024-04-06T09:40:51.469 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=91, tag_tid=14, entry_tid=14243], [object_number=90, tag_tid=14, entry_tid=14242], [object_number=89, tag_tid=14, entry_tid=14241], [object_number=88, tag_tid=14, entry_tid=14240]]], state=connected] 2024-04-06T09:40:51.472 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:40:51.475 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=93, tag_tid=15, entry_tid=1' 2024-04-06T09:40:51.475 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:40:51.475 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:40:51.475 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:40:51.475 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:40:51.475 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:40:51.475 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:40:51.475 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:40:51.475 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:40:51.475 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:40:51.475 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:40:51.476 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:40:51.476 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:40:51.476 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:40:51.476 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:40:51.476 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:40:51.523 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 22 2024-04-06T09:40:51.523 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 23 2024-04-06T09:40:51.523 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:40:51.523 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=93, tag_tid=15, entry_tid=1], [object_number=92, tag_tid=15, entry_tid=0], [object_number=95, tag_tid=14, entry_tid=19039], [object_number=94, tag_tid=14, entry_tid=19038]]], state=connected] 2024-04-06T09:40:51.524 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=91, tag_tid=14, entry_tid=14243], [object_number=90, tag_tid=14, entry_tid=14242], [object_number=89, tag_tid=14, entry_tid=14241], [object_number=88, tag_tid=14, entry_tid=14240]]], state=connected] 2024-04-06T09:40:51.527 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:40:51.528 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=91, tag_tid=14, entry_tid=14243' 2024-04-06T09:40:51.528 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=93, tag_tid=15, entry_tid=1' -a 'object_number=93, tag_tid=15, entry_tid=1' = 'object_number=91, tag_tid=14, entry_tid=14243' 2024-04-06T09:40:51.528 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=91, tag_tid=14, entry_tid=14243' '!=' 'object_number=88, tag_tid=14, entry_tid=14164' 2024-04-06T09:40:51.529 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:40:51.529 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=91, tag_tid=14, entry_tid=14243' = 'object_number=88, tag_tid=14, entry_tid=14164' 2024-04-06T09:40:51.529 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=91, tag_tid=14, entry_tid=14243' 2024-04-06T09:40:51.529 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=93, tag_tid=15, entry_tid=1' 2024-04-06T09:40:51.529 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:40:51.529 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:40:51.530 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=15 2024-04-06T09:40:51.531 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=91, tag_tid=14, entry_tid=14243' 2024-04-06T09:40:51.531 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:40:51.531 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:40:51.532 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=14 2024-04-06T09:40:51.532 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=93, tag_tid=15, entry_tid=1' 2024-04-06T09:40:51.532 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:40:51.532 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:40:51.533 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:40:51.534 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=91, tag_tid=14, entry_tid=14243' 2024-04-06T09:40:51.534 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:40:51.534 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:40:51.535 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=14243 2024-04-06T09:40:51.536 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 15 = 14 -a 1 -le 14243 2024-04-06T09:40:51.536 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:40:51.536 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:40:51.536 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:40:51.737 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:40:51.737 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:40:51.738 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:40:51.738 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:40:51.738 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:40:51.738 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:40:51.738 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:40:51.738 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:40:51.738 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:40:51.738 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:40:51.738 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:40:51.738 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:40:51.738 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:40:51.738 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:40:51.738 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:40:51.739 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:40:51.739 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:40:51.742 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:40:51.742 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:40:51.742 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:40:51.742 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:40:51.742 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:40:51.742 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:40:51.742 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:40:51.743 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:40:51.743 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:40:51.743 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:40:51.743 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:40:51.743 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:40:51.743 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:40:51.743 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:40:51.743 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:40:51.743 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:40:51.743 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:40:51.743 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:40:51.743 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:40:51.743 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:40:51.743 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:40:51.743 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:40:51.743 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:40:51.744 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:40:51.744 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:40:51.744 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:40:51.744 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:40:51.744 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:40:51.744 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:40:51.744 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:40:51.744 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:40:51.744 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:40:51.768 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:40:51.769 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:40:51.769 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:40:51.769 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:40:51.899 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:40:51.900 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:40:51.900 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:40:51.900 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:40:51.900 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:40:51.900 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:40:51.900 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:40:51.900 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:40:51.900 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:40:51.900 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:40:51.901 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:40:51.901 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:40:51.901 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:40:51.902 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:40:51.903 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:40:51.903 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:40:51.955 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 22 2024-04-06T09:40:51.955 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 23 2024-04-06T09:40:51.955 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:40:51.955 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=93, tag_tid=15, entry_tid=1], [object_number=92, tag_tid=15, entry_tid=0], [object_number=95, tag_tid=14, entry_tid=19039], [object_number=94, tag_tid=14, entry_tid=19038]]], state=connected] 2024-04-06T09:40:51.955 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=89, tag_tid=14, entry_tid=14329], [object_number=88, tag_tid=14, entry_tid=14328], [object_number=91, tag_tid=14, entry_tid=14327], [object_number=90, tag_tid=14, entry_tid=14326]]], state=connected] 2024-04-06T09:40:51.958 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:40:51.960 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=93, tag_tid=15, entry_tid=1' 2024-04-06T09:40:51.961 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:40:51.961 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:40:51.961 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:40:51.961 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:40:51.961 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:40:51.961 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:40:51.961 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:40:51.961 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:40:51.961 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:40:51.961 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:40:51.962 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:40:51.962 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:40:51.962 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:40:51.962 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:40:51.963 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:40:52.010 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 22 2024-04-06T09:40:52.010 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 23 2024-04-06T09:40:52.010 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:40:52.010 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=93, tag_tid=15, entry_tid=1], [object_number=92, tag_tid=15, entry_tid=0], [object_number=95, tag_tid=14, entry_tid=19039], [object_number=94, tag_tid=14, entry_tid=19038]]], state=connected] 2024-04-06T09:40:52.010 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=89, tag_tid=14, entry_tid=14329], [object_number=88, tag_tid=14, entry_tid=14328], [object_number=91, tag_tid=14, entry_tid=14327], [object_number=90, tag_tid=14, entry_tid=14326]]], state=connected] 2024-04-06T09:40:52.014 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:40:52.016 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=89, tag_tid=14, entry_tid=14329' 2024-04-06T09:40:52.016 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=93, tag_tid=15, entry_tid=1' -a 'object_number=93, tag_tid=15, entry_tid=1' = 'object_number=89, tag_tid=14, entry_tid=14329' 2024-04-06T09:40:52.016 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=89, tag_tid=14, entry_tid=14329' '!=' 'object_number=91, tag_tid=14, entry_tid=14243' 2024-04-06T09:40:52.016 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:40:52.016 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=89, tag_tid=14, entry_tid=14329' = 'object_number=91, tag_tid=14, entry_tid=14243' 2024-04-06T09:40:52.016 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=89, tag_tid=14, entry_tid=14329' 2024-04-06T09:40:52.016 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=93, tag_tid=15, entry_tid=1' 2024-04-06T09:40:52.016 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:40:52.016 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:40:52.017 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=15 2024-04-06T09:40:52.018 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=89, tag_tid=14, entry_tid=14329' 2024-04-06T09:40:52.018 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:40:52.018 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:40:52.019 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=14 2024-04-06T09:40:52.019 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=93, tag_tid=15, entry_tid=1' 2024-04-06T09:40:52.020 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:40:52.020 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:40:52.021 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:40:52.021 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=89, tag_tid=14, entry_tid=14329' 2024-04-06T09:40:52.021 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:40:52.021 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:40:52.023 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=14329 2024-04-06T09:40:52.023 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 15 = 14 -a 1 -le 14329 2024-04-06T09:40:52.023 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:40:52.023 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:40:52.023 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:40:52.224 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:40:52.224 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:40:52.224 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:40:52.225 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:40:52.225 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:40:52.225 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:40:52.225 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:40:52.225 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:40:52.225 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:40:52.225 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:40:52.225 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:40:52.225 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:40:52.225 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:40:52.225 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:40:52.225 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:40:52.226 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:40:52.226 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:40:52.228 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:40:52.229 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:40:52.229 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:40:52.229 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:40:52.229 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:40:52.229 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:40:52.229 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:40:52.229 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:40:52.229 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:40:52.229 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:40:52.229 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:40:52.229 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:40:52.229 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:40:52.229 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:40:52.229 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:40:52.229 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:40:52.230 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:40:52.230 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:40:52.231 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:40:52.231 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:40:52.231 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:40:52.231 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:40:52.231 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:40:52.231 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:40:52.231 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:40:52.231 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:40:52.231 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:40:52.231 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:40:52.231 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:40:52.231 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:40:52.231 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:40:52.231 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:40:52.256 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:40:52.256 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:40:52.256 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:40:52.256 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:40:52.384 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:40:52.385 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:40:52.385 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:40:52.385 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:40:52.385 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:40:52.385 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:40:52.385 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:40:52.386 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:40:52.386 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:40:52.386 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:40:52.386 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:40:52.386 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:40:52.387 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:40:52.389 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:40:52.390 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:40:52.390 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:40:52.442 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 22 2024-04-06T09:40:52.442 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 23 2024-04-06T09:40:52.442 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:40:52.442 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=93, tag_tid=15, entry_tid=1], [object_number=92, tag_tid=15, entry_tid=0], [object_number=95, tag_tid=14, entry_tid=19039], [object_number=94, tag_tid=14, entry_tid=19038]]], state=connected] 2024-04-06T09:40:52.442 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=89, tag_tid=14, entry_tid=14421], [object_number=88, tag_tid=14, entry_tid=14420], [object_number=91, tag_tid=14, entry_tid=14419], [object_number=90, tag_tid=14, entry_tid=14418]]], state=connected] 2024-04-06T09:40:52.445 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:40:52.447 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=93, tag_tid=15, entry_tid=1' 2024-04-06T09:40:52.448 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:40:52.448 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:40:52.448 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:40:52.448 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:40:52.449 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:40:52.449 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:40:52.449 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:40:52.449 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:40:52.449 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:40:52.449 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:40:52.449 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:40:52.449 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:40:52.450 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:40:52.450 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:40:52.450 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:40:52.495 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 22 2024-04-06T09:40:52.495 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 23 2024-04-06T09:40:52.495 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:40:52.495 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=93, tag_tid=15, entry_tid=1], [object_number=92, tag_tid=15, entry_tid=0], [object_number=95, tag_tid=14, entry_tid=19039], [object_number=94, tag_tid=14, entry_tid=19038]]], state=connected] 2024-04-06T09:40:52.495 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=89, tag_tid=14, entry_tid=14421], [object_number=88, tag_tid=14, entry_tid=14420], [object_number=91, tag_tid=14, entry_tid=14419], [object_number=90, tag_tid=14, entry_tid=14418]]], state=connected] 2024-04-06T09:40:52.499 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:40:52.501 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=89, tag_tid=14, entry_tid=14421' 2024-04-06T09:40:52.501 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=93, tag_tid=15, entry_tid=1' -a 'object_number=93, tag_tid=15, entry_tid=1' = 'object_number=89, tag_tid=14, entry_tid=14421' 2024-04-06T09:40:52.501 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=89, tag_tid=14, entry_tid=14421' '!=' 'object_number=89, tag_tid=14, entry_tid=14329' 2024-04-06T09:40:52.501 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:40:52.501 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=89, tag_tid=14, entry_tid=14421' = 'object_number=89, tag_tid=14, entry_tid=14329' 2024-04-06T09:40:52.501 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=89, tag_tid=14, entry_tid=14421' 2024-04-06T09:40:52.501 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=93, tag_tid=15, entry_tid=1' 2024-04-06T09:40:52.501 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:40:52.502 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:40:52.502 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=15 2024-04-06T09:40:52.503 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=89, tag_tid=14, entry_tid=14421' 2024-04-06T09:40:52.503 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:40:52.503 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:40:52.504 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=14 2024-04-06T09:40:52.504 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=93, tag_tid=15, entry_tid=1' 2024-04-06T09:40:52.504 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:40:52.505 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:40:52.505 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:40:52.506 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=89, tag_tid=14, entry_tid=14421' 2024-04-06T09:40:52.506 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:40:52.506 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:40:52.507 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=14421 2024-04-06T09:40:52.507 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 15 = 14 -a 1 -le 14421 2024-04-06T09:40:52.507 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:40:52.507 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:40:52.508 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:40:52.709 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:40:52.710 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:40:52.710 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:40:52.710 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:40:52.710 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:40:52.710 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:40:52.710 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:40:52.710 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:40:52.710 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:40:52.710 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:40:52.710 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:40:52.710 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:40:52.710 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:40:52.711 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:40:52.711 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:40:52.711 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:40:52.711 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:40:52.713 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:40:52.713 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:40:52.713 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:40:52.713 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:40:52.713 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:40:52.714 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:40:52.714 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:40:52.714 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:40:52.714 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:40:52.714 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:40:52.714 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:40:52.714 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:40:52.714 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:40:52.714 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:40:52.714 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:40:52.714 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:40:52.715 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:40:52.715 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:40:52.715 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:40:52.716 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:40:52.716 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:40:52.716 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:40:52.716 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:40:52.716 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:40:52.716 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:40:52.716 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:40:52.716 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:40:52.716 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:40:52.716 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:40:52.716 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:40:52.716 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:40:52.716 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:40:52.740 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:40:52.740 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:40:52.740 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:40:52.741 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:40:52.875 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:40:52.876 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:40:52.876 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:40:52.876 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:40:52.876 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:40:52.876 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:40:52.876 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:40:52.876 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:40:52.876 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:40:52.876 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:40:52.877 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:40:52.877 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:40:52.877 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:40:52.878 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:40:52.879 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:40:52.879 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:40:52.932 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 22 2024-04-06T09:40:52.932 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 23 2024-04-06T09:40:52.932 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:40:52.932 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=93, tag_tid=15, entry_tid=1], [object_number=92, tag_tid=15, entry_tid=0], [object_number=95, tag_tid=14, entry_tid=19039], [object_number=94, tag_tid=14, entry_tid=19038]]], state=connected] 2024-04-06T09:40:52.932 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=90, tag_tid=14, entry_tid=14510], [object_number=89, tag_tid=14, entry_tid=14509], [object_number=88, tag_tid=14, entry_tid=14508], [object_number=91, tag_tid=14, entry_tid=14507]]], state=connected] 2024-04-06T09:40:52.936 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:40:52.937 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=93, tag_tid=15, entry_tid=1' 2024-04-06T09:40:52.938 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:40:52.938 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:40:52.938 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:40:52.938 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:40:52.939 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:40:52.939 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:40:52.939 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:40:52.939 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:40:52.939 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:40:52.939 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:40:52.939 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:40:52.939 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:40:52.939 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:40:52.940 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:40:52.940 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:40:52.988 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 22 2024-04-06T09:40:52.988 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 23 2024-04-06T09:40:52.988 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:40:52.988 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=93, tag_tid=15, entry_tid=1], [object_number=92, tag_tid=15, entry_tid=0], [object_number=95, tag_tid=14, entry_tid=19039], [object_number=94, tag_tid=14, entry_tid=19038]]], state=connected] 2024-04-06T09:40:52.988 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=90, tag_tid=14, entry_tid=14510], [object_number=89, tag_tid=14, entry_tid=14509], [object_number=88, tag_tid=14, entry_tid=14508], [object_number=91, tag_tid=14, entry_tid=14507]]], state=connected] 2024-04-06T09:40:52.991 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:40:52.993 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=90, tag_tid=14, entry_tid=14510' 2024-04-06T09:40:52.993 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=93, tag_tid=15, entry_tid=1' -a 'object_number=93, tag_tid=15, entry_tid=1' = 'object_number=90, tag_tid=14, entry_tid=14510' 2024-04-06T09:40:52.993 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=90, tag_tid=14, entry_tid=14510' '!=' 'object_number=89, tag_tid=14, entry_tid=14421' 2024-04-06T09:40:52.994 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:40:52.994 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=90, tag_tid=14, entry_tid=14510' = 'object_number=89, tag_tid=14, entry_tid=14421' 2024-04-06T09:40:52.994 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=90, tag_tid=14, entry_tid=14510' 2024-04-06T09:40:52.994 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=93, tag_tid=15, entry_tid=1' 2024-04-06T09:40:52.994 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:40:52.994 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:40:52.994 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=15 2024-04-06T09:40:52.995 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=90, tag_tid=14, entry_tid=14510' 2024-04-06T09:40:52.995 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:40:52.995 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:40:52.996 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=14 2024-04-06T09:40:52.997 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=93, tag_tid=15, entry_tid=1' 2024-04-06T09:40:52.997 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:40:52.997 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:40:52.998 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:40:52.998 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=90, tag_tid=14, entry_tid=14510' 2024-04-06T09:40:52.999 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:40:52.999 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:40:53.000 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=14510 2024-04-06T09:40:53.000 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 15 = 14 -a 1 -le 14510 2024-04-06T09:40:53.000 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:40:53.000 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:40:53.000 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:40:53.203 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:40:53.203 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:40:53.203 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:40:53.203 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:40:53.203 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:40:53.203 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:40:53.204 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:40:53.204 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:40:53.204 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:40:53.204 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:40:53.204 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:40:53.204 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:40:53.204 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:40:53.204 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:40:53.205 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:40:53.209 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:40:53.209 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:40:53.209 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:40:53.209 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:40:53.209 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:40:53.210 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:40:53.210 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:40:53.210 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:40:53.210 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:40:53.210 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:40:53.210 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:40:53.210 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:40:53.211 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:40:53.211 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:40:53.211 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:40:53.211 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:40:53.211 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:40:53.211 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:40:53.211 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:40:53.211 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:40:53.212 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:40:53.212 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:40:53.212 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:40:53.212 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:40:53.212 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:40:53.212 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:40:53.212 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:40:53.212 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:40:53.213 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:40:53.213 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:40:53.213 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:40:53.213 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:40:53.213 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:40:53.213 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:40:53.232 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:40:53.232 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:40:53.233 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:40:53.233 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:40:53.360 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:40:53.361 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:40:53.361 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:40:53.362 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:40:53.362 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:40:53.362 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:40:53.362 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:40:53.362 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:40:53.362 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:40:53.362 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:40:53.363 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:40:53.364 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:40:53.364 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:40:53.364 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:40:53.364 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:40:53.365 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:40:53.416 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 22 2024-04-06T09:40:53.416 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 23 2024-04-06T09:40:53.416 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:40:53.416 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=93, tag_tid=15, entry_tid=1], [object_number=92, tag_tid=15, entry_tid=0], [object_number=95, tag_tid=14, entry_tid=19039], [object_number=94, tag_tid=14, entry_tid=19038]]], state=connected] 2024-04-06T09:40:53.416 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=91, tag_tid=14, entry_tid=14599], [object_number=90, tag_tid=14, entry_tid=14598], [object_number=89, tag_tid=14, entry_tid=14597], [object_number=88, tag_tid=14, entry_tid=14596]]], state=connected] 2024-04-06T09:40:53.420 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:40:53.421 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=93, tag_tid=15, entry_tid=1' 2024-04-06T09:40:53.422 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:40:53.422 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:40:53.422 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:40:53.422 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:40:53.422 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:40:53.422 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:40:53.423 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:40:53.423 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:40:53.423 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:40:53.423 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:40:53.423 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:40:53.423 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:40:53.424 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:40:53.424 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:40:53.424 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:40:53.469 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 22 2024-04-06T09:40:53.469 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 23 2024-04-06T09:40:53.470 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:40:53.470 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=93, tag_tid=15, entry_tid=1], [object_number=92, tag_tid=15, entry_tid=0], [object_number=95, tag_tid=14, entry_tid=19039], [object_number=94, tag_tid=14, entry_tid=19038]]], state=connected] 2024-04-06T09:40:53.470 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=91, tag_tid=14, entry_tid=14599], [object_number=90, tag_tid=14, entry_tid=14598], [object_number=89, tag_tid=14, entry_tid=14597], [object_number=88, tag_tid=14, entry_tid=14596]]], state=connected] 2024-04-06T09:40:53.473 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:40:53.475 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=91, tag_tid=14, entry_tid=14599' 2024-04-06T09:40:53.475 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=93, tag_tid=15, entry_tid=1' -a 'object_number=93, tag_tid=15, entry_tid=1' = 'object_number=91, tag_tid=14, entry_tid=14599' 2024-04-06T09:40:53.475 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=91, tag_tid=14, entry_tid=14599' '!=' 'object_number=90, tag_tid=14, entry_tid=14510' 2024-04-06T09:40:53.475 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:40:53.476 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=91, tag_tid=14, entry_tid=14599' = 'object_number=90, tag_tid=14, entry_tid=14510' 2024-04-06T09:40:53.476 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=91, tag_tid=14, entry_tid=14599' 2024-04-06T09:40:53.476 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=93, tag_tid=15, entry_tid=1' 2024-04-06T09:40:53.476 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:40:53.476 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:40:53.476 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=15 2024-04-06T09:40:53.477 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=91, tag_tid=14, entry_tid=14599' 2024-04-06T09:40:53.477 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:40:53.477 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:40:53.478 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=14 2024-04-06T09:40:53.479 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=93, tag_tid=15, entry_tid=1' 2024-04-06T09:40:53.479 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:40:53.479 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:40:53.480 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:40:53.480 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=91, tag_tid=14, entry_tid=14599' 2024-04-06T09:40:53.480 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:40:53.480 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:40:53.481 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=14599 2024-04-06T09:40:53.482 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 15 = 14 -a 1 -le 14599 2024-04-06T09:40:53.482 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:40:53.482 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:40:53.482 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:40:53.683 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:40:53.683 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:40:53.683 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:40:53.683 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:40:53.684 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:40:53.684 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:40:53.684 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:40:53.684 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:40:53.684 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:40:53.684 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:40:53.684 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:40:53.684 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:40:53.684 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:40:53.684 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:40:53.684 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:40:53.684 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:40:53.685 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:40:53.687 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:40:53.688 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:40:53.688 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:40:53.688 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:40:53.688 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:40:53.688 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:40:53.688 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:40:53.688 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:40:53.688 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:40:53.688 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:40:53.688 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:40:53.688 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:40:53.688 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:40:53.688 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:40:53.688 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:40:53.688 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:40:53.689 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:40:53.689 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:40:53.689 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:40:53.689 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:40:53.689 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:40:53.689 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:40:53.689 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:40:53.689 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:40:53.689 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:40:53.689 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:40:53.689 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:40:53.689 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:40:53.690 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:40:53.690 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:40:53.690 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:40:53.690 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:40:53.714 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:40:53.714 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:40:53.714 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:40:53.715 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:40:53.849 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:40:53.850 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:40:53.850 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:40:53.850 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:40:53.850 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:40:53.850 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:40:53.851 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:40:53.851 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:40:53.851 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:40:53.851 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:40:53.851 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:40:53.851 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:40:53.852 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:40:53.854 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:40:53.855 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:40:53.855 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:40:53.900 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 22 2024-04-06T09:40:53.901 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 23 2024-04-06T09:40:53.901 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:40:53.901 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=93, tag_tid=15, entry_tid=1], [object_number=92, tag_tid=15, entry_tid=0], [object_number=95, tag_tid=14, entry_tid=19039], [object_number=94, tag_tid=14, entry_tid=19038]]], state=connected] 2024-04-06T09:40:53.901 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=91, tag_tid=14, entry_tid=14695], [object_number=90, tag_tid=14, entry_tid=14694], [object_number=89, tag_tid=14, entry_tid=14693], [object_number=88, tag_tid=14, entry_tid=14692]]], state=connected] 2024-04-06T09:40:53.904 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:40:53.905 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=93, tag_tid=15, entry_tid=1' 2024-04-06T09:40:53.907 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:40:53.907 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:40:53.907 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:40:53.907 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:40:53.907 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:40:53.907 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:40:53.907 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:40:53.907 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:40:53.907 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:40:53.907 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:40:53.907 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:40:53.907 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:40:53.908 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:40:53.908 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:40:53.908 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:40:53.953 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 22 2024-04-06T09:40:53.953 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 23 2024-04-06T09:40:53.953 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:40:53.953 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=93, tag_tid=15, entry_tid=1], [object_number=92, tag_tid=15, entry_tid=0], [object_number=95, tag_tid=14, entry_tid=19039], [object_number=94, tag_tid=14, entry_tid=19038]]], state=connected] 2024-04-06T09:40:53.954 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=91, tag_tid=14, entry_tid=14695], [object_number=90, tag_tid=14, entry_tid=14694], [object_number=89, tag_tid=14, entry_tid=14693], [object_number=88, tag_tid=14, entry_tid=14692]]], state=connected] 2024-04-06T09:40:53.957 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:40:53.959 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=91, tag_tid=14, entry_tid=14695' 2024-04-06T09:40:53.959 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=93, tag_tid=15, entry_tid=1' -a 'object_number=93, tag_tid=15, entry_tid=1' = 'object_number=91, tag_tid=14, entry_tid=14695' 2024-04-06T09:40:53.959 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=91, tag_tid=14, entry_tid=14695' '!=' 'object_number=91, tag_tid=14, entry_tid=14599' 2024-04-06T09:40:53.959 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:40:53.959 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=91, tag_tid=14, entry_tid=14695' = 'object_number=91, tag_tid=14, entry_tid=14599' 2024-04-06T09:40:53.959 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=91, tag_tid=14, entry_tid=14695' 2024-04-06T09:40:53.959 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=93, tag_tid=15, entry_tid=1' 2024-04-06T09:40:53.959 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:40:53.960 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:40:53.960 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=15 2024-04-06T09:40:53.961 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=91, tag_tid=14, entry_tid=14695' 2024-04-06T09:40:53.961 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:40:53.961 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:40:53.962 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=14 2024-04-06T09:40:53.962 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=93, tag_tid=15, entry_tid=1' 2024-04-06T09:40:53.963 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:40:53.963 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:40:53.964 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:40:53.964 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=91, tag_tid=14, entry_tid=14695' 2024-04-06T09:40:53.964 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:40:53.964 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:40:53.965 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=14695 2024-04-06T09:40:53.966 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 15 = 14 -a 1 -le 14695 2024-04-06T09:40:53.966 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:40:53.966 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:40:53.966 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:40:54.167 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:40:54.167 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:40:54.167 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:40:54.167 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:40:54.168 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:40:54.168 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:40:54.168 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:40:54.168 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:40:54.168 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:40:54.168 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:40:54.168 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:40:54.168 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:40:54.168 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:40:54.168 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:40:54.168 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:40:54.169 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:40:54.169 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:40:54.173 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:40:54.173 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:40:54.173 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:40:54.173 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:40:54.173 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:40:54.173 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:40:54.173 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:40:54.173 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:40:54.173 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:40:54.173 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:40:54.174 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:40:54.174 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:40:54.174 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:40:54.174 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:40:54.174 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:40:54.174 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:40:54.174 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:40:54.174 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:40:54.174 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:40:54.174 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:40:54.174 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:40:54.174 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:40:54.175 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:40:54.175 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:40:54.175 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:40:54.175 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:40:54.175 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:40:54.175 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:40:54.175 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:40:54.175 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:40:54.175 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:40:54.175 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:40:54.199 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:40:54.199 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:40:54.199 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:40:54.199 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:40:54.327 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:40:54.337 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:40:54.337 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:40:54.337 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:40:54.338 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:40:54.338 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:40:54.338 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:40:54.338 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:40:54.338 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:40:54.338 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:40:54.338 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:40:54.338 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:40:54.338 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:40:54.338 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:40:54.338 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:40:54.338 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:40:54.381 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 22 2024-04-06T09:40:54.382 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 23 2024-04-06T09:40:54.382 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:40:54.382 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=93, tag_tid=15, entry_tid=1], [object_number=92, tag_tid=15, entry_tid=0], [object_number=95, tag_tid=14, entry_tid=19039], [object_number=94, tag_tid=14, entry_tid=19038]]], state=connected] 2024-04-06T09:40:54.382 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=90, tag_tid=14, entry_tid=14790], [object_number=89, tag_tid=14, entry_tid=14789], [object_number=88, tag_tid=14, entry_tid=14788], [object_number=91, tag_tid=14, entry_tid=14787]]], state=connected] 2024-04-06T09:40:54.385 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:40:54.386 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=93, tag_tid=15, entry_tid=1' 2024-04-06T09:40:54.387 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:40:54.388 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:40:54.388 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:40:54.388 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:40:54.388 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:40:54.388 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:40:54.388 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:40:54.388 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:40:54.388 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:40:54.388 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:40:54.389 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:40:54.389 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:40:54.390 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:40:54.390 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:40:54.390 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:40:54.434 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 22 2024-04-06T09:40:54.435 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 23 2024-04-06T09:40:54.435 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:40:54.435 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=93, tag_tid=15, entry_tid=1], [object_number=92, tag_tid=15, entry_tid=0], [object_number=95, tag_tid=14, entry_tid=19039], [object_number=94, tag_tid=14, entry_tid=19038]]], state=connected] 2024-04-06T09:40:54.435 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=90, tag_tid=14, entry_tid=14790], [object_number=89, tag_tid=14, entry_tid=14789], [object_number=88, tag_tid=14, entry_tid=14788], [object_number=91, tag_tid=14, entry_tid=14787]]], state=connected] 2024-04-06T09:40:54.438 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:40:54.440 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=90, tag_tid=14, entry_tid=14790' 2024-04-06T09:40:54.440 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=93, tag_tid=15, entry_tid=1' -a 'object_number=93, tag_tid=15, entry_tid=1' = 'object_number=90, tag_tid=14, entry_tid=14790' 2024-04-06T09:40:54.440 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=90, tag_tid=14, entry_tid=14790' '!=' 'object_number=91, tag_tid=14, entry_tid=14695' 2024-04-06T09:40:54.440 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:40:54.440 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=90, tag_tid=14, entry_tid=14790' = 'object_number=91, tag_tid=14, entry_tid=14695' 2024-04-06T09:40:54.440 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=90, tag_tid=14, entry_tid=14790' 2024-04-06T09:40:54.441 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=93, tag_tid=15, entry_tid=1' 2024-04-06T09:40:54.441 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:40:54.441 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:40:54.442 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=15 2024-04-06T09:40:54.442 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=90, tag_tid=14, entry_tid=14790' 2024-04-06T09:40:54.442 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:40:54.442 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:40:54.443 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=14 2024-04-06T09:40:54.444 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=93, tag_tid=15, entry_tid=1' 2024-04-06T09:40:54.444 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:40:54.444 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:40:54.444 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:40:54.445 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=90, tag_tid=14, entry_tid=14790' 2024-04-06T09:40:54.445 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:40:54.446 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:40:54.446 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=14790 2024-04-06T09:40:54.447 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 15 = 14 -a 1 -le 14790 2024-04-06T09:40:54.447 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:40:54.447 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:40:54.447 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:40:54.648 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:40:54.648 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:40:54.648 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:40:54.649 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:40:54.649 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:40:54.649 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:40:54.649 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:40:54.649 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:40:54.649 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:40:54.649 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:40:54.649 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:40:54.649 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:40:54.649 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:40:54.649 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:40:54.649 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:40:54.650 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:40:54.650 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:40:54.653 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:40:54.653 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:40:54.653 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:40:54.653 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:40:54.653 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:40:54.653 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:40:54.653 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:40:54.653 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:40:54.653 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:40:54.653 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:40:54.654 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:40:54.654 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:40:54.654 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:40:54.654 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:40:54.654 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:40:54.654 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:40:54.654 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:40:54.654 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:40:54.654 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:40:54.654 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:40:54.655 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:40:54.655 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:40:54.655 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:40:54.655 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:40:54.655 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:40:54.655 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:40:54.655 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:40:54.655 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:40:54.655 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:40:54.655 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:40:54.655 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:40:54.656 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:40:54.679 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:40:54.680 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:40:54.680 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:40:54.680 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:40:54.808 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:40:54.809 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:40:54.809 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:40:54.809 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:40:54.809 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:40:54.809 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:40:54.809 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:40:54.809 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:40:54.809 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:40:54.810 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:40:54.810 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:40:54.810 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:40:54.810 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:40:54.812 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:40:54.812 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:40:54.812 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:40:54.857 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 22 2024-04-06T09:40:54.857 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 23 2024-04-06T09:40:54.857 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:40:54.857 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=93, tag_tid=15, entry_tid=1], [object_number=92, tag_tid=15, entry_tid=0], [object_number=95, tag_tid=14, entry_tid=19039], [object_number=94, tag_tid=14, entry_tid=19038]]], state=connected] 2024-04-06T09:40:54.857 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=91, tag_tid=14, entry_tid=14887], [object_number=90, tag_tid=14, entry_tid=14886], [object_number=89, tag_tid=14, entry_tid=14885], [object_number=88, tag_tid=14, entry_tid=14884]]], state=connected] 2024-04-06T09:40:54.861 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:40:54.863 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=93, tag_tid=15, entry_tid=1' 2024-04-06T09:40:54.863 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:40:54.863 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:40:54.863 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:40:54.864 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:40:54.864 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:40:54.864 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:40:54.864 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:40:54.864 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:40:54.864 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:40:54.864 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:40:54.865 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:40:54.865 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:40:54.865 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:40:54.866 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:40:54.866 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:40:54.910 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 22 2024-04-06T09:40:54.911 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 23 2024-04-06T09:40:54.911 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:40:54.911 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=93, tag_tid=15, entry_tid=1], [object_number=92, tag_tid=15, entry_tid=0], [object_number=95, tag_tid=14, entry_tid=19039], [object_number=94, tag_tid=14, entry_tid=19038]]], state=connected] 2024-04-06T09:40:54.911 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=91, tag_tid=14, entry_tid=14887], [object_number=90, tag_tid=14, entry_tid=14886], [object_number=89, tag_tid=14, entry_tid=14885], [object_number=88, tag_tid=14, entry_tid=14884]]], state=connected] 2024-04-06T09:40:54.914 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:40:54.916 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=91, tag_tid=14, entry_tid=14887' 2024-04-06T09:40:54.916 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=93, tag_tid=15, entry_tid=1' -a 'object_number=93, tag_tid=15, entry_tid=1' = 'object_number=91, tag_tid=14, entry_tid=14887' 2024-04-06T09:40:54.916 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=91, tag_tid=14, entry_tid=14887' '!=' 'object_number=90, tag_tid=14, entry_tid=14790' 2024-04-06T09:40:54.916 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:40:54.916 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=91, tag_tid=14, entry_tid=14887' = 'object_number=90, tag_tid=14, entry_tid=14790' 2024-04-06T09:40:54.916 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=91, tag_tid=14, entry_tid=14887' 2024-04-06T09:40:54.916 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=93, tag_tid=15, entry_tid=1' 2024-04-06T09:40:54.917 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:40:54.917 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:40:54.918 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=15 2024-04-06T09:40:54.918 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=91, tag_tid=14, entry_tid=14887' 2024-04-06T09:40:54.918 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:40:54.918 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:40:54.919 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=14 2024-04-06T09:40:54.919 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=93, tag_tid=15, entry_tid=1' 2024-04-06T09:40:54.919 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:40:54.919 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:40:54.920 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:40:54.921 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=91, tag_tid=14, entry_tid=14887' 2024-04-06T09:40:54.921 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:40:54.921 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:40:54.922 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=14887 2024-04-06T09:40:54.922 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 15 = 14 -a 1 -le 14887 2024-04-06T09:40:54.922 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:40:54.923 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:40:54.923 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:40:55.124 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:40:55.125 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:40:55.125 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:40:55.125 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:40:55.125 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:40:55.125 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:40:55.125 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:40:55.125 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:40:55.125 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:40:55.125 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:40:55.125 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:40:55.125 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:40:55.125 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:40:55.125 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:40:55.126 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:40:55.126 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:40:55.126 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:40:55.128 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:40:55.128 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:40:55.128 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:40:55.129 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:40:55.129 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:40:55.129 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:40:55.129 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:40:55.129 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:40:55.129 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:40:55.129 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:40:55.129 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:40:55.129 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:40:55.129 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:40:55.129 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:40:55.129 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:40:55.129 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:40:55.129 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:40:55.129 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:40:55.130 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:40:55.130 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:40:55.130 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:40:55.130 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:40:55.130 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:40:55.130 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:40:55.130 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:40:55.130 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:40:55.130 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:40:55.130 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:40:55.130 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:40:55.130 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:40:55.130 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:40:55.131 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:40:55.156 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:40:55.156 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:40:55.156 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:40:55.156 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:40:55.282 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:40:55.283 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:40:55.283 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:40:55.283 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:40:55.283 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:40:55.283 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:40:55.283 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:40:55.283 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:40:55.283 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:40:55.283 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:40:55.284 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:40:55.284 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:40:55.284 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:40:55.286 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:40:55.287 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:40:55.287 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:40:55.341 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 22 2024-04-06T09:40:55.341 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 23 2024-04-06T09:40:55.341 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:40:55.341 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=93, tag_tid=15, entry_tid=1], [object_number=92, tag_tid=15, entry_tid=0], [object_number=95, tag_tid=14, entry_tid=19039], [object_number=94, tag_tid=14, entry_tid=19038]]], state=connected] 2024-04-06T09:40:55.342 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=88, tag_tid=14, entry_tid=14976], [object_number=91, tag_tid=14, entry_tid=14975], [object_number=90, tag_tid=14, entry_tid=14974], [object_number=89, tag_tid=14, entry_tid=14973]]], state=connected] 2024-04-06T09:40:55.345 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:40:55.347 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=93, tag_tid=15, entry_tid=1' 2024-04-06T09:40:55.347 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:40:55.347 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:40:55.347 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:40:55.348 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:40:55.348 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:40:55.348 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:40:55.348 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:40:55.348 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:40:55.348 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:40:55.348 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:40:55.348 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:40:55.348 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:40:55.349 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:40:55.349 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:40:55.349 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:40:55.395 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 22 2024-04-06T09:40:55.395 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 23 2024-04-06T09:40:55.395 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:40:55.395 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=93, tag_tid=15, entry_tid=1], [object_number=92, tag_tid=15, entry_tid=0], [object_number=95, tag_tid=14, entry_tid=19039], [object_number=94, tag_tid=14, entry_tid=19038]]], state=connected] 2024-04-06T09:40:55.395 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=88, tag_tid=14, entry_tid=14976], [object_number=91, tag_tid=14, entry_tid=14975], [object_number=90, tag_tid=14, entry_tid=14974], [object_number=89, tag_tid=14, entry_tid=14973]]], state=connected] 2024-04-06T09:40:55.399 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:40:55.400 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=88, tag_tid=14, entry_tid=14976' 2024-04-06T09:40:55.400 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=93, tag_tid=15, entry_tid=1' -a 'object_number=93, tag_tid=15, entry_tid=1' = 'object_number=88, tag_tid=14, entry_tid=14976' 2024-04-06T09:40:55.400 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=88, tag_tid=14, entry_tid=14976' '!=' 'object_number=91, tag_tid=14, entry_tid=14887' 2024-04-06T09:40:55.400 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:40:55.401 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=88, tag_tid=14, entry_tid=14976' = 'object_number=91, tag_tid=14, entry_tid=14887' 2024-04-06T09:40:55.401 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=88, tag_tid=14, entry_tid=14976' 2024-04-06T09:40:55.401 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=93, tag_tid=15, entry_tid=1' 2024-04-06T09:40:55.401 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:40:55.401 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:40:55.402 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=15 2024-04-06T09:40:55.403 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=88, tag_tid=14, entry_tid=14976' 2024-04-06T09:40:55.403 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:40:55.403 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:40:55.403 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=14 2024-04-06T09:40:55.404 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=93, tag_tid=15, entry_tid=1' 2024-04-06T09:40:55.405 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:40:55.405 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:40:55.405 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:40:55.406 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=88, tag_tid=14, entry_tid=14976' 2024-04-06T09:40:55.406 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:40:55.406 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:40:55.407 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=14976 2024-04-06T09:40:55.407 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 15 = 14 -a 1 -le 14976 2024-04-06T09:40:55.407 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:40:55.407 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:40:55.407 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:40:55.609 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:40:55.609 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:40:55.609 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:40:55.609 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:40:55.609 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:40:55.609 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:40:55.609 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:40:55.609 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:40:55.610 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:40:55.610 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:40:55.610 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:40:55.610 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:40:55.610 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:40:55.610 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:40:55.610 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:40:55.611 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:40:55.611 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:40:55.613 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:40:55.613 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:40:55.613 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:40:55.613 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:40:55.613 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:40:55.613 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:40:55.613 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:40:55.613 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:40:55.614 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:40:55.614 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:40:55.614 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:40:55.614 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:40:55.614 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:40:55.614 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:40:55.614 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:40:55.614 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:40:55.614 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:40:55.614 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:40:55.614 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:40:55.614 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:40:55.614 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:40:55.614 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:40:55.615 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:40:55.615 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:40:55.615 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:40:55.615 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:40:55.615 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:40:55.615 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:40:55.615 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:40:55.615 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:40:55.615 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:40:55.615 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:40:55.640 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:40:55.640 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:40:55.640 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:40:55.640 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:40:55.766 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:40:55.767 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:40:55.767 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:40:55.768 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:40:55.768 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:40:55.768 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:40:55.768 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:40:55.768 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:40:55.768 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:40:55.768 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:40:55.768 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:40:55.768 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:40:55.769 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:40:55.771 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:40:55.771 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:40:55.771 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:40:55.825 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 22 2024-04-06T09:40:55.825 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 23 2024-04-06T09:40:55.825 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:40:55.825 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=93, tag_tid=15, entry_tid=1], [object_number=92, tag_tid=15, entry_tid=0], [object_number=95, tag_tid=14, entry_tid=19039], [object_number=94, tag_tid=14, entry_tid=19038]]], state=connected] 2024-04-06T09:40:55.825 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=88, tag_tid=14, entry_tid=15064], [object_number=91, tag_tid=14, entry_tid=15063], [object_number=90, tag_tid=14, entry_tid=15062], [object_number=89, tag_tid=14, entry_tid=15061]]], state=connected] 2024-04-06T09:40:55.829 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:40:55.830 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=93, tag_tid=15, entry_tid=1' 2024-04-06T09:40:55.831 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:40:55.831 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:40:55.831 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:40:55.831 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:40:55.831 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:40:55.831 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:40:55.831 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:40:55.832 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:40:55.832 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:40:55.832 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:40:55.832 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:40:55.832 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:40:55.832 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:40:55.833 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:40:55.833 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:40:55.879 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 22 2024-04-06T09:40:55.879 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 23 2024-04-06T09:40:55.879 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:40:55.879 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=93, tag_tid=15, entry_tid=1], [object_number=92, tag_tid=15, entry_tid=0], [object_number=95, tag_tid=14, entry_tid=19039], [object_number=94, tag_tid=14, entry_tid=19038]]], state=connected] 2024-04-06T09:40:55.879 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=88, tag_tid=14, entry_tid=15064], [object_number=91, tag_tid=14, entry_tid=15063], [object_number=90, tag_tid=14, entry_tid=15062], [object_number=89, tag_tid=14, entry_tid=15061]]], state=connected] 2024-04-06T09:40:55.883 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:40:55.885 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=88, tag_tid=14, entry_tid=15064' 2024-04-06T09:40:55.885 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=93, tag_tid=15, entry_tid=1' -a 'object_number=93, tag_tid=15, entry_tid=1' = 'object_number=88, tag_tid=14, entry_tid=15064' 2024-04-06T09:40:55.885 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=88, tag_tid=14, entry_tid=15064' '!=' 'object_number=88, tag_tid=14, entry_tid=14976' 2024-04-06T09:40:55.885 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:40:55.885 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=88, tag_tid=14, entry_tid=15064' = 'object_number=88, tag_tid=14, entry_tid=14976' 2024-04-06T09:40:55.885 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=88, tag_tid=14, entry_tid=15064' 2024-04-06T09:40:55.885 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=93, tag_tid=15, entry_tid=1' 2024-04-06T09:40:55.885 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:40:55.885 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:40:55.886 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=15 2024-04-06T09:40:55.891 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=88, tag_tid=14, entry_tid=15064' 2024-04-06T09:40:55.891 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:40:55.891 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:40:55.892 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=14 2024-04-06T09:40:55.892 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=93, tag_tid=15, entry_tid=1' 2024-04-06T09:40:55.892 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:40:55.892 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:40:55.892 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:40:55.892 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=88, tag_tid=14, entry_tid=15064' 2024-04-06T09:40:55.892 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:40:55.892 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:40:55.892 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=15064 2024-04-06T09:40:55.892 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 15 = 14 -a 1 -le 15064 2024-04-06T09:40:55.892 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:40:55.892 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:40:55.892 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:40:56.095 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:40:56.095 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:40:56.095 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:40:56.095 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:40:56.095 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:40:56.095 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:40:56.095 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:40:56.095 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:40:56.095 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:40:56.095 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:40:56.095 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:40:56.096 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:40:56.096 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:40:56.096 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:40:56.096 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:40:56.096 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:40:56.098 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:40:56.098 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:40:56.098 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:40:56.098 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:40:56.099 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:40:56.099 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:40:56.099 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:40:56.099 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:40:56.099 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:40:56.099 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:40:56.099 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:40:56.099 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:40:56.099 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:40:56.099 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:40:56.099 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:40:56.099 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:40:56.099 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:40:56.099 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:40:56.099 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:40:56.100 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:40:56.100 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:40:56.100 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:40:56.100 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:40:56.100 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:40:56.100 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:40:56.100 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:40:56.100 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:40:56.100 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:40:56.100 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:40:56.100 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:40:56.100 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:40:56.100 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:40:56.100 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:40:56.118 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:40:56.119 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:40:56.119 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:40:56.119 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:40:56.241 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:40:56.242 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:40:56.243 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:40:56.243 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:40:56.243 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:40:56.243 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:40:56.243 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:40:56.243 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:40:56.243 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:40:56.243 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:40:56.243 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:40:56.244 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:40:56.244 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:40:56.245 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:40:56.246 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:40:56.246 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:40:56.300 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 22 2024-04-06T09:40:56.300 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 23 2024-04-06T09:40:56.300 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:40:56.300 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=93, tag_tid=15, entry_tid=1], [object_number=92, tag_tid=15, entry_tid=0], [object_number=95, tag_tid=14, entry_tid=19039], [object_number=94, tag_tid=14, entry_tid=19038]]], state=connected] 2024-04-06T09:40:56.300 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=89, tag_tid=14, entry_tid=15149], [object_number=88, tag_tid=14, entry_tid=15148], [object_number=91, tag_tid=14, entry_tid=15147], [object_number=90, tag_tid=14, entry_tid=15146]]], state=connected] 2024-04-06T09:40:56.303 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:40:56.304 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=93, tag_tid=15, entry_tid=1' 2024-04-06T09:40:56.305 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:40:56.305 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:40:56.305 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:40:56.305 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:40:56.306 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:40:56.306 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:40:56.306 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:40:56.306 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:40:56.306 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:40:56.306 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:40:56.306 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:40:56.306 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:40:56.307 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:40:56.307 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:40:56.307 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:40:56.350 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 22 2024-04-06T09:40:56.350 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 23 2024-04-06T09:40:56.350 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:40:56.350 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=93, tag_tid=15, entry_tid=1], [object_number=92, tag_tid=15, entry_tid=0], [object_number=95, tag_tid=14, entry_tid=19039], [object_number=94, tag_tid=14, entry_tid=19038]]], state=connected] 2024-04-06T09:40:56.351 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=89, tag_tid=14, entry_tid=15149], [object_number=88, tag_tid=14, entry_tid=15148], [object_number=91, tag_tid=14, entry_tid=15147], [object_number=90, tag_tid=14, entry_tid=15146]]], state=connected] 2024-04-06T09:40:56.353 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:40:56.355 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=89, tag_tid=14, entry_tid=15149' 2024-04-06T09:40:56.355 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=93, tag_tid=15, entry_tid=1' -a 'object_number=93, tag_tid=15, entry_tid=1' = 'object_number=89, tag_tid=14, entry_tid=15149' 2024-04-06T09:40:56.356 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=89, tag_tid=14, entry_tid=15149' '!=' 'object_number=88, tag_tid=14, entry_tid=15064' 2024-04-06T09:40:56.356 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:40:56.356 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=89, tag_tid=14, entry_tid=15149' = 'object_number=88, tag_tid=14, entry_tid=15064' 2024-04-06T09:40:56.356 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=89, tag_tid=14, entry_tid=15149' 2024-04-06T09:40:56.356 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=93, tag_tid=15, entry_tid=1' 2024-04-06T09:40:56.356 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:40:56.356 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:40:56.356 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=15 2024-04-06T09:40:56.357 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=89, tag_tid=14, entry_tid=15149' 2024-04-06T09:40:56.357 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:40:56.357 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:40:56.358 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=14 2024-04-06T09:40:56.358 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=93, tag_tid=15, entry_tid=1' 2024-04-06T09:40:56.359 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:40:56.359 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:40:56.359 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:40:56.360 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=89, tag_tid=14, entry_tid=15149' 2024-04-06T09:40:56.360 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:40:56.360 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:40:56.361 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=15149 2024-04-06T09:40:56.361 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 15 = 14 -a 1 -le 15149 2024-04-06T09:40:56.362 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:40:56.362 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:40:56.362 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:40:56.563 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:40:56.563 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:40:56.563 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:40:56.564 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:40:56.564 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:40:56.564 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:40:56.564 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:40:56.564 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:40:56.564 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:40:56.564 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:40:56.564 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:40:56.564 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:40:56.564 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:40:56.564 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:40:56.564 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:40:56.565 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:40:56.565 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:40:56.567 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:40:56.567 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:40:56.567 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:40:56.567 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:40:56.567 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:40:56.567 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:40:56.568 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:40:56.568 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:40:56.568 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:40:56.568 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:40:56.568 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:40:56.568 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:40:56.568 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:40:56.568 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:40:56.568 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:40:56.568 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:40:56.569 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:40:56.569 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:40:56.569 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:40:56.569 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:40:56.569 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:40:56.569 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:40:56.569 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:40:56.569 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:40:56.570 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:40:56.570 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:40:56.570 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:40:56.570 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:40:56.570 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:40:56.570 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:40:56.570 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:40:56.570 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:40:56.593 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:40:56.593 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:40:56.593 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:40:56.593 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:40:56.730 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:40:56.731 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:40:56.731 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:40:56.731 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:40:56.731 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:40:56.731 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:40:56.731 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:40:56.731 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:40:56.731 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:40:56.731 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:40:56.732 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:40:56.732 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:40:56.732 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:40:56.733 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:40:56.734 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:40:56.734 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:40:56.784 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 22 2024-04-06T09:40:56.784 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 23 2024-04-06T09:40:56.784 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:40:56.784 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=93, tag_tid=15, entry_tid=1], [object_number=92, tag_tid=15, entry_tid=0], [object_number=95, tag_tid=14, entry_tid=19039], [object_number=94, tag_tid=14, entry_tid=19038]]], state=connected] 2024-04-06T09:40:56.784 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=88, tag_tid=14, entry_tid=15232], [object_number=91, tag_tid=14, entry_tid=15231], [object_number=90, tag_tid=14, entry_tid=15230], [object_number=89, tag_tid=14, entry_tid=15229]]], state=connected] 2024-04-06T09:40:56.787 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:40:56.789 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=93, tag_tid=15, entry_tid=1' 2024-04-06T09:40:56.790 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:40:56.790 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:40:56.790 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:40:56.790 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:40:56.790 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:40:56.790 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:40:56.790 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:40:56.790 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:40:56.790 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:40:56.790 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:40:56.790 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:40:56.791 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:40:56.791 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:40:56.791 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:40:56.791 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:40:56.836 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 22 2024-04-06T09:40:56.837 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 23 2024-04-06T09:40:56.837 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:40:56.837 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=93, tag_tid=15, entry_tid=1], [object_number=92, tag_tid=15, entry_tid=0], [object_number=95, tag_tid=14, entry_tid=19039], [object_number=94, tag_tid=14, entry_tid=19038]]], state=connected] 2024-04-06T09:40:56.837 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=88, tag_tid=14, entry_tid=15232], [object_number=91, tag_tid=14, entry_tid=15231], [object_number=90, tag_tid=14, entry_tid=15230], [object_number=89, tag_tid=14, entry_tid=15229]]], state=connected] 2024-04-06T09:40:56.840 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:40:56.842 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=88, tag_tid=14, entry_tid=15232' 2024-04-06T09:40:56.843 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=93, tag_tid=15, entry_tid=1' -a 'object_number=93, tag_tid=15, entry_tid=1' = 'object_number=88, tag_tid=14, entry_tid=15232' 2024-04-06T09:40:56.843 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=88, tag_tid=14, entry_tid=15232' '!=' 'object_number=89, tag_tid=14, entry_tid=15149' 2024-04-06T09:40:56.843 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:40:56.843 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=88, tag_tid=14, entry_tid=15232' = 'object_number=89, tag_tid=14, entry_tid=15149' 2024-04-06T09:40:56.843 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=88, tag_tid=14, entry_tid=15232' 2024-04-06T09:40:56.843 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=93, tag_tid=15, entry_tid=1' 2024-04-06T09:40:56.844 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:40:56.844 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:40:56.846 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=15 2024-04-06T09:40:56.846 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=88, tag_tid=14, entry_tid=15232' 2024-04-06T09:40:56.846 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:40:56.846 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:40:56.846 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=14 2024-04-06T09:40:56.847 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=93, tag_tid=15, entry_tid=1' 2024-04-06T09:40:56.847 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:40:56.847 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:40:56.849 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:40:56.849 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=88, tag_tid=14, entry_tid=15232' 2024-04-06T09:40:56.849 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:40:56.849 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:40:56.849 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=15232 2024-04-06T09:40:56.849 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 15 = 14 -a 1 -le 15232 2024-04-06T09:40:56.849 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:40:56.850 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:40:56.850 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:40:57.050 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:40:57.051 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:40:57.051 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:40:57.051 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:40:57.051 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:40:57.051 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:40:57.051 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:40:57.051 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:40:57.051 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:40:57.052 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:40:57.052 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:40:57.052 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:40:57.052 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:40:57.052 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:40:57.052 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:40:57.052 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:40:57.057 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:40:57.057 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:40:57.057 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:40:57.057 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:40:57.057 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:40:57.058 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:40:57.058 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:40:57.058 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:40:57.058 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:40:57.058 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:40:57.058 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:40:57.058 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:40:57.058 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:40:57.059 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:40:57.059 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:40:57.059 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:40:57.059 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:40:57.059 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:40:57.059 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:40:57.059 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:40:57.060 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:40:57.060 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:40:57.060 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:40:57.060 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:40:57.060 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:40:57.060 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:40:57.061 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:40:57.061 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:40:57.061 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:40:57.061 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:40:57.061 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:40:57.061 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:40:57.062 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:40:57.079 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:40:57.080 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:40:57.080 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:40:57.080 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:40:57.212 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:40:57.213 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:40:57.213 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:40:57.213 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:40:57.213 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:40:57.213 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:40:57.213 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:40:57.213 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:40:57.213 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:40:57.214 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:40:57.214 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:40:57.214 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:40:57.214 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:40:57.216 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:40:57.216 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:40:57.216 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:40:57.262 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 22 2024-04-06T09:40:57.262 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 23 2024-04-06T09:40:57.262 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:40:57.262 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=93, tag_tid=15, entry_tid=1], [object_number=92, tag_tid=15, entry_tid=0], [object_number=95, tag_tid=14, entry_tid=19039], [object_number=94, tag_tid=14, entry_tid=19038]]], state=connected] 2024-04-06T09:40:57.262 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=88, tag_tid=14, entry_tid=15312], [object_number=91, tag_tid=14, entry_tid=15311], [object_number=90, tag_tid=14, entry_tid=15310], [object_number=89, tag_tid=14, entry_tid=15309]]], state=connected] 2024-04-06T09:40:57.265 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:40:57.268 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=93, tag_tid=15, entry_tid=1' 2024-04-06T09:40:57.268 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:40:57.268 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:40:57.268 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:40:57.268 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:40:57.268 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:40:57.268 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:40:57.268 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:40:57.268 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:40:57.268 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:40:57.268 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:40:57.268 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:40:57.268 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:40:57.269 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:40:57.269 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:40:57.269 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:40:57.313 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 22 2024-04-06T09:40:57.313 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 23 2024-04-06T09:40:57.313 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:40:57.313 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=93, tag_tid=15, entry_tid=1], [object_number=92, tag_tid=15, entry_tid=0], [object_number=95, tag_tid=14, entry_tid=19039], [object_number=94, tag_tid=14, entry_tid=19038]]], state=connected] 2024-04-06T09:40:57.314 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=88, tag_tid=14, entry_tid=15312], [object_number=91, tag_tid=14, entry_tid=15311], [object_number=90, tag_tid=14, entry_tid=15310], [object_number=89, tag_tid=14, entry_tid=15309]]], state=connected] 2024-04-06T09:40:57.316 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:40:57.318 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=88, tag_tid=14, entry_tid=15312' 2024-04-06T09:40:57.319 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=93, tag_tid=15, entry_tid=1' -a 'object_number=93, tag_tid=15, entry_tid=1' = 'object_number=88, tag_tid=14, entry_tid=15312' 2024-04-06T09:40:57.319 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=88, tag_tid=14, entry_tid=15312' '!=' 'object_number=88, tag_tid=14, entry_tid=15232' 2024-04-06T09:40:57.319 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:40:57.319 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=88, tag_tid=14, entry_tid=15312' = 'object_number=88, tag_tid=14, entry_tid=15232' 2024-04-06T09:40:57.319 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=88, tag_tid=14, entry_tid=15312' 2024-04-06T09:40:57.319 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=93, tag_tid=15, entry_tid=1' 2024-04-06T09:40:57.319 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:40:57.319 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:40:57.319 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=15 2024-04-06T09:40:57.320 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=88, tag_tid=14, entry_tid=15312' 2024-04-06T09:40:57.320 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:40:57.320 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:40:57.321 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=14 2024-04-06T09:40:57.322 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=93, tag_tid=15, entry_tid=1' 2024-04-06T09:40:57.322 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:40:57.322 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:40:57.322 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:40:57.323 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=88, tag_tid=14, entry_tid=15312' 2024-04-06T09:40:57.323 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:40:57.323 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:40:57.324 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=15312 2024-04-06T09:40:57.324 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 15 = 14 -a 1 -le 15312 2024-04-06T09:40:57.325 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:40:57.325 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:40:57.325 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:40:57.526 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:40:57.526 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:40:57.526 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:40:57.526 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:40:57.526 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:40:57.526 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:40:57.526 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:40:57.526 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:40:57.527 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:40:57.527 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:40:57.527 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:40:57.527 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:40:57.527 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:40:57.527 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:40:57.527 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:40:57.527 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:40:57.528 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:40:57.532 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:40:57.532 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:40:57.532 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:40:57.532 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:40:57.532 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:40:57.532 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:40:57.532 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:40:57.532 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:40:57.532 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:40:57.532 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:40:57.532 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:40:57.532 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:40:57.532 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:40:57.532 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:40:57.533 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:40:57.533 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:40:57.533 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:40:57.533 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:40:57.533 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:40:57.533 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:40:57.533 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:40:57.533 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:40:57.533 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:40:57.533 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:40:57.533 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:40:57.533 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:40:57.533 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:40:57.533 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:40:57.533 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:40:57.533 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:40:57.534 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:40:57.534 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:40:57.555 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:40:57.555 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:40:57.555 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:40:57.555 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:40:57.686 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:40:57.687 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:40:57.687 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:40:57.687 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:40:57.687 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:40:57.687 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:40:57.687 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:40:57.687 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:40:57.687 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:40:57.687 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:40:57.688 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:40:57.688 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:40:57.688 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:40:57.689 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:40:57.690 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:40:57.690 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:40:57.741 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 22 2024-04-06T09:40:57.741 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 23 2024-04-06T09:40:57.741 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:40:57.741 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=93, tag_tid=15, entry_tid=1], [object_number=92, tag_tid=15, entry_tid=0], [object_number=95, tag_tid=14, entry_tid=19039], [object_number=94, tag_tid=14, entry_tid=19038]]], state=connected] 2024-04-06T09:40:57.741 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=90, tag_tid=14, entry_tid=15394], [object_number=89, tag_tid=14, entry_tid=15393], [object_number=88, tag_tid=14, entry_tid=15392], [object_number=91, tag_tid=14, entry_tid=15391]]], state=connected] 2024-04-06T09:40:57.745 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:40:57.746 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=93, tag_tid=15, entry_tid=1' 2024-04-06T09:40:57.747 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:40:57.748 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:40:57.748 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:40:57.748 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:40:57.748 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:40:57.748 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:40:57.748 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:40:57.748 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:40:57.748 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:40:57.748 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:40:57.748 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:40:57.748 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:40:57.749 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:40:57.749 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:40:57.749 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:40:57.796 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 22 2024-04-06T09:40:57.796 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 23 2024-04-06T09:40:57.796 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:40:57.796 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=93, tag_tid=15, entry_tid=1], [object_number=92, tag_tid=15, entry_tid=0], [object_number=95, tag_tid=14, entry_tid=19039], [object_number=94, tag_tid=14, entry_tid=19038]]], state=connected] 2024-04-06T09:40:57.796 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=90, tag_tid=14, entry_tid=15394], [object_number=89, tag_tid=14, entry_tid=15393], [object_number=88, tag_tid=14, entry_tid=15392], [object_number=91, tag_tid=14, entry_tid=15391]]], state=connected] 2024-04-06T09:40:57.800 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:40:57.802 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=90, tag_tid=14, entry_tid=15394' 2024-04-06T09:40:57.802 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=93, tag_tid=15, entry_tid=1' -a 'object_number=93, tag_tid=15, entry_tid=1' = 'object_number=90, tag_tid=14, entry_tid=15394' 2024-04-06T09:40:57.802 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=90, tag_tid=14, entry_tid=15394' '!=' 'object_number=88, tag_tid=14, entry_tid=15312' 2024-04-06T09:40:57.802 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:40:57.802 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=90, tag_tid=14, entry_tid=15394' = 'object_number=88, tag_tid=14, entry_tid=15312' 2024-04-06T09:40:57.802 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=90, tag_tid=14, entry_tid=15394' 2024-04-06T09:40:57.803 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=93, tag_tid=15, entry_tid=1' 2024-04-06T09:40:57.803 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:40:57.803 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:40:57.803 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=15 2024-04-06T09:40:57.804 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=90, tag_tid=14, entry_tid=15394' 2024-04-06T09:40:57.804 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:40:57.804 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:40:57.805 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=14 2024-04-06T09:40:57.805 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=93, tag_tid=15, entry_tid=1' 2024-04-06T09:40:57.805 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:40:57.806 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:40:57.806 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:40:57.807 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=90, tag_tid=14, entry_tid=15394' 2024-04-06T09:40:57.807 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:40:57.807 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:40:57.808 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=15394 2024-04-06T09:40:57.808 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 15 = 14 -a 1 -le 15394 2024-04-06T09:40:57.808 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:40:57.808 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:40:57.809 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:40:58.010 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:40:58.010 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:40:58.011 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:40:58.011 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:40:58.011 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:40:58.011 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:40:58.011 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:40:58.011 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:40:58.011 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:40:58.011 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:40:58.011 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:40:58.011 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:40:58.011 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:40:58.011 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:40:58.012 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:40:58.012 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:40:58.012 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:40:58.015 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:40:58.015 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:40:58.015 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:40:58.015 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:40:58.015 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:40:58.015 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:40:58.015 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:40:58.015 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:40:58.015 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:40:58.015 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:40:58.016 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:40:58.016 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:40:58.016 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:40:58.016 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:40:58.016 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:40:58.016 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:40:58.016 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:40:58.016 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:40:58.016 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:40:58.016 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:40:58.016 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:40:58.016 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:40:58.016 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:40:58.016 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:40:58.017 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:40:58.017 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:40:58.017 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:40:58.017 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:40:58.017 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:40:58.017 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:40:58.017 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:40:58.017 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:40:58.039 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:40:58.040 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:40:58.040 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:40:58.040 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:40:58.171 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:40:58.172 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:40:58.172 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:40:58.172 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:40:58.173 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:40:58.173 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:40:58.173 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:40:58.173 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:40:58.173 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:40:58.173 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:40:58.173 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:40:58.174 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:40:58.174 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:40:58.175 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:40:58.175 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:40:58.175 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:40:58.227 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 22 2024-04-06T09:40:58.227 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 23 2024-04-06T09:40:58.227 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:40:58.227 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=93, tag_tid=15, entry_tid=1], [object_number=92, tag_tid=15, entry_tid=0], [object_number=95, tag_tid=14, entry_tid=19039], [object_number=94, tag_tid=14, entry_tid=19038]]], state=connected] 2024-04-06T09:40:58.227 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=90, tag_tid=14, entry_tid=15466], [object_number=89, tag_tid=14, entry_tid=15465], [object_number=88, tag_tid=14, entry_tid=15464], [object_number=91, tag_tid=14, entry_tid=15463]]], state=connected] 2024-04-06T09:40:58.230 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:40:58.233 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=93, tag_tid=15, entry_tid=1' 2024-04-06T09:40:58.233 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:40:58.233 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:40:58.233 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:40:58.233 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:40:58.233 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:40:58.233 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:40:58.233 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:40:58.233 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:40:58.233 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:40:58.234 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:40:58.234 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:40:58.234 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:40:58.234 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:40:58.234 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:40:58.235 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:40:58.281 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 22 2024-04-06T09:40:58.281 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 23 2024-04-06T09:40:58.281 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:40:58.281 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=93, tag_tid=15, entry_tid=1], [object_number=92, tag_tid=15, entry_tid=0], [object_number=95, tag_tid=14, entry_tid=19039], [object_number=94, tag_tid=14, entry_tid=19038]]], state=connected] 2024-04-06T09:40:58.281 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=90, tag_tid=14, entry_tid=15466], [object_number=89, tag_tid=14, entry_tid=15465], [object_number=88, tag_tid=14, entry_tid=15464], [object_number=91, tag_tid=14, entry_tid=15463]]], state=connected] 2024-04-06T09:40:58.285 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:40:58.287 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=90, tag_tid=14, entry_tid=15466' 2024-04-06T09:40:58.287 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=93, tag_tid=15, entry_tid=1' -a 'object_number=93, tag_tid=15, entry_tid=1' = 'object_number=90, tag_tid=14, entry_tid=15466' 2024-04-06T09:40:58.287 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=90, tag_tid=14, entry_tid=15466' '!=' 'object_number=90, tag_tid=14, entry_tid=15394' 2024-04-06T09:40:58.288 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:40:58.288 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=90, tag_tid=14, entry_tid=15466' = 'object_number=90, tag_tid=14, entry_tid=15394' 2024-04-06T09:40:58.288 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=90, tag_tid=14, entry_tid=15466' 2024-04-06T09:40:58.288 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=93, tag_tid=15, entry_tid=1' 2024-04-06T09:40:58.288 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:40:58.288 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:40:58.289 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=15 2024-04-06T09:40:58.289 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=90, tag_tid=14, entry_tid=15466' 2024-04-06T09:40:58.289 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:40:58.289 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:40:58.290 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=14 2024-04-06T09:40:58.291 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=93, tag_tid=15, entry_tid=1' 2024-04-06T09:40:58.291 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:40:58.291 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:40:58.292 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:40:58.292 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=90, tag_tid=14, entry_tid=15466' 2024-04-06T09:40:58.292 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:40:58.292 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:40:58.294 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=15466 2024-04-06T09:40:58.294 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 15 = 14 -a 1 -le 15466 2024-04-06T09:40:58.294 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:40:58.294 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:40:58.294 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:40:58.496 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:40:58.496 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:40:58.496 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:40:58.496 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:40:58.496 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:40:58.496 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:40:58.496 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:40:58.496 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:40:58.496 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:40:58.497 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:40:58.497 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:40:58.497 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:40:58.497 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:40:58.497 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:40:58.497 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:40:58.497 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:40:58.497 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:40:58.500 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:40:58.500 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:40:58.500 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:40:58.500 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:40:58.500 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:40:58.500 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:40:58.500 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:40:58.500 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:40:58.500 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:40:58.500 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:40:58.500 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:40:58.501 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:40:58.501 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:40:58.501 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:40:58.501 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:40:58.501 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:40:58.501 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:40:58.501 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:40:58.501 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:40:58.501 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:40:58.501 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:40:58.501 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:40:58.501 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:40:58.501 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:40:58.501 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:40:58.501 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:40:58.502 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:40:58.502 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:40:58.502 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:40:58.502 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:40:58.502 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:40:58.502 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:40:58.525 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:40:58.525 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:40:58.525 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:40:58.525 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:40:58.654 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:40:58.655 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:40:58.655 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:40:58.655 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:40:58.655 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:40:58.655 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:40:58.655 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:40:58.655 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:40:58.655 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:40:58.655 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:40:58.656 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:40:58.656 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:40:58.656 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:40:58.658 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:40:58.659 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:40:58.659 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:40:58.710 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 22 2024-04-06T09:40:58.710 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 23 2024-04-06T09:40:58.710 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:40:58.710 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=93, tag_tid=15, entry_tid=1], [object_number=92, tag_tid=15, entry_tid=0], [object_number=95, tag_tid=14, entry_tid=19039], [object_number=94, tag_tid=14, entry_tid=19038]]], state=connected] 2024-04-06T09:40:58.710 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=90, tag_tid=14, entry_tid=15546], [object_number=89, tag_tid=14, entry_tid=15545], [object_number=88, tag_tid=14, entry_tid=15544], [object_number=91, tag_tid=14, entry_tid=15543]]], state=connected] 2024-04-06T09:40:58.713 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:40:58.715 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=93, tag_tid=15, entry_tid=1' 2024-04-06T09:40:58.716 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:40:58.716 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:40:58.716 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:40:58.716 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:40:58.716 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:40:58.716 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:40:58.716 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:40:58.716 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:40:58.716 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:40:58.716 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:40:58.716 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:40:58.716 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:40:58.717 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:40:58.717 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:40:58.717 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:40:58.763 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 22 2024-04-06T09:40:58.763 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 23 2024-04-06T09:40:58.763 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:40:58.763 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=93, tag_tid=15, entry_tid=1], [object_number=92, tag_tid=15, entry_tid=0], [object_number=95, tag_tid=14, entry_tid=19039], [object_number=94, tag_tid=14, entry_tid=19038]]], state=connected] 2024-04-06T09:40:58.763 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=90, tag_tid=14, entry_tid=15546], [object_number=89, tag_tid=14, entry_tid=15545], [object_number=88, tag_tid=14, entry_tid=15544], [object_number=91, tag_tid=14, entry_tid=15543]]], state=connected] 2024-04-06T09:40:58.766 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:40:58.768 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=90, tag_tid=14, entry_tid=15546' 2024-04-06T09:40:58.768 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=93, tag_tid=15, entry_tid=1' -a 'object_number=93, tag_tid=15, entry_tid=1' = 'object_number=90, tag_tid=14, entry_tid=15546' 2024-04-06T09:40:58.769 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=90, tag_tid=14, entry_tid=15546' '!=' 'object_number=90, tag_tid=14, entry_tid=15466' 2024-04-06T09:40:58.769 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:40:58.769 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=90, tag_tid=14, entry_tid=15546' = 'object_number=90, tag_tid=14, entry_tid=15466' 2024-04-06T09:40:58.769 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=90, tag_tid=14, entry_tid=15546' 2024-04-06T09:40:58.769 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=93, tag_tid=15, entry_tid=1' 2024-04-06T09:40:58.769 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:40:58.769 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:40:58.770 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=15 2024-04-06T09:40:58.770 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=90, tag_tid=14, entry_tid=15546' 2024-04-06T09:40:58.771 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:40:58.771 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:40:58.771 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=14 2024-04-06T09:40:58.772 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=93, tag_tid=15, entry_tid=1' 2024-04-06T09:40:58.772 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:40:58.772 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:40:58.773 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:40:58.774 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=90, tag_tid=14, entry_tid=15546' 2024-04-06T09:40:58.774 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:40:58.774 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:40:58.775 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=15546 2024-04-06T09:40:58.775 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 15 = 14 -a 1 -le 15546 2024-04-06T09:40:58.775 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:40:58.775 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:40:58.775 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:40:58.977 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:40:58.977 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:40:58.977 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:40:58.977 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:40:58.977 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:40:58.977 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:40:58.977 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:40:58.978 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:40:58.978 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:40:58.978 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:40:58.978 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:40:58.978 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:40:58.978 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:40:58.978 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:40:58.978 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:40:58.978 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:40:58.979 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:40:58.981 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:40:58.981 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:40:58.981 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:40:58.981 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:40:58.981 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:40:58.981 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:40:58.981 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:40:58.981 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:40:58.981 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:40:58.981 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:40:58.981 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:40:58.982 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:40:58.982 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:40:58.982 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:40:58.982 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:40:58.982 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:40:58.983 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:40:58.983 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:40:58.983 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:40:58.983 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:40:58.983 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:40:58.983 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:40:58.983 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:40:58.983 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:40:58.983 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:40:58.984 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:40:58.984 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:40:58.984 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:40:58.984 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:40:58.984 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:40:58.984 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:40:58.984 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:40:59.008 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:40:59.008 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:40:59.008 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:40:59.008 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:40:59.135 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:40:59.136 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:40:59.136 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:40:59.136 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:40:59.136 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:40:59.136 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:40:59.136 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:40:59.136 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:40:59.136 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:40:59.136 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:40:59.137 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:40:59.137 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:40:59.137 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:40:59.138 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:40:59.139 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:40:59.139 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:40:59.190 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 22 2024-04-06T09:40:59.190 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 23 2024-04-06T09:40:59.190 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:40:59.190 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=93, tag_tid=15, entry_tid=1], [object_number=92, tag_tid=15, entry_tid=0], [object_number=95, tag_tid=14, entry_tid=19039], [object_number=94, tag_tid=14, entry_tid=19038]]], state=connected] 2024-04-06T09:40:59.190 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=88, tag_tid=14, entry_tid=15632], [object_number=91, tag_tid=14, entry_tid=15631], [object_number=90, tag_tid=14, entry_tid=15630], [object_number=89, tag_tid=14, entry_tid=15629]]], state=connected] 2024-04-06T09:40:59.194 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:40:59.195 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=93, tag_tid=15, entry_tid=1' 2024-04-06T09:40:59.196 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:40:59.196 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:40:59.196 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:40:59.196 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:40:59.196 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:40:59.197 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:40:59.197 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:40:59.197 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:40:59.197 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:40:59.197 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:40:59.197 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:40:59.197 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:40:59.198 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:40:59.198 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:40:59.198 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:40:59.243 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 22 2024-04-06T09:40:59.243 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 23 2024-04-06T09:40:59.243 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:40:59.243 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=93, tag_tid=15, entry_tid=1], [object_number=92, tag_tid=15, entry_tid=0], [object_number=95, tag_tid=14, entry_tid=19039], [object_number=94, tag_tid=14, entry_tid=19038]]], state=connected] 2024-04-06T09:40:59.243 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=88, tag_tid=14, entry_tid=15632], [object_number=91, tag_tid=14, entry_tid=15631], [object_number=90, tag_tid=14, entry_tid=15630], [object_number=89, tag_tid=14, entry_tid=15629]]], state=connected] 2024-04-06T09:40:59.247 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:40:59.249 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=88, tag_tid=14, entry_tid=15632' 2024-04-06T09:40:59.249 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=93, tag_tid=15, entry_tid=1' -a 'object_number=93, tag_tid=15, entry_tid=1' = 'object_number=88, tag_tid=14, entry_tid=15632' 2024-04-06T09:40:59.249 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=88, tag_tid=14, entry_tid=15632' '!=' 'object_number=90, tag_tid=14, entry_tid=15546' 2024-04-06T09:40:59.249 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:40:59.249 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=88, tag_tid=14, entry_tid=15632' = 'object_number=90, tag_tid=14, entry_tid=15546' 2024-04-06T09:40:59.249 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=88, tag_tid=14, entry_tid=15632' 2024-04-06T09:40:59.249 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=93, tag_tid=15, entry_tid=1' 2024-04-06T09:40:59.249 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:40:59.249 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:40:59.250 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=15 2024-04-06T09:40:59.250 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=88, tag_tid=14, entry_tid=15632' 2024-04-06T09:40:59.251 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:40:59.251 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:40:59.251 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=14 2024-04-06T09:40:59.252 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=93, tag_tid=15, entry_tid=1' 2024-04-06T09:40:59.252 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:40:59.252 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:40:59.253 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:40:59.254 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=88, tag_tid=14, entry_tid=15632' 2024-04-06T09:40:59.254 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:40:59.254 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:40:59.255 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=15632 2024-04-06T09:40:59.255 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 15 = 14 -a 1 -le 15632 2024-04-06T09:40:59.255 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:40:59.255 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:40:59.255 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:40:59.457 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:40:59.457 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:40:59.458 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:40:59.458 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:40:59.458 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:40:59.458 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:40:59.458 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:40:59.458 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:40:59.458 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:40:59.458 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:40:59.458 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:40:59.458 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:40:59.458 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:40:59.458 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:40:59.458 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:40:59.459 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:40:59.459 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:40:59.462 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:40:59.462 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:40:59.462 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:40:59.462 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:40:59.462 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:40:59.462 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:40:59.462 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:40:59.462 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:40:59.462 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:40:59.462 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:40:59.463 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:40:59.463 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:40:59.463 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:40:59.463 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:40:59.463 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:40:59.463 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:40:59.463 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:40:59.463 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:40:59.463 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:40:59.463 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:40:59.463 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:40:59.463 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:40:59.463 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:40:59.463 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:40:59.463 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:40:59.464 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:40:59.464 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:40:59.464 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:40:59.464 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:40:59.464 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:40:59.464 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:40:59.464 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:40:59.487 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:40:59.487 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:40:59.487 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:40:59.487 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:40:59.621 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:40:59.622 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:40:59.622 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:40:59.623 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:40:59.623 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:40:59.623 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:40:59.623 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:40:59.623 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:40:59.623 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:40:59.623 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:40:59.623 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:40:59.624 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:40:59.624 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:40:59.626 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:40:59.626 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:40:59.627 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:40:59.677 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 22 2024-04-06T09:40:59.677 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 23 2024-04-06T09:40:59.677 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:40:59.677 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=93, tag_tid=15, entry_tid=1], [object_number=92, tag_tid=15, entry_tid=0], [object_number=95, tag_tid=14, entry_tid=19039], [object_number=94, tag_tid=14, entry_tid=19038]]], state=connected] 2024-04-06T09:40:59.677 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=89, tag_tid=14, entry_tid=15721], [object_number=88, tag_tid=14, entry_tid=15720], [object_number=91, tag_tid=14, entry_tid=15719], [object_number=90, tag_tid=14, entry_tid=15718]]], state=connected] 2024-04-06T09:40:59.681 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:40:59.682 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=93, tag_tid=15, entry_tid=1' 2024-04-06T09:40:59.683 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:40:59.683 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:40:59.683 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:40:59.683 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:40:59.683 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:40:59.683 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:40:59.683 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:40:59.683 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:40:59.684 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:40:59.684 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:40:59.684 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:40:59.684 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:40:59.684 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:40:59.685 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:40:59.685 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:40:59.731 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 22 2024-04-06T09:40:59.731 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 23 2024-04-06T09:40:59.731 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:40:59.731 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=93, tag_tid=15, entry_tid=1], [object_number=92, tag_tid=15, entry_tid=0], [object_number=95, tag_tid=14, entry_tid=19039], [object_number=94, tag_tid=14, entry_tid=19038]]], state=connected] 2024-04-06T09:40:59.731 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=89, tag_tid=14, entry_tid=15721], [object_number=88, tag_tid=14, entry_tid=15720], [object_number=91, tag_tid=14, entry_tid=15719], [object_number=90, tag_tid=14, entry_tid=15718]]], state=connected] 2024-04-06T09:40:59.734 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:40:59.736 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=89, tag_tid=14, entry_tid=15721' 2024-04-06T09:40:59.736 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=93, tag_tid=15, entry_tid=1' -a 'object_number=93, tag_tid=15, entry_tid=1' = 'object_number=89, tag_tid=14, entry_tid=15721' 2024-04-06T09:40:59.736 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=89, tag_tid=14, entry_tid=15721' '!=' 'object_number=88, tag_tid=14, entry_tid=15632' 2024-04-06T09:40:59.736 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:40:59.736 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=89, tag_tid=14, entry_tid=15721' = 'object_number=88, tag_tid=14, entry_tid=15632' 2024-04-06T09:40:59.736 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=89, tag_tid=14, entry_tid=15721' 2024-04-06T09:40:59.737 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=93, tag_tid=15, entry_tid=1' 2024-04-06T09:40:59.737 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:40:59.737 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:40:59.738 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=15 2024-04-06T09:40:59.738 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=89, tag_tid=14, entry_tid=15721' 2024-04-06T09:40:59.738 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:40:59.738 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:40:59.739 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=14 2024-04-06T09:40:59.740 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=93, tag_tid=15, entry_tid=1' 2024-04-06T09:40:59.740 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:40:59.740 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:40:59.741 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:40:59.742 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=89, tag_tid=14, entry_tid=15721' 2024-04-06T09:40:59.742 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:40:59.742 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:40:59.743 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=15721 2024-04-06T09:40:59.743 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 15 = 14 -a 1 -le 15721 2024-04-06T09:40:59.743 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:40:59.743 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:40:59.743 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:40:59.945 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:40:59.945 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:40:59.945 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:40:59.945 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:40:59.945 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:40:59.945 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:40:59.945 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:40:59.945 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:40:59.946 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:40:59.946 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:40:59.946 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:40:59.946 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:40:59.946 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:40:59.946 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:40:59.946 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:40:59.946 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:40:59.947 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:40:59.949 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:40:59.949 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:40:59.949 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:40:59.949 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:40:59.949 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:40:59.949 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:40:59.949 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:40:59.950 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:40:59.950 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:40:59.950 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:40:59.950 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:40:59.950 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:40:59.950 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:40:59.950 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:40:59.950 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:40:59.950 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:40:59.950 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:40:59.950 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:40:59.951 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:40:59.951 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:40:59.951 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:40:59.951 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:40:59.951 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:40:59.951 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:40:59.951 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:40:59.951 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:40:59.951 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:40:59.951 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:40:59.951 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:40:59.951 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:40:59.951 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:40:59.952 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:40:59.975 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:40:59.975 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:40:59.975 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:40:59.976 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:41:00.103 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:41:00.104 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:41:00.104 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:41:00.104 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:41:00.105 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:41:00.105 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:41:00.105 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:41:00.105 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:41:00.105 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:41:00.105 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:41:00.105 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:41:00.105 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:41:00.106 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:41:00.108 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:41:00.108 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:41:00.108 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:41:00.160 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 22 2024-04-06T09:41:00.160 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 23 2024-04-06T09:41:00.160 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:41:00.160 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=93, tag_tid=15, entry_tid=1], [object_number=92, tag_tid=15, entry_tid=0], [object_number=95, tag_tid=14, entry_tid=19039], [object_number=94, tag_tid=14, entry_tid=19038]]], state=connected] 2024-04-06T09:41:00.161 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=89, tag_tid=14, entry_tid=15805], [object_number=88, tag_tid=14, entry_tid=15804], [object_number=91, tag_tid=14, entry_tid=15803], [object_number=90, tag_tid=14, entry_tid=15802]]], state=connected] 2024-04-06T09:41:00.164 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:41:00.165 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=93, tag_tid=15, entry_tid=1' 2024-04-06T09:41:00.167 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:41:00.167 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:41:00.167 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:41:00.167 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:41:00.167 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:41:00.167 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:41:00.167 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:41:00.167 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:41:00.167 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:41:00.167 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:41:00.168 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:41:00.168 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:41:00.168 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:41:00.168 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:41:00.169 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:41:00.215 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 22 2024-04-06T09:41:00.215 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 23 2024-04-06T09:41:00.215 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:41:00.215 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=93, tag_tid=15, entry_tid=1], [object_number=92, tag_tid=15, entry_tid=0], [object_number=95, tag_tid=14, entry_tid=19039], [object_number=94, tag_tid=14, entry_tid=19038]]], state=connected] 2024-04-06T09:41:00.215 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=89, tag_tid=14, entry_tid=15805], [object_number=88, tag_tid=14, entry_tid=15804], [object_number=91, tag_tid=14, entry_tid=15803], [object_number=90, tag_tid=14, entry_tid=15802]]], state=connected] 2024-04-06T09:41:00.219 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:41:00.221 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=89, tag_tid=14, entry_tid=15805' 2024-04-06T09:41:00.221 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=93, tag_tid=15, entry_tid=1' -a 'object_number=93, tag_tid=15, entry_tid=1' = 'object_number=89, tag_tid=14, entry_tid=15805' 2024-04-06T09:41:00.221 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=89, tag_tid=14, entry_tid=15805' '!=' 'object_number=89, tag_tid=14, entry_tid=15721' 2024-04-06T09:41:00.221 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:41:00.221 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=89, tag_tid=14, entry_tid=15805' = 'object_number=89, tag_tid=14, entry_tid=15721' 2024-04-06T09:41:00.221 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=89, tag_tid=14, entry_tid=15805' 2024-04-06T09:41:00.222 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=93, tag_tid=15, entry_tid=1' 2024-04-06T09:41:00.222 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:41:00.222 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:41:00.222 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=15 2024-04-06T09:41:00.223 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=89, tag_tid=14, entry_tid=15805' 2024-04-06T09:41:00.223 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:41:00.223 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:41:00.224 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=14 2024-04-06T09:41:00.224 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=93, tag_tid=15, entry_tid=1' 2024-04-06T09:41:00.225 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:41:00.225 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:41:00.225 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:41:00.226 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=89, tag_tid=14, entry_tid=15805' 2024-04-06T09:41:00.226 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:41:00.226 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:41:00.227 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=15805 2024-04-06T09:41:00.228 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 15 = 14 -a 1 -le 15805 2024-04-06T09:41:00.228 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:41:00.228 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:41:00.228 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:41:00.430 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:41:00.430 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:41:00.430 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:41:00.430 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:41:00.430 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:41:00.430 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:41:00.430 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:41:00.430 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:41:00.430 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:41:00.430 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:41:00.430 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:41:00.431 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:41:00.431 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:41:00.431 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:41:00.431 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:41:00.431 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:41:00.431 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:41:00.433 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:41:00.433 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:41:00.433 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:41:00.433 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:41:00.433 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:41:00.433 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:41:00.433 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:41:00.433 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:41:00.433 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:41:00.433 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:41:00.434 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:41:00.434 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:41:00.434 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:41:00.434 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:41:00.434 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:41:00.434 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:41:00.435 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:41:00.435 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:41:00.435 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:41:00.435 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:41:00.435 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:41:00.435 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:41:00.435 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:41:00.435 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:41:00.436 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:41:00.436 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:41:00.436 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:41:00.436 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:41:00.436 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:41:00.436 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:41:00.436 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:41:00.436 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:41:00.459 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:41:00.459 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:41:00.459 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:41:00.460 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:41:00.588 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:41:00.588 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:41:00.588 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:41:00.589 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:41:00.589 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:41:00.589 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:41:00.589 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:41:00.589 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:41:00.589 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:41:00.589 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:41:00.590 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:41:00.590 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:41:00.590 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:41:00.591 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:41:00.592 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:41:00.592 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:41:00.638 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 22 2024-04-06T09:41:00.638 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 23 2024-04-06T09:41:00.638 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:41:00.638 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=93, tag_tid=15, entry_tid=1], [object_number=92, tag_tid=15, entry_tid=0], [object_number=95, tag_tid=14, entry_tid=19039], [object_number=94, tag_tid=14, entry_tid=19038]]], state=connected] 2024-04-06T09:41:00.638 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=90, tag_tid=14, entry_tid=15882], [object_number=89, tag_tid=14, entry_tid=15881], [object_number=88, tag_tid=14, entry_tid=15880], [object_number=91, tag_tid=14, entry_tid=15879]]], state=connected] 2024-04-06T09:41:00.641 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:41:00.642 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=93, tag_tid=15, entry_tid=1' 2024-04-06T09:41:00.644 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:41:00.644 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:41:00.644 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:41:00.644 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:41:00.644 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:41:00.644 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:41:00.645 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:41:00.645 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:41:00.645 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:41:00.645 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:41:00.645 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:41:00.645 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:41:00.646 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:41:00.647 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:41:00.647 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:41:00.692 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 22 2024-04-06T09:41:00.692 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 23 2024-04-06T09:41:00.692 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:41:00.692 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=93, tag_tid=15, entry_tid=1], [object_number=92, tag_tid=15, entry_tid=0], [object_number=95, tag_tid=14, entry_tid=19039], [object_number=94, tag_tid=14, entry_tid=19038]]], state=connected] 2024-04-06T09:41:00.692 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=90, tag_tid=14, entry_tid=15882], [object_number=89, tag_tid=14, entry_tid=15881], [object_number=88, tag_tid=14, entry_tid=15880], [object_number=91, tag_tid=14, entry_tid=15879]]], state=connected] 2024-04-06T09:41:00.696 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:41:00.698 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=90, tag_tid=14, entry_tid=15882' 2024-04-06T09:41:00.698 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=93, tag_tid=15, entry_tid=1' -a 'object_number=93, tag_tid=15, entry_tid=1' = 'object_number=90, tag_tid=14, entry_tid=15882' 2024-04-06T09:41:00.698 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=90, tag_tid=14, entry_tid=15882' '!=' 'object_number=89, tag_tid=14, entry_tid=15805' 2024-04-06T09:41:00.698 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:41:00.698 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=90, tag_tid=14, entry_tid=15882' = 'object_number=89, tag_tid=14, entry_tid=15805' 2024-04-06T09:41:00.698 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=90, tag_tid=14, entry_tid=15882' 2024-04-06T09:41:00.698 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=93, tag_tid=15, entry_tid=1' 2024-04-06T09:41:00.698 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:41:00.698 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:41:00.699 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=15 2024-04-06T09:41:00.699 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=90, tag_tid=14, entry_tid=15882' 2024-04-06T09:41:00.700 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:41:00.700 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:41:00.700 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=14 2024-04-06T09:41:00.701 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=93, tag_tid=15, entry_tid=1' 2024-04-06T09:41:00.701 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:41:00.701 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:41:00.702 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:41:00.703 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=90, tag_tid=14, entry_tid=15882' 2024-04-06T09:41:00.703 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:41:00.703 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:41:00.704 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=15882 2024-04-06T09:41:00.704 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 15 = 14 -a 1 -le 15882 2024-04-06T09:41:00.704 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:41:00.705 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:41:00.705 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:41:00.906 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:41:00.906 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:41:00.906 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:41:00.906 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:41:00.906 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:41:00.906 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:41:00.906 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:41:00.906 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:41:00.907 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:41:00.907 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:41:00.907 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:41:00.907 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:41:00.907 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:41:00.907 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:41:00.907 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:41:00.907 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:41:00.908 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:41:00.912 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:41:00.912 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:41:00.912 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:41:00.912 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:41:00.912 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:41:00.912 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:41:00.912 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:41:00.912 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:41:00.912 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:41:00.912 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:41:00.912 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:41:00.912 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:41:00.912 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:41:00.912 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:41:00.912 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:41:00.913 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:41:00.913 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:41:00.913 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:41:00.913 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:41:00.913 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:41:00.913 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:41:00.913 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:41:00.913 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:41:00.913 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:41:00.913 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:41:00.913 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:41:00.913 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:41:00.913 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:41:00.913 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:41:00.913 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:41:00.913 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:41:00.914 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:41:00.938 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:41:00.939 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:41:00.939 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:41:00.939 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:41:01.067 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:41:01.068 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:41:01.068 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:41:01.068 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:41:01.068 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:41:01.068 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:41:01.068 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:41:01.068 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:41:01.068 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:41:01.068 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:41:01.069 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:41:01.069 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:41:01.069 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:41:01.071 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:41:01.071 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:41:01.071 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:41:01.122 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 22 2024-04-06T09:41:01.122 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 23 2024-04-06T09:41:01.122 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:41:01.122 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=93, tag_tid=15, entry_tid=1], [object_number=92, tag_tid=15, entry_tid=0], [object_number=95, tag_tid=14, entry_tid=19039], [object_number=94, tag_tid=14, entry_tid=19038]]], state=connected] 2024-04-06T09:41:01.123 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=90, tag_tid=14, entry_tid=15966], [object_number=89, tag_tid=14, entry_tid=15965], [object_number=88, tag_tid=14, entry_tid=15964], [object_number=91, tag_tid=14, entry_tid=15963]]], state=connected] 2024-04-06T09:41:01.126 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:41:01.127 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=93, tag_tid=15, entry_tid=1' 2024-04-06T09:41:01.129 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:41:01.129 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:41:01.129 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:41:01.129 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:41:01.129 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:41:01.129 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:41:01.129 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:41:01.129 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:41:01.129 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:41:01.129 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:41:01.129 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:41:01.130 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:41:01.130 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:41:01.130 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:41:01.131 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:41:01.176 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 22 2024-04-06T09:41:01.176 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 23 2024-04-06T09:41:01.177 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:41:01.177 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=93, tag_tid=15, entry_tid=1], [object_number=92, tag_tid=15, entry_tid=0], [object_number=95, tag_tid=14, entry_tid=19039], [object_number=94, tag_tid=14, entry_tid=19038]]], state=connected] 2024-04-06T09:41:01.177 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=90, tag_tid=14, entry_tid=15966], [object_number=89, tag_tid=14, entry_tid=15965], [object_number=88, tag_tid=14, entry_tid=15964], [object_number=91, tag_tid=14, entry_tid=15963]]], state=connected] 2024-04-06T09:41:01.180 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:41:01.182 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=90, tag_tid=14, entry_tid=15966' 2024-04-06T09:41:01.182 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=93, tag_tid=15, entry_tid=1' -a 'object_number=93, tag_tid=15, entry_tid=1' = 'object_number=90, tag_tid=14, entry_tid=15966' 2024-04-06T09:41:01.182 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=90, tag_tid=14, entry_tid=15966' '!=' 'object_number=90, tag_tid=14, entry_tid=15882' 2024-04-06T09:41:01.182 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:41:01.182 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=90, tag_tid=14, entry_tid=15966' = 'object_number=90, tag_tid=14, entry_tid=15882' 2024-04-06T09:41:01.182 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=90, tag_tid=14, entry_tid=15966' 2024-04-06T09:41:01.182 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=93, tag_tid=15, entry_tid=1' 2024-04-06T09:41:01.183 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:41:01.183 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:41:01.183 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=15 2024-04-06T09:41:01.184 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=90, tag_tid=14, entry_tid=15966' 2024-04-06T09:41:01.184 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:41:01.184 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:41:01.185 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=14 2024-04-06T09:41:01.186 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=93, tag_tid=15, entry_tid=1' 2024-04-06T09:41:01.186 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:41:01.186 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:41:01.187 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:41:01.187 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=90, tag_tid=14, entry_tid=15966' 2024-04-06T09:41:01.188 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:41:01.188 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:41:01.189 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=15966 2024-04-06T09:41:01.189 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 15 = 14 -a 1 -le 15966 2024-04-06T09:41:01.189 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:41:01.189 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:41:01.189 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:41:01.391 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:41:01.391 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:41:01.391 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:41:01.391 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:41:01.391 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:41:01.391 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:41:01.391 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:41:01.391 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:41:01.391 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:41:01.392 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:41:01.392 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:41:01.392 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:41:01.392 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:41:01.392 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:41:01.392 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:41:01.392 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:41:01.392 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:41:01.395 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:41:01.395 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:41:01.395 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:41:01.395 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:41:01.395 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:41:01.395 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:41:01.395 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:41:01.395 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:41:01.396 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:41:01.396 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:41:01.396 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:41:01.396 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:41:01.396 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:41:01.396 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:41:01.396 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:41:01.396 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:41:01.397 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:41:01.397 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:41:01.397 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:41:01.397 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:41:01.397 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:41:01.397 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:41:01.397 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:41:01.397 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:41:01.397 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:41:01.397 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:41:01.398 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:41:01.398 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:41:01.398 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:41:01.398 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:41:01.398 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:41:01.398 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:41:01.422 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:41:01.423 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:41:01.423 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:41:01.423 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:41:01.556 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:41:01.557 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:41:01.557 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:41:01.557 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:41:01.557 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:41:01.557 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:41:01.557 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:41:01.557 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:41:01.557 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:41:01.557 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:41:01.558 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:41:01.558 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:41:01.558 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:41:01.560 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:41:01.560 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:41:01.560 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:41:01.606 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 22 2024-04-06T09:41:01.606 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 23 2024-04-06T09:41:01.606 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:41:01.607 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=93, tag_tid=15, entry_tid=1], [object_number=92, tag_tid=15, entry_tid=0], [object_number=95, tag_tid=14, entry_tid=19039], [object_number=94, tag_tid=14, entry_tid=19038]]], state=connected] 2024-04-06T09:41:01.607 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=89, tag_tid=14, entry_tid=16049], [object_number=88, tag_tid=14, entry_tid=16048], [object_number=91, tag_tid=14, entry_tid=16047], [object_number=90, tag_tid=14, entry_tid=16046]]], state=connected] 2024-04-06T09:41:01.610 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:41:01.611 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=93, tag_tid=15, entry_tid=1' 2024-04-06T09:41:01.612 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:41:01.612 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:41:01.612 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:41:01.612 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:41:01.612 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:41:01.613 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:41:01.613 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:41:01.613 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:41:01.613 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:41:01.613 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:41:01.613 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:41:01.613 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:41:01.614 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:41:01.614 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:41:01.614 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:41:01.660 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 22 2024-04-06T09:41:01.660 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 23 2024-04-06T09:41:01.660 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:41:01.660 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=93, tag_tid=15, entry_tid=1], [object_number=92, tag_tid=15, entry_tid=0], [object_number=95, tag_tid=14, entry_tid=19039], [object_number=94, tag_tid=14, entry_tid=19038]]], state=connected] 2024-04-06T09:41:01.660 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=89, tag_tid=14, entry_tid=16049], [object_number=88, tag_tid=14, entry_tid=16048], [object_number=91, tag_tid=14, entry_tid=16047], [object_number=90, tag_tid=14, entry_tid=16046]]], state=connected] 2024-04-06T09:41:01.663 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:41:01.666 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=89, tag_tid=14, entry_tid=16049' 2024-04-06T09:41:01.666 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=93, tag_tid=15, entry_tid=1' -a 'object_number=93, tag_tid=15, entry_tid=1' = 'object_number=89, tag_tid=14, entry_tid=16049' 2024-04-06T09:41:01.666 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=89, tag_tid=14, entry_tid=16049' '!=' 'object_number=90, tag_tid=14, entry_tid=15966' 2024-04-06T09:41:01.666 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:41:01.666 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=89, tag_tid=14, entry_tid=16049' = 'object_number=90, tag_tid=14, entry_tid=15966' 2024-04-06T09:41:01.666 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=89, tag_tid=14, entry_tid=16049' 2024-04-06T09:41:01.666 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=93, tag_tid=15, entry_tid=1' 2024-04-06T09:41:01.667 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:41:01.667 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:41:01.667 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=15 2024-04-06T09:41:01.668 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=89, tag_tid=14, entry_tid=16049' 2024-04-06T09:41:01.668 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:41:01.668 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:41:01.668 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=14 2024-04-06T09:41:01.669 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=93, tag_tid=15, entry_tid=1' 2024-04-06T09:41:01.669 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:41:01.669 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:41:01.670 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:41:01.671 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=89, tag_tid=14, entry_tid=16049' 2024-04-06T09:41:01.671 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:41:01.671 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:41:01.672 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=16049 2024-04-06T09:41:01.672 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 15 = 14 -a 1 -le 16049 2024-04-06T09:41:01.672 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:41:01.672 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:41:01.672 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:41:01.874 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:41:01.874 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:41:01.874 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:41:01.874 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:41:01.875 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:41:01.875 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:41:01.875 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:41:01.875 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:41:01.875 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:41:01.875 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:41:01.875 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:41:01.875 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:41:01.875 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:41:01.875 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:41:01.875 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:41:01.876 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:41:01.876 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:41:01.879 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:41:01.880 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:41:01.880 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:41:01.880 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:41:01.880 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:41:01.880 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:41:01.880 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:41:01.880 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:41:01.880 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:41:01.880 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:41:01.880 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:41:01.880 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:41:01.880 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:41:01.881 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:41:01.881 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:41:01.881 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:41:01.881 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:41:01.881 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:41:01.881 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:41:01.881 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:41:01.881 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:41:01.881 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:41:01.881 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:41:01.881 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:41:01.881 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:41:01.882 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:41:01.882 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:41:01.882 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:41:01.882 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:41:01.882 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:41:01.882 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:41:01.882 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:41:01.906 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:41:01.906 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:41:01.906 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:41:01.906 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:41:02.041 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:41:02.042 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:41:02.043 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:41:02.043 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:41:02.043 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:41:02.043 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:41:02.043 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:41:02.043 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:41:02.043 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:41:02.043 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:41:02.044 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:41:02.044 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:41:02.044 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:41:02.046 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:41:02.047 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:41:02.047 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:41:02.093 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 22 2024-04-06T09:41:02.093 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 23 2024-04-06T09:41:02.093 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:41:02.093 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=93, tag_tid=15, entry_tid=1], [object_number=92, tag_tid=15, entry_tid=0], [object_number=95, tag_tid=14, entry_tid=19039], [object_number=94, tag_tid=14, entry_tid=19038]]], state=connected] 2024-04-06T09:41:02.093 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=90, tag_tid=14, entry_tid=16134], [object_number=89, tag_tid=14, entry_tid=16133], [object_number=88, tag_tid=14, entry_tid=16132], [object_number=91, tag_tid=14, entry_tid=16131]]], state=connected] 2024-04-06T09:41:02.096 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:41:02.099 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=93, tag_tid=15, entry_tid=1' 2024-04-06T09:41:02.099 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:41:02.099 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:41:02.099 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:41:02.099 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:41:02.099 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:41:02.099 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:41:02.099 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:41:02.100 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:41:02.100 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:41:02.100 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:41:02.100 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:41:02.100 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:41:02.101 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:41:02.101 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:41:02.101 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:41:02.147 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 22 2024-04-06T09:41:02.147 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 23 2024-04-06T09:41:02.147 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:41:02.147 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=93, tag_tid=15, entry_tid=1], [object_number=92, tag_tid=15, entry_tid=0], [object_number=95, tag_tid=14, entry_tid=19039], [object_number=94, tag_tid=14, entry_tid=19038]]], state=connected] 2024-04-06T09:41:02.147 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=90, tag_tid=14, entry_tid=16134], [object_number=89, tag_tid=14, entry_tid=16133], [object_number=88, tag_tid=14, entry_tid=16132], [object_number=91, tag_tid=14, entry_tid=16131]]], state=connected] 2024-04-06T09:41:02.150 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:41:02.152 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=90, tag_tid=14, entry_tid=16134' 2024-04-06T09:41:02.152 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=93, tag_tid=15, entry_tid=1' -a 'object_number=93, tag_tid=15, entry_tid=1' = 'object_number=90, tag_tid=14, entry_tid=16134' 2024-04-06T09:41:02.153 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=90, tag_tid=14, entry_tid=16134' '!=' 'object_number=89, tag_tid=14, entry_tid=16049' 2024-04-06T09:41:02.153 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:41:02.153 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=90, tag_tid=14, entry_tid=16134' = 'object_number=89, tag_tid=14, entry_tid=16049' 2024-04-06T09:41:02.153 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=90, tag_tid=14, entry_tid=16134' 2024-04-06T09:41:02.153 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=93, tag_tid=15, entry_tid=1' 2024-04-06T09:41:02.153 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:41:02.153 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:41:02.153 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=15 2024-04-06T09:41:02.154 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=90, tag_tid=14, entry_tid=16134' 2024-04-06T09:41:02.154 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:41:02.154 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:41:02.155 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=14 2024-04-06T09:41:02.156 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=93, tag_tid=15, entry_tid=1' 2024-04-06T09:41:02.156 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:41:02.156 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:41:02.157 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:41:02.158 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=90, tag_tid=14, entry_tid=16134' 2024-04-06T09:41:02.158 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:41:02.158 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:41:02.159 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=16134 2024-04-06T09:41:02.159 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 15 = 14 -a 1 -le 16134 2024-04-06T09:41:02.159 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:41:02.159 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:41:02.159 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:41:02.360 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:41:02.361 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:41:02.361 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:41:02.361 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:41:02.361 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:41:02.361 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:41:02.361 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:41:02.361 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:41:02.361 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:41:02.361 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:41:02.361 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:41:02.361 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:41:02.361 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:41:02.362 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:41:02.362 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:41:02.362 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:41:02.362 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:41:02.366 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:41:02.366 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:41:02.366 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:41:02.367 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:41:02.367 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:41:02.367 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:41:02.367 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:41:02.367 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:41:02.367 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:41:02.367 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:41:02.367 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:41:02.367 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:41:02.367 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:41:02.367 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:41:02.367 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:41:02.368 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:41:02.368 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:41:02.368 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:41:02.368 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:41:02.368 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:41:02.368 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:41:02.368 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:41:02.368 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:41:02.368 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:41:02.368 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:41:02.368 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:41:02.368 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:41:02.368 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:41:02.369 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:41:02.369 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:41:02.369 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:41:02.369 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:41:02.391 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:41:02.391 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:41:02.391 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:41:02.391 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:41:02.522 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:41:02.523 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:41:02.523 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:41:02.523 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:41:02.523 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:41:02.523 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:41:02.524 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:41:02.524 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:41:02.524 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:41:02.524 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:41:02.524 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:41:02.524 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:41:02.524 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:41:02.526 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:41:02.527 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:41:02.527 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:41:02.574 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 22 2024-04-06T09:41:02.574 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 23 2024-04-06T09:41:02.575 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:41:02.575 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=93, tag_tid=15, entry_tid=1], [object_number=92, tag_tid=15, entry_tid=0], [object_number=95, tag_tid=14, entry_tid=19039], [object_number=94, tag_tid=14, entry_tid=19038]]], state=connected] 2024-04-06T09:41:02.575 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=90, tag_tid=14, entry_tid=16238], [object_number=89, tag_tid=14, entry_tid=16237], [object_number=88, tag_tid=14, entry_tid=16236], [object_number=91, tag_tid=14, entry_tid=16235]]], state=connected] 2024-04-06T09:41:02.578 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:41:02.579 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=93, tag_tid=15, entry_tid=1' 2024-04-06T09:41:02.580 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:41:02.580 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:41:02.581 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:41:02.581 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:41:02.581 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:41:02.581 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:41:02.581 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:41:02.581 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:41:02.581 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:41:02.581 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:41:02.581 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:41:02.581 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:41:02.582 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:41:02.582 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:41:02.582 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:41:02.628 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 22 2024-04-06T09:41:02.628 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 23 2024-04-06T09:41:02.628 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:41:02.628 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=93, tag_tid=15, entry_tid=1], [object_number=92, tag_tid=15, entry_tid=0], [object_number=95, tag_tid=14, entry_tid=19039], [object_number=94, tag_tid=14, entry_tid=19038]]], state=connected] 2024-04-06T09:41:02.628 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=90, tag_tid=14, entry_tid=16238], [object_number=89, tag_tid=14, entry_tid=16237], [object_number=88, tag_tid=14, entry_tid=16236], [object_number=91, tag_tid=14, entry_tid=16235]]], state=connected] 2024-04-06T09:41:02.631 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:41:02.633 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=90, tag_tid=14, entry_tid=16238' 2024-04-06T09:41:02.633 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=93, tag_tid=15, entry_tid=1' -a 'object_number=93, tag_tid=15, entry_tid=1' = 'object_number=90, tag_tid=14, entry_tid=16238' 2024-04-06T09:41:02.633 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=90, tag_tid=14, entry_tid=16238' '!=' 'object_number=90, tag_tid=14, entry_tid=16134' 2024-04-06T09:41:02.633 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:41:02.633 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=90, tag_tid=14, entry_tid=16238' = 'object_number=90, tag_tid=14, entry_tid=16134' 2024-04-06T09:41:02.633 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=90, tag_tid=14, entry_tid=16238' 2024-04-06T09:41:02.634 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=93, tag_tid=15, entry_tid=1' 2024-04-06T09:41:02.634 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:41:02.634 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:41:02.634 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=15 2024-04-06T09:41:02.635 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=90, tag_tid=14, entry_tid=16238' 2024-04-06T09:41:02.635 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:41:02.635 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:41:02.636 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=14 2024-04-06T09:41:02.637 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=93, tag_tid=15, entry_tid=1' 2024-04-06T09:41:02.637 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:41:02.637 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:41:02.638 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:41:02.638 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=90, tag_tid=14, entry_tid=16238' 2024-04-06T09:41:02.639 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:41:02.639 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:41:02.640 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=16238 2024-04-06T09:41:02.640 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 15 = 14 -a 1 -le 16238 2024-04-06T09:41:02.640 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:41:02.640 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:41:02.640 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:41:02.842 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:41:02.843 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:41:02.843 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:41:02.843 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:41:02.843 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:41:02.843 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:41:02.843 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:41:02.843 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:41:02.843 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:41:02.843 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:41:02.843 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:41:02.843 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:41:02.843 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:41:02.843 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:41:02.844 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:41:02.844 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:41:02.845 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:41:02.845 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:41:02.845 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:41:02.845 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:41:02.845 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:41:02.845 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:41:02.846 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:41:02.846 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:41:02.846 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:41:02.846 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:41:02.846 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:41:02.846 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:41:02.846 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:41:02.846 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:41:02.846 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:41:02.846 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:41:02.846 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:41:02.848 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:41:02.848 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:41:02.848 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:41:02.848 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:41:02.848 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:41:02.848 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:41:02.848 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:41:02.848 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:41:02.848 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:41:02.848 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:41:02.848 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:41:02.848 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:41:02.848 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:41:02.848 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:41:02.849 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:41:02.849 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:41:02.872 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:41:02.872 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:41:02.872 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:41:02.872 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:41:03.000 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:41:03.001 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:41:03.001 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:41:03.001 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:41:03.001 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:41:03.001 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:41:03.002 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:41:03.002 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:41:03.002 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:41:03.002 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:41:03.002 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:41:03.002 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:41:03.002 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:41:03.004 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:41:03.004 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:41:03.005 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:41:03.056 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 22 2024-04-06T09:41:03.056 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 23 2024-04-06T09:41:03.056 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:41:03.057 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=93, tag_tid=15, entry_tid=1], [object_number=92, tag_tid=15, entry_tid=0], [object_number=95, tag_tid=14, entry_tid=19039], [object_number=94, tag_tid=14, entry_tid=19038]]], state=connected] 2024-04-06T09:41:03.057 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=88, tag_tid=14, entry_tid=16344], [object_number=91, tag_tid=14, entry_tid=16343], [object_number=90, tag_tid=14, entry_tid=16342], [object_number=89, tag_tid=14, entry_tid=16341]]], state=connected] 2024-04-06T09:41:03.060 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:41:03.061 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=93, tag_tid=15, entry_tid=1' 2024-04-06T09:41:03.062 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:41:03.062 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:41:03.062 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:41:03.062 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:41:03.063 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:41:03.063 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:41:03.063 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:41:03.063 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:41:03.063 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:41:03.063 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:41:03.063 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:41:03.063 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:41:03.064 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:41:03.064 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:41:03.064 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:41:03.110 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 22 2024-04-06T09:41:03.110 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 23 2024-04-06T09:41:03.110 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:41:03.110 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=93, tag_tid=15, entry_tid=1], [object_number=92, tag_tid=15, entry_tid=0], [object_number=95, tag_tid=14, entry_tid=19039], [object_number=94, tag_tid=14, entry_tid=19038]]], state=connected] 2024-04-06T09:41:03.110 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=88, tag_tid=14, entry_tid=16344], [object_number=91, tag_tid=14, entry_tid=16343], [object_number=90, tag_tid=14, entry_tid=16342], [object_number=89, tag_tid=14, entry_tid=16341]]], state=connected] 2024-04-06T09:41:03.114 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:41:03.116 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=88, tag_tid=14, entry_tid=16344' 2024-04-06T09:41:03.116 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=93, tag_tid=15, entry_tid=1' -a 'object_number=93, tag_tid=15, entry_tid=1' = 'object_number=88, tag_tid=14, entry_tid=16344' 2024-04-06T09:41:03.116 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=88, tag_tid=14, entry_tid=16344' '!=' 'object_number=90, tag_tid=14, entry_tid=16238' 2024-04-06T09:41:03.116 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:41:03.116 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=88, tag_tid=14, entry_tid=16344' = 'object_number=90, tag_tid=14, entry_tid=16238' 2024-04-06T09:41:03.116 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=88, tag_tid=14, entry_tid=16344' 2024-04-06T09:41:03.116 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=93, tag_tid=15, entry_tid=1' 2024-04-06T09:41:03.116 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:41:03.116 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:41:03.117 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=15 2024-04-06T09:41:03.118 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=88, tag_tid=14, entry_tid=16344' 2024-04-06T09:41:03.118 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:41:03.118 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:41:03.119 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=14 2024-04-06T09:41:03.119 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=93, tag_tid=15, entry_tid=1' 2024-04-06T09:41:03.119 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:41:03.119 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:41:03.120 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:41:03.121 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=88, tag_tid=14, entry_tid=16344' 2024-04-06T09:41:03.121 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:41:03.121 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:41:03.122 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=16344 2024-04-06T09:41:03.122 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 15 = 14 -a 1 -le 16344 2024-04-06T09:41:03.122 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:41:03.122 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:41:03.122 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:41:03.324 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:41:03.324 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:41:03.324 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:41:03.324 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:41:03.324 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:41:03.324 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:41:03.325 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:41:03.325 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:41:03.325 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:41:03.325 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:41:03.325 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:41:03.325 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:41:03.325 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:41:03.325 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:41:03.325 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:41:03.326 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:41:03.326 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:41:03.328 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:41:03.328 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:41:03.328 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:41:03.328 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:41:03.328 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:41:03.328 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:41:03.328 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:41:03.328 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:41:03.328 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:41:03.328 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:41:03.328 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:41:03.328 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:41:03.328 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:41:03.329 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:41:03.329 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:41:03.329 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:41:03.329 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:41:03.329 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:41:03.329 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:41:03.329 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:41:03.329 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:41:03.329 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:41:03.329 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:41:03.330 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:41:03.330 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:41:03.330 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:41:03.330 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:41:03.330 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:41:03.330 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:41:03.330 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:41:03.330 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:41:03.330 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:41:03.355 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:41:03.357 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:41:03.357 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:41:03.357 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:41:03.489 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:41:03.490 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:41:03.490 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:41:03.490 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:41:03.490 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:41:03.490 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:41:03.491 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:41:03.491 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:41:03.491 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:41:03.491 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:41:03.491 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:41:03.491 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:41:03.491 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:41:03.492 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:41:03.493 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:41:03.493 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:41:03.542 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 22 2024-04-06T09:41:03.542 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 23 2024-04-06T09:41:03.542 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:41:03.542 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=93, tag_tid=15, entry_tid=1], [object_number=92, tag_tid=15, entry_tid=0], [object_number=95, tag_tid=14, entry_tid=19039], [object_number=94, tag_tid=14, entry_tid=19038]]], state=connected] 2024-04-06T09:41:03.542 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=89, tag_tid=14, entry_tid=16441], [object_number=88, tag_tid=14, entry_tid=16440], [object_number=91, tag_tid=14, entry_tid=16439], [object_number=90, tag_tid=14, entry_tid=16438]]], state=connected] 2024-04-06T09:41:03.545 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:41:03.546 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=93, tag_tid=15, entry_tid=1' 2024-04-06T09:41:03.548 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:41:03.548 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:41:03.548 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:41:03.548 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:41:03.548 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:41:03.548 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:41:03.548 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:41:03.548 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:41:03.548 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:41:03.548 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:41:03.548 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:41:03.548 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:41:03.549 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:41:03.549 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:41:03.549 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:41:03.594 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 22 2024-04-06T09:41:03.595 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 23 2024-04-06T09:41:03.595 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:41:03.595 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=93, tag_tid=15, entry_tid=1], [object_number=92, tag_tid=15, entry_tid=0], [object_number=95, tag_tid=14, entry_tid=19039], [object_number=94, tag_tid=14, entry_tid=19038]]], state=connected] 2024-04-06T09:41:03.595 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=89, tag_tid=14, entry_tid=16441], [object_number=88, tag_tid=14, entry_tid=16440], [object_number=91, tag_tid=14, entry_tid=16439], [object_number=90, tag_tid=14, entry_tid=16438]]], state=connected] 2024-04-06T09:41:03.598 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:41:03.600 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=89, tag_tid=14, entry_tid=16441' 2024-04-06T09:41:03.600 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=93, tag_tid=15, entry_tid=1' -a 'object_number=93, tag_tid=15, entry_tid=1' = 'object_number=89, tag_tid=14, entry_tid=16441' 2024-04-06T09:41:03.600 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=89, tag_tid=14, entry_tid=16441' '!=' 'object_number=88, tag_tid=14, entry_tid=16344' 2024-04-06T09:41:03.600 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:41:03.600 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=89, tag_tid=14, entry_tid=16441' = 'object_number=88, tag_tid=14, entry_tid=16344' 2024-04-06T09:41:03.600 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=89, tag_tid=14, entry_tid=16441' 2024-04-06T09:41:03.601 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=93, tag_tid=15, entry_tid=1' 2024-04-06T09:41:03.601 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:41:03.601 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:41:03.601 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=15 2024-04-06T09:41:03.602 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=89, tag_tid=14, entry_tid=16441' 2024-04-06T09:41:03.602 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:41:03.602 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:41:03.603 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=14 2024-04-06T09:41:03.604 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=93, tag_tid=15, entry_tid=1' 2024-04-06T09:41:03.604 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:41:03.604 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:41:03.605 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:41:03.605 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=89, tag_tid=14, entry_tid=16441' 2024-04-06T09:41:03.605 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:41:03.606 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:41:03.607 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=16441 2024-04-06T09:41:03.607 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 15 = 14 -a 1 -le 16441 2024-04-06T09:41:03.607 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:41:03.607 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:41:03.607 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:41:03.808 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:41:03.809 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:41:03.809 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:41:03.809 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:41:03.809 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:41:03.809 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:41:03.809 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:41:03.809 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:41:03.809 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:41:03.809 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:41:03.809 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:41:03.809 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:41:03.809 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:41:03.810 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:41:03.810 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:41:03.810 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:41:03.811 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:41:03.812 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:41:03.813 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:41:03.813 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:41:03.813 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:41:03.813 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:41:03.813 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:41:03.813 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:41:03.813 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:41:03.813 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:41:03.813 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:41:03.813 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:41:03.813 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:41:03.813 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:41:03.813 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:41:03.814 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:41:03.814 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:41:03.815 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:41:03.815 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:41:03.815 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:41:03.815 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:41:03.815 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:41:03.815 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:41:03.815 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:41:03.815 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:41:03.815 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:41:03.815 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:41:03.815 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:41:03.815 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:41:03.815 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:41:03.815 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:41:03.815 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:41:03.816 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:41:03.839 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:41:03.839 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:41:03.840 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:41:03.840 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:41:03.966 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:41:03.967 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:41:03.967 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:41:03.967 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:41:03.967 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:41:03.967 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:41:03.967 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:41:03.967 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:41:03.967 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:41:03.967 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:41:03.968 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:41:03.968 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:41:03.968 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:41:03.970 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:41:03.971 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:41:03.971 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:41:04.021 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 22 2024-04-06T09:41:04.021 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 23 2024-04-06T09:41:04.022 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:41:04.022 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=93, tag_tid=15, entry_tid=1], [object_number=92, tag_tid=15, entry_tid=0], [object_number=95, tag_tid=14, entry_tid=19039], [object_number=94, tag_tid=14, entry_tid=19038]]], state=connected] 2024-04-06T09:41:04.022 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=89, tag_tid=14, entry_tid=16521], [object_number=88, tag_tid=14, entry_tid=16520], [object_number=91, tag_tid=14, entry_tid=16519], [object_number=90, tag_tid=14, entry_tid=16518]]], state=connected] 2024-04-06T09:41:04.025 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:41:04.026 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=93, tag_tid=15, entry_tid=1' 2024-04-06T09:41:04.027 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:41:04.027 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:41:04.027 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:41:04.027 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:41:04.027 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:41:04.028 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:41:04.028 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:41:04.028 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:41:04.028 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:41:04.028 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:41:04.028 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:41:04.028 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:41:04.028 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:41:04.029 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:41:04.029 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:41:04.076 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 22 2024-04-06T09:41:04.076 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 23 2024-04-06T09:41:04.076 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:41:04.076 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=93, tag_tid=15, entry_tid=1], [object_number=92, tag_tid=15, entry_tid=0], [object_number=95, tag_tid=14, entry_tid=19039], [object_number=94, tag_tid=14, entry_tid=19038]]], state=connected] 2024-04-06T09:41:04.076 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=89, tag_tid=14, entry_tid=16521], [object_number=88, tag_tid=14, entry_tid=16520], [object_number=91, tag_tid=14, entry_tid=16519], [object_number=90, tag_tid=14, entry_tid=16518]]], state=connected] 2024-04-06T09:41:04.079 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:41:04.081 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=89, tag_tid=14, entry_tid=16521' 2024-04-06T09:41:04.081 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=93, tag_tid=15, entry_tid=1' -a 'object_number=93, tag_tid=15, entry_tid=1' = 'object_number=89, tag_tid=14, entry_tid=16521' 2024-04-06T09:41:04.081 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=89, tag_tid=14, entry_tid=16521' '!=' 'object_number=89, tag_tid=14, entry_tid=16441' 2024-04-06T09:41:04.081 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:41:04.081 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=89, tag_tid=14, entry_tid=16521' = 'object_number=89, tag_tid=14, entry_tid=16441' 2024-04-06T09:41:04.082 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=89, tag_tid=14, entry_tid=16521' 2024-04-06T09:41:04.082 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=93, tag_tid=15, entry_tid=1' 2024-04-06T09:41:04.082 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:41:04.082 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:41:04.083 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=15 2024-04-06T09:41:04.083 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=89, tag_tid=14, entry_tid=16521' 2024-04-06T09:41:04.083 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:41:04.083 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:41:04.084 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=14 2024-04-06T09:41:04.085 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=93, tag_tid=15, entry_tid=1' 2024-04-06T09:41:04.085 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:41:04.085 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:41:04.086 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:41:04.087 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=89, tag_tid=14, entry_tid=16521' 2024-04-06T09:41:04.087 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:41:04.087 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:41:04.088 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=16521 2024-04-06T09:41:04.088 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 15 = 14 -a 1 -le 16521 2024-04-06T09:41:04.088 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:41:04.088 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:41:04.088 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:41:04.290 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:41:04.290 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:41:04.290 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:41:04.290 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:41:04.290 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:41:04.290 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:41:04.290 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:41:04.290 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:41:04.290 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:41:04.291 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:41:04.291 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:41:04.291 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:41:04.291 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:41:04.291 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:41:04.291 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:41:04.291 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:41:04.291 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:41:04.296 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:41:04.296 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:41:04.296 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:41:04.296 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:41:04.296 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:41:04.296 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:41:04.296 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:41:04.297 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:41:04.297 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:41:04.297 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:41:04.297 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:41:04.297 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:41:04.297 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:41:04.297 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:41:04.297 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:41:04.297 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:41:04.297 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:41:04.297 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:41:04.297 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:41:04.297 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:41:04.297 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:41:04.297 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:41:04.298 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:41:04.298 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:41:04.298 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:41:04.298 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:41:04.298 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:41:04.298 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:41:04.298 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:41:04.298 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:41:04.298 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:41:04.298 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:41:04.320 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:41:04.321 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:41:04.321 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:41:04.321 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:41:04.364 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:41:04.363+0000 7f2e84d66640 -1 reset not still connected to 0x560d2655fee0 2024-04-06T09:41:04.364 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:41:04.363+0000 7f2e84d66640 -1 reset not still connected to 0x560d27073930 2024-04-06T09:41:04.365 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:41:04.363+0000 7f2e84d66640 -1 reset not still connected to 0x560d27073ba0 2024-04-06T09:41:04.365 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:41:04.363+0000 7f2e84d66640 -1 reset not still connected to 0x560d27073d40 2024-04-06T09:41:04.451 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:41:04.453 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:41:04.453 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:41:04.453 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:41:04.453 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:41:04.453 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:41:04.453 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:41:04.453 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:41:04.453 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:41:04.453 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:41:04.453 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:41:04.454 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:41:04.454 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:41:04.456 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:41:04.456 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:41:04.456 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:41:04.502 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 22 2024-04-06T09:41:04.502 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 23 2024-04-06T09:41:04.502 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:41:04.502 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=93, tag_tid=15, entry_tid=1], [object_number=92, tag_tid=15, entry_tid=0], [object_number=95, tag_tid=14, entry_tid=19039], [object_number=94, tag_tid=14, entry_tid=19038]]], state=connected] 2024-04-06T09:41:04.502 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=91, tag_tid=14, entry_tid=16611], [object_number=90, tag_tid=14, entry_tid=16610], [object_number=89, tag_tid=14, entry_tid=16609], [object_number=88, tag_tid=14, entry_tid=16608]]], state=connected] 2024-04-06T09:41:04.506 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:41:04.507 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=93, tag_tid=15, entry_tid=1' 2024-04-06T09:41:04.508 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:41:04.508 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:41:04.508 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:41:04.508 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:41:04.508 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:41:04.508 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:41:04.508 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:41:04.508 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:41:04.508 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:41:04.509 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:41:04.509 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:41:04.509 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:41:04.509 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:41:04.510 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:41:04.510 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:41:04.559 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 22 2024-04-06T09:41:04.559 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 23 2024-04-06T09:41:04.559 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:41:04.559 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=93, tag_tid=15, entry_tid=1], [object_number=92, tag_tid=15, entry_tid=0], [object_number=95, tag_tid=14, entry_tid=19039], [object_number=94, tag_tid=14, entry_tid=19038]]], state=connected] 2024-04-06T09:41:04.559 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=91, tag_tid=14, entry_tid=16611], [object_number=90, tag_tid=14, entry_tid=16610], [object_number=89, tag_tid=14, entry_tid=16609], [object_number=88, tag_tid=14, entry_tid=16608]]], state=connected] 2024-04-06T09:41:04.562 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:41:04.565 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=91, tag_tid=14, entry_tid=16611' 2024-04-06T09:41:04.565 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=93, tag_tid=15, entry_tid=1' -a 'object_number=93, tag_tid=15, entry_tid=1' = 'object_number=91, tag_tid=14, entry_tid=16611' 2024-04-06T09:41:04.565 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=91, tag_tid=14, entry_tid=16611' '!=' 'object_number=89, tag_tid=14, entry_tid=16521' 2024-04-06T09:41:04.565 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:41:04.565 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=91, tag_tid=14, entry_tid=16611' = 'object_number=89, tag_tid=14, entry_tid=16521' 2024-04-06T09:41:04.565 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=91, tag_tid=14, entry_tid=16611' 2024-04-06T09:41:04.565 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=93, tag_tid=15, entry_tid=1' 2024-04-06T09:41:04.565 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:41:04.566 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:41:04.567 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=15 2024-04-06T09:41:04.567 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=91, tag_tid=14, entry_tid=16611' 2024-04-06T09:41:04.568 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:41:04.568 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:41:04.568 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=14 2024-04-06T09:41:04.570 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=93, tag_tid=15, entry_tid=1' 2024-04-06T09:41:04.570 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:41:04.570 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:41:04.570 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:41:04.570 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=91, tag_tid=14, entry_tid=16611' 2024-04-06T09:41:04.570 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:41:04.571 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:41:04.571 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=16611 2024-04-06T09:41:04.572 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 15 = 14 -a 1 -le 16611 2024-04-06T09:41:04.572 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:41:04.572 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:41:04.572 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:41:04.772 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:41:04.773 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:41:04.773 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:41:04.773 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:41:04.773 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:41:04.773 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:41:04.773 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:41:04.773 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:41:04.773 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:41:04.773 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:41:04.773 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:41:04.773 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:41:04.773 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:41:04.774 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:41:04.774 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:41:04.774 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:41:04.775 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:41:04.777 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:41:04.777 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:41:04.777 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:41:04.777 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:41:04.777 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:41:04.777 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:41:04.777 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:41:04.777 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:41:04.777 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:41:04.777 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:41:04.777 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:41:04.778 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:41:04.778 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:41:04.778 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:41:04.778 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:41:04.778 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:41:04.779 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:41:04.779 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:41:04.779 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:41:04.779 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:41:04.779 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:41:04.779 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:41:04.779 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:41:04.779 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:41:04.779 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:41:04.779 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:41:04.780 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:41:04.780 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:41:04.780 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:41:04.780 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:41:04.780 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:41:04.780 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:41:04.804 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:41:04.804 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:41:04.804 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:41:04.804 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:41:04.940 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:41:04.941 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:41:04.941 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:41:04.941 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:41:04.941 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:41:04.941 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:41:04.941 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:41:04.941 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:41:04.941 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:41:04.941 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:41:04.942 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:41:04.942 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:41:04.942 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:41:04.944 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:41:04.944 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:41:04.944 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:41:04.995 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 22 2024-04-06T09:41:04.995 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 23 2024-04-06T09:41:04.995 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:41:04.995 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=93, tag_tid=15, entry_tid=1], [object_number=92, tag_tid=15, entry_tid=0], [object_number=95, tag_tid=14, entry_tid=19039], [object_number=94, tag_tid=14, entry_tid=19038]]], state=connected] 2024-04-06T09:41:04.995 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=89, tag_tid=14, entry_tid=16709], [object_number=88, tag_tid=14, entry_tid=16708], [object_number=91, tag_tid=14, entry_tid=16707], [object_number=90, tag_tid=14, entry_tid=16706]]], state=connected] 2024-04-06T09:41:04.999 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:41:05.001 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=93, tag_tid=15, entry_tid=1' 2024-04-06T09:41:05.002 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:41:05.002 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:41:05.002 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:41:05.002 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:41:05.002 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:41:05.002 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:41:05.002 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:41:05.002 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:41:05.002 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:41:05.002 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:41:05.002 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:41:05.002 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:41:05.003 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:41:05.003 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:41:05.003 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:41:05.052 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 22 2024-04-06T09:41:05.052 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 23 2024-04-06T09:41:05.052 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:41:05.053 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=93, tag_tid=15, entry_tid=1], [object_number=92, tag_tid=15, entry_tid=0], [object_number=95, tag_tid=14, entry_tid=19039], [object_number=94, tag_tid=14, entry_tid=19038]]], state=connected] 2024-04-06T09:41:05.053 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=89, tag_tid=14, entry_tid=16709], [object_number=88, tag_tid=14, entry_tid=16708], [object_number=91, tag_tid=14, entry_tid=16707], [object_number=90, tag_tid=14, entry_tid=16706]]], state=connected] 2024-04-06T09:41:05.056 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:41:05.058 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=89, tag_tid=14, entry_tid=16709' 2024-04-06T09:41:05.058 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=93, tag_tid=15, entry_tid=1' -a 'object_number=93, tag_tid=15, entry_tid=1' = 'object_number=89, tag_tid=14, entry_tid=16709' 2024-04-06T09:41:05.058 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=89, tag_tid=14, entry_tid=16709' '!=' 'object_number=91, tag_tid=14, entry_tid=16611' 2024-04-06T09:41:05.058 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:41:05.058 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=89, tag_tid=14, entry_tid=16709' = 'object_number=91, tag_tid=14, entry_tid=16611' 2024-04-06T09:41:05.058 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=89, tag_tid=14, entry_tid=16709' 2024-04-06T09:41:05.058 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=93, tag_tid=15, entry_tid=1' 2024-04-06T09:41:05.058 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:41:05.058 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:41:05.059 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=15 2024-04-06T09:41:05.059 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=89, tag_tid=14, entry_tid=16709' 2024-04-06T09:41:05.059 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:41:05.060 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:41:05.060 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=14 2024-04-06T09:41:05.061 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=93, tag_tid=15, entry_tid=1' 2024-04-06T09:41:05.061 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:41:05.061 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:41:05.062 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:41:05.063 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=89, tag_tid=14, entry_tid=16709' 2024-04-06T09:41:05.063 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:41:05.063 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:41:05.064 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=16709 2024-04-06T09:41:05.064 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 15 = 14 -a 1 -le 16709 2024-04-06T09:41:05.064 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:41:05.064 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:41:05.064 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:41:05.267 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:41:05.267 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:41:05.267 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:41:05.267 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:41:05.267 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:41:05.267 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:41:05.267 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:41:05.267 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:41:05.267 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:41:05.267 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:41:05.267 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:41:05.267 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:41:05.268 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:41:05.268 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:41:05.268 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:41:05.268 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:41:05.268 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:41:05.270 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:41:05.270 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:41:05.270 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:41:05.270 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:41:05.270 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:41:05.270 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:41:05.271 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:41:05.271 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:41:05.271 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:41:05.271 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:41:05.271 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:41:05.271 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:41:05.271 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:41:05.271 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:41:05.272 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:41:05.272 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:41:05.273 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:41:05.273 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:41:05.273 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:41:05.273 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:41:05.273 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:41:05.273 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:41:05.273 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:41:05.273 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:41:05.273 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:41:05.273 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:41:05.273 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:41:05.273 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:41:05.273 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:41:05.274 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:41:05.274 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:41:05.274 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:41:05.297 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:41:05.297 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:41:05.297 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:41:05.297 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:41:05.436 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:41:05.436 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:41:05.436 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:41:05.436 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:41:05.436 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:41:05.436 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:41:05.436 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:41:05.436 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:41:05.437 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:41:05.437 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:41:05.437 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:41:05.437 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:41:05.438 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:41:05.439 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:41:05.440 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:41:05.440 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:41:05.496 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 23 2024-04-06T09:41:05.496 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 23 2024-04-06T09:41:05.496 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:41:05.496 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=93, tag_tid=15, entry_tid=1], [object_number=92, tag_tid=15, entry_tid=0], [object_number=95, tag_tid=14, entry_tid=19039], [object_number=94, tag_tid=14, entry_tid=19038]]], state=connected] 2024-04-06T09:41:05.496 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=95, tag_tid=14, entry_tid=16807], [object_number=94, tag_tid=14, entry_tid=16806], [object_number=93, tag_tid=14, entry_tid=16805], [object_number=92, tag_tid=14, entry_tid=16804]]], state=connected] 2024-04-06T09:41:05.500 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:41:05.501 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=93, tag_tid=15, entry_tid=1' 2024-04-06T09:41:05.502 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:41:05.502 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:41:05.502 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:41:05.502 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:41:05.502 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:41:05.502 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:41:05.502 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:41:05.503 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:41:05.503 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:41:05.503 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:41:05.503 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:41:05.503 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:41:05.503 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:41:05.503 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:41:05.504 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:41:05.549 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 23 2024-04-06T09:41:05.549 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 23 2024-04-06T09:41:05.549 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:41:05.549 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=93, tag_tid=15, entry_tid=1], [object_number=92, tag_tid=15, entry_tid=0], [object_number=95, tag_tid=14, entry_tid=19039], [object_number=94, tag_tid=14, entry_tid=19038]]], state=connected] 2024-04-06T09:41:05.550 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=95, tag_tid=14, entry_tid=16807], [object_number=94, tag_tid=14, entry_tid=16806], [object_number=93, tag_tid=14, entry_tid=16805], [object_number=92, tag_tid=14, entry_tid=16804]]], state=connected] 2024-04-06T09:41:05.552 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:41:05.554 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=95, tag_tid=14, entry_tid=16807' 2024-04-06T09:41:05.554 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=93, tag_tid=15, entry_tid=1' -a 'object_number=93, tag_tid=15, entry_tid=1' = 'object_number=95, tag_tid=14, entry_tid=16807' 2024-04-06T09:41:05.554 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=95, tag_tid=14, entry_tid=16807' '!=' 'object_number=89, tag_tid=14, entry_tid=16709' 2024-04-06T09:41:05.554 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:41:05.554 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=95, tag_tid=14, entry_tid=16807' = 'object_number=89, tag_tid=14, entry_tid=16709' 2024-04-06T09:41:05.554 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=95, tag_tid=14, entry_tid=16807' 2024-04-06T09:41:05.555 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=93, tag_tid=15, entry_tid=1' 2024-04-06T09:41:05.555 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:41:05.555 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:41:05.555 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=15 2024-04-06T09:41:05.556 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=95, tag_tid=14, entry_tid=16807' 2024-04-06T09:41:05.556 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:41:05.556 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:41:05.557 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=14 2024-04-06T09:41:05.558 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=93, tag_tid=15, entry_tid=1' 2024-04-06T09:41:05.558 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:41:05.558 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:41:05.559 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:41:05.560 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=95, tag_tid=14, entry_tid=16807' 2024-04-06T09:41:05.560 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:41:05.560 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:41:05.561 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=16807 2024-04-06T09:41:05.561 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 15 = 14 -a 1 -le 16807 2024-04-06T09:41:05.561 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:41:05.561 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:41:05.561 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:41:05.763 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:41:05.763 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:41:05.763 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:41:05.763 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:41:05.763 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:41:05.763 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:41:05.763 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:41:05.763 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:41:05.763 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:41:05.763 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:41:05.763 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:41:05.764 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:41:05.764 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:41:05.764 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:41:05.764 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:41:05.764 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:41:05.764 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:41:05.768 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:41:05.768 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:41:05.768 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:41:05.768 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:41:05.768 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:41:05.768 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:41:05.768 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:41:05.768 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:41:05.768 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:41:05.768 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:41:05.768 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:41:05.768 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:41:05.769 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:41:05.769 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:41:05.769 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:41:05.769 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:41:05.769 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:41:05.769 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:41:05.769 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:41:05.769 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:41:05.769 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:41:05.769 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:41:05.769 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:41:05.769 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:41:05.770 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:41:05.770 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:41:05.770 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:41:05.770 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:41:05.770 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:41:05.770 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:41:05.770 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:41:05.770 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:41:05.796 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:41:05.796 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:41:05.796 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:41:05.796 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:41:05.925 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:41:05.927 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:41:05.927 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:41:05.927 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:41:05.927 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:41:05.927 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:41:05.928 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:41:05.928 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:41:05.928 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:41:05.928 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:41:05.929 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:41:05.929 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:41:05.930 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:41:05.931 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:41:05.931 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:41:05.931 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:41:05.983 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 23 2024-04-06T09:41:05.983 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 23 2024-04-06T09:41:05.983 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:41:05.983 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=93, tag_tid=15, entry_tid=1], [object_number=92, tag_tid=15, entry_tid=0], [object_number=95, tag_tid=14, entry_tid=19039], [object_number=94, tag_tid=14, entry_tid=19038]]], state=connected] 2024-04-06T09:41:05.983 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=93, tag_tid=14, entry_tid=16917], [object_number=92, tag_tid=14, entry_tid=16916], [object_number=95, tag_tid=14, entry_tid=16915], [object_number=94, tag_tid=14, entry_tid=16914]]], state=connected] 2024-04-06T09:41:05.987 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:41:05.988 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=93, tag_tid=15, entry_tid=1' 2024-04-06T09:41:05.989 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:41:05.989 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:41:05.990 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:41:05.990 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:41:05.990 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:41:05.990 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:41:05.990 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:41:05.991 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:41:05.991 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:41:05.991 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:41:05.991 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:41:05.991 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:41:05.992 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:41:05.992 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:41:05.992 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:41:06.038 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 23 2024-04-06T09:41:06.038 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 23 2024-04-06T09:41:06.038 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:41:06.039 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=93, tag_tid=15, entry_tid=1], [object_number=92, tag_tid=15, entry_tid=0], [object_number=95, tag_tid=14, entry_tid=19039], [object_number=94, tag_tid=14, entry_tid=19038]]], state=connected] 2024-04-06T09:41:06.039 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=93, tag_tid=14, entry_tid=16917], [object_number=92, tag_tid=14, entry_tid=16916], [object_number=95, tag_tid=14, entry_tid=16915], [object_number=94, tag_tid=14, entry_tid=16914]]], state=connected] 2024-04-06T09:41:06.042 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:41:06.044 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=93, tag_tid=14, entry_tid=16917' 2024-04-06T09:41:06.044 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=93, tag_tid=15, entry_tid=1' -a 'object_number=93, tag_tid=15, entry_tid=1' = 'object_number=93, tag_tid=14, entry_tid=16917' 2024-04-06T09:41:06.044 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=93, tag_tid=14, entry_tid=16917' '!=' 'object_number=95, tag_tid=14, entry_tid=16807' 2024-04-06T09:41:06.045 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:41:06.045 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=93, tag_tid=14, entry_tid=16917' = 'object_number=95, tag_tid=14, entry_tid=16807' 2024-04-06T09:41:06.045 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=93, tag_tid=14, entry_tid=16917' 2024-04-06T09:41:06.045 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=93, tag_tid=15, entry_tid=1' 2024-04-06T09:41:06.045 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:41:06.045 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:41:06.046 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=15 2024-04-06T09:41:06.046 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=93, tag_tid=14, entry_tid=16917' 2024-04-06T09:41:06.047 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:41:06.047 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:41:06.048 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=14 2024-04-06T09:41:06.048 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=93, tag_tid=15, entry_tid=1' 2024-04-06T09:41:06.048 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:41:06.048 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:41:06.049 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:41:06.049 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=93, tag_tid=14, entry_tid=16917' 2024-04-06T09:41:06.049 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:41:06.049 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:41:06.050 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=16917 2024-04-06T09:41:06.050 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 15 = 14 -a 1 -le 16917 2024-04-06T09:41:06.051 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:41:06.051 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:41:06.051 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:41:06.252 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:41:06.252 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:41:06.252 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:41:06.252 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:41:06.252 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:41:06.252 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:41:06.252 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:41:06.253 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:41:06.253 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:41:06.253 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:41:06.253 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:41:06.253 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:41:06.253 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:41:06.253 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:41:06.253 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:41:06.253 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:41:06.253 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:41:06.257 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:41:06.257 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:41:06.257 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:41:06.257 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:41:06.257 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:41:06.257 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:41:06.257 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:41:06.257 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:41:06.257 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:41:06.257 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:41:06.257 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:41:06.257 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:41:06.258 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:41:06.258 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:41:06.258 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:41:06.258 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:41:06.258 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:41:06.258 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:41:06.258 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:41:06.258 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:41:06.258 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:41:06.258 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:41:06.258 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:41:06.258 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:41:06.258 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:41:06.258 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:41:06.258 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:41:06.259 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:41:06.259 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:41:06.259 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:41:06.259 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:41:06.259 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:41:06.282 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:41:06.283 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:41:06.283 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:41:06.283 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:41:06.416 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:41:06.417 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:41:06.417 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:41:06.417 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:41:06.417 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:41:06.417 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:41:06.417 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:41:06.417 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:41:06.417 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:41:06.417 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:41:06.418 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:41:06.418 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:41:06.418 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:41:06.420 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:41:06.420 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:41:06.420 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:41:06.467 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 23 2024-04-06T09:41:06.467 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 23 2024-04-06T09:41:06.467 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:41:06.467 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=93, tag_tid=15, entry_tid=1], [object_number=92, tag_tid=15, entry_tid=0], [object_number=95, tag_tid=14, entry_tid=19039], [object_number=94, tag_tid=14, entry_tid=19038]]], state=connected] 2024-04-06T09:41:06.467 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=93, tag_tid=14, entry_tid=17005], [object_number=92, tag_tid=14, entry_tid=17004], [object_number=95, tag_tid=14, entry_tid=17003], [object_number=94, tag_tid=14, entry_tid=17002]]], state=connected] 2024-04-06T09:41:06.470 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:41:06.472 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=93, tag_tid=15, entry_tid=1' 2024-04-06T09:41:06.473 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:41:06.473 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:41:06.473 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:41:06.473 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:41:06.473 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:41:06.473 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:41:06.473 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:41:06.474 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:41:06.474 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:41:06.474 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:41:06.474 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:41:06.474 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:41:06.474 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:41:06.474 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:41:06.475 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:41:06.520 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 23 2024-04-06T09:41:06.520 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 23 2024-04-06T09:41:06.520 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:41:06.520 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=93, tag_tid=15, entry_tid=1], [object_number=92, tag_tid=15, entry_tid=0], [object_number=95, tag_tid=14, entry_tid=19039], [object_number=94, tag_tid=14, entry_tid=19038]]], state=connected] 2024-04-06T09:41:06.520 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=93, tag_tid=14, entry_tid=17005], [object_number=92, tag_tid=14, entry_tid=17004], [object_number=95, tag_tid=14, entry_tid=17003], [object_number=94, tag_tid=14, entry_tid=17002]]], state=connected] 2024-04-06T09:41:06.523 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:41:06.525 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=93, tag_tid=14, entry_tid=17005' 2024-04-06T09:41:06.525 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=93, tag_tid=15, entry_tid=1' -a 'object_number=93, tag_tid=15, entry_tid=1' = 'object_number=93, tag_tid=14, entry_tid=17005' 2024-04-06T09:41:06.525 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=93, tag_tid=14, entry_tid=17005' '!=' 'object_number=93, tag_tid=14, entry_tid=16917' 2024-04-06T09:41:06.525 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:41:06.525 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=93, tag_tid=14, entry_tid=17005' = 'object_number=93, tag_tid=14, entry_tid=16917' 2024-04-06T09:41:06.525 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=93, tag_tid=14, entry_tid=17005' 2024-04-06T09:41:06.526 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=93, tag_tid=15, entry_tid=1' 2024-04-06T09:41:06.526 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:41:06.526 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:41:06.526 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=15 2024-04-06T09:41:06.527 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=93, tag_tid=14, entry_tid=17005' 2024-04-06T09:41:06.527 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:41:06.527 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:41:06.528 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=14 2024-04-06T09:41:06.529 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=93, tag_tid=15, entry_tid=1' 2024-04-06T09:41:06.529 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:41:06.529 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:41:06.530 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:41:06.530 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=93, tag_tid=14, entry_tid=17005' 2024-04-06T09:41:06.531 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:41:06.531 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:41:06.532 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=17005 2024-04-06T09:41:06.532 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 15 = 14 -a 1 -le 17005 2024-04-06T09:41:06.532 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:41:06.532 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:41:06.532 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:41:06.734 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:41:06.734 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:41:06.734 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:41:06.734 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:41:06.734 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:41:06.734 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:41:06.734 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:41:06.734 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:41:06.734 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:41:06.735 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:41:06.735 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:41:06.735 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:41:06.735 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:41:06.735 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:41:06.735 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:41:06.735 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:41:06.735 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:41:06.741 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:41:06.741 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:41:06.741 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:41:06.741 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:41:06.741 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:41:06.741 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:41:06.742 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:41:06.742 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:41:06.742 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:41:06.742 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:41:06.742 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:41:06.742 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:41:06.742 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:41:06.743 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:41:06.743 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:41:06.743 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:41:06.743 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:41:06.743 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:41:06.743 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:41:06.743 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:41:06.744 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:41:06.744 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:41:06.744 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:41:06.744 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:41:06.744 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:41:06.745 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:41:06.745 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:41:06.745 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:41:06.745 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:41:06.745 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:41:06.745 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:41:06.746 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:41:06.764 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:41:06.765 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:41:06.765 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:41:06.765 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:41:06.897 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:41:06.898 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:41:06.898 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:41:06.898 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:41:06.898 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:41:06.898 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:41:06.898 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:41:06.898 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:41:06.898 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:41:06.898 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:41:06.899 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:41:06.899 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:41:06.899 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:41:06.902 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:41:06.902 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:41:06.903 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:41:06.948 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 23 2024-04-06T09:41:06.948 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 23 2024-04-06T09:41:06.948 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:41:06.948 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=93, tag_tid=15, entry_tid=1], [object_number=92, tag_tid=15, entry_tid=0], [object_number=95, tag_tid=14, entry_tid=19039], [object_number=94, tag_tid=14, entry_tid=19038]]], state=connected] 2024-04-06T09:41:06.949 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=94, tag_tid=14, entry_tid=17090], [object_number=93, tag_tid=14, entry_tid=17089], [object_number=92, tag_tid=14, entry_tid=17088], [object_number=95, tag_tid=14, entry_tid=17087]]], state=connected] 2024-04-06T09:41:06.952 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:41:06.953 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=93, tag_tid=15, entry_tid=1' 2024-04-06T09:41:06.954 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:41:06.955 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:41:06.955 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:41:06.955 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:41:06.955 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:41:06.955 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:41:06.955 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:41:06.955 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:41:06.955 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:41:06.955 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:41:06.955 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:41:06.955 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:41:06.956 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:41:06.956 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:41:06.956 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:41:07.002 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 23 2024-04-06T09:41:07.002 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 23 2024-04-06T09:41:07.002 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:41:07.002 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=93, tag_tid=15, entry_tid=1], [object_number=92, tag_tid=15, entry_tid=0], [object_number=95, tag_tid=14, entry_tid=19039], [object_number=94, tag_tid=14, entry_tid=19038]]], state=connected] 2024-04-06T09:41:07.002 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=94, tag_tid=14, entry_tid=17090], [object_number=93, tag_tid=14, entry_tid=17089], [object_number=92, tag_tid=14, entry_tid=17088], [object_number=95, tag_tid=14, entry_tid=17087]]], state=connected] 2024-04-06T09:41:07.005 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:41:07.007 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=94, tag_tid=14, entry_tid=17090' 2024-04-06T09:41:07.007 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=93, tag_tid=15, entry_tid=1' -a 'object_number=93, tag_tid=15, entry_tid=1' = 'object_number=94, tag_tid=14, entry_tid=17090' 2024-04-06T09:41:07.008 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=94, tag_tid=14, entry_tid=17090' '!=' 'object_number=93, tag_tid=14, entry_tid=17005' 2024-04-06T09:41:07.008 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:41:07.008 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=94, tag_tid=14, entry_tid=17090' = 'object_number=93, tag_tid=14, entry_tid=17005' 2024-04-06T09:41:07.008 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=94, tag_tid=14, entry_tid=17090' 2024-04-06T09:41:07.008 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=93, tag_tid=15, entry_tid=1' 2024-04-06T09:41:07.008 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:41:07.008 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:41:07.008 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=15 2024-04-06T09:41:07.009 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=94, tag_tid=14, entry_tid=17090' 2024-04-06T09:41:07.009 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:41:07.009 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:41:07.010 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=14 2024-04-06T09:41:07.011 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=93, tag_tid=15, entry_tid=1' 2024-04-06T09:41:07.011 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:41:07.011 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:41:07.013 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:41:07.013 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=94, tag_tid=14, entry_tid=17090' 2024-04-06T09:41:07.013 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:41:07.013 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:41:07.014 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=17090 2024-04-06T09:41:07.014 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 15 = 14 -a 1 -le 17090 2024-04-06T09:41:07.014 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:41:07.014 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:41:07.014 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:41:07.216 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:41:07.216 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:41:07.216 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:41:07.216 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:41:07.216 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:41:07.216 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:41:07.216 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:41:07.216 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:41:07.216 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:41:07.216 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:41:07.216 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:41:07.216 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:41:07.216 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:41:07.216 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:41:07.217 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:41:07.217 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:41:07.217 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:41:07.220 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:41:07.220 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:41:07.220 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:41:07.220 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:41:07.220 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:41:07.220 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:41:07.220 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:41:07.220 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:41:07.221 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:41:07.221 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:41:07.221 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:41:07.221 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:41:07.221 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:41:07.221 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:41:07.221 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:41:07.221 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:41:07.222 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:41:07.222 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:41:07.222 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:41:07.222 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:41:07.222 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:41:07.223 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:41:07.223 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:41:07.223 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:41:07.223 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:41:07.223 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:41:07.223 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:41:07.223 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:41:07.223 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:41:07.223 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:41:07.223 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:41:07.223 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:41:07.250 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:41:07.250 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:41:07.250 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:41:07.250 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:41:07.383 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:41:07.384 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:41:07.384 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:41:07.385 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:41:07.385 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:41:07.385 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:41:07.385 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:41:07.385 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:41:07.385 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:41:07.385 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:41:07.386 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:41:07.386 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:41:07.386 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:41:07.388 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:41:07.388 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:41:07.388 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:41:07.437 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 23 2024-04-06T09:41:07.437 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 23 2024-04-06T09:41:07.437 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:41:07.437 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=93, tag_tid=15, entry_tid=1], [object_number=92, tag_tid=15, entry_tid=0], [object_number=95, tag_tid=14, entry_tid=19039], [object_number=94, tag_tid=14, entry_tid=19038]]], state=connected] 2024-04-06T09:41:07.437 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=93, tag_tid=14, entry_tid=17177], [object_number=92, tag_tid=14, entry_tid=17176], [object_number=95, tag_tid=14, entry_tid=17175], [object_number=94, tag_tid=14, entry_tid=17174]]], state=connected] 2024-04-06T09:41:07.441 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:41:07.442 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=93, tag_tid=15, entry_tid=1' 2024-04-06T09:41:07.443 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:41:07.443 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:41:07.443 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:41:07.443 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:41:07.443 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:41:07.443 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:41:07.443 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:41:07.443 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:41:07.443 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:41:07.444 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:41:07.444 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:41:07.444 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:41:07.444 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:41:07.444 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:41:07.444 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:41:07.491 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 23 2024-04-06T09:41:07.491 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 23 2024-04-06T09:41:07.491 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:41:07.491 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=93, tag_tid=15, entry_tid=1], [object_number=92, tag_tid=15, entry_tid=0], [object_number=95, tag_tid=14, entry_tid=19039], [object_number=94, tag_tid=14, entry_tid=19038]]], state=connected] 2024-04-06T09:41:07.491 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=93, tag_tid=14, entry_tid=17177], [object_number=92, tag_tid=14, entry_tid=17176], [object_number=95, tag_tid=14, entry_tid=17175], [object_number=94, tag_tid=14, entry_tid=17174]]], state=connected] 2024-04-06T09:41:07.494 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:41:07.496 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=93, tag_tid=14, entry_tid=17177' 2024-04-06T09:41:07.496 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=93, tag_tid=15, entry_tid=1' -a 'object_number=93, tag_tid=15, entry_tid=1' = 'object_number=93, tag_tid=14, entry_tid=17177' 2024-04-06T09:41:07.496 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=93, tag_tid=14, entry_tid=17177' '!=' 'object_number=94, tag_tid=14, entry_tid=17090' 2024-04-06T09:41:07.496 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:41:07.496 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=93, tag_tid=14, entry_tid=17177' = 'object_number=94, tag_tid=14, entry_tid=17090' 2024-04-06T09:41:07.496 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=93, tag_tid=14, entry_tid=17177' 2024-04-06T09:41:07.497 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=93, tag_tid=15, entry_tid=1' 2024-04-06T09:41:07.497 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:41:07.497 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:41:07.497 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=15 2024-04-06T09:41:07.498 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=93, tag_tid=14, entry_tid=17177' 2024-04-06T09:41:07.498 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:41:07.498 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:41:07.499 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=14 2024-04-06T09:41:07.500 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=93, tag_tid=15, entry_tid=1' 2024-04-06T09:41:07.500 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:41:07.500 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:41:07.501 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:41:07.501 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=93, tag_tid=14, entry_tid=17177' 2024-04-06T09:41:07.501 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:41:07.501 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:41:07.503 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=17177 2024-04-06T09:41:07.503 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 15 = 14 -a 1 -le 17177 2024-04-06T09:41:07.503 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:41:07.503 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:41:07.503 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:41:07.704 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:41:07.705 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:41:07.705 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:41:07.705 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:41:07.705 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:41:07.705 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:41:07.705 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:41:07.705 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:41:07.705 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:41:07.705 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:41:07.705 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:41:07.705 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:41:07.705 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:41:07.705 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:41:07.705 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:41:07.706 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:41:07.706 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:41:07.710 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:41:07.710 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:41:07.710 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:41:07.710 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:41:07.710 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:41:07.710 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:41:07.710 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:41:07.710 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:41:07.710 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:41:07.710 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:41:07.710 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:41:07.711 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:41:07.711 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:41:07.711 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:41:07.711 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:41:07.711 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:41:07.711 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:41:07.711 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:41:07.711 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:41:07.711 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:41:07.711 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:41:07.711 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:41:07.711 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:41:07.712 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:41:07.712 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:41:07.712 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:41:07.712 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:41:07.712 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:41:07.712 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:41:07.712 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:41:07.712 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:41:07.712 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:41:07.735 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:41:07.735 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:41:07.735 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:41:07.735 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:41:07.858 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:41:07.859 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:41:07.859 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:41:07.859 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:41:07.860 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:41:07.860 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:41:07.860 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:41:07.860 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:41:07.860 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:41:07.860 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:41:07.860 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:41:07.861 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:41:07.861 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:41:07.861 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:41:07.862 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:41:07.862 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:41:07.908 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 23 2024-04-06T09:41:07.908 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 23 2024-04-06T09:41:07.908 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:41:07.908 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=93, tag_tid=15, entry_tid=1], [object_number=92, tag_tid=15, entry_tid=0], [object_number=95, tag_tid=14, entry_tid=19039], [object_number=94, tag_tid=14, entry_tid=19038]]], state=connected] 2024-04-06T09:41:07.908 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=94, tag_tid=14, entry_tid=17266], [object_number=93, tag_tid=14, entry_tid=17265], [object_number=92, tag_tid=14, entry_tid=17264], [object_number=95, tag_tid=14, entry_tid=17263]]], state=connected] 2024-04-06T09:41:07.911 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:41:07.914 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=93, tag_tid=15, entry_tid=1' 2024-04-06T09:41:07.914 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:41:07.914 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:41:07.914 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:41:07.914 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:41:07.914 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:41:07.914 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:41:07.914 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:41:07.914 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:41:07.914 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:41:07.915 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:41:07.915 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:41:07.915 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:41:07.915 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:41:07.915 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:41:07.915 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:41:07.961 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 23 2024-04-06T09:41:07.962 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 23 2024-04-06T09:41:07.962 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:41:07.962 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=93, tag_tid=15, entry_tid=1], [object_number=92, tag_tid=15, entry_tid=0], [object_number=95, tag_tid=14, entry_tid=19039], [object_number=94, tag_tid=14, entry_tid=19038]]], state=connected] 2024-04-06T09:41:07.962 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=94, tag_tid=14, entry_tid=17266], [object_number=93, tag_tid=14, entry_tid=17265], [object_number=92, tag_tid=14, entry_tid=17264], [object_number=95, tag_tid=14, entry_tid=17263]]], state=connected] 2024-04-06T09:41:07.965 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:41:07.967 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=94, tag_tid=14, entry_tid=17266' 2024-04-06T09:41:07.967 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=93, tag_tid=15, entry_tid=1' -a 'object_number=93, tag_tid=15, entry_tid=1' = 'object_number=94, tag_tid=14, entry_tid=17266' 2024-04-06T09:41:07.967 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=94, tag_tid=14, entry_tid=17266' '!=' 'object_number=93, tag_tid=14, entry_tid=17177' 2024-04-06T09:41:07.967 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:41:07.968 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=94, tag_tid=14, entry_tid=17266' = 'object_number=93, tag_tid=14, entry_tid=17177' 2024-04-06T09:41:07.968 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=94, tag_tid=14, entry_tid=17266' 2024-04-06T09:41:07.968 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=93, tag_tid=15, entry_tid=1' 2024-04-06T09:41:07.968 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:41:07.968 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:41:07.968 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=15 2024-04-06T09:41:07.969 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=94, tag_tid=14, entry_tid=17266' 2024-04-06T09:41:07.969 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:41:07.969 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:41:07.970 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=14 2024-04-06T09:41:07.971 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=93, tag_tid=15, entry_tid=1' 2024-04-06T09:41:07.971 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:41:07.971 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:41:07.972 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:41:07.972 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=94, tag_tid=14, entry_tid=17266' 2024-04-06T09:41:07.972 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:41:07.973 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:41:07.974 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=17266 2024-04-06T09:41:07.974 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 15 = 14 -a 1 -le 17266 2024-04-06T09:41:07.974 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:41:07.974 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:41:07.974 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:41:08.034 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:41:08.031+0000 7f2e84d66640 -1 reset not still connected to 0x560d27073d40 2024-04-06T09:41:08.175 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:41:08.175 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:41:08.176 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:41:08.176 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:41:08.176 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:41:08.176 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:41:08.176 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:41:08.176 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:41:08.176 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:41:08.176 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:41:08.176 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:41:08.176 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:41:08.176 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:41:08.176 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:41:08.176 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:41:08.177 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:41:08.177 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:41:08.179 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:41:08.180 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:41:08.180 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:41:08.180 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:41:08.180 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:41:08.180 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:41:08.180 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:41:08.180 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:41:08.180 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:41:08.180 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:41:08.180 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:41:08.180 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:41:08.180 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:41:08.180 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:41:08.180 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:41:08.180 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:41:08.181 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:41:08.181 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:41:08.181 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:41:08.181 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:41:08.181 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:41:08.181 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:41:08.181 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:41:08.181 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:41:08.181 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:41:08.181 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:41:08.181 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:41:08.181 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:41:08.181 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:41:08.181 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:41:08.181 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:41:08.182 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:41:08.207 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:41:08.207 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:41:08.207 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:41:08.207 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:41:08.337 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:41:08.338 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:41:08.338 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:41:08.338 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:41:08.338 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:41:08.338 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:41:08.338 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:41:08.338 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:41:08.338 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:41:08.338 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:41:08.339 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:41:08.339 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:41:08.339 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:41:08.342 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:41:08.342 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:41:08.342 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:41:08.394 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 23 2024-04-06T09:41:08.394 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 23 2024-04-06T09:41:08.394 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:41:08.394 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=93, tag_tid=15, entry_tid=1], [object_number=92, tag_tid=15, entry_tid=0], [object_number=95, tag_tid=14, entry_tid=19039], [object_number=94, tag_tid=14, entry_tid=19038]]], state=connected] 2024-04-06T09:41:08.394 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=92, tag_tid=14, entry_tid=17356], [object_number=95, tag_tid=14, entry_tid=17355], [object_number=94, tag_tid=14, entry_tid=17354], [object_number=93, tag_tid=14, entry_tid=17353]]], state=connected] 2024-04-06T09:41:08.398 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:41:08.399 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=93, tag_tid=15, entry_tid=1' 2024-04-06T09:41:08.399 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:41:08.400 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:41:08.400 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:41:08.400 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:41:08.400 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:41:08.400 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:41:08.400 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:41:08.400 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:41:08.400 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:41:08.400 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:41:08.401 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:41:08.401 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:41:08.401 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:41:08.401 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:41:08.401 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:41:08.448 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 23 2024-04-06T09:41:08.448 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 23 2024-04-06T09:41:08.448 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:41:08.448 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=93, tag_tid=15, entry_tid=1], [object_number=92, tag_tid=15, entry_tid=0], [object_number=95, tag_tid=14, entry_tid=19039], [object_number=94, tag_tid=14, entry_tid=19038]]], state=connected] 2024-04-06T09:41:08.448 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=92, tag_tid=14, entry_tid=17356], [object_number=95, tag_tid=14, entry_tid=17355], [object_number=94, tag_tid=14, entry_tid=17354], [object_number=93, tag_tid=14, entry_tid=17353]]], state=connected] 2024-04-06T09:41:08.452 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:41:08.454 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=92, tag_tid=14, entry_tid=17356' 2024-04-06T09:41:08.454 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=93, tag_tid=15, entry_tid=1' -a 'object_number=93, tag_tid=15, entry_tid=1' = 'object_number=92, tag_tid=14, entry_tid=17356' 2024-04-06T09:41:08.454 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=92, tag_tid=14, entry_tid=17356' '!=' 'object_number=94, tag_tid=14, entry_tid=17266' 2024-04-06T09:41:08.454 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:41:08.454 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=92, tag_tid=14, entry_tid=17356' = 'object_number=94, tag_tid=14, entry_tid=17266' 2024-04-06T09:41:08.454 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=92, tag_tid=14, entry_tid=17356' 2024-04-06T09:41:08.454 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=93, tag_tid=15, entry_tid=1' 2024-04-06T09:41:08.454 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:41:08.454 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:41:08.455 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=15 2024-04-06T09:41:08.456 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=92, tag_tid=14, entry_tid=17356' 2024-04-06T09:41:08.456 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:41:08.456 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:41:08.457 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=14 2024-04-06T09:41:08.457 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=93, tag_tid=15, entry_tid=1' 2024-04-06T09:41:08.457 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:41:08.457 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:41:08.458 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:41:08.459 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=92, tag_tid=14, entry_tid=17356' 2024-04-06T09:41:08.459 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:41:08.459 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:41:08.460 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=17356 2024-04-06T09:41:08.460 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 15 = 14 -a 1 -le 17356 2024-04-06T09:41:08.460 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:41:08.460 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:41:08.461 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:41:08.662 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:41:08.662 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:41:08.662 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:41:08.663 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:41:08.663 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:41:08.663 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:41:08.663 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:41:08.663 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:41:08.663 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:41:08.663 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:41:08.663 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:41:08.663 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:41:08.663 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:41:08.663 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:41:08.663 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:41:08.664 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:41:08.664 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:41:08.666 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:41:08.667 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:41:08.667 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:41:08.667 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:41:08.667 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:41:08.667 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:41:08.667 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:41:08.667 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:41:08.667 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:41:08.667 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:41:08.667 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:41:08.667 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:41:08.667 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:41:08.667 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:41:08.667 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:41:08.667 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:41:08.668 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:41:08.668 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:41:08.669 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:41:08.669 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:41:08.669 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:41:08.669 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:41:08.669 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:41:08.669 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:41:08.669 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:41:08.669 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:41:08.669 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:41:08.669 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:41:08.669 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:41:08.669 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:41:08.669 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:41:08.669 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:41:08.695 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:41:08.695 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:41:08.695 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:41:08.695 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:41:08.821 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:41:08.822 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:41:08.822 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:41:08.822 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:41:08.822 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:41:08.822 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:41:08.822 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:41:08.822 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:41:08.822 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:41:08.822 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:41:08.823 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:41:08.823 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:41:08.823 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:41:08.825 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:41:08.825 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:41:08.825 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:41:08.872 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 23 2024-04-06T09:41:08.872 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 23 2024-04-06T09:41:08.872 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:41:08.872 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=93, tag_tid=15, entry_tid=1], [object_number=92, tag_tid=15, entry_tid=0], [object_number=95, tag_tid=14, entry_tid=19039], [object_number=94, tag_tid=14, entry_tid=19038]]], state=connected] 2024-04-06T09:41:08.872 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=94, tag_tid=14, entry_tid=17450], [object_number=93, tag_tid=14, entry_tid=17449], [object_number=92, tag_tid=14, entry_tid=17448], [object_number=95, tag_tid=14, entry_tid=17447]]], state=connected] 2024-04-06T09:41:08.876 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:41:08.877 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=93, tag_tid=15, entry_tid=1' 2024-04-06T09:41:08.878 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:41:08.878 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:41:08.878 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:41:08.878 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:41:08.878 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:41:08.878 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:41:08.878 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:41:08.878 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:41:08.879 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:41:08.879 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:41:08.879 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:41:08.879 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:41:08.879 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:41:08.880 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:41:08.880 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:41:08.925 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 23 2024-04-06T09:41:08.925 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 23 2024-04-06T09:41:08.925 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:41:08.925 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=93, tag_tid=15, entry_tid=1], [object_number=92, tag_tid=15, entry_tid=0], [object_number=95, tag_tid=14, entry_tid=19039], [object_number=94, tag_tid=14, entry_tid=19038]]], state=connected] 2024-04-06T09:41:08.925 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=94, tag_tid=14, entry_tid=17450], [object_number=93, tag_tid=14, entry_tid=17449], [object_number=92, tag_tid=14, entry_tid=17448], [object_number=95, tag_tid=14, entry_tid=17447]]], state=connected] 2024-04-06T09:41:08.929 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:41:08.931 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=94, tag_tid=14, entry_tid=17450' 2024-04-06T09:41:08.931 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=93, tag_tid=15, entry_tid=1' -a 'object_number=93, tag_tid=15, entry_tid=1' = 'object_number=94, tag_tid=14, entry_tid=17450' 2024-04-06T09:41:08.931 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=94, tag_tid=14, entry_tid=17450' '!=' 'object_number=92, tag_tid=14, entry_tid=17356' 2024-04-06T09:41:08.931 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:41:08.931 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=94, tag_tid=14, entry_tid=17450' = 'object_number=92, tag_tid=14, entry_tid=17356' 2024-04-06T09:41:08.931 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=94, tag_tid=14, entry_tid=17450' 2024-04-06T09:41:08.931 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=93, tag_tid=15, entry_tid=1' 2024-04-06T09:41:08.932 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:41:08.932 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:41:08.932 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=15 2024-04-06T09:41:08.933 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=94, tag_tid=14, entry_tid=17450' 2024-04-06T09:41:08.933 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:41:08.933 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:41:08.934 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=14 2024-04-06T09:41:08.935 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=93, tag_tid=15, entry_tid=1' 2024-04-06T09:41:08.935 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:41:08.935 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:41:08.936 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:41:08.936 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=94, tag_tid=14, entry_tid=17450' 2024-04-06T09:41:08.936 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:41:08.936 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:41:08.937 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=17450 2024-04-06T09:41:08.938 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 15 = 14 -a 1 -le 17450 2024-04-06T09:41:08.938 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:41:08.938 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:41:08.938 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:41:09.140 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:41:09.140 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:41:09.140 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:41:09.140 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:41:09.140 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:41:09.140 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:41:09.140 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:41:09.140 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:41:09.140 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:41:09.140 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:41:09.140 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:41:09.140 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:41:09.140 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:41:09.141 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:41:09.141 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:41:09.141 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:41:09.141 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:41:09.144 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:41:09.144 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:41:09.144 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:41:09.144 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:41:09.144 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:41:09.144 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:41:09.144 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:41:09.144 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:41:09.144 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:41:09.144 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:41:09.144 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:41:09.144 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:41:09.144 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:41:09.144 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:41:09.145 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:41:09.145 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:41:09.145 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:41:09.145 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:41:09.145 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:41:09.145 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:41:09.146 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:41:09.146 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:41:09.146 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:41:09.146 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:41:09.146 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:41:09.146 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:41:09.146 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:41:09.146 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:41:09.146 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:41:09.146 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:41:09.146 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:41:09.146 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:41:09.170 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:41:09.170 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:41:09.170 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:41:09.171 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:41:09.296 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:41:09.298 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:41:09.298 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:41:09.298 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:41:09.298 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:41:09.298 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:41:09.298 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:41:09.298 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:41:09.298 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:41:09.298 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:41:09.299 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:41:09.299 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:41:09.299 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:41:09.300 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:41:09.301 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:41:09.301 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:41:09.355 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 23 2024-04-06T09:41:09.355 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 23 2024-04-06T09:41:09.355 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:41:09.355 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=93, tag_tid=15, entry_tid=1], [object_number=92, tag_tid=15, entry_tid=0], [object_number=95, tag_tid=14, entry_tid=19039], [object_number=94, tag_tid=14, entry_tid=19038]]], state=connected] 2024-04-06T09:41:09.355 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=93, tag_tid=14, entry_tid=17541], [object_number=92, tag_tid=14, entry_tid=17540], [object_number=95, tag_tid=14, entry_tid=17539], [object_number=94, tag_tid=14, entry_tid=17538]]], state=connected] 2024-04-06T09:41:09.359 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:41:09.361 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=93, tag_tid=15, entry_tid=1' 2024-04-06T09:41:09.362 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:41:09.362 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:41:09.362 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:41:09.362 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:41:09.362 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:41:09.362 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:41:09.362 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:41:09.362 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:41:09.362 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:41:09.362 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:41:09.362 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:41:09.362 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:41:09.363 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:41:09.363 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:41:09.363 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:41:09.410 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 23 2024-04-06T09:41:09.410 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 23 2024-04-06T09:41:09.411 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:41:09.411 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=93, tag_tid=15, entry_tid=1], [object_number=92, tag_tid=15, entry_tid=0], [object_number=95, tag_tid=14, entry_tid=19039], [object_number=94, tag_tid=14, entry_tid=19038]]], state=connected] 2024-04-06T09:41:09.411 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=93, tag_tid=14, entry_tid=17541], [object_number=92, tag_tid=14, entry_tid=17540], [object_number=95, tag_tid=14, entry_tid=17539], [object_number=94, tag_tid=14, entry_tid=17538]]], state=connected] 2024-04-06T09:41:09.414 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:41:09.416 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=93, tag_tid=14, entry_tid=17541' 2024-04-06T09:41:09.416 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=93, tag_tid=15, entry_tid=1' -a 'object_number=93, tag_tid=15, entry_tid=1' = 'object_number=93, tag_tid=14, entry_tid=17541' 2024-04-06T09:41:09.416 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=93, tag_tid=14, entry_tid=17541' '!=' 'object_number=94, tag_tid=14, entry_tid=17450' 2024-04-06T09:41:09.416 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:41:09.417 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=93, tag_tid=14, entry_tid=17541' = 'object_number=94, tag_tid=14, entry_tid=17450' 2024-04-06T09:41:09.417 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=93, tag_tid=14, entry_tid=17541' 2024-04-06T09:41:09.417 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=93, tag_tid=15, entry_tid=1' 2024-04-06T09:41:09.417 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:41:09.417 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:41:09.418 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=15 2024-04-06T09:41:09.418 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=93, tag_tid=14, entry_tid=17541' 2024-04-06T09:41:09.418 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:41:09.418 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:41:09.419 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=14 2024-04-06T09:41:09.420 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=93, tag_tid=15, entry_tid=1' 2024-04-06T09:41:09.420 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:41:09.420 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:41:09.421 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:41:09.422 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=93, tag_tid=14, entry_tid=17541' 2024-04-06T09:41:09.422 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:41:09.422 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:41:09.423 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=17541 2024-04-06T09:41:09.423 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 15 = 14 -a 1 -le 17541 2024-04-06T09:41:09.423 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:41:09.424 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:41:09.424 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:41:09.625 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:41:09.625 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:41:09.625 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:41:09.625 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:41:09.626 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:41:09.626 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:41:09.626 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:41:09.626 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:41:09.626 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:41:09.626 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:41:09.626 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:41:09.626 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:41:09.626 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:41:09.626 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:41:09.626 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:41:09.627 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:41:09.627 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:41:09.628 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:41:09.628 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:41:09.628 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:41:09.628 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:41:09.628 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:41:09.629 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:41:09.629 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:41:09.629 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:41:09.629 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:41:09.629 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:41:09.629 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:41:09.629 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:41:09.629 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:41:09.629 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:41:09.629 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:41:09.629 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:41:09.631 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:41:09.631 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:41:09.631 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:41:09.631 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:41:09.631 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:41:09.631 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:41:09.631 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:41:09.631 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:41:09.631 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:41:09.631 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:41:09.631 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:41:09.631 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:41:09.632 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:41:09.632 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:41:09.632 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:41:09.632 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:41:09.655 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:41:09.655 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:41:09.656 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:41:09.656 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:41:09.784 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:41:09.785 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:41:09.785 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:41:09.785 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:41:09.785 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:41:09.785 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:41:09.785 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:41:09.785 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:41:09.785 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:41:09.786 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:41:09.786 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:41:09.786 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:41:09.786 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:41:09.788 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:41:09.788 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:41:09.788 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:41:09.840 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 23 2024-04-06T09:41:09.840 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 23 2024-04-06T09:41:09.840 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:41:09.840 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=93, tag_tid=15, entry_tid=1], [object_number=92, tag_tid=15, entry_tid=0], [object_number=95, tag_tid=14, entry_tid=19039], [object_number=94, tag_tid=14, entry_tid=19038]]], state=connected] 2024-04-06T09:41:09.840 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=92, tag_tid=14, entry_tid=17632], [object_number=95, tag_tid=14, entry_tid=17631], [object_number=94, tag_tid=14, entry_tid=17630], [object_number=93, tag_tid=14, entry_tid=17629]]], state=connected] 2024-04-06T09:41:09.844 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:41:09.845 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=93, tag_tid=15, entry_tid=1' 2024-04-06T09:41:09.846 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:41:09.846 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:41:09.846 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:41:09.846 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:41:09.847 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:41:09.847 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:41:09.847 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:41:09.847 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:41:09.847 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:41:09.847 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:41:09.847 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:41:09.847 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:41:09.848 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:41:09.848 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:41:09.848 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:41:09.895 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 23 2024-04-06T09:41:09.896 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 23 2024-04-06T09:41:09.896 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:41:09.896 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=93, tag_tid=15, entry_tid=1], [object_number=92, tag_tid=15, entry_tid=0], [object_number=95, tag_tid=14, entry_tid=19039], [object_number=94, tag_tid=14, entry_tid=19038]]], state=connected] 2024-04-06T09:41:09.896 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=92, tag_tid=14, entry_tid=17632], [object_number=95, tag_tid=14, entry_tid=17631], [object_number=94, tag_tid=14, entry_tid=17630], [object_number=93, tag_tid=14, entry_tid=17629]]], state=connected] 2024-04-06T09:41:09.899 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:41:09.901 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=92, tag_tid=14, entry_tid=17632' 2024-04-06T09:41:09.901 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=93, tag_tid=15, entry_tid=1' -a 'object_number=93, tag_tid=15, entry_tid=1' = 'object_number=92, tag_tid=14, entry_tid=17632' 2024-04-06T09:41:09.901 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=92, tag_tid=14, entry_tid=17632' '!=' 'object_number=93, tag_tid=14, entry_tid=17541' 2024-04-06T09:41:09.901 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:41:09.901 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=92, tag_tid=14, entry_tid=17632' = 'object_number=93, tag_tid=14, entry_tid=17541' 2024-04-06T09:41:09.901 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=92, tag_tid=14, entry_tid=17632' 2024-04-06T09:41:09.902 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=93, tag_tid=15, entry_tid=1' 2024-04-06T09:41:09.902 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:41:09.902 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:41:09.902 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=15 2024-04-06T09:41:09.903 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=92, tag_tid=14, entry_tid=17632' 2024-04-06T09:41:09.903 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:41:09.903 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:41:09.904 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=14 2024-04-06T09:41:09.905 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=93, tag_tid=15, entry_tid=1' 2024-04-06T09:41:09.905 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:41:09.905 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:41:09.906 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:41:09.906 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=92, tag_tid=14, entry_tid=17632' 2024-04-06T09:41:09.906 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:41:09.906 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:41:09.908 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=17632 2024-04-06T09:41:09.908 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 15 = 14 -a 1 -le 17632 2024-04-06T09:41:09.908 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:41:09.908 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:41:09.908 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:41:10.110 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:41:10.110 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:41:10.110 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:41:10.110 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:41:10.110 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:41:10.110 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:41:10.110 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:41:10.110 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:41:10.110 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:41:10.110 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:41:10.110 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:41:10.111 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:41:10.111 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:41:10.111 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:41:10.111 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:41:10.111 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:41:10.111 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:41:10.113 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:41:10.114 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:41:10.114 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:41:10.114 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:41:10.114 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:41:10.114 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:41:10.114 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:41:10.114 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:41:10.114 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:41:10.114 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:41:10.114 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:41:10.114 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:41:10.114 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:41:10.114 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:41:10.114 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:41:10.115 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:41:10.115 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:41:10.115 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:41:10.115 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:41:10.116 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:41:10.116 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:41:10.116 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:41:10.116 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:41:10.116 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:41:10.116 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:41:10.116 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:41:10.116 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:41:10.116 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:41:10.116 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:41:10.116 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:41:10.116 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:41:10.116 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:41:10.140 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:41:10.140 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:41:10.140 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:41:10.140 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:41:10.275 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:41:10.276 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:41:10.276 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:41:10.276 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:41:10.276 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:41:10.276 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:41:10.277 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:41:10.277 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:41:10.277 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:41:10.277 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:41:10.278 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:41:10.278 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:41:10.278 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:41:10.279 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:41:10.279 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:41:10.280 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:41:10.330 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 23 2024-04-06T09:41:10.330 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 23 2024-04-06T09:41:10.331 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:41:10.331 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=93, tag_tid=15, entry_tid=1], [object_number=92, tag_tid=15, entry_tid=0], [object_number=95, tag_tid=14, entry_tid=19039], [object_number=94, tag_tid=14, entry_tid=19038]]], state=connected] 2024-04-06T09:41:10.331 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=92, tag_tid=14, entry_tid=17724], [object_number=95, tag_tid=14, entry_tid=17723], [object_number=94, tag_tid=14, entry_tid=17722], [object_number=93, tag_tid=14, entry_tid=17721]]], state=connected] 2024-04-06T09:41:10.334 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:41:10.335 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=93, tag_tid=15, entry_tid=1' 2024-04-06T09:41:10.336 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:41:10.337 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:41:10.337 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:41:10.337 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:41:10.337 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:41:10.337 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:41:10.337 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:41:10.337 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:41:10.337 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:41:10.337 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:41:10.337 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:41:10.337 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:41:10.338 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:41:10.338 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:41:10.338 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:41:10.385 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 23 2024-04-06T09:41:10.385 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 23 2024-04-06T09:41:10.385 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:41:10.385 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=93, tag_tid=15, entry_tid=1], [object_number=92, tag_tid=15, entry_tid=0], [object_number=95, tag_tid=14, entry_tid=19039], [object_number=94, tag_tid=14, entry_tid=19038]]], state=connected] 2024-04-06T09:41:10.385 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=92, tag_tid=14, entry_tid=17724], [object_number=95, tag_tid=14, entry_tid=17723], [object_number=94, tag_tid=14, entry_tid=17722], [object_number=93, tag_tid=14, entry_tid=17721]]], state=connected] 2024-04-06T09:41:10.388 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:41:10.391 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=92, tag_tid=14, entry_tid=17724' 2024-04-06T09:41:10.391 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=93, tag_tid=15, entry_tid=1' -a 'object_number=93, tag_tid=15, entry_tid=1' = 'object_number=92, tag_tid=14, entry_tid=17724' 2024-04-06T09:41:10.391 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=92, tag_tid=14, entry_tid=17724' '!=' 'object_number=92, tag_tid=14, entry_tid=17632' 2024-04-06T09:41:10.391 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:41:10.391 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=92, tag_tid=14, entry_tid=17724' = 'object_number=92, tag_tid=14, entry_tid=17632' 2024-04-06T09:41:10.391 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=92, tag_tid=14, entry_tid=17724' 2024-04-06T09:41:10.391 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=93, tag_tid=15, entry_tid=1' 2024-04-06T09:41:10.391 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:41:10.391 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:41:10.392 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=15 2024-04-06T09:41:10.392 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=92, tag_tid=14, entry_tid=17724' 2024-04-06T09:41:10.392 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:41:10.392 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:41:10.393 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=14 2024-04-06T09:41:10.394 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=93, tag_tid=15, entry_tid=1' 2024-04-06T09:41:10.394 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:41:10.394 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:41:10.395 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:41:10.395 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=92, tag_tid=14, entry_tid=17724' 2024-04-06T09:41:10.396 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:41:10.396 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:41:10.397 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=17724 2024-04-06T09:41:10.397 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 15 = 14 -a 1 -le 17724 2024-04-06T09:41:10.397 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:41:10.397 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:41:10.397 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:41:10.600 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:41:10.600 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:41:10.600 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:41:10.600 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:41:10.601 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:41:10.601 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:41:10.601 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:41:10.601 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:41:10.601 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:41:10.601 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:41:10.601 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:41:10.601 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:41:10.601 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:41:10.601 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:41:10.601 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:41:10.601 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:41:10.602 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:41:10.603 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:41:10.603 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:41:10.603 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:41:10.604 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:41:10.604 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:41:10.604 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:41:10.604 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:41:10.604 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:41:10.604 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:41:10.604 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:41:10.604 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:41:10.605 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:41:10.605 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:41:10.605 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:41:10.605 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:41:10.605 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:41:10.605 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:41:10.605 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:41:10.605 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:41:10.605 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:41:10.605 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:41:10.605 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:41:10.605 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:41:10.606 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:41:10.606 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:41:10.606 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:41:10.606 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:41:10.606 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:41:10.606 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:41:10.606 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:41:10.606 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:41:10.606 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:41:10.631 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:41:10.631 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:41:10.631 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:41:10.631 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:41:10.759 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:41:10.759 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:41:10.760 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:41:10.760 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:41:10.760 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:41:10.760 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:41:10.760 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:41:10.760 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:41:10.760 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:41:10.760 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:41:10.761 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:41:10.761 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:41:10.761 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:41:10.763 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:41:10.764 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:41:10.764 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:41:10.820 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 23 2024-04-06T09:41:10.820 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 23 2024-04-06T09:41:10.821 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:41:10.821 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=93, tag_tid=15, entry_tid=1], [object_number=92, tag_tid=15, entry_tid=0], [object_number=95, tag_tid=14, entry_tid=19039], [object_number=94, tag_tid=14, entry_tid=19038]]], state=connected] 2024-04-06T09:41:10.821 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=93, tag_tid=14, entry_tid=17817], [object_number=92, tag_tid=14, entry_tid=17816], [object_number=95, tag_tid=14, entry_tid=17815], [object_number=94, tag_tid=14, entry_tid=17814]]], state=connected] 2024-04-06T09:41:10.824 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:41:10.826 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=93, tag_tid=15, entry_tid=1' 2024-04-06T09:41:10.827 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:41:10.827 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:41:10.827 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:41:10.827 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:41:10.827 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:41:10.827 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:41:10.827 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:41:10.827 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:41:10.828 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:41:10.828 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:41:10.828 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:41:10.828 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:41:10.828 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:41:10.829 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:41:10.829 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:41:10.875 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 23 2024-04-06T09:41:10.875 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 23 2024-04-06T09:41:10.875 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:41:10.875 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=93, tag_tid=15, entry_tid=1], [object_number=92, tag_tid=15, entry_tid=0], [object_number=95, tag_tid=14, entry_tid=19039], [object_number=94, tag_tid=14, entry_tid=19038]]], state=connected] 2024-04-06T09:41:10.875 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=93, tag_tid=14, entry_tid=17817], [object_number=92, tag_tid=14, entry_tid=17816], [object_number=95, tag_tid=14, entry_tid=17815], [object_number=94, tag_tid=14, entry_tid=17814]]], state=connected] 2024-04-06T09:41:10.879 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:41:10.880 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=93, tag_tid=14, entry_tid=17817' 2024-04-06T09:41:10.880 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=93, tag_tid=15, entry_tid=1' -a 'object_number=93, tag_tid=15, entry_tid=1' = 'object_number=93, tag_tid=14, entry_tid=17817' 2024-04-06T09:41:10.880 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=93, tag_tid=14, entry_tid=17817' '!=' 'object_number=92, tag_tid=14, entry_tid=17724' 2024-04-06T09:41:10.881 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:41:10.881 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=93, tag_tid=14, entry_tid=17817' = 'object_number=92, tag_tid=14, entry_tid=17724' 2024-04-06T09:41:10.881 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=93, tag_tid=14, entry_tid=17817' 2024-04-06T09:41:10.881 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=93, tag_tid=15, entry_tid=1' 2024-04-06T09:41:10.881 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:41:10.881 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:41:10.882 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=15 2024-04-06T09:41:10.882 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=93, tag_tid=14, entry_tid=17817' 2024-04-06T09:41:10.883 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:41:10.883 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:41:10.884 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=14 2024-04-06T09:41:10.884 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=93, tag_tid=15, entry_tid=1' 2024-04-06T09:41:10.884 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:41:10.884 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:41:10.885 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:41:10.886 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=93, tag_tid=14, entry_tid=17817' 2024-04-06T09:41:10.886 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:41:10.886 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:41:10.887 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=17817 2024-04-06T09:41:10.887 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 15 = 14 -a 1 -le 17817 2024-04-06T09:41:10.887 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:41:10.887 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:41:10.887 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:41:11.089 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:41:11.089 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:41:11.089 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:41:11.090 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:41:11.090 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:41:11.090 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:41:11.090 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:41:11.090 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:41:11.090 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:41:11.090 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:41:11.090 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:41:11.090 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:41:11.090 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:41:11.090 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:41:11.090 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:41:11.091 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:41:11.091 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:41:11.092 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:41:11.093 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:41:11.093 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:41:11.093 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:41:11.093 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:41:11.093 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:41:11.093 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:41:11.093 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:41:11.093 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:41:11.093 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:41:11.093 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:41:11.093 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:41:11.093 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:41:11.093 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:41:11.093 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:41:11.094 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:41:11.095 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:41:11.095 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:41:11.095 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:41:11.095 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:41:11.095 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:41:11.095 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:41:11.095 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:41:11.096 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:41:11.096 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:41:11.096 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:41:11.096 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:41:11.096 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:41:11.096 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:41:11.096 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:41:11.096 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:41:11.096 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:41:11.120 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:41:11.120 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:41:11.120 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:41:11.120 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:41:11.253 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:41:11.254 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:41:11.254 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:41:11.254 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:41:11.254 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:41:11.254 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:41:11.254 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:41:11.254 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:41:11.254 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:41:11.254 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:41:11.255 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:41:11.255 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:41:11.255 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:41:11.258 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:41:11.258 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:41:11.258 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:41:11.308 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 23 2024-04-06T09:41:11.308 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 23 2024-04-06T09:41:11.309 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:41:11.309 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=93, tag_tid=15, entry_tid=1], [object_number=92, tag_tid=15, entry_tid=0], [object_number=95, tag_tid=14, entry_tid=19039], [object_number=94, tag_tid=14, entry_tid=19038]]], state=connected] 2024-04-06T09:41:11.309 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=93, tag_tid=14, entry_tid=17921], [object_number=92, tag_tid=14, entry_tid=17920], [object_number=95, tag_tid=14, entry_tid=17919], [object_number=94, tag_tid=14, entry_tid=17918]]], state=connected] 2024-04-06T09:41:11.313 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:41:11.314 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=93, tag_tid=15, entry_tid=1' 2024-04-06T09:41:11.315 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:41:11.315 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:41:11.315 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:41:11.315 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:41:11.316 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:41:11.316 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:41:11.316 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:41:11.316 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:41:11.316 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:41:11.316 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:41:11.316 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:41:11.316 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:41:11.317 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:41:11.317 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:41:11.317 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:41:11.363 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 23 2024-04-06T09:41:11.363 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 23 2024-04-06T09:41:11.363 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:41:11.363 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=93, tag_tid=15, entry_tid=1], [object_number=92, tag_tid=15, entry_tid=0], [object_number=95, tag_tid=14, entry_tid=19039], [object_number=94, tag_tid=14, entry_tid=19038]]], state=connected] 2024-04-06T09:41:11.363 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=93, tag_tid=14, entry_tid=17921], [object_number=92, tag_tid=14, entry_tid=17920], [object_number=95, tag_tid=14, entry_tid=17919], [object_number=94, tag_tid=14, entry_tid=17918]]], state=connected] 2024-04-06T09:41:11.367 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:41:11.368 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=93, tag_tid=14, entry_tid=17921' 2024-04-06T09:41:11.369 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=93, tag_tid=15, entry_tid=1' -a 'object_number=93, tag_tid=15, entry_tid=1' = 'object_number=93, tag_tid=14, entry_tid=17921' 2024-04-06T09:41:11.369 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=93, tag_tid=14, entry_tid=17921' '!=' 'object_number=93, tag_tid=14, entry_tid=17817' 2024-04-06T09:41:11.369 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:41:11.369 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=93, tag_tid=14, entry_tid=17921' = 'object_number=93, tag_tid=14, entry_tid=17817' 2024-04-06T09:41:11.369 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=93, tag_tid=14, entry_tid=17921' 2024-04-06T09:41:11.369 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=93, tag_tid=15, entry_tid=1' 2024-04-06T09:41:11.370 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:41:11.370 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:41:11.370 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=15 2024-04-06T09:41:11.371 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=93, tag_tid=14, entry_tid=17921' 2024-04-06T09:41:11.371 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:41:11.371 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:41:11.372 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=14 2024-04-06T09:41:11.373 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=93, tag_tid=15, entry_tid=1' 2024-04-06T09:41:11.373 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:41:11.373 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:41:11.374 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:41:11.374 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=93, tag_tid=14, entry_tid=17921' 2024-04-06T09:41:11.375 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:41:11.375 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:41:11.375 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=17921 2024-04-06T09:41:11.376 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 15 = 14 -a 1 -le 17921 2024-04-06T09:41:11.376 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:41:11.376 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:41:11.376 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:41:11.578 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:41:11.578 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:41:11.578 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:41:11.578 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:41:11.578 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:41:11.578 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:41:11.578 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:41:11.578 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:41:11.579 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:41:11.579 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:41:11.579 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:41:11.579 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:41:11.579 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:41:11.579 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:41:11.579 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:41:11.579 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:41:11.580 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:41:11.583 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:41:11.583 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:41:11.583 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:41:11.583 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:41:11.583 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:41:11.583 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:41:11.583 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:41:11.584 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:41:11.584 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:41:11.584 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:41:11.584 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:41:11.584 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:41:11.584 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:41:11.584 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:41:11.584 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:41:11.584 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:41:11.584 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:41:11.584 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:41:11.584 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:41:11.584 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:41:11.584 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:41:11.585 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:41:11.585 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:41:11.585 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:41:11.585 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:41:11.585 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:41:11.585 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:41:11.585 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:41:11.585 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:41:11.585 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:41:11.585 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:41:11.585 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:41:11.609 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:41:11.609 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:41:11.610 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:41:11.610 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:41:11.738 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:41:11.739 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:41:11.739 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:41:11.739 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:41:11.739 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:41:11.739 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:41:11.739 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:41:11.739 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:41:11.739 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:41:11.739 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:41:11.739 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:41:11.740 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:41:11.740 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:41:11.742 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:41:11.742 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:41:11.742 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:41:12.001 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 23 2024-04-06T09:41:12.001 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 23 2024-04-06T09:41:12.001 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:41:12.001 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=93, tag_tid=15, entry_tid=1], [object_number=92, tag_tid=15, entry_tid=0], [object_number=95, tag_tid=14, entry_tid=19039], [object_number=94, tag_tid=14, entry_tid=19038]]], state=connected] 2024-04-06T09:41:12.001 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=94, tag_tid=14, entry_tid=18022], [object_number=93, tag_tid=14, entry_tid=18021], [object_number=92, tag_tid=14, entry_tid=18020], [object_number=95, tag_tid=14, entry_tid=18019]]], state=connected] 2024-04-06T09:41:12.007 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:41:12.009 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=93, tag_tid=15, entry_tid=1' 2024-04-06T09:41:12.011 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:41:12.011 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:41:12.011 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:41:12.011 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:41:12.011 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:41:12.011 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:41:12.011 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:41:12.011 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:41:12.011 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:41:12.011 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:41:12.012 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:41:12.012 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:41:12.013 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:41:12.014 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:41:12.014 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:41:12.072 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 23 2024-04-06T09:41:12.072 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 23 2024-04-06T09:41:12.072 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:41:12.072 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=93, tag_tid=15, entry_tid=1], [object_number=92, tag_tid=15, entry_tid=0], [object_number=95, tag_tid=14, entry_tid=19039], [object_number=94, tag_tid=14, entry_tid=19038]]], state=connected] 2024-04-06T09:41:12.072 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=94, tag_tid=14, entry_tid=18022], [object_number=93, tag_tid=14, entry_tid=18021], [object_number=92, tag_tid=14, entry_tid=18020], [object_number=95, tag_tid=14, entry_tid=18019]]], state=connected] 2024-04-06T09:41:12.076 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:41:12.078 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=94, tag_tid=14, entry_tid=18022' 2024-04-06T09:41:12.078 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=93, tag_tid=15, entry_tid=1' -a 'object_number=93, tag_tid=15, entry_tid=1' = 'object_number=94, tag_tid=14, entry_tid=18022' 2024-04-06T09:41:12.078 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=94, tag_tid=14, entry_tid=18022' '!=' 'object_number=93, tag_tid=14, entry_tid=17921' 2024-04-06T09:41:12.078 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:41:12.079 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=94, tag_tid=14, entry_tid=18022' = 'object_number=93, tag_tid=14, entry_tid=17921' 2024-04-06T09:41:12.079 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=94, tag_tid=14, entry_tid=18022' 2024-04-06T09:41:12.079 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=93, tag_tid=15, entry_tid=1' 2024-04-06T09:41:12.079 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:41:12.079 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:41:12.080 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=15 2024-04-06T09:41:12.080 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=94, tag_tid=14, entry_tid=18022' 2024-04-06T09:41:12.080 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:41:12.080 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:41:12.081 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=14 2024-04-06T09:41:12.082 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=93, tag_tid=15, entry_tid=1' 2024-04-06T09:41:12.082 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:41:12.082 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:41:12.083 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:41:12.084 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=94, tag_tid=14, entry_tid=18022' 2024-04-06T09:41:12.084 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:41:12.084 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:41:12.085 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=18022 2024-04-06T09:41:12.085 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 15 = 14 -a 1 -le 18022 2024-04-06T09:41:12.086 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:41:12.086 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:41:12.086 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:41:12.287 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:41:12.287 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:41:12.287 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:41:12.288 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:41:12.288 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:41:12.288 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:41:12.288 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:41:12.288 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:41:12.288 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:41:12.288 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:41:12.288 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:41:12.288 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:41:12.288 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:41:12.288 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:41:12.288 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:41:12.289 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:41:12.289 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:41:12.291 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:41:12.292 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:41:12.292 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:41:12.292 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:41:12.292 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:41:12.292 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:41:12.292 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:41:12.292 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:41:12.292 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:41:12.292 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:41:12.292 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:41:12.292 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:41:12.292 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:41:12.292 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:41:12.292 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:41:12.292 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:41:12.293 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:41:12.293 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:41:12.293 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:41:12.293 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:41:12.293 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:41:12.293 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:41:12.293 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:41:12.293 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:41:12.293 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:41:12.293 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:41:12.293 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:41:12.293 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:41:12.293 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:41:12.293 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:41:12.293 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:41:12.294 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:41:12.318 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:41:12.319 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:41:12.319 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:41:12.319 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:41:12.446 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:41:12.447 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:41:12.448 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:41:12.448 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:41:12.448 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:41:12.448 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:41:12.448 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:41:12.448 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:41:12.448 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:41:12.448 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:41:12.448 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:41:12.448 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:41:12.449 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:41:12.450 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:41:12.450 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:41:12.450 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:41:12.497 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 23 2024-04-06T09:41:12.497 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 23 2024-04-06T09:41:12.497 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:41:12.497 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=93, tag_tid=15, entry_tid=1], [object_number=92, tag_tid=15, entry_tid=0], [object_number=95, tag_tid=14, entry_tid=19039], [object_number=94, tag_tid=14, entry_tid=19038]]], state=connected] 2024-04-06T09:41:12.497 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=92, tag_tid=14, entry_tid=18188], [object_number=95, tag_tid=14, entry_tid=18187], [object_number=94, tag_tid=14, entry_tid=18186], [object_number=93, tag_tid=14, entry_tid=18185]]], state=connected] 2024-04-06T09:41:12.500 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:41:12.503 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=93, tag_tid=15, entry_tid=1' 2024-04-06T09:41:12.503 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:41:12.503 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:41:12.503 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:41:12.503 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:41:12.503 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:41:12.503 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:41:12.503 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:41:12.503 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:41:12.503 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:41:12.503 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:41:12.503 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:41:12.503 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:41:12.504 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:41:12.504 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:41:12.504 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:41:12.552 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 23 2024-04-06T09:41:12.552 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 23 2024-04-06T09:41:12.552 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:41:12.552 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=93, tag_tid=15, entry_tid=1], [object_number=92, tag_tid=15, entry_tid=0], [object_number=95, tag_tid=14, entry_tid=19039], [object_number=94, tag_tid=14, entry_tid=19038]]], state=connected] 2024-04-06T09:41:12.552 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=92, tag_tid=14, entry_tid=18188], [object_number=95, tag_tid=14, entry_tid=18187], [object_number=94, tag_tid=14, entry_tid=18186], [object_number=93, tag_tid=14, entry_tid=18185]]], state=connected] 2024-04-06T09:41:12.555 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:41:12.557 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=92, tag_tid=14, entry_tid=18188' 2024-04-06T09:41:12.557 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=93, tag_tid=15, entry_tid=1' -a 'object_number=93, tag_tid=15, entry_tid=1' = 'object_number=92, tag_tid=14, entry_tid=18188' 2024-04-06T09:41:12.557 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=92, tag_tid=14, entry_tid=18188' '!=' 'object_number=94, tag_tid=14, entry_tid=18022' 2024-04-06T09:41:12.557 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:41:12.557 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=92, tag_tid=14, entry_tid=18188' = 'object_number=94, tag_tid=14, entry_tid=18022' 2024-04-06T09:41:12.557 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=92, tag_tid=14, entry_tid=18188' 2024-04-06T09:41:12.558 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=93, tag_tid=15, entry_tid=1' 2024-04-06T09:41:12.558 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:41:12.558 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:41:12.559 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=15 2024-04-06T09:41:12.560 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=92, tag_tid=14, entry_tid=18188' 2024-04-06T09:41:12.560 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:41:12.560 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:41:12.560 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=14 2024-04-06T09:41:12.561 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=93, tag_tid=15, entry_tid=1' 2024-04-06T09:41:12.561 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:41:12.561 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:41:12.562 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:41:12.563 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=92, tag_tid=14, entry_tid=18188' 2024-04-06T09:41:12.563 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:41:12.563 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:41:12.564 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=18188 2024-04-06T09:41:12.564 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 15 = 14 -a 1 -le 18188 2024-04-06T09:41:12.564 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:41:12.564 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:41:12.565 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:41:12.766 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:41:12.767 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:41:12.767 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:41:12.767 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:41:12.767 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:41:12.767 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:41:12.767 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:41:12.767 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:41:12.767 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:41:12.767 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:41:12.767 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:41:12.767 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:41:12.767 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:41:12.768 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:41:12.768 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:41:12.768 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:41:12.768 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:41:12.770 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:41:12.771 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:41:12.771 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:41:12.771 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:41:12.771 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:41:12.771 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:41:12.771 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:41:12.771 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:41:12.771 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:41:12.771 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:41:12.771 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:41:12.771 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:41:12.771 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:41:12.771 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:41:12.771 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:41:12.771 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:41:12.772 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:41:12.772 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:41:12.773 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:41:12.773 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:41:12.773 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:41:12.773 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:41:12.773 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:41:12.773 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:41:12.773 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:41:12.773 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:41:12.773 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:41:12.773 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:41:12.773 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:41:12.773 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:41:12.773 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:41:12.773 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:41:12.797 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:41:12.797 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:41:12.798 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:41:12.798 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:41:12.933 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:41:12.934 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:41:12.934 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:41:12.934 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:41:12.935 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:41:12.935 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:41:12.935 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:41:12.935 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:41:12.935 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:41:12.935 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:41:12.935 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:41:12.935 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:41:12.936 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:41:12.938 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:41:12.938 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:41:12.938 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:41:12.985 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 23 2024-04-06T09:41:12.985 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 23 2024-04-06T09:41:12.985 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:41:12.985 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=93, tag_tid=15, entry_tid=1], [object_number=92, tag_tid=15, entry_tid=0], [object_number=95, tag_tid=14, entry_tid=19039], [object_number=94, tag_tid=14, entry_tid=19038]]], state=connected] 2024-04-06T09:41:12.986 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=92, tag_tid=14, entry_tid=18288], [object_number=95, tag_tid=14, entry_tid=18287], [object_number=94, tag_tid=14, entry_tid=18286], [object_number=93, tag_tid=14, entry_tid=18285]]], state=connected] 2024-04-06T09:41:12.989 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:41:12.990 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=93, tag_tid=15, entry_tid=1' 2024-04-06T09:41:12.991 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:41:12.991 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:41:12.992 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:41:12.992 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:41:12.992 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:41:12.992 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:41:12.992 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:41:12.992 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:41:12.992 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:41:12.992 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:41:12.992 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:41:12.992 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:41:12.993 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:41:12.993 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:41:12.993 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:41:13.039 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 23 2024-04-06T09:41:13.039 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 23 2024-04-06T09:41:13.039 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:41:13.040 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=93, tag_tid=15, entry_tid=1], [object_number=92, tag_tid=15, entry_tid=0], [object_number=95, tag_tid=14, entry_tid=19039], [object_number=94, tag_tid=14, entry_tid=19038]]], state=connected] 2024-04-06T09:41:13.040 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=92, tag_tid=14, entry_tid=18288], [object_number=95, tag_tid=14, entry_tid=18287], [object_number=94, tag_tid=14, entry_tid=18286], [object_number=93, tag_tid=14, entry_tid=18285]]], state=connected] 2024-04-06T09:41:13.043 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:41:13.045 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=92, tag_tid=14, entry_tid=18288' 2024-04-06T09:41:13.045 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=93, tag_tid=15, entry_tid=1' -a 'object_number=93, tag_tid=15, entry_tid=1' = 'object_number=92, tag_tid=14, entry_tid=18288' 2024-04-06T09:41:13.045 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=92, tag_tid=14, entry_tid=18288' '!=' 'object_number=92, tag_tid=14, entry_tid=18188' 2024-04-06T09:41:13.046 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:41:13.046 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=92, tag_tid=14, entry_tid=18288' = 'object_number=92, tag_tid=14, entry_tid=18188' 2024-04-06T09:41:13.046 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=92, tag_tid=14, entry_tid=18288' 2024-04-06T09:41:13.046 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=93, tag_tid=15, entry_tid=1' 2024-04-06T09:41:13.046 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:41:13.046 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:41:13.046 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=15 2024-04-06T09:41:13.047 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=92, tag_tid=14, entry_tid=18288' 2024-04-06T09:41:13.047 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:41:13.047 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:41:13.048 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=14 2024-04-06T09:41:13.049 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=93, tag_tid=15, entry_tid=1' 2024-04-06T09:41:13.049 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:41:13.049 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:41:13.050 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:41:13.051 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=92, tag_tid=14, entry_tid=18288' 2024-04-06T09:41:13.051 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:41:13.051 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:41:13.052 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=18288 2024-04-06T09:41:13.052 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 15 = 14 -a 1 -le 18288 2024-04-06T09:41:13.052 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:41:13.052 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:41:13.052 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:41:13.254 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:41:13.254 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:41:13.254 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:41:13.254 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:41:13.254 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:41:13.254 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:41:13.254 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:41:13.255 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:41:13.255 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:41:13.255 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:41:13.255 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:41:13.255 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:41:13.255 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:41:13.255 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:41:13.255 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:41:13.255 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:41:13.256 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:41:13.258 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:41:13.258 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:41:13.258 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:41:13.258 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:41:13.259 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:41:13.259 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:41:13.259 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:41:13.259 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:41:13.259 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:41:13.259 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:41:13.259 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:41:13.259 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:41:13.259 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:41:13.259 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:41:13.259 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:41:13.259 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:41:13.260 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:41:13.260 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:41:13.261 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:41:13.261 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:41:13.261 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:41:13.261 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:41:13.261 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:41:13.261 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:41:13.261 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:41:13.261 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:41:13.261 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:41:13.261 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:41:13.261 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:41:13.261 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:41:13.261 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:41:13.261 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:41:13.284 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:41:13.285 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:41:13.285 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:41:13.285 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:41:13.416 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:41:13.418 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:41:13.418 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:41:13.418 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:41:13.418 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:41:13.418 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:41:13.418 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:41:13.418 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:41:13.418 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:41:13.418 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:41:13.419 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:41:13.419 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:41:13.419 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:41:13.420 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:41:13.420 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:41:13.421 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:41:13.477 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 23 2024-04-06T09:41:13.477 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 23 2024-04-06T09:41:13.478 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:41:13.478 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=93, tag_tid=15, entry_tid=1], [object_number=92, tag_tid=15, entry_tid=0], [object_number=95, tag_tid=14, entry_tid=19039], [object_number=94, tag_tid=14, entry_tid=19038]]], state=connected] 2024-04-06T09:41:13.478 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=95, tag_tid=14, entry_tid=18379], [object_number=94, tag_tid=14, entry_tid=18378], [object_number=93, tag_tid=14, entry_tid=18377], [object_number=92, tag_tid=14, entry_tid=18376]]], state=connected] 2024-04-06T09:41:13.481 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:41:13.482 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=93, tag_tid=15, entry_tid=1' 2024-04-06T09:41:13.483 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:41:13.484 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:41:13.484 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:41:13.484 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:41:13.484 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:41:13.484 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:41:13.484 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:41:13.484 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:41:13.484 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:41:13.484 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:41:13.484 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:41:13.484 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:41:13.485 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:41:13.485 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:41:13.485 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:41:13.532 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 23 2024-04-06T09:41:13.532 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 23 2024-04-06T09:41:13.532 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:41:13.532 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=93, tag_tid=15, entry_tid=1], [object_number=92, tag_tid=15, entry_tid=0], [object_number=95, tag_tid=14, entry_tid=19039], [object_number=94, tag_tid=14, entry_tid=19038]]], state=connected] 2024-04-06T09:41:13.532 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=95, tag_tid=14, entry_tid=18379], [object_number=94, tag_tid=14, entry_tid=18378], [object_number=93, tag_tid=14, entry_tid=18377], [object_number=92, tag_tid=14, entry_tid=18376]]], state=connected] 2024-04-06T09:41:13.536 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:41:13.538 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=95, tag_tid=14, entry_tid=18379' 2024-04-06T09:41:13.538 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=93, tag_tid=15, entry_tid=1' -a 'object_number=93, tag_tid=15, entry_tid=1' = 'object_number=95, tag_tid=14, entry_tid=18379' 2024-04-06T09:41:13.538 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=95, tag_tid=14, entry_tid=18379' '!=' 'object_number=92, tag_tid=14, entry_tid=18288' 2024-04-06T09:41:13.538 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:41:13.538 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=95, tag_tid=14, entry_tid=18379' = 'object_number=92, tag_tid=14, entry_tid=18288' 2024-04-06T09:41:13.538 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=95, tag_tid=14, entry_tid=18379' 2024-04-06T09:41:13.538 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=93, tag_tid=15, entry_tid=1' 2024-04-06T09:41:13.538 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:41:13.538 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:41:13.539 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=15 2024-04-06T09:41:13.540 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=95, tag_tid=14, entry_tid=18379' 2024-04-06T09:41:13.540 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:41:13.540 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:41:13.540 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=14 2024-04-06T09:41:13.541 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=93, tag_tid=15, entry_tid=1' 2024-04-06T09:41:13.541 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:41:13.541 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:41:13.542 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:41:13.543 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=95, tag_tid=14, entry_tid=18379' 2024-04-06T09:41:13.543 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:41:13.543 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:41:13.544 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=18379 2024-04-06T09:41:13.544 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 15 = 14 -a 1 -le 18379 2024-04-06T09:41:13.544 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:41:13.545 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:41:13.545 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:41:13.747 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:41:13.747 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:41:13.747 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:41:13.747 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:41:13.747 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:41:13.747 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:41:13.747 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:41:13.747 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:41:13.747 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:41:13.747 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:41:13.748 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:41:13.748 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:41:13.748 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:41:13.748 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:41:13.748 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:41:13.748 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:41:13.749 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:41:13.749 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:41:13.750 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:41:13.750 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:41:13.750 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:41:13.750 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:41:13.750 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:41:13.750 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:41:13.750 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:41:13.750 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:41:13.750 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:41:13.750 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:41:13.750 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:41:13.750 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:41:13.751 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:41:13.751 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:41:13.751 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:41:13.752 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:41:13.752 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:41:13.752 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:41:13.752 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:41:13.753 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:41:13.753 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:41:13.753 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:41:13.753 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:41:13.753 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:41:13.753 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:41:13.753 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:41:13.753 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:41:13.753 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:41:13.753 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:41:13.753 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:41:13.753 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:41:13.776 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:41:13.776 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:41:13.776 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:41:13.776 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:41:13.903 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:41:13.904 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:41:13.904 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:41:13.904 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:41:13.904 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:41:13.904 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:41:13.904 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:41:13.904 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:41:13.904 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:41:13.904 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:41:13.905 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:41:13.905 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:41:13.905 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:41:13.906 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:41:13.907 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:41:13.907 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:41:13.960 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 23 2024-04-06T09:41:13.960 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 23 2024-04-06T09:41:13.960 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:41:13.960 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=93, tag_tid=15, entry_tid=1], [object_number=92, tag_tid=15, entry_tid=0], [object_number=95, tag_tid=14, entry_tid=19039], [object_number=94, tag_tid=14, entry_tid=19038]]], state=connected] 2024-04-06T09:41:13.960 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=94, tag_tid=14, entry_tid=18478], [object_number=93, tag_tid=14, entry_tid=18477], [object_number=92, tag_tid=14, entry_tid=18476], [object_number=95, tag_tid=14, entry_tid=18475]]], state=connected] 2024-04-06T09:41:13.963 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:41:13.965 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=93, tag_tid=15, entry_tid=1' 2024-04-06T09:41:13.966 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:41:13.966 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:41:13.966 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:41:13.966 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:41:13.966 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:41:13.966 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:41:13.966 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:41:13.966 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:41:13.966 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:41:13.967 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:41:13.967 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:41:13.967 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:41:13.967 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:41:13.967 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:41:13.967 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:41:14.014 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 23 2024-04-06T09:41:14.014 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 23 2024-04-06T09:41:14.014 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:41:14.014 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=93, tag_tid=15, entry_tid=1], [object_number=92, tag_tid=15, entry_tid=0], [object_number=95, tag_tid=14, entry_tid=19039], [object_number=94, tag_tid=14, entry_tid=19038]]], state=connected] 2024-04-06T09:41:14.014 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=94, tag_tid=14, entry_tid=18478], [object_number=93, tag_tid=14, entry_tid=18477], [object_number=92, tag_tid=14, entry_tid=18476], [object_number=95, tag_tid=14, entry_tid=18475]]], state=connected] 2024-04-06T09:41:14.017 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:41:14.019 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=94, tag_tid=14, entry_tid=18478' 2024-04-06T09:41:14.019 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=93, tag_tid=15, entry_tid=1' -a 'object_number=93, tag_tid=15, entry_tid=1' = 'object_number=94, tag_tid=14, entry_tid=18478' 2024-04-06T09:41:14.019 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=94, tag_tid=14, entry_tid=18478' '!=' 'object_number=95, tag_tid=14, entry_tid=18379' 2024-04-06T09:41:14.019 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:41:14.019 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=94, tag_tid=14, entry_tid=18478' = 'object_number=95, tag_tid=14, entry_tid=18379' 2024-04-06T09:41:14.020 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=94, tag_tid=14, entry_tid=18478' 2024-04-06T09:41:14.020 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=93, tag_tid=15, entry_tid=1' 2024-04-06T09:41:14.020 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:41:14.020 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:41:14.021 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=15 2024-04-06T09:41:14.021 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=94, tag_tid=14, entry_tid=18478' 2024-04-06T09:41:14.021 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:41:14.022 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:41:14.022 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=14 2024-04-06T09:41:14.023 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=93, tag_tid=15, entry_tid=1' 2024-04-06T09:41:14.023 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:41:14.023 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:41:14.024 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:41:14.025 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=94, tag_tid=14, entry_tid=18478' 2024-04-06T09:41:14.025 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:41:14.025 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:41:14.026 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=18478 2024-04-06T09:41:14.026 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 15 = 14 -a 1 -le 18478 2024-04-06T09:41:14.026 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:41:14.027 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:41:14.027 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:41:14.228 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:41:14.228 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:41:14.228 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:41:14.228 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:41:14.228 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:41:14.229 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:41:14.229 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:41:14.229 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:41:14.229 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:41:14.229 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:41:14.229 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:41:14.229 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:41:14.229 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:41:14.229 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:41:14.229 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:41:14.229 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:41:14.230 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:41:14.232 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:41:14.232 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:41:14.232 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:41:14.232 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:41:14.232 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:41:14.233 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:41:14.233 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:41:14.233 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:41:14.233 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:41:14.233 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:41:14.233 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:41:14.233 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:41:14.233 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:41:14.233 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:41:14.233 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:41:14.233 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:41:14.235 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:41:14.235 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:41:14.235 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:41:14.235 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:41:14.235 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:41:14.235 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:41:14.235 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:41:14.235 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:41:14.235 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:41:14.235 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:41:14.235 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:41:14.236 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:41:14.236 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:41:14.236 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:41:14.236 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:41:14.236 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:41:14.260 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:41:14.261 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:41:14.261 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:41:14.261 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:41:14.392 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:41:14.393 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:41:14.393 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:41:14.393 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:41:14.394 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:41:14.394 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:41:14.394 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:41:14.394 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:41:14.394 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:41:14.394 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:41:14.394 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:41:14.394 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:41:14.395 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:41:14.395 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:41:14.396 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:41:14.396 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:41:14.448 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 23 2024-04-06T09:41:14.449 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 23 2024-04-06T09:41:14.449 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:41:14.449 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=93, tag_tid=15, entry_tid=1], [object_number=92, tag_tid=15, entry_tid=0], [object_number=95, tag_tid=14, entry_tid=19039], [object_number=94, tag_tid=14, entry_tid=19038]]], state=connected] 2024-04-06T09:41:14.449 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=92, tag_tid=14, entry_tid=18576], [object_number=95, tag_tid=14, entry_tid=18575], [object_number=94, tag_tid=14, entry_tid=18574], [object_number=93, tag_tid=14, entry_tid=18573]]], state=connected] 2024-04-06T09:41:14.452 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:41:14.453 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=93, tag_tid=15, entry_tid=1' 2024-04-06T09:41:14.455 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:41:14.455 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:41:14.455 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:41:14.455 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:41:14.455 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:41:14.455 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:41:14.455 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:41:14.455 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:41:14.455 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:41:14.455 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:41:14.455 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:41:14.455 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:41:14.456 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:41:14.456 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:41:14.456 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:41:14.503 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 23 2024-04-06T09:41:14.504 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 23 2024-04-06T09:41:14.504 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:41:14.504 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=93, tag_tid=15, entry_tid=1], [object_number=92, tag_tid=15, entry_tid=0], [object_number=95, tag_tid=14, entry_tid=19039], [object_number=94, tag_tid=14, entry_tid=19038]]], state=connected] 2024-04-06T09:41:14.504 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=92, tag_tid=14, entry_tid=18576], [object_number=95, tag_tid=14, entry_tid=18575], [object_number=94, tag_tid=14, entry_tid=18574], [object_number=93, tag_tid=14, entry_tid=18573]]], state=connected] 2024-04-06T09:41:14.507 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:41:14.509 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=92, tag_tid=14, entry_tid=18576' 2024-04-06T09:41:14.509 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=93, tag_tid=15, entry_tid=1' -a 'object_number=93, tag_tid=15, entry_tid=1' = 'object_number=92, tag_tid=14, entry_tid=18576' 2024-04-06T09:41:14.509 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=92, tag_tid=14, entry_tid=18576' '!=' 'object_number=94, tag_tid=14, entry_tid=18478' 2024-04-06T09:41:14.509 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:41:14.509 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=92, tag_tid=14, entry_tid=18576' = 'object_number=94, tag_tid=14, entry_tid=18478' 2024-04-06T09:41:14.509 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=92, tag_tid=14, entry_tid=18576' 2024-04-06T09:41:14.510 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=93, tag_tid=15, entry_tid=1' 2024-04-06T09:41:14.510 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:41:14.510 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:41:14.510 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=15 2024-04-06T09:41:14.511 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=92, tag_tid=14, entry_tid=18576' 2024-04-06T09:41:14.511 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:41:14.511 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:41:14.512 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=14 2024-04-06T09:41:14.513 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=93, tag_tid=15, entry_tid=1' 2024-04-06T09:41:14.513 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:41:14.513 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:41:14.514 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:41:14.514 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=92, tag_tid=14, entry_tid=18576' 2024-04-06T09:41:14.515 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:41:14.515 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:41:14.516 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=18576 2024-04-06T09:41:14.516 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 15 = 14 -a 1 -le 18576 2024-04-06T09:41:14.516 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:41:14.516 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:41:14.516 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:41:14.718 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:41:14.718 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:41:14.718 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:41:14.718 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:41:14.718 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:41:14.718 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:41:14.718 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:41:14.718 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:41:14.718 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:41:14.718 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:41:14.718 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:41:14.719 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:41:14.719 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:41:14.719 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:41:14.719 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:41:14.719 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:41:14.719 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:41:14.726 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:41:14.726 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:41:14.726 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:41:14.726 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:41:14.726 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:41:14.726 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:41:14.726 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:41:14.726 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:41:14.726 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:41:14.726 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:41:14.726 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:41:14.726 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:41:14.727 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:41:14.727 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:41:14.727 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:41:14.727 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:41:14.727 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:41:14.727 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:41:14.727 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:41:14.727 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:41:14.727 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:41:14.727 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:41:14.727 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:41:14.727 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:41:14.727 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:41:14.727 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:41:14.728 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:41:14.728 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:41:14.728 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:41:14.728 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:41:14.728 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:41:14.728 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:41:14.749 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:41:14.749 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:41:14.749 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:41:14.749 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:41:14.877 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:41:14.879 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:41:14.879 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:41:14.879 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:41:14.879 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:41:14.879 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:41:14.879 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:41:14.879 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:41:14.879 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:41:14.879 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:41:14.879 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:41:14.880 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:41:14.880 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:41:14.880 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:41:14.881 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:41:14.881 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:41:14.938 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 23 2024-04-06T09:41:14.938 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 23 2024-04-06T09:41:14.938 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:41:14.938 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=93, tag_tid=15, entry_tid=1], [object_number=92, tag_tid=15, entry_tid=0], [object_number=95, tag_tid=14, entry_tid=19039], [object_number=94, tag_tid=14, entry_tid=19038]]], state=connected] 2024-04-06T09:41:14.938 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=95, tag_tid=14, entry_tid=18679], [object_number=94, tag_tid=14, entry_tid=18678], [object_number=93, tag_tid=14, entry_tid=18677], [object_number=92, tag_tid=14, entry_tid=18676]]], state=connected] 2024-04-06T09:41:14.940 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:41:14.941 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=93, tag_tid=15, entry_tid=1' 2024-04-06T09:41:14.942 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:41:14.942 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:41:14.943 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:41:14.943 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:41:14.943 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:41:14.943 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:41:14.943 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:41:14.943 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:41:14.943 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:41:14.943 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:41:14.943 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:41:14.943 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:41:14.944 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:41:14.944 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:41:14.944 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:41:14.990 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 23 2024-04-06T09:41:14.991 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 23 2024-04-06T09:41:14.991 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:41:14.991 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=93, tag_tid=15, entry_tid=1], [object_number=92, tag_tid=15, entry_tid=0], [object_number=95, tag_tid=14, entry_tid=19039], [object_number=94, tag_tid=14, entry_tid=19038]]], state=connected] 2024-04-06T09:41:14.991 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=95, tag_tid=14, entry_tid=18679], [object_number=94, tag_tid=14, entry_tid=18678], [object_number=93, tag_tid=14, entry_tid=18677], [object_number=92, tag_tid=14, entry_tid=18676]]], state=connected] 2024-04-06T09:41:14.994 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:41:14.996 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=95, tag_tid=14, entry_tid=18679' 2024-04-06T09:41:14.996 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=93, tag_tid=15, entry_tid=1' -a 'object_number=93, tag_tid=15, entry_tid=1' = 'object_number=95, tag_tid=14, entry_tid=18679' 2024-04-06T09:41:14.996 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=95, tag_tid=14, entry_tid=18679' '!=' 'object_number=92, tag_tid=14, entry_tid=18576' 2024-04-06T09:41:14.996 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:41:14.996 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=95, tag_tid=14, entry_tid=18679' = 'object_number=92, tag_tid=14, entry_tid=18576' 2024-04-06T09:41:14.996 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=95, tag_tid=14, entry_tid=18679' 2024-04-06T09:41:14.996 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=93, tag_tid=15, entry_tid=1' 2024-04-06T09:41:14.997 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:41:14.997 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:41:14.997 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=15 2024-04-06T09:41:14.998 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=95, tag_tid=14, entry_tid=18679' 2024-04-06T09:41:14.998 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:41:14.998 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:41:14.999 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=14 2024-04-06T09:41:15.000 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=93, tag_tid=15, entry_tid=1' 2024-04-06T09:41:15.000 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:41:15.000 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:41:15.001 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:41:15.002 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=95, tag_tid=14, entry_tid=18679' 2024-04-06T09:41:15.002 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:41:15.002 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:41:15.003 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=18679 2024-04-06T09:41:15.003 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 15 = 14 -a 1 -le 18679 2024-04-06T09:41:15.003 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:41:15.003 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:41:15.003 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:41:15.205 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:41:15.205 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:41:15.205 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:41:15.205 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:41:15.205 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:41:15.205 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:41:15.205 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:41:15.206 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:41:15.206 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:41:15.206 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:41:15.206 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:41:15.206 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:41:15.206 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:41:15.206 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:41:15.206 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:41:15.207 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:41:15.207 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:41:15.208 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:41:15.208 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:41:15.208 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:41:15.208 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:41:15.209 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:41:15.209 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:41:15.209 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:41:15.209 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:41:15.209 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:41:15.209 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:41:15.209 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:41:15.209 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:41:15.209 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:41:15.209 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:41:15.209 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:41:15.209 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:41:15.210 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:41:15.210 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:41:15.210 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:41:15.211 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:41:15.211 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:41:15.211 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:41:15.211 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:41:15.211 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:41:15.211 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:41:15.211 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:41:15.211 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:41:15.211 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:41:15.211 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:41:15.211 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:41:15.211 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:41:15.211 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:41:15.235 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:41:15.236 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:41:15.236 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:41:15.236 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:41:15.359 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:41:15.360 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:41:15.360 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:41:15.360 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:41:15.360 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:41:15.360 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:41:15.361 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:41:15.361 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:41:15.361 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:41:15.361 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:41:15.361 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:41:15.362 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:41:15.362 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:41:15.363 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:41:15.364 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:41:15.364 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:41:15.389 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.0 with skip (0.6565325988276913) > weight (0.25) 2024-04-06T09:41:15.389 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.1 with skip (0.36206417817030023) > weight (0.25) 2024-04-06T09:41:15.389 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.2 with skip (0.35043784174937076) > weight (0.25) 2024-04-06T09:41:15.389 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.3 with skip (0.8815937538584828) > weight (0.25) 2024-04-06T09:41:15.389 INFO:tasks.rbd_mirror_thrash:waiting for 72 secs before thrashing 2024-04-06T09:41:15.414 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 23 2024-04-06T09:41:15.414 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 23 2024-04-06T09:41:15.414 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:41:15.414 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=93, tag_tid=15, entry_tid=1], [object_number=92, tag_tid=15, entry_tid=0], [object_number=95, tag_tid=14, entry_tid=19039], [object_number=94, tag_tid=14, entry_tid=19038]]], state=connected] 2024-04-06T09:41:15.415 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=92, tag_tid=14, entry_tid=18776], [object_number=95, tag_tid=14, entry_tid=18775], [object_number=94, tag_tid=14, entry_tid=18774], [object_number=93, tag_tid=14, entry_tid=18773]]], state=connected] 2024-04-06T09:41:15.418 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:41:15.419 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=93, tag_tid=15, entry_tid=1' 2024-04-06T09:41:15.420 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:41:15.420 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:41:15.420 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:41:15.420 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:41:15.420 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:41:15.420 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:41:15.421 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:41:15.421 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:41:15.421 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:41:15.421 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:41:15.421 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:41:15.421 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:41:15.421 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:41:15.421 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:41:15.422 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:41:15.467 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 23 2024-04-06T09:41:15.467 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 23 2024-04-06T09:41:15.467 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:41:15.467 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=93, tag_tid=15, entry_tid=1], [object_number=92, tag_tid=15, entry_tid=0], [object_number=95, tag_tid=14, entry_tid=19039], [object_number=94, tag_tid=14, entry_tid=19038]]], state=connected] 2024-04-06T09:41:15.468 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=92, tag_tid=14, entry_tid=18776], [object_number=95, tag_tid=14, entry_tid=18775], [object_number=94, tag_tid=14, entry_tid=18774], [object_number=93, tag_tid=14, entry_tid=18773]]], state=connected] 2024-04-06T09:41:15.471 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:41:15.473 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=92, tag_tid=14, entry_tid=18776' 2024-04-06T09:41:15.473 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=93, tag_tid=15, entry_tid=1' -a 'object_number=93, tag_tid=15, entry_tid=1' = 'object_number=92, tag_tid=14, entry_tid=18776' 2024-04-06T09:41:15.473 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=92, tag_tid=14, entry_tid=18776' '!=' 'object_number=95, tag_tid=14, entry_tid=18679' 2024-04-06T09:41:15.473 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:41:15.473 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=92, tag_tid=14, entry_tid=18776' = 'object_number=95, tag_tid=14, entry_tid=18679' 2024-04-06T09:41:15.473 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=92, tag_tid=14, entry_tid=18776' 2024-04-06T09:41:15.474 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=93, tag_tid=15, entry_tid=1' 2024-04-06T09:41:15.474 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:41:15.474 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:41:15.475 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=15 2024-04-06T09:41:15.475 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=92, tag_tid=14, entry_tid=18776' 2024-04-06T09:41:15.475 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:41:15.475 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:41:15.476 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=14 2024-04-06T09:41:15.476 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=93, tag_tid=15, entry_tid=1' 2024-04-06T09:41:15.476 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:41:15.477 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:41:15.477 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:41:15.478 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=92, tag_tid=14, entry_tid=18776' 2024-04-06T09:41:15.478 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:41:15.478 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:41:15.479 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=18776 2024-04-06T09:41:15.479 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 15 = 14 -a 1 -le 18776 2024-04-06T09:41:15.480 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:41:15.480 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:41:15.480 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:41:15.681 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:41:15.682 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:41:15.682 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:41:15.682 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:41:15.682 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:41:15.682 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:41:15.682 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:41:15.682 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:41:15.682 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:41:15.682 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:41:15.682 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:41:15.682 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:41:15.682 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:41:15.682 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:41:15.683 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:41:15.683 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:41:15.683 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:41:15.686 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:41:15.686 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:41:15.686 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:41:15.686 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:41:15.686 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:41:15.686 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:41:15.686 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:41:15.686 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:41:15.687 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:41:15.687 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:41:15.687 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:41:15.687 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:41:15.687 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:41:15.687 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:41:15.687 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:41:15.687 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:41:15.688 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:41:15.688 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:41:15.688 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:41:15.688 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:41:15.688 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:41:15.688 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:41:15.689 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:41:15.689 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:41:15.689 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:41:15.689 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:41:15.689 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:41:15.689 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:41:15.689 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:41:15.689 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:41:15.689 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:41:15.689 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:41:15.714 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:41:15.714 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:41:15.714 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:41:15.715 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:41:15.840 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:41:15.841 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:41:15.841 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:41:15.841 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:41:15.841 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:41:15.841 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:41:15.841 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:41:15.841 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:41:15.841 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:41:15.841 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:41:15.842 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:41:15.842 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:41:15.842 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:41:15.844 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:41:15.844 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:41:15.844 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:41:15.890 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 23 2024-04-06T09:41:15.891 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 23 2024-04-06T09:41:15.891 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:41:15.891 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=93, tag_tid=15, entry_tid=1], [object_number=92, tag_tid=15, entry_tid=0], [object_number=95, tag_tid=14, entry_tid=19039], [object_number=94, tag_tid=14, entry_tid=19038]]], state=connected] 2024-04-06T09:41:15.891 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=92, tag_tid=14, entry_tid=18868], [object_number=95, tag_tid=14, entry_tid=18867], [object_number=94, tag_tid=14, entry_tid=18866], [object_number=93, tag_tid=14, entry_tid=18865]]], state=connected] 2024-04-06T09:41:15.894 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:41:15.896 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=93, tag_tid=15, entry_tid=1' 2024-04-06T09:41:15.897 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:41:15.897 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:41:15.898 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:41:15.898 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:41:15.898 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:41:15.898 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:41:15.898 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:41:15.898 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:41:15.898 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:41:15.898 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:41:15.898 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:41:15.898 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:41:15.899 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:41:15.899 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:41:15.899 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:41:15.946 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 23 2024-04-06T09:41:15.946 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 23 2024-04-06T09:41:15.946 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:41:15.946 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=93, tag_tid=15, entry_tid=1], [object_number=92, tag_tid=15, entry_tid=0], [object_number=95, tag_tid=14, entry_tid=19039], [object_number=94, tag_tid=14, entry_tid=19038]]], state=connected] 2024-04-06T09:41:15.946 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=92, tag_tid=14, entry_tid=18868], [object_number=95, tag_tid=14, entry_tid=18867], [object_number=94, tag_tid=14, entry_tid=18866], [object_number=93, tag_tid=14, entry_tid=18865]]], state=connected] 2024-04-06T09:41:15.949 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:41:15.951 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=92, tag_tid=14, entry_tid=18868' 2024-04-06T09:41:15.951 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=93, tag_tid=15, entry_tid=1' -a 'object_number=93, tag_tid=15, entry_tid=1' = 'object_number=92, tag_tid=14, entry_tid=18868' 2024-04-06T09:41:15.951 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=92, tag_tid=14, entry_tid=18868' '!=' 'object_number=92, tag_tid=14, entry_tid=18776' 2024-04-06T09:41:15.951 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:41:15.951 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=92, tag_tid=14, entry_tid=18868' = 'object_number=92, tag_tid=14, entry_tid=18776' 2024-04-06T09:41:15.951 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=92, tag_tid=14, entry_tid=18868' 2024-04-06T09:41:15.952 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=93, tag_tid=15, entry_tid=1' 2024-04-06T09:41:15.952 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:41:15.952 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:41:15.952 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=15 2024-04-06T09:41:15.953 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=92, tag_tid=14, entry_tid=18868' 2024-04-06T09:41:15.953 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:41:15.953 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:41:15.954 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=14 2024-04-06T09:41:15.955 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=93, tag_tid=15, entry_tid=1' 2024-04-06T09:41:15.955 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:41:15.955 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:41:15.956 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:41:15.956 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=92, tag_tid=14, entry_tid=18868' 2024-04-06T09:41:15.956 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:41:15.956 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:41:15.957 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=18868 2024-04-06T09:41:15.958 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 15 = 14 -a 1 -le 18868 2024-04-06T09:41:15.958 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:41:15.958 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:41:15.958 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:41:16.159 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:41:16.160 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:41:16.160 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:41:16.160 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:41:16.160 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:41:16.160 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:41:16.160 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:41:16.160 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:41:16.160 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:41:16.160 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:41:16.160 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:41:16.160 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:41:16.160 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:41:16.160 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:41:16.160 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:41:16.161 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:41:16.162 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:41:16.167 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:41:16.168 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:41:16.168 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:41:16.168 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:41:16.168 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:41:16.168 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:41:16.168 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:41:16.168 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:41:16.168 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:41:16.168 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:41:16.168 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:41:16.168 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:41:16.168 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:41:16.168 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:41:16.168 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:41:16.168 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:41:16.169 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:41:16.169 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:41:16.169 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:41:16.169 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:41:16.169 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:41:16.169 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:41:16.169 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:41:16.169 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:41:16.169 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:41:16.169 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:41:16.169 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:41:16.169 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:41:16.169 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:41:16.169 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:41:16.169 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:41:16.170 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:41:16.192 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:41:16.192 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:41:16.192 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:41:16.192 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:41:16.317 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:41:16.317 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:41:16.318 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:41:16.318 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:41:16.318 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:41:16.318 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:41:16.318 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:41:16.318 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:41:16.318 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:41:16.318 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:41:16.319 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:41:16.319 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:41:16.319 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:41:16.320 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:41:16.321 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:41:16.321 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:41:16.373 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 23 2024-04-06T09:41:16.374 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 23 2024-04-06T09:41:16.374 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:41:16.374 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=93, tag_tid=15, entry_tid=1], [object_number=92, tag_tid=15, entry_tid=0], [object_number=95, tag_tid=14, entry_tid=19039], [object_number=94, tag_tid=14, entry_tid=19038]]], state=connected] 2024-04-06T09:41:16.374 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=94, tag_tid=14, entry_tid=18962], [object_number=93, tag_tid=14, entry_tid=18961], [object_number=92, tag_tid=14, entry_tid=18960], [object_number=95, tag_tid=14, entry_tid=18959]]], state=connected] 2024-04-06T09:41:16.377 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:41:16.378 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=93, tag_tid=15, entry_tid=1' 2024-04-06T09:41:16.380 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:41:16.380 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:41:16.380 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:41:16.380 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:41:16.380 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:41:16.380 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:41:16.380 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:41:16.380 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:41:16.381 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:41:16.381 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:41:16.381 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:41:16.381 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:41:16.381 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:41:16.381 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:41:16.382 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:41:16.428 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 23 2024-04-06T09:41:16.428 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 23 2024-04-06T09:41:16.428 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:41:16.428 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=93, tag_tid=15, entry_tid=1], [object_number=92, tag_tid=15, entry_tid=0], [object_number=95, tag_tid=14, entry_tid=19039], [object_number=94, tag_tid=14, entry_tid=19038]]], state=connected] 2024-04-06T09:41:16.428 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=94, tag_tid=14, entry_tid=18962], [object_number=93, tag_tid=14, entry_tid=18961], [object_number=92, tag_tid=14, entry_tid=18960], [object_number=95, tag_tid=14, entry_tid=18959]]], state=connected] 2024-04-06T09:41:16.432 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:41:16.434 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=94, tag_tid=14, entry_tid=18962' 2024-04-06T09:41:16.434 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=93, tag_tid=15, entry_tid=1' -a 'object_number=93, tag_tid=15, entry_tid=1' = 'object_number=94, tag_tid=14, entry_tid=18962' 2024-04-06T09:41:16.434 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=94, tag_tid=14, entry_tid=18962' '!=' 'object_number=92, tag_tid=14, entry_tid=18868' 2024-04-06T09:41:16.434 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:41:16.434 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=94, tag_tid=14, entry_tid=18962' = 'object_number=92, tag_tid=14, entry_tid=18868' 2024-04-06T09:41:16.434 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=94, tag_tid=14, entry_tid=18962' 2024-04-06T09:41:16.434 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=93, tag_tid=15, entry_tid=1' 2024-04-06T09:41:16.435 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:41:16.435 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:41:16.435 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=15 2024-04-06T09:41:16.436 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=94, tag_tid=14, entry_tid=18962' 2024-04-06T09:41:16.436 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:41:16.436 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:41:16.437 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=14 2024-04-06T09:41:16.438 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=93, tag_tid=15, entry_tid=1' 2024-04-06T09:41:16.438 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:41:16.438 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:41:16.439 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:41:16.439 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=94, tag_tid=14, entry_tid=18962' 2024-04-06T09:41:16.439 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:41:16.440 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:41:16.441 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=18962 2024-04-06T09:41:16.441 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 15 = 14 -a 1 -le 18962 2024-04-06T09:41:16.441 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:41:16.441 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:41:16.441 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:41:16.643 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:41:16.643 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:41:16.643 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:41:16.643 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:41:16.643 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:41:16.643 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:41:16.643 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:41:16.643 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:41:16.643 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:41:16.644 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:41:16.644 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:41:16.644 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:41:16.644 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:41:16.644 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:41:16.644 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:41:16.644 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:41:16.644 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:41:16.645 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:41:16.646 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:41:16.648 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:41:16.648 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:41:16.648 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:41:16.648 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:41:16.648 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:41:16.648 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:41:16.648 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:41:16.648 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:41:16.648 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:41:16.648 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:41:16.648 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:41:16.649 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:41:16.649 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:41:16.649 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:41:16.649 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:41:16.649 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:41:16.649 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:41:16.649 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:41:16.649 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:41:16.649 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:41:16.649 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:41:16.649 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:41:16.649 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:41:16.649 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:41:16.649 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:41:16.650 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:41:16.650 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:41:16.650 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:41:16.650 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:41:16.650 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:41:16.674 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:41:16.674 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:41:16.674 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:41:16.674 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:41:16.800 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:41:16.801 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:41:16.801 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:41:16.801 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:41:16.801 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:41:16.801 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:41:16.801 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:41:16.801 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:41:16.801 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:41:16.801 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:41:16.802 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:41:16.802 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:41:16.802 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:41:16.803 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:41:16.804 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:41:16.804 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:41:16.856 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 23 2024-04-06T09:41:16.856 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 23 2024-04-06T09:41:16.856 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:41:16.856 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=93, tag_tid=15, entry_tid=1], [object_number=92, tag_tid=15, entry_tid=0], [object_number=95, tag_tid=14, entry_tid=19039], [object_number=94, tag_tid=14, entry_tid=19038]]], state=connected] 2024-04-06T09:41:16.856 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=93, tag_tid=14, entry_tid=19041], [object_number=92, tag_tid=14, entry_tid=19040], [object_number=95, tag_tid=14, entry_tid=19039], [object_number=94, tag_tid=14, entry_tid=19038]]], state=connected] 2024-04-06T09:41:16.859 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:41:16.861 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=93, tag_tid=15, entry_tid=1' 2024-04-06T09:41:16.862 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:41:16.862 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:41:16.862 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:41:16.862 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:41:16.862 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:41:16.862 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:41:16.862 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:41:16.862 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:41:16.862 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:41:16.863 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:41:16.863 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:41:16.863 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:41:16.863 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:41:16.863 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:41:16.863 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:41:16.910 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 23 2024-04-06T09:41:16.910 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 23 2024-04-06T09:41:16.910 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:41:16.911 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=93, tag_tid=15, entry_tid=1], [object_number=92, tag_tid=15, entry_tid=0], [object_number=95, tag_tid=14, entry_tid=19039], [object_number=94, tag_tid=14, entry_tid=19038]]], state=connected] 2024-04-06T09:41:16.911 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=93, tag_tid=14, entry_tid=19041], [object_number=92, tag_tid=14, entry_tid=19040], [object_number=95, tag_tid=14, entry_tid=19039], [object_number=94, tag_tid=14, entry_tid=19038]]], state=connected] 2024-04-06T09:41:16.914 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:41:16.916 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=93, tag_tid=14, entry_tid=19041' 2024-04-06T09:41:16.916 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=93, tag_tid=15, entry_tid=1' -a 'object_number=93, tag_tid=15, entry_tid=1' = 'object_number=93, tag_tid=14, entry_tid=19041' 2024-04-06T09:41:16.916 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=93, tag_tid=14, entry_tid=19041' '!=' 'object_number=94, tag_tid=14, entry_tid=18962' 2024-04-06T09:41:16.916 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:41:16.916 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=93, tag_tid=14, entry_tid=19041' = 'object_number=94, tag_tid=14, entry_tid=18962' 2024-04-06T09:41:16.916 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=93, tag_tid=14, entry_tid=19041' 2024-04-06T09:41:16.917 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=93, tag_tid=15, entry_tid=1' 2024-04-06T09:41:16.917 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:41:16.917 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:41:16.917 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=15 2024-04-06T09:41:16.918 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=93, tag_tid=14, entry_tid=19041' 2024-04-06T09:41:16.918 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:41:16.918 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:41:16.919 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=14 2024-04-06T09:41:16.920 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=93, tag_tid=15, entry_tid=1' 2024-04-06T09:41:16.920 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:41:16.920 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:41:16.921 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:41:16.921 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=93, tag_tid=14, entry_tid=19041' 2024-04-06T09:41:16.921 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:41:16.921 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:41:16.923 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=19041 2024-04-06T09:41:16.923 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 15 = 14 -a 1 -le 19041 2024-04-06T09:41:16.923 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:41:16.923 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:41:16.923 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:41:17.124 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:41:17.124 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:41:17.125 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:41:17.125 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:41:17.125 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:41:17.125 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:41:17.125 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:41:17.125 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:41:17.125 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:41:17.125 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:41:17.125 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:41:17.125 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:41:17.125 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:41:17.125 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:41:17.125 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:41:17.126 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:41:17.127 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:41:17.128 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:41:17.129 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:41:17.129 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:41:17.129 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:41:17.129 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:41:17.129 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:41:17.129 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:41:17.129 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:41:17.129 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:41:17.129 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:41:17.129 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:41:17.129 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:41:17.129 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:41:17.129 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:41:17.130 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:41:17.130 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:41:17.130 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:41:17.131 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:41:17.131 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:41:17.131 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:41:17.131 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:41:17.131 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:41:17.131 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:41:17.131 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:41:17.131 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:41:17.131 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:41:17.131 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:41:17.131 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:41:17.131 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:41:17.131 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:41:17.131 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:41:17.132 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:41:17.156 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:41:17.157 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:41:17.157 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:41:17.157 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:41:17.276 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:41:17.277 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:41:17.278 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:41:17.278 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:41:17.278 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:41:17.278 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:41:17.278 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:41:17.278 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:41:17.278 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:41:17.278 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:41:17.279 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:41:17.279 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:41:17.279 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:41:17.281 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:41:17.281 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:41:17.281 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:41:17.328 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 23 2024-04-06T09:41:17.328 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 23 2024-04-06T09:41:17.328 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:41:17.328 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=93, tag_tid=15, entry_tid=1], [object_number=92, tag_tid=15, entry_tid=0], [object_number=95, tag_tid=14, entry_tid=19039], [object_number=94, tag_tid=14, entry_tid=19038]]], state=connected] 2024-04-06T09:41:17.328 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=93, tag_tid=14, entry_tid=19041], [object_number=92, tag_tid=14, entry_tid=19040], [object_number=95, tag_tid=14, entry_tid=19039], [object_number=94, tag_tid=14, entry_tid=19038]]], state=connected] 2024-04-06T09:41:17.331 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:41:17.333 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=93, tag_tid=15, entry_tid=1' 2024-04-06T09:41:17.334 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:41:17.334 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:41:17.334 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:41:17.334 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:41:17.334 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:41:17.334 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:41:17.334 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:41:17.334 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:41:17.335 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:41:17.335 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:41:17.335 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:41:17.335 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:41:17.335 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:41:17.336 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:41:17.336 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:41:17.381 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 23 2024-04-06T09:41:17.381 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 23 2024-04-06T09:41:17.381 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:41:17.381 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=93, tag_tid=15, entry_tid=1], [object_number=92, tag_tid=15, entry_tid=0], [object_number=95, tag_tid=14, entry_tid=19039], [object_number=94, tag_tid=14, entry_tid=19038]]], state=connected] 2024-04-06T09:41:17.381 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=93, tag_tid=14, entry_tid=19041], [object_number=92, tag_tid=14, entry_tid=19040], [object_number=95, tag_tid=14, entry_tid=19039], [object_number=94, tag_tid=14, entry_tid=19038]]], state=connected] 2024-04-06T09:41:17.384 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:41:17.386 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=93, tag_tid=14, entry_tid=19041' 2024-04-06T09:41:17.386 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=93, tag_tid=15, entry_tid=1' -a 'object_number=93, tag_tid=15, entry_tid=1' = 'object_number=93, tag_tid=14, entry_tid=19041' 2024-04-06T09:41:17.386 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=93, tag_tid=14, entry_tid=19041' '!=' 'object_number=93, tag_tid=14, entry_tid=19041' 2024-04-06T09:41:17.386 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:41:17.386 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.4 2024-04-06T09:41:17.788 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:41:17.788 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:41:17.788 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:41:17.788 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:41:17.788 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:41:17.788 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:41:17.789 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:41:17.789 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:41:17.789 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:41:17.789 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:41:17.789 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:41:17.789 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:41:17.789 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:41:17.789 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:41:17.789 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:41:17.789 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:41:17.790 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:41:17.792 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:41:17.792 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:41:17.792 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:41:17.792 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:41:17.793 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:41:17.793 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:41:17.793 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:41:17.793 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:41:17.793 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:41:17.793 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:41:17.793 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:41:17.793 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:41:17.793 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:41:17.793 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:41:17.793 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:41:17.793 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:41:17.794 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:41:17.794 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:41:17.794 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:41:17.794 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:41:17.794 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:41:17.795 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:41:17.795 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:41:17.795 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:41:17.795 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:41:17.795 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:41:17.795 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:41:17.795 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:41:17.795 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:41:17.795 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:41:17.795 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:41:17.795 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:41:17.820 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:41:17.820 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:41:17.820 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:41:17.820 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:41:17.939 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:41:17.941 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:41:17.941 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:41:17.941 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:41:17.941 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:41:17.941 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:41:17.941 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:41:17.941 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:41:17.941 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:41:17.941 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:41:17.942 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:41:17.942 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:41:17.942 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:41:17.943 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:41:17.944 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:41:17.944 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:41:17.991 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 23 2024-04-06T09:41:17.991 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 23 2024-04-06T09:41:17.991 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:41:17.991 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=93, tag_tid=15, entry_tid=1], [object_number=92, tag_tid=15, entry_tid=0], [object_number=95, tag_tid=14, entry_tid=19039], [object_number=94, tag_tid=14, entry_tid=19038]]], state=connected] 2024-04-06T09:41:17.991 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=93, tag_tid=14, entry_tid=19041], [object_number=92, tag_tid=14, entry_tid=19040], [object_number=95, tag_tid=14, entry_tid=19039], [object_number=94, tag_tid=14, entry_tid=19038]]], state=connected] 2024-04-06T09:41:17.994 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:41:17.995 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=93, tag_tid=15, entry_tid=1' 2024-04-06T09:41:17.997 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:41:17.997 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:41:17.997 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:41:17.997 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:41:17.997 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:41:17.997 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:41:17.997 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:41:17.997 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:41:17.998 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:41:17.998 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:41:17.998 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:41:17.998 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:41:17.998 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:41:17.999 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:41:17.999 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:41:18.043 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 23 2024-04-06T09:41:18.043 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 23 2024-04-06T09:41:18.044 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:41:18.044 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=93, tag_tid=15, entry_tid=1], [object_number=92, tag_tid=15, entry_tid=0], [object_number=95, tag_tid=14, entry_tid=19039], [object_number=94, tag_tid=14, entry_tid=19038]]], state=connected] 2024-04-06T09:41:18.044 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=93, tag_tid=14, entry_tid=19041], [object_number=92, tag_tid=14, entry_tid=19040], [object_number=95, tag_tid=14, entry_tid=19039], [object_number=94, tag_tid=14, entry_tid=19038]]], state=connected] 2024-04-06T09:41:18.047 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:41:18.049 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=93, tag_tid=14, entry_tid=19041' 2024-04-06T09:41:18.049 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=93, tag_tid=15, entry_tid=1' -a 'object_number=93, tag_tid=15, entry_tid=1' = 'object_number=93, tag_tid=14, entry_tid=19041' 2024-04-06T09:41:18.049 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=93, tag_tid=14, entry_tid=19041' '!=' 'object_number=93, tag_tid=14, entry_tid=19041' 2024-04-06T09:41:18.049 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:41:18.049 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.8 2024-04-06T09:41:18.850 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:41:18.851 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:41:18.851 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:41:18.851 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:41:18.851 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:41:18.851 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:41:18.851 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:41:18.851 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:41:18.851 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:41:18.851 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:41:18.851 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:41:18.851 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:41:18.852 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:41:18.852 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:41:18.852 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:41:18.852 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:41:18.852 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:41:18.854 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:41:18.854 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:41:18.854 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:41:18.854 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:41:18.854 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:41:18.854 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:41:18.854 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:41:18.854 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:41:18.854 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:41:18.854 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:41:18.854 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:41:18.855 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:41:18.855 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:41:18.855 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:41:18.855 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:41:18.855 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:41:18.856 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:41:18.856 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:41:18.857 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:41:18.857 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:41:18.857 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:41:18.857 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:41:18.857 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:41:18.857 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:41:18.857 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:41:18.857 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:41:18.857 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:41:18.858 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:41:18.858 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:41:18.858 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:41:18.858 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:41:18.858 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:41:18.881 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:41:18.881 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:41:18.881 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:41:18.881 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:41:19.004 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:41:19.005 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:41:19.005 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:41:19.006 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:41:19.006 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:41:19.006 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:41:19.006 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:41:19.006 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:41:19.006 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:41:19.006 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:41:19.006 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:41:19.007 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:41:19.007 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:41:19.008 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:41:19.008 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:41:19.008 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:41:19.054 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 23 2024-04-06T09:41:19.054 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 23 2024-04-06T09:41:19.055 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:41:19.055 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=93, tag_tid=15, entry_tid=1], [object_number=92, tag_tid=15, entry_tid=0], [object_number=95, tag_tid=14, entry_tid=19039], [object_number=94, tag_tid=14, entry_tid=19038]]], state=connected] 2024-04-06T09:41:19.055 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=93, tag_tid=14, entry_tid=19041], [object_number=92, tag_tid=14, entry_tid=19040], [object_number=95, tag_tid=14, entry_tid=19039], [object_number=94, tag_tid=14, entry_tid=19038]]], state=connected] 2024-04-06T09:41:19.058 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:41:19.059 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=93, tag_tid=15, entry_tid=1' 2024-04-06T09:41:19.060 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:41:19.060 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:41:19.060 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:41:19.060 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:41:19.061 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:41:19.061 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:41:19.061 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:41:19.061 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:41:19.061 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:41:19.061 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:41:19.061 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:41:19.061 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:41:19.062 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:41:19.062 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:41:19.062 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:41:19.107 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 23 2024-04-06T09:41:19.107 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 23 2024-04-06T09:41:19.107 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:41:19.107 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=93, tag_tid=15, entry_tid=1], [object_number=92, tag_tid=15, entry_tid=0], [object_number=95, tag_tid=14, entry_tid=19039], [object_number=94, tag_tid=14, entry_tid=19038]]], state=connected] 2024-04-06T09:41:19.107 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=93, tag_tid=14, entry_tid=19041], [object_number=92, tag_tid=14, entry_tid=19040], [object_number=95, tag_tid=14, entry_tid=19039], [object_number=94, tag_tid=14, entry_tid=19038]]], state=connected] 2024-04-06T09:41:19.111 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:41:19.112 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=93, tag_tid=14, entry_tid=19041' 2024-04-06T09:41:19.112 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=93, tag_tid=15, entry_tid=1' -a 'object_number=93, tag_tid=15, entry_tid=1' = 'object_number=93, tag_tid=14, entry_tid=19041' 2024-04-06T09:41:19.113 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=93, tag_tid=14, entry_tid=19041' '!=' 'object_number=93, tag_tid=14, entry_tid=19041' 2024-04-06T09:41:19.113 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:41:19.113 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 1.6 2024-04-06T09:41:20.714 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:41:20.715 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:41:20.715 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:41:20.715 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:41:20.715 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:41:20.715 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:41:20.715 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:41:20.715 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:41:20.715 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:41:20.715 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:41:20.715 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:41:20.715 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:41:20.715 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:41:20.716 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:41:20.716 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:41:20.716 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:41:20.716 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:41:20.718 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:41:20.718 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:41:20.718 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:41:20.718 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:41:20.719 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:41:20.719 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:41:20.719 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:41:20.719 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:41:20.719 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:41:20.719 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:41:20.719 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:41:20.719 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:41:20.719 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:41:20.719 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:41:20.719 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:41:20.719 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:41:20.720 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:41:20.720 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:41:20.720 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:41:20.720 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:41:20.721 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:41:20.721 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:41:20.721 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:41:20.721 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:41:20.721 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:41:20.721 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:41:20.721 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:41:20.721 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:41:20.721 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:41:20.721 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:41:20.721 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:41:20.721 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:41:20.745 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:41:20.745 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:41:20.745 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:41:20.745 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:41:20.865 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:41:20.867 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:41:20.867 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:41:20.867 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:41:20.867 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:41:20.867 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:41:20.867 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:41:20.867 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:41:20.867 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:41:20.867 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:41:20.868 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:41:20.868 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:41:20.868 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:41:20.869 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:41:20.870 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:41:20.870 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:41:20.921 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 23 2024-04-06T09:41:20.921 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 23 2024-04-06T09:41:20.921 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:41:20.921 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=93, tag_tid=15, entry_tid=1], [object_number=92, tag_tid=15, entry_tid=0], [object_number=95, tag_tid=14, entry_tid=19039], [object_number=94, tag_tid=14, entry_tid=19038]]], state=connected] 2024-04-06T09:41:20.921 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=93, tag_tid=14, entry_tid=19041], [object_number=92, tag_tid=14, entry_tid=19040], [object_number=95, tag_tid=14, entry_tid=19039], [object_number=94, tag_tid=14, entry_tid=19038]]], state=connected] 2024-04-06T09:41:20.924 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:41:20.926 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=93, tag_tid=15, entry_tid=1' 2024-04-06T09:41:20.927 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:41:20.927 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:41:20.927 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:41:20.927 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:41:20.927 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:41:20.927 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:41:20.927 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:41:20.927 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:41:20.927 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:41:20.927 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:41:20.928 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:41:20.928 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:41:20.928 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:41:20.928 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:41:20.928 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:41:20.974 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 23 2024-04-06T09:41:20.974 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 23 2024-04-06T09:41:20.974 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:41:20.974 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=93, tag_tid=15, entry_tid=1], [object_number=92, tag_tid=15, entry_tid=0], [object_number=95, tag_tid=14, entry_tid=19039], [object_number=94, tag_tid=14, entry_tid=19038]]], state=connected] 2024-04-06T09:41:20.974 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=93, tag_tid=14, entry_tid=19041], [object_number=92, tag_tid=14, entry_tid=19040], [object_number=95, tag_tid=14, entry_tid=19039], [object_number=94, tag_tid=14, entry_tid=19038]]], state=connected] 2024-04-06T09:41:20.977 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:41:20.979 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=93, tag_tid=14, entry_tid=19041' 2024-04-06T09:41:20.979 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=93, tag_tid=15, entry_tid=1' -a 'object_number=93, tag_tid=15, entry_tid=1' = 'object_number=93, tag_tid=14, entry_tid=19041' 2024-04-06T09:41:20.979 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=93, tag_tid=14, entry_tid=19041' '!=' 'object_number=93, tag_tid=14, entry_tid=19041' 2024-04-06T09:41:20.979 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:41:20.979 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 2 2024-04-06T09:41:22.981 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:41:22.981 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:41:22.981 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:41:22.981 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:41:22.982 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:41:22.982 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:41:22.982 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:41:22.982 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:41:22.982 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:41:22.982 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:41:22.982 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:41:22.982 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:41:22.982 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:41:22.982 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:41:22.982 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:41:22.983 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:41:22.983 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:41:22.985 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:41:22.985 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:41:22.985 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:41:22.985 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:41:22.986 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:41:22.986 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:41:22.986 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:41:22.986 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:41:22.986 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:41:22.986 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:41:22.986 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:41:22.986 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:41:22.986 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:41:22.986 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:41:22.986 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:41:22.986 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:41:22.986 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:41:22.987 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:41:22.987 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:41:22.987 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:41:22.987 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:41:22.987 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:41:22.988 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:41:22.988 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:41:22.988 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:41:22.988 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:41:22.988 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:41:22.988 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:41:22.988 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:41:22.988 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:41:22.988 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:41:22.989 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:41:23.010 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:41:23.010 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:41:23.011 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:41:23.011 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:41:23.132 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:41:23.133 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:41:23.134 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:41:23.134 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:41:23.134 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:41:23.134 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:41:23.134 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:41:23.134 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:41:23.134 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:41:23.134 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:41:23.135 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:41:23.135 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:41:23.135 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:41:23.136 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:41:23.136 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:41:23.136 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:41:23.181 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 23 2024-04-06T09:41:23.181 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 23 2024-04-06T09:41:23.182 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:41:23.182 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=93, tag_tid=15, entry_tid=1], [object_number=92, tag_tid=15, entry_tid=0], [object_number=95, tag_tid=14, entry_tid=19039], [object_number=94, tag_tid=14, entry_tid=19038]]], state=connected] 2024-04-06T09:41:23.182 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=93, tag_tid=15, entry_tid=1], [object_number=92, tag_tid=15, entry_tid=0], [object_number=95, tag_tid=14, entry_tid=19039], [object_number=94, tag_tid=14, entry_tid=19038]]], state=connected] 2024-04-06T09:41:23.185 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:41:23.187 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=93, tag_tid=15, entry_tid=1' 2024-04-06T09:41:23.187 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:41:23.187 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:41:23.187 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:41:23.187 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:41:23.187 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:41:23.187 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:41:23.188 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:41:23.188 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:41:23.188 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:41:23.188 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:41:23.188 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:41:23.188 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:41:23.189 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:41:23.189 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:41:23.189 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:41:23.236 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 23 2024-04-06T09:41:23.236 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 23 2024-04-06T09:41:23.236 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:41:23.236 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=93, tag_tid=15, entry_tid=1], [object_number=92, tag_tid=15, entry_tid=0], [object_number=95, tag_tid=14, entry_tid=19039], [object_number=94, tag_tid=14, entry_tid=19038]]], state=connected] 2024-04-06T09:41:23.236 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=93, tag_tid=15, entry_tid=1], [object_number=92, tag_tid=15, entry_tid=0], [object_number=95, tag_tid=14, entry_tid=19039], [object_number=94, tag_tid=14, entry_tid=19038]]], state=connected] 2024-04-06T09:41:23.239 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:41:23.241 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=93, tag_tid=15, entry_tid=1' 2024-04-06T09:41:23.241 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=93, tag_tid=15, entry_tid=1' -a 'object_number=93, tag_tid=15, entry_tid=1' = 'object_number=93, tag_tid=15, entry_tid=1' 2024-04-06T09:41:23.241 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:41:23.242 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_snap_present cluster1 mirror test snap5 2024-04-06T09:41:23.242 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:41:23.242 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:41:23.242 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:41:23.242 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local snap_name=snap5 2024-04-06T09:41:23.242 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local s 2024-04-06T09:41:23.242 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-04-06T09:41:23.242 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 1 2024-04-06T09:41:24.243 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 info mirror/test@snap5 2024-04-06T09:41:24.314 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:rbd image 'test': 2024-04-06T09:41:24.314 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: size 512 MiB in 128 objects 2024-04-06T09:41:24.314 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: order 22 (4 MiB objects) 2024-04-06T09:41:24.314 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: snapshot_count: 9 2024-04-06T09:41:24.314 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: id: 10a56da0b207 2024-04-06T09:41:24.314 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: block_name_prefix: rbd_data.10a56da0b207 2024-04-06T09:41:24.314 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: format: 2 2024-04-06T09:41:24.314 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: features: layering, exclusive-lock, journaling 2024-04-06T09:41:24.314 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: op_features: 2024-04-06T09:41:24.314 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: flags: 2024-04-06T09:41:24.314 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: create_timestamp: Sat Apr 6 09:31:59 2024 2024-04-06T09:41:24.315 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: access_timestamp: Sat Apr 6 09:39:21 2024 2024-04-06T09:41:24.315 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: modify_timestamp: Sat Apr 6 09:41:07 2024 2024-04-06T09:41:24.315 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: protected: False 2024-04-06T09:41:24.315 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: journal: 10a56da0b207 2024-04-06T09:41:24.315 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: mirroring state: enabled 2024-04-06T09:41:24.315 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: mirroring mode: journal 2024-04-06T09:41:24.315 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: mirroring global id: 8ad5dedc-b3e0-425d-8993-b3e2231fc5ca 2024-04-06T09:41:24.315 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: mirroring primary: false 2024-04-06T09:41:24.319 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:41:24.319 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n snap4-clean ']' 2024-04-06T09:41:24.319 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ compare_image_snaps mirror test snap4-clean 2024-04-06T09:41:24.319 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:41:24.319 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:41:24.319 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local snap_name=snap4-clean 2024-04-06T09:41:24.319 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local ret=0 2024-04-06T09:41:24.320 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local rmt_export=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.export 2024-04-06T09:41:24.320 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loc_export=/tmp/tmp.BhW6xKmROU/cluster1-mirror-test.export 2024-04-06T09:41:24.320 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rm -f /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.export /tmp/tmp.BhW6xKmROU/cluster1-mirror-test.export 2024-04-06T09:41:24.320 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster2 -p mirror export test@snap4-clean /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.export 2024-04-06T09:41:24.908 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-04-06T09:41:24.924 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 -p mirror export test@snap4-clean /tmp/tmp.BhW6xKmROU/cluster1-mirror-test.export 2024-04-06T09:41:26.294 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-04-06T09:41:26.312 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmp /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.export /tmp/tmp.BhW6xKmROU/cluster1-mirror-test.export 2024-04-06T09:41:26.603 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rm -f /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.export /tmp/tmp.BhW6xKmROU/cluster1-mirror-test.export 2024-04-06T09:41:26.739 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:41:26.739 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ compare_image_snaps mirror test snap5 2024-04-06T09:41:26.739 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:41:26.739 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:41:26.739 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local snap_name=snap5 2024-04-06T09:41:26.739 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local ret=0 2024-04-06T09:41:26.739 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local rmt_export=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.export 2024-04-06T09:41:26.739 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loc_export=/tmp/tmp.BhW6xKmROU/cluster1-mirror-test.export 2024-04-06T09:41:26.739 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rm -f /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.export /tmp/tmp.BhW6xKmROU/cluster1-mirror-test.export 2024-04-06T09:41:26.740 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster2 -p mirror export test@snap5 /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.export 2024-04-06T09:41:27.362 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-04-06T09:41:27.377 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 -p mirror export test@snap5 /tmp/tmp.BhW6xKmROU/cluster1-mirror-test.export 2024-04-06T09:41:28.923 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-04-06T09:41:28.947 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmp /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.export /tmp/tmp.BhW6xKmROU/cluster1-mirror-test.export 2024-04-06T09:41:29.230 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rm -f /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.export /tmp/tmp.BhW6xKmROU/cluster1-mirror-test.export 2024-04-06T09:41:29.365 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:41:29.365 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ clean_snap_name=snap5-clean 2024-04-06T09:41:29.366 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ create_snap cluster2 mirror test snap5-clean 2024-04-06T09:41:29.366 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-04-06T09:41:29.366 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:41:29.366 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:41:29.366 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local snap_name=snap5-clean 2024-04-06T09:41:29.366 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap5-clean --debug-rbd=20 --debug-journaler=20 2024-04-06T09:41:29.625 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in `seq 1 10` 2024-04-06T09:41:29.625 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ stress_write_image cluster2 mirror test 2024-04-06T09:41:29.625 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-04-06T09:41:29.625 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:41:29.625 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:41:29.626 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-04-06T09:41:29.628 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local duration=13 2024-04-06T09:41:29.628 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set +e 2024-04-06T09:41:29.628 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ timeout 13s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror test --debug-rbd=20 --debug-journaler=20 2024-04-06T09:41:31.467 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:100 IOs, 67733504 bytes 2024-04-06T09:41:33.730 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:200 IOs, 123881472 bytes 2024-04-06T09:41:34.930 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:300 IOs, 151723008 bytes 2024-04-06T09:41:36.036 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:400 IOs, 185883648 bytes 2024-04-06T09:41:37.544 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:500 IOs, 217070592 bytes 2024-04-06T09:41:38.277 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:600 IOs, 230272000 bytes 2024-04-06T09:41:39.604 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:700 IOs, 257334272 bytes 2024-04-06T09:41:42.642 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ error_code=124 2024-04-06T09:41:42.642 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set -e 2024-04-06T09:41:42.642 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 124 -eq 124 ']' 2024-04-06T09:41:42.642 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:41:42.642 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+replaying 2024-04-06T09:41:42.643 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:41:42.643 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:41:42.643 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:41:42.643 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+replaying 2024-04-06T09:41:42.643 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local description_pattern= 2024-04-06T09:41:42.643 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local service_pattern= 2024-04-06T09:41:42.643 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:41:42.643 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 1 2024-04-06T09:41:43.643 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test_status_in_pool_dir cluster1 mirror test up+replaying '' '' 2024-04-06T09:41:43.644 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:41:43.644 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:41:43.644 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:41:43.644 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+replaying 2024-04-06T09:41:43.644 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local description_pattern= 2024-04-06T09:41:43.644 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local service_pattern= 2024-04-06T09:41:43.644 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local status 2024-04-06T09:41:43.644 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ CEPH_ARGS= 2024-04-06T09:41:43.644 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster1 mirror image status mirror/test 2024-04-06T09:41:43.736 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ status='test: 2024-04-06T09:41:43.737 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: global_id: 8ad5dedc-b3e0-425d-8993-b3e2231fc5ca 2024-04-06T09:41:43.737 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: state: up+replaying 2024-04-06T09:41:43.737 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: description: replaying, {"bytes_per_second":2434837.3,"entries_behind_primary":0,"entries_per_second":150.2,"non_primary_position":{"entry_tid":1,"object_number":93,"tag_tid":15},"primary_position":{"entry_tid":1,"object_number":93,"tag_tid":15}} 2024-04-06T09:41:43.737 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: service: mirror.0 on smithi192 2024-04-06T09:41:43.737 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: last_update: 2024-04-06 09:41:27 2024-04-06T09:41:43.737 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: peer_sites: 2024-04-06T09:41:43.737 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: name: cluster2 2024-04-06T09:41:43.737 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: state: up+stopped 2024-04-06T09:41:43.737 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: description: local image is primary 2024-04-06T09:41:43.737 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: last_update: 2024-04-06 09:41:41' 2024-04-06T09:41:43.737 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^ state: .*up+replaying' 2024-04-06T09:41:43.738 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: state: up+replaying 2024-04-06T09:41:43.738 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^ description: .*' 2024-04-06T09:41:43.739 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: description: replaying, {"bytes_per_second":2434837.3,"entries_behind_primary":0,"entries_per_second":150.2,"non_primary_position":{"entry_tid":1,"object_number":93,"tag_tid":15},"primary_position":{"entry_tid":1,"object_number":93,"tag_tid":15}} 2024-04-06T09:41:43.739 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n '' ']' 2024-04-06T09:41:43.739 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ echo up+replaying 2024-04-06T09:41:43.739 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^up+' 2024-04-06T09:41:43.740 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:up+replaying 2024-04-06T09:41:43.740 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep 'service: *mirror..* on ' 2024-04-06T09:41:43.742 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: service: mirror.0 on smithi192 2024-04-06T09:41:43.743 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local last_update 2024-04-06T09:41:43.743 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-04-06T09:41:43.746 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_update='2024-04-06 09:41:27' 2024-04-06T09:41:43.746 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+replaying '2024-04-06 09:41:27' 2024-04-06T09:41:43.746 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:41:43.746 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:41:43.746 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:41:43.746 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+replaying 2024-04-06T09:41:43.746 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'prev_last_update=2024-04-06 09:41:27' 2024-04-06T09:41:43.746 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local status 2024-04-06T09:41:43.746 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ CEPH_ARGS= 2024-04-06T09:41:43.746 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-04-06T09:41:43.834 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ status='OKOKOK142094268mirror.1smithi19218.2.2-1059-g4d05f4cafalseOK42584261mirror.0smithi19218.2.2-1059-g4d05f4catrueOK43194322mirror.3smithi19218.2.2-1059-g4d05f4cafalseOK43814384mirror.2smithi19218.2.2-1059-g4d05f4cafalseOKtest8ad5dedc-b3e0-425d-8993-b3e2231fc5caup+replayingreplaying, {"bytes_per_second":2434837.3,"entries_behind_primary":0,"entries_per_second":150.2,"non_primary_position":{"entry_tid":1,"object_number":93,"tag_tid":15},"primary_position":{"entry_tid":1,"object_number":93,"tag_tid":15}}42584261mirror.0smithi1922024-04-06 09:41:27cluster2c5e3bdc3-c286-4289-8fd0-6b8628b92376up+stoppedlocal image is primary2024-04-06 09:41:41' 2024-04-06T09:41:43.834 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local last_update state 2024-04-06T09:41:43.834 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2024-04-06T09:41:43.837 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_update='2024-04-06 09:41:27' 2024-04-06T09:41:43.838 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2024-04-06T09:41:43.841 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ state=up+replaying 2024-04-06T09:41:43.841 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ echo up+replaying 2024-04-06T09:41:43.841 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep up+replaying 2024-04-06T09:41:43.842 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:up+replaying 2024-04-06T09:41:43.842 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:41:43.842 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:41:43.843 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ snap_name=snap6 2024-04-06T09:41:43.843 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ create_snap cluster2 mirror test snap6 2024-04-06T09:41:43.843 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-04-06T09:41:43.843 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:41:43.843 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:41:43.843 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local snap_name=snap6 2024-04-06T09:41:43.843 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap6 --debug-rbd=20 --debug-journaler=20 2024-04-06T09:41:52.768 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:41:52.771+0000 7f2e84d66640 -1 reset not still connected to 0x560d27073d40 2024-04-06T09:42:07.391 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.0 with skip (0.41306439672428996) > weight (0.25) 2024-04-06T09:42:07.391 INFO:tasks.rbd_mirror_thrash:kill cluster2.client.mirror.1 2024-04-06T09:42:07.392 INFO:tasks.rbd_mirror.cluster2.client.mirror.1:Sent signal 15 2024-04-06T09:42:07.392 INFO:tasks.rbd_mirror_thrash:waiting for 7 secs before reviving daemons 2024-04-06T09:42:07.393 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi192.stderr:2024-04-06T09:42:07.394+0000 7f7cab45d640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.1 (PID: 27484) UID: 1000 2024-04-06T09:42:14.393 INFO:tasks.rbd_mirror_thrash:waiting for cluster2.client.mirror.1 2024-04-06T09:42:14.393 INFO:teuthology.orchestra.run:waiting for 600 2024-04-06T09:42:14.393 INFO:tasks.rbd_mirror_thrash:reviving cluster2.client.mirror.1 2024-04-06T09:42:14.393 INFO:tasks.rbd_mirror.cluster2.client.mirror.1:Restarting daemon 2024-04-06T09:42:14.394 DEBUG:teuthology.orchestra.run.smithi192:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.1 2024-04-06T09:42:14.397 INFO:tasks.rbd_mirror.cluster2.client.mirror.1:Started 2024-04-06T09:42:14.397 INFO:tasks.rbd_mirror_thrash:waiting for 73 secs before thrashing 2024-04-06T09:42:14.601 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_image_replay_started cluster1 mirror test 2024-04-06T09:42:14.601 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:42:14.601 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:42:14.601 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:42:14.601 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_image_replay_state cluster1 mirror test started 2024-04-06T09:42:14.601 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:42:14.601 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:42:14.601 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:42:14.601 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state=started 2024-04-06T09:42:14.601 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local s 2024-04-06T09:42:14.601 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:42:14.602 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 1 2024-04-06T09:42:15.184 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:42:15.182+0000 7f2e84d66640 -1 reset not still connected to 0x560d2655eea0 2024-04-06T09:42:15.184 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:42:15.182+0000 7f2e84d66640 -1 reset not still connected to 0x560d2655fee0 2024-04-06T09:42:15.184 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:42:15.182+0000 7f2e84d66640 -1 reset not still connected to 0x560d27073930 2024-04-06T09:42:15.184 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:42:15.186+0000 7f2e84d66640 -1 reset not still connected to 0x560d27073ba0 2024-04-06T09:42:15.184 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:42:15.186+0000 7f2e84d66640 -1 reset not still connected to 0x560d27073d40 2024-04-06T09:42:15.602 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test_image_replay_state cluster1 mirror test started 2024-04-06T09:42:15.602 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:42:15.602 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:42:15.602 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:42:15.602 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local test_state=started 2024-04-06T09:42:15.602 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local status_result 2024-04-06T09:42:15.603 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local current_state=stopped 2024-04-06T09:42:15.604 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2024-04-06T09:42:15.604 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -i state 2024-04-06T09:42:15.604 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_instance=cluster1 2024-04-06T09:42:15.604 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ shift 2024-04-06T09:42:15.604 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:42:15.604 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance=cluster1 2024-04-06T09:42:15.604 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local loop_instance 2024-04-06T09:42:15.604 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:42:15.604 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sleep 0 2024-04-06T09:42:15.605 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:42:15.606 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ seq 0 3 2024-04-06T09:42:15.608 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:42:15.608 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2024-04-06T09:42:15.609 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1:0 2024-04-06T09:42:15.609 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ shift 2024-04-06T09:42:15.609 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:42:15.609 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:42:15.609 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:42:15.609 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=cluster 2024-04-06T09:42:15.609 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:42:15.609 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:42:15.609 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:42:15.609 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:42:15.609 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval cluster=cluster1 2024-04-06T09:42:15.609 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ cluster=cluster1 2024-04-06T09:42:15.609 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:42:15.609 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:42:15.610 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:42:15.610 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local local_cluster=cluster1:0 2024-04-06T09:42:15.610 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local cluster=cluster1 2024-04-06T09:42:15.610 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local instance 2024-04-06T09:42:15.610 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:42:15.610 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local val=cluster1:0 2024-04-06T09:42:15.610 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local cluster_var_name=local_cluster 2024-04-06T09:42:15.610 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local instance_var_name=instance 2024-04-06T09:42:15.610 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ cluster=cluster1 2024-04-06T09:42:15.610 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:42:15.610 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ '[' 0 = cluster1:0 ']' 2024-04-06T09:42:15.610 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ eval local_cluster=cluster1 2024-04-06T09:42:15.610 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++++ local_cluster=cluster1 2024-04-06T09:42:15.610 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ eval instance=0 2024-04-06T09:42:15.611 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++++ instance=0 2024-04-06T09:42:15.611 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:42:15.634 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:42:15.634 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:42:15.634 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:42:15.634 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2024-04-06T09:42:15.753 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ return 0 2024-04-06T09:42:15.753 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ status_result=' "state": "Replaying"' 2024-04-06T09:42:15.754 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ echo ' "state": "Replaying"' 2024-04-06T09:42:15.754 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep -i Replaying 2024-04-06T09:42:15.755 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: "state": "Replaying" 2024-04-06T09:42:15.756 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ current_state=started 2024-04-06T09:42:15.756 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test started = started 2024-04-06T09:42:15.756 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:42:15.757 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror test 2024-04-06T09:42:15.757 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local local_cluster=cluster1 2024-04-06T09:42:15.757 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-04-06T09:42:15.757 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:42:15.757 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:42:15.757 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' journal = journal ']' 2024-04-06T09:42:15.757 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_journal_replay_complete cluster1 cluster2 mirror test 2024-04-06T09:42:15.757 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local local_cluster=cluster1 2024-04-06T09:42:15.757 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-04-06T09:42:15.758 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:42:15.758 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:42:15.758 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local s master_pos mirror_pos last_mirror_pos 2024-04-06T09:42:15.758 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local master_tag master_entry mirror_tag mirror_entry 2024-04-06T09:42:15.758 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:42:15.758 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:42:15.758 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:42:15.962 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:42:15.962 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:42:15.962 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:42:15.962 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:42:15.962 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:42:15.963 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:42:15.963 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:42:15.963 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:42:15.963 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:42:15.963 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:42:15.963 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:42:15.963 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:42:15.963 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:42:15.963 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:42:15.963 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:42:15.963 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:42:15.963 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:42:15.963 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:42:15.964 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:42:15.964 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:42:15.964 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:42:15.964 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:42:15.964 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:42:15.964 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:42:15.964 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:42:15.964 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:42:15.964 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:42:15.964 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:42:15.964 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:42:15.964 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:42:15.964 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:42:15.964 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:42:15.964 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:42:15.965 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:42:15.965 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:42:15.965 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:42:15.965 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:42:15.965 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:42:15.965 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:42:15.965 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:42:15.965 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:42:15.965 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:42:15.965 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:42:15.965 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:42:15.965 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:42:15.966 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:42:15.966 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:42:15.966 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:42:15.966 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:42:15.987 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:42:15.987 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:42:15.988 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:42:15.988 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:42:16.116 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:42:16.117 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:42:16.117 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:42:16.118 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:42:16.118 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:42:16.118 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:42:16.118 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:42:16.118 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:42:16.118 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:42:16.118 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:42:16.119 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:42:16.119 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:42:16.119 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:42:16.121 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:42:16.121 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:42:16.122 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:42:16.178 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 25 2024-04-06T09:42:16.178 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 28 2024-04-06T09:42:16.178 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:42:16.178 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=113, tag_tid=18, entry_tid=1], [object_number=112, tag_tid=18, entry_tid=0], [object_number=114, tag_tid=17, entry_tid=20202], [object_number=115, tag_tid=17, entry_tid=20199]]], state=connected] 2024-04-06T09:42:16.178 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=101, tag_tid=17, entry_tid=6689], [object_number=100, tag_tid=17, entry_tid=6688], [object_number=103, tag_tid=17, entry_tid=6687], [object_number=102, tag_tid=17, entry_tid=6686]]], state=connected] 2024-04-06T09:42:16.181 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:42:16.183 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=113, tag_tid=18, entry_tid=1' 2024-04-06T09:42:16.186 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:42:16.186 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:42:16.186 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:42:16.186 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:42:16.186 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:42:16.186 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:42:16.187 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:42:16.187 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:42:16.187 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:42:16.187 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:42:16.187 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:42:16.187 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:42:16.188 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:42:16.188 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:42:16.188 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:42:16.233 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 25 2024-04-06T09:42:16.233 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 28 2024-04-06T09:42:16.234 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:42:16.234 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=113, tag_tid=18, entry_tid=1], [object_number=112, tag_tid=18, entry_tid=0], [object_number=114, tag_tid=17, entry_tid=20202], [object_number=115, tag_tid=17, entry_tid=20199]]], state=connected] 2024-04-06T09:42:16.234 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=101, tag_tid=17, entry_tid=6689], [object_number=100, tag_tid=17, entry_tid=6688], [object_number=103, tag_tid=17, entry_tid=6687], [object_number=102, tag_tid=17, entry_tid=6686]]], state=connected] 2024-04-06T09:42:16.237 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:42:16.239 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=101, tag_tid=17, entry_tid=6689' 2024-04-06T09:42:16.239 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=113, tag_tid=18, entry_tid=1' -a 'object_number=113, tag_tid=18, entry_tid=1' = 'object_number=101, tag_tid=17, entry_tid=6689' 2024-04-06T09:42:16.239 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=101, tag_tid=17, entry_tid=6689' '!=' '' 2024-04-06T09:42:16.240 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:42:16.240 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=101, tag_tid=17, entry_tid=6689' = '' 2024-04-06T09:42:16.240 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=101, tag_tid=17, entry_tid=6689' 2024-04-06T09:42:16.240 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=113, tag_tid=18, entry_tid=1' 2024-04-06T09:42:16.240 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:42:16.240 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:42:16.240 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=18 2024-04-06T09:42:16.241 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=101, tag_tid=17, entry_tid=6689' 2024-04-06T09:42:16.241 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:42:16.241 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:42:16.242 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=17 2024-04-06T09:42:16.243 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=113, tag_tid=18, entry_tid=1' 2024-04-06T09:42:16.243 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:42:16.243 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:42:16.244 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:42:16.244 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=101, tag_tid=17, entry_tid=6689' 2024-04-06T09:42:16.245 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:42:16.245 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:42:16.246 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=6689 2024-04-06T09:42:16.246 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 18 = 17 -a 1 -le 6689 2024-04-06T09:42:16.246 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:42:16.246 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:42:16.246 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:42:16.448 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:42:16.448 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:42:16.449 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:42:16.449 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:42:16.449 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:42:16.449 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:42:16.449 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:42:16.449 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:42:16.449 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:42:16.449 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:42:16.449 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:42:16.449 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:42:16.449 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:42:16.449 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:42:16.449 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:42:16.450 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:42:16.450 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:42:16.453 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:42:16.453 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:42:16.453 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:42:16.453 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:42:16.453 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:42:16.454 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:42:16.454 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:42:16.454 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:42:16.454 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:42:16.454 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:42:16.454 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:42:16.454 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:42:16.454 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:42:16.454 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:42:16.454 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:42:16.455 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:42:16.455 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:42:16.455 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:42:16.455 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:42:16.455 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:42:16.455 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:42:16.455 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:42:16.455 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:42:16.455 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:42:16.455 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:42:16.455 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:42:16.455 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:42:16.456 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:42:16.456 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:42:16.456 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:42:16.456 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:42:16.456 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:42:16.478 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:42:16.479 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:42:16.479 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:42:16.479 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:42:16.606 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:42:16.607 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:42:16.607 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:42:16.608 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:42:16.608 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:42:16.608 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:42:16.608 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:42:16.608 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:42:16.608 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:42:16.608 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:42:16.609 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:42:16.609 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:42:16.609 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:42:16.610 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:42:16.610 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:42:16.611 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:42:16.657 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 25 2024-04-06T09:42:16.657 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 28 2024-04-06T09:42:16.657 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:42:16.657 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=113, tag_tid=18, entry_tid=1], [object_number=112, tag_tid=18, entry_tid=0], [object_number=114, tag_tid=17, entry_tid=20202], [object_number=115, tag_tid=17, entry_tid=20199]]], state=connected] 2024-04-06T09:42:16.657 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=100, tag_tid=17, entry_tid=6776], [object_number=103, tag_tid=17, entry_tid=6775], [object_number=102, tag_tid=17, entry_tid=6774], [object_number=101, tag_tid=17, entry_tid=6773]]], state=connected] 2024-04-06T09:42:16.660 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:42:16.662 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=113, tag_tid=18, entry_tid=1' 2024-04-06T09:42:16.663 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:42:16.663 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:42:16.663 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:42:16.663 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:42:16.663 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:42:16.663 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:42:16.664 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:42:16.664 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:42:16.664 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:42:16.664 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:42:16.664 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:42:16.664 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:42:16.665 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:42:16.665 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:42:16.665 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:42:16.710 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 25 2024-04-06T09:42:16.711 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 28 2024-04-06T09:42:16.711 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:42:16.711 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=113, tag_tid=18, entry_tid=1], [object_number=112, tag_tid=18, entry_tid=0], [object_number=114, tag_tid=17, entry_tid=20202], [object_number=115, tag_tid=17, entry_tid=20199]]], state=connected] 2024-04-06T09:42:16.711 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=100, tag_tid=17, entry_tid=6776], [object_number=103, tag_tid=17, entry_tid=6775], [object_number=102, tag_tid=17, entry_tid=6774], [object_number=101, tag_tid=17, entry_tid=6773]]], state=connected] 2024-04-06T09:42:16.714 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:42:16.716 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=100, tag_tid=17, entry_tid=6776' 2024-04-06T09:42:16.716 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=113, tag_tid=18, entry_tid=1' -a 'object_number=113, tag_tid=18, entry_tid=1' = 'object_number=100, tag_tid=17, entry_tid=6776' 2024-04-06T09:42:16.716 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=100, tag_tid=17, entry_tid=6776' '!=' 'object_number=101, tag_tid=17, entry_tid=6689' 2024-04-06T09:42:16.716 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:42:16.716 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=100, tag_tid=17, entry_tid=6776' = 'object_number=101, tag_tid=17, entry_tid=6689' 2024-04-06T09:42:16.716 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=100, tag_tid=17, entry_tid=6776' 2024-04-06T09:42:16.717 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=113, tag_tid=18, entry_tid=1' 2024-04-06T09:42:16.717 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:42:16.717 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:42:16.717 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=18 2024-04-06T09:42:16.718 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=100, tag_tid=17, entry_tid=6776' 2024-04-06T09:42:16.718 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:42:16.718 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:42:16.719 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=17 2024-04-06T09:42:16.720 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=113, tag_tid=18, entry_tid=1' 2024-04-06T09:42:16.720 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:42:16.720 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:42:16.721 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:42:16.722 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=100, tag_tid=17, entry_tid=6776' 2024-04-06T09:42:16.722 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:42:16.722 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:42:16.723 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=6776 2024-04-06T09:42:16.723 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 18 = 17 -a 1 -le 6776 2024-04-06T09:42:16.723 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:42:16.723 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:42:16.723 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:42:16.924 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:42:16.925 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:42:16.925 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:42:16.925 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:42:16.925 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:42:16.925 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:42:16.925 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:42:16.925 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:42:16.925 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:42:16.925 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:42:16.925 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:42:16.925 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:42:16.925 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:42:16.925 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:42:16.925 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:42:16.926 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:42:16.926 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:42:16.928 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:42:16.928 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:42:16.928 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:42:16.928 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:42:16.928 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:42:16.928 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:42:16.928 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:42:16.929 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:42:16.929 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:42:16.929 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:42:16.929 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:42:16.929 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:42:16.929 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:42:16.929 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:42:16.929 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:42:16.929 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:42:16.930 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:42:16.930 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:42:16.931 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:42:16.931 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:42:16.931 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:42:16.931 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:42:16.931 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:42:16.931 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:42:16.931 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:42:16.931 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:42:16.931 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:42:16.931 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:42:16.931 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:42:16.931 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:42:16.931 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:42:16.931 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:42:16.955 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:42:16.956 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:42:16.956 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:42:16.956 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:42:17.081 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:42:17.082 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:42:17.082 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:42:17.082 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:42:17.082 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:42:17.082 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:42:17.082 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:42:17.082 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:42:17.082 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:42:17.083 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:42:17.083 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:42:17.083 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:42:17.083 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:42:17.085 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:42:17.086 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:42:17.086 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:42:17.137 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 25 2024-04-06T09:42:17.137 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 28 2024-04-06T09:42:17.137 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:42:17.138 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=113, tag_tid=18, entry_tid=1], [object_number=112, tag_tid=18, entry_tid=0], [object_number=114, tag_tid=17, entry_tid=20202], [object_number=115, tag_tid=17, entry_tid=20199]]], state=connected] 2024-04-06T09:42:17.138 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=100, tag_tid=17, entry_tid=6856], [object_number=103, tag_tid=17, entry_tid=6855], [object_number=102, tag_tid=17, entry_tid=6854], [object_number=101, tag_tid=17, entry_tid=6853]]], state=connected] 2024-04-06T09:42:17.141 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:42:17.143 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=113, tag_tid=18, entry_tid=1' 2024-04-06T09:42:17.144 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:42:17.144 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:42:17.144 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:42:17.144 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:42:17.144 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:42:17.144 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:42:17.144 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:42:17.144 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:42:17.144 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:42:17.144 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:42:17.144 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:42:17.144 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:42:17.145 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:42:17.145 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:42:17.145 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:42:17.193 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 25 2024-04-06T09:42:17.193 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 28 2024-04-06T09:42:17.193 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:42:17.193 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=113, tag_tid=18, entry_tid=1], [object_number=112, tag_tid=18, entry_tid=0], [object_number=114, tag_tid=17, entry_tid=20202], [object_number=115, tag_tid=17, entry_tid=20199]]], state=connected] 2024-04-06T09:42:17.193 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=100, tag_tid=17, entry_tid=6856], [object_number=103, tag_tid=17, entry_tid=6855], [object_number=102, tag_tid=17, entry_tid=6854], [object_number=101, tag_tid=17, entry_tid=6853]]], state=connected] 2024-04-06T09:42:17.196 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:42:17.199 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=100, tag_tid=17, entry_tid=6856' 2024-04-06T09:42:17.199 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=113, tag_tid=18, entry_tid=1' -a 'object_number=113, tag_tid=18, entry_tid=1' = 'object_number=100, tag_tid=17, entry_tid=6856' 2024-04-06T09:42:17.199 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=100, tag_tid=17, entry_tid=6856' '!=' 'object_number=100, tag_tid=17, entry_tid=6776' 2024-04-06T09:42:17.199 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:42:17.199 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=100, tag_tid=17, entry_tid=6856' = 'object_number=100, tag_tid=17, entry_tid=6776' 2024-04-06T09:42:17.199 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=100, tag_tid=17, entry_tid=6856' 2024-04-06T09:42:17.199 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=113, tag_tid=18, entry_tid=1' 2024-04-06T09:42:17.199 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:42:17.200 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:42:17.201 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=18 2024-04-06T09:42:17.201 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=100, tag_tid=17, entry_tid=6856' 2024-04-06T09:42:17.201 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:42:17.201 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:42:17.201 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=17 2024-04-06T09:42:17.202 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=113, tag_tid=18, entry_tid=1' 2024-04-06T09:42:17.202 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:42:17.202 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:42:17.203 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:42:17.204 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=100, tag_tid=17, entry_tid=6856' 2024-04-06T09:42:17.204 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:42:17.204 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:42:17.205 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=6856 2024-04-06T09:42:17.205 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 18 = 17 -a 1 -le 6856 2024-04-06T09:42:17.205 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:42:17.205 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:42:17.206 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:42:17.407 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:42:17.407 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:42:17.408 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:42:17.408 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:42:17.408 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:42:17.408 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:42:17.408 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:42:17.408 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:42:17.408 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:42:17.408 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:42:17.408 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:42:17.408 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:42:17.408 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:42:17.408 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:42:17.409 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:42:17.409 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:42:17.409 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:42:17.412 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:42:17.412 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:42:17.412 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:42:17.412 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:42:17.412 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:42:17.412 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:42:17.412 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:42:17.412 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:42:17.412 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:42:17.412 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:42:17.413 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:42:17.413 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:42:17.413 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:42:17.413 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:42:17.413 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:42:17.413 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:42:17.414 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:42:17.414 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:42:17.414 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:42:17.414 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:42:17.414 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:42:17.414 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:42:17.414 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:42:17.414 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:42:17.414 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:42:17.414 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:42:17.415 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:42:17.415 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:42:17.415 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:42:17.415 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:42:17.415 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:42:17.415 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:42:17.438 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:42:17.439 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:42:17.439 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:42:17.439 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:42:17.573 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:42:17.574 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:42:17.574 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:42:17.574 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:42:17.574 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:42:17.574 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:42:17.574 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:42:17.574 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:42:17.574 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:42:17.574 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:42:17.575 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:42:17.575 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:42:17.575 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:42:17.576 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:42:17.577 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:42:17.577 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:42:17.628 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 25 2024-04-06T09:42:17.628 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 28 2024-04-06T09:42:17.628 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:42:17.628 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=113, tag_tid=18, entry_tid=1], [object_number=112, tag_tid=18, entry_tid=0], [object_number=114, tag_tid=17, entry_tid=20202], [object_number=115, tag_tid=17, entry_tid=20199]]], state=connected] 2024-04-06T09:42:17.628 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=100, tag_tid=17, entry_tid=6932], [object_number=103, tag_tid=17, entry_tid=6931], [object_number=102, tag_tid=17, entry_tid=6930], [object_number=101, tag_tid=17, entry_tid=6929]]], state=connected] 2024-04-06T09:42:17.632 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:42:17.633 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=113, tag_tid=18, entry_tid=1' 2024-04-06T09:42:17.634 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:42:17.634 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:42:17.635 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:42:17.635 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:42:17.635 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:42:17.635 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:42:17.635 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:42:17.635 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:42:17.635 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:42:17.635 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:42:17.635 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:42:17.635 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:42:17.636 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:42:17.636 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:42:17.636 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:42:17.682 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 25 2024-04-06T09:42:17.682 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 28 2024-04-06T09:42:17.683 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:42:17.683 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=113, tag_tid=18, entry_tid=1], [object_number=112, tag_tid=18, entry_tid=0], [object_number=114, tag_tid=17, entry_tid=20202], [object_number=115, tag_tid=17, entry_tid=20199]]], state=connected] 2024-04-06T09:42:17.683 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=100, tag_tid=17, entry_tid=6932], [object_number=103, tag_tid=17, entry_tid=6931], [object_number=102, tag_tid=17, entry_tid=6930], [object_number=101, tag_tid=17, entry_tid=6929]]], state=connected] 2024-04-06T09:42:17.686 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:42:17.688 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=100, tag_tid=17, entry_tid=6932' 2024-04-06T09:42:17.688 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=113, tag_tid=18, entry_tid=1' -a 'object_number=113, tag_tid=18, entry_tid=1' = 'object_number=100, tag_tid=17, entry_tid=6932' 2024-04-06T09:42:17.688 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=100, tag_tid=17, entry_tid=6932' '!=' 'object_number=100, tag_tid=17, entry_tid=6856' 2024-04-06T09:42:17.688 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:42:17.688 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=100, tag_tid=17, entry_tid=6932' = 'object_number=100, tag_tid=17, entry_tid=6856' 2024-04-06T09:42:17.688 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=100, tag_tid=17, entry_tid=6932' 2024-04-06T09:42:17.688 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=113, tag_tid=18, entry_tid=1' 2024-04-06T09:42:17.688 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:42:17.688 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:42:17.689 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=18 2024-04-06T09:42:17.690 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=100, tag_tid=17, entry_tid=6932' 2024-04-06T09:42:17.690 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:42:17.690 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:42:17.691 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=17 2024-04-06T09:42:17.691 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=113, tag_tid=18, entry_tid=1' 2024-04-06T09:42:17.691 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:42:17.691 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:42:17.692 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:42:17.693 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=100, tag_tid=17, entry_tid=6932' 2024-04-06T09:42:17.693 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:42:17.693 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:42:17.694 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=6932 2024-04-06T09:42:17.695 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 18 = 17 -a 1 -le 6932 2024-04-06T09:42:17.695 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:42:17.695 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:42:17.695 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:42:17.896 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:42:17.896 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:42:17.896 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:42:17.897 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:42:17.897 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:42:17.897 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:42:17.897 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:42:17.897 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:42:17.897 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:42:17.897 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:42:17.897 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:42:17.897 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:42:17.897 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:42:17.897 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:42:17.897 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:42:17.898 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:42:17.898 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:42:17.899 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:42:17.900 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:42:17.900 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:42:17.900 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:42:17.900 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:42:17.900 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:42:17.900 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:42:17.900 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:42:17.900 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:42:17.900 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:42:17.900 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:42:17.900 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:42:17.900 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:42:17.900 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:42:17.900 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:42:17.901 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:42:17.902 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:42:17.902 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:42:17.902 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:42:17.902 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:42:17.902 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:42:17.902 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:42:17.902 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:42:17.902 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:42:17.902 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:42:17.903 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:42:17.903 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:42:17.903 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:42:17.903 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:42:17.903 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:42:17.903 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:42:17.903 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:42:17.927 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:42:17.927 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:42:17.927 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:42:17.927 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:42:18.058 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:42:18.060 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:42:18.060 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:42:18.060 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:42:18.060 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:42:18.060 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:42:18.060 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:42:18.060 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:42:18.060 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:42:18.060 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:42:18.061 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:42:18.061 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:42:18.061 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:42:18.062 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:42:18.063 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:42:18.063 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:42:18.115 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 25 2024-04-06T09:42:18.115 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 28 2024-04-06T09:42:18.115 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:42:18.115 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=113, tag_tid=18, entry_tid=1], [object_number=112, tag_tid=18, entry_tid=0], [object_number=114, tag_tid=17, entry_tid=20202], [object_number=115, tag_tid=17, entry_tid=20199]]], state=connected] 2024-04-06T09:42:18.115 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=100, tag_tid=17, entry_tid=7008], [object_number=103, tag_tid=17, entry_tid=7007], [object_number=102, tag_tid=17, entry_tid=7006], [object_number=101, tag_tid=17, entry_tid=7005]]], state=connected] 2024-04-06T09:42:18.118 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:42:18.120 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=113, tag_tid=18, entry_tid=1' 2024-04-06T09:42:18.121 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:42:18.121 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:42:18.121 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:42:18.121 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:42:18.121 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:42:18.121 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:42:18.121 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:42:18.121 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:42:18.121 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:42:18.121 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:42:18.122 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:42:18.122 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:42:18.122 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:42:18.123 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:42:18.123 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:42:18.169 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 25 2024-04-06T09:42:18.169 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 28 2024-04-06T09:42:18.169 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:42:18.169 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=113, tag_tid=18, entry_tid=1], [object_number=112, tag_tid=18, entry_tid=0], [object_number=114, tag_tid=17, entry_tid=20202], [object_number=115, tag_tid=17, entry_tid=20199]]], state=connected] 2024-04-06T09:42:18.169 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=100, tag_tid=17, entry_tid=7008], [object_number=103, tag_tid=17, entry_tid=7007], [object_number=102, tag_tid=17, entry_tid=7006], [object_number=101, tag_tid=17, entry_tid=7005]]], state=connected] 2024-04-06T09:42:18.173 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:42:18.174 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=100, tag_tid=17, entry_tid=7008' 2024-04-06T09:42:18.174 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=113, tag_tid=18, entry_tid=1' -a 'object_number=113, tag_tid=18, entry_tid=1' = 'object_number=100, tag_tid=17, entry_tid=7008' 2024-04-06T09:42:18.175 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=100, tag_tid=17, entry_tid=7008' '!=' 'object_number=100, tag_tid=17, entry_tid=6932' 2024-04-06T09:42:18.175 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:42:18.175 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=100, tag_tid=17, entry_tid=7008' = 'object_number=100, tag_tid=17, entry_tid=6932' 2024-04-06T09:42:18.175 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=100, tag_tid=17, entry_tid=7008' 2024-04-06T09:42:18.175 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=113, tag_tid=18, entry_tid=1' 2024-04-06T09:42:18.175 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:42:18.176 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:42:18.176 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=18 2024-04-06T09:42:18.177 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=100, tag_tid=17, entry_tid=7008' 2024-04-06T09:42:18.177 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:42:18.177 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:42:18.178 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=17 2024-04-06T09:42:18.178 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=113, tag_tid=18, entry_tid=1' 2024-04-06T09:42:18.178 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:42:18.179 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:42:18.179 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:42:18.180 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=100, tag_tid=17, entry_tid=7008' 2024-04-06T09:42:18.180 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:42:18.180 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:42:18.181 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=7008 2024-04-06T09:42:18.181 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 18 = 17 -a 1 -le 7008 2024-04-06T09:42:18.182 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:42:18.182 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:42:18.182 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:42:18.383 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:42:18.384 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:42:18.384 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:42:18.384 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:42:18.384 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:42:18.384 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:42:18.384 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:42:18.384 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:42:18.384 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:42:18.384 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:42:18.384 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:42:18.384 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:42:18.384 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:42:18.384 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:42:18.385 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:42:18.385 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:42:18.385 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:42:18.387 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:42:18.387 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:42:18.387 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:42:18.387 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:42:18.387 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:42:18.387 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:42:18.388 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:42:18.388 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:42:18.388 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:42:18.388 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:42:18.388 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:42:18.388 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:42:18.388 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:42:18.388 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:42:18.388 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:42:18.388 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:42:18.388 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:42:18.388 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:42:18.388 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:42:18.388 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:42:18.388 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:42:18.389 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:42:18.389 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:42:18.389 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:42:18.389 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:42:18.389 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:42:18.389 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:42:18.389 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:42:18.389 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:42:18.389 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:42:18.389 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:42:18.390 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:42:18.413 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:42:18.413 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:42:18.413 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:42:18.413 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:42:18.552 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:42:18.553 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:42:18.553 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:42:18.554 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:42:18.554 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:42:18.554 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:42:18.554 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:42:18.554 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:42:18.554 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:42:18.554 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:42:18.555 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:42:18.555 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:42:18.555 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:42:18.556 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:42:18.556 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:42:18.557 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:42:18.614 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 25 2024-04-06T09:42:18.614 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 28 2024-04-06T09:42:18.614 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:42:18.614 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=113, tag_tid=18, entry_tid=1], [object_number=112, tag_tid=18, entry_tid=0], [object_number=114, tag_tid=17, entry_tid=20202], [object_number=115, tag_tid=17, entry_tid=20199]]], state=connected] 2024-04-06T09:42:18.615 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=101, tag_tid=17, entry_tid=7081], [object_number=100, tag_tid=17, entry_tid=7080], [object_number=103, tag_tid=17, entry_tid=7079], [object_number=102, tag_tid=17, entry_tid=7078]]], state=connected] 2024-04-06T09:42:18.618 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:42:18.619 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=113, tag_tid=18, entry_tid=1' 2024-04-06T09:42:18.620 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:42:18.620 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:42:18.620 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:42:18.620 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:42:18.620 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:42:18.620 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:42:18.620 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:42:18.621 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:42:18.621 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:42:18.621 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:42:18.621 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:42:18.621 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:42:18.622 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:42:18.622 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:42:18.622 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:42:18.669 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 25 2024-04-06T09:42:18.669 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 28 2024-04-06T09:42:18.669 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:42:18.670 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=113, tag_tid=18, entry_tid=1], [object_number=112, tag_tid=18, entry_tid=0], [object_number=114, tag_tid=17, entry_tid=20202], [object_number=115, tag_tid=17, entry_tid=20199]]], state=connected] 2024-04-06T09:42:18.670 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=101, tag_tid=17, entry_tid=7081], [object_number=100, tag_tid=17, entry_tid=7080], [object_number=103, tag_tid=17, entry_tid=7079], [object_number=102, tag_tid=17, entry_tid=7078]]], state=connected] 2024-04-06T09:42:18.673 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:42:18.674 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=101, tag_tid=17, entry_tid=7081' 2024-04-06T09:42:18.675 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=113, tag_tid=18, entry_tid=1' -a 'object_number=113, tag_tid=18, entry_tid=1' = 'object_number=101, tag_tid=17, entry_tid=7081' 2024-04-06T09:42:18.675 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=101, tag_tid=17, entry_tid=7081' '!=' 'object_number=100, tag_tid=17, entry_tid=7008' 2024-04-06T09:42:18.675 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:42:18.675 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=101, tag_tid=17, entry_tid=7081' = 'object_number=100, tag_tid=17, entry_tid=7008' 2024-04-06T09:42:18.675 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=101, tag_tid=17, entry_tid=7081' 2024-04-06T09:42:18.675 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=113, tag_tid=18, entry_tid=1' 2024-04-06T09:42:18.676 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:42:18.676 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:42:18.676 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=18 2024-04-06T09:42:18.677 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=101, tag_tid=17, entry_tid=7081' 2024-04-06T09:42:18.677 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:42:18.677 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:42:18.678 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=17 2024-04-06T09:42:18.678 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=113, tag_tid=18, entry_tid=1' 2024-04-06T09:42:18.679 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:42:18.679 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:42:18.679 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:42:18.680 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=101, tag_tid=17, entry_tid=7081' 2024-04-06T09:42:18.680 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:42:18.680 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:42:18.681 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=7081 2024-04-06T09:42:18.682 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 18 = 17 -a 1 -le 7081 2024-04-06T09:42:18.682 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:42:18.682 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:42:18.682 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:42:18.883 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:42:18.884 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:42:18.884 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:42:18.884 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:42:18.884 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:42:18.884 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:42:18.884 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:42:18.884 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:42:18.884 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:42:18.884 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:42:18.884 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:42:18.884 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:42:18.884 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:42:18.885 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:42:18.885 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:42:18.885 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:42:18.885 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:42:18.889 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:42:18.889 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:42:18.889 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:42:18.889 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:42:18.889 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:42:18.889 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:42:18.889 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:42:18.889 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:42:18.889 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:42:18.889 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:42:18.889 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:42:18.889 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:42:18.890 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:42:18.890 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:42:18.890 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:42:18.890 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:42:18.890 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:42:18.890 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:42:18.890 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:42:18.890 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:42:18.890 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:42:18.890 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:42:18.890 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:42:18.890 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:42:18.890 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:42:18.890 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:42:18.891 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:42:18.891 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:42:18.891 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:42:18.891 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:42:18.891 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:42:18.891 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:42:18.915 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:42:18.915 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:42:18.915 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:42:18.915 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:42:19.043 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:42:19.044 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:42:19.044 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:42:19.044 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:42:19.044 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:42:19.044 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:42:19.044 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:42:19.044 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:42:19.044 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:42:19.044 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:42:19.045 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:42:19.045 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:42:19.045 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:42:19.047 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:42:19.047 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:42:19.047 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:42:19.099 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 25 2024-04-06T09:42:19.099 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 28 2024-04-06T09:42:19.099 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:42:19.099 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=113, tag_tid=18, entry_tid=1], [object_number=112, tag_tid=18, entry_tid=0], [object_number=114, tag_tid=17, entry_tid=20202], [object_number=115, tag_tid=17, entry_tid=20199]]], state=connected] 2024-04-06T09:42:19.099 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=103, tag_tid=17, entry_tid=7167], [object_number=102, tag_tid=17, entry_tid=7166], [object_number=101, tag_tid=17, entry_tid=7165], [object_number=100, tag_tid=17, entry_tid=7164]]], state=connected] 2024-04-06T09:42:19.102 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:42:19.104 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=113, tag_tid=18, entry_tid=1' 2024-04-06T09:42:19.105 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:42:19.105 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:42:19.105 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:42:19.105 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:42:19.105 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:42:19.105 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:42:19.105 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:42:19.106 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:42:19.106 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:42:19.106 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:42:19.106 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:42:19.106 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:42:19.107 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:42:19.107 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:42:19.107 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:42:19.155 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 25 2024-04-06T09:42:19.156 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 28 2024-04-06T09:42:19.156 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:42:19.156 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=113, tag_tid=18, entry_tid=1], [object_number=112, tag_tid=18, entry_tid=0], [object_number=114, tag_tid=17, entry_tid=20202], [object_number=115, tag_tid=17, entry_tid=20199]]], state=connected] 2024-04-06T09:42:19.156 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=103, tag_tid=17, entry_tid=7167], [object_number=102, tag_tid=17, entry_tid=7166], [object_number=101, tag_tid=17, entry_tid=7165], [object_number=100, tag_tid=17, entry_tid=7164]]], state=connected] 2024-04-06T09:42:19.159 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:42:19.161 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=103, tag_tid=17, entry_tid=7167' 2024-04-06T09:42:19.161 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=113, tag_tid=18, entry_tid=1' -a 'object_number=113, tag_tid=18, entry_tid=1' = 'object_number=103, tag_tid=17, entry_tid=7167' 2024-04-06T09:42:19.162 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=103, tag_tid=17, entry_tid=7167' '!=' 'object_number=101, tag_tid=17, entry_tid=7081' 2024-04-06T09:42:19.162 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:42:19.162 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=103, tag_tid=17, entry_tid=7167' = 'object_number=101, tag_tid=17, entry_tid=7081' 2024-04-06T09:42:19.162 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=103, tag_tid=17, entry_tid=7167' 2024-04-06T09:42:19.162 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=113, tag_tid=18, entry_tid=1' 2024-04-06T09:42:19.162 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:42:19.162 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:42:19.162 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=18 2024-04-06T09:42:19.163 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=103, tag_tid=17, entry_tid=7167' 2024-04-06T09:42:19.163 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:42:19.163 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:42:19.164 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=17 2024-04-06T09:42:19.165 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=113, tag_tid=18, entry_tid=1' 2024-04-06T09:42:19.165 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:42:19.165 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:42:19.166 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:42:19.167 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=103, tag_tid=17, entry_tid=7167' 2024-04-06T09:42:19.167 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:42:19.167 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:42:19.168 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=7167 2024-04-06T09:42:19.168 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 18 = 17 -a 1 -le 7167 2024-04-06T09:42:19.168 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:42:19.168 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:42:19.168 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:42:19.370 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:42:19.370 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:42:19.370 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:42:19.370 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:42:19.370 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:42:19.370 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:42:19.370 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:42:19.370 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:42:19.370 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:42:19.370 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:42:19.370 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:42:19.370 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:42:19.371 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:42:19.371 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:42:19.371 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:42:19.371 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:42:19.371 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:42:19.373 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:42:19.373 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:42:19.374 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:42:19.374 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:42:19.374 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:42:19.374 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:42:19.374 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:42:19.374 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:42:19.374 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:42:19.374 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:42:19.374 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:42:19.374 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:42:19.374 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:42:19.374 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:42:19.374 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:42:19.374 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:42:19.376 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:42:19.376 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:42:19.376 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:42:19.376 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:42:19.376 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:42:19.376 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:42:19.376 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:42:19.376 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:42:19.377 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:42:19.377 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:42:19.377 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:42:19.377 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:42:19.377 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:42:19.377 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:42:19.377 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:42:19.377 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:42:19.399 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:42:19.400 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:42:19.400 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:42:19.400 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:42:19.529 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:42:19.530 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:42:19.530 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:42:19.530 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:42:19.530 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:42:19.530 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:42:19.530 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:42:19.530 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:42:19.530 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:42:19.530 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:42:19.531 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:42:19.531 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:42:19.531 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:42:19.533 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:42:19.534 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:42:19.534 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:42:19.587 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 25 2024-04-06T09:42:19.587 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 28 2024-04-06T09:42:19.587 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:42:19.588 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=113, tag_tid=18, entry_tid=1], [object_number=112, tag_tid=18, entry_tid=0], [object_number=114, tag_tid=17, entry_tid=20202], [object_number=115, tag_tid=17, entry_tid=20199]]], state=connected] 2024-04-06T09:42:19.588 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=103, tag_tid=17, entry_tid=7251], [object_number=102, tag_tid=17, entry_tid=7250], [object_number=101, tag_tid=17, entry_tid=7249], [object_number=100, tag_tid=17, entry_tid=7248]]], state=connected] 2024-04-06T09:42:19.591 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:42:19.592 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=113, tag_tid=18, entry_tid=1' 2024-04-06T09:42:19.593 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:42:19.593 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:42:19.593 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:42:19.593 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:42:19.593 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:42:19.593 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:42:19.594 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:42:19.594 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:42:19.594 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:42:19.594 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:42:19.594 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:42:19.594 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:42:19.594 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:42:19.595 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:42:19.595 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:42:19.643 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 25 2024-04-06T09:42:19.644 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 28 2024-04-06T09:42:19.644 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:42:19.644 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=113, tag_tid=18, entry_tid=1], [object_number=112, tag_tid=18, entry_tid=0], [object_number=114, tag_tid=17, entry_tid=20202], [object_number=115, tag_tid=17, entry_tid=20199]]], state=connected] 2024-04-06T09:42:19.644 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=103, tag_tid=17, entry_tid=7251], [object_number=102, tag_tid=17, entry_tid=7250], [object_number=101, tag_tid=17, entry_tid=7249], [object_number=100, tag_tid=17, entry_tid=7248]]], state=connected] 2024-04-06T09:42:19.647 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:42:19.649 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=103, tag_tid=17, entry_tid=7251' 2024-04-06T09:42:19.649 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=113, tag_tid=18, entry_tid=1' -a 'object_number=113, tag_tid=18, entry_tid=1' = 'object_number=103, tag_tid=17, entry_tid=7251' 2024-04-06T09:42:19.649 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=103, tag_tid=17, entry_tid=7251' '!=' 'object_number=103, tag_tid=17, entry_tid=7167' 2024-04-06T09:42:19.649 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:42:19.650 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=103, tag_tid=17, entry_tid=7251' = 'object_number=103, tag_tid=17, entry_tid=7167' 2024-04-06T09:42:19.650 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=103, tag_tid=17, entry_tid=7251' 2024-04-06T09:42:19.650 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=113, tag_tid=18, entry_tid=1' 2024-04-06T09:42:19.650 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:42:19.650 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:42:19.650 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=18 2024-04-06T09:42:19.651 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=103, tag_tid=17, entry_tid=7251' 2024-04-06T09:42:19.651 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:42:19.651 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:42:19.652 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=17 2024-04-06T09:42:19.653 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=113, tag_tid=18, entry_tid=1' 2024-04-06T09:42:19.653 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:42:19.653 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:42:19.654 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:42:19.654 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=103, tag_tid=17, entry_tid=7251' 2024-04-06T09:42:19.655 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:42:19.655 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:42:19.656 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=7251 2024-04-06T09:42:19.656 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 18 = 17 -a 1 -le 7251 2024-04-06T09:42:19.656 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:42:19.656 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:42:19.656 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:42:19.857 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:42:19.858 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:42:19.858 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:42:19.858 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:42:19.858 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:42:19.858 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:42:19.858 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:42:19.858 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:42:19.858 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:42:19.858 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:42:19.858 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:42:19.858 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:42:19.859 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:42:19.859 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:42:19.859 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:42:19.859 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:42:19.859 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:42:19.862 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:42:19.862 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:42:19.862 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:42:19.862 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:42:19.862 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:42:19.862 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:42:19.862 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:42:19.862 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:42:19.862 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:42:19.862 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:42:19.863 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:42:19.863 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:42:19.863 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:42:19.863 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:42:19.863 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:42:19.863 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:42:19.865 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:42:19.865 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:42:19.865 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:42:19.865 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:42:19.865 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:42:19.866 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:42:19.866 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:42:19.866 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:42:19.866 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:42:19.866 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:42:19.866 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:42:19.866 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:42:19.866 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:42:19.866 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:42:19.867 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:42:19.867 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:42:19.890 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:42:19.890 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:42:19.890 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:42:19.890 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:42:20.015 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:42:20.016 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:42:20.016 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:42:20.016 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:42:20.016 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:42:20.016 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:42:20.016 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:42:20.016 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:42:20.016 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:42:20.016 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:42:20.017 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:42:20.017 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:42:20.017 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:42:20.018 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:42:20.019 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:42:20.019 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:42:20.070 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 25 2024-04-06T09:42:20.071 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 28 2024-04-06T09:42:20.071 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:42:20.071 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=113, tag_tid=18, entry_tid=1], [object_number=112, tag_tid=18, entry_tid=0], [object_number=114, tag_tid=17, entry_tid=20202], [object_number=115, tag_tid=17, entry_tid=20199]]], state=connected] 2024-04-06T09:42:20.071 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=101, tag_tid=17, entry_tid=7333], [object_number=100, tag_tid=17, entry_tid=7332], [object_number=103, tag_tid=17, entry_tid=7331], [object_number=102, tag_tid=17, entry_tid=7330]]], state=connected] 2024-04-06T09:42:20.074 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:42:20.075 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=113, tag_tid=18, entry_tid=1' 2024-04-06T09:42:20.076 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:42:20.077 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:42:20.077 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:42:20.077 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:42:20.077 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:42:20.077 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:42:20.077 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:42:20.077 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:42:20.077 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:42:20.077 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:42:20.077 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:42:20.077 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:42:20.078 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:42:20.078 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:42:20.078 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:42:20.126 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 25 2024-04-06T09:42:20.127 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 28 2024-04-06T09:42:20.127 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:42:20.127 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=113, tag_tid=18, entry_tid=1], [object_number=112, tag_tid=18, entry_tid=0], [object_number=114, tag_tid=17, entry_tid=20202], [object_number=115, tag_tid=17, entry_tid=20199]]], state=connected] 2024-04-06T09:42:20.127 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=101, tag_tid=17, entry_tid=7333], [object_number=100, tag_tid=17, entry_tid=7332], [object_number=103, tag_tid=17, entry_tid=7331], [object_number=102, tag_tid=17, entry_tid=7330]]], state=connected] 2024-04-06T09:42:20.130 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:42:20.132 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=101, tag_tid=17, entry_tid=7333' 2024-04-06T09:42:20.132 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=113, tag_tid=18, entry_tid=1' -a 'object_number=113, tag_tid=18, entry_tid=1' = 'object_number=101, tag_tid=17, entry_tid=7333' 2024-04-06T09:42:20.133 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=101, tag_tid=17, entry_tid=7333' '!=' 'object_number=103, tag_tid=17, entry_tid=7251' 2024-04-06T09:42:20.133 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:42:20.133 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=101, tag_tid=17, entry_tid=7333' = 'object_number=103, tag_tid=17, entry_tid=7251' 2024-04-06T09:42:20.133 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=101, tag_tid=17, entry_tid=7333' 2024-04-06T09:42:20.133 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=113, tag_tid=18, entry_tid=1' 2024-04-06T09:42:20.133 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:42:20.133 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:42:20.133 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=18 2024-04-06T09:42:20.135 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=101, tag_tid=17, entry_tid=7333' 2024-04-06T09:42:20.135 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:42:20.135 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:42:20.135 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=17 2024-04-06T09:42:20.136 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=113, tag_tid=18, entry_tid=1' 2024-04-06T09:42:20.136 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:42:20.136 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:42:20.137 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:42:20.138 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=101, tag_tid=17, entry_tid=7333' 2024-04-06T09:42:20.138 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:42:20.138 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:42:20.139 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=7333 2024-04-06T09:42:20.139 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 18 = 17 -a 1 -le 7333 2024-04-06T09:42:20.139 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:42:20.139 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:42:20.140 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:42:20.341 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:42:20.341 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:42:20.341 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:42:20.341 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:42:20.341 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:42:20.341 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:42:20.341 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:42:20.342 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:42:20.342 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:42:20.342 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:42:20.342 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:42:20.342 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:42:20.342 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:42:20.342 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:42:20.342 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:42:20.342 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:42:20.343 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:42:20.346 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:42:20.346 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:42:20.347 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:42:20.347 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:42:20.347 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:42:20.347 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:42:20.347 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:42:20.347 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:42:20.347 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:42:20.347 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:42:20.347 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:42:20.347 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:42:20.347 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:42:20.347 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:42:20.347 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:42:20.347 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:42:20.347 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:42:20.348 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:42:20.348 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:42:20.348 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:42:20.348 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:42:20.348 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:42:20.348 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:42:20.348 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:42:20.348 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:42:20.348 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:42:20.348 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:42:20.348 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:42:20.348 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:42:20.348 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:42:20.348 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:42:20.349 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:42:20.373 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:42:20.373 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:42:20.373 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:42:20.373 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:42:20.501 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:42:20.502 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:42:20.502 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:42:20.502 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:42:20.502 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:42:20.502 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:42:20.502 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:42:20.502 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:42:20.502 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:42:20.502 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:42:20.503 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:42:20.503 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:42:20.503 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:42:20.505 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:42:20.505 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:42:20.506 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:42:20.559 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 25 2024-04-06T09:42:20.559 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 28 2024-04-06T09:42:20.559 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:42:20.559 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=113, tag_tid=18, entry_tid=1], [object_number=112, tag_tid=18, entry_tid=0], [object_number=114, tag_tid=17, entry_tid=20202], [object_number=115, tag_tid=17, entry_tid=20199]]], state=connected] 2024-04-06T09:42:20.559 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=101, tag_tid=17, entry_tid=7421], [object_number=100, tag_tid=17, entry_tid=7420], [object_number=103, tag_tid=17, entry_tid=7419], [object_number=102, tag_tid=17, entry_tid=7418]]], state=connected] 2024-04-06T09:42:20.563 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:42:20.564 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=113, tag_tid=18, entry_tid=1' 2024-04-06T09:42:20.565 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:42:20.565 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:42:20.565 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:42:20.566 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:42:20.566 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:42:20.566 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:42:20.566 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:42:20.566 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:42:20.566 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:42:20.566 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:42:20.566 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:42:20.566 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:42:20.567 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:42:20.567 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:42:20.567 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:42:20.615 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 25 2024-04-06T09:42:20.615 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 28 2024-04-06T09:42:20.615 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:42:20.615 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=113, tag_tid=18, entry_tid=1], [object_number=112, tag_tid=18, entry_tid=0], [object_number=114, tag_tid=17, entry_tid=20202], [object_number=115, tag_tid=17, entry_tid=20199]]], state=connected] 2024-04-06T09:42:20.616 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=101, tag_tid=17, entry_tid=7421], [object_number=100, tag_tid=17, entry_tid=7420], [object_number=103, tag_tid=17, entry_tid=7419], [object_number=102, tag_tid=17, entry_tid=7418]]], state=connected] 2024-04-06T09:42:20.619 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:42:20.621 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=101, tag_tid=17, entry_tid=7421' 2024-04-06T09:42:20.621 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=113, tag_tid=18, entry_tid=1' -a 'object_number=113, tag_tid=18, entry_tid=1' = 'object_number=101, tag_tid=17, entry_tid=7421' 2024-04-06T09:42:20.621 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=101, tag_tid=17, entry_tid=7421' '!=' 'object_number=101, tag_tid=17, entry_tid=7333' 2024-04-06T09:42:20.621 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:42:20.621 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=101, tag_tid=17, entry_tid=7421' = 'object_number=101, tag_tid=17, entry_tid=7333' 2024-04-06T09:42:20.621 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=101, tag_tid=17, entry_tid=7421' 2024-04-06T09:42:20.622 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=113, tag_tid=18, entry_tid=1' 2024-04-06T09:42:20.622 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:42:20.622 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:42:20.622 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=18 2024-04-06T09:42:20.623 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=101, tag_tid=17, entry_tid=7421' 2024-04-06T09:42:20.623 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:42:20.623 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:42:20.624 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=17 2024-04-06T09:42:20.624 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=113, tag_tid=18, entry_tid=1' 2024-04-06T09:42:20.624 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:42:20.624 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:42:20.625 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:42:20.626 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=101, tag_tid=17, entry_tid=7421' 2024-04-06T09:42:20.626 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:42:20.626 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:42:20.627 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=7421 2024-04-06T09:42:20.627 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 18 = 17 -a 1 -le 7421 2024-04-06T09:42:20.628 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:42:20.628 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:42:20.628 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:42:20.829 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:42:20.829 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:42:20.829 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:42:20.829 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:42:20.830 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:42:20.830 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:42:20.830 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:42:20.830 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:42:20.830 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:42:20.830 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:42:20.830 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:42:20.830 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:42:20.830 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:42:20.830 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:42:20.830 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:42:20.831 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:42:20.831 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:42:20.833 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:42:20.834 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:42:20.834 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:42:20.834 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:42:20.834 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:42:20.834 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:42:20.834 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:42:20.834 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:42:20.834 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:42:20.834 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:42:20.834 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:42:20.834 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:42:20.834 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:42:20.834 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:42:20.835 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:42:20.835 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:42:20.836 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:42:20.836 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:42:20.836 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:42:20.836 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:42:20.836 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:42:20.836 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:42:20.836 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:42:20.836 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:42:20.836 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:42:20.836 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:42:20.836 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:42:20.837 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:42:20.837 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:42:20.837 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:42:20.837 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:42:20.837 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:42:20.860 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:42:20.860 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:42:20.860 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:42:20.860 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:42:20.999 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:42:21.000 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:42:21.000 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:42:21.000 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:42:21.000 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:42:21.001 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:42:21.001 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:42:21.001 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:42:21.001 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:42:21.001 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:42:21.001 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:42:21.002 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:42:21.002 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:42:21.003 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:42:21.003 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:42:21.003 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:42:21.055 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 25 2024-04-06T09:42:21.055 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 28 2024-04-06T09:42:21.055 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:42:21.055 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=113, tag_tid=18, entry_tid=1], [object_number=112, tag_tid=18, entry_tid=0], [object_number=114, tag_tid=17, entry_tid=20202], [object_number=115, tag_tid=17, entry_tid=20199]]], state=connected] 2024-04-06T09:42:21.055 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=100, tag_tid=17, entry_tid=7508], [object_number=103, tag_tid=17, entry_tid=7507], [object_number=102, tag_tid=17, entry_tid=7506], [object_number=101, tag_tid=17, entry_tid=7505]]], state=connected] 2024-04-06T09:42:21.059 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:42:21.060 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=113, tag_tid=18, entry_tid=1' 2024-04-06T09:42:21.061 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:42:21.061 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:42:21.061 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:42:21.061 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:42:21.061 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:42:21.062 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:42:21.062 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:42:21.062 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:42:21.062 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:42:21.062 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:42:21.062 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:42:21.062 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:42:21.063 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:42:21.063 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:42:21.063 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:42:21.316 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 25 2024-04-06T09:42:21.316 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 28 2024-04-06T09:42:21.316 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:42:21.316 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=113, tag_tid=18, entry_tid=1], [object_number=112, tag_tid=18, entry_tid=0], [object_number=114, tag_tid=17, entry_tid=20202], [object_number=115, tag_tid=17, entry_tid=20199]]], state=connected] 2024-04-06T09:42:21.316 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=100, tag_tid=17, entry_tid=7508], [object_number=103, tag_tid=17, entry_tid=7507], [object_number=102, tag_tid=17, entry_tid=7506], [object_number=101, tag_tid=17, entry_tid=7505]]], state=connected] 2024-04-06T09:42:21.323 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:42:21.325 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=100, tag_tid=17, entry_tid=7508' 2024-04-06T09:42:21.325 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=113, tag_tid=18, entry_tid=1' -a 'object_number=113, tag_tid=18, entry_tid=1' = 'object_number=100, tag_tid=17, entry_tid=7508' 2024-04-06T09:42:21.325 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=100, tag_tid=17, entry_tid=7508' '!=' 'object_number=101, tag_tid=17, entry_tid=7421' 2024-04-06T09:42:21.325 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:42:21.325 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=100, tag_tid=17, entry_tid=7508' = 'object_number=101, tag_tid=17, entry_tid=7421' 2024-04-06T09:42:21.326 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=100, tag_tid=17, entry_tid=7508' 2024-04-06T09:42:21.326 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=113, tag_tid=18, entry_tid=1' 2024-04-06T09:42:21.326 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:42:21.326 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:42:21.327 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=18 2024-04-06T09:42:21.328 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=100, tag_tid=17, entry_tid=7508' 2024-04-06T09:42:21.328 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:42:21.329 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:42:21.330 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=17 2024-04-06T09:42:21.331 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=113, tag_tid=18, entry_tid=1' 2024-04-06T09:42:21.331 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:42:21.331 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:42:21.333 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:42:21.334 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=100, tag_tid=17, entry_tid=7508' 2024-04-06T09:42:21.334 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:42:21.334 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:42:21.336 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=7508 2024-04-06T09:42:21.336 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 18 = 17 -a 1 -le 7508 2024-04-06T09:42:21.336 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:42:21.336 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:42:21.336 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:42:21.539 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:42:21.540 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:42:21.540 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:42:21.540 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:42:21.540 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:42:21.540 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:42:21.540 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:42:21.540 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:42:21.540 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:42:21.540 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:42:21.540 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:42:21.540 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:42:21.540 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:42:21.540 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:42:21.540 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:42:21.541 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:42:21.541 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:42:21.543 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:42:21.543 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:42:21.544 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:42:21.544 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:42:21.544 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:42:21.544 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:42:21.544 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:42:21.544 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:42:21.544 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:42:21.544 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:42:21.544 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:42:21.544 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:42:21.544 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:42:21.544 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:42:21.544 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:42:21.544 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:42:21.545 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:42:21.545 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:42:21.545 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:42:21.545 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:42:21.545 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:42:21.545 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:42:21.545 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:42:21.545 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:42:21.545 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:42:21.545 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:42:21.545 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:42:21.545 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:42:21.545 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:42:21.545 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:42:21.545 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:42:21.546 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:42:21.568 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:42:21.569 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:42:21.569 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:42:21.569 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:42:21.705 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:42:21.706 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:42:21.706 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:42:21.707 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:42:21.707 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:42:21.707 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:42:21.707 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:42:21.707 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:42:21.707 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:42:21.707 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:42:21.707 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:42:21.707 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:42:21.708 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:42:21.710 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:42:21.710 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:42:21.711 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:42:21.757 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 25 2024-04-06T09:42:21.758 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 28 2024-04-06T09:42:21.758 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:42:21.758 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=113, tag_tid=18, entry_tid=1], [object_number=112, tag_tid=18, entry_tid=0], [object_number=114, tag_tid=17, entry_tid=20202], [object_number=115, tag_tid=17, entry_tid=20199]]], state=connected] 2024-04-06T09:42:21.758 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=102, tag_tid=17, entry_tid=7622], [object_number=101, tag_tid=17, entry_tid=7621], [object_number=100, tag_tid=17, entry_tid=7620], [object_number=103, tag_tid=17, entry_tid=7619]]], state=connected] 2024-04-06T09:42:21.761 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:42:21.762 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=113, tag_tid=18, entry_tid=1' 2024-04-06T09:42:21.763 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:42:21.764 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:42:21.764 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:42:21.764 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:42:21.764 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:42:21.764 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:42:21.764 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:42:21.764 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:42:21.764 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:42:21.764 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:42:21.764 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:42:21.764 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:42:21.765 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:42:21.765 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:42:21.765 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:42:21.811 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 25 2024-04-06T09:42:21.811 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 28 2024-04-06T09:42:21.811 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:42:21.811 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=113, tag_tid=18, entry_tid=1], [object_number=112, tag_tid=18, entry_tid=0], [object_number=114, tag_tid=17, entry_tid=20202], [object_number=115, tag_tid=17, entry_tid=20199]]], state=connected] 2024-04-06T09:42:21.811 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=102, tag_tid=17, entry_tid=7622], [object_number=101, tag_tid=17, entry_tid=7621], [object_number=100, tag_tid=17, entry_tid=7620], [object_number=103, tag_tid=17, entry_tid=7619]]], state=connected] 2024-04-06T09:42:21.814 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:42:21.816 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=102, tag_tid=17, entry_tid=7622' 2024-04-06T09:42:21.816 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=113, tag_tid=18, entry_tid=1' -a 'object_number=113, tag_tid=18, entry_tid=1' = 'object_number=102, tag_tid=17, entry_tid=7622' 2024-04-06T09:42:21.816 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=102, tag_tid=17, entry_tid=7622' '!=' 'object_number=100, tag_tid=17, entry_tid=7508' 2024-04-06T09:42:21.816 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:42:21.816 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=102, tag_tid=17, entry_tid=7622' = 'object_number=100, tag_tid=17, entry_tid=7508' 2024-04-06T09:42:21.816 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=102, tag_tid=17, entry_tid=7622' 2024-04-06T09:42:21.817 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=113, tag_tid=18, entry_tid=1' 2024-04-06T09:42:21.817 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:42:21.817 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:42:21.818 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=18 2024-04-06T09:42:21.818 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=102, tag_tid=17, entry_tid=7622' 2024-04-06T09:42:21.818 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:42:21.818 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:42:21.819 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=17 2024-04-06T09:42:21.820 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=113, tag_tid=18, entry_tid=1' 2024-04-06T09:42:21.820 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:42:21.820 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:42:21.821 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:42:21.822 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=102, tag_tid=17, entry_tid=7622' 2024-04-06T09:42:21.822 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:42:21.822 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:42:21.823 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=7622 2024-04-06T09:42:21.823 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 18 = 17 -a 1 -le 7622 2024-04-06T09:42:21.823 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:42:21.823 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:42:21.823 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:42:22.025 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:42:22.025 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:42:22.025 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:42:22.025 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:42:22.026 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:42:22.026 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:42:22.026 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:42:22.026 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:42:22.026 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:42:22.026 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:42:22.026 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:42:22.026 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:42:22.026 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:42:22.026 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:42:22.026 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:42:22.027 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:42:22.027 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:42:22.030 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:42:22.030 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:42:22.030 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:42:22.030 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:42:22.031 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:42:22.031 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:42:22.031 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:42:22.031 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:42:22.031 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:42:22.031 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:42:22.031 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:42:22.031 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:42:22.031 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:42:22.031 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:42:22.031 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:42:22.031 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:42:22.031 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:42:22.031 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:42:22.032 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:42:22.032 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:42:22.032 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:42:22.032 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:42:22.032 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:42:22.032 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:42:22.032 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:42:22.032 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:42:22.032 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:42:22.032 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:42:22.032 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:42:22.032 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:42:22.032 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:42:22.032 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:42:22.055 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:42:22.055 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:42:22.055 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:42:22.056 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:42:22.215 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:42:22.216 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:42:22.216 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:42:22.216 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:42:22.216 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:42:22.217 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:42:22.217 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:42:22.217 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:42:22.217 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:42:22.217 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:42:22.217 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:42:22.217 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:42:22.217 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:42:22.219 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:42:22.219 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:42:22.219 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:42:22.482 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 25 2024-04-06T09:42:22.482 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 28 2024-04-06T09:42:22.482 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:42:22.482 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=113, tag_tid=18, entry_tid=1], [object_number=112, tag_tid=18, entry_tid=0], [object_number=114, tag_tid=17, entry_tid=20202], [object_number=115, tag_tid=17, entry_tid=20199]]], state=connected] 2024-04-06T09:42:22.482 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=100, tag_tid=17, entry_tid=7684], [object_number=103, tag_tid=17, entry_tid=7683], [object_number=102, tag_tid=17, entry_tid=7682], [object_number=101, tag_tid=17, entry_tid=7681]]], state=connected] 2024-04-06T09:42:22.486 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:42:22.488 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=113, tag_tid=18, entry_tid=1' 2024-04-06T09:42:22.489 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:42:22.489 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:42:22.489 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:42:22.489 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:42:22.489 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:42:22.489 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:42:22.489 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:42:22.489 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:42:22.490 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:42:22.490 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:42:22.490 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:42:22.490 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:42:22.491 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:42:22.491 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:42:22.491 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:42:22.544 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 25 2024-04-06T09:42:22.544 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 28 2024-04-06T09:42:22.544 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:42:22.544 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=113, tag_tid=18, entry_tid=1], [object_number=112, tag_tid=18, entry_tid=0], [object_number=114, tag_tid=17, entry_tid=20202], [object_number=115, tag_tid=17, entry_tid=20199]]], state=connected] 2024-04-06T09:42:22.544 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=100, tag_tid=17, entry_tid=7684], [object_number=103, tag_tid=17, entry_tid=7683], [object_number=102, tag_tid=17, entry_tid=7682], [object_number=101, tag_tid=17, entry_tid=7681]]], state=connected] 2024-04-06T09:42:22.548 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:42:22.550 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=100, tag_tid=17, entry_tid=7684' 2024-04-06T09:42:22.550 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=113, tag_tid=18, entry_tid=1' -a 'object_number=113, tag_tid=18, entry_tid=1' = 'object_number=100, tag_tid=17, entry_tid=7684' 2024-04-06T09:42:22.550 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=100, tag_tid=17, entry_tid=7684' '!=' 'object_number=102, tag_tid=17, entry_tid=7622' 2024-04-06T09:42:22.550 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:42:22.550 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=100, tag_tid=17, entry_tid=7684' = 'object_number=102, tag_tid=17, entry_tid=7622' 2024-04-06T09:42:22.550 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=100, tag_tid=17, entry_tid=7684' 2024-04-06T09:42:22.550 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=113, tag_tid=18, entry_tid=1' 2024-04-06T09:42:22.550 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:42:22.550 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:42:22.551 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=18 2024-04-06T09:42:22.552 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=100, tag_tid=17, entry_tid=7684' 2024-04-06T09:42:22.552 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:42:22.552 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:42:22.553 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=17 2024-04-06T09:42:22.553 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=113, tag_tid=18, entry_tid=1' 2024-04-06T09:42:22.554 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:42:22.554 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:42:22.555 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:42:22.555 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=100, tag_tid=17, entry_tid=7684' 2024-04-06T09:42:22.556 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:42:22.556 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:42:22.557 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=7684 2024-04-06T09:42:22.557 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 18 = 17 -a 1 -le 7684 2024-04-06T09:42:22.557 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:42:22.557 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:42:22.557 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:42:22.760 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:42:22.760 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:42:22.760 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:42:22.760 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:42:22.760 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:42:22.760 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:42:22.760 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:42:22.760 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:42:22.760 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:42:22.761 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:42:22.761 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:42:22.761 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:42:22.761 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:42:22.761 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:42:22.761 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:42:22.761 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:42:22.761 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:42:22.764 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:42:22.764 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:42:22.764 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:42:22.764 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:42:22.764 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:42:22.765 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:42:22.765 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:42:22.765 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:42:22.765 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:42:22.765 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:42:22.765 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:42:22.765 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:42:22.765 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:42:22.765 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:42:22.765 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:42:22.765 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:42:22.765 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:42:22.765 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:42:22.765 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:42:22.765 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:42:22.766 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:42:22.766 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:42:22.766 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:42:22.766 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:42:22.766 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:42:22.766 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:42:22.766 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:42:22.766 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:42:22.766 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:42:22.766 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:42:22.766 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:42:22.766 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:42:22.791 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:42:22.792 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:42:22.792 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:42:22.792 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:42:22.931 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:42:22.932 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:42:22.932 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:42:22.932 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:42:22.932 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:42:22.932 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:42:22.932 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:42:22.932 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:42:22.932 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:42:22.932 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:42:22.933 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:42:22.933 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:42:22.933 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:42:22.934 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:42:22.935 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:42:22.935 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:42:22.983 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 25 2024-04-06T09:42:22.983 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 28 2024-04-06T09:42:22.983 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:42:22.983 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=113, tag_tid=18, entry_tid=1], [object_number=112, tag_tid=18, entry_tid=0], [object_number=114, tag_tid=17, entry_tid=20202], [object_number=115, tag_tid=17, entry_tid=20199]]], state=connected] 2024-04-06T09:42:22.984 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=103, tag_tid=17, entry_tid=7775], [object_number=102, tag_tid=17, entry_tid=7774], [object_number=101, tag_tid=17, entry_tid=7773], [object_number=100, tag_tid=17, entry_tid=7772]]], state=connected] 2024-04-06T09:42:22.987 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:42:22.988 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=113, tag_tid=18, entry_tid=1' 2024-04-06T09:42:22.989 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:42:22.989 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:42:22.989 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:42:22.989 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:42:22.989 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:42:22.990 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:42:22.990 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:42:22.990 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:42:22.990 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:42:22.990 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:42:22.990 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:42:22.990 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:42:22.991 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:42:22.991 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:42:22.991 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:42:23.037 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 25 2024-04-06T09:42:23.037 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 28 2024-04-06T09:42:23.037 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:42:23.037 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=113, tag_tid=18, entry_tid=1], [object_number=112, tag_tid=18, entry_tid=0], [object_number=114, tag_tid=17, entry_tid=20202], [object_number=115, tag_tid=17, entry_tid=20199]]], state=connected] 2024-04-06T09:42:23.038 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=103, tag_tid=17, entry_tid=7775], [object_number=102, tag_tid=17, entry_tid=7774], [object_number=101, tag_tid=17, entry_tid=7773], [object_number=100, tag_tid=17, entry_tid=7772]]], state=connected] 2024-04-06T09:42:23.041 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:42:23.043 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=103, tag_tid=17, entry_tid=7775' 2024-04-06T09:42:23.043 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=113, tag_tid=18, entry_tid=1' -a 'object_number=113, tag_tid=18, entry_tid=1' = 'object_number=103, tag_tid=17, entry_tid=7775' 2024-04-06T09:42:23.043 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=103, tag_tid=17, entry_tid=7775' '!=' 'object_number=100, tag_tid=17, entry_tid=7684' 2024-04-06T09:42:23.043 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:42:23.043 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=103, tag_tid=17, entry_tid=7775' = 'object_number=100, tag_tid=17, entry_tid=7684' 2024-04-06T09:42:23.043 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=103, tag_tid=17, entry_tid=7775' 2024-04-06T09:42:23.043 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=113, tag_tid=18, entry_tid=1' 2024-04-06T09:42:23.043 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:42:23.043 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:42:23.044 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=18 2024-04-06T09:42:23.045 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=103, tag_tid=17, entry_tid=7775' 2024-04-06T09:42:23.045 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:42:23.045 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:42:23.045 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=17 2024-04-06T09:42:23.046 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=113, tag_tid=18, entry_tid=1' 2024-04-06T09:42:23.046 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:42:23.046 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:42:23.047 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:42:23.048 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=103, tag_tid=17, entry_tid=7775' 2024-04-06T09:42:23.048 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:42:23.048 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:42:23.049 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=7775 2024-04-06T09:42:23.049 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 18 = 17 -a 1 -le 7775 2024-04-06T09:42:23.049 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:42:23.049 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:42:23.050 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:42:23.251 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:42:23.252 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:42:23.252 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:42:23.252 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:42:23.252 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:42:23.252 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:42:23.252 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:42:23.252 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:42:23.252 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:42:23.252 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:42:23.252 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:42:23.253 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:42:23.253 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:42:23.253 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:42:23.253 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:42:23.253 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:42:23.253 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:42:23.257 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:42:23.257 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:42:23.257 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:42:23.257 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:42:23.257 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:42:23.257 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:42:23.257 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:42:23.257 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:42:23.257 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:42:23.257 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:42:23.258 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:42:23.258 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:42:23.258 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:42:23.258 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:42:23.258 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:42:23.258 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:42:23.258 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:42:23.258 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:42:23.258 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:42:23.258 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:42:23.258 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:42:23.258 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:42:23.258 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:42:23.258 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:42:23.259 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:42:23.259 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:42:23.259 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:42:23.259 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:42:23.259 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:42:23.259 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:42:23.259 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:42:23.259 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:42:23.281 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:42:23.282 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:42:23.282 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:42:23.282 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:42:23.417 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:42:23.418 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:42:23.418 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:42:23.418 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:42:23.419 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:42:23.419 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:42:23.419 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:42:23.419 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:42:23.419 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:42:23.419 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:42:23.420 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:42:23.420 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:42:23.420 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:42:23.421 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:42:23.421 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:42:23.421 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:42:23.679 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 25 2024-04-06T09:42:23.680 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 28 2024-04-06T09:42:23.680 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:42:23.680 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=113, tag_tid=18, entry_tid=1], [object_number=112, tag_tid=18, entry_tid=0], [object_number=114, tag_tid=17, entry_tid=20202], [object_number=115, tag_tid=17, entry_tid=20199]]], state=connected] 2024-04-06T09:42:23.680 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=100, tag_tid=17, entry_tid=7860], [object_number=103, tag_tid=17, entry_tid=7859], [object_number=102, tag_tid=17, entry_tid=7858], [object_number=101, tag_tid=17, entry_tid=7857]]], state=connected] 2024-04-06T09:42:23.684 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:42:23.686 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=113, tag_tid=18, entry_tid=1' 2024-04-06T09:42:23.687 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:42:23.687 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:42:23.687 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:42:23.687 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:42:23.687 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:42:23.687 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:42:23.687 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:42:23.687 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:42:23.687 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:42:23.688 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:42:23.688 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:42:23.688 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:42:23.690 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:42:23.690 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:42:23.690 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:42:23.746 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 25 2024-04-06T09:42:23.746 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 28 2024-04-06T09:42:23.746 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:42:23.746 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=113, tag_tid=18, entry_tid=1], [object_number=112, tag_tid=18, entry_tid=0], [object_number=114, tag_tid=17, entry_tid=20202], [object_number=115, tag_tid=17, entry_tid=20199]]], state=connected] 2024-04-06T09:42:23.746 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=100, tag_tid=17, entry_tid=7860], [object_number=103, tag_tid=17, entry_tid=7859], [object_number=102, tag_tid=17, entry_tid=7858], [object_number=101, tag_tid=17, entry_tid=7857]]], state=connected] 2024-04-06T09:42:23.750 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:42:23.751 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=100, tag_tid=17, entry_tid=7860' 2024-04-06T09:42:23.752 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=113, tag_tid=18, entry_tid=1' -a 'object_number=113, tag_tid=18, entry_tid=1' = 'object_number=100, tag_tid=17, entry_tid=7860' 2024-04-06T09:42:23.752 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=100, tag_tid=17, entry_tid=7860' '!=' 'object_number=103, tag_tid=17, entry_tid=7775' 2024-04-06T09:42:23.752 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:42:23.752 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=100, tag_tid=17, entry_tid=7860' = 'object_number=103, tag_tid=17, entry_tid=7775' 2024-04-06T09:42:23.752 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=100, tag_tid=17, entry_tid=7860' 2024-04-06T09:42:23.752 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=113, tag_tid=18, entry_tid=1' 2024-04-06T09:42:23.753 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:42:23.753 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:42:23.753 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=18 2024-04-06T09:42:23.754 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=100, tag_tid=17, entry_tid=7860' 2024-04-06T09:42:23.754 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:42:23.754 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:42:23.756 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=17 2024-04-06T09:42:23.756 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=113, tag_tid=18, entry_tid=1' 2024-04-06T09:42:23.756 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:42:23.756 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:42:23.757 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:42:23.758 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=100, tag_tid=17, entry_tid=7860' 2024-04-06T09:42:23.758 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:42:23.758 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:42:23.759 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=7860 2024-04-06T09:42:23.759 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 18 = 17 -a 1 -le 7860 2024-04-06T09:42:23.759 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:42:23.759 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:42:23.759 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:42:23.961 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:42:23.961 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:42:23.961 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:42:23.961 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:42:23.962 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:42:23.962 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:42:23.962 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:42:23.962 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:42:23.962 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:42:23.962 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:42:23.962 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:42:23.962 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:42:23.962 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:42:23.962 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:42:23.962 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:42:23.963 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:42:23.963 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:42:23.966 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:42:23.966 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:42:23.966 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:42:23.966 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:42:23.966 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:42:23.966 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:42:23.967 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:42:23.967 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:42:23.967 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:42:23.967 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:42:23.967 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:42:23.967 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:42:23.967 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:42:23.967 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:42:23.967 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:42:23.967 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:42:23.967 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:42:23.967 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:42:23.967 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:42:23.967 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:42:23.967 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:42:23.968 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:42:23.968 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:42:23.968 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:42:23.968 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:42:23.968 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:42:23.968 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:42:23.968 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:42:23.968 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:42:23.968 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:42:23.968 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:42:23.968 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:42:23.991 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:42:23.991 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:42:23.991 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:42:23.991 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:42:24.121 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:42:24.122 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:42:24.123 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:42:24.123 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:42:24.123 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:42:24.123 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:42:24.123 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:42:24.123 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:42:24.123 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:42:24.123 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:42:24.123 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:42:24.124 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:42:24.124 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:42:24.126 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:42:24.126 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:42:24.126 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:42:24.178 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 25 2024-04-06T09:42:24.179 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 28 2024-04-06T09:42:24.179 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:42:24.179 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=113, tag_tid=18, entry_tid=1], [object_number=112, tag_tid=18, entry_tid=0], [object_number=114, tag_tid=17, entry_tid=20202], [object_number=115, tag_tid=17, entry_tid=20199]]], state=connected] 2024-04-06T09:42:24.179 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=103, tag_tid=17, entry_tid=7983], [object_number=102, tag_tid=17, entry_tid=7982], [object_number=101, tag_tid=17, entry_tid=7981], [object_number=100, tag_tid=17, entry_tid=7980]]], state=connected] 2024-04-06T09:42:24.182 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:42:24.183 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=113, tag_tid=18, entry_tid=1' 2024-04-06T09:42:24.185 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:42:24.185 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:42:24.185 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:42:24.185 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:42:24.185 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:42:24.185 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:42:24.185 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:42:24.185 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:42:24.185 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:42:24.185 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:42:24.185 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:42:24.186 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:42:24.186 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:42:24.186 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:42:24.186 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:42:24.233 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 25 2024-04-06T09:42:24.233 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 28 2024-04-06T09:42:24.233 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:42:24.233 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=113, tag_tid=18, entry_tid=1], [object_number=112, tag_tid=18, entry_tid=0], [object_number=114, tag_tid=17, entry_tid=20202], [object_number=115, tag_tid=17, entry_tid=20199]]], state=connected] 2024-04-06T09:42:24.234 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=103, tag_tid=17, entry_tid=7983], [object_number=102, tag_tid=17, entry_tid=7982], [object_number=101, tag_tid=17, entry_tid=7981], [object_number=100, tag_tid=17, entry_tid=7980]]], state=connected] 2024-04-06T09:42:24.237 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:42:24.239 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=103, tag_tid=17, entry_tid=7983' 2024-04-06T09:42:24.239 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=113, tag_tid=18, entry_tid=1' -a 'object_number=113, tag_tid=18, entry_tid=1' = 'object_number=103, tag_tid=17, entry_tid=7983' 2024-04-06T09:42:24.240 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=103, tag_tid=17, entry_tid=7983' '!=' 'object_number=100, tag_tid=17, entry_tid=7860' 2024-04-06T09:42:24.240 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:42:24.240 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=103, tag_tid=17, entry_tid=7983' = 'object_number=100, tag_tid=17, entry_tid=7860' 2024-04-06T09:42:24.240 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=103, tag_tid=17, entry_tid=7983' 2024-04-06T09:42:24.240 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=113, tag_tid=18, entry_tid=1' 2024-04-06T09:42:24.240 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:42:24.240 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:42:24.241 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=18 2024-04-06T09:42:24.241 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=103, tag_tid=17, entry_tid=7983' 2024-04-06T09:42:24.241 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:42:24.241 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:42:24.242 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=17 2024-04-06T09:42:24.243 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=113, tag_tid=18, entry_tid=1' 2024-04-06T09:42:24.243 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:42:24.243 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:42:24.244 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:42:24.244 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=103, tag_tid=17, entry_tid=7983' 2024-04-06T09:42:24.244 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:42:24.244 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:42:24.246 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=7983 2024-04-06T09:42:24.246 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 18 = 17 -a 1 -le 7983 2024-04-06T09:42:24.246 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:42:24.246 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:42:24.246 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:42:24.447 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:42:24.447 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:42:24.447 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:42:24.448 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:42:24.448 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:42:24.448 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:42:24.448 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:42:24.448 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:42:24.448 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:42:24.448 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:42:24.448 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:42:24.448 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:42:24.448 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:42:24.448 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:42:24.449 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:42:24.449 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:42:24.449 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:42:24.451 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:42:24.452 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:42:24.452 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:42:24.452 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:42:24.452 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:42:24.452 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:42:24.452 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:42:24.452 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:42:24.452 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:42:24.452 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:42:24.452 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:42:24.452 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:42:24.452 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:42:24.452 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:42:24.452 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:42:24.452 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:42:24.453 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:42:24.453 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:42:24.453 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:42:24.453 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:42:24.453 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:42:24.453 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:42:24.453 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:42:24.453 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:42:24.453 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:42:24.453 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:42:24.453 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:42:24.453 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:42:24.453 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:42:24.453 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:42:24.453 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:42:24.453 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:42:24.477 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:42:24.477 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:42:24.478 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:42:24.478 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:42:24.607 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:42:24.608 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:42:24.608 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:42:24.608 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:42:24.609 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:42:24.609 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:42:24.609 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:42:24.609 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:42:24.609 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:42:24.609 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:42:24.609 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:42:24.609 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:42:24.610 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:42:24.611 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:42:24.611 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:42:24.611 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:42:24.658 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 25 2024-04-06T09:42:24.658 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 28 2024-04-06T09:42:24.658 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:42:24.658 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=113, tag_tid=18, entry_tid=1], [object_number=112, tag_tid=18, entry_tid=0], [object_number=114, tag_tid=17, entry_tid=20202], [object_number=115, tag_tid=17, entry_tid=20199]]], state=connected] 2024-04-06T09:42:24.658 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=101, tag_tid=17, entry_tid=8089], [object_number=100, tag_tid=17, entry_tid=8088], [object_number=103, tag_tid=17, entry_tid=8087], [object_number=102, tag_tid=17, entry_tid=8086]]], state=connected] 2024-04-06T09:42:24.661 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:42:24.663 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=113, tag_tid=18, entry_tid=1' 2024-04-06T09:42:24.664 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:42:24.664 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:42:24.664 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:42:24.664 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:42:24.664 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:42:24.664 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:42:24.664 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:42:24.664 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:42:24.664 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:42:24.664 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:42:24.664 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:42:24.665 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:42:24.665 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:42:24.665 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:42:24.665 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:42:24.712 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 25 2024-04-06T09:42:24.712 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 28 2024-04-06T09:42:24.712 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:42:24.712 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=113, tag_tid=18, entry_tid=1], [object_number=112, tag_tid=18, entry_tid=0], [object_number=114, tag_tid=17, entry_tid=20202], [object_number=115, tag_tid=17, entry_tid=20199]]], state=connected] 2024-04-06T09:42:24.712 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=101, tag_tid=17, entry_tid=8089], [object_number=100, tag_tid=17, entry_tid=8088], [object_number=103, tag_tid=17, entry_tid=8087], [object_number=102, tag_tid=17, entry_tid=8086]]], state=connected] 2024-04-06T09:42:24.716 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:42:24.717 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=101, tag_tid=17, entry_tid=8089' 2024-04-06T09:42:24.717 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=113, tag_tid=18, entry_tid=1' -a 'object_number=113, tag_tid=18, entry_tid=1' = 'object_number=101, tag_tid=17, entry_tid=8089' 2024-04-06T09:42:24.718 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=101, tag_tid=17, entry_tid=8089' '!=' 'object_number=103, tag_tid=17, entry_tid=7983' 2024-04-06T09:42:24.718 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:42:24.718 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=101, tag_tid=17, entry_tid=8089' = 'object_number=103, tag_tid=17, entry_tid=7983' 2024-04-06T09:42:24.718 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=101, tag_tid=17, entry_tid=8089' 2024-04-06T09:42:24.718 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=113, tag_tid=18, entry_tid=1' 2024-04-06T09:42:24.718 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:42:24.719 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:42:24.719 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=18 2024-04-06T09:42:24.720 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=101, tag_tid=17, entry_tid=8089' 2024-04-06T09:42:24.720 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:42:24.720 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:42:24.721 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=17 2024-04-06T09:42:24.721 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=113, tag_tid=18, entry_tid=1' 2024-04-06T09:42:24.721 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:42:24.721 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:42:24.722 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:42:24.723 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=101, tag_tid=17, entry_tid=8089' 2024-04-06T09:42:24.723 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:42:24.723 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:42:24.724 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=8089 2024-04-06T09:42:24.724 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 18 = 17 -a 1 -le 8089 2024-04-06T09:42:24.725 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:42:24.725 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:42:24.725 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:42:24.927 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:42:24.927 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:42:24.927 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:42:24.927 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:42:24.927 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:42:24.927 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:42:24.927 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:42:24.927 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:42:24.928 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:42:24.928 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:42:24.928 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:42:24.928 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:42:24.928 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:42:24.928 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:42:24.928 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:42:24.928 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:42:24.929 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:42:24.930 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:42:24.930 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:42:24.930 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:42:24.930 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:42:24.930 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:42:24.930 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:42:24.930 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:42:24.930 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:42:24.930 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:42:24.930 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:42:24.931 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:42:24.931 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:42:24.931 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:42:24.931 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:42:24.931 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:42:24.931 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:42:24.932 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:42:24.933 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:42:24.933 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:42:24.933 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:42:24.933 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:42:24.933 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:42:24.933 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:42:24.933 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:42:24.933 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:42:24.933 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:42:24.933 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:42:24.933 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:42:24.933 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:42:24.933 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:42:24.933 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:42:24.934 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:42:24.959 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:42:24.959 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:42:24.959 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:42:24.959 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:42:25.089 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:42:25.091 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:42:25.091 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:42:25.091 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:42:25.091 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:42:25.091 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:42:25.091 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:42:25.091 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:42:25.091 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:42:25.091 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:42:25.091 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:42:25.092 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:42:25.092 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:42:25.094 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:42:25.095 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:42:25.095 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:42:25.147 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 25 2024-04-06T09:42:25.148 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 28 2024-04-06T09:42:25.148 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:42:25.148 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=113, tag_tid=18, entry_tid=1], [object_number=112, tag_tid=18, entry_tid=0], [object_number=114, tag_tid=17, entry_tid=20202], [object_number=115, tag_tid=17, entry_tid=20199]]], state=connected] 2024-04-06T09:42:25.148 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=100, tag_tid=17, entry_tid=8196], [object_number=103, tag_tid=17, entry_tid=8195], [object_number=102, tag_tid=17, entry_tid=8194], [object_number=101, tag_tid=17, entry_tid=8193]]], state=connected] 2024-04-06T09:42:25.151 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:42:25.153 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=113, tag_tid=18, entry_tid=1' 2024-04-06T09:42:25.154 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:42:25.154 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:42:25.154 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:42:25.154 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:42:25.154 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:42:25.154 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:42:25.154 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:42:25.154 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:42:25.154 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:42:25.154 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:42:25.155 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:42:25.155 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:42:25.155 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:42:25.155 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:42:25.155 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:42:25.202 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 25 2024-04-06T09:42:25.202 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 28 2024-04-06T09:42:25.202 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:42:25.202 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=113, tag_tid=18, entry_tid=1], [object_number=112, tag_tid=18, entry_tid=0], [object_number=114, tag_tid=17, entry_tid=20202], [object_number=115, tag_tid=17, entry_tid=20199]]], state=connected] 2024-04-06T09:42:25.202 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=100, tag_tid=17, entry_tid=8196], [object_number=103, tag_tid=17, entry_tid=8195], [object_number=102, tag_tid=17, entry_tid=8194], [object_number=101, tag_tid=17, entry_tid=8193]]], state=connected] 2024-04-06T09:42:25.206 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:42:25.207 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=100, tag_tid=17, entry_tid=8196' 2024-04-06T09:42:25.207 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=113, tag_tid=18, entry_tid=1' -a 'object_number=113, tag_tid=18, entry_tid=1' = 'object_number=100, tag_tid=17, entry_tid=8196' 2024-04-06T09:42:25.208 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=100, tag_tid=17, entry_tid=8196' '!=' 'object_number=101, tag_tid=17, entry_tid=8089' 2024-04-06T09:42:25.208 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:42:25.208 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=100, tag_tid=17, entry_tid=8196' = 'object_number=101, tag_tid=17, entry_tid=8089' 2024-04-06T09:42:25.208 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=100, tag_tid=17, entry_tid=8196' 2024-04-06T09:42:25.208 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=113, tag_tid=18, entry_tid=1' 2024-04-06T09:42:25.208 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:42:25.208 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:42:25.209 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=18 2024-04-06T09:42:25.210 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=100, tag_tid=17, entry_tid=8196' 2024-04-06T09:42:25.210 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:42:25.210 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:42:25.211 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=17 2024-04-06T09:42:25.211 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=113, tag_tid=18, entry_tid=1' 2024-04-06T09:42:25.211 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:42:25.212 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:42:25.212 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:42:25.213 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=100, tag_tid=17, entry_tid=8196' 2024-04-06T09:42:25.213 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:42:25.213 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:42:25.214 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=8196 2024-04-06T09:42:25.214 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 18 = 17 -a 1 -le 8196 2024-04-06T09:42:25.214 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:42:25.214 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:42:25.215 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:42:25.416 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:42:25.417 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:42:25.417 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:42:25.417 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:42:25.417 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:42:25.417 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:42:25.417 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:42:25.417 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:42:25.417 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:42:25.417 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:42:25.417 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:42:25.417 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:42:25.417 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:42:25.417 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:42:25.418 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:42:25.418 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:42:25.418 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:42:25.420 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:42:25.420 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:42:25.420 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:42:25.420 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:42:25.420 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:42:25.420 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:42:25.421 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:42:25.421 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:42:25.421 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:42:25.421 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:42:25.421 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:42:25.421 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:42:25.421 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:42:25.421 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:42:25.421 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:42:25.421 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:42:25.422 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:42:25.422 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:42:25.422 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:42:25.423 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:42:25.423 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:42:25.423 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:42:25.423 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:42:25.423 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:42:25.423 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:42:25.423 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:42:25.423 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:42:25.423 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:42:25.423 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:42:25.423 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:42:25.423 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:42:25.423 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:42:25.449 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:42:25.449 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:42:25.449 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:42:25.449 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:42:25.583 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:42:25.584 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:42:25.584 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:42:25.584 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:42:25.584 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:42:25.584 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:42:25.584 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:42:25.584 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:42:25.585 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:42:25.585 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:42:25.585 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:42:25.585 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:42:25.585 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:42:25.587 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:42:25.587 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:42:25.587 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:42:25.640 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 25 2024-04-06T09:42:25.640 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 28 2024-04-06T09:42:25.640 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:42:25.640 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=113, tag_tid=18, entry_tid=1], [object_number=112, tag_tid=18, entry_tid=0], [object_number=114, tag_tid=17, entry_tid=20202], [object_number=115, tag_tid=17, entry_tid=20199]]], state=connected] 2024-04-06T09:42:25.640 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=102, tag_tid=17, entry_tid=8290], [object_number=101, tag_tid=17, entry_tid=8289], [object_number=100, tag_tid=17, entry_tid=8288], [object_number=103, tag_tid=17, entry_tid=8287]]], state=connected] 2024-04-06T09:42:25.644 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:42:25.645 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=113, tag_tid=18, entry_tid=1' 2024-04-06T09:42:25.646 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:42:25.646 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:42:25.646 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:42:25.647 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:42:25.647 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:42:25.647 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:42:25.647 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:42:25.647 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:42:25.647 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:42:25.647 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:42:25.647 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:42:25.647 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:42:25.648 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:42:25.648 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:42:25.648 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:42:25.694 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 25 2024-04-06T09:42:25.694 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 28 2024-04-06T09:42:25.694 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:42:25.694 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=113, tag_tid=18, entry_tid=1], [object_number=112, tag_tid=18, entry_tid=0], [object_number=114, tag_tid=17, entry_tid=20202], [object_number=115, tag_tid=17, entry_tid=20199]]], state=connected] 2024-04-06T09:42:25.695 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=102, tag_tid=17, entry_tid=8290], [object_number=101, tag_tid=17, entry_tid=8289], [object_number=100, tag_tid=17, entry_tid=8288], [object_number=103, tag_tid=17, entry_tid=8287]]], state=connected] 2024-04-06T09:42:25.697 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:42:25.700 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=102, tag_tid=17, entry_tid=8290' 2024-04-06T09:42:25.700 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=113, tag_tid=18, entry_tid=1' -a 'object_number=113, tag_tid=18, entry_tid=1' = 'object_number=102, tag_tid=17, entry_tid=8290' 2024-04-06T09:42:25.700 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=102, tag_tid=17, entry_tid=8290' '!=' 'object_number=100, tag_tid=17, entry_tid=8196' 2024-04-06T09:42:25.700 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:42:25.700 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=102, tag_tid=17, entry_tid=8290' = 'object_number=100, tag_tid=17, entry_tid=8196' 2024-04-06T09:42:25.700 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=102, tag_tid=17, entry_tid=8290' 2024-04-06T09:42:25.700 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=113, tag_tid=18, entry_tid=1' 2024-04-06T09:42:25.700 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:42:25.700 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:42:25.701 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=18 2024-04-06T09:42:25.701 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=102, tag_tid=17, entry_tid=8290' 2024-04-06T09:42:25.702 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:42:25.702 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:42:25.702 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=17 2024-04-06T09:42:25.703 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=113, tag_tid=18, entry_tid=1' 2024-04-06T09:42:25.703 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:42:25.703 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:42:25.704 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:42:25.705 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=102, tag_tid=17, entry_tid=8290' 2024-04-06T09:42:25.705 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:42:25.705 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:42:25.706 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=8290 2024-04-06T09:42:25.706 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 18 = 17 -a 1 -le 8290 2024-04-06T09:42:25.706 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:42:25.706 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:42:25.707 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:42:25.908 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:42:25.908 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:42:25.908 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:42:25.908 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:42:25.908 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:42:25.908 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:42:25.908 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:42:25.908 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:42:25.909 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:42:25.909 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:42:25.909 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:42:25.909 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:42:25.909 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:42:25.909 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:42:25.909 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:42:25.909 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:42:25.910 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:42:25.912 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:42:25.912 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:42:25.912 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:42:25.912 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:42:25.912 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:42:25.912 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:42:25.912 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:42:25.912 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:42:25.912 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:42:25.913 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:42:25.913 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:42:25.913 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:42:25.913 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:42:25.913 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:42:25.913 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:42:25.913 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:42:25.913 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:42:25.913 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:42:25.913 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:42:25.913 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:42:25.913 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:42:25.913 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:42:25.913 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:42:25.914 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:42:25.914 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:42:25.914 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:42:25.914 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:42:25.914 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:42:25.914 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:42:25.915 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:42:25.915 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:42:25.915 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:42:25.938 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:42:25.938 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:42:25.938 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:42:25.938 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:42:26.068 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:42:26.069 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:42:26.069 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:42:26.069 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:42:26.069 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:42:26.070 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:42:26.070 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:42:26.070 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:42:26.070 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:42:26.070 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:42:26.070 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:42:26.070 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:42:26.070 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:42:26.072 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:42:26.072 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:42:26.072 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:42:26.124 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 25 2024-04-06T09:42:26.125 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 28 2024-04-06T09:42:26.125 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:42:26.125 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=113, tag_tid=18, entry_tid=1], [object_number=112, tag_tid=18, entry_tid=0], [object_number=114, tag_tid=17, entry_tid=20202], [object_number=115, tag_tid=17, entry_tid=20199]]], state=connected] 2024-04-06T09:42:26.125 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=102, tag_tid=17, entry_tid=8370], [object_number=101, tag_tid=17, entry_tid=8369], [object_number=100, tag_tid=17, entry_tid=8368], [object_number=103, tag_tid=17, entry_tid=8367]]], state=connected] 2024-04-06T09:42:26.128 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:42:26.129 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=113, tag_tid=18, entry_tid=1' 2024-04-06T09:42:26.131 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:42:26.131 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:42:26.131 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:42:26.131 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:42:26.131 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:42:26.131 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:42:26.131 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:42:26.131 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:42:26.131 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:42:26.131 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:42:26.131 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:42:26.131 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:42:26.132 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:42:26.132 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:42:26.132 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:42:26.180 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 25 2024-04-06T09:42:26.180 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 28 2024-04-06T09:42:26.180 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:42:26.181 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=113, tag_tid=18, entry_tid=1], [object_number=112, tag_tid=18, entry_tid=0], [object_number=114, tag_tid=17, entry_tid=20202], [object_number=115, tag_tid=17, entry_tid=20199]]], state=connected] 2024-04-06T09:42:26.181 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=102, tag_tid=17, entry_tid=8370], [object_number=101, tag_tid=17, entry_tid=8369], [object_number=100, tag_tid=17, entry_tid=8368], [object_number=103, tag_tid=17, entry_tid=8367]]], state=connected] 2024-04-06T09:42:26.185 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:42:26.187 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=102, tag_tid=17, entry_tid=8370' 2024-04-06T09:42:26.187 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=113, tag_tid=18, entry_tid=1' -a 'object_number=113, tag_tid=18, entry_tid=1' = 'object_number=102, tag_tid=17, entry_tid=8370' 2024-04-06T09:42:26.187 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=102, tag_tid=17, entry_tid=8370' '!=' 'object_number=102, tag_tid=17, entry_tid=8290' 2024-04-06T09:42:26.187 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:42:26.187 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=102, tag_tid=17, entry_tid=8370' = 'object_number=102, tag_tid=17, entry_tid=8290' 2024-04-06T09:42:26.187 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=102, tag_tid=17, entry_tid=8370' 2024-04-06T09:42:26.188 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=113, tag_tid=18, entry_tid=1' 2024-04-06T09:42:26.188 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:42:26.188 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:42:26.188 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=18 2024-04-06T09:42:26.188 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=102, tag_tid=17, entry_tid=8370' 2024-04-06T09:42:26.188 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:42:26.189 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:42:26.189 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=17 2024-04-06T09:42:26.189 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=113, tag_tid=18, entry_tid=1' 2024-04-06T09:42:26.190 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:42:26.190 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:42:26.190 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:42:26.191 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=102, tag_tid=17, entry_tid=8370' 2024-04-06T09:42:26.191 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:42:26.191 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:42:26.192 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=8370 2024-04-06T09:42:26.192 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 18 = 17 -a 1 -le 8370 2024-04-06T09:42:26.192 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:42:26.193 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:42:26.193 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:42:26.395 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:42:26.395 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:42:26.395 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:42:26.395 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:42:26.395 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:42:26.395 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:42:26.395 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:42:26.395 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:42:26.395 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:42:26.395 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:42:26.396 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:42:26.396 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:42:26.396 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:42:26.396 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:42:26.396 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:42:26.396 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:42:26.398 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:42:26.398 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:42:26.398 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:42:26.398 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:42:26.398 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:42:26.398 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:42:26.398 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:42:26.398 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:42:26.398 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:42:26.399 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:42:26.399 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:42:26.399 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:42:26.399 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:42:26.399 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:42:26.399 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:42:26.399 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:42:26.399 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:42:26.399 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:42:26.399 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:42:26.399 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:42:26.399 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:42:26.399 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:42:26.399 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:42:26.399 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:42:26.400 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:42:26.400 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:42:26.400 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:42:26.400 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:42:26.400 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:42:26.400 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:42:26.400 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:42:26.400 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:42:26.400 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:42:26.427 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:42:26.427 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:42:26.427 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:42:26.427 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:42:26.565 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:42:26.566 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:42:26.566 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:42:26.566 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:42:26.566 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:42:26.567 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:42:26.567 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:42:26.567 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:42:26.567 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:42:26.567 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:42:26.567 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:42:26.567 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:42:26.568 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:42:26.569 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:42:26.570 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:42:26.570 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:42:26.622 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 25 2024-04-06T09:42:26.622 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 28 2024-04-06T09:42:26.622 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:42:26.622 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=113, tag_tid=18, entry_tid=1], [object_number=112, tag_tid=18, entry_tid=0], [object_number=114, tag_tid=17, entry_tid=20202], [object_number=115, tag_tid=17, entry_tid=20199]]], state=connected] 2024-04-06T09:42:26.622 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=100, tag_tid=17, entry_tid=8456], [object_number=103, tag_tid=17, entry_tid=8455], [object_number=102, tag_tid=17, entry_tid=8454], [object_number=101, tag_tid=17, entry_tid=8453]]], state=connected] 2024-04-06T09:42:26.625 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:42:26.626 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=113, tag_tid=18, entry_tid=1' 2024-04-06T09:42:26.627 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:42:26.627 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:42:26.628 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:42:26.628 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:42:26.628 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:42:26.628 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:42:26.628 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:42:26.628 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:42:26.628 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:42:26.628 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:42:26.628 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:42:26.628 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:42:26.629 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:42:26.629 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:42:26.629 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:42:26.676 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 25 2024-04-06T09:42:26.676 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 28 2024-04-06T09:42:26.676 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:42:26.676 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=113, tag_tid=18, entry_tid=1], [object_number=112, tag_tid=18, entry_tid=0], [object_number=114, tag_tid=17, entry_tid=20202], [object_number=115, tag_tid=17, entry_tid=20199]]], state=connected] 2024-04-06T09:42:26.676 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=100, tag_tid=17, entry_tid=8456], [object_number=103, tag_tid=17, entry_tid=8455], [object_number=102, tag_tid=17, entry_tid=8454], [object_number=101, tag_tid=17, entry_tid=8453]]], state=connected] 2024-04-06T09:42:26.680 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:42:26.681 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=100, tag_tid=17, entry_tid=8456' 2024-04-06T09:42:26.681 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=113, tag_tid=18, entry_tid=1' -a 'object_number=113, tag_tid=18, entry_tid=1' = 'object_number=100, tag_tid=17, entry_tid=8456' 2024-04-06T09:42:26.682 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=100, tag_tid=17, entry_tid=8456' '!=' 'object_number=102, tag_tid=17, entry_tid=8370' 2024-04-06T09:42:26.682 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:42:26.682 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=100, tag_tid=17, entry_tid=8456' = 'object_number=102, tag_tid=17, entry_tid=8370' 2024-04-06T09:42:26.682 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=100, tag_tid=17, entry_tid=8456' 2024-04-06T09:42:26.682 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=113, tag_tid=18, entry_tid=1' 2024-04-06T09:42:26.682 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:42:26.683 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:42:26.683 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=18 2024-04-06T09:42:26.684 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=100, tag_tid=17, entry_tid=8456' 2024-04-06T09:42:26.684 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:42:26.684 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:42:26.685 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=17 2024-04-06T09:42:26.685 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=113, tag_tid=18, entry_tid=1' 2024-04-06T09:42:26.685 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:42:26.686 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:42:26.686 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:42:26.687 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=100, tag_tid=17, entry_tid=8456' 2024-04-06T09:42:26.687 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:42:26.687 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:42:26.688 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=8456 2024-04-06T09:42:26.688 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 18 = 17 -a 1 -le 8456 2024-04-06T09:42:26.689 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:42:26.689 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:42:26.689 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:42:26.891 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:42:26.891 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:42:26.891 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:42:26.891 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:42:26.891 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:42:26.891 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:42:26.891 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:42:26.891 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:42:26.892 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:42:26.892 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:42:26.892 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:42:26.892 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:42:26.892 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:42:26.892 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:42:26.892 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:42:26.892 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:42:26.892 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:42:26.895 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:42:26.895 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:42:26.895 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:42:26.895 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:42:26.895 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:42:26.895 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:42:26.895 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:42:26.896 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:42:26.896 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:42:26.896 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:42:26.896 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:42:26.896 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:42:26.896 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:42:26.896 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:42:26.896 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:42:26.896 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:42:26.896 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:42:26.898 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:42:26.899 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:42:26.899 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:42:26.899 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:42:26.899 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:42:26.899 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:42:26.900 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:42:26.900 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:42:26.900 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:42:26.900 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:42:26.900 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:42:26.900 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:42:26.901 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:42:26.901 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:42:26.901 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:42:26.921 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:42:26.921 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:42:26.921 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:42:26.921 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:42:27.048 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:42:27.049 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:42:27.050 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:42:27.050 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:42:27.050 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:42:27.050 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:42:27.050 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:42:27.050 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:42:27.050 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:42:27.050 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:42:27.051 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:42:27.051 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:42:27.051 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:42:27.052 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:42:27.052 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:42:27.052 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:42:27.101 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 25 2024-04-06T09:42:27.101 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 28 2024-04-06T09:42:27.101 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:42:27.102 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=113, tag_tid=18, entry_tid=1], [object_number=112, tag_tid=18, entry_tid=0], [object_number=114, tag_tid=17, entry_tid=20202], [object_number=115, tag_tid=17, entry_tid=20199]]], state=connected] 2024-04-06T09:42:27.102 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=103, tag_tid=17, entry_tid=8543], [object_number=102, tag_tid=17, entry_tid=8542], [object_number=101, tag_tid=17, entry_tid=8541], [object_number=100, tag_tid=17, entry_tid=8540]]], state=connected] 2024-04-06T09:42:27.105 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:42:27.106 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=113, tag_tid=18, entry_tid=1' 2024-04-06T09:42:27.107 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:42:27.107 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:42:27.108 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:42:27.108 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:42:27.108 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:42:27.108 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:42:27.108 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:42:27.108 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:42:27.108 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:42:27.108 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:42:27.108 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:42:27.108 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:42:27.108 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:42:27.109 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:42:27.109 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:42:27.156 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 25 2024-04-06T09:42:27.156 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 28 2024-04-06T09:42:27.156 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:42:27.156 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=113, tag_tid=18, entry_tid=1], [object_number=112, tag_tid=18, entry_tid=0], [object_number=114, tag_tid=17, entry_tid=20202], [object_number=115, tag_tid=17, entry_tid=20199]]], state=connected] 2024-04-06T09:42:27.156 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=103, tag_tid=17, entry_tid=8543], [object_number=102, tag_tid=17, entry_tid=8542], [object_number=101, tag_tid=17, entry_tid=8541], [object_number=100, tag_tid=17, entry_tid=8540]]], state=connected] 2024-04-06T09:42:27.159 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:42:27.161 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=103, tag_tid=17, entry_tid=8543' 2024-04-06T09:42:27.161 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=113, tag_tid=18, entry_tid=1' -a 'object_number=113, tag_tid=18, entry_tid=1' = 'object_number=103, tag_tid=17, entry_tid=8543' 2024-04-06T09:42:27.161 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=103, tag_tid=17, entry_tid=8543' '!=' 'object_number=100, tag_tid=17, entry_tid=8456' 2024-04-06T09:42:27.161 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:42:27.161 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=103, tag_tid=17, entry_tid=8543' = 'object_number=100, tag_tid=17, entry_tid=8456' 2024-04-06T09:42:27.161 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=103, tag_tid=17, entry_tid=8543' 2024-04-06T09:42:27.162 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=113, tag_tid=18, entry_tid=1' 2024-04-06T09:42:27.162 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:42:27.162 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:42:27.163 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=18 2024-04-06T09:42:27.164 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=103, tag_tid=17, entry_tid=8543' 2024-04-06T09:42:27.164 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:42:27.164 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:42:27.164 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=17 2024-04-06T09:42:27.165 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=113, tag_tid=18, entry_tid=1' 2024-04-06T09:42:27.165 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:42:27.165 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:42:27.166 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:42:27.167 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=103, tag_tid=17, entry_tid=8543' 2024-04-06T09:42:27.167 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:42:27.167 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:42:27.168 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=8543 2024-04-06T09:42:27.168 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 18 = 17 -a 1 -le 8543 2024-04-06T09:42:27.168 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:42:27.168 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:42:27.168 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:42:27.370 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:42:27.370 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:42:27.370 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:42:27.370 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:42:27.370 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:42:27.370 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:42:27.371 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:42:27.371 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:42:27.371 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:42:27.371 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:42:27.371 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:42:27.371 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:42:27.371 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:42:27.371 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:42:27.371 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:42:27.372 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:42:27.372 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:42:27.375 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:42:27.375 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:42:27.375 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:42:27.375 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:42:27.375 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:42:27.375 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:42:27.375 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:42:27.375 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:42:27.375 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:42:27.375 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:42:27.375 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:42:27.375 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:42:27.375 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:42:27.376 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:42:27.376 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:42:27.376 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:42:27.376 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:42:27.376 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:42:27.377 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:42:27.377 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:42:27.377 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:42:27.377 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:42:27.377 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:42:27.377 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:42:27.377 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:42:27.377 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:42:27.377 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:42:27.377 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:42:27.377 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:42:27.377 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:42:27.377 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:42:27.377 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:42:27.391 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.0 with skip (0.269606178270018) > weight (0.25) 2024-04-06T09:42:27.391 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.1 with skip (0.7563335217644218) > weight (0.25) 2024-04-06T09:42:27.391 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.2 with skip (0.8426239887712423) > weight (0.25) 2024-04-06T09:42:27.391 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.3 with skip (0.8953291985879269) > weight (0.25) 2024-04-06T09:42:27.391 INFO:tasks.rbd_mirror_thrash:waiting for 96 secs before thrashing 2024-04-06T09:42:27.404 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:42:27.405 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:42:27.405 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:42:27.405 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:42:27.549 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:42:27.550 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:42:27.550 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:42:27.550 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:42:27.550 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:42:27.551 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:42:27.551 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:42:27.551 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:42:27.551 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:42:27.551 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:42:27.551 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:42:27.551 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:42:27.551 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:42:27.553 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:42:27.553 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:42:27.553 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:42:27.606 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 25 2024-04-06T09:42:27.606 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 28 2024-04-06T09:42:27.606 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:42:27.606 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=113, tag_tid=18, entry_tid=1], [object_number=112, tag_tid=18, entry_tid=0], [object_number=114, tag_tid=17, entry_tid=20202], [object_number=115, tag_tid=17, entry_tid=20199]]], state=connected] 2024-04-06T09:42:27.606 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=101, tag_tid=17, entry_tid=8633], [object_number=100, tag_tid=17, entry_tid=8632], [object_number=103, tag_tid=17, entry_tid=8631], [object_number=102, tag_tid=17, entry_tid=8630]]], state=connected] 2024-04-06T09:42:27.609 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:42:27.611 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=113, tag_tid=18, entry_tid=1' 2024-04-06T09:42:27.612 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:42:27.612 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:42:27.612 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:42:27.612 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:42:27.612 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:42:27.612 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:42:27.612 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:42:27.612 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:42:27.613 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:42:27.613 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:42:27.613 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:42:27.613 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:42:27.613 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:42:27.613 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:42:27.614 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:42:27.659 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 25 2024-04-06T09:42:27.659 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 28 2024-04-06T09:42:27.660 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:42:27.660 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=113, tag_tid=18, entry_tid=1], [object_number=112, tag_tid=18, entry_tid=0], [object_number=114, tag_tid=17, entry_tid=20202], [object_number=115, tag_tid=17, entry_tid=20199]]], state=connected] 2024-04-06T09:42:27.660 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=101, tag_tid=17, entry_tid=8633], [object_number=100, tag_tid=17, entry_tid=8632], [object_number=103, tag_tid=17, entry_tid=8631], [object_number=102, tag_tid=17, entry_tid=8630]]], state=connected] 2024-04-06T09:42:27.663 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:42:27.665 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=101, tag_tid=17, entry_tid=8633' 2024-04-06T09:42:27.665 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=113, tag_tid=18, entry_tid=1' -a 'object_number=113, tag_tid=18, entry_tid=1' = 'object_number=101, tag_tid=17, entry_tid=8633' 2024-04-06T09:42:27.665 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=101, tag_tid=17, entry_tid=8633' '!=' 'object_number=103, tag_tid=17, entry_tid=8543' 2024-04-06T09:42:27.665 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:42:27.665 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=101, tag_tid=17, entry_tid=8633' = 'object_number=103, tag_tid=17, entry_tid=8543' 2024-04-06T09:42:27.665 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=101, tag_tid=17, entry_tid=8633' 2024-04-06T09:42:27.665 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=113, tag_tid=18, entry_tid=1' 2024-04-06T09:42:27.665 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:42:27.666 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:42:27.666 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=18 2024-04-06T09:42:27.667 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=101, tag_tid=17, entry_tid=8633' 2024-04-06T09:42:27.667 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:42:27.667 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:42:27.668 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=17 2024-04-06T09:42:27.669 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=113, tag_tid=18, entry_tid=1' 2024-04-06T09:42:27.669 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:42:27.669 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:42:27.669 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:42:27.670 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=101, tag_tid=17, entry_tid=8633' 2024-04-06T09:42:27.670 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:42:27.670 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:42:27.671 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=8633 2024-04-06T09:42:27.671 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 18 = 17 -a 1 -le 8633 2024-04-06T09:42:27.672 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:42:27.672 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:42:27.672 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:42:27.873 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:42:27.874 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:42:27.874 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:42:27.874 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:42:27.874 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:42:27.874 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:42:27.874 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:42:27.874 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:42:27.874 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:42:27.874 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:42:27.874 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:42:27.874 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:42:27.875 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:42:27.875 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:42:27.875 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:42:27.875 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:42:27.875 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:42:27.878 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:42:27.878 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:42:27.878 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:42:27.878 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:42:27.879 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:42:27.879 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:42:27.879 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:42:27.879 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:42:27.879 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:42:27.879 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:42:27.879 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:42:27.879 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:42:27.879 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:42:27.879 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:42:27.879 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:42:27.879 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:42:27.879 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:42:27.879 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:42:27.880 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:42:27.880 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:42:27.880 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:42:27.880 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:42:27.880 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:42:27.880 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:42:27.880 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:42:27.880 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:42:27.880 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:42:27.880 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:42:27.880 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:42:27.880 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:42:27.880 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:42:27.881 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:42:27.907 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:42:27.907 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:42:27.907 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:42:27.907 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:42:28.039 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:42:28.040 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:42:28.040 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:42:28.040 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:42:28.040 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:42:28.040 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:42:28.040 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:42:28.040 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:42:28.040 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:42:28.040 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:42:28.041 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:42:28.041 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:42:28.041 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:42:28.043 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:42:28.044 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:42:28.044 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:42:28.096 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 25 2024-04-06T09:42:28.096 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 28 2024-04-06T09:42:28.096 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:42:28.097 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=113, tag_tid=18, entry_tid=1], [object_number=112, tag_tid=18, entry_tid=0], [object_number=114, tag_tid=17, entry_tid=20202], [object_number=115, tag_tid=17, entry_tid=20199]]], state=connected] 2024-04-06T09:42:28.097 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=101, tag_tid=17, entry_tid=8733], [object_number=100, tag_tid=17, entry_tid=8732], [object_number=103, tag_tid=17, entry_tid=8731], [object_number=102, tag_tid=17, entry_tid=8730]]], state=connected] 2024-04-06T09:42:28.100 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:42:28.101 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=113, tag_tid=18, entry_tid=1' 2024-04-06T09:42:28.102 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:42:28.103 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:42:28.103 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:42:28.103 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:42:28.103 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:42:28.103 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:42:28.103 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:42:28.103 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:42:28.103 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:42:28.103 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:42:28.103 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:42:28.103 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:42:28.104 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:42:28.104 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:42:28.104 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:42:28.151 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 25 2024-04-06T09:42:28.151 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 28 2024-04-06T09:42:28.151 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:42:28.151 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=113, tag_tid=18, entry_tid=1], [object_number=112, tag_tid=18, entry_tid=0], [object_number=114, tag_tid=17, entry_tid=20202], [object_number=115, tag_tid=17, entry_tid=20199]]], state=connected] 2024-04-06T09:42:28.151 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=101, tag_tid=17, entry_tid=8733], [object_number=100, tag_tid=17, entry_tid=8732], [object_number=103, tag_tid=17, entry_tid=8731], [object_number=102, tag_tid=17, entry_tid=8730]]], state=connected] 2024-04-06T09:42:28.154 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:42:28.156 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=101, tag_tid=17, entry_tid=8733' 2024-04-06T09:42:28.156 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=113, tag_tid=18, entry_tid=1' -a 'object_number=113, tag_tid=18, entry_tid=1' = 'object_number=101, tag_tid=17, entry_tid=8733' 2024-04-06T09:42:28.156 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=101, tag_tid=17, entry_tid=8733' '!=' 'object_number=101, tag_tid=17, entry_tid=8633' 2024-04-06T09:42:28.156 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:42:28.156 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=101, tag_tid=17, entry_tid=8733' = 'object_number=101, tag_tid=17, entry_tid=8633' 2024-04-06T09:42:28.156 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=101, tag_tid=17, entry_tid=8733' 2024-04-06T09:42:28.156 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=113, tag_tid=18, entry_tid=1' 2024-04-06T09:42:28.156 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:42:28.156 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:42:28.157 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=18 2024-04-06T09:42:28.158 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=101, tag_tid=17, entry_tid=8733' 2024-04-06T09:42:28.158 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:42:28.158 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:42:28.159 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=17 2024-04-06T09:42:28.159 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=113, tag_tid=18, entry_tid=1' 2024-04-06T09:42:28.159 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:42:28.159 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:42:28.160 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:42:28.161 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=101, tag_tid=17, entry_tid=8733' 2024-04-06T09:42:28.161 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:42:28.161 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:42:28.162 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=8733 2024-04-06T09:42:28.163 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 18 = 17 -a 1 -le 8733 2024-04-06T09:42:28.163 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:42:28.163 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:42:28.163 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:42:28.364 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:42:28.364 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:42:28.365 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:42:28.365 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:42:28.365 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:42:28.365 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:42:28.365 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:42:28.365 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:42:28.365 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:42:28.365 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:42:28.365 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:42:28.365 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:42:28.365 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:42:28.365 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:42:28.365 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:42:28.366 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:42:28.366 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:42:28.369 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:42:28.369 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:42:28.369 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:42:28.369 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:42:28.369 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:42:28.369 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:42:28.369 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:42:28.370 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:42:28.370 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:42:28.370 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:42:28.370 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:42:28.370 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:42:28.370 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:42:28.370 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:42:28.370 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:42:28.370 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:42:28.370 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:42:28.370 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:42:28.370 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:42:28.370 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:42:28.370 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:42:28.371 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:42:28.371 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:42:28.371 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:42:28.371 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:42:28.371 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:42:28.371 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:42:28.371 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:42:28.371 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:42:28.371 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:42:28.371 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:42:28.371 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:42:28.394 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:42:28.394 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:42:28.395 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:42:28.395 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:42:28.530 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:42:28.531 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:42:28.531 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:42:28.531 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:42:28.531 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:42:28.531 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:42:28.531 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:42:28.531 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:42:28.531 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:42:28.531 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:42:28.532 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:42:28.532 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:42:28.532 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:42:28.533 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:42:28.534 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:42:28.534 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:42:28.587 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 25 2024-04-06T09:42:28.587 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 28 2024-04-06T09:42:28.587 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:42:28.587 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=113, tag_tid=18, entry_tid=1], [object_number=112, tag_tid=18, entry_tid=0], [object_number=114, tag_tid=17, entry_tid=20202], [object_number=115, tag_tid=17, entry_tid=20199]]], state=connected] 2024-04-06T09:42:28.587 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=103, tag_tid=17, entry_tid=8827], [object_number=102, tag_tid=17, entry_tid=8826], [object_number=101, tag_tid=17, entry_tid=8825], [object_number=100, tag_tid=17, entry_tid=8824]]], state=connected] 2024-04-06T09:42:28.591 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:42:28.592 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=113, tag_tid=18, entry_tid=1' 2024-04-06T09:42:28.593 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:42:28.593 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:42:28.593 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:42:28.593 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:42:28.593 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:42:28.594 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:42:28.594 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:42:28.594 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:42:28.594 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:42:28.594 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:42:28.594 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:42:28.594 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:42:28.595 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:42:28.595 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:42:28.595 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:42:28.642 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 25 2024-04-06T09:42:28.642 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 28 2024-04-06T09:42:28.642 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:42:28.642 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=113, tag_tid=18, entry_tid=1], [object_number=112, tag_tid=18, entry_tid=0], [object_number=114, tag_tid=17, entry_tid=20202], [object_number=115, tag_tid=17, entry_tid=20199]]], state=connected] 2024-04-06T09:42:28.642 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=103, tag_tid=17, entry_tid=8827], [object_number=102, tag_tid=17, entry_tid=8826], [object_number=101, tag_tid=17, entry_tid=8825], [object_number=100, tag_tid=17, entry_tid=8824]]], state=connected] 2024-04-06T09:42:28.646 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:42:28.647 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=103, tag_tid=17, entry_tid=8827' 2024-04-06T09:42:28.647 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=113, tag_tid=18, entry_tid=1' -a 'object_number=113, tag_tid=18, entry_tid=1' = 'object_number=103, tag_tid=17, entry_tid=8827' 2024-04-06T09:42:28.647 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=103, tag_tid=17, entry_tid=8827' '!=' 'object_number=101, tag_tid=17, entry_tid=8733' 2024-04-06T09:42:28.648 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:42:28.648 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=103, tag_tid=17, entry_tid=8827' = 'object_number=101, tag_tid=17, entry_tid=8733' 2024-04-06T09:42:28.648 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=103, tag_tid=17, entry_tid=8827' 2024-04-06T09:42:28.648 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=113, tag_tid=18, entry_tid=1' 2024-04-06T09:42:28.648 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:42:28.648 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:42:28.648 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=18 2024-04-06T09:42:28.649 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=103, tag_tid=17, entry_tid=8827' 2024-04-06T09:42:28.649 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:42:28.649 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:42:28.650 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=17 2024-04-06T09:42:28.651 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=113, tag_tid=18, entry_tid=1' 2024-04-06T09:42:28.651 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:42:28.651 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:42:28.652 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:42:28.653 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=103, tag_tid=17, entry_tid=8827' 2024-04-06T09:42:28.653 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:42:28.653 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:42:28.654 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=8827 2024-04-06T09:42:28.654 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 18 = 17 -a 1 -le 8827 2024-04-06T09:42:28.654 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:42:28.654 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:42:28.654 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:42:28.856 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:42:28.856 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:42:28.856 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:42:28.856 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:42:28.856 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:42:28.856 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:42:28.856 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:42:28.856 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:42:28.856 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:42:28.857 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:42:28.857 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:42:28.857 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:42:28.857 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:42:28.857 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:42:28.857 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:42:28.857 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:42:28.857 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:42:28.859 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:42:28.859 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:42:28.859 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:42:28.859 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:42:28.859 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:42:28.859 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:42:28.860 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:42:28.860 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:42:28.860 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:42:28.860 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:42:28.860 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:42:28.860 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:42:28.860 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:42:28.860 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:42:28.860 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:42:28.860 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:42:28.861 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:42:28.861 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:42:28.861 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:42:28.861 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:42:28.861 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:42:28.861 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:42:28.861 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:42:28.861 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:42:28.862 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:42:28.862 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:42:28.862 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:42:28.862 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:42:28.862 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:42:28.862 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:42:28.862 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:42:28.862 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:42:28.886 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:42:28.887 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:42:28.887 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:42:28.887 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:42:29.017 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:42:29.018 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:42:29.018 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:42:29.019 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:42:29.019 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:42:29.019 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:42:29.019 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:42:29.019 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:42:29.019 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:42:29.019 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:42:29.019 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:42:29.019 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:42:29.020 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:42:29.021 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:42:29.022 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:42:29.022 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:42:29.068 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 25 2024-04-06T09:42:29.068 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 28 2024-04-06T09:42:29.068 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:42:29.068 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=113, tag_tid=18, entry_tid=1], [object_number=112, tag_tid=18, entry_tid=0], [object_number=114, tag_tid=17, entry_tid=20202], [object_number=115, tag_tid=17, entry_tid=20199]]], state=connected] 2024-04-06T09:42:29.068 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=102, tag_tid=17, entry_tid=8926], [object_number=101, tag_tid=17, entry_tid=8925], [object_number=100, tag_tid=17, entry_tid=8924], [object_number=103, tag_tid=17, entry_tid=8923]]], state=connected] 2024-04-06T09:42:29.071 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:42:29.073 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=113, tag_tid=18, entry_tid=1' 2024-04-06T09:42:29.074 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:42:29.074 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:42:29.074 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:42:29.074 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:42:29.074 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:42:29.074 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:42:29.074 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:42:29.075 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:42:29.075 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:42:29.075 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:42:29.075 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:42:29.075 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:42:29.076 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:42:29.076 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:42:29.076 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:42:29.121 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 25 2024-04-06T09:42:29.121 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 28 2024-04-06T09:42:29.122 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:42:29.122 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=113, tag_tid=18, entry_tid=1], [object_number=112, tag_tid=18, entry_tid=0], [object_number=114, tag_tid=17, entry_tid=20202], [object_number=115, tag_tid=17, entry_tid=20199]]], state=connected] 2024-04-06T09:42:29.122 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=102, tag_tid=17, entry_tid=8926], [object_number=101, tag_tid=17, entry_tid=8925], [object_number=100, tag_tid=17, entry_tid=8924], [object_number=103, tag_tid=17, entry_tid=8923]]], state=connected] 2024-04-06T09:42:29.125 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:42:29.127 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=102, tag_tid=17, entry_tid=8926' 2024-04-06T09:42:29.127 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=113, tag_tid=18, entry_tid=1' -a 'object_number=113, tag_tid=18, entry_tid=1' = 'object_number=102, tag_tid=17, entry_tid=8926' 2024-04-06T09:42:29.127 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=102, tag_tid=17, entry_tid=8926' '!=' 'object_number=103, tag_tid=17, entry_tid=8827' 2024-04-06T09:42:29.127 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:42:29.127 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=102, tag_tid=17, entry_tid=8926' = 'object_number=103, tag_tid=17, entry_tid=8827' 2024-04-06T09:42:29.127 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=102, tag_tid=17, entry_tid=8926' 2024-04-06T09:42:29.128 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=113, tag_tid=18, entry_tid=1' 2024-04-06T09:42:29.128 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:42:29.128 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:42:29.128 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=18 2024-04-06T09:42:29.129 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=102, tag_tid=17, entry_tid=8926' 2024-04-06T09:42:29.129 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:42:29.129 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:42:29.130 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=17 2024-04-06T09:42:29.131 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=113, tag_tid=18, entry_tid=1' 2024-04-06T09:42:29.131 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:42:29.131 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:42:29.132 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:42:29.132 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=102, tag_tid=17, entry_tid=8926' 2024-04-06T09:42:29.133 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:42:29.133 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:42:29.134 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=8926 2024-04-06T09:42:29.134 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 18 = 17 -a 1 -le 8926 2024-04-06T09:42:29.134 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:42:29.134 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:42:29.134 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:42:29.335 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:42:29.335 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:42:29.336 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:42:29.336 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:42:29.336 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:42:29.336 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:42:29.336 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:42:29.336 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:42:29.336 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:42:29.336 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:42:29.336 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:42:29.336 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:42:29.336 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:42:29.336 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:42:29.336 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:42:29.337 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:42:29.337 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:42:29.339 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:42:29.339 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:42:29.339 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:42:29.339 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:42:29.339 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:42:29.339 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:42:29.339 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:42:29.339 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:42:29.340 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:42:29.340 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:42:29.340 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:42:29.340 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:42:29.340 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:42:29.340 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:42:29.340 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:42:29.340 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:42:29.341 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:42:29.341 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:42:29.341 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:42:29.341 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:42:29.341 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:42:29.342 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:42:29.342 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:42:29.342 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:42:29.342 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:42:29.342 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:42:29.342 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:42:29.342 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:42:29.342 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:42:29.342 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:42:29.342 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:42:29.342 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:42:29.367 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:42:29.367 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:42:29.367 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:42:29.367 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:42:29.496 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:42:29.497 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:42:29.498 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:42:29.498 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:42:29.498 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:42:29.498 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:42:29.498 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:42:29.498 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:42:29.498 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:42:29.498 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:42:29.498 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:42:29.499 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:42:29.499 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:42:29.501 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:42:29.501 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:42:29.501 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:42:29.553 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 25 2024-04-06T09:42:29.554 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 28 2024-04-06T09:42:29.554 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:42:29.554 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=113, tag_tid=18, entry_tid=1], [object_number=112, tag_tid=18, entry_tid=0], [object_number=114, tag_tid=17, entry_tid=20202], [object_number=115, tag_tid=17, entry_tid=20199]]], state=connected] 2024-04-06T09:42:29.554 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=100, tag_tid=17, entry_tid=9024], [object_number=103, tag_tid=17, entry_tid=9023], [object_number=102, tag_tid=17, entry_tid=9022], [object_number=101, tag_tid=17, entry_tid=9021]]], state=connected] 2024-04-06T09:42:29.557 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:42:29.558 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=113, tag_tid=18, entry_tid=1' 2024-04-06T09:42:29.559 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:42:29.560 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:42:29.560 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:42:29.560 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:42:29.560 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:42:29.560 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:42:29.560 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:42:29.560 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:42:29.560 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:42:29.560 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:42:29.560 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:42:29.560 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:42:29.561 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:42:29.561 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:42:29.561 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:42:29.608 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 25 2024-04-06T09:42:29.608 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 28 2024-04-06T09:42:29.608 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:42:29.608 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=113, tag_tid=18, entry_tid=1], [object_number=112, tag_tid=18, entry_tid=0], [object_number=114, tag_tid=17, entry_tid=20202], [object_number=115, tag_tid=17, entry_tid=20199]]], state=connected] 2024-04-06T09:42:29.608 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=100, tag_tid=17, entry_tid=9024], [object_number=103, tag_tid=17, entry_tid=9023], [object_number=102, tag_tid=17, entry_tid=9022], [object_number=101, tag_tid=17, entry_tid=9021]]], state=connected] 2024-04-06T09:42:29.612 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:42:29.613 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=100, tag_tid=17, entry_tid=9024' 2024-04-06T09:42:29.613 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=113, tag_tid=18, entry_tid=1' -a 'object_number=113, tag_tid=18, entry_tid=1' = 'object_number=100, tag_tid=17, entry_tid=9024' 2024-04-06T09:42:29.614 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=100, tag_tid=17, entry_tid=9024' '!=' 'object_number=102, tag_tid=17, entry_tid=8926' 2024-04-06T09:42:29.614 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:42:29.614 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=100, tag_tid=17, entry_tid=9024' = 'object_number=102, tag_tid=17, entry_tid=8926' 2024-04-06T09:42:29.614 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=100, tag_tid=17, entry_tid=9024' 2024-04-06T09:42:29.614 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=113, tag_tid=18, entry_tid=1' 2024-04-06T09:42:29.614 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:42:29.615 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:42:29.615 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=18 2024-04-06T09:42:29.616 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=100, tag_tid=17, entry_tid=9024' 2024-04-06T09:42:29.616 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:42:29.616 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:42:29.617 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=17 2024-04-06T09:42:29.617 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=113, tag_tid=18, entry_tid=1' 2024-04-06T09:42:29.617 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:42:29.617 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:42:29.618 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:42:29.619 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=100, tag_tid=17, entry_tid=9024' 2024-04-06T09:42:29.619 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:42:29.619 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:42:29.620 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=9024 2024-04-06T09:42:29.621 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 18 = 17 -a 1 -le 9024 2024-04-06T09:42:29.621 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:42:29.621 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:42:29.621 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:42:29.822 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:42:29.822 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:42:29.822 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:42:29.822 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:42:29.822 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:42:29.823 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:42:29.823 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:42:29.823 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:42:29.823 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:42:29.823 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:42:29.823 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:42:29.823 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:42:29.823 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:42:29.823 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:42:29.823 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:42:29.823 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:42:29.824 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:42:29.826 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:42:29.826 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:42:29.826 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:42:29.826 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:42:29.826 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:42:29.826 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:42:29.826 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:42:29.827 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:42:29.827 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:42:29.827 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:42:29.827 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:42:29.827 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:42:29.827 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:42:29.827 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:42:29.827 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:42:29.827 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:42:29.828 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:42:29.828 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:42:29.829 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:42:29.829 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:42:29.829 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:42:29.829 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:42:29.829 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:42:29.829 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:42:29.829 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:42:29.829 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:42:29.829 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:42:29.829 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:42:29.829 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:42:29.829 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:42:29.829 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:42:29.829 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:42:29.853 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:42:29.853 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:42:29.853 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:42:29.854 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:42:29.992 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:42:29.994 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:42:29.994 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:42:29.994 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:42:29.994 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:42:29.994 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:42:29.994 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:42:29.994 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:42:29.994 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:42:29.994 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:42:29.995 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:42:29.995 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:42:29.995 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:42:29.997 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:42:29.997 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:42:29.997 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:42:30.049 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 25 2024-04-06T09:42:30.049 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 28 2024-04-06T09:42:30.049 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:42:30.049 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=113, tag_tid=18, entry_tid=1], [object_number=112, tag_tid=18, entry_tid=0], [object_number=114, tag_tid=17, entry_tid=20202], [object_number=115, tag_tid=17, entry_tid=20199]]], state=connected] 2024-04-06T09:42:30.049 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=100, tag_tid=17, entry_tid=9120], [object_number=103, tag_tid=17, entry_tid=9119], [object_number=102, tag_tid=17, entry_tid=9118], [object_number=101, tag_tid=17, entry_tid=9117]]], state=connected] 2024-04-06T09:42:30.052 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:42:30.054 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=113, tag_tid=18, entry_tid=1' 2024-04-06T09:42:30.055 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:42:30.055 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:42:30.055 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:42:30.055 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:42:30.055 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:42:30.055 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:42:30.055 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:42:30.056 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:42:30.056 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:42:30.056 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:42:30.056 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:42:30.056 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:42:30.056 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:42:30.057 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:42:30.057 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:42:30.104 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 25 2024-04-06T09:42:30.104 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 28 2024-04-06T09:42:30.104 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:42:30.104 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=113, tag_tid=18, entry_tid=1], [object_number=112, tag_tid=18, entry_tid=0], [object_number=114, tag_tid=17, entry_tid=20202], [object_number=115, tag_tid=17, entry_tid=20199]]], state=connected] 2024-04-06T09:42:30.104 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=100, tag_tid=17, entry_tid=9120], [object_number=103, tag_tid=17, entry_tid=9119], [object_number=102, tag_tid=17, entry_tid=9118], [object_number=101, tag_tid=17, entry_tid=9117]]], state=connected] 2024-04-06T09:42:30.108 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:42:30.110 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=100, tag_tid=17, entry_tid=9120' 2024-04-06T09:42:30.110 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=113, tag_tid=18, entry_tid=1' -a 'object_number=113, tag_tid=18, entry_tid=1' = 'object_number=100, tag_tid=17, entry_tid=9120' 2024-04-06T09:42:30.110 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=100, tag_tid=17, entry_tid=9120' '!=' 'object_number=100, tag_tid=17, entry_tid=9024' 2024-04-06T09:42:30.110 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:42:30.110 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=100, tag_tid=17, entry_tid=9120' = 'object_number=100, tag_tid=17, entry_tid=9024' 2024-04-06T09:42:30.110 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=100, tag_tid=17, entry_tid=9120' 2024-04-06T09:42:30.110 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=113, tag_tid=18, entry_tid=1' 2024-04-06T09:42:30.110 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:42:30.110 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:42:30.111 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=18 2024-04-06T09:42:30.112 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=100, tag_tid=17, entry_tid=9120' 2024-04-06T09:42:30.112 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:42:30.112 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:42:30.113 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=17 2024-04-06T09:42:30.113 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=113, tag_tid=18, entry_tid=1' 2024-04-06T09:42:30.114 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:42:30.114 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:42:30.114 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:42:30.115 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=100, tag_tid=17, entry_tid=9120' 2024-04-06T09:42:30.115 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:42:30.115 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:42:30.117 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=9120 2024-04-06T09:42:30.117 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 18 = 17 -a 1 -le 9120 2024-04-06T09:42:30.117 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:42:30.117 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:42:30.117 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:42:30.318 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:42:30.318 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:42:30.319 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:42:30.319 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:42:30.319 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:42:30.319 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:42:30.319 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:42:30.319 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:42:30.319 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:42:30.319 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:42:30.319 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:42:30.319 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:42:30.319 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:42:30.319 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:42:30.319 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:42:30.320 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:42:30.321 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:42:30.324 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:42:30.324 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:42:30.324 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:42:30.324 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:42:30.324 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:42:30.324 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:42:30.324 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:42:30.324 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:42:30.324 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:42:30.324 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:42:30.324 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:42:30.324 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:42:30.324 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:42:30.325 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:42:30.325 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:42:30.325 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:42:30.325 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:42:30.325 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:42:30.325 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:42:30.325 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:42:30.325 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:42:30.325 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:42:30.325 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:42:30.325 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:42:30.325 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:42:30.325 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:42:30.325 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:42:30.325 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:42:30.326 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:42:30.326 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:42:30.326 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:42:30.326 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:42:30.352 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:42:30.352 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:42:30.352 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:42:30.352 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:42:30.491 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:42:30.491 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:42:30.491 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:42:30.492 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:42:30.492 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:42:30.492 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:42:30.492 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:42:30.492 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:42:30.492 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:42:30.492 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:42:30.492 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:42:30.492 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:42:30.492 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:42:30.493 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:42:30.493 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:42:30.493 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:42:30.541 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 25 2024-04-06T09:42:30.541 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 28 2024-04-06T09:42:30.541 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:42:30.541 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=113, tag_tid=18, entry_tid=1], [object_number=112, tag_tid=18, entry_tid=0], [object_number=114, tag_tid=17, entry_tid=20202], [object_number=115, tag_tid=17, entry_tid=20199]]], state=connected] 2024-04-06T09:42:30.542 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=103, tag_tid=17, entry_tid=9199], [object_number=102, tag_tid=17, entry_tid=9198], [object_number=101, tag_tid=17, entry_tid=9197], [object_number=100, tag_tid=17, entry_tid=9196]]], state=connected] 2024-04-06T09:42:30.545 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:42:30.546 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=113, tag_tid=18, entry_tid=1' 2024-04-06T09:42:30.547 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:42:30.548 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:42:30.548 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:42:30.548 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:42:30.548 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:42:30.548 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:42:30.548 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:42:30.548 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:42:30.548 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:42:30.548 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:42:30.548 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:42:30.548 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:42:30.549 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:42:30.549 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:42:30.549 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:42:30.595 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 25 2024-04-06T09:42:30.595 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 28 2024-04-06T09:42:30.595 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:42:30.595 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=113, tag_tid=18, entry_tid=1], [object_number=112, tag_tid=18, entry_tid=0], [object_number=114, tag_tid=17, entry_tid=20202], [object_number=115, tag_tid=17, entry_tid=20199]]], state=connected] 2024-04-06T09:42:30.595 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=103, tag_tid=17, entry_tid=9199], [object_number=102, tag_tid=17, entry_tid=9198], [object_number=101, tag_tid=17, entry_tid=9197], [object_number=100, tag_tid=17, entry_tid=9196]]], state=connected] 2024-04-06T09:42:30.598 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:42:30.601 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=103, tag_tid=17, entry_tid=9199' 2024-04-06T09:42:30.601 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=113, tag_tid=18, entry_tid=1' -a 'object_number=113, tag_tid=18, entry_tid=1' = 'object_number=103, tag_tid=17, entry_tid=9199' 2024-04-06T09:42:30.601 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=103, tag_tid=17, entry_tid=9199' '!=' 'object_number=100, tag_tid=17, entry_tid=9120' 2024-04-06T09:42:30.601 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:42:30.601 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=103, tag_tid=17, entry_tid=9199' = 'object_number=100, tag_tid=17, entry_tid=9120' 2024-04-06T09:42:30.601 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=103, tag_tid=17, entry_tid=9199' 2024-04-06T09:42:30.601 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=113, tag_tid=18, entry_tid=1' 2024-04-06T09:42:30.601 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:42:30.601 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:42:30.602 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=18 2024-04-06T09:42:30.602 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=103, tag_tid=17, entry_tid=9199' 2024-04-06T09:42:30.603 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:42:30.603 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:42:30.603 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=17 2024-04-06T09:42:30.604 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=113, tag_tid=18, entry_tid=1' 2024-04-06T09:42:30.604 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:42:30.604 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:42:30.605 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:42:30.606 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=103, tag_tid=17, entry_tid=9199' 2024-04-06T09:42:30.606 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:42:30.606 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:42:30.607 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=9199 2024-04-06T09:42:30.607 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 18 = 17 -a 1 -le 9199 2024-04-06T09:42:30.607 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:42:30.607 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:42:30.607 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:42:30.809 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:42:30.809 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:42:30.809 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:42:30.809 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:42:30.810 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:42:30.810 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:42:30.810 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:42:30.810 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:42:30.810 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:42:30.810 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:42:30.810 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:42:30.810 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:42:30.810 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:42:30.810 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:42:30.810 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:42:30.811 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:42:30.811 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:42:30.813 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:42:30.814 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:42:30.814 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:42:30.814 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:42:30.814 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:42:30.814 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:42:30.814 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:42:30.814 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:42:30.814 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:42:30.814 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:42:30.814 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:42:30.814 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:42:30.814 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:42:30.814 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:42:30.815 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:42:30.815 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:42:30.815 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:42:30.815 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:42:30.816 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:42:30.816 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:42:30.816 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:42:30.816 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:42:30.816 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:42:30.816 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:42:30.816 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:42:30.816 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:42:30.816 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:42:30.816 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:42:30.816 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:42:30.816 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:42:30.816 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:42:30.816 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:42:30.840 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:42:30.840 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:42:30.840 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:42:30.841 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:42:30.977 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:42:30.978 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:42:30.979 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:42:30.979 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:42:30.979 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:42:30.979 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:42:30.979 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:42:30.979 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:42:30.979 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:42:30.979 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:42:30.980 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:42:30.980 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:42:30.980 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:42:30.981 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:42:30.982 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:42:30.982 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:42:31.033 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 25 2024-04-06T09:42:31.033 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 28 2024-04-06T09:42:31.033 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:42:31.033 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=113, tag_tid=18, entry_tid=1], [object_number=112, tag_tid=18, entry_tid=0], [object_number=114, tag_tid=17, entry_tid=20202], [object_number=115, tag_tid=17, entry_tid=20199]]], state=connected] 2024-04-06T09:42:31.034 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=103, tag_tid=17, entry_tid=9283], [object_number=102, tag_tid=17, entry_tid=9282], [object_number=101, tag_tid=17, entry_tid=9281], [object_number=100, tag_tid=17, entry_tid=9280]]], state=connected] 2024-04-06T09:42:31.036 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:42:31.038 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=113, tag_tid=18, entry_tid=1' 2024-04-06T09:42:31.039 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:42:31.039 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:42:31.039 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:42:31.039 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:42:31.039 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:42:31.039 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:42:31.040 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:42:31.040 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:42:31.040 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:42:31.040 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:42:31.041 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:42:31.041 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:42:31.041 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:42:31.041 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:42:31.041 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:42:31.087 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 25 2024-04-06T09:42:31.087 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 28 2024-04-06T09:42:31.088 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:42:31.088 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=113, tag_tid=18, entry_tid=1], [object_number=112, tag_tid=18, entry_tid=0], [object_number=114, tag_tid=17, entry_tid=20202], [object_number=115, tag_tid=17, entry_tid=20199]]], state=connected] 2024-04-06T09:42:31.088 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=103, tag_tid=17, entry_tid=9283], [object_number=102, tag_tid=17, entry_tid=9282], [object_number=101, tag_tid=17, entry_tid=9281], [object_number=100, tag_tid=17, entry_tid=9280]]], state=connected] 2024-04-06T09:42:31.091 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:42:31.092 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=103, tag_tid=17, entry_tid=9283' 2024-04-06T09:42:31.093 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=113, tag_tid=18, entry_tid=1' -a 'object_number=113, tag_tid=18, entry_tid=1' = 'object_number=103, tag_tid=17, entry_tid=9283' 2024-04-06T09:42:31.093 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=103, tag_tid=17, entry_tid=9283' '!=' 'object_number=103, tag_tid=17, entry_tid=9199' 2024-04-06T09:42:31.093 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:42:31.093 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=103, tag_tid=17, entry_tid=9283' = 'object_number=103, tag_tid=17, entry_tid=9199' 2024-04-06T09:42:31.093 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=103, tag_tid=17, entry_tid=9283' 2024-04-06T09:42:31.093 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=113, tag_tid=18, entry_tid=1' 2024-04-06T09:42:31.094 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:42:31.094 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:42:31.094 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=18 2024-04-06T09:42:31.095 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=103, tag_tid=17, entry_tid=9283' 2024-04-06T09:42:31.095 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:42:31.095 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:42:31.096 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=17 2024-04-06T09:42:31.097 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=113, tag_tid=18, entry_tid=1' 2024-04-06T09:42:31.097 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:42:31.097 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:42:31.098 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:42:31.098 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=103, tag_tid=17, entry_tid=9283' 2024-04-06T09:42:31.099 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:42:31.099 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:42:31.102 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=9283 2024-04-06T09:42:31.102 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 18 = 17 -a 1 -le 9283 2024-04-06T09:42:31.102 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:42:31.102 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:42:31.103 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:42:31.302 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:42:31.302 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:42:31.302 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:42:31.302 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:42:31.305 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:42:31.306 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:42:31.306 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:42:31.306 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:42:31.306 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:42:31.306 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:42:31.306 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:42:31.306 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:42:31.306 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:42:31.307 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:42:31.307 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:42:31.309 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:42:31.309 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:42:31.309 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:42:31.309 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:42:31.310 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:42:31.310 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:42:31.310 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:42:31.310 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:42:31.310 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:42:31.310 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:42:31.310 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:42:31.310 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:42:31.310 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:42:31.310 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:42:31.310 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:42:31.310 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:42:31.311 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:42:31.311 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:42:31.311 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:42:31.311 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:42:31.311 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:42:31.311 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:42:31.311 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:42:31.311 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:42:31.311 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:42:31.311 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:42:31.311 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:42:31.311 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:42:31.311 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:42:31.311 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:42:31.312 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:42:31.312 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:42:31.312 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:42:31.312 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:42:31.333 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:42:31.333 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:42:31.333 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:42:31.333 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:42:31.469 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:42:31.470 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:42:31.470 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:42:31.470 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:42:31.471 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:42:31.471 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:42:31.471 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:42:31.471 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:42:31.471 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:42:31.471 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:42:31.472 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:42:31.472 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:42:31.472 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:42:31.473 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:42:31.473 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:42:31.474 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:42:31.522 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 25 2024-04-06T09:42:31.522 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 28 2024-04-06T09:42:31.522 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:42:31.522 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=113, tag_tid=18, entry_tid=1], [object_number=112, tag_tid=18, entry_tid=0], [object_number=114, tag_tid=17, entry_tid=20202], [object_number=115, tag_tid=17, entry_tid=20199]]], state=connected] 2024-04-06T09:42:31.523 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=103, tag_tid=17, entry_tid=9363], [object_number=102, tag_tid=17, entry_tid=9362], [object_number=101, tag_tid=17, entry_tid=9361], [object_number=100, tag_tid=17, entry_tid=9360]]], state=connected] 2024-04-06T09:42:31.526 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:42:31.527 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=113, tag_tid=18, entry_tid=1' 2024-04-06T09:42:31.528 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:42:31.528 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:42:31.528 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:42:31.528 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:42:31.529 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:42:31.529 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:42:31.529 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:42:31.529 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:42:31.529 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:42:31.529 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:42:31.529 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:42:31.529 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:42:31.530 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:42:31.530 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:42:31.530 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:42:31.576 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 25 2024-04-06T09:42:31.576 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 28 2024-04-06T09:42:31.576 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:42:31.576 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=113, tag_tid=18, entry_tid=1], [object_number=112, tag_tid=18, entry_tid=0], [object_number=114, tag_tid=17, entry_tid=20202], [object_number=115, tag_tid=17, entry_tid=20199]]], state=connected] 2024-04-06T09:42:31.576 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=103, tag_tid=17, entry_tid=9363], [object_number=102, tag_tid=17, entry_tid=9362], [object_number=101, tag_tid=17, entry_tid=9361], [object_number=100, tag_tid=17, entry_tid=9360]]], state=connected] 2024-04-06T09:42:31.579 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:42:31.581 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=103, tag_tid=17, entry_tid=9363' 2024-04-06T09:42:31.581 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=113, tag_tid=18, entry_tid=1' -a 'object_number=113, tag_tid=18, entry_tid=1' = 'object_number=103, tag_tid=17, entry_tid=9363' 2024-04-06T09:42:31.581 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=103, tag_tid=17, entry_tid=9363' '!=' 'object_number=103, tag_tid=17, entry_tid=9283' 2024-04-06T09:42:31.582 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:42:31.582 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=103, tag_tid=17, entry_tid=9363' = 'object_number=103, tag_tid=17, entry_tid=9283' 2024-04-06T09:42:31.582 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=103, tag_tid=17, entry_tid=9363' 2024-04-06T09:42:31.582 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=113, tag_tid=18, entry_tid=1' 2024-04-06T09:42:31.582 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:42:31.582 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:42:31.582 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=18 2024-04-06T09:42:31.583 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=103, tag_tid=17, entry_tid=9363' 2024-04-06T09:42:31.583 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:42:31.583 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:42:31.584 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=17 2024-04-06T09:42:31.585 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=113, tag_tid=18, entry_tid=1' 2024-04-06T09:42:31.585 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:42:31.585 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:42:31.586 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:42:31.587 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=103, tag_tid=17, entry_tid=9363' 2024-04-06T09:42:31.587 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:42:31.587 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:42:31.588 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=9363 2024-04-06T09:42:31.588 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 18 = 17 -a 1 -le 9363 2024-04-06T09:42:31.588 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:42:31.588 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:42:31.588 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:42:31.790 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:42:31.790 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:42:31.790 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:42:31.791 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:42:31.791 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:42:31.791 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:42:31.791 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:42:31.791 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:42:31.791 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:42:31.791 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:42:31.791 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:42:31.791 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:42:31.791 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:42:31.791 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:42:31.791 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:42:31.792 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:42:31.792 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:42:31.794 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:42:31.794 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:42:31.794 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:42:31.794 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:42:31.794 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:42:31.794 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:42:31.794 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:42:31.794 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:42:31.794 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:42:31.794 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:42:31.794 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:42:31.795 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:42:31.795 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:42:31.795 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:42:31.795 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:42:31.795 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:42:31.796 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:42:31.796 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:42:31.796 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:42:31.796 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:42:31.796 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:42:31.796 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:42:31.796 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:42:31.796 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:42:31.796 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:42:31.796 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:42:31.796 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:42:31.796 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:42:31.797 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:42:31.797 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:42:31.797 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:42:31.797 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:42:31.820 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:42:31.821 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:42:31.821 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:42:31.821 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:42:31.957 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:42:31.959 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:42:31.959 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:42:31.959 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:42:31.959 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:42:31.959 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:42:31.959 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:42:31.959 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:42:31.959 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:42:31.959 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:42:31.959 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:42:31.960 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:42:31.960 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:42:31.962 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:42:31.962 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:42:31.962 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:42:32.014 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 25 2024-04-06T09:42:32.014 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 28 2024-04-06T09:42:32.014 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:42:32.014 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=113, tag_tid=18, entry_tid=1], [object_number=112, tag_tid=18, entry_tid=0], [object_number=114, tag_tid=17, entry_tid=20202], [object_number=115, tag_tid=17, entry_tid=20199]]], state=connected] 2024-04-06T09:42:32.014 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=102, tag_tid=17, entry_tid=9446], [object_number=101, tag_tid=17, entry_tid=9445], [object_number=100, tag_tid=17, entry_tid=9444], [object_number=103, tag_tid=17, entry_tid=9443]]], state=connected] 2024-04-06T09:42:32.017 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:42:32.019 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=113, tag_tid=18, entry_tid=1' 2024-04-06T09:42:32.020 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:42:32.020 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:42:32.020 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:42:32.020 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:42:32.020 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:42:32.020 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:42:32.020 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:42:32.020 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:42:32.020 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:42:32.021 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:42:32.021 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:42:32.021 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:42:32.021 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:42:32.021 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:42:32.021 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:42:32.067 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 25 2024-04-06T09:42:32.067 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 28 2024-04-06T09:42:32.067 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:42:32.067 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=113, tag_tid=18, entry_tid=1], [object_number=112, tag_tid=18, entry_tid=0], [object_number=114, tag_tid=17, entry_tid=20202], [object_number=115, tag_tid=17, entry_tid=20199]]], state=connected] 2024-04-06T09:42:32.067 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=102, tag_tid=17, entry_tid=9446], [object_number=101, tag_tid=17, entry_tid=9445], [object_number=100, tag_tid=17, entry_tid=9444], [object_number=103, tag_tid=17, entry_tid=9443]]], state=connected] 2024-04-06T09:42:32.071 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:42:32.073 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=102, tag_tid=17, entry_tid=9446' 2024-04-06T09:42:32.073 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=113, tag_tid=18, entry_tid=1' -a 'object_number=113, tag_tid=18, entry_tid=1' = 'object_number=102, tag_tid=17, entry_tid=9446' 2024-04-06T09:42:32.073 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=102, tag_tid=17, entry_tid=9446' '!=' 'object_number=103, tag_tid=17, entry_tid=9363' 2024-04-06T09:42:32.073 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:42:32.073 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=102, tag_tid=17, entry_tid=9446' = 'object_number=103, tag_tid=17, entry_tid=9363' 2024-04-06T09:42:32.073 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=102, tag_tid=17, entry_tid=9446' 2024-04-06T09:42:32.073 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=113, tag_tid=18, entry_tid=1' 2024-04-06T09:42:32.074 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:42:32.074 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:42:32.074 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=18 2024-04-06T09:42:32.075 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=102, tag_tid=17, entry_tid=9446' 2024-04-06T09:42:32.075 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:42:32.075 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:42:32.076 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=17 2024-04-06T09:42:32.077 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=113, tag_tid=18, entry_tid=1' 2024-04-06T09:42:32.077 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:42:32.077 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:42:32.077 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:42:32.078 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=102, tag_tid=17, entry_tid=9446' 2024-04-06T09:42:32.078 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:42:32.078 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:42:32.079 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=9446 2024-04-06T09:42:32.080 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 18 = 17 -a 1 -le 9446 2024-04-06T09:42:32.080 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:42:32.080 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:42:32.080 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:42:32.281 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:42:32.282 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:42:32.282 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:42:32.282 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:42:32.282 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:42:32.282 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:42:32.282 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:42:32.282 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:42:32.282 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:42:32.282 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:42:32.282 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:42:32.282 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:42:32.282 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:42:32.282 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:42:32.282 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:42:32.283 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:42:32.283 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:42:32.287 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:42:32.287 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:42:32.287 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:42:32.287 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:42:32.287 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:42:32.287 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:42:32.287 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:42:32.287 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:42:32.287 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:42:32.287 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:42:32.287 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:42:32.288 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:42:32.288 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:42:32.288 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:42:32.288 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:42:32.288 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:42:32.288 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:42:32.288 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:42:32.288 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:42:32.288 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:42:32.288 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:42:32.288 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:42:32.288 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:42:32.288 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:42:32.288 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:42:32.288 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:42:32.288 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:42:32.289 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:42:32.289 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:42:32.289 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:42:32.289 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:42:32.289 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:42:32.312 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:42:32.312 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:42:32.312 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:42:32.312 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:42:32.448 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:42:32.449 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:42:32.449 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:42:32.449 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:42:32.449 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:42:32.449 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:42:32.449 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:42:32.449 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:42:32.450 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:42:32.450 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:42:32.450 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:42:32.450 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:42:32.450 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:42:32.452 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:42:32.453 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:42:32.453 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:42:32.505 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 25 2024-04-06T09:42:32.505 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 28 2024-04-06T09:42:32.505 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:42:32.505 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=113, tag_tid=18, entry_tid=1], [object_number=112, tag_tid=18, entry_tid=0], [object_number=114, tag_tid=17, entry_tid=20202], [object_number=115, tag_tid=17, entry_tid=20199]]], state=connected] 2024-04-06T09:42:32.505 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=102, tag_tid=17, entry_tid=9518], [object_number=101, tag_tid=17, entry_tid=9517], [object_number=100, tag_tid=17, entry_tid=9516], [object_number=103, tag_tid=17, entry_tid=9515]]], state=connected] 2024-04-06T09:42:32.508 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:42:32.510 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=113, tag_tid=18, entry_tid=1' 2024-04-06T09:42:32.511 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:42:32.511 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:42:32.511 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:42:32.511 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:42:32.511 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:42:32.511 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:42:32.511 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:42:32.512 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:42:32.512 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:42:32.512 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:42:32.512 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:42:32.512 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:42:32.513 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:42:32.513 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:42:32.513 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:42:32.559 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 25 2024-04-06T09:42:32.560 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 28 2024-04-06T09:42:32.560 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:42:32.560 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=113, tag_tid=18, entry_tid=1], [object_number=112, tag_tid=18, entry_tid=0], [object_number=114, tag_tid=17, entry_tid=20202], [object_number=115, tag_tid=17, entry_tid=20199]]], state=connected] 2024-04-06T09:42:32.560 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=102, tag_tid=17, entry_tid=9518], [object_number=101, tag_tid=17, entry_tid=9517], [object_number=100, tag_tid=17, entry_tid=9516], [object_number=103, tag_tid=17, entry_tid=9515]]], state=connected] 2024-04-06T09:42:32.563 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:42:32.565 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=102, tag_tid=17, entry_tid=9518' 2024-04-06T09:42:32.565 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=113, tag_tid=18, entry_tid=1' -a 'object_number=113, tag_tid=18, entry_tid=1' = 'object_number=102, tag_tid=17, entry_tid=9518' 2024-04-06T09:42:32.565 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=102, tag_tid=17, entry_tid=9518' '!=' 'object_number=102, tag_tid=17, entry_tid=9446' 2024-04-06T09:42:32.565 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:42:32.565 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=102, tag_tid=17, entry_tid=9518' = 'object_number=102, tag_tid=17, entry_tid=9446' 2024-04-06T09:42:32.565 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=102, tag_tid=17, entry_tid=9518' 2024-04-06T09:42:32.566 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=113, tag_tid=18, entry_tid=1' 2024-04-06T09:42:32.566 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:42:32.566 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:42:32.566 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=18 2024-04-06T09:42:32.567 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=102, tag_tid=17, entry_tid=9518' 2024-04-06T09:42:32.567 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:42:32.567 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:42:32.568 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=17 2024-04-06T09:42:32.568 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=113, tag_tid=18, entry_tid=1' 2024-04-06T09:42:32.568 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:42:32.569 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:42:32.569 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:42:32.570 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=102, tag_tid=17, entry_tid=9518' 2024-04-06T09:42:32.570 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:42:32.570 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:42:32.571 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=9518 2024-04-06T09:42:32.571 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 18 = 17 -a 1 -le 9518 2024-04-06T09:42:32.571 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:42:32.572 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:42:32.572 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:42:32.773 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:42:32.773 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:42:32.773 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:42:32.774 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:42:32.774 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:42:32.774 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:42:32.774 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:42:32.774 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:42:32.774 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:42:32.774 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:42:32.774 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:42:32.774 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:42:32.774 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:42:32.774 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:42:32.774 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:42:32.775 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:42:32.775 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:42:32.778 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:42:32.778 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:42:32.778 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:42:32.778 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:42:32.778 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:42:32.778 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:42:32.778 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:42:32.778 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:42:32.778 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:42:32.778 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:42:32.779 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:42:32.779 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:42:32.779 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:42:32.779 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:42:32.779 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:42:32.779 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:42:32.779 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:42:32.779 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:42:32.779 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:42:32.779 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:42:32.779 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:42:32.779 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:42:32.779 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:42:32.779 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:42:32.780 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:42:32.780 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:42:32.780 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:42:32.780 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:42:32.780 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:42:32.780 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:42:32.780 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:42:32.781 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:42:32.806 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:42:32.806 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:42:32.806 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:42:32.806 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:42:32.936 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:42:32.937 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:42:32.937 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:42:32.937 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:42:32.937 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:42:32.938 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:42:32.938 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:42:32.938 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:42:32.938 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:42:32.938 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:42:32.938 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:42:32.938 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:42:32.939 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:42:32.939 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:42:32.940 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:42:32.940 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:42:32.992 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 25 2024-04-06T09:42:32.992 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 28 2024-04-06T09:42:32.992 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:42:32.993 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=113, tag_tid=18, entry_tid=1], [object_number=112, tag_tid=18, entry_tid=0], [object_number=114, tag_tid=17, entry_tid=20202], [object_number=115, tag_tid=17, entry_tid=20199]]], state=connected] 2024-04-06T09:42:32.993 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=100, tag_tid=17, entry_tid=9608], [object_number=103, tag_tid=17, entry_tid=9607], [object_number=102, tag_tid=17, entry_tid=9606], [object_number=101, tag_tid=17, entry_tid=9605]]], state=connected] 2024-04-06T09:42:32.996 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:42:32.997 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=113, tag_tid=18, entry_tid=1' 2024-04-06T09:42:32.998 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:42:32.998 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:42:32.999 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:42:32.999 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:42:32.999 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:42:32.999 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:42:32.999 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:42:32.999 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:42:32.999 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:42:32.999 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:42:32.999 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:42:32.999 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:42:33.000 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:42:33.000 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:42:33.000 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:42:33.047 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 25 2024-04-06T09:42:33.047 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 28 2024-04-06T09:42:33.047 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:42:33.047 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=113, tag_tid=18, entry_tid=1], [object_number=112, tag_tid=18, entry_tid=0], [object_number=114, tag_tid=17, entry_tid=20202], [object_number=115, tag_tid=17, entry_tid=20199]]], state=connected] 2024-04-06T09:42:33.047 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=100, tag_tid=17, entry_tid=9608], [object_number=103, tag_tid=17, entry_tid=9607], [object_number=102, tag_tid=17, entry_tid=9606], [object_number=101, tag_tid=17, entry_tid=9605]]], state=connected] 2024-04-06T09:42:33.050 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:42:33.053 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=100, tag_tid=17, entry_tid=9608' 2024-04-06T09:42:33.053 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=113, tag_tid=18, entry_tid=1' -a 'object_number=113, tag_tid=18, entry_tid=1' = 'object_number=100, tag_tid=17, entry_tid=9608' 2024-04-06T09:42:33.053 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=100, tag_tid=17, entry_tid=9608' '!=' 'object_number=102, tag_tid=17, entry_tid=9518' 2024-04-06T09:42:33.053 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:42:33.053 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=100, tag_tid=17, entry_tid=9608' = 'object_number=102, tag_tid=17, entry_tid=9518' 2024-04-06T09:42:33.053 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=100, tag_tid=17, entry_tid=9608' 2024-04-06T09:42:33.053 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=113, tag_tid=18, entry_tid=1' 2024-04-06T09:42:33.053 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:42:33.054 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:42:33.054 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=18 2024-04-06T09:42:33.055 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=100, tag_tid=17, entry_tid=9608' 2024-04-06T09:42:33.055 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:42:33.055 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:42:33.056 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=17 2024-04-06T09:42:33.056 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=113, tag_tid=18, entry_tid=1' 2024-04-06T09:42:33.056 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:42:33.057 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:42:33.057 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:42:33.058 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=100, tag_tid=17, entry_tid=9608' 2024-04-06T09:42:33.058 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:42:33.058 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:42:33.059 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=9608 2024-04-06T09:42:33.059 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 18 = 17 -a 1 -le 9608 2024-04-06T09:42:33.059 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:42:33.059 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:42:33.060 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:42:33.262 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:42:33.262 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:42:33.262 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:42:33.262 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:42:33.262 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:42:33.262 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:42:33.262 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:42:33.262 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:42:33.262 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:42:33.262 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:42:33.263 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:42:33.263 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:42:33.263 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:42:33.263 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:42:33.263 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:42:33.263 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:42:33.263 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:42:33.264 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:42:33.265 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:42:33.265 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:42:33.265 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:42:33.265 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:42:33.265 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:42:33.265 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:42:33.265 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:42:33.265 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:42:33.265 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:42:33.265 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:42:33.265 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:42:33.265 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:42:33.265 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:42:33.265 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:42:33.266 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:42:33.267 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:42:33.267 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:42:33.267 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:42:33.268 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:42:33.268 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:42:33.268 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:42:33.268 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:42:33.268 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:42:33.268 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:42:33.268 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:42:33.268 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:42:33.268 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:42:33.268 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:42:33.268 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:42:33.268 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:42:33.268 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:42:33.291 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:42:33.292 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:42:33.292 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:42:33.292 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:42:33.424 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:42:33.425 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:42:33.425 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:42:33.425 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:42:33.425 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:42:33.425 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:42:33.425 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:42:33.425 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:42:33.425 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:42:33.425 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:42:33.426 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:42:33.426 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:42:33.426 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:42:33.429 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:42:33.429 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:42:33.429 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:42:33.480 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 25 2024-04-06T09:42:33.481 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 28 2024-04-06T09:42:33.481 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:42:33.481 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=113, tag_tid=18, entry_tid=1], [object_number=112, tag_tid=18, entry_tid=0], [object_number=114, tag_tid=17, entry_tid=20202], [object_number=115, tag_tid=17, entry_tid=20199]]], state=connected] 2024-04-06T09:42:33.481 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=103, tag_tid=17, entry_tid=9691], [object_number=102, tag_tid=17, entry_tid=9690], [object_number=101, tag_tid=17, entry_tid=9689], [object_number=100, tag_tid=17, entry_tid=9688]]], state=connected] 2024-04-06T09:42:33.484 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:42:33.485 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=113, tag_tid=18, entry_tid=1' 2024-04-06T09:42:33.487 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:42:33.487 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:42:33.487 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:42:33.487 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:42:33.487 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:42:33.487 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:42:33.487 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:42:33.487 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:42:33.487 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:42:33.487 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:42:33.487 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:42:33.487 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:42:33.488 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:42:33.488 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:42:33.488 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:42:33.536 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 25 2024-04-06T09:42:33.536 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 28 2024-04-06T09:42:33.536 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:42:33.536 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=113, tag_tid=18, entry_tid=1], [object_number=112, tag_tid=18, entry_tid=0], [object_number=114, tag_tid=17, entry_tid=20202], [object_number=115, tag_tid=17, entry_tid=20199]]], state=connected] 2024-04-06T09:42:33.536 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=103, tag_tid=17, entry_tid=9691], [object_number=102, tag_tid=17, entry_tid=9690], [object_number=101, tag_tid=17, entry_tid=9689], [object_number=100, tag_tid=17, entry_tid=9688]]], state=connected] 2024-04-06T09:42:33.539 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:42:33.541 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=103, tag_tid=17, entry_tid=9691' 2024-04-06T09:42:33.541 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=113, tag_tid=18, entry_tid=1' -a 'object_number=113, tag_tid=18, entry_tid=1' = 'object_number=103, tag_tid=17, entry_tid=9691' 2024-04-06T09:42:33.541 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=103, tag_tid=17, entry_tid=9691' '!=' 'object_number=100, tag_tid=17, entry_tid=9608' 2024-04-06T09:42:33.541 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:42:33.541 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=103, tag_tid=17, entry_tid=9691' = 'object_number=100, tag_tid=17, entry_tid=9608' 2024-04-06T09:42:33.541 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=103, tag_tid=17, entry_tid=9691' 2024-04-06T09:42:33.542 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=113, tag_tid=18, entry_tid=1' 2024-04-06T09:42:33.542 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:42:33.542 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:42:33.542 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=18 2024-04-06T09:42:33.543 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=103, tag_tid=17, entry_tid=9691' 2024-04-06T09:42:33.543 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:42:33.543 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:42:33.544 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=17 2024-04-06T09:42:33.545 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=113, tag_tid=18, entry_tid=1' 2024-04-06T09:42:33.545 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:42:33.545 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:42:33.546 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:42:33.547 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=103, tag_tid=17, entry_tid=9691' 2024-04-06T09:42:33.550 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:42:33.550 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:42:33.550 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=9691 2024-04-06T09:42:33.550 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 18 = 17 -a 1 -le 9691 2024-04-06T09:42:33.550 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:42:33.551 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:42:33.551 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:42:33.750 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:42:33.750 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:42:33.750 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:42:33.750 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:42:33.750 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:42:33.750 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:42:33.753 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:42:33.753 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:42:33.753 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:42:33.754 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:42:33.754 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:42:33.754 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:42:33.754 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:42:33.754 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:42:33.754 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:42:33.756 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:42:33.756 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:42:33.756 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:42:33.757 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:42:33.757 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:42:33.757 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:42:33.757 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:42:33.757 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:42:33.757 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:42:33.757 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:42:33.757 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:42:33.757 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:42:33.757 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:42:33.757 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:42:33.757 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:42:33.757 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:42:33.757 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:42:33.758 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:42:33.758 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:42:33.758 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:42:33.758 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:42:33.758 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:42:33.758 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:42:33.758 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:42:33.758 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:42:33.758 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:42:33.758 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:42:33.758 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:42:33.758 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:42:33.758 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:42:33.758 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:42:33.759 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:42:33.759 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:42:33.759 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:42:33.782 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:42:33.782 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:42:33.782 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:42:33.782 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:42:33.910 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:42:33.910 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:42:33.911 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:42:33.911 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:42:33.911 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:42:33.911 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:42:33.911 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:42:33.911 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:42:33.911 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:42:33.911 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:42:33.912 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:42:33.912 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:42:33.912 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:42:33.914 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:42:33.915 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:42:33.915 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:42:33.968 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 25 2024-04-06T09:42:33.969 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 28 2024-04-06T09:42:33.969 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:42:33.969 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=113, tag_tid=18, entry_tid=1], [object_number=112, tag_tid=18, entry_tid=0], [object_number=114, tag_tid=17, entry_tid=20202], [object_number=115, tag_tid=17, entry_tid=20199]]], state=connected] 2024-04-06T09:42:33.969 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=100, tag_tid=17, entry_tid=9776], [object_number=103, tag_tid=17, entry_tid=9775], [object_number=102, tag_tid=17, entry_tid=9774], [object_number=101, tag_tid=17, entry_tid=9773]]], state=connected] 2024-04-06T09:42:33.972 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:42:33.973 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=113, tag_tid=18, entry_tid=1' 2024-04-06T09:42:33.975 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:42:33.975 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:42:33.975 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:42:33.975 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:42:33.975 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:42:33.975 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:42:33.975 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:42:33.975 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:42:33.975 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:42:33.975 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:42:33.975 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:42:33.975 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:42:33.976 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:42:33.976 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:42:33.976 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:42:34.023 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 25 2024-04-06T09:42:34.023 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 28 2024-04-06T09:42:34.023 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:42:34.023 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=113, tag_tid=18, entry_tid=1], [object_number=112, tag_tid=18, entry_tid=0], [object_number=114, tag_tid=17, entry_tid=20202], [object_number=115, tag_tid=17, entry_tid=20199]]], state=connected] 2024-04-06T09:42:34.023 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=100, tag_tid=17, entry_tid=9776], [object_number=103, tag_tid=17, entry_tid=9775], [object_number=102, tag_tid=17, entry_tid=9774], [object_number=101, tag_tid=17, entry_tid=9773]]], state=connected] 2024-04-06T09:42:34.026 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:42:34.028 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=100, tag_tid=17, entry_tid=9776' 2024-04-06T09:42:34.028 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=113, tag_tid=18, entry_tid=1' -a 'object_number=113, tag_tid=18, entry_tid=1' = 'object_number=100, tag_tid=17, entry_tid=9776' 2024-04-06T09:42:34.028 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=100, tag_tid=17, entry_tid=9776' '!=' 'object_number=103, tag_tid=17, entry_tid=9691' 2024-04-06T09:42:34.028 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:42:34.029 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=100, tag_tid=17, entry_tid=9776' = 'object_number=103, tag_tid=17, entry_tid=9691' 2024-04-06T09:42:34.029 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=100, tag_tid=17, entry_tid=9776' 2024-04-06T09:42:34.029 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=113, tag_tid=18, entry_tid=1' 2024-04-06T09:42:34.029 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:42:34.029 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:42:34.030 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=18 2024-04-06T09:42:34.030 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=100, tag_tid=17, entry_tid=9776' 2024-04-06T09:42:34.030 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:42:34.031 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:42:34.031 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=17 2024-04-06T09:42:34.032 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=113, tag_tid=18, entry_tid=1' 2024-04-06T09:42:34.032 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:42:34.032 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:42:34.033 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:42:34.034 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=100, tag_tid=17, entry_tid=9776' 2024-04-06T09:42:34.034 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:42:34.034 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:42:34.035 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=9776 2024-04-06T09:42:34.035 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 18 = 17 -a 1 -le 9776 2024-04-06T09:42:34.035 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:42:34.036 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:42:34.036 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:42:34.237 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:42:34.237 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:42:34.237 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:42:34.237 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:42:34.237 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:42:34.237 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:42:34.237 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:42:34.237 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:42:34.237 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:42:34.237 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:42:34.238 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:42:34.238 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:42:34.238 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:42:34.238 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:42:34.238 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:42:34.238 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:42:34.239 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:42:34.242 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:42:34.243 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:42:34.243 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:42:34.243 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:42:34.243 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:42:34.243 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:42:34.243 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:42:34.243 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:42:34.243 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:42:34.243 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:42:34.243 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:42:34.243 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:42:34.243 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:42:34.243 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:42:34.244 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:42:34.244 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:42:34.244 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:42:34.244 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:42:34.244 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:42:34.244 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:42:34.244 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:42:34.244 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:42:34.244 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:42:34.244 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:42:34.244 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:42:34.244 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:42:34.244 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:42:34.244 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:42:34.244 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:42:34.245 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:42:34.245 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:42:34.245 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:42:34.268 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:42:34.268 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:42:34.268 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:42:34.268 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:42:34.406 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:42:34.407 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:42:34.407 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:42:34.407 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:42:34.407 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:42:34.408 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:42:34.408 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:42:34.408 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:42:34.408 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:42:34.408 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:42:34.408 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:42:34.408 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:42:34.408 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:42:34.409 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:42:34.410 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:42:34.410 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:42:34.462 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 25 2024-04-06T09:42:34.462 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 28 2024-04-06T09:42:34.462 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:42:34.462 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=113, tag_tid=18, entry_tid=1], [object_number=112, tag_tid=18, entry_tid=0], [object_number=114, tag_tid=17, entry_tid=20202], [object_number=115, tag_tid=17, entry_tid=20199]]], state=connected] 2024-04-06T09:42:34.462 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=101, tag_tid=17, entry_tid=9865], [object_number=100, tag_tid=17, entry_tid=9864], [object_number=103, tag_tid=17, entry_tid=9863], [object_number=102, tag_tid=17, entry_tid=9862]]], state=connected] 2024-04-06T09:42:34.465 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:42:34.467 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=113, tag_tid=18, entry_tid=1' 2024-04-06T09:42:34.468 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:42:34.468 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:42:34.468 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:42:34.468 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:42:34.468 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:42:34.468 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:42:34.468 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:42:34.469 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:42:34.469 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:42:34.469 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:42:34.469 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:42:34.469 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:42:34.469 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:42:34.470 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:42:34.470 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:42:34.517 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 25 2024-04-06T09:42:34.517 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 28 2024-04-06T09:42:34.517 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:42:34.517 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=113, tag_tid=18, entry_tid=1], [object_number=112, tag_tid=18, entry_tid=0], [object_number=114, tag_tid=17, entry_tid=20202], [object_number=115, tag_tid=17, entry_tid=20199]]], state=connected] 2024-04-06T09:42:34.517 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=101, tag_tid=17, entry_tid=9865], [object_number=100, tag_tid=17, entry_tid=9864], [object_number=103, tag_tid=17, entry_tid=9863], [object_number=102, tag_tid=17, entry_tid=9862]]], state=connected] 2024-04-06T09:42:34.521 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:42:34.523 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=101, tag_tid=17, entry_tid=9865' 2024-04-06T09:42:34.523 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=113, tag_tid=18, entry_tid=1' -a 'object_number=113, tag_tid=18, entry_tid=1' = 'object_number=101, tag_tid=17, entry_tid=9865' 2024-04-06T09:42:34.523 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=101, tag_tid=17, entry_tid=9865' '!=' 'object_number=100, tag_tid=17, entry_tid=9776' 2024-04-06T09:42:34.523 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:42:34.523 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=101, tag_tid=17, entry_tid=9865' = 'object_number=100, tag_tid=17, entry_tid=9776' 2024-04-06T09:42:34.524 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=101, tag_tid=17, entry_tid=9865' 2024-04-06T09:42:34.524 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=113, tag_tid=18, entry_tid=1' 2024-04-06T09:42:34.524 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:42:34.524 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:42:34.524 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=18 2024-04-06T09:42:34.525 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=101, tag_tid=17, entry_tid=9865' 2024-04-06T09:42:34.525 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:42:34.525 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:42:34.526 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=17 2024-04-06T09:42:34.527 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=113, tag_tid=18, entry_tid=1' 2024-04-06T09:42:34.527 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:42:34.527 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:42:34.528 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:42:34.529 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=101, tag_tid=17, entry_tid=9865' 2024-04-06T09:42:34.529 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:42:34.529 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:42:34.530 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=9865 2024-04-06T09:42:34.530 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 18 = 17 -a 1 -le 9865 2024-04-06T09:42:34.530 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:42:34.530 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:42:34.530 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:42:34.732 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:42:34.732 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:42:34.732 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:42:34.732 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:42:34.732 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:42:34.732 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:42:34.732 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:42:34.732 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:42:34.733 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:42:34.733 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:42:34.733 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:42:34.733 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:42:34.733 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:42:34.733 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:42:34.733 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:42:34.733 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:42:34.734 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:42:34.736 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:42:34.736 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:42:34.736 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:42:34.736 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:42:34.736 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:42:34.736 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:42:34.736 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:42:34.736 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:42:34.736 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:42:34.736 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:42:34.737 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:42:34.737 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:42:34.737 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:42:34.737 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:42:34.737 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:42:34.737 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:42:34.738 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:42:34.738 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:42:34.738 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:42:34.738 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:42:34.738 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:42:34.738 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:42:34.738 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:42:34.738 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:42:34.739 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:42:34.739 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:42:34.739 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:42:34.739 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:42:34.739 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:42:34.739 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:42:34.739 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:42:34.739 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:42:34.763 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:42:34.763 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:42:34.763 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:42:34.764 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:42:34.896 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:42:34.897 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:42:34.898 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:42:34.898 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:42:34.898 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:42:34.898 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:42:34.898 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:42:34.898 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:42:34.898 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:42:34.898 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:42:34.899 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:42:34.899 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:42:34.899 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:42:34.902 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:42:34.902 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:42:34.902 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:42:34.949 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 25 2024-04-06T09:42:34.950 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 28 2024-04-06T09:42:34.950 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:42:34.950 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=113, tag_tid=18, entry_tid=1], [object_number=112, tag_tid=18, entry_tid=0], [object_number=114, tag_tid=17, entry_tid=20202], [object_number=115, tag_tid=17, entry_tid=20199]]], state=connected] 2024-04-06T09:42:34.950 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=102, tag_tid=17, entry_tid=9950], [object_number=101, tag_tid=17, entry_tid=9949], [object_number=100, tag_tid=17, entry_tid=9948], [object_number=103, tag_tid=17, entry_tid=9947]]], state=connected] 2024-04-06T09:42:34.953 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:42:34.955 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=113, tag_tid=18, entry_tid=1' 2024-04-06T09:42:34.956 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:42:34.956 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:42:34.956 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:42:34.956 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:42:34.956 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:42:34.956 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:42:34.956 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:42:34.956 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:42:34.956 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:42:34.957 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:42:34.957 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:42:34.957 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:42:34.957 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:42:34.957 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:42:34.957 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:42:35.003 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 25 2024-04-06T09:42:35.003 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 28 2024-04-06T09:42:35.003 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:42:35.004 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=113, tag_tid=18, entry_tid=1], [object_number=112, tag_tid=18, entry_tid=0], [object_number=114, tag_tid=17, entry_tid=20202], [object_number=115, tag_tid=17, entry_tid=20199]]], state=connected] 2024-04-06T09:42:35.004 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=102, tag_tid=17, entry_tid=9950], [object_number=101, tag_tid=17, entry_tid=9949], [object_number=100, tag_tid=17, entry_tid=9948], [object_number=103, tag_tid=17, entry_tid=9947]]], state=connected] 2024-04-06T09:42:35.006 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:42:35.009 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=102, tag_tid=17, entry_tid=9950' 2024-04-06T09:42:35.009 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=113, tag_tid=18, entry_tid=1' -a 'object_number=113, tag_tid=18, entry_tid=1' = 'object_number=102, tag_tid=17, entry_tid=9950' 2024-04-06T09:42:35.009 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=102, tag_tid=17, entry_tid=9950' '!=' 'object_number=101, tag_tid=17, entry_tid=9865' 2024-04-06T09:42:35.009 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:42:35.009 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=102, tag_tid=17, entry_tid=9950' = 'object_number=101, tag_tid=17, entry_tid=9865' 2024-04-06T09:42:35.009 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=102, tag_tid=17, entry_tid=9950' 2024-04-06T09:42:35.009 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=113, tag_tid=18, entry_tid=1' 2024-04-06T09:42:35.009 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:42:35.009 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:42:35.010 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=18 2024-04-06T09:42:35.010 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=102, tag_tid=17, entry_tid=9950' 2024-04-06T09:42:35.011 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:42:35.011 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:42:35.011 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=17 2024-04-06T09:42:35.012 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=113, tag_tid=18, entry_tid=1' 2024-04-06T09:42:35.012 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:42:35.012 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:42:35.013 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:42:35.014 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=102, tag_tid=17, entry_tid=9950' 2024-04-06T09:42:35.014 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:42:35.014 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:42:35.015 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=9950 2024-04-06T09:42:35.015 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 18 = 17 -a 1 -le 9950 2024-04-06T09:42:35.015 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:42:35.015 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:42:35.015 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:42:35.217 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:42:35.217 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:42:35.217 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:42:35.217 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:42:35.217 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:42:35.217 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:42:35.217 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:42:35.217 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:42:35.218 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:42:35.218 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:42:35.218 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:42:35.218 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:42:35.218 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:42:35.218 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:42:35.218 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:42:35.218 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:42:35.219 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:42:35.221 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:42:35.221 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:42:35.221 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:42:35.221 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:42:35.221 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:42:35.221 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:42:35.221 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:42:35.221 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:42:35.221 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:42:35.222 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:42:35.222 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:42:35.222 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:42:35.222 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:42:35.222 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:42:35.222 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:42:35.222 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:42:35.223 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:42:35.223 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:42:35.223 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:42:35.223 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:42:35.223 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:42:35.223 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:42:35.223 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:42:35.223 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:42:35.223 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:42:35.223 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:42:35.224 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:42:35.224 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:42:35.224 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:42:35.224 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:42:35.224 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:42:35.224 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:42:35.247 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:42:35.248 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:42:35.248 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:42:35.248 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:42:35.389 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:42:35.390 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:42:35.390 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:42:35.390 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:42:35.390 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:42:35.390 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:42:35.390 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:42:35.391 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:42:35.391 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:42:35.391 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:42:35.391 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:42:35.391 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:42:35.391 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:42:35.394 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:42:35.394 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:42:35.394 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:42:35.441 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 25 2024-04-06T09:42:35.441 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 28 2024-04-06T09:42:35.441 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:42:35.441 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=113, tag_tid=18, entry_tid=1], [object_number=112, tag_tid=18, entry_tid=0], [object_number=114, tag_tid=17, entry_tid=20202], [object_number=115, tag_tid=17, entry_tid=20199]]], state=connected] 2024-04-06T09:42:35.442 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=101, tag_tid=17, entry_tid=10033], [object_number=100, tag_tid=17, entry_tid=10032], [object_number=103, tag_tid=17, entry_tid=10031], [object_number=102, tag_tid=17, entry_tid=10030]]], state=connected] 2024-04-06T09:42:35.445 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:42:35.447 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=113, tag_tid=18, entry_tid=1' 2024-04-06T09:42:35.447 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:42:35.447 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:42:35.447 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:42:35.447 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:42:35.448 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:42:35.448 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:42:35.448 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:42:35.448 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:42:35.448 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:42:35.448 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:42:35.448 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:42:35.448 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:42:35.448 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:42:35.449 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:42:35.449 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:42:35.495 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 25 2024-04-06T09:42:35.495 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 28 2024-04-06T09:42:35.495 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:42:35.495 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=113, tag_tid=18, entry_tid=1], [object_number=112, tag_tid=18, entry_tid=0], [object_number=114, tag_tid=17, entry_tid=20202], [object_number=115, tag_tid=17, entry_tid=20199]]], state=connected] 2024-04-06T09:42:35.495 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=101, tag_tid=17, entry_tid=10033], [object_number=100, tag_tid=17, entry_tid=10032], [object_number=103, tag_tid=17, entry_tid=10031], [object_number=102, tag_tid=17, entry_tid=10030]]], state=connected] 2024-04-06T09:42:35.499 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:42:35.501 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=101, tag_tid=17, entry_tid=10033' 2024-04-06T09:42:35.501 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=113, tag_tid=18, entry_tid=1' -a 'object_number=113, tag_tid=18, entry_tid=1' = 'object_number=101, tag_tid=17, entry_tid=10033' 2024-04-06T09:42:35.501 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=101, tag_tid=17, entry_tid=10033' '!=' 'object_number=102, tag_tid=17, entry_tid=9950' 2024-04-06T09:42:35.501 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:42:35.501 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=101, tag_tid=17, entry_tid=10033' = 'object_number=102, tag_tid=17, entry_tid=9950' 2024-04-06T09:42:35.501 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=101, tag_tid=17, entry_tid=10033' 2024-04-06T09:42:35.501 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=113, tag_tid=18, entry_tid=1' 2024-04-06T09:42:35.501 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:42:35.501 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:42:35.502 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=18 2024-04-06T09:42:35.502 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=101, tag_tid=17, entry_tid=10033' 2024-04-06T09:42:35.503 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:42:35.503 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:42:35.504 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=17 2024-04-06T09:42:35.504 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=113, tag_tid=18, entry_tid=1' 2024-04-06T09:42:35.504 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:42:35.505 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:42:35.505 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:42:35.506 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=101, tag_tid=17, entry_tid=10033' 2024-04-06T09:42:35.506 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:42:35.506 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:42:35.507 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=10033 2024-04-06T09:42:35.508 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 18 = 17 -a 1 -le 10033 2024-04-06T09:42:35.508 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:42:35.508 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:42:35.508 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:42:35.710 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:42:35.710 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:42:35.710 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:42:35.710 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:42:35.710 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:42:35.710 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:42:35.710 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:42:35.710 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:42:35.710 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:42:35.710 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:42:35.710 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:42:35.711 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:42:35.711 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:42:35.711 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:42:35.711 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:42:35.711 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:42:35.711 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:42:35.715 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:42:35.715 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:42:35.715 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:42:35.715 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:42:35.715 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:42:35.715 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:42:35.715 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:42:35.715 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:42:35.715 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:42:35.715 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:42:35.715 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:42:35.715 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:42:35.715 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:42:35.716 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:42:35.716 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:42:35.716 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:42:35.716 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:42:35.716 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:42:35.716 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:42:35.716 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:42:35.716 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:42:35.716 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:42:35.716 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:42:35.716 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:42:35.716 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:42:35.716 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:42:35.716 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:42:35.716 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:42:35.716 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:42:35.717 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:42:35.717 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:42:35.717 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:42:35.740 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:42:35.740 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:42:35.740 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:42:35.740 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:42:35.868 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:42:35.869 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:42:35.869 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:42:35.869 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:42:35.869 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:42:35.870 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:42:35.870 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:42:35.870 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:42:35.870 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:42:35.870 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:42:35.870 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:42:35.871 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:42:35.871 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:42:35.872 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:42:35.872 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:42:35.873 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:42:35.953 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 25 2024-04-06T09:42:35.953 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 28 2024-04-06T09:42:35.953 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:42:35.953 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=113, tag_tid=18, entry_tid=1], [object_number=112, tag_tid=18, entry_tid=0], [object_number=114, tag_tid=17, entry_tid=20202], [object_number=115, tag_tid=17, entry_tid=20199]]], state=connected] 2024-04-06T09:42:35.953 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=101, tag_tid=17, entry_tid=10113], [object_number=100, tag_tid=17, entry_tid=10112], [object_number=103, tag_tid=17, entry_tid=10111], [object_number=102, tag_tid=17, entry_tid=10110]]], state=connected] 2024-04-06T09:42:35.959 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:42:35.960 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=113, tag_tid=18, entry_tid=1' 2024-04-06T09:42:35.961 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:42:35.962 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:42:35.962 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:42:35.962 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:42:35.962 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:42:35.962 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:42:35.962 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:42:35.962 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:42:35.962 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:42:35.962 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:42:35.962 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:42:35.962 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:42:35.963 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:42:35.963 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:42:35.963 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:42:36.016 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 25 2024-04-06T09:42:36.016 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 28 2024-04-06T09:42:36.016 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:42:36.016 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=113, tag_tid=18, entry_tid=1], [object_number=112, tag_tid=18, entry_tid=0], [object_number=114, tag_tid=17, entry_tid=20202], [object_number=115, tag_tid=17, entry_tid=20199]]], state=connected] 2024-04-06T09:42:36.016 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=101, tag_tid=17, entry_tid=10113], [object_number=100, tag_tid=17, entry_tid=10112], [object_number=103, tag_tid=17, entry_tid=10111], [object_number=102, tag_tid=17, entry_tid=10110]]], state=connected] 2024-04-06T09:42:36.020 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:42:36.022 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=101, tag_tid=17, entry_tid=10113' 2024-04-06T09:42:36.022 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=113, tag_tid=18, entry_tid=1' -a 'object_number=113, tag_tid=18, entry_tid=1' = 'object_number=101, tag_tid=17, entry_tid=10113' 2024-04-06T09:42:36.022 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=101, tag_tid=17, entry_tid=10113' '!=' 'object_number=101, tag_tid=17, entry_tid=10033' 2024-04-06T09:42:36.022 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:42:36.022 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=101, tag_tid=17, entry_tid=10113' = 'object_number=101, tag_tid=17, entry_tid=10033' 2024-04-06T09:42:36.023 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=101, tag_tid=17, entry_tid=10113' 2024-04-06T09:42:36.023 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=113, tag_tid=18, entry_tid=1' 2024-04-06T09:42:36.023 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:42:36.023 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:42:36.023 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=18 2024-04-06T09:42:36.024 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=101, tag_tid=17, entry_tid=10113' 2024-04-06T09:42:36.024 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:42:36.024 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:42:36.025 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=17 2024-04-06T09:42:36.025 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=113, tag_tid=18, entry_tid=1' 2024-04-06T09:42:36.026 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:42:36.026 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:42:36.027 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:42:36.027 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=101, tag_tid=17, entry_tid=10113' 2024-04-06T09:42:36.027 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:42:36.027 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:42:36.028 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=10113 2024-04-06T09:42:36.029 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 18 = 17 -a 1 -le 10113 2024-04-06T09:42:36.029 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:42:36.029 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:42:36.029 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:42:36.230 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:42:36.231 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:42:36.231 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:42:36.231 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:42:36.231 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:42:36.231 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:42:36.231 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:42:36.231 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:42:36.231 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:42:36.231 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:42:36.231 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:42:36.231 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:42:36.231 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:42:36.231 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:42:36.231 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:42:36.232 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:42:36.232 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:42:36.235 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:42:36.235 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:42:36.235 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:42:36.235 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:42:36.236 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:42:36.236 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:42:36.236 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:42:36.236 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:42:36.236 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:42:36.236 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:42:36.236 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:42:36.236 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:42:36.236 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:42:36.236 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:42:36.236 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:42:36.237 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:42:36.237 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:42:36.237 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:42:36.237 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:42:36.237 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:42:36.237 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:42:36.237 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:42:36.237 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:42:36.237 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:42:36.237 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:42:36.237 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:42:36.237 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:42:36.237 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:42:36.237 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:42:36.237 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:42:36.238 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:42:36.238 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:42:36.260 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:42:36.260 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:42:36.261 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:42:36.261 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:42:36.396 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:42:36.397 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:42:36.397 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:42:36.397 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:42:36.397 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:42:36.397 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:42:36.397 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:42:36.397 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:42:36.397 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:42:36.398 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:42:36.398 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:42:36.398 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:42:36.398 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:42:36.400 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:42:36.401 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:42:36.401 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:42:36.455 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 26 2024-04-06T09:42:36.456 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 28 2024-04-06T09:42:36.456 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:42:36.456 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=113, tag_tid=18, entry_tid=1], [object_number=112, tag_tid=18, entry_tid=0], [object_number=114, tag_tid=17, entry_tid=20202], [object_number=115, tag_tid=17, entry_tid=20199]]], state=connected] 2024-04-06T09:42:36.456 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=104, tag_tid=17, entry_tid=10164], [object_number=107, tag_tid=17, entry_tid=10163], [object_number=106, tag_tid=17, entry_tid=10162], [object_number=105, tag_tid=17, entry_tid=10161]]], state=connected] 2024-04-06T09:42:36.458 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:42:36.460 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=113, tag_tid=18, entry_tid=1' 2024-04-06T09:42:36.461 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:42:36.461 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:42:36.461 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:42:36.461 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:42:36.461 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:42:36.461 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:42:36.461 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:42:36.461 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:42:36.462 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:42:36.462 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:42:36.462 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:42:36.462 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:42:36.462 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:42:36.462 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:42:36.462 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:42:36.508 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 26 2024-04-06T09:42:36.509 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 28 2024-04-06T09:42:36.509 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:42:36.509 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=113, tag_tid=18, entry_tid=1], [object_number=112, tag_tid=18, entry_tid=0], [object_number=114, tag_tid=17, entry_tid=20202], [object_number=115, tag_tid=17, entry_tid=20199]]], state=connected] 2024-04-06T09:42:36.509 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=104, tag_tid=17, entry_tid=10164], [object_number=107, tag_tid=17, entry_tid=10163], [object_number=106, tag_tid=17, entry_tid=10162], [object_number=105, tag_tid=17, entry_tid=10161]]], state=connected] 2024-04-06T09:42:36.512 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:42:36.514 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=104, tag_tid=17, entry_tid=10164' 2024-04-06T09:42:36.514 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=113, tag_tid=18, entry_tid=1' -a 'object_number=113, tag_tid=18, entry_tid=1' = 'object_number=104, tag_tid=17, entry_tid=10164' 2024-04-06T09:42:36.514 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=104, tag_tid=17, entry_tid=10164' '!=' 'object_number=101, tag_tid=17, entry_tid=10113' 2024-04-06T09:42:36.514 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:42:36.514 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=104, tag_tid=17, entry_tid=10164' = 'object_number=101, tag_tid=17, entry_tid=10113' 2024-04-06T09:42:36.514 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=104, tag_tid=17, entry_tid=10164' 2024-04-06T09:42:36.514 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=113, tag_tid=18, entry_tid=1' 2024-04-06T09:42:36.515 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:42:36.515 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:42:36.515 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=18 2024-04-06T09:42:36.516 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=104, tag_tid=17, entry_tid=10164' 2024-04-06T09:42:36.516 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:42:36.516 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:42:36.517 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=17 2024-04-06T09:42:36.518 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=113, tag_tid=18, entry_tid=1' 2024-04-06T09:42:36.518 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:42:36.518 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:42:36.519 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:42:36.519 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=104, tag_tid=17, entry_tid=10164' 2024-04-06T09:42:36.519 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:42:36.519 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:42:36.520 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=10164 2024-04-06T09:42:36.521 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 18 = 17 -a 1 -le 10164 2024-04-06T09:42:36.521 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:42:36.521 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:42:36.521 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:42:36.722 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:42:36.723 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:42:36.723 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:42:36.723 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:42:36.723 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:42:36.723 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:42:36.723 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:42:36.723 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:42:36.723 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:42:36.723 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:42:36.723 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:42:36.723 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:42:36.723 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:42:36.724 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:42:36.724 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:42:36.724 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:42:36.725 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:42:36.727 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:42:36.727 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:42:36.727 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:42:36.727 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:42:36.727 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:42:36.727 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:42:36.727 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:42:36.727 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:42:36.728 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:42:36.728 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:42:36.728 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:42:36.728 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:42:36.728 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:42:36.728 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:42:36.728 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:42:36.728 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:42:36.729 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:42:36.729 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:42:36.729 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:42:36.729 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:42:36.729 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:42:36.729 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:42:36.729 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:42:36.729 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:42:36.729 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:42:36.730 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:42:36.730 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:42:36.730 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:42:36.730 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:42:36.730 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:42:36.730 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:42:36.730 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:42:36.755 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:42:36.755 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:42:36.755 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:42:36.755 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:42:36.882 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:42:36.884 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:42:36.884 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:42:36.884 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:42:36.884 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:42:36.884 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:42:36.884 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:42:36.884 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:42:36.884 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:42:36.884 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:42:36.885 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:42:36.885 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:42:36.885 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:42:36.887 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:42:36.887 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:42:36.887 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:42:37.143 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 26 2024-04-06T09:42:37.143 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 28 2024-04-06T09:42:37.143 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:42:37.143 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=113, tag_tid=18, entry_tid=1], [object_number=112, tag_tid=18, entry_tid=0], [object_number=114, tag_tid=17, entry_tid=20202], [object_number=115, tag_tid=17, entry_tid=20199]]], state=connected] 2024-04-06T09:42:37.143 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=106, tag_tid=17, entry_tid=10246], [object_number=105, tag_tid=17, entry_tid=10245], [object_number=104, tag_tid=17, entry_tid=10244], [object_number=107, tag_tid=17, entry_tid=10243]]], state=connected] 2024-04-06T09:42:37.148 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:42:37.150 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=113, tag_tid=18, entry_tid=1' 2024-04-06T09:42:37.151 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:42:37.152 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:42:37.152 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:42:37.152 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:42:37.152 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:42:37.152 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:42:37.152 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:42:37.152 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:42:37.152 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:42:37.153 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:42:37.153 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:42:37.154 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:42:37.154 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:42:37.154 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:42:37.155 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:42:37.204 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 26 2024-04-06T09:42:37.204 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 28 2024-04-06T09:42:37.204 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:42:37.204 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=113, tag_tid=18, entry_tid=1], [object_number=112, tag_tid=18, entry_tid=0], [object_number=114, tag_tid=17, entry_tid=20202], [object_number=115, tag_tid=17, entry_tid=20199]]], state=connected] 2024-04-06T09:42:37.204 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=106, tag_tid=17, entry_tid=10246], [object_number=105, tag_tid=17, entry_tid=10245], [object_number=104, tag_tid=17, entry_tid=10244], [object_number=107, tag_tid=17, entry_tid=10243]]], state=connected] 2024-04-06T09:42:37.208 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:42:37.210 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=106, tag_tid=17, entry_tid=10246' 2024-04-06T09:42:37.210 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=113, tag_tid=18, entry_tid=1' -a 'object_number=113, tag_tid=18, entry_tid=1' = 'object_number=106, tag_tid=17, entry_tid=10246' 2024-04-06T09:42:37.210 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=106, tag_tid=17, entry_tid=10246' '!=' 'object_number=104, tag_tid=17, entry_tid=10164' 2024-04-06T09:42:37.210 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:42:37.210 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=106, tag_tid=17, entry_tid=10246' = 'object_number=104, tag_tid=17, entry_tid=10164' 2024-04-06T09:42:37.210 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=106, tag_tid=17, entry_tid=10246' 2024-04-06T09:42:37.211 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=113, tag_tid=18, entry_tid=1' 2024-04-06T09:42:37.211 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:42:37.211 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:42:37.212 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=18 2024-04-06T09:42:37.212 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=106, tag_tid=17, entry_tid=10246' 2024-04-06T09:42:37.213 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:42:37.213 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:42:37.213 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=17 2024-04-06T09:42:37.214 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=113, tag_tid=18, entry_tid=1' 2024-04-06T09:42:37.214 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:42:37.214 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:42:37.215 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:42:37.216 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=106, tag_tid=17, entry_tid=10246' 2024-04-06T09:42:37.216 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:42:37.216 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:42:37.218 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=10246 2024-04-06T09:42:37.218 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 18 = 17 -a 1 -le 10246 2024-04-06T09:42:37.218 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:42:37.218 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:42:37.218 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:42:37.419 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:42:37.419 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:42:37.419 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:42:37.420 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:42:37.420 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:42:37.420 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:42:37.420 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:42:37.420 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:42:37.420 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:42:37.420 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:42:37.420 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:42:37.420 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:42:37.420 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:42:37.420 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:42:37.420 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:42:37.421 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:42:37.421 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:42:37.423 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:42:37.424 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:42:37.424 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:42:37.424 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:42:37.424 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:42:37.424 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:42:37.424 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:42:37.424 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:42:37.424 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:42:37.424 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:42:37.424 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:42:37.424 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:42:37.424 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:42:37.424 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:42:37.425 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:42:37.425 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:42:37.426 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:42:37.426 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:42:37.426 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:42:37.426 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:42:37.426 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:42:37.426 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:42:37.426 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:42:37.426 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:42:37.426 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:42:37.426 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:42:37.426 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:42:37.426 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:42:37.426 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:42:37.427 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:42:37.427 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:42:37.427 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:42:37.451 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:42:37.452 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:42:37.452 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:42:37.452 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:42:37.585 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:42:37.587 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:42:37.587 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:42:37.587 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:42:37.587 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:42:37.587 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:42:37.587 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:42:37.587 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:42:37.587 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:42:37.587 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:42:37.588 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:42:37.588 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:42:37.588 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:42:37.590 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:42:37.591 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:42:37.591 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:42:37.645 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 26 2024-04-06T09:42:37.645 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 28 2024-04-06T09:42:37.645 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:42:37.645 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=113, tag_tid=18, entry_tid=1], [object_number=112, tag_tid=18, entry_tid=0], [object_number=114, tag_tid=17, entry_tid=20202], [object_number=115, tag_tid=17, entry_tid=20199]]], state=connected] 2024-04-06T09:42:37.646 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=104, tag_tid=17, entry_tid=10364], [object_number=107, tag_tid=17, entry_tid=10363], [object_number=106, tag_tid=17, entry_tid=10362], [object_number=105, tag_tid=17, entry_tid=10361]]], state=connected] 2024-04-06T09:42:37.649 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:42:37.650 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=113, tag_tid=18, entry_tid=1' 2024-04-06T09:42:37.651 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:42:37.651 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:42:37.651 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:42:37.652 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:42:37.652 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:42:37.652 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:42:37.652 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:42:37.652 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:42:37.652 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:42:37.652 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:42:37.652 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:42:37.652 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:42:37.653 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:42:37.653 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:42:37.653 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:42:37.700 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 26 2024-04-06T09:42:37.700 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 28 2024-04-06T09:42:37.700 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:42:37.700 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=113, tag_tid=18, entry_tid=1], [object_number=112, tag_tid=18, entry_tid=0], [object_number=114, tag_tid=17, entry_tid=20202], [object_number=115, tag_tid=17, entry_tid=20199]]], state=connected] 2024-04-06T09:42:37.700 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=104, tag_tid=17, entry_tid=10364], [object_number=107, tag_tid=17, entry_tid=10363], [object_number=106, tag_tid=17, entry_tid=10362], [object_number=105, tag_tid=17, entry_tid=10361]]], state=connected] 2024-04-06T09:42:37.703 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:42:37.705 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=104, tag_tid=17, entry_tid=10364' 2024-04-06T09:42:37.705 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=113, tag_tid=18, entry_tid=1' -a 'object_number=113, tag_tid=18, entry_tid=1' = 'object_number=104, tag_tid=17, entry_tid=10364' 2024-04-06T09:42:37.705 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=104, tag_tid=17, entry_tid=10364' '!=' 'object_number=106, tag_tid=17, entry_tid=10246' 2024-04-06T09:42:37.705 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:42:37.705 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=104, tag_tid=17, entry_tid=10364' = 'object_number=106, tag_tid=17, entry_tid=10246' 2024-04-06T09:42:37.706 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=104, tag_tid=17, entry_tid=10364' 2024-04-06T09:42:37.706 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=113, tag_tid=18, entry_tid=1' 2024-04-06T09:42:37.707 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:42:37.707 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:42:37.707 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=18 2024-04-06T09:42:37.708 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=104, tag_tid=17, entry_tid=10364' 2024-04-06T09:42:37.708 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:42:37.708 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:42:37.709 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=17 2024-04-06T09:42:37.709 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=113, tag_tid=18, entry_tid=1' 2024-04-06T09:42:37.709 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:42:37.709 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:42:37.710 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:42:37.711 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=104, tag_tid=17, entry_tid=10364' 2024-04-06T09:42:37.711 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:42:37.711 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:42:37.712 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=10364 2024-04-06T09:42:37.712 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 18 = 17 -a 1 -le 10364 2024-04-06T09:42:37.713 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:42:37.713 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:42:37.713 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:42:37.914 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:42:37.914 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:42:37.914 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:42:37.915 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:42:37.915 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:42:37.915 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:42:37.915 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:42:37.915 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:42:37.915 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:42:37.915 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:42:37.915 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:42:37.915 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:42:37.915 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:42:37.915 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:42:37.915 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:42:37.916 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:42:37.916 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:42:37.917 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:42:37.918 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:42:37.918 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:42:37.918 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:42:37.918 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:42:37.918 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:42:37.918 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:42:37.918 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:42:37.918 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:42:37.918 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:42:37.918 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:42:37.918 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:42:37.918 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:42:37.919 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:42:37.919 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:42:37.919 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:42:37.920 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:42:37.920 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:42:37.921 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:42:37.921 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:42:37.921 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:42:37.921 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:42:37.921 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:42:37.921 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:42:37.921 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:42:37.921 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:42:37.921 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:42:37.921 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:42:37.921 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:42:37.921 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:42:37.922 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:42:37.922 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:42:37.944 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:42:37.944 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:42:37.944 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:42:37.944 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:42:38.081 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:42:38.082 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:42:38.082 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:42:38.082 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:42:38.082 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:42:38.082 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:42:38.083 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:42:38.083 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:42:38.083 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:42:38.083 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:42:38.083 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:42:38.083 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:42:38.084 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:42:38.084 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:42:38.084 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:42:38.085 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:42:38.137 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 26 2024-04-06T09:42:38.137 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 28 2024-04-06T09:42:38.137 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:42:38.137 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=113, tag_tid=18, entry_tid=1], [object_number=112, tag_tid=18, entry_tid=0], [object_number=114, tag_tid=17, entry_tid=20202], [object_number=115, tag_tid=17, entry_tid=20199]]], state=connected] 2024-04-06T09:42:38.138 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=104, tag_tid=17, entry_tid=10448], [object_number=107, tag_tid=17, entry_tid=10447], [object_number=106, tag_tid=17, entry_tid=10446], [object_number=105, tag_tid=17, entry_tid=10445]]], state=connected] 2024-04-06T09:42:38.141 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:42:38.142 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=113, tag_tid=18, entry_tid=1' 2024-04-06T09:42:38.144 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:42:38.144 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:42:38.144 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:42:38.144 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:42:38.144 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:42:38.144 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:42:38.144 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:42:38.145 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:42:38.145 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:42:38.145 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:42:38.145 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:42:38.145 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:42:38.146 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:42:38.146 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:42:38.146 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:42:38.192 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 26 2024-04-06T09:42:38.192 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 28 2024-04-06T09:42:38.192 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:42:38.192 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=113, tag_tid=18, entry_tid=1], [object_number=112, tag_tid=18, entry_tid=0], [object_number=114, tag_tid=17, entry_tid=20202], [object_number=115, tag_tid=17, entry_tid=20199]]], state=connected] 2024-04-06T09:42:38.192 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=104, tag_tid=17, entry_tid=10448], [object_number=107, tag_tid=17, entry_tid=10447], [object_number=106, tag_tid=17, entry_tid=10446], [object_number=105, tag_tid=17, entry_tid=10445]]], state=connected] 2024-04-06T09:42:38.195 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:42:38.197 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=104, tag_tid=17, entry_tid=10448' 2024-04-06T09:42:38.198 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=113, tag_tid=18, entry_tid=1' -a 'object_number=113, tag_tid=18, entry_tid=1' = 'object_number=104, tag_tid=17, entry_tid=10448' 2024-04-06T09:42:38.198 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=104, tag_tid=17, entry_tid=10448' '!=' 'object_number=104, tag_tid=17, entry_tid=10364' 2024-04-06T09:42:38.198 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:42:38.198 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=104, tag_tid=17, entry_tid=10448' = 'object_number=104, tag_tid=17, entry_tid=10364' 2024-04-06T09:42:38.198 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=104, tag_tid=17, entry_tid=10448' 2024-04-06T09:42:38.198 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=113, tag_tid=18, entry_tid=1' 2024-04-06T09:42:38.198 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:42:38.198 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:42:38.199 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=18 2024-04-06T09:42:38.199 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=104, tag_tid=17, entry_tid=10448' 2024-04-06T09:42:38.200 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:42:38.200 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:42:38.200 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=17 2024-04-06T09:42:38.201 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=113, tag_tid=18, entry_tid=1' 2024-04-06T09:42:38.201 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:42:38.202 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:42:38.202 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:42:38.203 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=104, tag_tid=17, entry_tid=10448' 2024-04-06T09:42:38.203 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:42:38.203 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:42:38.204 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=10448 2024-04-06T09:42:38.204 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 18 = 17 -a 1 -le 10448 2024-04-06T09:42:38.204 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:42:38.205 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:42:38.205 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:42:38.406 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:42:38.407 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:42:38.407 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:42:38.407 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:42:38.407 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:42:38.407 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:42:38.407 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:42:38.407 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:42:38.407 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:42:38.407 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:42:38.407 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:42:38.407 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:42:38.408 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:42:38.408 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:42:38.408 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:42:38.408 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:42:38.408 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:42:38.412 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:42:38.412 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:42:38.412 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:42:38.412 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:42:38.412 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:42:38.412 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:42:38.412 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:42:38.412 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:42:38.412 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:42:38.412 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:42:38.412 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:42:38.412 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:42:38.412 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:42:38.413 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:42:38.413 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:42:38.413 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:42:38.413 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:42:38.413 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:42:38.413 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:42:38.413 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:42:38.413 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:42:38.413 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:42:38.413 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:42:38.413 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:42:38.413 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:42:38.413 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:42:38.413 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:42:38.414 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:42:38.414 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:42:38.414 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:42:38.414 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:42:38.414 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:42:38.437 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:42:38.437 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:42:38.437 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:42:38.437 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:42:38.581 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:42:38.582 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:42:38.582 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:42:38.582 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:42:38.583 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:42:38.583 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:42:38.583 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:42:38.583 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:42:38.583 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:42:38.583 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:42:38.583 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:42:38.584 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:42:38.584 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:42:38.586 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:42:38.586 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:42:38.586 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:42:38.637 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 26 2024-04-06T09:42:38.637 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 28 2024-04-06T09:42:38.637 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:42:38.637 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=113, tag_tid=18, entry_tid=1], [object_number=112, tag_tid=18, entry_tid=0], [object_number=114, tag_tid=17, entry_tid=20202], [object_number=115, tag_tid=17, entry_tid=20199]]], state=connected] 2024-04-06T09:42:38.638 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=107, tag_tid=17, entry_tid=10531], [object_number=106, tag_tid=17, entry_tid=10530], [object_number=105, tag_tid=17, entry_tid=10529], [object_number=104, tag_tid=17, entry_tid=10528]]], state=connected] 2024-04-06T09:42:38.641 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:42:38.642 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=113, tag_tid=18, entry_tid=1' 2024-04-06T09:42:38.643 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:42:38.643 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:42:38.643 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:42:38.644 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:42:38.644 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:42:38.644 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:42:38.644 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:42:38.644 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:42:38.644 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:42:38.644 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:42:38.644 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:42:38.644 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:42:38.645 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:42:38.645 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:42:38.645 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:42:38.691 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 26 2024-04-06T09:42:38.692 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 28 2024-04-06T09:42:38.692 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:42:38.692 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=113, tag_tid=18, entry_tid=1], [object_number=112, tag_tid=18, entry_tid=0], [object_number=114, tag_tid=17, entry_tid=20202], [object_number=115, tag_tid=17, entry_tid=20199]]], state=connected] 2024-04-06T09:42:38.692 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=107, tag_tid=17, entry_tid=10531], [object_number=106, tag_tid=17, entry_tid=10530], [object_number=105, tag_tid=17, entry_tid=10529], [object_number=104, tag_tid=17, entry_tid=10528]]], state=connected] 2024-04-06T09:42:38.695 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:42:38.697 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=107, tag_tid=17, entry_tid=10531' 2024-04-06T09:42:38.697 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=113, tag_tid=18, entry_tid=1' -a 'object_number=113, tag_tid=18, entry_tid=1' = 'object_number=107, tag_tid=17, entry_tid=10531' 2024-04-06T09:42:38.697 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=107, tag_tid=17, entry_tid=10531' '!=' 'object_number=104, tag_tid=17, entry_tid=10448' 2024-04-06T09:42:38.697 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:42:38.697 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=107, tag_tid=17, entry_tid=10531' = 'object_number=104, tag_tid=17, entry_tid=10448' 2024-04-06T09:42:38.697 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=107, tag_tid=17, entry_tid=10531' 2024-04-06T09:42:38.698 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=113, tag_tid=18, entry_tid=1' 2024-04-06T09:42:38.698 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:42:38.698 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:42:38.699 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=18 2024-04-06T09:42:38.699 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=107, tag_tid=17, entry_tid=10531' 2024-04-06T09:42:38.699 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:42:38.699 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:42:38.700 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=17 2024-04-06T09:42:38.701 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=113, tag_tid=18, entry_tid=1' 2024-04-06T09:42:38.701 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:42:38.701 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:42:38.703 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:42:38.703 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=107, tag_tid=17, entry_tid=10531' 2024-04-06T09:42:38.703 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:42:38.703 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:42:38.704 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=10531 2024-04-06T09:42:38.704 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 18 = 17 -a 1 -le 10531 2024-04-06T09:42:38.704 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:42:38.704 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:42:38.704 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:42:38.906 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:42:38.906 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:42:38.906 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:42:38.907 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:42:38.907 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:42:38.907 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:42:38.907 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:42:38.907 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:42:38.907 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:42:38.907 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:42:38.907 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:42:38.907 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:42:38.907 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:42:38.907 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:42:38.907 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:42:38.910 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:42:38.910 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:42:38.910 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:42:38.910 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:42:38.910 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:42:38.910 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:42:38.910 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:42:38.911 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:42:38.911 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:42:38.911 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:42:38.911 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:42:38.911 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:42:38.911 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:42:38.911 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:42:38.911 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:42:38.911 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:42:38.911 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:42:38.911 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:42:38.911 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:42:38.911 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:42:38.911 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:42:38.912 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:42:38.912 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:42:38.912 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:42:38.912 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:42:38.912 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:42:38.912 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:42:38.912 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:42:38.912 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:42:38.912 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:42:38.912 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:42:38.912 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:42:38.912 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:42:38.913 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:42:38.934 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:42:38.935 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:42:38.935 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:42:38.935 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:42:39.071 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:42:39.073 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:42:39.073 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:42:39.073 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:42:39.073 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:42:39.073 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:42:39.073 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:42:39.073 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:42:39.073 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:42:39.073 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:42:39.073 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:42:39.074 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:42:39.074 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:42:39.075 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:42:39.075 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:42:39.075 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:42:39.127 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 26 2024-04-06T09:42:39.127 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 28 2024-04-06T09:42:39.127 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:42:39.127 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=113, tag_tid=18, entry_tid=1], [object_number=112, tag_tid=18, entry_tid=0], [object_number=114, tag_tid=17, entry_tid=20202], [object_number=115, tag_tid=17, entry_tid=20199]]], state=connected] 2024-04-06T09:42:39.127 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=104, tag_tid=17, entry_tid=10616], [object_number=107, tag_tid=17, entry_tid=10615], [object_number=106, tag_tid=17, entry_tid=10614], [object_number=105, tag_tid=17, entry_tid=10613]]], state=connected] 2024-04-06T09:42:39.130 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:42:39.132 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=113, tag_tid=18, entry_tid=1' 2024-04-06T09:42:39.133 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:42:39.133 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:42:39.133 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:42:39.133 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:42:39.133 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:42:39.133 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:42:39.133 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:42:39.133 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:42:39.133 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:42:39.133 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:42:39.134 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:42:39.134 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:42:39.134 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:42:39.134 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:42:39.134 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:42:39.386 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 26 2024-04-06T09:42:39.386 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 28 2024-04-06T09:42:39.386 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:42:39.386 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=113, tag_tid=18, entry_tid=1], [object_number=112, tag_tid=18, entry_tid=0], [object_number=114, tag_tid=17, entry_tid=20202], [object_number=115, tag_tid=17, entry_tid=20199]]], state=connected] 2024-04-06T09:42:39.386 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=104, tag_tid=17, entry_tid=10616], [object_number=107, tag_tid=17, entry_tid=10615], [object_number=106, tag_tid=17, entry_tid=10614], [object_number=105, tag_tid=17, entry_tid=10613]]], state=connected] 2024-04-06T09:42:39.390 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:42:39.392 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=104, tag_tid=17, entry_tid=10616' 2024-04-06T09:42:39.392 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=113, tag_tid=18, entry_tid=1' -a 'object_number=113, tag_tid=18, entry_tid=1' = 'object_number=104, tag_tid=17, entry_tid=10616' 2024-04-06T09:42:39.392 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=104, tag_tid=17, entry_tid=10616' '!=' 'object_number=107, tag_tid=17, entry_tid=10531' 2024-04-06T09:42:39.392 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:42:39.393 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=104, tag_tid=17, entry_tid=10616' = 'object_number=107, tag_tid=17, entry_tid=10531' 2024-04-06T09:42:39.393 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=104, tag_tid=17, entry_tid=10616' 2024-04-06T09:42:39.393 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=113, tag_tid=18, entry_tid=1' 2024-04-06T09:42:39.393 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:42:39.393 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:42:39.394 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=18 2024-04-06T09:42:39.398 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=104, tag_tid=17, entry_tid=10616' 2024-04-06T09:42:39.399 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:42:39.399 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:42:39.399 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=17 2024-04-06T09:42:39.399 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=113, tag_tid=18, entry_tid=1' 2024-04-06T09:42:39.399 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:42:39.399 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:42:39.400 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:42:39.400 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=104, tag_tid=17, entry_tid=10616' 2024-04-06T09:42:39.400 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:42:39.400 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:42:39.402 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=10616 2024-04-06T09:42:39.402 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 18 = 17 -a 1 -le 10616 2024-04-06T09:42:39.402 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:42:39.402 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:42:39.402 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:42:39.604 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:42:39.604 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:42:39.604 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:42:39.604 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:42:39.604 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:42:39.604 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:42:39.605 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:42:39.605 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:42:39.605 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:42:39.605 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:42:39.605 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:42:39.605 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:42:39.605 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:42:39.605 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:42:39.605 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:42:39.605 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:42:39.606 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:42:39.608 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:42:39.608 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:42:39.608 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:42:39.608 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:42:39.608 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:42:39.608 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:42:39.608 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:42:39.609 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:42:39.609 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:42:39.609 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:42:39.609 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:42:39.609 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:42:39.609 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:42:39.609 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:42:39.609 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:42:39.609 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:42:39.611 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:42:39.611 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:42:39.611 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:42:39.611 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:42:39.611 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:42:39.611 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:42:39.611 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:42:39.611 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:42:39.611 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:42:39.611 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:42:39.611 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:42:39.612 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:42:39.612 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:42:39.612 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:42:39.612 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:42:39.612 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:42:39.636 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:42:39.636 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:42:39.636 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:42:39.637 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:42:39.769 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:42:39.770 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:42:39.770 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:42:39.771 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:42:39.771 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:42:39.771 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:42:39.771 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:42:39.771 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:42:39.771 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:42:39.771 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:42:39.771 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:42:39.771 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:42:39.772 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:42:39.773 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:42:39.774 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:42:39.774 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:42:39.821 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 26 2024-04-06T09:42:39.822 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 28 2024-04-06T09:42:39.822 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:42:39.822 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=113, tag_tid=18, entry_tid=1], [object_number=112, tag_tid=18, entry_tid=0], [object_number=114, tag_tid=17, entry_tid=20202], [object_number=115, tag_tid=17, entry_tid=20199]]], state=connected] 2024-04-06T09:42:39.822 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=104, tag_tid=17, entry_tid=10744], [object_number=107, tag_tid=17, entry_tid=10743], [object_number=106, tag_tid=17, entry_tid=10742], [object_number=105, tag_tid=17, entry_tid=10741]]], state=connected] 2024-04-06T09:42:39.825 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:42:39.826 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=113, tag_tid=18, entry_tid=1' 2024-04-06T09:42:39.827 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:42:39.827 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:42:39.828 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:42:39.828 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:42:39.828 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:42:39.828 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:42:39.828 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:42:39.828 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:42:39.828 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:42:39.828 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:42:39.828 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:42:39.828 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:42:39.829 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:42:39.829 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:42:39.829 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:42:39.875 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 26 2024-04-06T09:42:39.875 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 28 2024-04-06T09:42:39.876 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:42:39.876 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=113, tag_tid=18, entry_tid=1], [object_number=112, tag_tid=18, entry_tid=0], [object_number=114, tag_tid=17, entry_tid=20202], [object_number=115, tag_tid=17, entry_tid=20199]]], state=connected] 2024-04-06T09:42:39.876 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=104, tag_tid=17, entry_tid=10744], [object_number=107, tag_tid=17, entry_tid=10743], [object_number=106, tag_tid=17, entry_tid=10742], [object_number=105, tag_tid=17, entry_tid=10741]]], state=connected] 2024-04-06T09:42:39.879 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:42:39.880 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=104, tag_tid=17, entry_tid=10744' 2024-04-06T09:42:39.881 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=113, tag_tid=18, entry_tid=1' -a 'object_number=113, tag_tid=18, entry_tid=1' = 'object_number=104, tag_tid=17, entry_tid=10744' 2024-04-06T09:42:39.881 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=104, tag_tid=17, entry_tid=10744' '!=' 'object_number=104, tag_tid=17, entry_tid=10616' 2024-04-06T09:42:39.881 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:42:39.881 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=104, tag_tid=17, entry_tid=10744' = 'object_number=104, tag_tid=17, entry_tid=10616' 2024-04-06T09:42:39.881 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=104, tag_tid=17, entry_tid=10744' 2024-04-06T09:42:39.881 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=113, tag_tid=18, entry_tid=1' 2024-04-06T09:42:39.882 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:42:39.882 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:42:39.882 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=18 2024-04-06T09:42:39.883 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=104, tag_tid=17, entry_tid=10744' 2024-04-06T09:42:39.883 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:42:39.883 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:42:39.885 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=17 2024-04-06T09:42:39.885 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=113, tag_tid=18, entry_tid=1' 2024-04-06T09:42:39.885 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:42:39.885 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:42:39.886 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:42:39.887 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=104, tag_tid=17, entry_tid=10744' 2024-04-06T09:42:39.887 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:42:39.887 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:42:39.888 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=10744 2024-04-06T09:42:39.888 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 18 = 17 -a 1 -le 10744 2024-04-06T09:42:39.888 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:42:39.889 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:42:39.889 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:42:40.090 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:42:40.090 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:42:40.090 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:42:40.090 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:42:40.090 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:42:40.090 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:42:40.090 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:42:40.091 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:42:40.091 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:42:40.091 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:42:40.091 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:42:40.091 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:42:40.091 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:42:40.091 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:42:40.091 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:42:40.092 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:42:40.092 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:42:40.094 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:42:40.094 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:42:40.094 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:42:40.095 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:42:40.095 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:42:40.095 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:42:40.095 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:42:40.095 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:42:40.095 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:42:40.095 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:42:40.095 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:42:40.095 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:42:40.095 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:42:40.095 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:42:40.095 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:42:40.095 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:42:40.095 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:42:40.095 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:42:40.096 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:42:40.096 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:42:40.096 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:42:40.096 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:42:40.096 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:42:40.096 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:42:40.096 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:42:40.096 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:42:40.096 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:42:40.096 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:42:40.096 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:42:40.096 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:42:40.096 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:42:40.097 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:42:40.122 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:42:40.123 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:42:40.123 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:42:40.123 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:42:40.249 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:42:40.250 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:42:40.250 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:42:40.251 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:42:40.251 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:42:40.251 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:42:40.251 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:42:40.251 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:42:40.251 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:42:40.251 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:42:40.251 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:42:40.251 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:42:40.251 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:42:40.253 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:42:40.253 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:42:40.253 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:42:40.307 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 26 2024-04-06T09:42:40.307 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 28 2024-04-06T09:42:40.307 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:42:40.307 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=113, tag_tid=18, entry_tid=1], [object_number=112, tag_tid=18, entry_tid=0], [object_number=114, tag_tid=17, entry_tid=20202], [object_number=115, tag_tid=17, entry_tid=20199]]], state=connected] 2024-04-06T09:42:40.307 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=107, tag_tid=17, entry_tid=10823], [object_number=106, tag_tid=17, entry_tid=10822], [object_number=105, tag_tid=17, entry_tid=10821], [object_number=104, tag_tid=17, entry_tid=10820]]], state=connected] 2024-04-06T09:42:40.310 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:42:40.312 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=113, tag_tid=18, entry_tid=1' 2024-04-06T09:42:40.313 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:42:40.313 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:42:40.313 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:42:40.313 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:42:40.313 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:42:40.313 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:42:40.313 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:42:40.313 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:42:40.313 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:42:40.313 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:42:40.313 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:42:40.314 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:42:40.314 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:42:40.314 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:42:40.314 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:42:40.362 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 26 2024-04-06T09:42:40.362 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 28 2024-04-06T09:42:40.362 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:42:40.362 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=113, tag_tid=18, entry_tid=1], [object_number=112, tag_tid=18, entry_tid=0], [object_number=114, tag_tid=17, entry_tid=20202], [object_number=115, tag_tid=17, entry_tid=20199]]], state=connected] 2024-04-06T09:42:40.362 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=107, tag_tid=17, entry_tid=10823], [object_number=106, tag_tid=17, entry_tid=10822], [object_number=105, tag_tid=17, entry_tid=10821], [object_number=104, tag_tid=17, entry_tid=10820]]], state=connected] 2024-04-06T09:42:40.365 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:42:40.366 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=107, tag_tid=17, entry_tid=10823' 2024-04-06T09:42:40.367 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=113, tag_tid=18, entry_tid=1' -a 'object_number=113, tag_tid=18, entry_tid=1' = 'object_number=107, tag_tid=17, entry_tid=10823' 2024-04-06T09:42:40.367 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=107, tag_tid=17, entry_tid=10823' '!=' 'object_number=104, tag_tid=17, entry_tid=10744' 2024-04-06T09:42:40.367 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:42:40.367 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=107, tag_tid=17, entry_tid=10823' = 'object_number=104, tag_tid=17, entry_tid=10744' 2024-04-06T09:42:40.367 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=107, tag_tid=17, entry_tid=10823' 2024-04-06T09:42:40.367 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=113, tag_tid=18, entry_tid=1' 2024-04-06T09:42:40.368 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:42:40.368 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:42:40.368 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=18 2024-04-06T09:42:40.369 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=107, tag_tid=17, entry_tid=10823' 2024-04-06T09:42:40.369 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:42:40.369 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:42:40.370 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=17 2024-04-06T09:42:40.371 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=113, tag_tid=18, entry_tid=1' 2024-04-06T09:42:40.371 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:42:40.371 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:42:40.372 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:42:40.372 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=107, tag_tid=17, entry_tid=10823' 2024-04-06T09:42:40.372 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:42:40.373 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:42:40.374 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=10823 2024-04-06T09:42:40.374 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 18 = 17 -a 1 -le 10823 2024-04-06T09:42:40.374 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:42:40.374 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:42:40.374 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:42:40.575 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:42:40.576 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:42:40.576 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:42:40.576 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:42:40.576 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:42:40.576 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:42:40.577 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:42:40.577 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:42:40.577 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:42:40.577 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:42:40.577 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:42:40.577 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:42:40.577 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:42:40.577 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:42:40.577 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:42:40.578 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:42:40.578 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:42:40.580 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:42:40.580 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:42:40.580 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:42:40.580 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:42:40.580 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:42:40.580 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:42:40.580 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:42:40.580 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:42:40.580 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:42:40.581 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:42:40.581 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:42:40.581 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:42:40.581 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:42:40.581 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:42:40.581 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:42:40.581 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:42:40.581 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:42:40.581 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:42:40.581 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:42:40.581 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:42:40.581 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:42:40.581 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:42:40.581 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:42:40.582 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:42:40.582 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:42:40.582 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:42:40.582 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:42:40.582 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:42:40.582 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:42:40.582 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:42:40.582 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:42:40.582 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:42:40.608 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:42:40.608 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:42:40.608 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:42:40.609 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:42:40.741 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:42:40.743 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:42:40.743 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:42:40.743 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:42:40.743 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:42:40.743 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:42:40.743 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:42:40.743 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:42:40.743 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:42:40.743 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:42:40.743 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:42:40.744 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:42:40.744 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:42:40.746 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:42:40.746 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:42:40.747 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:42:40.794 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 26 2024-04-06T09:42:40.794 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 28 2024-04-06T09:42:40.794 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:42:40.794 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=113, tag_tid=18, entry_tid=1], [object_number=112, tag_tid=18, entry_tid=0], [object_number=114, tag_tid=17, entry_tid=20202], [object_number=115, tag_tid=17, entry_tid=20199]]], state=connected] 2024-04-06T09:42:40.794 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=104, tag_tid=17, entry_tid=10908], [object_number=107, tag_tid=17, entry_tid=10907], [object_number=106, tag_tid=17, entry_tid=10906], [object_number=105, tag_tid=17, entry_tid=10905]]], state=connected] 2024-04-06T09:42:40.797 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:42:40.798 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=113, tag_tid=18, entry_tid=1' 2024-04-06T09:42:40.799 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:42:40.799 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:42:40.800 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:42:40.800 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:42:40.800 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:42:40.800 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:42:40.800 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:42:40.800 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:42:40.800 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:42:40.800 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:42:40.800 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:42:40.800 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:42:40.801 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:42:40.801 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:42:40.801 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:42:40.849 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 26 2024-04-06T09:42:40.849 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 28 2024-04-06T09:42:40.849 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:42:40.849 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=113, tag_tid=18, entry_tid=1], [object_number=112, tag_tid=18, entry_tid=0], [object_number=114, tag_tid=17, entry_tid=20202], [object_number=115, tag_tid=17, entry_tid=20199]]], state=connected] 2024-04-06T09:42:40.849 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=104, tag_tid=17, entry_tid=10908], [object_number=107, tag_tid=17, entry_tid=10907], [object_number=106, tag_tid=17, entry_tid=10906], [object_number=105, tag_tid=17, entry_tid=10905]]], state=connected] 2024-04-06T09:42:40.852 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:42:40.854 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=104, tag_tid=17, entry_tid=10908' 2024-04-06T09:42:40.854 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=113, tag_tid=18, entry_tid=1' -a 'object_number=113, tag_tid=18, entry_tid=1' = 'object_number=104, tag_tid=17, entry_tid=10908' 2024-04-06T09:42:40.854 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=104, tag_tid=17, entry_tid=10908' '!=' 'object_number=107, tag_tid=17, entry_tid=10823' 2024-04-06T09:42:40.854 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:42:40.854 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=104, tag_tid=17, entry_tid=10908' = 'object_number=107, tag_tid=17, entry_tid=10823' 2024-04-06T09:42:40.854 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=104, tag_tid=17, entry_tid=10908' 2024-04-06T09:42:40.855 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=113, tag_tid=18, entry_tid=1' 2024-04-06T09:42:40.855 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:42:40.855 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:42:40.856 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=18 2024-04-06T09:42:40.856 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=104, tag_tid=17, entry_tid=10908' 2024-04-06T09:42:40.856 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:42:40.856 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:42:40.857 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=17 2024-04-06T09:42:40.858 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=113, tag_tid=18, entry_tid=1' 2024-04-06T09:42:40.859 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:42:40.859 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:42:40.859 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:42:40.860 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=104, tag_tid=17, entry_tid=10908' 2024-04-06T09:42:40.860 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:42:40.860 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:42:40.861 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=10908 2024-04-06T09:42:40.861 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 18 = 17 -a 1 -le 10908 2024-04-06T09:42:40.861 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:42:40.861 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:42:40.861 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:42:41.063 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:42:41.063 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:42:41.063 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:42:41.064 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:42:41.064 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:42:41.064 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:42:41.064 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:42:41.064 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:42:41.064 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:42:41.064 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:42:41.064 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:42:41.064 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:42:41.064 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:42:41.064 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:42:41.064 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:42:41.065 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:42:41.065 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:42:41.068 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:42:41.068 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:42:41.068 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:42:41.068 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:42:41.068 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:42:41.068 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:42:41.068 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:42:41.069 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:42:41.069 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:42:41.069 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:42:41.069 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:42:41.069 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:42:41.069 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:42:41.069 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:42:41.069 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:42:41.069 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:42:41.069 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:42:41.069 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:42:41.069 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:42:41.069 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:42:41.069 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:42:41.069 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:42:41.070 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:42:41.070 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:42:41.070 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:42:41.070 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:42:41.070 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:42:41.070 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:42:41.070 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:42:41.070 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:42:41.070 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:42:41.070 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:42:41.095 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:42:41.095 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:42:41.095 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:42:41.095 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:42:41.227 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:42:41.229 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:42:41.229 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:42:41.229 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:42:41.229 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:42:41.229 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:42:41.229 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:42:41.229 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:42:41.229 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:42:41.229 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:42:41.230 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:42:41.230 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:42:41.230 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:42:41.232 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:42:41.232 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:42:41.232 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:42:41.284 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 26 2024-04-06T09:42:41.284 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 28 2024-04-06T09:42:41.284 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:42:41.284 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=113, tag_tid=18, entry_tid=1], [object_number=112, tag_tid=18, entry_tid=0], [object_number=114, tag_tid=17, entry_tid=20202], [object_number=115, tag_tid=17, entry_tid=20199]]], state=connected] 2024-04-06T09:42:41.284 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=107, tag_tid=17, entry_tid=11007], [object_number=106, tag_tid=17, entry_tid=11006], [object_number=105, tag_tid=17, entry_tid=11005], [object_number=104, tag_tid=17, entry_tid=11004]]], state=connected] 2024-04-06T09:42:41.288 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:42:41.289 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=113, tag_tid=18, entry_tid=1' 2024-04-06T09:42:41.290 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:42:41.290 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:42:41.290 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:42:41.291 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:42:41.291 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:42:41.291 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:42:41.291 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:42:41.291 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:42:41.291 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:42:41.291 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:42:41.291 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:42:41.291 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:42:41.291 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:42:41.292 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:42:41.292 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:42:41.339 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 26 2024-04-06T09:42:41.339 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 28 2024-04-06T09:42:41.339 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:42:41.339 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=113, tag_tid=18, entry_tid=1], [object_number=112, tag_tid=18, entry_tid=0], [object_number=114, tag_tid=17, entry_tid=20202], [object_number=115, tag_tid=17, entry_tid=20199]]], state=connected] 2024-04-06T09:42:41.339 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=107, tag_tid=17, entry_tid=11007], [object_number=106, tag_tid=17, entry_tid=11006], [object_number=105, tag_tid=17, entry_tid=11005], [object_number=104, tag_tid=17, entry_tid=11004]]], state=connected] 2024-04-06T09:42:41.342 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:42:41.345 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=107, tag_tid=17, entry_tid=11007' 2024-04-06T09:42:41.345 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=113, tag_tid=18, entry_tid=1' -a 'object_number=113, tag_tid=18, entry_tid=1' = 'object_number=107, tag_tid=17, entry_tid=11007' 2024-04-06T09:42:41.346 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=107, tag_tid=17, entry_tid=11007' '!=' 'object_number=104, tag_tid=17, entry_tid=10908' 2024-04-06T09:42:41.346 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:42:41.346 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=107, tag_tid=17, entry_tid=11007' = 'object_number=104, tag_tid=17, entry_tid=10908' 2024-04-06T09:42:41.346 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=107, tag_tid=17, entry_tid=11007' 2024-04-06T09:42:41.346 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=113, tag_tid=18, entry_tid=1' 2024-04-06T09:42:41.346 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:42:41.346 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:42:41.347 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=18 2024-04-06T09:42:41.347 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=107, tag_tid=17, entry_tid=11007' 2024-04-06T09:42:41.347 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:42:41.347 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:42:41.348 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=17 2024-04-06T09:42:41.348 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=113, tag_tid=18, entry_tid=1' 2024-04-06T09:42:41.348 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:42:41.348 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:42:41.349 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:42:41.350 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=107, tag_tid=17, entry_tid=11007' 2024-04-06T09:42:41.350 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:42:41.350 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:42:41.351 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=11007 2024-04-06T09:42:41.351 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 18 = 17 -a 1 -le 11007 2024-04-06T09:42:41.351 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:42:41.352 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:42:41.352 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:42:41.553 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:42:41.553 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:42:41.553 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:42:41.553 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:42:41.553 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:42:41.554 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:42:41.554 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:42:41.554 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:42:41.554 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:42:41.554 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:42:41.554 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:42:41.554 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:42:41.554 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:42:41.554 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:42:41.554 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:42:41.555 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:42:41.555 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:42:41.558 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:42:41.558 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:42:41.558 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:42:41.558 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:42:41.558 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:42:41.558 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:42:41.558 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:42:41.558 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:42:41.558 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:42:41.558 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:42:41.559 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:42:41.559 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:42:41.559 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:42:41.559 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:42:41.559 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:42:41.559 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:42:41.559 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:42:41.559 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:42:41.559 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:42:41.559 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:42:41.559 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:42:41.559 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:42:41.559 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:42:41.559 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:42:41.560 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:42:41.560 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:42:41.560 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:42:41.560 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:42:41.560 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:42:41.560 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:42:41.560 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:42:41.560 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:42:41.583 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:42:41.583 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:42:41.583 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:42:41.583 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:42:41.716 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:42:41.717 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:42:41.718 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:42:41.718 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:42:41.718 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:42:41.718 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:42:41.718 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:42:41.718 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:42:41.718 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:42:41.718 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:42:41.719 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:42:41.719 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:42:41.719 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:42:41.721 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:42:41.722 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:42:41.722 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:42:41.774 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 26 2024-04-06T09:42:41.774 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 28 2024-04-06T09:42:41.775 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:42:41.775 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=113, tag_tid=18, entry_tid=1], [object_number=112, tag_tid=18, entry_tid=0], [object_number=114, tag_tid=17, entry_tid=20202], [object_number=115, tag_tid=17, entry_tid=20199]]], state=connected] 2024-04-06T09:42:41.775 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=105, tag_tid=17, entry_tid=11105], [object_number=104, tag_tid=17, entry_tid=11104], [object_number=107, tag_tid=17, entry_tid=11103], [object_number=106, tag_tid=17, entry_tid=11102]]], state=connected] 2024-04-06T09:42:41.778 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:42:41.779 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=113, tag_tid=18, entry_tid=1' 2024-04-06T09:42:41.780 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:42:41.780 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:42:41.781 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:42:41.781 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:42:41.781 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:42:41.781 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:42:41.781 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:42:41.781 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:42:41.781 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:42:41.781 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:42:41.781 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:42:41.781 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:42:41.782 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:42:41.782 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:42:41.782 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:42:41.828 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 26 2024-04-06T09:42:41.829 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 28 2024-04-06T09:42:41.829 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:42:41.829 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=113, tag_tid=18, entry_tid=1], [object_number=112, tag_tid=18, entry_tid=0], [object_number=114, tag_tid=17, entry_tid=20202], [object_number=115, tag_tid=17, entry_tid=20199]]], state=connected] 2024-04-06T09:42:41.829 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=105, tag_tid=17, entry_tid=11105], [object_number=104, tag_tid=17, entry_tid=11104], [object_number=107, tag_tid=17, entry_tid=11103], [object_number=106, tag_tid=17, entry_tid=11102]]], state=connected] 2024-04-06T09:42:41.832 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:42:41.834 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=105, tag_tid=17, entry_tid=11105' 2024-04-06T09:42:41.834 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=113, tag_tid=18, entry_tid=1' -a 'object_number=113, tag_tid=18, entry_tid=1' = 'object_number=105, tag_tid=17, entry_tid=11105' 2024-04-06T09:42:41.834 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=105, tag_tid=17, entry_tid=11105' '!=' 'object_number=107, tag_tid=17, entry_tid=11007' 2024-04-06T09:42:41.834 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:42:41.834 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=105, tag_tid=17, entry_tid=11105' = 'object_number=107, tag_tid=17, entry_tid=11007' 2024-04-06T09:42:41.835 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=105, tag_tid=17, entry_tid=11105' 2024-04-06T09:42:41.835 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=113, tag_tid=18, entry_tid=1' 2024-04-06T09:42:41.835 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:42:41.835 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:42:41.835 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=18 2024-04-06T09:42:41.836 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=105, tag_tid=17, entry_tid=11105' 2024-04-06T09:42:41.836 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:42:41.837 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:42:41.837 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=17 2024-04-06T09:42:41.838 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=113, tag_tid=18, entry_tid=1' 2024-04-06T09:42:41.838 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:42:41.838 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:42:41.839 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:42:41.839 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=105, tag_tid=17, entry_tid=11105' 2024-04-06T09:42:41.840 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:42:41.840 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:42:41.841 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=11105 2024-04-06T09:42:41.841 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 18 = 17 -a 1 -le 11105 2024-04-06T09:42:41.841 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:42:41.841 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:42:41.841 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:42:42.043 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:42:42.043 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:42:42.043 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:42:42.043 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:42:42.043 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:42:42.043 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:42:42.043 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:42:42.043 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:42:42.043 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:42:42.043 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:42:42.043 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:42:42.044 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:42:42.044 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:42:42.044 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:42:42.044 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:42:42.044 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:42:42.045 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:42:42.047 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:42:42.047 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:42:42.047 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:42:42.048 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:42:42.048 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:42:42.048 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:42:42.048 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:42:42.048 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:42:42.048 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:42:42.048 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:42:42.048 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:42:42.048 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:42:42.048 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:42:42.048 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:42:42.048 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:42:42.048 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:42:42.050 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:42:42.050 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:42:42.050 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:42:42.050 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:42:42.050 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:42:42.050 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:42:42.050 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:42:42.050 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:42:42.050 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:42:42.050 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:42:42.050 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:42:42.050 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:42:42.050 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:42:42.051 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:42:42.051 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:42:42.051 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:42:42.074 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:42:42.075 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:42:42.075 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:42:42.075 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:42:42.207 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:42:42.208 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:42:42.209 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:42:42.209 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:42:42.209 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:42:42.209 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:42:42.209 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:42:42.209 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:42:42.209 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:42:42.209 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:42:42.210 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:42:42.210 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:42:42.210 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:42:42.211 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:42:42.211 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:42:42.212 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:42:42.262 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 26 2024-04-06T09:42:42.262 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 28 2024-04-06T09:42:42.263 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:42:42.263 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=113, tag_tid=18, entry_tid=1], [object_number=112, tag_tid=18, entry_tid=0], [object_number=114, tag_tid=17, entry_tid=20202], [object_number=115, tag_tid=17, entry_tid=20199]]], state=connected] 2024-04-06T09:42:42.263 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=104, tag_tid=17, entry_tid=11188], [object_number=107, tag_tid=17, entry_tid=11187], [object_number=106, tag_tid=17, entry_tid=11186], [object_number=105, tag_tid=17, entry_tid=11185]]], state=connected] 2024-04-06T09:42:42.266 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:42:42.268 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=113, tag_tid=18, entry_tid=1' 2024-04-06T09:42:42.269 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:42:42.269 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:42:42.269 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:42:42.269 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:42:42.269 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:42:42.269 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:42:42.269 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:42:42.269 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:42:42.269 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:42:42.269 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:42:42.269 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:42:42.269 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:42:42.270 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:42:42.270 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:42:42.270 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:42:42.318 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 26 2024-04-06T09:42:42.318 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 28 2024-04-06T09:42:42.318 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:42:42.318 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=113, tag_tid=18, entry_tid=1], [object_number=112, tag_tid=18, entry_tid=0], [object_number=114, tag_tid=17, entry_tid=20202], [object_number=115, tag_tid=17, entry_tid=20199]]], state=connected] 2024-04-06T09:42:42.318 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=104, tag_tid=17, entry_tid=11188], [object_number=107, tag_tid=17, entry_tid=11187], [object_number=106, tag_tid=17, entry_tid=11186], [object_number=105, tag_tid=17, entry_tid=11185]]], state=connected] 2024-04-06T09:42:42.322 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:42:42.324 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=104, tag_tid=17, entry_tid=11188' 2024-04-06T09:42:42.324 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=113, tag_tid=18, entry_tid=1' -a 'object_number=113, tag_tid=18, entry_tid=1' = 'object_number=104, tag_tid=17, entry_tid=11188' 2024-04-06T09:42:42.324 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=104, tag_tid=17, entry_tid=11188' '!=' 'object_number=105, tag_tid=17, entry_tid=11105' 2024-04-06T09:42:42.324 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:42:42.324 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=104, tag_tid=17, entry_tid=11188' = 'object_number=105, tag_tid=17, entry_tid=11105' 2024-04-06T09:42:42.324 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=104, tag_tid=17, entry_tid=11188' 2024-04-06T09:42:42.325 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=113, tag_tid=18, entry_tid=1' 2024-04-06T09:42:42.325 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:42:42.325 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:42:42.325 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=18 2024-04-06T09:42:42.326 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=104, tag_tid=17, entry_tid=11188' 2024-04-06T09:42:42.326 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:42:42.326 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:42:42.327 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=17 2024-04-06T09:42:42.328 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=113, tag_tid=18, entry_tid=1' 2024-04-06T09:42:42.328 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:42:42.328 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:42:42.329 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:42:42.329 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=104, tag_tid=17, entry_tid=11188' 2024-04-06T09:42:42.330 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:42:42.330 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:42:42.331 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=11188 2024-04-06T09:42:42.331 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 18 = 17 -a 1 -le 11188 2024-04-06T09:42:42.331 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:42:42.331 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:42:42.331 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:42:42.533 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:42:42.533 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:42:42.533 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:42:42.533 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:42:42.533 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:42:42.533 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:42:42.533 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:42:42.533 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:42:42.533 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:42:42.533 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:42:42.534 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:42:42.534 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:42:42.534 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:42:42.534 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:42:42.534 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:42:42.534 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:42:42.535 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:42:42.537 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:42:42.537 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:42:42.537 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:42:42.537 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:42:42.537 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:42:42.538 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:42:42.538 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:42:42.538 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:42:42.538 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:42:42.538 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:42:42.538 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:42:42.538 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:42:42.538 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:42:42.538 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:42:42.538 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:42:42.538 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:42:42.540 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:42:42.540 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:42:42.540 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:42:42.540 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:42:42.540 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:42:42.540 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:42:42.540 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:42:42.540 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:42:42.540 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:42:42.540 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:42:42.540 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:42:42.540 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:42:42.540 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:42:42.541 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:42:42.541 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:42:42.541 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:42:42.564 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:42:42.564 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:42:42.564 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:42:42.564 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:42:42.693 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:42:42.694 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:42:42.694 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:42:42.694 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:42:42.694 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:42:42.695 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:42:42.695 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:42:42.695 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:42:42.695 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:42:42.695 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:42:42.695 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:42:42.695 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:42:42.695 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:42:42.697 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:42:42.698 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:42:42.698 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:42:42.744 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 26 2024-04-06T09:42:42.744 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 28 2024-04-06T09:42:42.744 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:42:42.744 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=113, tag_tid=18, entry_tid=1], [object_number=112, tag_tid=18, entry_tid=0], [object_number=114, tag_tid=17, entry_tid=20202], [object_number=115, tag_tid=17, entry_tid=20199]]], state=connected] 2024-04-06T09:42:42.744 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=104, tag_tid=17, entry_tid=11268], [object_number=107, tag_tid=17, entry_tid=11267], [object_number=106, tag_tid=17, entry_tid=11266], [object_number=105, tag_tid=17, entry_tid=11265]]], state=connected] 2024-04-06T09:42:42.747 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:42:42.748 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=113, tag_tid=18, entry_tid=1' 2024-04-06T09:42:42.749 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:42:42.750 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:42:42.750 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:42:42.750 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:42:42.750 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:42:42.750 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:42:42.750 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:42:42.750 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:42:42.750 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:42:42.750 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:42:42.750 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:42:42.750 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:42:42.751 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:42:42.751 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:42:42.751 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:42:42.797 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 26 2024-04-06T09:42:42.797 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 28 2024-04-06T09:42:42.797 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:42:42.797 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=113, tag_tid=18, entry_tid=1], [object_number=112, tag_tid=18, entry_tid=0], [object_number=114, tag_tid=17, entry_tid=20202], [object_number=115, tag_tid=17, entry_tid=20199]]], state=connected] 2024-04-06T09:42:42.797 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=104, tag_tid=17, entry_tid=11268], [object_number=107, tag_tid=17, entry_tid=11267], [object_number=106, tag_tid=17, entry_tid=11266], [object_number=105, tag_tid=17, entry_tid=11265]]], state=connected] 2024-04-06T09:42:42.800 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:42:42.803 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=104, tag_tid=17, entry_tid=11268' 2024-04-06T09:42:42.803 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=113, tag_tid=18, entry_tid=1' -a 'object_number=113, tag_tid=18, entry_tid=1' = 'object_number=104, tag_tid=17, entry_tid=11268' 2024-04-06T09:42:42.803 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=104, tag_tid=17, entry_tid=11268' '!=' 'object_number=104, tag_tid=17, entry_tid=11188' 2024-04-06T09:42:42.803 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:42:42.803 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=104, tag_tid=17, entry_tid=11268' = 'object_number=104, tag_tid=17, entry_tid=11188' 2024-04-06T09:42:42.803 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=104, tag_tid=17, entry_tid=11268' 2024-04-06T09:42:42.803 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=113, tag_tid=18, entry_tid=1' 2024-04-06T09:42:42.803 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:42:42.803 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:42:42.804 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=18 2024-04-06T09:42:42.804 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=104, tag_tid=17, entry_tid=11268' 2024-04-06T09:42:42.805 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:42:42.805 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:42:42.805 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=17 2024-04-06T09:42:42.806 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=113, tag_tid=18, entry_tid=1' 2024-04-06T09:42:42.806 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:42:42.806 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:42:42.807 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:42:42.808 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=104, tag_tid=17, entry_tid=11268' 2024-04-06T09:42:42.808 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:42:42.808 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:42:42.809 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=11268 2024-04-06T09:42:42.809 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 18 = 17 -a 1 -le 11268 2024-04-06T09:42:42.809 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:42:42.809 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:42:42.809 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:42:43.011 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:42:43.011 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:42:43.011 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:42:43.011 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:42:43.011 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:42:43.011 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:42:43.011 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:42:43.011 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:42:43.011 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:42:43.012 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:42:43.012 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:42:43.012 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:42:43.012 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:42:43.012 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:42:43.012 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:42:43.012 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:42:43.013 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:42:43.015 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:42:43.015 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:42:43.015 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:42:43.015 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:42:43.015 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:42:43.016 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:42:43.016 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:42:43.016 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:42:43.016 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:42:43.016 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:42:43.016 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:42:43.016 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:42:43.016 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:42:43.016 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:42:43.016 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:42:43.016 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:42:43.017 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:42:43.017 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:42:43.017 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:42:43.017 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:42:43.017 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:42:43.017 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:42:43.017 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:42:43.018 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:42:43.018 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:42:43.018 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:42:43.018 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:42:43.018 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:42:43.018 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:42:43.018 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:42:43.018 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:42:43.018 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:42:43.043 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:42:43.043 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:42:43.043 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:42:43.043 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:42:43.174 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:42:43.174 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:42:43.174 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:42:43.174 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:42:43.174 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:42:43.175 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:42:43.175 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:42:43.175 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:42:43.175 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:42:43.175 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:42:43.175 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:42:43.175 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:42:43.176 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:42:43.178 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:42:43.178 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:42:43.178 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:42:43.235 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 26 2024-04-06T09:42:43.235 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 28 2024-04-06T09:42:43.235 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:42:43.235 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=113, tag_tid=18, entry_tid=1], [object_number=112, tag_tid=18, entry_tid=0], [object_number=114, tag_tid=17, entry_tid=20202], [object_number=115, tag_tid=17, entry_tid=20199]]], state=connected] 2024-04-06T09:42:43.236 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=105, tag_tid=17, entry_tid=11357], [object_number=104, tag_tid=17, entry_tid=11356], [object_number=107, tag_tid=17, entry_tid=11355], [object_number=106, tag_tid=17, entry_tid=11354]]], state=connected] 2024-04-06T09:42:43.239 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:42:43.240 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=113, tag_tid=18, entry_tid=1' 2024-04-06T09:42:43.241 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:42:43.241 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:42:43.242 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:42:43.242 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:42:43.242 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:42:43.242 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:42:43.242 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:42:43.242 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:42:43.242 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:42:43.242 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:42:43.242 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:42:43.242 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:42:43.243 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:42:43.243 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:42:43.243 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:42:43.290 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 26 2024-04-06T09:42:43.290 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 28 2024-04-06T09:42:43.290 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:42:43.291 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=113, tag_tid=18, entry_tid=1], [object_number=112, tag_tid=18, entry_tid=0], [object_number=114, tag_tid=17, entry_tid=20202], [object_number=115, tag_tid=17, entry_tid=20199]]], state=connected] 2024-04-06T09:42:43.291 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=105, tag_tid=17, entry_tid=11357], [object_number=104, tag_tid=17, entry_tid=11356], [object_number=107, tag_tid=17, entry_tid=11355], [object_number=106, tag_tid=17, entry_tid=11354]]], state=connected] 2024-04-06T09:42:43.294 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:42:43.296 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=105, tag_tid=17, entry_tid=11357' 2024-04-06T09:42:43.296 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=113, tag_tid=18, entry_tid=1' -a 'object_number=113, tag_tid=18, entry_tid=1' = 'object_number=105, tag_tid=17, entry_tid=11357' 2024-04-06T09:42:43.296 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=105, tag_tid=17, entry_tid=11357' '!=' 'object_number=104, tag_tid=17, entry_tid=11268' 2024-04-06T09:42:43.296 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:42:43.296 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=105, tag_tid=17, entry_tid=11357' = 'object_number=104, tag_tid=17, entry_tid=11268' 2024-04-06T09:42:43.296 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=105, tag_tid=17, entry_tid=11357' 2024-04-06T09:42:43.296 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=113, tag_tid=18, entry_tid=1' 2024-04-06T09:42:43.296 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:42:43.296 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:42:43.297 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=18 2024-04-06T09:42:43.298 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=105, tag_tid=17, entry_tid=11357' 2024-04-06T09:42:43.298 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:42:43.298 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:42:43.299 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=17 2024-04-06T09:42:43.299 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=113, tag_tid=18, entry_tid=1' 2024-04-06T09:42:43.300 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:42:43.300 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:42:43.300 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:42:43.301 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=105, tag_tid=17, entry_tid=11357' 2024-04-06T09:42:43.301 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:42:43.301 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:42:43.303 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=11357 2024-04-06T09:42:43.303 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 18 = 17 -a 1 -le 11357 2024-04-06T09:42:43.303 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:42:43.303 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:42:43.303 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:42:43.504 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:42:43.504 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:42:43.504 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:42:43.505 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:42:43.505 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:42:43.505 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:42:43.505 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:42:43.505 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:42:43.505 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:42:43.505 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:42:43.505 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:42:43.505 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:42:43.505 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:42:43.505 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:42:43.505 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:42:43.506 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:42:43.506 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:42:43.509 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:42:43.509 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:42:43.509 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:42:43.509 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:42:43.509 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:42:43.509 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:42:43.509 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:42:43.509 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:42:43.509 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:42:43.509 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:42:43.510 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:42:43.510 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:42:43.510 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:42:43.510 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:42:43.510 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:42:43.510 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:42:43.510 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:42:43.510 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:42:43.510 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:42:43.510 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:42:43.510 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:42:43.510 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:42:43.510 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:42:43.510 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:42:43.511 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:42:43.511 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:42:43.511 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:42:43.511 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:42:43.511 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:42:43.511 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:42:43.511 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:42:43.511 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:42:43.535 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:42:43.535 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:42:43.535 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:42:43.536 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:42:43.674 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:42:43.675 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:42:43.675 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:42:43.675 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:42:43.675 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:42:43.676 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:42:43.676 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:42:43.676 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:42:43.676 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:42:43.676 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:42:43.676 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:42:43.676 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:42:43.677 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:42:43.678 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:42:43.679 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:42:43.679 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:42:43.730 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 26 2024-04-06T09:42:43.730 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 28 2024-04-06T09:42:43.730 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:42:43.730 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=113, tag_tid=18, entry_tid=1], [object_number=112, tag_tid=18, entry_tid=0], [object_number=114, tag_tid=17, entry_tid=20202], [object_number=115, tag_tid=17, entry_tid=20199]]], state=connected] 2024-04-06T09:42:43.730 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=106, tag_tid=17, entry_tid=11450], [object_number=105, tag_tid=17, entry_tid=11449], [object_number=104, tag_tid=17, entry_tid=11448], [object_number=107, tag_tid=17, entry_tid=11447]]], state=connected] 2024-04-06T09:42:43.734 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:42:43.736 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=113, tag_tid=18, entry_tid=1' 2024-04-06T09:42:43.737 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:42:43.737 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:42:43.737 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:42:43.737 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:42:43.737 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:42:43.737 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:42:43.737 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:42:43.737 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:42:43.737 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:42:43.738 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:42:43.738 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:42:43.738 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:42:43.738 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:42:43.739 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:42:43.739 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:42:43.785 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 26 2024-04-06T09:42:43.785 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 28 2024-04-06T09:42:43.785 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:42:43.785 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=113, tag_tid=18, entry_tid=1], [object_number=112, tag_tid=18, entry_tid=0], [object_number=114, tag_tid=17, entry_tid=20202], [object_number=115, tag_tid=17, entry_tid=20199]]], state=connected] 2024-04-06T09:42:43.785 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=106, tag_tid=17, entry_tid=11450], [object_number=105, tag_tid=17, entry_tid=11449], [object_number=104, tag_tid=17, entry_tid=11448], [object_number=107, tag_tid=17, entry_tid=11447]]], state=connected] 2024-04-06T09:42:43.788 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:42:43.790 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=106, tag_tid=17, entry_tid=11450' 2024-04-06T09:42:43.790 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=113, tag_tid=18, entry_tid=1' -a 'object_number=113, tag_tid=18, entry_tid=1' = 'object_number=106, tag_tid=17, entry_tid=11450' 2024-04-06T09:42:43.790 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=106, tag_tid=17, entry_tid=11450' '!=' 'object_number=105, tag_tid=17, entry_tid=11357' 2024-04-06T09:42:43.790 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:42:43.791 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=106, tag_tid=17, entry_tid=11450' = 'object_number=105, tag_tid=17, entry_tid=11357' 2024-04-06T09:42:43.791 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=106, tag_tid=17, entry_tid=11450' 2024-04-06T09:42:43.791 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=113, tag_tid=18, entry_tid=1' 2024-04-06T09:42:43.791 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:42:43.791 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:42:43.792 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=18 2024-04-06T09:42:43.793 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=106, tag_tid=17, entry_tid=11450' 2024-04-06T09:42:43.793 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:42:43.793 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:42:43.793 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=17 2024-04-06T09:42:43.794 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=113, tag_tid=18, entry_tid=1' 2024-04-06T09:42:43.794 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:42:43.794 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:42:43.795 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:42:43.796 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=106, tag_tid=17, entry_tid=11450' 2024-04-06T09:42:43.796 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:42:43.796 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:42:43.797 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=11450 2024-04-06T09:42:43.797 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 18 = 17 -a 1 -le 11450 2024-04-06T09:42:43.797 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:42:43.797 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:42:43.797 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:42:43.999 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:42:43.999 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:42:44.000 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:42:44.000 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:42:44.000 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:42:44.000 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:42:44.000 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:42:44.000 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:42:44.000 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:42:44.000 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:42:44.000 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:42:44.000 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:42:44.000 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:42:44.000 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:42:44.000 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:42:44.000 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:42:44.005 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:42:44.005 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:42:44.005 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:42:44.005 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:42:44.005 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:42:44.005 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:42:44.005 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:42:44.005 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:42:44.006 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:42:44.006 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:42:44.006 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:42:44.006 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:42:44.006 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:42:44.006 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:42:44.006 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:42:44.006 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:42:44.006 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:42:44.006 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:42:44.006 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:42:44.006 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:42:44.006 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:42:44.006 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:42:44.007 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:42:44.007 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:42:44.007 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:42:44.007 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:42:44.007 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:42:44.007 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:42:44.007 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:42:44.007 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:42:44.007 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:42:44.007 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:42:44.007 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:42:44.029 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:42:44.029 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:42:44.030 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:42:44.030 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:42:44.162 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:42:44.164 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:42:44.164 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:42:44.164 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:42:44.164 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:42:44.164 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:42:44.164 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:42:44.164 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:42:44.164 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:42:44.164 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:42:44.165 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:42:44.165 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:42:44.165 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:42:44.166 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:42:44.166 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:42:44.167 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:42:44.214 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 26 2024-04-06T09:42:44.214 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 28 2024-04-06T09:42:44.214 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:42:44.215 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=113, tag_tid=18, entry_tid=1], [object_number=112, tag_tid=18, entry_tid=0], [object_number=114, tag_tid=17, entry_tid=20202], [object_number=115, tag_tid=17, entry_tid=20199]]], state=connected] 2024-04-06T09:42:44.215 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=105, tag_tid=17, entry_tid=11545], [object_number=104, tag_tid=17, entry_tid=11544], [object_number=107, tag_tid=17, entry_tid=11543], [object_number=106, tag_tid=17, entry_tid=11542]]], state=connected] 2024-04-06T09:42:44.218 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:42:44.219 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=113, tag_tid=18, entry_tid=1' 2024-04-06T09:42:44.220 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:42:44.220 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:42:44.220 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:42:44.220 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:42:44.220 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:42:44.220 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:42:44.220 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:42:44.220 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:42:44.221 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:42:44.221 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:42:44.221 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:42:44.221 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:42:44.221 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:42:44.221 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:42:44.222 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:42:44.269 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 26 2024-04-06T09:42:44.269 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 28 2024-04-06T09:42:44.269 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:42:44.269 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=113, tag_tid=18, entry_tid=1], [object_number=112, tag_tid=18, entry_tid=0], [object_number=114, tag_tid=17, entry_tid=20202], [object_number=115, tag_tid=17, entry_tid=20199]]], state=connected] 2024-04-06T09:42:44.269 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=105, tag_tid=17, entry_tid=11545], [object_number=104, tag_tid=17, entry_tid=11544], [object_number=107, tag_tid=17, entry_tid=11543], [object_number=106, tag_tid=17, entry_tid=11542]]], state=connected] 2024-04-06T09:42:44.272 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:42:44.275 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=105, tag_tid=17, entry_tid=11545' 2024-04-06T09:42:44.275 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=113, tag_tid=18, entry_tid=1' -a 'object_number=113, tag_tid=18, entry_tid=1' = 'object_number=105, tag_tid=17, entry_tid=11545' 2024-04-06T09:42:44.275 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=105, tag_tid=17, entry_tid=11545' '!=' 'object_number=106, tag_tid=17, entry_tid=11450' 2024-04-06T09:42:44.275 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:42:44.275 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=105, tag_tid=17, entry_tid=11545' = 'object_number=106, tag_tid=17, entry_tid=11450' 2024-04-06T09:42:44.275 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=105, tag_tid=17, entry_tid=11545' 2024-04-06T09:42:44.275 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=113, tag_tid=18, entry_tid=1' 2024-04-06T09:42:44.275 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:42:44.275 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:42:44.276 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=18 2024-04-06T09:42:44.276 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=105, tag_tid=17, entry_tid=11545' 2024-04-06T09:42:44.277 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:42:44.277 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:42:44.277 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=17 2024-04-06T09:42:44.278 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=113, tag_tid=18, entry_tid=1' 2024-04-06T09:42:44.278 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:42:44.278 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:42:44.279 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:42:44.280 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=105, tag_tid=17, entry_tid=11545' 2024-04-06T09:42:44.280 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:42:44.280 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:42:44.281 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=11545 2024-04-06T09:42:44.281 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 18 = 17 -a 1 -le 11545 2024-04-06T09:42:44.281 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:42:44.281 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:42:44.282 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:42:44.483 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:42:44.483 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:42:44.483 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:42:44.483 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:42:44.483 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:42:44.483 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:42:44.484 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:42:44.484 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:42:44.484 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:42:44.484 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:42:44.484 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:42:44.484 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:42:44.484 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:42:44.484 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:42:44.484 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:42:44.484 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:42:44.485 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:42:44.487 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:42:44.488 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:42:44.488 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:42:44.488 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:42:44.488 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:42:44.488 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:42:44.488 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:42:44.488 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:42:44.488 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:42:44.488 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:42:44.488 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:42:44.488 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:42:44.488 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:42:44.488 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:42:44.488 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:42:44.489 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:42:44.489 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:42:44.489 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:42:44.489 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:42:44.489 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:42:44.489 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:42:44.489 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:42:44.489 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:42:44.489 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:42:44.489 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:42:44.490 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:42:44.490 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:42:44.490 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:42:44.490 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:42:44.490 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:42:44.490 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:42:44.490 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:42:44.514 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:42:44.515 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:42:44.515 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:42:44.515 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:42:44.649 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:42:44.650 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:42:44.650 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:42:44.650 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:42:44.650 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:42:44.651 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:42:44.651 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:42:44.651 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:42:44.651 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:42:44.651 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:42:44.651 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:42:44.651 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:42:44.651 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:42:44.651 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:42:44.652 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:42:44.652 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:42:44.699 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 26 2024-04-06T09:42:44.699 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 28 2024-04-06T09:42:44.699 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:42:44.699 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=113, tag_tid=18, entry_tid=1], [object_number=112, tag_tid=18, entry_tid=0], [object_number=114, tag_tid=17, entry_tid=20202], [object_number=115, tag_tid=17, entry_tid=20199]]], state=connected] 2024-04-06T09:42:44.699 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=104, tag_tid=17, entry_tid=11628], [object_number=107, tag_tid=17, entry_tid=11627], [object_number=106, tag_tid=17, entry_tid=11626], [object_number=105, tag_tid=17, entry_tid=11625]]], state=connected] 2024-04-06T09:42:44.703 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:42:44.704 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=113, tag_tid=18, entry_tid=1' 2024-04-06T09:42:44.705 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:42:44.705 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:42:44.705 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:42:44.705 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:42:44.705 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:42:44.706 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:42:44.706 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:42:44.706 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:42:44.706 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:42:44.706 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:42:44.706 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:42:44.706 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:42:44.707 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:42:44.707 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:42:44.707 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:42:44.753 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 26 2024-04-06T09:42:44.754 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 28 2024-04-06T09:42:44.754 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:42:44.754 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=113, tag_tid=18, entry_tid=1], [object_number=112, tag_tid=18, entry_tid=0], [object_number=114, tag_tid=17, entry_tid=20202], [object_number=115, tag_tid=17, entry_tid=20199]]], state=connected] 2024-04-06T09:42:44.754 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=104, tag_tid=17, entry_tid=11628], [object_number=107, tag_tid=17, entry_tid=11627], [object_number=106, tag_tid=17, entry_tid=11626], [object_number=105, tag_tid=17, entry_tid=11625]]], state=connected] 2024-04-06T09:42:44.757 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:42:44.759 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=104, tag_tid=17, entry_tid=11628' 2024-04-06T09:42:44.759 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=113, tag_tid=18, entry_tid=1' -a 'object_number=113, tag_tid=18, entry_tid=1' = 'object_number=104, tag_tid=17, entry_tid=11628' 2024-04-06T09:42:44.759 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=104, tag_tid=17, entry_tid=11628' '!=' 'object_number=105, tag_tid=17, entry_tid=11545' 2024-04-06T09:42:44.759 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:42:44.759 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=104, tag_tid=17, entry_tid=11628' = 'object_number=105, tag_tid=17, entry_tid=11545' 2024-04-06T09:42:44.759 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=104, tag_tid=17, entry_tid=11628' 2024-04-06T09:42:44.760 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=113, tag_tid=18, entry_tid=1' 2024-04-06T09:42:44.760 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:42:44.760 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:42:44.760 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=18 2024-04-06T09:42:44.761 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=104, tag_tid=17, entry_tid=11628' 2024-04-06T09:42:44.761 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:42:44.761 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:42:44.762 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=17 2024-04-06T09:42:44.763 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=113, tag_tid=18, entry_tid=1' 2024-04-06T09:42:44.763 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:42:44.763 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:42:44.764 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:42:44.765 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=104, tag_tid=17, entry_tid=11628' 2024-04-06T09:42:44.765 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:42:44.765 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:42:44.766 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=11628 2024-04-06T09:42:44.766 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 18 = 17 -a 1 -le 11628 2024-04-06T09:42:44.766 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:42:44.766 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:42:44.766 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:42:44.968 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:42:44.968 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:42:44.968 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:42:44.968 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:42:44.968 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:42:44.968 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:42:44.968 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:42:44.968 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:42:44.969 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:42:44.969 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:42:44.969 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:42:44.969 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:42:44.969 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:42:44.969 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:42:44.969 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:42:44.969 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:42:44.970 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:42:44.972 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:42:44.972 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:42:44.972 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:42:44.972 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:42:44.972 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:42:44.972 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:42:44.972 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:42:44.972 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:42:44.972 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:42:44.972 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:42:44.972 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:42:44.972 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:42:44.972 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:42:44.972 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:42:44.973 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:42:44.973 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:42:44.973 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:42:44.973 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:42:44.974 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:42:44.974 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:42:44.974 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:42:44.974 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:42:44.974 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:42:44.974 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:42:44.974 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:42:44.974 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:42:44.974 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:42:44.974 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:42:44.974 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:42:44.974 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:42:44.974 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:42:44.975 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:42:44.999 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:42:44.999 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:42:44.999 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:42:44.999 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:42:45.126 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:42:45.127 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:42:45.127 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:42:45.127 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:42:45.127 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:42:45.127 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:42:45.127 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:42:45.127 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:42:45.127 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:42:45.128 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:42:45.128 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:42:45.128 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:42:45.128 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:42:45.130 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:42:45.131 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:42:45.131 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:42:45.182 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 26 2024-04-06T09:42:45.182 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 28 2024-04-06T09:42:45.182 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:42:45.182 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=113, tag_tid=18, entry_tid=1], [object_number=112, tag_tid=18, entry_tid=0], [object_number=114, tag_tid=17, entry_tid=20202], [object_number=115, tag_tid=17, entry_tid=20199]]], state=connected] 2024-04-06T09:42:45.182 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=104, tag_tid=17, entry_tid=11728], [object_number=107, tag_tid=17, entry_tid=11727], [object_number=106, tag_tid=17, entry_tid=11726], [object_number=105, tag_tid=17, entry_tid=11725]]], state=connected] 2024-04-06T09:42:45.186 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:42:45.187 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=113, tag_tid=18, entry_tid=1' 2024-04-06T09:42:45.188 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:42:45.188 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:42:45.188 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:42:45.188 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:42:45.188 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:42:45.188 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:42:45.189 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:42:45.189 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:42:45.189 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:42:45.189 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:42:45.189 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:42:45.189 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:42:45.189 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:42:45.190 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:42:45.190 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:42:45.238 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 26 2024-04-06T09:42:45.239 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 28 2024-04-06T09:42:45.239 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:42:45.239 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=113, tag_tid=18, entry_tid=1], [object_number=112, tag_tid=18, entry_tid=0], [object_number=114, tag_tid=17, entry_tid=20202], [object_number=115, tag_tid=17, entry_tid=20199]]], state=connected] 2024-04-06T09:42:45.239 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=104, tag_tid=17, entry_tid=11728], [object_number=107, tag_tid=17, entry_tid=11727], [object_number=106, tag_tid=17, entry_tid=11726], [object_number=105, tag_tid=17, entry_tid=11725]]], state=connected] 2024-04-06T09:42:45.242 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:42:45.245 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=104, tag_tid=17, entry_tid=11728' 2024-04-06T09:42:45.245 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=113, tag_tid=18, entry_tid=1' -a 'object_number=113, tag_tid=18, entry_tid=1' = 'object_number=104, tag_tid=17, entry_tid=11728' 2024-04-06T09:42:45.245 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=104, tag_tid=17, entry_tid=11728' '!=' 'object_number=104, tag_tid=17, entry_tid=11628' 2024-04-06T09:42:45.245 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:42:45.245 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=104, tag_tid=17, entry_tid=11728' = 'object_number=104, tag_tid=17, entry_tid=11628' 2024-04-06T09:42:45.245 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=104, tag_tid=17, entry_tid=11728' 2024-04-06T09:42:45.245 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=113, tag_tid=18, entry_tid=1' 2024-04-06T09:42:45.245 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:42:45.245 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:42:45.246 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=18 2024-04-06T09:42:45.246 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=104, tag_tid=17, entry_tid=11728' 2024-04-06T09:42:45.247 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:42:45.247 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:42:45.247 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=17 2024-04-06T09:42:45.248 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=113, tag_tid=18, entry_tid=1' 2024-04-06T09:42:45.248 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:42:45.248 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:42:45.249 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:42:45.250 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=104, tag_tid=17, entry_tid=11728' 2024-04-06T09:42:45.250 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:42:45.250 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:42:45.251 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=11728 2024-04-06T09:42:45.251 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 18 = 17 -a 1 -le 11728 2024-04-06T09:42:45.251 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:42:45.251 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:42:45.251 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:42:45.453 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:42:45.453 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:42:45.453 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:42:45.453 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:42:45.453 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:42:45.454 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:42:45.454 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:42:45.454 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:42:45.454 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:42:45.454 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:42:45.454 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:42:45.454 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:42:45.454 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:42:45.454 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:42:45.454 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:42:45.455 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:42:45.455 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:42:45.458 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:42:45.458 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:42:45.458 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:42:45.458 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:42:45.458 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:42:45.458 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:42:45.459 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:42:45.459 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:42:45.459 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:42:45.459 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:42:45.459 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:42:45.459 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:42:45.459 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:42:45.459 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:42:45.459 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:42:45.459 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:42:45.459 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:42:45.459 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:42:45.459 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:42:45.459 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:42:45.460 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:42:45.460 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:42:45.460 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:42:45.460 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:42:45.460 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:42:45.460 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:42:45.460 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:42:45.460 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:42:45.460 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:42:45.460 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:42:45.460 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:42:45.460 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:42:45.483 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:42:45.483 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:42:45.483 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:42:45.483 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:42:45.612 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:42:45.613 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:42:45.613 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:42:45.613 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:42:45.613 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:42:45.613 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:42:45.613 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:42:45.613 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:42:45.613 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:42:45.613 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:42:45.614 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:42:45.614 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:42:45.614 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:42:45.617 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:42:45.617 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:42:45.617 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:42:45.663 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 26 2024-04-06T09:42:45.663 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 28 2024-04-06T09:42:45.663 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:42:45.663 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=113, tag_tid=18, entry_tid=1], [object_number=112, tag_tid=18, entry_tid=0], [object_number=114, tag_tid=17, entry_tid=20202], [object_number=115, tag_tid=17, entry_tid=20199]]], state=connected] 2024-04-06T09:42:45.663 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=107, tag_tid=17, entry_tid=11815], [object_number=106, tag_tid=17, entry_tid=11814], [object_number=105, tag_tid=17, entry_tid=11813], [object_number=104, tag_tid=17, entry_tid=11812]]], state=connected] 2024-04-06T09:42:45.667 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:42:45.668 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=113, tag_tid=18, entry_tid=1' 2024-04-06T09:42:45.669 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:42:45.669 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:42:45.670 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:42:45.670 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:42:45.670 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:42:45.670 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:42:45.670 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:42:45.670 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:42:45.670 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:42:45.670 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:42:45.670 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:42:45.670 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:42:45.671 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:42:45.671 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:42:45.671 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:42:45.717 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 26 2024-04-06T09:42:45.717 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 28 2024-04-06T09:42:45.717 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:42:45.717 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=113, tag_tid=18, entry_tid=1], [object_number=112, tag_tid=18, entry_tid=0], [object_number=114, tag_tid=17, entry_tid=20202], [object_number=115, tag_tid=17, entry_tid=20199]]], state=connected] 2024-04-06T09:42:45.718 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=107, tag_tid=17, entry_tid=11815], [object_number=106, tag_tid=17, entry_tid=11814], [object_number=105, tag_tid=17, entry_tid=11813], [object_number=104, tag_tid=17, entry_tid=11812]]], state=connected] 2024-04-06T09:42:45.721 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:42:45.723 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=107, tag_tid=17, entry_tid=11815' 2024-04-06T09:42:45.723 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=113, tag_tid=18, entry_tid=1' -a 'object_number=113, tag_tid=18, entry_tid=1' = 'object_number=107, tag_tid=17, entry_tid=11815' 2024-04-06T09:42:45.723 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=107, tag_tid=17, entry_tid=11815' '!=' 'object_number=104, tag_tid=17, entry_tid=11728' 2024-04-06T09:42:45.723 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:42:45.723 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=107, tag_tid=17, entry_tid=11815' = 'object_number=104, tag_tid=17, entry_tid=11728' 2024-04-06T09:42:45.723 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=107, tag_tid=17, entry_tid=11815' 2024-04-06T09:42:45.723 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=113, tag_tid=18, entry_tid=1' 2024-04-06T09:42:45.723 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:42:45.723 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:42:45.724 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=18 2024-04-06T09:42:45.725 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=107, tag_tid=17, entry_tid=11815' 2024-04-06T09:42:45.725 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:42:45.725 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:42:45.726 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=17 2024-04-06T09:42:45.726 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=113, tag_tid=18, entry_tid=1' 2024-04-06T09:42:45.726 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:42:45.726 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:42:45.727 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:42:45.728 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=107, tag_tid=17, entry_tid=11815' 2024-04-06T09:42:45.728 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:42:45.728 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:42:45.729 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=11815 2024-04-06T09:42:45.729 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 18 = 17 -a 1 -le 11815 2024-04-06T09:42:45.729 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:42:45.730 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:42:45.730 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:42:45.931 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:42:45.932 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:42:45.932 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:42:45.932 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:42:45.932 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:42:45.932 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:42:45.932 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:42:45.932 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:42:45.932 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:42:45.932 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:42:45.932 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:42:45.932 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:42:45.932 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:42:45.932 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:42:45.932 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:42:45.933 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:42:45.933 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:42:45.935 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:42:45.936 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:42:45.936 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:42:45.936 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:42:45.936 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:42:45.936 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:42:45.936 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:42:45.936 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:42:45.936 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:42:45.936 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:42:45.936 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:42:45.936 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:42:45.936 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:42:45.936 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:42:45.937 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:42:45.937 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:42:45.937 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:42:45.937 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:42:45.937 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:42:45.937 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:42:45.937 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:42:45.937 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:42:45.937 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:42:45.937 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:42:45.937 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:42:45.937 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:42:45.938 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:42:45.938 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:42:45.938 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:42:45.938 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:42:45.938 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:42:45.938 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:42:45.962 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:42:45.962 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:42:45.962 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:42:45.962 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:42:46.093 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:42:46.094 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:42:46.094 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:42:46.094 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:42:46.094 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:42:46.094 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:42:46.094 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:42:46.094 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:42:46.094 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:42:46.094 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:42:46.095 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:42:46.095 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:42:46.095 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:42:46.096 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:42:46.097 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:42:46.097 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:42:46.148 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 26 2024-04-06T09:42:46.149 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 28 2024-04-06T09:42:46.149 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:42:46.149 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=113, tag_tid=18, entry_tid=1], [object_number=112, tag_tid=18, entry_tid=0], [object_number=114, tag_tid=17, entry_tid=20202], [object_number=115, tag_tid=17, entry_tid=20199]]], state=connected] 2024-04-06T09:42:46.149 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=104, tag_tid=17, entry_tid=11896], [object_number=107, tag_tid=17, entry_tid=11895], [object_number=106, tag_tid=17, entry_tid=11894], [object_number=105, tag_tid=17, entry_tid=11893]]], state=connected] 2024-04-06T09:42:46.152 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:42:46.154 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=113, tag_tid=18, entry_tid=1' 2024-04-06T09:42:46.155 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:42:46.155 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:42:46.155 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:42:46.155 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:42:46.155 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:42:46.155 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:42:46.155 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:42:46.155 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:42:46.155 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:42:46.155 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:42:46.156 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:42:46.156 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:42:46.156 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:42:46.156 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:42:46.156 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:42:46.203 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 26 2024-04-06T09:42:46.203 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 28 2024-04-06T09:42:46.203 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:42:46.203 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=113, tag_tid=18, entry_tid=1], [object_number=112, tag_tid=18, entry_tid=0], [object_number=114, tag_tid=17, entry_tid=20202], [object_number=115, tag_tid=17, entry_tid=20199]]], state=connected] 2024-04-06T09:42:46.203 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=104, tag_tid=17, entry_tid=11896], [object_number=107, tag_tid=17, entry_tid=11895], [object_number=106, tag_tid=17, entry_tid=11894], [object_number=105, tag_tid=17, entry_tid=11893]]], state=connected] 2024-04-06T09:42:46.206 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:42:46.208 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=104, tag_tid=17, entry_tid=11896' 2024-04-06T09:42:46.208 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=113, tag_tid=18, entry_tid=1' -a 'object_number=113, tag_tid=18, entry_tid=1' = 'object_number=104, tag_tid=17, entry_tid=11896' 2024-04-06T09:42:46.208 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=104, tag_tid=17, entry_tid=11896' '!=' 'object_number=107, tag_tid=17, entry_tid=11815' 2024-04-06T09:42:46.208 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:42:46.208 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=104, tag_tid=17, entry_tid=11896' = 'object_number=107, tag_tid=17, entry_tid=11815' 2024-04-06T09:42:46.209 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=104, tag_tid=17, entry_tid=11896' 2024-04-06T09:42:46.209 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=113, tag_tid=18, entry_tid=1' 2024-04-06T09:42:46.209 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:42:46.209 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:42:46.209 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=18 2024-04-06T09:42:46.210 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=104, tag_tid=17, entry_tid=11896' 2024-04-06T09:42:46.210 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:42:46.210 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:42:46.211 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=17 2024-04-06T09:42:46.212 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=113, tag_tid=18, entry_tid=1' 2024-04-06T09:42:46.212 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:42:46.212 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:42:46.213 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:42:46.213 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=104, tag_tid=17, entry_tid=11896' 2024-04-06T09:42:46.213 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:42:46.213 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:42:46.215 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=11896 2024-04-06T09:42:46.215 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 18 = 17 -a 1 -le 11896 2024-04-06T09:42:46.215 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:42:46.215 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:42:46.215 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:42:46.417 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:42:46.417 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:42:46.417 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:42:46.418 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:42:46.418 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:42:46.418 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:42:46.418 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:42:46.418 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:42:46.418 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:42:46.418 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:42:46.418 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:42:46.418 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:42:46.418 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:42:46.418 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:42:46.418 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:42:46.421 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:42:46.421 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:42:46.421 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:42:46.421 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:42:46.421 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:42:46.421 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:42:46.421 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:42:46.421 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:42:46.422 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:42:46.422 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:42:46.422 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:42:46.422 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:42:46.422 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:42:46.422 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:42:46.422 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:42:46.422 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:42:46.422 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:42:46.422 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:42:46.422 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:42:46.422 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:42:46.422 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:42:46.422 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:42:46.423 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:42:46.423 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:42:46.423 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:42:46.423 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:42:46.423 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:42:46.423 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:42:46.423 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:42:46.423 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:42:46.423 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:42:46.423 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:42:46.423 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:42:46.423 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:42:46.446 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:42:46.446 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:42:46.446 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:42:46.446 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:42:46.575 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:42:46.577 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:42:46.577 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:42:46.577 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:42:46.577 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:42:46.577 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:42:46.577 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:42:46.577 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:42:46.577 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:42:46.577 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:42:46.578 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:42:46.578 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:42:46.578 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:42:46.581 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:42:46.581 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:42:46.581 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:42:46.634 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 26 2024-04-06T09:42:46.634 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 28 2024-04-06T09:42:46.635 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:42:46.635 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=113, tag_tid=18, entry_tid=1], [object_number=112, tag_tid=18, entry_tid=0], [object_number=114, tag_tid=17, entry_tid=20202], [object_number=115, tag_tid=17, entry_tid=20199]]], state=connected] 2024-04-06T09:42:46.635 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=107, tag_tid=17, entry_tid=11979], [object_number=106, tag_tid=17, entry_tid=11978], [object_number=105, tag_tid=17, entry_tid=11977], [object_number=104, tag_tid=17, entry_tid=11976]]], state=connected] 2024-04-06T09:42:46.638 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:42:46.641 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=113, tag_tid=18, entry_tid=1' 2024-04-06T09:42:46.641 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:42:46.641 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:42:46.641 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:42:46.641 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:42:46.641 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:42:46.641 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:42:46.642 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:42:46.642 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:42:46.642 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:42:46.642 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:42:46.642 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:42:46.642 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:42:46.642 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:42:46.643 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:42:46.643 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:42:46.688 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 26 2024-04-06T09:42:46.689 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 28 2024-04-06T09:42:46.689 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:42:46.689 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=113, tag_tid=18, entry_tid=1], [object_number=112, tag_tid=18, entry_tid=0], [object_number=114, tag_tid=17, entry_tid=20202], [object_number=115, tag_tid=17, entry_tid=20199]]], state=connected] 2024-04-06T09:42:46.689 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=107, tag_tid=17, entry_tid=11979], [object_number=106, tag_tid=17, entry_tid=11978], [object_number=105, tag_tid=17, entry_tid=11977], [object_number=104, tag_tid=17, entry_tid=11976]]], state=connected] 2024-04-06T09:42:46.692 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:42:46.693 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=107, tag_tid=17, entry_tid=11979' 2024-04-06T09:42:46.694 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=113, tag_tid=18, entry_tid=1' -a 'object_number=113, tag_tid=18, entry_tid=1' = 'object_number=107, tag_tid=17, entry_tid=11979' 2024-04-06T09:42:46.694 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=107, tag_tid=17, entry_tid=11979' '!=' 'object_number=104, tag_tid=17, entry_tid=11896' 2024-04-06T09:42:46.694 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:42:46.694 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=107, tag_tid=17, entry_tid=11979' = 'object_number=104, tag_tid=17, entry_tid=11896' 2024-04-06T09:42:46.694 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=107, tag_tid=17, entry_tid=11979' 2024-04-06T09:42:46.695 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=113, tag_tid=18, entry_tid=1' 2024-04-06T09:42:46.695 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:42:46.695 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:42:46.695 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=18 2024-04-06T09:42:46.696 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=107, tag_tid=17, entry_tid=11979' 2024-04-06T09:42:46.696 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:42:46.696 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:42:46.697 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=17 2024-04-06T09:42:46.697 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=113, tag_tid=18, entry_tid=1' 2024-04-06T09:42:46.697 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:42:46.698 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:42:46.698 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:42:46.699 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=107, tag_tid=17, entry_tid=11979' 2024-04-06T09:42:46.699 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:42:46.699 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:42:46.700 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=11979 2024-04-06T09:42:46.701 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 18 = 17 -a 1 -le 11979 2024-04-06T09:42:46.701 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:42:46.701 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:42:46.701 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:42:46.902 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:42:46.902 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:42:46.903 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:42:46.903 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:42:46.903 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:42:46.903 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:42:46.903 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:42:46.903 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:42:46.903 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:42:46.903 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:42:46.903 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:42:46.903 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:42:46.903 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:42:46.903 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:42:46.903 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:42:46.904 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:42:46.904 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:42:46.907 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:42:46.907 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:42:46.907 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:42:46.907 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:42:46.907 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:42:46.907 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:42:46.907 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:42:46.907 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:42:46.907 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:42:46.907 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:42:46.907 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:42:46.907 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:42:46.908 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:42:46.908 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:42:46.908 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:42:46.908 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:42:46.908 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:42:46.908 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:42:46.908 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:42:46.908 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:42:46.908 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:42:46.909 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:42:46.909 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:42:46.909 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:42:46.909 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:42:46.909 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:42:46.909 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:42:46.909 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:42:46.909 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:42:46.909 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:42:46.909 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:42:46.909 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:42:46.932 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:42:46.932 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:42:46.932 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:42:46.932 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:42:47.062 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:42:47.063 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:42:47.063 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:42:47.063 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:42:47.063 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:42:47.063 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:42:47.063 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:42:47.063 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:42:47.063 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:42:47.064 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:42:47.064 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:42:47.064 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:42:47.064 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:42:47.065 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:42:47.066 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:42:47.066 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:42:47.122 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 26 2024-04-06T09:42:47.145 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 28 2024-04-06T09:42:47.146 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:42:47.146 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=113, tag_tid=18, entry_tid=1], [object_number=112, tag_tid=18, entry_tid=0], [object_number=114, tag_tid=17, entry_tid=20202], [object_number=115, tag_tid=17, entry_tid=20199]]], state=connected] 2024-04-06T09:42:47.146 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=105, tag_tid=17, entry_tid=12065], [object_number=104, tag_tid=17, entry_tid=12064], [object_number=107, tag_tid=17, entry_tid=12063], [object_number=106, tag_tid=17, entry_tid=12062]]], state=connected] 2024-04-06T09:42:47.147 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:42:47.147 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=113, tag_tid=18, entry_tid=1' 2024-04-06T09:42:47.148 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:42:47.148 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:42:47.148 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:42:47.148 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:42:47.148 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:42:47.148 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:42:47.148 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:42:47.148 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:42:47.148 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:42:47.148 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:42:47.148 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:42:47.148 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:42:47.148 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:42:47.149 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:42:47.149 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:42:47.177 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 26 2024-04-06T09:42:47.177 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 28 2024-04-06T09:42:47.177 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:42:47.177 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=113, tag_tid=18, entry_tid=1], [object_number=112, tag_tid=18, entry_tid=0], [object_number=114, tag_tid=17, entry_tid=20202], [object_number=115, tag_tid=17, entry_tid=20199]]], state=connected] 2024-04-06T09:42:47.177 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=105, tag_tid=17, entry_tid=12065], [object_number=104, tag_tid=17, entry_tid=12064], [object_number=107, tag_tid=17, entry_tid=12063], [object_number=106, tag_tid=17, entry_tid=12062]]], state=connected] 2024-04-06T09:42:47.181 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:42:47.182 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=105, tag_tid=17, entry_tid=12065' 2024-04-06T09:42:47.183 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=113, tag_tid=18, entry_tid=1' -a 'object_number=113, tag_tid=18, entry_tid=1' = 'object_number=105, tag_tid=17, entry_tid=12065' 2024-04-06T09:42:47.183 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=105, tag_tid=17, entry_tid=12065' '!=' 'object_number=107, tag_tid=17, entry_tid=11979' 2024-04-06T09:42:47.183 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:42:47.183 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=105, tag_tid=17, entry_tid=12065' = 'object_number=107, tag_tid=17, entry_tid=11979' 2024-04-06T09:42:47.183 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=105, tag_tid=17, entry_tid=12065' 2024-04-06T09:42:47.184 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=113, tag_tid=18, entry_tid=1' 2024-04-06T09:42:47.184 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:42:47.184 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:42:47.184 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=18 2024-04-06T09:42:47.185 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=105, tag_tid=17, entry_tid=12065' 2024-04-06T09:42:47.185 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:42:47.185 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:42:47.186 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=17 2024-04-06T09:42:47.187 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=113, tag_tid=18, entry_tid=1' 2024-04-06T09:42:47.187 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:42:47.187 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:42:47.187 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:42:47.188 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=105, tag_tid=17, entry_tid=12065' 2024-04-06T09:42:47.188 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:42:47.188 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:42:47.190 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=12065 2024-04-06T09:42:47.190 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 18 = 17 -a 1 -le 12065 2024-04-06T09:42:47.190 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:42:47.190 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:42:47.190 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:42:47.391 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:42:47.392 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:42:47.392 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:42:47.392 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:42:47.392 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:42:47.392 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:42:47.392 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:42:47.392 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:42:47.392 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:42:47.392 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:42:47.392 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:42:47.392 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:42:47.392 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:42:47.392 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:42:47.392 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:42:47.393 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:42:47.393 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:42:47.395 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:42:47.396 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:42:47.396 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:42:47.396 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:42:47.396 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:42:47.396 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:42:47.396 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:42:47.396 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:42:47.396 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:42:47.396 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:42:47.396 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:42:47.396 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:42:47.396 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:42:47.397 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:42:47.397 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:42:47.397 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:42:47.397 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:42:47.397 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:42:47.397 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:42:47.397 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:42:47.397 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:42:47.397 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:42:47.397 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:42:47.397 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:42:47.397 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:42:47.397 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:42:47.397 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:42:47.398 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:42:47.398 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:42:47.398 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:42:47.398 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:42:47.398 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:42:47.422 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:42:47.422 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:42:47.422 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:42:47.422 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:42:47.556 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:42:47.557 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:42:47.557 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:42:47.558 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:42:47.558 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:42:47.558 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:42:47.558 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:42:47.558 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:42:47.558 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:42:47.558 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:42:47.559 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:42:47.559 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:42:47.559 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:42:47.561 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:42:47.561 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:42:47.561 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:42:47.612 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 26 2024-04-06T09:42:47.613 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 28 2024-04-06T09:42:47.613 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:42:47.613 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=113, tag_tid=18, entry_tid=1], [object_number=112, tag_tid=18, entry_tid=0], [object_number=114, tag_tid=17, entry_tid=20202], [object_number=115, tag_tid=17, entry_tid=20199]]], state=connected] 2024-04-06T09:42:47.613 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=106, tag_tid=17, entry_tid=12150], [object_number=105, tag_tid=17, entry_tid=12149], [object_number=104, tag_tid=17, entry_tid=12148], [object_number=107, tag_tid=17, entry_tid=12147]]], state=connected] 2024-04-06T09:42:47.616 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:42:47.617 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=113, tag_tid=18, entry_tid=1' 2024-04-06T09:42:47.618 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:42:47.618 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:42:47.618 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:42:47.618 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:42:47.618 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:42:47.619 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:42:47.619 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:42:47.619 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:42:47.619 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:42:47.619 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:42:47.619 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:42:47.619 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:42:47.619 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:42:47.620 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:42:47.620 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:42:47.667 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 26 2024-04-06T09:42:47.668 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 28 2024-04-06T09:42:47.668 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:42:47.668 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=113, tag_tid=18, entry_tid=1], [object_number=112, tag_tid=18, entry_tid=0], [object_number=114, tag_tid=17, entry_tid=20202], [object_number=115, tag_tid=17, entry_tid=20199]]], state=connected] 2024-04-06T09:42:47.668 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=106, tag_tid=17, entry_tid=12150], [object_number=105, tag_tid=17, entry_tid=12149], [object_number=104, tag_tid=17, entry_tid=12148], [object_number=107, tag_tid=17, entry_tid=12147]]], state=connected] 2024-04-06T09:42:47.671 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:42:47.673 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=106, tag_tid=17, entry_tid=12150' 2024-04-06T09:42:47.673 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=113, tag_tid=18, entry_tid=1' -a 'object_number=113, tag_tid=18, entry_tid=1' = 'object_number=106, tag_tid=17, entry_tid=12150' 2024-04-06T09:42:47.674 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=106, tag_tid=17, entry_tid=12150' '!=' 'object_number=105, tag_tid=17, entry_tid=12065' 2024-04-06T09:42:47.674 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:42:47.674 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=106, tag_tid=17, entry_tid=12150' = 'object_number=105, tag_tid=17, entry_tid=12065' 2024-04-06T09:42:47.674 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=106, tag_tid=17, entry_tid=12150' 2024-04-06T09:42:47.674 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=113, tag_tid=18, entry_tid=1' 2024-04-06T09:42:47.674 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:42:47.674 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:42:47.675 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=18 2024-04-06T09:42:47.675 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=106, tag_tid=17, entry_tid=12150' 2024-04-06T09:42:47.675 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:42:47.675 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:42:47.676 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=17 2024-04-06T09:42:47.677 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=113, tag_tid=18, entry_tid=1' 2024-04-06T09:42:47.677 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:42:47.677 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:42:47.678 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:42:47.679 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=106, tag_tid=17, entry_tid=12150' 2024-04-06T09:42:47.679 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:42:47.679 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:42:47.680 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=12150 2024-04-06T09:42:47.680 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 18 = 17 -a 1 -le 12150 2024-04-06T09:42:47.680 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:42:47.680 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:42:47.680 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:42:47.882 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:42:47.882 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:42:47.882 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:42:47.882 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:42:47.882 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:42:47.882 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:42:47.882 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:42:47.883 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:42:47.883 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:42:47.883 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:42:47.883 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:42:47.883 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:42:47.883 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:42:47.883 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:42:47.883 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:42:47.884 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:42:47.884 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:42:47.886 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:42:47.886 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:42:47.886 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:42:47.886 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:42:47.886 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:42:47.886 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:42:47.886 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:42:47.887 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:42:47.887 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:42:47.887 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:42:47.887 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:42:47.887 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:42:47.887 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:42:47.887 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:42:47.887 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:42:47.887 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:42:47.887 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:42:47.887 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:42:47.887 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:42:47.887 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:42:47.887 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:42:47.887 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:42:47.888 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:42:47.888 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:42:47.888 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:42:47.888 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:42:47.888 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:42:47.888 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:42:47.888 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:42:47.888 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:42:47.888 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:42:47.889 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:42:47.912 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:42:47.912 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:42:47.912 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:42:47.912 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:42:48.046 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:42:48.047 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:42:48.047 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:42:48.047 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:42:48.047 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:42:48.048 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:42:48.048 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:42:48.048 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:42:48.048 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:42:48.048 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:42:48.048 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:42:48.048 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:42:48.049 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:42:48.050 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:42:48.051 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:42:48.051 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:42:48.097 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 26 2024-04-06T09:42:48.098 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 28 2024-04-06T09:42:48.098 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:42:48.098 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=113, tag_tid=18, entry_tid=1], [object_number=112, tag_tid=18, entry_tid=0], [object_number=114, tag_tid=17, entry_tid=20202], [object_number=115, tag_tid=17, entry_tid=20199]]], state=connected] 2024-04-06T09:42:48.098 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=104, tag_tid=17, entry_tid=12236], [object_number=107, tag_tid=17, entry_tid=12235], [object_number=106, tag_tid=17, entry_tid=12234], [object_number=105, tag_tid=17, entry_tid=12233]]], state=connected] 2024-04-06T09:42:48.101 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:42:48.103 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=113, tag_tid=18, entry_tid=1' 2024-04-06T09:42:48.104 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:42:48.104 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:42:48.104 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:42:48.104 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:42:48.104 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:42:48.104 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:42:48.104 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:42:48.105 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:42:48.105 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:42:48.105 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:42:48.105 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:42:48.105 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:42:48.105 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:42:48.106 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:42:48.106 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:42:48.152 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 26 2024-04-06T09:42:48.152 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 28 2024-04-06T09:42:48.152 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:42:48.152 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=113, tag_tid=18, entry_tid=1], [object_number=112, tag_tid=18, entry_tid=0], [object_number=114, tag_tid=17, entry_tid=20202], [object_number=115, tag_tid=17, entry_tid=20199]]], state=connected] 2024-04-06T09:42:48.152 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=104, tag_tid=17, entry_tid=12236], [object_number=107, tag_tid=17, entry_tid=12235], [object_number=106, tag_tid=17, entry_tid=12234], [object_number=105, tag_tid=17, entry_tid=12233]]], state=connected] 2024-04-06T09:42:48.155 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:42:48.158 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=104, tag_tid=17, entry_tid=12236' 2024-04-06T09:42:48.158 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=113, tag_tid=18, entry_tid=1' -a 'object_number=113, tag_tid=18, entry_tid=1' = 'object_number=104, tag_tid=17, entry_tid=12236' 2024-04-06T09:42:48.158 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=104, tag_tid=17, entry_tid=12236' '!=' 'object_number=106, tag_tid=17, entry_tid=12150' 2024-04-06T09:42:48.158 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:42:48.158 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=104, tag_tid=17, entry_tid=12236' = 'object_number=106, tag_tid=17, entry_tid=12150' 2024-04-06T09:42:48.158 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=104, tag_tid=17, entry_tid=12236' 2024-04-06T09:42:48.158 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=113, tag_tid=18, entry_tid=1' 2024-04-06T09:42:48.158 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:42:48.158 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:42:48.159 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=18 2024-04-06T09:42:48.159 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=104, tag_tid=17, entry_tid=12236' 2024-04-06T09:42:48.159 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:42:48.159 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:42:48.160 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=17 2024-04-06T09:42:48.161 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=113, tag_tid=18, entry_tid=1' 2024-04-06T09:42:48.161 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:42:48.161 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:42:48.162 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:42:48.163 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=104, tag_tid=17, entry_tid=12236' 2024-04-06T09:42:48.163 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:42:48.163 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:42:48.164 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=12236 2024-04-06T09:42:48.164 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 18 = 17 -a 1 -le 12236 2024-04-06T09:42:48.164 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:42:48.164 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:42:48.164 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:42:48.366 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:42:48.366 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:42:48.366 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:42:48.366 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:42:48.366 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:42:48.366 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:42:48.366 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:42:48.366 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:42:48.366 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:42:48.367 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:42:48.367 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:42:48.367 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:42:48.367 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:42:48.367 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:42:48.367 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:42:48.367 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:42:48.368 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:42:48.370 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:42:48.370 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:42:48.370 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:42:48.371 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:42:48.371 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:42:48.371 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:42:48.371 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:42:48.371 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:42:48.371 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:42:48.371 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:42:48.371 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:42:48.371 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:42:48.371 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:42:48.371 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:42:48.371 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:42:48.371 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:42:48.371 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:42:48.372 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:42:48.372 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:42:48.372 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:42:48.372 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:42:48.372 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:42:48.372 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:42:48.372 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:42:48.372 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:42:48.372 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:42:48.372 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:42:48.372 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:42:48.372 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:42:48.372 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:42:48.372 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:42:48.373 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:42:48.397 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:42:48.397 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:42:48.397 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:42:48.397 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:42:48.532 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:42:48.533 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:42:48.534 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:42:48.534 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:42:48.534 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:42:48.534 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:42:48.534 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:42:48.534 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:42:48.534 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:42:48.534 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:42:48.535 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:42:48.535 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:42:48.535 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:42:48.536 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:42:48.537 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:42:48.537 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:42:48.595 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 26 2024-04-06T09:42:48.596 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 28 2024-04-06T09:42:48.596 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:42:48.596 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=113, tag_tid=18, entry_tid=1], [object_number=112, tag_tid=18, entry_tid=0], [object_number=114, tag_tid=17, entry_tid=20202], [object_number=115, tag_tid=17, entry_tid=20199]]], state=connected] 2024-04-06T09:42:48.596 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=105, tag_tid=17, entry_tid=12317], [object_number=104, tag_tid=17, entry_tid=12316], [object_number=107, tag_tid=17, entry_tid=12315], [object_number=106, tag_tid=17, entry_tid=12314]]], state=connected] 2024-04-06T09:42:48.599 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:42:48.601 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=113, tag_tid=18, entry_tid=1' 2024-04-06T09:42:48.602 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:42:48.602 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:42:48.602 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:42:48.602 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:42:48.602 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:42:48.602 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:42:48.602 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:42:48.602 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:42:48.602 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:42:48.602 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:42:48.602 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:42:48.603 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:42:48.603 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:42:48.603 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:42:48.603 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:42:48.650 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 26 2024-04-06T09:42:48.651 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 28 2024-04-06T09:42:48.651 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:42:48.651 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=113, tag_tid=18, entry_tid=1], [object_number=112, tag_tid=18, entry_tid=0], [object_number=114, tag_tid=17, entry_tid=20202], [object_number=115, tag_tid=17, entry_tid=20199]]], state=connected] 2024-04-06T09:42:48.651 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=105, tag_tid=17, entry_tid=12317], [object_number=104, tag_tid=17, entry_tid=12316], [object_number=107, tag_tid=17, entry_tid=12315], [object_number=106, tag_tid=17, entry_tid=12314]]], state=connected] 2024-04-06T09:42:48.654 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:42:48.657 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=105, tag_tid=17, entry_tid=12317' 2024-04-06T09:42:48.657 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=113, tag_tid=18, entry_tid=1' -a 'object_number=113, tag_tid=18, entry_tid=1' = 'object_number=105, tag_tid=17, entry_tid=12317' 2024-04-06T09:42:48.657 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=105, tag_tid=17, entry_tid=12317' '!=' 'object_number=104, tag_tid=17, entry_tid=12236' 2024-04-06T09:42:48.657 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:42:48.657 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=105, tag_tid=17, entry_tid=12317' = 'object_number=104, tag_tid=17, entry_tid=12236' 2024-04-06T09:42:48.658 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=105, tag_tid=17, entry_tid=12317' 2024-04-06T09:42:48.658 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=113, tag_tid=18, entry_tid=1' 2024-04-06T09:42:48.658 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:42:48.658 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:42:48.659 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=18 2024-04-06T09:42:48.659 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=105, tag_tid=17, entry_tid=12317' 2024-04-06T09:42:48.659 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:42:48.659 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:42:48.660 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=17 2024-04-06T09:42:48.660 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=113, tag_tid=18, entry_tid=1' 2024-04-06T09:42:48.660 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:42:48.660 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:42:48.661 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:42:48.661 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=105, tag_tid=17, entry_tid=12317' 2024-04-06T09:42:48.662 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:42:48.662 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:42:48.663 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=12317 2024-04-06T09:42:48.663 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 18 = 17 -a 1 -le 12317 2024-04-06T09:42:48.663 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:42:48.663 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:42:48.664 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:42:48.864 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:42:48.865 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:42:48.865 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:42:48.865 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:42:48.865 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:42:48.865 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:42:48.865 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:42:48.865 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:42:48.865 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:42:48.865 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:42:48.865 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:42:48.865 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:42:48.865 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:42:48.865 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:42:48.866 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:42:48.866 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:42:48.866 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:42:48.870 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:42:48.870 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:42:48.870 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:42:48.870 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:42:48.870 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:42:48.870 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:42:48.870 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:42:48.870 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:42:48.870 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:42:48.870 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:42:48.870 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:42:48.870 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:42:48.870 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:42:48.871 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:42:48.871 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:42:48.871 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:42:48.871 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:42:48.871 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:42:48.871 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:42:48.871 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:42:48.871 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:42:48.871 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:42:48.871 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:42:48.871 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:42:48.871 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:42:48.871 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:42:48.871 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:42:48.871 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:42:48.872 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:42:48.872 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:42:48.872 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:42:48.872 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:42:48.895 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:42:48.896 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:42:48.896 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:42:48.896 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:42:49.030 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:42:49.031 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:42:49.032 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:42:49.032 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:42:49.032 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:42:49.032 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:42:49.032 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:42:49.032 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:42:49.032 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:42:49.032 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:42:49.033 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:42:49.033 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:42:49.033 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:42:49.034 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:42:49.035 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:42:49.035 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:42:49.082 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 26 2024-04-06T09:42:49.082 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 28 2024-04-06T09:42:49.082 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:42:49.083 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=113, tag_tid=18, entry_tid=1], [object_number=112, tag_tid=18, entry_tid=0], [object_number=114, tag_tid=17, entry_tid=20202], [object_number=115, tag_tid=17, entry_tid=20199]]], state=connected] 2024-04-06T09:42:49.083 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=106, tag_tid=17, entry_tid=12402], [object_number=105, tag_tid=17, entry_tid=12401], [object_number=104, tag_tid=17, entry_tid=12400], [object_number=107, tag_tid=17, entry_tid=12399]]], state=connected] 2024-04-06T09:42:49.086 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:42:49.087 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=113, tag_tid=18, entry_tid=1' 2024-04-06T09:42:49.088 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:42:49.088 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:42:49.088 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:42:49.089 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:42:49.089 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:42:49.089 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:42:49.089 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:42:49.089 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:42:49.089 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:42:49.089 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:42:49.089 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:42:49.089 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:42:49.090 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:42:49.090 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:42:49.090 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:42:49.138 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 26 2024-04-06T09:42:49.138 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 28 2024-04-06T09:42:49.138 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:42:49.138 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=113, tag_tid=18, entry_tid=1], [object_number=112, tag_tid=18, entry_tid=0], [object_number=114, tag_tid=17, entry_tid=20202], [object_number=115, tag_tid=17, entry_tid=20199]]], state=connected] 2024-04-06T09:42:49.138 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=106, tag_tid=17, entry_tid=12402], [object_number=105, tag_tid=17, entry_tid=12401], [object_number=104, tag_tid=17, entry_tid=12400], [object_number=107, tag_tid=17, entry_tid=12399]]], state=connected] 2024-04-06T09:42:49.141 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:42:49.143 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=106, tag_tid=17, entry_tid=12402' 2024-04-06T09:42:49.143 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=113, tag_tid=18, entry_tid=1' -a 'object_number=113, tag_tid=18, entry_tid=1' = 'object_number=106, tag_tid=17, entry_tid=12402' 2024-04-06T09:42:49.143 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=106, tag_tid=17, entry_tid=12402' '!=' 'object_number=105, tag_tid=17, entry_tid=12317' 2024-04-06T09:42:49.143 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:42:49.143 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=106, tag_tid=17, entry_tid=12402' = 'object_number=105, tag_tid=17, entry_tid=12317' 2024-04-06T09:42:49.144 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=106, tag_tid=17, entry_tid=12402' 2024-04-06T09:42:49.144 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=113, tag_tid=18, entry_tid=1' 2024-04-06T09:42:49.144 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:42:49.144 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:42:49.145 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=18 2024-04-06T09:42:49.145 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=106, tag_tid=17, entry_tid=12402' 2024-04-06T09:42:49.146 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:42:49.146 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:42:49.146 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=17 2024-04-06T09:42:49.147 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=113, tag_tid=18, entry_tid=1' 2024-04-06T09:42:49.147 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:42:49.148 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:42:49.148 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:42:49.149 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=106, tag_tid=17, entry_tid=12402' 2024-04-06T09:42:49.149 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:42:49.149 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:42:49.150 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=12402 2024-04-06T09:42:49.150 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 18 = 17 -a 1 -le 12402 2024-04-06T09:42:49.150 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:42:49.150 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:42:49.150 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:42:49.352 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:42:49.352 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:42:49.352 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:42:49.352 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:42:49.353 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:42:49.353 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:42:49.353 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:42:49.353 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:42:49.353 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:42:49.353 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:42:49.353 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:42:49.353 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:42:49.353 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:42:49.353 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:42:49.353 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:42:49.354 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:42:49.354 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:42:49.356 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:42:49.356 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:42:49.356 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:42:49.356 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:42:49.356 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:42:49.356 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:42:49.356 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:42:49.356 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:42:49.356 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:42:49.357 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:42:49.357 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:42:49.357 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:42:49.357 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:42:49.357 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:42:49.357 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:42:49.357 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:42:49.358 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:42:49.358 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:42:49.358 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:42:49.358 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:42:49.358 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:42:49.358 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:42:49.358 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:42:49.359 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:42:49.359 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:42:49.359 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:42:49.359 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:42:49.359 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:42:49.359 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:42:49.359 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:42:49.359 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:42:49.359 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:42:49.383 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:42:49.383 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:42:49.383 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:42:49.383 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:42:49.522 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:42:49.523 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:42:49.523 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:42:49.523 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:42:49.524 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:42:49.524 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:42:49.524 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:42:49.524 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:42:49.524 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:42:49.524 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:42:49.524 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:42:49.525 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:42:49.525 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:42:49.526 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:42:49.527 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:42:49.527 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:42:49.578 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 26 2024-04-06T09:42:49.579 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 28 2024-04-06T09:42:49.579 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:42:49.579 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=113, tag_tid=18, entry_tid=1], [object_number=112, tag_tid=18, entry_tid=0], [object_number=114, tag_tid=17, entry_tid=20202], [object_number=115, tag_tid=17, entry_tid=20199]]], state=connected] 2024-04-06T09:42:49.579 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=107, tag_tid=17, entry_tid=12483], [object_number=106, tag_tid=17, entry_tid=12482], [object_number=105, tag_tid=17, entry_tid=12481], [object_number=104, tag_tid=17, entry_tid=12480]]], state=connected] 2024-04-06T09:42:49.582 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:42:49.583 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=113, tag_tid=18, entry_tid=1' 2024-04-06T09:42:49.584 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:42:49.584 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:42:49.585 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:42:49.585 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:42:49.585 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:42:49.585 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:42:49.585 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:42:49.585 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:42:49.585 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:42:49.585 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:42:49.585 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:42:49.585 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:42:49.586 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:42:49.586 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:42:49.586 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:42:49.633 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 26 2024-04-06T09:42:49.633 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 28 2024-04-06T09:42:49.633 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:42:49.633 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=113, tag_tid=18, entry_tid=1], [object_number=112, tag_tid=18, entry_tid=0], [object_number=114, tag_tid=17, entry_tid=20202], [object_number=115, tag_tid=17, entry_tid=20199]]], state=connected] 2024-04-06T09:42:49.633 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=107, tag_tid=17, entry_tid=12483], [object_number=106, tag_tid=17, entry_tid=12482], [object_number=105, tag_tid=17, entry_tid=12481], [object_number=104, tag_tid=17, entry_tid=12480]]], state=connected] 2024-04-06T09:42:49.636 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:42:49.639 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=107, tag_tid=17, entry_tid=12483' 2024-04-06T09:42:49.639 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=113, tag_tid=18, entry_tid=1' -a 'object_number=113, tag_tid=18, entry_tid=1' = 'object_number=107, tag_tid=17, entry_tid=12483' 2024-04-06T09:42:49.639 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=107, tag_tid=17, entry_tid=12483' '!=' 'object_number=106, tag_tid=17, entry_tid=12402' 2024-04-06T09:42:49.639 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:42:49.639 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=107, tag_tid=17, entry_tid=12483' = 'object_number=106, tag_tid=17, entry_tid=12402' 2024-04-06T09:42:49.639 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=107, tag_tid=17, entry_tid=12483' 2024-04-06T09:42:49.639 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=113, tag_tid=18, entry_tid=1' 2024-04-06T09:42:49.639 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:42:49.639 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:42:49.640 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=18 2024-04-06T09:42:49.640 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=107, tag_tid=17, entry_tid=12483' 2024-04-06T09:42:49.640 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:42:49.641 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:42:49.641 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=17 2024-04-06T09:42:49.642 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=113, tag_tid=18, entry_tid=1' 2024-04-06T09:42:49.642 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:42:49.642 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:42:49.643 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:42:49.644 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=107, tag_tid=17, entry_tid=12483' 2024-04-06T09:42:49.644 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:42:49.644 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:42:49.645 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=12483 2024-04-06T09:42:49.645 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 18 = 17 -a 1 -le 12483 2024-04-06T09:42:49.645 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:42:49.645 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:42:49.645 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:42:49.847 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:42:49.847 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:42:49.847 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:42:49.847 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:42:49.847 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:42:49.847 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:42:49.847 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:42:49.848 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:42:49.848 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:42:49.848 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:42:49.848 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:42:49.848 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:42:49.848 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:42:49.848 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:42:49.848 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:42:49.849 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:42:49.849 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:42:49.851 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:42:49.852 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:42:49.852 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:42:49.852 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:42:49.852 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:42:49.852 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:42:49.852 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:42:49.852 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:42:49.852 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:42:49.852 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:42:49.852 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:42:49.852 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:42:49.852 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:42:49.853 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:42:49.853 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:42:49.853 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:42:49.853 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:42:49.853 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:42:49.853 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:42:49.853 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:42:49.853 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:42:49.853 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:42:49.853 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:42:49.853 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:42:49.853 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:42:49.854 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:42:49.854 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:42:49.854 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:42:49.854 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:42:49.854 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:42:49.854 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:42:49.854 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:42:49.878 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:42:49.879 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:42:49.879 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:42:49.879 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:42:50.010 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:42:50.010 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:42:50.010 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:42:50.010 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:42:50.010 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:42:50.010 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:42:50.010 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:42:50.010 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:42:50.011 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:42:50.011 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:42:50.011 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:42:50.011 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:42:50.011 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:42:50.011 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:42:50.011 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:42:50.012 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:42:50.061 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 26 2024-04-06T09:42:50.061 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 28 2024-04-06T09:42:50.061 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:42:50.061 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=113, tag_tid=18, entry_tid=1], [object_number=112, tag_tid=18, entry_tid=0], [object_number=114, tag_tid=17, entry_tid=20202], [object_number=115, tag_tid=17, entry_tid=20199]]], state=connected] 2024-04-06T09:42:50.061 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=106, tag_tid=17, entry_tid=12578], [object_number=105, tag_tid=17, entry_tid=12577], [object_number=104, tag_tid=17, entry_tid=12576], [object_number=107, tag_tid=17, entry_tid=12575]]], state=connected] 2024-04-06T09:42:50.064 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:42:50.066 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=113, tag_tid=18, entry_tid=1' 2024-04-06T09:42:50.067 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:42:50.067 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:42:50.067 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:42:50.067 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:42:50.067 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:42:50.067 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:42:50.067 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:42:50.067 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:42:50.068 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:42:50.068 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:42:50.068 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:42:50.068 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:42:50.068 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:42:50.068 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:42:50.068 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:42:50.116 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 26 2024-04-06T09:42:50.116 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 28 2024-04-06T09:42:50.117 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:42:50.117 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=113, tag_tid=18, entry_tid=1], [object_number=112, tag_tid=18, entry_tid=0], [object_number=114, tag_tid=17, entry_tid=20202], [object_number=115, tag_tid=17, entry_tid=20199]]], state=connected] 2024-04-06T09:42:50.117 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=106, tag_tid=17, entry_tid=12578], [object_number=105, tag_tid=17, entry_tid=12577], [object_number=104, tag_tid=17, entry_tid=12576], [object_number=107, tag_tid=17, entry_tid=12575]]], state=connected] 2024-04-06T09:42:50.120 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:42:50.122 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=106, tag_tid=17, entry_tid=12578' 2024-04-06T09:42:50.122 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=113, tag_tid=18, entry_tid=1' -a 'object_number=113, tag_tid=18, entry_tid=1' = 'object_number=106, tag_tid=17, entry_tid=12578' 2024-04-06T09:42:50.122 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=106, tag_tid=17, entry_tid=12578' '!=' 'object_number=107, tag_tid=17, entry_tid=12483' 2024-04-06T09:42:50.122 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:42:50.122 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=106, tag_tid=17, entry_tid=12578' = 'object_number=107, tag_tid=17, entry_tid=12483' 2024-04-06T09:42:50.122 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=106, tag_tid=17, entry_tid=12578' 2024-04-06T09:42:50.122 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=113, tag_tid=18, entry_tid=1' 2024-04-06T09:42:50.122 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:42:50.123 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:42:50.123 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=18 2024-04-06T09:42:50.124 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=106, tag_tid=17, entry_tid=12578' 2024-04-06T09:42:50.124 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:42:50.124 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:42:50.124 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=17 2024-04-06T09:42:50.125 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=113, tag_tid=18, entry_tid=1' 2024-04-06T09:42:50.126 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:42:50.126 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:42:50.126 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:42:50.127 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=106, tag_tid=17, entry_tid=12578' 2024-04-06T09:42:50.127 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:42:50.127 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:42:50.128 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=12578 2024-04-06T09:42:50.128 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 18 = 17 -a 1 -le 12578 2024-04-06T09:42:50.129 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:42:50.129 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:42:50.129 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:42:50.330 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:42:50.330 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:42:50.330 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:42:50.330 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:42:50.331 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:42:50.331 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:42:50.331 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:42:50.331 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:42:50.331 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:42:50.331 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:42:50.331 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:42:50.331 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:42:50.331 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:42:50.331 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:42:50.331 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:42:50.332 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:42:50.332 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:42:50.335 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:42:50.335 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:42:50.335 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:42:50.336 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:42:50.336 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:42:50.336 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:42:50.336 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:42:50.336 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:42:50.336 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:42:50.336 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:42:50.336 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:42:50.336 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:42:50.336 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:42:50.336 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:42:50.336 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:42:50.336 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:42:50.337 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:42:50.337 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:42:50.337 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:42:50.337 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:42:50.337 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:42:50.337 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:42:50.337 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:42:50.337 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:42:50.337 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:42:50.337 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:42:50.337 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:42:50.338 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:42:50.338 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:42:50.338 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:42:50.338 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:42:50.338 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:42:50.362 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:42:50.362 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:42:50.363 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:42:50.363 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:42:50.494 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:42:50.496 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:42:50.496 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:42:50.496 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:42:50.496 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:42:50.496 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:42:50.496 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:42:50.496 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:42:50.496 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:42:50.496 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:42:50.497 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:42:50.497 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:42:50.497 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:42:50.498 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:42:50.499 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:42:50.499 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:42:50.548 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 26 2024-04-06T09:42:50.548 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 28 2024-04-06T09:42:50.548 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:42:50.548 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=113, tag_tid=18, entry_tid=1], [object_number=112, tag_tid=18, entry_tid=0], [object_number=114, tag_tid=17, entry_tid=20202], [object_number=115, tag_tid=17, entry_tid=20199]]], state=connected] 2024-04-06T09:42:50.548 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=107, tag_tid=17, entry_tid=12679], [object_number=106, tag_tid=17, entry_tid=12678], [object_number=105, tag_tid=17, entry_tid=12677], [object_number=104, tag_tid=17, entry_tid=12676]]], state=connected] 2024-04-06T09:42:50.551 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:42:50.553 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=113, tag_tid=18, entry_tid=1' 2024-04-06T09:42:50.554 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:42:50.554 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:42:50.554 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:42:50.554 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:42:50.554 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:42:50.554 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:42:50.555 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:42:50.555 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:42:50.555 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:42:50.555 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:42:50.555 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:42:50.555 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:42:50.556 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:42:50.556 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:42:50.556 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:42:50.602 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 26 2024-04-06T09:42:50.602 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 28 2024-04-06T09:42:50.602 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:42:50.602 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=113, tag_tid=18, entry_tid=1], [object_number=112, tag_tid=18, entry_tid=0], [object_number=114, tag_tid=17, entry_tid=20202], [object_number=115, tag_tid=17, entry_tid=20199]]], state=connected] 2024-04-06T09:42:50.602 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=107, tag_tid=17, entry_tid=12679], [object_number=106, tag_tid=17, entry_tid=12678], [object_number=105, tag_tid=17, entry_tid=12677], [object_number=104, tag_tid=17, entry_tid=12676]]], state=connected] 2024-04-06T09:42:50.605 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:42:50.607 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=107, tag_tid=17, entry_tid=12679' 2024-04-06T09:42:50.607 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=113, tag_tid=18, entry_tid=1' -a 'object_number=113, tag_tid=18, entry_tid=1' = 'object_number=107, tag_tid=17, entry_tid=12679' 2024-04-06T09:42:50.608 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=107, tag_tid=17, entry_tid=12679' '!=' 'object_number=106, tag_tid=17, entry_tid=12578' 2024-04-06T09:42:50.608 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:42:50.608 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=107, tag_tid=17, entry_tid=12679' = 'object_number=106, tag_tid=17, entry_tid=12578' 2024-04-06T09:42:50.608 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=107, tag_tid=17, entry_tid=12679' 2024-04-06T09:42:50.608 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=113, tag_tid=18, entry_tid=1' 2024-04-06T09:42:50.608 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:42:50.608 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:42:50.609 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=18 2024-04-06T09:42:50.609 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=107, tag_tid=17, entry_tid=12679' 2024-04-06T09:42:50.609 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:42:50.609 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:42:50.610 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=17 2024-04-06T09:42:50.611 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=113, tag_tid=18, entry_tid=1' 2024-04-06T09:42:50.611 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:42:50.611 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:42:50.612 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:42:50.613 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=107, tag_tid=17, entry_tid=12679' 2024-04-06T09:42:50.613 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:42:50.613 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:42:50.614 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=12679 2024-04-06T09:42:50.614 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 18 = 17 -a 1 -le 12679 2024-04-06T09:42:50.614 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:42:50.614 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:42:50.614 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:42:50.816 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:42:50.816 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:42:50.817 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:42:50.817 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:42:50.817 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:42:50.817 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:42:50.817 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:42:50.817 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:42:50.817 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:42:50.817 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:42:50.817 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:42:50.817 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:42:50.817 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:42:50.817 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:42:50.817 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:42:50.817 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:42:50.818 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:42:50.820 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:42:50.820 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:42:50.821 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:42:50.821 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:42:50.821 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:42:50.821 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:42:50.821 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:42:50.821 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:42:50.821 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:42:50.821 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:42:50.821 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:42:50.821 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:42:50.821 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:42:50.821 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:42:50.821 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:42:50.821 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:42:50.821 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:42:50.822 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:42:50.822 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:42:50.822 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:42:50.822 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:42:50.822 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:42:50.822 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:42:50.822 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:42:50.822 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:42:50.822 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:42:50.822 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:42:50.822 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:42:50.822 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:42:50.822 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:42:50.822 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:42:50.823 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:42:50.845 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:42:50.845 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:42:50.845 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:42:50.846 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:42:50.976 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:42:50.977 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:42:50.977 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:42:50.977 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:42:50.977 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:42:50.978 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:42:50.978 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:42:50.978 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:42:50.978 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:42:50.978 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:42:50.978 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:42:50.978 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:42:50.978 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:42:50.979 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:42:50.979 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:42:50.980 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:42:51.026 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 26 2024-04-06T09:42:51.026 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 28 2024-04-06T09:42:51.026 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:42:51.026 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=113, tag_tid=18, entry_tid=1], [object_number=112, tag_tid=18, entry_tid=0], [object_number=114, tag_tid=17, entry_tid=20202], [object_number=115, tag_tid=17, entry_tid=20199]]], state=connected] 2024-04-06T09:42:51.026 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=107, tag_tid=17, entry_tid=12771], [object_number=106, tag_tid=17, entry_tid=12770], [object_number=105, tag_tid=17, entry_tid=12769], [object_number=104, tag_tid=17, entry_tid=12768]]], state=connected] 2024-04-06T09:42:51.029 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:42:51.031 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=113, tag_tid=18, entry_tid=1' 2024-04-06T09:42:51.032 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:42:51.032 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:42:51.032 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:42:51.032 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:42:51.032 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:42:51.032 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:42:51.032 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:42:51.032 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:42:51.033 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:42:51.033 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:42:51.033 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:42:51.033 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:42:51.033 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:42:51.033 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:42:51.034 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:42:51.079 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 26 2024-04-06T09:42:51.079 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 28 2024-04-06T09:42:51.079 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:42:51.079 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=113, tag_tid=18, entry_tid=1], [object_number=112, tag_tid=18, entry_tid=0], [object_number=114, tag_tid=17, entry_tid=20202], [object_number=115, tag_tid=17, entry_tid=20199]]], state=connected] 2024-04-06T09:42:51.079 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=107, tag_tid=17, entry_tid=12771], [object_number=106, tag_tid=17, entry_tid=12770], [object_number=105, tag_tid=17, entry_tid=12769], [object_number=104, tag_tid=17, entry_tid=12768]]], state=connected] 2024-04-06T09:42:51.082 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:42:51.084 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=107, tag_tid=17, entry_tid=12771' 2024-04-06T09:42:51.084 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=113, tag_tid=18, entry_tid=1' -a 'object_number=113, tag_tid=18, entry_tid=1' = 'object_number=107, tag_tid=17, entry_tid=12771' 2024-04-06T09:42:51.084 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=107, tag_tid=17, entry_tid=12771' '!=' 'object_number=107, tag_tid=17, entry_tid=12679' 2024-04-06T09:42:51.084 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:42:51.084 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=107, tag_tid=17, entry_tid=12771' = 'object_number=107, tag_tid=17, entry_tid=12679' 2024-04-06T09:42:51.085 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=107, tag_tid=17, entry_tid=12771' 2024-04-06T09:42:51.085 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=113, tag_tid=18, entry_tid=1' 2024-04-06T09:42:51.085 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:42:51.085 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:42:51.085 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=18 2024-04-06T09:42:51.086 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=107, tag_tid=17, entry_tid=12771' 2024-04-06T09:42:51.086 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:42:51.086 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:42:51.087 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=17 2024-04-06T09:42:51.088 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=113, tag_tid=18, entry_tid=1' 2024-04-06T09:42:51.088 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:42:51.088 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:42:51.089 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:42:51.089 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=107, tag_tid=17, entry_tid=12771' 2024-04-06T09:42:51.089 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:42:51.089 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:42:51.090 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=12771 2024-04-06T09:42:51.091 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 18 = 17 -a 1 -le 12771 2024-04-06T09:42:51.091 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:42:51.091 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:42:51.091 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:42:51.293 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:42:51.293 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:42:51.293 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:42:51.293 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:42:51.293 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:42:51.293 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:42:51.293 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:42:51.293 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:42:51.293 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:42:51.293 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:42:51.293 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:42:51.294 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:42:51.294 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:42:51.294 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:42:51.294 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:42:51.295 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:42:51.295 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:42:51.295 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:42:51.295 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:42:51.295 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:42:51.296 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:42:51.296 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:42:51.296 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:42:51.296 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:42:51.296 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:42:51.296 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:42:51.296 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:42:51.296 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:42:51.296 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:42:51.296 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:42:51.296 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:42:51.296 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:42:51.296 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:42:51.298 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:42:51.298 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:42:51.298 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:42:51.298 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:42:51.298 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:42:51.298 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:42:51.298 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:42:51.298 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:42:51.298 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:42:51.298 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:42:51.299 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:42:51.299 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:42:51.299 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:42:51.299 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:42:51.299 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:42:51.299 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:42:51.322 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:42:51.323 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:42:51.323 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:42:51.323 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:42:51.454 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:42:51.455 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:42:51.455 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:42:51.455 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:42:51.455 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:42:51.456 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:42:51.456 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:42:51.456 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:42:51.456 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:42:51.456 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:42:51.456 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:42:51.456 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:42:51.456 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:42:51.458 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:42:51.459 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:42:51.459 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:42:51.512 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 26 2024-04-06T09:42:51.513 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 28 2024-04-06T09:42:51.513 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:42:51.513 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=113, tag_tid=18, entry_tid=1], [object_number=112, tag_tid=18, entry_tid=0], [object_number=114, tag_tid=17, entry_tid=20202], [object_number=115, tag_tid=17, entry_tid=20199]]], state=connected] 2024-04-06T09:42:51.513 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=105, tag_tid=17, entry_tid=12869], [object_number=104, tag_tid=17, entry_tid=12868], [object_number=107, tag_tid=17, entry_tid=12867], [object_number=106, tag_tid=17, entry_tid=12866]]], state=connected] 2024-04-06T09:42:51.516 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:42:51.517 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=113, tag_tid=18, entry_tid=1' 2024-04-06T09:42:51.518 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:42:51.519 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:42:51.519 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:42:51.519 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:42:51.519 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:42:51.519 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:42:51.519 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:42:51.519 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:42:51.519 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:42:51.519 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:42:51.519 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:42:51.519 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:42:51.520 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:42:51.520 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:42:51.520 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:42:51.774 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 26 2024-04-06T09:42:51.774 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 28 2024-04-06T09:42:51.774 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:42:51.774 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=113, tag_tid=18, entry_tid=1], [object_number=112, tag_tid=18, entry_tid=0], [object_number=114, tag_tid=17, entry_tid=20202], [object_number=115, tag_tid=17, entry_tid=20199]]], state=connected] 2024-04-06T09:42:51.774 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=105, tag_tid=17, entry_tid=12869], [object_number=104, tag_tid=17, entry_tid=12868], [object_number=107, tag_tid=17, entry_tid=12867], [object_number=106, tag_tid=17, entry_tid=12866]]], state=connected] 2024-04-06T09:42:51.779 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:42:51.781 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=105, tag_tid=17, entry_tid=12869' 2024-04-06T09:42:51.781 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=113, tag_tid=18, entry_tid=1' -a 'object_number=113, tag_tid=18, entry_tid=1' = 'object_number=105, tag_tid=17, entry_tid=12869' 2024-04-06T09:42:51.781 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=105, tag_tid=17, entry_tid=12869' '!=' 'object_number=107, tag_tid=17, entry_tid=12771' 2024-04-06T09:42:51.781 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:42:51.781 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=105, tag_tid=17, entry_tid=12869' = 'object_number=107, tag_tid=17, entry_tid=12771' 2024-04-06T09:42:51.781 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=105, tag_tid=17, entry_tid=12869' 2024-04-06T09:42:51.782 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=113, tag_tid=18, entry_tid=1' 2024-04-06T09:42:51.782 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:42:51.782 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:42:51.783 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=18 2024-04-06T09:42:51.784 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=105, tag_tid=17, entry_tid=12869' 2024-04-06T09:42:51.784 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:42:51.784 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:42:51.786 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=17 2024-04-06T09:42:51.786 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=113, tag_tid=18, entry_tid=1' 2024-04-06T09:42:51.787 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:42:51.787 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:42:51.788 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:42:51.789 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=105, tag_tid=17, entry_tid=12869' 2024-04-06T09:42:51.789 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:42:51.789 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:42:51.791 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=12869 2024-04-06T09:42:51.791 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 18 = 17 -a 1 -le 12869 2024-04-06T09:42:51.791 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:42:51.791 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:42:51.791 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:42:51.993 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:42:51.993 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:42:51.993 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:42:51.993 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:42:51.993 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:42:51.994 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:42:51.994 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:42:51.994 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:42:51.994 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:42:51.994 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:42:51.994 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:42:51.994 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:42:51.994 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:42:51.994 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:42:51.994 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:42:51.995 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:42:51.995 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:42:51.998 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:42:51.998 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:42:51.998 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:42:51.998 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:42:51.998 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:42:51.998 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:42:51.998 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:42:51.998 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:42:51.998 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:42:51.998 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:42:51.998 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:42:51.998 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:42:51.999 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:42:51.999 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:42:51.999 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:42:51.999 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:42:51.999 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:42:51.999 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:42:51.999 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:42:51.999 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:42:51.999 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:42:51.999 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:42:51.999 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:42:51.999 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:42:51.999 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:42:51.999 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:42:51.999 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:42:51.999 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:42:52.000 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:42:52.000 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:42:52.000 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:42:52.000 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:42:52.023 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:42:52.023 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:42:52.023 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:42:52.023 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:42:52.152 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:42:52.153 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:42:52.153 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:42:52.153 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:42:52.153 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:42:52.153 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:42:52.153 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:42:52.154 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:42:52.154 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:42:52.154 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:42:52.154 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:42:52.154 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:42:52.154 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:42:52.156 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:42:52.156 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:42:52.156 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:42:52.202 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 26 2024-04-06T09:42:52.202 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 28 2024-04-06T09:42:52.202 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:42:52.202 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=113, tag_tid=18, entry_tid=1], [object_number=112, tag_tid=18, entry_tid=0], [object_number=114, tag_tid=17, entry_tid=20202], [object_number=115, tag_tid=17, entry_tid=20199]]], state=connected] 2024-04-06T09:42:52.202 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=106, tag_tid=17, entry_tid=13018], [object_number=105, tag_tid=17, entry_tid=13017], [object_number=104, tag_tid=17, entry_tid=13016], [object_number=107, tag_tid=17, entry_tid=13015]]], state=connected] 2024-04-06T09:42:52.205 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:42:52.207 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=113, tag_tid=18, entry_tid=1' 2024-04-06T09:42:52.209 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:42:52.209 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:42:52.209 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:42:52.210 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:42:52.210 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:42:52.210 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:42:52.210 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:42:52.210 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:42:52.210 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:42:52.210 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:42:52.210 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:42:52.210 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:42:52.210 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:42:52.210 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:42:52.210 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:42:52.255 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 26 2024-04-06T09:42:52.255 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 28 2024-04-06T09:42:52.255 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:42:52.255 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=113, tag_tid=18, entry_tid=1], [object_number=112, tag_tid=18, entry_tid=0], [object_number=114, tag_tid=17, entry_tid=20202], [object_number=115, tag_tid=17, entry_tid=20199]]], state=connected] 2024-04-06T09:42:52.255 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=106, tag_tid=17, entry_tid=13018], [object_number=105, tag_tid=17, entry_tid=13017], [object_number=104, tag_tid=17, entry_tid=13016], [object_number=107, tag_tid=17, entry_tid=13015]]], state=connected] 2024-04-06T09:42:52.258 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:42:52.260 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=106, tag_tid=17, entry_tid=13018' 2024-04-06T09:42:52.260 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=113, tag_tid=18, entry_tid=1' -a 'object_number=113, tag_tid=18, entry_tid=1' = 'object_number=106, tag_tid=17, entry_tid=13018' 2024-04-06T09:42:52.260 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=106, tag_tid=17, entry_tid=13018' '!=' 'object_number=105, tag_tid=17, entry_tid=12869' 2024-04-06T09:42:52.260 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:42:52.260 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=106, tag_tid=17, entry_tid=13018' = 'object_number=105, tag_tid=17, entry_tid=12869' 2024-04-06T09:42:52.260 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=106, tag_tid=17, entry_tid=13018' 2024-04-06T09:42:52.261 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=113, tag_tid=18, entry_tid=1' 2024-04-06T09:42:52.261 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:42:52.261 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:42:52.261 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=18 2024-04-06T09:42:52.262 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=106, tag_tid=17, entry_tid=13018' 2024-04-06T09:42:52.262 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:42:52.262 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:42:52.263 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=17 2024-04-06T09:42:52.264 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=113, tag_tid=18, entry_tid=1' 2024-04-06T09:42:52.264 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:42:52.264 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:42:52.265 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:42:52.266 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=106, tag_tid=17, entry_tid=13018' 2024-04-06T09:42:52.266 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:42:52.266 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:42:52.267 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=13018 2024-04-06T09:42:52.267 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 18 = 17 -a 1 -le 13018 2024-04-06T09:42:52.267 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:42:52.267 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:42:52.267 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:42:52.469 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:42:52.469 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:42:52.469 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:42:52.469 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:42:52.469 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:42:52.469 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:42:52.469 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:42:52.469 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:42:52.469 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:42:52.470 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:42:52.470 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:42:52.470 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:42:52.470 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:42:52.470 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:42:52.470 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:42:52.470 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:42:52.471 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:42:52.475 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:42:52.475 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:42:52.475 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:42:52.475 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:42:52.475 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:42:52.475 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:42:52.475 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:42:52.475 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:42:52.475 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:42:52.476 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:42:52.476 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:42:52.476 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:42:52.476 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:42:52.476 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:42:52.476 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:42:52.476 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:42:52.476 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:42:52.476 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:42:52.476 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:42:52.477 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:42:52.477 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:42:52.477 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:42:52.477 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:42:52.477 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:42:52.477 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:42:52.477 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:42:52.477 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:42:52.477 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:42:52.477 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:42:52.477 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:42:52.477 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:42:52.477 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:42:52.502 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:42:52.502 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:42:52.502 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:42:52.502 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:42:52.637 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:42:52.638 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:42:52.638 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:42:52.638 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:42:52.638 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:42:52.638 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:42:52.639 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:42:52.639 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:42:52.639 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:42:52.639 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:42:52.639 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:42:52.640 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:42:52.640 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:42:52.642 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:42:52.643 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:42:52.644 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:42:52.689 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 26 2024-04-06T09:42:52.689 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 28 2024-04-06T09:42:52.689 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:42:52.690 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=113, tag_tid=18, entry_tid=1], [object_number=112, tag_tid=18, entry_tid=0], [object_number=114, tag_tid=17, entry_tid=20202], [object_number=115, tag_tid=17, entry_tid=20199]]], state=connected] 2024-04-06T09:42:52.690 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=107, tag_tid=17, entry_tid=13107], [object_number=106, tag_tid=17, entry_tid=13106], [object_number=105, tag_tid=17, entry_tid=13105], [object_number=104, tag_tid=17, entry_tid=13104]]], state=connected] 2024-04-06T09:42:52.693 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:42:52.694 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=113, tag_tid=18, entry_tid=1' 2024-04-06T09:42:52.695 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:42:52.695 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:42:52.696 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:42:52.696 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:42:52.696 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:42:52.696 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:42:52.696 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:42:52.696 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:42:52.696 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:42:52.696 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:42:52.696 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:42:52.696 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:42:52.697 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:42:52.697 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:42:52.697 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:42:52.744 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 26 2024-04-06T09:42:52.744 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 28 2024-04-06T09:42:52.744 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:42:52.744 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=113, tag_tid=18, entry_tid=1], [object_number=112, tag_tid=18, entry_tid=0], [object_number=114, tag_tid=17, entry_tid=20202], [object_number=115, tag_tid=17, entry_tid=20199]]], state=connected] 2024-04-06T09:42:52.745 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=107, tag_tid=17, entry_tid=13107], [object_number=106, tag_tid=17, entry_tid=13106], [object_number=105, tag_tid=17, entry_tid=13105], [object_number=104, tag_tid=17, entry_tid=13104]]], state=connected] 2024-04-06T09:42:52.748 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:42:52.750 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=107, tag_tid=17, entry_tid=13107' 2024-04-06T09:42:52.750 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=113, tag_tid=18, entry_tid=1' -a 'object_number=113, tag_tid=18, entry_tid=1' = 'object_number=107, tag_tid=17, entry_tid=13107' 2024-04-06T09:42:52.750 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=107, tag_tid=17, entry_tid=13107' '!=' 'object_number=106, tag_tid=17, entry_tid=13018' 2024-04-06T09:42:52.750 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:42:52.750 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=107, tag_tid=17, entry_tid=13107' = 'object_number=106, tag_tid=17, entry_tid=13018' 2024-04-06T09:42:52.750 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=107, tag_tid=17, entry_tid=13107' 2024-04-06T09:42:52.750 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=113, tag_tid=18, entry_tid=1' 2024-04-06T09:42:52.750 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:42:52.750 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:42:52.751 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=18 2024-04-06T09:42:52.751 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=107, tag_tid=17, entry_tid=13107' 2024-04-06T09:42:52.751 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:42:52.751 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:42:52.752 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=17 2024-04-06T09:42:52.753 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=113, tag_tid=18, entry_tid=1' 2024-04-06T09:42:52.753 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:42:52.753 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:42:52.754 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:42:52.754 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=107, tag_tid=17, entry_tid=13107' 2024-04-06T09:42:52.755 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:42:52.755 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:42:52.756 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=13107 2024-04-06T09:42:52.756 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 18 = 17 -a 1 -le 13107 2024-04-06T09:42:52.756 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:42:52.756 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:42:52.756 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:42:52.958 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:42:52.959 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:42:52.959 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:42:52.959 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:42:52.959 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:42:52.959 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:42:52.959 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:42:52.959 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:42:52.959 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:42:52.959 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:42:52.959 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:42:52.960 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:42:52.960 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:42:52.960 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:42:52.960 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:42:52.960 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:42:52.963 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:42:52.963 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:42:52.963 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:42:52.963 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:42:52.963 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:42:52.963 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:42:52.963 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:42:52.963 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:42:52.964 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:42:52.964 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:42:52.964 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:42:52.964 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:42:52.964 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:42:52.964 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:42:52.964 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:42:52.964 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:42:52.964 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:42:52.964 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:42:52.964 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:42:52.965 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:42:52.965 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:42:52.965 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:42:52.965 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:42:52.965 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:42:52.965 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:42:52.965 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:42:52.965 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:42:52.965 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:42:52.965 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:42:52.965 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:42:52.965 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:42:52.965 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:42:52.966 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:42:52.989 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:42:52.989 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:42:52.989 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:42:52.989 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:42:53.120 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:42:53.121 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:42:53.121 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:42:53.121 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:42:53.121 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:42:53.121 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:42:53.121 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:42:53.122 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:42:53.122 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:42:53.122 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:42:53.122 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:42:53.122 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:42:53.122 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:42:53.125 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:42:53.125 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:42:53.125 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:42:53.178 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 26 2024-04-06T09:42:53.178 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 28 2024-04-06T09:42:53.178 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:42:53.178 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=113, tag_tid=18, entry_tid=1], [object_number=112, tag_tid=18, entry_tid=0], [object_number=114, tag_tid=17, entry_tid=20202], [object_number=115, tag_tid=17, entry_tid=20199]]], state=connected] 2024-04-06T09:42:53.178 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=105, tag_tid=17, entry_tid=13205], [object_number=104, tag_tid=17, entry_tid=13204], [object_number=107, tag_tid=17, entry_tid=13203], [object_number=106, tag_tid=17, entry_tid=13202]]], state=connected] 2024-04-06T09:42:53.182 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:42:53.183 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=113, tag_tid=18, entry_tid=1' 2024-04-06T09:42:53.184 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:42:53.184 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:42:53.184 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:42:53.184 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:42:53.184 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:42:53.184 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:42:53.184 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:42:53.184 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:42:53.185 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:42:53.185 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:42:53.185 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:42:53.185 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:42:53.185 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:42:53.186 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:42:53.186 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:42:53.232 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 26 2024-04-06T09:42:53.233 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 28 2024-04-06T09:42:53.233 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:42:53.233 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=113, tag_tid=18, entry_tid=1], [object_number=112, tag_tid=18, entry_tid=0], [object_number=114, tag_tid=17, entry_tid=20202], [object_number=115, tag_tid=17, entry_tid=20199]]], state=connected] 2024-04-06T09:42:53.233 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=105, tag_tid=17, entry_tid=13205], [object_number=104, tag_tid=17, entry_tid=13204], [object_number=107, tag_tid=17, entry_tid=13203], [object_number=106, tag_tid=17, entry_tid=13202]]], state=connected] 2024-04-06T09:42:53.236 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:42:53.238 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=105, tag_tid=17, entry_tid=13205' 2024-04-06T09:42:53.238 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=113, tag_tid=18, entry_tid=1' -a 'object_number=113, tag_tid=18, entry_tid=1' = 'object_number=105, tag_tid=17, entry_tid=13205' 2024-04-06T09:42:53.238 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=105, tag_tid=17, entry_tid=13205' '!=' 'object_number=107, tag_tid=17, entry_tid=13107' 2024-04-06T09:42:53.238 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:42:53.238 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=105, tag_tid=17, entry_tid=13205' = 'object_number=107, tag_tid=17, entry_tid=13107' 2024-04-06T09:42:53.238 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=105, tag_tid=17, entry_tid=13205' 2024-04-06T09:42:53.239 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=113, tag_tid=18, entry_tid=1' 2024-04-06T09:42:53.239 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:42:53.239 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:42:53.239 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=18 2024-04-06T09:42:53.240 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=105, tag_tid=17, entry_tid=13205' 2024-04-06T09:42:53.240 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:42:53.240 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:42:53.241 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=17 2024-04-06T09:42:53.242 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=113, tag_tid=18, entry_tid=1' 2024-04-06T09:42:53.242 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:42:53.243 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:42:53.243 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:42:53.244 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=105, tag_tid=17, entry_tid=13205' 2024-04-06T09:42:53.244 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:42:53.244 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:42:53.245 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=13205 2024-04-06T09:42:53.245 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 18 = 17 -a 1 -le 13205 2024-04-06T09:42:53.245 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:42:53.245 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:42:53.245 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:42:53.447 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:42:53.447 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:42:53.447 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:42:53.447 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:42:53.447 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:42:53.448 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:42:53.448 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:42:53.448 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:42:53.448 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:42:53.448 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:42:53.448 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:42:53.448 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:42:53.448 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:42:53.448 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:42:53.448 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:42:53.448 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:42:53.449 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:42:53.454 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:42:53.454 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:42:53.454 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:42:53.454 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:42:53.454 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:42:53.454 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:42:53.454 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:42:53.455 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:42:53.455 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:42:53.455 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:42:53.455 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:42:53.455 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:42:53.455 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:42:53.455 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:42:53.455 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:42:53.455 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:42:53.455 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:42:53.455 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:42:53.455 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:42:53.455 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:42:53.455 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:42:53.455 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:42:53.456 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:42:53.456 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:42:53.456 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:42:53.456 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:42:53.456 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:42:53.456 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:42:53.456 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:42:53.456 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:42:53.456 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:42:53.456 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:42:53.479 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:42:53.479 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:42:53.479 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:42:53.479 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:42:53.608 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:42:53.609 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:42:53.609 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:42:53.609 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:42:53.610 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:42:53.610 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:42:53.610 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:42:53.610 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:42:53.610 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:42:53.610 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:42:53.611 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:42:53.611 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:42:53.611 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:42:53.613 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:42:53.613 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:42:53.613 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:42:53.665 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 26 2024-04-06T09:42:53.665 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 28 2024-04-06T09:42:53.665 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:42:53.666 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=113, tag_tid=18, entry_tid=1], [object_number=112, tag_tid=18, entry_tid=0], [object_number=114, tag_tid=17, entry_tid=20202], [object_number=115, tag_tid=17, entry_tid=20199]]], state=connected] 2024-04-06T09:42:53.666 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=104, tag_tid=17, entry_tid=13300], [object_number=107, tag_tid=17, entry_tid=13299], [object_number=106, tag_tid=17, entry_tid=13298], [object_number=105, tag_tid=17, entry_tid=13297]]], state=connected] 2024-04-06T09:42:53.669 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:42:53.672 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=113, tag_tid=18, entry_tid=1' 2024-04-06T09:42:53.672 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:42:53.672 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:42:53.672 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:42:53.672 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:42:53.672 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:42:53.672 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:42:53.672 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:42:53.672 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:42:53.672 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:42:53.672 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:42:53.673 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:42:53.673 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:42:53.673 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:42:53.673 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:42:53.673 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:42:53.721 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 26 2024-04-06T09:42:53.721 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 28 2024-04-06T09:42:53.721 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:42:53.721 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=113, tag_tid=18, entry_tid=1], [object_number=112, tag_tid=18, entry_tid=0], [object_number=114, tag_tid=17, entry_tid=20202], [object_number=115, tag_tid=17, entry_tid=20199]]], state=connected] 2024-04-06T09:42:53.721 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=104, tag_tid=17, entry_tid=13300], [object_number=107, tag_tid=17, entry_tid=13299], [object_number=106, tag_tid=17, entry_tid=13298], [object_number=105, tag_tid=17, entry_tid=13297]]], state=connected] 2024-04-06T09:42:53.724 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:42:53.727 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=104, tag_tid=17, entry_tid=13300' 2024-04-06T09:42:53.727 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=113, tag_tid=18, entry_tid=1' -a 'object_number=113, tag_tid=18, entry_tid=1' = 'object_number=104, tag_tid=17, entry_tid=13300' 2024-04-06T09:42:53.727 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=104, tag_tid=17, entry_tid=13300' '!=' 'object_number=105, tag_tid=17, entry_tid=13205' 2024-04-06T09:42:53.727 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:42:53.727 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=104, tag_tid=17, entry_tid=13300' = 'object_number=105, tag_tid=17, entry_tid=13205' 2024-04-06T09:42:53.727 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=104, tag_tid=17, entry_tid=13300' 2024-04-06T09:42:53.727 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=113, tag_tid=18, entry_tid=1' 2024-04-06T09:42:53.727 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:42:53.727 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:42:53.728 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=18 2024-04-06T09:42:53.728 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=104, tag_tid=17, entry_tid=13300' 2024-04-06T09:42:53.728 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:42:53.729 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:42:53.729 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=17 2024-04-06T09:42:53.730 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=113, tag_tid=18, entry_tid=1' 2024-04-06T09:42:53.730 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:42:53.730 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:42:53.731 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:42:53.731 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=104, tag_tid=17, entry_tid=13300' 2024-04-06T09:42:53.732 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:42:53.732 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:42:53.733 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=13300 2024-04-06T09:42:53.733 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 18 = 17 -a 1 -le 13300 2024-04-06T09:42:53.733 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:42:53.733 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:42:53.733 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:42:53.935 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:42:53.935 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:42:53.935 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:42:53.935 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:42:53.935 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:42:53.935 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:42:53.935 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:42:53.935 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:42:53.936 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:42:53.936 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:42:53.936 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:42:53.936 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:42:53.936 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:42:53.936 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:42:53.936 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:42:53.937 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:42:53.937 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:42:53.940 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:42:53.940 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:42:53.940 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:42:53.940 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:42:53.940 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:42:53.940 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:42:53.940 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:42:53.940 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:42:53.941 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:42:53.941 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:42:53.941 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:42:53.941 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:42:53.941 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:42:53.941 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:42:53.941 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:42:53.941 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:42:53.941 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:42:53.941 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:42:53.941 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:42:53.941 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:42:53.941 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:42:53.941 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:42:53.941 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:42:53.941 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:42:53.941 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:42:53.942 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:42:53.942 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:42:53.942 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:42:53.942 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:42:53.942 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:42:53.942 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:42:53.942 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:42:53.966 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:42:53.966 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:42:53.966 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:42:53.966 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:42:54.094 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:42:54.095 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:42:54.095 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:42:54.095 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:42:54.095 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:42:54.095 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:42:54.095 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:42:54.095 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:42:54.095 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:42:54.095 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:42:54.096 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:42:54.096 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:42:54.096 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:42:54.098 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:42:54.099 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:42:54.099 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:42:54.146 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 26 2024-04-06T09:42:54.146 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 28 2024-04-06T09:42:54.146 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:42:54.147 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=113, tag_tid=18, entry_tid=1], [object_number=112, tag_tid=18, entry_tid=0], [object_number=114, tag_tid=17, entry_tid=20202], [object_number=115, tag_tid=17, entry_tid=20199]]], state=connected] 2024-04-06T09:42:54.147 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=105, tag_tid=17, entry_tid=13385], [object_number=104, tag_tid=17, entry_tid=13384], [object_number=107, tag_tid=17, entry_tid=13383], [object_number=106, tag_tid=17, entry_tid=13382]]], state=connected] 2024-04-06T09:42:54.150 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:42:54.151 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=113, tag_tid=18, entry_tid=1' 2024-04-06T09:42:54.152 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:42:54.152 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:42:54.152 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:42:54.152 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:42:54.153 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:42:54.153 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:42:54.153 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:42:54.153 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:42:54.153 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:42:54.153 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:42:54.153 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:42:54.153 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:42:54.154 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:42:54.154 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:42:54.154 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:42:54.407 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 26 2024-04-06T09:42:54.407 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 28 2024-04-06T09:42:54.408 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:42:54.408 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=113, tag_tid=18, entry_tid=1], [object_number=112, tag_tid=18, entry_tid=0], [object_number=114, tag_tid=17, entry_tid=20202], [object_number=115, tag_tid=17, entry_tid=20199]]], state=connected] 2024-04-06T09:42:54.408 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=105, tag_tid=17, entry_tid=13385], [object_number=104, tag_tid=17, entry_tid=13384], [object_number=107, tag_tid=17, entry_tid=13383], [object_number=106, tag_tid=17, entry_tid=13382]]], state=connected] 2024-04-06T09:42:54.412 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:42:54.415 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=105, tag_tid=17, entry_tid=13385' 2024-04-06T09:42:54.415 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=113, tag_tid=18, entry_tid=1' -a 'object_number=113, tag_tid=18, entry_tid=1' = 'object_number=105, tag_tid=17, entry_tid=13385' 2024-04-06T09:42:54.415 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=105, tag_tid=17, entry_tid=13385' '!=' 'object_number=104, tag_tid=17, entry_tid=13300' 2024-04-06T09:42:54.415 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:42:54.415 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=105, tag_tid=17, entry_tid=13385' = 'object_number=104, tag_tid=17, entry_tid=13300' 2024-04-06T09:42:54.415 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=105, tag_tid=17, entry_tid=13385' 2024-04-06T09:42:54.416 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=113, tag_tid=18, entry_tid=1' 2024-04-06T09:42:54.416 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:42:54.416 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:42:54.418 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=18 2024-04-06T09:42:54.418 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=105, tag_tid=17, entry_tid=13385' 2024-04-06T09:42:54.419 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:42:54.419 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:42:54.421 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=17 2024-04-06T09:42:54.422 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=113, tag_tid=18, entry_tid=1' 2024-04-06T09:42:54.422 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:42:54.422 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:42:54.423 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:42:54.424 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=105, tag_tid=17, entry_tid=13385' 2024-04-06T09:42:54.424 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:42:54.425 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:42:54.426 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=13385 2024-04-06T09:42:54.426 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 18 = 17 -a 1 -le 13385 2024-04-06T09:42:54.426 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:42:54.426 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:42:54.427 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:42:54.628 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:42:54.628 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:42:54.628 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:42:54.629 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:42:54.629 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:42:54.629 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:42:54.629 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:42:54.629 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:42:54.629 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:42:54.629 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:42:54.629 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:42:54.629 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:42:54.629 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:42:54.629 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:42:54.629 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:42:54.630 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:42:54.630 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:42:54.632 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:42:54.632 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:42:54.632 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:42:54.632 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:42:54.632 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:42:54.632 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:42:54.632 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:42:54.632 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:42:54.632 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:42:54.632 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:42:54.632 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:42:54.632 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:42:54.632 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:42:54.632 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:42:54.633 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:42:54.633 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:42:54.634 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:42:54.634 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:42:54.634 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:42:54.634 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:42:54.634 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:42:54.634 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:42:54.634 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:42:54.634 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:42:54.634 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:42:54.634 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:42:54.635 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:42:54.635 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:42:54.635 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:42:54.635 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:42:54.635 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:42:54.635 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:42:54.659 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:42:54.659 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:42:54.659 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:42:54.659 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:42:54.789 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:42:54.790 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:42:54.790 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:42:54.790 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:42:54.790 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:42:54.790 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:42:54.790 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:42:54.790 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:42:54.790 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:42:54.790 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:42:54.791 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:42:54.791 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:42:54.791 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:42:54.793 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:42:54.793 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:42:54.793 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:42:54.840 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 26 2024-04-06T09:42:54.840 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 28 2024-04-06T09:42:54.840 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:42:54.840 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=113, tag_tid=18, entry_tid=1], [object_number=112, tag_tid=18, entry_tid=0], [object_number=114, tag_tid=17, entry_tid=20202], [object_number=115, tag_tid=17, entry_tid=20199]]], state=connected] 2024-04-06T09:42:54.840 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=107, tag_tid=17, entry_tid=13507], [object_number=106, tag_tid=17, entry_tid=13506], [object_number=105, tag_tid=17, entry_tid=13505], [object_number=104, tag_tid=17, entry_tid=13504]]], state=connected] 2024-04-06T09:42:54.843 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:42:54.845 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=113, tag_tid=18, entry_tid=1' 2024-04-06T09:42:54.846 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:42:54.846 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:42:54.846 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:42:54.846 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:42:54.846 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:42:54.846 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:42:54.846 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:42:54.846 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:42:54.846 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:42:54.847 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:42:54.847 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:42:54.847 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:42:54.847 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:42:54.847 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:42:54.848 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:42:54.894 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 26 2024-04-06T09:42:54.894 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 28 2024-04-06T09:42:54.894 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:42:54.894 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=113, tag_tid=18, entry_tid=1], [object_number=112, tag_tid=18, entry_tid=0], [object_number=114, tag_tid=17, entry_tid=20202], [object_number=115, tag_tid=17, entry_tid=20199]]], state=connected] 2024-04-06T09:42:54.894 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=107, tag_tid=17, entry_tid=13507], [object_number=106, tag_tid=17, entry_tid=13506], [object_number=105, tag_tid=17, entry_tid=13505], [object_number=104, tag_tid=17, entry_tid=13504]]], state=connected] 2024-04-06T09:42:54.898 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:42:54.900 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=107, tag_tid=17, entry_tid=13507' 2024-04-06T09:42:54.900 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=113, tag_tid=18, entry_tid=1' -a 'object_number=113, tag_tid=18, entry_tid=1' = 'object_number=107, tag_tid=17, entry_tid=13507' 2024-04-06T09:42:54.900 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=107, tag_tid=17, entry_tid=13507' '!=' 'object_number=105, tag_tid=17, entry_tid=13385' 2024-04-06T09:42:54.900 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:42:54.900 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=107, tag_tid=17, entry_tid=13507' = 'object_number=105, tag_tid=17, entry_tid=13385' 2024-04-06T09:42:54.900 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=107, tag_tid=17, entry_tid=13507' 2024-04-06T09:42:54.900 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=113, tag_tid=18, entry_tid=1' 2024-04-06T09:42:54.900 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:42:54.900 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:42:54.901 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=18 2024-04-06T09:42:54.902 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=107, tag_tid=17, entry_tid=13507' 2024-04-06T09:42:54.902 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:42:54.902 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:42:54.903 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=17 2024-04-06T09:42:54.903 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=113, tag_tid=18, entry_tid=1' 2024-04-06T09:42:54.903 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:42:54.903 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:42:54.904 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:42:54.905 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=107, tag_tid=17, entry_tid=13507' 2024-04-06T09:42:54.905 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:42:54.905 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:42:54.906 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=13507 2024-04-06T09:42:54.906 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 18 = 17 -a 1 -le 13507 2024-04-06T09:42:54.906 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:42:54.906 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:42:54.907 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:42:55.108 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:42:55.108 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:42:55.108 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:42:55.108 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:42:55.108 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:42:55.108 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:42:55.108 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:42:55.109 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:42:55.109 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:42:55.109 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:42:55.109 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:42:55.109 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:42:55.109 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:42:55.109 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:42:55.109 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:42:55.109 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:42:55.110 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:42:55.112 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:42:55.112 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:42:55.113 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:42:55.113 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:42:55.113 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:42:55.113 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:42:55.113 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:42:55.113 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:42:55.113 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:42:55.113 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:42:55.113 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:42:55.113 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:42:55.113 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:42:55.113 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:42:55.113 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:42:55.113 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:42:55.113 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:42:55.114 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:42:55.114 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:42:55.114 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:42:55.114 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:42:55.114 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:42:55.114 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:42:55.114 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:42:55.114 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:42:55.114 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:42:55.114 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:42:55.114 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:42:55.114 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:42:55.114 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:42:55.114 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:42:55.115 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:42:55.139 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:42:55.139 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:42:55.139 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:42:55.139 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:42:55.271 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:42:55.274 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:42:55.274 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:42:55.274 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:42:55.274 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:42:55.274 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:42:55.274 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:42:55.274 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:42:55.275 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:42:55.275 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:42:55.275 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:42:55.275 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:42:55.275 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:42:55.275 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:42:55.276 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:42:55.276 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:42:55.322 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 26 2024-04-06T09:42:55.323 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 28 2024-04-06T09:42:55.323 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:42:55.323 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=113, tag_tid=18, entry_tid=1], [object_number=112, tag_tid=18, entry_tid=0], [object_number=114, tag_tid=17, entry_tid=20202], [object_number=115, tag_tid=17, entry_tid=20199]]], state=connected] 2024-04-06T09:42:55.323 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=105, tag_tid=17, entry_tid=13601], [object_number=104, tag_tid=17, entry_tid=13600], [object_number=107, tag_tid=17, entry_tid=13599], [object_number=106, tag_tid=17, entry_tid=13598]]], state=connected] 2024-04-06T09:42:55.326 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:42:55.328 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=113, tag_tid=18, entry_tid=1' 2024-04-06T09:42:55.329 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:42:55.329 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:42:55.329 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:42:55.329 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:42:55.329 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:42:55.329 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:42:55.329 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:42:55.329 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:42:55.329 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:42:55.329 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:42:55.329 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:42:55.329 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:42:55.330 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:42:55.330 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:42:55.330 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:42:55.377 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 26 2024-04-06T09:42:55.377 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 28 2024-04-06T09:42:55.377 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:42:55.377 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=113, tag_tid=18, entry_tid=1], [object_number=112, tag_tid=18, entry_tid=0], [object_number=114, tag_tid=17, entry_tid=20202], [object_number=115, tag_tid=17, entry_tid=20199]]], state=connected] 2024-04-06T09:42:55.377 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=105, tag_tid=17, entry_tid=13601], [object_number=104, tag_tid=17, entry_tid=13600], [object_number=107, tag_tid=17, entry_tid=13599], [object_number=106, tag_tid=17, entry_tid=13598]]], state=connected] 2024-04-06T09:42:55.381 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:42:55.382 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=105, tag_tid=17, entry_tid=13601' 2024-04-06T09:42:55.382 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=113, tag_tid=18, entry_tid=1' -a 'object_number=113, tag_tid=18, entry_tid=1' = 'object_number=105, tag_tid=17, entry_tid=13601' 2024-04-06T09:42:55.382 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=105, tag_tid=17, entry_tid=13601' '!=' 'object_number=107, tag_tid=17, entry_tid=13507' 2024-04-06T09:42:55.383 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:42:55.383 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=105, tag_tid=17, entry_tid=13601' = 'object_number=107, tag_tid=17, entry_tid=13507' 2024-04-06T09:42:55.383 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=105, tag_tid=17, entry_tid=13601' 2024-04-06T09:42:55.383 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=113, tag_tid=18, entry_tid=1' 2024-04-06T09:42:55.383 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:42:55.383 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:42:55.384 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=18 2024-04-06T09:42:55.385 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=105, tag_tid=17, entry_tid=13601' 2024-04-06T09:42:55.385 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:42:55.385 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:42:55.386 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=17 2024-04-06T09:42:55.386 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=113, tag_tid=18, entry_tid=1' 2024-04-06T09:42:55.386 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:42:55.387 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:42:55.387 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:42:55.388 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=105, tag_tid=17, entry_tid=13601' 2024-04-06T09:42:55.388 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:42:55.388 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:42:55.389 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=13601 2024-04-06T09:42:55.389 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 18 = 17 -a 1 -le 13601 2024-04-06T09:42:55.389 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:42:55.390 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:42:55.390 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:42:55.591 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:42:55.591 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:42:55.591 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:42:55.591 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:42:55.592 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:42:55.592 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:42:55.592 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:42:55.592 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:42:55.592 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:42:55.592 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:42:55.592 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:42:55.592 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:42:55.592 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:42:55.592 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:42:55.592 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:42:55.593 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:42:55.593 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:42:55.596 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:42:55.597 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:42:55.597 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:42:55.597 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:42:55.597 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:42:55.597 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:42:55.597 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:42:55.597 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:42:55.597 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:42:55.597 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:42:55.597 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:42:55.597 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:42:55.597 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:42:55.597 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:42:55.598 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:42:55.598 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:42:55.598 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:42:55.598 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:42:55.598 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:42:55.598 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:42:55.598 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:42:55.598 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:42:55.598 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:42:55.598 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:42:55.598 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:42:55.598 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:42:55.599 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:42:55.599 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:42:55.599 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:42:55.599 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:42:55.599 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:42:55.599 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:42:55.622 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:42:55.622 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:42:55.622 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:42:55.622 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:42:55.749 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:42:55.750 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:42:55.750 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:42:55.751 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:42:55.751 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:42:55.751 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:42:55.751 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:42:55.751 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:42:55.751 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:42:55.751 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:42:55.751 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:42:55.752 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:42:55.752 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:42:55.753 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:42:55.753 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:42:55.754 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:42:55.805 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 26 2024-04-06T09:42:55.806 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 28 2024-04-06T09:42:55.806 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:42:55.806 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=113, tag_tid=18, entry_tid=1], [object_number=112, tag_tid=18, entry_tid=0], [object_number=114, tag_tid=17, entry_tid=20202], [object_number=115, tag_tid=17, entry_tid=20199]]], state=connected] 2024-04-06T09:42:55.807 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=105, tag_tid=17, entry_tid=13685], [object_number=104, tag_tid=17, entry_tid=13684], [object_number=107, tag_tid=17, entry_tid=13683], [object_number=106, tag_tid=17, entry_tid=13682]]], state=connected] 2024-04-06T09:42:55.809 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:42:55.810 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=113, tag_tid=18, entry_tid=1' 2024-04-06T09:42:55.812 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:42:55.812 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:42:55.812 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:42:55.812 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:42:55.812 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:42:55.812 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:42:55.812 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:42:55.812 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:42:55.812 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:42:55.812 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:42:55.812 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:42:55.812 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:42:55.813 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:42:55.813 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:42:55.813 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:42:55.860 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 26 2024-04-06T09:42:55.860 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 28 2024-04-06T09:42:55.860 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:42:55.861 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=113, tag_tid=18, entry_tid=1], [object_number=112, tag_tid=18, entry_tid=0], [object_number=114, tag_tid=17, entry_tid=20202], [object_number=115, tag_tid=17, entry_tid=20199]]], state=connected] 2024-04-06T09:42:55.861 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=105, tag_tid=17, entry_tid=13685], [object_number=104, tag_tid=17, entry_tid=13684], [object_number=107, tag_tid=17, entry_tid=13683], [object_number=106, tag_tid=17, entry_tid=13682]]], state=connected] 2024-04-06T09:42:55.864 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:42:55.866 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=105, tag_tid=17, entry_tid=13685' 2024-04-06T09:42:55.866 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=113, tag_tid=18, entry_tid=1' -a 'object_number=113, tag_tid=18, entry_tid=1' = 'object_number=105, tag_tid=17, entry_tid=13685' 2024-04-06T09:42:55.866 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=105, tag_tid=17, entry_tid=13685' '!=' 'object_number=105, tag_tid=17, entry_tid=13601' 2024-04-06T09:42:55.866 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:42:55.866 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=105, tag_tid=17, entry_tid=13685' = 'object_number=105, tag_tid=17, entry_tid=13601' 2024-04-06T09:42:55.866 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=105, tag_tid=17, entry_tid=13685' 2024-04-06T09:42:55.866 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=113, tag_tid=18, entry_tid=1' 2024-04-06T09:42:55.866 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:42:55.867 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:42:55.867 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=18 2024-04-06T09:42:55.868 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=105, tag_tid=17, entry_tid=13685' 2024-04-06T09:42:55.868 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:42:55.868 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:42:55.869 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=17 2024-04-06T09:42:55.869 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=113, tag_tid=18, entry_tid=1' 2024-04-06T09:42:55.869 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:42:55.869 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:42:55.870 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:42:55.871 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=105, tag_tid=17, entry_tid=13685' 2024-04-06T09:42:55.871 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:42:55.871 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:42:55.872 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=13685 2024-04-06T09:42:55.872 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 18 = 17 -a 1 -le 13685 2024-04-06T09:42:55.872 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:42:55.873 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:42:55.873 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:42:56.074 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:42:56.075 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:42:56.075 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:42:56.075 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:42:56.075 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:42:56.075 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:42:56.075 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:42:56.075 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:42:56.075 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:42:56.075 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:42:56.075 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:42:56.075 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:42:56.075 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:42:56.075 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:42:56.076 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:42:56.076 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:42:56.076 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:42:56.079 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:42:56.079 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:42:56.079 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:42:56.079 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:42:56.079 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:42:56.079 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:42:56.079 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:42:56.079 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:42:56.079 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:42:56.079 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:42:56.080 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:42:56.080 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:42:56.080 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:42:56.080 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:42:56.080 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:42:56.080 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:42:56.081 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:42:56.081 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:42:56.081 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:42:56.081 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:42:56.081 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:42:56.081 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:42:56.081 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:42:56.082 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:42:56.082 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:42:56.082 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:42:56.082 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:42:56.082 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:42:56.082 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:42:56.082 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:42:56.082 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:42:56.082 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:42:56.106 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:42:56.106 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:42:56.107 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:42:56.107 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:42:56.242 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:42:56.242 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:42:56.243 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:42:56.243 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:42:56.243 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:42:56.243 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:42:56.243 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:42:56.243 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:42:56.243 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:42:56.243 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:42:56.244 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:42:56.244 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:42:56.244 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:42:56.246 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:42:56.247 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:42:56.247 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:42:56.299 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 26 2024-04-06T09:42:56.299 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 28 2024-04-06T09:42:56.300 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:42:56.300 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=113, tag_tid=18, entry_tid=1], [object_number=112, tag_tid=18, entry_tid=0], [object_number=114, tag_tid=17, entry_tid=20202], [object_number=115, tag_tid=17, entry_tid=20199]]], state=connected] 2024-04-06T09:42:56.300 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=107, tag_tid=17, entry_tid=13779], [object_number=106, tag_tid=17, entry_tid=13778], [object_number=105, tag_tid=17, entry_tid=13777], [object_number=104, tag_tid=17, entry_tid=13776]]], state=connected] 2024-04-06T09:42:56.307 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:42:56.307 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=113, tag_tid=18, entry_tid=1' 2024-04-06T09:42:56.307 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:42:56.308 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:42:56.308 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:42:56.308 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:42:56.308 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:42:56.308 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:42:56.308 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:42:56.308 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:42:56.308 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:42:56.308 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:42:56.308 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:42:56.308 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:42:56.308 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:42:56.308 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:42:56.309 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:42:56.355 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 26 2024-04-06T09:42:56.356 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 28 2024-04-06T09:42:56.356 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:42:56.356 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=113, tag_tid=18, entry_tid=1], [object_number=112, tag_tid=18, entry_tid=0], [object_number=114, tag_tid=17, entry_tid=20202], [object_number=115, tag_tid=17, entry_tid=20199]]], state=connected] 2024-04-06T09:42:56.356 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=107, tag_tid=17, entry_tid=13779], [object_number=106, tag_tid=17, entry_tid=13778], [object_number=105, tag_tid=17, entry_tid=13777], [object_number=104, tag_tid=17, entry_tid=13776]]], state=connected] 2024-04-06T09:42:56.359 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:42:56.361 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=107, tag_tid=17, entry_tid=13779' 2024-04-06T09:42:56.361 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=113, tag_tid=18, entry_tid=1' -a 'object_number=113, tag_tid=18, entry_tid=1' = 'object_number=107, tag_tid=17, entry_tid=13779' 2024-04-06T09:42:56.361 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=107, tag_tid=17, entry_tid=13779' '!=' 'object_number=105, tag_tid=17, entry_tid=13685' 2024-04-06T09:42:56.361 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:42:56.361 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=107, tag_tid=17, entry_tid=13779' = 'object_number=105, tag_tid=17, entry_tid=13685' 2024-04-06T09:42:56.362 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=107, tag_tid=17, entry_tid=13779' 2024-04-06T09:42:56.362 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=113, tag_tid=18, entry_tid=1' 2024-04-06T09:42:56.362 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:42:56.362 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:42:56.362 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=18 2024-04-06T09:42:56.363 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=107, tag_tid=17, entry_tid=13779' 2024-04-06T09:42:56.363 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:42:56.363 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:42:56.364 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=17 2024-04-06T09:42:56.365 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=113, tag_tid=18, entry_tid=1' 2024-04-06T09:42:56.365 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:42:56.365 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:42:56.366 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:42:56.366 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=107, tag_tid=17, entry_tid=13779' 2024-04-06T09:42:56.367 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:42:56.367 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:42:56.368 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=13779 2024-04-06T09:42:56.368 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 18 = 17 -a 1 -le 13779 2024-04-06T09:42:56.368 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:42:56.368 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:42:56.368 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:42:56.569 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:42:56.570 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:42:56.570 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:42:56.570 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:42:56.570 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:42:56.570 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:42:56.570 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:42:56.570 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:42:56.570 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:42:56.570 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:42:56.570 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:42:56.570 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:42:56.571 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:42:56.571 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:42:56.571 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:42:56.571 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:42:56.571 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:42:56.575 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:42:56.575 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:42:56.575 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:42:56.576 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:42:56.576 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:42:56.576 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:42:56.576 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:42:56.576 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:42:56.576 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:42:56.576 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:42:56.576 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:42:56.576 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:42:56.577 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:42:56.577 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:42:56.577 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:42:56.577 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:42:56.577 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:42:56.577 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:42:56.577 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:42:56.577 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:42:56.577 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:42:56.577 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:42:56.577 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:42:56.577 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:42:56.578 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:42:56.578 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:42:56.578 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:42:56.578 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:42:56.578 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:42:56.578 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:42:56.578 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:42:56.578 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:42:56.600 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:42:56.600 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:42:56.600 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:42:56.600 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:42:56.723 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:42:56.725 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:42:56.725 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:42:56.725 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:42:56.725 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:42:56.725 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:42:56.725 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:42:56.725 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:42:56.726 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:42:56.726 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:42:56.726 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:42:56.726 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:42:56.726 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:42:56.727 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:42:56.728 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:42:56.728 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:42:56.784 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 26 2024-04-06T09:42:56.784 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 28 2024-04-06T09:42:56.784 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:42:56.784 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=113, tag_tid=18, entry_tid=1], [object_number=112, tag_tid=18, entry_tid=0], [object_number=114, tag_tid=17, entry_tid=20202], [object_number=115, tag_tid=17, entry_tid=20199]]], state=connected] 2024-04-06T09:42:56.784 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=104, tag_tid=17, entry_tid=13872], [object_number=107, tag_tid=17, entry_tid=13871], [object_number=106, tag_tid=17, entry_tid=13870], [object_number=105, tag_tid=17, entry_tid=13869]]], state=connected] 2024-04-06T09:42:56.788 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:42:56.789 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=113, tag_tid=18, entry_tid=1' 2024-04-06T09:42:56.790 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:42:56.790 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:42:56.790 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:42:56.790 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:42:56.791 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:42:56.791 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:42:56.791 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:42:56.791 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:42:56.791 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:42:56.791 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:42:56.791 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:42:56.791 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:42:56.792 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:42:56.792 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:42:56.792 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:42:56.839 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 26 2024-04-06T09:42:56.839 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 28 2024-04-06T09:42:56.840 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:42:56.840 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=113, tag_tid=18, entry_tid=1], [object_number=112, tag_tid=18, entry_tid=0], [object_number=114, tag_tid=17, entry_tid=20202], [object_number=115, tag_tid=17, entry_tid=20199]]], state=connected] 2024-04-06T09:42:56.840 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=104, tag_tid=17, entry_tid=13872], [object_number=107, tag_tid=17, entry_tid=13871], [object_number=106, tag_tid=17, entry_tid=13870], [object_number=105, tag_tid=17, entry_tid=13869]]], state=connected] 2024-04-06T09:42:56.843 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:42:56.845 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=104, tag_tid=17, entry_tid=13872' 2024-04-06T09:42:56.845 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=113, tag_tid=18, entry_tid=1' -a 'object_number=113, tag_tid=18, entry_tid=1' = 'object_number=104, tag_tid=17, entry_tid=13872' 2024-04-06T09:42:56.845 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=104, tag_tid=17, entry_tid=13872' '!=' 'object_number=107, tag_tid=17, entry_tid=13779' 2024-04-06T09:42:56.846 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:42:56.846 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=104, tag_tid=17, entry_tid=13872' = 'object_number=107, tag_tid=17, entry_tid=13779' 2024-04-06T09:42:56.846 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=104, tag_tid=17, entry_tid=13872' 2024-04-06T09:42:56.846 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=113, tag_tid=18, entry_tid=1' 2024-04-06T09:42:56.846 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:42:56.846 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:42:56.846 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=18 2024-04-06T09:42:56.847 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=104, tag_tid=17, entry_tid=13872' 2024-04-06T09:42:56.847 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:42:56.847 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:42:56.848 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=17 2024-04-06T09:42:56.849 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=113, tag_tid=18, entry_tid=1' 2024-04-06T09:42:56.849 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:42:56.849 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:42:56.850 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:42:56.850 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=104, tag_tid=17, entry_tid=13872' 2024-04-06T09:42:56.851 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:42:56.851 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:42:56.852 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=13872 2024-04-06T09:42:56.852 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 18 = 17 -a 1 -le 13872 2024-04-06T09:42:56.852 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:42:56.852 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:42:56.852 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:42:57.054 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:42:57.054 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:42:57.054 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:42:57.054 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:42:57.054 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:42:57.054 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:42:57.054 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:42:57.054 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:42:57.055 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:42:57.055 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:42:57.055 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:42:57.055 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:42:57.055 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:42:57.055 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:42:57.055 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:42:57.055 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:42:57.056 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:42:57.057 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:42:57.058 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:42:57.058 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:42:57.058 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:42:57.058 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:42:57.058 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:42:57.058 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:42:57.058 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:42:57.058 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:42:57.058 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:42:57.058 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:42:57.058 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:42:57.058 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:42:57.058 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:42:57.058 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:42:57.059 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:42:57.059 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:42:57.059 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:42:57.059 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:42:57.060 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:42:57.060 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:42:57.060 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:42:57.060 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:42:57.060 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:42:57.060 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:42:57.060 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:42:57.060 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:42:57.060 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:42:57.060 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:42:57.060 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:42:57.060 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:42:57.060 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:42:57.085 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:42:57.085 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:42:57.085 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:42:57.086 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:42:57.215 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:42:57.216 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:42:57.216 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:42:57.216 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:42:57.216 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:42:57.216 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:42:57.216 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:42:57.216 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:42:57.217 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:42:57.217 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:42:57.217 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:42:57.217 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:42:57.217 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:42:57.219 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:42:57.219 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:42:57.219 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:42:57.266 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 26 2024-04-06T09:42:57.266 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 28 2024-04-06T09:42:57.266 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:42:57.266 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=113, tag_tid=18, entry_tid=1], [object_number=112, tag_tid=18, entry_tid=0], [object_number=114, tag_tid=17, entry_tid=20202], [object_number=115, tag_tid=17, entry_tid=20199]]], state=connected] 2024-04-06T09:42:57.266 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=105, tag_tid=17, entry_tid=13977], [object_number=104, tag_tid=17, entry_tid=13976], [object_number=107, tag_tid=17, entry_tid=13975], [object_number=106, tag_tid=17, entry_tid=13974]]], state=connected] 2024-04-06T09:42:57.269 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:42:57.271 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=113, tag_tid=18, entry_tid=1' 2024-04-06T09:42:57.272 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:42:57.272 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:42:57.272 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:42:57.272 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:42:57.272 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:42:57.272 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:42:57.272 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:42:57.272 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:42:57.272 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:42:57.273 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:42:57.273 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:42:57.273 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:42:57.273 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:42:57.273 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:42:57.274 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:42:57.319 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 26 2024-04-06T09:42:57.320 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 28 2024-04-06T09:42:57.320 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:42:57.320 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=113, tag_tid=18, entry_tid=1], [object_number=112, tag_tid=18, entry_tid=0], [object_number=114, tag_tid=17, entry_tid=20202], [object_number=115, tag_tid=17, entry_tid=20199]]], state=connected] 2024-04-06T09:42:57.320 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=105, tag_tid=17, entry_tid=13977], [object_number=104, tag_tid=17, entry_tid=13976], [object_number=107, tag_tid=17, entry_tid=13975], [object_number=106, tag_tid=17, entry_tid=13974]]], state=connected] 2024-04-06T09:42:57.323 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:42:57.325 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=105, tag_tid=17, entry_tid=13977' 2024-04-06T09:42:57.325 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=113, tag_tid=18, entry_tid=1' -a 'object_number=113, tag_tid=18, entry_tid=1' = 'object_number=105, tag_tid=17, entry_tid=13977' 2024-04-06T09:42:57.325 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=105, tag_tid=17, entry_tid=13977' '!=' 'object_number=104, tag_tid=17, entry_tid=13872' 2024-04-06T09:42:57.325 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:42:57.325 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=105, tag_tid=17, entry_tid=13977' = 'object_number=104, tag_tid=17, entry_tid=13872' 2024-04-06T09:42:57.326 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=105, tag_tid=17, entry_tid=13977' 2024-04-06T09:42:57.326 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=113, tag_tid=18, entry_tid=1' 2024-04-06T09:42:57.326 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:42:57.326 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:42:57.327 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=18 2024-04-06T09:42:57.327 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=105, tag_tid=17, entry_tid=13977' 2024-04-06T09:42:57.327 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:42:57.327 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:42:57.330 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=17 2024-04-06T09:42:57.330 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=113, tag_tid=18, entry_tid=1' 2024-04-06T09:42:57.330 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:42:57.330 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:42:57.331 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:42:57.331 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=105, tag_tid=17, entry_tid=13977' 2024-04-06T09:42:57.331 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:42:57.331 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:42:57.332 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=13977 2024-04-06T09:42:57.332 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 18 = 17 -a 1 -le 13977 2024-04-06T09:42:57.332 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:42:57.333 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:42:57.333 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:42:57.535 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:42:57.535 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:42:57.535 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:42:57.535 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:42:57.536 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:42:57.536 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:42:57.536 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:42:57.536 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:42:57.536 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:42:57.536 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:42:57.536 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:42:57.536 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:42:57.536 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:42:57.536 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:42:57.536 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:42:57.536 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:42:57.537 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:42:57.538 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:42:57.538 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:42:57.538 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:42:57.538 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:42:57.538 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:42:57.538 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:42:57.539 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:42:57.539 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:42:57.539 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:42:57.539 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:42:57.539 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:42:57.539 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:42:57.539 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:42:57.539 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:42:57.539 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:42:57.539 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:42:57.540 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:42:57.540 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:42:57.540 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:42:57.540 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:42:57.540 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:42:57.541 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:42:57.541 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:42:57.541 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:42:57.541 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:42:57.541 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:42:57.541 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:42:57.541 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:42:57.541 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:42:57.541 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:42:57.541 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:42:57.541 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:42:57.566 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:42:57.566 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:42:57.567 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:42:57.567 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:42:57.695 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:42:57.696 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:42:57.696 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:42:57.697 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:42:57.697 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:42:57.697 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:42:57.697 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:42:57.697 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:42:57.697 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:42:57.697 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:42:57.697 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:42:57.698 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:42:57.698 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:42:57.699 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:42:57.699 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:42:57.699 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:42:57.959 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 26 2024-04-06T09:42:57.959 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 28 2024-04-06T09:42:57.959 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:42:57.959 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=113, tag_tid=18, entry_tid=1], [object_number=112, tag_tid=18, entry_tid=0], [object_number=114, tag_tid=17, entry_tid=20202], [object_number=115, tag_tid=17, entry_tid=20199]]], state=connected] 2024-04-06T09:42:57.959 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=107, tag_tid=17, entry_tid=14083], [object_number=106, tag_tid=17, entry_tid=14082], [object_number=105, tag_tid=17, entry_tid=14081], [object_number=104, tag_tid=17, entry_tid=14080]]], state=connected] 2024-04-06T09:42:57.963 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:42:57.965 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=113, tag_tid=18, entry_tid=1' 2024-04-06T09:42:57.966 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:42:57.966 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:42:57.966 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:42:57.966 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:42:57.966 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:42:57.966 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:42:57.966 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:42:57.967 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:42:57.967 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:42:57.967 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:42:57.967 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:42:57.967 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:42:57.967 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:42:57.968 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:42:57.968 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:42:58.018 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 26 2024-04-06T09:42:58.018 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 28 2024-04-06T09:42:58.018 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:42:58.018 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=113, tag_tid=18, entry_tid=1], [object_number=112, tag_tid=18, entry_tid=0], [object_number=114, tag_tid=17, entry_tid=20202], [object_number=115, tag_tid=17, entry_tid=20199]]], state=connected] 2024-04-06T09:42:58.018 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=107, tag_tid=17, entry_tid=14083], [object_number=106, tag_tid=17, entry_tid=14082], [object_number=105, tag_tid=17, entry_tid=14081], [object_number=104, tag_tid=17, entry_tid=14080]]], state=connected] 2024-04-06T09:42:58.022 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:42:58.025 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=107, tag_tid=17, entry_tid=14083' 2024-04-06T09:42:58.025 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=113, tag_tid=18, entry_tid=1' -a 'object_number=113, tag_tid=18, entry_tid=1' = 'object_number=107, tag_tid=17, entry_tid=14083' 2024-04-06T09:42:58.025 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=107, tag_tid=17, entry_tid=14083' '!=' 'object_number=105, tag_tid=17, entry_tid=13977' 2024-04-06T09:42:58.025 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:42:58.025 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=107, tag_tid=17, entry_tid=14083' = 'object_number=105, tag_tid=17, entry_tid=13977' 2024-04-06T09:42:58.025 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=107, tag_tid=17, entry_tid=14083' 2024-04-06T09:42:58.025 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=113, tag_tid=18, entry_tid=1' 2024-04-06T09:42:58.025 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:42:58.025 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:42:58.026 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=18 2024-04-06T09:42:58.026 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=107, tag_tid=17, entry_tid=14083' 2024-04-06T09:42:58.027 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:42:58.027 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:42:58.028 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=17 2024-04-06T09:42:58.028 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=113, tag_tid=18, entry_tid=1' 2024-04-06T09:42:58.028 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:42:58.029 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:42:58.029 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:42:58.030 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=107, tag_tid=17, entry_tid=14083' 2024-04-06T09:42:58.030 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:42:58.030 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:42:58.031 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=14083 2024-04-06T09:42:58.032 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 18 = 17 -a 1 -le 14083 2024-04-06T09:42:58.032 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:42:58.032 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:42:58.032 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:42:58.233 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:42:58.233 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:42:58.233 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:42:58.234 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:42:58.234 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:42:58.234 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:42:58.234 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:42:58.234 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:42:58.234 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:42:58.234 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:42:58.234 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:42:58.234 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:42:58.234 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:42:58.234 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:42:58.234 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:42:58.235 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:42:58.235 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:42:58.237 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:42:58.237 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:42:58.238 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:42:58.238 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:42:58.238 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:42:58.238 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:42:58.238 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:42:58.238 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:42:58.238 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:42:58.238 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:42:58.238 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:42:58.238 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:42:58.238 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:42:58.238 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:42:58.238 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:42:58.238 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:42:58.239 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:42:58.239 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:42:58.239 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:42:58.239 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:42:58.239 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:42:58.239 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:42:58.239 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:42:58.239 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:42:58.239 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:42:58.239 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:42:58.239 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:42:58.239 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:42:58.239 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:42:58.239 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:42:58.239 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:42:58.240 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:42:58.266 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:42:58.266 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:42:58.266 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:42:58.267 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:42:58.397 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:42:58.399 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:42:58.399 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:42:58.399 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:42:58.399 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:42:58.399 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:42:58.399 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:42:58.400 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:42:58.400 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:42:58.400 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:42:58.400 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:42:58.400 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:42:58.400 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:42:58.401 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:42:58.402 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:42:58.402 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:42:58.453 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 26 2024-04-06T09:42:58.454 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 28 2024-04-06T09:42:58.454 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:42:58.454 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=113, tag_tid=18, entry_tid=1], [object_number=112, tag_tid=18, entry_tid=0], [object_number=114, tag_tid=17, entry_tid=20202], [object_number=115, tag_tid=17, entry_tid=20199]]], state=connected] 2024-04-06T09:42:58.454 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=106, tag_tid=17, entry_tid=14214], [object_number=105, tag_tid=17, entry_tid=14213], [object_number=104, tag_tid=17, entry_tid=14212], [object_number=107, tag_tid=17, entry_tid=14211]]], state=connected] 2024-04-06T09:42:58.457 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:42:58.458 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=113, tag_tid=18, entry_tid=1' 2024-04-06T09:42:58.459 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:42:58.460 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:42:58.460 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:42:58.460 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:42:58.460 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:42:58.460 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:42:58.460 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:42:58.460 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:42:58.460 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:42:58.460 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:42:58.460 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:42:58.460 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:42:58.461 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:42:58.461 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:42:58.461 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:42:58.507 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 26 2024-04-06T09:42:58.507 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 28 2024-04-06T09:42:58.507 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:42:58.508 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=113, tag_tid=18, entry_tid=1], [object_number=112, tag_tid=18, entry_tid=0], [object_number=114, tag_tid=17, entry_tid=20202], [object_number=115, tag_tid=17, entry_tid=20199]]], state=connected] 2024-04-06T09:42:58.508 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=106, tag_tid=17, entry_tid=14214], [object_number=105, tag_tid=17, entry_tid=14213], [object_number=104, tag_tid=17, entry_tid=14212], [object_number=107, tag_tid=17, entry_tid=14211]]], state=connected] 2024-04-06T09:42:58.511 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:42:58.513 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=106, tag_tid=17, entry_tid=14214' 2024-04-06T09:42:58.513 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=113, tag_tid=18, entry_tid=1' -a 'object_number=113, tag_tid=18, entry_tid=1' = 'object_number=106, tag_tid=17, entry_tid=14214' 2024-04-06T09:42:58.513 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=106, tag_tid=17, entry_tid=14214' '!=' 'object_number=107, tag_tid=17, entry_tid=14083' 2024-04-06T09:42:58.513 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:42:58.513 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=106, tag_tid=17, entry_tid=14214' = 'object_number=107, tag_tid=17, entry_tid=14083' 2024-04-06T09:42:58.514 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=106, tag_tid=17, entry_tid=14214' 2024-04-06T09:42:58.514 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=113, tag_tid=18, entry_tid=1' 2024-04-06T09:42:58.514 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:42:58.514 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:42:58.514 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=18 2024-04-06T09:42:58.515 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=106, tag_tid=17, entry_tid=14214' 2024-04-06T09:42:58.515 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:42:58.515 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:42:58.516 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=17 2024-04-06T09:42:58.517 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=113, tag_tid=18, entry_tid=1' 2024-04-06T09:42:58.517 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:42:58.517 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:42:58.518 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:42:58.519 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=106, tag_tid=17, entry_tid=14214' 2024-04-06T09:42:58.519 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:42:58.519 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:42:58.520 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=14214 2024-04-06T09:42:58.520 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 18 = 17 -a 1 -le 14214 2024-04-06T09:42:58.520 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:42:58.520 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:42:58.520 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:42:58.722 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:42:58.722 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:42:58.722 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:42:58.722 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:42:58.722 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:42:58.723 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:42:58.723 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:42:58.723 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:42:58.723 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:42:58.723 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:42:58.723 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:42:58.723 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:42:58.723 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:42:58.723 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:42:58.723 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:42:58.724 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:42:58.724 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:42:58.727 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:42:58.727 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:42:58.727 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:42:58.727 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:42:58.728 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:42:58.728 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:42:58.728 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:42:58.728 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:42:58.728 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:42:58.728 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:42:58.728 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:42:58.728 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:42:58.728 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:42:58.728 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:42:58.728 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:42:58.728 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:42:58.728 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:42:58.728 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:42:58.729 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:42:58.729 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:42:58.729 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:42:58.729 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:42:58.729 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:42:58.729 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:42:58.729 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:42:58.729 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:42:58.729 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:42:58.729 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:42:58.729 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:42:58.729 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:42:58.729 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:42:58.729 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:42:58.752 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:42:58.752 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:42:58.752 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:42:58.753 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:42:58.885 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:42:58.886 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:42:58.886 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:42:58.886 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:42:58.886 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:42:58.886 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:42:58.886 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:42:58.887 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:42:58.887 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:42:58.887 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:42:58.887 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:42:58.887 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:42:58.887 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:42:58.888 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:42:58.889 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:42:58.889 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:42:58.945 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 26 2024-04-06T09:42:58.946 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 28 2024-04-06T09:42:58.946 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:42:58.946 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=113, tag_tid=18, entry_tid=1], [object_number=112, tag_tid=18, entry_tid=0], [object_number=114, tag_tid=17, entry_tid=20202], [object_number=115, tag_tid=17, entry_tid=20199]]], state=connected] 2024-04-06T09:42:58.946 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=107, tag_tid=17, entry_tid=14295], [object_number=106, tag_tid=17, entry_tid=14294], [object_number=105, tag_tid=17, entry_tid=14293], [object_number=104, tag_tid=17, entry_tid=14292]]], state=connected] 2024-04-06T09:42:58.949 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:42:58.950 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=113, tag_tid=18, entry_tid=1' 2024-04-06T09:42:58.952 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:42:58.952 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:42:58.952 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:42:58.952 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:42:58.952 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:42:58.952 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:42:58.952 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:42:58.952 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:42:58.952 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:42:58.952 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:42:58.952 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:42:58.952 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:42:58.953 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:42:58.953 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:42:58.953 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:42:59.000 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 26 2024-04-06T09:42:59.000 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 28 2024-04-06T09:42:59.001 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:42:59.001 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=113, tag_tid=18, entry_tid=1], [object_number=112, tag_tid=18, entry_tid=0], [object_number=114, tag_tid=17, entry_tid=20202], [object_number=115, tag_tid=17, entry_tid=20199]]], state=connected] 2024-04-06T09:42:59.001 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=107, tag_tid=17, entry_tid=14295], [object_number=106, tag_tid=17, entry_tid=14294], [object_number=105, tag_tid=17, entry_tid=14293], [object_number=104, tag_tid=17, entry_tid=14292]]], state=connected] 2024-04-06T09:42:59.004 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:42:59.006 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=107, tag_tid=17, entry_tid=14295' 2024-04-06T09:42:59.006 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=113, tag_tid=18, entry_tid=1' -a 'object_number=113, tag_tid=18, entry_tid=1' = 'object_number=107, tag_tid=17, entry_tid=14295' 2024-04-06T09:42:59.006 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=107, tag_tid=17, entry_tid=14295' '!=' 'object_number=106, tag_tid=17, entry_tid=14214' 2024-04-06T09:42:59.006 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:42:59.006 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=107, tag_tid=17, entry_tid=14295' = 'object_number=106, tag_tid=17, entry_tid=14214' 2024-04-06T09:42:59.006 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=107, tag_tid=17, entry_tid=14295' 2024-04-06T09:42:59.007 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=113, tag_tid=18, entry_tid=1' 2024-04-06T09:42:59.007 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:42:59.007 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:42:59.007 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=18 2024-04-06T09:42:59.008 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=107, tag_tid=17, entry_tid=14295' 2024-04-06T09:42:59.008 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:42:59.008 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:42:59.009 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=17 2024-04-06T09:42:59.010 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=113, tag_tid=18, entry_tid=1' 2024-04-06T09:42:59.010 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:42:59.010 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:42:59.010 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:42:59.011 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=107, tag_tid=17, entry_tid=14295' 2024-04-06T09:42:59.011 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:42:59.011 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:42:59.012 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=14295 2024-04-06T09:42:59.012 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 18 = 17 -a 1 -le 14295 2024-04-06T09:42:59.013 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:42:59.013 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:42:59.013 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:42:59.214 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:42:59.214 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:42:59.214 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:42:59.214 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:42:59.214 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:42:59.215 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:42:59.215 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:42:59.215 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:42:59.215 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:42:59.215 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:42:59.215 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:42:59.215 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:42:59.215 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:42:59.215 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:42:59.215 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:42:59.216 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:42:59.216 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:42:59.217 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:42:59.217 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:42:59.217 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:42:59.217 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:42:59.217 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:42:59.217 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:42:59.218 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:42:59.218 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:42:59.218 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:42:59.218 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:42:59.218 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:42:59.218 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:42:59.218 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:42:59.218 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:42:59.218 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:42:59.218 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:42:59.219 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:42:59.219 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:42:59.219 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:42:59.219 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:42:59.219 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:42:59.219 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:42:59.219 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:42:59.220 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:42:59.220 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:42:59.220 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:42:59.220 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:42:59.220 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:42:59.220 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:42:59.220 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:42:59.220 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:42:59.220 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:42:59.244 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:42:59.244 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:42:59.244 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:42:59.244 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:42:59.463 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:42:59.464 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:42:59.464 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:42:59.464 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:42:59.464 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:42:59.464 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:42:59.465 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:42:59.465 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:42:59.465 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:42:59.465 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:42:59.465 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:42:59.465 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:42:59.465 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:42:59.465 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:42:59.466 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:42:59.466 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:42:59.512 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 27 2024-04-06T09:42:59.512 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 28 2024-04-06T09:42:59.512 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:42:59.512 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=113, tag_tid=18, entry_tid=1], [object_number=112, tag_tid=18, entry_tid=0], [object_number=114, tag_tid=17, entry_tid=20202], [object_number=115, tag_tid=17, entry_tid=20199]]], state=connected] 2024-04-06T09:42:59.512 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=108, tag_tid=17, entry_tid=14336], [object_number=111, tag_tid=17, entry_tid=14335], [object_number=110, tag_tid=17, entry_tid=14334], [object_number=109, tag_tid=17, entry_tid=14333]]], state=connected] 2024-04-06T09:42:59.516 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:42:59.517 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=113, tag_tid=18, entry_tid=1' 2024-04-06T09:42:59.518 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:42:59.518 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:42:59.518 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:42:59.518 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:42:59.518 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:42:59.518 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:42:59.518 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:42:59.518 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:42:59.519 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:42:59.519 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:42:59.519 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:42:59.519 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:42:59.519 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:42:59.519 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:42:59.520 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:42:59.567 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 27 2024-04-06T09:42:59.567 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 28 2024-04-06T09:42:59.567 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:42:59.567 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=113, tag_tid=18, entry_tid=1], [object_number=112, tag_tid=18, entry_tid=0], [object_number=114, tag_tid=17, entry_tid=20202], [object_number=115, tag_tid=17, entry_tid=20199]]], state=connected] 2024-04-06T09:42:59.567 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=108, tag_tid=17, entry_tid=14336], [object_number=111, tag_tid=17, entry_tid=14335], [object_number=110, tag_tid=17, entry_tid=14334], [object_number=109, tag_tid=17, entry_tid=14333]]], state=connected] 2024-04-06T09:42:59.570 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:42:59.572 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=108, tag_tid=17, entry_tid=14336' 2024-04-06T09:42:59.572 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=113, tag_tid=18, entry_tid=1' -a 'object_number=113, tag_tid=18, entry_tid=1' = 'object_number=108, tag_tid=17, entry_tid=14336' 2024-04-06T09:42:59.572 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=108, tag_tid=17, entry_tid=14336' '!=' 'object_number=107, tag_tid=17, entry_tid=14295' 2024-04-06T09:42:59.573 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:42:59.573 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=108, tag_tid=17, entry_tid=14336' = 'object_number=107, tag_tid=17, entry_tid=14295' 2024-04-06T09:42:59.573 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=108, tag_tid=17, entry_tid=14336' 2024-04-06T09:42:59.573 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=113, tag_tid=18, entry_tid=1' 2024-04-06T09:42:59.573 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:42:59.573 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:42:59.573 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=18 2024-04-06T09:42:59.574 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=108, tag_tid=17, entry_tid=14336' 2024-04-06T09:42:59.574 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:42:59.574 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:42:59.575 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=17 2024-04-06T09:42:59.576 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=113, tag_tid=18, entry_tid=1' 2024-04-06T09:42:59.576 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:42:59.576 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:42:59.577 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:42:59.577 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=108, tag_tid=17, entry_tid=14336' 2024-04-06T09:42:59.578 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:42:59.578 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:42:59.579 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=14336 2024-04-06T09:42:59.579 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 18 = 17 -a 1 -le 14336 2024-04-06T09:42:59.579 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:42:59.579 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:42:59.579 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:42:59.781 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:42:59.781 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:42:59.781 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:42:59.781 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:42:59.781 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:42:59.781 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:42:59.781 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:42:59.781 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:42:59.781 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:42:59.781 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:42:59.781 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:42:59.782 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:42:59.782 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:42:59.782 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:42:59.782 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:42:59.782 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:42:59.782 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:42:59.785 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:42:59.785 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:42:59.785 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:42:59.785 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:42:59.785 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:42:59.785 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:42:59.785 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:42:59.785 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:42:59.785 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:42:59.786 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:42:59.786 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:42:59.786 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:42:59.786 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:42:59.786 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:42:59.786 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:42:59.786 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:42:59.790 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:42:59.790 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:42:59.790 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:42:59.790 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:42:59.790 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:42:59.790 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:42:59.790 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:42:59.791 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:42:59.791 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:42:59.791 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:42:59.791 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:42:59.791 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:42:59.791 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:42:59.791 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:42:59.791 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:42:59.791 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:42:59.813 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:42:59.814 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:42:59.814 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:42:59.814 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:42:59.940 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:42:59.941 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:42:59.941 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:42:59.941 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:42:59.941 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:42:59.941 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:42:59.942 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:42:59.942 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:42:59.942 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:42:59.942 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:42:59.942 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:42:59.943 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:42:59.943 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:42:59.945 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:42:59.945 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:42:59.945 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:43:00.003 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 27 2024-04-06T09:43:00.003 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 28 2024-04-06T09:43:00.003 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:43:00.003 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=113, tag_tid=18, entry_tid=1], [object_number=112, tag_tid=18, entry_tid=0], [object_number=114, tag_tid=17, entry_tid=20202], [object_number=115, tag_tid=17, entry_tid=20199]]], state=connected] 2024-04-06T09:43:00.003 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=111, tag_tid=17, entry_tid=14435], [object_number=110, tag_tid=17, entry_tid=14434], [object_number=109, tag_tid=17, entry_tid=14433], [object_number=108, tag_tid=17, entry_tid=14432]]], state=connected] 2024-04-06T09:43:00.006 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:43:00.007 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=113, tag_tid=18, entry_tid=1' 2024-04-06T09:43:00.009 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:43:00.009 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:43:00.009 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:43:00.009 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:43:00.009 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:43:00.009 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:43:00.009 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:43:00.009 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:43:00.009 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:43:00.009 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:43:00.009 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:43:00.009 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:43:00.010 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:43:00.010 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:43:00.010 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:43:00.056 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 27 2024-04-06T09:43:00.056 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 28 2024-04-06T09:43:00.057 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:43:00.057 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=113, tag_tid=18, entry_tid=1], [object_number=112, tag_tid=18, entry_tid=0], [object_number=114, tag_tid=17, entry_tid=20202], [object_number=115, tag_tid=17, entry_tid=20199]]], state=connected] 2024-04-06T09:43:00.057 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=111, tag_tid=17, entry_tid=14435], [object_number=110, tag_tid=17, entry_tid=14434], [object_number=109, tag_tid=17, entry_tid=14433], [object_number=108, tag_tid=17, entry_tid=14432]]], state=connected] 2024-04-06T09:43:00.060 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:43:00.061 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=111, tag_tid=17, entry_tid=14435' 2024-04-06T09:43:00.062 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=113, tag_tid=18, entry_tid=1' -a 'object_number=113, tag_tid=18, entry_tid=1' = 'object_number=111, tag_tid=17, entry_tid=14435' 2024-04-06T09:43:00.062 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=111, tag_tid=17, entry_tid=14435' '!=' 'object_number=108, tag_tid=17, entry_tid=14336' 2024-04-06T09:43:00.062 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:43:00.062 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=111, tag_tid=17, entry_tid=14435' = 'object_number=108, tag_tid=17, entry_tid=14336' 2024-04-06T09:43:00.062 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=111, tag_tid=17, entry_tid=14435' 2024-04-06T09:43:00.062 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=113, tag_tid=18, entry_tid=1' 2024-04-06T09:43:00.063 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:43:00.063 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:43:00.063 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=18 2024-04-06T09:43:00.064 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=111, tag_tid=17, entry_tid=14435' 2024-04-06T09:43:00.064 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:43:00.064 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:43:00.065 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=17 2024-04-06T09:43:00.065 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=113, tag_tid=18, entry_tid=1' 2024-04-06T09:43:00.066 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:43:00.066 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:43:00.066 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:43:00.067 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=111, tag_tid=17, entry_tid=14435' 2024-04-06T09:43:00.067 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:43:00.067 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:43:00.068 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=14435 2024-04-06T09:43:00.068 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 18 = 17 -a 1 -le 14435 2024-04-06T09:43:00.069 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:43:00.069 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:43:00.069 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:43:00.270 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:43:00.271 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:43:00.271 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:43:00.271 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:43:00.271 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:43:00.271 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:43:00.271 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:43:00.271 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:43:00.271 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:43:00.271 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:43:00.271 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:43:00.271 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:43:00.271 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:43:00.271 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:43:00.272 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:43:00.272 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:43:00.272 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:43:00.276 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:43:00.276 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:43:00.276 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:43:00.276 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:43:00.276 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:43:00.276 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:43:00.276 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:43:00.277 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:43:00.277 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:43:00.277 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:43:00.277 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:43:00.277 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:43:00.277 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:43:00.277 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:43:00.277 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:43:00.277 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:43:00.277 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:43:00.277 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:43:00.277 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:43:00.278 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:43:00.278 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:43:00.278 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:43:00.278 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:43:00.278 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:43:00.278 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:43:00.278 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:43:00.278 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:43:00.278 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:43:00.278 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:43:00.278 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:43:00.278 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:43:00.278 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:43:00.300 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:43:00.300 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:43:00.301 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:43:00.301 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:43:00.442 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:43:00.443 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:43:00.443 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:43:00.443 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:43:00.443 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:43:00.443 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:43:00.444 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:43:00.444 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:43:00.444 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:43:00.444 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:43:00.445 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:43:00.445 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:43:00.445 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:43:00.446 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:43:00.446 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:43:00.447 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:43:00.492 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 27 2024-04-06T09:43:00.492 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 28 2024-04-06T09:43:00.492 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:43:00.492 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=113, tag_tid=18, entry_tid=1], [object_number=112, tag_tid=18, entry_tid=0], [object_number=114, tag_tid=17, entry_tid=20202], [object_number=115, tag_tid=17, entry_tid=20199]]], state=connected] 2024-04-06T09:43:00.492 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=108, tag_tid=17, entry_tid=14528], [object_number=111, tag_tid=17, entry_tid=14527], [object_number=110, tag_tid=17, entry_tid=14526], [object_number=109, tag_tid=17, entry_tid=14525]]], state=connected] 2024-04-06T09:43:00.496 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:43:00.497 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=113, tag_tid=18, entry_tid=1' 2024-04-06T09:43:00.498 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:43:00.498 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:43:00.498 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:43:00.498 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:43:00.498 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:43:00.499 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:43:00.499 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:43:00.499 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:43:00.499 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:43:00.499 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:43:00.499 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:43:00.499 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:43:00.500 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:43:00.500 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:43:00.500 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:43:00.547 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 27 2024-04-06T09:43:00.548 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 28 2024-04-06T09:43:00.548 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:43:00.548 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=113, tag_tid=18, entry_tid=1], [object_number=112, tag_tid=18, entry_tid=0], [object_number=114, tag_tid=17, entry_tid=20202], [object_number=115, tag_tid=17, entry_tid=20199]]], state=connected] 2024-04-06T09:43:00.548 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=108, tag_tid=17, entry_tid=14528], [object_number=111, tag_tid=17, entry_tid=14527], [object_number=110, tag_tid=17, entry_tid=14526], [object_number=109, tag_tid=17, entry_tid=14525]]], state=connected] 2024-04-06T09:43:00.551 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:43:00.553 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=108, tag_tid=17, entry_tid=14528' 2024-04-06T09:43:00.553 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=113, tag_tid=18, entry_tid=1' -a 'object_number=113, tag_tid=18, entry_tid=1' = 'object_number=108, tag_tid=17, entry_tid=14528' 2024-04-06T09:43:00.553 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=108, tag_tid=17, entry_tid=14528' '!=' 'object_number=111, tag_tid=17, entry_tid=14435' 2024-04-06T09:43:00.553 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:43:00.553 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=108, tag_tid=17, entry_tid=14528' = 'object_number=111, tag_tid=17, entry_tid=14435' 2024-04-06T09:43:00.553 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=108, tag_tid=17, entry_tid=14528' 2024-04-06T09:43:00.553 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=113, tag_tid=18, entry_tid=1' 2024-04-06T09:43:00.554 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:43:00.554 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:43:00.554 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=18 2024-04-06T09:43:00.555 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=108, tag_tid=17, entry_tid=14528' 2024-04-06T09:43:00.555 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:43:00.555 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:43:00.556 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=17 2024-04-06T09:43:00.557 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=113, tag_tid=18, entry_tid=1' 2024-04-06T09:43:00.557 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:43:00.557 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:43:00.558 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:43:00.558 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=108, tag_tid=17, entry_tid=14528' 2024-04-06T09:43:00.559 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:43:00.559 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:43:00.560 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=14528 2024-04-06T09:43:00.560 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 18 = 17 -a 1 -le 14528 2024-04-06T09:43:00.560 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:43:00.560 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:43:00.560 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:43:00.762 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:43:00.762 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:43:00.762 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:43:00.762 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:43:00.762 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:43:00.762 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:43:00.762 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:43:00.763 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:43:00.763 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:43:00.763 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:43:00.763 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:43:00.763 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:43:00.763 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:43:00.763 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:43:00.763 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:43:00.767 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:43:00.767 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:43:00.767 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:43:00.767 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:43:00.767 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:43:00.767 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:43:00.767 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:43:00.767 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:43:00.767 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:43:00.767 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:43:00.767 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:43:00.768 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:43:00.768 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:43:00.768 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:43:00.768 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:43:00.768 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:43:00.768 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:43:00.768 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:43:00.768 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:43:00.768 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:43:00.768 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:43:00.769 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:43:00.769 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:43:00.769 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:43:00.769 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:43:00.769 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:43:00.769 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:43:00.769 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:43:00.769 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:43:00.769 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:43:00.769 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:43:00.769 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:43:00.770 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:43:00.770 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:43:00.791 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:43:00.792 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:43:00.792 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:43:00.792 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:43:00.922 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:43:00.923 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:43:00.923 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:43:00.923 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:43:00.923 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:43:00.923 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:43:00.923 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:43:00.923 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:43:00.923 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:43:00.924 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:43:00.924 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:43:00.924 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:43:00.924 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:43:00.925 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:43:00.926 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:43:00.926 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:43:00.978 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 27 2024-04-06T09:43:00.978 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 28 2024-04-06T09:43:00.978 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:43:00.978 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=113, tag_tid=18, entry_tid=1], [object_number=112, tag_tid=18, entry_tid=0], [object_number=114, tag_tid=17, entry_tid=20202], [object_number=115, tag_tid=17, entry_tid=20199]]], state=connected] 2024-04-06T09:43:00.978 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=110, tag_tid=17, entry_tid=14618], [object_number=109, tag_tid=17, entry_tid=14617], [object_number=108, tag_tid=17, entry_tid=14616], [object_number=111, tag_tid=17, entry_tid=14615]]], state=connected] 2024-04-06T09:43:00.981 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:43:00.983 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=113, tag_tid=18, entry_tid=1' 2024-04-06T09:43:00.984 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:43:00.984 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:43:00.984 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:43:00.985 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:43:00.985 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:43:00.985 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:43:00.985 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:43:00.985 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:43:00.985 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:43:00.985 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:43:00.985 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:43:00.985 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:43:00.986 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:43:00.986 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:43:00.986 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:43:01.032 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 27 2024-04-06T09:43:01.033 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 28 2024-04-06T09:43:01.033 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:43:01.033 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=113, tag_tid=18, entry_tid=1], [object_number=112, tag_tid=18, entry_tid=0], [object_number=114, tag_tid=17, entry_tid=20202], [object_number=115, tag_tid=17, entry_tid=20199]]], state=connected] 2024-04-06T09:43:01.033 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=110, tag_tid=17, entry_tid=14618], [object_number=109, tag_tid=17, entry_tid=14617], [object_number=108, tag_tid=17, entry_tid=14616], [object_number=111, tag_tid=17, entry_tid=14615]]], state=connected] 2024-04-06T09:43:01.036 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:43:01.039 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=110, tag_tid=17, entry_tid=14618' 2024-04-06T09:43:01.039 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=113, tag_tid=18, entry_tid=1' -a 'object_number=113, tag_tid=18, entry_tid=1' = 'object_number=110, tag_tid=17, entry_tid=14618' 2024-04-06T09:43:01.039 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=110, tag_tid=17, entry_tid=14618' '!=' 'object_number=108, tag_tid=17, entry_tid=14528' 2024-04-06T09:43:01.039 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:43:01.039 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=110, tag_tid=17, entry_tid=14618' = 'object_number=108, tag_tid=17, entry_tid=14528' 2024-04-06T09:43:01.039 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=110, tag_tid=17, entry_tid=14618' 2024-04-06T09:43:01.039 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=113, tag_tid=18, entry_tid=1' 2024-04-06T09:43:01.039 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:43:01.039 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:43:01.040 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=18 2024-04-06T09:43:01.040 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=110, tag_tid=17, entry_tid=14618' 2024-04-06T09:43:01.041 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:43:01.041 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:43:01.041 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=17 2024-04-06T09:43:01.042 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=113, tag_tid=18, entry_tid=1' 2024-04-06T09:43:01.042 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:43:01.042 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:43:01.043 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:43:01.044 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=110, tag_tid=17, entry_tid=14618' 2024-04-06T09:43:01.044 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:43:01.044 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:43:01.045 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=14618 2024-04-06T09:43:01.045 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 18 = 17 -a 1 -le 14618 2024-04-06T09:43:01.045 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:43:01.046 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:43:01.046 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:43:01.247 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:43:01.247 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:43:01.247 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:43:01.247 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:43:01.247 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:43:01.247 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:43:01.247 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:43:01.247 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:43:01.247 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:43:01.247 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:43:01.248 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:43:01.248 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:43:01.248 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:43:01.248 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:43:01.248 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:43:01.248 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:43:01.249 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:43:01.253 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:43:01.253 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:43:01.253 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:43:01.253 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:43:01.253 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:43:01.253 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:43:01.253 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:43:01.253 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:43:01.253 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:43:01.253 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:43:01.253 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:43:01.254 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:43:01.254 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:43:01.254 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:43:01.254 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:43:01.254 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:43:01.254 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:43:01.254 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:43:01.254 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:43:01.254 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:43:01.254 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:43:01.254 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:43:01.254 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:43:01.254 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:43:01.254 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:43:01.254 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:43:01.255 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:43:01.255 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:43:01.255 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:43:01.255 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:43:01.255 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:43:01.255 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:43:01.279 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:43:01.279 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:43:01.280 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:43:01.280 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:43:01.412 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:43:01.414 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:43:01.414 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:43:01.414 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:43:01.414 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:43:01.414 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:43:01.414 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:43:01.414 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:43:01.414 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:43:01.414 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:43:01.414 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:43:01.415 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:43:01.415 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:43:01.417 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:43:01.417 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:43:01.417 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:43:01.463 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 27 2024-04-06T09:43:01.463 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 28 2024-04-06T09:43:01.463 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:43:01.463 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=113, tag_tid=18, entry_tid=1], [object_number=112, tag_tid=18, entry_tid=0], [object_number=114, tag_tid=17, entry_tid=20202], [object_number=115, tag_tid=17, entry_tid=20199]]], state=connected] 2024-04-06T09:43:01.463 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=111, tag_tid=17, entry_tid=14711], [object_number=110, tag_tid=17, entry_tid=14710], [object_number=109, tag_tid=17, entry_tid=14709], [object_number=108, tag_tid=17, entry_tid=14708]]], state=connected] 2024-04-06T09:43:01.466 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:43:01.468 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=113, tag_tid=18, entry_tid=1' 2024-04-06T09:43:01.469 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:43:01.469 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:43:01.469 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:43:01.469 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:43:01.469 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:43:01.469 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:43:01.469 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:43:01.470 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:43:01.470 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:43:01.470 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:43:01.470 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:43:01.470 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:43:01.470 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:43:01.471 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:43:01.471 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:43:01.520 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 27 2024-04-06T09:43:01.520 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 28 2024-04-06T09:43:01.520 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:43:01.520 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=113, tag_tid=18, entry_tid=1], [object_number=112, tag_tid=18, entry_tid=0], [object_number=114, tag_tid=17, entry_tid=20202], [object_number=115, tag_tid=17, entry_tid=20199]]], state=connected] 2024-04-06T09:43:01.521 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=111, tag_tid=17, entry_tid=14711], [object_number=110, tag_tid=17, entry_tid=14710], [object_number=109, tag_tid=17, entry_tid=14709], [object_number=108, tag_tid=17, entry_tid=14708]]], state=connected] 2024-04-06T09:43:01.524 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:43:01.526 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=111, tag_tid=17, entry_tid=14711' 2024-04-06T09:43:01.526 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=113, tag_tid=18, entry_tid=1' -a 'object_number=113, tag_tid=18, entry_tid=1' = 'object_number=111, tag_tid=17, entry_tid=14711' 2024-04-06T09:43:01.526 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=111, tag_tid=17, entry_tid=14711' '!=' 'object_number=110, tag_tid=17, entry_tid=14618' 2024-04-06T09:43:01.526 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:43:01.526 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=111, tag_tid=17, entry_tid=14711' = 'object_number=110, tag_tid=17, entry_tid=14618' 2024-04-06T09:43:01.526 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=111, tag_tid=17, entry_tid=14711' 2024-04-06T09:43:01.526 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=113, tag_tid=18, entry_tid=1' 2024-04-06T09:43:01.527 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:43:01.527 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:43:01.527 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=18 2024-04-06T09:43:01.528 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=111, tag_tid=17, entry_tid=14711' 2024-04-06T09:43:01.528 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:43:01.528 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:43:01.528 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=17 2024-04-06T09:43:01.529 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=113, tag_tid=18, entry_tid=1' 2024-04-06T09:43:01.529 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:43:01.529 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:43:01.530 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:43:01.531 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=111, tag_tid=17, entry_tid=14711' 2024-04-06T09:43:01.531 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:43:01.531 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:43:01.532 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=14711 2024-04-06T09:43:01.532 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 18 = 17 -a 1 -le 14711 2024-04-06T09:43:01.532 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:43:01.532 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:43:01.533 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:43:01.734 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:43:01.734 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:43:01.735 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:43:01.735 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:43:01.735 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:43:01.735 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:43:01.735 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:43:01.735 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:43:01.735 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:43:01.735 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:43:01.735 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:43:01.735 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:43:01.735 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:43:01.736 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:43:01.736 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:43:01.736 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:43:01.736 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:43:01.740 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:43:01.740 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:43:01.740 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:43:01.740 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:43:01.740 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:43:01.740 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:43:01.740 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:43:01.740 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:43:01.740 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:43:01.741 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:43:01.741 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:43:01.741 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:43:01.741 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:43:01.741 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:43:01.741 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:43:01.741 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:43:01.741 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:43:01.741 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:43:01.741 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:43:01.741 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:43:01.741 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:43:01.741 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:43:01.742 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:43:01.742 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:43:01.742 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:43:01.742 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:43:01.742 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:43:01.742 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:43:01.742 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:43:01.742 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:43:01.742 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:43:01.742 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:43:01.767 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:43:01.767 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:43:01.768 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:43:01.768 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:43:01.893 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:43:01.895 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:43:01.895 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:43:01.895 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:43:01.895 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:43:01.895 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:43:01.895 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:43:01.895 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:43:01.895 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:43:01.895 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:43:01.895 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:43:01.896 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:43:01.896 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:43:01.898 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:43:01.898 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:43:01.898 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:43:01.950 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 27 2024-04-06T09:43:01.950 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 28 2024-04-06T09:43:01.950 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:43:01.951 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=113, tag_tid=18, entry_tid=1], [object_number=112, tag_tid=18, entry_tid=0], [object_number=114, tag_tid=17, entry_tid=20202], [object_number=115, tag_tid=17, entry_tid=20199]]], state=connected] 2024-04-06T09:43:01.951 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=111, tag_tid=17, entry_tid=14811], [object_number=110, tag_tid=17, entry_tid=14810], [object_number=109, tag_tid=17, entry_tid=14809], [object_number=108, tag_tid=17, entry_tid=14808]]], state=connected] 2024-04-06T09:43:01.954 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:43:01.955 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=113, tag_tid=18, entry_tid=1' 2024-04-06T09:43:01.957 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:43:01.957 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:43:01.957 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:43:01.957 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:43:01.957 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:43:01.957 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:43:01.957 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:43:01.957 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:43:01.957 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:43:01.957 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:43:01.957 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:43:01.958 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:43:01.958 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:43:01.958 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:43:01.958 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:43:02.007 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 27 2024-04-06T09:43:02.007 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 28 2024-04-06T09:43:02.007 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:43:02.007 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=113, tag_tid=18, entry_tid=1], [object_number=112, tag_tid=18, entry_tid=0], [object_number=114, tag_tid=17, entry_tid=20202], [object_number=115, tag_tid=17, entry_tid=20199]]], state=connected] 2024-04-06T09:43:02.007 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=111, tag_tid=17, entry_tid=14811], [object_number=110, tag_tid=17, entry_tid=14810], [object_number=109, tag_tid=17, entry_tid=14809], [object_number=108, tag_tid=17, entry_tid=14808]]], state=connected] 2024-04-06T09:43:02.010 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:43:02.012 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=111, tag_tid=17, entry_tid=14811' 2024-04-06T09:43:02.012 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=113, tag_tid=18, entry_tid=1' -a 'object_number=113, tag_tid=18, entry_tid=1' = 'object_number=111, tag_tid=17, entry_tid=14811' 2024-04-06T09:43:02.012 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=111, tag_tid=17, entry_tid=14811' '!=' 'object_number=111, tag_tid=17, entry_tid=14711' 2024-04-06T09:43:02.013 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:43:02.013 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=111, tag_tid=17, entry_tid=14811' = 'object_number=111, tag_tid=17, entry_tid=14711' 2024-04-06T09:43:02.013 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=111, tag_tid=17, entry_tid=14811' 2024-04-06T09:43:02.013 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=113, tag_tid=18, entry_tid=1' 2024-04-06T09:43:02.013 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:43:02.013 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:43:02.014 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=18 2024-04-06T09:43:02.014 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=111, tag_tid=17, entry_tid=14811' 2024-04-06T09:43:02.014 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:43:02.014 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:43:02.015 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=17 2024-04-06T09:43:02.016 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=113, tag_tid=18, entry_tid=1' 2024-04-06T09:43:02.016 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:43:02.016 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:43:02.017 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:43:02.018 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=111, tag_tid=17, entry_tid=14811' 2024-04-06T09:43:02.018 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:43:02.018 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:43:02.019 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=14811 2024-04-06T09:43:02.019 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 18 = 17 -a 1 -le 14811 2024-04-06T09:43:02.019 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:43:02.019 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:43:02.019 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:43:02.221 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:43:02.222 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:43:02.222 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:43:02.222 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:43:02.222 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:43:02.222 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:43:02.222 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:43:02.223 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:43:02.223 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:43:02.223 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:43:02.223 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:43:02.223 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:43:02.223 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:43:02.223 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:43:02.223 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:43:02.224 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:43:02.224 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:43:02.225 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:43:02.225 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:43:02.225 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:43:02.225 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:43:02.225 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:43:02.226 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:43:02.226 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:43:02.226 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:43:02.226 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:43:02.226 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:43:02.226 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:43:02.226 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:43:02.226 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:43:02.227 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:43:02.227 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:43:02.227 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:43:02.228 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:43:02.228 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:43:02.228 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:43:02.228 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:43:02.228 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:43:02.228 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:43:02.228 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:43:02.228 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:43:02.228 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:43:02.228 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:43:02.228 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:43:02.228 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:43:02.228 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:43:02.229 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:43:02.229 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:43:02.229 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:43:02.254 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:43:02.255 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:43:02.255 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:43:02.255 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:43:02.384 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:43:02.385 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:43:02.385 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:43:02.385 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:43:02.385 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:43:02.385 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:43:02.385 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:43:02.385 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:43:02.385 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:43:02.385 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:43:02.386 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:43:02.386 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:43:02.386 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:43:02.389 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:43:02.389 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:43:02.389 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:43:02.437 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 27 2024-04-06T09:43:02.437 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 28 2024-04-06T09:43:02.437 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:43:02.437 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=113, tag_tid=18, entry_tid=1], [object_number=112, tag_tid=18, entry_tid=0], [object_number=114, tag_tid=17, entry_tid=20202], [object_number=115, tag_tid=17, entry_tid=20199]]], state=connected] 2024-04-06T09:43:02.437 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=109, tag_tid=17, entry_tid=14913], [object_number=108, tag_tid=17, entry_tid=14912], [object_number=111, tag_tid=17, entry_tid=14911], [object_number=110, tag_tid=17, entry_tid=14910]]], state=connected] 2024-04-06T09:43:02.440 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:43:02.442 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=113, tag_tid=18, entry_tid=1' 2024-04-06T09:43:02.443 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:43:02.443 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:43:02.443 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:43:02.443 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:43:02.443 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:43:02.443 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:43:02.443 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:43:02.443 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:43:02.443 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:43:02.444 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:43:02.444 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:43:02.444 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:43:02.444 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:43:02.444 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:43:02.444 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:43:02.492 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 27 2024-04-06T09:43:02.492 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 28 2024-04-06T09:43:02.492 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:43:02.492 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=113, tag_tid=18, entry_tid=1], [object_number=112, tag_tid=18, entry_tid=0], [object_number=114, tag_tid=17, entry_tid=20202], [object_number=115, tag_tid=17, entry_tid=20199]]], state=connected] 2024-04-06T09:43:02.492 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=109, tag_tid=17, entry_tid=14913], [object_number=108, tag_tid=17, entry_tid=14912], [object_number=111, tag_tid=17, entry_tid=14911], [object_number=110, tag_tid=17, entry_tid=14910]]], state=connected] 2024-04-06T09:43:02.496 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:43:02.498 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=109, tag_tid=17, entry_tid=14913' 2024-04-06T09:43:02.498 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=113, tag_tid=18, entry_tid=1' -a 'object_number=113, tag_tid=18, entry_tid=1' = 'object_number=109, tag_tid=17, entry_tid=14913' 2024-04-06T09:43:02.498 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=109, tag_tid=17, entry_tid=14913' '!=' 'object_number=111, tag_tid=17, entry_tid=14811' 2024-04-06T09:43:02.498 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:43:02.498 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=109, tag_tid=17, entry_tid=14913' = 'object_number=111, tag_tid=17, entry_tid=14811' 2024-04-06T09:43:02.498 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=109, tag_tid=17, entry_tid=14913' 2024-04-06T09:43:02.498 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=113, tag_tid=18, entry_tid=1' 2024-04-06T09:43:02.499 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:43:02.499 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:43:02.499 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=18 2024-04-06T09:43:02.500 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=109, tag_tid=17, entry_tid=14913' 2024-04-06T09:43:02.500 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:43:02.500 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:43:02.501 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=17 2024-04-06T09:43:02.502 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=113, tag_tid=18, entry_tid=1' 2024-04-06T09:43:02.502 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:43:02.502 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:43:02.503 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:43:02.504 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=109, tag_tid=17, entry_tid=14913' 2024-04-06T09:43:02.504 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:43:02.504 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:43:02.505 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=14913 2024-04-06T09:43:02.505 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 18 = 17 -a 1 -le 14913 2024-04-06T09:43:02.505 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:43:02.505 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:43:02.505 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:43:02.707 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:43:02.707 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:43:02.707 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:43:02.707 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:43:02.707 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:43:02.707 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:43:02.707 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:43:02.707 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:43:02.708 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:43:02.708 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:43:02.708 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:43:02.708 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:43:02.708 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:43:02.708 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:43:02.708 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:43:02.708 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:43:02.709 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:43:02.710 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:43:02.711 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:43:02.711 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:43:02.711 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:43:02.711 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:43:02.711 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:43:02.711 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:43:02.711 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:43:02.711 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:43:02.711 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:43:02.711 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:43:02.711 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:43:02.711 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:43:02.711 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:43:02.712 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:43:02.712 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:43:02.713 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:43:02.713 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:43:02.713 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:43:02.713 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:43:02.713 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:43:02.713 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:43:02.713 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:43:02.714 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:43:02.714 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:43:02.714 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:43:02.714 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:43:02.714 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:43:02.714 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:43:02.714 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:43:02.714 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:43:02.714 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:43:02.739 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:43:02.739 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:43:02.739 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:43:02.739 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:43:02.872 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:43:02.873 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:43:02.873 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:43:02.873 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:43:02.873 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:43:02.873 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:43:02.873 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:43:02.873 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:43:02.873 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:43:02.874 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:43:02.874 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:43:02.874 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:43:02.874 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:43:02.875 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:43:02.875 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:43:02.875 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:43:03.134 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 27 2024-04-06T09:43:03.135 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 28 2024-04-06T09:43:03.135 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:43:03.135 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=113, tag_tid=18, entry_tid=1], [object_number=112, tag_tid=18, entry_tid=0], [object_number=114, tag_tid=17, entry_tid=20202], [object_number=115, tag_tid=17, entry_tid=20199]]], state=connected] 2024-04-06T09:43:03.135 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=108, tag_tid=17, entry_tid=15000], [object_number=111, tag_tid=17, entry_tid=14999], [object_number=110, tag_tid=17, entry_tid=14998], [object_number=109, tag_tid=17, entry_tid=14997]]], state=connected] 2024-04-06T09:43:03.139 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:43:03.140 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=113, tag_tid=18, entry_tid=1' 2024-04-06T09:43:03.141 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:43:03.141 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:43:03.142 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:43:03.142 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:43:03.142 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:43:03.142 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:43:03.142 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:43:03.142 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:43:03.142 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:43:03.142 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:43:03.142 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:43:03.142 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:43:03.143 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:43:03.143 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:43:03.143 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:43:03.194 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 27 2024-04-06T09:43:03.195 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 28 2024-04-06T09:43:03.195 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:43:03.195 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=113, tag_tid=18, entry_tid=1], [object_number=112, tag_tid=18, entry_tid=0], [object_number=114, tag_tid=17, entry_tid=20202], [object_number=115, tag_tid=17, entry_tid=20199]]], state=connected] 2024-04-06T09:43:03.195 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=108, tag_tid=17, entry_tid=15000], [object_number=111, tag_tid=17, entry_tid=14999], [object_number=110, tag_tid=17, entry_tid=14998], [object_number=109, tag_tid=17, entry_tid=14997]]], state=connected] 2024-04-06T09:43:03.199 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:43:03.201 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=108, tag_tid=17, entry_tid=15000' 2024-04-06T09:43:03.201 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=113, tag_tid=18, entry_tid=1' -a 'object_number=113, tag_tid=18, entry_tid=1' = 'object_number=108, tag_tid=17, entry_tid=15000' 2024-04-06T09:43:03.201 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=108, tag_tid=17, entry_tid=15000' '!=' 'object_number=109, tag_tid=17, entry_tid=14913' 2024-04-06T09:43:03.201 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:43:03.201 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=108, tag_tid=17, entry_tid=15000' = 'object_number=109, tag_tid=17, entry_tid=14913' 2024-04-06T09:43:03.201 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=108, tag_tid=17, entry_tid=15000' 2024-04-06T09:43:03.201 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=113, tag_tid=18, entry_tid=1' 2024-04-06T09:43:03.202 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:43:03.202 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:43:03.202 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=18 2024-04-06T09:43:03.203 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=108, tag_tid=17, entry_tid=15000' 2024-04-06T09:43:03.203 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:43:03.203 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:43:03.204 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=17 2024-04-06T09:43:03.205 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=113, tag_tid=18, entry_tid=1' 2024-04-06T09:43:03.205 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:43:03.205 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:43:03.206 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:43:03.207 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=108, tag_tid=17, entry_tid=15000' 2024-04-06T09:43:03.207 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:43:03.207 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:43:03.208 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=15000 2024-04-06T09:43:03.208 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 18 = 17 -a 1 -le 15000 2024-04-06T09:43:03.208 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:43:03.208 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:43:03.208 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:43:03.410 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:43:03.410 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:43:03.410 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:43:03.410 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:43:03.411 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:43:03.411 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:43:03.411 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:43:03.411 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:43:03.411 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:43:03.411 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:43:03.411 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:43:03.411 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:43:03.411 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:43:03.411 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:43:03.411 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:43:03.412 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:43:03.412 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:43:03.415 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:43:03.415 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:43:03.415 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:43:03.416 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:43:03.416 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:43:03.416 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:43:03.416 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:43:03.416 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:43:03.416 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:43:03.416 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:43:03.416 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:43:03.416 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:43:03.416 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:43:03.416 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:43:03.416 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:43:03.416 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:43:03.416 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:43:03.417 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:43:03.417 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:43:03.417 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:43:03.417 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:43:03.417 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:43:03.417 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:43:03.417 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:43:03.417 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:43:03.417 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:43:03.417 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:43:03.417 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:43:03.417 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:43:03.417 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:43:03.417 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:43:03.417 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:43:03.440 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:43:03.441 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:43:03.441 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:43:03.441 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:43:03.575 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:43:03.576 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:43:03.576 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:43:03.576 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:43:03.577 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:43:03.577 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:43:03.577 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:43:03.577 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:43:03.577 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:43:03.577 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:43:03.577 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:43:03.577 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:43:03.578 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:43:03.579 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:43:03.579 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:43:03.579 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:43:03.627 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 27 2024-04-06T09:43:03.627 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 28 2024-04-06T09:43:03.627 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:43:03.627 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=113, tag_tid=18, entry_tid=1], [object_number=112, tag_tid=18, entry_tid=0], [object_number=114, tag_tid=17, entry_tid=20202], [object_number=115, tag_tid=17, entry_tid=20199]]], state=connected] 2024-04-06T09:43:03.627 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=110, tag_tid=17, entry_tid=15114], [object_number=109, tag_tid=17, entry_tid=15113], [object_number=108, tag_tid=17, entry_tid=15112], [object_number=111, tag_tid=17, entry_tid=15111]]], state=connected] 2024-04-06T09:43:03.630 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:43:03.632 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=113, tag_tid=18, entry_tid=1' 2024-04-06T09:43:03.633 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:43:03.633 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:43:03.633 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:43:03.633 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:43:03.633 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:43:03.633 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:43:03.633 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:43:03.633 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:43:03.634 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:43:03.634 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:43:03.634 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:43:03.634 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:43:03.634 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:43:03.634 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:43:03.635 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:43:03.681 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 27 2024-04-06T09:43:03.681 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 28 2024-04-06T09:43:03.681 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:43:03.682 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=113, tag_tid=18, entry_tid=1], [object_number=112, tag_tid=18, entry_tid=0], [object_number=114, tag_tid=17, entry_tid=20202], [object_number=115, tag_tid=17, entry_tid=20199]]], state=connected] 2024-04-06T09:43:03.682 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=110, tag_tid=17, entry_tid=15114], [object_number=109, tag_tid=17, entry_tid=15113], [object_number=108, tag_tid=17, entry_tid=15112], [object_number=111, tag_tid=17, entry_tid=15111]]], state=connected] 2024-04-06T09:43:03.685 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:43:03.687 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=110, tag_tid=17, entry_tid=15114' 2024-04-06T09:43:03.687 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=113, tag_tid=18, entry_tid=1' -a 'object_number=113, tag_tid=18, entry_tid=1' = 'object_number=110, tag_tid=17, entry_tid=15114' 2024-04-06T09:43:03.687 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=110, tag_tid=17, entry_tid=15114' '!=' 'object_number=108, tag_tid=17, entry_tid=15000' 2024-04-06T09:43:03.687 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:43:03.687 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=110, tag_tid=17, entry_tid=15114' = 'object_number=108, tag_tid=17, entry_tid=15000' 2024-04-06T09:43:03.687 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=110, tag_tid=17, entry_tid=15114' 2024-04-06T09:43:03.688 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=113, tag_tid=18, entry_tid=1' 2024-04-06T09:43:03.688 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:43:03.688 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:43:03.688 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=18 2024-04-06T09:43:03.689 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=110, tag_tid=17, entry_tid=15114' 2024-04-06T09:43:03.689 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:43:03.689 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:43:03.690 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=17 2024-04-06T09:43:03.691 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=113, tag_tid=18, entry_tid=1' 2024-04-06T09:43:03.691 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:43:03.691 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:43:03.692 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:43:03.692 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=110, tag_tid=17, entry_tid=15114' 2024-04-06T09:43:03.692 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:43:03.693 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:43:03.694 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=15114 2024-04-06T09:43:03.694 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 18 = 17 -a 1 -le 15114 2024-04-06T09:43:03.694 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:43:03.694 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:43:03.694 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:43:03.895 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:43:03.895 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:43:03.896 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:43:03.896 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:43:03.896 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:43:03.896 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:43:03.896 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:43:03.896 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:43:03.896 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:43:03.896 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:43:03.896 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:43:03.896 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:43:03.896 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:43:03.896 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:43:03.896 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:43:03.897 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:43:03.897 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:43:03.900 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:43:03.901 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:43:03.901 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:43:03.901 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:43:03.901 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:43:03.901 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:43:03.901 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:43:03.901 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:43:03.901 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:43:03.901 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:43:03.901 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:43:03.901 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:43:03.901 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:43:03.902 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:43:03.902 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:43:03.902 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:43:03.902 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:43:03.902 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:43:03.902 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:43:03.902 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:43:03.902 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:43:03.902 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:43:03.902 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:43:03.903 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:43:03.903 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:43:03.903 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:43:03.903 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:43:03.903 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:43:03.903 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:43:03.903 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:43:03.903 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:43:03.903 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:43:03.926 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:43:03.926 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:43:03.926 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:43:03.926 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:43:04.052 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:43:04.053 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:43:04.053 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:43:04.053 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:43:04.053 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:43:04.053 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:43:04.053 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:43:04.053 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:43:04.053 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:43:04.054 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:43:04.054 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:43:04.054 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:43:04.054 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:43:04.057 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:43:04.057 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:43:04.057 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:43:04.115 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 27 2024-04-06T09:43:04.115 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 28 2024-04-06T09:43:04.115 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:43:04.115 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=113, tag_tid=18, entry_tid=1], [object_number=112, tag_tid=18, entry_tid=0], [object_number=114, tag_tid=17, entry_tid=20202], [object_number=115, tag_tid=17, entry_tid=20199]]], state=connected] 2024-04-06T09:43:04.115 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=109, tag_tid=17, entry_tid=15197], [object_number=108, tag_tid=17, entry_tid=15196], [object_number=111, tag_tid=17, entry_tid=15195], [object_number=110, tag_tid=17, entry_tid=15194]]], state=connected] 2024-04-06T09:43:04.119 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:43:04.120 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=113, tag_tid=18, entry_tid=1' 2024-04-06T09:43:04.121 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:43:04.122 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:43:04.122 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:43:04.122 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:43:04.122 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:43:04.122 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:43:04.122 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:43:04.122 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:43:04.122 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:43:04.122 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:43:04.122 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:43:04.122 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:43:04.123 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:43:04.123 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:43:04.123 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:43:04.171 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 27 2024-04-06T09:43:04.171 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 28 2024-04-06T09:43:04.171 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:43:04.171 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=113, tag_tid=18, entry_tid=1], [object_number=112, tag_tid=18, entry_tid=0], [object_number=114, tag_tid=17, entry_tid=20202], [object_number=115, tag_tid=17, entry_tid=20199]]], state=connected] 2024-04-06T09:43:04.171 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=109, tag_tid=17, entry_tid=15197], [object_number=108, tag_tid=17, entry_tid=15196], [object_number=111, tag_tid=17, entry_tid=15195], [object_number=110, tag_tid=17, entry_tid=15194]]], state=connected] 2024-04-06T09:43:04.174 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:43:04.176 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=109, tag_tid=17, entry_tid=15197' 2024-04-06T09:43:04.176 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=113, tag_tid=18, entry_tid=1' -a 'object_number=113, tag_tid=18, entry_tid=1' = 'object_number=109, tag_tid=17, entry_tid=15197' 2024-04-06T09:43:04.176 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=109, tag_tid=17, entry_tid=15197' '!=' 'object_number=110, tag_tid=17, entry_tid=15114' 2024-04-06T09:43:04.176 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:43:04.176 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=109, tag_tid=17, entry_tid=15197' = 'object_number=110, tag_tid=17, entry_tid=15114' 2024-04-06T09:43:04.176 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=109, tag_tid=17, entry_tid=15197' 2024-04-06T09:43:04.177 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=113, tag_tid=18, entry_tid=1' 2024-04-06T09:43:04.177 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:43:04.177 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:43:04.178 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=18 2024-04-06T09:43:04.178 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=109, tag_tid=17, entry_tid=15197' 2024-04-06T09:43:04.179 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:43:04.179 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:43:04.179 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=17 2024-04-06T09:43:04.180 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=113, tag_tid=18, entry_tid=1' 2024-04-06T09:43:04.180 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:43:04.180 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:43:04.181 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:43:04.182 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=109, tag_tid=17, entry_tid=15197' 2024-04-06T09:43:04.182 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:43:04.182 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:43:04.183 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=15197 2024-04-06T09:43:04.183 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 18 = 17 -a 1 -le 15197 2024-04-06T09:43:04.183 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:43:04.183 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:43:04.184 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:43:04.385 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:43:04.386 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:43:04.386 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:43:04.386 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:43:04.386 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:43:04.386 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:43:04.386 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:43:04.386 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:43:04.386 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:43:04.386 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:43:04.386 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:43:04.386 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:43:04.386 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:43:04.387 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:43:04.387 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:43:04.387 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:43:04.387 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:43:04.389 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:43:04.389 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:43:04.389 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:43:04.389 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:43:04.389 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:43:04.389 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:43:04.390 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:43:04.390 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:43:04.390 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:43:04.390 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:43:04.390 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:43:04.390 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:43:04.390 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:43:04.390 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:43:04.390 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:43:04.390 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:43:04.392 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:43:04.392 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:43:04.392 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:43:04.392 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:43:04.392 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:43:04.392 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:43:04.392 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:43:04.392 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:43:04.392 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:43:04.392 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:43:04.392 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:43:04.392 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:43:04.392 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:43:04.393 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:43:04.393 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:43:04.393 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:43:04.416 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:43:04.417 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:43:04.417 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:43:04.417 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:43:04.547 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:43:04.548 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:43:04.548 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:43:04.549 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:43:04.549 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:43:04.549 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:43:04.549 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:43:04.549 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:43:04.549 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:43:04.549 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:43:04.549 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:43:04.550 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:43:04.550 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:43:04.552 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:43:04.552 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:43:04.553 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:43:04.604 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 27 2024-04-06T09:43:04.604 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 28 2024-04-06T09:43:04.604 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:43:04.604 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=113, tag_tid=18, entry_tid=1], [object_number=112, tag_tid=18, entry_tid=0], [object_number=114, tag_tid=17, entry_tid=20202], [object_number=115, tag_tid=17, entry_tid=20199]]], state=connected] 2024-04-06T09:43:04.604 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=108, tag_tid=17, entry_tid=15304], [object_number=111, tag_tid=17, entry_tid=15303], [object_number=110, tag_tid=17, entry_tid=15302], [object_number=109, tag_tid=17, entry_tid=15301]]], state=connected] 2024-04-06T09:43:04.607 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:43:04.608 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=113, tag_tid=18, entry_tid=1' 2024-04-06T09:43:04.610 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:43:04.610 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:43:04.610 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:43:04.610 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:43:04.610 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:43:04.610 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:43:04.610 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:43:04.610 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:43:04.610 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:43:04.610 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:43:04.610 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:43:04.610 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:43:04.611 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:43:04.611 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:43:04.611 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:43:04.659 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 27 2024-04-06T09:43:04.659 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 28 2024-04-06T09:43:04.659 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:43:04.660 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=113, tag_tid=18, entry_tid=1], [object_number=112, tag_tid=18, entry_tid=0], [object_number=114, tag_tid=17, entry_tid=20202], [object_number=115, tag_tid=17, entry_tid=20199]]], state=connected] 2024-04-06T09:43:04.660 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=108, tag_tid=17, entry_tid=15304], [object_number=111, tag_tid=17, entry_tid=15303], [object_number=110, tag_tid=17, entry_tid=15302], [object_number=109, tag_tid=17, entry_tid=15301]]], state=connected] 2024-04-06T09:43:04.663 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:43:04.665 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=108, tag_tid=17, entry_tid=15304' 2024-04-06T09:43:04.665 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=113, tag_tid=18, entry_tid=1' -a 'object_number=113, tag_tid=18, entry_tid=1' = 'object_number=108, tag_tid=17, entry_tid=15304' 2024-04-06T09:43:04.665 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=108, tag_tid=17, entry_tid=15304' '!=' 'object_number=109, tag_tid=17, entry_tid=15197' 2024-04-06T09:43:04.665 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:43:04.665 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=108, tag_tid=17, entry_tid=15304' = 'object_number=109, tag_tid=17, entry_tid=15197' 2024-04-06T09:43:04.665 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=108, tag_tid=17, entry_tid=15304' 2024-04-06T09:43:04.665 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=113, tag_tid=18, entry_tid=1' 2024-04-06T09:43:04.666 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:43:04.666 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:43:04.666 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=18 2024-04-06T09:43:04.667 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=108, tag_tid=17, entry_tid=15304' 2024-04-06T09:43:04.667 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:43:04.667 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:43:04.668 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=17 2024-04-06T09:43:04.669 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=113, tag_tid=18, entry_tid=1' 2024-04-06T09:43:04.669 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:43:04.669 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:43:04.670 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:43:04.671 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=108, tag_tid=17, entry_tid=15304' 2024-04-06T09:43:04.671 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:43:04.671 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:43:04.672 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=15304 2024-04-06T09:43:04.672 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 18 = 17 -a 1 -le 15304 2024-04-06T09:43:04.672 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:43:04.672 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:43:04.672 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:43:04.874 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:43:04.874 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:43:04.874 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:43:04.874 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:43:04.874 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:43:04.874 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:43:04.874 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:43:04.874 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:43:04.874 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:43:04.874 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:43:04.874 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:43:04.874 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:43:04.875 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:43:04.875 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:43:04.875 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:43:04.875 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:43:04.875 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:43:04.878 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:43:04.878 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:43:04.879 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:43:04.879 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:43:04.879 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:43:04.879 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:43:04.879 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:43:04.879 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:43:04.879 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:43:04.879 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:43:04.879 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:43:04.879 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:43:04.879 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:43:04.879 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:43:04.879 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:43:04.879 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:43:04.879 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:43:04.880 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:43:04.880 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:43:04.880 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:43:04.880 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:43:04.880 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:43:04.880 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:43:04.880 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:43:04.880 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:43:04.880 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:43:04.880 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:43:04.880 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:43:04.880 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:43:04.880 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:43:04.880 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:43:04.881 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:43:04.905 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:43:04.905 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:43:04.905 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:43:04.905 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:43:05.032 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:43:05.034 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:43:05.034 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:43:05.034 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:43:05.034 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:43:05.034 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:43:05.034 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:43:05.034 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:43:05.034 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:43:05.034 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:43:05.035 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:43:05.035 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:43:05.035 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:43:05.037 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:43:05.038 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:43:05.038 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:43:05.089 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 27 2024-04-06T09:43:05.089 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 28 2024-04-06T09:43:05.089 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:43:05.089 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=113, tag_tid=18, entry_tid=1], [object_number=112, tag_tid=18, entry_tid=0], [object_number=114, tag_tid=17, entry_tid=20202], [object_number=115, tag_tid=17, entry_tid=20199]]], state=connected] 2024-04-06T09:43:05.089 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=109, tag_tid=17, entry_tid=15417], [object_number=108, tag_tid=17, entry_tid=15416], [object_number=111, tag_tid=17, entry_tid=15415], [object_number=110, tag_tid=17, entry_tid=15414]]], state=connected] 2024-04-06T09:43:05.093 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:43:05.094 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=113, tag_tid=18, entry_tid=1' 2024-04-06T09:43:05.095 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:43:05.095 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:43:05.095 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:43:05.095 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:43:05.095 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:43:05.096 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:43:05.096 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:43:05.096 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:43:05.096 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:43:05.096 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:43:05.096 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:43:05.096 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:43:05.096 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:43:05.097 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:43:05.097 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:43:05.143 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 27 2024-04-06T09:43:05.143 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 28 2024-04-06T09:43:05.144 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:43:05.144 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=113, tag_tid=18, entry_tid=1], [object_number=112, tag_tid=18, entry_tid=0], [object_number=114, tag_tid=17, entry_tid=20202], [object_number=115, tag_tid=17, entry_tid=20199]]], state=connected] 2024-04-06T09:43:05.144 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=109, tag_tid=17, entry_tid=15417], [object_number=108, tag_tid=17, entry_tid=15416], [object_number=111, tag_tid=17, entry_tid=15415], [object_number=110, tag_tid=17, entry_tid=15414]]], state=connected] 2024-04-06T09:43:05.147 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:43:05.149 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=109, tag_tid=17, entry_tid=15417' 2024-04-06T09:43:05.149 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=113, tag_tid=18, entry_tid=1' -a 'object_number=113, tag_tid=18, entry_tid=1' = 'object_number=109, tag_tid=17, entry_tid=15417' 2024-04-06T09:43:05.149 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=109, tag_tid=17, entry_tid=15417' '!=' 'object_number=108, tag_tid=17, entry_tid=15304' 2024-04-06T09:43:05.149 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:43:05.149 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=109, tag_tid=17, entry_tid=15417' = 'object_number=108, tag_tid=17, entry_tid=15304' 2024-04-06T09:43:05.149 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=109, tag_tid=17, entry_tid=15417' 2024-04-06T09:43:05.149 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=113, tag_tid=18, entry_tid=1' 2024-04-06T09:43:05.149 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:43:05.150 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:43:05.150 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=18 2024-04-06T09:43:05.151 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=109, tag_tid=17, entry_tid=15417' 2024-04-06T09:43:05.151 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:43:05.151 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:43:05.152 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=17 2024-04-06T09:43:05.152 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=113, tag_tid=18, entry_tid=1' 2024-04-06T09:43:05.153 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:43:05.153 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:43:05.154 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:43:05.154 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=109, tag_tid=17, entry_tid=15417' 2024-04-06T09:43:05.154 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:43:05.154 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:43:05.156 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=15417 2024-04-06T09:43:05.156 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 18 = 17 -a 1 -le 15417 2024-04-06T09:43:05.156 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:43:05.156 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:43:05.156 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:43:05.357 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:43:05.357 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:43:05.358 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:43:05.358 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:43:05.358 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:43:05.358 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:43:05.358 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:43:05.358 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:43:05.358 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:43:05.358 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:43:05.358 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:43:05.358 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:43:05.358 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:43:05.358 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:43:05.358 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:43:05.359 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:43:05.359 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:43:05.361 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:43:05.361 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:43:05.361 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:43:05.362 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:43:05.362 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:43:05.362 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:43:05.362 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:43:05.362 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:43:05.362 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:43:05.362 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:43:05.362 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:43:05.362 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:43:05.362 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:43:05.362 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:43:05.362 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:43:05.362 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:43:05.363 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:43:05.364 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:43:05.364 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:43:05.364 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:43:05.364 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:43:05.364 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:43:05.364 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:43:05.364 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:43:05.364 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:43:05.364 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:43:05.364 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:43:05.364 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:43:05.364 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:43:05.364 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:43:05.364 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:43:05.365 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:43:05.389 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:43:05.389 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:43:05.389 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:43:05.389 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:43:05.518 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:43:05.518 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:43:05.518 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:43:05.518 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:43:05.518 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:43:05.519 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:43:05.519 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:43:05.519 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:43:05.519 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:43:05.519 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:43:05.519 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:43:05.520 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:43:05.520 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:43:05.521 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:43:05.522 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:43:05.522 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:43:05.574 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 27 2024-04-06T09:43:05.574 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 28 2024-04-06T09:43:05.574 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:43:05.574 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=113, tag_tid=18, entry_tid=1], [object_number=112, tag_tid=18, entry_tid=0], [object_number=114, tag_tid=17, entry_tid=20202], [object_number=115, tag_tid=17, entry_tid=20199]]], state=connected] 2024-04-06T09:43:05.574 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=109, tag_tid=17, entry_tid=15521], [object_number=108, tag_tid=17, entry_tid=15520], [object_number=111, tag_tid=17, entry_tid=15519], [object_number=110, tag_tid=17, entry_tid=15518]]], state=connected] 2024-04-06T09:43:05.577 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:43:05.579 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=113, tag_tid=18, entry_tid=1' 2024-04-06T09:43:05.580 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:43:05.580 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:43:05.580 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:43:05.580 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:43:05.580 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:43:05.580 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:43:05.580 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:43:05.580 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:43:05.580 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:43:05.581 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:43:05.581 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:43:05.581 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:43:05.581 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:43:05.581 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:43:05.582 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:43:05.838 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 27 2024-04-06T09:43:05.838 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 28 2024-04-06T09:43:05.838 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:43:05.838 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=113, tag_tid=18, entry_tid=1], [object_number=112, tag_tid=18, entry_tid=0], [object_number=114, tag_tid=17, entry_tid=20202], [object_number=115, tag_tid=17, entry_tid=20199]]], state=connected] 2024-04-06T09:43:05.838 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=109, tag_tid=17, entry_tid=15521], [object_number=108, tag_tid=17, entry_tid=15520], [object_number=111, tag_tid=17, entry_tid=15519], [object_number=110, tag_tid=17, entry_tid=15518]]], state=connected] 2024-04-06T09:43:05.842 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:43:05.844 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=109, tag_tid=17, entry_tid=15521' 2024-04-06T09:43:05.844 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=113, tag_tid=18, entry_tid=1' -a 'object_number=113, tag_tid=18, entry_tid=1' = 'object_number=109, tag_tid=17, entry_tid=15521' 2024-04-06T09:43:05.844 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=109, tag_tid=17, entry_tid=15521' '!=' 'object_number=109, tag_tid=17, entry_tid=15417' 2024-04-06T09:43:05.844 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:43:05.844 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=109, tag_tid=17, entry_tid=15521' = 'object_number=109, tag_tid=17, entry_tid=15417' 2024-04-06T09:43:05.844 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=109, tag_tid=17, entry_tid=15521' 2024-04-06T09:43:05.845 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=113, tag_tid=18, entry_tid=1' 2024-04-06T09:43:05.845 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:43:05.845 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:43:05.846 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=18 2024-04-06T09:43:05.847 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=109, tag_tid=17, entry_tid=15521' 2024-04-06T09:43:05.847 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:43:05.847 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:43:05.848 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=17 2024-04-06T09:43:05.849 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=113, tag_tid=18, entry_tid=1' 2024-04-06T09:43:05.849 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:43:05.849 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:43:05.850 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:43:05.851 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=109, tag_tid=17, entry_tid=15521' 2024-04-06T09:43:05.851 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:43:05.851 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:43:05.853 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=15521 2024-04-06T09:43:05.853 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 18 = 17 -a 1 -le 15521 2024-04-06T09:43:05.853 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:43:05.853 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:43:05.853 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:43:06.055 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:43:06.055 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:43:06.055 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:43:06.055 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:43:06.055 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:43:06.055 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:43:06.055 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:43:06.056 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:43:06.056 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:43:06.056 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:43:06.056 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:43:06.056 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:43:06.056 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:43:06.056 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:43:06.056 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:43:06.057 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:43:06.057 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:43:06.059 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:43:06.060 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:43:06.060 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:43:06.060 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:43:06.060 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:43:06.060 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:43:06.060 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:43:06.060 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:43:06.060 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:43:06.060 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:43:06.060 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:43:06.060 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:43:06.060 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:43:06.060 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:43:06.060 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:43:06.061 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:43:06.061 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:43:06.061 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:43:06.061 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:43:06.061 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:43:06.061 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:43:06.061 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:43:06.061 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:43:06.061 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:43:06.061 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:43:06.061 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:43:06.061 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:43:06.061 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:43:06.061 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:43:06.061 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:43:06.062 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:43:06.062 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:43:06.086 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:43:06.086 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:43:06.086 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:43:06.086 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:43:06.214 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:43:06.215 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:43:06.215 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:43:06.215 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:43:06.215 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:43:06.215 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:43:06.215 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:43:06.215 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:43:06.215 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:43:06.215 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:43:06.216 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:43:06.216 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:43:06.216 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:43:06.218 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:43:06.218 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:43:06.218 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:43:06.272 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 27 2024-04-06T09:43:06.272 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 28 2024-04-06T09:43:06.272 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:43:06.272 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=113, tag_tid=18, entry_tid=1], [object_number=112, tag_tid=18, entry_tid=0], [object_number=114, tag_tid=17, entry_tid=20202], [object_number=115, tag_tid=17, entry_tid=20199]]], state=connected] 2024-04-06T09:43:06.272 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=109, tag_tid=17, entry_tid=15653], [object_number=108, tag_tid=17, entry_tid=15652], [object_number=111, tag_tid=17, entry_tid=15651], [object_number=110, tag_tid=17, entry_tid=15650]]], state=connected] 2024-04-06T09:43:06.276 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:43:06.277 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=113, tag_tid=18, entry_tid=1' 2024-04-06T09:43:06.278 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:43:06.278 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:43:06.278 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:43:06.278 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:43:06.278 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:43:06.279 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:43:06.279 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:43:06.279 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:43:06.279 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:43:06.279 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:43:06.279 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:43:06.279 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:43:06.279 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:43:06.280 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:43:06.280 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:43:06.326 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 27 2024-04-06T09:43:06.326 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 28 2024-04-06T09:43:06.326 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:43:06.326 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=113, tag_tid=18, entry_tid=1], [object_number=112, tag_tid=18, entry_tid=0], [object_number=114, tag_tid=17, entry_tid=20202], [object_number=115, tag_tid=17, entry_tid=20199]]], state=connected] 2024-04-06T09:43:06.326 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=109, tag_tid=17, entry_tid=15653], [object_number=108, tag_tid=17, entry_tid=15652], [object_number=111, tag_tid=17, entry_tid=15651], [object_number=110, tag_tid=17, entry_tid=15650]]], state=connected] 2024-04-06T09:43:06.329 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:43:06.331 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=109, tag_tid=17, entry_tid=15653' 2024-04-06T09:43:06.332 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=113, tag_tid=18, entry_tid=1' -a 'object_number=113, tag_tid=18, entry_tid=1' = 'object_number=109, tag_tid=17, entry_tid=15653' 2024-04-06T09:43:06.332 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=109, tag_tid=17, entry_tid=15653' '!=' 'object_number=109, tag_tid=17, entry_tid=15521' 2024-04-06T09:43:06.332 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:43:06.332 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=109, tag_tid=17, entry_tid=15653' = 'object_number=109, tag_tid=17, entry_tid=15521' 2024-04-06T09:43:06.332 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=109, tag_tid=17, entry_tid=15653' 2024-04-06T09:43:06.332 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=113, tag_tid=18, entry_tid=1' 2024-04-06T09:43:06.332 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:43:06.332 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:43:06.333 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=18 2024-04-06T09:43:06.333 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=109, tag_tid=17, entry_tid=15653' 2024-04-06T09:43:06.333 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:43:06.334 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:43:06.334 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=17 2024-04-06T09:43:06.335 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=113, tag_tid=18, entry_tid=1' 2024-04-06T09:43:06.335 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:43:06.335 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:43:06.336 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:43:06.337 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=109, tag_tid=17, entry_tid=15653' 2024-04-06T09:43:06.337 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:43:06.337 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:43:06.338 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=15653 2024-04-06T09:43:06.339 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 18 = 17 -a 1 -le 15653 2024-04-06T09:43:06.339 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:43:06.339 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:43:06.339 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:43:06.542 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:43:06.543 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:43:06.543 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:43:06.543 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:43:06.543 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:43:06.543 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:43:06.543 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:43:06.543 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:43:06.543 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:43:06.543 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:43:06.543 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:43:06.543 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:43:06.543 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:43:06.543 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:43:06.543 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:43:06.544 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:43:06.544 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:43:06.546 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:43:06.546 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:43:06.546 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:43:06.546 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:43:06.546 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:43:06.546 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:43:06.547 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:43:06.547 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:43:06.547 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:43:06.547 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:43:06.547 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:43:06.547 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:43:06.547 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:43:06.547 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:43:06.547 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:43:06.547 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:43:06.547 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:43:06.547 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:43:06.547 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:43:06.547 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:43:06.547 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:43:06.548 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:43:06.548 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:43:06.548 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:43:06.548 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:43:06.548 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:43:06.548 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:43:06.548 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:43:06.548 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:43:06.548 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:43:06.548 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:43:06.548 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:43:06.570 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:43:06.571 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:43:06.571 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:43:06.571 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:43:06.702 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:43:06.703 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:43:06.703 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:43:06.703 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:43:06.703 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:43:06.703 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:43:06.703 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:43:06.703 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:43:06.704 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:43:06.704 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:43:06.704 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:43:06.704 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:43:06.705 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:43:06.707 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:43:06.707 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:43:06.707 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:43:06.754 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 27 2024-04-06T09:43:06.754 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 28 2024-04-06T09:43:06.754 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:43:06.754 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=113, tag_tid=18, entry_tid=1], [object_number=112, tag_tid=18, entry_tid=0], [object_number=114, tag_tid=17, entry_tid=20202], [object_number=115, tag_tid=17, entry_tid=20199]]], state=connected] 2024-04-06T09:43:06.755 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=110, tag_tid=17, entry_tid=15742], [object_number=109, tag_tid=17, entry_tid=15741], [object_number=108, tag_tid=17, entry_tid=15740], [object_number=111, tag_tid=17, entry_tid=15739]]], state=connected] 2024-04-06T09:43:06.758 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:43:06.759 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=113, tag_tid=18, entry_tid=1' 2024-04-06T09:43:06.760 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:43:06.760 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:43:06.760 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:43:06.760 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:43:06.760 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:43:06.760 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:43:06.761 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:43:06.761 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:43:06.761 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:43:06.761 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:43:06.761 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:43:06.761 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:43:06.761 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:43:06.762 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:43:06.762 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:43:06.809 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 27 2024-04-06T09:43:06.809 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 28 2024-04-06T09:43:06.809 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:43:06.809 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=113, tag_tid=18, entry_tid=1], [object_number=112, tag_tid=18, entry_tid=0], [object_number=114, tag_tid=17, entry_tid=20202], [object_number=115, tag_tid=17, entry_tid=20199]]], state=connected] 2024-04-06T09:43:06.810 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=110, tag_tid=17, entry_tid=15742], [object_number=109, tag_tid=17, entry_tid=15741], [object_number=108, tag_tid=17, entry_tid=15740], [object_number=111, tag_tid=17, entry_tid=15739]]], state=connected] 2024-04-06T09:43:06.813 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:43:06.814 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=110, tag_tid=17, entry_tid=15742' 2024-04-06T09:43:06.815 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=113, tag_tid=18, entry_tid=1' -a 'object_number=113, tag_tid=18, entry_tid=1' = 'object_number=110, tag_tid=17, entry_tid=15742' 2024-04-06T09:43:06.815 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=110, tag_tid=17, entry_tid=15742' '!=' 'object_number=109, tag_tid=17, entry_tid=15653' 2024-04-06T09:43:06.815 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:43:06.815 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=110, tag_tid=17, entry_tid=15742' = 'object_number=109, tag_tid=17, entry_tid=15653' 2024-04-06T09:43:06.815 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=110, tag_tid=17, entry_tid=15742' 2024-04-06T09:43:06.815 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=113, tag_tid=18, entry_tid=1' 2024-04-06T09:43:06.816 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:43:06.816 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:43:06.816 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=18 2024-04-06T09:43:06.817 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=110, tag_tid=17, entry_tid=15742' 2024-04-06T09:43:06.817 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:43:06.817 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:43:06.818 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=17 2024-04-06T09:43:06.818 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=113, tag_tid=18, entry_tid=1' 2024-04-06T09:43:06.818 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:43:06.818 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:43:06.819 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:43:06.820 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=110, tag_tid=17, entry_tid=15742' 2024-04-06T09:43:06.820 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:43:06.820 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:43:06.821 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=15742 2024-04-06T09:43:06.821 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 18 = 17 -a 1 -le 15742 2024-04-06T09:43:06.821 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:43:06.822 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:43:06.822 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:43:07.023 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:43:07.023 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:43:07.023 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:43:07.023 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:43:07.024 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:43:07.024 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:43:07.024 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:43:07.024 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:43:07.024 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:43:07.024 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:43:07.024 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:43:07.024 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:43:07.024 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:43:07.024 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:43:07.024 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:43:07.025 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:43:07.025 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:43:07.028 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:43:07.028 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:43:07.028 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:43:07.029 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:43:07.029 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:43:07.029 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:43:07.029 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:43:07.029 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:43:07.029 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:43:07.029 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:43:07.029 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:43:07.029 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:43:07.029 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:43:07.029 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:43:07.029 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:43:07.029 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:43:07.029 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:43:07.030 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:43:07.030 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:43:07.030 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:43:07.030 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:43:07.030 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:43:07.030 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:43:07.030 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:43:07.030 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:43:07.030 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:43:07.030 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:43:07.030 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:43:07.030 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:43:07.030 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:43:07.031 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:43:07.031 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:43:07.054 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:43:07.055 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:43:07.055 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:43:07.055 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:43:07.186 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:43:07.187 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:43:07.187 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:43:07.187 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:43:07.187 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:43:07.187 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:43:07.187 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:43:07.187 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:43:07.187 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:43:07.187 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:43:07.188 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:43:07.188 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:43:07.188 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:43:07.190 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:43:07.190 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:43:07.191 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:43:07.237 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 27 2024-04-06T09:43:07.237 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 28 2024-04-06T09:43:07.237 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:43:07.238 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=113, tag_tid=18, entry_tid=1], [object_number=112, tag_tid=18, entry_tid=0], [object_number=114, tag_tid=17, entry_tid=20202], [object_number=115, tag_tid=17, entry_tid=20199]]], state=connected] 2024-04-06T09:43:07.238 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=110, tag_tid=17, entry_tid=15826], [object_number=109, tag_tid=17, entry_tid=15825], [object_number=108, tag_tid=17, entry_tid=15824], [object_number=111, tag_tid=17, entry_tid=15823]]], state=connected] 2024-04-06T09:43:07.241 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:43:07.242 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=113, tag_tid=18, entry_tid=1' 2024-04-06T09:43:07.243 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:43:07.243 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:43:07.243 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:43:07.244 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:43:07.244 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:43:07.244 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:43:07.244 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:43:07.244 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:43:07.244 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:43:07.244 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:43:07.244 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:43:07.244 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:43:07.245 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:43:07.245 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:43:07.245 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:43:07.291 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 27 2024-04-06T09:43:07.291 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 28 2024-04-06T09:43:07.291 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:43:07.291 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=113, tag_tid=18, entry_tid=1], [object_number=112, tag_tid=18, entry_tid=0], [object_number=114, tag_tid=17, entry_tid=20202], [object_number=115, tag_tid=17, entry_tid=20199]]], state=connected] 2024-04-06T09:43:07.292 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=110, tag_tid=17, entry_tid=15826], [object_number=109, tag_tid=17, entry_tid=15825], [object_number=108, tag_tid=17, entry_tid=15824], [object_number=111, tag_tid=17, entry_tid=15823]]], state=connected] 2024-04-06T09:43:07.295 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:43:07.297 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=110, tag_tid=17, entry_tid=15826' 2024-04-06T09:43:07.297 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=113, tag_tid=18, entry_tid=1' -a 'object_number=113, tag_tid=18, entry_tid=1' = 'object_number=110, tag_tid=17, entry_tid=15826' 2024-04-06T09:43:07.297 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=110, tag_tid=17, entry_tid=15826' '!=' 'object_number=110, tag_tid=17, entry_tid=15742' 2024-04-06T09:43:07.297 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:43:07.297 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=110, tag_tid=17, entry_tid=15826' = 'object_number=110, tag_tid=17, entry_tid=15742' 2024-04-06T09:43:07.297 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=110, tag_tid=17, entry_tid=15826' 2024-04-06T09:43:07.298 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=113, tag_tid=18, entry_tid=1' 2024-04-06T09:43:07.298 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:43:07.298 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:43:07.298 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=18 2024-04-06T09:43:07.299 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=110, tag_tid=17, entry_tid=15826' 2024-04-06T09:43:07.299 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:43:07.299 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:43:07.300 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=17 2024-04-06T09:43:07.301 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=113, tag_tid=18, entry_tid=1' 2024-04-06T09:43:07.301 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:43:07.301 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:43:07.302 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:43:07.302 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=110, tag_tid=17, entry_tid=15826' 2024-04-06T09:43:07.302 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:43:07.303 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:43:07.304 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=15826 2024-04-06T09:43:07.304 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 18 = 17 -a 1 -le 15826 2024-04-06T09:43:07.304 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:43:07.304 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:43:07.304 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:43:07.506 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:43:07.506 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:43:07.506 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:43:07.506 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:43:07.506 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:43:07.506 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:43:07.506 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:43:07.506 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:43:07.506 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:43:07.506 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:43:07.506 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:43:07.507 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:43:07.507 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:43:07.507 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:43:07.507 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:43:07.507 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:43:07.507 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:43:07.509 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:43:07.509 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:43:07.509 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:43:07.509 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:43:07.509 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:43:07.509 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:43:07.509 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:43:07.510 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:43:07.510 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:43:07.510 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:43:07.510 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:43:07.510 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:43:07.510 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:43:07.510 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:43:07.510 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:43:07.510 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:43:07.511 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:43:07.511 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:43:07.511 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:43:07.511 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:43:07.511 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:43:07.511 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:43:07.512 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:43:07.512 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:43:07.512 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:43:07.512 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:43:07.512 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:43:07.512 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:43:07.512 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:43:07.512 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:43:07.512 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:43:07.512 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:43:07.536 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:43:07.536 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:43:07.536 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:43:07.536 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:43:07.666 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:43:07.667 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:43:07.667 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:43:07.667 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:43:07.667 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:43:07.668 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:43:07.668 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:43:07.668 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:43:07.668 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:43:07.668 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:43:07.668 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:43:07.668 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:43:07.668 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:43:07.669 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:43:07.670 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:43:07.670 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:43:07.717 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 27 2024-04-06T09:43:07.717 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 28 2024-04-06T09:43:07.717 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:43:07.718 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=113, tag_tid=18, entry_tid=1], [object_number=112, tag_tid=18, entry_tid=0], [object_number=114, tag_tid=17, entry_tid=20202], [object_number=115, tag_tid=17, entry_tid=20199]]], state=connected] 2024-04-06T09:43:07.718 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=109, tag_tid=17, entry_tid=15909], [object_number=108, tag_tid=17, entry_tid=15908], [object_number=111, tag_tid=17, entry_tid=15907], [object_number=110, tag_tid=17, entry_tid=15906]]], state=connected] 2024-04-06T09:43:07.721 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:43:07.722 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=113, tag_tid=18, entry_tid=1' 2024-04-06T09:43:07.724 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:43:07.724 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:43:07.724 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:43:07.724 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:43:07.724 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:43:07.724 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:43:07.724 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:43:07.724 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:43:07.724 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:43:07.725 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:43:07.725 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:43:07.725 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:43:07.726 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:43:07.726 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:43:07.726 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:43:07.771 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 27 2024-04-06T09:43:07.771 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 28 2024-04-06T09:43:07.771 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:43:07.771 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=113, tag_tid=18, entry_tid=1], [object_number=112, tag_tid=18, entry_tid=0], [object_number=114, tag_tid=17, entry_tid=20202], [object_number=115, tag_tid=17, entry_tid=20199]]], state=connected] 2024-04-06T09:43:07.772 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=109, tag_tid=17, entry_tid=15909], [object_number=108, tag_tid=17, entry_tid=15908], [object_number=111, tag_tid=17, entry_tid=15907], [object_number=110, tag_tid=17, entry_tid=15906]]], state=connected] 2024-04-06T09:43:07.775 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:43:07.778 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=109, tag_tid=17, entry_tid=15909' 2024-04-06T09:43:07.778 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=113, tag_tid=18, entry_tid=1' -a 'object_number=113, tag_tid=18, entry_tid=1' = 'object_number=109, tag_tid=17, entry_tid=15909' 2024-04-06T09:43:07.778 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=109, tag_tid=17, entry_tid=15909' '!=' 'object_number=110, tag_tid=17, entry_tid=15826' 2024-04-06T09:43:07.778 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:43:07.778 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=109, tag_tid=17, entry_tid=15909' = 'object_number=110, tag_tid=17, entry_tid=15826' 2024-04-06T09:43:07.778 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=109, tag_tid=17, entry_tid=15909' 2024-04-06T09:43:07.778 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=113, tag_tid=18, entry_tid=1' 2024-04-06T09:43:07.778 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:43:07.778 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:43:07.779 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=18 2024-04-06T09:43:07.780 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=109, tag_tid=17, entry_tid=15909' 2024-04-06T09:43:07.780 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:43:07.780 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:43:07.781 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=17 2024-04-06T09:43:07.782 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=113, tag_tid=18, entry_tid=1' 2024-04-06T09:43:07.782 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:43:07.782 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:43:07.783 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:43:07.783 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=109, tag_tid=17, entry_tid=15909' 2024-04-06T09:43:07.783 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:43:07.784 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:43:07.784 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=15909 2024-04-06T09:43:07.784 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 18 = 17 -a 1 -le 15909 2024-04-06T09:43:07.785 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:43:07.785 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:43:07.785 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:43:07.986 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:43:07.986 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:43:07.986 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:43:07.986 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:43:07.986 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:43:07.986 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:43:07.986 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:43:07.986 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:43:07.986 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:43:07.986 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:43:07.986 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:43:07.987 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:43:07.987 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:43:07.987 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:43:07.987 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:43:07.987 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:43:07.987 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:43:07.990 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:43:07.991 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:43:07.991 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:43:07.991 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:43:07.991 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:43:07.991 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:43:07.991 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:43:07.991 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:43:07.991 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:43:07.991 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:43:07.991 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:43:07.991 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:43:07.991 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:43:07.991 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:43:07.991 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:43:07.992 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:43:07.992 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:43:07.992 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:43:07.992 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:43:07.992 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:43:07.992 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:43:07.992 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:43:07.992 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:43:07.992 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:43:07.992 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:43:07.992 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:43:07.992 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:43:07.992 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:43:07.992 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:43:07.993 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:43:07.993 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:43:07.993 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:43:08.017 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:43:08.018 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:43:08.018 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:43:08.018 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:43:08.146 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:43:08.148 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:43:08.148 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:43:08.148 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:43:08.148 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:43:08.148 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:43:08.148 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:43:08.148 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:43:08.148 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:43:08.148 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:43:08.149 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:43:08.149 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:43:08.149 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:43:08.150 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:43:08.151 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:43:08.151 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:43:08.204 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 27 2024-04-06T09:43:08.204 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 28 2024-04-06T09:43:08.204 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:43:08.205 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=113, tag_tid=18, entry_tid=1], [object_number=112, tag_tid=18, entry_tid=0], [object_number=114, tag_tid=17, entry_tid=20202], [object_number=115, tag_tid=17, entry_tid=20199]]], state=connected] 2024-04-06T09:43:08.205 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=109, tag_tid=17, entry_tid=15981], [object_number=108, tag_tid=17, entry_tid=15980], [object_number=111, tag_tid=17, entry_tid=15979], [object_number=110, tag_tid=17, entry_tid=15978]]], state=connected] 2024-04-06T09:43:08.208 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:43:08.210 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=113, tag_tid=18, entry_tid=1' 2024-04-06T09:43:08.210 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:43:08.210 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:43:08.210 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:43:08.210 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:43:08.210 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:43:08.211 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:43:08.211 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:43:08.211 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:43:08.211 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:43:08.211 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:43:08.211 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:43:08.211 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:43:08.212 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:43:08.212 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:43:08.212 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:43:08.258 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 27 2024-04-06T09:43:08.259 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 28 2024-04-06T09:43:08.259 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:43:08.259 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=113, tag_tid=18, entry_tid=1], [object_number=112, tag_tid=18, entry_tid=0], [object_number=114, tag_tid=17, entry_tid=20202], [object_number=115, tag_tid=17, entry_tid=20199]]], state=connected] 2024-04-06T09:43:08.259 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=109, tag_tid=17, entry_tid=15981], [object_number=108, tag_tid=17, entry_tid=15980], [object_number=111, tag_tid=17, entry_tid=15979], [object_number=110, tag_tid=17, entry_tid=15978]]], state=connected] 2024-04-06T09:43:08.262 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:43:08.264 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=109, tag_tid=17, entry_tid=15981' 2024-04-06T09:43:08.265 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=113, tag_tid=18, entry_tid=1' -a 'object_number=113, tag_tid=18, entry_tid=1' = 'object_number=109, tag_tid=17, entry_tid=15981' 2024-04-06T09:43:08.265 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=109, tag_tid=17, entry_tid=15981' '!=' 'object_number=109, tag_tid=17, entry_tid=15909' 2024-04-06T09:43:08.265 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:43:08.265 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=109, tag_tid=17, entry_tid=15981' = 'object_number=109, tag_tid=17, entry_tid=15909' 2024-04-06T09:43:08.265 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=109, tag_tid=17, entry_tid=15981' 2024-04-06T09:43:08.265 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=113, tag_tid=18, entry_tid=1' 2024-04-06T09:43:08.265 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:43:08.265 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:43:08.266 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=18 2024-04-06T09:43:08.266 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=109, tag_tid=17, entry_tid=15981' 2024-04-06T09:43:08.266 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:43:08.267 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:43:08.267 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=17 2024-04-06T09:43:08.268 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=113, tag_tid=18, entry_tid=1' 2024-04-06T09:43:08.268 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:43:08.268 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:43:08.269 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:43:08.270 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=109, tag_tid=17, entry_tid=15981' 2024-04-06T09:43:08.270 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:43:08.270 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:43:08.271 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=15981 2024-04-06T09:43:08.271 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 18 = 17 -a 1 -le 15981 2024-04-06T09:43:08.271 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:43:08.271 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:43:08.271 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:43:08.473 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:43:08.474 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:43:08.474 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:43:08.474 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:43:08.474 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:43:08.474 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:43:08.474 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:43:08.474 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:43:08.474 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:43:08.474 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:43:08.474 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:43:08.474 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:43:08.474 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:43:08.475 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:43:08.475 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:43:08.475 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:43:08.475 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:43:08.477 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:43:08.477 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:43:08.477 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:43:08.477 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:43:08.477 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:43:08.477 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:43:08.477 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:43:08.477 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:43:08.477 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:43:08.477 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:43:08.477 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:43:08.478 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:43:08.478 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:43:08.478 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:43:08.478 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:43:08.478 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:43:08.479 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:43:08.480 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:43:08.480 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:43:08.480 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:43:08.480 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:43:08.480 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:43:08.480 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:43:08.480 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:43:08.480 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:43:08.480 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:43:08.480 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:43:08.480 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:43:08.480 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:43:08.480 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:43:08.480 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:43:08.481 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:43:08.504 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:43:08.504 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:43:08.504 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:43:08.504 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:43:08.632 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:43:08.633 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:43:08.633 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:43:08.633 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:43:08.633 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:43:08.633 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:43:08.633 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:43:08.633 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:43:08.633 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:43:08.633 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:43:08.634 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:43:08.634 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:43:08.634 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:43:08.635 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:43:08.636 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:43:08.636 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:43:08.688 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 27 2024-04-06T09:43:08.688 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 28 2024-04-06T09:43:08.688 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:43:08.688 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=113, tag_tid=18, entry_tid=1], [object_number=112, tag_tid=18, entry_tid=0], [object_number=114, tag_tid=17, entry_tid=20202], [object_number=115, tag_tid=17, entry_tid=20199]]], state=connected] 2024-04-06T09:43:08.688 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=110, tag_tid=17, entry_tid=16062], [object_number=109, tag_tid=17, entry_tid=16061], [object_number=108, tag_tid=17, entry_tid=16060], [object_number=111, tag_tid=17, entry_tid=16059]]], state=connected] 2024-04-06T09:43:08.692 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:43:08.693 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=113, tag_tid=18, entry_tid=1' 2024-04-06T09:43:08.694 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:43:08.694 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:43:08.694 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:43:08.694 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:43:08.695 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:43:08.695 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:43:08.695 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:43:08.695 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:43:08.695 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:43:08.695 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:43:08.695 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:43:08.695 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:43:08.695 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:43:08.696 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:43:08.696 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:43:08.743 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 27 2024-04-06T09:43:08.743 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 28 2024-04-06T09:43:08.743 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:43:08.743 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=113, tag_tid=18, entry_tid=1], [object_number=112, tag_tid=18, entry_tid=0], [object_number=114, tag_tid=17, entry_tid=20202], [object_number=115, tag_tid=17, entry_tid=20199]]], state=connected] 2024-04-06T09:43:08.743 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=110, tag_tid=17, entry_tid=16062], [object_number=109, tag_tid=17, entry_tid=16061], [object_number=108, tag_tid=17, entry_tid=16060], [object_number=111, tag_tid=17, entry_tid=16059]]], state=connected] 2024-04-06T09:43:08.747 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:43:08.749 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=110, tag_tid=17, entry_tid=16062' 2024-04-06T09:43:08.749 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=113, tag_tid=18, entry_tid=1' -a 'object_number=113, tag_tid=18, entry_tid=1' = 'object_number=110, tag_tid=17, entry_tid=16062' 2024-04-06T09:43:08.749 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=110, tag_tid=17, entry_tid=16062' '!=' 'object_number=109, tag_tid=17, entry_tid=15981' 2024-04-06T09:43:08.749 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:43:08.750 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=110, tag_tid=17, entry_tid=16062' = 'object_number=109, tag_tid=17, entry_tid=15981' 2024-04-06T09:43:08.750 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=110, tag_tid=17, entry_tid=16062' 2024-04-06T09:43:08.750 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=113, tag_tid=18, entry_tid=1' 2024-04-06T09:43:08.750 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:43:08.750 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:43:08.750 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=18 2024-04-06T09:43:08.751 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=110, tag_tid=17, entry_tid=16062' 2024-04-06T09:43:08.751 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:43:08.751 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:43:08.752 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=17 2024-04-06T09:43:08.753 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=113, tag_tid=18, entry_tid=1' 2024-04-06T09:43:08.753 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:43:08.753 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:43:08.754 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:43:08.755 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=110, tag_tid=17, entry_tid=16062' 2024-04-06T09:43:08.755 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:43:08.755 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:43:08.756 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=16062 2024-04-06T09:43:08.756 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 18 = 17 -a 1 -le 16062 2024-04-06T09:43:08.756 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:43:08.756 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:43:08.756 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:43:08.958 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:43:08.958 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:43:08.958 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:43:08.958 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:43:08.958 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:43:08.958 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:43:08.958 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:43:08.958 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:43:08.958 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:43:08.959 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:43:08.959 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:43:08.959 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:43:08.959 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:43:08.959 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:43:08.959 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:43:08.959 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:43:08.960 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:43:08.963 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:43:08.963 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:43:08.963 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:43:08.963 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:43:08.963 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:43:08.963 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:43:08.963 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:43:08.963 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:43:08.964 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:43:08.964 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:43:08.964 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:43:08.964 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:43:08.964 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:43:08.964 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:43:08.964 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:43:08.964 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:43:08.964 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:43:08.964 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:43:08.964 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:43:08.964 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:43:08.964 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:43:08.964 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:43:08.965 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:43:08.965 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:43:08.965 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:43:08.965 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:43:08.965 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:43:08.965 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:43:08.965 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:43:08.965 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:43:08.965 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:43:08.965 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:43:08.990 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:43:08.990 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:43:08.990 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:43:08.990 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:43:09.123 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:43:09.124 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:43:09.125 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:43:09.125 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:43:09.125 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:43:09.125 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:43:09.125 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:43:09.125 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:43:09.125 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:43:09.125 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:43:09.125 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:43:09.126 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:43:09.126 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:43:09.127 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:43:09.128 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:43:09.128 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:43:09.184 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 27 2024-04-06T09:43:09.185 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 28 2024-04-06T09:43:09.185 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:43:09.185 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=113, tag_tid=18, entry_tid=1], [object_number=112, tag_tid=18, entry_tid=0], [object_number=114, tag_tid=17, entry_tid=20202], [object_number=115, tag_tid=17, entry_tid=20199]]], state=connected] 2024-04-06T09:43:09.185 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=108, tag_tid=17, entry_tid=16140], [object_number=111, tag_tid=17, entry_tid=16139], [object_number=110, tag_tid=17, entry_tid=16138], [object_number=109, tag_tid=17, entry_tid=16137]]], state=connected] 2024-04-06T09:43:09.189 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:43:09.190 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=113, tag_tid=18, entry_tid=1' 2024-04-06T09:43:09.191 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:43:09.191 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:43:09.191 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:43:09.191 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:43:09.192 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:43:09.192 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:43:09.192 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:43:09.192 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:43:09.192 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:43:09.192 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:43:09.192 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:43:09.192 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:43:09.193 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:43:09.193 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:43:09.193 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:43:09.239 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 27 2024-04-06T09:43:09.239 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 28 2024-04-06T09:43:09.239 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:43:09.239 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=113, tag_tid=18, entry_tid=1], [object_number=112, tag_tid=18, entry_tid=0], [object_number=114, tag_tid=17, entry_tid=20202], [object_number=115, tag_tid=17, entry_tid=20199]]], state=connected] 2024-04-06T09:43:09.239 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=108, tag_tid=17, entry_tid=16140], [object_number=111, tag_tid=17, entry_tid=16139], [object_number=110, tag_tid=17, entry_tid=16138], [object_number=109, tag_tid=17, entry_tid=16137]]], state=connected] 2024-04-06T09:43:09.242 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:43:09.244 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=108, tag_tid=17, entry_tid=16140' 2024-04-06T09:43:09.244 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=113, tag_tid=18, entry_tid=1' -a 'object_number=113, tag_tid=18, entry_tid=1' = 'object_number=108, tag_tid=17, entry_tid=16140' 2024-04-06T09:43:09.244 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=108, tag_tid=17, entry_tid=16140' '!=' 'object_number=110, tag_tid=17, entry_tid=16062' 2024-04-06T09:43:09.244 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:43:09.244 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=108, tag_tid=17, entry_tid=16140' = 'object_number=110, tag_tid=17, entry_tid=16062' 2024-04-06T09:43:09.244 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=108, tag_tid=17, entry_tid=16140' 2024-04-06T09:43:09.245 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=113, tag_tid=18, entry_tid=1' 2024-04-06T09:43:09.245 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:43:09.245 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:43:09.246 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=18 2024-04-06T09:43:09.246 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=108, tag_tid=17, entry_tid=16140' 2024-04-06T09:43:09.246 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:43:09.246 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:43:09.247 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=17 2024-04-06T09:43:09.248 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=113, tag_tid=18, entry_tid=1' 2024-04-06T09:43:09.248 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:43:09.248 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:43:09.249 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:43:09.250 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=108, tag_tid=17, entry_tid=16140' 2024-04-06T09:43:09.250 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:43:09.250 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:43:09.251 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=16140 2024-04-06T09:43:09.251 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 18 = 17 -a 1 -le 16140 2024-04-06T09:43:09.251 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:43:09.251 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:43:09.251 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:43:09.453 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:43:09.453 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:43:09.453 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:43:09.453 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:43:09.454 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:43:09.454 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:43:09.454 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:43:09.454 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:43:09.454 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:43:09.454 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:43:09.454 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:43:09.454 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:43:09.454 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:43:09.454 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:43:09.454 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:43:09.457 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:43:09.457 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:43:09.457 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:43:09.457 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:43:09.457 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:43:09.457 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:43:09.457 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:43:09.457 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:43:09.457 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:43:09.457 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:43:09.457 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:43:09.458 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:43:09.458 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:43:09.458 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:43:09.458 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:43:09.458 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:43:09.458 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:43:09.458 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:43:09.458 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:43:09.458 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:43:09.458 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:43:09.458 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:43:09.458 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:43:09.458 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:43:09.459 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:43:09.459 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:43:09.459 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:43:09.459 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:43:09.459 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:43:09.459 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:43:09.459 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:43:09.459 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:43:09.459 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:43:09.460 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:43:09.484 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:43:09.485 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:43:09.485 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:43:09.485 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:43:09.615 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:43:09.617 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:43:09.617 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:43:09.617 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:43:09.617 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:43:09.617 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:43:09.617 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:43:09.617 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:43:09.617 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:43:09.617 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:43:09.617 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:43:09.618 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:43:09.618 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:43:09.620 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:43:09.620 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:43:09.621 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:43:09.677 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 27 2024-04-06T09:43:09.677 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 28 2024-04-06T09:43:09.677 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:43:09.677 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=113, tag_tid=18, entry_tid=1], [object_number=112, tag_tid=18, entry_tid=0], [object_number=114, tag_tid=17, entry_tid=20202], [object_number=115, tag_tid=17, entry_tid=20199]]], state=connected] 2024-04-06T09:43:09.677 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=109, tag_tid=17, entry_tid=16221], [object_number=108, tag_tid=17, entry_tid=16220], [object_number=111, tag_tid=17, entry_tid=16219], [object_number=110, tag_tid=17, entry_tid=16218]]], state=connected] 2024-04-06T09:43:09.680 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:43:09.682 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=113, tag_tid=18, entry_tid=1' 2024-04-06T09:43:09.683 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:43:09.683 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:43:09.683 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:43:09.683 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:43:09.683 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:43:09.683 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:43:09.683 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:43:09.683 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:43:09.683 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:43:09.683 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:43:09.683 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:43:09.683 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:43:09.684 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:43:09.684 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:43:09.684 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:43:09.731 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 27 2024-04-06T09:43:09.731 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 28 2024-04-06T09:43:09.731 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:43:09.731 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=113, tag_tid=18, entry_tid=1], [object_number=112, tag_tid=18, entry_tid=0], [object_number=114, tag_tid=17, entry_tid=20202], [object_number=115, tag_tid=17, entry_tid=20199]]], state=connected] 2024-04-06T09:43:09.731 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=109, tag_tid=17, entry_tid=16221], [object_number=108, tag_tid=17, entry_tid=16220], [object_number=111, tag_tid=17, entry_tid=16219], [object_number=110, tag_tid=17, entry_tid=16218]]], state=connected] 2024-04-06T09:43:09.734 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:43:09.736 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=109, tag_tid=17, entry_tid=16221' 2024-04-06T09:43:09.736 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=113, tag_tid=18, entry_tid=1' -a 'object_number=113, tag_tid=18, entry_tid=1' = 'object_number=109, tag_tid=17, entry_tid=16221' 2024-04-06T09:43:09.736 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=109, tag_tid=17, entry_tid=16221' '!=' 'object_number=108, tag_tid=17, entry_tid=16140' 2024-04-06T09:43:09.736 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:43:09.736 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=109, tag_tid=17, entry_tid=16221' = 'object_number=108, tag_tid=17, entry_tid=16140' 2024-04-06T09:43:09.737 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=109, tag_tid=17, entry_tid=16221' 2024-04-06T09:43:09.737 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=113, tag_tid=18, entry_tid=1' 2024-04-06T09:43:09.737 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:43:09.737 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:43:09.738 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=18 2024-04-06T09:43:09.738 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=109, tag_tid=17, entry_tid=16221' 2024-04-06T09:43:09.738 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:43:09.739 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:43:09.739 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=17 2024-04-06T09:43:09.740 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=113, tag_tid=18, entry_tid=1' 2024-04-06T09:43:09.740 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:43:09.740 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:43:09.741 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:43:09.742 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=109, tag_tid=17, entry_tid=16221' 2024-04-06T09:43:09.742 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:43:09.742 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:43:09.743 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=16221 2024-04-06T09:43:09.743 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 18 = 17 -a 1 -le 16221 2024-04-06T09:43:09.743 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:43:09.743 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:43:09.744 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:43:09.945 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:43:09.945 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:43:09.945 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:43:09.945 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:43:09.945 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:43:09.945 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:43:09.945 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:43:09.945 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:43:09.946 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:43:09.946 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:43:09.946 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:43:09.946 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:43:09.946 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:43:09.946 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:43:09.946 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:43:09.946 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:43:09.947 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:43:09.950 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:43:09.950 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:43:09.950 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:43:09.950 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:43:09.950 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:43:09.950 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:43:09.950 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:43:09.950 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:43:09.950 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:43:09.951 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:43:09.951 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:43:09.951 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:43:09.951 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:43:09.951 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:43:09.951 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:43:09.951 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:43:09.951 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:43:09.951 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:43:09.951 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:43:09.951 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:43:09.951 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:43:09.951 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:43:09.951 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:43:09.951 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:43:09.951 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:43:09.952 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:43:09.952 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:43:09.952 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:43:09.952 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:43:09.952 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:43:09.952 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:43:09.952 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:43:09.976 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:43:09.976 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:43:09.976 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:43:09.976 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:43:10.105 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:43:10.105 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:43:10.106 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:43:10.106 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:43:10.106 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:43:10.106 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:43:10.106 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:43:10.106 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:43:10.106 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:43:10.106 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:43:10.107 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:43:10.107 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:43:10.107 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:43:10.109 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:43:10.109 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:43:10.109 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:43:10.161 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 27 2024-04-06T09:43:10.161 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 28 2024-04-06T09:43:10.161 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:43:10.161 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=113, tag_tid=18, entry_tid=1], [object_number=112, tag_tid=18, entry_tid=0], [object_number=114, tag_tid=17, entry_tid=20202], [object_number=115, tag_tid=17, entry_tid=20199]]], state=connected] 2024-04-06T09:43:10.161 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=108, tag_tid=17, entry_tid=16308], [object_number=111, tag_tid=17, entry_tid=16307], [object_number=110, tag_tid=17, entry_tid=16306], [object_number=109, tag_tid=17, entry_tid=16305]]], state=connected] 2024-04-06T09:43:10.164 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:43:10.165 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=113, tag_tid=18, entry_tid=1' 2024-04-06T09:43:10.167 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:43:10.167 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:43:10.167 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:43:10.167 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:43:10.167 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:43:10.167 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:43:10.167 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:43:10.167 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:43:10.167 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:43:10.167 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:43:10.168 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:43:10.168 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:43:10.168 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:43:10.168 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:43:10.169 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:43:10.214 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 27 2024-04-06T09:43:10.215 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 28 2024-04-06T09:43:10.215 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:43:10.215 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=113, tag_tid=18, entry_tid=1], [object_number=112, tag_tid=18, entry_tid=0], [object_number=114, tag_tid=17, entry_tid=20202], [object_number=115, tag_tid=17, entry_tid=20199]]], state=connected] 2024-04-06T09:43:10.215 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=108, tag_tid=17, entry_tid=16308], [object_number=111, tag_tid=17, entry_tid=16307], [object_number=110, tag_tid=17, entry_tid=16306], [object_number=109, tag_tid=17, entry_tid=16305]]], state=connected] 2024-04-06T09:43:10.218 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:43:10.223 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=108, tag_tid=17, entry_tid=16308' 2024-04-06T09:43:10.223 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=113, tag_tid=18, entry_tid=1' -a 'object_number=113, tag_tid=18, entry_tid=1' = 'object_number=108, tag_tid=17, entry_tid=16308' 2024-04-06T09:43:10.223 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=108, tag_tid=17, entry_tid=16308' '!=' 'object_number=109, tag_tid=17, entry_tid=16221' 2024-04-06T09:43:10.223 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:43:10.223 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=108, tag_tid=17, entry_tid=16308' = 'object_number=109, tag_tid=17, entry_tid=16221' 2024-04-06T09:43:10.223 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=108, tag_tid=17, entry_tid=16308' 2024-04-06T09:43:10.223 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=113, tag_tid=18, entry_tid=1' 2024-04-06T09:43:10.223 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:43:10.223 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:43:10.223 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=18 2024-04-06T09:43:10.223 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=108, tag_tid=17, entry_tid=16308' 2024-04-06T09:43:10.223 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:43:10.223 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:43:10.224 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=17 2024-04-06T09:43:10.224 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=113, tag_tid=18, entry_tid=1' 2024-04-06T09:43:10.224 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:43:10.224 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:43:10.225 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:43:10.225 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=108, tag_tid=17, entry_tid=16308' 2024-04-06T09:43:10.225 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:43:10.226 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:43:10.227 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=16308 2024-04-06T09:43:10.227 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 18 = 17 -a 1 -le 16308 2024-04-06T09:43:10.227 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:43:10.227 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:43:10.227 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:43:10.429 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:43:10.429 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:43:10.429 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:43:10.429 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:43:10.429 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:43:10.429 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:43:10.429 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:43:10.429 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:43:10.430 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:43:10.430 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:43:10.430 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:43:10.430 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:43:10.430 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:43:10.430 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:43:10.430 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:43:10.430 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:43:10.431 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:43:10.433 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:43:10.433 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:43:10.433 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:43:10.434 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:43:10.434 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:43:10.434 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:43:10.434 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:43:10.434 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:43:10.434 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:43:10.434 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:43:10.434 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:43:10.434 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:43:10.434 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:43:10.434 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:43:10.434 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:43:10.434 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:43:10.435 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:43:10.435 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:43:10.435 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:43:10.435 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:43:10.435 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:43:10.435 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:43:10.435 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:43:10.435 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:43:10.435 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:43:10.435 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:43:10.435 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:43:10.435 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:43:10.435 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:43:10.435 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:43:10.435 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:43:10.436 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:43:10.458 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:43:10.459 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:43:10.459 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:43:10.459 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:43:10.594 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:43:10.595 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:43:10.595 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:43:10.595 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:43:10.595 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:43:10.595 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:43:10.595 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:43:10.595 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:43:10.596 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:43:10.596 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:43:10.597 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:43:10.597 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:43:10.597 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:43:10.598 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:43:10.598 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:43:10.598 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:43:10.644 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 27 2024-04-06T09:43:10.645 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 28 2024-04-06T09:43:10.645 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:43:10.645 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=113, tag_tid=18, entry_tid=1], [object_number=112, tag_tid=18, entry_tid=0], [object_number=114, tag_tid=17, entry_tid=20202], [object_number=115, tag_tid=17, entry_tid=20199]]], state=connected] 2024-04-06T09:43:10.645 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=111, tag_tid=17, entry_tid=16411], [object_number=110, tag_tid=17, entry_tid=16410], [object_number=109, tag_tid=17, entry_tid=16409], [object_number=108, tag_tid=17, entry_tid=16408]]], state=connected] 2024-04-06T09:43:10.647 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:43:10.649 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=113, tag_tid=18, entry_tid=1' 2024-04-06T09:43:10.650 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:43:10.650 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:43:10.650 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:43:10.650 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:43:10.650 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:43:10.650 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:43:10.651 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:43:10.651 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:43:10.651 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:43:10.651 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:43:10.651 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:43:10.651 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:43:10.652 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:43:10.652 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:43:10.652 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:43:10.697 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 27 2024-04-06T09:43:10.697 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 28 2024-04-06T09:43:10.697 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:43:10.697 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=113, tag_tid=18, entry_tid=1], [object_number=112, tag_tid=18, entry_tid=0], [object_number=114, tag_tid=17, entry_tid=20202], [object_number=115, tag_tid=17, entry_tid=20199]]], state=connected] 2024-04-06T09:43:10.697 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=111, tag_tid=17, entry_tid=16411], [object_number=110, tag_tid=17, entry_tid=16410], [object_number=109, tag_tid=17, entry_tid=16409], [object_number=108, tag_tid=17, entry_tid=16408]]], state=connected] 2024-04-06T09:43:10.700 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:43:10.702 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=111, tag_tid=17, entry_tid=16411' 2024-04-06T09:43:10.702 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=113, tag_tid=18, entry_tid=1' -a 'object_number=113, tag_tid=18, entry_tid=1' = 'object_number=111, tag_tid=17, entry_tid=16411' 2024-04-06T09:43:10.702 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=111, tag_tid=17, entry_tid=16411' '!=' 'object_number=108, tag_tid=17, entry_tid=16308' 2024-04-06T09:43:10.702 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:43:10.703 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=111, tag_tid=17, entry_tid=16411' = 'object_number=108, tag_tid=17, entry_tid=16308' 2024-04-06T09:43:10.703 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=111, tag_tid=17, entry_tid=16411' 2024-04-06T09:43:10.703 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=113, tag_tid=18, entry_tid=1' 2024-04-06T09:43:10.703 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:43:10.703 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:43:10.703 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=18 2024-04-06T09:43:10.704 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=111, tag_tid=17, entry_tid=16411' 2024-04-06T09:43:10.704 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:43:10.704 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:43:10.705 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=17 2024-04-06T09:43:10.706 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=113, tag_tid=18, entry_tid=1' 2024-04-06T09:43:10.706 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:43:10.706 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:43:10.707 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:43:10.707 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=111, tag_tid=17, entry_tid=16411' 2024-04-06T09:43:10.707 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:43:10.708 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:43:10.709 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=16411 2024-04-06T09:43:10.709 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 18 = 17 -a 1 -le 16411 2024-04-06T09:43:10.709 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:43:10.709 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:43:10.709 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:43:10.911 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:43:10.911 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:43:10.911 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:43:10.911 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:43:10.911 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:43:10.911 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:43:10.911 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:43:10.911 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:43:10.911 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:43:10.911 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:43:10.911 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:43:10.911 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:43:10.912 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:43:10.912 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:43:10.912 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:43:10.912 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:43:10.912 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:43:10.915 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:43:10.915 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:43:10.915 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:43:10.915 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:43:10.916 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:43:10.916 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:43:10.916 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:43:10.916 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:43:10.916 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:43:10.916 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:43:10.916 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:43:10.916 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:43:10.916 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:43:10.916 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:43:10.916 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:43:10.916 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:43:10.916 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:43:10.916 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:43:10.917 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:43:10.917 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:43:10.917 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:43:10.917 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:43:10.917 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:43:10.917 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:43:10.917 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:43:10.917 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:43:10.917 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:43:10.917 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:43:10.917 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:43:10.917 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:43:10.917 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:43:10.918 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:43:10.941 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:43:10.941 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:43:10.941 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:43:10.941 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:43:11.072 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:43:11.073 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:43:11.073 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:43:11.073 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:43:11.073 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:43:11.073 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:43:11.073 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:43:11.073 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:43:11.074 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:43:11.074 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:43:11.074 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:43:11.074 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:43:11.074 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:43:11.076 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:43:11.077 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:43:11.077 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:43:11.128 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 27 2024-04-06T09:43:11.128 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 28 2024-04-06T09:43:11.128 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:43:11.128 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=113, tag_tid=18, entry_tid=1], [object_number=112, tag_tid=18, entry_tid=0], [object_number=114, tag_tid=17, entry_tid=20202], [object_number=115, tag_tid=17, entry_tid=20199]]], state=connected] 2024-04-06T09:43:11.129 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=109, tag_tid=17, entry_tid=16521], [object_number=108, tag_tid=17, entry_tid=16520], [object_number=111, tag_tid=17, entry_tid=16519], [object_number=110, tag_tid=17, entry_tid=16518]]], state=connected] 2024-04-06T09:43:11.132 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:43:11.133 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=113, tag_tid=18, entry_tid=1' 2024-04-06T09:43:11.134 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:43:11.134 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:43:11.134 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:43:11.134 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:43:11.134 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:43:11.135 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:43:11.135 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:43:11.135 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:43:11.135 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:43:11.135 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:43:11.135 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:43:11.135 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:43:11.136 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:43:11.136 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:43:11.136 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:43:11.182 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 27 2024-04-06T09:43:11.182 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 28 2024-04-06T09:43:11.182 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:43:11.182 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=113, tag_tid=18, entry_tid=1], [object_number=112, tag_tid=18, entry_tid=0], [object_number=114, tag_tid=17, entry_tid=20202], [object_number=115, tag_tid=17, entry_tid=20199]]], state=connected] 2024-04-06T09:43:11.183 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=109, tag_tid=17, entry_tid=16521], [object_number=108, tag_tid=17, entry_tid=16520], [object_number=111, tag_tid=17, entry_tid=16519], [object_number=110, tag_tid=17, entry_tid=16518]]], state=connected] 2024-04-06T09:43:11.186 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:43:11.187 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=109, tag_tid=17, entry_tid=16521' 2024-04-06T09:43:11.187 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=113, tag_tid=18, entry_tid=1' -a 'object_number=113, tag_tid=18, entry_tid=1' = 'object_number=109, tag_tid=17, entry_tid=16521' 2024-04-06T09:43:11.187 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=109, tag_tid=17, entry_tid=16521' '!=' 'object_number=111, tag_tid=17, entry_tid=16411' 2024-04-06T09:43:11.188 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:43:11.188 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=109, tag_tid=17, entry_tid=16521' = 'object_number=111, tag_tid=17, entry_tid=16411' 2024-04-06T09:43:11.188 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=109, tag_tid=17, entry_tid=16521' 2024-04-06T09:43:11.188 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=113, tag_tid=18, entry_tid=1' 2024-04-06T09:43:11.188 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:43:11.188 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:43:11.189 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=18 2024-04-06T09:43:11.190 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=109, tag_tid=17, entry_tid=16521' 2024-04-06T09:43:11.190 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:43:11.190 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:43:11.190 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=17 2024-04-06T09:43:11.191 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=113, tag_tid=18, entry_tid=1' 2024-04-06T09:43:11.191 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:43:11.191 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:43:11.192 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:43:11.193 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=109, tag_tid=17, entry_tid=16521' 2024-04-06T09:43:11.193 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:43:11.193 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:43:11.194 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=16521 2024-04-06T09:43:11.194 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 18 = 17 -a 1 -le 16521 2024-04-06T09:43:11.194 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:43:11.194 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:43:11.194 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:43:11.396 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:43:11.396 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:43:11.396 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:43:11.396 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:43:11.396 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:43:11.396 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:43:11.396 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:43:11.396 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:43:11.397 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:43:11.397 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:43:11.397 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:43:11.397 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:43:11.397 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:43:11.397 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:43:11.397 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:43:11.397 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:43:11.398 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:43:11.401 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:43:11.401 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:43:11.401 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:43:11.401 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:43:11.401 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:43:11.402 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:43:11.402 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:43:11.402 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:43:11.402 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:43:11.402 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:43:11.402 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:43:11.402 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:43:11.402 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:43:11.402 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:43:11.402 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:43:11.402 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:43:11.402 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:43:11.402 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:43:11.403 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:43:11.403 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:43:11.403 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:43:11.403 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:43:11.403 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:43:11.403 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:43:11.403 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:43:11.403 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:43:11.403 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:43:11.403 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:43:11.403 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:43:11.403 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:43:11.403 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:43:11.404 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:43:11.423 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:43:11.423 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:43:11.423 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:43:11.423 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:43:11.564 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:43:11.565 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:43:11.565 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:43:11.565 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:43:11.565 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:43:11.565 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:43:11.565 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:43:11.566 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:43:11.566 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:43:11.566 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:43:11.566 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:43:11.566 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:43:11.566 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:43:11.568 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:43:11.569 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:43:11.569 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:43:11.616 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 27 2024-04-06T09:43:11.616 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 28 2024-04-06T09:43:11.616 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:43:11.616 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=113, tag_tid=18, entry_tid=1], [object_number=112, tag_tid=18, entry_tid=0], [object_number=114, tag_tid=17, entry_tid=20202], [object_number=115, tag_tid=17, entry_tid=20199]]], state=connected] 2024-04-06T09:43:11.616 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=109, tag_tid=17, entry_tid=16621], [object_number=108, tag_tid=17, entry_tid=16620], [object_number=111, tag_tid=17, entry_tid=16619], [object_number=110, tag_tid=17, entry_tid=16618]]], state=connected] 2024-04-06T09:43:11.620 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:43:11.621 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=113, tag_tid=18, entry_tid=1' 2024-04-06T09:43:11.622 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:43:11.622 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:43:11.622 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:43:11.622 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:43:11.622 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:43:11.622 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:43:11.623 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:43:11.623 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:43:11.623 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:43:11.623 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:43:11.623 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:43:11.623 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:43:11.624 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:43:11.624 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:43:11.624 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:43:11.670 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 27 2024-04-06T09:43:11.670 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 28 2024-04-06T09:43:11.670 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:43:11.670 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=113, tag_tid=18, entry_tid=1], [object_number=112, tag_tid=18, entry_tid=0], [object_number=114, tag_tid=17, entry_tid=20202], [object_number=115, tag_tid=17, entry_tid=20199]]], state=connected] 2024-04-06T09:43:11.670 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=109, tag_tid=17, entry_tid=16621], [object_number=108, tag_tid=17, entry_tid=16620], [object_number=111, tag_tid=17, entry_tid=16619], [object_number=110, tag_tid=17, entry_tid=16618]]], state=connected] 2024-04-06T09:43:11.673 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:43:11.674 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=109, tag_tid=17, entry_tid=16621' 2024-04-06T09:43:11.675 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=113, tag_tid=18, entry_tid=1' -a 'object_number=113, tag_tid=18, entry_tid=1' = 'object_number=109, tag_tid=17, entry_tid=16621' 2024-04-06T09:43:11.675 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=109, tag_tid=17, entry_tid=16621' '!=' 'object_number=109, tag_tid=17, entry_tid=16521' 2024-04-06T09:43:11.675 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:43:11.675 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=109, tag_tid=17, entry_tid=16621' = 'object_number=109, tag_tid=17, entry_tid=16521' 2024-04-06T09:43:11.675 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=109, tag_tid=17, entry_tid=16621' 2024-04-06T09:43:11.675 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=113, tag_tid=18, entry_tid=1' 2024-04-06T09:43:11.676 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:43:11.676 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:43:11.676 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=18 2024-04-06T09:43:11.677 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=109, tag_tid=17, entry_tid=16621' 2024-04-06T09:43:11.677 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:43:11.677 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:43:11.678 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=17 2024-04-06T09:43:11.678 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=113, tag_tid=18, entry_tid=1' 2024-04-06T09:43:11.679 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:43:11.679 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:43:11.679 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:43:11.680 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=109, tag_tid=17, entry_tid=16621' 2024-04-06T09:43:11.680 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:43:11.680 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:43:11.681 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=16621 2024-04-06T09:43:11.682 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 18 = 17 -a 1 -le 16621 2024-04-06T09:43:11.682 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:43:11.682 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:43:11.682 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:43:11.883 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:43:11.883 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:43:11.883 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:43:11.884 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:43:11.884 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:43:11.884 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:43:11.884 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:43:11.884 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:43:11.884 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:43:11.884 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:43:11.884 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:43:11.884 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:43:11.884 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:43:11.884 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:43:11.884 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:43:11.885 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:43:11.885 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:43:11.888 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:43:11.888 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:43:11.888 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:43:11.888 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:43:11.888 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:43:11.888 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:43:11.889 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:43:11.889 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:43:11.889 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:43:11.889 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:43:11.889 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:43:11.889 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:43:11.889 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:43:11.889 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:43:11.889 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:43:11.889 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:43:11.889 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:43:11.889 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:43:11.889 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:43:11.889 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:43:11.889 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:43:11.890 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:43:11.890 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:43:11.890 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:43:11.890 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:43:11.890 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:43:11.890 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:43:11.890 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:43:11.890 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:43:11.890 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:43:11.890 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:43:11.890 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:43:11.914 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:43:11.914 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:43:11.914 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:43:11.914 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:43:12.048 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:43:12.049 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:43:12.049 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:43:12.050 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:43:12.050 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:43:12.050 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:43:12.050 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:43:12.050 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:43:12.050 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:43:12.050 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:43:12.050 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:43:12.051 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:43:12.051 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:43:12.053 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:43:12.053 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:43:12.054 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:43:12.105 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 27 2024-04-06T09:43:12.105 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 28 2024-04-06T09:43:12.105 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:43:12.105 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=113, tag_tid=18, entry_tid=1], [object_number=112, tag_tid=18, entry_tid=0], [object_number=114, tag_tid=17, entry_tid=20202], [object_number=115, tag_tid=17, entry_tid=20199]]], state=connected] 2024-04-06T09:43:12.105 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=109, tag_tid=17, entry_tid=16717], [object_number=108, tag_tid=17, entry_tid=16716], [object_number=111, tag_tid=17, entry_tid=16715], [object_number=110, tag_tid=17, entry_tid=16714]]], state=connected] 2024-04-06T09:43:12.109 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:43:12.110 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=113, tag_tid=18, entry_tid=1' 2024-04-06T09:43:12.111 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:43:12.111 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:43:12.111 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:43:12.111 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:43:12.111 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:43:12.111 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:43:12.112 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:43:12.112 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:43:12.112 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:43:12.112 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:43:12.112 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:43:12.112 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:43:12.112 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:43:12.113 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:43:12.113 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:43:12.160 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 27 2024-04-06T09:43:12.160 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 28 2024-04-06T09:43:12.160 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:43:12.160 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=113, tag_tid=18, entry_tid=1], [object_number=112, tag_tid=18, entry_tid=0], [object_number=114, tag_tid=17, entry_tid=20202], [object_number=115, tag_tid=17, entry_tid=20199]]], state=connected] 2024-04-06T09:43:12.160 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=109, tag_tid=17, entry_tid=16717], [object_number=108, tag_tid=17, entry_tid=16716], [object_number=111, tag_tid=17, entry_tid=16715], [object_number=110, tag_tid=17, entry_tid=16714]]], state=connected] 2024-04-06T09:43:12.163 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:43:12.165 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=109, tag_tid=17, entry_tid=16717' 2024-04-06T09:43:12.166 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=113, tag_tid=18, entry_tid=1' -a 'object_number=113, tag_tid=18, entry_tid=1' = 'object_number=109, tag_tid=17, entry_tid=16717' 2024-04-06T09:43:12.166 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=109, tag_tid=17, entry_tid=16717' '!=' 'object_number=109, tag_tid=17, entry_tid=16621' 2024-04-06T09:43:12.166 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:43:12.166 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=109, tag_tid=17, entry_tid=16717' = 'object_number=109, tag_tid=17, entry_tid=16621' 2024-04-06T09:43:12.166 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=109, tag_tid=17, entry_tid=16717' 2024-04-06T09:43:12.166 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=113, tag_tid=18, entry_tid=1' 2024-04-06T09:43:12.166 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:43:12.166 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:43:12.167 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=18 2024-04-06T09:43:12.167 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=109, tag_tid=17, entry_tid=16717' 2024-04-06T09:43:12.167 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:43:12.168 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:43:12.169 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=17 2024-04-06T09:43:12.169 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=113, tag_tid=18, entry_tid=1' 2024-04-06T09:43:12.169 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:43:12.169 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:43:12.170 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:43:12.171 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=109, tag_tid=17, entry_tid=16717' 2024-04-06T09:43:12.171 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:43:12.171 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:43:12.173 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=16717 2024-04-06T09:43:12.173 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 18 = 17 -a 1 -le 16717 2024-04-06T09:43:12.173 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:43:12.173 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:43:12.173 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:43:12.374 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:43:12.375 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:43:12.375 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:43:12.375 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:43:12.375 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:43:12.375 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:43:12.375 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:43:12.375 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:43:12.375 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:43:12.375 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:43:12.375 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:43:12.375 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:43:12.375 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:43:12.375 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:43:12.375 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:43:12.376 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:43:12.376 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:43:12.378 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:43:12.378 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:43:12.378 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:43:12.378 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:43:12.378 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:43:12.379 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:43:12.379 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:43:12.379 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:43:12.379 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:43:12.379 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:43:12.379 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:43:12.379 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:43:12.379 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:43:12.379 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:43:12.379 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:43:12.379 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:43:12.380 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:43:12.381 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:43:12.381 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:43:12.381 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:43:12.381 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:43:12.381 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:43:12.381 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:43:12.381 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:43:12.381 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:43:12.381 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:43:12.381 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:43:12.381 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:43:12.381 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:43:12.381 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:43:12.381 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:43:12.382 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:43:12.407 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:43:12.408 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:43:12.408 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:43:12.408 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:43:12.543 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:43:12.544 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:43:12.544 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:43:12.544 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:43:12.544 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:43:12.545 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:43:12.545 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:43:12.545 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:43:12.545 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:43:12.545 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:43:12.545 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:43:12.545 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:43:12.546 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:43:12.547 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:43:12.547 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:43:12.547 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:43:12.593 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 27 2024-04-06T09:43:12.593 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 28 2024-04-06T09:43:12.593 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:43:12.593 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=113, tag_tid=18, entry_tid=1], [object_number=112, tag_tid=18, entry_tid=0], [object_number=114, tag_tid=17, entry_tid=20202], [object_number=115, tag_tid=17, entry_tid=20199]]], state=connected] 2024-04-06T09:43:12.593 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=109, tag_tid=17, entry_tid=16813], [object_number=108, tag_tid=17, entry_tid=16812], [object_number=111, tag_tid=17, entry_tid=16811], [object_number=110, tag_tid=17, entry_tid=16810]]], state=connected] 2024-04-06T09:43:12.596 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:43:12.597 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=113, tag_tid=18, entry_tid=1' 2024-04-06T09:43:12.599 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:43:12.599 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:43:12.599 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:43:12.599 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:43:12.599 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:43:12.599 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:43:12.599 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:43:12.599 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:43:12.599 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:43:12.599 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:43:12.599 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:43:12.600 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:43:12.600 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:43:12.600 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:43:12.600 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:43:12.647 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 27 2024-04-06T09:43:12.647 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 28 2024-04-06T09:43:12.647 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:43:12.647 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=113, tag_tid=18, entry_tid=1], [object_number=112, tag_tid=18, entry_tid=0], [object_number=114, tag_tid=17, entry_tid=20202], [object_number=115, tag_tid=17, entry_tid=20199]]], state=connected] 2024-04-06T09:43:12.647 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=109, tag_tid=17, entry_tid=16813], [object_number=108, tag_tid=17, entry_tid=16812], [object_number=111, tag_tid=17, entry_tid=16811], [object_number=110, tag_tid=17, entry_tid=16810]]], state=connected] 2024-04-06T09:43:12.650 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:43:12.652 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=109, tag_tid=17, entry_tid=16813' 2024-04-06T09:43:12.652 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=113, tag_tid=18, entry_tid=1' -a 'object_number=113, tag_tid=18, entry_tid=1' = 'object_number=109, tag_tid=17, entry_tid=16813' 2024-04-06T09:43:12.652 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=109, tag_tid=17, entry_tid=16813' '!=' 'object_number=109, tag_tid=17, entry_tid=16717' 2024-04-06T09:43:12.652 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:43:12.652 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=109, tag_tid=17, entry_tid=16813' = 'object_number=109, tag_tid=17, entry_tid=16717' 2024-04-06T09:43:12.652 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=109, tag_tid=17, entry_tid=16813' 2024-04-06T09:43:12.653 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=113, tag_tid=18, entry_tid=1' 2024-04-06T09:43:12.653 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:43:12.653 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:43:12.654 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=18 2024-04-06T09:43:12.654 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=109, tag_tid=17, entry_tid=16813' 2024-04-06T09:43:12.654 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:43:12.655 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:43:12.655 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=17 2024-04-06T09:43:12.656 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=113, tag_tid=18, entry_tid=1' 2024-04-06T09:43:12.656 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:43:12.656 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:43:12.657 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:43:12.658 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=109, tag_tid=17, entry_tid=16813' 2024-04-06T09:43:12.658 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:43:12.658 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:43:12.659 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=16813 2024-04-06T09:43:12.659 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 18 = 17 -a 1 -le 16813 2024-04-06T09:43:12.659 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:43:12.659 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:43:12.659 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:43:12.861 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:43:12.861 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:43:12.861 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:43:12.861 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:43:12.861 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:43:12.861 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:43:12.861 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:43:12.861 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:43:12.862 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:43:12.862 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:43:12.862 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:43:12.862 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:43:12.862 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:43:12.862 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:43:12.862 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:43:12.862 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:43:12.863 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:43:12.864 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:43:12.864 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:43:12.865 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:43:12.865 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:43:12.865 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:43:12.865 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:43:12.865 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:43:12.865 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:43:12.865 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:43:12.865 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:43:12.865 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:43:12.865 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:43:12.865 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:43:12.865 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:43:12.865 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:43:12.865 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:43:12.866 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:43:12.867 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:43:12.867 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:43:12.867 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:43:12.867 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:43:12.867 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:43:12.867 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:43:12.867 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:43:12.867 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:43:12.867 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:43:12.867 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:43:12.867 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:43:12.867 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:43:12.867 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:43:12.867 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:43:12.868 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:43:12.891 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:43:12.892 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:43:12.892 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:43:12.892 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:43:13.027 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:43:13.028 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:43:13.028 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:43:13.028 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:43:13.028 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:43:13.028 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:43:13.028 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:43:13.029 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:43:13.029 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:43:13.029 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:43:13.029 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:43:13.029 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:43:13.030 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:43:13.030 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:43:13.031 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:43:13.031 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:43:13.084 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 27 2024-04-06T09:43:13.084 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 28 2024-04-06T09:43:13.084 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:43:13.084 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=113, tag_tid=18, entry_tid=1], [object_number=112, tag_tid=18, entry_tid=0], [object_number=114, tag_tid=17, entry_tid=20202], [object_number=115, tag_tid=17, entry_tid=20199]]], state=connected] 2024-04-06T09:43:13.084 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=109, tag_tid=17, entry_tid=16901], [object_number=108, tag_tid=17, entry_tid=16900], [object_number=111, tag_tid=17, entry_tid=16899], [object_number=110, tag_tid=17, entry_tid=16898]]], state=connected] 2024-04-06T09:43:13.087 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:43:13.089 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=113, tag_tid=18, entry_tid=1' 2024-04-06T09:43:13.090 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:43:13.090 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:43:13.090 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:43:13.090 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:43:13.090 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:43:13.090 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:43:13.090 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:43:13.090 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:43:13.091 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:43:13.091 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:43:13.091 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:43:13.091 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:43:13.091 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:43:13.091 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:43:13.092 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:43:13.139 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 27 2024-04-06T09:43:13.139 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 28 2024-04-06T09:43:13.139 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:43:13.140 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=113, tag_tid=18, entry_tid=1], [object_number=112, tag_tid=18, entry_tid=0], [object_number=114, tag_tid=17, entry_tid=20202], [object_number=115, tag_tid=17, entry_tid=20199]]], state=connected] 2024-04-06T09:43:13.140 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=109, tag_tid=17, entry_tid=16901], [object_number=108, tag_tid=17, entry_tid=16900], [object_number=111, tag_tid=17, entry_tid=16899], [object_number=110, tag_tid=17, entry_tid=16898]]], state=connected] 2024-04-06T09:43:13.143 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:43:13.145 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=109, tag_tid=17, entry_tid=16901' 2024-04-06T09:43:13.145 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=113, tag_tid=18, entry_tid=1' -a 'object_number=113, tag_tid=18, entry_tid=1' = 'object_number=109, tag_tid=17, entry_tid=16901' 2024-04-06T09:43:13.145 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=109, tag_tid=17, entry_tid=16901' '!=' 'object_number=109, tag_tid=17, entry_tid=16813' 2024-04-06T09:43:13.145 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:43:13.145 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=109, tag_tid=17, entry_tid=16901' = 'object_number=109, tag_tid=17, entry_tid=16813' 2024-04-06T09:43:13.145 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=109, tag_tid=17, entry_tid=16901' 2024-04-06T09:43:13.146 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=113, tag_tid=18, entry_tid=1' 2024-04-06T09:43:13.146 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:43:13.146 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:43:13.146 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=18 2024-04-06T09:43:13.147 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=109, tag_tid=17, entry_tid=16901' 2024-04-06T09:43:13.147 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:43:13.147 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:43:13.148 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=17 2024-04-06T09:43:13.149 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=113, tag_tid=18, entry_tid=1' 2024-04-06T09:43:13.149 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:43:13.149 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:43:13.150 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:43:13.151 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=109, tag_tid=17, entry_tid=16901' 2024-04-06T09:43:13.151 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:43:13.151 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:43:13.152 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=16901 2024-04-06T09:43:13.152 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 18 = 17 -a 1 -le 16901 2024-04-06T09:43:13.152 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:43:13.152 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:43:13.152 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:43:13.354 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:43:13.354 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:43:13.354 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:43:13.354 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:43:13.354 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:43:13.355 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:43:13.355 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:43:13.355 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:43:13.355 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:43:13.355 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:43:13.355 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:43:13.355 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:43:13.355 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:43:13.355 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:43:13.355 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:43:13.355 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:43:13.356 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:43:13.358 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:43:13.358 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:43:13.358 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:43:13.358 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:43:13.359 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:43:13.359 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:43:13.359 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:43:13.359 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:43:13.359 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:43:13.359 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:43:13.359 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:43:13.359 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:43:13.359 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:43:13.359 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:43:13.359 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:43:13.359 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:43:13.359 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:43:13.359 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:43:13.360 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:43:13.360 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:43:13.360 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:43:13.360 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:43:13.360 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:43:13.360 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:43:13.360 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:43:13.360 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:43:13.360 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:43:13.360 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:43:13.360 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:43:13.360 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:43:13.360 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:43:13.361 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:43:13.385 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:43:13.385 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:43:13.385 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:43:13.386 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:43:13.515 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:43:13.516 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:43:13.516 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:43:13.516 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:43:13.516 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:43:13.517 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:43:13.517 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:43:13.517 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:43:13.517 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:43:13.517 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:43:13.517 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:43:13.517 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:43:13.518 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:43:13.519 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:43:13.519 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:43:13.519 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:43:13.567 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 27 2024-04-06T09:43:13.567 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 28 2024-04-06T09:43:13.567 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:43:13.567 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=113, tag_tid=18, entry_tid=1], [object_number=112, tag_tid=18, entry_tid=0], [object_number=114, tag_tid=17, entry_tid=20202], [object_number=115, tag_tid=17, entry_tid=20199]]], state=connected] 2024-04-06T09:43:13.567 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=111, tag_tid=17, entry_tid=16995], [object_number=110, tag_tid=17, entry_tid=16994], [object_number=109, tag_tid=17, entry_tid=16993], [object_number=108, tag_tid=17, entry_tid=16992]]], state=connected] 2024-04-06T09:43:13.570 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:43:13.572 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=113, tag_tid=18, entry_tid=1' 2024-04-06T09:43:13.572 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:43:13.572 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:43:13.572 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:43:13.572 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:43:13.573 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:43:13.573 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:43:13.573 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:43:13.573 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:43:13.573 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:43:13.573 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:43:13.573 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:43:13.573 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:43:13.574 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:43:13.574 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:43:13.574 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:43:13.620 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 27 2024-04-06T09:43:13.620 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 28 2024-04-06T09:43:13.620 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:43:13.620 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=113, tag_tid=18, entry_tid=1], [object_number=112, tag_tid=18, entry_tid=0], [object_number=114, tag_tid=17, entry_tid=20202], [object_number=115, tag_tid=17, entry_tid=20199]]], state=connected] 2024-04-06T09:43:13.620 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=111, tag_tid=17, entry_tid=16995], [object_number=110, tag_tid=17, entry_tid=16994], [object_number=109, tag_tid=17, entry_tid=16993], [object_number=108, tag_tid=17, entry_tid=16992]]], state=connected] 2024-04-06T09:43:13.623 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:43:13.625 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=111, tag_tid=17, entry_tid=16995' 2024-04-06T09:43:13.625 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=113, tag_tid=18, entry_tid=1' -a 'object_number=113, tag_tid=18, entry_tid=1' = 'object_number=111, tag_tid=17, entry_tid=16995' 2024-04-06T09:43:13.626 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=111, tag_tid=17, entry_tid=16995' '!=' 'object_number=109, tag_tid=17, entry_tid=16901' 2024-04-06T09:43:13.626 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:43:13.626 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=111, tag_tid=17, entry_tid=16995' = 'object_number=109, tag_tid=17, entry_tid=16901' 2024-04-06T09:43:13.626 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=111, tag_tid=17, entry_tid=16995' 2024-04-06T09:43:13.626 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=113, tag_tid=18, entry_tid=1' 2024-04-06T09:43:13.626 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:43:13.626 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:43:13.626 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=18 2024-04-06T09:43:13.627 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=111, tag_tid=17, entry_tid=16995' 2024-04-06T09:43:13.627 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:43:13.627 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:43:13.628 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=17 2024-04-06T09:43:13.629 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=113, tag_tid=18, entry_tid=1' 2024-04-06T09:43:13.629 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:43:13.629 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:43:13.630 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:43:13.631 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=111, tag_tid=17, entry_tid=16995' 2024-04-06T09:43:13.631 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:43:13.631 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:43:13.632 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=16995 2024-04-06T09:43:13.632 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 18 = 17 -a 1 -le 16995 2024-04-06T09:43:13.632 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:43:13.632 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:43:13.632 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:43:13.834 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:43:13.834 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:43:13.834 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:43:13.834 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:43:13.834 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:43:13.834 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:43:13.834 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:43:13.834 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:43:13.834 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:43:13.835 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:43:13.835 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:43:13.835 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:43:13.835 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:43:13.835 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:43:13.835 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:43:13.835 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:43:13.835 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:43:13.839 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:43:13.839 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:43:13.839 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:43:13.839 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:43:13.839 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:43:13.840 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:43:13.840 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:43:13.840 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:43:13.840 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:43:13.840 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:43:13.840 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:43:13.840 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:43:13.840 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:43:13.840 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:43:13.840 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:43:13.840 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:43:13.840 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:43:13.840 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:43:13.841 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:43:13.841 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:43:13.841 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:43:13.841 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:43:13.841 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:43:13.841 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:43:13.841 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:43:13.841 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:43:13.841 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:43:13.841 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:43:13.842 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:43:13.842 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:43:13.842 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:43:13.842 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:43:13.864 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:43:13.864 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:43:13.864 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:43:13.864 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:43:13.993 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:43:13.994 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:43:13.994 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:43:13.994 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:43:13.994 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:43:13.995 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:43:13.995 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:43:13.995 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:43:13.995 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:43:13.995 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:43:13.995 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:43:13.995 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:43:13.995 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:43:13.997 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:43:13.998 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:43:13.998 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:43:14.052 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 27 2024-04-06T09:43:14.052 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 28 2024-04-06T09:43:14.052 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:43:14.052 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=113, tag_tid=18, entry_tid=1], [object_number=112, tag_tid=18, entry_tid=0], [object_number=114, tag_tid=17, entry_tid=20202], [object_number=115, tag_tid=17, entry_tid=20199]]], state=connected] 2024-04-06T09:43:14.053 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=110, tag_tid=17, entry_tid=17086], [object_number=109, tag_tid=17, entry_tid=17085], [object_number=108, tag_tid=17, entry_tid=17084], [object_number=111, tag_tid=17, entry_tid=17083]]], state=connected] 2024-04-06T09:43:14.056 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:43:14.057 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=113, tag_tid=18, entry_tid=1' 2024-04-06T09:43:14.058 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:43:14.058 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:43:14.058 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:43:14.058 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:43:14.059 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:43:14.059 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:43:14.059 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:43:14.059 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:43:14.059 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:43:14.059 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:43:14.059 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:43:14.059 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:43:14.060 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:43:14.060 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:43:14.060 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:43:14.107 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 27 2024-04-06T09:43:14.107 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 28 2024-04-06T09:43:14.108 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:43:14.108 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=113, tag_tid=18, entry_tid=1], [object_number=112, tag_tid=18, entry_tid=0], [object_number=114, tag_tid=17, entry_tid=20202], [object_number=115, tag_tid=17, entry_tid=20199]]], state=connected] 2024-04-06T09:43:14.108 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=110, tag_tid=17, entry_tid=17086], [object_number=109, tag_tid=17, entry_tid=17085], [object_number=108, tag_tid=17, entry_tid=17084], [object_number=111, tag_tid=17, entry_tid=17083]]], state=connected] 2024-04-06T09:43:14.111 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:43:14.113 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=110, tag_tid=17, entry_tid=17086' 2024-04-06T09:43:14.114 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=113, tag_tid=18, entry_tid=1' -a 'object_number=113, tag_tid=18, entry_tid=1' = 'object_number=110, tag_tid=17, entry_tid=17086' 2024-04-06T09:43:14.114 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=110, tag_tid=17, entry_tid=17086' '!=' 'object_number=111, tag_tid=17, entry_tid=16995' 2024-04-06T09:43:14.114 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:43:14.114 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=110, tag_tid=17, entry_tid=17086' = 'object_number=111, tag_tid=17, entry_tid=16995' 2024-04-06T09:43:14.114 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=110, tag_tid=17, entry_tid=17086' 2024-04-06T09:43:14.114 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=113, tag_tid=18, entry_tid=1' 2024-04-06T09:43:14.114 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:43:14.114 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:43:14.115 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=18 2024-04-06T09:43:14.115 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=110, tag_tid=17, entry_tid=17086' 2024-04-06T09:43:14.115 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:43:14.115 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:43:14.116 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=17 2024-04-06T09:43:14.117 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=113, tag_tid=18, entry_tid=1' 2024-04-06T09:43:14.117 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:43:14.117 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:43:14.118 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:43:14.119 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=110, tag_tid=17, entry_tid=17086' 2024-04-06T09:43:14.119 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:43:14.119 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:43:14.120 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=17086 2024-04-06T09:43:14.120 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 18 = 17 -a 1 -le 17086 2024-04-06T09:43:14.120 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:43:14.120 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:43:14.120 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:43:14.322 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:43:14.322 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:43:14.322 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:43:14.322 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:43:14.322 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:43:14.322 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:43:14.322 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:43:14.322 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:43:14.322 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:43:14.322 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:43:14.323 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:43:14.323 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:43:14.323 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:43:14.323 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:43:14.323 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:43:14.323 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:43:14.324 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:43:14.326 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:43:14.326 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:43:14.326 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:43:14.326 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:43:14.326 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:43:14.326 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:43:14.327 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:43:14.327 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:43:14.327 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:43:14.327 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:43:14.327 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:43:14.327 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:43:14.327 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:43:14.327 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:43:14.327 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:43:14.327 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:43:14.328 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:43:14.329 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:43:14.329 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:43:14.329 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:43:14.329 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:43:14.329 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:43:14.329 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:43:14.329 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:43:14.329 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:43:14.329 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:43:14.329 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:43:14.329 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:43:14.329 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:43:14.329 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:43:14.330 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:43:14.330 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:43:14.354 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:43:14.354 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:43:14.354 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:43:14.354 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:43:14.492 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:43:14.493 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:43:14.493 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:43:14.493 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:43:14.493 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:43:14.493 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:43:14.493 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:43:14.493 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:43:14.494 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:43:14.494 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:43:14.494 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:43:14.494 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:43:14.494 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:43:14.496 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:43:14.497 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:43:14.497 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:43:14.552 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 27 2024-04-06T09:43:14.552 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 28 2024-04-06T09:43:14.552 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:43:14.552 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=113, tag_tid=18, entry_tid=1], [object_number=112, tag_tid=18, entry_tid=0], [object_number=114, tag_tid=17, entry_tid=20202], [object_number=115, tag_tid=17, entry_tid=20199]]], state=connected] 2024-04-06T09:43:14.552 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=109, tag_tid=17, entry_tid=17177], [object_number=108, tag_tid=17, entry_tid=17176], [object_number=111, tag_tid=17, entry_tid=17175], [object_number=110, tag_tid=17, entry_tid=17174]]], state=connected] 2024-04-06T09:43:14.555 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:43:14.556 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=113, tag_tid=18, entry_tid=1' 2024-04-06T09:43:14.558 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:43:14.558 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:43:14.558 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:43:14.558 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:43:14.558 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:43:14.558 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:43:14.558 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:43:14.558 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:43:14.558 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:43:14.558 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:43:14.559 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:43:14.559 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:43:14.559 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:43:14.559 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:43:14.560 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:43:14.606 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 27 2024-04-06T09:43:14.606 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 28 2024-04-06T09:43:14.606 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:43:14.606 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=113, tag_tid=18, entry_tid=1], [object_number=112, tag_tid=18, entry_tid=0], [object_number=114, tag_tid=17, entry_tid=20202], [object_number=115, tag_tid=17, entry_tid=20199]]], state=connected] 2024-04-06T09:43:14.607 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=109, tag_tid=17, entry_tid=17177], [object_number=108, tag_tid=17, entry_tid=17176], [object_number=111, tag_tid=17, entry_tid=17175], [object_number=110, tag_tid=17, entry_tid=17174]]], state=connected] 2024-04-06T09:43:14.610 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:43:14.612 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=109, tag_tid=17, entry_tid=17177' 2024-04-06T09:43:14.612 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=113, tag_tid=18, entry_tid=1' -a 'object_number=113, tag_tid=18, entry_tid=1' = 'object_number=109, tag_tid=17, entry_tid=17177' 2024-04-06T09:43:14.612 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=109, tag_tid=17, entry_tid=17177' '!=' 'object_number=110, tag_tid=17, entry_tid=17086' 2024-04-06T09:43:14.612 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:43:14.612 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=109, tag_tid=17, entry_tid=17177' = 'object_number=110, tag_tid=17, entry_tid=17086' 2024-04-06T09:43:14.612 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=109, tag_tid=17, entry_tid=17177' 2024-04-06T09:43:14.613 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=113, tag_tid=18, entry_tid=1' 2024-04-06T09:43:14.613 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:43:14.613 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:43:14.613 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=18 2024-04-06T09:43:14.614 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=109, tag_tid=17, entry_tid=17177' 2024-04-06T09:43:14.614 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:43:14.614 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:43:14.615 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=17 2024-04-06T09:43:14.616 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=113, tag_tid=18, entry_tid=1' 2024-04-06T09:43:14.616 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:43:14.616 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:43:14.617 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:43:14.617 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=109, tag_tid=17, entry_tid=17177' 2024-04-06T09:43:14.617 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:43:14.618 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:43:14.619 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=17177 2024-04-06T09:43:14.619 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 18 = 17 -a 1 -le 17177 2024-04-06T09:43:14.619 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:43:14.619 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:43:14.619 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:43:14.821 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:43:14.821 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:43:14.821 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:43:14.821 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:43:14.821 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:43:14.821 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:43:14.821 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:43:14.821 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:43:14.821 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:43:14.821 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:43:14.821 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:43:14.821 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:43:14.822 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:43:14.822 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:43:14.822 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:43:14.822 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:43:14.822 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:43:14.825 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:43:14.825 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:43:14.825 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:43:14.825 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:43:14.825 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:43:14.825 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:43:14.825 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:43:14.825 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:43:14.825 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:43:14.825 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:43:14.825 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:43:14.825 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:43:14.825 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:43:14.825 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:43:14.825 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:43:14.826 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:43:14.826 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:43:14.826 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:43:14.826 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:43:14.826 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:43:14.827 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:43:14.827 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:43:14.827 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:43:14.827 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:43:14.827 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:43:14.827 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:43:14.827 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:43:14.827 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:43:14.827 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:43:14.827 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:43:14.827 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:43:14.827 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:43:14.851 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:43:14.851 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:43:14.851 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:43:14.851 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:43:14.975 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:43:14.976 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:43:14.976 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:43:14.976 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:43:14.976 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:43:14.976 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:43:14.976 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:43:14.976 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:43:14.976 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:43:14.977 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:43:14.977 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:43:14.977 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:43:14.977 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:43:14.978 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:43:14.979 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:43:14.979 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:43:15.025 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 27 2024-04-06T09:43:15.025 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 28 2024-04-06T09:43:15.025 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:43:15.025 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=113, tag_tid=18, entry_tid=1], [object_number=112, tag_tid=18, entry_tid=0], [object_number=114, tag_tid=17, entry_tid=20202], [object_number=115, tag_tid=17, entry_tid=20199]]], state=connected] 2024-04-06T09:43:15.025 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=109, tag_tid=17, entry_tid=17269], [object_number=108, tag_tid=17, entry_tid=17268], [object_number=111, tag_tid=17, entry_tid=17267], [object_number=110, tag_tid=17, entry_tid=17266]]], state=connected] 2024-04-06T09:43:15.028 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:43:15.029 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=113, tag_tid=18, entry_tid=1' 2024-04-06T09:43:15.030 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:43:15.030 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:43:15.030 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:43:15.030 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:43:15.031 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:43:15.031 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:43:15.031 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:43:15.031 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:43:15.031 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:43:15.031 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:43:15.031 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:43:15.031 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:43:15.032 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:43:15.032 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:43:15.032 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:43:15.076 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 27 2024-04-06T09:43:15.076 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 28 2024-04-06T09:43:15.076 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:43:15.076 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=113, tag_tid=18, entry_tid=1], [object_number=112, tag_tid=18, entry_tid=0], [object_number=114, tag_tid=17, entry_tid=20202], [object_number=115, tag_tid=17, entry_tid=20199]]], state=connected] 2024-04-06T09:43:15.076 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=109, tag_tid=17, entry_tid=17269], [object_number=108, tag_tid=17, entry_tid=17268], [object_number=111, tag_tid=17, entry_tid=17267], [object_number=110, tag_tid=17, entry_tid=17266]]], state=connected] 2024-04-06T09:43:15.079 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:43:15.081 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=109, tag_tid=17, entry_tid=17269' 2024-04-06T09:43:15.081 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=113, tag_tid=18, entry_tid=1' -a 'object_number=113, tag_tid=18, entry_tid=1' = 'object_number=109, tag_tid=17, entry_tid=17269' 2024-04-06T09:43:15.081 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=109, tag_tid=17, entry_tid=17269' '!=' 'object_number=109, tag_tid=17, entry_tid=17177' 2024-04-06T09:43:15.081 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:43:15.081 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=109, tag_tid=17, entry_tid=17269' = 'object_number=109, tag_tid=17, entry_tid=17177' 2024-04-06T09:43:15.081 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=109, tag_tid=17, entry_tid=17269' 2024-04-06T09:43:15.082 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=113, tag_tid=18, entry_tid=1' 2024-04-06T09:43:15.082 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:43:15.082 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:43:15.082 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=18 2024-04-06T09:43:15.083 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=109, tag_tid=17, entry_tid=17269' 2024-04-06T09:43:15.083 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:43:15.083 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:43:15.084 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=17 2024-04-06T09:43:15.084 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=113, tag_tid=18, entry_tid=1' 2024-04-06T09:43:15.084 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:43:15.084 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:43:15.085 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:43:15.086 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=109, tag_tid=17, entry_tid=17269' 2024-04-06T09:43:15.086 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:43:15.086 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:43:15.087 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=17269 2024-04-06T09:43:15.087 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 18 = 17 -a 1 -le 17269 2024-04-06T09:43:15.087 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:43:15.087 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:43:15.087 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:43:15.289 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:43:15.289 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:43:15.289 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:43:15.289 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:43:15.289 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:43:15.289 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:43:15.289 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:43:15.289 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:43:15.289 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:43:15.289 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:43:15.290 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:43:15.290 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:43:15.290 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:43:15.290 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:43:15.290 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:43:15.290 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:43:15.290 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:43:15.293 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:43:15.293 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:43:15.294 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:43:15.294 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:43:15.294 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:43:15.294 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:43:15.294 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:43:15.294 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:43:15.294 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:43:15.294 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:43:15.294 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:43:15.294 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:43:15.294 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:43:15.294 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:43:15.294 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:43:15.294 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:43:15.295 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:43:15.295 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:43:15.295 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:43:15.295 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:43:15.295 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:43:15.295 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:43:15.295 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:43:15.295 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:43:15.295 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:43:15.295 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:43:15.295 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:43:15.295 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:43:15.295 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:43:15.295 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:43:15.295 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:43:15.296 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:43:15.317 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:43:15.318 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:43:15.318 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:43:15.318 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:43:15.441 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:43:15.442 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:43:15.442 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:43:15.443 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:43:15.443 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:43:15.443 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:43:15.443 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:43:15.443 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:43:15.443 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:43:15.443 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:43:15.443 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:43:15.443 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:43:15.443 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:43:15.444 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:43:15.444 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:43:15.445 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:43:15.490 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 27 2024-04-06T09:43:15.490 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 28 2024-04-06T09:43:15.490 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:43:15.490 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=113, tag_tid=18, entry_tid=1], [object_number=112, tag_tid=18, entry_tid=0], [object_number=114, tag_tid=17, entry_tid=20202], [object_number=115, tag_tid=17, entry_tid=20199]]], state=connected] 2024-04-06T09:43:15.490 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=111, tag_tid=17, entry_tid=17355], [object_number=110, tag_tid=17, entry_tid=17354], [object_number=109, tag_tid=17, entry_tid=17353], [object_number=108, tag_tid=17, entry_tid=17352]]], state=connected] 2024-04-06T09:43:15.493 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:43:15.494 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=113, tag_tid=18, entry_tid=1' 2024-04-06T09:43:15.496 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:43:15.496 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:43:15.496 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:43:15.496 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:43:15.496 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:43:15.496 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:43:15.496 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:43:15.496 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:43:15.496 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:43:15.496 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:43:15.496 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:43:15.496 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:43:15.497 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:43:15.497 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:43:15.497 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:43:15.544 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 27 2024-04-06T09:43:15.544 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 28 2024-04-06T09:43:15.544 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:43:15.544 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=113, tag_tid=18, entry_tid=1], [object_number=112, tag_tid=18, entry_tid=0], [object_number=114, tag_tid=17, entry_tid=20202], [object_number=115, tag_tid=17, entry_tid=20199]]], state=connected] 2024-04-06T09:43:15.544 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=111, tag_tid=17, entry_tid=17355], [object_number=110, tag_tid=17, entry_tid=17354], [object_number=109, tag_tid=17, entry_tid=17353], [object_number=108, tag_tid=17, entry_tid=17352]]], state=connected] 2024-04-06T09:43:15.547 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:43:15.549 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=111, tag_tid=17, entry_tid=17355' 2024-04-06T09:43:15.549 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=113, tag_tid=18, entry_tid=1' -a 'object_number=113, tag_tid=18, entry_tid=1' = 'object_number=111, tag_tid=17, entry_tid=17355' 2024-04-06T09:43:15.549 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=111, tag_tid=17, entry_tid=17355' '!=' 'object_number=109, tag_tid=17, entry_tid=17269' 2024-04-06T09:43:15.549 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:43:15.549 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=111, tag_tid=17, entry_tid=17355' = 'object_number=109, tag_tid=17, entry_tid=17269' 2024-04-06T09:43:15.549 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=111, tag_tid=17, entry_tid=17355' 2024-04-06T09:43:15.549 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=113, tag_tid=18, entry_tid=1' 2024-04-06T09:43:15.549 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:43:15.550 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:43:15.550 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=18 2024-04-06T09:43:15.551 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=111, tag_tid=17, entry_tid=17355' 2024-04-06T09:43:15.551 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:43:15.551 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:43:15.552 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=17 2024-04-06T09:43:15.552 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=113, tag_tid=18, entry_tid=1' 2024-04-06T09:43:15.552 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:43:15.552 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:43:15.553 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:43:15.554 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=111, tag_tid=17, entry_tid=17355' 2024-04-06T09:43:15.554 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:43:15.554 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:43:15.555 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=17355 2024-04-06T09:43:15.555 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 18 = 17 -a 1 -le 17355 2024-04-06T09:43:15.555 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:43:15.555 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:43:15.556 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:43:15.757 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:43:15.757 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:43:15.757 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:43:15.757 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:43:15.757 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:43:15.757 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:43:15.757 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:43:15.758 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:43:15.758 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:43:15.758 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:43:15.758 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:43:15.758 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:43:15.758 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:43:15.758 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:43:15.758 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:43:15.759 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:43:15.759 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:43:15.762 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:43:15.762 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:43:15.762 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:43:15.762 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:43:15.762 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:43:15.762 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:43:15.762 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:43:15.763 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:43:15.763 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:43:15.763 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:43:15.763 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:43:15.763 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:43:15.763 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:43:15.763 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:43:15.763 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:43:15.763 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:43:15.763 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:43:15.763 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:43:15.763 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:43:15.763 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:43:15.763 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:43:15.763 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:43:15.763 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:43:15.764 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:43:15.764 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:43:15.764 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:43:15.764 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:43:15.764 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:43:15.764 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:43:15.764 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:43:15.764 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:43:15.764 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:43:15.786 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:43:15.787 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:43:15.787 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:43:15.787 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:43:15.917 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:43:15.918 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:43:15.918 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:43:15.918 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:43:15.918 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:43:15.918 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:43:15.918 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:43:15.918 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:43:15.918 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:43:15.918 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:43:15.919 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:43:15.919 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:43:15.919 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:43:15.920 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:43:15.921 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:43:15.921 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:43:15.971 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 27 2024-04-06T09:43:15.971 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 28 2024-04-06T09:43:15.971 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:43:15.971 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=113, tag_tid=18, entry_tid=1], [object_number=112, tag_tid=18, entry_tid=0], [object_number=114, tag_tid=17, entry_tid=20202], [object_number=115, tag_tid=17, entry_tid=20199]]], state=connected] 2024-04-06T09:43:15.972 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=108, tag_tid=17, entry_tid=17440], [object_number=111, tag_tid=17, entry_tid=17439], [object_number=110, tag_tid=17, entry_tid=17438], [object_number=109, tag_tid=17, entry_tid=17437]]], state=connected] 2024-04-06T09:43:15.975 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:43:15.976 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=113, tag_tid=18, entry_tid=1' 2024-04-06T09:43:15.977 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:43:15.978 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:43:15.978 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:43:15.978 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:43:15.978 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:43:15.978 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:43:15.978 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:43:15.979 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:43:15.979 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:43:15.979 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:43:15.979 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:43:15.979 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:43:15.980 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:43:15.980 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:43:15.980 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:43:16.023 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 27 2024-04-06T09:43:16.024 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 28 2024-04-06T09:43:16.024 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:43:16.024 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=113, tag_tid=18, entry_tid=1], [object_number=112, tag_tid=18, entry_tid=0], [object_number=114, tag_tid=17, entry_tid=20202], [object_number=115, tag_tid=17, entry_tid=20199]]], state=connected] 2024-04-06T09:43:16.024 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=108, tag_tid=17, entry_tid=17440], [object_number=111, tag_tid=17, entry_tid=17439], [object_number=110, tag_tid=17, entry_tid=17438], [object_number=109, tag_tid=17, entry_tid=17437]]], state=connected] 2024-04-06T09:43:16.027 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:43:16.029 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=108, tag_tid=17, entry_tid=17440' 2024-04-06T09:43:16.029 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=113, tag_tid=18, entry_tid=1' -a 'object_number=113, tag_tid=18, entry_tid=1' = 'object_number=108, tag_tid=17, entry_tid=17440' 2024-04-06T09:43:16.029 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=108, tag_tid=17, entry_tid=17440' '!=' 'object_number=111, tag_tid=17, entry_tid=17355' 2024-04-06T09:43:16.029 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:43:16.029 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=108, tag_tid=17, entry_tid=17440' = 'object_number=111, tag_tid=17, entry_tid=17355' 2024-04-06T09:43:16.029 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=108, tag_tid=17, entry_tid=17440' 2024-04-06T09:43:16.029 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=113, tag_tid=18, entry_tid=1' 2024-04-06T09:43:16.029 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:43:16.029 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:43:16.030 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=18 2024-04-06T09:43:16.030 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=108, tag_tid=17, entry_tid=17440' 2024-04-06T09:43:16.030 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:43:16.031 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:43:16.031 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=17 2024-04-06T09:43:16.032 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=113, tag_tid=18, entry_tid=1' 2024-04-06T09:43:16.032 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:43:16.032 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:43:16.033 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:43:16.034 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=108, tag_tid=17, entry_tid=17440' 2024-04-06T09:43:16.034 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:43:16.034 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:43:16.035 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=17440 2024-04-06T09:43:16.035 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 18 = 17 -a 1 -le 17440 2024-04-06T09:43:16.035 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:43:16.035 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:43:16.035 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:43:16.237 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:43:16.237 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:43:16.237 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:43:16.237 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:43:16.237 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:43:16.237 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:43:16.237 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:43:16.238 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:43:16.238 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:43:16.238 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:43:16.238 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:43:16.238 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:43:16.238 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:43:16.238 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:43:16.238 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:43:16.239 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:43:16.239 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:43:16.241 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:43:16.242 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:43:16.242 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:43:16.242 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:43:16.242 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:43:16.242 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:43:16.242 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:43:16.242 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:43:16.242 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:43:16.242 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:43:16.242 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:43:16.242 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:43:16.243 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:43:16.243 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:43:16.243 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:43:16.243 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:43:16.243 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:43:16.243 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:43:16.243 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:43:16.243 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:43:16.243 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:43:16.243 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:43:16.243 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:43:16.243 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:43:16.244 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:43:16.244 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:43:16.244 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:43:16.244 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:43:16.244 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:43:16.244 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:43:16.244 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:43:16.244 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:43:16.267 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:43:16.267 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:43:16.267 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:43:16.267 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:43:16.399 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:43:16.401 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:43:16.401 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:43:16.401 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:43:16.401 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:43:16.401 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:43:16.401 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:43:16.401 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:43:16.401 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:43:16.401 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:43:16.402 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:43:16.402 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:43:16.402 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:43:16.403 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:43:16.404 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:43:16.404 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:43:16.454 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 27 2024-04-06T09:43:16.454 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 28 2024-04-06T09:43:16.454 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:43:16.454 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=113, tag_tid=18, entry_tid=1], [object_number=112, tag_tid=18, entry_tid=0], [object_number=114, tag_tid=17, entry_tid=20202], [object_number=115, tag_tid=17, entry_tid=20199]]], state=connected] 2024-04-06T09:43:16.454 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=109, tag_tid=17, entry_tid=17525], [object_number=108, tag_tid=17, entry_tid=17524], [object_number=111, tag_tid=17, entry_tid=17523], [object_number=110, tag_tid=17, entry_tid=17522]]], state=connected] 2024-04-06T09:43:16.457 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:43:16.458 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=113, tag_tid=18, entry_tid=1' 2024-04-06T09:43:16.459 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:43:16.459 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:43:16.459 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:43:16.459 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:43:16.460 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:43:16.460 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:43:16.460 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:43:16.460 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:43:16.460 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:43:16.460 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:43:16.460 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:43:16.460 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:43:16.461 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:43:16.461 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:43:16.461 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:43:16.505 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 27 2024-04-06T09:43:16.505 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 28 2024-04-06T09:43:16.505 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:43:16.505 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=113, tag_tid=18, entry_tid=1], [object_number=112, tag_tid=18, entry_tid=0], [object_number=114, tag_tid=17, entry_tid=20202], [object_number=115, tag_tid=17, entry_tid=20199]]], state=connected] 2024-04-06T09:43:16.505 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=109, tag_tid=17, entry_tid=17525], [object_number=108, tag_tid=17, entry_tid=17524], [object_number=111, tag_tid=17, entry_tid=17523], [object_number=110, tag_tid=17, entry_tid=17522]]], state=connected] 2024-04-06T09:43:16.508 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:43:16.510 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=109, tag_tid=17, entry_tid=17525' 2024-04-06T09:43:16.510 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=113, tag_tid=18, entry_tid=1' -a 'object_number=113, tag_tid=18, entry_tid=1' = 'object_number=109, tag_tid=17, entry_tid=17525' 2024-04-06T09:43:16.510 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=109, tag_tid=17, entry_tid=17525' '!=' 'object_number=108, tag_tid=17, entry_tid=17440' 2024-04-06T09:43:16.510 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:43:16.510 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=109, tag_tid=17, entry_tid=17525' = 'object_number=108, tag_tid=17, entry_tid=17440' 2024-04-06T09:43:16.510 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=109, tag_tid=17, entry_tid=17525' 2024-04-06T09:43:16.511 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=113, tag_tid=18, entry_tid=1' 2024-04-06T09:43:16.511 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:43:16.511 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:43:16.511 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=18 2024-04-06T09:43:16.512 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=109, tag_tid=17, entry_tid=17525' 2024-04-06T09:43:16.512 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:43:16.512 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:43:16.513 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=17 2024-04-06T09:43:16.513 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=113, tag_tid=18, entry_tid=1' 2024-04-06T09:43:16.514 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:43:16.514 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:43:16.514 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:43:16.515 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=109, tag_tid=17, entry_tid=17525' 2024-04-06T09:43:16.515 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:43:16.515 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:43:16.516 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=17525 2024-04-06T09:43:16.516 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 18 = 17 -a 1 -le 17525 2024-04-06T09:43:16.516 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:43:16.516 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:43:16.517 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:43:16.718 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:43:16.719 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:43:16.719 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:43:16.719 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:43:16.719 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:43:16.719 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:43:16.719 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:43:16.719 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:43:16.719 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:43:16.719 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:43:16.719 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:43:16.720 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:43:16.720 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:43:16.720 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:43:16.720 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:43:16.720 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:43:16.720 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:43:16.723 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:43:16.723 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:43:16.723 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:43:16.723 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:43:16.723 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:43:16.723 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:43:16.723 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:43:16.723 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:43:16.724 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:43:16.724 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:43:16.724 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:43:16.724 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:43:16.724 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:43:16.724 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:43:16.724 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:43:16.724 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:43:16.724 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:43:16.724 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:43:16.724 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:43:16.724 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:43:16.724 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:43:16.725 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:43:16.725 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:43:16.725 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:43:16.725 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:43:16.725 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:43:16.725 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:43:16.725 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:43:16.725 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:43:16.725 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:43:16.725 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:43:16.726 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:43:16.751 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:43:16.751 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:43:16.751 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:43:16.751 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:43:16.879 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:43:16.880 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:43:16.880 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:43:16.880 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:43:16.881 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:43:16.881 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:43:16.881 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:43:16.881 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:43:16.881 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:43:16.881 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:43:16.882 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:43:16.882 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:43:16.882 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:43:16.883 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:43:16.883 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:43:16.884 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:43:16.935 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 27 2024-04-06T09:43:16.935 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 28 2024-04-06T09:43:16.935 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:43:16.935 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=113, tag_tid=18, entry_tid=1], [object_number=112, tag_tid=18, entry_tid=0], [object_number=114, tag_tid=17, entry_tid=20202], [object_number=115, tag_tid=17, entry_tid=20199]]], state=connected] 2024-04-06T09:43:16.935 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=109, tag_tid=17, entry_tid=17609], [object_number=108, tag_tid=17, entry_tid=17608], [object_number=111, tag_tid=17, entry_tid=17607], [object_number=110, tag_tid=17, entry_tid=17606]]], state=connected] 2024-04-06T09:43:16.938 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:43:16.940 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=113, tag_tid=18, entry_tid=1' 2024-04-06T09:43:16.941 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:43:16.941 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:43:16.941 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:43:16.941 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:43:16.941 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:43:16.941 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:43:16.941 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:43:16.942 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:43:16.942 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:43:16.942 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:43:16.942 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:43:16.942 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:43:16.942 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:43:16.943 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:43:16.943 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:43:16.989 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 27 2024-04-06T09:43:16.989 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 28 2024-04-06T09:43:16.989 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:43:16.989 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=113, tag_tid=18, entry_tid=1], [object_number=112, tag_tid=18, entry_tid=0], [object_number=114, tag_tid=17, entry_tid=20202], [object_number=115, tag_tid=17, entry_tid=20199]]], state=connected] 2024-04-06T09:43:16.989 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=109, tag_tid=17, entry_tid=17609], [object_number=108, tag_tid=17, entry_tid=17608], [object_number=111, tag_tid=17, entry_tid=17607], [object_number=110, tag_tid=17, entry_tid=17606]]], state=connected] 2024-04-06T09:43:16.993 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:43:16.995 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=109, tag_tid=17, entry_tid=17609' 2024-04-06T09:43:16.995 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=113, tag_tid=18, entry_tid=1' -a 'object_number=113, tag_tid=18, entry_tid=1' = 'object_number=109, tag_tid=17, entry_tid=17609' 2024-04-06T09:43:16.995 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=109, tag_tid=17, entry_tid=17609' '!=' 'object_number=109, tag_tid=17, entry_tid=17525' 2024-04-06T09:43:16.995 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:43:16.995 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=109, tag_tid=17, entry_tid=17609' = 'object_number=109, tag_tid=17, entry_tid=17525' 2024-04-06T09:43:16.995 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=109, tag_tid=17, entry_tid=17609' 2024-04-06T09:43:16.995 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=113, tag_tid=18, entry_tid=1' 2024-04-06T09:43:16.995 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:43:16.995 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:43:16.996 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=18 2024-04-06T09:43:16.997 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=109, tag_tid=17, entry_tid=17609' 2024-04-06T09:43:16.997 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:43:16.997 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:43:16.998 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=17 2024-04-06T09:43:16.998 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=113, tag_tid=18, entry_tid=1' 2024-04-06T09:43:16.999 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:43:16.999 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:43:16.999 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:43:17.000 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=109, tag_tid=17, entry_tid=17609' 2024-04-06T09:43:17.000 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:43:17.000 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:43:17.002 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=17609 2024-04-06T09:43:17.002 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 18 = 17 -a 1 -le 17609 2024-04-06T09:43:17.002 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:43:17.002 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:43:17.002 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:43:17.203 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:43:17.204 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:43:17.204 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:43:17.204 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:43:17.204 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:43:17.204 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:43:17.204 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:43:17.204 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:43:17.204 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:43:17.204 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:43:17.204 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:43:17.204 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:43:17.204 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:43:17.205 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:43:17.205 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:43:17.205 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:43:17.205 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:43:17.209 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:43:17.209 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:43:17.209 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:43:17.209 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:43:17.209 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:43:17.209 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:43:17.209 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:43:17.209 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:43:17.210 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:43:17.210 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:43:17.210 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:43:17.210 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:43:17.210 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:43:17.210 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:43:17.210 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:43:17.210 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:43:17.210 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:43:17.210 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:43:17.210 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:43:17.211 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:43:17.211 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:43:17.211 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:43:17.211 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:43:17.211 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:43:17.211 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:43:17.211 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:43:17.211 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:43:17.211 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:43:17.211 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:43:17.211 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:43:17.211 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:43:17.211 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:43:17.234 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:43:17.234 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:43:17.234 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:43:17.234 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:43:17.367 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:43:17.368 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:43:17.369 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:43:17.369 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:43:17.369 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:43:17.369 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:43:17.369 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:43:17.369 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:43:17.369 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:43:17.369 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:43:17.369 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:43:17.370 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:43:17.370 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:43:17.370 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:43:17.371 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:43:17.371 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:43:17.419 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 27 2024-04-06T09:43:17.419 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 28 2024-04-06T09:43:17.419 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:43:17.419 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=113, tag_tid=18, entry_tid=1], [object_number=112, tag_tid=18, entry_tid=0], [object_number=114, tag_tid=17, entry_tid=20202], [object_number=115, tag_tid=17, entry_tid=20199]]], state=connected] 2024-04-06T09:43:17.419 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=108, tag_tid=17, entry_tid=17696], [object_number=111, tag_tid=17, entry_tid=17695], [object_number=110, tag_tid=17, entry_tid=17694], [object_number=109, tag_tid=17, entry_tid=17693]]], state=connected] 2024-04-06T09:43:17.444 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:43:17.446 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=113, tag_tid=18, entry_tid=1' 2024-04-06T09:43:17.447 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:43:17.447 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:43:17.447 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:43:17.447 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:43:17.447 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:43:17.447 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:43:17.447 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:43:17.447 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:43:17.448 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:43:17.448 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:43:17.448 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:43:17.448 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:43:17.448 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:43:17.448 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:43:17.449 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:43:17.497 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 27 2024-04-06T09:43:17.497 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 28 2024-04-06T09:43:17.497 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:43:17.497 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=113, tag_tid=18, entry_tid=1], [object_number=112, tag_tid=18, entry_tid=0], [object_number=114, tag_tid=17, entry_tid=20202], [object_number=115, tag_tid=17, entry_tid=20199]]], state=connected] 2024-04-06T09:43:17.497 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=108, tag_tid=17, entry_tid=17696], [object_number=111, tag_tid=17, entry_tid=17695], [object_number=110, tag_tid=17, entry_tid=17694], [object_number=109, tag_tid=17, entry_tid=17693]]], state=connected] 2024-04-06T09:43:17.500 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:43:17.503 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=108, tag_tid=17, entry_tid=17696' 2024-04-06T09:43:17.503 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=113, tag_tid=18, entry_tid=1' -a 'object_number=113, tag_tid=18, entry_tid=1' = 'object_number=108, tag_tid=17, entry_tid=17696' 2024-04-06T09:43:17.503 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=108, tag_tid=17, entry_tid=17696' '!=' 'object_number=109, tag_tid=17, entry_tid=17609' 2024-04-06T09:43:17.503 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:43:17.503 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=108, tag_tid=17, entry_tid=17696' = 'object_number=109, tag_tid=17, entry_tid=17609' 2024-04-06T09:43:17.503 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=108, tag_tid=17, entry_tid=17696' 2024-04-06T09:43:17.503 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=113, tag_tid=18, entry_tid=1' 2024-04-06T09:43:17.503 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:43:17.503 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:43:17.504 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=18 2024-04-06T09:43:17.504 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=108, tag_tid=17, entry_tid=17696' 2024-04-06T09:43:17.504 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:43:17.505 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:43:17.505 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=17 2024-04-06T09:43:17.506 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=113, tag_tid=18, entry_tid=1' 2024-04-06T09:43:17.506 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:43:17.506 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:43:17.507 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:43:17.508 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=108, tag_tid=17, entry_tid=17696' 2024-04-06T09:43:17.508 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:43:17.508 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:43:17.509 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=17696 2024-04-06T09:43:17.509 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 18 = 17 -a 1 -le 17696 2024-04-06T09:43:17.509 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:43:17.509 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:43:17.509 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:43:17.711 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:43:17.711 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:43:17.711 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:43:17.711 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:43:17.711 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:43:17.711 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:43:17.712 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:43:17.712 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:43:17.712 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:43:17.712 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:43:17.712 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:43:17.712 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:43:17.712 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:43:17.712 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:43:17.712 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:43:17.713 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:43:17.713 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:43:17.717 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:43:17.717 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:43:17.717 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:43:17.717 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:43:17.717 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:43:17.717 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:43:17.718 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:43:17.718 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:43:17.718 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:43:17.718 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:43:17.718 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:43:17.718 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:43:17.718 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:43:17.718 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:43:17.718 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:43:17.719 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:43:17.719 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:43:17.719 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:43:17.719 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:43:17.719 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:43:17.719 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:43:17.719 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:43:17.719 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:43:17.719 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:43:17.719 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:43:17.719 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:43:17.719 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:43:17.719 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:43:17.720 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:43:17.720 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:43:17.720 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:43:17.720 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:43:17.742 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:43:17.742 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:43:17.742 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:43:17.742 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:43:17.871 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:43:17.891 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:43:17.891 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:43:17.892 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:43:17.892 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:43:17.892 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:43:17.892 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:43:17.892 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:43:17.892 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:43:17.892 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:43:17.892 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:43:17.892 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:43:17.892 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:43:17.892 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:43:17.892 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:43:17.892 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:43:17.922 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 27 2024-04-06T09:43:17.922 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 28 2024-04-06T09:43:17.923 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:43:17.923 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=113, tag_tid=18, entry_tid=1], [object_number=112, tag_tid=18, entry_tid=0], [object_number=114, tag_tid=17, entry_tid=20202], [object_number=115, tag_tid=17, entry_tid=20199]]], state=connected] 2024-04-06T09:43:17.923 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=109, tag_tid=17, entry_tid=17785], [object_number=108, tag_tid=17, entry_tid=17784], [object_number=111, tag_tid=17, entry_tid=17783], [object_number=110, tag_tid=17, entry_tid=17782]]], state=connected] 2024-04-06T09:43:17.926 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:43:17.927 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=113, tag_tid=18, entry_tid=1' 2024-04-06T09:43:17.928 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:43:17.928 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:43:17.928 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:43:17.928 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:43:17.928 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:43:17.929 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:43:17.929 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:43:17.929 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:43:17.929 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:43:17.929 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:43:17.929 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:43:17.929 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:43:17.929 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:43:17.930 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:43:17.930 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:43:17.978 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 27 2024-04-06T09:43:17.978 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 28 2024-04-06T09:43:17.978 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:43:17.978 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=113, tag_tid=18, entry_tid=1], [object_number=112, tag_tid=18, entry_tid=0], [object_number=114, tag_tid=17, entry_tid=20202], [object_number=115, tag_tid=17, entry_tid=20199]]], state=connected] 2024-04-06T09:43:17.978 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=109, tag_tid=17, entry_tid=17785], [object_number=108, tag_tid=17, entry_tid=17784], [object_number=111, tag_tid=17, entry_tid=17783], [object_number=110, tag_tid=17, entry_tid=17782]]], state=connected] 2024-04-06T09:43:17.980 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:43:17.982 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=109, tag_tid=17, entry_tid=17785' 2024-04-06T09:43:17.982 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=113, tag_tid=18, entry_tid=1' -a 'object_number=113, tag_tid=18, entry_tid=1' = 'object_number=109, tag_tid=17, entry_tid=17785' 2024-04-06T09:43:17.982 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=109, tag_tid=17, entry_tid=17785' '!=' 'object_number=108, tag_tid=17, entry_tid=17696' 2024-04-06T09:43:17.982 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:43:17.982 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=109, tag_tid=17, entry_tid=17785' = 'object_number=108, tag_tid=17, entry_tid=17696' 2024-04-06T09:43:17.982 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=109, tag_tid=17, entry_tid=17785' 2024-04-06T09:43:17.982 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=113, tag_tid=18, entry_tid=1' 2024-04-06T09:43:17.982 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:43:17.982 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:43:17.983 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=18 2024-04-06T09:43:17.983 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=109, tag_tid=17, entry_tid=17785' 2024-04-06T09:43:17.983 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:43:17.984 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:43:17.984 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=17 2024-04-06T09:43:17.985 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=113, tag_tid=18, entry_tid=1' 2024-04-06T09:43:17.985 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:43:17.985 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:43:17.986 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:43:17.987 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=109, tag_tid=17, entry_tid=17785' 2024-04-06T09:43:17.987 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:43:17.987 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:43:17.988 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=17785 2024-04-06T09:43:17.988 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 18 = 17 -a 1 -le 17785 2024-04-06T09:43:17.988 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:43:17.988 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:43:17.988 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:43:18.190 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:43:18.190 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:43:18.190 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:43:18.190 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:43:18.190 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:43:18.190 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:43:18.190 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:43:18.190 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:43:18.191 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:43:18.191 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:43:18.191 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:43:18.191 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:43:18.191 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:43:18.191 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:43:18.191 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:43:18.191 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:43:18.192 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:43:18.195 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:43:18.195 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:43:18.195 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:43:18.196 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:43:18.196 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:43:18.196 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:43:18.196 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:43:18.196 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:43:18.196 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:43:18.196 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:43:18.196 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:43:18.196 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:43:18.196 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:43:18.196 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:43:18.196 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:43:18.196 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:43:18.196 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:43:18.196 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:43:18.197 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:43:18.197 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:43:18.197 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:43:18.197 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:43:18.197 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:43:18.197 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:43:18.197 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:43:18.197 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:43:18.197 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:43:18.197 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:43:18.197 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:43:18.197 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:43:18.197 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:43:18.197 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:43:18.221 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:43:18.222 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:43:18.222 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:43:18.222 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:43:18.351 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:43:18.352 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:43:18.352 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:43:18.352 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:43:18.352 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:43:18.352 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:43:18.352 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:43:18.353 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:43:18.353 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:43:18.353 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:43:18.353 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:43:18.353 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:43:18.354 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:43:18.355 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:43:18.355 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:43:18.355 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:43:18.406 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 27 2024-04-06T09:43:18.406 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 28 2024-04-06T09:43:18.407 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:43:18.407 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=113, tag_tid=18, entry_tid=1], [object_number=112, tag_tid=18, entry_tid=0], [object_number=114, tag_tid=17, entry_tid=20202], [object_number=115, tag_tid=17, entry_tid=20199]]], state=connected] 2024-04-06T09:43:18.407 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=108, tag_tid=17, entry_tid=17872], [object_number=111, tag_tid=17, entry_tid=17871], [object_number=110, tag_tid=17, entry_tid=17870], [object_number=109, tag_tid=17, entry_tid=17869]]], state=connected] 2024-04-06T09:43:18.410 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:43:18.411 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=113, tag_tid=18, entry_tid=1' 2024-04-06T09:43:18.412 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:43:18.412 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:43:18.412 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:43:18.412 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:43:18.412 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:43:18.413 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:43:18.413 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:43:18.413 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:43:18.413 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:43:18.413 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:43:18.413 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:43:18.414 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:43:18.415 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:43:18.415 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:43:18.415 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:43:18.461 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 27 2024-04-06T09:43:18.461 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 28 2024-04-06T09:43:18.461 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:43:18.461 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=113, tag_tid=18, entry_tid=1], [object_number=112, tag_tid=18, entry_tid=0], [object_number=114, tag_tid=17, entry_tid=20202], [object_number=115, tag_tid=17, entry_tid=20199]]], state=connected] 2024-04-06T09:43:18.461 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=108, tag_tid=17, entry_tid=17872], [object_number=111, tag_tid=17, entry_tid=17871], [object_number=110, tag_tid=17, entry_tid=17870], [object_number=109, tag_tid=17, entry_tid=17869]]], state=connected] 2024-04-06T09:43:18.464 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:43:18.467 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=108, tag_tid=17, entry_tid=17872' 2024-04-06T09:43:18.467 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=113, tag_tid=18, entry_tid=1' -a 'object_number=113, tag_tid=18, entry_tid=1' = 'object_number=108, tag_tid=17, entry_tid=17872' 2024-04-06T09:43:18.467 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=108, tag_tid=17, entry_tid=17872' '!=' 'object_number=109, tag_tid=17, entry_tid=17785' 2024-04-06T09:43:18.467 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:43:18.467 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=108, tag_tid=17, entry_tid=17872' = 'object_number=109, tag_tid=17, entry_tid=17785' 2024-04-06T09:43:18.467 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=108, tag_tid=17, entry_tid=17872' 2024-04-06T09:43:18.467 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=113, tag_tid=18, entry_tid=1' 2024-04-06T09:43:18.467 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:43:18.467 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:43:18.468 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=18 2024-04-06T09:43:18.469 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=108, tag_tid=17, entry_tid=17872' 2024-04-06T09:43:18.469 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:43:18.469 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:43:18.469 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=17 2024-04-06T09:43:18.470 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=113, tag_tid=18, entry_tid=1' 2024-04-06T09:43:18.470 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:43:18.470 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:43:18.471 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:43:18.472 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=108, tag_tid=17, entry_tid=17872' 2024-04-06T09:43:18.472 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:43:18.472 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:43:18.473 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=17872 2024-04-06T09:43:18.473 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 18 = 17 -a 1 -le 17872 2024-04-06T09:43:18.473 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:43:18.473 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:43:18.473 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:43:18.675 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:43:18.675 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:43:18.675 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:43:18.676 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:43:18.676 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:43:18.676 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:43:18.676 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:43:18.676 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:43:18.676 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:43:18.676 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:43:18.676 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:43:18.676 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:43:18.676 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:43:18.676 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:43:18.676 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:43:18.677 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:43:18.677 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:43:18.680 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:43:18.680 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:43:18.680 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:43:18.680 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:43:18.681 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:43:18.681 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:43:18.681 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:43:18.681 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:43:18.681 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:43:18.681 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:43:18.681 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:43:18.681 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:43:18.681 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:43:18.681 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:43:18.681 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:43:18.681 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:43:18.681 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:43:18.681 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:43:18.682 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:43:18.682 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:43:18.682 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:43:18.682 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:43:18.682 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:43:18.682 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:43:18.682 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:43:18.682 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:43:18.682 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:43:18.682 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:43:18.682 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:43:18.682 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:43:18.682 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:43:18.682 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:43:18.706 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:43:18.706 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:43:18.706 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:43:18.706 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:43:18.839 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:43:18.840 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:43:18.840 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:43:18.841 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:43:18.841 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:43:18.841 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:43:18.841 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:43:18.841 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:43:18.841 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:43:18.841 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:43:18.841 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:43:18.842 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:43:18.842 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:43:18.843 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:43:18.844 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:43:18.844 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:43:18.890 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 27 2024-04-06T09:43:18.891 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 28 2024-04-06T09:43:18.891 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:43:18.891 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=113, tag_tid=18, entry_tid=1], [object_number=112, tag_tid=18, entry_tid=0], [object_number=114, tag_tid=17, entry_tid=20202], [object_number=115, tag_tid=17, entry_tid=20199]]], state=connected] 2024-04-06T09:43:18.891 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=111, tag_tid=17, entry_tid=17959], [object_number=110, tag_tid=17, entry_tid=17958], [object_number=109, tag_tid=17, entry_tid=17957], [object_number=108, tag_tid=17, entry_tid=17956]]], state=connected] 2024-04-06T09:43:18.894 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:43:18.895 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=113, tag_tid=18, entry_tid=1' 2024-04-06T09:43:18.896 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:43:18.896 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:43:18.897 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:43:18.897 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:43:18.897 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:43:18.897 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:43:18.897 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:43:18.897 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:43:18.897 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:43:18.897 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:43:18.897 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:43:18.897 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:43:18.898 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:43:18.898 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:43:18.899 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:43:18.945 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 27 2024-04-06T09:43:18.945 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 28 2024-04-06T09:43:18.945 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:43:18.945 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=113, tag_tid=18, entry_tid=1], [object_number=112, tag_tid=18, entry_tid=0], [object_number=114, tag_tid=17, entry_tid=20202], [object_number=115, tag_tid=17, entry_tid=20199]]], state=connected] 2024-04-06T09:43:18.945 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=111, tag_tid=17, entry_tid=17959], [object_number=110, tag_tid=17, entry_tid=17958], [object_number=109, tag_tid=17, entry_tid=17957], [object_number=108, tag_tid=17, entry_tid=17956]]], state=connected] 2024-04-06T09:43:18.948 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:43:18.951 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=111, tag_tid=17, entry_tid=17959' 2024-04-06T09:43:18.951 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=113, tag_tid=18, entry_tid=1' -a 'object_number=113, tag_tid=18, entry_tid=1' = 'object_number=111, tag_tid=17, entry_tid=17959' 2024-04-06T09:43:18.951 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=111, tag_tid=17, entry_tid=17959' '!=' 'object_number=108, tag_tid=17, entry_tid=17872' 2024-04-06T09:43:18.951 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:43:18.951 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=111, tag_tid=17, entry_tid=17959' = 'object_number=108, tag_tid=17, entry_tid=17872' 2024-04-06T09:43:18.951 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=111, tag_tid=17, entry_tid=17959' 2024-04-06T09:43:18.951 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=113, tag_tid=18, entry_tid=1' 2024-04-06T09:43:18.951 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:43:18.952 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:43:18.952 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=18 2024-04-06T09:43:18.952 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=111, tag_tid=17, entry_tid=17959' 2024-04-06T09:43:18.952 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:43:18.952 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:43:18.953 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=17 2024-04-06T09:43:18.954 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=113, tag_tid=18, entry_tid=1' 2024-04-06T09:43:18.954 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:43:18.954 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:43:18.955 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:43:18.955 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=111, tag_tid=17, entry_tid=17959' 2024-04-06T09:43:18.956 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:43:18.956 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:43:18.957 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=17959 2024-04-06T09:43:18.957 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 18 = 17 -a 1 -le 17959 2024-04-06T09:43:18.957 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:43:18.957 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:43:18.957 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:43:19.158 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:43:19.159 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:43:19.159 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:43:19.159 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:43:19.159 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:43:19.159 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:43:19.159 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:43:19.159 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:43:19.159 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:43:19.159 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:43:19.159 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:43:19.159 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:43:19.159 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:43:19.159 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:43:19.160 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:43:19.160 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:43:19.160 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:43:19.163 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:43:19.163 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:43:19.163 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:43:19.163 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:43:19.164 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:43:19.164 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:43:19.164 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:43:19.164 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:43:19.164 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:43:19.164 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:43:19.164 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:43:19.164 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:43:19.164 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:43:19.164 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:43:19.164 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:43:19.164 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:43:19.164 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:43:19.164 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:43:19.165 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:43:19.165 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:43:19.165 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:43:19.165 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:43:19.165 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:43:19.165 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:43:19.165 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:43:19.165 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:43:19.165 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:43:19.165 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:43:19.166 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:43:19.166 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:43:19.166 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:43:19.166 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:43:19.189 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:43:19.190 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:43:19.190 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:43:19.190 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:43:19.321 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:43:19.322 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:43:19.322 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:43:19.322 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:43:19.322 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:43:19.322 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:43:19.322 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:43:19.322 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:43:19.322 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:43:19.322 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:43:19.323 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:43:19.323 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:43:19.323 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:43:19.325 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:43:19.326 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:43:19.326 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:43:19.576 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 27 2024-04-06T09:43:19.576 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 28 2024-04-06T09:43:19.576 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:43:19.576 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=113, tag_tid=18, entry_tid=1], [object_number=112, tag_tid=18, entry_tid=0], [object_number=114, tag_tid=17, entry_tid=20202], [object_number=115, tag_tid=17, entry_tid=20199]]], state=connected] 2024-04-06T09:43:19.576 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=111, tag_tid=17, entry_tid=18043], [object_number=110, tag_tid=17, entry_tid=18042], [object_number=109, tag_tid=17, entry_tid=18041], [object_number=108, tag_tid=17, entry_tid=18040]]], state=connected] 2024-04-06T09:43:19.581 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:43:19.582 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=113, tag_tid=18, entry_tid=1' 2024-04-06T09:43:19.584 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:43:19.584 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:43:19.584 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:43:19.584 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:43:19.584 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:43:19.584 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:43:19.584 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:43:19.584 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:43:19.584 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:43:19.585 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:43:19.585 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:43:19.585 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:43:19.586 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:43:19.587 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:43:19.587 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:43:19.643 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 27 2024-04-06T09:43:19.643 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 28 2024-04-06T09:43:19.644 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:43:19.644 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=113, tag_tid=18, entry_tid=1], [object_number=112, tag_tid=18, entry_tid=0], [object_number=114, tag_tid=17, entry_tid=20202], [object_number=115, tag_tid=17, entry_tid=20199]]], state=connected] 2024-04-06T09:43:19.644 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=111, tag_tid=17, entry_tid=18043], [object_number=110, tag_tid=17, entry_tid=18042], [object_number=109, tag_tid=17, entry_tid=18041], [object_number=108, tag_tid=17, entry_tid=18040]]], state=connected] 2024-04-06T09:43:19.647 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:43:19.649 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=111, tag_tid=17, entry_tid=18043' 2024-04-06T09:43:19.649 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=113, tag_tid=18, entry_tid=1' -a 'object_number=113, tag_tid=18, entry_tid=1' = 'object_number=111, tag_tid=17, entry_tid=18043' 2024-04-06T09:43:19.649 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=111, tag_tid=17, entry_tid=18043' '!=' 'object_number=111, tag_tid=17, entry_tid=17959' 2024-04-06T09:43:19.650 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:43:19.650 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=111, tag_tid=17, entry_tid=18043' = 'object_number=111, tag_tid=17, entry_tid=17959' 2024-04-06T09:43:19.650 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=111, tag_tid=17, entry_tid=18043' 2024-04-06T09:43:19.650 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=113, tag_tid=18, entry_tid=1' 2024-04-06T09:43:19.650 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:43:19.650 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:43:19.651 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=18 2024-04-06T09:43:19.651 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=111, tag_tid=17, entry_tid=18043' 2024-04-06T09:43:19.651 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:43:19.651 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:43:19.652 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=17 2024-04-06T09:43:19.653 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=113, tag_tid=18, entry_tid=1' 2024-04-06T09:43:19.653 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:43:19.653 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:43:19.654 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:43:19.655 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=111, tag_tid=17, entry_tid=18043' 2024-04-06T09:43:19.655 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:43:19.655 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:43:19.656 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=18043 2024-04-06T09:43:19.656 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 18 = 17 -a 1 -le 18043 2024-04-06T09:43:19.656 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:43:19.657 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:43:19.657 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:43:19.858 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:43:19.858 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:43:19.859 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:43:19.859 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:43:19.859 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:43:19.859 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:43:19.859 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:43:19.859 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:43:19.859 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:43:19.859 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:43:19.859 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:43:19.859 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:43:19.859 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:43:19.859 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:43:19.860 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:43:19.860 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:43:19.860 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:43:19.864 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:43:19.865 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:43:19.865 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:43:19.865 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:43:19.865 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:43:19.865 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:43:19.865 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:43:19.865 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:43:19.865 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:43:19.865 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:43:19.865 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:43:19.865 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:43:19.865 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:43:19.865 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:43:19.865 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:43:19.865 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:43:19.866 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:43:19.866 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:43:19.866 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:43:19.866 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:43:19.866 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:43:19.866 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:43:19.866 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:43:19.866 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:43:19.866 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:43:19.866 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:43:19.866 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:43:19.866 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:43:19.866 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:43:19.866 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:43:19.867 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:43:19.867 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:43:19.889 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:43:19.889 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:43:19.889 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:43:19.889 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:43:20.024 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:43:20.026 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:43:20.026 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:43:20.026 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:43:20.026 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:43:20.026 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:43:20.026 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:43:20.026 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:43:20.026 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:43:20.026 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:43:20.027 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:43:20.027 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:43:20.027 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:43:20.028 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:43:20.029 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:43:20.029 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:43:20.086 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 27 2024-04-06T09:43:20.086 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 28 2024-04-06T09:43:20.086 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:43:20.086 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=113, tag_tid=18, entry_tid=1], [object_number=112, tag_tid=18, entry_tid=0], [object_number=114, tag_tid=17, entry_tid=20202], [object_number=115, tag_tid=17, entry_tid=20199]]], state=connected] 2024-04-06T09:43:20.086 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=108, tag_tid=17, entry_tid=18160], [object_number=111, tag_tid=17, entry_tid=18159], [object_number=110, tag_tid=17, entry_tid=18158], [object_number=109, tag_tid=17, entry_tid=18157]]], state=connected] 2024-04-06T09:43:20.090 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:43:20.091 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=113, tag_tid=18, entry_tid=1' 2024-04-06T09:43:20.093 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:43:20.093 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:43:20.093 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:43:20.093 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:43:20.093 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:43:20.093 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:43:20.093 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:43:20.093 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:43:20.093 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:43:20.093 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:43:20.093 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:43:20.093 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:43:20.094 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:43:20.094 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:43:20.094 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:43:20.349 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 27 2024-04-06T09:43:20.350 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 28 2024-04-06T09:43:20.350 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:43:20.350 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=113, tag_tid=18, entry_tid=1], [object_number=112, tag_tid=18, entry_tid=0], [object_number=114, tag_tid=17, entry_tid=20202], [object_number=115, tag_tid=17, entry_tid=20199]]], state=connected] 2024-04-06T09:43:20.351 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=108, tag_tid=17, entry_tid=18160], [object_number=111, tag_tid=17, entry_tid=18159], [object_number=110, tag_tid=17, entry_tid=18158], [object_number=109, tag_tid=17, entry_tid=18157]]], state=connected] 2024-04-06T09:43:20.354 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:43:20.356 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=108, tag_tid=17, entry_tid=18160' 2024-04-06T09:43:20.356 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=113, tag_tid=18, entry_tid=1' -a 'object_number=113, tag_tid=18, entry_tid=1' = 'object_number=108, tag_tid=17, entry_tid=18160' 2024-04-06T09:43:20.356 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=108, tag_tid=17, entry_tid=18160' '!=' 'object_number=111, tag_tid=17, entry_tid=18043' 2024-04-06T09:43:20.356 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:43:20.357 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=108, tag_tid=17, entry_tid=18160' = 'object_number=111, tag_tid=17, entry_tid=18043' 2024-04-06T09:43:20.357 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=108, tag_tid=17, entry_tid=18160' 2024-04-06T09:43:20.357 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=113, tag_tid=18, entry_tid=1' 2024-04-06T09:43:20.357 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:43:20.357 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:43:20.358 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=18 2024-04-06T09:43:20.359 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=108, tag_tid=17, entry_tid=18160' 2024-04-06T09:43:20.359 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:43:20.359 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:43:20.360 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=17 2024-04-06T09:43:20.361 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=113, tag_tid=18, entry_tid=1' 2024-04-06T09:43:20.361 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:43:20.361 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:43:20.362 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:43:20.363 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=108, tag_tid=17, entry_tid=18160' 2024-04-06T09:43:20.363 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:43:20.363 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:43:20.365 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=18160 2024-04-06T09:43:20.365 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 18 = 17 -a 1 -le 18160 2024-04-06T09:43:20.365 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:43:20.365 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:43:20.365 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:43:20.567 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:43:20.567 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:43:20.567 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:43:20.567 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:43:20.567 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:43:20.567 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:43:20.567 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:43:20.567 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:43:20.567 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:43:20.567 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:43:20.567 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:43:20.567 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:43:20.568 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:43:20.568 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:43:20.568 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:43:20.568 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:43:20.568 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:43:20.572 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:43:20.572 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:43:20.572 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:43:20.573 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:43:20.573 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:43:20.573 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:43:20.573 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:43:20.573 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:43:20.573 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:43:20.573 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:43:20.573 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:43:20.573 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:43:20.573 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:43:20.573 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:43:20.573 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:43:20.573 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:43:20.573 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:43:20.574 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:43:20.574 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:43:20.574 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:43:20.574 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:43:20.574 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:43:20.574 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:43:20.574 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:43:20.574 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:43:20.574 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:43:20.574 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:43:20.574 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:43:20.574 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:43:20.574 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:43:20.574 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:43:20.574 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:43:20.597 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:43:20.598 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:43:20.598 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:43:20.598 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:43:20.730 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:43:20.731 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:43:20.731 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:43:20.731 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:43:20.731 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:43:20.731 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:43:20.731 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:43:20.731 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:43:20.731 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:43:20.731 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:43:20.732 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:43:20.732 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:43:20.732 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:43:20.734 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:43:20.735 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:43:20.735 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:43:20.787 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 27 2024-04-06T09:43:20.787 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 28 2024-04-06T09:43:20.787 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:43:20.787 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=113, tag_tid=18, entry_tid=1], [object_number=112, tag_tid=18, entry_tid=0], [object_number=114, tag_tid=17, entry_tid=20202], [object_number=115, tag_tid=17, entry_tid=20199]]], state=connected] 2024-04-06T09:43:20.787 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=111, tag_tid=17, entry_tid=18267], [object_number=110, tag_tid=17, entry_tid=18266], [object_number=109, tag_tid=17, entry_tid=18265], [object_number=108, tag_tid=17, entry_tid=18264]]], state=connected] 2024-04-06T09:43:20.791 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:43:20.792 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=113, tag_tid=18, entry_tid=1' 2024-04-06T09:43:20.793 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:43:20.793 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:43:20.793 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:43:20.793 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:43:20.793 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:43:20.793 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:43:20.794 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:43:20.794 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:43:20.794 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:43:20.794 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:43:20.794 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:43:20.794 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:43:20.795 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:43:20.795 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:43:20.795 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:43:20.842 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 27 2024-04-06T09:43:20.842 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 28 2024-04-06T09:43:20.842 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:43:20.842 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=113, tag_tid=18, entry_tid=1], [object_number=112, tag_tid=18, entry_tid=0], [object_number=114, tag_tid=17, entry_tid=20202], [object_number=115, tag_tid=17, entry_tid=20199]]], state=connected] 2024-04-06T09:43:20.842 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=111, tag_tid=17, entry_tid=18267], [object_number=110, tag_tid=17, entry_tid=18266], [object_number=109, tag_tid=17, entry_tid=18265], [object_number=108, tag_tid=17, entry_tid=18264]]], state=connected] 2024-04-06T09:43:20.846 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:43:20.847 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=111, tag_tid=17, entry_tid=18267' 2024-04-06T09:43:20.847 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=113, tag_tid=18, entry_tid=1' -a 'object_number=113, tag_tid=18, entry_tid=1' = 'object_number=111, tag_tid=17, entry_tid=18267' 2024-04-06T09:43:20.847 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=111, tag_tid=17, entry_tid=18267' '!=' 'object_number=108, tag_tid=17, entry_tid=18160' 2024-04-06T09:43:20.847 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:43:20.847 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=111, tag_tid=17, entry_tid=18267' = 'object_number=108, tag_tid=17, entry_tid=18160' 2024-04-06T09:43:20.847 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=111, tag_tid=17, entry_tid=18267' 2024-04-06T09:43:20.848 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=113, tag_tid=18, entry_tid=1' 2024-04-06T09:43:20.848 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:43:20.848 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:43:20.848 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=18 2024-04-06T09:43:20.849 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=111, tag_tid=17, entry_tid=18267' 2024-04-06T09:43:20.849 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:43:20.849 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:43:20.850 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=17 2024-04-06T09:43:20.851 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=113, tag_tid=18, entry_tid=1' 2024-04-06T09:43:20.851 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:43:20.851 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:43:20.852 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:43:20.853 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=111, tag_tid=17, entry_tid=18267' 2024-04-06T09:43:20.853 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:43:20.853 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:43:20.854 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=18267 2024-04-06T09:43:20.854 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 18 = 17 -a 1 -le 18267 2024-04-06T09:43:20.854 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:43:20.854 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:43:20.854 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:43:21.056 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:43:21.056 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:43:21.056 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:43:21.056 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:43:21.056 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:43:21.056 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:43:21.057 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:43:21.057 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:43:21.057 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:43:21.057 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:43:21.057 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:43:21.057 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:43:21.057 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:43:21.057 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:43:21.057 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:43:21.057 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:43:21.058 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:43:21.061 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:43:21.061 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:43:21.061 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:43:21.061 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:43:21.061 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:43:21.061 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:43:21.061 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:43:21.061 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:43:21.061 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:43:21.061 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:43:21.061 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:43:21.061 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:43:21.061 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:43:21.061 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:43:21.061 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:43:21.062 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:43:21.062 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:43:21.062 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:43:21.062 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:43:21.062 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:43:21.062 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:43:21.062 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:43:21.062 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:43:21.062 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:43:21.062 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:43:21.062 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:43:21.062 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:43:21.062 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:43:21.063 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:43:21.063 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:43:21.063 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:43:21.063 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:43:21.086 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:43:21.086 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:43:21.086 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:43:21.086 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:43:21.222 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:43:21.223 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:43:21.224 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:43:21.224 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:43:21.224 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:43:21.224 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:43:21.224 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:43:21.224 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:43:21.224 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:43:21.224 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:43:21.225 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:43:21.225 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:43:21.225 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:43:21.226 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:43:21.226 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:43:21.227 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:43:21.278 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 27 2024-04-06T09:43:21.278 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 28 2024-04-06T09:43:21.278 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:43:21.278 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=113, tag_tid=18, entry_tid=1], [object_number=112, tag_tid=18, entry_tid=0], [object_number=114, tag_tid=17, entry_tid=20202], [object_number=115, tag_tid=17, entry_tid=20199]]], state=connected] 2024-04-06T09:43:21.278 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=109, tag_tid=17, entry_tid=18345], [object_number=108, tag_tid=17, entry_tid=18344], [object_number=111, tag_tid=17, entry_tid=18343], [object_number=110, tag_tid=17, entry_tid=18342]]], state=connected] 2024-04-06T09:43:21.281 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:43:21.283 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=113, tag_tid=18, entry_tid=1' 2024-04-06T09:43:21.284 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:43:21.284 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:43:21.284 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:43:21.284 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:43:21.284 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:43:21.284 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:43:21.284 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:43:21.284 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:43:21.284 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:43:21.285 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:43:21.285 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:43:21.285 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:43:21.285 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:43:21.285 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:43:21.286 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:43:21.331 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 27 2024-04-06T09:43:21.331 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 28 2024-04-06T09:43:21.331 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:43:21.331 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=113, tag_tid=18, entry_tid=1], [object_number=112, tag_tid=18, entry_tid=0], [object_number=114, tag_tid=17, entry_tid=20202], [object_number=115, tag_tid=17, entry_tid=20199]]], state=connected] 2024-04-06T09:43:21.332 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=109, tag_tid=17, entry_tid=18345], [object_number=108, tag_tid=17, entry_tid=18344], [object_number=111, tag_tid=17, entry_tid=18343], [object_number=110, tag_tid=17, entry_tid=18342]]], state=connected] 2024-04-06T09:43:21.335 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:43:21.337 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=109, tag_tid=17, entry_tid=18345' 2024-04-06T09:43:21.337 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=113, tag_tid=18, entry_tid=1' -a 'object_number=113, tag_tid=18, entry_tid=1' = 'object_number=109, tag_tid=17, entry_tid=18345' 2024-04-06T09:43:21.337 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=109, tag_tid=17, entry_tid=18345' '!=' 'object_number=111, tag_tid=17, entry_tid=18267' 2024-04-06T09:43:21.337 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:43:21.337 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=109, tag_tid=17, entry_tid=18345' = 'object_number=111, tag_tid=17, entry_tid=18267' 2024-04-06T09:43:21.337 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=109, tag_tid=17, entry_tid=18345' 2024-04-06T09:43:21.337 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=113, tag_tid=18, entry_tid=1' 2024-04-06T09:43:21.337 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:43:21.338 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:43:21.338 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=18 2024-04-06T09:43:21.339 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=109, tag_tid=17, entry_tid=18345' 2024-04-06T09:43:21.339 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:43:21.339 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:43:21.340 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=17 2024-04-06T09:43:21.341 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=113, tag_tid=18, entry_tid=1' 2024-04-06T09:43:21.341 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:43:21.341 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:43:21.342 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:43:21.342 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=109, tag_tid=17, entry_tid=18345' 2024-04-06T09:43:21.343 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:43:21.343 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:43:21.344 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=18345 2024-04-06T09:43:21.344 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 18 = 17 -a 1 -le 18345 2024-04-06T09:43:21.344 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:43:21.344 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:43:21.344 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:43:21.545 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:43:21.546 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:43:21.546 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:43:21.546 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:43:21.546 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:43:21.546 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:43:21.546 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:43:21.546 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:43:21.546 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:43:21.546 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:43:21.546 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:43:21.546 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:43:21.547 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:43:21.547 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:43:21.547 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:43:21.547 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:43:21.547 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:43:21.550 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:43:21.550 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:43:21.551 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:43:21.551 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:43:21.551 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:43:21.551 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:43:21.551 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:43:21.551 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:43:21.551 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:43:21.551 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:43:21.551 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:43:21.551 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:43:21.551 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:43:21.551 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:43:21.551 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:43:21.551 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:43:21.551 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:43:21.552 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:43:21.552 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:43:21.552 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:43:21.552 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:43:21.552 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:43:21.552 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:43:21.552 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:43:21.552 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:43:21.552 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:43:21.552 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:43:21.552 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:43:21.552 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:43:21.552 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:43:21.552 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:43:21.552 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:43:21.577 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:43:21.577 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:43:21.577 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:43:21.577 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:43:21.709 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:43:21.710 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:43:21.710 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:43:21.710 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:43:21.710 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:43:21.710 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:43:21.710 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:43:21.710 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:43:21.711 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:43:21.711 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:43:21.711 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:43:21.711 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:43:21.711 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:43:21.714 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:43:21.714 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:43:21.714 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:43:21.771 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 27 2024-04-06T09:43:21.771 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 28 2024-04-06T09:43:21.771 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:43:21.771 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=113, tag_tid=18, entry_tid=1], [object_number=112, tag_tid=18, entry_tid=0], [object_number=114, tag_tid=17, entry_tid=20202], [object_number=115, tag_tid=17, entry_tid=20199]]], state=connected] 2024-04-06T09:43:21.771 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=110, tag_tid=17, entry_tid=18418], [object_number=109, tag_tid=17, entry_tid=18417], [object_number=108, tag_tid=17, entry_tid=18416], [object_number=111, tag_tid=17, entry_tid=18415]]], state=connected] 2024-04-06T09:43:21.774 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:43:21.776 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=113, tag_tid=18, entry_tid=1' 2024-04-06T09:43:21.777 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:43:21.777 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:43:21.777 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:43:21.777 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:43:21.777 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:43:21.777 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:43:21.778 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:43:21.778 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:43:21.778 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:43:21.778 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:43:21.778 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:43:21.778 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:43:21.778 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:43:21.779 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:43:21.779 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:43:21.825 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 27 2024-04-06T09:43:21.826 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 28 2024-04-06T09:43:21.826 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:43:21.826 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=113, tag_tid=18, entry_tid=1], [object_number=112, tag_tid=18, entry_tid=0], [object_number=114, tag_tid=17, entry_tid=20202], [object_number=115, tag_tid=17, entry_tid=20199]]], state=connected] 2024-04-06T09:43:21.826 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=110, tag_tid=17, entry_tid=18418], [object_number=109, tag_tid=17, entry_tid=18417], [object_number=108, tag_tid=17, entry_tid=18416], [object_number=111, tag_tid=17, entry_tid=18415]]], state=connected] 2024-04-06T09:43:21.829 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:43:21.831 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=110, tag_tid=17, entry_tid=18418' 2024-04-06T09:43:21.831 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=113, tag_tid=18, entry_tid=1' -a 'object_number=113, tag_tid=18, entry_tid=1' = 'object_number=110, tag_tid=17, entry_tid=18418' 2024-04-06T09:43:21.832 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=110, tag_tid=17, entry_tid=18418' '!=' 'object_number=109, tag_tid=17, entry_tid=18345' 2024-04-06T09:43:21.832 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:43:21.832 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=110, tag_tid=17, entry_tid=18418' = 'object_number=109, tag_tid=17, entry_tid=18345' 2024-04-06T09:43:21.832 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=110, tag_tid=17, entry_tid=18418' 2024-04-06T09:43:21.832 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=113, tag_tid=18, entry_tid=1' 2024-04-06T09:43:21.832 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:43:21.832 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:43:21.832 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=18 2024-04-06T09:43:21.833 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=110, tag_tid=17, entry_tid=18418' 2024-04-06T09:43:21.834 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:43:21.834 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:43:21.834 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=17 2024-04-06T09:43:21.835 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=113, tag_tid=18, entry_tid=1' 2024-04-06T09:43:21.835 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:43:21.835 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:43:21.836 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:43:21.836 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=110, tag_tid=17, entry_tid=18418' 2024-04-06T09:43:21.837 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:43:21.837 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:43:21.838 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=18418 2024-04-06T09:43:21.838 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 18 = 17 -a 1 -le 18418 2024-04-06T09:43:21.838 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:43:21.838 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:43:21.838 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:43:22.040 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:43:22.040 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:43:22.040 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:43:22.040 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:43:22.040 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:43:22.040 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:43:22.041 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:43:22.041 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:43:22.041 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:43:22.041 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:43:22.041 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:43:22.041 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:43:22.041 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:43:22.041 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:43:22.041 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:43:22.042 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:43:22.042 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:43:22.045 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:43:22.045 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:43:22.045 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:43:22.045 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:43:22.045 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:43:22.045 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:43:22.045 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:43:22.046 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:43:22.046 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:43:22.046 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:43:22.046 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:43:22.046 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:43:22.046 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:43:22.046 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:43:22.046 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:43:22.046 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:43:22.046 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:43:22.046 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:43:22.046 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:43:22.046 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:43:22.046 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:43:22.046 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:43:22.047 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:43:22.047 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:43:22.047 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:43:22.047 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:43:22.047 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:43:22.047 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:43:22.047 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:43:22.047 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:43:22.047 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:43:22.047 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:43:22.070 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:43:22.070 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:43:22.071 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:43:22.071 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:43:22.200 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:43:22.201 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:43:22.201 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:43:22.201 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:43:22.201 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:43:22.201 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:43:22.201 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:43:22.201 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:43:22.201 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:43:22.201 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:43:22.202 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:43:22.202 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:43:22.202 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:43:22.205 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:43:22.205 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:43:22.205 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:43:22.256 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 27 2024-04-06T09:43:22.257 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 28 2024-04-06T09:43:22.257 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:43:22.257 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=113, tag_tid=18, entry_tid=1], [object_number=112, tag_tid=18, entry_tid=0], [object_number=114, tag_tid=17, entry_tid=20202], [object_number=115, tag_tid=17, entry_tid=20199]]], state=connected] 2024-04-06T09:43:22.257 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=111, tag_tid=17, entry_tid=18495], [object_number=110, tag_tid=17, entry_tid=18494], [object_number=109, tag_tid=17, entry_tid=18493], [object_number=108, tag_tid=17, entry_tid=18492]]], state=connected] 2024-04-06T09:43:22.260 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:43:22.261 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=113, tag_tid=18, entry_tid=1' 2024-04-06T09:43:22.263 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:43:22.263 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:43:22.263 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:43:22.263 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:43:22.263 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:43:22.263 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:43:22.263 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:43:22.263 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:43:22.263 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:43:22.263 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:43:22.264 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:43:22.264 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:43:22.264 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:43:22.264 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:43:22.264 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:43:22.310 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 27 2024-04-06T09:43:22.311 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 28 2024-04-06T09:43:22.311 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:43:22.311 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=113, tag_tid=18, entry_tid=1], [object_number=112, tag_tid=18, entry_tid=0], [object_number=114, tag_tid=17, entry_tid=20202], [object_number=115, tag_tid=17, entry_tid=20199]]], state=connected] 2024-04-06T09:43:22.311 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=111, tag_tid=17, entry_tid=18495], [object_number=110, tag_tid=17, entry_tid=18494], [object_number=109, tag_tid=17, entry_tid=18493], [object_number=108, tag_tid=17, entry_tid=18492]]], state=connected] 2024-04-06T09:43:22.314 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:43:22.316 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=111, tag_tid=17, entry_tid=18495' 2024-04-06T09:43:22.316 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=113, tag_tid=18, entry_tid=1' -a 'object_number=113, tag_tid=18, entry_tid=1' = 'object_number=111, tag_tid=17, entry_tid=18495' 2024-04-06T09:43:22.316 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=111, tag_tid=17, entry_tid=18495' '!=' 'object_number=110, tag_tid=17, entry_tid=18418' 2024-04-06T09:43:22.316 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:43:22.317 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=111, tag_tid=17, entry_tid=18495' = 'object_number=110, tag_tid=17, entry_tid=18418' 2024-04-06T09:43:22.317 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=111, tag_tid=17, entry_tid=18495' 2024-04-06T09:43:22.317 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=113, tag_tid=18, entry_tid=1' 2024-04-06T09:43:22.317 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:43:22.317 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:43:22.317 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=18 2024-04-06T09:43:22.318 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=111, tag_tid=17, entry_tid=18495' 2024-04-06T09:43:22.318 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:43:22.318 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:43:22.319 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=17 2024-04-06T09:43:22.320 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=113, tag_tid=18, entry_tid=1' 2024-04-06T09:43:22.320 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:43:22.320 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:43:22.321 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:43:22.321 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=111, tag_tid=17, entry_tid=18495' 2024-04-06T09:43:22.322 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:43:22.322 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:43:22.323 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=18495 2024-04-06T09:43:22.323 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 18 = 17 -a 1 -le 18495 2024-04-06T09:43:22.323 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:43:22.323 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:43:22.323 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:43:22.525 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:43:22.525 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:43:22.525 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:43:22.525 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:43:22.525 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:43:22.525 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:43:22.525 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:43:22.525 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:43:22.525 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:43:22.526 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:43:22.526 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:43:22.526 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:43:22.526 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:43:22.526 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:43:22.526 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:43:22.526 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:43:22.527 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:43:22.529 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:43:22.529 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:43:22.529 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:43:22.529 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:43:22.529 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:43:22.529 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:43:22.529 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:43:22.529 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:43:22.529 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:43:22.529 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:43:22.529 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:43:22.529 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:43:22.530 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:43:22.530 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:43:22.530 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:43:22.530 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:43:22.531 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:43:22.531 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:43:22.531 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:43:22.531 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:43:22.532 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:43:22.532 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:43:22.532 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:43:22.532 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:43:22.532 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:43:22.532 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:43:22.532 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:43:22.532 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:43:22.532 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:43:22.532 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:43:22.532 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:43:22.532 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:43:22.557 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:43:22.558 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:43:22.558 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:43:22.558 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:43:22.707 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:43:22.709 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:43:22.709 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:43:22.709 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:43:22.709 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:43:22.709 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:43:22.709 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:43:22.709 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:43:22.709 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:43:22.709 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:43:22.709 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:43:22.710 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:43:22.711 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:43:22.712 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:43:22.712 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:43:22.713 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:43:22.779 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 28 2024-04-06T09:43:22.779 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 28 2024-04-06T09:43:22.779 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:43:22.779 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=113, tag_tid=18, entry_tid=1], [object_number=112, tag_tid=18, entry_tid=0], [object_number=114, tag_tid=17, entry_tid=20202], [object_number=115, tag_tid=17, entry_tid=20199]]], state=connected] 2024-04-06T09:43:22.779 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=113, tag_tid=17, entry_tid=18553], [object_number=112, tag_tid=17, entry_tid=18552], [object_number=115, tag_tid=17, entry_tid=18551], [object_number=114, tag_tid=17, entry_tid=18550]]], state=connected] 2024-04-06T09:43:22.782 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:43:22.783 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=113, tag_tid=18, entry_tid=1' 2024-04-06T09:43:22.785 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:43:22.785 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:43:22.785 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:43:22.785 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:43:22.785 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:43:22.785 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:43:22.785 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:43:22.785 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:43:22.785 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:43:22.785 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:43:22.786 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:43:22.786 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:43:22.786 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:43:22.786 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:43:22.786 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:43:22.833 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 28 2024-04-06T09:43:22.833 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 28 2024-04-06T09:43:22.833 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:43:22.833 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=113, tag_tid=18, entry_tid=1], [object_number=112, tag_tid=18, entry_tid=0], [object_number=114, tag_tid=17, entry_tid=20202], [object_number=115, tag_tid=17, entry_tid=20199]]], state=connected] 2024-04-06T09:43:22.833 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=113, tag_tid=17, entry_tid=18553], [object_number=112, tag_tid=17, entry_tid=18552], [object_number=115, tag_tid=17, entry_tid=18551], [object_number=114, tag_tid=17, entry_tid=18550]]], state=connected] 2024-04-06T09:43:22.836 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:43:22.838 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=113, tag_tid=17, entry_tid=18553' 2024-04-06T09:43:22.838 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=113, tag_tid=18, entry_tid=1' -a 'object_number=113, tag_tid=18, entry_tid=1' = 'object_number=113, tag_tid=17, entry_tid=18553' 2024-04-06T09:43:22.838 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=113, tag_tid=17, entry_tid=18553' '!=' 'object_number=111, tag_tid=17, entry_tid=18495' 2024-04-06T09:43:22.838 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:43:22.839 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=113, tag_tid=17, entry_tid=18553' = 'object_number=111, tag_tid=17, entry_tid=18495' 2024-04-06T09:43:22.839 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=113, tag_tid=17, entry_tid=18553' 2024-04-06T09:43:22.839 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=113, tag_tid=18, entry_tid=1' 2024-04-06T09:43:22.839 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:43:22.839 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:43:22.839 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=18 2024-04-06T09:43:22.840 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=113, tag_tid=17, entry_tid=18553' 2024-04-06T09:43:22.840 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:43:22.840 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:43:22.841 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=17 2024-04-06T09:43:22.842 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=113, tag_tid=18, entry_tid=1' 2024-04-06T09:43:22.842 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:43:22.842 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:43:22.842 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:43:22.843 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=113, tag_tid=17, entry_tid=18553' 2024-04-06T09:43:22.843 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:43:22.843 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:43:22.845 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=18553 2024-04-06T09:43:22.845 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 18 = 17 -a 1 -le 18553 2024-04-06T09:43:22.845 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:43:22.845 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:43:22.845 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:43:23.046 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:43:23.047 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:43:23.047 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:43:23.047 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:43:23.047 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:43:23.047 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:43:23.047 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:43:23.047 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:43:23.047 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:43:23.047 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:43:23.047 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:43:23.047 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:43:23.048 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:43:23.048 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:43:23.048 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:43:23.048 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:43:23.048 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:43:23.052 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:43:23.052 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:43:23.052 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:43:23.052 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:43:23.052 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:43:23.052 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:43:23.052 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:43:23.053 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:43:23.053 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:43:23.053 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:43:23.053 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:43:23.053 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:43:23.053 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:43:23.053 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:43:23.054 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:43:23.054 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:43:23.054 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:43:23.054 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:43:23.054 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:43:23.054 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:43:23.054 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:43:23.055 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:43:23.055 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:43:23.055 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:43:23.055 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:43:23.055 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:43:23.055 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:43:23.055 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:43:23.055 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:43:23.056 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:43:23.056 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:43:23.056 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:43:23.078 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:43:23.078 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:43:23.078 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:43:23.078 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:43:23.205 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:43:23.206 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:43:23.207 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:43:23.207 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:43:23.207 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:43:23.207 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:43:23.207 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:43:23.207 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:43:23.207 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:43:23.207 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:43:23.207 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:43:23.207 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:43:23.207 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:43:23.209 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:43:23.209 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:43:23.209 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:43:23.255 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 28 2024-04-06T09:43:23.256 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 28 2024-04-06T09:43:23.256 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:43:23.256 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=113, tag_tid=18, entry_tid=1], [object_number=112, tag_tid=18, entry_tid=0], [object_number=114, tag_tid=17, entry_tid=20202], [object_number=115, tag_tid=17, entry_tid=20199]]], state=connected] 2024-04-06T09:43:23.256 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=112, tag_tid=17, entry_tid=18644], [object_number=115, tag_tid=17, entry_tid=18643], [object_number=114, tag_tid=17, entry_tid=18642], [object_number=113, tag_tid=17, entry_tid=18641]]], state=connected] 2024-04-06T09:43:23.259 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:43:23.260 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=113, tag_tid=18, entry_tid=1' 2024-04-06T09:43:23.261 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:43:23.261 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:43:23.261 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:43:23.262 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:43:23.262 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:43:23.262 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:43:23.262 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:43:23.262 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:43:23.262 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:43:23.262 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:43:23.262 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:43:23.262 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:43:23.263 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:43:23.263 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:43:23.263 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:43:23.309 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 28 2024-04-06T09:43:23.310 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 28 2024-04-06T09:43:23.310 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:43:23.310 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=113, tag_tid=18, entry_tid=1], [object_number=112, tag_tid=18, entry_tid=0], [object_number=114, tag_tid=17, entry_tid=20202], [object_number=115, tag_tid=17, entry_tid=20199]]], state=connected] 2024-04-06T09:43:23.310 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=112, tag_tid=17, entry_tid=18644], [object_number=115, tag_tid=17, entry_tid=18643], [object_number=114, tag_tid=17, entry_tid=18642], [object_number=113, tag_tid=17, entry_tid=18641]]], state=connected] 2024-04-06T09:43:23.313 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:43:23.315 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=112, tag_tid=17, entry_tid=18644' 2024-04-06T09:43:23.315 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=113, tag_tid=18, entry_tid=1' -a 'object_number=113, tag_tid=18, entry_tid=1' = 'object_number=112, tag_tid=17, entry_tid=18644' 2024-04-06T09:43:23.315 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=112, tag_tid=17, entry_tid=18644' '!=' 'object_number=113, tag_tid=17, entry_tid=18553' 2024-04-06T09:43:23.315 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:43:23.315 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=112, tag_tid=17, entry_tid=18644' = 'object_number=113, tag_tid=17, entry_tid=18553' 2024-04-06T09:43:23.315 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=112, tag_tid=17, entry_tid=18644' 2024-04-06T09:43:23.315 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=113, tag_tid=18, entry_tid=1' 2024-04-06T09:43:23.316 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:43:23.316 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:43:23.316 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=18 2024-04-06T09:43:23.317 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=112, tag_tid=17, entry_tid=18644' 2024-04-06T09:43:23.317 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:43:23.317 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:43:23.318 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=17 2024-04-06T09:43:23.320 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=113, tag_tid=18, entry_tid=1' 2024-04-06T09:43:23.320 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:43:23.320 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:43:23.320 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:43:23.321 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=112, tag_tid=17, entry_tid=18644' 2024-04-06T09:43:23.321 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:43:23.322 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:43:23.323 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=18644 2024-04-06T09:43:23.323 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 18 = 17 -a 1 -le 18644 2024-04-06T09:43:23.323 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:43:23.323 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:43:23.323 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:43:23.524 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:43:23.524 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:43:23.524 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:43:23.525 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:43:23.525 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:43:23.525 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:43:23.525 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:43:23.525 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:43:23.525 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:43:23.525 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:43:23.525 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:43:23.525 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:43:23.525 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:43:23.525 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:43:23.525 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:43:23.526 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:43:23.526 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:43:23.528 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:43:23.528 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:43:23.528 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:43:23.528 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:43:23.528 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:43:23.529 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:43:23.529 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:43:23.529 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:43:23.529 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:43:23.529 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:43:23.529 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:43:23.529 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:43:23.529 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:43:23.529 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:43:23.529 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:43:23.529 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:43:23.529 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:43:23.529 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:43:23.529 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:43:23.529 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:43:23.530 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:43:23.530 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:43:23.530 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:43:23.530 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:43:23.530 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:43:23.530 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:43:23.530 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:43:23.530 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:43:23.530 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:43:23.530 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:43:23.530 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:43:23.530 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:43:23.556 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:43:23.556 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:43:23.557 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:43:23.557 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:43:23.694 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:43:23.695 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:43:23.695 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:43:23.696 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:43:23.696 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:43:23.696 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:43:23.696 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:43:23.696 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:43:23.696 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:43:23.696 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:43:23.697 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:43:23.697 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:43:23.697 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:43:23.698 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:43:23.702 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:43:23.702 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:43:23.744 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 28 2024-04-06T09:43:23.744 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 28 2024-04-06T09:43:23.744 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:43:23.744 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=113, tag_tid=18, entry_tid=1], [object_number=112, tag_tid=18, entry_tid=0], [object_number=114, tag_tid=17, entry_tid=20202], [object_number=115, tag_tid=17, entry_tid=20199]]], state=connected] 2024-04-06T09:43:23.744 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=114, tag_tid=17, entry_tid=18730], [object_number=113, tag_tid=17, entry_tid=18729], [object_number=112, tag_tid=17, entry_tid=18728], [object_number=115, tag_tid=17, entry_tid=18727]]], state=connected] 2024-04-06T09:43:23.748 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:43:23.749 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=113, tag_tid=18, entry_tid=1' 2024-04-06T09:43:23.750 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:43:23.750 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:43:23.750 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:43:23.751 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:43:23.751 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:43:23.751 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:43:23.751 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:43:23.751 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:43:23.751 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:43:23.751 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:43:23.751 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:43:23.751 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:43:23.752 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:43:23.752 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:43:23.752 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:43:23.798 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 28 2024-04-06T09:43:23.798 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 28 2024-04-06T09:43:23.798 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:43:23.798 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=113, tag_tid=18, entry_tid=1], [object_number=112, tag_tid=18, entry_tid=0], [object_number=114, tag_tid=17, entry_tid=20202], [object_number=115, tag_tid=17, entry_tid=20199]]], state=connected] 2024-04-06T09:43:23.798 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=114, tag_tid=17, entry_tid=18730], [object_number=113, tag_tid=17, entry_tid=18729], [object_number=112, tag_tid=17, entry_tid=18728], [object_number=115, tag_tid=17, entry_tid=18727]]], state=connected] 2024-04-06T09:43:23.801 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:43:23.803 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=114, tag_tid=17, entry_tid=18730' 2024-04-06T09:43:23.803 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=113, tag_tid=18, entry_tid=1' -a 'object_number=113, tag_tid=18, entry_tid=1' = 'object_number=114, tag_tid=17, entry_tid=18730' 2024-04-06T09:43:23.804 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=114, tag_tid=17, entry_tid=18730' '!=' 'object_number=112, tag_tid=17, entry_tid=18644' 2024-04-06T09:43:23.804 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:43:23.804 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=114, tag_tid=17, entry_tid=18730' = 'object_number=112, tag_tid=17, entry_tid=18644' 2024-04-06T09:43:23.804 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=114, tag_tid=17, entry_tid=18730' 2024-04-06T09:43:23.804 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=113, tag_tid=18, entry_tid=1' 2024-04-06T09:43:23.804 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:43:23.804 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:43:23.804 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=18 2024-04-06T09:43:23.805 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=114, tag_tid=17, entry_tid=18730' 2024-04-06T09:43:23.805 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:43:23.805 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:43:23.806 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=17 2024-04-06T09:43:23.807 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=113, tag_tid=18, entry_tid=1' 2024-04-06T09:43:23.807 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:43:23.807 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:43:23.808 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:43:23.809 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=114, tag_tid=17, entry_tid=18730' 2024-04-06T09:43:23.809 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:43:23.809 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:43:23.810 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=18730 2024-04-06T09:43:23.810 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 18 = 17 -a 1 -le 18730 2024-04-06T09:43:23.810 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:43:23.810 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:43:23.810 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:43:24.012 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:43:24.012 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:43:24.012 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:43:24.012 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:43:24.012 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:43:24.012 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:43:24.013 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:43:24.013 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:43:24.013 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:43:24.013 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:43:24.013 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:43:24.013 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:43:24.013 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:43:24.013 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:43:24.013 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:43:24.014 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:43:24.014 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:43:24.015 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:43:24.016 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:43:24.016 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:43:24.016 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:43:24.016 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:43:24.016 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:43:24.016 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:43:24.016 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:43:24.016 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:43:24.016 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:43:24.016 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:43:24.016 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:43:24.016 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:43:24.016 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:43:24.016 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:43:24.017 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:43:24.017 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:43:24.017 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:43:24.018 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:43:24.018 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:43:24.018 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:43:24.018 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:43:24.018 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:43:24.018 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:43:24.018 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:43:24.018 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:43:24.018 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:43:24.018 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:43:24.018 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:43:24.018 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:43:24.018 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:43:24.018 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:43:24.042 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:43:24.042 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:43:24.043 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:43:24.043 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:43:24.176 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:43:24.177 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:43:24.177 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:43:24.177 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:43:24.177 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:43:24.177 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:43:24.177 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:43:24.177 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:43:24.178 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:43:24.178 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:43:24.178 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:43:24.178 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:43:24.179 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:43:24.180 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:43:24.180 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:43:24.180 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:43:24.226 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 28 2024-04-06T09:43:24.226 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 28 2024-04-06T09:43:24.226 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:43:24.226 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=113, tag_tid=18, entry_tid=1], [object_number=112, tag_tid=18, entry_tid=0], [object_number=114, tag_tid=17, entry_tid=20202], [object_number=115, tag_tid=17, entry_tid=20199]]], state=connected] 2024-04-06T09:43:24.226 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=115, tag_tid=17, entry_tid=18807], [object_number=114, tag_tid=17, entry_tid=18806], [object_number=113, tag_tid=17, entry_tid=18805], [object_number=112, tag_tid=17, entry_tid=18804]]], state=connected] 2024-04-06T09:43:24.229 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:43:24.231 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=113, tag_tid=18, entry_tid=1' 2024-04-06T09:43:24.232 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:43:24.232 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:43:24.232 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:43:24.232 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:43:24.232 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:43:24.232 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:43:24.232 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:43:24.233 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:43:24.233 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:43:24.233 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:43:24.233 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:43:24.233 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:43:24.233 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:43:24.233 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:43:24.233 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:43:24.280 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 28 2024-04-06T09:43:24.281 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 28 2024-04-06T09:43:24.281 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:43:24.281 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=113, tag_tid=18, entry_tid=1], [object_number=112, tag_tid=18, entry_tid=0], [object_number=114, tag_tid=17, entry_tid=20202], [object_number=115, tag_tid=17, entry_tid=20199]]], state=connected] 2024-04-06T09:43:24.281 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=115, tag_tid=17, entry_tid=18807], [object_number=114, tag_tid=17, entry_tid=18806], [object_number=113, tag_tid=17, entry_tid=18805], [object_number=112, tag_tid=17, entry_tid=18804]]], state=connected] 2024-04-06T09:43:24.284 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:43:24.286 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=115, tag_tid=17, entry_tid=18807' 2024-04-06T09:43:24.286 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=113, tag_tid=18, entry_tid=1' -a 'object_number=113, tag_tid=18, entry_tid=1' = 'object_number=115, tag_tid=17, entry_tid=18807' 2024-04-06T09:43:24.286 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=115, tag_tid=17, entry_tid=18807' '!=' 'object_number=114, tag_tid=17, entry_tid=18730' 2024-04-06T09:43:24.286 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:43:24.286 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=115, tag_tid=17, entry_tid=18807' = 'object_number=114, tag_tid=17, entry_tid=18730' 2024-04-06T09:43:24.286 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=115, tag_tid=17, entry_tid=18807' 2024-04-06T09:43:24.286 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=113, tag_tid=18, entry_tid=1' 2024-04-06T09:43:24.286 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:43:24.286 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:43:24.287 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=18 2024-04-06T09:43:24.288 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=115, tag_tid=17, entry_tid=18807' 2024-04-06T09:43:24.288 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:43:24.288 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:43:24.289 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=17 2024-04-06T09:43:24.290 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=113, tag_tid=18, entry_tid=1' 2024-04-06T09:43:24.290 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:43:24.290 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:43:24.291 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:43:24.291 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=115, tag_tid=17, entry_tid=18807' 2024-04-06T09:43:24.292 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:43:24.292 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:43:24.293 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=18807 2024-04-06T09:43:24.293 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 18 = 17 -a 1 -le 18807 2024-04-06T09:43:24.293 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:43:24.294 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:43:24.294 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:43:24.495 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:43:24.495 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:43:24.495 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:43:24.495 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:43:24.495 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:43:24.495 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:43:24.495 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:43:24.495 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:43:24.495 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:43:24.496 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:43:24.496 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:43:24.496 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:43:24.496 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:43:24.496 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:43:24.496 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:43:24.496 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:43:24.497 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:43:24.500 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:43:24.500 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:43:24.500 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:43:24.500 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:43:24.500 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:43:24.500 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:43:24.501 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:43:24.501 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:43:24.501 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:43:24.501 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:43:24.501 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:43:24.501 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:43:24.501 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:43:24.501 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:43:24.501 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:43:24.501 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:43:24.501 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:43:24.501 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:43:24.501 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:43:24.501 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:43:24.501 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:43:24.502 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:43:24.502 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:43:24.502 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:43:24.502 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:43:24.502 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:43:24.502 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:43:24.502 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:43:24.502 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:43:24.502 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:43:24.502 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:43:24.502 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:43:24.525 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:43:24.526 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:43:24.526 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:43:24.526 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:43:24.655 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:43:24.656 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:43:24.656 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:43:24.657 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:43:24.657 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:43:24.657 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:43:24.657 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:43:24.657 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:43:24.657 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:43:24.657 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:43:24.658 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:43:24.658 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:43:24.658 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:43:24.659 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:43:24.659 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:43:24.660 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:43:24.705 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 28 2024-04-06T09:43:24.706 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 28 2024-04-06T09:43:24.706 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:43:24.706 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=113, tag_tid=18, entry_tid=1], [object_number=112, tag_tid=18, entry_tid=0], [object_number=114, tag_tid=17, entry_tid=20202], [object_number=115, tag_tid=17, entry_tid=20199]]], state=connected] 2024-04-06T09:43:24.706 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=115, tag_tid=17, entry_tid=18887], [object_number=114, tag_tid=17, entry_tid=18886], [object_number=113, tag_tid=17, entry_tid=18885], [object_number=112, tag_tid=17, entry_tid=18884]]], state=connected] 2024-04-06T09:43:24.709 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:43:24.710 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=113, tag_tid=18, entry_tid=1' 2024-04-06T09:43:24.711 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:43:24.711 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:43:24.712 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:43:24.712 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:43:24.712 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:43:24.712 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:43:24.712 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:43:24.712 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:43:24.712 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:43:24.712 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:43:24.712 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:43:24.712 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:43:24.713 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:43:24.713 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:43:24.713 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:43:24.760 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 28 2024-04-06T09:43:24.761 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 28 2024-04-06T09:43:24.761 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:43:24.761 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=113, tag_tid=18, entry_tid=1], [object_number=112, tag_tid=18, entry_tid=0], [object_number=114, tag_tid=17, entry_tid=20202], [object_number=115, tag_tid=17, entry_tid=20199]]], state=connected] 2024-04-06T09:43:24.761 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=115, tag_tid=17, entry_tid=18887], [object_number=114, tag_tid=17, entry_tid=18886], [object_number=113, tag_tid=17, entry_tid=18885], [object_number=112, tag_tid=17, entry_tid=18884]]], state=connected] 2024-04-06T09:43:24.764 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:43:24.765 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=115, tag_tid=17, entry_tid=18887' 2024-04-06T09:43:24.765 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=113, tag_tid=18, entry_tid=1' -a 'object_number=113, tag_tid=18, entry_tid=1' = 'object_number=115, tag_tid=17, entry_tid=18887' 2024-04-06T09:43:24.765 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=115, tag_tid=17, entry_tid=18887' '!=' 'object_number=115, tag_tid=17, entry_tid=18807' 2024-04-06T09:43:24.766 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:43:24.766 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=115, tag_tid=17, entry_tid=18887' = 'object_number=115, tag_tid=17, entry_tid=18807' 2024-04-06T09:43:24.766 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=115, tag_tid=17, entry_tid=18887' 2024-04-06T09:43:24.766 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=113, tag_tid=18, entry_tid=1' 2024-04-06T09:43:24.766 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:43:24.766 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:43:24.767 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=18 2024-04-06T09:43:24.768 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=115, tag_tid=17, entry_tid=18887' 2024-04-06T09:43:24.768 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:43:24.768 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:43:24.769 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=17 2024-04-06T09:43:24.769 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=113, tag_tid=18, entry_tid=1' 2024-04-06T09:43:24.769 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:43:24.770 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:43:24.770 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:43:24.771 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=115, tag_tid=17, entry_tid=18887' 2024-04-06T09:43:24.771 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:43:24.771 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:43:24.772 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=18887 2024-04-06T09:43:24.772 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 18 = 17 -a 1 -le 18887 2024-04-06T09:43:24.773 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:43:24.773 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:43:24.773 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:43:24.975 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:43:24.975 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:43:24.976 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:43:24.976 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:43:24.976 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:43:24.976 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:43:24.976 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:43:24.976 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:43:24.976 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:43:24.976 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:43:24.976 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:43:24.976 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:43:24.976 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:43:24.976 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:43:24.976 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:43:24.976 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:43:24.977 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:43:24.980 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:43:24.980 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:43:24.981 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:43:24.981 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:43:24.981 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:43:24.981 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:43:24.981 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:43:24.981 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:43:24.981 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:43:24.981 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:43:24.981 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:43:24.981 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:43:24.981 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:43:24.981 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:43:24.981 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:43:24.982 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:43:24.982 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:43:24.982 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:43:24.982 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:43:24.982 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:43:24.982 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:43:24.982 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:43:24.982 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:43:24.982 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:43:24.982 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:43:24.982 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:43:24.982 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:43:24.982 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:43:24.982 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:43:24.983 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:43:24.983 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:43:24.983 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:43:25.007 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:43:25.007 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:43:25.007 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:43:25.007 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:43:25.143 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:43:25.145 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:43:25.145 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:43:25.145 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:43:25.145 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:43:25.145 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:43:25.145 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:43:25.145 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:43:25.145 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:43:25.145 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:43:25.145 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:43:25.146 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:43:25.146 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:43:25.148 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:43:25.149 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:43:25.149 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:43:25.204 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 28 2024-04-06T09:43:25.204 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 28 2024-04-06T09:43:25.204 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:43:25.204 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=113, tag_tid=18, entry_tid=1], [object_number=112, tag_tid=18, entry_tid=0], [object_number=114, tag_tid=17, entry_tid=20202], [object_number=115, tag_tid=17, entry_tid=20199]]], state=connected] 2024-04-06T09:43:25.204 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=112, tag_tid=17, entry_tid=18972], [object_number=115, tag_tid=17, entry_tid=18971], [object_number=114, tag_tid=17, entry_tid=18970], [object_number=113, tag_tid=17, entry_tid=18969]]], state=connected] 2024-04-06T09:43:25.208 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:43:25.209 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=113, tag_tid=18, entry_tid=1' 2024-04-06T09:43:25.210 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:43:25.210 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:43:25.210 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:43:25.210 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:43:25.210 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:43:25.211 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:43:25.211 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:43:25.211 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:43:25.211 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:43:25.211 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:43:25.211 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:43:25.211 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:43:25.212 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:43:25.212 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:43:25.212 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:43:25.258 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 28 2024-04-06T09:43:25.259 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 28 2024-04-06T09:43:25.259 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:43:25.259 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=113, tag_tid=18, entry_tid=1], [object_number=112, tag_tid=18, entry_tid=0], [object_number=114, tag_tid=17, entry_tid=20202], [object_number=115, tag_tid=17, entry_tid=20199]]], state=connected] 2024-04-06T09:43:25.259 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=112, tag_tid=17, entry_tid=18972], [object_number=115, tag_tid=17, entry_tid=18971], [object_number=114, tag_tid=17, entry_tid=18970], [object_number=113, tag_tid=17, entry_tid=18969]]], state=connected] 2024-04-06T09:43:25.262 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:43:25.264 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=112, tag_tid=17, entry_tid=18972' 2024-04-06T09:43:25.264 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=113, tag_tid=18, entry_tid=1' -a 'object_number=113, tag_tid=18, entry_tid=1' = 'object_number=112, tag_tid=17, entry_tid=18972' 2024-04-06T09:43:25.264 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=112, tag_tid=17, entry_tid=18972' '!=' 'object_number=115, tag_tid=17, entry_tid=18887' 2024-04-06T09:43:25.264 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:43:25.264 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=112, tag_tid=17, entry_tid=18972' = 'object_number=115, tag_tid=17, entry_tid=18887' 2024-04-06T09:43:25.264 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=112, tag_tid=17, entry_tid=18972' 2024-04-06T09:43:25.265 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=113, tag_tid=18, entry_tid=1' 2024-04-06T09:43:25.265 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:43:25.265 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:43:25.265 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=18 2024-04-06T09:43:25.266 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=112, tag_tid=17, entry_tid=18972' 2024-04-06T09:43:25.266 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:43:25.266 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:43:25.267 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=17 2024-04-06T09:43:25.268 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=113, tag_tid=18, entry_tid=1' 2024-04-06T09:43:25.268 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:43:25.268 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:43:25.269 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:43:25.270 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=112, tag_tid=17, entry_tid=18972' 2024-04-06T09:43:25.270 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:43:25.270 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:43:25.271 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=18972 2024-04-06T09:43:25.271 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 18 = 17 -a 1 -le 18972 2024-04-06T09:43:25.271 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:43:25.271 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:43:25.271 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:43:25.473 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:43:25.473 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:43:25.473 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:43:25.473 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:43:25.473 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:43:25.473 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:43:25.473 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:43:25.474 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:43:25.474 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:43:25.474 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:43:25.474 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:43:25.474 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:43:25.474 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:43:25.474 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:43:25.474 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:43:25.475 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:43:25.475 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:43:25.476 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:43:25.476 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:43:25.477 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:43:25.477 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:43:25.477 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:43:25.477 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:43:25.477 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:43:25.477 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:43:25.477 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:43:25.477 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:43:25.477 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:43:25.477 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:43:25.477 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:43:25.477 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:43:25.477 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:43:25.477 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:43:25.478 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:43:25.479 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:43:25.479 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:43:25.479 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:43:25.479 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:43:25.479 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:43:25.479 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:43:25.479 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:43:25.479 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:43:25.479 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:43:25.479 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:43:25.479 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:43:25.479 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:43:25.480 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:43:25.480 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:43:25.480 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:43:25.504 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:43:25.504 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:43:25.504 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:43:25.504 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:43:25.640 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:43:25.642 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:43:25.642 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:43:25.642 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:43:25.642 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:43:25.642 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:43:25.642 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:43:25.642 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:43:25.642 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:43:25.642 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:43:25.642 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:43:25.643 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:43:25.643 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:43:25.645 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:43:25.645 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:43:25.645 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:43:25.691 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 28 2024-04-06T09:43:25.691 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 28 2024-04-06T09:43:25.691 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:43:25.691 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=113, tag_tid=18, entry_tid=1], [object_number=112, tag_tid=18, entry_tid=0], [object_number=114, tag_tid=17, entry_tid=20202], [object_number=115, tag_tid=17, entry_tid=20199]]], state=connected] 2024-04-06T09:43:25.691 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=113, tag_tid=17, entry_tid=19057], [object_number=112, tag_tid=17, entry_tid=19056], [object_number=115, tag_tid=17, entry_tid=19055], [object_number=114, tag_tid=17, entry_tid=19054]]], state=connected] 2024-04-06T09:43:25.694 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:43:25.696 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=113, tag_tid=18, entry_tid=1' 2024-04-06T09:43:25.697 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:43:25.698 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:43:25.698 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:43:25.698 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:43:25.698 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:43:25.698 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:43:25.698 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:43:25.698 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:43:25.698 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:43:25.698 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:43:25.698 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:43:25.698 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:43:25.699 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:43:25.699 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:43:25.699 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:43:25.955 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 28 2024-04-06T09:43:25.955 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 28 2024-04-06T09:43:25.955 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:43:25.955 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=113, tag_tid=18, entry_tid=1], [object_number=112, tag_tid=18, entry_tid=0], [object_number=114, tag_tid=17, entry_tid=20202], [object_number=115, tag_tid=17, entry_tid=20199]]], state=connected] 2024-04-06T09:43:25.955 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=113, tag_tid=17, entry_tid=19057], [object_number=112, tag_tid=17, entry_tid=19056], [object_number=115, tag_tid=17, entry_tid=19055], [object_number=114, tag_tid=17, entry_tid=19054]]], state=connected] 2024-04-06T09:43:25.958 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:43:25.961 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=113, tag_tid=17, entry_tid=19057' 2024-04-06T09:43:25.961 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=113, tag_tid=18, entry_tid=1' -a 'object_number=113, tag_tid=18, entry_tid=1' = 'object_number=113, tag_tid=17, entry_tid=19057' 2024-04-06T09:43:25.961 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=113, tag_tid=17, entry_tid=19057' '!=' 'object_number=112, tag_tid=17, entry_tid=18972' 2024-04-06T09:43:25.961 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:43:25.961 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=113, tag_tid=17, entry_tid=19057' = 'object_number=112, tag_tid=17, entry_tid=18972' 2024-04-06T09:43:25.961 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=113, tag_tid=17, entry_tid=19057' 2024-04-06T09:43:25.961 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=113, tag_tid=18, entry_tid=1' 2024-04-06T09:43:25.962 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:43:25.962 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:43:25.964 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=18 2024-04-06T09:43:25.965 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=113, tag_tid=17, entry_tid=19057' 2024-04-06T09:43:25.965 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:43:25.965 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:43:25.967 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=17 2024-04-06T09:43:25.968 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=113, tag_tid=18, entry_tid=1' 2024-04-06T09:43:25.968 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:43:25.968 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:43:25.969 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:43:25.970 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=113, tag_tid=17, entry_tid=19057' 2024-04-06T09:43:25.970 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:43:25.970 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:43:25.972 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=19057 2024-04-06T09:43:25.972 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 18 = 17 -a 1 -le 19057 2024-04-06T09:43:25.972 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:43:25.972 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:43:25.972 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:43:26.175 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:43:26.175 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:43:26.175 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:43:26.175 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:43:26.175 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:43:26.175 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:43:26.175 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:43:26.175 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:43:26.176 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:43:26.176 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:43:26.176 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:43:26.176 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:43:26.176 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:43:26.176 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:43:26.176 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:43:26.176 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:43:26.178 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:43:26.179 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:43:26.179 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:43:26.179 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:43:26.179 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:43:26.179 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:43:26.179 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:43:26.179 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:43:26.179 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:43:26.179 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:43:26.179 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:43:26.179 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:43:26.179 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:43:26.179 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:43:26.179 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:43:26.180 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:43:26.180 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:43:26.180 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:43:26.180 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:43:26.180 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:43:26.180 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:43:26.180 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:43:26.180 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:43:26.180 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:43:26.180 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:43:26.180 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:43:26.180 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:43:26.180 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:43:26.180 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:43:26.180 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:43:26.180 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:43:26.181 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:43:26.181 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:43:26.205 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:43:26.205 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:43:26.205 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:43:26.205 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:43:26.330 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:43:26.332 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:43:26.332 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:43:26.332 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:43:26.332 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:43:26.332 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:43:26.332 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:43:26.332 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:43:26.332 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:43:26.332 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:43:26.332 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:43:26.333 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:43:26.333 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:43:26.334 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:43:26.334 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:43:26.335 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:43:26.386 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 28 2024-04-06T09:43:26.386 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 28 2024-04-06T09:43:26.386 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:43:26.386 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=113, tag_tid=18, entry_tid=1], [object_number=112, tag_tid=18, entry_tid=0], [object_number=114, tag_tid=17, entry_tid=20202], [object_number=115, tag_tid=17, entry_tid=20199]]], state=connected] 2024-04-06T09:43:26.386 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=115, tag_tid=17, entry_tid=19171], [object_number=114, tag_tid=17, entry_tid=19170], [object_number=113, tag_tid=17, entry_tid=19169], [object_number=112, tag_tid=17, entry_tid=19168]]], state=connected] 2024-04-06T09:43:26.390 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:43:26.391 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=113, tag_tid=18, entry_tid=1' 2024-04-06T09:43:26.392 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:43:26.392 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:43:26.392 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:43:26.392 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:43:26.392 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:43:26.392 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:43:26.393 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:43:26.393 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:43:26.393 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:43:26.393 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:43:26.393 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:43:26.393 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:43:26.393 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:43:26.394 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:43:26.394 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:43:26.440 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 28 2024-04-06T09:43:26.440 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 28 2024-04-06T09:43:26.440 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:43:26.440 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=113, tag_tid=18, entry_tid=1], [object_number=112, tag_tid=18, entry_tid=0], [object_number=114, tag_tid=17, entry_tid=20202], [object_number=115, tag_tid=17, entry_tid=20199]]], state=connected] 2024-04-06T09:43:26.441 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=115, tag_tid=17, entry_tid=19171], [object_number=114, tag_tid=17, entry_tid=19170], [object_number=113, tag_tid=17, entry_tid=19169], [object_number=112, tag_tid=17, entry_tid=19168]]], state=connected] 2024-04-06T09:43:26.444 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:43:26.445 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=115, tag_tid=17, entry_tid=19171' 2024-04-06T09:43:26.445 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=113, tag_tid=18, entry_tid=1' -a 'object_number=113, tag_tid=18, entry_tid=1' = 'object_number=115, tag_tid=17, entry_tid=19171' 2024-04-06T09:43:26.445 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=115, tag_tid=17, entry_tid=19171' '!=' 'object_number=113, tag_tid=17, entry_tid=19057' 2024-04-06T09:43:26.446 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:43:26.446 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=115, tag_tid=17, entry_tid=19171' = 'object_number=113, tag_tid=17, entry_tid=19057' 2024-04-06T09:43:26.446 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=115, tag_tid=17, entry_tid=19171' 2024-04-06T09:43:26.446 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=113, tag_tid=18, entry_tid=1' 2024-04-06T09:43:26.446 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:43:26.447 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:43:26.447 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=18 2024-04-06T09:43:26.447 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=115, tag_tid=17, entry_tid=19171' 2024-04-06T09:43:26.448 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:43:26.448 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:43:26.448 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=17 2024-04-06T09:43:26.449 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=113, tag_tid=18, entry_tid=1' 2024-04-06T09:43:26.449 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:43:26.449 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:43:26.450 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:43:26.451 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=115, tag_tid=17, entry_tid=19171' 2024-04-06T09:43:26.451 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:43:26.451 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:43:26.452 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=19171 2024-04-06T09:43:26.452 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 18 = 17 -a 1 -le 19171 2024-04-06T09:43:26.452 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:43:26.452 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:43:26.453 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:43:26.654 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:43:26.654 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:43:26.654 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:43:26.654 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:43:26.654 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:43:26.655 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:43:26.655 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:43:26.655 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:43:26.655 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:43:26.655 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:43:26.655 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:43:26.655 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:43:26.655 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:43:26.655 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:43:26.655 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:43:26.656 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:43:26.656 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:43:26.658 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:43:26.658 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:43:26.658 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:43:26.658 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:43:26.658 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:43:26.658 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:43:26.658 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:43:26.658 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:43:26.658 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:43:26.658 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:43:26.658 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:43:26.659 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:43:26.659 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:43:26.659 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:43:26.659 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:43:26.659 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:43:26.660 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:43:26.660 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:43:26.660 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:43:26.660 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:43:26.660 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:43:26.660 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:43:26.660 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:43:26.660 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:43:26.660 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:43:26.660 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:43:26.660 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:43:26.661 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:43:26.661 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:43:26.661 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:43:26.661 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:43:26.661 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:43:26.686 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:43:26.686 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:43:26.686 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:43:26.686 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:43:26.821 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:43:26.822 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:43:26.822 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:43:26.822 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:43:26.822 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:43:26.822 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:43:26.822 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:43:26.822 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:43:26.822 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:43:26.822 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:43:26.822 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:43:26.823 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:43:26.823 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:43:26.825 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:43:26.826 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:43:26.826 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:43:26.871 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 28 2024-04-06T09:43:26.872 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 28 2024-04-06T09:43:26.872 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:43:26.872 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=113, tag_tid=18, entry_tid=1], [object_number=112, tag_tid=18, entry_tid=0], [object_number=114, tag_tid=17, entry_tid=20202], [object_number=115, tag_tid=17, entry_tid=20199]]], state=connected] 2024-04-06T09:43:26.872 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=114, tag_tid=17, entry_tid=19246], [object_number=113, tag_tid=17, entry_tid=19245], [object_number=112, tag_tid=17, entry_tid=19244], [object_number=115, tag_tid=17, entry_tid=19243]]], state=connected] 2024-04-06T09:43:26.875 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:43:26.877 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=113, tag_tid=18, entry_tid=1' 2024-04-06T09:43:26.877 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:43:26.877 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:43:26.877 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:43:26.877 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:43:26.878 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:43:26.878 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:43:26.878 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:43:26.878 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:43:26.878 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:43:26.878 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:43:26.878 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:43:26.878 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:43:26.879 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:43:26.879 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:43:26.879 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:43:26.925 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 28 2024-04-06T09:43:26.925 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 28 2024-04-06T09:43:26.925 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:43:26.925 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=113, tag_tid=18, entry_tid=1], [object_number=112, tag_tid=18, entry_tid=0], [object_number=114, tag_tid=17, entry_tid=20202], [object_number=115, tag_tid=17, entry_tid=20199]]], state=connected] 2024-04-06T09:43:26.925 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=114, tag_tid=17, entry_tid=19246], [object_number=113, tag_tid=17, entry_tid=19245], [object_number=112, tag_tid=17, entry_tid=19244], [object_number=115, tag_tid=17, entry_tid=19243]]], state=connected] 2024-04-06T09:43:26.929 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:43:26.931 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=114, tag_tid=17, entry_tid=19246' 2024-04-06T09:43:26.931 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=113, tag_tid=18, entry_tid=1' -a 'object_number=113, tag_tid=18, entry_tid=1' = 'object_number=114, tag_tid=17, entry_tid=19246' 2024-04-06T09:43:26.931 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=114, tag_tid=17, entry_tid=19246' '!=' 'object_number=115, tag_tid=17, entry_tid=19171' 2024-04-06T09:43:26.931 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:43:26.931 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=114, tag_tid=17, entry_tid=19246' = 'object_number=115, tag_tid=17, entry_tid=19171' 2024-04-06T09:43:26.931 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=114, tag_tid=17, entry_tid=19246' 2024-04-06T09:43:26.931 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=113, tag_tid=18, entry_tid=1' 2024-04-06T09:43:26.931 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:43:26.931 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:43:26.932 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=18 2024-04-06T09:43:26.932 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=114, tag_tid=17, entry_tid=19246' 2024-04-06T09:43:26.932 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:43:26.933 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:43:26.933 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=17 2024-04-06T09:43:26.934 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=113, tag_tid=18, entry_tid=1' 2024-04-06T09:43:26.934 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:43:26.934 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:43:26.935 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:43:26.936 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=114, tag_tid=17, entry_tid=19246' 2024-04-06T09:43:26.936 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:43:26.936 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:43:26.937 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=19246 2024-04-06T09:43:26.937 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 18 = 17 -a 1 -le 19246 2024-04-06T09:43:26.937 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:43:26.937 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:43:26.937 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:43:27.139 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:43:27.139 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:43:27.139 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:43:27.139 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:43:27.139 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:43:27.139 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:43:27.139 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:43:27.139 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:43:27.139 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:43:27.139 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:43:27.140 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:43:27.140 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:43:27.140 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:43:27.140 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:43:27.140 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:43:27.140 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:43:27.140 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:43:27.146 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:43:27.146 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:43:27.146 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:43:27.146 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:43:27.146 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:43:27.147 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:43:27.147 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:43:27.147 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:43:27.147 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:43:27.147 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:43:27.147 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:43:27.147 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:43:27.147 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:43:27.147 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:43:27.147 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:43:27.147 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:43:27.147 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:43:27.147 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:43:27.147 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:43:27.147 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:43:27.148 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:43:27.148 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:43:27.148 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:43:27.148 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:43:27.148 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:43:27.148 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:43:27.148 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:43:27.148 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:43:27.148 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:43:27.148 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:43:27.148 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:43:27.148 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:43:27.170 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:43:27.170 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:43:27.171 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:43:27.171 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:43:27.300 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:43:27.302 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:43:27.302 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:43:27.302 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:43:27.302 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:43:27.302 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:43:27.302 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:43:27.302 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:43:27.302 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:43:27.303 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:43:27.303 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:43:27.303 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:43:27.303 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:43:27.304 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:43:27.304 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:43:27.304 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:43:27.352 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 28 2024-04-06T09:43:27.352 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 28 2024-04-06T09:43:27.352 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:43:27.353 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=113, tag_tid=18, entry_tid=1], [object_number=112, tag_tid=18, entry_tid=0], [object_number=114, tag_tid=17, entry_tid=20202], [object_number=115, tag_tid=17, entry_tid=20199]]], state=connected] 2024-04-06T09:43:27.353 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=113, tag_tid=17, entry_tid=19321], [object_number=112, tag_tid=17, entry_tid=19320], [object_number=115, tag_tid=17, entry_tid=19319], [object_number=114, tag_tid=17, entry_tid=19318]]], state=connected] 2024-04-06T09:43:27.356 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:43:27.357 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=113, tag_tid=18, entry_tid=1' 2024-04-06T09:43:27.358 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:43:27.358 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:43:27.358 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:43:27.358 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:43:27.359 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:43:27.359 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:43:27.359 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:43:27.359 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:43:27.359 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:43:27.359 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:43:27.359 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:43:27.359 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:43:27.360 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:43:27.360 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:43:27.360 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:43:27.397 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.0 with skip (0.9887173003022001) > weight (0.25) 2024-04-06T09:43:27.398 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.1 with skip (0.9120442841286961) > weight (0.25) 2024-04-06T09:43:27.398 INFO:tasks.rbd_mirror_thrash:kill cluster2.client.mirror.2 2024-04-06T09:43:27.398 INFO:tasks.rbd_mirror.cluster2.client.mirror.2:Sent signal 15 2024-04-06T09:43:27.398 INFO:tasks.rbd_mirror_thrash:waiting for 1 secs before reviving daemons 2024-04-06T09:43:27.399 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi192.stderr:2024-04-06T09:43:27.402+0000 7f6fd8fe0640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.2 (PID: 29034) UID: 1000 2024-04-06T09:43:27.409 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 28 2024-04-06T09:43:27.409 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 28 2024-04-06T09:43:27.409 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:43:27.410 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=113, tag_tid=18, entry_tid=1], [object_number=112, tag_tid=18, entry_tid=0], [object_number=114, tag_tid=17, entry_tid=20202], [object_number=115, tag_tid=17, entry_tid=20199]]], state=connected] 2024-04-06T09:43:27.410 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=113, tag_tid=17, entry_tid=19321], [object_number=112, tag_tid=17, entry_tid=19320], [object_number=115, tag_tid=17, entry_tid=19319], [object_number=114, tag_tid=17, entry_tid=19318]]], state=connected] 2024-04-06T09:43:27.417 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:43:27.419 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=113, tag_tid=17, entry_tid=19321' 2024-04-06T09:43:27.419 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=113, tag_tid=18, entry_tid=1' -a 'object_number=113, tag_tid=18, entry_tid=1' = 'object_number=113, tag_tid=17, entry_tid=19321' 2024-04-06T09:43:27.419 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=113, tag_tid=17, entry_tid=19321' '!=' 'object_number=114, tag_tid=17, entry_tid=19246' 2024-04-06T09:43:27.420 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:43:27.420 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=113, tag_tid=17, entry_tid=19321' = 'object_number=114, tag_tid=17, entry_tid=19246' 2024-04-06T09:43:27.420 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=113, tag_tid=17, entry_tid=19321' 2024-04-06T09:43:27.420 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:43:27.420 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:43:27.420 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=113, tag_tid=18, entry_tid=1' 2024-04-06T09:43:27.422 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=18 2024-04-06T09:43:27.422 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=113, tag_tid=17, entry_tid=19321' 2024-04-06T09:43:27.422 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:43:27.423 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:43:27.423 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=17 2024-04-06T09:43:27.424 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=113, tag_tid=18, entry_tid=1' 2024-04-06T09:43:27.424 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:43:27.424 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:43:27.425 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:43:27.426 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=113, tag_tid=17, entry_tid=19321' 2024-04-06T09:43:27.426 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:43:27.426 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:43:27.427 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=19321 2024-04-06T09:43:27.427 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 18 = 17 -a 1 -le 19321 2024-04-06T09:43:27.427 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:43:27.427 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:43:27.427 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:43:27.629 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:43:27.629 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:43:27.629 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:43:27.629 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:43:27.629 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:43:27.629 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:43:27.629 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:43:27.630 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:43:27.630 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:43:27.630 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:43:27.630 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:43:27.630 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:43:27.630 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:43:27.630 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:43:27.630 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:43:27.631 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:43:27.631 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:43:27.632 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:43:27.632 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:43:27.632 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:43:27.632 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:43:27.632 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:43:27.632 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:43:27.632 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:43:27.632 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:43:27.632 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:43:27.632 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:43:27.633 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:43:27.633 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:43:27.633 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:43:27.633 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:43:27.633 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:43:27.633 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:43:27.634 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:43:27.634 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:43:27.634 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:43:27.634 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:43:27.634 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:43:27.635 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:43:27.635 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:43:27.635 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:43:27.635 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:43:27.635 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:43:27.635 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:43:27.635 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:43:27.635 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:43:27.635 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:43:27.635 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:43:27.635 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:43:27.661 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:43:27.661 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:43:27.661 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:43:27.661 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:43:27.792 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:43:27.794 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:43:27.794 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:43:27.794 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:43:27.794 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:43:27.794 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:43:27.794 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:43:27.794 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:43:27.794 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:43:27.794 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:43:27.794 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:43:27.795 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:43:27.795 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:43:27.797 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:43:27.798 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:43:27.798 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:43:27.849 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 28 2024-04-06T09:43:27.849 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 28 2024-04-06T09:43:27.849 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:43:27.849 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=113, tag_tid=18, entry_tid=1], [object_number=112, tag_tid=18, entry_tid=0], [object_number=114, tag_tid=17, entry_tid=20202], [object_number=115, tag_tid=17, entry_tid=20199]]], state=connected] 2024-04-06T09:43:27.849 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=114, tag_tid=17, entry_tid=19402], [object_number=113, tag_tid=17, entry_tid=19401], [object_number=112, tag_tid=17, entry_tid=19400], [object_number=115, tag_tid=17, entry_tid=19399]]], state=connected] 2024-04-06T09:43:27.853 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:43:27.855 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=113, tag_tid=18, entry_tid=1' 2024-04-06T09:43:27.855 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:43:27.855 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:43:27.856 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:43:27.856 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:43:27.856 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:43:27.856 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:43:27.856 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:43:27.856 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:43:27.856 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:43:27.856 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:43:27.856 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:43:27.856 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:43:27.857 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:43:27.857 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:43:27.857 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:43:27.905 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 28 2024-04-06T09:43:27.905 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 28 2024-04-06T09:43:27.905 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:43:27.905 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=113, tag_tid=18, entry_tid=1], [object_number=112, tag_tid=18, entry_tid=0], [object_number=114, tag_tid=17, entry_tid=20202], [object_number=115, tag_tid=17, entry_tid=20199]]], state=connected] 2024-04-06T09:43:27.905 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=114, tag_tid=17, entry_tid=19402], [object_number=113, tag_tid=17, entry_tid=19401], [object_number=112, tag_tid=17, entry_tid=19400], [object_number=115, tag_tid=17, entry_tid=19399]]], state=connected] 2024-04-06T09:43:27.909 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:43:27.911 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=114, tag_tid=17, entry_tid=19402' 2024-04-06T09:43:27.911 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=113, tag_tid=18, entry_tid=1' -a 'object_number=113, tag_tid=18, entry_tid=1' = 'object_number=114, tag_tid=17, entry_tid=19402' 2024-04-06T09:43:27.911 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=114, tag_tid=17, entry_tid=19402' '!=' 'object_number=113, tag_tid=17, entry_tid=19321' 2024-04-06T09:43:27.911 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:43:27.911 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=114, tag_tid=17, entry_tid=19402' = 'object_number=113, tag_tid=17, entry_tid=19321' 2024-04-06T09:43:27.911 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=114, tag_tid=17, entry_tid=19402' 2024-04-06T09:43:27.911 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=113, tag_tid=18, entry_tid=1' 2024-04-06T09:43:27.911 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:43:27.912 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:43:27.912 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=18 2024-04-06T09:43:27.913 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=114, tag_tid=17, entry_tid=19402' 2024-04-06T09:43:27.913 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:43:27.913 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:43:27.914 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=17 2024-04-06T09:43:27.915 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=113, tag_tid=18, entry_tid=1' 2024-04-06T09:43:27.915 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:43:27.915 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:43:27.916 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:43:27.916 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=114, tag_tid=17, entry_tid=19402' 2024-04-06T09:43:27.916 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:43:27.917 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:43:27.918 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=19402 2024-04-06T09:43:27.918 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 18 = 17 -a 1 -le 19402 2024-04-06T09:43:27.918 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:43:27.918 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:43:27.918 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:43:28.120 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:43:28.120 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:43:28.120 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:43:28.120 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:43:28.120 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:43:28.120 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:43:28.120 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:43:28.120 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:43:28.120 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:43:28.120 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:43:28.120 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:43:28.120 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:43:28.121 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:43:28.121 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:43:28.121 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:43:28.121 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:43:28.121 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:43:28.123 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:43:28.123 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:43:28.123 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:43:28.123 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:43:28.123 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:43:28.123 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:43:28.123 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:43:28.123 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:43:28.124 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:43:28.124 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:43:28.124 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:43:28.124 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:43:28.124 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:43:28.124 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:43:28.124 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:43:28.124 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:43:28.125 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:43:28.125 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:43:28.125 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:43:28.125 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:43:28.125 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:43:28.125 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:43:28.126 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:43:28.126 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:43:28.126 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:43:28.126 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:43:28.126 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:43:28.126 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:43:28.126 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:43:28.126 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:43:28.126 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:43:28.126 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:43:28.151 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:43:28.151 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:43:28.151 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:43:28.151 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:43:28.278 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:43:28.279 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:43:28.280 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:43:28.280 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:43:28.280 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:43:28.280 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:43:28.280 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:43:28.280 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:43:28.280 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:43:28.280 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:43:28.281 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:43:28.281 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:43:28.281 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:43:28.282 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:43:28.282 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:43:28.282 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:43:28.329 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 28 2024-04-06T09:43:28.329 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 28 2024-04-06T09:43:28.329 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:43:28.329 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=113, tag_tid=18, entry_tid=1], [object_number=112, tag_tid=18, entry_tid=0], [object_number=114, tag_tid=17, entry_tid=20202], [object_number=115, tag_tid=17, entry_tid=20199]]], state=connected] 2024-04-06T09:43:28.330 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=113, tag_tid=17, entry_tid=19485], [object_number=112, tag_tid=17, entry_tid=19484], [object_number=115, tag_tid=17, entry_tid=19483], [object_number=114, tag_tid=17, entry_tid=19482]]], state=connected] 2024-04-06T09:43:28.333 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:43:28.334 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=113, tag_tid=18, entry_tid=1' 2024-04-06T09:43:28.335 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:43:28.335 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:43:28.335 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:43:28.336 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:43:28.336 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:43:28.336 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:43:28.336 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:43:28.336 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:43:28.336 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:43:28.336 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:43:28.336 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:43:28.336 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:43:28.337 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:43:28.337 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:43:28.337 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:43:28.382 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 28 2024-04-06T09:43:28.383 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 28 2024-04-06T09:43:28.383 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:43:28.383 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=113, tag_tid=18, entry_tid=1], [object_number=112, tag_tid=18, entry_tid=0], [object_number=114, tag_tid=17, entry_tid=20202], [object_number=115, tag_tid=17, entry_tid=20199]]], state=connected] 2024-04-06T09:43:28.383 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=113, tag_tid=17, entry_tid=19485], [object_number=112, tag_tid=17, entry_tid=19484], [object_number=115, tag_tid=17, entry_tid=19483], [object_number=114, tag_tid=17, entry_tid=19482]]], state=connected] 2024-04-06T09:43:28.386 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:43:28.388 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=113, tag_tid=17, entry_tid=19485' 2024-04-06T09:43:28.388 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=113, tag_tid=18, entry_tid=1' -a 'object_number=113, tag_tid=18, entry_tid=1' = 'object_number=113, tag_tid=17, entry_tid=19485' 2024-04-06T09:43:28.388 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=113, tag_tid=17, entry_tid=19485' '!=' 'object_number=114, tag_tid=17, entry_tid=19402' 2024-04-06T09:43:28.388 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:43:28.388 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=113, tag_tid=17, entry_tid=19485' = 'object_number=114, tag_tid=17, entry_tid=19402' 2024-04-06T09:43:28.389 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=113, tag_tid=17, entry_tid=19485' 2024-04-06T09:43:28.389 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=113, tag_tid=18, entry_tid=1' 2024-04-06T09:43:28.389 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:43:28.389 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:43:28.389 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=18 2024-04-06T09:43:28.390 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=113, tag_tid=17, entry_tid=19485' 2024-04-06T09:43:28.390 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:43:28.390 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:43:28.391 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=17 2024-04-06T09:43:28.392 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=113, tag_tid=18, entry_tid=1' 2024-04-06T09:43:28.392 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:43:28.392 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:43:28.393 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:43:28.393 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=113, tag_tid=17, entry_tid=19485' 2024-04-06T09:43:28.393 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:43:28.394 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:43:28.395 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=19485 2024-04-06T09:43:28.395 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 18 = 17 -a 1 -le 19485 2024-04-06T09:43:28.395 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:43:28.395 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:43:28.395 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:43:28.399 INFO:tasks.rbd_mirror_thrash:waiting for cluster2.client.mirror.2 2024-04-06T09:43:28.399 INFO:teuthology.orchestra.run:waiting for 600 2024-04-06T09:43:28.400 INFO:tasks.rbd_mirror_thrash:reviving cluster2.client.mirror.2 2024-04-06T09:43:28.400 INFO:tasks.rbd_mirror.cluster2.client.mirror.2:Restarting daemon 2024-04-06T09:43:28.400 DEBUG:teuthology.orchestra.run.smithi192:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.2 2024-04-06T09:43:28.402 INFO:tasks.rbd_mirror.cluster2.client.mirror.2:Started 2024-04-06T09:43:28.402 INFO:tasks.rbd_mirror_thrash:waiting for 101 secs before thrashing 2024-04-06T09:43:28.596 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:43:28.596 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:43:28.596 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:43:28.596 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:43:28.597 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:43:28.597 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:43:28.597 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:43:28.597 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:43:28.597 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:43:28.597 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:43:28.597 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:43:28.597 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:43:28.597 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:43:28.597 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:43:28.597 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:43:28.598 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:43:28.598 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:43:28.601 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:43:28.601 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:43:28.601 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:43:28.601 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:43:28.601 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:43:28.601 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:43:28.601 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:43:28.601 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:43:28.601 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:43:28.602 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:43:28.602 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:43:28.602 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:43:28.602 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:43:28.602 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:43:28.602 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:43:28.602 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:43:28.602 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:43:28.602 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:43:28.602 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:43:28.602 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:43:28.602 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:43:28.602 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:43:28.602 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:43:28.603 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:43:28.603 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:43:28.603 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:43:28.603 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:43:28.603 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:43:28.603 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:43:28.603 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:43:28.603 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:43:28.603 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:43:28.623 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:43:28.624 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:43:28.624 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:43:28.624 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:43:28.746 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:43:28.748 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:43:28.748 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:43:28.748 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:43:28.748 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:43:28.748 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:43:28.748 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:43:28.748 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:43:28.748 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:43:28.749 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:43:28.749 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:43:28.749 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:43:28.749 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:43:28.751 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:43:28.751 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:43:28.751 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:43:28.795 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 28 2024-04-06T09:43:28.795 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 28 2024-04-06T09:43:28.795 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:43:28.795 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=113, tag_tid=18, entry_tid=1], [object_number=112, tag_tid=18, entry_tid=0], [object_number=114, tag_tid=17, entry_tid=20202], [object_number=115, tag_tid=17, entry_tid=20199]]], state=connected] 2024-04-06T09:43:28.795 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=112, tag_tid=17, entry_tid=19564], [object_number=115, tag_tid=17, entry_tid=19563], [object_number=114, tag_tid=17, entry_tid=19562], [object_number=113, tag_tid=17, entry_tid=19561]]], state=connected] 2024-04-06T09:43:28.798 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:43:28.801 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=113, tag_tid=18, entry_tid=1' 2024-04-06T09:43:28.801 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:43:28.801 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:43:28.801 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:43:28.801 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:43:28.801 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:43:28.801 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:43:28.801 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:43:28.801 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:43:28.801 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:43:28.801 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:43:28.801 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:43:28.801 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:43:28.802 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:43:28.802 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:43:28.802 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:43:28.846 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 28 2024-04-06T09:43:28.846 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 28 2024-04-06T09:43:28.846 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:43:28.846 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=113, tag_tid=18, entry_tid=1], [object_number=112, tag_tid=18, entry_tid=0], [object_number=114, tag_tid=17, entry_tid=20202], [object_number=115, tag_tid=17, entry_tid=20199]]], state=connected] 2024-04-06T09:43:28.846 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=112, tag_tid=17, entry_tid=19564], [object_number=115, tag_tid=17, entry_tid=19563], [object_number=114, tag_tid=17, entry_tid=19562], [object_number=113, tag_tid=17, entry_tid=19561]]], state=connected] 2024-04-06T09:43:28.849 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:43:28.851 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=112, tag_tid=17, entry_tid=19564' 2024-04-06T09:43:28.851 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=113, tag_tid=18, entry_tid=1' -a 'object_number=113, tag_tid=18, entry_tid=1' = 'object_number=112, tag_tid=17, entry_tid=19564' 2024-04-06T09:43:28.851 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=112, tag_tid=17, entry_tid=19564' '!=' 'object_number=113, tag_tid=17, entry_tid=19485' 2024-04-06T09:43:28.851 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:43:28.851 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=112, tag_tid=17, entry_tid=19564' = 'object_number=113, tag_tid=17, entry_tid=19485' 2024-04-06T09:43:28.851 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=112, tag_tid=17, entry_tid=19564' 2024-04-06T09:43:28.851 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=113, tag_tid=18, entry_tid=1' 2024-04-06T09:43:28.851 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:43:28.851 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:43:28.852 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=18 2024-04-06T09:43:28.852 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=112, tag_tid=17, entry_tid=19564' 2024-04-06T09:43:28.853 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:43:28.853 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:43:28.853 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=17 2024-04-06T09:43:28.854 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=113, tag_tid=18, entry_tid=1' 2024-04-06T09:43:28.854 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:43:28.854 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:43:28.855 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:43:28.856 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=112, tag_tid=17, entry_tid=19564' 2024-04-06T09:43:28.856 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:43:28.856 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:43:28.857 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=19564 2024-04-06T09:43:28.857 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 18 = 17 -a 1 -le 19564 2024-04-06T09:43:28.857 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:43:28.857 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:43:28.858 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:43:29.059 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:43:29.059 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:43:29.060 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:43:29.060 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:43:29.060 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:43:29.060 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:43:29.060 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:43:29.060 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:43:29.060 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:43:29.060 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:43:29.060 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:43:29.060 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:43:29.060 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:43:29.060 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:43:29.060 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:43:29.061 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:43:29.061 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:43:29.063 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:43:29.064 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:43:29.064 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:43:29.064 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:43:29.064 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:43:29.064 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:43:29.064 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:43:29.064 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:43:29.064 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:43:29.064 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:43:29.064 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:43:29.064 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:43:29.064 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:43:29.064 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:43:29.064 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:43:29.064 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:43:29.065 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:43:29.065 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:43:29.065 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:43:29.065 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:43:29.065 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:43:29.065 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:43:29.065 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:43:29.065 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:43:29.065 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:43:29.065 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:43:29.065 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:43:29.065 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:43:29.065 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:43:29.065 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:43:29.065 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:43:29.066 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:43:29.088 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:43:29.088 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:43:29.088 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:43:29.088 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:43:29.226 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:43:29.232 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:43:29.233 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:43:29.233 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:43:29.233 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:43:29.233 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:43:29.233 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:43:29.233 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:43:29.233 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:43:29.233 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:43:29.233 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:43:29.233 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:43:29.233 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:43:29.233 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:43:29.233 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:43:29.233 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:43:29.275 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 28 2024-04-06T09:43:29.275 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 28 2024-04-06T09:43:29.275 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:43:29.275 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=113, tag_tid=18, entry_tid=1], [object_number=112, tag_tid=18, entry_tid=0], [object_number=114, tag_tid=17, entry_tid=20202], [object_number=115, tag_tid=17, entry_tid=20199]]], state=connected] 2024-04-06T09:43:29.275 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=112, tag_tid=17, entry_tid=19652], [object_number=115, tag_tid=17, entry_tid=19651], [object_number=114, tag_tid=17, entry_tid=19650], [object_number=113, tag_tid=17, entry_tid=19649]]], state=connected] 2024-04-06T09:43:29.279 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:43:29.281 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=113, tag_tid=18, entry_tid=1' 2024-04-06T09:43:29.281 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:43:29.281 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:43:29.281 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:43:29.281 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:43:29.281 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:43:29.281 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:43:29.281 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:43:29.281 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:43:29.282 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:43:29.282 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:43:29.282 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:43:29.282 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:43:29.282 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:43:29.282 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:43:29.283 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:43:29.327 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 28 2024-04-06T09:43:29.328 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 28 2024-04-06T09:43:29.328 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:43:29.328 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=113, tag_tid=18, entry_tid=1], [object_number=112, tag_tid=18, entry_tid=0], [object_number=114, tag_tid=17, entry_tid=20202], [object_number=115, tag_tid=17, entry_tid=20199]]], state=connected] 2024-04-06T09:43:29.328 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=112, tag_tid=17, entry_tid=19652], [object_number=115, tag_tid=17, entry_tid=19651], [object_number=114, tag_tid=17, entry_tid=19650], [object_number=113, tag_tid=17, entry_tid=19649]]], state=connected] 2024-04-06T09:43:29.331 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:43:29.333 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=112, tag_tid=17, entry_tid=19652' 2024-04-06T09:43:29.333 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=113, tag_tid=18, entry_tid=1' -a 'object_number=113, tag_tid=18, entry_tid=1' = 'object_number=112, tag_tid=17, entry_tid=19652' 2024-04-06T09:43:29.333 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=112, tag_tid=17, entry_tid=19652' '!=' 'object_number=112, tag_tid=17, entry_tid=19564' 2024-04-06T09:43:29.333 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:43:29.333 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=112, tag_tid=17, entry_tid=19652' = 'object_number=112, tag_tid=17, entry_tid=19564' 2024-04-06T09:43:29.333 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=112, tag_tid=17, entry_tid=19652' 2024-04-06T09:43:29.333 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=113, tag_tid=18, entry_tid=1' 2024-04-06T09:43:29.333 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:43:29.333 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:43:29.334 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=18 2024-04-06T09:43:29.334 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=112, tag_tid=17, entry_tid=19652' 2024-04-06T09:43:29.334 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:43:29.335 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:43:29.335 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=17 2024-04-06T09:43:29.336 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=113, tag_tid=18, entry_tid=1' 2024-04-06T09:43:29.336 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:43:29.336 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:43:29.337 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:43:29.337 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=112, tag_tid=17, entry_tid=19652' 2024-04-06T09:43:29.338 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:43:29.338 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:43:29.339 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=19652 2024-04-06T09:43:29.339 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 18 = 17 -a 1 -le 19652 2024-04-06T09:43:29.339 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:43:29.339 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:43:29.339 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:43:29.541 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:43:29.541 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:43:29.541 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:43:29.541 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:43:29.541 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:43:29.541 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:43:29.541 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:43:29.541 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:43:29.541 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:43:29.541 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:43:29.541 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:43:29.541 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:43:29.542 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:43:29.542 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:43:29.542 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:43:29.542 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:43:29.542 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:43:29.544 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:43:29.544 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:43:29.544 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:43:29.544 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:43:29.544 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:43:29.545 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:43:29.545 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:43:29.545 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:43:29.545 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:43:29.545 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:43:29.545 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:43:29.545 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:43:29.545 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:43:29.545 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:43:29.545 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:43:29.545 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:43:29.546 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:43:29.547 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:43:29.547 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:43:29.547 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:43:29.547 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:43:29.547 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:43:29.547 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:43:29.547 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:43:29.547 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:43:29.547 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:43:29.547 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:43:29.547 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:43:29.547 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:43:29.547 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:43:29.547 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:43:29.548 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:43:29.569 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:43:29.570 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:43:29.570 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:43:29.570 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:43:29.697 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:43:29.697 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:43:29.697 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:43:29.697 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:43:29.697 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:43:29.698 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:43:29.698 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:43:29.698 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:43:29.698 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:43:29.698 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:43:29.698 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:43:29.698 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:43:29.699 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:43:29.699 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:43:29.700 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:43:29.700 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:43:29.745 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 28 2024-04-06T09:43:29.745 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 28 2024-04-06T09:43:29.745 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:43:29.745 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=113, tag_tid=18, entry_tid=1], [object_number=112, tag_tid=18, entry_tid=0], [object_number=114, tag_tid=17, entry_tid=20202], [object_number=115, tag_tid=17, entry_tid=20199]]], state=connected] 2024-04-06T09:43:29.745 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=113, tag_tid=17, entry_tid=19729], [object_number=112, tag_tid=17, entry_tid=19728], [object_number=115, tag_tid=17, entry_tid=19727], [object_number=114, tag_tid=17, entry_tid=19726]]], state=connected] 2024-04-06T09:43:29.748 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:43:29.750 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=113, tag_tid=18, entry_tid=1' 2024-04-06T09:43:29.751 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:43:29.751 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:43:29.751 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:43:29.751 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:43:29.751 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:43:29.751 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:43:29.751 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:43:29.751 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:43:29.751 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:43:29.751 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:43:29.751 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:43:29.751 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:43:29.752 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:43:29.752 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:43:29.752 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:43:29.795 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 28 2024-04-06T09:43:29.799 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 28 2024-04-06T09:43:29.799 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:43:29.799 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=113, tag_tid=18, entry_tid=1], [object_number=112, tag_tid=18, entry_tid=0], [object_number=114, tag_tid=17, entry_tid=20202], [object_number=115, tag_tid=17, entry_tid=20199]]], state=connected] 2024-04-06T09:43:29.799 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=113, tag_tid=17, entry_tid=19729], [object_number=112, tag_tid=17, entry_tid=19728], [object_number=115, tag_tid=17, entry_tid=19727], [object_number=114, tag_tid=17, entry_tid=19726]]], state=connected] 2024-04-06T09:43:29.801 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:43:29.801 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=113, tag_tid=17, entry_tid=19729' 2024-04-06T09:43:29.801 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=113, tag_tid=18, entry_tid=1' -a 'object_number=113, tag_tid=18, entry_tid=1' = 'object_number=113, tag_tid=17, entry_tid=19729' 2024-04-06T09:43:29.801 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=113, tag_tid=17, entry_tid=19729' '!=' 'object_number=112, tag_tid=17, entry_tid=19652' 2024-04-06T09:43:29.801 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:43:29.801 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=113, tag_tid=17, entry_tid=19729' = 'object_number=112, tag_tid=17, entry_tid=19652' 2024-04-06T09:43:29.801 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=113, tag_tid=17, entry_tid=19729' 2024-04-06T09:43:29.801 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=113, tag_tid=18, entry_tid=1' 2024-04-06T09:43:29.801 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:43:29.801 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:43:29.802 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=18 2024-04-06T09:43:29.802 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=113, tag_tid=17, entry_tid=19729' 2024-04-06T09:43:29.802 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:43:29.802 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:43:29.803 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=17 2024-04-06T09:43:29.803 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=113, tag_tid=18, entry_tid=1' 2024-04-06T09:43:29.803 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:43:29.803 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:43:29.804 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:43:29.805 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=113, tag_tid=17, entry_tid=19729' 2024-04-06T09:43:29.805 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:43:29.805 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:43:29.806 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=19729 2024-04-06T09:43:29.806 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 18 = 17 -a 1 -le 19729 2024-04-06T09:43:29.806 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:43:29.806 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:43:29.807 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:43:30.008 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:43:30.008 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:43:30.008 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:43:30.008 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:43:30.008 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:43:30.008 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:43:30.008 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:43:30.008 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:43:30.008 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:43:30.008 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:43:30.009 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:43:30.009 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:43:30.009 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:43:30.009 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:43:30.009 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:43:30.009 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:43:30.009 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:43:30.011 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:43:30.011 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:43:30.011 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:43:30.011 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:43:30.011 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:43:30.011 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:43:30.011 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:43:30.012 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:43:30.012 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:43:30.012 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:43:30.012 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:43:30.012 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:43:30.012 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:43:30.012 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:43:30.012 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:43:30.012 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:43:30.013 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:43:30.013 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:43:30.013 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:43:30.014 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:43:30.014 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:43:30.014 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:43:30.014 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:43:30.014 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:43:30.014 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:43:30.014 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:43:30.014 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:43:30.014 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:43:30.014 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:43:30.014 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:43:30.014 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:43:30.014 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:43:30.037 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:43:30.037 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:43:30.037 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:43:30.037 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:43:30.162 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:43:30.163 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:43:30.163 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:43:30.163 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:43:30.163 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:43:30.163 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:43:30.163 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:43:30.163 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:43:30.164 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:43:30.164 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:43:30.164 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:43:30.164 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:43:30.164 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:43:30.166 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:43:30.167 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:43:30.167 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:43:30.212 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 28 2024-04-06T09:43:30.213 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 28 2024-04-06T09:43:30.213 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:43:30.213 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=113, tag_tid=18, entry_tid=1], [object_number=112, tag_tid=18, entry_tid=0], [object_number=114, tag_tid=17, entry_tid=20202], [object_number=115, tag_tid=17, entry_tid=20199]]], state=connected] 2024-04-06T09:43:30.213 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=114, tag_tid=17, entry_tid=19810], [object_number=113, tag_tid=17, entry_tid=19809], [object_number=112, tag_tid=17, entry_tid=19808], [object_number=115, tag_tid=17, entry_tid=19807]]], state=connected] 2024-04-06T09:43:30.216 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:43:30.217 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=113, tag_tid=18, entry_tid=1' 2024-04-06T09:43:30.219 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:43:30.219 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:43:30.219 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:43:30.219 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:43:30.219 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:43:30.219 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:43:30.219 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:43:30.219 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:43:30.219 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:43:30.219 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:43:30.219 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:43:30.220 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:43:30.220 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:43:30.220 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:43:30.220 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:43:30.269 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 28 2024-04-06T09:43:30.269 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 28 2024-04-06T09:43:30.269 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:43:30.269 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=113, tag_tid=18, entry_tid=1], [object_number=112, tag_tid=18, entry_tid=0], [object_number=114, tag_tid=17, entry_tid=20202], [object_number=115, tag_tid=17, entry_tid=20199]]], state=connected] 2024-04-06T09:43:30.269 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=114, tag_tid=17, entry_tid=19810], [object_number=113, tag_tid=17, entry_tid=19809], [object_number=112, tag_tid=17, entry_tid=19808], [object_number=115, tag_tid=17, entry_tid=19807]]], state=connected] 2024-04-06T09:43:30.272 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:43:30.274 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=114, tag_tid=17, entry_tid=19810' 2024-04-06T09:43:30.274 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=113, tag_tid=18, entry_tid=1' -a 'object_number=113, tag_tid=18, entry_tid=1' = 'object_number=114, tag_tid=17, entry_tid=19810' 2024-04-06T09:43:30.274 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=114, tag_tid=17, entry_tid=19810' '!=' 'object_number=113, tag_tid=17, entry_tid=19729' 2024-04-06T09:43:30.274 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:43:30.275 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=114, tag_tid=17, entry_tid=19810' = 'object_number=113, tag_tid=17, entry_tid=19729' 2024-04-06T09:43:30.275 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=114, tag_tid=17, entry_tid=19810' 2024-04-06T09:43:30.275 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=113, tag_tid=18, entry_tid=1' 2024-04-06T09:43:30.275 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:43:30.275 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:43:30.275 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=18 2024-04-06T09:43:30.276 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=114, tag_tid=17, entry_tid=19810' 2024-04-06T09:43:30.276 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:43:30.276 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:43:30.277 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=17 2024-04-06T09:43:30.278 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=113, tag_tid=18, entry_tid=1' 2024-04-06T09:43:30.278 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:43:30.278 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:43:30.279 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:43:30.280 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=114, tag_tid=17, entry_tid=19810' 2024-04-06T09:43:30.280 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:43:30.280 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:43:30.281 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=19810 2024-04-06T09:43:30.281 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 18 = 17 -a 1 -le 19810 2024-04-06T09:43:30.281 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:43:30.281 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:43:30.281 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:43:30.483 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:43:30.483 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:43:30.483 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:43:30.483 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:43:30.483 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:43:30.483 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:43:30.483 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:43:30.484 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:43:30.484 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:43:30.484 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:43:30.484 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:43:30.484 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:43:30.484 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:43:30.484 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:43:30.484 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:43:30.484 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:43:30.485 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:43:30.487 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:43:30.487 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:43:30.487 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:43:30.487 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:43:30.487 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:43:30.488 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:43:30.488 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:43:30.488 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:43:30.488 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:43:30.488 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:43:30.488 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:43:30.488 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:43:30.488 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:43:30.488 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:43:30.488 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:43:30.488 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:43:30.489 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:43:30.489 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:43:30.489 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:43:30.489 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:43:30.489 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:43:30.489 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:43:30.490 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:43:30.490 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:43:30.490 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:43:30.490 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:43:30.490 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:43:30.490 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:43:30.490 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:43:30.490 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:43:30.490 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:43:30.490 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:43:30.516 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:43:30.516 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:43:30.516 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:43:30.516 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:43:30.646 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:43:30.647 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:43:30.647 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:43:30.647 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:43:30.647 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:43:30.647 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:43:30.647 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:43:30.647 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:43:30.647 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:43:30.648 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:43:30.648 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:43:30.648 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:43:30.648 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:43:30.649 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:43:30.650 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:43:30.650 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:43:30.702 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 28 2024-04-06T09:43:30.702 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 28 2024-04-06T09:43:30.702 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:43:30.702 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=113, tag_tid=18, entry_tid=1], [object_number=112, tag_tid=18, entry_tid=0], [object_number=114, tag_tid=17, entry_tid=20202], [object_number=115, tag_tid=17, entry_tid=20199]]], state=connected] 2024-04-06T09:43:30.702 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=114, tag_tid=17, entry_tid=19890], [object_number=113, tag_tid=17, entry_tid=19889], [object_number=112, tag_tid=17, entry_tid=19888], [object_number=115, tag_tid=17, entry_tid=19887]]], state=connected] 2024-04-06T09:43:30.706 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:43:30.707 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=113, tag_tid=18, entry_tid=1' 2024-04-06T09:43:30.708 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:43:30.708 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:43:30.708 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:43:30.708 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:43:30.709 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:43:30.709 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:43:30.709 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:43:30.709 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:43:30.709 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:43:30.709 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:43:30.709 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:43:30.709 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:43:30.709 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:43:30.710 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:43:30.710 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:43:30.757 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 28 2024-04-06T09:43:30.757 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 28 2024-04-06T09:43:30.757 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:43:30.757 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=113, tag_tid=18, entry_tid=1], [object_number=112, tag_tid=18, entry_tid=0], [object_number=114, tag_tid=17, entry_tid=20202], [object_number=115, tag_tid=17, entry_tid=20199]]], state=connected] 2024-04-06T09:43:30.757 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=114, tag_tid=17, entry_tid=19890], [object_number=113, tag_tid=17, entry_tid=19889], [object_number=112, tag_tid=17, entry_tid=19888], [object_number=115, tag_tid=17, entry_tid=19887]]], state=connected] 2024-04-06T09:43:30.760 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:43:30.762 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=114, tag_tid=17, entry_tid=19890' 2024-04-06T09:43:30.762 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=113, tag_tid=18, entry_tid=1' -a 'object_number=113, tag_tid=18, entry_tid=1' = 'object_number=114, tag_tid=17, entry_tid=19890' 2024-04-06T09:43:30.762 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=114, tag_tid=17, entry_tid=19890' '!=' 'object_number=114, tag_tid=17, entry_tid=19810' 2024-04-06T09:43:30.762 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:43:30.762 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=114, tag_tid=17, entry_tid=19890' = 'object_number=114, tag_tid=17, entry_tid=19810' 2024-04-06T09:43:30.762 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=114, tag_tid=17, entry_tid=19890' 2024-04-06T09:43:30.763 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=113, tag_tid=18, entry_tid=1' 2024-04-06T09:43:30.763 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:43:30.763 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:43:30.763 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=18 2024-04-06T09:43:30.764 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=114, tag_tid=17, entry_tid=19890' 2024-04-06T09:43:30.764 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:43:30.764 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:43:30.765 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=17 2024-04-06T09:43:30.766 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=113, tag_tid=18, entry_tid=1' 2024-04-06T09:43:30.766 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:43:30.766 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:43:30.767 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:43:30.767 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=114, tag_tid=17, entry_tid=19890' 2024-04-06T09:43:30.768 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:43:30.768 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:43:30.769 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=19890 2024-04-06T09:43:30.769 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 18 = 17 -a 1 -le 19890 2024-04-06T09:43:30.769 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:43:30.769 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:43:30.769 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:43:30.971 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:43:30.971 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:43:30.971 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:43:30.971 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:43:30.971 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:43:30.971 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:43:30.971 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:43:30.971 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:43:30.971 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:43:30.971 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:43:30.971 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:43:30.972 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:43:30.972 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:43:30.972 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:43:30.972 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:43:30.972 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:43:30.973 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:43:30.975 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:43:30.975 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:43:30.975 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:43:30.975 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:43:30.975 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:43:30.975 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:43:30.975 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:43:30.976 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:43:30.976 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:43:30.976 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:43:30.976 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:43:30.976 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:43:30.976 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:43:30.976 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:43:30.976 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:43:30.976 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:43:30.977 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:43:30.977 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:43:30.978 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:43:30.978 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:43:30.978 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:43:30.978 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:43:30.978 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:43:30.978 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:43:30.978 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:43:30.978 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:43:30.978 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:43:30.978 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:43:30.978 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:43:30.978 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:43:30.978 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:43:30.979 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:43:31.002 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:43:31.002 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:43:31.002 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:43:31.002 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:43:31.139 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:43:31.140 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:43:31.140 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:43:31.140 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:43:31.140 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:43:31.140 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:43:31.140 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:43:31.140 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:43:31.141 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:43:31.141 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:43:31.141 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:43:31.141 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:43:31.141 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:43:31.143 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:43:31.143 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:43:31.143 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:43:31.195 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 28 2024-04-06T09:43:31.195 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 28 2024-04-06T09:43:31.195 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:43:31.195 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=113, tag_tid=18, entry_tid=1], [object_number=112, tag_tid=18, entry_tid=0], [object_number=114, tag_tid=17, entry_tid=20202], [object_number=115, tag_tid=17, entry_tid=20199]]], state=connected] 2024-04-06T09:43:31.195 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=113, tag_tid=17, entry_tid=19977], [object_number=112, tag_tid=17, entry_tid=19976], [object_number=115, tag_tid=17, entry_tid=19975], [object_number=114, tag_tid=17, entry_tid=19974]]], state=connected] 2024-04-06T09:43:31.198 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:43:31.200 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=113, tag_tid=18, entry_tid=1' 2024-04-06T09:43:31.201 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:43:31.201 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:43:31.201 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:43:31.201 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:43:31.201 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:43:31.201 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:43:31.201 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:43:31.202 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:43:31.202 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:43:31.202 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:43:31.202 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:43:31.202 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:43:31.202 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:43:31.203 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:43:31.203 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:43:31.251 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 28 2024-04-06T09:43:31.251 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 28 2024-04-06T09:43:31.251 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:43:31.251 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=113, tag_tid=18, entry_tid=1], [object_number=112, tag_tid=18, entry_tid=0], [object_number=114, tag_tid=17, entry_tid=20202], [object_number=115, tag_tid=17, entry_tid=20199]]], state=connected] 2024-04-06T09:43:31.251 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=113, tag_tid=17, entry_tid=19977], [object_number=112, tag_tid=17, entry_tid=19976], [object_number=115, tag_tid=17, entry_tid=19975], [object_number=114, tag_tid=17, entry_tid=19974]]], state=connected] 2024-04-06T09:43:31.254 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:43:31.256 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=113, tag_tid=17, entry_tid=19977' 2024-04-06T09:43:31.256 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=113, tag_tid=18, entry_tid=1' -a 'object_number=113, tag_tid=18, entry_tid=1' = 'object_number=113, tag_tid=17, entry_tid=19977' 2024-04-06T09:43:31.256 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=113, tag_tid=17, entry_tid=19977' '!=' 'object_number=114, tag_tid=17, entry_tid=19890' 2024-04-06T09:43:31.256 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:43:31.256 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=113, tag_tid=17, entry_tid=19977' = 'object_number=114, tag_tid=17, entry_tid=19890' 2024-04-06T09:43:31.257 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=113, tag_tid=17, entry_tid=19977' 2024-04-06T09:43:31.257 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=113, tag_tid=18, entry_tid=1' 2024-04-06T09:43:31.257 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:43:31.257 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:43:31.257 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=18 2024-04-06T09:43:31.258 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=113, tag_tid=17, entry_tid=19977' 2024-04-06T09:43:31.258 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:43:31.258 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:43:31.259 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=17 2024-04-06T09:43:31.260 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=113, tag_tid=18, entry_tid=1' 2024-04-06T09:43:31.260 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:43:31.260 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:43:31.261 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:43:31.262 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=113, tag_tid=17, entry_tid=19977' 2024-04-06T09:43:31.262 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:43:31.262 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:43:31.263 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=19977 2024-04-06T09:43:31.263 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 18 = 17 -a 1 -le 19977 2024-04-06T09:43:31.263 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:43:31.263 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:43:31.263 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:43:31.465 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:43:31.465 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:43:31.465 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:43:31.465 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:43:31.465 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:43:31.465 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:43:31.465 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:43:31.465 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:43:31.465 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:43:31.466 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:43:31.466 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:43:31.466 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:43:31.466 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:43:31.466 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:43:31.466 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:43:31.466 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:43:31.469 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:43:31.469 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:43:31.469 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:43:31.469 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:43:31.469 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:43:31.469 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:43:31.469 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:43:31.469 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:43:31.469 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:43:31.469 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:43:31.469 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:43:31.470 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:43:31.470 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:43:31.470 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:43:31.470 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:43:31.470 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:43:31.470 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:43:31.470 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:43:31.470 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:43:31.470 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:43:31.470 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:43:31.470 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:43:31.470 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:43:31.470 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:43:31.471 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:43:31.471 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:43:31.471 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:43:31.471 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:43:31.471 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:43:31.471 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:43:31.471 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:43:31.471 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:43:31.471 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:43:31.493 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:43:31.493 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:43:31.493 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:43:31.493 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:43:31.624 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:43:31.625 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:43:31.625 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:43:31.626 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:43:31.626 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:43:31.626 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:43:31.626 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:43:31.626 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:43:31.626 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:43:31.626 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:43:31.626 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:43:31.627 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:43:31.627 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:43:31.628 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:43:31.629 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:43:31.629 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:43:31.681 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 28 2024-04-06T09:43:31.681 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 28 2024-04-06T09:43:31.681 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:43:31.681 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=113, tag_tid=18, entry_tid=1], [object_number=112, tag_tid=18, entry_tid=0], [object_number=114, tag_tid=17, entry_tid=20202], [object_number=115, tag_tid=17, entry_tid=20199]]], state=connected] 2024-04-06T09:43:31.681 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=113, tag_tid=17, entry_tid=20065], [object_number=112, tag_tid=17, entry_tid=20064], [object_number=115, tag_tid=17, entry_tid=20063], [object_number=114, tag_tid=17, entry_tid=20062]]], state=connected] 2024-04-06T09:43:31.685 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:43:31.686 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=113, tag_tid=18, entry_tid=1' 2024-04-06T09:43:31.687 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:43:31.687 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:43:31.687 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:43:31.687 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:43:31.688 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:43:31.688 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:43:31.688 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:43:31.688 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:43:31.688 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:43:31.688 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:43:31.688 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:43:31.688 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:43:31.689 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:43:31.689 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:43:31.689 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:43:31.736 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 28 2024-04-06T09:43:31.736 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 28 2024-04-06T09:43:31.736 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:43:31.736 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=113, tag_tid=18, entry_tid=1], [object_number=112, tag_tid=18, entry_tid=0], [object_number=114, tag_tid=17, entry_tid=20202], [object_number=115, tag_tid=17, entry_tid=20199]]], state=connected] 2024-04-06T09:43:31.736 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=113, tag_tid=17, entry_tid=20065], [object_number=112, tag_tid=17, entry_tid=20064], [object_number=115, tag_tid=17, entry_tid=20063], [object_number=114, tag_tid=17, entry_tid=20062]]], state=connected] 2024-04-06T09:43:31.739 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:43:31.742 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=113, tag_tid=17, entry_tid=20065' 2024-04-06T09:43:31.742 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=113, tag_tid=18, entry_tid=1' -a 'object_number=113, tag_tid=18, entry_tid=1' = 'object_number=113, tag_tid=17, entry_tid=20065' 2024-04-06T09:43:31.742 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=113, tag_tid=17, entry_tid=20065' '!=' 'object_number=113, tag_tid=17, entry_tid=19977' 2024-04-06T09:43:31.742 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:43:31.742 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=113, tag_tid=17, entry_tid=20065' = 'object_number=113, tag_tid=17, entry_tid=19977' 2024-04-06T09:43:31.742 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=113, tag_tid=17, entry_tid=20065' 2024-04-06T09:43:31.742 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=113, tag_tid=18, entry_tid=1' 2024-04-06T09:43:31.742 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:43:31.742 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:43:31.743 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=18 2024-04-06T09:43:31.744 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=113, tag_tid=17, entry_tid=20065' 2024-04-06T09:43:31.744 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:43:31.744 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:43:31.745 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=17 2024-04-06T09:43:31.745 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=113, tag_tid=18, entry_tid=1' 2024-04-06T09:43:31.745 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:43:31.745 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:43:31.746 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:43:31.747 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=113, tag_tid=17, entry_tid=20065' 2024-04-06T09:43:31.747 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:43:31.747 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:43:31.748 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=20065 2024-04-06T09:43:31.748 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 18 = 17 -a 1 -le 20065 2024-04-06T09:43:31.748 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:43:31.749 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:43:31.749 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:43:31.951 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:43:31.951 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:43:31.951 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:43:31.951 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:43:31.951 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:43:31.951 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:43:31.951 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:43:31.952 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:43:31.952 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:43:31.952 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:43:31.952 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:43:31.952 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:43:31.952 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:43:31.952 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:43:31.952 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:43:31.955 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:43:31.955 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:43:31.955 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:43:31.956 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:43:31.956 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:43:31.956 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:43:31.956 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:43:31.956 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:43:31.956 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:43:31.956 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:43:31.956 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:43:31.956 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:43:31.956 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:43:31.956 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:43:31.957 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:43:31.957 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:43:31.957 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:43:31.957 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:43:31.957 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:43:31.957 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:43:31.957 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:43:31.957 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:43:31.957 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:43:31.957 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:43:31.957 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:43:31.957 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:43:31.957 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:43:31.958 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:43:31.958 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:43:31.958 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:43:31.958 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:43:31.958 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:43:31.958 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:43:31.958 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:43:31.980 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:43:31.981 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:43:31.981 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:43:31.981 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:43:32.114 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:43:32.114 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:43:32.115 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:43:32.115 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:43:32.115 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:43:32.115 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:43:32.115 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:43:32.115 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:43:32.115 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:43:32.115 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:43:32.116 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:43:32.116 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:43:32.116 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:43:32.119 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:43:32.119 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:43:32.119 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:43:32.170 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 28 2024-04-06T09:43:32.170 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 28 2024-04-06T09:43:32.170 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:43:32.170 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=113, tag_tid=18, entry_tid=1], [object_number=112, tag_tid=18, entry_tid=0], [object_number=114, tag_tid=17, entry_tid=20202], [object_number=115, tag_tid=17, entry_tid=20199]]], state=connected] 2024-04-06T09:43:32.170 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=114, tag_tid=17, entry_tid=20154], [object_number=113, tag_tid=17, entry_tid=20153], [object_number=112, tag_tid=17, entry_tid=20152], [object_number=115, tag_tid=17, entry_tid=20151]]], state=connected] 2024-04-06T09:43:32.173 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:43:32.175 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=113, tag_tid=18, entry_tid=1' 2024-04-06T09:43:32.176 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:43:32.176 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:43:32.176 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:43:32.176 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:43:32.176 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:43:32.176 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:43:32.176 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:43:32.176 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:43:32.177 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:43:32.177 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:43:32.177 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:43:32.177 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:43:32.177 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:43:32.178 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:43:32.178 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:43:32.225 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 28 2024-04-06T09:43:32.225 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 28 2024-04-06T09:43:32.225 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:43:32.225 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=113, tag_tid=18, entry_tid=1], [object_number=112, tag_tid=18, entry_tid=0], [object_number=114, tag_tid=17, entry_tid=20202], [object_number=115, tag_tid=17, entry_tid=20199]]], state=connected] 2024-04-06T09:43:32.225 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=114, tag_tid=17, entry_tid=20154], [object_number=113, tag_tid=17, entry_tid=20153], [object_number=112, tag_tid=17, entry_tid=20152], [object_number=115, tag_tid=17, entry_tid=20151]]], state=connected] 2024-04-06T09:43:32.228 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:43:32.230 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=114, tag_tid=17, entry_tid=20154' 2024-04-06T09:43:32.230 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=113, tag_tid=18, entry_tid=1' -a 'object_number=113, tag_tid=18, entry_tid=1' = 'object_number=114, tag_tid=17, entry_tid=20154' 2024-04-06T09:43:32.230 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=114, tag_tid=17, entry_tid=20154' '!=' 'object_number=113, tag_tid=17, entry_tid=20065' 2024-04-06T09:43:32.230 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:43:32.230 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=114, tag_tid=17, entry_tid=20154' = 'object_number=113, tag_tid=17, entry_tid=20065' 2024-04-06T09:43:32.230 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=114, tag_tid=17, entry_tid=20154' 2024-04-06T09:43:32.231 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=113, tag_tid=18, entry_tid=1' 2024-04-06T09:43:32.231 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:43:32.231 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:43:32.231 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=18 2024-04-06T09:43:32.232 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=114, tag_tid=17, entry_tid=20154' 2024-04-06T09:43:32.232 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:43:32.232 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:43:32.233 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=17 2024-04-06T09:43:32.234 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=113, tag_tid=18, entry_tid=1' 2024-04-06T09:43:32.234 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:43:32.234 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:43:32.235 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:43:32.236 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=114, tag_tid=17, entry_tid=20154' 2024-04-06T09:43:32.236 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:43:32.236 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:43:32.237 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=20154 2024-04-06T09:43:32.237 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 18 = 17 -a 1 -le 20154 2024-04-06T09:43:32.237 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:43:32.237 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:43:32.238 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:43:32.439 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:43:32.439 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:43:32.439 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:43:32.439 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:43:32.439 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:43:32.439 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:43:32.439 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:43:32.439 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:43:32.439 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:43:32.439 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:43:32.440 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:43:32.440 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:43:32.440 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:43:32.440 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:43:32.440 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:43:32.440 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:43:32.441 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:43:32.444 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:43:32.444 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:43:32.444 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:43:32.444 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:43:32.445 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:43:32.445 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:43:32.445 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:43:32.445 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:43:32.445 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:43:32.445 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:43:32.445 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:43:32.445 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:43:32.445 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:43:32.445 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:43:32.445 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:43:32.445 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:43:32.445 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:43:32.445 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:43:32.445 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:43:32.446 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:43:32.446 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:43:32.446 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:43:32.446 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:43:32.446 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:43:32.446 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:43:32.446 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:43:32.446 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:43:32.446 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:43:32.446 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:43:32.446 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:43:32.446 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:43:32.446 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:43:32.472 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:43:32.472 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:43:32.472 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:43:32.472 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:43:32.600 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:43:32.601 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:43:32.601 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:43:32.601 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:43:32.601 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:43:32.601 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:43:32.601 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:43:32.601 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:43:32.601 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:43:32.601 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:43:32.601 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:43:32.602 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:43:32.602 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:43:32.604 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:43:32.605 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:43:32.605 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:43:32.854 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 28 2024-04-06T09:43:32.855 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 28 2024-04-06T09:43:32.855 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:43:32.855 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=113, tag_tid=18, entry_tid=1], [object_number=112, tag_tid=18, entry_tid=0], [object_number=114, tag_tid=17, entry_tid=20202], [object_number=115, tag_tid=17, entry_tid=20199]]], state=connected] 2024-04-06T09:43:32.855 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=114, tag_tid=17, entry_tid=20202], [object_number=113, tag_tid=17, entry_tid=20201], [object_number=112, tag_tid=17, entry_tid=20200], [object_number=115, tag_tid=17, entry_tid=20199]]], state=connected] 2024-04-06T09:43:32.860 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:43:32.862 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=113, tag_tid=18, entry_tid=1' 2024-04-06T09:43:32.863 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:43:32.863 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:43:32.864 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:43:32.864 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:43:32.864 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:43:32.864 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:43:32.864 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:43:32.864 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:43:32.864 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:43:32.864 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:43:32.864 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:43:32.865 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:43:32.866 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:43:32.866 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:43:32.866 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:43:32.921 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 28 2024-04-06T09:43:32.921 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 28 2024-04-06T09:43:32.921 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:43:32.921 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=113, tag_tid=18, entry_tid=1], [object_number=112, tag_tid=18, entry_tid=0], [object_number=114, tag_tid=17, entry_tid=20202], [object_number=115, tag_tid=17, entry_tid=20199]]], state=connected] 2024-04-06T09:43:32.921 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=114, tag_tid=17, entry_tid=20202], [object_number=113, tag_tid=17, entry_tid=20201], [object_number=112, tag_tid=17, entry_tid=20200], [object_number=115, tag_tid=17, entry_tid=20199]]], state=connected] 2024-04-06T09:43:32.925 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:43:32.927 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=114, tag_tid=17, entry_tid=20202' 2024-04-06T09:43:32.927 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=113, tag_tid=18, entry_tid=1' -a 'object_number=113, tag_tid=18, entry_tid=1' = 'object_number=114, tag_tid=17, entry_tid=20202' 2024-04-06T09:43:32.927 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=114, tag_tid=17, entry_tid=20202' '!=' 'object_number=114, tag_tid=17, entry_tid=20154' 2024-04-06T09:43:32.927 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:43:32.927 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=114, tag_tid=17, entry_tid=20202' = 'object_number=114, tag_tid=17, entry_tid=20154' 2024-04-06T09:43:32.927 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=114, tag_tid=17, entry_tid=20202' 2024-04-06T09:43:32.927 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=113, tag_tid=18, entry_tid=1' 2024-04-06T09:43:32.927 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:43:32.927 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:43:32.928 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=18 2024-04-06T09:43:32.929 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=114, tag_tid=17, entry_tid=20202' 2024-04-06T09:43:32.929 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:43:32.929 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:43:32.930 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=17 2024-04-06T09:43:32.931 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=113, tag_tid=18, entry_tid=1' 2024-04-06T09:43:32.931 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:43:32.931 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:43:32.932 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:43:32.932 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=114, tag_tid=17, entry_tid=20202' 2024-04-06T09:43:32.933 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:43:32.933 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:43:32.934 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=20202 2024-04-06T09:43:32.934 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 18 = 17 -a 1 -le 20202 2024-04-06T09:43:32.934 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:43:32.934 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:43:32.934 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:43:33.136 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:43:33.136 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:43:33.136 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:43:33.136 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:43:33.136 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:43:33.136 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:43:33.136 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:43:33.136 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:43:33.136 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:43:33.136 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:43:33.137 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:43:33.137 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:43:33.137 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:43:33.137 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:43:33.137 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:43:33.137 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:43:33.137 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:43:33.141 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:43:33.141 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:43:33.141 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:43:33.141 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:43:33.141 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:43:33.141 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:43:33.141 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:43:33.141 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:43:33.141 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:43:33.141 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:43:33.141 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:43:33.141 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:43:33.141 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:43:33.142 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:43:33.142 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:43:33.142 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:43:33.142 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:43:33.142 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:43:33.142 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:43:33.142 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:43:33.142 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:43:33.142 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:43:33.142 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:43:33.142 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:43:33.142 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:43:33.142 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:43:33.142 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:43:33.143 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:43:33.143 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:43:33.143 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:43:33.143 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:43:33.143 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:43:33.166 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:43:33.166 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:43:33.166 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:43:33.166 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:43:33.287 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:43:33.289 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:43:33.289 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:43:33.289 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:43:33.289 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:43:33.289 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:43:33.289 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:43:33.289 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:43:33.289 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:43:33.289 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:43:33.290 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:43:33.290 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:43:33.290 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:43:33.291 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:43:33.291 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:43:33.291 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:43:33.357 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 28 2024-04-06T09:43:33.357 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 28 2024-04-06T09:43:33.357 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:43:33.357 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=113, tag_tid=18, entry_tid=1], [object_number=112, tag_tid=18, entry_tid=0], [object_number=114, tag_tid=17, entry_tid=20202], [object_number=115, tag_tid=17, entry_tid=20199]]], state=connected] 2024-04-06T09:43:33.357 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=114, tag_tid=17, entry_tid=20202], [object_number=113, tag_tid=17, entry_tid=20201], [object_number=112, tag_tid=17, entry_tid=20200], [object_number=115, tag_tid=17, entry_tid=20199]]], state=connected] 2024-04-06T09:43:33.357 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:43:33.357 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=113, tag_tid=18, entry_tid=1' 2024-04-06T09:43:33.357 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:43:33.357 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:43:33.357 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:43:33.358 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:43:33.358 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:43:33.358 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:43:33.358 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:43:33.358 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:43:33.358 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:43:33.358 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:43:33.358 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:43:33.358 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:43:33.358 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:43:33.358 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:43:33.358 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:43:33.400 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 28 2024-04-06T09:43:33.400 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 28 2024-04-06T09:43:33.400 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:43:33.400 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=113, tag_tid=18, entry_tid=1], [object_number=112, tag_tid=18, entry_tid=0], [object_number=114, tag_tid=17, entry_tid=20202], [object_number=115, tag_tid=17, entry_tid=20199]]], state=connected] 2024-04-06T09:43:33.400 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=114, tag_tid=17, entry_tid=20202], [object_number=113, tag_tid=17, entry_tid=20201], [object_number=112, tag_tid=17, entry_tid=20200], [object_number=115, tag_tid=17, entry_tid=20199]]], state=connected] 2024-04-06T09:43:33.403 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:43:33.404 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=114, tag_tid=17, entry_tid=20202' 2024-04-06T09:43:33.404 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=113, tag_tid=18, entry_tid=1' -a 'object_number=113, tag_tid=18, entry_tid=1' = 'object_number=114, tag_tid=17, entry_tid=20202' 2024-04-06T09:43:33.404 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=114, tag_tid=17, entry_tid=20202' '!=' 'object_number=114, tag_tid=17, entry_tid=20202' 2024-04-06T09:43:33.404 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:43:33.405 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.4 2024-04-06T09:43:33.805 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:43:33.805 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:43:33.805 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:43:33.805 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:43:33.806 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:43:33.806 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:43:33.806 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:43:33.806 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:43:33.806 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:43:33.806 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:43:33.806 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:43:33.806 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:43:33.806 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:43:33.806 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:43:33.806 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:43:33.807 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:43:33.807 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:43:33.810 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:43:33.811 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:43:33.811 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:43:33.811 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:43:33.811 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:43:33.811 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:43:33.811 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:43:33.811 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:43:33.811 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:43:33.811 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:43:33.811 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:43:33.811 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:43:33.811 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:43:33.811 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:43:33.812 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:43:33.812 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:43:33.812 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:43:33.812 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:43:33.812 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:43:33.812 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:43:33.812 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:43:33.812 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:43:33.812 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:43:33.812 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:43:33.812 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:43:33.812 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:43:33.812 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:43:33.812 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:43:33.813 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:43:33.813 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:43:33.813 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:43:33.813 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:43:33.835 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:43:33.836 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:43:33.836 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:43:33.836 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:43:33.954 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:43:33.955 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:43:33.955 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:43:33.956 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:43:33.956 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:43:33.956 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:43:33.956 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:43:33.956 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:43:33.956 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:43:33.956 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:43:33.956 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:43:33.957 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:43:33.957 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:43:33.959 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:43:33.959 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:43:33.959 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:43:34.010 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 28 2024-04-06T09:43:34.010 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 28 2024-04-06T09:43:34.010 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:43:34.011 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=113, tag_tid=18, entry_tid=1], [object_number=112, tag_tid=18, entry_tid=0], [object_number=114, tag_tid=17, entry_tid=20202], [object_number=115, tag_tid=17, entry_tid=20199]]], state=connected] 2024-04-06T09:43:34.011 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=114, tag_tid=17, entry_tid=20202], [object_number=113, tag_tid=17, entry_tid=20201], [object_number=112, tag_tid=17, entry_tid=20200], [object_number=115, tag_tid=17, entry_tid=20199]]], state=connected] 2024-04-06T09:43:34.014 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:43:34.015 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=113, tag_tid=18, entry_tid=1' 2024-04-06T09:43:34.016 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:43:34.016 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:43:34.017 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:43:34.017 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:43:34.017 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:43:34.017 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:43:34.017 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:43:34.017 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:43:34.017 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:43:34.017 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:43:34.017 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:43:34.017 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:43:34.017 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:43:34.018 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:43:34.018 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:43:34.065 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 28 2024-04-06T09:43:34.065 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 28 2024-04-06T09:43:34.065 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:43:34.065 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=113, tag_tid=18, entry_tid=1], [object_number=112, tag_tid=18, entry_tid=0], [object_number=114, tag_tid=17, entry_tid=20202], [object_number=115, tag_tid=17, entry_tid=20199]]], state=connected] 2024-04-06T09:43:34.065 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=114, tag_tid=17, entry_tid=20202], [object_number=113, tag_tid=17, entry_tid=20201], [object_number=112, tag_tid=17, entry_tid=20200], [object_number=115, tag_tid=17, entry_tid=20199]]], state=connected] 2024-04-06T09:43:34.069 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:43:34.070 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=114, tag_tid=17, entry_tid=20202' 2024-04-06T09:43:34.070 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=113, tag_tid=18, entry_tid=1' -a 'object_number=113, tag_tid=18, entry_tid=1' = 'object_number=114, tag_tid=17, entry_tid=20202' 2024-04-06T09:43:34.070 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=114, tag_tid=17, entry_tid=20202' '!=' 'object_number=114, tag_tid=17, entry_tid=20202' 2024-04-06T09:43:34.070 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:43:34.070 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.8 2024-04-06T09:43:34.872 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:43:34.872 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:43:34.872 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:43:34.872 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:43:34.872 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:43:34.872 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:43:34.873 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:43:34.873 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:43:34.873 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:43:34.873 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:43:34.873 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:43:34.873 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:43:34.873 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:43:34.873 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:43:34.873 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:43:34.873 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:43:34.874 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:43:34.875 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:43:34.875 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:43:34.876 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:43:34.876 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:43:34.876 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:43:34.876 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:43:34.876 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:43:34.876 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:43:34.876 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:43:34.876 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:43:34.876 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:43:34.876 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:43:34.876 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:43:34.876 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:43:34.878 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:43:34.878 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:43:34.878 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:43:34.878 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:43:34.878 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:43:34.878 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:43:34.879 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:43:34.879 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:43:34.879 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:43:34.879 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:43:34.879 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:43:34.879 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:43:34.879 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:43:34.879 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:43:34.879 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:43:34.879 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:43:34.879 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:43:34.879 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:43:34.902 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:43:34.903 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:43:34.903 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:43:34.903 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:43:35.021 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:43:35.022 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:43:35.022 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:43:35.022 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:43:35.022 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:43:35.022 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:43:35.022 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:43:35.022 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:43:35.022 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:43:35.022 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:43:35.023 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:43:35.023 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:43:35.023 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:43:35.025 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:43:35.026 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:43:35.026 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:43:35.081 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 28 2024-04-06T09:43:35.081 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 28 2024-04-06T09:43:35.082 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:43:35.082 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=113, tag_tid=18, entry_tid=1], [object_number=112, tag_tid=18, entry_tid=0], [object_number=114, tag_tid=17, entry_tid=20202], [object_number=115, tag_tid=17, entry_tid=20199]]], state=connected] 2024-04-06T09:43:35.082 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=114, tag_tid=17, entry_tid=20202], [object_number=113, tag_tid=17, entry_tid=20201], [object_number=112, tag_tid=17, entry_tid=20200], [object_number=115, tag_tid=17, entry_tid=20199]]], state=connected] 2024-04-06T09:43:35.085 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:43:35.086 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=113, tag_tid=18, entry_tid=1' 2024-04-06T09:43:35.087 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:43:35.088 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:43:35.088 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:43:35.088 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:43:35.088 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:43:35.088 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:43:35.088 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:43:35.088 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:43:35.088 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:43:35.088 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:43:35.088 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:43:35.088 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:43:35.089 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:43:35.089 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:43:35.089 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:43:35.136 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 28 2024-04-06T09:43:35.136 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 28 2024-04-06T09:43:35.136 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:43:35.136 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=113, tag_tid=18, entry_tid=1], [object_number=112, tag_tid=18, entry_tid=0], [object_number=114, tag_tid=17, entry_tid=20202], [object_number=115, tag_tid=17, entry_tid=20199]]], state=connected] 2024-04-06T09:43:35.136 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=114, tag_tid=17, entry_tid=20202], [object_number=113, tag_tid=17, entry_tid=20201], [object_number=112, tag_tid=17, entry_tid=20200], [object_number=115, tag_tid=17, entry_tid=20199]]], state=connected] 2024-04-06T09:43:35.140 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:43:35.142 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=114, tag_tid=17, entry_tid=20202' 2024-04-06T09:43:35.142 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=113, tag_tid=18, entry_tid=1' -a 'object_number=113, tag_tid=18, entry_tid=1' = 'object_number=114, tag_tid=17, entry_tid=20202' 2024-04-06T09:43:35.142 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=114, tag_tid=17, entry_tid=20202' '!=' 'object_number=114, tag_tid=17, entry_tid=20202' 2024-04-06T09:43:35.142 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:43:35.142 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 1.6 2024-04-06T09:43:36.744 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:43:36.744 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:43:36.744 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:43:36.745 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:43:36.745 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:43:36.745 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:43:36.745 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:43:36.745 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:43:36.745 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:43:36.745 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:43:36.745 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:43:36.745 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:43:36.745 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:43:36.745 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:43:36.745 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:43:36.746 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:43:36.746 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:43:36.748 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:43:36.748 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:43:36.748 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:43:36.748 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:43:36.748 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:43:36.748 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:43:36.748 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:43:36.748 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:43:36.748 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:43:36.748 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:43:36.749 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:43:36.749 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:43:36.749 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:43:36.749 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:43:36.749 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:43:36.749 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:43:36.750 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:43:36.750 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:43:36.751 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:43:36.751 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:43:36.751 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:43:36.751 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:43:36.751 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:43:36.751 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:43:36.751 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:43:36.751 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:43:36.751 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:43:36.751 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:43:36.751 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:43:36.751 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:43:36.751 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:43:36.751 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:43:36.774 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:43:36.775 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:43:36.775 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:43:36.775 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:43:36.894 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:43:36.895 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:43:36.895 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:43:36.895 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:43:36.896 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:43:36.896 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:43:36.896 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:43:36.896 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:43:36.896 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:43:36.896 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:43:36.896 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:43:36.896 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:43:36.896 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:43:36.898 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:43:36.899 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:43:36.899 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:43:36.952 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 28 2024-04-06T09:43:36.952 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 28 2024-04-06T09:43:36.952 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:43:36.952 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=113, tag_tid=18, entry_tid=1], [object_number=112, tag_tid=18, entry_tid=0], [object_number=114, tag_tid=17, entry_tid=20202], [object_number=115, tag_tid=17, entry_tid=20199]]], state=connected] 2024-04-06T09:43:36.952 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=114, tag_tid=17, entry_tid=20202], [object_number=113, tag_tid=17, entry_tid=20201], [object_number=112, tag_tid=17, entry_tid=20200], [object_number=115, tag_tid=17, entry_tid=20199]]], state=connected] 2024-04-06T09:43:36.956 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:43:36.957 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=113, tag_tid=18, entry_tid=1' 2024-04-06T09:43:36.958 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:43:36.958 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:43:36.958 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:43:36.958 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:43:36.958 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:43:36.958 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:43:36.959 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:43:36.959 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:43:36.959 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:43:36.959 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:43:36.959 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:43:36.959 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:43:36.960 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:43:36.960 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:43:36.960 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:43:37.009 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 28 2024-04-06T09:43:37.009 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 28 2024-04-06T09:43:37.009 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:43:37.009 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=113, tag_tid=18, entry_tid=1], [object_number=112, tag_tid=18, entry_tid=0], [object_number=114, tag_tid=17, entry_tid=20202], [object_number=115, tag_tid=17, entry_tid=20199]]], state=connected] 2024-04-06T09:43:37.009 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=114, tag_tid=17, entry_tid=20202], [object_number=113, tag_tid=17, entry_tid=20201], [object_number=112, tag_tid=17, entry_tid=20200], [object_number=115, tag_tid=17, entry_tid=20199]]], state=connected] 2024-04-06T09:43:37.013 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:43:37.014 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=114, tag_tid=17, entry_tid=20202' 2024-04-06T09:43:37.015 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=113, tag_tid=18, entry_tid=1' -a 'object_number=113, tag_tid=18, entry_tid=1' = 'object_number=114, tag_tid=17, entry_tid=20202' 2024-04-06T09:43:37.015 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=114, tag_tid=17, entry_tid=20202' '!=' 'object_number=114, tag_tid=17, entry_tid=20202' 2024-04-06T09:43:37.015 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:43:37.015 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 2 2024-04-06T09:43:39.016 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:43:39.017 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:43:39.017 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:43:39.017 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:43:39.017 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:43:39.017 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:43:39.017 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:43:39.017 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:43:39.017 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:43:39.017 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:43:39.017 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:43:39.017 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:43:39.018 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:43:39.018 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:43:39.018 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:43:39.018 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:43:39.018 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:43:39.023 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:43:39.023 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:43:39.023 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:43:39.023 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:43:39.023 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:43:39.023 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:43:39.023 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:43:39.023 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:43:39.023 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:43:39.023 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:43:39.024 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:43:39.024 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:43:39.024 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:43:39.024 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:43:39.024 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:43:39.024 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:43:39.024 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:43:39.024 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:43:39.024 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:43:39.024 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:43:39.024 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:43:39.024 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:43:39.024 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:43:39.024 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:43:39.025 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:43:39.025 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:43:39.025 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:43:39.025 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:43:39.025 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:43:39.025 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:43:39.025 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:43:39.025 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:43:39.047 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:43:39.048 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:43:39.048 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:43:39.048 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:43:39.173 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:43:39.175 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:43:39.175 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:43:39.175 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:43:39.175 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:43:39.175 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:43:39.175 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:43:39.175 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:43:39.175 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:43:39.176 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:43:39.176 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:43:39.176 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:43:39.176 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:43:39.177 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:43:39.177 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:43:39.177 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:43:39.232 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 28 2024-04-06T09:43:39.233 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 28 2024-04-06T09:43:39.233 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:43:39.233 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=113, tag_tid=18, entry_tid=1], [object_number=112, tag_tid=18, entry_tid=0], [object_number=114, tag_tid=17, entry_tid=20202], [object_number=115, tag_tid=17, entry_tid=20199]]], state=connected] 2024-04-06T09:43:39.233 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=113, tag_tid=18, entry_tid=1], [object_number=112, tag_tid=18, entry_tid=0], [object_number=114, tag_tid=17, entry_tid=20202], [object_number=115, tag_tid=17, entry_tid=20199]]], state=connected] 2024-04-06T09:43:39.236 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:43:39.237 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=113, tag_tid=18, entry_tid=1' 2024-04-06T09:43:39.239 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:43:39.239 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:43:39.239 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:43:39.239 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:43:39.239 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:43:39.239 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:43:39.239 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:43:39.239 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:43:39.239 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:43:39.239 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:43:39.239 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:43:39.240 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:43:39.240 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:43:39.240 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:43:39.240 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:43:39.287 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 28 2024-04-06T09:43:39.287 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 28 2024-04-06T09:43:39.287 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:43:39.287 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=113, tag_tid=18, entry_tid=1], [object_number=112, tag_tid=18, entry_tid=0], [object_number=114, tag_tid=17, entry_tid=20202], [object_number=115, tag_tid=17, entry_tid=20199]]], state=connected] 2024-04-06T09:43:39.288 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=113, tag_tid=18, entry_tid=1], [object_number=112, tag_tid=18, entry_tid=0], [object_number=114, tag_tid=17, entry_tid=20202], [object_number=115, tag_tid=17, entry_tid=20199]]], state=connected] 2024-04-06T09:43:39.291 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:43:39.292 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=113, tag_tid=18, entry_tid=1' 2024-04-06T09:43:39.293 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=113, tag_tid=18, entry_tid=1' -a 'object_number=113, tag_tid=18, entry_tid=1' = 'object_number=113, tag_tid=18, entry_tid=1' 2024-04-06T09:43:39.293 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:43:39.293 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_snap_present cluster1 mirror test snap6 2024-04-06T09:43:39.293 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:43:39.293 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:43:39.293 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:43:39.293 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local snap_name=snap6 2024-04-06T09:43:39.293 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local s 2024-04-06T09:43:39.293 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-04-06T09:43:39.293 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 1 2024-04-06T09:43:40.293 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 info mirror/test@snap6 2024-04-06T09:43:40.363 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:rbd image 'test': 2024-04-06T09:43:40.363 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: size 512 MiB in 128 objects 2024-04-06T09:43:40.363 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: order 22 (4 MiB objects) 2024-04-06T09:43:40.364 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: snapshot_count: 11 2024-04-06T09:43:40.364 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: id: 10a56da0b207 2024-04-06T09:43:40.364 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: block_name_prefix: rbd_data.10a56da0b207 2024-04-06T09:43:40.364 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: format: 2 2024-04-06T09:43:40.364 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: features: layering, exclusive-lock, journaling 2024-04-06T09:43:40.364 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: op_features: 2024-04-06T09:43:40.364 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: flags: 2024-04-06T09:43:40.364 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: create_timestamp: Sat Apr 6 09:31:59 2024 2024-04-06T09:43:40.364 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: access_timestamp: Sat Apr 6 09:41:24 2024 2024-04-06T09:43:40.364 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: modify_timestamp: Sat Apr 6 09:43:09 2024 2024-04-06T09:43:40.364 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: protected: False 2024-04-06T09:43:40.364 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: journal: 10a56da0b207 2024-04-06T09:43:40.364 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: mirroring state: enabled 2024-04-06T09:43:40.364 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: mirroring mode: journal 2024-04-06T09:43:40.365 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: mirroring global id: 8ad5dedc-b3e0-425d-8993-b3e2231fc5ca 2024-04-06T09:43:40.365 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: mirroring primary: false 2024-04-06T09:43:40.368 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:43:40.368 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n snap5-clean ']' 2024-04-06T09:43:40.368 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ compare_image_snaps mirror test snap5-clean 2024-04-06T09:43:40.368 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:43:40.369 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:43:40.369 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local snap_name=snap5-clean 2024-04-06T09:43:40.369 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local ret=0 2024-04-06T09:43:40.369 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local rmt_export=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.export 2024-04-06T09:43:40.369 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loc_export=/tmp/tmp.BhW6xKmROU/cluster1-mirror-test.export 2024-04-06T09:43:40.369 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rm -f /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.export /tmp/tmp.BhW6xKmROU/cluster1-mirror-test.export 2024-04-06T09:43:40.369 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster2 -p mirror export test@snap5-clean /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.export 2024-04-06T09:43:40.970 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-04-06T09:43:40.983 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 -p mirror export test@snap5-clean /tmp/tmp.BhW6xKmROU/cluster1-mirror-test.export 2024-04-06T09:43:42.391 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-04-06T09:43:42.417 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmp /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.export /tmp/tmp.BhW6xKmROU/cluster1-mirror-test.export 2024-04-06T09:43:42.701 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rm -f /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.export /tmp/tmp.BhW6xKmROU/cluster1-mirror-test.export 2024-04-06T09:43:42.835 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:43:42.835 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ compare_image_snaps mirror test snap6 2024-04-06T09:43:42.835 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:43:42.835 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:43:42.835 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local snap_name=snap6 2024-04-06T09:43:42.835 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local ret=0 2024-04-06T09:43:42.835 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local rmt_export=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.export 2024-04-06T09:43:42.835 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loc_export=/tmp/tmp.BhW6xKmROU/cluster1-mirror-test.export 2024-04-06T09:43:42.835 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rm -f /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.export /tmp/tmp.BhW6xKmROU/cluster1-mirror-test.export 2024-04-06T09:43:42.836 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster2 -p mirror export test@snap6 /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.export 2024-04-06T09:43:43.452 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-04-06T09:43:43.463 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 -p mirror export test@snap6 /tmp/tmp.BhW6xKmROU/cluster1-mirror-test.export 2024-04-06T09:43:45.106 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-04-06T09:43:45.128 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmp /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.export /tmp/tmp.BhW6xKmROU/cluster1-mirror-test.export 2024-04-06T09:43:45.411 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rm -f /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.export /tmp/tmp.BhW6xKmROU/cluster1-mirror-test.export 2024-04-06T09:43:45.547 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:43:45.547 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ clean_snap_name=snap6-clean 2024-04-06T09:43:45.547 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ create_snap cluster2 mirror test snap6-clean 2024-04-06T09:43:45.547 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-04-06T09:43:45.547 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:43:45.548 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:43:45.548 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local snap_name=snap6-clean 2024-04-06T09:43:45.548 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap6-clean --debug-rbd=20 --debug-journaler=20 2024-04-06T09:43:46.642 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in `seq 1 10` 2024-04-06T09:43:46.642 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ stress_write_image cluster2 mirror test 2024-04-06T09:43:46.643 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-04-06T09:43:46.643 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:43:46.643 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:43:46.643 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-04-06T09:43:46.645 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local duration=14 2024-04-06T09:43:46.645 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set +e 2024-04-06T09:43:46.645 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ timeout 14s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror test --debug-rbd=20 --debug-journaler=20 2024-04-06T09:43:48.888 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:100 IOs, 37498880 bytes 2024-04-06T09:43:49.798 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:200 IOs, 60086272 bytes 2024-04-06T09:43:49.995 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:300 IOs, 60709888 bytes 2024-04-06T09:43:54.953 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:400 IOs, 191441920 bytes 2024-04-06T09:43:57.458 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:500 IOs, 254694400 bytes 2024-04-06T09:43:59.374 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:600 IOs, 299652096 bytes 2024-04-06T09:44:00.216 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:700 IOs, 310516736 bytes 2024-04-06T09:44:00.655 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ error_code=124 2024-04-06T09:44:00.655 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set -e 2024-04-06T09:44:00.655 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 124 -eq 124 ']' 2024-04-06T09:44:00.655 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:44:00.655 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+replaying 2024-04-06T09:44:00.655 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:44:00.655 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:44:00.655 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:44:00.655 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+replaying 2024-04-06T09:44:00.655 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local description_pattern= 2024-04-06T09:44:00.655 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local service_pattern= 2024-04-06T09:44:00.655 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:44:00.656 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 1 2024-04-06T09:44:01.656 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test_status_in_pool_dir cluster1 mirror test up+replaying '' '' 2024-04-06T09:44:01.656 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:44:01.656 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:44:01.656 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:44:01.656 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+replaying 2024-04-06T09:44:01.656 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local description_pattern= 2024-04-06T09:44:01.656 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local service_pattern= 2024-04-06T09:44:01.656 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local status 2024-04-06T09:44:01.657 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ CEPH_ARGS= 2024-04-06T09:44:01.657 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster1 mirror image status mirror/test 2024-04-06T09:44:01.750 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ status='test: 2024-04-06T09:44:01.750 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: global_id: 8ad5dedc-b3e0-425d-8993-b3e2231fc5ca 2024-04-06T09:44:01.750 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: state: up+replaying 2024-04-06T09:44:01.750 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: description: replaying, {"bytes_per_second":1186008.4,"entries_behind_primary":1,"entries_per_second":73.5,"non_primary_position":{"entry_tid":1,"object_number":113,"tag_tid":18},"primary_position":{"entry_tid":1,"object_number":113,"tag_tid":19},"seconds_until_synced":0} 2024-04-06T09:44:01.750 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: service: mirror.0 on smithi192 2024-04-06T09:44:01.750 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: last_update: 2024-04-06 09:43:57 2024-04-06T09:44:01.750 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: peer_sites: 2024-04-06T09:44:01.750 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: name: cluster2 2024-04-06T09:44:01.750 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: state: up+stopped 2024-04-06T09:44:01.750 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: description: local image is primary 2024-04-06T09:44:01.750 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: last_update: 2024-04-06 09:43:44' 2024-04-06T09:44:01.750 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^ state: .*up+replaying' 2024-04-06T09:44:01.751 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: state: up+replaying 2024-04-06T09:44:01.751 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^ description: .*' 2024-04-06T09:44:01.752 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: description: replaying, {"bytes_per_second":1186008.4,"entries_behind_primary":1,"entries_per_second":73.5,"non_primary_position":{"entry_tid":1,"object_number":113,"tag_tid":18},"primary_position":{"entry_tid":1,"object_number":113,"tag_tid":19},"seconds_until_synced":0} 2024-04-06T09:44:01.752 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n '' ']' 2024-04-06T09:44:01.752 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ echo up+replaying 2024-04-06T09:44:01.752 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^up+' 2024-04-06T09:44:01.753 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:up+replaying 2024-04-06T09:44:01.753 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep 'service: *mirror..* on ' 2024-04-06T09:44:01.754 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: service: mirror.0 on smithi192 2024-04-06T09:44:01.754 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local last_update 2024-04-06T09:44:01.754 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-04-06T09:44:01.756 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_update='2024-04-06 09:43:57' 2024-04-06T09:44:01.756 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+replaying '2024-04-06 09:43:57' 2024-04-06T09:44:01.757 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:44:01.757 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:44:01.757 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:44:01.757 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+replaying 2024-04-06T09:44:01.757 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'prev_last_update=2024-04-06 09:43:57' 2024-04-06T09:44:01.757 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local status 2024-04-06T09:44:01.757 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ CEPH_ARGS= 2024-04-06T09:44:01.757 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-04-06T09:44:01.847 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ status='OKOKOK142094268mirror.1smithi19218.2.2-1059-g4d05f4cafalseOK42584261mirror.0smithi19218.2.2-1059-g4d05f4catrueOK43194322mirror.3smithi19218.2.2-1059-g4d05f4cafalseOK43814384mirror.2smithi19218.2.2-1059-g4d05f4cafalseOKtest8ad5dedc-b3e0-425d-8993-b3e2231fc5caup+replayingreplaying, {"bytes_per_second":1186008.4,"entries_behind_primary":1,"entries_per_second":73.5,"non_primary_position":{"entry_tid":1,"object_number":113,"tag_tid":18},"primary_position":{"entry_tid":1,"object_number":113,"tag_tid":19},"seconds_until_synced":0}42584261mirror.0smithi1922024-04-06 09:43:57cluster2c5e3bdc3-c286-4289-8fd0-6b8628b92376up+stoppedlocal image is primary2024-04-06 09:43:44' 2024-04-06T09:44:01.848 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local last_update state 2024-04-06T09:44:01.848 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2024-04-06T09:44:01.851 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_update='2024-04-06 09:43:57' 2024-04-06T09:44:01.851 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2024-04-06T09:44:01.855 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ state=up+replaying 2024-04-06T09:44:01.855 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ echo up+replaying 2024-04-06T09:44:01.855 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep up+replaying 2024-04-06T09:44:01.856 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:up+replaying 2024-04-06T09:44:01.857 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:44:01.857 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:44:01.857 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ snap_name=snap7 2024-04-06T09:44:01.857 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ create_snap cluster2 mirror test snap7 2024-04-06T09:44:01.857 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-04-06T09:44:01.857 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:44:01.857 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:44:01.858 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local snap_name=snap7 2024-04-06T09:44:01.858 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap7 --debug-rbd=20 --debug-journaler=20 2024-04-06T09:44:03.392 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.0 with skip (0.4170913272180351) > weight (0.25) 2024-04-06T09:44:03.393 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.1 with skip (0.6694549003462449) > weight (0.25) 2024-04-06T09:44:03.393 INFO:tasks.rbd_mirror_thrash:kill cluster1.client.mirror.2 2024-04-06T09:44:03.393 INFO:tasks.rbd_mirror.cluster1.client.mirror.2:Sent signal 15 2024-04-06T09:44:03.393 INFO:tasks.rbd_mirror_thrash:waiting for 0 secs before reviving daemons 2024-04-06T09:44:03.393 INFO:tasks.rbd_mirror_thrash:waiting for cluster1.client.mirror.2 2024-04-06T09:44:03.394 INFO:teuthology.orchestra.run:waiting for 600 2024-04-06T09:44:03.394 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi192.stderr:2024-04-06T09:44:03.401+0000 7f1197e74640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.2 (PID: 75802) UID: 1000 2024-04-06T09:44:09.495 INFO:tasks.rbd_mirror_thrash:reviving cluster1.client.mirror.2 2024-04-06T09:44:09.495 INFO:tasks.rbd_mirror.cluster1.client.mirror.2:Restarting daemon 2024-04-06T09:44:09.496 DEBUG:teuthology.orchestra.run.smithi192:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.2 2024-04-06T09:44:09.498 INFO:tasks.rbd_mirror.cluster1.client.mirror.2:Started 2024-04-06T09:44:09.498 INFO:tasks.rbd_mirror_thrash:waiting for 116 secs before thrashing 2024-04-06T09:44:18.573 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:44:18.573+0000 7f2e84d66640 -1 reset not still connected to 0x560d27073930 2024-04-06T09:44:18.575 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:44:18.573+0000 7f2e84d66640 -1 reset not still connected to 0x560d27073ba0 2024-04-06T09:44:18.575 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:44:18.573+0000 7f2e84d66640 -1 reset not still connected to 0x560d27073d40 2024-04-06T09:44:31.607 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_image_replay_started cluster1 mirror test 2024-04-06T09:44:31.607 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:44:31.607 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:44:31.607 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:44:31.608 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_image_replay_state cluster1 mirror test started 2024-04-06T09:44:31.608 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:44:31.608 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:44:31.608 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:44:31.608 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state=started 2024-04-06T09:44:31.608 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local s 2024-04-06T09:44:31.608 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:44:31.608 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 1 2024-04-06T09:44:32.609 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test_image_replay_state cluster1 mirror test started 2024-04-06T09:44:32.609 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:44:32.609 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:44:32.609 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:44:32.609 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local test_state=started 2024-04-06T09:44:32.609 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local status_result 2024-04-06T09:44:32.609 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local current_state=stopped 2024-04-06T09:44:32.611 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2024-04-06T09:44:32.612 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -i state 2024-04-06T09:44:32.612 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_instance=cluster1 2024-04-06T09:44:32.612 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ shift 2024-04-06T09:44:32.612 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:44:32.612 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance=cluster1 2024-04-06T09:44:32.613 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local loop_instance 2024-04-06T09:44:32.613 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:44:32.613 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sleep 0 2024-04-06T09:44:32.613 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:44:32.618 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ seq 0 3 2024-04-06T09:44:32.618 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:44:32.618 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2024-04-06T09:44:32.618 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1:0 2024-04-06T09:44:32.618 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ shift 2024-04-06T09:44:32.619 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:44:32.619 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:44:32.619 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:44:32.619 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=cluster 2024-04-06T09:44:32.619 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:44:32.619 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:44:32.620 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:44:32.620 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:44:32.620 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval cluster=cluster1 2024-04-06T09:44:32.620 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ cluster=cluster1 2024-04-06T09:44:32.621 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:44:32.621 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:44:32.621 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:44:32.621 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local local_cluster=cluster1:0 2024-04-06T09:44:32.621 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local cluster=cluster1 2024-04-06T09:44:32.621 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local instance 2024-04-06T09:44:32.621 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:44:32.622 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local val=cluster1:0 2024-04-06T09:44:32.622 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local cluster_var_name=local_cluster 2024-04-06T09:44:32.622 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local instance_var_name=instance 2024-04-06T09:44:32.623 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ cluster=cluster1 2024-04-06T09:44:32.624 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:44:32.624 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ '[' 0 = cluster1:0 ']' 2024-04-06T09:44:32.624 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ eval local_cluster=cluster1 2024-04-06T09:44:32.624 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++++ local_cluster=cluster1 2024-04-06T09:44:32.624 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ eval instance=0 2024-04-06T09:44:32.624 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++++ instance=0 2024-04-06T09:44:32.624 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:44:32.637 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:44:32.637 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:44:32.638 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:44:32.638 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2024-04-06T09:44:32.756 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ return 0 2024-04-06T09:44:32.756 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ status_result=' "state": "Replaying"' 2024-04-06T09:44:32.757 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ echo ' "state": "Replaying"' 2024-04-06T09:44:32.757 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep -i Replaying 2024-04-06T09:44:32.758 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: "state": "Replaying" 2024-04-06T09:44:32.759 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ current_state=started 2024-04-06T09:44:32.760 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test started = started 2024-04-06T09:44:32.760 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:44:32.760 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror test 2024-04-06T09:44:32.760 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local local_cluster=cluster1 2024-04-06T09:44:32.760 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-04-06T09:44:32.760 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:44:32.760 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:44:32.760 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' journal = journal ']' 2024-04-06T09:44:32.760 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_journal_replay_complete cluster1 cluster2 mirror test 2024-04-06T09:44:32.760 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local local_cluster=cluster1 2024-04-06T09:44:32.760 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-04-06T09:44:32.760 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:44:32.760 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:44:32.760 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local s master_pos mirror_pos last_mirror_pos 2024-04-06T09:44:32.761 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local master_tag master_entry mirror_tag mirror_entry 2024-04-06T09:44:32.761 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:44:32.761 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:44:32.761 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:44:32.962 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:44:32.962 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:44:32.962 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:44:32.962 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:44:32.962 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:44:32.962 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:44:32.963 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:44:32.963 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:44:32.963 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:44:32.963 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:44:32.963 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:44:32.963 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:44:32.963 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:44:32.964 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:44:32.964 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:44:32.969 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:44:32.970 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:44:32.970 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:44:32.970 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:44:32.970 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:44:32.970 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:44:32.970 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:44:32.970 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:44:32.971 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:44:32.971 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:44:32.971 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:44:32.971 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:44:32.971 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:44:32.971 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:44:32.971 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:44:32.971 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:44:32.972 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:44:32.972 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:44:32.972 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:44:32.972 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:44:32.972 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:44:32.972 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:44:32.972 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:44:32.973 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:44:32.973 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:44:32.973 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:44:32.973 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:44:32.973 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:44:32.973 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:44:32.973 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:44:32.974 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:44:32.974 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:44:32.974 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:44:32.974 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:44:32.991 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:44:32.991 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:44:32.991 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:44:32.992 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:44:33.133 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:44:33.134 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:44:33.134 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:44:33.135 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:44:33.135 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:44:33.135 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:44:33.135 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:44:33.135 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:44:33.135 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:44:33.135 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:44:33.135 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:44:33.136 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:44:33.136 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:44:33.138 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:44:33.139 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:44:33.139 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:44:33.197 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 30 2024-04-06T09:44:33.197 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 33 2024-04-06T09:44:33.197 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:44:33.197 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=133, tag_tid=21, entry_tid=1], [object_number=132, tag_tid=21, entry_tid=0], [object_number=135, tag_tid=20, entry_tid=19939], [object_number=134, tag_tid=20, entry_tid=19938]]], state=connected] 2024-04-06T09:44:33.197 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=121, tag_tid=20, entry_tid=7281], [object_number=120, tag_tid=20, entry_tid=7280], [object_number=123, tag_tid=20, entry_tid=7279], [object_number=122, tag_tid=20, entry_tid=7278]]], state=connected] 2024-04-06T09:44:33.201 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:44:33.202 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=133, tag_tid=21, entry_tid=1' 2024-04-06T09:44:33.203 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:44:33.203 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:44:33.203 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:44:33.203 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:44:33.203 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:44:33.203 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:44:33.203 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:44:33.204 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:44:33.204 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:44:33.204 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:44:33.204 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:44:33.204 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:44:33.204 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:44:33.205 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:44:33.205 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:44:33.252 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 30 2024-04-06T09:44:33.252 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 33 2024-04-06T09:44:33.252 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:44:33.252 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=133, tag_tid=21, entry_tid=1], [object_number=132, tag_tid=21, entry_tid=0], [object_number=135, tag_tid=20, entry_tid=19939], [object_number=134, tag_tid=20, entry_tid=19938]]], state=connected] 2024-04-06T09:44:33.253 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=121, tag_tid=20, entry_tid=7281], [object_number=120, tag_tid=20, entry_tid=7280], [object_number=123, tag_tid=20, entry_tid=7279], [object_number=122, tag_tid=20, entry_tid=7278]]], state=connected] 2024-04-06T09:44:33.256 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:44:33.258 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=121, tag_tid=20, entry_tid=7281' 2024-04-06T09:44:33.258 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=133, tag_tid=21, entry_tid=1' -a 'object_number=133, tag_tid=21, entry_tid=1' = 'object_number=121, tag_tid=20, entry_tid=7281' 2024-04-06T09:44:33.258 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=121, tag_tid=20, entry_tid=7281' '!=' '' 2024-04-06T09:44:33.258 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:44:33.258 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=121, tag_tid=20, entry_tid=7281' = '' 2024-04-06T09:44:33.258 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=121, tag_tid=20, entry_tid=7281' 2024-04-06T09:44:33.258 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=133, tag_tid=21, entry_tid=1' 2024-04-06T09:44:33.258 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:44:33.258 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:44:33.259 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=21 2024-04-06T09:44:33.260 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=121, tag_tid=20, entry_tid=7281' 2024-04-06T09:44:33.260 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:44:33.260 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:44:33.260 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=20 2024-04-06T09:44:33.261 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=133, tag_tid=21, entry_tid=1' 2024-04-06T09:44:33.261 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:44:33.261 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:44:33.262 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:44:33.263 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=121, tag_tid=20, entry_tid=7281' 2024-04-06T09:44:33.263 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:44:33.263 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:44:33.264 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=7281 2024-04-06T09:44:33.264 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 21 = 20 -a 1 -le 7281 2024-04-06T09:44:33.264 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:44:33.264 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:44:33.265 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:44:33.466 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:44:33.466 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:44:33.466 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:44:33.466 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:44:33.466 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:44:33.466 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:44:33.467 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:44:33.467 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:44:33.467 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:44:33.467 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:44:33.467 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:44:33.467 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:44:33.467 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:44:33.467 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:44:33.467 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:44:33.468 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:44:33.468 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:44:33.470 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:44:33.471 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:44:33.471 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:44:33.471 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:44:33.471 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:44:33.471 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:44:33.471 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:44:33.471 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:44:33.471 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:44:33.471 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:44:33.471 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:44:33.471 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:44:33.471 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:44:33.471 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:44:33.471 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:44:33.472 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:44:33.472 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:44:33.472 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:44:33.472 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:44:33.472 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:44:33.472 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:44:33.472 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:44:33.472 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:44:33.472 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:44:33.472 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:44:33.472 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:44:33.472 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:44:33.472 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:44:33.472 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:44:33.472 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:44:33.472 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:44:33.473 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:44:33.496 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:44:33.496 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:44:33.497 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:44:33.497 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:44:33.630 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:44:33.631 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:44:33.631 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:44:33.631 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:44:33.631 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:44:33.631 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:44:33.631 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:44:33.631 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:44:33.632 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:44:33.632 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:44:33.632 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:44:33.632 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:44:33.632 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:44:33.634 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:44:33.634 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:44:33.634 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:44:33.686 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 30 2024-04-06T09:44:33.686 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 33 2024-04-06T09:44:33.686 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:44:33.686 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=133, tag_tid=21, entry_tid=1], [object_number=132, tag_tid=21, entry_tid=0], [object_number=135, tag_tid=20, entry_tid=19939], [object_number=134, tag_tid=20, entry_tid=19938]]], state=connected] 2024-04-06T09:44:33.687 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=121, tag_tid=20, entry_tid=7369], [object_number=120, tag_tid=20, entry_tid=7368], [object_number=123, tag_tid=20, entry_tid=7367], [object_number=122, tag_tid=20, entry_tid=7366]]], state=connected] 2024-04-06T09:44:33.690 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:44:33.691 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=133, tag_tid=21, entry_tid=1' 2024-04-06T09:44:33.692 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:44:33.692 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:44:33.692 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:44:33.693 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:44:33.693 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:44:33.693 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:44:33.693 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:44:33.693 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:44:33.693 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:44:33.693 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:44:33.693 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:44:33.693 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:44:33.693 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:44:33.694 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:44:33.694 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:44:33.743 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 30 2024-04-06T09:44:33.743 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 33 2024-04-06T09:44:33.743 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:44:33.743 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=133, tag_tid=21, entry_tid=1], [object_number=132, tag_tid=21, entry_tid=0], [object_number=135, tag_tid=20, entry_tid=19939], [object_number=134, tag_tid=20, entry_tid=19938]]], state=connected] 2024-04-06T09:44:33.743 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=121, tag_tid=20, entry_tid=7369], [object_number=120, tag_tid=20, entry_tid=7368], [object_number=123, tag_tid=20, entry_tid=7367], [object_number=122, tag_tid=20, entry_tid=7366]]], state=connected] 2024-04-06T09:44:33.746 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:44:33.748 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=121, tag_tid=20, entry_tid=7369' 2024-04-06T09:44:33.748 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=133, tag_tid=21, entry_tid=1' -a 'object_number=133, tag_tid=21, entry_tid=1' = 'object_number=121, tag_tid=20, entry_tid=7369' 2024-04-06T09:44:33.748 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=121, tag_tid=20, entry_tid=7369' '!=' 'object_number=121, tag_tid=20, entry_tid=7281' 2024-04-06T09:44:33.748 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:44:33.748 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=121, tag_tid=20, entry_tid=7369' = 'object_number=121, tag_tid=20, entry_tid=7281' 2024-04-06T09:44:33.748 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=121, tag_tid=20, entry_tid=7369' 2024-04-06T09:44:33.749 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=133, tag_tid=21, entry_tid=1' 2024-04-06T09:44:33.749 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:44:33.749 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:44:33.749 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=21 2024-04-06T09:44:33.750 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=121, tag_tid=20, entry_tid=7369' 2024-04-06T09:44:33.750 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:44:33.750 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:44:33.751 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=20 2024-04-06T09:44:33.752 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=133, tag_tid=21, entry_tid=1' 2024-04-06T09:44:33.752 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:44:33.752 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:44:33.753 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:44:33.754 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=121, tag_tid=20, entry_tid=7369' 2024-04-06T09:44:33.754 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:44:33.754 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:44:33.755 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=7369 2024-04-06T09:44:33.755 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 21 = 20 -a 1 -le 7369 2024-04-06T09:44:33.755 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:44:33.755 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:44:33.755 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:44:33.957 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:44:33.957 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:44:33.957 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:44:33.957 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:44:33.957 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:44:33.957 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:44:33.958 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:44:33.958 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:44:33.958 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:44:33.958 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:44:33.958 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:44:33.958 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:44:33.958 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:44:33.958 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:44:33.958 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:44:33.959 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:44:33.959 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:44:33.961 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:44:33.962 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:44:33.962 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:44:33.962 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:44:33.962 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:44:33.962 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:44:33.962 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:44:33.962 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:44:33.962 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:44:33.962 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:44:33.962 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:44:33.962 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:44:33.962 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:44:33.962 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:44:33.962 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:44:33.963 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:44:33.963 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:44:33.963 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:44:33.963 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:44:33.963 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:44:33.963 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:44:33.963 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:44:33.963 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:44:33.963 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:44:33.963 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:44:33.963 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:44:33.963 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:44:33.963 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:44:33.963 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:44:33.963 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:44:33.964 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:44:33.964 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:44:33.987 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:44:33.987 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:44:33.987 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:44:33.987 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:44:34.118 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:44:34.119 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:44:34.119 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:44:34.119 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:44:34.119 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:44:34.119 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:44:34.119 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:44:34.119 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:44:34.119 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:44:34.120 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:44:34.120 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:44:34.120 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:44:34.120 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:44:34.122 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:44:34.122 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:44:34.122 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:44:34.171 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 30 2024-04-06T09:44:34.171 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 33 2024-04-06T09:44:34.171 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:44:34.171 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=133, tag_tid=21, entry_tid=1], [object_number=132, tag_tid=21, entry_tid=0], [object_number=135, tag_tid=20, entry_tid=19939], [object_number=134, tag_tid=20, entry_tid=19938]]], state=connected] 2024-04-06T09:44:34.172 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=121, tag_tid=20, entry_tid=7445], [object_number=120, tag_tid=20, entry_tid=7444], [object_number=123, tag_tid=20, entry_tid=7443], [object_number=122, tag_tid=20, entry_tid=7442]]], state=connected] 2024-04-06T09:44:34.175 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:44:34.176 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=133, tag_tid=21, entry_tid=1' 2024-04-06T09:44:34.177 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:44:34.177 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:44:34.177 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:44:34.177 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:44:34.177 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:44:34.177 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:44:34.177 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:44:34.177 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:44:34.177 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:44:34.178 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:44:34.178 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:44:34.178 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:44:34.179 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:44:34.179 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:44:34.179 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:44:34.226 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 30 2024-04-06T09:44:34.227 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 33 2024-04-06T09:44:34.227 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:44:34.227 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=133, tag_tid=21, entry_tid=1], [object_number=132, tag_tid=21, entry_tid=0], [object_number=135, tag_tid=20, entry_tid=19939], [object_number=134, tag_tid=20, entry_tid=19938]]], state=connected] 2024-04-06T09:44:34.227 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=121, tag_tid=20, entry_tid=7445], [object_number=120, tag_tid=20, entry_tid=7444], [object_number=123, tag_tid=20, entry_tid=7443], [object_number=122, tag_tid=20, entry_tid=7442]]], state=connected] 2024-04-06T09:44:34.230 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:44:34.231 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=121, tag_tid=20, entry_tid=7445' 2024-04-06T09:44:34.231 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=133, tag_tid=21, entry_tid=1' -a 'object_number=133, tag_tid=21, entry_tid=1' = 'object_number=121, tag_tid=20, entry_tid=7445' 2024-04-06T09:44:34.231 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=121, tag_tid=20, entry_tid=7445' '!=' 'object_number=121, tag_tid=20, entry_tid=7369' 2024-04-06T09:44:34.231 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:44:34.232 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=121, tag_tid=20, entry_tid=7445' = 'object_number=121, tag_tid=20, entry_tid=7369' 2024-04-06T09:44:34.232 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=121, tag_tid=20, entry_tid=7445' 2024-04-06T09:44:34.232 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=133, tag_tid=21, entry_tid=1' 2024-04-06T09:44:34.232 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:44:34.232 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:44:34.233 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=21 2024-04-06T09:44:34.233 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=121, tag_tid=20, entry_tid=7445' 2024-04-06T09:44:34.234 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:44:34.234 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:44:34.234 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=20 2024-04-06T09:44:34.235 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=133, tag_tid=21, entry_tid=1' 2024-04-06T09:44:34.235 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:44:34.235 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:44:34.236 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:44:34.237 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=121, tag_tid=20, entry_tid=7445' 2024-04-06T09:44:34.237 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:44:34.237 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:44:34.238 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=7445 2024-04-06T09:44:34.238 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 21 = 20 -a 1 -le 7445 2024-04-06T09:44:34.238 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:44:34.238 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:44:34.238 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:44:34.440 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:44:34.440 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:44:34.440 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:44:34.440 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:44:34.440 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:44:34.440 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:44:34.440 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:44:34.440 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:44:34.441 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:44:34.441 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:44:34.441 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:44:34.441 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:44:34.441 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:44:34.441 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:44:34.441 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:44:34.441 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:44:34.442 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:44:34.445 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:44:34.445 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:44:34.445 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:44:34.445 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:44:34.445 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:44:34.445 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:44:34.445 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:44:34.445 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:44:34.445 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:44:34.445 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:44:34.445 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:44:34.445 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:44:34.445 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:44:34.445 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:44:34.446 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:44:34.446 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:44:34.446 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:44:34.446 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:44:34.446 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:44:34.446 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:44:34.446 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:44:34.446 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:44:34.446 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:44:34.446 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:44:34.446 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:44:34.446 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:44:34.446 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:44:34.446 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:44:34.447 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:44:34.447 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:44:34.447 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:44:34.447 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:44:34.467 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:44:34.467 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:44:34.467 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:44:34.467 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:44:34.602 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:44:34.602 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:44:34.603 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:44:34.603 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:44:34.603 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:44:34.603 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:44:34.603 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:44:34.603 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:44:34.603 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:44:34.603 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:44:34.604 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:44:34.604 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:44:34.604 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:44:34.606 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:44:34.607 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:44:34.607 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:44:34.659 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 30 2024-04-06T09:44:34.659 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 33 2024-04-06T09:44:34.659 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:44:34.659 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=133, tag_tid=21, entry_tid=1], [object_number=132, tag_tid=21, entry_tid=0], [object_number=135, tag_tid=20, entry_tid=19939], [object_number=134, tag_tid=20, entry_tid=19938]]], state=connected] 2024-04-06T09:44:34.659 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=120, tag_tid=20, entry_tid=7488], [object_number=123, tag_tid=20, entry_tid=7487], [object_number=122, tag_tid=20, entry_tid=7486], [object_number=121, tag_tid=20, entry_tid=7485]]], state=connected] 2024-04-06T09:44:34.663 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:44:34.664 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=133, tag_tid=21, entry_tid=1' 2024-04-06T09:44:34.665 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:44:34.665 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:44:34.665 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:44:34.665 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:44:34.665 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:44:34.666 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:44:34.666 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:44:34.666 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:44:34.666 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:44:34.666 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:44:34.666 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:44:34.666 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:44:34.667 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:44:34.667 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:44:34.667 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:44:34.714 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 30 2024-04-06T09:44:34.714 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 33 2024-04-06T09:44:34.714 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:44:34.714 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=133, tag_tid=21, entry_tid=1], [object_number=132, tag_tid=21, entry_tid=0], [object_number=135, tag_tid=20, entry_tid=19939], [object_number=134, tag_tid=20, entry_tid=19938]]], state=connected] 2024-04-06T09:44:34.714 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=120, tag_tid=20, entry_tid=7488], [object_number=123, tag_tid=20, entry_tid=7487], [object_number=122, tag_tid=20, entry_tid=7486], [object_number=121, tag_tid=20, entry_tid=7485]]], state=connected] 2024-04-06T09:44:34.717 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:44:34.719 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=120, tag_tid=20, entry_tid=7488' 2024-04-06T09:44:34.719 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=133, tag_tid=21, entry_tid=1' -a 'object_number=133, tag_tid=21, entry_tid=1' = 'object_number=120, tag_tid=20, entry_tid=7488' 2024-04-06T09:44:34.719 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=120, tag_tid=20, entry_tid=7488' '!=' 'object_number=121, tag_tid=20, entry_tid=7445' 2024-04-06T09:44:34.719 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:44:34.719 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=120, tag_tid=20, entry_tid=7488' = 'object_number=121, tag_tid=20, entry_tid=7445' 2024-04-06T09:44:34.719 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=120, tag_tid=20, entry_tid=7488' 2024-04-06T09:44:34.720 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=133, tag_tid=21, entry_tid=1' 2024-04-06T09:44:34.720 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:44:34.720 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:44:34.721 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=21 2024-04-06T09:44:34.721 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=120, tag_tid=20, entry_tid=7488' 2024-04-06T09:44:34.721 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:44:34.721 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:44:34.722 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=20 2024-04-06T09:44:34.724 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=133, tag_tid=21, entry_tid=1' 2024-04-06T09:44:34.724 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:44:34.724 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:44:34.724 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:44:34.725 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=120, tag_tid=20, entry_tid=7488' 2024-04-06T09:44:34.725 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:44:34.725 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:44:34.726 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=7488 2024-04-06T09:44:34.726 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 21 = 20 -a 1 -le 7488 2024-04-06T09:44:34.726 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:44:34.726 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:44:34.726 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:44:34.928 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:44:34.928 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:44:34.928 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:44:34.928 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:44:34.929 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:44:34.929 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:44:34.929 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:44:34.929 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:44:34.929 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:44:34.929 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:44:34.929 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:44:34.929 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:44:34.929 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:44:34.929 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:44:34.929 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:44:34.930 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:44:34.930 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:44:34.931 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:44:34.931 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:44:34.931 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:44:34.931 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:44:34.932 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:44:34.932 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:44:34.932 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:44:34.932 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:44:34.932 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:44:34.932 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:44:34.932 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:44:34.932 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:44:34.932 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:44:34.932 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:44:34.932 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:44:34.932 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:44:34.934 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:44:34.934 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:44:34.934 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:44:34.934 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:44:34.934 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:44:34.934 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:44:34.934 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:44:34.934 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:44:34.934 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:44:34.934 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:44:34.934 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:44:34.934 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:44:34.934 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:44:34.934 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:44:34.935 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:44:34.935 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:44:34.959 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:44:34.959 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:44:34.959 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:44:34.959 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:44:35.092 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:44:35.093 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:44:35.093 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:44:35.093 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:44:35.093 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:44:35.093 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:44:35.093 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:44:35.093 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:44:35.093 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:44:35.093 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:44:35.094 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:44:35.094 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:44:35.094 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:44:35.095 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:44:35.095 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:44:35.096 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:44:35.142 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 30 2024-04-06T09:44:35.143 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 33 2024-04-06T09:44:35.143 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:44:35.143 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=133, tag_tid=21, entry_tid=1], [object_number=132, tag_tid=21, entry_tid=0], [object_number=135, tag_tid=20, entry_tid=19939], [object_number=134, tag_tid=20, entry_tid=19938]]], state=connected] 2024-04-06T09:44:35.143 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=122, tag_tid=20, entry_tid=7570], [object_number=121, tag_tid=20, entry_tid=7569], [object_number=120, tag_tid=20, entry_tid=7568], [object_number=123, tag_tid=20, entry_tid=7567]]], state=connected] 2024-04-06T09:44:35.146 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:44:35.147 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=133, tag_tid=21, entry_tid=1' 2024-04-06T09:44:35.148 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:44:35.148 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:44:35.148 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:44:35.149 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:44:35.149 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:44:35.149 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:44:35.149 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:44:35.149 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:44:35.149 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:44:35.149 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:44:35.149 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:44:35.149 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:44:35.150 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:44:35.150 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:44:35.150 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:44:35.197 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 30 2024-04-06T09:44:35.197 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 33 2024-04-06T09:44:35.197 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:44:35.197 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=133, tag_tid=21, entry_tid=1], [object_number=132, tag_tid=21, entry_tid=0], [object_number=135, tag_tid=20, entry_tid=19939], [object_number=134, tag_tid=20, entry_tid=19938]]], state=connected] 2024-04-06T09:44:35.197 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=122, tag_tid=20, entry_tid=7570], [object_number=121, tag_tid=20, entry_tid=7569], [object_number=120, tag_tid=20, entry_tid=7568], [object_number=123, tag_tid=20, entry_tid=7567]]], state=connected] 2024-04-06T09:44:35.201 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:44:35.203 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=122, tag_tid=20, entry_tid=7570' 2024-04-06T09:44:35.203 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=133, tag_tid=21, entry_tid=1' -a 'object_number=133, tag_tid=21, entry_tid=1' = 'object_number=122, tag_tid=20, entry_tid=7570' 2024-04-06T09:44:35.203 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=122, tag_tid=20, entry_tid=7570' '!=' 'object_number=120, tag_tid=20, entry_tid=7488' 2024-04-06T09:44:35.203 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:44:35.203 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=122, tag_tid=20, entry_tid=7570' = 'object_number=120, tag_tid=20, entry_tid=7488' 2024-04-06T09:44:35.203 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=122, tag_tid=20, entry_tid=7570' 2024-04-06T09:44:35.203 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=133, tag_tid=21, entry_tid=1' 2024-04-06T09:44:35.203 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:44:35.203 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:44:35.204 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=21 2024-04-06T09:44:35.205 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=122, tag_tid=20, entry_tid=7570' 2024-04-06T09:44:35.205 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:44:35.205 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:44:35.205 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=20 2024-04-06T09:44:35.206 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=133, tag_tid=21, entry_tid=1' 2024-04-06T09:44:35.206 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:44:35.206 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:44:35.207 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:44:35.208 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=122, tag_tid=20, entry_tid=7570' 2024-04-06T09:44:35.208 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:44:35.208 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:44:35.209 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=7570 2024-04-06T09:44:35.210 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 21 = 20 -a 1 -le 7570 2024-04-06T09:44:35.210 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:44:35.210 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:44:35.210 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:44:35.411 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:44:35.411 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:44:35.411 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:44:35.411 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:44:35.411 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:44:35.411 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:44:35.412 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:44:35.412 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:44:35.412 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:44:35.412 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:44:35.412 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:44:35.412 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:44:35.412 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:44:35.412 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:44:35.412 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:44:35.413 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:44:35.413 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:44:35.416 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:44:35.416 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:44:35.416 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:44:35.416 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:44:35.417 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:44:35.417 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:44:35.417 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:44:35.417 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:44:35.417 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:44:35.417 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:44:35.417 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:44:35.417 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:44:35.417 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:44:35.417 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:44:35.417 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:44:35.417 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:44:35.417 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:44:35.417 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:44:35.417 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:44:35.418 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:44:35.418 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:44:35.418 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:44:35.418 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:44:35.418 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:44:35.418 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:44:35.418 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:44:35.418 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:44:35.418 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:44:35.418 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:44:35.418 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:44:35.418 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:44:35.418 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:44:35.441 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:44:35.441 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:44:35.441 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:44:35.442 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:44:35.569 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:44:35.570 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:44:35.570 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:44:35.570 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:44:35.570 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:44:35.570 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:44:35.570 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:44:35.570 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:44:35.570 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:44:35.571 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:44:35.571 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:44:35.571 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:44:35.571 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:44:35.573 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:44:35.574 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:44:35.574 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:44:35.620 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 30 2024-04-06T09:44:35.620 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 33 2024-04-06T09:44:35.620 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:44:35.621 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=133, tag_tid=21, entry_tid=1], [object_number=132, tag_tid=21, entry_tid=0], [object_number=135, tag_tid=20, entry_tid=19939], [object_number=134, tag_tid=20, entry_tid=19938]]], state=connected] 2024-04-06T09:44:35.621 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=120, tag_tid=20, entry_tid=7652], [object_number=123, tag_tid=20, entry_tid=7651], [object_number=122, tag_tid=20, entry_tid=7650], [object_number=121, tag_tid=20, entry_tid=7649]]], state=connected] 2024-04-06T09:44:35.624 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:44:35.625 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=133, tag_tid=21, entry_tid=1' 2024-04-06T09:44:35.626 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:44:35.626 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:44:35.626 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:44:35.626 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:44:35.627 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:44:35.627 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:44:35.627 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:44:35.627 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:44:35.627 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:44:35.627 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:44:35.627 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:44:35.627 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:44:35.628 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:44:35.628 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:44:35.628 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:44:35.674 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 30 2024-04-06T09:44:35.674 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 33 2024-04-06T09:44:35.674 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:44:35.674 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=133, tag_tid=21, entry_tid=1], [object_number=132, tag_tid=21, entry_tid=0], [object_number=135, tag_tid=20, entry_tid=19939], [object_number=134, tag_tid=20, entry_tid=19938]]], state=connected] 2024-04-06T09:44:35.674 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=120, tag_tid=20, entry_tid=7652], [object_number=123, tag_tid=20, entry_tid=7651], [object_number=122, tag_tid=20, entry_tid=7650], [object_number=121, tag_tid=20, entry_tid=7649]]], state=connected] 2024-04-06T09:44:35.678 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:44:35.680 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=120, tag_tid=20, entry_tid=7652' 2024-04-06T09:44:35.680 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=133, tag_tid=21, entry_tid=1' -a 'object_number=133, tag_tid=21, entry_tid=1' = 'object_number=120, tag_tid=20, entry_tid=7652' 2024-04-06T09:44:35.680 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=120, tag_tid=20, entry_tid=7652' '!=' 'object_number=122, tag_tid=20, entry_tid=7570' 2024-04-06T09:44:35.680 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:44:35.680 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=120, tag_tid=20, entry_tid=7652' = 'object_number=122, tag_tid=20, entry_tid=7570' 2024-04-06T09:44:35.680 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=120, tag_tid=20, entry_tid=7652' 2024-04-06T09:44:35.680 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=133, tag_tid=21, entry_tid=1' 2024-04-06T09:44:35.680 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:44:35.681 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:44:35.681 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=21 2024-04-06T09:44:35.682 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=120, tag_tid=20, entry_tid=7652' 2024-04-06T09:44:35.682 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:44:35.682 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:44:35.683 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=20 2024-04-06T09:44:35.683 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=133, tag_tid=21, entry_tid=1' 2024-04-06T09:44:35.683 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:44:35.684 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:44:35.684 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:44:35.685 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=120, tag_tid=20, entry_tid=7652' 2024-04-06T09:44:35.685 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:44:35.685 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:44:35.686 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=7652 2024-04-06T09:44:35.686 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 21 = 20 -a 1 -le 7652 2024-04-06T09:44:35.686 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:44:35.687 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:44:35.687 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:44:35.888 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:44:35.888 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:44:35.888 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:44:35.889 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:44:35.889 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:44:35.889 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:44:35.889 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:44:35.889 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:44:35.889 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:44:35.889 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:44:35.889 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:44:35.889 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:44:35.889 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:44:35.889 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:44:35.889 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:44:35.890 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:44:35.890 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:44:35.892 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:44:35.892 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:44:35.892 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:44:35.892 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:44:35.892 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:44:35.893 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:44:35.893 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:44:35.893 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:44:35.893 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:44:35.893 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:44:35.893 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:44:35.893 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:44:35.893 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:44:35.893 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:44:35.894 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:44:35.894 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:44:35.894 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:44:35.894 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:44:35.894 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:44:35.894 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:44:35.894 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:44:35.894 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:44:35.894 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:44:35.894 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:44:35.894 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:44:35.894 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:44:35.894 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:44:35.895 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:44:35.895 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:44:35.895 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:44:35.895 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:44:35.895 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:44:35.919 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:44:35.919 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:44:35.919 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:44:35.919 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:44:36.049 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:44:36.050 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:44:36.050 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:44:36.050 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:44:36.050 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:44:36.050 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:44:36.050 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:44:36.051 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:44:36.051 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:44:36.051 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:44:36.051 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:44:36.051 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:44:36.051 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:44:36.052 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:44:36.053 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:44:36.053 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:44:36.105 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 30 2024-04-06T09:44:36.106 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 33 2024-04-06T09:44:36.106 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:44:36.106 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=133, tag_tid=21, entry_tid=1], [object_number=132, tag_tid=21, entry_tid=0], [object_number=135, tag_tid=20, entry_tid=19939], [object_number=134, tag_tid=20, entry_tid=19938]]], state=connected] 2024-04-06T09:44:36.106 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=120, tag_tid=20, entry_tid=7732], [object_number=123, tag_tid=20, entry_tid=7731], [object_number=122, tag_tid=20, entry_tid=7730], [object_number=121, tag_tid=20, entry_tid=7729]]], state=connected] 2024-04-06T09:44:36.109 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:44:36.110 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=133, tag_tid=21, entry_tid=1' 2024-04-06T09:44:36.111 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:44:36.112 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:44:36.112 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:44:36.112 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:44:36.112 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:44:36.112 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:44:36.112 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:44:36.112 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:44:36.112 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:44:36.112 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:44:36.112 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:44:36.112 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:44:36.113 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:44:36.113 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:44:36.113 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:44:36.160 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 30 2024-04-06T09:44:36.160 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 33 2024-04-06T09:44:36.160 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:44:36.160 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=133, tag_tid=21, entry_tid=1], [object_number=132, tag_tid=21, entry_tid=0], [object_number=135, tag_tid=20, entry_tid=19939], [object_number=134, tag_tid=20, entry_tid=19938]]], state=connected] 2024-04-06T09:44:36.161 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=120, tag_tid=20, entry_tid=7732], [object_number=123, tag_tid=20, entry_tid=7731], [object_number=122, tag_tid=20, entry_tid=7730], [object_number=121, tag_tid=20, entry_tid=7729]]], state=connected] 2024-04-06T09:44:36.164 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:44:36.166 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=120, tag_tid=20, entry_tid=7732' 2024-04-06T09:44:36.166 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=133, tag_tid=21, entry_tid=1' -a 'object_number=133, tag_tid=21, entry_tid=1' = 'object_number=120, tag_tid=20, entry_tid=7732' 2024-04-06T09:44:36.166 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=120, tag_tid=20, entry_tid=7732' '!=' 'object_number=120, tag_tid=20, entry_tid=7652' 2024-04-06T09:44:36.166 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:44:36.166 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=120, tag_tid=20, entry_tid=7732' = 'object_number=120, tag_tid=20, entry_tid=7652' 2024-04-06T09:44:36.166 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=120, tag_tid=20, entry_tid=7732' 2024-04-06T09:44:36.166 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=133, tag_tid=21, entry_tid=1' 2024-04-06T09:44:36.167 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:44:36.167 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:44:36.167 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=21 2024-04-06T09:44:36.168 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=120, tag_tid=20, entry_tid=7732' 2024-04-06T09:44:36.168 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:44:36.168 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:44:36.169 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=20 2024-04-06T09:44:36.169 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=133, tag_tid=21, entry_tid=1' 2024-04-06T09:44:36.170 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:44:36.170 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:44:36.171 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:44:36.171 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=120, tag_tid=20, entry_tid=7732' 2024-04-06T09:44:36.171 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:44:36.171 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:44:36.173 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=7732 2024-04-06T09:44:36.173 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 21 = 20 -a 1 -le 7732 2024-04-06T09:44:36.173 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:44:36.173 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:44:36.173 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:44:36.374 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:44:36.375 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:44:36.375 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:44:36.375 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:44:36.375 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:44:36.375 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:44:36.375 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:44:36.375 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:44:36.375 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:44:36.375 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:44:36.375 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:44:36.375 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:44:36.375 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:44:36.375 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:44:36.375 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:44:36.376 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:44:36.376 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:44:36.378 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:44:36.378 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:44:36.378 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:44:36.378 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:44:36.379 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:44:36.379 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:44:36.379 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:44:36.379 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:44:36.379 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:44:36.379 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:44:36.379 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:44:36.379 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:44:36.379 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:44:36.379 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:44:36.379 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:44:36.379 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:44:36.381 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:44:36.381 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:44:36.381 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:44:36.381 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:44:36.381 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:44:36.381 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:44:36.381 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:44:36.381 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:44:36.381 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:44:36.381 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:44:36.381 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:44:36.381 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:44:36.382 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:44:36.382 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:44:36.382 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:44:36.382 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:44:36.406 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:44:36.407 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:44:36.407 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:44:36.407 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:44:36.542 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:44:36.544 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:44:36.544 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:44:36.544 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:44:36.544 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:44:36.544 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:44:36.545 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:44:36.545 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:44:36.545 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:44:36.545 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:44:36.545 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:44:36.545 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:44:36.545 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:44:36.546 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:44:36.547 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:44:36.547 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:44:36.599 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 30 2024-04-06T09:44:36.599 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 33 2024-04-06T09:44:36.600 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:44:36.600 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=133, tag_tid=21, entry_tid=1], [object_number=132, tag_tid=21, entry_tid=0], [object_number=135, tag_tid=20, entry_tid=19939], [object_number=134, tag_tid=20, entry_tid=19938]]], state=connected] 2024-04-06T09:44:36.600 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=123, tag_tid=20, entry_tid=7823], [object_number=122, tag_tid=20, entry_tid=7822], [object_number=121, tag_tid=20, entry_tid=7821], [object_number=120, tag_tid=20, entry_tid=7820]]], state=connected] 2024-04-06T09:44:36.603 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:44:36.605 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=133, tag_tid=21, entry_tid=1' 2024-04-06T09:44:36.606 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:44:36.606 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:44:36.606 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:44:36.606 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:44:36.606 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:44:36.606 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:44:36.606 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:44:36.607 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:44:36.607 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:44:36.607 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:44:36.607 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:44:36.607 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:44:36.608 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:44:36.608 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:44:36.608 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:44:36.654 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 30 2024-04-06T09:44:36.655 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 33 2024-04-06T09:44:36.655 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:44:36.655 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=133, tag_tid=21, entry_tid=1], [object_number=132, tag_tid=21, entry_tid=0], [object_number=135, tag_tid=20, entry_tid=19939], [object_number=134, tag_tid=20, entry_tid=19938]]], state=connected] 2024-04-06T09:44:36.655 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=123, tag_tid=20, entry_tid=7823], [object_number=122, tag_tid=20, entry_tid=7822], [object_number=121, tag_tid=20, entry_tid=7821], [object_number=120, tag_tid=20, entry_tid=7820]]], state=connected] 2024-04-06T09:44:36.658 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:44:36.660 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=123, tag_tid=20, entry_tid=7823' 2024-04-06T09:44:36.660 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=133, tag_tid=21, entry_tid=1' -a 'object_number=133, tag_tid=21, entry_tid=1' = 'object_number=123, tag_tid=20, entry_tid=7823' 2024-04-06T09:44:36.660 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=123, tag_tid=20, entry_tid=7823' '!=' 'object_number=120, tag_tid=20, entry_tid=7732' 2024-04-06T09:44:36.660 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:44:36.660 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=123, tag_tid=20, entry_tid=7823' = 'object_number=120, tag_tid=20, entry_tid=7732' 2024-04-06T09:44:36.660 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=123, tag_tid=20, entry_tid=7823' 2024-04-06T09:44:36.661 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=133, tag_tid=21, entry_tid=1' 2024-04-06T09:44:36.661 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:44:36.661 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:44:36.661 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=21 2024-04-06T09:44:36.662 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=123, tag_tid=20, entry_tid=7823' 2024-04-06T09:44:36.662 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:44:36.662 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:44:36.663 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=20 2024-04-06T09:44:36.664 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=133, tag_tid=21, entry_tid=1' 2024-04-06T09:44:36.664 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:44:36.664 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:44:36.664 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:44:36.665 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=123, tag_tid=20, entry_tid=7823' 2024-04-06T09:44:36.665 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:44:36.665 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:44:36.667 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=7823 2024-04-06T09:44:36.667 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 21 = 20 -a 1 -le 7823 2024-04-06T09:44:36.667 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:44:36.667 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:44:36.667 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:44:36.868 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:44:36.869 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:44:36.869 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:44:36.869 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:44:36.869 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:44:36.869 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:44:36.869 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:44:36.869 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:44:36.869 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:44:36.869 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:44:36.869 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:44:36.869 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:44:36.869 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:44:36.869 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:44:36.869 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:44:36.870 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:44:36.870 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:44:36.872 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:44:36.873 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:44:36.873 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:44:36.873 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:44:36.873 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:44:36.873 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:44:36.873 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:44:36.873 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:44:36.873 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:44:36.873 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:44:36.873 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:44:36.873 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:44:36.873 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:44:36.873 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:44:36.873 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:44:36.873 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:44:36.874 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:44:36.874 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:44:36.875 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:44:36.875 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:44:36.875 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:44:36.875 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:44:36.875 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:44:36.875 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:44:36.875 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:44:36.875 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:44:36.875 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:44:36.875 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:44:36.875 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:44:36.875 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:44:36.875 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:44:36.875 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:44:36.899 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:44:36.900 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:44:36.900 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:44:36.900 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:44:37.039 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:44:37.040 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:44:37.040 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:44:37.040 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:44:37.040 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:44:37.041 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:44:37.041 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:44:37.041 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:44:37.041 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:44:37.041 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:44:37.041 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:44:37.042 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:44:37.042 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:44:37.043 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:44:37.043 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:44:37.044 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:44:37.091 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 30 2024-04-06T09:44:37.091 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 33 2024-04-06T09:44:37.091 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:44:37.092 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=133, tag_tid=21, entry_tid=1], [object_number=132, tag_tid=21, entry_tid=0], [object_number=135, tag_tid=20, entry_tid=19939], [object_number=134, tag_tid=20, entry_tid=19938]]], state=connected] 2024-04-06T09:44:37.092 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=123, tag_tid=20, entry_tid=7907], [object_number=122, tag_tid=20, entry_tid=7906], [object_number=121, tag_tid=20, entry_tid=7905], [object_number=120, tag_tid=20, entry_tid=7904]]], state=connected] 2024-04-06T09:44:37.095 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:44:37.096 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=133, tag_tid=21, entry_tid=1' 2024-04-06T09:44:37.097 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:44:37.097 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:44:37.098 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:44:37.098 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:44:37.098 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:44:37.098 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:44:37.098 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:44:37.098 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:44:37.098 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:44:37.098 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:44:37.098 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:44:37.098 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:44:37.099 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:44:37.099 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:44:37.099 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:44:37.147 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 30 2024-04-06T09:44:37.147 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 33 2024-04-06T09:44:37.148 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:44:37.148 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=133, tag_tid=21, entry_tid=1], [object_number=132, tag_tid=21, entry_tid=0], [object_number=135, tag_tid=20, entry_tid=19939], [object_number=134, tag_tid=20, entry_tid=19938]]], state=connected] 2024-04-06T09:44:37.148 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=123, tag_tid=20, entry_tid=7907], [object_number=122, tag_tid=20, entry_tid=7906], [object_number=121, tag_tid=20, entry_tid=7905], [object_number=120, tag_tid=20, entry_tid=7904]]], state=connected] 2024-04-06T09:44:37.151 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:44:37.153 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=123, tag_tid=20, entry_tid=7907' 2024-04-06T09:44:37.153 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=133, tag_tid=21, entry_tid=1' -a 'object_number=133, tag_tid=21, entry_tid=1' = 'object_number=123, tag_tid=20, entry_tid=7907' 2024-04-06T09:44:37.153 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=123, tag_tid=20, entry_tid=7907' '!=' 'object_number=123, tag_tid=20, entry_tid=7823' 2024-04-06T09:44:37.153 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:44:37.153 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=123, tag_tid=20, entry_tid=7907' = 'object_number=123, tag_tid=20, entry_tid=7823' 2024-04-06T09:44:37.153 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=123, tag_tid=20, entry_tid=7907' 2024-04-06T09:44:37.153 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=133, tag_tid=21, entry_tid=1' 2024-04-06T09:44:37.153 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:44:37.153 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:44:37.154 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=21 2024-04-06T09:44:37.155 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=123, tag_tid=20, entry_tid=7907' 2024-04-06T09:44:37.155 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:44:37.155 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:44:37.156 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=20 2024-04-06T09:44:37.156 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=133, tag_tid=21, entry_tid=1' 2024-04-06T09:44:37.156 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:44:37.156 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:44:37.157 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:44:37.158 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=123, tag_tid=20, entry_tid=7907' 2024-04-06T09:44:37.158 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:44:37.158 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:44:37.159 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=7907 2024-04-06T09:44:37.159 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 21 = 20 -a 1 -le 7907 2024-04-06T09:44:37.159 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:44:37.159 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:44:37.160 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:44:37.361 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:44:37.361 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:44:37.361 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:44:37.361 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:44:37.361 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:44:37.362 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:44:37.362 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:44:37.362 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:44:37.362 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:44:37.362 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:44:37.362 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:44:37.362 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:44:37.362 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:44:37.362 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:44:37.362 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:44:37.363 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:44:37.363 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:44:37.366 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:44:37.366 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:44:37.366 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:44:37.366 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:44:37.366 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:44:37.366 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:44:37.366 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:44:37.366 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:44:37.366 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:44:37.366 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:44:37.367 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:44:37.367 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:44:37.367 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:44:37.367 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:44:37.367 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:44:37.367 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:44:37.367 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:44:37.367 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:44:37.367 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:44:37.367 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:44:37.367 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:44:37.367 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:44:37.367 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:44:37.367 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:44:37.367 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:44:37.368 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:44:37.368 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:44:37.368 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:44:37.368 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:44:37.368 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:44:37.368 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:44:37.368 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:44:37.393 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:44:37.393 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:44:37.393 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:44:37.393 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:44:37.523 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:44:37.524 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:44:37.524 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:44:37.524 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:44:37.524 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:44:37.524 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:44:37.524 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:44:37.525 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:44:37.525 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:44:37.525 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:44:37.525 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:44:37.525 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:44:37.525 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:44:37.526 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:44:37.527 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:44:37.527 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:44:37.575 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 30 2024-04-06T09:44:37.576 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 33 2024-04-06T09:44:37.576 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:44:37.576 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=133, tag_tid=21, entry_tid=1], [object_number=132, tag_tid=21, entry_tid=0], [object_number=135, tag_tid=20, entry_tid=19939], [object_number=134, tag_tid=20, entry_tid=19938]]], state=connected] 2024-04-06T09:44:37.576 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=123, tag_tid=20, entry_tid=7995], [object_number=122, tag_tid=20, entry_tid=7994], [object_number=121, tag_tid=20, entry_tid=7993], [object_number=120, tag_tid=20, entry_tid=7992]]], state=connected] 2024-04-06T09:44:37.579 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:44:37.580 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=133, tag_tid=21, entry_tid=1' 2024-04-06T09:44:37.581 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:44:37.581 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:44:37.582 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:44:37.582 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:44:37.582 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:44:37.582 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:44:37.582 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:44:37.582 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:44:37.582 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:44:37.582 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:44:37.582 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:44:37.582 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:44:37.583 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:44:37.583 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:44:37.583 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:44:37.630 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 30 2024-04-06T09:44:37.630 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 33 2024-04-06T09:44:37.630 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:44:37.630 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=133, tag_tid=21, entry_tid=1], [object_number=132, tag_tid=21, entry_tid=0], [object_number=135, tag_tid=20, entry_tid=19939], [object_number=134, tag_tid=20, entry_tid=19938]]], state=connected] 2024-04-06T09:44:37.630 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=123, tag_tid=20, entry_tid=7995], [object_number=122, tag_tid=20, entry_tid=7994], [object_number=121, tag_tid=20, entry_tid=7993], [object_number=120, tag_tid=20, entry_tid=7992]]], state=connected] 2024-04-06T09:44:37.633 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:44:37.635 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=123, tag_tid=20, entry_tid=7995' 2024-04-06T09:44:37.636 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=133, tag_tid=21, entry_tid=1' -a 'object_number=133, tag_tid=21, entry_tid=1' = 'object_number=123, tag_tid=20, entry_tid=7995' 2024-04-06T09:44:37.636 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=123, tag_tid=20, entry_tid=7995' '!=' 'object_number=123, tag_tid=20, entry_tid=7907' 2024-04-06T09:44:37.636 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:44:37.636 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=123, tag_tid=20, entry_tid=7995' = 'object_number=123, tag_tid=20, entry_tid=7907' 2024-04-06T09:44:37.636 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=123, tag_tid=20, entry_tid=7995' 2024-04-06T09:44:37.636 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=133, tag_tid=21, entry_tid=1' 2024-04-06T09:44:37.636 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:44:37.636 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:44:37.636 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=21 2024-04-06T09:44:37.637 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=123, tag_tid=20, entry_tid=7995' 2024-04-06T09:44:37.637 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:44:37.637 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:44:37.638 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=20 2024-04-06T09:44:37.639 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=133, tag_tid=21, entry_tid=1' 2024-04-06T09:44:37.639 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:44:37.639 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:44:37.640 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:44:37.641 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=123, tag_tid=20, entry_tid=7995' 2024-04-06T09:44:37.641 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:44:37.641 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:44:37.642 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=7995 2024-04-06T09:44:37.642 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 21 = 20 -a 1 -le 7995 2024-04-06T09:44:37.642 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:44:37.642 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:44:37.642 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:44:37.843 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:44:37.844 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:44:37.844 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:44:37.844 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:44:37.844 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:44:37.844 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:44:37.844 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:44:37.844 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:44:37.844 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:44:37.844 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:44:37.844 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:44:37.844 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:44:37.844 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:44:37.844 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:44:37.845 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:44:37.845 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:44:37.845 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:44:37.847 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:44:37.847 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:44:37.848 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:44:37.848 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:44:37.848 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:44:37.848 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:44:37.848 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:44:37.848 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:44:37.848 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:44:37.848 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:44:37.848 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:44:37.848 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:44:37.848 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:44:37.848 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:44:37.848 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:44:37.848 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:44:37.849 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:44:37.849 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:44:37.849 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:44:37.850 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:44:37.850 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:44:37.850 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:44:37.850 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:44:37.850 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:44:37.850 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:44:37.850 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:44:37.850 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:44:37.850 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:44:37.850 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:44:37.850 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:44:37.850 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:44:37.850 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:44:37.874 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:44:37.875 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:44:37.875 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:44:37.875 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:44:38.006 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:44:38.007 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:44:38.007 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:44:38.007 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:44:38.008 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:44:38.008 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:44:38.008 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:44:38.008 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:44:38.008 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:44:38.008 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:44:38.008 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:44:38.009 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:44:38.009 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:44:38.011 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:44:38.011 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:44:38.011 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:44:38.064 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 30 2024-04-06T09:44:38.064 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 33 2024-04-06T09:44:38.064 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:44:38.064 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=133, tag_tid=21, entry_tid=1], [object_number=132, tag_tid=21, entry_tid=0], [object_number=135, tag_tid=20, entry_tid=19939], [object_number=134, tag_tid=20, entry_tid=19938]]], state=connected] 2024-04-06T09:44:38.064 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=123, tag_tid=20, entry_tid=8087], [object_number=122, tag_tid=20, entry_tid=8086], [object_number=121, tag_tid=20, entry_tid=8085], [object_number=120, tag_tid=20, entry_tid=8084]]], state=connected] 2024-04-06T09:44:38.067 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:44:38.068 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=133, tag_tid=21, entry_tid=1' 2024-04-06T09:44:38.069 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:44:38.070 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:44:38.070 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:44:38.070 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:44:38.070 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:44:38.070 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:44:38.070 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:44:38.070 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:44:38.070 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:44:38.070 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:44:38.070 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:44:38.070 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:44:38.071 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:44:38.071 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:44:38.071 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:44:38.325 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 30 2024-04-06T09:44:38.325 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 33 2024-04-06T09:44:38.325 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:44:38.325 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=133, tag_tid=21, entry_tid=1], [object_number=132, tag_tid=21, entry_tid=0], [object_number=135, tag_tid=20, entry_tid=19939], [object_number=134, tag_tid=20, entry_tid=19938]]], state=connected] 2024-04-06T09:44:38.325 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=123, tag_tid=20, entry_tid=8087], [object_number=122, tag_tid=20, entry_tid=8086], [object_number=121, tag_tid=20, entry_tid=8085], [object_number=120, tag_tid=20, entry_tid=8084]]], state=connected] 2024-04-06T09:44:38.330 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:44:38.333 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=123, tag_tid=20, entry_tid=8087' 2024-04-06T09:44:38.333 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=133, tag_tid=21, entry_tid=1' -a 'object_number=133, tag_tid=21, entry_tid=1' = 'object_number=123, tag_tid=20, entry_tid=8087' 2024-04-06T09:44:38.333 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=123, tag_tid=20, entry_tid=8087' '!=' 'object_number=123, tag_tid=20, entry_tid=7995' 2024-04-06T09:44:38.333 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:44:38.333 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=123, tag_tid=20, entry_tid=8087' = 'object_number=123, tag_tid=20, entry_tid=7995' 2024-04-06T09:44:38.333 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=123, tag_tid=20, entry_tid=8087' 2024-04-06T09:44:38.334 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=133, tag_tid=21, entry_tid=1' 2024-04-06T09:44:38.334 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:44:38.334 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:44:38.338 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=21 2024-04-06T09:44:38.338 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=123, tag_tid=20, entry_tid=8087' 2024-04-06T09:44:38.338 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:44:38.338 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:44:38.339 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=20 2024-04-06T09:44:38.340 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=133, tag_tid=21, entry_tid=1' 2024-04-06T09:44:38.340 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:44:38.340 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:44:38.342 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:44:38.343 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:44:38.343 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=123, tag_tid=20, entry_tid=8087' 2024-04-06T09:44:38.343 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:44:38.345 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=8087 2024-04-06T09:44:38.345 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 21 = 20 -a 1 -le 8087 2024-04-06T09:44:38.345 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:44:38.345 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:44:38.345 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:44:38.547 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:44:38.547 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:44:38.547 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:44:38.547 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:44:38.548 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:44:38.548 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:44:38.548 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:44:38.548 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:44:38.548 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:44:38.548 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:44:38.548 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:44:38.548 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:44:38.548 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:44:38.548 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:44:38.548 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:44:38.549 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:44:38.549 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:44:38.553 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:44:38.553 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:44:38.553 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:44:38.553 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:44:38.553 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:44:38.553 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:44:38.553 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:44:38.553 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:44:38.553 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:44:38.554 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:44:38.554 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:44:38.554 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:44:38.554 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:44:38.554 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:44:38.554 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:44:38.554 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:44:38.554 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:44:38.554 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:44:38.554 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:44:38.554 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:44:38.554 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:44:38.554 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:44:38.554 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:44:38.555 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:44:38.555 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:44:38.555 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:44:38.555 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:44:38.555 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:44:38.555 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:44:38.555 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:44:38.555 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:44:38.555 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:44:38.578 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:44:38.578 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:44:38.578 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:44:38.578 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:44:38.709 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:44:38.710 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:44:38.710 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:44:38.710 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:44:38.710 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:44:38.710 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:44:38.710 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:44:38.710 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:44:38.710 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:44:38.710 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:44:38.711 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:44:38.711 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:44:38.711 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:44:38.713 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:44:38.713 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:44:38.713 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:44:38.765 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 30 2024-04-06T09:44:38.766 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 33 2024-04-06T09:44:38.766 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:44:38.766 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=133, tag_tid=21, entry_tid=1], [object_number=132, tag_tid=21, entry_tid=0], [object_number=135, tag_tid=20, entry_tid=19939], [object_number=134, tag_tid=20, entry_tid=19938]]], state=connected] 2024-04-06T09:44:38.766 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=120, tag_tid=20, entry_tid=8216], [object_number=123, tag_tid=20, entry_tid=8215], [object_number=122, tag_tid=20, entry_tid=8214], [object_number=121, tag_tid=20, entry_tid=8213]]], state=connected] 2024-04-06T09:44:38.769 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:44:38.771 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=133, tag_tid=21, entry_tid=1' 2024-04-06T09:44:38.772 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:44:38.772 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:44:38.772 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:44:38.772 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:44:38.772 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:44:38.772 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:44:38.772 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:44:38.772 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:44:38.772 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:44:38.773 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:44:38.773 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:44:38.773 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:44:38.773 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:44:38.773 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:44:38.773 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:44:38.823 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 30 2024-04-06T09:44:38.823 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 33 2024-04-06T09:44:38.823 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:44:38.823 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=133, tag_tid=21, entry_tid=1], [object_number=132, tag_tid=21, entry_tid=0], [object_number=135, tag_tid=20, entry_tid=19939], [object_number=134, tag_tid=20, entry_tid=19938]]], state=connected] 2024-04-06T09:44:38.823 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=120, tag_tid=20, entry_tid=8216], [object_number=123, tag_tid=20, entry_tid=8215], [object_number=122, tag_tid=20, entry_tid=8214], [object_number=121, tag_tid=20, entry_tid=8213]]], state=connected] 2024-04-06T09:44:38.827 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:44:38.829 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=120, tag_tid=20, entry_tid=8216' 2024-04-06T09:44:38.829 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=133, tag_tid=21, entry_tid=1' -a 'object_number=133, tag_tid=21, entry_tid=1' = 'object_number=120, tag_tid=20, entry_tid=8216' 2024-04-06T09:44:38.829 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=120, tag_tid=20, entry_tid=8216' '!=' 'object_number=123, tag_tid=20, entry_tid=8087' 2024-04-06T09:44:38.829 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:44:38.829 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=120, tag_tid=20, entry_tid=8216' = 'object_number=123, tag_tid=20, entry_tid=8087' 2024-04-06T09:44:38.829 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=120, tag_tid=20, entry_tid=8216' 2024-04-06T09:44:38.829 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=133, tag_tid=21, entry_tid=1' 2024-04-06T09:44:38.829 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:44:38.829 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:44:38.830 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=21 2024-04-06T09:44:38.830 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=120, tag_tid=20, entry_tid=8216' 2024-04-06T09:44:38.831 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:44:38.831 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:44:38.832 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=20 2024-04-06T09:44:38.832 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=133, tag_tid=21, entry_tid=1' 2024-04-06T09:44:38.832 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:44:38.832 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:44:38.833 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:44:38.834 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=120, tag_tid=20, entry_tid=8216' 2024-04-06T09:44:38.834 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:44:38.834 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:44:38.835 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=8216 2024-04-06T09:44:38.835 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 21 = 20 -a 1 -le 8216 2024-04-06T09:44:38.835 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:44:38.835 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:44:38.836 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:44:39.037 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:44:39.037 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:44:39.037 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:44:39.037 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:44:39.037 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:44:39.037 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:44:39.038 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:44:39.038 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:44:39.038 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:44:39.038 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:44:39.038 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:44:39.038 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:44:39.038 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:44:39.038 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:44:39.038 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:44:39.038 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:44:39.039 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:44:39.042 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:44:39.043 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:44:39.043 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:44:39.043 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:44:39.043 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:44:39.043 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:44:39.043 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:44:39.043 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:44:39.043 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:44:39.043 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:44:39.043 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:44:39.043 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:44:39.043 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:44:39.043 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:44:39.043 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:44:39.044 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:44:39.044 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:44:39.044 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:44:39.044 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:44:39.044 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:44:39.044 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:44:39.044 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:44:39.044 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:44:39.044 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:44:39.044 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:44:39.044 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:44:39.044 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:44:39.044 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:44:39.044 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:44:39.044 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:44:39.044 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:44:39.045 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:44:39.067 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:44:39.068 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:44:39.068 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:44:39.068 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:44:39.208 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:44:39.209 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:44:39.210 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:44:39.210 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:44:39.210 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:44:39.210 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:44:39.210 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:44:39.210 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:44:39.210 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:44:39.210 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:44:39.210 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:44:39.211 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:44:39.211 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:44:39.213 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:44:39.213 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:44:39.213 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:44:39.266 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 30 2024-04-06T09:44:39.267 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 33 2024-04-06T09:44:39.267 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:44:39.267 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=133, tag_tid=21, entry_tid=1], [object_number=132, tag_tid=21, entry_tid=0], [object_number=135, tag_tid=20, entry_tid=19939], [object_number=134, tag_tid=20, entry_tid=19938]]], state=connected] 2024-04-06T09:44:39.267 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=122, tag_tid=20, entry_tid=8310], [object_number=121, tag_tid=20, entry_tid=8309], [object_number=120, tag_tid=20, entry_tid=8308], [object_number=123, tag_tid=20, entry_tid=8307]]], state=connected] 2024-04-06T09:44:39.270 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:44:39.271 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=133, tag_tid=21, entry_tid=1' 2024-04-06T09:44:39.272 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:44:39.273 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:44:39.273 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:44:39.273 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:44:39.273 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:44:39.273 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:44:39.273 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:44:39.273 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:44:39.273 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:44:39.273 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:44:39.273 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:44:39.273 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:44:39.274 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:44:39.274 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:44:39.274 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:44:39.321 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 30 2024-04-06T09:44:39.321 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 33 2024-04-06T09:44:39.321 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:44:39.321 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=133, tag_tid=21, entry_tid=1], [object_number=132, tag_tid=21, entry_tid=0], [object_number=135, tag_tid=20, entry_tid=19939], [object_number=134, tag_tid=20, entry_tid=19938]]], state=connected] 2024-04-06T09:44:39.321 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=122, tag_tid=20, entry_tid=8310], [object_number=121, tag_tid=20, entry_tid=8309], [object_number=120, tag_tid=20, entry_tid=8308], [object_number=123, tag_tid=20, entry_tid=8307]]], state=connected] 2024-04-06T09:44:39.325 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:44:39.326 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=122, tag_tid=20, entry_tid=8310' 2024-04-06T09:44:39.327 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=133, tag_tid=21, entry_tid=1' -a 'object_number=133, tag_tid=21, entry_tid=1' = 'object_number=122, tag_tid=20, entry_tid=8310' 2024-04-06T09:44:39.327 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=122, tag_tid=20, entry_tid=8310' '!=' 'object_number=120, tag_tid=20, entry_tid=8216' 2024-04-06T09:44:39.327 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:44:39.327 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=122, tag_tid=20, entry_tid=8310' = 'object_number=120, tag_tid=20, entry_tid=8216' 2024-04-06T09:44:39.327 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=122, tag_tid=20, entry_tid=8310' 2024-04-06T09:44:39.327 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=133, tag_tid=21, entry_tid=1' 2024-04-06T09:44:39.327 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:44:39.327 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:44:39.328 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=21 2024-04-06T09:44:39.328 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=122, tag_tid=20, entry_tid=8310' 2024-04-06T09:44:39.328 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:44:39.328 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:44:39.329 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=20 2024-04-06T09:44:39.330 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=133, tag_tid=21, entry_tid=1' 2024-04-06T09:44:39.330 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:44:39.330 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:44:39.331 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:44:39.332 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=122, tag_tid=20, entry_tid=8310' 2024-04-06T09:44:39.332 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:44:39.332 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:44:39.333 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=8310 2024-04-06T09:44:39.333 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 21 = 20 -a 1 -le 8310 2024-04-06T09:44:39.333 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:44:39.333 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:44:39.333 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:44:39.535 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:44:39.535 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:44:39.535 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:44:39.535 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:44:39.535 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:44:39.536 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:44:39.536 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:44:39.536 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:44:39.536 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:44:39.536 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:44:39.536 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:44:39.536 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:44:39.536 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:44:39.536 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:44:39.536 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:44:39.537 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:44:39.537 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:44:39.539 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:44:39.539 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:44:39.539 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:44:39.539 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:44:39.539 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:44:39.539 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:44:39.539 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:44:39.539 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:44:39.539 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:44:39.540 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:44:39.540 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:44:39.540 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:44:39.540 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:44:39.540 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:44:39.540 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:44:39.540 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:44:39.540 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:44:39.540 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:44:39.540 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:44:39.540 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:44:39.540 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:44:39.540 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:44:39.540 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:44:39.541 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:44:39.541 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:44:39.541 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:44:39.541 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:44:39.541 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:44:39.541 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:44:39.541 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:44:39.541 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:44:39.541 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:44:39.565 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:44:39.565 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:44:39.566 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:44:39.566 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:44:39.905 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:44:39.906 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:44:39.906 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:44:39.906 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:44:39.906 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:44:39.907 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:44:39.907 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:44:39.907 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:44:39.907 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:44:39.907 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:44:39.907 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:44:39.907 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:44:39.908 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:44:39.909 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:44:39.910 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:44:39.910 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:44:39.965 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 30 2024-04-06T09:44:39.965 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 33 2024-04-06T09:44:39.965 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:44:39.965 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=133, tag_tid=21, entry_tid=1], [object_number=132, tag_tid=21, entry_tid=0], [object_number=135, tag_tid=20, entry_tid=19939], [object_number=134, tag_tid=20, entry_tid=19938]]], state=connected] 2024-04-06T09:44:39.965 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=122, tag_tid=20, entry_tid=8406], [object_number=121, tag_tid=20, entry_tid=8405], [object_number=120, tag_tid=20, entry_tid=8404], [object_number=123, tag_tid=20, entry_tid=8403]]], state=connected] 2024-04-06T09:44:39.968 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:44:39.969 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=133, tag_tid=21, entry_tid=1' 2024-04-06T09:44:39.970 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:44:39.970 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:44:39.970 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:44:39.971 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:44:39.971 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:44:39.971 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:44:39.971 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:44:39.971 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:44:39.971 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:44:39.971 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:44:39.971 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:44:39.972 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:44:39.972 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:44:40.098 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:44:40.098 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:44:40.101 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 30 2024-04-06T09:44:40.101 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 33 2024-04-06T09:44:40.101 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:44:40.101 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=133, tag_tid=21, entry_tid=1], [object_number=132, tag_tid=21, entry_tid=0], [object_number=135, tag_tid=20, entry_tid=19939], [object_number=134, tag_tid=20, entry_tid=19938]]], state=connected] 2024-04-06T09:44:40.101 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=122, tag_tid=20, entry_tid=8406], [object_number=121, tag_tid=20, entry_tid=8405], [object_number=120, tag_tid=20, entry_tid=8404], [object_number=123, tag_tid=20, entry_tid=8403]]], state=connected] 2024-04-06T09:44:40.102 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:44:40.102 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=122, tag_tid=20, entry_tid=8406' 2024-04-06T09:44:40.102 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=133, tag_tid=21, entry_tid=1' -a 'object_number=133, tag_tid=21, entry_tid=1' = 'object_number=122, tag_tid=20, entry_tid=8406' 2024-04-06T09:44:40.102 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=122, tag_tid=20, entry_tid=8406' '!=' 'object_number=122, tag_tid=20, entry_tid=8310' 2024-04-06T09:44:40.102 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:44:40.102 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=122, tag_tid=20, entry_tid=8406' = 'object_number=122, tag_tid=20, entry_tid=8310' 2024-04-06T09:44:40.102 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=122, tag_tid=20, entry_tid=8406' 2024-04-06T09:44:40.102 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=133, tag_tid=21, entry_tid=1' 2024-04-06T09:44:40.102 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:44:40.102 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:44:40.102 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=21 2024-04-06T09:44:40.102 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=122, tag_tid=20, entry_tid=8406' 2024-04-06T09:44:40.102 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:44:40.103 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:44:40.103 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=20 2024-04-06T09:44:40.103 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=133, tag_tid=21, entry_tid=1' 2024-04-06T09:44:40.103 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:44:40.103 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:44:40.103 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:44:40.103 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=122, tag_tid=20, entry_tid=8406' 2024-04-06T09:44:40.103 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:44:40.103 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:44:40.103 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=8406 2024-04-06T09:44:40.103 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 21 = 20 -a 1 -le 8406 2024-04-06T09:44:40.103 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:44:40.103 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:44:40.103 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:44:40.235 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:44:40.235 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:44:40.235 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:44:40.235 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:44:40.235 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:44:40.235 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:44:40.235 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:44:40.235 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:44:40.235 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:44:40.235 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:44:40.235 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:44:40.235 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:44:40.236 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:44:40.236 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:44:40.236 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:44:40.236 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:44:40.237 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:44:40.239 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:44:40.239 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:44:40.239 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:44:40.239 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:44:40.239 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:44:40.239 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:44:40.239 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:44:40.239 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:44:40.239 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:44:40.240 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:44:40.240 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:44:40.240 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:44:40.240 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:44:40.240 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:44:40.240 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:44:40.240 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:44:40.241 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:44:40.241 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:44:40.241 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:44:40.241 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:44:40.241 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:44:40.241 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:44:40.241 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:44:40.241 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:44:40.242 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:44:40.242 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:44:40.242 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:44:40.242 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:44:40.242 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:44:40.242 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:44:40.242 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:44:40.242 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:44:40.266 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:44:40.266 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:44:40.266 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:44:40.266 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:44:40.407 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:44:40.409 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:44:40.409 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:44:40.409 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:44:40.409 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:44:40.409 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:44:40.409 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:44:40.409 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:44:40.409 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:44:40.409 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:44:40.410 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:44:40.410 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:44:40.410 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:44:40.411 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:44:40.412 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:44:40.412 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:44:40.469 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 30 2024-04-06T09:44:40.469 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 33 2024-04-06T09:44:40.469 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:44:40.469 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=133, tag_tid=21, entry_tid=1], [object_number=132, tag_tid=21, entry_tid=0], [object_number=135, tag_tid=20, entry_tid=19939], [object_number=134, tag_tid=20, entry_tid=19938]]], state=connected] 2024-04-06T09:44:40.469 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=121, tag_tid=20, entry_tid=8497], [object_number=120, tag_tid=20, entry_tid=8496], [object_number=123, tag_tid=20, entry_tid=8495], [object_number=122, tag_tid=20, entry_tid=8494]]], state=connected] 2024-04-06T09:44:40.545 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:44:40.547 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=133, tag_tid=21, entry_tid=1' 2024-04-06T09:44:40.549 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:44:40.549 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:44:40.549 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:44:40.549 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:44:40.549 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:44:40.549 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:44:40.549 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:44:40.549 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:44:40.549 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:44:40.549 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:44:40.550 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:44:40.550 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:44:40.551 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:44:40.552 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:44:40.552 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:44:40.608 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 30 2024-04-06T09:44:40.608 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 33 2024-04-06T09:44:40.609 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:44:40.609 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=133, tag_tid=21, entry_tid=1], [object_number=132, tag_tid=21, entry_tid=0], [object_number=135, tag_tid=20, entry_tid=19939], [object_number=134, tag_tid=20, entry_tid=19938]]], state=connected] 2024-04-06T09:44:40.609 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=121, tag_tid=20, entry_tid=8497], [object_number=120, tag_tid=20, entry_tid=8496], [object_number=123, tag_tid=20, entry_tid=8495], [object_number=122, tag_tid=20, entry_tid=8494]]], state=connected] 2024-04-06T09:44:40.612 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:44:40.614 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=121, tag_tid=20, entry_tid=8497' 2024-04-06T09:44:40.614 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=133, tag_tid=21, entry_tid=1' -a 'object_number=133, tag_tid=21, entry_tid=1' = 'object_number=121, tag_tid=20, entry_tid=8497' 2024-04-06T09:44:40.614 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=121, tag_tid=20, entry_tid=8497' '!=' 'object_number=122, tag_tid=20, entry_tid=8406' 2024-04-06T09:44:40.614 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:44:40.615 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=121, tag_tid=20, entry_tid=8497' = 'object_number=122, tag_tid=20, entry_tid=8406' 2024-04-06T09:44:40.615 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=121, tag_tid=20, entry_tid=8497' 2024-04-06T09:44:40.615 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=133, tag_tid=21, entry_tid=1' 2024-04-06T09:44:40.615 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:44:40.615 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:44:40.616 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=21 2024-04-06T09:44:40.617 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=121, tag_tid=20, entry_tid=8497' 2024-04-06T09:44:40.617 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:44:40.617 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:44:40.618 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=20 2024-04-06T09:44:40.618 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=133, tag_tid=21, entry_tid=1' 2024-04-06T09:44:40.619 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:44:40.619 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:44:40.620 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:44:40.620 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=121, tag_tid=20, entry_tid=8497' 2024-04-06T09:44:40.620 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:44:40.620 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:44:40.622 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=8497 2024-04-06T09:44:40.622 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 21 = 20 -a 1 -le 8497 2024-04-06T09:44:40.622 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:44:40.622 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:44:40.622 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:44:40.823 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:44:40.824 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:44:40.824 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:44:40.824 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:44:40.824 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:44:40.824 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:44:40.824 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:44:40.824 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:44:40.824 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:44:40.824 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:44:40.824 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:44:40.824 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:44:40.824 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:44:40.824 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:44:40.825 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:44:40.825 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:44:40.826 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:44:40.828 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:44:40.828 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:44:40.828 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:44:40.828 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:44:40.828 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:44:40.828 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:44:40.828 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:44:40.828 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:44:40.828 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:44:40.828 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:44:40.828 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:44:40.828 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:44:40.828 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:44:40.829 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:44:40.829 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:44:40.829 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:44:40.829 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:44:40.830 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:44:40.830 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:44:40.830 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:44:40.830 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:44:40.830 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:44:40.830 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:44:40.830 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:44:40.830 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:44:40.830 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:44:40.830 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:44:40.830 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:44:40.830 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:44:40.830 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:44:40.830 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:44:40.831 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:44:40.855 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:44:40.855 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:44:40.855 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:44:40.855 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:44:40.989 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:44:40.990 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:44:40.990 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:44:40.990 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:44:40.990 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:44:40.990 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:44:40.990 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:44:40.990 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:44:40.990 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:44:40.990 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:44:40.991 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:44:40.991 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:44:40.991 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:44:40.993 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:44:40.993 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:44:40.994 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:44:41.040 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 30 2024-04-06T09:44:41.040 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 33 2024-04-06T09:44:41.040 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:44:41.040 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=133, tag_tid=21, entry_tid=1], [object_number=132, tag_tid=21, entry_tid=0], [object_number=135, tag_tid=20, entry_tid=19939], [object_number=134, tag_tid=20, entry_tid=19938]]], state=connected] 2024-04-06T09:44:41.040 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=120, tag_tid=20, entry_tid=8604], [object_number=123, tag_tid=20, entry_tid=8603], [object_number=122, tag_tid=20, entry_tid=8602], [object_number=121, tag_tid=20, entry_tid=8601]]], state=connected] 2024-04-06T09:44:41.043 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:44:41.045 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=133, tag_tid=21, entry_tid=1' 2024-04-06T09:44:41.046 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:44:41.046 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:44:41.046 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:44:41.046 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:44:41.046 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:44:41.046 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:44:41.046 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:44:41.046 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:44:41.046 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:44:41.046 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:44:41.046 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:44:41.046 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:44:41.047 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:44:41.047 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:44:41.047 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:44:41.094 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 30 2024-04-06T09:44:41.094 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 33 2024-04-06T09:44:41.094 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:44:41.094 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=133, tag_tid=21, entry_tid=1], [object_number=132, tag_tid=21, entry_tid=0], [object_number=135, tag_tid=20, entry_tid=19939], [object_number=134, tag_tid=20, entry_tid=19938]]], state=connected] 2024-04-06T09:44:41.094 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=120, tag_tid=20, entry_tid=8604], [object_number=123, tag_tid=20, entry_tid=8603], [object_number=122, tag_tid=20, entry_tid=8602], [object_number=121, tag_tid=20, entry_tid=8601]]], state=connected] 2024-04-06T09:44:41.097 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:44:41.099 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=120, tag_tid=20, entry_tid=8604' 2024-04-06T09:44:41.099 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=133, tag_tid=21, entry_tid=1' -a 'object_number=133, tag_tid=21, entry_tid=1' = 'object_number=120, tag_tid=20, entry_tid=8604' 2024-04-06T09:44:41.100 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=120, tag_tid=20, entry_tid=8604' '!=' 'object_number=121, tag_tid=20, entry_tid=8497' 2024-04-06T09:44:41.100 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:44:41.100 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=120, tag_tid=20, entry_tid=8604' = 'object_number=121, tag_tid=20, entry_tid=8497' 2024-04-06T09:44:41.100 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=120, tag_tid=20, entry_tid=8604' 2024-04-06T09:44:41.100 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=133, tag_tid=21, entry_tid=1' 2024-04-06T09:44:41.100 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:44:41.100 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:44:41.101 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=21 2024-04-06T09:44:41.101 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=120, tag_tid=20, entry_tid=8604' 2024-04-06T09:44:41.101 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:44:41.101 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:44:41.102 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=20 2024-04-06T09:44:41.103 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=133, tag_tid=21, entry_tid=1' 2024-04-06T09:44:41.103 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:44:41.103 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:44:41.104 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:44:41.105 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=120, tag_tid=20, entry_tid=8604' 2024-04-06T09:44:41.105 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:44:41.105 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:44:41.106 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=8604 2024-04-06T09:44:41.106 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 21 = 20 -a 1 -le 8604 2024-04-06T09:44:41.106 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:44:41.106 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:44:41.106 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:44:41.307 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:44:41.308 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:44:41.308 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:44:41.308 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:44:41.308 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:44:41.308 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:44:41.308 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:44:41.308 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:44:41.308 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:44:41.308 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:44:41.308 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:44:41.308 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:44:41.308 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:44:41.308 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:44:41.308 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:44:41.309 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:44:41.309 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:44:41.311 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:44:41.311 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:44:41.311 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:44:41.311 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:44:41.311 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:44:41.311 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:44:41.311 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:44:41.311 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:44:41.311 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:44:41.311 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:44:41.311 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:44:41.312 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:44:41.312 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:44:41.312 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:44:41.312 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:44:41.312 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:44:41.313 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:44:41.313 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:44:41.313 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:44:41.313 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:44:41.313 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:44:41.313 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:44:41.313 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:44:41.313 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:44:41.313 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:44:41.314 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:44:41.314 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:44:41.314 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:44:41.314 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:44:41.314 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:44:41.314 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:44:41.314 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:44:41.338 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:44:41.338 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:44:41.338 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:44:41.338 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:44:41.462 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:44:41.463 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:44:41.464 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:44:41.464 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:44:41.464 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:44:41.464 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:44:41.464 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:44:41.464 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:44:41.464 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:44:41.464 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:44:41.465 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:44:41.465 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:44:41.465 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:44:41.467 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:44:41.467 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:44:41.467 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:44:41.514 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 30 2024-04-06T09:44:41.514 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 33 2024-04-06T09:44:41.514 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:44:41.514 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=133, tag_tid=21, entry_tid=1], [object_number=132, tag_tid=21, entry_tid=0], [object_number=135, tag_tid=20, entry_tid=19939], [object_number=134, tag_tid=20, entry_tid=19938]]], state=connected] 2024-04-06T09:44:41.514 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=120, tag_tid=20, entry_tid=8688], [object_number=123, tag_tid=20, entry_tid=8687], [object_number=122, tag_tid=20, entry_tid=8686], [object_number=121, tag_tid=20, entry_tid=8685]]], state=connected] 2024-04-06T09:44:41.517 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:44:41.519 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=133, tag_tid=21, entry_tid=1' 2024-04-06T09:44:41.520 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:44:41.520 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:44:41.520 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:44:41.520 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:44:41.520 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:44:41.520 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:44:41.520 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:44:41.520 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:44:41.520 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:44:41.520 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:44:41.521 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:44:41.521 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:44:41.522 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:44:41.522 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:44:41.522 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:44:41.569 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 30 2024-04-06T09:44:41.569 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 33 2024-04-06T09:44:41.569 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:44:41.569 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=133, tag_tid=21, entry_tid=1], [object_number=132, tag_tid=21, entry_tid=0], [object_number=135, tag_tid=20, entry_tid=19939], [object_number=134, tag_tid=20, entry_tid=19938]]], state=connected] 2024-04-06T09:44:41.570 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=120, tag_tid=20, entry_tid=8688], [object_number=123, tag_tid=20, entry_tid=8687], [object_number=122, tag_tid=20, entry_tid=8686], [object_number=121, tag_tid=20, entry_tid=8685]]], state=connected] 2024-04-06T09:44:41.572 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:44:41.575 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=120, tag_tid=20, entry_tid=8688' 2024-04-06T09:44:41.575 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=133, tag_tid=21, entry_tid=1' -a 'object_number=133, tag_tid=21, entry_tid=1' = 'object_number=120, tag_tid=20, entry_tid=8688' 2024-04-06T09:44:41.575 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=120, tag_tid=20, entry_tid=8688' '!=' 'object_number=120, tag_tid=20, entry_tid=8604' 2024-04-06T09:44:41.575 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:44:41.575 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=120, tag_tid=20, entry_tid=8688' = 'object_number=120, tag_tid=20, entry_tid=8604' 2024-04-06T09:44:41.575 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=120, tag_tid=20, entry_tid=8688' 2024-04-06T09:44:41.576 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=133, tag_tid=21, entry_tid=1' 2024-04-06T09:44:41.576 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:44:41.576 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:44:41.576 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=21 2024-04-06T09:44:41.577 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=120, tag_tid=20, entry_tid=8688' 2024-04-06T09:44:41.577 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:44:41.577 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:44:41.578 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=20 2024-04-06T09:44:41.579 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=133, tag_tid=21, entry_tid=1' 2024-04-06T09:44:41.579 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:44:41.579 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:44:41.580 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:44:41.580 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=120, tag_tid=20, entry_tid=8688' 2024-04-06T09:44:41.581 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:44:41.581 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:44:41.581 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=8688 2024-04-06T09:44:41.582 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 21 = 20 -a 1 -le 8688 2024-04-06T09:44:41.582 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:44:41.582 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:44:41.582 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:44:41.783 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:44:41.783 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:44:41.783 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:44:41.783 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:44:41.783 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:44:41.783 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:44:41.783 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:44:41.783 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:44:41.784 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:44:41.784 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:44:41.784 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:44:41.784 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:44:41.784 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:44:41.784 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:44:41.784 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:44:41.784 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:44:41.785 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:44:41.788 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:44:41.788 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:44:41.788 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:44:41.788 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:44:41.788 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:44:41.788 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:44:41.788 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:44:41.788 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:44:41.788 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:44:41.788 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:44:41.788 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:44:41.788 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:44:41.788 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:44:41.788 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:44:41.789 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:44:41.789 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:44:41.789 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:44:41.789 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:44:41.789 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:44:41.789 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:44:41.789 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:44:41.789 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:44:41.789 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:44:41.789 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:44:41.789 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:44:41.789 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:44:41.789 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:44:41.789 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:44:41.789 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:44:41.790 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:44:41.790 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:44:41.790 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:44:41.814 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:44:41.814 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:44:41.814 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:44:41.814 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:44:41.947 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:44:41.948 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:44:41.948 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:44:41.948 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:44:41.948 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:44:41.948 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:44:41.948 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:44:41.948 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:44:41.949 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:44:41.949 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:44:41.949 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:44:41.949 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:44:41.949 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:44:41.950 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:44:41.951 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:44:41.951 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:44:42.005 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 30 2024-04-06T09:44:42.005 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 33 2024-04-06T09:44:42.005 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:44:42.005 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=133, tag_tid=21, entry_tid=1], [object_number=132, tag_tid=21, entry_tid=0], [object_number=135, tag_tid=20, entry_tid=19939], [object_number=134, tag_tid=20, entry_tid=19938]]], state=connected] 2024-04-06T09:44:42.005 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=121, tag_tid=20, entry_tid=8773], [object_number=120, tag_tid=20, entry_tid=8772], [object_number=123, tag_tid=20, entry_tid=8771], [object_number=122, tag_tid=20, entry_tid=8770]]], state=connected] 2024-04-06T09:44:42.008 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:44:42.010 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=133, tag_tid=21, entry_tid=1' 2024-04-06T09:44:42.011 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:44:42.011 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:44:42.011 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:44:42.011 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:44:42.011 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:44:42.011 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:44:42.011 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:44:42.011 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:44:42.012 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:44:42.012 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:44:42.012 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:44:42.012 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:44:42.012 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:44:42.012 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:44:42.012 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:44:42.059 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 30 2024-04-06T09:44:42.060 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 33 2024-04-06T09:44:42.060 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:44:42.060 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=133, tag_tid=21, entry_tid=1], [object_number=132, tag_tid=21, entry_tid=0], [object_number=135, tag_tid=20, entry_tid=19939], [object_number=134, tag_tid=20, entry_tid=19938]]], state=connected] 2024-04-06T09:44:42.060 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=121, tag_tid=20, entry_tid=8773], [object_number=120, tag_tid=20, entry_tid=8772], [object_number=123, tag_tid=20, entry_tid=8771], [object_number=122, tag_tid=20, entry_tid=8770]]], state=connected] 2024-04-06T09:44:42.063 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:44:42.065 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=121, tag_tid=20, entry_tid=8773' 2024-04-06T09:44:42.065 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=133, tag_tid=21, entry_tid=1' -a 'object_number=133, tag_tid=21, entry_tid=1' = 'object_number=121, tag_tid=20, entry_tid=8773' 2024-04-06T09:44:42.065 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=121, tag_tid=20, entry_tid=8773' '!=' 'object_number=120, tag_tid=20, entry_tid=8688' 2024-04-06T09:44:42.065 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:44:42.065 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=121, tag_tid=20, entry_tid=8773' = 'object_number=120, tag_tid=20, entry_tid=8688' 2024-04-06T09:44:42.065 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=121, tag_tid=20, entry_tid=8773' 2024-04-06T09:44:42.066 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=133, tag_tid=21, entry_tid=1' 2024-04-06T09:44:42.066 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:44:42.066 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:44:42.066 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=21 2024-04-06T09:44:42.067 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=121, tag_tid=20, entry_tid=8773' 2024-04-06T09:44:42.067 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:44:42.067 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:44:42.068 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=20 2024-04-06T09:44:42.069 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=133, tag_tid=21, entry_tid=1' 2024-04-06T09:44:42.069 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:44:42.069 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:44:42.070 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:44:42.071 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=121, tag_tid=20, entry_tid=8773' 2024-04-06T09:44:42.071 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:44:42.071 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:44:42.072 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=8773 2024-04-06T09:44:42.072 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 21 = 20 -a 1 -le 8773 2024-04-06T09:44:42.072 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:44:42.072 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:44:42.072 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:44:42.274 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:44:42.274 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:44:42.274 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:44:42.274 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:44:42.274 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:44:42.274 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:44:42.274 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:44:42.274 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:44:42.274 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:44:42.274 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:44:42.274 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:44:42.275 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:44:42.275 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:44:42.275 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:44:42.275 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:44:42.275 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:44:42.276 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:44:42.278 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:44:42.278 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:44:42.278 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:44:42.278 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:44:42.278 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:44:42.278 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:44:42.278 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:44:42.278 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:44:42.278 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:44:42.278 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:44:42.278 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:44:42.278 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:44:42.278 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:44:42.279 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:44:42.279 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:44:42.279 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:44:42.280 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:44:42.280 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:44:42.280 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:44:42.280 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:44:42.280 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:44:42.280 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:44:42.280 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:44:42.280 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:44:42.280 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:44:42.281 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:44:42.281 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:44:42.281 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:44:42.281 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:44:42.281 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:44:42.281 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:44:42.281 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:44:42.306 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:44:42.306 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:44:42.306 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:44:42.306 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:44:42.434 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:44:42.434 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:44:42.435 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:44:42.435 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:44:42.435 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:44:42.435 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:44:42.435 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:44:42.435 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:44:42.435 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:44:42.435 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:44:42.436 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:44:42.436 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:44:42.436 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:44:42.438 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:44:42.439 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:44:42.439 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:44:42.490 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 30 2024-04-06T09:44:42.491 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 33 2024-04-06T09:44:42.491 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:44:42.491 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=133, tag_tid=21, entry_tid=1], [object_number=132, tag_tid=21, entry_tid=0], [object_number=135, tag_tid=20, entry_tid=19939], [object_number=134, tag_tid=20, entry_tid=19938]]], state=connected] 2024-04-06T09:44:42.491 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=121, tag_tid=20, entry_tid=8865], [object_number=120, tag_tid=20, entry_tid=8864], [object_number=123, tag_tid=20, entry_tid=8863], [object_number=122, tag_tid=20, entry_tid=8862]]], state=connected] 2024-04-06T09:44:42.494 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:44:42.495 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=133, tag_tid=21, entry_tid=1' 2024-04-06T09:44:42.496 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:44:42.497 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:44:42.497 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:44:42.497 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:44:42.497 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:44:42.497 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:44:42.497 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:44:42.497 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:44:42.497 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:44:42.497 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:44:42.497 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:44:42.497 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:44:42.498 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:44:42.498 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:44:42.498 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:44:42.545 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 30 2024-04-06T09:44:42.545 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 33 2024-04-06T09:44:42.545 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:44:42.545 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=133, tag_tid=21, entry_tid=1], [object_number=132, tag_tid=21, entry_tid=0], [object_number=135, tag_tid=20, entry_tid=19939], [object_number=134, tag_tid=20, entry_tid=19938]]], state=connected] 2024-04-06T09:44:42.545 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=121, tag_tid=20, entry_tid=8865], [object_number=120, tag_tid=20, entry_tid=8864], [object_number=123, tag_tid=20, entry_tid=8863], [object_number=122, tag_tid=20, entry_tid=8862]]], state=connected] 2024-04-06T09:44:42.548 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:44:42.550 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=121, tag_tid=20, entry_tid=8865' 2024-04-06T09:44:42.550 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=133, tag_tid=21, entry_tid=1' -a 'object_number=133, tag_tid=21, entry_tid=1' = 'object_number=121, tag_tid=20, entry_tid=8865' 2024-04-06T09:44:42.550 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=121, tag_tid=20, entry_tid=8865' '!=' 'object_number=121, tag_tid=20, entry_tid=8773' 2024-04-06T09:44:42.550 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:44:42.551 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=121, tag_tid=20, entry_tid=8865' = 'object_number=121, tag_tid=20, entry_tid=8773' 2024-04-06T09:44:42.551 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=121, tag_tid=20, entry_tid=8865' 2024-04-06T09:44:42.551 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=133, tag_tid=21, entry_tid=1' 2024-04-06T09:44:42.551 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:44:42.551 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:44:42.552 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=21 2024-04-06T09:44:42.552 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=121, tag_tid=20, entry_tid=8865' 2024-04-06T09:44:42.553 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:44:42.553 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:44:42.553 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=20 2024-04-06T09:44:42.554 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=133, tag_tid=21, entry_tid=1' 2024-04-06T09:44:42.554 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:44:42.554 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:44:42.555 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:44:42.556 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=121, tag_tid=20, entry_tid=8865' 2024-04-06T09:44:42.556 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:44:42.556 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:44:42.557 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=8865 2024-04-06T09:44:42.557 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 21 = 20 -a 1 -le 8865 2024-04-06T09:44:42.557 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:44:42.557 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:44:42.557 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:44:42.760 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:44:42.760 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:44:42.760 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:44:42.760 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:44:42.760 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:44:42.760 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:44:42.760 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:44:42.760 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:44:42.760 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:44:42.760 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:44:42.760 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:44:42.760 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:44:42.761 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:44:42.761 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:44:42.761 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:44:42.761 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:44:42.761 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:44:42.764 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:44:42.764 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:44:42.764 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:44:42.765 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:44:42.765 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:44:42.765 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:44:42.765 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:44:42.765 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:44:42.765 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:44:42.765 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:44:42.765 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:44:42.765 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:44:42.765 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:44:42.765 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:44:42.765 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:44:42.765 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:44:42.765 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:44:42.766 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:44:42.766 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:44:42.766 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:44:42.766 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:44:42.766 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:44:42.766 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:44:42.766 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:44:42.766 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:44:42.766 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:44:42.766 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:44:42.766 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:44:42.766 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:44:42.766 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:44:42.766 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:44:42.766 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:44:42.790 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:44:42.791 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:44:42.791 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:44:42.791 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:44:42.920 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:44:42.921 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:44:42.921 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:44:42.921 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:44:42.921 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:44:42.921 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:44:42.921 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:44:42.921 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:44:42.921 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:44:42.922 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:44:42.922 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:44:42.922 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:44:42.922 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:44:42.924 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:44:42.924 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:44:42.925 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:44:42.971 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 30 2024-04-06T09:44:42.971 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 33 2024-04-06T09:44:42.972 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:44:42.972 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=133, tag_tid=21, entry_tid=1], [object_number=132, tag_tid=21, entry_tid=0], [object_number=135, tag_tid=20, entry_tid=19939], [object_number=134, tag_tid=20, entry_tid=19938]]], state=connected] 2024-04-06T09:44:42.972 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=123, tag_tid=20, entry_tid=8955], [object_number=122, tag_tid=20, entry_tid=8954], [object_number=121, tag_tid=20, entry_tid=8953], [object_number=120, tag_tid=20, entry_tid=8952]]], state=connected] 2024-04-06T09:44:42.975 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:44:42.976 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=133, tag_tid=21, entry_tid=1' 2024-04-06T09:44:42.977 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:44:42.977 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:44:42.978 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:44:42.978 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:44:42.978 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:44:42.978 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:44:42.978 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:44:42.978 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:44:42.978 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:44:42.978 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:44:42.978 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:44:42.978 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:44:42.979 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:44:42.979 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:44:42.979 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:44:43.027 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 30 2024-04-06T09:44:43.027 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 33 2024-04-06T09:44:43.027 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:44:43.027 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=133, tag_tid=21, entry_tid=1], [object_number=132, tag_tid=21, entry_tid=0], [object_number=135, tag_tid=20, entry_tid=19939], [object_number=134, tag_tid=20, entry_tid=19938]]], state=connected] 2024-04-06T09:44:43.027 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=123, tag_tid=20, entry_tid=8955], [object_number=122, tag_tid=20, entry_tid=8954], [object_number=121, tag_tid=20, entry_tid=8953], [object_number=120, tag_tid=20, entry_tid=8952]]], state=connected] 2024-04-06T09:44:43.031 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:44:43.032 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=123, tag_tid=20, entry_tid=8955' 2024-04-06T09:44:43.032 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=133, tag_tid=21, entry_tid=1' -a 'object_number=133, tag_tid=21, entry_tid=1' = 'object_number=123, tag_tid=20, entry_tid=8955' 2024-04-06T09:44:43.032 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=123, tag_tid=20, entry_tid=8955' '!=' 'object_number=121, tag_tid=20, entry_tid=8865' 2024-04-06T09:44:43.032 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:44:43.033 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=123, tag_tid=20, entry_tid=8955' = 'object_number=121, tag_tid=20, entry_tid=8865' 2024-04-06T09:44:43.033 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=123, tag_tid=20, entry_tid=8955' 2024-04-06T09:44:43.033 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=133, tag_tid=21, entry_tid=1' 2024-04-06T09:44:43.033 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:44:43.033 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:44:43.034 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=21 2024-04-06T09:44:43.035 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=123, tag_tid=20, entry_tid=8955' 2024-04-06T09:44:43.035 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:44:43.035 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:44:43.035 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=20 2024-04-06T09:44:43.036 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=133, tag_tid=21, entry_tid=1' 2024-04-06T09:44:43.036 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:44:43.036 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:44:43.037 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:44:43.038 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=123, tag_tid=20, entry_tid=8955' 2024-04-06T09:44:43.038 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:44:43.038 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:44:43.039 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=8955 2024-04-06T09:44:43.039 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 21 = 20 -a 1 -le 8955 2024-04-06T09:44:43.039 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:44:43.039 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:44:43.039 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:44:43.097 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:44:43.097+0000 7f2e84d66640 -1 reset not still connected to 0x560d2655eea0 2024-04-06T09:44:43.097 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:44:43.097+0000 7f2e84d66640 -1 reset not still connected to 0x560d2655fee0 2024-04-06T09:44:43.097 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:44:43.097+0000 7f2e84d66640 -1 reset not still connected to 0x560d27073930 2024-04-06T09:44:43.097 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:44:43.097+0000 7f2e84d66640 -1 reset not still connected to 0x560d27073ba0 2024-04-06T09:44:43.097 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:44:43.097+0000 7f2e84d66640 -1 reset not still connected to 0x560d27073d40 2024-04-06T09:44:43.241 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:44:43.241 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:44:43.242 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:44:43.242 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:44:43.242 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:44:43.242 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:44:43.242 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:44:43.242 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:44:43.242 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:44:43.242 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:44:43.242 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:44:43.242 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:44:43.242 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:44:43.242 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:44:43.242 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:44:43.244 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:44:43.244 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:44:43.244 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:44:43.244 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:44:43.244 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:44:43.244 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:44:43.244 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:44:43.244 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:44:43.244 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:44:43.245 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:44:43.245 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:44:43.245 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:44:43.245 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:44:43.245 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:44:43.245 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:44:43.245 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:44:43.245 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:44:43.245 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:44:43.246 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:44:43.246 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:44:43.246 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:44:43.247 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:44:43.247 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:44:43.247 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:44:43.247 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:44:43.247 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:44:43.247 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:44:43.247 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:44:43.247 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:44:43.247 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:44:43.247 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:44:43.247 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:44:43.247 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:44:43.247 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:44:43.271 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:44:43.272 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:44:43.272 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:44:43.272 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:44:43.401 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:44:43.402 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:44:43.402 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:44:43.402 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:44:43.402 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:44:43.402 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:44:43.402 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:44:43.403 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:44:43.403 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:44:43.403 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:44:43.403 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:44:43.403 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:44:43.403 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:44:43.404 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:44:43.405 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:44:43.405 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:44:43.456 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 30 2024-04-06T09:44:43.457 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 33 2024-04-06T09:44:43.457 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:44:43.457 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=133, tag_tid=21, entry_tid=1], [object_number=132, tag_tid=21, entry_tid=0], [object_number=135, tag_tid=20, entry_tid=19939], [object_number=134, tag_tid=20, entry_tid=19938]]], state=connected] 2024-04-06T09:44:43.457 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=121, tag_tid=20, entry_tid=9041], [object_number=120, tag_tid=20, entry_tid=9040], [object_number=123, tag_tid=20, entry_tid=9039], [object_number=122, tag_tid=20, entry_tid=9038]]], state=connected] 2024-04-06T09:44:43.460 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:44:43.462 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=133, tag_tid=21, entry_tid=1' 2024-04-06T09:44:43.462 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:44:43.463 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:44:43.463 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:44:43.463 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:44:43.463 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:44:43.463 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:44:43.463 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:44:43.463 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:44:43.463 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:44:43.463 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:44:43.464 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:44:43.464 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:44:43.464 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:44:43.464 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:44:43.465 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:44:43.512 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 30 2024-04-06T09:44:43.512 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 33 2024-04-06T09:44:43.512 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:44:43.512 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=133, tag_tid=21, entry_tid=1], [object_number=132, tag_tid=21, entry_tid=0], [object_number=135, tag_tid=20, entry_tid=19939], [object_number=134, tag_tid=20, entry_tid=19938]]], state=connected] 2024-04-06T09:44:43.512 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=121, tag_tid=20, entry_tid=9041], [object_number=120, tag_tid=20, entry_tid=9040], [object_number=123, tag_tid=20, entry_tid=9039], [object_number=122, tag_tid=20, entry_tid=9038]]], state=connected] 2024-04-06T09:44:43.516 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:44:43.517 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=121, tag_tid=20, entry_tid=9041' 2024-04-06T09:44:43.518 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=133, tag_tid=21, entry_tid=1' -a 'object_number=133, tag_tid=21, entry_tid=1' = 'object_number=121, tag_tid=20, entry_tid=9041' 2024-04-06T09:44:43.518 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=121, tag_tid=20, entry_tid=9041' '!=' 'object_number=123, tag_tid=20, entry_tid=8955' 2024-04-06T09:44:43.518 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:44:43.518 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=121, tag_tid=20, entry_tid=9041' = 'object_number=123, tag_tid=20, entry_tid=8955' 2024-04-06T09:44:43.518 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=121, tag_tid=20, entry_tid=9041' 2024-04-06T09:44:43.519 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=133, tag_tid=21, entry_tid=1' 2024-04-06T09:44:43.519 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:44:43.519 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:44:43.519 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=21 2024-04-06T09:44:43.520 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=121, tag_tid=20, entry_tid=9041' 2024-04-06T09:44:43.520 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:44:43.520 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:44:43.521 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=20 2024-04-06T09:44:43.522 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=133, tag_tid=21, entry_tid=1' 2024-04-06T09:44:43.522 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:44:43.522 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:44:43.522 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:44:43.523 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=121, tag_tid=20, entry_tid=9041' 2024-04-06T09:44:43.523 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:44:43.523 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:44:43.525 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=9041 2024-04-06T09:44:43.525 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 21 = 20 -a 1 -le 9041 2024-04-06T09:44:43.525 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:44:43.525 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:44:43.525 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:44:43.726 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:44:43.727 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:44:43.727 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:44:43.727 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:44:43.727 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:44:43.727 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:44:43.727 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:44:43.727 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:44:43.727 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:44:43.727 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:44:43.727 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:44:43.727 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:44:43.728 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:44:43.728 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:44:43.728 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:44:43.728 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:44:43.728 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:44:43.731 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:44:43.731 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:44:43.731 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:44:43.731 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:44:43.731 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:44:43.731 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:44:43.731 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:44:43.731 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:44:43.731 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:44:43.731 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:44:43.731 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:44:43.731 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:44:43.731 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:44:43.732 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:44:43.732 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:44:43.732 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:44:43.732 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:44:43.732 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:44:43.732 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:44:43.732 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:44:43.732 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:44:43.732 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:44:43.732 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:44:43.732 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:44:43.732 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:44:43.732 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:44:43.732 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:44:43.732 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:44:43.733 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:44:43.733 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:44:43.733 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:44:43.733 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:44:43.756 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:44:43.757 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:44:43.757 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:44:43.757 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:44:43.901 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:44:43.902 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:44:43.902 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:44:43.902 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:44:43.902 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:44:43.903 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:44:43.903 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:44:43.903 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:44:43.903 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:44:43.903 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:44:43.904 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:44:43.904 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:44:43.904 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:44:43.906 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:44:43.906 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:44:43.907 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:44:43.955 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 30 2024-04-06T09:44:43.956 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 33 2024-04-06T09:44:43.956 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:44:43.956 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=133, tag_tid=21, entry_tid=1], [object_number=132, tag_tid=21, entry_tid=0], [object_number=135, tag_tid=20, entry_tid=19939], [object_number=134, tag_tid=20, entry_tid=19938]]], state=connected] 2024-04-06T09:44:43.956 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=123, tag_tid=20, entry_tid=9131], [object_number=122, tag_tid=20, entry_tid=9130], [object_number=121, tag_tid=20, entry_tid=9129], [object_number=120, tag_tid=20, entry_tid=9128]]], state=connected] 2024-04-06T09:44:43.959 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:44:43.960 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=133, tag_tid=21, entry_tid=1' 2024-04-06T09:44:43.961 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:44:43.961 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:44:43.962 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:44:43.962 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:44:43.962 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:44:43.962 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:44:43.962 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:44:43.962 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:44:43.962 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:44:43.962 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:44:43.962 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:44:43.962 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:44:43.963 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:44:43.963 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:44:43.963 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:44:44.009 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 30 2024-04-06T09:44:44.009 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 33 2024-04-06T09:44:44.009 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:44:44.009 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=133, tag_tid=21, entry_tid=1], [object_number=132, tag_tid=21, entry_tid=0], [object_number=135, tag_tid=20, entry_tid=19939], [object_number=134, tag_tid=20, entry_tid=19938]]], state=connected] 2024-04-06T09:44:44.009 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=123, tag_tid=20, entry_tid=9131], [object_number=122, tag_tid=20, entry_tid=9130], [object_number=121, tag_tid=20, entry_tid=9129], [object_number=120, tag_tid=20, entry_tid=9128]]], state=connected] 2024-04-06T09:44:44.013 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:44:44.015 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=123, tag_tid=20, entry_tid=9131' 2024-04-06T09:44:44.015 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=133, tag_tid=21, entry_tid=1' -a 'object_number=133, tag_tid=21, entry_tid=1' = 'object_number=123, tag_tid=20, entry_tid=9131' 2024-04-06T09:44:44.015 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=123, tag_tid=20, entry_tid=9131' '!=' 'object_number=121, tag_tid=20, entry_tid=9041' 2024-04-06T09:44:44.015 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:44:44.015 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=123, tag_tid=20, entry_tid=9131' = 'object_number=121, tag_tid=20, entry_tid=9041' 2024-04-06T09:44:44.015 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=123, tag_tid=20, entry_tid=9131' 2024-04-06T09:44:44.015 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=133, tag_tid=21, entry_tid=1' 2024-04-06T09:44:44.015 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:44:44.015 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:44:44.016 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=21 2024-04-06T09:44:44.017 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=123, tag_tid=20, entry_tid=9131' 2024-04-06T09:44:44.017 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:44:44.017 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:44:44.018 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=20 2024-04-06T09:44:44.018 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=133, tag_tid=21, entry_tid=1' 2024-04-06T09:44:44.018 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:44:44.019 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:44:44.019 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:44:44.020 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=123, tag_tid=20, entry_tid=9131' 2024-04-06T09:44:44.020 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:44:44.020 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:44:44.022 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=9131 2024-04-06T09:44:44.022 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 21 = 20 -a 1 -le 9131 2024-04-06T09:44:44.022 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:44:44.022 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:44:44.022 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:44:44.223 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:44:44.223 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:44:44.224 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:44:44.224 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:44:44.224 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:44:44.224 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:44:44.224 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:44:44.224 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:44:44.224 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:44:44.224 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:44:44.224 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:44:44.224 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:44:44.224 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:44:44.224 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:44:44.224 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:44:44.225 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:44:44.225 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:44:44.229 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:44:44.229 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:44:44.229 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:44:44.229 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:44:44.230 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:44:44.230 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:44:44.230 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:44:44.230 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:44:44.230 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:44:44.230 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:44:44.230 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:44:44.230 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:44:44.230 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:44:44.230 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:44:44.230 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:44:44.230 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:44:44.231 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:44:44.231 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:44:44.231 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:44:44.231 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:44:44.231 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:44:44.231 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:44:44.231 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:44:44.231 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:44:44.231 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:44:44.231 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:44:44.231 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:44:44.231 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:44:44.232 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:44:44.232 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:44:44.232 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:44:44.232 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:44:44.254 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:44:44.255 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:44:44.255 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:44:44.255 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:44:44.390 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:44:44.390 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:44:44.390 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:44:44.390 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:44:44.390 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:44:44.390 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:44:44.391 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:44:44.391 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:44:44.391 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:44:44.391 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:44:44.391 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:44:44.391 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:44:44.391 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:44:44.393 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:44:44.393 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:44:44.393 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:44:44.450 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 30 2024-04-06T09:44:44.450 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 33 2024-04-06T09:44:44.450 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:44:44.450 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=133, tag_tid=21, entry_tid=1], [object_number=132, tag_tid=21, entry_tid=0], [object_number=135, tag_tid=20, entry_tid=19939], [object_number=134, tag_tid=20, entry_tid=19938]]], state=connected] 2024-04-06T09:44:44.450 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=123, tag_tid=20, entry_tid=9219], [object_number=122, tag_tid=20, entry_tid=9218], [object_number=121, tag_tid=20, entry_tid=9217], [object_number=120, tag_tid=20, entry_tid=9216]]], state=connected] 2024-04-06T09:44:44.453 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:44:44.455 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=133, tag_tid=21, entry_tid=1' 2024-04-06T09:44:44.457 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:44:44.457 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:44:44.457 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:44:44.457 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:44:44.457 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:44:44.457 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:44:44.457 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:44:44.457 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:44:44.457 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:44:44.457 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:44:44.457 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:44:44.458 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:44:44.458 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:44:44.458 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:44:44.458 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:44:44.506 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 30 2024-04-06T09:44:44.506 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 33 2024-04-06T09:44:44.506 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:44:44.506 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=133, tag_tid=21, entry_tid=1], [object_number=132, tag_tid=21, entry_tid=0], [object_number=135, tag_tid=20, entry_tid=19939], [object_number=134, tag_tid=20, entry_tid=19938]]], state=connected] 2024-04-06T09:44:44.506 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=123, tag_tid=20, entry_tid=9219], [object_number=122, tag_tid=20, entry_tid=9218], [object_number=121, tag_tid=20, entry_tid=9217], [object_number=120, tag_tid=20, entry_tid=9216]]], state=connected] 2024-04-06T09:44:44.510 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:44:44.511 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=123, tag_tid=20, entry_tid=9219' 2024-04-06T09:44:44.512 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=133, tag_tid=21, entry_tid=1' -a 'object_number=133, tag_tid=21, entry_tid=1' = 'object_number=123, tag_tid=20, entry_tid=9219' 2024-04-06T09:44:44.512 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=123, tag_tid=20, entry_tid=9219' '!=' 'object_number=123, tag_tid=20, entry_tid=9131' 2024-04-06T09:44:44.512 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:44:44.512 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=123, tag_tid=20, entry_tid=9219' = 'object_number=123, tag_tid=20, entry_tid=9131' 2024-04-06T09:44:44.512 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=123, tag_tid=20, entry_tid=9219' 2024-04-06T09:44:44.512 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=133, tag_tid=21, entry_tid=1' 2024-04-06T09:44:44.512 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:44:44.513 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:44:44.513 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=21 2024-04-06T09:44:44.514 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=123, tag_tid=20, entry_tid=9219' 2024-04-06T09:44:44.514 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:44:44.514 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:44:44.515 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=20 2024-04-06T09:44:44.515 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=133, tag_tid=21, entry_tid=1' 2024-04-06T09:44:44.515 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:44:44.516 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:44:44.516 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:44:44.517 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=123, tag_tid=20, entry_tid=9219' 2024-04-06T09:44:44.517 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:44:44.517 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:44:44.518 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=9219 2024-04-06T09:44:44.518 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 21 = 20 -a 1 -le 9219 2024-04-06T09:44:44.518 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:44:44.519 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:44:44.519 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:44:44.720 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:44:44.720 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:44:44.721 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:44:44.721 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:44:44.721 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:44:44.721 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:44:44.721 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:44:44.721 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:44:44.721 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:44:44.721 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:44:44.721 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:44:44.721 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:44:44.721 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:44:44.721 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:44:44.721 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:44:44.722 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:44:44.722 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:44:44.724 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:44:44.724 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:44:44.724 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:44:44.724 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:44:44.724 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:44:44.724 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:44:44.724 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:44:44.724 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:44:44.724 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:44:44.724 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:44:44.724 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:44:44.724 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:44:44.724 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:44:44.725 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:44:44.725 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:44:44.725 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:44:44.726 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:44:44.726 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:44:44.726 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:44:44.726 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:44:44.726 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:44:44.726 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:44:44.726 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:44:44.726 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:44:44.726 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:44:44.726 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:44:44.726 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:44:44.727 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:44:44.727 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:44:44.727 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:44:44.727 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:44:44.727 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:44:44.751 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:44:44.752 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:44:44.752 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:44:44.752 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:44:44.881 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:44:44.882 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:44:44.882 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:44:44.882 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:44:44.882 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:44:44.882 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:44:44.882 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:44:44.882 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:44:44.883 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:44:44.883 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:44:44.883 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:44:44.883 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:44:44.883 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:44:44.886 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:44:44.886 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:44:44.887 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:44:44.937 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 30 2024-04-06T09:44:44.938 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 33 2024-04-06T09:44:44.938 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:44:44.938 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=133, tag_tid=21, entry_tid=1], [object_number=132, tag_tid=21, entry_tid=0], [object_number=135, tag_tid=20, entry_tid=19939], [object_number=134, tag_tid=20, entry_tid=19938]]], state=connected] 2024-04-06T09:44:44.938 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=121, tag_tid=20, entry_tid=9317], [object_number=120, tag_tid=20, entry_tid=9316], [object_number=123, tag_tid=20, entry_tid=9315], [object_number=122, tag_tid=20, entry_tid=9314]]], state=connected] 2024-04-06T09:44:44.941 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:44:44.943 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=133, tag_tid=21, entry_tid=1' 2024-04-06T09:44:44.944 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:44:44.944 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:44:44.944 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:44:44.944 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:44:44.944 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:44:44.944 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:44:44.944 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:44:44.944 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:44:44.944 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:44:44.944 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:44:44.945 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:44:44.945 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:44:44.945 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:44:44.945 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:44:44.945 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:44:44.992 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 30 2024-04-06T09:44:44.993 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 33 2024-04-06T09:44:44.993 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:44:44.993 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=133, tag_tid=21, entry_tid=1], [object_number=132, tag_tid=21, entry_tid=0], [object_number=135, tag_tid=20, entry_tid=19939], [object_number=134, tag_tid=20, entry_tid=19938]]], state=connected] 2024-04-06T09:44:44.993 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=121, tag_tid=20, entry_tid=9317], [object_number=120, tag_tid=20, entry_tid=9316], [object_number=123, tag_tid=20, entry_tid=9315], [object_number=122, tag_tid=20, entry_tid=9314]]], state=connected] 2024-04-06T09:44:44.996 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:44:44.998 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=121, tag_tid=20, entry_tid=9317' 2024-04-06T09:44:44.998 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=133, tag_tid=21, entry_tid=1' -a 'object_number=133, tag_tid=21, entry_tid=1' = 'object_number=121, tag_tid=20, entry_tid=9317' 2024-04-06T09:44:44.998 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=121, tag_tid=20, entry_tid=9317' '!=' 'object_number=123, tag_tid=20, entry_tid=9219' 2024-04-06T09:44:44.998 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:44:44.998 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=121, tag_tid=20, entry_tid=9317' = 'object_number=123, tag_tid=20, entry_tid=9219' 2024-04-06T09:44:44.999 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=121, tag_tid=20, entry_tid=9317' 2024-04-06T09:44:44.999 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=133, tag_tid=21, entry_tid=1' 2024-04-06T09:44:44.999 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:44:44.999 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:44:44.999 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=21 2024-04-06T09:44:45.000 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=121, tag_tid=20, entry_tid=9317' 2024-04-06T09:44:45.000 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:44:45.000 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:44:45.001 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=20 2024-04-06T09:44:45.002 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=133, tag_tid=21, entry_tid=1' 2024-04-06T09:44:45.002 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:44:45.002 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:44:45.002 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:44:45.003 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=121, tag_tid=20, entry_tid=9317' 2024-04-06T09:44:45.003 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:44:45.003 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:44:45.004 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=9317 2024-04-06T09:44:45.005 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 21 = 20 -a 1 -le 9317 2024-04-06T09:44:45.005 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:44:45.005 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:44:45.005 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:44:45.206 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:44:45.206 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:44:45.207 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:44:45.207 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:44:45.207 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:44:45.207 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:44:45.207 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:44:45.207 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:44:45.207 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:44:45.207 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:44:45.207 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:44:45.207 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:44:45.207 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:44:45.207 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:44:45.207 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:44:45.208 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:44:45.208 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:44:45.211 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:44:45.211 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:44:45.211 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:44:45.211 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:44:45.211 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:44:45.211 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:44:45.211 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:44:45.211 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:44:45.211 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:44:45.211 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:44:45.211 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:44:45.211 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:44:45.212 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:44:45.212 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:44:45.212 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:44:45.212 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:44:45.212 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:44:45.212 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:44:45.212 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:44:45.212 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:44:45.212 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:44:45.212 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:44:45.212 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:44:45.212 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:44:45.212 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:44:45.212 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:44:45.212 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:44:45.212 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:44:45.213 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:44:45.213 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:44:45.213 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:44:45.213 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:44:45.238 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:44:45.238 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:44:45.238 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:44:45.238 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:44:45.370 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:44:45.372 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:44:45.372 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:44:45.372 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:44:45.372 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:44:45.372 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:44:45.372 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:44:45.372 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:44:45.372 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:44:45.373 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:44:45.373 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:44:45.373 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:44:45.373 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:44:45.375 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:44:45.375 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:44:45.375 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:44:45.433 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 30 2024-04-06T09:44:45.433 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 33 2024-04-06T09:44:45.434 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:44:45.434 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=133, tag_tid=21, entry_tid=1], [object_number=132, tag_tid=21, entry_tid=0], [object_number=135, tag_tid=20, entry_tid=19939], [object_number=134, tag_tid=20, entry_tid=19938]]], state=connected] 2024-04-06T09:44:45.434 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=120, tag_tid=20, entry_tid=9396], [object_number=123, tag_tid=20, entry_tid=9395], [object_number=122, tag_tid=20, entry_tid=9394], [object_number=121, tag_tid=20, entry_tid=9393]]], state=connected] 2024-04-06T09:44:45.437 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:44:45.438 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=133, tag_tid=21, entry_tid=1' 2024-04-06T09:44:45.439 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:44:45.439 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:44:45.439 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:44:45.439 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:44:45.439 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:44:45.440 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:44:45.440 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:44:45.440 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:44:45.440 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:44:45.440 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:44:45.440 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:44:45.440 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:44:45.441 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:44:45.441 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:44:45.441 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:44:45.489 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 30 2024-04-06T09:44:45.489 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 33 2024-04-06T09:44:45.489 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:44:45.489 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=133, tag_tid=21, entry_tid=1], [object_number=132, tag_tid=21, entry_tid=0], [object_number=135, tag_tid=20, entry_tid=19939], [object_number=134, tag_tid=20, entry_tid=19938]]], state=connected] 2024-04-06T09:44:45.489 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=120, tag_tid=20, entry_tid=9396], [object_number=123, tag_tid=20, entry_tid=9395], [object_number=122, tag_tid=20, entry_tid=9394], [object_number=121, tag_tid=20, entry_tid=9393]]], state=connected] 2024-04-06T09:44:45.492 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:44:45.494 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=120, tag_tid=20, entry_tid=9396' 2024-04-06T09:44:45.494 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=133, tag_tid=21, entry_tid=1' -a 'object_number=133, tag_tid=21, entry_tid=1' = 'object_number=120, tag_tid=20, entry_tid=9396' 2024-04-06T09:44:45.494 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=120, tag_tid=20, entry_tid=9396' '!=' 'object_number=121, tag_tid=20, entry_tid=9317' 2024-04-06T09:44:45.494 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:44:45.494 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=120, tag_tid=20, entry_tid=9396' = 'object_number=121, tag_tid=20, entry_tid=9317' 2024-04-06T09:44:45.494 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=120, tag_tid=20, entry_tid=9396' 2024-04-06T09:44:45.495 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=133, tag_tid=21, entry_tid=1' 2024-04-06T09:44:45.495 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:44:45.495 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:44:45.495 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=21 2024-04-06T09:44:45.496 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=120, tag_tid=20, entry_tid=9396' 2024-04-06T09:44:45.496 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:44:45.496 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:44:45.497 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=20 2024-04-06T09:44:45.498 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=133, tag_tid=21, entry_tid=1' 2024-04-06T09:44:45.498 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:44:45.498 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:44:45.499 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:44:45.500 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=120, tag_tid=20, entry_tid=9396' 2024-04-06T09:44:45.500 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:44:45.500 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:44:45.501 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=9396 2024-04-06T09:44:45.501 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 21 = 20 -a 1 -le 9396 2024-04-06T09:44:45.501 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:44:45.501 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:44:45.501 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:44:45.703 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:44:45.703 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:44:45.703 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:44:45.703 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:44:45.703 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:44:45.703 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:44:45.703 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:44:45.703 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:44:45.703 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:44:45.703 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:44:45.703 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:44:45.703 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:44:45.704 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:44:45.704 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:44:45.704 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:44:45.704 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:44:45.704 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:44:45.705 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:44:45.706 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:44:45.706 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:44:45.706 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:44:45.706 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:44:45.706 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:44:45.706 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:44:45.706 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:44:45.706 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:44:45.706 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:44:45.706 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:44:45.706 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:44:45.706 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:44:45.706 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:44:45.706 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:44:45.707 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:44:45.708 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:44:45.708 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:44:45.708 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:44:45.708 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:44:45.708 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:44:45.708 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:44:45.708 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:44:45.708 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:44:45.708 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:44:45.708 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:44:45.708 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:44:45.709 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:44:45.709 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:44:45.709 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:44:45.709 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:44:45.709 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:44:45.732 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:44:45.733 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:44:45.733 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:44:45.733 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:44:45.860 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:44:45.862 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:44:45.862 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:44:45.862 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:44:45.862 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:44:45.862 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:44:45.862 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:44:45.862 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:44:45.862 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:44:45.862 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:44:45.863 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:44:45.863 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:44:45.863 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:44:45.864 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:44:45.864 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:44:45.864 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:44:45.919 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 30 2024-04-06T09:44:45.920 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 33 2024-04-06T09:44:45.920 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:44:45.920 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=133, tag_tid=21, entry_tid=1], [object_number=132, tag_tid=21, entry_tid=0], [object_number=135, tag_tid=20, entry_tid=19939], [object_number=134, tag_tid=20, entry_tid=19938]]], state=connected] 2024-04-06T09:44:45.920 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=123, tag_tid=20, entry_tid=9479], [object_number=122, tag_tid=20, entry_tid=9478], [object_number=121, tag_tid=20, entry_tid=9477], [object_number=120, tag_tid=20, entry_tid=9476]]], state=connected] 2024-04-06T09:44:45.923 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:44:45.924 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=133, tag_tid=21, entry_tid=1' 2024-04-06T09:44:45.925 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:44:45.925 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:44:45.926 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:44:45.926 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:44:45.926 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:44:45.926 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:44:45.926 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:44:45.926 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:44:45.926 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:44:45.926 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:44:45.926 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:44:45.926 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:44:45.927 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:44:45.927 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:44:45.927 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:44:45.976 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 30 2024-04-06T09:44:45.976 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 33 2024-04-06T09:44:45.976 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:44:45.976 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=133, tag_tid=21, entry_tid=1], [object_number=132, tag_tid=21, entry_tid=0], [object_number=135, tag_tid=20, entry_tid=19939], [object_number=134, tag_tid=20, entry_tid=19938]]], state=connected] 2024-04-06T09:44:45.976 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=123, tag_tid=20, entry_tid=9479], [object_number=122, tag_tid=20, entry_tid=9478], [object_number=121, tag_tid=20, entry_tid=9477], [object_number=120, tag_tid=20, entry_tid=9476]]], state=connected] 2024-04-06T09:44:45.979 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:44:45.981 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=123, tag_tid=20, entry_tid=9479' 2024-04-06T09:44:45.981 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=133, tag_tid=21, entry_tid=1' -a 'object_number=133, tag_tid=21, entry_tid=1' = 'object_number=123, tag_tid=20, entry_tid=9479' 2024-04-06T09:44:45.981 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=123, tag_tid=20, entry_tid=9479' '!=' 'object_number=120, tag_tid=20, entry_tid=9396' 2024-04-06T09:44:45.981 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:44:45.982 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=123, tag_tid=20, entry_tid=9479' = 'object_number=120, tag_tid=20, entry_tid=9396' 2024-04-06T09:44:45.982 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=123, tag_tid=20, entry_tid=9479' 2024-04-06T09:44:45.982 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=133, tag_tid=21, entry_tid=1' 2024-04-06T09:44:45.982 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:44:45.982 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:44:45.982 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=21 2024-04-06T09:44:45.983 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=123, tag_tid=20, entry_tid=9479' 2024-04-06T09:44:45.983 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:44:45.983 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:44:45.984 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=20 2024-04-06T09:44:45.985 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=133, tag_tid=21, entry_tid=1' 2024-04-06T09:44:45.985 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:44:45.985 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:44:45.986 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:44:45.986 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=123, tag_tid=20, entry_tid=9479' 2024-04-06T09:44:45.986 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:44:45.987 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:44:45.988 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=9479 2024-04-06T09:44:45.988 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 21 = 20 -a 1 -le 9479 2024-04-06T09:44:45.988 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:44:45.988 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:44:45.988 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:44:46.190 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:44:46.190 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:44:46.190 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:44:46.190 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:44:46.190 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:44:46.190 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:44:46.190 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:44:46.190 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:44:46.190 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:44:46.190 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:44:46.190 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:44:46.190 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:44:46.191 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:44:46.191 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:44:46.191 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:44:46.191 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:44:46.191 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:44:46.194 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:44:46.194 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:44:46.194 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:44:46.194 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:44:46.194 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:44:46.194 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:44:46.194 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:44:46.194 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:44:46.194 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:44:46.195 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:44:46.195 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:44:46.195 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:44:46.195 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:44:46.195 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:44:46.195 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:44:46.195 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:44:46.196 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:44:46.196 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:44:46.196 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:44:46.196 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:44:46.196 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:44:46.196 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:44:46.196 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:44:46.196 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:44:46.197 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:44:46.197 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:44:46.197 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:44:46.197 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:44:46.197 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:44:46.197 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:44:46.197 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:44:46.197 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:44:46.221 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:44:46.222 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:44:46.222 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:44:46.222 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:44:46.359 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:44:46.360 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:44:46.360 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:44:46.360 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:44:46.360 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:44:46.360 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:44:46.360 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:44:46.360 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:44:46.360 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:44:46.360 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:44:46.361 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:44:46.361 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:44:46.361 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:44:46.363 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:44:46.363 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:44:46.364 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:44:46.410 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 30 2024-04-06T09:44:46.410 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 33 2024-04-06T09:44:46.410 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:44:46.410 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=133, tag_tid=21, entry_tid=1], [object_number=132, tag_tid=21, entry_tid=0], [object_number=135, tag_tid=20, entry_tid=19939], [object_number=134, tag_tid=20, entry_tid=19938]]], state=connected] 2024-04-06T09:44:46.411 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=121, tag_tid=20, entry_tid=9565], [object_number=120, tag_tid=20, entry_tid=9564], [object_number=123, tag_tid=20, entry_tid=9563], [object_number=122, tag_tid=20, entry_tid=9562]]], state=connected] 2024-04-06T09:44:46.414 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:44:46.415 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=133, tag_tid=21, entry_tid=1' 2024-04-06T09:44:46.416 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:44:46.416 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:44:46.416 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:44:46.416 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:44:46.416 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:44:46.416 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:44:46.416 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:44:46.417 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:44:46.417 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:44:46.417 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:44:46.417 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:44:46.417 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:44:46.418 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:44:46.418 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:44:46.418 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:44:46.465 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 30 2024-04-06T09:44:46.465 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 33 2024-04-06T09:44:46.465 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:44:46.465 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=133, tag_tid=21, entry_tid=1], [object_number=132, tag_tid=21, entry_tid=0], [object_number=135, tag_tid=20, entry_tid=19939], [object_number=134, tag_tid=20, entry_tid=19938]]], state=connected] 2024-04-06T09:44:46.465 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=121, tag_tid=20, entry_tid=9565], [object_number=120, tag_tid=20, entry_tid=9564], [object_number=123, tag_tid=20, entry_tid=9563], [object_number=122, tag_tid=20, entry_tid=9562]]], state=connected] 2024-04-06T09:44:46.468 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:44:46.470 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=121, tag_tid=20, entry_tid=9565' 2024-04-06T09:44:46.470 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=133, tag_tid=21, entry_tid=1' -a 'object_number=133, tag_tid=21, entry_tid=1' = 'object_number=121, tag_tid=20, entry_tid=9565' 2024-04-06T09:44:46.470 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=121, tag_tid=20, entry_tid=9565' '!=' 'object_number=123, tag_tid=20, entry_tid=9479' 2024-04-06T09:44:46.470 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:44:46.470 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=121, tag_tid=20, entry_tid=9565' = 'object_number=123, tag_tid=20, entry_tid=9479' 2024-04-06T09:44:46.470 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=121, tag_tid=20, entry_tid=9565' 2024-04-06T09:44:46.471 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=133, tag_tid=21, entry_tid=1' 2024-04-06T09:44:46.471 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:44:46.471 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:44:46.471 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=21 2024-04-06T09:44:46.472 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=121, tag_tid=20, entry_tid=9565' 2024-04-06T09:44:46.472 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:44:46.472 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:44:46.473 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=20 2024-04-06T09:44:46.474 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=133, tag_tid=21, entry_tid=1' 2024-04-06T09:44:46.474 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:44:46.474 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:44:46.475 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:44:46.475 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=121, tag_tid=20, entry_tid=9565' 2024-04-06T09:44:46.475 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:44:46.476 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:44:46.477 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=9565 2024-04-06T09:44:46.477 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 21 = 20 -a 1 -le 9565 2024-04-06T09:44:46.477 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:44:46.477 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:44:46.477 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:44:46.679 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:44:46.679 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:44:46.679 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:44:46.679 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:44:46.679 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:44:46.679 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:44:46.679 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:44:46.679 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:44:46.679 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:44:46.679 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:44:46.679 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:44:46.680 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:44:46.680 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:44:46.680 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:44:46.680 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:44:46.680 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:44:46.681 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:44:46.683 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:44:46.683 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:44:46.683 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:44:46.683 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:44:46.683 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:44:46.683 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:44:46.684 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:44:46.684 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:44:46.684 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:44:46.684 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:44:46.684 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:44:46.684 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:44:46.684 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:44:46.684 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:44:46.684 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:44:46.684 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:44:46.685 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:44:46.685 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:44:46.685 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:44:46.685 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:44:46.685 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:44:46.685 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:44:46.686 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:44:46.686 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:44:46.686 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:44:46.686 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:44:46.686 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:44:46.686 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:44:46.686 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:44:46.686 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:44:46.686 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:44:46.686 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:44:46.712 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:44:46.712 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:44:46.712 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:44:46.712 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:44:46.840 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:44:46.841 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:44:46.841 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:44:46.841 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:44:46.841 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:44:46.841 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:44:46.842 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:44:46.842 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:44:46.842 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:44:46.842 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:44:46.842 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:44:46.842 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:44:46.843 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:44:46.844 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:44:46.844 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:44:46.844 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:44:46.896 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 30 2024-04-06T09:44:46.896 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 33 2024-04-06T09:44:46.896 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:44:46.896 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=133, tag_tid=21, entry_tid=1], [object_number=132, tag_tid=21, entry_tid=0], [object_number=135, tag_tid=20, entry_tid=19939], [object_number=134, tag_tid=20, entry_tid=19938]]], state=connected] 2024-04-06T09:44:46.896 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=121, tag_tid=20, entry_tid=9649], [object_number=120, tag_tid=20, entry_tid=9648], [object_number=123, tag_tid=20, entry_tid=9647], [object_number=122, tag_tid=20, entry_tid=9646]]], state=connected] 2024-04-06T09:44:46.899 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:44:46.901 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=133, tag_tid=21, entry_tid=1' 2024-04-06T09:44:46.902 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:44:46.902 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:44:46.902 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:44:46.902 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:44:46.902 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:44:46.902 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:44:46.903 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:44:46.903 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:44:46.903 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:44:46.903 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:44:46.903 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:44:46.903 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:44:46.903 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:44:46.904 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:44:46.904 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:44:47.158 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 30 2024-04-06T09:44:47.158 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 33 2024-04-06T09:44:47.159 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:44:47.159 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=133, tag_tid=21, entry_tid=1], [object_number=132, tag_tid=21, entry_tid=0], [object_number=135, tag_tid=20, entry_tid=19939], [object_number=134, tag_tid=20, entry_tid=19938]]], state=connected] 2024-04-06T09:44:47.159 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=121, tag_tid=20, entry_tid=9649], [object_number=120, tag_tid=20, entry_tid=9648], [object_number=123, tag_tid=20, entry_tid=9647], [object_number=122, tag_tid=20, entry_tid=9646]]], state=connected] 2024-04-06T09:44:47.164 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:44:47.166 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=121, tag_tid=20, entry_tid=9649' 2024-04-06T09:44:47.166 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=133, tag_tid=21, entry_tid=1' -a 'object_number=133, tag_tid=21, entry_tid=1' = 'object_number=121, tag_tid=20, entry_tid=9649' 2024-04-06T09:44:47.166 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=121, tag_tid=20, entry_tid=9649' '!=' 'object_number=121, tag_tid=20, entry_tid=9565' 2024-04-06T09:44:47.166 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:44:47.166 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=121, tag_tid=20, entry_tid=9649' = 'object_number=121, tag_tid=20, entry_tid=9565' 2024-04-06T09:44:47.166 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=121, tag_tid=20, entry_tid=9649' 2024-04-06T09:44:47.167 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=133, tag_tid=21, entry_tid=1' 2024-04-06T09:44:47.167 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:44:47.167 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:44:47.168 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=21 2024-04-06T09:44:47.169 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=121, tag_tid=20, entry_tid=9649' 2024-04-06T09:44:47.169 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:44:47.170 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:44:47.170 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=20 2024-04-06T09:44:47.171 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=133, tag_tid=21, entry_tid=1' 2024-04-06T09:44:47.172 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:44:47.172 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:44:47.172 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:44:47.173 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=121, tag_tid=20, entry_tid=9649' 2024-04-06T09:44:47.173 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:44:47.174 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:44:47.175 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=9649 2024-04-06T09:44:47.175 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 21 = 20 -a 1 -le 9649 2024-04-06T09:44:47.175 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:44:47.175 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:44:47.175 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:44:47.377 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:44:47.378 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:44:47.378 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:44:47.378 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:44:47.378 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:44:47.378 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:44:47.378 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:44:47.378 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:44:47.378 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:44:47.378 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:44:47.378 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:44:47.378 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:44:47.379 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:44:47.379 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:44:47.379 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:44:47.379 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:44:47.379 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:44:47.381 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:44:47.382 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:44:47.382 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:44:47.382 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:44:47.382 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:44:47.382 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:44:47.382 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:44:47.382 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:44:47.382 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:44:47.382 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:44:47.382 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:44:47.382 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:44:47.383 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:44:47.383 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:44:47.383 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:44:47.383 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:44:47.384 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:44:47.384 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:44:47.384 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:44:47.384 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:44:47.384 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:44:47.384 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:44:47.384 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:44:47.384 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:44:47.384 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:44:47.385 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:44:47.385 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:44:47.385 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:44:47.385 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:44:47.385 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:44:47.385 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:44:47.385 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:44:47.409 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:44:47.410 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:44:47.410 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:44:47.410 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:44:47.542 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:44:47.543 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:44:47.543 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:44:47.543 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:44:47.543 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:44:47.543 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:44:47.543 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:44:47.543 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:44:47.543 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:44:47.543 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:44:47.544 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:44:47.544 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:44:47.544 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:44:47.545 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:44:47.546 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:44:47.546 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:44:47.597 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 30 2024-04-06T09:44:47.597 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 33 2024-04-06T09:44:47.597 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:44:47.597 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=133, tag_tid=21, entry_tid=1], [object_number=132, tag_tid=21, entry_tid=0], [object_number=135, tag_tid=20, entry_tid=19939], [object_number=134, tag_tid=20, entry_tid=19938]]], state=connected] 2024-04-06T09:44:47.597 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=122, tag_tid=20, entry_tid=9762], [object_number=121, tag_tid=20, entry_tid=9761], [object_number=120, tag_tid=20, entry_tid=9760], [object_number=123, tag_tid=20, entry_tid=9759]]], state=connected] 2024-04-06T09:44:47.600 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:44:47.602 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=133, tag_tid=21, entry_tid=1' 2024-04-06T09:44:47.603 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:44:47.603 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:44:47.603 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:44:47.603 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:44:47.603 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:44:47.603 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:44:47.604 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:44:47.604 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:44:47.604 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:44:47.604 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:44:47.604 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:44:47.604 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:44:47.604 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:44:47.605 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:44:47.605 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:44:47.651 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 30 2024-04-06T09:44:47.651 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 33 2024-04-06T09:44:47.651 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:44:47.651 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=133, tag_tid=21, entry_tid=1], [object_number=132, tag_tid=21, entry_tid=0], [object_number=135, tag_tid=20, entry_tid=19939], [object_number=134, tag_tid=20, entry_tid=19938]]], state=connected] 2024-04-06T09:44:47.651 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=122, tag_tid=20, entry_tid=9762], [object_number=121, tag_tid=20, entry_tid=9761], [object_number=120, tag_tid=20, entry_tid=9760], [object_number=123, tag_tid=20, entry_tid=9759]]], state=connected] 2024-04-06T09:44:47.655 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:44:47.657 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=122, tag_tid=20, entry_tid=9762' 2024-04-06T09:44:47.657 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=133, tag_tid=21, entry_tid=1' -a 'object_number=133, tag_tid=21, entry_tid=1' = 'object_number=122, tag_tid=20, entry_tid=9762' 2024-04-06T09:44:47.657 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=122, tag_tid=20, entry_tid=9762' '!=' 'object_number=121, tag_tid=20, entry_tid=9649' 2024-04-06T09:44:47.657 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:44:47.657 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=122, tag_tid=20, entry_tid=9762' = 'object_number=121, tag_tid=20, entry_tid=9649' 2024-04-06T09:44:47.657 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=122, tag_tid=20, entry_tid=9762' 2024-04-06T09:44:47.657 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=133, tag_tid=21, entry_tid=1' 2024-04-06T09:44:47.657 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:44:47.657 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:44:47.658 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=21 2024-04-06T09:44:47.658 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=122, tag_tid=20, entry_tid=9762' 2024-04-06T09:44:47.659 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:44:47.659 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:44:47.659 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=20 2024-04-06T09:44:47.660 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=133, tag_tid=21, entry_tid=1' 2024-04-06T09:44:47.660 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:44:47.660 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:44:47.661 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:44:47.662 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=122, tag_tid=20, entry_tid=9762' 2024-04-06T09:44:47.662 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:44:47.662 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:44:47.663 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=9762 2024-04-06T09:44:47.663 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 21 = 20 -a 1 -le 9762 2024-04-06T09:44:47.663 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:44:47.663 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:44:47.664 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:44:47.865 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:44:47.865 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:44:47.865 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:44:47.865 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:44:47.866 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:44:47.866 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:44:47.866 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:44:47.866 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:44:47.866 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:44:47.866 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:44:47.866 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:44:47.866 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:44:47.866 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:44:47.866 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:44:47.866 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:44:47.867 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:44:47.867 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:44:47.870 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:44:47.870 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:44:47.870 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:44:47.870 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:44:47.870 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:44:47.870 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:44:47.871 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:44:47.871 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:44:47.871 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:44:47.871 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:44:47.871 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:44:47.871 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:44:47.871 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:44:47.871 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:44:47.871 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:44:47.871 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:44:47.871 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:44:47.871 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:44:47.871 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:44:47.871 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:44:47.871 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:44:47.872 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:44:47.872 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:44:47.872 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:44:47.872 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:44:47.872 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:44:47.872 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:44:47.872 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:44:47.872 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:44:47.872 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:44:47.872 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:44:47.872 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:44:47.895 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:44:47.896 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:44:47.896 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:44:47.896 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:44:48.027 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:44:48.029 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:44:48.029 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:44:48.029 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:44:48.029 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:44:48.029 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:44:48.029 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:44:48.029 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:44:48.029 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:44:48.030 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:44:48.030 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:44:48.030 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:44:48.030 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:44:48.031 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:44:48.031 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:44:48.032 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:44:48.081 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 30 2024-04-06T09:44:48.081 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 33 2024-04-06T09:44:48.081 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:44:48.082 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=133, tag_tid=21, entry_tid=1], [object_number=132, tag_tid=21, entry_tid=0], [object_number=135, tag_tid=20, entry_tid=19939], [object_number=134, tag_tid=20, entry_tid=19938]]], state=connected] 2024-04-06T09:44:48.082 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=121, tag_tid=20, entry_tid=9845], [object_number=120, tag_tid=20, entry_tid=9844], [object_number=123, tag_tid=20, entry_tid=9843], [object_number=122, tag_tid=20, entry_tid=9842]]], state=connected] 2024-04-06T09:44:48.085 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:44:48.086 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=133, tag_tid=21, entry_tid=1' 2024-04-06T09:44:48.087 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:44:48.088 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:44:48.088 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:44:48.088 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:44:48.088 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:44:48.088 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:44:48.088 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:44:48.088 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:44:48.088 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:44:48.088 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:44:48.088 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:44:48.088 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:44:48.089 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:44:48.089 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:44:48.089 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:44:48.136 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 30 2024-04-06T09:44:48.136 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 33 2024-04-06T09:44:48.136 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:44:48.136 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=133, tag_tid=21, entry_tid=1], [object_number=132, tag_tid=21, entry_tid=0], [object_number=135, tag_tid=20, entry_tid=19939], [object_number=134, tag_tid=20, entry_tid=19938]]], state=connected] 2024-04-06T09:44:48.136 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=121, tag_tid=20, entry_tid=9845], [object_number=120, tag_tid=20, entry_tid=9844], [object_number=123, tag_tid=20, entry_tid=9843], [object_number=122, tag_tid=20, entry_tid=9842]]], state=connected] 2024-04-06T09:44:48.139 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:44:48.142 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=121, tag_tid=20, entry_tid=9845' 2024-04-06T09:44:48.142 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=133, tag_tid=21, entry_tid=1' -a 'object_number=133, tag_tid=21, entry_tid=1' = 'object_number=121, tag_tid=20, entry_tid=9845' 2024-04-06T09:44:48.142 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=121, tag_tid=20, entry_tid=9845' '!=' 'object_number=122, tag_tid=20, entry_tid=9762' 2024-04-06T09:44:48.142 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:44:48.142 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=121, tag_tid=20, entry_tid=9845' = 'object_number=122, tag_tid=20, entry_tid=9762' 2024-04-06T09:44:48.142 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=121, tag_tid=20, entry_tid=9845' 2024-04-06T09:44:48.142 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=133, tag_tid=21, entry_tid=1' 2024-04-06T09:44:48.142 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:44:48.142 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:44:48.143 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=21 2024-04-06T09:44:48.144 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=121, tag_tid=20, entry_tid=9845' 2024-04-06T09:44:48.144 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:44:48.144 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:44:48.145 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=20 2024-04-06T09:44:48.145 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=133, tag_tid=21, entry_tid=1' 2024-04-06T09:44:48.146 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:44:48.146 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:44:48.146 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:44:48.147 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=121, tag_tid=20, entry_tid=9845' 2024-04-06T09:44:48.147 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:44:48.148 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:44:48.149 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=9845 2024-04-06T09:44:48.149 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 21 = 20 -a 1 -le 9845 2024-04-06T09:44:48.149 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:44:48.149 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:44:48.149 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:44:48.350 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:44:48.351 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:44:48.351 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:44:48.351 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:44:48.351 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:44:48.351 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:44:48.351 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:44:48.351 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:44:48.351 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:44:48.351 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:44:48.351 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:44:48.351 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:44:48.351 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:44:48.352 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:44:48.352 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:44:48.352 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:44:48.352 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:44:48.355 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:44:48.356 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:44:48.356 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:44:48.356 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:44:48.356 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:44:48.356 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:44:48.356 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:44:48.356 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:44:48.356 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:44:48.356 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:44:48.356 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:44:48.356 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:44:48.357 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:44:48.357 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:44:48.357 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:44:48.357 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:44:48.357 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:44:48.357 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:44:48.357 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:44:48.357 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:44:48.357 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:44:48.357 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:44:48.357 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:44:48.357 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:44:48.357 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:44:48.358 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:44:48.358 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:44:48.358 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:44:48.358 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:44:48.358 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:44:48.358 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:44:48.358 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:44:48.382 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:44:48.383 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:44:48.383 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:44:48.383 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:44:48.515 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:44:48.516 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:44:48.516 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:44:48.516 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:44:48.516 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:44:48.516 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:44:48.517 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:44:48.517 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:44:48.517 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:44:48.517 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:44:48.517 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:44:48.517 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:44:48.518 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:44:48.522 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:44:48.522 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:44:48.522 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:44:48.566 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 30 2024-04-06T09:44:48.566 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 33 2024-04-06T09:44:48.566 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:44:48.566 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=133, tag_tid=21, entry_tid=1], [object_number=132, tag_tid=21, entry_tid=0], [object_number=135, tag_tid=20, entry_tid=19939], [object_number=134, tag_tid=20, entry_tid=19938]]], state=connected] 2024-04-06T09:44:48.567 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=120, tag_tid=20, entry_tid=9928], [object_number=123, tag_tid=20, entry_tid=9927], [object_number=122, tag_tid=20, entry_tid=9926], [object_number=121, tag_tid=20, entry_tid=9925]]], state=connected] 2024-04-06T09:44:48.569 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:44:48.571 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=133, tag_tid=21, entry_tid=1' 2024-04-06T09:44:48.572 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:44:48.572 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:44:48.572 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:44:48.572 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:44:48.572 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:44:48.572 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:44:48.572 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:44:48.573 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:44:48.573 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:44:48.573 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:44:48.573 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:44:48.573 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:44:48.573 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:44:48.573 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:44:48.574 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:44:48.620 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 30 2024-04-06T09:44:48.620 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 33 2024-04-06T09:44:48.620 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:44:48.620 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=133, tag_tid=21, entry_tid=1], [object_number=132, tag_tid=21, entry_tid=0], [object_number=135, tag_tid=20, entry_tid=19939], [object_number=134, tag_tid=20, entry_tid=19938]]], state=connected] 2024-04-06T09:44:48.621 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=120, tag_tid=20, entry_tid=9928], [object_number=123, tag_tid=20, entry_tid=9927], [object_number=122, tag_tid=20, entry_tid=9926], [object_number=121, tag_tid=20, entry_tid=9925]]], state=connected] 2024-04-06T09:44:48.624 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:44:48.626 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=120, tag_tid=20, entry_tid=9928' 2024-04-06T09:44:48.626 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=133, tag_tid=21, entry_tid=1' -a 'object_number=133, tag_tid=21, entry_tid=1' = 'object_number=120, tag_tid=20, entry_tid=9928' 2024-04-06T09:44:48.626 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=120, tag_tid=20, entry_tid=9928' '!=' 'object_number=121, tag_tid=20, entry_tid=9845' 2024-04-06T09:44:48.626 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:44:48.626 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=120, tag_tid=20, entry_tid=9928' = 'object_number=121, tag_tid=20, entry_tid=9845' 2024-04-06T09:44:48.626 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=120, tag_tid=20, entry_tid=9928' 2024-04-06T09:44:48.626 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=133, tag_tid=21, entry_tid=1' 2024-04-06T09:44:48.626 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:44:48.626 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:44:48.627 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=21 2024-04-06T09:44:48.628 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=120, tag_tid=20, entry_tid=9928' 2024-04-06T09:44:48.628 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:44:48.628 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:44:48.629 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=20 2024-04-06T09:44:48.629 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=133, tag_tid=21, entry_tid=1' 2024-04-06T09:44:48.629 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:44:48.630 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:44:48.630 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:44:48.631 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=120, tag_tid=20, entry_tid=9928' 2024-04-06T09:44:48.631 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:44:48.631 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:44:48.632 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=9928 2024-04-06T09:44:48.632 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 21 = 20 -a 1 -le 9928 2024-04-06T09:44:48.632 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:44:48.633 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:44:48.633 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:44:48.834 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:44:48.834 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:44:48.834 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:44:48.835 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:44:48.835 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:44:48.835 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:44:48.835 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:44:48.835 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:44:48.835 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:44:48.835 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:44:48.835 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:44:48.835 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:44:48.835 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:44:48.835 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:44:48.835 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:44:48.836 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:44:48.836 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:44:48.838 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:44:48.838 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:44:48.838 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:44:48.838 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:44:48.839 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:44:48.839 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:44:48.839 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:44:48.839 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:44:48.839 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:44:48.839 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:44:48.839 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:44:48.839 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:44:48.839 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:44:48.839 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:44:48.839 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:44:48.839 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:44:48.839 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:44:48.839 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:44:48.839 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:44:48.840 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:44:48.840 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:44:48.840 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:44:48.840 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:44:48.840 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:44:48.840 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:44:48.840 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:44:48.840 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:44:48.840 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:44:48.840 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:44:48.840 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:44:48.840 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:44:48.841 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:44:48.865 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:44:48.865 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:44:48.865 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:44:48.866 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:44:49.000 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:44:49.003 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:44:49.003 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:44:49.003 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:44:49.003 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:44:49.003 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:44:49.003 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:44:49.004 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:44:49.004 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:44:49.004 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:44:49.004 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:44:49.004 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:44:49.004 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:44:49.005 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:44:49.005 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:44:49.005 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:44:49.060 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 30 2024-04-06T09:44:49.061 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 33 2024-04-06T09:44:49.061 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:44:49.061 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=133, tag_tid=21, entry_tid=1], [object_number=132, tag_tid=21, entry_tid=0], [object_number=135, tag_tid=20, entry_tid=19939], [object_number=134, tag_tid=20, entry_tid=19938]]], state=connected] 2024-04-06T09:44:49.061 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=123, tag_tid=20, entry_tid=10007], [object_number=122, tag_tid=20, entry_tid=10006], [object_number=121, tag_tid=20, entry_tid=10005], [object_number=120, tag_tid=20, entry_tid=10004]]], state=connected] 2024-04-06T09:44:49.064 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:44:49.065 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=133, tag_tid=21, entry_tid=1' 2024-04-06T09:44:49.068 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:44:49.068 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:44:49.068 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:44:49.069 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:44:49.069 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:44:49.069 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:44:49.069 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:44:49.069 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:44:49.069 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:44:49.069 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:44:49.070 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:44:49.070 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:44:49.070 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:44:49.070 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:44:49.070 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:44:49.115 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 30 2024-04-06T09:44:49.116 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 33 2024-04-06T09:44:49.116 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:44:49.116 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=133, tag_tid=21, entry_tid=1], [object_number=132, tag_tid=21, entry_tid=0], [object_number=135, tag_tid=20, entry_tid=19939], [object_number=134, tag_tid=20, entry_tid=19938]]], state=connected] 2024-04-06T09:44:49.116 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=123, tag_tid=20, entry_tid=10007], [object_number=122, tag_tid=20, entry_tid=10006], [object_number=121, tag_tid=20, entry_tid=10005], [object_number=120, tag_tid=20, entry_tid=10004]]], state=connected] 2024-04-06T09:44:49.119 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:44:49.121 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=123, tag_tid=20, entry_tid=10007' 2024-04-06T09:44:49.121 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=133, tag_tid=21, entry_tid=1' -a 'object_number=133, tag_tid=21, entry_tid=1' = 'object_number=123, tag_tid=20, entry_tid=10007' 2024-04-06T09:44:49.121 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=123, tag_tid=20, entry_tid=10007' '!=' 'object_number=120, tag_tid=20, entry_tid=9928' 2024-04-06T09:44:49.122 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:44:49.122 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=123, tag_tid=20, entry_tid=10007' = 'object_number=120, tag_tid=20, entry_tid=9928' 2024-04-06T09:44:49.122 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=123, tag_tid=20, entry_tid=10007' 2024-04-06T09:44:49.122 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=133, tag_tid=21, entry_tid=1' 2024-04-06T09:44:49.122 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:44:49.122 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:44:49.123 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=21 2024-04-06T09:44:49.123 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=123, tag_tid=20, entry_tid=10007' 2024-04-06T09:44:49.123 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:44:49.123 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:44:49.124 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=20 2024-04-06T09:44:49.125 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=133, tag_tid=21, entry_tid=1' 2024-04-06T09:44:49.125 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:44:49.125 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:44:49.126 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:44:49.126 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=123, tag_tid=20, entry_tid=10007' 2024-04-06T09:44:49.127 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:44:49.127 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:44:49.128 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=10007 2024-04-06T09:44:49.128 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 21 = 20 -a 1 -le 10007 2024-04-06T09:44:49.128 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:44:49.128 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:44:49.129 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:44:49.330 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:44:49.330 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:44:49.330 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:44:49.330 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:44:49.330 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:44:49.330 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:44:49.330 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:44:49.330 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:44:49.330 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:44:49.331 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:44:49.331 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:44:49.331 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:44:49.331 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:44:49.331 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:44:49.331 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:44:49.333 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:44:49.334 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:44:49.334 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:44:49.334 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:44:49.334 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:44:49.334 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:44:49.334 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:44:49.334 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:44:49.334 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:44:49.334 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:44:49.334 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:44:49.335 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:44:49.335 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:44:49.335 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:44:49.335 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:44:49.335 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:44:49.335 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:44:49.335 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:44:49.335 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:44:49.335 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:44:49.335 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:44:49.335 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:44:49.335 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:44:49.336 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:44:49.336 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:44:49.336 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:44:49.336 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:44:49.336 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:44:49.336 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:44:49.336 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:44:49.336 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:44:49.336 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:44:49.336 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:44:49.336 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:44:49.360 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:44:49.360 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:44:49.360 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:44:49.360 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:44:49.561 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:44:49.562 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:44:49.563 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:44:49.563 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:44:49.563 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:44:49.563 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:44:49.563 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:44:49.563 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:44:49.563 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:44:49.563 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:44:49.563 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:44:49.564 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:44:49.564 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:44:49.566 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:44:49.566 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:44:49.566 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:44:49.622 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 30 2024-04-06T09:44:49.623 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 33 2024-04-06T09:44:49.623 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:44:49.623 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=133, tag_tid=21, entry_tid=1], [object_number=132, tag_tid=21, entry_tid=0], [object_number=135, tag_tid=20, entry_tid=19939], [object_number=134, tag_tid=20, entry_tid=19938]]], state=connected] 2024-04-06T09:44:49.623 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=123, tag_tid=20, entry_tid=10083], [object_number=122, tag_tid=20, entry_tid=10082], [object_number=121, tag_tid=20, entry_tid=10081], [object_number=120, tag_tid=20, entry_tid=10080]]], state=connected] 2024-04-06T09:44:49.626 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:44:49.627 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=133, tag_tid=21, entry_tid=1' 2024-04-06T09:44:49.629 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:44:49.629 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:44:49.629 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:44:49.629 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:44:49.629 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:44:49.629 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:44:49.629 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:44:49.629 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:44:49.629 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:44:49.629 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:44:49.629 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:44:49.629 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:44:49.630 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:44:49.631 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:44:49.631 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:44:49.679 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 30 2024-04-06T09:44:49.679 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 33 2024-04-06T09:44:49.679 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:44:49.679 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=133, tag_tid=21, entry_tid=1], [object_number=132, tag_tid=21, entry_tid=0], [object_number=135, tag_tid=20, entry_tid=19939], [object_number=134, tag_tid=20, entry_tid=19938]]], state=connected] 2024-04-06T09:44:49.679 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=123, tag_tid=20, entry_tid=10083], [object_number=122, tag_tid=20, entry_tid=10082], [object_number=121, tag_tid=20, entry_tid=10081], [object_number=120, tag_tid=20, entry_tid=10080]]], state=connected] 2024-04-06T09:44:49.682 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:44:49.684 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=123, tag_tid=20, entry_tid=10083' 2024-04-06T09:44:49.684 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=133, tag_tid=21, entry_tid=1' -a 'object_number=133, tag_tid=21, entry_tid=1' = 'object_number=123, tag_tid=20, entry_tid=10083' 2024-04-06T09:44:49.684 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=123, tag_tid=20, entry_tid=10083' '!=' 'object_number=123, tag_tid=20, entry_tid=10007' 2024-04-06T09:44:49.684 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:44:49.684 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=123, tag_tid=20, entry_tid=10083' = 'object_number=123, tag_tid=20, entry_tid=10007' 2024-04-06T09:44:49.684 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=123, tag_tid=20, entry_tid=10083' 2024-04-06T09:44:49.685 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=133, tag_tid=21, entry_tid=1' 2024-04-06T09:44:49.685 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:44:49.685 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:44:49.686 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=21 2024-04-06T09:44:49.686 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=123, tag_tid=20, entry_tid=10083' 2024-04-06T09:44:49.686 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:44:49.687 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:44:49.687 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=20 2024-04-06T09:44:49.688 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=133, tag_tid=21, entry_tid=1' 2024-04-06T09:44:49.688 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:44:49.688 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:44:49.689 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:44:49.690 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=123, tag_tid=20, entry_tid=10083' 2024-04-06T09:44:49.690 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:44:49.690 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:44:49.691 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=10083 2024-04-06T09:44:49.691 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 21 = 20 -a 1 -le 10083 2024-04-06T09:44:49.691 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:44:49.691 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:44:49.691 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:44:49.893 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:44:49.893 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:44:49.893 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:44:49.893 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:44:49.893 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:44:49.893 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:44:49.893 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:44:49.893 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:44:49.893 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:44:49.894 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:44:49.894 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:44:49.894 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:44:49.894 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:44:49.894 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:44:49.894 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:44:49.894 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:44:49.895 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:44:49.898 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:44:49.899 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:44:49.899 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:44:49.899 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:44:49.899 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:44:49.899 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:44:49.899 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:44:49.899 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:44:49.899 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:44:49.899 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:44:49.899 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:44:49.899 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:44:49.899 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:44:49.899 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:44:49.900 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:44:49.900 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:44:49.900 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:44:49.900 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:44:49.900 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:44:49.900 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:44:49.900 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:44:49.900 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:44:49.900 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:44:49.900 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:44:49.900 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:44:49.900 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:44:49.900 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:44:49.900 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:44:49.900 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:44:49.901 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:44:49.901 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:44:49.901 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:44:49.924 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:44:49.924 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:44:49.924 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:44:49.924 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:44:50.050 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:44:50.052 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:44:50.052 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:44:50.052 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:44:50.052 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:44:50.052 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:44:50.052 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:44:50.052 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:44:50.052 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:44:50.052 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:44:50.053 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:44:50.053 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:44:50.053 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:44:50.054 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:44:50.054 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:44:50.055 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:44:50.113 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 30 2024-04-06T09:44:50.113 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 33 2024-04-06T09:44:50.113 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:44:50.113 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=133, tag_tid=21, entry_tid=1], [object_number=132, tag_tid=21, entry_tid=0], [object_number=135, tag_tid=20, entry_tid=19939], [object_number=134, tag_tid=20, entry_tid=19938]]], state=connected] 2024-04-06T09:44:50.113 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=122, tag_tid=20, entry_tid=10154], [object_number=121, tag_tid=20, entry_tid=10153], [object_number=120, tag_tid=20, entry_tid=10152], [object_number=123, tag_tid=20, entry_tid=10151]]], state=connected] 2024-04-06T09:44:50.117 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:44:50.118 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=133, tag_tid=21, entry_tid=1' 2024-04-06T09:44:50.119 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:44:50.119 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:44:50.119 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:44:50.119 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:44:50.120 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:44:50.120 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:44:50.120 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:44:50.120 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:44:50.120 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:44:50.120 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:44:50.120 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:44:50.120 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:44:50.121 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:44:50.121 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:44:50.121 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:44:50.372 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 30 2024-04-06T09:44:50.372 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 33 2024-04-06T09:44:50.373 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:44:50.373 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=133, tag_tid=21, entry_tid=1], [object_number=132, tag_tid=21, entry_tid=0], [object_number=135, tag_tid=20, entry_tid=19939], [object_number=134, tag_tid=20, entry_tid=19938]]], state=connected] 2024-04-06T09:44:50.373 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=122, tag_tid=20, entry_tid=10154], [object_number=121, tag_tid=20, entry_tid=10153], [object_number=120, tag_tid=20, entry_tid=10152], [object_number=123, tag_tid=20, entry_tid=10151]]], state=connected] 2024-04-06T09:44:50.377 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:44:50.380 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=122, tag_tid=20, entry_tid=10154' 2024-04-06T09:44:50.380 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=133, tag_tid=21, entry_tid=1' -a 'object_number=133, tag_tid=21, entry_tid=1' = 'object_number=122, tag_tid=20, entry_tid=10154' 2024-04-06T09:44:50.380 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=122, tag_tid=20, entry_tid=10154' '!=' 'object_number=123, tag_tid=20, entry_tid=10083' 2024-04-06T09:44:50.380 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:44:50.380 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=122, tag_tid=20, entry_tid=10154' = 'object_number=123, tag_tid=20, entry_tid=10083' 2024-04-06T09:44:50.380 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=122, tag_tid=20, entry_tid=10154' 2024-04-06T09:44:50.380 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=133, tag_tid=21, entry_tid=1' 2024-04-06T09:44:50.381 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:44:50.381 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:44:50.382 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=21 2024-04-06T09:44:50.383 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=122, tag_tid=20, entry_tid=10154' 2024-04-06T09:44:50.383 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:44:50.383 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:44:50.384 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=20 2024-04-06T09:44:50.385 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=133, tag_tid=21, entry_tid=1' 2024-04-06T09:44:50.385 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:44:50.386 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:44:50.387 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:44:50.388 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=122, tag_tid=20, entry_tid=10154' 2024-04-06T09:44:50.388 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:44:50.388 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:44:50.390 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=10154 2024-04-06T09:44:50.390 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 21 = 20 -a 1 -le 10154 2024-04-06T09:44:50.390 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:44:50.390 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:44:50.390 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:44:50.592 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:44:50.592 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:44:50.592 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:44:50.592 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:44:50.592 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:44:50.592 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:44:50.592 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:44:50.592 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:44:50.592 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:44:50.592 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:44:50.593 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:44:50.593 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:44:50.593 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:44:50.593 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:44:50.593 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:44:50.593 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:44:50.593 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:44:50.595 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:44:50.595 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:44:50.595 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:44:50.595 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:44:50.596 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:44:50.596 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:44:50.596 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:44:50.596 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:44:50.596 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:44:50.596 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:44:50.596 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:44:50.596 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:44:50.596 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:44:50.596 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:44:50.596 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:44:50.596 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:44:50.597 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:44:50.597 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:44:50.597 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:44:50.597 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:44:50.598 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:44:50.598 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:44:50.598 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:44:50.598 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:44:50.598 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:44:50.598 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:44:50.598 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:44:50.598 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:44:50.598 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:44:50.598 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:44:50.598 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:44:50.598 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:44:50.622 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:44:50.622 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:44:50.622 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:44:50.622 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:44:50.757 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:44:50.758 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:44:50.758 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:44:50.758 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:44:50.758 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:44:50.758 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:44:50.758 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:44:50.758 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:44:50.759 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:44:50.759 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:44:50.759 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:44:50.759 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:44:50.759 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:44:50.760 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:44:50.761 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:44:50.761 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:44:50.818 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 30 2024-04-06T09:44:50.818 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 33 2024-04-06T09:44:50.818 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:44:50.818 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=133, tag_tid=21, entry_tid=1], [object_number=132, tag_tid=21, entry_tid=0], [object_number=135, tag_tid=20, entry_tid=19939], [object_number=134, tag_tid=20, entry_tid=19938]]], state=connected] 2024-04-06T09:44:50.818 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=120, tag_tid=20, entry_tid=10288], [object_number=123, tag_tid=20, entry_tid=10287], [object_number=122, tag_tid=20, entry_tid=10286], [object_number=121, tag_tid=20, entry_tid=10285]]], state=connected] 2024-04-06T09:44:50.821 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:44:50.823 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=133, tag_tid=21, entry_tid=1' 2024-04-06T09:44:50.824 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:44:50.824 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:44:50.824 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:44:50.824 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:44:50.824 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:44:50.824 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:44:50.824 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:44:50.824 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:44:50.825 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:44:50.825 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:44:50.825 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:44:50.825 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:44:50.825 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:44:50.825 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:44:50.826 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:44:50.873 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 30 2024-04-06T09:44:50.873 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 33 2024-04-06T09:44:50.873 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:44:50.873 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=133, tag_tid=21, entry_tid=1], [object_number=132, tag_tid=21, entry_tid=0], [object_number=135, tag_tid=20, entry_tid=19939], [object_number=134, tag_tid=20, entry_tid=19938]]], state=connected] 2024-04-06T09:44:50.873 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=120, tag_tid=20, entry_tid=10288], [object_number=123, tag_tid=20, entry_tid=10287], [object_number=122, tag_tid=20, entry_tid=10286], [object_number=121, tag_tid=20, entry_tid=10285]]], state=connected] 2024-04-06T09:44:50.876 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:44:50.878 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=120, tag_tid=20, entry_tid=10288' 2024-04-06T09:44:50.879 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=133, tag_tid=21, entry_tid=1' -a 'object_number=133, tag_tid=21, entry_tid=1' = 'object_number=120, tag_tid=20, entry_tid=10288' 2024-04-06T09:44:50.879 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=120, tag_tid=20, entry_tid=10288' '!=' 'object_number=122, tag_tid=20, entry_tid=10154' 2024-04-06T09:44:50.879 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:44:50.879 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=120, tag_tid=20, entry_tid=10288' = 'object_number=122, tag_tid=20, entry_tid=10154' 2024-04-06T09:44:50.879 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=120, tag_tid=20, entry_tid=10288' 2024-04-06T09:44:50.879 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=133, tag_tid=21, entry_tid=1' 2024-04-06T09:44:50.879 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:44:50.879 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:44:50.880 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=21 2024-04-06T09:44:50.880 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=120, tag_tid=20, entry_tid=10288' 2024-04-06T09:44:50.880 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:44:50.881 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:44:50.881 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=20 2024-04-06T09:44:50.882 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=133, tag_tid=21, entry_tid=1' 2024-04-06T09:44:50.882 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:44:50.882 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:44:50.883 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:44:50.884 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=120, tag_tid=20, entry_tid=10288' 2024-04-06T09:44:50.884 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:44:50.884 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:44:50.885 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=10288 2024-04-06T09:44:50.885 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 21 = 20 -a 1 -le 10288 2024-04-06T09:44:50.885 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:44:50.885 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:44:50.885 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:44:51.087 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:44:51.087 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:44:51.087 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:44:51.087 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:44:51.087 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:44:51.087 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:44:51.087 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:44:51.087 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:44:51.088 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:44:51.088 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:44:51.088 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:44:51.088 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:44:51.088 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:44:51.088 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:44:51.088 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:44:51.088 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:44:51.089 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:44:51.090 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:44:51.091 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:44:51.091 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:44:51.091 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:44:51.091 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:44:51.091 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:44:51.091 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:44:51.091 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:44:51.091 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:44:51.091 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:44:51.091 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:44:51.091 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:44:51.091 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:44:51.091 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:44:51.091 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:44:51.092 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:44:51.092 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:44:51.092 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:44:51.093 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:44:51.093 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:44:51.093 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:44:51.093 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:44:51.093 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:44:51.093 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:44:51.093 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:44:51.093 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:44:51.093 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:44:51.093 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:44:51.093 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:44:51.093 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:44:51.093 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:44:51.093 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:44:51.118 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:44:51.118 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:44:51.118 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:44:51.119 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:44:51.244 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:44:51.245 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:44:51.245 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:44:51.245 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:44:51.245 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:44:51.246 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:44:51.246 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:44:51.246 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:44:51.246 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:44:51.246 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:44:51.246 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:44:51.247 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:44:51.247 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:44:51.248 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:44:51.248 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:44:51.248 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:44:51.304 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 30 2024-04-06T09:44:51.304 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 33 2024-04-06T09:44:51.305 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:44:51.305 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=133, tag_tid=21, entry_tid=1], [object_number=132, tag_tid=21, entry_tid=0], [object_number=135, tag_tid=20, entry_tid=19939], [object_number=134, tag_tid=20, entry_tid=19938]]], state=connected] 2024-04-06T09:44:51.305 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=123, tag_tid=20, entry_tid=10383], [object_number=122, tag_tid=20, entry_tid=10382], [object_number=121, tag_tid=20, entry_tid=10381], [object_number=120, tag_tid=20, entry_tid=10380]]], state=connected] 2024-04-06T09:44:51.308 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:44:51.309 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=133, tag_tid=21, entry_tid=1' 2024-04-06T09:44:51.310 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:44:51.310 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:44:51.311 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:44:51.311 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:44:51.311 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:44:51.311 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:44:51.311 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:44:51.311 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:44:51.311 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:44:51.311 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:44:51.311 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:44:51.311 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:44:51.312 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:44:51.312 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:44:51.312 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:44:51.359 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 30 2024-04-06T09:44:51.359 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 33 2024-04-06T09:44:51.360 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:44:51.360 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=133, tag_tid=21, entry_tid=1], [object_number=132, tag_tid=21, entry_tid=0], [object_number=135, tag_tid=20, entry_tid=19939], [object_number=134, tag_tid=20, entry_tid=19938]]], state=connected] 2024-04-06T09:44:51.360 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=123, tag_tid=20, entry_tid=10383], [object_number=122, tag_tid=20, entry_tid=10382], [object_number=121, tag_tid=20, entry_tid=10381], [object_number=120, tag_tid=20, entry_tid=10380]]], state=connected] 2024-04-06T09:44:51.363 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:44:51.365 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=123, tag_tid=20, entry_tid=10383' 2024-04-06T09:44:51.365 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=133, tag_tid=21, entry_tid=1' -a 'object_number=133, tag_tid=21, entry_tid=1' = 'object_number=123, tag_tid=20, entry_tid=10383' 2024-04-06T09:44:51.365 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=123, tag_tid=20, entry_tid=10383' '!=' 'object_number=120, tag_tid=20, entry_tid=10288' 2024-04-06T09:44:51.365 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:44:51.365 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=123, tag_tid=20, entry_tid=10383' = 'object_number=120, tag_tid=20, entry_tid=10288' 2024-04-06T09:44:51.365 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=123, tag_tid=20, entry_tid=10383' 2024-04-06T09:44:51.365 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=133, tag_tid=21, entry_tid=1' 2024-04-06T09:44:51.365 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:44:51.365 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:44:51.366 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=21 2024-04-06T09:44:51.367 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=123, tag_tid=20, entry_tid=10383' 2024-04-06T09:44:51.367 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:44:51.367 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:44:51.368 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=20 2024-04-06T09:44:51.368 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=133, tag_tid=21, entry_tid=1' 2024-04-06T09:44:51.368 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:44:51.368 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:44:51.369 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:44:51.370 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=123, tag_tid=20, entry_tid=10383' 2024-04-06T09:44:51.370 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:44:51.370 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:44:51.371 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=10383 2024-04-06T09:44:51.371 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 21 = 20 -a 1 -le 10383 2024-04-06T09:44:51.372 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:44:51.372 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:44:51.372 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:44:51.573 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:44:51.573 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:44:51.573 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:44:51.573 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:44:51.574 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:44:51.574 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:44:51.574 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:44:51.574 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:44:51.574 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:44:51.574 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:44:51.574 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:44:51.574 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:44:51.574 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:44:51.574 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:44:51.574 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:44:51.575 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:44:51.575 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:44:51.577 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:44:51.577 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:44:51.577 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:44:51.577 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:44:51.577 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:44:51.577 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:44:51.577 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:44:51.578 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:44:51.578 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:44:51.578 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:44:51.578 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:44:51.578 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:44:51.578 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:44:51.578 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:44:51.578 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:44:51.578 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:44:51.579 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:44:51.579 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:44:51.580 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:44:51.580 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:44:51.580 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:44:51.580 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:44:51.580 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:44:51.580 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:44:51.580 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:44:51.580 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:44:51.580 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:44:51.580 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:44:51.580 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:44:51.580 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:44:51.580 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:44:51.580 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:44:51.603 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:44:51.604 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:44:51.604 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:44:51.604 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:44:51.739 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:44:51.739 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:44:51.740 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:44:51.740 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:44:51.740 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:44:51.740 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:44:51.740 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:44:51.740 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:44:51.740 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:44:51.740 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:44:51.741 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:44:51.741 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:44:51.741 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:44:51.742 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:44:51.742 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:44:51.743 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:44:51.796 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 30 2024-04-06T09:44:51.796 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 33 2024-04-06T09:44:51.796 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:44:51.796 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=133, tag_tid=21, entry_tid=1], [object_number=132, tag_tid=21, entry_tid=0], [object_number=135, tag_tid=20, entry_tid=19939], [object_number=134, tag_tid=20, entry_tid=19938]]], state=connected] 2024-04-06T09:44:51.796 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=121, tag_tid=20, entry_tid=10481], [object_number=120, tag_tid=20, entry_tid=10480], [object_number=123, tag_tid=20, entry_tid=10479], [object_number=122, tag_tid=20, entry_tid=10478]]], state=connected] 2024-04-06T09:44:51.799 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:44:51.801 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=133, tag_tid=21, entry_tid=1' 2024-04-06T09:44:51.802 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:44:51.802 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:44:51.802 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:44:51.802 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:44:51.802 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:44:51.802 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:44:51.802 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:44:51.802 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:44:51.802 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:44:51.803 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:44:51.803 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:44:51.803 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:44:51.803 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:44:51.803 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:44:51.804 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:44:52.058 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 30 2024-04-06T09:44:52.058 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 33 2024-04-06T09:44:52.058 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:44:52.058 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=133, tag_tid=21, entry_tid=1], [object_number=132, tag_tid=21, entry_tid=0], [object_number=135, tag_tid=20, entry_tid=19939], [object_number=134, tag_tid=20, entry_tid=19938]]], state=connected] 2024-04-06T09:44:52.058 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=121, tag_tid=20, entry_tid=10481], [object_number=120, tag_tid=20, entry_tid=10480], [object_number=123, tag_tid=20, entry_tid=10479], [object_number=122, tag_tid=20, entry_tid=10478]]], state=connected] 2024-04-06T09:44:52.064 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:44:52.067 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=121, tag_tid=20, entry_tid=10481' 2024-04-06T09:44:52.067 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=133, tag_tid=21, entry_tid=1' -a 'object_number=133, tag_tid=21, entry_tid=1' = 'object_number=121, tag_tid=20, entry_tid=10481' 2024-04-06T09:44:52.067 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=121, tag_tid=20, entry_tid=10481' '!=' 'object_number=123, tag_tid=20, entry_tid=10383' 2024-04-06T09:44:52.067 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:44:52.067 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=121, tag_tid=20, entry_tid=10481' = 'object_number=123, tag_tid=20, entry_tid=10383' 2024-04-06T09:44:52.067 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=121, tag_tid=20, entry_tid=10481' 2024-04-06T09:44:52.068 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=133, tag_tid=21, entry_tid=1' 2024-04-06T09:44:52.068 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:44:52.068 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:44:52.069 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=21 2024-04-06T09:44:52.070 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=121, tag_tid=20, entry_tid=10481' 2024-04-06T09:44:52.070 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:44:52.070 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:44:52.072 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=20 2024-04-06T09:44:52.073 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=133, tag_tid=21, entry_tid=1' 2024-04-06T09:44:52.073 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:44:52.073 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:44:52.075 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:44:52.076 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=121, tag_tid=20, entry_tid=10481' 2024-04-06T09:44:52.076 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:44:52.076 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:44:52.078 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=10481 2024-04-06T09:44:52.078 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 21 = 20 -a 1 -le 10481 2024-04-06T09:44:52.078 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:44:52.078 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:44:52.078 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:44:52.280 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:44:52.280 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:44:52.280 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:44:52.280 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:44:52.280 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:44:52.280 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:44:52.280 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:44:52.280 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:44:52.280 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:44:52.280 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:44:52.280 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:44:52.281 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:44:52.281 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:44:52.281 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:44:52.281 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:44:52.281 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:44:52.281 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:44:52.284 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:44:52.284 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:44:52.284 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:44:52.284 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:44:52.284 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:44:52.284 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:44:52.284 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:44:52.284 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:44:52.284 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:44:52.284 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:44:52.284 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:44:52.284 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:44:52.285 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:44:52.285 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:44:52.285 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:44:52.285 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:44:52.285 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:44:52.285 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:44:52.285 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:44:52.285 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:44:52.285 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:44:52.285 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:44:52.285 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:44:52.285 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:44:52.285 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:44:52.285 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:44:52.285 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:44:52.286 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:44:52.286 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:44:52.286 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:44:52.286 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:44:52.286 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:44:52.311 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:44:52.311 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:44:52.311 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:44:52.311 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:44:52.443 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:44:52.444 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:44:52.444 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:44:52.444 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:44:52.444 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:44:52.445 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:44:52.445 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:44:52.445 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:44:52.445 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:44:52.445 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:44:52.445 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:44:52.445 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:44:52.445 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:44:52.447 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:44:52.447 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:44:52.447 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:44:52.503 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 30 2024-04-06T09:44:52.504 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 33 2024-04-06T09:44:52.504 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:44:52.504 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=133, tag_tid=21, entry_tid=1], [object_number=132, tag_tid=21, entry_tid=0], [object_number=135, tag_tid=20, entry_tid=19939], [object_number=134, tag_tid=20, entry_tid=19938]]], state=connected] 2024-04-06T09:44:52.504 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=121, tag_tid=20, entry_tid=10613], [object_number=120, tag_tid=20, entry_tid=10612], [object_number=123, tag_tid=20, entry_tid=10611], [object_number=122, tag_tid=20, entry_tid=10610]]], state=connected] 2024-04-06T09:44:52.507 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:44:52.508 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=133, tag_tid=21, entry_tid=1' 2024-04-06T09:44:52.510 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:44:52.510 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:44:52.510 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:44:52.510 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:44:52.510 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:44:52.510 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:44:52.510 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:44:52.510 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:44:52.510 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:44:52.510 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:44:52.510 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:44:52.510 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:44:52.511 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:44:52.511 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:44:52.511 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:44:52.559 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 30 2024-04-06T09:44:52.559 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 33 2024-04-06T09:44:52.559 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:44:52.559 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=133, tag_tid=21, entry_tid=1], [object_number=132, tag_tid=21, entry_tid=0], [object_number=135, tag_tid=20, entry_tid=19939], [object_number=134, tag_tid=20, entry_tid=19938]]], state=connected] 2024-04-06T09:44:52.559 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=121, tag_tid=20, entry_tid=10613], [object_number=120, tag_tid=20, entry_tid=10612], [object_number=123, tag_tid=20, entry_tid=10611], [object_number=122, tag_tid=20, entry_tid=10610]]], state=connected] 2024-04-06T09:44:52.562 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:44:52.564 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=121, tag_tid=20, entry_tid=10613' 2024-04-06T09:44:52.564 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=133, tag_tid=21, entry_tid=1' -a 'object_number=133, tag_tid=21, entry_tid=1' = 'object_number=121, tag_tid=20, entry_tid=10613' 2024-04-06T09:44:52.564 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=121, tag_tid=20, entry_tid=10613' '!=' 'object_number=121, tag_tid=20, entry_tid=10481' 2024-04-06T09:44:52.564 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:44:52.564 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=121, tag_tid=20, entry_tid=10613' = 'object_number=121, tag_tid=20, entry_tid=10481' 2024-04-06T09:44:52.564 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=121, tag_tid=20, entry_tid=10613' 2024-04-06T09:44:52.565 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=133, tag_tid=21, entry_tid=1' 2024-04-06T09:44:52.565 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:44:52.565 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:44:52.565 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=21 2024-04-06T09:44:52.566 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=121, tag_tid=20, entry_tid=10613' 2024-04-06T09:44:52.566 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:44:52.566 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:44:52.567 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=20 2024-04-06T09:44:52.568 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=133, tag_tid=21, entry_tid=1' 2024-04-06T09:44:52.568 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:44:52.568 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:44:52.569 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:44:52.570 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=121, tag_tid=20, entry_tid=10613' 2024-04-06T09:44:52.570 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:44:52.570 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:44:52.571 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=10613 2024-04-06T09:44:52.571 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 21 = 20 -a 1 -le 10613 2024-04-06T09:44:52.571 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:44:52.571 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:44:52.571 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:44:52.773 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:44:52.773 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:44:52.773 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:44:52.773 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:44:52.774 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:44:52.774 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:44:52.774 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:44:52.774 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:44:52.774 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:44:52.774 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:44:52.774 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:44:52.774 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:44:52.774 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:44:52.774 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:44:52.774 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:44:52.775 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:44:52.775 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:44:52.779 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:44:52.779 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:44:52.779 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:44:52.779 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:44:52.779 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:44:52.779 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:44:52.779 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:44:52.779 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:44:52.779 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:44:52.779 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:44:52.779 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:44:52.780 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:44:52.780 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:44:52.780 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:44:52.780 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:44:52.780 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:44:52.780 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:44:52.780 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:44:52.780 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:44:52.780 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:44:52.780 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:44:52.780 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:44:52.780 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:44:52.781 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:44:52.781 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:44:52.781 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:44:52.781 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:44:52.781 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:44:52.781 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:44:52.781 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:44:52.781 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:44:52.781 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:44:52.803 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:44:52.803 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:44:52.803 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:44:52.803 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:44:52.931 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:44:52.932 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:44:52.932 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:44:52.933 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:44:52.933 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:44:52.933 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:44:52.933 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:44:52.933 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:44:52.933 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:44:52.933 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:44:52.934 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:44:52.934 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:44:52.934 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:44:52.935 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:44:52.935 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:44:52.935 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:44:52.982 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 30 2024-04-06T09:44:52.982 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 33 2024-04-06T09:44:52.982 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:44:52.982 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=133, tag_tid=21, entry_tid=1], [object_number=132, tag_tid=21, entry_tid=0], [object_number=135, tag_tid=20, entry_tid=19939], [object_number=134, tag_tid=20, entry_tid=19938]]], state=connected] 2024-04-06T09:44:52.982 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=123, tag_tid=20, entry_tid=10699], [object_number=122, tag_tid=20, entry_tid=10698], [object_number=121, tag_tid=20, entry_tid=10697], [object_number=120, tag_tid=20, entry_tid=10696]]], state=connected] 2024-04-06T09:44:52.985 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:44:52.986 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=133, tag_tid=21, entry_tid=1' 2024-04-06T09:44:52.988 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:44:52.988 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:44:52.988 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:44:52.988 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:44:52.988 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:44:52.988 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:44:52.988 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:44:52.988 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:44:52.988 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:44:52.988 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:44:52.988 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:44:52.989 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:44:52.989 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:44:52.989 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:44:52.989 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:44:53.036 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 30 2024-04-06T09:44:53.036 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 33 2024-04-06T09:44:53.036 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:44:53.036 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=133, tag_tid=21, entry_tid=1], [object_number=132, tag_tid=21, entry_tid=0], [object_number=135, tag_tid=20, entry_tid=19939], [object_number=134, tag_tid=20, entry_tid=19938]]], state=connected] 2024-04-06T09:44:53.036 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=123, tag_tid=20, entry_tid=10699], [object_number=122, tag_tid=20, entry_tid=10698], [object_number=121, tag_tid=20, entry_tid=10697], [object_number=120, tag_tid=20, entry_tid=10696]]], state=connected] 2024-04-06T09:44:53.039 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:44:53.041 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=123, tag_tid=20, entry_tid=10699' 2024-04-06T09:44:53.041 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=133, tag_tid=21, entry_tid=1' -a 'object_number=133, tag_tid=21, entry_tid=1' = 'object_number=123, tag_tid=20, entry_tid=10699' 2024-04-06T09:44:53.042 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=123, tag_tid=20, entry_tid=10699' '!=' 'object_number=121, tag_tid=20, entry_tid=10613' 2024-04-06T09:44:53.042 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:44:53.042 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=123, tag_tid=20, entry_tid=10699' = 'object_number=121, tag_tid=20, entry_tid=10613' 2024-04-06T09:44:53.042 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=123, tag_tid=20, entry_tid=10699' 2024-04-06T09:44:53.042 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=133, tag_tid=21, entry_tid=1' 2024-04-06T09:44:53.042 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:44:53.042 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:44:53.042 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=21 2024-04-06T09:44:53.043 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=123, tag_tid=20, entry_tid=10699' 2024-04-06T09:44:53.043 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:44:53.043 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:44:53.044 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=20 2024-04-06T09:44:53.045 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=133, tag_tid=21, entry_tid=1' 2024-04-06T09:44:53.045 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:44:53.045 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:44:53.046 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:44:53.046 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=123, tag_tid=20, entry_tid=10699' 2024-04-06T09:44:53.046 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:44:53.047 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:44:53.048 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=10699 2024-04-06T09:44:53.048 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 21 = 20 -a 1 -le 10699 2024-04-06T09:44:53.048 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:44:53.048 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:44:53.048 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:44:53.250 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:44:53.250 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:44:53.250 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:44:53.250 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:44:53.250 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:44:53.251 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:44:53.251 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:44:53.251 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:44:53.251 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:44:53.251 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:44:53.251 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:44:53.251 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:44:53.251 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:44:53.251 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:44:53.251 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:44:53.252 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:44:53.252 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:44:53.256 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:44:53.256 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:44:53.256 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:44:53.257 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:44:53.257 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:44:53.257 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:44:53.257 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:44:53.257 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:44:53.257 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:44:53.257 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:44:53.257 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:44:53.258 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:44:53.258 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:44:53.258 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:44:53.258 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:44:53.258 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:44:53.258 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:44:53.258 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:44:53.258 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:44:53.258 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:44:53.258 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:44:53.258 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:44:53.259 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:44:53.259 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:44:53.259 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:44:53.259 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:44:53.259 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:44:53.259 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:44:53.259 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:44:53.259 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:44:53.259 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:44:53.259 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:44:53.280 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:44:53.280 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:44:53.280 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:44:53.280 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:44:53.413 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:44:53.414 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:44:53.414 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:44:53.414 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:44:53.414 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:44:53.414 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:44:53.415 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:44:53.415 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:44:53.415 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:44:53.415 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:44:53.415 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:44:53.415 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:44:53.415 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:44:53.417 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:44:53.418 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:44:53.418 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:44:53.470 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 30 2024-04-06T09:44:53.470 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 33 2024-04-06T09:44:53.470 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:44:53.470 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=133, tag_tid=21, entry_tid=1], [object_number=132, tag_tid=21, entry_tid=0], [object_number=135, tag_tid=20, entry_tid=19939], [object_number=134, tag_tid=20, entry_tid=19938]]], state=connected] 2024-04-06T09:44:53.470 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=122, tag_tid=20, entry_tid=10774], [object_number=121, tag_tid=20, entry_tid=10773], [object_number=120, tag_tid=20, entry_tid=10772], [object_number=123, tag_tid=20, entry_tid=10771]]], state=connected] 2024-04-06T09:44:53.474 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:44:53.475 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=133, tag_tid=21, entry_tid=1' 2024-04-06T09:44:53.476 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:44:53.476 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:44:53.476 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:44:53.476 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:44:53.477 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:44:53.477 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:44:53.477 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:44:53.477 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:44:53.477 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:44:53.477 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:44:53.477 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:44:53.477 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:44:53.478 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:44:53.478 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:44:53.478 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:44:53.527 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 30 2024-04-06T09:44:53.527 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 33 2024-04-06T09:44:53.528 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:44:53.528 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=133, tag_tid=21, entry_tid=1], [object_number=132, tag_tid=21, entry_tid=0], [object_number=135, tag_tid=20, entry_tid=19939], [object_number=134, tag_tid=20, entry_tid=19938]]], state=connected] 2024-04-06T09:44:53.528 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=122, tag_tid=20, entry_tid=10774], [object_number=121, tag_tid=20, entry_tid=10773], [object_number=120, tag_tid=20, entry_tid=10772], [object_number=123, tag_tid=20, entry_tid=10771]]], state=connected] 2024-04-06T09:44:53.530 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:44:53.532 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=122, tag_tid=20, entry_tid=10774' 2024-04-06T09:44:53.532 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=133, tag_tid=21, entry_tid=1' -a 'object_number=133, tag_tid=21, entry_tid=1' = 'object_number=122, tag_tid=20, entry_tid=10774' 2024-04-06T09:44:53.532 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=122, tag_tid=20, entry_tid=10774' '!=' 'object_number=123, tag_tid=20, entry_tid=10699' 2024-04-06T09:44:53.532 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:44:53.532 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=122, tag_tid=20, entry_tid=10774' = 'object_number=123, tag_tid=20, entry_tid=10699' 2024-04-06T09:44:53.532 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=122, tag_tid=20, entry_tid=10774' 2024-04-06T09:44:53.532 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=133, tag_tid=21, entry_tid=1' 2024-04-06T09:44:53.532 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:44:53.533 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:44:53.533 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=21 2024-04-06T09:44:53.534 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=122, tag_tid=20, entry_tid=10774' 2024-04-06T09:44:53.534 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:44:53.534 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:44:53.535 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=20 2024-04-06T09:44:53.536 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=133, tag_tid=21, entry_tid=1' 2024-04-06T09:44:53.536 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:44:53.536 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:44:53.536 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:44:53.537 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=122, tag_tid=20, entry_tid=10774' 2024-04-06T09:44:53.537 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:44:53.537 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:44:53.538 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=10774 2024-04-06T09:44:53.539 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 21 = 20 -a 1 -le 10774 2024-04-06T09:44:53.539 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:44:53.539 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:44:53.539 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:44:53.741 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:44:53.741 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:44:53.741 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:44:53.741 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:44:53.741 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:44:53.741 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:44:53.741 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:44:53.741 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:44:53.741 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:44:53.742 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:44:53.742 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:44:53.742 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:44:53.742 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:44:53.742 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:44:53.742 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:44:53.743 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:44:53.743 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:44:53.743 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:44:53.743 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:44:53.743 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:44:53.744 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:44:53.744 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:44:53.744 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:44:53.744 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:44:53.744 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:44:53.744 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:44:53.744 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:44:53.744 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:44:53.744 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:44:53.744 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:44:53.744 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:44:53.744 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:44:53.744 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:44:53.746 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:44:53.746 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:44:53.746 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:44:53.746 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:44:53.746 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:44:53.746 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:44:53.746 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:44:53.746 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:44:53.746 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:44:53.746 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:44:53.746 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:44:53.747 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:44:53.747 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:44:53.747 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:44:53.747 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:44:53.747 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:44:53.774 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:44:53.774 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:44:53.774 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:44:53.774 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:44:53.992 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:44:53.993 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:44:53.993 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:44:53.993 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:44:53.994 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:44:53.994 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:44:53.994 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:44:53.994 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:44:53.994 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:44:53.994 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:44:53.994 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:44:53.995 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:44:53.995 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:44:53.997 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:44:53.998 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:44:53.998 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:44:54.051 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 30 2024-04-06T09:44:54.051 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 33 2024-04-06T09:44:54.051 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:44:54.051 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=133, tag_tid=21, entry_tid=1], [object_number=132, tag_tid=21, entry_tid=0], [object_number=135, tag_tid=20, entry_tid=19939], [object_number=134, tag_tid=20, entry_tid=19938]]], state=connected] 2024-04-06T09:44:54.051 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=127, tag_tid=20, entry_tid=10831], [object_number=126, tag_tid=20, entry_tid=10830], [object_number=121, tag_tid=20, entry_tid=10829], [object_number=120, tag_tid=20, entry_tid=10828]]], state=connected] 2024-04-06T09:44:54.054 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:44:54.056 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=133, tag_tid=21, entry_tid=1' 2024-04-06T09:44:54.057 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:44:54.057 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:44:54.057 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:44:54.057 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:44:54.057 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:44:54.057 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:44:54.057 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:44:54.057 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:44:54.057 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:44:54.057 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:44:54.057 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:44:54.058 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:44:54.058 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:44:54.058 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:44:54.059 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:44:54.106 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 30 2024-04-06T09:44:54.106 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 33 2024-04-06T09:44:54.106 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:44:54.106 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=133, tag_tid=21, entry_tid=1], [object_number=132, tag_tid=21, entry_tid=0], [object_number=135, tag_tid=20, entry_tid=19939], [object_number=134, tag_tid=20, entry_tid=19938]]], state=connected] 2024-04-06T09:44:54.107 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=127, tag_tid=20, entry_tid=10831], [object_number=126, tag_tid=20, entry_tid=10830], [object_number=121, tag_tid=20, entry_tid=10829], [object_number=120, tag_tid=20, entry_tid=10828]]], state=connected] 2024-04-06T09:44:54.110 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:44:54.112 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=127, tag_tid=20, entry_tid=10831' 2024-04-06T09:44:54.112 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=133, tag_tid=21, entry_tid=1' -a 'object_number=133, tag_tid=21, entry_tid=1' = 'object_number=127, tag_tid=20, entry_tid=10831' 2024-04-06T09:44:54.112 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=127, tag_tid=20, entry_tid=10831' '!=' 'object_number=122, tag_tid=20, entry_tid=10774' 2024-04-06T09:44:54.112 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:44:54.112 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=127, tag_tid=20, entry_tid=10831' = 'object_number=122, tag_tid=20, entry_tid=10774' 2024-04-06T09:44:54.112 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=127, tag_tid=20, entry_tid=10831' 2024-04-06T09:44:54.112 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=133, tag_tid=21, entry_tid=1' 2024-04-06T09:44:54.112 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:44:54.112 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:44:54.113 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=21 2024-04-06T09:44:54.114 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=127, tag_tid=20, entry_tid=10831' 2024-04-06T09:44:54.114 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:44:54.114 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:44:54.115 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=20 2024-04-06T09:44:54.115 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=133, tag_tid=21, entry_tid=1' 2024-04-06T09:44:54.115 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:44:54.115 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:44:54.116 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:44:54.117 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=127, tag_tid=20, entry_tid=10831' 2024-04-06T09:44:54.117 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:44:54.117 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:44:54.118 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=10831 2024-04-06T09:44:54.118 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 21 = 20 -a 1 -le 10831 2024-04-06T09:44:54.119 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:44:54.119 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:44:54.119 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:44:54.320 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:44:54.320 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:44:54.320 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:44:54.321 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:44:54.321 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:44:54.321 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:44:54.321 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:44:54.321 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:44:54.321 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:44:54.321 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:44:54.321 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:44:54.321 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:44:54.321 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:44:54.321 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:44:54.321 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:44:54.322 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:44:54.322 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:44:54.325 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:44:54.325 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:44:54.325 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:44:54.325 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:44:54.325 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:44:54.325 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:44:54.325 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:44:54.325 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:44:54.325 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:44:54.325 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:44:54.325 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:44:54.325 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:44:54.326 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:44:54.326 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:44:54.326 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:44:54.326 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:44:54.327 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:44:54.327 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:44:54.327 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:44:54.327 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:44:54.327 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:44:54.327 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:44:54.327 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:44:54.327 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:44:54.327 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:44:54.327 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:44:54.327 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:44:54.327 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:44:54.327 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:44:54.328 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:44:54.328 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:44:54.328 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:44:54.349 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:44:54.349 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:44:54.349 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:44:54.349 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:44:54.489 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:44:54.490 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:44:54.491 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:44:54.491 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:44:54.491 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:44:54.491 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:44:54.491 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:44:54.491 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:44:54.491 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:44:54.491 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:44:54.491 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:44:54.492 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:44:54.493 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:44:54.494 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:44:54.494 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:44:54.495 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:44:54.561 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 31 2024-04-06T09:44:54.561 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 33 2024-04-06T09:44:54.561 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:44:54.561 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=133, tag_tid=21, entry_tid=1], [object_number=132, tag_tid=21, entry_tid=0], [object_number=135, tag_tid=20, entry_tid=19939], [object_number=134, tag_tid=20, entry_tid=19938]]], state=connected] 2024-04-06T09:44:54.561 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=124, tag_tid=20, entry_tid=10920], [object_number=127, tag_tid=20, entry_tid=10919], [object_number=126, tag_tid=20, entry_tid=10918], [object_number=125, tag_tid=20, entry_tid=10917]]], state=connected] 2024-04-06T09:44:54.564 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:44:54.565 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=133, tag_tid=21, entry_tid=1' 2024-04-06T09:44:54.567 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:44:54.567 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:44:54.567 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:44:54.567 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:44:54.567 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:44:54.567 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:44:54.567 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:44:54.567 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:44:54.567 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:44:54.567 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:44:54.567 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:44:54.567 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:44:54.568 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:44:54.568 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:44:54.568 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:44:54.614 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 31 2024-04-06T09:44:54.615 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 33 2024-04-06T09:44:54.615 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:44:54.615 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=133, tag_tid=21, entry_tid=1], [object_number=132, tag_tid=21, entry_tid=0], [object_number=135, tag_tid=20, entry_tid=19939], [object_number=134, tag_tid=20, entry_tid=19938]]], state=connected] 2024-04-06T09:44:54.615 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=124, tag_tid=20, entry_tid=10920], [object_number=127, tag_tid=20, entry_tid=10919], [object_number=126, tag_tid=20, entry_tid=10918], [object_number=125, tag_tid=20, entry_tid=10917]]], state=connected] 2024-04-06T09:44:54.618 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:44:54.620 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=124, tag_tid=20, entry_tid=10920' 2024-04-06T09:44:54.620 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=133, tag_tid=21, entry_tid=1' -a 'object_number=133, tag_tid=21, entry_tid=1' = 'object_number=124, tag_tid=20, entry_tid=10920' 2024-04-06T09:44:54.620 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=124, tag_tid=20, entry_tid=10920' '!=' 'object_number=127, tag_tid=20, entry_tid=10831' 2024-04-06T09:44:54.620 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:44:54.620 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=124, tag_tid=20, entry_tid=10920' = 'object_number=127, tag_tid=20, entry_tid=10831' 2024-04-06T09:44:54.620 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=124, tag_tid=20, entry_tid=10920' 2024-04-06T09:44:54.620 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=133, tag_tid=21, entry_tid=1' 2024-04-06T09:44:54.620 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:44:54.621 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:44:54.621 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=21 2024-04-06T09:44:54.622 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=124, tag_tid=20, entry_tid=10920' 2024-04-06T09:44:54.622 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:44:54.622 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:44:54.623 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=20 2024-04-06T09:44:54.623 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=133, tag_tid=21, entry_tid=1' 2024-04-06T09:44:54.624 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:44:54.624 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:44:54.624 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:44:54.625 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=124, tag_tid=20, entry_tid=10920' 2024-04-06T09:44:54.625 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:44:54.625 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:44:54.626 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=10920 2024-04-06T09:44:54.626 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 21 = 20 -a 1 -le 10920 2024-04-06T09:44:54.627 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:44:54.627 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:44:54.627 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:44:54.829 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:44:54.829 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:44:54.829 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:44:54.829 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:44:54.829 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:44:54.829 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:44:54.829 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:44:54.829 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:44:54.829 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:44:54.829 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:44:54.829 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:44:54.830 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:44:54.830 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:44:54.830 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:44:54.830 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:44:54.830 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:44:54.830 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:44:54.833 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:44:54.833 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:44:54.833 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:44:54.833 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:44:54.833 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:44:54.833 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:44:54.833 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:44:54.833 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:44:54.833 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:44:54.833 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:44:54.833 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:44:54.833 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:44:54.834 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:44:54.834 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:44:54.834 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:44:54.834 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:44:54.834 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:44:54.834 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:44:54.834 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:44:54.834 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:44:54.834 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:44:54.834 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:44:54.834 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:44:54.834 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:44:54.834 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:44:54.834 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:44:54.835 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:44:54.835 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:44:54.835 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:44:54.835 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:44:54.835 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:44:54.835 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:44:54.859 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:44:54.859 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:44:54.859 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:44:54.859 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:44:54.987 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:44:54.988 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:44:54.988 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:44:54.988 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:44:54.988 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:44:54.988 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:44:54.988 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:44:54.988 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:44:54.988 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:44:54.988 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:44:54.989 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:44:54.989 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:44:54.989 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:44:54.990 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:44:54.991 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:44:54.991 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:44:55.040 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 31 2024-04-06T09:44:55.040 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 33 2024-04-06T09:44:55.040 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:44:55.040 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=133, tag_tid=21, entry_tid=1], [object_number=132, tag_tid=21, entry_tid=0], [object_number=135, tag_tid=20, entry_tid=19939], [object_number=134, tag_tid=20, entry_tid=19938]]], state=connected] 2024-04-06T09:44:55.040 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=125, tag_tid=20, entry_tid=11033], [object_number=124, tag_tid=20, entry_tid=11032], [object_number=127, tag_tid=20, entry_tid=11031], [object_number=126, tag_tid=20, entry_tid=11030]]], state=connected] 2024-04-06T09:44:55.044 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:44:55.045 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=133, tag_tid=21, entry_tid=1' 2024-04-06T09:44:55.046 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:44:55.046 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:44:55.046 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:44:55.046 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:44:55.046 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:44:55.047 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:44:55.047 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:44:55.047 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:44:55.047 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:44:55.047 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:44:55.047 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:44:55.047 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:44:55.047 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:44:55.048 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:44:55.048 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:44:55.094 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 31 2024-04-06T09:44:55.094 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 33 2024-04-06T09:44:55.094 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:44:55.094 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=133, tag_tid=21, entry_tid=1], [object_number=132, tag_tid=21, entry_tid=0], [object_number=135, tag_tid=20, entry_tid=19939], [object_number=134, tag_tid=20, entry_tid=19938]]], state=connected] 2024-04-06T09:44:55.094 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=125, tag_tid=20, entry_tid=11033], [object_number=124, tag_tid=20, entry_tid=11032], [object_number=127, tag_tid=20, entry_tid=11031], [object_number=126, tag_tid=20, entry_tid=11030]]], state=connected] 2024-04-06T09:44:55.097 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:44:55.099 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=125, tag_tid=20, entry_tid=11033' 2024-04-06T09:44:55.099 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=133, tag_tid=21, entry_tid=1' -a 'object_number=133, tag_tid=21, entry_tid=1' = 'object_number=125, tag_tid=20, entry_tid=11033' 2024-04-06T09:44:55.099 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=125, tag_tid=20, entry_tid=11033' '!=' 'object_number=124, tag_tid=20, entry_tid=10920' 2024-04-06T09:44:55.099 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:44:55.099 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=125, tag_tid=20, entry_tid=11033' = 'object_number=124, tag_tid=20, entry_tid=10920' 2024-04-06T09:44:55.100 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=125, tag_tid=20, entry_tid=11033' 2024-04-06T09:44:55.100 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=133, tag_tid=21, entry_tid=1' 2024-04-06T09:44:55.100 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:44:55.100 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:44:55.101 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=21 2024-04-06T09:44:55.101 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=125, tag_tid=20, entry_tid=11033' 2024-04-06T09:44:55.101 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:44:55.101 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:44:55.102 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=20 2024-04-06T09:44:55.103 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=133, tag_tid=21, entry_tid=1' 2024-04-06T09:44:55.103 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:44:55.103 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:44:55.104 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:44:55.105 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=125, tag_tid=20, entry_tid=11033' 2024-04-06T09:44:55.105 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:44:55.105 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:44:55.106 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=11033 2024-04-06T09:44:55.106 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 21 = 20 -a 1 -le 11033 2024-04-06T09:44:55.106 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:44:55.107 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:44:55.107 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:44:55.308 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:44:55.308 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:44:55.308 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:44:55.308 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:44:55.308 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:44:55.308 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:44:55.308 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:44:55.308 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:44:55.308 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:44:55.309 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:44:55.309 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:44:55.309 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:44:55.309 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:44:55.309 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:44:55.309 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:44:55.309 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:44:55.310 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:44:55.311 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:44:55.311 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:44:55.312 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:44:55.312 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:44:55.312 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:44:55.312 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:44:55.312 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:44:55.312 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:44:55.312 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:44:55.312 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:44:55.312 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:44:55.313 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:44:55.313 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:44:55.313 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:44:55.313 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:44:55.313 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:44:55.314 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:44:55.314 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:44:55.314 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:44:55.315 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:44:55.315 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:44:55.315 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:44:55.315 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:44:55.315 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:44:55.315 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:44:55.315 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:44:55.315 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:44:55.315 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:44:55.315 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:44:55.315 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:44:55.315 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:44:55.315 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:44:55.340 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:44:55.340 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:44:55.340 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:44:55.340 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:44:55.470 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:44:55.471 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:44:55.471 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:44:55.471 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:44:55.471 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:44:55.471 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:44:55.472 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:44:55.472 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:44:55.472 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:44:55.472 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:44:55.472 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:44:55.472 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:44:55.473 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:44:55.475 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:44:55.475 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:44:55.475 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:44:55.526 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 31 2024-04-06T09:44:55.526 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 33 2024-04-06T09:44:55.526 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:44:55.526 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=133, tag_tid=21, entry_tid=1], [object_number=132, tag_tid=21, entry_tid=0], [object_number=135, tag_tid=20, entry_tid=19939], [object_number=134, tag_tid=20, entry_tid=19938]]], state=connected] 2024-04-06T09:44:55.526 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=126, tag_tid=20, entry_tid=11138], [object_number=125, tag_tid=20, entry_tid=11137], [object_number=124, tag_tid=20, entry_tid=11136], [object_number=127, tag_tid=20, entry_tid=11135]]], state=connected] 2024-04-06T09:44:55.529 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:44:55.530 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=133, tag_tid=21, entry_tid=1' 2024-04-06T09:44:55.532 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:44:55.532 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:44:55.532 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:44:55.532 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:44:55.532 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:44:55.532 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:44:55.532 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:44:55.532 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:44:55.532 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:44:55.532 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:44:55.532 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:44:55.532 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:44:55.533 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:44:55.533 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:44:55.533 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:44:55.580 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 31 2024-04-06T09:44:55.580 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 33 2024-04-06T09:44:55.580 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:44:55.580 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=133, tag_tid=21, entry_tid=1], [object_number=132, tag_tid=21, entry_tid=0], [object_number=135, tag_tid=20, entry_tid=19939], [object_number=134, tag_tid=20, entry_tid=19938]]], state=connected] 2024-04-06T09:44:55.580 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=126, tag_tid=20, entry_tid=11138], [object_number=125, tag_tid=20, entry_tid=11137], [object_number=124, tag_tid=20, entry_tid=11136], [object_number=127, tag_tid=20, entry_tid=11135]]], state=connected] 2024-04-06T09:44:55.583 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:44:55.585 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=126, tag_tid=20, entry_tid=11138' 2024-04-06T09:44:55.585 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=133, tag_tid=21, entry_tid=1' -a 'object_number=133, tag_tid=21, entry_tid=1' = 'object_number=126, tag_tid=20, entry_tid=11138' 2024-04-06T09:44:55.585 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=126, tag_tid=20, entry_tid=11138' '!=' 'object_number=125, tag_tid=20, entry_tid=11033' 2024-04-06T09:44:55.585 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:44:55.585 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=126, tag_tid=20, entry_tid=11138' = 'object_number=125, tag_tid=20, entry_tid=11033' 2024-04-06T09:44:55.586 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=126, tag_tid=20, entry_tid=11138' 2024-04-06T09:44:55.586 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=133, tag_tid=21, entry_tid=1' 2024-04-06T09:44:55.586 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:44:55.586 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:44:55.587 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=21 2024-04-06T09:44:55.587 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=126, tag_tid=20, entry_tid=11138' 2024-04-06T09:44:55.587 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:44:55.588 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:44:55.588 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=20 2024-04-06T09:44:55.589 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=133, tag_tid=21, entry_tid=1' 2024-04-06T09:44:55.589 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:44:55.589 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:44:55.590 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:44:55.591 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=126, tag_tid=20, entry_tid=11138' 2024-04-06T09:44:55.591 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:44:55.591 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:44:55.592 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=11138 2024-04-06T09:44:55.592 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 21 = 20 -a 1 -le 11138 2024-04-06T09:44:55.592 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:44:55.592 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:44:55.592 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:44:55.794 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:44:55.794 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:44:55.794 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:44:55.794 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:44:55.794 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:44:55.794 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:44:55.795 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:44:55.795 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:44:55.795 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:44:55.795 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:44:55.795 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:44:55.795 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:44:55.795 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:44:55.795 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:44:55.795 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:44:55.796 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:44:55.796 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:44:55.799 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:44:55.799 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:44:55.799 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:44:55.799 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:44:55.799 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:44:55.799 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:44:55.799 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:44:55.799 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:44:55.799 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:44:55.799 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:44:55.800 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:44:55.800 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:44:55.800 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:44:55.800 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:44:55.800 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:44:55.800 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:44:55.801 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:44:55.801 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:44:55.801 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:44:55.801 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:44:55.801 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:44:55.801 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:44:55.801 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:44:55.801 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:44:55.801 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:44:55.801 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:44:55.802 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:44:55.802 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:44:55.802 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:44:55.802 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:44:55.802 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:44:55.802 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:44:55.825 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:44:55.826 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:44:55.826 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:44:55.826 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:44:55.956 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:44:55.957 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:44:55.958 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:44:55.958 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:44:55.958 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:44:55.958 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:44:55.958 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:44:55.958 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:44:55.958 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:44:55.958 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:44:55.959 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:44:55.959 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:44:55.959 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:44:55.960 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:44:55.960 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:44:55.960 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:44:56.008 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 31 2024-04-06T09:44:56.008 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 33 2024-04-06T09:44:56.008 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:44:56.008 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=133, tag_tid=21, entry_tid=1], [object_number=132, tag_tid=21, entry_tid=0], [object_number=135, tag_tid=20, entry_tid=19939], [object_number=134, tag_tid=20, entry_tid=19938]]], state=connected] 2024-04-06T09:44:56.008 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=127, tag_tid=20, entry_tid=11231], [object_number=126, tag_tid=20, entry_tid=11230], [object_number=125, tag_tid=20, entry_tid=11229], [object_number=124, tag_tid=20, entry_tid=11228]]], state=connected] 2024-04-06T09:44:56.011 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:44:56.013 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=133, tag_tid=21, entry_tid=1' 2024-04-06T09:44:56.014 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:44:56.014 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:44:56.014 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:44:56.014 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:44:56.014 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:44:56.014 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:44:56.014 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:44:56.014 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:44:56.014 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:44:56.015 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:44:56.015 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:44:56.015 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:44:56.015 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:44:56.015 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:44:56.015 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:44:56.062 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 31 2024-04-06T09:44:56.062 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 33 2024-04-06T09:44:56.062 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:44:56.062 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=133, tag_tid=21, entry_tid=1], [object_number=132, tag_tid=21, entry_tid=0], [object_number=135, tag_tid=20, entry_tid=19939], [object_number=134, tag_tid=20, entry_tid=19938]]], state=connected] 2024-04-06T09:44:56.063 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=127, tag_tid=20, entry_tid=11231], [object_number=126, tag_tid=20, entry_tid=11230], [object_number=125, tag_tid=20, entry_tid=11229], [object_number=124, tag_tid=20, entry_tid=11228]]], state=connected] 2024-04-06T09:44:56.066 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:44:56.067 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=127, tag_tid=20, entry_tid=11231' 2024-04-06T09:44:56.067 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=133, tag_tid=21, entry_tid=1' -a 'object_number=133, tag_tid=21, entry_tid=1' = 'object_number=127, tag_tid=20, entry_tid=11231' 2024-04-06T09:44:56.067 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=127, tag_tid=20, entry_tid=11231' '!=' 'object_number=126, tag_tid=20, entry_tid=11138' 2024-04-06T09:44:56.068 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:44:56.068 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=127, tag_tid=20, entry_tid=11231' = 'object_number=126, tag_tid=20, entry_tid=11138' 2024-04-06T09:44:56.068 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=127, tag_tid=20, entry_tid=11231' 2024-04-06T09:44:56.068 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=133, tag_tid=21, entry_tid=1' 2024-04-06T09:44:56.068 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:44:56.068 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:44:56.069 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=21 2024-04-06T09:44:56.069 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=127, tag_tid=20, entry_tid=11231' 2024-04-06T09:44:56.070 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:44:56.070 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:44:56.071 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=20 2024-04-06T09:44:56.071 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=133, tag_tid=21, entry_tid=1' 2024-04-06T09:44:56.071 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:44:56.071 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:44:56.072 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:44:56.073 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=127, tag_tid=20, entry_tid=11231' 2024-04-06T09:44:56.073 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:44:56.073 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:44:56.074 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=11231 2024-04-06T09:44:56.074 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 21 = 20 -a 1 -le 11231 2024-04-06T09:44:56.074 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:44:56.075 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:44:56.075 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:44:56.276 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:44:56.276 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:44:56.276 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:44:56.277 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:44:56.277 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:44:56.277 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:44:56.277 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:44:56.277 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:44:56.277 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:44:56.277 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:44:56.277 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:44:56.277 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:44:56.277 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:44:56.277 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:44:56.277 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:44:56.278 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:44:56.278 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:44:56.280 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:44:56.280 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:44:56.280 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:44:56.280 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:44:56.280 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:44:56.281 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:44:56.281 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:44:56.281 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:44:56.281 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:44:56.281 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:44:56.281 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:44:56.281 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:44:56.281 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:44:56.281 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:44:56.281 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:44:56.281 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:44:56.282 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:44:56.282 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:44:56.282 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:44:56.282 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:44:56.282 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:44:56.282 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:44:56.283 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:44:56.283 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:44:56.283 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:44:56.283 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:44:56.283 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:44:56.283 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:44:56.283 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:44:56.283 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:44:56.283 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:44:56.283 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:44:56.308 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:44:56.308 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:44:56.308 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:44:56.308 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:44:56.439 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:44:56.440 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:44:56.440 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:44:56.440 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:44:56.440 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:44:56.440 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:44:56.441 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:44:56.441 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:44:56.441 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:44:56.441 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:44:56.441 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:44:56.441 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:44:56.441 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:44:56.443 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:44:56.443 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:44:56.443 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:44:56.490 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 31 2024-04-06T09:44:56.491 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 33 2024-04-06T09:44:56.491 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:44:56.491 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=133, tag_tid=21, entry_tid=1], [object_number=132, tag_tid=21, entry_tid=0], [object_number=135, tag_tid=20, entry_tid=19939], [object_number=134, tag_tid=20, entry_tid=19938]]], state=connected] 2024-04-06T09:44:56.491 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=127, tag_tid=20, entry_tid=11319], [object_number=126, tag_tid=20, entry_tid=11318], [object_number=125, tag_tid=20, entry_tid=11317], [object_number=124, tag_tid=20, entry_tid=11316]]], state=connected] 2024-04-06T09:44:56.494 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:44:56.495 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=133, tag_tid=21, entry_tid=1' 2024-04-06T09:44:56.496 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:44:56.496 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:44:56.497 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:44:56.497 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:44:56.497 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:44:56.497 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:44:56.497 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:44:56.497 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:44:56.497 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:44:56.497 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:44:56.497 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:44:56.497 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:44:56.498 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:44:56.498 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:44:56.498 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:44:56.544 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 31 2024-04-06T09:44:56.544 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 33 2024-04-06T09:44:56.544 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:44:56.544 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=133, tag_tid=21, entry_tid=1], [object_number=132, tag_tid=21, entry_tid=0], [object_number=135, tag_tid=20, entry_tid=19939], [object_number=134, tag_tid=20, entry_tid=19938]]], state=connected] 2024-04-06T09:44:56.544 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=127, tag_tid=20, entry_tid=11319], [object_number=126, tag_tid=20, entry_tid=11318], [object_number=125, tag_tid=20, entry_tid=11317], [object_number=124, tag_tid=20, entry_tid=11316]]], state=connected] 2024-04-06T09:44:56.547 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:44:56.550 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=127, tag_tid=20, entry_tid=11319' 2024-04-06T09:44:56.550 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=133, tag_tid=21, entry_tid=1' -a 'object_number=133, tag_tid=21, entry_tid=1' = 'object_number=127, tag_tid=20, entry_tid=11319' 2024-04-06T09:44:56.550 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=127, tag_tid=20, entry_tid=11319' '!=' 'object_number=127, tag_tid=20, entry_tid=11231' 2024-04-06T09:44:56.550 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:44:56.550 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=127, tag_tid=20, entry_tid=11319' = 'object_number=127, tag_tid=20, entry_tid=11231' 2024-04-06T09:44:56.550 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=127, tag_tid=20, entry_tid=11319' 2024-04-06T09:44:56.550 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=133, tag_tid=21, entry_tid=1' 2024-04-06T09:44:56.550 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:44:56.550 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:44:56.551 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=21 2024-04-06T09:44:56.551 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=127, tag_tid=20, entry_tid=11319' 2024-04-06T09:44:56.552 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:44:56.552 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:44:56.552 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=20 2024-04-06T09:44:56.553 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=133, tag_tid=21, entry_tid=1' 2024-04-06T09:44:56.553 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:44:56.553 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:44:56.554 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:44:56.555 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=127, tag_tid=20, entry_tid=11319' 2024-04-06T09:44:56.555 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:44:56.555 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:44:56.556 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=11319 2024-04-06T09:44:56.556 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 21 = 20 -a 1 -le 11319 2024-04-06T09:44:56.556 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:44:56.556 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:44:56.556 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:44:56.758 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:44:56.758 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:44:56.758 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:44:56.758 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:44:56.758 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:44:56.758 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:44:56.758 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:44:56.758 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:44:56.759 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:44:56.759 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:44:56.759 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:44:56.759 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:44:56.759 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:44:56.759 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:44:56.759 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:44:56.759 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:44:56.760 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:44:56.763 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:44:56.763 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:44:56.763 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:44:56.763 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:44:56.763 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:44:56.763 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:44:56.764 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:44:56.764 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:44:56.764 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:44:56.764 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:44:56.764 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:44:56.764 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:44:56.764 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:44:56.764 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:44:56.764 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:44:56.764 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:44:56.764 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:44:56.764 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:44:56.764 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:44:56.765 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:44:56.765 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:44:56.765 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:44:56.765 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:44:56.765 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:44:56.765 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:44:56.765 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:44:56.765 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:44:56.765 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:44:56.765 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:44:56.765 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:44:56.765 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:44:56.765 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:44:56.790 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:44:56.790 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:44:56.790 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:44:56.790 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:44:56.928 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:44:56.930 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:44:56.931 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:44:56.931 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:44:56.931 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:44:56.931 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:44:56.931 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:44:56.932 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:44:56.932 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:44:56.932 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:44:56.932 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:44:56.932 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:44:56.932 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:44:56.933 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:44:56.933 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:44:56.933 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:44:56.991 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 31 2024-04-06T09:44:56.991 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 33 2024-04-06T09:44:56.991 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:44:56.991 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=133, tag_tid=21, entry_tid=1], [object_number=132, tag_tid=21, entry_tid=0], [object_number=135, tag_tid=20, entry_tid=19939], [object_number=134, tag_tid=20, entry_tid=19938]]], state=connected] 2024-04-06T09:44:56.991 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=124, tag_tid=20, entry_tid=11404], [object_number=127, tag_tid=20, entry_tid=11403], [object_number=126, tag_tid=20, entry_tid=11402], [object_number=125, tag_tid=20, entry_tid=11401]]], state=connected] 2024-04-06T09:44:56.995 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:44:56.996 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=133, tag_tid=21, entry_tid=1' 2024-04-06T09:44:56.997 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:44:56.997 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:44:56.997 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:44:56.997 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:44:56.998 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:44:56.998 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:44:56.998 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:44:56.998 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:44:56.998 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:44:56.998 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:44:56.998 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:44:56.998 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:44:56.999 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:44:56.999 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:44:56.999 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:44:57.047 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 31 2024-04-06T09:44:57.047 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 33 2024-04-06T09:44:57.047 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:44:57.048 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=133, tag_tid=21, entry_tid=1], [object_number=132, tag_tid=21, entry_tid=0], [object_number=135, tag_tid=20, entry_tid=19939], [object_number=134, tag_tid=20, entry_tid=19938]]], state=connected] 2024-04-06T09:44:57.048 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=124, tag_tid=20, entry_tid=11404], [object_number=127, tag_tid=20, entry_tid=11403], [object_number=126, tag_tid=20, entry_tid=11402], [object_number=125, tag_tid=20, entry_tid=11401]]], state=connected] 2024-04-06T09:44:57.051 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:44:57.053 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=124, tag_tid=20, entry_tid=11404' 2024-04-06T09:44:57.053 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=133, tag_tid=21, entry_tid=1' -a 'object_number=133, tag_tid=21, entry_tid=1' = 'object_number=124, tag_tid=20, entry_tid=11404' 2024-04-06T09:44:57.054 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=124, tag_tid=20, entry_tid=11404' '!=' 'object_number=127, tag_tid=20, entry_tid=11319' 2024-04-06T09:44:57.054 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:44:57.054 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=124, tag_tid=20, entry_tid=11404' = 'object_number=127, tag_tid=20, entry_tid=11319' 2024-04-06T09:44:57.054 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=124, tag_tid=20, entry_tid=11404' 2024-04-06T09:44:57.054 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=133, tag_tid=21, entry_tid=1' 2024-04-06T09:44:57.054 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:44:57.055 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:44:57.055 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=21 2024-04-06T09:44:57.055 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=124, tag_tid=20, entry_tid=11404' 2024-04-06T09:44:57.055 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:44:57.055 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:44:57.056 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=20 2024-04-06T09:44:57.056 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=133, tag_tid=21, entry_tid=1' 2024-04-06T09:44:57.056 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:44:57.056 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:44:57.057 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:44:57.058 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=124, tag_tid=20, entry_tid=11404' 2024-04-06T09:44:57.058 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:44:57.059 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:44:57.060 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=11404 2024-04-06T09:44:57.060 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 21 = 20 -a 1 -le 11404 2024-04-06T09:44:57.060 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:44:57.060 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:44:57.060 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:44:57.262 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:44:57.262 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:44:57.262 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:44:57.262 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:44:57.262 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:44:57.262 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:44:57.262 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:44:57.262 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:44:57.262 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:44:57.262 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:44:57.263 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:44:57.263 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:44:57.263 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:44:57.263 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:44:57.263 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:44:57.263 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:44:57.265 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:44:57.265 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:44:57.265 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:44:57.265 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:44:57.266 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:44:57.266 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:44:57.266 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:44:57.266 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:44:57.266 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:44:57.266 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:44:57.266 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:44:57.266 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:44:57.266 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:44:57.266 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:44:57.266 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:44:57.266 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:44:57.266 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:44:57.266 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:44:57.267 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:44:57.267 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:44:57.267 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:44:57.267 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:44:57.267 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:44:57.267 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:44:57.267 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:44:57.267 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:44:57.267 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:44:57.267 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:44:57.267 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:44:57.267 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:44:57.267 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:44:57.267 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:44:57.268 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:44:57.292 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:44:57.292 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:44:57.292 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:44:57.292 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:44:57.423 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:44:57.424 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:44:57.425 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:44:57.425 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:44:57.425 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:44:57.425 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:44:57.425 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:44:57.425 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:44:57.425 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:44:57.425 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:44:57.425 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:44:57.425 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:44:57.425 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:44:57.427 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:44:57.427 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:44:57.428 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:44:57.476 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 31 2024-04-06T09:44:57.476 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 33 2024-04-06T09:44:57.476 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:44:57.476 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=133, tag_tid=21, entry_tid=1], [object_number=132, tag_tid=21, entry_tid=0], [object_number=135, tag_tid=20, entry_tid=19939], [object_number=134, tag_tid=20, entry_tid=19938]]], state=connected] 2024-04-06T09:44:57.476 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=127, tag_tid=20, entry_tid=11491], [object_number=126, tag_tid=20, entry_tid=11490], [object_number=125, tag_tid=20, entry_tid=11489], [object_number=124, tag_tid=20, entry_tid=11488]]], state=connected] 2024-04-06T09:44:57.479 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:44:57.481 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=133, tag_tid=21, entry_tid=1' 2024-04-06T09:44:57.482 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:44:57.482 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:44:57.482 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:44:57.482 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:44:57.482 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:44:57.482 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:44:57.482 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:44:57.482 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:44:57.482 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:44:57.482 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:44:57.482 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:44:57.483 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:44:57.483 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:44:57.483 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:44:57.483 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:44:57.531 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 31 2024-04-06T09:44:57.531 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 33 2024-04-06T09:44:57.531 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:44:57.531 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=133, tag_tid=21, entry_tid=1], [object_number=132, tag_tid=21, entry_tid=0], [object_number=135, tag_tid=20, entry_tid=19939], [object_number=134, tag_tid=20, entry_tid=19938]]], state=connected] 2024-04-06T09:44:57.531 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=127, tag_tid=20, entry_tid=11491], [object_number=126, tag_tid=20, entry_tid=11490], [object_number=125, tag_tid=20, entry_tid=11489], [object_number=124, tag_tid=20, entry_tid=11488]]], state=connected] 2024-04-06T09:44:57.534 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:44:57.536 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=127, tag_tid=20, entry_tid=11491' 2024-04-06T09:44:57.536 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=133, tag_tid=21, entry_tid=1' -a 'object_number=133, tag_tid=21, entry_tid=1' = 'object_number=127, tag_tid=20, entry_tid=11491' 2024-04-06T09:44:57.536 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=127, tag_tid=20, entry_tid=11491' '!=' 'object_number=124, tag_tid=20, entry_tid=11404' 2024-04-06T09:44:57.536 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:44:57.536 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=127, tag_tid=20, entry_tid=11491' = 'object_number=124, tag_tid=20, entry_tid=11404' 2024-04-06T09:44:57.537 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=127, tag_tid=20, entry_tid=11491' 2024-04-06T09:44:57.537 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=133, tag_tid=21, entry_tid=1' 2024-04-06T09:44:57.537 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:44:57.537 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:44:57.537 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=21 2024-04-06T09:44:57.538 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=127, tag_tid=20, entry_tid=11491' 2024-04-06T09:44:57.538 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:44:57.538 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:44:57.539 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=20 2024-04-06T09:44:57.540 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=133, tag_tid=21, entry_tid=1' 2024-04-06T09:44:57.540 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:44:57.540 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:44:57.541 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:44:57.541 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=127, tag_tid=20, entry_tid=11491' 2024-04-06T09:44:57.541 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:44:57.541 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:44:57.543 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=11491 2024-04-06T09:44:57.543 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 21 = 20 -a 1 -le 11491 2024-04-06T09:44:57.543 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:44:57.543 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:44:57.543 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:44:57.745 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:44:57.745 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:44:57.745 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:44:57.745 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:44:57.745 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:44:57.745 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:44:57.745 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:44:57.745 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:44:57.745 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:44:57.745 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:44:57.745 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:44:57.745 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:44:57.746 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:44:57.746 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:44:57.746 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:44:57.746 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:44:57.747 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:44:57.749 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:44:57.749 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:44:57.749 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:44:57.749 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:44:57.750 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:44:57.750 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:44:57.750 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:44:57.750 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:44:57.750 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:44:57.750 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:44:57.750 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:44:57.750 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:44:57.750 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:44:57.750 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:44:57.750 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:44:57.750 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:44:57.751 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:44:57.751 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:44:57.752 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:44:57.752 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:44:57.752 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:44:57.752 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:44:57.752 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:44:57.752 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:44:57.752 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:44:57.752 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:44:57.752 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:44:57.752 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:44:57.752 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:44:57.752 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:44:57.752 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:44:57.752 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:44:57.776 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:44:57.776 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:44:57.776 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:44:57.777 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:44:57.915 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:44:57.916 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:44:57.916 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:44:57.916 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:44:57.916 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:44:57.917 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:44:57.917 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:44:57.917 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:44:57.917 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:44:57.917 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:44:57.917 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:44:57.918 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:44:57.918 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:44:57.919 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:44:57.919 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:44:57.920 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:44:57.968 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 31 2024-04-06T09:44:57.968 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 33 2024-04-06T09:44:57.968 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:44:57.969 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=133, tag_tid=21, entry_tid=1], [object_number=132, tag_tid=21, entry_tid=0], [object_number=135, tag_tid=20, entry_tid=19939], [object_number=134, tag_tid=20, entry_tid=19938]]], state=connected] 2024-04-06T09:44:57.969 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=127, tag_tid=20, entry_tid=11579], [object_number=126, tag_tid=20, entry_tid=11578], [object_number=125, tag_tid=20, entry_tid=11577], [object_number=124, tag_tid=20, entry_tid=11576]]], state=connected] 2024-04-06T09:44:57.972 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:44:57.973 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=133, tag_tid=21, entry_tid=1' 2024-04-06T09:44:57.974 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:44:57.974 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:44:57.974 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:44:57.974 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:44:57.974 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:44:57.974 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:44:57.975 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:44:57.975 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:44:57.975 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:44:57.975 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:44:57.975 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:44:57.975 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:44:57.975 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:44:57.976 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:44:57.976 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:44:58.024 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 31 2024-04-06T09:44:58.024 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 33 2024-04-06T09:44:58.024 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:44:58.024 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=133, tag_tid=21, entry_tid=1], [object_number=132, tag_tid=21, entry_tid=0], [object_number=135, tag_tid=20, entry_tid=19939], [object_number=134, tag_tid=20, entry_tid=19938]]], state=connected] 2024-04-06T09:44:58.024 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=127, tag_tid=20, entry_tid=11579], [object_number=126, tag_tid=20, entry_tid=11578], [object_number=125, tag_tid=20, entry_tid=11577], [object_number=124, tag_tid=20, entry_tid=11576]]], state=connected] 2024-04-06T09:44:58.028 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:44:58.030 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=127, tag_tid=20, entry_tid=11579' 2024-04-06T09:44:58.030 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=133, tag_tid=21, entry_tid=1' -a 'object_number=133, tag_tid=21, entry_tid=1' = 'object_number=127, tag_tid=20, entry_tid=11579' 2024-04-06T09:44:58.030 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=127, tag_tid=20, entry_tid=11579' '!=' 'object_number=127, tag_tid=20, entry_tid=11491' 2024-04-06T09:44:58.030 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:44:58.031 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=127, tag_tid=20, entry_tid=11579' = 'object_number=127, tag_tid=20, entry_tid=11491' 2024-04-06T09:44:58.031 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=127, tag_tid=20, entry_tid=11579' 2024-04-06T09:44:58.031 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=133, tag_tid=21, entry_tid=1' 2024-04-06T09:44:58.031 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:44:58.031 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:44:58.032 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=21 2024-04-06T09:44:58.032 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=127, tag_tid=20, entry_tid=11579' 2024-04-06T09:44:58.032 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:44:58.032 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:44:58.033 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=20 2024-04-06T09:44:58.033 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=133, tag_tid=21, entry_tid=1' 2024-04-06T09:44:58.033 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:44:58.034 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:44:58.034 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:44:58.035 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=127, tag_tid=20, entry_tid=11579' 2024-04-06T09:44:58.035 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:44:58.035 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:44:58.036 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=11579 2024-04-06T09:44:58.037 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 21 = 20 -a 1 -le 11579 2024-04-06T09:44:58.037 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:44:58.037 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:44:58.037 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:44:58.239 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:44:58.239 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:44:58.239 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:44:58.239 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:44:58.239 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:44:58.239 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:44:58.239 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:44:58.240 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:44:58.240 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:44:58.240 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:44:58.240 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:44:58.240 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:44:58.240 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:44:58.240 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:44:58.240 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:44:58.240 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:44:58.241 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:44:58.244 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:44:58.244 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:44:58.245 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:44:58.245 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:44:58.245 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:44:58.245 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:44:58.245 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:44:58.245 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:44:58.245 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:44:58.245 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:44:58.245 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:44:58.245 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:44:58.245 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:44:58.245 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:44:58.245 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:44:58.245 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:44:58.246 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:44:58.246 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:44:58.246 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:44:58.246 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:44:58.246 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:44:58.246 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:44:58.246 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:44:58.246 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:44:58.246 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:44:58.246 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:44:58.246 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:44:58.246 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:44:58.246 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:44:58.246 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:44:58.247 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:44:58.247 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:44:58.270 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:44:58.270 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:44:58.270 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:44:58.270 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:44:58.400 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:44:58.401 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:44:58.401 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:44:58.401 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:44:58.401 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:44:58.401 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:44:58.401 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:44:58.401 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:44:58.401 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:44:58.401 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:44:58.402 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:44:58.402 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:44:58.402 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:44:58.403 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:44:58.404 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:44:58.404 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:44:58.457 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 31 2024-04-06T09:44:58.457 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 33 2024-04-06T09:44:58.457 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:44:58.457 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=133, tag_tid=21, entry_tid=1], [object_number=132, tag_tid=21, entry_tid=0], [object_number=135, tag_tid=20, entry_tid=19939], [object_number=134, tag_tid=20, entry_tid=19938]]], state=connected] 2024-04-06T09:44:58.457 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=125, tag_tid=20, entry_tid=11693], [object_number=124, tag_tid=20, entry_tid=11692], [object_number=127, tag_tid=20, entry_tid=11691], [object_number=126, tag_tid=20, entry_tid=11690]]], state=connected] 2024-04-06T09:44:58.460 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:44:58.462 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=133, tag_tid=21, entry_tid=1' 2024-04-06T09:44:58.463 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:44:58.463 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:44:58.463 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:44:58.463 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:44:58.463 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:44:58.463 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:44:58.463 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:44:58.463 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:44:58.463 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:44:58.464 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:44:58.464 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:44:58.464 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:44:58.464 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:44:58.464 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:44:58.464 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:44:58.512 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 31 2024-04-06T09:44:58.512 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 33 2024-04-06T09:44:58.512 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:44:58.512 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=133, tag_tid=21, entry_tid=1], [object_number=132, tag_tid=21, entry_tid=0], [object_number=135, tag_tid=20, entry_tid=19939], [object_number=134, tag_tid=20, entry_tid=19938]]], state=connected] 2024-04-06T09:44:58.512 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=125, tag_tid=20, entry_tid=11693], [object_number=124, tag_tid=20, entry_tid=11692], [object_number=127, tag_tid=20, entry_tid=11691], [object_number=126, tag_tid=20, entry_tid=11690]]], state=connected] 2024-04-06T09:44:58.515 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:44:58.517 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=125, tag_tid=20, entry_tid=11693' 2024-04-06T09:44:58.517 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=133, tag_tid=21, entry_tid=1' -a 'object_number=133, tag_tid=21, entry_tid=1' = 'object_number=125, tag_tid=20, entry_tid=11693' 2024-04-06T09:44:58.517 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=125, tag_tid=20, entry_tid=11693' '!=' 'object_number=127, tag_tid=20, entry_tid=11579' 2024-04-06T09:44:58.517 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:44:58.517 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=125, tag_tid=20, entry_tid=11693' = 'object_number=127, tag_tid=20, entry_tid=11579' 2024-04-06T09:44:58.518 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=125, tag_tid=20, entry_tid=11693' 2024-04-06T09:44:58.518 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=133, tag_tid=21, entry_tid=1' 2024-04-06T09:44:58.518 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:44:58.518 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:44:58.518 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=21 2024-04-06T09:44:58.519 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=125, tag_tid=20, entry_tid=11693' 2024-04-06T09:44:58.519 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:44:58.519 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:44:58.520 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=20 2024-04-06T09:44:58.521 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=133, tag_tid=21, entry_tid=1' 2024-04-06T09:44:58.521 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:44:58.521 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:44:58.522 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:44:58.523 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=125, tag_tid=20, entry_tid=11693' 2024-04-06T09:44:58.523 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:44:58.523 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:44:58.524 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=11693 2024-04-06T09:44:58.524 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 21 = 20 -a 1 -le 11693 2024-04-06T09:44:58.524 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:44:58.524 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:44:58.524 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:44:58.725 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:44:58.726 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:44:58.726 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:44:58.726 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:44:58.726 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:44:58.726 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:44:58.726 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:44:58.726 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:44:58.726 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:44:58.726 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:44:58.726 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:44:58.726 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:44:58.726 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:44:58.727 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:44:58.727 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:44:58.727 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:44:58.727 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:44:58.730 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:44:58.730 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:44:58.730 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:44:58.730 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:44:58.730 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:44:58.730 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:44:58.731 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:44:58.731 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:44:58.731 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:44:58.731 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:44:58.731 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:44:58.731 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:44:58.731 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:44:58.731 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:44:58.731 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:44:58.731 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:44:58.731 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:44:58.731 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:44:58.731 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:44:58.731 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:44:58.731 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:44:58.732 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:44:58.732 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:44:58.732 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:44:58.732 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:44:58.732 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:44:58.732 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:44:58.732 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:44:58.732 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:44:58.732 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:44:58.732 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:44:58.733 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:44:58.756 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:44:58.756 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:44:58.756 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:44:58.756 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:44:58.887 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:44:58.888 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:44:58.888 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:44:58.888 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:44:58.889 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:44:58.889 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:44:58.889 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:44:58.889 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:44:58.889 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:44:58.889 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:44:58.889 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:44:58.890 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:44:58.890 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:44:58.891 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:44:58.891 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:44:58.891 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:44:59.148 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 31 2024-04-06T09:44:59.148 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 33 2024-04-06T09:44:59.148 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:44:59.148 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=133, tag_tid=21, entry_tid=1], [object_number=132, tag_tid=21, entry_tid=0], [object_number=135, tag_tid=20, entry_tid=19939], [object_number=134, tag_tid=20, entry_tid=19938]]], state=connected] 2024-04-06T09:44:59.148 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=127, tag_tid=20, entry_tid=11803], [object_number=126, tag_tid=20, entry_tid=11802], [object_number=125, tag_tid=20, entry_tid=11801], [object_number=124, tag_tid=20, entry_tid=11800]]], state=connected] 2024-04-06T09:44:59.152 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:44:59.154 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=133, tag_tid=21, entry_tid=1' 2024-04-06T09:44:59.155 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:44:59.155 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:44:59.155 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:44:59.155 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:44:59.155 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:44:59.155 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:44:59.155 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:44:59.155 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:44:59.155 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:44:59.156 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:44:59.157 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:44:59.157 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:44:59.157 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:44:59.157 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:44:59.157 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:44:59.205 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 31 2024-04-06T09:44:59.206 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 33 2024-04-06T09:44:59.206 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:44:59.206 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=133, tag_tid=21, entry_tid=1], [object_number=132, tag_tid=21, entry_tid=0], [object_number=135, tag_tid=20, entry_tid=19939], [object_number=134, tag_tid=20, entry_tid=19938]]], state=connected] 2024-04-06T09:44:59.206 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=127, tag_tid=20, entry_tid=11803], [object_number=126, tag_tid=20, entry_tid=11802], [object_number=125, tag_tid=20, entry_tid=11801], [object_number=124, tag_tid=20, entry_tid=11800]]], state=connected] 2024-04-06T09:44:59.209 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:44:59.211 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=127, tag_tid=20, entry_tid=11803' 2024-04-06T09:44:59.212 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=133, tag_tid=21, entry_tid=1' -a 'object_number=133, tag_tid=21, entry_tid=1' = 'object_number=127, tag_tid=20, entry_tid=11803' 2024-04-06T09:44:59.212 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=127, tag_tid=20, entry_tid=11803' '!=' 'object_number=125, tag_tid=20, entry_tid=11693' 2024-04-06T09:44:59.212 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:44:59.212 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=127, tag_tid=20, entry_tid=11803' = 'object_number=125, tag_tid=20, entry_tid=11693' 2024-04-06T09:44:59.212 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=127, tag_tid=20, entry_tid=11803' 2024-04-06T09:44:59.212 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=133, tag_tid=21, entry_tid=1' 2024-04-06T09:44:59.212 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:44:59.212 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:44:59.212 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=21 2024-04-06T09:44:59.213 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=127, tag_tid=20, entry_tid=11803' 2024-04-06T09:44:59.213 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:44:59.213 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:44:59.214 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=20 2024-04-06T09:44:59.215 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=133, tag_tid=21, entry_tid=1' 2024-04-06T09:44:59.215 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:44:59.215 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:44:59.216 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:44:59.217 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=127, tag_tid=20, entry_tid=11803' 2024-04-06T09:44:59.217 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:44:59.217 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:44:59.218 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=11803 2024-04-06T09:44:59.218 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 21 = 20 -a 1 -le 11803 2024-04-06T09:44:59.218 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:44:59.218 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:44:59.218 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:44:59.420 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:44:59.420 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:44:59.420 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:44:59.420 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:44:59.420 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:44:59.420 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:44:59.420 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:44:59.421 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:44:59.421 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:44:59.421 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:44:59.421 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:44:59.421 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:44:59.421 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:44:59.421 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:44:59.421 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:44:59.421 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:44:59.422 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:44:59.424 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:44:59.424 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:44:59.424 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:44:59.424 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:44:59.424 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:44:59.424 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:44:59.424 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:44:59.425 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:44:59.425 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:44:59.425 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:44:59.425 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:44:59.425 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:44:59.425 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:44:59.425 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:44:59.425 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:44:59.425 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:44:59.426 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:44:59.426 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:44:59.426 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:44:59.427 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:44:59.427 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:44:59.427 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:44:59.427 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:44:59.427 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:44:59.427 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:44:59.427 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:44:59.427 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:44:59.427 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:44:59.427 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:44:59.427 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:44:59.427 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:44:59.427 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:44:59.452 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:44:59.453 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:44:59.453 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:44:59.453 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:44:59.584 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:44:59.585 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:44:59.585 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:44:59.585 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:44:59.585 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:44:59.585 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:44:59.585 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:44:59.585 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:44:59.585 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:44:59.585 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:44:59.586 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:44:59.586 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:44:59.586 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:44:59.588 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:44:59.588 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:44:59.588 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:44:59.634 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 31 2024-04-06T09:44:59.635 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 33 2024-04-06T09:44:59.635 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:44:59.635 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=133, tag_tid=21, entry_tid=1], [object_number=132, tag_tid=21, entry_tid=0], [object_number=135, tag_tid=20, entry_tid=19939], [object_number=134, tag_tid=20, entry_tid=19938]]], state=connected] 2024-04-06T09:44:59.635 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=124, tag_tid=20, entry_tid=11932], [object_number=127, tag_tid=20, entry_tid=11931], [object_number=126, tag_tid=20, entry_tid=11930], [object_number=125, tag_tid=20, entry_tid=11929]]], state=connected] 2024-04-06T09:44:59.639 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:44:59.640 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=133, tag_tid=21, entry_tid=1' 2024-04-06T09:44:59.641 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:44:59.641 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:44:59.641 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:44:59.641 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:44:59.641 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:44:59.641 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:44:59.641 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:44:59.642 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:44:59.642 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:44:59.642 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:44:59.642 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:44:59.642 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:44:59.642 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:44:59.642 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:44:59.643 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:44:59.690 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 31 2024-04-06T09:44:59.690 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 33 2024-04-06T09:44:59.690 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:44:59.690 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=133, tag_tid=21, entry_tid=1], [object_number=132, tag_tid=21, entry_tid=0], [object_number=135, tag_tid=20, entry_tid=19939], [object_number=134, tag_tid=20, entry_tid=19938]]], state=connected] 2024-04-06T09:44:59.690 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=124, tag_tid=20, entry_tid=11932], [object_number=127, tag_tid=20, entry_tid=11931], [object_number=126, tag_tid=20, entry_tid=11930], [object_number=125, tag_tid=20, entry_tid=11929]]], state=connected] 2024-04-06T09:44:59.693 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:44:59.695 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=124, tag_tid=20, entry_tid=11932' 2024-04-06T09:44:59.695 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=133, tag_tid=21, entry_tid=1' -a 'object_number=133, tag_tid=21, entry_tid=1' = 'object_number=124, tag_tid=20, entry_tid=11932' 2024-04-06T09:44:59.695 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=124, tag_tid=20, entry_tid=11932' '!=' 'object_number=127, tag_tid=20, entry_tid=11803' 2024-04-06T09:44:59.695 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:44:59.696 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=124, tag_tid=20, entry_tid=11932' = 'object_number=127, tag_tid=20, entry_tid=11803' 2024-04-06T09:44:59.696 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=124, tag_tid=20, entry_tid=11932' 2024-04-06T09:44:59.696 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=133, tag_tid=21, entry_tid=1' 2024-04-06T09:44:59.696 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:44:59.696 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:44:59.697 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=21 2024-04-06T09:44:59.697 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=124, tag_tid=20, entry_tid=11932' 2024-04-06T09:44:59.697 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:44:59.698 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:44:59.698 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=20 2024-04-06T09:44:59.699 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=133, tag_tid=21, entry_tid=1' 2024-04-06T09:44:59.699 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:44:59.699 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:44:59.700 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:44:59.701 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=124, tag_tid=20, entry_tid=11932' 2024-04-06T09:44:59.701 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:44:59.701 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:44:59.702 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=11932 2024-04-06T09:44:59.702 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 21 = 20 -a 1 -le 11932 2024-04-06T09:44:59.702 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:44:59.702 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:44:59.702 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:44:59.904 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:44:59.904 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:44:59.904 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:44:59.904 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:44:59.905 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:44:59.905 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:44:59.905 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:44:59.905 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:44:59.905 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:44:59.905 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:44:59.905 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:44:59.905 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:44:59.905 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:44:59.905 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:44:59.905 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:44:59.906 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:44:59.906 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:44:59.908 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:44:59.908 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:44:59.908 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:44:59.908 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:44:59.908 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:44:59.908 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:44:59.908 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:44:59.908 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:44:59.908 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:44:59.909 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:44:59.909 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:44:59.909 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:44:59.909 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:44:59.909 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:44:59.909 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:44:59.909 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:44:59.910 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:44:59.910 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:44:59.910 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:44:59.910 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:44:59.910 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:44:59.910 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:44:59.910 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:44:59.910 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:44:59.910 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:44:59.910 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:44:59.910 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:44:59.911 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:44:59.911 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:44:59.911 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:44:59.911 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:44:59.911 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:44:59.935 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:44:59.935 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:44:59.935 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:44:59.935 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:45:00.068 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:45:00.069 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:45:00.069 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:45:00.069 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:45:00.069 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:45:00.069 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:45:00.069 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:45:00.070 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:45:00.070 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:45:00.070 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:45:00.070 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:45:00.070 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:45:00.070 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:45:00.073 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:45:00.073 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:45:00.074 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:45:00.130 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 31 2024-04-06T09:45:00.130 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 33 2024-04-06T09:45:00.130 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:45:00.130 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=133, tag_tid=21, entry_tid=1], [object_number=132, tag_tid=21, entry_tid=0], [object_number=135, tag_tid=20, entry_tid=19939], [object_number=134, tag_tid=20, entry_tid=19938]]], state=connected] 2024-04-06T09:45:00.131 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=127, tag_tid=20, entry_tid=12003], [object_number=126, tag_tid=20, entry_tid=12002], [object_number=125, tag_tid=20, entry_tid=12001], [object_number=124, tag_tid=20, entry_tid=12000]]], state=connected] 2024-04-06T09:45:00.134 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:45:00.135 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=133, tag_tid=21, entry_tid=1' 2024-04-06T09:45:00.137 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:45:00.137 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:45:00.137 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:45:00.137 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:45:00.137 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:45:00.137 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:45:00.138 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:45:00.138 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:45:00.138 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:45:00.138 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:45:00.138 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:45:00.138 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:45:00.139 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:45:00.139 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:45:00.139 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:45:00.185 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 31 2024-04-06T09:45:00.185 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 33 2024-04-06T09:45:00.185 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:45:00.185 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=133, tag_tid=21, entry_tid=1], [object_number=132, tag_tid=21, entry_tid=0], [object_number=135, tag_tid=20, entry_tid=19939], [object_number=134, tag_tid=20, entry_tid=19938]]], state=connected] 2024-04-06T09:45:00.185 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=127, tag_tid=20, entry_tid=12003], [object_number=126, tag_tid=20, entry_tid=12002], [object_number=125, tag_tid=20, entry_tid=12001], [object_number=124, tag_tid=20, entry_tid=12000]]], state=connected] 2024-04-06T09:45:00.189 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:45:00.190 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=127, tag_tid=20, entry_tid=12003' 2024-04-06T09:45:00.191 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=133, tag_tid=21, entry_tid=1' -a 'object_number=133, tag_tid=21, entry_tid=1' = 'object_number=127, tag_tid=20, entry_tid=12003' 2024-04-06T09:45:00.191 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=127, tag_tid=20, entry_tid=12003' '!=' 'object_number=124, tag_tid=20, entry_tid=11932' 2024-04-06T09:45:00.191 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:45:00.191 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=127, tag_tid=20, entry_tid=12003' = 'object_number=124, tag_tid=20, entry_tid=11932' 2024-04-06T09:45:00.191 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=127, tag_tid=20, entry_tid=12003' 2024-04-06T09:45:00.191 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=133, tag_tid=21, entry_tid=1' 2024-04-06T09:45:00.192 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:45:00.192 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:45:00.192 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=21 2024-04-06T09:45:00.193 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=127, tag_tid=20, entry_tid=12003' 2024-04-06T09:45:00.193 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:45:00.193 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:45:00.194 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=20 2024-04-06T09:45:00.194 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=133, tag_tid=21, entry_tid=1' 2024-04-06T09:45:00.195 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:45:00.195 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:45:00.196 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:45:00.196 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=127, tag_tid=20, entry_tid=12003' 2024-04-06T09:45:00.196 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:45:00.196 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:45:00.198 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=12003 2024-04-06T09:45:00.198 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 21 = 20 -a 1 -le 12003 2024-04-06T09:45:00.198 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:45:00.198 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:45:00.198 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:45:00.400 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:45:00.400 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:45:00.400 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:45:00.400 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:45:00.400 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:45:00.400 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:45:00.400 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:45:00.401 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:45:00.401 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:45:00.401 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:45:00.401 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:45:00.401 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:45:00.401 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:45:00.401 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:45:00.401 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:45:00.401 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:45:00.402 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:45:00.402 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:45:00.402 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:45:00.402 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:45:00.403 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:45:00.403 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:45:00.403 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:45:00.403 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:45:00.403 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:45:00.403 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:45:00.403 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:45:00.403 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:45:00.403 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:45:00.403 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:45:00.403 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:45:00.403 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:45:00.403 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:45:00.405 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:45:00.405 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:45:00.405 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:45:00.405 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:45:00.405 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:45:00.405 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:45:00.405 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:45:00.405 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:45:00.405 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:45:00.405 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:45:00.405 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:45:00.405 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:45:00.405 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:45:00.406 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:45:00.406 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:45:00.406 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:45:00.430 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:45:00.431 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:45:00.431 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:45:00.431 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:45:00.567 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:45:00.567 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:45:00.567 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:45:00.567 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:45:00.568 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:45:00.568 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:45:00.568 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:45:00.568 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:45:00.568 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:45:00.568 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:45:00.568 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:45:00.568 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:45:00.569 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:45:00.570 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:45:00.570 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:45:00.570 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:45:00.618 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 31 2024-04-06T09:45:00.618 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 33 2024-04-06T09:45:00.618 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:45:00.618 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=133, tag_tid=21, entry_tid=1], [object_number=132, tag_tid=21, entry_tid=0], [object_number=135, tag_tid=20, entry_tid=19939], [object_number=134, tag_tid=20, entry_tid=19938]]], state=connected] 2024-04-06T09:45:00.618 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=125, tag_tid=20, entry_tid=12097], [object_number=124, tag_tid=20, entry_tid=12096], [object_number=127, tag_tid=20, entry_tid=12095], [object_number=126, tag_tid=20, entry_tid=12094]]], state=connected] 2024-04-06T09:45:00.621 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:45:00.623 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=133, tag_tid=21, entry_tid=1' 2024-04-06T09:45:00.624 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:45:00.624 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:45:00.624 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:45:00.624 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:45:00.624 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:45:00.624 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:45:00.624 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:45:00.624 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:45:00.624 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:45:00.624 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:45:00.624 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:45:00.624 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:45:00.625 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:45:00.625 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:45:00.626 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:45:00.672 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 31 2024-04-06T09:45:00.672 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 33 2024-04-06T09:45:00.672 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:45:00.672 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=133, tag_tid=21, entry_tid=1], [object_number=132, tag_tid=21, entry_tid=0], [object_number=135, tag_tid=20, entry_tid=19939], [object_number=134, tag_tid=20, entry_tid=19938]]], state=connected] 2024-04-06T09:45:00.672 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=125, tag_tid=20, entry_tid=12097], [object_number=124, tag_tid=20, entry_tid=12096], [object_number=127, tag_tid=20, entry_tid=12095], [object_number=126, tag_tid=20, entry_tid=12094]]], state=connected] 2024-04-06T09:45:00.676 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:45:00.678 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=125, tag_tid=20, entry_tid=12097' 2024-04-06T09:45:00.678 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=133, tag_tid=21, entry_tid=1' -a 'object_number=133, tag_tid=21, entry_tid=1' = 'object_number=125, tag_tid=20, entry_tid=12097' 2024-04-06T09:45:00.678 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=125, tag_tid=20, entry_tid=12097' '!=' 'object_number=127, tag_tid=20, entry_tid=12003' 2024-04-06T09:45:00.678 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:45:00.678 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=125, tag_tid=20, entry_tid=12097' = 'object_number=127, tag_tid=20, entry_tid=12003' 2024-04-06T09:45:00.678 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=125, tag_tid=20, entry_tid=12097' 2024-04-06T09:45:00.679 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=133, tag_tid=21, entry_tid=1' 2024-04-06T09:45:00.679 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:45:00.679 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:45:00.679 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=21 2024-04-06T09:45:00.680 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=125, tag_tid=20, entry_tid=12097' 2024-04-06T09:45:00.680 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:45:00.680 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:45:00.681 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=20 2024-04-06T09:45:00.681 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=133, tag_tid=21, entry_tid=1' 2024-04-06T09:45:00.682 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:45:00.682 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:45:00.683 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:45:00.683 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=125, tag_tid=20, entry_tid=12097' 2024-04-06T09:45:00.683 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:45:00.683 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:45:00.684 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=12097 2024-04-06T09:45:00.685 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 21 = 20 -a 1 -le 12097 2024-04-06T09:45:00.685 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:45:00.685 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:45:00.685 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:45:00.887 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:45:00.888 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:45:00.888 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:45:00.888 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:45:00.888 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:45:00.888 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:45:00.888 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:45:00.888 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:45:00.889 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:45:00.889 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:45:00.889 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:45:00.889 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:45:00.889 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:45:00.889 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:45:00.890 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:45:00.893 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:45:00.893 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:45:00.894 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:45:00.894 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:45:00.894 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:45:00.894 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:45:00.894 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:45:00.894 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:45:00.895 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:45:00.895 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:45:00.895 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:45:00.895 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:45:00.895 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:45:00.895 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:45:00.896 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:45:00.896 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:45:00.896 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:45:00.896 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:45:00.896 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:45:00.896 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:45:00.897 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:45:00.897 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:45:00.897 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:45:00.897 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:45:00.897 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:45:00.898 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:45:00.898 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:45:00.898 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:45:00.898 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:45:00.898 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:45:00.898 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:45:00.899 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:45:00.899 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:45:00.899 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:45:00.919 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:45:00.919 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:45:00.919 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:45:00.919 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:45:01.056 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:45:01.057 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:45:01.058 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:45:01.058 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:45:01.058 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:45:01.058 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:45:01.058 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:45:01.058 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:45:01.058 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:45:01.058 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:45:01.059 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:45:01.059 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:45:01.059 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:45:01.061 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:45:01.061 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:45:01.062 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:45:01.113 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 31 2024-04-06T09:45:01.114 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 33 2024-04-06T09:45:01.114 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:45:01.114 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=133, tag_tid=21, entry_tid=1], [object_number=132, tag_tid=21, entry_tid=0], [object_number=135, tag_tid=20, entry_tid=19939], [object_number=134, tag_tid=20, entry_tid=19938]]], state=connected] 2024-04-06T09:45:01.114 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=125, tag_tid=20, entry_tid=12193], [object_number=124, tag_tid=20, entry_tid=12192], [object_number=127, tag_tid=20, entry_tid=12191], [object_number=126, tag_tid=20, entry_tid=12190]]], state=connected] 2024-04-06T09:45:01.117 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:45:01.119 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=133, tag_tid=21, entry_tid=1' 2024-04-06T09:45:01.120 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:45:01.120 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:45:01.120 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:45:01.120 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:45:01.120 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:45:01.120 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:45:01.120 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:45:01.121 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:45:01.121 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:45:01.121 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:45:01.121 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:45:01.121 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:45:01.121 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:45:01.122 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:45:01.122 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:45:01.168 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 31 2024-04-06T09:45:01.168 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 33 2024-04-06T09:45:01.168 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:45:01.169 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=133, tag_tid=21, entry_tid=1], [object_number=132, tag_tid=21, entry_tid=0], [object_number=135, tag_tid=20, entry_tid=19939], [object_number=134, tag_tid=20, entry_tid=19938]]], state=connected] 2024-04-06T09:45:01.169 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=125, tag_tid=20, entry_tid=12193], [object_number=124, tag_tid=20, entry_tid=12192], [object_number=127, tag_tid=20, entry_tid=12191], [object_number=126, tag_tid=20, entry_tid=12190]]], state=connected] 2024-04-06T09:45:01.172 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:45:01.174 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=125, tag_tid=20, entry_tid=12193' 2024-04-06T09:45:01.174 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=133, tag_tid=21, entry_tid=1' -a 'object_number=133, tag_tid=21, entry_tid=1' = 'object_number=125, tag_tid=20, entry_tid=12193' 2024-04-06T09:45:01.174 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=125, tag_tid=20, entry_tid=12193' '!=' 'object_number=125, tag_tid=20, entry_tid=12097' 2024-04-06T09:45:01.174 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:45:01.174 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=125, tag_tid=20, entry_tid=12193' = 'object_number=125, tag_tid=20, entry_tid=12097' 2024-04-06T09:45:01.174 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=125, tag_tid=20, entry_tid=12193' 2024-04-06T09:45:01.174 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=133, tag_tid=21, entry_tid=1' 2024-04-06T09:45:01.175 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:45:01.175 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:45:01.175 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=21 2024-04-06T09:45:01.175 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=125, tag_tid=20, entry_tid=12193' 2024-04-06T09:45:01.176 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:45:01.176 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:45:01.176 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=20 2024-04-06T09:45:01.177 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=133, tag_tid=21, entry_tid=1' 2024-04-06T09:45:01.177 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:45:01.177 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:45:01.178 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:45:01.179 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=125, tag_tid=20, entry_tid=12193' 2024-04-06T09:45:01.179 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:45:01.179 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:45:01.180 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=12193 2024-04-06T09:45:01.181 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 21 = 20 -a 1 -le 12193 2024-04-06T09:45:01.181 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:45:01.181 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:45:01.181 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:45:01.382 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:45:01.382 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:45:01.382 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:45:01.382 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:45:01.382 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:45:01.383 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:45:01.383 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:45:01.383 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:45:01.383 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:45:01.383 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:45:01.383 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:45:01.383 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:45:01.383 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:45:01.383 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:45:01.383 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:45:01.384 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:45:01.384 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:45:01.387 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:45:01.387 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:45:01.387 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:45:01.387 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:45:01.387 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:45:01.387 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:45:01.387 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:45:01.387 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:45:01.387 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:45:01.387 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:45:01.387 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:45:01.387 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:45:01.388 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:45:01.388 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:45:01.388 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:45:01.388 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:45:01.388 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:45:01.388 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:45:01.388 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:45:01.388 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:45:01.388 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:45:01.388 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:45:01.388 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:45:01.388 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:45:01.388 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:45:01.388 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:45:01.388 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:45:01.389 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:45:01.389 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:45:01.389 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:45:01.389 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:45:01.389 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:45:01.414 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:45:01.414 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:45:01.414 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:45:01.415 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:45:01.540 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:45:01.541 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:45:01.541 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:45:01.541 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:45:01.541 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:45:01.541 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:45:01.541 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:45:01.541 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:45:01.541 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:45:01.541 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:45:01.542 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:45:01.542 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:45:01.542 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:45:01.543 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:45:01.543 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:45:01.544 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:45:01.596 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 31 2024-04-06T09:45:01.596 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 33 2024-04-06T09:45:01.596 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:45:01.596 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=133, tag_tid=21, entry_tid=1], [object_number=132, tag_tid=21, entry_tid=0], [object_number=135, tag_tid=20, entry_tid=19939], [object_number=134, tag_tid=20, entry_tid=19938]]], state=connected] 2024-04-06T09:45:01.596 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=125, tag_tid=20, entry_tid=12285], [object_number=124, tag_tid=20, entry_tid=12284], [object_number=127, tag_tid=20, entry_tid=12283], [object_number=126, tag_tid=20, entry_tid=12282]]], state=connected] 2024-04-06T09:45:01.600 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:45:01.601 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=133, tag_tid=21, entry_tid=1' 2024-04-06T09:45:01.603 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:45:01.603 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:45:01.603 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:45:01.603 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:45:01.603 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:45:01.603 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:45:01.603 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:45:01.603 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:45:01.603 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:45:01.603 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:45:01.603 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:45:01.603 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:45:01.604 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:45:01.604 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:45:01.604 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:45:01.652 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 31 2024-04-06T09:45:01.652 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 33 2024-04-06T09:45:01.652 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:45:01.652 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=133, tag_tid=21, entry_tid=1], [object_number=132, tag_tid=21, entry_tid=0], [object_number=135, tag_tid=20, entry_tid=19939], [object_number=134, tag_tid=20, entry_tid=19938]]], state=connected] 2024-04-06T09:45:01.652 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=125, tag_tid=20, entry_tid=12285], [object_number=124, tag_tid=20, entry_tid=12284], [object_number=127, tag_tid=20, entry_tid=12283], [object_number=126, tag_tid=20, entry_tid=12282]]], state=connected] 2024-04-06T09:45:01.656 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:45:01.658 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=125, tag_tid=20, entry_tid=12285' 2024-04-06T09:45:01.658 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=133, tag_tid=21, entry_tid=1' -a 'object_number=133, tag_tid=21, entry_tid=1' = 'object_number=125, tag_tid=20, entry_tid=12285' 2024-04-06T09:45:01.658 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=125, tag_tid=20, entry_tid=12285' '!=' 'object_number=125, tag_tid=20, entry_tid=12193' 2024-04-06T09:45:01.658 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:45:01.658 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=125, tag_tid=20, entry_tid=12285' = 'object_number=125, tag_tid=20, entry_tid=12193' 2024-04-06T09:45:01.658 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=125, tag_tid=20, entry_tid=12285' 2024-04-06T09:45:01.658 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=133, tag_tid=21, entry_tid=1' 2024-04-06T09:45:01.658 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:45:01.658 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:45:01.659 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=21 2024-04-06T09:45:01.660 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=125, tag_tid=20, entry_tid=12285' 2024-04-06T09:45:01.660 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:45:01.660 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:45:01.661 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=20 2024-04-06T09:45:01.661 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=133, tag_tid=21, entry_tid=1' 2024-04-06T09:45:01.661 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:45:01.662 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:45:01.662 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:45:01.663 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=125, tag_tid=20, entry_tid=12285' 2024-04-06T09:45:01.663 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:45:01.663 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:45:01.665 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=12285 2024-04-06T09:45:01.665 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 21 = 20 -a 1 -le 12285 2024-04-06T09:45:01.665 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:45:01.665 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:45:01.665 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:45:01.867 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:45:01.867 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:45:01.867 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:45:01.867 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:45:01.867 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:45:01.867 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:45:01.867 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:45:01.867 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:45:01.867 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:45:01.867 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:45:01.867 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:45:01.868 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:45:01.868 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:45:01.868 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:45:01.868 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:45:01.868 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:45:01.868 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:45:01.871 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:45:01.871 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:45:01.871 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:45:01.871 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:45:01.871 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:45:01.871 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:45:01.871 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:45:01.872 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:45:01.872 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:45:01.872 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:45:01.872 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:45:01.872 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:45:01.872 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:45:01.872 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:45:01.872 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:45:01.872 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:45:01.872 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:45:01.872 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:45:01.872 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:45:01.872 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:45:01.872 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:45:01.873 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:45:01.873 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:45:01.873 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:45:01.873 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:45:01.873 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:45:01.873 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:45:01.873 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:45:01.873 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:45:01.873 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:45:01.873 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:45:01.873 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:45:01.898 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:45:01.898 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:45:01.898 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:45:01.898 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:45:02.023 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:45:02.024 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:45:02.024 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:45:02.024 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:45:02.025 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:45:02.025 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:45:02.025 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:45:02.025 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:45:02.025 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:45:02.025 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:45:02.025 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:45:02.026 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:45:02.026 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:45:02.027 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:45:02.028 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:45:02.028 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:45:02.081 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 31 2024-04-06T09:45:02.081 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 33 2024-04-06T09:45:02.081 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:45:02.081 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=133, tag_tid=21, entry_tid=1], [object_number=132, tag_tid=21, entry_tid=0], [object_number=135, tag_tid=20, entry_tid=19939], [object_number=134, tag_tid=20, entry_tid=19938]]], state=connected] 2024-04-06T09:45:02.081 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=126, tag_tid=20, entry_tid=12374], [object_number=125, tag_tid=20, entry_tid=12373], [object_number=124, tag_tid=20, entry_tid=12372], [object_number=127, tag_tid=20, entry_tid=12371]]], state=connected] 2024-04-06T09:45:02.085 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:45:02.086 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=133, tag_tid=21, entry_tid=1' 2024-04-06T09:45:02.087 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:45:02.087 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:45:02.087 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:45:02.088 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:45:02.088 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:45:02.088 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:45:02.088 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:45:02.088 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:45:02.088 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:45:02.088 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:45:02.088 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:45:02.088 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:45:02.088 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:45:02.089 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:45:02.089 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:45:02.137 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 31 2024-04-06T09:45:02.137 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 33 2024-04-06T09:45:02.137 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:45:02.137 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=133, tag_tid=21, entry_tid=1], [object_number=132, tag_tid=21, entry_tid=0], [object_number=135, tag_tid=20, entry_tid=19939], [object_number=134, tag_tid=20, entry_tid=19938]]], state=connected] 2024-04-06T09:45:02.137 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=126, tag_tid=20, entry_tid=12374], [object_number=125, tag_tid=20, entry_tid=12373], [object_number=124, tag_tid=20, entry_tid=12372], [object_number=127, tag_tid=20, entry_tid=12371]]], state=connected] 2024-04-06T09:45:02.140 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:45:02.142 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=126, tag_tid=20, entry_tid=12374' 2024-04-06T09:45:02.142 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=133, tag_tid=21, entry_tid=1' -a 'object_number=133, tag_tid=21, entry_tid=1' = 'object_number=126, tag_tid=20, entry_tid=12374' 2024-04-06T09:45:02.142 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=126, tag_tid=20, entry_tid=12374' '!=' 'object_number=125, tag_tid=20, entry_tid=12285' 2024-04-06T09:45:02.142 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:45:02.142 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=126, tag_tid=20, entry_tid=12374' = 'object_number=125, tag_tid=20, entry_tid=12285' 2024-04-06T09:45:02.142 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=126, tag_tid=20, entry_tid=12374' 2024-04-06T09:45:02.142 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=133, tag_tid=21, entry_tid=1' 2024-04-06T09:45:02.142 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:45:02.143 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:45:02.143 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=21 2024-04-06T09:45:02.144 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=126, tag_tid=20, entry_tid=12374' 2024-04-06T09:45:02.144 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:45:02.144 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:45:02.145 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=20 2024-04-06T09:45:02.145 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=133, tag_tid=21, entry_tid=1' 2024-04-06T09:45:02.145 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:45:02.146 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:45:02.146 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:45:02.147 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=126, tag_tid=20, entry_tid=12374' 2024-04-06T09:45:02.147 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:45:02.147 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:45:02.149 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=12374 2024-04-06T09:45:02.149 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 21 = 20 -a 1 -le 12374 2024-04-06T09:45:02.149 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:45:02.149 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:45:02.149 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:45:02.350 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:45:02.350 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:45:02.350 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:45:02.350 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:45:02.351 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:45:02.351 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:45:02.351 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:45:02.351 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:45:02.351 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:45:02.351 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:45:02.351 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:45:02.351 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:45:02.351 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:45:02.351 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:45:02.351 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:45:02.352 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:45:02.352 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:45:02.355 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:45:02.355 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:45:02.355 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:45:02.355 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:45:02.355 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:45:02.356 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:45:02.356 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:45:02.356 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:45:02.356 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:45:02.356 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:45:02.356 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:45:02.356 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:45:02.356 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:45:02.356 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:45:02.356 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:45:02.356 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:45:02.356 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:45:02.356 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:45:02.356 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:45:02.356 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:45:02.356 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:45:02.357 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:45:02.357 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:45:02.357 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:45:02.357 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:45:02.357 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:45:02.357 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:45:02.357 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:45:02.357 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:45:02.357 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:45:02.357 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:45:02.357 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:45:02.382 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:45:02.382 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:45:02.382 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:45:02.382 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:45:02.511 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:45:02.512 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:45:02.512 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:45:02.512 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:45:02.512 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:45:02.512 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:45:02.512 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:45:02.512 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:45:02.512 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:45:02.512 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:45:02.513 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:45:02.513 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:45:02.513 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:45:02.515 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:45:02.515 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:45:02.515 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:45:02.564 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 31 2024-04-06T09:45:02.564 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 33 2024-04-06T09:45:02.565 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:45:02.565 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=133, tag_tid=21, entry_tid=1], [object_number=132, tag_tid=21, entry_tid=0], [object_number=135, tag_tid=20, entry_tid=19939], [object_number=134, tag_tid=20, entry_tid=19938]]], state=connected] 2024-04-06T09:45:02.565 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=124, tag_tid=20, entry_tid=12460], [object_number=127, tag_tid=20, entry_tid=12459], [object_number=126, tag_tid=20, entry_tid=12458], [object_number=125, tag_tid=20, entry_tid=12457]]], state=connected] 2024-04-06T09:45:02.568 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:45:02.569 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=133, tag_tid=21, entry_tid=1' 2024-04-06T09:45:02.571 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:45:02.571 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:45:02.571 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:45:02.571 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:45:02.571 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:45:02.571 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:45:02.571 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:45:02.571 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:45:02.571 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:45:02.571 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:45:02.571 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:45:02.571 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:45:02.572 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:45:02.572 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:45:02.572 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:45:02.620 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 31 2024-04-06T09:45:02.620 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 33 2024-04-06T09:45:02.620 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:45:02.620 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=133, tag_tid=21, entry_tid=1], [object_number=132, tag_tid=21, entry_tid=0], [object_number=135, tag_tid=20, entry_tid=19939], [object_number=134, tag_tid=20, entry_tid=19938]]], state=connected] 2024-04-06T09:45:02.620 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=124, tag_tid=20, entry_tid=12460], [object_number=127, tag_tid=20, entry_tid=12459], [object_number=126, tag_tid=20, entry_tid=12458], [object_number=125, tag_tid=20, entry_tid=12457]]], state=connected] 2024-04-06T09:45:02.623 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:45:02.625 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=124, tag_tid=20, entry_tid=12460' 2024-04-06T09:45:02.626 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=133, tag_tid=21, entry_tid=1' -a 'object_number=133, tag_tid=21, entry_tid=1' = 'object_number=124, tag_tid=20, entry_tid=12460' 2024-04-06T09:45:02.626 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=124, tag_tid=20, entry_tid=12460' '!=' 'object_number=126, tag_tid=20, entry_tid=12374' 2024-04-06T09:45:02.626 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:45:02.626 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=124, tag_tid=20, entry_tid=12460' = 'object_number=126, tag_tid=20, entry_tid=12374' 2024-04-06T09:45:02.626 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=124, tag_tid=20, entry_tid=12460' 2024-04-06T09:45:02.626 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=133, tag_tid=21, entry_tid=1' 2024-04-06T09:45:02.626 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:45:02.626 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:45:02.627 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=21 2024-04-06T09:45:02.627 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=124, tag_tid=20, entry_tid=12460' 2024-04-06T09:45:02.627 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:45:02.628 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:45:02.628 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=20 2024-04-06T09:45:02.629 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=133, tag_tid=21, entry_tid=1' 2024-04-06T09:45:02.629 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:45:02.629 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:45:02.630 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:45:02.631 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=124, tag_tid=20, entry_tid=12460' 2024-04-06T09:45:02.631 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:45:02.631 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:45:02.632 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=12460 2024-04-06T09:45:02.632 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 21 = 20 -a 1 -le 12460 2024-04-06T09:45:02.632 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:45:02.632 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:45:02.632 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:45:02.834 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:45:02.834 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:45:02.834 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:45:02.834 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:45:02.834 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:45:02.834 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:45:02.834 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:45:02.834 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:45:02.834 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:45:02.835 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:45:02.835 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:45:02.835 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:45:02.835 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:45:02.835 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:45:02.835 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:45:02.835 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:45:02.836 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:45:02.839 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:45:02.839 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:45:02.840 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:45:02.840 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:45:02.840 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:45:02.840 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:45:02.840 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:45:02.840 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:45:02.840 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:45:02.840 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:45:02.840 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:45:02.840 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:45:02.840 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:45:02.840 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:45:02.840 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:45:02.840 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:45:02.841 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:45:02.841 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:45:02.841 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:45:02.841 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:45:02.841 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:45:02.841 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:45:02.841 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:45:02.841 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:45:02.841 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:45:02.841 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:45:02.841 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:45:02.841 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:45:02.841 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:45:02.841 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:45:02.841 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:45:02.842 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:45:02.865 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:45:02.865 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:45:02.865 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:45:02.865 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:45:03.003 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:45:03.005 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:45:03.005 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:45:03.005 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:45:03.005 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:45:03.005 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:45:03.005 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:45:03.005 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:45:03.005 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:45:03.005 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:45:03.005 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:45:03.006 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:45:03.006 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:45:03.007 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:45:03.007 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:45:03.008 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:45:03.269 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 31 2024-04-06T09:45:03.269 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 33 2024-04-06T09:45:03.269 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:45:03.269 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=133, tag_tid=21, entry_tid=1], [object_number=132, tag_tid=21, entry_tid=0], [object_number=135, tag_tid=20, entry_tid=19939], [object_number=134, tag_tid=20, entry_tid=19938]]], state=connected] 2024-04-06T09:45:03.269 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=126, tag_tid=20, entry_tid=12550], [object_number=125, tag_tid=20, entry_tid=12549], [object_number=124, tag_tid=20, entry_tid=12548], [object_number=127, tag_tid=20, entry_tid=12547]]], state=connected] 2024-04-06T09:45:03.272 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:45:03.274 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=133, tag_tid=21, entry_tid=1' 2024-04-06T09:45:03.275 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:45:03.275 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:45:03.275 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:45:03.275 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:45:03.275 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:45:03.275 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:45:03.275 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:45:03.275 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:45:03.276 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:45:03.276 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:45:03.276 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:45:03.276 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:45:03.276 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:45:03.277 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:45:03.277 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:45:03.327 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 31 2024-04-06T09:45:03.327 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 33 2024-04-06T09:45:03.327 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:45:03.327 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=133, tag_tid=21, entry_tid=1], [object_number=132, tag_tid=21, entry_tid=0], [object_number=135, tag_tid=20, entry_tid=19939], [object_number=134, tag_tid=20, entry_tid=19938]]], state=connected] 2024-04-06T09:45:03.328 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=126, tag_tid=20, entry_tid=12550], [object_number=125, tag_tid=20, entry_tid=12549], [object_number=124, tag_tid=20, entry_tid=12548], [object_number=127, tag_tid=20, entry_tid=12547]]], state=connected] 2024-04-06T09:45:03.331 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:45:03.333 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=126, tag_tid=20, entry_tid=12550' 2024-04-06T09:45:03.333 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=133, tag_tid=21, entry_tid=1' -a 'object_number=133, tag_tid=21, entry_tid=1' = 'object_number=126, tag_tid=20, entry_tid=12550' 2024-04-06T09:45:03.333 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=126, tag_tid=20, entry_tid=12550' '!=' 'object_number=124, tag_tid=20, entry_tid=12460' 2024-04-06T09:45:03.333 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:45:03.333 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=126, tag_tid=20, entry_tid=12550' = 'object_number=124, tag_tid=20, entry_tid=12460' 2024-04-06T09:45:03.333 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=126, tag_tid=20, entry_tid=12550' 2024-04-06T09:45:03.334 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=133, tag_tid=21, entry_tid=1' 2024-04-06T09:45:03.334 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:45:03.334 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:45:03.334 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=21 2024-04-06T09:45:03.335 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=126, tag_tid=20, entry_tid=12550' 2024-04-06T09:45:03.335 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:45:03.335 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:45:03.336 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=20 2024-04-06T09:45:03.337 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=133, tag_tid=21, entry_tid=1' 2024-04-06T09:45:03.337 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:45:03.337 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:45:03.338 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:45:03.339 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=126, tag_tid=20, entry_tid=12550' 2024-04-06T09:45:03.339 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:45:03.339 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:45:03.340 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=12550 2024-04-06T09:45:03.340 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 21 = 20 -a 1 -le 12550 2024-04-06T09:45:03.341 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:45:03.341 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:45:03.341 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:45:03.543 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:45:03.543 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:45:03.543 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:45:03.543 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:45:03.543 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:45:03.543 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:45:03.543 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:45:03.543 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:45:03.543 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:45:03.543 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:45:03.544 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:45:03.544 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:45:03.544 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:45:03.544 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:45:03.544 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:45:03.544 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:45:03.544 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:45:03.547 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:45:03.547 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:45:03.547 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:45:03.547 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:45:03.547 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:45:03.547 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:45:03.547 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:45:03.547 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:45:03.547 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:45:03.548 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:45:03.548 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:45:03.548 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:45:03.548 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:45:03.548 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:45:03.548 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:45:03.548 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:45:03.548 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:45:03.548 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:45:03.548 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:45:03.548 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:45:03.548 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:45:03.548 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:45:03.548 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:45:03.548 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:45:03.549 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:45:03.549 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:45:03.549 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:45:03.549 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:45:03.549 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:45:03.549 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:45:03.549 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:45:03.549 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:45:03.572 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:45:03.573 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:45:03.573 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:45:03.573 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:45:03.712 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:45:03.713 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:45:03.713 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:45:03.714 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:45:03.714 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:45:03.714 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:45:03.714 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:45:03.714 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:45:03.714 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:45:03.714 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:45:03.715 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:45:03.715 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:45:03.715 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:45:03.717 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:45:03.718 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:45:03.718 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:45:03.774 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 31 2024-04-06T09:45:03.774 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 33 2024-04-06T09:45:03.775 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:45:03.775 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=133, tag_tid=21, entry_tid=1], [object_number=132, tag_tid=21, entry_tid=0], [object_number=135, tag_tid=20, entry_tid=19939], [object_number=134, tag_tid=20, entry_tid=19938]]], state=connected] 2024-04-06T09:45:03.775 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=125, tag_tid=20, entry_tid=12681], [object_number=124, tag_tid=20, entry_tid=12680], [object_number=127, tag_tid=20, entry_tid=12679], [object_number=126, tag_tid=20, entry_tid=12678]]], state=connected] 2024-04-06T09:45:03.778 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:45:03.779 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=133, tag_tid=21, entry_tid=1' 2024-04-06T09:45:03.780 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:45:03.780 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:45:03.780 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:45:03.780 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:45:03.780 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:45:03.780 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:45:03.780 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:45:03.780 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:45:03.780 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:45:03.781 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:45:03.781 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:45:03.781 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:45:03.782 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:45:03.782 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:45:03.782 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:45:03.829 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 31 2024-04-06T09:45:03.829 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 33 2024-04-06T09:45:03.829 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:45:03.829 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=133, tag_tid=21, entry_tid=1], [object_number=132, tag_tid=21, entry_tid=0], [object_number=135, tag_tid=20, entry_tid=19939], [object_number=134, tag_tid=20, entry_tid=19938]]], state=connected] 2024-04-06T09:45:03.830 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=125, tag_tid=20, entry_tid=12681], [object_number=124, tag_tid=20, entry_tid=12680], [object_number=127, tag_tid=20, entry_tid=12679], [object_number=126, tag_tid=20, entry_tid=12678]]], state=connected] 2024-04-06T09:45:03.833 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:45:03.835 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=125, tag_tid=20, entry_tid=12681' 2024-04-06T09:45:03.835 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=133, tag_tid=21, entry_tid=1' -a 'object_number=133, tag_tid=21, entry_tid=1' = 'object_number=125, tag_tid=20, entry_tid=12681' 2024-04-06T09:45:03.835 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=125, tag_tid=20, entry_tid=12681' '!=' 'object_number=126, tag_tid=20, entry_tid=12550' 2024-04-06T09:45:03.835 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:45:03.835 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=125, tag_tid=20, entry_tid=12681' = 'object_number=126, tag_tid=20, entry_tid=12550' 2024-04-06T09:45:03.836 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=125, tag_tid=20, entry_tid=12681' 2024-04-06T09:45:03.836 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=133, tag_tid=21, entry_tid=1' 2024-04-06T09:45:03.836 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:45:03.836 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:45:03.836 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=21 2024-04-06T09:45:03.837 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=125, tag_tid=20, entry_tid=12681' 2024-04-06T09:45:03.837 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:45:03.837 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:45:03.838 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=20 2024-04-06T09:45:03.839 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=133, tag_tid=21, entry_tid=1' 2024-04-06T09:45:03.839 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:45:03.839 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:45:03.839 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:45:03.840 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=125, tag_tid=20, entry_tid=12681' 2024-04-06T09:45:03.840 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:45:03.840 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:45:03.842 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=12681 2024-04-06T09:45:03.842 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 21 = 20 -a 1 -le 12681 2024-04-06T09:45:03.842 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:45:03.842 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:45:03.842 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:45:04.044 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:45:04.044 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:45:04.044 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:45:04.044 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:45:04.044 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:45:04.045 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:45:04.045 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:45:04.045 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:45:04.045 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:45:04.045 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:45:04.045 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:45:04.045 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:45:04.045 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:45:04.045 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:45:04.045 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:45:04.046 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:45:04.046 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:45:04.047 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:45:04.048 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:45:04.048 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:45:04.048 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:45:04.048 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:45:04.048 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:45:04.048 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:45:04.048 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:45:04.048 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:45:04.048 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:45:04.048 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:45:04.048 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:45:04.048 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:45:04.048 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:45:04.048 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:45:04.049 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:45:04.049 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:45:04.050 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:45:04.050 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:45:04.050 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:45:04.050 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:45:04.050 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:45:04.050 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:45:04.050 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:45:04.050 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:45:04.050 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:45:04.050 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:45:04.050 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:45:04.050 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:45:04.050 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:45:04.050 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:45:04.051 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:45:04.074 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:45:04.075 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:45:04.075 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:45:04.076 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:45:04.207 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:45:04.209 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:45:04.209 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:45:04.209 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:45:04.209 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:45:04.209 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:45:04.209 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:45:04.209 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:45:04.209 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:45:04.209 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:45:04.209 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:45:04.210 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:45:04.210 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:45:04.211 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:45:04.211 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:45:04.211 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:45:04.260 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 31 2024-04-06T09:45:04.261 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 33 2024-04-06T09:45:04.261 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:45:04.261 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=133, tag_tid=21, entry_tid=1], [object_number=132, tag_tid=21, entry_tid=0], [object_number=135, tag_tid=20, entry_tid=19939], [object_number=134, tag_tid=20, entry_tid=19938]]], state=connected] 2024-04-06T09:45:04.261 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=127, tag_tid=20, entry_tid=12767], [object_number=126, tag_tid=20, entry_tid=12766], [object_number=125, tag_tid=20, entry_tid=12765], [object_number=124, tag_tid=20, entry_tid=12764]]], state=connected] 2024-04-06T09:45:04.263 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:45:04.265 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=133, tag_tid=21, entry_tid=1' 2024-04-06T09:45:04.266 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:45:04.266 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:45:04.266 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:45:04.266 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:45:04.267 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:45:04.267 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:45:04.267 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:45:04.267 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:45:04.267 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:45:04.267 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:45:04.267 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:45:04.267 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:45:04.267 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:45:04.268 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:45:04.268 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:45:04.314 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 31 2024-04-06T09:45:04.315 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 33 2024-04-06T09:45:04.315 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:45:04.315 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=133, tag_tid=21, entry_tid=1], [object_number=132, tag_tid=21, entry_tid=0], [object_number=135, tag_tid=20, entry_tid=19939], [object_number=134, tag_tid=20, entry_tid=19938]]], state=connected] 2024-04-06T09:45:04.315 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=127, tag_tid=20, entry_tid=12767], [object_number=126, tag_tid=20, entry_tid=12766], [object_number=125, tag_tid=20, entry_tid=12765], [object_number=124, tag_tid=20, entry_tid=12764]]], state=connected] 2024-04-06T09:45:04.318 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:45:04.320 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=127, tag_tid=20, entry_tid=12767' 2024-04-06T09:45:04.321 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=133, tag_tid=21, entry_tid=1' -a 'object_number=133, tag_tid=21, entry_tid=1' = 'object_number=127, tag_tid=20, entry_tid=12767' 2024-04-06T09:45:04.321 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=127, tag_tid=20, entry_tid=12767' '!=' 'object_number=125, tag_tid=20, entry_tid=12681' 2024-04-06T09:45:04.321 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:45:04.321 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=127, tag_tid=20, entry_tid=12767' = 'object_number=125, tag_tid=20, entry_tid=12681' 2024-04-06T09:45:04.321 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=127, tag_tid=20, entry_tid=12767' 2024-04-06T09:45:04.321 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=133, tag_tid=21, entry_tid=1' 2024-04-06T09:45:04.321 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:45:04.321 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:45:04.322 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=21 2024-04-06T09:45:04.322 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=127, tag_tid=20, entry_tid=12767' 2024-04-06T09:45:04.323 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:45:04.323 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:45:04.323 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=20 2024-04-06T09:45:04.324 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=133, tag_tid=21, entry_tid=1' 2024-04-06T09:45:04.324 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:45:04.324 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:45:04.325 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:45:04.326 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=127, tag_tid=20, entry_tid=12767' 2024-04-06T09:45:04.326 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:45:04.326 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:45:04.327 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=12767 2024-04-06T09:45:04.327 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 21 = 20 -a 1 -le 12767 2024-04-06T09:45:04.327 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:45:04.328 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:45:04.328 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:45:04.529 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:45:04.529 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:45:04.529 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:45:04.530 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:45:04.530 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:45:04.530 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:45:04.530 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:45:04.530 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:45:04.530 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:45:04.530 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:45:04.530 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:45:04.530 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:45:04.531 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:45:04.531 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:45:04.531 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:45:04.531 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:45:04.531 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:45:04.534 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:45:04.534 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:45:04.535 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:45:04.535 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:45:04.535 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:45:04.535 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:45:04.535 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:45:04.535 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:45:04.535 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:45:04.535 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:45:04.535 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:45:04.535 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:45:04.535 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:45:04.536 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:45:04.536 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:45:04.536 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:45:04.536 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:45:04.536 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:45:04.536 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:45:04.536 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:45:04.536 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:45:04.536 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:45:04.536 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:45:04.536 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:45:04.536 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:45:04.536 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:45:04.536 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:45:04.536 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:45:04.537 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:45:04.537 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:45:04.537 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:45:04.537 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:45:04.560 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:45:04.560 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:45:04.560 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:45:04.560 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:45:04.692 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:45:04.693 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:45:04.693 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:45:04.693 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:45:04.693 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:45:04.693 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:45:04.693 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:45:04.693 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:45:04.694 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:45:04.694 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:45:04.694 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:45:04.694 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:45:04.694 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:45:04.697 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:45:04.697 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:45:04.697 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:45:04.750 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 31 2024-04-06T09:45:04.750 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 33 2024-04-06T09:45:04.750 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:45:04.750 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=133, tag_tid=21, entry_tid=1], [object_number=132, tag_tid=21, entry_tid=0], [object_number=135, tag_tid=20, entry_tid=19939], [object_number=134, tag_tid=20, entry_tid=19938]]], state=connected] 2024-04-06T09:45:04.750 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=126, tag_tid=20, entry_tid=12846], [object_number=125, tag_tid=20, entry_tid=12845], [object_number=124, tag_tid=20, entry_tid=12844], [object_number=127, tag_tid=20, entry_tid=12843]]], state=connected] 2024-04-06T09:45:04.754 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:45:04.755 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=133, tag_tid=21, entry_tid=1' 2024-04-06T09:45:04.756 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:45:04.756 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:45:04.756 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:45:04.756 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:45:04.756 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:45:04.756 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:45:04.756 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:45:04.756 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:45:04.756 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:45:04.756 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:45:04.756 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:45:04.756 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:45:04.757 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:45:04.757 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:45:04.758 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:45:04.804 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 31 2024-04-06T09:45:04.805 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 33 2024-04-06T09:45:04.805 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:45:04.805 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=133, tag_tid=21, entry_tid=1], [object_number=132, tag_tid=21, entry_tid=0], [object_number=135, tag_tid=20, entry_tid=19939], [object_number=134, tag_tid=20, entry_tid=19938]]], state=connected] 2024-04-06T09:45:04.805 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=126, tag_tid=20, entry_tid=12846], [object_number=125, tag_tid=20, entry_tid=12845], [object_number=124, tag_tid=20, entry_tid=12844], [object_number=127, tag_tid=20, entry_tid=12843]]], state=connected] 2024-04-06T09:45:04.808 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:45:04.811 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=126, tag_tid=20, entry_tid=12846' 2024-04-06T09:45:04.811 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=133, tag_tid=21, entry_tid=1' -a 'object_number=133, tag_tid=21, entry_tid=1' = 'object_number=126, tag_tid=20, entry_tid=12846' 2024-04-06T09:45:04.811 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=126, tag_tid=20, entry_tid=12846' '!=' 'object_number=127, tag_tid=20, entry_tid=12767' 2024-04-06T09:45:04.812 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:45:04.812 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=126, tag_tid=20, entry_tid=12846' = 'object_number=127, tag_tid=20, entry_tid=12767' 2024-04-06T09:45:04.812 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=126, tag_tid=20, entry_tid=12846' 2024-04-06T09:45:04.812 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=133, tag_tid=21, entry_tid=1' 2024-04-06T09:45:04.812 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:45:04.812 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:45:04.814 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=21 2024-04-06T09:45:04.814 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=126, tag_tid=20, entry_tid=12846' 2024-04-06T09:45:04.814 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:45:04.814 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:45:04.814 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=20 2024-04-06T09:45:04.815 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=133, tag_tid=21, entry_tid=1' 2024-04-06T09:45:04.815 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:45:04.815 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:45:04.815 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:45:04.816 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=126, tag_tid=20, entry_tid=12846' 2024-04-06T09:45:04.816 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:45:04.816 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:45:04.817 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=12846 2024-04-06T09:45:04.817 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 21 = 20 -a 1 -le 12846 2024-04-06T09:45:04.818 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:45:04.818 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:45:04.818 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:45:05.019 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:45:05.020 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:45:05.020 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:45:05.020 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:45:05.020 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:45:05.020 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:45:05.020 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:45:05.020 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:45:05.020 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:45:05.020 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:45:05.020 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:45:05.020 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:45:05.020 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:45:05.020 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:45:05.020 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:45:05.021 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:45:05.022 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:45:05.024 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:45:05.024 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:45:05.024 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:45:05.024 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:45:05.024 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:45:05.025 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:45:05.025 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:45:05.025 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:45:05.025 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:45:05.025 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:45:05.025 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:45:05.025 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:45:05.025 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:45:05.025 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:45:05.025 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:45:05.025 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:45:05.025 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:45:05.026 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:45:05.026 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:45:05.026 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:45:05.026 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:45:05.026 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:45:05.026 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:45:05.026 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:45:05.026 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:45:05.026 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:45:05.026 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:45:05.026 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:45:05.026 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:45:05.027 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:45:05.027 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:45:05.027 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:45:05.052 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:45:05.053 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:45:05.053 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:45:05.053 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:45:05.181 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:45:05.182 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:45:05.182 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:45:05.182 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:45:05.182 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:45:05.182 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:45:05.183 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:45:05.183 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:45:05.183 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:45:05.183 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:45:05.184 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:45:05.184 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:45:05.184 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:45:05.186 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:45:05.187 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:45:05.187 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:45:05.236 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 31 2024-04-06T09:45:05.236 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 33 2024-04-06T09:45:05.236 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:45:05.236 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=133, tag_tid=21, entry_tid=1], [object_number=132, tag_tid=21, entry_tid=0], [object_number=135, tag_tid=20, entry_tid=19939], [object_number=134, tag_tid=20, entry_tid=19938]]], state=connected] 2024-04-06T09:45:05.236 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=125, tag_tid=20, entry_tid=12929], [object_number=124, tag_tid=20, entry_tid=12928], [object_number=127, tag_tid=20, entry_tid=12927], [object_number=126, tag_tid=20, entry_tid=12926]]], state=connected] 2024-04-06T09:45:05.240 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:45:05.242 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=133, tag_tid=21, entry_tid=1' 2024-04-06T09:45:05.242 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:45:05.242 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:45:05.242 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:45:05.242 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:45:05.242 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:45:05.242 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:45:05.243 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:45:05.243 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:45:05.243 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:45:05.243 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:45:05.243 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:45:05.243 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:45:05.243 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:45:05.244 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:45:05.244 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:45:05.290 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 31 2024-04-06T09:45:05.290 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 33 2024-04-06T09:45:05.290 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:45:05.291 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=133, tag_tid=21, entry_tid=1], [object_number=132, tag_tid=21, entry_tid=0], [object_number=135, tag_tid=20, entry_tid=19939], [object_number=134, tag_tid=20, entry_tid=19938]]], state=connected] 2024-04-06T09:45:05.291 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=125, tag_tid=20, entry_tid=12929], [object_number=124, tag_tid=20, entry_tid=12928], [object_number=127, tag_tid=20, entry_tid=12927], [object_number=126, tag_tid=20, entry_tid=12926]]], state=connected] 2024-04-06T09:45:05.293 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:45:05.295 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=125, tag_tid=20, entry_tid=12929' 2024-04-06T09:45:05.295 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=133, tag_tid=21, entry_tid=1' -a 'object_number=133, tag_tid=21, entry_tid=1' = 'object_number=125, tag_tid=20, entry_tid=12929' 2024-04-06T09:45:05.295 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=125, tag_tid=20, entry_tid=12929' '!=' 'object_number=126, tag_tid=20, entry_tid=12846' 2024-04-06T09:45:05.295 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:45:05.295 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=125, tag_tid=20, entry_tid=12929' = 'object_number=126, tag_tid=20, entry_tid=12846' 2024-04-06T09:45:05.295 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=125, tag_tid=20, entry_tid=12929' 2024-04-06T09:45:05.296 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=133, tag_tid=21, entry_tid=1' 2024-04-06T09:45:05.296 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:45:05.296 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:45:05.297 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=21 2024-04-06T09:45:05.297 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=125, tag_tid=20, entry_tid=12929' 2024-04-06T09:45:05.298 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:45:05.298 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:45:05.298 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=20 2024-04-06T09:45:05.299 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=133, tag_tid=21, entry_tid=1' 2024-04-06T09:45:05.299 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:45:05.299 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:45:05.300 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:45:05.301 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=125, tag_tid=20, entry_tid=12929' 2024-04-06T09:45:05.301 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:45:05.302 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:45:05.302 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=12929 2024-04-06T09:45:05.302 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 21 = 20 -a 1 -le 12929 2024-04-06T09:45:05.303 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:45:05.303 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:45:05.303 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:45:05.504 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:45:05.505 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:45:05.505 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:45:05.505 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:45:05.505 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:45:05.505 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:45:05.505 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:45:05.505 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:45:05.505 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:45:05.505 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:45:05.505 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:45:05.505 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:45:05.505 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:45:05.505 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:45:05.505 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:45:05.509 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:45:05.509 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:45:05.509 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:45:05.509 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:45:05.509 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:45:05.509 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:45:05.509 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:45:05.509 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:45:05.509 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:45:05.509 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:45:05.509 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:45:05.509 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:45:05.510 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:45:05.510 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:45:05.510 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:45:05.510 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:45:05.510 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:45:05.510 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:45:05.510 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:45:05.510 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:45:05.510 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:45:05.510 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:45:05.510 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:45:05.510 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:45:05.510 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:45:05.511 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:45:05.511 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:45:05.511 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:45:05.511 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:45:05.511 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:45:05.511 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:45:05.511 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:45:05.511 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:45:05.511 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:45:05.535 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:45:05.535 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:45:05.536 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:45:05.536 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:45:05.672 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:45:05.675 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:45:05.675 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:45:05.675 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:45:05.675 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:45:05.675 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:45:05.676 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:45:05.676 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:45:05.676 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:45:05.676 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:45:05.676 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:45:05.676 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:45:05.676 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:45:05.677 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:45:05.678 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:45:05.678 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:45:05.734 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 31 2024-04-06T09:45:05.734 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 33 2024-04-06T09:45:05.734 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:45:05.734 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=133, tag_tid=21, entry_tid=1], [object_number=132, tag_tid=21, entry_tid=0], [object_number=135, tag_tid=20, entry_tid=19939], [object_number=134, tag_tid=20, entry_tid=19938]]], state=connected] 2024-04-06T09:45:05.734 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=126, tag_tid=20, entry_tid=13006], [object_number=125, tag_tid=20, entry_tid=13005], [object_number=124, tag_tid=20, entry_tid=13004], [object_number=127, tag_tid=20, entry_tid=13003]]], state=connected] 2024-04-06T09:45:05.737 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:45:05.738 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=133, tag_tid=21, entry_tid=1' 2024-04-06T09:45:05.740 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:45:05.740 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:45:05.740 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:45:05.740 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:45:05.740 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:45:05.740 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:45:05.740 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:45:05.740 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:45:05.740 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:45:05.740 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:45:05.740 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:45:05.740 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:45:05.741 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:45:05.741 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:45:05.741 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:45:05.788 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 31 2024-04-06T09:45:05.789 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 33 2024-04-06T09:45:05.789 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:45:05.789 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=133, tag_tid=21, entry_tid=1], [object_number=132, tag_tid=21, entry_tid=0], [object_number=135, tag_tid=20, entry_tid=19939], [object_number=134, tag_tid=20, entry_tid=19938]]], state=connected] 2024-04-06T09:45:05.789 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=126, tag_tid=20, entry_tid=13006], [object_number=125, tag_tid=20, entry_tid=13005], [object_number=124, tag_tid=20, entry_tid=13004], [object_number=127, tag_tid=20, entry_tid=13003]]], state=connected] 2024-04-06T09:45:05.792 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:45:05.794 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=126, tag_tid=20, entry_tid=13006' 2024-04-06T09:45:05.794 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=133, tag_tid=21, entry_tid=1' -a 'object_number=133, tag_tid=21, entry_tid=1' = 'object_number=126, tag_tid=20, entry_tid=13006' 2024-04-06T09:45:05.794 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=126, tag_tid=20, entry_tid=13006' '!=' 'object_number=125, tag_tid=20, entry_tid=12929' 2024-04-06T09:45:05.794 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:45:05.794 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=126, tag_tid=20, entry_tid=13006' = 'object_number=125, tag_tid=20, entry_tid=12929' 2024-04-06T09:45:05.794 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=126, tag_tid=20, entry_tid=13006' 2024-04-06T09:45:05.794 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=133, tag_tid=21, entry_tid=1' 2024-04-06T09:45:05.795 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:45:05.795 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:45:05.795 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=21 2024-04-06T09:45:05.796 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=126, tag_tid=20, entry_tid=13006' 2024-04-06T09:45:05.796 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:45:05.796 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:45:05.797 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=20 2024-04-06T09:45:05.798 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=133, tag_tid=21, entry_tid=1' 2024-04-06T09:45:05.798 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:45:05.798 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:45:05.799 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:45:05.799 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=126, tag_tid=20, entry_tid=13006' 2024-04-06T09:45:05.799 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:45:05.800 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:45:05.801 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=13006 2024-04-06T09:45:05.801 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 21 = 20 -a 1 -le 13006 2024-04-06T09:45:05.801 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:45:05.801 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:45:05.801 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:45:06.003 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:45:06.003 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:45:06.003 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:45:06.003 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:45:06.003 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:45:06.003 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:45:06.003 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:45:06.003 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:45:06.003 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:45:06.003 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:45:06.003 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:45:06.004 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:45:06.004 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:45:06.004 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:45:06.004 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:45:06.004 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:45:06.005 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:45:06.008 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:45:06.008 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:45:06.008 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:45:06.008 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:45:06.008 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:45:06.008 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:45:06.008 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:45:06.008 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:45:06.008 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:45:06.009 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:45:06.009 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:45:06.009 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:45:06.009 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:45:06.009 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:45:06.009 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:45:06.009 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:45:06.009 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:45:06.009 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:45:06.009 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:45:06.010 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:45:06.010 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:45:06.010 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:45:06.010 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:45:06.010 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:45:06.010 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:45:06.010 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:45:06.010 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:45:06.010 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:45:06.010 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:45:06.010 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:45:06.010 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:45:06.011 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:45:06.034 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:45:06.035 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:45:06.035 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:45:06.035 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:45:06.167 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:45:06.168 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:45:06.168 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:45:06.168 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:45:06.168 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:45:06.169 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:45:06.169 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:45:06.169 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:45:06.169 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:45:06.169 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:45:06.169 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:45:06.170 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:45:06.170 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:45:06.172 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:45:06.172 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:45:06.172 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:45:06.225 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 31 2024-04-06T09:45:06.225 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 33 2024-04-06T09:45:06.226 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:45:06.226 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=133, tag_tid=21, entry_tid=1], [object_number=132, tag_tid=21, entry_tid=0], [object_number=135, tag_tid=20, entry_tid=19939], [object_number=134, tag_tid=20, entry_tid=19938]]], state=connected] 2024-04-06T09:45:06.226 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=124, tag_tid=20, entry_tid=13092], [object_number=127, tag_tid=20, entry_tid=13091], [object_number=126, tag_tid=20, entry_tid=13090], [object_number=125, tag_tid=20, entry_tid=13089]]], state=connected] 2024-04-06T09:45:06.229 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:45:06.230 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=133, tag_tid=21, entry_tid=1' 2024-04-06T09:45:06.232 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:45:06.232 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:45:06.232 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:45:06.232 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:45:06.232 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:45:06.232 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:45:06.232 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:45:06.232 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:45:06.232 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:45:06.232 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:45:06.233 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:45:06.233 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:45:06.233 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:45:06.234 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:45:06.234 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:45:06.280 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 31 2024-04-06T09:45:06.281 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 33 2024-04-06T09:45:06.281 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:45:06.281 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=133, tag_tid=21, entry_tid=1], [object_number=132, tag_tid=21, entry_tid=0], [object_number=135, tag_tid=20, entry_tid=19939], [object_number=134, tag_tid=20, entry_tid=19938]]], state=connected] 2024-04-06T09:45:06.281 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=124, tag_tid=20, entry_tid=13092], [object_number=127, tag_tid=20, entry_tid=13091], [object_number=126, tag_tid=20, entry_tid=13090], [object_number=125, tag_tid=20, entry_tid=13089]]], state=connected] 2024-04-06T09:45:06.284 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:45:06.287 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=124, tag_tid=20, entry_tid=13092' 2024-04-06T09:45:06.287 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=133, tag_tid=21, entry_tid=1' -a 'object_number=133, tag_tid=21, entry_tid=1' = 'object_number=124, tag_tid=20, entry_tid=13092' 2024-04-06T09:45:06.287 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=124, tag_tid=20, entry_tid=13092' '!=' 'object_number=126, tag_tid=20, entry_tid=13006' 2024-04-06T09:45:06.287 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:45:06.287 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=124, tag_tid=20, entry_tid=13092' = 'object_number=126, tag_tid=20, entry_tid=13006' 2024-04-06T09:45:06.288 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=124, tag_tid=20, entry_tid=13092' 2024-04-06T09:45:06.288 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=133, tag_tid=21, entry_tid=1' 2024-04-06T09:45:06.288 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:45:06.288 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:45:06.288 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=21 2024-04-06T09:45:06.288 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=124, tag_tid=20, entry_tid=13092' 2024-04-06T09:45:06.288 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:45:06.289 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:45:06.289 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=20 2024-04-06T09:45:06.290 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=133, tag_tid=21, entry_tid=1' 2024-04-06T09:45:06.290 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:45:06.290 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:45:06.290 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:45:06.291 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=124, tag_tid=20, entry_tid=13092' 2024-04-06T09:45:06.291 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:45:06.291 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:45:06.292 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=13092 2024-04-06T09:45:06.293 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 21 = 20 -a 1 -le 13092 2024-04-06T09:45:06.293 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:45:06.293 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:45:06.293 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:45:06.495 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:45:06.495 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:45:06.495 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:45:06.495 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:45:06.496 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:45:06.496 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:45:06.496 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:45:06.496 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:45:06.496 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:45:06.496 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:45:06.497 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:45:06.497 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:45:06.497 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:45:06.497 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:45:06.497 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:45:06.499 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:45:06.499 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:45:06.499 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:45:06.499 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:45:06.499 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:45:06.499 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:45:06.499 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:45:06.499 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:45:06.500 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:45:06.500 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:45:06.500 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:45:06.500 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:45:06.500 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:45:06.500 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:45:06.500 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:45:06.500 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:45:06.500 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:45:06.500 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:45:06.501 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:45:06.501 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:45:06.501 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:45:06.502 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:45:06.502 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:45:06.502 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:45:06.502 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:45:06.502 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:45:06.502 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:45:06.502 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:45:06.502 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:45:06.502 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:45:06.502 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:45:06.502 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:45:06.503 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:45:06.503 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:45:06.527 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:45:06.527 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:45:06.527 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:45:06.527 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:45:06.659 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:45:06.660 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:45:06.660 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:45:06.660 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:45:06.660 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:45:06.660 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:45:06.660 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:45:06.660 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:45:06.660 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:45:06.661 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:45:06.661 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:45:06.661 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:45:06.661 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:45:06.663 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:45:06.663 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:45:06.663 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:45:06.711 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 31 2024-04-06T09:45:06.712 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 33 2024-04-06T09:45:06.712 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:45:06.712 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=133, tag_tid=21, entry_tid=1], [object_number=132, tag_tid=21, entry_tid=0], [object_number=135, tag_tid=20, entry_tid=19939], [object_number=134, tag_tid=20, entry_tid=19938]]], state=connected] 2024-04-06T09:45:06.712 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=125, tag_tid=20, entry_tid=13173], [object_number=124, tag_tid=20, entry_tid=13172], [object_number=127, tag_tid=20, entry_tid=13171], [object_number=126, tag_tid=20, entry_tid=13170]]], state=connected] 2024-04-06T09:45:06.715 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:45:06.716 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=133, tag_tid=21, entry_tid=1' 2024-04-06T09:45:06.717 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:45:06.717 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:45:06.717 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:45:06.717 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:45:06.717 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:45:06.718 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:45:06.718 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:45:06.718 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:45:06.718 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:45:06.718 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:45:06.718 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:45:06.718 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:45:06.719 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:45:06.719 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:45:06.719 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:45:06.767 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 31 2024-04-06T09:45:06.767 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 33 2024-04-06T09:45:06.767 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:45:06.767 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=133, tag_tid=21, entry_tid=1], [object_number=132, tag_tid=21, entry_tid=0], [object_number=135, tag_tid=20, entry_tid=19939], [object_number=134, tag_tid=20, entry_tid=19938]]], state=connected] 2024-04-06T09:45:06.767 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=125, tag_tid=20, entry_tid=13173], [object_number=124, tag_tid=20, entry_tid=13172], [object_number=127, tag_tid=20, entry_tid=13171], [object_number=126, tag_tid=20, entry_tid=13170]]], state=connected] 2024-04-06T09:45:06.770 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:45:06.772 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=125, tag_tid=20, entry_tid=13173' 2024-04-06T09:45:06.772 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=133, tag_tid=21, entry_tid=1' -a 'object_number=133, tag_tid=21, entry_tid=1' = 'object_number=125, tag_tid=20, entry_tid=13173' 2024-04-06T09:45:06.772 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=125, tag_tid=20, entry_tid=13173' '!=' 'object_number=124, tag_tid=20, entry_tid=13092' 2024-04-06T09:45:06.772 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:45:06.772 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=125, tag_tid=20, entry_tid=13173' = 'object_number=124, tag_tid=20, entry_tid=13092' 2024-04-06T09:45:06.773 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=125, tag_tid=20, entry_tid=13173' 2024-04-06T09:45:06.773 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=133, tag_tid=21, entry_tid=1' 2024-04-06T09:45:06.773 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:45:06.773 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:45:06.774 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=21 2024-04-06T09:45:06.775 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=125, tag_tid=20, entry_tid=13173' 2024-04-06T09:45:06.775 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:45:06.775 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:45:06.775 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=20 2024-04-06T09:45:06.776 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=133, tag_tid=21, entry_tid=1' 2024-04-06T09:45:06.776 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:45:06.776 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:45:06.777 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:45:06.778 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=125, tag_tid=20, entry_tid=13173' 2024-04-06T09:45:06.778 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:45:06.778 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:45:06.779 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=13173 2024-04-06T09:45:06.779 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 21 = 20 -a 1 -le 13173 2024-04-06T09:45:06.779 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:45:06.779 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:45:06.779 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:45:06.981 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:45:06.981 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:45:06.982 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:45:06.982 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:45:06.982 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:45:06.982 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:45:06.982 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:45:06.982 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:45:06.982 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:45:06.982 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:45:06.982 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:45:06.982 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:45:06.982 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:45:06.982 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:45:06.982 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:45:06.983 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:45:06.983 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:45:06.986 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:45:06.986 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:45:06.986 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:45:06.986 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:45:06.986 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:45:06.986 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:45:06.986 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:45:06.987 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:45:06.987 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:45:06.987 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:45:06.987 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:45:06.987 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:45:06.987 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:45:06.987 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:45:06.987 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:45:06.987 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:45:06.987 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:45:06.987 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:45:06.987 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:45:06.987 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:45:06.987 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:45:06.988 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:45:06.988 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:45:06.988 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:45:06.988 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:45:06.988 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:45:06.988 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:45:06.988 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:45:06.988 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:45:06.988 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:45:06.988 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:45:06.988 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:45:07.013 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:45:07.014 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:45:07.014 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:45:07.014 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:45:07.144 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:45:07.145 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:45:07.145 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:45:07.145 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:45:07.145 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:45:07.145 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:45:07.146 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:45:07.146 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:45:07.146 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:45:07.146 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:45:07.146 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:45:07.146 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:45:07.146 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:45:07.147 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:45:07.147 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:45:07.147 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:45:07.194 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 31 2024-04-06T09:45:07.195 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 33 2024-04-06T09:45:07.195 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:45:07.195 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=133, tag_tid=21, entry_tid=1], [object_number=132, tag_tid=21, entry_tid=0], [object_number=135, tag_tid=20, entry_tid=19939], [object_number=134, tag_tid=20, entry_tid=19938]]], state=connected] 2024-04-06T09:45:07.195 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=124, tag_tid=20, entry_tid=13252], [object_number=127, tag_tid=20, entry_tid=13251], [object_number=126, tag_tid=20, entry_tid=13250], [object_number=125, tag_tid=20, entry_tid=13249]]], state=connected] 2024-04-06T09:45:07.198 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:45:07.199 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=133, tag_tid=21, entry_tid=1' 2024-04-06T09:45:07.200 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:45:07.200 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:45:07.201 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:45:07.201 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:45:07.201 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:45:07.201 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:45:07.201 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:45:07.201 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:45:07.201 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:45:07.201 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:45:07.201 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:45:07.201 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:45:07.202 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:45:07.202 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:45:07.202 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:45:07.250 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 31 2024-04-06T09:45:07.250 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 33 2024-04-06T09:45:07.250 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:45:07.251 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=133, tag_tid=21, entry_tid=1], [object_number=132, tag_tid=21, entry_tid=0], [object_number=135, tag_tid=20, entry_tid=19939], [object_number=134, tag_tid=20, entry_tid=19938]]], state=connected] 2024-04-06T09:45:07.251 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=124, tag_tid=20, entry_tid=13252], [object_number=127, tag_tid=20, entry_tid=13251], [object_number=126, tag_tid=20, entry_tid=13250], [object_number=125, tag_tid=20, entry_tid=13249]]], state=connected] 2024-04-06T09:45:07.254 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:45:07.256 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=124, tag_tid=20, entry_tid=13252' 2024-04-06T09:45:07.256 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=133, tag_tid=21, entry_tid=1' -a 'object_number=133, tag_tid=21, entry_tid=1' = 'object_number=124, tag_tid=20, entry_tid=13252' 2024-04-06T09:45:07.256 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=124, tag_tid=20, entry_tid=13252' '!=' 'object_number=125, tag_tid=20, entry_tid=13173' 2024-04-06T09:45:07.256 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:45:07.256 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=124, tag_tid=20, entry_tid=13252' = 'object_number=125, tag_tid=20, entry_tid=13173' 2024-04-06T09:45:07.256 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=124, tag_tid=20, entry_tid=13252' 2024-04-06T09:45:07.256 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=133, tag_tid=21, entry_tid=1' 2024-04-06T09:45:07.256 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:45:07.256 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:45:07.257 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=21 2024-04-06T09:45:07.257 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=124, tag_tid=20, entry_tid=13252' 2024-04-06T09:45:07.258 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:45:07.258 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:45:07.259 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=20 2024-04-06T09:45:07.259 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=133, tag_tid=21, entry_tid=1' 2024-04-06T09:45:07.259 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:45:07.259 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:45:07.260 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:45:07.261 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=124, tag_tid=20, entry_tid=13252' 2024-04-06T09:45:07.261 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:45:07.261 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:45:07.262 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=13252 2024-04-06T09:45:07.262 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 21 = 20 -a 1 -le 13252 2024-04-06T09:45:07.262 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:45:07.263 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:45:07.263 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:45:07.464 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:45:07.464 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:45:07.464 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:45:07.464 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:45:07.464 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:45:07.465 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:45:07.465 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:45:07.465 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:45:07.465 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:45:07.465 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:45:07.465 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:45:07.465 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:45:07.465 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:45:07.465 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:45:07.465 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:45:07.465 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:45:07.466 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:45:07.468 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:45:07.468 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:45:07.468 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:45:07.468 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:45:07.468 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:45:07.468 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:45:07.468 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:45:07.468 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:45:07.468 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:45:07.468 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:45:07.469 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:45:07.469 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:45:07.469 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:45:07.469 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:45:07.469 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:45:07.469 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:45:07.470 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:45:07.470 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:45:07.470 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:45:07.470 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:45:07.471 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:45:07.471 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:45:07.471 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:45:07.471 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:45:07.471 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:45:07.471 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:45:07.471 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:45:07.471 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:45:07.471 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:45:07.471 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:45:07.471 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:45:07.471 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:45:07.495 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:45:07.495 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:45:07.496 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:45:07.496 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:45:07.626 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:45:07.627 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:45:07.627 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:45:07.627 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:45:07.627 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:45:07.627 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:45:07.627 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:45:07.627 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:45:07.627 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:45:07.627 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:45:07.627 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:45:07.628 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:45:07.628 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:45:07.631 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:45:07.631 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:45:07.631 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:45:07.677 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 31 2024-04-06T09:45:07.677 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 33 2024-04-06T09:45:07.678 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:45:07.678 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=133, tag_tid=21, entry_tid=1], [object_number=132, tag_tid=21, entry_tid=0], [object_number=135, tag_tid=20, entry_tid=19939], [object_number=134, tag_tid=20, entry_tid=19938]]], state=connected] 2024-04-06T09:45:07.678 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=124, tag_tid=20, entry_tid=13364], [object_number=127, tag_tid=20, entry_tid=13363], [object_number=126, tag_tid=20, entry_tid=13362], [object_number=125, tag_tid=20, entry_tid=13361]]], state=connected] 2024-04-06T09:45:07.681 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:45:07.683 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=133, tag_tid=21, entry_tid=1' 2024-04-06T09:45:07.683 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:45:07.683 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:45:07.683 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:45:07.683 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:45:07.683 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:45:07.683 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:45:07.684 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:45:07.684 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:45:07.684 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:45:07.684 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:45:07.684 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:45:07.684 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:45:07.684 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:45:07.684 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:45:07.685 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:45:07.731 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 31 2024-04-06T09:45:07.731 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 33 2024-04-06T09:45:07.731 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:45:07.731 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=133, tag_tid=21, entry_tid=1], [object_number=132, tag_tid=21, entry_tid=0], [object_number=135, tag_tid=20, entry_tid=19939], [object_number=134, tag_tid=20, entry_tid=19938]]], state=connected] 2024-04-06T09:45:07.731 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=124, tag_tid=20, entry_tid=13364], [object_number=127, tag_tid=20, entry_tid=13363], [object_number=126, tag_tid=20, entry_tid=13362], [object_number=125, tag_tid=20, entry_tid=13361]]], state=connected] 2024-04-06T09:45:07.735 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:45:07.737 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=124, tag_tid=20, entry_tid=13364' 2024-04-06T09:45:07.737 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=133, tag_tid=21, entry_tid=1' -a 'object_number=133, tag_tid=21, entry_tid=1' = 'object_number=124, tag_tid=20, entry_tid=13364' 2024-04-06T09:45:07.737 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=124, tag_tid=20, entry_tid=13364' '!=' 'object_number=124, tag_tid=20, entry_tid=13252' 2024-04-06T09:45:07.737 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:45:07.737 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=124, tag_tid=20, entry_tid=13364' = 'object_number=124, tag_tid=20, entry_tid=13252' 2024-04-06T09:45:07.737 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=124, tag_tid=20, entry_tid=13364' 2024-04-06T09:45:07.737 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=133, tag_tid=21, entry_tid=1' 2024-04-06T09:45:07.737 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:45:07.737 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:45:07.738 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=21 2024-04-06T09:45:07.739 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=124, tag_tid=20, entry_tid=13364' 2024-04-06T09:45:07.739 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:45:07.739 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:45:07.740 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=20 2024-04-06T09:45:07.740 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=133, tag_tid=21, entry_tid=1' 2024-04-06T09:45:07.740 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:45:07.741 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:45:07.741 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:45:07.742 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=124, tag_tid=20, entry_tid=13364' 2024-04-06T09:45:07.742 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:45:07.742 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:45:07.743 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=13364 2024-04-06T09:45:07.744 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 21 = 20 -a 1 -le 13364 2024-04-06T09:45:07.744 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:45:07.744 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:45:07.744 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:45:07.945 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:45:07.946 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:45:07.946 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:45:07.946 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:45:07.946 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:45:07.946 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:45:07.946 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:45:07.946 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:45:07.946 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:45:07.946 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:45:07.946 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:45:07.946 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:45:07.946 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:45:07.947 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:45:07.947 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:45:07.947 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:45:07.947 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:45:07.950 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:45:07.950 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:45:07.950 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:45:07.950 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:45:07.950 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:45:07.950 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:45:07.950 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:45:07.950 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:45:07.950 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:45:07.950 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:45:07.950 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:45:07.950 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:45:07.951 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:45:07.951 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:45:07.951 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:45:07.951 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:45:07.952 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:45:07.952 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:45:07.952 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:45:07.952 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:45:07.952 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:45:07.952 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:45:07.952 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:45:07.952 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:45:07.952 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:45:07.952 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:45:07.952 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:45:07.952 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:45:07.952 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:45:07.953 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:45:07.953 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:45:07.953 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:45:07.976 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:45:07.976 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:45:07.976 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:45:07.977 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:45:08.118 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:45:08.120 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:45:08.120 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:45:08.120 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:45:08.120 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:45:08.120 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:45:08.120 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:45:08.120 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:45:08.120 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:45:08.120 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:45:08.121 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:45:08.121 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:45:08.121 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:45:08.122 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:45:08.123 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:45:08.123 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:45:08.376 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 31 2024-04-06T09:45:08.376 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 33 2024-04-06T09:45:08.376 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:45:08.376 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=133, tag_tid=21, entry_tid=1], [object_number=132, tag_tid=21, entry_tid=0], [object_number=135, tag_tid=20, entry_tid=19939], [object_number=134, tag_tid=20, entry_tid=19938]]], state=connected] 2024-04-06T09:45:08.376 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=124, tag_tid=20, entry_tid=13460], [object_number=127, tag_tid=20, entry_tid=13459], [object_number=126, tag_tid=20, entry_tid=13458], [object_number=125, tag_tid=20, entry_tid=13457]]], state=connected] 2024-04-06T09:45:08.381 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:45:08.387 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=133, tag_tid=21, entry_tid=1' 2024-04-06T09:45:08.387 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:45:08.387 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:45:08.387 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:45:08.387 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:45:08.387 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:45:08.388 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:45:08.388 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:45:08.388 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:45:08.388 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:45:08.388 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:45:08.388 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:45:08.388 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:45:08.388 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:45:08.388 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:45:08.388 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:45:08.443 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 31 2024-04-06T09:45:08.443 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 33 2024-04-06T09:45:08.443 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:45:08.443 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=133, tag_tid=21, entry_tid=1], [object_number=132, tag_tid=21, entry_tid=0], [object_number=135, tag_tid=20, entry_tid=19939], [object_number=134, tag_tid=20, entry_tid=19938]]], state=connected] 2024-04-06T09:45:08.443 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=124, tag_tid=20, entry_tid=13460], [object_number=127, tag_tid=20, entry_tid=13459], [object_number=126, tag_tid=20, entry_tid=13458], [object_number=125, tag_tid=20, entry_tid=13457]]], state=connected] 2024-04-06T09:45:08.447 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:45:08.449 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=124, tag_tid=20, entry_tid=13460' 2024-04-06T09:45:08.449 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=133, tag_tid=21, entry_tid=1' -a 'object_number=133, tag_tid=21, entry_tid=1' = 'object_number=124, tag_tid=20, entry_tid=13460' 2024-04-06T09:45:08.449 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=124, tag_tid=20, entry_tid=13460' '!=' 'object_number=124, tag_tid=20, entry_tid=13364' 2024-04-06T09:45:08.449 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:45:08.449 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=124, tag_tid=20, entry_tid=13460' = 'object_number=124, tag_tid=20, entry_tid=13364' 2024-04-06T09:45:08.449 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=124, tag_tid=20, entry_tid=13460' 2024-04-06T09:45:08.450 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=133, tag_tid=21, entry_tid=1' 2024-04-06T09:45:08.450 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:45:08.450 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:45:08.450 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=21 2024-04-06T09:45:08.451 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=124, tag_tid=20, entry_tid=13460' 2024-04-06T09:45:08.451 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:45:08.451 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:45:08.452 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=20 2024-04-06T09:45:08.453 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=133, tag_tid=21, entry_tid=1' 2024-04-06T09:45:08.453 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:45:08.453 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:45:08.454 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:45:08.455 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=124, tag_tid=20, entry_tid=13460' 2024-04-06T09:45:08.455 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:45:08.455 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:45:08.456 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=13460 2024-04-06T09:45:08.456 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 21 = 20 -a 1 -le 13460 2024-04-06T09:45:08.456 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:45:08.457 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:45:08.457 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:45:08.658 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:45:08.658 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:45:08.658 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:45:08.658 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:45:08.658 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:45:08.659 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:45:08.659 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:45:08.659 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:45:08.659 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:45:08.659 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:45:08.659 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:45:08.659 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:45:08.659 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:45:08.659 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:45:08.659 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:45:08.659 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:45:08.660 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:45:08.663 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:45:08.663 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:45:08.663 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:45:08.663 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:45:08.663 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:45:08.663 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:45:08.664 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:45:08.664 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:45:08.664 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:45:08.664 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:45:08.664 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:45:08.664 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:45:08.664 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:45:08.664 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:45:08.664 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:45:08.664 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:45:08.664 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:45:08.664 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:45:08.664 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:45:08.664 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:45:08.664 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:45:08.664 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:45:08.665 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:45:08.665 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:45:08.665 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:45:08.665 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:45:08.665 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:45:08.665 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:45:08.665 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:45:08.665 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:45:08.665 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:45:08.665 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:45:08.688 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:45:08.689 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:45:08.689 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:45:08.689 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:45:08.821 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:45:08.822 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:45:08.822 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:45:08.822 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:45:08.823 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:45:08.823 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:45:08.823 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:45:08.823 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:45:08.823 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:45:08.823 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:45:08.823 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:45:08.823 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:45:08.823 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:45:08.825 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:45:08.826 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:45:08.826 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:45:08.878 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 31 2024-04-06T09:45:08.878 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 33 2024-04-06T09:45:08.878 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:45:08.878 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=133, tag_tid=21, entry_tid=1], [object_number=132, tag_tid=21, entry_tid=0], [object_number=135, tag_tid=20, entry_tid=19939], [object_number=134, tag_tid=20, entry_tid=19938]]], state=connected] 2024-04-06T09:45:08.878 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=126, tag_tid=20, entry_tid=13586], [object_number=125, tag_tid=20, entry_tid=13585], [object_number=124, tag_tid=20, entry_tid=13584], [object_number=127, tag_tid=20, entry_tid=13583]]], state=connected] 2024-04-06T09:45:08.881 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:45:08.883 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=133, tag_tid=21, entry_tid=1' 2024-04-06T09:45:08.884 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:45:08.884 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:45:08.884 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:45:08.884 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:45:08.884 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:45:08.884 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:45:08.884 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:45:08.884 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:45:08.885 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:45:08.885 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:45:08.885 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:45:08.885 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:45:08.885 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:45:08.886 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:45:08.886 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:45:08.933 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 31 2024-04-06T09:45:08.933 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 33 2024-04-06T09:45:08.934 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:45:08.934 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=133, tag_tid=21, entry_tid=1], [object_number=132, tag_tid=21, entry_tid=0], [object_number=135, tag_tid=20, entry_tid=19939], [object_number=134, tag_tid=20, entry_tid=19938]]], state=connected] 2024-04-06T09:45:08.934 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=126, tag_tid=20, entry_tid=13586], [object_number=125, tag_tid=20, entry_tid=13585], [object_number=124, tag_tid=20, entry_tid=13584], [object_number=127, tag_tid=20, entry_tid=13583]]], state=connected] 2024-04-06T09:45:08.937 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:45:08.939 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=126, tag_tid=20, entry_tid=13586' 2024-04-06T09:45:08.939 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=133, tag_tid=21, entry_tid=1' -a 'object_number=133, tag_tid=21, entry_tid=1' = 'object_number=126, tag_tid=20, entry_tid=13586' 2024-04-06T09:45:08.939 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=126, tag_tid=20, entry_tid=13586' '!=' 'object_number=124, tag_tid=20, entry_tid=13460' 2024-04-06T09:45:08.939 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:45:08.939 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=126, tag_tid=20, entry_tid=13586' = 'object_number=124, tag_tid=20, entry_tid=13460' 2024-04-06T09:45:08.939 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=126, tag_tid=20, entry_tid=13586' 2024-04-06T09:45:08.939 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=133, tag_tid=21, entry_tid=1' 2024-04-06T09:45:08.939 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:45:08.939 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:45:08.940 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=21 2024-04-06T09:45:08.941 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=126, tag_tid=20, entry_tid=13586' 2024-04-06T09:45:08.941 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:45:08.941 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:45:08.942 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=20 2024-04-06T09:45:08.943 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=133, tag_tid=21, entry_tid=1' 2024-04-06T09:45:08.943 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:45:08.943 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:45:08.944 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:45:08.944 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=126, tag_tid=20, entry_tid=13586' 2024-04-06T09:45:08.944 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:45:08.944 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:45:08.946 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=13586 2024-04-06T09:45:08.946 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 21 = 20 -a 1 -le 13586 2024-04-06T09:45:08.946 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:45:08.946 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:45:08.946 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:45:09.147 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:45:09.147 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:45:09.148 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:45:09.148 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:45:09.148 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:45:09.148 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:45:09.148 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:45:09.148 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:45:09.148 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:45:09.148 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:45:09.148 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:45:09.148 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:45:09.148 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:45:09.148 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:45:09.149 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:45:09.149 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:45:09.149 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:45:09.153 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:45:09.153 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:45:09.153 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:45:09.153 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:45:09.153 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:45:09.154 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:45:09.154 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:45:09.154 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:45:09.154 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:45:09.154 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:45:09.154 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:45:09.154 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:45:09.154 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:45:09.154 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:45:09.154 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:45:09.154 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:45:09.154 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:45:09.155 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:45:09.155 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:45:09.155 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:45:09.155 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:45:09.155 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:45:09.155 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:45:09.155 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:45:09.155 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:45:09.155 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:45:09.155 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:45:09.155 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:45:09.155 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:45:09.156 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:45:09.156 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:45:09.156 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:45:09.178 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:45:09.179 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:45:09.179 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:45:09.179 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:45:09.310 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:45:09.311 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:45:09.311 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:45:09.311 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:45:09.311 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:45:09.312 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:45:09.312 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:45:09.312 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:45:09.312 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:45:09.312 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:45:09.312 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:45:09.313 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:45:09.313 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:45:09.314 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:45:09.314 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:45:09.314 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:45:09.361 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 31 2024-04-06T09:45:09.361 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 33 2024-04-06T09:45:09.361 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:45:09.361 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=133, tag_tid=21, entry_tid=1], [object_number=132, tag_tid=21, entry_tid=0], [object_number=135, tag_tid=20, entry_tid=19939], [object_number=134, tag_tid=20, entry_tid=19938]]], state=connected] 2024-04-06T09:45:09.361 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=126, tag_tid=20, entry_tid=13670], [object_number=125, tag_tid=20, entry_tid=13669], [object_number=124, tag_tid=20, entry_tid=13668], [object_number=127, tag_tid=20, entry_tid=13667]]], state=connected] 2024-04-06T09:45:09.364 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:45:09.365 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=133, tag_tid=21, entry_tid=1' 2024-04-06T09:45:09.366 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:45:09.367 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:45:09.367 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:45:09.367 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:45:09.367 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:45:09.367 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:45:09.367 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:45:09.367 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:45:09.367 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:45:09.367 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:45:09.367 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:45:09.367 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:45:09.368 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:45:09.368 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:45:09.368 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:45:09.404 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.0 with skip (0.5439748953614193) > weight (0.25) 2024-04-06T09:45:09.404 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.1 with skip (0.9081167088262762) > weight (0.25) 2024-04-06T09:45:09.404 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.2 with skip (0.814511007426277) > weight (0.25) 2024-04-06T09:45:09.404 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.3 with skip (0.3238318488656716) > weight (0.25) 2024-04-06T09:45:09.404 INFO:tasks.rbd_mirror_thrash:waiting for 88 secs before thrashing 2024-04-06T09:45:09.414 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 31 2024-04-06T09:45:09.414 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 33 2024-04-06T09:45:09.414 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:45:09.414 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=133, tag_tid=21, entry_tid=1], [object_number=132, tag_tid=21, entry_tid=0], [object_number=135, tag_tid=20, entry_tid=19939], [object_number=134, tag_tid=20, entry_tid=19938]]], state=connected] 2024-04-06T09:45:09.414 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=126, tag_tid=20, entry_tid=13670], [object_number=125, tag_tid=20, entry_tid=13669], [object_number=124, tag_tid=20, entry_tid=13668], [object_number=127, tag_tid=20, entry_tid=13667]]], state=connected] 2024-04-06T09:45:09.418 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:45:09.420 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=126, tag_tid=20, entry_tid=13670' 2024-04-06T09:45:09.420 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=133, tag_tid=21, entry_tid=1' -a 'object_number=133, tag_tid=21, entry_tid=1' = 'object_number=126, tag_tid=20, entry_tid=13670' 2024-04-06T09:45:09.420 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=126, tag_tid=20, entry_tid=13670' '!=' 'object_number=126, tag_tid=20, entry_tid=13586' 2024-04-06T09:45:09.420 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:45:09.420 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=126, tag_tid=20, entry_tid=13670' = 'object_number=126, tag_tid=20, entry_tid=13586' 2024-04-06T09:45:09.420 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=126, tag_tid=20, entry_tid=13670' 2024-04-06T09:45:09.420 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=133, tag_tid=21, entry_tid=1' 2024-04-06T09:45:09.420 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:45:09.420 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:45:09.421 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=21 2024-04-06T09:45:09.422 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=126, tag_tid=20, entry_tid=13670' 2024-04-06T09:45:09.422 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:45:09.422 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:45:09.423 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=20 2024-04-06T09:45:09.423 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=133, tag_tid=21, entry_tid=1' 2024-04-06T09:45:09.423 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:45:09.423 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:45:09.424 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:45:09.425 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=126, tag_tid=20, entry_tid=13670' 2024-04-06T09:45:09.425 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:45:09.425 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:45:09.426 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=13670 2024-04-06T09:45:09.426 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 21 = 20 -a 1 -le 13670 2024-04-06T09:45:09.427 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:45:09.427 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:45:09.427 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:45:09.628 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:45:09.628 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:45:09.629 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:45:09.629 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:45:09.629 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:45:09.629 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:45:09.629 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:45:09.629 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:45:09.629 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:45:09.629 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:45:09.629 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:45:09.629 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:45:09.629 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:45:09.629 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:45:09.629 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:45:09.630 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:45:09.630 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:45:09.634 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:45:09.634 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:45:09.634 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:45:09.634 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:45:09.634 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:45:09.634 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:45:09.634 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:45:09.634 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:45:09.634 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:45:09.634 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:45:09.635 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:45:09.635 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:45:09.635 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:45:09.635 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:45:09.635 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:45:09.635 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:45:09.635 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:45:09.635 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:45:09.635 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:45:09.635 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:45:09.635 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:45:09.635 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:45:09.635 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:45:09.635 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:45:09.636 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:45:09.636 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:45:09.636 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:45:09.636 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:45:09.636 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:45:09.636 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:45:09.636 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:45:09.636 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:45:09.660 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:45:09.660 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:45:09.660 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:45:09.660 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:45:09.789 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:45:09.790 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:45:09.790 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:45:09.790 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:45:09.790 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:45:09.790 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:45:09.790 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:45:09.790 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:45:09.791 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:45:09.791 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:45:09.791 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:45:09.791 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:45:09.791 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:45:09.792 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:45:09.793 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:45:09.793 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:45:09.848 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 31 2024-04-06T09:45:09.848 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 33 2024-04-06T09:45:09.848 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:45:09.848 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=133, tag_tid=21, entry_tid=1], [object_number=132, tag_tid=21, entry_tid=0], [object_number=135, tag_tid=20, entry_tid=19939], [object_number=134, tag_tid=20, entry_tid=19938]]], state=connected] 2024-04-06T09:45:09.848 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=127, tag_tid=20, entry_tid=13759], [object_number=126, tag_tid=20, entry_tid=13758], [object_number=125, tag_tid=20, entry_tid=13757], [object_number=124, tag_tid=20, entry_tid=13756]]], state=connected] 2024-04-06T09:45:09.851 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:45:09.853 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=133, tag_tid=21, entry_tid=1' 2024-04-06T09:45:09.854 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:45:09.854 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:45:09.854 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:45:09.854 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:45:09.854 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:45:09.854 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:45:09.854 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:45:09.854 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:45:09.855 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:45:09.855 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:45:09.855 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:45:09.855 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:45:09.855 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:45:09.855 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:45:09.856 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:45:09.902 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 31 2024-04-06T09:45:09.902 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 33 2024-04-06T09:45:09.902 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:45:09.902 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=133, tag_tid=21, entry_tid=1], [object_number=132, tag_tid=21, entry_tid=0], [object_number=135, tag_tid=20, entry_tid=19939], [object_number=134, tag_tid=20, entry_tid=19938]]], state=connected] 2024-04-06T09:45:09.903 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=127, tag_tid=20, entry_tid=13759], [object_number=126, tag_tid=20, entry_tid=13758], [object_number=125, tag_tid=20, entry_tid=13757], [object_number=124, tag_tid=20, entry_tid=13756]]], state=connected] 2024-04-06T09:45:09.906 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:45:09.908 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=127, tag_tid=20, entry_tid=13759' 2024-04-06T09:45:09.908 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=133, tag_tid=21, entry_tid=1' -a 'object_number=133, tag_tid=21, entry_tid=1' = 'object_number=127, tag_tid=20, entry_tid=13759' 2024-04-06T09:45:09.908 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=127, tag_tid=20, entry_tid=13759' '!=' 'object_number=126, tag_tid=20, entry_tid=13670' 2024-04-06T09:45:09.908 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:45:09.908 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=127, tag_tid=20, entry_tid=13759' = 'object_number=126, tag_tid=20, entry_tid=13670' 2024-04-06T09:45:09.908 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=127, tag_tid=20, entry_tid=13759' 2024-04-06T09:45:09.908 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=133, tag_tid=21, entry_tid=1' 2024-04-06T09:45:09.908 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:45:09.908 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:45:09.909 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=21 2024-04-06T09:45:09.910 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=127, tag_tid=20, entry_tid=13759' 2024-04-06T09:45:09.910 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:45:09.910 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:45:09.911 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=20 2024-04-06T09:45:09.911 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=133, tag_tid=21, entry_tid=1' 2024-04-06T09:45:09.911 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:45:09.912 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:45:09.912 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:45:09.913 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=127, tag_tid=20, entry_tid=13759' 2024-04-06T09:45:09.913 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:45:09.913 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:45:09.914 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=13759 2024-04-06T09:45:09.915 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 21 = 20 -a 1 -le 13759 2024-04-06T09:45:09.915 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:45:09.915 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:45:09.915 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:45:10.116 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:45:10.117 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:45:10.117 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:45:10.117 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:45:10.117 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:45:10.117 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:45:10.117 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:45:10.117 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:45:10.117 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:45:10.117 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:45:10.117 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:45:10.117 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:45:10.117 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:45:10.117 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:45:10.117 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:45:10.118 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:45:10.118 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:45:10.120 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:45:10.120 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:45:10.120 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:45:10.120 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:45:10.120 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:45:10.120 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:45:10.121 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:45:10.121 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:45:10.121 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:45:10.121 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:45:10.121 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:45:10.121 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:45:10.121 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:45:10.121 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:45:10.121 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:45:10.121 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:45:10.121 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:45:10.121 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:45:10.121 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:45:10.121 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:45:10.122 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:45:10.122 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:45:10.122 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:45:10.122 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:45:10.122 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:45:10.122 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:45:10.122 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:45:10.122 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:45:10.122 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:45:10.122 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:45:10.122 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:45:10.123 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:45:10.149 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:45:10.149 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:45:10.149 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:45:10.149 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:45:10.282 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:45:10.283 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:45:10.283 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:45:10.283 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:45:10.284 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:45:10.284 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:45:10.284 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:45:10.284 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:45:10.284 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:45:10.284 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:45:10.284 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:45:10.284 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:45:10.284 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:45:10.285 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:45:10.286 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:45:10.286 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:45:10.339 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 31 2024-04-06T09:45:10.339 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 33 2024-04-06T09:45:10.339 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:45:10.339 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=133, tag_tid=21, entry_tid=1], [object_number=132, tag_tid=21, entry_tid=0], [object_number=135, tag_tid=20, entry_tid=19939], [object_number=134, tag_tid=20, entry_tid=19938]]], state=connected] 2024-04-06T09:45:10.339 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=124, tag_tid=20, entry_tid=13848], [object_number=127, tag_tid=20, entry_tid=13847], [object_number=126, tag_tid=20, entry_tid=13846], [object_number=125, tag_tid=20, entry_tid=13845]]], state=connected] 2024-04-06T09:45:10.342 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:45:10.344 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=133, tag_tid=21, entry_tid=1' 2024-04-06T09:45:10.345 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:45:10.345 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:45:10.345 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:45:10.345 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:45:10.345 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:45:10.345 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:45:10.345 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:45:10.345 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:45:10.345 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:45:10.345 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:45:10.346 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:45:10.346 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:45:10.346 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:45:10.346 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:45:10.346 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:45:10.394 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 31 2024-04-06T09:45:10.395 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 33 2024-04-06T09:45:10.395 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:45:10.395 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=133, tag_tid=21, entry_tid=1], [object_number=132, tag_tid=21, entry_tid=0], [object_number=135, tag_tid=20, entry_tid=19939], [object_number=134, tag_tid=20, entry_tid=19938]]], state=connected] 2024-04-06T09:45:10.395 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=124, tag_tid=20, entry_tid=13848], [object_number=127, tag_tid=20, entry_tid=13847], [object_number=126, tag_tid=20, entry_tid=13846], [object_number=125, tag_tid=20, entry_tid=13845]]], state=connected] 2024-04-06T09:45:10.398 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:45:10.400 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=124, tag_tid=20, entry_tid=13848' 2024-04-06T09:45:10.401 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=133, tag_tid=21, entry_tid=1' -a 'object_number=133, tag_tid=21, entry_tid=1' = 'object_number=124, tag_tid=20, entry_tid=13848' 2024-04-06T09:45:10.401 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=124, tag_tid=20, entry_tid=13848' '!=' 'object_number=127, tag_tid=20, entry_tid=13759' 2024-04-06T09:45:10.401 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:45:10.401 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=124, tag_tid=20, entry_tid=13848' = 'object_number=127, tag_tid=20, entry_tid=13759' 2024-04-06T09:45:10.401 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=124, tag_tid=20, entry_tid=13848' 2024-04-06T09:45:10.401 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=133, tag_tid=21, entry_tid=1' 2024-04-06T09:45:10.401 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:45:10.401 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:45:10.402 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=21 2024-04-06T09:45:10.402 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=124, tag_tid=20, entry_tid=13848' 2024-04-06T09:45:10.403 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:45:10.403 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:45:10.403 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=20 2024-04-06T09:45:10.404 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=133, tag_tid=21, entry_tid=1' 2024-04-06T09:45:10.404 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:45:10.404 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:45:10.405 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:45:10.406 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=124, tag_tid=20, entry_tid=13848' 2024-04-06T09:45:10.406 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:45:10.406 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:45:10.407 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=13848 2024-04-06T09:45:10.407 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 21 = 20 -a 1 -le 13848 2024-04-06T09:45:10.407 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:45:10.407 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:45:10.408 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:45:10.609 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:45:10.610 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:45:10.610 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:45:10.610 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:45:10.610 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:45:10.610 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:45:10.610 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:45:10.610 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:45:10.610 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:45:10.610 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:45:10.610 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:45:10.610 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:45:10.610 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:45:10.610 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:45:10.611 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:45:10.611 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:45:10.611 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:45:10.615 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:45:10.615 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:45:10.615 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:45:10.615 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:45:10.615 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:45:10.616 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:45:10.616 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:45:10.616 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:45:10.616 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:45:10.616 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:45:10.616 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:45:10.616 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:45:10.616 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:45:10.616 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:45:10.616 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:45:10.616 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:45:10.616 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:45:10.616 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:45:10.617 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:45:10.617 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:45:10.617 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:45:10.617 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:45:10.617 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:45:10.617 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:45:10.617 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:45:10.617 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:45:10.617 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:45:10.617 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:45:10.617 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:45:10.617 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:45:10.617 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:45:10.617 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:45:10.641 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:45:10.642 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:45:10.642 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:45:10.642 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:45:10.781 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:45:10.783 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:45:10.783 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:45:10.783 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:45:10.783 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:45:10.783 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:45:10.783 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:45:10.784 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:45:10.784 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:45:10.784 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:45:10.784 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:45:10.784 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:45:10.784 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:45:10.785 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:45:10.785 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:45:10.785 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:45:10.831 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 31 2024-04-06T09:45:10.831 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 33 2024-04-06T09:45:10.832 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:45:10.832 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=133, tag_tid=21, entry_tid=1], [object_number=132, tag_tid=21, entry_tid=0], [object_number=135, tag_tid=20, entry_tid=19939], [object_number=134, tag_tid=20, entry_tid=19938]]], state=connected] 2024-04-06T09:45:10.832 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=124, tag_tid=20, entry_tid=13944], [object_number=127, tag_tid=20, entry_tid=13943], [object_number=126, tag_tid=20, entry_tid=13942], [object_number=125, tag_tid=20, entry_tid=13941]]], state=connected] 2024-04-06T09:45:10.835 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:45:10.836 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=133, tag_tid=21, entry_tid=1' 2024-04-06T09:45:10.838 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:45:10.838 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:45:10.838 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:45:10.838 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:45:10.839 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:45:10.839 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:45:10.839 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:45:10.839 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:45:10.839 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:45:10.839 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:45:10.839 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:45:10.839 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:45:10.840 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:45:10.840 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:45:10.840 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:45:11.100 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 31 2024-04-06T09:45:11.100 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 33 2024-04-06T09:45:11.100 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:45:11.100 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=133, tag_tid=21, entry_tid=1], [object_number=132, tag_tid=21, entry_tid=0], [object_number=135, tag_tid=20, entry_tid=19939], [object_number=134, tag_tid=20, entry_tid=19938]]], state=connected] 2024-04-06T09:45:11.100 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=124, tag_tid=20, entry_tid=13944], [object_number=127, tag_tid=20, entry_tid=13943], [object_number=126, tag_tid=20, entry_tid=13942], [object_number=125, tag_tid=20, entry_tid=13941]]], state=connected] 2024-04-06T09:45:11.105 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:45:11.107 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=124, tag_tid=20, entry_tid=13944' 2024-04-06T09:45:11.107 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=133, tag_tid=21, entry_tid=1' -a 'object_number=133, tag_tid=21, entry_tid=1' = 'object_number=124, tag_tid=20, entry_tid=13944' 2024-04-06T09:45:11.107 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=124, tag_tid=20, entry_tid=13944' '!=' 'object_number=124, tag_tid=20, entry_tid=13848' 2024-04-06T09:45:11.107 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:45:11.107 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=124, tag_tid=20, entry_tid=13944' = 'object_number=124, tag_tid=20, entry_tid=13848' 2024-04-06T09:45:11.107 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=124, tag_tid=20, entry_tid=13944' 2024-04-06T09:45:11.107 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=133, tag_tid=21, entry_tid=1' 2024-04-06T09:45:11.108 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:45:11.108 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:45:11.108 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=21 2024-04-06T09:45:11.109 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=124, tag_tid=20, entry_tid=13944' 2024-04-06T09:45:11.110 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:45:11.110 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:45:11.111 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=20 2024-04-06T09:45:11.111 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=133, tag_tid=21, entry_tid=1' 2024-04-06T09:45:11.112 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:45:11.112 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:45:11.112 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:45:11.113 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=124, tag_tid=20, entry_tid=13944' 2024-04-06T09:45:11.113 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:45:11.113 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:45:11.115 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=13944 2024-04-06T09:45:11.115 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 21 = 20 -a 1 -le 13944 2024-04-06T09:45:11.115 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:45:11.115 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:45:11.115 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:45:11.318 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:45:11.318 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:45:11.318 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:45:11.318 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:45:11.318 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:45:11.318 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:45:11.318 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:45:11.318 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:45:11.319 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:45:11.319 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:45:11.319 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:45:11.319 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:45:11.319 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:45:11.319 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:45:11.319 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:45:11.319 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:45:11.322 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:45:11.323 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:45:11.323 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:45:11.323 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:45:11.323 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:45:11.323 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:45:11.323 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:45:11.323 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:45:11.323 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:45:11.323 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:45:11.324 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:45:11.324 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:45:11.324 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:45:11.324 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:45:11.324 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:45:11.324 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:45:11.324 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:45:11.324 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:45:11.325 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:45:11.325 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:45:11.325 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:45:11.325 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:45:11.325 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:45:11.325 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:45:11.325 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:45:11.325 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:45:11.326 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:45:11.326 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:45:11.326 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:45:11.326 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:45:11.326 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:45:11.326 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:45:11.326 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:45:11.347 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:45:11.348 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:45:11.348 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:45:11.348 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:45:11.478 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:45:11.480 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:45:11.480 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:45:11.480 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:45:11.480 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:45:11.480 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:45:11.480 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:45:11.480 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:45:11.480 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:45:11.480 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:45:11.481 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:45:11.481 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:45:11.481 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:45:11.482 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:45:11.482 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:45:11.483 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:45:11.531 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 31 2024-04-06T09:45:11.531 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 33 2024-04-06T09:45:11.531 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:45:11.531 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=133, tag_tid=21, entry_tid=1], [object_number=132, tag_tid=21, entry_tid=0], [object_number=135, tag_tid=20, entry_tid=19939], [object_number=134, tag_tid=20, entry_tid=19938]]], state=connected] 2024-04-06T09:45:11.531 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=125, tag_tid=20, entry_tid=14069], [object_number=124, tag_tid=20, entry_tid=14068], [object_number=127, tag_tid=20, entry_tid=14067], [object_number=126, tag_tid=20, entry_tid=14066]]], state=connected] 2024-04-06T09:45:11.535 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:45:11.536 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=133, tag_tid=21, entry_tid=1' 2024-04-06T09:45:11.537 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:45:11.537 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:45:11.537 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:45:11.537 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:45:11.537 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:45:11.537 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:45:11.537 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:45:11.537 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:45:11.537 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:45:11.538 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:45:11.538 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:45:11.538 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:45:11.538 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:45:11.539 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:45:11.539 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:45:11.585 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 31 2024-04-06T09:45:11.585 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 33 2024-04-06T09:45:11.585 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:45:11.585 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=133, tag_tid=21, entry_tid=1], [object_number=132, tag_tid=21, entry_tid=0], [object_number=135, tag_tid=20, entry_tid=19939], [object_number=134, tag_tid=20, entry_tid=19938]]], state=connected] 2024-04-06T09:45:11.585 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=125, tag_tid=20, entry_tid=14069], [object_number=124, tag_tid=20, entry_tid=14068], [object_number=127, tag_tid=20, entry_tid=14067], [object_number=126, tag_tid=20, entry_tid=14066]]], state=connected] 2024-04-06T09:45:11.588 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:45:11.590 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=125, tag_tid=20, entry_tid=14069' 2024-04-06T09:45:11.590 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=133, tag_tid=21, entry_tid=1' -a 'object_number=133, tag_tid=21, entry_tid=1' = 'object_number=125, tag_tid=20, entry_tid=14069' 2024-04-06T09:45:11.591 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=125, tag_tid=20, entry_tid=14069' '!=' 'object_number=124, tag_tid=20, entry_tid=13944' 2024-04-06T09:45:11.591 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:45:11.591 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=125, tag_tid=20, entry_tid=14069' = 'object_number=124, tag_tid=20, entry_tid=13944' 2024-04-06T09:45:11.591 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=125, tag_tid=20, entry_tid=14069' 2024-04-06T09:45:11.591 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=133, tag_tid=21, entry_tid=1' 2024-04-06T09:45:11.591 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:45:11.591 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:45:11.592 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=21 2024-04-06T09:45:11.592 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=125, tag_tid=20, entry_tid=14069' 2024-04-06T09:45:11.592 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:45:11.592 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:45:11.593 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=20 2024-04-06T09:45:11.594 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=133, tag_tid=21, entry_tid=1' 2024-04-06T09:45:11.594 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:45:11.594 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:45:11.595 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:45:11.595 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=125, tag_tid=20, entry_tid=14069' 2024-04-06T09:45:11.596 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:45:11.596 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:45:11.597 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=14069 2024-04-06T09:45:11.597 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 21 = 20 -a 1 -le 14069 2024-04-06T09:45:11.597 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:45:11.597 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:45:11.597 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:45:11.799 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:45:11.799 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:45:11.799 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:45:11.799 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:45:11.799 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:45:11.799 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:45:11.799 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:45:11.799 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:45:11.799 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:45:11.799 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:45:11.800 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:45:11.800 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:45:11.800 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:45:11.800 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:45:11.800 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:45:11.800 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:45:11.801 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:45:11.804 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:45:11.804 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:45:11.805 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:45:11.805 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:45:11.805 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:45:11.805 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:45:11.805 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:45:11.805 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:45:11.805 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:45:11.805 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:45:11.805 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:45:11.805 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:45:11.805 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:45:11.805 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:45:11.805 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:45:11.805 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:45:11.805 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:45:11.806 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:45:11.806 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:45:11.806 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:45:11.806 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:45:11.806 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:45:11.806 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:45:11.806 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:45:11.806 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:45:11.806 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:45:11.806 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:45:11.806 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:45:11.806 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:45:11.806 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:45:11.806 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:45:11.806 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:45:11.829 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:45:11.830 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:45:11.830 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:45:11.830 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:45:11.957 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:45:11.958 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:45:11.958 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:45:11.958 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:45:11.958 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:45:11.958 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:45:11.958 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:45:11.958 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:45:11.958 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:45:11.958 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:45:11.959 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:45:11.959 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:45:11.959 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:45:11.961 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:45:11.961 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:45:11.961 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:45:12.017 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 31 2024-04-06T09:45:12.017 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 33 2024-04-06T09:45:12.017 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:45:12.017 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=133, tag_tid=21, entry_tid=1], [object_number=132, tag_tid=21, entry_tid=0], [object_number=135, tag_tid=20, entry_tid=19939], [object_number=134, tag_tid=20, entry_tid=19938]]], state=connected] 2024-04-06T09:45:12.018 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=125, tag_tid=20, entry_tid=14157], [object_number=124, tag_tid=20, entry_tid=14156], [object_number=127, tag_tid=20, entry_tid=14155], [object_number=126, tag_tid=20, entry_tid=14154]]], state=connected] 2024-04-06T09:45:12.021 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:45:12.022 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=133, tag_tid=21, entry_tid=1' 2024-04-06T09:45:12.023 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:45:12.023 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:45:12.023 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:45:12.024 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:45:12.024 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:45:12.024 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:45:12.024 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:45:12.024 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:45:12.024 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:45:12.024 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:45:12.024 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:45:12.024 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:45:12.025 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:45:12.025 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:45:12.025 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:45:12.072 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 31 2024-04-06T09:45:12.072 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 33 2024-04-06T09:45:12.072 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:45:12.072 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=133, tag_tid=21, entry_tid=1], [object_number=132, tag_tid=21, entry_tid=0], [object_number=135, tag_tid=20, entry_tid=19939], [object_number=134, tag_tid=20, entry_tid=19938]]], state=connected] 2024-04-06T09:45:12.072 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=125, tag_tid=20, entry_tid=14157], [object_number=124, tag_tid=20, entry_tid=14156], [object_number=127, tag_tid=20, entry_tid=14155], [object_number=126, tag_tid=20, entry_tid=14154]]], state=connected] 2024-04-06T09:45:12.076 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:45:12.078 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=125, tag_tid=20, entry_tid=14157' 2024-04-06T09:45:12.078 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=133, tag_tid=21, entry_tid=1' -a 'object_number=133, tag_tid=21, entry_tid=1' = 'object_number=125, tag_tid=20, entry_tid=14157' 2024-04-06T09:45:12.078 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=125, tag_tid=20, entry_tid=14157' '!=' 'object_number=125, tag_tid=20, entry_tid=14069' 2024-04-06T09:45:12.078 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:45:12.078 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=125, tag_tid=20, entry_tid=14157' = 'object_number=125, tag_tid=20, entry_tid=14069' 2024-04-06T09:45:12.078 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=125, tag_tid=20, entry_tid=14157' 2024-04-06T09:45:12.078 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=133, tag_tid=21, entry_tid=1' 2024-04-06T09:45:12.078 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:45:12.079 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:45:12.079 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=21 2024-04-06T09:45:12.080 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=125, tag_tid=20, entry_tid=14157' 2024-04-06T09:45:12.080 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:45:12.080 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:45:12.080 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=20 2024-04-06T09:45:12.081 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=133, tag_tid=21, entry_tid=1' 2024-04-06T09:45:12.081 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:45:12.081 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:45:12.083 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:45:12.083 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=125, tag_tid=20, entry_tid=14157' 2024-04-06T09:45:12.083 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:45:12.083 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:45:12.085 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=14157 2024-04-06T09:45:12.085 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 21 = 20 -a 1 -le 14157 2024-04-06T09:45:12.085 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:45:12.085 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:45:12.085 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:45:12.287 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:45:12.287 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:45:12.287 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:45:12.287 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:45:12.287 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:45:12.287 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:45:12.287 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:45:12.287 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:45:12.287 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:45:12.288 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:45:12.288 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:45:12.288 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:45:12.288 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:45:12.288 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:45:12.288 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:45:12.289 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:45:12.289 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:45:12.291 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:45:12.291 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:45:12.291 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:45:12.291 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:45:12.292 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:45:12.292 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:45:12.292 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:45:12.292 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:45:12.292 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:45:12.292 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:45:12.292 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:45:12.292 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:45:12.292 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:45:12.292 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:45:12.292 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:45:12.292 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:45:12.293 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:45:12.293 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:45:12.294 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:45:12.294 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:45:12.294 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:45:12.294 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:45:12.294 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:45:12.294 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:45:12.294 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:45:12.294 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:45:12.294 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:45:12.294 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:45:12.295 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:45:12.295 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:45:12.295 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:45:12.295 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:45:12.317 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:45:12.318 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:45:12.318 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:45:12.318 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:45:12.443 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:45:12.444 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:45:12.444 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:45:12.444 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:45:12.444 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:45:12.444 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:45:12.444 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:45:12.444 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:45:12.445 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:45:12.445 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:45:12.445 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:45:12.445 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:45:12.445 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:45:12.447 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:45:12.447 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:45:12.447 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:45:12.499 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 31 2024-04-06T09:45:12.499 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 33 2024-04-06T09:45:12.499 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:45:12.499 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=133, tag_tid=21, entry_tid=1], [object_number=132, tag_tid=21, entry_tid=0], [object_number=135, tag_tid=20, entry_tid=19939], [object_number=134, tag_tid=20, entry_tid=19938]]], state=connected] 2024-04-06T09:45:12.499 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=127, tag_tid=20, entry_tid=14247], [object_number=126, tag_tid=20, entry_tid=14246], [object_number=125, tag_tid=20, entry_tid=14245], [object_number=124, tag_tid=20, entry_tid=14244]]], state=connected] 2024-04-06T09:45:12.503 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:45:12.504 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=133, tag_tid=21, entry_tid=1' 2024-04-06T09:45:12.505 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:45:12.505 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:45:12.506 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:45:12.506 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:45:12.506 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:45:12.506 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:45:12.506 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:45:12.506 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:45:12.506 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:45:12.506 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:45:12.506 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:45:12.506 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:45:12.507 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:45:12.507 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:45:12.507 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:45:12.553 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 31 2024-04-06T09:45:12.553 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 33 2024-04-06T09:45:12.553 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:45:12.554 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=133, tag_tid=21, entry_tid=1], [object_number=132, tag_tid=21, entry_tid=0], [object_number=135, tag_tid=20, entry_tid=19939], [object_number=134, tag_tid=20, entry_tid=19938]]], state=connected] 2024-04-06T09:45:12.554 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=127, tag_tid=20, entry_tid=14247], [object_number=126, tag_tid=20, entry_tid=14246], [object_number=125, tag_tid=20, entry_tid=14245], [object_number=124, tag_tid=20, entry_tid=14244]]], state=connected] 2024-04-06T09:45:12.557 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:45:12.559 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=127, tag_tid=20, entry_tid=14247' 2024-04-06T09:45:12.559 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=133, tag_tid=21, entry_tid=1' -a 'object_number=133, tag_tid=21, entry_tid=1' = 'object_number=127, tag_tid=20, entry_tid=14247' 2024-04-06T09:45:12.559 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=127, tag_tid=20, entry_tid=14247' '!=' 'object_number=125, tag_tid=20, entry_tid=14157' 2024-04-06T09:45:12.559 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:45:12.560 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=127, tag_tid=20, entry_tid=14247' = 'object_number=125, tag_tid=20, entry_tid=14157' 2024-04-06T09:45:12.560 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=127, tag_tid=20, entry_tid=14247' 2024-04-06T09:45:12.560 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=133, tag_tid=21, entry_tid=1' 2024-04-06T09:45:12.560 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:45:12.560 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:45:12.560 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=21 2024-04-06T09:45:12.561 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=127, tag_tid=20, entry_tid=14247' 2024-04-06T09:45:12.561 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:45:12.561 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:45:12.562 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=20 2024-04-06T09:45:12.563 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=133, tag_tid=21, entry_tid=1' 2024-04-06T09:45:12.563 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:45:12.563 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:45:12.564 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:45:12.565 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=127, tag_tid=20, entry_tid=14247' 2024-04-06T09:45:12.565 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:45:12.565 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:45:12.566 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=14247 2024-04-06T09:45:12.566 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 21 = 20 -a 1 -le 14247 2024-04-06T09:45:12.566 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:45:12.566 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:45:12.566 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:45:12.768 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:45:12.768 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:45:12.768 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:45:12.768 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:45:12.768 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:45:12.768 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:45:12.768 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:45:12.768 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:45:12.768 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:45:12.769 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:45:12.769 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:45:12.769 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:45:12.769 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:45:12.769 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:45:12.769 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:45:12.769 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:45:12.770 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:45:12.772 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:45:12.772 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:45:12.772 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:45:12.772 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:45:12.772 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:45:12.772 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:45:12.772 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:45:12.772 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:45:12.772 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:45:12.772 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:45:12.772 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:45:12.772 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:45:12.773 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:45:12.773 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:45:12.773 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:45:12.773 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:45:12.774 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:45:12.774 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:45:12.774 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:45:12.774 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:45:12.774 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:45:12.774 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:45:12.775 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:45:12.775 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:45:12.775 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:45:12.775 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:45:12.775 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:45:12.775 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:45:12.775 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:45:12.775 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:45:12.775 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:45:12.775 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:45:12.798 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:45:12.799 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:45:12.799 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:45:12.799 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:45:12.929 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:45:12.930 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:45:12.930 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:45:12.930 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:45:12.930 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:45:12.930 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:45:12.930 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:45:12.930 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:45:12.930 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:45:12.930 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:45:12.931 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:45:12.931 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:45:12.931 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:45:12.932 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:45:12.933 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:45:12.933 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:45:12.985 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 31 2024-04-06T09:45:12.985 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 33 2024-04-06T09:45:12.985 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:45:12.985 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=133, tag_tid=21, entry_tid=1], [object_number=132, tag_tid=21, entry_tid=0], [object_number=135, tag_tid=20, entry_tid=19939], [object_number=134, tag_tid=20, entry_tid=19938]]], state=connected] 2024-04-06T09:45:12.985 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=126, tag_tid=20, entry_tid=14338], [object_number=125, tag_tid=20, entry_tid=14337], [object_number=124, tag_tid=20, entry_tid=14336], [object_number=127, tag_tid=20, entry_tid=14335]]], state=connected] 2024-04-06T09:45:12.989 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:45:12.990 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=133, tag_tid=21, entry_tid=1' 2024-04-06T09:45:12.991 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:45:12.991 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:45:12.991 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:45:12.991 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:45:12.991 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:45:12.991 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:45:12.991 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:45:12.992 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:45:12.992 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:45:12.992 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:45:12.992 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:45:12.992 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:45:12.993 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:45:12.993 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:45:12.993 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:45:13.040 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 31 2024-04-06T09:45:13.040 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 33 2024-04-06T09:45:13.041 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:45:13.041 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=133, tag_tid=21, entry_tid=1], [object_number=132, tag_tid=21, entry_tid=0], [object_number=135, tag_tid=20, entry_tid=19939], [object_number=134, tag_tid=20, entry_tid=19938]]], state=connected] 2024-04-06T09:45:13.041 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=126, tag_tid=20, entry_tid=14338], [object_number=125, tag_tid=20, entry_tid=14337], [object_number=124, tag_tid=20, entry_tid=14336], [object_number=127, tag_tid=20, entry_tid=14335]]], state=connected] 2024-04-06T09:45:13.044 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:45:13.046 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=126, tag_tid=20, entry_tid=14338' 2024-04-06T09:45:13.046 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=133, tag_tid=21, entry_tid=1' -a 'object_number=133, tag_tid=21, entry_tid=1' = 'object_number=126, tag_tid=20, entry_tid=14338' 2024-04-06T09:45:13.046 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=126, tag_tid=20, entry_tid=14338' '!=' 'object_number=127, tag_tid=20, entry_tid=14247' 2024-04-06T09:45:13.046 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:45:13.046 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=126, tag_tid=20, entry_tid=14338' = 'object_number=127, tag_tid=20, entry_tid=14247' 2024-04-06T09:45:13.046 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=126, tag_tid=20, entry_tid=14338' 2024-04-06T09:45:13.046 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=133, tag_tid=21, entry_tid=1' 2024-04-06T09:45:13.046 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:45:13.046 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:45:13.047 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=21 2024-04-06T09:45:13.048 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=126, tag_tid=20, entry_tid=14338' 2024-04-06T09:45:13.048 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:45:13.048 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:45:13.049 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=20 2024-04-06T09:45:13.049 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=133, tag_tid=21, entry_tid=1' 2024-04-06T09:45:13.049 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:45:13.049 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:45:13.050 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:45:13.051 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=126, tag_tid=20, entry_tid=14338' 2024-04-06T09:45:13.051 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:45:13.051 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:45:13.053 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=14338 2024-04-06T09:45:13.053 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 21 = 20 -a 1 -le 14338 2024-04-06T09:45:13.053 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:45:13.053 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:45:13.053 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:45:13.254 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:45:13.254 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:45:13.254 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:45:13.254 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:45:13.254 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:45:13.254 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:45:13.254 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:45:13.254 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:45:13.255 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:45:13.255 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:45:13.255 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:45:13.255 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:45:13.255 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:45:13.255 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:45:13.255 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:45:13.258 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:45:13.258 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:45:13.258 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:45:13.258 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:45:13.258 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:45:13.258 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:45:13.258 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:45:13.258 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:45:13.258 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:45:13.258 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:45:13.259 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:45:13.259 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:45:13.259 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:45:13.259 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:45:13.259 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:45:13.259 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:45:13.259 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:45:13.259 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:45:13.259 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:45:13.259 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:45:13.259 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:45:13.259 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:45:13.259 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:45:13.259 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:45:13.259 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:45:13.260 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:45:13.260 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:45:13.260 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:45:13.260 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:45:13.260 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:45:13.260 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:45:13.260 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:45:13.260 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:45:13.260 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:45:13.284 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:45:13.284 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:45:13.284 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:45:13.284 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:45:13.413 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:45:13.415 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:45:13.415 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:45:13.415 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:45:13.415 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:45:13.415 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:45:13.415 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:45:13.415 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:45:13.415 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:45:13.415 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:45:13.415 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:45:13.416 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:45:13.416 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:45:13.417 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:45:13.417 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:45:13.418 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:45:13.473 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 31 2024-04-06T09:45:13.473 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 33 2024-04-06T09:45:13.473 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:45:13.473 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=133, tag_tid=21, entry_tid=1], [object_number=132, tag_tid=21, entry_tid=0], [object_number=135, tag_tid=20, entry_tid=19939], [object_number=134, tag_tid=20, entry_tid=19938]]], state=connected] 2024-04-06T09:45:13.473 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=127, tag_tid=20, entry_tid=14431], [object_number=126, tag_tid=20, entry_tid=14430], [object_number=125, tag_tid=20, entry_tid=14429], [object_number=124, tag_tid=20, entry_tid=14428]]], state=connected] 2024-04-06T09:45:13.476 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:45:13.478 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=133, tag_tid=21, entry_tid=1' 2024-04-06T09:45:13.479 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:45:13.479 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:45:13.479 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:45:13.479 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:45:13.479 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:45:13.479 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:45:13.480 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:45:13.480 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:45:13.480 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:45:13.480 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:45:13.480 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:45:13.480 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:45:13.480 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:45:13.481 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:45:13.481 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:45:13.528 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 31 2024-04-06T09:45:13.528 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 33 2024-04-06T09:45:13.528 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:45:13.528 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=133, tag_tid=21, entry_tid=1], [object_number=132, tag_tid=21, entry_tid=0], [object_number=135, tag_tid=20, entry_tid=19939], [object_number=134, tag_tid=20, entry_tid=19938]]], state=connected] 2024-04-06T09:45:13.528 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=127, tag_tid=20, entry_tid=14431], [object_number=126, tag_tid=20, entry_tid=14430], [object_number=125, tag_tid=20, entry_tid=14429], [object_number=124, tag_tid=20, entry_tid=14428]]], state=connected] 2024-04-06T09:45:13.531 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:45:13.533 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=127, tag_tid=20, entry_tid=14431' 2024-04-06T09:45:13.534 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=133, tag_tid=21, entry_tid=1' -a 'object_number=133, tag_tid=21, entry_tid=1' = 'object_number=127, tag_tid=20, entry_tid=14431' 2024-04-06T09:45:13.534 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=127, tag_tid=20, entry_tid=14431' '!=' 'object_number=126, tag_tid=20, entry_tid=14338' 2024-04-06T09:45:13.534 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:45:13.534 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=127, tag_tid=20, entry_tid=14431' = 'object_number=126, tag_tid=20, entry_tid=14338' 2024-04-06T09:45:13.534 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=127, tag_tid=20, entry_tid=14431' 2024-04-06T09:45:13.534 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=133, tag_tid=21, entry_tid=1' 2024-04-06T09:45:13.534 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:45:13.534 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:45:13.535 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=21 2024-04-06T09:45:13.535 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=127, tag_tid=20, entry_tid=14431' 2024-04-06T09:45:13.535 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:45:13.535 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:45:13.536 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=20 2024-04-06T09:45:13.537 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=133, tag_tid=21, entry_tid=1' 2024-04-06T09:45:13.537 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:45:13.537 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:45:13.538 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:45:13.539 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=127, tag_tid=20, entry_tid=14431' 2024-04-06T09:45:13.539 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:45:13.539 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:45:13.540 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=14431 2024-04-06T09:45:13.540 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 21 = 20 -a 1 -le 14431 2024-04-06T09:45:13.540 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:45:13.540 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:45:13.540 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:45:13.742 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:45:13.742 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:45:13.742 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:45:13.742 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:45:13.742 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:45:13.743 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:45:13.743 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:45:13.743 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:45:13.743 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:45:13.743 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:45:13.743 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:45:13.743 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:45:13.743 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:45:13.743 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:45:13.743 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:45:13.744 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:45:13.744 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:45:13.747 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:45:13.747 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:45:13.747 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:45:13.747 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:45:13.747 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:45:13.747 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:45:13.747 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:45:13.747 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:45:13.748 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:45:13.748 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:45:13.748 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:45:13.748 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:45:13.748 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:45:13.748 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:45:13.748 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:45:13.748 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:45:13.748 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:45:13.748 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:45:13.748 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:45:13.748 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:45:13.748 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:45:13.748 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:45:13.748 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:45:13.749 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:45:13.749 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:45:13.749 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:45:13.749 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:45:13.749 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:45:13.749 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:45:13.749 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:45:13.749 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:45:13.749 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:45:13.773 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:45:13.774 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:45:13.774 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:45:13.774 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:45:13.904 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:45:13.905 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:45:13.906 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:45:13.906 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:45:13.906 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:45:13.906 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:45:13.906 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:45:13.906 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:45:13.906 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:45:13.906 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:45:13.907 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:45:13.907 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:45:13.907 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:45:13.909 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:45:13.910 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:45:13.910 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:45:13.965 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 31 2024-04-06T09:45:13.965 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 33 2024-04-06T09:45:13.966 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:45:13.966 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=133, tag_tid=21, entry_tid=1], [object_number=132, tag_tid=21, entry_tid=0], [object_number=135, tag_tid=20, entry_tid=19939], [object_number=134, tag_tid=20, entry_tid=19938]]], state=connected] 2024-04-06T09:45:13.966 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=124, tag_tid=20, entry_tid=14532], [object_number=127, tag_tid=20, entry_tid=14531], [object_number=126, tag_tid=20, entry_tid=14530], [object_number=125, tag_tid=20, entry_tid=14529]]], state=connected] 2024-04-06T09:45:13.969 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:45:13.970 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=133, tag_tid=21, entry_tid=1' 2024-04-06T09:45:13.972 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:45:13.972 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:45:13.972 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:45:13.972 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:45:13.973 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:45:13.973 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:45:13.973 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:45:13.973 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:45:13.973 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:45:13.973 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:45:13.973 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:45:13.973 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:45:13.974 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:45:13.974 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:45:13.974 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:45:14.022 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 31 2024-04-06T09:45:14.023 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 33 2024-04-06T09:45:14.023 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:45:14.023 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=133, tag_tid=21, entry_tid=1], [object_number=132, tag_tid=21, entry_tid=0], [object_number=135, tag_tid=20, entry_tid=19939], [object_number=134, tag_tid=20, entry_tid=19938]]], state=connected] 2024-04-06T09:45:14.023 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=124, tag_tid=20, entry_tid=14532], [object_number=127, tag_tid=20, entry_tid=14531], [object_number=126, tag_tid=20, entry_tid=14530], [object_number=125, tag_tid=20, entry_tid=14529]]], state=connected] 2024-04-06T09:45:14.026 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:45:14.028 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=124, tag_tid=20, entry_tid=14532' 2024-04-06T09:45:14.028 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=133, tag_tid=21, entry_tid=1' -a 'object_number=133, tag_tid=21, entry_tid=1' = 'object_number=124, tag_tid=20, entry_tid=14532' 2024-04-06T09:45:14.028 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=124, tag_tid=20, entry_tid=14532' '!=' 'object_number=127, tag_tid=20, entry_tid=14431' 2024-04-06T09:45:14.028 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:45:14.028 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=124, tag_tid=20, entry_tid=14532' = 'object_number=127, tag_tid=20, entry_tid=14431' 2024-04-06T09:45:14.028 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=124, tag_tid=20, entry_tid=14532' 2024-04-06T09:45:14.029 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=133, tag_tid=21, entry_tid=1' 2024-04-06T09:45:14.029 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:45:14.029 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:45:14.029 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=21 2024-04-06T09:45:14.030 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=124, tag_tid=20, entry_tid=14532' 2024-04-06T09:45:14.030 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:45:14.030 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:45:14.031 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=20 2024-04-06T09:45:14.032 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=133, tag_tid=21, entry_tid=1' 2024-04-06T09:45:14.032 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:45:14.032 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:45:14.033 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:45:14.033 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=124, tag_tid=20, entry_tid=14532' 2024-04-06T09:45:14.033 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:45:14.033 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:45:14.035 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=14532 2024-04-06T09:45:14.035 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 21 = 20 -a 1 -le 14532 2024-04-06T09:45:14.035 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:45:14.035 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:45:14.035 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:45:14.236 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:45:14.237 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:45:14.237 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:45:14.237 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:45:14.237 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:45:14.237 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:45:14.237 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:45:14.237 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:45:14.237 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:45:14.237 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:45:14.237 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:45:14.237 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:45:14.237 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:45:14.237 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:45:14.238 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:45:14.238 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:45:14.238 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:45:14.241 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:45:14.241 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:45:14.241 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:45:14.241 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:45:14.242 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:45:14.242 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:45:14.242 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:45:14.242 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:45:14.242 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:45:14.242 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:45:14.242 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:45:14.242 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:45:14.242 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:45:14.242 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:45:14.242 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:45:14.242 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:45:14.242 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:45:14.242 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:45:14.242 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:45:14.243 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:45:14.243 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:45:14.243 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:45:14.243 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:45:14.243 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:45:14.243 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:45:14.243 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:45:14.243 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:45:14.243 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:45:14.243 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:45:14.243 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:45:14.243 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:45:14.244 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:45:14.267 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:45:14.267 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:45:14.267 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:45:14.267 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:45:14.402 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:45:14.402 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:45:14.402 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:45:14.403 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:45:14.403 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:45:14.403 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:45:14.403 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:45:14.403 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:45:14.403 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:45:14.403 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:45:14.404 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:45:14.404 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:45:14.404 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:45:14.406 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:45:14.407 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:45:14.407 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:45:14.460 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 31 2024-04-06T09:45:14.460 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 33 2024-04-06T09:45:14.460 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:45:14.460 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=133, tag_tid=21, entry_tid=1], [object_number=132, tag_tid=21, entry_tid=0], [object_number=135, tag_tid=20, entry_tid=19939], [object_number=134, tag_tid=20, entry_tid=19938]]], state=connected] 2024-04-06T09:45:14.460 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=124, tag_tid=20, entry_tid=14624], [object_number=127, tag_tid=20, entry_tid=14623], [object_number=126, tag_tid=20, entry_tid=14622], [object_number=125, tag_tid=20, entry_tid=14621]]], state=connected] 2024-04-06T09:45:14.464 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:45:14.465 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=133, tag_tid=21, entry_tid=1' 2024-04-06T09:45:14.466 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:45:14.466 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:45:14.466 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:45:14.466 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:45:14.467 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:45:14.467 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:45:14.467 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:45:14.467 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:45:14.467 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:45:14.467 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:45:14.467 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:45:14.467 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:45:14.468 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:45:14.468 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:45:14.468 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:45:14.515 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 31 2024-04-06T09:45:14.515 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 33 2024-04-06T09:45:14.515 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:45:14.515 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=133, tag_tid=21, entry_tid=1], [object_number=132, tag_tid=21, entry_tid=0], [object_number=135, tag_tid=20, entry_tid=19939], [object_number=134, tag_tid=20, entry_tid=19938]]], state=connected] 2024-04-06T09:45:14.515 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=124, tag_tid=20, entry_tid=14624], [object_number=127, tag_tid=20, entry_tid=14623], [object_number=126, tag_tid=20, entry_tid=14622], [object_number=125, tag_tid=20, entry_tid=14621]]], state=connected] 2024-04-06T09:45:14.519 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:45:14.520 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=124, tag_tid=20, entry_tid=14624' 2024-04-06T09:45:14.520 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=133, tag_tid=21, entry_tid=1' -a 'object_number=133, tag_tid=21, entry_tid=1' = 'object_number=124, tag_tid=20, entry_tid=14624' 2024-04-06T09:45:14.521 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=124, tag_tid=20, entry_tid=14624' '!=' 'object_number=124, tag_tid=20, entry_tid=14532' 2024-04-06T09:45:14.521 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:45:14.521 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=124, tag_tid=20, entry_tid=14624' = 'object_number=124, tag_tid=20, entry_tid=14532' 2024-04-06T09:45:14.521 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=124, tag_tid=20, entry_tid=14624' 2024-04-06T09:45:14.521 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=133, tag_tid=21, entry_tid=1' 2024-04-06T09:45:14.521 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:45:14.522 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:45:14.522 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=21 2024-04-06T09:45:14.523 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=124, tag_tid=20, entry_tid=14624' 2024-04-06T09:45:14.523 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:45:14.523 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:45:14.524 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=20 2024-04-06T09:45:14.524 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=133, tag_tid=21, entry_tid=1' 2024-04-06T09:45:14.525 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:45:14.525 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:45:14.525 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:45:14.526 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=124, tag_tid=20, entry_tid=14624' 2024-04-06T09:45:14.526 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:45:14.526 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:45:14.527 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=14624 2024-04-06T09:45:14.528 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 21 = 20 -a 1 -le 14624 2024-04-06T09:45:14.528 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:45:14.528 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:45:14.528 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:45:14.729 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:45:14.730 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:45:14.730 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:45:14.730 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:45:14.730 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:45:14.730 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:45:14.730 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:45:14.730 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:45:14.730 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:45:14.730 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:45:14.730 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:45:14.730 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:45:14.731 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:45:14.731 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:45:14.731 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:45:14.731 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:45:14.731 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:45:14.733 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:45:14.733 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:45:14.733 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:45:14.733 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:45:14.733 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:45:14.733 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:45:14.733 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:45:14.733 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:45:14.733 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:45:14.733 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:45:14.734 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:45:14.734 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:45:14.734 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:45:14.734 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:45:14.734 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:45:14.734 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:45:14.735 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:45:14.735 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:45:14.735 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:45:14.735 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:45:14.735 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:45:14.735 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:45:14.735 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:45:14.736 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:45:14.736 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:45:14.736 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:45:14.736 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:45:14.736 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:45:14.736 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:45:14.736 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:45:14.736 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:45:14.736 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:45:14.760 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:45:14.760 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:45:14.760 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:45:14.760 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:45:14.889 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:45:14.890 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:45:14.890 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:45:14.890 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:45:14.891 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:45:14.891 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:45:14.891 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:45:14.891 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:45:14.891 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:45:14.891 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:45:14.891 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:45:14.891 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:45:14.891 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:45:14.893 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:45:14.894 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:45:14.894 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:45:14.946 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 31 2024-04-06T09:45:14.946 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 33 2024-04-06T09:45:14.946 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:45:14.946 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=133, tag_tid=21, entry_tid=1], [object_number=132, tag_tid=21, entry_tid=0], [object_number=135, tag_tid=20, entry_tid=19939], [object_number=134, tag_tid=20, entry_tid=19938]]], state=connected] 2024-04-06T09:45:14.946 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=124, tag_tid=20, entry_tid=14712], [object_number=127, tag_tid=20, entry_tid=14711], [object_number=126, tag_tid=20, entry_tid=14710], [object_number=125, tag_tid=20, entry_tid=14709]]], state=connected] 2024-04-06T09:45:14.949 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:45:14.951 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=133, tag_tid=21, entry_tid=1' 2024-04-06T09:45:14.952 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:45:14.952 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:45:14.952 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:45:14.952 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:45:14.952 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:45:14.952 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:45:14.952 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:45:14.953 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:45:14.953 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:45:14.953 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:45:14.953 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:45:14.953 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:45:14.953 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:45:14.954 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:45:14.954 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:45:15.008 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 31 2024-04-06T09:45:15.008 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 33 2024-04-06T09:45:15.008 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:45:15.008 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=133, tag_tid=21, entry_tid=1], [object_number=132, tag_tid=21, entry_tid=0], [object_number=135, tag_tid=20, entry_tid=19939], [object_number=134, tag_tid=20, entry_tid=19938]]], state=connected] 2024-04-06T09:45:15.009 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=124, tag_tid=20, entry_tid=14712], [object_number=127, tag_tid=20, entry_tid=14711], [object_number=126, tag_tid=20, entry_tid=14710], [object_number=125, tag_tid=20, entry_tid=14709]]], state=connected] 2024-04-06T09:45:15.012 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:45:15.014 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=124, tag_tid=20, entry_tid=14712' 2024-04-06T09:45:15.014 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=133, tag_tid=21, entry_tid=1' -a 'object_number=133, tag_tid=21, entry_tid=1' = 'object_number=124, tag_tid=20, entry_tid=14712' 2024-04-06T09:45:15.014 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=124, tag_tid=20, entry_tid=14712' '!=' 'object_number=124, tag_tid=20, entry_tid=14624' 2024-04-06T09:45:15.014 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:45:15.014 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=124, tag_tid=20, entry_tid=14712' = 'object_number=124, tag_tid=20, entry_tid=14624' 2024-04-06T09:45:15.014 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=124, tag_tid=20, entry_tid=14712' 2024-04-06T09:45:15.015 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=133, tag_tid=21, entry_tid=1' 2024-04-06T09:45:15.015 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:45:15.015 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:45:15.015 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=21 2024-04-06T09:45:15.016 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=124, tag_tid=20, entry_tid=14712' 2024-04-06T09:45:15.016 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:45:15.016 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:45:15.017 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=20 2024-04-06T09:45:15.018 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=133, tag_tid=21, entry_tid=1' 2024-04-06T09:45:15.018 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:45:15.018 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:45:15.019 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:45:15.020 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=124, tag_tid=20, entry_tid=14712' 2024-04-06T09:45:15.020 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:45:15.020 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:45:15.021 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=14712 2024-04-06T09:45:15.021 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 21 = 20 -a 1 -le 14712 2024-04-06T09:45:15.021 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:45:15.021 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:45:15.021 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:45:15.223 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:45:15.223 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:45:15.223 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:45:15.223 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:45:15.223 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:45:15.223 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:45:15.223 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:45:15.223 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:45:15.223 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:45:15.224 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:45:15.224 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:45:15.224 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:45:15.224 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:45:15.224 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:45:15.224 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:45:15.224 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:45:15.224 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:45:15.228 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:45:15.228 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:45:15.228 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:45:15.228 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:45:15.228 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:45:15.228 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:45:15.229 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:45:15.229 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:45:15.229 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:45:15.229 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:45:15.229 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:45:15.229 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:45:15.229 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:45:15.229 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:45:15.229 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:45:15.230 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:45:15.230 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:45:15.230 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:45:15.230 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:45:15.230 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:45:15.230 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:45:15.230 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:45:15.230 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:45:15.230 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:45:15.230 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:45:15.230 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:45:15.230 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:45:15.230 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:45:15.230 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:45:15.230 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:45:15.231 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:45:15.231 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:45:15.255 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:45:15.256 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:45:15.256 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:45:15.256 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:45:15.383 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:45:15.384 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:45:15.384 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:45:15.384 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:45:15.384 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:45:15.384 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:45:15.384 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:45:15.384 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:45:15.385 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:45:15.385 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:45:15.385 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:45:15.385 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:45:15.385 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:45:15.387 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:45:15.387 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:45:15.387 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:45:15.434 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 31 2024-04-06T09:45:15.435 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 33 2024-04-06T09:45:15.435 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:45:15.435 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=133, tag_tid=21, entry_tid=1], [object_number=132, tag_tid=21, entry_tid=0], [object_number=135, tag_tid=20, entry_tid=19939], [object_number=134, tag_tid=20, entry_tid=19938]]], state=connected] 2024-04-06T09:45:15.435 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=124, tag_tid=20, entry_tid=14788], [object_number=127, tag_tid=20, entry_tid=14787], [object_number=126, tag_tid=20, entry_tid=14786], [object_number=125, tag_tid=20, entry_tid=14785]]], state=connected] 2024-04-06T09:45:15.438 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:45:15.439 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=133, tag_tid=21, entry_tid=1' 2024-04-06T09:45:15.440 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:45:15.440 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:45:15.441 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:45:15.441 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:45:15.441 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:45:15.441 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:45:15.441 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:45:15.441 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:45:15.441 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:45:15.441 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:45:15.441 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:45:15.441 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:45:15.442 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:45:15.442 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:45:15.442 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:45:15.488 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 31 2024-04-06T09:45:15.489 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 33 2024-04-06T09:45:15.489 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:45:15.489 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=133, tag_tid=21, entry_tid=1], [object_number=132, tag_tid=21, entry_tid=0], [object_number=135, tag_tid=20, entry_tid=19939], [object_number=134, tag_tid=20, entry_tid=19938]]], state=connected] 2024-04-06T09:45:15.489 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=124, tag_tid=20, entry_tid=14788], [object_number=127, tag_tid=20, entry_tid=14787], [object_number=126, tag_tid=20, entry_tid=14786], [object_number=125, tag_tid=20, entry_tid=14785]]], state=connected] 2024-04-06T09:45:15.492 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:45:15.494 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=124, tag_tid=20, entry_tid=14788' 2024-04-06T09:45:15.494 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=133, tag_tid=21, entry_tid=1' -a 'object_number=133, tag_tid=21, entry_tid=1' = 'object_number=124, tag_tid=20, entry_tid=14788' 2024-04-06T09:45:15.494 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=124, tag_tid=20, entry_tid=14788' '!=' 'object_number=124, tag_tid=20, entry_tid=14712' 2024-04-06T09:45:15.494 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:45:15.494 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=124, tag_tid=20, entry_tid=14788' = 'object_number=124, tag_tid=20, entry_tid=14712' 2024-04-06T09:45:15.494 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=124, tag_tid=20, entry_tid=14788' 2024-04-06T09:45:15.494 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=133, tag_tid=21, entry_tid=1' 2024-04-06T09:45:15.494 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:45:15.495 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:45:15.495 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=21 2024-04-06T09:45:15.496 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=124, tag_tid=20, entry_tid=14788' 2024-04-06T09:45:15.496 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:45:15.496 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:45:15.497 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=20 2024-04-06T09:45:15.497 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=133, tag_tid=21, entry_tid=1' 2024-04-06T09:45:15.497 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:45:15.498 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:45:15.498 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:45:15.499 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=124, tag_tid=20, entry_tid=14788' 2024-04-06T09:45:15.499 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:45:15.499 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:45:15.500 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=14788 2024-04-06T09:45:15.500 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 21 = 20 -a 1 -le 14788 2024-04-06T09:45:15.501 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:45:15.501 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:45:15.501 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:45:15.703 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:45:15.703 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:45:15.703 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:45:15.703 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:45:15.703 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:45:15.703 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:45:15.703 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:45:15.703 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:45:15.703 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:45:15.704 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:45:15.704 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:45:15.704 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:45:15.704 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:45:15.704 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:45:15.704 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:45:15.704 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:45:15.705 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:45:15.707 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:45:15.707 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:45:15.707 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:45:15.707 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:45:15.708 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:45:15.708 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:45:15.708 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:45:15.708 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:45:15.708 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:45:15.708 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:45:15.708 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:45:15.708 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:45:15.708 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:45:15.708 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:45:15.708 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:45:15.708 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:45:15.708 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:45:15.709 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:45:15.709 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:45:15.709 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:45:15.709 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:45:15.709 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:45:15.709 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:45:15.709 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:45:15.709 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:45:15.709 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:45:15.709 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:45:15.709 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:45:15.710 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:45:15.710 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:45:15.710 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:45:15.710 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:45:15.734 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:45:15.735 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:45:15.735 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:45:15.735 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:45:15.869 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:45:15.871 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:45:15.871 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:45:15.871 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:45:15.871 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:45:15.871 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:45:15.871 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:45:15.871 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:45:15.871 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:45:15.871 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:45:15.872 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:45:15.872 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:45:15.872 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:45:15.873 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:45:15.874 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:45:15.874 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:45:15.928 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 31 2024-04-06T09:45:15.928 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 33 2024-04-06T09:45:15.928 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:45:15.928 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=133, tag_tid=21, entry_tid=1], [object_number=132, tag_tid=21, entry_tid=0], [object_number=135, tag_tid=20, entry_tid=19939], [object_number=134, tag_tid=20, entry_tid=19938]]], state=connected] 2024-04-06T09:45:15.929 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=124, tag_tid=20, entry_tid=14864], [object_number=127, tag_tid=20, entry_tid=14863], [object_number=126, tag_tid=20, entry_tid=14862], [object_number=125, tag_tid=20, entry_tid=14861]]], state=connected] 2024-04-06T09:45:15.935 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:45:15.935 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=133, tag_tid=21, entry_tid=1' 2024-04-06T09:45:15.935 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:45:15.935 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:45:15.935 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:45:15.935 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:45:15.935 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:45:15.935 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:45:15.936 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:45:15.936 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:45:15.936 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:45:15.936 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:45:15.936 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:45:15.936 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:45:15.936 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:45:15.937 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:45:15.937 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:45:15.984 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 31 2024-04-06T09:45:15.985 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 33 2024-04-06T09:45:15.985 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:45:15.985 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=133, tag_tid=21, entry_tid=1], [object_number=132, tag_tid=21, entry_tid=0], [object_number=135, tag_tid=20, entry_tid=19939], [object_number=134, tag_tid=20, entry_tid=19938]]], state=connected] 2024-04-06T09:45:15.985 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=124, tag_tid=20, entry_tid=14864], [object_number=127, tag_tid=20, entry_tid=14863], [object_number=126, tag_tid=20, entry_tid=14862], [object_number=125, tag_tid=20, entry_tid=14861]]], state=connected] 2024-04-06T09:45:15.988 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:45:15.990 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=124, tag_tid=20, entry_tid=14864' 2024-04-06T09:45:15.990 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=133, tag_tid=21, entry_tid=1' -a 'object_number=133, tag_tid=21, entry_tid=1' = 'object_number=124, tag_tid=20, entry_tid=14864' 2024-04-06T09:45:15.990 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=124, tag_tid=20, entry_tid=14864' '!=' 'object_number=124, tag_tid=20, entry_tid=14788' 2024-04-06T09:45:15.990 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:45:15.990 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=124, tag_tid=20, entry_tid=14864' = 'object_number=124, tag_tid=20, entry_tid=14788' 2024-04-06T09:45:15.990 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=124, tag_tid=20, entry_tid=14864' 2024-04-06T09:45:15.991 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=133, tag_tid=21, entry_tid=1' 2024-04-06T09:45:15.991 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:45:15.991 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:45:15.991 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=21 2024-04-06T09:45:15.992 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=124, tag_tid=20, entry_tid=14864' 2024-04-06T09:45:15.992 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:45:15.992 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:45:15.993 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=20 2024-04-06T09:45:15.994 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=133, tag_tid=21, entry_tid=1' 2024-04-06T09:45:15.994 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:45:15.994 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:45:15.995 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:45:15.995 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=124, tag_tid=20, entry_tid=14864' 2024-04-06T09:45:15.996 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:45:15.996 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:45:15.997 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=14864 2024-04-06T09:45:15.997 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 21 = 20 -a 1 -le 14864 2024-04-06T09:45:15.997 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:45:15.997 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:45:15.997 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:45:16.199 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:45:16.199 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:45:16.199 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:45:16.199 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:45:16.199 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:45:16.199 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:45:16.199 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:45:16.199 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:45:16.199 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:45:16.199 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:45:16.199 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:45:16.200 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:45:16.200 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:45:16.200 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:45:16.200 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:45:16.200 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:45:16.201 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:45:16.203 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:45:16.203 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:45:16.203 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:45:16.203 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:45:16.203 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:45:16.203 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:45:16.203 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:45:16.204 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:45:16.204 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:45:16.204 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:45:16.204 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:45:16.204 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:45:16.204 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:45:16.204 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:45:16.204 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:45:16.204 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:45:16.205 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:45:16.205 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:45:16.205 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:45:16.205 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:45:16.205 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:45:16.205 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:45:16.205 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:45:16.205 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:45:16.206 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:45:16.206 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:45:16.206 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:45:16.206 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:45:16.206 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:45:16.206 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:45:16.206 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:45:16.206 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:45:16.231 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:45:16.231 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:45:16.231 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:45:16.231 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:45:16.360 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:45:16.361 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:45:16.362 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:45:16.362 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:45:16.362 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:45:16.362 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:45:16.362 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:45:16.362 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:45:16.362 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:45:16.362 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:45:16.363 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:45:16.363 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:45:16.363 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:45:16.365 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:45:16.366 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:45:16.366 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:45:16.419 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 31 2024-04-06T09:45:16.419 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 33 2024-04-06T09:45:16.419 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:45:16.419 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=133, tag_tid=21, entry_tid=1], [object_number=132, tag_tid=21, entry_tid=0], [object_number=135, tag_tid=20, entry_tid=19939], [object_number=134, tag_tid=20, entry_tid=19938]]], state=connected] 2024-04-06T09:45:16.419 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=126, tag_tid=20, entry_tid=14938], [object_number=125, tag_tid=20, entry_tid=14937], [object_number=124, tag_tid=20, entry_tid=14936], [object_number=127, tag_tid=20, entry_tid=14935]]], state=connected] 2024-04-06T09:45:16.423 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:45:16.424 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=133, tag_tid=21, entry_tid=1' 2024-04-06T09:45:16.425 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:45:16.425 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:45:16.425 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:45:16.425 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:45:16.425 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:45:16.426 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:45:16.426 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:45:16.426 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:45:16.426 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:45:16.426 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:45:16.426 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:45:16.426 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:45:16.427 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:45:16.427 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:45:16.427 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:45:16.476 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 31 2024-04-06T09:45:16.476 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 33 2024-04-06T09:45:16.476 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:45:16.476 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=133, tag_tid=21, entry_tid=1], [object_number=132, tag_tid=21, entry_tid=0], [object_number=135, tag_tid=20, entry_tid=19939], [object_number=134, tag_tid=20, entry_tid=19938]]], state=connected] 2024-04-06T09:45:16.477 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=126, tag_tid=20, entry_tid=14938], [object_number=125, tag_tid=20, entry_tid=14937], [object_number=124, tag_tid=20, entry_tid=14936], [object_number=127, tag_tid=20, entry_tid=14935]]], state=connected] 2024-04-06T09:45:16.480 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:45:16.482 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=126, tag_tid=20, entry_tid=14938' 2024-04-06T09:45:16.482 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=133, tag_tid=21, entry_tid=1' -a 'object_number=133, tag_tid=21, entry_tid=1' = 'object_number=126, tag_tid=20, entry_tid=14938' 2024-04-06T09:45:16.482 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=126, tag_tid=20, entry_tid=14938' '!=' 'object_number=124, tag_tid=20, entry_tid=14864' 2024-04-06T09:45:16.482 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:45:16.482 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=126, tag_tid=20, entry_tid=14938' = 'object_number=124, tag_tid=20, entry_tid=14864' 2024-04-06T09:45:16.482 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=126, tag_tid=20, entry_tid=14938' 2024-04-06T09:45:16.482 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=133, tag_tid=21, entry_tid=1' 2024-04-06T09:45:16.483 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:45:16.483 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:45:16.483 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=21 2024-04-06T09:45:16.484 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=126, tag_tid=20, entry_tid=14938' 2024-04-06T09:45:16.484 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:45:16.484 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:45:16.485 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=20 2024-04-06T09:45:16.485 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=133, tag_tid=21, entry_tid=1' 2024-04-06T09:45:16.486 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:45:16.486 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:45:16.486 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:45:16.487 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=126, tag_tid=20, entry_tid=14938' 2024-04-06T09:45:16.487 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:45:16.487 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:45:16.488 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=14938 2024-04-06T09:45:16.489 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 21 = 20 -a 1 -le 14938 2024-04-06T09:45:16.489 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:45:16.489 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:45:16.489 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:45:16.690 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:45:16.691 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:45:16.691 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:45:16.691 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:45:16.691 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:45:16.691 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:45:16.691 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:45:16.691 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:45:16.691 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:45:16.691 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:45:16.691 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:45:16.691 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:45:16.691 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:45:16.691 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:45:16.692 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:45:16.692 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:45:16.692 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:45:16.694 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:45:16.694 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:45:16.694 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:45:16.694 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:45:16.695 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:45:16.695 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:45:16.695 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:45:16.695 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:45:16.695 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:45:16.695 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:45:16.695 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:45:16.695 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:45:16.695 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:45:16.695 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:45:16.695 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:45:16.695 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:45:16.695 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:45:16.695 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:45:16.695 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:45:16.696 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:45:16.696 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:45:16.696 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:45:16.696 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:45:16.696 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:45:16.696 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:45:16.696 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:45:16.696 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:45:16.696 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:45:16.696 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:45:16.696 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:45:16.696 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:45:16.696 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:45:16.720 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:45:16.720 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:45:16.721 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:45:16.721 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:45:16.914 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:45:16.914 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:45:16.914 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:45:16.914 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:45:16.915 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:45:16.915 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:45:16.915 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:45:16.915 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:45:16.915 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:45:16.915 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:45:16.915 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:45:16.915 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:45:16.915 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:45:16.916 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:45:16.916 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:45:16.916 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:45:16.961 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 32 2024-04-06T09:45:16.962 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 33 2024-04-06T09:45:16.962 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:45:16.962 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=133, tag_tid=21, entry_tid=1], [object_number=132, tag_tid=21, entry_tid=0], [object_number=135, tag_tid=20, entry_tid=19939], [object_number=134, tag_tid=20, entry_tid=19938]]], state=connected] 2024-04-06T09:45:16.962 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=128, tag_tid=20, entry_tid=14976], [object_number=131, tag_tid=20, entry_tid=14975], [object_number=130, tag_tid=20, entry_tid=14974], [object_number=129, tag_tid=20, entry_tid=14973]]], state=connected] 2024-04-06T09:45:16.965 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:45:16.966 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=133, tag_tid=21, entry_tid=1' 2024-04-06T09:45:16.967 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:45:16.967 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:45:16.968 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:45:16.968 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:45:16.968 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:45:16.968 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:45:16.968 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:45:16.968 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:45:16.968 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:45:16.968 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:45:16.968 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:45:16.968 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:45:16.968 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:45:16.969 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:45:16.969 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:45:17.015 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 32 2024-04-06T09:45:17.015 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 33 2024-04-06T09:45:17.015 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:45:17.015 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=133, tag_tid=21, entry_tid=1], [object_number=132, tag_tid=21, entry_tid=0], [object_number=135, tag_tid=20, entry_tid=19939], [object_number=134, tag_tid=20, entry_tid=19938]]], state=connected] 2024-04-06T09:45:17.016 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=128, tag_tid=20, entry_tid=14976], [object_number=131, tag_tid=20, entry_tid=14975], [object_number=130, tag_tid=20, entry_tid=14974], [object_number=129, tag_tid=20, entry_tid=14973]]], state=connected] 2024-04-06T09:45:17.019 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:45:17.021 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=128, tag_tid=20, entry_tid=14976' 2024-04-06T09:45:17.021 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=133, tag_tid=21, entry_tid=1' -a 'object_number=133, tag_tid=21, entry_tid=1' = 'object_number=128, tag_tid=20, entry_tid=14976' 2024-04-06T09:45:17.021 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=128, tag_tid=20, entry_tid=14976' '!=' 'object_number=126, tag_tid=20, entry_tid=14938' 2024-04-06T09:45:17.021 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:45:17.021 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=128, tag_tid=20, entry_tid=14976' = 'object_number=126, tag_tid=20, entry_tid=14938' 2024-04-06T09:45:17.021 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=128, tag_tid=20, entry_tid=14976' 2024-04-06T09:45:17.021 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=133, tag_tid=21, entry_tid=1' 2024-04-06T09:45:17.022 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:45:17.022 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:45:17.022 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=21 2024-04-06T09:45:17.023 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=128, tag_tid=20, entry_tid=14976' 2024-04-06T09:45:17.023 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:45:17.023 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:45:17.024 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=20 2024-04-06T09:45:17.024 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=133, tag_tid=21, entry_tid=1' 2024-04-06T09:45:17.025 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:45:17.025 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:45:17.025 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:45:17.026 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=128, tag_tid=20, entry_tid=14976' 2024-04-06T09:45:17.026 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:45:17.026 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:45:17.027 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=14976 2024-04-06T09:45:17.027 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 21 = 20 -a 1 -le 14976 2024-04-06T09:45:17.028 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:45:17.028 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:45:17.028 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:45:17.229 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:45:17.230 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:45:17.230 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:45:17.230 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:45:17.230 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:45:17.230 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:45:17.230 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:45:17.230 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:45:17.230 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:45:17.230 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:45:17.230 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:45:17.230 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:45:17.231 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:45:17.231 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:45:17.231 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:45:17.231 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:45:17.231 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:45:17.233 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:45:17.233 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:45:17.233 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:45:17.233 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:45:17.233 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:45:17.233 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:45:17.233 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:45:17.233 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:45:17.233 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:45:17.233 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:45:17.233 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:45:17.233 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:45:17.234 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:45:17.234 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:45:17.234 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:45:17.234 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:45:17.235 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:45:17.235 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:45:17.235 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:45:17.235 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:45:17.235 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:45:17.236 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:45:17.236 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:45:17.236 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:45:17.236 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:45:17.236 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:45:17.236 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:45:17.236 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:45:17.236 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:45:17.236 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:45:17.236 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:45:17.236 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:45:17.259 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:45:17.260 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:45:17.260 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:45:17.260 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:45:17.399 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:45:17.400 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:45:17.400 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:45:17.400 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:45:17.400 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:45:17.400 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:45:17.400 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:45:17.400 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:45:17.400 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:45:17.401 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:45:17.401 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:45:17.401 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:45:17.401 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:45:17.403 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:45:17.403 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:45:17.403 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:45:17.451 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 32 2024-04-06T09:45:17.451 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 33 2024-04-06T09:45:17.451 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:45:17.451 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=133, tag_tid=21, entry_tid=1], [object_number=132, tag_tid=21, entry_tid=0], [object_number=135, tag_tid=20, entry_tid=19939], [object_number=134, tag_tid=20, entry_tid=19938]]], state=connected] 2024-04-06T09:45:17.451 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=130, tag_tid=20, entry_tid=15058], [object_number=129, tag_tid=20, entry_tid=15057], [object_number=128, tag_tid=20, entry_tid=15056], [object_number=131, tag_tid=20, entry_tid=15055]]], state=connected] 2024-04-06T09:45:17.455 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:45:17.456 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=133, tag_tid=21, entry_tid=1' 2024-04-06T09:45:17.457 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:45:17.457 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:45:17.457 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:45:17.457 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:45:17.457 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:45:17.457 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:45:17.457 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:45:17.458 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:45:17.458 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:45:17.458 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:45:17.458 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:45:17.458 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:45:17.458 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:45:17.459 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:45:17.459 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:45:17.505 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 32 2024-04-06T09:45:17.505 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 33 2024-04-06T09:45:17.505 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:45:17.506 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=133, tag_tid=21, entry_tid=1], [object_number=132, tag_tid=21, entry_tid=0], [object_number=135, tag_tid=20, entry_tid=19939], [object_number=134, tag_tid=20, entry_tid=19938]]], state=connected] 2024-04-06T09:45:17.506 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=130, tag_tid=20, entry_tid=15058], [object_number=129, tag_tid=20, entry_tid=15057], [object_number=128, tag_tid=20, entry_tid=15056], [object_number=131, tag_tid=20, entry_tid=15055]]], state=connected] 2024-04-06T09:45:17.509 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:45:17.511 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=130, tag_tid=20, entry_tid=15058' 2024-04-06T09:45:17.511 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=133, tag_tid=21, entry_tid=1' -a 'object_number=133, tag_tid=21, entry_tid=1' = 'object_number=130, tag_tid=20, entry_tid=15058' 2024-04-06T09:45:17.511 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=130, tag_tid=20, entry_tid=15058' '!=' 'object_number=128, tag_tid=20, entry_tid=14976' 2024-04-06T09:45:17.511 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:45:17.511 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=130, tag_tid=20, entry_tid=15058' = 'object_number=128, tag_tid=20, entry_tid=14976' 2024-04-06T09:45:17.511 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=130, tag_tid=20, entry_tid=15058' 2024-04-06T09:45:17.512 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=133, tag_tid=21, entry_tid=1' 2024-04-06T09:45:17.512 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:45:17.512 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:45:17.512 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=21 2024-04-06T09:45:17.513 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=130, tag_tid=20, entry_tid=15058' 2024-04-06T09:45:17.513 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:45:17.513 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:45:17.514 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=20 2024-04-06T09:45:17.515 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=133, tag_tid=21, entry_tid=1' 2024-04-06T09:45:17.515 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:45:17.515 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:45:17.516 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:45:17.517 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=130, tag_tid=20, entry_tid=15058' 2024-04-06T09:45:17.517 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:45:17.517 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:45:17.518 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=15058 2024-04-06T09:45:17.518 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 21 = 20 -a 1 -le 15058 2024-04-06T09:45:17.518 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:45:17.518 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:45:17.518 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:45:17.720 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:45:17.720 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:45:17.720 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:45:17.720 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:45:17.720 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:45:17.720 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:45:17.720 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:45:17.720 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:45:17.720 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:45:17.720 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:45:17.721 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:45:17.721 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:45:17.721 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:45:17.721 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:45:17.721 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:45:17.721 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:45:17.722 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:45:17.724 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:45:17.724 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:45:17.724 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:45:17.724 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:45:17.724 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:45:17.724 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:45:17.724 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:45:17.724 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:45:17.724 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:45:17.724 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:45:17.724 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:45:17.725 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:45:17.725 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:45:17.725 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:45:17.725 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:45:17.725 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:45:17.726 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:45:17.726 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:45:17.726 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:45:17.726 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:45:17.726 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:45:17.726 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:45:17.727 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:45:17.727 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:45:17.727 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:45:17.727 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:45:17.727 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:45:17.727 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:45:17.727 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:45:17.727 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:45:17.727 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:45:17.727 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:45:17.750 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:45:17.751 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:45:17.751 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:45:17.751 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:45:17.883 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:45:17.884 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:45:17.885 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:45:17.885 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:45:17.885 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:45:17.885 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:45:17.885 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:45:17.885 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:45:17.885 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:45:17.885 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:45:17.886 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:45:17.886 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:45:17.886 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:45:17.887 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:45:17.887 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:45:17.887 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:45:17.935 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 32 2024-04-06T09:45:17.935 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 33 2024-04-06T09:45:17.935 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:45:17.935 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=133, tag_tid=21, entry_tid=1], [object_number=132, tag_tid=21, entry_tid=0], [object_number=135, tag_tid=20, entry_tid=19939], [object_number=134, tag_tid=20, entry_tid=19938]]], state=connected] 2024-04-06T09:45:17.935 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=129, tag_tid=20, entry_tid=15137], [object_number=128, tag_tid=20, entry_tid=15136], [object_number=131, tag_tid=20, entry_tid=15135], [object_number=130, tag_tid=20, entry_tid=15134]]], state=connected] 2024-04-06T09:45:17.939 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:45:17.940 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=133, tag_tid=21, entry_tid=1' 2024-04-06T09:45:17.941 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:45:17.941 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:45:17.941 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:45:17.942 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:45:17.942 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:45:17.942 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:45:17.942 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:45:17.942 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:45:17.942 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:45:17.942 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:45:17.942 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:45:17.942 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:45:17.943 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:45:17.943 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:45:17.943 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:45:17.989 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 32 2024-04-06T09:45:17.990 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 33 2024-04-06T09:45:17.990 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:45:17.990 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=133, tag_tid=21, entry_tid=1], [object_number=132, tag_tid=21, entry_tid=0], [object_number=135, tag_tid=20, entry_tid=19939], [object_number=134, tag_tid=20, entry_tid=19938]]], state=connected] 2024-04-06T09:45:17.990 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=129, tag_tid=20, entry_tid=15137], [object_number=128, tag_tid=20, entry_tid=15136], [object_number=131, tag_tid=20, entry_tid=15135], [object_number=130, tag_tid=20, entry_tid=15134]]], state=connected] 2024-04-06T09:45:17.993 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:45:17.995 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=129, tag_tid=20, entry_tid=15137' 2024-04-06T09:45:17.995 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=133, tag_tid=21, entry_tid=1' -a 'object_number=133, tag_tid=21, entry_tid=1' = 'object_number=129, tag_tid=20, entry_tid=15137' 2024-04-06T09:45:17.995 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=129, tag_tid=20, entry_tid=15137' '!=' 'object_number=130, tag_tid=20, entry_tid=15058' 2024-04-06T09:45:17.995 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:45:17.995 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=129, tag_tid=20, entry_tid=15137' = 'object_number=130, tag_tid=20, entry_tid=15058' 2024-04-06T09:45:17.995 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=129, tag_tid=20, entry_tid=15137' 2024-04-06T09:45:17.995 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=133, tag_tid=21, entry_tid=1' 2024-04-06T09:45:17.996 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:45:17.996 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:45:17.996 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=21 2024-04-06T09:45:17.997 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=129, tag_tid=20, entry_tid=15137' 2024-04-06T09:45:17.997 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:45:17.997 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:45:17.998 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=20 2024-04-06T09:45:17.998 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=133, tag_tid=21, entry_tid=1' 2024-04-06T09:45:17.999 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:45:17.999 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:45:17.999 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:45:18.000 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=129, tag_tid=20, entry_tid=15137' 2024-04-06T09:45:18.000 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:45:18.000 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:45:18.001 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=15137 2024-04-06T09:45:18.002 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 21 = 20 -a 1 -le 15137 2024-04-06T09:45:18.002 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:45:18.002 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:45:18.002 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:45:18.203 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:45:18.203 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:45:18.203 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:45:18.204 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:45:18.204 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:45:18.204 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:45:18.204 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:45:18.204 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:45:18.204 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:45:18.204 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:45:18.204 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:45:18.204 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:45:18.204 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:45:18.204 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:45:18.204 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:45:18.205 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:45:18.205 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:45:18.208 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:45:18.208 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:45:18.208 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:45:18.208 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:45:18.208 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:45:18.209 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:45:18.209 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:45:18.209 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:45:18.209 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:45:18.209 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:45:18.209 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:45:18.209 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:45:18.209 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:45:18.209 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:45:18.209 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:45:18.209 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:45:18.209 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:45:18.209 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:45:18.209 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:45:18.210 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:45:18.210 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:45:18.210 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:45:18.210 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:45:18.210 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:45:18.210 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:45:18.210 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:45:18.210 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:45:18.210 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:45:18.210 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:45:18.210 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:45:18.210 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:45:18.210 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:45:18.235 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:45:18.235 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:45:18.235 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:45:18.235 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:45:18.366 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:45:18.367 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:45:18.368 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:45:18.368 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:45:18.368 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:45:18.368 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:45:18.368 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:45:18.368 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:45:18.368 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:45:18.368 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:45:18.369 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:45:18.369 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:45:18.369 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:45:18.370 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:45:18.370 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:45:18.370 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:45:18.423 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 32 2024-04-06T09:45:18.424 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 33 2024-04-06T09:45:18.424 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:45:18.424 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=133, tag_tid=21, entry_tid=1], [object_number=132, tag_tid=21, entry_tid=0], [object_number=135, tag_tid=20, entry_tid=19939], [object_number=134, tag_tid=20, entry_tid=19938]]], state=connected] 2024-04-06T09:45:18.424 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=130, tag_tid=20, entry_tid=15218], [object_number=129, tag_tid=20, entry_tid=15217], [object_number=128, tag_tid=20, entry_tid=15216], [object_number=131, tag_tid=20, entry_tid=15215]]], state=connected] 2024-04-06T09:45:18.427 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:45:18.428 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=133, tag_tid=21, entry_tid=1' 2024-04-06T09:45:18.430 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:45:18.430 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:45:18.430 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:45:18.430 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:45:18.430 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:45:18.430 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:45:18.430 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:45:18.430 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:45:18.430 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:45:18.430 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:45:18.430 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:45:18.430 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:45:18.431 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:45:18.431 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:45:18.431 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:45:18.478 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 32 2024-04-06T09:45:18.478 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 33 2024-04-06T09:45:18.478 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:45:18.479 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=133, tag_tid=21, entry_tid=1], [object_number=132, tag_tid=21, entry_tid=0], [object_number=135, tag_tid=20, entry_tid=19939], [object_number=134, tag_tid=20, entry_tid=19938]]], state=connected] 2024-04-06T09:45:18.479 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=130, tag_tid=20, entry_tid=15218], [object_number=129, tag_tid=20, entry_tid=15217], [object_number=128, tag_tid=20, entry_tid=15216], [object_number=131, tag_tid=20, entry_tid=15215]]], state=connected] 2024-04-06T09:45:18.482 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:45:18.484 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=130, tag_tid=20, entry_tid=15218' 2024-04-06T09:45:18.484 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=133, tag_tid=21, entry_tid=1' -a 'object_number=133, tag_tid=21, entry_tid=1' = 'object_number=130, tag_tid=20, entry_tid=15218' 2024-04-06T09:45:18.484 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=130, tag_tid=20, entry_tid=15218' '!=' 'object_number=129, tag_tid=20, entry_tid=15137' 2024-04-06T09:45:18.484 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:45:18.484 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=130, tag_tid=20, entry_tid=15218' = 'object_number=129, tag_tid=20, entry_tid=15137' 2024-04-06T09:45:18.484 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=130, tag_tid=20, entry_tid=15218' 2024-04-06T09:45:18.485 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=133, tag_tid=21, entry_tid=1' 2024-04-06T09:45:18.485 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:45:18.485 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:45:18.485 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=21 2024-04-06T09:45:18.486 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=130, tag_tid=20, entry_tid=15218' 2024-04-06T09:45:18.486 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:45:18.486 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:45:18.488 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=20 2024-04-06T09:45:18.488 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=133, tag_tid=21, entry_tid=1' 2024-04-06T09:45:18.488 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:45:18.488 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:45:18.489 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:45:18.490 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=130, tag_tid=20, entry_tid=15218' 2024-04-06T09:45:18.490 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:45:18.490 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:45:18.491 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=15218 2024-04-06T09:45:18.491 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 21 = 20 -a 1 -le 15218 2024-04-06T09:45:18.491 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:45:18.491 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:45:18.491 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:45:18.693 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:45:18.693 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:45:18.693 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:45:18.693 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:45:18.693 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:45:18.694 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:45:18.694 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:45:18.694 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:45:18.694 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:45:18.694 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:45:18.694 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:45:18.694 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:45:18.694 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:45:18.694 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:45:18.694 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:45:18.695 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:45:18.695 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:45:18.698 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:45:18.698 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:45:18.699 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:45:18.699 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:45:18.699 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:45:18.699 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:45:18.699 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:45:18.699 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:45:18.699 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:45:18.699 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:45:18.699 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:45:18.699 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:45:18.699 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:45:18.699 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:45:18.699 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:45:18.699 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:45:18.700 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:45:18.700 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:45:18.700 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:45:18.700 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:45:18.700 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:45:18.700 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:45:18.700 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:45:18.700 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:45:18.700 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:45:18.700 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:45:18.700 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:45:18.700 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:45:18.700 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:45:18.700 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:45:18.700 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:45:18.701 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:45:18.723 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:45:18.723 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:45:18.723 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:45:18.723 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:45:18.859 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:45:18.861 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:45:18.861 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:45:18.861 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:45:18.861 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:45:18.861 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:45:18.861 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:45:18.861 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:45:18.861 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:45:18.862 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:45:18.862 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:45:18.862 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:45:18.862 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:45:18.863 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:45:18.863 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:45:18.863 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:45:18.915 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 32 2024-04-06T09:45:18.915 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 33 2024-04-06T09:45:18.915 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:45:18.915 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=133, tag_tid=21, entry_tid=1], [object_number=132, tag_tid=21, entry_tid=0], [object_number=135, tag_tid=20, entry_tid=19939], [object_number=134, tag_tid=20, entry_tid=19938]]], state=connected] 2024-04-06T09:45:18.916 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=128, tag_tid=20, entry_tid=15300], [object_number=131, tag_tid=20, entry_tid=15299], [object_number=130, tag_tid=20, entry_tid=15298], [object_number=129, tag_tid=20, entry_tid=15297]]], state=connected] 2024-04-06T09:45:18.919 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:45:18.920 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=133, tag_tid=21, entry_tid=1' 2024-04-06T09:45:18.921 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:45:18.921 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:45:18.921 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:45:18.921 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:45:18.921 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:45:18.922 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:45:18.922 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:45:18.922 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:45:18.922 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:45:18.922 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:45:18.922 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:45:18.922 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:45:18.922 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:45:18.923 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:45:18.923 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:45:18.969 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 32 2024-04-06T09:45:18.969 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 33 2024-04-06T09:45:18.969 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:45:18.970 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=133, tag_tid=21, entry_tid=1], [object_number=132, tag_tid=21, entry_tid=0], [object_number=135, tag_tid=20, entry_tid=19939], [object_number=134, tag_tid=20, entry_tid=19938]]], state=connected] 2024-04-06T09:45:18.970 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=128, tag_tid=20, entry_tid=15300], [object_number=131, tag_tid=20, entry_tid=15299], [object_number=130, tag_tid=20, entry_tid=15298], [object_number=129, tag_tid=20, entry_tid=15297]]], state=connected] 2024-04-06T09:45:18.973 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:45:18.975 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=128, tag_tid=20, entry_tid=15300' 2024-04-06T09:45:18.975 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=133, tag_tid=21, entry_tid=1' -a 'object_number=133, tag_tid=21, entry_tid=1' = 'object_number=128, tag_tid=20, entry_tid=15300' 2024-04-06T09:45:18.975 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=128, tag_tid=20, entry_tid=15300' '!=' 'object_number=130, tag_tid=20, entry_tid=15218' 2024-04-06T09:45:18.975 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:45:18.975 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=128, tag_tid=20, entry_tid=15300' = 'object_number=130, tag_tid=20, entry_tid=15218' 2024-04-06T09:45:18.975 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=128, tag_tid=20, entry_tid=15300' 2024-04-06T09:45:18.975 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=133, tag_tid=21, entry_tid=1' 2024-04-06T09:45:18.975 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:45:18.975 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:45:18.976 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=21 2024-04-06T09:45:18.976 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=128, tag_tid=20, entry_tid=15300' 2024-04-06T09:45:18.977 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:45:18.977 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:45:18.978 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=20 2024-04-06T09:45:18.978 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=133, tag_tid=21, entry_tid=1' 2024-04-06T09:45:18.978 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:45:18.978 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:45:18.979 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:45:18.980 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=128, tag_tid=20, entry_tid=15300' 2024-04-06T09:45:18.980 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:45:18.980 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:45:18.981 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=15300 2024-04-06T09:45:18.981 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 21 = 20 -a 1 -le 15300 2024-04-06T09:45:18.981 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:45:18.981 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:45:18.982 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:45:19.183 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:45:19.183 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:45:19.183 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:45:19.183 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:45:19.183 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:45:19.184 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:45:19.184 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:45:19.184 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:45:19.184 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:45:19.184 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:45:19.184 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:45:19.184 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:45:19.184 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:45:19.184 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:45:19.184 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:45:19.185 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:45:19.185 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:45:19.188 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:45:19.188 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:45:19.188 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:45:19.188 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:45:19.189 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:45:19.189 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:45:19.189 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:45:19.189 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:45:19.189 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:45:19.189 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:45:19.189 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:45:19.189 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:45:19.189 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:45:19.189 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:45:19.189 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:45:19.189 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:45:19.189 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:45:19.190 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:45:19.190 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:45:19.190 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:45:19.190 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:45:19.190 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:45:19.190 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:45:19.190 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:45:19.190 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:45:19.190 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:45:19.190 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:45:19.190 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:45:19.190 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:45:19.190 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:45:19.190 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:45:19.191 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:45:19.214 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:45:19.214 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:45:19.215 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:45:19.215 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:45:19.345 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:45:19.346 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:45:19.346 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:45:19.346 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:45:19.346 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:45:19.346 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:45:19.346 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:45:19.346 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:45:19.346 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:45:19.346 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:45:19.347 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:45:19.347 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:45:19.347 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:45:19.349 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:45:19.350 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:45:19.350 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:45:19.396 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 32 2024-04-06T09:45:19.396 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 33 2024-04-06T09:45:19.396 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:45:19.396 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=133, tag_tid=21, entry_tid=1], [object_number=132, tag_tid=21, entry_tid=0], [object_number=135, tag_tid=20, entry_tid=19939], [object_number=134, tag_tid=20, entry_tid=19938]]], state=connected] 2024-04-06T09:45:19.396 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=128, tag_tid=20, entry_tid=15380], [object_number=131, tag_tid=20, entry_tid=15379], [object_number=130, tag_tid=20, entry_tid=15378], [object_number=129, tag_tid=20, entry_tid=15377]]], state=connected] 2024-04-06T09:45:19.404 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:45:19.404 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=133, tag_tid=21, entry_tid=1' 2024-04-06T09:45:19.404 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:45:19.404 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:45:19.404 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:45:19.404 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:45:19.404 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:45:19.404 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:45:19.404 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:45:19.404 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:45:19.404 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:45:19.405 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:45:19.405 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:45:19.405 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:45:19.405 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:45:19.405 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:45:19.405 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:45:19.451 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 32 2024-04-06T09:45:19.451 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 33 2024-04-06T09:45:19.451 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:45:19.451 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=133, tag_tid=21, entry_tid=1], [object_number=132, tag_tid=21, entry_tid=0], [object_number=135, tag_tid=20, entry_tid=19939], [object_number=134, tag_tid=20, entry_tid=19938]]], state=connected] 2024-04-06T09:45:19.452 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=128, tag_tid=20, entry_tid=15380], [object_number=131, tag_tid=20, entry_tid=15379], [object_number=130, tag_tid=20, entry_tid=15378], [object_number=129, tag_tid=20, entry_tid=15377]]], state=connected] 2024-04-06T09:45:19.455 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:45:19.457 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=128, tag_tid=20, entry_tid=15380' 2024-04-06T09:45:19.457 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=133, tag_tid=21, entry_tid=1' -a 'object_number=133, tag_tid=21, entry_tid=1' = 'object_number=128, tag_tid=20, entry_tid=15380' 2024-04-06T09:45:19.457 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=128, tag_tid=20, entry_tid=15380' '!=' 'object_number=128, tag_tid=20, entry_tid=15300' 2024-04-06T09:45:19.457 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:45:19.457 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=128, tag_tid=20, entry_tid=15380' = 'object_number=128, tag_tid=20, entry_tid=15300' 2024-04-06T09:45:19.457 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=128, tag_tid=20, entry_tid=15380' 2024-04-06T09:45:19.457 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=133, tag_tid=21, entry_tid=1' 2024-04-06T09:45:19.457 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:45:19.457 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:45:19.458 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=21 2024-04-06T09:45:19.459 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=128, tag_tid=20, entry_tid=15380' 2024-04-06T09:45:19.459 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:45:19.459 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:45:19.460 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=20 2024-04-06T09:45:19.460 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=133, tag_tid=21, entry_tid=1' 2024-04-06T09:45:19.460 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:45:19.460 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:45:19.461 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:45:19.462 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=128, tag_tid=20, entry_tid=15380' 2024-04-06T09:45:19.462 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:45:19.462 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:45:19.463 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=15380 2024-04-06T09:45:19.463 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 21 = 20 -a 1 -le 15380 2024-04-06T09:45:19.463 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:45:19.464 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:45:19.464 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:45:19.665 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:45:19.665 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:45:19.666 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:45:19.666 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:45:19.666 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:45:19.666 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:45:19.666 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:45:19.666 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:45:19.666 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:45:19.666 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:45:19.666 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:45:19.666 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:45:19.666 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:45:19.666 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:45:19.666 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:45:19.667 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:45:19.667 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:45:19.669 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:45:19.669 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:45:19.669 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:45:19.669 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:45:19.670 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:45:19.670 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:45:19.670 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:45:19.670 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:45:19.670 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:45:19.670 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:45:19.670 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:45:19.670 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:45:19.670 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:45:19.670 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:45:19.670 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:45:19.670 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:45:19.672 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:45:19.672 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:45:19.672 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:45:19.672 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:45:19.672 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:45:19.672 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:45:19.672 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:45:19.672 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:45:19.672 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:45:19.672 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:45:19.672 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:45:19.673 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:45:19.673 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:45:19.673 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:45:19.673 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:45:19.673 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:45:19.696 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:45:19.697 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:45:19.697 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:45:19.697 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:45:19.832 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:45:19.833 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:45:19.833 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:45:19.833 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:45:19.834 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:45:19.834 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:45:19.834 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:45:19.834 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:45:19.834 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:45:19.834 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:45:19.834 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:45:19.834 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:45:19.835 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:45:19.835 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:45:19.836 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:45:19.836 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:45:19.883 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 32 2024-04-06T09:45:19.883 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 33 2024-04-06T09:45:19.883 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:45:19.884 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=133, tag_tid=21, entry_tid=1], [object_number=132, tag_tid=21, entry_tid=0], [object_number=135, tag_tid=20, entry_tid=19939], [object_number=134, tag_tid=20, entry_tid=19938]]], state=connected] 2024-04-06T09:45:19.884 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=130, tag_tid=20, entry_tid=15466], [object_number=129, tag_tid=20, entry_tid=15465], [object_number=128, tag_tid=20, entry_tid=15464], [object_number=131, tag_tid=20, entry_tid=15463]]], state=connected] 2024-04-06T09:45:19.887 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:45:19.888 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=133, tag_tid=21, entry_tid=1' 2024-04-06T09:45:19.889 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:45:19.889 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:45:19.889 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:45:19.889 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:45:19.889 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:45:19.889 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:45:19.890 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:45:19.890 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:45:19.890 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:45:19.890 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:45:19.890 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:45:19.890 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:45:19.890 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:45:19.891 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:45:19.891 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:45:19.938 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 32 2024-04-06T09:45:19.938 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 33 2024-04-06T09:45:19.938 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:45:19.938 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=133, tag_tid=21, entry_tid=1], [object_number=132, tag_tid=21, entry_tid=0], [object_number=135, tag_tid=20, entry_tid=19939], [object_number=134, tag_tid=20, entry_tid=19938]]], state=connected] 2024-04-06T09:45:19.938 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=130, tag_tid=20, entry_tid=15466], [object_number=129, tag_tid=20, entry_tid=15465], [object_number=128, tag_tid=20, entry_tid=15464], [object_number=131, tag_tid=20, entry_tid=15463]]], state=connected] 2024-04-06T09:45:19.942 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:45:19.944 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=130, tag_tid=20, entry_tid=15466' 2024-04-06T09:45:19.944 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=133, tag_tid=21, entry_tid=1' -a 'object_number=133, tag_tid=21, entry_tid=1' = 'object_number=130, tag_tid=20, entry_tid=15466' 2024-04-06T09:45:19.944 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=130, tag_tid=20, entry_tid=15466' '!=' 'object_number=128, tag_tid=20, entry_tid=15380' 2024-04-06T09:45:19.944 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:45:19.944 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=130, tag_tid=20, entry_tid=15466' = 'object_number=128, tag_tid=20, entry_tid=15380' 2024-04-06T09:45:19.944 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=130, tag_tid=20, entry_tid=15466' 2024-04-06T09:45:19.944 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=133, tag_tid=21, entry_tid=1' 2024-04-06T09:45:19.944 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:45:19.944 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:45:19.945 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=21 2024-04-06T09:45:19.945 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=130, tag_tid=20, entry_tid=15466' 2024-04-06T09:45:19.946 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:45:19.946 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:45:19.947 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=20 2024-04-06T09:45:19.947 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=133, tag_tid=21, entry_tid=1' 2024-04-06T09:45:19.947 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:45:19.947 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:45:19.948 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:45:19.949 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=130, tag_tid=20, entry_tid=15466' 2024-04-06T09:45:19.949 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:45:19.949 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:45:19.950 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=15466 2024-04-06T09:45:19.951 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 21 = 20 -a 1 -le 15466 2024-04-06T09:45:19.951 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:45:19.951 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:45:19.951 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:45:20.152 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:45:20.152 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:45:20.152 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:45:20.152 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:45:20.152 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:45:20.153 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:45:20.153 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:45:20.153 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:45:20.153 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:45:20.153 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:45:20.153 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:45:20.153 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:45:20.153 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:45:20.153 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:45:20.153 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:45:20.154 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:45:20.154 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:45:20.157 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:45:20.157 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:45:20.157 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:45:20.157 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:45:20.157 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:45:20.157 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:45:20.157 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:45:20.157 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:45:20.157 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:45:20.157 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:45:20.157 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:45:20.157 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:45:20.157 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:45:20.158 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:45:20.158 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:45:20.158 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:45:20.158 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:45:20.158 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:45:20.158 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:45:20.158 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:45:20.158 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:45:20.158 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:45:20.158 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:45:20.158 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:45:20.158 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:45:20.158 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:45:20.158 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:45:20.158 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:45:20.159 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:45:20.159 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:45:20.159 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:45:20.159 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:45:20.182 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:45:20.183 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:45:20.183 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:45:20.183 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:45:20.312 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:45:20.313 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:45:20.313 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:45:20.313 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:45:20.313 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:45:20.314 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:45:20.314 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:45:20.314 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:45:20.314 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:45:20.314 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:45:20.314 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:45:20.314 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:45:20.315 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:45:20.317 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:45:20.317 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:45:20.317 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:45:20.373 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 32 2024-04-06T09:45:20.373 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 33 2024-04-06T09:45:20.373 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:45:20.373 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=133, tag_tid=21, entry_tid=1], [object_number=132, tag_tid=21, entry_tid=0], [object_number=135, tag_tid=20, entry_tid=19939], [object_number=134, tag_tid=20, entry_tid=19938]]], state=connected] 2024-04-06T09:45:20.373 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=131, tag_tid=20, entry_tid=15555], [object_number=130, tag_tid=20, entry_tid=15554], [object_number=129, tag_tid=20, entry_tid=15553], [object_number=128, tag_tid=20, entry_tid=15552]]], state=connected] 2024-04-06T09:45:20.376 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:45:20.378 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=133, tag_tid=21, entry_tid=1' 2024-04-06T09:45:20.379 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:45:20.379 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:45:20.379 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:45:20.379 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:45:20.379 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:45:20.379 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:45:20.379 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:45:20.379 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:45:20.379 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:45:20.379 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:45:20.380 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:45:20.380 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:45:20.380 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:45:20.381 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:45:20.381 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:45:20.428 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 32 2024-04-06T09:45:20.428 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 33 2024-04-06T09:45:20.428 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:45:20.428 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=133, tag_tid=21, entry_tid=1], [object_number=132, tag_tid=21, entry_tid=0], [object_number=135, tag_tid=20, entry_tid=19939], [object_number=134, tag_tid=20, entry_tid=19938]]], state=connected] 2024-04-06T09:45:20.428 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=131, tag_tid=20, entry_tid=15555], [object_number=130, tag_tid=20, entry_tid=15554], [object_number=129, tag_tid=20, entry_tid=15553], [object_number=128, tag_tid=20, entry_tid=15552]]], state=connected] 2024-04-06T09:45:20.431 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:45:20.433 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=131, tag_tid=20, entry_tid=15555' 2024-04-06T09:45:20.434 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=133, tag_tid=21, entry_tid=1' -a 'object_number=133, tag_tid=21, entry_tid=1' = 'object_number=131, tag_tid=20, entry_tid=15555' 2024-04-06T09:45:20.434 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=131, tag_tid=20, entry_tid=15555' '!=' 'object_number=130, tag_tid=20, entry_tid=15466' 2024-04-06T09:45:20.434 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:45:20.434 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=131, tag_tid=20, entry_tid=15555' = 'object_number=130, tag_tid=20, entry_tid=15466' 2024-04-06T09:45:20.434 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=131, tag_tid=20, entry_tid=15555' 2024-04-06T09:45:20.434 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=133, tag_tid=21, entry_tid=1' 2024-04-06T09:45:20.434 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:45:20.434 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:45:20.434 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=21 2024-04-06T09:45:20.435 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=131, tag_tid=20, entry_tid=15555' 2024-04-06T09:45:20.435 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:45:20.435 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:45:20.437 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=20 2024-04-06T09:45:20.437 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=133, tag_tid=21, entry_tid=1' 2024-04-06T09:45:20.437 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:45:20.437 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:45:20.438 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:45:20.439 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=131, tag_tid=20, entry_tid=15555' 2024-04-06T09:45:20.439 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:45:20.439 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:45:20.440 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=15555 2024-04-06T09:45:20.440 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 21 = 20 -a 1 -le 15555 2024-04-06T09:45:20.441 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:45:20.441 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:45:20.441 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:45:20.642 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:45:20.642 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:45:20.642 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:45:20.643 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:45:20.643 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:45:20.643 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:45:20.643 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:45:20.643 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:45:20.643 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:45:20.643 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:45:20.643 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:45:20.643 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:45:20.643 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:45:20.643 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:45:20.643 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:45:20.644 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:45:20.644 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:45:20.647 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:45:20.647 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:45:20.647 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:45:20.647 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:45:20.647 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:45:20.647 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:45:20.647 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:45:20.648 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:45:20.648 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:45:20.648 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:45:20.648 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:45:20.648 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:45:20.648 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:45:20.648 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:45:20.648 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:45:20.648 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:45:20.649 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:45:20.649 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:45:20.649 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:45:20.649 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:45:20.649 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:45:20.649 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:45:20.649 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:45:20.649 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:45:20.649 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:45:20.650 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:45:20.650 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:45:20.650 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:45:20.650 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:45:20.650 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:45:20.650 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:45:20.650 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:45:20.674 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:45:20.674 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:45:20.674 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:45:20.675 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:45:20.804 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:45:20.805 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:45:20.805 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:45:20.805 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:45:20.805 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:45:20.805 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:45:20.805 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:45:20.806 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:45:20.806 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:45:20.806 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:45:20.806 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:45:20.806 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:45:20.807 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:45:20.809 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:45:20.809 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:45:20.809 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:45:20.861 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 32 2024-04-06T09:45:20.861 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 33 2024-04-06T09:45:20.861 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:45:20.861 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=133, tag_tid=21, entry_tid=1], [object_number=132, tag_tid=21, entry_tid=0], [object_number=135, tag_tid=20, entry_tid=19939], [object_number=134, tag_tid=20, entry_tid=19938]]], state=connected] 2024-04-06T09:45:20.861 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=131, tag_tid=20, entry_tid=15647], [object_number=130, tag_tid=20, entry_tid=15646], [object_number=129, tag_tid=20, entry_tid=15645], [object_number=128, tag_tid=20, entry_tid=15644]]], state=connected] 2024-04-06T09:45:20.864 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:45:20.866 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=133, tag_tid=21, entry_tid=1' 2024-04-06T09:45:20.867 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:45:20.867 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:45:20.867 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:45:20.867 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:45:20.867 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:45:20.867 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:45:20.867 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:45:20.867 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:45:20.867 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:45:20.867 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:45:20.867 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:45:20.868 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:45:20.868 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:45:20.869 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:45:20.869 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:45:20.915 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 32 2024-04-06T09:45:20.915 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 33 2024-04-06T09:45:20.915 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:45:20.915 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=133, tag_tid=21, entry_tid=1], [object_number=132, tag_tid=21, entry_tid=0], [object_number=135, tag_tid=20, entry_tid=19939], [object_number=134, tag_tid=20, entry_tid=19938]]], state=connected] 2024-04-06T09:45:20.916 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=131, tag_tid=20, entry_tid=15647], [object_number=130, tag_tid=20, entry_tid=15646], [object_number=129, tag_tid=20, entry_tid=15645], [object_number=128, tag_tid=20, entry_tid=15644]]], state=connected] 2024-04-06T09:45:20.919 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:45:20.920 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=131, tag_tid=20, entry_tid=15647' 2024-04-06T09:45:20.920 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=133, tag_tid=21, entry_tid=1' -a 'object_number=133, tag_tid=21, entry_tid=1' = 'object_number=131, tag_tid=20, entry_tid=15647' 2024-04-06T09:45:20.920 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=131, tag_tid=20, entry_tid=15647' '!=' 'object_number=131, tag_tid=20, entry_tid=15555' 2024-04-06T09:45:20.921 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:45:20.921 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=131, tag_tid=20, entry_tid=15647' = 'object_number=131, tag_tid=20, entry_tid=15555' 2024-04-06T09:45:20.921 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=131, tag_tid=20, entry_tid=15647' 2024-04-06T09:45:20.921 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=133, tag_tid=21, entry_tid=1' 2024-04-06T09:45:20.921 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:45:20.921 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:45:20.922 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=21 2024-04-06T09:45:20.922 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=131, tag_tid=20, entry_tid=15647' 2024-04-06T09:45:20.923 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:45:20.923 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:45:20.923 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=20 2024-04-06T09:45:20.924 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=133, tag_tid=21, entry_tid=1' 2024-04-06T09:45:20.924 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:45:20.924 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:45:20.925 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:45:20.926 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=131, tag_tid=20, entry_tid=15647' 2024-04-06T09:45:20.926 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:45:20.926 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:45:20.927 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=15647 2024-04-06T09:45:20.927 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 21 = 20 -a 1 -le 15647 2024-04-06T09:45:20.927 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:45:20.927 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:45:20.927 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:45:21.129 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:45:21.129 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:45:21.129 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:45:21.129 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:45:21.130 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:45:21.130 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:45:21.130 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:45:21.130 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:45:21.130 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:45:21.130 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:45:21.130 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:45:21.130 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:45:21.130 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:45:21.130 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:45:21.130 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:45:21.131 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:45:21.131 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:45:21.134 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:45:21.134 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:45:21.134 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:45:21.134 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:45:21.134 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:45:21.134 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:45:21.134 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:45:21.134 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:45:21.134 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:45:21.134 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:45:21.134 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:45:21.134 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:45:21.134 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:45:21.135 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:45:21.135 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:45:21.135 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:45:21.136 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:45:21.136 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:45:21.136 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:45:21.136 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:45:21.136 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:45:21.136 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:45:21.136 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:45:21.136 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:45:21.136 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:45:21.136 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:45:21.136 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:45:21.136 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:45:21.136 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:45:21.136 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:45:21.137 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:45:21.137 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:45:21.160 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:45:21.160 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:45:21.161 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:45:21.161 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:45:21.295 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:45:21.296 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:45:21.296 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:45:21.296 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:45:21.296 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:45:21.296 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:45:21.297 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:45:21.297 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:45:21.297 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:45:21.297 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:45:21.297 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:45:21.297 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:45:21.298 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:45:21.299 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:45:21.299 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:45:21.300 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:45:21.346 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 32 2024-04-06T09:45:21.347 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 33 2024-04-06T09:45:21.347 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:45:21.347 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=133, tag_tid=21, entry_tid=1], [object_number=132, tag_tid=21, entry_tid=0], [object_number=135, tag_tid=20, entry_tid=19939], [object_number=134, tag_tid=20, entry_tid=19938]]], state=connected] 2024-04-06T09:45:21.347 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=128, tag_tid=20, entry_tid=15736], [object_number=131, tag_tid=20, entry_tid=15735], [object_number=130, tag_tid=20, entry_tid=15734], [object_number=129, tag_tid=20, entry_tid=15733]]], state=connected] 2024-04-06T09:45:21.350 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:45:21.352 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=133, tag_tid=21, entry_tid=1' 2024-04-06T09:45:21.352 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:45:21.352 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:45:21.352 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:45:21.352 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:45:21.353 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:45:21.353 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:45:21.353 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:45:21.353 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:45:21.353 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:45:21.353 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:45:21.353 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:45:21.353 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:45:21.354 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:45:21.354 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:45:21.354 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:45:21.400 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 32 2024-04-06T09:45:21.400 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 33 2024-04-06T09:45:21.400 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:45:21.400 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=133, tag_tid=21, entry_tid=1], [object_number=132, tag_tid=21, entry_tid=0], [object_number=135, tag_tid=20, entry_tid=19939], [object_number=134, tag_tid=20, entry_tid=19938]]], state=connected] 2024-04-06T09:45:21.400 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=128, tag_tid=20, entry_tid=15736], [object_number=131, tag_tid=20, entry_tid=15735], [object_number=130, tag_tid=20, entry_tid=15734], [object_number=129, tag_tid=20, entry_tid=15733]]], state=connected] 2024-04-06T09:45:21.403 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:45:21.405 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=128, tag_tid=20, entry_tid=15736' 2024-04-06T09:45:21.406 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=133, tag_tid=21, entry_tid=1' -a 'object_number=133, tag_tid=21, entry_tid=1' = 'object_number=128, tag_tid=20, entry_tid=15736' 2024-04-06T09:45:21.406 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=128, tag_tid=20, entry_tid=15736' '!=' 'object_number=131, tag_tid=20, entry_tid=15647' 2024-04-06T09:45:21.406 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:45:21.406 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=128, tag_tid=20, entry_tid=15736' = 'object_number=131, tag_tid=20, entry_tid=15647' 2024-04-06T09:45:21.406 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=128, tag_tid=20, entry_tid=15736' 2024-04-06T09:45:21.406 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=133, tag_tid=21, entry_tid=1' 2024-04-06T09:45:21.406 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:45:21.406 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:45:21.407 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=21 2024-04-06T09:45:21.407 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=128, tag_tid=20, entry_tid=15736' 2024-04-06T09:45:21.407 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:45:21.407 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:45:21.408 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=20 2024-04-06T09:45:21.409 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=133, tag_tid=21, entry_tid=1' 2024-04-06T09:45:21.409 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:45:21.409 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:45:21.410 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:45:21.411 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=128, tag_tid=20, entry_tid=15736' 2024-04-06T09:45:21.411 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:45:21.411 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:45:21.412 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=15736 2024-04-06T09:45:21.412 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 21 = 20 -a 1 -le 15736 2024-04-06T09:45:21.412 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:45:21.412 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:45:21.412 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:45:21.613 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:45:21.614 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:45:21.614 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:45:21.614 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:45:21.614 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:45:21.614 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:45:21.614 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:45:21.614 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:45:21.614 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:45:21.614 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:45:21.614 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:45:21.614 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:45:21.614 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:45:21.615 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:45:21.615 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:45:21.615 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:45:21.615 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:45:21.618 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:45:21.618 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:45:21.618 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:45:21.618 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:45:21.618 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:45:21.618 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:45:21.618 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:45:21.618 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:45:21.618 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:45:21.618 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:45:21.618 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:45:21.618 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:45:21.619 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:45:21.619 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:45:21.619 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:45:21.619 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:45:21.620 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:45:21.620 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:45:21.621 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:45:21.621 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:45:21.621 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:45:21.621 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:45:21.621 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:45:21.621 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:45:21.621 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:45:21.621 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:45:21.621 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:45:21.621 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:45:21.621 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:45:21.621 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:45:21.621 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:45:21.621 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:45:21.644 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:45:21.645 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:45:21.645 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:45:21.645 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:45:21.776 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:45:21.777 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:45:21.777 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:45:21.778 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:45:21.778 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:45:21.778 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:45:21.778 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:45:21.778 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:45:21.778 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:45:21.778 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:45:21.778 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:45:21.779 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:45:21.779 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:45:21.779 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:45:21.780 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:45:21.780 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:45:21.826 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 32 2024-04-06T09:45:21.827 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 33 2024-04-06T09:45:21.827 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:45:21.827 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=133, tag_tid=21, entry_tid=1], [object_number=132, tag_tid=21, entry_tid=0], [object_number=135, tag_tid=20, entry_tid=19939], [object_number=134, tag_tid=20, entry_tid=19938]]], state=connected] 2024-04-06T09:45:21.827 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=128, tag_tid=20, entry_tid=15828], [object_number=131, tag_tid=20, entry_tid=15827], [object_number=130, tag_tid=20, entry_tid=15826], [object_number=129, tag_tid=20, entry_tid=15825]]], state=connected] 2024-04-06T09:45:21.830 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:45:21.832 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=133, tag_tid=21, entry_tid=1' 2024-04-06T09:45:21.833 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:45:21.833 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:45:21.833 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:45:21.833 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:45:21.833 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:45:21.833 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:45:21.833 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:45:21.833 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:45:21.833 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:45:21.833 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:45:21.834 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:45:21.834 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:45:21.834 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:45:21.834 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:45:21.834 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:45:21.882 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 32 2024-04-06T09:45:21.883 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 33 2024-04-06T09:45:21.883 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:45:21.883 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=133, tag_tid=21, entry_tid=1], [object_number=132, tag_tid=21, entry_tid=0], [object_number=135, tag_tid=20, entry_tid=19939], [object_number=134, tag_tid=20, entry_tid=19938]]], state=connected] 2024-04-06T09:45:21.883 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=128, tag_tid=20, entry_tid=15828], [object_number=131, tag_tid=20, entry_tid=15827], [object_number=130, tag_tid=20, entry_tid=15826], [object_number=129, tag_tid=20, entry_tid=15825]]], state=connected] 2024-04-06T09:45:21.886 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:45:21.888 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=128, tag_tid=20, entry_tid=15828' 2024-04-06T09:45:21.888 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=133, tag_tid=21, entry_tid=1' -a 'object_number=133, tag_tid=21, entry_tid=1' = 'object_number=128, tag_tid=20, entry_tid=15828' 2024-04-06T09:45:21.888 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=128, tag_tid=20, entry_tid=15828' '!=' 'object_number=128, tag_tid=20, entry_tid=15736' 2024-04-06T09:45:21.888 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:45:21.888 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=128, tag_tid=20, entry_tid=15828' = 'object_number=128, tag_tid=20, entry_tid=15736' 2024-04-06T09:45:21.888 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=128, tag_tid=20, entry_tid=15828' 2024-04-06T09:45:21.888 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=133, tag_tid=21, entry_tid=1' 2024-04-06T09:45:21.889 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:45:21.889 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:45:21.889 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=21 2024-04-06T09:45:21.890 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=128, tag_tid=20, entry_tid=15828' 2024-04-06T09:45:21.890 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:45:21.890 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:45:21.891 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=20 2024-04-06T09:45:21.892 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=133, tag_tid=21, entry_tid=1' 2024-04-06T09:45:21.892 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:45:21.892 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:45:21.892 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:45:21.895 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=128, tag_tid=20, entry_tid=15828' 2024-04-06T09:45:21.895 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:45:21.895 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:45:21.895 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=15828 2024-04-06T09:45:21.895 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 21 = 20 -a 1 -le 15828 2024-04-06T09:45:21.895 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:45:21.895 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:45:21.895 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:45:22.096 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:45:22.096 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:45:22.096 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:45:22.096 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:45:22.097 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:45:22.097 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:45:22.097 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:45:22.097 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:45:22.097 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:45:22.097 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:45:22.097 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:45:22.097 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:45:22.097 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:45:22.097 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:45:22.097 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:45:22.098 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:45:22.098 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:45:22.100 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:45:22.100 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:45:22.100 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:45:22.100 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:45:22.100 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:45:22.100 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:45:22.100 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:45:22.100 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:45:22.100 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:45:22.101 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:45:22.101 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:45:22.101 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:45:22.101 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:45:22.101 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:45:22.101 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:45:22.101 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:45:22.102 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:45:22.102 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:45:22.102 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:45:22.102 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:45:22.102 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:45:22.103 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:45:22.103 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:45:22.103 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:45:22.103 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:45:22.103 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:45:22.103 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:45:22.103 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:45:22.103 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:45:22.103 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:45:22.103 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:45:22.103 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:45:22.126 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:45:22.127 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:45:22.127 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:45:22.127 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:45:22.261 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:45:22.263 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:45:22.263 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:45:22.263 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:45:22.263 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:45:22.263 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:45:22.263 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:45:22.263 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:45:22.263 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:45:22.263 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:45:22.263 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:45:22.264 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:45:22.264 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:45:22.265 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:45:22.266 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:45:22.266 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:45:22.318 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 32 2024-04-06T09:45:22.318 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 33 2024-04-06T09:45:22.319 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:45:22.319 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=133, tag_tid=21, entry_tid=1], [object_number=132, tag_tid=21, entry_tid=0], [object_number=135, tag_tid=20, entry_tid=19939], [object_number=134, tag_tid=20, entry_tid=19938]]], state=connected] 2024-04-06T09:45:22.319 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=131, tag_tid=20, entry_tid=15911], [object_number=130, tag_tid=20, entry_tid=15910], [object_number=129, tag_tid=20, entry_tid=15909], [object_number=128, tag_tid=20, entry_tid=15908]]], state=connected] 2024-04-06T09:45:22.322 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:45:22.324 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=133, tag_tid=21, entry_tid=1' 2024-04-06T09:45:22.325 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:45:22.325 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:45:22.325 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:45:22.325 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:45:22.325 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:45:22.325 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:45:22.326 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:45:22.326 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:45:22.326 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:45:22.326 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:45:22.326 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:45:22.326 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:45:22.327 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:45:22.327 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:45:22.327 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:45:22.580 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 32 2024-04-06T09:45:22.580 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 33 2024-04-06T09:45:22.580 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:45:22.581 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=133, tag_tid=21, entry_tid=1], [object_number=132, tag_tid=21, entry_tid=0], [object_number=135, tag_tid=20, entry_tid=19939], [object_number=134, tag_tid=20, entry_tid=19938]]], state=connected] 2024-04-06T09:45:22.581 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=131, tag_tid=20, entry_tid=15911], [object_number=130, tag_tid=20, entry_tid=15910], [object_number=129, tag_tid=20, entry_tid=15909], [object_number=128, tag_tid=20, entry_tid=15908]]], state=connected] 2024-04-06T09:45:22.585 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:45:22.588 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=131, tag_tid=20, entry_tid=15911' 2024-04-06T09:45:22.588 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=133, tag_tid=21, entry_tid=1' -a 'object_number=133, tag_tid=21, entry_tid=1' = 'object_number=131, tag_tid=20, entry_tid=15911' 2024-04-06T09:45:22.588 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=131, tag_tid=20, entry_tid=15911' '!=' 'object_number=128, tag_tid=20, entry_tid=15828' 2024-04-06T09:45:22.588 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:45:22.588 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=131, tag_tid=20, entry_tid=15911' = 'object_number=128, tag_tid=20, entry_tid=15828' 2024-04-06T09:45:22.588 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=131, tag_tid=20, entry_tid=15911' 2024-04-06T09:45:22.589 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=133, tag_tid=21, entry_tid=1' 2024-04-06T09:45:22.589 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:45:22.589 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:45:22.591 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=21 2024-04-06T09:45:22.592 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=131, tag_tid=20, entry_tid=15911' 2024-04-06T09:45:22.592 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:45:22.592 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:45:22.594 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=20 2024-04-06T09:45:22.595 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=133, tag_tid=21, entry_tid=1' 2024-04-06T09:45:22.595 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:45:22.595 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:45:22.596 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:45:22.597 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=131, tag_tid=20, entry_tid=15911' 2024-04-06T09:45:22.597 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:45:22.597 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:45:22.599 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=15911 2024-04-06T09:45:22.599 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 21 = 20 -a 1 -le 15911 2024-04-06T09:45:22.599 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:45:22.599 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:45:22.599 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:45:22.801 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:45:22.801 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:45:22.801 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:45:22.801 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:45:22.801 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:45:22.802 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:45:22.802 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:45:22.802 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:45:22.802 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:45:22.802 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:45:22.802 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:45:22.802 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:45:22.802 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:45:22.802 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:45:22.802 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:45:22.803 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:45:22.803 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:45:22.805 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:45:22.805 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:45:22.805 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:45:22.805 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:45:22.806 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:45:22.806 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:45:22.806 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:45:22.806 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:45:22.806 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:45:22.806 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:45:22.806 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:45:22.806 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:45:22.806 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:45:22.806 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:45:22.806 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:45:22.806 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:45:22.808 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:45:22.808 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:45:22.808 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:45:22.808 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:45:22.808 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:45:22.808 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:45:22.808 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:45:22.808 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:45:22.808 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:45:22.808 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:45:22.809 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:45:22.809 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:45:22.809 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:45:22.809 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:45:22.809 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:45:22.809 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:45:22.832 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:45:22.832 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:45:22.832 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:45:22.832 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:45:22.974 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:45:22.975 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:45:22.975 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:45:22.975 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:45:22.975 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:45:22.975 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:45:22.975 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:45:22.975 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:45:22.975 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:45:22.975 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:45:22.976 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:45:22.976 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:45:22.976 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:45:22.978 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:45:22.978 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:45:22.978 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:45:23.031 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 32 2024-04-06T09:45:23.031 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 33 2024-04-06T09:45:23.031 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:45:23.031 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=133, tag_tid=21, entry_tid=1], [object_number=132, tag_tid=21, entry_tid=0], [object_number=135, tag_tid=20, entry_tid=19939], [object_number=134, tag_tid=20, entry_tid=19938]]], state=connected] 2024-04-06T09:45:23.031 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=130, tag_tid=20, entry_tid=16030], [object_number=129, tag_tid=20, entry_tid=16029], [object_number=128, tag_tid=20, entry_tid=16028], [object_number=131, tag_tid=20, entry_tid=16027]]], state=connected] 2024-04-06T09:45:23.034 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:45:23.036 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=133, tag_tid=21, entry_tid=1' 2024-04-06T09:45:23.037 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:45:23.037 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:45:23.037 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:45:23.037 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:45:23.037 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:45:23.037 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:45:23.037 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:45:23.037 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:45:23.037 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:45:23.038 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:45:23.038 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:45:23.038 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:45:23.039 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:45:23.039 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:45:23.039 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:45:23.085 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 32 2024-04-06T09:45:23.086 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 33 2024-04-06T09:45:23.086 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:45:23.086 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=133, tag_tid=21, entry_tid=1], [object_number=132, tag_tid=21, entry_tid=0], [object_number=135, tag_tid=20, entry_tid=19939], [object_number=134, tag_tid=20, entry_tid=19938]]], state=connected] 2024-04-06T09:45:23.086 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=130, tag_tid=20, entry_tid=16030], [object_number=129, tag_tid=20, entry_tid=16029], [object_number=128, tag_tid=20, entry_tid=16028], [object_number=131, tag_tid=20, entry_tid=16027]]], state=connected] 2024-04-06T09:45:23.089 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:45:23.091 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=130, tag_tid=20, entry_tid=16030' 2024-04-06T09:45:23.091 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=133, tag_tid=21, entry_tid=1' -a 'object_number=133, tag_tid=21, entry_tid=1' = 'object_number=130, tag_tid=20, entry_tid=16030' 2024-04-06T09:45:23.091 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=130, tag_tid=20, entry_tid=16030' '!=' 'object_number=131, tag_tid=20, entry_tid=15911' 2024-04-06T09:45:23.091 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:45:23.091 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=130, tag_tid=20, entry_tid=16030' = 'object_number=131, tag_tid=20, entry_tid=15911' 2024-04-06T09:45:23.091 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=130, tag_tid=20, entry_tid=16030' 2024-04-06T09:45:23.092 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=133, tag_tid=21, entry_tid=1' 2024-04-06T09:45:23.092 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:45:23.092 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:45:23.092 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=21 2024-04-06T09:45:23.093 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=130, tag_tid=20, entry_tid=16030' 2024-04-06T09:45:23.093 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:45:23.093 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:45:23.094 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=20 2024-04-06T09:45:23.095 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=133, tag_tid=21, entry_tid=1' 2024-04-06T09:45:23.095 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:45:23.095 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:45:23.095 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:45:23.096 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=130, tag_tid=20, entry_tid=16030' 2024-04-06T09:45:23.096 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:45:23.096 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:45:23.097 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=16030 2024-04-06T09:45:23.098 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 21 = 20 -a 1 -le 16030 2024-04-06T09:45:23.098 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:45:23.098 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:45:23.098 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:45:23.299 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:45:23.299 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:45:23.299 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:45:23.299 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:45:23.300 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:45:23.300 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:45:23.300 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:45:23.300 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:45:23.300 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:45:23.300 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:45:23.300 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:45:23.300 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:45:23.300 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:45:23.300 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:45:23.300 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:45:23.301 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:45:23.301 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:45:23.303 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:45:23.303 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:45:23.303 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:45:23.303 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:45:23.303 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:45:23.303 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:45:23.303 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:45:23.303 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:45:23.303 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:45:23.304 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:45:23.304 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:45:23.304 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:45:23.304 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:45:23.304 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:45:23.304 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:45:23.304 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:45:23.305 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:45:23.305 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:45:23.305 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:45:23.305 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:45:23.306 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:45:23.306 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:45:23.306 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:45:23.306 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:45:23.306 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:45:23.306 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:45:23.306 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:45:23.306 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:45:23.306 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:45:23.306 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:45:23.306 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:45:23.306 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:45:23.332 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:45:23.332 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:45:23.332 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:45:23.332 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:45:23.463 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:45:23.464 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:45:23.464 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:45:23.465 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:45:23.465 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:45:23.465 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:45:23.465 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:45:23.465 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:45:23.465 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:45:23.465 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:45:23.466 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:45:23.466 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:45:23.466 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:45:23.467 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:45:23.467 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:45:23.468 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:45:23.515 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 32 2024-04-06T09:45:23.515 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 33 2024-04-06T09:45:23.515 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:45:23.515 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=133, tag_tid=21, entry_tid=1], [object_number=132, tag_tid=21, entry_tid=0], [object_number=135, tag_tid=20, entry_tid=19939], [object_number=134, tag_tid=20, entry_tid=19938]]], state=connected] 2024-04-06T09:45:23.515 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=129, tag_tid=20, entry_tid=16113], [object_number=128, tag_tid=20, entry_tid=16112], [object_number=131, tag_tid=20, entry_tid=16111], [object_number=130, tag_tid=20, entry_tid=16110]]], state=connected] 2024-04-06T09:45:23.519 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:45:23.520 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=133, tag_tid=21, entry_tid=1' 2024-04-06T09:45:23.521 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:45:23.521 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:45:23.521 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:45:23.521 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:45:23.521 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:45:23.521 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:45:23.522 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:45:23.522 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:45:23.522 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:45:23.522 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:45:23.522 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:45:23.522 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:45:23.523 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:45:23.523 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:45:23.523 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:45:23.569 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 32 2024-04-06T09:45:23.569 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 33 2024-04-06T09:45:23.570 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:45:23.570 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=133, tag_tid=21, entry_tid=1], [object_number=132, tag_tid=21, entry_tid=0], [object_number=135, tag_tid=20, entry_tid=19939], [object_number=134, tag_tid=20, entry_tid=19938]]], state=connected] 2024-04-06T09:45:23.570 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=129, tag_tid=20, entry_tid=16113], [object_number=128, tag_tid=20, entry_tid=16112], [object_number=131, tag_tid=20, entry_tid=16111], [object_number=130, tag_tid=20, entry_tid=16110]]], state=connected] 2024-04-06T09:45:23.573 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:45:23.575 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=129, tag_tid=20, entry_tid=16113' 2024-04-06T09:45:23.575 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=133, tag_tid=21, entry_tid=1' -a 'object_number=133, tag_tid=21, entry_tid=1' = 'object_number=129, tag_tid=20, entry_tid=16113' 2024-04-06T09:45:23.575 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=129, tag_tid=20, entry_tid=16113' '!=' 'object_number=130, tag_tid=20, entry_tid=16030' 2024-04-06T09:45:23.575 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:45:23.575 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=129, tag_tid=20, entry_tid=16113' = 'object_number=130, tag_tid=20, entry_tid=16030' 2024-04-06T09:45:23.575 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=129, tag_tid=20, entry_tid=16113' 2024-04-06T09:45:23.576 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=133, tag_tid=21, entry_tid=1' 2024-04-06T09:45:23.576 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:45:23.576 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:45:23.576 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=21 2024-04-06T09:45:23.577 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=129, tag_tid=20, entry_tid=16113' 2024-04-06T09:45:23.577 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:45:23.577 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:45:23.578 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=20 2024-04-06T09:45:23.578 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=133, tag_tid=21, entry_tid=1' 2024-04-06T09:45:23.578 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:45:23.579 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:45:23.579 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:45:23.580 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=129, tag_tid=20, entry_tid=16113' 2024-04-06T09:45:23.580 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:45:23.580 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:45:23.581 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=16113 2024-04-06T09:45:23.581 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 21 = 20 -a 1 -le 16113 2024-04-06T09:45:23.581 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:45:23.582 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:45:23.582 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:45:23.783 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:45:23.783 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:45:23.783 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:45:23.783 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:45:23.783 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:45:23.783 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:45:23.784 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:45:23.784 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:45:23.784 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:45:23.784 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:45:23.784 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:45:23.784 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:45:23.784 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:45:23.784 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:45:23.784 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:45:23.784 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:45:23.785 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:45:23.789 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:45:23.789 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:45:23.789 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:45:23.789 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:45:23.789 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:45:23.789 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:45:23.789 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:45:23.790 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:45:23.790 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:45:23.790 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:45:23.790 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:45:23.790 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:45:23.790 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:45:23.790 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:45:23.790 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:45:23.790 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:45:23.790 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:45:23.790 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:45:23.790 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:45:23.790 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:45:23.790 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:45:23.791 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:45:23.791 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:45:23.791 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:45:23.791 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:45:23.791 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:45:23.791 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:45:23.791 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:45:23.791 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:45:23.791 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:45:23.791 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:45:23.791 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:45:23.814 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:45:23.815 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:45:23.815 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:45:23.815 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:45:23.950 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:45:23.950 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:45:23.951 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:45:23.951 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:45:23.951 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:45:23.951 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:45:23.951 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:45:23.951 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:45:23.951 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:45:23.951 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:45:23.952 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:45:23.952 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:45:23.952 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:45:23.954 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:45:23.955 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:45:23.955 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:45:24.013 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 32 2024-04-06T09:45:24.013 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 33 2024-04-06T09:45:24.013 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:45:24.013 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=133, tag_tid=21, entry_tid=1], [object_number=132, tag_tid=21, entry_tid=0], [object_number=135, tag_tid=20, entry_tid=19939], [object_number=134, tag_tid=20, entry_tid=19938]]], state=connected] 2024-04-06T09:45:24.013 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=128, tag_tid=20, entry_tid=16200], [object_number=131, tag_tid=20, entry_tid=16199], [object_number=130, tag_tid=20, entry_tid=16198], [object_number=129, tag_tid=20, entry_tid=16197]]], state=connected] 2024-04-06T09:45:24.017 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:45:24.018 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=133, tag_tid=21, entry_tid=1' 2024-04-06T09:45:24.019 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:45:24.019 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:45:24.019 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:45:24.019 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:45:24.019 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:45:24.019 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:45:24.020 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:45:24.020 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:45:24.020 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:45:24.020 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:45:24.020 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:45:24.020 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:45:24.021 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:45:24.021 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:45:24.021 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:45:24.068 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 32 2024-04-06T09:45:24.068 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 33 2024-04-06T09:45:24.069 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:45:24.069 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=133, tag_tid=21, entry_tid=1], [object_number=132, tag_tid=21, entry_tid=0], [object_number=135, tag_tid=20, entry_tid=19939], [object_number=134, tag_tid=20, entry_tid=19938]]], state=connected] 2024-04-06T09:45:24.069 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=128, tag_tid=20, entry_tid=16200], [object_number=131, tag_tid=20, entry_tid=16199], [object_number=130, tag_tid=20, entry_tid=16198], [object_number=129, tag_tid=20, entry_tid=16197]]], state=connected] 2024-04-06T09:45:24.073 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:45:24.074 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=128, tag_tid=20, entry_tid=16200' 2024-04-06T09:45:24.074 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=133, tag_tid=21, entry_tid=1' -a 'object_number=133, tag_tid=21, entry_tid=1' = 'object_number=128, tag_tid=20, entry_tid=16200' 2024-04-06T09:45:24.074 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=128, tag_tid=20, entry_tid=16200' '!=' 'object_number=129, tag_tid=20, entry_tid=16113' 2024-04-06T09:45:24.074 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:45:24.074 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=128, tag_tid=20, entry_tid=16200' = 'object_number=129, tag_tid=20, entry_tid=16113' 2024-04-06T09:45:24.074 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=128, tag_tid=20, entry_tid=16200' 2024-04-06T09:45:24.075 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=133, tag_tid=21, entry_tid=1' 2024-04-06T09:45:24.075 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:45:24.075 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:45:24.075 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=21 2024-04-06T09:45:24.076 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=128, tag_tid=20, entry_tid=16200' 2024-04-06T09:45:24.076 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:45:24.076 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:45:24.077 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=20 2024-04-06T09:45:24.078 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=133, tag_tid=21, entry_tid=1' 2024-04-06T09:45:24.078 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:45:24.078 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:45:24.079 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:45:24.080 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=128, tag_tid=20, entry_tid=16200' 2024-04-06T09:45:24.080 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:45:24.080 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:45:24.081 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=16200 2024-04-06T09:45:24.081 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 21 = 20 -a 1 -le 16200 2024-04-06T09:45:24.081 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:45:24.081 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:45:24.081 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:45:24.283 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:45:24.283 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:45:24.283 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:45:24.283 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:45:24.283 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:45:24.283 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:45:24.283 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:45:24.284 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:45:24.284 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:45:24.284 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:45:24.284 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:45:24.284 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:45:24.284 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:45:24.284 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:45:24.284 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:45:24.284 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:45:24.285 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:45:24.287 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:45:24.287 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:45:24.287 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:45:24.287 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:45:24.287 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:45:24.287 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:45:24.287 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:45:24.287 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:45:24.288 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:45:24.288 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:45:24.288 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:45:24.288 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:45:24.288 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:45:24.288 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:45:24.288 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:45:24.288 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:45:24.289 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:45:24.289 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:45:24.289 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:45:24.289 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:45:24.289 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:45:24.289 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:45:24.289 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:45:24.289 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:45:24.289 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:45:24.289 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:45:24.290 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:45:24.290 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:45:24.290 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:45:24.290 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:45:24.290 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:45:24.290 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:45:24.314 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:45:24.315 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:45:24.315 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:45:24.315 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:45:24.445 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:45:24.446 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:45:24.446 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:45:24.446 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:45:24.446 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:45:24.446 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:45:24.446 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:45:24.447 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:45:24.447 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:45:24.447 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:45:24.447 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:45:24.447 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:45:24.447 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:45:24.449 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:45:24.449 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:45:24.450 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:45:24.497 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 32 2024-04-06T09:45:24.497 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 33 2024-04-06T09:45:24.498 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:45:24.498 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=133, tag_tid=21, entry_tid=1], [object_number=132, tag_tid=21, entry_tid=0], [object_number=135, tag_tid=20, entry_tid=19939], [object_number=134, tag_tid=20, entry_tid=19938]]], state=connected] 2024-04-06T09:45:24.498 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=131, tag_tid=20, entry_tid=16287], [object_number=130, tag_tid=20, entry_tid=16286], [object_number=129, tag_tid=20, entry_tid=16285], [object_number=128, tag_tid=20, entry_tid=16284]]], state=connected] 2024-04-06T09:45:24.501 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:45:24.502 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=133, tag_tid=21, entry_tid=1' 2024-04-06T09:45:24.503 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:45:24.503 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:45:24.503 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:45:24.503 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:45:24.503 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:45:24.504 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:45:24.504 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:45:24.504 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:45:24.504 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:45:24.504 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:45:24.504 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:45:24.504 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:45:24.505 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:45:24.505 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:45:24.505 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:45:24.551 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 32 2024-04-06T09:45:24.551 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 33 2024-04-06T09:45:24.551 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:45:24.551 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=133, tag_tid=21, entry_tid=1], [object_number=132, tag_tid=21, entry_tid=0], [object_number=135, tag_tid=20, entry_tid=19939], [object_number=134, tag_tid=20, entry_tid=19938]]], state=connected] 2024-04-06T09:45:24.551 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=131, tag_tid=20, entry_tid=16287], [object_number=130, tag_tid=20, entry_tid=16286], [object_number=129, tag_tid=20, entry_tid=16285], [object_number=128, tag_tid=20, entry_tid=16284]]], state=connected] 2024-04-06T09:45:24.555 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:45:24.557 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=131, tag_tid=20, entry_tid=16287' 2024-04-06T09:45:24.557 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=133, tag_tid=21, entry_tid=1' -a 'object_number=133, tag_tid=21, entry_tid=1' = 'object_number=131, tag_tid=20, entry_tid=16287' 2024-04-06T09:45:24.557 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=131, tag_tid=20, entry_tid=16287' '!=' 'object_number=128, tag_tid=20, entry_tid=16200' 2024-04-06T09:45:24.557 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:45:24.557 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=131, tag_tid=20, entry_tid=16287' = 'object_number=128, tag_tid=20, entry_tid=16200' 2024-04-06T09:45:24.557 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=131, tag_tid=20, entry_tid=16287' 2024-04-06T09:45:24.557 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=133, tag_tid=21, entry_tid=1' 2024-04-06T09:45:24.558 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:45:24.558 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:45:24.559 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=21 2024-04-06T09:45:24.559 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=131, tag_tid=20, entry_tid=16287' 2024-04-06T09:45:24.559 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:45:24.559 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:45:24.559 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=20 2024-04-06T09:45:24.560 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=133, tag_tid=21, entry_tid=1' 2024-04-06T09:45:24.561 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:45:24.561 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:45:24.561 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:45:24.562 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=131, tag_tid=20, entry_tid=16287' 2024-04-06T09:45:24.562 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:45:24.562 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:45:24.563 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=16287 2024-04-06T09:45:24.563 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 21 = 20 -a 1 -le 16287 2024-04-06T09:45:24.563 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:45:24.564 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:45:24.564 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:45:24.766 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:45:24.766 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:45:24.766 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:45:24.766 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:45:24.766 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:45:24.766 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:45:24.766 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:45:24.766 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:45:24.766 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:45:24.766 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:45:24.766 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:45:24.766 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:45:24.767 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:45:24.767 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:45:24.767 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:45:24.767 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:45:24.767 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:45:24.771 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:45:24.771 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:45:24.771 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:45:24.771 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:45:24.771 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:45:24.771 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:45:24.771 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:45:24.771 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:45:24.771 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:45:24.772 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:45:24.772 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:45:24.772 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:45:24.772 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:45:24.772 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:45:24.772 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:45:24.772 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:45:24.772 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:45:24.772 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:45:24.772 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:45:24.772 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:45:24.772 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:45:24.772 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:45:24.772 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:45:24.772 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:45:24.773 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:45:24.773 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:45:24.773 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:45:24.773 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:45:24.773 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:45:24.773 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:45:24.773 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:45:24.773 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:45:24.798 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:45:24.798 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:45:24.798 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:45:24.798 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:45:24.927 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:45:24.928 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:45:24.928 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:45:24.928 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:45:24.928 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:45:24.928 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:45:24.929 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:45:24.929 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:45:24.929 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:45:24.929 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:45:24.929 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:45:24.929 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:45:24.930 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:45:24.931 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:45:24.931 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:45:24.931 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:45:24.977 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 32 2024-04-06T09:45:24.978 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 33 2024-04-06T09:45:24.978 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:45:24.978 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=133, tag_tid=21, entry_tid=1], [object_number=132, tag_tid=21, entry_tid=0], [object_number=135, tag_tid=20, entry_tid=19939], [object_number=134, tag_tid=20, entry_tid=19938]]], state=connected] 2024-04-06T09:45:24.978 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=130, tag_tid=20, entry_tid=16374], [object_number=129, tag_tid=20, entry_tid=16373], [object_number=128, tag_tid=20, entry_tid=16372], [object_number=131, tag_tid=20, entry_tid=16371]]], state=connected] 2024-04-06T09:45:24.981 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:45:24.982 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=133, tag_tid=21, entry_tid=1' 2024-04-06T09:45:24.983 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:45:24.983 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:45:24.983 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:45:24.983 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:45:24.983 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:45:24.983 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:45:24.983 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:45:24.984 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:45:24.984 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:45:24.984 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:45:24.984 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:45:24.984 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:45:24.984 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:45:24.985 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:45:24.985 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:45:25.032 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 32 2024-04-06T09:45:25.032 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 33 2024-04-06T09:45:25.032 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:45:25.032 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=133, tag_tid=21, entry_tid=1], [object_number=132, tag_tid=21, entry_tid=0], [object_number=135, tag_tid=20, entry_tid=19939], [object_number=134, tag_tid=20, entry_tid=19938]]], state=connected] 2024-04-06T09:45:25.032 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=130, tag_tid=20, entry_tid=16374], [object_number=129, tag_tid=20, entry_tid=16373], [object_number=128, tag_tid=20, entry_tid=16372], [object_number=131, tag_tid=20, entry_tid=16371]]], state=connected] 2024-04-06T09:45:25.035 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:45:25.038 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=130, tag_tid=20, entry_tid=16374' 2024-04-06T09:45:25.038 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=133, tag_tid=21, entry_tid=1' -a 'object_number=133, tag_tid=21, entry_tid=1' = 'object_number=130, tag_tid=20, entry_tid=16374' 2024-04-06T09:45:25.038 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=130, tag_tid=20, entry_tid=16374' '!=' 'object_number=131, tag_tid=20, entry_tid=16287' 2024-04-06T09:45:25.038 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:45:25.038 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=130, tag_tid=20, entry_tid=16374' = 'object_number=131, tag_tid=20, entry_tid=16287' 2024-04-06T09:45:25.038 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=130, tag_tid=20, entry_tid=16374' 2024-04-06T09:45:25.038 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=133, tag_tid=21, entry_tid=1' 2024-04-06T09:45:25.038 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:45:25.038 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:45:25.039 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=21 2024-04-06T09:45:25.039 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=130, tag_tid=20, entry_tid=16374' 2024-04-06T09:45:25.040 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:45:25.040 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:45:25.040 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=20 2024-04-06T09:45:25.041 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=133, tag_tid=21, entry_tid=1' 2024-04-06T09:45:25.041 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:45:25.041 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:45:25.042 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:45:25.043 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=130, tag_tid=20, entry_tid=16374' 2024-04-06T09:45:25.043 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:45:25.043 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:45:25.044 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=16374 2024-04-06T09:45:25.044 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 21 = 20 -a 1 -le 16374 2024-04-06T09:45:25.044 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:45:25.044 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:45:25.044 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:45:25.246 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:45:25.246 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:45:25.246 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:45:25.246 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:45:25.246 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:45:25.246 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:45:25.246 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:45:25.246 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:45:25.246 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:45:25.246 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:45:25.247 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:45:25.247 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:45:25.247 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:45:25.247 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:45:25.247 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:45:25.247 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:45:25.247 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:45:25.249 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:45:25.249 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:45:25.249 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:45:25.249 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:45:25.250 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:45:25.250 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:45:25.250 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:45:25.250 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:45:25.250 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:45:25.250 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:45:25.250 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:45:25.250 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:45:25.250 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:45:25.250 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:45:25.250 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:45:25.250 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:45:25.252 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:45:25.252 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:45:25.252 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:45:25.252 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:45:25.252 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:45:25.252 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:45:25.253 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:45:25.253 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:45:25.253 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:45:25.253 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:45:25.253 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:45:25.253 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:45:25.253 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:45:25.253 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:45:25.253 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:45:25.253 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:45:25.276 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:45:25.276 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:45:25.276 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:45:25.277 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:45:25.402 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:45:25.404 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:45:25.404 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:45:25.404 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:45:25.404 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:45:25.404 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:45:25.404 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:45:25.405 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:45:25.405 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:45:25.405 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:45:25.405 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:45:25.405 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:45:25.406 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:45:25.406 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:45:25.407 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:45:25.407 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:45:25.453 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 32 2024-04-06T09:45:25.454 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 33 2024-04-06T09:45:25.454 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:45:25.454 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=133, tag_tid=21, entry_tid=1], [object_number=132, tag_tid=21, entry_tid=0], [object_number=135, tag_tid=20, entry_tid=19939], [object_number=134, tag_tid=20, entry_tid=19938]]], state=connected] 2024-04-06T09:45:25.454 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=129, tag_tid=20, entry_tid=16461], [object_number=128, tag_tid=20, entry_tid=16460], [object_number=131, tag_tid=20, entry_tid=16459], [object_number=130, tag_tid=20, entry_tid=16458]]], state=connected] 2024-04-06T09:45:25.457 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:45:25.458 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=133, tag_tid=21, entry_tid=1' 2024-04-06T09:45:25.459 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:45:25.459 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:45:25.459 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:45:25.459 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:45:25.460 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:45:25.460 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:45:25.460 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:45:25.460 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:45:25.460 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:45:25.460 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:45:25.460 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:45:25.460 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:45:25.461 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:45:25.462 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:45:25.462 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:45:25.508 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 32 2024-04-06T09:45:25.508 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 33 2024-04-06T09:45:25.508 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:45:25.508 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=133, tag_tid=21, entry_tid=1], [object_number=132, tag_tid=21, entry_tid=0], [object_number=135, tag_tid=20, entry_tid=19939], [object_number=134, tag_tid=20, entry_tid=19938]]], state=connected] 2024-04-06T09:45:25.508 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=129, tag_tid=20, entry_tid=16461], [object_number=128, tag_tid=20, entry_tid=16460], [object_number=131, tag_tid=20, entry_tid=16459], [object_number=130, tag_tid=20, entry_tid=16458]]], state=connected] 2024-04-06T09:45:25.511 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:45:25.513 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=129, tag_tid=20, entry_tid=16461' 2024-04-06T09:45:25.513 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=133, tag_tid=21, entry_tid=1' -a 'object_number=133, tag_tid=21, entry_tid=1' = 'object_number=129, tag_tid=20, entry_tid=16461' 2024-04-06T09:45:25.514 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=129, tag_tid=20, entry_tid=16461' '!=' 'object_number=130, tag_tid=20, entry_tid=16374' 2024-04-06T09:45:25.514 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:45:25.514 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=129, tag_tid=20, entry_tid=16461' = 'object_number=130, tag_tid=20, entry_tid=16374' 2024-04-06T09:45:25.514 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=129, tag_tid=20, entry_tid=16461' 2024-04-06T09:45:25.514 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=133, tag_tid=21, entry_tid=1' 2024-04-06T09:45:25.514 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:45:25.514 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:45:25.514 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=21 2024-04-06T09:45:25.515 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=129, tag_tid=20, entry_tid=16461' 2024-04-06T09:45:25.515 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:45:25.515 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:45:25.517 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=20 2024-04-06T09:45:25.517 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=133, tag_tid=21, entry_tid=1' 2024-04-06T09:45:25.517 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:45:25.518 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:45:25.518 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:45:25.519 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=129, tag_tid=20, entry_tid=16461' 2024-04-06T09:45:25.519 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:45:25.519 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:45:25.521 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=16461 2024-04-06T09:45:25.521 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 21 = 20 -a 1 -le 16461 2024-04-06T09:45:25.521 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:45:25.521 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:45:25.521 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:45:25.723 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:45:25.723 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:45:25.723 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:45:25.723 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:45:25.723 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:45:25.723 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:45:25.723 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:45:25.723 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:45:25.723 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:45:25.723 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:45:25.724 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:45:25.724 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:45:25.724 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:45:25.724 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:45:25.724 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:45:25.724 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:45:25.724 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:45:25.728 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:45:25.728 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:45:25.728 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:45:25.728 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:45:25.728 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:45:25.728 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:45:25.728 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:45:25.728 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:45:25.728 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:45:25.728 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:45:25.729 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:45:25.729 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:45:25.729 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:45:25.729 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:45:25.729 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:45:25.729 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:45:25.729 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:45:25.729 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:45:25.729 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:45:25.729 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:45:25.729 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:45:25.729 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:45:25.729 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:45:25.729 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:45:25.729 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:45:25.730 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:45:25.730 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:45:25.730 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:45:25.730 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:45:25.730 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:45:25.730 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:45:25.730 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:45:25.754 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:45:25.754 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:45:25.754 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:45:25.754 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:45:25.885 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:45:25.886 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:45:25.886 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:45:25.886 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:45:25.886 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:45:25.886 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:45:25.886 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:45:25.886 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:45:25.887 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:45:25.887 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:45:25.887 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:45:25.887 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:45:25.887 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:45:25.890 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:45:25.890 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:45:25.891 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:45:25.942 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 32 2024-04-06T09:45:25.942 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 33 2024-04-06T09:45:25.942 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:45:25.942 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=133, tag_tid=21, entry_tid=1], [object_number=132, tag_tid=21, entry_tid=0], [object_number=135, tag_tid=20, entry_tid=19939], [object_number=134, tag_tid=20, entry_tid=19938]]], state=connected] 2024-04-06T09:45:25.943 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=128, tag_tid=20, entry_tid=16544], [object_number=131, tag_tid=20, entry_tid=16543], [object_number=130, tag_tid=20, entry_tid=16542], [object_number=129, tag_tid=20, entry_tid=16541]]], state=connected] 2024-04-06T09:45:25.946 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:45:25.947 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=133, tag_tid=21, entry_tid=1' 2024-04-06T09:45:25.948 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:45:25.948 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:45:25.949 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:45:25.949 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:45:25.949 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:45:25.949 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:45:25.949 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:45:25.949 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:45:25.949 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:45:25.949 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:45:25.949 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:45:25.949 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:45:25.950 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:45:25.950 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:45:25.950 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:45:25.997 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 32 2024-04-06T09:45:25.997 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 33 2024-04-06T09:45:25.997 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:45:25.997 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=133, tag_tid=21, entry_tid=1], [object_number=132, tag_tid=21, entry_tid=0], [object_number=135, tag_tid=20, entry_tid=19939], [object_number=134, tag_tid=20, entry_tid=19938]]], state=connected] 2024-04-06T09:45:25.997 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=128, tag_tid=20, entry_tid=16544], [object_number=131, tag_tid=20, entry_tid=16543], [object_number=130, tag_tid=20, entry_tid=16542], [object_number=129, tag_tid=20, entry_tid=16541]]], state=connected] 2024-04-06T09:45:26.000 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:45:26.002 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=128, tag_tid=20, entry_tid=16544' 2024-04-06T09:45:26.002 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=133, tag_tid=21, entry_tid=1' -a 'object_number=133, tag_tid=21, entry_tid=1' = 'object_number=128, tag_tid=20, entry_tid=16544' 2024-04-06T09:45:26.002 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=128, tag_tid=20, entry_tid=16544' '!=' 'object_number=129, tag_tid=20, entry_tid=16461' 2024-04-06T09:45:26.002 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:45:26.003 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=128, tag_tid=20, entry_tid=16544' = 'object_number=129, tag_tid=20, entry_tid=16461' 2024-04-06T09:45:26.003 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=128, tag_tid=20, entry_tid=16544' 2024-04-06T09:45:26.003 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=133, tag_tid=21, entry_tid=1' 2024-04-06T09:45:26.003 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:45:26.003 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:45:26.004 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=21 2024-04-06T09:45:26.004 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=128, tag_tid=20, entry_tid=16544' 2024-04-06T09:45:26.004 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:45:26.005 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:45:26.005 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=20 2024-04-06T09:45:26.006 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=133, tag_tid=21, entry_tid=1' 2024-04-06T09:45:26.006 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:45:26.006 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:45:26.007 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:45:26.008 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=128, tag_tid=20, entry_tid=16544' 2024-04-06T09:45:26.008 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:45:26.008 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:45:26.009 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=16544 2024-04-06T09:45:26.009 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 21 = 20 -a 1 -le 16544 2024-04-06T09:45:26.009 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:45:26.009 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:45:26.010 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:45:26.222 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:45:26.222 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:45:26.222 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:45:26.222 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:45:26.222 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:45:26.222 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:45:26.223 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:45:26.223 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:45:26.223 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:45:26.223 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:45:26.223 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:45:26.223 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:45:26.223 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:45:26.223 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:45:26.223 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:45:26.223 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:45:26.223 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:45:26.223 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:45:26.223 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:45:26.224 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:45:26.224 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:45:26.224 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:45:26.224 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:45:26.224 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:45:26.224 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:45:26.224 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:45:26.224 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:45:26.224 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:45:26.224 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:45:26.224 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:45:26.224 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:45:26.224 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:45:26.224 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:45:26.224 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:45:26.225 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:45:26.225 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:45:26.225 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:45:26.225 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:45:26.225 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:45:26.225 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:45:26.225 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:45:26.225 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:45:26.225 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:45:26.225 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:45:26.225 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:45:26.225 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:45:26.225 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:45:26.225 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:45:26.225 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:45:26.245 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:45:26.245 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:45:26.245 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:45:26.245 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:45:26.371 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:45:26.372 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:45:26.372 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:45:26.373 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:45:26.373 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:45:26.373 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:45:26.373 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:45:26.373 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:45:26.373 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:45:26.373 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:45:26.373 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:45:26.374 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:45:26.374 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:45:26.375 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:45:26.376 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:45:26.376 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:45:26.428 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 32 2024-04-06T09:45:26.428 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 33 2024-04-06T09:45:26.428 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:45:26.428 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=133, tag_tid=21, entry_tid=1], [object_number=132, tag_tid=21, entry_tid=0], [object_number=135, tag_tid=20, entry_tid=19939], [object_number=134, tag_tid=20, entry_tid=19938]]], state=connected] 2024-04-06T09:45:26.428 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=130, tag_tid=20, entry_tid=16634], [object_number=129, tag_tid=20, entry_tid=16633], [object_number=128, tag_tid=20, entry_tid=16632], [object_number=131, tag_tid=20, entry_tid=16631]]], state=connected] 2024-04-06T09:45:26.431 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:45:26.433 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=133, tag_tid=21, entry_tid=1' 2024-04-06T09:45:26.434 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:45:26.434 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:45:26.434 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:45:26.434 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:45:26.434 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:45:26.434 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:45:26.434 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:45:26.434 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:45:26.434 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:45:26.434 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:45:26.435 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:45:26.435 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:45:26.435 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:45:26.435 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:45:26.435 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:45:26.483 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 32 2024-04-06T09:45:26.483 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 33 2024-04-06T09:45:26.483 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:45:26.483 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=133, tag_tid=21, entry_tid=1], [object_number=132, tag_tid=21, entry_tid=0], [object_number=135, tag_tid=20, entry_tid=19939], [object_number=134, tag_tid=20, entry_tid=19938]]], state=connected] 2024-04-06T09:45:26.483 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=130, tag_tid=20, entry_tid=16634], [object_number=129, tag_tid=20, entry_tid=16633], [object_number=128, tag_tid=20, entry_tid=16632], [object_number=131, tag_tid=20, entry_tid=16631]]], state=connected] 2024-04-06T09:45:26.486 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:45:26.488 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=130, tag_tid=20, entry_tid=16634' 2024-04-06T09:45:26.488 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=133, tag_tid=21, entry_tid=1' -a 'object_number=133, tag_tid=21, entry_tid=1' = 'object_number=130, tag_tid=20, entry_tid=16634' 2024-04-06T09:45:26.488 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=130, tag_tid=20, entry_tid=16634' '!=' 'object_number=128, tag_tid=20, entry_tid=16544' 2024-04-06T09:45:26.488 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:45:26.489 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=130, tag_tid=20, entry_tid=16634' = 'object_number=128, tag_tid=20, entry_tid=16544' 2024-04-06T09:45:26.489 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=130, tag_tid=20, entry_tid=16634' 2024-04-06T09:45:26.489 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=133, tag_tid=21, entry_tid=1' 2024-04-06T09:45:26.489 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:45:26.489 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:45:26.490 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=21 2024-04-06T09:45:26.490 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=130, tag_tid=20, entry_tid=16634' 2024-04-06T09:45:26.491 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:45:26.491 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:45:26.491 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=20 2024-04-06T09:45:26.492 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=133, tag_tid=21, entry_tid=1' 2024-04-06T09:45:26.492 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:45:26.493 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:45:26.493 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:45:26.494 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=130, tag_tid=20, entry_tid=16634' 2024-04-06T09:45:26.494 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:45:26.494 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:45:26.495 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=16634 2024-04-06T09:45:26.495 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 21 = 20 -a 1 -le 16634 2024-04-06T09:45:26.495 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:45:26.495 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:45:26.496 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:45:26.698 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:45:26.699 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:45:26.700 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:45:26.700 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:45:26.700 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:45:26.700 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:45:26.700 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:45:26.700 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:45:26.700 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:45:26.701 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:45:26.701 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:45:26.701 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:45:26.701 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:45:26.701 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:45:26.701 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:45:26.703 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:45:26.703 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:45:26.703 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:45:26.703 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:45:26.703 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:45:26.704 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:45:26.704 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:45:26.704 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:45:26.704 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:45:26.704 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:45:26.704 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:45:26.704 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:45:26.704 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:45:26.704 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:45:26.704 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:45:26.704 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:45:26.704 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:45:26.704 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:45:26.704 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:45:26.705 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:45:26.705 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:45:26.705 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:45:26.705 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:45:26.705 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:45:26.705 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:45:26.705 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:45:26.705 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:45:26.705 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:45:26.705 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:45:26.705 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:45:26.705 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:45:26.705 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:45:26.705 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:45:26.706 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:45:26.729 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:45:26.730 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:45:26.730 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:45:26.730 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:45:26.860 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:45:26.861 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:45:26.862 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:45:26.862 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:45:26.862 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:45:26.862 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:45:26.862 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:45:26.862 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:45:26.862 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:45:26.863 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:45:26.863 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:45:26.863 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:45:26.863 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:45:26.865 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:45:26.866 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:45:26.866 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:45:26.922 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 32 2024-04-06T09:45:26.923 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 33 2024-04-06T09:45:26.923 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:45:26.923 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=133, tag_tid=21, entry_tid=1], [object_number=132, tag_tid=21, entry_tid=0], [object_number=135, tag_tid=20, entry_tid=19939], [object_number=134, tag_tid=20, entry_tid=19938]]], state=connected] 2024-04-06T09:45:26.923 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=131, tag_tid=20, entry_tid=16723], [object_number=130, tag_tid=20, entry_tid=16722], [object_number=129, tag_tid=20, entry_tid=16721], [object_number=128, tag_tid=20, entry_tid=16720]]], state=connected] 2024-04-06T09:45:26.926 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:45:26.928 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=133, tag_tid=21, entry_tid=1' 2024-04-06T09:45:26.929 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:45:26.929 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:45:26.929 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:45:26.929 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:45:26.929 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:45:26.929 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:45:26.929 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:45:26.929 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:45:26.929 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:45:26.929 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:45:26.930 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:45:26.930 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:45:26.930 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:45:26.930 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:45:26.931 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:45:26.978 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 32 2024-04-06T09:45:26.978 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 33 2024-04-06T09:45:26.978 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:45:26.978 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=133, tag_tid=21, entry_tid=1], [object_number=132, tag_tid=21, entry_tid=0], [object_number=135, tag_tid=20, entry_tid=19939], [object_number=134, tag_tid=20, entry_tid=19938]]], state=connected] 2024-04-06T09:45:26.978 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=131, tag_tid=20, entry_tid=16723], [object_number=130, tag_tid=20, entry_tid=16722], [object_number=129, tag_tid=20, entry_tid=16721], [object_number=128, tag_tid=20, entry_tid=16720]]], state=connected] 2024-04-06T09:45:26.981 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:45:26.983 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=131, tag_tid=20, entry_tid=16723' 2024-04-06T09:45:26.983 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=133, tag_tid=21, entry_tid=1' -a 'object_number=133, tag_tid=21, entry_tid=1' = 'object_number=131, tag_tid=20, entry_tid=16723' 2024-04-06T09:45:26.983 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=131, tag_tid=20, entry_tid=16723' '!=' 'object_number=130, tag_tid=20, entry_tid=16634' 2024-04-06T09:45:26.983 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:45:26.983 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=131, tag_tid=20, entry_tid=16723' = 'object_number=130, tag_tid=20, entry_tid=16634' 2024-04-06T09:45:26.983 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=131, tag_tid=20, entry_tid=16723' 2024-04-06T09:45:26.984 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=133, tag_tid=21, entry_tid=1' 2024-04-06T09:45:26.984 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:45:26.984 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:45:26.984 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=21 2024-04-06T09:45:26.985 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=131, tag_tid=20, entry_tid=16723' 2024-04-06T09:45:26.985 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:45:26.985 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:45:26.986 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=20 2024-04-06T09:45:26.987 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=133, tag_tid=21, entry_tid=1' 2024-04-06T09:45:26.987 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:45:26.987 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:45:26.988 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:45:26.988 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=131, tag_tid=20, entry_tid=16723' 2024-04-06T09:45:26.989 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:45:26.989 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:45:26.990 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=16723 2024-04-06T09:45:26.990 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 21 = 20 -a 1 -le 16723 2024-04-06T09:45:26.990 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:45:26.990 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:45:26.990 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:45:27.192 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:45:27.192 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:45:27.192 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:45:27.192 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:45:27.192 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:45:27.192 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:45:27.192 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:45:27.192 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:45:27.192 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:45:27.192 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:45:27.193 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:45:27.193 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:45:27.193 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:45:27.193 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:45:27.193 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:45:27.193 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:45:27.193 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:45:27.197 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:45:27.197 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:45:27.197 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:45:27.197 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:45:27.197 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:45:27.197 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:45:27.197 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:45:27.198 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:45:27.198 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:45:27.198 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:45:27.198 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:45:27.198 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:45:27.198 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:45:27.198 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:45:27.198 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:45:27.198 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:45:27.198 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:45:27.198 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:45:27.198 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:45:27.198 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:45:27.198 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:45:27.199 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:45:27.199 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:45:27.199 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:45:27.199 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:45:27.199 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:45:27.199 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:45:27.199 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:45:27.199 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:45:27.199 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:45:27.199 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:45:27.199 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:45:27.223 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:45:27.224 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:45:27.224 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:45:27.224 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:45:27.355 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:45:27.356 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:45:27.356 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:45:27.356 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:45:27.356 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:45:27.356 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:45:27.357 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:45:27.357 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:45:27.357 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:45:27.357 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:45:27.357 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:45:27.357 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:45:27.357 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:45:27.358 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:45:27.359 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:45:27.359 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:45:27.411 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 32 2024-04-06T09:45:27.411 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 33 2024-04-06T09:45:27.411 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:45:27.411 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=133, tag_tid=21, entry_tid=1], [object_number=132, tag_tid=21, entry_tid=0], [object_number=135, tag_tid=20, entry_tid=19939], [object_number=134, tag_tid=20, entry_tid=19938]]], state=connected] 2024-04-06T09:45:27.411 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=129, tag_tid=20, entry_tid=16813], [object_number=128, tag_tid=20, entry_tid=16812], [object_number=131, tag_tid=20, entry_tid=16811], [object_number=130, tag_tid=20, entry_tid=16810]]], state=connected] 2024-04-06T09:45:27.415 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:45:27.419 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=133, tag_tid=21, entry_tid=1' 2024-04-06T09:45:27.419 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:45:27.419 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:45:27.419 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:45:27.419 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:45:27.419 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:45:27.419 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:45:27.419 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:45:27.419 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:45:27.420 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:45:27.420 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:45:27.420 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:45:27.420 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:45:27.420 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:45:27.420 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:45:27.420 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:45:27.466 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 32 2024-04-06T09:45:27.466 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 33 2024-04-06T09:45:27.466 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:45:27.466 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=133, tag_tid=21, entry_tid=1], [object_number=132, tag_tid=21, entry_tid=0], [object_number=135, tag_tid=20, entry_tid=19939], [object_number=134, tag_tid=20, entry_tid=19938]]], state=connected] 2024-04-06T09:45:27.466 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=129, tag_tid=20, entry_tid=16813], [object_number=128, tag_tid=20, entry_tid=16812], [object_number=131, tag_tid=20, entry_tid=16811], [object_number=130, tag_tid=20, entry_tid=16810]]], state=connected] 2024-04-06T09:45:27.470 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:45:27.471 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=129, tag_tid=20, entry_tid=16813' 2024-04-06T09:45:27.471 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=133, tag_tid=21, entry_tid=1' -a 'object_number=133, tag_tid=21, entry_tid=1' = 'object_number=129, tag_tid=20, entry_tid=16813' 2024-04-06T09:45:27.471 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=129, tag_tid=20, entry_tid=16813' '!=' 'object_number=131, tag_tid=20, entry_tid=16723' 2024-04-06T09:45:27.472 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:45:27.472 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=129, tag_tid=20, entry_tid=16813' = 'object_number=131, tag_tid=20, entry_tid=16723' 2024-04-06T09:45:27.472 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=129, tag_tid=20, entry_tid=16813' 2024-04-06T09:45:27.472 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=133, tag_tid=21, entry_tid=1' 2024-04-06T09:45:27.472 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:45:27.472 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:45:27.473 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=21 2024-04-06T09:45:27.474 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=129, tag_tid=20, entry_tid=16813' 2024-04-06T09:45:27.474 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:45:27.474 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:45:27.475 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=20 2024-04-06T09:45:27.475 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=133, tag_tid=21, entry_tid=1' 2024-04-06T09:45:27.475 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:45:27.475 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:45:27.476 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:45:27.477 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=129, tag_tid=20, entry_tid=16813' 2024-04-06T09:45:27.477 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:45:27.477 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:45:27.478 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=16813 2024-04-06T09:45:27.478 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 21 = 20 -a 1 -le 16813 2024-04-06T09:45:27.478 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:45:27.479 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:45:27.479 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:45:27.680 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:45:27.680 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:45:27.680 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:45:27.680 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:45:27.680 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:45:27.680 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:45:27.681 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:45:27.681 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:45:27.681 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:45:27.681 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:45:27.681 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:45:27.681 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:45:27.681 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:45:27.681 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:45:27.681 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:45:27.681 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:45:27.682 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:45:27.685 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:45:27.685 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:45:27.685 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:45:27.685 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:45:27.685 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:45:27.685 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:45:27.685 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:45:27.686 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:45:27.686 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:45:27.686 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:45:27.686 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:45:27.686 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:45:27.686 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:45:27.686 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:45:27.686 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:45:27.686 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:45:27.686 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:45:27.686 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:45:27.686 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:45:27.686 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:45:27.687 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:45:27.687 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:45:27.687 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:45:27.687 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:45:27.687 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:45:27.687 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:45:27.687 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:45:27.687 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:45:27.687 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:45:27.687 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:45:27.687 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:45:27.687 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:45:27.710 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:45:27.711 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:45:27.711 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:45:27.711 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:45:27.843 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:45:27.844 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:45:27.844 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:45:27.844 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:45:27.844 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:45:27.844 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:45:27.845 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:45:27.845 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:45:27.845 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:45:27.845 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:45:27.845 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:45:27.845 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:45:27.845 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:45:27.847 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:45:27.847 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:45:27.847 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:45:27.895 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 32 2024-04-06T09:45:27.895 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 33 2024-04-06T09:45:27.895 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:45:27.895 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=133, tag_tid=21, entry_tid=1], [object_number=132, tag_tid=21, entry_tid=0], [object_number=135, tag_tid=20, entry_tid=19939], [object_number=134, tag_tid=20, entry_tid=19938]]], state=connected] 2024-04-06T09:45:27.896 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=130, tag_tid=20, entry_tid=16902], [object_number=129, tag_tid=20, entry_tid=16901], [object_number=128, tag_tid=20, entry_tid=16900], [object_number=131, tag_tid=20, entry_tid=16899]]], state=connected] 2024-04-06T09:45:27.899 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:45:27.901 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=133, tag_tid=21, entry_tid=1' 2024-04-06T09:45:27.901 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:45:27.901 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:45:27.902 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:45:27.902 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:45:27.902 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:45:27.902 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:45:27.902 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:45:27.902 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:45:27.902 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:45:27.902 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:45:27.902 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:45:27.902 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:45:27.903 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:45:27.903 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:45:27.903 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:45:27.949 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 32 2024-04-06T09:45:27.949 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 33 2024-04-06T09:45:27.949 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:45:27.949 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=133, tag_tid=21, entry_tid=1], [object_number=132, tag_tid=21, entry_tid=0], [object_number=135, tag_tid=20, entry_tid=19939], [object_number=134, tag_tid=20, entry_tid=19938]]], state=connected] 2024-04-06T09:45:27.950 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=130, tag_tid=20, entry_tid=16902], [object_number=129, tag_tid=20, entry_tid=16901], [object_number=128, tag_tid=20, entry_tid=16900], [object_number=131, tag_tid=20, entry_tid=16899]]], state=connected] 2024-04-06T09:45:27.953 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:45:27.955 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=130, tag_tid=20, entry_tid=16902' 2024-04-06T09:45:27.955 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=133, tag_tid=21, entry_tid=1' -a 'object_number=133, tag_tid=21, entry_tid=1' = 'object_number=130, tag_tid=20, entry_tid=16902' 2024-04-06T09:45:27.955 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=130, tag_tid=20, entry_tid=16902' '!=' 'object_number=129, tag_tid=20, entry_tid=16813' 2024-04-06T09:45:27.955 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:45:27.955 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=130, tag_tid=20, entry_tid=16902' = 'object_number=129, tag_tid=20, entry_tid=16813' 2024-04-06T09:45:27.955 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=130, tag_tid=20, entry_tid=16902' 2024-04-06T09:45:27.955 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=133, tag_tid=21, entry_tid=1' 2024-04-06T09:45:27.955 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:45:27.955 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:45:27.956 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=21 2024-04-06T09:45:27.956 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=130, tag_tid=20, entry_tid=16902' 2024-04-06T09:45:27.957 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:45:27.957 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:45:27.958 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=20 2024-04-06T09:45:27.958 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=133, tag_tid=21, entry_tid=1' 2024-04-06T09:45:27.958 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:45:27.958 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:45:27.959 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:45:27.960 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=130, tag_tid=20, entry_tid=16902' 2024-04-06T09:45:27.960 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:45:27.960 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:45:27.961 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=16902 2024-04-06T09:45:27.961 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 21 = 20 -a 1 -le 16902 2024-04-06T09:45:27.961 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:45:27.961 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:45:27.961 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:45:28.163 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:45:28.163 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:45:28.163 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:45:28.163 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:45:28.163 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:45:28.163 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:45:28.163 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:45:28.163 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:45:28.163 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:45:28.164 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:45:28.164 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:45:28.164 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:45:28.164 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:45:28.164 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:45:28.164 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:45:28.164 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:45:28.165 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:45:28.167 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:45:28.167 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:45:28.167 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:45:28.167 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:45:28.167 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:45:28.167 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:45:28.168 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:45:28.168 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:45:28.168 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:45:28.168 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:45:28.168 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:45:28.168 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:45:28.168 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:45:28.168 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:45:28.168 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:45:28.168 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:45:28.168 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:45:28.168 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:45:28.168 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:45:28.168 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:45:28.168 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:45:28.168 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:45:28.168 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:45:28.169 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:45:28.169 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:45:28.169 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:45:28.169 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:45:28.169 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:45:28.169 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:45:28.169 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:45:28.169 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:45:28.169 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:45:28.194 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:45:28.194 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:45:28.194 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:45:28.194 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:45:28.317 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:45:28.318 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:45:28.318 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:45:28.318 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:45:28.318 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:45:28.318 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:45:28.318 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:45:28.318 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:45:28.318 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:45:28.318 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:45:28.319 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:45:28.319 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:45:28.319 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:45:28.321 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:45:28.322 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:45:28.322 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:45:28.373 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 32 2024-04-06T09:45:28.373 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 33 2024-04-06T09:45:28.374 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:45:28.374 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=133, tag_tid=21, entry_tid=1], [object_number=132, tag_tid=21, entry_tid=0], [object_number=135, tag_tid=20, entry_tid=19939], [object_number=134, tag_tid=20, entry_tid=19938]]], state=connected] 2024-04-06T09:45:28.374 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=129, tag_tid=20, entry_tid=16989], [object_number=128, tag_tid=20, entry_tid=16988], [object_number=131, tag_tid=20, entry_tid=16987], [object_number=130, tag_tid=20, entry_tid=16986]]], state=connected] 2024-04-06T09:45:28.377 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:45:28.378 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=133, tag_tid=21, entry_tid=1' 2024-04-06T09:45:28.380 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:45:28.380 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:45:28.380 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:45:28.380 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:45:28.380 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:45:28.380 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:45:28.380 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:45:28.380 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:45:28.380 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:45:28.380 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:45:28.380 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:45:28.380 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:45:28.381 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:45:28.381 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:45:28.381 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:45:28.428 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 32 2024-04-06T09:45:28.428 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 33 2024-04-06T09:45:28.428 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:45:28.428 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=133, tag_tid=21, entry_tid=1], [object_number=132, tag_tid=21, entry_tid=0], [object_number=135, tag_tid=20, entry_tid=19939], [object_number=134, tag_tid=20, entry_tid=19938]]], state=connected] 2024-04-06T09:45:28.428 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=129, tag_tid=20, entry_tid=16989], [object_number=128, tag_tid=20, entry_tid=16988], [object_number=131, tag_tid=20, entry_tid=16987], [object_number=130, tag_tid=20, entry_tid=16986]]], state=connected] 2024-04-06T09:45:28.431 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:45:28.433 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=129, tag_tid=20, entry_tid=16989' 2024-04-06T09:45:28.433 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=133, tag_tid=21, entry_tid=1' -a 'object_number=133, tag_tid=21, entry_tid=1' = 'object_number=129, tag_tid=20, entry_tid=16989' 2024-04-06T09:45:28.434 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=129, tag_tid=20, entry_tid=16989' '!=' 'object_number=130, tag_tid=20, entry_tid=16902' 2024-04-06T09:45:28.434 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:45:28.434 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=129, tag_tid=20, entry_tid=16989' = 'object_number=130, tag_tid=20, entry_tid=16902' 2024-04-06T09:45:28.434 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=129, tag_tid=20, entry_tid=16989' 2024-04-06T09:45:28.434 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=133, tag_tid=21, entry_tid=1' 2024-04-06T09:45:28.434 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:45:28.434 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:45:28.434 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=21 2024-04-06T09:45:28.435 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=129, tag_tid=20, entry_tid=16989' 2024-04-06T09:45:28.435 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:45:28.435 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:45:28.436 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=20 2024-04-06T09:45:28.437 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=133, tag_tid=21, entry_tid=1' 2024-04-06T09:45:28.437 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:45:28.437 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:45:28.438 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:45:28.438 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=129, tag_tid=20, entry_tid=16989' 2024-04-06T09:45:28.439 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:45:28.439 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:45:28.440 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=16989 2024-04-06T09:45:28.440 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 21 = 20 -a 1 -le 16989 2024-04-06T09:45:28.440 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:45:28.440 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:45:28.440 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:45:28.642 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:45:28.642 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:45:28.642 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:45:28.642 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:45:28.642 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:45:28.642 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:45:28.643 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:45:28.643 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:45:28.643 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:45:28.643 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:45:28.643 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:45:28.643 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:45:28.643 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:45:28.643 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:45:28.643 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:45:28.644 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:45:28.644 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:45:28.645 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:45:28.645 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:45:28.645 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:45:28.645 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:45:28.645 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:45:28.645 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:45:28.645 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:45:28.645 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:45:28.645 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:45:28.645 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:45:28.646 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:45:28.646 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:45:28.646 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:45:28.646 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:45:28.646 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:45:28.646 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:45:28.647 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:45:28.647 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:45:28.647 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:45:28.647 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:45:28.647 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:45:28.647 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:45:28.647 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:45:28.647 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:45:28.648 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:45:28.648 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:45:28.648 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:45:28.648 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:45:28.648 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:45:28.648 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:45:28.648 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:45:28.648 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:45:28.672 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:45:28.673 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:45:28.673 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:45:28.673 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:45:28.803 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:45:28.804 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:45:28.804 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:45:28.804 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:45:28.804 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:45:28.804 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:45:28.804 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:45:28.804 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:45:28.804 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:45:28.805 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:45:28.805 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:45:28.805 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:45:28.805 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:45:28.807 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:45:28.808 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:45:28.808 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:45:28.855 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 32 2024-04-06T09:45:28.855 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 33 2024-04-06T09:45:28.856 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:45:28.856 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=133, tag_tid=21, entry_tid=1], [object_number=132, tag_tid=21, entry_tid=0], [object_number=135, tag_tid=20, entry_tid=19939], [object_number=134, tag_tid=20, entry_tid=19938]]], state=connected] 2024-04-06T09:45:28.856 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=128, tag_tid=20, entry_tid=17076], [object_number=131, tag_tid=20, entry_tid=17075], [object_number=130, tag_tid=20, entry_tid=17074], [object_number=129, tag_tid=20, entry_tid=17073]]], state=connected] 2024-04-06T09:45:28.859 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:45:28.860 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=133, tag_tid=21, entry_tid=1' 2024-04-06T09:45:28.861 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:45:28.861 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:45:28.861 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:45:28.862 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:45:28.862 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:45:28.862 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:45:28.862 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:45:28.862 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:45:28.862 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:45:28.862 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:45:28.862 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:45:28.862 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:45:28.863 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:45:28.863 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:45:28.863 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:45:28.909 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 32 2024-04-06T09:45:28.909 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 33 2024-04-06T09:45:28.909 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:45:28.909 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=133, tag_tid=21, entry_tid=1], [object_number=132, tag_tid=21, entry_tid=0], [object_number=135, tag_tid=20, entry_tid=19939], [object_number=134, tag_tid=20, entry_tid=19938]]], state=connected] 2024-04-06T09:45:28.910 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=128, tag_tid=20, entry_tid=17076], [object_number=131, tag_tid=20, entry_tid=17075], [object_number=130, tag_tid=20, entry_tid=17074], [object_number=129, tag_tid=20, entry_tid=17073]]], state=connected] 2024-04-06T09:45:28.913 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:45:28.915 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=128, tag_tid=20, entry_tid=17076' 2024-04-06T09:45:28.915 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=133, tag_tid=21, entry_tid=1' -a 'object_number=133, tag_tid=21, entry_tid=1' = 'object_number=128, tag_tid=20, entry_tid=17076' 2024-04-06T09:45:28.915 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=128, tag_tid=20, entry_tid=17076' '!=' 'object_number=129, tag_tid=20, entry_tid=16989' 2024-04-06T09:45:28.915 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:45:28.915 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=128, tag_tid=20, entry_tid=17076' = 'object_number=129, tag_tid=20, entry_tid=16989' 2024-04-06T09:45:28.915 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=128, tag_tid=20, entry_tid=17076' 2024-04-06T09:45:28.915 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=133, tag_tid=21, entry_tid=1' 2024-04-06T09:45:28.915 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:45:28.915 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:45:28.916 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=21 2024-04-06T09:45:28.917 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=128, tag_tid=20, entry_tid=17076' 2024-04-06T09:45:28.917 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:45:28.917 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:45:28.918 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=20 2024-04-06T09:45:28.918 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=133, tag_tid=21, entry_tid=1' 2024-04-06T09:45:28.918 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:45:28.919 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:45:28.919 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:45:28.920 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=128, tag_tid=20, entry_tid=17076' 2024-04-06T09:45:28.920 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:45:28.920 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:45:28.921 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=17076 2024-04-06T09:45:28.921 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 21 = 20 -a 1 -le 17076 2024-04-06T09:45:28.921 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:45:28.921 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:45:28.921 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:45:29.123 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:45:29.124 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:45:29.124 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:45:29.124 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:45:29.124 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:45:29.124 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:45:29.124 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:45:29.124 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:45:29.124 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:45:29.124 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:45:29.124 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:45:29.124 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:45:29.124 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:45:29.124 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:45:29.124 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:45:29.125 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:45:29.125 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:45:29.128 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:45:29.128 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:45:29.128 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:45:29.128 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:45:29.128 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:45:29.128 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:45:29.129 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:45:29.129 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:45:29.129 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:45:29.129 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:45:29.129 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:45:29.129 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:45:29.129 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:45:29.129 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:45:29.129 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:45:29.129 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:45:29.129 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:45:29.129 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:45:29.129 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:45:29.129 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:45:29.130 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:45:29.130 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:45:29.130 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:45:29.130 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:45:29.130 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:45:29.130 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:45:29.130 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:45:29.130 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:45:29.130 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:45:29.130 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:45:29.130 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:45:29.131 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:45:29.154 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:45:29.154 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:45:29.154 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:45:29.154 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:45:29.285 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:45:29.286 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:45:29.286 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:45:29.286 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:45:29.286 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:45:29.286 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:45:29.286 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:45:29.286 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:45:29.286 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:45:29.286 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:45:29.287 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:45:29.287 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:45:29.287 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:45:29.290 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:45:29.290 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:45:29.290 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:45:29.336 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 32 2024-04-06T09:45:29.336 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 33 2024-04-06T09:45:29.336 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:45:29.336 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=133, tag_tid=21, entry_tid=1], [object_number=132, tag_tid=21, entry_tid=0], [object_number=135, tag_tid=20, entry_tid=19939], [object_number=134, tag_tid=20, entry_tid=19938]]], state=connected] 2024-04-06T09:45:29.336 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=128, tag_tid=20, entry_tid=17156], [object_number=131, tag_tid=20, entry_tid=17155], [object_number=130, tag_tid=20, entry_tid=17154], [object_number=129, tag_tid=20, entry_tid=17153]]], state=connected] 2024-04-06T09:45:29.340 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:45:29.341 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=133, tag_tid=21, entry_tid=1' 2024-04-06T09:45:29.342 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:45:29.342 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:45:29.342 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:45:29.342 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:45:29.343 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:45:29.343 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:45:29.343 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:45:29.343 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:45:29.343 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:45:29.343 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:45:29.343 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:45:29.343 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:45:29.344 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:45:29.344 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:45:29.344 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:45:29.390 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 32 2024-04-06T09:45:29.390 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 33 2024-04-06T09:45:29.390 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:45:29.390 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=133, tag_tid=21, entry_tid=1], [object_number=132, tag_tid=21, entry_tid=0], [object_number=135, tag_tid=20, entry_tid=19939], [object_number=134, tag_tid=20, entry_tid=19938]]], state=connected] 2024-04-06T09:45:29.391 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=128, tag_tid=20, entry_tid=17156], [object_number=131, tag_tid=20, entry_tid=17155], [object_number=130, tag_tid=20, entry_tid=17154], [object_number=129, tag_tid=20, entry_tid=17153]]], state=connected] 2024-04-06T09:45:29.394 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:45:29.395 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=128, tag_tid=20, entry_tid=17156' 2024-04-06T09:45:29.396 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=133, tag_tid=21, entry_tid=1' -a 'object_number=133, tag_tid=21, entry_tid=1' = 'object_number=128, tag_tid=20, entry_tid=17156' 2024-04-06T09:45:29.396 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=128, tag_tid=20, entry_tid=17156' '!=' 'object_number=128, tag_tid=20, entry_tid=17076' 2024-04-06T09:45:29.396 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:45:29.396 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=128, tag_tid=20, entry_tid=17156' = 'object_number=128, tag_tid=20, entry_tid=17076' 2024-04-06T09:45:29.396 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=128, tag_tid=20, entry_tid=17156' 2024-04-06T09:45:29.396 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=133, tag_tid=21, entry_tid=1' 2024-04-06T09:45:29.397 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:45:29.397 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:45:29.397 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=21 2024-04-06T09:45:29.398 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=128, tag_tid=20, entry_tid=17156' 2024-04-06T09:45:29.398 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:45:29.398 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:45:29.399 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=20 2024-04-06T09:45:29.399 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=133, tag_tid=21, entry_tid=1' 2024-04-06T09:45:29.399 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:45:29.400 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:45:29.400 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:45:29.401 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=128, tag_tid=20, entry_tid=17156' 2024-04-06T09:45:29.401 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:45:29.401 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:45:29.403 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=17156 2024-04-06T09:45:29.403 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 21 = 20 -a 1 -le 17156 2024-04-06T09:45:29.403 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:45:29.403 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:45:29.403 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:45:29.605 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:45:29.605 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:45:29.605 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:45:29.605 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:45:29.605 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:45:29.605 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:45:29.605 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:45:29.605 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:45:29.605 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:45:29.606 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:45:29.606 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:45:29.606 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:45:29.606 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:45:29.606 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:45:29.606 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:45:29.607 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:45:29.607 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:45:29.608 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:45:29.608 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:45:29.609 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:45:29.609 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:45:29.609 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:45:29.609 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:45:29.609 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:45:29.609 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:45:29.609 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:45:29.609 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:45:29.609 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:45:29.609 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:45:29.609 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:45:29.610 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:45:29.610 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:45:29.610 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:45:29.611 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:45:29.611 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:45:29.611 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:45:29.611 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:45:29.611 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:45:29.611 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:45:29.611 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:45:29.612 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:45:29.612 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:45:29.612 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:45:29.612 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:45:29.612 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:45:29.612 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:45:29.612 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:45:29.612 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:45:29.612 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:45:29.638 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:45:29.638 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:45:29.638 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:45:29.638 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:45:29.767 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:45:29.768 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:45:29.768 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:45:29.768 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:45:29.769 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:45:29.769 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:45:29.769 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:45:29.769 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:45:29.769 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:45:29.769 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:45:29.769 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:45:29.770 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:45:29.770 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:45:29.772 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:45:29.772 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:45:29.772 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:45:30.030 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 32 2024-04-06T09:45:30.030 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 33 2024-04-06T09:45:30.030 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:45:30.030 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=133, tag_tid=21, entry_tid=1], [object_number=132, tag_tid=21, entry_tid=0], [object_number=135, tag_tid=20, entry_tid=19939], [object_number=134, tag_tid=20, entry_tid=19938]]], state=connected] 2024-04-06T09:45:30.030 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=131, tag_tid=20, entry_tid=17231], [object_number=130, tag_tid=20, entry_tid=17230], [object_number=129, tag_tid=20, entry_tid=17229], [object_number=128, tag_tid=20, entry_tid=17228]]], state=connected] 2024-04-06T09:45:30.035 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:45:30.036 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=133, tag_tid=21, entry_tid=1' 2024-04-06T09:45:30.038 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:45:30.038 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:45:30.038 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:45:30.038 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:45:30.038 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:45:30.038 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:45:30.038 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:45:30.039 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:45:30.039 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:45:30.039 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:45:30.039 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:45:30.039 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:45:30.040 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:45:30.040 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:45:30.040 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:45:30.090 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 32 2024-04-06T09:45:30.090 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 33 2024-04-06T09:45:30.090 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:45:30.090 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=133, tag_tid=21, entry_tid=1], [object_number=132, tag_tid=21, entry_tid=0], [object_number=135, tag_tid=20, entry_tid=19939], [object_number=134, tag_tid=20, entry_tid=19938]]], state=connected] 2024-04-06T09:45:30.091 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=131, tag_tid=20, entry_tid=17231], [object_number=130, tag_tid=20, entry_tid=17230], [object_number=129, tag_tid=20, entry_tid=17229], [object_number=128, tag_tid=20, entry_tid=17228]]], state=connected] 2024-04-06T09:45:30.094 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:45:30.096 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=131, tag_tid=20, entry_tid=17231' 2024-04-06T09:45:30.096 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=133, tag_tid=21, entry_tid=1' -a 'object_number=133, tag_tid=21, entry_tid=1' = 'object_number=131, tag_tid=20, entry_tid=17231' 2024-04-06T09:45:30.096 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=131, tag_tid=20, entry_tid=17231' '!=' 'object_number=128, tag_tid=20, entry_tid=17156' 2024-04-06T09:45:30.096 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:45:30.096 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=131, tag_tid=20, entry_tid=17231' = 'object_number=128, tag_tid=20, entry_tid=17156' 2024-04-06T09:45:30.096 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=131, tag_tid=20, entry_tid=17231' 2024-04-06T09:45:30.097 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=133, tag_tid=21, entry_tid=1' 2024-04-06T09:45:30.097 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:45:30.097 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:45:30.097 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=21 2024-04-06T09:45:30.098 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=131, tag_tid=20, entry_tid=17231' 2024-04-06T09:45:30.098 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:45:30.098 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:45:30.099 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=20 2024-04-06T09:45:30.100 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=133, tag_tid=21, entry_tid=1' 2024-04-06T09:45:30.100 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:45:30.100 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:45:30.101 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:45:30.102 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=131, tag_tid=20, entry_tid=17231' 2024-04-06T09:45:30.102 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:45:30.102 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:45:30.103 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=17231 2024-04-06T09:45:30.103 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 21 = 20 -a 1 -le 17231 2024-04-06T09:45:30.103 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:45:30.104 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:45:30.104 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:45:30.305 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:45:30.306 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:45:30.306 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:45:30.306 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:45:30.306 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:45:30.306 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:45:30.306 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:45:30.306 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:45:30.306 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:45:30.306 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:45:30.306 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:45:30.306 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:45:30.306 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:45:30.307 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:45:30.307 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:45:30.307 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:45:30.307 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:45:30.310 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:45:30.310 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:45:30.310 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:45:30.310 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:45:30.310 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:45:30.310 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:45:30.310 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:45:30.310 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:45:30.310 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:45:30.311 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:45:30.311 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:45:30.311 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:45:30.311 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:45:30.311 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:45:30.311 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:45:30.311 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:45:30.312 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:45:30.312 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:45:30.312 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:45:30.312 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:45:30.312 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:45:30.312 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:45:30.313 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:45:30.313 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:45:30.313 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:45:30.313 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:45:30.313 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:45:30.313 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:45:30.313 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:45:30.313 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:45:30.313 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:45:30.313 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:45:30.336 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:45:30.337 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:45:30.337 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:45:30.337 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:45:30.471 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:45:30.472 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:45:30.472 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:45:30.472 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:45:30.472 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:45:30.472 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:45:30.472 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:45:30.473 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:45:30.473 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:45:30.473 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:45:30.473 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:45:30.473 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:45:30.473 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:45:30.474 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:45:30.475 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:45:30.475 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:45:30.525 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 32 2024-04-06T09:45:30.525 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 33 2024-04-06T09:45:30.525 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:45:30.525 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=133, tag_tid=21, entry_tid=1], [object_number=132, tag_tid=21, entry_tid=0], [object_number=135, tag_tid=20, entry_tid=19939], [object_number=134, tag_tid=20, entry_tid=19938]]], state=connected] 2024-04-06T09:45:30.525 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=129, tag_tid=20, entry_tid=17353], [object_number=128, tag_tid=20, entry_tid=17352], [object_number=131, tag_tid=20, entry_tid=17351], [object_number=130, tag_tid=20, entry_tid=17350]]], state=connected] 2024-04-06T09:45:30.528 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:45:30.530 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=133, tag_tid=21, entry_tid=1' 2024-04-06T09:45:30.531 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:45:30.531 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:45:30.531 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:45:30.531 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:45:30.531 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:45:30.531 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:45:30.531 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:45:30.531 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:45:30.531 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:45:30.532 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:45:30.532 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:45:30.532 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:45:30.532 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:45:30.532 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:45:30.532 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:45:30.580 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 32 2024-04-06T09:45:30.580 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 33 2024-04-06T09:45:30.580 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:45:30.580 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=133, tag_tid=21, entry_tid=1], [object_number=132, tag_tid=21, entry_tid=0], [object_number=135, tag_tid=20, entry_tid=19939], [object_number=134, tag_tid=20, entry_tid=19938]]], state=connected] 2024-04-06T09:45:30.580 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=129, tag_tid=20, entry_tid=17353], [object_number=128, tag_tid=20, entry_tid=17352], [object_number=131, tag_tid=20, entry_tid=17351], [object_number=130, tag_tid=20, entry_tid=17350]]], state=connected] 2024-04-06T09:45:30.583 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:45:30.585 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=129, tag_tid=20, entry_tid=17353' 2024-04-06T09:45:30.586 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=133, tag_tid=21, entry_tid=1' -a 'object_number=133, tag_tid=21, entry_tid=1' = 'object_number=129, tag_tid=20, entry_tid=17353' 2024-04-06T09:45:30.586 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=129, tag_tid=20, entry_tid=17353' '!=' 'object_number=131, tag_tid=20, entry_tid=17231' 2024-04-06T09:45:30.586 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:45:30.586 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=129, tag_tid=20, entry_tid=17353' = 'object_number=131, tag_tid=20, entry_tid=17231' 2024-04-06T09:45:30.586 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=129, tag_tid=20, entry_tid=17353' 2024-04-06T09:45:30.586 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=133, tag_tid=21, entry_tid=1' 2024-04-06T09:45:30.586 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:45:30.586 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:45:30.587 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=21 2024-04-06T09:45:30.587 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=129, tag_tid=20, entry_tid=17353' 2024-04-06T09:45:30.588 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:45:30.588 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:45:30.588 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=20 2024-04-06T09:45:30.589 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=133, tag_tid=21, entry_tid=1' 2024-04-06T09:45:30.589 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:45:30.589 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:45:30.590 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:45:30.591 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=129, tag_tid=20, entry_tid=17353' 2024-04-06T09:45:30.591 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:45:30.591 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:45:30.592 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=17353 2024-04-06T09:45:30.592 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 21 = 20 -a 1 -le 17353 2024-04-06T09:45:30.592 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:45:30.592 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:45:30.592 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:45:30.794 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:45:30.795 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:45:30.795 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:45:30.795 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:45:30.795 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:45:30.795 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:45:30.795 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:45:30.795 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:45:30.795 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:45:30.795 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:45:30.795 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:45:30.795 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:45:30.795 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:45:30.795 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:45:30.796 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:45:30.796 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:45:30.796 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:45:30.799 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:45:30.799 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:45:30.799 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:45:30.799 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:45:30.799 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:45:30.799 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:45:30.799 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:45:30.799 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:45:30.799 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:45:30.799 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:45:30.799 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:45:30.799 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:45:30.799 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:45:30.800 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:45:30.800 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:45:30.800 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:45:30.800 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:45:30.800 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:45:30.800 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:45:30.800 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:45:30.800 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:45:30.800 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:45:30.800 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:45:30.800 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:45:30.800 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:45:30.800 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:45:30.800 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:45:30.800 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:45:30.801 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:45:30.801 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:45:30.801 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:45:30.801 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:45:30.823 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:45:30.824 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:45:30.824 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:45:30.824 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:45:30.964 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:45:30.965 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:45:30.965 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:45:30.965 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:45:30.965 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:45:30.965 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:45:30.965 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:45:30.966 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:45:30.966 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:45:30.966 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:45:30.966 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:45:30.966 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:45:30.966 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:45:30.967 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:45:30.968 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:45:30.968 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:45:31.023 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 32 2024-04-06T09:45:31.024 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 33 2024-04-06T09:45:31.024 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:45:31.024 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=133, tag_tid=21, entry_tid=1], [object_number=132, tag_tid=21, entry_tid=0], [object_number=135, tag_tid=20, entry_tid=19939], [object_number=134, tag_tid=20, entry_tid=19938]]], state=connected] 2024-04-06T09:45:31.024 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=128, tag_tid=20, entry_tid=17444], [object_number=131, tag_tid=20, entry_tid=17443], [object_number=130, tag_tid=20, entry_tid=17442], [object_number=129, tag_tid=20, entry_tid=17441]]], state=connected] 2024-04-06T09:45:31.027 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:45:31.028 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=133, tag_tid=21, entry_tid=1' 2024-04-06T09:45:31.029 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:45:31.030 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:45:31.030 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:45:31.030 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:45:31.030 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:45:31.030 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:45:31.030 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:45:31.030 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:45:31.030 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:45:31.030 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:45:31.031 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:45:31.031 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:45:31.031 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:45:31.032 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:45:31.032 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:45:31.078 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 32 2024-04-06T09:45:31.078 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 33 2024-04-06T09:45:31.078 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:45:31.078 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=133, tag_tid=21, entry_tid=1], [object_number=132, tag_tid=21, entry_tid=0], [object_number=135, tag_tid=20, entry_tid=19939], [object_number=134, tag_tid=20, entry_tid=19938]]], state=connected] 2024-04-06T09:45:31.078 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=128, tag_tid=20, entry_tid=17444], [object_number=131, tag_tid=20, entry_tid=17443], [object_number=130, tag_tid=20, entry_tid=17442], [object_number=129, tag_tid=20, entry_tid=17441]]], state=connected] 2024-04-06T09:45:31.082 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:45:31.083 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=128, tag_tid=20, entry_tid=17444' 2024-04-06T09:45:31.084 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=133, tag_tid=21, entry_tid=1' -a 'object_number=133, tag_tid=21, entry_tid=1' = 'object_number=128, tag_tid=20, entry_tid=17444' 2024-04-06T09:45:31.084 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=128, tag_tid=20, entry_tid=17444' '!=' 'object_number=129, tag_tid=20, entry_tid=17353' 2024-04-06T09:45:31.084 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:45:31.084 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=128, tag_tid=20, entry_tid=17444' = 'object_number=129, tag_tid=20, entry_tid=17353' 2024-04-06T09:45:31.084 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=128, tag_tid=20, entry_tid=17444' 2024-04-06T09:45:31.084 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=133, tag_tid=21, entry_tid=1' 2024-04-06T09:45:31.085 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:45:31.085 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:45:31.085 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=21 2024-04-06T09:45:31.086 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=128, tag_tid=20, entry_tid=17444' 2024-04-06T09:45:31.086 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:45:31.086 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:45:31.087 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=20 2024-04-06T09:45:31.087 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=133, tag_tid=21, entry_tid=1' 2024-04-06T09:45:31.088 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:45:31.088 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:45:31.088 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:45:31.089 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=128, tag_tid=20, entry_tid=17444' 2024-04-06T09:45:31.089 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:45:31.089 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:45:31.090 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=17444 2024-04-06T09:45:31.090 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 21 = 20 -a 1 -le 17444 2024-04-06T09:45:31.090 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:45:31.091 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:45:31.091 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:45:31.292 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:45:31.293 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:45:31.293 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:45:31.293 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:45:31.293 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:45:31.293 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:45:31.293 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:45:31.293 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:45:31.293 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:45:31.293 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:45:31.293 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:45:31.294 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:45:31.294 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:45:31.294 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:45:31.294 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:45:31.296 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:45:31.296 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:45:31.296 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:45:31.296 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:45:31.296 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:45:31.297 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:45:31.297 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:45:31.297 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:45:31.297 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:45:31.297 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:45:31.297 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:45:31.297 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:45:31.297 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:45:31.297 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:45:31.297 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:45:31.297 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:45:31.297 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:45:31.297 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:45:31.299 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:45:31.299 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:45:31.299 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:45:31.299 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:45:31.299 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:45:31.299 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:45:31.299 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:45:31.299 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:45:31.299 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:45:31.299 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:45:31.299 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:45:31.299 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:45:31.299 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:45:31.299 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:45:31.300 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:45:31.300 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:45:31.325 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:45:31.325 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:45:31.325 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:45:31.325 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:45:31.455 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:45:31.456 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:45:31.456 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:45:31.456 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:45:31.456 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:45:31.456 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:45:31.456 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:45:31.457 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:45:31.457 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:45:31.457 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:45:31.457 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:45:31.457 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:45:31.457 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:45:31.459 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:45:31.459 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:45:31.459 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:45:31.505 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 32 2024-04-06T09:45:31.505 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 33 2024-04-06T09:45:31.505 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:45:31.506 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=133, tag_tid=21, entry_tid=1], [object_number=132, tag_tid=21, entry_tid=0], [object_number=135, tag_tid=20, entry_tid=19939], [object_number=134, tag_tid=20, entry_tid=19938]]], state=connected] 2024-04-06T09:45:31.506 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=131, tag_tid=20, entry_tid=17539], [object_number=130, tag_tid=20, entry_tid=17538], [object_number=129, tag_tid=20, entry_tid=17537], [object_number=128, tag_tid=20, entry_tid=17536]]], state=connected] 2024-04-06T09:45:31.509 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:45:31.510 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=133, tag_tid=21, entry_tid=1' 2024-04-06T09:45:31.511 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:45:31.511 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:45:31.512 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:45:31.512 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:45:31.512 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:45:31.512 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:45:31.512 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:45:31.512 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:45:31.512 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:45:31.512 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:45:31.512 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:45:31.512 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:45:31.513 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:45:31.513 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:45:31.513 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:45:31.559 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 32 2024-04-06T09:45:31.559 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 33 2024-04-06T09:45:31.559 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:45:31.560 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=133, tag_tid=21, entry_tid=1], [object_number=132, tag_tid=21, entry_tid=0], [object_number=135, tag_tid=20, entry_tid=19939], [object_number=134, tag_tid=20, entry_tid=19938]]], state=connected] 2024-04-06T09:45:31.560 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=131, tag_tid=20, entry_tid=17539], [object_number=130, tag_tid=20, entry_tid=17538], [object_number=129, tag_tid=20, entry_tid=17537], [object_number=128, tag_tid=20, entry_tid=17536]]], state=connected] 2024-04-06T09:45:31.563 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:45:31.564 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=131, tag_tid=20, entry_tid=17539' 2024-04-06T09:45:31.564 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=133, tag_tid=21, entry_tid=1' -a 'object_number=133, tag_tid=21, entry_tid=1' = 'object_number=131, tag_tid=20, entry_tid=17539' 2024-04-06T09:45:31.564 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=131, tag_tid=20, entry_tid=17539' '!=' 'object_number=128, tag_tid=20, entry_tid=17444' 2024-04-06T09:45:31.565 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:45:31.565 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=131, tag_tid=20, entry_tid=17539' = 'object_number=128, tag_tid=20, entry_tid=17444' 2024-04-06T09:45:31.565 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=131, tag_tid=20, entry_tid=17539' 2024-04-06T09:45:31.565 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=133, tag_tid=21, entry_tid=1' 2024-04-06T09:45:31.565 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:45:31.565 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:45:31.566 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=21 2024-04-06T09:45:31.566 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=131, tag_tid=20, entry_tid=17539' 2024-04-06T09:45:31.567 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:45:31.567 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:45:31.567 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=20 2024-04-06T09:45:31.568 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=133, tag_tid=21, entry_tid=1' 2024-04-06T09:45:31.568 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:45:31.568 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:45:31.569 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:45:31.570 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=131, tag_tid=20, entry_tid=17539' 2024-04-06T09:45:31.570 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:45:31.570 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:45:31.571 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=17539 2024-04-06T09:45:31.571 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 21 = 20 -a 1 -le 17539 2024-04-06T09:45:31.571 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:45:31.571 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:45:31.571 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:45:31.773 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:45:31.773 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:45:31.773 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:45:31.774 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:45:31.774 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:45:31.774 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:45:31.774 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:45:31.774 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:45:31.774 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:45:31.774 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:45:31.774 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:45:31.774 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:45:31.774 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:45:31.774 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:45:31.774 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:45:31.775 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:45:31.775 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:45:31.779 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:45:31.779 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:45:31.779 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:45:31.779 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:45:31.779 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:45:31.779 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:45:31.779 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:45:31.779 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:45:31.779 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:45:31.779 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:45:31.779 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:45:31.779 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:45:31.779 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:45:31.780 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:45:31.780 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:45:31.780 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:45:31.780 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:45:31.780 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:45:31.780 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:45:31.780 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:45:31.780 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:45:31.780 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:45:31.780 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:45:31.780 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:45:31.781 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:45:31.781 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:45:31.781 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:45:31.781 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:45:31.781 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:45:31.781 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:45:31.781 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:45:31.781 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:45:31.805 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:45:31.806 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:45:31.806 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:45:31.806 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:45:31.938 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:45:31.939 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:45:31.939 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:45:31.939 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:45:31.939 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:45:31.939 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:45:31.939 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:45:31.939 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:45:31.939 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:45:31.939 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:45:31.939 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:45:31.940 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:45:31.940 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:45:31.941 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:45:31.941 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:45:31.941 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:45:32.000 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 32 2024-04-06T09:45:32.000 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 33 2024-04-06T09:45:32.000 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:45:32.000 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=133, tag_tid=21, entry_tid=1], [object_number=132, tag_tid=21, entry_tid=0], [object_number=135, tag_tid=20, entry_tid=19939], [object_number=134, tag_tid=20, entry_tid=19938]]], state=connected] 2024-04-06T09:45:32.000 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=129, tag_tid=20, entry_tid=17629], [object_number=128, tag_tid=20, entry_tid=17628], [object_number=131, tag_tid=20, entry_tid=17627], [object_number=130, tag_tid=20, entry_tid=17626]]], state=connected] 2024-04-06T09:45:32.003 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:45:32.051 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=133, tag_tid=21, entry_tid=1' 2024-04-06T09:45:32.053 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:45:32.053 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:45:32.053 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:45:32.053 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:45:32.053 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:45:32.053 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:45:32.053 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:45:32.053 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:45:32.054 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:45:32.054 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:45:32.054 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:45:32.054 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:45:32.054 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:45:32.055 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:45:32.055 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:45:32.111 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 32 2024-04-06T09:45:32.111 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 33 2024-04-06T09:45:32.111 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:45:32.111 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=133, tag_tid=21, entry_tid=1], [object_number=132, tag_tid=21, entry_tid=0], [object_number=135, tag_tid=20, entry_tid=19939], [object_number=134, tag_tid=20, entry_tid=19938]]], state=connected] 2024-04-06T09:45:32.111 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=129, tag_tid=20, entry_tid=17629], [object_number=128, tag_tid=20, entry_tid=17628], [object_number=131, tag_tid=20, entry_tid=17627], [object_number=130, tag_tid=20, entry_tid=17626]]], state=connected] 2024-04-06T09:45:32.115 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:45:32.117 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=129, tag_tid=20, entry_tid=17629' 2024-04-06T09:45:32.117 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=133, tag_tid=21, entry_tid=1' -a 'object_number=133, tag_tid=21, entry_tid=1' = 'object_number=129, tag_tid=20, entry_tid=17629' 2024-04-06T09:45:32.117 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=129, tag_tid=20, entry_tid=17629' '!=' 'object_number=131, tag_tid=20, entry_tid=17539' 2024-04-06T09:45:32.117 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:45:32.117 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=129, tag_tid=20, entry_tid=17629' = 'object_number=131, tag_tid=20, entry_tid=17539' 2024-04-06T09:45:32.117 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=129, tag_tid=20, entry_tid=17629' 2024-04-06T09:45:32.118 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=133, tag_tid=21, entry_tid=1' 2024-04-06T09:45:32.118 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:45:32.118 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:45:32.118 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=21 2024-04-06T09:45:32.119 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=129, tag_tid=20, entry_tid=17629' 2024-04-06T09:45:32.119 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:45:32.119 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:45:32.120 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=20 2024-04-06T09:45:32.121 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=133, tag_tid=21, entry_tid=1' 2024-04-06T09:45:32.121 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:45:32.121 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:45:32.121 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:45:32.122 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=129, tag_tid=20, entry_tid=17629' 2024-04-06T09:45:32.123 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:45:32.123 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:45:32.123 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=17629 2024-04-06T09:45:32.124 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 21 = 20 -a 1 -le 17629 2024-04-06T09:45:32.124 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:45:32.124 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:45:32.124 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:45:32.325 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:45:32.326 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:45:32.326 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:45:32.326 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:45:32.326 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:45:32.326 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:45:32.326 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:45:32.326 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:45:32.326 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:45:32.326 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:45:32.326 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:45:32.326 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:45:32.326 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:45:32.327 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:45:32.327 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:45:32.327 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:45:32.327 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:45:32.330 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:45:32.331 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:45:32.331 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:45:32.331 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:45:32.331 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:45:32.331 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:45:32.331 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:45:32.331 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:45:32.331 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:45:32.331 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:45:32.331 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:45:32.331 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:45:32.332 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:45:32.332 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:45:32.332 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:45:32.332 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:45:32.332 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:45:32.332 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:45:32.332 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:45:32.332 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:45:32.332 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:45:32.332 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:45:32.332 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:45:32.332 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:45:32.333 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:45:32.333 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:45:32.333 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:45:32.333 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:45:32.333 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:45:32.333 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:45:32.333 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:45:32.333 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:45:32.355 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:45:32.356 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:45:32.356 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:45:32.356 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:45:32.494 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:45:32.495 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:45:32.495 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:45:32.495 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:45:32.495 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:45:32.495 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:45:32.495 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:45:32.495 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:45:32.496 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:45:32.496 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:45:32.496 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:45:32.496 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:45:32.496 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:45:32.498 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:45:32.498 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:45:32.498 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:45:32.547 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 32 2024-04-06T09:45:32.547 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 33 2024-04-06T09:45:32.547 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:45:32.547 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=133, tag_tid=21, entry_tid=1], [object_number=132, tag_tid=21, entry_tid=0], [object_number=135, tag_tid=20, entry_tid=19939], [object_number=134, tag_tid=20, entry_tid=19938]]], state=connected] 2024-04-06T09:45:32.547 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=129, tag_tid=20, entry_tid=17725], [object_number=128, tag_tid=20, entry_tid=17724], [object_number=131, tag_tid=20, entry_tid=17723], [object_number=130, tag_tid=20, entry_tid=17722]]], state=connected] 2024-04-06T09:45:32.550 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:45:32.551 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=133, tag_tid=21, entry_tid=1' 2024-04-06T09:45:32.553 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:45:32.553 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:45:32.553 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:45:32.553 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:45:32.553 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:45:32.553 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:45:32.553 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:45:32.553 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:45:32.553 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:45:32.553 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:45:32.553 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:45:32.554 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:45:32.554 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:45:32.554 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:45:32.554 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:45:32.600 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 32 2024-04-06T09:45:32.600 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 33 2024-04-06T09:45:32.600 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:45:32.600 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=133, tag_tid=21, entry_tid=1], [object_number=132, tag_tid=21, entry_tid=0], [object_number=135, tag_tid=20, entry_tid=19939], [object_number=134, tag_tid=20, entry_tid=19938]]], state=connected] 2024-04-06T09:45:32.600 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=129, tag_tid=20, entry_tid=17725], [object_number=128, tag_tid=20, entry_tid=17724], [object_number=131, tag_tid=20, entry_tid=17723], [object_number=130, tag_tid=20, entry_tid=17722]]], state=connected] 2024-04-06T09:45:32.604 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:45:32.606 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=129, tag_tid=20, entry_tid=17725' 2024-04-06T09:45:32.606 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=133, tag_tid=21, entry_tid=1' -a 'object_number=133, tag_tid=21, entry_tid=1' = 'object_number=129, tag_tid=20, entry_tid=17725' 2024-04-06T09:45:32.606 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=129, tag_tid=20, entry_tid=17725' '!=' 'object_number=129, tag_tid=20, entry_tid=17629' 2024-04-06T09:45:32.606 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:45:32.606 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=129, tag_tid=20, entry_tid=17725' = 'object_number=129, tag_tid=20, entry_tid=17629' 2024-04-06T09:45:32.607 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=129, tag_tid=20, entry_tid=17725' 2024-04-06T09:45:32.607 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=133, tag_tid=21, entry_tid=1' 2024-04-06T09:45:32.607 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:45:32.607 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:45:32.608 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=21 2024-04-06T09:45:32.608 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=129, tag_tid=20, entry_tid=17725' 2024-04-06T09:45:32.608 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:45:32.608 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:45:32.608 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=20 2024-04-06T09:45:32.609 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=133, tag_tid=21, entry_tid=1' 2024-04-06T09:45:32.609 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:45:32.610 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:45:32.610 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:45:32.611 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=129, tag_tid=20, entry_tid=17725' 2024-04-06T09:45:32.611 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:45:32.611 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:45:32.612 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=17725 2024-04-06T09:45:32.612 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 21 = 20 -a 1 -le 17725 2024-04-06T09:45:32.612 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:45:32.612 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:45:32.612 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:45:32.814 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:45:32.814 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:45:32.814 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:45:32.814 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:45:32.814 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:45:32.814 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:45:32.814 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:45:32.815 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:45:32.815 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:45:32.815 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:45:32.815 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:45:32.815 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:45:32.815 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:45:32.815 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:45:32.815 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:45:32.816 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:45:32.816 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:45:32.818 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:45:32.818 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:45:32.818 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:45:32.818 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:45:32.818 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:45:32.818 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:45:32.818 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:45:32.818 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:45:32.818 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:45:32.818 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:45:32.819 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:45:32.819 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:45:32.819 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:45:32.819 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:45:32.819 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:45:32.819 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:45:32.820 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:45:32.820 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:45:32.821 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:45:32.821 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:45:32.821 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:45:32.821 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:45:32.821 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:45:32.821 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:45:32.821 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:45:32.821 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:45:32.821 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:45:32.821 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:45:32.821 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:45:32.822 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:45:32.822 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:45:32.822 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:45:32.845 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:45:32.845 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:45:32.845 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:45:32.846 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:45:32.978 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:45:32.979 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:45:32.979 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:45:32.979 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:45:32.979 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:45:32.979 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:45:32.979 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:45:32.980 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:45:32.980 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:45:32.980 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:45:32.980 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:45:32.981 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:45:32.981 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:45:32.981 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:45:32.981 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:45:32.981 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:45:33.033 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 32 2024-04-06T09:45:33.033 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 33 2024-04-06T09:45:33.033 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:45:33.033 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=133, tag_tid=21, entry_tid=1], [object_number=132, tag_tid=21, entry_tid=0], [object_number=135, tag_tid=20, entry_tid=19939], [object_number=134, tag_tid=20, entry_tid=19938]]], state=connected] 2024-04-06T09:45:33.033 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=129, tag_tid=20, entry_tid=17809], [object_number=128, tag_tid=20, entry_tid=17808], [object_number=131, tag_tid=20, entry_tid=17807], [object_number=130, tag_tid=20, entry_tid=17806]]], state=connected] 2024-04-06T09:45:33.037 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:45:33.038 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=133, tag_tid=21, entry_tid=1' 2024-04-06T09:45:33.039 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:45:33.040 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:45:33.040 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:45:33.040 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:45:33.040 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:45:33.040 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:45:33.040 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:45:33.040 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:45:33.040 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:45:33.040 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:45:33.040 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:45:33.040 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:45:33.041 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:45:33.041 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:45:33.041 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:45:33.087 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 32 2024-04-06T09:45:33.088 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 33 2024-04-06T09:45:33.088 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:45:33.088 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=133, tag_tid=21, entry_tid=1], [object_number=132, tag_tid=21, entry_tid=0], [object_number=135, tag_tid=20, entry_tid=19939], [object_number=134, tag_tid=20, entry_tid=19938]]], state=connected] 2024-04-06T09:45:33.088 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=129, tag_tid=20, entry_tid=17809], [object_number=128, tag_tid=20, entry_tid=17808], [object_number=131, tag_tid=20, entry_tid=17807], [object_number=130, tag_tid=20, entry_tid=17806]]], state=connected] 2024-04-06T09:45:33.091 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:45:33.093 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=129, tag_tid=20, entry_tid=17809' 2024-04-06T09:45:33.094 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=133, tag_tid=21, entry_tid=1' -a 'object_number=133, tag_tid=21, entry_tid=1' = 'object_number=129, tag_tid=20, entry_tid=17809' 2024-04-06T09:45:33.094 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=129, tag_tid=20, entry_tid=17809' '!=' 'object_number=129, tag_tid=20, entry_tid=17725' 2024-04-06T09:45:33.094 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:45:33.094 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=129, tag_tid=20, entry_tid=17809' = 'object_number=129, tag_tid=20, entry_tid=17725' 2024-04-06T09:45:33.094 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=129, tag_tid=20, entry_tid=17809' 2024-04-06T09:45:33.094 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=133, tag_tid=21, entry_tid=1' 2024-04-06T09:45:33.094 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:45:33.094 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:45:33.095 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=21 2024-04-06T09:45:33.095 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=129, tag_tid=20, entry_tid=17809' 2024-04-06T09:45:33.096 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:45:33.096 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:45:33.096 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=20 2024-04-06T09:45:33.097 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=133, tag_tid=21, entry_tid=1' 2024-04-06T09:45:33.097 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:45:33.097 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:45:33.098 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:45:33.099 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=129, tag_tid=20, entry_tid=17809' 2024-04-06T09:45:33.099 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:45:33.099 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:45:33.101 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=17809 2024-04-06T09:45:33.102 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 21 = 20 -a 1 -le 17809 2024-04-06T09:45:33.102 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:45:33.102 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:45:33.102 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:45:33.302 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:45:33.302 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:45:33.302 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:45:33.302 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:45:33.302 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:45:33.302 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:45:33.303 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:45:33.303 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:45:33.303 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:45:33.303 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:45:33.303 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:45:33.303 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:45:33.303 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:45:33.303 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:45:33.303 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:45:33.308 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:45:33.308 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:45:33.308 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:45:33.308 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:45:33.308 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:45:33.308 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:45:33.308 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:45:33.308 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:45:33.308 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:45:33.309 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:45:33.309 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:45:33.309 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:45:33.309 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:45:33.309 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:45:33.309 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:45:33.309 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:45:33.309 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:45:33.309 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:45:33.309 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:45:33.309 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:45:33.309 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:45:33.310 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:45:33.310 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:45:33.310 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:45:33.310 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:45:33.310 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:45:33.310 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:45:33.310 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:45:33.310 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:45:33.310 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:45:33.310 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:45:33.310 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:45:33.310 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:45:33.311 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:45:33.334 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:45:33.334 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:45:33.334 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:45:33.334 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:45:33.464 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:45:33.465 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:45:33.466 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:45:33.466 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:45:33.466 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:45:33.466 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:45:33.466 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:45:33.466 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:45:33.466 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:45:33.466 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:45:33.467 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:45:33.467 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:45:33.467 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:45:33.468 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:45:33.469 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:45:33.469 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:45:33.522 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 32 2024-04-06T09:45:33.522 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 33 2024-04-06T09:45:33.522 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:45:33.522 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=133, tag_tid=21, entry_tid=1], [object_number=132, tag_tid=21, entry_tid=0], [object_number=135, tag_tid=20, entry_tid=19939], [object_number=134, tag_tid=20, entry_tid=19938]]], state=connected] 2024-04-06T09:45:33.522 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=130, tag_tid=20, entry_tid=17886], [object_number=129, tag_tid=20, entry_tid=17885], [object_number=128, tag_tid=20, entry_tid=17884], [object_number=131, tag_tid=20, entry_tid=17883]]], state=connected] 2024-04-06T09:45:33.526 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:45:33.527 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=133, tag_tid=21, entry_tid=1' 2024-04-06T09:45:33.528 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:45:33.528 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:45:33.528 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:45:33.528 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:45:33.528 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:45:33.528 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:45:33.528 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:45:33.528 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:45:33.528 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:45:33.528 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:45:33.529 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:45:33.529 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:45:33.529 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:45:33.529 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:45:33.529 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:45:33.577 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 32 2024-04-06T09:45:33.577 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 33 2024-04-06T09:45:33.577 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:45:33.577 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=133, tag_tid=21, entry_tid=1], [object_number=132, tag_tid=21, entry_tid=0], [object_number=135, tag_tid=20, entry_tid=19939], [object_number=134, tag_tid=20, entry_tid=19938]]], state=connected] 2024-04-06T09:45:33.577 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=130, tag_tid=20, entry_tid=17886], [object_number=129, tag_tid=20, entry_tid=17885], [object_number=128, tag_tid=20, entry_tid=17884], [object_number=131, tag_tid=20, entry_tid=17883]]], state=connected] 2024-04-06T09:45:33.580 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:45:33.582 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=130, tag_tid=20, entry_tid=17886' 2024-04-06T09:45:33.582 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=133, tag_tid=21, entry_tid=1' -a 'object_number=133, tag_tid=21, entry_tid=1' = 'object_number=130, tag_tid=20, entry_tid=17886' 2024-04-06T09:45:33.582 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=130, tag_tid=20, entry_tid=17886' '!=' 'object_number=129, tag_tid=20, entry_tid=17809' 2024-04-06T09:45:33.582 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:45:33.582 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=130, tag_tid=20, entry_tid=17886' = 'object_number=129, tag_tid=20, entry_tid=17809' 2024-04-06T09:45:33.583 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=130, tag_tid=20, entry_tid=17886' 2024-04-06T09:45:33.583 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=133, tag_tid=21, entry_tid=1' 2024-04-06T09:45:33.583 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:45:33.583 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:45:33.583 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=21 2024-04-06T09:45:33.584 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=130, tag_tid=20, entry_tid=17886' 2024-04-06T09:45:33.584 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:45:33.584 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:45:33.585 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=20 2024-04-06T09:45:33.586 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=133, tag_tid=21, entry_tid=1' 2024-04-06T09:45:33.586 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:45:33.586 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:45:33.586 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:45:33.587 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=130, tag_tid=20, entry_tid=17886' 2024-04-06T09:45:33.587 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:45:33.587 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:45:33.588 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=17886 2024-04-06T09:45:33.588 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 21 = 20 -a 1 -le 17886 2024-04-06T09:45:33.589 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:45:33.589 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:45:33.589 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:45:33.790 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:45:33.790 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:45:33.790 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:45:33.790 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:45:33.790 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:45:33.791 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:45:33.791 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:45:33.791 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:45:33.791 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:45:33.791 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:45:33.791 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:45:33.791 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:45:33.791 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:45:33.791 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:45:33.791 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:45:33.792 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:45:33.792 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:45:33.796 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:45:33.796 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:45:33.796 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:45:33.796 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:45:33.796 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:45:33.796 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:45:33.796 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:45:33.796 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:45:33.796 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:45:33.796 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:45:33.797 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:45:33.797 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:45:33.797 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:45:33.797 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:45:33.797 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:45:33.797 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:45:33.797 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:45:33.797 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:45:33.797 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:45:33.797 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:45:33.797 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:45:33.797 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:45:33.797 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:45:33.797 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:45:33.797 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:45:33.798 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:45:33.798 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:45:33.798 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:45:33.798 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:45:33.798 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:45:33.798 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:45:33.798 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:45:33.822 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:45:33.823 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:45:33.823 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:45:33.823 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:45:33.950 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:45:33.951 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:45:33.952 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:45:33.952 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:45:33.952 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:45:33.952 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:45:33.952 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:45:33.952 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:45:33.952 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:45:33.952 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:45:33.953 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:45:33.953 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:45:33.953 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:45:33.955 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:45:33.955 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:45:33.955 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:45:34.009 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 32 2024-04-06T09:45:34.010 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 33 2024-04-06T09:45:34.010 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:45:34.010 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=133, tag_tid=21, entry_tid=1], [object_number=132, tag_tid=21, entry_tid=0], [object_number=135, tag_tid=20, entry_tid=19939], [object_number=134, tag_tid=20, entry_tid=19938]]], state=connected] 2024-04-06T09:45:34.010 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=130, tag_tid=20, entry_tid=17966], [object_number=129, tag_tid=20, entry_tid=17965], [object_number=128, tag_tid=20, entry_tid=17964], [object_number=131, tag_tid=20, entry_tid=17963]]], state=connected] 2024-04-06T09:45:34.013 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:45:34.015 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=133, tag_tid=21, entry_tid=1' 2024-04-06T09:45:34.016 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:45:34.016 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:45:34.016 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:45:34.016 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:45:34.016 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:45:34.016 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:45:34.016 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:45:34.016 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:45:34.016 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:45:34.016 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:45:34.016 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:45:34.017 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:45:34.017 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:45:34.017 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:45:34.017 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:45:34.064 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 32 2024-04-06T09:45:34.064 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 33 2024-04-06T09:45:34.064 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:45:34.064 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=133, tag_tid=21, entry_tid=1], [object_number=132, tag_tid=21, entry_tid=0], [object_number=135, tag_tid=20, entry_tid=19939], [object_number=134, tag_tid=20, entry_tid=19938]]], state=connected] 2024-04-06T09:45:34.065 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=130, tag_tid=20, entry_tid=17966], [object_number=129, tag_tid=20, entry_tid=17965], [object_number=128, tag_tid=20, entry_tid=17964], [object_number=131, tag_tid=20, entry_tid=17963]]], state=connected] 2024-04-06T09:45:34.068 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:45:34.070 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=130, tag_tid=20, entry_tid=17966' 2024-04-06T09:45:34.070 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=133, tag_tid=21, entry_tid=1' -a 'object_number=133, tag_tid=21, entry_tid=1' = 'object_number=130, tag_tid=20, entry_tid=17966' 2024-04-06T09:45:34.070 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=130, tag_tid=20, entry_tid=17966' '!=' 'object_number=130, tag_tid=20, entry_tid=17886' 2024-04-06T09:45:34.070 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:45:34.070 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=130, tag_tid=20, entry_tid=17966' = 'object_number=130, tag_tid=20, entry_tid=17886' 2024-04-06T09:45:34.070 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=130, tag_tid=20, entry_tid=17966' 2024-04-06T09:45:34.070 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=133, tag_tid=21, entry_tid=1' 2024-04-06T09:45:34.070 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:45:34.070 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:45:34.071 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=21 2024-04-06T09:45:34.072 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=130, tag_tid=20, entry_tid=17966' 2024-04-06T09:45:34.072 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:45:34.072 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:45:34.073 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=20 2024-04-06T09:45:34.073 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=133, tag_tid=21, entry_tid=1' 2024-04-06T09:45:34.073 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:45:34.073 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:45:34.074 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:45:34.075 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=130, tag_tid=20, entry_tid=17966' 2024-04-06T09:45:34.075 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:45:34.075 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:45:34.076 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=17966 2024-04-06T09:45:34.076 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 21 = 20 -a 1 -le 17966 2024-04-06T09:45:34.076 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:45:34.077 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:45:34.077 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:45:34.278 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:45:34.278 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:45:34.278 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:45:34.278 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:45:34.279 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:45:34.279 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:45:34.279 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:45:34.279 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:45:34.279 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:45:34.279 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:45:34.279 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:45:34.279 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:45:34.279 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:45:34.279 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:45:34.279 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:45:34.280 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:45:34.280 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:45:34.283 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:45:34.284 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:45:34.284 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:45:34.284 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:45:34.284 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:45:34.284 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:45:34.284 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:45:34.284 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:45:34.284 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:45:34.284 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:45:34.284 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:45:34.284 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:45:34.284 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:45:34.284 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:45:34.285 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:45:34.285 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:45:34.285 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:45:34.285 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:45:34.285 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:45:34.285 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:45:34.285 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:45:34.285 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:45:34.285 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:45:34.285 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:45:34.285 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:45:34.285 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:45:34.285 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:45:34.285 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:45:34.285 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:45:34.286 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:45:34.286 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:45:34.286 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:45:34.310 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:45:34.310 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:45:34.310 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:45:34.310 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:45:34.444 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:45:34.445 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:45:34.446 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:45:34.446 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:45:34.446 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:45:34.446 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:45:34.446 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:45:34.446 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:45:34.446 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:45:34.446 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:45:34.446 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:45:34.447 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:45:34.447 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:45:34.447 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:45:34.448 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:45:34.448 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:45:34.504 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 32 2024-04-06T09:45:34.504 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 33 2024-04-06T09:45:34.504 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:45:34.504 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=133, tag_tid=21, entry_tid=1], [object_number=132, tag_tid=21, entry_tid=0], [object_number=135, tag_tid=20, entry_tid=19939], [object_number=134, tag_tid=20, entry_tid=19938]]], state=connected] 2024-04-06T09:45:34.504 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=131, tag_tid=20, entry_tid=18047], [object_number=130, tag_tid=20, entry_tid=18046], [object_number=129, tag_tid=20, entry_tid=18045], [object_number=128, tag_tid=20, entry_tid=18044]]], state=connected] 2024-04-06T09:45:34.508 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:45:34.509 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=133, tag_tid=21, entry_tid=1' 2024-04-06T09:45:34.510 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:45:34.510 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:45:34.510 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:45:34.510 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:45:34.510 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:45:34.510 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:45:34.511 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:45:34.511 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:45:34.511 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:45:34.511 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:45:34.511 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:45:34.511 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:45:34.511 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:45:34.512 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:45:34.512 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:45:34.772 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 32 2024-04-06T09:45:34.772 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 33 2024-04-06T09:45:34.772 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:45:34.773 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=133, tag_tid=21, entry_tid=1], [object_number=132, tag_tid=21, entry_tid=0], [object_number=135, tag_tid=20, entry_tid=19939], [object_number=134, tag_tid=20, entry_tid=19938]]], state=connected] 2024-04-06T09:45:34.773 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=131, tag_tid=20, entry_tid=18047], [object_number=130, tag_tid=20, entry_tid=18046], [object_number=129, tag_tid=20, entry_tid=18045], [object_number=128, tag_tid=20, entry_tid=18044]]], state=connected] 2024-04-06T09:45:34.777 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:45:34.779 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=131, tag_tid=20, entry_tid=18047' 2024-04-06T09:45:34.779 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=133, tag_tid=21, entry_tid=1' -a 'object_number=133, tag_tid=21, entry_tid=1' = 'object_number=131, tag_tid=20, entry_tid=18047' 2024-04-06T09:45:34.779 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=131, tag_tid=20, entry_tid=18047' '!=' 'object_number=130, tag_tid=20, entry_tid=17966' 2024-04-06T09:45:34.779 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:45:34.779 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=131, tag_tid=20, entry_tid=18047' = 'object_number=130, tag_tid=20, entry_tid=17966' 2024-04-06T09:45:34.779 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=131, tag_tid=20, entry_tid=18047' 2024-04-06T09:45:34.780 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=133, tag_tid=21, entry_tid=1' 2024-04-06T09:45:34.780 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:45:34.780 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:45:34.781 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=21 2024-04-06T09:45:34.782 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=131, tag_tid=20, entry_tid=18047' 2024-04-06T09:45:34.782 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:45:34.782 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:45:34.783 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=20 2024-04-06T09:45:34.784 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=133, tag_tid=21, entry_tid=1' 2024-04-06T09:45:34.784 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:45:34.784 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:45:34.785 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:45:34.786 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=131, tag_tid=20, entry_tid=18047' 2024-04-06T09:45:34.786 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:45:34.786 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:45:34.787 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=18047 2024-04-06T09:45:34.787 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 21 = 20 -a 1 -le 18047 2024-04-06T09:45:34.787 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:45:34.787 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:45:34.788 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:45:34.989 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:45:34.989 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:45:34.989 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:45:34.989 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:45:34.990 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:45:34.990 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:45:34.990 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:45:34.990 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:45:34.990 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:45:34.990 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:45:34.990 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:45:34.990 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:45:34.990 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:45:34.990 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:45:34.990 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:45:34.991 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:45:34.991 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:45:34.993 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:45:34.993 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:45:34.993 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:45:34.993 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:45:34.993 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:45:34.994 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:45:34.994 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:45:34.994 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:45:34.994 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:45:34.994 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:45:34.994 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:45:34.994 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:45:34.994 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:45:34.994 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:45:34.994 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:45:34.994 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:45:34.995 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:45:34.995 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:45:34.996 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:45:34.996 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:45:34.996 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:45:34.996 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:45:34.996 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:45:34.996 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:45:34.996 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:45:34.996 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:45:34.996 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:45:34.996 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:45:34.996 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:45:34.996 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:45:34.996 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:45:34.996 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:45:35.020 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:45:35.020 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:45:35.021 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:45:35.021 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:45:35.150 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:45:35.150 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:45:35.150 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:45:35.150 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:45:35.150 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:45:35.150 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:45:35.150 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:45:35.150 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:45:35.151 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:45:35.151 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:45:35.151 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:45:35.151 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:45:35.151 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:45:35.152 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:45:35.153 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:45:35.153 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:45:35.210 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 32 2024-04-06T09:45:35.210 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 33 2024-04-06T09:45:35.210 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:45:35.210 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=133, tag_tid=21, entry_tid=1], [object_number=132, tag_tid=21, entry_tid=0], [object_number=135, tag_tid=20, entry_tid=19939], [object_number=134, tag_tid=20, entry_tid=19938]]], state=connected] 2024-04-06T09:45:35.210 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=130, tag_tid=20, entry_tid=18178], [object_number=129, tag_tid=20, entry_tid=18177], [object_number=128, tag_tid=20, entry_tid=18176], [object_number=131, tag_tid=20, entry_tid=18175]]], state=connected] 2024-04-06T09:45:35.214 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:45:35.215 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=133, tag_tid=21, entry_tid=1' 2024-04-06T09:45:35.216 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:45:35.216 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:45:35.216 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:45:35.216 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:45:35.216 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:45:35.216 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:45:35.216 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:45:35.217 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:45:35.217 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:45:35.217 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:45:35.217 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:45:35.217 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:45:35.217 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:45:35.218 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:45:35.218 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:45:35.265 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 32 2024-04-06T09:45:35.265 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 33 2024-04-06T09:45:35.265 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:45:35.265 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=133, tag_tid=21, entry_tid=1], [object_number=132, tag_tid=21, entry_tid=0], [object_number=135, tag_tid=20, entry_tid=19939], [object_number=134, tag_tid=20, entry_tid=19938]]], state=connected] 2024-04-06T09:45:35.265 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=130, tag_tid=20, entry_tid=18178], [object_number=129, tag_tid=20, entry_tid=18177], [object_number=128, tag_tid=20, entry_tid=18176], [object_number=131, tag_tid=20, entry_tid=18175]]], state=connected] 2024-04-06T09:45:35.269 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:45:35.271 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=130, tag_tid=20, entry_tid=18178' 2024-04-06T09:45:35.271 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=133, tag_tid=21, entry_tid=1' -a 'object_number=133, tag_tid=21, entry_tid=1' = 'object_number=130, tag_tid=20, entry_tid=18178' 2024-04-06T09:45:35.271 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=130, tag_tid=20, entry_tid=18178' '!=' 'object_number=131, tag_tid=20, entry_tid=18047' 2024-04-06T09:45:35.271 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:45:35.271 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=130, tag_tid=20, entry_tid=18178' = 'object_number=131, tag_tid=20, entry_tid=18047' 2024-04-06T09:45:35.271 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=130, tag_tid=20, entry_tid=18178' 2024-04-06T09:45:35.271 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=133, tag_tid=21, entry_tid=1' 2024-04-06T09:45:35.271 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:45:35.271 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:45:35.272 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=21 2024-04-06T09:45:35.272 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=130, tag_tid=20, entry_tid=18178' 2024-04-06T09:45:35.273 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:45:35.273 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:45:35.273 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=20 2024-04-06T09:45:35.274 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=133, tag_tid=21, entry_tid=1' 2024-04-06T09:45:35.274 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:45:35.274 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:45:35.275 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:45:35.276 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=130, tag_tid=20, entry_tid=18178' 2024-04-06T09:45:35.276 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:45:35.276 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:45:35.277 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=18178 2024-04-06T09:45:35.277 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 21 = 20 -a 1 -le 18178 2024-04-06T09:45:35.277 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:45:35.277 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:45:35.277 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:45:35.479 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:45:35.479 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:45:35.479 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:45:35.479 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:45:35.479 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:45:35.479 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:45:35.480 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:45:35.480 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:45:35.480 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:45:35.480 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:45:35.480 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:45:35.480 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:45:35.480 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:45:35.480 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:45:35.480 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:45:35.480 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:45:35.481 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:45:35.483 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:45:35.483 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:45:35.484 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:45:35.484 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:45:35.484 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:45:35.484 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:45:35.484 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:45:35.484 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:45:35.484 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:45:35.484 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:45:35.484 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:45:35.484 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:45:35.485 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:45:35.485 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:45:35.485 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:45:35.485 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:45:35.486 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:45:35.486 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:45:35.486 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:45:35.486 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:45:35.486 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:45:35.486 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:45:35.486 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:45:35.486 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:45:35.486 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:45:35.486 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:45:35.487 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:45:35.487 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:45:35.487 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:45:35.487 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:45:35.487 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:45:35.487 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:45:35.513 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:45:35.513 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:45:35.513 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:45:35.513 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:45:35.643 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:45:35.644 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:45:35.644 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:45:35.644 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:45:35.644 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:45:35.644 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:45:35.644 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:45:35.644 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:45:35.644 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:45:35.644 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:45:35.645 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:45:35.645 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:45:35.645 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:45:35.647 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:45:35.647 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:45:35.647 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:45:35.701 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 32 2024-04-06T09:45:35.701 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 33 2024-04-06T09:45:35.701 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:45:35.701 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=133, tag_tid=21, entry_tid=1], [object_number=132, tag_tid=21, entry_tid=0], [object_number=135, tag_tid=20, entry_tid=19939], [object_number=134, tag_tid=20, entry_tid=19938]]], state=connected] 2024-04-06T09:45:35.701 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=128, tag_tid=20, entry_tid=18260], [object_number=131, tag_tid=20, entry_tid=18259], [object_number=130, tag_tid=20, entry_tid=18258], [object_number=129, tag_tid=20, entry_tid=18257]]], state=connected] 2024-04-06T09:45:35.704 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:45:35.706 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=133, tag_tid=21, entry_tid=1' 2024-04-06T09:45:35.707 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:45:35.707 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:45:35.707 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:45:35.707 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:45:35.707 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:45:35.707 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:45:35.707 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:45:35.707 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:45:35.707 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:45:35.707 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:45:35.708 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:45:35.708 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:45:35.708 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:45:35.708 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:45:35.708 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:45:35.757 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 32 2024-04-06T09:45:35.758 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 33 2024-04-06T09:45:35.758 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:45:35.758 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=133, tag_tid=21, entry_tid=1], [object_number=132, tag_tid=21, entry_tid=0], [object_number=135, tag_tid=20, entry_tid=19939], [object_number=134, tag_tid=20, entry_tid=19938]]], state=connected] 2024-04-06T09:45:35.758 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=128, tag_tid=20, entry_tid=18260], [object_number=131, tag_tid=20, entry_tid=18259], [object_number=130, tag_tid=20, entry_tid=18258], [object_number=129, tag_tid=20, entry_tid=18257]]], state=connected] 2024-04-06T09:45:35.761 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:45:35.763 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=128, tag_tid=20, entry_tid=18260' 2024-04-06T09:45:35.763 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=133, tag_tid=21, entry_tid=1' -a 'object_number=133, tag_tid=21, entry_tid=1' = 'object_number=128, tag_tid=20, entry_tid=18260' 2024-04-06T09:45:35.763 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=128, tag_tid=20, entry_tid=18260' '!=' 'object_number=130, tag_tid=20, entry_tid=18178' 2024-04-06T09:45:35.763 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:45:35.763 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=128, tag_tid=20, entry_tid=18260' = 'object_number=130, tag_tid=20, entry_tid=18178' 2024-04-06T09:45:35.763 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=128, tag_tid=20, entry_tid=18260' 2024-04-06T09:45:35.764 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=133, tag_tid=21, entry_tid=1' 2024-04-06T09:45:35.764 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:45:35.764 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:45:35.764 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=21 2024-04-06T09:45:35.765 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=128, tag_tid=20, entry_tid=18260' 2024-04-06T09:45:35.765 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:45:35.765 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:45:35.766 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=20 2024-04-06T09:45:35.767 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=133, tag_tid=21, entry_tid=1' 2024-04-06T09:45:35.767 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:45:35.767 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:45:35.768 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:45:35.768 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=128, tag_tid=20, entry_tid=18260' 2024-04-06T09:45:35.769 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:45:35.769 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:45:35.770 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=18260 2024-04-06T09:45:35.770 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 21 = 20 -a 1 -le 18260 2024-04-06T09:45:35.770 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:45:35.770 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:45:35.770 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:45:35.972 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:45:35.972 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:45:35.972 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:45:35.972 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:45:35.972 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:45:35.972 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:45:35.972 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:45:35.972 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:45:35.972 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:45:35.973 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:45:35.973 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:45:35.973 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:45:35.973 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:45:35.973 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:45:35.973 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:45:35.973 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:45:35.973 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:45:35.975 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:45:35.976 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:45:35.976 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:45:35.976 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:45:35.976 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:45:35.976 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:45:35.976 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:45:35.976 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:45:35.976 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:45:35.976 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:45:35.976 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:45:35.976 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:45:35.976 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:45:35.976 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:45:35.976 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:45:35.977 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:45:35.977 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:45:35.977 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:45:35.977 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:45:35.977 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:45:35.977 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:45:35.977 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:45:35.977 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:45:35.977 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:45:35.977 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:45:35.977 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:45:35.977 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:45:35.977 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:45:35.977 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:45:35.977 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:45:35.977 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:45:35.978 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:45:36.001 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:45:36.002 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:45:36.002 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:45:36.002 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:45:36.131 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:45:36.132 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:45:36.132 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:45:36.132 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:45:36.132 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:45:36.132 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:45:36.132 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:45:36.133 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:45:36.133 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:45:36.133 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:45:36.133 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:45:36.133 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:45:36.133 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:45:36.135 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:45:36.136 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:45:36.136 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:45:36.188 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 32 2024-04-06T09:45:36.188 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 33 2024-04-06T09:45:36.188 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:45:36.188 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=133, tag_tid=21, entry_tid=1], [object_number=132, tag_tid=21, entry_tid=0], [object_number=135, tag_tid=20, entry_tid=19939], [object_number=134, tag_tid=20, entry_tid=19938]]], state=connected] 2024-04-06T09:45:36.188 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=128, tag_tid=20, entry_tid=18364], [object_number=131, tag_tid=20, entry_tid=18363], [object_number=130, tag_tid=20, entry_tid=18362], [object_number=129, tag_tid=20, entry_tid=18361]]], state=connected] 2024-04-06T09:45:36.192 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:45:36.193 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=133, tag_tid=21, entry_tid=1' 2024-04-06T09:45:36.194 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:45:36.194 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:45:36.194 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:45:36.194 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:45:36.194 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:45:36.194 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:45:36.195 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:45:36.195 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:45:36.195 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:45:36.195 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:45:36.195 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:45:36.195 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:45:36.195 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:45:36.196 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:45:36.196 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:45:36.243 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 32 2024-04-06T09:45:36.243 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 33 2024-04-06T09:45:36.243 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:45:36.244 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=133, tag_tid=21, entry_tid=1], [object_number=132, tag_tid=21, entry_tid=0], [object_number=135, tag_tid=20, entry_tid=19939], [object_number=134, tag_tid=20, entry_tid=19938]]], state=connected] 2024-04-06T09:45:36.244 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=128, tag_tid=20, entry_tid=18364], [object_number=131, tag_tid=20, entry_tid=18363], [object_number=130, tag_tid=20, entry_tid=18362], [object_number=129, tag_tid=20, entry_tid=18361]]], state=connected] 2024-04-06T09:45:36.247 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:45:36.249 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=128, tag_tid=20, entry_tid=18364' 2024-04-06T09:45:36.249 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=133, tag_tid=21, entry_tid=1' -a 'object_number=133, tag_tid=21, entry_tid=1' = 'object_number=128, tag_tid=20, entry_tid=18364' 2024-04-06T09:45:36.249 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=128, tag_tid=20, entry_tid=18364' '!=' 'object_number=128, tag_tid=20, entry_tid=18260' 2024-04-06T09:45:36.249 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:45:36.249 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=128, tag_tid=20, entry_tid=18364' = 'object_number=128, tag_tid=20, entry_tid=18260' 2024-04-06T09:45:36.249 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=128, tag_tid=20, entry_tid=18364' 2024-04-06T09:45:36.249 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=133, tag_tid=21, entry_tid=1' 2024-04-06T09:45:36.249 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:45:36.250 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:45:36.250 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=21 2024-04-06T09:45:36.251 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=128, tag_tid=20, entry_tid=18364' 2024-04-06T09:45:36.251 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:45:36.251 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:45:36.252 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=20 2024-04-06T09:45:36.253 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=133, tag_tid=21, entry_tid=1' 2024-04-06T09:45:36.253 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:45:36.253 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:45:36.254 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:45:36.254 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=128, tag_tid=20, entry_tid=18364' 2024-04-06T09:45:36.254 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:45:36.255 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:45:36.255 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=18364 2024-04-06T09:45:36.255 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 21 = 20 -a 1 -le 18364 2024-04-06T09:45:36.255 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:45:36.256 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:45:36.256 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:45:36.457 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:45:36.457 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:45:36.457 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:45:36.457 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:45:36.458 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:45:36.458 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:45:36.458 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:45:36.458 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:45:36.458 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:45:36.458 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:45:36.458 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:45:36.458 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:45:36.458 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:45:36.458 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:45:36.458 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:45:36.459 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:45:36.459 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:45:36.461 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:45:36.461 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:45:36.461 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:45:36.461 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:45:36.462 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:45:36.462 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:45:36.462 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:45:36.462 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:45:36.462 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:45:36.462 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:45:36.462 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:45:36.462 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:45:36.462 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:45:36.462 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:45:36.462 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:45:36.462 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:45:36.462 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:45:36.462 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:45:36.463 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:45:36.463 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:45:36.463 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:45:36.463 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:45:36.463 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:45:36.463 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:45:36.463 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:45:36.463 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:45:36.463 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:45:36.463 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:45:36.463 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:45:36.463 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:45:36.463 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:45:36.463 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:45:36.487 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:45:36.487 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:45:36.487 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:45:36.487 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:45:36.617 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:45:36.618 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:45:36.618 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:45:36.618 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:45:36.618 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:45:36.618 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:45:36.619 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:45:36.619 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:45:36.619 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:45:36.619 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:45:36.619 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:45:36.619 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:45:36.619 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:45:36.620 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:45:36.621 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:45:36.621 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:45:36.678 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 32 2024-04-06T09:45:36.679 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 33 2024-04-06T09:45:36.679 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:45:36.679 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=133, tag_tid=21, entry_tid=1], [object_number=132, tag_tid=21, entry_tid=0], [object_number=135, tag_tid=20, entry_tid=19939], [object_number=134, tag_tid=20, entry_tid=19938]]], state=connected] 2024-04-06T09:45:36.679 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=130, tag_tid=20, entry_tid=18462], [object_number=129, tag_tid=20, entry_tid=18461], [object_number=128, tag_tid=20, entry_tid=18460], [object_number=131, tag_tid=20, entry_tid=18459]]], state=connected] 2024-04-06T09:45:36.682 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:45:36.683 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=133, tag_tid=21, entry_tid=1' 2024-04-06T09:45:36.684 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:45:36.685 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:45:36.685 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:45:36.685 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:45:36.685 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:45:36.685 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:45:36.685 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:45:36.685 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:45:36.685 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:45:36.685 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:45:36.685 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:45:36.685 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:45:36.686 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:45:36.686 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:45:36.686 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:45:36.733 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 32 2024-04-06T09:45:36.733 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 33 2024-04-06T09:45:36.733 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:45:36.733 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=133, tag_tid=21, entry_tid=1], [object_number=132, tag_tid=21, entry_tid=0], [object_number=135, tag_tid=20, entry_tid=19939], [object_number=134, tag_tid=20, entry_tid=19938]]], state=connected] 2024-04-06T09:45:36.733 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=130, tag_tid=20, entry_tid=18462], [object_number=129, tag_tid=20, entry_tid=18461], [object_number=128, tag_tid=20, entry_tid=18460], [object_number=131, tag_tid=20, entry_tid=18459]]], state=connected] 2024-04-06T09:45:36.736 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:45:36.738 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=130, tag_tid=20, entry_tid=18462' 2024-04-06T09:45:36.738 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=133, tag_tid=21, entry_tid=1' -a 'object_number=133, tag_tid=21, entry_tid=1' = 'object_number=130, tag_tid=20, entry_tid=18462' 2024-04-06T09:45:36.738 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=130, tag_tid=20, entry_tid=18462' '!=' 'object_number=128, tag_tid=20, entry_tid=18364' 2024-04-06T09:45:36.738 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:45:36.738 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=130, tag_tid=20, entry_tid=18462' = 'object_number=128, tag_tid=20, entry_tid=18364' 2024-04-06T09:45:36.739 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=130, tag_tid=20, entry_tid=18462' 2024-04-06T09:45:36.739 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=133, tag_tid=21, entry_tid=1' 2024-04-06T09:45:36.739 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:45:36.739 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:45:36.740 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=21 2024-04-06T09:45:36.740 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=130, tag_tid=20, entry_tid=18462' 2024-04-06T09:45:36.740 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:45:36.741 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:45:36.741 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=20 2024-04-06T09:45:36.742 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=133, tag_tid=21, entry_tid=1' 2024-04-06T09:45:36.742 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:45:36.742 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:45:36.743 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:45:36.744 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=130, tag_tid=20, entry_tid=18462' 2024-04-06T09:45:36.744 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:45:36.744 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:45:36.745 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=18462 2024-04-06T09:45:36.745 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 21 = 20 -a 1 -le 18462 2024-04-06T09:45:36.745 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:45:36.745 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:45:36.745 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:45:36.947 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:45:36.947 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:45:36.947 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:45:36.947 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:45:36.947 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:45:36.947 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:45:36.947 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:45:36.947 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:45:36.947 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:45:36.947 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:45:36.948 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:45:36.948 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:45:36.948 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:45:36.948 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:45:36.948 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:45:36.948 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:45:36.948 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:45:36.951 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:45:36.952 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:45:36.952 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:45:36.952 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:45:36.952 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:45:36.952 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:45:36.952 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:45:36.952 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:45:36.952 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:45:36.952 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:45:36.952 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:45:36.952 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:45:36.952 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:45:36.952 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:45:36.952 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:45:36.953 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:45:36.953 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:45:36.953 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:45:36.953 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:45:36.953 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:45:36.953 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:45:36.953 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:45:36.953 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:45:36.953 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:45:36.953 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:45:36.953 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:45:36.953 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:45:36.953 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:45:36.953 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:45:36.953 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:45:36.954 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:45:36.954 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:45:36.978 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:45:36.978 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:45:36.978 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:45:36.978 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:45:37.107 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:45:37.108 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:45:37.109 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:45:37.109 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:45:37.109 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:45:37.109 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:45:37.109 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:45:37.109 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:45:37.109 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:45:37.109 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:45:37.110 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:45:37.110 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:45:37.110 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:45:37.111 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:45:37.112 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:45:37.112 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:45:37.158 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 32 2024-04-06T09:45:37.158 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 33 2024-04-06T09:45:37.158 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:45:37.158 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=133, tag_tid=21, entry_tid=1], [object_number=132, tag_tid=21, entry_tid=0], [object_number=135, tag_tid=20, entry_tid=19939], [object_number=134, tag_tid=20, entry_tid=19938]]], state=connected] 2024-04-06T09:45:37.158 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=128, tag_tid=20, entry_tid=18556], [object_number=131, tag_tid=20, entry_tid=18555], [object_number=130, tag_tid=20, entry_tid=18554], [object_number=129, tag_tid=20, entry_tid=18553]]], state=connected] 2024-04-06T09:45:37.161 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:45:37.162 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=133, tag_tid=21, entry_tid=1' 2024-04-06T09:45:37.163 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:45:37.164 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:45:37.164 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:45:37.164 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:45:37.164 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:45:37.164 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:45:37.164 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:45:37.164 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:45:37.164 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:45:37.164 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:45:37.164 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:45:37.164 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:45:37.165 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:45:37.165 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:45:37.165 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:45:37.211 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 32 2024-04-06T09:45:37.211 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 33 2024-04-06T09:45:37.211 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:45:37.211 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=133, tag_tid=21, entry_tid=1], [object_number=132, tag_tid=21, entry_tid=0], [object_number=135, tag_tid=20, entry_tid=19939], [object_number=134, tag_tid=20, entry_tid=19938]]], state=connected] 2024-04-06T09:45:37.211 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=128, tag_tid=20, entry_tid=18556], [object_number=131, tag_tid=20, entry_tid=18555], [object_number=130, tag_tid=20, entry_tid=18554], [object_number=129, tag_tid=20, entry_tid=18553]]], state=connected] 2024-04-06T09:45:37.215 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:45:37.216 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=128, tag_tid=20, entry_tid=18556' 2024-04-06T09:45:37.216 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=133, tag_tid=21, entry_tid=1' -a 'object_number=133, tag_tid=21, entry_tid=1' = 'object_number=128, tag_tid=20, entry_tid=18556' 2024-04-06T09:45:37.216 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=128, tag_tid=20, entry_tid=18556' '!=' 'object_number=130, tag_tid=20, entry_tid=18462' 2024-04-06T09:45:37.217 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:45:37.217 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=128, tag_tid=20, entry_tid=18556' = 'object_number=130, tag_tid=20, entry_tid=18462' 2024-04-06T09:45:37.217 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=128, tag_tid=20, entry_tid=18556' 2024-04-06T09:45:37.217 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=133, tag_tid=21, entry_tid=1' 2024-04-06T09:45:37.217 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:45:37.217 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:45:37.218 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=21 2024-04-06T09:45:37.218 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=128, tag_tid=20, entry_tid=18556' 2024-04-06T09:45:37.219 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:45:37.219 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:45:37.220 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=20 2024-04-06T09:45:37.220 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=133, tag_tid=21, entry_tid=1' 2024-04-06T09:45:37.221 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:45:37.221 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:45:37.221 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:45:37.222 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=128, tag_tid=20, entry_tid=18556' 2024-04-06T09:45:37.222 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:45:37.222 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:45:37.223 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=18556 2024-04-06T09:45:37.223 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 21 = 20 -a 1 -le 18556 2024-04-06T09:45:37.223 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:45:37.224 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:45:37.224 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:45:37.425 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:45:37.425 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:45:37.425 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:45:37.426 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:45:37.426 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:45:37.426 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:45:37.426 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:45:37.426 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:45:37.426 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:45:37.426 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:45:37.426 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:45:37.426 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:45:37.426 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:45:37.426 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:45:37.426 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:45:37.427 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:45:37.427 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:45:37.429 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:45:37.430 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:45:37.430 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:45:37.430 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:45:37.430 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:45:37.430 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:45:37.430 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:45:37.430 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:45:37.430 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:45:37.430 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:45:37.430 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:45:37.430 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:45:37.430 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:45:37.430 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:45:37.430 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:45:37.431 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:45:37.431 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:45:37.432 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:45:37.432 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:45:37.432 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:45:37.432 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:45:37.432 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:45:37.432 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:45:37.432 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:45:37.432 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:45:37.432 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:45:37.432 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:45:37.432 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:45:37.432 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:45:37.432 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:45:37.432 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:45:37.433 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:45:37.457 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:45:37.457 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:45:37.458 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:45:37.458 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:45:37.586 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:45:37.587 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:45:37.587 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:45:37.587 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:45:37.587 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:45:37.587 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:45:37.587 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:45:37.587 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:45:37.587 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:45:37.587 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:45:37.588 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:45:37.588 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:45:37.588 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:45:37.591 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:45:37.591 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:45:37.591 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:45:37.639 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 32 2024-04-06T09:45:37.640 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 33 2024-04-06T09:45:37.640 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:45:37.640 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=133, tag_tid=21, entry_tid=1], [object_number=132, tag_tid=21, entry_tid=0], [object_number=135, tag_tid=20, entry_tid=19939], [object_number=134, tag_tid=20, entry_tid=19938]]], state=connected] 2024-04-06T09:45:37.640 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=128, tag_tid=20, entry_tid=18648], [object_number=131, tag_tid=20, entry_tid=18647], [object_number=130, tag_tid=20, entry_tid=18646], [object_number=129, tag_tid=20, entry_tid=18645]]], state=connected] 2024-04-06T09:45:37.643 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:45:37.644 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=133, tag_tid=21, entry_tid=1' 2024-04-06T09:45:37.646 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:45:37.646 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:45:37.646 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:45:37.646 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:45:37.646 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:45:37.646 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:45:37.646 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:45:37.646 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:45:37.646 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:45:37.646 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:45:37.646 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:45:37.646 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:45:37.647 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:45:37.647 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:45:37.647 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:45:37.695 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 32 2024-04-06T09:45:37.695 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 33 2024-04-06T09:45:37.695 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:45:37.695 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=133, tag_tid=21, entry_tid=1], [object_number=132, tag_tid=21, entry_tid=0], [object_number=135, tag_tid=20, entry_tid=19939], [object_number=134, tag_tid=20, entry_tid=19938]]], state=connected] 2024-04-06T09:45:37.695 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=128, tag_tid=20, entry_tid=18648], [object_number=131, tag_tid=20, entry_tid=18647], [object_number=130, tag_tid=20, entry_tid=18646], [object_number=129, tag_tid=20, entry_tid=18645]]], state=connected] 2024-04-06T09:45:37.698 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:45:37.700 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=128, tag_tid=20, entry_tid=18648' 2024-04-06T09:45:37.700 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=133, tag_tid=21, entry_tid=1' -a 'object_number=133, tag_tid=21, entry_tid=1' = 'object_number=128, tag_tid=20, entry_tid=18648' 2024-04-06T09:45:37.700 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=128, tag_tid=20, entry_tid=18648' '!=' 'object_number=128, tag_tid=20, entry_tid=18556' 2024-04-06T09:45:37.700 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:45:37.700 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=128, tag_tid=20, entry_tid=18648' = 'object_number=128, tag_tid=20, entry_tid=18556' 2024-04-06T09:45:37.700 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=128, tag_tid=20, entry_tid=18648' 2024-04-06T09:45:37.701 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=133, tag_tid=21, entry_tid=1' 2024-04-06T09:45:37.701 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:45:37.701 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:45:37.702 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=21 2024-04-06T09:45:37.702 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=128, tag_tid=20, entry_tid=18648' 2024-04-06T09:45:37.703 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:45:37.703 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:45:37.703 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=20 2024-04-06T09:45:37.704 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=133, tag_tid=21, entry_tid=1' 2024-04-06T09:45:37.704 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:45:37.704 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:45:37.705 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:45:37.706 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=128, tag_tid=20, entry_tid=18648' 2024-04-06T09:45:37.706 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:45:37.706 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:45:37.707 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=18648 2024-04-06T09:45:37.707 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 21 = 20 -a 1 -le 18648 2024-04-06T09:45:37.707 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:45:37.707 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:45:37.707 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:45:37.909 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:45:37.909 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:45:37.909 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:45:37.909 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:45:37.909 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:45:37.910 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:45:37.910 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:45:37.910 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:45:37.910 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:45:37.910 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:45:37.910 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:45:37.910 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:45:37.910 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:45:37.910 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:45:37.910 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:45:37.911 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:45:37.911 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:45:37.912 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:45:37.912 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:45:37.912 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:45:37.913 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:45:37.913 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:45:37.913 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:45:37.913 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:45:37.913 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:45:37.913 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:45:37.913 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:45:37.913 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:45:37.913 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:45:37.913 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:45:37.913 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:45:37.913 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:45:37.913 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:45:37.914 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:45:37.914 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:45:37.914 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:45:37.914 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:45:37.914 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:45:37.915 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:45:37.915 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:45:37.915 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:45:37.915 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:45:37.915 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:45:37.915 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:45:37.915 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:45:37.915 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:45:37.915 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:45:37.915 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:45:37.915 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:45:37.939 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:45:37.939 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:45:37.939 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:45:37.939 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:45:38.073 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:45:38.074 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:45:38.074 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:45:38.074 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:45:38.075 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:45:38.075 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:45:38.075 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:45:38.075 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:45:38.075 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:45:38.075 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:45:38.075 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:45:38.076 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:45:38.076 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:45:38.078 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:45:38.078 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:45:38.079 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:45:38.128 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 32 2024-04-06T09:45:38.128 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 33 2024-04-06T09:45:38.129 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:45:38.129 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=133, tag_tid=21, entry_tid=1], [object_number=132, tag_tid=21, entry_tid=0], [object_number=135, tag_tid=20, entry_tid=19939], [object_number=134, tag_tid=20, entry_tid=19938]]], state=connected] 2024-04-06T09:45:38.129 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=130, tag_tid=20, entry_tid=18734], [object_number=129, tag_tid=20, entry_tid=18733], [object_number=128, tag_tid=20, entry_tid=18732], [object_number=131, tag_tid=20, entry_tid=18731]]], state=connected] 2024-04-06T09:45:38.132 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:45:38.133 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=133, tag_tid=21, entry_tid=1' 2024-04-06T09:45:38.135 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:45:38.135 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:45:38.135 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:45:38.135 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:45:38.135 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:45:38.135 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:45:38.135 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:45:38.135 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:45:38.135 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:45:38.135 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:45:38.135 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:45:38.135 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:45:38.136 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:45:38.136 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:45:38.137 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:45:38.184 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 32 2024-04-06T09:45:38.184 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 33 2024-04-06T09:45:38.184 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:45:38.184 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=133, tag_tid=21, entry_tid=1], [object_number=132, tag_tid=21, entry_tid=0], [object_number=135, tag_tid=20, entry_tid=19939], [object_number=134, tag_tid=20, entry_tid=19938]]], state=connected] 2024-04-06T09:45:38.184 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=130, tag_tid=20, entry_tid=18734], [object_number=129, tag_tid=20, entry_tid=18733], [object_number=128, tag_tid=20, entry_tid=18732], [object_number=131, tag_tid=20, entry_tid=18731]]], state=connected] 2024-04-06T09:45:38.187 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:45:38.189 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=130, tag_tid=20, entry_tid=18734' 2024-04-06T09:45:38.189 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=133, tag_tid=21, entry_tid=1' -a 'object_number=133, tag_tid=21, entry_tid=1' = 'object_number=130, tag_tid=20, entry_tid=18734' 2024-04-06T09:45:38.189 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=130, tag_tid=20, entry_tid=18734' '!=' 'object_number=128, tag_tid=20, entry_tid=18648' 2024-04-06T09:45:38.189 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:45:38.190 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=130, tag_tid=20, entry_tid=18734' = 'object_number=128, tag_tid=20, entry_tid=18648' 2024-04-06T09:45:38.190 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=130, tag_tid=20, entry_tid=18734' 2024-04-06T09:45:38.190 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=133, tag_tid=21, entry_tid=1' 2024-04-06T09:45:38.190 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:45:38.190 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:45:38.190 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=21 2024-04-06T09:45:38.191 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=130, tag_tid=20, entry_tid=18734' 2024-04-06T09:45:38.191 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:45:38.191 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:45:38.192 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=20 2024-04-06T09:45:38.193 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=133, tag_tid=21, entry_tid=1' 2024-04-06T09:45:38.193 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:45:38.193 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:45:38.194 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:45:38.194 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=130, tag_tid=20, entry_tid=18734' 2024-04-06T09:45:38.195 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:45:38.195 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:45:38.196 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=18734 2024-04-06T09:45:38.196 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 21 = 20 -a 1 -le 18734 2024-04-06T09:45:38.196 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:45:38.196 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:45:38.196 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:45:38.398 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:45:38.398 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:45:38.398 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:45:38.398 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:45:38.398 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:45:38.398 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:45:38.399 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:45:38.399 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:45:38.399 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:45:38.399 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:45:38.399 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:45:38.399 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:45:38.399 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:45:38.399 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:45:38.399 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:45:38.400 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:45:38.400 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:45:38.402 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:45:38.402 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:45:38.402 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:45:38.402 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:45:38.403 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:45:38.403 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:45:38.403 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:45:38.403 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:45:38.403 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:45:38.403 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:45:38.403 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:45:38.403 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:45:38.403 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:45:38.403 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:45:38.403 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:45:38.403 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:45:38.404 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:45:38.404 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:45:38.404 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:45:38.404 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:45:38.405 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:45:38.405 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:45:38.405 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:45:38.405 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:45:38.405 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:45:38.405 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:45:38.405 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:45:38.405 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:45:38.405 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:45:38.405 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:45:38.405 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:45:38.405 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:45:38.431 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:45:38.431 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:45:38.432 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:45:38.432 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:45:38.569 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:45:38.571 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:45:38.571 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:45:38.571 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:45:38.571 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:45:38.571 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:45:38.571 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:45:38.571 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:45:38.571 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:45:38.571 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:45:38.571 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:45:38.572 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:45:38.572 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:45:38.574 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:45:38.574 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:45:38.574 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:45:38.621 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 32 2024-04-06T09:45:38.622 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 33 2024-04-06T09:45:38.622 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:45:38.622 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=133, tag_tid=21, entry_tid=1], [object_number=132, tag_tid=21, entry_tid=0], [object_number=135, tag_tid=20, entry_tid=19939], [object_number=134, tag_tid=20, entry_tid=19938]]], state=connected] 2024-04-06T09:45:38.622 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=128, tag_tid=20, entry_tid=18816], [object_number=131, tag_tid=20, entry_tid=18815], [object_number=130, tag_tid=20, entry_tid=18814], [object_number=129, tag_tid=20, entry_tid=18813]]], state=connected] 2024-04-06T09:45:38.625 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:45:38.627 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=133, tag_tid=21, entry_tid=1' 2024-04-06T09:45:38.628 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:45:38.628 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:45:38.628 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:45:38.628 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:45:38.628 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:45:38.628 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:45:38.628 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:45:38.628 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:45:38.628 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:45:38.629 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:45:38.629 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:45:38.629 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:45:38.629 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:45:38.630 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:45:38.630 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:45:38.676 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 32 2024-04-06T09:45:38.676 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 33 2024-04-06T09:45:38.676 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:45:38.676 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=133, tag_tid=21, entry_tid=1], [object_number=132, tag_tid=21, entry_tid=0], [object_number=135, tag_tid=20, entry_tid=19939], [object_number=134, tag_tid=20, entry_tid=19938]]], state=connected] 2024-04-06T09:45:38.676 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=128, tag_tid=20, entry_tid=18816], [object_number=131, tag_tid=20, entry_tid=18815], [object_number=130, tag_tid=20, entry_tid=18814], [object_number=129, tag_tid=20, entry_tid=18813]]], state=connected] 2024-04-06T09:45:38.679 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:45:38.681 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=128, tag_tid=20, entry_tid=18816' 2024-04-06T09:45:38.681 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=133, tag_tid=21, entry_tid=1' -a 'object_number=133, tag_tid=21, entry_tid=1' = 'object_number=128, tag_tid=20, entry_tid=18816' 2024-04-06T09:45:38.681 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=128, tag_tid=20, entry_tid=18816' '!=' 'object_number=130, tag_tid=20, entry_tid=18734' 2024-04-06T09:45:38.681 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:45:38.681 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=128, tag_tid=20, entry_tid=18816' = 'object_number=130, tag_tid=20, entry_tid=18734' 2024-04-06T09:45:38.681 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=128, tag_tid=20, entry_tid=18816' 2024-04-06T09:45:38.682 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=133, tag_tid=21, entry_tid=1' 2024-04-06T09:45:38.682 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:45:38.682 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:45:38.683 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=21 2024-04-06T09:45:38.683 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=128, tag_tid=20, entry_tid=18816' 2024-04-06T09:45:38.683 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:45:38.683 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:45:38.684 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=20 2024-04-06T09:45:38.685 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=133, tag_tid=21, entry_tid=1' 2024-04-06T09:45:38.685 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:45:38.685 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:45:38.686 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:45:38.687 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=128, tag_tid=20, entry_tid=18816' 2024-04-06T09:45:38.687 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:45:38.687 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:45:38.688 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=18816 2024-04-06T09:45:38.688 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 21 = 20 -a 1 -le 18816 2024-04-06T09:45:38.688 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:45:38.688 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:45:38.688 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:45:38.891 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:45:38.891 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:45:38.891 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:45:38.891 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:45:38.891 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:45:38.891 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:45:38.892 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:45:38.892 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:45:38.892 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:45:38.892 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:45:38.892 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:45:38.892 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:45:38.892 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:45:38.892 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:45:38.892 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:45:38.892 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:45:38.893 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:45:38.895 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:45:38.895 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:45:38.895 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:45:38.895 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:45:38.896 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:45:38.896 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:45:38.896 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:45:38.896 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:45:38.896 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:45:38.896 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:45:38.896 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:45:38.896 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:45:38.897 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:45:38.897 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:45:38.897 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:45:38.897 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:45:38.897 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:45:38.897 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:45:38.897 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:45:38.897 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:45:38.897 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:45:38.897 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:45:38.897 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:45:38.897 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:45:38.897 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:45:38.897 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:45:38.897 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:45:38.898 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:45:38.898 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:45:38.898 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:45:38.898 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:45:38.898 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:45:38.921 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:45:38.921 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:45:38.921 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:45:38.921 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:45:39.056 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:45:39.057 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:45:39.057 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:45:39.058 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:45:39.058 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:45:39.058 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:45:39.058 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:45:39.058 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:45:39.058 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:45:39.058 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:45:39.059 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:45:39.059 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:45:39.059 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:45:39.061 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:45:39.062 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:45:39.062 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:45:39.116 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 32 2024-04-06T09:45:39.116 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 33 2024-04-06T09:45:39.116 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:45:39.116 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=133, tag_tid=21, entry_tid=1], [object_number=132, tag_tid=21, entry_tid=0], [object_number=135, tag_tid=20, entry_tid=19939], [object_number=134, tag_tid=20, entry_tid=19938]]], state=connected] 2024-04-06T09:45:39.116 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=129, tag_tid=20, entry_tid=18901], [object_number=128, tag_tid=20, entry_tid=18900], [object_number=131, tag_tid=20, entry_tid=18899], [object_number=130, tag_tid=20, entry_tid=18898]]], state=connected] 2024-04-06T09:45:39.119 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:45:39.121 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=133, tag_tid=21, entry_tid=1' 2024-04-06T09:45:39.122 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:45:39.122 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:45:39.122 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:45:39.122 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:45:39.122 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:45:39.122 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:45:39.122 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:45:39.122 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:45:39.123 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:45:39.123 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:45:39.123 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:45:39.123 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:45:39.123 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:45:39.123 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:45:39.123 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:45:39.170 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 32 2024-04-06T09:45:39.170 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 33 2024-04-06T09:45:39.171 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:45:39.171 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=133, tag_tid=21, entry_tid=1], [object_number=132, tag_tid=21, entry_tid=0], [object_number=135, tag_tid=20, entry_tid=19939], [object_number=134, tag_tid=20, entry_tid=19938]]], state=connected] 2024-04-06T09:45:39.171 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=129, tag_tid=20, entry_tid=18901], [object_number=128, tag_tid=20, entry_tid=18900], [object_number=131, tag_tid=20, entry_tid=18899], [object_number=130, tag_tid=20, entry_tid=18898]]], state=connected] 2024-04-06T09:45:39.173 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:45:39.176 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=129, tag_tid=20, entry_tid=18901' 2024-04-06T09:45:39.176 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=133, tag_tid=21, entry_tid=1' -a 'object_number=133, tag_tid=21, entry_tid=1' = 'object_number=129, tag_tid=20, entry_tid=18901' 2024-04-06T09:45:39.176 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=129, tag_tid=20, entry_tid=18901' '!=' 'object_number=128, tag_tid=20, entry_tid=18816' 2024-04-06T09:45:39.177 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:45:39.177 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=129, tag_tid=20, entry_tid=18901' = 'object_number=128, tag_tid=20, entry_tid=18816' 2024-04-06T09:45:39.177 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=129, tag_tid=20, entry_tid=18901' 2024-04-06T09:45:39.177 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=133, tag_tid=21, entry_tid=1' 2024-04-06T09:45:39.177 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:45:39.177 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:45:39.180 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=21 2024-04-06T09:45:39.180 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=129, tag_tid=20, entry_tid=18901' 2024-04-06T09:45:39.180 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:45:39.180 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:45:39.180 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=20 2024-04-06T09:45:39.180 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=133, tag_tid=21, entry_tid=1' 2024-04-06T09:45:39.180 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:45:39.180 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:45:39.181 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:45:39.182 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=129, tag_tid=20, entry_tid=18901' 2024-04-06T09:45:39.182 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:45:39.182 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:45:39.183 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=18901 2024-04-06T09:45:39.183 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 21 = 20 -a 1 -le 18901 2024-04-06T09:45:39.183 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:45:39.183 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:45:39.183 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:45:39.384 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:45:39.384 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:45:39.384 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:45:39.385 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:45:39.385 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:45:39.385 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:45:39.385 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:45:39.386 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:45:39.386 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:45:39.386 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:45:39.386 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:45:39.386 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:45:39.387 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:45:39.387 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:45:39.387 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:45:39.391 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:45:39.391 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:45:39.391 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:45:39.392 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:45:39.392 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:45:39.392 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:45:39.392 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:45:39.392 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:45:39.392 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:45:39.392 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:45:39.392 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:45:39.392 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:45:39.392 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:45:39.392 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:45:39.392 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:45:39.392 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:45:39.392 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:45:39.393 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:45:39.393 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:45:39.393 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:45:39.393 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:45:39.393 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:45:39.393 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:45:39.393 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:45:39.393 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:45:39.394 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:45:39.394 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:45:39.394 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:45:39.394 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:45:39.394 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:45:39.394 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:45:39.394 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:45:39.394 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:45:39.394 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:45:39.416 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:45:39.416 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:45:39.416 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:45:39.416 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:45:39.558 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:45:39.559 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:45:39.559 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:45:39.559 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:45:39.559 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:45:39.559 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:45:39.559 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:45:39.559 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:45:39.559 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:45:39.559 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:45:39.560 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:45:39.560 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:45:39.560 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:45:39.563 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:45:39.563 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:45:39.563 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:45:39.610 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 32 2024-04-06T09:45:39.610 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 33 2024-04-06T09:45:39.610 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:45:39.611 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=133, tag_tid=21, entry_tid=1], [object_number=132, tag_tid=21, entry_tid=0], [object_number=135, tag_tid=20, entry_tid=19939], [object_number=134, tag_tid=20, entry_tid=19938]]], state=connected] 2024-04-06T09:45:39.611 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=128, tag_tid=20, entry_tid=18980], [object_number=131, tag_tid=20, entry_tid=18979], [object_number=130, tag_tid=20, entry_tid=18978], [object_number=129, tag_tid=20, entry_tid=18977]]], state=connected] 2024-04-06T09:45:39.614 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:45:39.615 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=133, tag_tid=21, entry_tid=1' 2024-04-06T09:45:39.616 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:45:39.616 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:45:39.616 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:45:39.617 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:45:39.617 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:45:39.617 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:45:39.617 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:45:39.617 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:45:39.617 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:45:39.617 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:45:39.617 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:45:39.617 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:45:39.618 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:45:39.618 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:45:39.618 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:45:39.664 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 32 2024-04-06T09:45:39.664 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 33 2024-04-06T09:45:39.664 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:45:39.664 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=133, tag_tid=21, entry_tid=1], [object_number=132, tag_tid=21, entry_tid=0], [object_number=135, tag_tid=20, entry_tid=19939], [object_number=134, tag_tid=20, entry_tid=19938]]], state=connected] 2024-04-06T09:45:39.664 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=128, tag_tid=20, entry_tid=18980], [object_number=131, tag_tid=20, entry_tid=18979], [object_number=130, tag_tid=20, entry_tid=18978], [object_number=129, tag_tid=20, entry_tid=18977]]], state=connected] 2024-04-06T09:45:39.667 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:45:39.670 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=128, tag_tid=20, entry_tid=18980' 2024-04-06T09:45:39.670 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=133, tag_tid=21, entry_tid=1' -a 'object_number=133, tag_tid=21, entry_tid=1' = 'object_number=128, tag_tid=20, entry_tid=18980' 2024-04-06T09:45:39.670 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=128, tag_tid=20, entry_tid=18980' '!=' 'object_number=129, tag_tid=20, entry_tid=18901' 2024-04-06T09:45:39.670 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:45:39.670 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=128, tag_tid=20, entry_tid=18980' = 'object_number=129, tag_tid=20, entry_tid=18901' 2024-04-06T09:45:39.670 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=128, tag_tid=20, entry_tid=18980' 2024-04-06T09:45:39.671 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=133, tag_tid=21, entry_tid=1' 2024-04-06T09:45:39.671 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:45:39.671 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:45:39.671 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=21 2024-04-06T09:45:39.672 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=128, tag_tid=20, entry_tid=18980' 2024-04-06T09:45:39.672 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:45:39.672 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:45:39.673 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=20 2024-04-06T09:45:39.674 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=133, tag_tid=21, entry_tid=1' 2024-04-06T09:45:39.674 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:45:39.674 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:45:39.675 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:45:39.676 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=128, tag_tid=20, entry_tid=18980' 2024-04-06T09:45:39.676 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:45:39.676 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:45:39.677 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=18980 2024-04-06T09:45:39.677 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 21 = 20 -a 1 -le 18980 2024-04-06T09:45:39.677 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:45:39.677 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:45:39.677 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:45:39.879 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:45:39.879 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:45:39.879 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:45:39.879 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:45:39.879 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:45:39.879 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:45:39.879 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:45:39.879 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:45:39.879 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:45:39.879 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:45:39.880 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:45:39.880 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:45:39.880 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:45:39.880 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:45:39.880 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:45:39.880 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:45:39.881 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:45:39.883 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:45:39.883 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:45:39.883 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:45:39.883 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:45:39.883 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:45:39.883 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:45:39.883 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:45:39.883 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:45:39.883 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:45:39.883 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:45:39.883 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:45:39.883 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:45:39.883 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:45:39.884 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:45:39.884 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:45:39.884 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:45:39.884 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:45:39.884 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:45:39.884 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:45:39.884 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:45:39.884 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:45:39.884 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:45:39.884 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:45:39.884 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:45:39.884 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:45:39.884 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:45:39.884 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:45:39.884 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:45:39.884 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:45:39.885 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:45:39.885 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:45:39.885 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:45:39.910 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:45:39.910 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:45:39.911 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:45:39.911 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:45:40.044 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:45:40.045 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:45:40.046 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:45:40.046 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:45:40.046 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:45:40.046 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:45:40.046 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:45:40.046 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:45:40.046 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:45:40.046 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:45:40.047 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:45:40.047 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:45:40.047 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:45:40.049 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:45:40.050 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:45:40.050 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:45:40.101 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 32 2024-04-06T09:45:40.102 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 33 2024-04-06T09:45:40.102 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:45:40.102 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=133, tag_tid=21, entry_tid=1], [object_number=132, tag_tid=21, entry_tid=0], [object_number=135, tag_tid=20, entry_tid=19939], [object_number=134, tag_tid=20, entry_tid=19938]]], state=connected] 2024-04-06T09:45:40.102 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=129, tag_tid=20, entry_tid=19061], [object_number=128, tag_tid=20, entry_tid=19060], [object_number=131, tag_tid=20, entry_tid=19059], [object_number=130, tag_tid=20, entry_tid=19058]]], state=connected] 2024-04-06T09:45:40.105 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:45:40.106 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=133, tag_tid=21, entry_tid=1' 2024-04-06T09:45:40.108 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:45:40.108 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:45:40.108 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:45:40.108 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:45:40.108 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:45:40.108 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:45:40.108 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:45:40.108 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:45:40.108 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:45:40.108 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:45:40.108 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:45:40.108 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:45:40.109 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:45:40.109 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:45:40.109 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:45:40.156 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 32 2024-04-06T09:45:40.156 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 33 2024-04-06T09:45:40.156 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:45:40.156 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=133, tag_tid=21, entry_tid=1], [object_number=132, tag_tid=21, entry_tid=0], [object_number=135, tag_tid=20, entry_tid=19939], [object_number=134, tag_tid=20, entry_tid=19938]]], state=connected] 2024-04-06T09:45:40.156 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=129, tag_tid=20, entry_tid=19061], [object_number=128, tag_tid=20, entry_tid=19060], [object_number=131, tag_tid=20, entry_tid=19059], [object_number=130, tag_tid=20, entry_tid=19058]]], state=connected] 2024-04-06T09:45:40.159 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:45:40.161 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=129, tag_tid=20, entry_tid=19061' 2024-04-06T09:45:40.161 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=133, tag_tid=21, entry_tid=1' -a 'object_number=133, tag_tid=21, entry_tid=1' = 'object_number=129, tag_tid=20, entry_tid=19061' 2024-04-06T09:45:40.162 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=129, tag_tid=20, entry_tid=19061' '!=' 'object_number=128, tag_tid=20, entry_tid=18980' 2024-04-06T09:45:40.162 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:45:40.162 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=129, tag_tid=20, entry_tid=19061' = 'object_number=128, tag_tid=20, entry_tid=18980' 2024-04-06T09:45:40.162 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=129, tag_tid=20, entry_tid=19061' 2024-04-06T09:45:40.162 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=133, tag_tid=21, entry_tid=1' 2024-04-06T09:45:40.162 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:45:40.162 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:45:40.162 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=21 2024-04-06T09:45:40.163 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=129, tag_tid=20, entry_tid=19061' 2024-04-06T09:45:40.163 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:45:40.163 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:45:40.164 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=20 2024-04-06T09:45:40.165 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=133, tag_tid=21, entry_tid=1' 2024-04-06T09:45:40.165 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:45:40.165 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:45:40.166 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:45:40.167 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=129, tag_tid=20, entry_tid=19061' 2024-04-06T09:45:40.167 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:45:40.167 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:45:40.168 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=19061 2024-04-06T09:45:40.168 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 21 = 20 -a 1 -le 19061 2024-04-06T09:45:40.168 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:45:40.168 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:45:40.168 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:45:40.370 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:45:40.370 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:45:40.371 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:45:40.371 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:45:40.371 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:45:40.371 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:45:40.371 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:45:40.371 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:45:40.371 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:45:40.371 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:45:40.371 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:45:40.371 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:45:40.372 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:45:40.372 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:45:40.372 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:45:40.374 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:45:40.374 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:45:40.375 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:45:40.375 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:45:40.375 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:45:40.375 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:45:40.375 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:45:40.375 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:45:40.375 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:45:40.375 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:45:40.375 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:45:40.375 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:45:40.375 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:45:40.375 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:45:40.375 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:45:40.376 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:45:40.376 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:45:40.376 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:45:40.376 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:45:40.376 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:45:40.376 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:45:40.376 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:45:40.376 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:45:40.376 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:45:40.376 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:45:40.376 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:45:40.376 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:45:40.376 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:45:40.376 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:45:40.377 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:45:40.377 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:45:40.377 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:45:40.377 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:45:40.377 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:45:40.399 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:45:40.400 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:45:40.400 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:45:40.400 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:45:40.534 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:45:40.535 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:45:40.536 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:45:40.536 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:45:40.536 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:45:40.536 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:45:40.536 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:45:40.537 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:45:40.537 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:45:40.537 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:45:40.537 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:45:40.538 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:45:40.539 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:45:40.539 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:45:40.539 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:45:40.539 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:45:40.590 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 32 2024-04-06T09:45:40.591 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 33 2024-04-06T09:45:40.591 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:45:40.591 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=133, tag_tid=21, entry_tid=1], [object_number=132, tag_tid=21, entry_tid=0], [object_number=135, tag_tid=20, entry_tid=19939], [object_number=134, tag_tid=20, entry_tid=19938]]], state=connected] 2024-04-06T09:45:40.591 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=129, tag_tid=20, entry_tid=19145], [object_number=128, tag_tid=20, entry_tid=19144], [object_number=131, tag_tid=20, entry_tid=19143], [object_number=130, tag_tid=20, entry_tid=19142]]], state=connected] 2024-04-06T09:45:40.594 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:45:40.596 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=133, tag_tid=21, entry_tid=1' 2024-04-06T09:45:40.597 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:45:40.597 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:45:40.597 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:45:40.597 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:45:40.597 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:45:40.597 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:45:40.597 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:45:40.597 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:45:40.598 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:45:40.598 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:45:40.598 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:45:40.598 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:45:40.598 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:45:40.598 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:45:40.599 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:45:40.648 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 32 2024-04-06T09:45:40.648 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 33 2024-04-06T09:45:40.648 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:45:40.648 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=133, tag_tid=21, entry_tid=1], [object_number=132, tag_tid=21, entry_tid=0], [object_number=135, tag_tid=20, entry_tid=19939], [object_number=134, tag_tid=20, entry_tid=19938]]], state=connected] 2024-04-06T09:45:40.648 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=129, tag_tid=20, entry_tid=19145], [object_number=128, tag_tid=20, entry_tid=19144], [object_number=131, tag_tid=20, entry_tid=19143], [object_number=130, tag_tid=20, entry_tid=19142]]], state=connected] 2024-04-06T09:45:40.652 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:45:40.654 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=129, tag_tid=20, entry_tid=19145' 2024-04-06T09:45:40.654 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=133, tag_tid=21, entry_tid=1' -a 'object_number=133, tag_tid=21, entry_tid=1' = 'object_number=129, tag_tid=20, entry_tid=19145' 2024-04-06T09:45:40.654 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=129, tag_tid=20, entry_tid=19145' '!=' 'object_number=129, tag_tid=20, entry_tid=19061' 2024-04-06T09:45:40.654 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:45:40.654 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=129, tag_tid=20, entry_tid=19145' = 'object_number=129, tag_tid=20, entry_tid=19061' 2024-04-06T09:45:40.654 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=129, tag_tid=20, entry_tid=19145' 2024-04-06T09:45:40.654 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=133, tag_tid=21, entry_tid=1' 2024-04-06T09:45:40.654 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:45:40.654 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:45:40.655 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=21 2024-04-06T09:45:40.656 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=129, tag_tid=20, entry_tid=19145' 2024-04-06T09:45:40.656 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:45:40.656 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:45:40.657 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=20 2024-04-06T09:45:40.657 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=133, tag_tid=21, entry_tid=1' 2024-04-06T09:45:40.657 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:45:40.657 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:45:40.658 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:45:40.659 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=129, tag_tid=20, entry_tid=19145' 2024-04-06T09:45:40.659 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:45:40.659 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:45:40.660 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=19145 2024-04-06T09:45:40.660 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 21 = 20 -a 1 -le 19145 2024-04-06T09:45:40.660 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:45:40.660 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:45:40.661 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:45:40.862 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:45:40.862 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:45:40.862 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:45:40.862 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:45:40.862 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:45:40.863 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:45:40.863 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:45:40.863 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:45:40.863 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:45:40.863 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:45:40.863 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:45:40.863 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:45:40.863 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:45:40.863 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:45:40.863 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:45:40.864 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:45:40.864 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:45:40.866 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:45:40.866 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:45:40.866 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:45:40.866 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:45:40.866 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:45:40.866 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:45:40.866 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:45:40.866 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:45:40.866 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:45:40.867 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:45:40.867 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:45:40.867 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:45:40.867 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:45:40.867 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:45:40.867 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:45:40.867 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:45:40.868 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:45:40.868 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:45:40.868 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:45:40.868 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:45:40.868 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:45:40.868 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:45:40.869 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:45:40.869 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:45:40.869 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:45:40.869 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:45:40.869 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:45:40.869 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:45:40.869 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:45:40.869 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:45:40.869 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:45:40.869 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:45:40.892 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:45:40.892 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:45:40.893 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:45:40.893 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:45:41.032 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:45:41.032 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:45:41.032 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:45:41.032 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:45:41.032 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:45:41.032 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:45:41.032 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:45:41.032 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:45:41.033 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:45:41.033 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:45:41.033 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:45:41.033 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:45:41.034 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:45:41.035 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:45:41.036 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:45:41.036 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:45:41.106 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 33 2024-04-06T09:45:41.106 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 33 2024-04-06T09:45:41.106 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:45:41.106 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=133, tag_tid=21, entry_tid=1], [object_number=132, tag_tid=21, entry_tid=0], [object_number=135, tag_tid=20, entry_tid=19939], [object_number=134, tag_tid=20, entry_tid=19938]]], state=connected] 2024-04-06T09:45:41.107 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=134, tag_tid=20, entry_tid=19222], [object_number=133, tag_tid=20, entry_tid=19221], [object_number=132, tag_tid=20, entry_tid=19220], [object_number=135, tag_tid=20, entry_tid=19219]]], state=connected] 2024-04-06T09:45:41.109 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:45:41.112 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=133, tag_tid=21, entry_tid=1' 2024-04-06T09:45:41.112 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:45:41.112 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:45:41.112 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:45:41.112 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:45:41.112 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:45:41.112 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:45:41.112 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:45:41.112 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:45:41.113 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:45:41.113 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:45:41.113 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:45:41.113 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:45:41.113 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:45:41.113 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:45:41.114 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:45:41.160 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 33 2024-04-06T09:45:41.160 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 33 2024-04-06T09:45:41.161 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:45:41.161 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=133, tag_tid=21, entry_tid=1], [object_number=132, tag_tid=21, entry_tid=0], [object_number=135, tag_tid=20, entry_tid=19939], [object_number=134, tag_tid=20, entry_tid=19938]]], state=connected] 2024-04-06T09:45:41.161 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=134, tag_tid=20, entry_tid=19222], [object_number=133, tag_tid=20, entry_tid=19221], [object_number=132, tag_tid=20, entry_tid=19220], [object_number=135, tag_tid=20, entry_tid=19219]]], state=connected] 2024-04-06T09:45:41.164 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:45:41.166 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=134, tag_tid=20, entry_tid=19222' 2024-04-06T09:45:41.166 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=133, tag_tid=21, entry_tid=1' -a 'object_number=133, tag_tid=21, entry_tid=1' = 'object_number=134, tag_tid=20, entry_tid=19222' 2024-04-06T09:45:41.166 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=134, tag_tid=20, entry_tid=19222' '!=' 'object_number=129, tag_tid=20, entry_tid=19145' 2024-04-06T09:45:41.167 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:45:41.167 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=134, tag_tid=20, entry_tid=19222' = 'object_number=129, tag_tid=20, entry_tid=19145' 2024-04-06T09:45:41.167 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=134, tag_tid=20, entry_tid=19222' 2024-04-06T09:45:41.167 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=133, tag_tid=21, entry_tid=1' 2024-04-06T09:45:41.167 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:45:41.167 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:45:41.167 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=21 2024-04-06T09:45:41.168 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=134, tag_tid=20, entry_tid=19222' 2024-04-06T09:45:41.168 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:45:41.168 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:45:41.169 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=20 2024-04-06T09:45:41.170 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=133, tag_tid=21, entry_tid=1' 2024-04-06T09:45:41.170 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:45:41.170 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:45:41.171 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:45:41.171 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=134, tag_tid=20, entry_tid=19222' 2024-04-06T09:45:41.171 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:45:41.172 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:45:41.173 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=19222 2024-04-06T09:45:41.173 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 21 = 20 -a 1 -le 19222 2024-04-06T09:45:41.173 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:45:41.173 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:45:41.173 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:45:41.375 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:45:41.375 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:45:41.375 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:45:41.376 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:45:41.376 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:45:41.376 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:45:41.376 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:45:41.376 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:45:41.376 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:45:41.377 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:45:41.377 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:45:41.377 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:45:41.377 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:45:41.377 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:45:41.377 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:45:41.382 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:45:41.382 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:45:41.382 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:45:41.382 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:45:41.382 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:45:41.382 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:45:41.382 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:45:41.383 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:45:41.383 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:45:41.383 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:45:41.383 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:45:41.383 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:45:41.383 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:45:41.383 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:45:41.383 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:45:41.383 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:45:41.383 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:45:41.383 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:45:41.383 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:45:41.384 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:45:41.384 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:45:41.384 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:45:41.384 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:45:41.384 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:45:41.384 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:45:41.384 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:45:41.384 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:45:41.384 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:45:41.384 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:45:41.384 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:45:41.384 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:45:41.384 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:45:41.384 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:45:41.384 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:45:41.406 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:45:41.407 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:45:41.407 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:45:41.407 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:45:41.538 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:45:41.539 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:45:41.539 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:45:41.539 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:45:41.539 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:45:41.539 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:45:41.540 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:45:41.540 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:45:41.540 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:45:41.540 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:45:41.540 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:45:41.540 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:45:41.541 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:45:41.543 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:45:41.543 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:45:41.543 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:45:41.590 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 33 2024-04-06T09:45:41.591 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 33 2024-04-06T09:45:41.591 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:45:41.591 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=133, tag_tid=21, entry_tid=1], [object_number=132, tag_tid=21, entry_tid=0], [object_number=135, tag_tid=20, entry_tid=19939], [object_number=134, tag_tid=20, entry_tid=19938]]], state=connected] 2024-04-06T09:45:41.591 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=132, tag_tid=20, entry_tid=19304], [object_number=135, tag_tid=20, entry_tid=19303], [object_number=134, tag_tid=20, entry_tid=19302], [object_number=133, tag_tid=20, entry_tid=19301]]], state=connected] 2024-04-06T09:45:41.593 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:45:41.595 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=133, tag_tid=21, entry_tid=1' 2024-04-06T09:45:41.596 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:45:41.596 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:45:41.596 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:45:41.596 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:45:41.596 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:45:41.597 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:45:41.597 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:45:41.597 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:45:41.597 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:45:41.597 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:45:41.597 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:45:41.597 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:45:41.597 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:45:41.598 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:45:41.598 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:45:41.644 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 33 2024-04-06T09:45:41.644 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 33 2024-04-06T09:45:41.644 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:45:41.644 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=133, tag_tid=21, entry_tid=1], [object_number=132, tag_tid=21, entry_tid=0], [object_number=135, tag_tid=20, entry_tid=19939], [object_number=134, tag_tid=20, entry_tid=19938]]], state=connected] 2024-04-06T09:45:41.644 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=132, tag_tid=20, entry_tid=19304], [object_number=135, tag_tid=20, entry_tid=19303], [object_number=134, tag_tid=20, entry_tid=19302], [object_number=133, tag_tid=20, entry_tid=19301]]], state=connected] 2024-04-06T09:45:41.648 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:45:41.650 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=132, tag_tid=20, entry_tid=19304' 2024-04-06T09:45:41.650 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=133, tag_tid=21, entry_tid=1' -a 'object_number=133, tag_tid=21, entry_tid=1' = 'object_number=132, tag_tid=20, entry_tid=19304' 2024-04-06T09:45:41.650 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=132, tag_tid=20, entry_tid=19304' '!=' 'object_number=134, tag_tid=20, entry_tid=19222' 2024-04-06T09:45:41.650 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:45:41.650 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=132, tag_tid=20, entry_tid=19304' = 'object_number=134, tag_tid=20, entry_tid=19222' 2024-04-06T09:45:41.650 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=132, tag_tid=20, entry_tid=19304' 2024-04-06T09:45:41.650 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=133, tag_tid=21, entry_tid=1' 2024-04-06T09:45:41.650 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:45:41.650 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:45:41.651 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=21 2024-04-06T09:45:41.652 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=132, tag_tid=20, entry_tid=19304' 2024-04-06T09:45:41.652 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:45:41.652 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:45:41.653 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=20 2024-04-06T09:45:41.653 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=133, tag_tid=21, entry_tid=1' 2024-04-06T09:45:41.653 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:45:41.654 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:45:41.654 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:45:41.655 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=132, tag_tid=20, entry_tid=19304' 2024-04-06T09:45:41.655 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:45:41.655 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:45:41.656 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=19304 2024-04-06T09:45:41.656 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 21 = 20 -a 1 -le 19304 2024-04-06T09:45:41.657 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:45:41.657 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:45:41.657 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:45:41.858 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:45:41.859 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:45:41.859 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:45:41.859 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:45:41.859 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:45:41.859 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:45:41.859 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:45:41.859 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:45:41.859 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:45:41.859 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:45:41.859 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:45:41.859 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:45:41.860 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:45:41.860 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:45:41.860 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:45:41.860 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:45:41.861 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:45:41.863 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:45:41.863 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:45:41.863 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:45:41.863 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:45:41.863 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:45:41.864 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:45:41.864 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:45:41.864 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:45:41.864 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:45:41.864 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:45:41.864 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:45:41.864 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:45:41.864 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:45:41.864 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:45:41.864 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:45:41.864 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:45:41.864 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:45:41.864 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:45:41.864 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:45:41.865 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:45:41.865 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:45:41.865 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:45:41.865 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:45:41.865 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:45:41.865 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:45:41.865 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:45:41.865 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:45:41.865 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:45:41.865 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:45:41.865 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:45:41.865 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:45:41.865 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:45:41.888 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:45:41.889 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:45:41.889 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:45:41.889 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:45:42.023 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:45:42.024 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:45:42.024 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:45:42.024 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:45:42.024 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:45:42.024 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:45:42.024 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:45:42.024 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:45:42.025 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:45:42.025 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:45:42.025 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:45:42.025 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:45:42.025 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:45:42.027 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:45:42.027 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:45:42.027 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:45:42.081 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 33 2024-04-06T09:45:42.081 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 33 2024-04-06T09:45:42.081 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:45:42.081 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=133, tag_tid=21, entry_tid=1], [object_number=132, tag_tid=21, entry_tid=0], [object_number=135, tag_tid=20, entry_tid=19939], [object_number=134, tag_tid=20, entry_tid=19938]]], state=connected] 2024-04-06T09:45:42.081 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=132, tag_tid=20, entry_tid=19388], [object_number=135, tag_tid=20, entry_tid=19387], [object_number=134, tag_tid=20, entry_tid=19386], [object_number=133, tag_tid=20, entry_tid=19385]]], state=connected] 2024-04-06T09:45:42.084 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:45:42.085 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=133, tag_tid=21, entry_tid=1' 2024-04-06T09:45:42.087 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:45:42.087 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:45:42.087 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:45:42.087 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:45:42.087 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:45:42.087 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:45:42.087 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:45:42.087 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:45:42.087 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:45:42.088 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:45:42.088 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:45:42.088 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:45:42.088 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:45:42.089 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:45:42.089 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:45:42.135 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 33 2024-04-06T09:45:42.135 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 33 2024-04-06T09:45:42.135 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:45:42.135 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=133, tag_tid=21, entry_tid=1], [object_number=132, tag_tid=21, entry_tid=0], [object_number=135, tag_tid=20, entry_tid=19939], [object_number=134, tag_tid=20, entry_tid=19938]]], state=connected] 2024-04-06T09:45:42.136 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=132, tag_tid=20, entry_tid=19388], [object_number=135, tag_tid=20, entry_tid=19387], [object_number=134, tag_tid=20, entry_tid=19386], [object_number=133, tag_tid=20, entry_tid=19385]]], state=connected] 2024-04-06T09:45:42.139 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:45:42.141 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=132, tag_tid=20, entry_tid=19388' 2024-04-06T09:45:42.141 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=133, tag_tid=21, entry_tid=1' -a 'object_number=133, tag_tid=21, entry_tid=1' = 'object_number=132, tag_tid=20, entry_tid=19388' 2024-04-06T09:45:42.141 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=132, tag_tid=20, entry_tid=19388' '!=' 'object_number=132, tag_tid=20, entry_tid=19304' 2024-04-06T09:45:42.141 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:45:42.141 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=132, tag_tid=20, entry_tid=19388' = 'object_number=132, tag_tid=20, entry_tid=19304' 2024-04-06T09:45:42.141 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=132, tag_tid=20, entry_tid=19388' 2024-04-06T09:45:42.141 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=133, tag_tid=21, entry_tid=1' 2024-04-06T09:45:42.141 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:45:42.141 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:45:42.142 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=21 2024-04-06T09:45:42.143 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=132, tag_tid=20, entry_tid=19388' 2024-04-06T09:45:42.143 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:45:42.143 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:45:42.144 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=20 2024-04-06T09:45:42.144 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=133, tag_tid=21, entry_tid=1' 2024-04-06T09:45:42.144 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:45:42.145 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:45:42.145 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:45:42.146 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=132, tag_tid=20, entry_tid=19388' 2024-04-06T09:45:42.146 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:45:42.146 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:45:42.147 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=19388 2024-04-06T09:45:42.147 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 21 = 20 -a 1 -le 19388 2024-04-06T09:45:42.148 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:45:42.148 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:45:42.148 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:45:42.349 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:45:42.350 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:45:42.350 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:45:42.350 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:45:42.350 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:45:42.350 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:45:42.350 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:45:42.350 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:45:42.350 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:45:42.350 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:45:42.350 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:45:42.350 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:45:42.351 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:45:42.351 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:45:42.351 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:45:42.351 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:45:42.351 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:45:42.354 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:45:42.354 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:45:42.354 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:45:42.354 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:45:42.354 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:45:42.354 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:45:42.354 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:45:42.354 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:45:42.355 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:45:42.355 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:45:42.355 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:45:42.355 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:45:42.355 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:45:42.355 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:45:42.355 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:45:42.355 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:45:42.355 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:45:42.355 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:45:42.355 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:45:42.355 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:45:42.355 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:45:42.355 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:45:42.356 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:45:42.356 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:45:42.356 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:45:42.356 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:45:42.356 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:45:42.356 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:45:42.356 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:45:42.356 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:45:42.356 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:45:42.356 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:45:42.381 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:45:42.382 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:45:42.382 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:45:42.382 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:45:42.518 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:45:42.519 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:45:42.519 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:45:42.519 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:45:42.519 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:45:42.519 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:45:42.519 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:45:42.519 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:45:42.519 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:45:42.519 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:45:42.519 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:45:42.519 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:45:42.519 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:45:42.520 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:45:42.520 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:45:42.520 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:45:42.571 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 33 2024-04-06T09:45:42.571 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 33 2024-04-06T09:45:42.571 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:45:42.571 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=133, tag_tid=21, entry_tid=1], [object_number=132, tag_tid=21, entry_tid=0], [object_number=135, tag_tid=20, entry_tid=19939], [object_number=134, tag_tid=20, entry_tid=19938]]], state=connected] 2024-04-06T09:45:42.571 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=133, tag_tid=20, entry_tid=19477], [object_number=132, tag_tid=20, entry_tid=19476], [object_number=135, tag_tid=20, entry_tid=19475], [object_number=134, tag_tid=20, entry_tid=19474]]], state=connected] 2024-04-06T09:45:42.574 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:45:42.576 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=133, tag_tid=21, entry_tid=1' 2024-04-06T09:45:42.577 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:45:42.577 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:45:42.577 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:45:42.577 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:45:42.577 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:45:42.577 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:45:42.577 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:45:42.578 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:45:42.578 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:45:42.578 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:45:42.578 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:45:42.578 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:45:42.579 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:45:42.579 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:45:42.579 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:45:42.625 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 33 2024-04-06T09:45:42.625 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 33 2024-04-06T09:45:42.626 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:45:42.626 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=133, tag_tid=21, entry_tid=1], [object_number=132, tag_tid=21, entry_tid=0], [object_number=135, tag_tid=20, entry_tid=19939], [object_number=134, tag_tid=20, entry_tid=19938]]], state=connected] 2024-04-06T09:45:42.626 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=133, tag_tid=20, entry_tid=19477], [object_number=132, tag_tid=20, entry_tid=19476], [object_number=135, tag_tid=20, entry_tid=19475], [object_number=134, tag_tid=20, entry_tid=19474]]], state=connected] 2024-04-06T09:45:42.629 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:45:42.631 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=133, tag_tid=20, entry_tid=19477' 2024-04-06T09:45:42.631 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=133, tag_tid=21, entry_tid=1' -a 'object_number=133, tag_tid=21, entry_tid=1' = 'object_number=133, tag_tid=20, entry_tid=19477' 2024-04-06T09:45:42.632 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=133, tag_tid=20, entry_tid=19477' '!=' 'object_number=132, tag_tid=20, entry_tid=19388' 2024-04-06T09:45:42.632 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:45:42.632 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=133, tag_tid=20, entry_tid=19477' = 'object_number=132, tag_tid=20, entry_tid=19388' 2024-04-06T09:45:42.632 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=133, tag_tid=20, entry_tid=19477' 2024-04-06T09:45:42.632 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=133, tag_tid=21, entry_tid=1' 2024-04-06T09:45:42.632 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:45:42.632 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:45:42.632 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=21 2024-04-06T09:45:42.633 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=133, tag_tid=20, entry_tid=19477' 2024-04-06T09:45:42.633 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:45:42.633 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:45:42.634 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=20 2024-04-06T09:45:42.635 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=133, tag_tid=21, entry_tid=1' 2024-04-06T09:45:42.635 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:45:42.635 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:45:42.636 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:45:42.636 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=133, tag_tid=20, entry_tid=19477' 2024-04-06T09:45:42.637 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:45:42.637 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:45:42.638 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=19477 2024-04-06T09:45:42.638 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 21 = 20 -a 1 -le 19477 2024-04-06T09:45:42.638 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:45:42.638 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:45:42.638 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:45:42.839 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:45:42.840 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:45:42.840 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:45:42.840 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:45:42.840 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:45:42.840 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:45:42.840 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:45:42.840 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:45:42.840 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:45:42.840 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:45:42.840 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:45:42.840 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:45:42.840 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:45:42.841 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:45:42.841 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:45:42.841 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:45:42.841 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:45:42.843 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:45:42.843 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:45:42.844 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:45:42.844 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:45:42.844 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:45:42.844 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:45:42.844 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:45:42.844 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:45:42.844 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:45:42.844 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:45:42.844 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:45:42.844 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:45:42.844 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:45:42.845 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:45:42.845 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:45:42.845 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:45:42.845 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:45:42.845 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:45:42.845 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:45:42.845 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:45:42.845 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:45:42.845 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:45:42.845 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:45:42.845 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:45:42.845 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:45:42.845 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:45:42.845 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:45:42.845 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:45:42.846 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:45:42.846 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:45:42.846 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:45:42.846 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:45:42.872 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:45:42.872 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:45:42.872 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:45:42.872 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:45:43.008 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:45:43.009 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:45:43.009 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:45:43.009 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:45:43.009 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:45:43.010 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:45:43.010 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:45:43.010 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:45:43.010 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:45:43.010 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:45:43.010 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:45:43.011 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:45:43.011 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:45:43.013 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:45:43.014 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:45:43.014 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:45:43.060 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 33 2024-04-06T09:45:43.060 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 33 2024-04-06T09:45:43.060 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:45:43.060 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=133, tag_tid=21, entry_tid=1], [object_number=132, tag_tid=21, entry_tid=0], [object_number=135, tag_tid=20, entry_tid=19939], [object_number=134, tag_tid=20, entry_tid=19938]]], state=connected] 2024-04-06T09:45:43.061 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=133, tag_tid=20, entry_tid=19561], [object_number=132, tag_tid=20, entry_tid=19560], [object_number=135, tag_tid=20, entry_tid=19559], [object_number=134, tag_tid=20, entry_tid=19558]]], state=connected] 2024-04-06T09:45:43.064 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:45:43.065 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=133, tag_tid=21, entry_tid=1' 2024-04-06T09:45:43.067 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:45:43.067 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:45:43.067 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:45:43.067 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:45:43.067 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:45:43.067 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:45:43.067 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:45:43.067 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:45:43.068 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:45:43.068 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:45:43.068 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:45:43.068 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:45:43.068 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:45:43.068 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:45:43.069 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:45:43.114 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 33 2024-04-06T09:45:43.114 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 33 2024-04-06T09:45:43.114 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:45:43.114 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=133, tag_tid=21, entry_tid=1], [object_number=132, tag_tid=21, entry_tid=0], [object_number=135, tag_tid=20, entry_tid=19939], [object_number=134, tag_tid=20, entry_tid=19938]]], state=connected] 2024-04-06T09:45:43.114 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=133, tag_tid=20, entry_tid=19561], [object_number=132, tag_tid=20, entry_tid=19560], [object_number=135, tag_tid=20, entry_tid=19559], [object_number=134, tag_tid=20, entry_tid=19558]]], state=connected] 2024-04-06T09:45:43.118 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:45:43.120 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=133, tag_tid=20, entry_tid=19561' 2024-04-06T09:45:43.120 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=133, tag_tid=21, entry_tid=1' -a 'object_number=133, tag_tid=21, entry_tid=1' = 'object_number=133, tag_tid=20, entry_tid=19561' 2024-04-06T09:45:43.120 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=133, tag_tid=20, entry_tid=19561' '!=' 'object_number=133, tag_tid=20, entry_tid=19477' 2024-04-06T09:45:43.120 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:45:43.120 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=133, tag_tid=20, entry_tid=19561' = 'object_number=133, tag_tid=20, entry_tid=19477' 2024-04-06T09:45:43.120 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=133, tag_tid=20, entry_tid=19561' 2024-04-06T09:45:43.120 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=133, tag_tid=21, entry_tid=1' 2024-04-06T09:45:43.121 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:45:43.121 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:45:43.121 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=21 2024-04-06T09:45:43.122 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=133, tag_tid=20, entry_tid=19561' 2024-04-06T09:45:43.122 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:45:43.122 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:45:43.122 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=20 2024-04-06T09:45:43.123 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=133, tag_tid=21, entry_tid=1' 2024-04-06T09:45:43.123 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:45:43.123 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:45:43.124 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:45:43.125 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=133, tag_tid=20, entry_tid=19561' 2024-04-06T09:45:43.125 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:45:43.125 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:45:43.126 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=19561 2024-04-06T09:45:43.126 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 21 = 20 -a 1 -le 19561 2024-04-06T09:45:43.126 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:45:43.126 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:45:43.126 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:45:43.328 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:45:43.328 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:45:43.328 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:45:43.328 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:45:43.328 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:45:43.329 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:45:43.329 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:45:43.329 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:45:43.329 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:45:43.329 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:45:43.329 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:45:43.329 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:45:43.329 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:45:43.329 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:45:43.329 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:45:43.330 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:45:43.330 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:45:43.333 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:45:43.333 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:45:43.333 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:45:43.333 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:45:43.333 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:45:43.333 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:45:43.333 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:45:43.334 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:45:43.334 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:45:43.334 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:45:43.334 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:45:43.334 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:45:43.334 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:45:43.334 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:45:43.334 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:45:43.334 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:45:43.334 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:45:43.334 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:45:43.334 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:45:43.334 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:45:43.334 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:45:43.335 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:45:43.335 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:45:43.335 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:45:43.335 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:45:43.335 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:45:43.335 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:45:43.335 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:45:43.335 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:45:43.335 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:45:43.335 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:45:43.335 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:45:43.358 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:45:43.359 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:45:43.359 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:45:43.359 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:45:43.491 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:45:43.491 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:45:43.492 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:45:43.492 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:45:43.492 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:45:43.492 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:45:43.492 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:45:43.492 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:45:43.492 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:45:43.492 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:45:43.492 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:45:43.493 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:45:43.493 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:45:43.494 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:45:43.495 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:45:43.495 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:45:43.555 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 33 2024-04-06T09:45:43.556 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 33 2024-04-06T09:45:43.556 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:45:43.556 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=133, tag_tid=21, entry_tid=1], [object_number=132, tag_tid=21, entry_tid=0], [object_number=135, tag_tid=20, entry_tid=19939], [object_number=134, tag_tid=20, entry_tid=19938]]], state=connected] 2024-04-06T09:45:43.556 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=134, tag_tid=20, entry_tid=19646], [object_number=133, tag_tid=20, entry_tid=19645], [object_number=132, tag_tid=20, entry_tid=19644], [object_number=135, tag_tid=20, entry_tid=19643]]], state=connected] 2024-04-06T09:45:43.559 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:45:43.560 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=133, tag_tid=21, entry_tid=1' 2024-04-06T09:45:43.561 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:45:43.561 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:45:43.561 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:45:43.561 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:45:43.562 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:45:43.562 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:45:43.562 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:45:43.562 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:45:43.562 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:45:43.562 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:45:43.562 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:45:43.562 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:45:43.562 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:45:43.563 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:45:43.563 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:45:43.612 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 33 2024-04-06T09:45:43.612 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 33 2024-04-06T09:45:43.612 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:45:43.612 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=133, tag_tid=21, entry_tid=1], [object_number=132, tag_tid=21, entry_tid=0], [object_number=135, tag_tid=20, entry_tid=19939], [object_number=134, tag_tid=20, entry_tid=19938]]], state=connected] 2024-04-06T09:45:43.612 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=134, tag_tid=20, entry_tid=19646], [object_number=133, tag_tid=20, entry_tid=19645], [object_number=132, tag_tid=20, entry_tid=19644], [object_number=135, tag_tid=20, entry_tid=19643]]], state=connected] 2024-04-06T09:45:43.615 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:45:43.618 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=134, tag_tid=20, entry_tid=19646' 2024-04-06T09:45:43.618 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=133, tag_tid=21, entry_tid=1' -a 'object_number=133, tag_tid=21, entry_tid=1' = 'object_number=134, tag_tid=20, entry_tid=19646' 2024-04-06T09:45:43.618 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=134, tag_tid=20, entry_tid=19646' '!=' 'object_number=133, tag_tid=20, entry_tid=19561' 2024-04-06T09:45:43.618 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:45:43.618 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=134, tag_tid=20, entry_tid=19646' = 'object_number=133, tag_tid=20, entry_tid=19561' 2024-04-06T09:45:43.618 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=134, tag_tid=20, entry_tid=19646' 2024-04-06T09:45:43.618 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=133, tag_tid=21, entry_tid=1' 2024-04-06T09:45:43.618 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:45:43.618 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:45:43.619 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=21 2024-04-06T09:45:43.619 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=134, tag_tid=20, entry_tid=19646' 2024-04-06T09:45:43.619 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:45:43.620 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:45:43.620 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=20 2024-04-06T09:45:43.621 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=133, tag_tid=21, entry_tid=1' 2024-04-06T09:45:43.621 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:45:43.621 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:45:43.622 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:45:43.623 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=134, tag_tid=20, entry_tid=19646' 2024-04-06T09:45:43.623 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:45:43.623 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:45:43.624 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=19646 2024-04-06T09:45:43.624 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 21 = 20 -a 1 -le 19646 2024-04-06T09:45:43.624 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:45:43.624 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:45:43.624 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:45:43.826 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:45:43.826 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:45:43.826 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:45:43.826 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:45:43.826 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:45:43.827 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:45:43.827 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:45:43.827 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:45:43.827 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:45:43.827 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:45:43.827 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:45:43.827 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:45:43.827 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:45:43.827 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:45:43.827 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:45:43.828 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:45:43.828 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:45:43.831 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:45:43.831 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:45:43.831 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:45:43.831 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:45:43.831 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:45:43.831 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:45:43.831 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:45:43.831 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:45:43.831 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:45:43.832 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:45:43.832 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:45:43.832 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:45:43.832 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:45:43.832 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:45:43.832 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:45:43.832 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:45:43.833 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:45:43.833 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:45:43.833 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:45:43.834 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:45:43.834 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:45:43.834 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:45:43.834 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:45:43.834 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:45:43.834 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:45:43.834 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:45:43.834 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:45:43.834 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:45:43.834 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:45:43.834 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:45:43.834 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:45:43.834 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:45:43.858 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:45:43.858 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:45:43.858 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:45:43.858 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:45:43.989 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:45:43.990 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:45:43.990 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:45:43.990 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:45:43.990 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:45:43.990 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:45:43.990 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:45:43.990 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:45:43.990 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:45:43.990 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:45:43.991 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:45:43.991 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:45:43.991 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:45:43.993 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:45:43.994 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:45:43.994 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:45:44.040 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 33 2024-04-06T09:45:44.040 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 33 2024-04-06T09:45:44.040 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:45:44.040 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=133, tag_tid=21, entry_tid=1], [object_number=132, tag_tid=21, entry_tid=0], [object_number=135, tag_tid=20, entry_tid=19939], [object_number=134, tag_tid=20, entry_tid=19938]]], state=connected] 2024-04-06T09:45:44.040 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=133, tag_tid=20, entry_tid=19741], [object_number=132, tag_tid=20, entry_tid=19740], [object_number=135, tag_tid=20, entry_tid=19739], [object_number=134, tag_tid=20, entry_tid=19738]]], state=connected] 2024-04-06T09:45:44.043 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:45:44.045 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=133, tag_tid=21, entry_tid=1' 2024-04-06T09:45:44.046 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:45:44.046 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:45:44.046 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:45:44.047 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:45:44.047 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:45:44.047 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:45:44.047 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:45:44.047 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:45:44.047 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:45:44.047 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:45:44.047 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:45:44.047 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:45:44.048 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:45:44.048 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:45:44.048 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:45:44.094 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 33 2024-04-06T09:45:44.094 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 33 2024-04-06T09:45:44.094 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:45:44.095 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=133, tag_tid=21, entry_tid=1], [object_number=132, tag_tid=21, entry_tid=0], [object_number=135, tag_tid=20, entry_tid=19939], [object_number=134, tag_tid=20, entry_tid=19938]]], state=connected] 2024-04-06T09:45:44.095 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=133, tag_tid=20, entry_tid=19741], [object_number=132, tag_tid=20, entry_tid=19740], [object_number=135, tag_tid=20, entry_tid=19739], [object_number=134, tag_tid=20, entry_tid=19738]]], state=connected] 2024-04-06T09:45:44.098 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:45:44.100 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=133, tag_tid=20, entry_tid=19741' 2024-04-06T09:45:44.100 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=133, tag_tid=21, entry_tid=1' -a 'object_number=133, tag_tid=21, entry_tid=1' = 'object_number=133, tag_tid=20, entry_tid=19741' 2024-04-06T09:45:44.100 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=133, tag_tid=20, entry_tid=19741' '!=' 'object_number=134, tag_tid=20, entry_tid=19646' 2024-04-06T09:45:44.100 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:45:44.100 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=133, tag_tid=20, entry_tid=19741' = 'object_number=134, tag_tid=20, entry_tid=19646' 2024-04-06T09:45:44.100 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=133, tag_tid=20, entry_tid=19741' 2024-04-06T09:45:44.101 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=133, tag_tid=21, entry_tid=1' 2024-04-06T09:45:44.101 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:45:44.101 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:45:44.101 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=21 2024-04-06T09:45:44.102 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=133, tag_tid=20, entry_tid=19741' 2024-04-06T09:45:44.102 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:45:44.102 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:45:44.103 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=20 2024-04-06T09:45:44.103 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=133, tag_tid=21, entry_tid=1' 2024-04-06T09:45:44.104 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:45:44.104 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:45:44.104 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:45:44.105 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=133, tag_tid=20, entry_tid=19741' 2024-04-06T09:45:44.105 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:45:44.105 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:45:44.106 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=19741 2024-04-06T09:45:44.107 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 21 = 20 -a 1 -le 19741 2024-04-06T09:45:44.107 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:45:44.107 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:45:44.107 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:45:44.308 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:45:44.308 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:45:44.309 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:45:44.309 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:45:44.309 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:45:44.309 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:45:44.309 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:45:44.309 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:45:44.309 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:45:44.309 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:45:44.309 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:45:44.309 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:45:44.309 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:45:44.309 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:45:44.309 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:45:44.310 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:45:44.310 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:45:44.312 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:45:44.312 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:45:44.312 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:45:44.312 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:45:44.312 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:45:44.312 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:45:44.312 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:45:44.312 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:45:44.312 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:45:44.312 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:45:44.313 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:45:44.313 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:45:44.313 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:45:44.313 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:45:44.313 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:45:44.313 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:45:44.314 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:45:44.314 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:45:44.315 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:45:44.315 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:45:44.315 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:45:44.315 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:45:44.315 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:45:44.315 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:45:44.315 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:45:44.315 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:45:44.315 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:45:44.315 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:45:44.315 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:45:44.315 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:45:44.315 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:45:44.315 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:45:44.339 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:45:44.340 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:45:44.340 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:45:44.340 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:45:44.477 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:45:44.478 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:45:44.478 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:45:44.479 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:45:44.479 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:45:44.479 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:45:44.479 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:45:44.479 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:45:44.479 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:45:44.479 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:45:44.479 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:45:44.479 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:45:44.480 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:45:44.481 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:45:44.481 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:45:44.481 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:45:44.745 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 33 2024-04-06T09:45:44.745 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 33 2024-04-06T09:45:44.745 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:45:44.745 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=133, tag_tid=21, entry_tid=1], [object_number=132, tag_tid=21, entry_tid=0], [object_number=135, tag_tid=20, entry_tid=19939], [object_number=134, tag_tid=20, entry_tid=19938]]], state=connected] 2024-04-06T09:45:44.745 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=135, tag_tid=20, entry_tid=19827], [object_number=134, tag_tid=20, entry_tid=19826], [object_number=133, tag_tid=20, entry_tid=19825], [object_number=132, tag_tid=20, entry_tid=19824]]], state=connected] 2024-04-06T09:45:44.750 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:45:44.751 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=133, tag_tid=21, entry_tid=1' 2024-04-06T09:45:44.753 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:45:44.753 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:45:44.753 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:45:44.753 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:45:44.753 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:45:44.753 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:45:44.753 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:45:44.753 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:45:44.753 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:45:44.753 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:45:44.753 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:45:44.753 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:45:44.754 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:45:44.755 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:45:44.755 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:45:44.805 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 33 2024-04-06T09:45:44.805 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 33 2024-04-06T09:45:44.805 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:45:44.806 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=133, tag_tid=21, entry_tid=1], [object_number=132, tag_tid=21, entry_tid=0], [object_number=135, tag_tid=20, entry_tid=19939], [object_number=134, tag_tid=20, entry_tid=19938]]], state=connected] 2024-04-06T09:45:44.806 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=135, tag_tid=20, entry_tid=19827], [object_number=134, tag_tid=20, entry_tid=19826], [object_number=133, tag_tid=20, entry_tid=19825], [object_number=132, tag_tid=20, entry_tid=19824]]], state=connected] 2024-04-06T09:45:44.809 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:45:44.811 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=135, tag_tid=20, entry_tid=19827' 2024-04-06T09:45:44.811 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=133, tag_tid=21, entry_tid=1' -a 'object_number=133, tag_tid=21, entry_tid=1' = 'object_number=135, tag_tid=20, entry_tid=19827' 2024-04-06T09:45:44.811 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=135, tag_tid=20, entry_tid=19827' '!=' 'object_number=133, tag_tid=20, entry_tid=19741' 2024-04-06T09:45:44.811 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:45:44.812 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=135, tag_tid=20, entry_tid=19827' = 'object_number=133, tag_tid=20, entry_tid=19741' 2024-04-06T09:45:44.812 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=135, tag_tid=20, entry_tid=19827' 2024-04-06T09:45:44.812 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=133, tag_tid=21, entry_tid=1' 2024-04-06T09:45:44.812 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:45:44.812 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:45:44.813 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=21 2024-04-06T09:45:44.814 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=135, tag_tid=20, entry_tid=19827' 2024-04-06T09:45:44.814 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:45:44.814 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:45:44.815 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=20 2024-04-06T09:45:44.815 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=133, tag_tid=21, entry_tid=1' 2024-04-06T09:45:44.816 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:45:44.816 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:45:44.816 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:45:44.817 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=135, tag_tid=20, entry_tid=19827' 2024-04-06T09:45:44.817 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:45:44.817 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:45:44.819 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=19827 2024-04-06T09:45:44.819 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 21 = 20 -a 1 -le 19827 2024-04-06T09:45:44.819 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:45:44.819 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:45:44.819 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:45:45.021 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:45:45.021 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:45:45.021 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:45:45.021 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:45:45.021 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:45:45.021 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:45:45.021 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:45:45.021 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:45:45.021 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:45:45.021 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:45:45.021 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:45:45.021 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:45:45.021 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:45:45.021 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:45:45.021 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:45:45.022 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:45:45.023 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:45:45.026 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:45:45.026 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:45:45.026 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:45:45.026 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:45:45.026 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:45:45.026 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:45:45.026 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:45:45.026 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:45:45.027 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:45:45.027 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:45:45.027 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:45:45.027 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:45:45.027 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:45:45.027 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:45:45.027 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:45:45.027 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:45:45.027 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:45:45.027 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:45:45.027 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:45:45.027 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:45:45.027 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:45:45.027 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:45:45.027 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:45:45.027 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:45:45.028 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:45:45.028 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:45:45.028 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:45:45.028 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:45:45.028 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:45:45.028 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:45:45.028 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:45:45.028 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:45:45.051 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:45:45.052 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:45:45.052 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:45:45.052 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:45:45.183 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:45:45.184 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:45:45.184 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:45:45.184 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:45:45.184 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:45:45.184 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:45:45.185 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:45:45.185 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:45:45.185 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:45:45.185 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:45:45.185 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:45:45.185 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:45:45.185 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:45:45.187 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:45:45.187 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:45:45.187 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:45:45.234 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 33 2024-04-06T09:45:45.234 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 33 2024-04-06T09:45:45.234 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:45:45.234 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=133, tag_tid=21, entry_tid=1], [object_number=132, tag_tid=21, entry_tid=0], [object_number=135, tag_tid=20, entry_tid=19939], [object_number=134, tag_tid=20, entry_tid=19938]]], state=connected] 2024-04-06T09:45:45.234 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=132, tag_tid=20, entry_tid=19940], [object_number=135, tag_tid=20, entry_tid=19939], [object_number=134, tag_tid=20, entry_tid=19938], [object_number=133, tag_tid=20, entry_tid=19937]]], state=connected] 2024-04-06T09:45:45.238 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:45:45.239 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=133, tag_tid=21, entry_tid=1' 2024-04-06T09:45:45.240 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:45:45.241 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:45:45.241 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:45:45.241 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:45:45.241 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:45:45.241 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:45:45.241 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:45:45.241 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:45:45.241 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:45:45.241 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:45:45.241 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:45:45.241 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:45:45.241 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:45:45.242 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:45:45.242 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:45:45.288 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 33 2024-04-06T09:45:45.289 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 33 2024-04-06T09:45:45.289 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:45:45.289 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=133, tag_tid=21, entry_tid=1], [object_number=132, tag_tid=21, entry_tid=0], [object_number=135, tag_tid=20, entry_tid=19939], [object_number=134, tag_tid=20, entry_tid=19938]]], state=connected] 2024-04-06T09:45:45.289 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=132, tag_tid=20, entry_tid=19940], [object_number=135, tag_tid=20, entry_tid=19939], [object_number=134, tag_tid=20, entry_tid=19938], [object_number=133, tag_tid=20, entry_tid=19937]]], state=connected] 2024-04-06T09:45:45.293 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:45:45.294 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=132, tag_tid=20, entry_tid=19940' 2024-04-06T09:45:45.295 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=133, tag_tid=21, entry_tid=1' -a 'object_number=133, tag_tid=21, entry_tid=1' = 'object_number=132, tag_tid=20, entry_tid=19940' 2024-04-06T09:45:45.295 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=132, tag_tid=20, entry_tid=19940' '!=' 'object_number=135, tag_tid=20, entry_tid=19827' 2024-04-06T09:45:45.295 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:45:45.295 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=132, tag_tid=20, entry_tid=19940' = 'object_number=135, tag_tid=20, entry_tid=19827' 2024-04-06T09:45:45.295 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=132, tag_tid=20, entry_tid=19940' 2024-04-06T09:45:45.295 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=133, tag_tid=21, entry_tid=1' 2024-04-06T09:45:45.295 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:45:45.295 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:45:45.295 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=21 2024-04-06T09:45:45.296 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=132, tag_tid=20, entry_tid=19940' 2024-04-06T09:45:45.296 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:45:45.296 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:45:45.297 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=20 2024-04-06T09:45:45.298 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=133, tag_tid=21, entry_tid=1' 2024-04-06T09:45:45.298 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:45:45.298 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:45:45.299 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:45:45.299 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=132, tag_tid=20, entry_tid=19940' 2024-04-06T09:45:45.300 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:45:45.300 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:45:45.301 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=19940 2024-04-06T09:45:45.301 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 21 = 20 -a 1 -le 19940 2024-04-06T09:45:45.301 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:45:45.301 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:45:45.301 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:45:45.503 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:45:45.503 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:45:45.503 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:45:45.503 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:45:45.503 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:45:45.503 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:45:45.503 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:45:45.503 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:45:45.503 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:45:45.503 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:45:45.503 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:45:45.503 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:45:45.504 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:45:45.504 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:45:45.504 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:45:45.504 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:45:45.504 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:45:45.507 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:45:45.507 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:45:45.507 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:45:45.507 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:45:45.507 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:45:45.507 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:45:45.507 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:45:45.507 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:45:45.507 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:45:45.507 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:45:45.508 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:45:45.508 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:45:45.508 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:45:45.508 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:45:45.508 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:45:45.508 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:45:45.508 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:45:45.508 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:45:45.508 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:45:45.508 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:45:45.508 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:45:45.508 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:45:45.508 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:45:45.508 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:45:45.508 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:45:45.509 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:45:45.509 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:45:45.509 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:45:45.509 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:45:45.509 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:45:45.509 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:45:45.509 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:45:45.532 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:45:45.533 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:45:45.533 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:45:45.533 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:45:45.653 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:45:45.654 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:45:45.654 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:45:45.654 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:45:45.655 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:45:45.655 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:45:45.655 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:45:45.655 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:45:45.655 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:45:45.655 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:45:45.655 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:45:45.656 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:45:45.656 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:45:45.658 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:45:45.659 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:45:45.659 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:45:45.711 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 33 2024-04-06T09:45:45.711 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 33 2024-04-06T09:45:45.711 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:45:45.711 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=133, tag_tid=21, entry_tid=1], [object_number=132, tag_tid=21, entry_tid=0], [object_number=135, tag_tid=20, entry_tid=19939], [object_number=134, tag_tid=20, entry_tid=19938]]], state=connected] 2024-04-06T09:45:45.711 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=132, tag_tid=20, entry_tid=19940], [object_number=135, tag_tid=20, entry_tid=19939], [object_number=134, tag_tid=20, entry_tid=19938], [object_number=133, tag_tid=20, entry_tid=19937]]], state=connected] 2024-04-06T09:45:45.715 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:45:45.716 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=133, tag_tid=21, entry_tid=1' 2024-04-06T09:45:45.717 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:45:45.717 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:45:45.718 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:45:45.718 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:45:45.718 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:45:45.718 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:45:45.718 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:45:45.718 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:45:45.718 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:45:45.718 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:45:45.718 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:45:45.718 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:45:45.719 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:45:45.719 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:45:45.719 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:45:45.766 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 33 2024-04-06T09:45:45.766 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 33 2024-04-06T09:45:45.766 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:45:45.766 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=133, tag_tid=21, entry_tid=1], [object_number=132, tag_tid=21, entry_tid=0], [object_number=135, tag_tid=20, entry_tid=19939], [object_number=134, tag_tid=20, entry_tid=19938]]], state=connected] 2024-04-06T09:45:45.767 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=132, tag_tid=20, entry_tid=19940], [object_number=135, tag_tid=20, entry_tid=19939], [object_number=134, tag_tid=20, entry_tid=19938], [object_number=133, tag_tid=20, entry_tid=19937]]], state=connected] 2024-04-06T09:45:45.770 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:45:45.772 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=132, tag_tid=20, entry_tid=19940' 2024-04-06T09:45:45.772 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=133, tag_tid=21, entry_tid=1' -a 'object_number=133, tag_tid=21, entry_tid=1' = 'object_number=132, tag_tid=20, entry_tid=19940' 2024-04-06T09:45:45.772 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=132, tag_tid=20, entry_tid=19940' '!=' 'object_number=132, tag_tid=20, entry_tid=19940' 2024-04-06T09:45:45.772 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:45:45.772 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.4 2024-04-06T09:45:46.174 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:45:46.174 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:45:46.174 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:45:46.174 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:45:46.174 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:45:46.174 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:45:46.174 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:45:46.174 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:45:46.174 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:45:46.174 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:45:46.174 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:45:46.175 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:45:46.175 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:45:46.175 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:45:46.175 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:45:46.175 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:45:46.175 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:45:46.179 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:45:46.179 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:45:46.179 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:45:46.179 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:45:46.179 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:45:46.179 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:45:46.179 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:45:46.179 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:45:46.179 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:45:46.179 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:45:46.179 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:45:46.180 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:45:46.180 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:45:46.180 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:45:46.180 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:45:46.180 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:45:46.180 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:45:46.180 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:45:46.180 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:45:46.180 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:45:46.180 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:45:46.180 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:45:46.180 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:45:46.180 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:45:46.180 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:45:46.181 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:45:46.181 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:45:46.181 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:45:46.181 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:45:46.181 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:45:46.181 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:45:46.181 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:45:46.204 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:45:46.205 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:45:46.205 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:45:46.205 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:45:46.330 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:45:46.331 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:45:46.331 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:45:46.332 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:45:46.332 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:45:46.332 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:45:46.332 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:45:46.332 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:45:46.332 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:45:46.332 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:45:46.332 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:45:46.333 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:45:46.333 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:45:46.335 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:45:46.335 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:45:46.335 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:45:46.388 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 33 2024-04-06T09:45:46.388 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 33 2024-04-06T09:45:46.388 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:45:46.388 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=133, tag_tid=21, entry_tid=1], [object_number=132, tag_tid=21, entry_tid=0], [object_number=135, tag_tid=20, entry_tid=19939], [object_number=134, tag_tid=20, entry_tid=19938]]], state=connected] 2024-04-06T09:45:46.388 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=133, tag_tid=21, entry_tid=1], [object_number=132, tag_tid=21, entry_tid=0], [object_number=135, tag_tid=20, entry_tid=19939], [object_number=134, tag_tid=20, entry_tid=19938]]], state=connected] 2024-04-06T09:45:46.391 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:45:46.393 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=133, tag_tid=21, entry_tid=1' 2024-04-06T09:45:46.394 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:45:46.394 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:45:46.394 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:45:46.394 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:45:46.394 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:45:46.394 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:45:46.395 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:45:46.395 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:45:46.395 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:45:46.395 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:45:46.395 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:45:46.395 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:45:46.399 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:45:46.399 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:45:46.399 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:45:46.443 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 33 2024-04-06T09:45:46.443 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 33 2024-04-06T09:45:46.443 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:45:46.443 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=133, tag_tid=21, entry_tid=1], [object_number=132, tag_tid=21, entry_tid=0], [object_number=135, tag_tid=20, entry_tid=19939], [object_number=134, tag_tid=20, entry_tid=19938]]], state=connected] 2024-04-06T09:45:46.443 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=133, tag_tid=21, entry_tid=1], [object_number=132, tag_tid=21, entry_tid=0], [object_number=135, tag_tid=20, entry_tid=19939], [object_number=134, tag_tid=20, entry_tid=19938]]], state=connected] 2024-04-06T09:45:46.446 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:45:46.448 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=133, tag_tid=21, entry_tid=1' 2024-04-06T09:45:46.448 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=133, tag_tid=21, entry_tid=1' -a 'object_number=133, tag_tid=21, entry_tid=1' = 'object_number=133, tag_tid=21, entry_tid=1' 2024-04-06T09:45:46.448 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:45:46.448 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_snap_present cluster1 mirror test snap7 2024-04-06T09:45:46.448 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:45:46.449 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:45:46.449 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:45:46.449 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local snap_name=snap7 2024-04-06T09:45:46.449 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local s 2024-04-06T09:45:46.449 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-04-06T09:45:46.449 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 1 2024-04-06T09:45:47.449 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 info mirror/test@snap7 2024-04-06T09:45:47.519 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:rbd image 'test': 2024-04-06T09:45:47.519 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: size 512 MiB in 128 objects 2024-04-06T09:45:47.519 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: order 22 (4 MiB objects) 2024-04-06T09:45:47.519 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: snapshot_count: 13 2024-04-06T09:45:47.519 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: id: 10a56da0b207 2024-04-06T09:45:47.519 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: block_name_prefix: rbd_data.10a56da0b207 2024-04-06T09:45:47.519 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: format: 2 2024-04-06T09:45:47.520 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: features: layering, exclusive-lock, journaling 2024-04-06T09:45:47.520 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: op_features: 2024-04-06T09:45:47.520 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: flags: 2024-04-06T09:45:47.520 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: create_timestamp: Sat Apr 6 09:31:59 2024 2024-04-06T09:45:47.520 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: access_timestamp: Sat Apr 6 09:43:41 2024 2024-04-06T09:45:47.520 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: modify_timestamp: Sat Apr 6 09:45:12 2024 2024-04-06T09:45:47.520 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: protected: False 2024-04-06T09:45:47.520 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: journal: 10a56da0b207 2024-04-06T09:45:47.520 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: mirroring state: enabled 2024-04-06T09:45:47.520 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: mirroring mode: journal 2024-04-06T09:45:47.520 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: mirroring global id: 8ad5dedc-b3e0-425d-8993-b3e2231fc5ca 2024-04-06T09:45:47.520 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: mirroring primary: false 2024-04-06T09:45:47.524 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:45:47.524 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n snap6-clean ']' 2024-04-06T09:45:47.524 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ compare_image_snaps mirror test snap6-clean 2024-04-06T09:45:47.525 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:45:47.525 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:45:47.525 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local snap_name=snap6-clean 2024-04-06T09:45:47.525 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local ret=0 2024-04-06T09:45:47.525 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local rmt_export=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.export 2024-04-06T09:45:47.525 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loc_export=/tmp/tmp.BhW6xKmROU/cluster1-mirror-test.export 2024-04-06T09:45:47.525 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rm -f /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.export /tmp/tmp.BhW6xKmROU/cluster1-mirror-test.export 2024-04-06T09:45:47.525 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster2 -p mirror export test@snap6-clean /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.export 2024-04-06T09:45:48.125 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-04-06T09:45:48.142 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 -p mirror export test@snap6-clean /tmp/tmp.BhW6xKmROU/cluster1-mirror-test.export 2024-04-06T09:45:49.542 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-04-06T09:45:49.564 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmp /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.export /tmp/tmp.BhW6xKmROU/cluster1-mirror-test.export 2024-04-06T09:45:49.849 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rm -f /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.export /tmp/tmp.BhW6xKmROU/cluster1-mirror-test.export 2024-04-06T09:45:49.984 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:45:49.984 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ compare_image_snaps mirror test snap7 2024-04-06T09:45:49.984 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:45:49.985 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:45:49.985 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local snap_name=snap7 2024-04-06T09:45:49.985 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local ret=0 2024-04-06T09:45:49.985 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local rmt_export=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.export 2024-04-06T09:45:49.985 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loc_export=/tmp/tmp.BhW6xKmROU/cluster1-mirror-test.export 2024-04-06T09:45:49.985 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rm -f /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.export /tmp/tmp.BhW6xKmROU/cluster1-mirror-test.export 2024-04-06T09:45:49.985 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster2 -p mirror export test@snap7 /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.export 2024-04-06T09:45:50.596 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-04-06T09:45:50.609 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 -p mirror export test@snap7 /tmp/tmp.BhW6xKmROU/cluster1-mirror-test.export 2024-04-06T09:45:52.119 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-04-06T09:45:52.145 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmp /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.export /tmp/tmp.BhW6xKmROU/cluster1-mirror-test.export 2024-04-06T09:45:52.423 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rm -f /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.export /tmp/tmp.BhW6xKmROU/cluster1-mirror-test.export 2024-04-06T09:45:52.561 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:45:52.561 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ clean_snap_name=snap7-clean 2024-04-06T09:45:52.561 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ create_snap cluster2 mirror test snap7-clean 2024-04-06T09:45:52.561 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-04-06T09:45:52.561 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:45:52.561 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:45:52.561 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local snap_name=snap7-clean 2024-04-06T09:45:52.561 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap7-clean --debug-rbd=20 --debug-journaler=20 2024-04-06T09:45:53.632 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in `seq 1 10` 2024-04-06T09:45:53.632 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ stress_write_image cluster2 mirror test 2024-04-06T09:45:53.632 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-04-06T09:45:53.632 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:45:53.632 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:45:53.632 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-04-06T09:45:53.635 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local duration=5 2024-04-06T09:45:53.635 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set +e 2024-04-06T09:45:53.635 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ timeout 5s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror test --debug-rbd=20 --debug-journaler=20 2024-04-06T09:45:55.181 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:100 IOs, 48398336 bytes 2024-04-06T09:45:58.341 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:200 IOs, 84904960 bytes 2024-04-06T09:45:58.643 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ error_code=124 2024-04-06T09:45:58.643 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set -e 2024-04-06T09:45:58.643 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 124 -eq 124 ']' 2024-04-06T09:45:58.643 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:45:58.643 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+replaying 2024-04-06T09:45:58.644 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:45:58.644 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:45:58.644 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:45:58.644 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+replaying 2024-04-06T09:45:58.644 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local description_pattern= 2024-04-06T09:45:58.644 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local service_pattern= 2024-04-06T09:45:58.644 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:45:58.644 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 1 2024-04-06T09:45:59.644 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test_status_in_pool_dir cluster1 mirror test up+replaying '' '' 2024-04-06T09:45:59.644 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:45:59.644 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:45:59.644 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:45:59.644 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+replaying 2024-04-06T09:45:59.645 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local description_pattern= 2024-04-06T09:45:59.645 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local service_pattern= 2024-04-06T09:45:59.645 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local status 2024-04-06T09:45:59.645 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ CEPH_ARGS= 2024-04-06T09:45:59.645 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster1 mirror image status mirror/test 2024-04-06T09:45:59.729 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ status='test: 2024-04-06T09:45:59.730 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: global_id: 8ad5dedc-b3e0-425d-8993-b3e2231fc5ca 2024-04-06T09:45:59.730 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: state: up+replaying 2024-04-06T09:45:59.730 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: description: replaying, {"bytes_per_second":2051713.43,"entries_behind_primary":1,"entries_per_second":130.17,"non_primary_position":{"entry_tid":1,"object_number":133,"tag_tid":21},"primary_position":{"entry_tid":1,"object_number":133,"tag_tid":22},"seconds_until_synced":0} 2024-04-06T09:45:59.730 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: service: mirror.0 on smithi192 2024-04-06T09:45:59.730 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: last_update: 2024-04-06 09:45:57 2024-04-06T09:45:59.730 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: peer_sites: 2024-04-06T09:45:59.730 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: name: cluster2 2024-04-06T09:45:59.730 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: state: up+stopped 2024-04-06T09:45:59.730 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: description: local image is primary 2024-04-06T09:45:59.730 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: last_update: 2024-04-06 09:45:44' 2024-04-06T09:45:59.730 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^ state: .*up+replaying' 2024-04-06T09:45:59.731 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: state: up+replaying 2024-04-06T09:45:59.731 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^ description: .*' 2024-04-06T09:45:59.732 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: description: replaying, {"bytes_per_second":2051713.43,"entries_behind_primary":1,"entries_per_second":130.17,"non_primary_position":{"entry_tid":1,"object_number":133,"tag_tid":21},"primary_position":{"entry_tid":1,"object_number":133,"tag_tid":22},"seconds_until_synced":0} 2024-04-06T09:45:59.732 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n '' ']' 2024-04-06T09:45:59.732 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ echo up+replaying 2024-04-06T09:45:59.732 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^up+' 2024-04-06T09:45:59.733 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:up+replaying 2024-04-06T09:45:59.733 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep 'service: *mirror..* on ' 2024-04-06T09:45:59.734 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: service: mirror.0 on smithi192 2024-04-06T09:45:59.734 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local last_update 2024-04-06T09:45:59.735 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-04-06T09:45:59.737 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_update='2024-04-06 09:45:57' 2024-04-06T09:45:59.737 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+replaying '2024-04-06 09:45:57' 2024-04-06T09:45:59.737 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:45:59.737 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:45:59.737 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:45:59.737 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+replaying 2024-04-06T09:45:59.737 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'prev_last_update=2024-04-06 09:45:57' 2024-04-06T09:45:59.737 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local status 2024-04-06T09:45:59.737 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ CEPH_ARGS= 2024-04-06T09:45:59.738 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-04-06T09:45:59.822 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ status='OKOKOK142094268mirror.1smithi19218.2.2-1059-g4d05f4cafalseOK42584261mirror.0smithi19218.2.2-1059-g4d05f4catrueOK43194322mirror.3smithi19218.2.2-1059-g4d05f4cafalseOK44424445mirror.2smithi19218.2.2-1059-g4d05f4cafalseOKtest8ad5dedc-b3e0-425d-8993-b3e2231fc5caup+replayingreplaying, {"bytes_per_second":2051713.43,"entries_behind_primary":1,"entries_per_second":130.17,"non_primary_position":{"entry_tid":1,"object_number":133,"tag_tid":21},"primary_position":{"entry_tid":1,"object_number":133,"tag_tid":22},"seconds_until_synced":0}42584261mirror.0smithi1922024-04-06 09:45:57cluster2c5e3bdc3-c286-4289-8fd0-6b8628b92376up+stoppedlocal image is primary2024-04-06 09:45:44' 2024-04-06T09:45:59.822 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local last_update state 2024-04-06T09:45:59.822 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2024-04-06T09:45:59.825 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_update='2024-04-06 09:45:57' 2024-04-06T09:45:59.825 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2024-04-06T09:45:59.828 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ state=up+replaying 2024-04-06T09:45:59.828 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ echo up+replaying 2024-04-06T09:45:59.828 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep up+replaying 2024-04-06T09:45:59.829 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:up+replaying 2024-04-06T09:45:59.829 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:45:59.830 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:45:59.830 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ snap_name=snap8 2024-04-06T09:45:59.830 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ create_snap cluster2 mirror test snap8 2024-04-06T09:45:59.830 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-04-06T09:45:59.830 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:45:59.830 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:45:59.830 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local snap_name=snap8 2024-04-06T09:45:59.830 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap8 --debug-rbd=20 --debug-journaler=20 2024-04-06T09:46:05.499 INFO:tasks.rbd_mirror_thrash:kill cluster1.client.mirror.0 2024-04-06T09:46:05.500 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Sent signal 15 2024-04-06T09:46:05.500 INFO:tasks.rbd_mirror_thrash:waiting for 9 secs before reviving daemons 2024-04-06T09:46:05.501 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi192.stderr:2024-04-06T09:46:05.502+0000 7f8afa92e640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.0 (PID: 28231) UID: 1000 2024-04-06T09:46:14.501 INFO:tasks.rbd_mirror_thrash:waiting for cluster1.client.mirror.0 2024-04-06T09:46:14.501 INFO:teuthology.orchestra.run:waiting for 600 2024-04-06T09:46:14.501 INFO:tasks.rbd_mirror_thrash:reviving cluster1.client.mirror.0 2024-04-06T09:46:14.502 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Restarting daemon 2024-04-06T09:46:14.502 DEBUG:teuthology.orchestra.run.smithi192:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.0 2024-04-06T09:46:14.505 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Started 2024-04-06T09:46:14.505 INFO:tasks.rbd_mirror_thrash:waiting for 64 secs before thrashing 2024-04-06T09:46:28.702 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_image_replay_started cluster1 mirror test 2024-04-06T09:46:28.702 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:46:28.702 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:46:28.702 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:46:28.703 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_image_replay_state cluster1 mirror test started 2024-04-06T09:46:28.703 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:46:28.703 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:46:28.703 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:46:28.703 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state=started 2024-04-06T09:46:28.703 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local s 2024-04-06T09:46:28.703 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:46:28.703 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 1 2024-04-06T09:46:29.703 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test_image_replay_state cluster1 mirror test started 2024-04-06T09:46:29.704 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:46:29.704 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:46:29.704 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:46:29.704 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local test_state=started 2024-04-06T09:46:29.704 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local status_result 2024-04-06T09:46:29.704 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local current_state=stopped 2024-04-06T09:46:29.705 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -i state 2024-04-06T09:46:29.705 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2024-04-06T09:46:29.705 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_instance=cluster1 2024-04-06T09:46:29.705 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ shift 2024-04-06T09:46:29.705 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:46:29.705 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance=cluster1 2024-04-06T09:46:29.705 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local loop_instance 2024-04-06T09:46:29.705 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:46:29.706 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sleep 0 2024-04-06T09:46:29.707 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:46:29.707 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ seq 0 3 2024-04-06T09:46:29.710 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:46:29.710 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2024-04-06T09:46:29.710 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1:0 2024-04-06T09:46:29.710 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ shift 2024-04-06T09:46:29.710 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:46:29.710 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:46:29.710 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:46:29.710 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=cluster 2024-04-06T09:46:29.711 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:46:29.711 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:46:29.711 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:46:29.711 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:46:29.711 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval cluster=cluster1 2024-04-06T09:46:29.711 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ cluster=cluster1 2024-04-06T09:46:29.711 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:46:29.711 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:46:29.711 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:46:29.711 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local local_cluster=cluster1:0 2024-04-06T09:46:29.711 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local cluster=cluster1 2024-04-06T09:46:29.711 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local instance 2024-04-06T09:46:29.711 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:46:29.711 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local val=cluster1:0 2024-04-06T09:46:29.711 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local cluster_var_name=local_cluster 2024-04-06T09:46:29.712 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local instance_var_name=instance 2024-04-06T09:46:29.712 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ cluster=cluster1 2024-04-06T09:46:29.712 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:46:29.712 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ '[' 0 = cluster1:0 ']' 2024-04-06T09:46:29.712 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ eval local_cluster=cluster1 2024-04-06T09:46:29.712 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++++ local_cluster=cluster1 2024-04-06T09:46:29.712 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ eval instance=0 2024-04-06T09:46:29.713 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++++ instance=0 2024-04-06T09:46:29.713 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:46:29.736 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:46:29.736 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:46:29.736 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:46:29.737 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2024-04-06T09:46:29.841 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-06T09:46:29.841 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:rbd mirror status : get status for rbd mirror 2024-04-06T09:46:29.841 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:admin_socket: invalid command 2024-04-06T09:46:29.845 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:46:29.845 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/test 2024-04-06T09:46:29.845 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1:1 2024-04-06T09:46:29.845 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ shift 2024-04-06T09:46:29.845 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:46:29.845 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-04-06T09:46:29.845 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:1 2024-04-06T09:46:29.845 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=cluster 2024-04-06T09:46:29.845 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:46:29.845 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:46:29.845 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=1 2024-04-06T09:46:29.845 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-06T09:46:29.845 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval cluster=cluster1 2024-04-06T09:46:29.846 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ cluster=cluster1 2024-04-06T09:46:29.846 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=1 2024-04-06T09:46:29.846 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=1 2024-04-06T09:46:29.846 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-04-06T09:46:29.846 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local local_cluster=cluster1:1 2024-04-06T09:46:29.846 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local cluster=cluster1 2024-04-06T09:46:29.846 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local instance 2024-04-06T09:46:29.846 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-06T09:46:29.846 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local val=cluster1:1 2024-04-06T09:46:29.846 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local cluster_var_name=local_cluster 2024-04-06T09:46:29.846 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local instance_var_name=instance 2024-04-06T09:46:29.846 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ cluster=cluster1 2024-04-06T09:46:29.846 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=1 2024-04-06T09:46:29.846 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ '[' 1 = cluster1:1 ']' 2024-04-06T09:46:29.847 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ eval local_cluster=cluster1 2024-04-06T09:46:29.847 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++++ local_cluster=cluster1 2024-04-06T09:46:29.847 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ eval instance=1 2024-04-06T09:46:29.847 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++++ instance=1 2024-04-06T09:46:29.847 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-06T09:46:29.869 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T09:46:29.869 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T09:46:29.869 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T09:46:29.869 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/test 2024-04-06T09:46:29.990 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ return 0 2024-04-06T09:46:29.991 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ status_result=' "state": "Replaying"' 2024-04-06T09:46:29.991 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ echo ' "state": "Replaying"' 2024-04-06T09:46:29.991 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep -i Replaying 2024-04-06T09:46:29.993 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: "state": "Replaying" 2024-04-06T09:46:29.994 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ current_state=started 2024-04-06T09:46:29.994 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test started = started 2024-04-06T09:46:29.994 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:46:29.994 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror test 2024-04-06T09:46:29.994 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local local_cluster=cluster1 2024-04-06T09:46:29.994 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-04-06T09:46:29.994 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:46:29.994 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:46:29.994 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' journal = journal ']' 2024-04-06T09:46:29.995 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_journal_replay_complete cluster1 cluster2 mirror test 2024-04-06T09:46:29.995 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local local_cluster=cluster1 2024-04-06T09:46:29.995 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-04-06T09:46:29.995 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:46:29.995 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:46:29.995 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local s master_pos mirror_pos last_mirror_pos 2024-04-06T09:46:29.995 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local master_tag master_entry mirror_tag mirror_entry 2024-04-06T09:46:29.995 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:46:29.995 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:46:29.995 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:46:30.196 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:46:30.196 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:46:30.196 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:46:30.196 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:46:30.196 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:46:30.196 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:46:30.196 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:46:30.196 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:46:30.196 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:46:30.197 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:46:30.197 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:46:30.197 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:46:30.197 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:46:30.197 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:46:30.197 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:46:30.197 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:46:30.198 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:46:30.201 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:46:30.202 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:46:30.202 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:46:30.202 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:46:30.202 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:46:30.202 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:46:30.202 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:46:30.202 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:46:30.202 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:46:30.202 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:46:30.202 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:46:30.202 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:46:30.203 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:46:30.203 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:46:30.203 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:46:30.203 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:46:30.203 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:46:30.203 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:46:30.203 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:46:30.203 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:46:30.203 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:46:30.203 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:46:30.203 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:46:30.203 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:46:30.204 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:46:30.204 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:46:30.204 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:46:30.204 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:46:30.204 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:46:30.204 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:46:30.204 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:46:30.204 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:46:30.226 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:46:30.227 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:46:30.227 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:46:30.227 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:46:30.333 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-06T09:46:30.333 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:rbd mirror flush : flush rbd mirror 2024-04-06T09:46:30.333 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:admin_socket: invalid command 2024-04-06T09:46:30.336 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:46:30.336 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/test 2024-04-06T09:46:30.336 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:1 2024-04-06T09:46:30.336 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:46:30.337 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:46:30.337 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-06T09:46:30.337 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:1 2024-04-06T09:46:30.337 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:46:30.337 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:46:30.337 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:46:30.337 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=1 2024-04-06T09:46:30.337 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-06T09:46:30.337 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:46:30.337 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:46:30.338 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=1 2024-04-06T09:46:30.338 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=1 2024-04-06T09:46:30.338 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-06T09:46:30.338 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:1 2024-04-06T09:46:30.338 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:46:30.338 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:46:30.338 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-06T09:46:30.338 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:1 2024-04-06T09:46:30.338 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:46:30.338 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:46:30.338 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:46:30.338 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=1 2024-04-06T09:46:30.338 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-06T09:46:30.339 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:46:30.339 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:46:30.339 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=1 2024-04-06T09:46:30.339 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=1 2024-04-06T09:46:30.339 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-06T09:46:30.355 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T09:46:30.356 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T09:46:30.356 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T09:46:30.356 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/test 2024-04-06T09:46:30.495 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:46:30.495 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:46:30.495 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:46:30.495 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:46:30.495 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:46:30.495 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:46:30.495 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:46:30.495 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:46:30.495 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:46:30.495 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:46:30.495 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:46:30.496 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:46:30.496 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:46:30.497 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:46:30.498 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:46:30.498 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:46:30.568 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 34 2024-04-06T09:46:30.569 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 34 2024-04-06T09:46:30.569 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:46:30.569 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=137, tag_tid=24, entry_tid=1], [object_number=136, tag_tid=24, entry_tid=0], [object_number=139, tag_tid=23, entry_tid=5615], [object_number=138, tag_tid=23, entry_tid=5614]]], state=connected] 2024-04-06T09:46:30.569 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=136, tag_tid=23, entry_tid=3776], [object_number=139, tag_tid=23, entry_tid=3775], [object_number=138, tag_tid=23, entry_tid=3774], [object_number=137, tag_tid=23, entry_tid=3773]]], state=connected] 2024-04-06T09:46:30.572 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:46:30.574 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=137, tag_tid=24, entry_tid=1' 2024-04-06T09:46:30.575 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:46:30.575 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:46:30.575 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:46:30.575 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:46:30.575 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:46:30.575 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:46:30.575 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:46:30.575 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:46:30.575 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:46:30.575 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:46:30.575 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:46:30.576 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:46:30.576 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:46:30.576 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:46:30.576 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:46:30.623 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 34 2024-04-06T09:46:30.623 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 34 2024-04-06T09:46:30.623 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:46:30.623 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=137, tag_tid=24, entry_tid=1], [object_number=136, tag_tid=24, entry_tid=0], [object_number=139, tag_tid=23, entry_tid=5615], [object_number=138, tag_tid=23, entry_tid=5614]]], state=connected] 2024-04-06T09:46:30.623 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=136, tag_tid=23, entry_tid=3776], [object_number=139, tag_tid=23, entry_tid=3775], [object_number=138, tag_tid=23, entry_tid=3774], [object_number=137, tag_tid=23, entry_tid=3773]]], state=connected] 2024-04-06T09:46:30.626 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:46:30.628 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=136, tag_tid=23, entry_tid=3776' 2024-04-06T09:46:30.628 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=137, tag_tid=24, entry_tid=1' -a 'object_number=137, tag_tid=24, entry_tid=1' = 'object_number=136, tag_tid=23, entry_tid=3776' 2024-04-06T09:46:30.629 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=136, tag_tid=23, entry_tid=3776' '!=' '' 2024-04-06T09:46:30.629 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:46:30.629 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=136, tag_tid=23, entry_tid=3776' = '' 2024-04-06T09:46:30.629 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=136, tag_tid=23, entry_tid=3776' 2024-04-06T09:46:30.629 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=137, tag_tid=24, entry_tid=1' 2024-04-06T09:46:30.629 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:46:30.629 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:46:30.630 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=24 2024-04-06T09:46:30.630 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=136, tag_tid=23, entry_tid=3776' 2024-04-06T09:46:30.630 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:46:30.631 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:46:30.631 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=23 2024-04-06T09:46:30.632 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=137, tag_tid=24, entry_tid=1' 2024-04-06T09:46:30.632 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:46:30.632 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:46:30.633 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:46:30.634 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=136, tag_tid=23, entry_tid=3776' 2024-04-06T09:46:30.634 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:46:30.634 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:46:30.635 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=3776 2024-04-06T09:46:30.635 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 24 = 23 -a 1 -le 3776 2024-04-06T09:46:30.635 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:46:30.635 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:46:30.635 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:46:30.837 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:46:30.837 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:46:30.837 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:46:30.837 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:46:30.837 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:46:30.837 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:46:30.837 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:46:30.837 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:46:30.838 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:46:30.838 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:46:30.838 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:46:30.838 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:46:30.838 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:46:30.838 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:46:30.838 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:46:30.839 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:46:30.839 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:46:30.841 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:46:30.841 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:46:30.841 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:46:30.841 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:46:30.842 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:46:30.842 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:46:30.842 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:46:30.842 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:46:30.842 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:46:30.842 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:46:30.842 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:46:30.842 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:46:30.842 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:46:30.842 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:46:30.842 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:46:30.842 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:46:30.843 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:46:30.843 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:46:30.843 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:46:30.843 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:46:30.843 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:46:30.843 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:46:30.843 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:46:30.843 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:46:30.843 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:46:30.843 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:46:30.843 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:46:30.843 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:46:30.843 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:46:30.844 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:46:30.844 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:46:30.844 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:46:30.866 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:46:30.866 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:46:30.867 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:46:30.867 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:46:30.972 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-06T09:46:30.973 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:rbd mirror flush : flush rbd mirror 2024-04-06T09:46:30.973 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:admin_socket: invalid command 2024-04-06T09:46:30.976 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:46:30.976 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/test 2024-04-06T09:46:30.976 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:1 2024-04-06T09:46:30.976 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:46:30.976 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:46:30.976 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-06T09:46:30.977 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:1 2024-04-06T09:46:30.977 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:46:30.977 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:46:30.977 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:46:30.977 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=1 2024-04-06T09:46:30.977 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-06T09:46:30.977 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:46:30.977 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:46:30.977 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=1 2024-04-06T09:46:30.977 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=1 2024-04-06T09:46:30.977 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-06T09:46:30.977 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:1 2024-04-06T09:46:30.977 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:46:30.978 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:46:30.978 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-06T09:46:30.978 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:1 2024-04-06T09:46:30.978 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:46:30.978 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:46:30.978 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:46:30.978 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=1 2024-04-06T09:46:30.978 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-06T09:46:30.978 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:46:30.978 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:46:30.978 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=1 2024-04-06T09:46:30.978 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=1 2024-04-06T09:46:30.978 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-06T09:46:30.995 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T09:46:30.995 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T09:46:30.995 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T09:46:30.995 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/test 2024-04-06T09:46:31.130 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:46:31.131 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:46:31.131 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:46:31.131 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:46:31.131 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:46:31.131 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:46:31.131 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:46:31.132 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:46:31.132 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:46:31.132 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:46:31.132 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:46:31.132 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:46:31.132 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:46:31.133 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:46:31.134 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:46:31.134 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:46:31.187 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 34 2024-04-06T09:46:31.187 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 34 2024-04-06T09:46:31.187 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:46:31.187 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=137, tag_tid=24, entry_tid=1], [object_number=136, tag_tid=24, entry_tid=0], [object_number=139, tag_tid=23, entry_tid=5615], [object_number=138, tag_tid=23, entry_tid=5614]]], state=connected] 2024-04-06T09:46:31.187 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=136, tag_tid=23, entry_tid=3892], [object_number=139, tag_tid=23, entry_tid=3891], [object_number=138, tag_tid=23, entry_tid=3890], [object_number=137, tag_tid=23, entry_tid=3889]]], state=connected] 2024-04-06T09:46:31.191 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:46:31.192 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=137, tag_tid=24, entry_tid=1' 2024-04-06T09:46:31.194 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:46:31.194 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:46:31.194 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:46:31.194 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:46:31.194 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:46:31.194 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:46:31.194 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:46:31.194 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:46:31.194 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:46:31.194 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:46:31.194 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:46:31.194 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:46:31.195 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:46:31.195 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:46:31.195 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:46:31.242 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 34 2024-04-06T09:46:31.242 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 34 2024-04-06T09:46:31.242 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:46:31.242 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=137, tag_tid=24, entry_tid=1], [object_number=136, tag_tid=24, entry_tid=0], [object_number=139, tag_tid=23, entry_tid=5615], [object_number=138, tag_tid=23, entry_tid=5614]]], state=connected] 2024-04-06T09:46:31.243 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=136, tag_tid=23, entry_tid=3892], [object_number=139, tag_tid=23, entry_tid=3891], [object_number=138, tag_tid=23, entry_tid=3890], [object_number=137, tag_tid=23, entry_tid=3889]]], state=connected] 2024-04-06T09:46:31.245 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:46:31.248 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=136, tag_tid=23, entry_tid=3892' 2024-04-06T09:46:31.248 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=137, tag_tid=24, entry_tid=1' -a 'object_number=137, tag_tid=24, entry_tid=1' = 'object_number=136, tag_tid=23, entry_tid=3892' 2024-04-06T09:46:31.248 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=136, tag_tid=23, entry_tid=3892' '!=' 'object_number=136, tag_tid=23, entry_tid=3776' 2024-04-06T09:46:31.248 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:46:31.248 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=136, tag_tid=23, entry_tid=3892' = 'object_number=136, tag_tid=23, entry_tid=3776' 2024-04-06T09:46:31.248 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=136, tag_tid=23, entry_tid=3892' 2024-04-06T09:46:31.248 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=137, tag_tid=24, entry_tid=1' 2024-04-06T09:46:31.248 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:46:31.248 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:46:31.249 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=24 2024-04-06T09:46:31.250 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=136, tag_tid=23, entry_tid=3892' 2024-04-06T09:46:31.250 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:46:31.250 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:46:31.250 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=23 2024-04-06T09:46:31.251 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=137, tag_tid=24, entry_tid=1' 2024-04-06T09:46:31.251 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:46:31.251 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:46:31.252 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:46:31.253 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=136, tag_tid=23, entry_tid=3892' 2024-04-06T09:46:31.253 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:46:31.253 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:46:31.254 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=3892 2024-04-06T09:46:31.254 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 24 = 23 -a 1 -le 3892 2024-04-06T09:46:31.254 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:46:31.254 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:46:31.255 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:46:31.456 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:46:31.456 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:46:31.456 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:46:31.456 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:46:31.456 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:46:31.457 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:46:31.457 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:46:31.457 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:46:31.457 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:46:31.457 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:46:31.457 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:46:31.457 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:46:31.457 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:46:31.457 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:46:31.457 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:46:31.458 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:46:31.458 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:46:31.460 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:46:31.460 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:46:31.460 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:46:31.460 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:46:31.461 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:46:31.461 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:46:31.461 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:46:31.461 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:46:31.461 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:46:31.461 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:46:31.461 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:46:31.461 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:46:31.461 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:46:31.461 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:46:31.461 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:46:31.461 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:46:31.463 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:46:31.463 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:46:31.463 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:46:31.463 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:46:31.464 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:46:31.464 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:46:31.464 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:46:31.464 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:46:31.464 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:46:31.464 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:46:31.464 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:46:31.464 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:46:31.464 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:46:31.464 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:46:31.464 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:46:31.464 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:46:31.488 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:46:31.489 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:46:31.489 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:46:31.489 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:46:31.595 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-06T09:46:31.596 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:rbd mirror flush : flush rbd mirror 2024-04-06T09:46:31.596 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:admin_socket: invalid command 2024-04-06T09:46:31.599 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:46:31.599 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/test 2024-04-06T09:46:31.599 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:1 2024-04-06T09:46:31.599 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:46:31.599 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:46:31.599 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-06T09:46:31.599 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:1 2024-04-06T09:46:31.599 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:46:31.599 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:46:31.599 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:46:31.599 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=1 2024-04-06T09:46:31.599 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-06T09:46:31.600 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:46:31.600 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:46:31.600 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=1 2024-04-06T09:46:31.600 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=1 2024-04-06T09:46:31.600 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-06T09:46:31.600 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:1 2024-04-06T09:46:31.600 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:46:31.600 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:46:31.600 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-06T09:46:31.600 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:1 2024-04-06T09:46:31.600 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:46:31.600 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:46:31.600 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:46:31.600 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=1 2024-04-06T09:46:31.600 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-06T09:46:31.601 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:46:31.601 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:46:31.601 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=1 2024-04-06T09:46:31.601 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=1 2024-04-06T09:46:31.601 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-06T09:46:31.624 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T09:46:31.625 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T09:46:31.625 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T09:46:31.625 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/test 2024-04-06T09:46:31.749 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:46:31.750 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:46:31.750 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:46:31.751 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:46:31.751 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:46:31.751 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:46:31.751 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:46:31.751 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:46:31.751 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:46:31.751 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:46:31.751 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:46:31.751 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:46:31.752 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:46:31.753 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:46:31.754 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:46:31.754 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:46:31.808 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 34 2024-04-06T09:46:31.809 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 34 2024-04-06T09:46:31.809 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:46:31.809 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=137, tag_tid=24, entry_tid=1], [object_number=136, tag_tid=24, entry_tid=0], [object_number=139, tag_tid=23, entry_tid=5615], [object_number=138, tag_tid=23, entry_tid=5614]]], state=connected] 2024-04-06T09:46:31.809 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=137, tag_tid=23, entry_tid=3997], [object_number=136, tag_tid=23, entry_tid=3996], [object_number=139, tag_tid=23, entry_tid=3995], [object_number=138, tag_tid=23, entry_tid=3994]]], state=connected] 2024-04-06T09:46:31.812 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:46:31.813 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=137, tag_tid=24, entry_tid=1' 2024-04-06T09:46:31.815 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:46:31.815 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:46:31.815 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:46:31.815 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:46:31.815 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:46:31.815 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:46:31.815 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:46:31.815 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:46:31.816 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:46:31.816 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:46:31.816 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:46:31.816 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:46:31.816 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:46:31.816 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:46:31.817 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:46:31.865 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 34 2024-04-06T09:46:31.866 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 34 2024-04-06T09:46:31.866 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:46:31.866 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=137, tag_tid=24, entry_tid=1], [object_number=136, tag_tid=24, entry_tid=0], [object_number=139, tag_tid=23, entry_tid=5615], [object_number=138, tag_tid=23, entry_tid=5614]]], state=connected] 2024-04-06T09:46:31.866 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=137, tag_tid=23, entry_tid=3997], [object_number=136, tag_tid=23, entry_tid=3996], [object_number=139, tag_tid=23, entry_tid=3995], [object_number=138, tag_tid=23, entry_tid=3994]]], state=connected] 2024-04-06T09:46:31.869 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:46:31.871 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=137, tag_tid=23, entry_tid=3997' 2024-04-06T09:46:31.871 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=137, tag_tid=24, entry_tid=1' -a 'object_number=137, tag_tid=24, entry_tid=1' = 'object_number=137, tag_tid=23, entry_tid=3997' 2024-04-06T09:46:31.871 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=137, tag_tid=23, entry_tid=3997' '!=' 'object_number=136, tag_tid=23, entry_tid=3892' 2024-04-06T09:46:31.872 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:46:31.872 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=137, tag_tid=23, entry_tid=3997' = 'object_number=136, tag_tid=23, entry_tid=3892' 2024-04-06T09:46:31.872 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=137, tag_tid=23, entry_tid=3997' 2024-04-06T09:46:31.872 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=137, tag_tid=24, entry_tid=1' 2024-04-06T09:46:31.873 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:46:31.873 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:46:31.873 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=24 2024-04-06T09:46:31.874 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=137, tag_tid=23, entry_tid=3997' 2024-04-06T09:46:31.874 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:46:31.874 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:46:31.875 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=23 2024-04-06T09:46:31.875 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=137, tag_tid=24, entry_tid=1' 2024-04-06T09:46:31.875 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:46:31.875 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:46:31.876 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:46:31.877 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=137, tag_tid=23, entry_tid=3997' 2024-04-06T09:46:31.877 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:46:31.877 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:46:31.878 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=3997 2024-04-06T09:46:31.878 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 24 = 23 -a 1 -le 3997 2024-04-06T09:46:31.879 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:46:31.879 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:46:31.879 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:46:32.080 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:46:32.080 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:46:32.080 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:46:32.081 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:46:32.081 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:46:32.081 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:46:32.081 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:46:32.081 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:46:32.081 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:46:32.081 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:46:32.081 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:46:32.081 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:46:32.081 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:46:32.081 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:46:32.081 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:46:32.082 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:46:32.082 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:46:32.084 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:46:32.084 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:46:32.084 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:46:32.084 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:46:32.084 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:46:32.084 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:46:32.084 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:46:32.085 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:46:32.085 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:46:32.085 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:46:32.085 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:46:32.085 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:46:32.085 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:46:32.085 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:46:32.085 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:46:32.085 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:46:32.086 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:46:32.086 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:46:32.086 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:46:32.087 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:46:32.087 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:46:32.087 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:46:32.087 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:46:32.087 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:46:32.087 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:46:32.087 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:46:32.087 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:46:32.087 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:46:32.087 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:46:32.087 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:46:32.087 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:46:32.087 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:46:32.111 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:46:32.111 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:46:32.111 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:46:32.111 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:46:32.219 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-06T09:46:32.219 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:rbd mirror flush : flush rbd mirror 2024-04-06T09:46:32.219 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:admin_socket: invalid command 2024-04-06T09:46:32.221 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:46:32.221 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/test 2024-04-06T09:46:32.221 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:1 2024-04-06T09:46:32.221 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:46:32.221 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:46:32.221 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-06T09:46:32.222 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:1 2024-04-06T09:46:32.222 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:46:32.222 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:46:32.222 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:46:32.222 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=1 2024-04-06T09:46:32.222 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-06T09:46:32.222 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:46:32.222 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:46:32.222 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=1 2024-04-06T09:46:32.222 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=1 2024-04-06T09:46:32.223 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-06T09:46:32.224 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:1 2024-04-06T09:46:32.224 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:46:32.224 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:46:32.224 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-06T09:46:32.224 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:1 2024-04-06T09:46:32.224 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:46:32.224 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:46:32.224 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:46:32.224 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=1 2024-04-06T09:46:32.224 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-06T09:46:32.224 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:46:32.224 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:46:32.224 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=1 2024-04-06T09:46:32.224 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=1 2024-04-06T09:46:32.224 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-06T09:46:32.248 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T09:46:32.248 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T09:46:32.248 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T09:46:32.248 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/test 2024-04-06T09:46:32.379 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:46:32.380 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:46:32.380 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:46:32.381 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:46:32.381 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:46:32.381 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:46:32.381 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:46:32.381 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:46:32.381 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:46:32.381 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:46:32.381 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:46:32.382 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:46:32.382 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:46:32.383 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:46:32.384 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:46:32.384 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:46:32.430 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 34 2024-04-06T09:46:32.430 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 34 2024-04-06T09:46:32.430 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:46:32.430 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=137, tag_tid=24, entry_tid=1], [object_number=136, tag_tid=24, entry_tid=0], [object_number=139, tag_tid=23, entry_tid=5615], [object_number=138, tag_tid=23, entry_tid=5614]]], state=connected] 2024-04-06T09:46:32.431 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=137, tag_tid=23, entry_tid=4101], [object_number=136, tag_tid=23, entry_tid=4100], [object_number=139, tag_tid=23, entry_tid=4099], [object_number=138, tag_tid=23, entry_tid=4098]]], state=connected] 2024-04-06T09:46:32.434 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:46:32.435 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=137, tag_tid=24, entry_tid=1' 2024-04-06T09:46:32.436 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:46:32.436 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:46:32.436 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:46:32.436 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:46:32.436 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:46:32.437 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:46:32.437 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:46:32.437 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:46:32.437 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:46:32.437 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:46:32.437 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:46:32.437 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:46:32.438 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:46:32.438 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:46:32.438 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:46:32.486 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 34 2024-04-06T09:46:32.486 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 34 2024-04-06T09:46:32.486 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:46:32.486 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=137, tag_tid=24, entry_tid=1], [object_number=136, tag_tid=24, entry_tid=0], [object_number=139, tag_tid=23, entry_tid=5615], [object_number=138, tag_tid=23, entry_tid=5614]]], state=connected] 2024-04-06T09:46:32.486 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=137, tag_tid=23, entry_tid=4101], [object_number=136, tag_tid=23, entry_tid=4100], [object_number=139, tag_tid=23, entry_tid=4099], [object_number=138, tag_tid=23, entry_tid=4098]]], state=connected] 2024-04-06T09:46:32.490 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:46:32.492 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=137, tag_tid=23, entry_tid=4101' 2024-04-06T09:46:32.492 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=137, tag_tid=24, entry_tid=1' -a 'object_number=137, tag_tid=24, entry_tid=1' = 'object_number=137, tag_tid=23, entry_tid=4101' 2024-04-06T09:46:32.492 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=137, tag_tid=23, entry_tid=4101' '!=' 'object_number=137, tag_tid=23, entry_tid=3997' 2024-04-06T09:46:32.492 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:46:32.492 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=137, tag_tid=23, entry_tid=4101' = 'object_number=137, tag_tid=23, entry_tid=3997' 2024-04-06T09:46:32.492 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=137, tag_tid=23, entry_tid=4101' 2024-04-06T09:46:32.492 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=137, tag_tid=24, entry_tid=1' 2024-04-06T09:46:32.492 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:46:32.492 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:46:32.493 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=24 2024-04-06T09:46:32.493 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=137, tag_tid=23, entry_tid=4101' 2024-04-06T09:46:32.494 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:46:32.494 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:46:32.495 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=23 2024-04-06T09:46:32.495 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=137, tag_tid=24, entry_tid=1' 2024-04-06T09:46:32.495 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:46:32.496 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:46:32.496 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:46:32.497 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=137, tag_tid=23, entry_tid=4101' 2024-04-06T09:46:32.497 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:46:32.497 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:46:32.498 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=4101 2024-04-06T09:46:32.499 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 24 = 23 -a 1 -le 4101 2024-04-06T09:46:32.499 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:46:32.499 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:46:32.499 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:46:32.700 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:46:32.700 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:46:32.701 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:46:32.701 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:46:32.701 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:46:32.701 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:46:32.701 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:46:32.701 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:46:32.701 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:46:32.701 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:46:32.701 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:46:32.701 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:46:32.701 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:46:32.701 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:46:32.701 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:46:32.702 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:46:32.702 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:46:32.704 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:46:32.704 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:46:32.704 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:46:32.704 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:46:32.704 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:46:32.704 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:46:32.704 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:46:32.704 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:46:32.704 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:46:32.705 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:46:32.705 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:46:32.705 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:46:32.705 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:46:32.705 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:46:32.705 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:46:32.705 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:46:32.706 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:46:32.706 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:46:32.706 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:46:32.706 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:46:32.706 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:46:32.707 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:46:32.707 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:46:32.707 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:46:32.707 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:46:32.707 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:46:32.707 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:46:32.707 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:46:32.707 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:46:32.707 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:46:32.707 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:46:32.707 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:46:32.731 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:46:32.732 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:46:32.732 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:46:32.732 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:46:32.837 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-06T09:46:32.837 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:rbd mirror flush : flush rbd mirror 2024-04-06T09:46:32.837 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:admin_socket: invalid command 2024-04-06T09:46:32.840 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:46:32.840 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/test 2024-04-06T09:46:32.840 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:1 2024-04-06T09:46:32.840 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:46:32.840 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:46:32.840 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-06T09:46:32.840 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:1 2024-04-06T09:46:32.840 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:46:32.840 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:46:32.840 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:46:32.840 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=1 2024-04-06T09:46:32.840 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-06T09:46:32.840 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:46:32.841 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:46:32.841 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=1 2024-04-06T09:46:32.841 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=1 2024-04-06T09:46:32.842 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-06T09:46:32.842 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:1 2024-04-06T09:46:32.842 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:46:32.842 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:46:32.842 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-06T09:46:32.843 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:1 2024-04-06T09:46:32.843 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:46:32.843 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:46:32.843 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:46:32.843 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=1 2024-04-06T09:46:32.843 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-06T09:46:32.843 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:46:32.843 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:46:32.843 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=1 2024-04-06T09:46:32.843 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=1 2024-04-06T09:46:32.843 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-06T09:46:32.867 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T09:46:32.867 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T09:46:32.868 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T09:46:32.868 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/test 2024-04-06T09:46:33.000 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:46:33.001 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:46:33.001 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:46:33.001 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:46:33.001 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:46:33.002 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:46:33.002 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:46:33.002 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:46:33.002 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:46:33.002 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:46:33.002 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:46:33.003 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:46:33.003 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:46:33.005 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:46:33.006 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:46:33.006 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:46:33.062 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 34 2024-04-06T09:46:33.062 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 34 2024-04-06T09:46:33.062 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:46:33.063 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=137, tag_tid=24, entry_tid=1], [object_number=136, tag_tid=24, entry_tid=0], [object_number=139, tag_tid=23, entry_tid=5615], [object_number=138, tag_tid=23, entry_tid=5614]]], state=connected] 2024-04-06T09:46:33.063 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=138, tag_tid=23, entry_tid=4206], [object_number=137, tag_tid=23, entry_tid=4205], [object_number=136, tag_tid=23, entry_tid=4204], [object_number=139, tag_tid=23, entry_tid=4203]]], state=connected] 2024-04-06T09:46:33.066 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:46:33.067 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=137, tag_tid=24, entry_tid=1' 2024-04-06T09:46:33.068 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:46:33.069 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:46:33.069 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:46:33.069 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:46:33.069 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:46:33.069 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:46:33.069 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:46:33.069 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:46:33.069 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:46:33.069 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:46:33.069 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:46:33.070 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:46:33.070 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:46:33.070 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:46:33.070 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:46:33.118 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 34 2024-04-06T09:46:33.118 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 34 2024-04-06T09:46:33.118 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:46:33.118 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=137, tag_tid=24, entry_tid=1], [object_number=136, tag_tid=24, entry_tid=0], [object_number=139, tag_tid=23, entry_tid=5615], [object_number=138, tag_tid=23, entry_tid=5614]]], state=connected] 2024-04-06T09:46:33.118 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=138, tag_tid=23, entry_tid=4206], [object_number=137, tag_tid=23, entry_tid=4205], [object_number=136, tag_tid=23, entry_tid=4204], [object_number=139, tag_tid=23, entry_tid=4203]]], state=connected] 2024-04-06T09:46:33.122 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:46:33.124 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=138, tag_tid=23, entry_tid=4206' 2024-04-06T09:46:33.124 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=137, tag_tid=24, entry_tid=1' -a 'object_number=137, tag_tid=24, entry_tid=1' = 'object_number=138, tag_tid=23, entry_tid=4206' 2024-04-06T09:46:33.124 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=138, tag_tid=23, entry_tid=4206' '!=' 'object_number=137, tag_tid=23, entry_tid=4101' 2024-04-06T09:46:33.124 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:46:33.124 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=138, tag_tid=23, entry_tid=4206' = 'object_number=137, tag_tid=23, entry_tid=4101' 2024-04-06T09:46:33.125 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=138, tag_tid=23, entry_tid=4206' 2024-04-06T09:46:33.125 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=137, tag_tid=24, entry_tid=1' 2024-04-06T09:46:33.125 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:46:33.125 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:46:33.125 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=24 2024-04-06T09:46:33.126 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=138, tag_tid=23, entry_tid=4206' 2024-04-06T09:46:33.126 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:46:33.126 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:46:33.127 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=23 2024-04-06T09:46:33.128 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=137, tag_tid=24, entry_tid=1' 2024-04-06T09:46:33.128 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:46:33.128 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:46:33.129 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:46:33.129 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=138, tag_tid=23, entry_tid=4206' 2024-04-06T09:46:33.129 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:46:33.130 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:46:33.131 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=4206 2024-04-06T09:46:33.131 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 24 = 23 -a 1 -le 4206 2024-04-06T09:46:33.131 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:46:33.131 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:46:33.131 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:46:33.333 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:46:33.333 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:46:33.333 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:46:33.333 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:46:33.333 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:46:33.333 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:46:33.333 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:46:33.333 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:46:33.333 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:46:33.333 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:46:33.333 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:46:33.333 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:46:33.333 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:46:33.334 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:46:33.334 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:46:33.334 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:46:33.334 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:46:33.337 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:46:33.337 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:46:33.337 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:46:33.337 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:46:33.337 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:46:33.338 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:46:33.338 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:46:33.338 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:46:33.338 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:46:33.338 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:46:33.338 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:46:33.338 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:46:33.338 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:46:33.338 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:46:33.338 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:46:33.338 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:46:33.338 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:46:33.338 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:46:33.338 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:46:33.339 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:46:33.339 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:46:33.339 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:46:33.339 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:46:33.339 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:46:33.339 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:46:33.339 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:46:33.339 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:46:33.339 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:46:33.339 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:46:33.339 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:46:33.339 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:46:33.339 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:46:33.362 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:46:33.362 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:46:33.363 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:46:33.363 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:46:33.464 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-06T09:46:33.464 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:rbd mirror flush : flush rbd mirror 2024-04-06T09:46:33.464 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:admin_socket: invalid command 2024-04-06T09:46:33.467 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:46:33.467 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/test 2024-04-06T09:46:33.467 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:1 2024-04-06T09:46:33.467 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:46:33.467 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:46:33.467 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-06T09:46:33.467 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:1 2024-04-06T09:46:33.468 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:46:33.468 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:46:33.468 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:46:33.468 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=1 2024-04-06T09:46:33.468 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-06T09:46:33.468 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:46:33.468 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:46:33.468 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=1 2024-04-06T09:46:33.468 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=1 2024-04-06T09:46:33.468 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-06T09:46:33.468 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:1 2024-04-06T09:46:33.468 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:46:33.468 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:46:33.468 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-06T09:46:33.469 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:1 2024-04-06T09:46:33.469 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:46:33.469 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:46:33.469 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:46:33.469 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=1 2024-04-06T09:46:33.469 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-06T09:46:33.469 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:46:33.469 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:46:33.469 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=1 2024-04-06T09:46:33.469 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=1 2024-04-06T09:46:33.470 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-06T09:46:33.492 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T09:46:33.492 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T09:46:33.492 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T09:46:33.492 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/test 2024-04-06T09:46:33.622 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:46:33.624 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:46:33.624 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:46:33.624 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:46:33.624 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:46:33.624 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:46:33.624 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:46:33.624 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:46:33.624 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:46:33.624 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:46:33.625 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:46:33.625 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:46:33.625 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:46:33.626 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:46:33.627 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:46:33.627 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:46:33.680 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 34 2024-04-06T09:46:33.680 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 34 2024-04-06T09:46:33.680 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:46:33.680 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=137, tag_tid=24, entry_tid=1], [object_number=136, tag_tid=24, entry_tid=0], [object_number=139, tag_tid=23, entry_tid=5615], [object_number=138, tag_tid=23, entry_tid=5614]]], state=connected] 2024-04-06T09:46:33.680 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=137, tag_tid=23, entry_tid=4313], [object_number=136, tag_tid=23, entry_tid=4312], [object_number=139, tag_tid=23, entry_tid=4311], [object_number=138, tag_tid=23, entry_tid=4310]]], state=connected] 2024-04-06T09:46:33.684 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:46:33.685 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=137, tag_tid=24, entry_tid=1' 2024-04-06T09:46:33.687 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:46:33.687 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:46:33.687 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:46:33.687 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:46:33.687 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:46:33.687 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:46:33.687 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:46:33.687 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:46:33.687 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:46:33.687 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:46:33.687 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:46:33.687 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:46:33.688 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:46:33.688 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:46:33.688 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:46:33.736 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 34 2024-04-06T09:46:33.736 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 34 2024-04-06T09:46:33.736 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:46:33.736 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=137, tag_tid=24, entry_tid=1], [object_number=136, tag_tid=24, entry_tid=0], [object_number=139, tag_tid=23, entry_tid=5615], [object_number=138, tag_tid=23, entry_tid=5614]]], state=connected] 2024-04-06T09:46:33.736 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=137, tag_tid=23, entry_tid=4313], [object_number=136, tag_tid=23, entry_tid=4312], [object_number=139, tag_tid=23, entry_tid=4311], [object_number=138, tag_tid=23, entry_tid=4310]]], state=connected] 2024-04-06T09:46:33.739 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:46:33.741 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=137, tag_tid=23, entry_tid=4313' 2024-04-06T09:46:33.742 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=137, tag_tid=24, entry_tid=1' -a 'object_number=137, tag_tid=24, entry_tid=1' = 'object_number=137, tag_tid=23, entry_tid=4313' 2024-04-06T09:46:33.742 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=137, tag_tid=23, entry_tid=4313' '!=' 'object_number=138, tag_tid=23, entry_tid=4206' 2024-04-06T09:46:33.742 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:46:33.742 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=137, tag_tid=23, entry_tid=4313' = 'object_number=138, tag_tid=23, entry_tid=4206' 2024-04-06T09:46:33.742 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=137, tag_tid=23, entry_tid=4313' 2024-04-06T09:46:33.742 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=137, tag_tid=24, entry_tid=1' 2024-04-06T09:46:33.742 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:46:33.742 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:46:33.743 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=24 2024-04-06T09:46:33.743 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=137, tag_tid=23, entry_tid=4313' 2024-04-06T09:46:33.743 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:46:33.743 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:46:33.744 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=23 2024-04-06T09:46:33.745 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=137, tag_tid=24, entry_tid=1' 2024-04-06T09:46:33.745 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:46:33.745 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:46:33.746 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:46:33.747 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=137, tag_tid=23, entry_tid=4313' 2024-04-06T09:46:33.747 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:46:33.747 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:46:33.748 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=4313 2024-04-06T09:46:33.748 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 24 = 23 -a 1 -le 4313 2024-04-06T09:46:33.748 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:46:33.748 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:46:33.748 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:46:33.950 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:46:33.951 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:46:33.951 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:46:33.951 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:46:33.951 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:46:33.951 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:46:33.951 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:46:33.951 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:46:33.951 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:46:33.951 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:46:33.952 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:46:33.952 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:46:33.952 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:46:33.952 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:46:33.952 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:46:33.952 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:46:33.953 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:46:33.955 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:46:33.956 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:46:33.956 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:46:33.956 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:46:33.956 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:46:33.956 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:46:33.956 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:46:33.956 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:46:33.956 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:46:33.956 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:46:33.956 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:46:33.957 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:46:33.957 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:46:33.957 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:46:33.957 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:46:33.957 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:46:33.957 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:46:33.957 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:46:33.957 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:46:33.958 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:46:33.958 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:46:33.958 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:46:33.958 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:46:33.958 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:46:33.958 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:46:33.958 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:46:33.958 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:46:33.958 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:46:33.958 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:46:33.958 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:46:33.958 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:46:33.959 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:46:33.981 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:46:33.982 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:46:33.982 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:46:33.982 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:46:34.092 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-06T09:46:34.092 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:rbd mirror flush : flush rbd mirror 2024-04-06T09:46:34.092 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:admin_socket: invalid command 2024-04-06T09:46:34.095 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:46:34.095 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/test 2024-04-06T09:46:34.095 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:1 2024-04-06T09:46:34.096 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:46:34.096 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:46:34.096 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-06T09:46:34.096 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:1 2024-04-06T09:46:34.096 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:46:34.096 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:46:34.096 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:46:34.096 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=1 2024-04-06T09:46:34.096 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-06T09:46:34.096 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:46:34.096 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:46:34.097 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=1 2024-04-06T09:46:34.097 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=1 2024-04-06T09:46:34.097 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-06T09:46:34.097 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:1 2024-04-06T09:46:34.097 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:46:34.097 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:46:34.097 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-06T09:46:34.097 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:1 2024-04-06T09:46:34.097 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:46:34.097 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:46:34.097 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:46:34.097 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=1 2024-04-06T09:46:34.098 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-06T09:46:34.098 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:46:34.098 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:46:34.098 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=1 2024-04-06T09:46:34.098 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=1 2024-04-06T09:46:34.098 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-06T09:46:34.120 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T09:46:34.121 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T09:46:34.121 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T09:46:34.121 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/test 2024-04-06T09:46:34.251 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:46:34.252 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:46:34.252 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:46:34.252 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:46:34.252 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:46:34.252 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:46:34.252 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:46:34.252 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:46:34.252 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:46:34.252 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:46:34.253 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:46:34.253 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:46:34.253 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:46:34.254 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:46:34.255 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:46:34.255 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:46:34.303 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 34 2024-04-06T09:46:34.303 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 34 2024-04-06T09:46:34.303 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:46:34.303 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=137, tag_tid=24, entry_tid=1], [object_number=136, tag_tid=24, entry_tid=0], [object_number=139, tag_tid=23, entry_tid=5615], [object_number=138, tag_tid=23, entry_tid=5614]]], state=connected] 2024-04-06T09:46:34.303 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=139, tag_tid=23, entry_tid=4419], [object_number=138, tag_tid=23, entry_tid=4418], [object_number=137, tag_tid=23, entry_tid=4417], [object_number=136, tag_tid=23, entry_tid=4416]]], state=connected] 2024-04-06T09:46:34.307 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:46:34.308 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=137, tag_tid=24, entry_tid=1' 2024-04-06T09:46:34.309 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:46:34.309 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:46:34.309 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:46:34.309 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:46:34.310 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:46:34.310 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:46:34.310 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:46:34.310 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:46:34.310 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:46:34.310 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:46:34.310 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:46:34.310 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:46:34.311 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:46:34.311 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:46:34.311 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:46:34.358 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 34 2024-04-06T09:46:34.358 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 34 2024-04-06T09:46:34.358 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:46:34.358 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=137, tag_tid=24, entry_tid=1], [object_number=136, tag_tid=24, entry_tid=0], [object_number=139, tag_tid=23, entry_tid=5615], [object_number=138, tag_tid=23, entry_tid=5614]]], state=connected] 2024-04-06T09:46:34.358 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=139, tag_tid=23, entry_tid=4419], [object_number=138, tag_tid=23, entry_tid=4418], [object_number=137, tag_tid=23, entry_tid=4417], [object_number=136, tag_tid=23, entry_tid=4416]]], state=connected] 2024-04-06T09:46:34.361 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:46:34.363 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=139, tag_tid=23, entry_tid=4419' 2024-04-06T09:46:34.363 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=137, tag_tid=24, entry_tid=1' -a 'object_number=137, tag_tid=24, entry_tid=1' = 'object_number=139, tag_tid=23, entry_tid=4419' 2024-04-06T09:46:34.363 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=139, tag_tid=23, entry_tid=4419' '!=' 'object_number=137, tag_tid=23, entry_tid=4313' 2024-04-06T09:46:34.364 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:46:34.364 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=139, tag_tid=23, entry_tid=4419' = 'object_number=137, tag_tid=23, entry_tid=4313' 2024-04-06T09:46:34.364 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=139, tag_tid=23, entry_tid=4419' 2024-04-06T09:46:34.364 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=137, tag_tid=24, entry_tid=1' 2024-04-06T09:46:34.364 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:46:34.364 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:46:34.364 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=24 2024-04-06T09:46:34.365 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=139, tag_tid=23, entry_tid=4419' 2024-04-06T09:46:34.365 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:46:34.365 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:46:34.366 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=23 2024-04-06T09:46:34.367 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=137, tag_tid=24, entry_tid=1' 2024-04-06T09:46:34.367 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:46:34.367 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:46:34.368 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:46:34.368 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=139, tag_tid=23, entry_tid=4419' 2024-04-06T09:46:34.369 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:46:34.369 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:46:34.370 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=4419 2024-04-06T09:46:34.370 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 24 = 23 -a 1 -le 4419 2024-04-06T09:46:34.370 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:46:34.370 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:46:34.370 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:46:34.572 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:46:34.572 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:46:34.572 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:46:34.572 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:46:34.572 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:46:34.572 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:46:34.573 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:46:34.573 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:46:34.573 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:46:34.573 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:46:34.573 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:46:34.573 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:46:34.573 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:46:34.573 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:46:34.573 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:46:34.574 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:46:34.574 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:46:34.577 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:46:34.577 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:46:34.577 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:46:34.577 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:46:34.577 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:46:34.577 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:46:34.577 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:46:34.577 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:46:34.578 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:46:34.578 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:46:34.578 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:46:34.578 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:46:34.578 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:46:34.578 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:46:34.578 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:46:34.578 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:46:34.578 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:46:34.578 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:46:34.578 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:46:34.578 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:46:34.578 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:46:34.578 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:46:34.579 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:46:34.579 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:46:34.579 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:46:34.579 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:46:34.579 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:46:34.579 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:46:34.579 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:46:34.579 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:46:34.579 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:46:34.579 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:46:34.602 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:46:34.603 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:46:34.603 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:46:34.603 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:46:34.710 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-06T09:46:34.711 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:rbd mirror flush : flush rbd mirror 2024-04-06T09:46:34.711 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:admin_socket: invalid command 2024-04-06T09:46:34.713 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:46:34.713 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/test 2024-04-06T09:46:34.713 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:1 2024-04-06T09:46:34.713 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:46:34.713 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:46:34.713 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-06T09:46:34.713 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:1 2024-04-06T09:46:34.713 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:46:34.713 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:46:34.714 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:46:34.714 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=1 2024-04-06T09:46:34.714 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-06T09:46:34.714 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:46:34.714 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:46:34.714 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=1 2024-04-06T09:46:34.714 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=1 2024-04-06T09:46:34.716 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-06T09:46:34.716 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:1 2024-04-06T09:46:34.716 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:46:34.716 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:46:34.716 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-06T09:46:34.716 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:1 2024-04-06T09:46:34.716 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:46:34.716 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:46:34.716 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:46:34.716 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=1 2024-04-06T09:46:34.717 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-06T09:46:34.717 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:46:34.717 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:46:34.717 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=1 2024-04-06T09:46:34.717 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=1 2024-04-06T09:46:34.717 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-06T09:46:34.740 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T09:46:34.740 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T09:46:34.740 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T09:46:34.740 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/test 2024-04-06T09:46:34.865 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:46:34.867 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:46:34.867 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:46:34.867 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:46:34.867 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:46:34.867 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:46:34.867 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:46:34.867 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:46:34.867 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:46:34.868 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:46:34.868 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:46:34.868 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:46:34.868 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:46:34.869 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:46:34.869 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:46:34.869 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:46:34.923 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 34 2024-04-06T09:46:34.923 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 34 2024-04-06T09:46:34.923 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:46:34.924 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=137, tag_tid=24, entry_tid=1], [object_number=136, tag_tid=24, entry_tid=0], [object_number=139, tag_tid=23, entry_tid=5615], [object_number=138, tag_tid=23, entry_tid=5614]]], state=connected] 2024-04-06T09:46:34.924 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=136, tag_tid=23, entry_tid=4512], [object_number=139, tag_tid=23, entry_tid=4511], [object_number=138, tag_tid=23, entry_tid=4510], [object_number=137, tag_tid=23, entry_tid=4509]]], state=connected] 2024-04-06T09:46:34.927 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:46:34.928 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=137, tag_tid=24, entry_tid=1' 2024-04-06T09:46:34.929 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:46:34.930 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:46:34.930 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:46:34.930 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:46:34.930 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:46:34.930 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:46:34.930 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:46:34.930 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:46:34.930 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:46:34.930 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:46:34.930 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:46:34.930 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:46:34.931 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:46:34.931 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:46:34.931 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:46:34.979 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 34 2024-04-06T09:46:34.979 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 34 2024-04-06T09:46:34.979 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:46:34.979 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=137, tag_tid=24, entry_tid=1], [object_number=136, tag_tid=24, entry_tid=0], [object_number=139, tag_tid=23, entry_tid=5615], [object_number=138, tag_tid=23, entry_tid=5614]]], state=connected] 2024-04-06T09:46:34.979 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=136, tag_tid=23, entry_tid=4512], [object_number=139, tag_tid=23, entry_tid=4511], [object_number=138, tag_tid=23, entry_tid=4510], [object_number=137, tag_tid=23, entry_tid=4509]]], state=connected] 2024-04-06T09:46:34.982 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:46:34.984 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=136, tag_tid=23, entry_tid=4512' 2024-04-06T09:46:34.985 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=137, tag_tid=24, entry_tid=1' -a 'object_number=137, tag_tid=24, entry_tid=1' = 'object_number=136, tag_tid=23, entry_tid=4512' 2024-04-06T09:46:34.985 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=136, tag_tid=23, entry_tid=4512' '!=' 'object_number=139, tag_tid=23, entry_tid=4419' 2024-04-06T09:46:34.985 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:46:34.985 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=136, tag_tid=23, entry_tid=4512' = 'object_number=139, tag_tid=23, entry_tid=4419' 2024-04-06T09:46:34.985 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=136, tag_tid=23, entry_tid=4512' 2024-04-06T09:46:34.985 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=137, tag_tid=24, entry_tid=1' 2024-04-06T09:46:34.985 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:46:34.985 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:46:34.986 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=24 2024-04-06T09:46:34.986 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=136, tag_tid=23, entry_tid=4512' 2024-04-06T09:46:34.987 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:46:34.987 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:46:34.987 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=23 2024-04-06T09:46:34.988 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=137, tag_tid=24, entry_tid=1' 2024-04-06T09:46:34.988 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:46:34.988 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:46:34.989 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:46:34.990 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=136, tag_tid=23, entry_tid=4512' 2024-04-06T09:46:34.990 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:46:34.990 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:46:34.991 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=4512 2024-04-06T09:46:34.991 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 24 = 23 -a 1 -le 4512 2024-04-06T09:46:34.991 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:46:34.991 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:46:34.992 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:46:35.193 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:46:35.194 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:46:35.194 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:46:35.194 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:46:35.194 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:46:35.194 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:46:35.194 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:46:35.194 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:46:35.195 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:46:35.195 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:46:35.195 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:46:35.195 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:46:35.195 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:46:35.195 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:46:35.195 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:46:35.195 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:46:35.196 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:46:35.199 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:46:35.200 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:46:35.200 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:46:35.200 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:46:35.200 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:46:35.200 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:46:35.200 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:46:35.200 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:46:35.200 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:46:35.200 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:46:35.200 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:46:35.200 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:46:35.200 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:46:35.200 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:46:35.201 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:46:35.201 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:46:35.201 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:46:35.201 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:46:35.201 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:46:35.201 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:46:35.201 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:46:35.201 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:46:35.201 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:46:35.201 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:46:35.201 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:46:35.201 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:46:35.201 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:46:35.201 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:46:35.201 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:46:35.201 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:46:35.202 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:46:35.202 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:46:35.225 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:46:35.226 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:46:35.226 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:46:35.226 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:46:35.332 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-06T09:46:35.332 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:rbd mirror flush : flush rbd mirror 2024-04-06T09:46:35.332 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:admin_socket: invalid command 2024-04-06T09:46:35.335 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:46:35.335 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/test 2024-04-06T09:46:35.335 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:1 2024-04-06T09:46:35.335 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:46:35.336 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:46:35.336 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-06T09:46:35.336 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:1 2024-04-06T09:46:35.336 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:46:35.336 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:46:35.336 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:46:35.336 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=1 2024-04-06T09:46:35.336 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-06T09:46:35.336 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:46:35.336 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:46:35.336 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=1 2024-04-06T09:46:35.336 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=1 2024-04-06T09:46:35.336 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-06T09:46:35.336 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:1 2024-04-06T09:46:35.337 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:46:35.337 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:46:35.337 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-06T09:46:35.337 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:1 2024-04-06T09:46:35.337 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:46:35.337 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:46:35.337 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:46:35.337 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=1 2024-04-06T09:46:35.337 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-06T09:46:35.337 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:46:35.337 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:46:35.337 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=1 2024-04-06T09:46:35.337 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=1 2024-04-06T09:46:35.338 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-06T09:46:35.360 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T09:46:35.361 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T09:46:35.361 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T09:46:35.361 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/test 2024-04-06T09:46:35.498 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:46:35.500 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:46:35.500 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:46:35.500 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:46:35.500 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:46:35.500 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:46:35.500 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:46:35.500 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:46:35.500 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:46:35.500 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:46:35.501 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:46:35.501 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:46:35.501 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:46:35.503 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:46:35.504 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:46:35.504 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:46:35.550 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 34 2024-04-06T09:46:35.551 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 34 2024-04-06T09:46:35.551 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:46:35.551 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=137, tag_tid=24, entry_tid=1], [object_number=136, tag_tid=24, entry_tid=0], [object_number=139, tag_tid=23, entry_tid=5615], [object_number=138, tag_tid=23, entry_tid=5614]]], state=connected] 2024-04-06T09:46:35.551 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=138, tag_tid=23, entry_tid=4618], [object_number=137, tag_tid=23, entry_tid=4617], [object_number=136, tag_tid=23, entry_tid=4616], [object_number=139, tag_tid=23, entry_tid=4615]]], state=connected] 2024-04-06T09:46:35.557 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:46:35.559 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=137, tag_tid=24, entry_tid=1' 2024-04-06T09:46:35.560 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:46:35.560 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:46:35.560 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:46:35.560 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:46:35.560 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:46:35.560 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:46:35.560 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:46:35.560 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:46:35.560 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:46:35.560 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:46:35.560 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:46:35.560 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:46:35.561 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:46:35.561 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:46:35.561 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:46:35.607 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 34 2024-04-06T09:46:35.608 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 34 2024-04-06T09:46:35.608 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:46:35.608 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=137, tag_tid=24, entry_tid=1], [object_number=136, tag_tid=24, entry_tid=0], [object_number=139, tag_tid=23, entry_tid=5615], [object_number=138, tag_tid=23, entry_tid=5614]]], state=connected] 2024-04-06T09:46:35.608 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=138, tag_tid=23, entry_tid=4618], [object_number=137, tag_tid=23, entry_tid=4617], [object_number=136, tag_tid=23, entry_tid=4616], [object_number=139, tag_tid=23, entry_tid=4615]]], state=connected] 2024-04-06T09:46:35.653 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:46:35.655 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=138, tag_tid=23, entry_tid=4618' 2024-04-06T09:46:35.655 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=137, tag_tid=24, entry_tid=1' -a 'object_number=137, tag_tid=24, entry_tid=1' = 'object_number=138, tag_tid=23, entry_tid=4618' 2024-04-06T09:46:35.655 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=138, tag_tid=23, entry_tid=4618' '!=' 'object_number=136, tag_tid=23, entry_tid=4512' 2024-04-06T09:46:35.656 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:46:35.656 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=138, tag_tid=23, entry_tid=4618' = 'object_number=136, tag_tid=23, entry_tid=4512' 2024-04-06T09:46:35.656 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=138, tag_tid=23, entry_tid=4618' 2024-04-06T09:46:35.656 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=137, tag_tid=24, entry_tid=1' 2024-04-06T09:46:35.656 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:46:35.657 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:46:35.659 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=24 2024-04-06T09:46:35.660 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=138, tag_tid=23, entry_tid=4618' 2024-04-06T09:46:35.660 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:46:35.660 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:46:35.661 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=23 2024-04-06T09:46:35.662 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=137, tag_tid=24, entry_tid=1' 2024-04-06T09:46:35.662 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:46:35.662 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:46:35.664 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:46:35.664 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=138, tag_tid=23, entry_tid=4618' 2024-04-06T09:46:35.664 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:46:35.664 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:46:35.666 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=4618 2024-04-06T09:46:35.666 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 24 = 23 -a 1 -le 4618 2024-04-06T09:46:35.666 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:46:35.666 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:46:35.666 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:46:35.868 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:46:35.868 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:46:35.868 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:46:35.868 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:46:35.868 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:46:35.868 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:46:35.868 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:46:35.868 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:46:35.868 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:46:35.868 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:46:35.868 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:46:35.869 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:46:35.869 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:46:35.869 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:46:35.869 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:46:35.869 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:46:35.869 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:46:35.872 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:46:35.873 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:46:35.873 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:46:35.873 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:46:35.873 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:46:35.873 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:46:35.873 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:46:35.873 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:46:35.873 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:46:35.873 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:46:35.873 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:46:35.873 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:46:35.873 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:46:35.873 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:46:35.873 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:46:35.873 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:46:35.873 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:46:35.874 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:46:35.874 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:46:35.874 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:46:35.874 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:46:35.874 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:46:35.874 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:46:35.874 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:46:35.874 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:46:35.874 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:46:35.874 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:46:35.874 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:46:35.874 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:46:35.874 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:46:35.875 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:46:35.875 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:46:35.900 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:46:35.901 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:46:35.901 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:46:35.901 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:46:35.991 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-04-06T09:46:35.987+0000 7fd1b6210640 -1 reset not still connected to 0x55ef99b88d00 2024-04-06T09:46:35.991 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-04-06T09:46:35.987+0000 7fd1b6210640 -1 reset not still connected to 0x55ef9ac60270 2024-04-06T09:46:36.007 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-06T09:46:36.007 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:rbd mirror flush : flush rbd mirror 2024-04-06T09:46:36.007 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:admin_socket: invalid command 2024-04-06T09:46:36.009 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:46:36.009 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/test 2024-04-06T09:46:36.009 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:1 2024-04-06T09:46:36.009 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:46:36.009 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:46:36.009 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-06T09:46:36.009 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:1 2024-04-06T09:46:36.009 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:46:36.009 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:46:36.010 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:46:36.010 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=1 2024-04-06T09:46:36.010 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-06T09:46:36.010 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:46:36.010 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:46:36.010 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=1 2024-04-06T09:46:36.010 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=1 2024-04-06T09:46:36.012 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-06T09:46:36.012 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:1 2024-04-06T09:46:36.012 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:46:36.012 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:46:36.012 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-06T09:46:36.012 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:1 2024-04-06T09:46:36.012 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:46:36.012 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:46:36.012 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:46:36.012 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=1 2024-04-06T09:46:36.012 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-06T09:46:36.012 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:46:36.013 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:46:36.013 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=1 2024-04-06T09:46:36.013 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=1 2024-04-06T09:46:36.013 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-06T09:46:36.029 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T09:46:36.029 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T09:46:36.030 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T09:46:36.030 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/test 2024-04-06T09:46:36.162 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:46:36.163 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:46:36.163 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:46:36.163 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:46:36.163 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:46:36.163 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:46:36.163 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:46:36.163 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:46:36.164 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:46:36.164 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:46:36.164 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:46:36.164 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:46:36.164 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:46:36.167 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:46:36.167 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:46:36.167 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:46:36.224 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 34 2024-04-06T09:46:36.224 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 34 2024-04-06T09:46:36.224 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:46:36.224 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=137, tag_tid=24, entry_tid=1], [object_number=136, tag_tid=24, entry_tid=0], [object_number=139, tag_tid=23, entry_tid=5615], [object_number=138, tag_tid=23, entry_tid=5614]]], state=connected] 2024-04-06T09:46:36.224 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=136, tag_tid=23, entry_tid=4736], [object_number=139, tag_tid=23, entry_tid=4735], [object_number=138, tag_tid=23, entry_tid=4734], [object_number=137, tag_tid=23, entry_tid=4733]]], state=connected] 2024-04-06T09:46:36.227 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:46:36.230 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=137, tag_tid=24, entry_tid=1' 2024-04-06T09:46:36.230 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:46:36.230 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:46:36.230 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:46:36.230 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:46:36.230 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:46:36.230 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:46:36.230 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:46:36.230 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:46:36.230 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:46:36.231 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:46:36.231 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:46:36.231 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:46:36.231 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:46:36.231 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:46:36.232 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:46:36.278 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 34 2024-04-06T09:46:36.279 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 34 2024-04-06T09:46:36.279 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:46:36.279 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=137, tag_tid=24, entry_tid=1], [object_number=136, tag_tid=24, entry_tid=0], [object_number=139, tag_tid=23, entry_tid=5615], [object_number=138, tag_tid=23, entry_tid=5614]]], state=connected] 2024-04-06T09:46:36.279 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=136, tag_tid=23, entry_tid=4736], [object_number=139, tag_tid=23, entry_tid=4735], [object_number=138, tag_tid=23, entry_tid=4734], [object_number=137, tag_tid=23, entry_tid=4733]]], state=connected] 2024-04-06T09:46:36.282 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:46:36.284 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=136, tag_tid=23, entry_tid=4736' 2024-04-06T09:46:36.284 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=137, tag_tid=24, entry_tid=1' -a 'object_number=137, tag_tid=24, entry_tid=1' = 'object_number=136, tag_tid=23, entry_tid=4736' 2024-04-06T09:46:36.285 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=136, tag_tid=23, entry_tid=4736' '!=' 'object_number=138, tag_tid=23, entry_tid=4618' 2024-04-06T09:46:36.285 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:46:36.285 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=136, tag_tid=23, entry_tid=4736' = 'object_number=138, tag_tid=23, entry_tid=4618' 2024-04-06T09:46:36.285 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=136, tag_tid=23, entry_tid=4736' 2024-04-06T09:46:36.285 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=137, tag_tid=24, entry_tid=1' 2024-04-06T09:46:36.285 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:46:36.285 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:46:36.285 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=24 2024-04-06T09:46:36.286 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=136, tag_tid=23, entry_tid=4736' 2024-04-06T09:46:36.286 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:46:36.287 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:46:36.287 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=23 2024-04-06T09:46:36.288 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=137, tag_tid=24, entry_tid=1' 2024-04-06T09:46:36.288 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:46:36.288 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:46:36.289 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:46:36.290 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=136, tag_tid=23, entry_tid=4736' 2024-04-06T09:46:36.290 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:46:36.290 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:46:36.291 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=4736 2024-04-06T09:46:36.291 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 24 = 23 -a 1 -le 4736 2024-04-06T09:46:36.291 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:46:36.291 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:46:36.291 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:46:36.493 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:46:36.493 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:46:36.493 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:46:36.493 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:46:36.493 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:46:36.493 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:46:36.493 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:46:36.493 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:46:36.493 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:46:36.493 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:46:36.493 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:46:36.493 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:46:36.493 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:46:36.494 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:46:36.494 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:46:36.494 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:46:36.494 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:46:36.497 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:46:36.497 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:46:36.497 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:46:36.497 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:46:36.497 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:46:36.497 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:46:36.497 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:46:36.497 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:46:36.497 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:46:36.497 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:46:36.498 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:46:36.498 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:46:36.498 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:46:36.498 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:46:36.498 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:46:36.498 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:46:36.499 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:46:36.499 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:46:36.499 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:46:36.499 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:46:36.499 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:46:36.499 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:46:36.499 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:46:36.500 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:46:36.500 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:46:36.500 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:46:36.500 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:46:36.500 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:46:36.500 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:46:36.500 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:46:36.500 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:46:36.500 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:46:36.526 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:46:36.526 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:46:36.526 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:46:36.526 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:46:36.629 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-06T09:46:36.629 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:rbd mirror flush : flush rbd mirror 2024-04-06T09:46:36.629 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:admin_socket: invalid command 2024-04-06T09:46:36.631 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:46:36.632 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/test 2024-04-06T09:46:36.632 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:1 2024-04-06T09:46:36.632 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:46:36.632 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:46:36.632 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-06T09:46:36.632 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:1 2024-04-06T09:46:36.632 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:46:36.632 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:46:36.632 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:46:36.632 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=1 2024-04-06T09:46:36.632 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-06T09:46:36.632 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:46:36.632 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:46:36.632 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=1 2024-04-06T09:46:36.632 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=1 2024-04-06T09:46:36.634 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-06T09:46:36.634 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:1 2024-04-06T09:46:36.634 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:46:36.634 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:46:36.634 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-06T09:46:36.634 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:1 2024-04-06T09:46:36.634 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:46:36.634 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:46:36.634 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:46:36.634 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=1 2024-04-06T09:46:36.634 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-06T09:46:36.634 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:46:36.635 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:46:36.635 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=1 2024-04-06T09:46:36.635 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=1 2024-04-06T09:46:36.635 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-06T09:46:36.657 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T09:46:36.657 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T09:46:36.657 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T09:46:36.657 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/test 2024-04-06T09:46:36.793 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:46:36.794 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:46:36.794 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:46:36.794 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:46:36.794 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:46:36.795 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:46:36.795 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:46:36.795 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:46:36.795 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:46:36.795 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:46:36.795 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:46:36.795 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:46:36.796 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:46:36.797 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:46:36.798 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:46:36.798 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:46:36.855 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 34 2024-04-06T09:46:36.855 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 34 2024-04-06T09:46:36.855 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:46:36.855 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=137, tag_tid=24, entry_tid=1], [object_number=136, tag_tid=24, entry_tid=0], [object_number=139, tag_tid=23, entry_tid=5615], [object_number=138, tag_tid=23, entry_tid=5614]]], state=connected] 2024-04-06T09:46:36.855 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=138, tag_tid=23, entry_tid=4802], [object_number=137, tag_tid=23, entry_tid=4801], [object_number=136, tag_tid=23, entry_tid=4800], [object_number=139, tag_tid=23, entry_tid=4799]]], state=connected] 2024-04-06T09:46:36.859 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:46:36.860 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=137, tag_tid=24, entry_tid=1' 2024-04-06T09:46:36.861 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:46:36.862 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:46:36.862 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:46:36.862 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:46:36.862 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:46:36.862 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:46:36.862 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:46:36.862 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:46:36.862 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:46:36.862 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:46:36.862 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:46:36.862 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:46:36.863 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:46:36.863 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:46:36.863 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:46:36.911 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 34 2024-04-06T09:46:36.912 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 34 2024-04-06T09:46:36.912 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:46:36.912 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=137, tag_tid=24, entry_tid=1], [object_number=136, tag_tid=24, entry_tid=0], [object_number=139, tag_tid=23, entry_tid=5615], [object_number=138, tag_tid=23, entry_tid=5614]]], state=connected] 2024-04-06T09:46:36.912 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=138, tag_tid=23, entry_tid=4802], [object_number=137, tag_tid=23, entry_tid=4801], [object_number=136, tag_tid=23, entry_tid=4800], [object_number=139, tag_tid=23, entry_tid=4799]]], state=connected] 2024-04-06T09:46:36.915 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:46:36.918 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=138, tag_tid=23, entry_tid=4802' 2024-04-06T09:46:36.918 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=137, tag_tid=24, entry_tid=1' -a 'object_number=137, tag_tid=24, entry_tid=1' = 'object_number=138, tag_tid=23, entry_tid=4802' 2024-04-06T09:46:36.918 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=138, tag_tid=23, entry_tid=4802' '!=' 'object_number=136, tag_tid=23, entry_tid=4736' 2024-04-06T09:46:36.918 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:46:36.918 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=138, tag_tid=23, entry_tid=4802' = 'object_number=136, tag_tid=23, entry_tid=4736' 2024-04-06T09:46:36.918 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=138, tag_tid=23, entry_tid=4802' 2024-04-06T09:46:36.918 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=137, tag_tid=24, entry_tid=1' 2024-04-06T09:46:36.918 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:46:36.919 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:46:36.920 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=24 2024-04-06T09:46:36.920 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=138, tag_tid=23, entry_tid=4802' 2024-04-06T09:46:36.920 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:46:36.920 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:46:36.921 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=23 2024-04-06T09:46:36.921 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=137, tag_tid=24, entry_tid=1' 2024-04-06T09:46:36.921 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:46:36.922 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:46:36.922 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:46:36.923 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=138, tag_tid=23, entry_tid=4802' 2024-04-06T09:46:36.923 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:46:36.923 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:46:36.924 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=4802 2024-04-06T09:46:36.924 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 24 = 23 -a 1 -le 4802 2024-04-06T09:46:36.924 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:46:36.924 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:46:36.924 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:46:37.126 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:46:37.126 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:46:37.126 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:46:37.126 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:46:37.126 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:46:37.126 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:46:37.126 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:46:37.126 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:46:37.126 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:46:37.127 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:46:37.127 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:46:37.127 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:46:37.127 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:46:37.127 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:46:37.127 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:46:37.128 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:46:37.128 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:46:37.130 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:46:37.130 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:46:37.130 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:46:37.130 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:46:37.130 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:46:37.130 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:46:37.130 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:46:37.130 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:46:37.131 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:46:37.131 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:46:37.131 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:46:37.131 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:46:37.131 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:46:37.131 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:46:37.131 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:46:37.131 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:46:37.132 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:46:37.133 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:46:37.133 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:46:37.133 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:46:37.133 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:46:37.133 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:46:37.133 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:46:37.133 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:46:37.133 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:46:37.133 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:46:37.133 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:46:37.134 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:46:37.134 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:46:37.134 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:46:37.134 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:46:37.134 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:46:37.157 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:46:37.158 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:46:37.158 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:46:37.158 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:46:37.261 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-06T09:46:37.261 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:rbd mirror flush : flush rbd mirror 2024-04-06T09:46:37.261 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:admin_socket: invalid command 2024-04-06T09:46:37.263 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:46:37.264 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/test 2024-04-06T09:46:37.264 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:1 2024-04-06T09:46:37.264 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:46:37.264 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:46:37.264 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-06T09:46:37.264 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:1 2024-04-06T09:46:37.264 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:46:37.264 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:46:37.265 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:46:37.265 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=1 2024-04-06T09:46:37.265 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-06T09:46:37.265 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:46:37.265 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:46:37.265 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=1 2024-04-06T09:46:37.265 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=1 2024-04-06T09:46:37.266 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-06T09:46:37.266 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:1 2024-04-06T09:46:37.266 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:46:37.266 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:46:37.267 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-06T09:46:37.267 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:1 2024-04-06T09:46:37.267 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:46:37.267 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:46:37.267 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:46:37.267 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=1 2024-04-06T09:46:37.267 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-06T09:46:37.267 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:46:37.267 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:46:37.268 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=1 2024-04-06T09:46:37.268 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=1 2024-04-06T09:46:37.268 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-06T09:46:37.289 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T09:46:37.289 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T09:46:37.289 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T09:46:37.289 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/test 2024-04-06T09:46:37.405 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.0 with skip (0.8676113845502291) > weight (0.25) 2024-04-06T09:46:37.405 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.1 with skip (0.6737248748108177) > weight (0.25) 2024-04-06T09:46:37.405 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.2 with skip (0.9156814124529014) > weight (0.25) 2024-04-06T09:46:37.405 INFO:tasks.rbd_mirror_thrash:kill cluster2.client.mirror.3 2024-04-06T09:46:37.406 INFO:tasks.rbd_mirror.cluster2.client.mirror.3:Sent signal 15 2024-04-06T09:46:37.406 INFO:tasks.rbd_mirror_thrash:waiting for 5 secs before reviving daemons 2024-04-06T09:46:37.407 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi192.stderr:2024-04-06T09:46:37.413+0000 7f87482e2640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.3 (PID: 27491) UID: 1000 2024-04-06T09:46:37.410 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:46:37.412 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:46:37.412 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:46:37.412 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:46:37.412 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:46:37.412 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:46:37.412 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:46:37.412 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:46:37.413 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:46:37.413 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:46:37.413 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:46:37.413 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:46:37.413 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:46:37.414 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:46:37.415 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:46:37.415 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:46:37.468 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 34 2024-04-06T09:46:37.468 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 34 2024-04-06T09:46:37.468 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:46:37.468 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=137, tag_tid=24, entry_tid=1], [object_number=136, tag_tid=24, entry_tid=0], [object_number=139, tag_tid=23, entry_tid=5615], [object_number=138, tag_tid=23, entry_tid=5614]]], state=connected] 2024-04-06T09:46:37.469 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=136, tag_tid=23, entry_tid=4872], [object_number=139, tag_tid=23, entry_tid=4871], [object_number=138, tag_tid=23, entry_tid=4870], [object_number=137, tag_tid=23, entry_tid=4869]]], state=connected] 2024-04-06T09:46:37.472 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:46:37.473 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=137, tag_tid=24, entry_tid=1' 2024-04-06T09:46:37.474 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:46:37.475 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:46:37.475 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:46:37.475 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:46:37.475 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:46:37.475 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:46:37.475 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:46:37.475 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:46:37.475 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:46:37.475 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:46:37.475 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:46:37.475 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:46:37.476 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:46:37.476 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:46:37.476 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:46:37.522 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 34 2024-04-06T09:46:37.523 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 34 2024-04-06T09:46:37.523 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:46:37.523 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=137, tag_tid=24, entry_tid=1], [object_number=136, tag_tid=24, entry_tid=0], [object_number=139, tag_tid=23, entry_tid=5615], [object_number=138, tag_tid=23, entry_tid=5614]]], state=connected] 2024-04-06T09:46:37.523 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=136, tag_tid=23, entry_tid=4872], [object_number=139, tag_tid=23, entry_tid=4871], [object_number=138, tag_tid=23, entry_tid=4870], [object_number=137, tag_tid=23, entry_tid=4869]]], state=connected] 2024-04-06T09:46:37.526 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:46:37.529 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=136, tag_tid=23, entry_tid=4872' 2024-04-06T09:46:37.529 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=137, tag_tid=24, entry_tid=1' -a 'object_number=137, tag_tid=24, entry_tid=1' = 'object_number=136, tag_tid=23, entry_tid=4872' 2024-04-06T09:46:37.529 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=136, tag_tid=23, entry_tid=4872' '!=' 'object_number=138, tag_tid=23, entry_tid=4802' 2024-04-06T09:46:37.529 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:46:37.529 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=136, tag_tid=23, entry_tid=4872' = 'object_number=138, tag_tid=23, entry_tid=4802' 2024-04-06T09:46:37.529 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=136, tag_tid=23, entry_tid=4872' 2024-04-06T09:46:37.529 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=137, tag_tid=24, entry_tid=1' 2024-04-06T09:46:37.529 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:46:37.529 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:46:37.530 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=24 2024-04-06T09:46:37.530 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=136, tag_tid=23, entry_tid=4872' 2024-04-06T09:46:37.531 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:46:37.531 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:46:37.531 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=23 2024-04-06T09:46:37.532 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=137, tag_tid=24, entry_tid=1' 2024-04-06T09:46:37.532 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:46:37.532 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:46:37.533 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:46:37.534 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=136, tag_tid=23, entry_tid=4872' 2024-04-06T09:46:37.534 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:46:37.534 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:46:37.535 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=4872 2024-04-06T09:46:37.535 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 24 = 23 -a 1 -le 4872 2024-04-06T09:46:37.535 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:46:37.535 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:46:37.535 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:46:37.738 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:46:37.738 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:46:37.738 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:46:37.738 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:46:37.738 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:46:37.738 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:46:37.738 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:46:37.738 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:46:37.738 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:46:37.738 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:46:37.738 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:46:37.739 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:46:37.739 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:46:37.739 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:46:37.739 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:46:37.739 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:46:37.739 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:46:37.743 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:46:37.743 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:46:37.743 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:46:37.743 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:46:37.743 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:46:37.743 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:46:37.743 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:46:37.744 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:46:37.744 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:46:37.744 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:46:37.744 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:46:37.744 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:46:37.744 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:46:37.744 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:46:37.744 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:46:37.744 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:46:37.744 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:46:37.744 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:46:37.744 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:46:37.744 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:46:37.744 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:46:37.745 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:46:37.745 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:46:37.745 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:46:37.745 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:46:37.745 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:46:37.745 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:46:37.745 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:46:37.745 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:46:37.745 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:46:37.745 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:46:37.745 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:46:37.768 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:46:37.769 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:46:37.769 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:46:37.769 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:46:37.875 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-06T09:46:37.875 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:rbd mirror flush : flush rbd mirror 2024-04-06T09:46:37.875 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:admin_socket: invalid command 2024-04-06T09:46:37.879 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:46:37.879 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/test 2024-04-06T09:46:37.879 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:1 2024-04-06T09:46:37.879 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:46:37.879 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:46:37.879 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-06T09:46:37.879 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:1 2024-04-06T09:46:37.879 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:46:37.879 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:46:37.879 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:46:37.879 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=1 2024-04-06T09:46:37.879 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-06T09:46:37.879 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:46:37.879 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:46:37.880 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=1 2024-04-06T09:46:37.880 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=1 2024-04-06T09:46:37.880 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-06T09:46:37.880 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:1 2024-04-06T09:46:37.880 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:46:37.880 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:46:37.880 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-06T09:46:37.880 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:1 2024-04-06T09:46:37.880 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:46:37.880 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:46:37.880 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:46:37.880 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=1 2024-04-06T09:46:37.880 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-06T09:46:37.880 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:46:37.880 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:46:37.881 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=1 2024-04-06T09:46:37.881 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=1 2024-04-06T09:46:37.881 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-06T09:46:37.903 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T09:46:37.904 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T09:46:37.904 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T09:46:37.904 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/test 2024-04-06T09:46:38.025 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:46:38.027 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:46:38.027 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:46:38.027 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:46:38.027 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:46:38.027 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:46:38.027 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:46:38.027 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:46:38.027 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:46:38.027 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:46:38.027 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:46:38.028 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:46:38.028 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:46:38.029 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:46:38.029 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:46:38.029 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:46:38.077 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 34 2024-04-06T09:46:38.077 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 34 2024-04-06T09:46:38.077 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:46:38.077 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=137, tag_tid=24, entry_tid=1], [object_number=136, tag_tid=24, entry_tid=0], [object_number=139, tag_tid=23, entry_tid=5615], [object_number=138, tag_tid=23, entry_tid=5614]]], state=connected] 2024-04-06T09:46:38.077 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=138, tag_tid=23, entry_tid=4994], [object_number=137, tag_tid=23, entry_tid=4993], [object_number=136, tag_tid=23, entry_tid=4992], [object_number=139, tag_tid=23, entry_tid=4991]]], state=connected] 2024-04-06T09:46:38.080 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:46:38.082 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=137, tag_tid=24, entry_tid=1' 2024-04-06T09:46:38.083 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:46:38.083 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:46:38.083 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:46:38.083 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:46:38.083 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:46:38.084 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:46:38.084 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:46:38.084 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:46:38.084 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:46:38.084 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:46:38.084 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:46:38.084 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:46:38.085 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:46:38.085 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:46:38.085 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:46:38.130 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 34 2024-04-06T09:46:38.131 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 34 2024-04-06T09:46:38.131 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:46:38.131 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=137, tag_tid=24, entry_tid=1], [object_number=136, tag_tid=24, entry_tid=0], [object_number=139, tag_tid=23, entry_tid=5615], [object_number=138, tag_tid=23, entry_tid=5614]]], state=connected] 2024-04-06T09:46:38.131 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=138, tag_tid=23, entry_tid=4994], [object_number=137, tag_tid=23, entry_tid=4993], [object_number=136, tag_tid=23, entry_tid=4992], [object_number=139, tag_tid=23, entry_tid=4991]]], state=connected] 2024-04-06T09:46:38.134 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:46:38.136 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=138, tag_tid=23, entry_tid=4994' 2024-04-06T09:46:38.136 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=137, tag_tid=24, entry_tid=1' -a 'object_number=137, tag_tid=24, entry_tid=1' = 'object_number=138, tag_tid=23, entry_tid=4994' 2024-04-06T09:46:38.136 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=138, tag_tid=23, entry_tid=4994' '!=' 'object_number=136, tag_tid=23, entry_tid=4872' 2024-04-06T09:46:38.136 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:46:38.136 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=138, tag_tid=23, entry_tid=4994' = 'object_number=136, tag_tid=23, entry_tid=4872' 2024-04-06T09:46:38.136 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=138, tag_tid=23, entry_tid=4994' 2024-04-06T09:46:38.137 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=137, tag_tid=24, entry_tid=1' 2024-04-06T09:46:38.137 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:46:38.137 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:46:38.137 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=24 2024-04-06T09:46:38.138 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=138, tag_tid=23, entry_tid=4994' 2024-04-06T09:46:38.138 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:46:38.138 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:46:38.139 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=23 2024-04-06T09:46:38.140 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=137, tag_tid=24, entry_tid=1' 2024-04-06T09:46:38.140 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:46:38.140 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:46:38.141 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:46:38.141 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=138, tag_tid=23, entry_tid=4994' 2024-04-06T09:46:38.142 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:46:38.142 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:46:38.143 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=4994 2024-04-06T09:46:38.143 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 24 = 23 -a 1 -le 4994 2024-04-06T09:46:38.143 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:46:38.143 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:46:38.143 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:46:38.345 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:46:38.345 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:46:38.345 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:46:38.345 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:46:38.345 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:46:38.345 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:46:38.345 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:46:38.345 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:46:38.345 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:46:38.345 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:46:38.345 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:46:38.345 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:46:38.346 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:46:38.346 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:46:38.346 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:46:38.346 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:46:38.346 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:46:38.349 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:46:38.349 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:46:38.349 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:46:38.349 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:46:38.349 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:46:38.349 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:46:38.349 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:46:38.349 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:46:38.349 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:46:38.349 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:46:38.349 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:46:38.349 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:46:38.349 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:46:38.350 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:46:38.350 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:46:38.350 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:46:38.350 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:46:38.350 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:46:38.350 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:46:38.350 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:46:38.350 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:46:38.350 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:46:38.350 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:46:38.350 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:46:38.350 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:46:38.350 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:46:38.350 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:46:38.351 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:46:38.351 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:46:38.351 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:46:38.351 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:46:38.351 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:46:38.376 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:46:38.377 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:46:38.377 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:46:38.377 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:46:38.482 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-06T09:46:38.482 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:rbd mirror flush : flush rbd mirror 2024-04-06T09:46:38.482 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:admin_socket: invalid command 2024-04-06T09:46:38.485 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:46:38.485 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/test 2024-04-06T09:46:38.485 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:1 2024-04-06T09:46:38.485 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:46:38.485 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:46:38.485 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-06T09:46:38.485 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:1 2024-04-06T09:46:38.485 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:46:38.485 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:46:38.485 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:46:38.485 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=1 2024-04-06T09:46:38.485 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-06T09:46:38.485 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:46:38.486 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:46:38.486 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=1 2024-04-06T09:46:38.486 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=1 2024-04-06T09:46:38.487 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-06T09:46:38.487 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:1 2024-04-06T09:46:38.487 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:46:38.487 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:46:38.487 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-06T09:46:38.487 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:1 2024-04-06T09:46:38.487 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:46:38.488 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:46:38.488 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:46:38.488 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=1 2024-04-06T09:46:38.488 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-06T09:46:38.488 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:46:38.488 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:46:38.488 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=1 2024-04-06T09:46:38.488 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=1 2024-04-06T09:46:38.488 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-06T09:46:38.506 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T09:46:38.506 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T09:46:38.506 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T09:46:38.506 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/test 2024-04-06T09:46:38.642 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:46:38.643 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:46:38.643 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:46:38.643 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:46:38.643 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:46:38.643 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:46:38.643 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:46:38.643 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:46:38.643 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:46:38.643 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:46:38.644 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:46:38.644 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:46:38.644 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:46:38.645 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:46:38.645 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:46:38.645 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:46:38.697 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 34 2024-04-06T09:46:38.697 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 34 2024-04-06T09:46:38.698 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:46:38.698 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=137, tag_tid=24, entry_tid=1], [object_number=136, tag_tid=24, entry_tid=0], [object_number=139, tag_tid=23, entry_tid=5615], [object_number=138, tag_tid=23, entry_tid=5614]]], state=connected] 2024-04-06T09:46:38.698 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=139, tag_tid=23, entry_tid=5095], [object_number=138, tag_tid=23, entry_tid=5094], [object_number=137, tag_tid=23, entry_tid=5093], [object_number=136, tag_tid=23, entry_tid=5092]]], state=connected] 2024-04-06T09:46:38.701 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:46:38.702 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=137, tag_tid=24, entry_tid=1' 2024-04-06T09:46:38.703 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:46:38.704 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:46:38.704 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:46:38.704 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:46:38.704 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:46:38.704 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:46:38.704 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:46:38.704 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:46:38.704 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:46:38.704 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:46:38.704 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:46:38.704 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:46:38.705 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:46:38.705 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:46:38.705 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:46:38.752 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 34 2024-04-06T09:46:38.752 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 34 2024-04-06T09:46:38.752 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:46:38.753 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=137, tag_tid=24, entry_tid=1], [object_number=136, tag_tid=24, entry_tid=0], [object_number=139, tag_tid=23, entry_tid=5615], [object_number=138, tag_tid=23, entry_tid=5614]]], state=connected] 2024-04-06T09:46:38.753 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=139, tag_tid=23, entry_tid=5095], [object_number=138, tag_tid=23, entry_tid=5094], [object_number=137, tag_tid=23, entry_tid=5093], [object_number=136, tag_tid=23, entry_tid=5092]]], state=connected] 2024-04-06T09:46:38.756 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:46:38.758 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=139, tag_tid=23, entry_tid=5095' 2024-04-06T09:46:38.758 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=137, tag_tid=24, entry_tid=1' -a 'object_number=137, tag_tid=24, entry_tid=1' = 'object_number=139, tag_tid=23, entry_tid=5095' 2024-04-06T09:46:38.758 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=139, tag_tid=23, entry_tid=5095' '!=' 'object_number=138, tag_tid=23, entry_tid=4994' 2024-04-06T09:46:38.758 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:46:38.758 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=139, tag_tid=23, entry_tid=5095' = 'object_number=138, tag_tid=23, entry_tid=4994' 2024-04-06T09:46:38.758 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=139, tag_tid=23, entry_tid=5095' 2024-04-06T09:46:38.758 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=137, tag_tid=24, entry_tid=1' 2024-04-06T09:46:38.758 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:46:38.759 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:46:38.759 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=24 2024-04-06T09:46:38.760 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=139, tag_tid=23, entry_tid=5095' 2024-04-06T09:46:38.760 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:46:38.760 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:46:38.761 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=23 2024-04-06T09:46:38.761 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=137, tag_tid=24, entry_tid=1' 2024-04-06T09:46:38.762 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:46:38.762 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:46:38.763 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:46:38.763 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=139, tag_tid=23, entry_tid=5095' 2024-04-06T09:46:38.763 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:46:38.763 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:46:38.765 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=5095 2024-04-06T09:46:38.765 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 24 = 23 -a 1 -le 5095 2024-04-06T09:46:38.765 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:46:38.765 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:46:38.765 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:46:38.967 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:46:38.967 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:46:38.967 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:46:38.967 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:46:38.967 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:46:38.967 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:46:38.967 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:46:38.967 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:46:38.967 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:46:38.967 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:46:38.967 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:46:38.967 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:46:38.968 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:46:38.968 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:46:38.968 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:46:38.968 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:46:38.968 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:46:38.972 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:46:38.972 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:46:38.972 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:46:38.972 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:46:38.972 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:46:38.972 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:46:38.972 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:46:38.972 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:46:38.973 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:46:38.973 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:46:38.973 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:46:38.973 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:46:38.973 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:46:38.973 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:46:38.973 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:46:38.973 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:46:38.973 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:46:38.973 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:46:38.973 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:46:38.973 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:46:38.973 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:46:38.973 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:46:38.973 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:46:38.974 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:46:38.974 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:46:38.974 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:46:38.974 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:46:38.974 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:46:38.974 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:46:38.974 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:46:38.974 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:46:38.974 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:46:38.998 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:46:38.998 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:46:38.998 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:46:38.999 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:46:39.104 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-06T09:46:39.105 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:rbd mirror flush : flush rbd mirror 2024-04-06T09:46:39.105 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:admin_socket: invalid command 2024-04-06T09:46:39.107 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:46:39.107 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/test 2024-04-06T09:46:39.107 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:1 2024-04-06T09:46:39.107 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:46:39.107 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:46:39.107 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-06T09:46:39.107 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:1 2024-04-06T09:46:39.107 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:46:39.107 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:46:39.108 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:46:39.108 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=1 2024-04-06T09:46:39.108 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-06T09:46:39.108 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:46:39.108 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:46:39.108 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=1 2024-04-06T09:46:39.108 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=1 2024-04-06T09:46:39.109 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-06T09:46:39.109 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:1 2024-04-06T09:46:39.109 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:46:39.110 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:46:39.110 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-06T09:46:39.110 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:1 2024-04-06T09:46:39.110 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:46:39.110 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:46:39.110 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:46:39.110 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=1 2024-04-06T09:46:39.110 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-06T09:46:39.110 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:46:39.110 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:46:39.110 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=1 2024-04-06T09:46:39.110 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=1 2024-04-06T09:46:39.111 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-06T09:46:39.133 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T09:46:39.133 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T09:46:39.133 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T09:46:39.133 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/test 2024-04-06T09:46:39.264 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:46:39.265 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:46:39.265 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:46:39.265 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:46:39.265 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:46:39.265 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:46:39.265 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:46:39.265 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:46:39.265 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:46:39.265 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:46:39.266 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:46:39.266 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:46:39.266 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:46:39.268 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:46:39.268 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:46:39.268 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:46:39.320 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 34 2024-04-06T09:46:39.320 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 34 2024-04-06T09:46:39.320 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:46:39.321 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=137, tag_tid=24, entry_tid=1], [object_number=136, tag_tid=24, entry_tid=0], [object_number=139, tag_tid=23, entry_tid=5615], [object_number=138, tag_tid=23, entry_tid=5614]]], state=connected] 2024-04-06T09:46:39.321 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=138, tag_tid=23, entry_tid=5210], [object_number=137, tag_tid=23, entry_tid=5209], [object_number=136, tag_tid=23, entry_tid=5208], [object_number=139, tag_tid=23, entry_tid=5207]]], state=connected] 2024-04-06T09:46:39.324 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:46:39.325 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=137, tag_tid=24, entry_tid=1' 2024-04-06T09:46:39.326 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:46:39.327 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:46:39.327 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:46:39.327 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:46:39.327 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:46:39.327 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:46:39.327 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:46:39.327 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:46:39.327 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:46:39.327 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:46:39.327 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:46:39.327 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:46:39.328 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:46:39.328 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:46:39.328 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:46:39.376 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 34 2024-04-06T09:46:39.376 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 34 2024-04-06T09:46:39.376 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:46:39.376 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=137, tag_tid=24, entry_tid=1], [object_number=136, tag_tid=24, entry_tid=0], [object_number=139, tag_tid=23, entry_tid=5615], [object_number=138, tag_tid=23, entry_tid=5614]]], state=connected] 2024-04-06T09:46:39.376 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=138, tag_tid=23, entry_tid=5210], [object_number=137, tag_tid=23, entry_tid=5209], [object_number=136, tag_tid=23, entry_tid=5208], [object_number=139, tag_tid=23, entry_tid=5207]]], state=connected] 2024-04-06T09:46:39.379 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:46:39.381 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=138, tag_tid=23, entry_tid=5210' 2024-04-06T09:46:39.382 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=137, tag_tid=24, entry_tid=1' -a 'object_number=137, tag_tid=24, entry_tid=1' = 'object_number=138, tag_tid=23, entry_tid=5210' 2024-04-06T09:46:39.382 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=138, tag_tid=23, entry_tid=5210' '!=' 'object_number=139, tag_tid=23, entry_tid=5095' 2024-04-06T09:46:39.382 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:46:39.382 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=138, tag_tid=23, entry_tid=5210' = 'object_number=139, tag_tid=23, entry_tid=5095' 2024-04-06T09:46:39.382 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=138, tag_tid=23, entry_tid=5210' 2024-04-06T09:46:39.382 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=137, tag_tid=24, entry_tid=1' 2024-04-06T09:46:39.382 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:46:39.382 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:46:39.383 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=24 2024-04-06T09:46:39.383 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=138, tag_tid=23, entry_tid=5210' 2024-04-06T09:46:39.383 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:46:39.384 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:46:39.384 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=23 2024-04-06T09:46:39.385 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=137, tag_tid=24, entry_tid=1' 2024-04-06T09:46:39.385 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:46:39.385 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:46:39.386 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:46:39.387 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=138, tag_tid=23, entry_tid=5210' 2024-04-06T09:46:39.387 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:46:39.387 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:46:39.388 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=5210 2024-04-06T09:46:39.388 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 24 = 23 -a 1 -le 5210 2024-04-06T09:46:39.388 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:46:39.389 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:46:39.389 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:46:39.590 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:46:39.591 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:46:39.591 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:46:39.591 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:46:39.591 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:46:39.591 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:46:39.591 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:46:39.591 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:46:39.591 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:46:39.591 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:46:39.591 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:46:39.592 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:46:39.592 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:46:39.592 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:46:39.592 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:46:39.592 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:46:39.592 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:46:39.595 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:46:39.595 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:46:39.595 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:46:39.596 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:46:39.596 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:46:39.596 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:46:39.596 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:46:39.596 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:46:39.596 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:46:39.596 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:46:39.596 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:46:39.596 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:46:39.596 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:46:39.596 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:46:39.596 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:46:39.597 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:46:39.597 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:46:39.597 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:46:39.597 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:46:39.597 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:46:39.597 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:46:39.597 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:46:39.597 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:46:39.597 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:46:39.597 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:46:39.597 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:46:39.597 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:46:39.597 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:46:39.597 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:46:39.597 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:46:39.598 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:46:39.598 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:46:39.621 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:46:39.621 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:46:39.622 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:46:39.622 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:46:39.729 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-06T09:46:39.729 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:rbd mirror flush : flush rbd mirror 2024-04-06T09:46:39.729 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:admin_socket: invalid command 2024-04-06T09:46:39.733 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:46:39.733 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/test 2024-04-06T09:46:39.733 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:1 2024-04-06T09:46:39.733 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:46:39.733 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:46:39.733 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-06T09:46:39.733 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:1 2024-04-06T09:46:39.733 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:46:39.734 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:46:39.734 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:46:39.734 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=1 2024-04-06T09:46:39.734 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-06T09:46:39.734 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:46:39.734 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:46:39.734 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=1 2024-04-06T09:46:39.734 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=1 2024-04-06T09:46:39.734 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-06T09:46:39.734 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:1 2024-04-06T09:46:39.734 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:46:39.734 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:46:39.735 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-06T09:46:39.735 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:1 2024-04-06T09:46:39.735 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:46:39.735 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:46:39.735 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:46:39.735 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=1 2024-04-06T09:46:39.735 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-06T09:46:39.735 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:46:39.735 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:46:39.735 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=1 2024-04-06T09:46:39.735 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=1 2024-04-06T09:46:39.735 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-06T09:46:39.752 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T09:46:39.752 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T09:46:39.752 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T09:46:39.752 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/test 2024-04-06T09:46:39.879 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:46:39.881 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:46:39.881 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:46:39.881 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:46:39.881 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:46:39.881 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:46:39.881 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:46:39.881 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:46:39.881 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:46:39.881 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:46:39.881 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:46:39.882 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:46:39.882 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:46:39.883 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:46:39.883 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:46:39.883 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:46:39.940 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 34 2024-04-06T09:46:39.941 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 34 2024-04-06T09:46:39.941 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:46:39.941 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=137, tag_tid=24, entry_tid=1], [object_number=136, tag_tid=24, entry_tid=0], [object_number=139, tag_tid=23, entry_tid=5615], [object_number=138, tag_tid=23, entry_tid=5614]]], state=connected] 2024-04-06T09:46:39.941 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=136, tag_tid=23, entry_tid=5308], [object_number=139, tag_tid=23, entry_tid=5307], [object_number=138, tag_tid=23, entry_tid=5306], [object_number=137, tag_tid=23, entry_tid=5305]]], state=connected] 2024-04-06T09:46:39.944 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:46:39.946 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=137, tag_tid=24, entry_tid=1' 2024-04-06T09:46:39.947 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:46:39.947 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:46:39.947 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:46:39.947 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:46:39.947 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:46:39.947 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:46:39.948 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:46:39.948 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:46:39.948 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:46:39.948 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:46:39.948 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:46:39.948 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:46:39.948 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:46:39.949 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:46:39.949 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:46:40.201 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 34 2024-04-06T09:46:40.201 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 34 2024-04-06T09:46:40.201 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:46:40.201 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=137, tag_tid=24, entry_tid=1], [object_number=136, tag_tid=24, entry_tid=0], [object_number=139, tag_tid=23, entry_tid=5615], [object_number=138, tag_tid=23, entry_tid=5614]]], state=connected] 2024-04-06T09:46:40.201 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=136, tag_tid=23, entry_tid=5308], [object_number=139, tag_tid=23, entry_tid=5307], [object_number=138, tag_tid=23, entry_tid=5306], [object_number=137, tag_tid=23, entry_tid=5305]]], state=connected] 2024-04-06T09:46:40.206 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:46:40.209 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=136, tag_tid=23, entry_tid=5308' 2024-04-06T09:46:40.209 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=137, tag_tid=24, entry_tid=1' -a 'object_number=137, tag_tid=24, entry_tid=1' = 'object_number=136, tag_tid=23, entry_tid=5308' 2024-04-06T09:46:40.209 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=136, tag_tid=23, entry_tid=5308' '!=' 'object_number=138, tag_tid=23, entry_tid=5210' 2024-04-06T09:46:40.209 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:46:40.209 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=136, tag_tid=23, entry_tid=5308' = 'object_number=138, tag_tid=23, entry_tid=5210' 2024-04-06T09:46:40.209 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=136, tag_tid=23, entry_tid=5308' 2024-04-06T09:46:40.210 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=137, tag_tid=24, entry_tid=1' 2024-04-06T09:46:40.210 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:46:40.210 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:46:40.211 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=24 2024-04-06T09:46:40.213 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=136, tag_tid=23, entry_tid=5308' 2024-04-06T09:46:40.213 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:46:40.213 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:46:40.214 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=23 2024-04-06T09:46:40.215 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=137, tag_tid=24, entry_tid=1' 2024-04-06T09:46:40.215 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:46:40.215 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:46:40.218 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:46:40.219 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=136, tag_tid=23, entry_tid=5308' 2024-04-06T09:46:40.219 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:46:40.219 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:46:40.221 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=5308 2024-04-06T09:46:40.221 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 24 = 23 -a 1 -le 5308 2024-04-06T09:46:40.221 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:46:40.221 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:46:40.221 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:46:40.423 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:46:40.423 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:46:40.423 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:46:40.423 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:46:40.423 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:46:40.423 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:46:40.423 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:46:40.423 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:46:40.424 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:46:40.424 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:46:40.424 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:46:40.424 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:46:40.424 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:46:40.424 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:46:40.424 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:46:40.424 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:46:40.425 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:46:40.428 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:46:40.428 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:46:40.428 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:46:40.428 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:46:40.428 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:46:40.429 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:46:40.429 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:46:40.429 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:46:40.429 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:46:40.429 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:46:40.429 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:46:40.429 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:46:40.429 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:46:40.429 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:46:40.429 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:46:40.429 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:46:40.430 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:46:40.430 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:46:40.430 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:46:40.430 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:46:40.430 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:46:40.430 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:46:40.430 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:46:40.430 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:46:40.430 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:46:40.430 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:46:40.430 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:46:40.430 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:46:40.431 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:46:40.431 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:46:40.431 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:46:40.431 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:46:40.454 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:46:40.455 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:46:40.455 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:46:40.455 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:46:40.561 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-06T09:46:40.561 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:rbd mirror flush : flush rbd mirror 2024-04-06T09:46:40.561 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:admin_socket: invalid command 2024-04-06T09:46:40.565 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:46:40.565 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/test 2024-04-06T09:46:40.565 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:1 2024-04-06T09:46:40.565 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:46:40.565 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:46:40.565 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-06T09:46:40.565 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:1 2024-04-06T09:46:40.565 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:46:40.565 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:46:40.565 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:46:40.566 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=1 2024-04-06T09:46:40.566 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-06T09:46:40.566 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:46:40.566 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:46:40.566 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=1 2024-04-06T09:46:40.566 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=1 2024-04-06T09:46:40.566 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-06T09:46:40.566 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:1 2024-04-06T09:46:40.567 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:46:40.567 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:46:40.567 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-06T09:46:40.567 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:1 2024-04-06T09:46:40.567 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:46:40.567 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:46:40.567 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:46:40.567 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=1 2024-04-06T09:46:40.567 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-06T09:46:40.567 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:46:40.567 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:46:40.567 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=1 2024-04-06T09:46:40.568 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=1 2024-04-06T09:46:40.568 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-06T09:46:40.588 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T09:46:40.589 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T09:46:40.589 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T09:46:40.589 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/test 2024-04-06T09:46:40.716 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:46:40.718 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:46:40.718 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:46:40.718 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:46:40.718 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:46:40.718 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:46:40.718 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:46:40.718 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:46:40.719 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:46:40.719 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:46:40.719 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:46:40.719 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:46:40.719 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:46:40.720 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:46:40.720 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:46:40.720 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:46:40.779 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 34 2024-04-06T09:46:40.779 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 34 2024-04-06T09:46:40.779 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:46:40.779 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=137, tag_tid=24, entry_tid=1], [object_number=136, tag_tid=24, entry_tid=0], [object_number=139, tag_tid=23, entry_tid=5615], [object_number=138, tag_tid=23, entry_tid=5614]]], state=connected] 2024-04-06T09:46:40.779 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=138, tag_tid=23, entry_tid=5438], [object_number=137, tag_tid=23, entry_tid=5437], [object_number=136, tag_tid=23, entry_tid=5436], [object_number=139, tag_tid=23, entry_tid=5435]]], state=connected] 2024-04-06T09:46:40.782 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:46:40.784 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=137, tag_tid=24, entry_tid=1' 2024-04-06T09:46:40.785 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:46:40.785 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:46:40.785 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:46:40.785 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:46:40.785 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:46:40.785 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:46:40.785 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:46:40.785 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:46:40.785 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:46:40.786 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:46:40.786 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:46:40.786 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:46:40.786 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:46:40.787 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:46:40.787 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:46:40.835 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 34 2024-04-06T09:46:40.835 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 34 2024-04-06T09:46:40.835 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:46:40.835 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=137, tag_tid=24, entry_tid=1], [object_number=136, tag_tid=24, entry_tid=0], [object_number=139, tag_tid=23, entry_tid=5615], [object_number=138, tag_tid=23, entry_tid=5614]]], state=connected] 2024-04-06T09:46:40.835 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=138, tag_tid=23, entry_tid=5438], [object_number=137, tag_tid=23, entry_tid=5437], [object_number=136, tag_tid=23, entry_tid=5436], [object_number=139, tag_tid=23, entry_tid=5435]]], state=connected] 2024-04-06T09:46:40.838 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:46:40.841 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=138, tag_tid=23, entry_tid=5438' 2024-04-06T09:46:40.841 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=137, tag_tid=24, entry_tid=1' -a 'object_number=137, tag_tid=24, entry_tid=1' = 'object_number=138, tag_tid=23, entry_tid=5438' 2024-04-06T09:46:40.841 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=138, tag_tid=23, entry_tid=5438' '!=' 'object_number=136, tag_tid=23, entry_tid=5308' 2024-04-06T09:46:40.841 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:46:40.841 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=138, tag_tid=23, entry_tid=5438' = 'object_number=136, tag_tid=23, entry_tid=5308' 2024-04-06T09:46:40.841 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=138, tag_tid=23, entry_tid=5438' 2024-04-06T09:46:40.841 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=137, tag_tid=24, entry_tid=1' 2024-04-06T09:46:40.841 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:46:40.841 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:46:40.842 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=24 2024-04-06T09:46:40.842 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=138, tag_tid=23, entry_tid=5438' 2024-04-06T09:46:40.843 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:46:40.843 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:46:40.843 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=23 2024-04-06T09:46:40.844 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=137, tag_tid=24, entry_tid=1' 2024-04-06T09:46:40.844 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:46:40.844 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:46:40.845 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:46:40.846 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=138, tag_tid=23, entry_tid=5438' 2024-04-06T09:46:40.846 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:46:40.846 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:46:40.847 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=5438 2024-04-06T09:46:40.847 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 24 = 23 -a 1 -le 5438 2024-04-06T09:46:40.847 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:46:40.848 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:46:40.848 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:46:41.049 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:46:41.049 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:46:41.049 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:46:41.049 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:46:41.049 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:46:41.049 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:46:41.050 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:46:41.050 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:46:41.050 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:46:41.050 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:46:41.050 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:46:41.050 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:46:41.050 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:46:41.050 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:46:41.050 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:46:41.051 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:46:41.051 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:46:41.053 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:46:41.053 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:46:41.053 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:46:41.053 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:46:41.053 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:46:41.053 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:46:41.053 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:46:41.053 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:46:41.054 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:46:41.054 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:46:41.054 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:46:41.054 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:46:41.054 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:46:41.054 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:46:41.054 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:46:41.054 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:46:41.055 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:46:41.056 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:46:41.056 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:46:41.056 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:46:41.056 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:46:41.056 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:46:41.056 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:46:41.056 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:46:41.056 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:46:41.056 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:46:41.056 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:46:41.056 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:46:41.056 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:46:41.056 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:46:41.056 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:46:41.057 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:46:41.080 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:46:41.080 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:46:41.080 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:46:41.080 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:46:41.187 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-06T09:46:41.187 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:rbd mirror flush : flush rbd mirror 2024-04-06T09:46:41.187 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:admin_socket: invalid command 2024-04-06T09:46:41.190 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:46:41.190 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/test 2024-04-06T09:46:41.191 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:1 2024-04-06T09:46:41.191 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:46:41.191 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:46:41.191 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-06T09:46:41.191 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:1 2024-04-06T09:46:41.191 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:46:41.191 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:46:41.191 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:46:41.191 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=1 2024-04-06T09:46:41.191 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-06T09:46:41.191 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:46:41.191 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:46:41.191 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=1 2024-04-06T09:46:41.191 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=1 2024-04-06T09:46:41.192 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-06T09:46:41.192 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:1 2024-04-06T09:46:41.192 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:46:41.192 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:46:41.192 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-06T09:46:41.192 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:1 2024-04-06T09:46:41.192 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:46:41.192 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:46:41.192 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:46:41.192 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=1 2024-04-06T09:46:41.192 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-06T09:46:41.192 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:46:41.192 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:46:41.192 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=1 2024-04-06T09:46:41.192 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=1 2024-04-06T09:46:41.193 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-06T09:46:41.212 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T09:46:41.212 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T09:46:41.212 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T09:46:41.212 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/test 2024-04-06T09:46:41.339 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:46:41.340 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:46:41.340 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:46:41.340 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:46:41.341 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:46:41.341 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:46:41.341 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:46:41.341 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:46:41.341 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:46:41.341 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:46:41.342 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:46:41.342 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:46:41.342 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:46:41.343 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:46:41.344 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:46:41.344 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:46:41.398 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 34 2024-04-06T09:46:41.398 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 34 2024-04-06T09:46:41.398 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:46:41.399 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=137, tag_tid=24, entry_tid=1], [object_number=136, tag_tid=24, entry_tid=0], [object_number=139, tag_tid=23, entry_tid=5615], [object_number=138, tag_tid=23, entry_tid=5614]]], state=connected] 2024-04-06T09:46:41.399 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=139, tag_tid=23, entry_tid=5559], [object_number=138, tag_tid=23, entry_tid=5558], [object_number=137, tag_tid=23, entry_tid=5557], [object_number=136, tag_tid=23, entry_tid=5556]]], state=connected] 2024-04-06T09:46:41.402 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:46:41.404 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=137, tag_tid=24, entry_tid=1' 2024-04-06T09:46:41.405 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:46:41.405 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:46:41.405 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:46:41.405 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:46:41.405 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:46:41.405 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:46:41.405 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:46:41.405 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:46:41.406 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:46:41.406 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:46:41.406 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:46:41.406 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:46:41.406 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:46:41.407 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:46:41.407 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:46:41.454 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 34 2024-04-06T09:46:41.454 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 34 2024-04-06T09:46:41.454 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:46:41.454 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=137, tag_tid=24, entry_tid=1], [object_number=136, tag_tid=24, entry_tid=0], [object_number=139, tag_tid=23, entry_tid=5615], [object_number=138, tag_tid=23, entry_tid=5614]]], state=connected] 2024-04-06T09:46:41.454 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=139, tag_tid=23, entry_tid=5559], [object_number=138, tag_tid=23, entry_tid=5558], [object_number=137, tag_tid=23, entry_tid=5557], [object_number=136, tag_tid=23, entry_tid=5556]]], state=connected] 2024-04-06T09:46:41.458 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:46:41.460 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=139, tag_tid=23, entry_tid=5559' 2024-04-06T09:46:41.460 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=137, tag_tid=24, entry_tid=1' -a 'object_number=137, tag_tid=24, entry_tid=1' = 'object_number=139, tag_tid=23, entry_tid=5559' 2024-04-06T09:46:41.460 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=139, tag_tid=23, entry_tid=5559' '!=' 'object_number=138, tag_tid=23, entry_tid=5438' 2024-04-06T09:46:41.460 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:46:41.460 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=139, tag_tid=23, entry_tid=5559' = 'object_number=138, tag_tid=23, entry_tid=5438' 2024-04-06T09:46:41.460 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=139, tag_tid=23, entry_tid=5559' 2024-04-06T09:46:41.460 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=137, tag_tid=24, entry_tid=1' 2024-04-06T09:46:41.461 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:46:41.461 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:46:41.461 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=24 2024-04-06T09:46:41.462 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=139, tag_tid=23, entry_tid=5559' 2024-04-06T09:46:41.462 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:46:41.462 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:46:41.463 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=23 2024-04-06T09:46:41.463 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=137, tag_tid=24, entry_tid=1' 2024-04-06T09:46:41.464 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:46:41.464 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:46:41.464 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:46:41.465 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=139, tag_tid=23, entry_tid=5559' 2024-04-06T09:46:41.465 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:46:41.465 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:46:41.467 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=5559 2024-04-06T09:46:41.467 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 24 = 23 -a 1 -le 5559 2024-04-06T09:46:41.467 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:46:41.467 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:46:41.467 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:46:41.668 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:46:41.669 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:46:41.669 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:46:41.669 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:46:41.669 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:46:41.669 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:46:41.669 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:46:41.669 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:46:41.669 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:46:41.669 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:46:41.669 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:46:41.669 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:46:41.669 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:46:41.669 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:46:41.669 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:46:41.670 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:46:41.670 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:46:41.673 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:46:41.673 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:46:41.673 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:46:41.673 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:46:41.673 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:46:41.674 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:46:41.674 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:46:41.674 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:46:41.674 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:46:41.674 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:46:41.674 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:46:41.674 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:46:41.674 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:46:41.674 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:46:41.674 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:46:41.674 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:46:41.674 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:46:41.674 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:46:41.675 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:46:41.675 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:46:41.675 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:46:41.675 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:46:41.675 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:46:41.675 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:46:41.675 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:46:41.675 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:46:41.675 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:46:41.675 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:46:41.675 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:46:41.675 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:46:41.675 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:46:41.676 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:46:41.698 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:46:41.699 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:46:41.699 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:46:41.699 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:46:41.806 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-06T09:46:41.806 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:rbd mirror flush : flush rbd mirror 2024-04-06T09:46:41.807 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:admin_socket: invalid command 2024-04-06T09:46:41.810 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:46:41.811 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/test 2024-04-06T09:46:41.811 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:1 2024-04-06T09:46:41.811 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:46:41.811 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:46:41.811 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-06T09:46:41.811 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:1 2024-04-06T09:46:41.811 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:46:41.811 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:46:41.811 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:46:41.811 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=1 2024-04-06T09:46:41.811 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-06T09:46:41.811 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:46:41.811 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:46:41.812 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=1 2024-04-06T09:46:41.812 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=1 2024-04-06T09:46:41.812 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-06T09:46:41.812 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:1 2024-04-06T09:46:41.812 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:46:41.812 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:46:41.812 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-06T09:46:41.812 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:1 2024-04-06T09:46:41.812 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:46:41.812 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:46:41.812 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:46:41.812 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=1 2024-04-06T09:46:41.812 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-06T09:46:41.812 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:46:41.813 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:46:41.813 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=1 2024-04-06T09:46:41.813 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=1 2024-04-06T09:46:41.813 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-06T09:46:41.829 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T09:46:41.830 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T09:46:41.830 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T09:46:41.830 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/test 2024-04-06T09:46:41.955 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:46:41.956 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:46:41.956 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:46:41.956 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:46:41.956 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:46:41.956 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:46:41.956 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:46:41.956 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:46:41.956 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:46:41.956 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:46:41.957 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:46:41.957 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:46:41.957 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:46:41.959 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:46:41.959 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:46:41.959 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:46:42.008 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 34 2024-04-06T09:46:42.008 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 34 2024-04-06T09:46:42.008 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:46:42.008 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=137, tag_tid=24, entry_tid=1], [object_number=136, tag_tid=24, entry_tid=0], [object_number=139, tag_tid=23, entry_tid=5615], [object_number=138, tag_tid=23, entry_tid=5614]]], state=connected] 2024-04-06T09:46:42.008 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=136, tag_tid=23, entry_tid=5616], [object_number=139, tag_tid=23, entry_tid=5615], [object_number=138, tag_tid=23, entry_tid=5614], [object_number=137, tag_tid=23, entry_tid=5613]]], state=connected] 2024-04-06T09:46:42.012 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:46:42.013 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=137, tag_tid=24, entry_tid=1' 2024-04-06T09:46:42.015 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:46:42.015 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:46:42.015 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:46:42.015 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:46:42.015 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:46:42.015 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:46:42.015 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:46:42.015 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:46:42.015 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:46:42.015 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:46:42.015 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:46:42.015 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:46:42.016 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:46:42.016 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:46:42.016 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:46:42.063 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 34 2024-04-06T09:46:42.063 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 34 2024-04-06T09:46:42.063 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:46:42.063 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=137, tag_tid=24, entry_tid=1], [object_number=136, tag_tid=24, entry_tid=0], [object_number=139, tag_tid=23, entry_tid=5615], [object_number=138, tag_tid=23, entry_tid=5614]]], state=connected] 2024-04-06T09:46:42.063 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=136, tag_tid=23, entry_tid=5616], [object_number=139, tag_tid=23, entry_tid=5615], [object_number=138, tag_tid=23, entry_tid=5614], [object_number=137, tag_tid=23, entry_tid=5613]]], state=connected] 2024-04-06T09:46:42.066 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:46:42.068 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=136, tag_tid=23, entry_tid=5616' 2024-04-06T09:46:42.068 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=137, tag_tid=24, entry_tid=1' -a 'object_number=137, tag_tid=24, entry_tid=1' = 'object_number=136, tag_tid=23, entry_tid=5616' 2024-04-06T09:46:42.068 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=136, tag_tid=23, entry_tid=5616' '!=' 'object_number=139, tag_tid=23, entry_tid=5559' 2024-04-06T09:46:42.068 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:46:42.068 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=136, tag_tid=23, entry_tid=5616' = 'object_number=139, tag_tid=23, entry_tid=5559' 2024-04-06T09:46:42.068 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=136, tag_tid=23, entry_tid=5616' 2024-04-06T09:46:42.069 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=137, tag_tid=24, entry_tid=1' 2024-04-06T09:46:42.069 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:46:42.069 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:46:42.070 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=24 2024-04-06T09:46:42.070 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=136, tag_tid=23, entry_tid=5616' 2024-04-06T09:46:42.070 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:46:42.071 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:46:42.071 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=23 2024-04-06T09:46:42.072 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=137, tag_tid=24, entry_tid=1' 2024-04-06T09:46:42.072 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:46:42.072 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:46:42.073 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:46:42.074 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=136, tag_tid=23, entry_tid=5616' 2024-04-06T09:46:42.074 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:46:42.074 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:46:42.075 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=5616 2024-04-06T09:46:42.075 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 24 = 23 -a 1 -le 5616 2024-04-06T09:46:42.075 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:46:42.075 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:46:42.075 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:46:42.277 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:46:42.277 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:46:42.277 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:46:42.277 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:46:42.277 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:46:42.277 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:46:42.277 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:46:42.277 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:46:42.277 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:46:42.277 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:46:42.277 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:46:42.277 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:46:42.277 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:46:42.278 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:46:42.278 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:46:42.278 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:46:42.279 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:46:42.283 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:46:42.283 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:46:42.283 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:46:42.283 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:46:42.283 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:46:42.283 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:46:42.283 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:46:42.283 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:46:42.283 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:46:42.283 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:46:42.283 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:46:42.283 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:46:42.283 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:46:42.283 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:46:42.283 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:46:42.284 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:46:42.284 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:46:42.284 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:46:42.284 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:46:42.284 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:46:42.284 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:46:42.284 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:46:42.284 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:46:42.284 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:46:42.284 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:46:42.284 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:46:42.284 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:46:42.284 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:46:42.284 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:46:42.284 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:46:42.284 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:46:42.285 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:46:42.307 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:46:42.308 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:46:42.308 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:46:42.308 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:46:42.406 INFO:tasks.rbd_mirror_thrash:waiting for cluster2.client.mirror.3 2024-04-06T09:46:42.407 INFO:teuthology.orchestra.run:waiting for 600 2024-04-06T09:46:42.407 INFO:tasks.rbd_mirror_thrash:reviving cluster2.client.mirror.3 2024-04-06T09:46:42.407 INFO:tasks.rbd_mirror.cluster2.client.mirror.3:Restarting daemon 2024-04-06T09:46:42.407 DEBUG:teuthology.orchestra.run.smithi192:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.3 2024-04-06T09:46:42.410 INFO:tasks.rbd_mirror.cluster2.client.mirror.3:Started 2024-04-06T09:46:42.410 INFO:tasks.rbd_mirror_thrash:waiting for 75 secs before thrashing 2024-04-06T09:46:42.424 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-06T09:46:42.424 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:rbd mirror flush : flush rbd mirror 2024-04-06T09:46:42.424 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:admin_socket: invalid command 2024-04-06T09:46:42.427 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:46:42.427 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/test 2024-04-06T09:46:42.427 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:1 2024-04-06T09:46:42.427 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:46:42.427 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:46:42.427 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-06T09:46:42.427 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:1 2024-04-06T09:46:42.427 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:46:42.427 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:46:42.427 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:46:42.427 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=1 2024-04-06T09:46:42.427 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-06T09:46:42.427 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:46:42.428 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:46:42.428 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=1 2024-04-06T09:46:42.428 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=1 2024-04-06T09:46:42.428 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-06T09:46:42.428 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:1 2024-04-06T09:46:42.428 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:46:42.428 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:46:42.428 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-06T09:46:42.428 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:1 2024-04-06T09:46:42.428 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:46:42.428 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:46:42.428 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:46:42.428 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=1 2024-04-06T09:46:42.428 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-06T09:46:42.428 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:46:42.429 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:46:42.429 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=1 2024-04-06T09:46:42.429 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=1 2024-04-06T09:46:42.429 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-06T09:46:42.447 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T09:46:42.447 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T09:46:42.447 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T09:46:42.447 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/test 2024-04-06T09:46:42.574 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:46:42.575 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:46:42.575 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:46:42.575 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:46:42.575 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:46:42.575 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:46:42.575 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:46:42.575 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:46:42.575 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:46:42.575 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:46:42.576 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:46:42.576 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:46:42.576 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:46:42.576 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:46:42.576 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:46:42.576 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:46:42.627 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 34 2024-04-06T09:46:42.628 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 34 2024-04-06T09:46:42.628 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:46:42.628 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=137, tag_tid=24, entry_tid=1], [object_number=136, tag_tid=24, entry_tid=0], [object_number=139, tag_tid=23, entry_tid=5615], [object_number=138, tag_tid=23, entry_tid=5614]]], state=connected] 2024-04-06T09:46:42.628 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=136, tag_tid=23, entry_tid=5616], [object_number=139, tag_tid=23, entry_tid=5615], [object_number=138, tag_tid=23, entry_tid=5614], [object_number=137, tag_tid=23, entry_tid=5613]]], state=connected] 2024-04-06T09:46:42.631 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:46:42.633 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=137, tag_tid=24, entry_tid=1' 2024-04-06T09:46:42.634 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:46:42.634 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:46:42.634 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:46:42.634 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:46:42.635 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:46:42.635 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:46:42.635 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:46:42.635 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:46:42.635 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:46:42.635 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:46:42.635 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:46:42.635 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:46:42.636 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:46:42.636 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:46:42.636 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:46:42.684 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 34 2024-04-06T09:46:42.684 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 34 2024-04-06T09:46:42.684 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:46:42.684 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=137, tag_tid=24, entry_tid=1], [object_number=136, tag_tid=24, entry_tid=0], [object_number=139, tag_tid=23, entry_tid=5615], [object_number=138, tag_tid=23, entry_tid=5614]]], state=connected] 2024-04-06T09:46:42.684 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=136, tag_tid=23, entry_tid=5616], [object_number=139, tag_tid=23, entry_tid=5615], [object_number=138, tag_tid=23, entry_tid=5614], [object_number=137, tag_tid=23, entry_tid=5613]]], state=connected] 2024-04-06T09:46:42.689 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:46:42.690 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=136, tag_tid=23, entry_tid=5616' 2024-04-06T09:46:42.690 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=137, tag_tid=24, entry_tid=1' -a 'object_number=137, tag_tid=24, entry_tid=1' = 'object_number=136, tag_tid=23, entry_tid=5616' 2024-04-06T09:46:42.690 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=136, tag_tid=23, entry_tid=5616' '!=' 'object_number=136, tag_tid=23, entry_tid=5616' 2024-04-06T09:46:42.690 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:46:42.690 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.4 2024-04-06T09:46:43.091 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:46:43.091 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:46:43.092 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:46:43.092 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:46:43.092 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:46:43.092 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:46:43.092 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:46:43.092 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:46:43.092 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:46:43.092 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:46:43.092 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:46:43.092 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:46:43.092 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:46:43.092 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:46:43.092 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:46:43.093 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:46:43.093 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:46:43.096 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:46:43.096 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:46:43.097 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:46:43.097 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:46:43.097 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:46:43.097 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:46:43.097 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:46:43.097 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:46:43.097 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:46:43.097 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:46:43.097 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:46:43.097 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:46:43.097 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:46:43.097 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:46:43.097 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:46:43.097 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:46:43.097 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:46:43.097 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:46:43.098 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:46:43.098 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:46:43.098 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:46:43.098 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:46:43.098 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:46:43.098 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:46:43.098 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:46:43.098 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:46:43.098 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:46:43.098 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:46:43.098 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:46:43.098 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:46:43.098 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:46:43.099 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:46:43.124 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:46:43.124 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:46:43.125 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:46:43.125 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:46:43.229 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-06T09:46:43.229 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:rbd mirror flush : flush rbd mirror 2024-04-06T09:46:43.229 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:admin_socket: invalid command 2024-04-06T09:46:43.231 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:46:43.231 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/test 2024-04-06T09:46:43.231 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:1 2024-04-06T09:46:43.231 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:46:43.231 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:46:43.231 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-06T09:46:43.232 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:1 2024-04-06T09:46:43.232 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:46:43.232 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:46:43.232 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:46:43.232 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=1 2024-04-06T09:46:43.232 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-06T09:46:43.232 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:46:43.232 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:46:43.232 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=1 2024-04-06T09:46:43.232 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=1 2024-04-06T09:46:43.233 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-06T09:46:43.234 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:1 2024-04-06T09:46:43.234 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:46:43.234 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:46:43.234 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-06T09:46:43.234 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:1 2024-04-06T09:46:43.234 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:46:43.234 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:46:43.234 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:46:43.234 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=1 2024-04-06T09:46:43.234 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-06T09:46:43.234 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:46:43.234 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:46:43.234 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=1 2024-04-06T09:46:43.235 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=1 2024-04-06T09:46:43.235 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-06T09:46:43.257 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T09:46:43.257 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T09:46:43.257 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T09:46:43.257 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/test 2024-04-06T09:46:43.382 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:46:43.382 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:46:43.382 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:46:43.383 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:46:43.383 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:46:43.383 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:46:43.383 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:46:43.383 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:46:43.383 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:46:43.383 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:46:43.384 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:46:43.384 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:46:43.384 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:46:43.386 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:46:43.387 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:46:43.387 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:46:43.439 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 34 2024-04-06T09:46:43.439 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 34 2024-04-06T09:46:43.439 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:46:43.440 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=137, tag_tid=24, entry_tid=1], [object_number=136, tag_tid=24, entry_tid=0], [object_number=139, tag_tid=23, entry_tid=5615], [object_number=138, tag_tid=23, entry_tid=5614]]], state=connected] 2024-04-06T09:46:43.440 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=136, tag_tid=23, entry_tid=5616], [object_number=139, tag_tid=23, entry_tid=5615], [object_number=138, tag_tid=23, entry_tid=5614], [object_number=137, tag_tid=23, entry_tid=5613]]], state=connected] 2024-04-06T09:46:43.443 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:46:43.444 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=137, tag_tid=24, entry_tid=1' 2024-04-06T09:46:43.445 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:46:43.445 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:46:43.445 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:46:43.445 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:46:43.446 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:46:43.446 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:46:43.446 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:46:43.446 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:46:43.446 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:46:43.446 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:46:43.446 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:46:43.446 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:46:43.447 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:46:43.447 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:46:43.447 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:46:43.494 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 34 2024-04-06T09:46:43.494 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 34 2024-04-06T09:46:43.494 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:46:43.494 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=137, tag_tid=24, entry_tid=1], [object_number=136, tag_tid=24, entry_tid=0], [object_number=139, tag_tid=23, entry_tid=5615], [object_number=138, tag_tid=23, entry_tid=5614]]], state=connected] 2024-04-06T09:46:43.494 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=136, tag_tid=23, entry_tid=5616], [object_number=139, tag_tid=23, entry_tid=5615], [object_number=138, tag_tid=23, entry_tid=5614], [object_number=137, tag_tid=23, entry_tid=5613]]], state=connected] 2024-04-06T09:46:43.498 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:46:43.499 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=136, tag_tid=23, entry_tid=5616' 2024-04-06T09:46:43.500 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=137, tag_tid=24, entry_tid=1' -a 'object_number=137, tag_tid=24, entry_tid=1' = 'object_number=136, tag_tid=23, entry_tid=5616' 2024-04-06T09:46:43.500 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=136, tag_tid=23, entry_tid=5616' '!=' 'object_number=136, tag_tid=23, entry_tid=5616' 2024-04-06T09:46:43.500 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:46:43.500 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.8 2024-04-06T09:46:44.301 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:46:44.302 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:46:44.302 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:46:44.302 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:46:44.302 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:46:44.302 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:46:44.302 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:46:44.302 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:46:44.302 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:46:44.302 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:46:44.302 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:46:44.302 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:46:44.302 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:46:44.302 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:46:44.302 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:46:44.303 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:46:44.303 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:46:44.306 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:46:44.306 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:46:44.306 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:46:44.306 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:46:44.307 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:46:44.307 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:46:44.307 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:46:44.307 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:46:44.307 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:46:44.307 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:46:44.307 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:46:44.307 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:46:44.307 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:46:44.307 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:46:44.307 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:46:44.307 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:46:44.307 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:46:44.307 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:46:44.307 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:46:44.307 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:46:44.308 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:46:44.309 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:46:44.309 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:46:44.309 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:46:44.309 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:46:44.309 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:46:44.309 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:46:44.309 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:46:44.309 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:46:44.309 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:46:44.309 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:46:44.309 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:46:44.333 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:46:44.333 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:46:44.333 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:46:44.333 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:46:44.436 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-06T09:46:44.436 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:rbd mirror flush : flush rbd mirror 2024-04-06T09:46:44.436 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:admin_socket: invalid command 2024-04-06T09:46:44.438 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:46:44.438 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/test 2024-04-06T09:46:44.438 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:1 2024-04-06T09:46:44.438 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:46:44.438 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:46:44.438 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-06T09:46:44.438 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:1 2024-04-06T09:46:44.438 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:46:44.439 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:46:44.439 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:46:44.439 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=1 2024-04-06T09:46:44.439 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-06T09:46:44.439 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:46:44.439 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:46:44.439 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=1 2024-04-06T09:46:44.439 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=1 2024-04-06T09:46:44.440 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-06T09:46:44.440 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:1 2024-04-06T09:46:44.440 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:46:44.441 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:46:44.441 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-06T09:46:44.441 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:1 2024-04-06T09:46:44.441 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:46:44.441 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:46:44.441 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:46:44.441 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=1 2024-04-06T09:46:44.441 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-06T09:46:44.441 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:46:44.441 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:46:44.441 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=1 2024-04-06T09:46:44.441 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=1 2024-04-06T09:46:44.441 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-06T09:46:44.458 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T09:46:44.459 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T09:46:44.459 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T09:46:44.459 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/test 2024-04-06T09:46:44.577 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:46:44.578 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:46:44.578 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:46:44.578 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:46:44.578 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:46:44.579 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:46:44.579 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:46:44.579 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:46:44.579 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:46:44.579 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:46:44.579 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:46:44.579 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:46:44.579 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:46:44.581 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:46:44.581 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:46:44.581 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:46:44.636 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 34 2024-04-06T09:46:44.636 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 34 2024-04-06T09:46:44.636 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:46:44.636 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=137, tag_tid=24, entry_tid=1], [object_number=136, tag_tid=24, entry_tid=0], [object_number=139, tag_tid=23, entry_tid=5615], [object_number=138, tag_tid=23, entry_tid=5614]]], state=connected] 2024-04-06T09:46:44.636 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=136, tag_tid=23, entry_tid=5616], [object_number=139, tag_tid=23, entry_tid=5615], [object_number=138, tag_tid=23, entry_tid=5614], [object_number=137, tag_tid=23, entry_tid=5613]]], state=connected] 2024-04-06T09:46:44.640 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:46:44.641 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=137, tag_tid=24, entry_tid=1' 2024-04-06T09:46:44.642 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:46:44.642 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:46:44.642 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:46:44.642 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:46:44.643 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:46:44.643 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:46:44.643 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:46:44.643 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:46:44.643 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:46:44.643 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:46:44.643 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:46:44.643 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:46:44.644 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:46:44.644 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:46:44.644 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:46:44.691 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 34 2024-04-06T09:46:44.691 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 34 2024-04-06T09:46:44.691 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:46:44.691 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=137, tag_tid=24, entry_tid=1], [object_number=136, tag_tid=24, entry_tid=0], [object_number=139, tag_tid=23, entry_tid=5615], [object_number=138, tag_tid=23, entry_tid=5614]]], state=connected] 2024-04-06T09:46:44.691 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=136, tag_tid=23, entry_tid=5616], [object_number=139, tag_tid=23, entry_tid=5615], [object_number=138, tag_tid=23, entry_tid=5614], [object_number=137, tag_tid=23, entry_tid=5613]]], state=connected] 2024-04-06T09:46:44.694 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:46:44.696 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=136, tag_tid=23, entry_tid=5616' 2024-04-06T09:46:44.696 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=137, tag_tid=24, entry_tid=1' -a 'object_number=137, tag_tid=24, entry_tid=1' = 'object_number=136, tag_tid=23, entry_tid=5616' 2024-04-06T09:46:44.696 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=136, tag_tid=23, entry_tid=5616' '!=' 'object_number=136, tag_tid=23, entry_tid=5616' 2024-04-06T09:46:44.696 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:46:44.696 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 1.6 2024-04-06T09:46:46.298 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:46:46.298 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:46:46.298 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:46:46.298 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:46:46.298 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:46:46.298 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:46:46.298 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:46:46.298 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:46:46.299 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:46:46.299 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:46:46.299 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:46:46.299 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:46:46.299 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:46:46.299 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:46:46.299 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:46:46.299 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:46:46.299 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:46:46.303 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:46:46.303 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:46:46.303 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:46:46.303 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:46:46.303 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:46:46.303 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:46:46.303 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:46:46.303 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:46:46.303 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:46:46.303 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:46:46.303 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:46:46.303 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:46:46.303 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:46:46.304 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:46:46.304 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:46:46.304 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:46:46.304 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:46:46.304 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:46:46.304 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:46:46.304 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:46:46.304 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:46:46.305 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:46:46.305 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:46:46.305 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:46:46.305 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:46:46.305 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:46:46.305 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:46:46.305 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:46:46.305 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:46:46.306 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:46:46.306 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:46:46.306 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:46:46.329 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:46:46.330 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:46:46.330 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:46:46.330 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:46:46.432 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-06T09:46:46.432 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:rbd mirror flush : flush rbd mirror 2024-04-06T09:46:46.432 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:admin_socket: invalid command 2024-04-06T09:46:46.435 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:46:46.435 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/test 2024-04-06T09:46:46.435 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:1 2024-04-06T09:46:46.435 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:46:46.435 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:46:46.435 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-06T09:46:46.435 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:1 2024-04-06T09:46:46.435 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:46:46.435 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:46:46.435 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:46:46.436 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=1 2024-04-06T09:46:46.436 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-06T09:46:46.436 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:46:46.436 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:46:46.436 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=1 2024-04-06T09:46:46.436 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=1 2024-04-06T09:46:46.436 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-06T09:46:46.436 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:1 2024-04-06T09:46:46.436 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:46:46.436 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:46:46.436 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-06T09:46:46.436 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:1 2024-04-06T09:46:46.436 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:46:46.436 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:46:46.436 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:46:46.437 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=1 2024-04-06T09:46:46.437 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-06T09:46:46.437 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:46:46.437 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:46:46.437 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=1 2024-04-06T09:46:46.437 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=1 2024-04-06T09:46:46.437 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-06T09:46:46.460 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T09:46:46.460 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T09:46:46.461 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T09:46:46.461 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/test 2024-04-06T09:46:46.582 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:46:46.583 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:46:46.583 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:46:46.583 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:46:46.583 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:46:46.583 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:46:46.583 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:46:46.583 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:46:46.583 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:46:46.583 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:46:46.584 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:46:46.584 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:46:46.584 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:46:46.587 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:46:46.587 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:46:46.587 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:46:46.646 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 34 2024-04-06T09:46:46.646 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 34 2024-04-06T09:46:46.647 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:46:46.647 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=137, tag_tid=24, entry_tid=1], [object_number=136, tag_tid=24, entry_tid=0], [object_number=139, tag_tid=23, entry_tid=5615], [object_number=138, tag_tid=23, entry_tid=5614]]], state=connected] 2024-04-06T09:46:46.647 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=136, tag_tid=23, entry_tid=5616], [object_number=139, tag_tid=23, entry_tid=5615], [object_number=138, tag_tid=23, entry_tid=5614], [object_number=137, tag_tid=23, entry_tid=5613]]], state=connected] 2024-04-06T09:46:46.650 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:46:46.652 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=137, tag_tid=24, entry_tid=1' 2024-04-06T09:46:46.653 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:46:46.653 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:46:46.653 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:46:46.653 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:46:46.653 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:46:46.654 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:46:46.654 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:46:46.654 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:46:46.654 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:46:46.654 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:46:46.654 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:46:46.654 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:46:46.655 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:46:46.655 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:46:46.655 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:46:46.704 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 34 2024-04-06T09:46:46.705 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 34 2024-04-06T09:46:46.705 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:46:46.705 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=137, tag_tid=24, entry_tid=1], [object_number=136, tag_tid=24, entry_tid=0], [object_number=139, tag_tid=23, entry_tid=5615], [object_number=138, tag_tid=23, entry_tid=5614]]], state=connected] 2024-04-06T09:46:46.705 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=136, tag_tid=23, entry_tid=5616], [object_number=139, tag_tid=23, entry_tid=5615], [object_number=138, tag_tid=23, entry_tid=5614], [object_number=137, tag_tid=23, entry_tid=5613]]], state=connected] 2024-04-06T09:46:46.708 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:46:46.710 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=136, tag_tid=23, entry_tid=5616' 2024-04-06T09:46:46.710 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=137, tag_tid=24, entry_tid=1' -a 'object_number=137, tag_tid=24, entry_tid=1' = 'object_number=136, tag_tid=23, entry_tid=5616' 2024-04-06T09:46:46.710 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=136, tag_tid=23, entry_tid=5616' '!=' 'object_number=136, tag_tid=23, entry_tid=5616' 2024-04-06T09:46:46.710 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:46:46.710 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 2 2024-04-06T09:46:48.711 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:46:48.712 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:46:48.712 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:46:48.712 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:46:48.712 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:46:48.712 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:46:48.712 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:46:48.712 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:46:48.712 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:46:48.712 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:46:48.712 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:46:48.712 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:46:48.712 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:46:48.712 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:46:48.712 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:46:48.713 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:46:48.713 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:46:48.715 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:46:48.715 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:46:48.715 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:46:48.715 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:46:48.715 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:46:48.716 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:46:48.716 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:46:48.716 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:46:48.716 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:46:48.716 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:46:48.716 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:46:48.716 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:46:48.716 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:46:48.716 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:46:48.718 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:46:48.718 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:46:48.718 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:46:48.718 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:46:48.718 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:46:48.718 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:46:48.718 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:46:48.718 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:46:48.719 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:46:48.719 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:46:48.719 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:46:48.719 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:46:48.719 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:46:48.719 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:46:48.719 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:46:48.719 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:46:48.719 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:46:48.719 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:46:48.743 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:46:48.743 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:46:48.743 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:46:48.743 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:46:48.848 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-06T09:46:48.848 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:rbd mirror flush : flush rbd mirror 2024-04-06T09:46:48.848 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:admin_socket: invalid command 2024-04-06T09:46:48.852 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:46:48.852 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/test 2024-04-06T09:46:48.852 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:1 2024-04-06T09:46:48.853 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:46:48.853 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:46:48.853 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-06T09:46:48.853 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:1 2024-04-06T09:46:48.853 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:46:48.853 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:46:48.853 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:46:48.853 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=1 2024-04-06T09:46:48.853 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-06T09:46:48.853 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:46:48.853 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:46:48.853 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=1 2024-04-06T09:46:48.853 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=1 2024-04-06T09:46:48.853 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-06T09:46:48.853 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:1 2024-04-06T09:46:48.854 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:46:48.854 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:46:48.854 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-06T09:46:48.854 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:1 2024-04-06T09:46:48.854 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:46:48.854 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:46:48.854 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:46:48.854 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=1 2024-04-06T09:46:48.854 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-06T09:46:48.854 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:46:48.854 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:46:48.854 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=1 2024-04-06T09:46:48.854 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=1 2024-04-06T09:46:48.854 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-06T09:46:48.877 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T09:46:48.877 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T09:46:48.877 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T09:46:48.877 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/test 2024-04-06T09:46:48.995 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:46:48.996 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:46:48.996 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:46:48.996 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:46:48.996 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:46:48.997 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:46:48.997 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:46:48.997 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:46:48.997 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:46:48.997 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:46:48.998 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:46:48.998 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:46:48.998 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:46:48.999 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:46:49.000 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:46:49.000 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:46:49.047 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 34 2024-04-06T09:46:49.047 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 34 2024-04-06T09:46:49.047 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:46:49.047 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=137, tag_tid=24, entry_tid=1], [object_number=136, tag_tid=24, entry_tid=0], [object_number=139, tag_tid=23, entry_tid=5615], [object_number=138, tag_tid=23, entry_tid=5614]]], state=connected] 2024-04-06T09:46:49.047 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=137, tag_tid=24, entry_tid=1], [object_number=136, tag_tid=24, entry_tid=0], [object_number=139, tag_tid=23, entry_tid=5615], [object_number=138, tag_tid=23, entry_tid=5614]]], state=connected] 2024-04-06T09:46:49.051 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:46:49.052 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=137, tag_tid=24, entry_tid=1' 2024-04-06T09:46:49.053 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:46:49.053 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:46:49.053 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:46:49.053 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:46:49.053 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:46:49.053 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:46:49.054 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:46:49.054 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:46:49.054 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:46:49.054 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:46:49.054 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:46:49.054 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:46:49.054 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:46:49.055 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:46:49.055 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:46:49.101 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 34 2024-04-06T09:46:49.101 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 34 2024-04-06T09:46:49.102 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:46:49.102 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=137, tag_tid=24, entry_tid=1], [object_number=136, tag_tid=24, entry_tid=0], [object_number=139, tag_tid=23, entry_tid=5615], [object_number=138, tag_tid=23, entry_tid=5614]]], state=connected] 2024-04-06T09:46:49.102 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=137, tag_tid=24, entry_tid=1], [object_number=136, tag_tid=24, entry_tid=0], [object_number=139, tag_tid=23, entry_tid=5615], [object_number=138, tag_tid=23, entry_tid=5614]]], state=connected] 2024-04-06T09:46:49.105 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:46:49.107 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=137, tag_tid=24, entry_tid=1' 2024-04-06T09:46:49.107 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=137, tag_tid=24, entry_tid=1' -a 'object_number=137, tag_tid=24, entry_tid=1' = 'object_number=137, tag_tid=24, entry_tid=1' 2024-04-06T09:46:49.107 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:46:49.107 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_snap_present cluster1 mirror test snap8 2024-04-06T09:46:49.107 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:46:49.107 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:46:49.107 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:46:49.107 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local snap_name=snap8 2024-04-06T09:46:49.107 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local s 2024-04-06T09:46:49.107 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-04-06T09:46:49.107 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 1 2024-04-06T09:46:50.108 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 info mirror/test@snap8 2024-04-06T09:46:50.173 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:rbd image 'test': 2024-04-06T09:46:50.173 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: size 512 MiB in 128 objects 2024-04-06T09:46:50.173 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: order 22 (4 MiB objects) 2024-04-06T09:46:50.173 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: snapshot_count: 15 2024-04-06T09:46:50.173 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: id: 10a56da0b207 2024-04-06T09:46:50.173 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: block_name_prefix: rbd_data.10a56da0b207 2024-04-06T09:46:50.173 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: format: 2 2024-04-06T09:46:50.173 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: features: layering, exclusive-lock, journaling 2024-04-06T09:46:50.174 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: op_features: 2024-04-06T09:46:50.174 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: flags: 2024-04-06T09:46:50.174 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: create_timestamp: Sat Apr 6 09:31:59 2024 2024-04-06T09:46:50.174 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: access_timestamp: Sat Apr 6 09:45:48 2024 2024-04-06T09:46:50.174 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: modify_timestamp: Sat Apr 6 09:46:15 2024 2024-04-06T09:46:50.174 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: protected: False 2024-04-06T09:46:50.174 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: journal: 10a56da0b207 2024-04-06T09:46:50.174 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: mirroring state: enabled 2024-04-06T09:46:50.174 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: mirroring mode: journal 2024-04-06T09:46:50.174 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: mirroring global id: 8ad5dedc-b3e0-425d-8993-b3e2231fc5ca 2024-04-06T09:46:50.174 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: mirroring primary: false 2024-04-06T09:46:50.179 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:46:50.179 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n snap7-clean ']' 2024-04-06T09:46:50.180 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ compare_image_snaps mirror test snap7-clean 2024-04-06T09:46:50.180 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:46:50.180 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:46:50.180 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local snap_name=snap7-clean 2024-04-06T09:46:50.180 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local ret=0 2024-04-06T09:46:50.180 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local rmt_export=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.export 2024-04-06T09:46:50.180 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loc_export=/tmp/tmp.BhW6xKmROU/cluster1-mirror-test.export 2024-04-06T09:46:50.180 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rm -f /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.export /tmp/tmp.BhW6xKmROU/cluster1-mirror-test.export 2024-04-06T09:46:50.180 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster2 -p mirror export test@snap7-clean /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.export 2024-04-06T09:46:50.781 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-04-06T09:46:50.797 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 -p mirror export test@snap7-clean /tmp/tmp.BhW6xKmROU/cluster1-mirror-test.export 2024-04-06T09:46:52.317 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-04-06T09:46:52.343 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmp /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.export /tmp/tmp.BhW6xKmROU/cluster1-mirror-test.export 2024-04-06T09:46:52.630 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rm -f /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.export /tmp/tmp.BhW6xKmROU/cluster1-mirror-test.export 2024-04-06T09:46:52.764 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:46:52.765 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ compare_image_snaps mirror test snap8 2024-04-06T09:46:52.765 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:46:52.765 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:46:52.765 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local snap_name=snap8 2024-04-06T09:46:52.765 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local ret=0 2024-04-06T09:46:52.765 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local rmt_export=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.export 2024-04-06T09:46:52.765 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loc_export=/tmp/tmp.BhW6xKmROU/cluster1-mirror-test.export 2024-04-06T09:46:52.765 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rm -f /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.export /tmp/tmp.BhW6xKmROU/cluster1-mirror-test.export 2024-04-06T09:46:52.766 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster2 -p mirror export test@snap8 /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.export 2024-04-06T09:46:53.402 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-04-06T09:46:53.419 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 -p mirror export test@snap8 /tmp/tmp.BhW6xKmROU/cluster1-mirror-test.export 2024-04-06T09:46:54.858 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-04-06T09:46:54.883 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmp /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.export /tmp/tmp.BhW6xKmROU/cluster1-mirror-test.export 2024-04-06T09:46:55.164 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rm -f /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.export /tmp/tmp.BhW6xKmROU/cluster1-mirror-test.export 2024-04-06T09:46:55.300 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:46:55.300 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ clean_snap_name=snap8-clean 2024-04-06T09:46:55.301 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ create_snap cluster2 mirror test snap8-clean 2024-04-06T09:46:55.301 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-04-06T09:46:55.301 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:46:55.301 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:46:55.301 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local snap_name=snap8-clean 2024-04-06T09:46:55.301 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap8-clean --debug-rbd=20 --debug-journaler=20 2024-04-06T09:46:55.638 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in `seq 1 10` 2024-04-06T09:46:55.638 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ stress_write_image cluster2 mirror test 2024-04-06T09:46:55.638 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-04-06T09:46:55.638 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:46:55.638 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:46:55.638 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-04-06T09:46:55.640 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local duration=7 2024-04-06T09:46:55.640 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set +e 2024-04-06T09:46:55.640 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ timeout 7s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror test --debug-rbd=20 --debug-journaler=20 2024-04-06T09:46:57.307 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:100 IOs, 22411264 bytes 2024-04-06T09:46:59.022 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:200 IOs, 53738496 bytes 2024-04-06T09:47:02.596 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:300 IOs, 142290944 bytes 2024-04-06T09:47:02.659 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ error_code=124 2024-04-06T09:47:02.659 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set -e 2024-04-06T09:47:02.660 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 124 -eq 124 ']' 2024-04-06T09:47:02.660 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:47:02.660 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+replaying 2024-04-06T09:47:02.660 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:47:02.660 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:47:02.660 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:47:02.661 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+replaying 2024-04-06T09:47:02.661 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local description_pattern= 2024-04-06T09:47:02.661 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local service_pattern= 2024-04-06T09:47:02.661 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:47:02.661 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 1 2024-04-06T09:47:03.660 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test_status_in_pool_dir cluster1 mirror test up+replaying '' '' 2024-04-06T09:47:03.660 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:47:03.660 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:47:03.660 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:47:03.660 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+replaying 2024-04-06T09:47:03.660 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local description_pattern= 2024-04-06T09:47:03.660 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local service_pattern= 2024-04-06T09:47:03.661 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local status 2024-04-06T09:47:03.661 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ CEPH_ARGS= 2024-04-06T09:47:03.661 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster1 mirror image status mirror/test 2024-04-06T09:47:03.753 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ status='test: 2024-04-06T09:47:03.753 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: global_id: 8ad5dedc-b3e0-425d-8993-b3e2231fc5ca 2024-04-06T09:47:03.753 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: state: up+replaying 2024-04-06T09:47:03.753 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: description: replaying, {"bytes_per_second":2380911.85,"entries_behind_primary":58,"entries_per_second":149.56,"non_primary_position":{"entry_tid":5559,"object_number":139,"tag_tid":23},"primary_position":{"entry_tid":1,"object_number":137,"tag_tid":24},"seconds_until_synced":0} 2024-04-06T09:47:03.754 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: service: mirror.1 on smithi192 2024-04-06T09:47:03.754 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: last_update: 2024-04-06 09:46:41 2024-04-06T09:47:03.754 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: peer_sites: 2024-04-06T09:47:03.754 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: name: cluster2 2024-04-06T09:47:03.754 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: state: up+stopped 2024-04-06T09:47:03.754 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: description: local image is primary 2024-04-06T09:47:03.754 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: last_update: 2024-04-06 09:46:44' 2024-04-06T09:47:03.754 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^ state: .*up+replaying' 2024-04-06T09:47:03.754 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: state: up+replaying 2024-04-06T09:47:03.755 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^ description: .*' 2024-04-06T09:47:03.757 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: description: replaying, {"bytes_per_second":2380911.85,"entries_behind_primary":58,"entries_per_second":149.56,"non_primary_position":{"entry_tid":5559,"object_number":139,"tag_tid":23},"primary_position":{"entry_tid":1,"object_number":137,"tag_tid":24},"seconds_until_synced":0} 2024-04-06T09:47:03.758 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n '' ']' 2024-04-06T09:47:03.758 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ echo up+replaying 2024-04-06T09:47:03.758 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^up+' 2024-04-06T09:47:03.758 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:up+replaying 2024-04-06T09:47:03.759 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep 'service: *mirror..* on ' 2024-04-06T09:47:03.759 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: service: mirror.1 on smithi192 2024-04-06T09:47:03.759 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local last_update 2024-04-06T09:47:03.759 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-04-06T09:47:03.761 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_update='2024-04-06 09:46:41' 2024-04-06T09:47:03.762 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+replaying '2024-04-06 09:46:41' 2024-04-06T09:47:03.762 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:47:03.762 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:47:03.762 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:47:03.762 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+replaying 2024-04-06T09:47:03.762 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'prev_last_update=2024-04-06 09:46:41' 2024-04-06T09:47:03.762 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local status 2024-04-06T09:47:03.762 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ CEPH_ARGS= 2024-04-06T09:47:03.762 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-04-06T09:47:03.854 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ status='OKOKOK142094268mirror.1smithi19218.2.2-1059-g4d05f4catrueOK43194322mirror.3smithi19218.2.2-1059-g4d05f4cafalseOK44424445mirror.2smithi19218.2.2-1059-g4d05f4cafalseOK44654468mirror.0smithi19218.2.2-1059-g4d05f4cafalseOKtest8ad5dedc-b3e0-425d-8993-b3e2231fc5caup+replayingreplaying, {"bytes_per_second":2380911.85,"entries_behind_primary":58,"entries_per_second":149.56,"non_primary_position":{"entry_tid":5559,"object_number":139,"tag_tid":23},"primary_position":{"entry_tid":1,"object_number":137,"tag_tid":24},"seconds_until_synced":0}42094268mirror.1smithi1922024-04-06 09:46:41cluster2c5e3bdc3-c286-4289-8fd0-6b8628b92376up+stoppedlocal image is primary2024-04-06 09:46:44' 2024-04-06T09:47:03.854 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local last_update state 2024-04-06T09:47:03.854 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2024-04-06T09:47:03.858 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_update='2024-04-06 09:46:41' 2024-04-06T09:47:03.859 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2024-04-06T09:47:03.862 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ state=up+replaying 2024-04-06T09:47:03.863 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ echo up+replaying 2024-04-06T09:47:03.863 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep up+replaying 2024-04-06T09:47:03.864 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:up+replaying 2024-04-06T09:47:03.864 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:47:03.864 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:47:03.864 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ snap_name=snap9 2024-04-06T09:47:03.864 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ create_snap cluster2 mirror test snap9 2024-04-06T09:47:03.864 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-04-06T09:47:03.864 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:47:03.865 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:47:03.865 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local snap_name=snap9 2024-04-06T09:47:03.865 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap9 --debug-rbd=20 --debug-journaler=20 2024-04-06T09:47:18.507 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.0 with skip (0.6418002669047136) > weight (0.25) 2024-04-06T09:47:18.507 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.1 with skip (0.45980946197059636) > weight (0.25) 2024-04-06T09:47:18.507 INFO:tasks.rbd_mirror_thrash:kill cluster1.client.mirror.2 2024-04-06T09:47:18.508 INFO:tasks.rbd_mirror.cluster1.client.mirror.2:Sent signal 15 2024-04-06T09:47:18.508 INFO:tasks.rbd_mirror_thrash:waiting for 2 secs before reviving daemons 2024-04-06T09:47:18.509 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi192.stderr:2024-04-06T09:47:18.512+0000 7ffa09d15640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.2 (PID: 103723) UID: 1000 2024-04-06T09:47:20.509 INFO:tasks.rbd_mirror_thrash:waiting for cluster1.client.mirror.2 2024-04-06T09:47:20.509 INFO:teuthology.orchestra.run:waiting for 600 2024-04-06T09:47:20.509 INFO:tasks.rbd_mirror_thrash:reviving cluster1.client.mirror.2 2024-04-06T09:47:20.510 INFO:tasks.rbd_mirror.cluster1.client.mirror.2:Restarting daemon 2024-04-06T09:47:20.510 DEBUG:teuthology.orchestra.run.smithi192:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.2 2024-04-06T09:47:20.513 INFO:tasks.rbd_mirror.cluster1.client.mirror.2:Started 2024-04-06T09:47:20.513 INFO:tasks.rbd_mirror_thrash:waiting for 93 secs before thrashing 2024-04-06T09:47:34.730 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_image_replay_started cluster1 mirror test 2024-04-06T09:47:34.730 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:47:34.730 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:47:34.730 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:47:34.730 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_image_replay_state cluster1 mirror test started 2024-04-06T09:47:34.730 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:47:34.730 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:47:34.730 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:47:34.730 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state=started 2024-04-06T09:47:34.730 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local s 2024-04-06T09:47:34.731 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:47:34.731 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 1 2024-04-06T09:47:35.731 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test_image_replay_state cluster1 mirror test started 2024-04-06T09:47:35.731 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:47:35.732 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:47:35.732 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:47:35.732 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local test_state=started 2024-04-06T09:47:35.732 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local status_result 2024-04-06T09:47:35.732 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local current_state=stopped 2024-04-06T09:47:35.733 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2024-04-06T09:47:35.733 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -i state 2024-04-06T09:47:35.733 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_instance=cluster1 2024-04-06T09:47:35.733 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ shift 2024-04-06T09:47:35.733 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:47:35.733 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance=cluster1 2024-04-06T09:47:35.733 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local loop_instance 2024-04-06T09:47:35.733 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:47:35.734 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sleep 0 2024-04-06T09:47:35.734 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:47:35.735 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ seq 0 3 2024-04-06T09:47:35.737 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:47:35.737 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2024-04-06T09:47:35.737 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1:0 2024-04-06T09:47:35.737 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ shift 2024-04-06T09:47:35.737 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:47:35.737 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:47:35.737 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:47:35.737 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=cluster 2024-04-06T09:47:35.737 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:47:35.738 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:47:35.738 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:47:35.738 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:47:35.738 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval cluster=cluster1 2024-04-06T09:47:35.738 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ cluster=cluster1 2024-04-06T09:47:35.738 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:47:35.738 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:47:35.738 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:47:35.740 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local local_cluster=cluster1:0 2024-04-06T09:47:35.740 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local cluster=cluster1 2024-04-06T09:47:35.740 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local instance 2024-04-06T09:47:35.740 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:47:35.740 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local val=cluster1:0 2024-04-06T09:47:35.740 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local cluster_var_name=local_cluster 2024-04-06T09:47:35.740 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local instance_var_name=instance 2024-04-06T09:47:35.740 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ cluster=cluster1 2024-04-06T09:47:35.740 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:47:35.740 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ '[' 0 = cluster1:0 ']' 2024-04-06T09:47:35.740 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ eval local_cluster=cluster1 2024-04-06T09:47:35.740 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++++ local_cluster=cluster1 2024-04-06T09:47:35.741 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ eval instance=0 2024-04-06T09:47:35.741 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++++ instance=0 2024-04-06T09:47:35.741 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:47:35.759 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:47:35.759 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:47:35.759 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:47:35.759 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2024-04-06T09:47:35.867 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-06T09:47:35.867 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:rbd mirror status : get status for rbd mirror 2024-04-06T09:47:35.867 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:admin_socket: invalid command 2024-04-06T09:47:35.871 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:47:35.871 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/test 2024-04-06T09:47:35.871 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1:1 2024-04-06T09:47:35.871 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ shift 2024-04-06T09:47:35.871 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:47:35.871 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-04-06T09:47:35.871 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:1 2024-04-06T09:47:35.871 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=cluster 2024-04-06T09:47:35.871 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:47:35.871 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:47:35.871 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=1 2024-04-06T09:47:35.871 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-06T09:47:35.871 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval cluster=cluster1 2024-04-06T09:47:35.871 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ cluster=cluster1 2024-04-06T09:47:35.872 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=1 2024-04-06T09:47:35.872 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=1 2024-04-06T09:47:35.872 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-04-06T09:47:35.872 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local local_cluster=cluster1:1 2024-04-06T09:47:35.872 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local cluster=cluster1 2024-04-06T09:47:35.872 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local instance 2024-04-06T09:47:35.872 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-06T09:47:35.872 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local val=cluster1:1 2024-04-06T09:47:35.872 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local cluster_var_name=local_cluster 2024-04-06T09:47:35.872 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local instance_var_name=instance 2024-04-06T09:47:35.872 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ cluster=cluster1 2024-04-06T09:47:35.872 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=1 2024-04-06T09:47:35.872 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ '[' 1 = cluster1:1 ']' 2024-04-06T09:47:35.872 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ eval local_cluster=cluster1 2024-04-06T09:47:35.872 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++++ local_cluster=cluster1 2024-04-06T09:47:35.873 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ eval instance=1 2024-04-06T09:47:35.873 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++++ instance=1 2024-04-06T09:47:35.873 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-06T09:47:35.890 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T09:47:35.890 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T09:47:35.890 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T09:47:35.890 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/test 2024-04-06T09:47:36.008 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ return 0 2024-04-06T09:47:36.008 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ status_result=' "state": "Replaying"' 2024-04-06T09:47:36.009 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep -i Replaying 2024-04-06T09:47:36.009 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ echo ' "state": "Replaying"' 2024-04-06T09:47:36.010 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: "state": "Replaying" 2024-04-06T09:47:36.012 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ current_state=started 2024-04-06T09:47:36.012 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test started = started 2024-04-06T09:47:36.012 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:47:36.012 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror test 2024-04-06T09:47:36.012 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local local_cluster=cluster1 2024-04-06T09:47:36.012 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-04-06T09:47:36.012 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:47:36.012 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:47:36.012 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' journal = journal ']' 2024-04-06T09:47:36.012 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_journal_replay_complete cluster1 cluster2 mirror test 2024-04-06T09:47:36.012 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local local_cluster=cluster1 2024-04-06T09:47:36.013 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-04-06T09:47:36.013 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:47:36.013 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:47:36.013 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local s master_pos mirror_pos last_mirror_pos 2024-04-06T09:47:36.013 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local master_tag master_entry mirror_tag mirror_entry 2024-04-06T09:47:36.013 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:47:36.013 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:47:36.013 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:47:36.214 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:47:36.214 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:47:36.214 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:47:36.214 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:47:36.214 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:47:36.214 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:47:36.214 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:47:36.214 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:47:36.214 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:47:36.214 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:47:36.215 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:47:36.215 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:47:36.215 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:47:36.215 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:47:36.215 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:47:36.215 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:47:36.215 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:47:36.217 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:47:36.217 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:47:36.217 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:47:36.217 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:47:36.217 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:47:36.218 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:47:36.218 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:47:36.218 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:47:36.218 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:47:36.218 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:47:36.218 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:47:36.218 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:47:36.218 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:47:36.218 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:47:36.218 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:47:36.218 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:47:36.219 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:47:36.219 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:47:36.220 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:47:36.220 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:47:36.220 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:47:36.220 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:47:36.220 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:47:36.220 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:47:36.220 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:47:36.220 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:47:36.220 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:47:36.220 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:47:36.220 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:47:36.220 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:47:36.220 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:47:36.220 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:47:36.243 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:47:36.244 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:47:36.244 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:47:36.244 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:47:36.350 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-06T09:47:36.351 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:rbd mirror flush : flush rbd mirror 2024-04-06T09:47:36.351 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:admin_socket: invalid command 2024-04-06T09:47:36.355 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:47:36.355 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/test 2024-04-06T09:47:36.355 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:1 2024-04-06T09:47:36.355 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:47:36.355 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:47:36.355 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-06T09:47:36.355 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:1 2024-04-06T09:47:36.355 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:47:36.355 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:47:36.355 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:47:36.356 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=1 2024-04-06T09:47:36.356 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-06T09:47:36.356 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:47:36.356 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:47:36.356 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=1 2024-04-06T09:47:36.356 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=1 2024-04-06T09:47:36.356 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-06T09:47:36.356 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:1 2024-04-06T09:47:36.356 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:47:36.356 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:47:36.356 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-06T09:47:36.356 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:1 2024-04-06T09:47:36.356 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:47:36.356 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:47:36.356 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:47:36.357 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=1 2024-04-06T09:47:36.357 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-06T09:47:36.357 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:47:36.357 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:47:36.357 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=1 2024-04-06T09:47:36.357 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=1 2024-04-06T09:47:36.357 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-06T09:47:36.374 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T09:47:36.374 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T09:47:36.374 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T09:47:36.374 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/test 2024-04-06T09:47:36.508 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:47:36.509 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:47:36.510 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:47:36.510 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:47:36.510 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:47:36.510 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:47:36.510 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:47:36.510 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:47:36.510 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:47:36.510 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:47:36.511 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:47:36.511 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:47:36.511 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:47:36.512 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:47:36.512 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:47:36.512 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:47:36.560 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 35 2024-04-06T09:47:36.560 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 36 2024-04-06T09:47:36.560 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:47:36.560 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=145, tag_tid=27, entry_tid=1], [object_number=144, tag_tid=27, entry_tid=0], [object_number=147, tag_tid=26, entry_tid=8935], [object_number=146, tag_tid=26, entry_tid=8934]]], state=connected] 2024-04-06T09:47:36.560 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=141, tag_tid=26, entry_tid=4869], [object_number=140, tag_tid=26, entry_tid=4868], [object_number=143, tag_tid=26, entry_tid=4867], [object_number=142, tag_tid=26, entry_tid=4866]]], state=connected] 2024-04-06T09:47:36.564 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:47:36.565 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=145, tag_tid=27, entry_tid=1' 2024-04-06T09:47:36.566 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:47:36.566 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:47:36.567 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:47:36.567 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:47:36.567 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:47:36.567 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:47:36.567 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:47:36.567 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:47:36.567 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:47:36.567 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:47:36.567 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:47:36.567 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:47:36.568 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:47:36.568 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:47:36.568 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:47:36.615 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 35 2024-04-06T09:47:36.615 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 36 2024-04-06T09:47:36.615 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:47:36.615 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=145, tag_tid=27, entry_tid=1], [object_number=144, tag_tid=27, entry_tid=0], [object_number=147, tag_tid=26, entry_tid=8935], [object_number=146, tag_tid=26, entry_tid=8934]]], state=connected] 2024-04-06T09:47:36.615 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=141, tag_tid=26, entry_tid=4869], [object_number=140, tag_tid=26, entry_tid=4868], [object_number=143, tag_tid=26, entry_tid=4867], [object_number=142, tag_tid=26, entry_tid=4866]]], state=connected] 2024-04-06T09:47:36.618 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:47:36.620 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=141, tag_tid=26, entry_tid=4869' 2024-04-06T09:47:36.620 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=145, tag_tid=27, entry_tid=1' -a 'object_number=145, tag_tid=27, entry_tid=1' = 'object_number=141, tag_tid=26, entry_tid=4869' 2024-04-06T09:47:36.620 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=141, tag_tid=26, entry_tid=4869' '!=' '' 2024-04-06T09:47:36.620 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:47:36.620 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=141, tag_tid=26, entry_tid=4869' = '' 2024-04-06T09:47:36.620 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=141, tag_tid=26, entry_tid=4869' 2024-04-06T09:47:36.621 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=145, tag_tid=27, entry_tid=1' 2024-04-06T09:47:36.621 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:47:36.621 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:47:36.621 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=27 2024-04-06T09:47:36.622 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=141, tag_tid=26, entry_tid=4869' 2024-04-06T09:47:36.622 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:47:36.622 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:47:36.623 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=26 2024-04-06T09:47:36.623 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=145, tag_tid=27, entry_tid=1' 2024-04-06T09:47:36.624 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:47:36.624 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:47:36.624 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:47:36.625 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=141, tag_tid=26, entry_tid=4869' 2024-04-06T09:47:36.625 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:47:36.625 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:47:36.626 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=4869 2024-04-06T09:47:36.627 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 27 = 26 -a 1 -le 4869 2024-04-06T09:47:36.627 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:47:36.627 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:47:36.627 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:47:36.829 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:47:36.829 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:47:36.829 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:47:36.829 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:47:36.829 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:47:36.829 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:47:36.829 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:47:36.829 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:47:36.829 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:47:36.829 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:47:36.830 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:47:36.830 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:47:36.830 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:47:36.830 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:47:36.830 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:47:36.830 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:47:36.830 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:47:36.832 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:47:36.832 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:47:36.832 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:47:36.832 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:47:36.832 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:47:36.832 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:47:36.833 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:47:36.833 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:47:36.833 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:47:36.833 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:47:36.833 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:47:36.833 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:47:36.833 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:47:36.833 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:47:36.833 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:47:36.833 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:47:36.834 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:47:36.834 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:47:36.835 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:47:36.835 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:47:36.835 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:47:36.835 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:47:36.835 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:47:36.835 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:47:36.835 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:47:36.835 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:47:36.835 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:47:36.835 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:47:36.836 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:47:36.836 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:47:36.836 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:47:36.836 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:47:36.859 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:47:36.860 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:47:36.860 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:47:36.860 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:47:36.974 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-06T09:47:36.974 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:rbd mirror flush : flush rbd mirror 2024-04-06T09:47:36.974 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:admin_socket: invalid command 2024-04-06T09:47:36.976 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:47:36.976 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/test 2024-04-06T09:47:36.976 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:1 2024-04-06T09:47:36.976 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:47:36.976 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:47:36.976 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-06T09:47:36.976 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:1 2024-04-06T09:47:36.976 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:47:36.976 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:47:36.977 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:47:36.977 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=1 2024-04-06T09:47:36.977 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-06T09:47:36.977 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:47:36.977 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:47:36.977 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=1 2024-04-06T09:47:36.977 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=1 2024-04-06T09:47:36.978 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-06T09:47:36.978 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:1 2024-04-06T09:47:36.978 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:47:36.978 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:47:36.978 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-06T09:47:36.979 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:1 2024-04-06T09:47:36.979 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:47:36.979 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:47:36.979 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:47:36.979 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=1 2024-04-06T09:47:36.979 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-06T09:47:36.979 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:47:36.979 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:47:36.979 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=1 2024-04-06T09:47:36.979 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=1 2024-04-06T09:47:36.979 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-06T09:47:37.003 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T09:47:37.004 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T09:47:37.004 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T09:47:37.004 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/test 2024-04-06T09:47:37.144 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:47:37.146 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:47:37.146 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:47:37.146 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:47:37.146 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:47:37.146 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:47:37.146 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:47:37.146 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:47:37.146 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:47:37.146 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:47:37.146 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:47:37.147 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:47:37.147 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:47:37.148 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:47:37.148 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:47:37.148 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:47:37.204 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 35 2024-04-06T09:47:37.205 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 36 2024-04-06T09:47:37.205 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:47:37.205 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=145, tag_tid=27, entry_tid=1], [object_number=144, tag_tid=27, entry_tid=0], [object_number=147, tag_tid=26, entry_tid=8935], [object_number=146, tag_tid=26, entry_tid=8934]]], state=connected] 2024-04-06T09:47:37.205 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=143, tag_tid=26, entry_tid=4975], [object_number=142, tag_tid=26, entry_tid=4974], [object_number=141, tag_tid=26, entry_tid=4973], [object_number=140, tag_tid=26, entry_tid=4972]]], state=connected] 2024-04-06T09:47:37.208 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:47:37.209 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=145, tag_tid=27, entry_tid=1' 2024-04-06T09:47:37.211 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:47:37.211 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:47:37.211 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:47:37.211 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:47:37.211 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:47:37.211 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:47:37.211 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:47:37.211 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:47:37.211 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:47:37.211 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:47:37.211 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:47:37.211 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:47:37.212 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:47:37.212 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:47:37.212 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:47:37.259 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 35 2024-04-06T09:47:37.259 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 36 2024-04-06T09:47:37.259 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:47:37.259 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=145, tag_tid=27, entry_tid=1], [object_number=144, tag_tid=27, entry_tid=0], [object_number=147, tag_tid=26, entry_tid=8935], [object_number=146, tag_tid=26, entry_tid=8934]]], state=connected] 2024-04-06T09:47:37.259 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=143, tag_tid=26, entry_tid=4975], [object_number=142, tag_tid=26, entry_tid=4974], [object_number=141, tag_tid=26, entry_tid=4973], [object_number=140, tag_tid=26, entry_tid=4972]]], state=connected] 2024-04-06T09:47:37.263 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:47:37.265 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=143, tag_tid=26, entry_tid=4975' 2024-04-06T09:47:37.265 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=145, tag_tid=27, entry_tid=1' -a 'object_number=145, tag_tid=27, entry_tid=1' = 'object_number=143, tag_tid=26, entry_tid=4975' 2024-04-06T09:47:37.265 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=143, tag_tid=26, entry_tid=4975' '!=' 'object_number=141, tag_tid=26, entry_tid=4869' 2024-04-06T09:47:37.265 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:47:37.265 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=143, tag_tid=26, entry_tid=4975' = 'object_number=141, tag_tid=26, entry_tid=4869' 2024-04-06T09:47:37.265 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=143, tag_tid=26, entry_tid=4975' 2024-04-06T09:47:37.265 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=145, tag_tid=27, entry_tid=1' 2024-04-06T09:47:37.265 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:47:37.265 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:47:37.266 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=27 2024-04-06T09:47:37.266 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=143, tag_tid=26, entry_tid=4975' 2024-04-06T09:47:37.267 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:47:37.267 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:47:37.267 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=26 2024-04-06T09:47:37.268 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=145, tag_tid=27, entry_tid=1' 2024-04-06T09:47:37.269 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:47:37.269 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:47:37.269 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:47:37.270 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=143, tag_tid=26, entry_tid=4975' 2024-04-06T09:47:37.270 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:47:37.270 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:47:37.271 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=4975 2024-04-06T09:47:37.271 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 27 = 26 -a 1 -le 4975 2024-04-06T09:47:37.271 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:47:37.271 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:47:37.271 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:47:37.473 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:47:37.473 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:47:37.474 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:47:37.474 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:47:37.474 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:47:37.474 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:47:37.474 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:47:37.474 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:47:37.474 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:47:37.474 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:47:37.474 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:47:37.474 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:47:37.474 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:47:37.475 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:47:37.475 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:47:37.475 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:47:37.475 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:47:37.477 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:47:37.477 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:47:37.477 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:47:37.477 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:47:37.477 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:47:37.478 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:47:37.478 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:47:37.478 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:47:37.478 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:47:37.478 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:47:37.478 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:47:37.478 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:47:37.478 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:47:37.478 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:47:37.478 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:47:37.479 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:47:37.480 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:47:37.480 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:47:37.480 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:47:37.480 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:47:37.481 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:47:37.481 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:47:37.481 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:47:37.481 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:47:37.481 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:47:37.481 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:47:37.481 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:47:37.481 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:47:37.482 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:47:37.482 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:47:37.482 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:47:37.482 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:47:37.506 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:47:37.506 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:47:37.506 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:47:37.506 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:47:37.613 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-06T09:47:37.613 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:rbd mirror flush : flush rbd mirror 2024-04-06T09:47:37.613 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:admin_socket: invalid command 2024-04-06T09:47:37.615 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:47:37.616 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/test 2024-04-06T09:47:37.616 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:1 2024-04-06T09:47:37.616 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:47:37.616 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:47:37.616 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-06T09:47:37.616 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:1 2024-04-06T09:47:37.616 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:47:37.616 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:47:37.616 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:47:37.616 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=1 2024-04-06T09:47:37.616 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-06T09:47:37.616 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:47:37.616 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:47:37.616 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=1 2024-04-06T09:47:37.617 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=1 2024-04-06T09:47:37.618 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-06T09:47:37.618 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:1 2024-04-06T09:47:37.618 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:47:37.619 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:47:37.619 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-06T09:47:37.619 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:1 2024-04-06T09:47:37.619 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:47:37.619 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:47:37.619 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:47:37.619 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=1 2024-04-06T09:47:37.619 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-06T09:47:37.619 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:47:37.619 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:47:37.619 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=1 2024-04-06T09:47:37.619 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=1 2024-04-06T09:47:37.619 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-06T09:47:37.636 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T09:47:37.636 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T09:47:37.637 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T09:47:37.637 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/test 2024-04-06T09:47:37.757 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:47:37.758 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:47:37.758 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:47:37.758 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:47:37.759 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:47:37.759 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:47:37.759 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:47:37.759 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:47:37.759 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:47:37.759 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:47:37.759 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:47:37.759 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:47:37.759 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:47:37.761 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:47:37.761 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:47:37.762 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:47:37.814 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 35 2024-04-06T09:47:37.814 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 36 2024-04-06T09:47:37.814 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:47:37.814 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=145, tag_tid=27, entry_tid=1], [object_number=144, tag_tid=27, entry_tid=0], [object_number=147, tag_tid=26, entry_tid=8935], [object_number=146, tag_tid=26, entry_tid=8934]]], state=connected] 2024-04-06T09:47:37.814 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=141, tag_tid=26, entry_tid=5069], [object_number=140, tag_tid=26, entry_tid=5068], [object_number=143, tag_tid=26, entry_tid=5067], [object_number=142, tag_tid=26, entry_tid=5066]]], state=connected] 2024-04-06T09:47:37.818 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:47:37.819 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=145, tag_tid=27, entry_tid=1' 2024-04-06T09:47:37.820 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:47:37.820 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:47:37.821 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:47:37.821 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:47:37.821 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:47:37.821 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:47:37.821 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:47:37.821 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:47:37.821 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:47:37.821 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:47:37.821 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:47:37.821 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:47:37.822 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:47:37.822 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:47:37.822 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:47:37.869 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 35 2024-04-06T09:47:37.869 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 36 2024-04-06T09:47:37.870 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:47:37.870 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=145, tag_tid=27, entry_tid=1], [object_number=144, tag_tid=27, entry_tid=0], [object_number=147, tag_tid=26, entry_tid=8935], [object_number=146, tag_tid=26, entry_tid=8934]]], state=connected] 2024-04-06T09:47:37.870 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=141, tag_tid=26, entry_tid=5069], [object_number=140, tag_tid=26, entry_tid=5068], [object_number=143, tag_tid=26, entry_tid=5067], [object_number=142, tag_tid=26, entry_tid=5066]]], state=connected] 2024-04-06T09:47:37.873 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:47:37.875 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=141, tag_tid=26, entry_tid=5069' 2024-04-06T09:47:37.875 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=145, tag_tid=27, entry_tid=1' -a 'object_number=145, tag_tid=27, entry_tid=1' = 'object_number=141, tag_tid=26, entry_tid=5069' 2024-04-06T09:47:37.875 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=141, tag_tid=26, entry_tid=5069' '!=' 'object_number=143, tag_tid=26, entry_tid=4975' 2024-04-06T09:47:37.875 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:47:37.875 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=141, tag_tid=26, entry_tid=5069' = 'object_number=143, tag_tid=26, entry_tid=4975' 2024-04-06T09:47:37.875 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=141, tag_tid=26, entry_tid=5069' 2024-04-06T09:47:37.875 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=145, tag_tid=27, entry_tid=1' 2024-04-06T09:47:37.875 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:47:37.875 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:47:37.876 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=27 2024-04-06T09:47:37.876 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=141, tag_tid=26, entry_tid=5069' 2024-04-06T09:47:37.877 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:47:37.877 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:47:37.877 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=26 2024-04-06T09:47:37.878 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=145, tag_tid=27, entry_tid=1' 2024-04-06T09:47:37.878 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:47:37.878 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:47:37.879 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:47:37.880 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=141, tag_tid=26, entry_tid=5069' 2024-04-06T09:47:37.880 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:47:37.880 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:47:37.881 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=5069 2024-04-06T09:47:37.881 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 27 = 26 -a 1 -le 5069 2024-04-06T09:47:37.881 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:47:37.881 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:47:37.881 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:47:38.083 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:47:38.083 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:47:38.083 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:47:38.083 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:47:38.083 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:47:38.083 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:47:38.084 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:47:38.084 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:47:38.084 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:47:38.084 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:47:38.084 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:47:38.084 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:47:38.084 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:47:38.084 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:47:38.084 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:47:38.085 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:47:38.085 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:47:38.087 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:47:38.087 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:47:38.087 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:47:38.087 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:47:38.087 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:47:38.087 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:47:38.087 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:47:38.087 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:47:38.087 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:47:38.088 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:47:38.088 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:47:38.088 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:47:38.088 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:47:38.088 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:47:38.088 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:47:38.088 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:47:38.088 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:47:38.088 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:47:38.088 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:47:38.088 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:47:38.088 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:47:38.088 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:47:38.088 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:47:38.088 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:47:38.088 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:47:38.088 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:47:38.089 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:47:38.089 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:47:38.089 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:47:38.089 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:47:38.089 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:47:38.089 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:47:38.113 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:47:38.114 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:47:38.114 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:47:38.114 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:47:38.217 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-06T09:47:38.217 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:rbd mirror flush : flush rbd mirror 2024-04-06T09:47:38.217 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:admin_socket: invalid command 2024-04-06T09:47:38.219 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:47:38.220 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/test 2024-04-06T09:47:38.220 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:1 2024-04-06T09:47:38.220 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:47:38.220 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:47:38.220 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-06T09:47:38.220 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:1 2024-04-06T09:47:38.220 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:47:38.220 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:47:38.220 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:47:38.220 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=1 2024-04-06T09:47:38.220 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-06T09:47:38.220 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:47:38.220 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:47:38.221 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=1 2024-04-06T09:47:38.221 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=1 2024-04-06T09:47:38.222 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-06T09:47:38.222 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:1 2024-04-06T09:47:38.222 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:47:38.222 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:47:38.222 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-06T09:47:38.222 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:1 2024-04-06T09:47:38.222 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:47:38.223 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:47:38.223 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:47:38.223 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=1 2024-04-06T09:47:38.223 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-06T09:47:38.223 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:47:38.223 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:47:38.223 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=1 2024-04-06T09:47:38.223 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=1 2024-04-06T09:47:38.223 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-06T09:47:38.245 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T09:47:38.245 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T09:47:38.246 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T09:47:38.246 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/test 2024-04-06T09:47:38.366 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:47:38.367 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:47:38.367 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:47:38.367 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:47:38.368 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:47:38.368 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:47:38.368 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:47:38.368 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:47:38.368 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:47:38.368 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:47:38.368 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:47:38.369 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:47:38.369 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:47:38.370 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:47:38.371 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:47:38.371 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:47:38.428 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 35 2024-04-06T09:47:38.428 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 36 2024-04-06T09:47:38.428 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:47:38.428 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=145, tag_tid=27, entry_tid=1], [object_number=144, tag_tid=27, entry_tid=0], [object_number=147, tag_tid=26, entry_tid=8935], [object_number=146, tag_tid=26, entry_tid=8934]]], state=connected] 2024-04-06T09:47:38.428 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=142, tag_tid=26, entry_tid=5162], [object_number=141, tag_tid=26, entry_tid=5161], [object_number=140, tag_tid=26, entry_tid=5160], [object_number=143, tag_tid=26, entry_tid=5159]]], state=connected] 2024-04-06T09:47:38.431 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:47:38.432 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=145, tag_tid=27, entry_tid=1' 2024-04-06T09:47:38.434 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:47:38.434 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:47:38.434 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:47:38.434 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:47:38.434 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:47:38.434 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:47:38.434 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:47:38.434 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:47:38.434 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:47:38.434 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:47:38.434 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:47:38.434 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:47:38.435 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:47:38.435 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:47:38.435 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:47:38.482 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 35 2024-04-06T09:47:38.482 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 36 2024-04-06T09:47:38.482 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:47:38.483 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=145, tag_tid=27, entry_tid=1], [object_number=144, tag_tid=27, entry_tid=0], [object_number=147, tag_tid=26, entry_tid=8935], [object_number=146, tag_tid=26, entry_tid=8934]]], state=connected] 2024-04-06T09:47:38.483 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=142, tag_tid=26, entry_tid=5162], [object_number=141, tag_tid=26, entry_tid=5161], [object_number=140, tag_tid=26, entry_tid=5160], [object_number=143, tag_tid=26, entry_tid=5159]]], state=connected] 2024-04-06T09:47:38.486 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:47:38.488 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=142, tag_tid=26, entry_tid=5162' 2024-04-06T09:47:38.488 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=145, tag_tid=27, entry_tid=1' -a 'object_number=145, tag_tid=27, entry_tid=1' = 'object_number=142, tag_tid=26, entry_tid=5162' 2024-04-06T09:47:38.488 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=142, tag_tid=26, entry_tid=5162' '!=' 'object_number=141, tag_tid=26, entry_tid=5069' 2024-04-06T09:47:38.488 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:47:38.488 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=142, tag_tid=26, entry_tid=5162' = 'object_number=141, tag_tid=26, entry_tid=5069' 2024-04-06T09:47:38.488 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=142, tag_tid=26, entry_tid=5162' 2024-04-06T09:47:38.489 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=145, tag_tid=27, entry_tid=1' 2024-04-06T09:47:38.489 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:47:38.489 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:47:38.489 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=27 2024-04-06T09:47:38.490 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=142, tag_tid=26, entry_tid=5162' 2024-04-06T09:47:38.490 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:47:38.490 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:47:38.491 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=26 2024-04-06T09:47:38.492 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=145, tag_tid=27, entry_tid=1' 2024-04-06T09:47:38.492 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:47:38.492 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:47:38.493 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:47:38.494 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=142, tag_tid=26, entry_tid=5162' 2024-04-06T09:47:38.494 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:47:38.494 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:47:38.495 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=5162 2024-04-06T09:47:38.495 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 27 = 26 -a 1 -le 5162 2024-04-06T09:47:38.495 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:47:38.495 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:47:38.495 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:47:38.697 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:47:38.697 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:47:38.697 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:47:38.697 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:47:38.697 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:47:38.697 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:47:38.698 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:47:38.698 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:47:38.698 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:47:38.698 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:47:38.698 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:47:38.698 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:47:38.698 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:47:38.698 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:47:38.698 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:47:38.699 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:47:38.699 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:47:38.701 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:47:38.702 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:47:38.702 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:47:38.702 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:47:38.702 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:47:38.702 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:47:38.702 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:47:38.702 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:47:38.702 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:47:38.702 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:47:38.702 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:47:38.702 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:47:38.703 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:47:38.703 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:47:38.703 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:47:38.703 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:47:38.704 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:47:38.704 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:47:38.704 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:47:38.704 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:47:38.704 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:47:38.704 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:47:38.704 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:47:38.704 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:47:38.704 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:47:38.704 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:47:38.704 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:47:38.704 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:47:38.704 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:47:38.704 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:47:38.705 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:47:38.705 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:47:38.727 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:47:38.728 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:47:38.728 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:47:38.728 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:47:38.839 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-06T09:47:38.839 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:rbd mirror flush : flush rbd mirror 2024-04-06T09:47:38.839 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:admin_socket: invalid command 2024-04-06T09:47:38.841 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:47:38.841 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/test 2024-04-06T09:47:38.842 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:1 2024-04-06T09:47:38.842 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:47:38.842 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:47:38.842 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-06T09:47:38.842 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:1 2024-04-06T09:47:38.842 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:47:38.842 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:47:38.842 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:47:38.842 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=1 2024-04-06T09:47:38.842 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-06T09:47:38.842 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:47:38.842 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:47:38.842 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=1 2024-04-06T09:47:38.843 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=1 2024-04-06T09:47:38.844 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-06T09:47:38.844 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:1 2024-04-06T09:47:38.844 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:47:38.844 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:47:38.844 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-06T09:47:38.844 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:1 2024-04-06T09:47:38.844 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:47:38.844 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:47:38.844 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:47:38.844 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=1 2024-04-06T09:47:38.844 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-06T09:47:38.844 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:47:38.845 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:47:38.845 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=1 2024-04-06T09:47:38.845 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=1 2024-04-06T09:47:38.845 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-06T09:47:38.863 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T09:47:38.864 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T09:47:38.864 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T09:47:38.864 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/test 2024-04-06T09:47:38.995 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:47:38.997 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:47:38.997 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:47:38.997 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:47:38.997 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:47:38.997 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:47:38.997 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:47:38.997 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:47:38.997 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:47:38.997 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:47:38.997 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:47:38.998 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:47:38.998 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:47:38.999 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:47:38.999 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:47:38.999 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:47:39.051 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 35 2024-04-06T09:47:39.051 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 36 2024-04-06T09:47:39.051 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:47:39.051 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=145, tag_tid=27, entry_tid=1], [object_number=144, tag_tid=27, entry_tid=0], [object_number=147, tag_tid=26, entry_tid=8935], [object_number=146, tag_tid=26, entry_tid=8934]]], state=connected] 2024-04-06T09:47:39.051 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=140, tag_tid=26, entry_tid=5260], [object_number=143, tag_tid=26, entry_tid=5259], [object_number=142, tag_tid=26, entry_tid=5258], [object_number=141, tag_tid=26, entry_tid=5257]]], state=connected] 2024-04-06T09:47:39.054 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:47:39.056 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=145, tag_tid=27, entry_tid=1' 2024-04-06T09:47:39.057 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:47:39.057 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:47:39.057 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:47:39.057 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:47:39.057 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:47:39.057 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:47:39.057 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:47:39.057 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:47:39.058 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:47:39.058 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:47:39.058 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:47:39.058 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:47:39.058 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:47:39.059 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:47:39.059 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:47:39.312 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 35 2024-04-06T09:47:39.312 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 36 2024-04-06T09:47:39.312 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:47:39.312 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=145, tag_tid=27, entry_tid=1], [object_number=144, tag_tid=27, entry_tid=0], [object_number=147, tag_tid=26, entry_tid=8935], [object_number=146, tag_tid=26, entry_tid=8934]]], state=connected] 2024-04-06T09:47:39.312 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=140, tag_tid=26, entry_tid=5260], [object_number=143, tag_tid=26, entry_tid=5259], [object_number=142, tag_tid=26, entry_tid=5258], [object_number=141, tag_tid=26, entry_tid=5257]]], state=connected] 2024-04-06T09:47:39.317 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:47:39.320 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=140, tag_tid=26, entry_tid=5260' 2024-04-06T09:47:39.320 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=145, tag_tid=27, entry_tid=1' -a 'object_number=145, tag_tid=27, entry_tid=1' = 'object_number=140, tag_tid=26, entry_tid=5260' 2024-04-06T09:47:39.320 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=140, tag_tid=26, entry_tid=5260' '!=' 'object_number=142, tag_tid=26, entry_tid=5162' 2024-04-06T09:47:39.320 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:47:39.320 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=140, tag_tid=26, entry_tid=5260' = 'object_number=142, tag_tid=26, entry_tid=5162' 2024-04-06T09:47:39.320 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=140, tag_tid=26, entry_tid=5260' 2024-04-06T09:47:39.321 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=145, tag_tid=27, entry_tid=1' 2024-04-06T09:47:39.321 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:47:39.321 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:47:39.322 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=27 2024-04-06T09:47:39.323 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=140, tag_tid=26, entry_tid=5260' 2024-04-06T09:47:39.324 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:47:39.324 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:47:39.325 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=26 2024-04-06T09:47:39.326 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=145, tag_tid=27, entry_tid=1' 2024-04-06T09:47:39.326 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:47:39.326 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:47:39.327 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:47:39.328 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=140, tag_tid=26, entry_tid=5260' 2024-04-06T09:47:39.328 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:47:39.329 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:47:39.330 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=5260 2024-04-06T09:47:39.330 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 27 = 26 -a 1 -le 5260 2024-04-06T09:47:39.330 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:47:39.331 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:47:39.331 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:47:39.532 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:47:39.532 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:47:39.532 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:47:39.533 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:47:39.533 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:47:39.533 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:47:39.533 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:47:39.533 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:47:39.533 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:47:39.533 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:47:39.533 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:47:39.533 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:47:39.533 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:47:39.533 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:47:39.533 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:47:39.534 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:47:39.534 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:47:39.536 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:47:39.537 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:47:39.537 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:47:39.537 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:47:39.537 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:47:39.537 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:47:39.537 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:47:39.537 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:47:39.537 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:47:39.537 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:47:39.537 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:47:39.537 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:47:39.537 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:47:39.537 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:47:39.537 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:47:39.538 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:47:39.538 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:47:39.538 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:47:39.538 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:47:39.538 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:47:39.538 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:47:39.538 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:47:39.538 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:47:39.538 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:47:39.538 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:47:39.538 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:47:39.538 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:47:39.538 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:47:39.538 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:47:39.538 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:47:39.539 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:47:39.539 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:47:39.563 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:47:39.563 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:47:39.563 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:47:39.563 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:47:39.671 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-06T09:47:39.671 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:rbd mirror flush : flush rbd mirror 2024-04-06T09:47:39.671 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:admin_socket: invalid command 2024-04-06T09:47:39.675 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:47:39.675 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/test 2024-04-06T09:47:39.675 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:1 2024-04-06T09:47:39.675 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:47:39.675 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:47:39.675 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-06T09:47:39.676 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:1 2024-04-06T09:47:39.676 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:47:39.676 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:47:39.676 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:47:39.676 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=1 2024-04-06T09:47:39.676 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-06T09:47:39.676 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:47:39.676 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:47:39.676 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=1 2024-04-06T09:47:39.676 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=1 2024-04-06T09:47:39.676 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-06T09:47:39.676 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:1 2024-04-06T09:47:39.676 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:47:39.676 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:47:39.677 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-06T09:47:39.677 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:1 2024-04-06T09:47:39.677 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:47:39.677 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:47:39.677 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:47:39.677 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=1 2024-04-06T09:47:39.677 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-06T09:47:39.677 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:47:39.677 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:47:39.677 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=1 2024-04-06T09:47:39.677 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=1 2024-04-06T09:47:39.677 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-06T09:47:39.699 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T09:47:39.700 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T09:47:39.700 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T09:47:39.700 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/test 2024-04-06T09:47:39.825 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:47:39.827 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:47:39.827 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:47:39.827 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:47:39.827 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:47:39.827 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:47:39.827 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:47:39.827 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:47:39.827 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:47:39.827 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:47:39.827 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:47:39.828 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:47:39.828 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:47:39.829 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:47:39.829 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:47:39.829 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:47:39.891 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 35 2024-04-06T09:47:39.891 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 36 2024-04-06T09:47:39.891 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:47:39.891 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=145, tag_tid=27, entry_tid=1], [object_number=144, tag_tid=27, entry_tid=0], [object_number=147, tag_tid=26, entry_tid=8935], [object_number=146, tag_tid=26, entry_tid=8934]]], state=connected] 2024-04-06T09:47:39.891 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=142, tag_tid=26, entry_tid=5394], [object_number=141, tag_tid=26, entry_tid=5393], [object_number=140, tag_tid=26, entry_tid=5392], [object_number=143, tag_tid=26, entry_tid=5391]]], state=connected] 2024-04-06T09:47:39.895 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:47:39.896 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=145, tag_tid=27, entry_tid=1' 2024-04-06T09:47:39.897 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:47:39.897 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:47:39.897 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:47:39.897 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:47:39.898 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:47:39.898 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:47:39.898 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:47:39.898 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:47:39.898 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:47:39.898 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:47:39.898 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:47:39.898 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:47:39.899 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:47:39.899 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:47:39.899 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:47:39.950 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 35 2024-04-06T09:47:39.951 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 36 2024-04-06T09:47:39.951 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:47:39.951 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=145, tag_tid=27, entry_tid=1], [object_number=144, tag_tid=27, entry_tid=0], [object_number=147, tag_tid=26, entry_tid=8935], [object_number=146, tag_tid=26, entry_tid=8934]]], state=connected] 2024-04-06T09:47:39.951 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=142, tag_tid=26, entry_tid=5394], [object_number=141, tag_tid=26, entry_tid=5393], [object_number=140, tag_tid=26, entry_tid=5392], [object_number=143, tag_tid=26, entry_tid=5391]]], state=connected] 2024-04-06T09:47:39.981 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:47:39.984 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=142, tag_tid=26, entry_tid=5394' 2024-04-06T09:47:39.984 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=145, tag_tid=27, entry_tid=1' -a 'object_number=145, tag_tid=27, entry_tid=1' = 'object_number=142, tag_tid=26, entry_tid=5394' 2024-04-06T09:47:39.984 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=142, tag_tid=26, entry_tid=5394' '!=' 'object_number=140, tag_tid=26, entry_tid=5260' 2024-04-06T09:47:39.984 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:47:39.984 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=142, tag_tid=26, entry_tid=5394' = 'object_number=140, tag_tid=26, entry_tid=5260' 2024-04-06T09:47:39.984 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=142, tag_tid=26, entry_tid=5394' 2024-04-06T09:47:39.984 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=145, tag_tid=27, entry_tid=1' 2024-04-06T09:47:39.984 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:47:39.984 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:47:39.985 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=27 2024-04-06T09:47:39.985 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=142, tag_tid=26, entry_tid=5394' 2024-04-06T09:47:39.986 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:47:39.986 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:47:39.987 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=26 2024-04-06T09:47:39.987 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=145, tag_tid=27, entry_tid=1' 2024-04-06T09:47:39.987 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:47:39.987 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:47:39.988 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:47:39.989 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=142, tag_tid=26, entry_tid=5394' 2024-04-06T09:47:39.989 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:47:39.989 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:47:39.990 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=5394 2024-04-06T09:47:39.991 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 27 = 26 -a 1 -le 5394 2024-04-06T09:47:39.991 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:47:39.991 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:47:39.991 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:47:40.192 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:47:40.193 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:47:40.193 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:47:40.193 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:47:40.193 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:47:40.193 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:47:40.193 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:47:40.193 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:47:40.193 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:47:40.193 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:47:40.193 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:47:40.193 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:47:40.193 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:47:40.193 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:47:40.194 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:47:40.194 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:47:40.194 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:47:40.197 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:47:40.197 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:47:40.197 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:47:40.197 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:47:40.198 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:47:40.198 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:47:40.198 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:47:40.198 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:47:40.198 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:47:40.198 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:47:40.198 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:47:40.198 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:47:40.198 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:47:40.198 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:47:40.198 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:47:40.198 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:47:40.198 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:47:40.198 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:47:40.199 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:47:40.199 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:47:40.199 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:47:40.199 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:47:40.199 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:47:40.199 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:47:40.199 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:47:40.199 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:47:40.199 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:47:40.199 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:47:40.199 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:47:40.199 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:47:40.199 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:47:40.199 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:47:40.222 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:47:40.223 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:47:40.223 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:47:40.223 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:47:40.328 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-06T09:47:40.328 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:rbd mirror flush : flush rbd mirror 2024-04-06T09:47:40.328 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:admin_socket: invalid command 2024-04-06T09:47:40.330 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:47:40.330 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/test 2024-04-06T09:47:40.330 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:1 2024-04-06T09:47:40.330 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:47:40.330 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:47:40.330 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-06T09:47:40.331 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:1 2024-04-06T09:47:40.331 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:47:40.331 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:47:40.331 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:47:40.331 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=1 2024-04-06T09:47:40.331 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-06T09:47:40.331 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:47:40.331 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:47:40.331 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=1 2024-04-06T09:47:40.331 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=1 2024-04-06T09:47:40.332 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-06T09:47:40.332 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:1 2024-04-06T09:47:40.333 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:47:40.333 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:47:40.333 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-06T09:47:40.333 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:1 2024-04-06T09:47:40.333 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:47:40.333 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:47:40.333 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:47:40.333 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=1 2024-04-06T09:47:40.333 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-06T09:47:40.333 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:47:40.333 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:47:40.333 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=1 2024-04-06T09:47:40.333 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=1 2024-04-06T09:47:40.333 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-06T09:47:40.351 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T09:47:40.352 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T09:47:40.352 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T09:47:40.352 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/test 2024-04-06T09:47:40.476 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:47:40.477 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:47:40.477 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:47:40.477 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:47:40.477 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:47:40.477 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:47:40.478 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:47:40.478 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:47:40.478 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:47:40.478 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:47:40.478 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:47:40.478 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:47:40.478 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:47:40.479 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:47:40.480 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:47:40.480 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:47:40.532 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 35 2024-04-06T09:47:40.532 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 36 2024-04-06T09:47:40.532 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:47:40.532 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=145, tag_tid=27, entry_tid=1], [object_number=144, tag_tid=27, entry_tid=0], [object_number=147, tag_tid=26, entry_tid=8935], [object_number=146, tag_tid=26, entry_tid=8934]]], state=connected] 2024-04-06T09:47:40.532 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=143, tag_tid=26, entry_tid=5507], [object_number=142, tag_tid=26, entry_tid=5506], [object_number=141, tag_tid=26, entry_tid=5505], [object_number=140, tag_tid=26, entry_tid=5504]]], state=connected] 2024-04-06T09:47:40.536 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:47:40.537 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=145, tag_tid=27, entry_tid=1' 2024-04-06T09:47:40.538 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:47:40.538 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:47:40.538 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:47:40.538 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:47:40.538 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:47:40.538 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:47:40.538 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:47:40.539 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:47:40.539 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:47:40.539 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:47:40.539 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:47:40.539 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:47:40.539 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:47:40.540 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:47:40.540 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:47:40.588 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 35 2024-04-06T09:47:40.588 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 36 2024-04-06T09:47:40.588 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:47:40.588 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=145, tag_tid=27, entry_tid=1], [object_number=144, tag_tid=27, entry_tid=0], [object_number=147, tag_tid=26, entry_tid=8935], [object_number=146, tag_tid=26, entry_tid=8934]]], state=connected] 2024-04-06T09:47:40.588 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=143, tag_tid=26, entry_tid=5507], [object_number=142, tag_tid=26, entry_tid=5506], [object_number=141, tag_tid=26, entry_tid=5505], [object_number=140, tag_tid=26, entry_tid=5504]]], state=connected] 2024-04-06T09:47:40.591 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:47:40.594 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=143, tag_tid=26, entry_tid=5507' 2024-04-06T09:47:40.594 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=145, tag_tid=27, entry_tid=1' -a 'object_number=145, tag_tid=27, entry_tid=1' = 'object_number=143, tag_tid=26, entry_tid=5507' 2024-04-06T09:47:40.594 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=143, tag_tid=26, entry_tid=5507' '!=' 'object_number=142, tag_tid=26, entry_tid=5394' 2024-04-06T09:47:40.594 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:47:40.594 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=143, tag_tid=26, entry_tid=5507' = 'object_number=142, tag_tid=26, entry_tid=5394' 2024-04-06T09:47:40.594 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=143, tag_tid=26, entry_tid=5507' 2024-04-06T09:47:40.594 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=145, tag_tid=27, entry_tid=1' 2024-04-06T09:47:40.594 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:47:40.594 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:47:40.595 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=27 2024-04-06T09:47:40.595 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=143, tag_tid=26, entry_tid=5507' 2024-04-06T09:47:40.596 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:47:40.596 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:47:40.597 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=26 2024-04-06T09:47:40.597 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=145, tag_tid=27, entry_tid=1' 2024-04-06T09:47:40.597 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:47:40.597 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:47:40.599 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:47:40.599 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=143, tag_tid=26, entry_tid=5507' 2024-04-06T09:47:40.599 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:47:40.599 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:47:40.600 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=5507 2024-04-06T09:47:40.601 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 27 = 26 -a 1 -le 5507 2024-04-06T09:47:40.601 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:47:40.601 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:47:40.601 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:47:40.802 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:47:40.802 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:47:40.802 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:47:40.803 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:47:40.803 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:47:40.803 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:47:40.803 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:47:40.803 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:47:40.803 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:47:40.803 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:47:40.803 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:47:40.803 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:47:40.803 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:47:40.803 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:47:40.803 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:47:40.804 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:47:40.804 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:47:40.807 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:47:40.807 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:47:40.807 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:47:40.807 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:47:40.807 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:47:40.807 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:47:40.807 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:47:40.807 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:47:40.807 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:47:40.808 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:47:40.808 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:47:40.808 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:47:40.808 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:47:40.808 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:47:40.808 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:47:40.808 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:47:40.809 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:47:40.809 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:47:40.809 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:47:40.809 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:47:40.809 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:47:40.809 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:47:40.810 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:47:40.810 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:47:40.810 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:47:40.810 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:47:40.810 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:47:40.810 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:47:40.810 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:47:40.810 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:47:40.810 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:47:40.810 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:47:40.834 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:47:40.834 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:47:40.835 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:47:40.835 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:47:40.940 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-06T09:47:40.940 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:rbd mirror flush : flush rbd mirror 2024-04-06T09:47:40.940 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:admin_socket: invalid command 2024-04-06T09:47:40.943 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:47:40.943 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/test 2024-04-06T09:47:40.943 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:1 2024-04-06T09:47:40.943 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:47:40.943 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:47:40.943 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-06T09:47:40.943 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:1 2024-04-06T09:47:40.943 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:47:40.943 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:47:40.943 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:47:40.943 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=1 2024-04-06T09:47:40.943 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-06T09:47:40.943 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:47:40.944 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:47:40.944 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=1 2024-04-06T09:47:40.944 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=1 2024-04-06T09:47:40.944 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-06T09:47:40.944 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:1 2024-04-06T09:47:40.944 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:47:40.944 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:47:40.944 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-06T09:47:40.944 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:1 2024-04-06T09:47:40.944 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:47:40.944 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:47:40.944 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:47:40.944 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=1 2024-04-06T09:47:40.944 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-06T09:47:40.944 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:47:40.944 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:47:40.945 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=1 2024-04-06T09:47:40.945 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=1 2024-04-06T09:47:40.945 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-06T09:47:40.968 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T09:47:40.968 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T09:47:40.968 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T09:47:40.968 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/test 2024-04-06T09:47:41.098 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:47:41.098 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:47:41.099 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:47:41.099 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:47:41.099 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:47:41.099 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:47:41.099 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:47:41.099 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:47:41.099 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:47:41.099 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:47:41.100 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:47:41.100 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:47:41.100 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:47:41.101 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:47:41.102 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:47:41.102 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:47:41.154 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 35 2024-04-06T09:47:41.154 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 36 2024-04-06T09:47:41.154 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:47:41.154 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=145, tag_tid=27, entry_tid=1], [object_number=144, tag_tid=27, entry_tid=0], [object_number=147, tag_tid=26, entry_tid=8935], [object_number=146, tag_tid=26, entry_tid=8934]]], state=connected] 2024-04-06T09:47:41.154 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=142, tag_tid=26, entry_tid=5618], [object_number=141, tag_tid=26, entry_tid=5617], [object_number=140, tag_tid=26, entry_tid=5616], [object_number=143, tag_tid=26, entry_tid=5615]]], state=connected] 2024-04-06T09:47:41.158 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:47:41.159 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=145, tag_tid=27, entry_tid=1' 2024-04-06T09:47:41.160 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:47:41.160 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:47:41.160 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:47:41.160 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:47:41.160 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:47:41.161 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:47:41.161 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:47:41.161 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:47:41.161 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:47:41.161 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:47:41.161 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:47:41.161 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:47:41.162 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:47:41.162 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:47:41.162 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:47:41.209 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 35 2024-04-06T09:47:41.210 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 36 2024-04-06T09:47:41.210 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:47:41.210 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=145, tag_tid=27, entry_tid=1], [object_number=144, tag_tid=27, entry_tid=0], [object_number=147, tag_tid=26, entry_tid=8935], [object_number=146, tag_tid=26, entry_tid=8934]]], state=connected] 2024-04-06T09:47:41.210 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=142, tag_tid=26, entry_tid=5618], [object_number=141, tag_tid=26, entry_tid=5617], [object_number=140, tag_tid=26, entry_tid=5616], [object_number=143, tag_tid=26, entry_tid=5615]]], state=connected] 2024-04-06T09:47:41.213 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:47:41.215 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=142, tag_tid=26, entry_tid=5618' 2024-04-06T09:47:41.215 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=145, tag_tid=27, entry_tid=1' -a 'object_number=145, tag_tid=27, entry_tid=1' = 'object_number=142, tag_tid=26, entry_tid=5618' 2024-04-06T09:47:41.216 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=142, tag_tid=26, entry_tid=5618' '!=' 'object_number=143, tag_tid=26, entry_tid=5507' 2024-04-06T09:47:41.216 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:47:41.216 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=142, tag_tid=26, entry_tid=5618' = 'object_number=143, tag_tid=26, entry_tid=5507' 2024-04-06T09:47:41.216 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=142, tag_tid=26, entry_tid=5618' 2024-04-06T09:47:41.216 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=145, tag_tid=27, entry_tid=1' 2024-04-06T09:47:41.216 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:47:41.216 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:47:41.216 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=27 2024-04-06T09:47:41.217 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=142, tag_tid=26, entry_tid=5618' 2024-04-06T09:47:41.217 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:47:41.217 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:47:41.218 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=26 2024-04-06T09:47:41.219 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=145, tag_tid=27, entry_tid=1' 2024-04-06T09:47:41.219 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:47:41.219 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:47:41.220 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:47:41.220 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=142, tag_tid=26, entry_tid=5618' 2024-04-06T09:47:41.221 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:47:41.221 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:47:41.222 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=5618 2024-04-06T09:47:41.222 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 27 = 26 -a 1 -le 5618 2024-04-06T09:47:41.222 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:47:41.222 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:47:41.222 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:47:41.423 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:47:41.423 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:47:41.424 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:47:41.424 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:47:41.424 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:47:41.424 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:47:41.424 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:47:41.424 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:47:41.424 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:47:41.424 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:47:41.424 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:47:41.424 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:47:41.424 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:47:41.424 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:47:41.424 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:47:41.425 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:47:41.425 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:47:41.428 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:47:41.428 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:47:41.428 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:47:41.428 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:47:41.428 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:47:41.428 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:47:41.428 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:47:41.428 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:47:41.428 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:47:41.428 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:47:41.428 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:47:41.429 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:47:41.429 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:47:41.429 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:47:41.429 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:47:41.429 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:47:41.429 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:47:41.429 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:47:41.429 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:47:41.429 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:47:41.429 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:47:41.429 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:47:41.429 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:47:41.429 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:47:41.429 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:47:41.429 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:47:41.430 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:47:41.430 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:47:41.430 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:47:41.430 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:47:41.430 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:47:41.430 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:47:41.454 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:47:41.455 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:47:41.455 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:47:41.455 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:47:41.562 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-06T09:47:41.563 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:rbd mirror flush : flush rbd mirror 2024-04-06T09:47:41.563 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:admin_socket: invalid command 2024-04-06T09:47:41.567 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:47:41.568 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/test 2024-04-06T09:47:41.568 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:1 2024-04-06T09:47:41.568 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:47:41.568 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:47:41.568 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-06T09:47:41.568 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:1 2024-04-06T09:47:41.569 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:47:41.569 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:47:41.569 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:47:41.569 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=1 2024-04-06T09:47:41.569 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-06T09:47:41.569 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:47:41.570 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:47:41.570 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=1 2024-04-06T09:47:41.570 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=1 2024-04-06T09:47:41.570 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-06T09:47:41.570 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:1 2024-04-06T09:47:41.571 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:47:41.571 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:47:41.571 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-06T09:47:41.571 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:1 2024-04-06T09:47:41.571 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:47:41.571 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:47:41.571 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:47:41.572 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=1 2024-04-06T09:47:41.572 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-06T09:47:41.572 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:47:41.572 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:47:41.572 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=1 2024-04-06T09:47:41.573 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=1 2024-04-06T09:47:41.573 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-06T09:47:41.590 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T09:47:41.590 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T09:47:41.590 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T09:47:41.590 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/test 2024-04-06T09:47:41.719 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:47:41.720 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:47:41.720 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:47:41.720 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:47:41.720 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:47:41.720 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:47:41.720 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:47:41.721 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:47:41.721 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:47:41.721 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:47:41.721 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:47:41.722 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:47:41.722 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:47:41.723 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:47:41.724 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:47:41.724 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:47:41.781 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 35 2024-04-06T09:47:41.781 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 36 2024-04-06T09:47:41.781 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:47:41.781 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=145, tag_tid=27, entry_tid=1], [object_number=144, tag_tid=27, entry_tid=0], [object_number=147, tag_tid=26, entry_tid=8935], [object_number=146, tag_tid=26, entry_tid=8934]]], state=connected] 2024-04-06T09:47:41.781 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=143, tag_tid=26, entry_tid=5727], [object_number=142, tag_tid=26, entry_tid=5726], [object_number=141, tag_tid=26, entry_tid=5725], [object_number=140, tag_tid=26, entry_tid=5724]]], state=connected] 2024-04-06T09:47:41.785 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:47:41.786 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=145, tag_tid=27, entry_tid=1' 2024-04-06T09:47:41.787 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:47:41.787 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:47:41.787 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:47:41.787 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:47:41.788 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:47:41.788 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:47:41.788 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:47:41.788 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:47:41.788 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:47:41.788 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:47:41.788 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:47:41.788 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:47:41.789 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:47:41.789 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:47:41.789 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:47:41.836 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 35 2024-04-06T09:47:41.836 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 36 2024-04-06T09:47:41.836 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:47:41.836 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=145, tag_tid=27, entry_tid=1], [object_number=144, tag_tid=27, entry_tid=0], [object_number=147, tag_tid=26, entry_tid=8935], [object_number=146, tag_tid=26, entry_tid=8934]]], state=connected] 2024-04-06T09:47:41.837 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=143, tag_tid=26, entry_tid=5727], [object_number=142, tag_tid=26, entry_tid=5726], [object_number=141, tag_tid=26, entry_tid=5725], [object_number=140, tag_tid=26, entry_tid=5724]]], state=connected] 2024-04-06T09:47:41.840 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:47:41.842 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=143, tag_tid=26, entry_tid=5727' 2024-04-06T09:47:41.842 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=145, tag_tid=27, entry_tid=1' -a 'object_number=145, tag_tid=27, entry_tid=1' = 'object_number=143, tag_tid=26, entry_tid=5727' 2024-04-06T09:47:41.842 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=143, tag_tid=26, entry_tid=5727' '!=' 'object_number=142, tag_tid=26, entry_tid=5618' 2024-04-06T09:47:41.842 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:47:41.843 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=143, tag_tid=26, entry_tid=5727' = 'object_number=142, tag_tid=26, entry_tid=5618' 2024-04-06T09:47:41.843 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=143, tag_tid=26, entry_tid=5727' 2024-04-06T09:47:41.843 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=145, tag_tid=27, entry_tid=1' 2024-04-06T09:47:41.843 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:47:41.843 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:47:41.843 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=27 2024-04-06T09:47:41.844 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=143, tag_tid=26, entry_tid=5727' 2024-04-06T09:47:41.844 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:47:41.844 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:47:41.845 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=26 2024-04-06T09:47:41.846 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=145, tag_tid=27, entry_tid=1' 2024-04-06T09:47:41.846 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:47:41.846 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:47:41.847 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:47:41.848 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=143, tag_tid=26, entry_tid=5727' 2024-04-06T09:47:41.848 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:47:41.848 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:47:41.849 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=5727 2024-04-06T09:47:41.849 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 27 = 26 -a 1 -le 5727 2024-04-06T09:47:41.849 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:47:41.849 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:47:41.849 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:47:42.051 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:47:42.051 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:47:42.051 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:47:42.051 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:47:42.051 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:47:42.051 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:47:42.051 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:47:42.051 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:47:42.051 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:47:42.051 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:47:42.052 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:47:42.052 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:47:42.052 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:47:42.052 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:47:42.052 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:47:42.053 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:47:42.053 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:47:42.053 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:47:42.053 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:47:42.054 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:47:42.054 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:47:42.054 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:47:42.054 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:47:42.054 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:47:42.054 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:47:42.054 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:47:42.054 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:47:42.054 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:47:42.054 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:47:42.054 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:47:42.054 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:47:42.054 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:47:42.054 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:47:42.056 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:47:42.056 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:47:42.056 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:47:42.056 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:47:42.056 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:47:42.056 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:47:42.056 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:47:42.056 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:47:42.057 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:47:42.057 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:47:42.057 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:47:42.057 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:47:42.057 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:47:42.057 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:47:42.057 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:47:42.057 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:47:42.080 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:47:42.080 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:47:42.080 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:47:42.080 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:47:42.183 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-06T09:47:42.183 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:rbd mirror flush : flush rbd mirror 2024-04-06T09:47:42.183 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:admin_socket: invalid command 2024-04-06T09:47:42.186 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:47:42.186 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/test 2024-04-06T09:47:42.186 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:1 2024-04-06T09:47:42.186 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:47:42.186 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:47:42.186 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-06T09:47:42.186 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:1 2024-04-06T09:47:42.186 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:47:42.186 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:47:42.186 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:47:42.186 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=1 2024-04-06T09:47:42.186 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-06T09:47:42.186 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:47:42.187 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:47:42.187 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=1 2024-04-06T09:47:42.187 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=1 2024-04-06T09:47:42.188 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-06T09:47:42.188 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:1 2024-04-06T09:47:42.188 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:47:42.188 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:47:42.188 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-06T09:47:42.188 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:1 2024-04-06T09:47:42.188 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:47:42.188 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:47:42.189 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:47:42.189 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=1 2024-04-06T09:47:42.189 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-06T09:47:42.189 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:47:42.189 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:47:42.189 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=1 2024-04-06T09:47:42.189 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=1 2024-04-06T09:47:42.189 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-06T09:47:42.213 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T09:47:42.213 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T09:47:42.213 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T09:47:42.213 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/test 2024-04-06T09:47:42.337 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:47:42.339 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:47:42.339 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:47:42.339 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:47:42.339 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:47:42.340 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:47:42.340 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:47:42.340 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:47:42.340 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:47:42.340 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:47:42.340 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:47:42.340 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:47:42.340 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:47:42.341 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:47:42.341 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:47:42.341 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:47:42.604 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 35 2024-04-06T09:47:42.604 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 36 2024-04-06T09:47:42.604 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:47:42.605 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=145, tag_tid=27, entry_tid=1], [object_number=144, tag_tid=27, entry_tid=0], [object_number=147, tag_tid=26, entry_tid=8935], [object_number=146, tag_tid=26, entry_tid=8934]]], state=connected] 2024-04-06T09:47:42.605 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=140, tag_tid=26, entry_tid=5832], [object_number=143, tag_tid=26, entry_tid=5831], [object_number=142, tag_tid=26, entry_tid=5830], [object_number=141, tag_tid=26, entry_tid=5829]]], state=connected] 2024-04-06T09:47:42.610 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:47:42.612 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=145, tag_tid=27, entry_tid=1' 2024-04-06T09:47:42.613 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:47:42.613 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:47:42.613 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:47:42.613 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:47:42.613 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:47:42.613 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:47:42.614 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:47:42.614 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:47:42.614 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:47:42.614 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:47:42.614 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:47:42.615 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:47:42.616 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:47:42.616 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:47:42.616 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:47:42.673 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 35 2024-04-06T09:47:42.674 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 36 2024-04-06T09:47:42.674 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:47:42.674 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=145, tag_tid=27, entry_tid=1], [object_number=144, tag_tid=27, entry_tid=0], [object_number=147, tag_tid=26, entry_tid=8935], [object_number=146, tag_tid=26, entry_tid=8934]]], state=connected] 2024-04-06T09:47:42.674 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=140, tag_tid=26, entry_tid=5832], [object_number=143, tag_tid=26, entry_tid=5831], [object_number=142, tag_tid=26, entry_tid=5830], [object_number=141, tag_tid=26, entry_tid=5829]]], state=connected] 2024-04-06T09:47:42.677 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:47:42.680 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=140, tag_tid=26, entry_tid=5832' 2024-04-06T09:47:42.680 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=145, tag_tid=27, entry_tid=1' -a 'object_number=145, tag_tid=27, entry_tid=1' = 'object_number=140, tag_tid=26, entry_tid=5832' 2024-04-06T09:47:42.680 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=140, tag_tid=26, entry_tid=5832' '!=' 'object_number=143, tag_tid=26, entry_tid=5727' 2024-04-06T09:47:42.680 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:47:42.680 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=140, tag_tid=26, entry_tid=5832' = 'object_number=143, tag_tid=26, entry_tid=5727' 2024-04-06T09:47:42.680 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=140, tag_tid=26, entry_tid=5832' 2024-04-06T09:47:42.680 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=145, tag_tid=27, entry_tid=1' 2024-04-06T09:47:42.680 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:47:42.680 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:47:42.681 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=27 2024-04-06T09:47:42.682 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=140, tag_tid=26, entry_tid=5832' 2024-04-06T09:47:42.682 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:47:42.682 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:47:42.683 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=26 2024-04-06T09:47:42.683 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=145, tag_tid=27, entry_tid=1' 2024-04-06T09:47:42.683 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:47:42.684 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:47:42.684 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:47:42.685 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=140, tag_tid=26, entry_tid=5832' 2024-04-06T09:47:42.685 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:47:42.685 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:47:42.686 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=5832 2024-04-06T09:47:42.687 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 27 = 26 -a 1 -le 5832 2024-04-06T09:47:42.687 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:47:42.687 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:47:42.687 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:47:42.888 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:47:42.888 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:47:42.888 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:47:42.888 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:47:42.888 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:47:42.889 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:47:42.889 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:47:42.889 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:47:42.889 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:47:42.889 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:47:42.889 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:47:42.889 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:47:42.889 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:47:42.889 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:47:42.890 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:47:42.892 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:47:42.892 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:47:42.893 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:47:42.893 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:47:42.893 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:47:42.893 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:47:42.893 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:47:42.893 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:47:42.893 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:47:42.893 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:47:42.893 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:47:42.893 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:47:42.893 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:47:42.894 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:47:42.894 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:47:42.894 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:47:42.894 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:47:42.894 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:47:42.894 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:47:42.894 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:47:42.894 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:47:42.894 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:47:42.894 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:47:42.895 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:47:42.895 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:47:42.895 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:47:42.895 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:47:42.895 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:47:42.895 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:47:42.895 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:47:42.895 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:47:42.895 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:47:42.895 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:47:42.895 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:47:42.917 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:47:42.918 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:47:42.918 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:47:42.918 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:47:43.023 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-06T09:47:43.024 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:rbd mirror flush : flush rbd mirror 2024-04-06T09:47:43.024 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:admin_socket: invalid command 2024-04-06T09:47:43.027 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:47:43.027 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/test 2024-04-06T09:47:43.027 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:1 2024-04-06T09:47:43.027 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:47:43.027 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:47:43.027 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-06T09:47:43.027 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:1 2024-04-06T09:47:43.027 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:47:43.027 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:47:43.027 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:47:43.027 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=1 2024-04-06T09:47:43.027 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-06T09:47:43.027 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:47:43.028 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:47:43.028 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=1 2024-04-06T09:47:43.028 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=1 2024-04-06T09:47:43.028 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-06T09:47:43.028 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:1 2024-04-06T09:47:43.028 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:47:43.028 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:47:43.028 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-06T09:47:43.028 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:1 2024-04-06T09:47:43.028 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:47:43.028 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:47:43.028 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:47:43.028 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=1 2024-04-06T09:47:43.028 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-06T09:47:43.029 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:47:43.029 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:47:43.029 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=1 2024-04-06T09:47:43.029 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=1 2024-04-06T09:47:43.029 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-06T09:47:43.053 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T09:47:43.053 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T09:47:43.053 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T09:47:43.053 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/test 2024-04-06T09:47:43.177 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:47:43.178 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:47:43.178 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:47:43.178 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:47:43.178 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:47:43.178 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:47:43.178 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:47:43.179 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:47:43.179 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:47:43.179 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:47:43.179 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:47:43.179 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:47:43.179 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:47:43.180 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:47:43.181 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:47:43.181 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:47:43.234 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 35 2024-04-06T09:47:43.234 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 36 2024-04-06T09:47:43.234 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:47:43.234 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=145, tag_tid=27, entry_tid=1], [object_number=144, tag_tid=27, entry_tid=0], [object_number=147, tag_tid=26, entry_tid=8935], [object_number=146, tag_tid=26, entry_tid=8934]]], state=connected] 2024-04-06T09:47:43.234 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=142, tag_tid=26, entry_tid=5966], [object_number=141, tag_tid=26, entry_tid=5965], [object_number=140, tag_tid=26, entry_tid=5964], [object_number=143, tag_tid=26, entry_tid=5963]]], state=connected] 2024-04-06T09:47:43.237 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:47:43.239 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=145, tag_tid=27, entry_tid=1' 2024-04-06T09:47:43.240 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:47:43.240 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:47:43.240 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:47:43.240 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:47:43.240 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:47:43.240 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:47:43.240 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:47:43.240 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:47:43.241 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:47:43.241 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:47:43.241 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:47:43.241 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:47:43.241 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:47:43.242 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:47:43.242 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:47:43.289 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 35 2024-04-06T09:47:43.289 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 36 2024-04-06T09:47:43.290 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:47:43.290 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=145, tag_tid=27, entry_tid=1], [object_number=144, tag_tid=27, entry_tid=0], [object_number=147, tag_tid=26, entry_tid=8935], [object_number=146, tag_tid=26, entry_tid=8934]]], state=connected] 2024-04-06T09:47:43.290 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=142, tag_tid=26, entry_tid=5966], [object_number=141, tag_tid=26, entry_tid=5965], [object_number=140, tag_tid=26, entry_tid=5964], [object_number=143, tag_tid=26, entry_tid=5963]]], state=connected] 2024-04-06T09:47:43.293 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:47:43.295 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=142, tag_tid=26, entry_tid=5966' 2024-04-06T09:47:43.295 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=145, tag_tid=27, entry_tid=1' -a 'object_number=145, tag_tid=27, entry_tid=1' = 'object_number=142, tag_tid=26, entry_tid=5966' 2024-04-06T09:47:43.295 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=142, tag_tid=26, entry_tid=5966' '!=' 'object_number=140, tag_tid=26, entry_tid=5832' 2024-04-06T09:47:43.295 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:47:43.295 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=142, tag_tid=26, entry_tid=5966' = 'object_number=140, tag_tid=26, entry_tid=5832' 2024-04-06T09:47:43.295 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=142, tag_tid=26, entry_tid=5966' 2024-04-06T09:47:43.296 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=145, tag_tid=27, entry_tid=1' 2024-04-06T09:47:43.296 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:47:43.296 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:47:43.296 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=27 2024-04-06T09:47:43.297 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=142, tag_tid=26, entry_tid=5966' 2024-04-06T09:47:43.297 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:47:43.297 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:47:43.298 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=26 2024-04-06T09:47:43.299 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=145, tag_tid=27, entry_tid=1' 2024-04-06T09:47:43.299 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:47:43.299 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:47:43.300 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:47:43.300 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=142, tag_tid=26, entry_tid=5966' 2024-04-06T09:47:43.300 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:47:43.301 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:47:43.301 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=5966 2024-04-06T09:47:43.302 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 27 = 26 -a 1 -le 5966 2024-04-06T09:47:43.302 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:47:43.302 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:47:43.302 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:47:43.503 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:47:43.503 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:47:43.503 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:47:43.503 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:47:43.503 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:47:43.503 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:47:43.504 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:47:43.504 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:47:43.504 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:47:43.504 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:47:43.504 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:47:43.504 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:47:43.504 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:47:43.504 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:47:43.504 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:47:43.504 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:47:43.505 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:47:43.508 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:47:43.508 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:47:43.508 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:47:43.508 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:47:43.508 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:47:43.508 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:47:43.508 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:47:43.508 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:47:43.508 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:47:43.508 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:47:43.509 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:47:43.509 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:47:43.509 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:47:43.509 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:47:43.509 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:47:43.509 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:47:43.509 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:47:43.509 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:47:43.509 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:47:43.509 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:47:43.509 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:47:43.509 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:47:43.509 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:47:43.509 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:47:43.510 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:47:43.510 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:47:43.510 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:47:43.510 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:47:43.510 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:47:43.510 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:47:43.510 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:47:43.510 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:47:43.534 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:47:43.535 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:47:43.535 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:47:43.535 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:47:43.642 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-06T09:47:43.642 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:rbd mirror flush : flush rbd mirror 2024-04-06T09:47:43.642 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:admin_socket: invalid command 2024-04-06T09:47:43.645 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:47:43.646 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/test 2024-04-06T09:47:43.646 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:1 2024-04-06T09:47:43.646 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:47:43.646 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:47:43.646 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-06T09:47:43.646 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:1 2024-04-06T09:47:43.646 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:47:43.646 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:47:43.646 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:47:43.646 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=1 2024-04-06T09:47:43.646 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-06T09:47:43.646 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:47:43.646 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:47:43.646 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=1 2024-04-06T09:47:43.647 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=1 2024-04-06T09:47:43.647 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-06T09:47:43.647 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:1 2024-04-06T09:47:43.647 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:47:43.647 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:47:43.647 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-06T09:47:43.647 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:1 2024-04-06T09:47:43.647 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:47:43.647 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:47:43.647 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:47:43.647 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=1 2024-04-06T09:47:43.647 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-06T09:47:43.647 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:47:43.647 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:47:43.647 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=1 2024-04-06T09:47:43.648 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=1 2024-04-06T09:47:43.648 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-06T09:47:43.671 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T09:47:43.671 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T09:47:43.671 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T09:47:43.671 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/test 2024-04-06T09:47:43.791 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:47:43.792 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:47:43.793 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:47:43.793 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:47:43.793 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:47:43.793 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:47:43.793 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:47:43.793 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:47:43.793 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:47:43.793 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:47:43.794 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:47:43.794 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:47:43.794 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:47:43.795 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:47:43.796 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:47:43.796 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:47:43.851 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 35 2024-04-06T09:47:43.851 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 36 2024-04-06T09:47:43.851 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:47:43.851 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=145, tag_tid=27, entry_tid=1], [object_number=144, tag_tid=27, entry_tid=0], [object_number=147, tag_tid=26, entry_tid=8935], [object_number=146, tag_tid=26, entry_tid=8934]]], state=connected] 2024-04-06T09:47:43.851 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=143, tag_tid=26, entry_tid=6075], [object_number=142, tag_tid=26, entry_tid=6074], [object_number=141, tag_tid=26, entry_tid=6073], [object_number=140, tag_tid=26, entry_tid=6072]]], state=connected] 2024-04-06T09:47:43.854 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:47:43.855 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=145, tag_tid=27, entry_tid=1' 2024-04-06T09:47:43.856 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:47:43.856 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:47:43.856 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:47:43.857 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:47:43.857 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:47:43.857 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:47:43.857 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:47:43.857 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:47:43.857 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:47:43.857 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:47:43.857 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:47:43.857 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:47:43.857 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:47:43.858 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:47:43.858 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:47:43.906 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 35 2024-04-06T09:47:43.906 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 36 2024-04-06T09:47:43.906 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:47:43.906 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=145, tag_tid=27, entry_tid=1], [object_number=144, tag_tid=27, entry_tid=0], [object_number=147, tag_tid=26, entry_tid=8935], [object_number=146, tag_tid=26, entry_tid=8934]]], state=connected] 2024-04-06T09:47:43.906 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=143, tag_tid=26, entry_tid=6075], [object_number=142, tag_tid=26, entry_tid=6074], [object_number=141, tag_tid=26, entry_tid=6073], [object_number=140, tag_tid=26, entry_tid=6072]]], state=connected] 2024-04-06T09:47:43.910 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:47:43.912 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=143, tag_tid=26, entry_tid=6075' 2024-04-06T09:47:43.912 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=145, tag_tid=27, entry_tid=1' -a 'object_number=145, tag_tid=27, entry_tid=1' = 'object_number=143, tag_tid=26, entry_tid=6075' 2024-04-06T09:47:43.912 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=143, tag_tid=26, entry_tid=6075' '!=' 'object_number=142, tag_tid=26, entry_tid=5966' 2024-04-06T09:47:43.912 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:47:43.912 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=143, tag_tid=26, entry_tid=6075' = 'object_number=142, tag_tid=26, entry_tid=5966' 2024-04-06T09:47:43.912 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=143, tag_tid=26, entry_tid=6075' 2024-04-06T09:47:43.912 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=145, tag_tid=27, entry_tid=1' 2024-04-06T09:47:43.912 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:47:43.912 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:47:43.913 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=27 2024-04-06T09:47:43.914 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=143, tag_tid=26, entry_tid=6075' 2024-04-06T09:47:43.914 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:47:43.914 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:47:43.915 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=26 2024-04-06T09:47:43.915 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=145, tag_tid=27, entry_tid=1' 2024-04-06T09:47:43.915 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:47:43.915 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:47:43.916 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:47:43.917 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=143, tag_tid=26, entry_tid=6075' 2024-04-06T09:47:43.917 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:47:43.917 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:47:43.918 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=6075 2024-04-06T09:47:43.918 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 27 = 26 -a 1 -le 6075 2024-04-06T09:47:43.918 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:47:43.919 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:47:43.919 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:47:44.120 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:47:44.120 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:47:44.121 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:47:44.121 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:47:44.121 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:47:44.121 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:47:44.121 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:47:44.121 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:47:44.121 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:47:44.121 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:47:44.121 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:47:44.121 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:47:44.121 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:47:44.121 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:47:44.121 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:47:44.122 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:47:44.122 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:47:44.125 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:47:44.126 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:47:44.126 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:47:44.126 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:47:44.126 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:47:44.126 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:47:44.126 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:47:44.126 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:47:44.126 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:47:44.126 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:47:44.126 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:47:44.126 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:47:44.126 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:47:44.127 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:47:44.127 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:47:44.127 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:47:44.127 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:47:44.127 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:47:44.127 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:47:44.127 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:47:44.127 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:47:44.127 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:47:44.127 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:47:44.127 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:47:44.127 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:47:44.127 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:47:44.127 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:47:44.127 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:47:44.128 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:47:44.128 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:47:44.128 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:47:44.128 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:47:44.151 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:47:44.152 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:47:44.152 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:47:44.152 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:47:44.255 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-06T09:47:44.256 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:rbd mirror flush : flush rbd mirror 2024-04-06T09:47:44.256 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:admin_socket: invalid command 2024-04-06T09:47:44.258 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:47:44.258 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/test 2024-04-06T09:47:44.258 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:1 2024-04-06T09:47:44.258 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:47:44.258 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:47:44.258 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-06T09:47:44.258 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:1 2024-04-06T09:47:44.258 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:47:44.258 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:47:44.259 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:47:44.259 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=1 2024-04-06T09:47:44.259 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-06T09:47:44.259 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:47:44.259 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:47:44.259 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=1 2024-04-06T09:47:44.259 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=1 2024-04-06T09:47:44.260 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-06T09:47:44.260 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:1 2024-04-06T09:47:44.260 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:47:44.261 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:47:44.261 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-06T09:47:44.261 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:1 2024-04-06T09:47:44.261 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:47:44.261 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:47:44.261 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:47:44.261 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=1 2024-04-06T09:47:44.261 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-06T09:47:44.261 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:47:44.261 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:47:44.261 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=1 2024-04-06T09:47:44.261 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=1 2024-04-06T09:47:44.261 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-06T09:47:44.278 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T09:47:44.279 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T09:47:44.279 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T09:47:44.279 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/test 2024-04-06T09:47:44.413 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:47:44.414 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:47:44.414 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:47:44.414 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:47:44.414 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:47:44.414 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:47:44.414 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:47:44.414 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:47:44.414 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:47:44.415 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:47:44.415 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:47:44.415 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:47:44.415 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:47:44.415 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:47:44.416 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:47:44.416 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:47:44.472 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 35 2024-04-06T09:47:44.472 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 36 2024-04-06T09:47:44.472 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:47:44.472 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=145, tag_tid=27, entry_tid=1], [object_number=144, tag_tid=27, entry_tid=0], [object_number=147, tag_tid=26, entry_tid=8935], [object_number=146, tag_tid=26, entry_tid=8934]]], state=connected] 2024-04-06T09:47:44.473 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=142, tag_tid=26, entry_tid=6174], [object_number=141, tag_tid=26, entry_tid=6173], [object_number=140, tag_tid=26, entry_tid=6172], [object_number=143, tag_tid=26, entry_tid=6171]]], state=connected] 2024-04-06T09:47:44.477 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:47:44.478 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=145, tag_tid=27, entry_tid=1' 2024-04-06T09:47:44.479 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:47:44.479 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:47:44.480 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:47:44.480 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:47:44.480 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:47:44.480 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:47:44.480 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:47:44.480 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:47:44.480 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:47:44.480 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:47:44.480 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:47:44.480 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:47:44.481 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:47:44.481 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:47:44.481 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:47:44.531 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 35 2024-04-06T09:47:44.531 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 36 2024-04-06T09:47:44.532 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:47:44.532 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=145, tag_tid=27, entry_tid=1], [object_number=144, tag_tid=27, entry_tid=0], [object_number=147, tag_tid=26, entry_tid=8935], [object_number=146, tag_tid=26, entry_tid=8934]]], state=connected] 2024-04-06T09:47:44.532 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=142, tag_tid=26, entry_tid=6174], [object_number=141, tag_tid=26, entry_tid=6173], [object_number=140, tag_tid=26, entry_tid=6172], [object_number=143, tag_tid=26, entry_tid=6171]]], state=connected] 2024-04-06T09:47:44.535 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:47:44.536 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=142, tag_tid=26, entry_tid=6174' 2024-04-06T09:47:44.537 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=145, tag_tid=27, entry_tid=1' -a 'object_number=145, tag_tid=27, entry_tid=1' = 'object_number=142, tag_tid=26, entry_tid=6174' 2024-04-06T09:47:44.537 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=142, tag_tid=26, entry_tid=6174' '!=' 'object_number=143, tag_tid=26, entry_tid=6075' 2024-04-06T09:47:44.537 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:47:44.537 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=142, tag_tid=26, entry_tid=6174' = 'object_number=143, tag_tid=26, entry_tid=6075' 2024-04-06T09:47:44.537 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=142, tag_tid=26, entry_tid=6174' 2024-04-06T09:47:44.537 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=145, tag_tid=27, entry_tid=1' 2024-04-06T09:47:44.538 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:47:44.538 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:47:44.538 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=27 2024-04-06T09:47:44.539 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=142, tag_tid=26, entry_tid=6174' 2024-04-06T09:47:44.539 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:47:44.539 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:47:44.540 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=26 2024-04-06T09:47:44.541 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=145, tag_tid=27, entry_tid=1' 2024-04-06T09:47:44.541 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:47:44.541 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:47:44.541 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:47:44.542 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=142, tag_tid=26, entry_tid=6174' 2024-04-06T09:47:44.542 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:47:44.542 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:47:44.543 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=6174 2024-04-06T09:47:44.544 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 27 = 26 -a 1 -le 6174 2024-04-06T09:47:44.544 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:47:44.544 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:47:44.544 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:47:44.745 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:47:44.745 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:47:44.746 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:47:44.746 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:47:44.746 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:47:44.746 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:47:44.746 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:47:44.746 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:47:44.746 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:47:44.746 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:47:44.746 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:47:44.746 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:47:44.746 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:47:44.746 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:47:44.746 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:47:44.747 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:47:44.747 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:47:44.751 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:47:44.751 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:47:44.751 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:47:44.751 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:47:44.751 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:47:44.751 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:47:44.751 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:47:44.751 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:47:44.751 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:47:44.751 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:47:44.752 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:47:44.752 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:47:44.752 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:47:44.752 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:47:44.752 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:47:44.752 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:47:44.752 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:47:44.752 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:47:44.752 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:47:44.752 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:47:44.752 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:47:44.752 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:47:44.752 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:47:44.752 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:47:44.752 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:47:44.752 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:47:44.753 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:47:44.753 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:47:44.753 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:47:44.753 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:47:44.753 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:47:44.753 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:47:44.777 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:47:44.778 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:47:44.778 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:47:44.778 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:47:44.885 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-06T09:47:44.885 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:rbd mirror flush : flush rbd mirror 2024-04-06T09:47:44.885 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:admin_socket: invalid command 2024-04-06T09:47:44.887 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:47:44.887 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/test 2024-04-06T09:47:44.887 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:1 2024-04-06T09:47:44.887 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:47:44.887 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:47:44.888 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-06T09:47:44.888 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:1 2024-04-06T09:47:44.888 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:47:44.888 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:47:44.888 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:47:44.888 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=1 2024-04-06T09:47:44.888 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-06T09:47:44.888 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:47:44.888 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:47:44.888 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=1 2024-04-06T09:47:44.888 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=1 2024-04-06T09:47:44.890 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-06T09:47:44.890 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:1 2024-04-06T09:47:44.890 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:47:44.890 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:47:44.890 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-06T09:47:44.890 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:1 2024-04-06T09:47:44.890 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:47:44.891 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:47:44.891 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:47:44.891 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=1 2024-04-06T09:47:44.891 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-06T09:47:44.891 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:47:44.891 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:47:44.891 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=1 2024-04-06T09:47:44.891 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=1 2024-04-06T09:47:44.891 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-06T09:47:44.912 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T09:47:44.912 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T09:47:44.912 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T09:47:44.912 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/test 2024-04-06T09:47:45.050 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:47:45.051 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:47:45.051 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:47:45.051 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:47:45.051 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:47:45.051 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:47:45.051 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:47:45.052 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:47:45.052 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:47:45.052 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:47:45.052 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:47:45.052 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:47:45.052 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:47:45.056 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:47:45.056 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:47:45.056 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:47:45.121 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 36 2024-04-06T09:47:45.121 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 36 2024-04-06T09:47:45.121 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:47:45.122 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=145, tag_tid=27, entry_tid=1], [object_number=144, tag_tid=27, entry_tid=0], [object_number=147, tag_tid=26, entry_tid=8935], [object_number=146, tag_tid=26, entry_tid=8934]]], state=connected] 2024-04-06T09:47:45.122 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=147, tag_tid=26, entry_tid=6263], [object_number=146, tag_tid=26, entry_tid=6262], [object_number=145, tag_tid=26, entry_tid=6261], [object_number=144, tag_tid=26, entry_tid=6260]]], state=connected] 2024-04-06T09:47:45.125 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:47:45.126 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=145, tag_tid=27, entry_tid=1' 2024-04-06T09:47:45.128 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:47:45.128 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:47:45.128 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:47:45.128 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:47:45.128 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:47:45.128 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:47:45.128 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:47:45.128 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:47:45.128 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:47:45.128 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:47:45.128 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:47:45.128 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:47:45.129 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:47:45.129 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:47:45.129 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:47:45.176 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 36 2024-04-06T09:47:45.176 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 36 2024-04-06T09:47:45.176 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:47:45.176 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=145, tag_tid=27, entry_tid=1], [object_number=144, tag_tid=27, entry_tid=0], [object_number=147, tag_tid=26, entry_tid=8935], [object_number=146, tag_tid=26, entry_tid=8934]]], state=connected] 2024-04-06T09:47:45.176 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=147, tag_tid=26, entry_tid=6263], [object_number=146, tag_tid=26, entry_tid=6262], [object_number=145, tag_tid=26, entry_tid=6261], [object_number=144, tag_tid=26, entry_tid=6260]]], state=connected] 2024-04-06T09:47:45.180 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:47:45.182 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=147, tag_tid=26, entry_tid=6263' 2024-04-06T09:47:45.182 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=145, tag_tid=27, entry_tid=1' -a 'object_number=145, tag_tid=27, entry_tid=1' = 'object_number=147, tag_tid=26, entry_tid=6263' 2024-04-06T09:47:45.182 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=147, tag_tid=26, entry_tid=6263' '!=' 'object_number=142, tag_tid=26, entry_tid=6174' 2024-04-06T09:47:45.182 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:47:45.182 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=147, tag_tid=26, entry_tid=6263' = 'object_number=142, tag_tid=26, entry_tid=6174' 2024-04-06T09:47:45.182 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=147, tag_tid=26, entry_tid=6263' 2024-04-06T09:47:45.182 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=145, tag_tid=27, entry_tid=1' 2024-04-06T09:47:45.182 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:47:45.183 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:47:45.183 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=27 2024-04-06T09:47:45.184 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=147, tag_tid=26, entry_tid=6263' 2024-04-06T09:47:45.184 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:47:45.184 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:47:45.185 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=26 2024-04-06T09:47:45.185 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=145, tag_tid=27, entry_tid=1' 2024-04-06T09:47:45.185 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:47:45.185 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:47:45.186 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:47:45.187 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=147, tag_tid=26, entry_tid=6263' 2024-04-06T09:47:45.187 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:47:45.187 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:47:45.188 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=6263 2024-04-06T09:47:45.188 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 27 = 26 -a 1 -le 6263 2024-04-06T09:47:45.188 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:47:45.188 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:47:45.189 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:47:45.390 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:47:45.390 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:47:45.391 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:47:45.391 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:47:45.391 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:47:45.391 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:47:45.391 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:47:45.391 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:47:45.391 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:47:45.391 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:47:45.391 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:47:45.391 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:47:45.391 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:47:45.391 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:47:45.391 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:47:45.394 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:47:45.395 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:47:45.395 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:47:45.395 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:47:45.395 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:47:45.395 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:47:45.395 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:47:45.395 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:47:45.395 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:47:45.395 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:47:45.395 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:47:45.395 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:47:45.395 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:47:45.395 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:47:45.396 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:47:45.396 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:47:45.396 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:47:45.396 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:47:45.396 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:47:45.396 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:47:45.396 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:47:45.396 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:47:45.396 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:47:45.396 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:47:45.396 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:47:45.396 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:47:45.396 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:47:45.396 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:47:45.396 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:47:45.397 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:47:45.397 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:47:45.397 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:47:45.397 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:47:45.397 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:47:45.419 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:47:45.420 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:47:45.420 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:47:45.420 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:47:45.534 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-06T09:47:45.534 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:rbd mirror flush : flush rbd mirror 2024-04-06T09:47:45.534 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:admin_socket: invalid command 2024-04-06T09:47:45.536 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:47:45.536 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/test 2024-04-06T09:47:45.536 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:1 2024-04-06T09:47:45.537 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:47:45.537 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:47:45.537 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-06T09:47:45.537 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:1 2024-04-06T09:47:45.537 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:47:45.537 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:47:45.537 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:47:45.537 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=1 2024-04-06T09:47:45.537 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-06T09:47:45.537 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:47:45.537 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:47:45.537 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=1 2024-04-06T09:47:45.537 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=1 2024-04-06T09:47:45.539 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-06T09:47:45.539 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:1 2024-04-06T09:47:45.539 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:47:45.539 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:47:45.539 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-06T09:47:45.539 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:1 2024-04-06T09:47:45.539 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:47:45.539 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:47:45.539 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:47:45.539 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=1 2024-04-06T09:47:45.539 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-06T09:47:45.540 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:47:45.540 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:47:45.540 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=1 2024-04-06T09:47:45.540 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=1 2024-04-06T09:47:45.540 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-06T09:47:45.561 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T09:47:45.562 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T09:47:45.562 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T09:47:45.562 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/test 2024-04-06T09:47:45.690 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:47:45.690 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:47:45.690 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:47:45.690 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:47:45.691 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:47:45.691 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:47:45.691 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:47:45.691 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:47:45.691 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:47:45.691 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:47:45.691 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:47:45.692 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:47:45.692 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:47:45.693 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:47:45.693 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:47:45.693 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:47:45.740 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 36 2024-04-06T09:47:45.740 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 36 2024-04-06T09:47:45.740 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:47:45.741 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=145, tag_tid=27, entry_tid=1], [object_number=144, tag_tid=27, entry_tid=0], [object_number=147, tag_tid=26, entry_tid=8935], [object_number=146, tag_tid=26, entry_tid=8934]]], state=connected] 2024-04-06T09:47:45.741 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=147, tag_tid=26, entry_tid=6383], [object_number=146, tag_tid=26, entry_tid=6382], [object_number=145, tag_tid=26, entry_tid=6381], [object_number=144, tag_tid=26, entry_tid=6380]]], state=connected] 2024-04-06T09:47:45.744 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:47:45.746 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=145, tag_tid=27, entry_tid=1' 2024-04-06T09:47:45.747 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:47:45.747 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:47:45.747 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:47:45.747 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:47:45.747 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:47:45.747 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:47:45.747 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:47:45.747 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:47:45.747 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:47:45.747 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:47:45.748 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:47:45.748 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:47:45.748 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:47:45.749 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:47:45.749 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:47:45.796 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 36 2024-04-06T09:47:45.796 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 36 2024-04-06T09:47:45.796 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:47:45.796 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=145, tag_tid=27, entry_tid=1], [object_number=144, tag_tid=27, entry_tid=0], [object_number=147, tag_tid=26, entry_tid=8935], [object_number=146, tag_tid=26, entry_tid=8934]]], state=connected] 2024-04-06T09:47:45.796 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=147, tag_tid=26, entry_tid=6383], [object_number=146, tag_tid=26, entry_tid=6382], [object_number=145, tag_tid=26, entry_tid=6381], [object_number=144, tag_tid=26, entry_tid=6380]]], state=connected] 2024-04-06T09:47:45.799 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:47:45.802 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=147, tag_tid=26, entry_tid=6383' 2024-04-06T09:47:45.802 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=145, tag_tid=27, entry_tid=1' -a 'object_number=145, tag_tid=27, entry_tid=1' = 'object_number=147, tag_tid=26, entry_tid=6383' 2024-04-06T09:47:45.802 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=147, tag_tid=26, entry_tid=6383' '!=' 'object_number=147, tag_tid=26, entry_tid=6263' 2024-04-06T09:47:45.802 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:47:45.802 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=147, tag_tid=26, entry_tid=6383' = 'object_number=147, tag_tid=26, entry_tid=6263' 2024-04-06T09:47:45.802 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=147, tag_tid=26, entry_tid=6383' 2024-04-06T09:47:45.802 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=145, tag_tid=27, entry_tid=1' 2024-04-06T09:47:45.802 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:47:45.802 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:47:45.803 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=27 2024-04-06T09:47:45.803 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=147, tag_tid=26, entry_tid=6383' 2024-04-06T09:47:45.804 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:47:45.804 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:47:45.804 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=26 2024-04-06T09:47:45.805 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=145, tag_tid=27, entry_tid=1' 2024-04-06T09:47:45.805 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:47:45.805 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:47:45.806 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:47:45.807 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=147, tag_tid=26, entry_tid=6383' 2024-04-06T09:47:45.807 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:47:45.807 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:47:45.808 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=6383 2024-04-06T09:47:45.808 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 27 = 26 -a 1 -le 6383 2024-04-06T09:47:45.808 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:47:45.808 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:47:45.808 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:47:46.010 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:47:46.010 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:47:46.010 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:47:46.011 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:47:46.011 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:47:46.011 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:47:46.011 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:47:46.011 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:47:46.011 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:47:46.011 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:47:46.011 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:47:46.011 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:47:46.011 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:47:46.011 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:47:46.011 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:47:46.013 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:47:46.013 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:47:46.013 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:47:46.013 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:47:46.013 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:47:46.013 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:47:46.013 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:47:46.013 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:47:46.013 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:47:46.013 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:47:46.014 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:47:46.014 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:47:46.014 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:47:46.014 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:47:46.014 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:47:46.014 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:47:46.014 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:47:46.014 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:47:46.015 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:47:46.015 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:47:46.015 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:47:46.015 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:47:46.016 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:47:46.016 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:47:46.016 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:47:46.016 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:47:46.016 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:47:46.016 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:47:46.016 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:47:46.016 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:47:46.016 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:47:46.016 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:47:46.016 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:47:46.016 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:47:46.039 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:47:46.040 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:47:46.040 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:47:46.040 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:47:46.148 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-06T09:47:46.148 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:rbd mirror flush : flush rbd mirror 2024-04-06T09:47:46.148 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:admin_socket: invalid command 2024-04-06T09:47:46.151 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:47:46.151 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/test 2024-04-06T09:47:46.151 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:1 2024-04-06T09:47:46.152 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:47:46.152 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:47:46.152 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-06T09:47:46.152 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:1 2024-04-06T09:47:46.152 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:47:46.152 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:47:46.152 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:47:46.152 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=1 2024-04-06T09:47:46.152 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-06T09:47:46.152 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:47:46.152 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:47:46.152 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=1 2024-04-06T09:47:46.152 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=1 2024-04-06T09:47:46.152 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-06T09:47:46.152 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:1 2024-04-06T09:47:46.152 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:47:46.153 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:47:46.153 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-06T09:47:46.153 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:1 2024-04-06T09:47:46.153 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:47:46.153 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:47:46.153 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:47:46.153 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=1 2024-04-06T09:47:46.153 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-06T09:47:46.153 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:47:46.153 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:47:46.153 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=1 2024-04-06T09:47:46.153 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=1 2024-04-06T09:47:46.154 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-06T09:47:46.176 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T09:47:46.176 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T09:47:46.177 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T09:47:46.177 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/test 2024-04-06T09:47:46.312 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:47:46.313 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:47:46.313 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:47:46.313 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:47:46.314 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:47:46.314 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:47:46.314 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:47:46.314 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:47:46.314 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:47:46.314 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:47:46.314 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:47:46.315 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:47:46.315 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:47:46.317 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:47:46.317 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:47:46.318 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:47:46.374 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 36 2024-04-06T09:47:46.374 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 36 2024-04-06T09:47:46.374 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:47:46.375 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=145, tag_tid=27, entry_tid=1], [object_number=144, tag_tid=27, entry_tid=0], [object_number=147, tag_tid=26, entry_tid=8935], [object_number=146, tag_tid=26, entry_tid=8934]]], state=connected] 2024-04-06T09:47:46.375 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=145, tag_tid=26, entry_tid=6509], [object_number=144, tag_tid=26, entry_tid=6508], [object_number=147, tag_tid=26, entry_tid=6507], [object_number=146, tag_tid=26, entry_tid=6506]]], state=connected] 2024-04-06T09:47:46.378 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:47:46.380 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=145, tag_tid=27, entry_tid=1' 2024-04-06T09:47:46.381 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:47:46.381 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:47:46.381 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:47:46.381 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:47:46.381 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:47:46.381 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:47:46.381 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:47:46.381 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:47:46.381 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:47:46.381 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:47:46.381 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:47:46.381 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:47:46.382 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:47:46.382 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:47:46.382 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:47:46.431 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 36 2024-04-06T09:47:46.432 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 36 2024-04-06T09:47:46.432 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:47:46.432 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=145, tag_tid=27, entry_tid=1], [object_number=144, tag_tid=27, entry_tid=0], [object_number=147, tag_tid=26, entry_tid=8935], [object_number=146, tag_tid=26, entry_tid=8934]]], state=connected] 2024-04-06T09:47:46.432 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=145, tag_tid=26, entry_tid=6509], [object_number=144, tag_tid=26, entry_tid=6508], [object_number=147, tag_tid=26, entry_tid=6507], [object_number=146, tag_tid=26, entry_tid=6506]]], state=connected] 2024-04-06T09:47:46.435 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:47:46.438 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=145, tag_tid=26, entry_tid=6509' 2024-04-06T09:47:46.438 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=145, tag_tid=27, entry_tid=1' -a 'object_number=145, tag_tid=27, entry_tid=1' = 'object_number=145, tag_tid=26, entry_tid=6509' 2024-04-06T09:47:46.438 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=145, tag_tid=26, entry_tid=6509' '!=' 'object_number=147, tag_tid=26, entry_tid=6383' 2024-04-06T09:47:46.438 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:47:46.438 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=145, tag_tid=26, entry_tid=6509' = 'object_number=147, tag_tid=26, entry_tid=6383' 2024-04-06T09:47:46.438 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=145, tag_tid=26, entry_tid=6509' 2024-04-06T09:47:46.438 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=145, tag_tid=27, entry_tid=1' 2024-04-06T09:47:46.438 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:47:46.438 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:47:46.439 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=27 2024-04-06T09:47:46.439 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=145, tag_tid=26, entry_tid=6509' 2024-04-06T09:47:46.439 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:47:46.439 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:47:46.440 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=26 2024-04-06T09:47:46.441 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=145, tag_tid=27, entry_tid=1' 2024-04-06T09:47:46.441 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:47:46.441 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:47:46.442 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:47:46.443 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=145, tag_tid=26, entry_tid=6509' 2024-04-06T09:47:46.443 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:47:46.443 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:47:46.444 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=6509 2024-04-06T09:47:46.444 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 27 = 26 -a 1 -le 6509 2024-04-06T09:47:46.444 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:47:46.444 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:47:46.444 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:47:46.646 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:47:46.646 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:47:46.646 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:47:46.646 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:47:46.646 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:47:46.646 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:47:46.647 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:47:46.647 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:47:46.647 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:47:46.647 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:47:46.647 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:47:46.647 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:47:46.647 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:47:46.647 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:47:46.647 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:47:46.647 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:47:46.648 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:47:46.651 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:47:46.651 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:47:46.652 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:47:46.652 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:47:46.652 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:47:46.652 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:47:46.652 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:47:46.652 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:47:46.652 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:47:46.652 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:47:46.652 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:47:46.652 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:47:46.652 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:47:46.653 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:47:46.653 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:47:46.653 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:47:46.653 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:47:46.653 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:47:46.653 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:47:46.653 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:47:46.653 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:47:46.653 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:47:46.653 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:47:46.653 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:47:46.653 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:47:46.654 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:47:46.654 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:47:46.654 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:47:46.654 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:47:46.654 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:47:46.654 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:47:46.654 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:47:46.679 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:47:46.679 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:47:46.679 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:47:46.679 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:47:46.783 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-06T09:47:46.783 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:rbd mirror flush : flush rbd mirror 2024-04-06T09:47:46.783 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:admin_socket: invalid command 2024-04-06T09:47:46.785 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:47:46.785 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/test 2024-04-06T09:47:46.785 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:1 2024-04-06T09:47:46.785 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:47:46.786 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:47:46.786 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-06T09:47:46.786 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:1 2024-04-06T09:47:46.786 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:47:46.786 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:47:46.786 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:47:46.786 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=1 2024-04-06T09:47:46.786 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-06T09:47:46.786 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:47:46.786 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:47:46.786 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=1 2024-04-06T09:47:46.786 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=1 2024-04-06T09:47:46.788 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-06T09:47:46.788 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:1 2024-04-06T09:47:46.788 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:47:46.788 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:47:46.788 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-06T09:47:46.788 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:1 2024-04-06T09:47:46.788 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:47:46.788 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:47:46.788 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:47:46.788 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=1 2024-04-06T09:47:46.789 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-06T09:47:46.789 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:47:46.789 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:47:46.789 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=1 2024-04-06T09:47:46.789 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=1 2024-04-06T09:47:46.789 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-06T09:47:46.812 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T09:47:46.812 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T09:47:46.812 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T09:47:46.812 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/test 2024-04-06T09:47:46.948 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:47:46.949 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:47:46.949 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:47:46.949 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:47:46.949 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:47:46.949 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:47:46.949 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:47:46.949 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:47:46.949 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:47:46.949 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:47:46.950 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:47:46.950 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:47:46.950 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:47:46.953 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:47:46.953 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:47:46.953 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:47:47.007 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 36 2024-04-06T09:47:47.007 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 36 2024-04-06T09:47:47.007 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:47:47.007 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=145, tag_tid=27, entry_tid=1], [object_number=144, tag_tid=27, entry_tid=0], [object_number=147, tag_tid=26, entry_tid=8935], [object_number=146, tag_tid=26, entry_tid=8934]]], state=connected] 2024-04-06T09:47:47.007 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=147, tag_tid=26, entry_tid=6631], [object_number=146, tag_tid=26, entry_tid=6630], [object_number=145, tag_tid=26, entry_tid=6629], [object_number=144, tag_tid=26, entry_tid=6628]]], state=connected] 2024-04-06T09:47:47.011 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:47:47.012 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=145, tag_tid=27, entry_tid=1' 2024-04-06T09:47:47.013 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:47:47.013 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:47:47.013 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:47:47.013 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:47:47.013 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:47:47.013 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:47:47.013 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:47:47.014 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:47:47.014 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:47:47.014 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:47:47.014 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:47:47.014 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:47:47.014 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:47:47.015 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:47:47.015 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:47:47.063 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 36 2024-04-06T09:47:47.063 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 36 2024-04-06T09:47:47.063 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:47:47.063 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=145, tag_tid=27, entry_tid=1], [object_number=144, tag_tid=27, entry_tid=0], [object_number=147, tag_tid=26, entry_tid=8935], [object_number=146, tag_tid=26, entry_tid=8934]]], state=connected] 2024-04-06T09:47:47.063 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=147, tag_tid=26, entry_tid=6631], [object_number=146, tag_tid=26, entry_tid=6630], [object_number=145, tag_tid=26, entry_tid=6629], [object_number=144, tag_tid=26, entry_tid=6628]]], state=connected] 2024-04-06T09:47:47.066 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:47:47.068 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=147, tag_tid=26, entry_tid=6631' 2024-04-06T09:47:47.068 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=145, tag_tid=27, entry_tid=1' -a 'object_number=145, tag_tid=27, entry_tid=1' = 'object_number=147, tag_tid=26, entry_tid=6631' 2024-04-06T09:47:47.068 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=147, tag_tid=26, entry_tid=6631' '!=' 'object_number=145, tag_tid=26, entry_tid=6509' 2024-04-06T09:47:47.068 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:47:47.068 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=147, tag_tid=26, entry_tid=6631' = 'object_number=145, tag_tid=26, entry_tid=6509' 2024-04-06T09:47:47.068 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=147, tag_tid=26, entry_tid=6631' 2024-04-06T09:47:47.069 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=145, tag_tid=27, entry_tid=1' 2024-04-06T09:47:47.069 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:47:47.069 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:47:47.069 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=27 2024-04-06T09:47:47.070 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=147, tag_tid=26, entry_tid=6631' 2024-04-06T09:47:47.070 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:47:47.070 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:47:47.071 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=26 2024-04-06T09:47:47.072 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=145, tag_tid=27, entry_tid=1' 2024-04-06T09:47:47.072 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:47:47.072 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:47:47.073 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:47:47.074 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=147, tag_tid=26, entry_tid=6631' 2024-04-06T09:47:47.074 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:47:47.074 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:47:47.075 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=6631 2024-04-06T09:47:47.075 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 27 = 26 -a 1 -le 6631 2024-04-06T09:47:47.075 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:47:47.075 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:47:47.075 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:47:47.277 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:47:47.277 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:47:47.277 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:47:47.277 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:47:47.277 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:47:47.277 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:47:47.277 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:47:47.277 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:47:47.278 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:47:47.278 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:47:47.278 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:47:47.278 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:47:47.278 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:47:47.278 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:47:47.278 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:47:47.278 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:47:47.279 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:47:47.281 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:47:47.281 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:47:47.282 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:47:47.282 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:47:47.282 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:47:47.282 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:47:47.282 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:47:47.282 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:47:47.282 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:47:47.282 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:47:47.282 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:47:47.282 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:47:47.282 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:47:47.282 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:47:47.282 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:47:47.282 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:47:47.283 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:47:47.283 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:47:47.284 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:47:47.284 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:47:47.284 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:47:47.284 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:47:47.284 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:47:47.284 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:47:47.284 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:47:47.284 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:47:47.284 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:47:47.284 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:47:47.284 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:47:47.284 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:47:47.284 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:47:47.284 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:47:47.309 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:47:47.310 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:47:47.310 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:47:47.310 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:47:47.416 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-06T09:47:47.416 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:rbd mirror flush : flush rbd mirror 2024-04-06T09:47:47.416 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:admin_socket: invalid command 2024-04-06T09:47:47.419 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:47:47.419 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/test 2024-04-06T09:47:47.419 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:1 2024-04-06T09:47:47.419 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:47:47.419 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:47:47.419 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-06T09:47:47.419 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:1 2024-04-06T09:47:47.419 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:47:47.420 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:47:47.420 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:47:47.420 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=1 2024-04-06T09:47:47.420 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-06T09:47:47.420 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:47:47.420 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:47:47.420 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=1 2024-04-06T09:47:47.420 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=1 2024-04-06T09:47:47.420 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-06T09:47:47.420 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:1 2024-04-06T09:47:47.420 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:47:47.420 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:47:47.420 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-06T09:47:47.420 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:1 2024-04-06T09:47:47.421 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:47:47.421 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:47:47.421 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:47:47.421 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=1 2024-04-06T09:47:47.421 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-06T09:47:47.421 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:47:47.421 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:47:47.421 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=1 2024-04-06T09:47:47.421 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=1 2024-04-06T09:47:47.421 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-06T09:47:47.444 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T09:47:47.444 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T09:47:47.445 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T09:47:47.445 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/test 2024-04-06T09:47:47.575 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:47:47.576 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:47:47.576 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:47:47.576 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:47:47.576 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:47:47.576 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:47:47.577 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:47:47.577 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:47:47.577 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:47:47.577 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:47:47.577 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:47:47.577 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:47:47.577 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:47:47.579 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:47:47.580 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:47:47.580 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:47:47.627 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 36 2024-04-06T09:47:47.627 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 36 2024-04-06T09:47:47.627 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:47:47.627 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=145, tag_tid=27, entry_tid=1], [object_number=144, tag_tid=27, entry_tid=0], [object_number=147, tag_tid=26, entry_tid=8935], [object_number=146, tag_tid=26, entry_tid=8934]]], state=connected] 2024-04-06T09:47:47.627 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=146, tag_tid=26, entry_tid=6758], [object_number=145, tag_tid=26, entry_tid=6757], [object_number=144, tag_tid=26, entry_tid=6756], [object_number=147, tag_tid=26, entry_tid=6755]]], state=connected] 2024-04-06T09:47:47.631 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:47:47.632 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=145, tag_tid=27, entry_tid=1' 2024-04-06T09:47:47.633 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:47:47.633 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:47:47.633 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:47:47.633 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:47:47.633 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:47:47.634 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:47:47.634 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:47:47.634 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:47:47.634 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:47:47.634 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:47:47.634 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:47:47.634 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:47:47.635 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:47:47.635 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:47:47.635 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:47:47.681 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 36 2024-04-06T09:47:47.682 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 36 2024-04-06T09:47:47.682 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:47:47.682 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=145, tag_tid=27, entry_tid=1], [object_number=144, tag_tid=27, entry_tid=0], [object_number=147, tag_tid=26, entry_tid=8935], [object_number=146, tag_tid=26, entry_tid=8934]]], state=connected] 2024-04-06T09:47:47.682 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=146, tag_tid=26, entry_tid=6758], [object_number=145, tag_tid=26, entry_tid=6757], [object_number=144, tag_tid=26, entry_tid=6756], [object_number=147, tag_tid=26, entry_tid=6755]]], state=connected] 2024-04-06T09:47:47.685 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:47:47.687 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=146, tag_tid=26, entry_tid=6758' 2024-04-06T09:47:47.687 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=145, tag_tid=27, entry_tid=1' -a 'object_number=145, tag_tid=27, entry_tid=1' = 'object_number=146, tag_tid=26, entry_tid=6758' 2024-04-06T09:47:47.687 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=146, tag_tid=26, entry_tid=6758' '!=' 'object_number=147, tag_tid=26, entry_tid=6631' 2024-04-06T09:47:47.687 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:47:47.687 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=146, tag_tid=26, entry_tid=6758' = 'object_number=147, tag_tid=26, entry_tid=6631' 2024-04-06T09:47:47.687 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=146, tag_tid=26, entry_tid=6758' 2024-04-06T09:47:47.688 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=145, tag_tid=27, entry_tid=1' 2024-04-06T09:47:47.688 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:47:47.688 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:47:47.688 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=27 2024-04-06T09:47:47.689 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=146, tag_tid=26, entry_tid=6758' 2024-04-06T09:47:47.689 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:47:47.689 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:47:47.690 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=26 2024-04-06T09:47:47.691 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=145, tag_tid=27, entry_tid=1' 2024-04-06T09:47:47.691 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:47:47.691 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:47:47.692 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:47:47.693 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=146, tag_tid=26, entry_tid=6758' 2024-04-06T09:47:47.693 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:47:47.693 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:47:47.694 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=6758 2024-04-06T09:47:47.694 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 27 = 26 -a 1 -le 6758 2024-04-06T09:47:47.694 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:47:47.694 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:47:47.694 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:47:47.896 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:47:47.896 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:47:47.896 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:47:47.896 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:47:47.896 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:47:47.896 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:47:47.896 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:47:47.896 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:47:47.896 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:47:47.897 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:47:47.897 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:47:47.897 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:47:47.897 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:47:47.897 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:47:47.897 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:47:47.897 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:47:47.898 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:47:47.900 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:47:47.900 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:47:47.900 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:47:47.900 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:47:47.900 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:47:47.900 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:47:47.900 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:47:47.900 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:47:47.900 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:47:47.900 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:47:47.900 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:47:47.901 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:47:47.901 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:47:47.901 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:47:47.901 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:47:47.901 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:47:47.902 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:47:47.902 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:47:47.902 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:47:47.903 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:47:47.903 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:47:47.903 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:47:47.903 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:47:47.903 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:47:47.903 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:47:47.903 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:47:47.903 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:47:47.903 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:47:47.903 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:47:47.903 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:47:47.903 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:47:47.903 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:47:47.929 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:47:47.929 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:47:47.929 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:47:47.929 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:47:48.035 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-06T09:47:48.036 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:rbd mirror flush : flush rbd mirror 2024-04-06T09:47:48.036 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:admin_socket: invalid command 2024-04-06T09:47:48.038 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:47:48.038 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/test 2024-04-06T09:47:48.038 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:1 2024-04-06T09:47:48.038 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:47:48.038 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:47:48.038 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-06T09:47:48.038 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:1 2024-04-06T09:47:48.038 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:47:48.038 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:47:48.038 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:47:48.038 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=1 2024-04-06T09:47:48.039 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-06T09:47:48.039 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:47:48.039 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:47:48.039 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=1 2024-04-06T09:47:48.039 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=1 2024-04-06T09:47:48.040 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-06T09:47:48.040 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:1 2024-04-06T09:47:48.040 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:47:48.040 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:47:48.040 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-06T09:47:48.040 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:1 2024-04-06T09:47:48.040 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:47:48.040 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:47:48.040 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:47:48.040 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=1 2024-04-06T09:47:48.040 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-06T09:47:48.040 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:47:48.040 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:47:48.041 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=1 2024-04-06T09:47:48.041 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=1 2024-04-06T09:47:48.041 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-06T09:47:48.065 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T09:47:48.065 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T09:47:48.065 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T09:47:48.065 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/test 2024-04-06T09:47:48.192 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:47:48.193 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:47:48.193 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:47:48.194 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:47:48.194 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:47:48.194 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:47:48.194 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:47:48.194 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:47:48.194 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:47:48.194 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:47:48.195 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:47:48.195 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:47:48.195 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:47:48.198 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:47:48.198 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:47:48.198 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:47:48.247 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 36 2024-04-06T09:47:48.247 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 36 2024-04-06T09:47:48.247 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:47:48.247 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=145, tag_tid=27, entry_tid=1], [object_number=144, tag_tid=27, entry_tid=0], [object_number=147, tag_tid=26, entry_tid=8935], [object_number=146, tag_tid=26, entry_tid=8934]]], state=connected] 2024-04-06T09:47:48.247 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=147, tag_tid=26, entry_tid=6883], [object_number=146, tag_tid=26, entry_tid=6882], [object_number=145, tag_tid=26, entry_tid=6881], [object_number=144, tag_tid=26, entry_tid=6880]]], state=connected] 2024-04-06T09:47:48.250 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:47:48.253 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=145, tag_tid=27, entry_tid=1' 2024-04-06T09:47:48.253 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:47:48.253 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:47:48.253 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:47:48.253 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:47:48.253 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:47:48.253 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:47:48.253 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:47:48.253 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:47:48.253 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:47:48.254 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:47:48.254 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:47:48.254 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:47:48.254 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:47:48.254 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:47:48.254 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:47:48.508 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 36 2024-04-06T09:47:48.508 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 36 2024-04-06T09:47:48.508 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:47:48.508 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=145, tag_tid=27, entry_tid=1], [object_number=144, tag_tid=27, entry_tid=0], [object_number=147, tag_tid=26, entry_tid=8935], [object_number=146, tag_tid=26, entry_tid=8934]]], state=connected] 2024-04-06T09:47:48.508 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=147, tag_tid=26, entry_tid=6883], [object_number=146, tag_tid=26, entry_tid=6882], [object_number=145, tag_tid=26, entry_tid=6881], [object_number=144, tag_tid=26, entry_tid=6880]]], state=connected] 2024-04-06T09:47:48.514 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:47:48.516 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=147, tag_tid=26, entry_tid=6883' 2024-04-06T09:47:48.516 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=145, tag_tid=27, entry_tid=1' -a 'object_number=145, tag_tid=27, entry_tid=1' = 'object_number=147, tag_tid=26, entry_tid=6883' 2024-04-06T09:47:48.516 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=147, tag_tid=26, entry_tid=6883' '!=' 'object_number=146, tag_tid=26, entry_tid=6758' 2024-04-06T09:47:48.516 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:47:48.517 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=147, tag_tid=26, entry_tid=6883' = 'object_number=146, tag_tid=26, entry_tid=6758' 2024-04-06T09:47:48.517 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=147, tag_tid=26, entry_tid=6883' 2024-04-06T09:47:48.517 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=145, tag_tid=27, entry_tid=1' 2024-04-06T09:47:48.517 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:47:48.517 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:47:48.519 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=27 2024-04-06T09:47:48.520 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=147, tag_tid=26, entry_tid=6883' 2024-04-06T09:47:48.520 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:47:48.520 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:47:48.521 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=26 2024-04-06T09:47:48.523 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=145, tag_tid=27, entry_tid=1' 2024-04-06T09:47:48.523 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:47:48.523 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:47:48.524 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:47:48.525 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=147, tag_tid=26, entry_tid=6883' 2024-04-06T09:47:48.525 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:47:48.525 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:47:48.527 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=6883 2024-04-06T09:47:48.527 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 27 = 26 -a 1 -le 6883 2024-04-06T09:47:48.527 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:47:48.527 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:47:48.528 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:47:48.729 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:47:48.729 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:47:48.729 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:47:48.729 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:47:48.729 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:47:48.729 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:47:48.730 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:47:48.730 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:47:48.730 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:47:48.730 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:47:48.730 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:47:48.730 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:47:48.730 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:47:48.730 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:47:48.730 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:47:48.731 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:47:48.731 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:47:48.735 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:47:48.735 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:47:48.735 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:47:48.735 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:47:48.735 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:47:48.735 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:47:48.735 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:47:48.735 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:47:48.735 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:47:48.735 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:47:48.735 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:47:48.735 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:47:48.735 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:47:48.735 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:47:48.736 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:47:48.736 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:47:48.736 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:47:48.736 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:47:48.736 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:47:48.736 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:47:48.736 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:47:48.736 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:47:48.736 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:47:48.736 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:47:48.736 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:47:48.736 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:47:48.736 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:47:48.736 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:47:48.736 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:47:48.737 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:47:48.737 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:47:48.737 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:47:48.760 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:47:48.760 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:47:48.760 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:47:48.760 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:47:48.867 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-06T09:47:48.867 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:rbd mirror flush : flush rbd mirror 2024-04-06T09:47:48.867 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:admin_socket: invalid command 2024-04-06T09:47:48.871 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:47:48.871 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/test 2024-04-06T09:47:48.871 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:1 2024-04-06T09:47:48.871 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:47:48.871 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:47:48.871 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-06T09:47:48.871 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:1 2024-04-06T09:47:48.871 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:47:48.871 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:47:48.872 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:47:48.872 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=1 2024-04-06T09:47:48.872 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-06T09:47:48.872 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:47:48.872 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:47:48.872 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=1 2024-04-06T09:47:48.872 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=1 2024-04-06T09:47:48.872 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-06T09:47:48.872 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:1 2024-04-06T09:47:48.872 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:47:48.872 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:47:48.872 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-06T09:47:48.872 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:1 2024-04-06T09:47:48.873 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:47:48.873 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:47:48.873 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:47:48.873 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=1 2024-04-06T09:47:48.873 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-06T09:47:48.873 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:47:48.873 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:47:48.873 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=1 2024-04-06T09:47:48.873 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=1 2024-04-06T09:47:48.873 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-06T09:47:48.896 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T09:47:48.896 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T09:47:48.896 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T09:47:48.896 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/test 2024-04-06T09:47:49.020 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:47:49.021 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:47:49.021 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:47:49.021 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:47:49.021 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:47:49.021 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:47:49.021 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:47:49.022 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:47:49.022 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:47:49.022 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:47:49.022 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:47:49.022 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:47:49.022 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:47:49.025 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:47:49.025 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:47:49.025 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:47:49.073 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 36 2024-04-06T09:47:49.074 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 36 2024-04-06T09:47:49.074 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:47:49.074 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=145, tag_tid=27, entry_tid=1], [object_number=144, tag_tid=27, entry_tid=0], [object_number=147, tag_tid=26, entry_tid=8935], [object_number=146, tag_tid=26, entry_tid=8934]]], state=connected] 2024-04-06T09:47:49.074 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=144, tag_tid=26, entry_tid=7024], [object_number=147, tag_tid=26, entry_tid=7023], [object_number=146, tag_tid=26, entry_tid=7022], [object_number=145, tag_tid=26, entry_tid=7021]]], state=connected] 2024-04-06T09:47:49.077 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:47:49.079 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=145, tag_tid=27, entry_tid=1' 2024-04-06T09:47:49.080 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:47:49.080 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:47:49.080 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:47:49.080 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:47:49.080 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:47:49.080 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:47:49.080 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:47:49.080 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:47:49.080 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:47:49.080 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:47:49.080 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:47:49.080 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:47:49.081 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:47:49.081 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:47:49.081 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:47:49.128 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 36 2024-04-06T09:47:49.128 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 36 2024-04-06T09:47:49.128 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:47:49.128 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=145, tag_tid=27, entry_tid=1], [object_number=144, tag_tid=27, entry_tid=0], [object_number=147, tag_tid=26, entry_tid=8935], [object_number=146, tag_tid=26, entry_tid=8934]]], state=connected] 2024-04-06T09:47:49.128 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=144, tag_tid=26, entry_tid=7024], [object_number=147, tag_tid=26, entry_tid=7023], [object_number=146, tag_tid=26, entry_tid=7022], [object_number=145, tag_tid=26, entry_tid=7021]]], state=connected] 2024-04-06T09:47:49.132 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:47:49.134 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=144, tag_tid=26, entry_tid=7024' 2024-04-06T09:47:49.134 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=145, tag_tid=27, entry_tid=1' -a 'object_number=145, tag_tid=27, entry_tid=1' = 'object_number=144, tag_tid=26, entry_tid=7024' 2024-04-06T09:47:49.134 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=144, tag_tid=26, entry_tid=7024' '!=' 'object_number=147, tag_tid=26, entry_tid=6883' 2024-04-06T09:47:49.134 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:47:49.134 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=144, tag_tid=26, entry_tid=7024' = 'object_number=147, tag_tid=26, entry_tid=6883' 2024-04-06T09:47:49.134 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=144, tag_tid=26, entry_tid=7024' 2024-04-06T09:47:49.134 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=145, tag_tid=27, entry_tid=1' 2024-04-06T09:47:49.135 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:47:49.135 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:47:49.135 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=27 2024-04-06T09:47:49.136 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=144, tag_tid=26, entry_tid=7024' 2024-04-06T09:47:49.136 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:47:49.136 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:47:49.137 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=26 2024-04-06T09:47:49.137 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=145, tag_tid=27, entry_tid=1' 2024-04-06T09:47:49.137 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:47:49.137 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:47:49.138 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:47:49.139 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=144, tag_tid=26, entry_tid=7024' 2024-04-06T09:47:49.139 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:47:49.139 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:47:49.140 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=7024 2024-04-06T09:47:49.140 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 27 = 26 -a 1 -le 7024 2024-04-06T09:47:49.140 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:47:49.141 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:47:49.141 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:47:49.342 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:47:49.342 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:47:49.342 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:47:49.342 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:47:49.342 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:47:49.342 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:47:49.343 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:47:49.343 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:47:49.343 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:47:49.343 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:47:49.343 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:47:49.343 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:47:49.343 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:47:49.343 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:47:49.343 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:47:49.344 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:47:49.344 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:47:49.346 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:47:49.346 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:47:49.346 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:47:49.346 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:47:49.346 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:47:49.346 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:47:49.346 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:47:49.346 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:47:49.347 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:47:49.347 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:47:49.347 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:47:49.347 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:47:49.347 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:47:49.347 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:47:49.347 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:47:49.347 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:47:49.347 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:47:49.347 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:47:49.347 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:47:49.347 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:47:49.347 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:47:49.347 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:47:49.348 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:47:49.348 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:47:49.348 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:47:49.348 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:47:49.348 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:47:49.348 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:47:49.348 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:47:49.348 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:47:49.348 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:47:49.348 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:47:49.371 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:47:49.372 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:47:49.372 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:47:49.372 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:47:49.478 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-06T09:47:49.479 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:rbd mirror flush : flush rbd mirror 2024-04-06T09:47:49.479 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:admin_socket: invalid command 2024-04-06T09:47:49.481 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:47:49.481 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/test 2024-04-06T09:47:49.481 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:1 2024-04-06T09:47:49.481 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:47:49.481 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:47:49.481 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-06T09:47:49.481 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:1 2024-04-06T09:47:49.481 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:47:49.482 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:47:49.482 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:47:49.482 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=1 2024-04-06T09:47:49.482 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-06T09:47:49.482 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:47:49.482 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:47:49.482 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=1 2024-04-06T09:47:49.482 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=1 2024-04-06T09:47:49.483 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-06T09:47:49.483 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:1 2024-04-06T09:47:49.483 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:47:49.483 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:47:49.484 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-06T09:47:49.484 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:1 2024-04-06T09:47:49.484 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:47:49.484 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:47:49.484 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:47:49.484 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=1 2024-04-06T09:47:49.484 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-06T09:47:49.484 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:47:49.484 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:47:49.484 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=1 2024-04-06T09:47:49.484 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=1 2024-04-06T09:47:49.484 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-06T09:47:49.507 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T09:47:49.508 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T09:47:49.508 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T09:47:49.508 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/test 2024-04-06T09:47:49.630 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:47:49.631 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:47:49.631 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:47:49.631 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:47:49.631 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:47:49.632 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:47:49.632 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:47:49.632 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:47:49.632 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:47:49.632 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:47:49.632 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:47:49.633 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:47:49.633 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:47:49.635 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:47:49.635 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:47:49.635 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:47:49.683 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 36 2024-04-06T09:47:49.683 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 36 2024-04-06T09:47:49.683 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:47:49.683 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=145, tag_tid=27, entry_tid=1], [object_number=144, tag_tid=27, entry_tid=0], [object_number=147, tag_tid=26, entry_tid=8935], [object_number=146, tag_tid=26, entry_tid=8934]]], state=connected] 2024-04-06T09:47:49.684 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=147, tag_tid=26, entry_tid=7123], [object_number=146, tag_tid=26, entry_tid=7122], [object_number=145, tag_tid=26, entry_tid=7121], [object_number=144, tag_tid=26, entry_tid=7120]]], state=connected] 2024-04-06T09:47:49.687 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:47:49.688 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=145, tag_tid=27, entry_tid=1' 2024-04-06T09:47:49.689 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:47:49.689 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:47:49.689 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:47:49.689 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:47:49.689 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:47:49.689 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:47:49.689 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:47:49.690 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:47:49.690 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:47:49.690 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:47:49.690 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:47:49.690 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:47:49.690 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:47:49.691 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:47:49.691 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:47:49.738 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 36 2024-04-06T09:47:49.738 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 36 2024-04-06T09:47:49.738 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:47:49.738 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=145, tag_tid=27, entry_tid=1], [object_number=144, tag_tid=27, entry_tid=0], [object_number=147, tag_tid=26, entry_tid=8935], [object_number=146, tag_tid=26, entry_tid=8934]]], state=connected] 2024-04-06T09:47:49.738 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=147, tag_tid=26, entry_tid=7123], [object_number=146, tag_tid=26, entry_tid=7122], [object_number=145, tag_tid=26, entry_tid=7121], [object_number=144, tag_tid=26, entry_tid=7120]]], state=connected] 2024-04-06T09:47:49.742 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:47:49.744 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=147, tag_tid=26, entry_tid=7123' 2024-04-06T09:47:49.744 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=145, tag_tid=27, entry_tid=1' -a 'object_number=145, tag_tid=27, entry_tid=1' = 'object_number=147, tag_tid=26, entry_tid=7123' 2024-04-06T09:47:49.744 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=147, tag_tid=26, entry_tid=7123' '!=' 'object_number=144, tag_tid=26, entry_tid=7024' 2024-04-06T09:47:49.744 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:47:49.744 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=147, tag_tid=26, entry_tid=7123' = 'object_number=144, tag_tid=26, entry_tid=7024' 2024-04-06T09:47:49.744 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=147, tag_tid=26, entry_tid=7123' 2024-04-06T09:47:49.744 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=145, tag_tid=27, entry_tid=1' 2024-04-06T09:47:49.744 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:47:49.745 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:47:49.745 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=27 2024-04-06T09:47:49.746 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=147, tag_tid=26, entry_tid=7123' 2024-04-06T09:47:49.746 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:47:49.746 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:47:49.747 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=26 2024-04-06T09:47:49.748 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=145, tag_tid=27, entry_tid=1' 2024-04-06T09:47:49.748 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:47:49.748 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:47:49.748 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:47:49.749 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=147, tag_tid=26, entry_tid=7123' 2024-04-06T09:47:49.749 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:47:49.749 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:47:49.750 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=7123 2024-04-06T09:47:49.751 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 27 = 26 -a 1 -le 7123 2024-04-06T09:47:49.751 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:47:49.751 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:47:49.751 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:47:49.952 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:47:49.952 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:47:49.952 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:47:49.953 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:47:49.953 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:47:49.953 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:47:49.953 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:47:49.953 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:47:49.953 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:47:49.953 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:47:49.953 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:47:49.953 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:47:49.953 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:47:49.953 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:47:49.953 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:47:49.954 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:47:49.954 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:47:49.956 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:47:49.956 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:47:49.956 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:47:49.956 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:47:49.957 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:47:49.957 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:47:49.957 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:47:49.957 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:47:49.957 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:47:49.957 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:47:49.957 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:47:49.957 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:47:49.957 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:47:49.957 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:47:49.957 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:47:49.957 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:47:49.958 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:47:49.958 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:47:49.959 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:47:49.959 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:47:49.959 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:47:49.959 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:47:49.959 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:47:49.959 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:47:49.959 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:47:49.959 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:47:49.959 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:47:49.959 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:47:49.959 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:47:49.960 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:47:49.960 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:47:49.960 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:47:49.982 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:47:49.983 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:47:49.983 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:47:49.983 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:47:50.089 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-06T09:47:50.089 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:rbd mirror flush : flush rbd mirror 2024-04-06T09:47:50.089 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:admin_socket: invalid command 2024-04-06T09:47:50.091 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:47:50.091 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/test 2024-04-06T09:47:50.091 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:1 2024-04-06T09:47:50.092 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:47:50.092 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:47:50.092 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-06T09:47:50.092 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:1 2024-04-06T09:47:50.092 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:47:50.092 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:47:50.092 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:47:50.092 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=1 2024-04-06T09:47:50.092 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-06T09:47:50.092 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:47:50.092 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:47:50.092 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=1 2024-04-06T09:47:50.092 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=1 2024-04-06T09:47:50.093 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-06T09:47:50.094 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:1 2024-04-06T09:47:50.094 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:47:50.094 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:47:50.094 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-06T09:47:50.094 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:1 2024-04-06T09:47:50.094 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:47:50.094 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:47:50.094 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:47:50.094 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=1 2024-04-06T09:47:50.094 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-06T09:47:50.094 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:47:50.094 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:47:50.094 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=1 2024-04-06T09:47:50.095 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=1 2024-04-06T09:47:50.095 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-06T09:47:50.119 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T09:47:50.119 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T09:47:50.119 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T09:47:50.119 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/test 2024-04-06T09:47:50.243 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:47:50.245 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:47:50.245 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:47:50.245 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:47:50.245 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:47:50.245 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:47:50.245 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:47:50.245 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:47:50.245 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:47:50.245 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:47:50.246 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:47:50.246 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:47:50.246 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:47:50.248 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:47:50.248 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:47:50.248 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:47:50.301 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 36 2024-04-06T09:47:50.301 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 36 2024-04-06T09:47:50.301 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:47:50.302 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=145, tag_tid=27, entry_tid=1], [object_number=144, tag_tid=27, entry_tid=0], [object_number=147, tag_tid=26, entry_tid=8935], [object_number=146, tag_tid=26, entry_tid=8934]]], state=connected] 2024-04-06T09:47:50.302 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=144, tag_tid=26, entry_tid=7240], [object_number=147, tag_tid=26, entry_tid=7239], [object_number=146, tag_tid=26, entry_tid=7238], [object_number=145, tag_tid=26, entry_tid=7237]]], state=connected] 2024-04-06T09:47:50.305 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:47:50.306 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=145, tag_tid=27, entry_tid=1' 2024-04-06T09:47:50.307 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:47:50.308 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:47:50.308 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:47:50.308 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:47:50.308 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:47:50.308 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:47:50.308 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:47:50.308 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:47:50.308 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:47:50.308 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:47:50.308 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:47:50.308 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:47:50.309 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:47:50.309 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:47:50.309 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:47:50.356 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 36 2024-04-06T09:47:50.356 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 36 2024-04-06T09:47:50.356 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:47:50.356 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=145, tag_tid=27, entry_tid=1], [object_number=144, tag_tid=27, entry_tid=0], [object_number=147, tag_tid=26, entry_tid=8935], [object_number=146, tag_tid=26, entry_tid=8934]]], state=connected] 2024-04-06T09:47:50.357 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=144, tag_tid=26, entry_tid=7240], [object_number=147, tag_tid=26, entry_tid=7239], [object_number=146, tag_tid=26, entry_tid=7238], [object_number=145, tag_tid=26, entry_tid=7237]]], state=connected] 2024-04-06T09:47:50.360 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:47:50.362 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=144, tag_tid=26, entry_tid=7240' 2024-04-06T09:47:50.362 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=145, tag_tid=27, entry_tid=1' -a 'object_number=145, tag_tid=27, entry_tid=1' = 'object_number=144, tag_tid=26, entry_tid=7240' 2024-04-06T09:47:50.362 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=144, tag_tid=26, entry_tid=7240' '!=' 'object_number=147, tag_tid=26, entry_tid=7123' 2024-04-06T09:47:50.362 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:47:50.362 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=144, tag_tid=26, entry_tid=7240' = 'object_number=147, tag_tid=26, entry_tid=7123' 2024-04-06T09:47:50.362 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=144, tag_tid=26, entry_tid=7240' 2024-04-06T09:47:50.362 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=145, tag_tid=27, entry_tid=1' 2024-04-06T09:47:50.362 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:47:50.362 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:47:50.363 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=27 2024-04-06T09:47:50.364 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=144, tag_tid=26, entry_tid=7240' 2024-04-06T09:47:50.364 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:47:50.364 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:47:50.365 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=26 2024-04-06T09:47:50.365 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=145, tag_tid=27, entry_tid=1' 2024-04-06T09:47:50.365 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:47:50.365 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:47:50.367 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:47:50.367 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=144, tag_tid=26, entry_tid=7240' 2024-04-06T09:47:50.367 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:47:50.367 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:47:50.368 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=7240 2024-04-06T09:47:50.369 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 27 = 26 -a 1 -le 7240 2024-04-06T09:47:50.369 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:47:50.369 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:47:50.369 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:47:50.570 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:47:50.570 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:47:50.570 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:47:50.570 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:47:50.570 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:47:50.570 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:47:50.570 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:47:50.570 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:47:50.570 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:47:50.571 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:47:50.571 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:47:50.571 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:47:50.571 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:47:50.571 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:47:50.571 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:47:50.571 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:47:50.571 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:47:50.573 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:47:50.573 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:47:50.574 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:47:50.574 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:47:50.574 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:47:50.574 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:47:50.574 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:47:50.574 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:47:50.574 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:47:50.574 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:47:50.574 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:47:50.574 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:47:50.574 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:47:50.574 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:47:50.574 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:47:50.575 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:47:50.576 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:47:50.576 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:47:50.576 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:47:50.576 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:47:50.576 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:47:50.576 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:47:50.576 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:47:50.576 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:47:50.576 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:47:50.577 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:47:50.577 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:47:50.577 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:47:50.577 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:47:50.577 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:47:50.577 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:47:50.577 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:47:50.600 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:47:50.600 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:47:50.600 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:47:50.600 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:47:50.704 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-06T09:47:50.704 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:rbd mirror flush : flush rbd mirror 2024-04-06T09:47:50.704 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:admin_socket: invalid command 2024-04-06T09:47:50.707 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:47:50.707 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/test 2024-04-06T09:47:50.707 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:1 2024-04-06T09:47:50.707 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:47:50.707 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:47:50.707 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-06T09:47:50.707 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:1 2024-04-06T09:47:50.707 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:47:50.707 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:47:50.707 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:47:50.707 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=1 2024-04-06T09:47:50.707 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-06T09:47:50.708 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:47:50.708 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:47:50.708 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=1 2024-04-06T09:47:50.708 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=1 2024-04-06T09:47:50.708 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-06T09:47:50.708 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:1 2024-04-06T09:47:50.708 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:47:50.708 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:47:50.708 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-06T09:47:50.708 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:1 2024-04-06T09:47:50.708 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:47:50.708 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:47:50.708 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:47:50.708 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=1 2024-04-06T09:47:50.708 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-06T09:47:50.708 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:47:50.709 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:47:50.709 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=1 2024-04-06T09:47:50.709 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=1 2024-04-06T09:47:50.709 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-06T09:47:50.732 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T09:47:50.732 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T09:47:50.732 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T09:47:50.733 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/test 2024-04-06T09:47:50.870 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:47:50.871 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:47:50.871 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:47:50.871 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:47:50.871 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:47:50.871 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:47:50.871 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:47:50.871 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:47:50.871 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:47:50.871 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:47:50.872 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:47:50.872 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:47:50.872 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:47:50.874 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:47:50.874 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:47:50.874 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:47:50.927 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 36 2024-04-06T09:47:50.927 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 36 2024-04-06T09:47:50.927 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:47:50.927 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=145, tag_tid=27, entry_tid=1], [object_number=144, tag_tid=27, entry_tid=0], [object_number=147, tag_tid=26, entry_tid=8935], [object_number=146, tag_tid=26, entry_tid=8934]]], state=connected] 2024-04-06T09:47:50.927 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=144, tag_tid=26, entry_tid=7360], [object_number=147, tag_tid=26, entry_tid=7359], [object_number=146, tag_tid=26, entry_tid=7358], [object_number=145, tag_tid=26, entry_tid=7357]]], state=connected] 2024-04-06T09:47:50.931 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:47:50.933 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=145, tag_tid=27, entry_tid=1' 2024-04-06T09:47:50.933 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:47:50.933 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:47:50.933 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:47:50.933 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:47:50.933 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:47:50.934 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:47:50.934 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:47:50.934 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:47:50.934 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:47:50.934 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:47:50.934 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:47:50.935 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:47:50.935 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:47:50.935 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:47:50.935 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:47:50.983 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 36 2024-04-06T09:47:50.983 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 36 2024-04-06T09:47:50.983 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:47:50.983 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=145, tag_tid=27, entry_tid=1], [object_number=144, tag_tid=27, entry_tid=0], [object_number=147, tag_tid=26, entry_tid=8935], [object_number=146, tag_tid=26, entry_tid=8934]]], state=connected] 2024-04-06T09:47:50.984 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=144, tag_tid=26, entry_tid=7360], [object_number=147, tag_tid=26, entry_tid=7359], [object_number=146, tag_tid=26, entry_tid=7358], [object_number=145, tag_tid=26, entry_tid=7357]]], state=connected] 2024-04-06T09:47:50.987 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:47:50.989 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=144, tag_tid=26, entry_tid=7360' 2024-04-06T09:47:50.990 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=145, tag_tid=27, entry_tid=1' -a 'object_number=145, tag_tid=27, entry_tid=1' = 'object_number=144, tag_tid=26, entry_tid=7360' 2024-04-06T09:47:50.990 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=144, tag_tid=26, entry_tid=7360' '!=' 'object_number=144, tag_tid=26, entry_tid=7240' 2024-04-06T09:47:50.990 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:47:50.990 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=144, tag_tid=26, entry_tid=7360' = 'object_number=144, tag_tid=26, entry_tid=7240' 2024-04-06T09:47:50.990 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=144, tag_tid=26, entry_tid=7360' 2024-04-06T09:47:50.990 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=145, tag_tid=27, entry_tid=1' 2024-04-06T09:47:50.990 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:47:50.990 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:47:50.991 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=27 2024-04-06T09:47:50.991 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=144, tag_tid=26, entry_tid=7360' 2024-04-06T09:47:50.991 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:47:50.992 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:47:50.992 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=26 2024-04-06T09:47:50.993 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=145, tag_tid=27, entry_tid=1' 2024-04-06T09:47:50.993 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:47:50.993 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:47:50.994 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:47:50.995 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=144, tag_tid=26, entry_tid=7360' 2024-04-06T09:47:50.995 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:47:50.995 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:47:50.996 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=7360 2024-04-06T09:47:50.996 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 27 = 26 -a 1 -le 7360 2024-04-06T09:47:50.996 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:47:50.997 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:47:50.997 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:47:51.198 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:47:51.198 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:47:51.198 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:47:51.198 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:47:51.199 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:47:51.199 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:47:51.199 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:47:51.199 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:47:51.199 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:47:51.199 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:47:51.199 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:47:51.199 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:47:51.199 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:47:51.199 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:47:51.199 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:47:51.200 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:47:51.200 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:47:51.202 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:47:51.203 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:47:51.203 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:47:51.203 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:47:51.203 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:47:51.203 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:47:51.203 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:47:51.203 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:47:51.203 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:47:51.203 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:47:51.203 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:47:51.203 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:47:51.203 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:47:51.203 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:47:51.203 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:47:51.204 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:47:51.204 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:47:51.205 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:47:51.205 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:47:51.205 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:47:51.205 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:47:51.205 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:47:51.205 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:47:51.205 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:47:51.205 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:47:51.205 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:47:51.205 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:47:51.205 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:47:51.205 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:47:51.205 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:47:51.205 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:47:51.206 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:47:51.229 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:47:51.230 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:47:51.230 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:47:51.230 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:47:51.336 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-06T09:47:51.336 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:rbd mirror flush : flush rbd mirror 2024-04-06T09:47:51.336 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:admin_socket: invalid command 2024-04-06T09:47:51.338 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:47:51.339 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/test 2024-04-06T09:47:51.339 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:1 2024-04-06T09:47:51.339 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:47:51.339 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:47:51.339 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-06T09:47:51.339 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:1 2024-04-06T09:47:51.339 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:47:51.339 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:47:51.339 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:47:51.339 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=1 2024-04-06T09:47:51.339 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-06T09:47:51.339 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:47:51.339 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:47:51.339 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=1 2024-04-06T09:47:51.339 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=1 2024-04-06T09:47:51.341 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-06T09:47:51.341 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:1 2024-04-06T09:47:51.341 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:47:51.341 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:47:51.341 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-06T09:47:51.341 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:1 2024-04-06T09:47:51.341 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:47:51.341 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:47:51.341 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:47:51.341 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=1 2024-04-06T09:47:51.341 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-06T09:47:51.341 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:47:51.341 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:47:51.342 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=1 2024-04-06T09:47:51.342 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=1 2024-04-06T09:47:51.342 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-06T09:47:51.365 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T09:47:51.365 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T09:47:51.365 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T09:47:51.365 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/test 2024-04-06T09:47:51.501 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:47:51.502 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:47:51.502 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:47:51.502 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:47:51.502 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:47:51.502 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:47:51.502 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:47:51.502 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:47:51.502 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:47:51.502 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:47:51.503 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:47:51.503 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:47:51.503 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:47:51.506 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:47:51.506 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:47:51.506 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:47:51.562 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 36 2024-04-06T09:47:51.562 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 36 2024-04-06T09:47:51.562 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:47:51.562 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=145, tag_tid=27, entry_tid=1], [object_number=144, tag_tid=27, entry_tid=0], [object_number=147, tag_tid=26, entry_tid=8935], [object_number=146, tag_tid=26, entry_tid=8934]]], state=connected] 2024-04-06T09:47:51.562 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=144, tag_tid=26, entry_tid=7444], [object_number=147, tag_tid=26, entry_tid=7443], [object_number=146, tag_tid=26, entry_tid=7442], [object_number=145, tag_tid=26, entry_tid=7441]]], state=connected] 2024-04-06T09:47:51.566 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:47:51.567 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=145, tag_tid=27, entry_tid=1' 2024-04-06T09:47:51.568 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:47:51.568 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:47:51.568 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:47:51.568 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:47:51.569 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:47:51.569 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:47:51.569 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:47:51.569 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:47:51.569 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:47:51.569 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:47:51.569 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:47:51.569 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:47:51.569 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:47:51.570 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:47:51.570 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:47:51.620 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 36 2024-04-06T09:47:51.620 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 36 2024-04-06T09:47:51.620 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:47:51.620 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=145, tag_tid=27, entry_tid=1], [object_number=144, tag_tid=27, entry_tid=0], [object_number=147, tag_tid=26, entry_tid=8935], [object_number=146, tag_tid=26, entry_tid=8934]]], state=connected] 2024-04-06T09:47:51.620 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=144, tag_tid=26, entry_tid=7444], [object_number=147, tag_tid=26, entry_tid=7443], [object_number=146, tag_tid=26, entry_tid=7442], [object_number=145, tag_tid=26, entry_tid=7441]]], state=connected] 2024-04-06T09:47:51.623 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:47:51.625 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=144, tag_tid=26, entry_tid=7444' 2024-04-06T09:47:51.625 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=145, tag_tid=27, entry_tid=1' -a 'object_number=145, tag_tid=27, entry_tid=1' = 'object_number=144, tag_tid=26, entry_tid=7444' 2024-04-06T09:47:51.625 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=144, tag_tid=26, entry_tid=7444' '!=' 'object_number=144, tag_tid=26, entry_tid=7360' 2024-04-06T09:47:51.625 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:47:51.626 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=144, tag_tid=26, entry_tid=7444' = 'object_number=144, tag_tid=26, entry_tid=7360' 2024-04-06T09:47:51.626 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=144, tag_tid=26, entry_tid=7444' 2024-04-06T09:47:51.626 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=145, tag_tid=27, entry_tid=1' 2024-04-06T09:47:51.626 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:47:51.626 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:47:51.627 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=27 2024-04-06T09:47:51.627 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=144, tag_tid=26, entry_tid=7444' 2024-04-06T09:47:51.628 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:47:51.628 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:47:51.629 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=26 2024-04-06T09:47:51.629 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=145, tag_tid=27, entry_tid=1' 2024-04-06T09:47:51.629 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:47:51.629 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:47:51.630 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:47:51.631 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=144, tag_tid=26, entry_tid=7444' 2024-04-06T09:47:51.631 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:47:51.631 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:47:51.633 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=7444 2024-04-06T09:47:51.633 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 27 = 26 -a 1 -le 7444 2024-04-06T09:47:51.633 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:47:51.633 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:47:51.633 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:47:51.834 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:47:51.834 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:47:51.834 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:47:51.834 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:47:51.835 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:47:51.835 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:47:51.835 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:47:51.835 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:47:51.835 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:47:51.835 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:47:51.835 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:47:51.835 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:47:51.835 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:47:51.835 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:47:51.835 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:47:51.836 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:47:51.836 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:47:51.839 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:47:51.839 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:47:51.839 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:47:51.839 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:47:51.839 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:47:51.839 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:47:51.840 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:47:51.840 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:47:51.840 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:47:51.840 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:47:51.840 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:47:51.840 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:47:51.840 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:47:51.840 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:47:51.840 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:47:51.840 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:47:51.841 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:47:51.841 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:47:51.841 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:47:51.841 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:47:51.841 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:47:51.841 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:47:51.841 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:47:51.841 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:47:51.841 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:47:51.842 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:47:51.842 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:47:51.842 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:47:51.842 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:47:51.842 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:47:51.842 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:47:51.842 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:47:51.866 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:47:51.867 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:47:51.867 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:47:51.867 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:47:51.975 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-06T09:47:51.975 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:rbd mirror flush : flush rbd mirror 2024-04-06T09:47:51.975 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:admin_socket: invalid command 2024-04-06T09:47:51.978 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:47:51.978 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/test 2024-04-06T09:47:51.978 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:1 2024-04-06T09:47:51.978 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:47:51.978 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:47:51.978 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-06T09:47:51.978 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:1 2024-04-06T09:47:51.978 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:47:51.978 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:47:51.978 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:47:51.978 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=1 2024-04-06T09:47:51.978 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-06T09:47:51.979 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:47:51.979 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:47:51.979 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=1 2024-04-06T09:47:51.979 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=1 2024-04-06T09:47:51.980 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-06T09:47:51.980 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:1 2024-04-06T09:47:51.980 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:47:51.980 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:47:51.980 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-06T09:47:51.980 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:1 2024-04-06T09:47:51.980 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:47:51.980 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:47:51.980 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:47:51.980 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=1 2024-04-06T09:47:51.980 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-06T09:47:51.980 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:47:51.981 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:47:51.981 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=1 2024-04-06T09:47:51.981 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=1 2024-04-06T09:47:51.981 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-06T09:47:52.004 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T09:47:52.004 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T09:47:52.005 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T09:47:52.005 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/test 2024-04-06T09:47:52.140 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:47:52.140 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:47:52.140 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:47:52.140 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:47:52.140 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:47:52.141 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:47:52.141 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:47:52.141 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:47:52.141 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:47:52.141 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:47:52.141 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:47:52.141 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:47:52.142 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:47:52.143 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:47:52.143 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:47:52.143 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:47:52.194 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 36 2024-04-06T09:47:52.194 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 36 2024-04-06T09:47:52.195 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:47:52.195 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=145, tag_tid=27, entry_tid=1], [object_number=144, tag_tid=27, entry_tid=0], [object_number=147, tag_tid=26, entry_tid=8935], [object_number=146, tag_tid=26, entry_tid=8934]]], state=connected] 2024-04-06T09:47:52.195 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=144, tag_tid=26, entry_tid=7572], [object_number=147, tag_tid=26, entry_tid=7571], [object_number=146, tag_tid=26, entry_tid=7570], [object_number=145, tag_tid=26, entry_tid=7569]]], state=connected] 2024-04-06T09:47:52.198 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:47:52.199 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=145, tag_tid=27, entry_tid=1' 2024-04-06T09:47:52.200 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:47:52.200 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:47:52.200 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:47:52.201 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:47:52.201 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:47:52.201 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:47:52.201 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:47:52.201 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:47:52.201 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:47:52.201 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:47:52.201 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:47:52.201 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:47:52.202 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:47:52.202 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:47:52.202 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:47:52.250 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 36 2024-04-06T09:47:52.250 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 36 2024-04-06T09:47:52.250 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:47:52.250 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=145, tag_tid=27, entry_tid=1], [object_number=144, tag_tid=27, entry_tid=0], [object_number=147, tag_tid=26, entry_tid=8935], [object_number=146, tag_tid=26, entry_tid=8934]]], state=connected] 2024-04-06T09:47:52.250 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=144, tag_tid=26, entry_tid=7572], [object_number=147, tag_tid=26, entry_tid=7571], [object_number=146, tag_tid=26, entry_tid=7570], [object_number=145, tag_tid=26, entry_tid=7569]]], state=connected] 2024-04-06T09:47:52.253 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:47:52.255 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=144, tag_tid=26, entry_tid=7572' 2024-04-06T09:47:52.256 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=145, tag_tid=27, entry_tid=1' -a 'object_number=145, tag_tid=27, entry_tid=1' = 'object_number=144, tag_tid=26, entry_tid=7572' 2024-04-06T09:47:52.256 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=144, tag_tid=26, entry_tid=7572' '!=' 'object_number=144, tag_tid=26, entry_tid=7444' 2024-04-06T09:47:52.256 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:47:52.256 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=144, tag_tid=26, entry_tid=7572' = 'object_number=144, tag_tid=26, entry_tid=7444' 2024-04-06T09:47:52.256 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=144, tag_tid=26, entry_tid=7572' 2024-04-06T09:47:52.256 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=145, tag_tid=27, entry_tid=1' 2024-04-06T09:47:52.256 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:47:52.256 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:47:52.256 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=27 2024-04-06T09:47:52.257 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=144, tag_tid=26, entry_tid=7572' 2024-04-06T09:47:52.257 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:47:52.257 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:47:52.258 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=26 2024-04-06T09:47:52.259 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=145, tag_tid=27, entry_tid=1' 2024-04-06T09:47:52.259 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:47:52.259 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:47:52.260 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:47:52.260 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=144, tag_tid=26, entry_tid=7572' 2024-04-06T09:47:52.261 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:47:52.261 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:47:52.262 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=7572 2024-04-06T09:47:52.262 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 27 = 26 -a 1 -le 7572 2024-04-06T09:47:52.262 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:47:52.262 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:47:52.262 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:47:52.464 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:47:52.464 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:47:52.464 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:47:52.464 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:47:52.464 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:47:52.464 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:47:52.464 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:47:52.464 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:47:52.464 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:47:52.464 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:47:52.464 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:47:52.464 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:47:52.464 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:47:52.465 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:47:52.465 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:47:52.465 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:47:52.465 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:47:52.467 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:47:52.468 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:47:52.468 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:47:52.468 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:47:52.468 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:47:52.468 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:47:52.468 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:47:52.468 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:47:52.468 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:47:52.468 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:47:52.468 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:47:52.468 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:47:52.468 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:47:52.468 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:47:52.468 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:47:52.469 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:47:52.469 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:47:52.469 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:47:52.470 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:47:52.470 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:47:52.470 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:47:52.470 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:47:52.470 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:47:52.470 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:47:52.470 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:47:52.470 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:47:52.470 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:47:52.470 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:47:52.470 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:47:52.470 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:47:52.470 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:47:52.471 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:47:52.495 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:47:52.495 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:47:52.495 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:47:52.495 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:47:52.603 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-06T09:47:52.604 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:rbd mirror flush : flush rbd mirror 2024-04-06T09:47:52.604 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:admin_socket: invalid command 2024-04-06T09:47:52.607 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:47:52.607 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/test 2024-04-06T09:47:52.607 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:1 2024-04-06T09:47:52.607 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:47:52.607 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:47:52.607 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-06T09:47:52.607 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:1 2024-04-06T09:47:52.607 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:47:52.607 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:47:52.607 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:47:52.607 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=1 2024-04-06T09:47:52.607 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-06T09:47:52.608 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:47:52.608 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:47:52.608 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=1 2024-04-06T09:47:52.608 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=1 2024-04-06T09:47:52.608 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-06T09:47:52.608 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:1 2024-04-06T09:47:52.608 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:47:52.608 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:47:52.608 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-06T09:47:52.608 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:1 2024-04-06T09:47:52.608 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:47:52.608 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:47:52.608 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:47:52.608 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=1 2024-04-06T09:47:52.609 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-06T09:47:52.609 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:47:52.609 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:47:52.609 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=1 2024-04-06T09:47:52.609 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=1 2024-04-06T09:47:52.609 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-06T09:47:52.631 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T09:47:52.631 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T09:47:52.631 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T09:47:52.631 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/test 2024-04-06T09:47:52.762 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:47:52.763 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:47:52.763 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:47:52.763 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:47:52.764 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:47:52.764 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:47:52.764 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:47:52.764 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:47:52.764 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:47:52.764 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:47:52.764 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:47:52.765 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:47:52.765 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:47:52.766 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:47:52.767 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:47:52.767 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:47:52.813 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 36 2024-04-06T09:47:52.813 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 36 2024-04-06T09:47:52.813 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:47:52.813 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=145, tag_tid=27, entry_tid=1], [object_number=144, tag_tid=27, entry_tid=0], [object_number=147, tag_tid=26, entry_tid=8935], [object_number=146, tag_tid=26, entry_tid=8934]]], state=connected] 2024-04-06T09:47:52.814 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=144, tag_tid=26, entry_tid=7700], [object_number=147, tag_tid=26, entry_tid=7699], [object_number=146, tag_tid=26, entry_tid=7698], [object_number=145, tag_tid=26, entry_tid=7697]]], state=connected] 2024-04-06T09:47:52.817 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:47:52.819 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=145, tag_tid=27, entry_tid=1' 2024-04-06T09:47:52.819 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:47:52.819 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:47:52.819 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:47:52.820 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:47:52.820 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:47:52.820 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:47:52.820 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:47:52.820 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:47:52.820 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:47:52.820 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:47:52.820 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:47:52.820 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:47:52.821 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:47:52.821 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:47:52.821 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:47:53.082 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 36 2024-04-06T09:47:53.082 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 36 2024-04-06T09:47:53.082 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:47:53.082 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=145, tag_tid=27, entry_tid=1], [object_number=144, tag_tid=27, entry_tid=0], [object_number=147, tag_tid=26, entry_tid=8935], [object_number=146, tag_tid=26, entry_tid=8934]]], state=connected] 2024-04-06T09:47:53.082 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=144, tag_tid=26, entry_tid=7740], [object_number=147, tag_tid=26, entry_tid=7739], [object_number=146, tag_tid=26, entry_tid=7738], [object_number=145, tag_tid=26, entry_tid=7737]]], state=connected] 2024-04-06T09:47:53.086 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:47:53.088 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=144, tag_tid=26, entry_tid=7740' 2024-04-06T09:47:53.088 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=145, tag_tid=27, entry_tid=1' -a 'object_number=145, tag_tid=27, entry_tid=1' = 'object_number=144, tag_tid=26, entry_tid=7740' 2024-04-06T09:47:53.088 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=144, tag_tid=26, entry_tid=7740' '!=' 'object_number=144, tag_tid=26, entry_tid=7572' 2024-04-06T09:47:53.089 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:47:53.089 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=144, tag_tid=26, entry_tid=7740' = 'object_number=144, tag_tid=26, entry_tid=7572' 2024-04-06T09:47:53.089 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=144, tag_tid=26, entry_tid=7740' 2024-04-06T09:47:53.089 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=145, tag_tid=27, entry_tid=1' 2024-04-06T09:47:53.089 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:47:53.089 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:47:53.090 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=27 2024-04-06T09:47:53.091 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=144, tag_tid=26, entry_tid=7740' 2024-04-06T09:47:53.092 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:47:53.092 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:47:53.093 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=26 2024-04-06T09:47:53.094 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=145, tag_tid=27, entry_tid=1' 2024-04-06T09:47:53.094 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:47:53.094 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:47:53.095 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:47:53.096 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=144, tag_tid=26, entry_tid=7740' 2024-04-06T09:47:53.096 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:47:53.096 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:47:53.098 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=7740 2024-04-06T09:47:53.098 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 27 = 26 -a 1 -le 7740 2024-04-06T09:47:53.098 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:47:53.098 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:47:53.098 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:47:53.300 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:47:53.300 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:47:53.300 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:47:53.300 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:47:53.300 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:47:53.300 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:47:53.300 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:47:53.301 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:47:53.301 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:47:53.301 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:47:53.301 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:47:53.301 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:47:53.301 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:47:53.301 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:47:53.301 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:47:53.301 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:47:53.301 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:47:53.303 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:47:53.303 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:47:53.303 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:47:53.304 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:47:53.304 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:47:53.304 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:47:53.304 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:47:53.304 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:47:53.304 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:47:53.304 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:47:53.304 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:47:53.304 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:47:53.304 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:47:53.304 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:47:53.304 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:47:53.304 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:47:53.305 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:47:53.305 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:47:53.305 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:47:53.305 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:47:53.306 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:47:53.306 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:47:53.306 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:47:53.306 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:47:53.306 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:47:53.306 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:47:53.306 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:47:53.306 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:47:53.306 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:47:53.306 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:47:53.306 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:47:53.306 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:47:53.330 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:47:53.331 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:47:53.331 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:47:53.331 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:47:53.436 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-06T09:47:53.436 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:rbd mirror flush : flush rbd mirror 2024-04-06T09:47:53.436 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:admin_socket: invalid command 2024-04-06T09:47:53.439 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:47:53.439 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/test 2024-04-06T09:47:53.439 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:1 2024-04-06T09:47:53.439 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:47:53.439 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:47:53.439 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-06T09:47:53.439 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:1 2024-04-06T09:47:53.440 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:47:53.440 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:47:53.440 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:47:53.440 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=1 2024-04-06T09:47:53.440 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-06T09:47:53.440 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:47:53.440 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:47:53.440 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=1 2024-04-06T09:47:53.440 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=1 2024-04-06T09:47:53.440 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-06T09:47:53.440 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:1 2024-04-06T09:47:53.440 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:47:53.440 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:47:53.440 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-06T09:47:53.441 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:1 2024-04-06T09:47:53.441 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:47:53.441 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:47:53.441 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:47:53.441 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=1 2024-04-06T09:47:53.441 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-06T09:47:53.441 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:47:53.441 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:47:53.441 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=1 2024-04-06T09:47:53.441 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=1 2024-04-06T09:47:53.442 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-06T09:47:53.464 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T09:47:53.464 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T09:47:53.464 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T09:47:53.464 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/test 2024-04-06T09:47:53.585 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:47:53.586 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:47:53.586 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:47:53.586 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:47:53.586 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:47:53.586 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:47:53.586 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:47:53.586 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:47:53.586 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:47:53.587 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:47:53.587 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:47:53.587 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:47:53.587 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:47:53.589 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:47:53.589 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:47:53.589 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:47:53.643 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 36 2024-04-06T09:47:53.643 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 36 2024-04-06T09:47:53.643 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:47:53.643 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=145, tag_tid=27, entry_tid=1], [object_number=144, tag_tid=27, entry_tid=0], [object_number=147, tag_tid=26, entry_tid=8935], [object_number=146, tag_tid=26, entry_tid=8934]]], state=connected] 2024-04-06T09:47:53.643 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=144, tag_tid=26, entry_tid=7876], [object_number=147, tag_tid=26, entry_tid=7875], [object_number=146, tag_tid=26, entry_tid=7874], [object_number=145, tag_tid=26, entry_tid=7873]]], state=connected] 2024-04-06T09:47:53.647 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:47:53.648 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=145, tag_tid=27, entry_tid=1' 2024-04-06T09:47:53.649 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:47:53.649 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:47:53.649 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:47:53.649 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:47:53.650 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:47:53.650 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:47:53.650 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:47:53.650 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:47:53.650 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:47:53.650 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:47:53.650 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:47:53.650 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:47:53.651 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:47:53.651 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:47:53.651 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:47:53.698 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 36 2024-04-06T09:47:53.698 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 36 2024-04-06T09:47:53.698 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:47:53.698 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=145, tag_tid=27, entry_tid=1], [object_number=144, tag_tid=27, entry_tid=0], [object_number=147, tag_tid=26, entry_tid=8935], [object_number=146, tag_tid=26, entry_tid=8934]]], state=connected] 2024-04-06T09:47:53.698 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=144, tag_tid=26, entry_tid=7876], [object_number=147, tag_tid=26, entry_tid=7875], [object_number=146, tag_tid=26, entry_tid=7874], [object_number=145, tag_tid=26, entry_tid=7873]]], state=connected] 2024-04-06T09:47:53.702 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:47:53.704 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=144, tag_tid=26, entry_tid=7876' 2024-04-06T09:47:53.704 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=145, tag_tid=27, entry_tid=1' -a 'object_number=145, tag_tid=27, entry_tid=1' = 'object_number=144, tag_tid=26, entry_tid=7876' 2024-04-06T09:47:53.704 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=144, tag_tid=26, entry_tid=7876' '!=' 'object_number=144, tag_tid=26, entry_tid=7740' 2024-04-06T09:47:53.704 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:47:53.704 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=144, tag_tid=26, entry_tid=7876' = 'object_number=144, tag_tid=26, entry_tid=7740' 2024-04-06T09:47:53.704 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=144, tag_tid=26, entry_tid=7876' 2024-04-06T09:47:53.704 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=145, tag_tid=27, entry_tid=1' 2024-04-06T09:47:53.704 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:47:53.704 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:47:53.705 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=27 2024-04-06T09:47:53.706 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=144, tag_tid=26, entry_tid=7876' 2024-04-06T09:47:53.706 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:47:53.706 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:47:53.707 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=26 2024-04-06T09:47:53.708 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=145, tag_tid=27, entry_tid=1' 2024-04-06T09:47:53.708 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:47:53.708 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:47:53.709 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:47:53.710 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=144, tag_tid=26, entry_tid=7876' 2024-04-06T09:47:53.710 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:47:53.710 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:47:53.711 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=7876 2024-04-06T09:47:53.711 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 27 = 26 -a 1 -le 7876 2024-04-06T09:47:53.712 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:47:53.712 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:47:53.712 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:47:53.913 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:47:53.913 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:47:53.913 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:47:53.913 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:47:53.913 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:47:53.913 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:47:53.913 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:47:53.913 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:47:53.913 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:47:53.913 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:47:53.913 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:47:53.913 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:47:53.914 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:47:53.914 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:47:53.914 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:47:53.914 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:47:53.914 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:47:53.915 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:47:53.916 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:47:53.916 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:47:53.916 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:47:53.916 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:47:53.916 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:47:53.916 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:47:53.916 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:47:53.916 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:47:53.916 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:47:53.916 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:47:53.916 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:47:53.916 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:47:53.916 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:47:53.916 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:47:53.916 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:47:53.917 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:47:53.918 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:47:53.918 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:47:53.918 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:47:53.918 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:47:53.918 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:47:53.918 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:47:53.918 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:47:53.918 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:47:53.918 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:47:53.918 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:47:53.918 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:47:53.918 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:47:53.918 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:47:53.918 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:47:53.919 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:47:53.943 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:47:53.944 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:47:53.944 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:47:53.944 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:47:54.050 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-06T09:47:54.050 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:rbd mirror flush : flush rbd mirror 2024-04-06T09:47:54.050 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:admin_socket: invalid command 2024-04-06T09:47:54.053 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:47:54.054 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/test 2024-04-06T09:47:54.054 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:1 2024-04-06T09:47:54.054 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:47:54.054 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:47:54.054 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-06T09:47:54.054 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:1 2024-04-06T09:47:54.054 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:47:54.054 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:47:54.054 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:47:54.054 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=1 2024-04-06T09:47:54.054 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-06T09:47:54.055 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:47:54.055 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:47:54.055 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=1 2024-04-06T09:47:54.055 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=1 2024-04-06T09:47:54.055 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-06T09:47:54.055 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:1 2024-04-06T09:47:54.055 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:47:54.055 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:47:54.055 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-06T09:47:54.055 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:1 2024-04-06T09:47:54.055 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:47:54.055 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:47:54.055 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:47:54.055 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=1 2024-04-06T09:47:54.056 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-06T09:47:54.056 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:47:54.056 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:47:54.056 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=1 2024-04-06T09:47:54.056 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=1 2024-04-06T09:47:54.056 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-06T09:47:54.077 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T09:47:54.078 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T09:47:54.078 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T09:47:54.078 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/test 2024-04-06T09:47:54.205 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:47:54.206 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:47:54.206 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:47:54.207 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:47:54.207 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:47:54.207 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:47:54.207 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:47:54.207 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:47:54.207 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:47:54.207 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:47:54.207 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:47:54.208 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:47:54.208 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:47:54.209 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:47:54.209 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:47:54.209 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:47:54.262 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 36 2024-04-06T09:47:54.262 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 36 2024-04-06T09:47:54.262 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:47:54.262 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=145, tag_tid=27, entry_tid=1], [object_number=144, tag_tid=27, entry_tid=0], [object_number=147, tag_tid=26, entry_tid=8935], [object_number=146, tag_tid=26, entry_tid=8934]]], state=connected] 2024-04-06T09:47:54.262 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=144, tag_tid=26, entry_tid=8016], [object_number=147, tag_tid=26, entry_tid=8015], [object_number=146, tag_tid=26, entry_tid=8014], [object_number=145, tag_tid=26, entry_tid=8013]]], state=connected] 2024-04-06T09:47:54.266 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:47:54.268 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=145, tag_tid=27, entry_tid=1' 2024-04-06T09:47:54.268 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:47:54.268 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:47:54.268 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:47:54.268 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:47:54.268 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:47:54.268 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:47:54.269 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:47:54.269 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:47:54.269 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:47:54.269 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:47:54.269 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:47:54.269 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:47:54.269 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:47:54.270 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:47:54.270 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:47:54.317 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 36 2024-04-06T09:47:54.318 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 36 2024-04-06T09:47:54.318 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:47:54.318 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=145, tag_tid=27, entry_tid=1], [object_number=144, tag_tid=27, entry_tid=0], [object_number=147, tag_tid=26, entry_tid=8935], [object_number=146, tag_tid=26, entry_tid=8934]]], state=connected] 2024-04-06T09:47:54.318 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=144, tag_tid=26, entry_tid=8016], [object_number=147, tag_tid=26, entry_tid=8015], [object_number=146, tag_tid=26, entry_tid=8014], [object_number=145, tag_tid=26, entry_tid=8013]]], state=connected] 2024-04-06T09:47:54.321 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:47:54.323 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=144, tag_tid=26, entry_tid=8016' 2024-04-06T09:47:54.323 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=145, tag_tid=27, entry_tid=1' -a 'object_number=145, tag_tid=27, entry_tid=1' = 'object_number=144, tag_tid=26, entry_tid=8016' 2024-04-06T09:47:54.323 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=144, tag_tid=26, entry_tid=8016' '!=' 'object_number=144, tag_tid=26, entry_tid=7876' 2024-04-06T09:47:54.323 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:47:54.323 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=144, tag_tid=26, entry_tid=8016' = 'object_number=144, tag_tid=26, entry_tid=7876' 2024-04-06T09:47:54.323 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=144, tag_tid=26, entry_tid=8016' 2024-04-06T09:47:54.324 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=145, tag_tid=27, entry_tid=1' 2024-04-06T09:47:54.324 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:47:54.324 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:47:54.324 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=27 2024-04-06T09:47:54.325 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=144, tag_tid=26, entry_tid=8016' 2024-04-06T09:47:54.325 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:47:54.325 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:47:54.326 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=26 2024-04-06T09:47:54.327 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=145, tag_tid=27, entry_tid=1' 2024-04-06T09:47:54.327 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:47:54.327 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:47:54.328 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:47:54.329 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=144, tag_tid=26, entry_tid=8016' 2024-04-06T09:47:54.329 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:47:54.329 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:47:54.330 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=8016 2024-04-06T09:47:54.330 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 27 = 26 -a 1 -le 8016 2024-04-06T09:47:54.330 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:47:54.330 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:47:54.331 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:47:54.532 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:47:54.532 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:47:54.532 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:47:54.532 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:47:54.532 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:47:54.532 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:47:54.532 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:47:54.532 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:47:54.532 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:47:54.533 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:47:54.533 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:47:54.533 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:47:54.533 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:47:54.533 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:47:54.533 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:47:54.533 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:47:54.534 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:47:54.537 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:47:54.537 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:47:54.537 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:47:54.537 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:47:54.537 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:47:54.537 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:47:54.537 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:47:54.538 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:47:54.538 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:47:54.538 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:47:54.538 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:47:54.538 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:47:54.538 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:47:54.538 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:47:54.538 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:47:54.538 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:47:54.538 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:47:54.538 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:47:54.538 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:47:54.538 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:47:54.538 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:47:54.539 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:47:54.539 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:47:54.539 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:47:54.539 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:47:54.539 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:47:54.539 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:47:54.539 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:47:54.539 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:47:54.539 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:47:54.539 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:47:54.539 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:47:54.562 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:47:54.563 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:47:54.563 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:47:54.563 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:47:54.670 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-06T09:47:54.671 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:rbd mirror flush : flush rbd mirror 2024-04-06T09:47:54.671 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:admin_socket: invalid command 2024-04-06T09:47:54.673 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:47:54.673 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/test 2024-04-06T09:47:54.673 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:1 2024-04-06T09:47:54.673 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:47:54.673 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:47:54.673 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-06T09:47:54.674 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:1 2024-04-06T09:47:54.674 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:47:54.674 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:47:54.674 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:47:54.674 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=1 2024-04-06T09:47:54.674 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-06T09:47:54.674 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:47:54.674 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:47:54.674 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=1 2024-04-06T09:47:54.674 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=1 2024-04-06T09:47:54.676 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-06T09:47:54.676 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:1 2024-04-06T09:47:54.676 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:47:54.676 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:47:54.677 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-06T09:47:54.677 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:1 2024-04-06T09:47:54.677 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:47:54.677 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:47:54.677 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:47:54.677 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=1 2024-04-06T09:47:54.677 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-06T09:47:54.677 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:47:54.677 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:47:54.677 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=1 2024-04-06T09:47:54.677 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=1 2024-04-06T09:47:54.677 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-06T09:47:54.699 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T09:47:54.700 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T09:47:54.700 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T09:47:54.700 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/test 2024-04-06T09:47:54.827 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:47:54.828 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:47:54.828 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:47:54.829 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:47:54.829 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:47:54.829 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:47:54.829 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:47:54.829 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:47:54.829 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:47:54.829 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:47:54.830 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:47:54.830 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:47:54.830 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:47:54.831 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:47:54.831 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:47:54.831 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:47:54.886 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 36 2024-04-06T09:47:54.886 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 36 2024-04-06T09:47:54.886 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:47:54.886 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=145, tag_tid=27, entry_tid=1], [object_number=144, tag_tid=27, entry_tid=0], [object_number=147, tag_tid=26, entry_tid=8935], [object_number=146, tag_tid=26, entry_tid=8934]]], state=connected] 2024-04-06T09:47:54.886 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=146, tag_tid=26, entry_tid=8158], [object_number=145, tag_tid=26, entry_tid=8157], [object_number=144, tag_tid=26, entry_tid=8156], [object_number=147, tag_tid=26, entry_tid=8155]]], state=connected] 2024-04-06T09:47:54.890 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:47:54.891 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=145, tag_tid=27, entry_tid=1' 2024-04-06T09:47:54.892 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:47:54.892 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:47:54.892 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:47:54.892 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:47:54.892 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:47:54.893 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:47:54.893 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:47:54.893 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:47:54.893 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:47:54.893 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:47:54.893 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:47:54.893 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:47:54.893 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:47:54.894 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:47:54.894 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:47:54.941 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 36 2024-04-06T09:47:54.942 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 36 2024-04-06T09:47:54.942 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:47:54.942 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=145, tag_tid=27, entry_tid=1], [object_number=144, tag_tid=27, entry_tid=0], [object_number=147, tag_tid=26, entry_tid=8935], [object_number=146, tag_tid=26, entry_tid=8934]]], state=connected] 2024-04-06T09:47:54.942 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=146, tag_tid=26, entry_tid=8158], [object_number=145, tag_tid=26, entry_tid=8157], [object_number=144, tag_tid=26, entry_tid=8156], [object_number=147, tag_tid=26, entry_tid=8155]]], state=connected] 2024-04-06T09:47:54.945 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:47:54.947 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=146, tag_tid=26, entry_tid=8158' 2024-04-06T09:47:54.947 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=145, tag_tid=27, entry_tid=1' -a 'object_number=145, tag_tid=27, entry_tid=1' = 'object_number=146, tag_tid=26, entry_tid=8158' 2024-04-06T09:47:54.948 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=146, tag_tid=26, entry_tid=8158' '!=' 'object_number=144, tag_tid=26, entry_tid=8016' 2024-04-06T09:47:54.948 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:47:54.948 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=146, tag_tid=26, entry_tid=8158' = 'object_number=144, tag_tid=26, entry_tid=8016' 2024-04-06T09:47:54.948 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=146, tag_tid=26, entry_tid=8158' 2024-04-06T09:47:54.948 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=145, tag_tid=27, entry_tid=1' 2024-04-06T09:47:54.948 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:47:54.948 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:47:54.948 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=27 2024-04-06T09:47:54.949 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=146, tag_tid=26, entry_tid=8158' 2024-04-06T09:47:54.950 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:47:54.950 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:47:54.950 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=26 2024-04-06T09:47:54.951 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=145, tag_tid=27, entry_tid=1' 2024-04-06T09:47:54.951 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:47:54.951 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:47:54.952 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:47:54.953 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=146, tag_tid=26, entry_tid=8158' 2024-04-06T09:47:54.953 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:47:54.953 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:47:54.954 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=8158 2024-04-06T09:47:54.954 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 27 = 26 -a 1 -le 8158 2024-04-06T09:47:54.954 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:47:54.955 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:47:54.955 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:47:55.156 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:47:55.156 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:47:55.156 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:47:55.156 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:47:55.156 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:47:55.156 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:47:55.156 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:47:55.156 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:47:55.156 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:47:55.156 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:47:55.157 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:47:55.157 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:47:55.157 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:47:55.157 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:47:55.157 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:47:55.157 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:47:55.157 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:47:55.161 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:47:55.161 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:47:55.161 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:47:55.161 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:47:55.162 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:47:55.162 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:47:55.162 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:47:55.162 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:47:55.162 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:47:55.162 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:47:55.162 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:47:55.162 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:47:55.162 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:47:55.162 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:47:55.162 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:47:55.162 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:47:55.162 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:47:55.162 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:47:55.163 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:47:55.163 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:47:55.163 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:47:55.163 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:47:55.163 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:47:55.163 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:47:55.163 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:47:55.163 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:47:55.163 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:47:55.163 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:47:55.163 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:47:55.163 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:47:55.164 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:47:55.164 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:47:55.186 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:47:55.186 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:47:55.186 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:47:55.186 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:47:55.289 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-06T09:47:55.290 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:rbd mirror flush : flush rbd mirror 2024-04-06T09:47:55.290 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:admin_socket: invalid command 2024-04-06T09:47:55.292 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:47:55.292 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/test 2024-04-06T09:47:55.292 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:1 2024-04-06T09:47:55.292 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:47:55.292 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:47:55.292 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-06T09:47:55.292 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:1 2024-04-06T09:47:55.292 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:47:55.292 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:47:55.293 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:47:55.293 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=1 2024-04-06T09:47:55.293 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-06T09:47:55.293 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:47:55.293 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:47:55.293 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=1 2024-04-06T09:47:55.293 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=1 2024-04-06T09:47:55.294 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-06T09:47:55.344 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:1 2024-04-06T09:47:55.345 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:47:55.345 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:47:55.345 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-06T09:47:55.345 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:1 2024-04-06T09:47:55.345 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:47:55.345 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:47:55.345 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:47:55.345 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=1 2024-04-06T09:47:55.345 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-06T09:47:55.345 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:47:55.345 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:47:55.345 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=1 2024-04-06T09:47:55.346 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=1 2024-04-06T09:47:55.346 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-06T09:47:55.346 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T09:47:55.346 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T09:47:55.347 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T09:47:55.347 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/test 2024-04-06T09:47:55.447 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:47:55.448 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:47:55.448 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:47:55.448 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:47:55.448 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:47:55.449 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:47:55.449 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:47:55.449 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:47:55.449 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:47:55.449 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:47:55.449 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:47:55.450 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:47:55.450 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:47:55.451 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:47:55.452 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:47:55.452 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:47:55.503 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 36 2024-04-06T09:47:55.503 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 36 2024-04-06T09:47:55.503 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:47:55.504 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=145, tag_tid=27, entry_tid=1], [object_number=144, tag_tid=27, entry_tid=0], [object_number=147, tag_tid=26, entry_tid=8935], [object_number=146, tag_tid=26, entry_tid=8934]]], state=connected] 2024-04-06T09:47:55.504 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=147, tag_tid=26, entry_tid=8275], [object_number=146, tag_tid=26, entry_tid=8274], [object_number=145, tag_tid=26, entry_tid=8273], [object_number=144, tag_tid=26, entry_tid=8272]]], state=connected] 2024-04-06T09:47:55.507 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:47:55.509 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=145, tag_tid=27, entry_tid=1' 2024-04-06T09:47:55.510 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:47:55.510 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:47:55.510 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:47:55.510 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:47:55.510 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:47:55.510 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:47:55.510 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:47:55.510 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:47:55.510 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:47:55.511 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:47:55.511 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:47:55.511 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:47:55.511 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:47:55.511 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:47:55.512 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:47:55.561 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 36 2024-04-06T09:47:55.561 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 36 2024-04-06T09:47:55.561 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:47:55.561 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=145, tag_tid=27, entry_tid=1], [object_number=144, tag_tid=27, entry_tid=0], [object_number=147, tag_tid=26, entry_tid=8935], [object_number=146, tag_tid=26, entry_tid=8934]]], state=connected] 2024-04-06T09:47:55.561 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=147, tag_tid=26, entry_tid=8275], [object_number=146, tag_tid=26, entry_tid=8274], [object_number=145, tag_tid=26, entry_tid=8273], [object_number=144, tag_tid=26, entry_tid=8272]]], state=connected] 2024-04-06T09:47:55.565 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:47:55.567 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=147, tag_tid=26, entry_tid=8275' 2024-04-06T09:47:55.567 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=145, tag_tid=27, entry_tid=1' -a 'object_number=145, tag_tid=27, entry_tid=1' = 'object_number=147, tag_tid=26, entry_tid=8275' 2024-04-06T09:47:55.567 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=147, tag_tid=26, entry_tid=8275' '!=' 'object_number=146, tag_tid=26, entry_tid=8158' 2024-04-06T09:47:55.567 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:47:55.567 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=147, tag_tid=26, entry_tid=8275' = 'object_number=146, tag_tid=26, entry_tid=8158' 2024-04-06T09:47:55.567 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=147, tag_tid=26, entry_tid=8275' 2024-04-06T09:47:55.568 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=145, tag_tid=27, entry_tid=1' 2024-04-06T09:47:55.568 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:47:55.568 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:47:55.568 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=27 2024-04-06T09:47:55.569 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=147, tag_tid=26, entry_tid=8275' 2024-04-06T09:47:55.569 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:47:55.569 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:47:55.570 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=26 2024-04-06T09:47:55.571 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=145, tag_tid=27, entry_tid=1' 2024-04-06T09:47:55.571 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:47:55.571 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:47:55.572 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:47:55.572 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=147, tag_tid=26, entry_tid=8275' 2024-04-06T09:47:55.572 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:47:55.572 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:47:55.574 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=8275 2024-04-06T09:47:55.574 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 27 = 26 -a 1 -le 8275 2024-04-06T09:47:55.574 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:47:55.574 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:47:55.574 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:47:55.775 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:47:55.776 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:47:55.776 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:47:55.776 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:47:55.776 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:47:55.776 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:47:55.776 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:47:55.776 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:47:55.776 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:47:55.776 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:47:55.776 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:47:55.776 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:47:55.776 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:47:55.776 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:47:55.776 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:47:55.777 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:47:55.778 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:47:55.780 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:47:55.780 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:47:55.780 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:47:55.780 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:47:55.780 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:47:55.780 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:47:55.780 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:47:55.780 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:47:55.780 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:47:55.780 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:47:55.781 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:47:55.781 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:47:55.781 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:47:55.781 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:47:55.781 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:47:55.781 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:47:55.782 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:47:55.782 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:47:55.782 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:47:55.782 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:47:55.782 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:47:55.782 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:47:55.782 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:47:55.782 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:47:55.783 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:47:55.783 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:47:55.783 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:47:55.783 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:47:55.783 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:47:55.783 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:47:55.783 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:47:55.783 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:47:55.807 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:47:55.807 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:47:55.807 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:47:55.807 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:47:55.913 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-06T09:47:55.913 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:rbd mirror flush : flush rbd mirror 2024-04-06T09:47:55.913 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:admin_socket: invalid command 2024-04-06T09:47:55.915 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:47:55.916 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/test 2024-04-06T09:47:55.916 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:1 2024-04-06T09:47:55.916 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:47:55.916 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:47:55.916 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-06T09:47:55.916 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:1 2024-04-06T09:47:55.916 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:47:55.916 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:47:55.916 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:47:55.916 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=1 2024-04-06T09:47:55.916 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-06T09:47:55.916 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:47:55.916 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:47:55.917 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=1 2024-04-06T09:47:55.917 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=1 2024-04-06T09:47:55.918 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-06T09:47:55.918 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:1 2024-04-06T09:47:55.918 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:47:55.918 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:47:55.918 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-06T09:47:55.918 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:1 2024-04-06T09:47:55.918 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:47:55.918 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:47:55.918 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:47:55.919 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=1 2024-04-06T09:47:55.919 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-06T09:47:55.919 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:47:55.919 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:47:55.919 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=1 2024-04-06T09:47:55.919 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=1 2024-04-06T09:47:55.919 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-06T09:47:55.941 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T09:47:55.941 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T09:47:55.942 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T09:47:55.942 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/test 2024-04-06T09:47:56.077 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:47:56.079 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:47:56.079 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:47:56.079 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:47:56.079 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:47:56.079 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:47:56.079 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:47:56.079 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:47:56.079 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:47:56.079 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:47:56.080 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:47:56.080 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:47:56.080 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:47:56.082 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:47:56.082 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:47:56.082 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:47:56.128 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 36 2024-04-06T09:47:56.128 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 36 2024-04-06T09:47:56.128 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:47:56.128 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=145, tag_tid=27, entry_tid=1], [object_number=144, tag_tid=27, entry_tid=0], [object_number=147, tag_tid=26, entry_tid=8935], [object_number=146, tag_tid=26, entry_tid=8934]]], state=connected] 2024-04-06T09:47:56.128 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=147, tag_tid=26, entry_tid=8391], [object_number=146, tag_tid=26, entry_tid=8390], [object_number=145, tag_tid=26, entry_tid=8389], [object_number=144, tag_tid=26, entry_tid=8388]]], state=connected] 2024-04-06T09:47:56.132 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:47:56.133 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=145, tag_tid=27, entry_tid=1' 2024-04-06T09:47:56.135 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:47:56.135 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:47:56.135 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:47:56.135 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:47:56.135 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:47:56.135 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:47:56.135 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:47:56.135 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:47:56.135 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:47:56.135 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:47:56.135 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:47:56.135 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:47:56.136 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:47:56.136 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:47:56.136 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:47:56.185 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 36 2024-04-06T09:47:56.185 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 36 2024-04-06T09:47:56.185 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:47:56.185 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=145, tag_tid=27, entry_tid=1], [object_number=144, tag_tid=27, entry_tid=0], [object_number=147, tag_tid=26, entry_tid=8935], [object_number=146, tag_tid=26, entry_tid=8934]]], state=connected] 2024-04-06T09:47:56.186 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=147, tag_tid=26, entry_tid=8391], [object_number=146, tag_tid=26, entry_tid=8390], [object_number=145, tag_tid=26, entry_tid=8389], [object_number=144, tag_tid=26, entry_tid=8388]]], state=connected] 2024-04-06T09:47:56.189 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:47:56.191 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=147, tag_tid=26, entry_tid=8391' 2024-04-06T09:47:56.191 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=145, tag_tid=27, entry_tid=1' -a 'object_number=145, tag_tid=27, entry_tid=1' = 'object_number=147, tag_tid=26, entry_tid=8391' 2024-04-06T09:47:56.191 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=147, tag_tid=26, entry_tid=8391' '!=' 'object_number=147, tag_tid=26, entry_tid=8275' 2024-04-06T09:47:56.191 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:47:56.192 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=147, tag_tid=26, entry_tid=8391' = 'object_number=147, tag_tid=26, entry_tid=8275' 2024-04-06T09:47:56.192 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=147, tag_tid=26, entry_tid=8391' 2024-04-06T09:47:56.192 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=145, tag_tid=27, entry_tid=1' 2024-04-06T09:47:56.192 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:47:56.192 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:47:56.192 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=27 2024-04-06T09:47:56.193 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=147, tag_tid=26, entry_tid=8391' 2024-04-06T09:47:56.193 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:47:56.193 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:47:56.194 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=26 2024-04-06T09:47:56.194 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=145, tag_tid=27, entry_tid=1' 2024-04-06T09:47:56.195 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:47:56.195 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:47:56.195 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:47:56.196 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=147, tag_tid=26, entry_tid=8391' 2024-04-06T09:47:56.196 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:47:56.196 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:47:56.197 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=8391 2024-04-06T09:47:56.198 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 27 = 26 -a 1 -le 8391 2024-04-06T09:47:56.198 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:47:56.198 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:47:56.198 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:47:56.399 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:47:56.399 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:47:56.399 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:47:56.399 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:47:56.400 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:47:56.400 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:47:56.400 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:47:56.400 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:47:56.400 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:47:56.400 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:47:56.400 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:47:56.400 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:47:56.400 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:47:56.400 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:47:56.400 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:47:56.401 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:47:56.401 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:47:56.405 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:47:56.405 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:47:56.405 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:47:56.405 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:47:56.405 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:47:56.405 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:47:56.405 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:47:56.405 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:47:56.405 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:47:56.405 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:47:56.406 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:47:56.406 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:47:56.406 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:47:56.406 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:47:56.406 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:47:56.406 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:47:56.406 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:47:56.406 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:47:56.406 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:47:56.406 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:47:56.406 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:47:56.406 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:47:56.406 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:47:56.407 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:47:56.407 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:47:56.407 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:47:56.407 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:47:56.407 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:47:56.407 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:47:56.407 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:47:56.407 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:47:56.407 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:47:56.430 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:47:56.431 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:47:56.431 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:47:56.431 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:47:56.536 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-06T09:47:56.536 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:rbd mirror flush : flush rbd mirror 2024-04-06T09:47:56.536 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:admin_socket: invalid command 2024-04-06T09:47:56.538 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:47:56.538 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/test 2024-04-06T09:47:56.539 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:1 2024-04-06T09:47:56.539 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:47:56.539 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:47:56.539 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-06T09:47:56.539 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:1 2024-04-06T09:47:56.539 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:47:56.539 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:47:56.539 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:47:56.539 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=1 2024-04-06T09:47:56.539 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-06T09:47:56.539 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:47:56.539 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:47:56.539 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=1 2024-04-06T09:47:56.539 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=1 2024-04-06T09:47:56.540 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-06T09:47:56.540 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:1 2024-04-06T09:47:56.540 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:47:56.540 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:47:56.540 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-06T09:47:56.540 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:1 2024-04-06T09:47:56.540 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:47:56.540 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:47:56.540 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:47:56.540 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=1 2024-04-06T09:47:56.540 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-06T09:47:56.540 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:47:56.540 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:47:56.540 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=1 2024-04-06T09:47:56.540 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=1 2024-04-06T09:47:56.541 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-06T09:47:56.564 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T09:47:56.564 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T09:47:56.565 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T09:47:56.565 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/test 2024-04-06T09:47:56.687 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:47:56.688 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:47:56.688 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:47:56.688 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:47:56.688 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:47:56.688 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:47:56.688 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:47:56.688 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:47:56.688 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:47:56.688 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:47:56.689 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:47:56.689 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:47:56.689 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:47:56.691 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:47:56.691 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:47:56.691 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:47:56.739 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 36 2024-04-06T09:47:56.740 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 36 2024-04-06T09:47:56.740 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:47:56.740 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=145, tag_tid=27, entry_tid=1], [object_number=144, tag_tid=27, entry_tid=0], [object_number=147, tag_tid=26, entry_tid=8935], [object_number=146, tag_tid=26, entry_tid=8934]]], state=connected] 2024-04-06T09:47:56.740 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=146, tag_tid=26, entry_tid=8506], [object_number=145, tag_tid=26, entry_tid=8505], [object_number=144, tag_tid=26, entry_tid=8504], [object_number=147, tag_tid=26, entry_tid=8503]]], state=connected] 2024-04-06T09:47:56.743 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:47:56.746 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=145, tag_tid=27, entry_tid=1' 2024-04-06T09:47:56.746 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:47:56.746 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:47:56.746 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:47:56.746 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:47:56.746 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:47:56.746 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:47:56.746 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:47:56.746 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:47:56.746 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:47:56.746 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:47:56.746 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:47:56.747 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:47:56.747 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:47:56.747 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:47:56.747 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:47:56.794 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 36 2024-04-06T09:47:56.794 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 36 2024-04-06T09:47:56.794 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:47:56.794 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=145, tag_tid=27, entry_tid=1], [object_number=144, tag_tid=27, entry_tid=0], [object_number=147, tag_tid=26, entry_tid=8935], [object_number=146, tag_tid=26, entry_tid=8934]]], state=connected] 2024-04-06T09:47:56.794 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=146, tag_tid=26, entry_tid=8506], [object_number=145, tag_tid=26, entry_tid=8505], [object_number=144, tag_tid=26, entry_tid=8504], [object_number=147, tag_tid=26, entry_tid=8503]]], state=connected] 2024-04-06T09:47:56.798 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:47:56.800 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=146, tag_tid=26, entry_tid=8506' 2024-04-06T09:47:56.800 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=145, tag_tid=27, entry_tid=1' -a 'object_number=145, tag_tid=27, entry_tid=1' = 'object_number=146, tag_tid=26, entry_tid=8506' 2024-04-06T09:47:56.800 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=146, tag_tid=26, entry_tid=8506' '!=' 'object_number=147, tag_tid=26, entry_tid=8391' 2024-04-06T09:47:56.800 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:47:56.800 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=146, tag_tid=26, entry_tid=8506' = 'object_number=147, tag_tid=26, entry_tid=8391' 2024-04-06T09:47:56.801 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=146, tag_tid=26, entry_tid=8506' 2024-04-06T09:47:56.801 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=145, tag_tid=27, entry_tid=1' 2024-04-06T09:47:56.801 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:47:56.801 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:47:56.801 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=27 2024-04-06T09:47:56.802 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=146, tag_tid=26, entry_tid=8506' 2024-04-06T09:47:56.802 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:47:56.802 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:47:56.803 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=26 2024-04-06T09:47:56.803 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=145, tag_tid=27, entry_tid=1' 2024-04-06T09:47:56.804 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:47:56.804 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:47:56.804 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:47:56.805 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=146, tag_tid=26, entry_tid=8506' 2024-04-06T09:47:56.805 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:47:56.805 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:47:56.806 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=8506 2024-04-06T09:47:56.807 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 27 = 26 -a 1 -le 8506 2024-04-06T09:47:56.807 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:47:56.807 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:47:56.807 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:47:57.008 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:47:57.008 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:47:57.008 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:47:57.008 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:47:57.009 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:47:57.009 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:47:57.009 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:47:57.009 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:47:57.009 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:47:57.009 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:47:57.009 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:47:57.009 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:47:57.009 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:47:57.009 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:47:57.009 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:47:57.010 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:47:57.010 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:47:57.012 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:47:57.012 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:47:57.013 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:47:57.013 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:47:57.013 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:47:57.013 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:47:57.013 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:47:57.013 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:47:57.013 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:47:57.013 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:47:57.013 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:47:57.013 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:47:57.013 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:47:57.013 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:47:57.013 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:47:57.013 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:47:57.014 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:47:57.014 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:47:57.014 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:47:57.014 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:47:57.014 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:47:57.014 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:47:57.014 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:47:57.014 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:47:57.014 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:47:57.014 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:47:57.014 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:47:57.014 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:47:57.014 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:47:57.015 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:47:57.015 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:47:57.015 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:47:57.038 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:47:57.039 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:47:57.039 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:47:57.039 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:47:57.145 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-06T09:47:57.146 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:rbd mirror flush : flush rbd mirror 2024-04-06T09:47:57.146 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:admin_socket: invalid command 2024-04-06T09:47:57.148 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:47:57.148 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/test 2024-04-06T09:47:57.148 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:1 2024-04-06T09:47:57.148 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:47:57.148 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:47:57.148 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-06T09:47:57.148 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:1 2024-04-06T09:47:57.148 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:47:57.148 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:47:57.149 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:47:57.149 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=1 2024-04-06T09:47:57.149 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-06T09:47:57.149 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:47:57.149 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:47:57.149 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=1 2024-04-06T09:47:57.149 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=1 2024-04-06T09:47:57.151 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-06T09:47:57.151 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:1 2024-04-06T09:47:57.151 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:47:57.151 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:47:57.151 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-06T09:47:57.151 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:1 2024-04-06T09:47:57.151 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:47:57.151 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:47:57.151 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:47:57.151 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=1 2024-04-06T09:47:57.151 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-06T09:47:57.151 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:47:57.151 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:47:57.152 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=1 2024-04-06T09:47:57.152 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=1 2024-04-06T09:47:57.152 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-06T09:47:57.174 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T09:47:57.175 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T09:47:57.175 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T09:47:57.175 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/test 2024-04-06T09:47:57.301 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:47:57.302 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:47:57.302 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:47:57.302 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:47:57.302 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:47:57.302 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:47:57.303 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:47:57.303 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:47:57.303 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:47:57.303 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:47:57.303 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:47:57.303 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:47:57.303 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:47:57.305 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:47:57.306 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:47:57.306 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:47:57.358 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 36 2024-04-06T09:47:57.359 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 36 2024-04-06T09:47:57.359 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:47:57.359 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=145, tag_tid=27, entry_tid=1], [object_number=144, tag_tid=27, entry_tid=0], [object_number=147, tag_tid=26, entry_tid=8935], [object_number=146, tag_tid=26, entry_tid=8934]]], state=connected] 2024-04-06T09:47:57.359 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=144, tag_tid=26, entry_tid=8620], [object_number=147, tag_tid=26, entry_tid=8619], [object_number=146, tag_tid=26, entry_tid=8618], [object_number=145, tag_tid=26, entry_tid=8617]]], state=connected] 2024-04-06T09:47:57.362 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:47:57.364 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=145, tag_tid=27, entry_tid=1' 2024-04-06T09:47:57.365 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:47:57.365 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:47:57.365 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:47:57.365 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:47:57.365 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:47:57.365 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:47:57.365 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:47:57.365 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:47:57.366 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:47:57.366 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:47:57.366 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:47:57.366 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:47:57.366 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:47:57.367 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:47:57.367 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:47:57.411 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.0 with skip (0.8179547033117065) > weight (0.25) 2024-04-06T09:47:57.411 INFO:tasks.rbd_mirror_thrash:kill cluster2.client.mirror.1 2024-04-06T09:47:57.412 INFO:tasks.rbd_mirror.cluster2.client.mirror.1:Sent signal 15 2024-04-06T09:47:57.412 INFO:tasks.rbd_mirror_thrash:waiting for 1 secs before reviving daemons 2024-04-06T09:47:57.412 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi192.stderr:2024-04-06T09:47:57.415+0000 7f574295a640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.1 (PID: 88727) UID: 1000 2024-04-06T09:47:57.416 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 36 2024-04-06T09:47:57.416 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 36 2024-04-06T09:47:57.416 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:47:57.416 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=145, tag_tid=27, entry_tid=1], [object_number=144, tag_tid=27, entry_tid=0], [object_number=147, tag_tid=26, entry_tid=8935], [object_number=146, tag_tid=26, entry_tid=8934]]], state=connected] 2024-04-06T09:47:57.416 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=144, tag_tid=26, entry_tid=8620], [object_number=147, tag_tid=26, entry_tid=8619], [object_number=146, tag_tid=26, entry_tid=8618], [object_number=145, tag_tid=26, entry_tid=8617]]], state=connected] 2024-04-06T09:47:57.420 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:47:57.422 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=144, tag_tid=26, entry_tid=8620' 2024-04-06T09:47:57.422 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=145, tag_tid=27, entry_tid=1' -a 'object_number=145, tag_tid=27, entry_tid=1' = 'object_number=144, tag_tid=26, entry_tid=8620' 2024-04-06T09:47:57.423 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=144, tag_tid=26, entry_tid=8620' '!=' 'object_number=146, tag_tid=26, entry_tid=8506' 2024-04-06T09:47:57.423 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:47:57.423 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=144, tag_tid=26, entry_tid=8620' = 'object_number=146, tag_tid=26, entry_tid=8506' 2024-04-06T09:47:57.423 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=144, tag_tid=26, entry_tid=8620' 2024-04-06T09:47:57.423 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=145, tag_tid=27, entry_tid=1' 2024-04-06T09:47:57.423 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:47:57.423 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:47:57.424 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=27 2024-04-06T09:47:57.425 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=144, tag_tid=26, entry_tid=8620' 2024-04-06T09:47:57.425 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:47:57.425 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:47:57.426 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=26 2024-04-06T09:47:57.427 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=145, tag_tid=27, entry_tid=1' 2024-04-06T09:47:57.427 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:47:57.427 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:47:57.428 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:47:57.429 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=144, tag_tid=26, entry_tid=8620' 2024-04-06T09:47:57.429 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:47:57.429 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:47:57.430 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=8620 2024-04-06T09:47:57.430 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 27 = 26 -a 1 -le 8620 2024-04-06T09:47:57.430 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:47:57.430 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:47:57.430 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:47:57.631 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:47:57.632 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:47:57.632 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:47:57.632 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:47:57.632 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:47:57.632 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:47:57.632 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:47:57.632 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:47:57.632 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:47:57.632 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:47:57.632 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:47:57.632 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:47:57.632 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:47:57.632 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:47:57.633 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:47:57.633 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:47:57.634 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:47:57.637 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:47:57.637 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:47:57.637 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:47:57.637 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:47:57.637 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:47:57.637 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:47:57.637 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:47:57.638 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:47:57.638 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:47:57.638 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:47:57.638 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:47:57.638 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:47:57.638 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:47:57.638 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:47:57.638 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:47:57.638 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:47:57.638 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:47:57.638 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:47:57.638 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:47:57.638 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:47:57.638 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:47:57.639 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:47:57.639 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:47:57.639 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:47:57.639 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:47:57.639 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:47:57.639 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:47:57.639 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:47:57.639 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:47:57.639 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:47:57.639 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:47:57.639 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:47:57.663 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:47:57.663 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:47:57.663 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:47:57.663 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:47:57.767 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-06T09:47:57.767 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:rbd mirror flush : flush rbd mirror 2024-04-06T09:47:57.767 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:admin_socket: invalid command 2024-04-06T09:47:57.769 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:47:57.769 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/test 2024-04-06T09:47:57.769 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:1 2024-04-06T09:47:57.769 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:47:57.769 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:47:57.769 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-06T09:47:57.769 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:1 2024-04-06T09:47:57.769 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:47:57.770 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:47:57.770 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:47:57.770 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=1 2024-04-06T09:47:57.770 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-06T09:47:57.770 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:47:57.770 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:47:57.770 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=1 2024-04-06T09:47:57.770 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=1 2024-04-06T09:47:57.771 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-06T09:47:57.771 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:1 2024-04-06T09:47:57.772 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:47:57.772 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:47:57.772 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-06T09:47:57.772 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:1 2024-04-06T09:47:57.772 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:47:57.772 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:47:57.772 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:47:57.772 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=1 2024-04-06T09:47:57.772 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-06T09:47:57.772 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:47:57.772 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:47:57.772 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=1 2024-04-06T09:47:57.772 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=1 2024-04-06T09:47:57.772 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-06T09:47:57.789 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T09:47:57.790 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T09:47:57.790 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T09:47:57.790 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/test 2024-04-06T09:47:57.911 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:47:57.912 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:47:57.912 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:47:57.912 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:47:57.912 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:47:57.913 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:47:57.913 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:47:57.913 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:47:57.913 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:47:57.913 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:47:57.913 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:47:57.914 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:47:57.914 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:47:57.915 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:47:57.915 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:47:57.916 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:47:57.963 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 36 2024-04-06T09:47:57.963 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 36 2024-04-06T09:47:57.963 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:47:57.963 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=145, tag_tid=27, entry_tid=1], [object_number=144, tag_tid=27, entry_tid=0], [object_number=147, tag_tid=26, entry_tid=8935], [object_number=146, tag_tid=26, entry_tid=8934]]], state=connected] 2024-04-06T09:47:57.963 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=144, tag_tid=26, entry_tid=8736], [object_number=147, tag_tid=26, entry_tid=8735], [object_number=146, tag_tid=26, entry_tid=8734], [object_number=145, tag_tid=26, entry_tid=8733]]], state=connected] 2024-04-06T09:47:57.967 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:47:57.969 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=145, tag_tid=27, entry_tid=1' 2024-04-06T09:47:57.969 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:47:57.969 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:47:57.969 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:47:57.969 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:47:57.970 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:47:57.970 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:47:57.970 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:47:57.970 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:47:57.970 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:47:57.970 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:47:57.970 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:47:57.970 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:47:57.971 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:47:57.971 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:47:57.971 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:47:58.018 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 36 2024-04-06T09:47:58.018 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 36 2024-04-06T09:47:58.018 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:47:58.018 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=145, tag_tid=27, entry_tid=1], [object_number=144, tag_tid=27, entry_tid=0], [object_number=147, tag_tid=26, entry_tid=8935], [object_number=146, tag_tid=26, entry_tid=8934]]], state=connected] 2024-04-06T09:47:58.018 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=144, tag_tid=26, entry_tid=8736], [object_number=147, tag_tid=26, entry_tid=8735], [object_number=146, tag_tid=26, entry_tid=8734], [object_number=145, tag_tid=26, entry_tid=8733]]], state=connected] 2024-04-06T09:47:58.021 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:47:58.024 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=144, tag_tid=26, entry_tid=8736' 2024-04-06T09:47:58.024 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=145, tag_tid=27, entry_tid=1' -a 'object_number=145, tag_tid=27, entry_tid=1' = 'object_number=144, tag_tid=26, entry_tid=8736' 2024-04-06T09:47:58.024 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=144, tag_tid=26, entry_tid=8736' '!=' 'object_number=144, tag_tid=26, entry_tid=8620' 2024-04-06T09:47:58.024 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:47:58.024 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=144, tag_tid=26, entry_tid=8736' = 'object_number=144, tag_tid=26, entry_tid=8620' 2024-04-06T09:47:58.024 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=144, tag_tid=26, entry_tid=8736' 2024-04-06T09:47:58.024 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=145, tag_tid=27, entry_tid=1' 2024-04-06T09:47:58.024 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:47:58.024 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:47:58.025 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=27 2024-04-06T09:47:58.025 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=144, tag_tid=26, entry_tid=8736' 2024-04-06T09:47:58.025 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:47:58.026 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:47:58.026 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=26 2024-04-06T09:47:58.027 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=145, tag_tid=27, entry_tid=1' 2024-04-06T09:47:58.027 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:47:58.027 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:47:58.028 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:47:58.029 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=144, tag_tid=26, entry_tid=8736' 2024-04-06T09:47:58.029 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:47:58.029 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:47:58.030 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=8736 2024-04-06T09:47:58.030 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 27 = 26 -a 1 -le 8736 2024-04-06T09:47:58.030 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:47:58.030 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:47:58.030 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:47:58.232 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:47:58.232 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:47:58.232 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:47:58.232 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:47:58.232 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:47:58.232 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:47:58.233 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:47:58.233 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:47:58.233 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:47:58.233 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:47:58.233 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:47:58.233 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:47:58.233 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:47:58.233 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:47:58.233 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:47:58.234 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:47:58.234 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:47:58.237 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:47:58.237 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:47:58.237 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:47:58.238 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:47:58.238 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:47:58.238 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:47:58.238 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:47:58.238 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:47:58.238 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:47:58.238 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:47:58.238 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:47:58.238 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:47:58.238 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:47:58.238 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:47:58.238 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:47:58.238 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:47:58.239 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:47:58.239 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:47:58.239 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:47:58.239 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:47:58.239 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:47:58.239 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:47:58.239 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:47:58.239 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:47:58.239 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:47:58.239 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:47:58.239 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:47:58.239 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:47:58.239 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:47:58.239 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:47:58.240 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:47:58.240 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:47:58.264 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:47:58.264 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:47:58.264 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:47:58.265 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:47:58.372 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-06T09:47:58.373 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:rbd mirror flush : flush rbd mirror 2024-04-06T09:47:58.373 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:admin_socket: invalid command 2024-04-06T09:47:58.375 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:47:58.375 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/test 2024-04-06T09:47:58.375 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:1 2024-04-06T09:47:58.375 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:47:58.375 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:47:58.376 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-06T09:47:58.376 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:1 2024-04-06T09:47:58.376 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:47:58.376 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:47:58.376 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:47:58.376 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=1 2024-04-06T09:47:58.376 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-06T09:47:58.376 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:47:58.376 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:47:58.376 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=1 2024-04-06T09:47:58.376 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=1 2024-04-06T09:47:58.378 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-06T09:47:58.378 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:1 2024-04-06T09:47:58.378 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:47:58.378 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:47:58.378 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-06T09:47:58.378 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:1 2024-04-06T09:47:58.378 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:47:58.378 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:47:58.378 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:47:58.378 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=1 2024-04-06T09:47:58.378 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-06T09:47:58.378 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:47:58.378 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:47:58.378 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=1 2024-04-06T09:47:58.378 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=1 2024-04-06T09:47:58.379 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-06T09:47:58.401 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T09:47:58.401 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T09:47:58.401 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T09:47:58.401 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/test 2024-04-06T09:47:58.413 INFO:tasks.rbd_mirror_thrash:waiting for cluster2.client.mirror.1 2024-04-06T09:47:58.413 INFO:teuthology.orchestra.run:waiting for 600 2024-04-06T09:47:58.413 INFO:tasks.rbd_mirror_thrash:reviving cluster2.client.mirror.1 2024-04-06T09:47:58.413 INFO:tasks.rbd_mirror.cluster2.client.mirror.1:Restarting daemon 2024-04-06T09:47:58.413 DEBUG:teuthology.orchestra.run.smithi192:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.1 2024-04-06T09:47:58.415 INFO:tasks.rbd_mirror.cluster2.client.mirror.1:Started 2024-04-06T09:47:58.416 INFO:tasks.rbd_mirror_thrash:waiting for 71 secs before thrashing 2024-04-06T09:47:58.529 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:47:58.529 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:47:58.529 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:47:58.529 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:47:58.529 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:47:58.529 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:47:58.529 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:47:58.529 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:47:58.529 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:47:58.529 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:47:58.529 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:47:58.530 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:47:58.530 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:47:58.530 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:47:58.530 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:47:58.531 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:47:58.582 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 36 2024-04-06T09:47:58.582 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 36 2024-04-06T09:47:58.583 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:47:58.583 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=145, tag_tid=27, entry_tid=1], [object_number=144, tag_tid=27, entry_tid=0], [object_number=147, tag_tid=26, entry_tid=8935], [object_number=146, tag_tid=26, entry_tid=8934]]], state=connected] 2024-04-06T09:47:58.583 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=144, tag_tid=26, entry_tid=8860], [object_number=147, tag_tid=26, entry_tid=8859], [object_number=146, tag_tid=26, entry_tid=8858], [object_number=145, tag_tid=26, entry_tid=8857]]], state=connected] 2024-04-06T09:47:58.587 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:47:58.588 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=145, tag_tid=27, entry_tid=1' 2024-04-06T09:47:58.589 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:47:58.590 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:47:58.590 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:47:58.590 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:47:58.590 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:47:58.590 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:47:58.590 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:47:58.590 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:47:58.590 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:47:58.590 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:47:58.591 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:47:58.591 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:47:58.592 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:47:58.592 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:47:58.592 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:47:58.649 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 36 2024-04-06T09:47:58.650 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 36 2024-04-06T09:47:58.650 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:47:58.650 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=145, tag_tid=27, entry_tid=1], [object_number=144, tag_tid=27, entry_tid=0], [object_number=147, tag_tid=26, entry_tid=8935], [object_number=146, tag_tid=26, entry_tid=8934]]], state=connected] 2024-04-06T09:47:58.650 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=144, tag_tid=26, entry_tid=8860], [object_number=147, tag_tid=26, entry_tid=8859], [object_number=146, tag_tid=26, entry_tid=8858], [object_number=145, tag_tid=26, entry_tid=8857]]], state=connected] 2024-04-06T09:47:58.653 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:47:58.655 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=144, tag_tid=26, entry_tid=8860' 2024-04-06T09:47:58.655 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=145, tag_tid=27, entry_tid=1' -a 'object_number=145, tag_tid=27, entry_tid=1' = 'object_number=144, tag_tid=26, entry_tid=8860' 2024-04-06T09:47:58.655 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=144, tag_tid=26, entry_tid=8860' '!=' 'object_number=144, tag_tid=26, entry_tid=8736' 2024-04-06T09:47:58.655 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:47:58.655 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=144, tag_tid=26, entry_tid=8860' = 'object_number=144, tag_tid=26, entry_tid=8736' 2024-04-06T09:47:58.656 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=144, tag_tid=26, entry_tid=8860' 2024-04-06T09:47:58.656 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=145, tag_tid=27, entry_tid=1' 2024-04-06T09:47:58.656 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:47:58.656 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:47:58.657 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=27 2024-04-06T09:47:58.657 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=144, tag_tid=26, entry_tid=8860' 2024-04-06T09:47:58.657 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:47:58.658 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:47:58.658 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=26 2024-04-06T09:47:58.659 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=145, tag_tid=27, entry_tid=1' 2024-04-06T09:47:58.659 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:47:58.659 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:47:58.660 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:47:58.661 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=144, tag_tid=26, entry_tid=8860' 2024-04-06T09:47:58.661 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:47:58.661 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:47:58.662 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=8860 2024-04-06T09:47:58.662 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 27 = 26 -a 1 -le 8860 2024-04-06T09:47:58.662 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:47:58.662 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:47:58.662 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:47:58.864 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:47:58.864 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:47:58.864 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:47:58.864 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:47:58.864 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:47:58.865 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:47:58.865 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:47:58.865 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:47:58.865 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:47:58.865 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:47:58.865 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:47:58.865 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:47:58.865 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:47:58.865 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:47:58.865 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:47:58.866 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:47:58.866 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:47:58.870 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:47:58.870 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:47:58.870 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:47:58.870 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:47:58.870 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:47:58.870 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:47:58.870 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:47:58.870 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:47:58.870 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:47:58.871 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:47:58.871 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:47:58.871 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:47:58.871 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:47:58.871 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:47:58.871 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:47:58.871 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:47:58.871 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:47:58.871 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:47:58.871 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:47:58.871 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:47:58.871 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:47:58.871 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:47:58.871 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:47:58.872 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:47:58.872 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:47:58.872 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:47:58.872 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:47:58.872 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:47:58.872 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:47:58.872 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:47:58.872 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:47:58.872 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:47:58.894 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:47:58.895 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:47:58.895 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:47:58.895 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:47:59.000 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-06T09:47:59.001 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:rbd mirror flush : flush rbd mirror 2024-04-06T09:47:59.001 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:admin_socket: invalid command 2024-04-06T09:47:59.003 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:47:59.004 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/test 2024-04-06T09:47:59.004 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:1 2024-04-06T09:47:59.004 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:47:59.004 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:47:59.004 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-06T09:47:59.004 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:1 2024-04-06T09:47:59.004 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:47:59.004 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:47:59.004 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:47:59.004 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=1 2024-04-06T09:47:59.005 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-06T09:47:59.005 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:47:59.005 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:47:59.005 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=1 2024-04-06T09:47:59.005 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=1 2024-04-06T09:47:59.005 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-06T09:47:59.005 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:1 2024-04-06T09:47:59.005 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:47:59.005 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:47:59.005 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-06T09:47:59.006 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:1 2024-04-06T09:47:59.006 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:47:59.006 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:47:59.006 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:47:59.007 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=1 2024-04-06T09:47:59.007 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-06T09:47:59.007 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:47:59.007 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:47:59.007 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=1 2024-04-06T09:47:59.007 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=1 2024-04-06T09:47:59.008 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-06T09:47:59.028 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T09:47:59.028 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T09:47:59.028 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T09:47:59.028 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/test 2024-04-06T09:47:59.145 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:47:59.146 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:47:59.146 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:47:59.146 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:47:59.146 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:47:59.146 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:47:59.146 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:47:59.147 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:47:59.147 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:47:59.147 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:47:59.147 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:47:59.147 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:47:59.148 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:47:59.150 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:47:59.150 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:47:59.151 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:47:59.201 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 36 2024-04-06T09:47:59.201 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 36 2024-04-06T09:47:59.201 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:47:59.201 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=145, tag_tid=27, entry_tid=1], [object_number=144, tag_tid=27, entry_tid=0], [object_number=147, tag_tid=26, entry_tid=8935], [object_number=146, tag_tid=26, entry_tid=8934]]], state=connected] 2024-04-06T09:47:59.202 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=144, tag_tid=26, entry_tid=8936], [object_number=147, tag_tid=26, entry_tid=8935], [object_number=146, tag_tid=26, entry_tid=8934], [object_number=145, tag_tid=26, entry_tid=8933]]], state=connected] 2024-04-06T09:47:59.205 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:47:59.206 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=145, tag_tid=27, entry_tid=1' 2024-04-06T09:47:59.207 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:47:59.208 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:47:59.208 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:47:59.208 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:47:59.208 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:47:59.208 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:47:59.208 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:47:59.208 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:47:59.208 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:47:59.208 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:47:59.208 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:47:59.208 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:47:59.209 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:47:59.209 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:47:59.209 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:47:59.256 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 36 2024-04-06T09:47:59.256 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 36 2024-04-06T09:47:59.256 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:47:59.256 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=145, tag_tid=27, entry_tid=1], [object_number=144, tag_tid=27, entry_tid=0], [object_number=147, tag_tid=26, entry_tid=8935], [object_number=146, tag_tid=26, entry_tid=8934]]], state=connected] 2024-04-06T09:47:59.256 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=144, tag_tid=26, entry_tid=8936], [object_number=147, tag_tid=26, entry_tid=8935], [object_number=146, tag_tid=26, entry_tid=8934], [object_number=145, tag_tid=26, entry_tid=8933]]], state=connected] 2024-04-06T09:47:59.259 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:47:59.261 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=144, tag_tid=26, entry_tid=8936' 2024-04-06T09:47:59.261 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=145, tag_tid=27, entry_tid=1' -a 'object_number=145, tag_tid=27, entry_tid=1' = 'object_number=144, tag_tid=26, entry_tid=8936' 2024-04-06T09:47:59.261 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=144, tag_tid=26, entry_tid=8936' '!=' 'object_number=144, tag_tid=26, entry_tid=8860' 2024-04-06T09:47:59.261 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:47:59.261 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=144, tag_tid=26, entry_tid=8936' = 'object_number=144, tag_tid=26, entry_tid=8860' 2024-04-06T09:47:59.261 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=144, tag_tid=26, entry_tid=8936' 2024-04-06T09:47:59.261 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=145, tag_tid=27, entry_tid=1' 2024-04-06T09:47:59.261 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:47:59.262 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:47:59.262 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=27 2024-04-06T09:47:59.263 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=144, tag_tid=26, entry_tid=8936' 2024-04-06T09:47:59.263 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:47:59.263 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:47:59.264 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=26 2024-04-06T09:47:59.264 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=145, tag_tid=27, entry_tid=1' 2024-04-06T09:47:59.265 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:47:59.265 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:47:59.265 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:47:59.266 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=144, tag_tid=26, entry_tid=8936' 2024-04-06T09:47:59.266 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:47:59.266 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:47:59.267 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=8936 2024-04-06T09:47:59.267 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 27 = 26 -a 1 -le 8936 2024-04-06T09:47:59.268 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:47:59.268 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:47:59.268 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:47:59.469 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:47:59.469 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:47:59.469 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:47:59.469 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:47:59.470 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:47:59.470 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:47:59.470 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:47:59.470 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:47:59.470 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:47:59.470 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:47:59.470 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:47:59.470 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:47:59.470 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:47:59.470 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:47:59.470 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:47:59.471 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:47:59.471 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:47:59.474 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:47:59.474 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:47:59.474 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:47:59.475 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:47:59.475 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:47:59.475 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:47:59.475 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:47:59.475 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:47:59.475 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:47:59.475 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:47:59.475 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:47:59.475 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:47:59.475 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:47:59.475 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:47:59.475 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:47:59.475 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:47:59.475 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:47:59.475 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:47:59.476 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:47:59.476 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:47:59.476 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:47:59.476 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:47:59.476 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:47:59.476 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:47:59.476 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:47:59.476 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:47:59.476 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:47:59.476 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:47:59.476 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:47:59.476 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:47:59.476 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:47:59.476 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:47:59.502 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:47:59.503 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:47:59.503 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:47:59.503 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:47:59.605 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-06T09:47:59.605 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:rbd mirror flush : flush rbd mirror 2024-04-06T09:47:59.605 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:admin_socket: invalid command 2024-04-06T09:47:59.607 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:47:59.607 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/test 2024-04-06T09:47:59.607 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:1 2024-04-06T09:47:59.607 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:47:59.607 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:47:59.607 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-06T09:47:59.608 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:1 2024-04-06T09:47:59.608 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:47:59.608 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:47:59.608 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:47:59.608 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=1 2024-04-06T09:47:59.608 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-06T09:47:59.608 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:47:59.608 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:47:59.608 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=1 2024-04-06T09:47:59.608 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=1 2024-04-06T09:47:59.609 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-06T09:47:59.610 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:1 2024-04-06T09:47:59.610 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:47:59.610 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:47:59.610 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-06T09:47:59.610 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:1 2024-04-06T09:47:59.610 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:47:59.610 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:47:59.610 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:47:59.610 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=1 2024-04-06T09:47:59.610 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-06T09:47:59.610 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:47:59.610 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:47:59.610 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=1 2024-04-06T09:47:59.611 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=1 2024-04-06T09:47:59.611 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-06T09:47:59.633 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T09:47:59.633 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T09:47:59.633 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T09:47:59.633 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/test 2024-04-06T09:47:59.759 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:47:59.760 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:47:59.760 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:47:59.760 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:47:59.760 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:47:59.760 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:47:59.760 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:47:59.760 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:47:59.761 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:47:59.761 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:47:59.761 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:47:59.761 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:47:59.761 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:47:59.762 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:47:59.763 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:47:59.763 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:47:59.815 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 36 2024-04-06T09:47:59.815 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 36 2024-04-06T09:47:59.815 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:47:59.815 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=145, tag_tid=27, entry_tid=1], [object_number=144, tag_tid=27, entry_tid=0], [object_number=147, tag_tid=26, entry_tid=8935], [object_number=146, tag_tid=26, entry_tid=8934]]], state=connected] 2024-04-06T09:47:59.815 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=145, tag_tid=27, entry_tid=1], [object_number=144, tag_tid=27, entry_tid=0], [object_number=147, tag_tid=26, entry_tid=8935], [object_number=146, tag_tid=26, entry_tid=8934]]], state=connected] 2024-04-06T09:47:59.819 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:47:59.820 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=145, tag_tid=27, entry_tid=1' 2024-04-06T09:47:59.822 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:47:59.822 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:47:59.822 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:47:59.822 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:47:59.822 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:47:59.822 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:47:59.822 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:47:59.822 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:47:59.822 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:47:59.822 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:47:59.822 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:47:59.823 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:47:59.823 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:47:59.823 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:47:59.823 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:47:59.871 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 36 2024-04-06T09:47:59.871 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 36 2024-04-06T09:47:59.871 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:47:59.871 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=145, tag_tid=27, entry_tid=1], [object_number=144, tag_tid=27, entry_tid=0], [object_number=147, tag_tid=26, entry_tid=8935], [object_number=146, tag_tid=26, entry_tid=8934]]], state=connected] 2024-04-06T09:47:59.871 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=145, tag_tid=27, entry_tid=1], [object_number=144, tag_tid=27, entry_tid=0], [object_number=147, tag_tid=26, entry_tid=8935], [object_number=146, tag_tid=26, entry_tid=8934]]], state=connected] 2024-04-06T09:47:59.875 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:47:59.877 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=145, tag_tid=27, entry_tid=1' 2024-04-06T09:47:59.877 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=145, tag_tid=27, entry_tid=1' -a 'object_number=145, tag_tid=27, entry_tid=1' = 'object_number=145, tag_tid=27, entry_tid=1' 2024-04-06T09:47:59.877 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:47:59.877 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_snap_present cluster1 mirror test snap9 2024-04-06T09:47:59.877 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:47:59.877 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:47:59.877 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:47:59.877 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local snap_name=snap9 2024-04-06T09:47:59.877 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local s 2024-04-06T09:47:59.877 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-04-06T09:47:59.878 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 1 2024-04-06T09:48:00.878 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 info mirror/test@snap9 2024-04-06T09:48:00.951 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:rbd image 'test': 2024-04-06T09:48:00.951 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: size 512 MiB in 128 objects 2024-04-06T09:48:00.951 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: order 22 (4 MiB objects) 2024-04-06T09:48:00.951 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: snapshot_count: 17 2024-04-06T09:48:00.951 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: id: 10a56da0b207 2024-04-06T09:48:00.951 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: block_name_prefix: rbd_data.10a56da0b207 2024-04-06T09:48:00.951 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: format: 2 2024-04-06T09:48:00.951 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: features: layering, exclusive-lock, journaling 2024-04-06T09:48:00.952 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: op_features: 2024-04-06T09:48:00.952 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: flags: 2024-04-06T09:48:00.952 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: create_timestamp: Sat Apr 6 09:31:59 2024 2024-04-06T09:48:00.952 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: access_timestamp: Sat Apr 6 09:46:50 2024 2024-04-06T09:48:00.952 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: modify_timestamp: Sat Apr 6 09:47:15 2024 2024-04-06T09:48:00.952 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: protected: False 2024-04-06T09:48:00.952 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: journal: 10a56da0b207 2024-04-06T09:48:00.952 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: mirroring state: enabled 2024-04-06T09:48:00.952 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: mirroring mode: journal 2024-04-06T09:48:00.952 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: mirroring global id: 8ad5dedc-b3e0-425d-8993-b3e2231fc5ca 2024-04-06T09:48:00.952 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: mirroring primary: false 2024-04-06T09:48:00.956 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:48:00.956 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n snap8-clean ']' 2024-04-06T09:48:00.956 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ compare_image_snaps mirror test snap8-clean 2024-04-06T09:48:00.956 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:48:00.957 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:48:00.957 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local snap_name=snap8-clean 2024-04-06T09:48:00.957 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local ret=0 2024-04-06T09:48:00.957 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local rmt_export=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.export 2024-04-06T09:48:00.957 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loc_export=/tmp/tmp.BhW6xKmROU/cluster1-mirror-test.export 2024-04-06T09:48:00.957 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rm -f /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.export /tmp/tmp.BhW6xKmROU/cluster1-mirror-test.export 2024-04-06T09:48:00.958 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster2 -p mirror export test@snap8-clean /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.export 2024-04-06T09:48:01.591 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-04-06T09:48:01.602 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 -p mirror export test@snap8-clean /tmp/tmp.BhW6xKmROU/cluster1-mirror-test.export 2024-04-06T09:48:03.105 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-04-06T09:48:03.128 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmp /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.export /tmp/tmp.BhW6xKmROU/cluster1-mirror-test.export 2024-04-06T09:48:03.411 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rm -f /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.export /tmp/tmp.BhW6xKmROU/cluster1-mirror-test.export 2024-04-06T09:48:03.548 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:48:03.548 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ compare_image_snaps mirror test snap9 2024-04-06T09:48:03.548 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:48:03.548 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:48:03.548 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local snap_name=snap9 2024-04-06T09:48:03.548 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local ret=0 2024-04-06T09:48:03.548 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local rmt_export=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.export 2024-04-06T09:48:03.548 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loc_export=/tmp/tmp.BhW6xKmROU/cluster1-mirror-test.export 2024-04-06T09:48:03.548 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rm -f /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.export /tmp/tmp.BhW6xKmROU/cluster1-mirror-test.export 2024-04-06T09:48:03.549 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster2 -p mirror export test@snap9 /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.export 2024-04-06T09:48:04.187 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-04-06T09:48:04.201 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 -p mirror export test@snap9 /tmp/tmp.BhW6xKmROU/cluster1-mirror-test.export 2024-04-06T09:48:05.780 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-04-06T09:48:05.800 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmp /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.export /tmp/tmp.BhW6xKmROU/cluster1-mirror-test.export 2024-04-06T09:48:06.089 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rm -f /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.export /tmp/tmp.BhW6xKmROU/cluster1-mirror-test.export 2024-04-06T09:48:06.225 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:48:06.225 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ clean_snap_name=snap9-clean 2024-04-06T09:48:06.225 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ create_snap cluster2 mirror test snap9-clean 2024-04-06T09:48:06.226 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-04-06T09:48:06.226 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:48:06.226 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:48:06.226 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local snap_name=snap9-clean 2024-04-06T09:48:06.226 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap9-clean --debug-rbd=20 --debug-journaler=20 2024-04-06T09:48:06.666 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in `seq 1 10` 2024-04-06T09:48:06.666 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ stress_write_image cluster2 mirror test 2024-04-06T09:48:06.666 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-04-06T09:48:06.666 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:48:06.667 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:48:06.667 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-04-06T09:48:06.668 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local duration=9 2024-04-06T09:48:06.668 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set +e 2024-04-06T09:48:06.668 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ timeout 9s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror test --debug-rbd=20 --debug-journaler=20 2024-04-06T09:48:07.568 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:100 IOs, 10954752 bytes 2024-04-06T09:48:12.413 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:200 IOs, 81038336 bytes 2024-04-06T09:48:12.631 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:300 IOs, 81669120 bytes 2024-04-06T09:48:15.436 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:400 IOs, 144034816 bytes 2024-04-06T09:48:15.685 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ error_code=124 2024-04-06T09:48:15.686 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set -e 2024-04-06T09:48:15.686 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 124 -eq 124 ']' 2024-04-06T09:48:15.686 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:48:15.686 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+replaying 2024-04-06T09:48:15.686 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:48:15.686 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:48:15.686 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:48:15.686 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+replaying 2024-04-06T09:48:15.686 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local description_pattern= 2024-04-06T09:48:15.686 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local service_pattern= 2024-04-06T09:48:15.686 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:48:15.686 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 1 2024-04-06T09:48:16.687 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test_status_in_pool_dir cluster1 mirror test up+replaying '' '' 2024-04-06T09:48:16.687 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:48:16.687 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:48:16.687 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:48:16.687 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+replaying 2024-04-06T09:48:16.688 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local description_pattern= 2024-04-06T09:48:16.688 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local service_pattern= 2024-04-06T09:48:16.688 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local status 2024-04-06T09:48:16.688 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ CEPH_ARGS= 2024-04-06T09:48:16.688 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster1 mirror image status mirror/test 2024-04-06T09:48:16.780 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ status='test: 2024-04-06T09:48:16.780 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: global_id: 8ad5dedc-b3e0-425d-8993-b3e2231fc5ca 2024-04-06T09:48:16.780 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: state: up+replaying 2024-04-06T09:48:16.781 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: description: replaying, {"bytes_per_second":2423655.03,"entries_behind_primary":0,"entries_per_second":150.0,"non_primary_position":{"entry_tid":1,"object_number":145,"tag_tid":27},"primary_position":{"entry_tid":1,"object_number":145,"tag_tid":27}} 2024-04-06T09:48:16.781 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: service: mirror.1 on smithi192 2024-04-06T09:48:16.781 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: last_update: 2024-04-06 09:48:11 2024-04-06T09:48:16.781 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: peer_sites: 2024-04-06T09:48:16.781 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: name: cluster2 2024-04-06T09:48:16.781 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: state: up+stopped 2024-04-06T09:48:16.781 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: description: local image is primary 2024-04-06T09:48:16.781 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: last_update: 2024-04-06 09:48:01' 2024-04-06T09:48:16.781 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^ state: .*up+replaying' 2024-04-06T09:48:16.782 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: state: up+replaying 2024-04-06T09:48:16.782 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^ description: .*' 2024-04-06T09:48:16.783 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: description: replaying, {"bytes_per_second":2423655.03,"entries_behind_primary":0,"entries_per_second":150.0,"non_primary_position":{"entry_tid":1,"object_number":145,"tag_tid":27},"primary_position":{"entry_tid":1,"object_number":145,"tag_tid":27}} 2024-04-06T09:48:16.783 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n '' ']' 2024-04-06T09:48:16.783 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ echo up+replaying 2024-04-06T09:48:16.784 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^up+' 2024-04-06T09:48:16.784 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:up+replaying 2024-04-06T09:48:16.784 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep 'service: *mirror..* on ' 2024-04-06T09:48:16.785 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: service: mirror.1 on smithi192 2024-04-06T09:48:16.785 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local last_update 2024-04-06T09:48:16.785 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-04-06T09:48:16.788 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_update='2024-04-06 09:48:11' 2024-04-06T09:48:16.788 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+replaying '2024-04-06 09:48:11' 2024-04-06T09:48:16.788 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:48:16.788 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:48:16.788 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:48:16.788 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+replaying 2024-04-06T09:48:16.788 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'prev_last_update=2024-04-06 09:48:11' 2024-04-06T09:48:16.788 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local status 2024-04-06T09:48:16.788 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ CEPH_ARGS= 2024-04-06T09:48:16.788 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-04-06T09:48:16.875 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ status='OKOKOK142094268mirror.1smithi19218.2.2-1059-g4d05f4catrueOK43194322mirror.3smithi19218.2.2-1059-g4d05f4cafalseOK44654468mirror.0smithi19218.2.2-1059-g4d05f4cafalseOK44954498mirror.2smithi19218.2.2-1059-g4d05f4cafalseOKtest8ad5dedc-b3e0-425d-8993-b3e2231fc5caup+replayingreplaying, {"bytes_per_second":2423655.03,"entries_behind_primary":0,"entries_per_second":150.0,"non_primary_position":{"entry_tid":1,"object_number":145,"tag_tid":27},"primary_position":{"entry_tid":1,"object_number":145,"tag_tid":27}}42094268mirror.1smithi1922024-04-06 09:48:11cluster2c5e3bdc3-c286-4289-8fd0-6b8628b92376up+stoppedlocal image is primary2024-04-06 09:48:01' 2024-04-06T09:48:16.875 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local last_update state 2024-04-06T09:48:16.876 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2024-04-06T09:48:16.880 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_update='2024-04-06 09:48:11' 2024-04-06T09:48:16.880 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2024-04-06T09:48:16.883 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ state=up+replaying 2024-04-06T09:48:16.883 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ echo up+replaying 2024-04-06T09:48:16.884 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep up+replaying 2024-04-06T09:48:16.884 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:up+replaying 2024-04-06T09:48:16.885 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:48:16.885 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:48:16.885 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ snap_name=snap10 2024-04-06T09:48:16.885 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ create_snap cluster2 mirror test snap10 2024-04-06T09:48:16.886 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-04-06T09:48:16.886 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:48:16.886 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:48:16.886 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local snap_name=snap10 2024-04-06T09:48:16.886 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap10 --debug-rbd=20 --debug-journaler=20 2024-04-06T09:48:44.745 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_image_replay_started cluster1 mirror test 2024-04-06T09:48:44.745 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:48:44.745 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:48:44.745 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:48:44.745 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_image_replay_state cluster1 mirror test started 2024-04-06T09:48:44.745 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:48:44.745 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:48:44.745 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:48:44.745 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state=started 2024-04-06T09:48:44.746 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local s 2024-04-06T09:48:44.746 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:48:44.746 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 1 2024-04-06T09:48:45.746 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test_image_replay_state cluster1 mirror test started 2024-04-06T09:48:45.746 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:48:45.746 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:48:45.747 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:48:45.747 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local test_state=started 2024-04-06T09:48:45.747 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local status_result 2024-04-06T09:48:45.747 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local current_state=stopped 2024-04-06T09:48:45.747 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2024-04-06T09:48:45.748 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -i state 2024-04-06T09:48:45.748 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_instance=cluster1 2024-04-06T09:48:45.748 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ shift 2024-04-06T09:48:45.748 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:48:45.748 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance=cluster1 2024-04-06T09:48:45.748 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local loop_instance 2024-04-06T09:48:45.748 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:48:45.748 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sleep 0 2024-04-06T09:48:45.749 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:48:45.750 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ seq 0 3 2024-04-06T09:48:45.752 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:48:45.753 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2024-04-06T09:48:45.753 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1:0 2024-04-06T09:48:45.753 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ shift 2024-04-06T09:48:45.753 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:48:45.753 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:48:45.753 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:48:45.753 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=cluster 2024-04-06T09:48:45.753 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:48:45.753 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:48:45.753 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:48:45.753 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:48:45.753 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval cluster=cluster1 2024-04-06T09:48:45.753 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ cluster=cluster1 2024-04-06T09:48:45.753 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:48:45.754 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:48:45.754 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:48:45.754 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local local_cluster=cluster1:0 2024-04-06T09:48:45.754 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local cluster=cluster1 2024-04-06T09:48:45.754 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local instance 2024-04-06T09:48:45.754 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:48:45.754 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local val=cluster1:0 2024-04-06T09:48:45.754 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local cluster_var_name=local_cluster 2024-04-06T09:48:45.754 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local instance_var_name=instance 2024-04-06T09:48:45.754 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ cluster=cluster1 2024-04-06T09:48:45.754 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:48:45.754 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ '[' 0 = cluster1:0 ']' 2024-04-06T09:48:45.754 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ eval local_cluster=cluster1 2024-04-06T09:48:45.754 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++++ local_cluster=cluster1 2024-04-06T09:48:45.755 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ eval instance=0 2024-04-06T09:48:45.755 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++++ instance=0 2024-04-06T09:48:45.755 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:48:45.778 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:48:45.779 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:48:45.779 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:48:45.779 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2024-04-06T09:48:45.886 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-06T09:48:45.886 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:rbd mirror status : get status for rbd mirror 2024-04-06T09:48:45.886 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:admin_socket: invalid command 2024-04-06T09:48:45.888 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:48:45.888 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/test 2024-04-06T09:48:45.888 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1:1 2024-04-06T09:48:45.888 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ shift 2024-04-06T09:48:45.889 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:48:45.889 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-04-06T09:48:45.889 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:1 2024-04-06T09:48:45.889 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=cluster 2024-04-06T09:48:45.889 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:48:45.889 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:48:45.889 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=1 2024-04-06T09:48:45.889 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-06T09:48:45.889 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval cluster=cluster1 2024-04-06T09:48:45.889 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ cluster=cluster1 2024-04-06T09:48:45.889 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=1 2024-04-06T09:48:45.889 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=1 2024-04-06T09:48:45.891 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-04-06T09:48:45.891 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local local_cluster=cluster1:1 2024-04-06T09:48:45.891 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local cluster=cluster1 2024-04-06T09:48:45.891 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local instance 2024-04-06T09:48:45.891 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-06T09:48:45.891 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local val=cluster1:1 2024-04-06T09:48:45.891 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local cluster_var_name=local_cluster 2024-04-06T09:48:45.891 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local instance_var_name=instance 2024-04-06T09:48:45.891 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ cluster=cluster1 2024-04-06T09:48:45.891 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=1 2024-04-06T09:48:45.891 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ '[' 1 = cluster1:1 ']' 2024-04-06T09:48:45.891 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ eval local_cluster=cluster1 2024-04-06T09:48:45.892 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++++ local_cluster=cluster1 2024-04-06T09:48:45.892 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ eval instance=1 2024-04-06T09:48:45.892 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++++ instance=1 2024-04-06T09:48:45.892 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-06T09:48:45.913 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T09:48:45.914 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T09:48:45.914 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T09:48:45.914 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/test 2024-04-06T09:48:46.026 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ return 0 2024-04-06T09:48:46.027 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ status_result=' "state": "Replaying"' 2024-04-06T09:48:46.027 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ echo ' "state": "Replaying"' 2024-04-06T09:48:46.027 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep -i Replaying 2024-04-06T09:48:46.029 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: "state": "Replaying" 2024-04-06T09:48:46.031 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ current_state=started 2024-04-06T09:48:46.031 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test started = started 2024-04-06T09:48:46.031 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:48:46.031 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror test 2024-04-06T09:48:46.031 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local local_cluster=cluster1 2024-04-06T09:48:46.031 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-04-06T09:48:46.031 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:48:46.031 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:48:46.031 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' journal = journal ']' 2024-04-06T09:48:46.031 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_journal_replay_complete cluster1 cluster2 mirror test 2024-04-06T09:48:46.031 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local local_cluster=cluster1 2024-04-06T09:48:46.032 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-04-06T09:48:46.032 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:48:46.032 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:48:46.032 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local s master_pos mirror_pos last_mirror_pos 2024-04-06T09:48:46.032 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local master_tag master_entry mirror_tag mirror_entry 2024-04-06T09:48:46.032 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:48:46.032 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:48:46.032 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:48:46.232 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:48:46.232 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:48:46.232 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:48:46.233 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:48:46.233 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:48:46.233 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:48:46.233 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:48:46.233 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:48:46.233 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:48:46.233 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:48:46.233 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:48:46.233 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:48:46.233 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:48:46.233 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:48:46.233 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:48:46.233 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:48:46.235 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:48:46.236 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:48:46.236 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:48:46.236 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:48:46.236 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:48:46.236 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:48:46.236 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:48:46.236 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:48:46.236 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:48:46.236 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:48:46.236 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:48:46.236 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:48:46.236 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:48:46.236 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:48:46.236 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:48:46.237 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:48:46.237 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:48:46.237 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:48:46.237 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:48:46.237 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:48:46.237 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:48:46.237 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:48:46.237 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:48:46.237 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:48:46.237 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:48:46.237 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:48:46.237 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:48:46.237 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:48:46.237 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:48:46.238 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:48:46.238 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:48:46.238 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:48:46.238 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:48:46.262 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:48:46.263 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:48:46.263 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:48:46.263 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:48:46.370 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-06T09:48:46.370 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:rbd mirror flush : flush rbd mirror 2024-04-06T09:48:46.370 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:admin_socket: invalid command 2024-04-06T09:48:46.372 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:48:46.372 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/test 2024-04-06T09:48:46.372 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:1 2024-04-06T09:48:46.372 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:48:46.373 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:48:46.373 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-06T09:48:46.373 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:1 2024-04-06T09:48:46.373 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:48:46.373 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:48:46.373 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:48:46.373 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=1 2024-04-06T09:48:46.373 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-06T09:48:46.373 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:48:46.373 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:48:46.373 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=1 2024-04-06T09:48:46.373 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=1 2024-04-06T09:48:46.375 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-06T09:48:46.375 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:1 2024-04-06T09:48:46.375 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:48:46.375 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:48:46.375 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-06T09:48:46.375 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:1 2024-04-06T09:48:46.375 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:48:46.375 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:48:46.375 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:48:46.376 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=1 2024-04-06T09:48:46.376 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-06T09:48:46.376 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:48:46.376 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:48:46.376 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=1 2024-04-06T09:48:46.376 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=1 2024-04-06T09:48:46.376 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-06T09:48:46.398 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T09:48:46.398 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T09:48:46.398 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T09:48:46.398 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/test 2024-04-06T09:48:46.532 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:48:46.533 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:48:46.533 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:48:46.533 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:48:46.533 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:48:46.533 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:48:46.533 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:48:46.534 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:48:46.534 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:48:46.534 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:48:46.534 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:48:46.534 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:48:46.534 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:48:46.536 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:48:46.536 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:48:46.536 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:48:46.591 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 37 2024-04-06T09:48:46.591 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 38 2024-04-06T09:48:46.591 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:48:46.591 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=153, tag_tid=30, entry_tid=1], [object_number=152, tag_tid=30, entry_tid=0], [object_number=155, tag_tid=29, entry_tid=9059], [object_number=154, tag_tid=29, entry_tid=9058]]], state=connected] 2024-04-06T09:48:46.591 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=149, tag_tid=29, entry_tid=5373], [object_number=148, tag_tid=29, entry_tid=5372], [object_number=151, tag_tid=29, entry_tid=5371], [object_number=150, tag_tid=29, entry_tid=5370]]], state=connected] 2024-04-06T09:48:46.595 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:48:46.596 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=153, tag_tid=30, entry_tid=1' 2024-04-06T09:48:46.597 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:48:46.597 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:48:46.597 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:48:46.597 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:48:46.597 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:48:46.597 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:48:46.597 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:48:46.598 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:48:46.598 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:48:46.598 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:48:46.598 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:48:46.598 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:48:46.599 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:48:46.599 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:48:46.599 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:48:46.647 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 37 2024-04-06T09:48:46.647 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 38 2024-04-06T09:48:46.647 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:48:46.647 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=153, tag_tid=30, entry_tid=1], [object_number=152, tag_tid=30, entry_tid=0], [object_number=155, tag_tid=29, entry_tid=9059], [object_number=154, tag_tid=29, entry_tid=9058]]], state=connected] 2024-04-06T09:48:46.648 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=149, tag_tid=29, entry_tid=5373], [object_number=148, tag_tid=29, entry_tid=5372], [object_number=151, tag_tid=29, entry_tid=5371], [object_number=150, tag_tid=29, entry_tid=5370]]], state=connected] 2024-04-06T09:48:46.651 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:48:46.653 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=149, tag_tid=29, entry_tid=5373' 2024-04-06T09:48:46.653 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=153, tag_tid=30, entry_tid=1' -a 'object_number=153, tag_tid=30, entry_tid=1' = 'object_number=149, tag_tid=29, entry_tid=5373' 2024-04-06T09:48:46.653 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=149, tag_tid=29, entry_tid=5373' '!=' '' 2024-04-06T09:48:46.653 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:48:46.653 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=149, tag_tid=29, entry_tid=5373' = '' 2024-04-06T09:48:46.653 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=149, tag_tid=29, entry_tid=5373' 2024-04-06T09:48:46.654 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=153, tag_tid=30, entry_tid=1' 2024-04-06T09:48:46.654 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:48:46.654 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:48:46.654 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=30 2024-04-06T09:48:46.655 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=149, tag_tid=29, entry_tid=5373' 2024-04-06T09:48:46.655 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:48:46.655 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:48:46.656 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=29 2024-04-06T09:48:46.656 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=153, tag_tid=30, entry_tid=1' 2024-04-06T09:48:46.657 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:48:46.657 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:48:46.657 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:48:46.658 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=149, tag_tid=29, entry_tid=5373' 2024-04-06T09:48:46.658 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:48:46.658 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:48:46.659 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=5373 2024-04-06T09:48:46.660 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 30 = 29 -a 1 -le 5373 2024-04-06T09:48:46.660 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:48:46.660 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:48:46.660 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:48:46.861 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:48:46.861 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:48:46.861 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:48:46.861 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:48:46.861 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:48:46.861 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:48:46.862 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:48:46.862 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:48:46.862 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:48:46.862 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:48:46.862 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:48:46.862 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:48:46.862 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:48:46.862 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:48:46.862 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:48:46.862 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:48:46.863 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:48:46.865 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:48:46.865 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:48:46.865 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:48:46.865 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:48:46.865 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:48:46.865 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:48:46.865 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:48:46.865 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:48:46.865 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:48:46.866 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:48:46.866 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:48:46.866 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:48:46.866 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:48:46.866 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:48:46.866 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:48:46.866 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:48:46.867 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:48:46.867 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:48:46.867 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:48:46.867 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:48:46.867 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:48:46.867 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:48:46.867 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:48:46.867 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:48:46.867 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:48:46.868 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:48:46.868 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:48:46.868 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:48:46.868 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:48:46.868 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:48:46.868 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:48:46.868 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:48:46.892 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:48:46.892 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:48:46.892 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:48:46.892 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:48:47.004 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-06T09:48:47.005 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:rbd mirror flush : flush rbd mirror 2024-04-06T09:48:47.005 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:admin_socket: invalid command 2024-04-06T09:48:47.007 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:48:47.007 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/test 2024-04-06T09:48:47.007 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:1 2024-04-06T09:48:47.007 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:48:47.007 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:48:47.007 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-06T09:48:47.007 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:1 2024-04-06T09:48:47.008 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:48:47.008 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:48:47.008 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:48:47.008 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=1 2024-04-06T09:48:47.008 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-06T09:48:47.008 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:48:47.008 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:48:47.008 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=1 2024-04-06T09:48:47.008 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=1 2024-04-06T09:48:47.009 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-06T09:48:47.009 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:1 2024-04-06T09:48:47.010 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:48:47.010 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:48:47.010 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-06T09:48:47.010 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:1 2024-04-06T09:48:47.010 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:48:47.010 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:48:47.010 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:48:47.010 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=1 2024-04-06T09:48:47.010 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-06T09:48:47.010 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:48:47.010 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:48:47.010 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=1 2024-04-06T09:48:47.010 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=1 2024-04-06T09:48:47.010 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-06T09:48:47.035 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T09:48:47.035 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T09:48:47.035 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T09:48:47.036 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/test 2024-04-06T09:48:47.160 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:48:47.161 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:48:47.161 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:48:47.162 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:48:47.162 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:48:47.162 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:48:47.162 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:48:47.162 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:48:47.162 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:48:47.162 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:48:47.163 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:48:47.163 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:48:47.163 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:48:47.164 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:48:47.164 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:48:47.164 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:48:47.425 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 37 2024-04-06T09:48:47.425 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 38 2024-04-06T09:48:47.425 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:48:47.425 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=153, tag_tid=30, entry_tid=1], [object_number=152, tag_tid=30, entry_tid=0], [object_number=155, tag_tid=29, entry_tid=9059], [object_number=154, tag_tid=29, entry_tid=9058]]], state=connected] 2024-04-06T09:48:47.425 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=148, tag_tid=29, entry_tid=5480], [object_number=151, tag_tid=29, entry_tid=5479], [object_number=150, tag_tid=29, entry_tid=5478], [object_number=149, tag_tid=29, entry_tid=5477]]], state=connected] 2024-04-06T09:48:47.431 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:48:47.433 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=153, tag_tid=30, entry_tid=1' 2024-04-06T09:48:47.434 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:48:47.434 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:48:47.435 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:48:47.435 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:48:47.435 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:48:47.435 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:48:47.435 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:48:47.435 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:48:47.435 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:48:47.435 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:48:47.436 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:48:47.436 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:48:47.436 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:48:47.437 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:48:47.437 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:48:47.491 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 37 2024-04-06T09:48:47.491 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 38 2024-04-06T09:48:47.491 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:48:47.491 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=153, tag_tid=30, entry_tid=1], [object_number=152, tag_tid=30, entry_tid=0], [object_number=155, tag_tid=29, entry_tid=9059], [object_number=154, tag_tid=29, entry_tid=9058]]], state=connected] 2024-04-06T09:48:47.491 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=148, tag_tid=29, entry_tid=5480], [object_number=151, tag_tid=29, entry_tid=5479], [object_number=150, tag_tid=29, entry_tid=5478], [object_number=149, tag_tid=29, entry_tid=5477]]], state=connected] 2024-04-06T09:48:47.495 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:48:47.497 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=148, tag_tid=29, entry_tid=5480' 2024-04-06T09:48:47.497 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=153, tag_tid=30, entry_tid=1' -a 'object_number=153, tag_tid=30, entry_tid=1' = 'object_number=148, tag_tid=29, entry_tid=5480' 2024-04-06T09:48:47.497 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=148, tag_tid=29, entry_tid=5480' '!=' 'object_number=149, tag_tid=29, entry_tid=5373' 2024-04-06T09:48:47.497 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:48:47.497 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=148, tag_tid=29, entry_tid=5480' = 'object_number=149, tag_tid=29, entry_tid=5373' 2024-04-06T09:48:47.497 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=148, tag_tid=29, entry_tid=5480' 2024-04-06T09:48:47.498 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=153, tag_tid=30, entry_tid=1' 2024-04-06T09:48:47.498 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:48:47.498 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:48:47.498 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=30 2024-04-06T09:48:47.499 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=148, tag_tid=29, entry_tid=5480' 2024-04-06T09:48:47.499 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:48:47.499 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:48:47.500 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=29 2024-04-06T09:48:47.501 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=153, tag_tid=30, entry_tid=1' 2024-04-06T09:48:47.501 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:48:47.501 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:48:47.502 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:48:47.503 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=148, tag_tid=29, entry_tid=5480' 2024-04-06T09:48:47.503 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:48:47.504 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:48:47.505 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=5480 2024-04-06T09:48:47.505 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 30 = 29 -a 1 -le 5480 2024-04-06T09:48:47.505 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:48:47.505 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:48:47.505 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:48:47.707 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:48:47.707 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:48:47.707 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:48:47.707 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:48:47.707 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:48:47.707 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:48:47.707 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:48:47.707 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:48:47.708 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:48:47.708 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:48:47.708 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:48:47.708 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:48:47.708 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:48:47.708 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:48:47.708 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:48:47.708 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:48:47.709 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:48:47.712 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:48:47.713 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:48:47.713 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:48:47.713 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:48:47.713 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:48:47.713 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:48:47.713 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:48:47.713 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:48:47.713 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:48:47.713 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:48:47.713 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:48:47.713 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:48:47.714 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:48:47.714 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:48:47.714 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:48:47.714 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:48:47.714 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:48:47.714 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:48:47.714 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:48:47.714 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:48:47.714 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:48:47.714 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:48:47.714 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:48:47.714 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:48:47.714 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:48:47.714 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:48:47.715 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:48:47.715 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:48:47.715 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:48:47.715 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:48:47.715 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:48:47.715 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:48:47.738 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:48:47.739 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:48:47.739 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:48:47.739 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:48:47.844 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-06T09:48:47.844 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:rbd mirror flush : flush rbd mirror 2024-04-06T09:48:47.844 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:admin_socket: invalid command 2024-04-06T09:48:47.847 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:48:47.847 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/test 2024-04-06T09:48:47.847 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:1 2024-04-06T09:48:47.847 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:48:47.847 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:48:47.847 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-06T09:48:47.847 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:1 2024-04-06T09:48:47.847 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:48:47.847 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:48:47.847 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:48:47.847 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=1 2024-04-06T09:48:47.848 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-06T09:48:47.848 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:48:47.848 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:48:47.848 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=1 2024-04-06T09:48:47.848 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=1 2024-04-06T09:48:47.848 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-06T09:48:47.848 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:1 2024-04-06T09:48:47.848 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:48:47.848 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:48:47.848 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-06T09:48:47.848 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:1 2024-04-06T09:48:47.848 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:48:47.848 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:48:47.848 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:48:47.848 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=1 2024-04-06T09:48:47.849 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-06T09:48:47.849 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:48:47.849 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:48:47.849 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=1 2024-04-06T09:48:47.849 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=1 2024-04-06T09:48:47.849 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-06T09:48:47.866 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T09:48:47.867 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T09:48:47.867 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T09:48:47.867 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/test 2024-04-06T09:48:47.988 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:48:47.989 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:48:47.989 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:48:47.989 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:48:47.990 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:48:47.990 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:48:47.990 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:48:47.990 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:48:47.990 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:48:47.990 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:48:47.990 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:48:47.990 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:48:47.990 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:48:47.991 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:48:47.991 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:48:47.992 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:48:48.061 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 37 2024-04-06T09:48:48.061 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 38 2024-04-06T09:48:48.061 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:48:48.062 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=153, tag_tid=30, entry_tid=1], [object_number=152, tag_tid=30, entry_tid=0], [object_number=155, tag_tid=29, entry_tid=9059], [object_number=154, tag_tid=29, entry_tid=9058]]], state=connected] 2024-04-06T09:48:48.062 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=150, tag_tid=29, entry_tid=5610], [object_number=149, tag_tid=29, entry_tid=5609], [object_number=148, tag_tid=29, entry_tid=5608], [object_number=151, tag_tid=29, entry_tid=5607]]], state=connected] 2024-04-06T09:48:48.065 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:48:48.066 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=153, tag_tid=30, entry_tid=1' 2024-04-06T09:48:48.068 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:48:48.068 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:48:48.068 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:48:48.068 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:48:48.068 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:48:48.068 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:48:48.068 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:48:48.068 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:48:48.069 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:48:48.069 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:48:48.069 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:48:48.069 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:48:48.069 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:48:48.069 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:48:48.070 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:48:48.122 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 37 2024-04-06T09:48:48.123 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 38 2024-04-06T09:48:48.123 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:48:48.123 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=153, tag_tid=30, entry_tid=1], [object_number=152, tag_tid=30, entry_tid=0], [object_number=155, tag_tid=29, entry_tid=9059], [object_number=154, tag_tid=29, entry_tid=9058]]], state=connected] 2024-04-06T09:48:48.123 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=150, tag_tid=29, entry_tid=5610], [object_number=149, tag_tid=29, entry_tid=5609], [object_number=148, tag_tid=29, entry_tid=5608], [object_number=151, tag_tid=29, entry_tid=5607]]], state=connected] 2024-04-06T09:48:48.128 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:48:48.130 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=150, tag_tid=29, entry_tid=5610' 2024-04-06T09:48:48.130 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=153, tag_tid=30, entry_tid=1' -a 'object_number=153, tag_tid=30, entry_tid=1' = 'object_number=150, tag_tid=29, entry_tid=5610' 2024-04-06T09:48:48.130 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=150, tag_tid=29, entry_tid=5610' '!=' 'object_number=148, tag_tid=29, entry_tid=5480' 2024-04-06T09:48:48.130 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:48:48.130 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=150, tag_tid=29, entry_tid=5610' = 'object_number=148, tag_tid=29, entry_tid=5480' 2024-04-06T09:48:48.130 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=150, tag_tid=29, entry_tid=5610' 2024-04-06T09:48:48.130 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=153, tag_tid=30, entry_tid=1' 2024-04-06T09:48:48.130 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:48:48.130 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:48:48.131 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=30 2024-04-06T09:48:48.132 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=150, tag_tid=29, entry_tid=5610' 2024-04-06T09:48:48.132 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:48:48.132 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:48:48.133 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=29 2024-04-06T09:48:48.133 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=153, tag_tid=30, entry_tid=1' 2024-04-06T09:48:48.134 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:48:48.134 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:48:48.135 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:48:48.135 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=150, tag_tid=29, entry_tid=5610' 2024-04-06T09:48:48.135 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:48:48.136 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:48:48.137 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=5610 2024-04-06T09:48:48.137 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 30 = 29 -a 1 -le 5610 2024-04-06T09:48:48.138 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:48:48.138 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:48:48.138 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:48:48.339 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:48:48.340 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:48:48.340 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:48:48.340 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:48:48.340 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:48:48.340 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:48:48.340 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:48:48.340 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:48:48.340 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:48:48.340 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:48:48.340 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:48:48.340 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:48:48.340 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:48:48.341 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:48:48.341 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:48:48.341 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:48:48.341 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:48:48.344 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:48:48.344 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:48:48.344 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:48:48.344 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:48:48.344 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:48:48.344 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:48:48.345 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:48:48.345 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:48:48.345 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:48:48.345 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:48:48.345 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:48:48.345 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:48:48.345 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:48:48.345 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:48:48.345 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:48:48.345 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:48:48.345 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:48:48.345 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:48:48.345 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:48:48.345 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:48:48.346 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:48:48.346 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:48:48.346 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:48:48.346 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:48:48.346 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:48:48.346 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:48:48.346 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:48:48.346 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:48:48.346 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:48:48.346 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:48:48.346 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:48:48.346 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:48:48.370 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:48:48.370 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:48:48.370 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:48:48.370 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:48:48.476 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-06T09:48:48.476 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:rbd mirror flush : flush rbd mirror 2024-04-06T09:48:48.476 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:admin_socket: invalid command 2024-04-06T09:48:48.478 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:48:48.478 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/test 2024-04-06T09:48:48.478 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:1 2024-04-06T09:48:48.478 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:48:48.478 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:48:48.479 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-06T09:48:48.479 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:1 2024-04-06T09:48:48.479 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:48:48.479 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:48:48.479 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:48:48.479 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=1 2024-04-06T09:48:48.479 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-06T09:48:48.479 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:48:48.479 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:48:48.479 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=1 2024-04-06T09:48:48.479 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=1 2024-04-06T09:48:48.480 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-06T09:48:48.480 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:1 2024-04-06T09:48:48.480 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:48:48.480 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:48:48.480 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-06T09:48:48.480 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:1 2024-04-06T09:48:48.480 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:48:48.481 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:48:48.481 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:48:48.481 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=1 2024-04-06T09:48:48.481 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-06T09:48:48.481 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:48:48.481 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:48:48.481 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=1 2024-04-06T09:48:48.481 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=1 2024-04-06T09:48:48.481 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-06T09:48:48.504 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T09:48:48.504 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T09:48:48.504 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T09:48:48.505 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/test 2024-04-06T09:48:48.642 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:48:48.643 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:48:48.643 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:48:48.643 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:48:48.643 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:48:48.643 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:48:48.644 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:48:48.644 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:48:48.644 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:48:48.644 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:48:48.644 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:48:48.644 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:48:48.644 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:48:48.646 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:48:48.646 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:48:48.646 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:48:48.715 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 37 2024-04-06T09:48:48.715 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 38 2024-04-06T09:48:48.715 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:48:48.715 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=153, tag_tid=30, entry_tid=1], [object_number=152, tag_tid=30, entry_tid=0], [object_number=155, tag_tid=29, entry_tid=9059], [object_number=154, tag_tid=29, entry_tid=9058]]], state=connected] 2024-04-06T09:48:48.716 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=153, tag_tid=29, entry_tid=5693], [object_number=152, tag_tid=29, entry_tid=5692], [object_number=155, tag_tid=29, entry_tid=5691], [object_number=154, tag_tid=29, entry_tid=5690]]], state=connected] 2024-04-06T09:48:48.719 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:48:48.721 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=153, tag_tid=30, entry_tid=1' 2024-04-06T09:48:48.722 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:48:48.722 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:48:48.722 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:48:48.722 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:48:48.722 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:48:48.722 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:48:48.723 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:48:48.723 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:48:48.723 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:48:48.723 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:48:48.723 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:48:48.723 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:48:48.723 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:48:48.724 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:48:48.724 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:48:48.771 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 38 2024-04-06T09:48:48.771 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 38 2024-04-06T09:48:48.771 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:48:48.771 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=153, tag_tid=30, entry_tid=1], [object_number=152, tag_tid=30, entry_tid=0], [object_number=155, tag_tid=29, entry_tid=9059], [object_number=154, tag_tid=29, entry_tid=9058]]], state=connected] 2024-04-06T09:48:48.772 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=153, tag_tid=29, entry_tid=5693], [object_number=152, tag_tid=29, entry_tid=5692], [object_number=155, tag_tid=29, entry_tid=5691], [object_number=154, tag_tid=29, entry_tid=5690]]], state=connected] 2024-04-06T09:48:48.775 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:48:48.777 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=153, tag_tid=29, entry_tid=5693' 2024-04-06T09:48:48.777 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=153, tag_tid=30, entry_tid=1' -a 'object_number=153, tag_tid=30, entry_tid=1' = 'object_number=153, tag_tid=29, entry_tid=5693' 2024-04-06T09:48:48.777 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=153, tag_tid=29, entry_tid=5693' '!=' 'object_number=150, tag_tid=29, entry_tid=5610' 2024-04-06T09:48:48.777 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:48:48.777 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=153, tag_tid=29, entry_tid=5693' = 'object_number=150, tag_tid=29, entry_tid=5610' 2024-04-06T09:48:48.777 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=153, tag_tid=29, entry_tid=5693' 2024-04-06T09:48:48.777 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=153, tag_tid=30, entry_tid=1' 2024-04-06T09:48:48.777 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:48:48.777 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:48:48.778 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=30 2024-04-06T09:48:48.779 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=153, tag_tid=29, entry_tid=5693' 2024-04-06T09:48:48.779 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:48:48.779 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:48:48.780 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=29 2024-04-06T09:48:48.780 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=153, tag_tid=30, entry_tid=1' 2024-04-06T09:48:48.780 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:48:48.780 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:48:48.781 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:48:48.782 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=153, tag_tid=29, entry_tid=5693' 2024-04-06T09:48:48.782 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:48:48.782 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:48:48.784 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=5693 2024-04-06T09:48:48.784 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 30 = 29 -a 1 -le 5693 2024-04-06T09:48:48.784 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:48:48.784 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:48:48.784 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:48:48.985 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:48:48.985 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:48:48.985 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:48:48.986 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:48:48.986 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:48:48.986 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:48:48.986 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:48:48.986 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:48:48.986 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:48:48.986 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:48:48.986 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:48:48.986 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:48:48.986 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:48:48.986 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:48:48.986 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:48:48.987 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:48:48.987 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:48:48.990 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:48:48.990 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:48:48.990 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:48:48.990 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:48:48.990 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:48:48.990 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:48:48.990 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:48:48.991 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:48:48.991 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:48:48.991 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:48:48.991 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:48:48.991 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:48:48.991 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:48:48.991 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:48:48.991 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:48:48.991 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:48:48.991 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:48:48.991 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:48:48.991 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:48:48.991 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:48:48.991 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:48:48.991 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:48:48.992 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:48:48.992 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:48:48.992 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:48:48.992 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:48:48.992 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:48:48.992 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:48:48.992 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:48:48.992 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:48:48.992 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:48:48.992 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:48:49.016 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:48:49.016 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:48:49.016 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:48:49.016 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:48:49.123 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-06T09:48:49.123 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:rbd mirror flush : flush rbd mirror 2024-04-06T09:48:49.124 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:admin_socket: invalid command 2024-04-06T09:48:49.125 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:48:49.126 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/test 2024-04-06T09:48:49.126 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:1 2024-04-06T09:48:49.126 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:48:49.126 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:48:49.126 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-06T09:48:49.126 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:1 2024-04-06T09:48:49.126 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:48:49.126 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:48:49.126 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:48:49.126 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=1 2024-04-06T09:48:49.126 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-06T09:48:49.126 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:48:49.126 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:48:49.127 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=1 2024-04-06T09:48:49.127 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=1 2024-04-06T09:48:49.128 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-06T09:48:49.128 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:1 2024-04-06T09:48:49.128 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:48:49.128 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:48:49.128 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-06T09:48:49.128 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:1 2024-04-06T09:48:49.128 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:48:49.128 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:48:49.128 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:48:49.128 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=1 2024-04-06T09:48:49.128 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-06T09:48:49.129 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:48:49.129 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:48:49.129 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=1 2024-04-06T09:48:49.129 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=1 2024-04-06T09:48:49.129 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-06T09:48:49.152 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T09:48:49.153 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T09:48:49.153 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T09:48:49.153 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/test 2024-04-06T09:48:49.273 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:48:49.274 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:48:49.274 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:48:49.275 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:48:49.275 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:48:49.275 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:48:49.275 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:48:49.275 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:48:49.275 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:48:49.275 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:48:49.275 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:48:49.275 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:48:49.276 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:48:49.277 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:48:49.278 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:48:49.278 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:48:49.327 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 38 2024-04-06T09:48:49.327 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 38 2024-04-06T09:48:49.327 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:48:49.327 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=153, tag_tid=30, entry_tid=1], [object_number=152, tag_tid=30, entry_tid=0], [object_number=155, tag_tid=29, entry_tid=9059], [object_number=154, tag_tid=29, entry_tid=9058]]], state=connected] 2024-04-06T09:48:49.327 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=155, tag_tid=29, entry_tid=5815], [object_number=154, tag_tid=29, entry_tid=5814], [object_number=153, tag_tid=29, entry_tid=5813], [object_number=152, tag_tid=29, entry_tid=5812]]], state=connected] 2024-04-06T09:48:49.330 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:48:49.331 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=153, tag_tid=30, entry_tid=1' 2024-04-06T09:48:49.333 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:48:49.333 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:48:49.333 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:48:49.333 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:48:49.333 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:48:49.333 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:48:49.333 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:48:49.333 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:48:49.333 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:48:49.333 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:48:49.333 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:48:49.333 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:48:49.334 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:48:49.334 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:48:49.334 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:48:49.382 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 38 2024-04-06T09:48:49.382 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 38 2024-04-06T09:48:49.382 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:48:49.383 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=153, tag_tid=30, entry_tid=1], [object_number=152, tag_tid=30, entry_tid=0], [object_number=155, tag_tid=29, entry_tid=9059], [object_number=154, tag_tid=29, entry_tid=9058]]], state=connected] 2024-04-06T09:48:49.383 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=155, tag_tid=29, entry_tid=5815], [object_number=154, tag_tid=29, entry_tid=5814], [object_number=153, tag_tid=29, entry_tid=5813], [object_number=152, tag_tid=29, entry_tid=5812]]], state=connected] 2024-04-06T09:48:49.386 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:48:49.387 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=155, tag_tid=29, entry_tid=5815' 2024-04-06T09:48:49.387 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=153, tag_tid=30, entry_tid=1' -a 'object_number=153, tag_tid=30, entry_tid=1' = 'object_number=155, tag_tid=29, entry_tid=5815' 2024-04-06T09:48:49.388 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=155, tag_tid=29, entry_tid=5815' '!=' 'object_number=153, tag_tid=29, entry_tid=5693' 2024-04-06T09:48:49.388 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:48:49.388 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=155, tag_tid=29, entry_tid=5815' = 'object_number=153, tag_tid=29, entry_tid=5693' 2024-04-06T09:48:49.388 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=155, tag_tid=29, entry_tid=5815' 2024-04-06T09:48:49.388 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=153, tag_tid=30, entry_tid=1' 2024-04-06T09:48:49.388 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:48:49.389 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:48:49.389 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=30 2024-04-06T09:48:49.390 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=155, tag_tid=29, entry_tid=5815' 2024-04-06T09:48:49.390 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:48:49.390 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:48:49.391 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=29 2024-04-06T09:48:49.391 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=153, tag_tid=30, entry_tid=1' 2024-04-06T09:48:49.391 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:48:49.391 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:48:49.392 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:48:49.393 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=155, tag_tid=29, entry_tid=5815' 2024-04-06T09:48:49.393 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:48:49.393 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:48:49.394 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=5815 2024-04-06T09:48:49.395 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 30 = 29 -a 1 -le 5815 2024-04-06T09:48:49.395 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:48:49.395 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:48:49.395 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:48:49.596 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:48:49.596 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:48:49.596 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:48:49.596 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:48:49.597 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:48:49.597 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:48:49.597 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:48:49.597 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:48:49.597 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:48:49.597 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:48:49.597 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:48:49.597 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:48:49.597 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:48:49.597 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:48:49.597 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:48:49.598 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:48:49.598 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:48:49.600 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:48:49.600 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:48:49.601 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:48:49.601 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:48:49.601 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:48:49.601 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:48:49.601 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:48:49.601 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:48:49.601 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:48:49.601 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:48:49.601 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:48:49.601 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:48:49.601 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:48:49.601 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:48:49.601 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:48:49.601 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:48:49.602 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:48:49.602 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:48:49.602 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:48:49.602 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:48:49.602 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:48:49.602 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:48:49.602 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:48:49.602 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:48:49.602 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:48:49.602 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:48:49.602 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:48:49.602 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:48:49.602 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:48:49.602 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:48:49.602 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:48:49.603 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:48:49.627 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:48:49.627 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:48:49.628 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:48:49.628 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:48:49.733 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-06T09:48:49.733 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:rbd mirror flush : flush rbd mirror 2024-04-06T09:48:49.733 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:admin_socket: invalid command 2024-04-06T09:48:49.735 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:48:49.735 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/test 2024-04-06T09:48:49.735 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:1 2024-04-06T09:48:49.735 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:48:49.735 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:48:49.735 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-06T09:48:49.736 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:1 2024-04-06T09:48:49.736 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:48:49.736 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:48:49.736 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:48:49.736 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=1 2024-04-06T09:48:49.736 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-06T09:48:49.736 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:48:49.736 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:48:49.736 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=1 2024-04-06T09:48:49.736 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=1 2024-04-06T09:48:49.737 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-06T09:48:49.737 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:1 2024-04-06T09:48:49.738 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:48:49.738 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:48:49.738 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-06T09:48:49.738 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:1 2024-04-06T09:48:49.738 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:48:49.738 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:48:49.738 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:48:49.738 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=1 2024-04-06T09:48:49.738 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-06T09:48:49.738 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:48:49.738 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:48:49.738 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=1 2024-04-06T09:48:49.738 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=1 2024-04-06T09:48:49.739 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-06T09:48:49.761 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T09:48:49.762 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T09:48:49.762 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T09:48:49.762 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/test 2024-04-06T09:48:49.887 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:48:49.888 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:48:49.888 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:48:49.888 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:48:49.888 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:48:49.888 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:48:49.888 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:48:49.888 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:48:49.888 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:48:49.888 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:48:49.889 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:48:49.889 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:48:49.889 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:48:49.891 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:48:49.891 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:48:49.891 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:48:49.939 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 38 2024-04-06T09:48:49.939 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 38 2024-04-06T09:48:49.940 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:48:49.940 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=153, tag_tid=30, entry_tid=1], [object_number=152, tag_tid=30, entry_tid=0], [object_number=155, tag_tid=29, entry_tid=9059], [object_number=154, tag_tid=29, entry_tid=9058]]], state=connected] 2024-04-06T09:48:49.940 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=155, tag_tid=29, entry_tid=5943], [object_number=154, tag_tid=29, entry_tid=5942], [object_number=153, tag_tid=29, entry_tid=5941], [object_number=152, tag_tid=29, entry_tid=5940]]], state=connected] 2024-04-06T09:48:49.943 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:48:49.944 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=153, tag_tid=30, entry_tid=1' 2024-04-06T09:48:49.945 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:48:49.945 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:48:49.946 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:48:49.946 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:48:49.946 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:48:49.946 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:48:49.946 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:48:49.946 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:48:49.946 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:48:49.946 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:48:49.946 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:48:49.946 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:48:49.946 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:48:49.947 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:48:49.947 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:48:49.995 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 38 2024-04-06T09:48:49.996 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 38 2024-04-06T09:48:49.996 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:48:49.996 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=153, tag_tid=30, entry_tid=1], [object_number=152, tag_tid=30, entry_tid=0], [object_number=155, tag_tid=29, entry_tid=9059], [object_number=154, tag_tid=29, entry_tid=9058]]], state=connected] 2024-04-06T09:48:49.996 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=155, tag_tid=29, entry_tid=5943], [object_number=154, tag_tid=29, entry_tid=5942], [object_number=153, tag_tid=29, entry_tid=5941], [object_number=152, tag_tid=29, entry_tid=5940]]], state=connected] 2024-04-06T09:48:50.001 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:48:50.003 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=155, tag_tid=29, entry_tid=5943' 2024-04-06T09:48:50.003 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=153, tag_tid=30, entry_tid=1' -a 'object_number=153, tag_tid=30, entry_tid=1' = 'object_number=155, tag_tid=29, entry_tid=5943' 2024-04-06T09:48:50.003 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=155, tag_tid=29, entry_tid=5943' '!=' 'object_number=155, tag_tid=29, entry_tid=5815' 2024-04-06T09:48:50.003 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:48:50.003 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=155, tag_tid=29, entry_tid=5943' = 'object_number=155, tag_tid=29, entry_tid=5815' 2024-04-06T09:48:50.003 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=155, tag_tid=29, entry_tid=5943' 2024-04-06T09:48:50.004 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=153, tag_tid=30, entry_tid=1' 2024-04-06T09:48:50.004 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:48:50.004 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:48:50.004 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=30 2024-04-06T09:48:50.005 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=155, tag_tid=29, entry_tid=5943' 2024-04-06T09:48:50.005 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:48:50.005 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:48:50.006 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=29 2024-04-06T09:48:50.007 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=153, tag_tid=30, entry_tid=1' 2024-04-06T09:48:50.007 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:48:50.007 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:48:50.008 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:48:50.008 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=155, tag_tid=29, entry_tid=5943' 2024-04-06T09:48:50.008 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:48:50.009 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:48:50.010 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=5943 2024-04-06T09:48:50.010 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 30 = 29 -a 1 -le 5943 2024-04-06T09:48:50.010 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:48:50.010 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:48:50.010 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:48:50.211 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:48:50.212 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:48:50.212 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:48:50.212 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:48:50.212 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:48:50.212 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:48:50.212 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:48:50.212 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:48:50.212 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:48:50.212 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:48:50.212 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:48:50.212 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:48:50.212 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:48:50.212 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:48:50.212 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:48:50.213 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:48:50.213 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:48:50.216 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:48:50.216 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:48:50.216 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:48:50.216 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:48:50.216 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:48:50.216 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:48:50.216 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:48:50.216 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:48:50.216 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:48:50.217 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:48:50.217 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:48:50.217 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:48:50.217 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:48:50.217 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:48:50.217 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:48:50.217 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:48:50.217 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:48:50.217 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:48:50.217 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:48:50.217 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:48:50.217 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:48:50.217 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:48:50.217 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:48:50.218 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:48:50.218 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:48:50.218 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:48:50.218 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:48:50.218 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:48:50.218 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:48:50.218 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:48:50.218 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:48:50.218 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:48:50.241 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:48:50.242 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:48:50.242 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:48:50.242 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:48:50.343 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-06T09:48:50.343 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:rbd mirror flush : flush rbd mirror 2024-04-06T09:48:50.343 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:admin_socket: invalid command 2024-04-06T09:48:50.345 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:48:50.345 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/test 2024-04-06T09:48:50.345 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:1 2024-04-06T09:48:50.345 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:48:50.345 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:48:50.346 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-06T09:48:50.346 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:1 2024-04-06T09:48:50.346 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:48:50.346 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:48:50.346 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:48:50.346 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=1 2024-04-06T09:48:50.346 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-06T09:48:50.346 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:48:50.346 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:48:50.346 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=1 2024-04-06T09:48:50.346 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=1 2024-04-06T09:48:50.347 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-06T09:48:50.348 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:1 2024-04-06T09:48:50.348 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:48:50.348 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:48:50.348 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-06T09:48:50.348 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:1 2024-04-06T09:48:50.348 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:48:50.348 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:48:50.348 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:48:50.348 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=1 2024-04-06T09:48:50.348 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-06T09:48:50.348 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:48:50.348 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:48:50.348 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=1 2024-04-06T09:48:50.349 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=1 2024-04-06T09:48:50.349 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-06T09:48:50.365 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T09:48:50.366 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T09:48:50.366 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T09:48:50.366 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/test 2024-04-06T09:48:50.497 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:48:50.498 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:48:50.498 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:48:50.498 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:48:50.498 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:48:50.498 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:48:50.498 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:48:50.499 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:48:50.499 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:48:50.499 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:48:50.499 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:48:50.499 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:48:50.499 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:48:50.502 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:48:50.502 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:48:50.502 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:48:50.555 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 38 2024-04-06T09:48:50.555 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 38 2024-04-06T09:48:50.555 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:48:50.555 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=153, tag_tid=30, entry_tid=1], [object_number=152, tag_tid=30, entry_tid=0], [object_number=155, tag_tid=29, entry_tid=9059], [object_number=154, tag_tid=29, entry_tid=9058]]], state=connected] 2024-04-06T09:48:50.556 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=153, tag_tid=29, entry_tid=6081], [object_number=152, tag_tid=29, entry_tid=6080], [object_number=155, tag_tid=29, entry_tid=6079], [object_number=154, tag_tid=29, entry_tid=6078]]], state=connected] 2024-04-06T09:48:50.559 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:48:50.560 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=153, tag_tid=30, entry_tid=1' 2024-04-06T09:48:50.561 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:48:50.561 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:48:50.561 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:48:50.562 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:48:50.562 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:48:50.562 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:48:50.562 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:48:50.562 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:48:50.562 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:48:50.562 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:48:50.562 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:48:50.562 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:48:50.563 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:48:50.563 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:48:50.563 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:48:50.610 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 38 2024-04-06T09:48:50.611 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 38 2024-04-06T09:48:50.611 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:48:50.611 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=153, tag_tid=30, entry_tid=1], [object_number=152, tag_tid=30, entry_tid=0], [object_number=155, tag_tid=29, entry_tid=9059], [object_number=154, tag_tid=29, entry_tid=9058]]], state=connected] 2024-04-06T09:48:50.611 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=153, tag_tid=29, entry_tid=6081], [object_number=152, tag_tid=29, entry_tid=6080], [object_number=155, tag_tid=29, entry_tid=6079], [object_number=154, tag_tid=29, entry_tid=6078]]], state=connected] 2024-04-06T09:48:50.614 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:48:50.616 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=153, tag_tid=29, entry_tid=6081' 2024-04-06T09:48:50.616 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=153, tag_tid=30, entry_tid=1' -a 'object_number=153, tag_tid=30, entry_tid=1' = 'object_number=153, tag_tid=29, entry_tid=6081' 2024-04-06T09:48:50.616 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=153, tag_tid=29, entry_tid=6081' '!=' 'object_number=155, tag_tid=29, entry_tid=5943' 2024-04-06T09:48:50.616 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:48:50.616 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=153, tag_tid=29, entry_tid=6081' = 'object_number=155, tag_tid=29, entry_tid=5943' 2024-04-06T09:48:50.616 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=153, tag_tid=29, entry_tid=6081' 2024-04-06T09:48:50.617 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=153, tag_tid=30, entry_tid=1' 2024-04-06T09:48:50.617 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:48:50.617 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:48:50.617 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=30 2024-04-06T09:48:50.618 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=153, tag_tid=29, entry_tid=6081' 2024-04-06T09:48:50.618 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:48:50.618 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:48:50.619 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=29 2024-04-06T09:48:50.620 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=153, tag_tid=30, entry_tid=1' 2024-04-06T09:48:50.620 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:48:50.620 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:48:50.621 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:48:50.621 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=153, tag_tid=29, entry_tid=6081' 2024-04-06T09:48:50.621 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:48:50.621 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:48:50.623 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=6081 2024-04-06T09:48:50.623 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 30 = 29 -a 1 -le 6081 2024-04-06T09:48:50.623 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:48:50.623 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:48:50.623 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:48:50.824 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:48:50.825 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:48:50.825 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:48:50.825 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:48:50.825 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:48:50.825 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:48:50.825 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:48:50.825 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:48:50.825 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:48:50.825 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:48:50.825 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:48:50.825 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:48:50.826 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:48:50.826 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:48:50.826 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:48:50.826 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:48:50.827 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:48:50.830 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:48:50.831 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:48:50.831 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:48:50.831 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:48:50.831 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:48:50.831 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:48:50.831 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:48:50.831 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:48:50.831 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:48:50.831 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:48:50.831 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:48:50.831 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:48:50.831 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:48:50.831 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:48:50.831 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:48:50.832 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:48:50.832 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:48:50.832 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:48:50.832 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:48:50.832 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:48:50.832 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:48:50.832 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:48:50.832 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:48:50.833 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:48:50.833 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:48:50.833 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:48:50.833 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:48:50.833 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:48:50.833 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:48:50.833 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:48:50.833 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:48:50.833 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:48:50.855 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:48:50.856 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:48:50.856 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:48:50.856 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:48:50.965 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-06T09:48:50.965 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:rbd mirror flush : flush rbd mirror 2024-04-06T09:48:50.965 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:admin_socket: invalid command 2024-04-06T09:48:50.969 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:48:50.969 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/test 2024-04-06T09:48:50.970 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:1 2024-04-06T09:48:50.970 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:48:50.970 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:48:50.970 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-06T09:48:50.970 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:1 2024-04-06T09:48:50.970 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:48:50.970 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:48:50.970 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:48:50.970 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=1 2024-04-06T09:48:50.971 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-06T09:48:50.971 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:48:50.971 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:48:50.971 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=1 2024-04-06T09:48:50.971 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=1 2024-04-06T09:48:50.971 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-06T09:48:50.971 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:1 2024-04-06T09:48:50.971 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:48:50.972 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:48:50.972 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-06T09:48:50.972 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:1 2024-04-06T09:48:50.972 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:48:50.972 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:48:50.972 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:48:50.972 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=1 2024-04-06T09:48:50.972 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-06T09:48:50.973 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:48:50.973 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:48:50.973 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=1 2024-04-06T09:48:50.973 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=1 2024-04-06T09:48:50.973 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-06T09:48:50.993 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T09:48:50.994 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T09:48:50.994 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T09:48:50.994 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/test 2024-04-06T09:48:51.125 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:48:51.126 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:48:51.126 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:48:51.127 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:48:51.127 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:48:51.127 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:48:51.127 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:48:51.127 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:48:51.127 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:48:51.127 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:48:51.127 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:48:51.128 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:48:51.128 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:48:51.129 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:48:51.130 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:48:51.130 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:48:51.187 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 38 2024-04-06T09:48:51.188 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 38 2024-04-06T09:48:51.188 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:48:51.188 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=153, tag_tid=30, entry_tid=1], [object_number=152, tag_tid=30, entry_tid=0], [object_number=155, tag_tid=29, entry_tid=9059], [object_number=154, tag_tid=29, entry_tid=9058]]], state=connected] 2024-04-06T09:48:51.188 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=153, tag_tid=29, entry_tid=6189], [object_number=152, tag_tid=29, entry_tid=6188], [object_number=155, tag_tid=29, entry_tid=6187], [object_number=154, tag_tid=29, entry_tid=6186]]], state=connected] 2024-04-06T09:48:51.191 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:48:51.192 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=153, tag_tid=30, entry_tid=1' 2024-04-06T09:48:51.194 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:48:51.194 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:48:51.194 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:48:51.194 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:48:51.194 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:48:51.194 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:48:51.194 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:48:51.194 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:48:51.194 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:48:51.194 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:48:51.194 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:48:51.194 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:48:51.195 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:48:51.195 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:48:51.195 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:48:51.246 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 38 2024-04-06T09:48:51.246 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 38 2024-04-06T09:48:51.246 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:48:51.247 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=153, tag_tid=30, entry_tid=1], [object_number=152, tag_tid=30, entry_tid=0], [object_number=155, tag_tid=29, entry_tid=9059], [object_number=154, tag_tid=29, entry_tid=9058]]], state=connected] 2024-04-06T09:48:51.247 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=153, tag_tid=29, entry_tid=6189], [object_number=152, tag_tid=29, entry_tid=6188], [object_number=155, tag_tid=29, entry_tid=6187], [object_number=154, tag_tid=29, entry_tid=6186]]], state=connected] 2024-04-06T09:48:51.250 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:48:51.252 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=153, tag_tid=29, entry_tid=6189' 2024-04-06T09:48:51.252 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=153, tag_tid=30, entry_tid=1' -a 'object_number=153, tag_tid=30, entry_tid=1' = 'object_number=153, tag_tid=29, entry_tid=6189' 2024-04-06T09:48:51.252 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=153, tag_tid=29, entry_tid=6189' '!=' 'object_number=153, tag_tid=29, entry_tid=6081' 2024-04-06T09:48:51.252 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:48:51.252 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=153, tag_tid=29, entry_tid=6189' = 'object_number=153, tag_tid=29, entry_tid=6081' 2024-04-06T09:48:51.252 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=153, tag_tid=29, entry_tid=6189' 2024-04-06T09:48:51.253 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=153, tag_tid=30, entry_tid=1' 2024-04-06T09:48:51.253 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:48:51.253 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:48:51.253 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=30 2024-04-06T09:48:51.254 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=153, tag_tid=29, entry_tid=6189' 2024-04-06T09:48:51.254 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:48:51.254 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:48:51.255 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=29 2024-04-06T09:48:51.256 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=153, tag_tid=30, entry_tid=1' 2024-04-06T09:48:51.256 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:48:51.256 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:48:51.257 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:48:51.257 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=153, tag_tid=29, entry_tid=6189' 2024-04-06T09:48:51.258 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:48:51.258 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:48:51.259 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=6189 2024-04-06T09:48:51.259 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 30 = 29 -a 1 -le 6189 2024-04-06T09:48:51.259 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:48:51.259 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:48:51.259 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:48:51.462 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:48:51.462 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:48:51.462 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:48:51.463 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:48:51.463 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:48:51.463 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:48:51.463 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:48:51.463 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:48:51.463 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:48:51.463 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:48:51.463 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:48:51.463 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:48:51.464 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:48:51.464 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:48:51.464 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:48:51.464 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:48:51.468 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:48:51.468 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:48:51.468 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:48:51.468 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:48:51.468 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:48:51.468 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:48:51.468 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:48:51.468 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:48:51.468 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:48:51.468 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:48:51.468 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:48:51.468 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:48:51.468 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:48:51.469 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:48:51.469 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:48:51.469 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:48:51.469 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:48:51.469 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:48:51.469 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:48:51.469 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:48:51.469 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:48:51.469 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:48:51.469 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:48:51.469 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:48:51.469 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:48:51.469 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:48:51.469 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:48:51.470 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:48:51.470 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:48:51.470 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:48:51.470 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:48:51.470 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:48:51.470 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:48:51.492 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:48:51.492 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:48:51.492 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:48:51.492 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:48:51.598 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-06T09:48:51.598 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:rbd mirror flush : flush rbd mirror 2024-04-06T09:48:51.599 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:admin_socket: invalid command 2024-04-06T09:48:51.600 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:48:51.601 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/test 2024-04-06T09:48:51.601 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:1 2024-04-06T09:48:51.601 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:48:51.601 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:48:51.601 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-06T09:48:51.601 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:1 2024-04-06T09:48:51.601 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:48:51.601 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:48:51.601 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:48:51.601 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=1 2024-04-06T09:48:51.601 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-06T09:48:51.601 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:48:51.602 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:48:51.602 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=1 2024-04-06T09:48:51.602 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=1 2024-04-06T09:48:51.603 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-06T09:48:51.603 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:1 2024-04-06T09:48:51.603 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:48:51.603 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:48:51.603 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-06T09:48:51.603 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:1 2024-04-06T09:48:51.603 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:48:51.603 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:48:51.603 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:48:51.604 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=1 2024-04-06T09:48:51.604 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-06T09:48:51.604 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:48:51.604 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:48:51.604 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=1 2024-04-06T09:48:51.604 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=1 2024-04-06T09:48:51.604 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-06T09:48:51.628 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T09:48:51.628 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T09:48:51.628 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T09:48:51.628 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/test 2024-04-06T09:48:51.746 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:48:51.748 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:48:51.748 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:48:51.748 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:48:51.748 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:48:51.748 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:48:51.748 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:48:51.748 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:48:51.748 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:48:51.748 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:48:51.748 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:48:51.749 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:48:51.749 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:48:51.750 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:48:51.750 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:48:51.750 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:48:51.800 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 38 2024-04-06T09:48:51.800 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 38 2024-04-06T09:48:51.801 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:48:51.801 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=153, tag_tid=30, entry_tid=1], [object_number=152, tag_tid=30, entry_tid=0], [object_number=155, tag_tid=29, entry_tid=9059], [object_number=154, tag_tid=29, entry_tid=9058]]], state=connected] 2024-04-06T09:48:51.801 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=152, tag_tid=29, entry_tid=6304], [object_number=155, tag_tid=29, entry_tid=6303], [object_number=154, tag_tid=29, entry_tid=6302], [object_number=153, tag_tid=29, entry_tid=6301]]], state=connected] 2024-04-06T09:48:51.804 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:48:51.805 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=153, tag_tid=30, entry_tid=1' 2024-04-06T09:48:51.806 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:48:51.806 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:48:51.807 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:48:51.807 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:48:51.807 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:48:51.807 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:48:51.807 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:48:51.807 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:48:51.807 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:48:51.807 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:48:51.807 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:48:51.807 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:48:51.808 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:48:51.808 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:48:51.808 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:48:51.858 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 38 2024-04-06T09:48:51.858 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 38 2024-04-06T09:48:51.858 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:48:51.858 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=153, tag_tid=30, entry_tid=1], [object_number=152, tag_tid=30, entry_tid=0], [object_number=155, tag_tid=29, entry_tid=9059], [object_number=154, tag_tid=29, entry_tid=9058]]], state=connected] 2024-04-06T09:48:51.858 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=152, tag_tid=29, entry_tid=6304], [object_number=155, tag_tid=29, entry_tid=6303], [object_number=154, tag_tid=29, entry_tid=6302], [object_number=153, tag_tid=29, entry_tid=6301]]], state=connected] 2024-04-06T09:48:51.861 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:48:51.863 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=152, tag_tid=29, entry_tid=6304' 2024-04-06T09:48:51.863 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=153, tag_tid=30, entry_tid=1' -a 'object_number=153, tag_tid=30, entry_tid=1' = 'object_number=152, tag_tid=29, entry_tid=6304' 2024-04-06T09:48:51.863 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=152, tag_tid=29, entry_tid=6304' '!=' 'object_number=153, tag_tid=29, entry_tid=6189' 2024-04-06T09:48:51.863 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:48:51.863 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=152, tag_tid=29, entry_tid=6304' = 'object_number=153, tag_tid=29, entry_tid=6189' 2024-04-06T09:48:51.863 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=152, tag_tid=29, entry_tid=6304' 2024-04-06T09:48:51.864 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=153, tag_tid=30, entry_tid=1' 2024-04-06T09:48:51.864 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:48:51.864 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:48:51.864 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=30 2024-04-06T09:48:51.865 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=152, tag_tid=29, entry_tid=6304' 2024-04-06T09:48:51.865 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:48:51.865 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:48:51.866 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=29 2024-04-06T09:48:51.867 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=153, tag_tid=30, entry_tid=1' 2024-04-06T09:48:51.867 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:48:51.867 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:48:51.868 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:48:51.868 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=152, tag_tid=29, entry_tid=6304' 2024-04-06T09:48:51.869 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:48:51.869 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:48:51.870 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=6304 2024-04-06T09:48:51.870 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 30 = 29 -a 1 -le 6304 2024-04-06T09:48:51.870 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:48:51.870 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:48:51.870 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:48:52.071 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:48:52.071 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:48:52.071 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:48:52.071 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:48:52.071 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:48:52.072 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:48:52.072 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:48:52.072 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:48:52.072 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:48:52.072 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:48:52.072 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:48:52.072 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:48:52.072 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:48:52.072 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:48:52.072 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:48:52.072 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:48:52.073 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:48:52.075 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:48:52.075 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:48:52.075 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:48:52.075 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:48:52.076 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:48:52.076 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:48:52.076 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:48:52.076 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:48:52.076 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:48:52.076 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:48:52.076 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:48:52.076 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:48:52.076 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:48:52.076 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:48:52.076 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:48:52.076 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:48:52.076 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:48:52.076 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:48:52.076 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:48:52.077 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:48:52.077 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:48:52.077 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:48:52.077 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:48:52.077 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:48:52.077 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:48:52.077 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:48:52.077 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:48:52.077 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:48:52.077 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:48:52.077 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:48:52.077 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:48:52.078 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:48:52.102 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:48:52.102 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:48:52.102 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:48:52.102 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:48:52.212 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-06T09:48:52.212 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:rbd mirror flush : flush rbd mirror 2024-04-06T09:48:52.212 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:admin_socket: invalid command 2024-04-06T09:48:52.215 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:48:52.215 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/test 2024-04-06T09:48:52.215 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:1 2024-04-06T09:48:52.215 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:48:52.215 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:48:52.215 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-06T09:48:52.215 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:1 2024-04-06T09:48:52.215 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:48:52.216 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:48:52.216 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:48:52.216 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=1 2024-04-06T09:48:52.216 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-06T09:48:52.216 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:48:52.216 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:48:52.216 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=1 2024-04-06T09:48:52.216 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=1 2024-04-06T09:48:52.216 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-06T09:48:52.216 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:1 2024-04-06T09:48:52.216 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:48:52.216 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:48:52.216 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-06T09:48:52.216 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:1 2024-04-06T09:48:52.217 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:48:52.217 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:48:52.217 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:48:52.217 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=1 2024-04-06T09:48:52.217 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-06T09:48:52.217 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:48:52.217 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:48:52.217 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=1 2024-04-06T09:48:52.217 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=1 2024-04-06T09:48:52.217 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-06T09:48:52.241 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T09:48:52.241 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T09:48:52.241 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T09:48:52.242 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/test 2024-04-06T09:48:52.367 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:48:52.369 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:48:52.369 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:48:52.369 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:48:52.369 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:48:52.369 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:48:52.369 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:48:52.369 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:48:52.369 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:48:52.369 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:48:52.370 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:48:52.370 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:48:52.370 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:48:52.371 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:48:52.372 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:48:52.372 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:48:52.419 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 38 2024-04-06T09:48:52.419 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 38 2024-04-06T09:48:52.419 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:48:52.419 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=153, tag_tid=30, entry_tid=1], [object_number=152, tag_tid=30, entry_tid=0], [object_number=155, tag_tid=29, entry_tid=9059], [object_number=154, tag_tid=29, entry_tid=9058]]], state=connected] 2024-04-06T09:48:52.419 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=154, tag_tid=29, entry_tid=6398], [object_number=153, tag_tid=29, entry_tid=6397], [object_number=152, tag_tid=29, entry_tid=6396], [object_number=155, tag_tid=29, entry_tid=6395]]], state=connected] 2024-04-06T09:48:52.422 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:48:52.424 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=153, tag_tid=30, entry_tid=1' 2024-04-06T09:48:52.425 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:48:52.425 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:48:52.425 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:48:52.425 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:48:52.425 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:48:52.425 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:48:52.425 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:48:52.425 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:48:52.425 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:48:52.425 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:48:52.425 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:48:52.426 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:48:52.426 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:48:52.426 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:48:52.426 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:48:52.474 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 38 2024-04-06T09:48:52.474 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 38 2024-04-06T09:48:52.474 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:48:52.474 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=153, tag_tid=30, entry_tid=1], [object_number=152, tag_tid=30, entry_tid=0], [object_number=155, tag_tid=29, entry_tid=9059], [object_number=154, tag_tid=29, entry_tid=9058]]], state=connected] 2024-04-06T09:48:52.474 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=154, tag_tid=29, entry_tid=6398], [object_number=153, tag_tid=29, entry_tid=6397], [object_number=152, tag_tid=29, entry_tid=6396], [object_number=155, tag_tid=29, entry_tid=6395]]], state=connected] 2024-04-06T09:48:52.477 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:48:52.479 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=154, tag_tid=29, entry_tid=6398' 2024-04-06T09:48:52.479 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=153, tag_tid=30, entry_tid=1' -a 'object_number=153, tag_tid=30, entry_tid=1' = 'object_number=154, tag_tid=29, entry_tid=6398' 2024-04-06T09:48:52.479 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=154, tag_tid=29, entry_tid=6398' '!=' 'object_number=152, tag_tid=29, entry_tid=6304' 2024-04-06T09:48:52.479 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:48:52.479 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=154, tag_tid=29, entry_tid=6398' = 'object_number=152, tag_tid=29, entry_tid=6304' 2024-04-06T09:48:52.479 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=154, tag_tid=29, entry_tid=6398' 2024-04-06T09:48:52.480 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=153, tag_tid=30, entry_tid=1' 2024-04-06T09:48:52.480 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:48:52.480 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:48:52.480 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=30 2024-04-06T09:48:52.481 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=154, tag_tid=29, entry_tid=6398' 2024-04-06T09:48:52.481 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:48:52.481 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:48:52.482 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=29 2024-04-06T09:48:52.483 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=153, tag_tid=30, entry_tid=1' 2024-04-06T09:48:52.483 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:48:52.483 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:48:52.484 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:48:52.484 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=154, tag_tid=29, entry_tid=6398' 2024-04-06T09:48:52.485 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:48:52.485 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:48:52.485 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=6398 2024-04-06T09:48:52.486 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 30 = 29 -a 1 -le 6398 2024-04-06T09:48:52.486 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:48:52.486 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:48:52.486 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:48:52.688 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:48:52.688 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:48:52.688 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:48:52.688 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:48:52.688 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:48:52.688 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:48:52.688 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:48:52.688 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:48:52.688 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:48:52.688 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:48:52.688 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:48:52.688 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:48:52.689 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:48:52.689 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:48:52.689 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:48:52.689 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:48:52.689 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:48:52.691 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:48:52.691 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:48:52.692 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:48:52.692 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:48:52.692 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:48:52.692 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:48:52.692 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:48:52.692 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:48:52.692 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:48:52.692 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:48:52.692 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:48:52.692 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:48:52.692 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:48:52.692 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:48:52.692 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:48:52.692 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:48:52.693 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:48:52.693 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:48:52.693 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:48:52.693 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:48:52.693 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:48:52.693 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:48:52.693 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:48:52.693 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:48:52.693 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:48:52.693 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:48:52.693 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:48:52.693 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:48:52.693 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:48:52.693 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:48:52.693 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:48:52.694 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:48:52.719 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:48:52.719 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:48:52.719 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:48:52.719 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:48:52.824 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-06T09:48:52.825 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:rbd mirror flush : flush rbd mirror 2024-04-06T09:48:52.825 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:admin_socket: invalid command 2024-04-06T09:48:52.827 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:48:52.827 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/test 2024-04-06T09:48:52.827 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:1 2024-04-06T09:48:52.827 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:48:52.827 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:48:52.827 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-06T09:48:52.827 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:1 2024-04-06T09:48:52.827 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:48:52.827 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:48:52.828 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:48:52.828 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=1 2024-04-06T09:48:52.828 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-06T09:48:52.828 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:48:52.828 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:48:52.828 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=1 2024-04-06T09:48:52.828 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=1 2024-04-06T09:48:52.828 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-06T09:48:52.828 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:1 2024-04-06T09:48:52.828 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:48:52.828 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:48:52.828 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-06T09:48:52.828 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:1 2024-04-06T09:48:52.828 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:48:52.828 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:48:52.829 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:48:52.829 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=1 2024-04-06T09:48:52.829 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-06T09:48:52.829 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:48:52.829 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:48:52.829 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=1 2024-04-06T09:48:52.829 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=1 2024-04-06T09:48:52.829 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-06T09:48:52.852 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T09:48:52.852 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T09:48:52.852 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T09:48:52.852 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/test 2024-04-06T09:48:52.978 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:48:52.980 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:48:52.980 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:48:52.980 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:48:52.980 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:48:52.980 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:48:52.980 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:48:52.980 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:48:52.980 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:48:52.980 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:48:52.980 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:48:52.981 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:48:52.981 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:48:52.983 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:48:52.983 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:48:52.984 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:48:53.038 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 38 2024-04-06T09:48:53.038 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 38 2024-04-06T09:48:53.038 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:48:53.038 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=153, tag_tid=30, entry_tid=1], [object_number=152, tag_tid=30, entry_tid=0], [object_number=155, tag_tid=29, entry_tid=9059], [object_number=154, tag_tid=29, entry_tid=9058]]], state=connected] 2024-04-06T09:48:53.038 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=152, tag_tid=29, entry_tid=6496], [object_number=155, tag_tid=29, entry_tid=6495], [object_number=154, tag_tid=29, entry_tid=6494], [object_number=153, tag_tid=29, entry_tid=6493]]], state=connected] 2024-04-06T09:48:53.041 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:48:53.043 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=153, tag_tid=30, entry_tid=1' 2024-04-06T09:48:53.044 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:48:53.044 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:48:53.044 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:48:53.044 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:48:53.044 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:48:53.044 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:48:53.044 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:48:53.044 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:48:53.045 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:48:53.045 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:48:53.045 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:48:53.045 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:48:53.045 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:48:53.046 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:48:53.046 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:48:53.093 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 38 2024-04-06T09:48:53.093 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 38 2024-04-06T09:48:53.093 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:48:53.094 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=153, tag_tid=30, entry_tid=1], [object_number=152, tag_tid=30, entry_tid=0], [object_number=155, tag_tid=29, entry_tid=9059], [object_number=154, tag_tid=29, entry_tid=9058]]], state=connected] 2024-04-06T09:48:53.094 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=152, tag_tid=29, entry_tid=6496], [object_number=155, tag_tid=29, entry_tid=6495], [object_number=154, tag_tid=29, entry_tid=6494], [object_number=153, tag_tid=29, entry_tid=6493]]], state=connected] 2024-04-06T09:48:53.097 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:48:53.099 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=152, tag_tid=29, entry_tid=6496' 2024-04-06T09:48:53.099 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=153, tag_tid=30, entry_tid=1' -a 'object_number=153, tag_tid=30, entry_tid=1' = 'object_number=152, tag_tid=29, entry_tid=6496' 2024-04-06T09:48:53.099 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=152, tag_tid=29, entry_tid=6496' '!=' 'object_number=154, tag_tid=29, entry_tid=6398' 2024-04-06T09:48:53.099 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:48:53.099 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=152, tag_tid=29, entry_tid=6496' = 'object_number=154, tag_tid=29, entry_tid=6398' 2024-04-06T09:48:53.100 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=152, tag_tid=29, entry_tid=6496' 2024-04-06T09:48:53.100 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=153, tag_tid=30, entry_tid=1' 2024-04-06T09:48:53.100 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:48:53.100 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:48:53.101 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=30 2024-04-06T09:48:53.102 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=152, tag_tid=29, entry_tid=6496' 2024-04-06T09:48:53.102 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:48:53.102 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:48:53.102 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=29 2024-04-06T09:48:53.103 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=153, tag_tid=30, entry_tid=1' 2024-04-06T09:48:53.103 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:48:53.103 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:48:53.104 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:48:53.104 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=152, tag_tid=29, entry_tid=6496' 2024-04-06T09:48:53.104 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:48:53.105 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:48:53.106 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=6496 2024-04-06T09:48:53.106 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 30 = 29 -a 1 -le 6496 2024-04-06T09:48:53.106 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:48:53.106 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:48:53.106 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:48:53.308 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:48:53.308 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:48:53.308 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:48:53.308 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:48:53.308 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:48:53.308 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:48:53.309 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:48:53.309 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:48:53.309 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:48:53.309 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:48:53.309 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:48:53.309 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:48:53.309 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:48:53.309 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:48:53.309 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:48:53.310 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:48:53.310 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:48:53.312 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:48:53.312 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:48:53.312 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:48:53.312 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:48:53.313 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:48:53.313 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:48:53.313 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:48:53.313 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:48:53.313 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:48:53.313 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:48:53.313 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:48:53.313 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:48:53.313 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:48:53.313 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:48:53.313 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:48:53.313 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:48:53.313 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:48:53.314 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:48:53.314 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:48:53.314 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:48:53.314 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:48:53.314 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:48:53.314 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:48:53.314 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:48:53.314 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:48:53.314 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:48:53.314 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:48:53.314 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:48:53.314 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:48:53.314 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:48:53.314 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:48:53.315 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:48:53.339 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:48:53.339 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:48:53.339 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:48:53.339 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:48:53.444 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-06T09:48:53.444 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:rbd mirror flush : flush rbd mirror 2024-04-06T09:48:53.444 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:admin_socket: invalid command 2024-04-06T09:48:53.446 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:48:53.446 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/test 2024-04-06T09:48:53.446 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:1 2024-04-06T09:48:53.446 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:48:53.446 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:48:53.446 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-06T09:48:53.446 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:1 2024-04-06T09:48:53.446 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:48:53.447 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:48:53.447 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:48:53.447 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=1 2024-04-06T09:48:53.447 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-06T09:48:53.447 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:48:53.447 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:48:53.447 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=1 2024-04-06T09:48:53.447 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=1 2024-04-06T09:48:53.448 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-06T09:48:53.449 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:1 2024-04-06T09:48:53.449 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:48:53.449 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:48:53.449 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-06T09:48:53.449 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:1 2024-04-06T09:48:53.449 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:48:53.449 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:48:53.449 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:48:53.449 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=1 2024-04-06T09:48:53.449 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-06T09:48:53.450 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:48:53.450 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:48:53.450 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=1 2024-04-06T09:48:53.450 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=1 2024-04-06T09:48:53.450 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-06T09:48:53.466 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T09:48:53.466 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T09:48:53.467 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T09:48:53.467 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/test 2024-04-06T09:48:53.513 INFO:tasks.rbd_mirror_thrash:kill cluster1.client.mirror.0 2024-04-06T09:48:53.514 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Sent signal 15 2024-04-06T09:48:53.514 INFO:tasks.rbd_mirror_thrash:waiting for 3 secs before reviving daemons 2024-04-06T09:48:53.515 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi192.stderr:2024-04-06T09:48:53.518+0000 7f78a718a640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.0 (PID: 117441) UID: 1000 2024-04-06T09:48:53.598 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:48:53.599 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:48:53.599 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:48:53.600 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:48:53.600 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:48:53.600 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:48:53.600 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:48:53.600 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:48:53.600 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:48:53.600 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:48:53.600 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:48:53.601 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:48:53.601 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:48:53.602 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:48:53.602 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:48:53.602 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:48:53.653 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 38 2024-04-06T09:48:53.653 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 38 2024-04-06T09:48:53.653 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:48:53.653 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=153, tag_tid=30, entry_tid=1], [object_number=152, tag_tid=30, entry_tid=0], [object_number=155, tag_tid=29, entry_tid=9059], [object_number=154, tag_tid=29, entry_tid=9058]]], state=connected] 2024-04-06T09:48:53.654 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=152, tag_tid=29, entry_tid=6600], [object_number=155, tag_tid=29, entry_tid=6599], [object_number=154, tag_tid=29, entry_tid=6598], [object_number=153, tag_tid=29, entry_tid=6597]]], state=connected] 2024-04-06T09:48:53.657 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:48:53.658 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=153, tag_tid=30, entry_tid=1' 2024-04-06T09:48:53.659 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:48:53.659 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:48:53.659 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:48:53.659 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:48:53.659 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:48:53.660 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:48:53.660 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:48:53.660 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:48:53.660 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:48:53.660 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:48:53.660 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:48:53.660 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:48:53.660 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:48:53.661 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:48:53.661 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:48:53.708 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 38 2024-04-06T09:48:53.709 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 38 2024-04-06T09:48:53.709 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:48:53.709 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=153, tag_tid=30, entry_tid=1], [object_number=152, tag_tid=30, entry_tid=0], [object_number=155, tag_tid=29, entry_tid=9059], [object_number=154, tag_tid=29, entry_tid=9058]]], state=connected] 2024-04-06T09:48:53.709 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=152, tag_tid=29, entry_tid=6600], [object_number=155, tag_tid=29, entry_tid=6599], [object_number=154, tag_tid=29, entry_tid=6598], [object_number=153, tag_tid=29, entry_tid=6597]]], state=connected] 2024-04-06T09:48:53.712 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:48:53.714 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=152, tag_tid=29, entry_tid=6600' 2024-04-06T09:48:53.714 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=153, tag_tid=30, entry_tid=1' -a 'object_number=153, tag_tid=30, entry_tid=1' = 'object_number=152, tag_tid=29, entry_tid=6600' 2024-04-06T09:48:53.714 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=152, tag_tid=29, entry_tid=6600' '!=' 'object_number=152, tag_tid=29, entry_tid=6496' 2024-04-06T09:48:53.714 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:48:53.714 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=152, tag_tid=29, entry_tid=6600' = 'object_number=152, tag_tid=29, entry_tid=6496' 2024-04-06T09:48:53.714 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=152, tag_tid=29, entry_tid=6600' 2024-04-06T09:48:53.715 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=153, tag_tid=30, entry_tid=1' 2024-04-06T09:48:53.715 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:48:53.715 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:48:53.715 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=30 2024-04-06T09:48:53.716 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=152, tag_tid=29, entry_tid=6600' 2024-04-06T09:48:53.716 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:48:53.716 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:48:53.717 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=29 2024-04-06T09:48:53.718 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=153, tag_tid=30, entry_tid=1' 2024-04-06T09:48:53.718 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:48:53.718 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:48:53.719 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:48:53.719 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=152, tag_tid=29, entry_tid=6600' 2024-04-06T09:48:53.720 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:48:53.720 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:48:53.721 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=6600 2024-04-06T09:48:53.721 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 30 = 29 -a 1 -le 6600 2024-04-06T09:48:53.721 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:48:53.721 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:48:53.721 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:48:53.923 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:48:53.923 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:48:53.923 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:48:53.923 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:48:53.923 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:48:53.923 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:48:53.923 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:48:53.923 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:48:53.923 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:48:53.924 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:48:53.924 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:48:53.924 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:48:53.924 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:48:53.924 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:48:53.924 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:48:53.924 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:48:53.925 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:48:53.926 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:48:53.926 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:48:53.926 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:48:53.926 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:48:53.926 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:48:53.927 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:48:53.927 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:48:53.927 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:48:53.927 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:48:53.927 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:48:53.927 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:48:53.927 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:48:53.927 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:48:53.927 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:48:53.927 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:48:53.927 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:48:53.928 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:48:53.928 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:48:53.928 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:48:53.929 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:48:53.929 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:48:53.929 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:48:53.929 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:48:53.929 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:48:53.929 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:48:53.929 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:48:53.929 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:48:53.929 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:48:53.929 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:48:53.929 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:48:53.929 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:48:53.929 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:48:53.953 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:48:53.953 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:48:53.953 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:48:53.953 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:48:54.052 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:admin_socket: exception getting command descriptions: [Errno 2] No such file or directory 2024-04-06T09:48:54.054 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:48:54.054 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/test 2024-04-06T09:48:54.054 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:1 2024-04-06T09:48:54.054 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:48:54.054 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:48:54.054 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-06T09:48:54.055 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:1 2024-04-06T09:48:54.055 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:48:54.055 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:48:54.055 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:48:54.055 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=1 2024-04-06T09:48:54.055 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-06T09:48:54.055 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:48:54.055 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:48:54.055 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=1 2024-04-06T09:48:54.055 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=1 2024-04-06T09:48:54.056 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-06T09:48:54.056 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:1 2024-04-06T09:48:54.057 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:48:54.057 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:48:54.057 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-06T09:48:54.057 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:1 2024-04-06T09:48:54.057 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:48:54.057 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:48:54.057 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:48:54.057 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=1 2024-04-06T09:48:54.057 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-06T09:48:54.057 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:48:54.057 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:48:54.057 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=1 2024-04-06T09:48:54.057 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=1 2024-04-06T09:48:54.057 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-06T09:48:54.080 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T09:48:54.080 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T09:48:54.080 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T09:48:54.081 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/test 2024-04-06T09:48:54.219 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:48:54.220 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:48:54.221 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:48:54.221 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:48:54.221 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:48:54.221 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:48:54.221 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:48:54.221 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:48:54.221 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:48:54.221 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:48:54.221 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:48:54.222 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:48:54.222 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:48:54.223 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:48:54.223 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:48:54.224 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:48:54.272 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 38 2024-04-06T09:48:54.272 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 38 2024-04-06T09:48:54.272 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:48:54.272 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=153, tag_tid=30, entry_tid=1], [object_number=152, tag_tid=30, entry_tid=0], [object_number=155, tag_tid=29, entry_tid=9059], [object_number=154, tag_tid=29, entry_tid=9058]]], state=connected] 2024-04-06T09:48:54.272 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=154, tag_tid=29, entry_tid=6726], [object_number=153, tag_tid=29, entry_tid=6725], [object_number=152, tag_tid=29, entry_tid=6724], [object_number=155, tag_tid=29, entry_tid=6723]]], state=connected] 2024-04-06T09:48:54.275 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:48:54.277 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=153, tag_tid=30, entry_tid=1' 2024-04-06T09:48:54.278 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:48:54.278 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:48:54.278 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:48:54.278 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:48:54.278 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:48:54.278 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:48:54.278 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:48:54.278 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:48:54.278 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:48:54.278 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:48:54.279 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:48:54.279 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:48:54.279 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:48:54.279 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:48:54.279 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:48:54.327 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 38 2024-04-06T09:48:54.328 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 38 2024-04-06T09:48:54.328 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:48:54.328 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=153, tag_tid=30, entry_tid=1], [object_number=152, tag_tid=30, entry_tid=0], [object_number=155, tag_tid=29, entry_tid=9059], [object_number=154, tag_tid=29, entry_tid=9058]]], state=connected] 2024-04-06T09:48:54.328 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=154, tag_tid=29, entry_tid=6726], [object_number=153, tag_tid=29, entry_tid=6725], [object_number=152, tag_tid=29, entry_tid=6724], [object_number=155, tag_tid=29, entry_tid=6723]]], state=connected] 2024-04-06T09:48:54.331 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:48:54.333 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=154, tag_tid=29, entry_tid=6726' 2024-04-06T09:48:54.333 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=153, tag_tid=30, entry_tid=1' -a 'object_number=153, tag_tid=30, entry_tid=1' = 'object_number=154, tag_tid=29, entry_tid=6726' 2024-04-06T09:48:54.333 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=154, tag_tid=29, entry_tid=6726' '!=' 'object_number=152, tag_tid=29, entry_tid=6600' 2024-04-06T09:48:54.333 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:48:54.333 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=154, tag_tid=29, entry_tid=6726' = 'object_number=152, tag_tid=29, entry_tid=6600' 2024-04-06T09:48:54.333 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=154, tag_tid=29, entry_tid=6726' 2024-04-06T09:48:54.334 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=153, tag_tid=30, entry_tid=1' 2024-04-06T09:48:54.334 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:48:54.334 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:48:54.335 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=30 2024-04-06T09:48:54.335 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=154, tag_tid=29, entry_tid=6726' 2024-04-06T09:48:54.335 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:48:54.335 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:48:54.336 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=29 2024-04-06T09:48:54.337 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=153, tag_tid=30, entry_tid=1' 2024-04-06T09:48:54.337 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:48:54.337 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:48:54.338 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:48:54.339 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=154, tag_tid=29, entry_tid=6726' 2024-04-06T09:48:54.339 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:48:54.339 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:48:54.340 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=6726 2024-04-06T09:48:54.340 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 30 = 29 -a 1 -le 6726 2024-04-06T09:48:54.340 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:48:54.340 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:48:54.340 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:48:54.542 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:48:54.542 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:48:54.542 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:48:54.542 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:48:54.542 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:48:54.542 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:48:54.543 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:48:54.543 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:48:54.543 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:48:54.543 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:48:54.543 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:48:54.543 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:48:54.543 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:48:54.543 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:48:54.543 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:48:54.544 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:48:54.544 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:48:54.547 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:48:54.547 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:48:54.547 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:48:54.547 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:48:54.547 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:48:54.548 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:48:54.548 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:48:54.548 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:48:54.548 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:48:54.548 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:48:54.548 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:48:54.548 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:48:54.548 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:48:54.548 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:48:54.548 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:48:54.548 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:48:54.548 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:48:54.548 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:48:54.548 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:48:54.549 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:48:54.549 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:48:54.549 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:48:54.549 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:48:54.549 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:48:54.549 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:48:54.549 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:48:54.549 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:48:54.549 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:48:54.550 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:48:54.550 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:48:54.550 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:48:54.550 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:48:54.573 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:48:54.573 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:48:54.573 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:48:54.573 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:48:54.672 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:admin_socket: exception getting command descriptions: [Errno 2] No such file or directory 2024-04-06T09:48:54.676 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:48:54.676 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/test 2024-04-06T09:48:54.676 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:1 2024-04-06T09:48:54.676 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:48:54.676 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:48:54.676 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-06T09:48:54.676 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:1 2024-04-06T09:48:54.676 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:48:54.676 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:48:54.676 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:48:54.676 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=1 2024-04-06T09:48:54.677 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-06T09:48:54.677 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:48:54.677 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:48:54.677 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=1 2024-04-06T09:48:54.677 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=1 2024-04-06T09:48:54.677 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-06T09:48:54.677 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:1 2024-04-06T09:48:54.677 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:48:54.677 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:48:54.677 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-06T09:48:54.677 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:1 2024-04-06T09:48:54.677 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:48:54.677 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:48:54.677 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:48:54.677 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=1 2024-04-06T09:48:54.678 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-06T09:48:54.678 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:48:54.678 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:48:54.678 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=1 2024-04-06T09:48:54.678 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=1 2024-04-06T09:48:54.678 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-06T09:48:54.695 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T09:48:54.695 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T09:48:54.695 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T09:48:54.695 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/test 2024-04-06T09:48:54.818 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:48:54.819 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:48:54.819 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:48:54.819 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:48:54.819 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:48:54.819 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:48:54.819 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:48:54.819 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:48:54.819 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:48:54.820 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:48:54.820 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:48:54.820 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:48:54.820 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:48:54.822 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:48:54.822 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:48:54.822 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:48:54.870 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 38 2024-04-06T09:48:54.870 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 38 2024-04-06T09:48:54.870 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:48:54.870 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=153, tag_tid=30, entry_tid=1], [object_number=152, tag_tid=30, entry_tid=0], [object_number=155, tag_tid=29, entry_tid=9059], [object_number=154, tag_tid=29, entry_tid=9058]]], state=connected] 2024-04-06T09:48:54.870 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=154, tag_tid=29, entry_tid=6834], [object_number=153, tag_tid=29, entry_tid=6833], [object_number=152, tag_tid=29, entry_tid=6832], [object_number=155, tag_tid=29, entry_tid=6831]]], state=connected] 2024-04-06T09:48:54.873 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:48:54.874 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=153, tag_tid=30, entry_tid=1' 2024-04-06T09:48:54.875 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:48:54.876 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:48:54.876 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:48:54.876 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:48:54.876 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:48:54.876 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:48:54.876 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:48:54.876 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:48:54.876 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:48:54.876 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:48:54.876 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:48:54.876 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:48:54.877 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:48:54.877 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:48:54.877 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:48:54.925 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 38 2024-04-06T09:48:54.926 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 38 2024-04-06T09:48:54.926 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:48:54.926 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=153, tag_tid=30, entry_tid=1], [object_number=152, tag_tid=30, entry_tid=0], [object_number=155, tag_tid=29, entry_tid=9059], [object_number=154, tag_tid=29, entry_tid=9058]]], state=connected] 2024-04-06T09:48:54.926 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=154, tag_tid=29, entry_tid=6834], [object_number=153, tag_tid=29, entry_tid=6833], [object_number=152, tag_tid=29, entry_tid=6832], [object_number=155, tag_tid=29, entry_tid=6831]]], state=connected] 2024-04-06T09:48:54.929 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:48:54.931 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=154, tag_tid=29, entry_tid=6834' 2024-04-06T09:48:54.931 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=153, tag_tid=30, entry_tid=1' -a 'object_number=153, tag_tid=30, entry_tid=1' = 'object_number=154, tag_tid=29, entry_tid=6834' 2024-04-06T09:48:54.931 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=154, tag_tid=29, entry_tid=6834' '!=' 'object_number=154, tag_tid=29, entry_tid=6726' 2024-04-06T09:48:54.931 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:48:54.931 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=154, tag_tid=29, entry_tid=6834' = 'object_number=154, tag_tid=29, entry_tid=6726' 2024-04-06T09:48:54.932 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=154, tag_tid=29, entry_tid=6834' 2024-04-06T09:48:54.932 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=153, tag_tid=30, entry_tid=1' 2024-04-06T09:48:54.932 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:48:54.932 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:48:54.932 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=30 2024-04-06T09:48:54.933 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=154, tag_tid=29, entry_tid=6834' 2024-04-06T09:48:54.933 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:48:54.933 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:48:54.934 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=29 2024-04-06T09:48:54.934 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=153, tag_tid=30, entry_tid=1' 2024-04-06T09:48:54.935 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:48:54.935 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:48:54.936 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:48:54.936 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=154, tag_tid=29, entry_tid=6834' 2024-04-06T09:48:54.936 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:48:54.936 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:48:54.938 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=6834 2024-04-06T09:48:54.938 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 30 = 29 -a 1 -le 6834 2024-04-06T09:48:54.938 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:48:54.938 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:48:54.938 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:48:55.139 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:48:55.139 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:48:55.140 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:48:55.140 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:48:55.140 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:48:55.140 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:48:55.140 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:48:55.140 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:48:55.140 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:48:55.140 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:48:55.140 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:48:55.140 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:48:55.140 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:48:55.140 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:48:55.140 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:48:55.141 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:48:55.141 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:48:55.144 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:48:55.144 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:48:55.144 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:48:55.144 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:48:55.144 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:48:55.144 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:48:55.144 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:48:55.144 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:48:55.144 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:48:55.144 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:48:55.144 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:48:55.144 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:48:55.144 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:48:55.145 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:48:55.145 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:48:55.145 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:48:55.145 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:48:55.145 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:48:55.145 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:48:55.145 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:48:55.145 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:48:55.145 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:48:55.145 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:48:55.145 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:48:55.145 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:48:55.145 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:48:55.145 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:48:55.146 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:48:55.146 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:48:55.146 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:48:55.146 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:48:55.146 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:48:55.170 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:48:55.171 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:48:55.171 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:48:55.171 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:48:55.270 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:admin_socket: exception getting command descriptions: [Errno 2] No such file or directory 2024-04-06T09:48:55.272 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:48:55.272 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/test 2024-04-06T09:48:55.272 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:1 2024-04-06T09:48:55.272 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:48:55.272 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:48:55.272 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-06T09:48:55.272 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:1 2024-04-06T09:48:55.272 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:48:55.272 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:48:55.273 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:48:55.273 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=1 2024-04-06T09:48:55.273 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-06T09:48:55.273 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:48:55.273 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:48:55.273 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=1 2024-04-06T09:48:55.273 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=1 2024-04-06T09:48:55.274 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-06T09:48:55.275 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:1 2024-04-06T09:48:55.275 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:48:55.275 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:48:55.275 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-06T09:48:55.275 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:1 2024-04-06T09:48:55.275 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:48:55.275 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:48:55.275 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:48:55.275 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=1 2024-04-06T09:48:55.275 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-06T09:48:55.275 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:48:55.275 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:48:55.275 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=1 2024-04-06T09:48:55.276 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=1 2024-04-06T09:48:55.276 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-06T09:48:55.298 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T09:48:55.298 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T09:48:55.298 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T09:48:55.298 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/test 2024-04-06T09:48:55.424 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:48:55.424 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:48:55.424 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:48:55.425 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:48:55.425 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:48:55.425 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:48:55.425 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:48:55.425 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:48:55.425 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:48:55.425 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:48:55.425 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:48:55.426 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:48:55.426 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:48:55.427 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:48:55.427 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:48:55.427 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:48:55.476 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 38 2024-04-06T09:48:55.477 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 38 2024-04-06T09:48:55.477 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:48:55.477 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=153, tag_tid=30, entry_tid=1], [object_number=152, tag_tid=30, entry_tid=0], [object_number=155, tag_tid=29, entry_tid=9059], [object_number=154, tag_tid=29, entry_tid=9058]]], state=connected] 2024-04-06T09:48:55.477 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=152, tag_tid=29, entry_tid=6936], [object_number=155, tag_tid=29, entry_tid=6935], [object_number=154, tag_tid=29, entry_tid=6934], [object_number=153, tag_tid=29, entry_tid=6933]]], state=connected] 2024-04-06T09:48:55.480 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:48:55.481 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=153, tag_tid=30, entry_tid=1' 2024-04-06T09:48:55.483 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:48:55.483 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:48:55.483 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:48:55.483 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:48:55.483 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:48:55.483 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:48:55.483 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:48:55.483 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:48:55.483 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:48:55.483 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:48:55.483 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:48:55.483 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:48:55.484 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:48:55.484 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:48:55.484 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:48:55.535 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 38 2024-04-06T09:48:55.535 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 38 2024-04-06T09:48:55.535 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:48:55.535 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=153, tag_tid=30, entry_tid=1], [object_number=152, tag_tid=30, entry_tid=0], [object_number=155, tag_tid=29, entry_tid=9059], [object_number=154, tag_tid=29, entry_tid=9058]]], state=connected] 2024-04-06T09:48:55.535 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=152, tag_tid=29, entry_tid=6936], [object_number=155, tag_tid=29, entry_tid=6935], [object_number=154, tag_tid=29, entry_tid=6934], [object_number=153, tag_tid=29, entry_tid=6933]]], state=connected] 2024-04-06T09:48:55.538 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:48:55.540 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=152, tag_tid=29, entry_tid=6936' 2024-04-06T09:48:55.540 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=153, tag_tid=30, entry_tid=1' -a 'object_number=153, tag_tid=30, entry_tid=1' = 'object_number=152, tag_tid=29, entry_tid=6936' 2024-04-06T09:48:55.541 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=152, tag_tid=29, entry_tid=6936' '!=' 'object_number=154, tag_tid=29, entry_tid=6834' 2024-04-06T09:48:55.541 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:48:55.541 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=152, tag_tid=29, entry_tid=6936' = 'object_number=154, tag_tid=29, entry_tid=6834' 2024-04-06T09:48:55.541 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=152, tag_tid=29, entry_tid=6936' 2024-04-06T09:48:55.541 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=153, tag_tid=30, entry_tid=1' 2024-04-06T09:48:55.541 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:48:55.541 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:48:55.541 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=30 2024-04-06T09:48:55.542 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=152, tag_tid=29, entry_tid=6936' 2024-04-06T09:48:55.542 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:48:55.542 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:48:55.543 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=29 2024-04-06T09:48:55.544 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=153, tag_tid=30, entry_tid=1' 2024-04-06T09:48:55.544 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:48:55.544 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:48:55.545 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:48:55.546 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=152, tag_tid=29, entry_tid=6936' 2024-04-06T09:48:55.546 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:48:55.546 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:48:55.547 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=6936 2024-04-06T09:48:55.547 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 30 = 29 -a 1 -le 6936 2024-04-06T09:48:55.547 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:48:55.547 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:48:55.547 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:48:55.749 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:48:55.749 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:48:55.749 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:48:55.749 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:48:55.749 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:48:55.749 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:48:55.749 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:48:55.749 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:48:55.749 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:48:55.750 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:48:55.750 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:48:55.750 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:48:55.750 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:48:55.750 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:48:55.750 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:48:55.750 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:48:55.751 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:48:55.754 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:48:55.754 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:48:55.754 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:48:55.754 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:48:55.754 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:48:55.754 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:48:55.754 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:48:55.754 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:48:55.754 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:48:55.754 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:48:55.754 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:48:55.755 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:48:55.755 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:48:55.755 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:48:55.755 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:48:55.755 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:48:55.755 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:48:55.755 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:48:55.755 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:48:55.755 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:48:55.755 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:48:55.755 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:48:55.755 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:48:55.755 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:48:55.755 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:48:55.755 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:48:55.756 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:48:55.756 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:48:55.756 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:48:55.756 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:48:55.756 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:48:55.756 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:48:55.779 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:48:55.780 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:48:55.780 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:48:55.780 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:48:55.884 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:admin_socket: exception getting command descriptions: [Errno 2] No such file or directory 2024-04-06T09:48:55.884 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:48:55.884 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/test 2024-04-06T09:48:55.884 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:1 2024-04-06T09:48:55.884 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:48:55.884 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:48:55.884 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-06T09:48:55.884 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:1 2024-04-06T09:48:55.884 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:48:55.884 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:48:55.884 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:48:55.884 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=1 2024-04-06T09:48:55.885 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-06T09:48:55.885 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:48:55.885 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:48:55.885 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=1 2024-04-06T09:48:55.885 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=1 2024-04-06T09:48:55.885 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-06T09:48:55.885 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:1 2024-04-06T09:48:55.885 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:48:55.885 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:48:55.885 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-06T09:48:55.885 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:1 2024-04-06T09:48:55.885 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:48:55.886 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:48:55.886 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:48:55.886 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=1 2024-04-06T09:48:55.886 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-06T09:48:55.886 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:48:55.886 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:48:55.886 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=1 2024-04-06T09:48:55.886 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=1 2024-04-06T09:48:55.886 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-06T09:48:55.908 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T09:48:55.908 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T09:48:55.908 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T09:48:55.909 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/test 2024-04-06T09:48:56.030 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:48:56.031 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:48:56.031 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:48:56.032 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:48:56.032 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:48:56.032 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:48:56.032 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:48:56.032 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:48:56.032 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:48:56.032 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:48:56.033 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:48:56.033 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:48:56.033 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:48:56.035 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:48:56.035 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:48:56.035 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:48:56.090 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 38 2024-04-06T09:48:56.090 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 38 2024-04-06T09:48:56.090 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:48:56.090 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=153, tag_tid=30, entry_tid=1], [object_number=152, tag_tid=30, entry_tid=0], [object_number=155, tag_tid=29, entry_tid=9059], [object_number=154, tag_tid=29, entry_tid=9058]]], state=connected] 2024-04-06T09:48:56.090 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=152, tag_tid=29, entry_tid=7040], [object_number=155, tag_tid=29, entry_tid=7039], [object_number=154, tag_tid=29, entry_tid=7038], [object_number=153, tag_tid=29, entry_tid=7037]]], state=connected] 2024-04-06T09:48:56.094 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:48:56.095 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=153, tag_tid=30, entry_tid=1' 2024-04-06T09:48:56.096 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:48:56.096 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:48:56.096 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:48:56.096 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:48:56.096 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:48:56.096 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:48:56.097 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:48:56.097 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:48:56.097 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:48:56.097 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:48:56.097 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:48:56.097 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:48:56.097 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:48:56.098 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:48:56.098 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:48:56.145 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 38 2024-04-06T09:48:56.146 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 38 2024-04-06T09:48:56.146 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:48:56.146 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=153, tag_tid=30, entry_tid=1], [object_number=152, tag_tid=30, entry_tid=0], [object_number=155, tag_tid=29, entry_tid=9059], [object_number=154, tag_tid=29, entry_tid=9058]]], state=connected] 2024-04-06T09:48:56.146 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=152, tag_tid=29, entry_tid=7040], [object_number=155, tag_tid=29, entry_tid=7039], [object_number=154, tag_tid=29, entry_tid=7038], [object_number=153, tag_tid=29, entry_tid=7037]]], state=connected] 2024-04-06T09:48:56.149 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:48:56.152 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=152, tag_tid=29, entry_tid=7040' 2024-04-06T09:48:56.152 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=153, tag_tid=30, entry_tid=1' -a 'object_number=153, tag_tid=30, entry_tid=1' = 'object_number=152, tag_tid=29, entry_tid=7040' 2024-04-06T09:48:56.152 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=152, tag_tid=29, entry_tid=7040' '!=' 'object_number=152, tag_tid=29, entry_tid=6936' 2024-04-06T09:48:56.152 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:48:56.152 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=152, tag_tid=29, entry_tid=7040' = 'object_number=152, tag_tid=29, entry_tid=6936' 2024-04-06T09:48:56.152 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=152, tag_tid=29, entry_tid=7040' 2024-04-06T09:48:56.152 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=153, tag_tid=30, entry_tid=1' 2024-04-06T09:48:56.152 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:48:56.152 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:48:56.153 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=30 2024-04-06T09:48:56.153 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=152, tag_tid=29, entry_tid=7040' 2024-04-06T09:48:56.154 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:48:56.154 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:48:56.155 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=29 2024-04-06T09:48:56.155 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=153, tag_tid=30, entry_tid=1' 2024-04-06T09:48:56.155 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:48:56.156 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:48:56.156 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:48:56.157 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=152, tag_tid=29, entry_tid=7040' 2024-04-06T09:48:56.157 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:48:56.157 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:48:56.158 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=7040 2024-04-06T09:48:56.159 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 30 = 29 -a 1 -le 7040 2024-04-06T09:48:56.159 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:48:56.159 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:48:56.159 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:48:56.360 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:48:56.360 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:48:56.360 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:48:56.361 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:48:56.361 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:48:56.361 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:48:56.361 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:48:56.361 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:48:56.361 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:48:56.361 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:48:56.361 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:48:56.361 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:48:56.361 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:48:56.361 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:48:56.361 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:48:56.362 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:48:56.362 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:48:56.364 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:48:56.365 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:48:56.365 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:48:56.365 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:48:56.365 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:48:56.365 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:48:56.365 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:48:56.365 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:48:56.365 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:48:56.365 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:48:56.365 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:48:56.365 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:48:56.365 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:48:56.365 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:48:56.366 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:48:56.366 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:48:56.366 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:48:56.366 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:48:56.366 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:48:56.366 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:48:56.366 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:48:56.366 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:48:56.366 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:48:56.366 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:48:56.366 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:48:56.366 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:48:56.366 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:48:56.366 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:48:56.366 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:48:56.367 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:48:56.367 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:48:56.367 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:48:56.391 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:48:56.392 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:48:56.392 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:48:56.392 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:48:56.490 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:admin_socket: exception getting command descriptions: [Errno 2] No such file or directory 2024-04-06T09:48:56.493 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:48:56.493 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/test 2024-04-06T09:48:56.493 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:1 2024-04-06T09:48:56.493 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:48:56.493 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:48:56.493 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-06T09:48:56.493 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:1 2024-04-06T09:48:56.493 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:48:56.493 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:48:56.494 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:48:56.494 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=1 2024-04-06T09:48:56.494 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-06T09:48:56.494 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:48:56.494 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:48:56.494 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=1 2024-04-06T09:48:56.494 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=1 2024-04-06T09:48:56.494 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-06T09:48:56.494 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:1 2024-04-06T09:48:56.494 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:48:56.494 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:48:56.494 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-06T09:48:56.494 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:1 2024-04-06T09:48:56.494 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:48:56.495 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:48:56.495 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:48:56.495 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=1 2024-04-06T09:48:56.495 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-06T09:48:56.495 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:48:56.495 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:48:56.495 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=1 2024-04-06T09:48:56.495 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=1 2024-04-06T09:48:56.495 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-06T09:48:56.516 INFO:tasks.rbd_mirror_thrash:waiting for cluster1.client.mirror.0 2024-04-06T09:48:56.516 INFO:teuthology.orchestra.run:waiting for 600 2024-04-06T09:48:56.516 INFO:tasks.rbd_mirror_thrash:reviving cluster1.client.mirror.0 2024-04-06T09:48:56.516 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Restarting daemon 2024-04-06T09:48:56.517 DEBUG:teuthology.orchestra.run.smithi192:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.0 2024-04-06T09:48:56.518 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T09:48:56.518 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T09:48:56.518 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T09:48:56.518 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/test 2024-04-06T09:48:56.520 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Started 2024-04-06T09:48:56.520 INFO:tasks.rbd_mirror_thrash:waiting for 88 secs before thrashing 2024-04-06T09:48:56.644 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:48:56.645 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:48:56.645 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:48:56.645 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:48:56.645 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:48:56.645 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:48:56.645 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:48:56.645 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:48:56.645 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:48:56.645 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:48:56.645 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:48:56.645 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:48:56.645 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:48:56.646 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:48:56.646 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:48:56.646 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:48:56.698 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 38 2024-04-06T09:48:56.698 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 38 2024-04-06T09:48:56.699 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:48:56.699 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=153, tag_tid=30, entry_tid=1], [object_number=152, tag_tid=30, entry_tid=0], [object_number=155, tag_tid=29, entry_tid=9059], [object_number=154, tag_tid=29, entry_tid=9058]]], state=connected] 2024-04-06T09:48:56.699 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=154, tag_tid=29, entry_tid=7146], [object_number=153, tag_tid=29, entry_tid=7145], [object_number=152, tag_tid=29, entry_tid=7144], [object_number=155, tag_tid=29, entry_tid=7143]]], state=connected] 2024-04-06T09:48:56.699 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:48:56.700 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=153, tag_tid=30, entry_tid=1' 2024-04-06T09:48:56.701 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:48:56.701 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:48:56.701 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:48:56.701 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:48:56.701 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:48:56.701 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:48:56.702 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:48:56.702 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:48:56.702 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:48:56.702 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:48:56.702 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:48:56.702 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:48:56.702 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:48:56.703 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:48:56.703 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:48:56.755 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 38 2024-04-06T09:48:56.756 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 38 2024-04-06T09:48:56.756 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:48:56.756 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=153, tag_tid=30, entry_tid=1], [object_number=152, tag_tid=30, entry_tid=0], [object_number=155, tag_tid=29, entry_tid=9059], [object_number=154, tag_tid=29, entry_tid=9058]]], state=connected] 2024-04-06T09:48:56.756 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=154, tag_tid=29, entry_tid=7146], [object_number=153, tag_tid=29, entry_tid=7145], [object_number=152, tag_tid=29, entry_tid=7144], [object_number=155, tag_tid=29, entry_tid=7143]]], state=connected] 2024-04-06T09:48:56.759 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:48:56.761 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=154, tag_tid=29, entry_tid=7146' 2024-04-06T09:48:56.761 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=153, tag_tid=30, entry_tid=1' -a 'object_number=153, tag_tid=30, entry_tid=1' = 'object_number=154, tag_tid=29, entry_tid=7146' 2024-04-06T09:48:56.761 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=154, tag_tid=29, entry_tid=7146' '!=' 'object_number=152, tag_tid=29, entry_tid=7040' 2024-04-06T09:48:56.761 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:48:56.761 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=154, tag_tid=29, entry_tid=7146' = 'object_number=152, tag_tid=29, entry_tid=7040' 2024-04-06T09:48:56.761 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=154, tag_tid=29, entry_tid=7146' 2024-04-06T09:48:56.762 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=153, tag_tid=30, entry_tid=1' 2024-04-06T09:48:56.762 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:48:56.762 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:48:56.762 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=30 2024-04-06T09:48:56.763 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=154, tag_tid=29, entry_tid=7146' 2024-04-06T09:48:56.763 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:48:56.763 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:48:56.764 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=29 2024-04-06T09:48:56.765 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=153, tag_tid=30, entry_tid=1' 2024-04-06T09:48:56.765 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:48:56.765 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:48:56.766 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:48:56.767 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=154, tag_tid=29, entry_tid=7146' 2024-04-06T09:48:56.767 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:48:56.767 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:48:56.768 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=7146 2024-04-06T09:48:56.768 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 30 = 29 -a 1 -le 7146 2024-04-06T09:48:56.768 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:48:56.768 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:48:56.769 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:48:56.971 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:48:56.971 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:48:56.971 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:48:56.971 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:48:56.971 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:48:56.971 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:48:56.971 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:48:56.971 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:48:56.971 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:48:56.972 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:48:56.972 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:48:56.972 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:48:56.972 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:48:56.972 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:48:56.972 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:48:56.972 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:48:56.972 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:48:56.975 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:48:56.975 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:48:56.975 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:48:56.975 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:48:56.975 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:48:56.975 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:48:56.975 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:48:56.975 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:48:56.976 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:48:56.976 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:48:56.976 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:48:56.976 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:48:56.976 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:48:56.976 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:48:56.976 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:48:56.976 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:48:56.976 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:48:56.976 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:48:56.976 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:48:56.976 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:48:56.976 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:48:56.977 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:48:56.977 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:48:56.977 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:48:56.977 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:48:56.977 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:48:56.977 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:48:56.977 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:48:56.977 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:48:56.977 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:48:56.977 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:48:56.977 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:48:57.002 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:48:57.002 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:48:57.002 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:48:57.002 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:48:57.106 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-06T09:48:57.106 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:rbd mirror flush : flush rbd mirror 2024-04-06T09:48:57.106 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:admin_socket: invalid command 2024-04-06T09:48:57.108 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:48:57.108 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/test 2024-04-06T09:48:57.108 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:1 2024-04-06T09:48:57.109 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:48:57.109 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:48:57.109 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-06T09:48:57.109 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:1 2024-04-06T09:48:57.109 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:48:57.109 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:48:57.109 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:48:57.109 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=1 2024-04-06T09:48:57.109 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-06T09:48:57.110 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:48:57.110 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:48:57.110 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=1 2024-04-06T09:48:57.110 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=1 2024-04-06T09:48:57.111 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-06T09:48:57.111 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:1 2024-04-06T09:48:57.111 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:48:57.111 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:48:57.111 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-06T09:48:57.111 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:1 2024-04-06T09:48:57.112 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:48:57.112 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:48:57.112 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:48:57.112 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=1 2024-04-06T09:48:57.112 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-06T09:48:57.112 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:48:57.112 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:48:57.112 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=1 2024-04-06T09:48:57.112 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=1 2024-04-06T09:48:57.112 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-06T09:48:57.134 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T09:48:57.134 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T09:48:57.134 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T09:48:57.134 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/test 2024-04-06T09:48:57.260 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:48:57.261 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:48:57.261 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:48:57.262 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:48:57.262 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:48:57.262 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:48:57.262 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:48:57.262 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:48:57.262 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:48:57.262 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:48:57.263 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:48:57.263 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:48:57.263 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:48:57.264 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:48:57.264 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:48:57.264 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:48:57.314 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 38 2024-04-06T09:48:57.315 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 38 2024-04-06T09:48:57.315 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:48:57.315 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=153, tag_tid=30, entry_tid=1], [object_number=152, tag_tid=30, entry_tid=0], [object_number=155, tag_tid=29, entry_tid=9059], [object_number=154, tag_tid=29, entry_tid=9058]]], state=connected] 2024-04-06T09:48:57.315 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=155, tag_tid=29, entry_tid=7255], [object_number=154, tag_tid=29, entry_tid=7254], [object_number=153, tag_tid=29, entry_tid=7253], [object_number=152, tag_tid=29, entry_tid=7252]]], state=connected] 2024-04-06T09:48:57.315 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:48:57.316 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=153, tag_tid=30, entry_tid=1' 2024-04-06T09:48:57.318 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:48:57.318 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:48:57.318 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:48:57.318 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:48:57.318 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:48:57.318 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:48:57.318 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:48:57.318 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:48:57.318 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:48:57.318 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:48:57.318 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:48:57.318 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:48:57.319 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:48:57.319 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:48:57.319 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:48:57.367 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 38 2024-04-06T09:48:57.367 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 38 2024-04-06T09:48:57.367 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:48:57.367 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=153, tag_tid=30, entry_tid=1], [object_number=152, tag_tid=30, entry_tid=0], [object_number=155, tag_tid=29, entry_tid=9059], [object_number=154, tag_tid=29, entry_tid=9058]]], state=connected] 2024-04-06T09:48:57.367 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=155, tag_tid=29, entry_tid=7255], [object_number=154, tag_tid=29, entry_tid=7254], [object_number=153, tag_tid=29, entry_tid=7253], [object_number=152, tag_tid=29, entry_tid=7252]]], state=connected] 2024-04-06T09:48:57.371 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:48:57.373 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=155, tag_tid=29, entry_tid=7255' 2024-04-06T09:48:57.373 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=153, tag_tid=30, entry_tid=1' -a 'object_number=153, tag_tid=30, entry_tid=1' = 'object_number=155, tag_tid=29, entry_tid=7255' 2024-04-06T09:48:57.373 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=155, tag_tid=29, entry_tid=7255' '!=' 'object_number=154, tag_tid=29, entry_tid=7146' 2024-04-06T09:48:57.373 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:48:57.373 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=155, tag_tid=29, entry_tid=7255' = 'object_number=154, tag_tid=29, entry_tid=7146' 2024-04-06T09:48:57.373 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=155, tag_tid=29, entry_tid=7255' 2024-04-06T09:48:57.373 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=153, tag_tid=30, entry_tid=1' 2024-04-06T09:48:57.373 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:48:57.373 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:48:57.374 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=30 2024-04-06T09:48:57.375 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=155, tag_tid=29, entry_tid=7255' 2024-04-06T09:48:57.375 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:48:57.375 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:48:57.375 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=29 2024-04-06T09:48:57.376 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=153, tag_tid=30, entry_tid=1' 2024-04-06T09:48:57.376 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:48:57.376 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:48:57.378 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:48:57.378 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=155, tag_tid=29, entry_tid=7255' 2024-04-06T09:48:57.378 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:48:57.378 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:48:57.379 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=7255 2024-04-06T09:48:57.380 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 30 = 29 -a 1 -le 7255 2024-04-06T09:48:57.380 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:48:57.380 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:48:57.380 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:48:57.582 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:48:57.582 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:48:57.582 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:48:57.582 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:48:57.582 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:48:57.582 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:48:57.582 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:48:57.582 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:48:57.582 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:48:57.582 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:48:57.582 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:48:57.583 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:48:57.583 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:48:57.583 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:48:57.583 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:48:57.583 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:48:57.584 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:48:57.586 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:48:57.587 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:48:57.587 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:48:57.587 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:48:57.587 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:48:57.587 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:48:57.587 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:48:57.587 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:48:57.587 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:48:57.587 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:48:57.587 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:48:57.587 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:48:57.587 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:48:57.587 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:48:57.587 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:48:57.588 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:48:57.588 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:48:57.588 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:48:57.588 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:48:57.589 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:48:57.589 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:48:57.589 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:48:57.589 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:48:57.589 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:48:57.589 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:48:57.589 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:48:57.589 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:48:57.589 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:48:57.589 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:48:57.589 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:48:57.589 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:48:57.589 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:48:57.615 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:48:57.616 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:48:57.616 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:48:57.616 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:48:57.724 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-06T09:48:57.724 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:rbd mirror flush : flush rbd mirror 2024-04-06T09:48:57.724 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:admin_socket: invalid command 2024-04-06T09:48:57.727 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:48:57.727 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/test 2024-04-06T09:48:57.728 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:1 2024-04-06T09:48:57.728 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:48:57.728 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:48:57.728 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-06T09:48:57.728 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:1 2024-04-06T09:48:57.728 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:48:57.728 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:48:57.728 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:48:57.728 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=1 2024-04-06T09:48:57.728 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-06T09:48:57.728 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:48:57.728 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:48:57.728 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=1 2024-04-06T09:48:57.728 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=1 2024-04-06T09:48:57.729 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-06T09:48:57.729 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:1 2024-04-06T09:48:57.729 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:48:57.729 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:48:57.729 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-06T09:48:57.729 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:1 2024-04-06T09:48:57.729 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:48:57.729 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:48:57.729 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:48:57.729 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=1 2024-04-06T09:48:57.729 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-06T09:48:57.729 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:48:57.729 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:48:57.729 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=1 2024-04-06T09:48:57.729 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=1 2024-04-06T09:48:57.730 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-06T09:48:57.753 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T09:48:57.753 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T09:48:57.753 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T09:48:57.753 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/test 2024-04-06T09:48:57.874 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:48:57.875 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:48:57.875 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:48:57.875 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:48:57.875 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:48:57.875 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:48:57.875 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:48:57.876 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:48:57.876 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:48:57.876 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:48:57.876 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:48:57.876 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:48:57.876 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:48:57.877 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:48:57.878 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:48:57.878 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:48:57.936 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 38 2024-04-06T09:48:57.937 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 38 2024-04-06T09:48:57.937 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:48:57.937 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=153, tag_tid=30, entry_tid=1], [object_number=152, tag_tid=30, entry_tid=0], [object_number=155, tag_tid=29, entry_tid=9059], [object_number=154, tag_tid=29, entry_tid=9058]]], state=connected] 2024-04-06T09:48:57.937 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=154, tag_tid=29, entry_tid=7354], [object_number=153, tag_tid=29, entry_tid=7353], [object_number=152, tag_tid=29, entry_tid=7352], [object_number=155, tag_tid=29, entry_tid=7351]]], state=connected] 2024-04-06T09:48:57.941 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:48:57.942 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=153, tag_tid=30, entry_tid=1' 2024-04-06T09:48:57.943 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:48:57.943 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:48:57.943 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:48:57.943 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:48:57.943 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:48:57.944 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:48:57.944 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:48:57.944 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:48:57.944 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:48:57.944 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:48:57.944 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:48:57.944 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:48:57.944 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:48:57.945 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:48:57.945 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:48:57.995 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 38 2024-04-06T09:48:57.995 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 38 2024-04-06T09:48:57.995 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:48:57.995 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=153, tag_tid=30, entry_tid=1], [object_number=152, tag_tid=30, entry_tid=0], [object_number=155, tag_tid=29, entry_tid=9059], [object_number=154, tag_tid=29, entry_tid=9058]]], state=connected] 2024-04-06T09:48:57.995 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=154, tag_tid=29, entry_tid=7354], [object_number=153, tag_tid=29, entry_tid=7353], [object_number=152, tag_tid=29, entry_tid=7352], [object_number=155, tag_tid=29, entry_tid=7351]]], state=connected] 2024-04-06T09:48:57.999 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:48:58.000 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=154, tag_tid=29, entry_tid=7354' 2024-04-06T09:48:58.001 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=153, tag_tid=30, entry_tid=1' -a 'object_number=153, tag_tid=30, entry_tid=1' = 'object_number=154, tag_tid=29, entry_tid=7354' 2024-04-06T09:48:58.001 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=154, tag_tid=29, entry_tid=7354' '!=' 'object_number=155, tag_tid=29, entry_tid=7255' 2024-04-06T09:48:58.001 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:48:58.001 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=154, tag_tid=29, entry_tid=7354' = 'object_number=155, tag_tid=29, entry_tid=7255' 2024-04-06T09:48:58.001 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=154, tag_tid=29, entry_tid=7354' 2024-04-06T09:48:58.001 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=153, tag_tid=30, entry_tid=1' 2024-04-06T09:48:58.002 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:48:58.002 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:48:58.002 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=30 2024-04-06T09:48:58.003 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=154, tag_tid=29, entry_tid=7354' 2024-04-06T09:48:58.003 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:48:58.003 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:48:58.004 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=29 2024-04-06T09:48:58.005 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=153, tag_tid=30, entry_tid=1' 2024-04-06T09:48:58.005 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:48:58.005 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:48:58.006 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:48:58.006 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=154, tag_tid=29, entry_tid=7354' 2024-04-06T09:48:58.006 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:48:58.007 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:48:58.008 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=7354 2024-04-06T09:48:58.008 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 30 = 29 -a 1 -le 7354 2024-04-06T09:48:58.008 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:48:58.008 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:48:58.008 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:48:58.210 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:48:58.210 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:48:58.210 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:48:58.210 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:48:58.210 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:48:58.210 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:48:58.210 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:48:58.210 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:48:58.211 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:48:58.211 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:48:58.211 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:48:58.211 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:48:58.211 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:48:58.211 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:48:58.211 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:48:58.211 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:48:58.212 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:48:58.214 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:48:58.214 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:48:58.214 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:48:58.214 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:48:58.214 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:48:58.214 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:48:58.215 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:48:58.215 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:48:58.215 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:48:58.215 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:48:58.215 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:48:58.215 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:48:58.215 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:48:58.215 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:48:58.215 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:48:58.215 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:48:58.216 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:48:58.216 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:48:58.216 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:48:58.216 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:48:58.216 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:48:58.216 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:48:58.216 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:48:58.217 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:48:58.217 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:48:58.217 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:48:58.217 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:48:58.217 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:48:58.217 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:48:58.217 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:48:58.217 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:48:58.217 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:48:58.242 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:48:58.243 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:48:58.243 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:48:58.243 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:48:58.347 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-06T09:48:58.347 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:rbd mirror flush : flush rbd mirror 2024-04-06T09:48:58.348 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:admin_socket: invalid command 2024-04-06T09:48:58.350 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:48:58.350 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/test 2024-04-06T09:48:58.350 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:1 2024-04-06T09:48:58.350 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:48:58.350 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:48:58.350 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-06T09:48:58.351 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:1 2024-04-06T09:48:58.351 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:48:58.351 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:48:58.351 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:48:58.351 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=1 2024-04-06T09:48:58.351 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-06T09:48:58.351 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:48:58.351 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:48:58.351 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=1 2024-04-06T09:48:58.351 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=1 2024-04-06T09:48:58.352 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-06T09:48:58.352 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:1 2024-04-06T09:48:58.352 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:48:58.352 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:48:58.352 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-06T09:48:58.353 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:1 2024-04-06T09:48:58.353 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:48:58.353 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:48:58.353 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:48:58.353 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=1 2024-04-06T09:48:58.353 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-06T09:48:58.353 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:48:58.353 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:48:58.353 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=1 2024-04-06T09:48:58.353 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=1 2024-04-06T09:48:58.353 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-06T09:48:58.372 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T09:48:58.373 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T09:48:58.373 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T09:48:58.373 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/test 2024-04-06T09:48:58.508 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:48:58.509 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:48:58.509 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:48:58.509 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:48:58.509 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:48:58.509 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:48:58.509 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:48:58.509 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:48:58.509 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:48:58.509 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:48:58.510 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:48:58.510 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:48:58.510 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:48:58.511 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:48:58.512 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:48:58.512 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:48:58.559 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 38 2024-04-06T09:48:58.559 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 38 2024-04-06T09:48:58.559 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:48:58.559 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=153, tag_tid=30, entry_tid=1], [object_number=152, tag_tid=30, entry_tid=0], [object_number=155, tag_tid=29, entry_tid=9059], [object_number=154, tag_tid=29, entry_tid=9058]]], state=connected] 2024-04-06T09:48:58.560 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=153, tag_tid=29, entry_tid=7461], [object_number=152, tag_tid=29, entry_tid=7460], [object_number=155, tag_tid=29, entry_tid=7459], [object_number=154, tag_tid=29, entry_tid=7458]]], state=connected] 2024-04-06T09:48:58.563 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:48:58.565 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=153, tag_tid=30, entry_tid=1' 2024-04-06T09:48:58.566 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:48:58.566 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:48:58.566 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:48:58.566 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:48:58.566 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:48:58.566 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:48:58.566 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:48:58.566 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:48:58.566 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:48:58.566 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:48:58.566 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:48:58.567 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:48:58.567 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:48:58.567 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:48:58.567 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:48:58.615 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 38 2024-04-06T09:48:58.615 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 38 2024-04-06T09:48:58.616 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:48:58.616 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=153, tag_tid=30, entry_tid=1], [object_number=152, tag_tid=30, entry_tid=0], [object_number=155, tag_tid=29, entry_tid=9059], [object_number=154, tag_tid=29, entry_tid=9058]]], state=connected] 2024-04-06T09:48:58.616 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=153, tag_tid=29, entry_tid=7461], [object_number=152, tag_tid=29, entry_tid=7460], [object_number=155, tag_tid=29, entry_tid=7459], [object_number=154, tag_tid=29, entry_tid=7458]]], state=connected] 2024-04-06T09:48:58.619 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:48:58.621 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=153, tag_tid=29, entry_tid=7461' 2024-04-06T09:48:58.621 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=153, tag_tid=30, entry_tid=1' -a 'object_number=153, tag_tid=30, entry_tid=1' = 'object_number=153, tag_tid=29, entry_tid=7461' 2024-04-06T09:48:58.621 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=153, tag_tid=29, entry_tid=7461' '!=' 'object_number=154, tag_tid=29, entry_tid=7354' 2024-04-06T09:48:58.621 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:48:58.621 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=153, tag_tid=29, entry_tid=7461' = 'object_number=154, tag_tid=29, entry_tid=7354' 2024-04-06T09:48:58.621 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=153, tag_tid=29, entry_tid=7461' 2024-04-06T09:48:58.621 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=153, tag_tid=30, entry_tid=1' 2024-04-06T09:48:58.621 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:48:58.621 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:48:58.622 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=30 2024-04-06T09:48:58.623 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=153, tag_tid=29, entry_tid=7461' 2024-04-06T09:48:58.623 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:48:58.623 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:48:58.624 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=29 2024-04-06T09:48:58.624 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=153, tag_tid=30, entry_tid=1' 2024-04-06T09:48:58.624 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:48:58.624 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:48:58.625 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:48:58.626 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=153, tag_tid=29, entry_tid=7461' 2024-04-06T09:48:58.626 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:48:58.626 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:48:58.627 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=7461 2024-04-06T09:48:58.627 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 30 = 29 -a 1 -le 7461 2024-04-06T09:48:58.628 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:48:58.628 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:48:58.628 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:48:58.829 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:48:58.829 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:48:58.829 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:48:58.829 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:48:58.829 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:48:58.829 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:48:58.829 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:48:58.830 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:48:58.830 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:48:58.830 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:48:58.830 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:48:58.830 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:48:58.830 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:48:58.830 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:48:58.831 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:48:58.831 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:48:58.831 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:48:58.834 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:48:58.834 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:48:58.834 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:48:58.834 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:48:58.834 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:48:58.834 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:48:58.834 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:48:58.834 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:48:58.834 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:48:58.835 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:48:58.835 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:48:58.835 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:48:58.835 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:48:58.835 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:48:58.835 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:48:58.835 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:48:58.835 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:48:58.835 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:48:58.835 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:48:58.835 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:48:58.835 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:48:58.835 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:48:58.835 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:48:58.835 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:48:58.836 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:48:58.836 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:48:58.836 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:48:58.836 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:48:58.836 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:48:58.836 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:48:58.836 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:48:58.836 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:48:58.860 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:48:58.860 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:48:58.860 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:48:58.860 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:48:58.967 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-06T09:48:58.967 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:rbd mirror flush : flush rbd mirror 2024-04-06T09:48:58.968 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:admin_socket: invalid command 2024-04-06T09:48:58.971 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:48:58.971 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/test 2024-04-06T09:48:58.971 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:1 2024-04-06T09:48:58.971 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:48:58.971 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:48:58.971 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-06T09:48:58.971 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:1 2024-04-06T09:48:58.971 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:48:58.971 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:48:58.971 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:48:58.972 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=1 2024-04-06T09:48:58.972 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-06T09:48:58.972 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:48:58.972 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:48:58.972 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=1 2024-04-06T09:48:58.972 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=1 2024-04-06T09:48:58.972 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-06T09:48:58.972 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:1 2024-04-06T09:48:58.972 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:48:58.972 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:48:58.972 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-06T09:48:58.972 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:1 2024-04-06T09:48:58.972 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:48:58.972 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:48:58.972 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:48:58.973 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=1 2024-04-06T09:48:58.973 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-06T09:48:58.973 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:48:58.973 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:48:58.973 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=1 2024-04-06T09:48:58.973 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=1 2024-04-06T09:48:58.973 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-06T09:48:58.994 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T09:48:58.994 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T09:48:58.994 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T09:48:58.994 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/test 2024-04-06T09:48:59.122 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:48:59.123 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:48:59.123 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:48:59.123 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:48:59.123 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:48:59.123 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:48:59.124 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:48:59.124 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:48:59.124 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:48:59.124 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:48:59.124 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:48:59.124 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:48:59.124 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:48:59.126 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:48:59.127 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:48:59.127 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:48:59.182 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 38 2024-04-06T09:48:59.182 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 38 2024-04-06T09:48:59.182 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:48:59.182 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=153, tag_tid=30, entry_tid=1], [object_number=152, tag_tid=30, entry_tid=0], [object_number=155, tag_tid=29, entry_tid=9059], [object_number=154, tag_tid=29, entry_tid=9058]]], state=connected] 2024-04-06T09:48:59.182 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=152, tag_tid=29, entry_tid=7560], [object_number=155, tag_tid=29, entry_tid=7559], [object_number=154, tag_tid=29, entry_tid=7558], [object_number=153, tag_tid=29, entry_tid=7557]]], state=connected] 2024-04-06T09:48:59.186 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:48:59.187 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=153, tag_tid=30, entry_tid=1' 2024-04-06T09:48:59.188 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:48:59.188 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:48:59.188 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:48:59.188 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:48:59.188 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:48:59.188 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:48:59.189 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:48:59.189 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:48:59.189 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:48:59.189 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:48:59.189 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:48:59.189 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:48:59.189 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:48:59.190 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:48:59.190 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:48:59.237 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 38 2024-04-06T09:48:59.237 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 38 2024-04-06T09:48:59.238 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:48:59.238 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=153, tag_tid=30, entry_tid=1], [object_number=152, tag_tid=30, entry_tid=0], [object_number=155, tag_tid=29, entry_tid=9059], [object_number=154, tag_tid=29, entry_tid=9058]]], state=connected] 2024-04-06T09:48:59.238 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=152, tag_tid=29, entry_tid=7560], [object_number=155, tag_tid=29, entry_tid=7559], [object_number=154, tag_tid=29, entry_tid=7558], [object_number=153, tag_tid=29, entry_tid=7557]]], state=connected] 2024-04-06T09:48:59.241 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:48:59.243 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=152, tag_tid=29, entry_tid=7560' 2024-04-06T09:48:59.244 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=153, tag_tid=30, entry_tid=1' -a 'object_number=153, tag_tid=30, entry_tid=1' = 'object_number=152, tag_tid=29, entry_tid=7560' 2024-04-06T09:48:59.244 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=152, tag_tid=29, entry_tid=7560' '!=' 'object_number=153, tag_tid=29, entry_tid=7461' 2024-04-06T09:48:59.244 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:48:59.244 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=152, tag_tid=29, entry_tid=7560' = 'object_number=153, tag_tid=29, entry_tid=7461' 2024-04-06T09:48:59.244 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=152, tag_tid=29, entry_tid=7560' 2024-04-06T09:48:59.244 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=153, tag_tid=30, entry_tid=1' 2024-04-06T09:48:59.244 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:48:59.244 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:48:59.245 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=30 2024-04-06T09:48:59.245 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=152, tag_tid=29, entry_tid=7560' 2024-04-06T09:48:59.245 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:48:59.245 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:48:59.246 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=29 2024-04-06T09:48:59.247 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=153, tag_tid=30, entry_tid=1' 2024-04-06T09:48:59.247 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:48:59.247 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:48:59.248 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:48:59.249 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=152, tag_tid=29, entry_tid=7560' 2024-04-06T09:48:59.249 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:48:59.249 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:48:59.250 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=7560 2024-04-06T09:48:59.250 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 30 = 29 -a 1 -le 7560 2024-04-06T09:48:59.250 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:48:59.250 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:48:59.250 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:48:59.452 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:48:59.452 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:48:59.452 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:48:59.452 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:48:59.452 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:48:59.452 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:48:59.452 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:48:59.452 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:48:59.452 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:48:59.452 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:48:59.452 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:48:59.452 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:48:59.453 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:48:59.453 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:48:59.453 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:48:59.453 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:48:59.454 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:48:59.456 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:48:59.456 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:48:59.457 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:48:59.457 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:48:59.457 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:48:59.457 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:48:59.457 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:48:59.457 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:48:59.457 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:48:59.457 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:48:59.457 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:48:59.458 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:48:59.458 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:48:59.458 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:48:59.458 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:48:59.458 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:48:59.458 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:48:59.458 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:48:59.458 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:48:59.458 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:48:59.458 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:48:59.458 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:48:59.458 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:48:59.458 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:48:59.458 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:48:59.459 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:48:59.459 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:48:59.459 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:48:59.459 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:48:59.459 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:48:59.459 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:48:59.459 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:48:59.483 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:48:59.483 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:48:59.483 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:48:59.483 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:48:59.588 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-06T09:48:59.588 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:rbd mirror flush : flush rbd mirror 2024-04-06T09:48:59.589 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:admin_socket: invalid command 2024-04-06T09:48:59.592 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:48:59.592 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/test 2024-04-06T09:48:59.592 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:1 2024-04-06T09:48:59.592 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:48:59.592 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:48:59.592 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-06T09:48:59.592 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:1 2024-04-06T09:48:59.592 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:48:59.592 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:48:59.592 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:48:59.593 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=1 2024-04-06T09:48:59.593 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-06T09:48:59.593 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:48:59.593 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:48:59.593 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=1 2024-04-06T09:48:59.593 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=1 2024-04-06T09:48:59.593 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-06T09:48:59.593 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:1 2024-04-06T09:48:59.593 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:48:59.593 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:48:59.593 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-06T09:48:59.593 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:1 2024-04-06T09:48:59.593 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:48:59.593 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:48:59.593 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:48:59.594 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=1 2024-04-06T09:48:59.594 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-06T09:48:59.594 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:48:59.594 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:48:59.594 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=1 2024-04-06T09:48:59.594 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=1 2024-04-06T09:48:59.594 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-06T09:48:59.617 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T09:48:59.617 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T09:48:59.617 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T09:48:59.617 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/test 2024-04-06T09:48:59.740 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:48:59.740 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:48:59.741 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:48:59.741 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:48:59.741 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:48:59.741 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:48:59.741 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:48:59.741 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:48:59.741 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:48:59.741 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:48:59.742 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:48:59.742 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:48:59.742 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:48:59.743 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:48:59.744 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:48:59.744 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:48:59.796 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 38 2024-04-06T09:48:59.796 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 38 2024-04-06T09:48:59.797 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:48:59.797 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=153, tag_tid=30, entry_tid=1], [object_number=152, tag_tid=30, entry_tid=0], [object_number=155, tag_tid=29, entry_tid=9059], [object_number=154, tag_tid=29, entry_tid=9058]]], state=connected] 2024-04-06T09:48:59.797 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=155, tag_tid=29, entry_tid=7687], [object_number=154, tag_tid=29, entry_tid=7686], [object_number=153, tag_tid=29, entry_tid=7685], [object_number=152, tag_tid=29, entry_tid=7684]]], state=connected] 2024-04-06T09:48:59.800 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:48:59.802 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=153, tag_tid=30, entry_tid=1' 2024-04-06T09:48:59.803 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:48:59.803 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:48:59.803 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:48:59.803 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:48:59.803 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:48:59.803 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:48:59.803 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:48:59.803 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:48:59.803 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:48:59.803 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:48:59.804 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:48:59.804 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:48:59.804 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:48:59.804 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:48:59.805 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:48:59.852 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 38 2024-04-06T09:48:59.852 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 38 2024-04-06T09:48:59.852 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:48:59.852 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=153, tag_tid=30, entry_tid=1], [object_number=152, tag_tid=30, entry_tid=0], [object_number=155, tag_tid=29, entry_tid=9059], [object_number=154, tag_tid=29, entry_tid=9058]]], state=connected] 2024-04-06T09:48:59.852 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=155, tag_tid=29, entry_tid=7687], [object_number=154, tag_tid=29, entry_tid=7686], [object_number=153, tag_tid=29, entry_tid=7685], [object_number=152, tag_tid=29, entry_tid=7684]]], state=connected] 2024-04-06T09:48:59.856 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:48:59.858 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=155, tag_tid=29, entry_tid=7687' 2024-04-06T09:48:59.858 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=153, tag_tid=30, entry_tid=1' -a 'object_number=153, tag_tid=30, entry_tid=1' = 'object_number=155, tag_tid=29, entry_tid=7687' 2024-04-06T09:48:59.858 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=155, tag_tid=29, entry_tid=7687' '!=' 'object_number=152, tag_tid=29, entry_tid=7560' 2024-04-06T09:48:59.858 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:48:59.858 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=155, tag_tid=29, entry_tid=7687' = 'object_number=152, tag_tid=29, entry_tid=7560' 2024-04-06T09:48:59.858 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=155, tag_tid=29, entry_tid=7687' 2024-04-06T09:48:59.859 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=153, tag_tid=30, entry_tid=1' 2024-04-06T09:48:59.859 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:48:59.859 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:48:59.859 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=30 2024-04-06T09:48:59.860 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=155, tag_tid=29, entry_tid=7687' 2024-04-06T09:48:59.860 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:48:59.860 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:48:59.862 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=29 2024-04-06T09:48:59.862 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=153, tag_tid=30, entry_tid=1' 2024-04-06T09:48:59.862 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:48:59.862 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:48:59.863 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:48:59.863 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=155, tag_tid=29, entry_tid=7687' 2024-04-06T09:48:59.863 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:48:59.864 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:48:59.865 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=7687 2024-04-06T09:48:59.865 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 30 = 29 -a 1 -le 7687 2024-04-06T09:48:59.865 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:48:59.865 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:48:59.865 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:49:00.067 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:49:00.067 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:49:00.067 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:49:00.067 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:49:00.067 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:49:00.067 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:49:00.067 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:49:00.067 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:49:00.067 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:49:00.068 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:49:00.068 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:49:00.068 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:49:00.068 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:49:00.068 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:49:00.068 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:49:00.068 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:49:00.068 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:49:00.071 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:49:00.071 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:49:00.071 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:49:00.071 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:49:00.072 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:49:00.072 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:49:00.072 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:49:00.072 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:49:00.072 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:49:00.072 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:49:00.072 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:49:00.072 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:49:00.072 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:49:00.072 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:49:00.072 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:49:00.072 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:49:00.072 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:49:00.072 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:49:00.072 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:49:00.073 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:49:00.073 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:49:00.073 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:49:00.073 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:49:00.073 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:49:00.073 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:49:00.073 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:49:00.073 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:49:00.073 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:49:00.073 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:49:00.073 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:49:00.073 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:49:00.073 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:49:00.097 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:49:00.097 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:49:00.097 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:49:00.097 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:49:00.208 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-06T09:49:00.208 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:rbd mirror flush : flush rbd mirror 2024-04-06T09:49:00.208 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:admin_socket: invalid command 2024-04-06T09:49:00.210 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:49:00.210 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/test 2024-04-06T09:49:00.210 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:1 2024-04-06T09:49:00.211 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:49:00.211 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:49:00.211 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-06T09:49:00.211 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:1 2024-04-06T09:49:00.211 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:49:00.211 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:49:00.211 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:49:00.211 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=1 2024-04-06T09:49:00.211 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-06T09:49:00.211 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:49:00.211 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:49:00.211 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=1 2024-04-06T09:49:00.211 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=1 2024-04-06T09:49:00.212 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-06T09:49:00.212 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:1 2024-04-06T09:49:00.212 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:49:00.212 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:49:00.212 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-06T09:49:00.213 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:1 2024-04-06T09:49:00.213 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:49:00.213 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:49:00.213 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:49:00.213 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=1 2024-04-06T09:49:00.213 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-06T09:49:00.213 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:49:00.213 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:49:00.213 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=1 2024-04-06T09:49:00.213 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=1 2024-04-06T09:49:00.213 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-06T09:49:00.237 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T09:49:00.237 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T09:49:00.237 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T09:49:00.237 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/test 2024-04-06T09:49:00.362 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:49:00.363 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:49:00.363 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:49:00.363 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:49:00.363 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:49:00.363 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:49:00.363 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:49:00.363 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:49:00.363 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:49:00.363 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:49:00.364 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:49:00.364 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:49:00.364 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:49:00.367 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:49:00.367 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:49:00.367 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:49:00.630 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 38 2024-04-06T09:49:00.630 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 38 2024-04-06T09:49:00.630 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:49:00.630 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=153, tag_tid=30, entry_tid=1], [object_number=152, tag_tid=30, entry_tid=0], [object_number=155, tag_tid=29, entry_tid=9059], [object_number=154, tag_tid=29, entry_tid=9058]]], state=connected] 2024-04-06T09:49:00.630 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=154, tag_tid=29, entry_tid=7798], [object_number=153, tag_tid=29, entry_tid=7797], [object_number=152, tag_tid=29, entry_tid=7796], [object_number=155, tag_tid=29, entry_tid=7795]]], state=connected] 2024-04-06T09:49:00.634 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:49:00.636 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=153, tag_tid=30, entry_tid=1' 2024-04-06T09:49:00.637 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:49:00.637 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:49:00.637 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:49:00.637 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:49:00.637 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:49:00.637 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:49:00.637 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:49:00.637 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:49:00.637 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:49:00.637 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:49:00.638 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:49:00.638 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:49:00.639 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:49:00.639 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:49:00.639 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:49:00.691 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 38 2024-04-06T09:49:00.691 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 38 2024-04-06T09:49:00.691 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:49:00.691 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=153, tag_tid=30, entry_tid=1], [object_number=152, tag_tid=30, entry_tid=0], [object_number=155, tag_tid=29, entry_tid=9059], [object_number=154, tag_tid=29, entry_tid=9058]]], state=connected] 2024-04-06T09:49:00.691 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=154, tag_tid=29, entry_tid=7798], [object_number=153, tag_tid=29, entry_tid=7797], [object_number=152, tag_tid=29, entry_tid=7796], [object_number=155, tag_tid=29, entry_tid=7795]]], state=connected] 2024-04-06T09:49:00.694 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:49:00.696 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=154, tag_tid=29, entry_tid=7798' 2024-04-06T09:49:00.696 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=153, tag_tid=30, entry_tid=1' -a 'object_number=153, tag_tid=30, entry_tid=1' = 'object_number=154, tag_tid=29, entry_tid=7798' 2024-04-06T09:49:00.696 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=154, tag_tid=29, entry_tid=7798' '!=' 'object_number=155, tag_tid=29, entry_tid=7687' 2024-04-06T09:49:00.696 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:49:00.696 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=154, tag_tid=29, entry_tid=7798' = 'object_number=155, tag_tid=29, entry_tid=7687' 2024-04-06T09:49:00.696 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=154, tag_tid=29, entry_tid=7798' 2024-04-06T09:49:00.697 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=153, tag_tid=30, entry_tid=1' 2024-04-06T09:49:00.697 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:49:00.697 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:49:00.698 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=30 2024-04-06T09:49:00.698 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=154, tag_tid=29, entry_tid=7798' 2024-04-06T09:49:00.699 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:49:00.699 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:49:00.699 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=29 2024-04-06T09:49:00.700 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=153, tag_tid=30, entry_tid=1' 2024-04-06T09:49:00.700 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:49:00.700 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:49:00.701 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:49:00.702 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=154, tag_tid=29, entry_tid=7798' 2024-04-06T09:49:00.702 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:49:00.702 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:49:00.703 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=7798 2024-04-06T09:49:00.704 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 30 = 29 -a 1 -le 7798 2024-04-06T09:49:00.704 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:49:00.704 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:49:00.704 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:49:00.905 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:49:00.906 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:49:00.906 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:49:00.906 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:49:00.906 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:49:00.906 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:49:00.906 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:49:00.906 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:49:00.906 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:49:00.906 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:49:00.907 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:49:00.907 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:49:00.907 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:49:00.907 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:49:00.907 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:49:00.907 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:49:00.908 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:49:00.909 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:49:00.910 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:49:00.910 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:49:00.910 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:49:00.910 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:49:00.910 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:49:00.910 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:49:00.910 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:49:00.910 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:49:00.910 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:49:00.910 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:49:00.910 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:49:00.910 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:49:00.910 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:49:00.910 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:49:00.911 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:49:00.911 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:49:00.911 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:49:00.911 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:49:00.911 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:49:00.911 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:49:00.911 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:49:00.911 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:49:00.911 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:49:00.911 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:49:00.911 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:49:00.911 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:49:00.911 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:49:00.912 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:49:00.912 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:49:00.912 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:49:00.912 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:49:00.936 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:49:00.936 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:49:00.937 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:49:00.937 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:49:01.038 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-06T09:49:01.038 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:rbd mirror flush : flush rbd mirror 2024-04-06T09:49:01.038 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:admin_socket: invalid command 2024-04-06T09:49:01.040 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:49:01.040 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/test 2024-04-06T09:49:01.040 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:1 2024-04-06T09:49:01.040 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:49:01.040 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:49:01.040 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-06T09:49:01.040 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:1 2024-04-06T09:49:01.040 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:49:01.040 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:49:01.040 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:49:01.040 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=1 2024-04-06T09:49:01.040 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-06T09:49:01.041 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:49:01.041 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:49:01.041 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=1 2024-04-06T09:49:01.041 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=1 2024-04-06T09:49:01.042 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-06T09:49:01.043 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:1 2024-04-06T09:49:01.043 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:49:01.043 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:49:01.043 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-06T09:49:01.043 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:1 2024-04-06T09:49:01.043 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:49:01.043 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:49:01.043 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:49:01.043 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=1 2024-04-06T09:49:01.043 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-06T09:49:01.043 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:49:01.043 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:49:01.043 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=1 2024-04-06T09:49:01.043 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=1 2024-04-06T09:49:01.044 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-06T09:49:01.065 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T09:49:01.066 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T09:49:01.066 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T09:49:01.066 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/test 2024-04-06T09:49:01.186 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:49:01.187 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:49:01.187 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:49:01.187 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:49:01.187 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:49:01.187 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:49:01.187 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:49:01.188 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:49:01.188 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:49:01.188 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:49:01.188 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:49:01.188 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:49:01.188 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:49:01.189 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:49:01.190 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:49:01.190 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:49:01.442 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 38 2024-04-06T09:49:01.443 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 38 2024-04-06T09:49:01.443 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:49:01.443 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=153, tag_tid=30, entry_tid=1], [object_number=152, tag_tid=30, entry_tid=0], [object_number=155, tag_tid=29, entry_tid=9059], [object_number=154, tag_tid=29, entry_tid=9058]]], state=connected] 2024-04-06T09:49:01.443 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=155, tag_tid=29, entry_tid=7935], [object_number=154, tag_tid=29, entry_tid=7934], [object_number=153, tag_tid=29, entry_tid=7933], [object_number=152, tag_tid=29, entry_tid=7932]]], state=connected] 2024-04-06T09:49:01.448 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:49:01.449 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=153, tag_tid=30, entry_tid=1' 2024-04-06T09:49:01.451 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:49:01.451 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:49:01.451 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:49:01.451 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:49:01.451 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:49:01.451 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:49:01.451 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:49:01.451 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:49:01.451 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:49:01.452 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:49:01.452 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:49:01.452 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:49:01.453 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:49:01.453 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:49:01.453 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:49:01.506 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 38 2024-04-06T09:49:01.506 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 38 2024-04-06T09:49:01.506 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:49:01.506 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=153, tag_tid=30, entry_tid=1], [object_number=152, tag_tid=30, entry_tid=0], [object_number=155, tag_tid=29, entry_tid=9059], [object_number=154, tag_tid=29, entry_tid=9058]]], state=connected] 2024-04-06T09:49:01.506 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=155, tag_tid=29, entry_tid=7935], [object_number=154, tag_tid=29, entry_tid=7934], [object_number=153, tag_tid=29, entry_tid=7933], [object_number=152, tag_tid=29, entry_tid=7932]]], state=connected] 2024-04-06T09:49:01.510 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:49:01.512 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=155, tag_tid=29, entry_tid=7935' 2024-04-06T09:49:01.512 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=153, tag_tid=30, entry_tid=1' -a 'object_number=153, tag_tid=30, entry_tid=1' = 'object_number=155, tag_tid=29, entry_tid=7935' 2024-04-06T09:49:01.512 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=155, tag_tid=29, entry_tid=7935' '!=' 'object_number=154, tag_tid=29, entry_tid=7798' 2024-04-06T09:49:01.512 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:49:01.512 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=155, tag_tid=29, entry_tid=7935' = 'object_number=154, tag_tid=29, entry_tid=7798' 2024-04-06T09:49:01.512 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=155, tag_tid=29, entry_tid=7935' 2024-04-06T09:49:01.513 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=153, tag_tid=30, entry_tid=1' 2024-04-06T09:49:01.513 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:49:01.513 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:49:01.514 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=30 2024-04-06T09:49:01.514 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=155, tag_tid=29, entry_tid=7935' 2024-04-06T09:49:01.514 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:49:01.514 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:49:01.515 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=29 2024-04-06T09:49:01.516 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=153, tag_tid=30, entry_tid=1' 2024-04-06T09:49:01.516 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:49:01.516 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:49:01.517 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:49:01.518 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=155, tag_tid=29, entry_tid=7935' 2024-04-06T09:49:01.518 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:49:01.518 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:49:01.519 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=7935 2024-04-06T09:49:01.520 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 30 = 29 -a 1 -le 7935 2024-04-06T09:49:01.520 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:49:01.520 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:49:01.520 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:49:01.722 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:49:01.722 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:49:01.722 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:49:01.722 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:49:01.722 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:49:01.722 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:49:01.722 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:49:01.722 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:49:01.722 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:49:01.723 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:49:01.723 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:49:01.723 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:49:01.723 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:49:01.723 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:49:01.723 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:49:01.723 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:49:01.723 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:49:01.727 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:49:01.727 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:49:01.727 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:49:01.727 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:49:01.727 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:49:01.727 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:49:01.727 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:49:01.727 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:49:01.727 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:49:01.727 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:49:01.728 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:49:01.728 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:49:01.728 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:49:01.728 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:49:01.728 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:49:01.728 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:49:01.728 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:49:01.728 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:49:01.728 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:49:01.728 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:49:01.728 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:49:01.728 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:49:01.728 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:49:01.728 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:49:01.728 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:49:01.728 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:49:01.729 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:49:01.729 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:49:01.729 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:49:01.729 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:49:01.729 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:49:01.729 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:49:01.752 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:49:01.752 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:49:01.752 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:49:01.752 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:49:01.858 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-06T09:49:01.859 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:rbd mirror flush : flush rbd mirror 2024-04-06T09:49:01.859 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:admin_socket: invalid command 2024-04-06T09:49:01.861 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:49:01.861 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/test 2024-04-06T09:49:01.861 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:1 2024-04-06T09:49:01.861 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:49:01.861 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:49:01.861 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-06T09:49:01.861 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:1 2024-04-06T09:49:01.861 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:49:01.861 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:49:01.861 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:49:01.861 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=1 2024-04-06T09:49:01.861 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-06T09:49:01.862 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:49:01.862 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:49:01.862 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=1 2024-04-06T09:49:01.862 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=1 2024-04-06T09:49:01.864 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-06T09:49:01.864 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:1 2024-04-06T09:49:01.864 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:49:01.864 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:49:01.864 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-06T09:49:01.864 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:1 2024-04-06T09:49:01.864 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:49:01.864 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:49:01.864 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:49:01.864 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=1 2024-04-06T09:49:01.864 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-06T09:49:01.864 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:49:01.865 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:49:01.865 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=1 2024-04-06T09:49:01.865 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=1 2024-04-06T09:49:01.865 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-06T09:49:01.886 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T09:49:01.886 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T09:49:01.886 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T09:49:01.886 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/test 2024-04-06T09:49:02.012 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:49:02.013 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:49:02.013 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:49:02.013 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:49:02.014 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:49:02.014 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:49:02.014 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:49:02.014 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:49:02.014 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:49:02.014 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:49:02.015 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:49:02.015 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:49:02.015 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:49:02.017 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:49:02.018 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:49:02.018 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:49:02.282 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 38 2024-04-06T09:49:02.282 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 38 2024-04-06T09:49:02.282 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:49:02.282 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=153, tag_tid=30, entry_tid=1], [object_number=152, tag_tid=30, entry_tid=0], [object_number=155, tag_tid=29, entry_tid=9059], [object_number=154, tag_tid=29, entry_tid=9058]]], state=connected] 2024-04-06T09:49:02.282 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=152, tag_tid=29, entry_tid=8080], [object_number=155, tag_tid=29, entry_tid=8079], [object_number=154, tag_tid=29, entry_tid=8078], [object_number=153, tag_tid=29, entry_tid=8077]]], state=connected] 2024-04-06T09:49:02.287 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:49:02.289 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=153, tag_tid=30, entry_tid=1' 2024-04-06T09:49:02.289 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:49:02.289 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:49:02.289 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:49:02.289 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:49:02.290 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:49:02.290 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:49:02.290 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:49:02.290 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:49:02.290 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:49:02.291 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:49:02.291 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:49:02.291 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:49:02.292 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:49:02.292 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:49:02.292 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:49:02.348 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 38 2024-04-06T09:49:02.348 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 38 2024-04-06T09:49:02.349 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:49:02.349 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=153, tag_tid=30, entry_tid=1], [object_number=152, tag_tid=30, entry_tid=0], [object_number=155, tag_tid=29, entry_tid=9059], [object_number=154, tag_tid=29, entry_tid=9058]]], state=connected] 2024-04-06T09:49:02.349 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=152, tag_tid=29, entry_tid=8080], [object_number=155, tag_tid=29, entry_tid=8079], [object_number=154, tag_tid=29, entry_tid=8078], [object_number=153, tag_tid=29, entry_tid=8077]]], state=connected] 2024-04-06T09:49:02.352 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:49:02.354 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=152, tag_tid=29, entry_tid=8080' 2024-04-06T09:49:02.354 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=153, tag_tid=30, entry_tid=1' -a 'object_number=153, tag_tid=30, entry_tid=1' = 'object_number=152, tag_tid=29, entry_tid=8080' 2024-04-06T09:49:02.354 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=152, tag_tid=29, entry_tid=8080' '!=' 'object_number=155, tag_tid=29, entry_tid=7935' 2024-04-06T09:49:02.354 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:49:02.354 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=152, tag_tid=29, entry_tid=8080' = 'object_number=155, tag_tid=29, entry_tid=7935' 2024-04-06T09:49:02.355 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=152, tag_tid=29, entry_tid=8080' 2024-04-06T09:49:02.355 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=153, tag_tid=30, entry_tid=1' 2024-04-06T09:49:02.355 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:49:02.355 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:49:02.356 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=30 2024-04-06T09:49:02.357 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=152, tag_tid=29, entry_tid=8080' 2024-04-06T09:49:02.357 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:49:02.357 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:49:02.358 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=29 2024-04-06T09:49:02.359 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=153, tag_tid=30, entry_tid=1' 2024-04-06T09:49:02.359 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:49:02.359 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:49:02.359 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:49:02.360 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=152, tag_tid=29, entry_tid=8080' 2024-04-06T09:49:02.360 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:49:02.360 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:49:02.362 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=8080 2024-04-06T09:49:02.362 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 30 = 29 -a 1 -le 8080 2024-04-06T09:49:02.362 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:49:02.362 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:49:02.362 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:49:02.563 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:49:02.564 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:49:02.564 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:49:02.564 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:49:02.564 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:49:02.564 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:49:02.564 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:49:02.564 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:49:02.564 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:49:02.564 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:49:02.564 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:49:02.564 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:49:02.564 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:49:02.565 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:49:02.565 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:49:02.565 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:49:02.565 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:49:02.569 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:49:02.569 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:49:02.569 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:49:02.569 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:49:02.569 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:49:02.569 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:49:02.569 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:49:02.569 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:49:02.570 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:49:02.570 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:49:02.570 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:49:02.570 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:49:02.570 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:49:02.570 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:49:02.570 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:49:02.570 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:49:02.570 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:49:02.570 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:49:02.570 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:49:02.570 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:49:02.570 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:49:02.570 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:49:02.570 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:49:02.571 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:49:02.571 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:49:02.571 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:49:02.571 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:49:02.571 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:49:02.571 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:49:02.571 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:49:02.571 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:49:02.571 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:49:02.595 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:49:02.595 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:49:02.595 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:49:02.595 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:49:02.704 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-06T09:49:02.704 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:rbd mirror flush : flush rbd mirror 2024-04-06T09:49:02.704 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:admin_socket: invalid command 2024-04-06T09:49:02.707 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:49:02.707 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/test 2024-04-06T09:49:02.708 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:1 2024-04-06T09:49:02.708 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:49:02.708 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:49:02.708 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-06T09:49:02.708 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:1 2024-04-06T09:49:02.708 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:49:02.708 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:49:02.708 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:49:02.708 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=1 2024-04-06T09:49:02.708 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-06T09:49:02.708 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:49:02.708 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:49:02.709 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=1 2024-04-06T09:49:02.709 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=1 2024-04-06T09:49:02.709 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-06T09:49:02.709 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:1 2024-04-06T09:49:02.709 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:49:02.709 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:49:02.709 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-06T09:49:02.709 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:1 2024-04-06T09:49:02.709 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:49:02.709 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:49:02.709 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:49:02.709 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=1 2024-04-06T09:49:02.709 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-06T09:49:02.709 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:49:02.709 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:49:02.710 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=1 2024-04-06T09:49:02.710 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=1 2024-04-06T09:49:02.710 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-06T09:49:02.733 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T09:49:02.733 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T09:49:02.733 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T09:49:02.733 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/test 2024-04-06T09:49:02.855 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:49:02.856 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:49:02.857 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:49:02.857 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:49:02.857 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:49:02.857 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:49:02.857 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:49:02.857 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:49:02.857 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:49:02.857 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:49:02.858 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:49:02.858 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:49:02.858 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:49:02.859 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:49:02.860 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:49:02.860 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:49:03.119 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 38 2024-04-06T09:49:03.119 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 38 2024-04-06T09:49:03.119 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:49:03.119 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=153, tag_tid=30, entry_tid=1], [object_number=152, tag_tid=30, entry_tid=0], [object_number=155, tag_tid=29, entry_tid=9059], [object_number=154, tag_tid=29, entry_tid=9058]]], state=connected] 2024-04-06T09:49:03.120 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=153, tag_tid=29, entry_tid=8241], [object_number=152, tag_tid=29, entry_tid=8240], [object_number=155, tag_tid=29, entry_tid=8239], [object_number=154, tag_tid=29, entry_tid=8238]]], state=connected] 2024-04-06T09:49:03.125 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:49:03.127 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=153, tag_tid=30, entry_tid=1' 2024-04-06T09:49:03.128 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:49:03.128 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:49:03.128 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:49:03.128 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:49:03.128 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:49:03.128 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:49:03.128 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:49:03.128 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:49:03.128 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:49:03.128 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:49:03.129 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:49:03.129 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:49:03.129 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:49:03.130 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:49:03.130 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:49:03.183 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 38 2024-04-06T09:49:03.183 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 38 2024-04-06T09:49:03.183 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:49:03.183 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=153, tag_tid=30, entry_tid=1], [object_number=152, tag_tid=30, entry_tid=0], [object_number=155, tag_tid=29, entry_tid=9059], [object_number=154, tag_tid=29, entry_tid=9058]]], state=connected] 2024-04-06T09:49:03.183 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=153, tag_tid=29, entry_tid=8241], [object_number=152, tag_tid=29, entry_tid=8240], [object_number=155, tag_tid=29, entry_tid=8239], [object_number=154, tag_tid=29, entry_tid=8238]]], state=connected] 2024-04-06T09:49:03.186 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:49:03.189 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=153, tag_tid=29, entry_tid=8241' 2024-04-06T09:49:03.189 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=153, tag_tid=30, entry_tid=1' -a 'object_number=153, tag_tid=30, entry_tid=1' = 'object_number=153, tag_tid=29, entry_tid=8241' 2024-04-06T09:49:03.189 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=153, tag_tid=29, entry_tid=8241' '!=' 'object_number=152, tag_tid=29, entry_tid=8080' 2024-04-06T09:49:03.189 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:49:03.189 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=153, tag_tid=29, entry_tid=8241' = 'object_number=152, tag_tid=29, entry_tid=8080' 2024-04-06T09:49:03.189 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=153, tag_tid=29, entry_tid=8241' 2024-04-06T09:49:03.189 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=153, tag_tid=30, entry_tid=1' 2024-04-06T09:49:03.189 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:49:03.189 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:49:03.190 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=30 2024-04-06T09:49:03.191 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=153, tag_tid=29, entry_tid=8241' 2024-04-06T09:49:03.191 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:49:03.191 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:49:03.192 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=29 2024-04-06T09:49:03.193 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=153, tag_tid=30, entry_tid=1' 2024-04-06T09:49:03.193 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:49:03.193 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:49:03.194 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:49:03.194 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=153, tag_tid=29, entry_tid=8241' 2024-04-06T09:49:03.195 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:49:03.195 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:49:03.196 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=8241 2024-04-06T09:49:03.196 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 30 = 29 -a 1 -le 8241 2024-04-06T09:49:03.196 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:49:03.196 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:49:03.196 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:49:03.398 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:49:03.398 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:49:03.398 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:49:03.398 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:49:03.398 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:49:03.398 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:49:03.398 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:49:03.398 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:49:03.398 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:49:03.398 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:49:03.398 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:49:03.399 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:49:03.399 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:49:03.399 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:49:03.399 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:49:03.399 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:49:03.399 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:49:03.401 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:49:03.401 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:49:03.401 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:49:03.401 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:49:03.401 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:49:03.401 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:49:03.401 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:49:03.401 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:49:03.401 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:49:03.401 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:49:03.402 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:49:03.402 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:49:03.402 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:49:03.402 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:49:03.402 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:49:03.402 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:49:03.403 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:49:03.403 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:49:03.403 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:49:03.403 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:49:03.403 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:49:03.403 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:49:03.403 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:49:03.403 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:49:03.404 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:49:03.404 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:49:03.404 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:49:03.404 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:49:03.404 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:49:03.404 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:49:03.404 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:49:03.404 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:49:03.427 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:49:03.428 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:49:03.428 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:49:03.428 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:49:03.537 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-06T09:49:03.537 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:rbd mirror flush : flush rbd mirror 2024-04-06T09:49:03.537 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:admin_socket: invalid command 2024-04-06T09:49:03.541 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:49:03.541 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/test 2024-04-06T09:49:03.541 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:1 2024-04-06T09:49:03.541 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:49:03.542 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:49:03.542 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-06T09:49:03.542 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:1 2024-04-06T09:49:03.542 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:49:03.542 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:49:03.542 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:49:03.542 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=1 2024-04-06T09:49:03.542 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-06T09:49:03.542 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:49:03.542 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:49:03.542 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=1 2024-04-06T09:49:03.542 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=1 2024-04-06T09:49:03.542 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-06T09:49:03.542 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:1 2024-04-06T09:49:03.543 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:49:03.543 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:49:03.543 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-06T09:49:03.543 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:1 2024-04-06T09:49:03.543 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:49:03.543 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:49:03.543 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:49:03.543 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=1 2024-04-06T09:49:03.543 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-06T09:49:03.543 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:49:03.543 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:49:03.543 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=1 2024-04-06T09:49:03.543 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=1 2024-04-06T09:49:03.543 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-06T09:49:03.563 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T09:49:03.563 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T09:49:03.563 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T09:49:03.563 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/test 2024-04-06T09:49:03.687 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:49:03.688 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:49:03.688 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:49:03.688 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:49:03.689 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:49:03.689 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:49:03.689 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:49:03.689 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:49:03.689 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:49:03.689 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:49:03.689 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:49:03.689 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:49:03.689 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:49:03.690 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:49:03.691 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:49:03.691 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:49:03.740 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 38 2024-04-06T09:49:03.740 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 38 2024-04-06T09:49:03.740 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:49:03.740 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=153, tag_tid=30, entry_tid=1], [object_number=152, tag_tid=30, entry_tid=0], [object_number=155, tag_tid=29, entry_tid=9059], [object_number=154, tag_tid=29, entry_tid=9058]]], state=connected] 2024-04-06T09:49:03.740 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=154, tag_tid=29, entry_tid=8386], [object_number=153, tag_tid=29, entry_tid=8385], [object_number=152, tag_tid=29, entry_tid=8384], [object_number=155, tag_tid=29, entry_tid=8383]]], state=connected] 2024-04-06T09:49:03.743 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:49:03.744 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=153, tag_tid=30, entry_tid=1' 2024-04-06T09:49:03.746 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:49:03.746 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:49:03.746 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:49:03.746 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:49:03.746 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:49:03.746 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:49:03.746 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:49:03.746 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:49:03.746 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:49:03.746 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:49:03.746 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:49:03.747 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:49:03.747 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:49:03.747 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:49:03.747 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:49:03.795 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 38 2024-04-06T09:49:03.795 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 38 2024-04-06T09:49:03.795 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:49:03.795 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=153, tag_tid=30, entry_tid=1], [object_number=152, tag_tid=30, entry_tid=0], [object_number=155, tag_tid=29, entry_tid=9059], [object_number=154, tag_tid=29, entry_tid=9058]]], state=connected] 2024-04-06T09:49:03.795 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=154, tag_tid=29, entry_tid=8386], [object_number=153, tag_tid=29, entry_tid=8385], [object_number=152, tag_tid=29, entry_tid=8384], [object_number=155, tag_tid=29, entry_tid=8383]]], state=connected] 2024-04-06T09:49:03.798 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:49:03.800 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=154, tag_tid=29, entry_tid=8386' 2024-04-06T09:49:03.800 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=153, tag_tid=30, entry_tid=1' -a 'object_number=153, tag_tid=30, entry_tid=1' = 'object_number=154, tag_tid=29, entry_tid=8386' 2024-04-06T09:49:03.800 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=154, tag_tid=29, entry_tid=8386' '!=' 'object_number=153, tag_tid=29, entry_tid=8241' 2024-04-06T09:49:03.800 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:49:03.800 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=154, tag_tid=29, entry_tid=8386' = 'object_number=153, tag_tid=29, entry_tid=8241' 2024-04-06T09:49:03.801 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=154, tag_tid=29, entry_tid=8386' 2024-04-06T09:49:03.801 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=153, tag_tid=30, entry_tid=1' 2024-04-06T09:49:03.801 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:49:03.801 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:49:03.802 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=30 2024-04-06T09:49:03.802 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=154, tag_tid=29, entry_tid=8386' 2024-04-06T09:49:03.803 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:49:03.803 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:49:03.803 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=29 2024-04-06T09:49:03.804 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=153, tag_tid=30, entry_tid=1' 2024-04-06T09:49:03.804 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:49:03.804 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:49:03.805 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:49:03.806 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=154, tag_tid=29, entry_tid=8386' 2024-04-06T09:49:03.806 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:49:03.806 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:49:03.807 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=8386 2024-04-06T09:49:03.807 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 30 = 29 -a 1 -le 8386 2024-04-06T09:49:03.807 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:49:03.807 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:49:03.807 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:49:04.009 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:49:04.009 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:49:04.009 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:49:04.009 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:49:04.009 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:49:04.009 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:49:04.009 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:49:04.009 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:49:04.009 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:49:04.009 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:49:04.010 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:49:04.010 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:49:04.010 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:49:04.010 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:49:04.010 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:49:04.010 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:49:04.011 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:49:04.013 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:49:04.013 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:49:04.014 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:49:04.014 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:49:04.014 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:49:04.014 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:49:04.014 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:49:04.014 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:49:04.014 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:49:04.014 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:49:04.014 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:49:04.014 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:49:04.014 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:49:04.014 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:49:04.015 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:49:04.015 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:49:04.015 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:49:04.015 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:49:04.015 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:49:04.015 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:49:04.015 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:49:04.015 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:49:04.015 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:49:04.015 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:49:04.015 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:49:04.015 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:49:04.015 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:49:04.015 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:49:04.015 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:49:04.016 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:49:04.016 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:49:04.016 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:49:04.038 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:49:04.039 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:49:04.039 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:49:04.039 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:49:04.142 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-06T09:49:04.142 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:rbd mirror flush : flush rbd mirror 2024-04-06T09:49:04.142 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:admin_socket: invalid command 2024-04-06T09:49:04.144 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:49:04.145 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/test 2024-04-06T09:49:04.145 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:1 2024-04-06T09:49:04.145 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:49:04.145 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:49:04.145 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-06T09:49:04.145 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:1 2024-04-06T09:49:04.145 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:49:04.145 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:49:04.145 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:49:04.145 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=1 2024-04-06T09:49:04.145 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-06T09:49:04.145 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:49:04.145 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:49:04.146 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=1 2024-04-06T09:49:04.146 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=1 2024-04-06T09:49:04.147 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-06T09:49:04.147 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:1 2024-04-06T09:49:04.147 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:49:04.147 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:49:04.147 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-06T09:49:04.147 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:1 2024-04-06T09:49:04.148 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:49:04.148 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:49:04.148 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:49:04.148 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=1 2024-04-06T09:49:04.148 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-06T09:49:04.148 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:49:04.148 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:49:04.148 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=1 2024-04-06T09:49:04.148 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=1 2024-04-06T09:49:04.148 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-06T09:49:04.170 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T09:49:04.170 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T09:49:04.171 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T09:49:04.171 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/test 2024-04-06T09:49:04.296 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:49:04.298 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:49:04.298 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:49:04.298 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:49:04.298 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:49:04.298 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:49:04.298 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:49:04.298 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:49:04.298 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:49:04.298 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:49:04.299 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:49:04.299 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:49:04.299 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:49:04.301 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:49:04.302 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:49:04.302 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:49:04.356 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 38 2024-04-06T09:49:04.356 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 38 2024-04-06T09:49:04.356 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:49:04.356 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=153, tag_tid=30, entry_tid=1], [object_number=152, tag_tid=30, entry_tid=0], [object_number=155, tag_tid=29, entry_tid=9059], [object_number=154, tag_tid=29, entry_tid=9058]]], state=connected] 2024-04-06T09:49:04.356 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=152, tag_tid=29, entry_tid=8496], [object_number=155, tag_tid=29, entry_tid=8495], [object_number=154, tag_tid=29, entry_tid=8494], [object_number=153, tag_tid=29, entry_tid=8493]]], state=connected] 2024-04-06T09:49:04.360 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:49:04.362 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=153, tag_tid=30, entry_tid=1' 2024-04-06T09:49:04.363 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:49:04.363 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:49:04.363 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:49:04.363 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:49:04.363 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:49:04.363 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:49:04.363 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:49:04.363 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:49:04.363 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:49:04.363 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:49:04.363 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:49:04.363 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:49:04.364 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:49:04.365 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:49:04.365 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:49:04.415 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 38 2024-04-06T09:49:04.415 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 38 2024-04-06T09:49:04.415 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:49:04.415 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=153, tag_tid=30, entry_tid=1], [object_number=152, tag_tid=30, entry_tid=0], [object_number=155, tag_tid=29, entry_tid=9059], [object_number=154, tag_tid=29, entry_tid=9058]]], state=connected] 2024-04-06T09:49:04.415 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=152, tag_tid=29, entry_tid=8496], [object_number=155, tag_tid=29, entry_tid=8495], [object_number=154, tag_tid=29, entry_tid=8494], [object_number=153, tag_tid=29, entry_tid=8493]]], state=connected] 2024-04-06T09:49:04.419 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:49:04.421 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=152, tag_tid=29, entry_tid=8496' 2024-04-06T09:49:04.421 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=153, tag_tid=30, entry_tid=1' -a 'object_number=153, tag_tid=30, entry_tid=1' = 'object_number=152, tag_tid=29, entry_tid=8496' 2024-04-06T09:49:04.421 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=152, tag_tid=29, entry_tid=8496' '!=' 'object_number=154, tag_tid=29, entry_tid=8386' 2024-04-06T09:49:04.421 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:49:04.421 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=152, tag_tid=29, entry_tid=8496' = 'object_number=154, tag_tid=29, entry_tid=8386' 2024-04-06T09:49:04.421 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=152, tag_tid=29, entry_tid=8496' 2024-04-06T09:49:04.421 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=153, tag_tid=30, entry_tid=1' 2024-04-06T09:49:04.421 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:49:04.422 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:49:04.422 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=30 2024-04-06T09:49:04.423 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=152, tag_tid=29, entry_tid=8496' 2024-04-06T09:49:04.423 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:49:04.423 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:49:04.423 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=29 2024-04-06T09:49:04.424 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=153, tag_tid=30, entry_tid=1' 2024-04-06T09:49:04.424 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:49:04.424 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:49:04.425 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:49:04.426 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=152, tag_tid=29, entry_tid=8496' 2024-04-06T09:49:04.426 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:49:04.426 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:49:04.427 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=8496 2024-04-06T09:49:04.427 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 30 = 29 -a 1 -le 8496 2024-04-06T09:49:04.428 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:49:04.428 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:49:04.428 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:49:04.629 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:49:04.630 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:49:04.630 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:49:04.630 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:49:04.630 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:49:04.630 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:49:04.630 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:49:04.630 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:49:04.630 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:49:04.630 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:49:04.630 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:49:04.630 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:49:04.630 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:49:04.630 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:49:04.631 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:49:04.631 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:49:04.631 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:49:04.633 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:49:04.633 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:49:04.634 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:49:04.634 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:49:04.634 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:49:04.634 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:49:04.634 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:49:04.634 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:49:04.634 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:49:04.634 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:49:04.634 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:49:04.634 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:49:04.634 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:49:04.634 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:49:04.634 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:49:04.635 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:49:04.636 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:49:04.636 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:49:04.636 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:49:04.636 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:49:04.636 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:49:04.636 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:49:04.636 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:49:04.636 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:49:04.636 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:49:04.636 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:49:04.636 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:49:04.637 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:49:04.637 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:49:04.637 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:49:04.637 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:49:04.637 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:49:04.661 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:49:04.662 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:49:04.662 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:49:04.662 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:49:04.768 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-06T09:49:04.768 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:rbd mirror flush : flush rbd mirror 2024-04-06T09:49:04.768 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:admin_socket: invalid command 2024-04-06T09:49:04.771 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:49:04.771 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/test 2024-04-06T09:49:04.771 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:1 2024-04-06T09:49:04.771 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:49:04.771 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:49:04.771 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-06T09:49:04.771 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:1 2024-04-06T09:49:04.771 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:49:04.771 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:49:04.771 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:49:04.771 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=1 2024-04-06T09:49:04.771 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-06T09:49:04.771 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:49:04.772 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:49:04.772 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=1 2024-04-06T09:49:04.772 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=1 2024-04-06T09:49:04.772 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-06T09:49:04.772 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:1 2024-04-06T09:49:04.772 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:49:04.772 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:49:04.772 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-06T09:49:04.772 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:1 2024-04-06T09:49:04.772 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:49:04.772 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:49:04.772 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:49:04.772 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=1 2024-04-06T09:49:04.772 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-06T09:49:04.773 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:49:04.773 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:49:04.773 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=1 2024-04-06T09:49:04.773 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=1 2024-04-06T09:49:04.773 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-06T09:49:04.797 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T09:49:04.797 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T09:49:04.797 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T09:49:04.797 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/test 2024-04-06T09:49:04.935 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:49:04.936 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:49:04.936 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:49:04.936 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:49:04.936 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:49:04.936 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:49:04.936 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:49:04.936 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:49:04.936 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:49:04.936 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:49:04.937 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:49:04.937 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:49:04.937 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:49:04.938 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:49:04.939 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:49:04.939 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:49:04.991 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 38 2024-04-06T09:49:04.992 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 38 2024-04-06T09:49:04.992 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:49:04.992 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=153, tag_tid=30, entry_tid=1], [object_number=152, tag_tid=30, entry_tid=0], [object_number=155, tag_tid=29, entry_tid=9059], [object_number=154, tag_tid=29, entry_tid=9058]]], state=connected] 2024-04-06T09:49:04.992 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=155, tag_tid=29, entry_tid=8603], [object_number=154, tag_tid=29, entry_tid=8602], [object_number=153, tag_tid=29, entry_tid=8601], [object_number=152, tag_tid=29, entry_tid=8600]]], state=connected] 2024-04-06T09:49:04.995 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:49:04.997 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=153, tag_tid=30, entry_tid=1' 2024-04-06T09:49:04.998 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:49:04.998 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:49:04.998 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:49:04.998 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:49:04.998 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:49:04.998 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:49:04.998 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:49:04.998 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:49:04.998 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:49:04.999 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:49:04.999 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:49:04.999 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:49:04.999 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:49:04.999 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:49:04.999 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:49:05.048 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 38 2024-04-06T09:49:05.048 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 38 2024-04-06T09:49:05.049 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:49:05.049 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=153, tag_tid=30, entry_tid=1], [object_number=152, tag_tid=30, entry_tid=0], [object_number=155, tag_tid=29, entry_tid=9059], [object_number=154, tag_tid=29, entry_tid=9058]]], state=connected] 2024-04-06T09:49:05.049 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=155, tag_tid=29, entry_tid=8603], [object_number=154, tag_tid=29, entry_tid=8602], [object_number=153, tag_tid=29, entry_tid=8601], [object_number=152, tag_tid=29, entry_tid=8600]]], state=connected] 2024-04-06T09:49:05.052 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:49:05.054 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=155, tag_tid=29, entry_tid=8603' 2024-04-06T09:49:05.054 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=153, tag_tid=30, entry_tid=1' -a 'object_number=153, tag_tid=30, entry_tid=1' = 'object_number=155, tag_tid=29, entry_tid=8603' 2024-04-06T09:49:05.054 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=155, tag_tid=29, entry_tid=8603' '!=' 'object_number=152, tag_tid=29, entry_tid=8496' 2024-04-06T09:49:05.054 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:49:05.054 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=155, tag_tid=29, entry_tid=8603' = 'object_number=152, tag_tid=29, entry_tid=8496' 2024-04-06T09:49:05.054 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=155, tag_tid=29, entry_tid=8603' 2024-04-06T09:49:05.054 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=153, tag_tid=30, entry_tid=1' 2024-04-06T09:49:05.055 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:49:05.055 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:49:05.055 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=30 2024-04-06T09:49:05.056 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=155, tag_tid=29, entry_tid=8603' 2024-04-06T09:49:05.056 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:49:05.056 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:49:05.057 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=29 2024-04-06T09:49:05.058 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=153, tag_tid=30, entry_tid=1' 2024-04-06T09:49:05.058 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:49:05.058 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:49:05.059 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:49:05.059 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=155, tag_tid=29, entry_tid=8603' 2024-04-06T09:49:05.059 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:49:05.060 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:49:05.061 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=8603 2024-04-06T09:49:05.061 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 30 = 29 -a 1 -le 8603 2024-04-06T09:49:05.061 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:49:05.061 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:49:05.061 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:49:05.263 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:49:05.263 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:49:05.263 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:49:05.263 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:49:05.263 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:49:05.263 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:49:05.263 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:49:05.264 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:49:05.264 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:49:05.264 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:49:05.264 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:49:05.264 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:49:05.264 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:49:05.264 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:49:05.264 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:49:05.265 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:49:05.265 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:49:05.268 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:49:05.268 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:49:05.268 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:49:05.268 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:49:05.268 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:49:05.268 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:49:05.268 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:49:05.268 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:49:05.268 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:49:05.268 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:49:05.268 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:49:05.269 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:49:05.269 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:49:05.269 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:49:05.269 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:49:05.269 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:49:05.269 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:49:05.269 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:49:05.269 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:49:05.269 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:49:05.269 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:49:05.269 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:49:05.269 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:49:05.269 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:49:05.269 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:49:05.270 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:49:05.270 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:49:05.270 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:49:05.270 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:49:05.270 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:49:05.270 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:49:05.270 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:49:05.294 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:49:05.295 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:49:05.295 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:49:05.295 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:49:05.401 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-06T09:49:05.401 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:rbd mirror flush : flush rbd mirror 2024-04-06T09:49:05.401 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:admin_socket: invalid command 2024-04-06T09:49:05.404 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:49:05.404 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/test 2024-04-06T09:49:05.404 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:1 2024-04-06T09:49:05.404 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:49:05.404 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:49:05.404 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-06T09:49:05.404 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:1 2024-04-06T09:49:05.404 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:49:05.404 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:49:05.404 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:49:05.404 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=1 2024-04-06T09:49:05.404 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-06T09:49:05.405 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:49:05.405 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:49:05.405 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=1 2024-04-06T09:49:05.405 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=1 2024-04-06T09:49:05.406 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-06T09:49:05.406 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:1 2024-04-06T09:49:05.406 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:49:05.406 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:49:05.406 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-06T09:49:05.406 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:1 2024-04-06T09:49:05.407 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:49:05.407 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:49:05.407 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:49:05.407 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=1 2024-04-06T09:49:05.407 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-06T09:49:05.407 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:49:05.407 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:49:05.407 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=1 2024-04-06T09:49:05.407 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=1 2024-04-06T09:49:05.407 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-06T09:49:05.430 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T09:49:05.430 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T09:49:05.430 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T09:49:05.430 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/test 2024-04-06T09:49:05.561 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:49:05.562 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:49:05.562 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:49:05.563 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:49:05.563 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:49:05.563 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:49:05.563 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:49:05.563 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:49:05.563 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:49:05.563 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:49:05.563 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:49:05.564 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:49:05.564 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:49:05.564 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:49:05.565 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:49:05.565 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:49:05.613 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 38 2024-04-06T09:49:05.613 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 38 2024-04-06T09:49:05.614 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:49:05.614 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=153, tag_tid=30, entry_tid=1], [object_number=152, tag_tid=30, entry_tid=0], [object_number=155, tag_tid=29, entry_tid=9059], [object_number=154, tag_tid=29, entry_tid=9058]]], state=connected] 2024-04-06T09:49:05.614 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=155, tag_tid=29, entry_tid=8711], [object_number=154, tag_tid=29, entry_tid=8710], [object_number=153, tag_tid=29, entry_tid=8709], [object_number=152, tag_tid=29, entry_tid=8708]]], state=connected] 2024-04-06T09:49:05.617 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:49:05.619 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=153, tag_tid=30, entry_tid=1' 2024-04-06T09:49:05.620 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:49:05.620 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:49:05.620 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:49:05.620 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:49:05.620 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:49:05.620 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:49:05.620 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:49:05.620 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:49:05.620 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:49:05.620 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:49:05.620 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:49:05.620 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:49:05.621 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:49:05.622 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:49:05.622 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:49:05.670 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 38 2024-04-06T09:49:05.670 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 38 2024-04-06T09:49:05.670 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:49:05.670 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=153, tag_tid=30, entry_tid=1], [object_number=152, tag_tid=30, entry_tid=0], [object_number=155, tag_tid=29, entry_tid=9059], [object_number=154, tag_tid=29, entry_tid=9058]]], state=connected] 2024-04-06T09:49:05.670 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=155, tag_tid=29, entry_tid=8711], [object_number=154, tag_tid=29, entry_tid=8710], [object_number=153, tag_tid=29, entry_tid=8709], [object_number=152, tag_tid=29, entry_tid=8708]]], state=connected] 2024-04-06T09:49:05.673 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:49:05.675 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=155, tag_tid=29, entry_tid=8711' 2024-04-06T09:49:05.675 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=153, tag_tid=30, entry_tid=1' -a 'object_number=153, tag_tid=30, entry_tid=1' = 'object_number=155, tag_tid=29, entry_tid=8711' 2024-04-06T09:49:05.675 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=155, tag_tid=29, entry_tid=8711' '!=' 'object_number=155, tag_tid=29, entry_tid=8603' 2024-04-06T09:49:05.675 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:49:05.675 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=155, tag_tid=29, entry_tid=8711' = 'object_number=155, tag_tid=29, entry_tid=8603' 2024-04-06T09:49:05.675 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=155, tag_tid=29, entry_tid=8711' 2024-04-06T09:49:05.676 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=153, tag_tid=30, entry_tid=1' 2024-04-06T09:49:05.676 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:49:05.676 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:49:05.676 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=30 2024-04-06T09:49:05.677 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=155, tag_tid=29, entry_tid=8711' 2024-04-06T09:49:05.677 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:49:05.677 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:49:05.678 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=29 2024-04-06T09:49:05.679 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=153, tag_tid=30, entry_tid=1' 2024-04-06T09:49:05.679 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:49:05.679 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:49:05.680 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:49:05.681 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=155, tag_tid=29, entry_tid=8711' 2024-04-06T09:49:05.681 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:49:05.681 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:49:05.682 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=8711 2024-04-06T09:49:05.682 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 30 = 29 -a 1 -le 8711 2024-04-06T09:49:05.682 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:49:05.682 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:49:05.682 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:49:05.884 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:49:05.884 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:49:05.884 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:49:05.884 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:49:05.884 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:49:05.884 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:49:05.884 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:49:05.884 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:49:05.884 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:49:05.884 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:49:05.884 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:49:05.884 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:49:05.884 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:49:05.885 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:49:05.885 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:49:05.885 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:49:05.885 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:49:05.888 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:49:05.888 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:49:05.888 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:49:05.888 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:49:05.888 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:49:05.888 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:49:05.888 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:49:05.888 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:49:05.888 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:49:05.888 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:49:05.888 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:49:05.889 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:49:05.889 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:49:05.889 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:49:05.889 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:49:05.889 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:49:05.890 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:49:05.890 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:49:05.890 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:49:05.890 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:49:05.890 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:49:05.890 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:49:05.890 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:49:05.890 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:49:05.890 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:49:05.890 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:49:05.890 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:49:05.891 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:49:05.891 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:49:05.891 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:49:05.891 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:49:05.891 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:49:05.915 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:49:05.916 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:49:05.916 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:49:05.916 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:49:06.022 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-06T09:49:06.022 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:rbd mirror flush : flush rbd mirror 2024-04-06T09:49:06.022 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:admin_socket: invalid command 2024-04-06T09:49:06.027 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:49:06.027 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/test 2024-04-06T09:49:06.027 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:1 2024-04-06T09:49:06.027 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:49:06.027 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:49:06.027 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-06T09:49:06.027 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:1 2024-04-06T09:49:06.027 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:49:06.027 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:49:06.027 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:49:06.027 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=1 2024-04-06T09:49:06.027 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-06T09:49:06.027 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:49:06.028 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:49:06.028 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=1 2024-04-06T09:49:06.028 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=1 2024-04-06T09:49:06.028 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-06T09:49:06.028 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:1 2024-04-06T09:49:06.028 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:49:06.028 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:49:06.028 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-06T09:49:06.028 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:1 2024-04-06T09:49:06.028 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:49:06.028 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:49:06.028 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:49:06.028 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=1 2024-04-06T09:49:06.028 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-06T09:49:06.028 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:49:06.028 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:49:06.029 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=1 2024-04-06T09:49:06.029 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=1 2024-04-06T09:49:06.029 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-06T09:49:06.050 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T09:49:06.051 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T09:49:06.051 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T09:49:06.051 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/test 2024-04-06T09:49:06.178 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:49:06.179 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:49:06.179 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:49:06.179 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:49:06.179 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:49:06.179 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:49:06.179 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:49:06.179 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:49:06.179 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:49:06.179 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:49:06.179 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:49:06.180 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:49:06.180 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:49:06.182 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:49:06.182 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:49:06.182 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:49:06.230 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 38 2024-04-06T09:49:06.230 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 38 2024-04-06T09:49:06.230 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:49:06.230 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=153, tag_tid=30, entry_tid=1], [object_number=152, tag_tid=30, entry_tid=0], [object_number=155, tag_tid=29, entry_tid=9059], [object_number=154, tag_tid=29, entry_tid=9058]]], state=connected] 2024-04-06T09:49:06.230 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=152, tag_tid=29, entry_tid=8808], [object_number=155, tag_tid=29, entry_tid=8807], [object_number=154, tag_tid=29, entry_tid=8806], [object_number=153, tag_tid=29, entry_tid=8805]]], state=connected] 2024-04-06T09:49:06.233 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:49:06.235 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=153, tag_tid=30, entry_tid=1' 2024-04-06T09:49:06.236 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:49:06.236 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:49:06.236 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:49:06.236 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:49:06.236 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:49:06.236 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:49:06.236 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:49:06.236 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:49:06.236 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:49:06.236 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:49:06.237 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:49:06.237 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:49:06.237 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:49:06.237 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:49:06.237 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:49:06.284 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 38 2024-04-06T09:49:06.285 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 38 2024-04-06T09:49:06.285 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:49:06.285 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=153, tag_tid=30, entry_tid=1], [object_number=152, tag_tid=30, entry_tid=0], [object_number=155, tag_tid=29, entry_tid=9059], [object_number=154, tag_tid=29, entry_tid=9058]]], state=connected] 2024-04-06T09:49:06.285 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=152, tag_tid=29, entry_tid=8808], [object_number=155, tag_tid=29, entry_tid=8807], [object_number=154, tag_tid=29, entry_tid=8806], [object_number=153, tag_tid=29, entry_tid=8805]]], state=connected] 2024-04-06T09:49:06.288 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:49:06.290 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=152, tag_tid=29, entry_tid=8808' 2024-04-06T09:49:06.290 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=153, tag_tid=30, entry_tid=1' -a 'object_number=153, tag_tid=30, entry_tid=1' = 'object_number=152, tag_tid=29, entry_tid=8808' 2024-04-06T09:49:06.290 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=152, tag_tid=29, entry_tid=8808' '!=' 'object_number=155, tag_tid=29, entry_tid=8711' 2024-04-06T09:49:06.290 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:49:06.290 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=152, tag_tid=29, entry_tid=8808' = 'object_number=155, tag_tid=29, entry_tid=8711' 2024-04-06T09:49:06.290 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=152, tag_tid=29, entry_tid=8808' 2024-04-06T09:49:06.291 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=153, tag_tid=30, entry_tid=1' 2024-04-06T09:49:06.291 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:49:06.291 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:49:06.291 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=30 2024-04-06T09:49:06.292 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=152, tag_tid=29, entry_tid=8808' 2024-04-06T09:49:06.292 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:49:06.292 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:49:06.293 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=29 2024-04-06T09:49:06.294 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=153, tag_tid=30, entry_tid=1' 2024-04-06T09:49:06.294 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:49:06.294 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:49:06.295 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:49:06.296 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=152, tag_tid=29, entry_tid=8808' 2024-04-06T09:49:06.296 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:49:06.296 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:49:06.297 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=8808 2024-04-06T09:49:06.297 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 30 = 29 -a 1 -le 8808 2024-04-06T09:49:06.297 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:49:06.297 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:49:06.297 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:49:06.499 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:49:06.499 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:49:06.499 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:49:06.499 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:49:06.499 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:49:06.499 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:49:06.499 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:49:06.499 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:49:06.500 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:49:06.500 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:49:06.500 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:49:06.500 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:49:06.500 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:49:06.500 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:49:06.500 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:49:06.500 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:49:06.501 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:49:06.502 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:49:06.502 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:49:06.503 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:49:06.503 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:49:06.503 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:49:06.503 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:49:06.503 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:49:06.503 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:49:06.503 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:49:06.503 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:49:06.503 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:49:06.504 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:49:06.504 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:49:06.504 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:49:06.505 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:49:06.505 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:49:06.505 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:49:06.505 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:49:06.505 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:49:06.505 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:49:06.505 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:49:06.505 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:49:06.505 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:49:06.505 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:49:06.505 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:49:06.505 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:49:06.505 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:49:06.505 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:49:06.506 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:49:06.506 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:49:06.506 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:49:06.506 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:49:06.531 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:49:06.531 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:49:06.531 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:49:06.531 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:49:06.634 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-06T09:49:06.634 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:rbd mirror flush : flush rbd mirror 2024-04-06T09:49:06.634 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:admin_socket: invalid command 2024-04-06T09:49:06.636 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:49:06.636 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/test 2024-04-06T09:49:06.636 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:1 2024-04-06T09:49:06.636 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:49:06.636 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:49:06.637 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-06T09:49:06.637 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:1 2024-04-06T09:49:06.637 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:49:06.637 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:49:06.637 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:49:06.637 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=1 2024-04-06T09:49:06.637 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-06T09:49:06.637 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:49:06.637 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:49:06.637 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=1 2024-04-06T09:49:06.637 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=1 2024-04-06T09:49:06.638 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-06T09:49:06.639 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:1 2024-04-06T09:49:06.639 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:49:06.639 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:49:06.639 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-06T09:49:06.639 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:1 2024-04-06T09:49:06.639 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:49:06.639 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:49:06.639 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:49:06.639 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=1 2024-04-06T09:49:06.639 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-06T09:49:06.639 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:49:06.639 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:49:06.639 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=1 2024-04-06T09:49:06.639 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=1 2024-04-06T09:49:06.640 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-06T09:49:06.663 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T09:49:06.663 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T09:49:06.663 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T09:49:06.663 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/test 2024-04-06T09:49:06.789 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:49:06.790 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:49:06.790 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:49:06.790 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:49:06.791 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:49:06.791 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:49:06.791 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:49:06.791 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:49:06.791 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:49:06.791 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:49:06.791 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:49:06.792 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:49:06.792 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:49:06.793 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:49:06.794 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:49:06.794 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:49:06.847 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 38 2024-04-06T09:49:06.847 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 38 2024-04-06T09:49:06.847 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:49:06.847 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=153, tag_tid=30, entry_tid=1], [object_number=152, tag_tid=30, entry_tid=0], [object_number=155, tag_tid=29, entry_tid=9059], [object_number=154, tag_tid=29, entry_tid=9058]]], state=connected] 2024-04-06T09:49:06.847 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=152, tag_tid=29, entry_tid=8912], [object_number=155, tag_tid=29, entry_tid=8911], [object_number=154, tag_tid=29, entry_tid=8910], [object_number=153, tag_tid=29, entry_tid=8909]]], state=connected] 2024-04-06T09:49:06.851 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:49:06.852 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=153, tag_tid=30, entry_tid=1' 2024-04-06T09:49:06.853 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:49:06.853 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:49:06.854 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:49:06.854 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:49:06.854 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:49:06.854 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:49:06.854 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:49:06.854 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:49:06.854 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:49:06.854 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:49:06.854 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:49:06.854 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:49:06.855 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:49:06.855 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:49:06.855 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:49:06.903 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 38 2024-04-06T09:49:06.903 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 38 2024-04-06T09:49:06.903 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:49:06.904 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=153, tag_tid=30, entry_tid=1], [object_number=152, tag_tid=30, entry_tid=0], [object_number=155, tag_tid=29, entry_tid=9059], [object_number=154, tag_tid=29, entry_tid=9058]]], state=connected] 2024-04-06T09:49:06.904 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=152, tag_tid=29, entry_tid=8912], [object_number=155, tag_tid=29, entry_tid=8911], [object_number=154, tag_tid=29, entry_tid=8910], [object_number=153, tag_tid=29, entry_tid=8909]]], state=connected] 2024-04-06T09:49:06.907 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:49:06.908 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=152, tag_tid=29, entry_tid=8912' 2024-04-06T09:49:06.909 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=153, tag_tid=30, entry_tid=1' -a 'object_number=153, tag_tid=30, entry_tid=1' = 'object_number=152, tag_tid=29, entry_tid=8912' 2024-04-06T09:49:06.909 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=152, tag_tid=29, entry_tid=8912' '!=' 'object_number=152, tag_tid=29, entry_tid=8808' 2024-04-06T09:49:06.909 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:49:06.909 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=152, tag_tid=29, entry_tid=8912' = 'object_number=152, tag_tid=29, entry_tid=8808' 2024-04-06T09:49:06.909 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=152, tag_tid=29, entry_tid=8912' 2024-04-06T09:49:06.909 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=153, tag_tid=30, entry_tid=1' 2024-04-06T09:49:06.910 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:49:06.910 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:49:06.910 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=30 2024-04-06T09:49:06.911 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=152, tag_tid=29, entry_tid=8912' 2024-04-06T09:49:06.911 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:49:06.911 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:49:06.912 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=29 2024-04-06T09:49:06.913 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=153, tag_tid=30, entry_tid=1' 2024-04-06T09:49:06.913 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:49:06.913 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:49:06.914 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:49:06.914 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=152, tag_tid=29, entry_tid=8912' 2024-04-06T09:49:06.914 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:49:06.914 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:49:06.916 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=8912 2024-04-06T09:49:06.916 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 30 = 29 -a 1 -le 8912 2024-04-06T09:49:06.916 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:49:06.916 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:49:06.916 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:49:07.118 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:49:07.118 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:49:07.118 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:49:07.118 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:49:07.118 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:49:07.118 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:49:07.118 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:49:07.118 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:49:07.118 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:49:07.118 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:49:07.119 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:49:07.119 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:49:07.119 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:49:07.119 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:49:07.119 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:49:07.119 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:49:07.119 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:49:07.122 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:49:07.122 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:49:07.122 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:49:07.122 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:49:07.122 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:49:07.123 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:49:07.123 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:49:07.123 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:49:07.123 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:49:07.123 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:49:07.123 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:49:07.123 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:49:07.123 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:49:07.123 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:49:07.123 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:49:07.123 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:49:07.123 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:49:07.124 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:49:07.124 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:49:07.124 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:49:07.124 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:49:07.124 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:49:07.124 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:49:07.124 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:49:07.124 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:49:07.124 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:49:07.124 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:49:07.124 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:49:07.124 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:49:07.124 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:49:07.124 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:49:07.124 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:49:07.148 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:49:07.148 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:49:07.148 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:49:07.148 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:49:07.253 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-06T09:49:07.253 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:rbd mirror flush : flush rbd mirror 2024-04-06T09:49:07.253 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:admin_socket: invalid command 2024-04-06T09:49:07.255 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:49:07.255 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/test 2024-04-06T09:49:07.255 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:1 2024-04-06T09:49:07.255 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:49:07.255 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:49:07.255 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-06T09:49:07.255 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:1 2024-04-06T09:49:07.255 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:49:07.255 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:49:07.255 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:49:07.256 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=1 2024-04-06T09:49:07.256 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-06T09:49:07.256 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:49:07.256 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:49:07.256 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=1 2024-04-06T09:49:07.256 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=1 2024-04-06T09:49:07.257 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-06T09:49:07.257 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:1 2024-04-06T09:49:07.258 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:49:07.258 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:49:07.258 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-06T09:49:07.258 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:1 2024-04-06T09:49:07.258 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:49:07.258 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:49:07.258 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:49:07.258 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=1 2024-04-06T09:49:07.258 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-06T09:49:07.258 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:49:07.258 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:49:07.258 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=1 2024-04-06T09:49:07.258 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=1 2024-04-06T09:49:07.258 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-06T09:49:07.282 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T09:49:07.282 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T09:49:07.282 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T09:49:07.282 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/test 2024-04-06T09:49:07.409 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:49:07.410 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:49:07.411 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:49:07.411 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:49:07.411 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:49:07.411 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:49:07.411 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:49:07.411 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:49:07.411 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:49:07.411 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:49:07.412 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:49:07.412 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:49:07.412 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:49:07.413 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:49:07.414 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:49:07.414 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:49:07.467 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 38 2024-04-06T09:49:07.468 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 38 2024-04-06T09:49:07.468 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:49:07.468 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=153, tag_tid=30, entry_tid=1], [object_number=152, tag_tid=30, entry_tid=0], [object_number=155, tag_tid=29, entry_tid=9059], [object_number=154, tag_tid=29, entry_tid=9058]]], state=connected] 2024-04-06T09:49:07.468 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=155, tag_tid=29, entry_tid=9019], [object_number=154, tag_tid=29, entry_tid=9018], [object_number=153, tag_tid=29, entry_tid=9017], [object_number=152, tag_tid=29, entry_tid=9016]]], state=connected] 2024-04-06T09:49:07.471 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:49:07.473 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=153, tag_tid=30, entry_tid=1' 2024-04-06T09:49:07.474 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:49:07.474 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:49:07.474 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:49:07.474 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:49:07.474 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:49:07.474 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:49:07.474 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:49:07.474 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:49:07.474 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:49:07.474 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:49:07.474 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:49:07.474 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:49:07.475 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:49:07.475 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:49:07.475 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:49:07.523 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 38 2024-04-06T09:49:07.523 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 38 2024-04-06T09:49:07.523 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:49:07.523 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=153, tag_tid=30, entry_tid=1], [object_number=152, tag_tid=30, entry_tid=0], [object_number=155, tag_tid=29, entry_tid=9059], [object_number=154, tag_tid=29, entry_tid=9058]]], state=connected] 2024-04-06T09:49:07.523 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=155, tag_tid=29, entry_tid=9019], [object_number=154, tag_tid=29, entry_tid=9018], [object_number=153, tag_tid=29, entry_tid=9017], [object_number=152, tag_tid=29, entry_tid=9016]]], state=connected] 2024-04-06T09:49:07.526 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:49:07.529 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=155, tag_tid=29, entry_tid=9019' 2024-04-06T09:49:07.529 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=153, tag_tid=30, entry_tid=1' -a 'object_number=153, tag_tid=30, entry_tid=1' = 'object_number=155, tag_tid=29, entry_tid=9019' 2024-04-06T09:49:07.529 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=155, tag_tid=29, entry_tid=9019' '!=' 'object_number=152, tag_tid=29, entry_tid=8912' 2024-04-06T09:49:07.529 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:49:07.529 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=155, tag_tid=29, entry_tid=9019' = 'object_number=152, tag_tid=29, entry_tid=8912' 2024-04-06T09:49:07.529 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=155, tag_tid=29, entry_tid=9019' 2024-04-06T09:49:07.529 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=153, tag_tid=30, entry_tid=1' 2024-04-06T09:49:07.529 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:49:07.529 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:49:07.530 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=30 2024-04-06T09:49:07.530 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=155, tag_tid=29, entry_tid=9019' 2024-04-06T09:49:07.531 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:49:07.531 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:49:07.532 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=29 2024-04-06T09:49:07.532 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=153, tag_tid=30, entry_tid=1' 2024-04-06T09:49:07.532 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:49:07.532 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:49:07.533 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:49:07.534 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=155, tag_tid=29, entry_tid=9019' 2024-04-06T09:49:07.534 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:49:07.534 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:49:07.535 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=9019 2024-04-06T09:49:07.535 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 30 = 29 -a 1 -le 9019 2024-04-06T09:49:07.535 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:49:07.535 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:49:07.536 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:49:07.737 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:49:07.738 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:49:07.738 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:49:07.738 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:49:07.738 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:49:07.738 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:49:07.738 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:49:07.738 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:49:07.738 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:49:07.738 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:49:07.738 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:49:07.738 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:49:07.739 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:49:07.739 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:49:07.739 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:49:07.739 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:49:07.739 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:49:07.742 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:49:07.742 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:49:07.742 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:49:07.743 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:49:07.743 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:49:07.743 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:49:07.743 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:49:07.743 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:49:07.743 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:49:07.743 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:49:07.743 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:49:07.743 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:49:07.743 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:49:07.743 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:49:07.743 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:49:07.743 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:49:07.744 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:49:07.744 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:49:07.744 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:49:07.744 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:49:07.745 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:49:07.745 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:49:07.745 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:49:07.745 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:49:07.745 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:49:07.745 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:49:07.745 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:49:07.745 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:49:07.745 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:49:07.745 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:49:07.745 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:49:07.745 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:49:07.770 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:49:07.770 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:49:07.770 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:49:07.771 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:49:07.872 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-06T09:49:07.873 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:rbd mirror flush : flush rbd mirror 2024-04-06T09:49:07.873 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:admin_socket: invalid command 2024-04-06T09:49:07.876 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:49:07.876 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/test 2024-04-06T09:49:07.876 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:1 2024-04-06T09:49:07.876 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:49:07.876 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:49:07.876 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-06T09:49:07.876 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:1 2024-04-06T09:49:07.876 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:49:07.877 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:49:07.877 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:49:07.877 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=1 2024-04-06T09:49:07.877 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-06T09:49:07.877 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:49:07.877 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:49:07.877 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=1 2024-04-06T09:49:07.877 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=1 2024-04-06T09:49:07.877 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-06T09:49:07.877 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:1 2024-04-06T09:49:07.877 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:49:07.877 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:49:07.877 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-06T09:49:07.877 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:1 2024-04-06T09:49:07.877 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:49:07.878 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:49:07.878 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:49:07.878 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=1 2024-04-06T09:49:07.878 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-06T09:49:07.878 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:49:07.878 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:49:07.878 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=1 2024-04-06T09:49:07.878 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=1 2024-04-06T09:49:07.878 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-06T09:49:07.901 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T09:49:07.901 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T09:49:07.901 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T09:49:07.901 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/test 2024-04-06T09:49:08.021 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:49:08.023 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:49:08.024 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:49:08.024 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:49:08.024 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:49:08.024 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:49:08.024 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:49:08.024 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:49:08.024 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:49:08.024 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:49:08.024 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:49:08.024 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:49:08.024 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:49:08.025 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:49:08.025 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:49:08.025 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:49:08.074 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 38 2024-04-06T09:49:08.074 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 38 2024-04-06T09:49:08.074 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:49:08.074 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=153, tag_tid=30, entry_tid=1], [object_number=152, tag_tid=30, entry_tid=0], [object_number=155, tag_tid=29, entry_tid=9059], [object_number=154, tag_tid=29, entry_tid=9058]]], state=connected] 2024-04-06T09:49:08.074 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=153, tag_tid=29, entry_tid=9061], [object_number=152, tag_tid=29, entry_tid=9060], [object_number=155, tag_tid=29, entry_tid=9059], [object_number=154, tag_tid=29, entry_tid=9058]]], state=connected] 2024-04-06T09:49:08.077 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:49:08.079 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=153, tag_tid=30, entry_tid=1' 2024-04-06T09:49:08.080 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:49:08.080 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:49:08.080 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:49:08.080 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:49:08.080 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:49:08.080 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:49:08.080 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:49:08.080 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:49:08.081 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:49:08.081 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:49:08.081 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:49:08.081 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:49:08.081 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:49:08.081 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:49:08.082 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:49:08.131 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 38 2024-04-06T09:49:08.131 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 38 2024-04-06T09:49:08.131 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:49:08.131 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=153, tag_tid=30, entry_tid=1], [object_number=152, tag_tid=30, entry_tid=0], [object_number=155, tag_tid=29, entry_tid=9059], [object_number=154, tag_tid=29, entry_tid=9058]]], state=connected] 2024-04-06T09:49:08.132 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=153, tag_tid=29, entry_tid=9061], [object_number=152, tag_tid=29, entry_tid=9060], [object_number=155, tag_tid=29, entry_tid=9059], [object_number=154, tag_tid=29, entry_tid=9058]]], state=connected] 2024-04-06T09:49:08.135 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:49:08.137 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=153, tag_tid=29, entry_tid=9061' 2024-04-06T09:49:08.137 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=153, tag_tid=30, entry_tid=1' -a 'object_number=153, tag_tid=30, entry_tid=1' = 'object_number=153, tag_tid=29, entry_tid=9061' 2024-04-06T09:49:08.137 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=153, tag_tid=29, entry_tid=9061' '!=' 'object_number=155, tag_tid=29, entry_tid=9019' 2024-04-06T09:49:08.137 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:49:08.137 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=153, tag_tid=29, entry_tid=9061' = 'object_number=155, tag_tid=29, entry_tid=9019' 2024-04-06T09:49:08.137 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=153, tag_tid=29, entry_tid=9061' 2024-04-06T09:49:08.137 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=153, tag_tid=30, entry_tid=1' 2024-04-06T09:49:08.138 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:49:08.138 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:49:08.138 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=30 2024-04-06T09:49:08.139 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=153, tag_tid=29, entry_tid=9061' 2024-04-06T09:49:08.139 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:49:08.139 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:49:08.140 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=29 2024-04-06T09:49:08.140 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=153, tag_tid=30, entry_tid=1' 2024-04-06T09:49:08.141 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:49:08.141 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:49:08.142 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:49:08.142 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=153, tag_tid=29, entry_tid=9061' 2024-04-06T09:49:08.142 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:49:08.142 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:49:08.144 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=9061 2024-04-06T09:49:08.144 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 30 = 29 -a 1 -le 9061 2024-04-06T09:49:08.144 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:49:08.144 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:49:08.144 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:49:08.346 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:49:08.346 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:49:08.346 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:49:08.346 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:49:08.346 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:49:08.346 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:49:08.346 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:49:08.346 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:49:08.347 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:49:08.347 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:49:08.347 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:49:08.347 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:49:08.347 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:49:08.347 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:49:08.347 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:49:08.347 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:49:08.349 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:49:08.349 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:49:08.349 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:49:08.349 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:49:08.349 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:49:08.349 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:49:08.349 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:49:08.349 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:49:08.349 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:49:08.349 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:49:08.349 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:49:08.350 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:49:08.350 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:49:08.350 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:49:08.350 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:49:08.350 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:49:08.350 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:49:08.350 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:49:08.350 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:49:08.350 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:49:08.350 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:49:08.350 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:49:08.350 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:49:08.350 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:49:08.350 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:49:08.351 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:49:08.351 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:49:08.351 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:49:08.351 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:49:08.351 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:49:08.351 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:49:08.351 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:49:08.351 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:49:08.375 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:49:08.375 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:49:08.376 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:49:08.376 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:49:08.481 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-06T09:49:08.481 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:rbd mirror flush : flush rbd mirror 2024-04-06T09:49:08.481 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:admin_socket: invalid command 2024-04-06T09:49:08.483 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:49:08.483 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/test 2024-04-06T09:49:08.483 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:1 2024-04-06T09:49:08.483 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:49:08.483 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:49:08.483 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-06T09:49:08.483 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:1 2024-04-06T09:49:08.483 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:49:08.483 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:49:08.483 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:49:08.483 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=1 2024-04-06T09:49:08.484 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-06T09:49:08.484 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:49:08.484 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:49:08.484 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=1 2024-04-06T09:49:08.484 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=1 2024-04-06T09:49:08.485 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-06T09:49:08.485 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:1 2024-04-06T09:49:08.485 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:49:08.485 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:49:08.485 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-06T09:49:08.485 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:1 2024-04-06T09:49:08.486 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:49:08.486 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:49:08.486 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:49:08.486 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=1 2024-04-06T09:49:08.486 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-06T09:49:08.486 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:49:08.486 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:49:08.486 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=1 2024-04-06T09:49:08.486 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=1 2024-04-06T09:49:08.486 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-06T09:49:08.509 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T09:49:08.510 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T09:49:08.510 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T09:49:08.510 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/test 2024-04-06T09:49:08.628 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:49:08.630 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:49:08.630 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:49:08.630 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:49:08.630 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:49:08.630 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:49:08.630 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:49:08.630 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:49:08.631 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:49:08.631 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:49:08.631 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:49:08.631 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:49:08.631 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:49:08.632 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:49:08.632 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:49:08.632 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:49:08.679 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 38 2024-04-06T09:49:08.679 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 38 2024-04-06T09:49:08.679 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:49:08.679 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=153, tag_tid=30, entry_tid=1], [object_number=152, tag_tid=30, entry_tid=0], [object_number=155, tag_tid=29, entry_tid=9059], [object_number=154, tag_tid=29, entry_tid=9058]]], state=connected] 2024-04-06T09:49:08.679 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=153, tag_tid=29, entry_tid=9061], [object_number=152, tag_tid=29, entry_tid=9060], [object_number=155, tag_tid=29, entry_tid=9059], [object_number=154, tag_tid=29, entry_tid=9058]]], state=connected] 2024-04-06T09:49:08.683 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:49:08.684 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=153, tag_tid=30, entry_tid=1' 2024-04-06T09:49:08.685 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:49:08.685 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:49:08.685 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:49:08.685 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:49:08.685 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:49:08.685 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:49:08.686 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:49:08.686 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:49:08.686 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:49:08.686 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:49:08.686 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:49:08.686 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:49:08.687 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:49:08.687 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:49:08.687 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:49:08.734 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 38 2024-04-06T09:49:08.734 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 38 2024-04-06T09:49:08.734 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:49:08.734 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=153, tag_tid=30, entry_tid=1], [object_number=152, tag_tid=30, entry_tid=0], [object_number=155, tag_tid=29, entry_tid=9059], [object_number=154, tag_tid=29, entry_tid=9058]]], state=connected] 2024-04-06T09:49:08.734 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=153, tag_tid=29, entry_tid=9061], [object_number=152, tag_tid=29, entry_tid=9060], [object_number=155, tag_tid=29, entry_tid=9059], [object_number=154, tag_tid=29, entry_tid=9058]]], state=connected] 2024-04-06T09:49:08.737 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:49:08.739 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=153, tag_tid=29, entry_tid=9061' 2024-04-06T09:49:08.739 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=153, tag_tid=30, entry_tid=1' -a 'object_number=153, tag_tid=30, entry_tid=1' = 'object_number=153, tag_tid=29, entry_tid=9061' 2024-04-06T09:49:08.739 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=153, tag_tid=29, entry_tid=9061' '!=' 'object_number=153, tag_tid=29, entry_tid=9061' 2024-04-06T09:49:08.739 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:49:08.739 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.4 2024-04-06T09:49:09.141 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:49:09.141 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:49:09.142 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:49:09.142 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:49:09.142 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:49:09.142 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:49:09.142 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:49:09.142 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:49:09.142 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:49:09.142 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:49:09.142 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:49:09.142 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:49:09.143 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:49:09.143 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:49:09.143 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:49:09.143 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:49:09.143 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:49:09.147 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:49:09.147 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:49:09.147 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:49:09.147 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:49:09.147 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:49:09.148 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:49:09.148 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:49:09.148 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:49:09.148 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:49:09.148 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:49:09.148 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:49:09.148 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:49:09.148 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:49:09.148 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:49:09.148 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:49:09.148 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:49:09.148 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:49:09.148 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:49:09.149 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:49:09.149 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:49:09.149 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:49:09.149 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:49:09.149 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:49:09.149 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:49:09.149 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:49:09.149 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:49:09.149 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:49:09.149 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:49:09.149 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:49:09.149 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:49:09.149 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:49:09.149 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:49:09.173 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:49:09.173 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:49:09.173 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:49:09.173 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:49:09.279 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-06T09:49:09.279 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:rbd mirror flush : flush rbd mirror 2024-04-06T09:49:09.279 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:admin_socket: invalid command 2024-04-06T09:49:09.282 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:49:09.282 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/test 2024-04-06T09:49:09.282 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:1 2024-04-06T09:49:09.283 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:49:09.283 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:49:09.283 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-06T09:49:09.283 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:1 2024-04-06T09:49:09.283 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:49:09.283 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:49:09.283 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:49:09.283 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=1 2024-04-06T09:49:09.283 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-06T09:49:09.284 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:49:09.284 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:49:09.284 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=1 2024-04-06T09:49:09.284 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=1 2024-04-06T09:49:09.284 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-06T09:49:09.284 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:1 2024-04-06T09:49:09.284 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:49:09.284 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:49:09.284 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-06T09:49:09.284 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:1 2024-04-06T09:49:09.284 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:49:09.284 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:49:09.285 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:49:09.285 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=1 2024-04-06T09:49:09.285 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-06T09:49:09.285 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:49:09.285 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:49:09.285 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=1 2024-04-06T09:49:09.285 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=1 2024-04-06T09:49:09.285 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-06T09:49:09.307 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T09:49:09.307 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T09:49:09.307 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T09:49:09.307 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/test 2024-04-06T09:49:09.416 INFO:tasks.rbd_mirror_thrash:kill cluster2.client.mirror.0 2024-04-06T09:49:09.417 INFO:tasks.rbd_mirror.cluster2.client.mirror.0:Sent signal 15 2024-04-06T09:49:09.417 INFO:tasks.rbd_mirror_thrash:waiting for 2 secs before reviving daemons 2024-04-06T09:49:09.417 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:49:09.418 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi192.stderr:2024-04-06T09:49:09.417+0000 7fcba503e640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.0 (PID: 77156) UID: 1000 2024-04-06T09:49:09.419 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:49:09.419 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:49:09.419 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:49:09.419 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:49:09.419 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:49:09.419 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:49:09.419 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:49:09.419 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:49:09.419 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:49:09.420 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:49:09.420 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:49:09.420 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:49:09.422 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:49:09.423 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:49:09.423 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:49:09.479 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 38 2024-04-06T09:49:09.479 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 38 2024-04-06T09:49:09.480 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:49:09.480 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=153, tag_tid=30, entry_tid=1], [object_number=152, tag_tid=30, entry_tid=0], [object_number=155, tag_tid=29, entry_tid=9059], [object_number=154, tag_tid=29, entry_tid=9058]]], state=connected] 2024-04-06T09:49:09.480 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=153, tag_tid=29, entry_tid=9061], [object_number=152, tag_tid=29, entry_tid=9060], [object_number=155, tag_tid=29, entry_tid=9059], [object_number=154, tag_tid=29, entry_tid=9058]]], state=connected] 2024-04-06T09:49:09.483 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:49:09.484 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=153, tag_tid=30, entry_tid=1' 2024-04-06T09:49:09.485 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:49:09.486 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:49:09.486 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:49:09.486 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:49:09.486 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:49:09.486 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:49:09.486 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:49:09.486 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:49:09.486 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:49:09.486 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:49:09.486 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:49:09.486 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:49:09.487 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:49:09.487 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:49:09.487 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:49:09.535 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 38 2024-04-06T09:49:09.536 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 38 2024-04-06T09:49:09.536 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:49:09.536 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=153, tag_tid=30, entry_tid=1], [object_number=152, tag_tid=30, entry_tid=0], [object_number=155, tag_tid=29, entry_tid=9059], [object_number=154, tag_tid=29, entry_tid=9058]]], state=connected] 2024-04-06T09:49:09.536 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=153, tag_tid=29, entry_tid=9061], [object_number=152, tag_tid=29, entry_tid=9060], [object_number=155, tag_tid=29, entry_tid=9059], [object_number=154, tag_tid=29, entry_tid=9058]]], state=connected] 2024-04-06T09:49:09.539 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:49:09.540 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=153, tag_tid=29, entry_tid=9061' 2024-04-06T09:49:09.540 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=153, tag_tid=30, entry_tid=1' -a 'object_number=153, tag_tid=30, entry_tid=1' = 'object_number=153, tag_tid=29, entry_tid=9061' 2024-04-06T09:49:09.541 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=153, tag_tid=29, entry_tid=9061' '!=' 'object_number=153, tag_tid=29, entry_tid=9061' 2024-04-06T09:49:09.541 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:49:09.541 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.8 2024-04-06T09:49:10.343 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:49:10.343 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:49:10.343 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:49:10.343 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:49:10.343 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:49:10.343 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:49:10.343 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:49:10.343 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:49:10.344 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:49:10.344 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:49:10.344 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:49:10.344 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:49:10.344 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:49:10.344 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:49:10.344 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:49:10.344 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:49:10.345 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:49:10.346 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:49:10.347 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:49:10.347 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:49:10.347 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:49:10.347 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:49:10.347 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:49:10.347 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:49:10.347 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:49:10.347 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:49:10.347 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:49:10.347 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:49:10.349 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:49:10.349 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:49:10.349 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:49:10.349 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:49:10.350 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:49:10.350 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:49:10.350 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:49:10.350 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:49:10.350 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:49:10.350 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:49:10.350 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:49:10.350 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:49:10.350 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:49:10.350 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:49:10.350 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:49:10.350 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:49:10.351 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:49:10.351 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:49:10.351 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:49:10.351 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:49:10.351 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:49:10.374 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:49:10.374 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:49:10.374 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:49:10.374 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:49:10.479 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-06T09:49:10.479 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:rbd mirror flush : flush rbd mirror 2024-04-06T09:49:10.479 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:admin_socket: invalid command 2024-04-06T09:49:10.481 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:49:10.481 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/test 2024-04-06T09:49:10.481 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:1 2024-04-06T09:49:10.481 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:49:10.482 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:49:10.482 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-06T09:49:10.482 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:1 2024-04-06T09:49:10.482 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:49:10.482 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:49:10.482 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:49:10.482 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=1 2024-04-06T09:49:10.482 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-06T09:49:10.482 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:49:10.482 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:49:10.482 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=1 2024-04-06T09:49:10.482 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=1 2024-04-06T09:49:10.484 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-06T09:49:10.484 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:1 2024-04-06T09:49:10.484 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:49:10.484 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:49:10.485 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-06T09:49:10.485 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:1 2024-04-06T09:49:10.485 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:49:10.485 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:49:10.485 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:49:10.485 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=1 2024-04-06T09:49:10.485 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-06T09:49:10.485 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:49:10.485 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:49:10.485 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=1 2024-04-06T09:49:10.485 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=1 2024-04-06T09:49:10.485 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-06T09:49:10.507 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T09:49:10.507 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T09:49:10.508 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T09:49:10.508 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/test 2024-04-06T09:49:10.626 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:49:10.627 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:49:10.627 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:49:10.627 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:49:10.627 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:49:10.627 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:49:10.627 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:49:10.627 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:49:10.627 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:49:10.627 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:49:10.627 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:49:10.627 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:49:10.627 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:49:10.629 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:49:10.629 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:49:10.629 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:49:10.677 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 38 2024-04-06T09:49:10.677 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 38 2024-04-06T09:49:10.677 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:49:10.677 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=153, tag_tid=30, entry_tid=1], [object_number=152, tag_tid=30, entry_tid=0], [object_number=155, tag_tid=29, entry_tid=9059], [object_number=154, tag_tid=29, entry_tid=9058]]], state=connected] 2024-04-06T09:49:10.677 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=153, tag_tid=29, entry_tid=9061], [object_number=152, tag_tid=29, entry_tid=9060], [object_number=155, tag_tid=29, entry_tid=9059], [object_number=154, tag_tid=29, entry_tid=9058]]], state=connected] 2024-04-06T09:49:10.680 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:49:10.682 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=153, tag_tid=30, entry_tid=1' 2024-04-06T09:49:10.683 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:49:10.683 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:49:10.684 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:49:10.684 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:49:10.684 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:49:10.684 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:49:10.684 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:49:10.684 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:49:10.684 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:49:10.684 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:49:10.684 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:49:10.684 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:49:10.685 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:49:10.685 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:49:10.685 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:49:10.733 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 38 2024-04-06T09:49:10.733 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 38 2024-04-06T09:49:10.733 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:49:10.733 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=153, tag_tid=30, entry_tid=1], [object_number=152, tag_tid=30, entry_tid=0], [object_number=155, tag_tid=29, entry_tid=9059], [object_number=154, tag_tid=29, entry_tid=9058]]], state=connected] 2024-04-06T09:49:10.733 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=153, tag_tid=29, entry_tid=9061], [object_number=152, tag_tid=29, entry_tid=9060], [object_number=155, tag_tid=29, entry_tid=9059], [object_number=154, tag_tid=29, entry_tid=9058]]], state=connected] 2024-04-06T09:49:10.736 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:49:10.738 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=153, tag_tid=29, entry_tid=9061' 2024-04-06T09:49:10.738 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=153, tag_tid=30, entry_tid=1' -a 'object_number=153, tag_tid=30, entry_tid=1' = 'object_number=153, tag_tid=29, entry_tid=9061' 2024-04-06T09:49:10.738 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=153, tag_tid=29, entry_tid=9061' '!=' 'object_number=153, tag_tid=29, entry_tid=9061' 2024-04-06T09:49:10.738 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:49:10.738 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 1.6 2024-04-06T09:49:11.417 INFO:tasks.rbd_mirror_thrash:waiting for cluster2.client.mirror.0 2024-04-06T09:49:11.417 INFO:teuthology.orchestra.run:waiting for 600 2024-04-06T09:49:11.418 INFO:tasks.rbd_mirror_thrash:reviving cluster2.client.mirror.0 2024-04-06T09:49:11.418 INFO:tasks.rbd_mirror.cluster2.client.mirror.0:Restarting daemon 2024-04-06T09:49:11.418 DEBUG:teuthology.orchestra.run.smithi192:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.0 2024-04-06T09:49:11.421 INFO:tasks.rbd_mirror.cluster2.client.mirror.0:Started 2024-04-06T09:49:11.421 INFO:tasks.rbd_mirror_thrash:waiting for 92 secs before thrashing 2024-04-06T09:49:12.339 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:49:12.340 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:49:12.340 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:49:12.340 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:49:12.340 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:49:12.340 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:49:12.340 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:49:12.340 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:49:12.340 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:49:12.340 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:49:12.340 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:49:12.340 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:49:12.340 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:49:12.340 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:49:12.340 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:49:12.341 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:49:12.342 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:49:12.344 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:49:12.344 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:49:12.344 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:49:12.344 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:49:12.344 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:49:12.344 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:49:12.344 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:49:12.344 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:49:12.344 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:49:12.344 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:49:12.345 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:49:12.345 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:49:12.345 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:49:12.345 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:49:12.345 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:49:12.345 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:49:12.345 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:49:12.345 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:49:12.345 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:49:12.345 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:49:12.345 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:49:12.345 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:49:12.345 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:49:12.345 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:49:12.345 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:49:12.346 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:49:12.346 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:49:12.346 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:49:12.346 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:49:12.346 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:49:12.346 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:49:12.346 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:49:12.372 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:49:12.372 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:49:12.372 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:49:12.372 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:49:12.478 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-06T09:49:12.479 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:rbd mirror flush : flush rbd mirror 2024-04-06T09:49:12.479 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:admin_socket: invalid command 2024-04-06T09:49:12.481 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:49:12.481 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/test 2024-04-06T09:49:12.481 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:1 2024-04-06T09:49:12.481 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:49:12.481 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:49:12.481 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-06T09:49:12.481 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:1 2024-04-06T09:49:12.481 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:49:12.481 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:49:12.481 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:49:12.481 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=1 2024-04-06T09:49:12.482 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-06T09:49:12.482 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:49:12.482 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:49:12.482 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=1 2024-04-06T09:49:12.482 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=1 2024-04-06T09:49:12.483 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-06T09:49:12.483 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:1 2024-04-06T09:49:12.483 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:49:12.483 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:49:12.483 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-06T09:49:12.483 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:1 2024-04-06T09:49:12.484 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:49:12.484 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:49:12.484 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:49:12.484 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=1 2024-04-06T09:49:12.484 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-06T09:49:12.484 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:49:12.484 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:49:12.484 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=1 2024-04-06T09:49:12.484 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=1 2024-04-06T09:49:12.484 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-06T09:49:12.508 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T09:49:12.508 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T09:49:12.508 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T09:49:12.508 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/test 2024-04-06T09:49:12.620 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:49:12.621 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:49:12.621 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:49:12.621 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:49:12.621 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:49:12.621 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:49:12.621 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:49:12.621 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:49:12.621 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:49:12.621 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:49:12.622 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:49:12.622 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:49:12.622 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:49:12.624 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:49:12.624 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:49:12.624 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:49:12.675 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 38 2024-04-06T09:49:12.675 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 38 2024-04-06T09:49:12.675 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:49:12.675 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=153, tag_tid=30, entry_tid=1], [object_number=152, tag_tid=30, entry_tid=0], [object_number=155, tag_tid=29, entry_tid=9059], [object_number=154, tag_tid=29, entry_tid=9058]]], state=connected] 2024-04-06T09:49:12.675 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=153, tag_tid=29, entry_tid=9061], [object_number=152, tag_tid=29, entry_tid=9060], [object_number=155, tag_tid=29, entry_tid=9059], [object_number=154, tag_tid=29, entry_tid=9058]]], state=connected] 2024-04-06T09:49:12.678 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:49:12.680 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=153, tag_tid=30, entry_tid=1' 2024-04-06T09:49:12.681 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:49:12.681 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:49:12.681 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:49:12.681 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:49:12.681 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:49:12.681 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:49:12.681 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:49:12.681 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:49:12.681 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:49:12.682 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:49:12.682 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:49:12.682 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:49:12.682 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:49:12.682 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:49:12.682 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:49:12.730 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 38 2024-04-06T09:49:12.731 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 38 2024-04-06T09:49:12.731 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:49:12.731 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=153, tag_tid=30, entry_tid=1], [object_number=152, tag_tid=30, entry_tid=0], [object_number=155, tag_tid=29, entry_tid=9059], [object_number=154, tag_tid=29, entry_tid=9058]]], state=connected] 2024-04-06T09:49:12.731 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=153, tag_tid=29, entry_tid=9061], [object_number=152, tag_tid=29, entry_tid=9060], [object_number=155, tag_tid=29, entry_tid=9059], [object_number=154, tag_tid=29, entry_tid=9058]]], state=connected] 2024-04-06T09:49:12.734 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:49:12.736 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=153, tag_tid=29, entry_tid=9061' 2024-04-06T09:49:12.736 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=153, tag_tid=30, entry_tid=1' -a 'object_number=153, tag_tid=30, entry_tid=1' = 'object_number=153, tag_tid=29, entry_tid=9061' 2024-04-06T09:49:12.736 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=153, tag_tid=29, entry_tid=9061' '!=' 'object_number=153, tag_tid=29, entry_tid=9061' 2024-04-06T09:49:12.736 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:49:12.736 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 2 2024-04-06T09:49:14.738 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:49:14.738 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:49:14.738 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:49:14.738 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:49:14.738 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:49:14.738 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:49:14.738 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:49:14.738 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:49:14.738 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:49:14.738 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:49:14.738 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:49:14.739 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:49:14.739 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:49:14.739 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:49:14.739 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:49:14.739 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:49:14.739 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:49:14.740 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:49:14.741 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:49:14.741 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:49:14.741 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:49:14.741 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:49:14.741 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:49:14.741 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:49:14.741 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:49:14.741 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:49:14.741 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:49:14.741 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:49:14.741 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:49:14.741 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:49:14.741 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:49:14.741 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:49:14.742 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:49:14.743 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:49:14.743 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:49:14.743 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:49:14.743 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:49:14.743 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:49:14.743 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:49:14.743 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:49:14.743 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:49:14.743 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:49:14.743 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:49:14.743 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:49:14.743 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:49:14.743 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:49:14.744 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:49:14.744 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:49:14.744 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:49:14.767 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:49:14.768 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:49:14.768 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:49:14.768 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:49:14.873 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-06T09:49:14.873 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:rbd mirror flush : flush rbd mirror 2024-04-06T09:49:14.873 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:admin_socket: invalid command 2024-04-06T09:49:14.876 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:49:14.876 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/test 2024-04-06T09:49:14.876 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:1 2024-04-06T09:49:14.876 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:49:14.876 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:49:14.876 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-06T09:49:14.876 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:1 2024-04-06T09:49:14.876 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:49:14.876 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:49:14.876 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:49:14.876 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=1 2024-04-06T09:49:14.876 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-06T09:49:14.877 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:49:14.877 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:49:14.877 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=1 2024-04-06T09:49:14.877 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=1 2024-04-06T09:49:14.878 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-06T09:49:14.878 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:1 2024-04-06T09:49:14.878 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:49:14.878 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:49:14.878 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-06T09:49:14.878 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:1 2024-04-06T09:49:14.878 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:49:14.879 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:49:14.879 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:49:14.879 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=1 2024-04-06T09:49:14.879 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-06T09:49:14.879 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:49:14.879 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:49:14.879 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=1 2024-04-06T09:49:14.879 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=1 2024-04-06T09:49:14.879 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-06T09:49:14.901 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T09:49:14.901 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T09:49:14.901 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T09:49:14.901 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/test 2024-04-06T09:49:15.034 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:49:15.035 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:49:15.035 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:49:15.035 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:49:15.035 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:49:15.035 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:49:15.035 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:49:15.035 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:49:15.036 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:49:15.036 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:49:15.036 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:49:15.036 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:49:15.036 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:49:15.039 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:49:15.039 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:49:15.039 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:49:15.092 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 38 2024-04-06T09:49:15.092 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 38 2024-04-06T09:49:15.092 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:49:15.092 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=153, tag_tid=30, entry_tid=1], [object_number=152, tag_tid=30, entry_tid=0], [object_number=155, tag_tid=29, entry_tid=9059], [object_number=154, tag_tid=29, entry_tid=9058]]], state=connected] 2024-04-06T09:49:15.092 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=153, tag_tid=30, entry_tid=1], [object_number=152, tag_tid=30, entry_tid=0], [object_number=155, tag_tid=29, entry_tid=9059], [object_number=154, tag_tid=29, entry_tid=9058]]], state=connected] 2024-04-06T09:49:15.095 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:49:15.097 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=153, tag_tid=30, entry_tid=1' 2024-04-06T09:49:15.098 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:49:15.098 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:49:15.098 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:49:15.098 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:49:15.098 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:49:15.098 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:49:15.099 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:49:15.099 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:49:15.099 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:49:15.099 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:49:15.099 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:49:15.099 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:49:15.099 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:49:15.100 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:49:15.100 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:49:15.147 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 38 2024-04-06T09:49:15.148 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 38 2024-04-06T09:49:15.148 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:49:15.148 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=153, tag_tid=30, entry_tid=1], [object_number=152, tag_tid=30, entry_tid=0], [object_number=155, tag_tid=29, entry_tid=9059], [object_number=154, tag_tid=29, entry_tid=9058]]], state=connected] 2024-04-06T09:49:15.148 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=153, tag_tid=30, entry_tid=1], [object_number=152, tag_tid=30, entry_tid=0], [object_number=155, tag_tid=29, entry_tid=9059], [object_number=154, tag_tid=29, entry_tid=9058]]], state=connected] 2024-04-06T09:49:15.151 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:49:15.154 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=153, tag_tid=30, entry_tid=1' 2024-04-06T09:49:15.154 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=153, tag_tid=30, entry_tid=1' -a 'object_number=153, tag_tid=30, entry_tid=1' = 'object_number=153, tag_tid=30, entry_tid=1' 2024-04-06T09:49:15.154 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:49:15.154 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_snap_present cluster1 mirror test snap10 2024-04-06T09:49:15.154 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:49:15.154 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:49:15.154 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:49:15.155 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local snap_name=snap10 2024-04-06T09:49:15.155 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local s 2024-04-06T09:49:15.155 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-04-06T09:49:15.155 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 1 2024-04-06T09:49:16.154 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 info mirror/test@snap10 2024-04-06T09:49:16.237 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:rbd image 'test': 2024-04-06T09:49:16.237 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: size 512 MiB in 128 objects 2024-04-06T09:49:16.237 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: order 22 (4 MiB objects) 2024-04-06T09:49:16.237 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: snapshot_count: 19 2024-04-06T09:49:16.237 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: id: 10a56da0b207 2024-04-06T09:49:16.237 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: block_name_prefix: rbd_data.10a56da0b207 2024-04-06T09:49:16.238 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: format: 2 2024-04-06T09:49:16.238 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: features: layering, exclusive-lock, journaling 2024-04-06T09:49:16.238 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: op_features: 2024-04-06T09:49:16.238 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: flags: 2024-04-06T09:49:16.238 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: create_timestamp: Sat Apr 6 09:31:59 2024 2024-04-06T09:49:16.238 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: access_timestamp: Sat Apr 6 09:48:01 2024 2024-04-06T09:49:16.238 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: modify_timestamp: Sat Apr 6 09:48:17 2024 2024-04-06T09:49:16.238 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: protected: False 2024-04-06T09:49:16.238 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: journal: 10a56da0b207 2024-04-06T09:49:16.238 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: mirroring state: enabled 2024-04-06T09:49:16.238 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: mirroring mode: journal 2024-04-06T09:49:16.238 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: mirroring global id: 8ad5dedc-b3e0-425d-8993-b3e2231fc5ca 2024-04-06T09:49:16.238 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: mirroring primary: false 2024-04-06T09:49:16.243 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:49:16.243 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n snap9-clean ']' 2024-04-06T09:49:16.244 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ compare_image_snaps mirror test snap9-clean 2024-04-06T09:49:16.244 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:49:16.244 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:49:16.244 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local snap_name=snap9-clean 2024-04-06T09:49:16.244 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local ret=0 2024-04-06T09:49:16.244 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local rmt_export=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.export 2024-04-06T09:49:16.244 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loc_export=/tmp/tmp.BhW6xKmROU/cluster1-mirror-test.export 2024-04-06T09:49:16.244 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rm -f /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.export /tmp/tmp.BhW6xKmROU/cluster1-mirror-test.export 2024-04-06T09:49:16.244 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster2 -p mirror export test@snap9-clean /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.export 2024-04-06T09:49:16.918 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-04-06T09:49:16.938 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 -p mirror export test@snap9-clean /tmp/tmp.BhW6xKmROU/cluster1-mirror-test.export 2024-04-06T09:49:18.493 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-04-06T09:49:18.518 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmp /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.export /tmp/tmp.BhW6xKmROU/cluster1-mirror-test.export 2024-04-06T09:49:18.800 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rm -f /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.export /tmp/tmp.BhW6xKmROU/cluster1-mirror-test.export 2024-04-06T09:49:18.936 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:49:18.936 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ compare_image_snaps mirror test snap10 2024-04-06T09:49:18.936 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:49:18.936 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:49:18.936 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local snap_name=snap10 2024-04-06T09:49:18.937 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local ret=0 2024-04-06T09:49:18.937 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local rmt_export=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.export 2024-04-06T09:49:18.937 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loc_export=/tmp/tmp.BhW6xKmROU/cluster1-mirror-test.export 2024-04-06T09:49:18.937 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rm -f /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.export /tmp/tmp.BhW6xKmROU/cluster1-mirror-test.export 2024-04-06T09:49:18.937 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster2 -p mirror export test@snap10 /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.export 2024-04-06T09:49:19.578 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-04-06T09:49:19.590 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 -p mirror export test@snap10 /tmp/tmp.BhW6xKmROU/cluster1-mirror-test.export 2024-04-06T09:49:21.186 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-04-06T09:49:21.203 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmp /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.export /tmp/tmp.BhW6xKmROU/cluster1-mirror-test.export 2024-04-06T09:49:21.484 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rm -f /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.export /tmp/tmp.BhW6xKmROU/cluster1-mirror-test.export 2024-04-06T09:49:21.621 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:49:21.621 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ clean_snap_name=snap10-clean 2024-04-06T09:49:21.621 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ create_snap cluster2 mirror test snap10-clean 2024-04-06T09:49:21.621 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-04-06T09:49:21.621 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:49:21.622 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:49:21.622 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local snap_name=snap10-clean 2024-04-06T09:49:21.622 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap10-clean --debug-rbd=20 --debug-journaler=20 2024-04-06T09:49:22.734 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_image_replay_started cluster1 mirror test 2024-04-06T09:49:22.734 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:49:22.734 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:49:22.734 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:49:22.734 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_image_replay_state cluster1 mirror test started 2024-04-06T09:49:22.734 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:49:22.734 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:49:22.734 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:49:22.734 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state=started 2024-04-06T09:49:22.734 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local s 2024-04-06T09:49:22.734 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:49:22.734 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 1 2024-04-06T09:49:23.735 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test_image_replay_state cluster1 mirror test started 2024-04-06T09:49:23.735 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:49:23.735 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:49:23.736 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:49:23.736 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local test_state=started 2024-04-06T09:49:23.736 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local status_result 2024-04-06T09:49:23.736 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local current_state=stopped 2024-04-06T09:49:23.737 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2024-04-06T09:49:23.737 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -i state 2024-04-06T09:49:23.737 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_instance=cluster1 2024-04-06T09:49:23.737 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ shift 2024-04-06T09:49:23.737 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:49:23.737 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance=cluster1 2024-04-06T09:49:23.737 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local loop_instance 2024-04-06T09:49:23.737 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:49:23.737 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sleep 0 2024-04-06T09:49:23.738 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:49:23.739 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ seq 0 3 2024-04-06T09:49:23.743 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:49:23.743 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2024-04-06T09:49:23.743 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1:0 2024-04-06T09:49:23.743 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ shift 2024-04-06T09:49:23.744 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:49:23.744 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:49:23.744 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:49:23.744 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=cluster 2024-04-06T09:49:23.744 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:49:23.744 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:49:23.744 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:49:23.744 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:49:23.744 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval cluster=cluster1 2024-04-06T09:49:23.744 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ cluster=cluster1 2024-04-06T09:49:23.744 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:49:23.744 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:49:23.745 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:49:23.745 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local local_cluster=cluster1:0 2024-04-06T09:49:23.745 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local cluster=cluster1 2024-04-06T09:49:23.745 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local instance 2024-04-06T09:49:23.745 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:49:23.745 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local val=cluster1:0 2024-04-06T09:49:23.745 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local cluster_var_name=local_cluster 2024-04-06T09:49:23.745 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local instance_var_name=instance 2024-04-06T09:49:23.745 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ cluster=cluster1 2024-04-06T09:49:23.745 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:49:23.745 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ '[' 0 = cluster1:0 ']' 2024-04-06T09:49:23.746 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ eval local_cluster=cluster1 2024-04-06T09:49:23.746 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++++ local_cluster=cluster1 2024-04-06T09:49:23.746 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ eval instance=0 2024-04-06T09:49:23.746 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++++ instance=0 2024-04-06T09:49:23.746 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:49:23.764 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:49:23.764 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:49:23.764 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:49:23.764 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2024-04-06T09:49:23.866 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-06T09:49:23.867 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:rbd mirror status : get status for rbd mirror 2024-04-06T09:49:23.867 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:admin_socket: invalid command 2024-04-06T09:49:23.868 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:49:23.868 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/test 2024-04-06T09:49:23.869 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1:1 2024-04-06T09:49:23.869 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ shift 2024-04-06T09:49:23.869 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:49:23.869 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-04-06T09:49:23.869 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:1 2024-04-06T09:49:23.869 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=cluster 2024-04-06T09:49:23.869 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:49:23.869 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:49:23.869 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=1 2024-04-06T09:49:23.869 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-06T09:49:23.869 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval cluster=cluster1 2024-04-06T09:49:23.870 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ cluster=cluster1 2024-04-06T09:49:23.870 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=1 2024-04-06T09:49:23.870 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=1 2024-04-06T09:49:23.871 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-04-06T09:49:23.871 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local local_cluster=cluster1:1 2024-04-06T09:49:23.871 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local cluster=cluster1 2024-04-06T09:49:23.871 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local instance 2024-04-06T09:49:23.872 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-06T09:49:23.872 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local val=cluster1:1 2024-04-06T09:49:23.872 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local cluster_var_name=local_cluster 2024-04-06T09:49:23.872 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local instance_var_name=instance 2024-04-06T09:49:23.872 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ cluster=cluster1 2024-04-06T09:49:23.872 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=1 2024-04-06T09:49:23.872 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ '[' 1 = cluster1:1 ']' 2024-04-06T09:49:23.872 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ eval local_cluster=cluster1 2024-04-06T09:49:23.872 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++++ local_cluster=cluster1 2024-04-06T09:49:23.872 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ eval instance=1 2024-04-06T09:49:23.872 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++++ instance=1 2024-04-06T09:49:23.872 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-06T09:49:23.894 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T09:49:23.894 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T09:49:23.894 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T09:49:23.894 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/test 2024-04-06T09:49:24.014 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ return 0 2024-04-06T09:49:24.015 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ status_result=' "state": "Replaying"' 2024-04-06T09:49:24.016 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ echo ' "state": "Replaying"' 2024-04-06T09:49:24.016 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep -i Replaying 2024-04-06T09:49:24.017 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: "state": "Replaying" 2024-04-06T09:49:24.020 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ current_state=started 2024-04-06T09:49:24.020 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test started = started 2024-04-06T09:49:24.020 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:49:24.020 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror test 2024-04-06T09:49:24.020 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local local_cluster=cluster1 2024-04-06T09:49:24.020 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-04-06T09:49:24.020 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:49:24.020 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:49:24.020 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' journal = journal ']' 2024-04-06T09:49:24.020 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_journal_replay_complete cluster1 cluster2 mirror test 2024-04-06T09:49:24.020 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local local_cluster=cluster1 2024-04-06T09:49:24.020 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-04-06T09:49:24.021 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:49:24.021 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:49:24.021 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local s master_pos mirror_pos last_mirror_pos 2024-04-06T09:49:24.021 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local master_tag master_entry mirror_tag mirror_entry 2024-04-06T09:49:24.021 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:49:24.021 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:49:24.021 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:49:24.221 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:49:24.221 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:49:24.221 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:49:24.221 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:49:24.221 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:49:24.221 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:49:24.221 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:49:24.221 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:49:24.221 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:49:24.222 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:49:24.222 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:49:24.222 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:49:24.222 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:49:24.222 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:49:24.222 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:49:24.223 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:49:24.223 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:49:24.226 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:49:24.226 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:49:24.226 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:49:24.226 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:49:24.226 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:49:24.226 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:49:24.226 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:49:24.226 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:49:24.226 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:49:24.226 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:49:24.227 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:49:24.227 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:49:24.227 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:49:24.227 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:49:24.227 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:49:24.227 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:49:24.227 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:49:24.227 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:49:24.227 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:49:24.227 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:49:24.227 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:49:24.227 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:49:24.227 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:49:24.228 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:49:24.228 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:49:24.228 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:49:24.228 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:49:24.228 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:49:24.228 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:49:24.228 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:49:24.228 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:49:24.228 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:49:24.249 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:49:24.250 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:49:24.250 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:49:24.250 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:49:24.360 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-06T09:49:24.360 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:rbd mirror flush : flush rbd mirror 2024-04-06T09:49:24.360 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:admin_socket: invalid command 2024-04-06T09:49:24.362 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:49:24.362 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/test 2024-04-06T09:49:24.362 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:1 2024-04-06T09:49:24.362 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:49:24.362 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:49:24.362 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-06T09:49:24.363 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:1 2024-04-06T09:49:24.363 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:49:24.363 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:49:24.363 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:49:24.363 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=1 2024-04-06T09:49:24.363 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-06T09:49:24.363 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:49:24.363 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:49:24.363 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=1 2024-04-06T09:49:24.363 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=1 2024-04-06T09:49:24.364 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-06T09:49:24.364 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:1 2024-04-06T09:49:24.364 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:49:24.364 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:49:24.365 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-06T09:49:24.365 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:1 2024-04-06T09:49:24.365 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:49:24.365 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:49:24.365 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:49:24.365 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=1 2024-04-06T09:49:24.365 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-06T09:49:24.365 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:49:24.365 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:49:24.365 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=1 2024-04-06T09:49:24.365 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=1 2024-04-06T09:49:24.365 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-06T09:49:24.383 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T09:49:24.383 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T09:49:24.383 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T09:49:24.383 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/test 2024-04-06T09:49:24.502 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:49:24.503 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:49:24.504 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:49:24.504 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:49:24.504 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:49:24.504 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:49:24.504 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:49:24.504 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:49:24.504 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:49:24.504 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:49:24.504 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:49:24.504 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:49:24.505 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:49:24.506 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:49:24.506 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:49:24.506 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:49:24.554 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 38 2024-04-06T09:49:24.554 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 38 2024-04-06T09:49:24.554 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:49:24.554 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=153, tag_tid=31, entry_tid=1], [object_number=152, tag_tid=31, entry_tid=0], [object_number=155, tag_tid=29, entry_tid=9059], [object_number=154, tag_tid=29, entry_tid=9058]]], state=connected] 2024-04-06T09:49:24.554 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=153, tag_tid=30, entry_tid=1], [object_number=152, tag_tid=30, entry_tid=0], [object_number=155, tag_tid=29, entry_tid=9059], [object_number=154, tag_tid=29, entry_tid=9058]]], state=connected] 2024-04-06T09:49:24.558 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:49:24.559 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=153, tag_tid=31, entry_tid=1' 2024-04-06T09:49:24.560 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:49:24.560 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:49:24.560 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:49:24.560 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:49:24.560 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:49:24.560 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:49:24.560 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:49:24.560 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:49:24.561 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:49:24.561 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:49:24.561 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:49:24.561 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:49:24.561 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:49:24.561 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:49:24.562 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:49:24.608 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 38 2024-04-06T09:49:24.608 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 38 2024-04-06T09:49:24.609 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:49:24.609 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=153, tag_tid=31, entry_tid=1], [object_number=152, tag_tid=31, entry_tid=0], [object_number=155, tag_tid=29, entry_tid=9059], [object_number=154, tag_tid=29, entry_tid=9058]]], state=connected] 2024-04-06T09:49:24.609 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=153, tag_tid=30, entry_tid=1], [object_number=152, tag_tid=30, entry_tid=0], [object_number=155, tag_tid=29, entry_tid=9059], [object_number=154, tag_tid=29, entry_tid=9058]]], state=connected] 2024-04-06T09:49:24.614 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:49:24.614 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=153, tag_tid=30, entry_tid=1' 2024-04-06T09:49:24.614 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=153, tag_tid=31, entry_tid=1' -a 'object_number=153, tag_tid=31, entry_tid=1' = 'object_number=153, tag_tid=30, entry_tid=1' 2024-04-06T09:49:24.614 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=153, tag_tid=30, entry_tid=1' '!=' '' 2024-04-06T09:49:24.614 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:49:24.614 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=153, tag_tid=30, entry_tid=1' = '' 2024-04-06T09:49:24.614 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=153, tag_tid=30, entry_tid=1' 2024-04-06T09:49:24.615 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=153, tag_tid=31, entry_tid=1' 2024-04-06T09:49:24.615 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:49:24.615 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:49:24.616 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=31 2024-04-06T09:49:24.617 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=153, tag_tid=30, entry_tid=1' 2024-04-06T09:49:24.617 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:49:24.617 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:49:24.617 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=30 2024-04-06T09:49:24.618 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=153, tag_tid=31, entry_tid=1' 2024-04-06T09:49:24.618 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:49:24.618 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:49:24.619 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:49:24.620 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=153, tag_tid=30, entry_tid=1' 2024-04-06T09:49:24.620 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:49:24.620 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:49:24.621 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=1 2024-04-06T09:49:24.621 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 31 = 30 -a 1 -le 1 2024-04-06T09:49:24.621 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:49:24.621 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:49:24.622 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:49:24.823 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:49:24.823 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:49:24.824 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:49:24.824 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:49:24.824 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:49:24.824 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:49:24.824 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:49:24.824 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:49:24.824 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:49:24.824 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:49:24.824 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:49:24.824 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:49:24.824 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:49:24.824 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:49:24.824 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:49:24.825 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:49:24.825 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:49:24.827 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:49:24.828 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:49:24.828 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:49:24.828 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:49:24.828 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:49:24.828 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:49:24.828 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:49:24.828 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:49:24.828 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:49:24.828 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:49:24.828 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:49:24.828 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:49:24.828 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:49:24.828 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:49:24.828 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:49:24.829 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:49:24.829 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:49:24.829 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:49:24.830 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:49:24.830 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:49:24.830 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:49:24.830 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:49:24.830 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:49:24.830 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:49:24.830 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:49:24.830 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:49:24.830 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:49:24.830 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:49:24.830 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:49:24.830 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:49:24.830 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:49:24.831 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:49:24.855 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:49:24.855 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:49:24.855 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:49:24.855 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:49:24.960 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-06T09:49:24.960 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:rbd mirror flush : flush rbd mirror 2024-04-06T09:49:24.960 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:admin_socket: invalid command 2024-04-06T09:49:24.963 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:49:24.963 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/test 2024-04-06T09:49:24.963 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:1 2024-04-06T09:49:24.963 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:49:24.963 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:49:24.963 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-06T09:49:24.963 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:1 2024-04-06T09:49:24.963 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:49:24.963 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:49:24.963 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:49:24.963 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=1 2024-04-06T09:49:24.964 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-06T09:49:24.964 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:49:24.964 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:49:24.964 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=1 2024-04-06T09:49:24.964 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=1 2024-04-06T09:49:24.965 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-06T09:49:24.965 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:1 2024-04-06T09:49:24.965 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:49:24.965 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:49:24.965 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-06T09:49:24.965 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:1 2024-04-06T09:49:24.965 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:49:24.966 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:49:24.966 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:49:24.966 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=1 2024-04-06T09:49:24.966 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-06T09:49:24.966 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:49:24.966 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:49:24.966 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=1 2024-04-06T09:49:24.966 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=1 2024-04-06T09:49:24.966 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-06T09:49:24.990 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T09:49:24.991 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T09:49:24.991 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T09:49:24.991 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/test 2024-04-06T09:49:25.108 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:49:25.109 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:49:25.109 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:49:25.109 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:49:25.109 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:49:25.109 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:49:25.109 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:49:25.109 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:49:25.109 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:49:25.110 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:49:25.110 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:49:25.110 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:49:25.110 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:49:25.111 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:49:25.112 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:49:25.112 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:49:25.159 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 38 2024-04-06T09:49:25.160 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 38 2024-04-06T09:49:25.160 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:49:25.160 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=153, tag_tid=31, entry_tid=1], [object_number=152, tag_tid=31, entry_tid=0], [object_number=155, tag_tid=29, entry_tid=9059], [object_number=154, tag_tid=29, entry_tid=9058]]], state=connected] 2024-04-06T09:49:25.160 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=153, tag_tid=30, entry_tid=1], [object_number=152, tag_tid=30, entry_tid=0], [object_number=155, tag_tid=29, entry_tid=9059], [object_number=154, tag_tid=29, entry_tid=9058]]], state=connected] 2024-04-06T09:49:25.163 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:49:25.165 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=153, tag_tid=31, entry_tid=1' 2024-04-06T09:49:25.166 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:49:25.166 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:49:25.166 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:49:25.166 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:49:25.166 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:49:25.166 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:49:25.166 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:49:25.166 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:49:25.166 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:49:25.166 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:49:25.167 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:49:25.167 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:49:25.167 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:49:25.167 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:49:25.167 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:49:25.215 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 38 2024-04-06T09:49:25.215 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 38 2024-04-06T09:49:25.215 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:49:25.215 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=153, tag_tid=31, entry_tid=1], [object_number=152, tag_tid=31, entry_tid=0], [object_number=155, tag_tid=29, entry_tid=9059], [object_number=154, tag_tid=29, entry_tid=9058]]], state=connected] 2024-04-06T09:49:25.215 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=153, tag_tid=30, entry_tid=1], [object_number=152, tag_tid=30, entry_tid=0], [object_number=155, tag_tid=29, entry_tid=9059], [object_number=154, tag_tid=29, entry_tid=9058]]], state=connected] 2024-04-06T09:49:25.218 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:49:25.220 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=153, tag_tid=30, entry_tid=1' 2024-04-06T09:49:25.220 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=153, tag_tid=31, entry_tid=1' -a 'object_number=153, tag_tid=31, entry_tid=1' = 'object_number=153, tag_tid=30, entry_tid=1' 2024-04-06T09:49:25.220 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=153, tag_tid=30, entry_tid=1' '!=' 'object_number=153, tag_tid=30, entry_tid=1' 2024-04-06T09:49:25.220 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:49:25.220 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.4 2024-04-06T09:49:25.622 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:49:25.622 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:49:25.622 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:49:25.622 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:49:25.623 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:49:25.623 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:49:25.623 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:49:25.623 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:49:25.623 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:49:25.623 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:49:25.623 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:49:25.623 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:49:25.623 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:49:25.623 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:49:25.623 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:49:25.624 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:49:25.624 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:49:25.626 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:49:25.626 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:49:25.627 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:49:25.627 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:49:25.627 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:49:25.627 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:49:25.627 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:49:25.627 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:49:25.627 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:49:25.627 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:49:25.627 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:49:25.627 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:49:25.627 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:49:25.627 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:49:25.627 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:49:25.627 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:49:25.629 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:49:25.629 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:49:25.629 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:49:25.629 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:49:25.629 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:49:25.629 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:49:25.629 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:49:25.629 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:49:25.629 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:49:25.629 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:49:25.629 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:49:25.629 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:49:25.629 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:49:25.630 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:49:25.630 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:49:25.630 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:49:25.655 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:49:25.655 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:49:25.655 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:49:25.655 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:49:25.762 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-06T09:49:25.762 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:rbd mirror flush : flush rbd mirror 2024-04-06T09:49:25.762 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:admin_socket: invalid command 2024-04-06T09:49:25.764 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:49:25.764 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/test 2024-04-06T09:49:25.764 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:1 2024-04-06T09:49:25.764 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:49:25.764 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:49:25.764 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-06T09:49:25.764 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:1 2024-04-06T09:49:25.764 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:49:25.764 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:49:25.764 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:49:25.764 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=1 2024-04-06T09:49:25.764 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-06T09:49:25.764 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:49:25.764 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:49:25.764 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=1 2024-04-06T09:49:25.765 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=1 2024-04-06T09:49:25.766 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-06T09:49:25.766 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:1 2024-04-06T09:49:25.766 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:49:25.766 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:49:25.766 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-06T09:49:25.766 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:1 2024-04-06T09:49:25.766 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:49:25.766 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:49:25.766 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:49:25.766 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=1 2024-04-06T09:49:25.766 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-06T09:49:25.767 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:49:25.767 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:49:25.767 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=1 2024-04-06T09:49:25.767 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=1 2024-04-06T09:49:25.767 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-06T09:49:25.789 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T09:49:25.790 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T09:49:25.790 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T09:49:25.790 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/test 2024-04-06T09:49:25.915 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:49:25.916 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:49:25.916 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:49:25.916 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:49:25.916 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:49:25.916 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:49:25.917 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:49:25.917 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:49:25.917 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:49:25.917 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:49:25.917 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:49:25.917 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:49:25.917 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:49:25.920 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:49:25.920 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:49:25.920 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:49:25.967 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 38 2024-04-06T09:49:25.967 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 38 2024-04-06T09:49:25.967 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:49:25.967 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=153, tag_tid=31, entry_tid=1], [object_number=152, tag_tid=31, entry_tid=0], [object_number=155, tag_tid=29, entry_tid=9059], [object_number=154, tag_tid=29, entry_tid=9058]]], state=connected] 2024-04-06T09:49:25.968 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=153, tag_tid=30, entry_tid=1], [object_number=152, tag_tid=30, entry_tid=0], [object_number=155, tag_tid=29, entry_tid=9059], [object_number=154, tag_tid=29, entry_tid=9058]]], state=connected] 2024-04-06T09:49:25.971 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:49:25.972 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=153, tag_tid=31, entry_tid=1' 2024-04-06T09:49:25.973 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:49:25.973 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:49:25.973 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:49:25.973 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:49:25.973 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:49:25.974 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:49:25.974 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:49:25.974 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:49:25.974 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:49:25.974 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:49:25.974 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:49:25.974 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:49:25.975 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:49:25.975 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:49:25.975 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:49:26.022 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 38 2024-04-06T09:49:26.022 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 38 2024-04-06T09:49:26.022 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:49:26.022 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=153, tag_tid=31, entry_tid=1], [object_number=152, tag_tid=31, entry_tid=0], [object_number=155, tag_tid=29, entry_tid=9059], [object_number=154, tag_tid=29, entry_tid=9058]]], state=connected] 2024-04-06T09:49:26.022 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=153, tag_tid=30, entry_tid=1], [object_number=152, tag_tid=30, entry_tid=0], [object_number=155, tag_tid=29, entry_tid=9059], [object_number=154, tag_tid=29, entry_tid=9058]]], state=connected] 2024-04-06T09:49:26.025 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:49:26.027 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=153, tag_tid=30, entry_tid=1' 2024-04-06T09:49:26.027 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=153, tag_tid=31, entry_tid=1' -a 'object_number=153, tag_tid=31, entry_tid=1' = 'object_number=153, tag_tid=30, entry_tid=1' 2024-04-06T09:49:26.027 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=153, tag_tid=30, entry_tid=1' '!=' 'object_number=153, tag_tid=30, entry_tid=1' 2024-04-06T09:49:26.027 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:49:26.027 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.8 2024-04-06T09:49:26.829 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:49:26.829 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:49:26.829 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:49:26.829 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:49:26.829 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:49:26.829 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:49:26.829 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:49:26.829 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:49:26.829 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:49:26.829 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:49:26.829 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:49:26.829 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:49:26.830 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:49:26.830 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:49:26.830 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:49:26.830 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:49:26.830 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:49:26.833 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:49:26.833 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:49:26.833 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:49:26.833 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:49:26.833 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:49:26.833 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:49:26.833 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:49:26.833 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:49:26.833 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:49:26.833 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:49:26.834 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:49:26.834 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:49:26.834 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:49:26.834 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:49:26.834 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:49:26.834 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:49:26.834 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:49:26.834 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:49:26.834 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:49:26.834 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:49:26.834 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:49:26.834 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:49:26.834 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:49:26.834 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:49:26.834 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:49:26.835 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:49:26.835 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:49:26.835 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:49:26.835 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:49:26.835 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:49:26.835 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:49:26.835 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:49:26.859 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:49:26.860 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:49:26.860 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:49:26.860 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:49:26.964 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-06T09:49:26.965 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:rbd mirror flush : flush rbd mirror 2024-04-06T09:49:26.965 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:admin_socket: invalid command 2024-04-06T09:49:26.967 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:49:26.967 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/test 2024-04-06T09:49:26.967 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:1 2024-04-06T09:49:26.967 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:49:26.967 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:49:26.967 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-06T09:49:26.967 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:1 2024-04-06T09:49:26.967 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:49:26.967 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:49:26.967 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:49:26.967 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=1 2024-04-06T09:49:26.967 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-06T09:49:26.968 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:49:26.968 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:49:26.968 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=1 2024-04-06T09:49:26.968 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=1 2024-04-06T09:49:26.969 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-06T09:49:26.969 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:1 2024-04-06T09:49:26.969 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:49:26.969 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:49:26.969 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-06T09:49:26.970 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:1 2024-04-06T09:49:26.970 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:49:26.970 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:49:26.970 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:49:26.970 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=1 2024-04-06T09:49:26.970 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-06T09:49:26.970 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:49:26.970 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:49:26.970 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=1 2024-04-06T09:49:26.970 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=1 2024-04-06T09:49:26.970 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-06T09:49:26.987 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T09:49:26.987 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T09:49:26.987 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T09:49:26.988 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/test 2024-04-06T09:49:27.104 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:49:27.106 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:49:27.106 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:49:27.106 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:49:27.106 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:49:27.106 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:49:27.106 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:49:27.106 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:49:27.106 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:49:27.106 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:49:27.107 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:49:27.107 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:49:27.107 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:49:27.108 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:49:27.109 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:49:27.109 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:49:27.166 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 38 2024-04-06T09:49:27.166 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 38 2024-04-06T09:49:27.166 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:49:27.166 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=153, tag_tid=31, entry_tid=1], [object_number=152, tag_tid=31, entry_tid=0], [object_number=155, tag_tid=29, entry_tid=9059], [object_number=154, tag_tid=29, entry_tid=9058]]], state=connected] 2024-04-06T09:49:27.166 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=153, tag_tid=30, entry_tid=1], [object_number=152, tag_tid=30, entry_tid=0], [object_number=155, tag_tid=29, entry_tid=9059], [object_number=154, tag_tid=29, entry_tid=9058]]], state=connected] 2024-04-06T09:49:27.170 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:49:27.171 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=153, tag_tid=31, entry_tid=1' 2024-04-06T09:49:27.172 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:49:27.172 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:49:27.172 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:49:27.172 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:49:27.173 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:49:27.173 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:49:27.173 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:49:27.173 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:49:27.173 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:49:27.173 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:49:27.173 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:49:27.173 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:49:27.173 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:49:27.174 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:49:27.174 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:49:27.224 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 38 2024-04-06T09:49:27.224 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 38 2024-04-06T09:49:27.224 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:49:27.224 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=153, tag_tid=31, entry_tid=1], [object_number=152, tag_tid=31, entry_tid=0], [object_number=155, tag_tid=29, entry_tid=9059], [object_number=154, tag_tid=29, entry_tid=9058]]], state=connected] 2024-04-06T09:49:27.225 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=153, tag_tid=30, entry_tid=1], [object_number=152, tag_tid=30, entry_tid=0], [object_number=155, tag_tid=29, entry_tid=9059], [object_number=154, tag_tid=29, entry_tid=9058]]], state=connected] 2024-04-06T09:49:27.228 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:49:27.229 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=153, tag_tid=30, entry_tid=1' 2024-04-06T09:49:27.230 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=153, tag_tid=31, entry_tid=1' -a 'object_number=153, tag_tid=31, entry_tid=1' = 'object_number=153, tag_tid=30, entry_tid=1' 2024-04-06T09:49:27.230 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=153, tag_tid=30, entry_tid=1' '!=' 'object_number=153, tag_tid=30, entry_tid=1' 2024-04-06T09:49:27.230 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:49:27.230 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 1.6 2024-04-06T09:49:28.831 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror test 2024-04-06T09:49:28.832 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:49:28.832 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:49:28.832 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:49:28.832 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:49:28.832 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n test ']' 2024-04-06T09:49:28.832 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/test' 2024-04-06T09:49:28.832 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-04-06T09:49:28.832 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:49:28.832 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:49:28.832 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:49:28.832 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:49:28.832 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:49:28.832 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:49:28.832 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:49:28.833 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:49:28.833 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:49:28.836 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:49:28.836 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-04-06T09:49:28.836 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:49:28.836 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:49:28.836 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:49:28.836 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:49:28.836 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:49:28.836 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:49:28.836 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:49:28.836 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:49:28.836 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:49:28.836 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:49:28.836 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:49:28.836 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:49:28.837 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:49:28.837 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:49:28.837 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:49:28.837 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:49:28.838 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:49:28.838 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:49:28.838 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:49:28.838 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:49:28.838 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:49:28.838 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:49:28.838 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:49:28.838 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:49:28.839 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:49:28.839 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:49:28.839 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:49:28.839 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:49:28.839 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:49:28.839 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:49:28.862 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:49:28.863 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:49:28.863 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:49:28.863 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-04-06T09:49:28.968 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-06T09:49:28.969 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:rbd mirror flush : flush rbd mirror 2024-04-06T09:49:28.969 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:admin_socket: invalid command 2024-04-06T09:49:28.971 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:49:28.971 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/test 2024-04-06T09:49:28.971 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:1 2024-04-06T09:49:28.971 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:49:28.971 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:49:28.972 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-06T09:49:28.972 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:1 2024-04-06T09:49:28.972 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:49:28.972 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:49:28.972 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:49:28.972 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=1 2024-04-06T09:49:28.972 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-06T09:49:28.972 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:49:28.972 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:49:28.972 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=1 2024-04-06T09:49:28.972 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=1 2024-04-06T09:49:28.972 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-06T09:49:28.972 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:1 2024-04-06T09:49:28.972 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:49:28.973 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:49:28.973 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-06T09:49:28.973 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:1 2024-04-06T09:49:28.973 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:49:28.973 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:49:28.973 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:49:28.973 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=1 2024-04-06T09:49:28.973 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-06T09:49:28.973 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:49:28.973 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:49:28.973 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=1 2024-04-06T09:49:28.973 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=1 2024-04-06T09:49:28.974 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-06T09:49:28.996 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T09:49:28.996 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T09:49:28.997 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T09:49:28.997 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/test 2024-04-06T09:49:29.121 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:49:29.123 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror test 2024-04-06T09:49:29.123 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:49:29.123 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:49:29.123 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:49:29.123 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '' 2024-04-06T09:49:29.123 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:49:29.123 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:49:29.123 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:49:29.123 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:49:29.124 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:49:29.124 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:49:29.124 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:49:29.125 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:49:29.125 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:49:29.126 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:49:29.174 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 38 2024-04-06T09:49:29.175 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 38 2024-04-06T09:49:29.175 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:49:29.175 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=153, tag_tid=31, entry_tid=1], [object_number=152, tag_tid=31, entry_tid=0], [object_number=155, tag_tid=29, entry_tid=9059], [object_number=154, tag_tid=29, entry_tid=9058]]], state=connected] 2024-04-06T09:49:29.175 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=153, tag_tid=31, entry_tid=1], [object_number=152, tag_tid=31, entry_tid=0], [object_number=155, tag_tid=29, entry_tid=9059], [object_number=154, tag_tid=29, entry_tid=9058]]], state=connected] 2024-04-06T09:49:29.178 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:49:29.180 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=153, tag_tid=31, entry_tid=1' 2024-04-06T09:49:29.181 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-04-06T09:49:29.181 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:49:29.181 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:49:29.181 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:49:29.181 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-04-06T09:49:29.181 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:49:29.181 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:49:29.181 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:49:29.181 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:49:29.181 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-test.status 2024-04-06T09:49:29.181 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-test.status 2024-04-06T09:49:29.182 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:49:29.182 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:49:29.182 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-04-06T09:49:29.183 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:49:29.231 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 38 2024-04-06T09:49:29.231 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 38 2024-04-06T09:49:29.231 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:49:29.231 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=153, tag_tid=31, entry_tid=1], [object_number=152, tag_tid=31, entry_tid=0], [object_number=155, tag_tid=29, entry_tid=9059], [object_number=154, tag_tid=29, entry_tid=9058]]], state=connected] 2024-04-06T09:49:29.231 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=153, tag_tid=31, entry_tid=1], [object_number=152, tag_tid=31, entry_tid=0], [object_number=155, tag_tid=29, entry_tid=9059], [object_number=154, tag_tid=29, entry_tid=9058]]], state=connected] 2024-04-06T09:49:29.234 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.status 2024-04-06T09:49:29.236 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=153, tag_tid=31, entry_tid=1' 2024-04-06T09:49:29.236 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=153, tag_tid=31, entry_tid=1' -a 'object_number=153, tag_tid=31, entry_tid=1' = 'object_number=153, tag_tid=31, entry_tid=1' 2024-04-06T09:49:29.236 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:49:29.237 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_snap_present cluster1 mirror test snap10-clean 2024-04-06T09:49:29.237 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:49:29.237 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:49:29.237 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:49:29.237 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local snap_name=snap10-clean 2024-04-06T09:49:29.237 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local s 2024-04-06T09:49:29.237 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-04-06T09:49:29.237 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 1 2024-04-06T09:49:30.238 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 info mirror/test@snap10-clean 2024-04-06T09:49:30.323 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:rbd image 'test': 2024-04-06T09:49:30.323 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: size 512 MiB in 128 objects 2024-04-06T09:49:30.323 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: order 22 (4 MiB objects) 2024-04-06T09:49:30.323 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: snapshot_count: 20 2024-04-06T09:49:30.323 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: id: 10a56da0b207 2024-04-06T09:49:30.323 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: block_name_prefix: rbd_data.10a56da0b207 2024-04-06T09:49:30.323 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: format: 2 2024-04-06T09:49:30.323 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: features: layering, exclusive-lock, journaling 2024-04-06T09:49:30.323 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: op_features: 2024-04-06T09:49:30.323 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: flags: 2024-04-06T09:49:30.324 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: create_timestamp: Sat Apr 6 09:31:59 2024 2024-04-06T09:49:30.324 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: access_timestamp: Sat Apr 6 09:49:17 2024 2024-04-06T09:49:30.324 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: modify_timestamp: Sat Apr 6 09:48:17 2024 2024-04-06T09:49:30.324 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: protected: False 2024-04-06T09:49:30.324 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: journal: 10a56da0b207 2024-04-06T09:49:30.324 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: mirroring state: enabled 2024-04-06T09:49:30.324 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: mirroring mode: journal 2024-04-06T09:49:30.324 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: mirroring global id: 8ad5dedc-b3e0-425d-8993-b3e2231fc5ca 2024-04-06T09:49:30.324 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: mirroring primary: false 2024-04-06T09:49:30.328 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:49:30.328 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 1 10 2024-04-06T09:49:30.330 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in `seq 1 10` 2024-04-06T09:49:30.331 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ snap_name=snap1 2024-04-06T09:49:30.331 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ compare_image_snaps mirror test snap1 2024-04-06T09:49:30.331 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:49:30.331 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:49:30.331 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local snap_name=snap1 2024-04-06T09:49:30.331 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local ret=0 2024-04-06T09:49:30.331 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local rmt_export=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.export 2024-04-06T09:49:30.331 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loc_export=/tmp/tmp.BhW6xKmROU/cluster1-mirror-test.export 2024-04-06T09:49:30.331 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rm -f /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.export /tmp/tmp.BhW6xKmROU/cluster1-mirror-test.export 2024-04-06T09:49:30.331 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster2 -p mirror export test@snap1 /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.export 2024-04-06T09:49:30.855 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-04-06T09:49:30.867 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 -p mirror export test@snap1 /tmp/tmp.BhW6xKmROU/cluster1-mirror-test.export 2024-04-06T09:49:31.898 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-04-06T09:49:31.914 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmp /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.export /tmp/tmp.BhW6xKmROU/cluster1-mirror-test.export 2024-04-06T09:49:32.193 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rm -f /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.export /tmp/tmp.BhW6xKmROU/cluster1-mirror-test.export 2024-04-06T09:49:32.329 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:49:32.329 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ snap_name=snap1-clean 2024-04-06T09:49:32.329 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ compare_image_snaps mirror test snap1-clean 2024-04-06T09:49:32.329 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:49:32.329 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:49:32.329 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local snap_name=snap1-clean 2024-04-06T09:49:32.329 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local ret=0 2024-04-06T09:49:32.329 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local rmt_export=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.export 2024-04-06T09:49:32.329 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loc_export=/tmp/tmp.BhW6xKmROU/cluster1-mirror-test.export 2024-04-06T09:49:32.329 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rm -f /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.export /tmp/tmp.BhW6xKmROU/cluster1-mirror-test.export 2024-04-06T09:49:32.330 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster2 -p mirror export test@snap1-clean /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.export 2024-04-06T09:49:32.869 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-04-06T09:49:32.884 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 -p mirror export test@snap1-clean /tmp/tmp.BhW6xKmROU/cluster1-mirror-test.export 2024-04-06T09:49:33.947 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-04-06T09:49:33.966 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmp /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.export /tmp/tmp.BhW6xKmROU/cluster1-mirror-test.export 2024-04-06T09:49:34.246 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rm -f /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.export /tmp/tmp.BhW6xKmROU/cluster1-mirror-test.export 2024-04-06T09:49:34.381 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:49:34.382 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in `seq 1 10` 2024-04-06T09:49:34.382 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ snap_name=snap2 2024-04-06T09:49:34.382 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ compare_image_snaps mirror test snap2 2024-04-06T09:49:34.382 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:49:34.382 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:49:34.382 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local snap_name=snap2 2024-04-06T09:49:34.382 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local ret=0 2024-04-06T09:49:34.382 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local rmt_export=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.export 2024-04-06T09:49:34.382 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loc_export=/tmp/tmp.BhW6xKmROU/cluster1-mirror-test.export 2024-04-06T09:49:34.382 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rm -f /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.export /tmp/tmp.BhW6xKmROU/cluster1-mirror-test.export 2024-04-06T09:49:34.383 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster2 -p mirror export test@snap2 /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.export 2024-04-06T09:49:34.935 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-04-06T09:49:34.949 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 -p mirror export test@snap2 /tmp/tmp.BhW6xKmROU/cluster1-mirror-test.export 2024-04-06T09:49:36.200 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-04-06T09:49:36.220 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmp /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.export /tmp/tmp.BhW6xKmROU/cluster1-mirror-test.export 2024-04-06T09:49:36.499 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rm -f /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.export /tmp/tmp.BhW6xKmROU/cluster1-mirror-test.export 2024-04-06T09:49:36.636 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:49:36.637 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ snap_name=snap2-clean 2024-04-06T09:49:36.637 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ compare_image_snaps mirror test snap2-clean 2024-04-06T09:49:36.637 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:49:36.637 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:49:36.637 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local snap_name=snap2-clean 2024-04-06T09:49:36.637 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local ret=0 2024-04-06T09:49:36.637 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local rmt_export=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.export 2024-04-06T09:49:36.637 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loc_export=/tmp/tmp.BhW6xKmROU/cluster1-mirror-test.export 2024-04-06T09:49:36.637 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rm -f /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.export /tmp/tmp.BhW6xKmROU/cluster1-mirror-test.export 2024-04-06T09:49:36.638 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster2 -p mirror export test@snap2-clean /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.export 2024-04-06T09:49:37.193 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-04-06T09:49:37.209 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 -p mirror export test@snap2-clean /tmp/tmp.BhW6xKmROU/cluster1-mirror-test.export 2024-04-06T09:49:38.407 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-04-06T09:49:38.426 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmp /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.export /tmp/tmp.BhW6xKmROU/cluster1-mirror-test.export 2024-04-06T09:49:38.707 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rm -f /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.export /tmp/tmp.BhW6xKmROU/cluster1-mirror-test.export 2024-04-06T09:49:38.845 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:49:38.845 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in `seq 1 10` 2024-04-06T09:49:38.846 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ snap_name=snap3 2024-04-06T09:49:38.846 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ compare_image_snaps mirror test snap3 2024-04-06T09:49:38.846 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:49:38.846 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:49:38.846 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local snap_name=snap3 2024-04-06T09:49:38.846 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local ret=0 2024-04-06T09:49:38.846 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local rmt_export=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.export 2024-04-06T09:49:38.846 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loc_export=/tmp/tmp.BhW6xKmROU/cluster1-mirror-test.export 2024-04-06T09:49:38.846 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rm -f /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.export /tmp/tmp.BhW6xKmROU/cluster1-mirror-test.export 2024-04-06T09:49:38.847 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster2 -p mirror export test@snap3 /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.export 2024-04-06T09:49:39.429 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-04-06T09:49:39.441 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 -p mirror export test@snap3 /tmp/tmp.BhW6xKmROU/cluster1-mirror-test.export 2024-04-06T09:49:40.895 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-04-06T09:49:40.916 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmp /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.export /tmp/tmp.BhW6xKmROU/cluster1-mirror-test.export 2024-04-06T09:49:41.200 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rm -f /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.export /tmp/tmp.BhW6xKmROU/cluster1-mirror-test.export 2024-04-06T09:49:41.336 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:49:41.337 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ snap_name=snap3-clean 2024-04-06T09:49:41.337 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ compare_image_snaps mirror test snap3-clean 2024-04-06T09:49:41.337 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:49:41.337 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:49:41.337 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local snap_name=snap3-clean 2024-04-06T09:49:41.337 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local ret=0 2024-04-06T09:49:41.337 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local rmt_export=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.export 2024-04-06T09:49:41.337 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loc_export=/tmp/tmp.BhW6xKmROU/cluster1-mirror-test.export 2024-04-06T09:49:41.337 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rm -f /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.export /tmp/tmp.BhW6xKmROU/cluster1-mirror-test.export 2024-04-06T09:49:41.337 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster2 -p mirror export test@snap3-clean /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.export 2024-04-06T09:49:41.924 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-04-06T09:49:41.939 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 -p mirror export test@snap3-clean /tmp/tmp.BhW6xKmROU/cluster1-mirror-test.export 2024-04-06T09:49:43.459 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-04-06T09:49:43.488 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmp /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.export /tmp/tmp.BhW6xKmROU/cluster1-mirror-test.export 2024-04-06T09:49:43.773 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rm -f /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.export /tmp/tmp.BhW6xKmROU/cluster1-mirror-test.export 2024-04-06T09:49:43.910 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:49:43.910 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in `seq 1 10` 2024-04-06T09:49:43.910 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ snap_name=snap4 2024-04-06T09:49:43.910 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ compare_image_snaps mirror test snap4 2024-04-06T09:49:43.910 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:49:43.910 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:49:43.910 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local snap_name=snap4 2024-04-06T09:49:43.910 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local ret=0 2024-04-06T09:49:43.910 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local rmt_export=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.export 2024-04-06T09:49:43.910 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loc_export=/tmp/tmp.BhW6xKmROU/cluster1-mirror-test.export 2024-04-06T09:49:43.910 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rm -f /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.export /tmp/tmp.BhW6xKmROU/cluster1-mirror-test.export 2024-04-06T09:49:43.911 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster2 -p mirror export test@snap4 /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.export 2024-04-06T09:49:44.511 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-04-06T09:49:44.529 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 -p mirror export test@snap4 /tmp/tmp.BhW6xKmROU/cluster1-mirror-test.export 2024-04-06T09:49:45.861 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-04-06T09:49:45.881 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmp /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.export /tmp/tmp.BhW6xKmROU/cluster1-mirror-test.export 2024-04-06T09:49:46.163 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rm -f /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.export /tmp/tmp.BhW6xKmROU/cluster1-mirror-test.export 2024-04-06T09:49:46.300 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:49:46.300 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ snap_name=snap4-clean 2024-04-06T09:49:46.300 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ compare_image_snaps mirror test snap4-clean 2024-04-06T09:49:46.300 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:49:46.300 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:49:46.301 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local snap_name=snap4-clean 2024-04-06T09:49:46.301 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local ret=0 2024-04-06T09:49:46.301 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local rmt_export=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.export 2024-04-06T09:49:46.301 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loc_export=/tmp/tmp.BhW6xKmROU/cluster1-mirror-test.export 2024-04-06T09:49:46.301 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rm -f /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.export /tmp/tmp.BhW6xKmROU/cluster1-mirror-test.export 2024-04-06T09:49:46.301 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster2 -p mirror export test@snap4-clean /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.export 2024-04-06T09:49:46.877 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-04-06T09:49:46.890 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 -p mirror export test@snap4-clean /tmp/tmp.BhW6xKmROU/cluster1-mirror-test.export 2024-04-06T09:49:48.357 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-04-06T09:49:48.379 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmp /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.export /tmp/tmp.BhW6xKmROU/cluster1-mirror-test.export 2024-04-06T09:49:48.664 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rm -f /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.export /tmp/tmp.BhW6xKmROU/cluster1-mirror-test.export 2024-04-06T09:49:48.802 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:49:48.802 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in `seq 1 10` 2024-04-06T09:49:48.802 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ snap_name=snap5 2024-04-06T09:49:48.802 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ compare_image_snaps mirror test snap5 2024-04-06T09:49:48.802 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:49:48.802 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:49:48.802 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local snap_name=snap5 2024-04-06T09:49:48.802 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local ret=0 2024-04-06T09:49:48.802 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local rmt_export=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.export 2024-04-06T09:49:48.803 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loc_export=/tmp/tmp.BhW6xKmROU/cluster1-mirror-test.export 2024-04-06T09:49:48.803 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rm -f /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.export /tmp/tmp.BhW6xKmROU/cluster1-mirror-test.export 2024-04-06T09:49:48.806 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster2 -p mirror export test@snap5 /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.export 2024-04-06T09:49:49.412 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-04-06T09:49:49.425 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 -p mirror export test@snap5 /tmp/tmp.BhW6xKmROU/cluster1-mirror-test.export 2024-04-06T09:49:49.920 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:49:49.917+0000 7f0954d28640 -1 bdev(0x56329d428e00 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 1 2024-04-06T09:49:49.920 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:49:49.917+0000 7f0955d2a640 -1 bdev(0x56329d428e00 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 3 2024-04-06T09:49:49.920 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:49:49.921+0000 7f095652b640 -1 bdev(0x56329d428e00 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 3 2024-04-06T09:49:50.800 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-04-06T09:49:50.816 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmp /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.export /tmp/tmp.BhW6xKmROU/cluster1-mirror-test.export 2024-04-06T09:49:51.099 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rm -f /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.export /tmp/tmp.BhW6xKmROU/cluster1-mirror-test.export 2024-04-06T09:49:51.236 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:49:51.236 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ snap_name=snap5-clean 2024-04-06T09:49:51.236 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ compare_image_snaps mirror test snap5-clean 2024-04-06T09:49:51.236 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:49:51.236 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:49:51.237 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local snap_name=snap5-clean 2024-04-06T09:49:51.237 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local ret=0 2024-04-06T09:49:51.237 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local rmt_export=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.export 2024-04-06T09:49:51.237 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loc_export=/tmp/tmp.BhW6xKmROU/cluster1-mirror-test.export 2024-04-06T09:49:51.237 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rm -f /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.export /tmp/tmp.BhW6xKmROU/cluster1-mirror-test.export 2024-04-06T09:49:51.237 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster2 -p mirror export test@snap5-clean /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.export 2024-04-06T09:49:52.098 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-04-06T09:49:52.110 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 -p mirror export test@snap5-clean /tmp/tmp.BhW6xKmROU/cluster1-mirror-test.export 2024-04-06T09:49:53.501 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-04-06T09:49:53.527 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmp /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.export /tmp/tmp.BhW6xKmROU/cluster1-mirror-test.export 2024-04-06T09:49:53.814 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rm -f /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.export /tmp/tmp.BhW6xKmROU/cluster1-mirror-test.export 2024-04-06T09:49:53.950 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:49:53.950 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in `seq 1 10` 2024-04-06T09:49:53.950 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ snap_name=snap6 2024-04-06T09:49:53.950 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ compare_image_snaps mirror test snap6 2024-04-06T09:49:53.950 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:49:53.950 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:49:53.950 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local snap_name=snap6 2024-04-06T09:49:53.950 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local ret=0 2024-04-06T09:49:53.951 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local rmt_export=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.export 2024-04-06T09:49:53.951 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loc_export=/tmp/tmp.BhW6xKmROU/cluster1-mirror-test.export 2024-04-06T09:49:53.951 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rm -f /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.export /tmp/tmp.BhW6xKmROU/cluster1-mirror-test.export 2024-04-06T09:49:53.951 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster2 -p mirror export test@snap6 /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.export 2024-04-06T09:49:54.563 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-04-06T09:49:54.576 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 -p mirror export test@snap6 /tmp/tmp.BhW6xKmROU/cluster1-mirror-test.export 2024-04-06T09:49:56.033 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-04-06T09:49:56.046 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmp /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.export /tmp/tmp.BhW6xKmROU/cluster1-mirror-test.export 2024-04-06T09:49:56.326 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rm -f /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.export /tmp/tmp.BhW6xKmROU/cluster1-mirror-test.export 2024-04-06T09:49:56.463 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:49:56.464 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ snap_name=snap6-clean 2024-04-06T09:49:56.464 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ compare_image_snaps mirror test snap6-clean 2024-04-06T09:49:56.464 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:49:56.464 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:49:56.464 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local snap_name=snap6-clean 2024-04-06T09:49:56.464 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local ret=0 2024-04-06T09:49:56.464 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local rmt_export=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.export 2024-04-06T09:49:56.464 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loc_export=/tmp/tmp.BhW6xKmROU/cluster1-mirror-test.export 2024-04-06T09:49:56.464 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rm -f /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.export /tmp/tmp.BhW6xKmROU/cluster1-mirror-test.export 2024-04-06T09:49:56.465 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster2 -p mirror export test@snap6-clean /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.export 2024-04-06T09:49:57.108 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-04-06T09:49:57.135 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 -p mirror export test@snap6-clean /tmp/tmp.BhW6xKmROU/cluster1-mirror-test.export 2024-04-06T09:49:58.562 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-04-06T09:49:58.576 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmp /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.export /tmp/tmp.BhW6xKmROU/cluster1-mirror-test.export 2024-04-06T09:49:58.859 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rm -f /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.export /tmp/tmp.BhW6xKmROU/cluster1-mirror-test.export 2024-04-06T09:49:58.998 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:49:58.998 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in `seq 1 10` 2024-04-06T09:49:58.998 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ snap_name=snap7 2024-04-06T09:49:58.998 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ compare_image_snaps mirror test snap7 2024-04-06T09:49:58.998 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:49:58.998 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:49:58.998 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local snap_name=snap7 2024-04-06T09:49:58.998 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local ret=0 2024-04-06T09:49:58.998 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local rmt_export=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.export 2024-04-06T09:49:58.998 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loc_export=/tmp/tmp.BhW6xKmROU/cluster1-mirror-test.export 2024-04-06T09:49:58.998 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rm -f /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.export /tmp/tmp.BhW6xKmROU/cluster1-mirror-test.export 2024-04-06T09:49:58.999 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster2 -p mirror export test@snap7 /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.export 2024-04-06T09:49:59.644 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-04-06T09:49:59.657 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 -p mirror export test@snap7 /tmp/tmp.BhW6xKmROU/cluster1-mirror-test.export 2024-04-06T09:50:01.203 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-04-06T09:50:01.224 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmp /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.export /tmp/tmp.BhW6xKmROU/cluster1-mirror-test.export 2024-04-06T09:50:01.506 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rm -f /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.export /tmp/tmp.BhW6xKmROU/cluster1-mirror-test.export 2024-04-06T09:50:01.643 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:50:01.643 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ snap_name=snap7-clean 2024-04-06T09:50:01.643 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ compare_image_snaps mirror test snap7-clean 2024-04-06T09:50:01.643 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:50:01.643 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:50:01.643 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local snap_name=snap7-clean 2024-04-06T09:50:01.643 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local ret=0 2024-04-06T09:50:01.643 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local rmt_export=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.export 2024-04-06T09:50:01.643 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loc_export=/tmp/tmp.BhW6xKmROU/cluster1-mirror-test.export 2024-04-06T09:50:01.643 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rm -f /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.export /tmp/tmp.BhW6xKmROU/cluster1-mirror-test.export 2024-04-06T09:50:01.644 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster2 -p mirror export test@snap7-clean /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.export 2024-04-06T09:50:02.250 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-04-06T09:50:02.262 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 -p mirror export test@snap7-clean /tmp/tmp.BhW6xKmROU/cluster1-mirror-test.export 2024-04-06T09:50:03.663 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-04-06T09:50:03.681 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmp /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.export /tmp/tmp.BhW6xKmROU/cluster1-mirror-test.export 2024-04-06T09:50:03.964 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rm -f /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.export /tmp/tmp.BhW6xKmROU/cluster1-mirror-test.export 2024-04-06T09:50:04.102 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:50:04.103 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in `seq 1 10` 2024-04-06T09:50:04.103 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ snap_name=snap8 2024-04-06T09:50:04.103 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ compare_image_snaps mirror test snap8 2024-04-06T09:50:04.103 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:50:04.103 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:50:04.103 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local snap_name=snap8 2024-04-06T09:50:04.103 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local ret=0 2024-04-06T09:50:04.103 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local rmt_export=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.export 2024-04-06T09:50:04.103 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loc_export=/tmp/tmp.BhW6xKmROU/cluster1-mirror-test.export 2024-04-06T09:50:04.103 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rm -f /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.export /tmp/tmp.BhW6xKmROU/cluster1-mirror-test.export 2024-04-06T09:50:04.104 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster2 -p mirror export test@snap8 /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.export 2024-04-06T09:50:04.719 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-04-06T09:50:04.733 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 -p mirror export test@snap8 /tmp/tmp.BhW6xKmROU/cluster1-mirror-test.export 2024-04-06T09:50:06.118 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-04-06T09:50:06.136 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmp /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.export /tmp/tmp.BhW6xKmROU/cluster1-mirror-test.export 2024-04-06T09:50:06.421 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rm -f /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.export /tmp/tmp.BhW6xKmROU/cluster1-mirror-test.export 2024-04-06T09:50:06.562 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:50:06.562 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ snap_name=snap8-clean 2024-04-06T09:50:06.562 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ compare_image_snaps mirror test snap8-clean 2024-04-06T09:50:06.562 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:50:06.562 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:50:06.562 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local snap_name=snap8-clean 2024-04-06T09:50:06.562 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local ret=0 2024-04-06T09:50:06.563 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local rmt_export=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.export 2024-04-06T09:50:06.563 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loc_export=/tmp/tmp.BhW6xKmROU/cluster1-mirror-test.export 2024-04-06T09:50:06.563 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rm -f /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.export /tmp/tmp.BhW6xKmROU/cluster1-mirror-test.export 2024-04-06T09:50:06.563 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster2 -p mirror export test@snap8-clean /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.export 2024-04-06T09:50:07.185 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-04-06T09:50:07.198 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 -p mirror export test@snap8-clean /tmp/tmp.BhW6xKmROU/cluster1-mirror-test.export 2024-04-06T09:50:07.664 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:50:07.661+0000 7f0954527640 -1 bdev(0x56329d428e00 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 1 2024-04-06T09:50:07.670 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:50:07.669+0000 7f0954d28640 -1 bdev(0x56329d428e00 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 1 2024-04-06T09:50:08.715 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-04-06T09:50:08.739 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmp /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.export /tmp/tmp.BhW6xKmROU/cluster1-mirror-test.export 2024-04-06T09:50:09.021 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rm -f /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.export /tmp/tmp.BhW6xKmROU/cluster1-mirror-test.export 2024-04-06T09:50:09.158 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:50:09.158 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in `seq 1 10` 2024-04-06T09:50:09.158 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ snap_name=snap9 2024-04-06T09:50:09.159 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ compare_image_snaps mirror test snap9 2024-04-06T09:50:09.159 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:50:09.159 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:50:09.159 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local snap_name=snap9 2024-04-06T09:50:09.159 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local ret=0 2024-04-06T09:50:09.159 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local rmt_export=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.export 2024-04-06T09:50:09.159 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loc_export=/tmp/tmp.BhW6xKmROU/cluster1-mirror-test.export 2024-04-06T09:50:09.159 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rm -f /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.export /tmp/tmp.BhW6xKmROU/cluster1-mirror-test.export 2024-04-06T09:50:09.159 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster2 -p mirror export test@snap9 /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.export 2024-04-06T09:50:09.778 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-04-06T09:50:09.790 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 -p mirror export test@snap9 /tmp/tmp.BhW6xKmROU/cluster1-mirror-test.export 2024-04-06T09:50:11.121 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-04-06T09:50:11.141 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmp /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.export /tmp/tmp.BhW6xKmROU/cluster1-mirror-test.export 2024-04-06T09:50:11.424 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rm -f /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.export /tmp/tmp.BhW6xKmROU/cluster1-mirror-test.export 2024-04-06T09:50:11.563 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:50:11.563 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ snap_name=snap9-clean 2024-04-06T09:50:11.563 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ compare_image_snaps mirror test snap9-clean 2024-04-06T09:50:11.563 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:50:11.563 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:50:11.563 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local snap_name=snap9-clean 2024-04-06T09:50:11.563 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local ret=0 2024-04-06T09:50:11.563 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local rmt_export=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.export 2024-04-06T09:50:11.563 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loc_export=/tmp/tmp.BhW6xKmROU/cluster1-mirror-test.export 2024-04-06T09:50:11.564 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rm -f /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.export /tmp/tmp.BhW6xKmROU/cluster1-mirror-test.export 2024-04-06T09:50:11.564 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster2 -p mirror export test@snap9-clean /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.export 2024-04-06T09:50:12.197 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-04-06T09:50:12.214 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 -p mirror export test@snap9-clean /tmp/tmp.BhW6xKmROU/cluster1-mirror-test.export 2024-04-06T09:50:12.761 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:50:12.757+0000 7f0955d2a640 -1 bdev(0x56329d428e00 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 4 2024-04-06T09:50:12.761 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:50:12.757+0000 7f0952523640 -1 bdev(0x56329d428e00 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 5 2024-04-06T09:50:12.761 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:50:12.757+0000 7f0954d28640 -1 bdev(0x56329d428e00 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 4 2024-04-06T09:50:12.761 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:50:12.761+0000 7f0954527640 -1 bdev(0x56329d428e00 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 4 2024-04-06T09:50:13.621 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-04-06T09:50:13.645 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmp /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.export /tmp/tmp.BhW6xKmROU/cluster1-mirror-test.export 2024-04-06T09:50:13.932 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rm -f /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.export /tmp/tmp.BhW6xKmROU/cluster1-mirror-test.export 2024-04-06T09:50:14.069 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:50:14.069 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in `seq 1 10` 2024-04-06T09:50:14.069 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ snap_name=snap10 2024-04-06T09:50:14.070 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ compare_image_snaps mirror test snap10 2024-04-06T09:50:14.070 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:50:14.070 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:50:14.070 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local snap_name=snap10 2024-04-06T09:50:14.070 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local ret=0 2024-04-06T09:50:14.070 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local rmt_export=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.export 2024-04-06T09:50:14.070 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loc_export=/tmp/tmp.BhW6xKmROU/cluster1-mirror-test.export 2024-04-06T09:50:14.070 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rm -f /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.export /tmp/tmp.BhW6xKmROU/cluster1-mirror-test.export 2024-04-06T09:50:14.070 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster2 -p mirror export test@snap10 /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.export 2024-04-06T09:50:14.711 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-04-06T09:50:14.723 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 -p mirror export test@snap10 /tmp/tmp.BhW6xKmROU/cluster1-mirror-test.export 2024-04-06T09:50:16.348 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-04-06T09:50:16.373 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmp /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.export /tmp/tmp.BhW6xKmROU/cluster1-mirror-test.export 2024-04-06T09:50:16.656 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rm -f /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.export /tmp/tmp.BhW6xKmROU/cluster1-mirror-test.export 2024-04-06T09:50:16.795 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:50:16.795 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ snap_name=snap10-clean 2024-04-06T09:50:16.795 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ compare_image_snaps mirror test snap10-clean 2024-04-06T09:50:16.796 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:50:16.796 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:50:16.796 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local snap_name=snap10-clean 2024-04-06T09:50:16.796 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local ret=0 2024-04-06T09:50:16.796 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local rmt_export=/tmp/tmp.BhW6xKmROU/cluster2-mirror-test.export 2024-04-06T09:50:16.796 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loc_export=/tmp/tmp.BhW6xKmROU/cluster1-mirror-test.export 2024-04-06T09:50:16.796 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rm -f /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.export /tmp/tmp.BhW6xKmROU/cluster1-mirror-test.export 2024-04-06T09:50:16.796 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster2 -p mirror export test@snap10-clean /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.export 2024-04-06T09:50:17.450 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-04-06T09:50:17.473 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 -p mirror export test@snap10-clean /tmp/tmp.BhW6xKmROU/cluster1-mirror-test.export 2024-04-06T09:50:19.047 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-04-06T09:50:19.068 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmp /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.export /tmp/tmp.BhW6xKmROU/cluster1-mirror-test.export 2024-04-06T09:50:19.352 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rm -f /tmp/tmp.BhW6xKmROU/cluster2-mirror-test.export /tmp/tmp.BhW6xKmROU/cluster1-mirror-test.export 2024-04-06T09:50:19.490 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:50:19.490 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 1 10 2024-04-06T09:50:19.492 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in `seq 1 10` 2024-04-06T09:50:19.492 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ snap_name=snap1 2024-04-06T09:50:19.492 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ remove_snapshot cluster2 mirror test snap1 2024-04-06T09:50:19.492 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-04-06T09:50:19.492 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:50:19.492 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:50:19.492 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local snap=snap1 2024-04-06T09:50:19.492 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap1 2024-04-06T09:50:19.720 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: Removing snap: 100% complete...done. 2024-04-06T09:50:19.751 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ snap_name=snap1-clean 2024-04-06T09:50:19.751 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ remove_snapshot cluster2 mirror test snap1-clean 2024-04-06T09:50:19.751 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-04-06T09:50:19.751 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:50:19.751 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:50:19.751 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local snap=snap1-clean 2024-04-06T09:50:19.751 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap1-clean 2024-04-06T09:50:20.765 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: Removing snap: 100% complete...done. 2024-04-06T09:50:20.766 INFO:tasks.ceph.cluster2.osd.1.smithi192.stderr:2024-04-06T09:50:20.768+0000 7f0de0719640 -1 reset not still connected to 0x55bfa3e74f70 2024-04-06T09:50:20.766 INFO:tasks.ceph.cluster2.osd.1.smithi192.stderr:2024-04-06T09:50:20.768+0000 7f0de0719640 -1 reset not still connected to 0x55bfa468d450 2024-04-06T09:50:20.766 INFO:tasks.ceph.cluster2.osd.1.smithi192.stderr:2024-04-06T09:50:20.768+0000 7f0de0719640 -1 reset not still connected to 0x55bfa468dee0 2024-04-06T09:50:20.804 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in `seq 1 10` 2024-04-06T09:50:20.804 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ snap_name=snap2 2024-04-06T09:50:20.804 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ remove_snapshot cluster2 mirror test snap2 2024-04-06T09:50:20.804 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-04-06T09:50:20.804 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:50:20.804 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:50:20.804 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local snap=snap2 2024-04-06T09:50:20.804 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap2 2024-04-06T09:50:21.743 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: Removing snap: 100% complete...done. 2024-04-06T09:50:21.775 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ snap_name=snap2-clean 2024-04-06T09:50:21.775 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ remove_snapshot cluster2 mirror test snap2-clean 2024-04-06T09:50:21.775 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-04-06T09:50:21.775 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:50:21.775 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:50:21.775 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local snap=snap2-clean 2024-04-06T09:50:21.776 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap2-clean 2024-04-06T09:50:22.771 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: Removing snap: 100% complete...done. 2024-04-06T09:50:22.813 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in `seq 1 10` 2024-04-06T09:50:22.813 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ snap_name=snap3 2024-04-06T09:50:22.813 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ remove_snapshot cluster2 mirror test snap3 2024-04-06T09:50:22.813 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-04-06T09:50:22.813 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:50:22.813 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:50:22.813 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local snap=snap3 2024-04-06T09:50:22.813 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap3 2024-04-06T09:50:23.781 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: Removing snap: 100% complete...done. 2024-04-06T09:50:23.823 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ snap_name=snap3-clean 2024-04-06T09:50:23.823 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ remove_snapshot cluster2 mirror test snap3-clean 2024-04-06T09:50:23.823 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-04-06T09:50:23.823 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:50:23.823 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:50:23.823 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local snap=snap3-clean 2024-04-06T09:50:23.824 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap3-clean 2024-04-06T09:50:24.521 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.0 with skip (0.6433871048402414) > weight (0.25) 2024-04-06T09:50:24.521 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.1 with skip (0.4484008664856908) > weight (0.25) 2024-04-06T09:50:24.521 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.2 with skip (0.8938790381516559) > weight (0.25) 2024-04-06T09:50:24.521 INFO:tasks.rbd_mirror_thrash:kill cluster1.client.mirror.3 2024-04-06T09:50:24.522 INFO:tasks.rbd_mirror.cluster1.client.mirror.3:Sent signal 15 2024-04-06T09:50:24.522 INFO:tasks.rbd_mirror_thrash:waiting for 7 secs before reviving daemons 2024-04-06T09:50:24.523 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi192.stderr:2024-04-06T09:50:24.528+0000 7fdf93461640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.3 (PID: 43008) UID: 1000 2024-04-06T09:50:24.781 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: Removing snap: 100% complete...done. 2024-04-06T09:50:24.817 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in `seq 1 10` 2024-04-06T09:50:24.817 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ snap_name=snap4 2024-04-06T09:50:24.817 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ remove_snapshot cluster2 mirror test snap4 2024-04-06T09:50:24.817 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-04-06T09:50:24.817 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:50:24.817 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:50:24.817 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local snap=snap4 2024-04-06T09:50:24.817 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap4 2024-04-06T09:50:25.648 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: Removing snap: 100% complete...done. 2024-04-06T09:50:25.675 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ snap_name=snap4-clean 2024-04-06T09:50:25.675 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ remove_snapshot cluster2 mirror test snap4-clean 2024-04-06T09:50:25.675 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-04-06T09:50:25.676 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:50:25.676 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:50:25.676 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local snap=snap4-clean 2024-04-06T09:50:25.676 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap4-clean 2024-04-06T09:50:26.790 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: Removing snap: 100% complete...done. 2024-04-06T09:50:26.821 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in `seq 1 10` 2024-04-06T09:50:26.821 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ snap_name=snap5 2024-04-06T09:50:26.821 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ remove_snapshot cluster2 mirror test snap5 2024-04-06T09:50:26.821 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-04-06T09:50:26.821 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:50:26.821 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:50:26.822 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local snap=snap5 2024-04-06T09:50:26.822 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap5 2024-04-06T09:50:26.951 INFO:tasks.ceph.cluster2.osd.0.smithi192.stderr:2024-04-06T09:50:26.952+0000 7f789d271640 -1 bdev(0x56041f2dce00 /var/lib/ceph/osd/cluster2-0/block) aio_submit retries 7 2024-04-06T09:50:26.953 INFO:tasks.ceph.cluster2.osd.0.smithi192.stderr:2024-04-06T09:50:26.956+0000 7f78a0a78640 -1 bdev(0x56041f2dce00 /var/lib/ceph/osd/cluster2-0/block) aio_submit retries 18 2024-04-06T09:50:27.783 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: Removing snap: 100% complete...done. 2024-04-06T09:50:27.816 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ snap_name=snap5-clean 2024-04-06T09:50:27.817 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ remove_snapshot cluster2 mirror test snap5-clean 2024-04-06T09:50:27.817 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-04-06T09:50:27.817 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:50:27.817 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:50:27.817 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local snap=snap5-clean 2024-04-06T09:50:27.817 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap5-clean 2024-04-06T09:50:28.800 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: Removing snap: 100% complete...done. 2024-04-06T09:50:28.833 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in `seq 1 10` 2024-04-06T09:50:28.834 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ snap_name=snap6 2024-04-06T09:50:28.834 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ remove_snapshot cluster2 mirror test snap6 2024-04-06T09:50:28.834 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-04-06T09:50:28.834 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:50:28.834 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:50:28.834 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local snap=snap6 2024-04-06T09:50:28.834 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap6 2024-04-06T09:50:29.795 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: Removing snap: 100% complete...done. 2024-04-06T09:50:29.832 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ snap_name=snap6-clean 2024-04-06T09:50:29.832 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ remove_snapshot cluster2 mirror test snap6-clean 2024-04-06T09:50:29.832 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-04-06T09:50:29.832 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:50:29.833 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:50:29.833 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local snap=snap6-clean 2024-04-06T09:50:29.833 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap6-clean 2024-04-06T09:50:29.936 INFO:tasks.ceph.cluster2.osd.0.smithi192.stderr:2024-04-06T09:50:29.939+0000 7f789ca70640 -1 bdev(0x56041f2dce00 /var/lib/ceph/osd/cluster2-0/block) aio_submit retries 10 2024-04-06T09:50:29.938 INFO:tasks.ceph.cluster2.osd.0.smithi192.stderr:2024-04-06T09:50:29.939+0000 7f789d271640 -1 bdev(0x56041f2dce00 /var/lib/ceph/osd/cluster2-0/block) aio_submit retries 14 2024-04-06T09:50:30.837 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: Removing snap: 100% complete...done. 2024-04-06T09:50:30.868 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in `seq 1 10` 2024-04-06T09:50:30.868 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ snap_name=snap7 2024-04-06T09:50:30.868 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ remove_snapshot cluster2 mirror test snap7 2024-04-06T09:50:30.868 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-04-06T09:50:30.869 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:50:30.869 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:50:30.869 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local snap=snap7 2024-04-06T09:50:30.869 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap7 2024-04-06T09:50:31.523 INFO:tasks.rbd_mirror_thrash:waiting for cluster1.client.mirror.3 2024-04-06T09:50:31.523 INFO:teuthology.orchestra.run:waiting for 600 2024-04-06T09:50:31.523 INFO:tasks.rbd_mirror_thrash:reviving cluster1.client.mirror.3 2024-04-06T09:50:31.524 INFO:tasks.rbd_mirror.cluster1.client.mirror.3:Restarting daemon 2024-04-06T09:50:31.524 DEBUG:teuthology.orchestra.run.smithi192:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.3 2024-04-06T09:50:31.527 INFO:tasks.rbd_mirror.cluster1.client.mirror.3:Started 2024-04-06T09:50:31.527 INFO:tasks.rbd_mirror_thrash:waiting for 119 secs before thrashing 2024-04-06T09:50:31.806 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: Removing snap: 100% complete...done. 2024-04-06T09:50:31.838 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ snap_name=snap7-clean 2024-04-06T09:50:31.838 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ remove_snapshot cluster2 mirror test snap7-clean 2024-04-06T09:50:31.838 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-04-06T09:50:31.838 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:50:31.838 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:50:31.838 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local snap=snap7-clean 2024-04-06T09:50:31.838 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap7-clean 2024-04-06T09:50:32.830 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: Removing snap: 100% complete...done. 2024-04-06T09:50:32.869 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in `seq 1 10` 2024-04-06T09:50:32.869 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ snap_name=snap8 2024-04-06T09:50:32.869 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ remove_snapshot cluster2 mirror test snap8 2024-04-06T09:50:32.869 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-04-06T09:50:32.869 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:50:32.869 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:50:32.869 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local snap=snap8 2024-04-06T09:50:32.869 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap8 2024-04-06T09:50:33.832 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: Removing snap: 100% complete...done. 2024-04-06T09:50:33.874 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ snap_name=snap8-clean 2024-04-06T09:50:33.874 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ remove_snapshot cluster2 mirror test snap8-clean 2024-04-06T09:50:33.874 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-04-06T09:50:33.874 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:50:33.874 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:50:33.875 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local snap=snap8-clean 2024-04-06T09:50:33.875 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap8-clean 2024-04-06T09:50:34.874 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: Removing snap: 100% complete...done. 2024-04-06T09:50:34.912 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in `seq 1 10` 2024-04-06T09:50:34.913 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ snap_name=snap9 2024-04-06T09:50:34.913 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ remove_snapshot cluster2 mirror test snap9 2024-04-06T09:50:34.913 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-04-06T09:50:34.913 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:50:34.913 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:50:34.913 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local snap=snap9 2024-04-06T09:50:34.914 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap9 2024-04-06T09:50:35.667 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: Removing snap: 100% complete...done. 2024-04-06T09:50:35.697 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ snap_name=snap9-clean 2024-04-06T09:50:35.697 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ remove_snapshot cluster2 mirror test snap9-clean 2024-04-06T09:50:35.697 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-04-06T09:50:35.697 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:50:35.697 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:50:35.697 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local snap=snap9-clean 2024-04-06T09:50:35.697 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap9-clean 2024-04-06T09:50:35.812 INFO:tasks.ceph.cluster2.osd.0.smithi192.stderr:2024-04-06T09:50:35.815+0000 7f789ca70640 -1 bdev(0x56041f2dce00 /var/lib/ceph/osd/cluster2-0/block) aio_submit retries 7 2024-04-06T09:50:36.854 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: Removing snap: 100% complete...done. 2024-04-06T09:50:36.890 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in `seq 1 10` 2024-04-06T09:50:36.890 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ snap_name=snap10 2024-04-06T09:50:36.890 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ remove_snapshot cluster2 mirror test snap10 2024-04-06T09:50:36.891 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-04-06T09:50:36.891 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:50:36.891 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:50:36.891 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local snap=snap10 2024-04-06T09:50:36.891 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap10 2024-04-06T09:50:37.863 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: Removing snap: 100% complete...done. 2024-04-06T09:50:37.880 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ snap_name=snap10-clean 2024-04-06T09:50:37.880 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ remove_snapshot cluster2 mirror test snap10-clean 2024-04-06T09:50:37.880 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-04-06T09:50:37.880 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:50:37.880 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:50:37.880 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local snap=snap10-clean 2024-04-06T09:50:37.880 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap10-clean 2024-04-06T09:50:38.871 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: Removing snap: 100% complete...done. 2024-04-06T09:50:38.895 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ remove_image_retry cluster2 mirror test 2024-04-06T09:50:38.895 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-04-06T09:50:38.895 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:50:38.895 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:50:38.895 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 16 32 2024-04-06T09:50:38.895 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:50:38.897 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ remove_image cluster2 mirror test 2024-04-06T09:50:38.897 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-04-06T09:50:38.897 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:50:38.897 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:50:38.897 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster=cluster2 snap purge mirror/test 2024-04-06T09:50:38.950 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster=cluster2 rm mirror/test 2024-04-06T09:50:39.316 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2024-04-06T09:50:39.321 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:50:39.322 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_image_present cluster1 mirror test deleted 2024-04-06T09:50:39.322 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:50:39.322 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:50:39.322 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:50:39.322 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state=deleted 2024-04-06T09:50:39.322 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image_id= 2024-04-06T09:50:39.322 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local s 2024-04-06T09:50:39.322 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n '' 2024-04-06T09:50:39.322 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_image_id cluster1 mirror test 2024-04-06T09:50:39.322 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:50:39.322 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:50:39.322 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:50:39.322 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster=cluster1 info mirror/test 2024-04-06T09:50:39.322 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-04-06T09:50:39.360 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:rbd: error opening image test: (2) No such file or directory 2024-04-06T09:50:39.364 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ image_id= 2024-04-06T09:50:39.364 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-04-06T09:50:39.364 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.1 2024-04-06T09:50:39.465 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test_image_present cluster1 mirror test deleted '' 2024-04-06T09:50:39.465 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:50:39.465 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:50:39.466 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=test 2024-04-06T09:50:39.466 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local test_state=deleted 2024-04-06T09:50:39.466 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image_id= 2024-04-06T09:50:39.466 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local current_state=deleted 2024-04-06T09:50:39.466 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local current_image_id 2024-04-06T09:50:39.466 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_image_id cluster1 mirror test 2024-04-06T09:50:39.466 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:50:39.467 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:50:39.467 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=test 2024-04-06T09:50:39.467 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster=cluster1 info mirror/test 2024-04-06T09:50:39.467 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-04-06T09:50:39.510 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:rbd: error opening image test: (2) No such file or directory 2024-04-06T09:50:39.515 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ current_image_id= 2024-04-06T09:50:39.515 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n '' 2024-04-06T09:50:39.516 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test deleted = deleted 2024-04-06T09:50:39.516 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:50:39.516 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ testlog 'TEST: create many images' 2024-04-06T09:50:39.516 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ tee -a /tmp/tmp.BhW6xKmROU/rbd-mirror.test.log 2024-04-06T09:50:39.516 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ date '+%F %T' 2024-04-06T09:50:39.516 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ echo 2024-04-06 09:50:39 TEST: create many images 2024-04-06T09:50:39.517 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:2024-04-06 09:50:39 TEST: create many images 2024-04-06T09:50:39.517 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ snap_name=snap 2024-04-06T09:50:39.517 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 1 50 2024-04-06T09:50:39.518 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-06T09:50:39.519 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ image=image_1 2024-04-06T09:50:39.519 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ create_image_and_enable_mirror cluster2 mirror image_1 journal 128M 2024-04-06T09:50:39.519 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-04-06T09:50:39.519 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:50:39.519 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:50:39.519 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:50:39.519 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_1 2024-04-06T09:50:39.519 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:50:39.519 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local mode=journal 2024-04-06T09:50:39.519 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n journal ']' 2024-04-06T09:50:39.519 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:50:39.519 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ create_image cluster2 mirror image_1 128M 2024-04-06T09:50:39.519 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-04-06T09:50:39.519 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:50:39.520 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:50:39.520 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:50:39.520 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_1 2024-04-06T09:50:39.520 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:50:39.520 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local size=128 2024-04-06T09:50:39.520 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n 128M ']' 2024-04-06T09:50:39.520 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ size=128M 2024-04-06T09:50:39.520 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:50:39.520 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,journaling mirror/image_1 2024-04-06T09:50:39.582 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' pool = image ']' 2024-04-06T09:50:39.582 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' mirror = mirror_parent ']' 2024-04-06T09:50:39.582 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n '' ']' 2024-04-06T09:50:39.582 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ stress_write_image cluster2 mirror image_1 2024-04-06T09:50:39.582 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-04-06T09:50:39.582 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:50:39.582 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_1 2024-04-06T09:50:39.582 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-04-06T09:50:39.583 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local duration=10 2024-04-06T09:50:39.584 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set +e 2024-04-06T09:50:39.584 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ timeout 10s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_1 --debug-rbd=20 --debug-journaler=20 2024-04-06T09:50:41.109 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:100 IOs, 26059776 bytes 2024-04-06T09:50:43.421 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.0 with skip (0.36633192350246135) > weight (0.25) 2024-04-06T09:50:43.421 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.1 with skip (0.32778169886976705) > weight (0.25) 2024-04-06T09:50:43.422 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.2 with skip (0.9964343214799235) > weight (0.25) 2024-04-06T09:50:43.422 INFO:tasks.rbd_mirror_thrash:kill cluster2.client.mirror.3 2024-04-06T09:50:43.422 INFO:tasks.rbd_mirror.cluster2.client.mirror.3:Sent signal 15 2024-04-06T09:50:43.422 INFO:tasks.rbd_mirror_thrash:waiting for 6 secs before reviving daemons 2024-04-06T09:50:43.423 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi192.stderr:2024-04-06T09:50:43.427+0000 7f45fca80640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.3 (PID: 119416) UID: 1000 2024-04-06T09:50:44.034 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:200 IOs, 92664832 bytes 2024-04-06T09:50:46.366 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:300 IOs, 125028352 bytes 2024-04-06T09:50:47.063 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:400 IOs, 138323968 bytes 2024-04-06T09:50:47.485 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:500 IOs, 143970304 bytes 2024-04-06T09:50:49.368 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:600 IOs, 186372096 bytes 2024-04-06T09:50:49.423 INFO:tasks.rbd_mirror_thrash:waiting for cluster2.client.mirror.3 2024-04-06T09:50:49.423 INFO:teuthology.orchestra.run:waiting for 600 2024-04-06T09:50:49.424 INFO:tasks.rbd_mirror_thrash:reviving cluster2.client.mirror.3 2024-04-06T09:50:49.424 INFO:tasks.rbd_mirror.cluster2.client.mirror.3:Restarting daemon 2024-04-06T09:50:49.424 DEBUG:teuthology.orchestra.run.smithi192:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.3 2024-04-06T09:50:49.426 INFO:tasks.rbd_mirror.cluster2.client.mirror.3:Started 2024-04-06T09:50:49.426 INFO:tasks.rbd_mirror_thrash:waiting for 99 secs before thrashing 2024-04-06T09:50:49.595 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ error_code=124 2024-04-06T09:50:49.595 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set -e 2024-04-06T09:50:49.595 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 124 -eq 124 ']' 2024-04-06T09:50:49.595 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:50:49.595 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-06T09:50:49.595 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ image=image_2 2024-04-06T09:50:49.596 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ create_image_and_enable_mirror cluster2 mirror image_2 journal 128M 2024-04-06T09:50:49.596 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-04-06T09:50:49.596 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:50:49.596 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:50:49.596 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:50:49.596 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_2 2024-04-06T09:50:49.596 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:50:49.596 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local mode=journal 2024-04-06T09:50:49.596 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n journal ']' 2024-04-06T09:50:49.596 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:50:49.596 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ create_image cluster2 mirror image_2 128M 2024-04-06T09:50:49.596 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-04-06T09:50:49.596 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:50:49.596 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:50:49.597 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:50:49.597 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_2 2024-04-06T09:50:49.597 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:50:49.597 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local size=128 2024-04-06T09:50:49.597 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n 128M ']' 2024-04-06T09:50:49.597 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ size=128M 2024-04-06T09:50:49.597 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:50:49.597 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,journaling mirror/image_2 2024-04-06T09:50:49.665 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' pool = image ']' 2024-04-06T09:50:49.665 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' mirror = mirror_parent ']' 2024-04-06T09:50:49.665 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n '' ']' 2024-04-06T09:50:49.665 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ stress_write_image cluster2 mirror image_2 2024-04-06T09:50:49.665 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-04-06T09:50:49.665 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:50:49.665 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_2 2024-04-06T09:50:49.665 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-04-06T09:50:49.666 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local duration=9 2024-04-06T09:50:49.666 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set +e 2024-04-06T09:50:49.666 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ timeout 9s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_2 --debug-rbd=20 --debug-journaler=20 2024-04-06T09:50:53.855 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:100 IOs, 44883968 bytes 2024-04-06T09:50:55.515 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:200 IOs, 67255296 bytes 2024-04-06T09:50:57.522 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:300 IOs, 103390208 bytes 2024-04-06T09:50:58.673 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ error_code=124 2024-04-06T09:50:58.673 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set -e 2024-04-06T09:50:58.674 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 124 -eq 124 ']' 2024-04-06T09:50:58.674 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:50:58.674 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-06T09:50:58.674 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ image=image_3 2024-04-06T09:50:58.674 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ create_image_and_enable_mirror cluster2 mirror image_3 journal 128M 2024-04-06T09:50:58.674 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-04-06T09:50:58.674 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:50:58.674 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:50:58.674 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:50:58.674 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_3 2024-04-06T09:50:58.674 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:50:58.674 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local mode=journal 2024-04-06T09:50:58.674 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n journal ']' 2024-04-06T09:50:58.675 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:50:58.675 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ create_image cluster2 mirror image_3 128M 2024-04-06T09:50:58.675 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-04-06T09:50:58.675 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:50:58.675 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:50:58.675 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:50:58.675 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_3 2024-04-06T09:50:58.675 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:50:58.675 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local size=128 2024-04-06T09:50:58.675 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n 128M ']' 2024-04-06T09:50:58.675 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ size=128M 2024-04-06T09:50:58.675 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:50:58.675 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,journaling mirror/image_3 2024-04-06T09:50:58.729 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' pool = image ']' 2024-04-06T09:50:58.730 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' mirror = mirror_parent ']' 2024-04-06T09:50:58.730 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n '' ']' 2024-04-06T09:50:58.730 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ stress_write_image cluster2 mirror image_3 2024-04-06T09:50:58.730 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-04-06T09:50:58.730 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:50:58.730 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_3 2024-04-06T09:50:58.730 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-04-06T09:50:58.731 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local duration=10 2024-04-06T09:50:58.731 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set +e 2024-04-06T09:50:58.731 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ timeout 10s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_3 --debug-rbd=20 --debug-journaler=20 2024-04-06T09:51:00.762 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:100 IOs, 60236800 bytes 2024-04-06T09:51:02.945 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:200 IOs, 72565760 bytes 2024-04-06T09:51:03.519 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:300 IOs, 83212288 bytes 2024-04-06T09:51:05.488 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:400 IOs, 113346560 bytes 2024-04-06T09:51:06.637 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:500 IOs, 131251200 bytes 2024-04-06T09:51:08.549 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:600 IOs, 173927424 bytes 2024-04-06T09:51:08.738 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ error_code=124 2024-04-06T09:51:08.739 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set -e 2024-04-06T09:51:08.739 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 124 -eq 124 ']' 2024-04-06T09:51:08.739 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:51:08.739 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-06T09:51:08.739 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ image=image_4 2024-04-06T09:51:08.739 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ create_image_and_enable_mirror cluster2 mirror image_4 journal 128M 2024-04-06T09:51:08.739 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-04-06T09:51:08.739 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:51:08.739 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:51:08.739 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:51:08.739 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_4 2024-04-06T09:51:08.739 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:51:08.739 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local mode=journal 2024-04-06T09:51:08.739 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n journal ']' 2024-04-06T09:51:08.739 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:51:08.740 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ create_image cluster2 mirror image_4 128M 2024-04-06T09:51:08.740 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-04-06T09:51:08.740 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:51:08.740 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:51:08.740 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:51:08.740 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_4 2024-04-06T09:51:08.740 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:51:08.740 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local size=128 2024-04-06T09:51:08.740 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n 128M ']' 2024-04-06T09:51:08.740 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ size=128M 2024-04-06T09:51:08.740 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:51:08.740 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,journaling mirror/image_4 2024-04-06T09:51:08.793 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' pool = image ']' 2024-04-06T09:51:08.793 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' mirror = mirror_parent ']' 2024-04-06T09:51:08.794 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n '' ']' 2024-04-06T09:51:08.794 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ stress_write_image cluster2 mirror image_4 2024-04-06T09:51:08.794 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-04-06T09:51:08.794 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:51:08.794 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_4 2024-04-06T09:51:08.794 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-04-06T09:51:08.795 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local duration=11 2024-04-06T09:51:08.795 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set +e 2024-04-06T09:51:08.795 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ timeout 11s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_4 --debug-rbd=20 --debug-journaler=20 2024-04-06T09:51:09.183 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:100 IOs, 2902016 bytes 2024-04-06T09:51:11.526 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-04-06T09:51:11.524+0000 7fd1b6210640 -1 reset not still connected to 0x55ef997761a0 2024-04-06T09:51:11.527 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-04-06T09:51:11.524+0000 7fd1b6210640 -1 reset not still connected to 0x55ef99776680 2024-04-06T09:51:11.528 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-04-06T09:51:11.528+0000 7fd1b6210640 -1 reset not still connected to 0x55ef99b88d00 2024-04-06T09:51:11.528 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-04-06T09:51:11.528+0000 7fd1b6210640 -1 reset not still connected to 0x55ef9ac60270 2024-04-06T09:51:11.528 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-04-06T09:51:11.528+0000 7fd1b6210640 -1 reset not still connected to 0x55ef9e88f520 2024-04-06T09:51:11.803 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:200 IOs, 31098880 bytes 2024-04-06T09:51:14.429 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:300 IOs, 96915456 bytes 2024-04-06T09:51:16.993 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:400 IOs, 141599744 bytes 2024-04-06T09:51:18.524 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:500 IOs, 169171968 bytes 2024-04-06T09:51:19.806 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ error_code=124 2024-04-06T09:51:19.807 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set -e 2024-04-06T09:51:19.807 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 124 -eq 124 ']' 2024-04-06T09:51:19.807 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:51:19.807 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-06T09:51:19.807 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ image=image_5 2024-04-06T09:51:19.807 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ create_image_and_enable_mirror cluster2 mirror image_5 journal 128M 2024-04-06T09:51:19.807 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-04-06T09:51:19.807 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:51:19.807 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:51:19.807 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:51:19.807 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_5 2024-04-06T09:51:19.807 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:51:19.807 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local mode=journal 2024-04-06T09:51:19.807 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n journal ']' 2024-04-06T09:51:19.808 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:51:19.808 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ create_image cluster2 mirror image_5 128M 2024-04-06T09:51:19.808 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-04-06T09:51:19.808 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:51:19.808 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:51:19.808 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:51:19.808 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_5 2024-04-06T09:51:19.808 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:51:19.808 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local size=128 2024-04-06T09:51:19.808 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n 128M ']' 2024-04-06T09:51:19.808 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ size=128M 2024-04-06T09:51:19.808 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:51:19.808 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,journaling mirror/image_5 2024-04-06T09:51:19.885 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' pool = image ']' 2024-04-06T09:51:19.885 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' mirror = mirror_parent ']' 2024-04-06T09:51:19.885 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n '' ']' 2024-04-06T09:51:19.885 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ stress_write_image cluster2 mirror image_5 2024-04-06T09:51:19.885 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-04-06T09:51:19.885 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:51:19.885 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_5 2024-04-06T09:51:19.885 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-04-06T09:51:19.886 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local duration=12 2024-04-06T09:51:19.886 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set +e 2024-04-06T09:51:19.886 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ timeout 12s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_5 --debug-rbd=20 --debug-journaler=20 2024-04-06T09:51:22.182 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:100 IOs, 35651584 bytes 2024-04-06T09:51:25.545 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:200 IOs, 85966848 bytes 2024-04-06T09:51:28.773 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:300 IOs, 143722496 bytes 2024-04-06T09:51:31.894 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ error_code=124 2024-04-06T09:51:31.895 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set -e 2024-04-06T09:51:31.895 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 124 -eq 124 ']' 2024-04-06T09:51:31.895 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:51:31.895 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-06T09:51:31.895 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ image=image_6 2024-04-06T09:51:31.895 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ create_image_and_enable_mirror cluster2 mirror image_6 journal 128M 2024-04-06T09:51:31.895 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-04-06T09:51:31.895 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:51:31.895 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:51:31.895 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:51:31.895 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_6 2024-04-06T09:51:31.895 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:51:31.895 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local mode=journal 2024-04-06T09:51:31.895 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n journal ']' 2024-04-06T09:51:31.896 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:51:31.896 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ create_image cluster2 mirror image_6 128M 2024-04-06T09:51:31.896 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-04-06T09:51:31.896 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:51:31.896 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:51:31.896 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:51:31.896 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_6 2024-04-06T09:51:31.896 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:51:31.896 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local size=128 2024-04-06T09:51:31.896 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n 128M ']' 2024-04-06T09:51:31.896 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ size=128M 2024-04-06T09:51:31.896 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:51:31.896 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,journaling mirror/image_6 2024-04-06T09:51:31.953 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' pool = image ']' 2024-04-06T09:51:31.954 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' mirror = mirror_parent ']' 2024-04-06T09:51:31.954 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n '' ']' 2024-04-06T09:51:31.954 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ stress_write_image cluster2 mirror image_6 2024-04-06T09:51:31.954 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-04-06T09:51:31.954 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:51:31.954 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_6 2024-04-06T09:51:31.954 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-04-06T09:51:31.955 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local duration=14 2024-04-06T09:51:31.955 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set +e 2024-04-06T09:51:31.955 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ timeout 14s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_6 --debug-rbd=20 --debug-journaler=20 2024-04-06T09:51:33.277 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:51:33.276+0000 7f2e84d66640 -1 reset not still connected to 0x560d27c3edd0 2024-04-06T09:51:33.277 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:51:33.276+0000 7f2e84d66640 -1 reset not still connected to 0x560d27c3f2b0 2024-04-06T09:51:33.277 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:51:33.276+0000 7f2e84d66640 -1 reset not still connected to 0x560d2a17d6c0 2024-04-06T09:51:33.279 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:51:33.276+0000 7f2e84d66640 -1 reset not still connected to 0x560d2a2336c0 2024-04-06T09:51:36.623 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:100 IOs, 67289088 bytes 2024-04-06T09:51:38.746 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:200 IOs, 99280896 bytes 2024-04-06T09:51:38.971 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:300 IOs, 100141056 bytes 2024-04-06T09:51:41.332 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:400 IOs, 158845952 bytes 2024-04-06T09:51:42.564 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:500 IOs, 177841152 bytes 2024-04-06T09:51:43.759 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:600 IOs, 195095552 bytes 2024-04-06T09:51:44.294 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:700 IOs, 219481088 bytes 2024-04-06T09:51:45.963 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ error_code=124 2024-04-06T09:51:45.963 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set -e 2024-04-06T09:51:45.963 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 124 -eq 124 ']' 2024-04-06T09:51:45.963 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:51:45.963 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-06T09:51:45.963 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ image=image_7 2024-04-06T09:51:45.963 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ create_image_and_enable_mirror cluster2 mirror image_7 journal 128M 2024-04-06T09:51:45.964 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-04-06T09:51:45.964 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:51:45.964 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:51:45.964 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:51:45.964 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_7 2024-04-06T09:51:45.964 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:51:45.964 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local mode=journal 2024-04-06T09:51:45.964 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n journal ']' 2024-04-06T09:51:45.964 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:51:45.964 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ create_image cluster2 mirror image_7 128M 2024-04-06T09:51:45.964 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-04-06T09:51:45.964 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:51:45.964 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:51:45.964 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:51:45.965 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_7 2024-04-06T09:51:45.965 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:51:45.965 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local size=128 2024-04-06T09:51:45.965 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n 128M ']' 2024-04-06T09:51:45.965 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ size=128M 2024-04-06T09:51:45.965 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:51:45.965 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,journaling mirror/image_7 2024-04-06T09:51:46.030 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' pool = image ']' 2024-04-06T09:51:46.030 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' mirror = mirror_parent ']' 2024-04-06T09:51:46.030 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n '' ']' 2024-04-06T09:51:46.030 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ stress_write_image cluster2 mirror image_7 2024-04-06T09:51:46.030 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-04-06T09:51:46.030 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:51:46.031 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_7 2024-04-06T09:51:46.031 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-04-06T09:51:46.031 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local duration=13 2024-04-06T09:51:46.031 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set +e 2024-04-06T09:51:46.032 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ timeout 13s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_7 --debug-rbd=20 --debug-journaler=20 2024-04-06T09:51:51.477 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:100 IOs, 88559616 bytes 2024-04-06T09:51:56.659 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:200 IOs, 203707392 bytes 2024-04-06T09:51:58.996 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:51:58.995+0000 7f2e84d66640 -1 reset not still connected to 0x560d2a17d450 2024-04-06T09:51:58.996 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:51:58.995+0000 7f2e84d66640 -1 reset not still connected to 0x560d2a17d6c0 2024-04-06T09:51:58.996 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:51:58.995+0000 7f2e84d66640 -1 reset not still connected to 0x560d2a2336c0 2024-04-06T09:51:58.997 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:51:58.995+0000 7f2e84d66640 -1 reset not still connected to 0x560d368e4f70 2024-04-06T09:51:59.041 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ error_code=124 2024-04-06T09:51:59.041 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set -e 2024-04-06T09:51:59.041 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 124 -eq 124 ']' 2024-04-06T09:51:59.042 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:51:59.042 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-06T09:51:59.042 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ image=image_8 2024-04-06T09:51:59.042 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ create_image_and_enable_mirror cluster2 mirror image_8 journal 128M 2024-04-06T09:51:59.042 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-04-06T09:51:59.042 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:51:59.042 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:51:59.042 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:51:59.042 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_8 2024-04-06T09:51:59.042 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:51:59.042 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local mode=journal 2024-04-06T09:51:59.042 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n journal ']' 2024-04-06T09:51:59.043 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:51:59.043 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ create_image cluster2 mirror image_8 128M 2024-04-06T09:51:59.043 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-04-06T09:51:59.043 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:51:59.043 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:51:59.043 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:51:59.043 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_8 2024-04-06T09:51:59.043 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:51:59.043 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local size=128 2024-04-06T09:51:59.043 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n 128M ']' 2024-04-06T09:51:59.043 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ size=128M 2024-04-06T09:51:59.043 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:51:59.043 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,journaling mirror/image_8 2024-04-06T09:51:59.100 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' pool = image ']' 2024-04-06T09:51:59.101 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' mirror = mirror_parent ']' 2024-04-06T09:51:59.101 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n '' ']' 2024-04-06T09:51:59.101 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ stress_write_image cluster2 mirror image_8 2024-04-06T09:51:59.101 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-04-06T09:51:59.101 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:51:59.101 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_8 2024-04-06T09:51:59.101 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-04-06T09:51:59.102 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local duration=5 2024-04-06T09:51:59.102 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set +e 2024-04-06T09:51:59.102 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ timeout 5s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_8 --debug-rbd=20 --debug-journaler=20 2024-04-06T09:52:00.662 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:52:00.659+0000 7f2e84d66640 -1 reset not still connected to 0x560d2a2336c0 2024-04-06T09:52:00.662 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:52:00.659+0000 7f2e84d66640 -1 reset not still connected to 0x560d368e4f70 2024-04-06T09:52:00.676 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-04-06T09:52:00.675+0000 7fd1b6210640 -1 reset not still connected to 0x55ef99b88d00 2024-04-06T09:52:00.676 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-04-06T09:52:00.675+0000 7fd1b6210640 -1 reset not still connected to 0x55ef99b89ad0 2024-04-06T09:52:00.676 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-04-06T09:52:00.675+0000 7fd1b6210640 -1 reset not still connected to 0x55ef9ac60270 2024-04-06T09:52:00.676 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-04-06T09:52:00.675+0000 7fd1b6210640 -1 reset not still connected to 0x55ef9e88f520 2024-04-06T09:52:01.072 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:100 IOs, 30169088 bytes 2024-04-06T09:52:01.592 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:52:01.591+0000 7f2e84d66640 -1 reset not still connected to 0x560d2a2336c0 2024-04-06T09:52:01.598 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:52:01.595+0000 7f2e84d66640 -1 reset not still connected to 0x560d368e4f70 2024-04-06T09:52:01.609 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:52:01.607+0000 7f2e84d66640 -1 reset not still connected to 0x560d2a17d6c0 2024-04-06T09:52:01.609 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:52:01.607+0000 7f2e84d66640 -1 reset not still connected to 0x560d2a2336c0 2024-04-06T09:52:01.609 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:52:01.607+0000 7f2e84d66640 -1 reset not still connected to 0x560d368e4f70 2024-04-06T09:52:04.112 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ error_code=124 2024-04-06T09:52:04.113 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set -e 2024-04-06T09:52:04.113 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 124 -eq 124 ']' 2024-04-06T09:52:04.113 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:52:04.113 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-06T09:52:04.113 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ image=image_9 2024-04-06T09:52:04.113 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ create_image_and_enable_mirror cluster2 mirror image_9 journal 128M 2024-04-06T09:52:04.113 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-04-06T09:52:04.113 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:52:04.113 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:52:04.113 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:52:04.113 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_9 2024-04-06T09:52:04.113 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:52:04.113 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local mode=journal 2024-04-06T09:52:04.114 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n journal ']' 2024-04-06T09:52:04.114 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:52:04.114 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ create_image cluster2 mirror image_9 128M 2024-04-06T09:52:04.114 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-04-06T09:52:04.114 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:52:04.114 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:52:04.114 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:52:04.114 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_9 2024-04-06T09:52:04.114 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:52:04.114 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local size=128 2024-04-06T09:52:04.114 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n 128M ']' 2024-04-06T09:52:04.114 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ size=128M 2024-04-06T09:52:04.115 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:52:04.115 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,journaling mirror/image_9 2024-04-06T09:52:04.183 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' pool = image ']' 2024-04-06T09:52:04.183 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' mirror = mirror_parent ']' 2024-04-06T09:52:04.184 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n '' ']' 2024-04-06T09:52:04.184 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ stress_write_image cluster2 mirror image_9 2024-04-06T09:52:04.184 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-04-06T09:52:04.184 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:52:04.184 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_9 2024-04-06T09:52:04.184 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-04-06T09:52:04.185 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local duration=7 2024-04-06T09:52:04.185 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set +e 2024-04-06T09:52:04.185 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ timeout 7s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_9 --debug-rbd=20 --debug-journaler=20 2024-04-06T09:52:04.266 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:52:04.259+0000 7f2e84d66640 -1 reset not still connected to 0x560d2a2336c0 2024-04-06T09:52:09.100 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:100 IOs, 79300608 bytes 2024-04-06T09:52:11.198 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ error_code=124 2024-04-06T09:52:11.198 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set -e 2024-04-06T09:52:11.198 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 124 -eq 124 ']' 2024-04-06T09:52:11.198 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:52:11.198 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-06T09:52:11.198 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ image=image_10 2024-04-06T09:52:11.198 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ create_image_and_enable_mirror cluster2 mirror image_10 journal 128M 2024-04-06T09:52:11.198 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-04-06T09:52:11.198 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:52:11.199 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:52:11.199 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:52:11.199 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_10 2024-04-06T09:52:11.199 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:52:11.199 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local mode=journal 2024-04-06T09:52:11.199 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n journal ']' 2024-04-06T09:52:11.199 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:52:11.199 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ create_image cluster2 mirror image_10 128M 2024-04-06T09:52:11.199 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-04-06T09:52:11.199 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:52:11.199 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:52:11.199 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:52:11.199 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_10 2024-04-06T09:52:11.199 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:52:11.199 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local size=128 2024-04-06T09:52:11.200 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n 128M ']' 2024-04-06T09:52:11.200 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ size=128M 2024-04-06T09:52:11.200 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:52:11.200 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,journaling mirror/image_10 2024-04-06T09:52:11.259 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' pool = image ']' 2024-04-06T09:52:11.259 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' mirror = mirror_parent ']' 2024-04-06T09:52:11.259 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n '' ']' 2024-04-06T09:52:11.259 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ stress_write_image cluster2 mirror image_10 2024-04-06T09:52:11.259 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-04-06T09:52:11.259 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:52:11.259 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_10 2024-04-06T09:52:11.259 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-04-06T09:52:11.260 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local duration=13 2024-04-06T09:52:11.260 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set +e 2024-04-06T09:52:11.260 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ timeout 13s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_10 --debug-rbd=20 --debug-journaler=20 2024-04-06T09:52:11.694 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:52:11.691+0000 7f0965192640 -1 reset not still connected to 0x5632e16851e0 2024-04-06T09:52:11.929 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:52:11.927+0000 7f0965192640 -1 reset not still connected to 0x5632e16851e0 2024-04-06T09:52:12.700 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi192.stderr:2024-04-06T09:52:12.701+0000 7f3d28bce640 -1 rbd::mirror::InstanceWatcher: C_NotifyInstanceRequest: 0x560663839980 finish: resending after timeout 2024-04-06T09:52:12.700 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi192.stderr:2024-04-06T09:52:12.701+0000 7f3d293cf640 -1 rbd::mirror::InstanceWatcher: C_NotifyInstanceRequest: 0x560664b11200 finish: resending after timeout 2024-04-06T09:52:15.730 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:100 IOs, 51867648 bytes 2024-04-06T09:52:16.944 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:200 IOs, 66072576 bytes 2024-04-06T09:52:17.932 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:300 IOs, 89604096 bytes 2024-04-06T09:52:20.571 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:400 IOs, 126185472 bytes 2024-04-06T09:52:22.012 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:500 IOs, 164555776 bytes 2024-04-06T09:52:23.691 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:600 IOs, 189671424 bytes 2024-04-06T09:52:24.270 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ error_code=124 2024-04-06T09:52:24.270 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set -e 2024-04-06T09:52:24.270 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 124 -eq 124 ']' 2024-04-06T09:52:24.271 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:52:24.271 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-06T09:52:24.271 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ image=image_11 2024-04-06T09:52:24.271 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ create_image_and_enable_mirror cluster2 mirror image_11 journal 128M 2024-04-06T09:52:24.271 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-04-06T09:52:24.271 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:52:24.271 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:52:24.271 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:52:24.271 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_11 2024-04-06T09:52:24.271 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:52:24.271 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local mode=journal 2024-04-06T09:52:24.271 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n journal ']' 2024-04-06T09:52:24.271 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:52:24.271 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ create_image cluster2 mirror image_11 128M 2024-04-06T09:52:24.272 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-04-06T09:52:24.272 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:52:24.272 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:52:24.272 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:52:24.272 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_11 2024-04-06T09:52:24.272 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:52:24.272 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local size=128 2024-04-06T09:52:24.272 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n 128M ']' 2024-04-06T09:52:24.272 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ size=128M 2024-04-06T09:52:24.272 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:52:24.272 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,journaling mirror/image_11 2024-04-06T09:52:24.342 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' pool = image ']' 2024-04-06T09:52:24.342 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' mirror = mirror_parent ']' 2024-04-06T09:52:24.342 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n '' ']' 2024-04-06T09:52:24.342 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ stress_write_image cluster2 mirror image_11 2024-04-06T09:52:24.342 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-04-06T09:52:24.343 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:52:24.343 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_11 2024-04-06T09:52:24.343 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-04-06T09:52:24.343 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local duration=13 2024-04-06T09:52:24.343 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set +e 2024-04-06T09:52:24.344 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ timeout 13s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_11 --debug-rbd=20 --debug-journaler=20 2024-04-06T09:52:26.756 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:100 IOs, 29661184 bytes 2024-04-06T09:52:28.427 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.0 with skip (0.7401759282389895) > weight (0.25) 2024-04-06T09:52:28.428 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.1 with skip (0.8837569674925152) > weight (0.25) 2024-04-06T09:52:28.428 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.2 with skip (0.29243406147204143) > weight (0.25) 2024-04-06T09:52:28.428 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.3 with skip (0.720606494310274) > weight (0.25) 2024-04-06T09:52:28.428 INFO:tasks.rbd_mirror_thrash:waiting for 97 secs before thrashing 2024-04-06T09:52:28.442 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:52:28.439+0000 7f2e84d66640 -1 reset not still connected to 0x560d27839930 2024-04-06T09:52:28.442 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:52:28.439+0000 7f2e84d66640 -1 reset not still connected to 0x560d27c3edd0 2024-04-06T09:52:28.442 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:52:28.439+0000 7f2e84d66640 -1 reset not still connected to 0x560d27c3f2b0 2024-04-06T09:52:28.442 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:52:28.439+0000 7f2e84d66640 -1 reset not still connected to 0x560d294908f0 2024-04-06T09:52:28.443 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:52:28.439+0000 7f2e84d66640 -1 reset not still connected to 0x560d2a17d6c0 2024-04-06T09:52:28.443 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:52:28.439+0000 7f2e84d66640 -1 reset not still connected to 0x560d2a2336c0 2024-04-06T09:52:28.443 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:52:28.439+0000 7f2e84d66640 -1 reset not still connected to 0x560d368e4f70 2024-04-06T09:52:30.244 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi192.stderr:2024-04-06T09:52:30.249+0000 7f3d28bce640 -1 rbd::mirror::InstanceWatcher: C_NotifyInstanceRequest: 0x560663839980 finish: resending after timeout 2024-04-06T09:52:30.245 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi192.stderr:2024-04-06T09:52:30.249+0000 7f3d293cf640 -1 rbd::mirror::InstanceWatcher: C_NotifyInstanceRequest: 0x560664b11200 finish: resending after timeout 2024-04-06T09:52:30.430 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:52:30.427+0000 7f0965192640 -1 reset not still connected to 0x5632a4cda000 2024-04-06T09:52:30.430 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:52:30.427+0000 7f0965192640 -1 reset not still connected to 0x5632a5e9dee0 2024-04-06T09:52:30.431 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:52:30.427+0000 7f0965192640 -1 reset not still connected to 0x5632abc81d40 2024-04-06T09:52:30.431 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:52:30.427+0000 7f0965192640 -1 reset not still connected to 0x5632abcbe820 2024-04-06T09:52:30.431 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:52:30.427+0000 7f0965192640 -1 reset not still connected to 0x5632cb3ac000 2024-04-06T09:52:30.431 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:52:30.427+0000 7f0965192640 -1 reset not still connected to 0x5632e16851e0 2024-04-06T09:52:30.528 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.0 with skip (0.3401440097521815) > weight (0.25) 2024-04-06T09:52:30.528 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.1 with skip (0.7747568910887165) > weight (0.25) 2024-04-06T09:52:30.528 INFO:tasks.rbd_mirror_thrash:kill cluster1.client.mirror.2 2024-04-06T09:52:30.528 INFO:tasks.rbd_mirror.cluster1.client.mirror.2:Sent signal 15 2024-04-06T09:52:30.529 INFO:tasks.rbd_mirror_thrash:waiting for 0 secs before reviving daemons 2024-04-06T09:52:30.529 INFO:tasks.rbd_mirror_thrash:waiting for cluster1.client.mirror.2 2024-04-06T09:52:30.529 INFO:teuthology.orchestra.run:waiting for 600 2024-04-06T09:52:30.529 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi192.stderr:2024-04-06T09:52:30.533+0000 7fde6ed30640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.2 (PID: 120149) UID: 1000 2024-04-06T09:52:31.747 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:200 IOs, 102264832 bytes 2024-04-06T09:52:36.307 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:52:36.303+0000 7f2e84d66640 -1 reset not still connected to 0x560d27c3edd0 2024-04-06T09:52:36.308 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:52:36.303+0000 7f2e84d66640 -1 reset not still connected to 0x560d27c3f2b0 2024-04-06T09:52:36.308 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:52:36.303+0000 7f2e84d66640 -1 reset not still connected to 0x560d294908f0 2024-04-06T09:52:36.308 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:52:36.303+0000 7f2e84d66640 -1 reset not still connected to 0x560d29cd8f70 2024-04-06T09:52:36.308 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:52:36.303+0000 7f2e84d66640 -1 reset not still connected to 0x560d29d5b520 2024-04-06T09:52:36.308 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:52:36.303+0000 7f2e84d66640 -1 reset not still connected to 0x560d2a17d450 2024-04-06T09:52:36.308 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:52:36.303+0000 7f2e84d66640 -1 reset not still connected to 0x560d2a17d6c0 2024-04-06T09:52:36.308 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:52:36.303+0000 7f2e84d66640 -1 reset not still connected to 0x560d2a2336c0 2024-04-06T09:52:36.631 INFO:tasks.rbd_mirror_thrash:reviving cluster1.client.mirror.2 2024-04-06T09:52:36.631 INFO:tasks.rbd_mirror.cluster1.client.mirror.2:Restarting daemon 2024-04-06T09:52:36.631 DEBUG:teuthology.orchestra.run.smithi192:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.2 2024-04-06T09:52:36.634 INFO:tasks.rbd_mirror.cluster1.client.mirror.2:Started 2024-04-06T09:52:36.634 INFO:tasks.rbd_mirror_thrash:waiting for 96 secs before thrashing 2024-04-06T09:52:37.034 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:300 IOs, 187123712 bytes 2024-04-06T09:52:37.355 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ error_code=124 2024-04-06T09:52:37.355 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set -e 2024-04-06T09:52:37.355 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 124 -eq 124 ']' 2024-04-06T09:52:37.356 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:52:37.356 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-06T09:52:37.356 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ image=image_12 2024-04-06T09:52:37.356 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ create_image_and_enable_mirror cluster2 mirror image_12 journal 128M 2024-04-06T09:52:37.356 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-04-06T09:52:37.356 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:52:37.356 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:52:37.356 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:52:37.356 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_12 2024-04-06T09:52:37.356 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:52:37.356 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local mode=journal 2024-04-06T09:52:37.356 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n journal ']' 2024-04-06T09:52:37.356 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:52:37.356 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ create_image cluster2 mirror image_12 128M 2024-04-06T09:52:37.356 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-04-06T09:52:37.357 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:52:37.357 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:52:37.357 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:52:37.357 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_12 2024-04-06T09:52:37.357 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:52:37.357 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local size=128 2024-04-06T09:52:37.357 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n 128M ']' 2024-04-06T09:52:37.357 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ size=128M 2024-04-06T09:52:37.357 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:52:37.357 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,journaling mirror/image_12 2024-04-06T09:52:37.424 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' pool = image ']' 2024-04-06T09:52:37.424 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' mirror = mirror_parent ']' 2024-04-06T09:52:37.424 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n '' ']' 2024-04-06T09:52:37.424 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ stress_write_image cluster2 mirror image_12 2024-04-06T09:52:37.424 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-04-06T09:52:37.424 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:52:37.424 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_12 2024-04-06T09:52:37.425 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-04-06T09:52:37.426 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local duration=7 2024-04-06T09:52:37.426 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set +e 2024-04-06T09:52:37.426 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ timeout 7s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_12 --debug-rbd=20 --debug-journaler=20 2024-04-06T09:52:38.848 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:100 IOs, 25014272 bytes 2024-04-06T09:52:39.476 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi192.stderr:2024-04-06T09:52:39.481+0000 7f3d28bce640 -1 rbd::mirror::InstanceWatcher: C_NotifyInstanceRequest: 0x560663839980 finish: resending after timeout 2024-04-06T09:52:39.476 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi192.stderr:2024-04-06T09:52:39.481+0000 7f3d293cf640 -1 rbd::mirror::InstanceWatcher: C_NotifyInstanceRequest: 0x560664b11200 finish: resending after timeout 2024-04-06T09:52:40.451 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-04-06T09:52:40.447+0000 7fd1b6210640 -1 reset not still connected to 0x55ef9ac60270 2024-04-06T09:52:40.454 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-04-06T09:52:40.451+0000 7fd1b6210640 -1 reset not still connected to 0x55ef9c103110 2024-04-06T09:52:40.454 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-04-06T09:52:40.451+0000 7fd1b6210640 -1 reset not still connected to 0x55ef9e88f520 2024-04-06T09:52:40.455 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-04-06T09:52:40.451+0000 7fd1b6210640 -1 reset not still connected to 0x55ef9eee21a0 2024-04-06T09:52:42.630 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:200 IOs, 50776064 bytes 2024-04-06T09:52:44.437 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ error_code=124 2024-04-06T09:52:44.437 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set -e 2024-04-06T09:52:44.437 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 124 -eq 124 ']' 2024-04-06T09:52:44.437 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:52:44.437 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-06T09:52:44.437 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ image=image_13 2024-04-06T09:52:44.437 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ create_image_and_enable_mirror cluster2 mirror image_13 journal 128M 2024-04-06T09:52:44.438 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-04-06T09:52:44.438 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:52:44.438 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:52:44.438 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:52:44.438 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_13 2024-04-06T09:52:44.438 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:52:44.438 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local mode=journal 2024-04-06T09:52:44.438 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n journal ']' 2024-04-06T09:52:44.438 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:52:44.438 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ create_image cluster2 mirror image_13 128M 2024-04-06T09:52:44.438 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-04-06T09:52:44.438 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:52:44.438 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:52:44.438 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:52:44.438 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_13 2024-04-06T09:52:44.439 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:52:44.439 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local size=128 2024-04-06T09:52:44.439 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n 128M ']' 2024-04-06T09:52:44.439 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ size=128M 2024-04-06T09:52:44.439 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:52:44.439 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,journaling mirror/image_13 2024-04-06T09:52:44.524 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' pool = image ']' 2024-04-06T09:52:44.524 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' mirror = mirror_parent ']' 2024-04-06T09:52:44.524 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n '' ']' 2024-04-06T09:52:44.525 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ stress_write_image cluster2 mirror image_13 2024-04-06T09:52:44.525 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-04-06T09:52:44.525 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:52:44.525 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_13 2024-04-06T09:52:44.525 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-04-06T09:52:44.526 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local duration=7 2024-04-06T09:52:44.526 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set +e 2024-04-06T09:52:44.526 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ timeout 7s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_13 --debug-rbd=20 --debug-journaler=20 2024-04-06T09:52:48.386 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:52:48.387+0000 7f0965192640 -1 reset not still connected to 0x5632a417e750 2024-04-06T09:52:48.386 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:52:48.387+0000 7f0965192640 -1 reset not still connected to 0x5632a5e9dee0 2024-04-06T09:52:48.387 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:52:48.387+0000 7f0965192640 -1 reset not still connected to 0x5632abc81d40 2024-04-06T09:52:48.387 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:52:48.387+0000 7f0965192640 -1 reset not still connected to 0x5632e16851e0 2024-04-06T09:52:49.605 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:52:49.603+0000 7f0965192640 -1 reset not still connected to 0x5632a5e9dee0 2024-04-06T09:52:49.605 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:52:49.603+0000 7f0965192640 -1 reset not still connected to 0x5632abc81d40 2024-04-06T09:52:49.605 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:52:49.607+0000 7f0965192640 -1 reset not still connected to 0x5632abcbe820 2024-04-06T09:52:49.606 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:52:49.607+0000 7f0965192640 -1 reset not still connected to 0x5632e16851e0 2024-04-06T09:52:49.697 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:52:49.695+0000 7f0965192640 -1 reset not still connected to 0x5632a5e9dee0 2024-04-06T09:52:49.697 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:52:49.699+0000 7f0965192640 -1 reset not still connected to 0x5632abcbe820 2024-04-06T09:52:49.697 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:52:49.699+0000 7f0965192640 -1 reset not still connected to 0x5632e16851e0 2024-04-06T09:52:49.730 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:100 IOs, 64843776 bytes 2024-04-06T09:52:49.994 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:200 IOs, 66685952 bytes 2024-04-06T09:52:50.058 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:52:50.059+0000 7f0965192640 -1 reset not still connected to 0x5632a4cda000 2024-04-06T09:52:50.059 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:52:50.059+0000 7f0965192640 -1 reset not still connected to 0x5632a5e9dee0 2024-04-06T09:52:50.059 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:52:50.059+0000 7f0965192640 -1 reset not still connected to 0x5632abc81d40 2024-04-06T09:52:50.059 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:52:50.059+0000 7f0965192640 -1 reset not still connected to 0x5632abcbe820 2024-04-06T09:52:50.061 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:52:50.063+0000 7f0965192640 -1 reset not still connected to 0x5632cb3ac000 2024-04-06T09:52:50.062 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:52:50.063+0000 7f0965192640 -1 reset not still connected to 0x5632e16851e0 2024-04-06T09:52:51.536 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ error_code=124 2024-04-06T09:52:51.536 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set -e 2024-04-06T09:52:51.536 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 124 -eq 124 ']' 2024-04-06T09:52:51.536 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:52:51.536 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-06T09:52:51.536 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ image=image_14 2024-04-06T09:52:51.536 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ create_image_and_enable_mirror cluster2 mirror image_14 journal 128M 2024-04-06T09:52:51.536 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-04-06T09:52:51.536 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:52:51.536 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:52:51.536 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:52:51.536 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_14 2024-04-06T09:52:51.536 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:52:51.536 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local mode=journal 2024-04-06T09:52:51.537 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n journal ']' 2024-04-06T09:52:51.537 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:52:51.537 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ create_image cluster2 mirror image_14 128M 2024-04-06T09:52:51.537 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-04-06T09:52:51.537 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:52:51.537 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:52:51.537 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:52:51.537 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_14 2024-04-06T09:52:51.537 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:52:51.537 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local size=128 2024-04-06T09:52:51.537 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n 128M ']' 2024-04-06T09:52:51.537 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ size=128M 2024-04-06T09:52:51.537 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:52:51.537 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,journaling mirror/image_14 2024-04-06T09:52:51.613 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' pool = image ']' 2024-04-06T09:52:51.614 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' mirror = mirror_parent ']' 2024-04-06T09:52:51.614 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n '' ']' 2024-04-06T09:52:51.614 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ stress_write_image cluster2 mirror image_14 2024-04-06T09:52:51.614 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-04-06T09:52:51.614 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:52:51.614 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_14 2024-04-06T09:52:51.614 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-04-06T09:52:51.615 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local duration=7 2024-04-06T09:52:51.615 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set +e 2024-04-06T09:52:51.615 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ timeout 7s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_14 --debug-rbd=20 --debug-journaler=20 2024-04-06T09:52:52.648 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:100 IOs, 29774848 bytes 2024-04-06T09:52:56.804 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:200 IOs, 54725632 bytes 2024-04-06T09:52:58.067 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:300 IOs, 101150720 bytes 2024-04-06T09:52:58.625 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ error_code=124 2024-04-06T09:52:58.625 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set -e 2024-04-06T09:52:58.625 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 124 -eq 124 ']' 2024-04-06T09:52:58.625 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:52:58.625 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-06T09:52:58.625 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ image=image_15 2024-04-06T09:52:58.625 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ create_image_and_enable_mirror cluster2 mirror image_15 journal 128M 2024-04-06T09:52:58.625 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-04-06T09:52:58.625 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:52:58.626 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:52:58.626 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:52:58.626 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_15 2024-04-06T09:52:58.626 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:52:58.626 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local mode=journal 2024-04-06T09:52:58.626 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n journal ']' 2024-04-06T09:52:58.626 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:52:58.626 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ create_image cluster2 mirror image_15 128M 2024-04-06T09:52:58.626 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-04-06T09:52:58.626 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:52:58.626 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:52:58.626 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:52:58.626 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_15 2024-04-06T09:52:58.626 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:52:58.627 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local size=128 2024-04-06T09:52:58.627 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n 128M ']' 2024-04-06T09:52:58.627 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ size=128M 2024-04-06T09:52:58.627 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:52:58.627 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,journaling mirror/image_15 2024-04-06T09:52:58.701 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' pool = image ']' 2024-04-06T09:52:58.702 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' mirror = mirror_parent ']' 2024-04-06T09:52:58.702 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n '' ']' 2024-04-06T09:52:58.702 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ stress_write_image cluster2 mirror image_15 2024-04-06T09:52:58.702 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-04-06T09:52:58.702 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:52:58.702 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_15 2024-04-06T09:52:58.702 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-04-06T09:52:58.703 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local duration=6 2024-04-06T09:52:58.703 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set +e 2024-04-06T09:52:58.703 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ timeout 6s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_15 --debug-rbd=20 --debug-journaler=20 2024-04-06T09:53:01.228 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:100 IOs, 33624064 bytes 2024-04-06T09:53:01.333 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:53:01.327+0000 7f2e84d66640 -1 reset not still connected to 0x560d27073860 2024-04-06T09:53:01.333 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:53:01.327+0000 7f2e84d66640 -1 reset not still connected to 0x560d27567380 2024-04-06T09:53:01.333 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:53:01.327+0000 7f2e84d66640 -1 reset not still connected to 0x560d27839380 2024-04-06T09:53:01.333 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:53:01.327+0000 7f2e84d66640 -1 reset not still connected to 0x560d27839930 2024-04-06T09:53:01.333 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:53:01.327+0000 7f2e84d66640 -1 reset not still connected to 0x560d27c3edd0 2024-04-06T09:53:01.333 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:53:01.327+0000 7f2e84d66640 -1 reset not still connected to 0x560d27c3f2b0 2024-04-06T09:53:01.333 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:53:01.331+0000 7f2e84d66640 -1 reset not still connected to 0x560d29cd8f70 2024-04-06T09:53:01.334 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:53:01.331+0000 7f2e84d66640 -1 reset not still connected to 0x560d29d5b520 2024-04-06T09:53:01.334 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:53:01.331+0000 7f2e84d66640 -1 reset not still connected to 0x560d2a17d450 2024-04-06T09:53:01.334 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:53:01.331+0000 7f2e84d66640 -1 reset not still connected to 0x560d2a17d6c0 2024-04-06T09:53:01.334 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:53:01.331+0000 7f2e84d66640 -1 reset not still connected to 0x560d2a2336c0 2024-04-06T09:53:01.337 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:53:01.339+0000 7f2e84d66640 -1 reset not still connected to 0x560d368e4f70 2024-04-06T09:53:01.337 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:53:01.339+0000 7f2e84d66640 -1 reset not still connected to 0x560d36912ea0 2024-04-06T09:53:02.422 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:53:02.423+0000 7f2e84d66640 -1 reset not still connected to 0x560d27072820 2024-04-06T09:53:02.422 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:53:02.423+0000 7f2e84d66640 -1 reset not still connected to 0x560d27073860 2024-04-06T09:53:02.422 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:53:02.423+0000 7f2e84d66640 -1 reset not still connected to 0x560d2722e5b0 2024-04-06T09:53:02.422 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:53:02.423+0000 7f2e84d66640 -1 reset not still connected to 0x560d27567380 2024-04-06T09:53:02.423 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:53:02.423+0000 7f2e84d66640 -1 reset not still connected to 0x560d27839380 2024-04-06T09:53:02.423 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:53:02.423+0000 7f2e84d66640 -1 reset not still connected to 0x560d27839930 2024-04-06T09:53:02.423 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:53:02.423+0000 7f2e84d66640 -1 reset not still connected to 0x560d27c3edd0 2024-04-06T09:53:02.423 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:53:02.423+0000 7f2e84d66640 -1 reset not still connected to 0x560d27c3f2b0 2024-04-06T09:53:02.423 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:53:02.423+0000 7f2e84d66640 -1 reset not still connected to 0x560d294908f0 2024-04-06T09:53:02.425 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:53:02.423+0000 7f2e84d66640 -1 reset not still connected to 0x560d29cd8f70 2024-04-06T09:53:02.425 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:53:02.423+0000 7f2e84d66640 -1 reset not still connected to 0x560d29d5b520 2024-04-06T09:53:02.425 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:53:02.423+0000 7f2e84d66640 -1 reset not still connected to 0x560d2a17d450 2024-04-06T09:53:02.425 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:53:02.423+0000 7f2e84d66640 -1 reset not still connected to 0x560d2a17d6c0 2024-04-06T09:53:02.425 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:53:02.423+0000 7f2e84d66640 -1 reset not still connected to 0x560d2a2336c0 2024-04-06T09:53:02.425 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:53:02.423+0000 7f2e84d66640 -1 reset not still connected to 0x560d368e4f70 2024-04-06T09:53:02.425 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:53:02.423+0000 7f2e84d66640 -1 reset not still connected to 0x560d36912ea0 2024-04-06T09:53:02.610 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:200 IOs, 43304960 bytes 2024-04-06T09:53:03.593 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:53:03.595+0000 7f2e84d66640 -1 reset not still connected to 0x560d27567380 2024-04-06T09:53:03.594 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:53:03.595+0000 7f2e84d66640 -1 reset not still connected to 0x560d27839380 2024-04-06T09:53:03.594 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:53:03.595+0000 7f2e84d66640 -1 reset not still connected to 0x560d27839930 2024-04-06T09:53:03.597 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:53:03.595+0000 7f2e84d66640 -1 reset not still connected to 0x560d27c3edd0 2024-04-06T09:53:03.602 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:53:03.599+0000 7f2e84d66640 -1 reset not still connected to 0x560d27c3f2b0 2024-04-06T09:53:03.603 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:53:03.599+0000 7f2e84d66640 -1 reset not still connected to 0x560d294908f0 2024-04-06T09:53:03.603 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:53:03.603+0000 7f2e84d66640 -1 reset not still connected to 0x560d29cd8f70 2024-04-06T09:53:03.603 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:53:03.603+0000 7f2e84d66640 -1 reset not still connected to 0x560d29d5b520 2024-04-06T09:53:03.603 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:53:03.603+0000 7f2e84d66640 -1 reset not still connected to 0x560d2a17d450 2024-04-06T09:53:03.603 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:53:03.603+0000 7f2e84d66640 -1 reset not still connected to 0x560d2a17d6c0 2024-04-06T09:53:03.603 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:53:03.603+0000 7f2e84d66640 -1 reset not still connected to 0x560d2a2336c0 2024-04-06T09:53:03.603 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:53:03.607+0000 7f2e84d66640 -1 reset not still connected to 0x560d368e4f70 2024-04-06T09:53:03.604 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:53:03.607+0000 7f2e84d66640 -1 reset not still connected to 0x560d36912ea0 2024-04-06T09:53:04.163 INFO:tasks.ceph.cluster2.osd.2.smithi192.stderr:2024-04-06T09:53:04.164+0000 7fa534aba640 -1 reset not still connected to 0x559338f41040 2024-04-06T09:53:04.164 INFO:tasks.ceph.cluster2.osd.2.smithi192.stderr:2024-04-06T09:53:04.168+0000 7fa534aba640 -1 reset not still connected to 0x55933ce91110 2024-04-06T09:53:04.164 INFO:tasks.ceph.cluster2.osd.2.smithi192.stderr:2024-04-06T09:53:04.168+0000 7fa534aba640 -1 reset not still connected to 0x55933ce91520 2024-04-06T09:53:04.164 INFO:tasks.ceph.cluster2.osd.2.smithi192.stderr:2024-04-06T09:53:04.168+0000 7fa534aba640 -1 reset not still connected to 0x55935bfc9d40 2024-04-06T09:53:04.199 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:300 IOs, 61696000 bytes 2024-04-06T09:53:04.711 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ error_code=124 2024-04-06T09:53:04.711 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set -e 2024-04-06T09:53:04.711 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 124 -eq 124 ']' 2024-04-06T09:53:04.711 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:53:04.711 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-06T09:53:04.712 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ image=image_16 2024-04-06T09:53:04.712 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ create_image_and_enable_mirror cluster2 mirror image_16 journal 128M 2024-04-06T09:53:04.712 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-04-06T09:53:04.712 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:53:04.712 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:53:04.712 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:53:04.712 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_16 2024-04-06T09:53:04.712 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:53:04.712 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local mode=journal 2024-04-06T09:53:04.712 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n journal ']' 2024-04-06T09:53:04.712 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:53:04.712 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ create_image cluster2 mirror image_16 128M 2024-04-06T09:53:04.712 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-04-06T09:53:04.712 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:53:04.712 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:53:04.713 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:53:04.713 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_16 2024-04-06T09:53:04.713 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:53:04.713 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local size=128 2024-04-06T09:53:04.713 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n 128M ']' 2024-04-06T09:53:04.713 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ size=128M 2024-04-06T09:53:04.713 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:53:04.713 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,journaling mirror/image_16 2024-04-06T09:53:04.792 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' pool = image ']' 2024-04-06T09:53:04.792 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' mirror = mirror_parent ']' 2024-04-06T09:53:04.792 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n '' ']' 2024-04-06T09:53:04.792 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ stress_write_image cluster2 mirror image_16 2024-04-06T09:53:04.792 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-04-06T09:53:04.792 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:53:04.792 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_16 2024-04-06T09:53:04.792 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-04-06T09:53:04.793 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local duration=6 2024-04-06T09:53:04.794 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set +e 2024-04-06T09:53:04.794 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ timeout 6s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_16 --debug-rbd=20 --debug-journaler=20 2024-04-06T09:53:05.625 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:100 IOs, 9497600 bytes 2024-04-06T09:53:07.905 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:53:07.903+0000 7f2e84d66640 -1 reset not still connected to 0x560d29d5b520 2024-04-06T09:53:07.905 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:53:07.903+0000 7f2e84d66640 -1 reset not still connected to 0x560d2a17d450 2024-04-06T09:53:07.905 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:53:07.903+0000 7f2e84d66640 -1 reset not still connected to 0x560d2a17d6c0 2024-04-06T09:53:07.905 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:53:07.903+0000 7f2e84d66640 -1 reset not still connected to 0x560d2a2336c0 2024-04-06T09:53:07.906 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:53:07.907+0000 7f2e84d66640 -1 reset not still connected to 0x560d368e4f70 2024-04-06T09:53:07.923 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:53:07.923+0000 7f2e84d66640 -1 reset not still connected to 0x560d36912ea0 2024-04-06T09:53:08.091 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-04-06T09:53:08.091+0000 7fd1b6210640 -1 reset not still connected to 0x55ef9c103930 2024-04-06T09:53:08.091 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-04-06T09:53:08.091+0000 7fd1b6210640 -1 reset not still connected to 0x55ef9e698c30 2024-04-06T09:53:08.091 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-04-06T09:53:08.091+0000 7fd1b6210640 -1 reset not still connected to 0x55ef9eee21a0 2024-04-06T09:53:09.814 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:200 IOs, 63195136 bytes 2024-04-06T09:53:10.805 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ error_code=124 2024-04-06T09:53:10.805 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set -e 2024-04-06T09:53:10.805 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 124 -eq 124 ']' 2024-04-06T09:53:10.805 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:53:10.805 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-06T09:53:10.806 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ image=image_17 2024-04-06T09:53:10.806 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ create_image_and_enable_mirror cluster2 mirror image_17 journal 128M 2024-04-06T09:53:10.806 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-04-06T09:53:10.806 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:53:10.806 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:53:10.806 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:53:10.806 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_17 2024-04-06T09:53:10.806 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:53:10.806 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local mode=journal 2024-04-06T09:53:10.806 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n journal ']' 2024-04-06T09:53:10.806 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:53:10.806 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ create_image cluster2 mirror image_17 128M 2024-04-06T09:53:10.806 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-04-06T09:53:10.807 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:53:10.807 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:53:10.807 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:53:10.807 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_17 2024-04-06T09:53:10.807 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:53:10.807 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local size=128 2024-04-06T09:53:10.807 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n 128M ']' 2024-04-06T09:53:10.807 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ size=128M 2024-04-06T09:53:10.807 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:53:10.807 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,journaling mirror/image_17 2024-04-06T09:53:10.895 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' pool = image ']' 2024-04-06T09:53:10.895 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' mirror = mirror_parent ']' 2024-04-06T09:53:10.895 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n '' ']' 2024-04-06T09:53:10.895 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ stress_write_image cluster2 mirror image_17 2024-04-06T09:53:10.895 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-04-06T09:53:10.895 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:53:10.895 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_17 2024-04-06T09:53:10.895 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-04-06T09:53:10.896 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local duration=5 2024-04-06T09:53:10.896 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set +e 2024-04-06T09:53:10.896 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ timeout 5s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_17 --debug-rbd=20 --debug-journaler=20 2024-04-06T09:53:15.921 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ error_code=124 2024-04-06T09:53:15.922 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set -e 2024-04-06T09:53:15.922 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 124 -eq 124 ']' 2024-04-06T09:53:15.922 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:53:15.922 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-06T09:53:15.922 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ image=image_18 2024-04-06T09:53:15.922 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ create_image_and_enable_mirror cluster2 mirror image_18 journal 128M 2024-04-06T09:53:15.922 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-04-06T09:53:15.922 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:53:15.922 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:53:15.922 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:53:15.922 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_18 2024-04-06T09:53:15.922 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:53:15.923 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local mode=journal 2024-04-06T09:53:15.923 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n journal ']' 2024-04-06T09:53:15.923 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:53:15.923 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ create_image cluster2 mirror image_18 128M 2024-04-06T09:53:15.923 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-04-06T09:53:15.923 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:53:15.923 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:53:15.925 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:53:15.925 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_18 2024-04-06T09:53:15.925 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:53:15.926 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local size=128 2024-04-06T09:53:15.926 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n 128M ']' 2024-04-06T09:53:15.926 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ size=128M 2024-04-06T09:53:15.926 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:53:15.926 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,journaling mirror/image_18 2024-04-06T09:53:15.986 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' pool = image ']' 2024-04-06T09:53:15.986 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' mirror = mirror_parent ']' 2024-04-06T09:53:15.987 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n '' ']' 2024-04-06T09:53:15.987 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ stress_write_image cluster2 mirror image_18 2024-04-06T09:53:15.987 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-04-06T09:53:15.987 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:53:15.987 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_18 2024-04-06T09:53:15.987 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-04-06T09:53:15.988 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local duration=12 2024-04-06T09:53:15.988 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set +e 2024-04-06T09:53:15.988 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ timeout 12s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_18 --debug-rbd=20 --debug-journaler=20 2024-04-06T09:53:16.278 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:100 IOs, 598016 bytes 2024-04-06T09:53:18.695 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:200 IOs, 31933440 bytes 2024-04-06T09:53:18.878 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-04-06T09:53:18.879+0000 7fd1b6210640 -1 reset not still connected to 0x55ef9bbe9ee0 2024-04-06T09:53:18.878 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-04-06T09:53:18.879+0000 7fd1b6210640 -1 reset not still connected to 0x55ef9bd5d520 2024-04-06T09:53:18.878 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-04-06T09:53:18.879+0000 7fd1b6210640 -1 reset not still connected to 0x55ef9c103930 2024-04-06T09:53:18.878 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-04-06T09:53:18.879+0000 7fd1b6210640 -1 reset not still connected to 0x55ef9e698c30 2024-04-06T09:53:19.131 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-04-06T09:53:19.131+0000 7fd1b6210640 -1 reset not still connected to 0x55ef9ac60270 2024-04-06T09:53:19.131 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-04-06T09:53:19.131+0000 7fd1b6210640 -1 reset not still connected to 0x55ef9bbe9ee0 2024-04-06T09:53:19.131 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-04-06T09:53:19.131+0000 7fd1b6210640 -1 reset not still connected to 0x55ef9bd5d520 2024-04-06T09:53:19.136 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-04-06T09:53:19.139+0000 7fd1b6210640 -1 reset not still connected to 0x55ef9bd86820 2024-04-06T09:53:19.136 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-04-06T09:53:19.139+0000 7fd1b6210640 -1 reset not still connected to 0x55ef9c103110 2024-04-06T09:53:19.136 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-04-06T09:53:19.139+0000 7fd1b6210640 -1 reset not still connected to 0x55ef9c103930 2024-04-06T09:53:19.136 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-04-06T09:53:19.139+0000 7fd1b6210640 -1 reset not still connected to 0x55ef9e698c30 2024-04-06T09:53:19.136 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-04-06T09:53:19.139+0000 7fd1b6210640 -1 reset not still connected to 0x55ef9e88f520 2024-04-06T09:53:19.136 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-04-06T09:53:19.139+0000 7fd1b6210640 -1 reset not still connected to 0x55ef9eee21a0 2024-04-06T09:53:22.002 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:300 IOs, 89295872 bytes 2024-04-06T09:53:22.502 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:53:22.498+0000 7f2e84d66640 -1 reset not still connected to 0x560d27567380 2024-04-06T09:53:22.502 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:53:22.498+0000 7f2e84d66640 -1 reset not still connected to 0x560d27839380 2024-04-06T09:53:22.502 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:53:22.502+0000 7f2e84d66640 -1 reset not still connected to 0x560d27839930 2024-04-06T09:53:22.505 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:53:22.506+0000 7f2e84d66640 -1 reset not still connected to 0x560d27c3edd0 2024-04-06T09:53:22.507 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:53:22.506+0000 7f2e84d66640 -1 reset not still connected to 0x560d27c3f2b0 2024-04-06T09:53:22.507 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:53:22.506+0000 7f2e84d66640 -1 reset not still connected to 0x560d281875f0 2024-04-06T09:53:22.507 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:53:22.506+0000 7f2e84d66640 -1 reset not still connected to 0x560d294908f0 2024-04-06T09:53:22.507 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:53:22.506+0000 7f2e84d66640 -1 reset not still connected to 0x560d29cd8f70 2024-04-06T09:53:22.507 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:53:22.506+0000 7f2e84d66640 -1 reset not still connected to 0x560d29d5b520 2024-04-06T09:53:22.507 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:53:22.506+0000 7f2e84d66640 -1 reset not still connected to 0x560d2a17d450 2024-04-06T09:53:22.507 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:53:22.506+0000 7f2e84d66640 -1 reset not still connected to 0x560d2a17d6c0 2024-04-06T09:53:22.507 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:53:22.506+0000 7f2e84d66640 -1 reset not still connected to 0x560d2a204b60 2024-04-06T09:53:22.508 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:53:22.506+0000 7f2e84d66640 -1 reset not still connected to 0x560d2a2336c0 2024-04-06T09:53:22.508 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:53:22.506+0000 7f2e84d66640 -1 reset not still connected to 0x560d368e4f70 2024-04-06T09:53:22.508 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:53:22.506+0000 7f2e84d66640 -1 reset not still connected to 0x560d36912ea0 2024-04-06T09:53:23.537 INFO:tasks.ceph.cluster2.osd.1.smithi192.stderr:2024-04-06T09:53:23.539+0000 7f0de0719640 -1 reset not still connected to 0x55bfb063cf70 2024-04-06T09:53:24.720 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:400 IOs, 104617984 bytes 2024-04-06T09:53:25.437 INFO:tasks.ceph.cluster2.osd.1.smithi192.stderr:2024-04-06T09:53:25.439+0000 7f0dccaa8640 -1 bdev(0x55bfa195ce00 /var/lib/ceph/osd/cluster2-1/block) aio_submit retries 1 2024-04-06T09:53:26.123 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-04-06T09:53:26.122+0000 7fd1b6210640 -1 reset not still connected to 0x55ef9ac60270 2024-04-06T09:53:26.124 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-04-06T09:53:26.122+0000 7fd1b6210640 -1 reset not still connected to 0x55ef9bbe9ee0 2024-04-06T09:53:26.124 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-04-06T09:53:26.122+0000 7fd1b6210640 -1 reset not still connected to 0x55ef9bd5c9c0 2024-04-06T09:53:26.124 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-04-06T09:53:26.122+0000 7fd1b6210640 -1 reset not still connected to 0x55ef9bd5d520 2024-04-06T09:53:26.124 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-04-06T09:53:26.122+0000 7fd1b6210640 -1 reset not still connected to 0x55ef9bd86820 2024-04-06T09:53:26.124 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-04-06T09:53:26.122+0000 7fd1b6210640 -1 reset not still connected to 0x55ef9c103110 2024-04-06T09:53:26.124 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-04-06T09:53:26.122+0000 7fd1b6210640 -1 reset not still connected to 0x55ef9c103930 2024-04-06T09:53:26.124 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-04-06T09:53:26.122+0000 7fd1b6210640 -1 reset not still connected to 0x55ef9e698c30 2024-04-06T09:53:26.124 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-04-06T09:53:26.122+0000 7fd1b6210640 -1 reset not still connected to 0x55ef9e88f520 2024-04-06T09:53:26.124 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-04-06T09:53:26.122+0000 7fd1b6210640 -1 reset not still connected to 0x55ef9eee21a0 2024-04-06T09:53:26.274 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:500 IOs, 122568704 bytes 2024-04-06T09:53:26.807 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:53:26.806+0000 7f0965192640 -1 reset not still connected to 0x5632a5e9dee0 2024-04-06T09:53:26.808 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:53:26.810+0000 7f0965192640 -1 reset not still connected to 0x5632abc81d40 2024-04-06T09:53:26.808 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:53:26.810+0000 7f0965192640 -1 reset not still connected to 0x5632e16851e0 2024-04-06T09:53:27.996 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ error_code=124 2024-04-06T09:53:27.996 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set -e 2024-04-06T09:53:27.996 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 124 -eq 124 ']' 2024-04-06T09:53:27.996 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:53:27.996 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-06T09:53:27.996 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ image=image_19 2024-04-06T09:53:27.996 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ create_image_and_enable_mirror cluster2 mirror image_19 journal 128M 2024-04-06T09:53:27.996 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-04-06T09:53:27.996 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:53:27.996 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:53:27.996 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:53:27.996 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_19 2024-04-06T09:53:27.996 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:53:27.997 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local mode=journal 2024-04-06T09:53:27.997 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n journal ']' 2024-04-06T09:53:27.997 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:53:27.997 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ create_image cluster2 mirror image_19 128M 2024-04-06T09:53:27.997 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-04-06T09:53:27.997 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:53:27.997 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:53:27.997 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:53:27.997 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_19 2024-04-06T09:53:27.997 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:53:27.997 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local size=128 2024-04-06T09:53:27.997 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n 128M ']' 2024-04-06T09:53:27.997 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ size=128M 2024-04-06T09:53:27.997 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:53:27.997 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,journaling mirror/image_19 2024-04-06T09:53:28.081 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' pool = image ']' 2024-04-06T09:53:28.081 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' mirror = mirror_parent ']' 2024-04-06T09:53:28.081 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n '' ']' 2024-04-06T09:53:28.081 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ stress_write_image cluster2 mirror image_19 2024-04-06T09:53:28.081 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-04-06T09:53:28.081 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:53:28.081 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_19 2024-04-06T09:53:28.081 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-04-06T09:53:28.082 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local duration=10 2024-04-06T09:53:28.082 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set +e 2024-04-06T09:53:28.082 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ timeout 10s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_19 --debug-rbd=20 --debug-journaler=20 2024-04-06T09:53:30.642 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:100 IOs, 55342080 bytes 2024-04-06T09:53:31.121 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:53:31.122+0000 7f2e84d66640 -1 reset not still connected to 0x560d27567380 2024-04-06T09:53:31.121 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:53:31.122+0000 7f2e84d66640 -1 reset not still connected to 0x560d27839380 2024-04-06T09:53:31.121 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:53:31.122+0000 7f2e84d66640 -1 reset not still connected to 0x560d27c3edd0 2024-04-06T09:53:31.121 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:53:31.122+0000 7f2e84d66640 -1 reset not still connected to 0x560d27c3f2b0 2024-04-06T09:53:31.121 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:53:31.122+0000 7f2e84d66640 -1 reset not still connected to 0x560d294908f0 2024-04-06T09:53:31.127 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:53:31.126+0000 7f2e84d66640 -1 reset not still connected to 0x560d29cd8f70 2024-04-06T09:53:31.127 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:53:31.126+0000 7f2e84d66640 -1 reset not still connected to 0x560d29d5b520 2024-04-06T09:53:31.127 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:53:31.126+0000 7f2e84d66640 -1 reset not still connected to 0x560d2a17d450 2024-04-06T09:53:31.127 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:53:31.126+0000 7f2e84d66640 -1 reset not still connected to 0x560d2a17d6c0 2024-04-06T09:53:31.132 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:53:31.130+0000 7f2e84d66640 -1 reset not still connected to 0x560d2a2336c0 2024-04-06T09:53:31.132 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:53:31.130+0000 7f2e84d66640 -1 reset not still connected to 0x560d368e4f70 2024-04-06T09:53:31.132 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:53:31.130+0000 7f2e84d66640 -1 reset not still connected to 0x560d36912ea0 2024-04-06T09:53:34.234 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:200 IOs, 94566400 bytes 2024-04-06T09:53:36.461 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:300 IOs, 134049792 bytes 2024-04-06T09:53:38.091 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ error_code=124 2024-04-06T09:53:38.091 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set -e 2024-04-06T09:53:38.091 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 124 -eq 124 ']' 2024-04-06T09:53:38.091 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:53:38.092 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-06T09:53:38.092 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ image=image_20 2024-04-06T09:53:38.092 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ create_image_and_enable_mirror cluster2 mirror image_20 journal 128M 2024-04-06T09:53:38.092 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-04-06T09:53:38.092 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:53:38.092 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:53:38.092 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:53:38.092 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_20 2024-04-06T09:53:38.092 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:53:38.092 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local mode=journal 2024-04-06T09:53:38.092 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n journal ']' 2024-04-06T09:53:38.092 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:53:38.092 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ create_image cluster2 mirror image_20 128M 2024-04-06T09:53:38.092 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-04-06T09:53:38.092 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:53:38.093 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:53:38.093 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:53:38.093 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_20 2024-04-06T09:53:38.093 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:53:38.093 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local size=128 2024-04-06T09:53:38.093 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n 128M ']' 2024-04-06T09:53:38.093 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ size=128M 2024-04-06T09:53:38.093 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:53:38.093 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,journaling mirror/image_20 2024-04-06T09:53:38.176 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' pool = image ']' 2024-04-06T09:53:38.176 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' mirror = mirror_parent ']' 2024-04-06T09:53:38.176 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n '' ']' 2024-04-06T09:53:38.176 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ stress_write_image cluster2 mirror image_20 2024-04-06T09:53:38.176 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-04-06T09:53:38.176 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:53:38.176 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_20 2024-04-06T09:53:38.176 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-04-06T09:53:38.178 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local duration=12 2024-04-06T09:53:38.178 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set +e 2024-04-06T09:53:38.179 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ timeout 12s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_20 --debug-rbd=20 --debug-journaler=20 2024-04-06T09:53:40.007 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:100 IOs, 39911424 bytes 2024-04-06T09:53:40.155 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi192.stderr:2024-04-06T09:53:40.155+0000 7f3d293cf640 -1 rbd::mirror::InstanceWatcher: C_NotifyInstanceRequest: 0x56066544d8c0 finish: resending after timeout 2024-04-06T09:53:40.843 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:53:40.842+0000 7f2e84d66640 -1 reset not still connected to 0x560d294908f0 2024-04-06T09:53:40.845 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:53:40.846+0000 7f2e84d66640 -1 reset not still connected to 0x560d2a17d450 2024-04-06T09:53:40.845 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:53:40.846+0000 7f2e84d66640 -1 reset not still connected to 0x560d2a17d6c0 2024-04-06T09:53:40.845 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:53:40.846+0000 7f2e84d66640 -1 reset not still connected to 0x560d2a2336c0 2024-04-06T09:53:40.846 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:53:40.846+0000 7f2e84d66640 -1 reset not still connected to 0x560d368e4f70 2024-04-06T09:53:44.189 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:200 IOs, 75495424 bytes 2024-04-06T09:53:45.388 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-04-06T09:53:45.386+0000 7fd1b6210640 -1 reset not still connected to 0x55ef993061a0 2024-04-06T09:53:45.388 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-04-06T09:53:45.386+0000 7fd1b6210640 -1 reset not still connected to 0x55ef997761a0 2024-04-06T09:53:45.390 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-04-06T09:53:45.386+0000 7fd1b6210640 -1 reset not still connected to 0x55ef99776680 2024-04-06T09:53:45.390 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-04-06T09:53:45.386+0000 7fd1b6210640 -1 reset not still connected to 0x55ef99b88d00 2024-04-06T09:53:45.390 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-04-06T09:53:45.386+0000 7fd1b6210640 -1 reset not still connected to 0x55ef99b89ad0 2024-04-06T09:53:45.390 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-04-06T09:53:45.386+0000 7fd1b6210640 -1 reset not still connected to 0x55ef9ac60270 2024-04-06T09:53:45.392 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-04-06T09:53:45.390+0000 7fd1b6210640 -1 reset not still connected to 0x55ef9b657e10 2024-04-06T09:53:45.392 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-04-06T09:53:45.390+0000 7fd1b6210640 -1 reset not still connected to 0x55ef9bbe9ee0 2024-04-06T09:53:45.392 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-04-06T09:53:45.390+0000 7fd1b6210640 -1 reset not still connected to 0x55ef9bd5c9c0 2024-04-06T09:53:45.392 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-04-06T09:53:45.394+0000 7fd1b6210640 -1 reset not still connected to 0x55ef9bd5d520 2024-04-06T09:53:45.392 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-04-06T09:53:45.394+0000 7fd1b6210640 -1 reset not still connected to 0x55ef9bd86820 2024-04-06T09:53:45.392 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-04-06T09:53:45.394+0000 7fd1b6210640 -1 reset not still connected to 0x55ef9c103110 2024-04-06T09:53:45.392 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-04-06T09:53:45.394+0000 7fd1b6210640 -1 reset not still connected to 0x55ef9c103930 2024-04-06T09:53:45.392 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-04-06T09:53:45.394+0000 7fd1b6210640 -1 reset not still connected to 0x55ef9e698c30 2024-04-06T09:53:45.392 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-04-06T09:53:45.394+0000 7fd1b6210640 -1 reset not still connected to 0x55ef9e88f520 2024-04-06T09:53:45.392 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-04-06T09:53:45.394+0000 7fd1b6210640 -1 reset not still connected to 0x55ef9eee21a0 2024-04-06T09:53:47.011 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:300 IOs, 114635776 bytes 2024-04-06T09:53:48.086 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:400 IOs, 126104576 bytes 2024-04-06T09:53:48.845 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:53:48.846+0000 7f0965192640 -1 reset not still connected to 0x5632e16851e0 2024-04-06T09:53:50.196 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ error_code=124 2024-04-06T09:53:50.196 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set -e 2024-04-06T09:53:50.196 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 124 -eq 124 ']' 2024-04-06T09:53:50.196 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:53:50.196 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-06T09:53:50.196 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ image=image_21 2024-04-06T09:53:50.196 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ create_image_and_enable_mirror cluster2 mirror image_21 journal 128M 2024-04-06T09:53:50.196 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-04-06T09:53:50.197 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:53:50.197 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:53:50.197 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:53:50.197 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_21 2024-04-06T09:53:50.197 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:53:50.197 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local mode=journal 2024-04-06T09:53:50.197 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n journal ']' 2024-04-06T09:53:50.197 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:53:50.197 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ create_image cluster2 mirror image_21 128M 2024-04-06T09:53:50.197 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-04-06T09:53:50.197 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:53:50.197 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:53:50.198 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:53:50.198 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_21 2024-04-06T09:53:50.198 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:53:50.198 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local size=128 2024-04-06T09:53:50.198 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n 128M ']' 2024-04-06T09:53:50.198 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ size=128M 2024-04-06T09:53:50.198 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:53:50.198 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,journaling mirror/image_21 2024-04-06T09:53:50.275 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' pool = image ']' 2024-04-06T09:53:50.275 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' mirror = mirror_parent ']' 2024-04-06T09:53:50.275 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n '' ']' 2024-04-06T09:53:50.275 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ stress_write_image cluster2 mirror image_21 2024-04-06T09:53:50.275 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-04-06T09:53:50.275 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:53:50.275 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_21 2024-04-06T09:53:50.275 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-04-06T09:53:50.277 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local duration=11 2024-04-06T09:53:50.277 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set +e 2024-04-06T09:53:50.277 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ timeout 11s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_21 --debug-rbd=20 --debug-journaler=20 2024-04-06T09:53:50.549 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-04-06T09:53:50.550+0000 7fd1b6210640 -1 reset not still connected to 0x55ef9c103930 2024-04-06T09:53:50.550 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-04-06T09:53:50.550+0000 7fd1b6210640 -1 reset not still connected to 0x55ef9e698c30 2024-04-06T09:53:50.550 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-04-06T09:53:50.550+0000 7fd1b6210640 -1 reset not still connected to 0x55ef9eee21a0 2024-04-06T09:53:51.131 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:53:51.126+0000 7f2e84d66640 -1 reset not still connected to 0x560d2a17d6c0 2024-04-06T09:53:51.137 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:53:51.138+0000 7f2e84d66640 -1 reset not still connected to 0x560d2a2336c0 2024-04-06T09:53:51.137 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:53:51.138+0000 7f2e84d66640 -1 reset not still connected to 0x560d2a5705b0 2024-04-06T09:53:51.137 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:53:51.138+0000 7f2e84d66640 -1 reset not still connected to 0x560d368e4f70 2024-04-06T09:53:51.339 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:53:51.338+0000 7f2e84d66640 -1 reset not still connected to 0x560d2a17d450 2024-04-06T09:53:51.339 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:53:51.338+0000 7f2e84d66640 -1 reset not still connected to 0x560d2a17d6c0 2024-04-06T09:53:51.339 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:53:51.338+0000 7f2e84d66640 -1 reset not still connected to 0x560d2a204b60 2024-04-06T09:53:51.339 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:53:51.338+0000 7f2e84d66640 -1 reset not still connected to 0x560d2a2336c0 2024-04-06T09:53:51.340 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:53:51.338+0000 7f2e84d66640 -1 reset not still connected to 0x560d2a5705b0 2024-04-06T09:53:51.340 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:53:51.338+0000 7f2e84d66640 -1 reset not still connected to 0x560d368e4f70 2024-04-06T09:53:51.340 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:53:51.338+0000 7f2e84d66640 -1 reset not still connected to 0x560d36912ea0 2024-04-06T09:53:52.748 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-04-06T09:53:52.746+0000 7fd1b6210640 -1 reset not still connected to 0x55ef9e698c30 2024-04-06T09:53:52.751 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-04-06T09:53:52.746+0000 7fd1b6210640 -1 reset not still connected to 0x55ef9e88f520 2024-04-06T09:53:52.751 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-04-06T09:53:52.746+0000 7fd1b6210640 -1 reset not still connected to 0x55ef9eee21a0 2024-04-06T09:53:52.751 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-04-06T09:53:52.746+0000 7fd1b6210640 -1 reset not still connected to 0x55ef99776680 2024-04-06T09:53:52.752 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-04-06T09:53:52.746+0000 7fd1b6210640 -1 reset not still connected to 0x55ef99b88d00 2024-04-06T09:53:53.133 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-04-06T09:53:53.134+0000 7fd1b6210640 -1 reset not still connected to 0x55ef9bbe9ee0 2024-04-06T09:53:53.135 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-04-06T09:53:53.138+0000 7fd1b6210640 -1 reset not still connected to 0x55ef9bd5c9c0 2024-04-06T09:53:53.136 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-04-06T09:53:53.138+0000 7fd1b6210640 -1 reset not still connected to 0x55ef9bd5d520 2024-04-06T09:53:53.141 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-04-06T09:53:53.142+0000 7fd1b6210640 -1 reset not still connected to 0x55ef9c103930 2024-04-06T09:53:53.141 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-04-06T09:53:53.142+0000 7fd1b6210640 -1 reset not still connected to 0x55ef9e698c30 2024-04-06T09:53:53.141 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-04-06T09:53:53.142+0000 7fd1b6210640 -1 reset not still connected to 0x55ef9e88f520 2024-04-06T09:53:53.141 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-04-06T09:53:53.142+0000 7fd1b6210640 -1 reset not still connected to 0x55ef9eee21a0 2024-04-06T09:53:54.112 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-04-06T09:53:54.106+0000 7fd1b6210640 -1 reset not still connected to 0x55ef9bbe9ee0 2024-04-06T09:53:54.112 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-04-06T09:53:54.106+0000 7fd1b6210640 -1 reset not still connected to 0x55ef9bd5c9c0 2024-04-06T09:53:54.112 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-04-06T09:53:54.106+0000 7fd1b6210640 -1 reset not still connected to 0x55ef9bd5d520 2024-04-06T09:53:54.112 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-04-06T09:53:54.110+0000 7fd1b6210640 -1 reset not still connected to 0x55ef9c103930 2024-04-06T09:53:54.112 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-04-06T09:53:54.110+0000 7fd1b6210640 -1 reset not still connected to 0x55ef9e698c30 2024-04-06T09:53:54.113 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-04-06T09:53:54.114+0000 7fd1b6210640 -1 reset not still connected to 0x55ef9e88f520 2024-04-06T09:53:54.114 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-04-06T09:53:54.114+0000 7fd1b6210640 -1 reset not still connected to 0x55ef9eee21a0 2024-04-06T09:53:56.043 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-04-06T09:53:56.042+0000 7fd1b6210640 -1 reset not still connected to 0x55ef9c103930 2024-04-06T09:53:56.043 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-04-06T09:53:56.042+0000 7fd1b6210640 -1 reset not still connected to 0x55ef9e698c30 2024-04-06T09:53:56.044 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-04-06T09:53:56.046+0000 7fd1b6210640 -1 reset not still connected to 0x55ef9e88f520 2024-04-06T09:53:56.044 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-04-06T09:53:56.046+0000 7fd1b6210640 -1 reset not still connected to 0x55ef9eee21a0 2024-04-06T09:53:56.746 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:100 IOs, 108442624 bytes 2024-04-06T09:53:57.866 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-04-06T09:53:57.866+0000 7fd1b6210640 -1 reset not still connected to 0x55ef997761a0 2024-04-06T09:53:57.867 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-04-06T09:53:57.866+0000 7fd1b6210640 -1 reset not still connected to 0x55ef99776680 2024-04-06T09:53:57.872 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-04-06T09:53:57.870+0000 7fd1b6210640 -1 reset not still connected to 0x55ef99b88d00 2024-04-06T09:53:57.873 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-04-06T09:53:57.870+0000 7fd1b6210640 -1 reset not still connected to 0x55ef99b89ad0 2024-04-06T09:53:57.873 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-04-06T09:53:57.870+0000 7fd1b6210640 -1 reset not still connected to 0x55ef99ebf6c0 2024-04-06T09:53:57.873 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-04-06T09:53:57.870+0000 7fd1b6210640 -1 reset not still connected to 0x55ef9ac60270 2024-04-06T09:53:57.873 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-04-06T09:53:57.870+0000 7fd1b6210640 -1 reset not still connected to 0x55ef9bbe9ee0 2024-04-06T09:53:57.873 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-04-06T09:53:57.870+0000 7fd1b6210640 -1 reset not still connected to 0x55ef9bd5c9c0 2024-04-06T09:53:57.873 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-04-06T09:53:57.870+0000 7fd1b6210640 -1 reset not still connected to 0x55ef9bd5d520 2024-04-06T09:53:57.874 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-04-06T09:53:57.874+0000 7fd1b6210640 -1 reset not still connected to 0x55ef9c103110 2024-04-06T09:53:57.874 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-04-06T09:53:57.874+0000 7fd1b6210640 -1 reset not still connected to 0x55ef9c103930 2024-04-06T09:53:57.874 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-04-06T09:53:57.874+0000 7fd1b6210640 -1 reset not still connected to 0x55ef9e698c30 2024-04-06T09:53:57.874 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-04-06T09:53:57.874+0000 7fd1b6210640 -1 reset not still connected to 0x55ef9e88f520 2024-04-06T09:53:57.874 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-04-06T09:53:57.874+0000 7fd1b6210640 -1 reset not still connected to 0x55ef9eee21a0 2024-04-06T09:53:59.310 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:53:59.310+0000 7f0965192640 -1 reset not still connected to 0x5632a0ff5110 2024-04-06T09:53:59.310 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:53:59.310+0000 7f0965192640 -1 reset not still connected to 0x5632a14ac820 2024-04-06T09:53:59.310 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:53:59.310+0000 7f0965192640 -1 reset not still connected to 0x5632a1988000 2024-04-06T09:53:59.310 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:53:59.310+0000 7f0965192640 -1 reset not still connected to 0x5632a1b73ee0 2024-04-06T09:53:59.310 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:53:59.310+0000 7f0965192640 -1 reset not still connected to 0x5632a2793520 2024-04-06T09:53:59.310 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:53:59.310+0000 7f0965192640 -1 reset not still connected to 0x5632a30f0270 2024-04-06T09:53:59.311 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:53:59.310+0000 7f0965192640 -1 reset not still connected to 0x5632a30f0f70 2024-04-06T09:53:59.311 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:53:59.310+0000 7f0965192640 -1 reset not still connected to 0x5632a417e750 2024-04-06T09:53:59.311 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:53:59.310+0000 7f0965192640 -1 reset not still connected to 0x5632a4cda000 2024-04-06T09:53:59.311 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:53:59.310+0000 7f0965192640 -1 reset not still connected to 0x5632a5e9dee0 2024-04-06T09:53:59.311 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:53:59.310+0000 7f0965192640 -1 reset not still connected to 0x5632abc81d40 2024-04-06T09:53:59.311 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:53:59.310+0000 7f0965192640 -1 reset not still connected to 0x5632abcbe820 2024-04-06T09:53:59.311 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:53:59.310+0000 7f0965192640 -1 reset not still connected to 0x5632b9593110 2024-04-06T09:53:59.312 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:53:59.310+0000 7f0965192640 -1 reset not still connected to 0x5632cb3ac000 2024-04-06T09:53:59.312 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:53:59.310+0000 7f0965192640 -1 reset not still connected to 0x5632e16851e0 2024-04-06T09:54:00.971 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:200 IOs, 153721856 bytes 2024-04-06T09:54:01.286 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ error_code=124 2024-04-06T09:54:01.286 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set -e 2024-04-06T09:54:01.287 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 124 -eq 124 ']' 2024-04-06T09:54:01.287 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:54:01.287 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-06T09:54:01.287 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ image=image_22 2024-04-06T09:54:01.287 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ create_image_and_enable_mirror cluster2 mirror image_22 journal 128M 2024-04-06T09:54:01.287 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-04-06T09:54:01.287 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:54:01.287 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:54:01.287 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:54:01.287 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_22 2024-04-06T09:54:01.287 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:54:01.287 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local mode=journal 2024-04-06T09:54:01.287 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n journal ']' 2024-04-06T09:54:01.288 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:54:01.288 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ create_image cluster2 mirror image_22 128M 2024-04-06T09:54:01.288 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-04-06T09:54:01.288 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:54:01.288 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:54:01.288 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:54:01.288 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_22 2024-04-06T09:54:01.288 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:54:01.288 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local size=128 2024-04-06T09:54:01.288 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n 128M ']' 2024-04-06T09:54:01.288 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ size=128M 2024-04-06T09:54:01.288 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:54:01.288 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,journaling mirror/image_22 2024-04-06T09:54:01.348 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' pool = image ']' 2024-04-06T09:54:01.349 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' mirror = mirror_parent ']' 2024-04-06T09:54:01.349 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n '' ']' 2024-04-06T09:54:01.349 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ stress_write_image cluster2 mirror image_22 2024-04-06T09:54:01.349 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-04-06T09:54:01.349 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:54:01.349 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_22 2024-04-06T09:54:01.349 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-04-06T09:54:01.350 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local duration=11 2024-04-06T09:54:01.350 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set +e 2024-04-06T09:54:01.350 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ timeout 11s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_22 --debug-rbd=20 --debug-journaler=20 2024-04-06T09:54:01.432 INFO:tasks.ceph.cluster2.osd.2.smithi192.stderr:2024-04-06T09:54:01.430+0000 7fa524650640 -1 bdev(0x55933350ee00 /var/lib/ceph/osd/cluster2-2/block) aio_submit retries 1 2024-04-06T09:54:01.432 INFO:tasks.ceph.cluster2.osd.2.smithi192.stderr:2024-04-06T09:54:01.434+0000 7fa526e55640 -1 bdev(0x55933350ee00 /var/lib/ceph/osd/cluster2-2/block) aio_submit retries 5 2024-04-06T09:54:03.048 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:100 IOs, 37573632 bytes 2024-04-06T09:54:03.240 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-04-06T09:54:03.242+0000 7fd1b6210640 -1 reset not still connected to 0x55ef9e698c30 2024-04-06T09:54:03.251 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-04-06T09:54:03.254+0000 7fd1b6210640 -1 reset not still connected to 0x55ef9e88f520 2024-04-06T09:54:03.251 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-04-06T09:54:03.254+0000 7fd1b6210640 -1 reset not still connected to 0x55ef9eee21a0 2024-04-06T09:54:03.303 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:54:03.298+0000 7f0965192640 -1 reset not still connected to 0x5632cb3ac000 2024-04-06T09:54:03.304 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:54:03.298+0000 7f0965192640 -1 reset not still connected to 0x5632e16851e0 2024-04-06T09:54:05.180 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:200 IOs, 60770304 bytes 2024-04-06T09:54:05.429 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.0 with skip (0.8986505384206669) > weight (0.25) 2024-04-06T09:54:05.429 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.1 with skip (0.6804173030381009) > weight (0.25) 2024-04-06T09:54:05.429 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.2 with skip (0.3760675005025498) > weight (0.25) 2024-04-06T09:54:05.429 INFO:tasks.rbd_mirror_thrash:kill cluster2.client.mirror.3 2024-04-06T09:54:05.429 INFO:tasks.rbd_mirror.cluster2.client.mirror.3:Sent signal 15 2024-04-06T09:54:05.430 INFO:tasks.rbd_mirror_thrash:waiting for 5 secs before reviving daemons 2024-04-06T09:54:05.430 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi192.stderr:2024-04-06T09:54:05.430+0000 7f9515239640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.3 (PID: 130219) UID: 1000 2024-04-06T09:54:06.205 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:54:06.206+0000 7f0965192640 -1 reset not still connected to 0x5632a1988000 2024-04-06T09:54:06.205 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:54:06.206+0000 7f0965192640 -1 reset not still connected to 0x5632a1b73ee0 2024-04-06T09:54:06.209 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:54:06.210+0000 7f0965192640 -1 reset not still connected to 0x5632a2793520 2024-04-06T09:54:06.209 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:54:06.210+0000 7f0965192640 -1 reset not still connected to 0x5632a30f0270 2024-04-06T09:54:06.209 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:54:06.210+0000 7f0965192640 -1 reset not still connected to 0x5632a30f0f70 2024-04-06T09:54:06.210 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:54:06.210+0000 7f0965192640 -1 reset not still connected to 0x5632a417e750 2024-04-06T09:54:06.210 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:54:06.210+0000 7f0965192640 -1 reset not still connected to 0x5632a4cda000 2024-04-06T09:54:06.210 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:54:06.210+0000 7f0965192640 -1 reset not still connected to 0x5632a5e9dee0 2024-04-06T09:54:06.210 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:54:06.210+0000 7f0965192640 -1 reset not still connected to 0x5632abc81d40 2024-04-06T09:54:06.210 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:54:06.210+0000 7f0965192640 -1 reset not still connected to 0x5632abcbe820 2024-04-06T09:54:06.210 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:54:06.210+0000 7f0965192640 -1 reset not still connected to 0x5632b9593110 2024-04-06T09:54:06.210 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:54:06.210+0000 7f0965192640 -1 reset not still connected to 0x5632cb3ac000 2024-04-06T09:54:06.210 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:54:06.210+0000 7f0965192640 -1 reset not still connected to 0x5632e16851e0 2024-04-06T09:54:06.796 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:54:06.794+0000 7f0965192640 -1 reset not still connected to 0x5632a1988000 2024-04-06T09:54:06.797 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:54:06.794+0000 7f0965192640 -1 reset not still connected to 0x5632a1b73ee0 2024-04-06T09:54:06.797 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:54:06.794+0000 7f0965192640 -1 reset not still connected to 0x5632a2793520 2024-04-06T09:54:06.797 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:54:06.794+0000 7f0965192640 -1 reset not still connected to 0x5632a30f0270 2024-04-06T09:54:06.797 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:54:06.794+0000 7f0965192640 -1 reset not still connected to 0x5632a417e750 2024-04-06T09:54:06.797 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:54:06.794+0000 7f0965192640 -1 reset not still connected to 0x5632a4cda000 2024-04-06T09:54:06.797 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:54:06.794+0000 7f0965192640 -1 reset not still connected to 0x5632a5e9dee0 2024-04-06T09:54:06.797 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:54:06.794+0000 7f0965192640 -1 reset not still connected to 0x5632abc81d40 2024-04-06T09:54:06.797 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:54:06.798+0000 7f0965192640 -1 reset not still connected to 0x5632abcbe820 2024-04-06T09:54:06.797 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:54:06.798+0000 7f0965192640 -1 reset not still connected to 0x5632b9593110 2024-04-06T09:54:06.797 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:54:06.798+0000 7f0965192640 -1 reset not still connected to 0x5632cb3ac000 2024-04-06T09:54:06.797 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:54:06.798+0000 7f0965192640 -1 reset not still connected to 0x5632e16851e0 2024-04-06T09:54:08.106 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:300 IOs, 120857600 bytes 2024-04-06T09:54:09.420 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:400 IOs, 163657728 bytes 2024-04-06T09:54:10.431 INFO:tasks.rbd_mirror_thrash:waiting for cluster2.client.mirror.3 2024-04-06T09:54:10.431 INFO:teuthology.orchestra.run:waiting for 600 2024-04-06T09:54:10.431 INFO:tasks.rbd_mirror_thrash:reviving cluster2.client.mirror.3 2024-04-06T09:54:10.432 INFO:tasks.rbd_mirror.cluster2.client.mirror.3:Restarting daemon 2024-04-06T09:54:10.432 DEBUG:teuthology.orchestra.run.smithi192:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.3 2024-04-06T09:54:10.434 INFO:tasks.rbd_mirror.cluster2.client.mirror.3:Started 2024-04-06T09:54:10.434 INFO:tasks.rbd_mirror_thrash:waiting for 113 secs before thrashing 2024-04-06T09:54:10.539 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-04-06T09:54:10.538+0000 7fd1b6210640 -1 reset not still connected to 0x55ef9c103110 2024-04-06T09:54:10.543 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-04-06T09:54:10.542+0000 7fd1b6210640 -1 reset not still connected to 0x55ef9c103930 2024-04-06T09:54:10.543 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-04-06T09:54:10.542+0000 7fd1b6210640 -1 reset not still connected to 0x55ef9e698c30 2024-04-06T09:54:10.543 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-04-06T09:54:10.542+0000 7fd1b6210640 -1 reset not still connected to 0x55ef9e88f520 2024-04-06T09:54:10.543 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-04-06T09:54:10.542+0000 7fd1b6210640 -1 reset not still connected to 0x55ef9eee21a0 2024-04-06T09:54:12.359 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ error_code=124 2024-04-06T09:54:12.359 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set -e 2024-04-06T09:54:12.359 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 124 -eq 124 ']' 2024-04-06T09:54:12.359 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:54:12.359 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-06T09:54:12.359 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ image=image_23 2024-04-06T09:54:12.360 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ create_image_and_enable_mirror cluster2 mirror image_23 journal 128M 2024-04-06T09:54:12.360 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-04-06T09:54:12.360 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:54:12.360 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:54:12.360 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:54:12.360 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_23 2024-04-06T09:54:12.360 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:54:12.360 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local mode=journal 2024-04-06T09:54:12.360 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n journal ']' 2024-04-06T09:54:12.360 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:54:12.360 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ create_image cluster2 mirror image_23 128M 2024-04-06T09:54:12.360 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-04-06T09:54:12.360 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:54:12.360 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:54:12.361 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:54:12.361 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_23 2024-04-06T09:54:12.361 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:54:12.361 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local size=128 2024-04-06T09:54:12.361 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n 128M ']' 2024-04-06T09:54:12.361 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ size=128M 2024-04-06T09:54:12.361 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:54:12.361 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,journaling mirror/image_23 2024-04-06T09:54:12.425 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' pool = image ']' 2024-04-06T09:54:12.426 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' mirror = mirror_parent ']' 2024-04-06T09:54:12.426 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n '' ']' 2024-04-06T09:54:12.426 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ stress_write_image cluster2 mirror image_23 2024-04-06T09:54:12.426 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-04-06T09:54:12.426 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:54:12.426 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_23 2024-04-06T09:54:12.426 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-04-06T09:54:12.427 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local duration=7 2024-04-06T09:54:12.427 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set +e 2024-04-06T09:54:12.427 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ timeout 7s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_23 --debug-rbd=20 --debug-journaler=20 2024-04-06T09:54:12.635 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.0 with skip (0.6449879747779041) > weight (0.25) 2024-04-06T09:54:12.635 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.1 with skip (0.3524768310192944) > weight (0.25) 2024-04-06T09:54:12.635 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.2 with skip (0.7599643070102768) > weight (0.25) 2024-04-06T09:54:12.635 INFO:tasks.rbd_mirror_thrash:kill cluster1.client.mirror.3 2024-04-06T09:54:12.635 INFO:tasks.rbd_mirror.cluster1.client.mirror.3:Sent signal 15 2024-04-06T09:54:12.635 INFO:tasks.rbd_mirror_thrash:waiting for 6 secs before reviving daemons 2024-04-06T09:54:12.636 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi192.stderr:2024-04-06T09:54:12.638+0000 7f0b5c846640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.3 (PID: 129820) UID: 1000 2024-04-06T09:54:13.517 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-04-06T09:54:13.518+0000 7fd1b6210640 -1 reset not still connected to 0x55ef9ac60270 2024-04-06T09:54:13.517 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-04-06T09:54:13.518+0000 7fd1b6210640 -1 reset not still connected to 0x55ef9bbe9ee0 2024-04-06T09:54:13.518 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-04-06T09:54:13.518+0000 7fd1b6210640 -1 reset not still connected to 0x55ef9bd5c9c0 2024-04-06T09:54:13.518 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-04-06T09:54:13.518+0000 7fd1b6210640 -1 reset not still connected to 0x55ef9bd5d520 2024-04-06T09:54:13.519 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-04-06T09:54:13.518+0000 7fd1b6210640 -1 reset not still connected to 0x55ef9c103110 2024-04-06T09:54:13.519 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-04-06T09:54:13.518+0000 7fd1b6210640 -1 reset not still connected to 0x55ef9c103930 2024-04-06T09:54:13.519 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-04-06T09:54:13.518+0000 7fd1b6210640 -1 reset not still connected to 0x55ef9e698c30 2024-04-06T09:54:13.520 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-04-06T09:54:13.518+0000 7fd1b6210640 -1 reset not still connected to 0x55ef9e88f520 2024-04-06T09:54:13.520 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-04-06T09:54:13.522+0000 7fd1b6210640 -1 reset not still connected to 0x55ef9eee21a0 2024-04-06T09:54:14.765 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:54:14.766+0000 7f0965192640 -1 reset not still connected to 0x5632a2793520 2024-04-06T09:54:14.765 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:54:14.766+0000 7f0965192640 -1 reset not still connected to 0x5632a30f0f70 2024-04-06T09:54:14.766 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:54:14.766+0000 7f0965192640 -1 reset not still connected to 0x5632a417e750 2024-04-06T09:54:14.766 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:54:14.766+0000 7f0965192640 -1 reset not still connected to 0x5632a4cda000 2024-04-06T09:54:14.767 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:54:14.766+0000 7f0965192640 -1 reset not still connected to 0x5632abcbe820 2024-04-06T09:54:14.767 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:54:14.766+0000 7f0965192640 -1 reset not still connected to 0x5632cb3ac000 2024-04-06T09:54:14.820 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:100 IOs, 31952896 bytes 2024-04-06T09:54:14.828 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:54:14.830+0000 7f0965192640 -1 reset not still connected to 0x5632a14ac820 2024-04-06T09:54:14.834 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:54:14.834+0000 7f0965192640 -1 reset not still connected to 0x5632a1988000 2024-04-06T09:54:14.836 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:54:14.838+0000 7f0965192640 -1 reset not still connected to 0x5632a1b73ee0 2024-04-06T09:54:14.836 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:54:14.838+0000 7f0965192640 -1 reset not still connected to 0x5632a2793520 2024-04-06T09:54:14.837 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:54:14.838+0000 7f0965192640 -1 reset not still connected to 0x5632a30f0270 2024-04-06T09:54:14.837 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:54:14.838+0000 7f0965192640 -1 reset not still connected to 0x5632a30f0f70 2024-04-06T09:54:14.837 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:54:14.838+0000 7f0965192640 -1 reset not still connected to 0x5632a417e750 2024-04-06T09:54:14.837 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:54:14.838+0000 7f0965192640 -1 reset not still connected to 0x5632a4cda000 2024-04-06T09:54:14.837 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:54:14.838+0000 7f0965192640 -1 reset not still connected to 0x5632a5e9dee0 2024-04-06T09:54:14.837 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:54:14.838+0000 7f0965192640 -1 reset not still connected to 0x5632abc81d40 2024-04-06T09:54:14.837 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:54:14.838+0000 7f0965192640 -1 reset not still connected to 0x5632abcbe820 2024-04-06T09:54:14.837 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:54:14.838+0000 7f0965192640 -1 reset not still connected to 0x5632b9593110 2024-04-06T09:54:14.837 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:54:14.838+0000 7f0965192640 -1 reset not still connected to 0x5632cb3ac000 2024-04-06T09:54:14.837 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:54:14.838+0000 7f0965192640 -1 reset not still connected to 0x5632e16851e0 2024-04-06T09:54:15.905 INFO:tasks.ceph.cluster2.osd.1.smithi192.stderr:2024-04-06T09:54:15.902+0000 7f0de0719640 -1 reset not still connected to 0x55bfa4b36340 2024-04-06T09:54:15.906 INFO:tasks.ceph.cluster2.osd.1.smithi192.stderr:2024-04-06T09:54:15.902+0000 7f0de0719640 -1 reset not still connected to 0x55bfa5642b60 2024-04-06T09:54:15.906 INFO:tasks.ceph.cluster2.osd.1.smithi192.stderr:2024-04-06T09:54:15.902+0000 7f0de0719640 -1 reset not still connected to 0x55bfa6e48410 2024-04-06T09:54:15.906 INFO:tasks.ceph.cluster2.osd.1.smithi192.stderr:2024-04-06T09:54:15.902+0000 7f0de0719640 -1 reset not still connected to 0x55bfaef3ba00 2024-04-06T09:54:15.926 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:200 IOs, 32602112 bytes 2024-04-06T09:54:18.636 INFO:tasks.rbd_mirror_thrash:waiting for cluster1.client.mirror.3 2024-04-06T09:54:18.636 INFO:teuthology.orchestra.run:waiting for 600 2024-04-06T09:54:18.637 INFO:tasks.rbd_mirror_thrash:reviving cluster1.client.mirror.3 2024-04-06T09:54:18.637 INFO:tasks.rbd_mirror.cluster1.client.mirror.3:Restarting daemon 2024-04-06T09:54:18.637 DEBUG:teuthology.orchestra.run.smithi192:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.3 2024-04-06T09:54:18.639 INFO:tasks.rbd_mirror.cluster1.client.mirror.3:Started 2024-04-06T09:54:18.639 INFO:tasks.rbd_mirror_thrash:waiting for 92 secs before thrashing 2024-04-06T09:54:19.124 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:300 IOs, 78324736 bytes 2024-04-06T09:54:19.281 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-04-06T09:54:19.282+0000 7fd1b6210640 -1 reset not still connected to 0x55ef9ac60270 2024-04-06T09:54:19.281 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-04-06T09:54:19.282+0000 7fd1b6210640 -1 reset not still connected to 0x55ef9b657e10 2024-04-06T09:54:19.281 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-04-06T09:54:19.282+0000 7fd1b6210640 -1 reset not still connected to 0x55ef9bbe9ee0 2024-04-06T09:54:19.281 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-04-06T09:54:19.282+0000 7fd1b6210640 -1 reset not still connected to 0x55ef9bd5c9c0 2024-04-06T09:54:19.281 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-04-06T09:54:19.282+0000 7fd1b6210640 -1 reset not still connected to 0x55ef9bd5d520 2024-04-06T09:54:19.281 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-04-06T09:54:19.282+0000 7fd1b6210640 -1 reset not still connected to 0x55ef9bd5dd40 2024-04-06T09:54:19.282 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-04-06T09:54:19.282+0000 7fd1b6210640 -1 reset not still connected to 0x55ef9bd86820 2024-04-06T09:54:19.282 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-04-06T09:54:19.282+0000 7fd1b6210640 -1 reset not still connected to 0x55ef9c103110 2024-04-06T09:54:19.282 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-04-06T09:54:19.282+0000 7fd1b6210640 -1 reset not still connected to 0x55ef9c103930 2024-04-06T09:54:19.282 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-04-06T09:54:19.282+0000 7fd1b6210640 -1 reset not still connected to 0x55ef9e698c30 2024-04-06T09:54:19.282 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-04-06T09:54:19.282+0000 7fd1b6210640 -1 reset not still connected to 0x55ef9e88f520 2024-04-06T09:54:19.282 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-04-06T09:54:19.282+0000 7fd1b6210640 -1 reset not still connected to 0x55ef9eee21a0 2024-04-06T09:54:19.282 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-04-06T09:54:19.282+0000 7fd1b6210640 -1 reset not still connected to 0x55ef9eee3a00 2024-04-06T09:54:19.404 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:54:19.406+0000 7f2e84d66640 -1 reset not still connected to 0x560d29d5b520 2024-04-06T09:54:19.404 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:54:19.406+0000 7f2e84d66640 -1 reset not still connected to 0x560d2a17d450 2024-04-06T09:54:19.409 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:54:19.410+0000 7f2e84d66640 -1 reset not still connected to 0x560d2a17d6c0 2024-04-06T09:54:19.410 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:54:19.410+0000 7f2e84d66640 -1 reset not still connected to 0x560d2a204b60 2024-04-06T09:54:19.410 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:54:19.410+0000 7f2e84d66640 -1 reset not still connected to 0x560d2a2336c0 2024-04-06T09:54:19.410 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:54:19.410+0000 7f2e84d66640 -1 reset not still connected to 0x560d2a5705b0 2024-04-06T09:54:19.410 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:54:19.410+0000 7f2e84d66640 -1 reset not still connected to 0x560d368e4f70 2024-04-06T09:54:19.436 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ error_code=124 2024-04-06T09:54:19.436 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set -e 2024-04-06T09:54:19.436 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 124 -eq 124 ']' 2024-04-06T09:54:19.436 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:54:19.436 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-06T09:54:19.437 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ image=image_24 2024-04-06T09:54:19.437 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ create_image_and_enable_mirror cluster2 mirror image_24 journal 128M 2024-04-06T09:54:19.437 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-04-06T09:54:19.437 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:54:19.437 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:54:19.437 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:54:19.437 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_24 2024-04-06T09:54:19.437 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:54:19.437 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local mode=journal 2024-04-06T09:54:19.437 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n journal ']' 2024-04-06T09:54:19.437 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:54:19.437 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ create_image cluster2 mirror image_24 128M 2024-04-06T09:54:19.438 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-04-06T09:54:19.438 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:54:19.438 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:54:19.438 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:54:19.438 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_24 2024-04-06T09:54:19.438 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:54:19.439 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local size=128 2024-04-06T09:54:19.439 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n 128M ']' 2024-04-06T09:54:19.439 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ size=128M 2024-04-06T09:54:19.439 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:54:19.439 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,journaling mirror/image_24 2024-04-06T09:54:19.482 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-04-06T09:54:19.474+0000 7fd1b6210640 -1 reset not still connected to 0x55ef9ac60270 2024-04-06T09:54:19.482 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-04-06T09:54:19.474+0000 7fd1b6210640 -1 reset not still connected to 0x55ef9b657e10 2024-04-06T09:54:19.482 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-04-06T09:54:19.474+0000 7fd1b6210640 -1 reset not still connected to 0x55ef9bbe9ee0 2024-04-06T09:54:19.482 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-04-06T09:54:19.474+0000 7fd1b6210640 -1 reset not still connected to 0x55ef9bd5c9c0 2024-04-06T09:54:19.482 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-04-06T09:54:19.474+0000 7fd1b6210640 -1 reset not still connected to 0x55ef9bd5d520 2024-04-06T09:54:19.482 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-04-06T09:54:19.478+0000 7fd1b6210640 -1 reset not still connected to 0x55ef9bd5dd40 2024-04-06T09:54:19.482 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-04-06T09:54:19.478+0000 7fd1b6210640 -1 reset not still connected to 0x55ef9bd86820 2024-04-06T09:54:19.482 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-04-06T09:54:19.478+0000 7fd1b6210640 -1 reset not still connected to 0x55ef9c103110 2024-04-06T09:54:19.484 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-04-06T09:54:19.486+0000 7fd1b6210640 -1 reset not still connected to 0x55ef9c103930 2024-04-06T09:54:19.484 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-04-06T09:54:19.486+0000 7fd1b6210640 -1 reset not still connected to 0x55ef9e698c30 2024-04-06T09:54:19.484 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-04-06T09:54:19.486+0000 7fd1b6210640 -1 reset not still connected to 0x55ef9e88f520 2024-04-06T09:54:19.486 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-04-06T09:54:19.486+0000 7fd1b6210640 -1 reset not still connected to 0x55ef9eee21a0 2024-04-06T09:54:19.486 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-04-06T09:54:19.486+0000 7fd1b6210640 -1 reset not still connected to 0x55ef9eee3a00 2024-04-06T09:54:19.499 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' pool = image ']' 2024-04-06T09:54:19.499 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' mirror = mirror_parent ']' 2024-04-06T09:54:19.499 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n '' ']' 2024-04-06T09:54:19.499 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ stress_write_image cluster2 mirror image_24 2024-04-06T09:54:19.499 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-04-06T09:54:19.500 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:54:19.500 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_24 2024-04-06T09:54:19.500 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-04-06T09:54:19.501 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local duration=9 2024-04-06T09:54:19.501 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set +e 2024-04-06T09:54:19.501 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ timeout 9s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_24 --debug-rbd=20 --debug-journaler=20 2024-04-06T09:54:22.453 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:54:22.454+0000 7f2e84d66640 -1 reset not still connected to 0x560d29d5b520 2024-04-06T09:54:22.453 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:54:22.454+0000 7f2e84d66640 -1 reset not still connected to 0x560d2a17d450 2024-04-06T09:54:22.453 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:54:22.454+0000 7f2e84d66640 -1 reset not still connected to 0x560d2a17d6c0 2024-04-06T09:54:22.453 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:54:22.454+0000 7f2e84d66640 -1 reset not still connected to 0x560d2a204b60 2024-04-06T09:54:22.453 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:54:22.454+0000 7f2e84d66640 -1 reset not still connected to 0x560d2a2336c0 2024-04-06T09:54:22.453 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:54:22.454+0000 7f2e84d66640 -1 reset not still connected to 0x560d2a5705b0 2024-04-06T09:54:22.456 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:54:22.458+0000 7f2e84d66640 -1 reset not still connected to 0x560d368e4f70 2024-04-06T09:54:22.456 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:54:22.458+0000 7f2e84d66640 -1 reset not still connected to 0x560d36912ea0 2024-04-06T09:54:23.262 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:54:23.262+0000 7f0965192640 -1 reset not still connected to 0x5632a1988000 2024-04-06T09:54:23.262 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:54:23.262+0000 7f0965192640 -1 reset not still connected to 0x5632a1b73ee0 2024-04-06T09:54:23.262 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:54:23.262+0000 7f0965192640 -1 reset not still connected to 0x5632a2793520 2024-04-06T09:54:23.262 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:54:23.262+0000 7f0965192640 -1 reset not still connected to 0x5632a30f0270 2024-04-06T09:54:23.263 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:54:23.262+0000 7f0965192640 -1 reset not still connected to 0x5632a30f0f70 2024-04-06T09:54:23.263 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:54:23.262+0000 7f0965192640 -1 reset not still connected to 0x5632a417e750 2024-04-06T09:54:23.264 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:54:23.262+0000 7f0965192640 -1 reset not still connected to 0x5632a4cda000 2024-04-06T09:54:23.264 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:54:23.262+0000 7f0965192640 -1 reset not still connected to 0x5632a5e9dee0 2024-04-06T09:54:23.264 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:54:23.262+0000 7f0965192640 -1 reset not still connected to 0x5632abc812b0 2024-04-06T09:54:23.264 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:54:23.262+0000 7f0965192640 -1 reset not still connected to 0x5632abc81d40 2024-04-06T09:54:23.264 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:54:23.262+0000 7f0965192640 -1 reset not still connected to 0x5632abcbe820 2024-04-06T09:54:23.267 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:54:23.270+0000 7f0965192640 -1 reset not still connected to 0x5632b9593110 2024-04-06T09:54:23.268 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:54:23.270+0000 7f0965192640 -1 reset not still connected to 0x5632cb3ac000 2024-04-06T09:54:23.268 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:54:23.270+0000 7f0965192640 -1 reset not still connected to 0x5632dac09040 2024-04-06T09:54:23.268 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:54:23.270+0000 7f0965192640 -1 reset not still connected to 0x5632e16851e0 2024-04-06T09:54:23.330 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:100 IOs, 62550016 bytes 2024-04-06T09:54:23.894 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:54:23.894+0000 7f0965192640 -1 reset not still connected to 0x5632b9593110 2024-04-06T09:54:23.897 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:54:23.898+0000 7f0965192640 -1 reset not still connected to 0x5632cb3ac000 2024-04-06T09:54:23.898 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:54:23.898+0000 7f0965192640 -1 reset not still connected to 0x5632dac09040 2024-04-06T09:54:23.898 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:54:23.898+0000 7f0965192640 -1 reset not still connected to 0x5632e16851e0 2024-04-06T09:54:26.302 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:200 IOs, 77687808 bytes 2024-04-06T09:54:28.510 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ error_code=124 2024-04-06T09:54:28.510 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set -e 2024-04-06T09:54:28.510 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 124 -eq 124 ']' 2024-04-06T09:54:28.510 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:54:28.510 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-06T09:54:28.510 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ image=image_25 2024-04-06T09:54:28.510 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ create_image_and_enable_mirror cluster2 mirror image_25 journal 128M 2024-04-06T09:54:28.510 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-04-06T09:54:28.510 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:54:28.510 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:54:28.511 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:54:28.511 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_25 2024-04-06T09:54:28.511 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:54:28.511 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local mode=journal 2024-04-06T09:54:28.511 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n journal ']' 2024-04-06T09:54:28.511 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:54:28.511 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ create_image cluster2 mirror image_25 128M 2024-04-06T09:54:28.511 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-04-06T09:54:28.511 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:54:28.511 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:54:28.511 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:54:28.511 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_25 2024-04-06T09:54:28.511 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:54:28.511 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local size=128 2024-04-06T09:54:28.511 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n 128M ']' 2024-04-06T09:54:28.512 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ size=128M 2024-04-06T09:54:28.512 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:54:28.512 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,journaling mirror/image_25 2024-04-06T09:54:28.619 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' pool = image ']' 2024-04-06T09:54:28.619 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' mirror = mirror_parent ']' 2024-04-06T09:54:28.619 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n '' ']' 2024-04-06T09:54:28.619 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ stress_write_image cluster2 mirror image_25 2024-04-06T09:54:28.619 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-04-06T09:54:28.619 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:54:28.620 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_25 2024-04-06T09:54:28.620 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-04-06T09:54:28.621 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local duration=9 2024-04-06T09:54:28.621 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set +e 2024-04-06T09:54:28.621 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ timeout 9s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_25 --debug-rbd=20 --debug-journaler=20 2024-04-06T09:54:30.128 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:54:30.126+0000 7f0965192640 -1 reset not still connected to 0x5632e16851e0 2024-04-06T09:54:30.589 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:54:30.586+0000 7f2e84d66640 -1 reset not still connected to 0x560d27072820 2024-04-06T09:54:30.590 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:54:30.586+0000 7f2e84d66640 -1 reset not still connected to 0x560d27073860 2024-04-06T09:54:30.590 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:54:30.590+0000 7f2e84d66640 -1 reset not still connected to 0x560d2722e5b0 2024-04-06T09:54:30.591 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:54:30.590+0000 7f2e84d66640 -1 reset not still connected to 0x560d27566ea0 2024-04-06T09:54:30.591 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:54:30.590+0000 7f2e84d66640 -1 reset not still connected to 0x560d27567380 2024-04-06T09:54:30.600 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:54:30.598+0000 7f2e84d66640 -1 reset not still connected to 0x560d27839930 2024-04-06T09:54:30.600 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:54:30.598+0000 7f2e84d66640 -1 reset not still connected to 0x560d27c3edd0 2024-04-06T09:54:30.600 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:54:30.598+0000 7f2e84d66640 -1 reset not still connected to 0x560d27c3f2b0 2024-04-06T09:54:30.600 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:54:30.598+0000 7f2e84d66640 -1 reset not still connected to 0x560d281875f0 2024-04-06T09:54:30.600 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:54:30.598+0000 7f2e84d66640 -1 reset not still connected to 0x560d294908f0 2024-04-06T09:54:30.600 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:54:30.598+0000 7f2e84d66640 -1 reset not still connected to 0x560d29cd8f70 2024-04-06T09:54:30.603 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:54:30.602+0000 7f2e84d66640 -1 reset not still connected to 0x560d29d5a340 2024-04-06T09:54:30.603 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:54:30.602+0000 7f2e84d66640 -1 reset not still connected to 0x560d29d5b520 2024-04-06T09:54:30.606 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:54:30.606+0000 7f2e84d66640 -1 reset not still connected to 0x560d2a17d450 2024-04-06T09:54:30.606 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:54:30.606+0000 7f2e84d66640 -1 reset not still connected to 0x560d2a17d6c0 2024-04-06T09:54:30.610 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:54:30.606+0000 7f2e84d66640 -1 reset not still connected to 0x560d2a204b60 2024-04-06T09:54:30.610 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:54:30.606+0000 7f2e84d66640 -1 reset not still connected to 0x560d2a2336c0 2024-04-06T09:54:30.610 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:54:30.606+0000 7f2e84d66640 -1 reset not still connected to 0x560d2a5705b0 2024-04-06T09:54:30.610 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:54:30.606+0000 7f2e84d66640 -1 reset not still connected to 0x560d368e4f70 2024-04-06T09:54:30.611 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:54:30.606+0000 7f2e84d66640 -1 reset not still connected to 0x560d36912ea0 2024-04-06T09:54:30.809 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:54:30.810+0000 7f0965192640 -1 reset not still connected to 0x5632a5e9dee0 2024-04-06T09:54:30.809 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:54:30.810+0000 7f0965192640 -1 reset not still connected to 0x5632abc812b0 2024-04-06T09:54:30.809 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:54:30.810+0000 7f0965192640 -1 reset not still connected to 0x5632abc81d40 2024-04-06T09:54:30.812 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:54:30.814+0000 7f0965192640 -1 reset not still connected to 0x5632dac09040 2024-04-06T09:54:30.812 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:54:30.814+0000 7f0965192640 -1 reset not still connected to 0x5632e16851e0 2024-04-06T09:54:31.082 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:54:31.082+0000 7f2e84d66640 -1 reset not still connected to 0x560d27c3edd0 2024-04-06T09:54:31.082 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:54:31.082+0000 7f2e84d66640 -1 reset not still connected to 0x560d27c3f2b0 2024-04-06T09:54:31.082 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:54:31.082+0000 7f2e84d66640 -1 reset not still connected to 0x560d294908f0 2024-04-06T09:54:31.087 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:54:31.086+0000 7f2e84d66640 -1 reset not still connected to 0x560d29cd8f70 2024-04-06T09:54:31.089 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:54:31.090+0000 7f2e84d66640 -1 reset not still connected to 0x560d29cd9ba0 2024-04-06T09:54:31.089 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:54:31.090+0000 7f2e84d66640 -1 reset not still connected to 0x560d29d5a340 2024-04-06T09:54:31.089 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:54:31.090+0000 7f2e84d66640 -1 reset not still connected to 0x560d29d5b520 2024-04-06T09:54:31.089 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:54:31.090+0000 7f2e84d66640 -1 reset not still connected to 0x560d2a17d450 2024-04-06T09:54:31.089 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:54:31.090+0000 7f2e84d66640 -1 reset not still connected to 0x560d2a17d6c0 2024-04-06T09:54:31.092 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:54:31.094+0000 7f2e84d66640 -1 reset not still connected to 0x560d2a2336c0 2024-04-06T09:54:31.092 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:54:31.094+0000 7f2e84d66640 -1 reset not still connected to 0x560d2a5705b0 2024-04-06T09:54:31.092 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:54:31.094+0000 7f2e84d66640 -1 reset not still connected to 0x560d368e4f70 2024-04-06T09:54:31.092 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:54:31.094+0000 7f2e84d66640 -1 reset not still connected to 0x560d36912ea0 2024-04-06T09:54:32.661 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:54:32.658+0000 7f0965192640 -1 reset not still connected to 0x5632a1988000 2024-04-06T09:54:32.663 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:54:32.662+0000 7f0965192640 -1 reset not still connected to 0x5632a1b73ee0 2024-04-06T09:54:32.663 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:54:32.662+0000 7f0965192640 -1 reset not still connected to 0x5632a2793520 2024-04-06T09:54:32.663 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:54:32.662+0000 7f0965192640 -1 reset not still connected to 0x5632a30f0270 2024-04-06T09:54:32.663 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:54:32.662+0000 7f0965192640 -1 reset not still connected to 0x5632a30f0f70 2024-04-06T09:54:32.665 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:54:32.666+0000 7f0965192640 -1 reset not still connected to 0x5632a417e750 2024-04-06T09:54:32.665 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:54:32.666+0000 7f0965192640 -1 reset not still connected to 0x5632a4cda000 2024-04-06T09:54:32.665 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:54:32.666+0000 7f0965192640 -1 reset not still connected to 0x5632a5e9dee0 2024-04-06T09:54:32.665 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:54:32.666+0000 7f0965192640 -1 reset not still connected to 0x5632abc812b0 2024-04-06T09:54:32.665 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:54:32.666+0000 7f0965192640 -1 reset not still connected to 0x5632abc81d40 2024-04-06T09:54:32.665 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:54:32.666+0000 7f0965192640 -1 reset not still connected to 0x5632abcbe820 2024-04-06T09:54:32.666 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:54:32.666+0000 7f0965192640 -1 reset not still connected to 0x5632b9593110 2024-04-06T09:54:32.666 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:54:32.666+0000 7f0965192640 -1 reset not still connected to 0x5632cb3ac000 2024-04-06T09:54:32.666 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:54:32.666+0000 7f0965192640 -1 reset not still connected to 0x5632dac09040 2024-04-06T09:54:32.666 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:54:32.666+0000 7f0965192640 -1 reset not still connected to 0x5632e16851e0 2024-04-06T09:54:34.164 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:100 IOs, 71083008 bytes 2024-04-06T09:54:35.833 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-04-06T09:54:35.830+0000 7fd1b6210640 -1 reset not still connected to 0x55ef99306ea0 2024-04-06T09:54:35.833 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-04-06T09:54:35.830+0000 7fd1b6210640 -1 reset not still connected to 0x55ef997761a0 2024-04-06T09:54:35.833 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-04-06T09:54:35.830+0000 7fd1b6210640 -1 reset not still connected to 0x55ef99776680 2024-04-06T09:54:35.833 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-04-06T09:54:35.830+0000 7fd1b6210640 -1 reset not still connected to 0x55ef99b88d00 2024-04-06T09:54:35.833 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-04-06T09:54:35.830+0000 7fd1b6210640 -1 reset not still connected to 0x55ef99b89ad0 2024-04-06T09:54:35.833 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-04-06T09:54:35.830+0000 7fd1b6210640 -1 reset not still connected to 0x55ef9ac60270 2024-04-06T09:54:35.834 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-04-06T09:54:35.834+0000 7fd1b6210640 -1 reset not still connected to 0x55ef9bbe9ee0 2024-04-06T09:54:35.835 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-04-06T09:54:35.834+0000 7fd1b6210640 -1 reset not still connected to 0x55ef9bd5c9c0 2024-04-06T09:54:35.835 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-04-06T09:54:35.834+0000 7fd1b6210640 -1 reset not still connected to 0x55ef9bd5d520 2024-04-06T09:54:35.835 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-04-06T09:54:35.834+0000 7fd1b6210640 -1 reset not still connected to 0x55ef9bd5dd40 2024-04-06T09:54:35.838 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-04-06T09:54:35.838+0000 7fd1b6210640 -1 reset not still connected to 0x55ef9c103110 2024-04-06T09:54:35.838 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-04-06T09:54:35.838+0000 7fd1b6210640 -1 reset not still connected to 0x55ef9c103930 2024-04-06T09:54:35.839 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-04-06T09:54:35.838+0000 7fd1b6210640 -1 reset not still connected to 0x55ef9e698c30 2024-04-06T09:54:35.839 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-04-06T09:54:35.838+0000 7fd1b6210640 -1 reset not still connected to 0x55ef9e88f520 2024-04-06T09:54:35.839 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-04-06T09:54:35.838+0000 7fd1b6210640 -1 reset not still connected to 0x55ef9eee21a0 2024-04-06T09:54:37.488 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:54:37.490+0000 7f0965192640 -1 reset not still connected to 0x5632a417e750 2024-04-06T09:54:37.488 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:54:37.490+0000 7f0965192640 -1 reset not still connected to 0x5632abc812b0 2024-04-06T09:54:37.496 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:54:37.494+0000 7f0965192640 -1 reset not still connected to 0x5632abcbe820 2024-04-06T09:54:37.496 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:54:37.494+0000 7f0965192640 -1 reset not still connected to 0x5632b9593110 2024-04-06T09:54:37.496 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:54:37.494+0000 7f0965192640 -1 reset not still connected to 0x5632cb3ac000 2024-04-06T09:54:37.496 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:54:37.494+0000 7f0965192640 -1 reset not still connected to 0x5632dac08dd0 2024-04-06T09:54:37.497 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:54:37.498+0000 7f0965192640 -1 reset not still connected to 0x5632e16851e0 2024-04-06T09:54:37.497 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:54:37.498+0000 7f0965192640 -1 reset not still connected to 0x5632a417e750 2024-04-06T09:54:37.497 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:54:37.498+0000 7f0965192640 -1 reset not still connected to 0x5632abc812b0 2024-04-06T09:54:37.497 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:54:37.498+0000 7f0965192640 -1 reset not still connected to 0x5632abcbe820 2024-04-06T09:54:37.630 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ error_code=124 2024-04-06T09:54:37.631 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set -e 2024-04-06T09:54:37.631 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 124 -eq 124 ']' 2024-04-06T09:54:37.631 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:54:37.631 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-06T09:54:37.631 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ image=image_26 2024-04-06T09:54:37.631 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ create_image_and_enable_mirror cluster2 mirror image_26 journal 128M 2024-04-06T09:54:37.631 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-04-06T09:54:37.631 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:54:37.631 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:54:37.631 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:54:37.631 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_26 2024-04-06T09:54:37.631 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:54:37.631 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local mode=journal 2024-04-06T09:54:37.631 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n journal ']' 2024-04-06T09:54:37.632 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:54:37.632 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ create_image cluster2 mirror image_26 128M 2024-04-06T09:54:37.632 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-04-06T09:54:37.632 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:54:37.632 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:54:37.632 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:54:37.632 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_26 2024-04-06T09:54:37.632 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:54:37.632 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local size=128 2024-04-06T09:54:37.632 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n 128M ']' 2024-04-06T09:54:37.632 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ size=128M 2024-04-06T09:54:37.632 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:54:37.632 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,journaling mirror/image_26 2024-04-06T09:54:37.694 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' pool = image ']' 2024-04-06T09:54:37.694 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' mirror = mirror_parent ']' 2024-04-06T09:54:37.694 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n '' ']' 2024-04-06T09:54:37.694 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ stress_write_image cluster2 mirror image_26 2024-04-06T09:54:37.694 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-04-06T09:54:37.695 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:54:37.695 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_26 2024-04-06T09:54:37.695 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-04-06T09:54:37.695 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local duration=8 2024-04-06T09:54:37.695 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set +e 2024-04-06T09:54:37.696 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ timeout 8s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_26 --debug-rbd=20 --debug-journaler=20 2024-04-06T09:54:40.269 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:54:40.270+0000 7f0965192640 -1 reset not still connected to 0x5632a14ac820 2024-04-06T09:54:40.269 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:54:40.270+0000 7f0965192640 -1 reset not still connected to 0x5632a1988000 2024-04-06T09:54:40.269 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:54:40.270+0000 7f0965192640 -1 reset not still connected to 0x5632a2793520 2024-04-06T09:54:40.269 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:54:40.270+0000 7f0965192640 -1 reset not still connected to 0x5632a417e750 2024-04-06T09:54:40.269 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:54:40.270+0000 7f0965192640 -1 reset not still connected to 0x5632a4cda000 2024-04-06T09:54:40.269 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:54:40.270+0000 7f0965192640 -1 reset not still connected to 0x5632a5e9dee0 2024-04-06T09:54:40.275 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:54:40.274+0000 7f0965192640 -1 reset not still connected to 0x5632abc812b0 2024-04-06T09:54:40.275 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:54:40.274+0000 7f0965192640 -1 reset not still connected to 0x5632abc81d40 2024-04-06T09:54:40.275 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:54:40.274+0000 7f0965192640 -1 reset not still connected to 0x5632abcbe820 2024-04-06T09:54:40.278 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:54:40.278+0000 7f0965192640 -1 reset not still connected to 0x5632b9593110 2024-04-06T09:54:40.278 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:54:40.278+0000 7f0965192640 -1 reset not still connected to 0x5632cb3ac000 2024-04-06T09:54:40.278 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:54:40.278+0000 7f0965192640 -1 reset not still connected to 0x5632dac08dd0 2024-04-06T09:54:40.278 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:54:40.278+0000 7f0965192640 -1 reset not still connected to 0x5632dac09040 2024-04-06T09:54:40.278 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:54:40.278+0000 7f0965192640 -1 reset not still connected to 0x5632e16851e0 2024-04-06T09:54:40.282 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:54:40.282+0000 7f0965192640 -1 reset not still connected to 0x56329f99c0d0 2024-04-06T09:54:40.282 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:54:40.282+0000 7f0965192640 -1 reset not still connected to 0x5632a14ac820 2024-04-06T09:54:40.289 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:54:40.290+0000 7f0965192640 -1 reset not still connected to 0x5632a1988000 2024-04-06T09:54:40.324 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:54:40.290+0000 7f0965192640 -1 reset not still connected to 0x5632a1b73ee0 2024-04-06T09:54:40.324 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:54:40.290+0000 7f0965192640 -1 reset not still connected to 0x5632a2793520 2024-04-06T09:54:40.324 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:54:40.290+0000 7f0965192640 -1 reset not still connected to 0x5632a30f0270 2024-04-06T09:54:40.326 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:54:40.290+0000 7f0965192640 -1 reset not still connected to 0x5632a30f0f70 2024-04-06T09:54:40.326 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:54:40.290+0000 7f0965192640 -1 reset not still connected to 0x5632a417e750 2024-04-06T09:54:40.326 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:54:40.290+0000 7f0965192640 -1 reset not still connected to 0x5632a4cda000 2024-04-06T09:54:40.327 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:54:40.290+0000 7f0965192640 -1 reset not still connected to 0x5632a5e9dee0 2024-04-06T09:54:40.327 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:54:40.290+0000 7f0965192640 -1 reset not still connected to 0x5632abc812b0 2024-04-06T09:54:40.327 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:54:40.294+0000 7f0965192640 -1 reset not still connected to 0x5632abc81d40 2024-04-06T09:54:40.327 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:54:40.294+0000 7f0965192640 -1 reset not still connected to 0x5632abcbe820 2024-04-06T09:54:40.327 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:54:40.294+0000 7f0965192640 -1 reset not still connected to 0x5632b9593110 2024-04-06T09:54:40.327 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:54:40.294+0000 7f0965192640 -1 reset not still connected to 0x5632cb3ac000 2024-04-06T09:54:40.327 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:54:40.294+0000 7f0965192640 -1 reset not still connected to 0x5632dac08dd0 2024-04-06T09:54:40.327 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:54:40.294+0000 7f0965192640 -1 reset not still connected to 0x5632dac09040 2024-04-06T09:54:40.327 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:54:40.294+0000 7f0965192640 -1 reset not still connected to 0x5632e16851e0 2024-04-06T09:54:40.887 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:54:40.886+0000 7f0965192640 -1 reset not still connected to 0x5632a14ac820 2024-04-06T09:54:40.888 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:54:40.886+0000 7f0965192640 -1 reset not still connected to 0x5632a1988000 2024-04-06T09:54:40.888 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:54:40.886+0000 7f0965192640 -1 reset not still connected to 0x5632a1b73ee0 2024-04-06T09:54:40.888 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:54:40.886+0000 7f0965192640 -1 reset not still connected to 0x5632a2793520 2024-04-06T09:54:40.888 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:54:40.886+0000 7f0965192640 -1 reset not still connected to 0x5632a30f0270 2024-04-06T09:54:40.888 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:54:40.886+0000 7f0965192640 -1 reset not still connected to 0x5632a417e750 2024-04-06T09:54:40.888 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:54:40.886+0000 7f0965192640 -1 reset not still connected to 0x5632a4cda000 2024-04-06T09:54:40.888 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:54:40.886+0000 7f0965192640 -1 reset not still connected to 0x5632a5e9dee0 2024-04-06T09:54:40.888 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:54:40.886+0000 7f0965192640 -1 reset not still connected to 0x5632abc812b0 2024-04-06T09:54:40.888 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:54:40.886+0000 7f0965192640 -1 reset not still connected to 0x5632abc81d40 2024-04-06T09:54:40.888 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:54:40.886+0000 7f0965192640 -1 reset not still connected to 0x5632abcbe820 2024-04-06T09:54:40.888 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:54:40.886+0000 7f0965192640 -1 reset not still connected to 0x5632b9593110 2024-04-06T09:54:40.888 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:54:40.886+0000 7f0965192640 -1 reset not still connected to 0x5632cb3ac000 2024-04-06T09:54:40.888 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:54:40.886+0000 7f0965192640 -1 reset not still connected to 0x5632dac08dd0 2024-04-06T09:54:40.888 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:54:40.886+0000 7f0965192640 -1 reset not still connected to 0x5632dac09040 2024-04-06T09:54:40.889 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:54:40.886+0000 7f0965192640 -1 reset not still connected to 0x5632e16851e0 2024-04-06T09:54:41.430 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:54:41.430+0000 7f0965192640 -1 reset not still connected to 0x5632a1988000 2024-04-06T09:54:41.436 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:54:41.434+0000 7f0965192640 -1 reset not still connected to 0x5632a1b73ee0 2024-04-06T09:54:41.436 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:54:41.434+0000 7f0965192640 -1 reset not still connected to 0x5632a2793520 2024-04-06T09:54:41.436 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:54:41.434+0000 7f0965192640 -1 reset not still connected to 0x5632a30f0270 2024-04-06T09:54:41.438 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:54:41.438+0000 7f0965192640 -1 reset not still connected to 0x5632a30f0f70 2024-04-06T09:54:41.438 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:54:41.438+0000 7f0965192640 -1 reset not still connected to 0x5632a417e750 2024-04-06T09:54:41.438 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:54:41.438+0000 7f0965192640 -1 reset not still connected to 0x5632a4cda000 2024-04-06T09:54:41.439 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:54:41.438+0000 7f0965192640 -1 reset not still connected to 0x5632a4cdbba0 2024-04-06T09:54:41.439 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:54:41.438+0000 7f0965192640 -1 reset not still connected to 0x5632a5e9dee0 2024-04-06T09:54:41.439 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:54:41.438+0000 7f0965192640 -1 reset not still connected to 0x5632abc812b0 2024-04-06T09:54:41.441 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:54:41.438+0000 7f0965192640 -1 reset not still connected to 0x5632abc81d40 2024-04-06T09:54:41.441 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:54:41.438+0000 7f0965192640 -1 reset not still connected to 0x5632abcbe820 2024-04-06T09:54:41.441 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:54:41.438+0000 7f0965192640 -1 reset not still connected to 0x5632b9593110 2024-04-06T09:54:41.441 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:54:41.442+0000 7f0965192640 -1 reset not still connected to 0x5632cb3ac000 2024-04-06T09:54:41.441 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:54:41.442+0000 7f0965192640 -1 reset not still connected to 0x5632dac09040 2024-04-06T09:54:41.441 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:54:41.442+0000 7f0965192640 -1 reset not still connected to 0x5632e16851e0 2024-04-06T09:54:43.692 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:54:43.694+0000 7f2e84d66640 -1 reset not still connected to 0x560d2a5705b0 2024-04-06T09:54:43.694 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:54:43.694+0000 7f2e84d66640 -1 reset not still connected to 0x560d368e4f70 2024-04-06T09:54:43.694 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:54:43.694+0000 7f2e84d66640 -1 reset not still connected to 0x560d36912ea0 2024-04-06T09:54:43.767 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:100 IOs, 94231552 bytes 2024-04-06T09:54:45.706 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ error_code=124 2024-04-06T09:54:45.707 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set -e 2024-04-06T09:54:45.707 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 124 -eq 124 ']' 2024-04-06T09:54:45.707 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:54:45.707 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-06T09:54:45.707 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ image=image_27 2024-04-06T09:54:45.707 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ create_image_and_enable_mirror cluster2 mirror image_27 journal 128M 2024-04-06T09:54:45.707 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-04-06T09:54:45.707 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:54:45.707 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:54:45.707 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:54:45.707 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_27 2024-04-06T09:54:45.707 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:54:45.708 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local mode=journal 2024-04-06T09:54:45.708 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n journal ']' 2024-04-06T09:54:45.708 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:54:45.708 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ create_image cluster2 mirror image_27 128M 2024-04-06T09:54:45.708 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-04-06T09:54:45.708 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:54:45.708 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:54:45.708 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:54:45.708 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_27 2024-04-06T09:54:45.708 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:54:45.708 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local size=128 2024-04-06T09:54:45.708 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n 128M ']' 2024-04-06T09:54:45.708 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ size=128M 2024-04-06T09:54:45.708 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:54:45.708 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,journaling mirror/image_27 2024-04-06T09:54:45.770 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' pool = image ']' 2024-04-06T09:54:45.770 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' mirror = mirror_parent ']' 2024-04-06T09:54:45.770 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n '' ']' 2024-04-06T09:54:45.770 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ stress_write_image cluster2 mirror image_27 2024-04-06T09:54:45.770 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-04-06T09:54:45.770 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:54:45.770 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_27 2024-04-06T09:54:45.770 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-04-06T09:54:45.771 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local duration=5 2024-04-06T09:54:45.771 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set +e 2024-04-06T09:54:45.771 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ timeout 5s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_27 --debug-rbd=20 --debug-journaler=20 2024-04-06T09:54:46.952 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-04-06T09:54:46.954+0000 7fd1b6210640 -1 reset not still connected to 0x55ef99306ea0 2024-04-06T09:54:46.955 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-04-06T09:54:46.954+0000 7fd1b6210640 -1 reset not still connected to 0x55ef997761a0 2024-04-06T09:54:46.955 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-04-06T09:54:46.954+0000 7fd1b6210640 -1 reset not still connected to 0x55ef99776680 2024-04-06T09:54:46.955 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-04-06T09:54:46.954+0000 7fd1b6210640 -1 reset not still connected to 0x55ef99b88d00 2024-04-06T09:54:46.955 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-04-06T09:54:46.954+0000 7fd1b6210640 -1 reset not still connected to 0x55ef99b89ad0 2024-04-06T09:54:46.955 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-04-06T09:54:46.954+0000 7fd1b6210640 -1 reset not still connected to 0x55ef99ebf6c0 2024-04-06T09:54:46.955 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-04-06T09:54:46.954+0000 7fd1b6210640 -1 reset not still connected to 0x55ef9ac60270 2024-04-06T09:54:46.955 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-04-06T09:54:46.954+0000 7fd1b6210640 -1 reset not still connected to 0x55ef9b657e10 2024-04-06T09:54:46.955 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-04-06T09:54:46.954+0000 7fd1b6210640 -1 reset not still connected to 0x55ef9bbe9ee0 2024-04-06T09:54:46.955 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-04-06T09:54:46.954+0000 7fd1b6210640 -1 reset not still connected to 0x55ef9bd5c9c0 2024-04-06T09:54:46.955 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-04-06T09:54:46.954+0000 7fd1b6210640 -1 reset not still connected to 0x55ef9bd5dd40 2024-04-06T09:54:46.956 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-04-06T09:54:46.954+0000 7fd1b6210640 -1 reset not still connected to 0x55ef9bd86820 2024-04-06T09:54:46.956 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-04-06T09:54:46.954+0000 7fd1b6210640 -1 reset not still connected to 0x55ef9c103110 2024-04-06T09:54:46.956 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-04-06T09:54:46.954+0000 7fd1b6210640 -1 reset not still connected to 0x55ef9c103930 2024-04-06T09:54:46.956 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-04-06T09:54:46.954+0000 7fd1b6210640 -1 reset not still connected to 0x55ef9e698c30 2024-04-06T09:54:46.956 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-04-06T09:54:46.954+0000 7fd1b6210640 -1 reset not still connected to 0x55ef9e88f520 2024-04-06T09:54:46.956 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-04-06T09:54:46.954+0000 7fd1b6210640 -1 reset not still connected to 0x55ef9eee21a0 2024-04-06T09:54:46.956 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-04-06T09:54:46.954+0000 7fd1b6210640 -1 reset not still connected to 0x55ef9eee3a00 2024-04-06T09:54:48.796 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:100 IOs, 39706624 bytes 2024-04-06T09:54:50.235 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:54:50.234+0000 7f2e84d66640 -1 reset not still connected to 0x560d2722e5b0 2024-04-06T09:54:50.235 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:54:50.234+0000 7f2e84d66640 -1 reset not still connected to 0x560d27566ea0 2024-04-06T09:54:50.235 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:54:50.234+0000 7f2e84d66640 -1 reset not still connected to 0x560d27567380 2024-04-06T09:54:50.235 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:54:50.234+0000 7f2e84d66640 -1 reset not still connected to 0x560d27839380 2024-04-06T09:54:50.235 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:54:50.234+0000 7f2e84d66640 -1 reset not still connected to 0x560d27839930 2024-04-06T09:54:50.235 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:54:50.234+0000 7f2e84d66640 -1 reset not still connected to 0x560d27c3edd0 2024-04-06T09:54:50.235 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:54:50.234+0000 7f2e84d66640 -1 reset not still connected to 0x560d27c3f2b0 2024-04-06T09:54:50.235 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:54:50.234+0000 7f2e84d66640 -1 reset not still connected to 0x560d281875f0 2024-04-06T09:54:50.235 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:54:50.234+0000 7f2e84d66640 -1 reset not still connected to 0x560d294908f0 2024-04-06T09:54:50.235 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:54:50.234+0000 7f2e84d66640 -1 reset not still connected to 0x560d29cd8f70 2024-04-06T09:54:50.235 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:54:50.234+0000 7f2e84d66640 -1 reset not still connected to 0x560d29cd9ba0 2024-04-06T09:54:50.236 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:54:50.234+0000 7f2e84d66640 -1 reset not still connected to 0x560d29d5a340 2024-04-06T09:54:50.236 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:54:50.234+0000 7f2e84d66640 -1 reset not still connected to 0x560d29d5b520 2024-04-06T09:54:50.236 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:54:50.234+0000 7f2e84d66640 -1 reset not still connected to 0x560d2a17d450 2024-04-06T09:54:50.236 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:54:50.234+0000 7f2e84d66640 -1 reset not still connected to 0x560d2a17d6c0 2024-04-06T09:54:50.236 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:54:50.234+0000 7f2e84d66640 -1 reset not still connected to 0x560d2a204b60 2024-04-06T09:54:50.236 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:54:50.234+0000 7f2e84d66640 -1 reset not still connected to 0x560d2a2336c0 2024-04-06T09:54:50.236 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:54:50.234+0000 7f2e84d66640 -1 reset not still connected to 0x560d2a5705b0 2024-04-06T09:54:50.236 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:54:50.234+0000 7f2e84d66640 -1 reset not still connected to 0x560d368e4f70 2024-04-06T09:54:50.236 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:54:50.234+0000 7f2e84d66640 -1 reset not still connected to 0x560d36912ea0 2024-04-06T09:54:50.236 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:54:50.234+0000 7f2e84d66640 -1 reset not still connected to 0x560d381dd5f0 2024-04-06T09:54:50.236 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:54:50.234+0000 7f2e84d66640 -1 reset not still connected to 0x560d390dca90 2024-04-06T09:54:50.781 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ error_code=124 2024-04-06T09:54:50.781 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set -e 2024-04-06T09:54:50.782 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 124 -eq 124 ']' 2024-04-06T09:54:50.782 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:54:50.782 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-06T09:54:50.782 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ image=image_28 2024-04-06T09:54:50.782 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ create_image_and_enable_mirror cluster2 mirror image_28 journal 128M 2024-04-06T09:54:50.782 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-04-06T09:54:50.782 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:54:50.782 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:54:50.782 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:54:50.782 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_28 2024-04-06T09:54:50.782 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:54:50.782 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local mode=journal 2024-04-06T09:54:50.782 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n journal ']' 2024-04-06T09:54:50.782 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:54:50.783 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ create_image cluster2 mirror image_28 128M 2024-04-06T09:54:50.783 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-04-06T09:54:50.783 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:54:50.783 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:54:50.783 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:54:50.783 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_28 2024-04-06T09:54:50.783 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:54:50.783 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local size=128 2024-04-06T09:54:50.783 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n 128M ']' 2024-04-06T09:54:50.783 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ size=128M 2024-04-06T09:54:50.783 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:54:50.783 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,journaling mirror/image_28 2024-04-06T09:54:50.841 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' pool = image ']' 2024-04-06T09:54:50.841 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' mirror = mirror_parent ']' 2024-04-06T09:54:50.842 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n '' ']' 2024-04-06T09:54:50.842 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ stress_write_image cluster2 mirror image_28 2024-04-06T09:54:50.842 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-04-06T09:54:50.842 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:54:50.842 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_28 2024-04-06T09:54:50.842 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-04-06T09:54:50.843 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local duration=11 2024-04-06T09:54:50.843 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set +e 2024-04-06T09:54:50.843 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ timeout 11s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_28 --debug-rbd=20 --debug-journaler=20 2024-04-06T09:54:50.903 INFO:tasks.ceph.cluster2.osd.0.smithi192.stderr:2024-04-06T09:54:50.905+0000 7f78af6df640 -1 reset not still connected to 0x5604222dc1a0 2024-04-06T09:54:50.904 INFO:tasks.ceph.cluster2.osd.0.smithi192.stderr:2024-04-06T09:54:50.909+0000 7f78af6df640 -1 reset not still connected to 0x5604459921a0 2024-04-06T09:54:51.033 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:54:51.034+0000 7f0965192640 -1 reset not still connected to 0x5632a14ac820 2024-04-06T09:54:51.033 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:54:51.034+0000 7f0965192640 -1 reset not still connected to 0x5632a1988000 2024-04-06T09:54:51.033 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:54:51.034+0000 7f0965192640 -1 reset not still connected to 0x5632a1b73ee0 2024-04-06T09:54:51.033 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:54:51.034+0000 7f0965192640 -1 reset not still connected to 0x5632a2793520 2024-04-06T09:54:51.033 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:54:51.034+0000 7f0965192640 -1 reset not still connected to 0x5632a30f0270 2024-04-06T09:54:51.033 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:54:51.034+0000 7f0965192640 -1 reset not still connected to 0x5632a417e750 2024-04-06T09:54:51.033 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:54:51.034+0000 7f0965192640 -1 reset not still connected to 0x5632a4cda000 2024-04-06T09:54:51.033 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:54:51.034+0000 7f0965192640 -1 reset not still connected to 0x5632abc81d40 2024-04-06T09:54:51.033 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:54:51.034+0000 7f0965192640 -1 reset not still connected to 0x5632abcbe820 2024-04-06T09:54:51.033 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:54:51.034+0000 7f0965192640 -1 reset not still connected to 0x5632b9593110 2024-04-06T09:54:51.033 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:54:51.034+0000 7f0965192640 -1 reset not still connected to 0x5632cb3ac000 2024-04-06T09:54:51.034 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:54:51.034+0000 7f0965192640 -1 reset not still connected to 0x5632e16851e0 2024-04-06T09:54:51.058 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:54:51.058+0000 7f2e84d66640 -1 reset not still connected to 0x560d27c3edd0 2024-04-06T09:54:51.059 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:54:51.058+0000 7f2e84d66640 -1 reset not still connected to 0x560d27c3f2b0 2024-04-06T09:54:51.069 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:54:51.066+0000 7f2e84d66640 -1 reset not still connected to 0x560d2a17d450 2024-04-06T09:54:51.069 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:54:51.066+0000 7f2e84d66640 -1 reset not still connected to 0x560d2a17d6c0 2024-04-06T09:54:52.793 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:54:52.790+0000 7f0965192640 -1 reset not still connected to 0x5632a1988000 2024-04-06T09:54:52.803 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:54:52.802+0000 7f0965192640 -1 reset not still connected to 0x5632a1b73ee0 2024-04-06T09:54:52.803 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:54:52.802+0000 7f0965192640 -1 reset not still connected to 0x5632a2793520 2024-04-06T09:54:52.803 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:54:52.802+0000 7f0965192640 -1 reset not still connected to 0x5632a30f0270 2024-04-06T09:54:52.803 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:54:52.802+0000 7f0965192640 -1 reset not still connected to 0x5632a30f0f70 2024-04-06T09:54:52.804 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:54:52.802+0000 7f0965192640 -1 reset not still connected to 0x5632a417e750 2024-04-06T09:54:52.804 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:54:52.802+0000 7f0965192640 -1 reset not still connected to 0x5632a4cda000 2024-04-06T09:54:52.804 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:54:52.802+0000 7f0965192640 -1 reset not still connected to 0x5632a4cdbba0 2024-04-06T09:54:52.804 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:54:52.802+0000 7f0965192640 -1 reset not still connected to 0x5632a5e9dee0 2024-04-06T09:54:52.804 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:54:52.802+0000 7f0965192640 -1 reset not still connected to 0x5632abc812b0 2024-04-06T09:54:52.804 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:54:52.802+0000 7f0965192640 -1 reset not still connected to 0x5632abc81d40 2024-04-06T09:54:52.804 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:54:52.802+0000 7f0965192640 -1 reset not still connected to 0x5632abcbe820 2024-04-06T09:54:52.804 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:54:52.802+0000 7f0965192640 -1 reset not still connected to 0x5632b9593110 2024-04-06T09:54:52.804 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:54:52.802+0000 7f0965192640 -1 reset not still connected to 0x5632cb3ac000 2024-04-06T09:54:52.804 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:54:52.802+0000 7f0965192640 -1 reset not still connected to 0x5632dac08dd0 2024-04-06T09:54:52.804 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:54:52.802+0000 7f0965192640 -1 reset not still connected to 0x5632dac09040 2024-04-06T09:54:52.804 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:54:52.802+0000 7f0965192640 -1 reset not still connected to 0x5632e16851e0 2024-04-06T09:54:55.403 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:100 IOs, 63542272 bytes 2024-04-06T09:54:58.865 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:200 IOs, 136451072 bytes 2024-04-06T09:54:59.725 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:300 IOs, 170269696 bytes 2024-04-06T09:55:01.856 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ error_code=124 2024-04-06T09:55:01.856 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set -e 2024-04-06T09:55:01.856 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 124 -eq 124 ']' 2024-04-06T09:55:01.856 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:55:01.856 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-06T09:55:01.856 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ image=image_29 2024-04-06T09:55:01.856 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ create_image_and_enable_mirror cluster2 mirror image_29 journal 128M 2024-04-06T09:55:01.857 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-04-06T09:55:01.857 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:55:01.857 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:55:01.857 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:55:01.857 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_29 2024-04-06T09:55:01.857 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:55:01.857 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local mode=journal 2024-04-06T09:55:01.857 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n journal ']' 2024-04-06T09:55:01.857 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:55:01.857 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ create_image cluster2 mirror image_29 128M 2024-04-06T09:55:01.857 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-04-06T09:55:01.857 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:55:01.857 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:55:01.857 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:55:01.857 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_29 2024-04-06T09:55:01.857 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:55:01.857 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local size=128 2024-04-06T09:55:01.858 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n 128M ']' 2024-04-06T09:55:01.858 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ size=128M 2024-04-06T09:55:01.858 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:55:01.858 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,journaling mirror/image_29 2024-04-06T09:55:01.939 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' pool = image ']' 2024-04-06T09:55:01.939 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' mirror = mirror_parent ']' 2024-04-06T09:55:01.940 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n '' ']' 2024-04-06T09:55:01.940 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ stress_write_image cluster2 mirror image_29 2024-04-06T09:55:01.940 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-04-06T09:55:01.940 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:55:01.940 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_29 2024-04-06T09:55:01.940 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-04-06T09:55:01.993 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local duration=14 2024-04-06T09:55:01.993 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set +e 2024-04-06T09:55:01.993 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ timeout 14s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_29 --debug-rbd=20 --debug-journaler=20 2024-04-06T09:55:03.600 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:55:03.601+0000 7f0965192640 -1 reset not still connected to 0x5632abc81d40 2024-04-06T09:55:03.601 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:55:03.601+0000 7f0965192640 -1 reset not still connected to 0x5632abcbe820 2024-04-06T09:55:03.616 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:55:03.613+0000 7f0965192640 -1 reset not still connected to 0x5632cb3ac000 2024-04-06T09:55:03.617 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:55:03.613+0000 7f0965192640 -1 reset not still connected to 0x5632dac09040 2024-04-06T09:55:03.617 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:55:03.613+0000 7f0965192640 -1 reset not still connected to 0x5632e16851e0 2024-04-06T09:55:04.370 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:100 IOs, 37596160 bytes 2024-04-06T09:55:06.266 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:200 IOs, 57443328 bytes 2024-04-06T09:55:06.490 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:300 IOs, 58060800 bytes 2024-04-06T09:55:07.147 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi192.stderr:2024-04-06T09:55:07.149+0000 7f3d293cf640 -1 rbd::mirror::InstanceWatcher: C_NotifyInstanceRequest: 0x56066544c480 finish: resending after timeout 2024-04-06T09:55:07.148 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi192.stderr:2024-04-06T09:55:07.149+0000 7f3d293cf640 -1 rbd::mirror::InstanceWatcher: C_NotifyInstanceRequest: 0x56066bd9a600 finish: resending after timeout 2024-04-06T09:55:07.148 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi192.stderr:2024-04-06T09:55:07.149+0000 7f3d293cf640 -1 rbd::mirror::InstanceWatcher: C_NotifyInstanceRequest: 0x56066bd9b740 finish: resending after timeout 2024-04-06T09:55:08.513 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:400 IOs, 96729088 bytes 2024-04-06T09:55:09.281 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:55:09.281+0000 7f0965192640 -1 reset not still connected to 0x5632abc812b0 2024-04-06T09:55:09.284 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:55:09.285+0000 7f0965192640 -1 reset not still connected to 0x5632abc81d40 2024-04-06T09:55:09.284 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:55:09.285+0000 7f0965192640 -1 reset not still connected to 0x5632abcbe820 2024-04-06T09:55:09.286 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:55:09.285+0000 7f0965192640 -1 reset not still connected to 0x5632cb3ac000 2024-04-06T09:55:09.286 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:55:09.285+0000 7f0965192640 -1 reset not still connected to 0x5632dac08dd0 2024-04-06T09:55:09.291 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:55:09.289+0000 7f0965192640 -1 reset not still connected to 0x5632e1684ea0 2024-04-06T09:55:09.291 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:55:09.289+0000 7f0965192640 -1 reset not still connected to 0x5632e16851e0 2024-04-06T09:55:12.149 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi192.stderr:2024-04-06T09:55:12.153+0000 7f3d293cf640 -1 rbd::mirror::InstanceWatcher: C_NotifyInstanceRequest: 0x56066544c480 finish: resending after timeout 2024-04-06T09:55:12.150 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi192.stderr:2024-04-06T09:55:12.153+0000 7f3d293cf640 -1 rbd::mirror::InstanceWatcher: C_NotifyInstanceRequest: 0x56066bd9a600 finish: resending after timeout 2024-04-06T09:55:12.150 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi192.stderr:2024-04-06T09:55:12.153+0000 7f3d293cf640 -1 rbd::mirror::InstanceWatcher: C_NotifyInstanceRequest: 0x56066bd9b740 finish: resending after timeout 2024-04-06T09:55:13.517 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:500 IOs, 167016448 bytes 2024-04-06T09:55:14.651 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:55:14.649+0000 7f0965192640 -1 reset not still connected to 0x5632a1988000 2024-04-06T09:55:14.651 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:55:14.649+0000 7f0965192640 -1 reset not still connected to 0x5632a1b73ee0 2024-04-06T09:55:14.657 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:55:14.657+0000 7f0965192640 -1 reset not still connected to 0x5632a2793520 2024-04-06T09:55:14.657 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:55:14.657+0000 7f0965192640 -1 reset not still connected to 0x5632a30f0270 2024-04-06T09:55:14.658 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:55:14.657+0000 7f0965192640 -1 reset not still connected to 0x5632a417e750 2024-04-06T09:55:14.658 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:55:14.657+0000 7f0965192640 -1 reset not still connected to 0x5632a4cda000 2024-04-06T09:55:14.664 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:55:14.661+0000 7f0965192640 -1 reset not still connected to 0x5632a4cda9c0 2024-04-06T09:55:14.664 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:55:14.661+0000 7f0965192640 -1 reset not still connected to 0x5632a4cdbba0 2024-04-06T09:55:14.664 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:55:14.661+0000 7f0965192640 -1 reset not still connected to 0x5632a5e9dee0 2024-04-06T09:55:14.664 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:55:14.661+0000 7f0965192640 -1 reset not still connected to 0x5632a89f4750 2024-04-06T09:55:14.665 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:55:14.661+0000 7f0965192640 -1 reset not still connected to 0x5632abc812b0 2024-04-06T09:55:14.665 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:55:14.661+0000 7f0965192640 -1 reset not still connected to 0x5632abc81d40 2024-04-06T09:55:14.665 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:55:14.661+0000 7f0965192640 -1 reset not still connected to 0x5632abcbe820 2024-04-06T09:55:14.665 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:55:14.661+0000 7f0965192640 -1 reset not still connected to 0x5632b9593110 2024-04-06T09:55:14.665 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:55:14.661+0000 7f0965192640 -1 reset not still connected to 0x5632cb3ac000 2024-04-06T09:55:14.665 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:55:14.661+0000 7f0965192640 -1 reset not still connected to 0x5632dac08dd0 2024-04-06T09:55:14.665 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:55:14.661+0000 7f0965192640 -1 reset not still connected to 0x5632dac09040 2024-04-06T09:55:14.665 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:55:14.661+0000 7f0965192640 -1 reset not still connected to 0x5632e1684ea0 2024-04-06T09:55:14.665 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:55:14.661+0000 7f0965192640 -1 reset not still connected to 0x5632e16851e0 2024-04-06T09:55:16.021 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ error_code=124 2024-04-06T09:55:16.022 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set -e 2024-04-06T09:55:16.022 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 124 -eq 124 ']' 2024-04-06T09:55:16.022 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:55:16.022 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-06T09:55:16.022 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ image=image_30 2024-04-06T09:55:16.022 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ create_image_and_enable_mirror cluster2 mirror image_30 journal 128M 2024-04-06T09:55:16.022 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-04-06T09:55:16.022 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:55:16.022 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:55:16.022 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:55:16.022 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_30 2024-04-06T09:55:16.022 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:55:16.022 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local mode=journal 2024-04-06T09:55:16.023 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n journal ']' 2024-04-06T09:55:16.023 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:55:16.023 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ create_image cluster2 mirror image_30 128M 2024-04-06T09:55:16.023 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-04-06T09:55:16.023 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:55:16.023 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:55:16.023 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:55:16.023 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_30 2024-04-06T09:55:16.023 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:55:16.023 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local size=128 2024-04-06T09:55:16.023 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n 128M ']' 2024-04-06T09:55:16.023 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ size=128M 2024-04-06T09:55:16.023 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:55:16.023 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,journaling mirror/image_30 2024-04-06T09:55:16.081 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:55:16.077+0000 7f2e84d66640 -1 reset not still connected to 0x560d2a17d450 2024-04-06T09:55:16.081 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:55:16.077+0000 7f2e84d66640 -1 reset not still connected to 0x560d2a17d6c0 2024-04-06T09:55:16.082 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:55:16.081+0000 7f2e84d66640 -1 reset not still connected to 0x560d2a204b60 2024-04-06T09:55:16.082 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:55:16.081+0000 7f2e84d66640 -1 reset not still connected to 0x560d2a2336c0 2024-04-06T09:55:16.082 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:55:16.081+0000 7f2e84d66640 -1 reset not still connected to 0x560d2a570000 2024-04-06T09:55:16.088 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:55:16.085+0000 7f2e84d66640 -1 reset not still connected to 0x560d2a5705b0 2024-04-06T09:55:16.088 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:55:16.085+0000 7f2e84d66640 -1 reset not still connected to 0x560d368e4f70 2024-04-06T09:55:16.088 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:55:16.085+0000 7f2e84d66640 -1 reset not still connected to 0x560d36912ea0 2024-04-06T09:55:16.088 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:55:16.085+0000 7f2e84d66640 -1 reset not still connected to 0x560d381dd5f0 2024-04-06T09:55:16.088 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:55:16.085+0000 7f2e84d66640 -1 reset not still connected to 0x560d390dca90 2024-04-06T09:55:16.281 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' pool = image ']' 2024-04-06T09:55:16.281 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' mirror = mirror_parent ']' 2024-04-06T09:55:16.282 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n '' ']' 2024-04-06T09:55:16.282 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ stress_write_image cluster2 mirror image_30 2024-04-06T09:55:16.282 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-04-06T09:55:16.282 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:55:16.282 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_30 2024-04-06T09:55:16.282 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-04-06T09:55:16.295 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local duration=9 2024-04-06T09:55:16.295 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set +e 2024-04-06T09:55:16.295 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ timeout 9s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_30 --debug-rbd=20 --debug-journaler=20 2024-04-06T09:55:17.548 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi192.stderr:2024-04-06T09:55:17.548+0000 7f3d28bce640 -1 rbd::mirror::InstanceWatcher: C_NotifyInstanceRequest: 0x56066544c480 finish: resending after timeout 2024-04-06T09:55:17.548 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi192.stderr:2024-04-06T09:55:17.548+0000 7f3d28bce640 -1 rbd::mirror::InstanceWatcher: C_NotifyInstanceRequest: 0x56066bd9a600 finish: resending after timeout 2024-04-06T09:55:17.548 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi192.stderr:2024-04-06T09:55:17.548+0000 7f3d28bce640 -1 rbd::mirror::InstanceWatcher: C_NotifyInstanceRequest: 0x56066bd9b740 finish: resending after timeout 2024-04-06T09:55:19.811 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:100 IOs, 49941504 bytes 2024-04-06T09:55:20.081 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:55:20.081+0000 7f0965192640 -1 reset not still connected to 0x5632a5e9dee0 2024-04-06T09:55:20.083 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:55:20.085+0000 7f0965192640 -1 reset not still connected to 0x5632abcbe820 2024-04-06T09:55:20.083 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:55:20.085+0000 7f0965192640 -1 reset not still connected to 0x5632e16851e0 2024-04-06T09:55:23.359 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:200 IOs, 75654144 bytes 2024-04-06T09:55:24.609 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:55:24.601+0000 7f0965192640 -1 reset not still connected to 0x56329f99c0d0 2024-04-06T09:55:24.609 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:55:24.601+0000 7f0965192640 -1 reset not still connected to 0x5632a0ff5110 2024-04-06T09:55:24.609 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:55:24.601+0000 7f0965192640 -1 reset not still connected to 0x5632a14ac820 2024-04-06T09:55:24.609 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:55:24.601+0000 7f0965192640 -1 reset not still connected to 0x5632a1988000 2024-04-06T09:55:24.611 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:55:24.613+0000 7f0965192640 -1 reset not still connected to 0x5632a1989040 2024-04-06T09:55:24.611 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:55:24.613+0000 7f0965192640 -1 reset not still connected to 0x5632a1b73ee0 2024-04-06T09:55:24.611 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:55:24.613+0000 7f0965192640 -1 reset not still connected to 0x5632a2793520 2024-04-06T09:55:24.612 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:55:24.613+0000 7f0965192640 -1 reset not still connected to 0x5632a30f0f70 2024-04-06T09:55:24.612 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:55:24.613+0000 7f0965192640 -1 reset not still connected to 0x5632a417e750 2024-04-06T09:55:24.612 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:55:24.613+0000 7f0965192640 -1 reset not still connected to 0x5632a4cda000 2024-04-06T09:55:24.612 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:55:24.613+0000 7f0965192640 -1 reset not still connected to 0x5632a4cda9c0 2024-04-06T09:55:24.612 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:55:24.613+0000 7f0965192640 -1 reset not still connected to 0x5632a4cdbba0 2024-04-06T09:55:24.612 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:55:24.613+0000 7f0965192640 -1 reset not still connected to 0x5632a5e9dee0 2024-04-06T09:55:24.614 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:55:24.613+0000 7f0965192640 -1 reset not still connected to 0x5632a89f4750 2024-04-06T09:55:24.616 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:55:24.613+0000 7f0965192640 -1 reset not still connected to 0x5632abc812b0 2024-04-06T09:55:24.616 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:55:24.613+0000 7f0965192640 -1 reset not still connected to 0x5632abc81d40 2024-04-06T09:55:24.616 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:55:24.613+0000 7f0965192640 -1 reset not still connected to 0x5632abcbe820 2024-04-06T09:55:24.616 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:55:24.613+0000 7f0965192640 -1 reset not still connected to 0x5632b9592000 2024-04-06T09:55:24.616 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:55:24.613+0000 7f0965192640 -1 reset not still connected to 0x5632b9593110 2024-04-06T09:55:24.616 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:55:24.613+0000 7f0965192640 -1 reset not still connected to 0x5632cb3ac000 2024-04-06T09:55:24.616 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:55:24.613+0000 7f0965192640 -1 reset not still connected to 0x5632dac08dd0 2024-04-06T09:55:24.616 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:55:24.613+0000 7f0965192640 -1 reset not still connected to 0x5632dac09040 2024-04-06T09:55:24.616 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:55:24.613+0000 7f0965192640 -1 reset not still connected to 0x5632e1684ea0 2024-04-06T09:55:24.616 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:55:24.613+0000 7f0965192640 -1 reset not still connected to 0x5632e16851e0 2024-04-06T09:55:24.617 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:55:24.613+0000 7f0965192640 -1 reset not still connected to 0x56329f99c0d0 2024-04-06T09:55:24.617 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:55:24.613+0000 7f0965192640 -1 reset not still connected to 0x5632a417e750 2024-04-06T09:55:24.617 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:55:24.613+0000 7f0965192640 -1 reset not still connected to 0x5632a5e9dee0 2024-04-06T09:55:24.617 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:55:24.613+0000 7f0965192640 -1 reset not still connected to 0x5632abcbe820 2024-04-06T09:55:24.617 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:55:24.613+0000 7f0965192640 -1 reset not still connected to 0x5632e16851e0 2024-04-06T09:55:25.306 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ error_code=124 2024-04-06T09:55:25.306 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set -e 2024-04-06T09:55:25.306 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 124 -eq 124 ']' 2024-04-06T09:55:25.306 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:55:25.306 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-06T09:55:25.306 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ image=image_31 2024-04-06T09:55:25.306 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ create_image_and_enable_mirror cluster2 mirror image_31 journal 128M 2024-04-06T09:55:25.307 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-04-06T09:55:25.307 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:55:25.307 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:55:25.307 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:55:25.307 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_31 2024-04-06T09:55:25.307 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:55:25.307 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local mode=journal 2024-04-06T09:55:25.307 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n journal ']' 2024-04-06T09:55:25.307 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:55:25.307 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ create_image cluster2 mirror image_31 128M 2024-04-06T09:55:25.307 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-04-06T09:55:25.307 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:55:25.307 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:55:25.307 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:55:25.307 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_31 2024-04-06T09:55:25.308 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:55:25.308 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local size=128 2024-04-06T09:55:25.308 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n 128M ']' 2024-04-06T09:55:25.308 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ size=128M 2024-04-06T09:55:25.308 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:55:25.308 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,journaling mirror/image_31 2024-04-06T09:55:25.587 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' pool = image ']' 2024-04-06T09:55:25.587 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' mirror = mirror_parent ']' 2024-04-06T09:55:25.587 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n '' ']' 2024-04-06T09:55:25.587 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ stress_write_image cluster2 mirror image_31 2024-04-06T09:55:25.587 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-04-06T09:55:25.587 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:55:25.587 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_31 2024-04-06T09:55:25.587 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-04-06T09:55:26.207 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-04-06T09:55:26.205+0000 7fd1b6210640 -1 reset not still connected to 0x55ef9e88f520 2024-04-06T09:55:26.211 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-04-06T09:55:26.213+0000 7fd1b6210640 -1 reset not still connected to 0x55efbb630d00 2024-04-06T09:55:26.268 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local duration=5 2024-04-06T09:55:26.268 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set +e 2024-04-06T09:55:26.268 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ timeout 5s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_31 --debug-rbd=20 --debug-journaler=20 2024-04-06T09:55:26.610 INFO:tasks.ceph.cluster2.osd.2.smithi192.stderr:2024-04-06T09:55:26.612+0000 7fa534aba640 -1 reset not still connected to 0x55933fc2dd40 2024-04-06T09:55:26.610 INFO:tasks.ceph.cluster2.osd.2.smithi192.stderr:2024-04-06T09:55:26.612+0000 7fa534aba640 -1 reset not still connected to 0x559341658b60 2024-04-06T09:55:26.610 INFO:tasks.ceph.cluster2.osd.2.smithi192.stderr:2024-04-06T09:55:26.612+0000 7fa534aba640 -1 reset not still connected to 0x55934811f6c0 2024-04-06T09:55:27.912 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:55:27.905+0000 7f0965192640 -1 reset not still connected to 0x5632e16851e0 2024-04-06T09:55:28.218 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:55:28.217+0000 7f0965192640 -1 reset not still connected to 0x5632a1988000 2024-04-06T09:55:28.218 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:55:28.217+0000 7f0965192640 -1 reset not still connected to 0x5632a2793520 2024-04-06T09:55:28.218 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:55:28.217+0000 7f0965192640 -1 reset not still connected to 0x5632a417e750 2024-04-06T09:55:28.219 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:55:28.221+0000 7f0965192640 -1 reset not still connected to 0x5632a4cda000 2024-04-06T09:55:28.220 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:55:28.221+0000 7f0965192640 -1 reset not still connected to 0x5632a4cdbba0 2024-04-06T09:55:28.223 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:55:28.221+0000 7f0965192640 -1 reset not still connected to 0x5632abc812b0 2024-04-06T09:55:28.223 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:55:28.221+0000 7f0965192640 -1 reset not still connected to 0x5632abc81d40 2024-04-06T09:55:28.223 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:55:28.221+0000 7f0965192640 -1 reset not still connected to 0x5632abcbe820 2024-04-06T09:55:28.223 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:55:28.221+0000 7f0965192640 -1 reset not still connected to 0x5632b9592000 2024-04-06T09:55:28.223 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:55:28.221+0000 7f0965192640 -1 reset not still connected to 0x5632c0cfa000 2024-04-06T09:55:28.224 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:55:28.225+0000 7f0965192640 -1 reset not still connected to 0x5632cb3ac000 2024-04-06T09:55:28.224 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:55:28.225+0000 7f0965192640 -1 reset not still connected to 0x5632dac08dd0 2024-04-06T09:55:28.235 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:55:28.233+0000 7f0965192640 -1 reset not still connected to 0x5632dac09040 2024-04-06T09:55:28.236 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:55:28.233+0000 7f0965192640 -1 reset not still connected to 0x5632e1684ea0 2024-04-06T09:55:28.236 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:55:28.233+0000 7f0965192640 -1 reset not still connected to 0x5632e16851e0 2024-04-06T09:55:28.858 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:55:28.857+0000 7f2e84d66640 -1 reset not still connected to 0x560d27c3edd0 2024-04-06T09:55:28.858 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:55:28.857+0000 7f2e84d66640 -1 reset not still connected to 0x560d27c3f2b0 2024-04-06T09:55:28.859 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:55:28.857+0000 7f2e84d66640 -1 reset not still connected to 0x560d29cd8f70 2024-04-06T09:55:28.859 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:55:28.857+0000 7f2e84d66640 -1 reset not still connected to 0x560d2a17d6c0 2024-04-06T09:55:28.859 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:55:28.857+0000 7f2e84d66640 -1 reset not still connected to 0x560d2a2336c0 2024-04-06T09:55:29.937 INFO:tasks.ceph.cluster2.osd.0.smithi192.stderr:2024-04-06T09:55:29.940+0000 7f78af6df640 -1 reset not still connected to 0x5604216ff790 2024-04-06T09:55:29.945 INFO:tasks.ceph.cluster2.osd.0.smithi192.stderr:2024-04-06T09:55:29.944+0000 7f78af6df640 -1 reset not still connected to 0x560422aa4ea0 2024-04-06T09:55:29.946 INFO:tasks.ceph.cluster2.osd.0.smithi192.stderr:2024-04-06T09:55:29.944+0000 7f78af6df640 -1 reset not still connected to 0x560422aa52b0 2024-04-06T09:55:29.946 INFO:tasks.ceph.cluster2.osd.0.smithi192.stderr:2024-04-06T09:55:29.944+0000 7f78af6df640 -1 reset not still connected to 0x5604239371e0 2024-04-06T09:55:29.953 INFO:tasks.ceph.cluster2.osd.0.smithi192.stderr:2024-04-06T09:55:29.956+0000 7f78af6df640 -1 reset not still connected to 0x56042bd3fc70 2024-04-06T09:55:29.954 INFO:tasks.ceph.cluster2.osd.0.smithi192.stderr:2024-04-06T09:55:29.956+0000 7f78af6df640 -1 reset not still connected to 0x560436199450 2024-04-06T09:55:29.954 INFO:tasks.ceph.cluster2.osd.0.smithi192.stderr:2024-04-06T09:55:29.956+0000 7f78af6df640 -1 reset not still connected to 0x56043d1216c0 2024-04-06T09:55:29.954 INFO:tasks.ceph.cluster2.osd.0.smithi192.stderr:2024-04-06T09:55:29.956+0000 7f78af6df640 -1 reset not still connected to 0x560444f6c820 2024-04-06T09:55:29.954 INFO:tasks.ceph.cluster2.osd.0.smithi192.stderr:2024-04-06T09:55:29.956+0000 7f78af6df640 -1 reset not still connected to 0x56044cc1e1a0 2024-04-06T09:55:29.954 INFO:tasks.ceph.cluster2.osd.0.smithi192.stderr:2024-04-06T09:55:29.956+0000 7f78af6df640 -1 reset not still connected to 0x56044cc1fa00 2024-04-06T09:55:31.277 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ error_code=124 2024-04-06T09:55:31.278 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set -e 2024-04-06T09:55:31.278 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 124 -eq 124 ']' 2024-04-06T09:55:31.278 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:55:31.278 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-06T09:55:31.278 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ image=image_32 2024-04-06T09:55:31.278 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ create_image_and_enable_mirror cluster2 mirror image_32 journal 128M 2024-04-06T09:55:31.278 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-04-06T09:55:31.278 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:55:31.278 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:55:31.278 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:55:31.278 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_32 2024-04-06T09:55:31.278 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:55:31.278 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local mode=journal 2024-04-06T09:55:31.279 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n journal ']' 2024-04-06T09:55:31.279 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:55:31.279 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ create_image cluster2 mirror image_32 128M 2024-04-06T09:55:31.279 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-04-06T09:55:31.279 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:55:31.279 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:55:31.279 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:55:31.279 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_32 2024-04-06T09:55:31.279 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:55:31.279 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local size=128 2024-04-06T09:55:31.279 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n 128M ']' 2024-04-06T09:55:31.279 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ size=128M 2024-04-06T09:55:31.279 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:55:31.279 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,journaling mirror/image_32 2024-04-06T09:55:31.692 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:55:31.693+0000 7f0965192640 -1 reset not still connected to 0x5632cb3ac000 2024-04-06T09:55:31.692 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:55:31.693+0000 7f0965192640 -1 reset not still connected to 0x5632dac08dd0 2024-04-06T09:55:31.692 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:55:31.693+0000 7f0965192640 -1 reset not still connected to 0x5632dac09040 2024-04-06T09:55:31.692 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:55:31.693+0000 7f0965192640 -1 reset not still connected to 0x5632e1684ea0 2024-04-06T09:55:31.693 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:55:31.693+0000 7f0965192640 -1 reset not still connected to 0x5632e16851e0 2024-04-06T09:55:31.789 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' pool = image ']' 2024-04-06T09:55:31.789 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' mirror = mirror_parent ']' 2024-04-06T09:55:31.789 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n '' ']' 2024-04-06T09:55:31.789 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ stress_write_image cluster2 mirror image_32 2024-04-06T09:55:31.790 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-04-06T09:55:31.790 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:55:31.790 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_32 2024-04-06T09:55:31.815 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-04-06T09:55:31.948 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local duration=7 2024-04-06T09:55:31.948 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set +e 2024-04-06T09:55:31.948 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ timeout 7s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_32 --debug-rbd=20 --debug-journaler=20 2024-04-06T09:55:33.398 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:55:33.397+0000 7f2e84d66640 -1 reset not still connected to 0x560d2a5705b0 2024-04-06T09:55:33.400 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:55:33.401+0000 7f2e84d66640 -1 reset not still connected to 0x560d368e4f70 2024-04-06T09:55:33.402 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:55:33.401+0000 7f2e84d66640 -1 reset not still connected to 0x560d381dd5f0 2024-04-06T09:55:36.060 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:55:36.061+0000 7f2e84d66640 -1 reset not still connected to 0x560d27839380 2024-04-06T09:55:36.068 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:55:36.069+0000 7f2e84d66640 -1 reset not still connected to 0x560d27c3edd0 2024-04-06T09:55:36.069 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:55:36.069+0000 7f2e84d66640 -1 reset not still connected to 0x560d27c3f2b0 2024-04-06T09:55:36.069 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:55:36.069+0000 7f2e84d66640 -1 reset not still connected to 0x560d281875f0 2024-04-06T09:55:36.069 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:55:36.069+0000 7f2e84d66640 -1 reset not still connected to 0x560d294908f0 2024-04-06T09:55:36.072 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:55:36.073+0000 7f2e84d66640 -1 reset not still connected to 0x560d29cd8f70 2024-04-06T09:55:36.073 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:55:36.073+0000 7f2e84d66640 -1 reset not still connected to 0x560d29cd9ba0 2024-04-06T09:55:36.073 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:55:36.073+0000 7f2e84d66640 -1 reset not still connected to 0x560d29d5a340 2024-04-06T09:55:36.073 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:55:36.073+0000 7f2e84d66640 -1 reset not still connected to 0x560d29d5b520 2024-04-06T09:55:36.073 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:55:36.073+0000 7f2e84d66640 -1 reset not still connected to 0x560d2a17d450 2024-04-06T09:55:36.073 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:55:36.073+0000 7f2e84d66640 -1 reset not still connected to 0x560d2a17d6c0 2024-04-06T09:55:36.074 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:55:36.073+0000 7f2e84d66640 -1 reset not still connected to 0x560d2a204b60 2024-04-06T09:55:36.074 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:55:36.073+0000 7f2e84d66640 -1 reset not still connected to 0x560d2a2336c0 2024-04-06T09:55:36.079 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:55:36.081+0000 7f2e84d66640 -1 reset not still connected to 0x560d2a570000 2024-04-06T09:55:36.079 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:55:36.081+0000 7f2e84d66640 -1 reset not still connected to 0x560d2a5705b0 2024-04-06T09:55:36.082 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:55:36.081+0000 7f2e84d66640 -1 reset not still connected to 0x560d368e4f70 2024-04-06T09:55:36.082 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:55:36.081+0000 7f2e84d66640 -1 reset not still connected to 0x560d36912ea0 2024-04-06T09:55:36.082 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:55:36.081+0000 7f2e84d66640 -1 reset not still connected to 0x560d381dd5f0 2024-04-06T09:55:36.082 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:55:36.081+0000 7f2e84d66640 -1 reset not still connected to 0x560d390dca90 2024-04-06T09:55:36.830 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:55:36.829+0000 7f2e84d66640 -1 reset not still connected to 0x560d27567380 2024-04-06T09:55:36.832 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:55:36.829+0000 7f2e84d66640 -1 reset not still connected to 0x560d27839380 2024-04-06T09:55:36.832 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:55:36.829+0000 7f2e84d66640 -1 reset not still connected to 0x560d27839930 2024-04-06T09:55:36.833 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:55:36.833+0000 7f2e84d66640 -1 reset not still connected to 0x560d27c3edd0 2024-04-06T09:55:36.839 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:55:36.837+0000 7f2e84d66640 -1 reset not still connected to 0x560d27c3f2b0 2024-04-06T09:55:36.839 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:55:36.837+0000 7f2e84d66640 -1 reset not still connected to 0x560d281875f0 2024-04-06T09:55:36.839 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:55:36.837+0000 7f2e84d66640 -1 reset not still connected to 0x560d294908f0 2024-04-06T09:55:36.839 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:55:36.837+0000 7f2e84d66640 -1 reset not still connected to 0x560d29cd8f70 2024-04-06T09:55:36.839 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:55:36.837+0000 7f2e84d66640 -1 reset not still connected to 0x560d29cd9ba0 2024-04-06T09:55:36.839 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:55:36.837+0000 7f2e84d66640 -1 reset not still connected to 0x560d29d5a340 2024-04-06T09:55:36.839 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:55:36.837+0000 7f2e84d66640 -1 reset not still connected to 0x560d29d5b520 2024-04-06T09:55:36.840 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:55:36.841+0000 7f2e84d66640 -1 reset not still connected to 0x560d2a17d450 2024-04-06T09:55:36.840 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:55:36.841+0000 7f2e84d66640 -1 reset not still connected to 0x560d2a17d6c0 2024-04-06T09:55:36.840 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:55:36.841+0000 7f2e84d66640 -1 reset not still connected to 0x560d2a204b60 2024-04-06T09:55:36.840 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:55:36.841+0000 7f2e84d66640 -1 reset not still connected to 0x560d2a2336c0 2024-04-06T09:55:36.841 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:55:36.841+0000 7f2e84d66640 -1 reset not still connected to 0x560d2a570000 2024-04-06T09:55:36.841 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:55:36.841+0000 7f2e84d66640 -1 reset not still connected to 0x560d2a5705b0 2024-04-06T09:55:36.844 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:55:36.845+0000 7f2e84d66640 -1 reset not still connected to 0x560d368e4f70 2024-04-06T09:55:36.844 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:55:36.845+0000 7f2e84d66640 -1 reset not still connected to 0x560d36912ea0 2024-04-06T09:55:36.844 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:55:36.845+0000 7f2e84d66640 -1 reset not still connected to 0x560d381dd5f0 2024-04-06T09:55:36.844 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:55:36.845+0000 7f2e84d66640 -1 reset not still connected to 0x560d390dca90 2024-04-06T09:55:37.227 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:100 IOs, 59219968 bytes 2024-04-06T09:55:38.996 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ error_code=124 2024-04-06T09:55:38.997 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set -e 2024-04-06T09:55:38.997 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 124 -eq 124 ']' 2024-04-06T09:55:38.997 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:55:38.997 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-06T09:55:38.997 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ image=image_33 2024-04-06T09:55:38.997 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ create_image_and_enable_mirror cluster2 mirror image_33 journal 128M 2024-04-06T09:55:38.997 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-04-06T09:55:38.997 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:55:38.998 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:55:38.998 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:55:38.998 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_33 2024-04-06T09:55:38.998 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:55:38.998 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local mode=journal 2024-04-06T09:55:38.998 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n journal ']' 2024-04-06T09:55:38.998 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:55:38.998 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ create_image cluster2 mirror image_33 128M 2024-04-06T09:55:38.998 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-04-06T09:55:38.998 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:55:38.998 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:55:38.998 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:55:38.998 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_33 2024-04-06T09:55:38.999 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:55:38.999 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local size=128 2024-04-06T09:55:38.999 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n 128M ']' 2024-04-06T09:55:38.999 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ size=128M 2024-04-06T09:55:38.999 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:55:38.999 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,journaling mirror/image_33 2024-04-06T09:55:39.180 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' pool = image ']' 2024-04-06T09:55:39.180 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' mirror = mirror_parent ']' 2024-04-06T09:55:39.180 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n '' ']' 2024-04-06T09:55:39.180 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ stress_write_image cluster2 mirror image_33 2024-04-06T09:55:39.181 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-04-06T09:55:39.181 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:55:39.181 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_33 2024-04-06T09:55:39.181 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-04-06T09:55:39.182 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local duration=8 2024-04-06T09:55:39.182 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set +e 2024-04-06T09:55:39.182 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ timeout 8s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_33 --debug-rbd=20 --debug-journaler=20 2024-04-06T09:55:39.752 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi192.stderr:2024-04-06T09:55:39.752+0000 7f3d28bce640 -1 rbd::mirror::InstanceWatcher: C_NotifyInstanceRequest: 0x56066bd9a600 finish: resending after timeout 2024-04-06T09:55:39.758 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi192.stderr:2024-04-06T09:55:39.760+0000 7f3d293cf640 -1 rbd::mirror::InstanceWatcher: C_NotifyInstanceRequest: 0x560668d73200 finish: resending after timeout 2024-04-06T09:55:40.528 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:55:40.525+0000 7f2e84d66640 -1 reset not still connected to 0x560d27567380 2024-04-06T09:55:40.533 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:55:40.533+0000 7f2e84d66640 -1 reset not still connected to 0x560d29d5b520 2024-04-06T09:55:40.533 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:55:40.533+0000 7f2e84d66640 -1 reset not still connected to 0x560d2a17d450 2024-04-06T09:55:40.533 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:55:40.533+0000 7f2e84d66640 -1 reset not still connected to 0x560d2a17d6c0 2024-04-06T09:55:40.538 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:55:40.537+0000 7f2e84d66640 -1 reset not still connected to 0x560d2a2336c0 2024-04-06T09:55:40.538 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:55:40.537+0000 7f2e84d66640 -1 reset not still connected to 0x560d2a5705b0 2024-04-06T09:55:40.539 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:55:40.537+0000 7f2e84d66640 -1 reset not still connected to 0x560d368e4f70 2024-04-06T09:55:40.539 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:55:40.541+0000 7f2e84d66640 -1 reset not still connected to 0x560d36912ea0 2024-04-06T09:55:40.539 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:55:40.541+0000 7f2e84d66640 -1 reset not still connected to 0x560d381dd5f0 2024-04-06T09:55:40.642 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi192.stderr:2024-04-06T09:55:40.644+0000 7f3d293cf640 -1 rbd::mirror::InstanceWatcher: C_NotifyInstanceRequest: 0x56067b2cdec0 finish: resending after timeout 2024-04-06T09:55:41.908 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:55:41.905+0000 7f0965192640 -1 reset not still connected to 0x5632a417e750 2024-04-06T09:55:41.913 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:55:41.905+0000 7f0965192640 -1 reset not still connected to 0x5632a5e9dee0 2024-04-06T09:55:41.914 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:55:41.909+0000 7f0965192640 -1 reset not still connected to 0x5632abc812b0 2024-04-06T09:55:41.914 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:55:41.909+0000 7f0965192640 -1 reset not still connected to 0x5632abcbe820 2024-04-06T09:55:41.914 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:55:41.909+0000 7f0965192640 -1 reset not still connected to 0x5632c0cfa000 2024-04-06T09:55:41.914 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:55:41.909+0000 7f0965192640 -1 reset not still connected to 0x5632cb3ac000 2024-04-06T09:55:41.915 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:55:41.909+0000 7f0965192640 -1 reset not still connected to 0x5632dac08dd0 2024-04-06T09:55:41.915 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:55:41.913+0000 7f0965192640 -1 reset not still connected to 0x5632dac09040 2024-04-06T09:55:41.916 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:55:41.913+0000 7f0965192640 -1 reset not still connected to 0x5632e1684ea0 2024-04-06T09:55:41.920 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:55:41.921+0000 7f0965192640 -1 reset not still connected to 0x5632e16851e0 2024-04-06T09:55:42.156 INFO:tasks.ceph.cluster2.osd.2.smithi192.stderr:2024-04-06T09:55:42.156+0000 7fa534aba640 -1 reset not still connected to 0x559338f41040 2024-04-06T09:55:42.156 INFO:tasks.ceph.cluster2.osd.2.smithi192.stderr:2024-04-06T09:55:42.156+0000 7fa534aba640 -1 reset not still connected to 0x55933ce91110 2024-04-06T09:55:42.156 INFO:tasks.ceph.cluster2.osd.2.smithi192.stderr:2024-04-06T09:55:42.156+0000 7fa534aba640 -1 reset not still connected to 0x55933ecf01a0 2024-04-06T09:55:42.156 INFO:tasks.ceph.cluster2.osd.2.smithi192.stderr:2024-04-06T09:55:42.156+0000 7fa534aba640 -1 reset not still connected to 0x55933fc2dd40 2024-04-06T09:55:42.157 INFO:tasks.ceph.cluster2.osd.2.smithi192.stderr:2024-04-06T09:55:42.156+0000 7fa534aba640 -1 reset not still connected to 0x559341658b60 2024-04-06T09:55:42.157 INFO:tasks.ceph.cluster2.osd.2.smithi192.stderr:2024-04-06T09:55:42.156+0000 7fa534aba640 -1 reset not still connected to 0x55934811eb60 2024-04-06T09:55:42.157 INFO:tasks.ceph.cluster2.osd.2.smithi192.stderr:2024-04-06T09:55:42.156+0000 7fa534aba640 -1 reset not still connected to 0x55934811f6c0 2024-04-06T09:55:42.157 INFO:tasks.ceph.cluster2.osd.2.smithi192.stderr:2024-04-06T09:55:42.156+0000 7fa534aba640 -1 reset not still connected to 0x55935bfc9d40 2024-04-06T09:55:42.755 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:100 IOs, 34884608 bytes 2024-04-06T09:55:43.418 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:200 IOs, 40812544 bytes 2024-04-06T09:55:44.865 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:300 IOs, 66586624 bytes 2024-04-06T09:55:47.001 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:55:46.997+0000 7f2e84d66640 -1 reset not still connected to 0x560d2722e5b0 2024-04-06T09:55:47.009 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:55:47.005+0000 7f2e84d66640 -1 reset not still connected to 0x560d27566ea0 2024-04-06T09:55:47.009 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:55:47.005+0000 7f2e84d66640 -1 reset not still connected to 0x560d27567380 2024-04-06T09:55:47.009 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:55:47.005+0000 7f2e84d66640 -1 reset not still connected to 0x560d27839380 2024-04-06T09:55:47.009 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:55:47.009+0000 7f2e84d66640 -1 reset not still connected to 0x560d27c3edd0 2024-04-06T09:55:47.009 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:55:47.009+0000 7f2e84d66640 -1 reset not still connected to 0x560d27c3f2b0 2024-04-06T09:55:47.010 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:55:47.009+0000 7f2e84d66640 -1 reset not still connected to 0x560d294908f0 2024-04-06T09:55:47.011 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:55:47.009+0000 7f2e84d66640 -1 reset not still connected to 0x560d29cd8f70 2024-04-06T09:55:47.011 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:55:47.009+0000 7f2e84d66640 -1 reset not still connected to 0x560d29cd9ba0 2024-04-06T09:55:47.011 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:55:47.009+0000 7f2e84d66640 -1 reset not still connected to 0x560d29d5b520 2024-04-06T09:55:47.011 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:55:47.009+0000 7f2e84d66640 -1 reset not still connected to 0x560d2a17d450 2024-04-06T09:55:47.011 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:55:47.009+0000 7f2e84d66640 -1 reset not still connected to 0x560d2a17d6c0 2024-04-06T09:55:47.011 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:55:47.009+0000 7f2e84d66640 -1 reset not still connected to 0x560d2a204b60 2024-04-06T09:55:47.011 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:55:47.009+0000 7f2e84d66640 -1 reset not still connected to 0x560d2a2336c0 2024-04-06T09:55:47.011 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:55:47.009+0000 7f2e84d66640 -1 reset not still connected to 0x560d2a5705b0 2024-04-06T09:55:47.011 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:55:47.009+0000 7f2e84d66640 -1 reset not still connected to 0x560d368e4f70 2024-04-06T09:55:47.012 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:55:47.009+0000 7f2e84d66640 -1 reset not still connected to 0x560d36912ea0 2024-04-06T09:55:47.012 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:55:47.009+0000 7f2e84d66640 -1 reset not still connected to 0x560d381dd5f0 2024-04-06T09:55:47.012 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:55:47.009+0000 7f2e84d66640 -1 reset not still connected to 0x560d50adba00 2024-04-06T09:55:47.196 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ error_code=124 2024-04-06T09:55:47.197 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set -e 2024-04-06T09:55:47.198 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 124 -eq 124 ']' 2024-04-06T09:55:47.199 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:55:47.199 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-06T09:55:47.199 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ image=image_34 2024-04-06T09:55:47.199 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ create_image_and_enable_mirror cluster2 mirror image_34 journal 128M 2024-04-06T09:55:47.199 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-04-06T09:55:47.199 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:55:47.199 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:55:47.199 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:55:47.199 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_34 2024-04-06T09:55:47.199 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:55:47.199 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local mode=journal 2024-04-06T09:55:47.199 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n journal ']' 2024-04-06T09:55:47.200 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:55:47.200 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ create_image cluster2 mirror image_34 128M 2024-04-06T09:55:47.200 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-04-06T09:55:47.200 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:55:47.200 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:55:47.200 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:55:47.200 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_34 2024-04-06T09:55:47.200 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:55:47.200 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local size=128 2024-04-06T09:55:47.200 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n 128M ']' 2024-04-06T09:55:47.200 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ size=128M 2024-04-06T09:55:47.200 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:55:47.201 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,journaling mirror/image_34 2024-04-06T09:55:47.297 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' pool = image ']' 2024-04-06T09:55:47.297 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' mirror = mirror_parent ']' 2024-04-06T09:55:47.297 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n '' ']' 2024-04-06T09:55:47.297 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ stress_write_image cluster2 mirror image_34 2024-04-06T09:55:47.297 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-04-06T09:55:47.297 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:55:47.297 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_34 2024-04-06T09:55:47.297 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-04-06T09:55:47.298 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local duration=10 2024-04-06T09:55:47.298 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set +e 2024-04-06T09:55:47.299 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ timeout 10s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_34 --debug-rbd=20 --debug-journaler=20 2024-04-06T09:55:50.641 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.0 with skip (0.8156193347442152) > weight (0.25) 2024-04-06T09:55:50.641 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.1 with skip (0.31486068507349796) > weight (0.25) 2024-04-06T09:55:50.641 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.2 with skip (0.5539763816852896) > weight (0.25) 2024-04-06T09:55:50.641 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.3 with skip (0.7805068826977491) > weight (0.25) 2024-04-06T09:55:50.641 INFO:tasks.rbd_mirror_thrash:waiting for 99 secs before thrashing 2024-04-06T09:55:50.774 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:100 IOs, 58341376 bytes 2024-04-06T09:55:51.028 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:55:51.029+0000 7f0965192640 -1 reset not still connected to 0x5632a417e750 2024-04-06T09:55:51.040 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:55:51.041+0000 7f0965192640 -1 reset not still connected to 0x5632a4cdbba0 2024-04-06T09:55:51.040 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:55:51.041+0000 7f0965192640 -1 reset not still connected to 0x5632a5e9dee0 2024-04-06T09:55:51.040 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:55:51.041+0000 7f0965192640 -1 reset not still connected to 0x5632a89f4750 2024-04-06T09:55:51.040 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:55:51.041+0000 7f0965192640 -1 reset not still connected to 0x5632abc812b0 2024-04-06T09:55:51.040 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:55:51.041+0000 7f0965192640 -1 reset not still connected to 0x5632abc81d40 2024-04-06T09:55:51.040 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:55:51.041+0000 7f0965192640 -1 reset not still connected to 0x5632abcbe820 2024-04-06T09:55:51.040 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:55:51.041+0000 7f0965192640 -1 reset not still connected to 0x5632b9592000 2024-04-06T09:55:51.040 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:55:51.041+0000 7f0965192640 -1 reset not still connected to 0x5632c0cfa000 2024-04-06T09:55:51.040 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:55:51.041+0000 7f0965192640 -1 reset not still connected to 0x5632dac09040 2024-04-06T09:55:51.041 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:55:51.041+0000 7f0965192640 -1 reset not still connected to 0x5632e1684ea0 2024-04-06T09:55:51.041 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:55:51.041+0000 7f0965192640 -1 reset not still connected to 0x5632e16851e0 2024-04-06T09:55:51.442 INFO:tasks.ceph.cluster2.osd.2.smithi192.stderr:2024-04-06T09:55:51.444+0000 7fa534aba640 -1 reset not still connected to 0x55933ce91110 2024-04-06T09:55:51.687 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:55:51.677+0000 7f0965192640 -1 reset not still connected to 0x5632a0ff4410 2024-04-06T09:55:51.687 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:55:51.677+0000 7f0965192640 -1 reset not still connected to 0x5632a0ff5110 2024-04-06T09:55:51.688 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:55:51.677+0000 7f0965192640 -1 reset not still connected to 0x5632a1988000 2024-04-06T09:55:51.688 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:55:51.677+0000 7f0965192640 -1 reset not still connected to 0x5632a1989040 2024-04-06T09:55:51.690 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:55:51.689+0000 7f0965192640 -1 reset not still connected to 0x5632a1b73ee0 2024-04-06T09:55:51.690 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:55:51.689+0000 7f0965192640 -1 reset not still connected to 0x5632a2793520 2024-04-06T09:55:51.690 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:55:51.689+0000 7f0965192640 -1 reset not still connected to 0x5632a30f0270 2024-04-06T09:55:51.691 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:55:51.689+0000 7f0965192640 -1 reset not still connected to 0x5632a30f0f70 2024-04-06T09:55:51.691 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:55:51.689+0000 7f0965192640 -1 reset not still connected to 0x5632a417e750 2024-04-06T09:55:51.691 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:55:51.689+0000 7f0965192640 -1 reset not still connected to 0x5632a4cda000 2024-04-06T09:55:51.697 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:55:51.693+0000 7f0965192640 -1 reset not still connected to 0x5632a4cdbba0 2024-04-06T09:55:51.697 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:55:51.693+0000 7f0965192640 -1 reset not still connected to 0x5632a5e9dee0 2024-04-06T09:55:51.697 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:55:51.693+0000 7f0965192640 -1 reset not still connected to 0x5632a89f4750 2024-04-06T09:55:51.697 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:55:51.693+0000 7f0965192640 -1 reset not still connected to 0x5632abc812b0 2024-04-06T09:55:51.697 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:55:51.693+0000 7f0965192640 -1 reset not still connected to 0x5632abc81d40 2024-04-06T09:55:51.697 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:55:51.693+0000 7f0965192640 -1 reset not still connected to 0x5632abcbe820 2024-04-06T09:55:51.697 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:55:51.693+0000 7f0965192640 -1 reset not still connected to 0x5632b9592000 2024-04-06T09:55:51.697 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:55:51.693+0000 7f0965192640 -1 reset not still connected to 0x5632b9593110 2024-04-06T09:55:51.697 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:55:51.693+0000 7f0965192640 -1 reset not still connected to 0x5632c0cfa000 2024-04-06T09:55:51.697 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:55:51.697+0000 7f0965192640 -1 reset not still connected to 0x5632cb3ac000 2024-04-06T09:55:51.698 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:55:51.697+0000 7f0965192640 -1 reset not still connected to 0x5632dac08dd0 2024-04-06T09:55:51.698 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:55:51.697+0000 7f0965192640 -1 reset not still connected to 0x5632dac09040 2024-04-06T09:55:51.698 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:55:51.697+0000 7f0965192640 -1 reset not still connected to 0x5632e1684ea0 2024-04-06T09:55:51.698 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:55:51.697+0000 7f0965192640 -1 reset not still connected to 0x5632e16851e0 2024-04-06T09:55:51.973 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:55:51.973+0000 7f2e84d66640 -1 reset not still connected to 0x560d27c3edd0 2024-04-06T09:55:51.978 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:55:51.977+0000 7f2e84d66640 -1 reset not still connected to 0x560d27c3f2b0 2024-04-06T09:55:51.978 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:55:51.977+0000 7f2e84d66640 -1 reset not still connected to 0x560d29cd9ba0 2024-04-06T09:55:51.978 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:55:51.977+0000 7f2e84d66640 -1 reset not still connected to 0x560d29d5b520 2024-04-06T09:55:51.979 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:55:51.977+0000 7f2e84d66640 -1 reset not still connected to 0x560d2a17d450 2024-04-06T09:55:51.979 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:55:51.977+0000 7f2e84d66640 -1 reset not still connected to 0x560d2a17d6c0 2024-04-06T09:55:51.979 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:55:51.981+0000 7f2e84d66640 -1 reset not still connected to 0x560d2a2336c0 2024-04-06T09:55:51.980 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:55:51.981+0000 7f2e84d66640 -1 reset not still connected to 0x560d2a5705b0 2024-04-06T09:55:51.995 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:55:51.997+0000 7f2e84d66640 -1 reset not still connected to 0x560d368e4f70 2024-04-06T09:55:52.002 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:55:52.001+0000 7f2e84d66640 -1 reset not still connected to 0x560d36912ea0 2024-04-06T09:55:52.002 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:55:52.001+0000 7f2e84d66640 -1 reset not still connected to 0x560d381dd5f0 2024-04-06T09:55:52.004 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:55:52.005+0000 7f2e84d66640 -1 reset not still connected to 0x560d50adba00 2024-04-06T09:55:52.005 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:55:52.005+0000 7f2e84d66640 -1 reset not still connected to 0x560d27073860 2024-04-06T09:55:52.005 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:55:52.005+0000 7f2e84d66640 -1 reset not still connected to 0x560d27567380 2024-04-06T09:55:52.005 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:55:52.005+0000 7f2e84d66640 -1 reset not still connected to 0x560d27c3edd0 2024-04-06T09:55:52.005 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:55:52.005+0000 7f2e84d66640 -1 reset not still connected to 0x560d27c3f2b0 2024-04-06T09:55:52.005 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:55:52.005+0000 7f2e84d66640 -1 reset not still connected to 0x560d2a17d6c0 2024-04-06T09:55:52.005 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:55:52.005+0000 7f2e84d66640 -1 reset not still connected to 0x560d2a2336c0 2024-04-06T09:55:52.699 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:55:52.701+0000 7f0965192640 -1 reset not still connected to 0x5632a1988000 2024-04-06T09:55:52.700 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:55:52.701+0000 7f0965192640 -1 reset not still connected to 0x5632a1b73ee0 2024-04-06T09:55:52.707 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:55:52.705+0000 7f0965192640 -1 reset not still connected to 0x5632a2793520 2024-04-06T09:55:52.709 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:55:52.705+0000 7f0965192640 -1 reset not still connected to 0x5632a30f0270 2024-04-06T09:55:52.709 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:55:52.709+0000 7f0965192640 -1 reset not still connected to 0x5632a417e750 2024-04-06T09:55:52.711 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:55:52.709+0000 7f0965192640 -1 reset not still connected to 0x5632a4cda000 2024-04-06T09:55:52.713 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:55:52.713+0000 7f0965192640 -1 reset not still connected to 0x5632a4cdbba0 2024-04-06T09:55:52.713 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:55:52.713+0000 7f0965192640 -1 reset not still connected to 0x5632a5e9dee0 2024-04-06T09:55:52.713 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:55:52.713+0000 7f0965192640 -1 reset not still connected to 0x5632a89f4750 2024-04-06T09:55:52.716 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:55:52.713+0000 7f0965192640 -1 reset not still connected to 0x5632abc812b0 2024-04-06T09:55:52.716 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:55:52.713+0000 7f0965192640 -1 reset not still connected to 0x5632abc81d40 2024-04-06T09:55:52.716 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:55:52.713+0000 7f0965192640 -1 reset not still connected to 0x5632abcbe820 2024-04-06T09:55:52.717 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:55:52.713+0000 7f0965192640 -1 reset not still connected to 0x5632b9592000 2024-04-06T09:55:52.717 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:55:52.713+0000 7f0965192640 -1 reset not still connected to 0x5632b9593110 2024-04-06T09:55:52.717 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:55:52.713+0000 7f0965192640 -1 reset not still connected to 0x5632c0cfa000 2024-04-06T09:55:52.717 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:55:52.713+0000 7f0965192640 -1 reset not still connected to 0x5632cb3ac000 2024-04-06T09:55:52.717 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:55:52.713+0000 7f0965192640 -1 reset not still connected to 0x5632dac09040 2024-04-06T09:55:52.717 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:55:52.713+0000 7f0965192640 -1 reset not still connected to 0x5632e16851e0 2024-04-06T09:55:53.986 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:55:53.985+0000 7f0965192640 -1 reset not still connected to 0x5632dac09040 2024-04-06T09:55:53.987 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:55:53.985+0000 7f0965192640 -1 reset not still connected to 0x5632e1684ea0 2024-04-06T09:55:53.987 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:55:53.985+0000 7f0965192640 -1 reset not still connected to 0x5632e16851e0 2024-04-06T09:55:54.048 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:55:54.041+0000 7f0965192640 -1 reset not still connected to 0x5632a417e750 2024-04-06T09:55:54.048 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:55:54.041+0000 7f0965192640 -1 reset not still connected to 0x5632a4cda000 2024-04-06T09:55:54.052 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:55:54.045+0000 7f0965192640 -1 reset not still connected to 0x5632a4cdbba0 2024-04-06T09:55:54.052 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:55:54.045+0000 7f0965192640 -1 reset not still connected to 0x5632a5e9dee0 2024-04-06T09:55:54.052 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:55:54.045+0000 7f0965192640 -1 reset not still connected to 0x5632a89f4750 2024-04-06T09:55:54.052 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:55:54.045+0000 7f0965192640 -1 reset not still connected to 0x5632abc812b0 2024-04-06T09:55:54.053 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:55:54.045+0000 7f0965192640 -1 reset not still connected to 0x5632abc81d40 2024-04-06T09:55:54.053 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:55:54.045+0000 7f0965192640 -1 reset not still connected to 0x5632abcbe820 2024-04-06T09:55:54.053 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:55:54.045+0000 7f0965192640 -1 reset not still connected to 0x5632b9592000 2024-04-06T09:55:54.053 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:55:54.045+0000 7f0965192640 -1 reset not still connected to 0x5632b9593110 2024-04-06T09:55:54.053 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:55:54.045+0000 7f0965192640 -1 reset not still connected to 0x5632c0cfa000 2024-04-06T09:55:54.053 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:55:54.045+0000 7f0965192640 -1 reset not still connected to 0x5632cb3ac000 2024-04-06T09:55:54.053 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:55:54.045+0000 7f0965192640 -1 reset not still connected to 0x5632dac09040 2024-04-06T09:55:54.054 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:55:54.053+0000 7f0965192640 -1 reset not still connected to 0x5632e1684ea0 2024-04-06T09:55:54.054 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:55:54.053+0000 7f0965192640 -1 reset not still connected to 0x5632e16851e0 2024-04-06T09:55:54.065 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:200 IOs, 95229952 bytes 2024-04-06T09:55:54.280 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:55:54.277+0000 7f0965192640 -1 reset not still connected to 0x5632a4cdbba0 2024-04-06T09:55:54.280 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:55:54.277+0000 7f0965192640 -1 reset not still connected to 0x5632a5e9dee0 2024-04-06T09:55:54.280 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:55:54.277+0000 7f0965192640 -1 reset not still connected to 0x5632a89f4750 2024-04-06T09:55:54.280 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:55:54.277+0000 7f0965192640 -1 reset not still connected to 0x5632abc812b0 2024-04-06T09:55:54.280 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:55:54.277+0000 7f0965192640 -1 reset not still connected to 0x5632abc81d40 2024-04-06T09:55:54.282 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:55:54.281+0000 7f0965192640 -1 reset not still connected to 0x5632abcbe820 2024-04-06T09:55:54.283 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:55:54.281+0000 7f0965192640 -1 reset not still connected to 0x5632b9592000 2024-04-06T09:55:54.283 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:55:54.281+0000 7f0965192640 -1 reset not still connected to 0x5632b9593110 2024-04-06T09:55:54.283 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:55:54.281+0000 7f0965192640 -1 reset not still connected to 0x5632c0cfa000 2024-04-06T09:55:54.283 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:55:54.281+0000 7f0965192640 -1 reset not still connected to 0x5632dac09040 2024-04-06T09:55:54.283 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:55:54.281+0000 7f0965192640 -1 reset not still connected to 0x5632e1684ea0 2024-04-06T09:55:54.283 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:55:54.281+0000 7f0965192640 -1 reset not still connected to 0x5632e16851e0 2024-04-06T09:55:56.471 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:300 IOs, 118142976 bytes 2024-04-06T09:55:56.637 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:55:56.637+0000 7f2e84d66640 -1 reset not still connected to 0x560d294908f0 2024-04-06T09:55:56.637 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:55:56.637+0000 7f2e84d66640 -1 reset not still connected to 0x560d29cd9ba0 2024-04-06T09:55:56.638 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:55:56.637+0000 7f2e84d66640 -1 reset not still connected to 0x560d29d5b520 2024-04-06T09:55:56.638 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:55:56.637+0000 7f2e84d66640 -1 reset not still connected to 0x560d2a17d450 2024-04-06T09:55:56.638 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:55:56.637+0000 7f2e84d66640 -1 reset not still connected to 0x560d2a17d6c0 2024-04-06T09:55:56.638 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:55:56.637+0000 7f2e84d66640 -1 reset not still connected to 0x560d2a204b60 2024-04-06T09:55:56.638 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:55:56.637+0000 7f2e84d66640 -1 reset not still connected to 0x560d2a2336c0 2024-04-06T09:55:56.638 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:55:56.637+0000 7f2e84d66640 -1 reset not still connected to 0x560d2a5705b0 2024-04-06T09:55:56.643 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:55:56.641+0000 7f2e84d66640 -1 reset not still connected to 0x560d3620b520 2024-04-06T09:55:56.643 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:55:56.641+0000 7f2e84d66640 -1 reset not still connected to 0x560d368e4f70 2024-04-06T09:55:56.652 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:55:56.653+0000 7f2e84d66640 -1 reset not still connected to 0x560d36912ea0 2024-04-06T09:55:56.653 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:55:56.653+0000 7f2e84d66640 -1 reset not still connected to 0x560d381dd5f0 2024-04-06T09:55:56.653 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:55:56.653+0000 7f2e84d66640 -1 reset not still connected to 0x560d50adba00 2024-04-06T09:55:57.306 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ error_code=124 2024-04-06T09:55:57.307 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set -e 2024-04-06T09:55:57.307 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 124 -eq 124 ']' 2024-04-06T09:55:57.307 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:55:57.307 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-06T09:55:57.307 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ image=image_35 2024-04-06T09:55:57.307 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ create_image_and_enable_mirror cluster2 mirror image_35 journal 128M 2024-04-06T09:55:57.307 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-04-06T09:55:57.307 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:55:57.307 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:55:57.307 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:55:57.307 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_35 2024-04-06T09:55:57.307 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:55:57.307 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local mode=journal 2024-04-06T09:55:57.308 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n journal ']' 2024-04-06T09:55:57.308 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:55:57.308 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ create_image cluster2 mirror image_35 128M 2024-04-06T09:55:57.308 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-04-06T09:55:57.308 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:55:57.308 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:55:57.308 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:55:57.308 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_35 2024-04-06T09:55:57.308 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:55:57.308 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local size=128 2024-04-06T09:55:57.308 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n 128M ']' 2024-04-06T09:55:57.308 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ size=128M 2024-04-06T09:55:57.308 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:55:57.308 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,journaling mirror/image_35 2024-04-06T09:55:57.374 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' pool = image ']' 2024-04-06T09:55:57.374 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' mirror = mirror_parent ']' 2024-04-06T09:55:57.374 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n '' ']' 2024-04-06T09:55:57.374 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ stress_write_image cluster2 mirror image_35 2024-04-06T09:55:57.374 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-04-06T09:55:57.374 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:55:57.375 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_35 2024-04-06T09:55:57.375 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-04-06T09:55:57.375 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local duration=5 2024-04-06T09:55:57.376 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set +e 2024-04-06T09:55:57.376 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ timeout 5s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_35 --debug-rbd=20 --debug-journaler=20 2024-04-06T09:55:57.916 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:55:57.913+0000 7f2e84d66640 -1 reset not still connected to 0x560d2a17d6c0 2024-04-06T09:55:57.916 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:55:57.917+0000 7f2e84d66640 -1 reset not still connected to 0x560d2a2336c0 2024-04-06T09:55:57.917 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:55:57.917+0000 7f2e84d66640 -1 reset not still connected to 0x560d2a5705b0 2024-04-06T09:55:57.926 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:55:57.925+0000 7f2e84d66640 -1 reset not still connected to 0x560d3620b520 2024-04-06T09:55:57.926 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:55:57.925+0000 7f2e84d66640 -1 reset not still connected to 0x560d368e4f70 2024-04-06T09:55:57.926 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:55:57.925+0000 7f2e84d66640 -1 reset not still connected to 0x560d36912ea0 2024-04-06T09:55:57.926 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:55:57.925+0000 7f2e84d66640 -1 reset not still connected to 0x560d381dd5f0 2024-04-06T09:55:57.926 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:55:57.925+0000 7f2e84d66640 -1 reset not still connected to 0x560d50adba00 2024-04-06T09:55:58.748 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:55:58.745+0000 7f0965192640 -1 reset not still connected to 0x5632b9593110 2024-04-06T09:55:58.748 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:55:58.745+0000 7f0965192640 -1 reset not still connected to 0x5632c0cfa000 2024-04-06T09:55:58.748 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:55:58.745+0000 7f0965192640 -1 reset not still connected to 0x5632cb3ac000 2024-04-06T09:55:58.748 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:55:58.745+0000 7f0965192640 -1 reset not still connected to 0x5632dac09040 2024-04-06T09:55:58.748 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:55:58.745+0000 7f0965192640 -1 reset not still connected to 0x5632e1684ea0 2024-04-06T09:55:58.752 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:55:58.753+0000 7f0965192640 -1 reset not still connected to 0x5632e16851e0 2024-04-06T09:55:58.924 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:55:58.921+0000 7f0965192640 -1 reset not still connected to 0x5632a30f0270 2024-04-06T09:55:58.926 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:55:58.925+0000 7f0965192640 -1 reset not still connected to 0x5632a30f0f70 2024-04-06T09:55:58.927 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:55:58.925+0000 7f0965192640 -1 reset not still connected to 0x5632a417e750 2024-04-06T09:55:58.927 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:55:58.925+0000 7f0965192640 -1 reset not still connected to 0x5632a4cda000 2024-04-06T09:55:58.930 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:55:58.929+0000 7f0965192640 -1 reset not still connected to 0x5632a4cdbba0 2024-04-06T09:55:58.930 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:55:58.929+0000 7f0965192640 -1 reset not still connected to 0x5632a5e9dee0 2024-04-06T09:55:58.930 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:55:58.929+0000 7f0965192640 -1 reset not still connected to 0x5632a89f4750 2024-04-06T09:55:58.931 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:55:58.929+0000 7f0965192640 -1 reset not still connected to 0x5632abc812b0 2024-04-06T09:55:58.931 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:55:58.929+0000 7f0965192640 -1 reset not still connected to 0x5632abc81d40 2024-04-06T09:55:58.931 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:55:58.929+0000 7f0965192640 -1 reset not still connected to 0x5632abcbe820 2024-04-06T09:55:58.931 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:55:58.929+0000 7f0965192640 -1 reset not still connected to 0x5632b9592000 2024-04-06T09:55:58.931 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:55:58.929+0000 7f0965192640 -1 reset not still connected to 0x5632b9593110 2024-04-06T09:55:58.931 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:55:58.929+0000 7f0965192640 -1 reset not still connected to 0x5632c0cfa000 2024-04-06T09:55:58.935 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:55:58.933+0000 7f0965192640 -1 reset not still connected to 0x5632cb3ac000 2024-04-06T09:55:58.935 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:55:58.933+0000 7f0965192640 -1 reset not still connected to 0x5632dac08dd0 2024-04-06T09:55:58.935 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:55:58.933+0000 7f0965192640 -1 reset not still connected to 0x5632dac09040 2024-04-06T09:55:58.935 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:55:58.933+0000 7f0965192640 -1 reset not still connected to 0x5632e1684ea0 2024-04-06T09:55:58.935 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:55:58.933+0000 7f0965192640 -1 reset not still connected to 0x5632e16851e0 2024-04-06T09:55:59.532 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:55:59.529+0000 7f2e84d66640 -1 reset not still connected to 0x560d25ab2680 2024-04-06T09:55:59.534 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:55:59.529+0000 7f2e84d66640 -1 reset not still connected to 0x560d27073860 2024-04-06T09:55:59.534 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:55:59.529+0000 7f2e84d66640 -1 reset not still connected to 0x560d27567380 2024-04-06T09:55:59.534 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:55:59.529+0000 7f2e84d66640 -1 reset not still connected to 0x560d27839930 2024-04-06T09:55:59.534 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:55:59.529+0000 7f2e84d66640 -1 reset not still connected to 0x560d27c3edd0 2024-04-06T09:55:59.535 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:55:59.529+0000 7f2e84d66640 -1 reset not still connected to 0x560d27c3f2b0 2024-04-06T09:55:59.535 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:55:59.529+0000 7f2e84d66640 -1 reset not still connected to 0x560d294908f0 2024-04-06T09:55:59.535 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:55:59.529+0000 7f2e84d66640 -1 reset not still connected to 0x560d29cd8f70 2024-04-06T09:55:59.535 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:55:59.529+0000 7f2e84d66640 -1 reset not still connected to 0x560d2a2336c0 2024-04-06T09:55:59.535 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:55:59.529+0000 7f2e84d66640 -1 reset not still connected to 0x560d36912ea0 2024-04-06T09:55:59.535 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:55:59.529+0000 7f2e84d66640 -1 reset not still connected to 0x560d381dd5f0 2024-04-06T09:55:59.769 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:100 IOs, 32049152 bytes 2024-04-06T09:55:59.904 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-04-06T09:55:59.901+0000 7fd1b6210640 -1 reset not still connected to 0x55ef9ac60270 2024-04-06T09:55:59.911 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-04-06T09:55:59.905+0000 7fd1b6210640 -1 reset not still connected to 0x55ef9b657e10 2024-04-06T09:55:59.912 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-04-06T09:55:59.913+0000 7fd1b6210640 -1 reset not still connected to 0x55ef9bbe84e0 2024-04-06T09:55:59.912 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-04-06T09:55:59.913+0000 7fd1b6210640 -1 reset not still connected to 0x55ef9bbe9ee0 2024-04-06T09:55:59.912 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-04-06T09:55:59.913+0000 7fd1b6210640 -1 reset not still connected to 0x55ef9bd5c9c0 2024-04-06T09:55:59.912 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-04-06T09:55:59.913+0000 7fd1b6210640 -1 reset not still connected to 0x55ef9bd5d520 2024-04-06T09:55:59.912 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-04-06T09:55:59.913+0000 7fd1b6210640 -1 reset not still connected to 0x55ef9bd5dd40 2024-04-06T09:55:59.913 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-04-06T09:55:59.913+0000 7fd1b6210640 -1 reset not still connected to 0x55ef9bd86820 2024-04-06T09:55:59.913 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-04-06T09:55:59.913+0000 7fd1b6210640 -1 reset not still connected to 0x55ef9c103110 2024-04-06T09:55:59.913 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-04-06T09:55:59.913+0000 7fd1b6210640 -1 reset not still connected to 0x55ef9c103930 2024-04-06T09:55:59.913 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-04-06T09:55:59.913+0000 7fd1b6210640 -1 reset not still connected to 0x55ef9c103d40 2024-04-06T09:55:59.915 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-04-06T09:55:59.917+0000 7fd1b6210640 -1 reset not still connected to 0x55ef9e698c30 2024-04-06T09:55:59.916 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-04-06T09:55:59.917+0000 7fd1b6210640 -1 reset not still connected to 0x55ef9e88f520 2024-04-06T09:55:59.916 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-04-06T09:55:59.917+0000 7fd1b6210640 -1 reset not still connected to 0x55ef9eee21a0 2024-04-06T09:55:59.916 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-04-06T09:55:59.917+0000 7fd1b6210640 -1 reset not still connected to 0x55ef9eee3a00 2024-04-06T09:55:59.916 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-04-06T09:55:59.917+0000 7fd1b6210640 -1 reset not still connected to 0x55efa18cc270 2024-04-06T09:55:59.921 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-04-06T09:55:59.917+0000 7fd1b6210640 -1 reset not still connected to 0x55efbb630d00 2024-04-06T09:56:01.399 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:56:01.393+0000 7f2e84d66640 -1 reset not still connected to 0x560d27567380 2024-04-06T09:56:01.399 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:56:01.393+0000 7f2e84d66640 -1 reset not still connected to 0x560d27839380 2024-04-06T09:56:01.399 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:56:01.393+0000 7f2e84d66640 -1 reset not still connected to 0x560d27839930 2024-04-06T09:56:01.399 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:56:01.393+0000 7f2e84d66640 -1 reset not still connected to 0x560d27c3edd0 2024-04-06T09:56:01.399 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:56:01.393+0000 7f2e84d66640 -1 reset not still connected to 0x560d281875f0 2024-04-06T09:56:01.400 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:56:01.393+0000 7f2e84d66640 -1 reset not still connected to 0x560d29cd8f70 2024-04-06T09:56:01.400 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:56:01.393+0000 7f2e84d66640 -1 reset not still connected to 0x560d29cd9ba0 2024-04-06T09:56:01.400 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:56:01.393+0000 7f2e84d66640 -1 reset not still connected to 0x560d29d5a340 2024-04-06T09:56:01.400 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:56:01.393+0000 7f2e84d66640 -1 reset not still connected to 0x560d29d5b520 2024-04-06T09:56:01.400 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:56:01.393+0000 7f2e84d66640 -1 reset not still connected to 0x560d2a17d450 2024-04-06T09:56:01.400 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:56:01.393+0000 7f2e84d66640 -1 reset not still connected to 0x560d2a17d6c0 2024-04-06T09:56:01.400 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:56:01.393+0000 7f2e84d66640 -1 reset not still connected to 0x560d2a204b60 2024-04-06T09:56:01.401 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:56:01.401+0000 7f2e84d66640 -1 reset not still connected to 0x560d2a2336c0 2024-04-06T09:56:01.406 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:56:01.401+0000 7f2e84d66640 -1 reset not still connected to 0x560d2a570000 2024-04-06T09:56:01.408 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:56:01.401+0000 7f2e84d66640 -1 reset not still connected to 0x560d2a5705b0 2024-04-06T09:56:01.408 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:56:01.401+0000 7f2e84d66640 -1 reset not still connected to 0x560d3620b520 2024-04-06T09:56:01.409 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:56:01.409+0000 7f2e84d66640 -1 reset not still connected to 0x560d368e4f70 2024-04-06T09:56:01.411 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:56:01.409+0000 7f2e84d66640 -1 reset not still connected to 0x560d36912ea0 2024-04-06T09:56:01.411 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:56:01.409+0000 7f2e84d66640 -1 reset not still connected to 0x560d381dd5f0 2024-04-06T09:56:01.412 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:56:01.409+0000 7f2e84d66640 -1 reset not still connected to 0x560d390dca90 2024-04-06T09:56:01.412 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:56:01.409+0000 7f2e84d66640 -1 reset not still connected to 0x560d4bc7e8f0 2024-04-06T09:56:01.412 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:56:01.409+0000 7f2e84d66640 -1 reset not still connected to 0x560d50adba00 2024-04-06T09:56:01.484 INFO:tasks.ceph.cluster2.osd.0.smithi192.stderr:2024-04-06T09:56:01.483+0000 7f789aa6c640 -1 bdev(0x56041f2dce00 /var/lib/ceph/osd/cluster2-0/block) aio_submit retries 5 2024-04-06T09:56:02.138 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:56:02.133+0000 7f2e84d66640 -1 reset not still connected to 0x560d29d5b520 2024-04-06T09:56:02.138 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:56:02.133+0000 7f2e84d66640 -1 reset not still connected to 0x560d2a17d450 2024-04-06T09:56:02.138 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:56:02.133+0000 7f2e84d66640 -1 reset not still connected to 0x560d2a17d6c0 2024-04-06T09:56:02.138 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:56:02.137+0000 7f2e84d66640 -1 reset not still connected to 0x560d2a2336c0 2024-04-06T09:56:02.138 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:56:02.137+0000 7f2e84d66640 -1 reset not still connected to 0x560d2a5705b0 2024-04-06T09:56:02.142 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:56:02.141+0000 7f2e84d66640 -1 reset not still connected to 0x560d368e4f70 2024-04-06T09:56:02.156 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:56:02.153+0000 7f2e84d66640 -1 reset not still connected to 0x560d36912ea0 2024-04-06T09:56:02.156 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:56:02.153+0000 7f2e84d66640 -1 reset not still connected to 0x560d381dd5f0 2024-04-06T09:56:02.157 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:56:02.157+0000 7f2e84d66640 -1 reset not still connected to 0x560d4bc7e8f0 2024-04-06T09:56:02.157 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:56:02.157+0000 7f2e84d66640 -1 reset not still connected to 0x560d50adba00 2024-04-06T09:56:02.158 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:56:02.157+0000 7f2e84d66640 -1 reset not still connected to 0x560d25ab2680 2024-04-06T09:56:02.171 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:56:02.169+0000 7f2e84d66640 -1 reset not still connected to 0x560d27567380 2024-04-06T09:56:02.171 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:56:02.169+0000 7f2e84d66640 -1 reset not still connected to 0x560d27839380 2024-04-06T09:56:02.171 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:56:02.169+0000 7f2e84d66640 -1 reset not still connected to 0x560d27839930 2024-04-06T09:56:02.171 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:56:02.169+0000 7f2e84d66640 -1 reset not still connected to 0x560d27c3edd0 2024-04-06T09:56:02.179 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:56:02.177+0000 7f2e84d66640 -1 reset not still connected to 0x560d27c3f2b0 2024-04-06T09:56:02.179 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:56:02.177+0000 7f2e84d66640 -1 reset not still connected to 0x560d294908f0 2024-04-06T09:56:02.180 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:56:02.177+0000 7f2e84d66640 -1 reset not still connected to 0x560d29cd8f70 2024-04-06T09:56:02.180 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:56:02.177+0000 7f2e84d66640 -1 reset not still connected to 0x560d29cd9ba0 2024-04-06T09:56:02.180 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:56:02.177+0000 7f2e84d66640 -1 reset not still connected to 0x560d29d5b520 2024-04-06T09:56:02.180 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:56:02.177+0000 7f2e84d66640 -1 reset not still connected to 0x560d2a17d450 2024-04-06T09:56:02.181 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:56:02.177+0000 7f2e84d66640 -1 reset not still connected to 0x560d2a17d6c0 2024-04-06T09:56:02.181 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:56:02.177+0000 7f2e84d66640 -1 reset not still connected to 0x560d2a204b60 2024-04-06T09:56:02.181 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:56:02.177+0000 7f2e84d66640 -1 reset not still connected to 0x560d2a2336c0 2024-04-06T09:56:02.181 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:56:02.177+0000 7f2e84d66640 -1 reset not still connected to 0x560d2a5705b0 2024-04-06T09:56:02.181 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:56:02.177+0000 7f2e84d66640 -1 reset not still connected to 0x560d3620b520 2024-04-06T09:56:02.181 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:56:02.177+0000 7f2e84d66640 -1 reset not still connected to 0x560d368e4f70 2024-04-06T09:56:02.181 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:56:02.177+0000 7f2e84d66640 -1 reset not still connected to 0x560d381dd5f0 2024-04-06T09:56:02.181 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:56:02.177+0000 7f2e84d66640 -1 reset not still connected to 0x560d4bc7e8f0 2024-04-06T09:56:02.186 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:56:02.185+0000 7f2e84d66640 -1 reset not still connected to 0x560d50adba00 2024-04-06T09:56:02.383 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ error_code=124 2024-04-06T09:56:02.383 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set -e 2024-04-06T09:56:02.383 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 124 -eq 124 ']' 2024-04-06T09:56:02.383 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:56:02.383 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-06T09:56:02.383 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ image=image_36 2024-04-06T09:56:02.383 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ create_image_and_enable_mirror cluster2 mirror image_36 journal 128M 2024-04-06T09:56:02.383 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-04-06T09:56:02.383 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:56:02.384 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:56:02.384 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:56:02.384 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_36 2024-04-06T09:56:02.384 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:56:02.384 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local mode=journal 2024-04-06T09:56:02.384 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n journal ']' 2024-04-06T09:56:02.384 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:56:02.384 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ create_image cluster2 mirror image_36 128M 2024-04-06T09:56:02.384 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-04-06T09:56:02.384 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:56:02.384 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:56:02.384 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:56:02.384 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_36 2024-04-06T09:56:02.385 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:56:02.385 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local size=128 2024-04-06T09:56:02.385 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n 128M ']' 2024-04-06T09:56:02.385 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ size=128M 2024-04-06T09:56:02.385 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:56:02.385 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,journaling mirror/image_36 2024-04-06T09:56:02.466 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' pool = image ']' 2024-04-06T09:56:02.467 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' mirror = mirror_parent ']' 2024-04-06T09:56:02.467 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n '' ']' 2024-04-06T09:56:02.467 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ stress_write_image cluster2 mirror image_36 2024-04-06T09:56:02.467 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-04-06T09:56:02.467 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:56:02.467 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_36 2024-04-06T09:56:02.467 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-04-06T09:56:02.468 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local duration=10 2024-04-06T09:56:02.468 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set +e 2024-04-06T09:56:02.468 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ timeout 10s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_36 --debug-rbd=20 --debug-journaler=20 2024-04-06T09:56:03.435 INFO:tasks.rbd_mirror_thrash:kill cluster2.client.mirror.0 2024-04-06T09:56:03.435 INFO:tasks.rbd_mirror.cluster2.client.mirror.0:Sent signal 15 2024-04-06T09:56:03.436 INFO:tasks.rbd_mirror_thrash:waiting for 7 secs before reviving daemons 2024-04-06T09:56:03.437 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi192.stderr:2024-04-06T09:56:03.439+0000 7f86e36e0640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.0 (PID: 127826) UID: 1000 2024-04-06T09:56:04.097 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:56:04.097+0000 7f2e84d66640 -1 reset not still connected to 0x560d29cd8f70 2024-04-06T09:56:04.098 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:56:04.097+0000 7f2e84d66640 -1 reset not still connected to 0x560d29cd9ba0 2024-04-06T09:56:04.098 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:56:04.097+0000 7f2e84d66640 -1 reset not still connected to 0x560d29d5b520 2024-04-06T09:56:04.098 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:56:04.097+0000 7f2e84d66640 -1 reset not still connected to 0x560d2a17d450 2024-04-06T09:56:04.099 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:56:04.097+0000 7f2e84d66640 -1 reset not still connected to 0x560d2a17d6c0 2024-04-06T09:56:04.105 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:56:04.105+0000 7f2e84d66640 -1 reset not still connected to 0x560d2a2336c0 2024-04-06T09:56:04.105 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:56:04.105+0000 7f2e84d66640 -1 reset not still connected to 0x560d2a5705b0 2024-04-06T09:56:04.109 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:56:04.109+0000 7f2e84d66640 -1 reset not still connected to 0x560d36912ea0 2024-04-06T09:56:04.109 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:56:04.109+0000 7f2e84d66640 -1 reset not still connected to 0x560d381dd5f0 2024-04-06T09:56:05.329 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi192.stderr:2024-04-06T09:56:05.331+0000 7f3d28bce640 -1 rbd::mirror::InstanceWatcher: C_NotifyInstanceRequest: 0x560668d73200 finish: resending after timeout 2024-04-06T09:56:05.329 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi192.stderr:2024-04-06T09:56:05.331+0000 7f3d293cf640 -1 rbd::mirror::InstanceWatcher: C_NotifyInstanceRequest: 0x56067b2cdec0 finish: resending after timeout 2024-04-06T09:56:05.329 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi192.stderr:2024-04-06T09:56:05.331+0000 7f3d28bce640 -1 rbd::mirror::InstanceWatcher: C_NotifyInstanceRequest: 0x560668d72600 finish: resending after timeout 2024-04-06T09:56:05.477 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:56:05.473+0000 7f2e84d66640 -1 reset not still connected to 0x560d27839930 2024-04-06T09:56:05.481 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:56:05.481+0000 7f2e84d66640 -1 reset not still connected to 0x560d27c3edd0 2024-04-06T09:56:05.484 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:56:05.485+0000 7f2e84d66640 -1 reset not still connected to 0x560d27c3f2b0 2024-04-06T09:56:05.484 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:56:05.485+0000 7f2e84d66640 -1 reset not still connected to 0x560d294908f0 2024-04-06T09:56:05.490 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:56:05.489+0000 7f2e84d66640 -1 reset not still connected to 0x560d29cd8f70 2024-04-06T09:56:05.490 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:56:05.489+0000 7f2e84d66640 -1 reset not still connected to 0x560d29cd9ba0 2024-04-06T09:56:05.490 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:56:05.489+0000 7f2e84d66640 -1 reset not still connected to 0x560d29d5b520 2024-04-06T09:56:05.490 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:56:05.489+0000 7f2e84d66640 -1 reset not still connected to 0x560d2a17d450 2024-04-06T09:56:05.491 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:56:05.489+0000 7f2e84d66640 -1 reset not still connected to 0x560d2a17d6c0 2024-04-06T09:56:05.491 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:56:05.489+0000 7f2e84d66640 -1 reset not still connected to 0x560d2a204b60 2024-04-06T09:56:05.491 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:56:05.489+0000 7f2e84d66640 -1 reset not still connected to 0x560d2a2336c0 2024-04-06T09:56:05.491 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:56:05.489+0000 7f2e84d66640 -1 reset not still connected to 0x560d2a5705b0 2024-04-06T09:56:05.491 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:56:05.489+0000 7f2e84d66640 -1 reset not still connected to 0x560d3620b520 2024-04-06T09:56:05.491 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:56:05.489+0000 7f2e84d66640 -1 reset not still connected to 0x560d368e4f70 2024-04-06T09:56:05.491 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:56:05.489+0000 7f2e84d66640 -1 reset not still connected to 0x560d381dd5f0 2024-04-06T09:56:05.491 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:56:05.489+0000 7f2e84d66640 -1 reset not still connected to 0x560d4bc7e8f0 2024-04-06T09:56:05.498 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:56:05.497+0000 7f2e84d66640 -1 reset not still connected to 0x560d50adba00 2024-04-06T09:56:05.722 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-04-06T09:56:05.717+0000 7fd1b6210640 -1 reset not still connected to 0x55ef9ac60270 2024-04-06T09:56:05.722 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-04-06T09:56:05.717+0000 7fd1b6210640 -1 reset not still connected to 0x55ef9bbe84e0 2024-04-06T09:56:05.722 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-04-06T09:56:05.717+0000 7fd1b6210640 -1 reset not still connected to 0x55ef9bbe9ee0 2024-04-06T09:56:05.722 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-04-06T09:56:05.717+0000 7fd1b6210640 -1 reset not still connected to 0x55ef9bd5c9c0 2024-04-06T09:56:05.722 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-04-06T09:56:05.717+0000 7fd1b6210640 -1 reset not still connected to 0x55ef9bd5d520 2024-04-06T09:56:05.723 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-04-06T09:56:05.721+0000 7fd1b6210640 -1 reset not still connected to 0x55ef9bd5dd40 2024-04-06T09:56:05.731 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-04-06T09:56:05.729+0000 7fd1b6210640 -1 reset not still connected to 0x55ef9c103110 2024-04-06T09:56:05.731 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-04-06T09:56:05.729+0000 7fd1b6210640 -1 reset not still connected to 0x55ef9c103930 2024-04-06T09:56:05.731 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-04-06T09:56:05.729+0000 7fd1b6210640 -1 reset not still connected to 0x55ef9c103d40 2024-04-06T09:56:05.731 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-04-06T09:56:05.729+0000 7fd1b6210640 -1 reset not still connected to 0x55ef9e698c30 2024-04-06T09:56:05.731 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-04-06T09:56:05.729+0000 7fd1b6210640 -1 reset not still connected to 0x55ef9e88f520 2024-04-06T09:56:05.731 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-04-06T09:56:05.729+0000 7fd1b6210640 -1 reset not still connected to 0x55ef9eee21a0 2024-04-06T09:56:05.731 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-04-06T09:56:05.729+0000 7fd1b6210640 -1 reset not still connected to 0x55efa18cc270 2024-04-06T09:56:05.731 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-04-06T09:56:05.729+0000 7fd1b6210640 -1 reset not still connected to 0x55efd81ee8f0 2024-04-06T09:56:06.202 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:56:06.197+0000 7f0965192640 -1 reset not still connected to 0x5632a1988000 2024-04-06T09:56:06.203 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:56:06.197+0000 7f0965192640 -1 reset not still connected to 0x5632a1989040 2024-04-06T09:56:06.203 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:56:06.197+0000 7f0965192640 -1 reset not still connected to 0x5632a1b73ee0 2024-04-06T09:56:06.203 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:56:06.197+0000 7f0965192640 -1 reset not still connected to 0x5632a2793520 2024-04-06T09:56:06.203 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:56:06.197+0000 7f0965192640 -1 reset not still connected to 0x5632a30f0270 2024-04-06T09:56:06.203 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:56:06.197+0000 7f0965192640 -1 reset not still connected to 0x5632a30f0f70 2024-04-06T09:56:06.203 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:56:06.197+0000 7f0965192640 -1 reset not still connected to 0x5632a4cdbba0 2024-04-06T09:56:06.203 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:56:06.197+0000 7f0965192640 -1 reset not still connected to 0x5632a5e9dee0 2024-04-06T09:56:06.203 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:56:06.197+0000 7f0965192640 -1 reset not still connected to 0x5632a89f4750 2024-04-06T09:56:06.203 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:56:06.197+0000 7f0965192640 -1 reset not still connected to 0x5632a89f52b0 2024-04-06T09:56:06.203 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:56:06.197+0000 7f0965192640 -1 reset not still connected to 0x5632abc812b0 2024-04-06T09:56:06.203 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:56:06.197+0000 7f0965192640 -1 reset not still connected to 0x5632abc81d40 2024-04-06T09:56:06.204 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:56:06.197+0000 7f0965192640 -1 reset not still connected to 0x5632b9592000 2024-04-06T09:56:06.204 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:56:06.197+0000 7f0965192640 -1 reset not still connected to 0x5632b9593110 2024-04-06T09:56:06.204 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:56:06.197+0000 7f0965192640 -1 reset not still connected to 0x5632c0cfa000 2024-04-06T09:56:06.204 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:56:06.197+0000 7f0965192640 -1 reset not still connected to 0x5632dac09040 2024-04-06T09:56:06.204 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:56:06.197+0000 7f0965192640 -1 reset not still connected to 0x5632e1684ea0 2024-04-06T09:56:06.204 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:56:06.197+0000 7f0965192640 -1 reset not still connected to 0x5632e16851e0 2024-04-06T09:56:07.099 INFO:tasks.ceph.cluster2.osd.0.smithi192.stderr:2024-04-06T09:56:07.099+0000 7f78af6df640 -1 reset not still connected to 0x56044cc1e1a0 2024-04-06T09:56:07.474 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:100 IOs, 52806656 bytes 2024-04-06T09:56:07.628 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:56:07.621+0000 7f0965192640 -1 reset not still connected to 0x5632c0cfa000 2024-04-06T09:56:07.628 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:56:07.621+0000 7f0965192640 -1 reset not still connected to 0x5632cb3ac000 2024-04-06T09:56:07.629 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:56:07.629+0000 7f0965192640 -1 reset not still connected to 0x5632dac09040 2024-04-06T09:56:07.629 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:56:07.629+0000 7f0965192640 -1 reset not still connected to 0x5632e1684ea0 2024-04-06T09:56:07.629 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:56:07.629+0000 7f0965192640 -1 reset not still connected to 0x5632e16851e0 2024-04-06T09:56:07.916 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:56:07.913+0000 7f2e84d66640 -1 reset not still connected to 0x560d368e4f70 2024-04-06T09:56:07.922 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:56:07.921+0000 7f2e84d66640 -1 reset not still connected to 0x560d36912ea0 2024-04-06T09:56:07.922 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:56:07.921+0000 7f2e84d66640 -1 reset not still connected to 0x560d381dd5f0 2024-04-06T09:56:07.922 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:56:07.921+0000 7f2e84d66640 -1 reset not still connected to 0x560d4bc7e8f0 2024-04-06T09:56:07.922 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:56:07.921+0000 7f2e84d66640 -1 reset not still connected to 0x560d50adba00 2024-04-06T09:56:09.285 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:200 IOs, 80750592 bytes 2024-04-06T09:56:10.437 INFO:tasks.rbd_mirror_thrash:waiting for cluster2.client.mirror.0 2024-04-06T09:56:10.437 INFO:teuthology.orchestra.run:waiting for 600 2024-04-06T09:56:10.437 INFO:tasks.rbd_mirror_thrash:reviving cluster2.client.mirror.0 2024-04-06T09:56:10.437 INFO:tasks.rbd_mirror.cluster2.client.mirror.0:Restarting daemon 2024-04-06T09:56:10.438 DEBUG:teuthology.orchestra.run.smithi192:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.0 2024-04-06T09:56:10.440 INFO:tasks.rbd_mirror.cluster2.client.mirror.0:Started 2024-04-06T09:56:10.440 INFO:tasks.rbd_mirror_thrash:waiting for 91 secs before thrashing 2024-04-06T09:56:10.585 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:56:10.585+0000 7f0965192640 -1 reset not still connected to 0x5632a417e750 2024-04-06T09:56:10.586 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:56:10.585+0000 7f0965192640 -1 reset not still connected to 0x5632a4cda000 2024-04-06T09:56:10.590 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:56:10.585+0000 7f0965192640 -1 reset not still connected to 0x5632a4cda9c0 2024-04-06T09:56:10.590 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:56:10.585+0000 7f0965192640 -1 reset not still connected to 0x5632a4cdbba0 2024-04-06T09:56:10.590 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:56:10.585+0000 7f0965192640 -1 reset not still connected to 0x5632a5e9dee0 2024-04-06T09:56:10.590 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:56:10.585+0000 7f0965192640 -1 reset not still connected to 0x5632a89f4750 2024-04-06T09:56:10.590 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:56:10.585+0000 7f0965192640 -1 reset not still connected to 0x5632a89f52b0 2024-04-06T09:56:10.591 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:56:10.589+0000 7f0965192640 -1 reset not still connected to 0x5632abc812b0 2024-04-06T09:56:10.591 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:56:10.589+0000 7f0965192640 -1 reset not still connected to 0x5632abc81d40 2024-04-06T09:56:10.591 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:56:10.589+0000 7f0965192640 -1 reset not still connected to 0x5632abcbe820 2024-04-06T09:56:10.591 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:56:10.589+0000 7f0965192640 -1 reset not still connected to 0x5632b9592000 2024-04-06T09:56:10.591 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:56:10.589+0000 7f0965192640 -1 reset not still connected to 0x5632b9593110 2024-04-06T09:56:10.591 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:56:10.589+0000 7f0965192640 -1 reset not still connected to 0x5632c0cfa000 2024-04-06T09:56:10.592 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:56:10.589+0000 7f0965192640 -1 reset not still connected to 0x5632cb3ac000 2024-04-06T09:56:10.592 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:56:10.589+0000 7f0965192640 -1 reset not still connected to 0x5632dac08dd0 2024-04-06T09:56:10.592 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:56:10.589+0000 7f0965192640 -1 reset not still connected to 0x5632dac09040 2024-04-06T09:56:10.592 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:56:10.589+0000 7f0965192640 -1 reset not still connected to 0x5632e1684ea0 2024-04-06T09:56:10.592 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:56:10.589+0000 7f0965192640 -1 reset not still connected to 0x5632e16851e0 2024-04-06T09:56:10.859 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:300 IOs, 101210112 bytes 2024-04-06T09:56:11.189 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:56:11.181+0000 7f0965192640 -1 reset not still connected to 0x5632abc81d40 2024-04-06T09:56:11.189 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:56:11.181+0000 7f0965192640 -1 reset not still connected to 0x5632abcbe820 2024-04-06T09:56:11.193 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:56:11.181+0000 7f0965192640 -1 reset not still connected to 0x5632b9592000 2024-04-06T09:56:11.194 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:56:11.181+0000 7f0965192640 -1 reset not still connected to 0x5632c0cfa000 2024-04-06T09:56:11.194 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:56:11.181+0000 7f0965192640 -1 reset not still connected to 0x5632cb3ac000 2024-04-06T09:56:11.194 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:56:11.181+0000 7f0965192640 -1 reset not still connected to 0x5632dac08dd0 2024-04-06T09:56:11.194 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:56:11.181+0000 7f0965192640 -1 reset not still connected to 0x5632dac09040 2024-04-06T09:56:11.194 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:56:11.181+0000 7f0965192640 -1 reset not still connected to 0x5632e16851e0 2024-04-06T09:56:11.639 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:56:11.633+0000 7f2e84d66640 -1 reset not still connected to 0x560d27567380 2024-04-06T09:56:11.641 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:56:11.637+0000 7f2e84d66640 -1 reset not still connected to 0x560d27839380 2024-04-06T09:56:11.641 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:56:11.637+0000 7f2e84d66640 -1 reset not still connected to 0x560d27839930 2024-04-06T09:56:11.641 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:56:11.637+0000 7f2e84d66640 -1 reset not still connected to 0x560d27c3edd0 2024-04-06T09:56:11.641 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:56:11.637+0000 7f2e84d66640 -1 reset not still connected to 0x560d27c3f2b0 2024-04-06T09:56:11.648 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:56:11.645+0000 7f2e84d66640 -1 reset not still connected to 0x560d294908f0 2024-04-06T09:56:11.649 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:56:11.649+0000 7f2e84d66640 -1 reset not still connected to 0x560d29cd8f70 2024-04-06T09:56:11.649 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:56:11.649+0000 7f2e84d66640 -1 reset not still connected to 0x560d29cd9ba0 2024-04-06T09:56:11.649 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:56:11.649+0000 7f2e84d66640 -1 reset not still connected to 0x560d29d5b520 2024-04-06T09:56:11.649 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:56:11.649+0000 7f2e84d66640 -1 reset not still connected to 0x560d2a17d450 2024-04-06T09:56:11.649 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:56:11.649+0000 7f2e84d66640 -1 reset not still connected to 0x560d2a17d6c0 2024-04-06T09:56:11.649 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:56:11.649+0000 7f2e84d66640 -1 reset not still connected to 0x560d2a204b60 2024-04-06T09:56:11.649 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:56:11.649+0000 7f2e84d66640 -1 reset not still connected to 0x560d2a2336c0 2024-04-06T09:56:11.649 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:56:11.649+0000 7f2e84d66640 -1 reset not still connected to 0x560d2a5705b0 2024-04-06T09:56:11.650 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:56:11.649+0000 7f2e84d66640 -1 reset not still connected to 0x560d3620b520 2024-04-06T09:56:11.650 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:56:11.649+0000 7f2e84d66640 -1 reset not still connected to 0x560d368e4f70 2024-04-06T09:56:11.650 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:56:11.649+0000 7f2e84d66640 -1 reset not still connected to 0x560d381dd5f0 2024-04-06T09:56:11.650 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:56:11.649+0000 7f2e84d66640 -1 reset not still connected to 0x560d4bc7e8f0 2024-04-06T09:56:12.479 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ error_code=124 2024-04-06T09:56:12.479 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set -e 2024-04-06T09:56:12.479 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 124 -eq 124 ']' 2024-04-06T09:56:12.479 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:56:12.479 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-06T09:56:12.479 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ image=image_37 2024-04-06T09:56:12.479 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ create_image_and_enable_mirror cluster2 mirror image_37 journal 128M 2024-04-06T09:56:12.479 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-04-06T09:56:12.480 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:56:12.480 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:56:12.480 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:56:12.480 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_37 2024-04-06T09:56:12.480 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:56:12.480 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local mode=journal 2024-04-06T09:56:12.480 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n journal ']' 2024-04-06T09:56:12.480 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:56:12.480 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ create_image cluster2 mirror image_37 128M 2024-04-06T09:56:12.480 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-04-06T09:56:12.480 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:56:12.480 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:56:12.480 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:56:12.480 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_37 2024-04-06T09:56:12.480 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:56:12.481 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local size=128 2024-04-06T09:56:12.481 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n 128M ']' 2024-04-06T09:56:12.481 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ size=128M 2024-04-06T09:56:12.481 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:56:12.481 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,journaling mirror/image_37 2024-04-06T09:56:12.576 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' pool = image ']' 2024-04-06T09:56:12.576 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' mirror = mirror_parent ']' 2024-04-06T09:56:12.576 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n '' ']' 2024-04-06T09:56:12.576 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ stress_write_image cluster2 mirror image_37 2024-04-06T09:56:12.577 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-04-06T09:56:12.577 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:56:12.577 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_37 2024-04-06T09:56:12.577 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-04-06T09:56:12.577 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local duration=5 2024-04-06T09:56:12.577 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set +e 2024-04-06T09:56:12.578 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ timeout 5s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_37 --debug-rbd=20 --debug-journaler=20 2024-04-06T09:56:13.596 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:56:13.597+0000 7f0965192640 -1 reset not still connected to 0x5632c0cfa000 2024-04-06T09:56:13.596 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:56:13.597+0000 7f0965192640 -1 reset not still connected to 0x5632cb3ac000 2024-04-06T09:56:13.597 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:56:13.597+0000 7f0965192640 -1 reset not still connected to 0x5632dac08dd0 2024-04-06T09:56:13.597 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:56:13.597+0000 7f0965192640 -1 reset not still connected to 0x5632dac09040 2024-04-06T09:56:13.597 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:56:13.597+0000 7f0965192640 -1 reset not still connected to 0x5632e1684ea0 2024-04-06T09:56:13.597 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:56:13.597+0000 7f0965192640 -1 reset not still connected to 0x5632e16851e0 2024-04-06T09:56:14.301 INFO:tasks.ceph.cluster2.osd.1.smithi192.stderr:2024-04-06T09:56:14.303+0000 7f0de0719640 -1 reset not still connected to 0x55bfa5642b60 2024-04-06T09:56:14.302 INFO:tasks.ceph.cluster2.osd.1.smithi192.stderr:2024-04-06T09:56:14.303+0000 7f0de0719640 -1 reset not still connected to 0x55bfa6e48410 2024-04-06T09:56:14.302 INFO:tasks.ceph.cluster2.osd.1.smithi192.stderr:2024-04-06T09:56:14.303+0000 7f0de0719640 -1 reset not still connected to 0x55bfaef3ba00 2024-04-06T09:56:14.303 INFO:tasks.ceph.cluster2.osd.1.smithi192.stderr:2024-04-06T09:56:14.303+0000 7f0de0719640 -1 reset not still connected to 0x55bfd74fb380 2024-04-06T09:56:14.303 INFO:tasks.ceph.cluster2.osd.1.smithi192.stderr:2024-04-06T09:56:14.303+0000 7f0de0719640 -1 reset not still connected to 0x55bfd74fb450 2024-04-06T09:56:15.417 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:100 IOs, 29138944 bytes 2024-04-06T09:56:16.135 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:56:16.133+0000 7f0965192640 -1 reset not still connected to 0x5632a89f52b0 2024-04-06T09:56:16.135 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:56:16.133+0000 7f0965192640 -1 reset not still connected to 0x5632abc81d40 2024-04-06T09:56:16.135 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:56:16.133+0000 7f0965192640 -1 reset not still connected to 0x5632abcbe820 2024-04-06T09:56:16.136 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:56:16.133+0000 7f0965192640 -1 reset not still connected to 0x5632b9592000 2024-04-06T09:56:16.137 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:56:16.133+0000 7f0965192640 -1 reset not still connected to 0x5632b9593110 2024-04-06T09:56:16.137 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:56:16.133+0000 7f0965192640 -1 reset not still connected to 0x5632c0cfa000 2024-04-06T09:56:16.140 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:56:16.141+0000 7f0965192640 -1 reset not still connected to 0x5632cb3ac000 2024-04-06T09:56:16.143 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:56:16.145+0000 7f0965192640 -1 reset not still connected to 0x5632dac09040 2024-04-06T09:56:16.143 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:56:16.145+0000 7f0965192640 -1 reset not still connected to 0x5632e1684ea0 2024-04-06T09:56:16.144 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:56:16.145+0000 7f0965192640 -1 reset not still connected to 0x5632e16851e0 2024-04-06T09:56:16.602 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:56:16.601+0000 7f2e84d66640 -1 reset not still connected to 0x560d2a17d450 2024-04-06T09:56:16.602 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:56:16.601+0000 7f2e84d66640 -1 reset not still connected to 0x560d2a17d6c0 2024-04-06T09:56:16.614 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:56:16.613+0000 7f2e84d66640 -1 reset not still connected to 0x560d2a204b60 2024-04-06T09:56:16.614 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:56:16.613+0000 7f2e84d66640 -1 reset not still connected to 0x560d2a205ba0 2024-04-06T09:56:16.614 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:56:16.613+0000 7f2e84d66640 -1 reset not still connected to 0x560d2a2336c0 2024-04-06T09:56:16.614 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:56:16.613+0000 7f2e84d66640 -1 reset not still connected to 0x560d2a570000 2024-04-06T09:56:16.614 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:56:16.613+0000 7f2e84d66640 -1 reset not still connected to 0x560d2a5705b0 2024-04-06T09:56:16.614 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:56:16.613+0000 7f2e84d66640 -1 reset not still connected to 0x560d3620b520 2024-04-06T09:56:16.615 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:56:16.613+0000 7f2e84d66640 -1 reset not still connected to 0x560d368e4f70 2024-04-06T09:56:16.615 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:56:16.613+0000 7f2e84d66640 -1 reset not still connected to 0x560d36912ea0 2024-04-06T09:56:16.615 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:56:16.613+0000 7f2e84d66640 -1 reset not still connected to 0x560d381dd5f0 2024-04-06T09:56:16.615 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:56:16.613+0000 7f2e84d66640 -1 reset not still connected to 0x560d390dca90 2024-04-06T09:56:16.615 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:56:16.613+0000 7f2e84d66640 -1 reset not still connected to 0x560d4bc7e8f0 2024-04-06T09:56:16.615 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:56:16.613+0000 7f2e84d66640 -1 reset not still connected to 0x560d50adba00 2024-04-06T09:56:16.716 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:56:16.713+0000 7f0965192640 -1 reset not still connected to 0x5632abcbe820 2024-04-06T09:56:17.393 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:56:17.393+0000 7f0965192640 -1 reset not still connected to 0x5632abcbe820 2024-04-06T09:56:17.395 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:56:17.393+0000 7f0965192640 -1 reset not still connected to 0x5632cb3ac000 2024-04-06T09:56:17.402 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:56:17.393+0000 7f0965192640 -1 reset not still connected to 0x5632e16851e0 2024-04-06T09:56:17.584 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ error_code=124 2024-04-06T09:56:17.585 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set -e 2024-04-06T09:56:17.585 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 124 -eq 124 ']' 2024-04-06T09:56:17.585 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:56:17.585 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-06T09:56:17.585 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ image=image_38 2024-04-06T09:56:17.585 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ create_image_and_enable_mirror cluster2 mirror image_38 journal 128M 2024-04-06T09:56:17.585 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-04-06T09:56:17.585 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:56:17.585 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:56:17.585 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:56:17.585 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_38 2024-04-06T09:56:17.585 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:56:17.585 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local mode=journal 2024-04-06T09:56:17.585 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n journal ']' 2024-04-06T09:56:17.586 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:56:17.586 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ create_image cluster2 mirror image_38 128M 2024-04-06T09:56:17.586 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-04-06T09:56:17.586 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:56:17.586 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:56:17.586 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:56:17.586 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_38 2024-04-06T09:56:17.586 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:56:17.586 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local size=128 2024-04-06T09:56:17.586 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n 128M ']' 2024-04-06T09:56:17.586 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ size=128M 2024-04-06T09:56:17.586 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:56:17.586 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,journaling mirror/image_38 2024-04-06T09:56:17.711 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' pool = image ']' 2024-04-06T09:56:17.712 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' mirror = mirror_parent ']' 2024-04-06T09:56:17.712 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n '' ']' 2024-04-06T09:56:17.712 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ stress_write_image cluster2 mirror image_38 2024-04-06T09:56:17.712 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-04-06T09:56:17.712 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:56:17.712 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_38 2024-04-06T09:56:17.712 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-04-06T09:56:17.713 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local duration=13 2024-04-06T09:56:17.713 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set +e 2024-04-06T09:56:17.713 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ timeout 13s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_38 --debug-rbd=20 --debug-journaler=20 2024-04-06T09:56:18.061 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:56:18.061+0000 7f0965192640 -1 reset not still connected to 0x5632abc812b0 2024-04-06T09:56:18.061 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:56:18.061+0000 7f0965192640 -1 reset not still connected to 0x5632abc81d40 2024-04-06T09:56:18.061 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:56:18.061+0000 7f0965192640 -1 reset not still connected to 0x5632abcbe820 2024-04-06T09:56:18.061 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:56:18.061+0000 7f0965192640 -1 reset not still connected to 0x5632b9592000 2024-04-06T09:56:18.061 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:56:18.061+0000 7f0965192640 -1 reset not still connected to 0x5632b9593110 2024-04-06T09:56:18.061 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:56:18.061+0000 7f0965192640 -1 reset not still connected to 0x5632cb3ac000 2024-04-06T09:56:18.061 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:56:18.061+0000 7f0965192640 -1 reset not still connected to 0x5632dac08dd0 2024-04-06T09:56:18.061 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:56:18.061+0000 7f0965192640 -1 reset not still connected to 0x5632dac09040 2024-04-06T09:56:18.061 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:56:18.061+0000 7f0965192640 -1 reset not still connected to 0x5632e1684ea0 2024-04-06T09:56:18.061 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:56:18.061+0000 7f0965192640 -1 reset not still connected to 0x5632e16851e0 2024-04-06T09:56:18.613 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:56:18.601+0000 7f2e84d66640 -1 reset not still connected to 0x560d2a17d6c0 2024-04-06T09:56:18.614 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:56:18.609+0000 7f2e84d66640 -1 reset not still connected to 0x560d2a204b60 2024-04-06T09:56:18.621 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:56:18.621+0000 7f2e84d66640 -1 reset not still connected to 0x560d2a205ba0 2024-04-06T09:56:18.621 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:56:18.621+0000 7f2e84d66640 -1 reset not still connected to 0x560d2a2336c0 2024-04-06T09:56:18.621 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:56:18.621+0000 7f2e84d66640 -1 reset not still connected to 0x560d2a5705b0 2024-04-06T09:56:18.621 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:56:18.621+0000 7f2e84d66640 -1 reset not still connected to 0x560d3620b520 2024-04-06T09:56:18.621 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:56:18.621+0000 7f2e84d66640 -1 reset not still connected to 0x560d368e4f70 2024-04-06T09:56:18.621 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:56:18.621+0000 7f2e84d66640 -1 reset not still connected to 0x560d36912ea0 2024-04-06T09:56:18.622 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:56:18.621+0000 7f2e84d66640 -1 reset not still connected to 0x560d381dd5f0 2024-04-06T09:56:18.622 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:56:18.621+0000 7f2e84d66640 -1 reset not still connected to 0x560d4bc7e8f0 2024-04-06T09:56:18.622 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:56:18.621+0000 7f2e84d66640 -1 reset not still connected to 0x560d50adba00 2024-04-06T09:56:20.218 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:56:20.213+0000 7f0965192640 -1 reset not still connected to 0x5632a1b73ee0 2024-04-06T09:56:20.226 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:56:20.221+0000 7f0965192640 -1 reset not still connected to 0x5632a30f0270 2024-04-06T09:56:20.226 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:56:20.221+0000 7f0965192640 -1 reset not still connected to 0x5632a30f0f70 2024-04-06T09:56:20.226 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:56:20.221+0000 7f0965192640 -1 reset not still connected to 0x5632a417e750 2024-04-06T09:56:20.227 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:56:20.225+0000 7f0965192640 -1 reset not still connected to 0x5632a5e9dee0 2024-04-06T09:56:20.228 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:56:20.225+0000 7f0965192640 -1 reset not still connected to 0x5632a89f4750 2024-04-06T09:56:20.228 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:56:20.225+0000 7f0965192640 -1 reset not still connected to 0x5632a89f52b0 2024-04-06T09:56:20.228 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:56:20.225+0000 7f0965192640 -1 reset not still connected to 0x5632abc812b0 2024-04-06T09:56:20.228 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:56:20.225+0000 7f0965192640 -1 reset not still connected to 0x5632abc81d40 2024-04-06T09:56:20.228 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:56:20.225+0000 7f0965192640 -1 reset not still connected to 0x5632b9592000 2024-04-06T09:56:20.228 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:56:20.225+0000 7f0965192640 -1 reset not still connected to 0x5632b9593110 2024-04-06T09:56:20.228 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:56:20.225+0000 7f0965192640 -1 reset not still connected to 0x5632c0cfa000 2024-04-06T09:56:20.228 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:56:20.225+0000 7f0965192640 -1 reset not still connected to 0x5632dac09040 2024-04-06T09:56:20.230 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:56:20.225+0000 7f0965192640 -1 reset not still connected to 0x5632e1684ea0 2024-04-06T09:56:20.230 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:56:20.225+0000 7f0965192640 -1 reset not still connected to 0x5632e16851e0 2024-04-06T09:56:21.046 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:56:21.045+0000 7f2e84d66640 -1 reset not still connected to 0x560d27c3f2b0 2024-04-06T09:56:21.047 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:56:21.045+0000 7f2e84d66640 -1 reset not still connected to 0x560d2a17d6c0 2024-04-06T09:56:21.106 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:100 IOs, 30912512 bytes 2024-04-06T09:56:21.248 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi192.stderr:2024-04-06T09:56:21.247+0000 7f3d28bce640 -1 rbd::mirror::InstanceWatcher: C_NotifyInstanceRequest: 0x56067b2cdec0 finish: resending after timeout 2024-04-06T09:56:21.248 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi192.stderr:2024-04-06T09:56:21.247+0000 7f3d28bce640 -1 rbd::mirror::InstanceWatcher: C_NotifyInstanceRequest: 0x560668d72600 finish: resending after timeout 2024-04-06T09:56:21.251 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi192.stderr:2024-04-06T09:56:21.251+0000 7f3d293cf640 -1 rbd::mirror::InstanceWatcher: C_NotifyInstanceRequest: 0x56067b2cd380 finish: resending after timeout 2024-04-06T09:56:21.352 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:56:21.349+0000 7f2e84d66640 -1 reset not still connected to 0x560d29cd9ba0 2024-04-06T09:56:21.352 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:56:21.349+0000 7f2e84d66640 -1 reset not still connected to 0x560d29d5b520 2024-04-06T09:56:21.352 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:56:21.349+0000 7f2e84d66640 -1 reset not still connected to 0x560d2a17d450 2024-04-06T09:56:21.353 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:56:21.349+0000 7f2e84d66640 -1 reset not still connected to 0x560d2a17d6c0 2024-04-06T09:56:21.354 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:56:21.353+0000 7f2e84d66640 -1 reset not still connected to 0x560d2a205ba0 2024-04-06T09:56:21.354 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:56:21.353+0000 7f2e84d66640 -1 reset not still connected to 0x560d2a2336c0 2024-04-06T09:56:21.354 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:56:21.353+0000 7f2e84d66640 -1 reset not still connected to 0x560d2a5705b0 2024-04-06T09:56:21.356 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:56:21.353+0000 7f2e84d66640 -1 reset not still connected to 0x560d368e4f70 2024-04-06T09:56:21.365 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:56:21.361+0000 7f2e84d66640 -1 reset not still connected to 0x560d36912ea0 2024-04-06T09:56:21.365 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:56:21.361+0000 7f2e84d66640 -1 reset not still connected to 0x560d381dd5f0 2024-04-06T09:56:21.365 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:56:21.361+0000 7f2e84d66640 -1 reset not still connected to 0x560d4bc7e8f0 2024-04-06T09:56:21.365 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:56:21.361+0000 7f2e84d66640 -1 reset not still connected to 0x560d50adba00 2024-04-06T09:56:21.759 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:56:21.757+0000 7f2e84d66640 -1 reset not still connected to 0x560d27c3edd0 2024-04-06T09:56:21.762 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:56:21.761+0000 7f2e84d66640 -1 reset not still connected to 0x560d27c3f2b0 2024-04-06T09:56:21.764 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:56:21.765+0000 7f2e84d66640 -1 reset not still connected to 0x560d294908f0 2024-04-06T09:56:21.778 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:56:21.777+0000 7f2e84d66640 -1 reset not still connected to 0x560d29cd9ba0 2024-04-06T09:56:21.778 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:56:21.777+0000 7f2e84d66640 -1 reset not still connected to 0x560d29d5b520 2024-04-06T09:56:21.778 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:56:21.777+0000 7f2e84d66640 -1 reset not still connected to 0x560d2a17d450 2024-04-06T09:56:21.778 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:56:21.777+0000 7f2e84d66640 -1 reset not still connected to 0x560d2a17d6c0 2024-04-06T09:56:21.778 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:56:21.777+0000 7f2e84d66640 -1 reset not still connected to 0x560d2a204b60 2024-04-06T09:56:21.778 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:56:21.777+0000 7f2e84d66640 -1 reset not still connected to 0x560d2a205ba0 2024-04-06T09:56:21.778 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:56:21.777+0000 7f2e84d66640 -1 reset not still connected to 0x560d2a2336c0 2024-04-06T09:56:21.778 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:56:21.777+0000 7f2e84d66640 -1 reset not still connected to 0x560d2a5705b0 2024-04-06T09:56:21.778 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:56:21.777+0000 7f2e84d66640 -1 reset not still connected to 0x560d3620b520 2024-04-06T09:56:21.778 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:56:21.777+0000 7f2e84d66640 -1 reset not still connected to 0x560d368e4f70 2024-04-06T09:56:21.779 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:56:21.777+0000 7f2e84d66640 -1 reset not still connected to 0x560d381dd5f0 2024-04-06T09:56:21.779 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:56:21.777+0000 7f2e84d66640 -1 reset not still connected to 0x560d4bc7e8f0 2024-04-06T09:56:21.779 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:56:21.777+0000 7f2e84d66640 -1 reset not still connected to 0x560d50adba00 2024-04-06T09:56:21.802 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:200 IOs, 38157312 bytes 2024-04-06T09:56:23.318 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:56:23.317+0000 7f0965192640 -1 reset not still connected to 0x5632a4cdbba0 2024-04-06T09:56:23.318 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:56:23.317+0000 7f0965192640 -1 reset not still connected to 0x5632a5e9dee0 2024-04-06T09:56:23.318 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:56:23.317+0000 7f0965192640 -1 reset not still connected to 0x5632a89f4750 2024-04-06T09:56:23.328 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:56:23.325+0000 7f0965192640 -1 reset not still connected to 0x5632a89f52b0 2024-04-06T09:56:23.328 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:56:23.325+0000 7f0965192640 -1 reset not still connected to 0x5632abc812b0 2024-04-06T09:56:23.328 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:56:23.325+0000 7f0965192640 -1 reset not still connected to 0x5632abc81d40 2024-04-06T09:56:23.328 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:56:23.325+0000 7f0965192640 -1 reset not still connected to 0x5632b9592000 2024-04-06T09:56:23.328 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:56:23.325+0000 7f0965192640 -1 reset not still connected to 0x5632b9593110 2024-04-06T09:56:23.328 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:56:23.325+0000 7f0965192640 -1 reset not still connected to 0x5632c0cfa000 2024-04-06T09:56:23.328 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:56:23.325+0000 7f0965192640 -1 reset not still connected to 0x5632dac09040 2024-04-06T09:56:23.328 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:56:23.325+0000 7f0965192640 -1 reset not still connected to 0x5632e1684ea0 2024-04-06T09:56:23.328 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:56:23.325+0000 7f0965192640 -1 reset not still connected to 0x5632e16851e0 2024-04-06T09:56:23.439 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:56:23.441+0000 7f0965192640 -1 reset not still connected to 0x5632a1988000 2024-04-06T09:56:23.440 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:56:23.441+0000 7f0965192640 -1 reset not still connected to 0x5632a1989040 2024-04-06T09:56:23.440 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:56:23.441+0000 7f0965192640 -1 reset not still connected to 0x5632a1b73ee0 2024-04-06T09:56:23.440 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:56:23.441+0000 7f0965192640 -1 reset not still connected to 0x5632a2793520 2024-04-06T09:56:23.442 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:56:23.441+0000 7f0965192640 -1 reset not still connected to 0x5632a30f0f70 2024-04-06T09:56:23.451 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:56:23.449+0000 7f0965192640 -1 reset not still connected to 0x5632a417e750 2024-04-06T09:56:23.451 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:56:23.449+0000 7f0965192640 -1 reset not still connected to 0x5632a4cda000 2024-04-06T09:56:23.451 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:56:23.449+0000 7f0965192640 -1 reset not still connected to 0x5632a4cdbba0 2024-04-06T09:56:23.451 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:56:23.449+0000 7f0965192640 -1 reset not still connected to 0x5632a5e9dee0 2024-04-06T09:56:23.451 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:56:23.449+0000 7f0965192640 -1 reset not still connected to 0x5632a89f4750 2024-04-06T09:56:23.451 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:56:23.449+0000 7f0965192640 -1 reset not still connected to 0x5632a89f52b0 2024-04-06T09:56:23.452 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:56:23.449+0000 7f0965192640 -1 reset not still connected to 0x5632abc812b0 2024-04-06T09:56:23.452 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:56:23.449+0000 7f0965192640 -1 reset not still connected to 0x5632abc81d40 2024-04-06T09:56:23.452 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:56:23.449+0000 7f0965192640 -1 reset not still connected to 0x5632abcbe820 2024-04-06T09:56:23.452 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:56:23.449+0000 7f0965192640 -1 reset not still connected to 0x5632b9592000 2024-04-06T09:56:23.452 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:56:23.449+0000 7f0965192640 -1 reset not still connected to 0x5632b9593110 2024-04-06T09:56:23.454 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:56:23.453+0000 7f0965192640 -1 reset not still connected to 0x5632c0cfa000 2024-04-06T09:56:23.459 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:56:23.457+0000 7f0965192640 -1 reset not still connected to 0x5632cb3ac000 2024-04-06T09:56:23.459 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:56:23.457+0000 7f0965192640 -1 reset not still connected to 0x5632dac08dd0 2024-04-06T09:56:23.459 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:56:23.457+0000 7f0965192640 -1 reset not still connected to 0x5632dac09040 2024-04-06T09:56:23.459 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:56:23.457+0000 7f0965192640 -1 reset not still connected to 0x5632e1684ea0 2024-04-06T09:56:23.459 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:56:23.457+0000 7f0965192640 -1 reset not still connected to 0x5632e16851e0 2024-04-06T09:56:23.518 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:56:23.517+0000 7f0965192640 -1 reset not still connected to 0x5632abc812b0 2024-04-06T09:56:23.518 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:56:23.517+0000 7f0965192640 -1 reset not still connected to 0x5632abc81d40 2024-04-06T09:56:23.518 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:56:23.517+0000 7f0965192640 -1 reset not still connected to 0x5632abcbe820 2024-04-06T09:56:23.518 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:56:23.517+0000 7f0965192640 -1 reset not still connected to 0x5632b9592000 2024-04-06T09:56:23.526 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:56:23.525+0000 7f0965192640 -1 reset not still connected to 0x5632b9593110 2024-04-06T09:56:23.527 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:56:23.525+0000 7f0965192640 -1 reset not still connected to 0x5632c0cfa000 2024-04-06T09:56:23.528 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:56:23.529+0000 7f0965192640 -1 reset not still connected to 0x5632cb3ac000 2024-04-06T09:56:23.528 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:56:23.529+0000 7f0965192640 -1 reset not still connected to 0x5632dac08dd0 2024-04-06T09:56:23.528 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:56:23.529+0000 7f0965192640 -1 reset not still connected to 0x5632dac09040 2024-04-06T09:56:23.528 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:56:23.529+0000 7f0965192640 -1 reset not still connected to 0x5632e1684ea0 2024-04-06T09:56:23.528 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:56:23.529+0000 7f0965192640 -1 reset not still connected to 0x5632e16851e0 2024-04-06T09:56:24.410 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:56:24.409+0000 7f0965192640 -1 reset not still connected to 0x5632a5e9dee0 2024-04-06T09:56:24.411 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:56:24.413+0000 7f0965192640 -1 reset not still connected to 0x5632a89f4750 2024-04-06T09:56:24.413 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:56:24.413+0000 7f0965192640 -1 reset not still connected to 0x5632abc812b0 2024-04-06T09:56:24.413 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:56:24.413+0000 7f0965192640 -1 reset not still connected to 0x5632abc81d40 2024-04-06T09:56:24.415 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:56:24.417+0000 7f0965192640 -1 reset not still connected to 0x5632b9592000 2024-04-06T09:56:24.416 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:56:24.417+0000 7f0965192640 -1 reset not still connected to 0x5632b9593110 2024-04-06T09:56:24.418 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:56:24.417+0000 7f0965192640 -1 reset not still connected to 0x5632c0cfa000 2024-04-06T09:56:24.418 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:56:24.417+0000 7f0965192640 -1 reset not still connected to 0x5632dac09040 2024-04-06T09:56:24.418 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:56:24.417+0000 7f0965192640 -1 reset not still connected to 0x5632e1684ea0 2024-04-06T09:56:24.419 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:56:24.417+0000 7f0965192640 -1 reset not still connected to 0x5632e16851e0 2024-04-06T09:56:24.775 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:56:24.773+0000 7f0965192640 -1 reset not still connected to 0x5632abcbe820 2024-04-06T09:56:24.775 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:56:24.773+0000 7f0965192640 -1 reset not still connected to 0x5632dac09040 2024-04-06T09:56:24.780 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:56:24.777+0000 7f0965192640 -1 reset not still connected to 0x5632e1684ea0 2024-04-06T09:56:24.786 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:56:24.781+0000 7f0965192640 -1 reset not still connected to 0x5632e16851e0 2024-04-06T09:56:25.708 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:56:25.705+0000 7f2e84d66640 -1 reset not still connected to 0x560d27567380 2024-04-06T09:56:25.708 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:56:25.705+0000 7f2e84d66640 -1 reset not still connected to 0x560d27839930 2024-04-06T09:56:25.708 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:56:25.705+0000 7f2e84d66640 -1 reset not still connected to 0x560d27c3edd0 2024-04-06T09:56:25.709 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:56:25.705+0000 7f2e84d66640 -1 reset not still connected to 0x560d27c3f2b0 2024-04-06T09:56:25.709 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:56:25.705+0000 7f2e84d66640 -1 reset not still connected to 0x560d294908f0 2024-04-06T09:56:25.709 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:56:25.705+0000 7f2e84d66640 -1 reset not still connected to 0x560d29cd9ba0 2024-04-06T09:56:25.709 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:56:25.705+0000 7f2e84d66640 -1 reset not still connected to 0x560d29d5b520 2024-04-06T09:56:25.709 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:56:25.709+0000 7f2e84d66640 -1 reset not still connected to 0x560d2a17d450 2024-04-06T09:56:25.715 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:56:25.709+0000 7f2e84d66640 -1 reset not still connected to 0x560d2a17d6c0 2024-04-06T09:56:25.715 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:56:25.709+0000 7f2e84d66640 -1 reset not still connected to 0x560d2a204b60 2024-04-06T09:56:25.717 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:56:25.709+0000 7f2e84d66640 -1 reset not still connected to 0x560d2a205ba0 2024-04-06T09:56:25.717 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:56:25.709+0000 7f2e84d66640 -1 reset not still connected to 0x560d2a2336c0 2024-04-06T09:56:25.717 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:56:25.709+0000 7f2e84d66640 -1 reset not still connected to 0x560d2a5705b0 2024-04-06T09:56:25.717 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:56:25.709+0000 7f2e84d66640 -1 reset not still connected to 0x560d3620b520 2024-04-06T09:56:25.717 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:56:25.709+0000 7f2e84d66640 -1 reset not still connected to 0x560d368e4f70 2024-04-06T09:56:25.723 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:56:25.721+0000 7f2e84d66640 -1 reset not still connected to 0x560d381dd5f0 2024-04-06T09:56:25.723 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:56:25.721+0000 7f2e84d66640 -1 reset not still connected to 0x560d4bc7e8f0 2024-04-06T09:56:25.723 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:56:25.721+0000 7f2e84d66640 -1 reset not still connected to 0x560d50adba00 2024-04-06T09:56:26.003 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:56:26.001+0000 7f0965192640 -1 reset not still connected to 0x5632a89f52b0 2024-04-06T09:56:26.005 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:56:26.005+0000 7f0965192640 -1 reset not still connected to 0x5632abc812b0 2024-04-06T09:56:26.006 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:56:26.005+0000 7f0965192640 -1 reset not still connected to 0x5632abc81d40 2024-04-06T09:56:26.006 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:56:26.005+0000 7f0965192640 -1 reset not still connected to 0x5632abcbe820 2024-04-06T09:56:26.006 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:56:26.005+0000 7f0965192640 -1 reset not still connected to 0x5632b9592000 2024-04-06T09:56:26.007 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:56:26.009+0000 7f0965192640 -1 reset not still connected to 0x5632b9593110 2024-04-06T09:56:26.008 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:56:26.009+0000 7f0965192640 -1 reset not still connected to 0x5632c0cfa000 2024-04-06T09:56:26.008 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:56:26.009+0000 7f0965192640 -1 reset not still connected to 0x5632cb3ac000 2024-04-06T09:56:26.008 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:56:26.009+0000 7f0965192640 -1 reset not still connected to 0x5632dac09040 2024-04-06T09:56:26.008 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:56:26.009+0000 7f0965192640 -1 reset not still connected to 0x5632e1684ea0 2024-04-06T09:56:26.008 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:56:26.009+0000 7f0965192640 -1 reset not still connected to 0x5632e16851e0 2024-04-06T09:56:26.087 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:56:26.085+0000 7f2e84d66640 -1 reset not still connected to 0x560d27567380 2024-04-06T09:56:26.087 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:56:26.085+0000 7f2e84d66640 -1 reset not still connected to 0x560d27839380 2024-04-06T09:56:26.087 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:56:26.085+0000 7f2e84d66640 -1 reset not still connected to 0x560d27839930 2024-04-06T09:56:26.087 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:56:26.085+0000 7f2e84d66640 -1 reset not still connected to 0x560d27c3edd0 2024-04-06T09:56:26.100 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:56:26.097+0000 7f2e84d66640 -1 reset not still connected to 0x560d27c3f2b0 2024-04-06T09:56:26.100 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:56:26.097+0000 7f2e84d66640 -1 reset not still connected to 0x560d294908f0 2024-04-06T09:56:26.103 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:56:26.101+0000 7f2e84d66640 -1 reset not still connected to 0x560d29cd8f70 2024-04-06T09:56:26.103 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:56:26.101+0000 7f2e84d66640 -1 reset not still connected to 0x560d29cd9ba0 2024-04-06T09:56:26.103 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:56:26.101+0000 7f2e84d66640 -1 reset not still connected to 0x560d29d5b520 2024-04-06T09:56:26.103 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:56:26.101+0000 7f2e84d66640 -1 reset not still connected to 0x560d2a17d450 2024-04-06T09:56:26.103 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:56:26.101+0000 7f2e84d66640 -1 reset not still connected to 0x560d2a17d6c0 2024-04-06T09:56:26.103 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:56:26.101+0000 7f2e84d66640 -1 reset not still connected to 0x560d2a204b60 2024-04-06T09:56:26.103 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:56:26.101+0000 7f2e84d66640 -1 reset not still connected to 0x560d2a205ba0 2024-04-06T09:56:26.103 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:56:26.101+0000 7f2e84d66640 -1 reset not still connected to 0x560d2a2336c0 2024-04-06T09:56:26.103 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:56:26.101+0000 7f2e84d66640 -1 reset not still connected to 0x560d2a5705b0 2024-04-06T09:56:26.103 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:56:26.101+0000 7f2e84d66640 -1 reset not still connected to 0x560d3620b520 2024-04-06T09:56:26.104 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:56:26.101+0000 7f2e84d66640 -1 reset not still connected to 0x560d368e4f70 2024-04-06T09:56:26.106 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:56:26.105+0000 7f2e84d66640 -1 reset not still connected to 0x560d36912ea0 2024-04-06T09:56:26.106 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:56:26.105+0000 7f2e84d66640 -1 reset not still connected to 0x560d381dd5f0 2024-04-06T09:56:26.106 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:56:26.105+0000 7f2e84d66640 -1 reset not still connected to 0x560d4bc7e8f0 2024-04-06T09:56:26.108 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:56:26.109+0000 7f2e84d66640 -1 reset not still connected to 0x560d50adba00 2024-04-06T09:56:26.633 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:56:26.629+0000 7f0965192640 -1 reset not still connected to 0x5632a08055f0 2024-04-06T09:56:26.633 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:56:26.629+0000 7f0965192640 -1 reset not still connected to 0x5632a0ff4410 2024-04-06T09:56:26.633 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:56:26.629+0000 7f0965192640 -1 reset not still connected to 0x5632a14ac820 2024-04-06T09:56:26.633 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:56:26.629+0000 7f0965192640 -1 reset not still connected to 0x5632a1988000 2024-04-06T09:56:26.633 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:56:26.629+0000 7f0965192640 -1 reset not still connected to 0x5632a1b73ee0 2024-04-06T09:56:26.633 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:56:26.629+0000 7f0965192640 -1 reset not still connected to 0x5632a2793520 2024-04-06T09:56:26.633 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:56:26.629+0000 7f0965192640 -1 reset not still connected to 0x5632a30f0270 2024-04-06T09:56:26.633 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:56:26.629+0000 7f0965192640 -1 reset not still connected to 0x5632a417e750 2024-04-06T09:56:26.634 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:56:26.629+0000 7f0965192640 -1 reset not still connected to 0x5632a4cda000 2024-04-06T09:56:26.634 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:56:26.629+0000 7f0965192640 -1 reset not still connected to 0x5632a4cda9c0 2024-04-06T09:56:26.634 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:56:26.629+0000 7f0965192640 -1 reset not still connected to 0x5632a4cdbba0 2024-04-06T09:56:26.634 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:56:26.629+0000 7f0965192640 -1 reset not still connected to 0x5632a5e9dee0 2024-04-06T09:56:26.634 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:56:26.629+0000 7f0965192640 -1 reset not still connected to 0x5632a89f4750 2024-04-06T09:56:26.636 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:56:26.637+0000 7f0965192640 -1 reset not still connected to 0x5632a89f52b0 2024-04-06T09:56:26.636 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:56:26.637+0000 7f0965192640 -1 reset not still connected to 0x5632abc812b0 2024-04-06T09:56:26.636 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:56:26.637+0000 7f0965192640 -1 reset not still connected to 0x5632abc81d40 2024-04-06T09:56:26.636 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:56:26.637+0000 7f0965192640 -1 reset not still connected to 0x5632abcbe820 2024-04-06T09:56:26.636 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:56:26.637+0000 7f0965192640 -1 reset not still connected to 0x5632b9592000 2024-04-06T09:56:26.637 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:56:26.637+0000 7f0965192640 -1 reset not still connected to 0x5632b9593110 2024-04-06T09:56:26.644 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:56:26.645+0000 7f0965192640 -1 reset not still connected to 0x5632c0cfa000 2024-04-06T09:56:26.647 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:56:26.645+0000 7f0965192640 -1 reset not still connected to 0x5632cb3ac000 2024-04-06T09:56:26.647 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:56:26.645+0000 7f0965192640 -1 reset not still connected to 0x5632dac08dd0 2024-04-06T09:56:26.647 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:56:26.645+0000 7f0965192640 -1 reset not still connected to 0x5632dac09040 2024-04-06T09:56:26.648 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:56:26.645+0000 7f0965192640 -1 reset not still connected to 0x5632e1684ea0 2024-04-06T09:56:26.648 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:56:26.645+0000 7f0965192640 -1 reset not still connected to 0x5632e16851e0 2024-04-06T09:56:26.846 INFO:tasks.ceph.cluster2.osd.1.smithi192.stderr:2024-04-06T09:56:26.847+0000 7f0de0719640 -1 reset not still connected to 0x55bfa5642b60 2024-04-06T09:56:26.922 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:56:26.921+0000 7f0965192640 -1 reset not still connected to 0x5632cb3ac000 2024-04-06T09:56:27.610 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:300 IOs, 112848896 bytes 2024-04-06T09:56:29.427 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:56:29.421+0000 7f0965192640 -1 reset not still connected to 0x5632dac09040 2024-04-06T09:56:29.427 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:56:29.421+0000 7f0965192640 -1 reset not still connected to 0x56329f99c0d0 2024-04-06T09:56:29.428 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:56:29.425+0000 7f0965192640 -1 reset not still connected to 0x5632a5e9dee0 2024-04-06T09:56:29.428 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:56:29.425+0000 7f0965192640 -1 reset not still connected to 0x5632abc812b0 2024-04-06T09:56:30.281 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:56:30.281+0000 7f0965192640 -1 reset not still connected to 0x5632abc812b0 2024-04-06T09:56:30.288 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:56:30.289+0000 7f0965192640 -1 reset not still connected to 0x5632b9593110 2024-04-06T09:56:30.288 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:56:30.289+0000 7f0965192640 -1 reset not still connected to 0x5632c0cfa000 2024-04-06T09:56:30.289 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:56:30.289+0000 7f0965192640 -1 reset not still connected to 0x5632cb3ac000 2024-04-06T09:56:30.289 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:56:30.289+0000 7f0965192640 -1 reset not still connected to 0x5632dac09040 2024-04-06T09:56:30.289 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:56:30.289+0000 7f0965192640 -1 reset not still connected to 0x5632e16851e0 2024-04-06T09:56:30.724 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ error_code=124 2024-04-06T09:56:30.724 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set -e 2024-04-06T09:56:30.724 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 124 -eq 124 ']' 2024-04-06T09:56:30.724 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:56:30.724 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-06T09:56:30.724 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ image=image_39 2024-04-06T09:56:30.725 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ create_image_and_enable_mirror cluster2 mirror image_39 journal 128M 2024-04-06T09:56:30.725 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-04-06T09:56:30.725 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:56:30.725 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:56:30.725 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:56:30.725 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_39 2024-04-06T09:56:30.725 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:56:30.725 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local mode=journal 2024-04-06T09:56:30.725 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n journal ']' 2024-04-06T09:56:30.725 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:56:30.725 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ create_image cluster2 mirror image_39 128M 2024-04-06T09:56:30.725 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-04-06T09:56:30.725 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:56:30.725 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:56:30.725 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:56:30.726 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_39 2024-04-06T09:56:30.726 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:56:30.726 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local size=128 2024-04-06T09:56:30.726 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n 128M ']' 2024-04-06T09:56:30.726 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ size=128M 2024-04-06T09:56:30.726 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:56:30.726 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,journaling mirror/image_39 2024-04-06T09:56:30.849 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' pool = image ']' 2024-04-06T09:56:30.849 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' mirror = mirror_parent ']' 2024-04-06T09:56:30.849 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n '' ']' 2024-04-06T09:56:30.849 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ stress_write_image cluster2 mirror image_39 2024-04-06T09:56:30.849 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-04-06T09:56:30.849 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:56:30.849 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_39 2024-04-06T09:56:30.849 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-04-06T09:56:30.851 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local duration=11 2024-04-06T09:56:30.851 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set +e 2024-04-06T09:56:30.851 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ timeout 11s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_39 --debug-rbd=20 --debug-journaler=20 2024-04-06T09:56:31.368 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:100 IOs, 623616 bytes 2024-04-06T09:56:31.930 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:56:31.929+0000 7f0965192640 -1 reset not still connected to 0x5632c0cfa000 2024-04-06T09:56:31.930 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:56:31.929+0000 7f0965192640 -1 reset not still connected to 0x5632cb3ac000 2024-04-06T09:56:31.930 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:56:31.929+0000 7f0965192640 -1 reset not still connected to 0x5632dac08dd0 2024-04-06T09:56:31.930 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:56:31.929+0000 7f0965192640 -1 reset not still connected to 0x5632dac09040 2024-04-06T09:56:31.930 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:56:31.929+0000 7f0965192640 -1 reset not still connected to 0x5632e1684ea0 2024-04-06T09:56:31.930 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:56:31.929+0000 7f0965192640 -1 reset not still connected to 0x5632e16851e0 2024-04-06T09:56:33.884 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:200 IOs, 21003264 bytes 2024-04-06T09:56:34.651 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:56:34.649+0000 7f0965192640 -1 reset not still connected to 0x5632a4cdbba0 2024-04-06T09:56:34.651 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:56:34.649+0000 7f0965192640 -1 reset not still connected to 0x5632a5e9dee0 2024-04-06T09:56:34.653 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:56:34.653+0000 7f0965192640 -1 reset not still connected to 0x5632a89f4750 2024-04-06T09:56:34.654 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:56:34.653+0000 7f0965192640 -1 reset not still connected to 0x5632a89f52b0 2024-04-06T09:56:34.654 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:56:34.653+0000 7f0965192640 -1 reset not still connected to 0x5632abc812b0 2024-04-06T09:56:34.654 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:56:34.653+0000 7f0965192640 -1 reset not still connected to 0x5632abc81d40 2024-04-06T09:56:34.662 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:56:34.661+0000 7f0965192640 -1 reset not still connected to 0x5632abcbe820 2024-04-06T09:56:34.662 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:56:34.661+0000 7f0965192640 -1 reset not still connected to 0x5632b81f3a00 2024-04-06T09:56:34.663 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:56:34.661+0000 7f0965192640 -1 reset not still connected to 0x5632b9592000 2024-04-06T09:56:34.663 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:56:34.661+0000 7f0965192640 -1 reset not still connected to 0x5632b9593110 2024-04-06T09:56:34.663 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:56:34.661+0000 7f0965192640 -1 reset not still connected to 0x5632c0cfa000 2024-04-06T09:56:34.663 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:56:34.661+0000 7f0965192640 -1 reset not still connected to 0x5632cb3ac000 2024-04-06T09:56:34.663 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:56:34.661+0000 7f0965192640 -1 reset not still connected to 0x5632dac08dd0 2024-04-06T09:56:34.663 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:56:34.661+0000 7f0965192640 -1 reset not still connected to 0x5632dac09040 2024-04-06T09:56:34.663 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:56:34.661+0000 7f0965192640 -1 reset not still connected to 0x5632e1684ea0 2024-04-06T09:56:34.663 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:56:34.661+0000 7f0965192640 -1 reset not still connected to 0x5632e16851e0 2024-04-06T09:56:35.625 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:300 IOs, 57664512 bytes 2024-04-06T09:56:38.907 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:56:38.904+0000 7f0965192640 -1 reset not still connected to 0x5632a1988000 2024-04-06T09:56:38.908 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:56:38.904+0000 7f0965192640 -1 reset not still connected to 0x5632a1b73ee0 2024-04-06T09:56:38.910 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:56:38.908+0000 7f0965192640 -1 reset not still connected to 0x5632a2793520 2024-04-06T09:56:38.912 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:56:38.908+0000 7f0965192640 -1 reset not still connected to 0x5632a30f0270 2024-04-06T09:56:38.913 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:56:38.912+0000 7f0965192640 -1 reset not still connected to 0x5632a30f0f70 2024-04-06T09:56:38.915 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:56:38.912+0000 7f0965192640 -1 reset not still connected to 0x5632a417e750 2024-04-06T09:56:38.916 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:56:38.912+0000 7f0965192640 -1 reset not still connected to 0x5632a4cda000 2024-04-06T09:56:38.917 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:56:38.916+0000 7f0965192640 -1 reset not still connected to 0x5632a4cdbba0 2024-04-06T09:56:38.917 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:56:38.916+0000 7f0965192640 -1 reset not still connected to 0x5632a5e9dee0 2024-04-06T09:56:38.920 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:56:38.920+0000 7f0965192640 -1 reset not still connected to 0x5632a89f4750 2024-04-06T09:56:38.920 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:56:38.920+0000 7f0965192640 -1 reset not still connected to 0x5632a89f52b0 2024-04-06T09:56:38.920 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:56:38.920+0000 7f0965192640 -1 reset not still connected to 0x5632abc812b0 2024-04-06T09:56:38.920 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:56:38.920+0000 7f0965192640 -1 reset not still connected to 0x5632abc81d40 2024-04-06T09:56:38.921 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:56:38.920+0000 7f0965192640 -1 reset not still connected to 0x5632abcbe820 2024-04-06T09:56:38.921 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:56:38.920+0000 7f0965192640 -1 reset not still connected to 0x5632b81f3a00 2024-04-06T09:56:38.922 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:56:38.920+0000 7f0965192640 -1 reset not still connected to 0x5632b9592000 2024-04-06T09:56:38.922 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:56:38.920+0000 7f0965192640 -1 reset not still connected to 0x5632b9593110 2024-04-06T09:56:38.922 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:56:38.920+0000 7f0965192640 -1 reset not still connected to 0x5632c0cfa000 2024-04-06T09:56:38.923 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:56:38.920+0000 7f0965192640 -1 reset not still connected to 0x5632cb3ac000 2024-04-06T09:56:38.923 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:56:38.920+0000 7f0965192640 -1 reset not still connected to 0x5632dac08dd0 2024-04-06T09:56:38.923 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:56:38.920+0000 7f0965192640 -1 reset not still connected to 0x5632dac09040 2024-04-06T09:56:38.923 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:56:38.920+0000 7f0965192640 -1 reset not still connected to 0x5632e1684ea0 2024-04-06T09:56:38.923 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:56:38.920+0000 7f0965192640 -1 reset not still connected to 0x5632e16851e0 2024-04-06T09:56:41.298 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi192.stderr:2024-04-06T09:56:41.299+0000 7f3d28bce640 -1 rbd::mirror::InstanceWatcher: C_NotifyInstanceRequest: 0x560668d72600 finish: resending after timeout 2024-04-06T09:56:41.298 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi192.stderr:2024-04-06T09:56:41.299+0000 7f3d28bce640 -1 rbd::mirror::InstanceWatcher: C_NotifyInstanceRequest: 0x56067b2cd380 finish: resending after timeout 2024-04-06T09:56:41.398 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:56:41.396+0000 7f2e84d66640 -1 reset not still connected to 0x560d2a205ba0 2024-04-06T09:56:41.398 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:56:41.396+0000 7f2e84d66640 -1 reset not still connected to 0x560d2a2336c0 2024-04-06T09:56:41.398 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:56:41.396+0000 7f2e84d66640 -1 reset not still connected to 0x560d2a5705b0 2024-04-06T09:56:41.410 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:56:41.408+0000 7f2e84d66640 -1 reset not still connected to 0x560d368e4f70 2024-04-06T09:56:41.410 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:56:41.408+0000 7f2e84d66640 -1 reset not still connected to 0x560d36912ea0 2024-04-06T09:56:41.410 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:56:41.408+0000 7f2e84d66640 -1 reset not still connected to 0x560d381dd5f0 2024-04-06T09:56:41.410 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:56:41.408+0000 7f2e84d66640 -1 reset not still connected to 0x560d390dca90 2024-04-06T09:56:41.410 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:56:41.408+0000 7f2e84d66640 -1 reset not still connected to 0x560d4bc7e8f0 2024-04-06T09:56:41.410 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:56:41.408+0000 7f2e84d66640 -1 reset not still connected to 0x560d50adba00 2024-04-06T09:56:41.862 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ error_code=124 2024-04-06T09:56:41.862 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set -e 2024-04-06T09:56:41.862 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 124 -eq 124 ']' 2024-04-06T09:56:41.862 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:56:41.862 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-06T09:56:41.862 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ image=image_40 2024-04-06T09:56:41.862 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ create_image_and_enable_mirror cluster2 mirror image_40 journal 128M 2024-04-06T09:56:41.862 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-04-06T09:56:41.862 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:56:41.863 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:56:41.863 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:56:41.863 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_40 2024-04-06T09:56:41.863 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:56:41.863 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local mode=journal 2024-04-06T09:56:41.863 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n journal ']' 2024-04-06T09:56:41.863 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:56:41.863 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ create_image cluster2 mirror image_40 128M 2024-04-06T09:56:41.863 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-04-06T09:56:41.863 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:56:41.863 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:56:41.863 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:56:41.864 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_40 2024-04-06T09:56:41.864 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:56:41.864 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local size=128 2024-04-06T09:56:41.864 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n 128M ']' 2024-04-06T09:56:41.864 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ size=128M 2024-04-06T09:56:41.864 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:56:41.864 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,journaling mirror/image_40 2024-04-06T09:56:41.939 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' pool = image ']' 2024-04-06T09:56:41.939 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' mirror = mirror_parent ']' 2024-04-06T09:56:41.939 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n '' ']' 2024-04-06T09:56:41.939 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ stress_write_image cluster2 mirror image_40 2024-04-06T09:56:41.939 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-04-06T09:56:41.939 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:56:41.939 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_40 2024-04-06T09:56:41.939 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-04-06T09:56:41.940 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local duration=5 2024-04-06T09:56:41.940 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set +e 2024-04-06T09:56:41.940 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ timeout 5s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_40 --debug-rbd=20 --debug-journaler=20 2024-04-06T09:56:42.011 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:56:42.008+0000 7f0965192640 -1 reset not still connected to 0x5632a0ff5110 2024-04-06T09:56:42.011 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:56:42.008+0000 7f0965192640 -1 reset not still connected to 0x5632a1988000 2024-04-06T09:56:42.013 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:56:42.012+0000 7f0965192640 -1 reset not still connected to 0x5632a1b73ee0 2024-04-06T09:56:42.024 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:56:42.024+0000 7f0965192640 -1 reset not still connected to 0x5632a2793520 2024-04-06T09:56:42.024 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:56:42.024+0000 7f0965192640 -1 reset not still connected to 0x5632a30f0270 2024-04-06T09:56:42.025 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:56:42.024+0000 7f0965192640 -1 reset not still connected to 0x5632a30f0f70 2024-04-06T09:56:42.025 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:56:42.024+0000 7f0965192640 -1 reset not still connected to 0x5632a417e750 2024-04-06T09:56:42.027 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:56:42.024+0000 7f0965192640 -1 reset not still connected to 0x5632a4cda000 2024-04-06T09:56:42.032 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:56:42.028+0000 7f0965192640 -1 reset not still connected to 0x5632a4cda9c0 2024-04-06T09:56:42.033 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:56:42.028+0000 7f0965192640 -1 reset not still connected to 0x5632a4cdbba0 2024-04-06T09:56:42.033 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:56:42.028+0000 7f0965192640 -1 reset not still connected to 0x5632a5e9dee0 2024-04-06T09:56:42.034 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:56:42.028+0000 7f0965192640 -1 reset not still connected to 0x5632a89f4750 2024-04-06T09:56:42.034 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:56:42.028+0000 7f0965192640 -1 reset not still connected to 0x5632a89f52b0 2024-04-06T09:56:42.034 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:56:42.028+0000 7f0965192640 -1 reset not still connected to 0x5632abc812b0 2024-04-06T09:56:42.034 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:56:42.028+0000 7f0965192640 -1 reset not still connected to 0x5632abc81d40 2024-04-06T09:56:42.036 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:56:42.036+0000 7f0965192640 -1 reset not still connected to 0x5632abcbe820 2024-04-06T09:56:42.036 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:56:42.036+0000 7f0965192640 -1 reset not still connected to 0x5632b81f3a00 2024-04-06T09:56:42.036 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:56:42.036+0000 7f0965192640 -1 reset not still connected to 0x5632b9592000 2024-04-06T09:56:42.036 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:56:42.036+0000 7f0965192640 -1 reset not still connected to 0x5632b9593110 2024-04-06T09:56:42.036 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:56:42.036+0000 7f0965192640 -1 reset not still connected to 0x5632c0cfa000 2024-04-06T09:56:42.036 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:56:42.036+0000 7f0965192640 -1 reset not still connected to 0x5632cb3ac000 2024-04-06T09:56:42.037 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:56:42.036+0000 7f0965192640 -1 reset not still connected to 0x5632dac08dd0 2024-04-06T09:56:42.037 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:56:42.036+0000 7f0965192640 -1 reset not still connected to 0x5632dac09040 2024-04-06T09:56:42.037 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:56:42.036+0000 7f0965192640 -1 reset not still connected to 0x5632e1684ea0 2024-04-06T09:56:42.037 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:56:42.036+0000 7f0965192640 -1 reset not still connected to 0x5632e16851e0 2024-04-06T09:56:43.672 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:100 IOs, 18724864 bytes 2024-04-06T09:56:43.834 INFO:tasks.ceph.cluster2.osd.0.smithi192.stderr:2024-04-06T09:56:43.835+0000 7f78af6df640 -1 reset not still connected to 0x5604242c76c0 2024-04-06T09:56:43.834 INFO:tasks.ceph.cluster2.osd.0.smithi192.stderr:2024-04-06T09:56:43.835+0000 7f78af6df640 -1 reset not still connected to 0x560436199450 2024-04-06T09:56:46.651 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:200 IOs, 48586752 bytes 2024-04-06T09:56:46.950 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ error_code=124 2024-04-06T09:56:46.950 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set -e 2024-04-06T09:56:46.950 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 124 -eq 124 ']' 2024-04-06T09:56:46.950 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:56:46.950 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-06T09:56:46.950 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ image=image_41 2024-04-06T09:56:46.950 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ create_image_and_enable_mirror cluster2 mirror image_41 journal 128M 2024-04-06T09:56:46.950 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-04-06T09:56:46.950 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:56:46.950 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:56:46.951 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:56:46.951 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_41 2024-04-06T09:56:46.951 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:56:46.951 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local mode=journal 2024-04-06T09:56:46.951 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n journal ']' 2024-04-06T09:56:46.951 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:56:46.951 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ create_image cluster2 mirror image_41 128M 2024-04-06T09:56:46.951 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-04-06T09:56:46.951 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:56:46.951 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:56:46.951 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:56:46.951 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_41 2024-04-06T09:56:46.951 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:56:46.951 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local size=128 2024-04-06T09:56:46.951 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n 128M ']' 2024-04-06T09:56:46.951 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ size=128M 2024-04-06T09:56:46.952 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:56:46.952 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,journaling mirror/image_41 2024-04-06T09:56:47.104 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' pool = image ']' 2024-04-06T09:56:47.104 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' mirror = mirror_parent ']' 2024-04-06T09:56:47.104 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n '' ']' 2024-04-06T09:56:47.104 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ stress_write_image cluster2 mirror image_41 2024-04-06T09:56:47.104 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-04-06T09:56:47.104 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:56:47.104 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_41 2024-04-06T09:56:47.104 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-04-06T09:56:47.105 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local duration=14 2024-04-06T09:56:47.105 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set +e 2024-04-06T09:56:47.105 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ timeout 14s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_41 --debug-rbd=20 --debug-journaler=20 2024-04-06T09:56:48.909 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:56:48.908+0000 7f2e84d66640 -1 reset not still connected to 0x560d2722e5b0 2024-04-06T09:56:48.909 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:56:48.908+0000 7f2e84d66640 -1 reset not still connected to 0x560d27566ea0 2024-04-06T09:56:48.909 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:56:48.908+0000 7f2e84d66640 -1 reset not still connected to 0x560d27567380 2024-04-06T09:56:48.909 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:56:48.908+0000 7f2e84d66640 -1 reset not still connected to 0x560d27839380 2024-04-06T09:56:48.910 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:56:48.908+0000 7f2e84d66640 -1 reset not still connected to 0x560d27839930 2024-04-06T09:56:48.915 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:56:48.912+0000 7f2e84d66640 -1 reset not still connected to 0x560d27c3edd0 2024-04-06T09:56:48.916 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:56:48.912+0000 7f2e84d66640 -1 reset not still connected to 0x560d27c3f2b0 2024-04-06T09:56:48.916 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:56:48.916+0000 7f2e84d66640 -1 reset not still connected to 0x560d281875f0 2024-04-06T09:56:48.916 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:56:48.916+0000 7f2e84d66640 -1 reset not still connected to 0x560d294908f0 2024-04-06T09:56:48.916 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:56:48.916+0000 7f2e84d66640 -1 reset not still connected to 0x560d29cd8f70 2024-04-06T09:56:48.916 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:56:48.916+0000 7f2e84d66640 -1 reset not still connected to 0x560d29cd9ba0 2024-04-06T09:56:48.918 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:56:48.916+0000 7f2e84d66640 -1 reset not still connected to 0x560d29d5a340 2024-04-06T09:56:48.918 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:56:48.916+0000 7f2e84d66640 -1 reset not still connected to 0x560d29d5b520 2024-04-06T09:56:48.918 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:56:48.916+0000 7f2e84d66640 -1 reset not still connected to 0x560d2a17d450 2024-04-06T09:56:48.918 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:56:48.916+0000 7f2e84d66640 -1 reset not still connected to 0x560d2a17d6c0 2024-04-06T09:56:48.919 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:56:48.916+0000 7f2e84d66640 -1 reset not still connected to 0x560d2a204b60 2024-04-06T09:56:48.919 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:56:48.916+0000 7f2e84d66640 -1 reset not still connected to 0x560d2a205ba0 2024-04-06T09:56:48.919 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:56:48.916+0000 7f2e84d66640 -1 reset not still connected to 0x560d2a2336c0 2024-04-06T09:56:48.919 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:56:48.916+0000 7f2e84d66640 -1 reset not still connected to 0x560d2a5705b0 2024-04-06T09:56:48.920 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:56:48.916+0000 7f2e84d66640 -1 reset not still connected to 0x560d3620b520 2024-04-06T09:56:48.920 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:56:48.916+0000 7f2e84d66640 -1 reset not still connected to 0x560d368e4f70 2024-04-06T09:56:48.920 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:56:48.916+0000 7f2e84d66640 -1 reset not still connected to 0x560d36912ea0 2024-04-06T09:56:48.920 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:56:48.916+0000 7f2e84d66640 -1 reset not still connected to 0x560d381dd5f0 2024-04-06T09:56:48.920 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:56:48.916+0000 7f2e84d66640 -1 reset not still connected to 0x560d390dca90 2024-04-06T09:56:48.924 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:56:48.924+0000 7f2e84d66640 -1 reset not still connected to 0x560d4bc7e8f0 2024-04-06T09:56:48.924 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:56:48.924+0000 7f2e84d66640 -1 reset not still connected to 0x560d50adba00 2024-04-06T09:56:50.818 INFO:tasks.ceph.cluster2.osd.1.smithi192.stderr:2024-04-06T09:56:50.818+0000 7f0de0719640 -1 reset not still connected to 0x55bfa4b36340 2024-04-06T09:56:50.824 INFO:tasks.ceph.cluster2.osd.1.smithi192.stderr:2024-04-06T09:56:50.826+0000 7f0de0719640 -1 reset not still connected to 0x55bfa5642b60 2024-04-06T09:56:50.824 INFO:tasks.ceph.cluster2.osd.1.smithi192.stderr:2024-04-06T09:56:50.826+0000 7f0de0719640 -1 reset not still connected to 0x55bfa6e48410 2024-04-06T09:56:50.825 INFO:tasks.ceph.cluster2.osd.1.smithi192.stderr:2024-04-06T09:56:50.826+0000 7f0de0719640 -1 reset not still connected to 0x55bfa8bcf6c0 2024-04-06T09:56:50.825 INFO:tasks.ceph.cluster2.osd.1.smithi192.stderr:2024-04-06T09:56:50.826+0000 7f0de0719640 -1 reset not still connected to 0x55bfaef3ba00 2024-04-06T09:56:50.825 INFO:tasks.ceph.cluster2.osd.1.smithi192.stderr:2024-04-06T09:56:50.826+0000 7f0de0719640 -1 reset not still connected to 0x55bfb063c680 2024-04-06T09:56:50.826 INFO:tasks.ceph.cluster2.osd.1.smithi192.stderr:2024-04-06T09:56:50.826+0000 7f0de0719640 -1 reset not still connected to 0x55bfbd8f9d40 2024-04-06T09:56:50.826 INFO:tasks.ceph.cluster2.osd.1.smithi192.stderr:2024-04-06T09:56:50.826+0000 7f0de0719640 -1 reset not still connected to 0x55bfd3db9d40 2024-04-06T09:56:50.826 INFO:tasks.ceph.cluster2.osd.1.smithi192.stderr:2024-04-06T09:56:50.826+0000 7f0de0719640 -1 reset not still connected to 0x55bfd74fb450 2024-04-06T09:56:50.892 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-04-06T09:56:50.884+0000 7fd1b6210640 -1 reset not still connected to 0x55ef99b89ad0 2024-04-06T09:56:50.899 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-04-06T09:56:50.884+0000 7fd1b6210640 -1 reset not still connected to 0x55ef9ac60270 2024-04-06T09:56:50.900 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-04-06T09:56:50.892+0000 7fd1b6210640 -1 reset not still connected to 0x55ef9bbe84e0 2024-04-06T09:56:50.900 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-04-06T09:56:50.892+0000 7fd1b6210640 -1 reset not still connected to 0x55ef9bbe9ee0 2024-04-06T09:56:50.901 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-04-06T09:56:50.892+0000 7fd1b6210640 -1 reset not still connected to 0x55ef9bd5c9c0 2024-04-06T09:56:50.901 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-04-06T09:56:50.892+0000 7fd1b6210640 -1 reset not still connected to 0x55ef9bd5d520 2024-04-06T09:56:50.901 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-04-06T09:56:50.892+0000 7fd1b6210640 -1 reset not still connected to 0x55ef9bd5dd40 2024-04-06T09:56:50.901 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-04-06T09:56:50.900+0000 7fd1b6210640 -1 reset not still connected to 0x55ef9c103110 2024-04-06T09:56:50.901 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-04-06T09:56:50.900+0000 7fd1b6210640 -1 reset not still connected to 0x55ef9c103930 2024-04-06T09:56:50.901 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-04-06T09:56:50.900+0000 7fd1b6210640 -1 reset not still connected to 0x55ef9c103d40 2024-04-06T09:56:50.901 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-04-06T09:56:50.900+0000 7fd1b6210640 -1 reset not still connected to 0x55ef9e698c30 2024-04-06T09:56:50.901 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-04-06T09:56:50.900+0000 7fd1b6210640 -1 reset not still connected to 0x55ef9e88f520 2024-04-06T09:56:50.907 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-04-06T09:56:50.908+0000 7fd1b6210640 -1 reset not still connected to 0x55ef9eee21a0 2024-04-06T09:56:50.912 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-04-06T09:56:50.912+0000 7fd1b6210640 -1 reset not still connected to 0x55ef9eee3a00 2024-04-06T09:56:50.912 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-04-06T09:56:50.912+0000 7fd1b6210640 -1 reset not still connected to 0x55efa18cc270 2024-04-06T09:56:50.913 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-04-06T09:56:50.912+0000 7fd1b6210640 -1 reset not still connected to 0x55efa18cc9c0 2024-04-06T09:56:50.913 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-04-06T09:56:50.912+0000 7fd1b6210640 -1 reset not still connected to 0x55efa18cca90 2024-04-06T09:56:50.913 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-04-06T09:56:50.912+0000 7fd1b6210640 -1 reset not still connected to 0x55efbb630d00 2024-04-06T09:56:50.913 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-04-06T09:56:50.912+0000 7fd1b6210640 -1 reset not still connected to 0x55efd81ee8f0 2024-04-06T09:56:50.913 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-04-06T09:56:50.912+0000 7fd1b6210640 -1 reset not still connected to 0x55efe06a9860 2024-04-06T09:56:51.770 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:100 IOs, 52269056 bytes 2024-04-06T09:56:52.597 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:56:52.596+0000 7f2e84d66640 -1 reset not still connected to 0x560d381dd5f0 2024-04-06T09:56:52.607 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:56:52.608+0000 7f2e84d66640 -1 reset not still connected to 0x560d50adba00 2024-04-06T09:56:53.026 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:56:53.016+0000 7f2e84d66640 -1 reset not still connected to 0x560d29d5b520 2024-04-06T09:56:53.026 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:56:53.016+0000 7f2e84d66640 -1 reset not still connected to 0x560d2a17d450 2024-04-06T09:56:53.027 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:56:53.024+0000 7f2e84d66640 -1 reset not still connected to 0x560d2a17d6c0 2024-04-06T09:56:53.027 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:56:53.024+0000 7f2e84d66640 -1 reset not still connected to 0x560d2a205ba0 2024-04-06T09:56:53.027 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:56:53.024+0000 7f2e84d66640 -1 reset not still connected to 0x560d2a2336c0 2024-04-06T09:56:53.027 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:56:53.024+0000 7f2e84d66640 -1 reset not still connected to 0x560d2a5705b0 2024-04-06T09:56:53.027 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:56:53.024+0000 7f2e84d66640 -1 reset not still connected to 0x560d368e4f70 2024-04-06T09:56:53.028 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:56:53.028+0000 7f2e84d66640 -1 reset not still connected to 0x560d381dd5f0 2024-04-06T09:56:53.028 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:56:53.028+0000 7f2e84d66640 -1 reset not still connected to 0x560d50adba00 2024-04-06T09:56:54.247 INFO:tasks.ceph.cluster2.osd.0.smithi192.stderr:2024-04-06T09:56:54.246+0000 7f78af6df640 -1 reset not still connected to 0x56044cc1e1a0 2024-04-06T09:56:54.387 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:56:54.380+0000 7f0965192640 -1 reset not still connected to 0x5632a1b73ee0 2024-04-06T09:56:54.388 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:56:54.388+0000 7f0965192640 -1 reset not still connected to 0x5632a2793520 2024-04-06T09:56:54.388 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:56:54.388+0000 7f0965192640 -1 reset not still connected to 0x5632a30f0270 2024-04-06T09:56:54.389 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:56:54.388+0000 7f0965192640 -1 reset not still connected to 0x5632a30f0f70 2024-04-06T09:56:54.389 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:56:54.388+0000 7f0965192640 -1 reset not still connected to 0x5632a417e750 2024-04-06T09:56:54.389 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:56:54.388+0000 7f0965192640 -1 reset not still connected to 0x5632a4cda000 2024-04-06T09:56:54.398 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:56:54.396+0000 7f0965192640 -1 reset not still connected to 0x5632a4cda9c0 2024-04-06T09:56:54.399 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:56:54.396+0000 7f0965192640 -1 reset not still connected to 0x5632a4cdbba0 2024-04-06T09:56:54.399 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:56:54.396+0000 7f0965192640 -1 reset not still connected to 0x5632a5e9dee0 2024-04-06T09:56:54.399 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:56:54.396+0000 7f0965192640 -1 reset not still connected to 0x5632a89f4750 2024-04-06T09:56:54.399 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:56:54.396+0000 7f0965192640 -1 reset not still connected to 0x5632a89f52b0 2024-04-06T09:56:54.399 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:56:54.396+0000 7f0965192640 -1 reset not still connected to 0x5632abc812b0 2024-04-06T09:56:54.399 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:56:54.396+0000 7f0965192640 -1 reset not still connected to 0x5632abc81d40 2024-04-06T09:56:54.399 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:56:54.396+0000 7f0965192640 -1 reset not still connected to 0x5632abcbe820 2024-04-06T09:56:54.399 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:56:54.396+0000 7f0965192640 -1 reset not still connected to 0x5632b81f3a00 2024-04-06T09:56:54.399 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:56:54.396+0000 7f0965192640 -1 reset not still connected to 0x5632b9592000 2024-04-06T09:56:54.399 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:56:54.396+0000 7f0965192640 -1 reset not still connected to 0x5632b9593110 2024-04-06T09:56:54.399 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:56:54.396+0000 7f0965192640 -1 reset not still connected to 0x5632c0cfa000 2024-04-06T09:56:54.399 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:56:54.396+0000 7f0965192640 -1 reset not still connected to 0x5632c0cfba00 2024-04-06T09:56:54.399 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:56:54.396+0000 7f0965192640 -1 reset not still connected to 0x5632cb3ac000 2024-04-06T09:56:54.399 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:56:54.396+0000 7f0965192640 -1 reset not still connected to 0x5632dac08dd0 2024-04-06T09:56:54.399 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:56:54.396+0000 7f0965192640 -1 reset not still connected to 0x5632dac09040 2024-04-06T09:56:54.400 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:56:54.396+0000 7f0965192640 -1 reset not still connected to 0x5632e1684ea0 2024-04-06T09:56:54.400 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:56:54.396+0000 7f0965192640 -1 reset not still connected to 0x5632e16851e0 2024-04-06T09:56:56.565 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:56:56.564+0000 7f0965192640 -1 reset not still connected to 0x5632a0ff5110 2024-04-06T09:56:56.570 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:56:56.564+0000 7f0965192640 -1 reset not still connected to 0x5632a14ac820 2024-04-06T09:56:56.570 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:56:56.564+0000 7f0965192640 -1 reset not still connected to 0x5632a1988000 2024-04-06T09:56:56.570 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:56:56.564+0000 7f0965192640 -1 reset not still connected to 0x5632a1989040 2024-04-06T09:56:56.570 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:56:56.564+0000 7f0965192640 -1 reset not still connected to 0x5632a1b73ee0 2024-04-06T09:56:56.570 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:56:56.564+0000 7f0965192640 -1 reset not still connected to 0x5632a2793520 2024-04-06T09:56:56.571 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:56:56.564+0000 7f0965192640 -1 reset not still connected to 0x5632a30f0270 2024-04-06T09:56:56.571 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:56:56.568+0000 7f0965192640 -1 reset not still connected to 0x5632a30f0f70 2024-04-06T09:56:56.571 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:56:56.568+0000 7f0965192640 -1 reset not still connected to 0x5632a417e750 2024-04-06T09:56:56.571 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:56:56.568+0000 7f0965192640 -1 reset not still connected to 0x5632a4cda000 2024-04-06T09:56:56.571 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:56:56.568+0000 7f0965192640 -1 reset not still connected to 0x5632a4cda9c0 2024-04-06T09:56:56.571 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:56:56.568+0000 7f0965192640 -1 reset not still connected to 0x5632a4cdbba0 2024-04-06T09:56:56.571 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:56:56.568+0000 7f0965192640 -1 reset not still connected to 0x5632a5e9dee0 2024-04-06T09:56:56.571 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:56:56.568+0000 7f0965192640 -1 reset not still connected to 0x5632a89f4750 2024-04-06T09:56:56.571 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:56:56.568+0000 7f0965192640 -1 reset not still connected to 0x5632a89f52b0 2024-04-06T09:56:56.571 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:56:56.568+0000 7f0965192640 -1 reset not still connected to 0x5632abc812b0 2024-04-06T09:56:56.571 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:56:56.568+0000 7f0965192640 -1 reset not still connected to 0x5632abc81d40 2024-04-06T09:56:56.576 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:56:56.572+0000 7f0965192640 -1 reset not still connected to 0x5632abcbe820 2024-04-06T09:56:56.576 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:56:56.572+0000 7f0965192640 -1 reset not still connected to 0x5632b81f3a00 2024-04-06T09:56:56.576 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:56:56.572+0000 7f0965192640 -1 reset not still connected to 0x5632b9592000 2024-04-06T09:56:56.576 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:56:56.572+0000 7f0965192640 -1 reset not still connected to 0x5632b9593110 2024-04-06T09:56:56.577 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:56:56.576+0000 7f0965192640 -1 reset not still connected to 0x5632c0cfa000 2024-04-06T09:56:56.577 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:56:56.576+0000 7f0965192640 -1 reset not still connected to 0x5632c0cfba00 2024-04-06T09:56:56.577 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:56:56.576+0000 7f0965192640 -1 reset not still connected to 0x5632cb3ac000 2024-04-06T09:56:56.577 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:56:56.576+0000 7f0965192640 -1 reset not still connected to 0x5632dac08dd0 2024-04-06T09:56:56.577 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:56:56.576+0000 7f0965192640 -1 reset not still connected to 0x5632dac09040 2024-04-06T09:56:56.577 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:56:56.576+0000 7f0965192640 -1 reset not still connected to 0x5632e1684ea0 2024-04-06T09:56:56.577 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:56:56.576+0000 7f0965192640 -1 reset not still connected to 0x5632e16851e0 2024-04-06T09:56:57.246 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:200 IOs, 143811584 bytes 2024-04-06T09:56:58.322 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-04-06T09:56:58.320+0000 7fd1b6210640 -1 reset not still connected to 0x55efd81ee8f0 2024-04-06T09:56:59.654 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-04-06T09:56:59.652+0000 7fd1b6210640 -1 reset not still connected to 0x55ef9c103d40 2024-04-06T09:56:59.654 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-04-06T09:56:59.652+0000 7fd1b6210640 -1 reset not still connected to 0x55ef9e698c30 2024-04-06T09:56:59.659 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-04-06T09:56:59.660+0000 7fd1b6210640 -1 reset not still connected to 0x55ef9e88f520 2024-04-06T09:56:59.659 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-04-06T09:56:59.660+0000 7fd1b6210640 -1 reset not still connected to 0x55ef9eee21a0 2024-04-06T09:56:59.659 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-04-06T09:56:59.660+0000 7fd1b6210640 -1 reset not still connected to 0x55efa18cc270 2024-04-06T09:56:59.659 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-04-06T09:56:59.660+0000 7fd1b6210640 -1 reset not still connected to 0x55efa18cc9c0 2024-04-06T09:56:59.664 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-04-06T09:56:59.664+0000 7fd1b6210640 -1 reset not still connected to 0x55efd81ee8f0 2024-04-06T09:57:00.755 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:300 IOs, 186981376 bytes 2024-04-06T09:57:01.118 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ error_code=124 2024-04-06T09:57:01.118 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set -e 2024-04-06T09:57:01.118 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 124 -eq 124 ']' 2024-04-06T09:57:01.118 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:57:01.118 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-06T09:57:01.118 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ image=image_42 2024-04-06T09:57:01.119 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ create_image_and_enable_mirror cluster2 mirror image_42 journal 128M 2024-04-06T09:57:01.119 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-04-06T09:57:01.119 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:57:01.119 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:57:01.119 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:57:01.119 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_42 2024-04-06T09:57:01.119 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:57:01.119 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local mode=journal 2024-04-06T09:57:01.119 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n journal ']' 2024-04-06T09:57:01.119 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:57:01.119 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ create_image cluster2 mirror image_42 128M 2024-04-06T09:57:01.119 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-04-06T09:57:01.119 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:57:01.119 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:57:01.120 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:57:01.120 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_42 2024-04-06T09:57:01.120 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:57:01.120 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local size=128 2024-04-06T09:57:01.120 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n 128M ']' 2024-04-06T09:57:01.120 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ size=128M 2024-04-06T09:57:01.120 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:57:01.120 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,journaling mirror/image_42 2024-04-06T09:57:01.186 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' pool = image ']' 2024-04-06T09:57:01.186 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' mirror = mirror_parent ']' 2024-04-06T09:57:01.187 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n '' ']' 2024-04-06T09:57:01.187 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ stress_write_image cluster2 mirror image_42 2024-04-06T09:57:01.187 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-04-06T09:57:01.187 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:57:01.187 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_42 2024-04-06T09:57:01.187 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-04-06T09:57:01.188 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local duration=9 2024-04-06T09:57:01.188 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set +e 2024-04-06T09:57:01.188 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ timeout 9s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_42 --debug-rbd=20 --debug-journaler=20 2024-04-06T09:57:01.495 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:100 IOs, 1314816 bytes 2024-04-06T09:57:01.651 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-04-06T09:57:01.648+0000 7fd1b6210640 -1 reset not still connected to 0x55ef99b88000 2024-04-06T09:57:01.652 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-04-06T09:57:01.648+0000 7fd1b6210640 -1 reset not still connected to 0x55ef99b88d00 2024-04-06T09:57:01.652 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-04-06T09:57:01.648+0000 7fd1b6210640 -1 reset not still connected to 0x55ef99b89ad0 2024-04-06T09:57:01.652 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-04-06T09:57:01.648+0000 7fd1b6210640 -1 reset not still connected to 0x55ef99ebf6c0 2024-04-06T09:57:01.652 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-04-06T09:57:01.648+0000 7fd1b6210640 -1 reset not still connected to 0x55ef9ac60270 2024-04-06T09:57:01.652 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-04-06T09:57:01.652+0000 7fd1b6210640 -1 reset not still connected to 0x55ef9bbe9ee0 2024-04-06T09:57:01.652 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-04-06T09:57:01.652+0000 7fd1b6210640 -1 reset not still connected to 0x55ef9bd5c9c0 2024-04-06T09:57:01.666 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-04-06T09:57:01.664+0000 7fd1b6210640 -1 reset not still connected to 0x55ef9c103930 2024-04-06T09:57:01.666 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-04-06T09:57:01.664+0000 7fd1b6210640 -1 reset not still connected to 0x55ef9c103d40 2024-04-06T09:57:01.666 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-04-06T09:57:01.664+0000 7fd1b6210640 -1 reset not still connected to 0x55ef9e698c30 2024-04-06T09:57:01.666 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-04-06T09:57:01.664+0000 7fd1b6210640 -1 reset not still connected to 0x55ef9e88f520 2024-04-06T09:57:01.666 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-04-06T09:57:01.664+0000 7fd1b6210640 -1 reset not still connected to 0x55ef9eee21a0 2024-04-06T09:57:01.669 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-04-06T09:57:01.668+0000 7fd1b6210640 -1 reset not still connected to 0x55ef9eee3a00 2024-04-06T09:57:01.669 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-04-06T09:57:01.668+0000 7fd1b6210640 -1 reset not still connected to 0x55efa18cc270 2024-04-06T09:57:01.669 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-04-06T09:57:01.668+0000 7fd1b6210640 -1 reset not still connected to 0x55efa18cc9c0 2024-04-06T09:57:01.669 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-04-06T09:57:01.668+0000 7fd1b6210640 -1 reset not still connected to 0x55efa18cca90 2024-04-06T09:57:01.669 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-04-06T09:57:01.668+0000 7fd1b6210640 -1 reset not still connected to 0x55efbb630d00 2024-04-06T09:57:01.669 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-04-06T09:57:01.668+0000 7fd1b6210640 -1 reset not still connected to 0x55efd81ee8f0 2024-04-06T09:57:01.669 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-04-06T09:57:01.668+0000 7fd1b6210640 -1 reset not still connected to 0x55efe06a9860 2024-04-06T09:57:01.731 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-04-06T09:57:01.728+0000 7fd1b6210640 -1 reset not still connected to 0x55ef9bbe84e0 2024-04-06T09:57:01.735 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-04-06T09:57:01.732+0000 7fd1b6210640 -1 reset not still connected to 0x55ef9bbe9ee0 2024-04-06T09:57:01.735 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-04-06T09:57:01.732+0000 7fd1b6210640 -1 reset not still connected to 0x55ef9bd5c9c0 2024-04-06T09:57:01.735 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-04-06T09:57:01.732+0000 7fd1b6210640 -1 reset not still connected to 0x55ef9bd5d520 2024-04-06T09:57:01.735 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-04-06T09:57:01.732+0000 7fd1b6210640 -1 reset not still connected to 0x55ef9bd5dd40 2024-04-06T09:57:01.737 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-04-06T09:57:01.736+0000 7fd1b6210640 -1 reset not still connected to 0x55ef9bd86820 2024-04-06T09:57:01.737 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-04-06T09:57:01.736+0000 7fd1b6210640 -1 reset not still connected to 0x55ef9c103110 2024-04-06T09:57:01.738 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-04-06T09:57:01.736+0000 7fd1b6210640 -1 reset not still connected to 0x55ef9c103930 2024-04-06T09:57:01.738 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-04-06T09:57:01.736+0000 7fd1b6210640 -1 reset not still connected to 0x55ef9c103d40 2024-04-06T09:57:01.738 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-04-06T09:57:01.736+0000 7fd1b6210640 -1 reset not still connected to 0x55ef9e698c30 2024-04-06T09:57:01.738 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-04-06T09:57:01.736+0000 7fd1b6210640 -1 reset not still connected to 0x55ef9e88f520 2024-04-06T09:57:01.738 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-04-06T09:57:01.736+0000 7fd1b6210640 -1 reset not still connected to 0x55ef9eee21a0 2024-04-06T09:57:01.738 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-04-06T09:57:01.736+0000 7fd1b6210640 -1 reset not still connected to 0x55ef9eee3a00 2024-04-06T09:57:01.738 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-04-06T09:57:01.736+0000 7fd1b6210640 -1 reset not still connected to 0x55efa18cc270 2024-04-06T09:57:01.738 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-04-06T09:57:01.736+0000 7fd1b6210640 -1 reset not still connected to 0x55efa18cc9c0 2024-04-06T09:57:01.738 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-04-06T09:57:01.736+0000 7fd1b6210640 -1 reset not still connected to 0x55efa18cca90 2024-04-06T09:57:01.738 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-04-06T09:57:01.736+0000 7fd1b6210640 -1 reset not still connected to 0x55efbb630d00 2024-04-06T09:57:01.738 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-04-06T09:57:01.736+0000 7fd1b6210640 -1 reset not still connected to 0x55efd81ee8f0 2024-04-06T09:57:01.738 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-04-06T09:57:01.736+0000 7fd1b6210640 -1 reset not still connected to 0x55efe06a9860 2024-04-06T09:57:03.792 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:57:03.792+0000 7f2e84d66640 -1 reset not still connected to 0x560d27073860 2024-04-06T09:57:03.792 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:57:03.792+0000 7f2e84d66640 -1 reset not still connected to 0x560d2722e5b0 2024-04-06T09:57:03.792 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:57:03.792+0000 7f2e84d66640 -1 reset not still connected to 0x560d27566ea0 2024-04-06T09:57:03.792 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:57:03.792+0000 7f2e84d66640 -1 reset not still connected to 0x560d27567380 2024-04-06T09:57:03.792 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:57:03.792+0000 7f2e84d66640 -1 reset not still connected to 0x560d27839380 2024-04-06T09:57:03.806 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:57:03.796+0000 7f2e84d66640 -1 reset not still connected to 0x560d27839930 2024-04-06T09:57:03.807 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:57:03.796+0000 7f2e84d66640 -1 reset not still connected to 0x560d27c3edd0 2024-04-06T09:57:03.807 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:57:03.796+0000 7f2e84d66640 -1 reset not still connected to 0x560d27c3f2b0 2024-04-06T09:57:03.807 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:57:03.796+0000 7f2e84d66640 -1 reset not still connected to 0x560d281875f0 2024-04-06T09:57:03.807 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:57:03.796+0000 7f2e84d66640 -1 reset not still connected to 0x560d294908f0 2024-04-06T09:57:03.807 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:57:03.796+0000 7f2e84d66640 -1 reset not still connected to 0x560d29cd8f70 2024-04-06T09:57:03.807 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:57:03.796+0000 7f2e84d66640 -1 reset not still connected to 0x560d29cd9ba0 2024-04-06T09:57:03.807 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:57:03.796+0000 7f2e84d66640 -1 reset not still connected to 0x560d29d5a340 2024-04-06T09:57:03.808 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:57:03.796+0000 7f2e84d66640 -1 reset not still connected to 0x560d29d5b520 2024-04-06T09:57:03.808 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:57:03.796+0000 7f2e84d66640 -1 reset not still connected to 0x560d2a17d450 2024-04-06T09:57:03.808 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:57:03.796+0000 7f2e84d66640 -1 reset not still connected to 0x560d2a17d6c0 2024-04-06T09:57:03.808 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:57:03.796+0000 7f2e84d66640 -1 reset not still connected to 0x560d2a204b60 2024-04-06T09:57:03.808 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:57:03.796+0000 7f2e84d66640 -1 reset not still connected to 0x560d2a205ba0 2024-04-06T09:57:03.808 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:57:03.796+0000 7f2e84d66640 -1 reset not still connected to 0x560d2a2336c0 2024-04-06T09:57:03.808 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:57:03.796+0000 7f2e84d66640 -1 reset not still connected to 0x560d2a570000 2024-04-06T09:57:03.808 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:57:03.796+0000 7f2e84d66640 -1 reset not still connected to 0x560d2a5705b0 2024-04-06T09:57:03.808 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:57:03.796+0000 7f2e84d66640 -1 reset not still connected to 0x560d35349380 2024-04-06T09:57:03.808 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:57:03.796+0000 7f2e84d66640 -1 reset not still connected to 0x560d3620b520 2024-04-06T09:57:03.808 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:57:03.796+0000 7f2e84d66640 -1 reset not still connected to 0x560d368e4f70 2024-04-06T09:57:03.808 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:57:03.796+0000 7f2e84d66640 -1 reset not still connected to 0x560d36912ea0 2024-04-06T09:57:03.809 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:57:03.800+0000 7f2e84d66640 -1 reset not still connected to 0x560d381dcc30 2024-04-06T09:57:03.809 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:57:03.800+0000 7f2e84d66640 -1 reset not still connected to 0x560d381dd5f0 2024-04-06T09:57:03.809 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:57:03.800+0000 7f2e84d66640 -1 reset not still connected to 0x560d390dca90 2024-04-06T09:57:03.809 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:57:03.800+0000 7f2e84d66640 -1 reset not still connected to 0x560d4bc7e8f0 2024-04-06T09:57:03.809 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:57:03.800+0000 7f2e84d66640 -1 reset not still connected to 0x560d50adba00 2024-04-06T09:57:04.517 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-04-06T09:57:04.516+0000 7fd1b6210640 -1 reset not still connected to 0x55ef997761a0 2024-04-06T09:57:04.521 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-04-06T09:57:04.516+0000 7fd1b6210640 -1 reset not still connected to 0x55ef99776680 2024-04-06T09:57:04.521 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-04-06T09:57:04.516+0000 7fd1b6210640 -1 reset not still connected to 0x55ef99b88000 2024-04-06T09:57:04.521 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-04-06T09:57:04.516+0000 7fd1b6210640 -1 reset not still connected to 0x55ef99b88d00 2024-04-06T09:57:04.521 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-04-06T09:57:04.516+0000 7fd1b6210640 -1 reset not still connected to 0x55ef99b89ad0 2024-04-06T09:57:04.522 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-04-06T09:57:04.516+0000 7fd1b6210640 -1 reset not still connected to 0x55ef99ebf6c0 2024-04-06T09:57:04.522 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-04-06T09:57:04.516+0000 7fd1b6210640 -1 reset not still connected to 0x55ef9ac60270 2024-04-06T09:57:04.526 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-04-06T09:57:04.524+0000 7fd1b6210640 -1 reset not still connected to 0x55ef9b657e10 2024-04-06T09:57:04.526 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-04-06T09:57:04.524+0000 7fd1b6210640 -1 reset not still connected to 0x55ef9bbe84e0 2024-04-06T09:57:04.526 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-04-06T09:57:04.524+0000 7fd1b6210640 -1 reset not still connected to 0x55ef9bbe9ee0 2024-04-06T09:57:04.526 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-04-06T09:57:04.524+0000 7fd1b6210640 -1 reset not still connected to 0x55ef9bd5c9c0 2024-04-06T09:57:04.526 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-04-06T09:57:04.524+0000 7fd1b6210640 -1 reset not still connected to 0x55ef9bd5d520 2024-04-06T09:57:04.526 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-04-06T09:57:04.524+0000 7fd1b6210640 -1 reset not still connected to 0x55ef9bd5dd40 2024-04-06T09:57:04.526 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-04-06T09:57:04.524+0000 7fd1b6210640 -1 reset not still connected to 0x55ef9bd86820 2024-04-06T09:57:04.526 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-04-06T09:57:04.524+0000 7fd1b6210640 -1 reset not still connected to 0x55ef9c103110 2024-04-06T09:57:04.526 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-04-06T09:57:04.524+0000 7fd1b6210640 -1 reset not still connected to 0x55ef9c103930 2024-04-06T09:57:04.526 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-04-06T09:57:04.524+0000 7fd1b6210640 -1 reset not still connected to 0x55ef9c103d40 2024-04-06T09:57:04.526 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-04-06T09:57:04.524+0000 7fd1b6210640 -1 reset not still connected to 0x55ef9e698c30 2024-04-06T09:57:04.530 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-04-06T09:57:04.528+0000 7fd1b6210640 -1 reset not still connected to 0x55ef9e88f520 2024-04-06T09:57:04.531 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-04-06T09:57:04.528+0000 7fd1b6210640 -1 reset not still connected to 0x55ef9eee21a0 2024-04-06T09:57:04.531 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-04-06T09:57:04.528+0000 7fd1b6210640 -1 reset not still connected to 0x55ef9eee3a00 2024-04-06T09:57:04.531 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-04-06T09:57:04.528+0000 7fd1b6210640 -1 reset not still connected to 0x55efa18cc270 2024-04-06T09:57:04.531 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-04-06T09:57:04.528+0000 7fd1b6210640 -1 reset not still connected to 0x55efa18cc9c0 2024-04-06T09:57:04.531 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-04-06T09:57:04.528+0000 7fd1b6210640 -1 reset not still connected to 0x55efa18cca90 2024-04-06T09:57:04.532 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-04-06T09:57:04.528+0000 7fd1b6210640 -1 reset not still connected to 0x55efbb630d00 2024-04-06T09:57:04.532 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-04-06T09:57:04.528+0000 7fd1b6210640 -1 reset not still connected to 0x55efd81ee8f0 2024-04-06T09:57:04.532 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-04-06T09:57:04.528+0000 7fd1b6210640 -1 reset not still connected to 0x55efe06a9860 2024-04-06T09:57:04.843 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:200 IOs, 46173184 bytes 2024-04-06T09:57:05.098 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-04-06T09:57:05.096+0000 7fd1b6210640 -1 reset not still connected to 0x55ef99b88d00 2024-04-06T09:57:05.098 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-04-06T09:57:05.096+0000 7fd1b6210640 -1 reset not still connected to 0x55ef99b89ad0 2024-04-06T09:57:05.098 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-04-06T09:57:05.096+0000 7fd1b6210640 -1 reset not still connected to 0x55ef99ebf6c0 2024-04-06T09:57:05.098 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-04-06T09:57:05.096+0000 7fd1b6210640 -1 reset not still connected to 0x55ef9ac60270 2024-04-06T09:57:05.098 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-04-06T09:57:05.096+0000 7fd1b6210640 -1 reset not still connected to 0x55ef9b657e10 2024-04-06T09:57:05.098 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-04-06T09:57:05.096+0000 7fd1b6210640 -1 reset not still connected to 0x55ef9bbe84e0 2024-04-06T09:57:05.100 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-04-06T09:57:05.096+0000 7fd1b6210640 -1 reset not still connected to 0x55ef9bbe8d00 2024-04-06T09:57:05.105 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-04-06T09:57:05.100+0000 7fd1b6210640 -1 reset not still connected to 0x55ef9bbe9ee0 2024-04-06T09:57:05.105 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-04-06T09:57:05.100+0000 7fd1b6210640 -1 reset not still connected to 0x55ef9bd5c9c0 2024-04-06T09:57:05.105 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-04-06T09:57:05.100+0000 7fd1b6210640 -1 reset not still connected to 0x55ef9bd5d520 2024-04-06T09:57:05.105 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-04-06T09:57:05.100+0000 7fd1b6210640 -1 reset not still connected to 0x55ef9bd5dd40 2024-04-06T09:57:05.106 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-04-06T09:57:05.100+0000 7fd1b6210640 -1 reset not still connected to 0x55ef9bd86820 2024-04-06T09:57:05.106 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-04-06T09:57:05.100+0000 7fd1b6210640 -1 reset not still connected to 0x55ef9c103110 2024-04-06T09:57:05.107 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-04-06T09:57:05.104+0000 7fd1b6210640 -1 reset not still connected to 0x55ef9c103930 2024-04-06T09:57:05.107 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-04-06T09:57:05.104+0000 7fd1b6210640 -1 reset not still connected to 0x55ef9c103d40 2024-04-06T09:57:05.107 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-04-06T09:57:05.104+0000 7fd1b6210640 -1 reset not still connected to 0x55ef9e698c30 2024-04-06T09:57:05.107 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-04-06T09:57:05.104+0000 7fd1b6210640 -1 reset not still connected to 0x55ef9e88f520 2024-04-06T09:57:05.107 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-04-06T09:57:05.104+0000 7fd1b6210640 -1 reset not still connected to 0x55ef9eee21a0 2024-04-06T09:57:05.107 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-04-06T09:57:05.104+0000 7fd1b6210640 -1 reset not still connected to 0x55ef9eee3a00 2024-04-06T09:57:05.107 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-04-06T09:57:05.104+0000 7fd1b6210640 -1 reset not still connected to 0x55efa18cc270 2024-04-06T09:57:05.107 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-04-06T09:57:05.104+0000 7fd1b6210640 -1 reset not still connected to 0x55efa18cc9c0 2024-04-06T09:57:05.107 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-04-06T09:57:05.104+0000 7fd1b6210640 -1 reset not still connected to 0x55efa18cca90 2024-04-06T09:57:05.108 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-04-06T09:57:05.104+0000 7fd1b6210640 -1 reset not still connected to 0x55efbb630d00 2024-04-06T09:57:05.108 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-04-06T09:57:05.104+0000 7fd1b6210640 -1 reset not still connected to 0x55efd81ee8f0 2024-04-06T09:57:05.108 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-04-06T09:57:05.104+0000 7fd1b6210640 -1 reset not still connected to 0x55efe06a9860 2024-04-06T09:57:07.665 INFO:tasks.ceph.cluster2.osd.2.smithi192.stderr:2024-04-06T09:57:07.662+0000 7fa534aba640 -1 reset not still connected to 0x55934811f6c0 2024-04-06T09:57:07.666 INFO:tasks.ceph.cluster2.osd.2.smithi192.stderr:2024-04-06T09:57:07.662+0000 7fa534aba640 -1 reset not still connected to 0x559348d9ad00 2024-04-06T09:57:07.667 INFO:tasks.ceph.cluster2.osd.2.smithi192.stderr:2024-04-06T09:57:07.666+0000 7fa534aba640 -1 reset not still connected to 0x55934feb6000 2024-04-06T09:57:07.667 INFO:tasks.ceph.cluster2.osd.2.smithi192.stderr:2024-04-06T09:57:07.666+0000 7fa534aba640 -1 reset not still connected to 0x55935bfc9d40 2024-04-06T09:57:07.667 INFO:tasks.ceph.cluster2.osd.2.smithi192.stderr:2024-04-06T09:57:07.666+0000 7fa534aba640 -1 reset not still connected to 0x55936c6f44e0 2024-04-06T09:57:07.918 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:300 IOs, 96739328 bytes 2024-04-06T09:57:08.694 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:57:08.692+0000 7f0965192640 -1 reset not still connected to 0x5632a1988000 2024-04-06T09:57:08.702 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:57:08.700+0000 7f0965192640 -1 reset not still connected to 0x5632a1b73ee0 2024-04-06T09:57:08.702 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:57:08.700+0000 7f0965192640 -1 reset not still connected to 0x5632a2793520 2024-04-06T09:57:08.702 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:57:08.700+0000 7f0965192640 -1 reset not still connected to 0x5632a30f0f70 2024-04-06T09:57:08.702 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:57:08.700+0000 7f0965192640 -1 reset not still connected to 0x5632a417e750 2024-04-06T09:57:08.702 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:57:08.700+0000 7f0965192640 -1 reset not still connected to 0x5632a4cda000 2024-04-06T09:57:08.702 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:57:08.700+0000 7f0965192640 -1 reset not still connected to 0x5632a4cdbba0 2024-04-06T09:57:08.706 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:57:08.704+0000 7f0965192640 -1 reset not still connected to 0x5632a5e9dee0 2024-04-06T09:57:08.713 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:57:08.708+0000 7f0965192640 -1 reset not still connected to 0x5632a89f4750 2024-04-06T09:57:08.714 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:57:08.708+0000 7f0965192640 -1 reset not still connected to 0x5632a89f52b0 2024-04-06T09:57:08.714 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:57:08.708+0000 7f0965192640 -1 reset not still connected to 0x5632abc812b0 2024-04-06T09:57:08.714 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:57:08.708+0000 7f0965192640 -1 reset not still connected to 0x5632abc81d40 2024-04-06T09:57:08.714 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:57:08.708+0000 7f0965192640 -1 reset not still connected to 0x5632abcbe820 2024-04-06T09:57:08.714 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:57:08.708+0000 7f0965192640 -1 reset not still connected to 0x5632b81f3a00 2024-04-06T09:57:08.714 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:57:08.708+0000 7f0965192640 -1 reset not still connected to 0x5632b9592000 2024-04-06T09:57:08.714 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:57:08.708+0000 7f0965192640 -1 reset not still connected to 0x5632b9593110 2024-04-06T09:57:08.714 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:57:08.708+0000 7f0965192640 -1 reset not still connected to 0x5632c0cfa000 2024-04-06T09:57:08.714 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:57:08.708+0000 7f0965192640 -1 reset not still connected to 0x5632c0cfba00 2024-04-06T09:57:08.714 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:57:08.708+0000 7f0965192640 -1 reset not still connected to 0x5632cb3ac000 2024-04-06T09:57:08.714 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:57:08.708+0000 7f0965192640 -1 reset not still connected to 0x5632d8f88820 2024-04-06T09:57:08.714 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:57:08.708+0000 7f0965192640 -1 reset not still connected to 0x5632dac08dd0 2024-04-06T09:57:08.715 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:57:08.716+0000 7f0965192640 -1 reset not still connected to 0x5632dac09040 2024-04-06T09:57:08.715 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:57:08.716+0000 7f0965192640 -1 reset not still connected to 0x5632e1684ea0 2024-04-06T09:57:08.716 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:57:08.716+0000 7f0965192640 -1 reset not still connected to 0x5632e16851e0 2024-04-06T09:57:08.896 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:57:08.896+0000 7f0965192640 -1 reset not still connected to 0x56329f99c0d0 2024-04-06T09:57:08.909 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:57:08.904+0000 7f0965192640 -1 reset not still connected to 0x5632a08055f0 2024-04-06T09:57:08.909 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:57:08.904+0000 7f0965192640 -1 reset not still connected to 0x5632a0ff4410 2024-04-06T09:57:08.909 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:57:08.904+0000 7f0965192640 -1 reset not still connected to 0x5632a0ff5110 2024-04-06T09:57:08.909 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:57:08.904+0000 7f0965192640 -1 reset not still connected to 0x5632a14ac820 2024-04-06T09:57:08.910 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:57:08.904+0000 7f0965192640 -1 reset not still connected to 0x5632a1988000 2024-04-06T09:57:08.910 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:57:08.904+0000 7f0965192640 -1 reset not still connected to 0x5632a1989040 2024-04-06T09:57:08.910 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:57:08.904+0000 7f0965192640 -1 reset not still connected to 0x5632a1b73ee0 2024-04-06T09:57:08.910 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:57:08.904+0000 7f0965192640 -1 reset not still connected to 0x5632a2793520 2024-04-06T09:57:08.910 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:57:08.904+0000 7f0965192640 -1 reset not still connected to 0x5632a30f0270 2024-04-06T09:57:08.910 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:57:08.904+0000 7f0965192640 -1 reset not still connected to 0x5632a30f0f70 2024-04-06T09:57:08.910 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:57:08.904+0000 7f0965192640 -1 reset not still connected to 0x5632a417e750 2024-04-06T09:57:08.910 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:57:08.904+0000 7f0965192640 -1 reset not still connected to 0x5632a4cda000 2024-04-06T09:57:08.910 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:57:08.904+0000 7f0965192640 -1 reset not still connected to 0x5632a4cda9c0 2024-04-06T09:57:08.910 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:57:08.904+0000 7f0965192640 -1 reset not still connected to 0x5632a4cdbba0 2024-04-06T09:57:08.910 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:57:08.904+0000 7f0965192640 -1 reset not still connected to 0x5632a5e9dee0 2024-04-06T09:57:08.910 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:57:08.904+0000 7f0965192640 -1 reset not still connected to 0x5632a89f4750 2024-04-06T09:57:08.910 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:57:08.904+0000 7f0965192640 -1 reset not still connected to 0x5632a89f52b0 2024-04-06T09:57:08.911 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:57:08.904+0000 7f0965192640 -1 reset not still connected to 0x5632abc812b0 2024-04-06T09:57:08.911 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:57:08.904+0000 7f0965192640 -1 reset not still connected to 0x5632abc81d40 2024-04-06T09:57:08.911 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:57:08.908+0000 7f0965192640 -1 reset not still connected to 0x5632abcbe820 2024-04-06T09:57:08.911 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:57:08.908+0000 7f0965192640 -1 reset not still connected to 0x5632b81f3a00 2024-04-06T09:57:08.911 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:57:08.908+0000 7f0965192640 -1 reset not still connected to 0x5632b9592000 2024-04-06T09:57:08.911 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:57:08.908+0000 7f0965192640 -1 reset not still connected to 0x5632b9593110 2024-04-06T09:57:08.911 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:57:08.908+0000 7f0965192640 -1 reset not still connected to 0x5632c0cfa000 2024-04-06T09:57:08.911 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:57:08.908+0000 7f0965192640 -1 reset not still connected to 0x5632c0cfba00 2024-04-06T09:57:08.911 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:57:08.908+0000 7f0965192640 -1 reset not still connected to 0x5632cb3ac000 2024-04-06T09:57:08.911 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:57:08.908+0000 7f0965192640 -1 reset not still connected to 0x5632d8f88820 2024-04-06T09:57:08.911 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:57:08.908+0000 7f0965192640 -1 reset not still connected to 0x5632dac08dd0 2024-04-06T09:57:08.911 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:57:08.908+0000 7f0965192640 -1 reset not still connected to 0x5632dac09040 2024-04-06T09:57:08.911 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:57:08.908+0000 7f0965192640 -1 reset not still connected to 0x5632e1684ea0 2024-04-06T09:57:08.911 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:57:08.908+0000 7f0965192640 -1 reset not still connected to 0x5632e16851e0 2024-04-06T09:57:09.889 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-04-06T09:57:09.884+0000 7fd1b6210640 -1 reset not still connected to 0x55ef9ac60270 2024-04-06T09:57:09.890 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-04-06T09:57:09.884+0000 7fd1b6210640 -1 reset not still connected to 0x55ef9b657e10 2024-04-06T09:57:09.891 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-04-06T09:57:09.888+0000 7fd1b6210640 -1 reset not still connected to 0x55ef9bbe84e0 2024-04-06T09:57:09.891 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-04-06T09:57:09.888+0000 7fd1b6210640 -1 reset not still connected to 0x55ef9bbe8d00 2024-04-06T09:57:09.891 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-04-06T09:57:09.888+0000 7fd1b6210640 -1 reset not still connected to 0x55ef9bbe9ee0 2024-04-06T09:57:09.891 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-04-06T09:57:09.888+0000 7fd1b6210640 -1 reset not still connected to 0x55ef9bd5c9c0 2024-04-06T09:57:09.891 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-04-06T09:57:09.888+0000 7fd1b6210640 -1 reset not still connected to 0x55ef9bd5d520 2024-04-06T09:57:09.891 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-04-06T09:57:09.888+0000 7fd1b6210640 -1 reset not still connected to 0x55ef9bd5dd40 2024-04-06T09:57:09.891 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-04-06T09:57:09.888+0000 7fd1b6210640 -1 reset not still connected to 0x55ef9bd86820 2024-04-06T09:57:09.891 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-04-06T09:57:09.888+0000 7fd1b6210640 -1 reset not still connected to 0x55ef9c103110 2024-04-06T09:57:09.891 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-04-06T09:57:09.888+0000 7fd1b6210640 -1 reset not still connected to 0x55ef9c103930 2024-04-06T09:57:09.891 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-04-06T09:57:09.888+0000 7fd1b6210640 -1 reset not still connected to 0x55ef9c103d40 2024-04-06T09:57:09.892 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-04-06T09:57:09.888+0000 7fd1b6210640 -1 reset not still connected to 0x55ef9e698c30 2024-04-06T09:57:09.892 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-04-06T09:57:09.888+0000 7fd1b6210640 -1 reset not still connected to 0x55ef9e88f520 2024-04-06T09:57:09.892 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-04-06T09:57:09.888+0000 7fd1b6210640 -1 reset not still connected to 0x55ef9eee21a0 2024-04-06T09:57:09.892 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-04-06T09:57:09.888+0000 7fd1b6210640 -1 reset not still connected to 0x55ef9eee3a00 2024-04-06T09:57:09.892 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-04-06T09:57:09.888+0000 7fd1b6210640 -1 reset not still connected to 0x55efa18cc270 2024-04-06T09:57:09.892 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-04-06T09:57:09.888+0000 7fd1b6210640 -1 reset not still connected to 0x55efa18cc9c0 2024-04-06T09:57:09.892 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-04-06T09:57:09.888+0000 7fd1b6210640 -1 reset not still connected to 0x55efd81ee8f0 2024-04-06T09:57:10.197 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ error_code=124 2024-04-06T09:57:10.197 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set -e 2024-04-06T09:57:10.197 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 124 -eq 124 ']' 2024-04-06T09:57:10.197 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:57:10.197 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-06T09:57:10.198 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ image=image_43 2024-04-06T09:57:10.198 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ create_image_and_enable_mirror cluster2 mirror image_43 journal 128M 2024-04-06T09:57:10.198 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-04-06T09:57:10.198 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:57:10.198 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:57:10.198 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:57:10.198 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_43 2024-04-06T09:57:10.198 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:57:10.198 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local mode=journal 2024-04-06T09:57:10.198 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n journal ']' 2024-04-06T09:57:10.198 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:57:10.198 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ create_image cluster2 mirror image_43 128M 2024-04-06T09:57:10.198 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-04-06T09:57:10.198 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:57:10.199 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:57:10.199 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:57:10.199 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_43 2024-04-06T09:57:10.199 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:57:10.199 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local size=128 2024-04-06T09:57:10.199 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n 128M ']' 2024-04-06T09:57:10.199 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ size=128M 2024-04-06T09:57:10.199 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:57:10.199 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,journaling mirror/image_43 2024-04-06T09:57:10.265 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' pool = image ']' 2024-04-06T09:57:10.266 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' mirror = mirror_parent ']' 2024-04-06T09:57:10.266 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n '' ']' 2024-04-06T09:57:10.266 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ stress_write_image cluster2 mirror image_43 2024-04-06T09:57:10.266 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-04-06T09:57:10.266 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:57:10.266 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_43 2024-04-06T09:57:10.266 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-04-06T09:57:10.267 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local duration=10 2024-04-06T09:57:10.267 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set +e 2024-04-06T09:57:10.267 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ timeout 10s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_43 --debug-rbd=20 --debug-journaler=20 2024-04-06T09:57:10.742 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:57:10.736+0000 7f0965192640 -1 reset not still connected to 0x5632abc81d40 2024-04-06T09:57:10.743 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:57:10.736+0000 7f0965192640 -1 reset not still connected to 0x5632abcbe820 2024-04-06T09:57:10.743 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:57:10.736+0000 7f0965192640 -1 reset not still connected to 0x5632e16851e0 2024-04-06T09:57:10.750 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:57:10.748+0000 7f2e84d66640 -1 reset not still connected to 0x560d27567380 2024-04-06T09:57:10.753 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:57:10.752+0000 7f2e84d66640 -1 reset not still connected to 0x560d27839380 2024-04-06T09:57:10.769 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:57:10.768+0000 7f2e84d66640 -1 reset not still connected to 0x560d27c3edd0 2024-04-06T09:57:10.772 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:57:10.768+0000 7f2e84d66640 -1 reset not still connected to 0x560d27c3f2b0 2024-04-06T09:57:10.772 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:57:10.768+0000 7f2e84d66640 -1 reset not still connected to 0x560d29cd9ba0 2024-04-06T09:57:10.772 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:57:10.768+0000 7f2e84d66640 -1 reset not still connected to 0x560d29d5b520 2024-04-06T09:57:10.772 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:57:10.768+0000 7f2e84d66640 -1 reset not still connected to 0x560d2a17d450 2024-04-06T09:57:10.772 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:57:10.768+0000 7f2e84d66640 -1 reset not still connected to 0x560d2a17d6c0 2024-04-06T09:57:10.772 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:57:10.768+0000 7f2e84d66640 -1 reset not still connected to 0x560d2a205ba0 2024-04-06T09:57:10.772 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:57:10.768+0000 7f2e84d66640 -1 reset not still connected to 0x560d2a2336c0 2024-04-06T09:57:10.772 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:57:10.768+0000 7f2e84d66640 -1 reset not still connected to 0x560d2a5705b0 2024-04-06T09:57:10.772 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:57:10.768+0000 7f2e84d66640 -1 reset not still connected to 0x560d381dd5f0 2024-04-06T09:57:11.779 INFO:tasks.ceph.cluster2.osd.0.smithi192.stderr:2024-04-06T09:57:11.778+0000 7f78af6df640 -1 reset not still connected to 0x5604225c85b0 2024-04-06T09:57:11.779 INFO:tasks.ceph.cluster2.osd.0.smithi192.stderr:2024-04-06T09:57:11.778+0000 7f78af6df640 -1 reset not still connected to 0x560422aa4ea0 2024-04-06T09:57:11.779 INFO:tasks.ceph.cluster2.osd.0.smithi192.stderr:2024-04-06T09:57:11.778+0000 7f78af6df640 -1 reset not still connected to 0x560422aa52b0 2024-04-06T09:57:11.779 INFO:tasks.ceph.cluster2.osd.0.smithi192.stderr:2024-04-06T09:57:11.778+0000 7f78af6df640 -1 reset not still connected to 0x560423937520 2024-04-06T09:57:11.780 INFO:tasks.ceph.cluster2.osd.0.smithi192.stderr:2024-04-06T09:57:11.778+0000 7f78af6df640 -1 reset not still connected to 0x5604242c76c0 2024-04-06T09:57:11.780 INFO:tasks.ceph.cluster2.osd.0.smithi192.stderr:2024-04-06T09:57:11.778+0000 7f78af6df640 -1 reset not still connected to 0x5604247b84e0 2024-04-06T09:57:11.780 INFO:tasks.ceph.cluster2.osd.0.smithi192.stderr:2024-04-06T09:57:11.778+0000 7f78af6df640 -1 reset not still connected to 0x560436199450 2024-04-06T09:57:11.780 INFO:tasks.ceph.cluster2.osd.0.smithi192.stderr:2024-04-06T09:57:11.778+0000 7f78af6df640 -1 reset not still connected to 0x56043d1216c0 2024-04-06T09:57:11.780 INFO:tasks.ceph.cluster2.osd.0.smithi192.stderr:2024-04-06T09:57:11.778+0000 7f78af6df640 -1 reset not still connected to 0x560444f6c820 2024-04-06T09:57:11.780 INFO:tasks.ceph.cluster2.osd.0.smithi192.stderr:2024-04-06T09:57:11.778+0000 7f78af6df640 -1 reset not still connected to 0x560449684410 2024-04-06T09:57:11.780 INFO:tasks.ceph.cluster2.osd.0.smithi192.stderr:2024-04-06T09:57:11.778+0000 7f78af6df640 -1 reset not still connected to 0x560456d576c0 2024-04-06T09:57:12.086 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:57:12.080+0000 7f0965192640 -1 reset not still connected to 0x5632a4cdbba0 2024-04-06T09:57:12.089 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:57:12.084+0000 7f0965192640 -1 reset not still connected to 0x5632a5e9dee0 2024-04-06T09:57:12.090 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:57:12.084+0000 7f0965192640 -1 reset not still connected to 0x5632a89f4750 2024-04-06T09:57:12.090 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:57:12.084+0000 7f0965192640 -1 reset not still connected to 0x5632a89f52b0 2024-04-06T09:57:12.090 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:57:12.084+0000 7f0965192640 -1 reset not still connected to 0x5632abc812b0 2024-04-06T09:57:12.090 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:57:12.084+0000 7f0965192640 -1 reset not still connected to 0x5632abc81d40 2024-04-06T09:57:12.090 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:57:12.084+0000 7f0965192640 -1 reset not still connected to 0x5632abcbe820 2024-04-06T09:57:12.090 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:57:12.084+0000 7f0965192640 -1 reset not still connected to 0x5632b81f3a00 2024-04-06T09:57:12.090 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:57:12.084+0000 7f0965192640 -1 reset not still connected to 0x5632b9592000 2024-04-06T09:57:12.090 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:57:12.084+0000 7f0965192640 -1 reset not still connected to 0x5632b9593110 2024-04-06T09:57:12.090 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:57:12.084+0000 7f0965192640 -1 reset not still connected to 0x5632c0cfa000 2024-04-06T09:57:12.090 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:57:12.084+0000 7f0965192640 -1 reset not still connected to 0x5632c0cfba00 2024-04-06T09:57:12.091 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:57:12.084+0000 7f0965192640 -1 reset not still connected to 0x5632d8f88820 2024-04-06T09:57:12.091 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:57:12.088+0000 7f0965192640 -1 reset not still connected to 0x5632dac09040 2024-04-06T09:57:12.091 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:57:12.088+0000 7f0965192640 -1 reset not still connected to 0x5632e16851e0 2024-04-06T09:57:12.619 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-04-06T09:57:12.616+0000 7fd1b6210640 -1 reset not still connected to 0x55ef9ac60270 2024-04-06T09:57:12.619 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-04-06T09:57:12.616+0000 7fd1b6210640 -1 reset not still connected to 0x55ef9b657e10 2024-04-06T09:57:12.619 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-04-06T09:57:12.616+0000 7fd1b6210640 -1 reset not still connected to 0x55ef9bbe84e0 2024-04-06T09:57:12.619 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-04-06T09:57:12.616+0000 7fd1b6210640 -1 reset not still connected to 0x55ef9bbe8d00 2024-04-06T09:57:12.620 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-04-06T09:57:12.616+0000 7fd1b6210640 -1 reset not still connected to 0x55ef9bbe9ee0 2024-04-06T09:57:12.620 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-04-06T09:57:12.616+0000 7fd1b6210640 -1 reset not still connected to 0x55ef9bd5c9c0 2024-04-06T09:57:12.620 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-04-06T09:57:12.616+0000 7fd1b6210640 -1 reset not still connected to 0x55ef9bd5d520 2024-04-06T09:57:12.620 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-04-06T09:57:12.616+0000 7fd1b6210640 -1 reset not still connected to 0x55ef9bd5dd40 2024-04-06T09:57:12.620 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-04-06T09:57:12.616+0000 7fd1b6210640 -1 reset not still connected to 0x55ef9bd86820 2024-04-06T09:57:12.620 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-04-06T09:57:12.616+0000 7fd1b6210640 -1 reset not still connected to 0x55ef9c103110 2024-04-06T09:57:12.620 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-04-06T09:57:12.616+0000 7fd1b6210640 -1 reset not still connected to 0x55ef9c103930 2024-04-06T09:57:12.620 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-04-06T09:57:12.616+0000 7fd1b6210640 -1 reset not still connected to 0x55ef9c103d40 2024-04-06T09:57:12.620 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-04-06T09:57:12.616+0000 7fd1b6210640 -1 reset not still connected to 0x55ef9e698c30 2024-04-06T09:57:12.620 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-04-06T09:57:12.616+0000 7fd1b6210640 -1 reset not still connected to 0x55ef9e88f520 2024-04-06T09:57:12.622 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-04-06T09:57:12.620+0000 7fd1b6210640 -1 reset not still connected to 0x55ef9eee21a0 2024-04-06T09:57:12.622 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-04-06T09:57:12.620+0000 7fd1b6210640 -1 reset not still connected to 0x55ef9eee3a00 2024-04-06T09:57:12.622 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-04-06T09:57:12.620+0000 7fd1b6210640 -1 reset not still connected to 0x55efa18cc270 2024-04-06T09:57:12.623 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-04-06T09:57:12.620+0000 7fd1b6210640 -1 reset not still connected to 0x55efa18cc9c0 2024-04-06T09:57:12.623 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-04-06T09:57:12.620+0000 7fd1b6210640 -1 reset not still connected to 0x55efa18cca90 2024-04-06T09:57:12.623 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-04-06T09:57:12.624+0000 7fd1b6210640 -1 reset not still connected to 0x55efbb630d00 2024-04-06T09:57:12.623 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-04-06T09:57:12.624+0000 7fd1b6210640 -1 reset not still connected to 0x55efd81ee8f0 2024-04-06T09:57:12.623 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-04-06T09:57:12.624+0000 7fd1b6210640 -1 reset not still connected to 0x55efe06a9860 2024-04-06T09:57:13.286 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:100 IOs, 41668608 bytes 2024-04-06T09:57:15.846 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:200 IOs, 65292288 bytes 2024-04-06T09:57:18.475 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-04-06T09:57:18.472+0000 7fd1b6210640 -1 reset not still connected to 0x55efa18cc270 2024-04-06T09:57:18.475 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-04-06T09:57:18.472+0000 7fd1b6210640 -1 reset not still connected to 0x55efd81ee8f0 2024-04-06T09:57:19.149 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:57:19.144+0000 7f2e84d66640 -1 reset not still connected to 0x560d29cd9ba0 2024-04-06T09:57:19.149 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:57:19.144+0000 7f2e84d66640 -1 reset not still connected to 0x560d29d5b520 2024-04-06T09:57:19.149 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:57:19.144+0000 7f2e84d66640 -1 reset not still connected to 0x560d2a17d450 2024-04-06T09:57:19.149 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:57:19.144+0000 7f2e84d66640 -1 reset not still connected to 0x560d2a17d6c0 2024-04-06T09:57:19.149 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:57:19.144+0000 7f2e84d66640 -1 reset not still connected to 0x560d2a204b60 2024-04-06T09:57:19.152 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:57:19.152+0000 7f2e84d66640 -1 reset not still connected to 0x560d2a205ba0 2024-04-06T09:57:19.153 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:57:19.152+0000 7f2e84d66640 -1 reset not still connected to 0x560d2a2336c0 2024-04-06T09:57:19.153 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:57:19.152+0000 7f2e84d66640 -1 reset not still connected to 0x560d2a5705b0 2024-04-06T09:57:19.153 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:57:19.152+0000 7f2e84d66640 -1 reset not still connected to 0x560d35349380 2024-04-06T09:57:19.153 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:57:19.152+0000 7f2e84d66640 -1 reset not still connected to 0x560d3620b520 2024-04-06T09:57:19.155 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:57:19.152+0000 7f2e84d66640 -1 reset not still connected to 0x560d3620bee0 2024-04-06T09:57:19.155 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:57:19.152+0000 7f2e84d66640 -1 reset not still connected to 0x560d368e4f70 2024-04-06T09:57:19.162 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:57:19.156+0000 7f2e84d66640 -1 reset not still connected to 0x560d36912ea0 2024-04-06T09:57:19.162 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:57:19.156+0000 7f2e84d66640 -1 reset not still connected to 0x560d381dd5f0 2024-04-06T09:57:19.162 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:57:19.156+0000 7f2e84d66640 -1 reset not still connected to 0x560d4bc7e8f0 2024-04-06T09:57:19.162 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:57:19.156+0000 7f2e84d66640 -1 reset not still connected to 0x560d4bc7ec30 2024-04-06T09:57:19.162 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:57:19.156+0000 7f2e84d66640 -1 reset not still connected to 0x560d50adba00 2024-04-06T09:57:20.039 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:57:20.024+0000 7f2e84d66640 -1 reset not still connected to 0x560d27567380 2024-04-06T09:57:20.039 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:57:20.028+0000 7f2e84d66640 -1 reset not still connected to 0x560d27839930 2024-04-06T09:57:20.039 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:57:20.028+0000 7f2e84d66640 -1 reset not still connected to 0x560d27c3edd0 2024-04-06T09:57:20.044 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:57:20.036+0000 7f2e84d66640 -1 reset not still connected to 0x560d27c3f2b0 2024-04-06T09:57:20.044 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:57:20.036+0000 7f2e84d66640 -1 reset not still connected to 0x560d281875f0 2024-04-06T09:57:20.044 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:57:20.036+0000 7f2e84d66640 -1 reset not still connected to 0x560d29cd8f70 2024-04-06T09:57:20.044 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:57:20.036+0000 7f2e84d66640 -1 reset not still connected to 0x560d29cd9ba0 2024-04-06T09:57:20.044 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:57:20.036+0000 7f2e84d66640 -1 reset not still connected to 0x560d29d5a340 2024-04-06T09:57:20.044 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:57:20.036+0000 7f2e84d66640 -1 reset not still connected to 0x560d29d5b520 2024-04-06T09:57:20.044 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:57:20.036+0000 7f2e84d66640 -1 reset not still connected to 0x560d2a17d450 2024-04-06T09:57:20.044 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:57:20.036+0000 7f2e84d66640 -1 reset not still connected to 0x560d2a17d6c0 2024-04-06T09:57:20.045 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:57:20.040+0000 7f2e84d66640 -1 reset not still connected to 0x560d2a204b60 2024-04-06T09:57:20.045 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:57:20.040+0000 7f2e84d66640 -1 reset not still connected to 0x560d2a205ba0 2024-04-06T09:57:20.045 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:57:20.040+0000 7f2e84d66640 -1 reset not still connected to 0x560d2a2336c0 2024-04-06T09:57:20.045 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:57:20.040+0000 7f2e84d66640 -1 reset not still connected to 0x560d2a570000 2024-04-06T09:57:20.045 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:57:20.040+0000 7f2e84d66640 -1 reset not still connected to 0x560d2a5705b0 2024-04-06T09:57:20.045 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:57:20.040+0000 7f2e84d66640 -1 reset not still connected to 0x560d35349380 2024-04-06T09:57:20.045 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:57:20.040+0000 7f2e84d66640 -1 reset not still connected to 0x560d3620b520 2024-04-06T09:57:20.048 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:57:20.044+0000 7f2e84d66640 -1 reset not still connected to 0x560d368e4f70 2024-04-06T09:57:20.048 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:57:20.044+0000 7f2e84d66640 -1 reset not still connected to 0x560d36912ea0 2024-04-06T09:57:20.048 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:57:20.044+0000 7f2e84d66640 -1 reset not still connected to 0x560d381dcc30 2024-04-06T09:57:20.048 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:57:20.044+0000 7f2e84d66640 -1 reset not still connected to 0x560d381dd5f0 2024-04-06T09:57:20.048 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:57:20.044+0000 7f2e84d66640 -1 reset not still connected to 0x560d390dca90 2024-04-06T09:57:20.048 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:57:20.044+0000 7f2e84d66640 -1 reset not still connected to 0x560d4bc7e8f0 2024-04-06T09:57:20.048 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:57:20.044+0000 7f2e84d66640 -1 reset not still connected to 0x560d4bc7ec30 2024-04-06T09:57:20.277 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ error_code=124 2024-04-06T09:57:20.278 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set -e 2024-04-06T09:57:20.278 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 124 -eq 124 ']' 2024-04-06T09:57:20.278 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:57:20.278 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-06T09:57:20.278 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ image=image_44 2024-04-06T09:57:20.278 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ create_image_and_enable_mirror cluster2 mirror image_44 journal 128M 2024-04-06T09:57:20.278 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-04-06T09:57:20.278 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:57:20.278 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:57:20.278 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:57:20.278 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_44 2024-04-06T09:57:20.279 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:57:20.279 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local mode=journal 2024-04-06T09:57:20.279 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n journal ']' 2024-04-06T09:57:20.279 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:57:20.279 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ create_image cluster2 mirror image_44 128M 2024-04-06T09:57:20.279 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-04-06T09:57:20.279 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:57:20.279 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:57:20.279 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:57:20.279 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_44 2024-04-06T09:57:20.279 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:57:20.279 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local size=128 2024-04-06T09:57:20.279 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n 128M ']' 2024-04-06T09:57:20.279 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ size=128M 2024-04-06T09:57:20.279 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:57:20.280 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,journaling mirror/image_44 2024-04-06T09:57:20.371 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' pool = image ']' 2024-04-06T09:57:20.371 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' mirror = mirror_parent ']' 2024-04-06T09:57:20.372 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n '' ']' 2024-04-06T09:57:20.372 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ stress_write_image cluster2 mirror image_44 2024-04-06T09:57:20.372 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-04-06T09:57:20.372 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:57:20.372 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_44 2024-04-06T09:57:20.372 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-04-06T09:57:20.373 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local duration=5 2024-04-06T09:57:20.373 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set +e 2024-04-06T09:57:20.373 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ timeout 5s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_44 --debug-rbd=20 --debug-journaler=20 2024-04-06T09:57:22.097 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:100 IOs, 18680832 bytes 2024-04-06T09:57:22.383 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:57:22.380+0000 7f2e84d66640 -1 reset not still connected to 0x560d27072820 2024-04-06T09:57:22.383 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:57:22.380+0000 7f2e84d66640 -1 reset not still connected to 0x560d27073860 2024-04-06T09:57:22.383 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:57:22.380+0000 7f2e84d66640 -1 reset not still connected to 0x560d2722e5b0 2024-04-06T09:57:22.383 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:57:22.380+0000 7f2e84d66640 -1 reset not still connected to 0x560d27566ea0 2024-04-06T09:57:22.383 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:57:22.380+0000 7f2e84d66640 -1 reset not still connected to 0x560d27567380 2024-04-06T09:57:22.383 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:57:22.380+0000 7f2e84d66640 -1 reset not still connected to 0x560d27839380 2024-04-06T09:57:22.383 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:57:22.380+0000 7f2e84d66640 -1 reset not still connected to 0x560d27839930 2024-04-06T09:57:22.383 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:57:22.380+0000 7f2e84d66640 -1 reset not still connected to 0x560d27c3edd0 2024-04-06T09:57:22.383 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:57:22.380+0000 7f2e84d66640 -1 reset not still connected to 0x560d27c3f2b0 2024-04-06T09:57:22.384 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:57:22.380+0000 7f2e84d66640 -1 reset not still connected to 0x560d294908f0 2024-04-06T09:57:22.386 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:57:22.384+0000 7f2e84d66640 -1 reset not still connected to 0x560d29cd8f70 2024-04-06T09:57:22.387 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:57:22.384+0000 7f2e84d66640 -1 reset not still connected to 0x560d29cd9ba0 2024-04-06T09:57:22.387 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:57:22.384+0000 7f2e84d66640 -1 reset not still connected to 0x560d29d5b520 2024-04-06T09:57:22.387 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:57:22.384+0000 7f2e84d66640 -1 reset not still connected to 0x560d2a17d450 2024-04-06T09:57:22.387 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:57:22.384+0000 7f2e84d66640 -1 reset not still connected to 0x560d2a17d6c0 2024-04-06T09:57:22.387 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:57:22.384+0000 7f2e84d66640 -1 reset not still connected to 0x560d2a204b60 2024-04-06T09:57:22.387 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:57:22.384+0000 7f2e84d66640 -1 reset not still connected to 0x560d2a205ba0 2024-04-06T09:57:22.387 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:57:22.384+0000 7f2e84d66640 -1 reset not still connected to 0x560d2a2336c0 2024-04-06T09:57:22.387 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:57:22.384+0000 7f2e84d66640 -1 reset not still connected to 0x560d2a5705b0 2024-04-06T09:57:22.387 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:57:22.384+0000 7f2e84d66640 -1 reset not still connected to 0x560d35349380 2024-04-06T09:57:22.387 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:57:22.384+0000 7f2e84d66640 -1 reset not still connected to 0x560d3620b520 2024-04-06T09:57:22.387 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:57:22.384+0000 7f2e84d66640 -1 reset not still connected to 0x560d3620bee0 2024-04-06T09:57:22.387 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:57:22.384+0000 7f2e84d66640 -1 reset not still connected to 0x560d368e4f70 2024-04-06T09:57:22.387 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:57:22.384+0000 7f2e84d66640 -1 reset not still connected to 0x560d36912ea0 2024-04-06T09:57:22.387 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:57:22.384+0000 7f2e84d66640 -1 reset not still connected to 0x560d381dd5f0 2024-04-06T09:57:22.388 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:57:22.384+0000 7f2e84d66640 -1 reset not still connected to 0x560d4bc7e8f0 2024-04-06T09:57:22.388 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:57:22.384+0000 7f2e84d66640 -1 reset not still connected to 0x560d4bc7ec30 2024-04-06T09:57:22.388 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:57:22.384+0000 7f2e84d66640 -1 reset not still connected to 0x560d50adba00 2024-04-06T09:57:22.621 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:57:22.616+0000 7f0965192640 -1 reset not still connected to 0x5632a0ff5110 2024-04-06T09:57:22.621 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:57:22.616+0000 7f0965192640 -1 reset not still connected to 0x5632a1988000 2024-04-06T09:57:22.621 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:57:22.616+0000 7f0965192640 -1 reset not still connected to 0x5632a1b73ee0 2024-04-06T09:57:22.621 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:57:22.616+0000 7f0965192640 -1 reset not still connected to 0x5632a2793520 2024-04-06T09:57:22.621 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:57:22.616+0000 7f0965192640 -1 reset not still connected to 0x5632a30f0270 2024-04-06T09:57:22.627 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:57:22.624+0000 7f0965192640 -1 reset not still connected to 0x5632a30f0f70 2024-04-06T09:57:22.628 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:57:22.624+0000 7f0965192640 -1 reset not still connected to 0x5632a417e750 2024-04-06T09:57:22.628 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:57:22.624+0000 7f0965192640 -1 reset not still connected to 0x5632a4cda000 2024-04-06T09:57:22.632 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:57:22.628+0000 7f0965192640 -1 reset not still connected to 0x5632a4cdbba0 2024-04-06T09:57:22.634 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:57:22.632+0000 7f0965192640 -1 reset not still connected to 0x5632a5e9dee0 2024-04-06T09:57:22.634 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:57:22.632+0000 7f0965192640 -1 reset not still connected to 0x5632a89f4750 2024-04-06T09:57:22.635 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:57:22.632+0000 7f0965192640 -1 reset not still connected to 0x5632a89f52b0 2024-04-06T09:57:22.635 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:57:22.632+0000 7f0965192640 -1 reset not still connected to 0x5632abc812b0 2024-04-06T09:57:22.635 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:57:22.632+0000 7f0965192640 -1 reset not still connected to 0x5632abc81d40 2024-04-06T09:57:22.635 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:57:22.632+0000 7f0965192640 -1 reset not still connected to 0x5632abcbe820 2024-04-06T09:57:22.635 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:57:22.632+0000 7f0965192640 -1 reset not still connected to 0x5632abcbf930 2024-04-06T09:57:22.635 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:57:22.632+0000 7f0965192640 -1 reset not still connected to 0x5632b81f3a00 2024-04-06T09:57:22.635 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:57:22.632+0000 7f0965192640 -1 reset not still connected to 0x5632b9592000 2024-04-06T09:57:22.635 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:57:22.632+0000 7f0965192640 -1 reset not still connected to 0x5632b9593110 2024-04-06T09:57:22.636 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:57:22.632+0000 7f0965192640 -1 reset not still connected to 0x5632c0cfa000 2024-04-06T09:57:22.636 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:57:22.632+0000 7f0965192640 -1 reset not still connected to 0x5632c0cfba00 2024-04-06T09:57:22.636 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:57:22.632+0000 7f0965192640 -1 reset not still connected to 0x5632cb3ac000 2024-04-06T09:57:22.636 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:57:22.632+0000 7f0965192640 -1 reset not still connected to 0x5632d8f88820 2024-04-06T09:57:22.636 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:57:22.632+0000 7f0965192640 -1 reset not still connected to 0x5632dac08dd0 2024-04-06T09:57:22.636 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:57:22.632+0000 7f0965192640 -1 reset not still connected to 0x5632dac09040 2024-04-06T09:57:22.636 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:57:22.632+0000 7f0965192640 -1 reset not still connected to 0x5632e1684ea0 2024-04-06T09:57:22.637 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:57:22.632+0000 7f0965192640 -1 reset not still connected to 0x5632e16851e0 2024-04-06T09:57:22.680 INFO:tasks.ceph.cluster2.osd.2.smithi192.stderr:2024-04-06T09:57:22.678+0000 7fa534aba640 -1 reset not still connected to 0x559338f415f0 2024-04-06T09:57:22.680 INFO:tasks.ceph.cluster2.osd.2.smithi192.stderr:2024-04-06T09:57:22.678+0000 7fa534aba640 -1 reset not still connected to 0x55933bc92d00 2024-04-06T09:57:22.680 INFO:tasks.ceph.cluster2.osd.2.smithi192.stderr:2024-04-06T09:57:22.678+0000 7fa534aba640 -1 reset not still connected to 0x55933ce91110 2024-04-06T09:57:22.680 INFO:tasks.ceph.cluster2.osd.2.smithi192.stderr:2024-04-06T09:57:22.678+0000 7fa534aba640 -1 reset not still connected to 0x55933ecf01a0 2024-04-06T09:57:22.680 INFO:tasks.ceph.cluster2.osd.2.smithi192.stderr:2024-04-06T09:57:22.678+0000 7fa534aba640 -1 reset not still connected to 0x55933fc2dd40 2024-04-06T09:57:22.681 INFO:tasks.ceph.cluster2.osd.2.smithi192.stderr:2024-04-06T09:57:22.678+0000 7fa534aba640 -1 reset not still connected to 0x559341658b60 2024-04-06T09:57:22.681 INFO:tasks.ceph.cluster2.osd.2.smithi192.stderr:2024-04-06T09:57:22.678+0000 7fa534aba640 -1 reset not still connected to 0x55934811f6c0 2024-04-06T09:57:22.681 INFO:tasks.ceph.cluster2.osd.2.smithi192.stderr:2024-04-06T09:57:22.678+0000 7fa534aba640 -1 reset not still connected to 0x55934feb6000 2024-04-06T09:57:22.681 INFO:tasks.ceph.cluster2.osd.2.smithi192.stderr:2024-04-06T09:57:22.678+0000 7fa534aba640 -1 reset not still connected to 0x55935bfc9d40 2024-04-06T09:57:23.672 INFO:tasks.ceph.cluster2.osd.2.smithi192.stderr:2024-04-06T09:57:23.670+0000 7fa534aba640 -1 reset not still connected to 0x559338f415f0 2024-04-06T09:57:23.673 INFO:tasks.ceph.cluster2.osd.2.smithi192.stderr:2024-04-06T09:57:23.674+0000 7fa534aba640 -1 reset not still connected to 0x55933ce91110 2024-04-06T09:57:23.673 INFO:tasks.ceph.cluster2.osd.2.smithi192.stderr:2024-04-06T09:57:23.674+0000 7fa534aba640 -1 reset not still connected to 0x55933ecf01a0 2024-04-06T09:57:23.673 INFO:tasks.ceph.cluster2.osd.2.smithi192.stderr:2024-04-06T09:57:23.674+0000 7fa534aba640 -1 reset not still connected to 0x55933fc2dd40 2024-04-06T09:57:23.673 INFO:tasks.ceph.cluster2.osd.2.smithi192.stderr:2024-04-06T09:57:23.674+0000 7fa534aba640 -1 reset not still connected to 0x559341658b60 2024-04-06T09:57:23.674 INFO:tasks.ceph.cluster2.osd.2.smithi192.stderr:2024-04-06T09:57:23.674+0000 7fa534aba640 -1 reset not still connected to 0x55934811f6c0 2024-04-06T09:57:23.675 INFO:tasks.ceph.cluster2.osd.2.smithi192.stderr:2024-04-06T09:57:23.674+0000 7fa534aba640 -1 reset not still connected to 0x559348f02820 2024-04-06T09:57:23.680 INFO:tasks.ceph.cluster2.osd.2.smithi192.stderr:2024-04-06T09:57:23.678+0000 7fa534aba640 -1 reset not still connected to 0x5593550955f0 2024-04-06T09:57:23.680 INFO:tasks.ceph.cluster2.osd.2.smithi192.stderr:2024-04-06T09:57:23.678+0000 7fa534aba640 -1 reset not still connected to 0x55935bfc9d40 2024-04-06T09:57:23.680 INFO:tasks.ceph.cluster2.osd.2.smithi192.stderr:2024-04-06T09:57:23.678+0000 7fa534aba640 -1 reset not still connected to 0x55936c6f44e0 2024-04-06T09:57:24.043 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-04-06T09:57:24.036+0000 7fd1b6210640 -1 reset not still connected to 0x55ef9bd5d520 2024-04-06T09:57:24.045 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-04-06T09:57:24.040+0000 7fd1b6210640 -1 reset not still connected to 0x55ef9c103110 2024-04-06T09:57:24.047 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-04-06T09:57:24.044+0000 7fd1b6210640 -1 reset not still connected to 0x55ef9c103930 2024-04-06T09:57:24.057 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-04-06T09:57:24.056+0000 7fd1b6210640 -1 reset not still connected to 0x55ef9c103d40 2024-04-06T09:57:24.057 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-04-06T09:57:24.056+0000 7fd1b6210640 -1 reset not still connected to 0x55ef9e698c30 2024-04-06T09:57:24.057 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-04-06T09:57:24.056+0000 7fd1b6210640 -1 reset not still connected to 0x55ef9e88f520 2024-04-06T09:57:24.057 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-04-06T09:57:24.056+0000 7fd1b6210640 -1 reset not still connected to 0x55ef9eee21a0 2024-04-06T09:57:24.066 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-04-06T09:57:24.060+0000 7fd1b6210640 -1 reset not still connected to 0x55efa18cc270 2024-04-06T09:57:24.066 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-04-06T09:57:24.060+0000 7fd1b6210640 -1 reset not still connected to 0x55efa18cc9c0 2024-04-06T09:57:24.066 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-04-06T09:57:24.060+0000 7fd1b6210640 -1 reset not still connected to 0x55efa18cca90 2024-04-06T09:57:24.066 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-04-06T09:57:24.060+0000 7fd1b6210640 -1 reset not still connected to 0x55efbb630d00 2024-04-06T09:57:24.066 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-04-06T09:57:24.060+0000 7fd1b6210640 -1 reset not still connected to 0x55efd81ee8f0 2024-04-06T09:57:24.067 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-04-06T09:57:24.060+0000 7fd1b6210640 -1 reset not still connected to 0x55efe06a9860 2024-04-06T09:57:24.546 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:200 IOs, 36009984 bytes 2024-04-06T09:57:25.384 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ error_code=124 2024-04-06T09:57:25.384 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set -e 2024-04-06T09:57:25.384 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 124 -eq 124 ']' 2024-04-06T09:57:25.384 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:57:25.384 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-06T09:57:25.384 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ image=image_45 2024-04-06T09:57:25.384 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ create_image_and_enable_mirror cluster2 mirror image_45 journal 128M 2024-04-06T09:57:25.384 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-04-06T09:57:25.384 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:57:25.384 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:57:25.384 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:57:25.385 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_45 2024-04-06T09:57:25.385 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:57:25.385 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local mode=journal 2024-04-06T09:57:25.385 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n journal ']' 2024-04-06T09:57:25.385 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:57:25.385 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ create_image cluster2 mirror image_45 128M 2024-04-06T09:57:25.385 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-04-06T09:57:25.385 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:57:25.385 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:57:25.385 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:57:25.385 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_45 2024-04-06T09:57:25.385 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:57:25.385 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local size=128 2024-04-06T09:57:25.385 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n 128M ']' 2024-04-06T09:57:25.385 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ size=128M 2024-04-06T09:57:25.386 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:57:25.386 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,journaling mirror/image_45 2024-04-06T09:57:25.467 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' pool = image ']' 2024-04-06T09:57:25.468 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' mirror = mirror_parent ']' 2024-04-06T09:57:25.468 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n '' ']' 2024-04-06T09:57:25.468 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ stress_write_image cluster2 mirror image_45 2024-04-06T09:57:25.468 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-04-06T09:57:25.468 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:57:25.468 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_45 2024-04-06T09:57:25.468 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-04-06T09:57:25.469 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local duration=6 2024-04-06T09:57:25.469 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set +e 2024-04-06T09:57:25.469 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ timeout 6s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_45 --debug-rbd=20 --debug-journaler=20 2024-04-06T09:57:25.702 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:57:25.692+0000 7f0965192640 -1 reset not still connected to 0x56329f99da00 2024-04-06T09:57:25.702 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:57:25.692+0000 7f0965192640 -1 reset not still connected to 0x5632a08055f0 2024-04-06T09:57:25.702 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:57:25.692+0000 7f0965192640 -1 reset not still connected to 0x5632a0ff4410 2024-04-06T09:57:25.702 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:57:25.692+0000 7f0965192640 -1 reset not still connected to 0x5632a14ac820 2024-04-06T09:57:25.702 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:57:25.692+0000 7f0965192640 -1 reset not still connected to 0x5632a1988000 2024-04-06T09:57:25.702 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:57:25.692+0000 7f0965192640 -1 reset not still connected to 0x5632a1b73ee0 2024-04-06T09:57:25.702 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:57:25.692+0000 7f0965192640 -1 reset not still connected to 0x5632a2793520 2024-04-06T09:57:25.702 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:57:25.692+0000 7f0965192640 -1 reset not still connected to 0x5632a30f0270 2024-04-06T09:57:25.703 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:57:25.692+0000 7f0965192640 -1 reset not still connected to 0x5632a417e750 2024-04-06T09:57:25.703 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:57:25.692+0000 7f0965192640 -1 reset not still connected to 0x5632a4cda000 2024-04-06T09:57:25.703 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:57:25.692+0000 7f0965192640 -1 reset not still connected to 0x5632a4cda9c0 2024-04-06T09:57:25.703 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:57:25.692+0000 7f0965192640 -1 reset not still connected to 0x5632a4cdbba0 2024-04-06T09:57:25.703 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:57:25.692+0000 7f0965192640 -1 reset not still connected to 0x5632a5e9dee0 2024-04-06T09:57:25.703 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:57:25.692+0000 7f0965192640 -1 reset not still connected to 0x5632a89f4750 2024-04-06T09:57:25.703 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:57:25.692+0000 7f0965192640 -1 reset not still connected to 0x5632a89f52b0 2024-04-06T09:57:25.703 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:57:25.692+0000 7f0965192640 -1 reset not still connected to 0x5632abc812b0 2024-04-06T09:57:25.703 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:57:25.692+0000 7f0965192640 -1 reset not still connected to 0x5632abc81d40 2024-04-06T09:57:25.703 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:57:25.692+0000 7f0965192640 -1 reset not still connected to 0x5632abcbe820 2024-04-06T09:57:25.703 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:57:25.696+0000 7f0965192640 -1 reset not still connected to 0x5632abcbf930 2024-04-06T09:57:25.703 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:57:25.696+0000 7f0965192640 -1 reset not still connected to 0x5632b81f3a00 2024-04-06T09:57:25.703 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:57:25.696+0000 7f0965192640 -1 reset not still connected to 0x5632b9592000 2024-04-06T09:57:25.703 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:57:25.696+0000 7f0965192640 -1 reset not still connected to 0x5632b9593110 2024-04-06T09:57:25.703 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:57:25.696+0000 7f0965192640 -1 reset not still connected to 0x5632c0cfa000 2024-04-06T09:57:25.703 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:57:25.696+0000 7f0965192640 -1 reset not still connected to 0x5632c0cfba00 2024-04-06T09:57:25.704 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:57:25.696+0000 7f0965192640 -1 reset not still connected to 0x5632cb3ac000 2024-04-06T09:57:25.704 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:57:25.696+0000 7f0965192640 -1 reset not still connected to 0x5632d8f88820 2024-04-06T09:57:25.704 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:57:25.696+0000 7f0965192640 -1 reset not still connected to 0x5632dac08dd0 2024-04-06T09:57:25.704 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:57:25.696+0000 7f0965192640 -1 reset not still connected to 0x5632dac09040 2024-04-06T09:57:25.704 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:57:25.704+0000 7f0965192640 -1 reset not still connected to 0x5632e16851e0 2024-04-06T09:57:26.260 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:57:26.260+0000 7f2e84d66640 -1 reset not still connected to 0x560d27c3edd0 2024-04-06T09:57:26.260 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:57:26.260+0000 7f2e84d66640 -1 reset not still connected to 0x560d27c3f2b0 2024-04-06T09:57:26.260 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:57:26.260+0000 7f2e84d66640 -1 reset not still connected to 0x560d294908f0 2024-04-06T09:57:26.260 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:57:26.260+0000 7f2e84d66640 -1 reset not still connected to 0x560d29cd8f70 2024-04-06T09:57:26.260 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:57:26.260+0000 7f2e84d66640 -1 reset not still connected to 0x560d29cd9ba0 2024-04-06T09:57:26.260 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:57:26.260+0000 7f2e84d66640 -1 reset not still connected to 0x560d29d5b520 2024-04-06T09:57:26.261 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:57:26.260+0000 7f2e84d66640 -1 reset not still connected to 0x560d2a17d450 2024-04-06T09:57:26.261 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:57:26.260+0000 7f2e84d66640 -1 reset not still connected to 0x560d2a17d6c0 2024-04-06T09:57:26.261 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:57:26.260+0000 7f2e84d66640 -1 reset not still connected to 0x560d2a204b60 2024-04-06T09:57:26.261 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:57:26.260+0000 7f2e84d66640 -1 reset not still connected to 0x560d2a205ba0 2024-04-06T09:57:26.261 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:57:26.260+0000 7f2e84d66640 -1 reset not still connected to 0x560d2a2336c0 2024-04-06T09:57:26.261 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:57:26.260+0000 7f2e84d66640 -1 reset not still connected to 0x560d2a5705b0 2024-04-06T09:57:26.263 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:57:26.260+0000 7f2e84d66640 -1 reset not still connected to 0x560d3620b520 2024-04-06T09:57:26.263 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:57:26.260+0000 7f2e84d66640 -1 reset not still connected to 0x560d3620bee0 2024-04-06T09:57:26.265 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:57:26.264+0000 7f2e84d66640 -1 reset not still connected to 0x560d368e4f70 2024-04-06T09:57:26.265 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:57:26.264+0000 7f2e84d66640 -1 reset not still connected to 0x560d36912ea0 2024-04-06T09:57:26.265 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:57:26.264+0000 7f2e84d66640 -1 reset not still connected to 0x560d381dd5f0 2024-04-06T09:57:26.273 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:57:26.272+0000 7f2e84d66640 -1 reset not still connected to 0x560d47e9aa90 2024-04-06T09:57:26.273 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:57:26.272+0000 7f2e84d66640 -1 reset not still connected to 0x560d4bc7e8f0 2024-04-06T09:57:26.273 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:57:26.272+0000 7f2e84d66640 -1 reset not still connected to 0x560d4bc7ec30 2024-04-06T09:57:26.273 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:57:26.272+0000 7f2e84d66640 -1 reset not still connected to 0x560d50adba00 2024-04-06T09:57:26.772 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:57:26.768+0000 7f0965192640 -1 reset not still connected to 0x56329f99da00 2024-04-06T09:57:26.772 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:57:26.772+0000 7f0965192640 -1 reset not still connected to 0x5632a0ff4410 2024-04-06T09:57:26.775 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:57:26.772+0000 7f0965192640 -1 reset not still connected to 0x5632a0ff5110 2024-04-06T09:57:26.775 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:57:26.772+0000 7f0965192640 -1 reset not still connected to 0x5632a1988000 2024-04-06T09:57:26.775 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:57:26.772+0000 7f0965192640 -1 reset not still connected to 0x5632a1b73ee0 2024-04-06T09:57:26.775 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:57:26.772+0000 7f0965192640 -1 reset not still connected to 0x5632a2793520 2024-04-06T09:57:26.775 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:57:26.772+0000 7f0965192640 -1 reset not still connected to 0x5632a30f0270 2024-04-06T09:57:26.775 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:57:26.772+0000 7f0965192640 -1 reset not still connected to 0x5632a417e750 2024-04-06T09:57:26.775 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:57:26.772+0000 7f0965192640 -1 reset not still connected to 0x5632a4cda000 2024-04-06T09:57:26.783 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:57:26.776+0000 7f0965192640 -1 reset not still connected to 0x5632a4cdbba0 2024-04-06T09:57:26.783 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:57:26.776+0000 7f0965192640 -1 reset not still connected to 0x5632a5e9dee0 2024-04-06T09:57:26.784 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:57:26.784+0000 7f0965192640 -1 reset not still connected to 0x5632a89f4750 2024-04-06T09:57:26.785 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:57:26.784+0000 7f0965192640 -1 reset not still connected to 0x5632a89f52b0 2024-04-06T09:57:26.785 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:57:26.784+0000 7f0965192640 -1 reset not still connected to 0x5632abc812b0 2024-04-06T09:57:26.785 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:57:26.784+0000 7f0965192640 -1 reset not still connected to 0x5632abc81d40 2024-04-06T09:57:26.789 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:57:26.784+0000 7f0965192640 -1 reset not still connected to 0x5632abcbe820 2024-04-06T09:57:26.789 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:57:26.784+0000 7f0965192640 -1 reset not still connected to 0x5632abcbf930 2024-04-06T09:57:26.789 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:57:26.788+0000 7f0965192640 -1 reset not still connected to 0x5632b81f3a00 2024-04-06T09:57:26.789 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:57:26.788+0000 7f0965192640 -1 reset not still connected to 0x5632b9592000 2024-04-06T09:57:26.789 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:57:26.788+0000 7f0965192640 -1 reset not still connected to 0x5632b9593110 2024-04-06T09:57:26.789 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:57:26.788+0000 7f0965192640 -1 reset not still connected to 0x5632c0cfa000 2024-04-06T09:57:26.789 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:57:26.788+0000 7f0965192640 -1 reset not still connected to 0x5632c0cfba00 2024-04-06T09:57:26.789 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:57:26.788+0000 7f0965192640 -1 reset not still connected to 0x5632cb3ac000 2024-04-06T09:57:26.789 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:57:26.788+0000 7f0965192640 -1 reset not still connected to 0x5632d8f88820 2024-04-06T09:57:26.789 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:57:26.788+0000 7f0965192640 -1 reset not still connected to 0x5632dac08dd0 2024-04-06T09:57:26.790 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:57:26.788+0000 7f0965192640 -1 reset not still connected to 0x5632dac09040 2024-04-06T09:57:26.793 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:57:26.788+0000 7f0965192640 -1 reset not still connected to 0x5632e1684ea0 2024-04-06T09:57:26.793 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:57:26.788+0000 7f0965192640 -1 reset not still connected to 0x5632e16851e0 2024-04-06T09:57:29.033 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-04-06T09:57:29.032+0000 7fd1b6210640 -1 reset not still connected to 0x55ef9b657e10 2024-04-06T09:57:29.033 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-04-06T09:57:29.032+0000 7fd1b6210640 -1 reset not still connected to 0x55ef9bbe84e0 2024-04-06T09:57:29.033 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-04-06T09:57:29.032+0000 7fd1b6210640 -1 reset not still connected to 0x55ef9bbe8d00 2024-04-06T09:57:29.037 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-04-06T09:57:29.036+0000 7fd1b6210640 -1 reset not still connected to 0x55ef9bbe9ee0 2024-04-06T09:57:29.042 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-04-06T09:57:29.040+0000 7fd1b6210640 -1 reset not still connected to 0x55ef9bd5c9c0 2024-04-06T09:57:29.042 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-04-06T09:57:29.040+0000 7fd1b6210640 -1 reset not still connected to 0x55ef9bd5d520 2024-04-06T09:57:29.042 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-04-06T09:57:29.040+0000 7fd1b6210640 -1 reset not still connected to 0x55ef9bd5dd40 2024-04-06T09:57:29.042 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-04-06T09:57:29.040+0000 7fd1b6210640 -1 reset not still connected to 0x55ef9bd86820 2024-04-06T09:57:29.042 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-04-06T09:57:29.040+0000 7fd1b6210640 -1 reset not still connected to 0x55ef9c103110 2024-04-06T09:57:29.045 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-04-06T09:57:29.044+0000 7fd1b6210640 -1 reset not still connected to 0x55ef9c103930 2024-04-06T09:57:29.045 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-04-06T09:57:29.044+0000 7fd1b6210640 -1 reset not still connected to 0x55ef9c103d40 2024-04-06T09:57:29.045 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-04-06T09:57:29.044+0000 7fd1b6210640 -1 reset not still connected to 0x55ef9e698c30 2024-04-06T09:57:29.046 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-04-06T09:57:29.044+0000 7fd1b6210640 -1 reset not still connected to 0x55ef9e88f520 2024-04-06T09:57:29.046 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-04-06T09:57:29.044+0000 7fd1b6210640 -1 reset not still connected to 0x55ef9eee21a0 2024-04-06T09:57:29.046 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-04-06T09:57:29.044+0000 7fd1b6210640 -1 reset not still connected to 0x55ef9eee3a00 2024-04-06T09:57:29.046 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-04-06T09:57:29.044+0000 7fd1b6210640 -1 reset not still connected to 0x55efa18cc270 2024-04-06T09:57:29.046 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-04-06T09:57:29.044+0000 7fd1b6210640 -1 reset not still connected to 0x55efa18cca90 2024-04-06T09:57:29.046 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-04-06T09:57:29.044+0000 7fd1b6210640 -1 reset not still connected to 0x55efbb630d00 2024-04-06T09:57:29.047 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-04-06T09:57:29.044+0000 7fd1b6210640 -1 reset not still connected to 0x55efd81ee8f0 2024-04-06T09:57:29.047 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-04-06T09:57:29.044+0000 7fd1b6210640 -1 reset not still connected to 0x55efe06a9860 2024-04-06T09:57:29.397 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:100 IOs, 78555136 bytes 2024-04-06T09:57:29.643 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.0 with skip (0.6347705174918779) > weight (0.25) 2024-04-06T09:57:29.643 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.1 with skip (0.9689010164827763) > weight (0.25) 2024-04-06T09:57:29.643 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.2 with skip (0.5312561890332567) > weight (0.25) 2024-04-06T09:57:29.643 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.3 with skip (0.3476478001415856) > weight (0.25) 2024-04-06T09:57:29.643 INFO:tasks.rbd_mirror_thrash:waiting for 102 secs before thrashing 2024-04-06T09:57:30.090 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-04-06T09:57:30.084+0000 7fd1b6210640 -1 reset not still connected to 0x55ef9eee3a00 2024-04-06T09:57:30.090 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-04-06T09:57:30.084+0000 7fd1b6210640 -1 reset not still connected to 0x55efa18cc270 2024-04-06T09:57:30.090 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-04-06T09:57:30.084+0000 7fd1b6210640 -1 reset not still connected to 0x55efa18cc9c0 2024-04-06T09:57:30.090 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-04-06T09:57:30.084+0000 7fd1b6210640 -1 reset not still connected to 0x55efa18cca90 2024-04-06T09:57:30.090 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-04-06T09:57:30.084+0000 7fd1b6210640 -1 reset not still connected to 0x55efbb630d00 2024-04-06T09:57:30.090 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-04-06T09:57:30.084+0000 7fd1b6210640 -1 reset not still connected to 0x55efd81ee8f0 2024-04-06T09:57:30.090 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-04-06T09:57:30.084+0000 7fd1b6210640 -1 reset not still connected to 0x55efe06a9860 2024-04-06T09:57:31.480 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ error_code=124 2024-04-06T09:57:31.480 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set -e 2024-04-06T09:57:31.480 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 124 -eq 124 ']' 2024-04-06T09:57:31.480 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:57:31.480 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-06T09:57:31.480 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ image=image_46 2024-04-06T09:57:31.480 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ create_image_and_enable_mirror cluster2 mirror image_46 journal 128M 2024-04-06T09:57:31.480 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-04-06T09:57:31.480 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:57:31.480 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:57:31.481 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:57:31.481 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_46 2024-04-06T09:57:31.481 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:57:31.481 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local mode=journal 2024-04-06T09:57:31.481 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n journal ']' 2024-04-06T09:57:31.481 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:57:31.481 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ create_image cluster2 mirror image_46 128M 2024-04-06T09:57:31.481 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-04-06T09:57:31.481 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:57:31.481 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:57:31.481 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:57:31.481 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_46 2024-04-06T09:57:31.481 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:57:31.481 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local size=128 2024-04-06T09:57:31.482 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n 128M ']' 2024-04-06T09:57:31.482 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ size=128M 2024-04-06T09:57:31.482 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:57:31.482 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,journaling mirror/image_46 2024-04-06T09:57:31.586 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' pool = image ']' 2024-04-06T09:57:31.586 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' mirror = mirror_parent ']' 2024-04-06T09:57:31.586 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n '' ']' 2024-04-06T09:57:31.586 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ stress_write_image cluster2 mirror image_46 2024-04-06T09:57:31.586 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-04-06T09:57:31.586 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:57:31.586 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_46 2024-04-06T09:57:31.586 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-04-06T09:57:31.587 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local duration=6 2024-04-06T09:57:31.587 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set +e 2024-04-06T09:57:31.587 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ timeout 6s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_46 --debug-rbd=20 --debug-journaler=20 2024-04-06T09:57:33.139 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:57:33.136+0000 7f2e84d66640 -1 reset not still connected to 0x560d3620bee0 2024-04-06T09:57:33.139 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:57:33.136+0000 7f2e84d66640 -1 reset not still connected to 0x560d368e4f70 2024-04-06T09:57:33.139 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:57:33.136+0000 7f2e84d66640 -1 reset not still connected to 0x560d36912ea0 2024-04-06T09:57:33.140 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:57:33.136+0000 7f2e84d66640 -1 reset not still connected to 0x560d381dd5f0 2024-04-06T09:57:33.140 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:57:33.136+0000 7f2e84d66640 -1 reset not still connected to 0x560d47e9aa90 2024-04-06T09:57:33.140 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:57:33.136+0000 7f2e84d66640 -1 reset not still connected to 0x560d4bc7e8f0 2024-04-06T09:57:33.140 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:57:33.136+0000 7f2e84d66640 -1 reset not still connected to 0x560d50adba00 2024-04-06T09:57:33.469 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:100 IOs, 24296448 bytes 2024-04-06T09:57:34.196 INFO:tasks.ceph.cluster2.osd.1.smithi192.stderr:2024-04-06T09:57:34.194+0000 7f0de0719640 -1 reset not still connected to 0x55bfa3e74f70 2024-04-06T09:57:34.196 INFO:tasks.ceph.cluster2.osd.1.smithi192.stderr:2024-04-06T09:57:34.194+0000 7f0de0719640 -1 reset not still connected to 0x55bfa468d450 2024-04-06T09:57:34.196 INFO:tasks.ceph.cluster2.osd.1.smithi192.stderr:2024-04-06T09:57:34.194+0000 7f0de0719640 -1 reset not still connected to 0x55bfa4b36340 2024-04-06T09:57:34.197 INFO:tasks.ceph.cluster2.osd.1.smithi192.stderr:2024-04-06T09:57:34.198+0000 7f0de0719640 -1 reset not still connected to 0x55bfa5642b60 2024-04-06T09:57:34.197 INFO:tasks.ceph.cluster2.osd.1.smithi192.stderr:2024-04-06T09:57:34.198+0000 7f0de0719640 -1 reset not still connected to 0x55bfa5642c30 2024-04-06T09:57:34.198 INFO:tasks.ceph.cluster2.osd.1.smithi192.stderr:2024-04-06T09:57:34.198+0000 7f0de0719640 -1 reset not still connected to 0x55bfa6e48410 2024-04-06T09:57:34.199 INFO:tasks.ceph.cluster2.osd.1.smithi192.stderr:2024-04-06T09:57:34.198+0000 7f0de0719640 -1 reset not still connected to 0x55bfa8bcf6c0 2024-04-06T09:57:34.199 INFO:tasks.ceph.cluster2.osd.1.smithi192.stderr:2024-04-06T09:57:34.198+0000 7f0de0719640 -1 reset not still connected to 0x55bfaef3ba00 2024-04-06T09:57:34.199 INFO:tasks.ceph.cluster2.osd.1.smithi192.stderr:2024-04-06T09:57:34.198+0000 7f0de0719640 -1 reset not still connected to 0x55bfaef3bc70 2024-04-06T09:57:34.199 INFO:tasks.ceph.cluster2.osd.1.smithi192.stderr:2024-04-06T09:57:34.198+0000 7f0de0719640 -1 reset not still connected to 0x55bfb5cb6000 2024-04-06T09:57:34.200 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:57:34.200+0000 7f2e84d66640 -1 reset not still connected to 0x560d27567380 2024-04-06T09:57:34.200 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:57:34.200+0000 7f2e84d66640 -1 reset not still connected to 0x560d27839380 2024-04-06T09:57:34.200 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:57:34.200+0000 7f2e84d66640 -1 reset not still connected to 0x560d27839930 2024-04-06T09:57:34.200 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:57:34.200+0000 7f2e84d66640 -1 reset not still connected to 0x560d27c3edd0 2024-04-06T09:57:34.202 INFO:tasks.ceph.cluster2.osd.1.smithi192.stderr:2024-04-06T09:57:34.202+0000 7f0de0719640 -1 reset not still connected to 0x55bfbd8f9d40 2024-04-06T09:57:34.202 INFO:tasks.ceph.cluster2.osd.1.smithi192.stderr:2024-04-06T09:57:34.202+0000 7f0de0719640 -1 reset not still connected to 0x55bfd74fb450 2024-04-06T09:57:34.204 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:57:34.200+0000 7f2e84d66640 -1 reset not still connected to 0x560d29cd8f70 2024-04-06T09:57:34.204 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:57:34.200+0000 7f2e84d66640 -1 reset not still connected to 0x560d29cd9ba0 2024-04-06T09:57:34.204 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:57:34.200+0000 7f2e84d66640 -1 reset not still connected to 0x560d29d5b520 2024-04-06T09:57:34.204 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:57:34.200+0000 7f2e84d66640 -1 reset not still connected to 0x560d2a17d450 2024-04-06T09:57:34.204 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:57:34.200+0000 7f2e84d66640 -1 reset not still connected to 0x560d2a17d6c0 2024-04-06T09:57:34.204 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:57:34.200+0000 7f2e84d66640 -1 reset not still connected to 0x560d2a204b60 2024-04-06T09:57:34.204 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:57:34.200+0000 7f2e84d66640 -1 reset not still connected to 0x560d2a205ba0 2024-04-06T09:57:34.204 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:57:34.200+0000 7f2e84d66640 -1 reset not still connected to 0x560d2a2336c0 2024-04-06T09:57:34.204 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:57:34.200+0000 7f2e84d66640 -1 reset not still connected to 0x560d2a5705b0 2024-04-06T09:57:34.210 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:57:34.208+0000 7f2e84d66640 -1 reset not still connected to 0x560d35349380 2024-04-06T09:57:34.210 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:57:34.208+0000 7f2e84d66640 -1 reset not still connected to 0x560d3620b520 2024-04-06T09:57:34.212 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:57:34.212+0000 7f2e84d66640 -1 reset not still connected to 0x560d36666d00 2024-04-06T09:57:34.217 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:57:34.216+0000 7f2e84d66640 -1 reset not still connected to 0x560d368e4f70 2024-04-06T09:57:34.217 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:57:34.216+0000 7f2e84d66640 -1 reset not still connected to 0x560d36912ea0 2024-04-06T09:57:34.219 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:57:34.216+0000 7f2e84d66640 -1 reset not still connected to 0x560d381dcc30 2024-04-06T09:57:34.219 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:57:34.216+0000 7f2e84d66640 -1 reset not still connected to 0x560d381dd5f0 2024-04-06T09:57:34.219 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:57:34.216+0000 7f2e84d66640 -1 reset not still connected to 0x560d390dca90 2024-04-06T09:57:34.219 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:57:34.216+0000 7f2e84d66640 -1 reset not still connected to 0x560d4bc7e8f0 2024-04-06T09:57:34.219 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:57:34.216+0000 7f2e84d66640 -1 reset not still connected to 0x560d4bc7ec30 2024-04-06T09:57:34.226 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:57:34.224+0000 7f2e84d66640 -1 reset not still connected to 0x560d50adba00 2024-04-06T09:57:34.226 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:57:34.224+0000 7f2e84d66640 -1 reset not still connected to 0x560d27073860 2024-04-06T09:57:34.226 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:57:34.224+0000 7f2e84d66640 -1 reset not still connected to 0x560d27567380 2024-04-06T09:57:34.226 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:57:34.224+0000 7f2e84d66640 -1 reset not still connected to 0x560d29d5b520 2024-04-06T09:57:34.236 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:57:34.232+0000 7f2e84d66640 -1 reset not still connected to 0x560d2a17d450 2024-04-06T09:57:34.236 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:57:34.232+0000 7f2e84d66640 -1 reset not still connected to 0x560d2a17d6c0 2024-04-06T09:57:34.236 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:57:34.236+0000 7f2e84d66640 -1 reset not still connected to 0x560d25ab2680 2024-04-06T09:57:34.238 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:57:34.236+0000 7f2e84d66640 -1 reset not still connected to 0x560d27072820 2024-04-06T09:57:34.238 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:57:34.236+0000 7f2e84d66640 -1 reset not still connected to 0x560d27073860 2024-04-06T09:57:34.242 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:57:34.240+0000 7f2e84d66640 -1 reset not still connected to 0x560d27566ea0 2024-04-06T09:57:34.242 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:57:34.240+0000 7f2e84d66640 -1 reset not still connected to 0x560d27567380 2024-04-06T09:57:34.242 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:57:34.240+0000 7f2e84d66640 -1 reset not still connected to 0x560d27839380 2024-04-06T09:57:34.242 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:57:34.240+0000 7f2e84d66640 -1 reset not still connected to 0x560d27839930 2024-04-06T09:57:34.242 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:57:34.240+0000 7f2e84d66640 -1 reset not still connected to 0x560d27c3edd0 2024-04-06T09:57:34.245 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:57:34.244+0000 7f2e84d66640 -1 reset not still connected to 0x560d27c3f2b0 2024-04-06T09:57:34.246 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:57:34.244+0000 7f2e84d66640 -1 reset not still connected to 0x560d281875f0 2024-04-06T09:57:34.246 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:57:34.244+0000 7f2e84d66640 -1 reset not still connected to 0x560d294908f0 2024-04-06T09:57:34.249 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:57:34.248+0000 7f2e84d66640 -1 reset not still connected to 0x560d29cd8f70 2024-04-06T09:57:34.249 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:57:34.248+0000 7f2e84d66640 -1 reset not still connected to 0x560d29cd9ba0 2024-04-06T09:57:34.249 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:57:34.248+0000 7f2e84d66640 -1 reset not still connected to 0x560d29d5a340 2024-04-06T09:57:34.249 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:57:34.248+0000 7f2e84d66640 -1 reset not still connected to 0x560d29d5b520 2024-04-06T09:57:34.249 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:57:34.248+0000 7f2e84d66640 -1 reset not still connected to 0x560d2a17d450 2024-04-06T09:57:34.249 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:57:34.248+0000 7f2e84d66640 -1 reset not still connected to 0x560d2a17d6c0 2024-04-06T09:57:34.249 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:57:34.248+0000 7f2e84d66640 -1 reset not still connected to 0x560d2a204b60 2024-04-06T09:57:34.249 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:57:34.248+0000 7f2e84d66640 -1 reset not still connected to 0x560d2a205ba0 2024-04-06T09:57:34.249 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:57:34.248+0000 7f2e84d66640 -1 reset not still connected to 0x560d2a2336c0 2024-04-06T09:57:34.249 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:57:34.248+0000 7f2e84d66640 -1 reset not still connected to 0x560d2a570000 2024-04-06T09:57:34.249 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:57:34.248+0000 7f2e84d66640 -1 reset not still connected to 0x560d2a5705b0 2024-04-06T09:57:34.249 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:57:34.248+0000 7f2e84d66640 -1 reset not still connected to 0x560d3620b520 2024-04-06T09:57:34.249 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:57:34.248+0000 7f2e84d66640 -1 reset not still connected to 0x560d3620bee0 2024-04-06T09:57:34.250 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:57:34.248+0000 7f2e84d66640 -1 reset not still connected to 0x560d368e4f70 2024-04-06T09:57:34.250 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:57:34.248+0000 7f2e84d66640 -1 reset not still connected to 0x560d36912ea0 2024-04-06T09:57:34.251 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:57:34.248+0000 7f2e84d66640 -1 reset not still connected to 0x560d381dcc30 2024-04-06T09:57:34.251 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:57:34.248+0000 7f2e84d66640 -1 reset not still connected to 0x560d381dd5f0 2024-04-06T09:57:34.251 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:57:34.248+0000 7f2e84d66640 -1 reset not still connected to 0x560d390dca90 2024-04-06T09:57:34.251 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:57:34.248+0000 7f2e84d66640 -1 reset not still connected to 0x560d4bc7e8f0 2024-04-06T09:57:34.254 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:57:34.252+0000 7f2e84d66640 -1 reset not still connected to 0x560d50adba00 2024-04-06T09:57:34.803 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:200 IOs, 27561984 bytes 2024-04-06T09:57:34.926 INFO:tasks.ceph.cluster2.osd.2.smithi192.stderr:2024-04-06T09:57:34.926+0000 7fa534aba640 -1 reset not still connected to 0x559338f41040 2024-04-06T09:57:34.926 INFO:tasks.ceph.cluster2.osd.2.smithi192.stderr:2024-04-06T09:57:34.926+0000 7fa534aba640 -1 reset not still connected to 0x559338f415f0 2024-04-06T09:57:34.926 INFO:tasks.ceph.cluster2.osd.2.smithi192.stderr:2024-04-06T09:57:34.926+0000 7fa534aba640 -1 reset not still connected to 0x55933ce91110 2024-04-06T09:57:34.926 INFO:tasks.ceph.cluster2.osd.2.smithi192.stderr:2024-04-06T09:57:34.926+0000 7fa534aba640 -1 reset not still connected to 0x55933ecf01a0 2024-04-06T09:57:34.927 INFO:tasks.ceph.cluster2.osd.2.smithi192.stderr:2024-04-06T09:57:34.926+0000 7fa534aba640 -1 reset not still connected to 0x55933fc2dd40 2024-04-06T09:57:34.927 INFO:tasks.ceph.cluster2.osd.2.smithi192.stderr:2024-04-06T09:57:34.926+0000 7fa534aba640 -1 reset not still connected to 0x559341658b60 2024-04-06T09:57:34.927 INFO:tasks.ceph.cluster2.osd.2.smithi192.stderr:2024-04-06T09:57:34.926+0000 7fa534aba640 -1 reset not still connected to 0x55934811f520 2024-04-06T09:57:34.927 INFO:tasks.ceph.cluster2.osd.2.smithi192.stderr:2024-04-06T09:57:34.926+0000 7fa534aba640 -1 reset not still connected to 0x55934811f6c0 2024-04-06T09:57:34.927 INFO:tasks.ceph.cluster2.osd.2.smithi192.stderr:2024-04-06T09:57:34.926+0000 7fa534aba640 -1 reset not still connected to 0x559348f02820 2024-04-06T09:57:34.927 INFO:tasks.ceph.cluster2.osd.2.smithi192.stderr:2024-04-06T09:57:34.926+0000 7fa534aba640 -1 reset not still connected to 0x5593550955f0 2024-04-06T09:57:34.927 INFO:tasks.ceph.cluster2.osd.2.smithi192.stderr:2024-04-06T09:57:34.926+0000 7fa534aba640 -1 reset not still connected to 0x55935bfc9d40 2024-04-06T09:57:34.927 INFO:tasks.ceph.cluster2.osd.2.smithi192.stderr:2024-04-06T09:57:34.926+0000 7fa534aba640 -1 reset not still connected to 0x55936c6f44e0 2024-04-06T09:57:35.086 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi192.stderr:2024-04-06T09:57:35.086+0000 7f3d293cf640 -1 rbd::mirror::InstanceWatcher: C_NotifyInstanceRequest: 0x56067b284cc0 finish: resending after timeout 2024-04-06T09:57:35.086 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi192.stderr:2024-04-06T09:57:35.086+0000 7f3d28bce640 -1 rbd::mirror::InstanceWatcher: C_NotifyInstanceRequest: 0x560668d738c0 finish: resending after timeout 2024-04-06T09:57:35.088 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi192.stderr:2024-04-06T09:57:35.090+0000 7f3d293cf640 -1 rbd::mirror::InstanceWatcher: C_NotifyInstanceRequest: 0x560663838600 finish: resending after timeout 2024-04-06T09:57:35.941 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:57:35.940+0000 7f2e84d66640 -1 reset not still connected to 0x560d2722e5b0 2024-04-06T09:57:35.942 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:57:35.940+0000 7f2e84d66640 -1 reset not still connected to 0x560d27566ea0 2024-04-06T09:57:35.942 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:57:35.940+0000 7f2e84d66640 -1 reset not still connected to 0x560d27567380 2024-04-06T09:57:35.942 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:57:35.940+0000 7f2e84d66640 -1 reset not still connected to 0x560d27839380 2024-04-06T09:57:35.942 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:57:35.940+0000 7f2e84d66640 -1 reset not still connected to 0x560d27839930 2024-04-06T09:57:35.942 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:57:35.940+0000 7f2e84d66640 -1 reset not still connected to 0x560d27c3edd0 2024-04-06T09:57:35.942 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:57:35.940+0000 7f2e84d66640 -1 reset not still connected to 0x560d27c3f2b0 2024-04-06T09:57:35.942 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:57:35.940+0000 7f2e84d66640 -1 reset not still connected to 0x560d281875f0 2024-04-06T09:57:35.942 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:57:35.940+0000 7f2e84d66640 -1 reset not still connected to 0x560d294908f0 2024-04-06T09:57:35.943 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:57:35.940+0000 7f2e84d66640 -1 reset not still connected to 0x560d294916c0 2024-04-06T09:57:35.943 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:57:35.940+0000 7f2e84d66640 -1 reset not still connected to 0x560d29cd8f70 2024-04-06T09:57:35.943 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:57:35.940+0000 7f2e84d66640 -1 reset not still connected to 0x560d29cd9ba0 2024-04-06T09:57:35.947 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:57:35.944+0000 7f2e84d66640 -1 reset not still connected to 0x560d29d5a340 2024-04-06T09:57:35.947 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:57:35.944+0000 7f2e84d66640 -1 reset not still connected to 0x560d29d5b520 2024-04-06T09:57:35.947 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:57:35.944+0000 7f2e84d66640 -1 reset not still connected to 0x560d2a17d450 2024-04-06T09:57:35.947 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:57:35.944+0000 7f2e84d66640 -1 reset not still connected to 0x560d2a17d6c0 2024-04-06T09:57:35.947 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:57:35.944+0000 7f2e84d66640 -1 reset not still connected to 0x560d2a204b60 2024-04-06T09:57:35.947 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:57:35.944+0000 7f2e84d66640 -1 reset not still connected to 0x560d2a205ba0 2024-04-06T09:57:35.947 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:57:35.944+0000 7f2e84d66640 -1 reset not still connected to 0x560d2a2336c0 2024-04-06T09:57:35.947 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:57:35.944+0000 7f2e84d66640 -1 reset not still connected to 0x560d2a570000 2024-04-06T09:57:35.947 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:57:35.944+0000 7f2e84d66640 -1 reset not still connected to 0x560d2a5705b0 2024-04-06T09:57:35.947 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:57:35.944+0000 7f2e84d66640 -1 reset not still connected to 0x560d35349380 2024-04-06T09:57:35.947 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:57:35.944+0000 7f2e84d66640 -1 reset not still connected to 0x560d3620b520 2024-04-06T09:57:35.947 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:57:35.944+0000 7f2e84d66640 -1 reset not still connected to 0x560d3620bee0 2024-04-06T09:57:35.947 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:57:35.944+0000 7f2e84d66640 -1 reset not still connected to 0x560d36666d00 2024-04-06T09:57:35.947 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:57:35.944+0000 7f2e84d66640 -1 reset not still connected to 0x560d368e4f70 2024-04-06T09:57:35.948 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:57:35.944+0000 7f2e84d66640 -1 reset not still connected to 0x560d36912ea0 2024-04-06T09:57:35.949 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:57:35.948+0000 7f2e84d66640 -1 reset not still connected to 0x560d381dcc30 2024-04-06T09:57:35.949 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:57:35.948+0000 7f2e84d66640 -1 reset not still connected to 0x560d381dd5f0 2024-04-06T09:57:35.949 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:57:35.948+0000 7f2e84d66640 -1 reset not still connected to 0x560d390dc8f0 2024-04-06T09:57:35.949 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:57:35.948+0000 7f2e84d66640 -1 reset not still connected to 0x560d390dca90 2024-04-06T09:57:35.949 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:57:35.948+0000 7f2e84d66640 -1 reset not still connected to 0x560d47e9aa90 2024-04-06T09:57:35.949 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:57:35.948+0000 7f2e84d66640 -1 reset not still connected to 0x560d4bc7e8f0 2024-04-06T09:57:35.949 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:57:35.948+0000 7f2e84d66640 -1 reset not still connected to 0x560d4bc7ec30 2024-04-06T09:57:35.950 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:57:35.948+0000 7f2e84d66640 -1 reset not still connected to 0x560d50adba00 2024-04-06T09:57:37.464 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:57:37.464+0000 7f2e84d66640 -1 reset not still connected to 0x560d25ab2680 2024-04-06T09:57:37.465 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:57:37.464+0000 7f2e84d66640 -1 reset not still connected to 0x560d27072820 2024-04-06T09:57:37.465 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:57:37.464+0000 7f2e84d66640 -1 reset not still connected to 0x560d27073860 2024-04-06T09:57:37.471 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:57:37.464+0000 7f2e84d66640 -1 reset not still connected to 0x560d2722e5b0 2024-04-06T09:57:37.471 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:57:37.464+0000 7f2e84d66640 -1 reset not still connected to 0x560d27567380 2024-04-06T09:57:37.476 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:57:37.468+0000 7f2e84d66640 -1 reset not still connected to 0x560d27839380 2024-04-06T09:57:37.476 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:57:37.468+0000 7f2e84d66640 -1 reset not still connected to 0x560d27839930 2024-04-06T09:57:37.477 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:57:37.472+0000 7f2e84d66640 -1 reset not still connected to 0x560d27c3edd0 2024-04-06T09:57:37.477 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:57:37.472+0000 7f2e84d66640 -1 reset not still connected to 0x560d27c3f2b0 2024-04-06T09:57:37.477 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:57:37.472+0000 7f2e84d66640 -1 reset not still connected to 0x560d281875f0 2024-04-06T09:57:37.477 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:57:37.472+0000 7f2e84d66640 -1 reset not still connected to 0x560d294908f0 2024-04-06T09:57:37.477 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:57:37.472+0000 7f2e84d66640 -1 reset not still connected to 0x560d294916c0 2024-04-06T09:57:37.477 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:57:37.472+0000 7f2e84d66640 -1 reset not still connected to 0x560d29cd8f70 2024-04-06T09:57:37.477 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:57:37.472+0000 7f2e84d66640 -1 reset not still connected to 0x560d29cd9ba0 2024-04-06T09:57:37.477 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:57:37.472+0000 7f2e84d66640 -1 reset not still connected to 0x560d29d5a340 2024-04-06T09:57:37.477 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:57:37.472+0000 7f2e84d66640 -1 reset not still connected to 0x560d29d5b520 2024-04-06T09:57:37.477 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:57:37.472+0000 7f2e84d66640 -1 reset not still connected to 0x560d2a17d450 2024-04-06T09:57:37.477 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:57:37.472+0000 7f2e84d66640 -1 reset not still connected to 0x560d2a17d6c0 2024-04-06T09:57:37.477 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:57:37.472+0000 7f2e84d66640 -1 reset not still connected to 0x560d2a204b60 2024-04-06T09:57:37.477 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:57:37.472+0000 7f2e84d66640 -1 reset not still connected to 0x560d2a205ba0 2024-04-06T09:57:37.477 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:57:37.472+0000 7f2e84d66640 -1 reset not still connected to 0x560d2a2336c0 2024-04-06T09:57:37.478 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:57:37.472+0000 7f2e84d66640 -1 reset not still connected to 0x560d2a570000 2024-04-06T09:57:37.478 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:57:37.472+0000 7f2e84d66640 -1 reset not still connected to 0x560d2a5705b0 2024-04-06T09:57:37.478 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:57:37.472+0000 7f2e84d66640 -1 reset not still connected to 0x560d35349380 2024-04-06T09:57:37.478 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:57:37.472+0000 7f2e84d66640 -1 reset not still connected to 0x560d3620b520 2024-04-06T09:57:37.478 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:57:37.472+0000 7f2e84d66640 -1 reset not still connected to 0x560d3620bee0 2024-04-06T09:57:37.478 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:57:37.472+0000 7f2e84d66640 -1 reset not still connected to 0x560d36666d00 2024-04-06T09:57:37.478 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:57:37.472+0000 7f2e84d66640 -1 reset not still connected to 0x560d368e4f70 2024-04-06T09:57:37.478 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:57:37.472+0000 7f2e84d66640 -1 reset not still connected to 0x560d36912ea0 2024-04-06T09:57:37.486 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:57:37.484+0000 7f2e84d66640 -1 reset not still connected to 0x560d381dcc30 2024-04-06T09:57:37.486 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:57:37.484+0000 7f2e84d66640 -1 reset not still connected to 0x560d381dd5f0 2024-04-06T09:57:37.486 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:57:37.484+0000 7f2e84d66640 -1 reset not still connected to 0x560d390dc8f0 2024-04-06T09:57:37.486 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:57:37.484+0000 7f2e84d66640 -1 reset not still connected to 0x560d390dca90 2024-04-06T09:57:37.486 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:57:37.484+0000 7f2e84d66640 -1 reset not still connected to 0x560d47e9aa90 2024-04-06T09:57:37.486 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:57:37.484+0000 7f2e84d66640 -1 reset not still connected to 0x560d4bc7e8f0 2024-04-06T09:57:37.486 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:57:37.484+0000 7f2e84d66640 -1 reset not still connected to 0x560d4bc7ec30 2024-04-06T09:57:37.487 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:57:37.484+0000 7f2e84d66640 -1 reset not still connected to 0x560d50adba00 2024-04-06T09:57:37.596 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ error_code=124 2024-04-06T09:57:37.596 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set -e 2024-04-06T09:57:37.597 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 124 -eq 124 ']' 2024-04-06T09:57:37.597 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:57:37.597 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-06T09:57:37.597 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ image=image_47 2024-04-06T09:57:37.597 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ create_image_and_enable_mirror cluster2 mirror image_47 journal 128M 2024-04-06T09:57:37.597 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-04-06T09:57:37.597 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:57:37.597 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:57:37.597 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:57:37.597 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_47 2024-04-06T09:57:37.597 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:57:37.597 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local mode=journal 2024-04-06T09:57:37.597 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n journal ']' 2024-04-06T09:57:37.597 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:57:37.598 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ create_image cluster2 mirror image_47 128M 2024-04-06T09:57:37.598 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-04-06T09:57:37.598 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:57:37.598 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:57:37.598 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:57:37.598 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_47 2024-04-06T09:57:37.598 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:57:37.598 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local size=128 2024-04-06T09:57:37.598 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n 128M ']' 2024-04-06T09:57:37.598 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ size=128M 2024-04-06T09:57:37.598 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:57:37.598 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,journaling mirror/image_47 2024-04-06T09:57:37.837 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' pool = image ']' 2024-04-06T09:57:37.837 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' mirror = mirror_parent ']' 2024-04-06T09:57:37.837 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n '' ']' 2024-04-06T09:57:37.837 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ stress_write_image cluster2 mirror image_47 2024-04-06T09:57:37.837 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-04-06T09:57:37.837 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:57:37.837 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_47 2024-04-06T09:57:37.837 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-04-06T09:57:37.838 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local duration=11 2024-04-06T09:57:37.838 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set +e 2024-04-06T09:57:37.838 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ timeout 11s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_47 --debug-rbd=20 --debug-journaler=20 2024-04-06T09:57:38.746 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi192.stderr:2024-04-06T09:57:38.746+0000 7f3d28bce640 -1 rbd::mirror::InstanceWatcher: C_NotifyInstanceRequest: 0x560663839800 finish: resending after timeout 2024-04-06T09:57:38.823 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-04-06T09:57:38.812+0000 7fd1b6210640 -1 reset not still connected to 0x55ef99b89ad0 2024-04-06T09:57:38.824 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-04-06T09:57:38.820+0000 7fd1b6210640 -1 reset not still connected to 0x55ef99ebf6c0 2024-04-06T09:57:38.824 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-04-06T09:57:38.820+0000 7fd1b6210640 -1 reset not still connected to 0x55ef9ac60270 2024-04-06T09:57:38.824 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-04-06T09:57:38.820+0000 7fd1b6210640 -1 reset not still connected to 0x55ef9bbe84e0 2024-04-06T09:57:38.825 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-04-06T09:57:38.824+0000 7fd1b6210640 -1 reset not still connected to 0x55ef9bbe9ee0 2024-04-06T09:57:38.825 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-04-06T09:57:38.824+0000 7fd1b6210640 -1 reset not still connected to 0x55ef9bd5c9c0 2024-04-06T09:57:38.825 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-04-06T09:57:38.824+0000 7fd1b6210640 -1 reset not still connected to 0x55ef9bd5d520 2024-04-06T09:57:38.825 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-04-06T09:57:38.824+0000 7fd1b6210640 -1 reset not still connected to 0x55ef9bd5dd40 2024-04-06T09:57:38.833 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-04-06T09:57:38.832+0000 7fd1b6210640 -1 reset not still connected to 0x55ef9bd86820 2024-04-06T09:57:38.833 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-04-06T09:57:38.832+0000 7fd1b6210640 -1 reset not still connected to 0x55ef9c103110 2024-04-06T09:57:38.833 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-04-06T09:57:38.832+0000 7fd1b6210640 -1 reset not still connected to 0x55ef9c103930 2024-04-06T09:57:38.833 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-04-06T09:57:38.832+0000 7fd1b6210640 -1 reset not still connected to 0x55ef9c103d40 2024-04-06T09:57:38.833 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-04-06T09:57:38.832+0000 7fd1b6210640 -1 reset not still connected to 0x55ef9e698c30 2024-04-06T09:57:38.833 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-04-06T09:57:38.832+0000 7fd1b6210640 -1 reset not still connected to 0x55ef9e88f520 2024-04-06T09:57:38.833 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-04-06T09:57:38.832+0000 7fd1b6210640 -1 reset not still connected to 0x55ef9eee21a0 2024-04-06T09:57:38.833 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-04-06T09:57:38.832+0000 7fd1b6210640 -1 reset not still connected to 0x55ef9eee3a00 2024-04-06T09:57:38.833 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-04-06T09:57:38.832+0000 7fd1b6210640 -1 reset not still connected to 0x55efa18cc270 2024-04-06T09:57:38.833 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-04-06T09:57:38.832+0000 7fd1b6210640 -1 reset not still connected to 0x55efa18cc9c0 2024-04-06T09:57:38.834 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-04-06T09:57:38.832+0000 7fd1b6210640 -1 reset not still connected to 0x55efa18cca90 2024-04-06T09:57:38.835 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-04-06T09:57:38.832+0000 7fd1b6210640 -1 reset not still connected to 0x55efae5ffe10 2024-04-06T09:57:38.835 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-04-06T09:57:38.832+0000 7fd1b6210640 -1 reset not still connected to 0x55efbb630d00 2024-04-06T09:57:38.835 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-04-06T09:57:38.832+0000 7fd1b6210640 -1 reset not still connected to 0x55efd81ee8f0 2024-04-06T09:57:38.835 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-04-06T09:57:38.836+0000 7fd1b6210640 -1 reset not still connected to 0x55efe06a9860 2024-04-06T09:57:39.122 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:57:39.120+0000 7f2e84d66640 -1 reset not still connected to 0x560d27072820 2024-04-06T09:57:39.123 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:57:39.120+0000 7f2e84d66640 -1 reset not still connected to 0x560d27073860 2024-04-06T09:57:39.123 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:57:39.124+0000 7f2e84d66640 -1 reset not still connected to 0x560d2722e5b0 2024-04-06T09:57:39.126 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:57:39.124+0000 7f2e84d66640 -1 reset not still connected to 0x560d27566ea0 2024-04-06T09:57:39.126 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:57:39.124+0000 7f2e84d66640 -1 reset not still connected to 0x560d27567380 2024-04-06T09:57:39.126 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:57:39.124+0000 7f2e84d66640 -1 reset not still connected to 0x560d27839380 2024-04-06T09:57:39.126 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:57:39.124+0000 7f2e84d66640 -1 reset not still connected to 0x560d27839930 2024-04-06T09:57:39.126 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:57:39.124+0000 7f2e84d66640 -1 reset not still connected to 0x560d27c3edd0 2024-04-06T09:57:39.126 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:57:39.124+0000 7f2e84d66640 -1 reset not still connected to 0x560d27c3f2b0 2024-04-06T09:57:39.126 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:57:39.124+0000 7f2e84d66640 -1 reset not still connected to 0x560d281875f0 2024-04-06T09:57:39.126 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:57:39.124+0000 7f2e84d66640 -1 reset not still connected to 0x560d294908f0 2024-04-06T09:57:39.130 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:57:39.124+0000 7f2e84d66640 -1 reset not still connected to 0x560d29cd8f70 2024-04-06T09:57:39.130 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:57:39.124+0000 7f2e84d66640 -1 reset not still connected to 0x560d29cd9ba0 2024-04-06T09:57:39.130 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:57:39.128+0000 7f2e84d66640 -1 reset not still connected to 0x560d29d5a340 2024-04-06T09:57:39.130 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:57:39.128+0000 7f2e84d66640 -1 reset not still connected to 0x560d29d5b520 2024-04-06T09:57:39.130 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:57:39.128+0000 7f2e84d66640 -1 reset not still connected to 0x560d2a17d450 2024-04-06T09:57:39.130 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:57:39.128+0000 7f2e84d66640 -1 reset not still connected to 0x560d2a17d6c0 2024-04-06T09:57:39.131 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:57:39.128+0000 7f2e84d66640 -1 reset not still connected to 0x560d2a204b60 2024-04-06T09:57:39.131 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:57:39.128+0000 7f2e84d66640 -1 reset not still connected to 0x560d2a205ba0 2024-04-06T09:57:39.131 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:57:39.128+0000 7f2e84d66640 -1 reset not still connected to 0x560d2a2336c0 2024-04-06T09:57:39.131 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:57:39.128+0000 7f2e84d66640 -1 reset not still connected to 0x560d2a570000 2024-04-06T09:57:39.131 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:57:39.128+0000 7f2e84d66640 -1 reset not still connected to 0x560d2a5705b0 2024-04-06T09:57:39.131 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:57:39.128+0000 7f2e84d66640 -1 reset not still connected to 0x560d35349380 2024-04-06T09:57:39.131 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:57:39.128+0000 7f2e84d66640 -1 reset not still connected to 0x560d3620b520 2024-04-06T09:57:39.131 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:57:39.128+0000 7f2e84d66640 -1 reset not still connected to 0x560d3620bee0 2024-04-06T09:57:39.134 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:57:39.132+0000 7f2e84d66640 -1 reset not still connected to 0x560d36666d00 2024-04-06T09:57:39.137 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:57:39.136+0000 7f2e84d66640 -1 reset not still connected to 0x560d368e4f70 2024-04-06T09:57:39.137 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:57:39.136+0000 7f2e84d66640 -1 reset not still connected to 0x560d36912ea0 2024-04-06T09:57:39.137 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:57:39.136+0000 7f2e84d66640 -1 reset not still connected to 0x560d381dcc30 2024-04-06T09:57:39.137 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:57:39.136+0000 7f2e84d66640 -1 reset not still connected to 0x560d381dd5f0 2024-04-06T09:57:39.137 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:57:39.136+0000 7f2e84d66640 -1 reset not still connected to 0x560d390dc8f0 2024-04-06T09:57:39.137 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:57:39.136+0000 7f2e84d66640 -1 reset not still connected to 0x560d390dca90 2024-04-06T09:57:39.137 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:57:39.136+0000 7f2e84d66640 -1 reset not still connected to 0x560d47e9aa90 2024-04-06T09:57:39.137 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:57:39.136+0000 7f2e84d66640 -1 reset not still connected to 0x560d4bc7e8f0 2024-04-06T09:57:39.137 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:57:39.136+0000 7f2e84d66640 -1 reset not still connected to 0x560d4bc7ec30 2024-04-06T09:57:39.138 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:57:39.136+0000 7f2e84d66640 -1 reset not still connected to 0x560d50adba00 2024-04-06T09:57:40.287 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:57:40.284+0000 7f0965192640 -1 reset not still connected to 0x5632a0ff4410 2024-04-06T09:57:40.287 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:57:40.284+0000 7f0965192640 -1 reset not still connected to 0x5632a0ff5110 2024-04-06T09:57:40.287 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:57:40.284+0000 7f0965192640 -1 reset not still connected to 0x5632a14ac820 2024-04-06T09:57:40.287 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:57:40.284+0000 7f0965192640 -1 reset not still connected to 0x5632a1988000 2024-04-06T09:57:40.287 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:57:40.284+0000 7f0965192640 -1 reset not still connected to 0x5632a1989040 2024-04-06T09:57:40.289 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:57:40.288+0000 7f0965192640 -1 reset not still connected to 0x5632a1b73ee0 2024-04-06T09:57:40.289 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:57:40.288+0000 7f0965192640 -1 reset not still connected to 0x5632a2793520 2024-04-06T09:57:40.289 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:57:40.288+0000 7f0965192640 -1 reset not still connected to 0x5632a30f0270 2024-04-06T09:57:40.289 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:57:40.288+0000 7f0965192640 -1 reset not still connected to 0x5632a30f0f70 2024-04-06T09:57:40.289 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:57:40.288+0000 7f0965192640 -1 reset not still connected to 0x5632a417e750 2024-04-06T09:57:40.290 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:57:40.288+0000 7f0965192640 -1 reset not still connected to 0x5632a4cda000 2024-04-06T09:57:40.290 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:57:40.288+0000 7f0965192640 -1 reset not still connected to 0x5632a4cda9c0 2024-04-06T09:57:40.290 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:57:40.288+0000 7f0965192640 -1 reset not still connected to 0x5632a4cdbba0 2024-04-06T09:57:40.290 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:57:40.288+0000 7f0965192640 -1 reset not still connected to 0x5632a5e9dee0 2024-04-06T09:57:40.290 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:57:40.288+0000 7f0965192640 -1 reset not still connected to 0x5632a89f4750 2024-04-06T09:57:40.290 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:57:40.288+0000 7f0965192640 -1 reset not still connected to 0x5632a89f52b0 2024-04-06T09:57:40.291 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:57:40.288+0000 7f0965192640 -1 reset not still connected to 0x5632abc812b0 2024-04-06T09:57:40.291 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:57:40.288+0000 7f0965192640 -1 reset not still connected to 0x5632abc81d40 2024-04-06T09:57:40.291 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:57:40.288+0000 7f0965192640 -1 reset not still connected to 0x5632abcbe820 2024-04-06T09:57:40.291 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:57:40.288+0000 7f0965192640 -1 reset not still connected to 0x5632abcbf930 2024-04-06T09:57:40.291 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:57:40.288+0000 7f0965192640 -1 reset not still connected to 0x5632b81f3a00 2024-04-06T09:57:40.291 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:57:40.288+0000 7f0965192640 -1 reset not still connected to 0x5632b9592000 2024-04-06T09:57:40.291 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:57:40.288+0000 7f0965192640 -1 reset not still connected to 0x5632b9593110 2024-04-06T09:57:40.291 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:57:40.288+0000 7f0965192640 -1 reset not still connected to 0x5632c0cfa000 2024-04-06T09:57:40.291 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:57:40.288+0000 7f0965192640 -1 reset not still connected to 0x5632c0cfba00 2024-04-06T09:57:40.291 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:57:40.288+0000 7f0965192640 -1 reset not still connected to 0x5632c1c23790 2024-04-06T09:57:40.291 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:57:40.288+0000 7f0965192640 -1 reset not still connected to 0x5632cb3ac000 2024-04-06T09:57:40.291 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:57:40.288+0000 7f0965192640 -1 reset not still connected to 0x5632d8f88820 2024-04-06T09:57:40.291 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:57:40.288+0000 7f0965192640 -1 reset not still connected to 0x5632dac08dd0 2024-04-06T09:57:40.291 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:57:40.288+0000 7f0965192640 -1 reset not still connected to 0x5632dac09040 2024-04-06T09:57:40.291 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:57:40.288+0000 7f0965192640 -1 reset not still connected to 0x5632e1684ea0 2024-04-06T09:57:40.292 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:57:40.288+0000 7f0965192640 -1 reset not still connected to 0x5632e16851e0 2024-04-06T09:57:41.441 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.0 with skip (0.424228239194929) > weight (0.25) 2024-04-06T09:57:41.441 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.1 with skip (0.43761933613167325) > weight (0.25) 2024-04-06T09:57:41.441 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.2 with skip (0.7103997259522) > weight (0.25) 2024-04-06T09:57:41.441 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.3 with skip (0.8857428020706054) > weight (0.25) 2024-04-06T09:57:41.441 INFO:tasks.rbd_mirror_thrash:waiting for 106 secs before thrashing 2024-04-06T09:57:42.211 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:57:42.204+0000 7f0965192640 -1 reset not still connected to 0x5632a1b73ee0 2024-04-06T09:57:42.212 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:57:42.208+0000 7f0965192640 -1 reset not still connected to 0x5632a2793520 2024-04-06T09:57:42.212 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:57:42.208+0000 7f0965192640 -1 reset not still connected to 0x5632a30f0270 2024-04-06T09:57:42.212 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:57:42.208+0000 7f0965192640 -1 reset not still connected to 0x5632a30f0f70 2024-04-06T09:57:42.212 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:57:42.208+0000 7f0965192640 -1 reset not still connected to 0x5632a417e750 2024-04-06T09:57:42.212 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:57:42.208+0000 7f0965192640 -1 reset not still connected to 0x5632a4cda000 2024-04-06T09:57:42.212 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:57:42.208+0000 7f0965192640 -1 reset not still connected to 0x5632a4cda9c0 2024-04-06T09:57:42.212 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:57:42.212+0000 7f0965192640 -1 reset not still connected to 0x5632a4cdbba0 2024-04-06T09:57:42.212 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:57:42.212+0000 7f0965192640 -1 reset not still connected to 0x5632a5e9dee0 2024-04-06T09:57:42.224 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:57:42.220+0000 7f0965192640 -1 reset not still connected to 0x5632a89f4750 2024-04-06T09:57:42.224 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:57:42.220+0000 7f0965192640 -1 reset not still connected to 0x5632a89f52b0 2024-04-06T09:57:42.224 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:57:42.220+0000 7f0965192640 -1 reset not still connected to 0x5632abc812b0 2024-04-06T09:57:42.224 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:57:42.220+0000 7f0965192640 -1 reset not still connected to 0x5632abc81d40 2024-04-06T09:57:42.224 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:57:42.220+0000 7f0965192640 -1 reset not still connected to 0x5632abcbe820 2024-04-06T09:57:42.225 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:57:42.224+0000 7f0965192640 -1 reset not still connected to 0x5632b81f3a00 2024-04-06T09:57:42.225 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:57:42.224+0000 7f0965192640 -1 reset not still connected to 0x5632b9592000 2024-04-06T09:57:42.225 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:57:42.224+0000 7f0965192640 -1 reset not still connected to 0x5632b9593110 2024-04-06T09:57:42.225 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:57:42.224+0000 7f0965192640 -1 reset not still connected to 0x5632c0cfa000 2024-04-06T09:57:42.229 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:57:42.228+0000 7f0965192640 -1 reset not still connected to 0x5632c0cfba00 2024-04-06T09:57:42.229 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:57:42.228+0000 7f0965192640 -1 reset not still connected to 0x5632c1c23790 2024-04-06T09:57:42.229 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:57:42.228+0000 7f0965192640 -1 reset not still connected to 0x5632cb3ac000 2024-04-06T09:57:42.229 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:57:42.228+0000 7f0965192640 -1 reset not still connected to 0x5632d8f88820 2024-04-06T09:57:42.229 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:57:42.228+0000 7f0965192640 -1 reset not still connected to 0x5632dac08dd0 2024-04-06T09:57:42.229 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:57:42.228+0000 7f0965192640 -1 reset not still connected to 0x5632dac09040 2024-04-06T09:57:42.229 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:57:42.228+0000 7f0965192640 -1 reset not still connected to 0x5632e1684ea0 2024-04-06T09:57:42.229 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:57:42.228+0000 7f0965192640 -1 reset not still connected to 0x5632e16851e0 2024-04-06T09:57:42.278 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:57:42.276+0000 7f2e84d66640 -1 reset not still connected to 0x560d27073860 2024-04-06T09:57:42.496 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:57:42.488+0000 7f0965192640 -1 reset not still connected to 0x5632a14ac820 2024-04-06T09:57:42.497 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:57:42.496+0000 7f0965192640 -1 reset not still connected to 0x5632a1988000 2024-04-06T09:57:42.497 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:57:42.496+0000 7f0965192640 -1 reset not still connected to 0x5632a1989040 2024-04-06T09:57:42.497 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:57:42.496+0000 7f0965192640 -1 reset not still connected to 0x5632a1b73ee0 2024-04-06T09:57:42.497 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:57:42.496+0000 7f0965192640 -1 reset not still connected to 0x5632a2793520 2024-04-06T09:57:42.497 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:57:42.496+0000 7f0965192640 -1 reset not still connected to 0x5632a30f0270 2024-04-06T09:57:42.497 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:57:42.496+0000 7f0965192640 -1 reset not still connected to 0x5632a30f0f70 2024-04-06T09:57:42.497 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:57:42.496+0000 7f0965192640 -1 reset not still connected to 0x5632a417e750 2024-04-06T09:57:42.497 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:57:42.496+0000 7f0965192640 -1 reset not still connected to 0x5632a4cda000 2024-04-06T09:57:42.497 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:57:42.496+0000 7f0965192640 -1 reset not still connected to 0x5632a4cda9c0 2024-04-06T09:57:42.497 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:57:42.496+0000 7f0965192640 -1 reset not still connected to 0x5632a4cdbba0 2024-04-06T09:57:42.498 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:57:42.496+0000 7f0965192640 -1 reset not still connected to 0x5632a5e9dee0 2024-04-06T09:57:42.498 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:57:42.496+0000 7f0965192640 -1 reset not still connected to 0x5632a89f4750 2024-04-06T09:57:42.498 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:57:42.496+0000 7f0965192640 -1 reset not still connected to 0x5632a89f52b0 2024-04-06T09:57:42.498 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:57:42.496+0000 7f0965192640 -1 reset not still connected to 0x5632abc812b0 2024-04-06T09:57:42.499 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:57:42.496+0000 7f0965192640 -1 reset not still connected to 0x5632abc81d40 2024-04-06T09:57:42.499 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:57:42.496+0000 7f0965192640 -1 reset not still connected to 0x5632abcbe820 2024-04-06T09:57:42.499 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:57:42.496+0000 7f0965192640 -1 reset not still connected to 0x5632abcbf930 2024-04-06T09:57:42.499 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:57:42.496+0000 7f0965192640 -1 reset not still connected to 0x5632b81f3a00 2024-04-06T09:57:42.499 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:57:42.496+0000 7f0965192640 -1 reset not still connected to 0x5632b9592000 2024-04-06T09:57:42.499 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:57:42.496+0000 7f0965192640 -1 reset not still connected to 0x5632b9593110 2024-04-06T09:57:42.499 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:57:42.496+0000 7f0965192640 -1 reset not still connected to 0x5632c0cfa000 2024-04-06T09:57:42.499 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:57:42.496+0000 7f0965192640 -1 reset not still connected to 0x5632c0cfba00 2024-04-06T09:57:42.499 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:57:42.496+0000 7f0965192640 -1 reset not still connected to 0x5632c1c23790 2024-04-06T09:57:42.499 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:57:42.496+0000 7f0965192640 -1 reset not still connected to 0x5632cb3ac000 2024-04-06T09:57:42.499 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:57:42.496+0000 7f0965192640 -1 reset not still connected to 0x5632d8f88820 2024-04-06T09:57:42.499 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:57:42.496+0000 7f0965192640 -1 reset not still connected to 0x5632dac08dd0 2024-04-06T09:57:42.499 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:57:42.496+0000 7f0965192640 -1 reset not still connected to 0x5632dac09040 2024-04-06T09:57:42.500 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:57:42.496+0000 7f0965192640 -1 reset not still connected to 0x5632e1684ea0 2024-04-06T09:57:42.500 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:57:42.496+0000 7f0965192640 -1 reset not still connected to 0x5632e16851e0 2024-04-06T09:57:42.627 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:57:42.624+0000 7f0965192640 -1 reset not still connected to 0x5632a14ac820 2024-04-06T09:57:42.627 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:57:42.624+0000 7f0965192640 -1 reset not still connected to 0x5632a1988000 2024-04-06T09:57:42.627 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:57:42.624+0000 7f0965192640 -1 reset not still connected to 0x5632a1b73ee0 2024-04-06T09:57:42.627 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:57:42.624+0000 7f0965192640 -1 reset not still connected to 0x5632a2793520 2024-04-06T09:57:42.627 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:57:42.624+0000 7f0965192640 -1 reset not still connected to 0x5632a30f0270 2024-04-06T09:57:42.627 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:57:42.624+0000 7f0965192640 -1 reset not still connected to 0x5632a417e750 2024-04-06T09:57:42.627 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:57:42.624+0000 7f0965192640 -1 reset not still connected to 0x5632a4cda000 2024-04-06T09:57:42.627 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:57:42.624+0000 7f0965192640 -1 reset not still connected to 0x5632a5e9dee0 2024-04-06T09:57:42.627 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:57:42.624+0000 7f0965192640 -1 reset not still connected to 0x5632abcbe820 2024-04-06T09:57:42.627 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:57:42.624+0000 7f0965192640 -1 reset not still connected to 0x5632cb3ac000 2024-04-06T09:57:42.627 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:57:42.624+0000 7f0965192640 -1 reset not still connected to 0x5632e16851e0 2024-04-06T09:57:43.657 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:57:43.656+0000 7f0965192640 -1 reset not still connected to 0x5632a0ff5110 2024-04-06T09:57:43.663 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:57:43.656+0000 7f0965192640 -1 reset not still connected to 0x5632a14ac820 2024-04-06T09:57:43.663 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:57:43.656+0000 7f0965192640 -1 reset not still connected to 0x5632a1988000 2024-04-06T09:57:43.663 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:57:43.656+0000 7f0965192640 -1 reset not still connected to 0x5632a1989040 2024-04-06T09:57:43.663 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:57:43.656+0000 7f0965192640 -1 reset not still connected to 0x5632a1b73ee0 2024-04-06T09:57:43.663 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:57:43.656+0000 7f0965192640 -1 reset not still connected to 0x5632a2793520 2024-04-06T09:57:43.663 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:57:43.656+0000 7f0965192640 -1 reset not still connected to 0x5632a30f0270 2024-04-06T09:57:43.663 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:57:43.656+0000 7f0965192640 -1 reset not still connected to 0x5632a30f0f70 2024-04-06T09:57:43.663 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:57:43.656+0000 7f0965192640 -1 reset not still connected to 0x5632a417e750 2024-04-06T09:57:43.663 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:57:43.656+0000 7f0965192640 -1 reset not still connected to 0x5632a4cda000 2024-04-06T09:57:43.663 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:57:43.656+0000 7f0965192640 -1 reset not still connected to 0x5632a4cda9c0 2024-04-06T09:57:43.663 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:57:43.656+0000 7f0965192640 -1 reset not still connected to 0x5632a4cdbba0 2024-04-06T09:57:43.669 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:57:43.668+0000 7f0965192640 -1 reset not still connected to 0x5632a5e9dee0 2024-04-06T09:57:43.675 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:57:43.672+0000 7f0965192640 -1 reset not still connected to 0x5632a89f4750 2024-04-06T09:57:43.675 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:57:43.672+0000 7f0965192640 -1 reset not still connected to 0x5632a89f52b0 2024-04-06T09:57:43.675 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:57:43.672+0000 7f0965192640 -1 reset not still connected to 0x5632abc812b0 2024-04-06T09:57:43.675 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:57:43.672+0000 7f0965192640 -1 reset not still connected to 0x5632abc81d40 2024-04-06T09:57:43.675 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:57:43.672+0000 7f0965192640 -1 reset not still connected to 0x5632abcbe820 2024-04-06T09:57:43.675 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:57:43.672+0000 7f0965192640 -1 reset not still connected to 0x5632abcbf930 2024-04-06T09:57:43.676 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:57:43.672+0000 7f0965192640 -1 reset not still connected to 0x5632b81f3a00 2024-04-06T09:57:43.676 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:57:43.672+0000 7f0965192640 -1 reset not still connected to 0x5632b9592000 2024-04-06T09:57:43.676 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:57:43.672+0000 7f0965192640 -1 reset not still connected to 0x5632b9593110 2024-04-06T09:57:43.676 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:57:43.672+0000 7f0965192640 -1 reset not still connected to 0x5632c0cfa000 2024-04-06T09:57:43.676 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:57:43.672+0000 7f0965192640 -1 reset not still connected to 0x5632c0cfba00 2024-04-06T09:57:43.676 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:57:43.672+0000 7f0965192640 -1 reset not still connected to 0x5632c1c23790 2024-04-06T09:57:43.676 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:57:43.672+0000 7f0965192640 -1 reset not still connected to 0x5632cb3ac000 2024-04-06T09:57:43.676 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:57:43.672+0000 7f0965192640 -1 reset not still connected to 0x5632d8f88820 2024-04-06T09:57:43.676 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:57:43.672+0000 7f0965192640 -1 reset not still connected to 0x5632dac08dd0 2024-04-06T09:57:43.678 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:57:43.676+0000 7f0965192640 -1 reset not still connected to 0x5632dac09040 2024-04-06T09:57:43.678 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:57:43.676+0000 7f0965192640 -1 reset not still connected to 0x5632e1684ea0 2024-04-06T09:57:43.678 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:57:43.676+0000 7f0965192640 -1 reset not still connected to 0x5632e16851e0 2024-04-06T09:57:44.109 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:100 IOs, 73994240 bytes 2024-04-06T09:57:44.573 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:57:44.568+0000 7f2e84d66640 -1 reset not still connected to 0x560d2a17d450 2024-04-06T09:57:44.573 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:57:44.568+0000 7f2e84d66640 -1 reset not still connected to 0x560d2a17d6c0 2024-04-06T09:57:44.575 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:57:44.572+0000 7f2e84d66640 -1 reset not still connected to 0x560d2a205ba0 2024-04-06T09:57:44.575 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:57:44.572+0000 7f2e84d66640 -1 reset not still connected to 0x560d2a2336c0 2024-04-06T09:57:44.575 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:57:44.572+0000 7f2e84d66640 -1 reset not still connected to 0x560d2a5705b0 2024-04-06T09:57:44.582 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:57:44.580+0000 7f2e84d66640 -1 reset not still connected to 0x560d3620b520 2024-04-06T09:57:44.586 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:57:44.584+0000 7f2e84d66640 -1 reset not still connected to 0x560d36666d00 2024-04-06T09:57:44.587 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:57:44.584+0000 7f2e84d66640 -1 reset not still connected to 0x560d368e4f70 2024-04-06T09:57:44.587 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:57:44.584+0000 7f2e84d66640 -1 reset not still connected to 0x560d36912ea0 2024-04-06T09:57:44.587 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:57:44.584+0000 7f2e84d66640 -1 reset not still connected to 0x560d381dd5f0 2024-04-06T09:57:44.587 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:57:44.584+0000 7f2e84d66640 -1 reset not still connected to 0x560d390dca90 2024-04-06T09:57:44.587 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:57:44.584+0000 7f2e84d66640 -1 reset not still connected to 0x560d4bc7e8f0 2024-04-06T09:57:44.587 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:57:44.584+0000 7f2e84d66640 -1 reset not still connected to 0x560d4bc7ec30 2024-04-06T09:57:44.605 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:57:44.604+0000 7f2e84d66640 -1 reset not still connected to 0x560d27073860 2024-04-06T09:57:44.607 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:57:44.604+0000 7f2e84d66640 -1 reset not still connected to 0x560d27567380 2024-04-06T09:57:44.607 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:57:44.604+0000 7f2e84d66640 -1 reset not still connected to 0x560d27839380 2024-04-06T09:57:44.607 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:57:44.604+0000 7f2e84d66640 -1 reset not still connected to 0x560d27839930 2024-04-06T09:57:44.607 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:57:44.604+0000 7f2e84d66640 -1 reset not still connected to 0x560d27c3edd0 2024-04-06T09:57:44.607 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:57:44.604+0000 7f2e84d66640 -1 reset not still connected to 0x560d27c3f2b0 2024-04-06T09:57:44.620 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:57:44.616+0000 7f2e84d66640 -1 reset not still connected to 0x560d29cd8f70 2024-04-06T09:57:44.620 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:57:44.616+0000 7f2e84d66640 -1 reset not still connected to 0x560d29cd9ba0 2024-04-06T09:57:44.620 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:57:44.616+0000 7f2e84d66640 -1 reset not still connected to 0x560d29d5b520 2024-04-06T09:57:44.620 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:57:44.616+0000 7f2e84d66640 -1 reset not still connected to 0x560d2a17d450 2024-04-06T09:57:44.620 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:57:44.616+0000 7f2e84d66640 -1 reset not still connected to 0x560d2a17d6c0 2024-04-06T09:57:44.620 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:57:44.616+0000 7f2e84d66640 -1 reset not still connected to 0x560d2a204b60 2024-04-06T09:57:44.620 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:57:44.616+0000 7f2e84d66640 -1 reset not still connected to 0x560d2a205ba0 2024-04-06T09:57:44.620 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:57:44.616+0000 7f2e84d66640 -1 reset not still connected to 0x560d2a2336c0 2024-04-06T09:57:44.624 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:57:44.620+0000 7f2e84d66640 -1 reset not still connected to 0x560d2a5705b0 2024-04-06T09:57:44.634 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:57:44.624+0000 7f2e84d66640 -1 reset not still connected to 0x560d35349380 2024-04-06T09:57:44.634 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:57:44.624+0000 7f2e84d66640 -1 reset not still connected to 0x560d3620b520 2024-04-06T09:57:44.634 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:57:44.628+0000 7f2e84d66640 -1 reset not still connected to 0x560d368e4f70 2024-04-06T09:57:44.634 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:57:44.628+0000 7f2e84d66640 -1 reset not still connected to 0x560d36912ea0 2024-04-06T09:57:44.634 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:57:44.628+0000 7f2e84d66640 -1 reset not still connected to 0x560d381dcc30 2024-04-06T09:57:44.635 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:57:44.628+0000 7f2e84d66640 -1 reset not still connected to 0x560d381dd5f0 2024-04-06T09:57:44.635 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:57:44.628+0000 7f2e84d66640 -1 reset not still connected to 0x560d390dc8f0 2024-04-06T09:57:44.635 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:57:44.628+0000 7f2e84d66640 -1 reset not still connected to 0x560d390dca90 2024-04-06T09:57:44.635 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:57:44.628+0000 7f2e84d66640 -1 reset not still connected to 0x560d4bc7e8f0 2024-04-06T09:57:44.635 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:57:44.632+0000 7f2e84d66640 -1 reset not still connected to 0x560d4bc7ec30 2024-04-06T09:57:44.635 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:57:44.632+0000 7f2e84d66640 -1 reset not still connected to 0x560d50adba00 2024-04-06T09:57:45.151 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi192.stderr:2024-04-06T09:57:45.149+0000 7f3d28bce640 -1 rbd::mirror::InstanceWatcher: C_NotifyInstanceRequest: 0x560668d738c0 finish: resending after timeout 2024-04-06T09:57:45.151 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi192.stderr:2024-04-06T09:57:45.149+0000 7f3d293cf640 -1 rbd::mirror::InstanceWatcher: C_NotifyInstanceRequest: 0x560663838600 finish: resending after timeout 2024-04-06T09:57:45.151 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi192.stderr:2024-04-06T09:57:45.149+0000 7f3d28bce640 -1 rbd::mirror::InstanceWatcher: C_NotifyInstanceRequest: 0x560663839800 finish: resending after timeout 2024-04-06T09:57:47.198 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:57:47.188+0000 7f0965192640 -1 reset not still connected to 0x5632a417e750 2024-04-06T09:57:47.200 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:57:47.188+0000 7f0965192640 -1 reset not still connected to 0x5632abc812b0 2024-04-06T09:57:47.200 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:57:47.188+0000 7f0965192640 -1 reset not still connected to 0x5632abcbe820 2024-04-06T09:57:47.206 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:57:47.200+0000 7f0965192640 -1 reset not still connected to 0x5632e16851e0 2024-04-06T09:57:47.631 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:57:47.624+0000 7f0965192640 -1 reset not still connected to 0x5632a417e750 2024-04-06T09:57:47.632 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:57:47.628+0000 7f0965192640 -1 reset not still connected to 0x5632a4cda000 2024-04-06T09:57:47.638 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:57:47.636+0000 7f0965192640 -1 reset not still connected to 0x5632a5e9dee0 2024-04-06T09:57:47.638 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:57:47.636+0000 7f0965192640 -1 reset not still connected to 0x5632abc812b0 2024-04-06T09:57:47.638 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:57:47.636+0000 7f0965192640 -1 reset not still connected to 0x5632abc81d40 2024-04-06T09:57:47.638 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:57:47.636+0000 7f0965192640 -1 reset not still connected to 0x5632abcbe820 2024-04-06T09:57:47.638 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:57:47.636+0000 7f0965192640 -1 reset not still connected to 0x5632b9593110 2024-04-06T09:57:47.638 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:57:47.636+0000 7f0965192640 -1 reset not still connected to 0x5632cb3ac000 2024-04-06T09:57:47.638 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:57:47.636+0000 7f0965192640 -1 reset not still connected to 0x5632dac08dd0 2024-04-06T09:57:47.639 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:57:47.636+0000 7f0965192640 -1 reset not still connected to 0x5632dac09040 2024-04-06T09:57:47.639 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:57:47.636+0000 7f0965192640 -1 reset not still connected to 0x5632e16851e0 2024-04-06T09:57:47.664 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:57:47.660+0000 7f2e84d66640 -1 reset not still connected to 0x560d294908f0 2024-04-06T09:57:47.665 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:57:47.664+0000 7f2e84d66640 -1 reset not still connected to 0x560d29cd8f70 2024-04-06T09:57:47.665 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:57:47.664+0000 7f2e84d66640 -1 reset not still connected to 0x560d29cd9ba0 2024-04-06T09:57:47.665 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:57:47.664+0000 7f2e84d66640 -1 reset not still connected to 0x560d29d5b520 2024-04-06T09:57:47.666 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:57:47.664+0000 7f2e84d66640 -1 reset not still connected to 0x560d2a17d450 2024-04-06T09:57:47.666 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:57:47.664+0000 7f2e84d66640 -1 reset not still connected to 0x560d2a17d6c0 2024-04-06T09:57:47.671 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:57:47.668+0000 7f2e84d66640 -1 reset not still connected to 0x560d2a205ba0 2024-04-06T09:57:47.671 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:57:47.668+0000 7f2e84d66640 -1 reset not still connected to 0x560d2a2336c0 2024-04-06T09:57:47.671 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:57:47.668+0000 7f2e84d66640 -1 reset not still connected to 0x560d2a5705b0 2024-04-06T09:57:47.671 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:57:47.668+0000 7f2e84d66640 -1 reset not still connected to 0x560d3620bee0 2024-04-06T09:57:47.671 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:57:47.668+0000 7f2e84d66640 -1 reset not still connected to 0x560d368e4f70 2024-04-06T09:57:47.672 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:57:47.672+0000 7f2e84d66640 -1 reset not still connected to 0x560d381dd5f0 2024-04-06T09:57:47.675 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:57:47.672+0000 7f2e84d66640 -1 reset not still connected to 0x560d47e9aa90 2024-04-06T09:57:47.675 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:57:47.672+0000 7f2e84d66640 -1 reset not still connected to 0x560d4bc7e8f0 2024-04-06T09:57:47.678 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:57:47.676+0000 7f2e84d66640 -1 reset not still connected to 0x560d50adba00 2024-04-06T09:57:48.643 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:200 IOs, 137234432 bytes 2024-04-06T09:57:48.848 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ error_code=124 2024-04-06T09:57:48.848 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set -e 2024-04-06T09:57:48.848 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 124 -eq 124 ']' 2024-04-06T09:57:48.848 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:57:48.848 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-06T09:57:48.848 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ image=image_48 2024-04-06T09:57:48.848 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ create_image_and_enable_mirror cluster2 mirror image_48 journal 128M 2024-04-06T09:57:48.849 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-04-06T09:57:48.849 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:57:48.849 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:57:48.849 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:57:48.849 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_48 2024-04-06T09:57:48.849 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:57:48.849 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local mode=journal 2024-04-06T09:57:48.849 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n journal ']' 2024-04-06T09:57:48.849 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:57:48.849 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ create_image cluster2 mirror image_48 128M 2024-04-06T09:57:48.849 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-04-06T09:57:48.849 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:57:48.849 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:57:48.849 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:57:48.850 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_48 2024-04-06T09:57:48.850 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:57:48.850 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local size=128 2024-04-06T09:57:48.850 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n 128M ']' 2024-04-06T09:57:48.850 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ size=128M 2024-04-06T09:57:48.850 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:57:48.850 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,journaling mirror/image_48 2024-04-06T09:57:48.924 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' pool = image ']' 2024-04-06T09:57:48.924 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' mirror = mirror_parent ']' 2024-04-06T09:57:48.924 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n '' ']' 2024-04-06T09:57:48.924 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ stress_write_image cluster2 mirror image_48 2024-04-06T09:57:48.924 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-04-06T09:57:48.924 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:57:48.924 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_48 2024-04-06T09:57:48.924 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-04-06T09:57:48.925 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local duration=7 2024-04-06T09:57:48.925 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set +e 2024-04-06T09:57:48.925 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ timeout 7s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_48 --debug-rbd=20 --debug-journaler=20 2024-04-06T09:57:52.634 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:57:52.624+0000 7f2e84d66640 -1 reset not still connected to 0x560d27073860 2024-04-06T09:57:52.642 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:57:52.640+0000 7f2e84d66640 -1 reset not still connected to 0x560d27839380 2024-04-06T09:57:52.642 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:57:52.640+0000 7f2e84d66640 -1 reset not still connected to 0x560d27839930 2024-04-06T09:57:52.642 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:57:52.640+0000 7f2e84d66640 -1 reset not still connected to 0x560d27c3edd0 2024-04-06T09:57:52.663 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:57:52.660+0000 7f2e84d66640 -1 reset not still connected to 0x560d27c3f2b0 2024-04-06T09:57:52.663 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:57:52.660+0000 7f2e84d66640 -1 reset not still connected to 0x560d2811bad0 2024-04-06T09:57:52.663 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:57:52.660+0000 7f2e84d66640 -1 reset not still connected to 0x560d281875f0 2024-04-06T09:57:52.663 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:57:52.660+0000 7f2e84d66640 -1 reset not still connected to 0x560d294908f0 2024-04-06T09:57:52.663 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:57:52.660+0000 7f2e84d66640 -1 reset not still connected to 0x560d294916c0 2024-04-06T09:57:52.663 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:57:52.660+0000 7f2e84d66640 -1 reset not still connected to 0x560d29cd8f70 2024-04-06T09:57:52.663 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:57:52.660+0000 7f2e84d66640 -1 reset not still connected to 0x560d29cd9ba0 2024-04-06T09:57:52.663 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:57:52.660+0000 7f2e84d66640 -1 reset not still connected to 0x560d29d5a340 2024-04-06T09:57:52.663 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:57:52.660+0000 7f2e84d66640 -1 reset not still connected to 0x560d29d5b520 2024-04-06T09:57:52.663 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:57:52.660+0000 7f2e84d66640 -1 reset not still connected to 0x560d2a17d450 2024-04-06T09:57:52.663 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:57:52.660+0000 7f2e84d66640 -1 reset not still connected to 0x560d2a17d6c0 2024-04-06T09:57:52.664 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:57:52.660+0000 7f2e84d66640 -1 reset not still connected to 0x560d2a204b60 2024-04-06T09:57:52.664 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:57:52.660+0000 7f2e84d66640 -1 reset not still connected to 0x560d2a205ba0 2024-04-06T09:57:52.664 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:57:52.660+0000 7f2e84d66640 -1 reset not still connected to 0x560d2a2336c0 2024-04-06T09:57:52.664 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:57:52.660+0000 7f2e84d66640 -1 reset not still connected to 0x560d2a570000 2024-04-06T09:57:52.664 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:57:52.660+0000 7f2e84d66640 -1 reset not still connected to 0x560d2a5705b0 2024-04-06T09:57:52.664 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:57:52.660+0000 7f2e84d66640 -1 reset not still connected to 0x560d35349380 2024-04-06T09:57:52.664 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:57:52.660+0000 7f2e84d66640 -1 reset not still connected to 0x560d3620b520 2024-04-06T09:57:52.664 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:57:52.660+0000 7f2e84d66640 -1 reset not still connected to 0x560d36666d00 2024-04-06T09:57:52.664 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:57:52.660+0000 7f2e84d66640 -1 reset not still connected to 0x560d368e4f70 2024-04-06T09:57:52.664 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:57:52.660+0000 7f2e84d66640 -1 reset not still connected to 0x560d36912ea0 2024-04-06T09:57:52.664 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:57:52.660+0000 7f2e84d66640 -1 reset not still connected to 0x560d381dcc30 2024-04-06T09:57:52.664 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:57:52.660+0000 7f2e84d66640 -1 reset not still connected to 0x560d381dd5f0 2024-04-06T09:57:52.664 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:57:52.660+0000 7f2e84d66640 -1 reset not still connected to 0x560d390dc8f0 2024-04-06T09:57:52.664 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:57:52.660+0000 7f2e84d66640 -1 reset not still connected to 0x560d390dca90 2024-04-06T09:57:52.665 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:57:52.660+0000 7f2e84d66640 -1 reset not still connected to 0x560d4bc7e8f0 2024-04-06T09:57:52.665 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:57:52.660+0000 7f2e84d66640 -1 reset not still connected to 0x560d4bc7ec30 2024-04-06T09:57:52.665 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:57:52.660+0000 7f2e84d66640 -1 reset not still connected to 0x560d59635ad0 2024-04-06T09:57:53.624 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:57:53.624+0000 7f0965192640 -1 reset not still connected to 0x5632dac09040 2024-04-06T09:57:54.546 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:100 IOs, 73652224 bytes 2024-04-06T09:57:55.935 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ error_code=124 2024-04-06T09:57:55.935 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set -e 2024-04-06T09:57:55.935 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 124 -eq 124 ']' 2024-04-06T09:57:55.935 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:57:55.935 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-06T09:57:55.935 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ image=image_49 2024-04-06T09:57:55.935 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ create_image_and_enable_mirror cluster2 mirror image_49 journal 128M 2024-04-06T09:57:55.935 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-04-06T09:57:55.936 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:57:55.936 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:57:55.936 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:57:55.936 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_49 2024-04-06T09:57:55.936 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:57:55.936 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local mode=journal 2024-04-06T09:57:55.936 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n journal ']' 2024-04-06T09:57:55.936 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:57:55.936 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ create_image cluster2 mirror image_49 128M 2024-04-06T09:57:55.936 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-04-06T09:57:55.936 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:57:55.936 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:57:55.936 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:57:55.936 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_49 2024-04-06T09:57:55.937 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:57:55.937 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local size=128 2024-04-06T09:57:55.937 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n 128M ']' 2024-04-06T09:57:55.937 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ size=128M 2024-04-06T09:57:55.937 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:57:55.937 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,journaling mirror/image_49 2024-04-06T09:57:56.009 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' pool = image ']' 2024-04-06T09:57:56.009 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' mirror = mirror_parent ']' 2024-04-06T09:57:56.010 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n '' ']' 2024-04-06T09:57:56.010 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ stress_write_image cluster2 mirror image_49 2024-04-06T09:57:56.010 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-04-06T09:57:56.010 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:57:56.010 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_49 2024-04-06T09:57:56.010 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-04-06T09:57:56.011 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local duration=11 2024-04-06T09:57:56.011 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set +e 2024-04-06T09:57:56.011 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ timeout 11s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_49 --debug-rbd=20 --debug-journaler=20 2024-04-06T09:57:57.024 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi192.stderr:2024-04-06T09:57:57.025+0000 7f3d28bce640 -1 rbd::mirror::InstanceWatcher: C_NotifyInstanceRequest: 0x560668d738c0 finish: resending after timeout 2024-04-06T09:57:57.025 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi192.stderr:2024-04-06T09:57:57.025+0000 7f3d293cf640 -1 rbd::mirror::InstanceWatcher: C_NotifyInstanceRequest: 0x560663838600 finish: resending after timeout 2024-04-06T09:57:57.025 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi192.stderr:2024-04-06T09:57:57.025+0000 7f3d28bce640 -1 rbd::mirror::InstanceWatcher: C_NotifyInstanceRequest: 0x560663839800 finish: resending after timeout 2024-04-06T09:57:57.026 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi192.stderr:2024-04-06T09:57:57.025+0000 7f3d293cf640 -1 rbd::mirror::InstanceWatcher: C_NotifyInstanceRequest: 0x56067b284d80 finish: resending after timeout 2024-04-06T09:57:58.501 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-04-06T09:57:58.496+0000 7fd1b6210640 -1 reset not still connected to 0x55ef9bd5c9c0 2024-04-06T09:57:58.501 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-04-06T09:57:58.496+0000 7fd1b6210640 -1 reset not still connected to 0x55ef9bd5d520 2024-04-06T09:57:58.501 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-04-06T09:57:58.496+0000 7fd1b6210640 -1 reset not still connected to 0x55ef9bd5dd40 2024-04-06T09:57:58.502 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-04-06T09:57:58.496+0000 7fd1b6210640 -1 reset not still connected to 0x55ef9bd86820 2024-04-06T09:57:58.502 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-04-06T09:57:58.496+0000 7fd1b6210640 -1 reset not still connected to 0x55ef9c103110 2024-04-06T09:57:58.502 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-04-06T09:57:58.496+0000 7fd1b6210640 -1 reset not still connected to 0x55ef9c103930 2024-04-06T09:57:58.504 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-04-06T09:57:58.504+0000 7fd1b6210640 -1 reset not still connected to 0x55ef9c103d40 2024-04-06T09:57:58.504 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-04-06T09:57:58.504+0000 7fd1b6210640 -1 reset not still connected to 0x55ef9e698c30 2024-04-06T09:57:58.504 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-04-06T09:57:58.504+0000 7fd1b6210640 -1 reset not still connected to 0x55ef9e88f520 2024-04-06T09:57:58.504 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-04-06T09:57:58.504+0000 7fd1b6210640 -1 reset not still connected to 0x55ef9eee21a0 2024-04-06T09:57:58.505 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-04-06T09:57:58.504+0000 7fd1b6210640 -1 reset not still connected to 0x55ef9eee3a00 2024-04-06T09:57:58.505 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-04-06T09:57:58.504+0000 7fd1b6210640 -1 reset not still connected to 0x55efa18cc9c0 2024-04-06T09:57:58.505 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-04-06T09:57:58.504+0000 7fd1b6210640 -1 reset not still connected to 0x55efa18cca90 2024-04-06T09:57:58.505 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-04-06T09:57:58.504+0000 7fd1b6210640 -1 reset not still connected to 0x55efae5ffe10 2024-04-06T09:57:58.505 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-04-06T09:57:58.504+0000 7fd1b6210640 -1 reset not still connected to 0x55efbb630d00 2024-04-06T09:57:58.505 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-04-06T09:57:58.504+0000 7fd1b6210640 -1 reset not still connected to 0x55efd81ee8f0 2024-04-06T09:57:58.505 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-04-06T09:57:58.504+0000 7fd1b6210640 -1 reset not still connected to 0x55efe06a9860 2024-04-06T09:58:00.618 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:58:00.612+0000 7f2e84d66640 -1 reset not still connected to 0x560d27c3edd0 2024-04-06T09:58:00.620 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:58:00.612+0000 7f2e84d66640 -1 reset not still connected to 0x560d27c3f2b0 2024-04-06T09:58:00.620 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:58:00.612+0000 7f2e84d66640 -1 reset not still connected to 0x560d294908f0 2024-04-06T09:58:00.620 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:58:00.616+0000 7f2e84d66640 -1 reset not still connected to 0x560d29cd8f70 2024-04-06T09:58:00.621 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:58:00.616+0000 7f2e84d66640 -1 reset not still connected to 0x560d29cd9ba0 2024-04-06T09:58:00.621 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:58:00.616+0000 7f2e84d66640 -1 reset not still connected to 0x560d29d5b520 2024-04-06T09:58:00.621 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:58:00.616+0000 7f2e84d66640 -1 reset not still connected to 0x560d2a17d450 2024-04-06T09:58:00.621 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:58:00.616+0000 7f2e84d66640 -1 reset not still connected to 0x560d2a17d6c0 2024-04-06T09:58:00.621 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:58:00.616+0000 7f2e84d66640 -1 reset not still connected to 0x560d2a204b60 2024-04-06T09:58:00.622 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:58:00.616+0000 7f2e84d66640 -1 reset not still connected to 0x560d2a205ba0 2024-04-06T09:58:00.622 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:58:00.616+0000 7f2e84d66640 -1 reset not still connected to 0x560d2a2336c0 2024-04-06T09:58:00.622 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:58:00.616+0000 7f2e84d66640 -1 reset not still connected to 0x560d2a5705b0 2024-04-06T09:58:00.622 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:58:00.616+0000 7f2e84d66640 -1 reset not still connected to 0x560d35349380 2024-04-06T09:58:00.622 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:58:00.616+0000 7f2e84d66640 -1 reset not still connected to 0x560d3620b520 2024-04-06T09:58:00.623 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:58:00.620+0000 7f2e84d66640 -1 reset not still connected to 0x560d36666d00 2024-04-06T09:58:00.623 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:58:00.620+0000 7f2e84d66640 -1 reset not still connected to 0x560d368e4f70 2024-04-06T09:58:00.623 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:58:00.620+0000 7f2e84d66640 -1 reset not still connected to 0x560d36912ea0 2024-04-06T09:58:00.623 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:58:00.620+0000 7f2e84d66640 -1 reset not still connected to 0x560d381dd5f0 2024-04-06T09:58:00.623 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:58:00.620+0000 7f2e84d66640 -1 reset not still connected to 0x560d4bc7e8f0 2024-04-06T09:58:00.624 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:58:00.620+0000 7f2e84d66640 -1 reset not still connected to 0x560d4bc7ec30 2024-04-06T09:58:00.624 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:58:00.620+0000 7f2e84d66640 -1 reset not still connected to 0x560d50adba00 2024-04-06T09:58:01.317 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-04-06T09:58:01.316+0000 7fd1b6210640 -1 reset not still connected to 0x55ef9c103d40 2024-04-06T09:58:01.317 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-04-06T09:58:01.316+0000 7fd1b6210640 -1 reset not still connected to 0x55ef9e698c30 2024-04-06T09:58:01.322 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-04-06T09:58:01.320+0000 7fd1b6210640 -1 reset not still connected to 0x55ef9e88f520 2024-04-06T09:58:01.322 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-04-06T09:58:01.320+0000 7fd1b6210640 -1 reset not still connected to 0x55ef9eee21a0 2024-04-06T09:58:01.322 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-04-06T09:58:01.320+0000 7fd1b6210640 -1 reset not still connected to 0x55efa18cc270 2024-04-06T09:58:01.332 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-04-06T09:58:01.332+0000 7fd1b6210640 -1 reset not still connected to 0x55efa18cc9c0 2024-04-06T09:58:01.332 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-04-06T09:58:01.332+0000 7fd1b6210640 -1 reset not still connected to 0x55efd81ee8f0 2024-04-06T09:58:01.332 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-04-06T09:58:01.332+0000 7fd1b6210640 -1 reset not still connected to 0x55efe06a9860 2024-04-06T09:58:01.571 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:58:01.568+0000 7f2e84d66640 -1 reset not still connected to 0x560d2a205ba0 2024-04-06T09:58:01.571 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:58:01.568+0000 7f2e84d66640 -1 reset not still connected to 0x560d2a2336c0 2024-04-06T09:58:01.579 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:58:01.576+0000 7f2e84d66640 -1 reset not still connected to 0x560d2a570000 2024-04-06T09:58:01.579 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:58:01.576+0000 7f2e84d66640 -1 reset not still connected to 0x560d2a5705b0 2024-04-06T09:58:01.579 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:58:01.576+0000 7f2e84d66640 -1 reset not still connected to 0x560d35349380 2024-04-06T09:58:01.579 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:58:01.576+0000 7f2e84d66640 -1 reset not still connected to 0x560d3620b520 2024-04-06T09:58:01.579 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:58:01.576+0000 7f2e84d66640 -1 reset not still connected to 0x560d3620bee0 2024-04-06T09:58:01.579 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:58:01.576+0000 7f2e84d66640 -1 reset not still connected to 0x560d36666d00 2024-04-06T09:58:01.579 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:58:01.576+0000 7f2e84d66640 -1 reset not still connected to 0x560d368e4f70 2024-04-06T09:58:01.579 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:58:01.576+0000 7f2e84d66640 -1 reset not still connected to 0x560d36912ea0 2024-04-06T09:58:01.579 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:58:01.576+0000 7f2e84d66640 -1 reset not still connected to 0x560d381dcc30 2024-04-06T09:58:01.579 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:58:01.576+0000 7f2e84d66640 -1 reset not still connected to 0x560d381dd5f0 2024-04-06T09:58:01.579 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:58:01.576+0000 7f2e84d66640 -1 reset not still connected to 0x560d390dc8f0 2024-04-06T09:58:01.579 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:58:01.576+0000 7f2e84d66640 -1 reset not still connected to 0x560d390dca90 2024-04-06T09:58:01.581 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:58:01.576+0000 7f2e84d66640 -1 reset not still connected to 0x560d47e9aa90 2024-04-06T09:58:01.581 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:58:01.576+0000 7f2e84d66640 -1 reset not still connected to 0x560d4bc7e8f0 2024-04-06T09:58:01.581 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:58:01.576+0000 7f2e84d66640 -1 reset not still connected to 0x560d4bc7ec30 2024-04-06T09:58:01.587 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:58:01.584+0000 7f2e84d66640 -1 reset not still connected to 0x560d4e9ddba0 2024-04-06T09:58:01.587 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:58:01.584+0000 7f2e84d66640 -1 reset not still connected to 0x560d50adba00 2024-04-06T09:58:01.587 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:58:01.584+0000 7f2e84d66640 -1 reset not still connected to 0x560d59635ad0 2024-04-06T09:58:03.021 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-04-06T09:58:03.020+0000 7fd1b6210640 -1 reset not still connected to 0x55ef9ac60270 2024-04-06T09:58:03.022 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-04-06T09:58:03.020+0000 7fd1b6210640 -1 reset not still connected to 0x55ef9b657e10 2024-04-06T09:58:03.022 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-04-06T09:58:03.020+0000 7fd1b6210640 -1 reset not still connected to 0x55ef9bbe84e0 2024-04-06T09:58:03.022 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-04-06T09:58:03.020+0000 7fd1b6210640 -1 reset not still connected to 0x55ef9bbe8d00 2024-04-06T09:58:03.022 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-04-06T09:58:03.020+0000 7fd1b6210640 -1 reset not still connected to 0x55ef9bbe9ee0 2024-04-06T09:58:03.022 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-04-06T09:58:03.020+0000 7fd1b6210640 -1 reset not still connected to 0x55ef9bd5c9c0 2024-04-06T09:58:03.022 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-04-06T09:58:03.020+0000 7fd1b6210640 -1 reset not still connected to 0x55ef9bd5d520 2024-04-06T09:58:03.022 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-04-06T09:58:03.020+0000 7fd1b6210640 -1 reset not still connected to 0x55ef9bd86820 2024-04-06T09:58:03.022 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-04-06T09:58:03.020+0000 7fd1b6210640 -1 reset not still connected to 0x55ef9c103110 2024-04-06T09:58:03.026 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-04-06T09:58:03.024+0000 7fd1b6210640 -1 reset not still connected to 0x55ef9c103930 2024-04-06T09:58:03.026 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-04-06T09:58:03.024+0000 7fd1b6210640 -1 reset not still connected to 0x55ef9c103d40 2024-04-06T09:58:03.026 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-04-06T09:58:03.024+0000 7fd1b6210640 -1 reset not still connected to 0x55ef9e698c30 2024-04-06T09:58:03.026 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-04-06T09:58:03.024+0000 7fd1b6210640 -1 reset not still connected to 0x55ef9e88f520 2024-04-06T09:58:03.026 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-04-06T09:58:03.024+0000 7fd1b6210640 -1 reset not still connected to 0x55ef9eee21a0 2024-04-06T09:58:03.026 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-04-06T09:58:03.024+0000 7fd1b6210640 -1 reset not still connected to 0x55ef9eee3a00 2024-04-06T09:58:03.026 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-04-06T09:58:03.024+0000 7fd1b6210640 -1 reset not still connected to 0x55efa18cc270 2024-04-06T09:58:03.028 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-04-06T09:58:03.028+0000 7fd1b6210640 -1 reset not still connected to 0x55efbb630d00 2024-04-06T09:58:03.028 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-04-06T09:58:03.028+0000 7fd1b6210640 -1 reset not still connected to 0x55efd81ee8f0 2024-04-06T09:58:03.029 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-04-06T09:58:03.028+0000 7fd1b6210640 -1 reset not still connected to 0x55efe06a9860 2024-04-06T09:58:03.799 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:100 IOs, 124337152 bytes 2024-04-06T09:58:07.020 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ error_code=124 2024-04-06T09:58:07.020 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set -e 2024-04-06T09:58:07.020 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 124 -eq 124 ']' 2024-04-06T09:58:07.020 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:58:07.021 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-06T09:58:07.021 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ image=image_50 2024-04-06T09:58:07.021 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ create_image_and_enable_mirror cluster2 mirror image_50 journal 128M 2024-04-06T09:58:07.021 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-04-06T09:58:07.021 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:58:07.021 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:58:07.021 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:58:07.021 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_50 2024-04-06T09:58:07.021 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:58:07.021 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local mode=journal 2024-04-06T09:58:07.021 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n journal ']' 2024-04-06T09:58:07.021 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:58:07.021 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ create_image cluster2 mirror image_50 128M 2024-04-06T09:58:07.021 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-04-06T09:58:07.021 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:58:07.022 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:58:07.022 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:58:07.022 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_50 2024-04-06T09:58:07.022 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:58:07.022 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local size=128 2024-04-06T09:58:07.022 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n 128M ']' 2024-04-06T09:58:07.022 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ size=128M 2024-04-06T09:58:07.022 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:58:07.022 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,journaling mirror/image_50 2024-04-06T09:58:07.134 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' pool = image ']' 2024-04-06T09:58:07.134 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' mirror = mirror_parent ']' 2024-04-06T09:58:07.134 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n '' ']' 2024-04-06T09:58:07.134 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ stress_write_image cluster2 mirror image_50 2024-04-06T09:58:07.134 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-04-06T09:58:07.134 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:58:07.134 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_50 2024-04-06T09:58:07.134 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-04-06T09:58:07.135 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local duration=7 2024-04-06T09:58:07.135 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set +e 2024-04-06T09:58:07.135 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ timeout 7s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_50 --debug-rbd=20 --debug-journaler=20 2024-04-06T09:58:07.843 INFO:tasks.ceph.cluster2.osd.0.smithi192.stderr:2024-04-06T09:58:07.845+0000 7f78af6df640 -1 reset not still connected to 0x5604239371e0 2024-04-06T09:58:07.843 INFO:tasks.ceph.cluster2.osd.0.smithi192.stderr:2024-04-06T09:58:07.845+0000 7f78af6df640 -1 reset not still connected to 0x5604242c76c0 2024-04-06T09:58:07.843 INFO:tasks.ceph.cluster2.osd.0.smithi192.stderr:2024-04-06T09:58:07.845+0000 7f78af6df640 -1 reset not still connected to 0x56042bd3fc70 2024-04-06T09:58:07.843 INFO:tasks.ceph.cluster2.osd.0.smithi192.stderr:2024-04-06T09:58:07.845+0000 7f78af6df640 -1 reset not still connected to 0x56043a5848f0 2024-04-06T09:58:07.843 INFO:tasks.ceph.cluster2.osd.0.smithi192.stderr:2024-04-06T09:58:07.845+0000 7f78af6df640 -1 reset not still connected to 0x5604442b6000 2024-04-06T09:58:07.843 INFO:tasks.ceph.cluster2.osd.0.smithi192.stderr:2024-04-06T09:58:07.845+0000 7f78af6df640 -1 reset not still connected to 0x560444f6c820 2024-04-06T09:58:07.844 INFO:tasks.ceph.cluster2.osd.0.smithi192.stderr:2024-04-06T09:58:07.845+0000 7f78af6df640 -1 reset not still connected to 0x56044cc1e1a0 2024-04-06T09:58:07.844 INFO:tasks.ceph.cluster2.osd.0.smithi192.stderr:2024-04-06T09:58:07.845+0000 7f78af6df640 -1 reset not still connected to 0x560456d576c0 2024-04-06T09:58:08.065 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:58:08.064+0000 7f2e84d66640 -1 reset not still connected to 0x560d294908f0 2024-04-06T09:58:08.066 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:58:08.064+0000 7f2e84d66640 -1 reset not still connected to 0x560d29cd9ba0 2024-04-06T09:58:08.066 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:58:08.064+0000 7f2e84d66640 -1 reset not still connected to 0x560d29d5b520 2024-04-06T09:58:08.066 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:58:08.064+0000 7f2e84d66640 -1 reset not still connected to 0x560d2a17d450 2024-04-06T09:58:08.066 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:58:08.064+0000 7f2e84d66640 -1 reset not still connected to 0x560d2a17d6c0 2024-04-06T09:58:08.066 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:58:08.064+0000 7f2e84d66640 -1 reset not still connected to 0x560d2a204b60 2024-04-06T09:58:08.068 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:58:08.068+0000 7f2e84d66640 -1 reset not still connected to 0x560d2a205ba0 2024-04-06T09:58:08.068 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:58:08.068+0000 7f2e84d66640 -1 reset not still connected to 0x560d2a2336c0 2024-04-06T09:58:08.069 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:58:08.068+0000 7f2e84d66640 -1 reset not still connected to 0x560d2a5705b0 2024-04-06T09:58:08.073 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:58:08.072+0000 7f2e84d66640 -1 reset not still connected to 0x560d35349380 2024-04-06T09:58:08.073 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:58:08.072+0000 7f2e84d66640 -1 reset not still connected to 0x560d3620b520 2024-04-06T09:58:08.079 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:58:08.076+0000 7f2e84d66640 -1 reset not still connected to 0x560d3620bee0 2024-04-06T09:58:08.079 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:58:08.076+0000 7f2e84d66640 -1 reset not still connected to 0x560d36666d00 2024-04-06T09:58:08.079 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:58:08.076+0000 7f2e84d66640 -1 reset not still connected to 0x560d368e4f70 2024-04-06T09:58:08.079 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:58:08.076+0000 7f2e84d66640 -1 reset not still connected to 0x560d36912ea0 2024-04-06T09:58:08.083 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:58:08.080+0000 7f2e84d66640 -1 reset not still connected to 0x560d381dcc30 2024-04-06T09:58:08.083 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:58:08.080+0000 7f2e84d66640 -1 reset not still connected to 0x560d381dd5f0 2024-04-06T09:58:08.083 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:58:08.080+0000 7f2e84d66640 -1 reset not still connected to 0x560d390dc8f0 2024-04-06T09:58:08.083 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:58:08.080+0000 7f2e84d66640 -1 reset not still connected to 0x560d390dca90 2024-04-06T09:58:08.083 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:58:08.080+0000 7f2e84d66640 -1 reset not still connected to 0x560d47e9aa90 2024-04-06T09:58:08.083 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:58:08.080+0000 7f2e84d66640 -1 reset not still connected to 0x560d4bc7e8f0 2024-04-06T09:58:08.083 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:58:08.080+0000 7f2e84d66640 -1 reset not still connected to 0x560d4bc7ec30 2024-04-06T09:58:08.083 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:58:08.080+0000 7f2e84d66640 -1 reset not still connected to 0x560d4e9ddba0 2024-04-06T09:58:08.083 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:58:08.080+0000 7f2e84d66640 -1 reset not still connected to 0x560d50adba00 2024-04-06T09:58:08.083 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:58:08.080+0000 7f2e84d66640 -1 reset not still connected to 0x560d59635ad0 2024-04-06T09:58:09.337 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:58:09.336+0000 7f2e84d66640 -1 reset not still connected to 0x560d29d5b520 2024-04-06T09:58:09.339 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:58:09.340+0000 7f2e84d66640 -1 reset not still connected to 0x560d2a17d450 2024-04-06T09:58:09.368 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:58:09.368+0000 7f2e84d66640 -1 reset not still connected to 0x560d2a17d6c0 2024-04-06T09:58:09.368 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:58:09.368+0000 7f2e84d66640 -1 reset not still connected to 0x560d2a204b60 2024-04-06T09:58:09.378 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:58:09.376+0000 7f2e84d66640 -1 reset not still connected to 0x560d2a205ba0 2024-04-06T09:58:09.379 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:58:09.376+0000 7f2e84d66640 -1 reset not still connected to 0x560d2a2336c0 2024-04-06T09:58:09.379 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:58:09.376+0000 7f2e84d66640 -1 reset not still connected to 0x560d2a570000 2024-04-06T09:58:09.379 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:58:09.376+0000 7f2e84d66640 -1 reset not still connected to 0x560d2a5705b0 2024-04-06T09:58:09.379 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:58:09.376+0000 7f2e84d66640 -1 reset not still connected to 0x560d35349380 2024-04-06T09:58:09.379 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:58:09.376+0000 7f2e84d66640 -1 reset not still connected to 0x560d3620b520 2024-04-06T09:58:09.385 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:58:09.380+0000 7f2e84d66640 -1 reset not still connected to 0x560d3620bee0 2024-04-06T09:58:09.385 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:58:09.380+0000 7f2e84d66640 -1 reset not still connected to 0x560d36666d00 2024-04-06T09:58:09.385 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:58:09.380+0000 7f2e84d66640 -1 reset not still connected to 0x560d368e4f70 2024-04-06T09:58:09.385 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:58:09.380+0000 7f2e84d66640 -1 reset not still connected to 0x560d36912ea0 2024-04-06T09:58:09.386 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:58:09.380+0000 7f2e84d66640 -1 reset not still connected to 0x560d381dcc30 2024-04-06T09:58:09.386 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:58:09.380+0000 7f2e84d66640 -1 reset not still connected to 0x560d381dd5f0 2024-04-06T09:58:09.386 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:58:09.380+0000 7f2e84d66640 -1 reset not still connected to 0x560d390dc8f0 2024-04-06T09:58:09.386 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:58:09.380+0000 7f2e84d66640 -1 reset not still connected to 0x560d390dca90 2024-04-06T09:58:09.386 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:58:09.380+0000 7f2e84d66640 -1 reset not still connected to 0x560d47e9aa90 2024-04-06T09:58:09.386 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:58:09.380+0000 7f2e84d66640 -1 reset not still connected to 0x560d4bc7e8f0 2024-04-06T09:58:09.386 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:58:09.380+0000 7f2e84d66640 -1 reset not still connected to 0x560d4bc7ec30 2024-04-06T09:58:09.386 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:58:09.380+0000 7f2e84d66640 -1 reset not still connected to 0x560d4e9ddba0 2024-04-06T09:58:09.386 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:58:09.380+0000 7f2e84d66640 -1 reset not still connected to 0x560d50adba00 2024-04-06T09:58:09.386 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:58:09.380+0000 7f2e84d66640 -1 reset not still connected to 0x560d59635ad0 2024-04-06T09:58:09.745 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:58:09.744+0000 7f2e84d66640 -1 reset not still connected to 0x560d2722e5b0 2024-04-06T09:58:09.745 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:58:09.744+0000 7f2e84d66640 -1 reset not still connected to 0x560d27567380 2024-04-06T09:58:09.748 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:58:09.748+0000 7f2e84d66640 -1 reset not still connected to 0x560d27839380 2024-04-06T09:58:09.748 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:58:09.748+0000 7f2e84d66640 -1 reset not still connected to 0x560d27839930 2024-04-06T09:58:09.749 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:58:09.748+0000 7f2e84d66640 -1 reset not still connected to 0x560d27c3edd0 2024-04-06T09:58:09.754 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:58:09.752+0000 7f2e84d66640 -1 reset not still connected to 0x560d27c3f2b0 2024-04-06T09:58:09.757 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:58:09.756+0000 7f2e84d66640 -1 reset not still connected to 0x560d2811bad0 2024-04-06T09:58:09.757 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:58:09.756+0000 7f2e84d66640 -1 reset not still connected to 0x560d281875f0 2024-04-06T09:58:09.762 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:58:09.760+0000 7f2e84d66640 -1 reset not still connected to 0x560d294908f0 2024-04-06T09:58:09.762 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:58:09.760+0000 7f2e84d66640 -1 reset not still connected to 0x560d29490ea0 2024-04-06T09:58:09.762 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:58:09.760+0000 7f2e84d66640 -1 reset not still connected to 0x560d294916c0 2024-04-06T09:58:09.762 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:58:09.760+0000 7f2e84d66640 -1 reset not still connected to 0x560d29cd8f70 2024-04-06T09:58:09.763 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:58:09.760+0000 7f2e84d66640 -1 reset not still connected to 0x560d29cd9ba0 2024-04-06T09:58:09.763 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:58:09.760+0000 7f2e84d66640 -1 reset not still connected to 0x560d29d5a340 2024-04-06T09:58:09.763 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:58:09.760+0000 7f2e84d66640 -1 reset not still connected to 0x560d29d5b520 2024-04-06T09:58:09.763 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:58:09.760+0000 7f2e84d66640 -1 reset not still connected to 0x560d2a17d450 2024-04-06T09:58:09.763 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:58:09.760+0000 7f2e84d66640 -1 reset not still connected to 0x560d2a17d6c0 2024-04-06T09:58:09.763 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:58:09.760+0000 7f2e84d66640 -1 reset not still connected to 0x560d2a204b60 2024-04-06T09:58:09.763 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:58:09.760+0000 7f2e84d66640 -1 reset not still connected to 0x560d2a205ba0 2024-04-06T09:58:09.763 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:58:09.760+0000 7f2e84d66640 -1 reset not still connected to 0x560d2a2336c0 2024-04-06T09:58:09.763 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:58:09.760+0000 7f2e84d66640 -1 reset not still connected to 0x560d2a5705b0 2024-04-06T09:58:09.763 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:58:09.760+0000 7f2e84d66640 -1 reset not still connected to 0x560d35349380 2024-04-06T09:58:09.763 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:58:09.760+0000 7f2e84d66640 -1 reset not still connected to 0x560d3620b520 2024-04-06T09:58:09.763 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:58:09.760+0000 7f2e84d66640 -1 reset not still connected to 0x560d3620bee0 2024-04-06T09:58:09.763 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:58:09.760+0000 7f2e84d66640 -1 reset not still connected to 0x560d36666d00 2024-04-06T09:58:09.763 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:58:09.760+0000 7f2e84d66640 -1 reset not still connected to 0x560d368e4f70 2024-04-06T09:58:09.764 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:58:09.760+0000 7f2e84d66640 -1 reset not still connected to 0x560d36912ea0 2024-04-06T09:58:09.764 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:58:09.760+0000 7f2e84d66640 -1 reset not still connected to 0x560d381dd5f0 2024-04-06T09:58:09.764 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:58:09.760+0000 7f2e84d66640 -1 reset not still connected to 0x560d390dca90 2024-04-06T09:58:09.764 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:58:09.760+0000 7f2e84d66640 -1 reset not still connected to 0x560d47e9aa90 2024-04-06T09:58:09.764 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:58:09.760+0000 7f2e84d66640 -1 reset not still connected to 0x560d4bc7e8f0 2024-04-06T09:58:09.764 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:58:09.760+0000 7f2e84d66640 -1 reset not still connected to 0x560d4bc7ec30 2024-04-06T09:58:09.764 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:58:09.760+0000 7f2e84d66640 -1 reset not still connected to 0x560d50adba00 2024-04-06T09:58:10.833 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-04-06T09:58:10.832+0000 7fd1b6210640 -1 reset not still connected to 0x55ef9bd5c9c0 2024-04-06T09:58:10.833 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-04-06T09:58:10.832+0000 7fd1b6210640 -1 reset not still connected to 0x55ef9bd5d520 2024-04-06T09:58:10.833 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-04-06T09:58:10.832+0000 7fd1b6210640 -1 reset not still connected to 0x55ef9bd5dd40 2024-04-06T09:58:10.838 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-04-06T09:58:10.836+0000 7fd1b6210640 -1 reset not still connected to 0x55ef9bd86820 2024-04-06T09:58:10.838 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-04-06T09:58:10.836+0000 7fd1b6210640 -1 reset not still connected to 0x55ef9c103110 2024-04-06T09:58:10.838 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-04-06T09:58:10.836+0000 7fd1b6210640 -1 reset not still connected to 0x55ef9c103930 2024-04-06T09:58:10.838 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-04-06T09:58:10.836+0000 7fd1b6210640 -1 reset not still connected to 0x55ef9c103d40 2024-04-06T09:58:10.838 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-04-06T09:58:10.836+0000 7fd1b6210640 -1 reset not still connected to 0x55ef9e698c30 2024-04-06T09:58:10.838 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-04-06T09:58:10.836+0000 7fd1b6210640 -1 reset not still connected to 0x55ef9e88f520 2024-04-06T09:58:10.838 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-04-06T09:58:10.836+0000 7fd1b6210640 -1 reset not still connected to 0x55ef9eee21a0 2024-04-06T09:58:10.838 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-04-06T09:58:10.836+0000 7fd1b6210640 -1 reset not still connected to 0x55ef9eee3a00 2024-04-06T09:58:10.838 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-04-06T09:58:10.836+0000 7fd1b6210640 -1 reset not still connected to 0x55efa18cc270 2024-04-06T09:58:10.838 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-04-06T09:58:10.836+0000 7fd1b6210640 -1 reset not still connected to 0x55efa18cc9c0 2024-04-06T09:58:10.838 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-04-06T09:58:10.836+0000 7fd1b6210640 -1 reset not still connected to 0x55efa18cca90 2024-04-06T09:58:10.838 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-04-06T09:58:10.836+0000 7fd1b6210640 -1 reset not still connected to 0x55efa96c96c0 2024-04-06T09:58:10.839 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-04-06T09:58:10.836+0000 7fd1b6210640 -1 reset not still connected to 0x55efae5ffe10 2024-04-06T09:58:10.839 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-04-06T09:58:10.836+0000 7fd1b6210640 -1 reset not still connected to 0x55efbb630d00 2024-04-06T09:58:10.839 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-04-06T09:58:10.836+0000 7fd1b6210640 -1 reset not still connected to 0x55efd81ee8f0 2024-04-06T09:58:10.839 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-04-06T09:58:10.836+0000 7fd1b6210640 -1 reset not still connected to 0x55efe06a9860 2024-04-06T09:58:11.647 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:100 IOs, 40714240 bytes 2024-04-06T09:58:12.045 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:58:12.040+0000 7f2e84d66640 -1 reset not still connected to 0x560d27839380 2024-04-06T09:58:12.046 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:58:12.044+0000 7f2e84d66640 -1 reset not still connected to 0x560d27c3edd0 2024-04-06T09:58:12.052 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:58:12.048+0000 7f2e84d66640 -1 reset not still connected to 0x560d27c3f2b0 2024-04-06T09:58:12.054 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:58:12.052+0000 7f2e84d66640 -1 reset not still connected to 0x560d281875f0 2024-04-06T09:58:12.054 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:58:12.052+0000 7f2e84d66640 -1 reset not still connected to 0x560d294908f0 2024-04-06T09:58:12.054 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:58:12.052+0000 7f2e84d66640 -1 reset not still connected to 0x560d29490ea0 2024-04-06T09:58:12.063 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:58:12.060+0000 7f2e84d66640 -1 reset not still connected to 0x560d29cd8f70 2024-04-06T09:58:12.070 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:58:12.068+0000 7f2e84d66640 -1 reset not still connected to 0x560d29cd9ba0 2024-04-06T09:58:12.070 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:58:12.068+0000 7f2e84d66640 -1 reset not still connected to 0x560d29d5a340 2024-04-06T09:58:12.071 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:58:12.068+0000 7f2e84d66640 -1 reset not still connected to 0x560d29d5b520 2024-04-06T09:58:12.071 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:58:12.068+0000 7f2e84d66640 -1 reset not still connected to 0x560d2a17d450 2024-04-06T09:58:12.071 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:58:12.068+0000 7f2e84d66640 -1 reset not still connected to 0x560d2a17d6c0 2024-04-06T09:58:12.071 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:58:12.068+0000 7f2e84d66640 -1 reset not still connected to 0x560d2a204b60 2024-04-06T09:58:12.071 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:58:12.068+0000 7f2e84d66640 -1 reset not still connected to 0x560d2a205ba0 2024-04-06T09:58:12.071 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:58:12.068+0000 7f2e84d66640 -1 reset not still connected to 0x560d2a2336c0 2024-04-06T09:58:12.071 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:58:12.068+0000 7f2e84d66640 -1 reset not still connected to 0x560d2a570000 2024-04-06T09:58:12.071 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:58:12.068+0000 7f2e84d66640 -1 reset not still connected to 0x560d2a5705b0 2024-04-06T09:58:12.071 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:58:12.068+0000 7f2e84d66640 -1 reset not still connected to 0x560d35349380 2024-04-06T09:58:12.071 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:58:12.068+0000 7f2e84d66640 -1 reset not still connected to 0x560d3620b520 2024-04-06T09:58:12.071 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:58:12.068+0000 7f2e84d66640 -1 reset not still connected to 0x560d3620bee0 2024-04-06T09:58:12.071 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:58:12.068+0000 7f2e84d66640 -1 reset not still connected to 0x560d36666d00 2024-04-06T09:58:12.071 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:58:12.068+0000 7f2e84d66640 -1 reset not still connected to 0x560d368e4f70 2024-04-06T09:58:12.071 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:58:12.068+0000 7f2e84d66640 -1 reset not still connected to 0x560d36912ea0 2024-04-06T09:58:12.072 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:58:12.068+0000 7f2e84d66640 -1 reset not still connected to 0x560d381dcc30 2024-04-06T09:58:12.072 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:58:12.068+0000 7f2e84d66640 -1 reset not still connected to 0x560d381dd5f0 2024-04-06T09:58:12.072 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:58:12.068+0000 7f2e84d66640 -1 reset not still connected to 0x560d390dc8f0 2024-04-06T09:58:12.072 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:58:12.068+0000 7f2e84d66640 -1 reset not still connected to 0x560d390dca90 2024-04-06T09:58:12.072 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:58:12.068+0000 7f2e84d66640 -1 reset not still connected to 0x560d47e9aa90 2024-04-06T09:58:12.072 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:58:12.068+0000 7f2e84d66640 -1 reset not still connected to 0x560d4bc7e8f0 2024-04-06T09:58:12.072 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:58:12.068+0000 7f2e84d66640 -1 reset not still connected to 0x560d4bc7ec30 2024-04-06T09:58:12.073 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:58:12.072+0000 7f2e84d66640 -1 reset not still connected to 0x560d4e9ddba0 2024-04-06T09:58:12.073 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:58:12.072+0000 7f2e84d66640 -1 reset not still connected to 0x560d50adba00 2024-04-06T09:58:12.073 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:58:12.072+0000 7f2e84d66640 -1 reset not still connected to 0x560d59635ad0 2024-04-06T09:58:12.206 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-04-06T09:58:12.204+0000 7fd1b6210640 -1 reset not still connected to 0x55efa18cc9c0 2024-04-06T09:58:12.207 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-04-06T09:58:12.204+0000 7fd1b6210640 -1 reset not still connected to 0x55efa18cca90 2024-04-06T09:58:12.208 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-04-06T09:58:12.204+0000 7fd1b6210640 -1 reset not still connected to 0x55efa96c96c0 2024-04-06T09:58:12.208 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-04-06T09:58:12.204+0000 7fd1b6210640 -1 reset not still connected to 0x55efae5ffe10 2024-04-06T09:58:12.208 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-04-06T09:58:12.204+0000 7fd1b6210640 -1 reset not still connected to 0x55efbb630d00 2024-04-06T09:58:12.208 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-04-06T09:58:12.204+0000 7fd1b6210640 -1 reset not still connected to 0x55efd81ee8f0 2024-04-06T09:58:12.208 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-04-06T09:58:12.204+0000 7fd1b6210640 -1 reset not still connected to 0x55efe06a9860 2024-04-06T09:58:14.145 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ error_code=124 2024-04-06T09:58:14.145 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set -e 2024-04-06T09:58:14.145 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 124 -eq 124 ']' 2024-04-06T09:58:14.145 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:58:14.145 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_pool_images cluster2 mirror 50 2024-04-06T09:58:14.145 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-04-06T09:58:14.145 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:58:14.145 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image_count=50 2024-04-06T09:58:14.146 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local s 2024-04-06T09:58:14.146 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local count 2024-04-06T09:58:14.146 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local last_count=0 2024-04-06T09:58:14.146 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:58:14.147 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 1 40 2024-04-06T09:58:14.325 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in `seq 1 40` 2024-04-06T09:58:14.325 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 1 -ne 1 2024-04-06T09:58:14.326 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 -p mirror mirror pool status 2024-04-06T09:58:14.326 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep 'images: ' 2024-04-06T09:58:14.326 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut '-d ' -f 2 2024-04-06T09:58:14.401 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:rbd: failed to query services: (13) Permission denied 2024-04-06T09:58:14.405 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ count=50 2024-04-06T09:58:14.406 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 50 = 50 2024-04-06T09:58:14.406 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:58:14.406 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_pool_healthy cluster2 mirror 2024-04-06T09:58:14.406 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-04-06T09:58:14.406 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:58:14.406 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local s 2024-04-06T09:58:14.406 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state 2024-04-06T09:58:14.406 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 1 40 2024-04-06T09:58:14.407 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in `seq 1 40` 2024-04-06T09:58:14.407 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 1 -ne 1 2024-04-06T09:58:14.407 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 -p mirror mirror pool status 2024-04-06T09:58:14.407 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep 'image health:' 2024-04-06T09:58:14.407 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut '-d ' -f 3 2024-04-06T09:58:14.448 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:rbd: failed to query services: (13) Permission denied 2024-04-06T09:58:14.452 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ state=WARNING 2024-04-06T09:58:14.452 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test WARNING = ERROR 2024-04-06T09:58:14.452 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test WARNING = OK 2024-04-06T09:58:14.452 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in `seq 1 40` 2024-04-06T09:58:14.453 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 2 -ne 1 2024-04-06T09:58:14.453 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 30 2024-04-06T09:58:14.710 INFO:tasks.ceph.cluster2.osd.2.smithi192.stderr:2024-04-06T09:58:14.709+0000 7fa534aba640 -1 reset not still connected to 0x55933ce91110 2024-04-06T09:58:14.710 INFO:tasks.ceph.cluster2.osd.2.smithi192.stderr:2024-04-06T09:58:14.709+0000 7fa534aba640 -1 reset not still connected to 0x55933ecf01a0 2024-04-06T09:58:14.710 INFO:tasks.ceph.cluster2.osd.2.smithi192.stderr:2024-04-06T09:58:14.709+0000 7fa534aba640 -1 reset not still connected to 0x55933fc2dd40 2024-04-06T09:58:14.710 INFO:tasks.ceph.cluster2.osd.2.smithi192.stderr:2024-04-06T09:58:14.709+0000 7fa534aba640 -1 reset not still connected to 0x559341658b60 2024-04-06T09:58:14.714 INFO:tasks.ceph.cluster2.osd.2.smithi192.stderr:2024-04-06T09:58:14.709+0000 7fa534aba640 -1 reset not still connected to 0x55934696eea0 2024-04-06T09:58:14.715 INFO:tasks.ceph.cluster2.osd.2.smithi192.stderr:2024-04-06T09:58:14.713+0000 7fa534aba640 -1 reset not still connected to 0x55934811f520 2024-04-06T09:58:14.715 INFO:tasks.ceph.cluster2.osd.2.smithi192.stderr:2024-04-06T09:58:14.713+0000 7fa534aba640 -1 reset not still connected to 0x55934811f6c0 2024-04-06T09:58:14.715 INFO:tasks.ceph.cluster2.osd.2.smithi192.stderr:2024-04-06T09:58:14.717+0000 7fa534aba640 -1 reset not still connected to 0x559348f02820 2024-04-06T09:58:14.721 INFO:tasks.ceph.cluster2.osd.2.smithi192.stderr:2024-04-06T09:58:14.725+0000 7fa534aba640 -1 reset not still connected to 0x55934feb6000 2024-04-06T09:58:14.721 INFO:tasks.ceph.cluster2.osd.2.smithi192.stderr:2024-04-06T09:58:14.725+0000 7fa534aba640 -1 reset not still connected to 0x5593550955f0 2024-04-06T09:58:14.721 INFO:tasks.ceph.cluster2.osd.2.smithi192.stderr:2024-04-06T09:58:14.725+0000 7fa534aba640 -1 reset not still connected to 0x55935bfc9d40 2024-04-06T09:58:14.721 INFO:tasks.ceph.cluster2.osd.2.smithi192.stderr:2024-04-06T09:58:14.725+0000 7fa534aba640 -1 reset not still connected to 0x55936c6f44e0 2024-04-06T09:58:15.544 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:58:15.544+0000 7f0965192640 -1 reset not still connected to 0x5632e16851e0 2024-04-06T09:58:17.205 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi192.stderr:2024-04-06T09:58:17.209+0000 7f3d28bce640 -1 rbd::mirror::InstanceWatcher: C_NotifyInstanceRequest: 0x56067b284d80 finish: resending after timeout 2024-04-06T09:58:17.206 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi192.stderr:2024-04-06T09:58:17.209+0000 7f3d293cf640 -1 rbd::mirror::InstanceWatcher: C_NotifyInstanceRequest: 0x560668d73c80 finish: resending after timeout 2024-04-06T09:58:17.206 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi192.stderr:2024-04-06T09:58:17.209+0000 7f3d28bce640 -1 rbd::mirror::InstanceWatcher: C_NotifyInstanceRequest: 0x56067b1d8b40 finish: resending after timeout 2024-04-06T09:58:17.247 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:58:17.244+0000 7f2e84d66640 -1 reset not still connected to 0x560d27839930 2024-04-06T09:58:17.256 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:58:17.252+0000 7f2e84d66640 -1 reset not still connected to 0x560d27c3edd0 2024-04-06T09:58:17.258 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:58:17.256+0000 7f2e84d66640 -1 reset not still connected to 0x560d27c3f2b0 2024-04-06T09:58:17.258 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:58:17.256+0000 7f2e84d66640 -1 reset not still connected to 0x560d294908f0 2024-04-06T09:58:17.258 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:58:17.256+0000 7f2e84d66640 -1 reset not still connected to 0x560d29490ea0 2024-04-06T09:58:17.258 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:58:17.256+0000 7f2e84d66640 -1 reset not still connected to 0x560d29cd9ba0 2024-04-06T09:58:17.258 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:58:17.256+0000 7f2e84d66640 -1 reset not still connected to 0x560d29d5b520 2024-04-06T09:58:17.264 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:58:17.264+0000 7f2e84d66640 -1 reset not still connected to 0x560d2a17d450 2024-04-06T09:58:17.264 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:58:17.264+0000 7f2e84d66640 -1 reset not still connected to 0x560d2a17d6c0 2024-04-06T09:58:17.264 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:58:17.264+0000 7f2e84d66640 -1 reset not still connected to 0x560d2a204b60 2024-04-06T09:58:17.264 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:58:17.264+0000 7f2e84d66640 -1 reset not still connected to 0x560d2a205ba0 2024-04-06T09:58:17.264 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:58:17.264+0000 7f2e84d66640 -1 reset not still connected to 0x560d2a2336c0 2024-04-06T09:58:17.264 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:58:17.264+0000 7f2e84d66640 -1 reset not still connected to 0x560d2a5705b0 2024-04-06T09:58:17.264 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:58:17.264+0000 7f2e84d66640 -1 reset not still connected to 0x560d35349380 2024-04-06T09:58:17.264 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:58:17.264+0000 7f2e84d66640 -1 reset not still connected to 0x560d3620b520 2024-04-06T09:58:17.264 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:58:17.264+0000 7f2e84d66640 -1 reset not still connected to 0x560d3620bee0 2024-04-06T09:58:17.265 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:58:17.264+0000 7f2e84d66640 -1 reset not still connected to 0x560d36666d00 2024-04-06T09:58:17.265 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:58:17.264+0000 7f2e84d66640 -1 reset not still connected to 0x560d368e4f70 2024-04-06T09:58:17.269 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:58:17.268+0000 7f2e84d66640 -1 reset not still connected to 0x560d381dd5f0 2024-04-06T09:58:17.269 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:58:17.268+0000 7f2e84d66640 -1 reset not still connected to 0x560d47e9aa90 2024-04-06T09:58:17.269 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:58:17.268+0000 7f2e84d66640 -1 reset not still connected to 0x560d4bc7e8f0 2024-04-06T09:58:17.270 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:58:17.268+0000 7f2e84d66640 -1 reset not still connected to 0x560d4bc7ec30 2024-04-06T09:58:17.270 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:58:17.268+0000 7f2e84d66640 -1 reset not still connected to 0x560d50adba00 2024-04-06T09:58:19.628 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:58:19.616+0000 7f0965192640 -1 reset not still connected to 0x5632a89f52b0 2024-04-06T09:58:19.628 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:58:19.620+0000 7f0965192640 -1 reset not still connected to 0x5632abc812b0 2024-04-06T09:58:19.631 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:58:19.624+0000 7f0965192640 -1 reset not still connected to 0x5632abc81d40 2024-04-06T09:58:19.631 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:58:19.624+0000 7f0965192640 -1 reset not still connected to 0x5632abcbe820 2024-04-06T09:58:19.634 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:58:19.628+0000 7f0965192640 -1 reset not still connected to 0x5632abcbf930 2024-04-06T09:58:19.635 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:58:19.628+0000 7f0965192640 -1 reset not still connected to 0x5632b81f3a00 2024-04-06T09:58:19.635 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:58:19.628+0000 7f0965192640 -1 reset not still connected to 0x5632b9592000 2024-04-06T09:58:19.635 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:58:19.628+0000 7f0965192640 -1 reset not still connected to 0x5632b9593110 2024-04-06T09:58:19.635 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:58:19.628+0000 7f0965192640 -1 reset not still connected to 0x5632c0cfa000 2024-04-06T09:58:19.635 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:58:19.628+0000 7f0965192640 -1 reset not still connected to 0x5632c0cfba00 2024-04-06T09:58:19.643 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:58:19.640+0000 7f0965192640 -1 reset not still connected to 0x5632c1c23790 2024-04-06T09:58:19.643 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:58:19.640+0000 7f0965192640 -1 reset not still connected to 0x5632c99855f0 2024-04-06T09:58:19.643 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:58:19.640+0000 7f0965192640 -1 reset not still connected to 0x5632cb3ac000 2024-04-06T09:58:19.643 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:58:19.640+0000 7f0965192640 -1 reset not still connected to 0x5632ceff5790 2024-04-06T09:58:19.643 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:58:19.640+0000 7f0965192640 -1 reset not still connected to 0x5632d8f88820 2024-04-06T09:58:19.643 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:58:19.640+0000 7f0965192640 -1 reset not still connected to 0x5632dac08dd0 2024-04-06T09:58:19.644 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:58:19.640+0000 7f0965192640 -1 reset not still connected to 0x5632dac09040 2024-04-06T09:58:19.644 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:58:19.640+0000 7f0965192640 -1 reset not still connected to 0x5632e1684ea0 2024-04-06T09:58:19.644 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:58:19.640+0000 7f0965192640 -1 reset not still connected to 0x5632e16851e0 2024-04-06T09:58:20.296 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:58:20.291+0000 7f2e84d66640 -1 reset not still connected to 0x560d27073860 2024-04-06T09:58:20.296 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:58:20.291+0000 7f2e84d66640 -1 reset not still connected to 0x560d27567380 2024-04-06T09:58:20.296 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:58:20.291+0000 7f2e84d66640 -1 reset not still connected to 0x560d27c3edd0 2024-04-06T09:58:20.296 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:58:20.291+0000 7f2e84d66640 -1 reset not still connected to 0x560d27c3f2b0 2024-04-06T09:58:20.296 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:58:20.291+0000 7f2e84d66640 -1 reset not still connected to 0x560d2a17d450 2024-04-06T09:58:20.296 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:58:20.291+0000 7f2e84d66640 -1 reset not still connected to 0x560d2a17d6c0 2024-04-06T09:58:20.296 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:58:20.291+0000 7f2e84d66640 -1 reset not still connected to 0x560d2a2336c0 2024-04-06T09:58:22.800 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:58:22.791+0000 7f2e84d66640 -1 reset not still connected to 0x560d27c3f2b0 2024-04-06T09:58:22.801 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:58:22.795+0000 7f2e84d66640 -1 reset not still connected to 0x560d29d5b520 2024-04-06T09:58:22.801 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:58:22.795+0000 7f2e84d66640 -1 reset not still connected to 0x560d2a17d450 2024-04-06T09:58:22.801 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:58:22.795+0000 7f2e84d66640 -1 reset not still connected to 0x560d2a17d6c0 2024-04-06T09:58:22.806 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:58:22.803+0000 7f2e84d66640 -1 reset not still connected to 0x560d2a205ba0 2024-04-06T09:58:22.806 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:58:22.803+0000 7f2e84d66640 -1 reset not still connected to 0x560d2a2336c0 2024-04-06T09:58:22.806 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:58:22.803+0000 7f2e84d66640 -1 reset not still connected to 0x560d2a5705b0 2024-04-06T09:58:22.807 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:58:22.803+0000 7f2e84d66640 -1 reset not still connected to 0x560d368e4f70 2024-04-06T09:58:22.808 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:58:22.803+0000 7f2e84d66640 -1 reset not still connected to 0x560d381dd5f0 2024-04-06T09:58:22.808 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:58:22.807+0000 7f2e84d66640 -1 reset not still connected to 0x560d50adba00 2024-04-06T09:58:23.826 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:58:23.823+0000 7f2e84d66640 -1 reset not still connected to 0x560d2a17d450 2024-04-06T09:58:23.831 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:58:23.827+0000 7f2e84d66640 -1 reset not still connected to 0x560d2a17d6c0 2024-04-06T09:58:23.832 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:58:23.827+0000 7f2e84d66640 -1 reset not still connected to 0x560d2a204b60 2024-04-06T09:58:23.832 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:58:23.827+0000 7f2e84d66640 -1 reset not still connected to 0x560d2a205ba0 2024-04-06T09:58:23.832 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:58:23.827+0000 7f2e84d66640 -1 reset not still connected to 0x560d2a2336c0 2024-04-06T09:58:23.832 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:58:23.827+0000 7f2e84d66640 -1 reset not still connected to 0x560d2a5705b0 2024-04-06T09:58:23.833 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:58:23.831+0000 7f2e84d66640 -1 reset not still connected to 0x560d3620b520 2024-04-06T09:58:23.846 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:58:23.843+0000 7f2e84d66640 -1 reset not still connected to 0x560d36666d00 2024-04-06T09:58:23.849 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:58:23.847+0000 7f2e84d66640 -1 reset not still connected to 0x560d368e4f70 2024-04-06T09:58:23.849 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:58:23.847+0000 7f2e84d66640 -1 reset not still connected to 0x560d36912ea0 2024-04-06T09:58:23.849 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:58:23.847+0000 7f2e84d66640 -1 reset not still connected to 0x560d381dcc30 2024-04-06T09:58:23.849 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:58:23.847+0000 7f2e84d66640 -1 reset not still connected to 0x560d381dd5f0 2024-04-06T09:58:23.849 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:58:23.847+0000 7f2e84d66640 -1 reset not still connected to 0x560d390dc8f0 2024-04-06T09:58:23.849 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:58:23.847+0000 7f2e84d66640 -1 reset not still connected to 0x560d390dca90 2024-04-06T09:58:23.849 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:58:23.847+0000 7f2e84d66640 -1 reset not still connected to 0x560d4bc7e8f0 2024-04-06T09:58:23.849 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:58:23.847+0000 7f2e84d66640 -1 reset not still connected to 0x560d4bc7ec30 2024-04-06T09:58:23.850 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:58:23.847+0000 7f2e84d66640 -1 reset not still connected to 0x560d4e9ddba0 2024-04-06T09:58:23.852 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:58:23.851+0000 7f2e84d66640 -1 reset not still connected to 0x560d50adba00 2024-04-06T09:58:23.852 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:58:23.851+0000 7f2e84d66640 -1 reset not still connected to 0x560d59635ad0 2024-04-06T09:58:24.057 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:58:24.055+0000 7f2e84d66640 -1 reset not still connected to 0x560d27567380 2024-04-06T09:58:24.059 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:58:24.055+0000 7f2e84d66640 -1 reset not still connected to 0x560d27c3edd0 2024-04-06T09:58:24.060 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:58:24.059+0000 7f2e84d66640 -1 reset not still connected to 0x560d2a17d450 2024-04-06T09:58:24.060 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:58:24.059+0000 7f2e84d66640 -1 reset not still connected to 0x560d2a17d6c0 2024-04-06T09:58:24.060 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:58:24.059+0000 7f2e84d66640 -1 reset not still connected to 0x560d2a2336c0 2024-04-06T09:58:27.568 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-04-06T09:58:27.567+0000 7fd1b6210640 -1 reset not still connected to 0x55ef9bd5c9c0 2024-04-06T09:58:27.568 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-04-06T09:58:27.567+0000 7fd1b6210640 -1 reset not still connected to 0x55ef9bd5d520 2024-04-06T09:58:27.568 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-04-06T09:58:27.567+0000 7fd1b6210640 -1 reset not still connected to 0x55ef9bd5dd40 2024-04-06T09:58:27.568 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-04-06T09:58:27.567+0000 7fd1b6210640 -1 reset not still connected to 0x55ef9bd86820 2024-04-06T09:58:27.569 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-04-06T09:58:27.567+0000 7fd1b6210640 -1 reset not still connected to 0x55ef9c103110 2024-04-06T09:58:27.569 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-04-06T09:58:27.567+0000 7fd1b6210640 -1 reset not still connected to 0x55ef9c103930 2024-04-06T09:58:27.569 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-04-06T09:58:27.567+0000 7fd1b6210640 -1 reset not still connected to 0x55ef9e698c30 2024-04-06T09:58:27.569 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-04-06T09:58:27.567+0000 7fd1b6210640 -1 reset not still connected to 0x55ef9e88f520 2024-04-06T09:58:27.569 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-04-06T09:58:27.567+0000 7fd1b6210640 -1 reset not still connected to 0x55ef9eee21a0 2024-04-06T09:58:27.569 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-04-06T09:58:27.567+0000 7fd1b6210640 -1 reset not still connected to 0x55ef9eee3a00 2024-04-06T09:58:27.574 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-04-06T09:58:27.571+0000 7fd1b6210640 -1 reset not still connected to 0x55efa18cc9c0 2024-04-06T09:58:27.574 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-04-06T09:58:27.571+0000 7fd1b6210640 -1 reset not still connected to 0x55efa18cca90 2024-04-06T09:58:27.574 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-04-06T09:58:27.571+0000 7fd1b6210640 -1 reset not still connected to 0x55efa96c96c0 2024-04-06T09:58:27.574 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-04-06T09:58:27.571+0000 7fd1b6210640 -1 reset not still connected to 0x55efae5ffe10 2024-04-06T09:58:27.574 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-04-06T09:58:27.571+0000 7fd1b6210640 -1 reset not still connected to 0x55efbb630d00 2024-04-06T09:58:27.576 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-04-06T09:58:27.575+0000 7fd1b6210640 -1 reset not still connected to 0x55efd81ee8f0 2024-04-06T09:58:27.576 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-04-06T09:58:27.575+0000 7fd1b6210640 -1 reset not still connected to 0x55efe06a9860 2024-04-06T09:58:29.434 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:58:29.427+0000 7f2e84d66640 -1 reset not still connected to 0x560d27839930 2024-04-06T09:58:29.434 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:58:29.427+0000 7f2e84d66640 -1 reset not still connected to 0x560d27c3edd0 2024-04-06T09:58:29.434 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:58:29.431+0000 7f2e84d66640 -1 reset not still connected to 0x560d27c3f2b0 2024-04-06T09:58:29.434 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:58:29.431+0000 7f2e84d66640 -1 reset not still connected to 0x560d294908f0 2024-04-06T09:58:29.446 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:58:29.443+0000 7f2e84d66640 -1 reset not still connected to 0x560d29cd8f70 2024-04-06T09:58:29.446 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:58:29.443+0000 7f2e84d66640 -1 reset not still connected to 0x560d29cd9ba0 2024-04-06T09:58:29.446 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:58:29.443+0000 7f2e84d66640 -1 reset not still connected to 0x560d29d5b520 2024-04-06T09:58:29.446 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:58:29.443+0000 7f2e84d66640 -1 reset not still connected to 0x560d2a17d450 2024-04-06T09:58:29.446 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:58:29.443+0000 7f2e84d66640 -1 reset not still connected to 0x560d2a17d6c0 2024-04-06T09:58:29.446 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:58:29.443+0000 7f2e84d66640 -1 reset not still connected to 0x560d2a204b60 2024-04-06T09:58:29.447 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:58:29.443+0000 7f2e84d66640 -1 reset not still connected to 0x560d2a205ba0 2024-04-06T09:58:29.447 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:58:29.443+0000 7f2e84d66640 -1 reset not still connected to 0x560d2a2336c0 2024-04-06T09:58:29.447 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:58:29.443+0000 7f2e84d66640 -1 reset not still connected to 0x560d2a5705b0 2024-04-06T09:58:29.447 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:58:29.443+0000 7f2e84d66640 -1 reset not still connected to 0x560d35349380 2024-04-06T09:58:29.447 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:58:29.443+0000 7f2e84d66640 -1 reset not still connected to 0x560d3620b520 2024-04-06T09:58:29.447 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:58:29.443+0000 7f2e84d66640 -1 reset not still connected to 0x560d3620bee0 2024-04-06T09:58:29.451 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:58:29.451+0000 7f2e84d66640 -1 reset not still connected to 0x560d36666d00 2024-04-06T09:58:29.451 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:58:29.451+0000 7f2e84d66640 -1 reset not still connected to 0x560d368e4f70 2024-04-06T09:58:29.452 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:58:29.451+0000 7f2e84d66640 -1 reset not still connected to 0x560d36912ea0 2024-04-06T09:58:29.452 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:58:29.451+0000 7f2e84d66640 -1 reset not still connected to 0x560d381dd5f0 2024-04-06T09:58:29.452 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:58:29.451+0000 7f2e84d66640 -1 reset not still connected to 0x560d390dca90 2024-04-06T09:58:29.452 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:58:29.451+0000 7f2e84d66640 -1 reset not still connected to 0x560d47e9aa90 2024-04-06T09:58:29.452 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:58:29.451+0000 7f2e84d66640 -1 reset not still connected to 0x560d4bc7e8f0 2024-04-06T09:58:29.452 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:58:29.451+0000 7f2e84d66640 -1 reset not still connected to 0x560d4bc7ec30 2024-04-06T09:58:29.452 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:58:29.451+0000 7f2e84d66640 -1 reset not still connected to 0x560d50adba00 2024-04-06T09:58:31.522 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:58:31.519+0000 7f2e84d66640 -1 reset not still connected to 0x560d381dd5f0 2024-04-06T09:58:31.534 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:58:31.527+0000 7f2e84d66640 -1 reset not still connected to 0x560d50adba00 2024-04-06T09:58:32.875 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:58:32.875+0000 7f2e84d66640 -1 reset not still connected to 0x560d36912ea0 2024-04-06T09:58:32.876 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:58:32.875+0000 7f2e84d66640 -1 reset not still connected to 0x560d381dd5f0 2024-04-06T09:58:32.876 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:58:32.875+0000 7f2e84d66640 -1 reset not still connected to 0x560d390dca90 2024-04-06T09:58:32.876 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:58:32.875+0000 7f2e84d66640 -1 reset not still connected to 0x560d4bc7e8f0 2024-04-06T09:58:32.876 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:58:32.875+0000 7f2e84d66640 -1 reset not still connected to 0x560d4bc7ec30 2024-04-06T09:58:33.350 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-04-06T09:58:33.347+0000 7fd1b6210640 -1 reset not still connected to 0x55ef99306ea0 2024-04-06T09:58:33.356 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-04-06T09:58:33.351+0000 7fd1b6210640 -1 reset not still connected to 0x55ef997761a0 2024-04-06T09:58:33.357 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-04-06T09:58:33.351+0000 7fd1b6210640 -1 reset not still connected to 0x55ef99776680 2024-04-06T09:58:33.357 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-04-06T09:58:33.351+0000 7fd1b6210640 -1 reset not still connected to 0x55ef99b88d00 2024-04-06T09:58:33.358 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-04-06T09:58:33.351+0000 7fd1b6210640 -1 reset not still connected to 0x55ef99b89ad0 2024-04-06T09:58:33.358 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-04-06T09:58:33.351+0000 7fd1b6210640 -1 reset not still connected to 0x55ef9ac60270 2024-04-06T09:58:33.358 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-04-06T09:58:33.351+0000 7fd1b6210640 -1 reset not still connected to 0x55ef9bbe9ee0 2024-04-06T09:58:33.358 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-04-06T09:58:33.351+0000 7fd1b6210640 -1 reset not still connected to 0x55ef9e698c30 2024-04-06T09:58:33.364 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-04-06T09:58:33.359+0000 7fd1b6210640 -1 reset not still connected to 0x55ef9eee21a0 2024-04-06T09:58:33.765 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:58:33.763+0000 7f2e84d66640 -1 reset not still connected to 0x560d27839380 2024-04-06T09:58:33.773 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:58:33.771+0000 7f2e84d66640 -1 reset not still connected to 0x560d27c3edd0 2024-04-06T09:58:33.779 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:58:33.775+0000 7f2e84d66640 -1 reset not still connected to 0x560d27c3f2b0 2024-04-06T09:58:33.779 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:58:33.775+0000 7f2e84d66640 -1 reset not still connected to 0x560d281875f0 2024-04-06T09:58:33.780 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:58:33.775+0000 7f2e84d66640 -1 reset not still connected to 0x560d294908f0 2024-04-06T09:58:33.780 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:58:33.775+0000 7f2e84d66640 -1 reset not still connected to 0x560d29cd8f70 2024-04-06T09:58:33.780 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:58:33.775+0000 7f2e84d66640 -1 reset not still connected to 0x560d29cd9ba0 2024-04-06T09:58:33.780 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:58:33.775+0000 7f2e84d66640 -1 reset not still connected to 0x560d29d5a340 2024-04-06T09:58:33.780 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:58:33.775+0000 7f2e84d66640 -1 reset not still connected to 0x560d29d5b520 2024-04-06T09:58:33.780 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:58:33.775+0000 7f2e84d66640 -1 reset not still connected to 0x560d2a17d450 2024-04-06T09:58:33.780 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:58:33.775+0000 7f2e84d66640 -1 reset not still connected to 0x560d2a17d6c0 2024-04-06T09:58:33.782 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:58:33.779+0000 7f2e84d66640 -1 reset not still connected to 0x560d2a204b60 2024-04-06T09:58:33.782 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:58:33.779+0000 7f2e84d66640 -1 reset not still connected to 0x560d2a205ba0 2024-04-06T09:58:33.782 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:58:33.779+0000 7f2e84d66640 -1 reset not still connected to 0x560d2a2336c0 2024-04-06T09:58:33.782 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:58:33.779+0000 7f2e84d66640 -1 reset not still connected to 0x560d2a570000 2024-04-06T09:58:33.782 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:58:33.779+0000 7f2e84d66640 -1 reset not still connected to 0x560d2a5705b0 2024-04-06T09:58:33.786 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:58:33.783+0000 7f2e84d66640 -1 reset not still connected to 0x560d3620b520 2024-04-06T09:58:33.792 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:58:33.787+0000 7f2e84d66640 -1 reset not still connected to 0x560d368e4f70 2024-04-06T09:58:33.792 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:58:33.787+0000 7f2e84d66640 -1 reset not still connected to 0x560d36912ea0 2024-04-06T09:58:33.792 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:58:33.787+0000 7f2e84d66640 -1 reset not still connected to 0x560d381dcc30 2024-04-06T09:58:33.792 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:58:33.787+0000 7f2e84d66640 -1 reset not still connected to 0x560d381dd5f0 2024-04-06T09:58:33.792 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:58:33.787+0000 7f2e84d66640 -1 reset not still connected to 0x560d390dc8f0 2024-04-06T09:58:33.793 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:58:33.787+0000 7f2e84d66640 -1 reset not still connected to 0x560d390dca90 2024-04-06T09:58:33.793 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:58:33.791+0000 7f2e84d66640 -1 reset not still connected to 0x560d47e9aa90 2024-04-06T09:58:33.794 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:58:33.791+0000 7f2e84d66640 -1 reset not still connected to 0x560d4bc7e8f0 2024-04-06T09:58:33.794 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:58:33.791+0000 7f2e84d66640 -1 reset not still connected to 0x560d4bc7ec30 2024-04-06T09:58:33.794 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:58:33.791+0000 7f2e84d66640 -1 reset not still connected to 0x560d4e9ddba0 2024-04-06T09:58:33.794 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:58:33.791+0000 7f2e84d66640 -1 reset not still connected to 0x560d50adba00 2024-04-06T09:58:33.795 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:58:33.791+0000 7f2e84d66640 -1 reset not still connected to 0x560d59635ad0 2024-04-06T09:58:33.795 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:58:33.791+0000 7f2e84d66640 -1 reset not still connected to 0x560d27073860 2024-04-06T09:58:33.795 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:58:33.791+0000 7f2e84d66640 -1 reset not still connected to 0x560d27c3f2b0 2024-04-06T09:58:33.795 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:58:33.791+0000 7f2e84d66640 -1 reset not still connected to 0x560d2a17d450 2024-04-06T09:58:33.795 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:58:33.791+0000 7f2e84d66640 -1 reset not still connected to 0x560d2a2336c0 2024-04-06T09:58:35.607 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-04-06T09:58:35.603+0000 7fd1b6210640 -1 reset not still connected to 0x55ef993061a0 2024-04-06T09:58:35.612 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-04-06T09:58:35.611+0000 7fd1b6210640 -1 reset not still connected to 0x55ef99306ea0 2024-04-06T09:58:35.612 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-04-06T09:58:35.611+0000 7fd1b6210640 -1 reset not still connected to 0x55ef997761a0 2024-04-06T09:58:35.613 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-04-06T09:58:35.611+0000 7fd1b6210640 -1 reset not still connected to 0x55ef99776680 2024-04-06T09:58:35.613 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-04-06T09:58:35.611+0000 7fd1b6210640 -1 reset not still connected to 0x55ef99b88000 2024-04-06T09:58:35.613 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-04-06T09:58:35.611+0000 7fd1b6210640 -1 reset not still connected to 0x55ef99b88d00 2024-04-06T09:58:35.613 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-04-06T09:58:35.611+0000 7fd1b6210640 -1 reset not still connected to 0x55ef99b89ad0 2024-04-06T09:58:35.613 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-04-06T09:58:35.611+0000 7fd1b6210640 -1 reset not still connected to 0x55ef99ebf6c0 2024-04-06T09:58:35.613 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-04-06T09:58:35.611+0000 7fd1b6210640 -1 reset not still connected to 0x55ef9ac60270 2024-04-06T09:58:35.613 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-04-06T09:58:35.611+0000 7fd1b6210640 -1 reset not still connected to 0x55ef9b657e10 2024-04-06T09:58:35.613 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-04-06T09:58:35.611+0000 7fd1b6210640 -1 reset not still connected to 0x55ef9bbe84e0 2024-04-06T09:58:35.613 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-04-06T09:58:35.611+0000 7fd1b6210640 -1 reset not still connected to 0x55ef9bbe8d00 2024-04-06T09:58:35.613 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-04-06T09:58:35.611+0000 7fd1b6210640 -1 reset not still connected to 0x55ef9bd5c9c0 2024-04-06T09:58:35.613 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-04-06T09:58:35.611+0000 7fd1b6210640 -1 reset not still connected to 0x55ef9bd5d520 2024-04-06T09:58:35.613 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-04-06T09:58:35.611+0000 7fd1b6210640 -1 reset not still connected to 0x55ef9bd5dd40 2024-04-06T09:58:35.613 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-04-06T09:58:35.611+0000 7fd1b6210640 -1 reset not still connected to 0x55ef9bd86820 2024-04-06T09:58:35.613 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-04-06T09:58:35.611+0000 7fd1b6210640 -1 reset not still connected to 0x55ef9c103110 2024-04-06T09:58:35.613 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-04-06T09:58:35.611+0000 7fd1b6210640 -1 reset not still connected to 0x55ef9c103930 2024-04-06T09:58:35.614 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-04-06T09:58:35.611+0000 7fd1b6210640 -1 reset not still connected to 0x55ef9e698c30 2024-04-06T09:58:35.614 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-04-06T09:58:35.611+0000 7fd1b6210640 -1 reset not still connected to 0x55ef9e88f520 2024-04-06T09:58:35.614 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-04-06T09:58:35.611+0000 7fd1b6210640 -1 reset not still connected to 0x55ef9eee21a0 2024-04-06T09:58:35.614 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-04-06T09:58:35.611+0000 7fd1b6210640 -1 reset not still connected to 0x55ef9eee3a00 2024-04-06T09:58:35.614 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-04-06T09:58:35.611+0000 7fd1b6210640 -1 reset not still connected to 0x55efbb630d00 2024-04-06T09:58:35.614 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-04-06T09:58:35.611+0000 7fd1b6210640 -1 reset not still connected to 0x55efd81ee8f0 2024-04-06T09:58:35.614 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-04-06T09:58:35.611+0000 7fd1b6210640 -1 reset not still connected to 0x55efe06a9860 2024-04-06T09:58:36.966 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-04-06T09:58:36.963+0000 7fd1b6210640 -1 reset not still connected to 0x55ef99776680 2024-04-06T09:58:36.966 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-04-06T09:58:36.963+0000 7fd1b6210640 -1 reset not still connected to 0x55ef99b88000 2024-04-06T09:58:36.966 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-04-06T09:58:36.963+0000 7fd1b6210640 -1 reset not still connected to 0x55ef99b88d00 2024-04-06T09:58:36.966 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-04-06T09:58:36.963+0000 7fd1b6210640 -1 reset not still connected to 0x55ef99b89ad0 2024-04-06T09:58:36.966 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-04-06T09:58:36.963+0000 7fd1b6210640 -1 reset not still connected to 0x55ef99ebf6c0 2024-04-06T09:58:36.966 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-04-06T09:58:36.963+0000 7fd1b6210640 -1 reset not still connected to 0x55ef9b657e10 2024-04-06T09:58:36.966 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-04-06T09:58:36.963+0000 7fd1b6210640 -1 reset not still connected to 0x55ef9bbe84e0 2024-04-06T09:58:36.966 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-04-06T09:58:36.963+0000 7fd1b6210640 -1 reset not still connected to 0x55ef9bbe8d00 2024-04-06T09:58:36.967 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-04-06T09:58:36.963+0000 7fd1b6210640 -1 reset not still connected to 0x55ef9bbe9ee0 2024-04-06T09:58:36.967 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-04-06T09:58:36.963+0000 7fd1b6210640 -1 reset not still connected to 0x55ef9bd5d520 2024-04-06T09:58:36.970 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-04-06T09:58:36.967+0000 7fd1b6210640 -1 reset not still connected to 0x55ef9bd86820 2024-04-06T09:58:36.970 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-04-06T09:58:36.967+0000 7fd1b6210640 -1 reset not still connected to 0x55ef9c103110 2024-04-06T09:58:36.971 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-04-06T09:58:36.967+0000 7fd1b6210640 -1 reset not still connected to 0x55ef9c103930 2024-04-06T09:58:36.971 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-04-06T09:58:36.967+0000 7fd1b6210640 -1 reset not still connected to 0x55ef9c103d40 2024-04-06T09:58:36.971 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-04-06T09:58:36.967+0000 7fd1b6210640 -1 reset not still connected to 0x55ef9e698c30 2024-04-06T09:58:36.971 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-04-06T09:58:36.967+0000 7fd1b6210640 -1 reset not still connected to 0x55ef9e88f520 2024-04-06T09:58:36.973 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-04-06T09:58:36.971+0000 7fd1b6210640 -1 reset not still connected to 0x55ef9eee21a0 2024-04-06T09:58:36.973 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-04-06T09:58:36.971+0000 7fd1b6210640 -1 reset not still connected to 0x55ef9eee3a00 2024-04-06T09:58:36.973 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-04-06T09:58:36.971+0000 7fd1b6210640 -1 reset not still connected to 0x55efa18cc270 2024-04-06T09:58:36.973 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-04-06T09:58:36.971+0000 7fd1b6210640 -1 reset not still connected to 0x55efa18cc9c0 2024-04-06T09:58:36.973 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-04-06T09:58:36.971+0000 7fd1b6210640 -1 reset not still connected to 0x55efa18cca90 2024-04-06T09:58:36.973 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-04-06T09:58:36.971+0000 7fd1b6210640 -1 reset not still connected to 0x55efa96c96c0 2024-04-06T09:58:36.973 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-04-06T09:58:36.971+0000 7fd1b6210640 -1 reset not still connected to 0x55efae5ffe10 2024-04-06T09:58:36.973 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-04-06T09:58:36.971+0000 7fd1b6210640 -1 reset not still connected to 0x55efbb630d00 2024-04-06T09:58:36.974 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-04-06T09:58:36.971+0000 7fd1b6210640 -1 reset not still connected to 0x55efd81ee8f0 2024-04-06T09:58:36.974 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-04-06T09:58:36.971+0000 7fd1b6210640 -1 reset not still connected to 0x55efe06a9860 2024-04-06T09:58:38.912 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-04-06T09:58:38.907+0000 7fd1b6210640 -1 reset not still connected to 0x55efd81ee8f0 2024-04-06T09:58:39.129 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-04-06T09:58:39.127+0000 7fd1b6210640 -1 reset not still connected to 0x55ef993061a0 2024-04-06T09:58:39.130 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-04-06T09:58:39.127+0000 7fd1b6210640 -1 reset not still connected to 0x55ef99306ea0 2024-04-06T09:58:39.130 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-04-06T09:58:39.127+0000 7fd1b6210640 -1 reset not still connected to 0x55ef997761a0 2024-04-06T09:58:39.130 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-04-06T09:58:39.127+0000 7fd1b6210640 -1 reset not still connected to 0x55ef99776680 2024-04-06T09:58:39.130 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-04-06T09:58:39.127+0000 7fd1b6210640 -1 reset not still connected to 0x55ef99b88000 2024-04-06T09:58:39.130 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-04-06T09:58:39.127+0000 7fd1b6210640 -1 reset not still connected to 0x55ef99b88d00 2024-04-06T09:58:39.130 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-04-06T09:58:39.127+0000 7fd1b6210640 -1 reset not still connected to 0x55ef99b89ad0 2024-04-06T09:58:39.130 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-04-06T09:58:39.127+0000 7fd1b6210640 -1 reset not still connected to 0x55ef99ebf6c0 2024-04-06T09:58:39.130 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-04-06T09:58:39.127+0000 7fd1b6210640 -1 reset not still connected to 0x55ef9ac60270 2024-04-06T09:58:39.130 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-04-06T09:58:39.127+0000 7fd1b6210640 -1 reset not still connected to 0x55ef9b657e10 2024-04-06T09:58:39.130 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-04-06T09:58:39.127+0000 7fd1b6210640 -1 reset not still connected to 0x55ef9bbe84e0 2024-04-06T09:58:39.133 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-04-06T09:58:39.127+0000 7fd1b6210640 -1 reset not still connected to 0x55ef9bbe8d00 2024-04-06T09:58:39.133 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-04-06T09:58:39.127+0000 7fd1b6210640 -1 reset not still connected to 0x55ef9bbe9ee0 2024-04-06T09:58:39.133 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-04-06T09:58:39.127+0000 7fd1b6210640 -1 reset not still connected to 0x55ef9bd5c9c0 2024-04-06T09:58:39.133 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-04-06T09:58:39.127+0000 7fd1b6210640 -1 reset not still connected to 0x55ef9bd5d520 2024-04-06T09:58:39.133 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-04-06T09:58:39.127+0000 7fd1b6210640 -1 reset not still connected to 0x55ef9bd5dd40 2024-04-06T09:58:39.133 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-04-06T09:58:39.127+0000 7fd1b6210640 -1 reset not still connected to 0x55ef9bd86820 2024-04-06T09:58:39.133 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-04-06T09:58:39.127+0000 7fd1b6210640 -1 reset not still connected to 0x55ef9c103110 2024-04-06T09:58:39.133 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-04-06T09:58:39.127+0000 7fd1b6210640 -1 reset not still connected to 0x55ef9c103930 2024-04-06T09:58:39.133 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-04-06T09:58:39.127+0000 7fd1b6210640 -1 reset not still connected to 0x55ef9c103d40 2024-04-06T09:58:39.133 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-04-06T09:58:39.127+0000 7fd1b6210640 -1 reset not still connected to 0x55ef9e698c30 2024-04-06T09:58:39.133 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-04-06T09:58:39.127+0000 7fd1b6210640 -1 reset not still connected to 0x55ef9e88f520 2024-04-06T09:58:39.133 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-04-06T09:58:39.127+0000 7fd1b6210640 -1 reset not still connected to 0x55ef9eee21a0 2024-04-06T09:58:39.133 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-04-06T09:58:39.127+0000 7fd1b6210640 -1 reset not still connected to 0x55ef9eee3a00 2024-04-06T09:58:39.133 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-04-06T09:58:39.127+0000 7fd1b6210640 -1 reset not still connected to 0x55efa18cc270 2024-04-06T09:58:39.134 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-04-06T09:58:39.127+0000 7fd1b6210640 -1 reset not still connected to 0x55efa18cc9c0 2024-04-06T09:58:39.137 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-04-06T09:58:39.131+0000 7fd1b6210640 -1 reset not still connected to 0x55efa18cca90 2024-04-06T09:58:39.137 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-04-06T09:58:39.131+0000 7fd1b6210640 -1 reset not still connected to 0x55efa96c96c0 2024-04-06T09:58:39.137 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-04-06T09:58:39.131+0000 7fd1b6210640 -1 reset not still connected to 0x55efae5ffe10 2024-04-06T09:58:39.137 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-04-06T09:58:39.131+0000 7fd1b6210640 -1 reset not still connected to 0x55efbb630d00 2024-04-06T09:58:39.137 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-04-06T09:58:39.131+0000 7fd1b6210640 -1 reset not still connected to 0x55efd81ee8f0 2024-04-06T09:58:39.137 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-04-06T09:58:39.131+0000 7fd1b6210640 -1 reset not still connected to 0x55efe06a9860 2024-04-06T09:58:40.527 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:58:40.523+0000 7f0965192640 -1 reset not still connected to 0x5632abcbe820 2024-04-06T09:58:40.527 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:58:40.523+0000 7f0965192640 -1 reset not still connected to 0x5632c0cfa000 2024-04-06T09:58:40.536 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:58:40.531+0000 7f0965192640 -1 reset not still connected to 0x5632c99855f0 2024-04-06T09:58:40.537 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:58:40.531+0000 7f0965192640 -1 reset not still connected to 0x5632cb3ac000 2024-04-06T09:58:40.537 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:58:40.531+0000 7f0965192640 -1 reset not still connected to 0x5632d8f88820 2024-04-06T09:58:40.537 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:58:40.531+0000 7f0965192640 -1 reset not still connected to 0x5632dac09040 2024-04-06T09:58:40.537 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:58:40.531+0000 7f0965192640 -1 reset not still connected to 0x5632e1684ea0 2024-04-06T09:58:40.541 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:58:40.539+0000 7f0965192640 -1 reset not still connected to 0x5632e16851e0 2024-04-06T09:58:42.042 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:58:42.039+0000 7f2e84d66640 -1 reset not still connected to 0x560d27073860 2024-04-06T09:58:42.042 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:58:42.039+0000 7f2e84d66640 -1 reset not still connected to 0x560d27567380 2024-04-06T09:58:43.000 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-04-06T09:58:42.991+0000 7fd1b6210640 -1 reset not still connected to 0x55ef9ac60270 2024-04-06T09:58:43.000 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-04-06T09:58:42.995+0000 7fd1b6210640 -1 reset not still connected to 0x55ef9b657e10 2024-04-06T09:58:43.001 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-04-06T09:58:42.995+0000 7fd1b6210640 -1 reset not still connected to 0x55ef9bbe84e0 2024-04-06T09:58:43.001 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-04-06T09:58:42.995+0000 7fd1b6210640 -1 reset not still connected to 0x55ef9bbe9ee0 2024-04-06T09:58:43.001 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-04-06T09:58:42.995+0000 7fd1b6210640 -1 reset not still connected to 0x55ef9bd5c9c0 2024-04-06T09:58:43.001 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-04-06T09:58:42.995+0000 7fd1b6210640 -1 reset not still connected to 0x55ef9bd5d520 2024-04-06T09:58:43.001 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-04-06T09:58:42.995+0000 7fd1b6210640 -1 reset not still connected to 0x55ef9bd5dd40 2024-04-06T09:58:43.001 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-04-06T09:58:42.995+0000 7fd1b6210640 -1 reset not still connected to 0x55ef9bd86820 2024-04-06T09:58:43.001 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-04-06T09:58:42.995+0000 7fd1b6210640 -1 reset not still connected to 0x55ef9c103110 2024-04-06T09:58:43.005 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-04-06T09:58:42.999+0000 7fd1b6210640 -1 reset not still connected to 0x55ef9c103930 2024-04-06T09:58:43.005 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-04-06T09:58:42.999+0000 7fd1b6210640 -1 reset not still connected to 0x55ef9c103d40 2024-04-06T09:58:43.006 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-04-06T09:58:42.999+0000 7fd1b6210640 -1 reset not still connected to 0x55ef9e698c30 2024-04-06T09:58:43.006 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-04-06T09:58:42.999+0000 7fd1b6210640 -1 reset not still connected to 0x55ef9e88f520 2024-04-06T09:58:43.006 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-04-06T09:58:42.999+0000 7fd1b6210640 -1 reset not still connected to 0x55ef9eee21a0 2024-04-06T09:58:43.006 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-04-06T09:58:42.999+0000 7fd1b6210640 -1 reset not still connected to 0x55ef9eee3a00 2024-04-06T09:58:43.006 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-04-06T09:58:42.999+0000 7fd1b6210640 -1 reset not still connected to 0x55efa18cc270 2024-04-06T09:58:43.006 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-04-06T09:58:42.999+0000 7fd1b6210640 -1 reset not still connected to 0x55efa18cc9c0 2024-04-06T09:58:43.006 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-04-06T09:58:43.003+0000 7fd1b6210640 -1 reset not still connected to 0x55efa96c96c0 2024-04-06T09:58:43.006 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-04-06T09:58:43.003+0000 7fd1b6210640 -1 reset not still connected to 0x55efbb630d00 2024-04-06T09:58:43.006 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-04-06T09:58:43.003+0000 7fd1b6210640 -1 reset not still connected to 0x55efd81ee8f0 2024-04-06T09:58:43.006 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-04-06T09:58:43.003+0000 7fd1b6210640 -1 reset not still connected to 0x55efe06a9860 2024-04-06T09:58:43.118 INFO:tasks.ceph.cluster2.osd.1.smithi192.stderr:2024-04-06T09:58:43.120+0000 7f0de0719640 -1 reset not still connected to 0x55bfa8bcf6c0 2024-04-06T09:58:43.118 INFO:tasks.ceph.cluster2.osd.1.smithi192.stderr:2024-04-06T09:58:43.120+0000 7f0de0719640 -1 reset not still connected to 0x55bfaef3ba00 2024-04-06T09:58:43.118 INFO:tasks.ceph.cluster2.osd.1.smithi192.stderr:2024-04-06T09:58:43.120+0000 7f0de0719640 -1 reset not still connected to 0x55bfaef3bc70 2024-04-06T09:58:43.118 INFO:tasks.ceph.cluster2.osd.1.smithi192.stderr:2024-04-06T09:58:43.120+0000 7f0de0719640 -1 reset not still connected to 0x55bfb5cb6270 2024-04-06T09:58:43.118 INFO:tasks.ceph.cluster2.osd.1.smithi192.stderr:2024-04-06T09:58:43.120+0000 7f0de0719640 -1 reset not still connected to 0x55bfbd8f9d40 2024-04-06T09:58:43.119 INFO:tasks.ceph.cluster2.osd.1.smithi192.stderr:2024-04-06T09:58:43.120+0000 7f0de0719640 -1 reset not still connected to 0x55bfd0d3b450 2024-04-06T09:58:43.119 INFO:tasks.ceph.cluster2.osd.1.smithi192.stderr:2024-04-06T09:58:43.120+0000 7f0de0719640 -1 reset not still connected to 0x55bfd74fb450 2024-04-06T09:58:43.119 INFO:tasks.ceph.cluster2.osd.1.smithi192.stderr:2024-04-06T09:58:43.120+0000 7f0de0719640 -1 reset not still connected to 0x55bfd764cdd0 2024-04-06T09:58:43.277 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:58:43.271+0000 7f2e84d66640 -1 reset not still connected to 0x560d27c3edd0 2024-04-06T09:58:43.277 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:58:43.271+0000 7f2e84d66640 -1 reset not still connected to 0x560d27c3f2b0 2024-04-06T09:58:43.277 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:58:43.271+0000 7f2e84d66640 -1 reset not still connected to 0x560d2a17d450 2024-04-06T09:58:43.278 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:58:43.271+0000 7f2e84d66640 -1 reset not still connected to 0x560d2a2336c0 2024-04-06T09:58:44.467 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 -p mirror mirror pool status 2024-04-06T09:58:44.468 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep 'image health:' 2024-04-06T09:58:44.468 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut '-d ' -f 3 2024-04-06T09:58:44.538 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:rbd: failed to query services: (13) Permission denied 2024-04-06T09:58:44.542 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ state=WARNING 2024-04-06T09:58:44.542 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test WARNING = ERROR 2024-04-06T09:58:44.543 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test WARNING = OK 2024-04-06T09:58:44.543 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in `seq 1 40` 2024-04-06T09:58:44.543 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 3 -ne 1 2024-04-06T09:58:44.543 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 30 2024-04-06T09:58:45.915 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:58:45.907+0000 7f2e84d66640 -1 reset not still connected to 0x560d294908f0 2024-04-06T09:58:45.915 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:58:45.907+0000 7f2e84d66640 -1 reset not still connected to 0x560d29d5b520 2024-04-06T09:58:45.917 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:58:45.915+0000 7f2e84d66640 -1 reset not still connected to 0x560d2a17d450 2024-04-06T09:58:45.917 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:58:45.915+0000 7f2e84d66640 -1 reset not still connected to 0x560d2a17d6c0 2024-04-06T09:58:45.917 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:58:45.915+0000 7f2e84d66640 -1 reset not still connected to 0x560d2a204b60 2024-04-06T09:58:45.918 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:58:45.915+0000 7f2e84d66640 -1 reset not still connected to 0x560d2a205ba0 2024-04-06T09:58:45.918 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:58:45.915+0000 7f2e84d66640 -1 reset not still connected to 0x560d2a2336c0 2024-04-06T09:58:45.918 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:58:45.915+0000 7f2e84d66640 -1 reset not still connected to 0x560d2a5705b0 2024-04-06T09:58:45.918 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:58:45.915+0000 7f2e84d66640 -1 reset not still connected to 0x560d35349380 2024-04-06T09:58:45.918 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:58:45.915+0000 7f2e84d66640 -1 reset not still connected to 0x560d3620b520 2024-04-06T09:58:45.919 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:58:45.919+0000 7f2e84d66640 -1 reset not still connected to 0x560d3620bee0 2024-04-06T09:58:45.925 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:58:45.923+0000 7f2e84d66640 -1 reset not still connected to 0x560d36666d00 2024-04-06T09:58:45.925 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:58:45.923+0000 7f2e84d66640 -1 reset not still connected to 0x560d368e4f70 2024-04-06T09:58:45.932 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:58:45.927+0000 7f2e84d66640 -1 reset not still connected to 0x560d36912ea0 2024-04-06T09:58:45.932 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:58:45.927+0000 7f2e84d66640 -1 reset not still connected to 0x560d381dd5f0 2024-04-06T09:58:45.932 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:58:45.927+0000 7f2e84d66640 -1 reset not still connected to 0x560d47e9aa90 2024-04-06T09:58:45.932 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:58:45.927+0000 7f2e84d66640 -1 reset not still connected to 0x560d4bc7e8f0 2024-04-06T09:58:45.932 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:58:45.927+0000 7f2e84d66640 -1 reset not still connected to 0x560d4bc7ec30 2024-04-06T09:58:45.932 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T09:58:45.927+0000 7f2e84d66640 -1 reset not still connected to 0x560d50adba00 2024-04-06T09:58:46.265 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:58:46.263+0000 7f0965192640 -1 reset not still connected to 0x5632e16851e0 2024-04-06T09:58:46.329 INFO:tasks.ceph.cluster2.osd.1.smithi192.stderr:2024-04-06T09:58:46.328+0000 7f0de0719640 -1 reset not still connected to 0x55bfa5642c30 2024-04-06T09:58:46.329 INFO:tasks.ceph.cluster2.osd.1.smithi192.stderr:2024-04-06T09:58:46.328+0000 7f0de0719640 -1 reset not still connected to 0x55bfa6e48410 2024-04-06T09:58:46.329 INFO:tasks.ceph.cluster2.osd.1.smithi192.stderr:2024-04-06T09:58:46.328+0000 7f0de0719640 -1 reset not still connected to 0x55bfa88ff040 2024-04-06T09:58:46.330 INFO:tasks.ceph.cluster2.osd.1.smithi192.stderr:2024-04-06T09:58:46.332+0000 7f0de0719640 -1 reset not still connected to 0x55bfaef3ba00 2024-04-06T09:58:46.330 INFO:tasks.ceph.cluster2.osd.1.smithi192.stderr:2024-04-06T09:58:46.332+0000 7f0de0719640 -1 reset not still connected to 0x55bfaef3bc70 2024-04-06T09:58:46.330 INFO:tasks.ceph.cluster2.osd.1.smithi192.stderr:2024-04-06T09:58:46.332+0000 7f0de0719640 -1 reset not still connected to 0x55bfbd8f9d40 2024-04-06T09:58:46.330 INFO:tasks.ceph.cluster2.osd.1.smithi192.stderr:2024-04-06T09:58:46.332+0000 7f0de0719640 -1 reset not still connected to 0x55bfd74fb450 2024-04-06T09:58:46.330 INFO:tasks.ceph.cluster2.osd.1.smithi192.stderr:2024-04-06T09:58:46.332+0000 7f0de0719640 -1 reset not still connected to 0x55bfd764cdd0 2024-04-06T09:58:47.734 INFO:tasks.ceph.cluster2.osd.2.smithi192.stderr:2024-04-06T09:58:47.732+0000 7fa534aba640 -1 reset not still connected to 0x55933640f520 2024-04-06T09:58:47.734 INFO:tasks.ceph.cluster2.osd.2.smithi192.stderr:2024-04-06T09:58:47.732+0000 7fa534aba640 -1 reset not still connected to 0x55933640fba0 2024-04-06T09:58:47.735 INFO:tasks.ceph.cluster2.osd.2.smithi192.stderr:2024-04-06T09:58:47.732+0000 7fa534aba640 -1 reset not still connected to 0x559337f8c340 2024-04-06T09:58:47.735 INFO:tasks.ceph.cluster2.osd.2.smithi192.stderr:2024-04-06T09:58:47.732+0000 7fa534aba640 -1 reset not still connected to 0x559338f41040 2024-04-06T09:58:47.735 INFO:tasks.ceph.cluster2.osd.2.smithi192.stderr:2024-04-06T09:58:47.736+0000 7fa534aba640 -1 reset not still connected to 0x55933bc92d00 2024-04-06T09:58:47.735 INFO:tasks.ceph.cluster2.osd.2.smithi192.stderr:2024-04-06T09:58:47.736+0000 7fa534aba640 -1 reset not still connected to 0x55933ce91110 2024-04-06T09:58:47.735 INFO:tasks.ceph.cluster2.osd.2.smithi192.stderr:2024-04-06T09:58:47.736+0000 7fa534aba640 -1 reset not still connected to 0x55933ecf01a0 2024-04-06T09:58:47.735 INFO:tasks.ceph.cluster2.osd.2.smithi192.stderr:2024-04-06T09:58:47.736+0000 7fa534aba640 -1 reset not still connected to 0x55933fc2dd40 2024-04-06T09:58:47.735 INFO:tasks.ceph.cluster2.osd.2.smithi192.stderr:2024-04-06T09:58:47.736+0000 7fa534aba640 -1 reset not still connected to 0x559341658b60 2024-04-06T09:58:47.735 INFO:tasks.ceph.cluster2.osd.2.smithi192.stderr:2024-04-06T09:58:47.736+0000 7fa534aba640 -1 reset not still connected to 0x55934811f6c0 2024-04-06T09:58:47.735 INFO:tasks.ceph.cluster2.osd.2.smithi192.stderr:2024-04-06T09:58:47.736+0000 7fa534aba640 -1 reset not still connected to 0x559348d9ad00 2024-04-06T09:58:47.742 INFO:tasks.ceph.cluster2.osd.2.smithi192.stderr:2024-04-06T09:58:47.736+0000 7fa534aba640 -1 reset not still connected to 0x559348d9b860 2024-04-06T09:58:47.743 INFO:tasks.ceph.cluster2.osd.2.smithi192.stderr:2024-04-06T09:58:47.736+0000 7fa534aba640 -1 reset not still connected to 0x559348f02820 2024-04-06T09:58:47.743 INFO:tasks.ceph.cluster2.osd.2.smithi192.stderr:2024-04-06T09:58:47.736+0000 7fa534aba640 -1 reset not still connected to 0x55934feb6000 2024-04-06T09:58:47.743 INFO:tasks.ceph.cluster2.osd.2.smithi192.stderr:2024-04-06T09:58:47.736+0000 7fa534aba640 -1 reset not still connected to 0x55935bfc9d40 2024-04-06T09:58:47.743 INFO:tasks.ceph.cluster2.osd.2.smithi192.stderr:2024-04-06T09:58:47.740+0000 7fa534aba640 -1 reset not still connected to 0x55936c6f44e0 2024-04-06T09:58:48.089 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-04-06T09:58:48.087+0000 7fd1b6210640 -1 reset not still connected to 0x55ef9ac60270 2024-04-06T09:58:48.090 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-04-06T09:58:48.087+0000 7fd1b6210640 -1 reset not still connected to 0x55ef9b657e10 2024-04-06T09:58:48.090 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-04-06T09:58:48.087+0000 7fd1b6210640 -1 reset not still connected to 0x55ef9bbe84e0 2024-04-06T09:58:48.090 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-04-06T09:58:48.087+0000 7fd1b6210640 -1 reset not still connected to 0x55ef9bbe8d00 2024-04-06T09:58:48.090 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-04-06T09:58:48.087+0000 7fd1b6210640 -1 reset not still connected to 0x55ef9bbe9ee0 2024-04-06T09:58:48.090 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-04-06T09:58:48.087+0000 7fd1b6210640 -1 reset not still connected to 0x55ef9bd5c9c0 2024-04-06T09:58:48.090 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-04-06T09:58:48.087+0000 7fd1b6210640 -1 reset not still connected to 0x55ef9bd5d520 2024-04-06T09:58:48.090 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-04-06T09:58:48.087+0000 7fd1b6210640 -1 reset not still connected to 0x55ef9bd5dd40 2024-04-06T09:58:48.090 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-04-06T09:58:48.087+0000 7fd1b6210640 -1 reset not still connected to 0x55ef9bd86820 2024-04-06T09:58:48.090 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-04-06T09:58:48.087+0000 7fd1b6210640 -1 reset not still connected to 0x55ef9c103110 2024-04-06T09:58:48.090 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-04-06T09:58:48.087+0000 7fd1b6210640 -1 reset not still connected to 0x55ef9c103930 2024-04-06T09:58:48.090 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-04-06T09:58:48.087+0000 7fd1b6210640 -1 reset not still connected to 0x55ef9c103d40 2024-04-06T09:58:48.091 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-04-06T09:58:48.087+0000 7fd1b6210640 -1 reset not still connected to 0x55ef9e698c30 2024-04-06T09:58:48.091 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-04-06T09:58:48.087+0000 7fd1b6210640 -1 reset not still connected to 0x55ef9e88f520 2024-04-06T09:58:48.091 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-04-06T09:58:48.087+0000 7fd1b6210640 -1 reset not still connected to 0x55ef9eee21a0 2024-04-06T09:58:48.091 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-04-06T09:58:48.087+0000 7fd1b6210640 -1 reset not still connected to 0x55ef9eee3a00 2024-04-06T09:58:48.091 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-04-06T09:58:48.087+0000 7fd1b6210640 -1 reset not still connected to 0x55efa18cc270 2024-04-06T09:58:48.091 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-04-06T09:58:48.087+0000 7fd1b6210640 -1 reset not still connected to 0x55efa18cc9c0 2024-04-06T09:58:48.091 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-04-06T09:58:48.087+0000 7fd1b6210640 -1 reset not still connected to 0x55efa96c96c0 2024-04-06T09:58:48.094 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-04-06T09:58:48.091+0000 7fd1b6210640 -1 reset not still connected to 0x55efbb630d00 2024-04-06T09:58:48.094 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-04-06T09:58:48.091+0000 7fd1b6210640 -1 reset not still connected to 0x55efd81ee8f0 2024-04-06T09:58:48.096 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-04-06T09:58:48.095+0000 7fd1b6210640 -1 reset not still connected to 0x55efe06a9860 2024-04-06T09:58:51.820 INFO:tasks.ceph.cluster2.osd.1.smithi192.stderr:2024-04-06T09:58:51.820+0000 7f0dd0ab0640 -1 bdev(0x55bfa195ce00 /var/lib/ceph/osd/cluster2-1/block) aio_submit retries 4 2024-04-06T09:58:51.824 INFO:tasks.ceph.cluster2.osd.1.smithi192.stderr:2024-04-06T09:58:51.824+0000 7f0dd32b5640 -1 bdev(0x55bfa195ce00 /var/lib/ceph/osd/cluster2-1/block) aio_submit retries 13 2024-04-06T09:58:53.096 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi192.stderr:2024-04-06T09:58:53.096+0000 7f3d28bce640 -1 rbd::mirror::InstanceWatcher: C_NotifyInstanceRequest: 0x560668d73c80 finish: resending after timeout 2024-04-06T09:58:56.534 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-04-06T09:58:56.531+0000 7fd1b6210640 -1 reset not still connected to 0x55ef9bbe9ee0 2024-04-06T09:58:56.544 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-04-06T09:58:56.543+0000 7fd1b6210640 -1 reset not still connected to 0x55ef9bd5c9c0 2024-04-06T09:58:56.545 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-04-06T09:58:56.543+0000 7fd1b6210640 -1 reset not still connected to 0x55ef9bd5d520 2024-04-06T09:58:56.545 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-04-06T09:58:56.543+0000 7fd1b6210640 -1 reset not still connected to 0x55ef9bd5dd40 2024-04-06T09:58:56.545 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-04-06T09:58:56.543+0000 7fd1b6210640 -1 reset not still connected to 0x55ef9bd86820 2024-04-06T09:58:56.545 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-04-06T09:58:56.543+0000 7fd1b6210640 -1 reset not still connected to 0x55ef9c103110 2024-04-06T09:58:56.546 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-04-06T09:58:56.543+0000 7fd1b6210640 -1 reset not still connected to 0x55ef9c103930 2024-04-06T09:58:56.547 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-04-06T09:58:56.543+0000 7fd1b6210640 -1 reset not still connected to 0x55ef9c103d40 2024-04-06T09:58:56.547 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-04-06T09:58:56.543+0000 7fd1b6210640 -1 reset not still connected to 0x55ef9e698c30 2024-04-06T09:58:56.547 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-04-06T09:58:56.543+0000 7fd1b6210640 -1 reset not still connected to 0x55ef9e88f520 2024-04-06T09:58:56.547 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-04-06T09:58:56.543+0000 7fd1b6210640 -1 reset not still connected to 0x55ef9eee21a0 2024-04-06T09:58:56.553 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-04-06T09:58:56.551+0000 7fd1b6210640 -1 reset not still connected to 0x55ef9eee3a00 2024-04-06T09:58:56.553 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-04-06T09:58:56.551+0000 7fd1b6210640 -1 reset not still connected to 0x55efa18cc270 2024-04-06T09:58:56.554 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-04-06T09:58:56.551+0000 7fd1b6210640 -1 reset not still connected to 0x55efa18cc9c0 2024-04-06T09:58:56.554 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-04-06T09:58:56.551+0000 7fd1b6210640 -1 reset not still connected to 0x55efa18cca90 2024-04-06T09:58:56.554 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-04-06T09:58:56.551+0000 7fd1b6210640 -1 reset not still connected to 0x55efa96c96c0 2024-04-06T09:58:56.554 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-04-06T09:58:56.551+0000 7fd1b6210640 -1 reset not still connected to 0x55efae5ffe10 2024-04-06T09:58:56.554 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-04-06T09:58:56.551+0000 7fd1b6210640 -1 reset not still connected to 0x55efbb630d00 2024-04-06T09:58:56.554 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-04-06T09:58:56.551+0000 7fd1b6210640 -1 reset not still connected to 0x55efd81ee8f0 2024-04-06T09:58:56.554 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-04-06T09:58:56.551+0000 7fd1b6210640 -1 reset not still connected to 0x55efe06a9860 2024-04-06T09:58:57.099 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-04-06T09:58:57.095+0000 7fd1b6210640 -1 reset not still connected to 0x55ef9e88f520 2024-04-06T09:58:57.109 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-04-06T09:58:57.107+0000 7fd1b6210640 -1 reset not still connected to 0x55efa96c96c0 2024-04-06T09:58:57.115 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-04-06T09:58:57.111+0000 7fd1b6210640 -1 reset not still connected to 0x55efae5ffe10 2024-04-06T09:58:57.115 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-04-06T09:58:57.111+0000 7fd1b6210640 -1 reset not still connected to 0x55efbb630d00 2024-04-06T09:58:57.115 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-04-06T09:58:57.111+0000 7fd1b6210640 -1 reset not still connected to 0x55efd81ee8f0 2024-04-06T09:58:57.115 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-04-06T09:58:57.111+0000 7fd1b6210640 -1 reset not still connected to 0x55efe06a9860 2024-04-06T09:58:58.839 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T09:58:58.835+0000 7f0965192640 -1 reset not still connected to 0x5632e16851e0 2024-04-06T09:59:02.918 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-04-06T09:59:02.903+0000 7fd1b6210640 -1 reset not still connected to 0x55ef99b88000 2024-04-06T09:59:02.918 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-04-06T09:59:02.903+0000 7fd1b6210640 -1 reset not still connected to 0x55ef99b88d00 2024-04-06T09:59:02.918 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-04-06T09:59:02.903+0000 7fd1b6210640 -1 reset not still connected to 0x55ef99b89ad0 2024-04-06T09:59:02.918 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-04-06T09:59:02.903+0000 7fd1b6210640 -1 reset not still connected to 0x55ef99ebf6c0 2024-04-06T09:59:02.918 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-04-06T09:59:02.903+0000 7fd1b6210640 -1 reset not still connected to 0x55ef9ac60270 2024-04-06T09:59:02.919 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-04-06T09:59:02.903+0000 7fd1b6210640 -1 reset not still connected to 0x55ef9bbe84e0 2024-04-06T09:59:02.919 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-04-06T09:59:02.911+0000 7fd1b6210640 -1 reset not still connected to 0x55ef9bbe9ee0 2024-04-06T09:59:02.919 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-04-06T09:59:02.911+0000 7fd1b6210640 -1 reset not still connected to 0x55ef9bd5c9c0 2024-04-06T09:59:02.919 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-04-06T09:59:02.911+0000 7fd1b6210640 -1 reset not still connected to 0x55ef9bd5d520 2024-04-06T09:59:02.919 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-04-06T09:59:02.911+0000 7fd1b6210640 -1 reset not still connected to 0x55ef9bd86820 2024-04-06T09:59:02.919 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-04-06T09:59:02.911+0000 7fd1b6210640 -1 reset not still connected to 0x55ef9c103110 2024-04-06T09:59:02.921 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-04-06T09:59:02.919+0000 7fd1b6210640 -1 reset not still connected to 0x55ef9c103d40 2024-04-06T09:59:02.922 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-04-06T09:59:02.919+0000 7fd1b6210640 -1 reset not still connected to 0x55ef9e698c30 2024-04-06T09:59:02.922 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-04-06T09:59:02.919+0000 7fd1b6210640 -1 reset not still connected to 0x55ef9e88f520 2024-04-06T09:59:02.922 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-04-06T09:59:02.919+0000 7fd1b6210640 -1 reset not still connected to 0x55ef9eee21a0 2024-04-06T09:59:02.922 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-04-06T09:59:02.919+0000 7fd1b6210640 -1 reset not still connected to 0x55ef9eee3a00 2024-04-06T09:59:02.922 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-04-06T09:59:02.919+0000 7fd1b6210640 -1 reset not still connected to 0x55efa18cc270 2024-04-06T09:59:02.922 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-04-06T09:59:02.919+0000 7fd1b6210640 -1 reset not still connected to 0x55efa18cca90 2024-04-06T09:59:02.922 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-04-06T09:59:02.919+0000 7fd1b6210640 -1 reset not still connected to 0x55efa96c96c0 2024-04-06T09:59:02.922 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-04-06T09:59:02.919+0000 7fd1b6210640 -1 reset not still connected to 0x55efae5ffe10 2024-04-06T09:59:02.922 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-04-06T09:59:02.919+0000 7fd1b6210640 -1 reset not still connected to 0x55efbb630d00 2024-04-06T09:59:02.923 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-04-06T09:59:02.919+0000 7fd1b6210640 -1 reset not still connected to 0x55efd81ee8f0 2024-04-06T09:59:02.923 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-04-06T09:59:02.919+0000 7fd1b6210640 -1 reset not still connected to 0x55efe06a9860 2024-04-06T09:59:03.317 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-04-06T09:59:03.315+0000 7fd1b6210640 -1 reset not still connected to 0x55ef993061a0 2024-04-06T09:59:03.317 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-04-06T09:59:03.315+0000 7fd1b6210640 -1 reset not still connected to 0x55ef997761a0 2024-04-06T09:59:03.326 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-04-06T09:59:03.315+0000 7fd1b6210640 -1 reset not still connected to 0x55ef99776680 2024-04-06T09:59:03.326 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-04-06T09:59:03.315+0000 7fd1b6210640 -1 reset not still connected to 0x55ef99b88d00 2024-04-06T09:59:03.327 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-04-06T09:59:03.315+0000 7fd1b6210640 -1 reset not still connected to 0x55ef99b89ad0 2024-04-06T09:59:03.327 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-04-06T09:59:03.327+0000 7fd1b6210640 -1 reset not still connected to 0x55ef9ac60270 2024-04-06T09:59:11.644 INFO:tasks.rbd_mirror_thrash:kill cluster1.client.mirror.0 2024-04-06T09:59:11.644 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Sent signal 15 2024-04-06T09:59:11.644 INFO:tasks.rbd_mirror_thrash:waiting for 5 secs before reviving daemons 2024-04-06T09:59:11.645 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi192.stderr:2024-04-06T09:59:11.648+0000 7fcbdfabe640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.0 (PID: 125752) UID: 1000 2024-04-06T09:59:14.545 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 -p mirror mirror pool status 2024-04-06T09:59:14.545 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep 'image health:' 2024-04-06T09:59:14.545 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut '-d ' -f 3 2024-04-06T09:59:14.595 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:rbd: failed to query services: (13) Permission denied 2024-04-06T09:59:14.599 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ state=OK 2024-04-06T09:59:14.599 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test OK = ERROR 2024-04-06T09:59:14.600 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test OK = OK 2024-04-06T09:59:14.600 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:59:14.600 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_pool_images cluster1 mirror 50 2024-04-06T09:59:14.600 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:59:14.600 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:59:14.600 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image_count=50 2024-04-06T09:59:14.600 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local s 2024-04-06T09:59:14.600 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local count 2024-04-06T09:59:14.600 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local last_count=0 2024-04-06T09:59:14.600 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:59:14.600 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 1 40 2024-04-06T09:59:14.601 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in `seq 1 40` 2024-04-06T09:59:14.601 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 1 -ne 1 2024-04-06T09:59:14.601 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster1 -p mirror mirror pool status 2024-04-06T09:59:14.601 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep 'images: ' 2024-04-06T09:59:14.602 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut '-d ' -f 2 2024-04-06T09:59:14.651 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:rbd: failed to query services: (13) Permission denied 2024-04-06T09:59:14.655 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ count=50 2024-04-06T09:59:14.655 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 50 = 50 2024-04-06T09:59:14.655 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:59:14.655 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_pool_healthy cluster1 mirror 2024-04-06T09:59:14.655 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:59:14.656 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:59:14.656 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local s 2024-04-06T09:59:14.656 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state 2024-04-06T09:59:14.656 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 1 40 2024-04-06T09:59:14.656 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in `seq 1 40` 2024-04-06T09:59:14.656 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 1 -ne 1 2024-04-06T09:59:14.657 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster1 -p mirror mirror pool status 2024-04-06T09:59:14.657 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep 'image health:' 2024-04-06T09:59:14.657 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut '-d ' -f 3 2024-04-06T09:59:14.709 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:rbd: failed to query services: (13) Permission denied 2024-04-06T09:59:14.713 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ state=OK 2024-04-06T09:59:14.713 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test OK = ERROR 2024-04-06T09:59:14.713 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test OK = OK 2024-04-06T09:59:14.714 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:59:14.714 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ testlog 'TEST: compare many images' 2024-04-06T09:59:14.714 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ tee -a /tmp/tmp.BhW6xKmROU/rbd-mirror.test.log 2024-04-06T09:59:14.714 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ date '+%F %T' 2024-04-06T09:59:14.733 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ echo 2024-04-06 09:59:14 TEST: compare many images 2024-04-06T09:59:14.735 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:2024-04-06 09:59:14 TEST: compare many images 2024-04-06T09:59:14.743 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 1 50 2024-04-06T09:59:14.744 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-06T09:59:14.744 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ image=image_1 2024-04-06T09:59:14.744 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ create_snap cluster2 mirror image_1 snap 2024-04-06T09:59:14.744 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-04-06T09:59:14.744 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:59:14.744 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_1 2024-04-06T09:59:14.744 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local snap_name=snap 2024-04-06T09:59:14.745 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster2 -p mirror snap create image_1@snap --debug-rbd=20 --debug-journaler=20 2024-04-06T09:59:16.645 INFO:tasks.rbd_mirror_thrash:waiting for cluster1.client.mirror.0 2024-04-06T09:59:16.646 INFO:teuthology.orchestra.run:waiting for 600 2024-04-06T09:59:16.646 INFO:tasks.rbd_mirror_thrash:reviving cluster1.client.mirror.0 2024-04-06T09:59:16.646 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Restarting daemon 2024-04-06T09:59:16.646 DEBUG:teuthology.orchestra.run.smithi192:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.0 2024-04-06T09:59:16.649 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Started 2024-04-06T09:59:16.649 INFO:tasks.rbd_mirror_thrash:waiting for 116 secs before thrashing 2024-04-06T09:59:17.775 INFO:tasks.ceph.cluster2.osd.0.smithi192.stderr:2024-04-06T09:59:17.776+0000 7f78af6df640 -1 reset not still connected to 0x56044cc1e1a0 2024-04-06T09:59:17.775 INFO:tasks.ceph.cluster2.osd.0.smithi192.stderr:2024-04-06T09:59:17.776+0000 7f78af6df640 -1 reset not still connected to 0x560456d576c0 2024-04-06T09:59:18.862 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_image_replay_started cluster1 mirror image_1 2024-04-06T09:59:18.862 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:59:18.862 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:59:18.862 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_1 2024-04-06T09:59:18.862 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_image_replay_state cluster1 mirror image_1 started 2024-04-06T09:59:18.863 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:59:18.863 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:59:18.863 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_1 2024-04-06T09:59:18.863 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state=started 2024-04-06T09:59:18.863 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local s 2024-04-06T09:59:18.863 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:59:18.863 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 1 2024-04-06T09:59:19.864 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test_image_replay_state cluster1 mirror image_1 started 2024-04-06T09:59:19.864 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:59:19.864 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:59:19.864 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_1 2024-04-06T09:59:19.864 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local test_state=started 2024-04-06T09:59:19.864 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local status_result 2024-04-06T09:59:19.864 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local current_state=stopped 2024-04-06T09:59:19.865 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -i state 2024-04-06T09:59:19.865 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_1 2024-04-06T09:59:19.865 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_instance=cluster1 2024-04-06T09:59:19.865 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ shift 2024-04-06T09:59:19.866 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:59:19.866 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance=cluster1 2024-04-06T09:59:19.866 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local loop_instance 2024-04-06T09:59:19.866 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:59:19.866 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sleep 0 2024-04-06T09:59:19.867 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:59:19.867 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ seq 0 3 2024-04-06T09:59:19.870 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:59:19.870 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_1 2024-04-06T09:59:19.870 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1:0 2024-04-06T09:59:19.870 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ shift 2024-04-06T09:59:19.870 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:59:19.870 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:59:19.870 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:59:19.870 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=cluster 2024-04-06T09:59:19.870 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:59:19.871 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:59:19.871 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:59:19.871 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:59:19.871 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval cluster=cluster1 2024-04-06T09:59:19.871 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ cluster=cluster1 2024-04-06T09:59:19.871 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:59:19.871 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:59:19.871 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:59:19.871 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local local_cluster=cluster1:0 2024-04-06T09:59:19.871 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local cluster=cluster1 2024-04-06T09:59:19.871 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local instance 2024-04-06T09:59:19.871 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:59:19.873 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local val=cluster1:0 2024-04-06T09:59:19.873 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local cluster_var_name=local_cluster 2024-04-06T09:59:19.873 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local instance_var_name=instance 2024-04-06T09:59:19.874 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ cluster=cluster1 2024-04-06T09:59:19.874 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:59:19.874 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ '[' 0 = cluster1:0 ']' 2024-04-06T09:59:19.874 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ eval local_cluster=cluster1 2024-04-06T09:59:19.874 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++++ local_cluster=cluster1 2024-04-06T09:59:19.874 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ eval instance=0 2024-04-06T09:59:19.874 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++++ instance=0 2024-04-06T09:59:19.874 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:59:19.906 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:59:19.906 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:59:19.906 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:59:19.906 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_1 2024-04-06T09:59:20.080 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:Invalid command: unused arguments: ['mirror/image_1'] 2024-04-06T09:59:20.080 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:rbd mirror status : get status for rbd mirror 2024-04-06T09:59:20.080 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:admin_socket: invalid command 2024-04-06T09:59:20.082 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:59:20.082 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_1 2024-04-06T09:59:20.082 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1:1 2024-04-06T09:59:20.083 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ shift 2024-04-06T09:59:20.083 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:59:20.083 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-04-06T09:59:20.083 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:1 2024-04-06T09:59:20.083 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=cluster 2024-04-06T09:59:20.083 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:59:20.083 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:59:20.083 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=1 2024-04-06T09:59:20.083 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-06T09:59:20.083 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval cluster=cluster1 2024-04-06T09:59:20.083 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ cluster=cluster1 2024-04-06T09:59:20.083 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=1 2024-04-06T09:59:20.083 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=1 2024-04-06T09:59:20.085 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-04-06T09:59:20.085 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local local_cluster=cluster1:1 2024-04-06T09:59:20.085 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local cluster=cluster1 2024-04-06T09:59:20.085 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local instance 2024-04-06T09:59:20.085 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-06T09:59:20.085 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local val=cluster1:1 2024-04-06T09:59:20.085 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local cluster_var_name=local_cluster 2024-04-06T09:59:20.085 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local instance_var_name=instance 2024-04-06T09:59:20.085 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ cluster=cluster1 2024-04-06T09:59:20.085 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=1 2024-04-06T09:59:20.085 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ '[' 1 = cluster1:1 ']' 2024-04-06T09:59:20.086 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ eval local_cluster=cluster1 2024-04-06T09:59:20.086 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++++ local_cluster=cluster1 2024-04-06T09:59:20.086 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ eval instance=1 2024-04-06T09:59:20.086 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++++ instance=1 2024-04-06T09:59:20.086 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-06T09:59:20.110 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T09:59:20.110 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T09:59:20.110 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T09:59:20.110 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_1 2024-04-06T09:59:20.257 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ return 0 2024-04-06T09:59:20.257 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ status_result=' "state": "Replaying"' 2024-04-06T09:59:20.258 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ echo ' "state": "Replaying"' 2024-04-06T09:59:20.258 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep -i Replaying 2024-04-06T09:59:20.260 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: "state": "Replaying" 2024-04-06T09:59:20.262 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ current_state=started 2024-04-06T09:59:20.262 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test started = started 2024-04-06T09:59:20.262 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:59:20.262 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_1 2024-04-06T09:59:20.262 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local local_cluster=cluster1 2024-04-06T09:59:20.262 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-04-06T09:59:20.263 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:59:20.263 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_1 2024-04-06T09:59:20.263 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' journal = journal ']' 2024-04-06T09:59:20.263 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_journal_replay_complete cluster1 cluster2 mirror image_1 2024-04-06T09:59:20.263 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local local_cluster=cluster1 2024-04-06T09:59:20.263 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-04-06T09:59:20.265 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:59:20.265 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_1 2024-04-06T09:59:20.265 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local s master_pos mirror_pos last_mirror_pos 2024-04-06T09:59:20.265 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local master_tag master_entry mirror_tag mirror_entry 2024-04-06T09:59:20.265 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:59:20.265 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:59:20.265 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:59:20.463 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror image_1 2024-04-06T09:59:20.463 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:59:20.464 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:59:20.464 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_1 2024-04-06T09:59:20.464 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:59:20.464 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n image_1 ']' 2024-04-06T09:59:20.464 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/image_1' 2024-04-06T09:59:20.464 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_1 2024-04-06T09:59:20.464 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:59:20.464 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:59:20.464 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:59:20.464 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:59:20.464 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:59:20.464 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:59:20.464 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:59:20.465 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:59:20.465 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:59:20.469 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:59:20.469 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_1 2024-04-06T09:59:20.469 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:59:20.469 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:59:20.469 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:59:20.469 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:59:20.469 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:59:20.469 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:59:20.469 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:59:20.470 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:59:20.470 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:59:20.470 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:59:20.470 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:59:20.470 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:59:20.470 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:59:20.470 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:59:20.470 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:59:20.470 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:59:20.470 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:59:20.470 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:59:20.470 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:59:20.470 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:59:20.470 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:59:20.471 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:59:20.471 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:59:20.471 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:59:20.471 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:59:20.471 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:59:20.471 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:59:20.471 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:59:20.471 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:59:20.471 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:59:20.494 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:59:20.494 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:59:20.494 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:59:20.495 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_1 2024-04-06T09:59:20.601 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:Invalid command: unused arguments: ['mirror/image_1'] 2024-04-06T09:59:20.601 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:rbd mirror flush : flush rbd mirror 2024-04-06T09:59:20.601 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:admin_socket: invalid command 2024-04-06T09:59:20.603 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:59:20.603 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/image_1 2024-04-06T09:59:20.603 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:1 2024-04-06T09:59:20.603 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:59:20.603 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:59:20.603 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-06T09:59:20.603 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:1 2024-04-06T09:59:20.604 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:59:20.604 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:59:20.604 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:59:20.604 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=1 2024-04-06T09:59:20.604 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-06T09:59:20.604 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:59:20.604 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:59:20.604 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=1 2024-04-06T09:59:20.604 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=1 2024-04-06T09:59:20.606 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-06T09:59:20.606 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:1 2024-04-06T09:59:20.606 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:59:20.606 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:59:20.606 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-06T09:59:20.606 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:1 2024-04-06T09:59:20.606 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:59:20.606 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:59:20.606 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:59:20.606 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=1 2024-04-06T09:59:20.606 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-06T09:59:20.606 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:59:20.607 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:59:20.607 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=1 2024-04-06T09:59:20.607 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=1 2024-04-06T09:59:20.607 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-06T09:59:20.629 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T09:59:20.630 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T09:59:20.630 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T09:59:20.630 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/image_1 2024-04-06T09:59:20.776 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:59:20.777 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror image_1 2024-04-06T09:59:20.777 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:59:20.777 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:59:20.777 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_1 2024-04-06T09:59:20.777 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror image_1 '' 2024-04-06T09:59:20.777 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:59:20.777 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:59:20.777 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_1 2024-04-06T09:59:20.777 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:59:20.778 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-image_1.status 2024-04-06T09:59:20.778 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-image_1.status 2024-04-06T09:59:20.778 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:59:20.800 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-image_1.status 2024-04-06T09:59:20.801 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_1 2024-04-06T09:59:20.801 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_1.status 2024-04-06T09:59:20.852 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 2 2024-04-06T09:59:20.852 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 2 2024-04-06T09:59:20.852 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:59:20.852 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=9, tag_tid=2, entry_tid=1], [object_number=8, tag_tid=2, entry_tid=0], [object_number=11, tag_tid=1, entry_tid=11543], [object_number=10, tag_tid=1, entry_tid=11542]]], state=connected] 2024-04-06T09:59:20.852 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=8, tag_tid=1, entry_tid=11544], [object_number=11, tag_tid=1, entry_tid=11543], [object_number=10, tag_tid=1, entry_tid=11542], [object_number=9, tag_tid=1, entry_tid=11541]]], state=connected] 2024-04-06T09:59:20.856 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_1.status 2024-04-06T09:59:20.857 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=9, tag_tid=2, entry_tid=1' 2024-04-06T09:59:20.858 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror image_1 2024-04-06T09:59:20.859 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:59:20.859 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:59:20.859 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_1 2024-04-06T09:59:20.859 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror image_1 '..*' 2024-04-06T09:59:20.859 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:59:20.859 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:59:20.859 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_1 2024-04-06T09:59:20.859 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:59:20.859 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-image_1.status 2024-04-06T09:59:20.859 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-image_1.status 2024-04-06T09:59:20.859 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:59:20.860 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-image_1.status 2024-04-06T09:59:20.860 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_1 2024-04-06T09:59:20.860 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_1.status 2024-04-06T09:59:20.907 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 2 2024-04-06T09:59:20.907 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 2 2024-04-06T09:59:20.907 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:59:20.907 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=9, tag_tid=2, entry_tid=1], [object_number=8, tag_tid=2, entry_tid=0], [object_number=11, tag_tid=1, entry_tid=11543], [object_number=10, tag_tid=1, entry_tid=11542]]], state=connected] 2024-04-06T09:59:20.907 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=8, tag_tid=1, entry_tid=11544], [object_number=11, tag_tid=1, entry_tid=11543], [object_number=10, tag_tid=1, entry_tid=11542], [object_number=9, tag_tid=1, entry_tid=11541]]], state=connected] 2024-04-06T09:59:20.911 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_1.status 2024-04-06T09:59:20.913 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=8, tag_tid=1, entry_tid=11544' 2024-04-06T09:59:20.913 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=9, tag_tid=2, entry_tid=1' -a 'object_number=9, tag_tid=2, entry_tid=1' = 'object_number=8, tag_tid=1, entry_tid=11544' 2024-04-06T09:59:20.913 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=8, tag_tid=1, entry_tid=11544' '!=' '' 2024-04-06T09:59:20.913 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:59:20.913 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=8, tag_tid=1, entry_tid=11544' = '' 2024-04-06T09:59:20.913 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=8, tag_tid=1, entry_tid=11544' 2024-04-06T09:59:20.913 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=9, tag_tid=2, entry_tid=1' 2024-04-06T09:59:20.914 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:59:20.914 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:59:20.914 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=2 2024-04-06T09:59:20.915 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=8, tag_tid=1, entry_tid=11544' 2024-04-06T09:59:20.915 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:59:20.915 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:59:20.916 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=1 2024-04-06T09:59:20.917 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=9, tag_tid=2, entry_tid=1' 2024-04-06T09:59:20.917 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:59:20.917 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:59:20.918 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:59:20.918 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=8, tag_tid=1, entry_tid=11544' 2024-04-06T09:59:20.918 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:59:20.919 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:59:20.920 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=11544 2024-04-06T09:59:20.920 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 2 = 1 -a 1 -le 11544 2024-04-06T09:59:20.920 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:59:20.920 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:59:20.920 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:59:21.121 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror image_1 2024-04-06T09:59:21.122 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:59:21.122 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:59:21.122 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_1 2024-04-06T09:59:21.122 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:59:21.122 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n image_1 ']' 2024-04-06T09:59:21.122 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/image_1' 2024-04-06T09:59:21.122 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_1 2024-04-06T09:59:21.122 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:59:21.122 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:59:21.122 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:59:21.122 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:59:21.122 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:59:21.122 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:59:21.123 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:59:21.123 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:59:21.123 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:59:21.126 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:59:21.126 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_1 2024-04-06T09:59:21.126 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:59:21.126 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:59:21.126 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:59:21.126 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:59:21.126 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:59:21.126 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:59:21.126 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:59:21.126 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:59:21.126 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:59:21.126 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:59:21.126 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:59:21.127 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:59:21.127 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:59:21.127 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:59:21.128 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:59:21.128 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:59:21.128 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:59:21.128 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:59:21.128 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:59:21.128 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:59:21.128 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:59:21.129 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:59:21.129 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:59:21.129 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:59:21.129 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:59:21.129 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:59:21.129 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:59:21.129 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:59:21.129 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:59:21.129 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:59:21.153 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:59:21.153 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:59:21.153 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:59:21.153 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_1 2024-04-06T09:59:21.259 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:Invalid command: unused arguments: ['mirror/image_1'] 2024-04-06T09:59:21.259 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:rbd mirror flush : flush rbd mirror 2024-04-06T09:59:21.259 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:admin_socket: invalid command 2024-04-06T09:59:21.261 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:59:21.262 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/image_1 2024-04-06T09:59:21.262 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:1 2024-04-06T09:59:21.262 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:59:21.262 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:59:21.262 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-06T09:59:21.262 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:1 2024-04-06T09:59:21.262 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:59:21.262 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:59:21.262 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:59:21.262 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=1 2024-04-06T09:59:21.263 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-06T09:59:21.263 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:59:21.263 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:59:21.263 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=1 2024-04-06T09:59:21.263 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=1 2024-04-06T09:59:21.264 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-06T09:59:21.265 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:1 2024-04-06T09:59:21.265 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:59:21.265 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:59:21.265 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-06T09:59:21.265 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:1 2024-04-06T09:59:21.265 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:59:21.265 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:59:21.265 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:59:21.265 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=1 2024-04-06T09:59:21.265 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-06T09:59:21.265 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:59:21.265 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:59:21.265 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=1 2024-04-06T09:59:21.265 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=1 2024-04-06T09:59:21.266 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-06T09:59:21.288 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T09:59:21.288 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T09:59:21.288 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T09:59:21.288 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/image_1 2024-04-06T09:59:21.439 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:59:21.440 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror image_1 2024-04-06T09:59:21.440 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:59:21.440 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:59:21.440 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_1 2024-04-06T09:59:21.440 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror image_1 '' 2024-04-06T09:59:21.440 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:59:21.440 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:59:21.440 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_1 2024-04-06T09:59:21.440 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:59:21.441 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-image_1.status 2024-04-06T09:59:21.441 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-image_1.status 2024-04-06T09:59:21.441 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:59:21.443 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-image_1.status 2024-04-06T09:59:21.444 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_1 2024-04-06T09:59:21.444 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_1.status 2024-04-06T09:59:21.496 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 2 2024-04-06T09:59:21.496 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 2 2024-04-06T09:59:21.496 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:59:21.497 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=9, tag_tid=2, entry_tid=1], [object_number=8, tag_tid=2, entry_tid=0], [object_number=11, tag_tid=1, entry_tid=11543], [object_number=10, tag_tid=1, entry_tid=11542]]], state=connected] 2024-04-06T09:59:21.497 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=9, tag_tid=2, entry_tid=1], [object_number=8, tag_tid=2, entry_tid=0], [object_number=11, tag_tid=1, entry_tid=11543], [object_number=10, tag_tid=1, entry_tid=11542]]], state=connected] 2024-04-06T09:59:21.500 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_1.status 2024-04-06T09:59:21.501 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=9, tag_tid=2, entry_tid=1' 2024-04-06T09:59:21.502 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror image_1 2024-04-06T09:59:21.503 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:59:21.503 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:59:21.503 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_1 2024-04-06T09:59:21.503 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror image_1 '..*' 2024-04-06T09:59:21.503 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:59:21.503 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:59:21.503 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_1 2024-04-06T09:59:21.503 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:59:21.503 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-image_1.status 2024-04-06T09:59:21.503 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-image_1.status 2024-04-06T09:59:21.503 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:59:21.504 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-image_1.status 2024-04-06T09:59:21.504 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_1 2024-04-06T09:59:21.504 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_1.status 2024-04-06T09:59:21.549 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 2 2024-04-06T09:59:21.549 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 2 2024-04-06T09:59:21.549 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:59:21.549 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=9, tag_tid=2, entry_tid=1], [object_number=8, tag_tid=2, entry_tid=0], [object_number=11, tag_tid=1, entry_tid=11543], [object_number=10, tag_tid=1, entry_tid=11542]]], state=connected] 2024-04-06T09:59:21.549 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=9, tag_tid=2, entry_tid=1], [object_number=8, tag_tid=2, entry_tid=0], [object_number=11, tag_tid=1, entry_tid=11543], [object_number=10, tag_tid=1, entry_tid=11542]]], state=connected] 2024-04-06T09:59:21.552 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_1.status 2024-04-06T09:59:21.554 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=9, tag_tid=2, entry_tid=1' 2024-04-06T09:59:21.555 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=9, tag_tid=2, entry_tid=1' -a 'object_number=9, tag_tid=2, entry_tid=1' = 'object_number=9, tag_tid=2, entry_tid=1' 2024-04-06T09:59:21.555 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:59:21.555 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_snap_present cluster1 mirror image_1 snap 2024-04-06T09:59:21.555 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:59:21.555 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:59:21.555 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_1 2024-04-06T09:59:21.555 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local snap_name=snap 2024-04-06T09:59:21.555 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local s 2024-04-06T09:59:21.555 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-04-06T09:59:21.555 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 1 2024-04-06T09:59:22.556 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 info mirror/image_1@snap 2024-04-06T09:59:22.625 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:rbd image 'image_1': 2024-04-06T09:59:22.626 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: size 128 MiB in 32 objects 2024-04-06T09:59:22.626 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: order 22 (4 MiB objects) 2024-04-06T09:59:22.626 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: snapshot_count: 1 2024-04-06T09:59:22.626 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: id: 10ac4624ab4b 2024-04-06T09:59:22.626 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: block_name_prefix: rbd_data.10ac4624ab4b 2024-04-06T09:59:22.626 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: format: 2 2024-04-06T09:59:22.626 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: features: layering, exclusive-lock, journaling 2024-04-06T09:59:22.626 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: op_features: 2024-04-06T09:59:22.626 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: flags: 2024-04-06T09:59:22.626 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: create_timestamp: Sat Apr 6 09:50:41 2024 2024-04-06T09:59:22.627 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: access_timestamp: Sat Apr 6 09:50:41 2024 2024-04-06T09:59:22.627 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: modify_timestamp: Sat Apr 6 09:51:42 2024 2024-04-06T09:59:22.627 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: protected: False 2024-04-06T09:59:22.627 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: journal: 10ac4624ab4b 2024-04-06T09:59:22.627 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: mirroring state: enabled 2024-04-06T09:59:22.627 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: mirroring mode: journal 2024-04-06T09:59:22.627 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: mirroring global id: 25b51a43-8d8d-40cb-8708-26d849ca6acf 2024-04-06T09:59:22.627 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: mirroring primary: false 2024-04-06T09:59:22.631 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:59:22.631 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ compare_image_snaps mirror image_1 snap 2024-04-06T09:59:22.631 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:59:22.631 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_1 2024-04-06T09:59:22.631 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local snap_name=snap 2024-04-06T09:59:22.632 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local ret=0 2024-04-06T09:59:22.632 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local rmt_export=/tmp/tmp.BhW6xKmROU/cluster2-mirror-image_1.export 2024-04-06T09:59:22.632 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loc_export=/tmp/tmp.BhW6xKmROU/cluster1-mirror-image_1.export 2024-04-06T09:59:22.632 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rm -f /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_1.export /tmp/tmp.BhW6xKmROU/cluster1-mirror-image_1.export 2024-04-06T09:59:22.639 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster2 -p mirror export image_1@snap /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_1.export 2024-04-06T09:59:22.821 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-04-06T09:59:22.830 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 -p mirror export image_1@snap /tmp/tmp.BhW6xKmROU/cluster1-mirror-image_1.export 2024-04-06T09:59:23.155 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-04-06T09:59:23.167 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmp /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_1.export /tmp/tmp.BhW6xKmROU/cluster1-mirror-image_1.export 2024-04-06T09:59:23.247 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rm -f /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_1.export /tmp/tmp.BhW6xKmROU/cluster1-mirror-image_1.export 2024-04-06T09:59:23.284 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:59:23.284 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-06T09:59:23.284 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ image=image_2 2024-04-06T09:59:23.284 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ create_snap cluster2 mirror image_2 snap 2024-04-06T09:59:23.284 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-04-06T09:59:23.285 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:59:23.285 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_2 2024-04-06T09:59:23.285 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local snap_name=snap 2024-04-06T09:59:23.285 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster2 -p mirror snap create image_2@snap --debug-rbd=20 --debug-journaler=20 2024-04-06T09:59:25.811 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_image_replay_started cluster1 mirror image_2 2024-04-06T09:59:25.811 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:59:25.811 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:59:25.811 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_2 2024-04-06T09:59:25.811 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_image_replay_state cluster1 mirror image_2 started 2024-04-06T09:59:25.811 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:59:25.811 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:59:25.811 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_2 2024-04-06T09:59:25.811 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state=started 2024-04-06T09:59:25.811 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local s 2024-04-06T09:59:25.811 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:59:25.812 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 1 2024-04-06T09:59:26.812 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test_image_replay_state cluster1 mirror image_2 started 2024-04-06T09:59:26.812 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:59:26.813 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:59:26.813 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_2 2024-04-06T09:59:26.813 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local test_state=started 2024-04-06T09:59:26.813 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local status_result 2024-04-06T09:59:26.813 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local current_state=stopped 2024-04-06T09:59:26.814 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -i state 2024-04-06T09:59:26.814 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_2 2024-04-06T09:59:26.814 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_instance=cluster1 2024-04-06T09:59:26.814 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ shift 2024-04-06T09:59:26.814 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:59:26.814 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance=cluster1 2024-04-06T09:59:26.814 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local loop_instance 2024-04-06T09:59:26.815 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:59:26.815 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sleep 0 2024-04-06T09:59:26.815 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:59:26.816 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ seq 0 3 2024-04-06T09:59:26.819 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:59:26.819 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_2 2024-04-06T09:59:26.819 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1:0 2024-04-06T09:59:26.819 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ shift 2024-04-06T09:59:26.819 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:59:26.820 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:59:26.820 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:59:26.820 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=cluster 2024-04-06T09:59:26.820 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:59:26.820 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:59:26.820 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:59:26.820 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:59:26.820 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval cluster=cluster1 2024-04-06T09:59:26.820 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ cluster=cluster1 2024-04-06T09:59:26.820 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:59:26.820 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:59:26.820 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:59:26.821 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local local_cluster=cluster1:0 2024-04-06T09:59:26.821 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local cluster=cluster1 2024-04-06T09:59:26.821 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local instance 2024-04-06T09:59:26.821 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:59:26.821 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local val=cluster1:0 2024-04-06T09:59:26.821 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local cluster_var_name=local_cluster 2024-04-06T09:59:26.821 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local instance_var_name=instance 2024-04-06T09:59:26.821 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ cluster=cluster1 2024-04-06T09:59:26.821 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:59:26.821 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ '[' 0 = cluster1:0 ']' 2024-04-06T09:59:26.821 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ eval local_cluster=cluster1 2024-04-06T09:59:26.821 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++++ local_cluster=cluster1 2024-04-06T09:59:26.821 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ eval instance=0 2024-04-06T09:59:26.821 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++++ instance=0 2024-04-06T09:59:26.822 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:59:26.844 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:59:26.844 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:59:26.844 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:59:26.844 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_2 2024-04-06T09:59:26.948 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:Invalid command: unused arguments: ['mirror/image_2'] 2024-04-06T09:59:26.948 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:rbd mirror status : get status for rbd mirror 2024-04-06T09:59:26.948 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:admin_socket: invalid command 2024-04-06T09:59:26.950 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:59:26.951 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_2 2024-04-06T09:59:26.951 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1:1 2024-04-06T09:59:26.951 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ shift 2024-04-06T09:59:26.951 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:59:26.951 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-04-06T09:59:26.951 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:1 2024-04-06T09:59:26.951 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=cluster 2024-04-06T09:59:26.951 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:59:26.951 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:59:26.951 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=1 2024-04-06T09:59:26.951 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-06T09:59:26.951 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval cluster=cluster1 2024-04-06T09:59:26.951 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ cluster=cluster1 2024-04-06T09:59:26.952 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=1 2024-04-06T09:59:26.952 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=1 2024-04-06T09:59:26.953 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-04-06T09:59:26.953 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local local_cluster=cluster1:1 2024-04-06T09:59:26.953 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local cluster=cluster1 2024-04-06T09:59:26.953 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local instance 2024-04-06T09:59:26.953 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-06T09:59:26.953 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local val=cluster1:1 2024-04-06T09:59:26.953 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local cluster_var_name=local_cluster 2024-04-06T09:59:26.953 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local instance_var_name=instance 2024-04-06T09:59:26.953 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ cluster=cluster1 2024-04-06T09:59:26.954 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=1 2024-04-06T09:59:26.954 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ '[' 1 = cluster1:1 ']' 2024-04-06T09:59:26.954 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ eval local_cluster=cluster1 2024-04-06T09:59:26.954 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++++ local_cluster=cluster1 2024-04-06T09:59:26.954 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ eval instance=1 2024-04-06T09:59:26.954 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++++ instance=1 2024-04-06T09:59:26.954 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-06T09:59:26.975 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T09:59:26.975 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T09:59:26.976 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T09:59:26.976 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_2 2024-04-06T09:59:27.126 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ return 0 2024-04-06T09:59:27.126 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ status_result=' "state": "Replaying"' 2024-04-06T09:59:27.127 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ echo ' "state": "Replaying"' 2024-04-06T09:59:27.127 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep -i Replaying 2024-04-06T09:59:27.129 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: "state": "Replaying" 2024-04-06T09:59:27.131 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ current_state=started 2024-04-06T09:59:27.131 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test started = started 2024-04-06T09:59:27.131 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:59:27.131 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_2 2024-04-06T09:59:27.131 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local local_cluster=cluster1 2024-04-06T09:59:27.131 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-04-06T09:59:27.131 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:59:27.131 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_2 2024-04-06T09:59:27.132 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' journal = journal ']' 2024-04-06T09:59:27.132 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_journal_replay_complete cluster1 cluster2 mirror image_2 2024-04-06T09:59:27.132 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local local_cluster=cluster1 2024-04-06T09:59:27.132 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-04-06T09:59:27.132 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:59:27.132 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_2 2024-04-06T09:59:27.132 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local s master_pos mirror_pos last_mirror_pos 2024-04-06T09:59:27.132 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local master_tag master_entry mirror_tag mirror_entry 2024-04-06T09:59:27.132 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:59:27.132 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:59:27.132 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:59:27.332 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror image_2 2024-04-06T09:59:27.333 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:59:27.333 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:59:27.333 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_2 2024-04-06T09:59:27.333 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:59:27.333 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n image_2 ']' 2024-04-06T09:59:27.333 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/image_2' 2024-04-06T09:59:27.333 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_2 2024-04-06T09:59:27.333 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:59:27.333 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:59:27.333 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:59:27.333 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:59:27.333 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:59:27.333 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:59:27.333 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:59:27.334 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:59:27.334 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:59:27.337 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:59:27.338 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_2 2024-04-06T09:59:27.338 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:59:27.338 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:59:27.338 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:59:27.338 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:59:27.338 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:59:27.338 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:59:27.338 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:59:27.338 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:59:27.338 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:59:27.338 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:59:27.338 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:59:27.338 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:59:27.339 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:59:27.339 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:59:27.339 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:59:27.339 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:59:27.339 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:59:27.339 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:59:27.339 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:59:27.339 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:59:27.339 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:59:27.339 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:59:27.339 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:59:27.339 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:59:27.339 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:59:27.339 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:59:27.339 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:59:27.340 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:59:27.340 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:59:27.340 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:59:27.365 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:59:27.365 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:59:27.365 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:59:27.365 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_2 2024-04-06T09:59:27.442 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.0 with skip (0.6711457330451687) > weight (0.25) 2024-04-06T09:59:27.442 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.1 with skip (0.4366550293585134) > weight (0.25) 2024-04-06T09:59:27.442 INFO:tasks.rbd_mirror_thrash:kill cluster2.client.mirror.2 2024-04-06T09:59:27.442 INFO:tasks.rbd_mirror.cluster2.client.mirror.2:Sent signal 15 2024-04-06T09:59:27.442 INFO:tasks.rbd_mirror_thrash:waiting for 5 secs before reviving daemons 2024-04-06T09:59:27.444 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi192.stderr:2024-04-06T09:59:27.444+0000 7fc684046640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.2 (PID: 102138) UID: 1000 2024-04-06T09:59:27.491 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:Invalid command: unused arguments: ['mirror/image_2'] 2024-04-06T09:59:27.491 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:rbd mirror flush : flush rbd mirror 2024-04-06T09:59:27.491 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:admin_socket: invalid command 2024-04-06T09:59:27.496 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:59:27.496 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/image_2 2024-04-06T09:59:27.496 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:1 2024-04-06T09:59:27.496 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:59:27.496 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:59:27.496 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-06T09:59:27.496 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:1 2024-04-06T09:59:27.496 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:59:27.496 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:59:27.496 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:59:27.496 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=1 2024-04-06T09:59:27.497 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-06T09:59:27.497 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:59:27.497 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:59:27.497 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=1 2024-04-06T09:59:27.497 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=1 2024-04-06T09:59:27.497 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-06T09:59:27.497 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:1 2024-04-06T09:59:27.497 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:59:27.497 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:59:27.497 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-06T09:59:27.497 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:1 2024-04-06T09:59:27.497 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:59:27.497 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:59:27.497 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:59:27.497 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=1 2024-04-06T09:59:27.497 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-06T09:59:27.498 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:59:27.498 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:59:27.498 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=1 2024-04-06T09:59:27.498 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=1 2024-04-06T09:59:27.498 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-06T09:59:27.519 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T09:59:27.519 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T09:59:27.519 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T09:59:27.519 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/image_2 2024-04-06T09:59:27.669 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:59:27.670 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror image_2 2024-04-06T09:59:27.670 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:59:27.671 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:59:27.671 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_2 2024-04-06T09:59:27.671 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror image_2 '' 2024-04-06T09:59:27.671 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:59:27.671 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:59:27.671 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_2 2024-04-06T09:59:27.671 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:59:27.672 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-image_2.status 2024-04-06T09:59:27.672 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-image_2.status 2024-04-06T09:59:27.672 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:59:27.674 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-image_2.status 2024-04-06T09:59:27.674 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_2.status 2024-04-06T09:59:27.674 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_2 2024-04-06T09:59:27.724 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 1 2024-04-06T09:59:27.724 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 1 2024-04-06T09:59:27.724 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:59:27.724 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=5, tag_tid=2, entry_tid=1], [object_number=4, tag_tid=2, entry_tid=0], [object_number=7, tag_tid=1, entry_tid=7771], [object_number=6, tag_tid=1, entry_tid=7770]]], state=connected] 2024-04-06T09:59:27.724 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=7, tag_tid=1, entry_tid=7771], [object_number=6, tag_tid=1, entry_tid=7770], [object_number=5, tag_tid=1, entry_tid=7769], [object_number=4, tag_tid=1, entry_tid=7768]]], state=connected] 2024-04-06T09:59:27.728 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_2.status 2024-04-06T09:59:27.729 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=5, tag_tid=2, entry_tid=1' 2024-04-06T09:59:27.730 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror image_2 2024-04-06T09:59:27.730 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:59:27.730 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:59:27.730 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_2 2024-04-06T09:59:27.730 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror image_2 '..*' 2024-04-06T09:59:27.731 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:59:27.731 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:59:27.731 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_2 2024-04-06T09:59:27.731 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:59:27.731 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-image_2.status 2024-04-06T09:59:27.731 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-image_2.status 2024-04-06T09:59:27.731 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:59:27.732 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-image_2.status 2024-04-06T09:59:27.732 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_2 2024-04-06T09:59:27.732 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_2.status 2024-04-06T09:59:27.780 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 1 2024-04-06T09:59:27.780 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 1 2024-04-06T09:59:27.780 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:59:27.780 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=5, tag_tid=2, entry_tid=1], [object_number=4, tag_tid=2, entry_tid=0], [object_number=7, tag_tid=1, entry_tid=7771], [object_number=6, tag_tid=1, entry_tid=7770]]], state=connected] 2024-04-06T09:59:27.780 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=7, tag_tid=1, entry_tid=7771], [object_number=6, tag_tid=1, entry_tid=7770], [object_number=5, tag_tid=1, entry_tid=7769], [object_number=4, tag_tid=1, entry_tid=7768]]], state=connected] 2024-04-06T09:59:27.784 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_2.status 2024-04-06T09:59:27.785 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=7, tag_tid=1, entry_tid=7771' 2024-04-06T09:59:27.786 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=5, tag_tid=2, entry_tid=1' -a 'object_number=5, tag_tid=2, entry_tid=1' = 'object_number=7, tag_tid=1, entry_tid=7771' 2024-04-06T09:59:27.786 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=7, tag_tid=1, entry_tid=7771' '!=' '' 2024-04-06T09:59:27.786 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:59:27.786 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=7, tag_tid=1, entry_tid=7771' = '' 2024-04-06T09:59:27.786 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=7, tag_tid=1, entry_tid=7771' 2024-04-06T09:59:27.787 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=5, tag_tid=2, entry_tid=1' 2024-04-06T09:59:27.787 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:59:27.787 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:59:27.787 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=2 2024-04-06T09:59:27.788 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=7, tag_tid=1, entry_tid=7771' 2024-04-06T09:59:27.788 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:59:27.788 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:59:27.789 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=1 2024-04-06T09:59:27.789 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=5, tag_tid=2, entry_tid=1' 2024-04-06T09:59:27.789 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:59:27.790 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:59:27.790 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:59:27.791 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=7, tag_tid=1, entry_tid=7771' 2024-04-06T09:59:27.791 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:59:27.791 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:59:27.792 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=7771 2024-04-06T09:59:27.793 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 2 = 1 -a 1 -le 7771 2024-04-06T09:59:27.793 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:59:27.793 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:59:27.793 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:59:27.994 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror image_2 2024-04-06T09:59:27.995 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:59:27.995 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:59:27.995 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_2 2024-04-06T09:59:27.995 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:59:27.995 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n image_2 ']' 2024-04-06T09:59:27.995 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/image_2' 2024-04-06T09:59:27.995 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_2 2024-04-06T09:59:27.995 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:59:27.995 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:59:27.995 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:59:27.995 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:59:27.995 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:59:27.995 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:59:27.996 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:59:27.996 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:59:27.996 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:59:27.998 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:59:27.998 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_2 2024-04-06T09:59:27.998 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:59:27.998 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:59:27.998 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:59:27.999 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:59:27.999 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:59:27.999 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:59:27.999 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:59:27.999 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:59:27.999 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:59:27.999 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:59:27.999 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:59:27.999 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:59:27.999 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:59:27.999 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:59:28.000 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:59:28.000 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:59:28.001 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:59:28.001 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:59:28.001 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:59:28.001 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:59:28.001 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:59:28.001 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:59:28.001 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:59:28.001 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:59:28.001 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:59:28.001 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:59:28.001 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:59:28.001 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:59:28.001 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:59:28.001 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:59:28.026 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:59:28.026 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:59:28.026 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:59:28.026 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_2 2024-04-06T09:59:28.132 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:Invalid command: unused arguments: ['mirror/image_2'] 2024-04-06T09:59:28.132 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:rbd mirror flush : flush rbd mirror 2024-04-06T09:59:28.132 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:admin_socket: invalid command 2024-04-06T09:59:28.135 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:59:28.135 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/image_2 2024-04-06T09:59:28.135 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:1 2024-04-06T09:59:28.135 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:59:28.135 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:59:28.135 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-06T09:59:28.135 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:1 2024-04-06T09:59:28.135 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:59:28.135 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:59:28.135 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:59:28.135 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=1 2024-04-06T09:59:28.135 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-06T09:59:28.136 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:59:28.136 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:59:28.136 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=1 2024-04-06T09:59:28.136 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=1 2024-04-06T09:59:28.137 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-06T09:59:28.137 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:1 2024-04-06T09:59:28.137 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:59:28.137 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:59:28.137 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-06T09:59:28.137 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:1 2024-04-06T09:59:28.138 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:59:28.138 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:59:28.138 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:59:28.138 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=1 2024-04-06T09:59:28.138 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-06T09:59:28.138 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:59:28.138 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:59:28.138 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=1 2024-04-06T09:59:28.138 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=1 2024-04-06T09:59:28.138 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-06T09:59:28.162 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T09:59:28.162 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T09:59:28.162 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T09:59:28.162 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/image_2 2024-04-06T09:59:28.316 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:59:28.317 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror image_2 2024-04-06T09:59:28.317 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:59:28.317 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:59:28.317 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_2 2024-04-06T09:59:28.317 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror image_2 '' 2024-04-06T09:59:28.318 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:59:28.318 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:59:28.318 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_2 2024-04-06T09:59:28.318 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:59:28.318 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-image_2.status 2024-04-06T09:59:28.318 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-image_2.status 2024-04-06T09:59:28.318 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:59:28.321 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-image_2.status 2024-04-06T09:59:28.321 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_2 2024-04-06T09:59:28.321 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_2.status 2024-04-06T09:59:28.367 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 1 2024-04-06T09:59:28.368 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 1 2024-04-06T09:59:28.368 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:59:28.368 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=5, tag_tid=2, entry_tid=1], [object_number=4, tag_tid=2, entry_tid=0], [object_number=7, tag_tid=1, entry_tid=7771], [object_number=6, tag_tid=1, entry_tid=7770]]], state=connected] 2024-04-06T09:59:28.368 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=7, tag_tid=1, entry_tid=7771], [object_number=6, tag_tid=1, entry_tid=7770], [object_number=5, tag_tid=1, entry_tid=7769], [object_number=4, tag_tid=1, entry_tid=7768]]], state=connected] 2024-04-06T09:59:28.371 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_2.status 2024-04-06T09:59:28.372 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=5, tag_tid=2, entry_tid=1' 2024-04-06T09:59:28.374 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror image_2 2024-04-06T09:59:28.374 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:59:28.374 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:59:28.374 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_2 2024-04-06T09:59:28.374 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror image_2 '..*' 2024-04-06T09:59:28.374 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:59:28.374 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:59:28.374 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_2 2024-04-06T09:59:28.374 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:59:28.374 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-image_2.status 2024-04-06T09:59:28.374 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-image_2.status 2024-04-06T09:59:28.374 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:59:28.375 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-image_2.status 2024-04-06T09:59:28.375 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_2 2024-04-06T09:59:28.375 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_2.status 2024-04-06T09:59:28.420 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 1 2024-04-06T09:59:28.420 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 1 2024-04-06T09:59:28.420 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:59:28.420 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=5, tag_tid=2, entry_tid=1], [object_number=4, tag_tid=2, entry_tid=0], [object_number=7, tag_tid=1, entry_tid=7771], [object_number=6, tag_tid=1, entry_tid=7770]]], state=connected] 2024-04-06T09:59:28.420 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=7, tag_tid=1, entry_tid=7771], [object_number=6, tag_tid=1, entry_tid=7770], [object_number=5, tag_tid=1, entry_tid=7769], [object_number=4, tag_tid=1, entry_tid=7768]]], state=connected] 2024-04-06T09:59:28.424 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_2.status 2024-04-06T09:59:28.425 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=7, tag_tid=1, entry_tid=7771' 2024-04-06T09:59:28.425 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=5, tag_tid=2, entry_tid=1' -a 'object_number=5, tag_tid=2, entry_tid=1' = 'object_number=7, tag_tid=1, entry_tid=7771' 2024-04-06T09:59:28.425 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=7, tag_tid=1, entry_tid=7771' '!=' 'object_number=7, tag_tid=1, entry_tid=7771' 2024-04-06T09:59:28.426 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:59:28.426 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.4 2024-04-06T09:59:28.827 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror image_2 2024-04-06T09:59:28.828 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:59:28.828 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:59:28.828 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_2 2024-04-06T09:59:28.828 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:59:28.828 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n image_2 ']' 2024-04-06T09:59:28.828 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/image_2' 2024-04-06T09:59:28.828 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_2 2024-04-06T09:59:28.828 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:59:28.828 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:59:28.828 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:59:28.829 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:59:28.829 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:59:28.829 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:59:28.829 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:59:28.829 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:59:28.829 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:59:28.833 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:59:28.833 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_2 2024-04-06T09:59:28.833 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:59:28.833 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:59:28.833 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:59:28.833 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:59:28.833 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:59:28.833 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:59:28.833 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:59:28.834 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:59:28.834 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:59:28.834 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:59:28.834 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:59:28.834 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:59:28.834 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:59:28.834 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:59:28.834 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:59:28.834 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:59:28.834 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:59:28.834 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:59:28.834 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:59:28.834 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:59:28.834 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:59:28.835 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:59:28.835 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:59:28.835 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:59:28.835 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:59:28.835 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:59:28.835 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:59:28.835 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:59:28.835 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:59:28.835 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:59:28.858 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:59:28.858 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:59:28.858 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:59:28.859 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_2 2024-04-06T09:59:28.969 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:Invalid command: unused arguments: ['mirror/image_2'] 2024-04-06T09:59:28.970 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:rbd mirror flush : flush rbd mirror 2024-04-06T09:59:28.970 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:admin_socket: invalid command 2024-04-06T09:59:28.972 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:59:28.972 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/image_2 2024-04-06T09:59:28.972 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:1 2024-04-06T09:59:28.972 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:59:28.972 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:59:28.972 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-06T09:59:28.972 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:1 2024-04-06T09:59:28.972 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:59:28.972 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:59:28.972 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:59:28.973 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=1 2024-04-06T09:59:28.973 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-06T09:59:28.973 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:59:28.973 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:59:28.973 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=1 2024-04-06T09:59:28.973 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=1 2024-04-06T09:59:28.974 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-06T09:59:28.974 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:1 2024-04-06T09:59:28.975 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:59:28.975 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:59:28.975 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-06T09:59:28.975 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:1 2024-04-06T09:59:28.975 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:59:28.975 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:59:28.975 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:59:28.975 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=1 2024-04-06T09:59:28.975 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-06T09:59:28.975 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:59:28.975 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:59:28.976 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=1 2024-04-06T09:59:28.976 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=1 2024-04-06T09:59:28.976 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-06T09:59:28.998 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T09:59:28.998 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T09:59:28.999 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T09:59:28.999 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/image_2 2024-04-06T09:59:29.151 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:59:29.152 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror image_2 2024-04-06T09:59:29.152 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:59:29.152 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:59:29.152 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_2 2024-04-06T09:59:29.152 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror image_2 '' 2024-04-06T09:59:29.152 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:59:29.152 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:59:29.152 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_2 2024-04-06T09:59:29.152 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:59:29.153 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-image_2.status 2024-04-06T09:59:29.153 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-image_2.status 2024-04-06T09:59:29.153 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:59:29.155 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-image_2.status 2024-04-06T09:59:29.156 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_2 2024-04-06T09:59:29.156 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_2.status 2024-04-06T09:59:29.209 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 1 2024-04-06T09:59:29.210 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 1 2024-04-06T09:59:29.210 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:59:29.210 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=5, tag_tid=2, entry_tid=1], [object_number=4, tag_tid=2, entry_tid=0], [object_number=7, tag_tid=1, entry_tid=7771], [object_number=6, tag_tid=1, entry_tid=7770]]], state=connected] 2024-04-06T09:59:29.210 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=7, tag_tid=1, entry_tid=7771], [object_number=6, tag_tid=1, entry_tid=7770], [object_number=5, tag_tid=1, entry_tid=7769], [object_number=4, tag_tid=1, entry_tid=7768]]], state=connected] 2024-04-06T09:59:29.213 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_2.status 2024-04-06T09:59:29.215 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=5, tag_tid=2, entry_tid=1' 2024-04-06T09:59:29.216 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror image_2 2024-04-06T09:59:29.216 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:59:29.216 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:59:29.216 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_2 2024-04-06T09:59:29.216 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror image_2 '..*' 2024-04-06T09:59:29.216 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:59:29.216 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:59:29.216 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_2 2024-04-06T09:59:29.216 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:59:29.217 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-image_2.status 2024-04-06T09:59:29.217 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-image_2.status 2024-04-06T09:59:29.217 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:59:29.217 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-image_2.status 2024-04-06T09:59:29.218 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_2 2024-04-06T09:59:29.218 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_2.status 2024-04-06T09:59:29.263 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 1 2024-04-06T09:59:29.263 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 1 2024-04-06T09:59:29.263 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:59:29.264 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=5, tag_tid=2, entry_tid=1], [object_number=4, tag_tid=2, entry_tid=0], [object_number=7, tag_tid=1, entry_tid=7771], [object_number=6, tag_tid=1, entry_tid=7770]]], state=connected] 2024-04-06T09:59:29.264 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=7, tag_tid=1, entry_tid=7771], [object_number=6, tag_tid=1, entry_tid=7770], [object_number=5, tag_tid=1, entry_tid=7769], [object_number=4, tag_tid=1, entry_tid=7768]]], state=connected] 2024-04-06T09:59:29.267 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_2.status 2024-04-06T09:59:29.269 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=7, tag_tid=1, entry_tid=7771' 2024-04-06T09:59:29.269 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=5, tag_tid=2, entry_tid=1' -a 'object_number=5, tag_tid=2, entry_tid=1' = 'object_number=7, tag_tid=1, entry_tid=7771' 2024-04-06T09:59:29.269 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=7, tag_tid=1, entry_tid=7771' '!=' 'object_number=7, tag_tid=1, entry_tid=7771' 2024-04-06T09:59:29.269 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:59:29.269 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.8 2024-04-06T09:59:30.071 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror image_2 2024-04-06T09:59:30.071 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:59:30.071 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:59:30.071 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_2 2024-04-06T09:59:30.071 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:59:30.071 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n image_2 ']' 2024-04-06T09:59:30.071 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/image_2' 2024-04-06T09:59:30.071 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_2 2024-04-06T09:59:30.072 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:59:30.072 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:59:30.072 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:59:30.072 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:59:30.072 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:59:30.072 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:59:30.072 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:59:30.072 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:59:30.073 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:59:30.074 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:59:30.075 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_2 2024-04-06T09:59:30.075 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:59:30.075 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:59:30.075 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:59:30.075 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:59:30.075 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:59:30.075 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:59:30.075 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:59:30.075 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:59:30.075 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:59:30.075 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:59:30.075 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:59:30.076 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:59:30.076 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:59:30.076 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:59:30.076 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:59:30.076 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:59:30.076 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:59:30.076 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:59:30.076 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:59:30.076 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:59:30.076 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:59:30.076 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:59:30.076 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:59:30.076 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:59:30.076 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:59:30.077 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:59:30.077 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:59:30.077 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:59:30.077 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:59:30.077 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:59:30.104 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:59:30.104 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:59:30.104 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:59:30.104 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_2 2024-04-06T09:59:30.214 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:Invalid command: unused arguments: ['mirror/image_2'] 2024-04-06T09:59:30.215 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:rbd mirror flush : flush rbd mirror 2024-04-06T09:59:30.215 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:admin_socket: invalid command 2024-04-06T09:59:30.217 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:59:30.217 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/image_2 2024-04-06T09:59:30.217 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:1 2024-04-06T09:59:30.217 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:59:30.217 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:59:30.217 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-06T09:59:30.217 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:1 2024-04-06T09:59:30.217 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:59:30.217 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:59:30.218 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:59:30.218 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=1 2024-04-06T09:59:30.218 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-06T09:59:30.218 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:59:30.218 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:59:30.218 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=1 2024-04-06T09:59:30.218 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=1 2024-04-06T09:59:30.219 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-06T09:59:30.219 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:1 2024-04-06T09:59:30.219 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:59:30.219 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:59:30.219 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-06T09:59:30.219 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:1 2024-04-06T09:59:30.220 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:59:30.220 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:59:30.220 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:59:30.220 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=1 2024-04-06T09:59:30.220 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-06T09:59:30.220 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:59:30.220 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:59:30.220 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=1 2024-04-06T09:59:30.220 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=1 2024-04-06T09:59:30.220 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-06T09:59:30.239 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T09:59:30.239 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T09:59:30.239 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T09:59:30.239 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/image_2 2024-04-06T09:59:30.391 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:59:30.392 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror image_2 2024-04-06T09:59:30.393 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:59:30.393 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:59:30.393 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_2 2024-04-06T09:59:30.393 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror image_2 '' 2024-04-06T09:59:30.393 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:59:30.393 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:59:30.393 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_2 2024-04-06T09:59:30.393 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:59:30.394 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-image_2.status 2024-04-06T09:59:30.394 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-image_2.status 2024-04-06T09:59:30.394 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:59:30.395 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-image_2.status 2024-04-06T09:59:30.395 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_2 2024-04-06T09:59:30.396 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_2.status 2024-04-06T09:59:30.446 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 1 2024-04-06T09:59:30.446 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 1 2024-04-06T09:59:30.446 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:59:30.446 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=5, tag_tid=2, entry_tid=1], [object_number=4, tag_tid=2, entry_tid=0], [object_number=7, tag_tid=1, entry_tid=7771], [object_number=6, tag_tid=1, entry_tid=7770]]], state=connected] 2024-04-06T09:59:30.446 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=5, tag_tid=2, entry_tid=1], [object_number=4, tag_tid=2, entry_tid=0], [object_number=7, tag_tid=1, entry_tid=7771], [object_number=6, tag_tid=1, entry_tid=7770]]], state=connected] 2024-04-06T09:59:30.450 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_2.status 2024-04-06T09:59:30.451 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=5, tag_tid=2, entry_tid=1' 2024-04-06T09:59:30.452 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror image_2 2024-04-06T09:59:30.452 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:59:30.452 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:59:30.453 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_2 2024-04-06T09:59:30.453 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror image_2 '..*' 2024-04-06T09:59:30.453 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:59:30.453 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:59:30.453 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_2 2024-04-06T09:59:30.453 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:59:30.453 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-image_2.status 2024-04-06T09:59:30.453 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-image_2.status 2024-04-06T09:59:30.453 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:59:30.453 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-image_2.status 2024-04-06T09:59:30.454 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_2 2024-04-06T09:59:30.454 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_2.status 2024-04-06T09:59:30.500 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 1 2024-04-06T09:59:30.500 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 1 2024-04-06T09:59:30.500 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:59:30.500 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=5, tag_tid=2, entry_tid=1], [object_number=4, tag_tid=2, entry_tid=0], [object_number=7, tag_tid=1, entry_tid=7771], [object_number=6, tag_tid=1, entry_tid=7770]]], state=connected] 2024-04-06T09:59:30.500 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=5, tag_tid=2, entry_tid=1], [object_number=4, tag_tid=2, entry_tid=0], [object_number=7, tag_tid=1, entry_tid=7771], [object_number=6, tag_tid=1, entry_tid=7770]]], state=connected] 2024-04-06T09:59:30.503 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_2.status 2024-04-06T09:59:30.505 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=5, tag_tid=2, entry_tid=1' 2024-04-06T09:59:30.505 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=5, tag_tid=2, entry_tid=1' -a 'object_number=5, tag_tid=2, entry_tid=1' = 'object_number=5, tag_tid=2, entry_tid=1' 2024-04-06T09:59:30.505 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:59:30.505 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_snap_present cluster1 mirror image_2 snap 2024-04-06T09:59:30.505 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:59:30.505 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:59:30.505 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_2 2024-04-06T09:59:30.506 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local snap_name=snap 2024-04-06T09:59:30.506 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local s 2024-04-06T09:59:30.506 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-04-06T09:59:30.506 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 1 2024-04-06T09:59:31.506 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 info mirror/image_2@snap 2024-04-06T09:59:31.578 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:rbd image 'image_2': 2024-04-06T09:59:31.579 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: size 128 MiB in 32 objects 2024-04-06T09:59:31.579 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: order 22 (4 MiB objects) 2024-04-06T09:59:31.579 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: snapshot_count: 1 2024-04-06T09:59:31.579 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: id: 10acec481509 2024-04-06T09:59:31.579 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: block_name_prefix: rbd_data.10acec481509 2024-04-06T09:59:31.579 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: format: 2 2024-04-06T09:59:31.579 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: features: layering, exclusive-lock, journaling 2024-04-06T09:59:31.579 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: op_features: 2024-04-06T09:59:31.579 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: flags: 2024-04-06T09:59:31.579 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: create_timestamp: Sat Apr 6 09:50:52 2024 2024-04-06T09:59:31.579 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: access_timestamp: Sat Apr 6 09:50:52 2024 2024-04-06T09:59:31.579 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: modify_timestamp: Sat Apr 6 09:50:52 2024 2024-04-06T09:59:31.579 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: protected: False 2024-04-06T09:59:31.579 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: journal: 10acec481509 2024-04-06T09:59:31.579 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: mirroring state: enabled 2024-04-06T09:59:31.579 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: mirroring mode: journal 2024-04-06T09:59:31.580 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: mirroring global id: ad6f887b-7129-4257-84a8-71b1bb7cfdac 2024-04-06T09:59:31.580 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: mirroring primary: false 2024-04-06T09:59:31.584 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:59:31.585 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ compare_image_snaps mirror image_2 snap 2024-04-06T09:59:31.585 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:59:31.585 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_2 2024-04-06T09:59:31.585 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local snap_name=snap 2024-04-06T09:59:31.585 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local ret=0 2024-04-06T09:59:31.585 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local rmt_export=/tmp/tmp.BhW6xKmROU/cluster2-mirror-image_2.export 2024-04-06T09:59:31.585 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loc_export=/tmp/tmp.BhW6xKmROU/cluster1-mirror-image_2.export 2024-04-06T09:59:31.585 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rm -f /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_2.export /tmp/tmp.BhW6xKmROU/cluster1-mirror-image_2.export 2024-04-06T09:59:31.585 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster2 -p mirror export image_2@snap /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_2.export 2024-04-06T09:59:31.761 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-04-06T09:59:31.771 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 -p mirror export image_2@snap /tmp/tmp.BhW6xKmROU/cluster1-mirror-image_2.export 2024-04-06T09:59:32.063 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-04-06T09:59:32.073 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmp /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_2.export /tmp/tmp.BhW6xKmROU/cluster1-mirror-image_2.export 2024-04-06T09:59:32.153 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rm -f /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_2.export /tmp/tmp.BhW6xKmROU/cluster1-mirror-image_2.export 2024-04-06T09:59:32.196 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:59:32.197 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-06T09:59:32.197 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ image=image_3 2024-04-06T09:59:32.197 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ create_snap cluster2 mirror image_3 snap 2024-04-06T09:59:32.197 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-04-06T09:59:32.197 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:59:32.197 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_3 2024-04-06T09:59:32.197 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local snap_name=snap 2024-04-06T09:59:32.197 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster2 -p mirror snap create image_3@snap --debug-rbd=20 --debug-journaler=20 2024-04-06T09:59:32.443 INFO:tasks.rbd_mirror_thrash:waiting for cluster2.client.mirror.2 2024-04-06T09:59:32.443 INFO:teuthology.orchestra.run:waiting for 600 2024-04-06T09:59:32.443 INFO:tasks.rbd_mirror_thrash:reviving cluster2.client.mirror.2 2024-04-06T09:59:32.443 INFO:tasks.rbd_mirror.cluster2.client.mirror.2:Restarting daemon 2024-04-06T09:59:32.444 DEBUG:teuthology.orchestra.run.smithi192:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.2 2024-04-06T09:59:32.447 INFO:tasks.rbd_mirror.cluster2.client.mirror.2:Started 2024-04-06T09:59:32.447 INFO:tasks.rbd_mirror_thrash:waiting for 71 secs before thrashing 2024-04-06T09:59:34.862 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_image_replay_started cluster1 mirror image_3 2024-04-06T09:59:34.863 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:59:34.863 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:59:34.863 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_3 2024-04-06T09:59:34.863 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_image_replay_state cluster1 mirror image_3 started 2024-04-06T09:59:34.863 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:59:34.863 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:59:34.863 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_3 2024-04-06T09:59:34.863 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state=started 2024-04-06T09:59:34.863 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local s 2024-04-06T09:59:34.863 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:59:34.863 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 1 2024-04-06T09:59:35.865 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test_image_replay_state cluster1 mirror image_3 started 2024-04-06T09:59:35.865 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:59:35.865 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:59:35.865 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_3 2024-04-06T09:59:35.865 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local test_state=started 2024-04-06T09:59:35.865 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local status_result 2024-04-06T09:59:35.865 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local current_state=stopped 2024-04-06T09:59:35.866 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_3 2024-04-06T09:59:35.867 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -i state 2024-04-06T09:59:35.867 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_instance=cluster1 2024-04-06T09:59:35.867 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ shift 2024-04-06T09:59:35.867 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:59:35.867 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance=cluster1 2024-04-06T09:59:35.867 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local loop_instance 2024-04-06T09:59:35.867 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:59:35.867 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sleep 0 2024-04-06T09:59:35.867 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:59:35.868 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ seq 0 3 2024-04-06T09:59:35.869 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:59:35.869 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_3 2024-04-06T09:59:35.869 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1:0 2024-04-06T09:59:35.869 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ shift 2024-04-06T09:59:35.869 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:59:35.869 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:59:35.869 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:59:35.869 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=cluster 2024-04-06T09:59:35.870 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:59:35.870 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:59:35.870 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:59:35.870 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:59:35.870 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval cluster=cluster1 2024-04-06T09:59:35.870 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ cluster=cluster1 2024-04-06T09:59:35.870 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:59:35.870 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:59:35.871 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:59:35.871 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local local_cluster=cluster1:0 2024-04-06T09:59:35.872 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local cluster=cluster1 2024-04-06T09:59:35.872 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local instance 2024-04-06T09:59:35.872 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:59:35.872 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local val=cluster1:0 2024-04-06T09:59:35.872 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local cluster_var_name=local_cluster 2024-04-06T09:59:35.872 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local instance_var_name=instance 2024-04-06T09:59:35.872 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ cluster=cluster1 2024-04-06T09:59:35.872 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:59:35.872 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ '[' 0 = cluster1:0 ']' 2024-04-06T09:59:35.872 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ eval local_cluster=cluster1 2024-04-06T09:59:35.872 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++++ local_cluster=cluster1 2024-04-06T09:59:35.873 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ eval instance=0 2024-04-06T09:59:35.873 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++++ instance=0 2024-04-06T09:59:35.873 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:59:35.892 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:59:35.893 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:59:35.893 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:59:35.893 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_3 2024-04-06T09:59:36.000 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:Invalid command: unused arguments: ['mirror/image_3'] 2024-04-06T09:59:36.000 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:rbd mirror status : get status for rbd mirror 2024-04-06T09:59:36.000 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:admin_socket: invalid command 2024-04-06T09:59:36.002 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:59:36.002 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_3 2024-04-06T09:59:36.002 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1:1 2024-04-06T09:59:36.002 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ shift 2024-04-06T09:59:36.002 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:59:36.002 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-04-06T09:59:36.002 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:1 2024-04-06T09:59:36.002 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=cluster 2024-04-06T09:59:36.002 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:59:36.002 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:59:36.002 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=1 2024-04-06T09:59:36.003 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-06T09:59:36.003 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval cluster=cluster1 2024-04-06T09:59:36.003 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ cluster=cluster1 2024-04-06T09:59:36.003 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=1 2024-04-06T09:59:36.003 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=1 2024-04-06T09:59:36.004 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-04-06T09:59:36.004 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local local_cluster=cluster1:1 2024-04-06T09:59:36.004 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local cluster=cluster1 2024-04-06T09:59:36.004 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local instance 2024-04-06T09:59:36.004 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-06T09:59:36.004 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local val=cluster1:1 2024-04-06T09:59:36.004 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local cluster_var_name=local_cluster 2024-04-06T09:59:36.005 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local instance_var_name=instance 2024-04-06T09:59:36.005 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ cluster=cluster1 2024-04-06T09:59:36.005 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=1 2024-04-06T09:59:36.005 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ '[' 1 = cluster1:1 ']' 2024-04-06T09:59:36.005 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ eval local_cluster=cluster1 2024-04-06T09:59:36.005 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++++ local_cluster=cluster1 2024-04-06T09:59:36.005 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ eval instance=1 2024-04-06T09:59:36.005 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++++ instance=1 2024-04-06T09:59:36.005 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-06T09:59:36.028 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T09:59:36.028 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T09:59:36.028 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T09:59:36.028 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_3 2024-04-06T09:59:36.192 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ return 0 2024-04-06T09:59:36.193 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ status_result=' "state": "Replaying"' 2024-04-06T09:59:36.194 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ echo ' "state": "Replaying"' 2024-04-06T09:59:36.194 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep -i Replaying 2024-04-06T09:59:36.196 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: "state": "Replaying" 2024-04-06T09:59:36.198 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ current_state=started 2024-04-06T09:59:36.198 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test started = started 2024-04-06T09:59:36.198 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:59:36.198 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_3 2024-04-06T09:59:36.198 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local local_cluster=cluster1 2024-04-06T09:59:36.198 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-04-06T09:59:36.198 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:59:36.198 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_3 2024-04-06T09:59:36.198 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' journal = journal ']' 2024-04-06T09:59:36.199 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_journal_replay_complete cluster1 cluster2 mirror image_3 2024-04-06T09:59:36.199 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local local_cluster=cluster1 2024-04-06T09:59:36.199 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-04-06T09:59:36.199 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:59:36.199 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_3 2024-04-06T09:59:36.199 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local s master_pos mirror_pos last_mirror_pos 2024-04-06T09:59:36.199 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local master_tag master_entry mirror_tag mirror_entry 2024-04-06T09:59:36.199 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:59:36.199 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:59:36.199 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:59:36.399 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror image_3 2024-04-06T09:59:36.399 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:59:36.399 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:59:36.399 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_3 2024-04-06T09:59:36.400 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:59:36.400 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n image_3 ']' 2024-04-06T09:59:36.400 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/image_3' 2024-04-06T09:59:36.400 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_3 2024-04-06T09:59:36.400 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:59:36.400 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:59:36.400 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:59:36.400 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:59:36.400 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:59:36.400 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:59:36.400 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:59:36.401 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:59:36.401 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:59:36.404 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:59:36.405 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_3 2024-04-06T09:59:36.405 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:59:36.405 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:59:36.405 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:59:36.405 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:59:36.405 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:59:36.405 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:59:36.405 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:59:36.405 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:59:36.405 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:59:36.405 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:59:36.405 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:59:36.406 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:59:36.406 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:59:36.406 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:59:36.406 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:59:36.406 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:59:36.406 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:59:36.406 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:59:36.406 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:59:36.406 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:59:36.406 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:59:36.406 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:59:36.406 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:59:36.406 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:59:36.406 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:59:36.406 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:59:36.407 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:59:36.407 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:59:36.407 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:59:36.407 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:59:36.430 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:59:36.431 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:59:36.431 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:59:36.431 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_3 2024-04-06T09:59:36.536 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:Invalid command: unused arguments: ['mirror/image_3'] 2024-04-06T09:59:36.536 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:rbd mirror flush : flush rbd mirror 2024-04-06T09:59:36.536 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:admin_socket: invalid command 2024-04-06T09:59:36.538 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:59:36.538 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/image_3 2024-04-06T09:59:36.538 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:1 2024-04-06T09:59:36.538 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:59:36.539 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:59:36.539 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-06T09:59:36.539 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:1 2024-04-06T09:59:36.539 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:59:36.539 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:59:36.539 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:59:36.539 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=1 2024-04-06T09:59:36.539 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-06T09:59:36.539 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:59:36.539 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:59:36.539 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=1 2024-04-06T09:59:36.539 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=1 2024-04-06T09:59:36.541 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-06T09:59:36.541 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:1 2024-04-06T09:59:36.541 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:59:36.541 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:59:36.541 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-06T09:59:36.541 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:1 2024-04-06T09:59:36.541 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:59:36.541 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:59:36.541 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:59:36.542 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=1 2024-04-06T09:59:36.542 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-06T09:59:36.542 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:59:36.542 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:59:36.542 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=1 2024-04-06T09:59:36.542 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=1 2024-04-06T09:59:36.542 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-06T09:59:36.564 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T09:59:36.564 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T09:59:36.564 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T09:59:36.564 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/image_3 2024-04-06T09:59:36.705 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:59:36.706 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror image_3 2024-04-06T09:59:36.706 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:59:36.706 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:59:36.706 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_3 2024-04-06T09:59:36.706 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror image_3 '' 2024-04-06T09:59:36.706 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:59:36.706 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:59:36.707 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_3 2024-04-06T09:59:36.707 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:59:36.707 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-image_3.status 2024-04-06T09:59:36.707 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-image_3.status 2024-04-06T09:59:36.707 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:59:36.709 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-image_3.status 2024-04-06T09:59:36.710 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_3 2024-04-06T09:59:36.710 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_3.status 2024-04-06T09:59:36.756 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 2 2024-04-06T09:59:36.756 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 2 2024-04-06T09:59:36.756 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:59:36.756 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=9, tag_tid=2, entry_tid=1], [object_number=8, tag_tid=2, entry_tid=0], [object_number=11, tag_tid=1, entry_tid=11107], [object_number=10, tag_tid=1, entry_tid=11106]]], state=connected] 2024-04-06T09:59:36.756 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=8, tag_tid=1, entry_tid=11108], [object_number=11, tag_tid=1, entry_tid=11107], [object_number=10, tag_tid=1, entry_tid=11106], [object_number=9, tag_tid=1, entry_tid=11105]]], state=connected] 2024-04-06T09:59:36.759 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_3.status 2024-04-06T09:59:36.761 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=9, tag_tid=2, entry_tid=1' 2024-04-06T09:59:36.762 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror image_3 2024-04-06T09:59:36.762 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:59:36.762 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:59:36.762 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_3 2024-04-06T09:59:36.762 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror image_3 '..*' 2024-04-06T09:59:36.762 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:59:36.763 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:59:36.763 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_3 2024-04-06T09:59:36.763 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:59:36.763 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-image_3.status 2024-04-06T09:59:36.763 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-image_3.status 2024-04-06T09:59:36.763 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:59:36.763 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-image_3.status 2024-04-06T09:59:36.764 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_3 2024-04-06T09:59:36.764 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_3.status 2024-04-06T09:59:36.809 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 2 2024-04-06T09:59:36.809 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 2 2024-04-06T09:59:36.810 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:59:36.810 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=9, tag_tid=2, entry_tid=1], [object_number=8, tag_tid=2, entry_tid=0], [object_number=11, tag_tid=1, entry_tid=11107], [object_number=10, tag_tid=1, entry_tid=11106]]], state=connected] 2024-04-06T09:59:36.810 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=8, tag_tid=1, entry_tid=11108], [object_number=11, tag_tid=1, entry_tid=11107], [object_number=10, tag_tid=1, entry_tid=11106], [object_number=9, tag_tid=1, entry_tid=11105]]], state=connected] 2024-04-06T09:59:36.813 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_3.status 2024-04-06T09:59:36.815 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=8, tag_tid=1, entry_tid=11108' 2024-04-06T09:59:36.815 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=9, tag_tid=2, entry_tid=1' -a 'object_number=9, tag_tid=2, entry_tid=1' = 'object_number=8, tag_tid=1, entry_tid=11108' 2024-04-06T09:59:36.815 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=8, tag_tid=1, entry_tid=11108' '!=' '' 2024-04-06T09:59:36.815 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:59:36.815 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=8, tag_tid=1, entry_tid=11108' = '' 2024-04-06T09:59:36.815 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=8, tag_tid=1, entry_tid=11108' 2024-04-06T09:59:36.816 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=9, tag_tid=2, entry_tid=1' 2024-04-06T09:59:36.816 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:59:36.816 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:59:36.816 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=2 2024-04-06T09:59:36.817 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=8, tag_tid=1, entry_tid=11108' 2024-04-06T09:59:36.817 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:59:36.817 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:59:36.818 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=1 2024-04-06T09:59:36.819 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=9, tag_tid=2, entry_tid=1' 2024-04-06T09:59:36.819 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:59:36.819 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:59:36.820 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:59:36.821 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=8, tag_tid=1, entry_tid=11108' 2024-04-06T09:59:36.821 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:59:36.821 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:59:36.822 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=11108 2024-04-06T09:59:36.822 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 2 = 1 -a 1 -le 11108 2024-04-06T09:59:36.822 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:59:36.822 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:59:36.822 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:59:37.024 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror image_3 2024-04-06T09:59:37.024 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:59:37.024 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:59:37.024 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_3 2024-04-06T09:59:37.024 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:59:37.024 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n image_3 ']' 2024-04-06T09:59:37.024 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/image_3' 2024-04-06T09:59:37.025 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_3 2024-04-06T09:59:37.025 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:59:37.025 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:59:37.025 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:59:37.025 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:59:37.025 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:59:37.025 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:59:37.025 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:59:37.025 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:59:37.026 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:59:37.028 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:59:37.028 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_3 2024-04-06T09:59:37.028 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:59:37.028 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:59:37.028 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:59:37.028 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:59:37.028 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:59:37.028 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:59:37.028 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:59:37.028 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:59:37.029 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:59:37.029 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:59:37.029 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:59:37.029 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:59:37.029 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:59:37.029 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:59:37.029 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:59:37.029 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:59:37.029 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:59:37.029 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:59:37.029 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:59:37.029 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:59:37.029 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:59:37.029 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:59:37.029 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:59:37.030 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:59:37.030 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:59:37.030 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:59:37.030 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:59:37.030 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:59:37.030 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:59:37.030 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:59:37.053 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:59:37.054 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:59:37.054 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:59:37.054 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_3 2024-04-06T09:59:37.160 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:Invalid command: unused arguments: ['mirror/image_3'] 2024-04-06T09:59:37.160 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:rbd mirror flush : flush rbd mirror 2024-04-06T09:59:37.160 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:admin_socket: invalid command 2024-04-06T09:59:37.163 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:59:37.163 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/image_3 2024-04-06T09:59:37.163 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:1 2024-04-06T09:59:37.163 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:59:37.163 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:59:37.163 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-06T09:59:37.163 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:1 2024-04-06T09:59:37.163 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:59:37.163 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:59:37.163 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:59:37.163 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=1 2024-04-06T09:59:37.163 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-06T09:59:37.164 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:59:37.164 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:59:37.164 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=1 2024-04-06T09:59:37.164 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=1 2024-04-06T09:59:37.165 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-06T09:59:37.165 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:1 2024-04-06T09:59:37.165 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:59:37.165 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:59:37.165 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-06T09:59:37.165 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:1 2024-04-06T09:59:37.165 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:59:37.166 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:59:37.166 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:59:37.166 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=1 2024-04-06T09:59:37.166 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-06T09:59:37.166 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:59:37.166 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:59:37.166 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=1 2024-04-06T09:59:37.166 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=1 2024-04-06T09:59:37.166 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-06T09:59:37.189 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T09:59:37.189 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T09:59:37.189 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T09:59:37.189 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/image_3 2024-04-06T09:59:37.341 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:59:37.342 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror image_3 2024-04-06T09:59:37.342 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:59:37.342 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:59:37.343 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_3 2024-04-06T09:59:37.343 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror image_3 '' 2024-04-06T09:59:37.343 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:59:37.343 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:59:37.343 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_3 2024-04-06T09:59:37.343 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:59:37.343 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-image_3.status 2024-04-06T09:59:37.344 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-image_3.status 2024-04-06T09:59:37.344 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:59:37.345 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-image_3.status 2024-04-06T09:59:37.346 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_3 2024-04-06T09:59:37.346 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_3.status 2024-04-06T09:59:37.396 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 2 2024-04-06T09:59:37.396 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 2 2024-04-06T09:59:37.396 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:59:37.396 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=9, tag_tid=2, entry_tid=1], [object_number=8, tag_tid=2, entry_tid=0], [object_number=11, tag_tid=1, entry_tid=11107], [object_number=10, tag_tid=1, entry_tid=11106]]], state=connected] 2024-04-06T09:59:37.396 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=8, tag_tid=1, entry_tid=11108], [object_number=11, tag_tid=1, entry_tid=11107], [object_number=10, tag_tid=1, entry_tid=11106], [object_number=9, tag_tid=1, entry_tid=11105]]], state=connected] 2024-04-06T09:59:37.399 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_3.status 2024-04-06T09:59:37.402 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=9, tag_tid=2, entry_tid=1' 2024-04-06T09:59:37.402 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror image_3 2024-04-06T09:59:37.402 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:59:37.402 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:59:37.402 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_3 2024-04-06T09:59:37.402 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror image_3 '..*' 2024-04-06T09:59:37.402 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:59:37.402 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:59:37.403 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_3 2024-04-06T09:59:37.403 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:59:37.403 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-image_3.status 2024-04-06T09:59:37.403 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-image_3.status 2024-04-06T09:59:37.403 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:59:37.403 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-image_3.status 2024-04-06T09:59:37.404 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_3.status 2024-04-06T09:59:37.404 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_3 2024-04-06T09:59:37.449 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 2 2024-04-06T09:59:37.450 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 2 2024-04-06T09:59:37.450 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:59:37.450 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=9, tag_tid=2, entry_tid=1], [object_number=8, tag_tid=2, entry_tid=0], [object_number=11, tag_tid=1, entry_tid=11107], [object_number=10, tag_tid=1, entry_tid=11106]]], state=connected] 2024-04-06T09:59:37.450 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=8, tag_tid=1, entry_tid=11108], [object_number=11, tag_tid=1, entry_tid=11107], [object_number=10, tag_tid=1, entry_tid=11106], [object_number=9, tag_tid=1, entry_tid=11105]]], state=connected] 2024-04-06T09:59:37.453 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_3.status 2024-04-06T09:59:37.455 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=8, tag_tid=1, entry_tid=11108' 2024-04-06T09:59:37.455 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=9, tag_tid=2, entry_tid=1' -a 'object_number=9, tag_tid=2, entry_tid=1' = 'object_number=8, tag_tid=1, entry_tid=11108' 2024-04-06T09:59:37.455 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=8, tag_tid=1, entry_tid=11108' '!=' 'object_number=8, tag_tid=1, entry_tid=11108' 2024-04-06T09:59:37.455 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:59:37.455 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.4 2024-04-06T09:59:37.857 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror image_3 2024-04-06T09:59:37.857 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:59:37.858 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:59:37.858 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_3 2024-04-06T09:59:37.858 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:59:37.858 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n image_3 ']' 2024-04-06T09:59:37.858 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/image_3' 2024-04-06T09:59:37.858 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_3 2024-04-06T09:59:37.858 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:59:37.858 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:59:37.858 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:59:37.858 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:59:37.858 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:59:37.858 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:59:37.859 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:59:37.860 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:59:37.860 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:59:37.860 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:59:37.860 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_3 2024-04-06T09:59:37.860 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:59:37.860 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:59:37.860 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:59:37.860 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:59:37.860 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:59:37.860 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:59:37.861 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:59:37.861 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:59:37.861 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:59:37.861 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:59:37.861 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:59:37.861 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:59:37.861 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:59:37.861 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:59:37.862 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:59:37.862 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:59:37.863 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:59:37.863 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:59:37.863 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:59:37.863 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:59:37.863 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:59:37.863 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:59:37.863 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:59:37.863 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:59:37.863 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:59:37.863 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:59:37.863 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:59:37.863 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:59:37.863 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:59:37.863 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:59:37.887 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:59:37.887 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:59:37.887 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:59:37.887 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_3 2024-04-06T09:59:37.993 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:Invalid command: unused arguments: ['mirror/image_3'] 2024-04-06T09:59:37.993 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:rbd mirror flush : flush rbd mirror 2024-04-06T09:59:37.993 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:admin_socket: invalid command 2024-04-06T09:59:37.995 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:59:37.996 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/image_3 2024-04-06T09:59:37.996 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:1 2024-04-06T09:59:37.996 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:59:37.996 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:59:37.996 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-06T09:59:37.996 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:1 2024-04-06T09:59:37.996 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:59:37.996 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:59:37.996 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:59:37.996 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=1 2024-04-06T09:59:37.996 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-06T09:59:37.996 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:59:37.996 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:59:37.996 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=1 2024-04-06T09:59:37.996 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=1 2024-04-06T09:59:37.997 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-06T09:59:37.998 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:1 2024-04-06T09:59:37.998 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:59:37.998 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:59:37.998 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-06T09:59:37.998 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:1 2024-04-06T09:59:37.998 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:59:37.998 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:59:37.998 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:59:37.998 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=1 2024-04-06T09:59:37.998 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-06T09:59:37.998 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:59:37.998 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:59:37.999 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=1 2024-04-06T09:59:37.999 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=1 2024-04-06T09:59:37.999 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-06T09:59:38.016 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T09:59:38.016 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T09:59:38.016 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T09:59:38.016 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/image_3 2024-04-06T09:59:38.159 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:59:38.160 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror image_3 2024-04-06T09:59:38.160 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:59:38.160 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:59:38.160 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_3 2024-04-06T09:59:38.161 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror image_3 '' 2024-04-06T09:59:38.161 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:59:38.161 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:59:38.161 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_3 2024-04-06T09:59:38.161 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:59:38.161 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-image_3.status 2024-04-06T09:59:38.161 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-image_3.status 2024-04-06T09:59:38.162 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:59:38.163 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-image_3.status 2024-04-06T09:59:38.164 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_3 2024-04-06T09:59:38.164 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_3.status 2024-04-06T09:59:38.209 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 2 2024-04-06T09:59:38.209 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 2 2024-04-06T09:59:38.209 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:59:38.209 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=9, tag_tid=2, entry_tid=1], [object_number=8, tag_tid=2, entry_tid=0], [object_number=11, tag_tid=1, entry_tid=11107], [object_number=10, tag_tid=1, entry_tid=11106]]], state=connected] 2024-04-06T09:59:38.209 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=8, tag_tid=1, entry_tid=11108], [object_number=11, tag_tid=1, entry_tid=11107], [object_number=10, tag_tid=1, entry_tid=11106], [object_number=9, tag_tid=1, entry_tid=11105]]], state=connected] 2024-04-06T09:59:38.212 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_3.status 2024-04-06T09:59:38.214 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=9, tag_tid=2, entry_tid=1' 2024-04-06T09:59:38.215 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror image_3 2024-04-06T09:59:38.215 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:59:38.215 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:59:38.215 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_3 2024-04-06T09:59:38.215 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror image_3 '..*' 2024-04-06T09:59:38.215 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:59:38.215 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:59:38.215 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_3 2024-04-06T09:59:38.216 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:59:38.216 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-image_3.status 2024-04-06T09:59:38.216 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-image_3.status 2024-04-06T09:59:38.216 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:59:38.216 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-image_3.status 2024-04-06T09:59:38.216 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_3 2024-04-06T09:59:38.217 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_3.status 2024-04-06T09:59:38.263 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 2 2024-04-06T09:59:38.263 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 2 2024-04-06T09:59:38.263 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:59:38.263 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=9, tag_tid=2, entry_tid=1], [object_number=8, tag_tid=2, entry_tid=0], [object_number=11, tag_tid=1, entry_tid=11107], [object_number=10, tag_tid=1, entry_tid=11106]]], state=connected] 2024-04-06T09:59:38.263 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=8, tag_tid=1, entry_tid=11108], [object_number=11, tag_tid=1, entry_tid=11107], [object_number=10, tag_tid=1, entry_tid=11106], [object_number=9, tag_tid=1, entry_tid=11105]]], state=connected] 2024-04-06T09:59:38.266 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_3.status 2024-04-06T09:59:38.268 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=8, tag_tid=1, entry_tid=11108' 2024-04-06T09:59:38.268 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=9, tag_tid=2, entry_tid=1' -a 'object_number=9, tag_tid=2, entry_tid=1' = 'object_number=8, tag_tid=1, entry_tid=11108' 2024-04-06T09:59:38.268 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=8, tag_tid=1, entry_tid=11108' '!=' 'object_number=8, tag_tid=1, entry_tid=11108' 2024-04-06T09:59:38.268 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:59:38.268 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.8 2024-04-06T09:59:39.070 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror image_3 2024-04-06T09:59:39.071 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:59:39.071 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:59:39.071 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_3 2024-04-06T09:59:39.071 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:59:39.071 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n image_3 ']' 2024-04-06T09:59:39.071 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/image_3' 2024-04-06T09:59:39.071 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_3 2024-04-06T09:59:39.071 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:59:39.071 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:59:39.071 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:59:39.071 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:59:39.072 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:59:39.072 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:59:39.072 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:59:39.072 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:59:39.072 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:59:39.076 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:59:39.076 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_3 2024-04-06T09:59:39.076 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:59:39.076 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:59:39.076 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:59:39.076 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:59:39.076 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:59:39.076 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:59:39.076 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:59:39.076 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:59:39.076 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:59:39.077 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:59:39.077 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:59:39.077 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:59:39.077 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:59:39.077 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:59:39.077 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:59:39.077 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:59:39.077 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:59:39.077 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:59:39.077 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:59:39.077 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:59:39.077 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:59:39.077 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:59:39.077 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:59:39.077 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:59:39.078 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:59:39.078 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:59:39.078 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:59:39.078 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:59:39.078 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:59:39.078 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:59:39.102 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:59:39.102 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:59:39.102 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:59:39.102 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_3 2024-04-06T09:59:39.207 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:Invalid command: unused arguments: ['mirror/image_3'] 2024-04-06T09:59:39.208 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:rbd mirror flush : flush rbd mirror 2024-04-06T09:59:39.208 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:admin_socket: invalid command 2024-04-06T09:59:39.210 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:59:39.210 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/image_3 2024-04-06T09:59:39.210 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:1 2024-04-06T09:59:39.210 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:59:39.210 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:59:39.210 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-06T09:59:39.210 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:1 2024-04-06T09:59:39.210 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:59:39.211 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:59:39.211 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:59:39.211 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=1 2024-04-06T09:59:39.211 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-06T09:59:39.211 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:59:39.211 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:59:39.211 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=1 2024-04-06T09:59:39.211 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=1 2024-04-06T09:59:39.213 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-06T09:59:39.213 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:1 2024-04-06T09:59:39.213 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:59:39.213 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:59:39.213 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-06T09:59:39.213 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:1 2024-04-06T09:59:39.213 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:59:39.214 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:59:39.214 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:59:39.214 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=1 2024-04-06T09:59:39.214 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-06T09:59:39.214 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:59:39.214 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:59:39.214 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=1 2024-04-06T09:59:39.214 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=1 2024-04-06T09:59:39.214 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-06T09:59:39.235 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T09:59:39.236 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T09:59:39.236 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T09:59:39.236 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/image_3 2024-04-06T09:59:39.379 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:59:39.381 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror image_3 2024-04-06T09:59:39.381 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:59:39.381 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:59:39.381 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_3 2024-04-06T09:59:39.381 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror image_3 '' 2024-04-06T09:59:39.381 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:59:39.381 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:59:39.381 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_3 2024-04-06T09:59:39.381 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:59:39.382 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-image_3.status 2024-04-06T09:59:39.382 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-image_3.status 2024-04-06T09:59:39.382 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:59:39.384 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-image_3.status 2024-04-06T09:59:39.385 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_3 2024-04-06T09:59:39.385 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_3.status 2024-04-06T09:59:39.436 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 2 2024-04-06T09:59:39.436 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 2 2024-04-06T09:59:39.436 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:59:39.436 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=9, tag_tid=2, entry_tid=1], [object_number=8, tag_tid=2, entry_tid=0], [object_number=11, tag_tid=1, entry_tid=11107], [object_number=10, tag_tid=1, entry_tid=11106]]], state=connected] 2024-04-06T09:59:39.436 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=8, tag_tid=1, entry_tid=11108], [object_number=11, tag_tid=1, entry_tid=11107], [object_number=10, tag_tid=1, entry_tid=11106], [object_number=9, tag_tid=1, entry_tid=11105]]], state=connected] 2024-04-06T09:59:39.439 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_3.status 2024-04-06T09:59:39.441 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=9, tag_tid=2, entry_tid=1' 2024-04-06T09:59:39.442 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror image_3 2024-04-06T09:59:39.442 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:59:39.442 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:59:39.442 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_3 2024-04-06T09:59:39.442 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror image_3 '..*' 2024-04-06T09:59:39.442 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:59:39.442 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:59:39.442 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_3 2024-04-06T09:59:39.443 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:59:39.443 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-image_3.status 2024-04-06T09:59:39.443 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-image_3.status 2024-04-06T09:59:39.443 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:59:39.443 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-image_3.status 2024-04-06T09:59:39.444 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_3 2024-04-06T09:59:39.444 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_3.status 2024-04-06T09:59:39.491 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 2 2024-04-06T09:59:39.491 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 2 2024-04-06T09:59:39.491 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:59:39.491 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=9, tag_tid=2, entry_tid=1], [object_number=8, tag_tid=2, entry_tid=0], [object_number=11, tag_tid=1, entry_tid=11107], [object_number=10, tag_tid=1, entry_tid=11106]]], state=connected] 2024-04-06T09:59:39.491 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=8, tag_tid=1, entry_tid=11108], [object_number=11, tag_tid=1, entry_tid=11107], [object_number=10, tag_tid=1, entry_tid=11106], [object_number=9, tag_tid=1, entry_tid=11105]]], state=connected] 2024-04-06T09:59:39.494 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_3.status 2024-04-06T09:59:39.496 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=8, tag_tid=1, entry_tid=11108' 2024-04-06T09:59:39.496 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=9, tag_tid=2, entry_tid=1' -a 'object_number=9, tag_tid=2, entry_tid=1' = 'object_number=8, tag_tid=1, entry_tid=11108' 2024-04-06T09:59:39.496 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=8, tag_tid=1, entry_tid=11108' '!=' 'object_number=8, tag_tid=1, entry_tid=11108' 2024-04-06T09:59:39.496 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:59:39.496 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 1.6 2024-04-06T09:59:41.098 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror image_3 2024-04-06T09:59:41.098 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:59:41.098 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:59:41.099 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_3 2024-04-06T09:59:41.099 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:59:41.099 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n image_3 ']' 2024-04-06T09:59:41.099 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/image_3' 2024-04-06T09:59:41.099 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_3 2024-04-06T09:59:41.099 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:59:41.099 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:59:41.099 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:59:41.099 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:59:41.099 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:59:41.099 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:59:41.099 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:59:41.100 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:59:41.100 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:59:41.102 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:59:41.102 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_3 2024-04-06T09:59:41.102 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:59:41.102 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:59:41.102 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:59:41.103 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:59:41.103 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:59:41.103 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:59:41.103 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:59:41.103 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:59:41.103 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:59:41.103 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:59:41.103 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:59:41.103 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:59:41.103 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:59:41.103 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:59:41.104 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:59:41.104 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:59:41.104 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:59:41.104 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:59:41.104 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:59:41.105 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:59:41.105 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:59:41.105 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:59:41.105 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:59:41.105 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:59:41.105 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:59:41.105 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:59:41.105 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:59:41.105 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:59:41.105 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:59:41.105 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:59:41.129 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:59:41.130 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:59:41.130 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:59:41.130 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_3 2024-04-06T09:59:41.236 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:Invalid command: unused arguments: ['mirror/image_3'] 2024-04-06T09:59:41.236 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:rbd mirror flush : flush rbd mirror 2024-04-06T09:59:41.236 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:admin_socket: invalid command 2024-04-06T09:59:41.238 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:59:41.238 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/image_3 2024-04-06T09:59:41.238 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:1 2024-04-06T09:59:41.238 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:59:41.238 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:59:41.238 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-06T09:59:41.239 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:1 2024-04-06T09:59:41.239 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:59:41.239 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:59:41.239 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:59:41.239 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=1 2024-04-06T09:59:41.239 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-06T09:59:41.239 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:59:41.239 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:59:41.239 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=1 2024-04-06T09:59:41.239 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=1 2024-04-06T09:59:41.240 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-06T09:59:41.240 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:1 2024-04-06T09:59:41.240 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:59:41.240 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:59:41.240 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-06T09:59:41.240 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:1 2024-04-06T09:59:41.240 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:59:41.240 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:59:41.240 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:59:41.241 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=1 2024-04-06T09:59:41.241 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-06T09:59:41.241 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:59:41.241 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:59:41.241 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=1 2024-04-06T09:59:41.241 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=1 2024-04-06T09:59:41.241 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-06T09:59:41.264 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T09:59:41.264 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T09:59:41.264 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T09:59:41.264 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/image_3 2024-04-06T09:59:41.436 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:59:41.436 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror image_3 2024-04-06T09:59:41.436 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:59:41.436 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:59:41.436 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_3 2024-04-06T09:59:41.436 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror image_3 '' 2024-04-06T09:59:41.436 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:59:41.436 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:59:41.436 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_3 2024-04-06T09:59:41.437 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:59:41.437 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-image_3.status 2024-04-06T09:59:41.437 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-image_3.status 2024-04-06T09:59:41.437 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:59:41.439 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-image_3.status 2024-04-06T09:59:41.439 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_3 2024-04-06T09:59:41.439 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_3.status 2024-04-06T09:59:41.488 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 2 2024-04-06T09:59:41.488 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 2 2024-04-06T09:59:41.489 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:59:41.489 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=9, tag_tid=2, entry_tid=1], [object_number=8, tag_tid=2, entry_tid=0], [object_number=11, tag_tid=1, entry_tid=11107], [object_number=10, tag_tid=1, entry_tid=11106]]], state=connected] 2024-04-06T09:59:41.489 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=9, tag_tid=2, entry_tid=1], [object_number=8, tag_tid=2, entry_tid=0], [object_number=11, tag_tid=1, entry_tid=11107], [object_number=10, tag_tid=1, entry_tid=11106]]], state=connected] 2024-04-06T09:59:41.492 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_3.status 2024-04-06T09:59:41.494 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=9, tag_tid=2, entry_tid=1' 2024-04-06T09:59:41.495 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror image_3 2024-04-06T09:59:41.495 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:59:41.495 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:59:41.495 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_3 2024-04-06T09:59:41.495 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror image_3 '..*' 2024-04-06T09:59:41.495 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:59:41.495 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:59:41.495 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_3 2024-04-06T09:59:41.495 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:59:41.495 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-image_3.status 2024-04-06T09:59:41.495 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-image_3.status 2024-04-06T09:59:41.495 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:59:41.496 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-image_3.status 2024-04-06T09:59:41.496 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_3 2024-04-06T09:59:41.496 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_3.status 2024-04-06T09:59:41.542 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 2 2024-04-06T09:59:41.542 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 2 2024-04-06T09:59:41.542 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:59:41.542 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=9, tag_tid=2, entry_tid=1], [object_number=8, tag_tid=2, entry_tid=0], [object_number=11, tag_tid=1, entry_tid=11107], [object_number=10, tag_tid=1, entry_tid=11106]]], state=connected] 2024-04-06T09:59:41.542 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=9, tag_tid=2, entry_tid=1], [object_number=8, tag_tid=2, entry_tid=0], [object_number=11, tag_tid=1, entry_tid=11107], [object_number=10, tag_tid=1, entry_tid=11106]]], state=connected] 2024-04-06T09:59:41.545 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_3.status 2024-04-06T09:59:41.547 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=9, tag_tid=2, entry_tid=1' 2024-04-06T09:59:41.547 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=9, tag_tid=2, entry_tid=1' -a 'object_number=9, tag_tid=2, entry_tid=1' = 'object_number=9, tag_tid=2, entry_tid=1' 2024-04-06T09:59:41.547 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:59:41.547 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_snap_present cluster1 mirror image_3 snap 2024-04-06T09:59:41.548 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:59:41.548 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:59:41.548 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_3 2024-04-06T09:59:41.548 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local snap_name=snap 2024-04-06T09:59:41.548 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local s 2024-04-06T09:59:41.548 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-04-06T09:59:41.548 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 1 2024-04-06T09:59:41.649 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-04-06T09:59:41.643+0000 7fd1b6210640 -1 reset not still connected to 0x55ef993061a0 2024-04-06T09:59:41.661 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-04-06T09:59:41.659+0000 7fd1b6210640 -1 reset not still connected to 0x55ef99306ea0 2024-04-06T09:59:41.663 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-04-06T09:59:41.667+0000 7fd1b6210640 -1 reset not still connected to 0x55ef997761a0 2024-04-06T09:59:41.664 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-04-06T09:59:41.667+0000 7fd1b6210640 -1 reset not still connected to 0x55ef99776680 2024-04-06T09:59:41.664 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-04-06T09:59:41.667+0000 7fd1b6210640 -1 reset not still connected to 0x55ef99b88000 2024-04-06T09:59:41.666 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-04-06T09:59:41.667+0000 7fd1b6210640 -1 reset not still connected to 0x55ef99b88d00 2024-04-06T09:59:41.666 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-04-06T09:59:41.667+0000 7fd1b6210640 -1 reset not still connected to 0x55ef99b89ad0 2024-04-06T09:59:41.666 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-04-06T09:59:41.667+0000 7fd1b6210640 -1 reset not still connected to 0x55ef99ebf6c0 2024-04-06T09:59:41.666 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-04-06T09:59:41.667+0000 7fd1b6210640 -1 reset not still connected to 0x55ef9ac60270 2024-04-06T09:59:41.666 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-04-06T09:59:41.667+0000 7fd1b6210640 -1 reset not still connected to 0x55ef9b657e10 2024-04-06T09:59:41.666 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-04-06T09:59:41.667+0000 7fd1b6210640 -1 reset not still connected to 0x55ef9bbe84e0 2024-04-06T09:59:41.666 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-04-06T09:59:41.667+0000 7fd1b6210640 -1 reset not still connected to 0x55ef9bbe8d00 2024-04-06T09:59:41.666 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-04-06T09:59:41.667+0000 7fd1b6210640 -1 reset not still connected to 0x55ef9bbe9ee0 2024-04-06T09:59:41.666 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-04-06T09:59:41.667+0000 7fd1b6210640 -1 reset not still connected to 0x55ef9bd5c9c0 2024-04-06T09:59:41.666 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-04-06T09:59:41.667+0000 7fd1b6210640 -1 reset not still connected to 0x55ef9bd5d520 2024-04-06T09:59:41.667 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-04-06T09:59:41.667+0000 7fd1b6210640 -1 reset not still connected to 0x55ef9bd5dd40 2024-04-06T09:59:41.667 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-04-06T09:59:41.667+0000 7fd1b6210640 -1 reset not still connected to 0x55ef9bd86820 2024-04-06T09:59:41.667 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-04-06T09:59:41.667+0000 7fd1b6210640 -1 reset not still connected to 0x55ef9c103110 2024-04-06T09:59:41.667 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-04-06T09:59:41.667+0000 7fd1b6210640 -1 reset not still connected to 0x55ef9c103930 2024-04-06T09:59:41.667 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-04-06T09:59:41.667+0000 7fd1b6210640 -1 reset not still connected to 0x55ef9c103d40 2024-04-06T09:59:41.667 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-04-06T09:59:41.667+0000 7fd1b6210640 -1 reset not still connected to 0x55ef9e698c30 2024-04-06T09:59:41.667 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-04-06T09:59:41.667+0000 7fd1b6210640 -1 reset not still connected to 0x55ef9e88f520 2024-04-06T09:59:41.667 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-04-06T09:59:41.667+0000 7fd1b6210640 -1 reset not still connected to 0x55ef9eee21a0 2024-04-06T09:59:41.667 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-04-06T09:59:41.667+0000 7fd1b6210640 -1 reset not still connected to 0x55ef9eee3a00 2024-04-06T09:59:41.667 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-04-06T09:59:41.667+0000 7fd1b6210640 -1 reset not still connected to 0x55efa18cc270 2024-04-06T09:59:41.667 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-04-06T09:59:41.667+0000 7fd1b6210640 -1 reset not still connected to 0x55efa18cc9c0 2024-04-06T09:59:41.667 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-04-06T09:59:41.667+0000 7fd1b6210640 -1 reset not still connected to 0x55efa18cca90 2024-04-06T09:59:41.667 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-04-06T09:59:41.667+0000 7fd1b6210640 -1 reset not still connected to 0x55efa96c96c0 2024-04-06T09:59:41.667 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-04-06T09:59:41.667+0000 7fd1b6210640 -1 reset not still connected to 0x55efae5ffe10 2024-04-06T09:59:41.667 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-04-06T09:59:41.667+0000 7fd1b6210640 -1 reset not still connected to 0x55efbb630d00 2024-04-06T09:59:41.668 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-04-06T09:59:41.667+0000 7fd1b6210640 -1 reset not still connected to 0x55efd81ee8f0 2024-04-06T09:59:41.668 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-04-06T09:59:41.667+0000 7fd1b6210640 -1 reset not still connected to 0x55efe06a9860 2024-04-06T09:59:42.548 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 info mirror/image_3@snap 2024-04-06T09:59:42.612 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:rbd image 'image_3': 2024-04-06T09:59:42.612 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: size 128 MiB in 32 objects 2024-04-06T09:59:42.612 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: order 22 (4 MiB objects) 2024-04-06T09:59:42.612 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: snapshot_count: 1 2024-04-06T09:59:42.612 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: id: 10ac62e76b57 2024-04-06T09:59:42.612 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: block_name_prefix: rbd_data.10ac62e76b57 2024-04-06T09:59:42.612 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: format: 2 2024-04-06T09:59:42.612 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: features: layering, exclusive-lock, journaling 2024-04-06T09:59:42.612 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: op_features: 2024-04-06T09:59:42.613 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: flags: 2024-04-06T09:59:42.613 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: create_timestamp: Sat Apr 6 09:51:01 2024 2024-04-06T09:59:42.613 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: access_timestamp: Sat Apr 6 09:51:01 2024 2024-04-06T09:59:42.613 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: modify_timestamp: Sat Apr 6 09:52:02 2024 2024-04-06T09:59:42.613 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: protected: False 2024-04-06T09:59:42.613 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: journal: 10ac62e76b57 2024-04-06T09:59:42.613 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: mirroring state: enabled 2024-04-06T09:59:42.613 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: mirroring mode: journal 2024-04-06T09:59:42.613 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: mirroring global id: 5d5515f6-48c6-40ae-8779-1015b5272235 2024-04-06T09:59:42.613 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: mirroring primary: false 2024-04-06T09:59:42.617 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:59:42.617 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ compare_image_snaps mirror image_3 snap 2024-04-06T09:59:42.617 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:59:42.617 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_3 2024-04-06T09:59:42.617 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local snap_name=snap 2024-04-06T09:59:42.617 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local ret=0 2024-04-06T09:59:42.617 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local rmt_export=/tmp/tmp.BhW6xKmROU/cluster2-mirror-image_3.export 2024-04-06T09:59:42.617 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loc_export=/tmp/tmp.BhW6xKmROU/cluster1-mirror-image_3.export 2024-04-06T09:59:42.617 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rm -f /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_3.export /tmp/tmp.BhW6xKmROU/cluster1-mirror-image_3.export 2024-04-06T09:59:42.618 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster2 -p mirror export image_3@snap /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_3.export 2024-04-06T09:59:42.801 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-04-06T09:59:42.809 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 -p mirror export image_3@snap /tmp/tmp.BhW6xKmROU/cluster1-mirror-image_3.export 2024-04-06T09:59:43.173 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-04-06T09:59:43.185 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmp /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_3.export /tmp/tmp.BhW6xKmROU/cluster1-mirror-image_3.export 2024-04-06T09:59:43.266 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rm -f /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_3.export /tmp/tmp.BhW6xKmROU/cluster1-mirror-image_3.export 2024-04-06T09:59:43.307 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:59:43.307 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-06T09:59:43.307 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ image=image_4 2024-04-06T09:59:43.308 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ create_snap cluster2 mirror image_4 snap 2024-04-06T09:59:43.308 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-04-06T09:59:43.308 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:59:43.308 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_4 2024-04-06T09:59:43.308 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local snap_name=snap 2024-04-06T09:59:43.308 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster2 -p mirror snap create image_4@snap --debug-rbd=20 --debug-journaler=20 2024-04-06T09:59:44.832 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_image_replay_started cluster1 mirror image_4 2024-04-06T09:59:44.832 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:59:44.832 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:59:44.832 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_4 2024-04-06T09:59:44.832 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_image_replay_state cluster1 mirror image_4 started 2024-04-06T09:59:44.832 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:59:44.832 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:59:44.832 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_4 2024-04-06T09:59:44.832 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state=started 2024-04-06T09:59:44.832 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local s 2024-04-06T09:59:44.832 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:59:44.832 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 1 2024-04-06T09:59:45.833 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test_image_replay_state cluster1 mirror image_4 started 2024-04-06T09:59:45.833 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:59:45.833 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:59:45.833 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_4 2024-04-06T09:59:45.833 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local test_state=started 2024-04-06T09:59:45.834 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local status_result 2024-04-06T09:59:45.834 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local current_state=stopped 2024-04-06T09:59:45.835 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_4 2024-04-06T09:59:45.835 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -i state 2024-04-06T09:59:45.835 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_instance=cluster1 2024-04-06T09:59:45.835 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ shift 2024-04-06T09:59:45.835 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:59:45.835 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance=cluster1 2024-04-06T09:59:45.835 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local loop_instance 2024-04-06T09:59:45.836 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:59:45.836 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sleep 0 2024-04-06T09:59:45.836 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:59:45.837 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ seq 0 3 2024-04-06T09:59:45.839 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:59:45.839 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_4 2024-04-06T09:59:45.839 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1:0 2024-04-06T09:59:45.839 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ shift 2024-04-06T09:59:45.839 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:59:45.839 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:59:45.839 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:59:45.839 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=cluster 2024-04-06T09:59:45.839 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:59:45.839 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:59:45.839 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:59:45.840 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:59:45.840 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval cluster=cluster1 2024-04-06T09:59:45.840 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ cluster=cluster1 2024-04-06T09:59:45.840 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:59:45.840 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:59:45.841 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:59:45.841 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local local_cluster=cluster1:0 2024-04-06T09:59:45.841 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local cluster=cluster1 2024-04-06T09:59:45.841 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local instance 2024-04-06T09:59:45.841 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:59:45.841 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local val=cluster1:0 2024-04-06T09:59:45.841 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local cluster_var_name=local_cluster 2024-04-06T09:59:45.842 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local instance_var_name=instance 2024-04-06T09:59:45.842 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ cluster=cluster1 2024-04-06T09:59:45.842 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:59:45.842 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ '[' 0 = cluster1:0 ']' 2024-04-06T09:59:45.842 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ eval local_cluster=cluster1 2024-04-06T09:59:45.842 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++++ local_cluster=cluster1 2024-04-06T09:59:45.842 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ eval instance=0 2024-04-06T09:59:45.842 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++++ instance=0 2024-04-06T09:59:45.842 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:59:45.862 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:59:45.862 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:59:45.862 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:59:45.862 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_4 2024-04-06T09:59:45.964 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:Invalid command: unused arguments: ['mirror/image_4'] 2024-04-06T09:59:45.964 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:rbd mirror status : get status for rbd mirror 2024-04-06T09:59:45.964 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:admin_socket: invalid command 2024-04-06T09:59:45.966 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:59:45.966 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_4 2024-04-06T09:59:45.966 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1:1 2024-04-06T09:59:45.967 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ shift 2024-04-06T09:59:45.967 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:59:45.967 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-04-06T09:59:45.967 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:1 2024-04-06T09:59:45.967 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=cluster 2024-04-06T09:59:45.967 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:59:45.967 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:59:45.967 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=1 2024-04-06T09:59:45.967 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-06T09:59:45.967 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval cluster=cluster1 2024-04-06T09:59:45.967 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ cluster=cluster1 2024-04-06T09:59:45.967 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=1 2024-04-06T09:59:45.967 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=1 2024-04-06T09:59:45.969 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-04-06T09:59:45.969 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local local_cluster=cluster1:1 2024-04-06T09:59:45.969 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local cluster=cluster1 2024-04-06T09:59:45.969 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local instance 2024-04-06T09:59:45.969 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-06T09:59:45.969 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local val=cluster1:1 2024-04-06T09:59:45.969 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local cluster_var_name=local_cluster 2024-04-06T09:59:45.969 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local instance_var_name=instance 2024-04-06T09:59:45.969 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ cluster=cluster1 2024-04-06T09:59:45.969 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=1 2024-04-06T09:59:45.969 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ '[' 1 = cluster1:1 ']' 2024-04-06T09:59:45.969 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ eval local_cluster=cluster1 2024-04-06T09:59:45.970 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++++ local_cluster=cluster1 2024-04-06T09:59:45.970 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ eval instance=1 2024-04-06T09:59:45.970 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++++ instance=1 2024-04-06T09:59:45.970 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-06T09:59:45.991 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T09:59:45.991 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T09:59:45.991 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T09:59:45.991 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_4 2024-04-06T09:59:46.146 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ return 0 2024-04-06T09:59:46.147 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ status_result=' "state": "Replaying"' 2024-04-06T09:59:46.148 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ echo ' "state": "Replaying"' 2024-04-06T09:59:46.148 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep -i Replaying 2024-04-06T09:59:46.149 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: "state": "Replaying" 2024-04-06T09:59:46.151 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ current_state=started 2024-04-06T09:59:46.151 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test started = started 2024-04-06T09:59:46.151 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:59:46.151 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_4 2024-04-06T09:59:46.151 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local local_cluster=cluster1 2024-04-06T09:59:46.151 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-04-06T09:59:46.151 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:59:46.151 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_4 2024-04-06T09:59:46.151 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' journal = journal ']' 2024-04-06T09:59:46.151 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_journal_replay_complete cluster1 cluster2 mirror image_4 2024-04-06T09:59:46.151 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local local_cluster=cluster1 2024-04-06T09:59:46.151 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-04-06T09:59:46.152 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:59:46.152 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_4 2024-04-06T09:59:46.152 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local s master_pos mirror_pos last_mirror_pos 2024-04-06T09:59:46.152 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local master_tag master_entry mirror_tag mirror_entry 2024-04-06T09:59:46.152 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:59:46.152 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:59:46.152 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:59:46.353 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror image_4 2024-04-06T09:59:46.353 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:59:46.353 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:59:46.353 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_4 2024-04-06T09:59:46.354 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:59:46.354 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n image_4 ']' 2024-04-06T09:59:46.354 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/image_4' 2024-04-06T09:59:46.354 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_4 2024-04-06T09:59:46.354 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:59:46.354 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:59:46.354 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:59:46.354 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:59:46.354 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:59:46.354 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:59:46.354 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:59:46.355 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:59:46.355 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:59:46.359 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:59:46.359 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_4 2024-04-06T09:59:46.359 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:59:46.359 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:59:46.359 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:59:46.359 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:59:46.359 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:59:46.359 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:59:46.359 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:59:46.359 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:59:46.359 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:59:46.359 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:59:46.359 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:59:46.360 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:59:46.360 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:59:46.360 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:59:46.360 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:59:46.360 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:59:46.360 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:59:46.360 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:59:46.360 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:59:46.360 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:59:46.360 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:59:46.361 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:59:46.361 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:59:46.361 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:59:46.361 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:59:46.361 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:59:46.361 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:59:46.362 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:59:46.362 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:59:46.362 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:59:46.384 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:59:46.385 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:59:46.385 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:59:46.385 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_4 2024-04-06T09:59:46.490 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:Invalid command: unused arguments: ['mirror/image_4'] 2024-04-06T09:59:46.490 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:rbd mirror flush : flush rbd mirror 2024-04-06T09:59:46.490 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:admin_socket: invalid command 2024-04-06T09:59:46.494 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:59:46.494 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/image_4 2024-04-06T09:59:46.494 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:1 2024-04-06T09:59:46.494 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:59:46.494 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:59:46.494 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-06T09:59:46.494 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:1 2024-04-06T09:59:46.495 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:59:46.495 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:59:46.495 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:59:46.495 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=1 2024-04-06T09:59:46.495 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-06T09:59:46.495 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:59:46.495 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:59:46.495 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=1 2024-04-06T09:59:46.495 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=1 2024-04-06T09:59:46.495 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-06T09:59:46.495 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:1 2024-04-06T09:59:46.495 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:59:46.496 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:59:46.496 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-06T09:59:46.496 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:1 2024-04-06T09:59:46.496 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:59:46.496 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:59:46.496 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:59:46.496 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=1 2024-04-06T09:59:46.496 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-06T09:59:46.496 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:59:46.496 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:59:46.496 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=1 2024-04-06T09:59:46.496 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=1 2024-04-06T09:59:46.496 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-06T09:59:46.512 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T09:59:46.513 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T09:59:46.513 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T09:59:46.513 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/image_4 2024-04-06T09:59:46.649 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:59:46.650 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror image_4 2024-04-06T09:59:46.650 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:59:46.651 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:59:46.651 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_4 2024-04-06T09:59:46.651 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror image_4 '' 2024-04-06T09:59:46.651 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:59:46.651 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:59:46.651 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_4 2024-04-06T09:59:46.651 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:59:46.651 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-image_4.status 2024-04-06T09:59:46.651 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-image_4.status 2024-04-06T09:59:46.651 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:59:46.653 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-image_4.status 2024-04-06T09:59:46.654 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_4 2024-04-06T09:59:46.654 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_4.status 2024-04-06T09:59:46.703 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 2 2024-04-06T09:59:46.704 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 2 2024-04-06T09:59:46.704 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:59:46.704 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=9, tag_tid=2, entry_tid=1], [object_number=8, tag_tid=2, entry_tid=0], [object_number=11, tag_tid=1, entry_tid=11667], [object_number=10, tag_tid=1, entry_tid=11666]]], state=connected] 2024-04-06T09:59:46.704 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=8, tag_tid=1, entry_tid=11668], [object_number=11, tag_tid=1, entry_tid=11667], [object_number=10, tag_tid=1, entry_tid=11666], [object_number=9, tag_tid=1, entry_tid=11665]]], state=connected] 2024-04-06T09:59:46.707 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_4.status 2024-04-06T09:59:46.709 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=9, tag_tid=2, entry_tid=1' 2024-04-06T09:59:46.710 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror image_4 2024-04-06T09:59:46.710 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:59:46.710 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:59:46.710 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_4 2024-04-06T09:59:46.710 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror image_4 '..*' 2024-04-06T09:59:46.710 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:59:46.711 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:59:46.711 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_4 2024-04-06T09:59:46.711 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:59:46.711 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-image_4.status 2024-04-06T09:59:46.711 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-image_4.status 2024-04-06T09:59:46.711 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:59:46.712 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-image_4.status 2024-04-06T09:59:46.712 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_4 2024-04-06T09:59:46.712 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_4.status 2024-04-06T09:59:46.757 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 2 2024-04-06T09:59:46.757 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 2 2024-04-06T09:59:46.758 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:59:46.758 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=9, tag_tid=2, entry_tid=1], [object_number=8, tag_tid=2, entry_tid=0], [object_number=11, tag_tid=1, entry_tid=11667], [object_number=10, tag_tid=1, entry_tid=11666]]], state=connected] 2024-04-06T09:59:46.758 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=8, tag_tid=1, entry_tid=11668], [object_number=11, tag_tid=1, entry_tid=11667], [object_number=10, tag_tid=1, entry_tid=11666], [object_number=9, tag_tid=1, entry_tid=11665]]], state=connected] 2024-04-06T09:59:46.761 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_4.status 2024-04-06T09:59:46.763 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=8, tag_tid=1, entry_tid=11668' 2024-04-06T09:59:46.763 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=9, tag_tid=2, entry_tid=1' -a 'object_number=9, tag_tid=2, entry_tid=1' = 'object_number=8, tag_tid=1, entry_tid=11668' 2024-04-06T09:59:46.763 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=8, tag_tid=1, entry_tid=11668' '!=' '' 2024-04-06T09:59:46.763 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:59:46.763 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=8, tag_tid=1, entry_tid=11668' = '' 2024-04-06T09:59:46.763 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=8, tag_tid=1, entry_tid=11668' 2024-04-06T09:59:46.763 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=9, tag_tid=2, entry_tid=1' 2024-04-06T09:59:46.763 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:59:46.763 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:59:46.764 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=2 2024-04-06T09:59:46.765 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=8, tag_tid=1, entry_tid=11668' 2024-04-06T09:59:46.765 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:59:46.765 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:59:46.766 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=1 2024-04-06T09:59:46.767 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=9, tag_tid=2, entry_tid=1' 2024-04-06T09:59:46.767 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:59:46.767 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:59:46.768 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:59:46.768 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=8, tag_tid=1, entry_tid=11668' 2024-04-06T09:59:46.768 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:59:46.768 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:59:46.770 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=11668 2024-04-06T09:59:46.770 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 2 = 1 -a 1 -le 11668 2024-04-06T09:59:46.770 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:59:46.770 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:59:46.770 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:59:46.972 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror image_4 2024-04-06T09:59:46.972 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:59:46.972 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:59:46.972 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_4 2024-04-06T09:59:46.972 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:59:46.972 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n image_4 ']' 2024-04-06T09:59:46.972 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/image_4' 2024-04-06T09:59:46.972 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_4 2024-04-06T09:59:46.972 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:59:46.973 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:59:46.973 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:59:46.973 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:59:46.973 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:59:46.973 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:59:46.973 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:59:46.973 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:59:46.974 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:59:46.975 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:59:46.976 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_4 2024-04-06T09:59:46.976 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:59:46.976 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:59:46.976 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:59:46.976 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:59:46.976 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:59:46.976 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:59:46.976 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:59:46.976 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:59:46.976 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:59:46.976 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:59:46.976 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:59:46.976 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:59:46.976 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:59:46.977 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:59:46.978 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:59:46.978 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:59:46.978 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:59:46.978 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:59:46.978 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:59:46.978 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:59:46.978 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:59:46.978 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:59:46.978 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:59:46.978 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:59:46.978 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:59:46.978 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:59:46.979 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:59:46.979 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:59:46.979 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:59:46.979 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:59:47.003 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:59:47.003 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:59:47.003 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:59:47.003 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_4 2024-04-06T09:59:47.108 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:Invalid command: unused arguments: ['mirror/image_4'] 2024-04-06T09:59:47.108 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:rbd mirror flush : flush rbd mirror 2024-04-06T09:59:47.108 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:admin_socket: invalid command 2024-04-06T09:59:47.110 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:59:47.110 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/image_4 2024-04-06T09:59:47.111 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:1 2024-04-06T09:59:47.111 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:59:47.111 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:59:47.111 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-06T09:59:47.111 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:1 2024-04-06T09:59:47.111 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:59:47.111 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:59:47.111 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:59:47.111 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=1 2024-04-06T09:59:47.111 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-06T09:59:47.111 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:59:47.111 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:59:47.111 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=1 2024-04-06T09:59:47.111 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=1 2024-04-06T09:59:47.113 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-06T09:59:47.113 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:1 2024-04-06T09:59:47.113 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:59:47.113 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:59:47.113 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-06T09:59:47.113 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:1 2024-04-06T09:59:47.113 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:59:47.113 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:59:47.113 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:59:47.113 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=1 2024-04-06T09:59:47.114 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-06T09:59:47.114 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:59:47.114 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:59:47.114 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=1 2024-04-06T09:59:47.114 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=1 2024-04-06T09:59:47.114 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-06T09:59:47.138 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T09:59:47.138 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T09:59:47.138 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T09:59:47.138 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/image_4 2024-04-06T09:59:47.280 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:59:47.281 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror image_4 2024-04-06T09:59:47.281 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:59:47.282 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:59:47.282 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_4 2024-04-06T09:59:47.282 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror image_4 '' 2024-04-06T09:59:47.282 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:59:47.282 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:59:47.282 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_4 2024-04-06T09:59:47.282 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:59:47.283 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-image_4.status 2024-04-06T09:59:47.283 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-image_4.status 2024-04-06T09:59:47.283 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:59:47.285 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-image_4.status 2024-04-06T09:59:47.285 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_4 2024-04-06T09:59:47.285 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_4.status 2024-04-06T09:59:47.337 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 2 2024-04-06T09:59:47.338 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 2 2024-04-06T09:59:47.338 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:59:47.338 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=9, tag_tid=2, entry_tid=1], [object_number=8, tag_tid=2, entry_tid=0], [object_number=11, tag_tid=1, entry_tid=11667], [object_number=10, tag_tid=1, entry_tid=11666]]], state=connected] 2024-04-06T09:59:47.338 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=8, tag_tid=1, entry_tid=11668], [object_number=11, tag_tid=1, entry_tid=11667], [object_number=10, tag_tid=1, entry_tid=11666], [object_number=9, tag_tid=1, entry_tid=11665]]], state=connected] 2024-04-06T09:59:47.342 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_4.status 2024-04-06T09:59:47.342 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=9, tag_tid=2, entry_tid=1' 2024-04-06T09:59:47.344 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror image_4 2024-04-06T09:59:47.344 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:59:47.344 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:59:47.344 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_4 2024-04-06T09:59:47.344 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror image_4 '..*' 2024-04-06T09:59:47.344 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:59:47.344 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:59:47.344 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_4 2024-04-06T09:59:47.344 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:59:47.344 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-image_4.status 2024-04-06T09:59:47.344 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-image_4.status 2024-04-06T09:59:47.344 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:59:47.345 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-image_4.status 2024-04-06T09:59:47.345 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_4 2024-04-06T09:59:47.345 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_4.status 2024-04-06T09:59:47.391 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 2 2024-04-06T09:59:47.391 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 2 2024-04-06T09:59:47.391 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:59:47.391 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=9, tag_tid=2, entry_tid=1], [object_number=8, tag_tid=2, entry_tid=0], [object_number=11, tag_tid=1, entry_tid=11667], [object_number=10, tag_tid=1, entry_tid=11666]]], state=connected] 2024-04-06T09:59:47.391 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=8, tag_tid=1, entry_tid=11668], [object_number=11, tag_tid=1, entry_tid=11667], [object_number=10, tag_tid=1, entry_tid=11666], [object_number=9, tag_tid=1, entry_tid=11665]]], state=connected] 2024-04-06T09:59:47.394 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_4.status 2024-04-06T09:59:47.396 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=8, tag_tid=1, entry_tid=11668' 2024-04-06T09:59:47.397 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=9, tag_tid=2, entry_tid=1' -a 'object_number=9, tag_tid=2, entry_tid=1' = 'object_number=8, tag_tid=1, entry_tid=11668' 2024-04-06T09:59:47.397 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=8, tag_tid=1, entry_tid=11668' '!=' 'object_number=8, tag_tid=1, entry_tid=11668' 2024-04-06T09:59:47.397 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:59:47.397 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.4 2024-04-06T09:59:47.798 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror image_4 2024-04-06T09:59:47.798 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:59:47.799 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:59:47.799 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_4 2024-04-06T09:59:47.799 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:59:47.799 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n image_4 ']' 2024-04-06T09:59:47.799 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/image_4' 2024-04-06T09:59:47.799 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_4 2024-04-06T09:59:47.799 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:59:47.799 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:59:47.799 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:59:47.799 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:59:47.799 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:59:47.799 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:59:47.799 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:59:47.800 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:59:47.800 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:59:47.802 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:59:47.802 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_4 2024-04-06T09:59:47.802 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:59:47.802 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:59:47.802 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:59:47.803 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:59:47.803 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:59:47.803 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:59:47.803 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:59:47.803 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:59:47.803 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:59:47.803 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:59:47.803 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:59:47.803 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:59:47.803 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:59:47.803 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:59:47.804 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:59:47.804 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:59:47.804 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:59:47.804 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:59:47.804 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:59:47.804 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:59:47.804 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:59:47.805 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:59:47.805 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:59:47.805 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:59:47.805 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:59:47.805 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:59:47.805 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:59:47.805 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:59:47.805 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:59:47.805 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:59:47.828 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:59:47.829 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:59:47.829 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:59:47.829 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_4 2024-04-06T09:59:47.937 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:Invalid command: unused arguments: ['mirror/image_4'] 2024-04-06T09:59:47.937 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:rbd mirror flush : flush rbd mirror 2024-04-06T09:59:47.937 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:admin_socket: invalid command 2024-04-06T09:59:47.939 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:59:47.939 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/image_4 2024-04-06T09:59:47.939 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:1 2024-04-06T09:59:47.939 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:59:47.940 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:59:47.940 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-06T09:59:47.940 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:1 2024-04-06T09:59:47.940 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:59:47.940 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:59:47.940 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:59:47.940 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=1 2024-04-06T09:59:47.940 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-06T09:59:47.940 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:59:47.940 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:59:47.940 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=1 2024-04-06T09:59:47.940 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=1 2024-04-06T09:59:47.942 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-06T09:59:47.942 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:1 2024-04-06T09:59:47.942 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:59:47.942 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:59:47.942 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-06T09:59:47.942 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:1 2024-04-06T09:59:47.943 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:59:47.943 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:59:47.943 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:59:47.943 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=1 2024-04-06T09:59:47.943 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-06T09:59:47.943 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:59:47.943 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:59:47.943 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=1 2024-04-06T09:59:47.944 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=1 2024-04-06T09:59:47.944 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-06T09:59:47.960 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T09:59:47.960 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T09:59:47.960 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T09:59:47.960 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/image_4 2024-04-06T09:59:48.104 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:59:48.105 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror image_4 2024-04-06T09:59:48.105 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:59:48.105 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:59:48.105 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_4 2024-04-06T09:59:48.105 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror image_4 '' 2024-04-06T09:59:48.106 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:59:48.106 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:59:48.106 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_4 2024-04-06T09:59:48.106 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:59:48.106 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-image_4.status 2024-04-06T09:59:48.106 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-image_4.status 2024-04-06T09:59:48.106 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:59:48.108 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-image_4.status 2024-04-06T09:59:48.109 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_4 2024-04-06T09:59:48.109 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_4.status 2024-04-06T09:59:48.158 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 2 2024-04-06T09:59:48.158 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 2 2024-04-06T09:59:48.158 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:59:48.158 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=9, tag_tid=2, entry_tid=1], [object_number=8, tag_tid=2, entry_tid=0], [object_number=11, tag_tid=1, entry_tid=11667], [object_number=10, tag_tid=1, entry_tid=11666]]], state=connected] 2024-04-06T09:59:48.158 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=8, tag_tid=1, entry_tid=11668], [object_number=11, tag_tid=1, entry_tid=11667], [object_number=10, tag_tid=1, entry_tid=11666], [object_number=9, tag_tid=1, entry_tid=11665]]], state=connected] 2024-04-06T09:59:48.162 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_4.status 2024-04-06T09:59:48.163 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=9, tag_tid=2, entry_tid=1' 2024-04-06T09:59:48.164 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror image_4 2024-04-06T09:59:48.164 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:59:48.164 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:59:48.164 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_4 2024-04-06T09:59:48.165 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror image_4 '..*' 2024-04-06T09:59:48.165 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:59:48.165 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:59:48.165 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_4 2024-04-06T09:59:48.165 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:59:48.165 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-image_4.status 2024-04-06T09:59:48.165 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-image_4.status 2024-04-06T09:59:48.165 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:59:48.165 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-image_4.status 2024-04-06T09:59:48.166 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_4 2024-04-06T09:59:48.166 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_4.status 2024-04-06T09:59:48.212 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 2 2024-04-06T09:59:48.212 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 2 2024-04-06T09:59:48.212 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:59:48.212 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=9, tag_tid=2, entry_tid=1], [object_number=8, tag_tid=2, entry_tid=0], [object_number=11, tag_tid=1, entry_tid=11667], [object_number=10, tag_tid=1, entry_tid=11666]]], state=connected] 2024-04-06T09:59:48.212 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=8, tag_tid=1, entry_tid=11668], [object_number=11, tag_tid=1, entry_tid=11667], [object_number=10, tag_tid=1, entry_tid=11666], [object_number=9, tag_tid=1, entry_tid=11665]]], state=connected] 2024-04-06T09:59:48.215 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_4.status 2024-04-06T09:59:48.217 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=8, tag_tid=1, entry_tid=11668' 2024-04-06T09:59:48.217 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=9, tag_tid=2, entry_tid=1' -a 'object_number=9, tag_tid=2, entry_tid=1' = 'object_number=8, tag_tid=1, entry_tid=11668' 2024-04-06T09:59:48.217 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=8, tag_tid=1, entry_tid=11668' '!=' 'object_number=8, tag_tid=1, entry_tid=11668' 2024-04-06T09:59:48.217 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:59:48.217 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.8 2024-04-06T09:59:49.019 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror image_4 2024-04-06T09:59:49.019 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:59:49.019 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:59:49.019 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_4 2024-04-06T09:59:49.019 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:59:49.019 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n image_4 ']' 2024-04-06T09:59:49.019 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/image_4' 2024-04-06T09:59:49.019 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_4 2024-04-06T09:59:49.019 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:59:49.020 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:59:49.020 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:59:49.020 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:59:49.020 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:59:49.020 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:59:49.020 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:59:49.021 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:59:49.021 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:59:49.023 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:59:49.023 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_4 2024-04-06T09:59:49.023 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:59:49.024 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:59:49.024 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:59:49.024 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:59:49.024 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:59:49.024 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:59:49.024 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:59:49.024 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:59:49.024 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:59:49.024 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:59:49.024 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:59:49.024 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:59:49.024 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:59:49.024 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:59:49.024 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:59:49.024 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:59:49.025 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:59:49.025 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:59:49.025 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:59:49.025 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:59:49.025 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:59:49.025 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:59:49.026 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:59:49.026 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:59:49.026 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:59:49.026 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:59:49.026 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:59:49.026 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:59:49.026 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:59:49.026 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:59:49.049 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:59:49.049 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:59:49.050 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:59:49.050 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_4 2024-04-06T09:59:49.155 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:Invalid command: unused arguments: ['mirror/image_4'] 2024-04-06T09:59:49.156 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:rbd mirror flush : flush rbd mirror 2024-04-06T09:59:49.156 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:admin_socket: invalid command 2024-04-06T09:59:49.158 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:59:49.158 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/image_4 2024-04-06T09:59:49.158 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:1 2024-04-06T09:59:49.158 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:59:49.158 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:59:49.158 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-06T09:59:49.158 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:1 2024-04-06T09:59:49.158 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:59:49.158 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:59:49.158 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:59:49.159 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=1 2024-04-06T09:59:49.159 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-06T09:59:49.159 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:59:49.159 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:59:49.159 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=1 2024-04-06T09:59:49.159 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=1 2024-04-06T09:59:49.160 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-06T09:59:49.160 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:1 2024-04-06T09:59:49.160 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:59:49.160 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:59:49.160 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-06T09:59:49.161 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:1 2024-04-06T09:59:49.161 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:59:49.161 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:59:49.161 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:59:49.161 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=1 2024-04-06T09:59:49.161 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-06T09:59:49.161 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:59:49.161 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:59:49.161 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=1 2024-04-06T09:59:49.161 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=1 2024-04-06T09:59:49.161 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-06T09:59:49.187 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T09:59:49.187 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T09:59:49.187 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T09:59:49.187 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/image_4 2024-04-06T09:59:49.326 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:59:49.327 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror image_4 2024-04-06T09:59:49.328 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:59:49.328 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:59:49.328 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_4 2024-04-06T09:59:49.328 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror image_4 '' 2024-04-06T09:59:49.328 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:59:49.328 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:59:49.328 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_4 2024-04-06T09:59:49.328 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:59:49.329 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-image_4.status 2024-04-06T09:59:49.329 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-image_4.status 2024-04-06T09:59:49.329 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:59:49.331 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-image_4.status 2024-04-06T09:59:49.332 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_4 2024-04-06T09:59:49.332 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_4.status 2024-04-06T09:59:49.384 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 2 2024-04-06T09:59:49.384 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 2 2024-04-06T09:59:49.384 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:59:49.384 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=9, tag_tid=2, entry_tid=1], [object_number=8, tag_tid=2, entry_tid=0], [object_number=11, tag_tid=1, entry_tid=11667], [object_number=10, tag_tid=1, entry_tid=11666]]], state=connected] 2024-04-06T09:59:49.384 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=8, tag_tid=1, entry_tid=11668], [object_number=11, tag_tid=1, entry_tid=11667], [object_number=10, tag_tid=1, entry_tid=11666], [object_number=9, tag_tid=1, entry_tid=11665]]], state=connected] 2024-04-06T09:59:49.391 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_4.status 2024-04-06T09:59:49.391 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=9, tag_tid=2, entry_tid=1' 2024-04-06T09:59:49.391 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror image_4 2024-04-06T09:59:49.391 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:59:49.391 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:59:49.391 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_4 2024-04-06T09:59:49.391 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror image_4 '..*' 2024-04-06T09:59:49.391 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:59:49.391 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:59:49.391 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_4 2024-04-06T09:59:49.392 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:59:49.392 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-image_4.status 2024-04-06T09:59:49.392 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-image_4.status 2024-04-06T09:59:49.392 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:59:49.392 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-image_4.status 2024-04-06T09:59:49.392 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_4 2024-04-06T09:59:49.393 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_4.status 2024-04-06T09:59:49.437 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 2 2024-04-06T09:59:49.437 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 2 2024-04-06T09:59:49.437 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:59:49.437 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=9, tag_tid=2, entry_tid=1], [object_number=8, tag_tid=2, entry_tid=0], [object_number=11, tag_tid=1, entry_tid=11667], [object_number=10, tag_tid=1, entry_tid=11666]]], state=connected] 2024-04-06T09:59:49.437 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=8, tag_tid=1, entry_tid=11668], [object_number=11, tag_tid=1, entry_tid=11667], [object_number=10, tag_tid=1, entry_tid=11666], [object_number=9, tag_tid=1, entry_tid=11665]]], state=connected] 2024-04-06T09:59:49.440 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_4.status 2024-04-06T09:59:49.442 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=8, tag_tid=1, entry_tid=11668' 2024-04-06T09:59:49.442 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=9, tag_tid=2, entry_tid=1' -a 'object_number=9, tag_tid=2, entry_tid=1' = 'object_number=8, tag_tid=1, entry_tid=11668' 2024-04-06T09:59:49.443 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=8, tag_tid=1, entry_tid=11668' '!=' 'object_number=8, tag_tid=1, entry_tid=11668' 2024-04-06T09:59:49.443 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:59:49.443 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 1.6 2024-04-06T09:59:51.045 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror image_4 2024-04-06T09:59:51.045 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:59:51.045 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:59:51.045 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_4 2024-04-06T09:59:51.045 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:59:51.045 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n image_4 ']' 2024-04-06T09:59:51.045 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/image_4' 2024-04-06T09:59:51.045 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_4 2024-04-06T09:59:51.045 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:59:51.045 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:59:51.045 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:59:51.045 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:59:51.046 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:59:51.046 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:59:51.046 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:59:51.046 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:59:51.046 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:59:51.049 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:59:51.049 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_4 2024-04-06T09:59:51.049 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:59:51.049 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:59:51.049 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:59:51.049 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:59:51.049 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:59:51.049 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:59:51.049 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:59:51.049 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:59:51.049 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:59:51.050 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:59:51.050 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:59:51.052 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:59:51.052 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:59:51.052 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:59:51.052 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:59:51.052 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:59:51.052 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:59:51.052 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:59:51.052 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:59:51.052 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:59:51.052 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:59:51.052 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:59:51.052 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:59:51.053 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:59:51.053 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:59:51.053 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:59:51.053 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:59:51.053 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:59:51.053 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:59:51.053 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:59:51.076 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:59:51.076 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:59:51.077 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:59:51.077 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_4 2024-04-06T09:59:51.179 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:Invalid command: unused arguments: ['mirror/image_4'] 2024-04-06T09:59:51.179 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:rbd mirror flush : flush rbd mirror 2024-04-06T09:59:51.179 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:admin_socket: invalid command 2024-04-06T09:59:51.183 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:59:51.183 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/image_4 2024-04-06T09:59:51.183 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:1 2024-04-06T09:59:51.183 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:59:51.183 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:59:51.183 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-06T09:59:51.184 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:1 2024-04-06T09:59:51.184 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:59:51.184 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:59:51.184 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:59:51.184 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=1 2024-04-06T09:59:51.184 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-06T09:59:51.184 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:59:51.184 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:59:51.184 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=1 2024-04-06T09:59:51.184 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=1 2024-04-06T09:59:51.184 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-06T09:59:51.184 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:1 2024-04-06T09:59:51.184 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:59:51.184 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:59:51.185 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-06T09:59:51.185 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:1 2024-04-06T09:59:51.185 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:59:51.185 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:59:51.185 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:59:51.185 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=1 2024-04-06T09:59:51.185 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-06T09:59:51.185 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:59:51.185 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:59:51.186 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=1 2024-04-06T09:59:51.186 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=1 2024-04-06T09:59:51.186 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-06T09:59:51.207 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T09:59:51.207 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T09:59:51.207 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T09:59:51.208 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/image_4 2024-04-06T09:59:51.367 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:59:51.368 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror image_4 2024-04-06T09:59:51.368 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:59:51.368 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:59:51.368 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_4 2024-04-06T09:59:51.368 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror image_4 '' 2024-04-06T09:59:51.368 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:59:51.368 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:59:51.368 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_4 2024-04-06T09:59:51.368 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:59:51.369 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-image_4.status 2024-04-06T09:59:51.369 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:59:51.369 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-image_4.status 2024-04-06T09:59:51.370 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-image_4.status 2024-04-06T09:59:51.370 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_4 2024-04-06T09:59:51.370 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_4.status 2024-04-06T09:59:51.427 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 2 2024-04-06T09:59:51.427 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 2 2024-04-06T09:59:51.427 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:59:51.428 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=9, tag_tid=2, entry_tid=1], [object_number=8, tag_tid=2, entry_tid=0], [object_number=11, tag_tid=1, entry_tid=11667], [object_number=10, tag_tid=1, entry_tid=11666]]], state=connected] 2024-04-06T09:59:51.428 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=9, tag_tid=2, entry_tid=1], [object_number=8, tag_tid=2, entry_tid=0], [object_number=11, tag_tid=1, entry_tid=11667], [object_number=10, tag_tid=1, entry_tid=11666]]], state=connected] 2024-04-06T09:59:51.431 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_4.status 2024-04-06T09:59:51.432 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=9, tag_tid=2, entry_tid=1' 2024-04-06T09:59:51.434 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror image_4 2024-04-06T09:59:51.434 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:59:51.435 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:59:51.435 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_4 2024-04-06T09:59:51.435 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror image_4 '..*' 2024-04-06T09:59:51.435 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:59:51.435 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:59:51.435 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_4 2024-04-06T09:59:51.436 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:59:51.436 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-image_4.status 2024-04-06T09:59:51.436 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-image_4.status 2024-04-06T09:59:51.436 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:59:51.437 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-image_4.status 2024-04-06T09:59:51.437 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_4 2024-04-06T09:59:51.437 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_4.status 2024-04-06T09:59:51.480 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 2 2024-04-06T09:59:51.481 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 2 2024-04-06T09:59:51.481 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:59:51.481 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=9, tag_tid=2, entry_tid=1], [object_number=8, tag_tid=2, entry_tid=0], [object_number=11, tag_tid=1, entry_tid=11667], [object_number=10, tag_tid=1, entry_tid=11666]]], state=connected] 2024-04-06T09:59:51.481 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=9, tag_tid=2, entry_tid=1], [object_number=8, tag_tid=2, entry_tid=0], [object_number=11, tag_tid=1, entry_tid=11667], [object_number=10, tag_tid=1, entry_tid=11666]]], state=connected] 2024-04-06T09:59:51.484 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_4.status 2024-04-06T09:59:51.486 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=9, tag_tid=2, entry_tid=1' 2024-04-06T09:59:51.486 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=9, tag_tid=2, entry_tid=1' -a 'object_number=9, tag_tid=2, entry_tid=1' = 'object_number=9, tag_tid=2, entry_tid=1' 2024-04-06T09:59:51.486 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:59:51.486 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_snap_present cluster1 mirror image_4 snap 2024-04-06T09:59:51.486 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:59:51.486 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:59:51.486 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_4 2024-04-06T09:59:51.486 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local snap_name=snap 2024-04-06T09:59:51.486 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local s 2024-04-06T09:59:51.486 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-04-06T09:59:51.487 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 1 2024-04-06T09:59:52.487 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 info mirror/image_4@snap 2024-04-06T09:59:52.554 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:rbd image 'image_4': 2024-04-06T09:59:52.554 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: size 128 MiB in 32 objects 2024-04-06T09:59:52.554 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: order 22 (4 MiB objects) 2024-04-06T09:59:52.554 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: snapshot_count: 1 2024-04-06T09:59:52.554 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: id: 10acd7922c7a 2024-04-06T09:59:52.554 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: block_name_prefix: rbd_data.10acd7922c7a 2024-04-06T09:59:52.554 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: format: 2 2024-04-06T09:59:52.554 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: features: layering, exclusive-lock, journaling 2024-04-06T09:59:52.554 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: op_features: 2024-04-06T09:59:52.555 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: flags: 2024-04-06T09:59:52.555 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: create_timestamp: Sat Apr 6 09:51:10 2024 2024-04-06T09:59:52.555 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: access_timestamp: Sat Apr 6 09:51:10 2024 2024-04-06T09:59:52.555 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: modify_timestamp: Sat Apr 6 09:52:11 2024 2024-04-06T09:59:52.555 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: protected: False 2024-04-06T09:59:52.555 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: journal: 10acd7922c7a 2024-04-06T09:59:52.555 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: mirroring state: enabled 2024-04-06T09:59:52.555 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: mirroring mode: journal 2024-04-06T09:59:52.555 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: mirroring global id: 231f381a-e9f8-49fa-9b2a-78e9ffc448d9 2024-04-06T09:59:52.555 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: mirroring primary: false 2024-04-06T09:59:52.559 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:59:52.559 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ compare_image_snaps mirror image_4 snap 2024-04-06T09:59:52.559 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:59:52.559 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_4 2024-04-06T09:59:52.559 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local snap_name=snap 2024-04-06T09:59:52.559 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local ret=0 2024-04-06T09:59:52.560 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local rmt_export=/tmp/tmp.BhW6xKmROU/cluster2-mirror-image_4.export 2024-04-06T09:59:52.560 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loc_export=/tmp/tmp.BhW6xKmROU/cluster1-mirror-image_4.export 2024-04-06T09:59:52.560 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rm -f /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_4.export /tmp/tmp.BhW6xKmROU/cluster1-mirror-image_4.export 2024-04-06T09:59:52.560 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster2 -p mirror export image_4@snap /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_4.export 2024-04-06T09:59:52.738 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-04-06T09:59:52.747 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 -p mirror export image_4@snap /tmp/tmp.BhW6xKmROU/cluster1-mirror-image_4.export 2024-04-06T09:59:53.155 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-04-06T09:59:53.170 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmp /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_4.export /tmp/tmp.BhW6xKmROU/cluster1-mirror-image_4.export 2024-04-06T09:59:53.251 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rm -f /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_4.export /tmp/tmp.BhW6xKmROU/cluster1-mirror-image_4.export 2024-04-06T09:59:53.295 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:59:53.295 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-06T09:59:53.295 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ image=image_5 2024-04-06T09:59:53.295 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ create_snap cluster2 mirror image_5 snap 2024-04-06T09:59:53.295 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-04-06T09:59:53.295 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:59:53.295 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_5 2024-04-06T09:59:53.295 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local snap_name=snap 2024-04-06T09:59:53.295 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster2 -p mirror snap create image_5@snap --debug-rbd=20 --debug-journaler=20 2024-04-06T09:59:54.393 INFO:tasks.ceph.cluster2.osd.0.smithi192.stderr:2024-04-06T09:59:54.395+0000 7f78af6df640 -1 reset not still connected to 0x560422aa4ea0 2024-04-06T09:59:54.393 INFO:tasks.ceph.cluster2.osd.0.smithi192.stderr:2024-04-06T09:59:54.395+0000 7f78af6df640 -1 reset not still connected to 0x560422aa52b0 2024-04-06T09:59:54.393 INFO:tasks.ceph.cluster2.osd.0.smithi192.stderr:2024-04-06T09:59:54.395+0000 7f78af6df640 -1 reset not still connected to 0x5604239371e0 2024-04-06T09:59:54.394 INFO:tasks.ceph.cluster2.osd.0.smithi192.stderr:2024-04-06T09:59:54.395+0000 7f78af6df640 -1 reset not still connected to 0x560423937520 2024-04-06T09:59:54.394 INFO:tasks.ceph.cluster2.osd.0.smithi192.stderr:2024-04-06T09:59:54.395+0000 7f78af6df640 -1 reset not still connected to 0x5604242c76c0 2024-04-06T09:59:54.394 INFO:tasks.ceph.cluster2.osd.0.smithi192.stderr:2024-04-06T09:59:54.395+0000 7f78af6df640 -1 reset not still connected to 0x5604247b84e0 2024-04-06T09:59:54.394 INFO:tasks.ceph.cluster2.osd.0.smithi192.stderr:2024-04-06T09:59:54.395+0000 7f78af6df640 -1 reset not still connected to 0x56042bd3fc70 2024-04-06T09:59:54.394 INFO:tasks.ceph.cluster2.osd.0.smithi192.stderr:2024-04-06T09:59:54.395+0000 7f78af6df640 -1 reset not still connected to 0x560436199450 2024-04-06T09:59:54.398 INFO:tasks.ceph.cluster2.osd.0.smithi192.stderr:2024-04-06T09:59:54.399+0000 7f78af6df640 -1 reset not still connected to 0x560436e7b040 2024-04-06T09:59:54.398 INFO:tasks.ceph.cluster2.osd.0.smithi192.stderr:2024-04-06T09:59:54.399+0000 7f78af6df640 -1 reset not still connected to 0x56043d1216c0 2024-04-06T09:59:54.398 INFO:tasks.ceph.cluster2.osd.0.smithi192.stderr:2024-04-06T09:59:54.399+0000 7f78af6df640 -1 reset not still connected to 0x5604442b6000 2024-04-06T09:59:54.398 INFO:tasks.ceph.cluster2.osd.0.smithi192.stderr:2024-04-06T09:59:54.399+0000 7f78af6df640 -1 reset not still connected to 0x560444f6c820 2024-04-06T09:59:54.399 INFO:tasks.ceph.cluster2.osd.0.smithi192.stderr:2024-04-06T09:59:54.399+0000 7f78af6df640 -1 reset not still connected to 0x560448e60680 2024-04-06T09:59:54.399 INFO:tasks.ceph.cluster2.osd.0.smithi192.stderr:2024-04-06T09:59:54.399+0000 7f78af6df640 -1 reset not still connected to 0x560449684410 2024-04-06T09:59:54.399 INFO:tasks.ceph.cluster2.osd.0.smithi192.stderr:2024-04-06T09:59:54.399+0000 7f78af6df640 -1 reset not still connected to 0x56044cc1e1a0 2024-04-06T09:59:54.399 INFO:tasks.ceph.cluster2.osd.0.smithi192.stderr:2024-04-06T09:59:54.399+0000 7f78af6df640 -1 reset not still connected to 0x560456d576c0 2024-04-06T09:59:55.612 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_image_replay_started cluster1 mirror image_5 2024-04-06T09:59:55.613 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:59:55.613 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:59:55.613 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_5 2024-04-06T09:59:55.613 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_image_replay_state cluster1 mirror image_5 started 2024-04-06T09:59:55.613 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:59:55.613 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:59:55.613 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_5 2024-04-06T09:59:55.613 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state=started 2024-04-06T09:59:55.613 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local s 2024-04-06T09:59:55.613 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:59:55.613 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 1 2024-04-06T09:59:56.614 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test_image_replay_state cluster1 mirror image_5 started 2024-04-06T09:59:56.614 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:59:56.614 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:59:56.614 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_5 2024-04-06T09:59:56.614 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local test_state=started 2024-04-06T09:59:56.615 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local status_result 2024-04-06T09:59:56.615 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local current_state=stopped 2024-04-06T09:59:56.615 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_5 2024-04-06T09:59:56.616 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -i state 2024-04-06T09:59:56.616 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_instance=cluster1 2024-04-06T09:59:56.616 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ shift 2024-04-06T09:59:56.616 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:59:56.616 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance=cluster1 2024-04-06T09:59:56.616 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local loop_instance 2024-04-06T09:59:56.616 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:59:56.616 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sleep 0 2024-04-06T09:59:56.617 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:59:56.617 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ seq 0 3 2024-04-06T09:59:56.619 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:59:56.619 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_5 2024-04-06T09:59:56.619 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1:0 2024-04-06T09:59:56.619 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ shift 2024-04-06T09:59:56.619 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:59:56.619 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:59:56.619 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:59:56.620 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=cluster 2024-04-06T09:59:56.620 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:59:56.620 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:59:56.620 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:59:56.620 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:59:56.620 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval cluster=cluster1 2024-04-06T09:59:56.620 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ cluster=cluster1 2024-04-06T09:59:56.620 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:59:56.620 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:59:56.622 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:59:56.622 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local local_cluster=cluster1:0 2024-04-06T09:59:56.622 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local cluster=cluster1 2024-04-06T09:59:56.622 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local instance 2024-04-06T09:59:56.622 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:59:56.622 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local val=cluster1:0 2024-04-06T09:59:56.622 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local cluster_var_name=local_cluster 2024-04-06T09:59:56.622 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local instance_var_name=instance 2024-04-06T09:59:56.623 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ cluster=cluster1 2024-04-06T09:59:56.623 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:59:56.623 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ '[' 0 = cluster1:0 ']' 2024-04-06T09:59:56.623 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ eval local_cluster=cluster1 2024-04-06T09:59:56.623 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++++ local_cluster=cluster1 2024-04-06T09:59:56.623 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ eval instance=0 2024-04-06T09:59:56.623 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++++ instance=0 2024-04-06T09:59:56.623 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:59:56.647 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:59:56.647 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:59:56.647 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:59:56.647 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_5 2024-04-06T09:59:56.751 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:Invalid command: unused arguments: ['mirror/image_5'] 2024-04-06T09:59:56.751 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:rbd mirror status : get status for rbd mirror 2024-04-06T09:59:56.751 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:admin_socket: invalid command 2024-04-06T09:59:56.753 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:59:56.753 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_5 2024-04-06T09:59:56.754 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1:1 2024-04-06T09:59:56.754 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ shift 2024-04-06T09:59:56.754 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:59:56.754 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-04-06T09:59:56.754 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:1 2024-04-06T09:59:56.754 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=cluster 2024-04-06T09:59:56.754 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:59:56.754 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:59:56.754 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=1 2024-04-06T09:59:56.755 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-06T09:59:56.755 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval cluster=cluster1 2024-04-06T09:59:56.755 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ cluster=cluster1 2024-04-06T09:59:56.755 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=1 2024-04-06T09:59:56.755 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=1 2024-04-06T09:59:56.756 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-04-06T09:59:56.756 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local local_cluster=cluster1:1 2024-04-06T09:59:56.756 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local cluster=cluster1 2024-04-06T09:59:56.756 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local instance 2024-04-06T09:59:56.756 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-06T09:59:56.756 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local val=cluster1:1 2024-04-06T09:59:56.756 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local cluster_var_name=local_cluster 2024-04-06T09:59:56.756 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local instance_var_name=instance 2024-04-06T09:59:56.756 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ cluster=cluster1 2024-04-06T09:59:56.757 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=1 2024-04-06T09:59:56.757 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ '[' 1 = cluster1:1 ']' 2024-04-06T09:59:56.757 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ eval local_cluster=cluster1 2024-04-06T09:59:56.757 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++++ local_cluster=cluster1 2024-04-06T09:59:56.757 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ eval instance=1 2024-04-06T09:59:56.757 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++++ instance=1 2024-04-06T09:59:56.757 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-06T09:59:56.779 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T09:59:56.780 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T09:59:56.780 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T09:59:56.780 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_5 2024-04-06T09:59:56.943 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ return 0 2024-04-06T09:59:56.944 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ status_result=' "state": "Replaying"' 2024-04-06T09:59:56.944 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ echo ' "state": "Replaying"' 2024-04-06T09:59:56.944 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep -i Replaying 2024-04-06T09:59:56.946 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: "state": "Replaying" 2024-04-06T09:59:56.948 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ current_state=started 2024-04-06T09:59:56.948 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test started = started 2024-04-06T09:59:56.948 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:59:56.948 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_5 2024-04-06T09:59:56.948 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local local_cluster=cluster1 2024-04-06T09:59:56.948 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-04-06T09:59:56.948 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:59:56.948 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_5 2024-04-06T09:59:56.948 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' journal = journal ']' 2024-04-06T09:59:56.948 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_journal_replay_complete cluster1 cluster2 mirror image_5 2024-04-06T09:59:56.948 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local local_cluster=cluster1 2024-04-06T09:59:56.948 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-04-06T09:59:56.948 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:59:56.949 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_5 2024-04-06T09:59:56.949 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local s master_pos mirror_pos last_mirror_pos 2024-04-06T09:59:56.949 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local master_tag master_entry mirror_tag mirror_entry 2024-04-06T09:59:56.949 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:59:56.949 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:59:56.949 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:59:57.149 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror image_5 2024-04-06T09:59:57.150 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:59:57.150 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:59:57.150 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_5 2024-04-06T09:59:57.150 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:59:57.150 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n image_5 ']' 2024-04-06T09:59:57.150 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/image_5' 2024-04-06T09:59:57.150 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_5 2024-04-06T09:59:57.150 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:59:57.150 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:59:57.150 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:59:57.150 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:59:57.151 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:59:57.151 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:59:57.151 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:59:57.151 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:59:57.152 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:59:57.153 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:59:57.154 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_5 2024-04-06T09:59:57.154 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:59:57.154 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:59:57.154 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:59:57.154 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:59:57.154 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:59:57.154 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:59:57.154 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:59:57.154 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:59:57.154 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:59:57.154 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:59:57.154 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:59:57.155 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:59:57.155 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:59:57.155 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:59:57.155 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:59:57.155 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:59:57.156 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:59:57.156 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:59:57.156 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:59:57.156 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:59:57.156 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:59:57.156 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:59:57.156 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:59:57.156 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:59:57.156 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:59:57.156 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:59:57.156 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:59:57.156 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:59:57.156 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:59:57.156 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:59:57.181 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:59:57.181 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:59:57.181 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:59:57.181 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_5 2024-04-06T09:59:57.282 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:Invalid command: unused arguments: ['mirror/image_5'] 2024-04-06T09:59:57.282 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:rbd mirror flush : flush rbd mirror 2024-04-06T09:59:57.282 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:admin_socket: invalid command 2024-04-06T09:59:57.285 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:59:57.285 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/image_5 2024-04-06T09:59:57.285 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:1 2024-04-06T09:59:57.285 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:59:57.285 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:59:57.285 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-06T09:59:57.285 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:1 2024-04-06T09:59:57.285 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:59:57.285 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:59:57.285 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:59:57.286 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=1 2024-04-06T09:59:57.286 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-06T09:59:57.286 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:59:57.286 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:59:57.286 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=1 2024-04-06T09:59:57.286 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=1 2024-04-06T09:59:57.286 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-06T09:59:57.286 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:1 2024-04-06T09:59:57.286 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:59:57.286 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:59:57.287 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-06T09:59:57.287 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:1 2024-04-06T09:59:57.287 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:59:57.287 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:59:57.287 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:59:57.287 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=1 2024-04-06T09:59:57.287 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-06T09:59:57.287 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:59:57.287 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:59:57.287 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=1 2024-04-06T09:59:57.287 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=1 2024-04-06T09:59:57.288 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-06T09:59:57.310 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T09:59:57.310 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T09:59:57.310 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T09:59:57.311 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/image_5 2024-04-06T09:59:57.449 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:59:57.450 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror image_5 2024-04-06T09:59:57.450 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:59:57.450 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:59:57.450 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_5 2024-04-06T09:59:57.450 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror image_5 '' 2024-04-06T09:59:57.450 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:59:57.450 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:59:57.450 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_5 2024-04-06T09:59:57.450 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:59:57.450 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-image_5.status 2024-04-06T09:59:57.451 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-image_5.status 2024-04-06T09:59:57.451 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:59:57.453 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-image_5.status 2024-04-06T09:59:57.454 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_5 2024-04-06T09:59:57.454 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_5.status 2024-04-06T09:59:57.506 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 2 2024-04-06T09:59:57.506 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 2 2024-04-06T09:59:57.506 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:59:57.506 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=9, tag_tid=2, entry_tid=1], [object_number=8, tag_tid=2, entry_tid=0], [object_number=11, tag_tid=1, entry_tid=12183], [object_number=10, tag_tid=1, entry_tid=12182]]], state=connected] 2024-04-06T09:59:57.506 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=8, tag_tid=1, entry_tid=12184], [object_number=11, tag_tid=1, entry_tid=12183], [object_number=10, tag_tid=1, entry_tid=12182], [object_number=9, tag_tid=1, entry_tid=12181]]], state=connected] 2024-04-06T09:59:57.510 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_5.status 2024-04-06T09:59:57.511 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=9, tag_tid=2, entry_tid=1' 2024-04-06T09:59:57.512 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror image_5 2024-04-06T09:59:57.513 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:59:57.513 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:59:57.513 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_5 2024-04-06T09:59:57.513 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror image_5 '..*' 2024-04-06T09:59:57.513 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:59:57.513 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:59:57.513 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_5 2024-04-06T09:59:57.513 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:59:57.513 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-image_5.status 2024-04-06T09:59:57.513 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-image_5.status 2024-04-06T09:59:57.513 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:59:57.514 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-image_5.status 2024-04-06T09:59:57.514 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_5 2024-04-06T09:59:57.514 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_5.status 2024-04-06T09:59:57.768 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 2 2024-04-06T09:59:57.769 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 2 2024-04-06T09:59:57.769 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:59:57.769 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=9, tag_tid=2, entry_tid=1], [object_number=8, tag_tid=2, entry_tid=0], [object_number=11, tag_tid=1, entry_tid=12183], [object_number=10, tag_tid=1, entry_tid=12182]]], state=connected] 2024-04-06T09:59:57.769 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=8, tag_tid=1, entry_tid=12184], [object_number=11, tag_tid=1, entry_tid=12183], [object_number=10, tag_tid=1, entry_tid=12182], [object_number=9, tag_tid=1, entry_tid=12181]]], state=connected] 2024-04-06T09:59:57.774 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_5.status 2024-04-06T09:59:57.776 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=8, tag_tid=1, entry_tid=12184' 2024-04-06T09:59:57.776 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=9, tag_tid=2, entry_tid=1' -a 'object_number=9, tag_tid=2, entry_tid=1' = 'object_number=8, tag_tid=1, entry_tid=12184' 2024-04-06T09:59:57.776 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=8, tag_tid=1, entry_tid=12184' '!=' '' 2024-04-06T09:59:57.776 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T09:59:57.776 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=8, tag_tid=1, entry_tid=12184' = '' 2024-04-06T09:59:57.776 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=8, tag_tid=1, entry_tid=12184' 2024-04-06T09:59:57.777 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=9, tag_tid=2, entry_tid=1' 2024-04-06T09:59:57.777 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:59:57.777 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:59:57.777 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=2 2024-04-06T09:59:57.778 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=8, tag_tid=1, entry_tid=12184' 2024-04-06T09:59:57.778 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T09:59:57.778 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:59:57.779 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=1 2024-04-06T09:59:57.780 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=9, tag_tid=2, entry_tid=1' 2024-04-06T09:59:57.782 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:59:57.782 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:59:57.783 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T09:59:57.784 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=8, tag_tid=1, entry_tid=12184' 2024-04-06T09:59:57.784 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T09:59:57.784 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T09:59:57.785 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=12184 2024-04-06T09:59:57.785 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 2 = 1 -a 1 -le 12184 2024-04-06T09:59:57.785 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T09:59:57.786 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:59:57.786 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T09:59:57.987 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror image_5 2024-04-06T09:59:57.987 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:59:57.987 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:59:57.988 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_5 2024-04-06T09:59:57.988 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:59:57.988 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n image_5 ']' 2024-04-06T09:59:57.988 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/image_5' 2024-04-06T09:59:57.988 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_5 2024-04-06T09:59:57.988 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:59:57.988 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:59:57.988 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:59:57.988 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:59:57.988 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:59:57.988 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:59:57.988 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:59:57.989 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:59:57.989 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:59:57.992 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:59:57.992 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_5 2024-04-06T09:59:57.992 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:59:57.993 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:59:57.993 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:59:57.993 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:59:57.993 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:59:57.993 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:59:57.993 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:59:57.993 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:59:57.993 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:59:57.993 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:59:57.993 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:59:57.993 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:59:57.993 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:59:57.993 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:59:57.993 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:59:57.994 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:59:57.994 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:59:57.994 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:59:57.994 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:59:57.994 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:59:57.994 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:59:57.994 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:59:57.994 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:59:57.994 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:59:57.994 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:59:57.994 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:59:57.994 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:59:57.994 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:59:57.994 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:59:57.995 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:59:58.017 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:59:58.017 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:59:58.017 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:59:58.017 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_5 2024-04-06T09:59:58.125 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:Invalid command: unused arguments: ['mirror/image_5'] 2024-04-06T09:59:58.125 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:rbd mirror flush : flush rbd mirror 2024-04-06T09:59:58.125 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:admin_socket: invalid command 2024-04-06T09:59:58.127 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:59:58.127 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/image_5 2024-04-06T09:59:58.127 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:1 2024-04-06T09:59:58.127 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:59:58.127 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:59:58.128 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-06T09:59:58.128 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:1 2024-04-06T09:59:58.128 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:59:58.128 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:59:58.128 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:59:58.128 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=1 2024-04-06T09:59:58.128 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-06T09:59:58.128 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:59:58.128 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:59:58.128 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=1 2024-04-06T09:59:58.128 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=1 2024-04-06T09:59:58.129 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-06T09:59:58.129 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:1 2024-04-06T09:59:58.130 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:59:58.130 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:59:58.130 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-06T09:59:58.130 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:1 2024-04-06T09:59:58.130 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:59:58.130 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:59:58.130 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:59:58.130 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=1 2024-04-06T09:59:58.130 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-06T09:59:58.130 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:59:58.130 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:59:58.130 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=1 2024-04-06T09:59:58.131 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=1 2024-04-06T09:59:58.131 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-06T09:59:58.154 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T09:59:58.154 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T09:59:58.154 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T09:59:58.155 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/image_5 2024-04-06T09:59:58.306 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:59:58.308 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror image_5 2024-04-06T09:59:58.308 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:59:58.308 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:59:58.308 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_5 2024-04-06T09:59:58.308 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror image_5 '' 2024-04-06T09:59:58.308 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:59:58.308 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:59:58.308 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_5 2024-04-06T09:59:58.308 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:59:58.308 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-image_5.status 2024-04-06T09:59:58.309 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:59:58.309 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-image_5.status 2024-04-06T09:59:58.309 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-image_5.status 2024-04-06T09:59:58.310 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_5 2024-04-06T09:59:58.310 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_5.status 2024-04-06T09:59:58.355 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 2 2024-04-06T09:59:58.355 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 2 2024-04-06T09:59:58.356 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:59:58.356 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=9, tag_tid=2, entry_tid=1], [object_number=8, tag_tid=2, entry_tid=0], [object_number=11, tag_tid=1, entry_tid=12183], [object_number=10, tag_tid=1, entry_tid=12182]]], state=connected] 2024-04-06T09:59:58.356 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=8, tag_tid=1, entry_tid=12184], [object_number=11, tag_tid=1, entry_tid=12183], [object_number=10, tag_tid=1, entry_tid=12182], [object_number=9, tag_tid=1, entry_tid=12181]]], state=connected] 2024-04-06T09:59:58.359 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_5.status 2024-04-06T09:59:58.360 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=9, tag_tid=2, entry_tid=1' 2024-04-06T09:59:58.362 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror image_5 2024-04-06T09:59:58.362 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:59:58.362 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:59:58.362 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_5 2024-04-06T09:59:58.362 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror image_5 '..*' 2024-04-06T09:59:58.362 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:59:58.362 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:59:58.362 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_5 2024-04-06T09:59:58.363 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:59:58.363 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-image_5.status 2024-04-06T09:59:58.363 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-image_5.status 2024-04-06T09:59:58.363 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:59:58.364 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-image_5.status 2024-04-06T09:59:58.364 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_5 2024-04-06T09:59:58.364 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_5.status 2024-04-06T09:59:58.408 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 2 2024-04-06T09:59:58.408 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 2 2024-04-06T09:59:58.408 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:59:58.408 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=9, tag_tid=2, entry_tid=1], [object_number=8, tag_tid=2, entry_tid=0], [object_number=11, tag_tid=1, entry_tid=12183], [object_number=10, tag_tid=1, entry_tid=12182]]], state=connected] 2024-04-06T09:59:58.408 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=8, tag_tid=1, entry_tid=12184], [object_number=11, tag_tid=1, entry_tid=12183], [object_number=10, tag_tid=1, entry_tid=12182], [object_number=9, tag_tid=1, entry_tid=12181]]], state=connected] 2024-04-06T09:59:58.411 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_5.status 2024-04-06T09:59:58.413 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=8, tag_tid=1, entry_tid=12184' 2024-04-06T09:59:58.413 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=9, tag_tid=2, entry_tid=1' -a 'object_number=9, tag_tid=2, entry_tid=1' = 'object_number=8, tag_tid=1, entry_tid=12184' 2024-04-06T09:59:58.413 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=8, tag_tid=1, entry_tid=12184' '!=' 'object_number=8, tag_tid=1, entry_tid=12184' 2024-04-06T09:59:58.413 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:59:58.413 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.4 2024-04-06T09:59:58.815 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror image_5 2024-04-06T09:59:58.815 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:59:58.815 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T09:59:58.815 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_5 2024-04-06T09:59:58.815 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T09:59:58.816 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n image_5 ']' 2024-04-06T09:59:58.816 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/image_5' 2024-04-06T09:59:58.816 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_5 2024-04-06T09:59:58.816 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T09:59:58.816 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:59:58.816 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T09:59:58.816 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T09:59:58.816 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T09:59:58.816 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T09:59:58.816 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T09:59:58.817 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T09:59:58.817 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T09:59:58.820 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:59:58.820 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_5 2024-04-06T09:59:58.820 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T09:59:58.820 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:59:58.820 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:59:58.820 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T09:59:58.820 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T09:59:58.820 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:59:58.820 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:59:58.820 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:59:58.820 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T09:59:58.821 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T09:59:58.821 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:59:58.821 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:59:58.821 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T09:59:58.821 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:59:58.821 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T09:59:58.821 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T09:59:58.821 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:59:58.821 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:59:58.821 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T09:59:58.821 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T09:59:58.821 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:59:58.821 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:59:58.821 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:59:58.822 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T09:59:58.822 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T09:59:58.822 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:59:58.822 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:59:58.822 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T09:59:58.822 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T09:59:58.822 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T09:59:58.845 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:59:58.845 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:59:58.845 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T09:59:58.845 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_5 2024-04-06T09:59:58.952 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:Invalid command: unused arguments: ['mirror/image_5'] 2024-04-06T09:59:58.952 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:rbd mirror flush : flush rbd mirror 2024-04-06T09:59:58.952 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:admin_socket: invalid command 2024-04-06T09:59:58.954 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T09:59:58.955 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/image_5 2024-04-06T09:59:58.955 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:1 2024-04-06T09:59:58.955 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T09:59:58.955 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T09:59:58.955 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-06T09:59:58.955 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:1 2024-04-06T09:59:58.955 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T09:59:58.955 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T09:59:58.955 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T09:59:58.955 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=1 2024-04-06T09:59:58.955 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-06T09:59:58.955 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T09:59:58.955 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:59:58.955 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=1 2024-04-06T09:59:58.956 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=1 2024-04-06T09:59:58.957 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-06T09:59:58.957 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:1 2024-04-06T09:59:58.957 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T09:59:58.957 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T09:59:58.957 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-06T09:59:58.957 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:1 2024-04-06T09:59:58.957 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T09:59:58.957 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T09:59:58.958 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T09:59:58.958 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=1 2024-04-06T09:59:58.958 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-06T09:59:58.958 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T09:59:58.958 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T09:59:58.958 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=1 2024-04-06T09:59:58.958 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=1 2024-04-06T09:59:58.958 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-06T09:59:58.979 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T09:59:58.979 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T09:59:58.979 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T09:59:58.980 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/image_5 2024-04-06T09:59:59.132 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T09:59:59.133 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror image_5 2024-04-06T09:59:59.133 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:59:59.134 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:59:59.134 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_5 2024-04-06T09:59:59.134 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror image_5 '' 2024-04-06T09:59:59.134 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:59:59.134 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:59:59.134 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_5 2024-04-06T09:59:59.134 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T09:59:59.135 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-image_5.status 2024-04-06T09:59:59.135 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-image_5.status 2024-04-06T09:59:59.135 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:59:59.137 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-image_5.status 2024-04-06T09:59:59.137 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_5 2024-04-06T09:59:59.138 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_5.status 2024-04-06T09:59:59.190 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 2 2024-04-06T09:59:59.190 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 2 2024-04-06T09:59:59.190 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:59:59.190 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=9, tag_tid=2, entry_tid=1], [object_number=8, tag_tid=2, entry_tid=0], [object_number=11, tag_tid=1, entry_tid=12183], [object_number=10, tag_tid=1, entry_tid=12182]]], state=connected] 2024-04-06T09:59:59.191 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=8, tag_tid=1, entry_tid=12184], [object_number=11, tag_tid=1, entry_tid=12183], [object_number=10, tag_tid=1, entry_tid=12182], [object_number=9, tag_tid=1, entry_tid=12181]]], state=connected] 2024-04-06T09:59:59.194 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_5.status 2024-04-06T09:59:59.196 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=9, tag_tid=2, entry_tid=1' 2024-04-06T09:59:59.197 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror image_5 2024-04-06T09:59:59.197 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:59:59.197 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:59:59.197 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_5 2024-04-06T09:59:59.197 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror image_5 '..*' 2024-04-06T09:59:59.197 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T09:59:59.197 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T09:59:59.197 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_5 2024-04-06T09:59:59.197 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T09:59:59.198 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-image_5.status 2024-04-06T09:59:59.198 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-image_5.status 2024-04-06T09:59:59.198 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T09:59:59.198 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-image_5.status 2024-04-06T09:59:59.198 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_5 2024-04-06T09:59:59.198 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_5.status 2024-04-06T09:59:59.244 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 2 2024-04-06T09:59:59.244 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 2 2024-04-06T09:59:59.244 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T09:59:59.244 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=9, tag_tid=2, entry_tid=1], [object_number=8, tag_tid=2, entry_tid=0], [object_number=11, tag_tid=1, entry_tid=12183], [object_number=10, tag_tid=1, entry_tid=12182]]], state=connected] 2024-04-06T09:59:59.245 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=8, tag_tid=1, entry_tid=12184], [object_number=11, tag_tid=1, entry_tid=12183], [object_number=10, tag_tid=1, entry_tid=12182], [object_number=9, tag_tid=1, entry_tid=12181]]], state=connected] 2024-04-06T09:59:59.248 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_5.status 2024-04-06T09:59:59.249 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=8, tag_tid=1, entry_tid=12184' 2024-04-06T09:59:59.250 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=9, tag_tid=2, entry_tid=1' -a 'object_number=9, tag_tid=2, entry_tid=1' = 'object_number=8, tag_tid=1, entry_tid=12184' 2024-04-06T09:59:59.250 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=8, tag_tid=1, entry_tid=12184' '!=' 'object_number=8, tag_tid=1, entry_tid=12184' 2024-04-06T09:59:59.250 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T09:59:59.250 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.8 2024-04-06T10:00:00.052 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror image_5 2024-04-06T10:00:00.052 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T10:00:00.052 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:00:00.052 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_5 2024-04-06T10:00:00.052 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T10:00:00.052 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n image_5 ']' 2024-04-06T10:00:00.052 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/image_5' 2024-04-06T10:00:00.052 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_5 2024-04-06T10:00:00.052 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T10:00:00.052 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T10:00:00.052 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T10:00:00.052 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T10:00:00.053 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T10:00:00.053 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T10:00:00.053 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T10:00:00.053 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T10:00:00.053 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T10:00:00.057 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T10:00:00.057 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_5 2024-04-06T10:00:00.057 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T10:00:00.057 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T10:00:00.057 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T10:00:00.057 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T10:00:00.057 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T10:00:00.057 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T10:00:00.057 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T10:00:00.057 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T10:00:00.057 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T10:00:00.058 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T10:00:00.058 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T10:00:00.058 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:00:00.058 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T10:00:00.058 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T10:00:00.058 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T10:00:00.058 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T10:00:00.058 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T10:00:00.058 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T10:00:00.058 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T10:00:00.058 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T10:00:00.058 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T10:00:00.058 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T10:00:00.058 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:00:00.059 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T10:00:00.059 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T10:00:00.059 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T10:00:00.059 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T10:00:00.059 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T10:00:00.059 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T10:00:00.059 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T10:00:00.083 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T10:00:00.083 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T10:00:00.083 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T10:00:00.083 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_5 2024-04-06T10:00:00.187 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:Invalid command: unused arguments: ['mirror/image_5'] 2024-04-06T10:00:00.188 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:rbd mirror flush : flush rbd mirror 2024-04-06T10:00:00.188 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:admin_socket: invalid command 2024-04-06T10:00:00.190 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T10:00:00.190 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/image_5 2024-04-06T10:00:00.190 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:1 2024-04-06T10:00:00.190 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T10:00:00.190 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T10:00:00.190 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-06T10:00:00.190 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:1 2024-04-06T10:00:00.190 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T10:00:00.190 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T10:00:00.190 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T10:00:00.191 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=1 2024-04-06T10:00:00.191 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-06T10:00:00.191 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T10:00:00.191 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:00:00.191 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=1 2024-04-06T10:00:00.191 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=1 2024-04-06T10:00:00.192 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-06T10:00:00.192 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:1 2024-04-06T10:00:00.192 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T10:00:00.193 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T10:00:00.193 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-06T10:00:00.193 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:1 2024-04-06T10:00:00.193 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T10:00:00.193 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T10:00:00.193 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:00:00.193 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=1 2024-04-06T10:00:00.193 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-06T10:00:00.193 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T10:00:00.193 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T10:00:00.193 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=1 2024-04-06T10:00:00.193 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=1 2024-04-06T10:00:00.193 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-06T10:00:00.216 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T10:00:00.216 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T10:00:00.216 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T10:00:00.216 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/image_5 2024-04-06T10:00:00.367 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T10:00:00.368 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror image_5 2024-04-06T10:00:00.368 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:00:00.368 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:00:00.369 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_5 2024-04-06T10:00:00.369 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror image_5 '' 2024-04-06T10:00:00.369 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:00:00.369 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:00:00.369 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_5 2024-04-06T10:00:00.369 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T10:00:00.369 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-image_5.status 2024-04-06T10:00:00.369 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-image_5.status 2024-04-06T10:00:00.370 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T10:00:00.371 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-image_5.status 2024-04-06T10:00:00.372 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_5 2024-04-06T10:00:00.372 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_5.status 2024-04-06T10:00:00.423 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 2 2024-04-06T10:00:00.423 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 2 2024-04-06T10:00:00.423 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T10:00:00.423 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=9, tag_tid=2, entry_tid=1], [object_number=8, tag_tid=2, entry_tid=0], [object_number=11, tag_tid=1, entry_tid=12183], [object_number=10, tag_tid=1, entry_tid=12182]]], state=connected] 2024-04-06T10:00:00.423 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=8, tag_tid=1, entry_tid=12184], [object_number=11, tag_tid=1, entry_tid=12183], [object_number=10, tag_tid=1, entry_tid=12182], [object_number=9, tag_tid=1, entry_tid=12181]]], state=connected] 2024-04-06T10:00:00.425 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_5.status 2024-04-06T10:00:00.426 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=9, tag_tid=2, entry_tid=1' 2024-04-06T10:00:00.428 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror image_5 2024-04-06T10:00:00.428 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:00:00.428 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:00:00.428 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_5 2024-04-06T10:00:00.428 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror image_5 '..*' 2024-04-06T10:00:00.428 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:00:00.428 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:00:00.428 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_5 2024-04-06T10:00:00.428 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T10:00:00.428 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-image_5.status 2024-04-06T10:00:00.428 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-image_5.status 2024-04-06T10:00:00.428 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T10:00:00.429 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-image_5.status 2024-04-06T10:00:00.429 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_5 2024-04-06T10:00:00.429 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_5.status 2024-04-06T10:00:00.474 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 2 2024-04-06T10:00:00.475 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 2 2024-04-06T10:00:00.475 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T10:00:00.475 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=9, tag_tid=2, entry_tid=1], [object_number=8, tag_tid=2, entry_tid=0], [object_number=11, tag_tid=1, entry_tid=12183], [object_number=10, tag_tid=1, entry_tid=12182]]], state=connected] 2024-04-06T10:00:00.475 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=8, tag_tid=1, entry_tid=12184], [object_number=11, tag_tid=1, entry_tid=12183], [object_number=10, tag_tid=1, entry_tid=12182], [object_number=9, tag_tid=1, entry_tid=12181]]], state=connected] 2024-04-06T10:00:00.478 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_5.status 2024-04-06T10:00:00.480 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=8, tag_tid=1, entry_tid=12184' 2024-04-06T10:00:00.480 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=9, tag_tid=2, entry_tid=1' -a 'object_number=9, tag_tid=2, entry_tid=1' = 'object_number=8, tag_tid=1, entry_tid=12184' 2024-04-06T10:00:00.480 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=8, tag_tid=1, entry_tid=12184' '!=' 'object_number=8, tag_tid=1, entry_tid=12184' 2024-04-06T10:00:00.480 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T10:00:00.480 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 1.6 2024-04-06T10:00:02.082 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror image_5 2024-04-06T10:00:02.082 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T10:00:02.082 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:00:02.082 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_5 2024-04-06T10:00:02.082 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T10:00:02.082 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n image_5 ']' 2024-04-06T10:00:02.082 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/image_5' 2024-04-06T10:00:02.083 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_5 2024-04-06T10:00:02.083 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T10:00:02.083 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T10:00:02.083 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T10:00:02.083 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T10:00:02.083 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T10:00:02.083 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T10:00:02.083 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T10:00:02.083 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T10:00:02.084 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T10:00:02.087 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T10:00:02.088 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_5 2024-04-06T10:00:02.088 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T10:00:02.088 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T10:00:02.088 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T10:00:02.088 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T10:00:02.088 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T10:00:02.088 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T10:00:02.088 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T10:00:02.088 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T10:00:02.089 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T10:00:02.089 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T10:00:02.089 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T10:00:02.089 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:00:02.089 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T10:00:02.089 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T10:00:02.089 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T10:00:02.089 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T10:00:02.089 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T10:00:02.089 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T10:00:02.089 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T10:00:02.089 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T10:00:02.089 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T10:00:02.090 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T10:00:02.090 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:00:02.090 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T10:00:02.090 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T10:00:02.090 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T10:00:02.090 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T10:00:02.090 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T10:00:02.090 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T10:00:02.090 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T10:00:02.115 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T10:00:02.115 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T10:00:02.115 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T10:00:02.115 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_5 2024-04-06T10:00:02.221 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:Invalid command: unused arguments: ['mirror/image_5'] 2024-04-06T10:00:02.221 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:rbd mirror flush : flush rbd mirror 2024-04-06T10:00:02.221 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:admin_socket: invalid command 2024-04-06T10:00:02.223 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T10:00:02.223 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/image_5 2024-04-06T10:00:02.223 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:1 2024-04-06T10:00:02.223 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T10:00:02.223 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T10:00:02.223 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-06T10:00:02.223 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:1 2024-04-06T10:00:02.223 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T10:00:02.223 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T10:00:02.224 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T10:00:02.224 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=1 2024-04-06T10:00:02.224 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-06T10:00:02.224 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T10:00:02.224 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:00:02.224 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=1 2024-04-06T10:00:02.224 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=1 2024-04-06T10:00:02.225 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-06T10:00:02.225 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:1 2024-04-06T10:00:02.225 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T10:00:02.225 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T10:00:02.226 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-06T10:00:02.226 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:1 2024-04-06T10:00:02.226 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T10:00:02.226 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T10:00:02.226 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:00:02.226 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=1 2024-04-06T10:00:02.226 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-06T10:00:02.226 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T10:00:02.226 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T10:00:02.226 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=1 2024-04-06T10:00:02.226 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=1 2024-04-06T10:00:02.226 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-06T10:00:02.244 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T10:00:02.245 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T10:00:02.245 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T10:00:02.245 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/image_5 2024-04-06T10:00:02.399 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T10:00:02.401 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror image_5 2024-04-06T10:00:02.401 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:00:02.401 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:00:02.401 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_5 2024-04-06T10:00:02.401 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror image_5 '' 2024-04-06T10:00:02.401 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:00:02.401 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:00:02.402 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_5 2024-04-06T10:00:02.402 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T10:00:02.402 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-image_5.status 2024-04-06T10:00:02.402 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T10:00:02.402 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-image_5.status 2024-04-06T10:00:02.403 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-image_5.status 2024-04-06T10:00:02.403 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_5 2024-04-06T10:00:02.404 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_5.status 2024-04-06T10:00:02.458 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 2 2024-04-06T10:00:02.458 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 2 2024-04-06T10:00:02.458 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T10:00:02.458 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=9, tag_tid=2, entry_tid=1], [object_number=8, tag_tid=2, entry_tid=0], [object_number=11, tag_tid=1, entry_tid=12183], [object_number=10, tag_tid=1, entry_tid=12182]]], state=connected] 2024-04-06T10:00:02.458 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=9, tag_tid=2, entry_tid=1], [object_number=8, tag_tid=2, entry_tid=0], [object_number=11, tag_tid=1, entry_tid=12183], [object_number=10, tag_tid=1, entry_tid=12182]]], state=connected] 2024-04-06T10:00:02.461 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_5.status 2024-04-06T10:00:02.463 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=9, tag_tid=2, entry_tid=1' 2024-04-06T10:00:02.464 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror image_5 2024-04-06T10:00:02.464 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:00:02.464 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:00:02.464 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_5 2024-04-06T10:00:02.464 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror image_5 '..*' 2024-04-06T10:00:02.464 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:00:02.464 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:00:02.464 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_5 2024-04-06T10:00:02.465 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T10:00:02.465 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-image_5.status 2024-04-06T10:00:02.465 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-image_5.status 2024-04-06T10:00:02.465 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T10:00:02.465 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-image_5.status 2024-04-06T10:00:02.465 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_5 2024-04-06T10:00:02.466 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_5.status 2024-04-06T10:00:02.511 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 2 2024-04-06T10:00:02.511 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 2 2024-04-06T10:00:02.512 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T10:00:02.512 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=9, tag_tid=2, entry_tid=1], [object_number=8, tag_tid=2, entry_tid=0], [object_number=11, tag_tid=1, entry_tid=12183], [object_number=10, tag_tid=1, entry_tid=12182]]], state=connected] 2024-04-06T10:00:02.512 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=9, tag_tid=2, entry_tid=1], [object_number=8, tag_tid=2, entry_tid=0], [object_number=11, tag_tid=1, entry_tid=12183], [object_number=10, tag_tid=1, entry_tid=12182]]], state=connected] 2024-04-06T10:00:02.515 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_5.status 2024-04-06T10:00:02.517 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=9, tag_tid=2, entry_tid=1' 2024-04-06T10:00:02.517 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=9, tag_tid=2, entry_tid=1' -a 'object_number=9, tag_tid=2, entry_tid=1' = 'object_number=9, tag_tid=2, entry_tid=1' 2024-04-06T10:00:02.517 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T10:00:02.517 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_snap_present cluster1 mirror image_5 snap 2024-04-06T10:00:02.517 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T10:00:02.517 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:00:02.517 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_5 2024-04-06T10:00:02.517 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local snap_name=snap 2024-04-06T10:00:02.518 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local s 2024-04-06T10:00:02.518 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-04-06T10:00:02.518 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 1 2024-04-06T10:00:03.518 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 info mirror/image_5@snap 2024-04-06T10:00:03.591 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:rbd image 'image_5': 2024-04-06T10:00:03.591 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: size 128 MiB in 32 objects 2024-04-06T10:00:03.591 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: order 22 (4 MiB objects) 2024-04-06T10:00:03.591 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: snapshot_count: 1 2024-04-06T10:00:03.591 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: id: 10ac99722d9f 2024-04-06T10:00:03.591 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: block_name_prefix: rbd_data.10ac99722d9f 2024-04-06T10:00:03.591 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: format: 2 2024-04-06T10:00:03.591 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: features: layering, exclusive-lock, journaling 2024-04-06T10:00:03.592 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: op_features: 2024-04-06T10:00:03.592 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: flags: 2024-04-06T10:00:03.592 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: create_timestamp: Sat Apr 6 09:51:22 2024 2024-04-06T10:00:03.592 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: access_timestamp: Sat Apr 6 09:51:22 2024 2024-04-06T10:00:03.592 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: modify_timestamp: Sat Apr 6 09:52:23 2024 2024-04-06T10:00:03.592 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: protected: False 2024-04-06T10:00:03.592 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: journal: 10ac99722d9f 2024-04-06T10:00:03.592 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: mirroring state: enabled 2024-04-06T10:00:03.592 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: mirroring mode: journal 2024-04-06T10:00:03.592 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: mirroring global id: 6177013a-a089-41ca-89d3-00420a642439 2024-04-06T10:00:03.592 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: mirroring primary: false 2024-04-06T10:00:03.596 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T10:00:03.596 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ compare_image_snaps mirror image_5 snap 2024-04-06T10:00:03.596 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:00:03.596 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_5 2024-04-06T10:00:03.596 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local snap_name=snap 2024-04-06T10:00:03.596 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local ret=0 2024-04-06T10:00:03.597 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local rmt_export=/tmp/tmp.BhW6xKmROU/cluster2-mirror-image_5.export 2024-04-06T10:00:03.597 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loc_export=/tmp/tmp.BhW6xKmROU/cluster1-mirror-image_5.export 2024-04-06T10:00:03.597 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rm -f /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_5.export /tmp/tmp.BhW6xKmROU/cluster1-mirror-image_5.export 2024-04-06T10:00:03.597 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster2 -p mirror export image_5@snap /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_5.export 2024-04-06T10:00:03.782 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-04-06T10:00:03.791 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 -p mirror export image_5@snap /tmp/tmp.BhW6xKmROU/cluster1-mirror-image_5.export 2024-04-06T10:00:04.160 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-04-06T10:00:04.174 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmp /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_5.export /tmp/tmp.BhW6xKmROU/cluster1-mirror-image_5.export 2024-04-06T10:00:04.254 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rm -f /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_5.export /tmp/tmp.BhW6xKmROU/cluster1-mirror-image_5.export 2024-04-06T10:00:04.296 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T10:00:04.297 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-06T10:00:04.297 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ image=image_6 2024-04-06T10:00:04.297 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ create_snap cluster2 mirror image_6 snap 2024-04-06T10:00:04.297 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-04-06T10:00:04.297 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:00:04.297 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_6 2024-04-06T10:00:04.297 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local snap_name=snap 2024-04-06T10:00:04.297 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster2 -p mirror snap create image_6@snap --debug-rbd=20 --debug-journaler=20 2024-04-06T10:00:06.850 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_image_replay_started cluster1 mirror image_6 2024-04-06T10:00:06.850 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T10:00:06.850 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:00:06.850 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_6 2024-04-06T10:00:06.850 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_image_replay_state cluster1 mirror image_6 started 2024-04-06T10:00:06.850 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T10:00:06.850 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:00:06.850 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_6 2024-04-06T10:00:06.850 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state=started 2024-04-06T10:00:06.850 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local s 2024-04-06T10:00:06.850 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T10:00:06.851 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 1 2024-04-06T10:00:07.851 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test_image_replay_state cluster1 mirror image_6 started 2024-04-06T10:00:07.851 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T10:00:07.851 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:00:07.851 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_6 2024-04-06T10:00:07.852 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local test_state=started 2024-04-06T10:00:07.852 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local status_result 2024-04-06T10:00:07.852 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local current_state=stopped 2024-04-06T10:00:07.852 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_6 2024-04-06T10:00:07.853 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_instance=cluster1 2024-04-06T10:00:07.853 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ shift 2024-04-06T10:00:07.853 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T10:00:07.853 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance=cluster1 2024-04-06T10:00:07.853 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local loop_instance 2024-04-06T10:00:07.853 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T10:00:07.853 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sleep 0 2024-04-06T10:00:07.853 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -i state 2024-04-06T10:00:07.854 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-04-06T10:00:07.854 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ seq 0 3 2024-04-06T10:00:07.857 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T10:00:07.857 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_6 2024-04-06T10:00:07.857 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1:0 2024-04-06T10:00:07.857 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ shift 2024-04-06T10:00:07.857 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T10:00:07.857 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-04-06T10:00:07.857 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T10:00:07.857 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=cluster 2024-04-06T10:00:07.857 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T10:00:07.857 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:00:07.857 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T10:00:07.857 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T10:00:07.857 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval cluster=cluster1 2024-04-06T10:00:07.858 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ cluster=cluster1 2024-04-06T10:00:07.858 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T10:00:07.858 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T10:00:07.858 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-04-06T10:00:07.858 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local local_cluster=cluster1:0 2024-04-06T10:00:07.858 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local cluster=cluster1 2024-04-06T10:00:07.858 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local instance 2024-04-06T10:00:07.858 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T10:00:07.858 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local val=cluster1:0 2024-04-06T10:00:07.858 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local cluster_var_name=local_cluster 2024-04-06T10:00:07.858 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local instance_var_name=instance 2024-04-06T10:00:07.858 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ cluster=cluster1 2024-04-06T10:00:07.858 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T10:00:07.858 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ '[' 0 = cluster1:0 ']' 2024-04-06T10:00:07.859 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ eval local_cluster=cluster1 2024-04-06T10:00:07.859 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++++ local_cluster=cluster1 2024-04-06T10:00:07.859 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ eval instance=0 2024-04-06T10:00:07.859 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++++ instance=0 2024-04-06T10:00:07.859 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T10:00:07.884 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T10:00:07.884 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T10:00:07.884 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T10:00:07.884 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_6 2024-04-06T10:00:07.988 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:Invalid command: unused arguments: ['mirror/image_6'] 2024-04-06T10:00:07.988 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:rbd mirror status : get status for rbd mirror 2024-04-06T10:00:07.988 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:admin_socket: invalid command 2024-04-06T10:00:07.990 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T10:00:07.990 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_6 2024-04-06T10:00:07.990 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1:1 2024-04-06T10:00:07.990 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ shift 2024-04-06T10:00:07.990 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T10:00:07.990 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-04-06T10:00:07.991 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:1 2024-04-06T10:00:07.991 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=cluster 2024-04-06T10:00:07.991 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T10:00:07.991 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:00:07.991 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=1 2024-04-06T10:00:07.991 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-06T10:00:07.991 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval cluster=cluster1 2024-04-06T10:00:07.991 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ cluster=cluster1 2024-04-06T10:00:07.991 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=1 2024-04-06T10:00:07.991 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=1 2024-04-06T10:00:07.993 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-04-06T10:00:07.993 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local local_cluster=cluster1:1 2024-04-06T10:00:07.993 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local cluster=cluster1 2024-04-06T10:00:07.993 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local instance 2024-04-06T10:00:07.993 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-06T10:00:07.993 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local val=cluster1:1 2024-04-06T10:00:07.994 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local cluster_var_name=local_cluster 2024-04-06T10:00:07.994 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local instance_var_name=instance 2024-04-06T10:00:07.994 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ cluster=cluster1 2024-04-06T10:00:07.994 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=1 2024-04-06T10:00:07.994 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ '[' 1 = cluster1:1 ']' 2024-04-06T10:00:07.994 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ eval local_cluster=cluster1 2024-04-06T10:00:07.994 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++++ local_cluster=cluster1 2024-04-06T10:00:07.994 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ eval instance=1 2024-04-06T10:00:07.994 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++++ instance=1 2024-04-06T10:00:07.994 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-06T10:00:08.011 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T10:00:08.011 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T10:00:08.011 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T10:00:08.012 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_6 2024-04-06T10:00:08.160 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ return 0 2024-04-06T10:00:08.161 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ status_result=' "state": "Replaying"' 2024-04-06T10:00:08.161 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ echo ' "state": "Replaying"' 2024-04-06T10:00:08.161 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep -i Replaying 2024-04-06T10:00:08.163 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: "state": "Replaying" 2024-04-06T10:00:08.164 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ current_state=started 2024-04-06T10:00:08.164 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test started = started 2024-04-06T10:00:08.164 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T10:00:08.165 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_6 2024-04-06T10:00:08.165 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local local_cluster=cluster1 2024-04-06T10:00:08.165 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-04-06T10:00:08.165 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:00:08.165 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_6 2024-04-06T10:00:08.165 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' journal = journal ']' 2024-04-06T10:00:08.165 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_journal_replay_complete cluster1 cluster2 mirror image_6 2024-04-06T10:00:08.165 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local local_cluster=cluster1 2024-04-06T10:00:08.165 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-04-06T10:00:08.165 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:00:08.165 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_6 2024-04-06T10:00:08.165 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local s master_pos mirror_pos last_mirror_pos 2024-04-06T10:00:08.165 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local master_tag master_entry mirror_tag mirror_entry 2024-04-06T10:00:08.165 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T10:00:08.166 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T10:00:08.166 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T10:00:08.366 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror image_6 2024-04-06T10:00:08.366 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T10:00:08.366 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:00:08.366 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_6 2024-04-06T10:00:08.367 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T10:00:08.367 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n image_6 ']' 2024-04-06T10:00:08.367 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/image_6' 2024-04-06T10:00:08.367 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_6 2024-04-06T10:00:08.367 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T10:00:08.367 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T10:00:08.367 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T10:00:08.367 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T10:00:08.367 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T10:00:08.367 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T10:00:08.367 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T10:00:08.368 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T10:00:08.368 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T10:00:08.371 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T10:00:08.371 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_6 2024-04-06T10:00:08.371 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T10:00:08.371 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T10:00:08.372 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T10:00:08.372 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T10:00:08.372 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T10:00:08.372 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T10:00:08.372 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T10:00:08.372 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T10:00:08.372 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T10:00:08.372 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T10:00:08.372 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T10:00:08.372 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:00:08.372 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T10:00:08.372 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T10:00:08.372 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T10:00:08.372 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T10:00:08.373 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T10:00:08.373 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T10:00:08.373 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T10:00:08.373 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T10:00:08.373 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T10:00:08.373 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T10:00:08.373 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:00:08.373 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T10:00:08.373 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T10:00:08.373 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T10:00:08.373 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T10:00:08.373 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T10:00:08.373 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T10:00:08.374 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T10:00:08.397 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T10:00:08.397 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T10:00:08.397 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T10:00:08.398 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_6 2024-04-06T10:00:08.503 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:Invalid command: unused arguments: ['mirror/image_6'] 2024-04-06T10:00:08.503 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:rbd mirror flush : flush rbd mirror 2024-04-06T10:00:08.503 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:admin_socket: invalid command 2024-04-06T10:00:08.505 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T10:00:08.505 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/image_6 2024-04-06T10:00:08.505 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:1 2024-04-06T10:00:08.505 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T10:00:08.505 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T10:00:08.505 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-06T10:00:08.506 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:1 2024-04-06T10:00:08.506 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T10:00:08.506 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T10:00:08.506 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T10:00:08.506 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=1 2024-04-06T10:00:08.506 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-06T10:00:08.506 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T10:00:08.506 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:00:08.506 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=1 2024-04-06T10:00:08.506 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=1 2024-04-06T10:00:08.507 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-06T10:00:08.508 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:1 2024-04-06T10:00:08.508 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T10:00:08.508 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T10:00:08.508 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-06T10:00:08.508 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:1 2024-04-06T10:00:08.508 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T10:00:08.508 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T10:00:08.508 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:00:08.508 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=1 2024-04-06T10:00:08.508 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-06T10:00:08.508 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T10:00:08.508 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T10:00:08.508 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=1 2024-04-06T10:00:08.508 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=1 2024-04-06T10:00:08.509 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-06T10:00:08.531 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T10:00:08.531 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T10:00:08.531 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T10:00:08.531 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/image_6 2024-04-06T10:00:08.680 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T10:00:08.680 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror image_6 2024-04-06T10:00:08.681 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:00:08.681 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:00:08.681 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_6 2024-04-06T10:00:08.681 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror image_6 '' 2024-04-06T10:00:08.681 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:00:08.681 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:00:08.681 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_6 2024-04-06T10:00:08.681 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T10:00:08.682 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-image_6.status 2024-04-06T10:00:08.682 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-image_6.status 2024-04-06T10:00:08.682 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T10:00:08.684 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-image_6.status 2024-04-06T10:00:08.685 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_6.status 2024-04-06T10:00:08.685 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_6 2024-04-06T10:00:08.736 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 3 2024-04-06T10:00:08.736 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 3 2024-04-06T10:00:08.736 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T10:00:08.736 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=13, tag_tid=2, entry_tid=1], [object_number=12, tag_tid=2, entry_tid=0], [object_number=14, tag_tid=1, entry_tid=15170], [object_number=15, tag_tid=1, entry_tid=15167]]], state=connected] 2024-04-06T10:00:08.736 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=14, tag_tid=1, entry_tid=15170], [object_number=13, tag_tid=1, entry_tid=15169], [object_number=12, tag_tid=1, entry_tid=15168], [object_number=15, tag_tid=1, entry_tid=15167]]], state=connected] 2024-04-06T10:00:08.739 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_6.status 2024-04-06T10:00:08.741 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=13, tag_tid=2, entry_tid=1' 2024-04-06T10:00:08.742 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror image_6 2024-04-06T10:00:08.742 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:00:08.742 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:00:08.742 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_6 2024-04-06T10:00:08.742 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror image_6 '..*' 2024-04-06T10:00:08.742 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:00:08.742 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:00:08.742 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_6 2024-04-06T10:00:08.743 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T10:00:08.743 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-image_6.status 2024-04-06T10:00:08.743 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-image_6.status 2024-04-06T10:00:08.743 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T10:00:08.743 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-image_6.status 2024-04-06T10:00:08.744 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_6 2024-04-06T10:00:08.744 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_6.status 2024-04-06T10:00:09.001 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 3 2024-04-06T10:00:09.001 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 3 2024-04-06T10:00:09.001 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T10:00:09.001 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=13, tag_tid=2, entry_tid=1], [object_number=12, tag_tid=2, entry_tid=0], [object_number=14, tag_tid=1, entry_tid=15170], [object_number=15, tag_tid=1, entry_tid=15167]]], state=connected] 2024-04-06T10:00:09.001 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=14, tag_tid=1, entry_tid=15170], [object_number=13, tag_tid=1, entry_tid=15169], [object_number=12, tag_tid=1, entry_tid=15168], [object_number=15, tag_tid=1, entry_tid=15167]]], state=connected] 2024-04-06T10:00:09.005 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_6.status 2024-04-06T10:00:09.008 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=14, tag_tid=1, entry_tid=15170' 2024-04-06T10:00:09.008 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=13, tag_tid=2, entry_tid=1' -a 'object_number=13, tag_tid=2, entry_tid=1' = 'object_number=14, tag_tid=1, entry_tid=15170' 2024-04-06T10:00:09.008 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=14, tag_tid=1, entry_tid=15170' '!=' '' 2024-04-06T10:00:09.008 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T10:00:09.008 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=14, tag_tid=1, entry_tid=15170' = '' 2024-04-06T10:00:09.008 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=14, tag_tid=1, entry_tid=15170' 2024-04-06T10:00:09.009 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=13, tag_tid=2, entry_tid=1' 2024-04-06T10:00:09.009 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T10:00:09.009 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T10:00:09.010 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=2 2024-04-06T10:00:09.011 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=14, tag_tid=1, entry_tid=15170' 2024-04-06T10:00:09.011 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T10:00:09.011 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T10:00:09.012 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=1 2024-04-06T10:00:09.013 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=13, tag_tid=2, entry_tid=1' 2024-04-06T10:00:09.013 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T10:00:09.013 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T10:00:09.014 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T10:00:09.015 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=14, tag_tid=1, entry_tid=15170' 2024-04-06T10:00:09.015 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T10:00:09.015 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T10:00:09.016 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=15170 2024-04-06T10:00:09.017 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 2 = 1 -a 1 -le 15170 2024-04-06T10:00:09.017 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T10:00:09.017 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T10:00:09.017 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T10:00:09.219 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror image_6 2024-04-06T10:00:09.219 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T10:00:09.219 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:00:09.219 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_6 2024-04-06T10:00:09.219 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T10:00:09.219 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n image_6 ']' 2024-04-06T10:00:09.219 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/image_6' 2024-04-06T10:00:09.220 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_6 2024-04-06T10:00:09.220 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T10:00:09.220 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T10:00:09.220 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T10:00:09.220 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T10:00:09.220 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T10:00:09.220 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T10:00:09.220 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T10:00:09.220 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T10:00:09.220 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T10:00:09.222 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T10:00:09.222 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_6 2024-04-06T10:00:09.222 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T10:00:09.222 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T10:00:09.222 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T10:00:09.222 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T10:00:09.222 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T10:00:09.222 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T10:00:09.222 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T10:00:09.222 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T10:00:09.222 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T10:00:09.222 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T10:00:09.222 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T10:00:09.223 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:00:09.223 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T10:00:09.223 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T10:00:09.223 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T10:00:09.224 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T10:00:09.224 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T10:00:09.224 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T10:00:09.224 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T10:00:09.224 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T10:00:09.224 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T10:00:09.224 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T10:00:09.224 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:00:09.224 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T10:00:09.224 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T10:00:09.224 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T10:00:09.224 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T10:00:09.224 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T10:00:09.224 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T10:00:09.225 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T10:00:09.250 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T10:00:09.250 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T10:00:09.250 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T10:00:09.250 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_6 2024-04-06T10:00:09.356 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:Invalid command: unused arguments: ['mirror/image_6'] 2024-04-06T10:00:09.356 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:rbd mirror flush : flush rbd mirror 2024-04-06T10:00:09.356 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:admin_socket: invalid command 2024-04-06T10:00:09.358 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T10:00:09.358 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/image_6 2024-04-06T10:00:09.358 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:1 2024-04-06T10:00:09.358 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T10:00:09.358 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T10:00:09.358 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-06T10:00:09.358 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:1 2024-04-06T10:00:09.358 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T10:00:09.358 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T10:00:09.358 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T10:00:09.359 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=1 2024-04-06T10:00:09.359 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-06T10:00:09.359 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T10:00:09.359 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:00:09.359 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=1 2024-04-06T10:00:09.359 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=1 2024-04-06T10:00:09.360 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-06T10:00:09.360 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:1 2024-04-06T10:00:09.360 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T10:00:09.360 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T10:00:09.360 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-06T10:00:09.361 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:1 2024-04-06T10:00:09.361 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T10:00:09.361 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T10:00:09.361 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:00:09.361 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=1 2024-04-06T10:00:09.361 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-06T10:00:09.361 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T10:00:09.361 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T10:00:09.361 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=1 2024-04-06T10:00:09.361 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=1 2024-04-06T10:00:09.361 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-06T10:00:09.383 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T10:00:09.384 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T10:00:09.384 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T10:00:09.384 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/image_6 2024-04-06T10:00:09.525 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T10:00:09.525 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror image_6 2024-04-06T10:00:09.525 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:00:09.525 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:00:09.525 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_6 2024-04-06T10:00:09.525 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror image_6 '' 2024-04-06T10:00:09.525 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:00:09.526 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:00:09.526 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_6 2024-04-06T10:00:09.526 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T10:00:09.526 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-image_6.status 2024-04-06T10:00:09.526 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-image_6.status 2024-04-06T10:00:09.526 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T10:00:09.527 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-image_6.status 2024-04-06T10:00:09.528 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_6.status 2024-04-06T10:00:09.528 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_6 2024-04-06T10:00:09.586 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 3 2024-04-06T10:00:09.586 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 3 2024-04-06T10:00:09.586 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T10:00:09.586 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=13, tag_tid=2, entry_tid=1], [object_number=12, tag_tid=2, entry_tid=0], [object_number=14, tag_tid=1, entry_tid=15170], [object_number=15, tag_tid=1, entry_tid=15167]]], state=connected] 2024-04-06T10:00:09.586 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=14, tag_tid=1, entry_tid=15170], [object_number=13, tag_tid=1, entry_tid=15169], [object_number=12, tag_tid=1, entry_tid=15168], [object_number=15, tag_tid=1, entry_tid=15167]]], state=connected] 2024-04-06T10:00:09.589 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_6.status 2024-04-06T10:00:09.592 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=13, tag_tid=2, entry_tid=1' 2024-04-06T10:00:09.592 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror image_6 2024-04-06T10:00:09.592 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:00:09.593 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:00:09.593 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_6 2024-04-06T10:00:09.593 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror image_6 '..*' 2024-04-06T10:00:09.593 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:00:09.593 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:00:09.593 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_6 2024-04-06T10:00:09.593 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T10:00:09.593 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-image_6.status 2024-04-06T10:00:09.593 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-image_6.status 2024-04-06T10:00:09.593 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T10:00:09.594 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-image_6.status 2024-04-06T10:00:09.594 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_6 2024-04-06T10:00:09.594 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_6.status 2024-04-06T10:00:09.639 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 3 2024-04-06T10:00:09.640 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 3 2024-04-06T10:00:09.640 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T10:00:09.640 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=13, tag_tid=2, entry_tid=1], [object_number=12, tag_tid=2, entry_tid=0], [object_number=14, tag_tid=1, entry_tid=15170], [object_number=15, tag_tid=1, entry_tid=15167]]], state=connected] 2024-04-06T10:00:09.640 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=14, tag_tid=1, entry_tid=15170], [object_number=13, tag_tid=1, entry_tid=15169], [object_number=12, tag_tid=1, entry_tid=15168], [object_number=15, tag_tid=1, entry_tid=15167]]], state=connected] 2024-04-06T10:00:09.643 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_6.status 2024-04-06T10:00:09.645 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=14, tag_tid=1, entry_tid=15170' 2024-04-06T10:00:09.645 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=13, tag_tid=2, entry_tid=1' -a 'object_number=13, tag_tid=2, entry_tid=1' = 'object_number=14, tag_tid=1, entry_tid=15170' 2024-04-06T10:00:09.645 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=14, tag_tid=1, entry_tid=15170' '!=' 'object_number=14, tag_tid=1, entry_tid=15170' 2024-04-06T10:00:09.645 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T10:00:09.645 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.4 2024-04-06T10:00:10.047 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror image_6 2024-04-06T10:00:10.047 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T10:00:10.047 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:00:10.047 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_6 2024-04-06T10:00:10.047 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T10:00:10.047 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n image_6 ']' 2024-04-06T10:00:10.047 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/image_6' 2024-04-06T10:00:10.048 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_6 2024-04-06T10:00:10.048 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T10:00:10.048 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T10:00:10.048 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T10:00:10.048 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T10:00:10.048 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T10:00:10.048 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T10:00:10.048 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T10:00:10.049 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T10:00:10.049 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T10:00:10.053 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T10:00:10.053 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_6 2024-04-06T10:00:10.053 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T10:00:10.053 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T10:00:10.053 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T10:00:10.053 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T10:00:10.053 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T10:00:10.054 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T10:00:10.054 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T10:00:10.054 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T10:00:10.054 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T10:00:10.054 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T10:00:10.054 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T10:00:10.054 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:00:10.054 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T10:00:10.054 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T10:00:10.054 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T10:00:10.054 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T10:00:10.054 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T10:00:10.054 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T10:00:10.055 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T10:00:10.055 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T10:00:10.055 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T10:00:10.055 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T10:00:10.055 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:00:10.055 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T10:00:10.055 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T10:00:10.055 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T10:00:10.055 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T10:00:10.055 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T10:00:10.055 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T10:00:10.055 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T10:00:10.078 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T10:00:10.078 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T10:00:10.078 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T10:00:10.079 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_6 2024-04-06T10:00:10.180 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:Invalid command: unused arguments: ['mirror/image_6'] 2024-04-06T10:00:10.180 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:rbd mirror flush : flush rbd mirror 2024-04-06T10:00:10.180 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:admin_socket: invalid command 2024-04-06T10:00:10.183 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T10:00:10.183 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/image_6 2024-04-06T10:00:10.183 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:1 2024-04-06T10:00:10.183 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T10:00:10.183 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T10:00:10.183 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-06T10:00:10.183 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:1 2024-04-06T10:00:10.183 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T10:00:10.183 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T10:00:10.183 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T10:00:10.183 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=1 2024-04-06T10:00:10.183 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-06T10:00:10.183 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T10:00:10.183 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:00:10.184 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=1 2024-04-06T10:00:10.184 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=1 2024-04-06T10:00:10.185 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-06T10:00:10.185 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:1 2024-04-06T10:00:10.185 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T10:00:10.186 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T10:00:10.186 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-06T10:00:10.186 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:1 2024-04-06T10:00:10.186 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T10:00:10.186 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T10:00:10.186 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:00:10.186 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=1 2024-04-06T10:00:10.186 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-06T10:00:10.186 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T10:00:10.186 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T10:00:10.186 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=1 2024-04-06T10:00:10.186 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=1 2024-04-06T10:00:10.186 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-06T10:00:10.208 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T10:00:10.208 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T10:00:10.208 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T10:00:10.208 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/image_6 2024-04-06T10:00:10.346 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T10:00:10.347 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror image_6 2024-04-06T10:00:10.347 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:00:10.347 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:00:10.347 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_6 2024-04-06T10:00:10.347 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror image_6 '' 2024-04-06T10:00:10.347 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:00:10.347 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:00:10.347 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_6 2024-04-06T10:00:10.348 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T10:00:10.348 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-image_6.status 2024-04-06T10:00:10.348 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-image_6.status 2024-04-06T10:00:10.348 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T10:00:10.350 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-image_6.status 2024-04-06T10:00:10.350 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_6 2024-04-06T10:00:10.350 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_6.status 2024-04-06T10:00:10.406 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 3 2024-04-06T10:00:10.407 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 3 2024-04-06T10:00:10.407 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T10:00:10.407 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=13, tag_tid=2, entry_tid=1], [object_number=12, tag_tid=2, entry_tid=0], [object_number=14, tag_tid=1, entry_tid=15170], [object_number=15, tag_tid=1, entry_tid=15167]]], state=connected] 2024-04-06T10:00:10.407 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=14, tag_tid=1, entry_tid=15170], [object_number=13, tag_tid=1, entry_tid=15169], [object_number=12, tag_tid=1, entry_tid=15168], [object_number=15, tag_tid=1, entry_tid=15167]]], state=connected] 2024-04-06T10:00:10.410 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_6.status 2024-04-06T10:00:10.411 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=13, tag_tid=2, entry_tid=1' 2024-04-06T10:00:10.412 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror image_6 2024-04-06T10:00:10.412 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:00:10.412 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:00:10.413 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_6 2024-04-06T10:00:10.413 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror image_6 '..*' 2024-04-06T10:00:10.413 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:00:10.413 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:00:10.413 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_6 2024-04-06T10:00:10.413 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T10:00:10.413 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-image_6.status 2024-04-06T10:00:10.413 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-image_6.status 2024-04-06T10:00:10.413 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T10:00:10.414 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-image_6.status 2024-04-06T10:00:10.414 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_6 2024-04-06T10:00:10.414 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_6.status 2024-04-06T10:00:10.459 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 3 2024-04-06T10:00:10.460 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 3 2024-04-06T10:00:10.460 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T10:00:10.460 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=13, tag_tid=2, entry_tid=1], [object_number=12, tag_tid=2, entry_tid=0], [object_number=14, tag_tid=1, entry_tid=15170], [object_number=15, tag_tid=1, entry_tid=15167]]], state=connected] 2024-04-06T10:00:10.460 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=14, tag_tid=1, entry_tid=15170], [object_number=13, tag_tid=1, entry_tid=15169], [object_number=12, tag_tid=1, entry_tid=15168], [object_number=15, tag_tid=1, entry_tid=15167]]], state=connected] 2024-04-06T10:00:10.463 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_6.status 2024-04-06T10:00:10.465 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=14, tag_tid=1, entry_tid=15170' 2024-04-06T10:00:10.465 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=13, tag_tid=2, entry_tid=1' -a 'object_number=13, tag_tid=2, entry_tid=1' = 'object_number=14, tag_tid=1, entry_tid=15170' 2024-04-06T10:00:10.465 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=14, tag_tid=1, entry_tid=15170' '!=' 'object_number=14, tag_tid=1, entry_tid=15170' 2024-04-06T10:00:10.465 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T10:00:10.465 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.8 2024-04-06T10:00:11.267 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror image_6 2024-04-06T10:00:11.268 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T10:00:11.268 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:00:11.268 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_6 2024-04-06T10:00:11.268 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T10:00:11.268 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n image_6 ']' 2024-04-06T10:00:11.268 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/image_6' 2024-04-06T10:00:11.268 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_6 2024-04-06T10:00:11.268 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T10:00:11.268 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T10:00:11.268 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T10:00:11.268 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T10:00:11.268 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T10:00:11.268 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T10:00:11.268 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T10:00:11.269 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T10:00:11.269 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T10:00:11.272 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T10:00:11.272 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_6 2024-04-06T10:00:11.272 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T10:00:11.272 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T10:00:11.272 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T10:00:11.273 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T10:00:11.273 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T10:00:11.273 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T10:00:11.273 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T10:00:11.273 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T10:00:11.273 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T10:00:11.273 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T10:00:11.273 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T10:00:11.273 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:00:11.273 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T10:00:11.273 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T10:00:11.275 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T10:00:11.275 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T10:00:11.275 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T10:00:11.275 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T10:00:11.275 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T10:00:11.275 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T10:00:11.275 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T10:00:11.275 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T10:00:11.275 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:00:11.275 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T10:00:11.275 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T10:00:11.275 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T10:00:11.275 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T10:00:11.275 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T10:00:11.276 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T10:00:11.276 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T10:00:11.309 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T10:00:11.309 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T10:00:11.309 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T10:00:11.309 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_6 2024-04-06T10:00:11.423 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:Invalid command: unused arguments: ['mirror/image_6'] 2024-04-06T10:00:11.423 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:rbd mirror flush : flush rbd mirror 2024-04-06T10:00:11.423 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:admin_socket: invalid command 2024-04-06T10:00:11.427 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T10:00:11.427 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/image_6 2024-04-06T10:00:11.427 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:1 2024-04-06T10:00:11.427 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T10:00:11.427 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T10:00:11.427 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-06T10:00:11.427 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:1 2024-04-06T10:00:11.427 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T10:00:11.427 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T10:00:11.427 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T10:00:11.427 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=1 2024-04-06T10:00:11.427 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-06T10:00:11.427 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T10:00:11.428 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:00:11.428 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=1 2024-04-06T10:00:11.428 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=1 2024-04-06T10:00:11.428 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-06T10:00:11.428 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:1 2024-04-06T10:00:11.428 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T10:00:11.428 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T10:00:11.428 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-06T10:00:11.428 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:1 2024-04-06T10:00:11.428 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T10:00:11.428 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T10:00:11.428 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:00:11.428 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=1 2024-04-06T10:00:11.428 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-06T10:00:11.429 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T10:00:11.429 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T10:00:11.429 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=1 2024-04-06T10:00:11.429 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=1 2024-04-06T10:00:11.429 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-06T10:00:11.446 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T10:00:11.446 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T10:00:11.446 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T10:00:11.446 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/image_6 2024-04-06T10:00:11.584 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T10:00:11.585 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror image_6 2024-04-06T10:00:11.586 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:00:11.586 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:00:11.586 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_6 2024-04-06T10:00:11.586 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror image_6 '' 2024-04-06T10:00:11.586 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:00:11.586 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:00:11.586 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_6 2024-04-06T10:00:11.586 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T10:00:11.587 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-image_6.status 2024-04-06T10:00:11.587 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-image_6.status 2024-04-06T10:00:11.587 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T10:00:11.588 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-image_6.status 2024-04-06T10:00:11.588 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_6 2024-04-06T10:00:11.588 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_6.status 2024-04-06T10:00:11.639 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 3 2024-04-06T10:00:11.639 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 3 2024-04-06T10:00:11.639 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T10:00:11.639 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=13, tag_tid=2, entry_tid=1], [object_number=12, tag_tid=2, entry_tid=0], [object_number=14, tag_tid=1, entry_tid=15170], [object_number=15, tag_tid=1, entry_tid=15167]]], state=connected] 2024-04-06T10:00:11.639 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=14, tag_tid=1, entry_tid=15170], [object_number=13, tag_tid=1, entry_tid=15169], [object_number=12, tag_tid=1, entry_tid=15168], [object_number=15, tag_tid=1, entry_tid=15167]]], state=connected] 2024-04-06T10:00:11.642 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_6.status 2024-04-06T10:00:11.644 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=13, tag_tid=2, entry_tid=1' 2024-04-06T10:00:11.645 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror image_6 2024-04-06T10:00:11.645 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:00:11.645 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:00:11.645 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_6 2024-04-06T10:00:11.645 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror image_6 '..*' 2024-04-06T10:00:11.645 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:00:11.645 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:00:11.645 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_6 2024-04-06T10:00:11.645 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T10:00:11.645 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-image_6.status 2024-04-06T10:00:11.646 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-image_6.status 2024-04-06T10:00:11.646 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T10:00:11.646 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-image_6.status 2024-04-06T10:00:11.646 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_6 2024-04-06T10:00:11.647 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_6.status 2024-04-06T10:00:11.692 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 3 2024-04-06T10:00:11.692 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 3 2024-04-06T10:00:11.692 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T10:00:11.692 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=13, tag_tid=2, entry_tid=1], [object_number=12, tag_tid=2, entry_tid=0], [object_number=14, tag_tid=1, entry_tid=15170], [object_number=15, tag_tid=1, entry_tid=15167]]], state=connected] 2024-04-06T10:00:11.692 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=14, tag_tid=1, entry_tid=15170], [object_number=13, tag_tid=1, entry_tid=15169], [object_number=12, tag_tid=1, entry_tid=15168], [object_number=15, tag_tid=1, entry_tid=15167]]], state=connected] 2024-04-06T10:00:11.696 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_6.status 2024-04-06T10:00:11.698 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=14, tag_tid=1, entry_tid=15170' 2024-04-06T10:00:11.698 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=13, tag_tid=2, entry_tid=1' -a 'object_number=13, tag_tid=2, entry_tid=1' = 'object_number=14, tag_tid=1, entry_tid=15170' 2024-04-06T10:00:11.698 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=14, tag_tid=1, entry_tid=15170' '!=' 'object_number=14, tag_tid=1, entry_tid=15170' 2024-04-06T10:00:11.698 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T10:00:11.698 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 1.6 2024-04-06T10:00:13.300 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror image_6 2024-04-06T10:00:13.300 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T10:00:13.300 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:00:13.300 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_6 2024-04-06T10:00:13.300 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T10:00:13.300 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n image_6 ']' 2024-04-06T10:00:13.300 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/image_6' 2024-04-06T10:00:13.300 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_6 2024-04-06T10:00:13.301 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T10:00:13.301 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T10:00:13.301 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T10:00:13.301 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T10:00:13.301 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T10:00:13.301 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T10:00:13.301 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T10:00:13.301 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T10:00:13.302 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T10:00:13.303 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T10:00:13.303 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_6 2024-04-06T10:00:13.303 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T10:00:13.303 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T10:00:13.303 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T10:00:13.304 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T10:00:13.304 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T10:00:13.304 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T10:00:13.304 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T10:00:13.304 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T10:00:13.304 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T10:00:13.304 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T10:00:13.304 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T10:00:13.306 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:00:13.306 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T10:00:13.306 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T10:00:13.306 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T10:00:13.306 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T10:00:13.306 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T10:00:13.307 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T10:00:13.307 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T10:00:13.307 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T10:00:13.307 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T10:00:13.307 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T10:00:13.307 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:00:13.307 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T10:00:13.307 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T10:00:13.307 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T10:00:13.307 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T10:00:13.307 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T10:00:13.307 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T10:00:13.307 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T10:00:13.331 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T10:00:13.332 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T10:00:13.332 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T10:00:13.332 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_6 2024-04-06T10:00:13.436 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:Invalid command: unused arguments: ['mirror/image_6'] 2024-04-06T10:00:13.436 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:rbd mirror flush : flush rbd mirror 2024-04-06T10:00:13.436 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:admin_socket: invalid command 2024-04-06T10:00:13.438 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T10:00:13.438 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/image_6 2024-04-06T10:00:13.438 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:1 2024-04-06T10:00:13.438 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T10:00:13.438 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T10:00:13.438 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-06T10:00:13.439 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:1 2024-04-06T10:00:13.439 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T10:00:13.439 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T10:00:13.439 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T10:00:13.439 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=1 2024-04-06T10:00:13.439 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-06T10:00:13.439 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T10:00:13.439 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:00:13.439 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=1 2024-04-06T10:00:13.439 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=1 2024-04-06T10:00:13.440 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-06T10:00:13.440 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:1 2024-04-06T10:00:13.440 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T10:00:13.441 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T10:00:13.441 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-06T10:00:13.441 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:1 2024-04-06T10:00:13.441 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T10:00:13.441 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T10:00:13.441 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:00:13.441 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=1 2024-04-06T10:00:13.441 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-06T10:00:13.441 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T10:00:13.441 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T10:00:13.441 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=1 2024-04-06T10:00:13.441 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=1 2024-04-06T10:00:13.441 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-06T10:00:13.464 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T10:00:13.464 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T10:00:13.464 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T10:00:13.464 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/image_6 2024-04-06T10:00:13.615 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T10:00:13.617 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror image_6 2024-04-06T10:00:13.617 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:00:13.617 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:00:13.617 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_6 2024-04-06T10:00:13.617 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror image_6 '' 2024-04-06T10:00:13.617 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:00:13.617 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:00:13.617 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_6 2024-04-06T10:00:13.617 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T10:00:13.617 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-image_6.status 2024-04-06T10:00:13.618 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-image_6.status 2024-04-06T10:00:13.618 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T10:00:13.619 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-image_6.status 2024-04-06T10:00:13.619 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_6 2024-04-06T10:00:13.619 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_6.status 2024-04-06T10:00:13.670 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 3 2024-04-06T10:00:13.670 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 3 2024-04-06T10:00:13.670 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T10:00:13.670 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=13, tag_tid=2, entry_tid=1], [object_number=12, tag_tid=2, entry_tid=0], [object_number=14, tag_tid=1, entry_tid=15170], [object_number=15, tag_tid=1, entry_tid=15167]]], state=connected] 2024-04-06T10:00:13.670 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=13, tag_tid=2, entry_tid=1], [object_number=12, tag_tid=2, entry_tid=0], [object_number=14, tag_tid=1, entry_tid=15170], [object_number=15, tag_tid=1, entry_tid=15167]]], state=connected] 2024-04-06T10:00:13.674 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_6.status 2024-04-06T10:00:13.675 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=13, tag_tid=2, entry_tid=1' 2024-04-06T10:00:13.676 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror image_6 2024-04-06T10:00:13.676 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:00:13.676 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:00:13.676 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_6 2024-04-06T10:00:13.676 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror image_6 '..*' 2024-04-06T10:00:13.677 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:00:13.677 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:00:13.677 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_6 2024-04-06T10:00:13.677 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T10:00:13.677 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-image_6.status 2024-04-06T10:00:13.677 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-image_6.status 2024-04-06T10:00:13.677 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T10:00:13.678 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-image_6.status 2024-04-06T10:00:13.678 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_6 2024-04-06T10:00:13.679 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_6.status 2024-04-06T10:00:13.928 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 3 2024-04-06T10:00:13.929 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 3 2024-04-06T10:00:13.929 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T10:00:13.929 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=13, tag_tid=2, entry_tid=1], [object_number=12, tag_tid=2, entry_tid=0], [object_number=14, tag_tid=1, entry_tid=15170], [object_number=15, tag_tid=1, entry_tid=15167]]], state=connected] 2024-04-06T10:00:13.929 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=13, tag_tid=2, entry_tid=1], [object_number=12, tag_tid=2, entry_tid=0], [object_number=14, tag_tid=1, entry_tid=15170], [object_number=15, tag_tid=1, entry_tid=15167]]], state=connected] 2024-04-06T10:00:13.933 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_6.status 2024-04-06T10:00:13.935 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=13, tag_tid=2, entry_tid=1' 2024-04-06T10:00:13.935 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=13, tag_tid=2, entry_tid=1' -a 'object_number=13, tag_tid=2, entry_tid=1' = 'object_number=13, tag_tid=2, entry_tid=1' 2024-04-06T10:00:13.935 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T10:00:13.935 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_snap_present cluster1 mirror image_6 snap 2024-04-06T10:00:13.935 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T10:00:13.935 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:00:13.935 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_6 2024-04-06T10:00:13.935 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local snap_name=snap 2024-04-06T10:00:13.935 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local s 2024-04-06T10:00:13.935 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-04-06T10:00:13.936 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 1 2024-04-06T10:00:14.936 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 info mirror/image_6@snap 2024-04-06T10:00:14.999 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:rbd image 'image_6': 2024-04-06T10:00:15.000 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: size 128 MiB in 32 objects 2024-04-06T10:00:15.000 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: order 22 (4 MiB objects) 2024-04-06T10:00:15.000 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: snapshot_count: 1 2024-04-06T10:00:15.000 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: id: 10ac325cbb01 2024-04-06T10:00:15.000 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: block_name_prefix: rbd_data.10ac325cbb01 2024-04-06T10:00:15.000 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: format: 2 2024-04-06T10:00:15.000 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: features: layering, exclusive-lock, journaling 2024-04-06T10:00:15.000 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: op_features: 2024-04-06T10:00:15.000 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: flags: 2024-04-06T10:00:15.000 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: create_timestamp: Sat Apr 6 09:51:34 2024 2024-04-06T10:00:15.000 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: access_timestamp: Sat Apr 6 09:51:34 2024 2024-04-06T10:00:15.000 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: modify_timestamp: Sat Apr 6 09:52:34 2024 2024-04-06T10:00:15.001 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: protected: False 2024-04-06T10:00:15.001 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: journal: 10ac325cbb01 2024-04-06T10:00:15.001 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: mirroring state: enabled 2024-04-06T10:00:15.001 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: mirroring mode: journal 2024-04-06T10:00:15.001 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: mirroring global id: fb8976a9-b6cd-417c-aafb-723f5c63ac98 2024-04-06T10:00:15.001 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: mirroring primary: false 2024-04-06T10:00:15.004 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T10:00:15.005 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ compare_image_snaps mirror image_6 snap 2024-04-06T10:00:15.005 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:00:15.005 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_6 2024-04-06T10:00:15.005 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local snap_name=snap 2024-04-06T10:00:15.005 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local ret=0 2024-04-06T10:00:15.005 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local rmt_export=/tmp/tmp.BhW6xKmROU/cluster2-mirror-image_6.export 2024-04-06T10:00:15.005 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loc_export=/tmp/tmp.BhW6xKmROU/cluster1-mirror-image_6.export 2024-04-06T10:00:15.005 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rm -f /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_6.export /tmp/tmp.BhW6xKmROU/cluster1-mirror-image_6.export 2024-04-06T10:00:15.005 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster2 -p mirror export image_6@snap /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_6.export 2024-04-06T10:00:15.179 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-04-06T10:00:15.187 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 -p mirror export image_6@snap /tmp/tmp.BhW6xKmROU/cluster1-mirror-image_6.export 2024-04-06T10:00:15.638 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-04-06T10:00:15.652 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmp /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_6.export /tmp/tmp.BhW6xKmROU/cluster1-mirror-image_6.export 2024-04-06T10:00:15.732 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rm -f /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_6.export /tmp/tmp.BhW6xKmROU/cluster1-mirror-image_6.export 2024-04-06T10:00:15.767 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T10:00:15.767 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-06T10:00:15.767 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ image=image_7 2024-04-06T10:00:15.767 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ create_snap cluster2 mirror image_7 snap 2024-04-06T10:00:15.768 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-04-06T10:00:15.768 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:00:15.768 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_7 2024-04-06T10:00:15.768 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local snap_name=snap 2024-04-06T10:00:15.768 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster2 -p mirror snap create image_7@snap --debug-rbd=20 --debug-journaler=20 2024-04-06T10:00:18.854 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_image_replay_started cluster1 mirror image_7 2024-04-06T10:00:18.854 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T10:00:18.854 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:00:18.854 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_7 2024-04-06T10:00:18.854 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_image_replay_state cluster1 mirror image_7 started 2024-04-06T10:00:18.854 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T10:00:18.854 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:00:18.854 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_7 2024-04-06T10:00:18.855 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state=started 2024-04-06T10:00:18.855 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local s 2024-04-06T10:00:18.855 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T10:00:18.855 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 1 2024-04-06T10:00:19.856 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test_image_replay_state cluster1 mirror image_7 started 2024-04-06T10:00:19.857 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T10:00:19.857 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:00:19.857 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_7 2024-04-06T10:00:19.857 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local test_state=started 2024-04-06T10:00:19.857 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local status_result 2024-04-06T10:00:19.857 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local current_state=stopped 2024-04-06T10:00:19.859 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_7 2024-04-06T10:00:19.859 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_instance=cluster1 2024-04-06T10:00:19.859 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ shift 2024-04-06T10:00:19.859 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T10:00:19.860 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance=cluster1 2024-04-06T10:00:19.860 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local loop_instance 2024-04-06T10:00:19.860 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T10:00:19.860 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sleep 0 2024-04-06T10:00:19.860 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -i state 2024-04-06T10:00:19.860 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-04-06T10:00:19.860 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ seq 0 3 2024-04-06T10:00:19.865 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T10:00:19.865 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_7 2024-04-06T10:00:19.865 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1:0 2024-04-06T10:00:19.865 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ shift 2024-04-06T10:00:19.865 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T10:00:19.865 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-04-06T10:00:19.866 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T10:00:19.866 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=cluster 2024-04-06T10:00:19.866 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T10:00:19.866 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:00:19.866 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T10:00:19.866 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T10:00:19.866 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval cluster=cluster1 2024-04-06T10:00:19.867 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ cluster=cluster1 2024-04-06T10:00:19.867 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T10:00:19.867 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T10:00:19.867 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-04-06T10:00:19.867 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local local_cluster=cluster1:0 2024-04-06T10:00:19.867 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local cluster=cluster1 2024-04-06T10:00:19.867 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local instance 2024-04-06T10:00:19.868 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T10:00:19.868 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local val=cluster1:0 2024-04-06T10:00:19.868 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local cluster_var_name=local_cluster 2024-04-06T10:00:19.868 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local instance_var_name=instance 2024-04-06T10:00:19.868 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ cluster=cluster1 2024-04-06T10:00:19.868 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T10:00:19.868 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ '[' 0 = cluster1:0 ']' 2024-04-06T10:00:19.869 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ eval local_cluster=cluster1 2024-04-06T10:00:19.869 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++++ local_cluster=cluster1 2024-04-06T10:00:19.869 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ eval instance=0 2024-04-06T10:00:19.869 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++++ instance=0 2024-04-06T10:00:19.869 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T10:00:19.889 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T10:00:19.890 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T10:00:19.890 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T10:00:19.890 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_7 2024-04-06T10:00:19.996 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:Invalid command: unused arguments: ['mirror/image_7'] 2024-04-06T10:00:19.996 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:rbd mirror status : get status for rbd mirror 2024-04-06T10:00:19.996 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:admin_socket: invalid command 2024-04-06T10:00:19.998 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T10:00:19.998 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_7 2024-04-06T10:00:19.998 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1:1 2024-04-06T10:00:19.998 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ shift 2024-04-06T10:00:19.998 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T10:00:19.999 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-04-06T10:00:19.999 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:1 2024-04-06T10:00:19.999 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=cluster 2024-04-06T10:00:19.999 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T10:00:19.999 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:00:19.999 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=1 2024-04-06T10:00:19.999 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-06T10:00:19.999 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval cluster=cluster1 2024-04-06T10:00:19.999 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ cluster=cluster1 2024-04-06T10:00:19.999 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=1 2024-04-06T10:00:19.999 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=1 2024-04-06T10:00:20.001 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-04-06T10:00:20.001 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local local_cluster=cluster1:1 2024-04-06T10:00:20.001 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local cluster=cluster1 2024-04-06T10:00:20.002 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local instance 2024-04-06T10:00:20.002 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-06T10:00:20.002 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local val=cluster1:1 2024-04-06T10:00:20.002 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local cluster_var_name=local_cluster 2024-04-06T10:00:20.002 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local instance_var_name=instance 2024-04-06T10:00:20.002 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ cluster=cluster1 2024-04-06T10:00:20.002 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=1 2024-04-06T10:00:20.002 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ '[' 1 = cluster1:1 ']' 2024-04-06T10:00:20.002 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ eval local_cluster=cluster1 2024-04-06T10:00:20.002 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++++ local_cluster=cluster1 2024-04-06T10:00:20.002 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ eval instance=1 2024-04-06T10:00:20.002 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++++ instance=1 2024-04-06T10:00:20.002 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-06T10:00:20.019 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T10:00:20.019 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T10:00:20.019 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T10:00:20.019 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_7 2024-04-06T10:00:20.169 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ return 0 2024-04-06T10:00:20.171 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ status_result=' "state": "Replaying"' 2024-04-06T10:00:20.171 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ echo ' "state": "Replaying"' 2024-04-06T10:00:20.171 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep -i Replaying 2024-04-06T10:00:20.172 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: "state": "Replaying" 2024-04-06T10:00:20.174 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ current_state=started 2024-04-06T10:00:20.174 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test started = started 2024-04-06T10:00:20.174 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T10:00:20.174 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_7 2024-04-06T10:00:20.174 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local local_cluster=cluster1 2024-04-06T10:00:20.174 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-04-06T10:00:20.174 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:00:20.175 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_7 2024-04-06T10:00:20.175 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' journal = journal ']' 2024-04-06T10:00:20.175 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_journal_replay_complete cluster1 cluster2 mirror image_7 2024-04-06T10:00:20.175 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local local_cluster=cluster1 2024-04-06T10:00:20.175 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-04-06T10:00:20.175 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:00:20.175 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_7 2024-04-06T10:00:20.175 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local s master_pos mirror_pos last_mirror_pos 2024-04-06T10:00:20.175 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local master_tag master_entry mirror_tag mirror_entry 2024-04-06T10:00:20.175 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T10:00:20.175 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T10:00:20.175 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T10:00:20.376 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror image_7 2024-04-06T10:00:20.376 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T10:00:20.376 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:00:20.376 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_7 2024-04-06T10:00:20.376 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T10:00:20.376 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n image_7 ']' 2024-04-06T10:00:20.376 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/image_7' 2024-04-06T10:00:20.376 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_7 2024-04-06T10:00:20.376 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T10:00:20.376 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T10:00:20.377 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T10:00:20.377 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T10:00:20.377 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T10:00:20.377 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T10:00:20.377 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T10:00:20.377 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T10:00:20.377 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T10:00:20.380 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T10:00:20.380 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_7 2024-04-06T10:00:20.381 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T10:00:20.381 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T10:00:20.381 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T10:00:20.381 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T10:00:20.381 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T10:00:20.381 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T10:00:20.381 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T10:00:20.381 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T10:00:20.381 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T10:00:20.381 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T10:00:20.381 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T10:00:20.381 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:00:20.381 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T10:00:20.382 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T10:00:20.382 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T10:00:20.382 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T10:00:20.382 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T10:00:20.383 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T10:00:20.383 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T10:00:20.383 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T10:00:20.383 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T10:00:20.383 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T10:00:20.383 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:00:20.383 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T10:00:20.383 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T10:00:20.383 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T10:00:20.383 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T10:00:20.383 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T10:00:20.383 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T10:00:20.384 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T10:00:20.405 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T10:00:20.406 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T10:00:20.406 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T10:00:20.406 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_7 2024-04-06T10:00:20.518 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:Invalid command: unused arguments: ['mirror/image_7'] 2024-04-06T10:00:20.518 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:rbd mirror flush : flush rbd mirror 2024-04-06T10:00:20.518 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:admin_socket: invalid command 2024-04-06T10:00:20.520 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T10:00:20.520 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/image_7 2024-04-06T10:00:20.520 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:1 2024-04-06T10:00:20.520 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T10:00:20.520 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T10:00:20.520 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-06T10:00:20.520 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:1 2024-04-06T10:00:20.520 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T10:00:20.520 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T10:00:20.520 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T10:00:20.521 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=1 2024-04-06T10:00:20.521 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-06T10:00:20.521 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T10:00:20.521 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:00:20.521 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=1 2024-04-06T10:00:20.521 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=1 2024-04-06T10:00:20.522 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-06T10:00:20.522 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:1 2024-04-06T10:00:20.522 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T10:00:20.522 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T10:00:20.523 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-06T10:00:20.523 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:1 2024-04-06T10:00:20.523 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T10:00:20.523 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T10:00:20.523 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:00:20.523 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=1 2024-04-06T10:00:20.523 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-06T10:00:20.523 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T10:00:20.523 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T10:00:20.523 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=1 2024-04-06T10:00:20.523 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=1 2024-04-06T10:00:20.523 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-06T10:00:20.541 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T10:00:20.541 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T10:00:20.541 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T10:00:20.541 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/image_7 2024-04-06T10:00:20.684 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T10:00:20.685 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror image_7 2024-04-06T10:00:20.686 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:00:20.686 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:00:20.686 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_7 2024-04-06T10:00:20.686 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror image_7 '' 2024-04-06T10:00:20.686 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:00:20.686 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:00:20.686 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_7 2024-04-06T10:00:20.686 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T10:00:20.687 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-image_7.status 2024-04-06T10:00:20.687 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T10:00:20.687 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-image_7.status 2024-04-06T10:00:20.688 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-image_7.status 2024-04-06T10:00:20.688 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_7 2024-04-06T10:00:20.688 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_7.status 2024-04-06T10:00:20.740 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 3 2024-04-06T10:00:20.740 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 3 2024-04-06T10:00:20.740 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T10:00:20.740 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=13, tag_tid=2, entry_tid=1], [object_number=12, tag_tid=2, entry_tid=0], [object_number=15, tag_tid=1, entry_tid=14631], [object_number=14, tag_tid=1, entry_tid=14630]]], state=connected] 2024-04-06T10:00:20.740 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=12, tag_tid=1, entry_tid=14632], [object_number=15, tag_tid=1, entry_tid=14631], [object_number=14, tag_tid=1, entry_tid=14630], [object_number=13, tag_tid=1, entry_tid=14629]]], state=connected] 2024-04-06T10:00:20.743 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_7.status 2024-04-06T10:00:20.744 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=13, tag_tid=2, entry_tid=1' 2024-04-06T10:00:20.746 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror image_7 2024-04-06T10:00:20.746 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:00:20.746 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:00:20.746 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_7 2024-04-06T10:00:20.746 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror image_7 '..*' 2024-04-06T10:00:20.746 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:00:20.746 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:00:20.746 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_7 2024-04-06T10:00:20.746 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T10:00:20.746 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-image_7.status 2024-04-06T10:00:20.746 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-image_7.status 2024-04-06T10:00:20.747 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T10:00:20.747 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-image_7.status 2024-04-06T10:00:20.747 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_7 2024-04-06T10:00:20.747 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_7.status 2024-04-06T10:00:20.793 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 3 2024-04-06T10:00:20.793 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 3 2024-04-06T10:00:20.793 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T10:00:20.793 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=13, tag_tid=2, entry_tid=1], [object_number=12, tag_tid=2, entry_tid=0], [object_number=15, tag_tid=1, entry_tid=14631], [object_number=14, tag_tid=1, entry_tid=14630]]], state=connected] 2024-04-06T10:00:20.793 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=12, tag_tid=1, entry_tid=14632], [object_number=15, tag_tid=1, entry_tid=14631], [object_number=14, tag_tid=1, entry_tid=14630], [object_number=13, tag_tid=1, entry_tid=14629]]], state=connected] 2024-04-06T10:00:20.796 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_7.status 2024-04-06T10:00:20.798 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=12, tag_tid=1, entry_tid=14632' 2024-04-06T10:00:20.798 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=13, tag_tid=2, entry_tid=1' -a 'object_number=13, tag_tid=2, entry_tid=1' = 'object_number=12, tag_tid=1, entry_tid=14632' 2024-04-06T10:00:20.798 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=12, tag_tid=1, entry_tid=14632' '!=' '' 2024-04-06T10:00:20.798 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T10:00:20.798 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=12, tag_tid=1, entry_tid=14632' = '' 2024-04-06T10:00:20.799 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=12, tag_tid=1, entry_tid=14632' 2024-04-06T10:00:20.799 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=13, tag_tid=2, entry_tid=1' 2024-04-06T10:00:20.799 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T10:00:20.799 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T10:00:20.800 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=2 2024-04-06T10:00:20.800 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=12, tag_tid=1, entry_tid=14632' 2024-04-06T10:00:20.801 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T10:00:20.801 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T10:00:20.801 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=1 2024-04-06T10:00:20.802 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=13, tag_tid=2, entry_tid=1' 2024-04-06T10:00:20.802 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T10:00:20.802 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T10:00:20.803 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T10:00:20.804 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=12, tag_tid=1, entry_tid=14632' 2024-04-06T10:00:20.804 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T10:00:20.804 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T10:00:20.805 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=14632 2024-04-06T10:00:20.805 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 2 = 1 -a 1 -le 14632 2024-04-06T10:00:20.805 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T10:00:20.805 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T10:00:20.805 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T10:00:21.007 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror image_7 2024-04-06T10:00:21.008 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T10:00:21.008 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:00:21.008 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_7 2024-04-06T10:00:21.008 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T10:00:21.008 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n image_7 ']' 2024-04-06T10:00:21.008 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/image_7' 2024-04-06T10:00:21.008 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_7 2024-04-06T10:00:21.008 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T10:00:21.008 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T10:00:21.008 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T10:00:21.008 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T10:00:21.008 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T10:00:21.008 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T10:00:21.008 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T10:00:21.010 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T10:00:21.010 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T10:00:21.010 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T10:00:21.010 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_7 2024-04-06T10:00:21.010 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T10:00:21.010 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T10:00:21.010 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T10:00:21.010 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T10:00:21.010 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T10:00:21.011 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T10:00:21.011 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T10:00:21.011 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T10:00:21.011 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T10:00:21.011 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T10:00:21.011 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T10:00:21.011 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:00:21.011 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T10:00:21.011 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T10:00:21.012 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T10:00:21.012 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T10:00:21.012 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T10:00:21.013 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T10:00:21.013 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T10:00:21.013 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T10:00:21.013 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T10:00:21.013 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T10:00:21.013 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:00:21.013 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T10:00:21.013 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T10:00:21.013 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T10:00:21.013 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T10:00:21.014 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T10:00:21.014 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T10:00:21.014 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T10:00:21.038 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T10:00:21.039 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T10:00:21.039 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T10:00:21.039 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_7 2024-04-06T10:00:21.152 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:Invalid command: unused arguments: ['mirror/image_7'] 2024-04-06T10:00:21.152 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:rbd mirror flush : flush rbd mirror 2024-04-06T10:00:21.152 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:admin_socket: invalid command 2024-04-06T10:00:21.154 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T10:00:21.154 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/image_7 2024-04-06T10:00:21.154 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:1 2024-04-06T10:00:21.154 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T10:00:21.154 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T10:00:21.154 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-06T10:00:21.154 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:1 2024-04-06T10:00:21.154 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T10:00:21.154 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T10:00:21.155 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T10:00:21.155 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=1 2024-04-06T10:00:21.155 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-06T10:00:21.155 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T10:00:21.155 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:00:21.155 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=1 2024-04-06T10:00:21.155 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=1 2024-04-06T10:00:21.156 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-06T10:00:21.156 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:1 2024-04-06T10:00:21.156 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T10:00:21.157 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T10:00:21.157 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-06T10:00:21.157 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:1 2024-04-06T10:00:21.157 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T10:00:21.157 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T10:00:21.157 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:00:21.157 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=1 2024-04-06T10:00:21.157 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-06T10:00:21.157 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T10:00:21.157 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T10:00:21.157 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=1 2024-04-06T10:00:21.157 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=1 2024-04-06T10:00:21.157 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-06T10:00:21.179 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T10:00:21.180 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T10:00:21.180 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T10:00:21.180 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/image_7 2024-04-06T10:00:21.340 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T10:00:21.341 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror image_7 2024-04-06T10:00:21.341 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:00:21.341 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:00:21.341 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_7 2024-04-06T10:00:21.341 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror image_7 '' 2024-04-06T10:00:21.341 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:00:21.341 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:00:21.341 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_7 2024-04-06T10:00:21.342 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T10:00:21.342 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-image_7.status 2024-04-06T10:00:21.343 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-image_7.status 2024-04-06T10:00:21.343 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T10:00:21.345 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-image_7.status 2024-04-06T10:00:21.345 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_7 2024-04-06T10:00:21.346 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_7.status 2024-04-06T10:00:21.393 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 3 2024-04-06T10:00:21.393 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 3 2024-04-06T10:00:21.393 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T10:00:21.393 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=13, tag_tid=2, entry_tid=1], [object_number=12, tag_tid=2, entry_tid=0], [object_number=15, tag_tid=1, entry_tid=14631], [object_number=14, tag_tid=1, entry_tid=14630]]], state=connected] 2024-04-06T10:00:21.393 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=13, tag_tid=2, entry_tid=1], [object_number=12, tag_tid=2, entry_tid=0], [object_number=15, tag_tid=1, entry_tid=14631], [object_number=14, tag_tid=1, entry_tid=14630]]], state=connected] 2024-04-06T10:00:21.397 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_7.status 2024-04-06T10:00:21.398 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=13, tag_tid=2, entry_tid=1' 2024-04-06T10:00:21.399 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror image_7 2024-04-06T10:00:21.399 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:00:21.399 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:00:21.399 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_7 2024-04-06T10:00:21.400 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror image_7 '..*' 2024-04-06T10:00:21.400 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:00:21.400 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:00:21.400 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_7 2024-04-06T10:00:21.400 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T10:00:21.400 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-image_7.status 2024-04-06T10:00:21.400 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-image_7.status 2024-04-06T10:00:21.400 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T10:00:21.401 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-image_7.status 2024-04-06T10:00:21.401 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_7 2024-04-06T10:00:21.401 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_7.status 2024-04-06T10:00:21.446 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 3 2024-04-06T10:00:21.446 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 3 2024-04-06T10:00:21.446 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T10:00:21.446 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=13, tag_tid=2, entry_tid=1], [object_number=12, tag_tid=2, entry_tid=0], [object_number=15, tag_tid=1, entry_tid=14631], [object_number=14, tag_tid=1, entry_tid=14630]]], state=connected] 2024-04-06T10:00:21.447 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=13, tag_tid=2, entry_tid=1], [object_number=12, tag_tid=2, entry_tid=0], [object_number=15, tag_tid=1, entry_tid=14631], [object_number=14, tag_tid=1, entry_tid=14630]]], state=connected] 2024-04-06T10:00:21.450 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_7.status 2024-04-06T10:00:21.452 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=13, tag_tid=2, entry_tid=1' 2024-04-06T10:00:21.452 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=13, tag_tid=2, entry_tid=1' -a 'object_number=13, tag_tid=2, entry_tid=1' = 'object_number=13, tag_tid=2, entry_tid=1' 2024-04-06T10:00:21.452 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T10:00:21.452 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_snap_present cluster1 mirror image_7 snap 2024-04-06T10:00:21.452 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T10:00:21.452 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:00:21.452 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_7 2024-04-06T10:00:21.453 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local snap_name=snap 2024-04-06T10:00:21.453 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local s 2024-04-06T10:00:21.453 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-04-06T10:00:21.453 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 1 2024-04-06T10:00:22.453 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 info mirror/image_7@snap 2024-04-06T10:00:22.521 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:rbd image 'image_7': 2024-04-06T10:00:22.521 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: size 128 MiB in 32 objects 2024-04-06T10:00:22.521 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: order 22 (4 MiB objects) 2024-04-06T10:00:22.521 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: snapshot_count: 1 2024-04-06T10:00:22.521 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: id: 10ac4ab72fb9 2024-04-06T10:00:22.521 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: block_name_prefix: rbd_data.10ac4ab72fb9 2024-04-06T10:00:22.521 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: format: 2 2024-04-06T10:00:22.521 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: features: layering, exclusive-lock, journaling 2024-04-06T10:00:22.521 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: op_features: 2024-04-06T10:00:22.521 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: flags: 2024-04-06T10:00:22.521 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: create_timestamp: Sat Apr 6 09:51:49 2024 2024-04-06T10:00:22.521 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: access_timestamp: Sat Apr 6 09:51:49 2024 2024-04-06T10:00:22.522 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: modify_timestamp: Sat Apr 6 09:52:50 2024 2024-04-06T10:00:22.522 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: protected: False 2024-04-06T10:00:22.522 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: journal: 10ac4ab72fb9 2024-04-06T10:00:22.522 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: mirroring state: enabled 2024-04-06T10:00:22.522 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: mirroring mode: journal 2024-04-06T10:00:22.522 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: mirroring global id: 3f0b2fac-de78-4dc3-ab24-d4b8a954ba4a 2024-04-06T10:00:22.522 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: mirroring primary: false 2024-04-06T10:00:22.526 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T10:00:22.526 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ compare_image_snaps mirror image_7 snap 2024-04-06T10:00:22.526 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:00:22.526 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_7 2024-04-06T10:00:22.526 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local snap_name=snap 2024-04-06T10:00:22.526 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local ret=0 2024-04-06T10:00:22.526 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local rmt_export=/tmp/tmp.BhW6xKmROU/cluster2-mirror-image_7.export 2024-04-06T10:00:22.526 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loc_export=/tmp/tmp.BhW6xKmROU/cluster1-mirror-image_7.export 2024-04-06T10:00:22.526 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rm -f /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_7.export /tmp/tmp.BhW6xKmROU/cluster1-mirror-image_7.export 2024-04-06T10:00:22.527 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster2 -p mirror export image_7@snap /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_7.export 2024-04-06T10:00:22.720 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-04-06T10:00:22.729 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 -p mirror export image_7@snap /tmp/tmp.BhW6xKmROU/cluster1-mirror-image_7.export 2024-04-06T10:00:22.920 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-04-06T10:00:22.914+0000 7f0951d22640 -1 bdev(0x56329d428e00 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 1 2024-04-06T10:00:23.084 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-04-06T10:00:23.095 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmp /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_7.export /tmp/tmp.BhW6xKmROU/cluster1-mirror-image_7.export 2024-04-06T10:00:23.174 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rm -f /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_7.export /tmp/tmp.BhW6xKmROU/cluster1-mirror-image_7.export 2024-04-06T10:00:23.215 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T10:00:23.215 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-06T10:00:23.215 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ image=image_8 2024-04-06T10:00:23.215 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ create_snap cluster2 mirror image_8 snap 2024-04-06T10:00:23.215 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-04-06T10:00:23.215 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:00:23.215 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_8 2024-04-06T10:00:23.216 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local snap_name=snap 2024-04-06T10:00:23.216 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster2 -p mirror snap create image_8@snap --debug-rbd=20 --debug-journaler=20 2024-04-06T10:00:23.818 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_image_replay_started cluster1 mirror image_8 2024-04-06T10:00:23.819 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T10:00:23.819 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:00:23.819 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_8 2024-04-06T10:00:23.819 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_image_replay_state cluster1 mirror image_8 started 2024-04-06T10:00:23.819 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T10:00:23.819 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:00:23.819 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_8 2024-04-06T10:00:23.819 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state=started 2024-04-06T10:00:23.819 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local s 2024-04-06T10:00:23.819 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T10:00:23.819 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 1 2024-04-06T10:00:24.821 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test_image_replay_state cluster1 mirror image_8 started 2024-04-06T10:00:24.821 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T10:00:24.821 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:00:24.821 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_8 2024-04-06T10:00:24.821 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local test_state=started 2024-04-06T10:00:24.821 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local status_result 2024-04-06T10:00:24.821 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local current_state=stopped 2024-04-06T10:00:24.822 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_8 2024-04-06T10:00:24.823 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -i state 2024-04-06T10:00:24.823 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_instance=cluster1 2024-04-06T10:00:24.823 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ shift 2024-04-06T10:00:24.823 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T10:00:24.823 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance=cluster1 2024-04-06T10:00:24.823 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local loop_instance 2024-04-06T10:00:24.823 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T10:00:24.823 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sleep 0 2024-04-06T10:00:24.823 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-04-06T10:00:24.827 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ seq 0 3 2024-04-06T10:00:24.827 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T10:00:24.828 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_8 2024-04-06T10:00:24.828 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1:0 2024-04-06T10:00:24.828 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ shift 2024-04-06T10:00:24.828 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T10:00:24.828 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-04-06T10:00:24.829 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T10:00:24.829 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=cluster 2024-04-06T10:00:24.829 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T10:00:24.829 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:00:24.829 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T10:00:24.829 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T10:00:24.829 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval cluster=cluster1 2024-04-06T10:00:24.829 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ cluster=cluster1 2024-04-06T10:00:24.829 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T10:00:24.829 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T10:00:24.830 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-04-06T10:00:24.830 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local local_cluster=cluster1:0 2024-04-06T10:00:24.830 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local cluster=cluster1 2024-04-06T10:00:24.830 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local instance 2024-04-06T10:00:24.830 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T10:00:24.830 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local val=cluster1:0 2024-04-06T10:00:24.830 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local cluster_var_name=local_cluster 2024-04-06T10:00:24.830 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local instance_var_name=instance 2024-04-06T10:00:24.830 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ cluster=cluster1 2024-04-06T10:00:24.830 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T10:00:24.830 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ '[' 0 = cluster1:0 ']' 2024-04-06T10:00:24.830 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ eval local_cluster=cluster1 2024-04-06T10:00:24.830 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++++ local_cluster=cluster1 2024-04-06T10:00:24.830 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ eval instance=0 2024-04-06T10:00:24.831 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++++ instance=0 2024-04-06T10:00:24.831 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T10:00:24.848 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T10:00:24.849 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T10:00:24.849 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T10:00:24.849 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_8 2024-04-06T10:00:24.955 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:Invalid command: unused arguments: ['mirror/image_8'] 2024-04-06T10:00:24.955 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:rbd mirror status : get status for rbd mirror 2024-04-06T10:00:24.955 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:admin_socket: invalid command 2024-04-06T10:00:24.958 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T10:00:24.958 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_8 2024-04-06T10:00:24.958 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1:1 2024-04-06T10:00:24.959 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ shift 2024-04-06T10:00:24.959 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T10:00:24.959 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-04-06T10:00:24.959 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:1 2024-04-06T10:00:24.959 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=cluster 2024-04-06T10:00:24.959 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T10:00:24.959 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:00:24.959 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=1 2024-04-06T10:00:24.959 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-06T10:00:24.959 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval cluster=cluster1 2024-04-06T10:00:24.959 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ cluster=cluster1 2024-04-06T10:00:24.959 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=1 2024-04-06T10:00:24.959 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=1 2024-04-06T10:00:24.959 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-04-06T10:00:24.960 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local local_cluster=cluster1:1 2024-04-06T10:00:24.960 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local cluster=cluster1 2024-04-06T10:00:24.960 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local instance 2024-04-06T10:00:24.960 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-06T10:00:24.960 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local val=cluster1:1 2024-04-06T10:00:24.960 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local cluster_var_name=local_cluster 2024-04-06T10:00:24.960 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local instance_var_name=instance 2024-04-06T10:00:24.960 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ cluster=cluster1 2024-04-06T10:00:24.960 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=1 2024-04-06T10:00:24.960 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ '[' 1 = cluster1:1 ']' 2024-04-06T10:00:24.960 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ eval local_cluster=cluster1 2024-04-06T10:00:24.960 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++++ local_cluster=cluster1 2024-04-06T10:00:24.960 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ eval instance=1 2024-04-06T10:00:24.960 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++++ instance=1 2024-04-06T10:00:24.960 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-06T10:00:24.983 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T10:00:24.983 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T10:00:24.984 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T10:00:24.984 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_8 2024-04-06T10:00:25.141 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ return 0 2024-04-06T10:00:25.141 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ status_result=' "state": "Replaying"' 2024-04-06T10:00:25.142 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ echo ' "state": "Replaying"' 2024-04-06T10:00:25.142 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep -i Replaying 2024-04-06T10:00:25.143 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: "state": "Replaying" 2024-04-06T10:00:25.145 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ current_state=started 2024-04-06T10:00:25.145 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test started = started 2024-04-06T10:00:25.145 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T10:00:25.145 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_8 2024-04-06T10:00:25.145 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local local_cluster=cluster1 2024-04-06T10:00:25.145 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-04-06T10:00:25.145 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:00:25.145 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_8 2024-04-06T10:00:25.145 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' journal = journal ']' 2024-04-06T10:00:25.145 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_journal_replay_complete cluster1 cluster2 mirror image_8 2024-04-06T10:00:25.146 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local local_cluster=cluster1 2024-04-06T10:00:25.146 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-04-06T10:00:25.146 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:00:25.146 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_8 2024-04-06T10:00:25.146 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local s master_pos mirror_pos last_mirror_pos 2024-04-06T10:00:25.146 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local master_tag master_entry mirror_tag mirror_entry 2024-04-06T10:00:25.146 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T10:00:25.146 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T10:00:25.146 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T10:00:25.346 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror image_8 2024-04-06T10:00:25.346 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T10:00:25.347 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:00:25.347 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_8 2024-04-06T10:00:25.347 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T10:00:25.347 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n image_8 ']' 2024-04-06T10:00:25.347 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/image_8' 2024-04-06T10:00:25.347 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_8 2024-04-06T10:00:25.347 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T10:00:25.347 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T10:00:25.347 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T10:00:25.347 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T10:00:25.347 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T10:00:25.347 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T10:00:25.347 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T10:00:25.348 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T10:00:25.348 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T10:00:25.352 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T10:00:25.352 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_8 2024-04-06T10:00:25.352 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T10:00:25.352 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T10:00:25.352 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T10:00:25.352 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T10:00:25.352 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T10:00:25.352 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T10:00:25.352 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T10:00:25.352 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T10:00:25.353 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T10:00:25.353 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T10:00:25.353 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T10:00:25.353 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:00:25.353 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T10:00:25.353 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T10:00:25.353 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T10:00:25.353 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T10:00:25.353 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T10:00:25.353 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T10:00:25.353 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T10:00:25.353 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T10:00:25.353 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T10:00:25.353 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T10:00:25.353 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:00:25.354 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T10:00:25.354 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T10:00:25.354 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T10:00:25.354 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T10:00:25.354 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T10:00:25.354 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T10:00:25.354 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T10:00:25.376 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T10:00:25.377 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T10:00:25.377 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T10:00:25.377 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_8 2024-04-06T10:00:25.481 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:Invalid command: unused arguments: ['mirror/image_8'] 2024-04-06T10:00:25.481 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:rbd mirror flush : flush rbd mirror 2024-04-06T10:00:25.481 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:admin_socket: invalid command 2024-04-06T10:00:25.484 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T10:00:25.484 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/image_8 2024-04-06T10:00:25.484 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:1 2024-04-06T10:00:25.485 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T10:00:25.485 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T10:00:25.485 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-06T10:00:25.485 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:1 2024-04-06T10:00:25.485 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T10:00:25.485 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T10:00:25.485 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T10:00:25.485 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=1 2024-04-06T10:00:25.485 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-06T10:00:25.485 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T10:00:25.485 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:00:25.485 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=1 2024-04-06T10:00:25.485 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=1 2024-04-06T10:00:25.485 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-06T10:00:25.486 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:1 2024-04-06T10:00:25.486 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T10:00:25.486 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T10:00:25.486 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-06T10:00:25.486 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:1 2024-04-06T10:00:25.486 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T10:00:25.486 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T10:00:25.486 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:00:25.486 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=1 2024-04-06T10:00:25.486 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-06T10:00:25.486 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T10:00:25.486 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T10:00:25.486 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=1 2024-04-06T10:00:25.487 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=1 2024-04-06T10:00:25.487 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-06T10:00:25.504 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T10:00:25.504 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T10:00:25.504 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T10:00:25.504 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/image_8 2024-04-06T10:00:25.640 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T10:00:25.641 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror image_8 2024-04-06T10:00:25.641 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:00:25.642 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:00:25.642 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_8 2024-04-06T10:00:25.642 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror image_8 '' 2024-04-06T10:00:25.642 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:00:25.642 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:00:25.642 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_8 2024-04-06T10:00:25.642 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T10:00:25.642 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-image_8.status 2024-04-06T10:00:25.642 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-image_8.status 2024-04-06T10:00:25.643 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T10:00:25.644 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-image_8.status 2024-04-06T10:00:25.644 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_8 2024-04-06T10:00:25.644 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_8.status 2024-04-06T10:00:25.688 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 0 2024-04-06T10:00:25.688 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 0 2024-04-06T10:00:25.688 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T10:00:25.688 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=1, tag_tid=3, entry_tid=1], [object_number=0, tag_tid=3, entry_tid=0], [object_number=3, tag_tid=2, entry_tid=3], [object_number=2, tag_tid=2, entry_tid=2]]], state=connected] 2024-04-06T10:00:25.688 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=3, tag_tid=2, entry_tid=3], [object_number=2, tag_tid=2, entry_tid=2], [object_number=1, tag_tid=2, entry_tid=1], [object_number=0, tag_tid=2, entry_tid=0]]], state=connected] 2024-04-06T10:00:25.691 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_8.status 2024-04-06T10:00:25.693 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=1, tag_tid=3, entry_tid=1' 2024-04-06T10:00:25.694 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror image_8 2024-04-06T10:00:25.694 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:00:25.694 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:00:25.694 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_8 2024-04-06T10:00:25.694 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror image_8 '..*' 2024-04-06T10:00:25.694 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:00:25.694 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:00:25.694 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_8 2024-04-06T10:00:25.694 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T10:00:25.694 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-image_8.status 2024-04-06T10:00:25.695 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-image_8.status 2024-04-06T10:00:25.695 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T10:00:25.695 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-image_8.status 2024-04-06T10:00:25.695 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_8 2024-04-06T10:00:25.696 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_8.status 2024-04-06T10:00:25.740 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 0 2024-04-06T10:00:25.740 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 0 2024-04-06T10:00:25.740 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T10:00:25.740 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=1, tag_tid=3, entry_tid=1], [object_number=0, tag_tid=3, entry_tid=0], [object_number=3, tag_tid=2, entry_tid=3], [object_number=2, tag_tid=2, entry_tid=2]]], state=connected] 2024-04-06T10:00:25.740 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=3, tag_tid=2, entry_tid=3], [object_number=2, tag_tid=2, entry_tid=2], [object_number=1, tag_tid=2, entry_tid=1], [object_number=0, tag_tid=2, entry_tid=0]]], state=connected] 2024-04-06T10:00:25.743 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_8.status 2024-04-06T10:00:25.745 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=3, tag_tid=2, entry_tid=3' 2024-04-06T10:00:25.745 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=1, tag_tid=3, entry_tid=1' -a 'object_number=1, tag_tid=3, entry_tid=1' = 'object_number=3, tag_tid=2, entry_tid=3' 2024-04-06T10:00:25.745 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=3, tag_tid=2, entry_tid=3' '!=' '' 2024-04-06T10:00:25.746 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T10:00:25.746 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=3, tag_tid=2, entry_tid=3' = '' 2024-04-06T10:00:25.746 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=3, tag_tid=2, entry_tid=3' 2024-04-06T10:00:25.746 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=1, tag_tid=3, entry_tid=1' 2024-04-06T10:00:25.746 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T10:00:25.746 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T10:00:25.746 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=3 2024-04-06T10:00:25.747 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=3, tag_tid=2, entry_tid=3' 2024-04-06T10:00:25.747 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T10:00:25.747 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T10:00:25.748 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=2 2024-04-06T10:00:25.749 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=1, tag_tid=3, entry_tid=1' 2024-04-06T10:00:25.749 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T10:00:25.749 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T10:00:25.750 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T10:00:25.751 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=3, tag_tid=2, entry_tid=3' 2024-04-06T10:00:25.751 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T10:00:25.751 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T10:00:25.752 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=3 2024-04-06T10:00:25.752 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 3 = 2 -a 1 -le 3 2024-04-06T10:00:25.752 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T10:00:25.752 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T10:00:25.752 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T10:00:25.954 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror image_8 2024-04-06T10:00:25.955 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T10:00:25.955 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:00:25.955 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_8 2024-04-06T10:00:25.955 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T10:00:25.955 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n image_8 ']' 2024-04-06T10:00:25.955 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/image_8' 2024-04-06T10:00:25.955 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_8 2024-04-06T10:00:25.955 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T10:00:25.956 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T10:00:25.956 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T10:00:25.956 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T10:00:25.956 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T10:00:25.956 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T10:00:25.956 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T10:00:25.959 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T10:00:25.959 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T10:00:25.959 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T10:00:25.959 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_8 2024-04-06T10:00:25.959 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T10:00:25.959 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T10:00:25.959 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T10:00:25.959 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T10:00:25.960 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T10:00:25.960 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T10:00:25.960 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T10:00:25.960 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T10:00:25.960 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T10:00:25.960 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T10:00:25.960 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T10:00:25.960 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:00:25.960 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T10:00:25.960 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T10:00:25.961 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T10:00:25.961 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T10:00:25.961 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T10:00:25.961 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T10:00:25.961 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T10:00:25.961 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T10:00:25.961 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T10:00:25.961 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T10:00:25.962 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:00:25.962 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T10:00:25.962 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T10:00:25.962 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T10:00:25.962 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T10:00:25.962 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T10:00:25.962 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T10:00:25.962 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T10:00:25.984 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T10:00:25.985 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T10:00:25.985 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T10:00:25.985 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_8 2024-04-06T10:00:26.089 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:Invalid command: unused arguments: ['mirror/image_8'] 2024-04-06T10:00:26.089 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:rbd mirror flush : flush rbd mirror 2024-04-06T10:00:26.089 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:admin_socket: invalid command 2024-04-06T10:00:26.091 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T10:00:26.091 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/image_8 2024-04-06T10:00:26.091 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:1 2024-04-06T10:00:26.091 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T10:00:26.091 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T10:00:26.091 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-06T10:00:26.091 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:1 2024-04-06T10:00:26.092 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T10:00:26.092 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T10:00:26.092 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T10:00:26.092 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=1 2024-04-06T10:00:26.092 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-06T10:00:26.092 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T10:00:26.092 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:00:26.092 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=1 2024-04-06T10:00:26.092 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=1 2024-04-06T10:00:26.093 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-06T10:00:26.093 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:1 2024-04-06T10:00:26.093 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T10:00:26.093 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T10:00:26.094 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-06T10:00:26.094 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:1 2024-04-06T10:00:26.094 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T10:00:26.094 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T10:00:26.094 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:00:26.094 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=1 2024-04-06T10:00:26.094 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-06T10:00:26.094 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T10:00:26.094 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T10:00:26.094 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=1 2024-04-06T10:00:26.094 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=1 2024-04-06T10:00:26.094 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-06T10:00:26.112 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T10:00:26.112 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T10:00:26.112 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T10:00:26.112 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/image_8 2024-04-06T10:00:26.255 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T10:00:26.256 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror image_8 2024-04-06T10:00:26.256 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:00:26.256 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:00:26.256 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_8 2024-04-06T10:00:26.256 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror image_8 '' 2024-04-06T10:00:26.256 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:00:26.257 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:00:26.257 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_8 2024-04-06T10:00:26.257 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T10:00:26.257 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-image_8.status 2024-04-06T10:00:26.257 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-image_8.status 2024-04-06T10:00:26.257 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T10:00:26.262 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-image_8.status 2024-04-06T10:00:26.262 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_8 2024-04-06T10:00:26.262 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_8.status 2024-04-06T10:00:26.305 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 0 2024-04-06T10:00:26.305 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 0 2024-04-06T10:00:26.305 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T10:00:26.305 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=1, tag_tid=3, entry_tid=1], [object_number=0, tag_tid=3, entry_tid=0], [object_number=3, tag_tid=2, entry_tid=3], [object_number=2, tag_tid=2, entry_tid=2]]], state=connected] 2024-04-06T10:00:26.305 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=3, tag_tid=2, entry_tid=3], [object_number=2, tag_tid=2, entry_tid=2], [object_number=1, tag_tid=2, entry_tid=1], [object_number=0, tag_tid=2, entry_tid=0]]], state=connected] 2024-04-06T10:00:26.308 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_8.status 2024-04-06T10:00:26.310 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=1, tag_tid=3, entry_tid=1' 2024-04-06T10:00:26.311 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror image_8 2024-04-06T10:00:26.311 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:00:26.311 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:00:26.311 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_8 2024-04-06T10:00:26.311 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror image_8 '..*' 2024-04-06T10:00:26.311 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:00:26.311 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:00:26.311 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_8 2024-04-06T10:00:26.311 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T10:00:26.311 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-image_8.status 2024-04-06T10:00:26.312 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-image_8.status 2024-04-06T10:00:26.312 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T10:00:26.312 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-image_8.status 2024-04-06T10:00:26.312 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_8 2024-04-06T10:00:26.312 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_8.status 2024-04-06T10:00:26.358 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 0 2024-04-06T10:00:26.358 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 0 2024-04-06T10:00:26.358 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T10:00:26.358 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=1, tag_tid=3, entry_tid=1], [object_number=0, tag_tid=3, entry_tid=0], [object_number=3, tag_tid=2, entry_tid=3], [object_number=2, tag_tid=2, entry_tid=2]]], state=connected] 2024-04-06T10:00:26.358 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=3, tag_tid=2, entry_tid=3], [object_number=2, tag_tid=2, entry_tid=2], [object_number=1, tag_tid=2, entry_tid=1], [object_number=0, tag_tid=2, entry_tid=0]]], state=connected] 2024-04-06T10:00:26.361 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_8.status 2024-04-06T10:00:26.363 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=3, tag_tid=2, entry_tid=3' 2024-04-06T10:00:26.363 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=1, tag_tid=3, entry_tid=1' -a 'object_number=1, tag_tid=3, entry_tid=1' = 'object_number=3, tag_tid=2, entry_tid=3' 2024-04-06T10:00:26.363 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=3, tag_tid=2, entry_tid=3' '!=' 'object_number=3, tag_tid=2, entry_tid=3' 2024-04-06T10:00:26.363 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T10:00:26.363 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.4 2024-04-06T10:00:26.765 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror image_8 2024-04-06T10:00:26.765 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T10:00:26.765 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:00:26.765 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_8 2024-04-06T10:00:26.765 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T10:00:26.765 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n image_8 ']' 2024-04-06T10:00:26.765 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/image_8' 2024-04-06T10:00:26.765 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_8 2024-04-06T10:00:26.765 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T10:00:26.766 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T10:00:26.766 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T10:00:26.766 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T10:00:26.766 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T10:00:26.766 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T10:00:26.766 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T10:00:26.766 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T10:00:26.767 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T10:00:26.770 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T10:00:26.770 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_8 2024-04-06T10:00:26.770 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T10:00:26.770 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T10:00:26.770 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T10:00:26.770 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T10:00:26.771 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T10:00:26.771 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T10:00:26.771 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T10:00:26.771 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T10:00:26.771 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T10:00:26.771 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T10:00:26.771 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T10:00:26.771 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:00:26.771 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T10:00:26.771 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T10:00:26.771 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T10:00:26.771 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T10:00:26.771 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T10:00:26.771 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T10:00:26.771 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T10:00:26.772 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T10:00:26.772 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T10:00:26.772 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T10:00:26.772 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:00:26.772 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T10:00:26.772 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T10:00:26.772 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T10:00:26.772 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T10:00:26.772 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T10:00:26.772 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T10:00:26.773 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T10:00:26.797 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T10:00:26.797 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T10:00:26.798 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T10:00:26.798 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_8 2024-04-06T10:00:26.902 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:Invalid command: unused arguments: ['mirror/image_8'] 2024-04-06T10:00:26.902 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:rbd mirror flush : flush rbd mirror 2024-04-06T10:00:26.903 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:admin_socket: invalid command 2024-04-06T10:00:26.905 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T10:00:26.905 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/image_8 2024-04-06T10:00:26.905 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:1 2024-04-06T10:00:26.905 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T10:00:26.905 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T10:00:26.905 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-06T10:00:26.905 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:1 2024-04-06T10:00:26.905 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T10:00:26.905 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T10:00:26.906 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T10:00:26.906 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=1 2024-04-06T10:00:26.906 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-06T10:00:26.906 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T10:00:26.906 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:00:26.906 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=1 2024-04-06T10:00:26.906 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=1 2024-04-06T10:00:26.908 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-06T10:00:26.908 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:1 2024-04-06T10:00:26.908 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T10:00:26.908 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T10:00:26.908 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-06T10:00:26.908 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:1 2024-04-06T10:00:26.908 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T10:00:26.908 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T10:00:26.908 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:00:26.908 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=1 2024-04-06T10:00:26.908 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-06T10:00:26.908 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T10:00:26.909 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T10:00:26.909 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=1 2024-04-06T10:00:26.909 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=1 2024-04-06T10:00:26.909 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-06T10:00:26.931 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T10:00:26.932 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T10:00:26.932 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T10:00:26.932 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/image_8 2024-04-06T10:00:27.082 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T10:00:27.083 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror image_8 2024-04-06T10:00:27.083 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:00:27.083 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:00:27.084 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_8 2024-04-06T10:00:27.084 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror image_8 '' 2024-04-06T10:00:27.084 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:00:27.084 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:00:27.084 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_8 2024-04-06T10:00:27.084 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T10:00:27.084 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-image_8.status 2024-04-06T10:00:27.085 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-image_8.status 2024-04-06T10:00:27.085 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T10:00:27.087 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-image_8.status 2024-04-06T10:00:27.087 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_8 2024-04-06T10:00:27.088 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_8.status 2024-04-06T10:00:27.133 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 0 2024-04-06T10:00:27.133 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 0 2024-04-06T10:00:27.133 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T10:00:27.133 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=1, tag_tid=3, entry_tid=1], [object_number=0, tag_tid=3, entry_tid=0], [object_number=3, tag_tid=2, entry_tid=3], [object_number=2, tag_tid=2, entry_tid=2]]], state=connected] 2024-04-06T10:00:27.133 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=1, tag_tid=3, entry_tid=1], [object_number=0, tag_tid=3, entry_tid=0], [object_number=3, tag_tid=2, entry_tid=3], [object_number=2, tag_tid=2, entry_tid=2]]], state=connected] 2024-04-06T10:00:27.137 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_8.status 2024-04-06T10:00:27.139 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=1, tag_tid=3, entry_tid=1' 2024-04-06T10:00:27.139 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror image_8 2024-04-06T10:00:27.140 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:00:27.140 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:00:27.140 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_8 2024-04-06T10:00:27.140 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror image_8 '..*' 2024-04-06T10:00:27.140 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:00:27.140 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:00:27.140 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_8 2024-04-06T10:00:27.140 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T10:00:27.140 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-image_8.status 2024-04-06T10:00:27.140 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-image_8.status 2024-04-06T10:00:27.140 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T10:00:27.142 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-image_8.status 2024-04-06T10:00:27.142 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_8 2024-04-06T10:00:27.142 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_8.status 2024-04-06T10:00:27.186 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 0 2024-04-06T10:00:27.186 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 0 2024-04-06T10:00:27.186 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T10:00:27.186 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=1, tag_tid=3, entry_tid=1], [object_number=0, tag_tid=3, entry_tid=0], [object_number=3, tag_tid=2, entry_tid=3], [object_number=2, tag_tid=2, entry_tid=2]]], state=connected] 2024-04-06T10:00:27.186 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=1, tag_tid=3, entry_tid=1], [object_number=0, tag_tid=3, entry_tid=0], [object_number=3, tag_tid=2, entry_tid=3], [object_number=2, tag_tid=2, entry_tid=2]]], state=connected] 2024-04-06T10:00:27.216 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_8.status 2024-04-06T10:00:27.217 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=1, tag_tid=3, entry_tid=1' 2024-04-06T10:00:27.217 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=1, tag_tid=3, entry_tid=1' -a 'object_number=1, tag_tid=3, entry_tid=1' = 'object_number=1, tag_tid=3, entry_tid=1' 2024-04-06T10:00:27.218 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T10:00:27.218 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_snap_present cluster1 mirror image_8 snap 2024-04-06T10:00:27.218 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T10:00:27.218 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:00:27.218 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_8 2024-04-06T10:00:27.218 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local snap_name=snap 2024-04-06T10:00:27.218 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local s 2024-04-06T10:00:27.218 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-04-06T10:00:27.218 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 1 2024-04-06T10:00:28.218 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 info mirror/image_8@snap 2024-04-06T10:00:28.283 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:rbd image 'image_8': 2024-04-06T10:00:28.283 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: size 128 MiB in 32 objects 2024-04-06T10:00:28.284 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: order 22 (4 MiB objects) 2024-04-06T10:00:28.284 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: snapshot_count: 1 2024-04-06T10:00:28.284 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: id: 10acc10221f1 2024-04-06T10:00:28.284 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: block_name_prefix: rbd_data.10acc10221f1 2024-04-06T10:00:28.284 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: format: 2 2024-04-06T10:00:28.284 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: features: layering, exclusive-lock, journaling 2024-04-06T10:00:28.284 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: op_features: 2024-04-06T10:00:28.284 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: flags: 2024-04-06T10:00:28.284 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: create_timestamp: Sat Apr 6 09:52:02 2024 2024-04-06T10:00:28.284 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: access_timestamp: Sat Apr 6 09:52:02 2024 2024-04-06T10:00:28.284 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: modify_timestamp: Sat Apr 6 09:53:03 2024 2024-04-06T10:00:28.284 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: protected: False 2024-04-06T10:00:28.284 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: journal: 10acc10221f1 2024-04-06T10:00:28.284 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: mirroring state: enabled 2024-04-06T10:00:28.285 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: mirroring mode: journal 2024-04-06T10:00:28.285 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: mirroring global id: e70e214a-d8aa-47c6-a0b7-54b9a4159ffe 2024-04-06T10:00:28.285 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: mirroring primary: false 2024-04-06T10:00:28.288 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T10:00:28.288 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ compare_image_snaps mirror image_8 snap 2024-04-06T10:00:28.288 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:00:28.289 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_8 2024-04-06T10:00:28.289 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local snap_name=snap 2024-04-06T10:00:28.289 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local ret=0 2024-04-06T10:00:28.289 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local rmt_export=/tmp/tmp.BhW6xKmROU/cluster2-mirror-image_8.export 2024-04-06T10:00:28.289 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loc_export=/tmp/tmp.BhW6xKmROU/cluster1-mirror-image_8.export 2024-04-06T10:00:28.289 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rm -f /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_8.export /tmp/tmp.BhW6xKmROU/cluster1-mirror-image_8.export 2024-04-06T10:00:28.289 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster2 -p mirror export image_8@snap /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_8.export 2024-04-06T10:00:28.438 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-04-06T10:00:28.446 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 -p mirror export image_8@snap /tmp/tmp.BhW6xKmROU/cluster1-mirror-image_8.export 2024-04-06T10:00:28.714 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-04-06T10:00:28.724 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmp /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_8.export /tmp/tmp.BhW6xKmROU/cluster1-mirror-image_8.export 2024-04-06T10:00:28.803 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rm -f /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_8.export /tmp/tmp.BhW6xKmROU/cluster1-mirror-image_8.export 2024-04-06T10:00:28.844 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T10:00:28.844 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-06T10:00:28.844 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ image=image_9 2024-04-06T10:00:28.844 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ create_snap cluster2 mirror image_9 snap 2024-04-06T10:00:28.844 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-04-06T10:00:28.844 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:00:28.844 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_9 2024-04-06T10:00:28.844 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local snap_name=snap 2024-04-06T10:00:28.844 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster2 -p mirror snap create image_9@snap --debug-rbd=20 --debug-journaler=20 2024-04-06T10:00:29.846 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_image_replay_started cluster1 mirror image_9 2024-04-06T10:00:29.847 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T10:00:29.847 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:00:29.847 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_9 2024-04-06T10:00:29.847 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_image_replay_state cluster1 mirror image_9 started 2024-04-06T10:00:29.847 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T10:00:29.847 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:00:29.847 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_9 2024-04-06T10:00:29.847 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state=started 2024-04-06T10:00:29.847 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local s 2024-04-06T10:00:29.847 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T10:00:29.847 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 1 2024-04-06T10:00:30.850 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test_image_replay_state cluster1 mirror image_9 started 2024-04-06T10:00:30.851 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T10:00:30.851 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:00:30.851 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_9 2024-04-06T10:00:30.851 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local test_state=started 2024-04-06T10:00:30.851 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local status_result 2024-04-06T10:00:30.851 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local current_state=stopped 2024-04-06T10:00:30.851 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_9 2024-04-06T10:00:30.851 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -i state 2024-04-06T10:00:30.851 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_instance=cluster1 2024-04-06T10:00:30.852 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ shift 2024-04-06T10:00:30.852 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T10:00:30.852 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance=cluster1 2024-04-06T10:00:30.852 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local loop_instance 2024-04-06T10:00:30.852 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T10:00:30.852 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sleep 0 2024-04-06T10:00:30.856 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-04-06T10:00:30.856 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ seq 0 3 2024-04-06T10:00:30.856 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T10:00:30.856 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_9 2024-04-06T10:00:30.856 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1:0 2024-04-06T10:00:30.856 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ shift 2024-04-06T10:00:30.856 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T10:00:30.856 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-04-06T10:00:30.857 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T10:00:30.857 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=cluster 2024-04-06T10:00:30.857 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T10:00:30.857 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:00:30.857 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T10:00:30.857 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T10:00:30.857 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval cluster=cluster1 2024-04-06T10:00:30.857 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ cluster=cluster1 2024-04-06T10:00:30.857 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T10:00:30.858 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T10:00:30.858 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-04-06T10:00:30.858 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local local_cluster=cluster1:0 2024-04-06T10:00:30.858 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local cluster=cluster1 2024-04-06T10:00:30.858 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local instance 2024-04-06T10:00:30.858 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T10:00:30.858 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local val=cluster1:0 2024-04-06T10:00:30.858 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local cluster_var_name=local_cluster 2024-04-06T10:00:30.858 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local instance_var_name=instance 2024-04-06T10:00:30.859 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ cluster=cluster1 2024-04-06T10:00:30.859 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T10:00:30.859 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ '[' 0 = cluster1:0 ']' 2024-04-06T10:00:30.859 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ eval local_cluster=cluster1 2024-04-06T10:00:30.859 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++++ local_cluster=cluster1 2024-04-06T10:00:30.859 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ eval instance=0 2024-04-06T10:00:30.859 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++++ instance=0 2024-04-06T10:00:30.859 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T10:00:30.880 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T10:00:30.880 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T10:00:30.881 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T10:00:30.881 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_9 2024-04-06T10:00:30.987 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:Invalid command: unused arguments: ['mirror/image_9'] 2024-04-06T10:00:30.987 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:rbd mirror status : get status for rbd mirror 2024-04-06T10:00:30.987 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:admin_socket: invalid command 2024-04-06T10:00:30.989 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T10:00:30.989 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_9 2024-04-06T10:00:30.989 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1:1 2024-04-06T10:00:30.989 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ shift 2024-04-06T10:00:30.989 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T10:00:30.989 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-04-06T10:00:30.990 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:1 2024-04-06T10:00:30.990 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=cluster 2024-04-06T10:00:30.990 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T10:00:30.990 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:00:30.990 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=1 2024-04-06T10:00:30.990 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-06T10:00:30.990 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval cluster=cluster1 2024-04-06T10:00:30.990 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ cluster=cluster1 2024-04-06T10:00:30.990 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=1 2024-04-06T10:00:30.990 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=1 2024-04-06T10:00:30.992 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-04-06T10:00:30.992 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local local_cluster=cluster1:1 2024-04-06T10:00:30.992 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local cluster=cluster1 2024-04-06T10:00:30.992 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local instance 2024-04-06T10:00:30.992 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-06T10:00:30.992 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local val=cluster1:1 2024-04-06T10:00:30.992 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local cluster_var_name=local_cluster 2024-04-06T10:00:30.992 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local instance_var_name=instance 2024-04-06T10:00:30.992 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ cluster=cluster1 2024-04-06T10:00:30.992 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=1 2024-04-06T10:00:30.992 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ '[' 1 = cluster1:1 ']' 2024-04-06T10:00:30.992 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ eval local_cluster=cluster1 2024-04-06T10:00:30.992 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++++ local_cluster=cluster1 2024-04-06T10:00:30.992 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ eval instance=1 2024-04-06T10:00:30.993 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++++ instance=1 2024-04-06T10:00:30.993 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-06T10:00:31.015 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T10:00:31.015 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T10:00:31.015 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T10:00:31.015 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_9 2024-04-06T10:00:31.172 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ return 0 2024-04-06T10:00:31.173 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ status_result=' "state": "Replaying"' 2024-04-06T10:00:31.173 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ echo ' "state": "Replaying"' 2024-04-06T10:00:31.173 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep -i Replaying 2024-04-06T10:00:31.176 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: "state": "Replaying" 2024-04-06T10:00:31.178 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ current_state=started 2024-04-06T10:00:31.178 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test started = started 2024-04-06T10:00:31.178 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T10:00:31.178 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_9 2024-04-06T10:00:31.178 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local local_cluster=cluster1 2024-04-06T10:00:31.178 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-04-06T10:00:31.178 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:00:31.178 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_9 2024-04-06T10:00:31.178 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' journal = journal ']' 2024-04-06T10:00:31.179 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_journal_replay_complete cluster1 cluster2 mirror image_9 2024-04-06T10:00:31.179 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local local_cluster=cluster1 2024-04-06T10:00:31.179 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-04-06T10:00:31.179 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:00:31.179 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_9 2024-04-06T10:00:31.179 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local s master_pos mirror_pos last_mirror_pos 2024-04-06T10:00:31.179 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local master_tag master_entry mirror_tag mirror_entry 2024-04-06T10:00:31.179 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T10:00:31.179 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T10:00:31.179 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T10:00:31.379 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror image_9 2024-04-06T10:00:31.379 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T10:00:31.379 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:00:31.379 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_9 2024-04-06T10:00:31.379 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T10:00:31.380 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n image_9 ']' 2024-04-06T10:00:31.380 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/image_9' 2024-04-06T10:00:31.380 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_9 2024-04-06T10:00:31.380 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T10:00:31.380 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T10:00:31.380 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T10:00:31.380 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T10:00:31.380 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T10:00:31.380 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T10:00:31.380 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T10:00:31.381 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T10:00:31.381 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T10:00:31.383 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T10:00:31.383 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_9 2024-04-06T10:00:31.383 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T10:00:31.383 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T10:00:31.383 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T10:00:31.384 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T10:00:31.384 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T10:00:31.384 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T10:00:31.384 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T10:00:31.384 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T10:00:31.384 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T10:00:31.384 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T10:00:31.385 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T10:00:31.385 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:00:31.385 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T10:00:31.385 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T10:00:31.385 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T10:00:31.385 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T10:00:31.385 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T10:00:31.385 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T10:00:31.385 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T10:00:31.385 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T10:00:31.385 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T10:00:31.385 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T10:00:31.385 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:00:31.386 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T10:00:31.386 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T10:00:31.386 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T10:00:31.386 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T10:00:31.386 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T10:00:31.386 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T10:00:31.386 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T10:00:31.409 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T10:00:31.409 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T10:00:31.409 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T10:00:31.409 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_9 2024-04-06T10:00:31.515 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:Invalid command: unused arguments: ['mirror/image_9'] 2024-04-06T10:00:31.515 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:rbd mirror flush : flush rbd mirror 2024-04-06T10:00:31.515 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:admin_socket: invalid command 2024-04-06T10:00:31.518 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T10:00:31.518 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/image_9 2024-04-06T10:00:31.518 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:1 2024-04-06T10:00:31.519 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T10:00:31.519 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T10:00:31.519 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-06T10:00:31.519 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:1 2024-04-06T10:00:31.519 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T10:00:31.519 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T10:00:31.519 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T10:00:31.519 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=1 2024-04-06T10:00:31.519 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-06T10:00:31.519 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T10:00:31.519 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:00:31.519 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=1 2024-04-06T10:00:31.519 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=1 2024-04-06T10:00:31.519 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-06T10:00:31.520 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:1 2024-04-06T10:00:31.520 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T10:00:31.520 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T10:00:31.520 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-06T10:00:31.520 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:1 2024-04-06T10:00:31.520 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T10:00:31.520 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T10:00:31.520 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:00:31.520 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=1 2024-04-06T10:00:31.520 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-06T10:00:31.520 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T10:00:31.520 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T10:00:31.520 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=1 2024-04-06T10:00:31.520 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=1 2024-04-06T10:00:31.521 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-06T10:00:31.543 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T10:00:31.543 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T10:00:31.543 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T10:00:31.543 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/image_9 2024-04-06T10:00:31.704 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T10:00:31.706 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror image_9 2024-04-06T10:00:31.706 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:00:31.706 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:00:31.706 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_9 2024-04-06T10:00:31.706 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror image_9 '' 2024-04-06T10:00:31.706 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:00:31.706 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:00:31.707 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_9 2024-04-06T10:00:31.707 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T10:00:31.707 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-image_9.status 2024-04-06T10:00:31.707 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-image_9.status 2024-04-06T10:00:31.707 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T10:00:31.709 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-image_9.status 2024-04-06T10:00:31.709 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_9 2024-04-06T10:00:31.709 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_9.status 2024-04-06T10:00:31.759 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 1 2024-04-06T10:00:31.759 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 1 2024-04-06T10:00:31.759 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T10:00:31.759 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=5, tag_tid=3, entry_tid=1], [object_number=4, tag_tid=3, entry_tid=0], [object_number=6, tag_tid=1, entry_tid=6010], [object_number=7, tag_tid=1, entry_tid=6007]]], state=connected] 2024-04-06T10:00:31.760 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=5, tag_tid=3, entry_tid=1], [object_number=4, tag_tid=3, entry_tid=0], [object_number=6, tag_tid=1, entry_tid=6010], [object_number=7, tag_tid=1, entry_tid=6007]]], state=connected] 2024-04-06T10:00:31.763 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_9.status 2024-04-06T10:00:31.764 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=5, tag_tid=3, entry_tid=1' 2024-04-06T10:00:31.765 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror image_9 2024-04-06T10:00:31.766 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:00:31.766 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:00:31.766 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_9 2024-04-06T10:00:31.766 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror image_9 '..*' 2024-04-06T10:00:31.766 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:00:31.766 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:00:31.766 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_9 2024-04-06T10:00:31.766 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T10:00:31.766 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-image_9.status 2024-04-06T10:00:31.766 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-image_9.status 2024-04-06T10:00:31.766 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T10:00:31.767 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-image_9.status 2024-04-06T10:00:31.767 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_9 2024-04-06T10:00:31.767 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_9.status 2024-04-06T10:00:31.813 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 1 2024-04-06T10:00:31.813 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 1 2024-04-06T10:00:31.813 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T10:00:31.813 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=5, tag_tid=3, entry_tid=1], [object_number=4, tag_tid=3, entry_tid=0], [object_number=6, tag_tid=1, entry_tid=6010], [object_number=7, tag_tid=1, entry_tid=6007]]], state=connected] 2024-04-06T10:00:31.813 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=5, tag_tid=3, entry_tid=1], [object_number=4, tag_tid=3, entry_tid=0], [object_number=6, tag_tid=1, entry_tid=6010], [object_number=7, tag_tid=1, entry_tid=6007]]], state=connected] 2024-04-06T10:00:31.816 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_9.status 2024-04-06T10:00:31.818 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=5, tag_tid=3, entry_tid=1' 2024-04-06T10:00:31.819 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=5, tag_tid=3, entry_tid=1' -a 'object_number=5, tag_tid=3, entry_tid=1' = 'object_number=5, tag_tid=3, entry_tid=1' 2024-04-06T10:00:31.819 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T10:00:31.819 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_snap_present cluster1 mirror image_9 snap 2024-04-06T10:00:31.819 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T10:00:31.819 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:00:31.819 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_9 2024-04-06T10:00:31.819 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local snap_name=snap 2024-04-06T10:00:31.819 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local s 2024-04-06T10:00:31.819 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-04-06T10:00:31.819 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 1 2024-04-06T10:00:32.819 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 info mirror/image_9@snap 2024-04-06T10:00:32.887 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:rbd image 'image_9': 2024-04-06T10:00:32.887 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: size 128 MiB in 32 objects 2024-04-06T10:00:32.887 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: order 22 (4 MiB objects) 2024-04-06T10:00:32.887 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: snapshot_count: 1 2024-04-06T10:00:32.887 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: id: 10acfbe9f823 2024-04-06T10:00:32.887 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: block_name_prefix: rbd_data.10acfbe9f823 2024-04-06T10:00:32.887 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: format: 2 2024-04-06T10:00:32.887 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: features: layering, exclusive-lock, journaling 2024-04-06T10:00:32.887 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: op_features: 2024-04-06T10:00:32.887 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: flags: 2024-04-06T10:00:32.888 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: create_timestamp: Sat Apr 6 09:52:06 2024 2024-04-06T10:00:32.888 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: access_timestamp: Sat Apr 6 09:52:06 2024 2024-04-06T10:00:32.888 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: modify_timestamp: Sat Apr 6 09:53:07 2024 2024-04-06T10:00:32.888 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: protected: False 2024-04-06T10:00:32.888 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: journal: 10acfbe9f823 2024-04-06T10:00:32.888 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: mirroring state: enabled 2024-04-06T10:00:32.888 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: mirroring mode: journal 2024-04-06T10:00:32.888 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: mirroring global id: e70b32c3-be14-466d-aae8-e1e58b0d3233 2024-04-06T10:00:32.888 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: mirroring primary: false 2024-04-06T10:00:32.892 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T10:00:32.892 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ compare_image_snaps mirror image_9 snap 2024-04-06T10:00:32.892 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:00:32.892 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_9 2024-04-06T10:00:32.892 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local snap_name=snap 2024-04-06T10:00:32.892 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local ret=0 2024-04-06T10:00:32.892 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local rmt_export=/tmp/tmp.BhW6xKmROU/cluster2-mirror-image_9.export 2024-04-06T10:00:32.893 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loc_export=/tmp/tmp.BhW6xKmROU/cluster1-mirror-image_9.export 2024-04-06T10:00:32.893 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rm -f /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_9.export /tmp/tmp.BhW6xKmROU/cluster1-mirror-image_9.export 2024-04-06T10:00:32.893 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster2 -p mirror export image_9@snap /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_9.export 2024-04-06T10:00:33.067 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-04-06T10:00:33.075 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 -p mirror export image_9@snap /tmp/tmp.BhW6xKmROU/cluster1-mirror-image_9.export 2024-04-06T10:00:33.430 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-04-06T10:00:33.442 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmp /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_9.export /tmp/tmp.BhW6xKmROU/cluster1-mirror-image_9.export 2024-04-06T10:00:33.524 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rm -f /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_9.export /tmp/tmp.BhW6xKmROU/cluster1-mirror-image_9.export 2024-04-06T10:00:33.564 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T10:00:33.564 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-06T10:00:33.564 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ image=image_10 2024-04-06T10:00:33.564 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ create_snap cluster2 mirror image_10 snap 2024-04-06T10:00:33.564 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-04-06T10:00:33.564 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:00:33.564 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_10 2024-04-06T10:00:33.564 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local snap_name=snap 2024-04-06T10:00:33.565 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster2 -p mirror snap create image_10@snap --debug-rbd=20 --debug-journaler=20 2024-04-06T10:00:35.855 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_image_replay_started cluster1 mirror image_10 2024-04-06T10:00:35.855 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T10:00:35.856 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:00:35.856 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_10 2024-04-06T10:00:35.856 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_image_replay_state cluster1 mirror image_10 started 2024-04-06T10:00:35.856 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T10:00:35.856 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:00:35.856 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_10 2024-04-06T10:00:35.856 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state=started 2024-04-06T10:00:35.856 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local s 2024-04-06T10:00:35.856 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T10:00:35.856 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 1 2024-04-06T10:00:36.857 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test_image_replay_state cluster1 mirror image_10 started 2024-04-06T10:00:36.857 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T10:00:36.857 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:00:36.857 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_10 2024-04-06T10:00:36.857 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local test_state=started 2024-04-06T10:00:36.857 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local status_result 2024-04-06T10:00:36.857 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local current_state=stopped 2024-04-06T10:00:36.858 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_10 2024-04-06T10:00:36.859 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -i state 2024-04-06T10:00:36.859 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_instance=cluster1 2024-04-06T10:00:36.859 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ shift 2024-04-06T10:00:36.859 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T10:00:36.859 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance=cluster1 2024-04-06T10:00:36.859 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local loop_instance 2024-04-06T10:00:36.859 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T10:00:36.859 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sleep 0 2024-04-06T10:00:36.860 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-04-06T10:00:36.860 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ seq 0 3 2024-04-06T10:00:36.863 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T10:00:36.864 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_10 2024-04-06T10:00:36.864 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1:0 2024-04-06T10:00:36.864 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ shift 2024-04-06T10:00:36.864 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T10:00:36.864 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-04-06T10:00:36.864 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T10:00:36.864 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=cluster 2024-04-06T10:00:36.864 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T10:00:36.864 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:00:36.864 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T10:00:36.864 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T10:00:36.864 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval cluster=cluster1 2024-04-06T10:00:36.864 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ cluster=cluster1 2024-04-06T10:00:36.865 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T10:00:36.865 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T10:00:36.865 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-04-06T10:00:36.865 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local local_cluster=cluster1:0 2024-04-06T10:00:36.865 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local cluster=cluster1 2024-04-06T10:00:36.865 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local instance 2024-04-06T10:00:36.865 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T10:00:36.865 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local val=cluster1:0 2024-04-06T10:00:36.865 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local cluster_var_name=local_cluster 2024-04-06T10:00:36.865 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local instance_var_name=instance 2024-04-06T10:00:36.865 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ cluster=cluster1 2024-04-06T10:00:36.865 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T10:00:36.865 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ '[' 0 = cluster1:0 ']' 2024-04-06T10:00:36.865 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ eval local_cluster=cluster1 2024-04-06T10:00:36.865 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++++ local_cluster=cluster1 2024-04-06T10:00:36.866 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ eval instance=0 2024-04-06T10:00:36.866 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++++ instance=0 2024-04-06T10:00:36.866 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T10:00:36.885 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T10:00:36.885 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T10:00:36.885 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T10:00:36.886 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_10 2024-04-06T10:00:36.987 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:Invalid command: unused arguments: ['mirror/image_10'] 2024-04-06T10:00:36.988 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:rbd mirror status : get status for rbd mirror 2024-04-06T10:00:36.988 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:admin_socket: invalid command 2024-04-06T10:00:36.991 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T10:00:36.992 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_10 2024-04-06T10:00:36.992 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1:1 2024-04-06T10:00:36.992 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ shift 2024-04-06T10:00:36.992 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T10:00:36.992 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-04-06T10:00:36.992 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:1 2024-04-06T10:00:36.992 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=cluster 2024-04-06T10:00:36.992 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T10:00:36.992 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:00:36.992 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=1 2024-04-06T10:00:36.992 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-06T10:00:36.992 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval cluster=cluster1 2024-04-06T10:00:36.992 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ cluster=cluster1 2024-04-06T10:00:36.993 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=1 2024-04-06T10:00:36.993 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=1 2024-04-06T10:00:36.993 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-04-06T10:00:36.993 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local local_cluster=cluster1:1 2024-04-06T10:00:36.993 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local cluster=cluster1 2024-04-06T10:00:36.993 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local instance 2024-04-06T10:00:36.993 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-06T10:00:36.993 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local val=cluster1:1 2024-04-06T10:00:36.993 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local cluster_var_name=local_cluster 2024-04-06T10:00:36.993 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local instance_var_name=instance 2024-04-06T10:00:36.993 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ cluster=cluster1 2024-04-06T10:00:36.993 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=1 2024-04-06T10:00:36.993 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ '[' 1 = cluster1:1 ']' 2024-04-06T10:00:36.993 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ eval local_cluster=cluster1 2024-04-06T10:00:36.993 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++++ local_cluster=cluster1 2024-04-06T10:00:36.994 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ eval instance=1 2024-04-06T10:00:36.994 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++++ instance=1 2024-04-06T10:00:36.994 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-06T10:00:37.011 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T10:00:37.012 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T10:00:37.012 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T10:00:37.012 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_10 2024-04-06T10:00:37.154 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ return 0 2024-04-06T10:00:37.154 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ status_result=' "state": "Replaying"' 2024-04-06T10:00:37.155 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ echo ' "state": "Replaying"' 2024-04-06T10:00:37.155 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep -i Replaying 2024-04-06T10:00:37.156 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: "state": "Replaying" 2024-04-06T10:00:37.158 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ current_state=started 2024-04-06T10:00:37.158 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test started = started 2024-04-06T10:00:37.158 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T10:00:37.158 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_10 2024-04-06T10:00:37.159 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local local_cluster=cluster1 2024-04-06T10:00:37.159 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-04-06T10:00:37.159 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:00:37.159 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_10 2024-04-06T10:00:37.159 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' journal = journal ']' 2024-04-06T10:00:37.159 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_journal_replay_complete cluster1 cluster2 mirror image_10 2024-04-06T10:00:37.159 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local local_cluster=cluster1 2024-04-06T10:00:37.159 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-04-06T10:00:37.159 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:00:37.159 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_10 2024-04-06T10:00:37.159 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local s master_pos mirror_pos last_mirror_pos 2024-04-06T10:00:37.159 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local master_tag master_entry mirror_tag mirror_entry 2024-04-06T10:00:37.159 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T10:00:37.159 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T10:00:37.159 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T10:00:37.360 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror image_10 2024-04-06T10:00:37.360 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T10:00:37.360 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:00:37.360 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_10 2024-04-06T10:00:37.360 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T10:00:37.360 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n image_10 ']' 2024-04-06T10:00:37.360 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/image_10' 2024-04-06T10:00:37.360 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_10 2024-04-06T10:00:37.360 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T10:00:37.360 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T10:00:37.360 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T10:00:37.361 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T10:00:37.361 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T10:00:37.361 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T10:00:37.361 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T10:00:37.361 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T10:00:37.361 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T10:00:37.364 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T10:00:37.365 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_10 2024-04-06T10:00:37.365 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T10:00:37.365 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T10:00:37.365 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T10:00:37.365 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T10:00:37.365 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T10:00:37.365 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T10:00:37.365 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T10:00:37.365 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T10:00:37.365 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T10:00:37.365 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T10:00:37.365 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T10:00:37.365 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:00:37.365 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T10:00:37.366 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T10:00:37.366 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T10:00:37.366 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T10:00:37.366 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T10:00:37.366 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T10:00:37.366 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T10:00:37.366 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T10:00:37.366 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T10:00:37.366 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T10:00:37.366 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:00:37.366 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T10:00:37.366 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T10:00:37.366 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T10:00:37.366 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T10:00:37.367 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T10:00:37.367 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T10:00:37.367 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T10:00:37.388 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T10:00:37.389 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T10:00:37.389 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T10:00:37.389 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_10 2024-04-06T10:00:37.485 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:Invalid command: unused arguments: ['mirror/image_10'] 2024-04-06T10:00:37.485 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:rbd mirror flush : flush rbd mirror 2024-04-06T10:00:37.486 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:admin_socket: invalid command 2024-04-06T10:00:37.489 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T10:00:37.489 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/image_10 2024-04-06T10:00:37.489 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:1 2024-04-06T10:00:37.489 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T10:00:37.489 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T10:00:37.489 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-06T10:00:37.489 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:1 2024-04-06T10:00:37.489 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T10:00:37.489 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T10:00:37.489 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T10:00:37.489 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=1 2024-04-06T10:00:37.489 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-06T10:00:37.489 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T10:00:37.489 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:00:37.490 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=1 2024-04-06T10:00:37.490 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=1 2024-04-06T10:00:37.490 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-06T10:00:37.490 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:1 2024-04-06T10:00:37.490 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T10:00:37.490 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T10:00:37.490 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-06T10:00:37.490 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:1 2024-04-06T10:00:37.490 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T10:00:37.490 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T10:00:37.490 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:00:37.490 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=1 2024-04-06T10:00:37.490 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-06T10:00:37.490 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T10:00:37.490 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T10:00:37.491 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=1 2024-04-06T10:00:37.491 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=1 2024-04-06T10:00:37.491 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-06T10:00:37.508 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T10:00:37.508 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T10:00:37.508 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T10:00:37.508 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/image_10 2024-04-06T10:00:37.635 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T10:00:37.636 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror image_10 2024-04-06T10:00:37.636 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:00:37.636 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:00:37.636 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_10 2024-04-06T10:00:37.636 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror image_10 '' 2024-04-06T10:00:37.636 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:00:37.636 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:00:37.637 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_10 2024-04-06T10:00:37.637 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T10:00:37.637 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-image_10.status 2024-04-06T10:00:37.637 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-image_10.status 2024-04-06T10:00:37.637 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T10:00:37.640 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-image_10.status 2024-04-06T10:00:37.640 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_10 2024-04-06T10:00:37.640 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_10.status 2024-04-06T10:00:37.688 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 2 2024-04-06T10:00:37.689 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 2 2024-04-06T10:00:37.689 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T10:00:37.689 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=9, tag_tid=2, entry_tid=1], [object_number=8, tag_tid=2, entry_tid=0], [object_number=10, tag_tid=1, entry_tid=12550], [object_number=11, tag_tid=1, entry_tid=12547]]], state=connected] 2024-04-06T10:00:37.689 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=10, tag_tid=1, entry_tid=12550], [object_number=9, tag_tid=1, entry_tid=12549], [object_number=8, tag_tid=1, entry_tid=12548], [object_number=11, tag_tid=1, entry_tid=12547]]], state=connected] 2024-04-06T10:00:37.691 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_10.status 2024-04-06T10:00:37.693 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=9, tag_tid=2, entry_tid=1' 2024-04-06T10:00:37.694 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror image_10 2024-04-06T10:00:37.694 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:00:37.694 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:00:37.694 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_10 2024-04-06T10:00:37.694 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror image_10 '..*' 2024-04-06T10:00:37.694 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:00:37.694 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:00:37.695 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_10 2024-04-06T10:00:37.695 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T10:00:37.695 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-image_10.status 2024-04-06T10:00:37.695 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-image_10.status 2024-04-06T10:00:37.695 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T10:00:37.695 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-image_10.status 2024-04-06T10:00:37.696 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_10 2024-04-06T10:00:37.696 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_10.status 2024-04-06T10:00:37.738 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 2 2024-04-06T10:00:37.738 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 2 2024-04-06T10:00:37.738 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T10:00:37.739 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=9, tag_tid=2, entry_tid=1], [object_number=8, tag_tid=2, entry_tid=0], [object_number=10, tag_tid=1, entry_tid=12550], [object_number=11, tag_tid=1, entry_tid=12547]]], state=connected] 2024-04-06T10:00:37.739 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=10, tag_tid=1, entry_tid=12550], [object_number=9, tag_tid=1, entry_tid=12549], [object_number=8, tag_tid=1, entry_tid=12548], [object_number=11, tag_tid=1, entry_tid=12547]]], state=connected] 2024-04-06T10:00:37.744 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_10.status 2024-04-06T10:00:37.746 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=10, tag_tid=1, entry_tid=12550' 2024-04-06T10:00:37.746 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=9, tag_tid=2, entry_tid=1' -a 'object_number=9, tag_tid=2, entry_tid=1' = 'object_number=10, tag_tid=1, entry_tid=12550' 2024-04-06T10:00:37.746 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=10, tag_tid=1, entry_tid=12550' '!=' '' 2024-04-06T10:00:37.746 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T10:00:37.746 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=10, tag_tid=1, entry_tid=12550' = '' 2024-04-06T10:00:37.746 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=10, tag_tid=1, entry_tid=12550' 2024-04-06T10:00:37.747 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=9, tag_tid=2, entry_tid=1' 2024-04-06T10:00:37.747 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T10:00:37.747 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T10:00:37.748 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=2 2024-04-06T10:00:37.749 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=10, tag_tid=1, entry_tid=12550' 2024-04-06T10:00:37.749 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T10:00:37.749 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T10:00:37.750 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=1 2024-04-06T10:00:37.750 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=9, tag_tid=2, entry_tid=1' 2024-04-06T10:00:37.750 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T10:00:37.750 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T10:00:37.751 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T10:00:37.751 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=10, tag_tid=1, entry_tid=12550' 2024-04-06T10:00:37.752 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T10:00:37.752 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T10:00:37.753 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=12550 2024-04-06T10:00:37.753 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 2 = 1 -a 1 -le 12550 2024-04-06T10:00:37.753 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T10:00:37.753 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T10:00:37.753 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T10:00:37.962 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror image_10 2024-04-06T10:00:37.962 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T10:00:37.962 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:00:37.962 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_10 2024-04-06T10:00:37.962 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T10:00:37.962 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n image_10 ']' 2024-04-06T10:00:37.962 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/image_10' 2024-04-06T10:00:37.962 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_10 2024-04-06T10:00:37.963 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T10:00:37.963 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T10:00:37.963 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T10:00:37.963 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T10:00:37.963 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T10:00:37.963 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T10:00:37.963 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T10:00:37.963 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T10:00:37.963 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T10:00:37.963 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T10:00:37.963 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_10 2024-04-06T10:00:37.963 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T10:00:37.963 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T10:00:37.964 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T10:00:37.964 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T10:00:37.964 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T10:00:37.964 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T10:00:37.964 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T10:00:37.964 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T10:00:37.965 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T10:00:37.965 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T10:00:37.965 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T10:00:37.965 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:00:37.965 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T10:00:37.965 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T10:00:37.966 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T10:00:37.966 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T10:00:37.966 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T10:00:37.966 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T10:00:37.966 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T10:00:37.967 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T10:00:37.967 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T10:00:37.967 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T10:00:37.967 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:00:37.967 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T10:00:37.967 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T10:00:37.968 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T10:00:37.968 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T10:00:37.968 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T10:00:37.968 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T10:00:37.968 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T10:00:37.985 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T10:00:37.985 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T10:00:37.985 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T10:00:37.985 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_10 2024-04-06T10:00:38.086 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:Invalid command: unused arguments: ['mirror/image_10'] 2024-04-06T10:00:38.086 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:rbd mirror flush : flush rbd mirror 2024-04-06T10:00:38.086 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:admin_socket: invalid command 2024-04-06T10:00:38.089 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T10:00:38.089 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/image_10 2024-04-06T10:00:38.090 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:1 2024-04-06T10:00:38.090 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T10:00:38.090 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T10:00:38.090 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-06T10:00:38.090 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:1 2024-04-06T10:00:38.090 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T10:00:38.090 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T10:00:38.090 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T10:00:38.090 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=1 2024-04-06T10:00:38.090 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-06T10:00:38.090 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T10:00:38.090 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:00:38.090 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=1 2024-04-06T10:00:38.091 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=1 2024-04-06T10:00:38.091 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-06T10:00:38.091 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:1 2024-04-06T10:00:38.091 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T10:00:38.091 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T10:00:38.091 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-06T10:00:38.091 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:1 2024-04-06T10:00:38.091 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T10:00:38.091 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T10:00:38.091 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:00:38.091 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=1 2024-04-06T10:00:38.091 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-06T10:00:38.091 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T10:00:38.091 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T10:00:38.092 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=1 2024-04-06T10:00:38.092 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=1 2024-04-06T10:00:38.092 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-06T10:00:38.114 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T10:00:38.114 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T10:00:38.115 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T10:00:38.115 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/image_10 2024-04-06T10:00:38.255 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T10:00:38.257 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror image_10 2024-04-06T10:00:38.257 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:00:38.257 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:00:38.257 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_10 2024-04-06T10:00:38.257 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror image_10 '' 2024-04-06T10:00:38.257 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:00:38.257 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:00:38.257 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_10 2024-04-06T10:00:38.257 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T10:00:38.257 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-image_10.status 2024-04-06T10:00:38.258 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-image_10.status 2024-04-06T10:00:38.258 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T10:00:38.261 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-image_10.status 2024-04-06T10:00:38.261 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_10 2024-04-06T10:00:38.261 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_10.status 2024-04-06T10:00:38.304 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 2 2024-04-06T10:00:38.305 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 2 2024-04-06T10:00:38.305 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T10:00:38.305 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=9, tag_tid=2, entry_tid=1], [object_number=8, tag_tid=2, entry_tid=0], [object_number=10, tag_tid=1, entry_tid=12550], [object_number=11, tag_tid=1, entry_tid=12547]]], state=connected] 2024-04-06T10:00:38.305 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=10, tag_tid=1, entry_tid=12550], [object_number=9, tag_tid=1, entry_tid=12549], [object_number=8, tag_tid=1, entry_tid=12548], [object_number=11, tag_tid=1, entry_tid=12547]]], state=connected] 2024-04-06T10:00:38.308 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_10.status 2024-04-06T10:00:38.309 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=9, tag_tid=2, entry_tid=1' 2024-04-06T10:00:38.310 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror image_10 2024-04-06T10:00:38.310 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:00:38.310 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:00:38.310 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_10 2024-04-06T10:00:38.310 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror image_10 '..*' 2024-04-06T10:00:38.311 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:00:38.311 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:00:38.311 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_10 2024-04-06T10:00:38.311 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T10:00:38.311 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-image_10.status 2024-04-06T10:00:38.311 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-image_10.status 2024-04-06T10:00:38.311 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T10:00:38.312 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-image_10.status 2024-04-06T10:00:38.312 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_10 2024-04-06T10:00:38.312 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_10.status 2024-04-06T10:00:38.355 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 2 2024-04-06T10:00:38.355 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 2 2024-04-06T10:00:38.355 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T10:00:38.356 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=9, tag_tid=2, entry_tid=1], [object_number=8, tag_tid=2, entry_tid=0], [object_number=10, tag_tid=1, entry_tid=12550], [object_number=11, tag_tid=1, entry_tid=12547]]], state=connected] 2024-04-06T10:00:38.356 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=10, tag_tid=1, entry_tid=12550], [object_number=9, tag_tid=1, entry_tid=12549], [object_number=8, tag_tid=1, entry_tid=12548], [object_number=11, tag_tid=1, entry_tid=12547]]], state=connected] 2024-04-06T10:00:38.359 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_10.status 2024-04-06T10:00:38.361 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=10, tag_tid=1, entry_tid=12550' 2024-04-06T10:00:38.361 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=9, tag_tid=2, entry_tid=1' -a 'object_number=9, tag_tid=2, entry_tid=1' = 'object_number=10, tag_tid=1, entry_tid=12550' 2024-04-06T10:00:38.361 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=10, tag_tid=1, entry_tid=12550' '!=' 'object_number=10, tag_tid=1, entry_tid=12550' 2024-04-06T10:00:38.361 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T10:00:38.361 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.4 2024-04-06T10:00:38.762 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror image_10 2024-04-06T10:00:38.762 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T10:00:38.762 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:00:38.762 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_10 2024-04-06T10:00:38.763 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T10:00:38.763 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n image_10 ']' 2024-04-06T10:00:38.763 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/image_10' 2024-04-06T10:00:38.763 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_10 2024-04-06T10:00:38.763 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T10:00:38.763 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T10:00:38.763 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T10:00:38.763 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T10:00:38.763 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T10:00:38.763 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T10:00:38.763 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T10:00:38.764 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T10:00:38.764 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T10:00:38.766 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T10:00:38.766 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_10 2024-04-06T10:00:38.766 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T10:00:38.766 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T10:00:38.766 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T10:00:38.766 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T10:00:38.766 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T10:00:38.766 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T10:00:38.767 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T10:00:38.767 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T10:00:38.767 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T10:00:38.767 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T10:00:38.767 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T10:00:38.767 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:00:38.767 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T10:00:38.767 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T10:00:38.768 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T10:00:38.768 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T10:00:38.768 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T10:00:38.769 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T10:00:38.769 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T10:00:38.769 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T10:00:38.769 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T10:00:38.769 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T10:00:38.769 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:00:38.769 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T10:00:38.769 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T10:00:38.769 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T10:00:38.769 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T10:00:38.769 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T10:00:38.769 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T10:00:38.769 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T10:00:38.793 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T10:00:38.793 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T10:00:38.794 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T10:00:38.794 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_10 2024-04-06T10:00:38.899 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:Invalid command: unused arguments: ['mirror/image_10'] 2024-04-06T10:00:38.899 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:rbd mirror flush : flush rbd mirror 2024-04-06T10:00:38.899 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:admin_socket: invalid command 2024-04-06T10:00:38.902 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T10:00:38.902 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/image_10 2024-04-06T10:00:38.902 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:1 2024-04-06T10:00:38.902 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T10:00:38.902 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T10:00:38.902 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-06T10:00:38.902 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:1 2024-04-06T10:00:38.902 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T10:00:38.902 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T10:00:38.902 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T10:00:38.902 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=1 2024-04-06T10:00:38.903 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-06T10:00:38.903 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T10:00:38.903 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:00:38.903 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=1 2024-04-06T10:00:38.903 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=1 2024-04-06T10:00:38.904 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-06T10:00:38.904 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:1 2024-04-06T10:00:38.904 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T10:00:38.904 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T10:00:38.904 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-06T10:00:38.905 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:1 2024-04-06T10:00:38.905 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T10:00:38.905 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T10:00:38.905 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:00:38.905 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=1 2024-04-06T10:00:38.905 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-06T10:00:38.905 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T10:00:38.905 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T10:00:38.905 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=1 2024-04-06T10:00:38.905 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=1 2024-04-06T10:00:38.905 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-06T10:00:38.928 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T10:00:38.929 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T10:00:38.929 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T10:00:38.929 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/image_10 2024-04-06T10:00:39.068 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T10:00:39.069 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror image_10 2024-04-06T10:00:39.069 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:00:39.069 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:00:39.069 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_10 2024-04-06T10:00:39.069 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror image_10 '' 2024-04-06T10:00:39.069 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:00:39.069 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:00:39.069 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_10 2024-04-06T10:00:39.069 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T10:00:39.070 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-image_10.status 2024-04-06T10:00:39.070 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-image_10.status 2024-04-06T10:00:39.070 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T10:00:39.071 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-image_10.status 2024-04-06T10:00:39.072 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_10 2024-04-06T10:00:39.072 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_10.status 2024-04-06T10:00:39.123 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 2 2024-04-06T10:00:39.123 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 2 2024-04-06T10:00:39.123 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T10:00:39.123 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=9, tag_tid=2, entry_tid=1], [object_number=8, tag_tid=2, entry_tid=0], [object_number=10, tag_tid=1, entry_tid=12550], [object_number=11, tag_tid=1, entry_tid=12547]]], state=connected] 2024-04-06T10:00:39.123 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=10, tag_tid=1, entry_tid=12550], [object_number=9, tag_tid=1, entry_tid=12549], [object_number=8, tag_tid=1, entry_tid=12548], [object_number=11, tag_tid=1, entry_tid=12547]]], state=connected] 2024-04-06T10:00:39.126 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_10.status 2024-04-06T10:00:39.127 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=9, tag_tid=2, entry_tid=1' 2024-04-06T10:00:39.129 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror image_10 2024-04-06T10:00:39.129 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:00:39.129 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:00:39.129 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_10 2024-04-06T10:00:39.129 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror image_10 '..*' 2024-04-06T10:00:39.129 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:00:39.129 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:00:39.129 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_10 2024-04-06T10:00:39.129 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T10:00:39.129 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-image_10.status 2024-04-06T10:00:39.129 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-image_10.status 2024-04-06T10:00:39.129 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T10:00:39.130 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-image_10.status 2024-04-06T10:00:39.130 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_10 2024-04-06T10:00:39.130 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_10.status 2024-04-06T10:00:39.176 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 2 2024-04-06T10:00:39.176 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 2 2024-04-06T10:00:39.176 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T10:00:39.176 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=9, tag_tid=2, entry_tid=1], [object_number=8, tag_tid=2, entry_tid=0], [object_number=10, tag_tid=1, entry_tid=12550], [object_number=11, tag_tid=1, entry_tid=12547]]], state=connected] 2024-04-06T10:00:39.176 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=10, tag_tid=1, entry_tid=12550], [object_number=9, tag_tid=1, entry_tid=12549], [object_number=8, tag_tid=1, entry_tid=12548], [object_number=11, tag_tid=1, entry_tid=12547]]], state=connected] 2024-04-06T10:00:39.179 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_10.status 2024-04-06T10:00:39.180 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=10, tag_tid=1, entry_tid=12550' 2024-04-06T10:00:39.181 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=9, tag_tid=2, entry_tid=1' -a 'object_number=9, tag_tid=2, entry_tid=1' = 'object_number=10, tag_tid=1, entry_tid=12550' 2024-04-06T10:00:39.181 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=10, tag_tid=1, entry_tid=12550' '!=' 'object_number=10, tag_tid=1, entry_tid=12550' 2024-04-06T10:00:39.181 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T10:00:39.181 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.8 2024-04-06T10:00:39.983 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror image_10 2024-04-06T10:00:39.983 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T10:00:39.983 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:00:39.983 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_10 2024-04-06T10:00:39.983 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T10:00:39.983 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n image_10 ']' 2024-04-06T10:00:39.983 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/image_10' 2024-04-06T10:00:39.983 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_10 2024-04-06T10:00:39.983 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T10:00:39.983 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T10:00:39.984 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T10:00:39.984 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T10:00:39.984 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T10:00:39.984 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T10:00:39.984 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T10:00:39.984 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T10:00:39.984 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T10:00:39.986 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T10:00:39.986 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_10 2024-04-06T10:00:39.986 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T10:00:39.987 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T10:00:39.987 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T10:00:39.987 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T10:00:39.987 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T10:00:39.987 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T10:00:39.987 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T10:00:39.987 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T10:00:39.987 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T10:00:39.987 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T10:00:39.987 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T10:00:39.987 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:00:39.987 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T10:00:39.987 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T10:00:39.989 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T10:00:39.989 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T10:00:39.989 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T10:00:39.989 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T10:00:39.989 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T10:00:39.989 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T10:00:39.989 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T10:00:39.989 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T10:00:39.989 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:00:39.989 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T10:00:39.989 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T10:00:39.989 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T10:00:39.990 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T10:00:39.990 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T10:00:39.990 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T10:00:39.990 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T10:00:40.014 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T10:00:40.014 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T10:00:40.014 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T10:00:40.014 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_10 2024-04-06T10:00:40.121 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:Invalid command: unused arguments: ['mirror/image_10'] 2024-04-06T10:00:40.122 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:rbd mirror flush : flush rbd mirror 2024-04-06T10:00:40.122 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:admin_socket: invalid command 2024-04-06T10:00:40.124 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T10:00:40.124 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/image_10 2024-04-06T10:00:40.124 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:1 2024-04-06T10:00:40.124 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T10:00:40.124 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T10:00:40.124 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-06T10:00:40.124 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:1 2024-04-06T10:00:40.124 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T10:00:40.124 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T10:00:40.124 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T10:00:40.125 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=1 2024-04-06T10:00:40.125 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-06T10:00:40.125 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T10:00:40.125 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:00:40.125 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=1 2024-04-06T10:00:40.125 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=1 2024-04-06T10:00:40.126 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-06T10:00:40.126 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:1 2024-04-06T10:00:40.126 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T10:00:40.126 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T10:00:40.126 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-06T10:00:40.126 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:1 2024-04-06T10:00:40.126 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T10:00:40.126 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T10:00:40.126 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:00:40.127 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=1 2024-04-06T10:00:40.127 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-06T10:00:40.127 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T10:00:40.127 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T10:00:40.127 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=1 2024-04-06T10:00:40.127 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=1 2024-04-06T10:00:40.127 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-06T10:00:40.151 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T10:00:40.151 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T10:00:40.151 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T10:00:40.151 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/image_10 2024-04-06T10:00:40.302 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T10:00:40.303 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror image_10 2024-04-06T10:00:40.303 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:00:40.303 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:00:40.303 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_10 2024-04-06T10:00:40.303 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror image_10 '' 2024-04-06T10:00:40.303 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:00:40.303 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:00:40.303 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_10 2024-04-06T10:00:40.303 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T10:00:40.304 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-image_10.status 2024-04-06T10:00:40.304 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-image_10.status 2024-04-06T10:00:40.304 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T10:00:40.306 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-image_10.status 2024-04-06T10:00:40.306 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_10.status 2024-04-06T10:00:40.306 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_10 2024-04-06T10:00:40.352 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 2 2024-04-06T10:00:40.352 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 2 2024-04-06T10:00:40.352 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T10:00:40.352 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=9, tag_tid=2, entry_tid=1], [object_number=8, tag_tid=2, entry_tid=0], [object_number=10, tag_tid=1, entry_tid=12550], [object_number=11, tag_tid=1, entry_tid=12547]]], state=connected] 2024-04-06T10:00:40.352 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=10, tag_tid=1, entry_tid=12550], [object_number=9, tag_tid=1, entry_tid=12549], [object_number=8, tag_tid=1, entry_tid=12548], [object_number=11, tag_tid=1, entry_tid=12547]]], state=connected] 2024-04-06T10:00:40.355 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_10.status 2024-04-06T10:00:40.357 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=9, tag_tid=2, entry_tid=1' 2024-04-06T10:00:40.358 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror image_10 2024-04-06T10:00:40.358 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:00:40.358 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:00:40.358 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_10 2024-04-06T10:00:40.358 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror image_10 '..*' 2024-04-06T10:00:40.358 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:00:40.358 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:00:40.358 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_10 2024-04-06T10:00:40.359 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T10:00:40.359 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-image_10.status 2024-04-06T10:00:40.359 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-image_10.status 2024-04-06T10:00:40.359 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T10:00:40.359 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-image_10.status 2024-04-06T10:00:40.360 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_10 2024-04-06T10:00:40.360 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_10.status 2024-04-06T10:00:40.405 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 2 2024-04-06T10:00:40.405 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 2 2024-04-06T10:00:40.405 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T10:00:40.405 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=9, tag_tid=2, entry_tid=1], [object_number=8, tag_tid=2, entry_tid=0], [object_number=10, tag_tid=1, entry_tid=12550], [object_number=11, tag_tid=1, entry_tid=12547]]], state=connected] 2024-04-06T10:00:40.405 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=10, tag_tid=1, entry_tid=12550], [object_number=9, tag_tid=1, entry_tid=12549], [object_number=8, tag_tid=1, entry_tid=12548], [object_number=11, tag_tid=1, entry_tid=12547]]], state=connected] 2024-04-06T10:00:40.408 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_10.status 2024-04-06T10:00:40.410 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=10, tag_tid=1, entry_tid=12550' 2024-04-06T10:00:40.410 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=9, tag_tid=2, entry_tid=1' -a 'object_number=9, tag_tid=2, entry_tid=1' = 'object_number=10, tag_tid=1, entry_tid=12550' 2024-04-06T10:00:40.410 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=10, tag_tid=1, entry_tid=12550' '!=' 'object_number=10, tag_tid=1, entry_tid=12550' 2024-04-06T10:00:40.410 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T10:00:40.411 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 1.6 2024-04-06T10:00:42.012 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror image_10 2024-04-06T10:00:42.012 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T10:00:42.012 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:00:42.013 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_10 2024-04-06T10:00:42.013 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T10:00:42.013 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n image_10 ']' 2024-04-06T10:00:42.013 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/image_10' 2024-04-06T10:00:42.013 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_10 2024-04-06T10:00:42.013 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T10:00:42.013 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T10:00:42.013 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T10:00:42.013 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T10:00:42.013 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T10:00:42.013 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T10:00:42.014 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T10:00:42.016 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T10:00:42.016 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T10:00:42.016 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T10:00:42.016 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_10 2024-04-06T10:00:42.016 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T10:00:42.016 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T10:00:42.016 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T10:00:42.017 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T10:00:42.017 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T10:00:42.017 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T10:00:42.017 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T10:00:42.017 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T10:00:42.017 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T10:00:42.017 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T10:00:42.017 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T10:00:42.017 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:00:42.017 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T10:00:42.017 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T10:00:42.017 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T10:00:42.018 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T10:00:42.018 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T10:00:42.018 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T10:00:42.018 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T10:00:42.019 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T10:00:42.019 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T10:00:42.019 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T10:00:42.019 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:00:42.019 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T10:00:42.020 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T10:00:42.020 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T10:00:42.020 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T10:00:42.020 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T10:00:42.020 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T10:00:42.020 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T10:00:42.042 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T10:00:42.042 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T10:00:42.042 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T10:00:42.042 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_10 2024-04-06T10:00:42.144 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:Invalid command: unused arguments: ['mirror/image_10'] 2024-04-06T10:00:42.144 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:rbd mirror flush : flush rbd mirror 2024-04-06T10:00:42.144 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:admin_socket: invalid command 2024-04-06T10:00:42.146 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T10:00:42.146 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/image_10 2024-04-06T10:00:42.146 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:1 2024-04-06T10:00:42.146 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T10:00:42.146 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T10:00:42.146 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-06T10:00:42.146 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:1 2024-04-06T10:00:42.146 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T10:00:42.147 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T10:00:42.147 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T10:00:42.147 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=1 2024-04-06T10:00:42.147 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-06T10:00:42.147 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T10:00:42.147 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:00:42.147 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=1 2024-04-06T10:00:42.147 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=1 2024-04-06T10:00:42.148 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-06T10:00:42.149 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:1 2024-04-06T10:00:42.149 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T10:00:42.149 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T10:00:42.149 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-06T10:00:42.149 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:1 2024-04-06T10:00:42.149 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T10:00:42.149 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T10:00:42.149 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:00:42.149 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=1 2024-04-06T10:00:42.149 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-06T10:00:42.149 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T10:00:42.149 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T10:00:42.149 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=1 2024-04-06T10:00:42.149 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=1 2024-04-06T10:00:42.150 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-06T10:00:42.172 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T10:00:42.172 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T10:00:42.172 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T10:00:42.172 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/image_10 2024-04-06T10:00:42.312 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T10:00:42.313 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror image_10 2024-04-06T10:00:42.313 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:00:42.313 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:00:42.313 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_10 2024-04-06T10:00:42.313 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror image_10 '' 2024-04-06T10:00:42.313 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:00:42.314 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:00:42.314 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_10 2024-04-06T10:00:42.314 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T10:00:42.314 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-image_10.status 2024-04-06T10:00:42.314 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-image_10.status 2024-04-06T10:00:42.315 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T10:00:42.317 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-image_10.status 2024-04-06T10:00:42.317 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_10 2024-04-06T10:00:42.317 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_10.status 2024-04-06T10:00:42.367 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 2 2024-04-06T10:00:42.367 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 2 2024-04-06T10:00:42.367 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T10:00:42.367 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=9, tag_tid=2, entry_tid=1], [object_number=8, tag_tid=2, entry_tid=0], [object_number=10, tag_tid=1, entry_tid=12550], [object_number=11, tag_tid=1, entry_tid=12547]]], state=connected] 2024-04-06T10:00:42.368 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=10, tag_tid=1, entry_tid=12550], [object_number=9, tag_tid=1, entry_tid=12549], [object_number=8, tag_tid=1, entry_tid=12548], [object_number=11, tag_tid=1, entry_tid=12547]]], state=connected] 2024-04-06T10:00:42.371 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_10.status 2024-04-06T10:00:42.372 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=9, tag_tid=2, entry_tid=1' 2024-04-06T10:00:42.373 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror image_10 2024-04-06T10:00:42.374 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:00:42.374 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:00:42.374 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_10 2024-04-06T10:00:42.374 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror image_10 '..*' 2024-04-06T10:00:42.374 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:00:42.374 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:00:42.374 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_10 2024-04-06T10:00:42.374 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T10:00:42.374 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-image_10.status 2024-04-06T10:00:42.374 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-image_10.status 2024-04-06T10:00:42.374 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T10:00:42.375 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-image_10.status 2024-04-06T10:00:42.375 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_10 2024-04-06T10:00:42.375 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_10.status 2024-04-06T10:00:42.420 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 2 2024-04-06T10:00:42.420 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 2 2024-04-06T10:00:42.420 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T10:00:42.420 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=9, tag_tid=2, entry_tid=1], [object_number=8, tag_tid=2, entry_tid=0], [object_number=10, tag_tid=1, entry_tid=12550], [object_number=11, tag_tid=1, entry_tid=12547]]], state=connected] 2024-04-06T10:00:42.420 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=10, tag_tid=1, entry_tid=12550], [object_number=9, tag_tid=1, entry_tid=12549], [object_number=8, tag_tid=1, entry_tid=12548], [object_number=11, tag_tid=1, entry_tid=12547]]], state=connected] 2024-04-06T10:00:42.424 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_10.status 2024-04-06T10:00:42.425 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=10, tag_tid=1, entry_tid=12550' 2024-04-06T10:00:42.425 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=9, tag_tid=2, entry_tid=1' -a 'object_number=9, tag_tid=2, entry_tid=1' = 'object_number=10, tag_tid=1, entry_tid=12550' 2024-04-06T10:00:42.425 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=10, tag_tid=1, entry_tid=12550' '!=' 'object_number=10, tag_tid=1, entry_tid=12550' 2024-04-06T10:00:42.425 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T10:00:42.426 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 2 2024-04-06T10:00:43.447 INFO:tasks.rbd_mirror_thrash:kill cluster2.client.mirror.0 2024-04-06T10:00:43.448 INFO:tasks.rbd_mirror.cluster2.client.mirror.0:Sent signal 15 2024-04-06T10:00:43.448 INFO:tasks.rbd_mirror_thrash:waiting for 7 secs before reviving daemons 2024-04-06T10:00:43.449 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi192.stderr:2024-04-06T10:00:43.450+0000 7f4f0fee6640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.0 (PID: 133260) UID: 1000 2024-04-06T10:00:44.427 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror image_10 2024-04-06T10:00:44.427 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T10:00:44.428 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:00:44.428 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_10 2024-04-06T10:00:44.428 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T10:00:44.428 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n image_10 ']' 2024-04-06T10:00:44.428 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/image_10' 2024-04-06T10:00:44.428 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_10 2024-04-06T10:00:44.428 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T10:00:44.428 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T10:00:44.428 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T10:00:44.428 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T10:00:44.428 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T10:00:44.428 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T10:00:44.428 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T10:00:44.429 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T10:00:44.429 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T10:00:44.432 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T10:00:44.432 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_10 2024-04-06T10:00:44.432 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T10:00:44.432 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T10:00:44.432 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T10:00:44.432 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T10:00:44.433 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T10:00:44.433 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T10:00:44.433 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T10:00:44.433 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T10:00:44.433 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T10:00:44.433 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T10:00:44.433 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T10:00:44.433 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:00:44.433 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T10:00:44.433 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T10:00:44.433 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T10:00:44.433 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T10:00:44.433 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T10:00:44.433 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T10:00:44.433 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T10:00:44.434 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T10:00:44.434 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T10:00:44.434 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T10:00:44.434 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:00:44.434 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T10:00:44.434 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T10:00:44.434 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T10:00:44.434 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T10:00:44.435 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T10:00:44.435 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T10:00:44.435 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T10:00:44.460 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T10:00:44.460 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T10:00:44.460 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T10:00:44.460 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_10 2024-04-06T10:00:44.570 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:Invalid command: unused arguments: ['mirror/image_10'] 2024-04-06T10:00:44.570 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:rbd mirror flush : flush rbd mirror 2024-04-06T10:00:44.570 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:admin_socket: invalid command 2024-04-06T10:00:44.573 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T10:00:44.573 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/image_10 2024-04-06T10:00:44.573 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:1 2024-04-06T10:00:44.574 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T10:00:44.574 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T10:00:44.574 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-06T10:00:44.574 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:1 2024-04-06T10:00:44.574 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T10:00:44.574 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T10:00:44.574 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T10:00:44.574 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=1 2024-04-06T10:00:44.574 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-06T10:00:44.574 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T10:00:44.574 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:00:44.574 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=1 2024-04-06T10:00:44.574 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=1 2024-04-06T10:00:44.575 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-06T10:00:44.575 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:1 2024-04-06T10:00:44.575 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T10:00:44.575 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T10:00:44.575 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-06T10:00:44.575 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:1 2024-04-06T10:00:44.575 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T10:00:44.575 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T10:00:44.575 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:00:44.575 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=1 2024-04-06T10:00:44.575 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-06T10:00:44.575 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T10:00:44.575 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T10:00:44.575 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=1 2024-04-06T10:00:44.576 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=1 2024-04-06T10:00:44.576 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-06T10:00:44.599 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T10:00:44.599 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T10:00:44.599 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T10:00:44.599 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/image_10 2024-04-06T10:00:44.740 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T10:00:44.741 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror image_10 2024-04-06T10:00:44.741 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:00:44.741 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:00:44.741 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_10 2024-04-06T10:00:44.741 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror image_10 '' 2024-04-06T10:00:44.741 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:00:44.741 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:00:44.742 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_10 2024-04-06T10:00:44.742 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T10:00:44.742 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-image_10.status 2024-04-06T10:00:44.742 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-image_10.status 2024-04-06T10:00:44.742 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T10:00:44.745 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-image_10.status 2024-04-06T10:00:44.745 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_10 2024-04-06T10:00:44.745 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_10.status 2024-04-06T10:00:44.795 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 2 2024-04-06T10:00:44.795 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 2 2024-04-06T10:00:44.795 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T10:00:44.795 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=9, tag_tid=2, entry_tid=1], [object_number=8, tag_tid=2, entry_tid=0], [object_number=10, tag_tid=1, entry_tid=12550], [object_number=11, tag_tid=1, entry_tid=12547]]], state=connected] 2024-04-06T10:00:44.796 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=10, tag_tid=1, entry_tid=12550], [object_number=9, tag_tid=1, entry_tid=12549], [object_number=8, tag_tid=1, entry_tid=12548], [object_number=11, tag_tid=1, entry_tid=12547]]], state=connected] 2024-04-06T10:00:44.799 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_10.status 2024-04-06T10:00:44.800 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=9, tag_tid=2, entry_tid=1' 2024-04-06T10:00:44.802 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror image_10 2024-04-06T10:00:44.802 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:00:44.802 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:00:44.802 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_10 2024-04-06T10:00:44.802 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror image_10 '..*' 2024-04-06T10:00:44.802 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:00:44.803 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:00:44.803 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_10 2024-04-06T10:00:44.803 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T10:00:44.803 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-image_10.status 2024-04-06T10:00:44.803 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-image_10.status 2024-04-06T10:00:44.803 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T10:00:44.804 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-image_10.status 2024-04-06T10:00:44.804 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_10 2024-04-06T10:00:44.804 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_10.status 2024-04-06T10:00:44.849 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 2 2024-04-06T10:00:44.849 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 2 2024-04-06T10:00:44.849 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T10:00:44.849 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=9, tag_tid=2, entry_tid=1], [object_number=8, tag_tid=2, entry_tid=0], [object_number=10, tag_tid=1, entry_tid=12550], [object_number=11, tag_tid=1, entry_tid=12547]]], state=connected] 2024-04-06T10:00:44.849 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=10, tag_tid=1, entry_tid=12550], [object_number=9, tag_tid=1, entry_tid=12549], [object_number=8, tag_tid=1, entry_tid=12548], [object_number=11, tag_tid=1, entry_tid=12547]]], state=connected] 2024-04-06T10:00:44.853 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_10.status 2024-04-06T10:00:44.855 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=10, tag_tid=1, entry_tid=12550' 2024-04-06T10:00:44.855 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=9, tag_tid=2, entry_tid=1' -a 'object_number=9, tag_tid=2, entry_tid=1' = 'object_number=10, tag_tid=1, entry_tid=12550' 2024-04-06T10:00:44.855 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=10, tag_tid=1, entry_tid=12550' '!=' 'object_number=10, tag_tid=1, entry_tid=12550' 2024-04-06T10:00:44.855 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T10:00:44.855 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 2 2024-04-06T10:00:46.856 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror image_10 2024-04-06T10:00:46.857 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T10:00:46.857 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:00:46.857 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_10 2024-04-06T10:00:46.857 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T10:00:46.857 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n image_10 ']' 2024-04-06T10:00:46.857 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/image_10' 2024-04-06T10:00:46.857 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_10 2024-04-06T10:00:46.857 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T10:00:46.857 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T10:00:46.857 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T10:00:46.857 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T10:00:46.857 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T10:00:46.858 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T10:00:46.858 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T10:00:46.858 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T10:00:46.858 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T10:00:46.863 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T10:00:46.863 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_10 2024-04-06T10:00:46.863 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T10:00:46.863 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T10:00:46.863 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T10:00:46.863 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T10:00:46.863 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T10:00:46.863 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T10:00:46.863 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T10:00:46.863 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T10:00:46.864 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T10:00:46.864 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T10:00:46.864 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T10:00:46.864 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:00:46.864 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T10:00:46.864 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T10:00:46.864 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T10:00:46.864 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T10:00:46.864 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T10:00:46.864 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T10:00:46.864 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T10:00:46.864 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T10:00:46.864 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T10:00:46.864 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T10:00:46.865 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:00:46.865 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T10:00:46.865 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T10:00:46.865 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T10:00:46.865 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T10:00:46.865 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T10:00:46.865 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T10:00:46.865 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T10:00:46.888 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T10:00:46.889 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T10:00:46.889 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T10:00:46.889 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_10 2024-04-06T10:00:46.995 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:Invalid command: unused arguments: ['mirror/image_10'] 2024-04-06T10:00:46.995 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:rbd mirror flush : flush rbd mirror 2024-04-06T10:00:46.995 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:admin_socket: invalid command 2024-04-06T10:00:46.999 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T10:00:46.999 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/image_10 2024-04-06T10:00:46.999 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:1 2024-04-06T10:00:46.999 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T10:00:46.999 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T10:00:47.000 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-06T10:00:47.000 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:1 2024-04-06T10:00:47.000 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T10:00:47.000 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T10:00:47.000 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T10:00:47.000 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=1 2024-04-06T10:00:47.000 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-06T10:00:47.000 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T10:00:47.000 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:00:47.000 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=1 2024-04-06T10:00:47.000 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=1 2024-04-06T10:00:47.000 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-06T10:00:47.001 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:1 2024-04-06T10:00:47.001 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T10:00:47.001 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T10:00:47.001 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-06T10:00:47.001 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:1 2024-04-06T10:00:47.001 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T10:00:47.001 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T10:00:47.001 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:00:47.001 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=1 2024-04-06T10:00:47.001 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-06T10:00:47.001 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T10:00:47.001 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T10:00:47.001 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=1 2024-04-06T10:00:47.001 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=1 2024-04-06T10:00:47.001 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-06T10:00:47.018 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T10:00:47.019 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T10:00:47.019 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T10:00:47.019 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/image_10 2024-04-06T10:00:47.165 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T10:00:47.167 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror image_10 2024-04-06T10:00:47.167 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:00:47.167 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:00:47.167 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_10 2024-04-06T10:00:47.167 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror image_10 '' 2024-04-06T10:00:47.167 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:00:47.167 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:00:47.167 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_10 2024-04-06T10:00:47.167 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T10:00:47.167 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-image_10.status 2024-04-06T10:00:47.168 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-image_10.status 2024-04-06T10:00:47.168 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T10:00:47.169 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-image_10.status 2024-04-06T10:00:47.169 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_10 2024-04-06T10:00:47.169 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_10.status 2024-04-06T10:00:47.224 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 2 2024-04-06T10:00:47.224 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 2 2024-04-06T10:00:47.224 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T10:00:47.224 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=9, tag_tid=2, entry_tid=1], [object_number=8, tag_tid=2, entry_tid=0], [object_number=10, tag_tid=1, entry_tid=12550], [object_number=11, tag_tid=1, entry_tid=12547]]], state=connected] 2024-04-06T10:00:47.224 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=9, tag_tid=2, entry_tid=1], [object_number=8, tag_tid=2, entry_tid=0], [object_number=10, tag_tid=1, entry_tid=12550], [object_number=11, tag_tid=1, entry_tid=12547]]], state=connected] 2024-04-06T10:00:47.228 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_10.status 2024-04-06T10:00:47.230 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=9, tag_tid=2, entry_tid=1' 2024-04-06T10:00:47.230 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror image_10 2024-04-06T10:00:47.231 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:00:47.231 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:00:47.231 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_10 2024-04-06T10:00:47.231 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror image_10 '..*' 2024-04-06T10:00:47.231 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:00:47.231 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:00:47.231 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_10 2024-04-06T10:00:47.231 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T10:00:47.231 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-image_10.status 2024-04-06T10:00:47.231 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-image_10.status 2024-04-06T10:00:47.231 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T10:00:47.232 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-image_10.status 2024-04-06T10:00:47.232 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_10 2024-04-06T10:00:47.232 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_10.status 2024-04-06T10:00:47.278 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 2 2024-04-06T10:00:47.278 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 2 2024-04-06T10:00:47.278 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T10:00:47.278 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=9, tag_tid=2, entry_tid=1], [object_number=8, tag_tid=2, entry_tid=0], [object_number=10, tag_tid=1, entry_tid=12550], [object_number=11, tag_tid=1, entry_tid=12547]]], state=connected] 2024-04-06T10:00:47.278 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=9, tag_tid=2, entry_tid=1], [object_number=8, tag_tid=2, entry_tid=0], [object_number=10, tag_tid=1, entry_tid=12550], [object_number=11, tag_tid=1, entry_tid=12547]]], state=connected] 2024-04-06T10:00:47.281 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_10.status 2024-04-06T10:00:47.283 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=9, tag_tid=2, entry_tid=1' 2024-04-06T10:00:47.284 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=9, tag_tid=2, entry_tid=1' -a 'object_number=9, tag_tid=2, entry_tid=1' = 'object_number=9, tag_tid=2, entry_tid=1' 2024-04-06T10:00:47.284 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T10:00:47.284 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_snap_present cluster1 mirror image_10 snap 2024-04-06T10:00:47.284 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T10:00:47.284 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:00:47.284 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_10 2024-04-06T10:00:47.284 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local snap_name=snap 2024-04-06T10:00:47.284 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local s 2024-04-06T10:00:47.284 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-04-06T10:00:47.284 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 1 2024-04-06T10:00:48.285 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 info mirror/image_10@snap 2024-04-06T10:00:48.350 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:rbd image 'image_10': 2024-04-06T10:00:48.350 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: size 128 MiB in 32 objects 2024-04-06T10:00:48.350 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: order 22 (4 MiB objects) 2024-04-06T10:00:48.350 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: snapshot_count: 1 2024-04-06T10:00:48.350 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: id: 10aca1e0e882 2024-04-06T10:00:48.350 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: block_name_prefix: rbd_data.10aca1e0e882 2024-04-06T10:00:48.350 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: format: 2 2024-04-06T10:00:48.350 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: features: layering, exclusive-lock, journaling 2024-04-06T10:00:48.350 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: op_features: 2024-04-06T10:00:48.350 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: flags: 2024-04-06T10:00:48.350 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: create_timestamp: Sat Apr 6 09:52:13 2024 2024-04-06T10:00:48.351 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: access_timestamp: Sat Apr 6 09:52:13 2024 2024-04-06T10:00:48.351 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: modify_timestamp: Sat Apr 6 09:53:14 2024 2024-04-06T10:00:48.351 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: protected: False 2024-04-06T10:00:48.351 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: journal: 10aca1e0e882 2024-04-06T10:00:48.351 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: mirroring state: enabled 2024-04-06T10:00:48.351 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: mirroring mode: journal 2024-04-06T10:00:48.351 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: mirroring global id: 9e513123-1516-4450-b0f8-921f0336d667 2024-04-06T10:00:48.351 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: mirroring primary: false 2024-04-06T10:00:48.355 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T10:00:48.355 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ compare_image_snaps mirror image_10 snap 2024-04-06T10:00:48.355 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:00:48.355 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_10 2024-04-06T10:00:48.355 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local snap_name=snap 2024-04-06T10:00:48.355 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local ret=0 2024-04-06T10:00:48.355 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local rmt_export=/tmp/tmp.BhW6xKmROU/cluster2-mirror-image_10.export 2024-04-06T10:00:48.356 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loc_export=/tmp/tmp.BhW6xKmROU/cluster1-mirror-image_10.export 2024-04-06T10:00:48.356 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rm -f /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_10.export /tmp/tmp.BhW6xKmROU/cluster1-mirror-image_10.export 2024-04-06T10:00:48.356 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster2 -p mirror export image_10@snap /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_10.export 2024-04-06T10:00:48.762 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-04-06T10:00:48.770 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 -p mirror export image_10@snap /tmp/tmp.BhW6xKmROU/cluster1-mirror-image_10.export 2024-04-06T10:00:49.127 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-04-06T10:00:49.139 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmp /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_10.export /tmp/tmp.BhW6xKmROU/cluster1-mirror-image_10.export 2024-04-06T10:00:49.219 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rm -f /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_10.export /tmp/tmp.BhW6xKmROU/cluster1-mirror-image_10.export 2024-04-06T10:00:49.260 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T10:00:49.260 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-06T10:00:49.260 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ image=image_11 2024-04-06T10:00:49.260 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ create_snap cluster2 mirror image_11 snap 2024-04-06T10:00:49.260 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-04-06T10:00:49.260 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:00:49.260 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_11 2024-04-06T10:00:49.260 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local snap_name=snap 2024-04-06T10:00:49.260 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster2 -p mirror snap create image_11@snap --debug-rbd=20 --debug-journaler=20 2024-04-06T10:00:50.449 INFO:tasks.rbd_mirror_thrash:waiting for cluster2.client.mirror.0 2024-04-06T10:00:50.449 INFO:teuthology.orchestra.run:waiting for 600 2024-04-06T10:00:50.449 INFO:tasks.rbd_mirror_thrash:reviving cluster2.client.mirror.0 2024-04-06T10:00:50.449 INFO:tasks.rbd_mirror.cluster2.client.mirror.0:Restarting daemon 2024-04-06T10:00:50.450 DEBUG:teuthology.orchestra.run.smithi192:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.0 2024-04-06T10:00:50.452 INFO:tasks.rbd_mirror.cluster2.client.mirror.0:Started 2024-04-06T10:00:50.452 INFO:tasks.rbd_mirror_thrash:waiting for 61 secs before thrashing 2024-04-06T10:00:50.837 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_image_replay_started cluster1 mirror image_11 2024-04-06T10:00:50.837 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T10:00:50.838 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:00:50.838 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_11 2024-04-06T10:00:50.838 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_image_replay_state cluster1 mirror image_11 started 2024-04-06T10:00:50.838 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T10:00:50.838 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:00:50.838 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_11 2024-04-06T10:00:50.838 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state=started 2024-04-06T10:00:50.838 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local s 2024-04-06T10:00:50.839 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T10:00:50.839 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 1 2024-04-06T10:00:51.838 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test_image_replay_state cluster1 mirror image_11 started 2024-04-06T10:00:51.838 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T10:00:51.838 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:00:51.838 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_11 2024-04-06T10:00:51.839 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local test_state=started 2024-04-06T10:00:51.839 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local status_result 2024-04-06T10:00:51.839 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local current_state=stopped 2024-04-06T10:00:51.840 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_11 2024-04-06T10:00:51.840 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_instance=cluster1 2024-04-06T10:00:51.840 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -i state 2024-04-06T10:00:51.840 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ shift 2024-04-06T10:00:51.840 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T10:00:51.840 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance=cluster1 2024-04-06T10:00:51.840 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local loop_instance 2024-04-06T10:00:51.840 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T10:00:51.840 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sleep 0 2024-04-06T10:00:51.841 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-04-06T10:00:51.841 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ seq 0 3 2024-04-06T10:00:51.844 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T10:00:51.844 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_11 2024-04-06T10:00:51.844 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1:0 2024-04-06T10:00:51.844 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ shift 2024-04-06T10:00:51.844 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T10:00:51.844 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-04-06T10:00:51.844 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T10:00:51.844 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=cluster 2024-04-06T10:00:51.844 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T10:00:51.845 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:00:51.845 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T10:00:51.845 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T10:00:51.845 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval cluster=cluster1 2024-04-06T10:00:51.845 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ cluster=cluster1 2024-04-06T10:00:51.845 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T10:00:51.845 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T10:00:51.845 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-04-06T10:00:51.845 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local local_cluster=cluster1:0 2024-04-06T10:00:51.845 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local cluster=cluster1 2024-04-06T10:00:51.845 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local instance 2024-04-06T10:00:51.845 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T10:00:51.845 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local val=cluster1:0 2024-04-06T10:00:51.845 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local cluster_var_name=local_cluster 2024-04-06T10:00:51.845 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local instance_var_name=instance 2024-04-06T10:00:51.845 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ cluster=cluster1 2024-04-06T10:00:51.846 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T10:00:51.846 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ '[' 0 = cluster1:0 ']' 2024-04-06T10:00:51.846 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ eval local_cluster=cluster1 2024-04-06T10:00:51.846 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++++ local_cluster=cluster1 2024-04-06T10:00:51.846 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ eval instance=0 2024-04-06T10:00:51.846 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++++ instance=0 2024-04-06T10:00:51.846 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T10:00:51.866 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T10:00:51.866 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T10:00:51.866 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T10:00:51.866 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_11 2024-04-06T10:00:51.970 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:Invalid command: unused arguments: ['mirror/image_11'] 2024-04-06T10:00:51.971 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:rbd mirror status : get status for rbd mirror 2024-04-06T10:00:51.971 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:admin_socket: invalid command 2024-04-06T10:00:51.973 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T10:00:51.973 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_11 2024-04-06T10:00:51.973 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1:1 2024-04-06T10:00:51.973 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ shift 2024-04-06T10:00:51.973 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T10:00:51.973 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-04-06T10:00:51.973 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:1 2024-04-06T10:00:51.973 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=cluster 2024-04-06T10:00:51.973 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T10:00:51.973 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:00:51.973 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=1 2024-04-06T10:00:51.973 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-06T10:00:51.973 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval cluster=cluster1 2024-04-06T10:00:51.973 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ cluster=cluster1 2024-04-06T10:00:51.974 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=1 2024-04-06T10:00:51.974 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=1 2024-04-06T10:00:51.975 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-04-06T10:00:51.975 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local local_cluster=cluster1:1 2024-04-06T10:00:51.975 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local cluster=cluster1 2024-04-06T10:00:51.975 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local instance 2024-04-06T10:00:51.975 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-06T10:00:51.975 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local val=cluster1:1 2024-04-06T10:00:51.975 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local cluster_var_name=local_cluster 2024-04-06T10:00:51.975 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local instance_var_name=instance 2024-04-06T10:00:51.975 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ cluster=cluster1 2024-04-06T10:00:51.976 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=1 2024-04-06T10:00:51.976 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ '[' 1 = cluster1:1 ']' 2024-04-06T10:00:51.976 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ eval local_cluster=cluster1 2024-04-06T10:00:51.976 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++++ local_cluster=cluster1 2024-04-06T10:00:51.976 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ eval instance=1 2024-04-06T10:00:51.976 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++++ instance=1 2024-04-06T10:00:51.976 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-06T10:00:51.993 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T10:00:51.993 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T10:00:51.993 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T10:00:51.994 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_11 2024-04-06T10:00:52.144 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ return 0 2024-04-06T10:00:52.145 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ status_result=' "state": "Replaying"' 2024-04-06T10:00:52.145 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ echo ' "state": "Replaying"' 2024-04-06T10:00:52.145 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep -i Replaying 2024-04-06T10:00:52.146 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: "state": "Replaying" 2024-04-06T10:00:52.148 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ current_state=started 2024-04-06T10:00:52.148 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test started = started 2024-04-06T10:00:52.148 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T10:00:52.148 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_11 2024-04-06T10:00:52.148 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local local_cluster=cluster1 2024-04-06T10:00:52.148 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-04-06T10:00:52.148 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:00:52.149 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_11 2024-04-06T10:00:52.149 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' journal = journal ']' 2024-04-06T10:00:52.149 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_journal_replay_complete cluster1 cluster2 mirror image_11 2024-04-06T10:00:52.149 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local local_cluster=cluster1 2024-04-06T10:00:52.149 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-04-06T10:00:52.149 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:00:52.149 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_11 2024-04-06T10:00:52.149 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local s master_pos mirror_pos last_mirror_pos 2024-04-06T10:00:52.149 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local master_tag master_entry mirror_tag mirror_entry 2024-04-06T10:00:52.149 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T10:00:52.149 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T10:00:52.149 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T10:00:52.350 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror image_11 2024-04-06T10:00:52.350 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T10:00:52.350 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:00:52.350 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_11 2024-04-06T10:00:52.351 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T10:00:52.351 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n image_11 ']' 2024-04-06T10:00:52.351 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/image_11' 2024-04-06T10:00:52.351 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_11 2024-04-06T10:00:52.351 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T10:00:52.351 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T10:00:52.351 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T10:00:52.351 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T10:00:52.351 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T10:00:52.351 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T10:00:52.351 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T10:00:52.352 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T10:00:52.352 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T10:00:52.354 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T10:00:52.354 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_11 2024-04-06T10:00:52.354 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T10:00:52.354 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T10:00:52.354 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T10:00:52.354 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T10:00:52.354 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T10:00:52.354 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T10:00:52.354 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T10:00:52.355 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T10:00:52.355 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T10:00:52.355 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T10:00:52.355 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T10:00:52.355 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:00:52.355 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T10:00:52.355 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T10:00:52.357 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T10:00:52.357 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T10:00:52.357 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T10:00:52.357 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T10:00:52.357 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T10:00:52.357 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T10:00:52.357 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T10:00:52.357 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T10:00:52.357 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:00:52.357 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T10:00:52.357 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T10:00:52.358 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T10:00:52.358 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T10:00:52.358 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T10:00:52.358 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T10:00:52.358 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T10:00:52.381 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T10:00:52.381 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T10:00:52.382 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T10:00:52.382 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_11 2024-04-06T10:00:52.486 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:Invalid command: unused arguments: ['mirror/image_11'] 2024-04-06T10:00:52.487 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:rbd mirror flush : flush rbd mirror 2024-04-06T10:00:52.487 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:admin_socket: invalid command 2024-04-06T10:00:52.490 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T10:00:52.490 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/image_11 2024-04-06T10:00:52.490 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:1 2024-04-06T10:00:52.490 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T10:00:52.490 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T10:00:52.490 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-06T10:00:52.490 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:1 2024-04-06T10:00:52.490 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T10:00:52.490 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T10:00:52.490 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T10:00:52.490 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=1 2024-04-06T10:00:52.490 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-06T10:00:52.491 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T10:00:52.491 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:00:52.491 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=1 2024-04-06T10:00:52.491 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=1 2024-04-06T10:00:52.491 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-06T10:00:52.491 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:1 2024-04-06T10:00:52.491 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T10:00:52.491 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T10:00:52.491 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-06T10:00:52.491 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:1 2024-04-06T10:00:52.491 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T10:00:52.491 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T10:00:52.491 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:00:52.491 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=1 2024-04-06T10:00:52.492 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-06T10:00:52.492 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T10:00:52.492 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T10:00:52.492 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=1 2024-04-06T10:00:52.492 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=1 2024-04-06T10:00:52.492 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-06T10:00:52.516 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T10:00:52.516 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T10:00:52.516 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T10:00:52.516 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/image_11 2024-04-06T10:00:52.664 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T10:00:52.665 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror image_11 2024-04-06T10:00:52.665 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:00:52.665 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:00:52.665 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_11 2024-04-06T10:00:52.665 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror image_11 '' 2024-04-06T10:00:52.665 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:00:52.665 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:00:52.665 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_11 2024-04-06T10:00:52.665 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T10:00:52.666 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-image_11.status 2024-04-06T10:00:52.666 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-image_11.status 2024-04-06T10:00:52.666 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T10:00:52.668 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-image_11.status 2024-04-06T10:00:52.669 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_11.status 2024-04-06T10:00:52.669 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_11 2024-04-06T10:00:52.720 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 2 2024-04-06T10:00:52.720 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 2 2024-04-06T10:00:52.721 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T10:00:52.721 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=9, tag_tid=2, entry_tid=1], [object_number=8, tag_tid=2, entry_tid=0], [object_number=11, tag_tid=1, entry_tid=11819], [object_number=10, tag_tid=1, entry_tid=11818]]], state=connected] 2024-04-06T10:00:52.721 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=9, tag_tid=2, entry_tid=1], [object_number=8, tag_tid=2, entry_tid=0], [object_number=11, tag_tid=1, entry_tid=11819], [object_number=10, tag_tid=1, entry_tid=11818]]], state=connected] 2024-04-06T10:00:52.724 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_11.status 2024-04-06T10:00:52.725 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=9, tag_tid=2, entry_tid=1' 2024-04-06T10:00:52.727 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror image_11 2024-04-06T10:00:52.727 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:00:52.727 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:00:52.727 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_11 2024-04-06T10:00:52.727 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror image_11 '..*' 2024-04-06T10:00:52.727 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:00:52.727 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:00:52.727 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_11 2024-04-06T10:00:52.727 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T10:00:52.727 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-image_11.status 2024-04-06T10:00:52.727 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-image_11.status 2024-04-06T10:00:52.727 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T10:00:52.728 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-image_11.status 2024-04-06T10:00:52.728 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_11 2024-04-06T10:00:52.728 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_11.status 2024-04-06T10:00:52.773 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 2 2024-04-06T10:00:52.774 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 2 2024-04-06T10:00:52.774 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T10:00:52.774 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=9, tag_tid=2, entry_tid=1], [object_number=8, tag_tid=2, entry_tid=0], [object_number=11, tag_tid=1, entry_tid=11819], [object_number=10, tag_tid=1, entry_tid=11818]]], state=connected] 2024-04-06T10:00:52.774 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=9, tag_tid=2, entry_tid=1], [object_number=8, tag_tid=2, entry_tid=0], [object_number=11, tag_tid=1, entry_tid=11819], [object_number=10, tag_tid=1, entry_tid=11818]]], state=connected] 2024-04-06T10:00:52.777 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_11.status 2024-04-06T10:00:52.779 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=9, tag_tid=2, entry_tid=1' 2024-04-06T10:00:52.779 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=9, tag_tid=2, entry_tid=1' -a 'object_number=9, tag_tid=2, entry_tid=1' = 'object_number=9, tag_tid=2, entry_tid=1' 2024-04-06T10:00:52.779 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T10:00:52.779 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_snap_present cluster1 mirror image_11 snap 2024-04-06T10:00:52.780 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T10:00:52.780 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:00:52.780 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_11 2024-04-06T10:00:52.780 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local snap_name=snap 2024-04-06T10:00:52.780 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local s 2024-04-06T10:00:52.780 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-04-06T10:00:52.780 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 1 2024-04-06T10:00:53.781 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 info mirror/image_11@snap 2024-04-06T10:00:53.851 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:rbd image 'image_11': 2024-04-06T10:00:53.851 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: size 128 MiB in 32 objects 2024-04-06T10:00:53.851 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: order 22 (4 MiB objects) 2024-04-06T10:00:53.851 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: snapshot_count: 1 2024-04-06T10:00:53.851 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: id: 10ac70b134fa 2024-04-06T10:00:53.852 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: block_name_prefix: rbd_data.10ac70b134fa 2024-04-06T10:00:53.852 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: format: 2 2024-04-06T10:00:53.852 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: features: layering, exclusive-lock, journaling 2024-04-06T10:00:53.852 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: op_features: 2024-04-06T10:00:53.852 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: flags: 2024-04-06T10:00:53.852 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: create_timestamp: Sat Apr 6 09:52:26 2024 2024-04-06T10:00:53.852 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: access_timestamp: Sat Apr 6 09:52:26 2024 2024-04-06T10:00:53.852 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: modify_timestamp: Sat Apr 6 09:53:26 2024 2024-04-06T10:00:53.852 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: protected: False 2024-04-06T10:00:53.852 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: journal: 10ac70b134fa 2024-04-06T10:00:53.852 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: mirroring state: enabled 2024-04-06T10:00:53.852 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: mirroring mode: journal 2024-04-06T10:00:53.852 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: mirroring global id: 42f18cd7-c0fb-4a5e-982f-901df33520ed 2024-04-06T10:00:53.852 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: mirroring primary: false 2024-04-06T10:00:53.857 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T10:00:53.857 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ compare_image_snaps mirror image_11 snap 2024-04-06T10:00:53.857 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:00:53.857 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_11 2024-04-06T10:00:53.857 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local snap_name=snap 2024-04-06T10:00:53.857 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local ret=0 2024-04-06T10:00:53.857 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local rmt_export=/tmp/tmp.BhW6xKmROU/cluster2-mirror-image_11.export 2024-04-06T10:00:53.857 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loc_export=/tmp/tmp.BhW6xKmROU/cluster1-mirror-image_11.export 2024-04-06T10:00:53.857 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rm -f /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_11.export /tmp/tmp.BhW6xKmROU/cluster1-mirror-image_11.export 2024-04-06T10:00:53.858 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster2 -p mirror export image_11@snap /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_11.export 2024-04-06T10:00:54.030 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-04-06T10:00:54.038 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 -p mirror export image_11@snap /tmp/tmp.BhW6xKmROU/cluster1-mirror-image_11.export 2024-04-06T10:00:54.393 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-04-06T10:00:54.406 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmp /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_11.export /tmp/tmp.BhW6xKmROU/cluster1-mirror-image_11.export 2024-04-06T10:00:54.487 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rm -f /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_11.export /tmp/tmp.BhW6xKmROU/cluster1-mirror-image_11.export 2024-04-06T10:00:54.528 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T10:00:54.528 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-06T10:00:54.528 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ image=image_12 2024-04-06T10:00:54.528 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ create_snap cluster2 mirror image_12 snap 2024-04-06T10:00:54.528 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-04-06T10:00:54.528 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:00:54.528 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_12 2024-04-06T10:00:54.528 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local snap_name=snap 2024-04-06T10:00:54.528 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster2 -p mirror snap create image_12@snap --debug-rbd=20 --debug-journaler=20 2024-04-06T10:00:55.739 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_image_replay_started cluster1 mirror image_12 2024-04-06T10:00:55.739 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T10:00:55.739 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:00:55.739 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_12 2024-04-06T10:00:55.739 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_image_replay_state cluster1 mirror image_12 started 2024-04-06T10:00:55.739 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T10:00:55.739 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:00:55.739 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_12 2024-04-06T10:00:55.739 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state=started 2024-04-06T10:00:55.739 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local s 2024-04-06T10:00:55.739 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T10:00:55.740 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 1 2024-04-06T10:00:56.741 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test_image_replay_state cluster1 mirror image_12 started 2024-04-06T10:00:56.741 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T10:00:56.741 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:00:56.741 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_12 2024-04-06T10:00:56.741 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local test_state=started 2024-04-06T10:00:56.741 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local status_result 2024-04-06T10:00:56.741 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local current_state=stopped 2024-04-06T10:00:56.742 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_12 2024-04-06T10:00:56.742 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -i state 2024-04-06T10:00:56.742 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_instance=cluster1 2024-04-06T10:00:56.742 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ shift 2024-04-06T10:00:56.743 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T10:00:56.743 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance=cluster1 2024-04-06T10:00:56.743 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local loop_instance 2024-04-06T10:00:56.743 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T10:00:56.743 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sleep 0 2024-04-06T10:00:56.743 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-04-06T10:00:56.744 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ seq 0 3 2024-04-06T10:00:56.746 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T10:00:56.746 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_12 2024-04-06T10:00:56.747 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1:0 2024-04-06T10:00:56.747 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ shift 2024-04-06T10:00:56.747 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T10:00:56.747 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-04-06T10:00:56.747 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T10:00:56.747 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=cluster 2024-04-06T10:00:56.747 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T10:00:56.747 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:00:56.747 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T10:00:56.747 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T10:00:56.747 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval cluster=cluster1 2024-04-06T10:00:56.747 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ cluster=cluster1 2024-04-06T10:00:56.747 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T10:00:56.747 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T10:00:56.748 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-04-06T10:00:56.748 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local local_cluster=cluster1:0 2024-04-06T10:00:56.748 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local cluster=cluster1 2024-04-06T10:00:56.748 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local instance 2024-04-06T10:00:56.748 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T10:00:56.748 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local val=cluster1:0 2024-04-06T10:00:56.748 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local cluster_var_name=local_cluster 2024-04-06T10:00:56.749 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local instance_var_name=instance 2024-04-06T10:00:56.749 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ cluster=cluster1 2024-04-06T10:00:56.749 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T10:00:56.749 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ '[' 0 = cluster1:0 ']' 2024-04-06T10:00:56.749 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ eval local_cluster=cluster1 2024-04-06T10:00:56.749 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++++ local_cluster=cluster1 2024-04-06T10:00:56.749 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ eval instance=0 2024-04-06T10:00:56.749 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++++ instance=0 2024-04-06T10:00:56.750 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T10:00:56.769 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T10:00:56.769 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T10:00:56.769 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T10:00:56.769 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_12 2024-04-06T10:00:56.875 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:Invalid command: unused arguments: ['mirror/image_12'] 2024-04-06T10:00:56.875 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:rbd mirror status : get status for rbd mirror 2024-04-06T10:00:56.875 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:admin_socket: invalid command 2024-04-06T10:00:56.877 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T10:00:56.877 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_12 2024-04-06T10:00:56.877 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1:1 2024-04-06T10:00:56.877 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ shift 2024-04-06T10:00:56.877 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T10:00:56.878 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-04-06T10:00:56.878 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:1 2024-04-06T10:00:56.878 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=cluster 2024-04-06T10:00:56.878 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T10:00:56.878 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:00:56.878 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=1 2024-04-06T10:00:56.878 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-06T10:00:56.878 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval cluster=cluster1 2024-04-06T10:00:56.878 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ cluster=cluster1 2024-04-06T10:00:56.878 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=1 2024-04-06T10:00:56.878 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=1 2024-04-06T10:00:56.880 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-04-06T10:00:56.880 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local local_cluster=cluster1:1 2024-04-06T10:00:56.880 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local cluster=cluster1 2024-04-06T10:00:56.880 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local instance 2024-04-06T10:00:56.880 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-06T10:00:56.880 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local val=cluster1:1 2024-04-06T10:00:56.880 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local cluster_var_name=local_cluster 2024-04-06T10:00:56.880 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local instance_var_name=instance 2024-04-06T10:00:56.880 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ cluster=cluster1 2024-04-06T10:00:56.880 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=1 2024-04-06T10:00:56.880 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ '[' 1 = cluster1:1 ']' 2024-04-06T10:00:56.880 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ eval local_cluster=cluster1 2024-04-06T10:00:56.880 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++++ local_cluster=cluster1 2024-04-06T10:00:56.881 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ eval instance=1 2024-04-06T10:00:56.881 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++++ instance=1 2024-04-06T10:00:56.881 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-06T10:00:56.903 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T10:00:56.903 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T10:00:56.903 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T10:00:56.903 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_12 2024-04-06T10:00:57.050 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ return 0 2024-04-06T10:00:57.051 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ status_result=' "state": "Replaying"' 2024-04-06T10:00:57.051 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ echo ' "state": "Replaying"' 2024-04-06T10:00:57.052 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep -i Replaying 2024-04-06T10:00:57.053 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: "state": "Replaying" 2024-04-06T10:00:57.055 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ current_state=started 2024-04-06T10:00:57.055 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test started = started 2024-04-06T10:00:57.055 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T10:00:57.056 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_12 2024-04-06T10:00:57.056 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local local_cluster=cluster1 2024-04-06T10:00:57.056 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-04-06T10:00:57.056 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:00:57.056 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_12 2024-04-06T10:00:57.056 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' journal = journal ']' 2024-04-06T10:00:57.056 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_journal_replay_complete cluster1 cluster2 mirror image_12 2024-04-06T10:00:57.056 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local local_cluster=cluster1 2024-04-06T10:00:57.056 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-04-06T10:00:57.056 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:00:57.056 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_12 2024-04-06T10:00:57.057 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local s master_pos mirror_pos last_mirror_pos 2024-04-06T10:00:57.057 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local master_tag master_entry mirror_tag mirror_entry 2024-04-06T10:00:57.057 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T10:00:57.057 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T10:00:57.057 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T10:00:57.256 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror image_12 2024-04-06T10:00:57.256 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T10:00:57.256 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:00:57.256 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_12 2024-04-06T10:00:57.257 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T10:00:57.257 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n image_12 ']' 2024-04-06T10:00:57.257 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/image_12' 2024-04-06T10:00:57.257 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_12 2024-04-06T10:00:57.257 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T10:00:57.257 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T10:00:57.257 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T10:00:57.257 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T10:00:57.257 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T10:00:57.257 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T10:00:57.257 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T10:00:57.258 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T10:00:57.258 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T10:00:57.262 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T10:00:57.262 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_12 2024-04-06T10:00:57.262 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T10:00:57.262 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T10:00:57.262 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T10:00:57.262 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T10:00:57.262 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T10:00:57.262 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T10:00:57.262 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T10:00:57.262 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T10:00:57.262 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T10:00:57.263 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T10:00:57.263 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T10:00:57.263 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:00:57.263 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T10:00:57.263 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T10:00:57.263 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T10:00:57.263 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T10:00:57.263 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T10:00:57.263 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T10:00:57.263 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T10:00:57.263 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T10:00:57.263 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T10:00:57.263 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T10:00:57.263 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:00:57.263 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T10:00:57.264 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T10:00:57.264 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T10:00:57.264 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T10:00:57.264 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T10:00:57.264 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T10:00:57.264 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T10:00:57.287 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T10:00:57.287 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T10:00:57.287 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T10:00:57.287 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_12 2024-04-06T10:00:57.390 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:Invalid command: unused arguments: ['mirror/image_12'] 2024-04-06T10:00:57.390 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:rbd mirror flush : flush rbd mirror 2024-04-06T10:00:57.390 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:admin_socket: invalid command 2024-04-06T10:00:57.392 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T10:00:57.392 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/image_12 2024-04-06T10:00:57.392 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:1 2024-04-06T10:00:57.392 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T10:00:57.392 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T10:00:57.392 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-06T10:00:57.392 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:1 2024-04-06T10:00:57.393 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T10:00:57.393 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T10:00:57.393 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T10:00:57.393 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=1 2024-04-06T10:00:57.393 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-06T10:00:57.393 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T10:00:57.393 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:00:57.393 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=1 2024-04-06T10:00:57.393 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=1 2024-04-06T10:00:57.394 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-06T10:00:57.394 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:1 2024-04-06T10:00:57.394 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T10:00:57.394 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T10:00:57.394 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-06T10:00:57.395 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:1 2024-04-06T10:00:57.395 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T10:00:57.395 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T10:00:57.395 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:00:57.395 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=1 2024-04-06T10:00:57.395 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-06T10:00:57.395 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T10:00:57.395 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T10:00:57.395 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=1 2024-04-06T10:00:57.395 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=1 2024-04-06T10:00:57.395 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-06T10:00:57.414 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T10:00:57.415 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T10:00:57.415 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T10:00:57.415 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/image_12 2024-04-06T10:00:57.556 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T10:00:57.557 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror image_12 2024-04-06T10:00:57.557 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:00:57.558 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:00:57.558 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_12 2024-04-06T10:00:57.558 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror image_12 '' 2024-04-06T10:00:57.558 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:00:57.558 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:00:57.558 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_12 2024-04-06T10:00:57.558 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T10:00:57.559 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-image_12.status 2024-04-06T10:00:57.559 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-image_12.status 2024-04-06T10:00:57.559 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T10:00:57.561 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-image_12.status 2024-04-06T10:00:57.561 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_12 2024-04-06T10:00:57.561 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_12.status 2024-04-06T10:00:57.606 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 1 2024-04-06T10:00:57.606 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 1 2024-04-06T10:00:57.606 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T10:00:57.606 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=5, tag_tid=2, entry_tid=1], [object_number=4, tag_tid=2, entry_tid=0], [object_number=7, tag_tid=1, entry_tid=4711], [object_number=6, tag_tid=1, entry_tid=4710]]], state=connected] 2024-04-06T10:00:57.606 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=7, tag_tid=1, entry_tid=4711], [object_number=6, tag_tid=1, entry_tid=4710], [object_number=5, tag_tid=1, entry_tid=4709], [object_number=4, tag_tid=1, entry_tid=4708]]], state=connected] 2024-04-06T10:00:57.610 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_12.status 2024-04-06T10:00:57.611 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=5, tag_tid=2, entry_tid=1' 2024-04-06T10:00:57.612 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror image_12 2024-04-06T10:00:57.612 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:00:57.612 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:00:57.612 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_12 2024-04-06T10:00:57.612 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror image_12 '..*' 2024-04-06T10:00:57.613 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:00:57.613 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:00:57.613 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_12 2024-04-06T10:00:57.613 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T10:00:57.613 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-image_12.status 2024-04-06T10:00:57.613 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-image_12.status 2024-04-06T10:00:57.613 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T10:00:57.613 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-image_12.status 2024-04-06T10:00:57.614 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_12 2024-04-06T10:00:57.614 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_12.status 2024-04-06T10:00:57.659 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 1 2024-04-06T10:00:57.659 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 1 2024-04-06T10:00:57.659 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T10:00:57.659 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=5, tag_tid=2, entry_tid=1], [object_number=4, tag_tid=2, entry_tid=0], [object_number=7, tag_tid=1, entry_tid=4711], [object_number=6, tag_tid=1, entry_tid=4710]]], state=connected] 2024-04-06T10:00:57.659 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=7, tag_tid=1, entry_tid=4711], [object_number=6, tag_tid=1, entry_tid=4710], [object_number=5, tag_tid=1, entry_tid=4709], [object_number=4, tag_tid=1, entry_tid=4708]]], state=connected] 2024-04-06T10:00:57.662 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_12.status 2024-04-06T10:00:57.664 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=7, tag_tid=1, entry_tid=4711' 2024-04-06T10:00:57.664 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=5, tag_tid=2, entry_tid=1' -a 'object_number=5, tag_tid=2, entry_tid=1' = 'object_number=7, tag_tid=1, entry_tid=4711' 2024-04-06T10:00:57.664 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=7, tag_tid=1, entry_tid=4711' '!=' '' 2024-04-06T10:00:57.664 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T10:00:57.664 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=7, tag_tid=1, entry_tid=4711' = '' 2024-04-06T10:00:57.665 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=7, tag_tid=1, entry_tid=4711' 2024-04-06T10:00:57.665 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=5, tag_tid=2, entry_tid=1' 2024-04-06T10:00:57.665 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T10:00:57.665 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T10:00:57.665 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=2 2024-04-06T10:00:57.666 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=7, tag_tid=1, entry_tid=4711' 2024-04-06T10:00:57.666 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T10:00:57.666 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T10:00:57.667 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=1 2024-04-06T10:00:57.668 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=5, tag_tid=2, entry_tid=1' 2024-04-06T10:00:57.668 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T10:00:57.668 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T10:00:57.669 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T10:00:57.669 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=7, tag_tid=1, entry_tid=4711' 2024-04-06T10:00:57.669 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T10:00:57.670 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T10:00:57.671 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=4711 2024-04-06T10:00:57.671 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 2 = 1 -a 1 -le 4711 2024-04-06T10:00:57.671 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T10:00:57.671 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T10:00:57.671 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T10:00:57.872 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror image_12 2024-04-06T10:00:57.872 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T10:00:57.873 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:00:57.873 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_12 2024-04-06T10:00:57.873 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T10:00:57.873 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n image_12 ']' 2024-04-06T10:00:57.873 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/image_12' 2024-04-06T10:00:57.873 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_12 2024-04-06T10:00:57.873 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T10:00:57.873 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T10:00:57.873 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T10:00:57.873 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T10:00:57.873 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T10:00:57.873 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T10:00:57.873 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T10:00:57.874 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T10:00:57.874 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T10:00:57.877 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T10:00:57.877 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_12 2024-04-06T10:00:57.877 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T10:00:57.877 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T10:00:57.878 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T10:00:57.878 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T10:00:57.878 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T10:00:57.878 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T10:00:57.878 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T10:00:57.878 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T10:00:57.878 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T10:00:57.878 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T10:00:57.878 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T10:00:57.878 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:00:57.878 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T10:00:57.878 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T10:00:57.878 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T10:00:57.879 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T10:00:57.879 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T10:00:57.879 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T10:00:57.879 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T10:00:57.879 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T10:00:57.879 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T10:00:57.879 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T10:00:57.879 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:00:57.879 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T10:00:57.879 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T10:00:57.879 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T10:00:57.879 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T10:00:57.879 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T10:00:57.879 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T10:00:57.880 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T10:00:57.902 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T10:00:57.903 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T10:00:57.903 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T10:00:57.903 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_12 2024-04-06T10:00:58.007 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:Invalid command: unused arguments: ['mirror/image_12'] 2024-04-06T10:00:58.007 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:rbd mirror flush : flush rbd mirror 2024-04-06T10:00:58.007 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:admin_socket: invalid command 2024-04-06T10:00:58.010 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T10:00:58.010 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/image_12 2024-04-06T10:00:58.010 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:1 2024-04-06T10:00:58.010 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T10:00:58.010 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T10:00:58.010 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-06T10:00:58.010 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:1 2024-04-06T10:00:58.010 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T10:00:58.010 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T10:00:58.011 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T10:00:58.011 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=1 2024-04-06T10:00:58.011 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-06T10:00:58.011 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T10:00:58.011 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:00:58.011 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=1 2024-04-06T10:00:58.011 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=1 2024-04-06T10:00:58.012 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-06T10:00:58.012 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:1 2024-04-06T10:00:58.012 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T10:00:58.012 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T10:00:58.012 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-06T10:00:58.013 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:1 2024-04-06T10:00:58.013 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T10:00:58.013 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T10:00:58.013 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:00:58.013 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=1 2024-04-06T10:00:58.013 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-06T10:00:58.013 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T10:00:58.013 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T10:00:58.013 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=1 2024-04-06T10:00:58.013 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=1 2024-04-06T10:00:58.013 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-06T10:00:58.035 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T10:00:58.035 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T10:00:58.035 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T10:00:58.035 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/image_12 2024-04-06T10:00:58.184 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T10:00:58.185 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror image_12 2024-04-06T10:00:58.185 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:00:58.185 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:00:58.185 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_12 2024-04-06T10:00:58.185 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror image_12 '' 2024-04-06T10:00:58.185 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:00:58.186 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:00:58.186 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_12 2024-04-06T10:00:58.186 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T10:00:58.186 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-image_12.status 2024-04-06T10:00:58.186 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-image_12.status 2024-04-06T10:00:58.186 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T10:00:58.188 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-image_12.status 2024-04-06T10:00:58.188 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_12 2024-04-06T10:00:58.188 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_12.status 2024-04-06T10:00:58.238 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 1 2024-04-06T10:00:58.238 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 1 2024-04-06T10:00:58.238 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T10:00:58.238 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=5, tag_tid=2, entry_tid=1], [object_number=4, tag_tid=2, entry_tid=0], [object_number=7, tag_tid=1, entry_tid=4711], [object_number=6, tag_tid=1, entry_tid=4710]]], state=connected] 2024-04-06T10:00:58.239 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=7, tag_tid=1, entry_tid=4711], [object_number=6, tag_tid=1, entry_tid=4710], [object_number=5, tag_tid=1, entry_tid=4709], [object_number=4, tag_tid=1, entry_tid=4708]]], state=connected] 2024-04-06T10:00:58.242 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_12.status 2024-04-06T10:00:58.243 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=5, tag_tid=2, entry_tid=1' 2024-04-06T10:00:58.244 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror image_12 2024-04-06T10:00:58.244 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:00:58.244 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:00:58.245 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_12 2024-04-06T10:00:58.245 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror image_12 '..*' 2024-04-06T10:00:58.245 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:00:58.245 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:00:58.245 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_12 2024-04-06T10:00:58.245 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T10:00:58.245 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-image_12.status 2024-04-06T10:00:58.245 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-image_12.status 2024-04-06T10:00:58.245 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T10:00:58.245 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-image_12.status 2024-04-06T10:00:58.246 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_12 2024-04-06T10:00:58.246 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_12.status 2024-04-06T10:00:58.291 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 1 2024-04-06T10:00:58.291 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 1 2024-04-06T10:00:58.291 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T10:00:58.292 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=5, tag_tid=2, entry_tid=1], [object_number=4, tag_tid=2, entry_tid=0], [object_number=7, tag_tid=1, entry_tid=4711], [object_number=6, tag_tid=1, entry_tid=4710]]], state=connected] 2024-04-06T10:00:58.292 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=7, tag_tid=1, entry_tid=4711], [object_number=6, tag_tid=1, entry_tid=4710], [object_number=5, tag_tid=1, entry_tid=4709], [object_number=4, tag_tid=1, entry_tid=4708]]], state=connected] 2024-04-06T10:00:58.295 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_12.status 2024-04-06T10:00:58.297 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=7, tag_tid=1, entry_tid=4711' 2024-04-06T10:00:58.297 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=5, tag_tid=2, entry_tid=1' -a 'object_number=5, tag_tid=2, entry_tid=1' = 'object_number=7, tag_tid=1, entry_tid=4711' 2024-04-06T10:00:58.297 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=7, tag_tid=1, entry_tid=4711' '!=' 'object_number=7, tag_tid=1, entry_tid=4711' 2024-04-06T10:00:58.297 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T10:00:58.297 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.4 2024-04-06T10:00:58.699 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror image_12 2024-04-06T10:00:58.699 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T10:00:58.700 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:00:58.700 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_12 2024-04-06T10:00:58.700 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T10:00:58.700 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n image_12 ']' 2024-04-06T10:00:58.700 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/image_12' 2024-04-06T10:00:58.700 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_12 2024-04-06T10:00:58.700 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T10:00:58.700 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T10:00:58.700 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T10:00:58.700 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T10:00:58.700 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T10:00:58.700 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T10:00:58.701 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T10:00:58.701 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T10:00:58.701 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T10:00:58.704 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T10:00:58.704 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_12 2024-04-06T10:00:58.704 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T10:00:58.704 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T10:00:58.705 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T10:00:58.705 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T10:00:58.705 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T10:00:58.705 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T10:00:58.705 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T10:00:58.705 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T10:00:58.705 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T10:00:58.705 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T10:00:58.705 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T10:00:58.705 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:00:58.705 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T10:00:58.705 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T10:00:58.706 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T10:00:58.706 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T10:00:58.706 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T10:00:58.706 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T10:00:58.706 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T10:00:58.706 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T10:00:58.706 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T10:00:58.706 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T10:00:58.706 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:00:58.706 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T10:00:58.706 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T10:00:58.706 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T10:00:58.707 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T10:00:58.707 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T10:00:58.707 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T10:00:58.707 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T10:00:58.730 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T10:00:58.730 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T10:00:58.730 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T10:00:58.730 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_12 2024-04-06T10:00:58.835 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:Invalid command: unused arguments: ['mirror/image_12'] 2024-04-06T10:00:58.835 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:rbd mirror flush : flush rbd mirror 2024-04-06T10:00:58.835 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:admin_socket: invalid command 2024-04-06T10:00:58.837 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T10:00:58.837 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/image_12 2024-04-06T10:00:58.838 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:1 2024-04-06T10:00:58.838 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T10:00:58.838 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T10:00:58.838 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-06T10:00:58.838 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:1 2024-04-06T10:00:58.838 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T10:00:58.838 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T10:00:58.838 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T10:00:58.838 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=1 2024-04-06T10:00:58.838 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-06T10:00:58.838 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T10:00:58.838 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:00:58.838 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=1 2024-04-06T10:00:58.839 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=1 2024-04-06T10:00:58.840 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-06T10:00:58.840 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:1 2024-04-06T10:00:58.840 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T10:00:58.840 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T10:00:58.840 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-06T10:00:58.840 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:1 2024-04-06T10:00:58.840 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T10:00:58.840 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T10:00:58.840 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:00:58.840 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=1 2024-04-06T10:00:58.841 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-06T10:00:58.841 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T10:00:58.841 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T10:00:58.841 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=1 2024-04-06T10:00:58.841 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=1 2024-04-06T10:00:58.841 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-06T10:00:58.865 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T10:00:58.865 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T10:00:58.865 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T10:00:58.866 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/image_12 2024-04-06T10:00:59.000 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T10:00:59.001 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror image_12 2024-04-06T10:00:59.001 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:00:59.001 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:00:59.001 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_12 2024-04-06T10:00:59.001 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror image_12 '' 2024-04-06T10:00:59.001 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:00:59.001 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:00:59.002 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_12 2024-04-06T10:00:59.002 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T10:00:59.002 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-image_12.status 2024-04-06T10:00:59.002 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-image_12.status 2024-04-06T10:00:59.002 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T10:00:59.005 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-image_12.status 2024-04-06T10:00:59.005 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_12 2024-04-06T10:00:59.005 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_12.status 2024-04-06T10:00:59.051 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 1 2024-04-06T10:00:59.051 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 1 2024-04-06T10:00:59.051 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T10:00:59.051 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=5, tag_tid=2, entry_tid=1], [object_number=4, tag_tid=2, entry_tid=0], [object_number=7, tag_tid=1, entry_tid=4711], [object_number=6, tag_tid=1, entry_tid=4710]]], state=connected] 2024-04-06T10:00:59.051 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=7, tag_tid=1, entry_tid=4711], [object_number=6, tag_tid=1, entry_tid=4710], [object_number=5, tag_tid=1, entry_tid=4709], [object_number=4, tag_tid=1, entry_tid=4708]]], state=connected] 2024-04-06T10:00:59.055 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_12.status 2024-04-06T10:00:59.056 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=5, tag_tid=2, entry_tid=1' 2024-04-06T10:00:59.057 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror image_12 2024-04-06T10:00:59.057 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:00:59.057 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:00:59.057 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_12 2024-04-06T10:00:59.057 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror image_12 '..*' 2024-04-06T10:00:59.057 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:00:59.057 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:00:59.058 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_12 2024-04-06T10:00:59.058 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T10:00:59.058 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-image_12.status 2024-04-06T10:00:59.058 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-image_12.status 2024-04-06T10:00:59.058 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T10:00:59.059 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-image_12.status 2024-04-06T10:00:59.059 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_12 2024-04-06T10:00:59.059 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_12.status 2024-04-06T10:00:59.104 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 1 2024-04-06T10:00:59.104 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 1 2024-04-06T10:00:59.104 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T10:00:59.104 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=5, tag_tid=2, entry_tid=1], [object_number=4, tag_tid=2, entry_tid=0], [object_number=7, tag_tid=1, entry_tid=4711], [object_number=6, tag_tid=1, entry_tid=4710]]], state=connected] 2024-04-06T10:00:59.104 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=7, tag_tid=1, entry_tid=4711], [object_number=6, tag_tid=1, entry_tid=4710], [object_number=5, tag_tid=1, entry_tid=4709], [object_number=4, tag_tid=1, entry_tid=4708]]], state=connected] 2024-04-06T10:00:59.107 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_12.status 2024-04-06T10:00:59.109 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=7, tag_tid=1, entry_tid=4711' 2024-04-06T10:00:59.109 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=5, tag_tid=2, entry_tid=1' -a 'object_number=5, tag_tid=2, entry_tid=1' = 'object_number=7, tag_tid=1, entry_tid=4711' 2024-04-06T10:00:59.110 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=7, tag_tid=1, entry_tid=4711' '!=' 'object_number=7, tag_tid=1, entry_tid=4711' 2024-04-06T10:00:59.110 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T10:00:59.110 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.8 2024-04-06T10:00:59.911 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror image_12 2024-04-06T10:00:59.911 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T10:00:59.911 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:00:59.911 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_12 2024-04-06T10:00:59.911 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T10:00:59.911 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n image_12 ']' 2024-04-06T10:00:59.911 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/image_12' 2024-04-06T10:00:59.912 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_12 2024-04-06T10:00:59.912 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T10:00:59.912 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T10:00:59.912 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T10:00:59.912 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T10:00:59.912 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T10:00:59.913 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T10:00:59.913 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T10:00:59.918 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T10:00:59.918 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T10:00:59.918 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T10:00:59.919 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_12 2024-04-06T10:00:59.919 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T10:00:59.919 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T10:00:59.919 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T10:00:59.919 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T10:00:59.919 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T10:00:59.919 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T10:00:59.919 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T10:00:59.919 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T10:00:59.919 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T10:00:59.919 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T10:00:59.919 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T10:00:59.919 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:00:59.919 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T10:00:59.919 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T10:00:59.920 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T10:00:59.920 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T10:00:59.920 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T10:00:59.920 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T10:00:59.920 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T10:00:59.920 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T10:00:59.920 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T10:00:59.920 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T10:00:59.920 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:00:59.920 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T10:00:59.920 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T10:00:59.920 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T10:00:59.920 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T10:00:59.920 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T10:00:59.921 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T10:00:59.921 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T10:00:59.943 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T10:00:59.944 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T10:00:59.944 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T10:00:59.944 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_12 2024-04-06T10:01:00.050 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:Invalid command: unused arguments: ['mirror/image_12'] 2024-04-06T10:01:00.050 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:rbd mirror flush : flush rbd mirror 2024-04-06T10:01:00.050 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:admin_socket: invalid command 2024-04-06T10:01:00.053 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T10:01:00.053 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/image_12 2024-04-06T10:01:00.053 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:1 2024-04-06T10:01:00.053 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T10:01:00.053 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T10:01:00.053 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-06T10:01:00.053 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:1 2024-04-06T10:01:00.054 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T10:01:00.054 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T10:01:00.054 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T10:01:00.054 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=1 2024-04-06T10:01:00.054 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-06T10:01:00.054 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T10:01:00.054 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:01:00.054 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=1 2024-04-06T10:01:00.054 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=1 2024-04-06T10:01:00.054 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-06T10:01:00.054 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:1 2024-04-06T10:01:00.054 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T10:01:00.054 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T10:01:00.054 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-06T10:01:00.055 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:1 2024-04-06T10:01:00.055 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T10:01:00.055 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T10:01:00.055 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:01:00.055 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=1 2024-04-06T10:01:00.055 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-06T10:01:00.055 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T10:01:00.055 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T10:01:00.055 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=1 2024-04-06T10:01:00.055 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=1 2024-04-06T10:01:00.055 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-06T10:01:00.075 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T10:01:00.075 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T10:01:00.075 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T10:01:00.075 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/image_12 2024-04-06T10:01:00.216 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T10:01:00.217 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror image_12 2024-04-06T10:01:00.217 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:01:00.217 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:01:00.217 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_12 2024-04-06T10:01:00.217 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror image_12 '' 2024-04-06T10:01:00.217 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:01:00.217 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:01:00.217 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_12 2024-04-06T10:01:00.217 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T10:01:00.218 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-image_12.status 2024-04-06T10:01:00.218 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-image_12.status 2024-04-06T10:01:00.218 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T10:01:00.220 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-image_12.status 2024-04-06T10:01:00.221 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_12 2024-04-06T10:01:00.221 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_12.status 2024-04-06T10:01:00.266 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 1 2024-04-06T10:01:00.266 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 1 2024-04-06T10:01:00.266 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T10:01:00.266 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=5, tag_tid=2, entry_tid=1], [object_number=4, tag_tid=2, entry_tid=0], [object_number=7, tag_tid=1, entry_tid=4711], [object_number=6, tag_tid=1, entry_tid=4710]]], state=connected] 2024-04-06T10:01:00.266 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=7, tag_tid=1, entry_tid=4711], [object_number=6, tag_tid=1, entry_tid=4710], [object_number=5, tag_tid=1, entry_tid=4709], [object_number=4, tag_tid=1, entry_tid=4708]]], state=connected] 2024-04-06T10:01:00.270 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_12.status 2024-04-06T10:01:00.271 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=5, tag_tid=2, entry_tid=1' 2024-04-06T10:01:00.272 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror image_12 2024-04-06T10:01:00.272 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:01:00.273 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:01:00.273 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_12 2024-04-06T10:01:00.273 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror image_12 '..*' 2024-04-06T10:01:00.273 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:01:00.273 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:01:00.273 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_12 2024-04-06T10:01:00.273 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T10:01:00.273 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-image_12.status 2024-04-06T10:01:00.273 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-image_12.status 2024-04-06T10:01:00.273 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T10:01:00.274 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-image_12.status 2024-04-06T10:01:00.274 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_12 2024-04-06T10:01:00.274 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_12.status 2024-04-06T10:01:00.319 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 1 2024-04-06T10:01:00.319 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 1 2024-04-06T10:01:00.319 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T10:01:00.319 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=5, tag_tid=2, entry_tid=1], [object_number=4, tag_tid=2, entry_tid=0], [object_number=7, tag_tid=1, entry_tid=4711], [object_number=6, tag_tid=1, entry_tid=4710]]], state=connected] 2024-04-06T10:01:00.319 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=7, tag_tid=1, entry_tid=4711], [object_number=6, tag_tid=1, entry_tid=4710], [object_number=5, tag_tid=1, entry_tid=4709], [object_number=4, tag_tid=1, entry_tid=4708]]], state=connected] 2024-04-06T10:01:00.322 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_12.status 2024-04-06T10:01:00.324 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=7, tag_tid=1, entry_tid=4711' 2024-04-06T10:01:00.324 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=5, tag_tid=2, entry_tid=1' -a 'object_number=5, tag_tid=2, entry_tid=1' = 'object_number=7, tag_tid=1, entry_tid=4711' 2024-04-06T10:01:00.324 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=7, tag_tid=1, entry_tid=4711' '!=' 'object_number=7, tag_tid=1, entry_tid=4711' 2024-04-06T10:01:00.324 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T10:01:00.324 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 1.6 2024-04-06T10:01:01.926 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror image_12 2024-04-06T10:01:01.926 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T10:01:01.926 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:01:01.926 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_12 2024-04-06T10:01:01.927 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T10:01:01.927 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n image_12 ']' 2024-04-06T10:01:01.927 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/image_12' 2024-04-06T10:01:01.927 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_12 2024-04-06T10:01:01.927 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T10:01:01.927 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T10:01:01.927 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T10:01:01.927 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T10:01:01.927 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T10:01:01.927 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T10:01:01.927 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T10:01:01.928 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T10:01:01.928 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T10:01:01.931 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T10:01:01.931 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_12 2024-04-06T10:01:01.931 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T10:01:01.931 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T10:01:01.931 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T10:01:01.931 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T10:01:01.932 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T10:01:01.932 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T10:01:01.932 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T10:01:01.932 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T10:01:01.932 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T10:01:01.932 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T10:01:01.932 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T10:01:01.932 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:01:01.932 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T10:01:01.932 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T10:01:01.932 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T10:01:01.932 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T10:01:01.932 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T10:01:01.932 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T10:01:01.932 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T10:01:01.933 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T10:01:01.933 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T10:01:01.933 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T10:01:01.933 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:01:01.933 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T10:01:01.933 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T10:01:01.933 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T10:01:01.933 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T10:01:01.933 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T10:01:01.933 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T10:01:01.933 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T10:01:01.956 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T10:01:01.957 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T10:01:01.957 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T10:01:01.957 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_12 2024-04-06T10:01:02.061 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:Invalid command: unused arguments: ['mirror/image_12'] 2024-04-06T10:01:02.061 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:rbd mirror flush : flush rbd mirror 2024-04-06T10:01:02.061 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:admin_socket: invalid command 2024-04-06T10:01:02.063 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T10:01:02.063 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/image_12 2024-04-06T10:01:02.063 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:1 2024-04-06T10:01:02.063 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T10:01:02.063 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T10:01:02.064 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-06T10:01:02.064 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:1 2024-04-06T10:01:02.064 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T10:01:02.064 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T10:01:02.064 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T10:01:02.064 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=1 2024-04-06T10:01:02.064 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-06T10:01:02.064 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T10:01:02.064 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:01:02.064 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=1 2024-04-06T10:01:02.064 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=1 2024-04-06T10:01:02.066 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-06T10:01:02.066 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:1 2024-04-06T10:01:02.066 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T10:01:02.066 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T10:01:02.066 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-06T10:01:02.066 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:1 2024-04-06T10:01:02.066 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T10:01:02.066 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T10:01:02.066 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:01:02.066 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=1 2024-04-06T10:01:02.066 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-06T10:01:02.066 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T10:01:02.067 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T10:01:02.067 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=1 2024-04-06T10:01:02.067 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=1 2024-04-06T10:01:02.067 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-06T10:01:02.085 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T10:01:02.086 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T10:01:02.086 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T10:01:02.086 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/image_12 2024-04-06T10:01:02.236 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T10:01:02.237 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror image_12 2024-04-06T10:01:02.237 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:01:02.237 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:01:02.237 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_12 2024-04-06T10:01:02.238 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror image_12 '' 2024-04-06T10:01:02.238 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:01:02.238 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:01:02.238 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_12 2024-04-06T10:01:02.238 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T10:01:02.238 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-image_12.status 2024-04-06T10:01:02.239 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-image_12.status 2024-04-06T10:01:02.239 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T10:01:02.241 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-image_12.status 2024-04-06T10:01:02.241 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_12 2024-04-06T10:01:02.241 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_12.status 2024-04-06T10:01:02.291 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 1 2024-04-06T10:01:02.291 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 1 2024-04-06T10:01:02.291 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T10:01:02.291 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=5, tag_tid=2, entry_tid=1], [object_number=4, tag_tid=2, entry_tid=0], [object_number=7, tag_tid=1, entry_tid=4711], [object_number=6, tag_tid=1, entry_tid=4710]]], state=connected] 2024-04-06T10:01:02.291 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=5, tag_tid=2, entry_tid=1], [object_number=4, tag_tid=2, entry_tid=0], [object_number=7, tag_tid=1, entry_tid=4711], [object_number=6, tag_tid=1, entry_tid=4710]]], state=connected] 2024-04-06T10:01:02.295 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_12.status 2024-04-06T10:01:02.296 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=5, tag_tid=2, entry_tid=1' 2024-04-06T10:01:02.297 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror image_12 2024-04-06T10:01:02.297 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:01:02.297 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:01:02.298 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_12 2024-04-06T10:01:02.298 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror image_12 '..*' 2024-04-06T10:01:02.298 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:01:02.298 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:01:02.298 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_12 2024-04-06T10:01:02.298 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T10:01:02.298 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-image_12.status 2024-04-06T10:01:02.298 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-image_12.status 2024-04-06T10:01:02.298 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T10:01:02.299 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-image_12.status 2024-04-06T10:01:02.299 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_12 2024-04-06T10:01:02.299 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_12.status 2024-04-06T10:01:02.343 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 1 2024-04-06T10:01:02.343 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 1 2024-04-06T10:01:02.344 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T10:01:02.344 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=5, tag_tid=2, entry_tid=1], [object_number=4, tag_tid=2, entry_tid=0], [object_number=7, tag_tid=1, entry_tid=4711], [object_number=6, tag_tid=1, entry_tid=4710]]], state=connected] 2024-04-06T10:01:02.344 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=5, tag_tid=2, entry_tid=1], [object_number=4, tag_tid=2, entry_tid=0], [object_number=7, tag_tid=1, entry_tid=4711], [object_number=6, tag_tid=1, entry_tid=4710]]], state=connected] 2024-04-06T10:01:02.347 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_12.status 2024-04-06T10:01:02.352 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=5, tag_tid=2, entry_tid=1' 2024-04-06T10:01:02.352 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=5, tag_tid=2, entry_tid=1' -a 'object_number=5, tag_tid=2, entry_tid=1' = 'object_number=5, tag_tid=2, entry_tid=1' 2024-04-06T10:01:02.352 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T10:01:02.352 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_snap_present cluster1 mirror image_12 snap 2024-04-06T10:01:02.352 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T10:01:02.352 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:01:02.352 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_12 2024-04-06T10:01:02.352 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local snap_name=snap 2024-04-06T10:01:02.352 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local s 2024-04-06T10:01:02.352 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-04-06T10:01:02.352 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 1 2024-04-06T10:01:03.350 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 info mirror/image_12@snap 2024-04-06T10:01:03.419 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:rbd image 'image_12': 2024-04-06T10:01:03.419 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: size 128 MiB in 32 objects 2024-04-06T10:01:03.419 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: order 22 (4 MiB objects) 2024-04-06T10:01:03.420 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: snapshot_count: 1 2024-04-06T10:01:03.420 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: id: 10ac6bf2d8c3 2024-04-06T10:01:03.420 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: block_name_prefix: rbd_data.10ac6bf2d8c3 2024-04-06T10:01:03.420 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: format: 2 2024-04-06T10:01:03.420 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: features: layering, exclusive-lock, journaling 2024-04-06T10:01:03.420 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: op_features: 2024-04-06T10:01:03.420 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: flags: 2024-04-06T10:01:03.420 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: create_timestamp: Sat Apr 6 09:52:40 2024 2024-04-06T10:01:03.420 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: access_timestamp: Sat Apr 6 09:52:40 2024 2024-04-06T10:01:03.420 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: modify_timestamp: Sat Apr 6 09:52:40 2024 2024-04-06T10:01:03.420 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: protected: False 2024-04-06T10:01:03.420 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: journal: 10ac6bf2d8c3 2024-04-06T10:01:03.420 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: mirroring state: enabled 2024-04-06T10:01:03.420 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: mirroring mode: journal 2024-04-06T10:01:03.420 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: mirroring global id: 66c7b32b-5481-41da-b834-43f6ec04d33b 2024-04-06T10:01:03.421 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: mirroring primary: false 2024-04-06T10:01:03.425 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T10:01:03.425 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ compare_image_snaps mirror image_12 snap 2024-04-06T10:01:03.425 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:01:03.425 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_12 2024-04-06T10:01:03.425 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local snap_name=snap 2024-04-06T10:01:03.425 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local ret=0 2024-04-06T10:01:03.425 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local rmt_export=/tmp/tmp.BhW6xKmROU/cluster2-mirror-image_12.export 2024-04-06T10:01:03.425 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loc_export=/tmp/tmp.BhW6xKmROU/cluster1-mirror-image_12.export 2024-04-06T10:01:03.425 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rm -f /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_12.export /tmp/tmp.BhW6xKmROU/cluster1-mirror-image_12.export 2024-04-06T10:01:03.426 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster2 -p mirror export image_12@snap /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_12.export 2024-04-06T10:01:03.581 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-04-06T10:01:03.590 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 -p mirror export image_12@snap /tmp/tmp.BhW6xKmROU/cluster1-mirror-image_12.export 2024-04-06T10:01:03.914 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-04-06T10:01:03.925 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmp /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_12.export /tmp/tmp.BhW6xKmROU/cluster1-mirror-image_12.export 2024-04-06T10:01:04.002 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rm -f /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_12.export /tmp/tmp.BhW6xKmROU/cluster1-mirror-image_12.export 2024-04-06T10:01:04.036 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T10:01:04.036 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-06T10:01:04.036 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ image=image_13 2024-04-06T10:01:04.036 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ create_snap cluster2 mirror image_13 snap 2024-04-06T10:01:04.037 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-04-06T10:01:04.037 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:01:04.037 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_13 2024-04-06T10:01:04.037 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local snap_name=snap 2024-04-06T10:01:04.037 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster2 -p mirror snap create image_13@snap --debug-rbd=20 --debug-journaler=20 2024-04-06T10:01:04.859 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_image_replay_started cluster1 mirror image_13 2024-04-06T10:01:04.859 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T10:01:04.860 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:01:04.860 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_13 2024-04-06T10:01:04.860 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_image_replay_state cluster1 mirror image_13 started 2024-04-06T10:01:04.860 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T10:01:04.860 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:01:04.860 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_13 2024-04-06T10:01:04.861 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state=started 2024-04-06T10:01:04.861 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local s 2024-04-06T10:01:04.861 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T10:01:04.861 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 1 2024-04-06T10:01:05.861 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test_image_replay_state cluster1 mirror image_13 started 2024-04-06T10:01:05.862 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T10:01:05.862 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:01:05.862 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_13 2024-04-06T10:01:05.862 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local test_state=started 2024-04-06T10:01:05.862 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local status_result 2024-04-06T10:01:05.862 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local current_state=stopped 2024-04-06T10:01:05.863 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_13 2024-04-06T10:01:05.863 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_instance=cluster1 2024-04-06T10:01:05.863 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ shift 2024-04-06T10:01:05.864 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T10:01:05.864 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance=cluster1 2024-04-06T10:01:05.864 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local loop_instance 2024-04-06T10:01:05.864 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T10:01:05.864 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sleep 0 2024-04-06T10:01:05.864 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -i state 2024-04-06T10:01:05.864 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-04-06T10:01:05.865 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ seq 0 3 2024-04-06T10:01:05.867 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T10:01:05.867 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_13 2024-04-06T10:01:05.867 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1:0 2024-04-06T10:01:05.867 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ shift 2024-04-06T10:01:05.867 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T10:01:05.868 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-04-06T10:01:05.868 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T10:01:05.868 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=cluster 2024-04-06T10:01:05.868 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T10:01:05.868 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:01:05.868 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T10:01:05.868 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T10:01:05.868 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval cluster=cluster1 2024-04-06T10:01:05.868 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ cluster=cluster1 2024-04-06T10:01:05.868 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T10:01:05.868 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T10:01:05.869 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-04-06T10:01:05.870 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local local_cluster=cluster1:0 2024-04-06T10:01:05.870 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local cluster=cluster1 2024-04-06T10:01:05.870 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local instance 2024-04-06T10:01:05.870 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T10:01:05.870 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local val=cluster1:0 2024-04-06T10:01:05.870 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local cluster_var_name=local_cluster 2024-04-06T10:01:05.870 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local instance_var_name=instance 2024-04-06T10:01:05.870 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ cluster=cluster1 2024-04-06T10:01:05.870 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T10:01:05.870 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ '[' 0 = cluster1:0 ']' 2024-04-06T10:01:05.870 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ eval local_cluster=cluster1 2024-04-06T10:01:05.870 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++++ local_cluster=cluster1 2024-04-06T10:01:05.871 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ eval instance=0 2024-04-06T10:01:05.871 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++++ instance=0 2024-04-06T10:01:05.871 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T10:01:05.895 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T10:01:05.895 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T10:01:05.895 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T10:01:05.895 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_13 2024-04-06T10:01:06.001 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:Invalid command: unused arguments: ['mirror/image_13'] 2024-04-06T10:01:06.002 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:rbd mirror status : get status for rbd mirror 2024-04-06T10:01:06.002 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:admin_socket: invalid command 2024-04-06T10:01:06.005 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T10:01:06.006 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_13 2024-04-06T10:01:06.006 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1:1 2024-04-06T10:01:06.006 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ shift 2024-04-06T10:01:06.006 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T10:01:06.006 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-04-06T10:01:06.006 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:1 2024-04-06T10:01:06.006 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=cluster 2024-04-06T10:01:06.007 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T10:01:06.007 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:01:06.007 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=1 2024-04-06T10:01:06.007 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-06T10:01:06.007 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval cluster=cluster1 2024-04-06T10:01:06.007 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ cluster=cluster1 2024-04-06T10:01:06.007 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=1 2024-04-06T10:01:06.007 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=1 2024-04-06T10:01:06.007 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-04-06T10:01:06.007 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local local_cluster=cluster1:1 2024-04-06T10:01:06.007 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local cluster=cluster1 2024-04-06T10:01:06.007 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local instance 2024-04-06T10:01:06.007 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-06T10:01:06.007 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local val=cluster1:1 2024-04-06T10:01:06.008 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local cluster_var_name=local_cluster 2024-04-06T10:01:06.008 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local instance_var_name=instance 2024-04-06T10:01:06.008 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ cluster=cluster1 2024-04-06T10:01:06.008 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=1 2024-04-06T10:01:06.008 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ '[' 1 = cluster1:1 ']' 2024-04-06T10:01:06.008 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ eval local_cluster=cluster1 2024-04-06T10:01:06.008 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++++ local_cluster=cluster1 2024-04-06T10:01:06.008 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ eval instance=1 2024-04-06T10:01:06.008 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++++ instance=1 2024-04-06T10:01:06.008 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-06T10:01:06.025 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T10:01:06.025 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T10:01:06.025 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T10:01:06.025 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_13 2024-04-06T10:01:06.177 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ return 0 2024-04-06T10:01:06.177 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ status_result=' "state": "Replaying"' 2024-04-06T10:01:06.178 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ echo ' "state": "Replaying"' 2024-04-06T10:01:06.178 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep -i Replaying 2024-04-06T10:01:06.181 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: "state": "Replaying" 2024-04-06T10:01:06.183 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ current_state=started 2024-04-06T10:01:06.184 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test started = started 2024-04-06T10:01:06.184 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T10:01:06.184 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_13 2024-04-06T10:01:06.184 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local local_cluster=cluster1 2024-04-06T10:01:06.184 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-04-06T10:01:06.184 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:01:06.184 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_13 2024-04-06T10:01:06.184 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' journal = journal ']' 2024-04-06T10:01:06.184 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_journal_replay_complete cluster1 cluster2 mirror image_13 2024-04-06T10:01:06.184 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local local_cluster=cluster1 2024-04-06T10:01:06.184 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-04-06T10:01:06.184 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:01:06.184 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_13 2024-04-06T10:01:06.184 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local s master_pos mirror_pos last_mirror_pos 2024-04-06T10:01:06.185 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local master_tag master_entry mirror_tag mirror_entry 2024-04-06T10:01:06.185 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T10:01:06.185 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T10:01:06.185 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T10:01:06.383 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror image_13 2024-04-06T10:01:06.383 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T10:01:06.383 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:01:06.383 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_13 2024-04-06T10:01:06.383 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T10:01:06.383 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n image_13 ']' 2024-04-06T10:01:06.383 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/image_13' 2024-04-06T10:01:06.384 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_13 2024-04-06T10:01:06.384 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T10:01:06.384 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T10:01:06.384 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T10:01:06.384 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T10:01:06.384 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T10:01:06.384 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T10:01:06.384 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T10:01:06.384 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T10:01:06.385 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T10:01:06.387 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T10:01:06.387 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_13 2024-04-06T10:01:06.387 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T10:01:06.387 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T10:01:06.387 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T10:01:06.387 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T10:01:06.387 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T10:01:06.387 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T10:01:06.387 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T10:01:06.387 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T10:01:06.388 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T10:01:06.388 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T10:01:06.388 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T10:01:06.388 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:01:06.388 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T10:01:06.388 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T10:01:06.388 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T10:01:06.388 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T10:01:06.388 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T10:01:06.388 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T10:01:06.388 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T10:01:06.388 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T10:01:06.388 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T10:01:06.388 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T10:01:06.388 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:01:06.389 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T10:01:06.389 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T10:01:06.389 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T10:01:06.389 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T10:01:06.389 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T10:01:06.389 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T10:01:06.389 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T10:01:06.413 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T10:01:06.413 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T10:01:06.413 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T10:01:06.414 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_13 2024-04-06T10:01:06.520 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:Invalid command: unused arguments: ['mirror/image_13'] 2024-04-06T10:01:06.520 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:rbd mirror flush : flush rbd mirror 2024-04-06T10:01:06.520 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:admin_socket: invalid command 2024-04-06T10:01:06.522 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T10:01:06.522 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/image_13 2024-04-06T10:01:06.522 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:1 2024-04-06T10:01:06.522 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T10:01:06.522 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T10:01:06.523 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-06T10:01:06.523 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:1 2024-04-06T10:01:06.523 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T10:01:06.523 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T10:01:06.523 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T10:01:06.523 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=1 2024-04-06T10:01:06.523 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-06T10:01:06.523 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T10:01:06.523 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:01:06.523 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=1 2024-04-06T10:01:06.523 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=1 2024-04-06T10:01:06.524 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-06T10:01:06.525 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:1 2024-04-06T10:01:06.525 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T10:01:06.525 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T10:01:06.525 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-06T10:01:06.525 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:1 2024-04-06T10:01:06.525 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T10:01:06.525 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T10:01:06.525 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:01:06.525 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=1 2024-04-06T10:01:06.525 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-06T10:01:06.525 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T10:01:06.525 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T10:01:06.525 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=1 2024-04-06T10:01:06.525 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=1 2024-04-06T10:01:06.525 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-06T10:01:06.548 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T10:01:06.548 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T10:01:06.548 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T10:01:06.548 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/image_13 2024-04-06T10:01:06.696 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T10:01:06.697 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror image_13 2024-04-06T10:01:06.697 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:01:06.697 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:01:06.697 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_13 2024-04-06T10:01:06.697 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror image_13 '' 2024-04-06T10:01:06.697 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:01:06.697 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:01:06.697 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_13 2024-04-06T10:01:06.697 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T10:01:06.698 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-image_13.status 2024-04-06T10:01:06.698 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-image_13.status 2024-04-06T10:01:06.699 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T10:01:06.700 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-image_13.status 2024-04-06T10:01:06.701 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_13.status 2024-04-06T10:01:06.701 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_13 2024-04-06T10:01:06.959 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 1 2024-04-06T10:01:06.959 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 1 2024-04-06T10:01:06.959 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T10:01:06.960 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=5, tag_tid=3, entry_tid=1], [object_number=4, tag_tid=3, entry_tid=0], [object_number=7, tag_tid=1, entry_tid=5555], [object_number=6, tag_tid=1, entry_tid=5554]]], state=connected] 2024-04-06T10:01:06.960 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=5, tag_tid=2, entry_tid=1], [object_number=4, tag_tid=2, entry_tid=0], [object_number=7, tag_tid=1, entry_tid=5555], [object_number=6, tag_tid=1, entry_tid=5554]]], state=connected] 2024-04-06T10:01:06.963 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_13.status 2024-04-06T10:01:06.965 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=5, tag_tid=3, entry_tid=1' 2024-04-06T10:01:06.969 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror image_13 2024-04-06T10:01:06.969 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:01:06.969 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:01:06.969 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_13 2024-04-06T10:01:06.969 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror image_13 '..*' 2024-04-06T10:01:06.970 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:01:06.970 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:01:06.970 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_13 2024-04-06T10:01:06.970 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T10:01:06.970 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-image_13.status 2024-04-06T10:01:06.970 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-image_13.status 2024-04-06T10:01:06.971 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T10:01:06.971 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-image_13.status 2024-04-06T10:01:06.971 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_13 2024-04-06T10:01:06.971 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_13.status 2024-04-06T10:01:07.017 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 1 2024-04-06T10:01:07.018 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 1 2024-04-06T10:01:07.018 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T10:01:07.018 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=5, tag_tid=3, entry_tid=1], [object_number=4, tag_tid=3, entry_tid=0], [object_number=7, tag_tid=1, entry_tid=5555], [object_number=6, tag_tid=1, entry_tid=5554]]], state=connected] 2024-04-06T10:01:07.018 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=5, tag_tid=2, entry_tid=1], [object_number=4, tag_tid=2, entry_tid=0], [object_number=7, tag_tid=1, entry_tid=5555], [object_number=6, tag_tid=1, entry_tid=5554]]], state=connected] 2024-04-06T10:01:07.021 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_13.status 2024-04-06T10:01:07.024 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=5, tag_tid=2, entry_tid=1' 2024-04-06T10:01:07.024 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=5, tag_tid=3, entry_tid=1' -a 'object_number=5, tag_tid=3, entry_tid=1' = 'object_number=5, tag_tid=2, entry_tid=1' 2024-04-06T10:01:07.024 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=5, tag_tid=2, entry_tid=1' '!=' '' 2024-04-06T10:01:07.024 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T10:01:07.024 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=5, tag_tid=2, entry_tid=1' = '' 2024-04-06T10:01:07.024 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=5, tag_tid=2, entry_tid=1' 2024-04-06T10:01:07.024 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=5, tag_tid=3, entry_tid=1' 2024-04-06T10:01:07.024 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T10:01:07.024 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T10:01:07.025 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=3 2024-04-06T10:01:07.025 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=5, tag_tid=2, entry_tid=1' 2024-04-06T10:01:07.025 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T10:01:07.025 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T10:01:07.026 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=2 2024-04-06T10:01:07.027 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=5, tag_tid=3, entry_tid=1' 2024-04-06T10:01:07.027 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T10:01:07.027 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T10:01:07.028 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T10:01:07.029 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=5, tag_tid=2, entry_tid=1' 2024-04-06T10:01:07.029 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T10:01:07.029 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T10:01:07.031 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=1 2024-04-06T10:01:07.031 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 3 = 2 -a 1 -le 1 2024-04-06T10:01:07.031 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T10:01:07.031 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T10:01:07.031 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T10:01:07.232 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror image_13 2024-04-06T10:01:07.233 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T10:01:07.233 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:01:07.233 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_13 2024-04-06T10:01:07.233 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T10:01:07.233 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n image_13 ']' 2024-04-06T10:01:07.233 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/image_13' 2024-04-06T10:01:07.233 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_13 2024-04-06T10:01:07.233 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T10:01:07.233 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T10:01:07.233 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T10:01:07.233 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T10:01:07.233 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T10:01:07.233 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T10:01:07.234 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T10:01:07.234 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T10:01:07.234 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T10:01:07.238 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T10:01:07.238 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_13 2024-04-06T10:01:07.238 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T10:01:07.239 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T10:01:07.239 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T10:01:07.239 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T10:01:07.239 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T10:01:07.239 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T10:01:07.239 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T10:01:07.239 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T10:01:07.239 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T10:01:07.239 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T10:01:07.239 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T10:01:07.240 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:01:07.240 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T10:01:07.240 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T10:01:07.240 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T10:01:07.240 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T10:01:07.240 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T10:01:07.240 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T10:01:07.240 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T10:01:07.240 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T10:01:07.240 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T10:01:07.240 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T10:01:07.240 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:01:07.240 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T10:01:07.240 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T10:01:07.240 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T10:01:07.241 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T10:01:07.241 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T10:01:07.241 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T10:01:07.241 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T10:01:07.266 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T10:01:07.266 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T10:01:07.266 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T10:01:07.266 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_13 2024-04-06T10:01:07.372 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:Invalid command: unused arguments: ['mirror/image_13'] 2024-04-06T10:01:07.372 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:rbd mirror flush : flush rbd mirror 2024-04-06T10:01:07.372 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:admin_socket: invalid command 2024-04-06T10:01:07.377 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T10:01:07.377 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/image_13 2024-04-06T10:01:07.377 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:1 2024-04-06T10:01:07.377 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T10:01:07.377 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T10:01:07.377 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-06T10:01:07.377 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:1 2024-04-06T10:01:07.377 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T10:01:07.377 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T10:01:07.377 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T10:01:07.377 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=1 2024-04-06T10:01:07.378 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-06T10:01:07.378 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T10:01:07.378 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:01:07.378 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=1 2024-04-06T10:01:07.378 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=1 2024-04-06T10:01:07.378 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-06T10:01:07.378 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:1 2024-04-06T10:01:07.378 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T10:01:07.378 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T10:01:07.378 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-06T10:01:07.378 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:1 2024-04-06T10:01:07.378 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T10:01:07.378 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T10:01:07.378 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:01:07.379 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=1 2024-04-06T10:01:07.379 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-06T10:01:07.379 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T10:01:07.379 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T10:01:07.379 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=1 2024-04-06T10:01:07.379 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=1 2024-04-06T10:01:07.379 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-06T10:01:07.400 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T10:01:07.400 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T10:01:07.400 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T10:01:07.400 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/image_13 2024-04-06T10:01:07.537 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T10:01:07.538 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror image_13 2024-04-06T10:01:07.538 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:01:07.538 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:01:07.539 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_13 2024-04-06T10:01:07.539 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror image_13 '' 2024-04-06T10:01:07.539 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:01:07.539 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:01:07.539 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_13 2024-04-06T10:01:07.539 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T10:01:07.539 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-image_13.status 2024-04-06T10:01:07.539 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-image_13.status 2024-04-06T10:01:07.540 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T10:01:07.541 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-image_13.status 2024-04-06T10:01:07.542 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_13 2024-04-06T10:01:07.542 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_13.status 2024-04-06T10:01:07.591 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 1 2024-04-06T10:01:07.592 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 1 2024-04-06T10:01:07.592 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T10:01:07.592 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=5, tag_tid=3, entry_tid=1], [object_number=4, tag_tid=3, entry_tid=0], [object_number=7, tag_tid=1, entry_tid=5555], [object_number=6, tag_tid=1, entry_tid=5554]]], state=connected] 2024-04-06T10:01:07.592 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=5, tag_tid=2, entry_tid=1], [object_number=4, tag_tid=2, entry_tid=0], [object_number=7, tag_tid=1, entry_tid=5555], [object_number=6, tag_tid=1, entry_tid=5554]]], state=connected] 2024-04-06T10:01:07.595 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_13.status 2024-04-06T10:01:07.597 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=5, tag_tid=3, entry_tid=1' 2024-04-06T10:01:07.597 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror image_13 2024-04-06T10:01:07.598 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:01:07.598 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:01:07.598 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_13 2024-04-06T10:01:07.598 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror image_13 '..*' 2024-04-06T10:01:07.598 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:01:07.598 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:01:07.598 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_13 2024-04-06T10:01:07.598 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T10:01:07.598 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-image_13.status 2024-04-06T10:01:07.599 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-image_13.status 2024-04-06T10:01:07.599 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T10:01:07.599 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-image_13.status 2024-04-06T10:01:07.599 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_13 2024-04-06T10:01:07.599 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_13.status 2024-04-06T10:01:07.645 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 1 2024-04-06T10:01:07.645 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 1 2024-04-06T10:01:07.645 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T10:01:07.645 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=5, tag_tid=3, entry_tid=1], [object_number=4, tag_tid=3, entry_tid=0], [object_number=7, tag_tid=1, entry_tid=5555], [object_number=6, tag_tid=1, entry_tid=5554]]], state=connected] 2024-04-06T10:01:07.645 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=5, tag_tid=2, entry_tid=1], [object_number=4, tag_tid=2, entry_tid=0], [object_number=7, tag_tid=1, entry_tid=5555], [object_number=6, tag_tid=1, entry_tid=5554]]], state=connected] 2024-04-06T10:01:07.648 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_13.status 2024-04-06T10:01:07.650 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=5, tag_tid=2, entry_tid=1' 2024-04-06T10:01:07.650 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=5, tag_tid=3, entry_tid=1' -a 'object_number=5, tag_tid=3, entry_tid=1' = 'object_number=5, tag_tid=2, entry_tid=1' 2024-04-06T10:01:07.650 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=5, tag_tid=2, entry_tid=1' '!=' 'object_number=5, tag_tid=2, entry_tid=1' 2024-04-06T10:01:07.650 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T10:01:07.650 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.4 2024-04-06T10:01:08.052 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror image_13 2024-04-06T10:01:08.052 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T10:01:08.052 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:01:08.052 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_13 2024-04-06T10:01:08.052 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T10:01:08.052 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n image_13 ']' 2024-04-06T10:01:08.053 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/image_13' 2024-04-06T10:01:08.053 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_13 2024-04-06T10:01:08.053 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T10:01:08.053 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T10:01:08.053 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T10:01:08.053 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T10:01:08.053 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T10:01:08.053 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T10:01:08.053 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T10:01:08.054 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T10:01:08.054 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T10:01:08.058 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T10:01:08.058 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_13 2024-04-06T10:01:08.058 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T10:01:08.058 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T10:01:08.058 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T10:01:08.058 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T10:01:08.059 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T10:01:08.059 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T10:01:08.059 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T10:01:08.059 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T10:01:08.059 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T10:01:08.059 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T10:01:08.059 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T10:01:08.059 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:01:08.059 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T10:01:08.059 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T10:01:08.059 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T10:01:08.059 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T10:01:08.060 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T10:01:08.060 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T10:01:08.060 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T10:01:08.060 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T10:01:08.060 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T10:01:08.060 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T10:01:08.060 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:01:08.060 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T10:01:08.060 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T10:01:08.060 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T10:01:08.060 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T10:01:08.060 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T10:01:08.061 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T10:01:08.061 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T10:01:08.084 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T10:01:08.084 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T10:01:08.084 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T10:01:08.084 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_13 2024-04-06T10:01:08.188 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:Invalid command: unused arguments: ['mirror/image_13'] 2024-04-06T10:01:08.189 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:rbd mirror flush : flush rbd mirror 2024-04-06T10:01:08.189 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:admin_socket: invalid command 2024-04-06T10:01:08.195 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T10:01:08.195 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/image_13 2024-04-06T10:01:08.195 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:1 2024-04-06T10:01:08.196 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T10:01:08.196 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T10:01:08.196 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-06T10:01:08.196 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:1 2024-04-06T10:01:08.196 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T10:01:08.196 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T10:01:08.196 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T10:01:08.196 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=1 2024-04-06T10:01:08.196 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-06T10:01:08.196 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T10:01:08.196 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:01:08.197 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=1 2024-04-06T10:01:08.197 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=1 2024-04-06T10:01:08.197 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-06T10:01:08.197 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:1 2024-04-06T10:01:08.197 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T10:01:08.197 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T10:01:08.197 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-06T10:01:08.197 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:1 2024-04-06T10:01:08.197 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T10:01:08.197 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T10:01:08.197 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:01:08.197 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=1 2024-04-06T10:01:08.198 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-06T10:01:08.198 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T10:01:08.198 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T10:01:08.198 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=1 2024-04-06T10:01:08.198 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=1 2024-04-06T10:01:08.198 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-06T10:01:08.216 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T10:01:08.217 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T10:01:08.217 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T10:01:08.217 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/image_13 2024-04-06T10:01:08.360 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T10:01:08.361 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror image_13 2024-04-06T10:01:08.361 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:01:08.362 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:01:08.362 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_13 2024-04-06T10:01:08.362 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror image_13 '' 2024-04-06T10:01:08.362 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:01:08.362 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:01:08.362 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_13 2024-04-06T10:01:08.362 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T10:01:08.363 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-image_13.status 2024-04-06T10:01:08.363 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-image_13.status 2024-04-06T10:01:08.363 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T10:01:08.364 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-image_13.status 2024-04-06T10:01:08.365 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_13 2024-04-06T10:01:08.365 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_13.status 2024-04-06T10:01:08.410 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 1 2024-04-06T10:01:08.410 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 1 2024-04-06T10:01:08.410 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T10:01:08.411 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=5, tag_tid=3, entry_tid=1], [object_number=4, tag_tid=3, entry_tid=0], [object_number=7, tag_tid=1, entry_tid=5555], [object_number=6, tag_tid=1, entry_tid=5554]]], state=connected] 2024-04-06T10:01:08.411 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=5, tag_tid=2, entry_tid=1], [object_number=4, tag_tid=2, entry_tid=0], [object_number=7, tag_tid=1, entry_tid=5555], [object_number=6, tag_tid=1, entry_tid=5554]]], state=connected] 2024-04-06T10:01:08.414 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_13.status 2024-04-06T10:01:08.416 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=5, tag_tid=3, entry_tid=1' 2024-04-06T10:01:08.416 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror image_13 2024-04-06T10:01:08.416 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:01:08.416 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:01:08.416 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_13 2024-04-06T10:01:08.417 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror image_13 '..*' 2024-04-06T10:01:08.417 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:01:08.417 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:01:08.417 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_13 2024-04-06T10:01:08.417 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T10:01:08.417 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-image_13.status 2024-04-06T10:01:08.417 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-image_13.status 2024-04-06T10:01:08.417 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T10:01:08.418 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-image_13.status 2024-04-06T10:01:08.418 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_13 2024-04-06T10:01:08.418 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_13.status 2024-04-06T10:01:08.463 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 1 2024-04-06T10:01:08.463 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 1 2024-04-06T10:01:08.463 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T10:01:08.463 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=5, tag_tid=3, entry_tid=1], [object_number=4, tag_tid=3, entry_tid=0], [object_number=7, tag_tid=1, entry_tid=5555], [object_number=6, tag_tid=1, entry_tid=5554]]], state=connected] 2024-04-06T10:01:08.463 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=5, tag_tid=2, entry_tid=1], [object_number=4, tag_tid=2, entry_tid=0], [object_number=7, tag_tid=1, entry_tid=5555], [object_number=6, tag_tid=1, entry_tid=5554]]], state=connected] 2024-04-06T10:01:08.466 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_13.status 2024-04-06T10:01:08.468 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=5, tag_tid=2, entry_tid=1' 2024-04-06T10:01:08.468 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=5, tag_tid=3, entry_tid=1' -a 'object_number=5, tag_tid=3, entry_tid=1' = 'object_number=5, tag_tid=2, entry_tid=1' 2024-04-06T10:01:08.468 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=5, tag_tid=2, entry_tid=1' '!=' 'object_number=5, tag_tid=2, entry_tid=1' 2024-04-06T10:01:08.468 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T10:01:08.468 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.8 2024-04-06T10:01:09.271 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror image_13 2024-04-06T10:01:09.271 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T10:01:09.271 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:01:09.271 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_13 2024-04-06T10:01:09.271 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T10:01:09.271 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n image_13 ']' 2024-04-06T10:01:09.271 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/image_13' 2024-04-06T10:01:09.271 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_13 2024-04-06T10:01:09.271 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T10:01:09.271 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T10:01:09.272 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T10:01:09.272 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T10:01:09.272 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T10:01:09.272 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T10:01:09.272 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T10:01:09.272 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T10:01:09.272 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T10:01:09.274 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T10:01:09.274 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_13 2024-04-06T10:01:09.274 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T10:01:09.274 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T10:01:09.274 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T10:01:09.274 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T10:01:09.274 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T10:01:09.274 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T10:01:09.274 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T10:01:09.274 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T10:01:09.274 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T10:01:09.275 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T10:01:09.275 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T10:01:09.275 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:01:09.275 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T10:01:09.275 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T10:01:09.276 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T10:01:09.276 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T10:01:09.276 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T10:01:09.276 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T10:01:09.276 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T10:01:09.277 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T10:01:09.277 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T10:01:09.277 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T10:01:09.277 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:01:09.277 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T10:01:09.277 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T10:01:09.277 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T10:01:09.277 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T10:01:09.277 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T10:01:09.277 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T10:01:09.277 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T10:01:09.301 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T10:01:09.301 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T10:01:09.301 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T10:01:09.301 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_13 2024-04-06T10:01:09.407 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:Invalid command: unused arguments: ['mirror/image_13'] 2024-04-06T10:01:09.407 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:rbd mirror flush : flush rbd mirror 2024-04-06T10:01:09.407 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:admin_socket: invalid command 2024-04-06T10:01:09.411 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T10:01:09.411 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/image_13 2024-04-06T10:01:09.411 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:1 2024-04-06T10:01:09.411 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T10:01:09.411 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T10:01:09.411 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-06T10:01:09.411 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:1 2024-04-06T10:01:09.411 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T10:01:09.411 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T10:01:09.411 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T10:01:09.412 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=1 2024-04-06T10:01:09.412 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-06T10:01:09.412 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T10:01:09.412 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:01:09.412 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=1 2024-04-06T10:01:09.412 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=1 2024-04-06T10:01:09.412 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-06T10:01:09.412 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:1 2024-04-06T10:01:09.412 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T10:01:09.412 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T10:01:09.412 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-06T10:01:09.412 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:1 2024-04-06T10:01:09.412 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T10:01:09.412 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T10:01:09.412 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:01:09.413 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=1 2024-04-06T10:01:09.413 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-06T10:01:09.413 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T10:01:09.413 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T10:01:09.413 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=1 2024-04-06T10:01:09.413 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=1 2024-04-06T10:01:09.413 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-06T10:01:09.436 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T10:01:09.436 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T10:01:09.436 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T10:01:09.436 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/image_13 2024-04-06T10:01:09.574 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T10:01:09.575 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror image_13 2024-04-06T10:01:09.575 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:01:09.575 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:01:09.575 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_13 2024-04-06T10:01:09.575 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror image_13 '' 2024-04-06T10:01:09.575 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:01:09.575 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:01:09.576 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_13 2024-04-06T10:01:09.576 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T10:01:09.576 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-image_13.status 2024-04-06T10:01:09.576 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-image_13.status 2024-04-06T10:01:09.577 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T10:01:09.577 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-image_13.status 2024-04-06T10:01:09.578 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_13 2024-04-06T10:01:09.578 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_13.status 2024-04-06T10:01:09.622 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 1 2024-04-06T10:01:09.623 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 1 2024-04-06T10:01:09.623 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T10:01:09.623 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=5, tag_tid=3, entry_tid=1], [object_number=4, tag_tid=3, entry_tid=0], [object_number=7, tag_tid=1, entry_tid=5555], [object_number=6, tag_tid=1, entry_tid=5554]]], state=connected] 2024-04-06T10:01:09.623 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=5, tag_tid=2, entry_tid=1], [object_number=4, tag_tid=2, entry_tid=0], [object_number=7, tag_tid=1, entry_tid=5555], [object_number=6, tag_tid=1, entry_tid=5554]]], state=connected] 2024-04-06T10:01:09.626 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_13.status 2024-04-06T10:01:09.629 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=5, tag_tid=3, entry_tid=1' 2024-04-06T10:01:09.629 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror image_13 2024-04-06T10:01:09.629 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:01:09.629 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:01:09.629 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_13 2024-04-06T10:01:09.629 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror image_13 '..*' 2024-04-06T10:01:09.629 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:01:09.629 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:01:09.629 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_13 2024-04-06T10:01:09.629 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T10:01:09.630 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-image_13.status 2024-04-06T10:01:09.630 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-image_13.status 2024-04-06T10:01:09.630 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T10:01:09.630 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-image_13.status 2024-04-06T10:01:09.630 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_13 2024-04-06T10:01:09.630 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_13.status 2024-04-06T10:01:09.674 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 1 2024-04-06T10:01:09.674 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 1 2024-04-06T10:01:09.674 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T10:01:09.675 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=5, tag_tid=3, entry_tid=1], [object_number=4, tag_tid=3, entry_tid=0], [object_number=7, tag_tid=1, entry_tid=5555], [object_number=6, tag_tid=1, entry_tid=5554]]], state=connected] 2024-04-06T10:01:09.675 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=5, tag_tid=2, entry_tid=1], [object_number=4, tag_tid=2, entry_tid=0], [object_number=7, tag_tid=1, entry_tid=5555], [object_number=6, tag_tid=1, entry_tid=5554]]], state=connected] 2024-04-06T10:01:09.678 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_13.status 2024-04-06T10:01:09.679 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=5, tag_tid=2, entry_tid=1' 2024-04-06T10:01:09.680 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=5, tag_tid=3, entry_tid=1' -a 'object_number=5, tag_tid=3, entry_tid=1' = 'object_number=5, tag_tid=2, entry_tid=1' 2024-04-06T10:01:09.680 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=5, tag_tid=2, entry_tid=1' '!=' 'object_number=5, tag_tid=2, entry_tid=1' 2024-04-06T10:01:09.680 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T10:01:09.680 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 1.6 2024-04-06T10:01:11.281 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror image_13 2024-04-06T10:01:11.281 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T10:01:11.281 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:01:11.281 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_13 2024-04-06T10:01:11.282 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T10:01:11.282 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n image_13 ']' 2024-04-06T10:01:11.282 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/image_13' 2024-04-06T10:01:11.282 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_13 2024-04-06T10:01:11.282 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T10:01:11.282 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T10:01:11.282 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T10:01:11.282 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T10:01:11.282 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T10:01:11.282 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T10:01:11.282 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T10:01:11.283 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T10:01:11.283 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T10:01:11.286 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T10:01:11.286 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_13 2024-04-06T10:01:11.287 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T10:01:11.287 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T10:01:11.287 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T10:01:11.287 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T10:01:11.287 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T10:01:11.287 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T10:01:11.287 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T10:01:11.287 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T10:01:11.287 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T10:01:11.287 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T10:01:11.287 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T10:01:11.287 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:01:11.287 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T10:01:11.287 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T10:01:11.288 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T10:01:11.288 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T10:01:11.288 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T10:01:11.288 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T10:01:11.288 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T10:01:11.288 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T10:01:11.288 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T10:01:11.288 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T10:01:11.288 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:01:11.288 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T10:01:11.288 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T10:01:11.288 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T10:01:11.288 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T10:01:11.288 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T10:01:11.288 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T10:01:11.288 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T10:01:11.312 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T10:01:11.313 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T10:01:11.313 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T10:01:11.313 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_13 2024-04-06T10:01:11.418 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:Invalid command: unused arguments: ['mirror/image_13'] 2024-04-06T10:01:11.418 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:rbd mirror flush : flush rbd mirror 2024-04-06T10:01:11.418 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:admin_socket: invalid command 2024-04-06T10:01:11.420 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T10:01:11.420 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/image_13 2024-04-06T10:01:11.421 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:1 2024-04-06T10:01:11.421 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T10:01:11.421 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T10:01:11.421 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-06T10:01:11.421 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:1 2024-04-06T10:01:11.421 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T10:01:11.421 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T10:01:11.421 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T10:01:11.421 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=1 2024-04-06T10:01:11.421 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-06T10:01:11.421 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T10:01:11.421 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:01:11.421 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=1 2024-04-06T10:01:11.421 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=1 2024-04-06T10:01:11.423 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-06T10:01:11.423 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:1 2024-04-06T10:01:11.423 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T10:01:11.423 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T10:01:11.423 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-06T10:01:11.423 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:1 2024-04-06T10:01:11.423 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T10:01:11.424 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T10:01:11.424 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:01:11.424 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=1 2024-04-06T10:01:11.424 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-06T10:01:11.424 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T10:01:11.424 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T10:01:11.424 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=1 2024-04-06T10:01:11.424 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=1 2024-04-06T10:01:11.424 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-06T10:01:11.446 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T10:01:11.447 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T10:01:11.447 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T10:01:11.447 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/image_13 2024-04-06T10:01:11.592 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T10:01:11.594 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror image_13 2024-04-06T10:01:11.594 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:01:11.594 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:01:11.594 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_13 2024-04-06T10:01:11.595 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror image_13 '' 2024-04-06T10:01:11.595 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:01:11.595 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:01:11.596 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_13 2024-04-06T10:01:11.596 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T10:01:11.596 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-image_13.status 2024-04-06T10:01:11.596 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-image_13.status 2024-04-06T10:01:11.597 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T10:01:11.598 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-image_13.status 2024-04-06T10:01:11.598 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_13 2024-04-06T10:01:11.598 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_13.status 2024-04-06T10:01:11.648 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 1 2024-04-06T10:01:11.648 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 1 2024-04-06T10:01:11.648 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T10:01:11.648 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=5, tag_tid=3, entry_tid=1], [object_number=4, tag_tid=3, entry_tid=0], [object_number=7, tag_tid=1, entry_tid=5555], [object_number=6, tag_tid=1, entry_tid=5554]]], state=connected] 2024-04-06T10:01:11.649 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=5, tag_tid=2, entry_tid=1], [object_number=4, tag_tid=2, entry_tid=0], [object_number=7, tag_tid=1, entry_tid=5555], [object_number=6, tag_tid=1, entry_tid=5554]]], state=connected] 2024-04-06T10:01:11.652 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_13.status 2024-04-06T10:01:11.653 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=5, tag_tid=3, entry_tid=1' 2024-04-06T10:01:11.654 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror image_13 2024-04-06T10:01:11.654 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:01:11.654 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:01:11.654 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_13 2024-04-06T10:01:11.654 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror image_13 '..*' 2024-04-06T10:01:11.655 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:01:11.655 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:01:11.655 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_13 2024-04-06T10:01:11.655 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T10:01:11.655 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-image_13.status 2024-04-06T10:01:11.655 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-image_13.status 2024-04-06T10:01:11.655 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T10:01:11.656 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-image_13.status 2024-04-06T10:01:11.656 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_13 2024-04-06T10:01:11.656 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_13.status 2024-04-06T10:01:11.701 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 1 2024-04-06T10:01:11.701 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 1 2024-04-06T10:01:11.701 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T10:01:11.701 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=5, tag_tid=3, entry_tid=1], [object_number=4, tag_tid=3, entry_tid=0], [object_number=7, tag_tid=1, entry_tid=5555], [object_number=6, tag_tid=1, entry_tid=5554]]], state=connected] 2024-04-06T10:01:11.702 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=5, tag_tid=2, entry_tid=1], [object_number=4, tag_tid=2, entry_tid=0], [object_number=7, tag_tid=1, entry_tid=5555], [object_number=6, tag_tid=1, entry_tid=5554]]], state=connected] 2024-04-06T10:01:11.705 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_13.status 2024-04-06T10:01:11.706 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=5, tag_tid=2, entry_tid=1' 2024-04-06T10:01:11.707 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=5, tag_tid=3, entry_tid=1' -a 'object_number=5, tag_tid=3, entry_tid=1' = 'object_number=5, tag_tid=2, entry_tid=1' 2024-04-06T10:01:11.707 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=5, tag_tid=2, entry_tid=1' '!=' 'object_number=5, tag_tid=2, entry_tid=1' 2024-04-06T10:01:11.707 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T10:01:11.707 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 2 2024-04-06T10:01:12.649 INFO:tasks.rbd_mirror_thrash:kill cluster1.client.mirror.0 2024-04-06T10:01:12.650 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Sent signal 15 2024-04-06T10:01:12.650 INFO:tasks.rbd_mirror_thrash:waiting for 8 secs before reviving daemons 2024-04-06T10:01:12.651 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi192.stderr:2024-04-06T10:01:12.650+0000 7f67c73fc640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.0 (PID: 135331) UID: 1000 2024-04-06T10:01:13.709 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror image_13 2024-04-06T10:01:13.709 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T10:01:13.709 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:01:13.709 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_13 2024-04-06T10:01:13.709 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T10:01:13.709 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n image_13 ']' 2024-04-06T10:01:13.709 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/image_13' 2024-04-06T10:01:13.709 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_13 2024-04-06T10:01:13.709 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T10:01:13.709 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T10:01:13.709 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T10:01:13.709 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T10:01:13.710 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T10:01:13.710 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T10:01:13.710 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T10:01:13.710 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T10:01:13.710 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T10:01:13.713 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T10:01:13.713 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_13 2024-04-06T10:01:13.713 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T10:01:13.714 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T10:01:13.714 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T10:01:13.714 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T10:01:13.714 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T10:01:13.714 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T10:01:13.714 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T10:01:13.714 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T10:01:13.714 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T10:01:13.714 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T10:01:13.714 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T10:01:13.714 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:01:13.714 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T10:01:13.714 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T10:01:13.714 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T10:01:13.715 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T10:01:13.715 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T10:01:13.715 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T10:01:13.715 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T10:01:13.715 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T10:01:13.715 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T10:01:13.715 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T10:01:13.715 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:01:13.715 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T10:01:13.715 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T10:01:13.715 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T10:01:13.715 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T10:01:13.715 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T10:01:13.715 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T10:01:13.715 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T10:01:13.738 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T10:01:13.739 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T10:01:13.739 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T10:01:13.739 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_13 2024-04-06T10:01:13.836 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:admin_socket: exception getting command descriptions: [Errno 2] No such file or directory 2024-04-06T10:01:13.840 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T10:01:13.840 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/image_13 2024-04-06T10:01:13.840 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:1 2024-04-06T10:01:13.840 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T10:01:13.840 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T10:01:13.840 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-06T10:01:13.840 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:1 2024-04-06T10:01:13.840 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T10:01:13.840 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T10:01:13.840 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T10:01:13.841 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=1 2024-04-06T10:01:13.841 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-06T10:01:13.841 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T10:01:13.841 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:01:13.841 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=1 2024-04-06T10:01:13.841 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=1 2024-04-06T10:01:13.841 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-06T10:01:13.841 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:1 2024-04-06T10:01:13.841 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T10:01:13.841 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T10:01:13.841 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-06T10:01:13.841 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:1 2024-04-06T10:01:13.841 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T10:01:13.842 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T10:01:13.842 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:01:13.842 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=1 2024-04-06T10:01:13.842 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-06T10:01:13.842 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T10:01:13.842 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T10:01:13.842 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=1 2024-04-06T10:01:13.842 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=1 2024-04-06T10:01:13.842 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-06T10:01:13.864 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T10:01:13.864 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T10:01:13.864 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T10:01:13.864 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/image_13 2024-04-06T10:01:14.007 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T10:01:14.008 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror image_13 2024-04-06T10:01:14.008 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:01:14.008 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:01:14.008 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_13 2024-04-06T10:01:14.008 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror image_13 '' 2024-04-06T10:01:14.008 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:01:14.009 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:01:14.009 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_13 2024-04-06T10:01:14.009 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T10:01:14.009 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-image_13.status 2024-04-06T10:01:14.009 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-image_13.status 2024-04-06T10:01:14.009 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T10:01:14.010 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-image_13.status 2024-04-06T10:01:14.011 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_13 2024-04-06T10:01:14.011 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_13.status 2024-04-06T10:01:14.062 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 1 2024-04-06T10:01:14.062 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 1 2024-04-06T10:01:14.062 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T10:01:14.062 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=5, tag_tid=3, entry_tid=1], [object_number=4, tag_tid=3, entry_tid=0], [object_number=7, tag_tid=1, entry_tid=5555], [object_number=6, tag_tid=1, entry_tid=5554]]], state=connected] 2024-04-06T10:01:14.062 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=5, tag_tid=3, entry_tid=1], [object_number=4, tag_tid=3, entry_tid=0], [object_number=7, tag_tid=1, entry_tid=5555], [object_number=6, tag_tid=1, entry_tid=5554]]], state=connected] 2024-04-06T10:01:14.065 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_13.status 2024-04-06T10:01:14.066 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=5, tag_tid=3, entry_tid=1' 2024-04-06T10:01:14.067 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror image_13 2024-04-06T10:01:14.067 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:01:14.067 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:01:14.067 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_13 2024-04-06T10:01:14.067 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror image_13 '..*' 2024-04-06T10:01:14.068 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:01:14.068 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:01:14.068 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_13 2024-04-06T10:01:14.068 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T10:01:14.068 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-image_13.status 2024-04-06T10:01:14.068 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-image_13.status 2024-04-06T10:01:14.068 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T10:01:14.068 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-image_13.status 2024-04-06T10:01:14.069 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_13 2024-04-06T10:01:14.069 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_13.status 2024-04-06T10:01:14.114 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 1 2024-04-06T10:01:14.114 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 1 2024-04-06T10:01:14.114 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T10:01:14.115 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=5, tag_tid=3, entry_tid=1], [object_number=4, tag_tid=3, entry_tid=0], [object_number=7, tag_tid=1, entry_tid=5555], [object_number=6, tag_tid=1, entry_tid=5554]]], state=connected] 2024-04-06T10:01:14.115 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=5, tag_tid=3, entry_tid=1], [object_number=4, tag_tid=3, entry_tid=0], [object_number=7, tag_tid=1, entry_tid=5555], [object_number=6, tag_tid=1, entry_tid=5554]]], state=connected] 2024-04-06T10:01:14.118 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_13.status 2024-04-06T10:01:14.120 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=5, tag_tid=3, entry_tid=1' 2024-04-06T10:01:14.120 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=5, tag_tid=3, entry_tid=1' -a 'object_number=5, tag_tid=3, entry_tid=1' = 'object_number=5, tag_tid=3, entry_tid=1' 2024-04-06T10:01:14.120 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T10:01:14.120 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_snap_present cluster1 mirror image_13 snap 2024-04-06T10:01:14.120 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T10:01:14.120 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:01:14.120 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_13 2024-04-06T10:01:14.120 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local snap_name=snap 2024-04-06T10:01:14.120 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local s 2024-04-06T10:01:14.121 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-04-06T10:01:14.121 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 1 2024-04-06T10:01:15.121 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 info mirror/image_13@snap 2024-04-06T10:01:15.190 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:rbd image 'image_13': 2024-04-06T10:01:15.190 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: size 128 MiB in 32 objects 2024-04-06T10:01:15.190 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: order 22 (4 MiB objects) 2024-04-06T10:01:15.190 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: snapshot_count: 1 2024-04-06T10:01:15.190 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: id: 10accabe13d2 2024-04-06T10:01:15.190 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: block_name_prefix: rbd_data.10accabe13d2 2024-04-06T10:01:15.190 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: format: 2 2024-04-06T10:01:15.190 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: features: layering, exclusive-lock, journaling 2024-04-06T10:01:15.190 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: op_features: 2024-04-06T10:01:15.191 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: flags: 2024-04-06T10:01:15.191 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: create_timestamp: Sat Apr 6 09:52:46 2024 2024-04-06T10:01:15.191 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: access_timestamp: Sat Apr 6 09:52:46 2024 2024-04-06T10:01:15.191 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: modify_timestamp: Sat Apr 6 09:53:47 2024 2024-04-06T10:01:15.191 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: protected: False 2024-04-06T10:01:15.191 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: journal: 10accabe13d2 2024-04-06T10:01:15.191 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: mirroring state: enabled 2024-04-06T10:01:15.191 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: mirroring mode: journal 2024-04-06T10:01:15.191 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: mirroring global id: caca7bbf-36fc-4492-9238-3e136ca92124 2024-04-06T10:01:15.191 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: mirroring primary: false 2024-04-06T10:01:15.194 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T10:01:15.195 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ compare_image_snaps mirror image_13 snap 2024-04-06T10:01:15.195 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:01:15.195 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_13 2024-04-06T10:01:15.195 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local snap_name=snap 2024-04-06T10:01:15.195 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local ret=0 2024-04-06T10:01:15.195 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local rmt_export=/tmp/tmp.BhW6xKmROU/cluster2-mirror-image_13.export 2024-04-06T10:01:15.195 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loc_export=/tmp/tmp.BhW6xKmROU/cluster1-mirror-image_13.export 2024-04-06T10:01:15.195 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rm -f /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_13.export /tmp/tmp.BhW6xKmROU/cluster1-mirror-image_13.export 2024-04-06T10:01:15.196 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster2 -p mirror export image_13@snap /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_13.export 2024-04-06T10:01:15.362 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-04-06T10:01:15.370 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 -p mirror export image_13@snap /tmp/tmp.BhW6xKmROU/cluster1-mirror-image_13.export 2024-04-06T10:01:15.619 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-04-06T10:01:15.630 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmp /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_13.export /tmp/tmp.BhW6xKmROU/cluster1-mirror-image_13.export 2024-04-06T10:01:15.709 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rm -f /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_13.export /tmp/tmp.BhW6xKmROU/cluster1-mirror-image_13.export 2024-04-06T10:01:15.751 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T10:01:15.751 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-06T10:01:15.751 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ image=image_14 2024-04-06T10:01:15.751 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ create_snap cluster2 mirror image_14 snap 2024-04-06T10:01:15.751 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-04-06T10:01:15.751 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:01:15.751 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_14 2024-04-06T10:01:15.751 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local snap_name=snap 2024-04-06T10:01:15.751 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster2 -p mirror snap create image_14@snap --debug-rbd=20 --debug-journaler=20 2024-04-06T10:01:18.890 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_image_replay_started cluster1 mirror image_14 2024-04-06T10:01:18.891 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T10:01:18.891 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:01:18.891 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_14 2024-04-06T10:01:18.891 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_image_replay_state cluster1 mirror image_14 started 2024-04-06T10:01:18.891 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T10:01:18.891 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:01:18.891 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_14 2024-04-06T10:01:18.891 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state=started 2024-04-06T10:01:18.891 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local s 2024-04-06T10:01:18.891 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T10:01:18.891 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 1 2024-04-06T10:01:19.893 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test_image_replay_state cluster1 mirror image_14 started 2024-04-06T10:01:19.893 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T10:01:19.893 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:01:19.893 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_14 2024-04-06T10:01:19.893 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local test_state=started 2024-04-06T10:01:19.893 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local status_result 2024-04-06T10:01:19.893 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local current_state=stopped 2024-04-06T10:01:19.894 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_14 2024-04-06T10:01:19.894 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -i state 2024-04-06T10:01:19.894 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_instance=cluster1 2024-04-06T10:01:19.894 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ shift 2024-04-06T10:01:19.894 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T10:01:19.894 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance=cluster1 2024-04-06T10:01:19.894 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local loop_instance 2024-04-06T10:01:19.895 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T10:01:19.895 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sleep 0 2024-04-06T10:01:19.896 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-04-06T10:01:19.896 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ seq 0 3 2024-04-06T10:01:19.899 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T10:01:19.899 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_14 2024-04-06T10:01:19.899 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1:0 2024-04-06T10:01:19.899 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ shift 2024-04-06T10:01:19.899 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T10:01:19.899 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-04-06T10:01:19.899 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T10:01:19.899 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=cluster 2024-04-06T10:01:19.899 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T10:01:19.899 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:01:19.899 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T10:01:19.899 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T10:01:19.900 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval cluster=cluster1 2024-04-06T10:01:19.900 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ cluster=cluster1 2024-04-06T10:01:19.900 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T10:01:19.900 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T10:01:19.900 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-04-06T10:01:19.900 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local local_cluster=cluster1:0 2024-04-06T10:01:19.900 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local cluster=cluster1 2024-04-06T10:01:19.900 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local instance 2024-04-06T10:01:19.900 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T10:01:19.900 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local val=cluster1:0 2024-04-06T10:01:19.900 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local cluster_var_name=local_cluster 2024-04-06T10:01:19.900 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local instance_var_name=instance 2024-04-06T10:01:19.900 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ cluster=cluster1 2024-04-06T10:01:19.900 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T10:01:19.901 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ '[' 0 = cluster1:0 ']' 2024-04-06T10:01:19.901 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ eval local_cluster=cluster1 2024-04-06T10:01:19.901 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++++ local_cluster=cluster1 2024-04-06T10:01:19.901 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ eval instance=0 2024-04-06T10:01:19.901 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++++ instance=0 2024-04-06T10:01:19.902 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T10:01:19.921 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T10:01:19.922 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T10:01:19.922 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T10:01:19.922 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_14 2024-04-06T10:01:20.018 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:admin_socket: exception getting command descriptions: [Errno 2] No such file or directory 2024-04-06T10:01:20.022 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T10:01:20.022 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_14 2024-04-06T10:01:20.022 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1:1 2024-04-06T10:01:20.022 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ shift 2024-04-06T10:01:20.022 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T10:01:20.022 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-04-06T10:01:20.022 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:1 2024-04-06T10:01:20.022 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=cluster 2024-04-06T10:01:20.022 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T10:01:20.023 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:01:20.023 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=1 2024-04-06T10:01:20.023 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-06T10:01:20.023 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval cluster=cluster1 2024-04-06T10:01:20.023 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ cluster=cluster1 2024-04-06T10:01:20.023 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=1 2024-04-06T10:01:20.023 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=1 2024-04-06T10:01:20.023 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-04-06T10:01:20.023 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local local_cluster=cluster1:1 2024-04-06T10:01:20.023 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local cluster=cluster1 2024-04-06T10:01:20.023 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local instance 2024-04-06T10:01:20.023 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-06T10:01:20.023 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local val=cluster1:1 2024-04-06T10:01:20.023 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local cluster_var_name=local_cluster 2024-04-06T10:01:20.023 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local instance_var_name=instance 2024-04-06T10:01:20.024 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ cluster=cluster1 2024-04-06T10:01:20.024 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=1 2024-04-06T10:01:20.024 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ '[' 1 = cluster1:1 ']' 2024-04-06T10:01:20.024 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ eval local_cluster=cluster1 2024-04-06T10:01:20.024 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++++ local_cluster=cluster1 2024-04-06T10:01:20.024 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ eval instance=1 2024-04-06T10:01:20.024 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++++ instance=1 2024-04-06T10:01:20.024 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-06T10:01:20.045 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T10:01:20.045 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T10:01:20.045 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T10:01:20.045 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_14 2024-04-06T10:01:20.200 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ return 0 2024-04-06T10:01:20.201 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ status_result=' "state": "Replaying"' 2024-04-06T10:01:20.201 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ echo ' "state": "Replaying"' 2024-04-06T10:01:20.202 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep -i Replaying 2024-04-06T10:01:20.204 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: "state": "Replaying" 2024-04-06T10:01:20.205 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ current_state=started 2024-04-06T10:01:20.205 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test started = started 2024-04-06T10:01:20.205 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T10:01:20.205 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_14 2024-04-06T10:01:20.205 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local local_cluster=cluster1 2024-04-06T10:01:20.205 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-04-06T10:01:20.206 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:01:20.206 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_14 2024-04-06T10:01:20.206 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' journal = journal ']' 2024-04-06T10:01:20.206 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_journal_replay_complete cluster1 cluster2 mirror image_14 2024-04-06T10:01:20.206 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local local_cluster=cluster1 2024-04-06T10:01:20.206 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-04-06T10:01:20.206 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:01:20.206 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_14 2024-04-06T10:01:20.206 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local s master_pos mirror_pos last_mirror_pos 2024-04-06T10:01:20.206 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local master_tag master_entry mirror_tag mirror_entry 2024-04-06T10:01:20.206 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T10:01:20.206 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T10:01:20.206 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T10:01:20.407 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror image_14 2024-04-06T10:01:20.407 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T10:01:20.407 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:01:20.407 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_14 2024-04-06T10:01:20.407 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T10:01:20.407 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n image_14 ']' 2024-04-06T10:01:20.407 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/image_14' 2024-04-06T10:01:20.408 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_14 2024-04-06T10:01:20.408 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T10:01:20.408 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T10:01:20.408 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T10:01:20.408 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T10:01:20.408 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T10:01:20.408 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T10:01:20.408 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T10:01:20.409 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T10:01:20.409 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T10:01:20.410 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T10:01:20.410 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_14 2024-04-06T10:01:20.410 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T10:01:20.410 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T10:01:20.411 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T10:01:20.411 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T10:01:20.411 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T10:01:20.411 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T10:01:20.411 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T10:01:20.411 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T10:01:20.411 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T10:01:20.411 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T10:01:20.411 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T10:01:20.411 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:01:20.411 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T10:01:20.411 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T10:01:20.413 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T10:01:20.413 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T10:01:20.413 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T10:01:20.413 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T10:01:20.413 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T10:01:20.413 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T10:01:20.413 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T10:01:20.414 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T10:01:20.414 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:01:20.414 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T10:01:20.414 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T10:01:20.414 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T10:01:20.414 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T10:01:20.414 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T10:01:20.414 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T10:01:20.414 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T10:01:20.438 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T10:01:20.439 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T10:01:20.439 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T10:01:20.439 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_14 2024-04-06T10:01:20.535 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:admin_socket: exception getting command descriptions: [Errno 2] No such file or directory 2024-04-06T10:01:20.538 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T10:01:20.538 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/image_14 2024-04-06T10:01:20.538 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:1 2024-04-06T10:01:20.538 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T10:01:20.538 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T10:01:20.538 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-06T10:01:20.538 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:1 2024-04-06T10:01:20.538 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T10:01:20.539 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T10:01:20.539 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T10:01:20.539 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=1 2024-04-06T10:01:20.539 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-06T10:01:20.539 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T10:01:20.539 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:01:20.539 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=1 2024-04-06T10:01:20.539 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=1 2024-04-06T10:01:20.541 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-06T10:01:20.541 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:1 2024-04-06T10:01:20.541 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T10:01:20.541 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T10:01:20.541 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-06T10:01:20.542 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:1 2024-04-06T10:01:20.542 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T10:01:20.542 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T10:01:20.542 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:01:20.542 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=1 2024-04-06T10:01:20.542 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-06T10:01:20.542 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T10:01:20.542 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T10:01:20.542 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=1 2024-04-06T10:01:20.543 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=1 2024-04-06T10:01:20.543 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-06T10:01:20.558 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T10:01:20.559 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T10:01:20.559 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T10:01:20.559 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/image_14 2024-04-06T10:01:20.651 INFO:tasks.rbd_mirror_thrash:waiting for cluster1.client.mirror.0 2024-04-06T10:01:20.651 INFO:teuthology.orchestra.run:waiting for 600 2024-04-06T10:01:20.651 INFO:tasks.rbd_mirror_thrash:reviving cluster1.client.mirror.0 2024-04-06T10:01:20.651 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Restarting daemon 2024-04-06T10:01:20.652 DEBUG:teuthology.orchestra.run.smithi192:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.0 2024-04-06T10:01:20.654 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Started 2024-04-06T10:01:20.654 INFO:tasks.rbd_mirror_thrash:waiting for 109 secs before thrashing 2024-04-06T10:01:20.703 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T10:01:20.704 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror image_14 2024-04-06T10:01:20.704 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:01:20.704 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:01:20.704 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_14 2024-04-06T10:01:20.704 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror image_14 '' 2024-04-06T10:01:20.704 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:01:20.704 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:01:20.704 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_14 2024-04-06T10:01:20.704 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T10:01:20.705 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-image_14.status 2024-04-06T10:01:20.705 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-image_14.status 2024-04-06T10:01:20.705 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T10:01:20.707 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-image_14.status 2024-04-06T10:01:20.708 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_14 2024-04-06T10:01:20.708 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_14.status 2024-04-06T10:01:20.755 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 1 2024-04-06T10:01:20.755 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 1 2024-04-06T10:01:20.755 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T10:01:20.755 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=5, tag_tid=2, entry_tid=1], [object_number=4, tag_tid=2, entry_tid=0], [object_number=6, tag_tid=1, entry_tid=5794], [object_number=7, tag_tid=1, entry_tid=5791]]], state=connected] 2024-04-06T10:01:20.755 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=6, tag_tid=1, entry_tid=5794], [object_number=5, tag_tid=1, entry_tid=5793], [object_number=4, tag_tid=1, entry_tid=5792], [object_number=7, tag_tid=1, entry_tid=5791]]], state=connected] 2024-04-06T10:01:20.760 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_14.status 2024-04-06T10:01:20.761 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=5, tag_tid=2, entry_tid=1' 2024-04-06T10:01:20.763 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror image_14 2024-04-06T10:01:20.763 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:01:20.763 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:01:20.763 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_14 2024-04-06T10:01:20.763 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror image_14 '..*' 2024-04-06T10:01:20.763 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:01:20.763 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:01:20.763 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_14 2024-04-06T10:01:20.763 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T10:01:20.764 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-image_14.status 2024-04-06T10:01:20.764 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-image_14.status 2024-04-06T10:01:20.764 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T10:01:20.764 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-image_14.status 2024-04-06T10:01:20.764 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_14 2024-04-06T10:01:20.765 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_14.status 2024-04-06T10:01:20.811 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 1 2024-04-06T10:01:20.811 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 1 2024-04-06T10:01:20.812 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T10:01:20.812 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=5, tag_tid=2, entry_tid=1], [object_number=4, tag_tid=2, entry_tid=0], [object_number=6, tag_tid=1, entry_tid=5794], [object_number=7, tag_tid=1, entry_tid=5791]]], state=connected] 2024-04-06T10:01:20.812 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=6, tag_tid=1, entry_tid=5794], [object_number=5, tag_tid=1, entry_tid=5793], [object_number=4, tag_tid=1, entry_tid=5792], [object_number=7, tag_tid=1, entry_tid=5791]]], state=connected] 2024-04-06T10:01:20.815 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_14.status 2024-04-06T10:01:20.817 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=6, tag_tid=1, entry_tid=5794' 2024-04-06T10:01:20.817 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=5, tag_tid=2, entry_tid=1' -a 'object_number=5, tag_tid=2, entry_tid=1' = 'object_number=6, tag_tid=1, entry_tid=5794' 2024-04-06T10:01:20.817 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=6, tag_tid=1, entry_tid=5794' '!=' '' 2024-04-06T10:01:20.817 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T10:01:20.818 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=6, tag_tid=1, entry_tid=5794' = '' 2024-04-06T10:01:20.818 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=6, tag_tid=1, entry_tid=5794' 2024-04-06T10:01:20.818 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=5, tag_tid=2, entry_tid=1' 2024-04-06T10:01:20.818 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T10:01:20.818 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T10:01:20.818 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=2 2024-04-06T10:01:20.819 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=6, tag_tid=1, entry_tid=5794' 2024-04-06T10:01:20.819 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T10:01:20.819 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T10:01:20.820 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=1 2024-04-06T10:01:20.821 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=5, tag_tid=2, entry_tid=1' 2024-04-06T10:01:20.821 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T10:01:20.822 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T10:01:20.822 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T10:01:20.823 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=6, tag_tid=1, entry_tid=5794' 2024-04-06T10:01:20.823 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T10:01:20.823 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T10:01:20.824 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=5794 2024-04-06T10:01:20.824 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 2 = 1 -a 1 -le 5794 2024-04-06T10:01:20.824 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T10:01:20.824 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T10:01:20.824 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T10:01:21.026 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror image_14 2024-04-06T10:01:21.026 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T10:01:21.026 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:01:21.026 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_14 2024-04-06T10:01:21.026 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T10:01:21.027 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n image_14 ']' 2024-04-06T10:01:21.027 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/image_14' 2024-04-06T10:01:21.027 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_14 2024-04-06T10:01:21.027 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T10:01:21.027 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T10:01:21.027 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T10:01:21.027 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T10:01:21.027 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T10:01:21.027 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T10:01:21.027 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T10:01:21.028 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T10:01:21.028 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T10:01:21.030 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T10:01:21.030 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_14 2024-04-06T10:01:21.030 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T10:01:21.030 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T10:01:21.030 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T10:01:21.030 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T10:01:21.030 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T10:01:21.030 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T10:01:21.031 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T10:01:21.031 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T10:01:21.031 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T10:01:21.031 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T10:01:21.031 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T10:01:21.031 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:01:21.031 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T10:01:21.031 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T10:01:21.032 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T10:01:21.032 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T10:01:21.032 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T10:01:21.032 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T10:01:21.032 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T10:01:21.033 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T10:01:21.033 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T10:01:21.033 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T10:01:21.033 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:01:21.033 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T10:01:21.033 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T10:01:21.033 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T10:01:21.033 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T10:01:21.033 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T10:01:21.033 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T10:01:21.033 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T10:01:21.056 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T10:01:21.057 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T10:01:21.057 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T10:01:21.057 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_14 2024-04-06T10:01:21.161 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:Invalid command: unused arguments: ['mirror/image_14'] 2024-04-06T10:01:21.161 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:rbd mirror flush : flush rbd mirror 2024-04-06T10:01:21.161 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:admin_socket: invalid command 2024-04-06T10:01:21.164 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T10:01:21.164 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/image_14 2024-04-06T10:01:21.164 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:1 2024-04-06T10:01:21.164 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T10:01:21.165 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T10:01:21.165 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-06T10:01:21.165 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:1 2024-04-06T10:01:21.165 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T10:01:21.165 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T10:01:21.165 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T10:01:21.165 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=1 2024-04-06T10:01:21.165 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-06T10:01:21.165 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T10:01:21.165 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:01:21.165 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=1 2024-04-06T10:01:21.165 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=1 2024-04-06T10:01:21.165 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-06T10:01:21.165 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:1 2024-04-06T10:01:21.165 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T10:01:21.166 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T10:01:21.166 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-06T10:01:21.166 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:1 2024-04-06T10:01:21.166 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T10:01:21.167 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T10:01:21.167 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:01:21.167 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=1 2024-04-06T10:01:21.167 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-06T10:01:21.167 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T10:01:21.167 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T10:01:21.167 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=1 2024-04-06T10:01:21.167 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=1 2024-04-06T10:01:21.168 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-06T10:01:21.191 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T10:01:21.191 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T10:01:21.191 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T10:01:21.191 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/image_14 2024-04-06T10:01:21.325 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T10:01:21.326 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror image_14 2024-04-06T10:01:21.326 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:01:21.326 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:01:21.327 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_14 2024-04-06T10:01:21.327 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror image_14 '' 2024-04-06T10:01:21.327 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:01:21.327 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:01:21.327 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_14 2024-04-06T10:01:21.327 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T10:01:21.328 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-image_14.status 2024-04-06T10:01:21.328 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-image_14.status 2024-04-06T10:01:21.328 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T10:01:21.329 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-image_14.status 2024-04-06T10:01:21.330 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_14 2024-04-06T10:01:21.330 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_14.status 2024-04-06T10:01:21.375 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 1 2024-04-06T10:01:21.375 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 1 2024-04-06T10:01:21.375 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T10:01:21.375 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=5, tag_tid=2, entry_tid=1], [object_number=4, tag_tid=2, entry_tid=0], [object_number=6, tag_tid=1, entry_tid=5794], [object_number=7, tag_tid=1, entry_tid=5791]]], state=connected] 2024-04-06T10:01:21.375 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=6, tag_tid=1, entry_tid=5794], [object_number=5, tag_tid=1, entry_tid=5793], [object_number=4, tag_tid=1, entry_tid=5792], [object_number=7, tag_tid=1, entry_tid=5791]]], state=connected] 2024-04-06T10:01:21.378 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_14.status 2024-04-06T10:01:21.379 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=5, tag_tid=2, entry_tid=1' 2024-04-06T10:01:21.381 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror image_14 2024-04-06T10:01:21.381 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:01:21.381 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:01:21.381 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_14 2024-04-06T10:01:21.381 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror image_14 '..*' 2024-04-06T10:01:21.381 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:01:21.381 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:01:21.381 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_14 2024-04-06T10:01:21.381 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T10:01:21.381 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-image_14.status 2024-04-06T10:01:21.381 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-image_14.status 2024-04-06T10:01:21.381 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T10:01:21.382 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-image_14.status 2024-04-06T10:01:21.382 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_14 2024-04-06T10:01:21.382 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_14.status 2024-04-06T10:01:21.426 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 1 2024-04-06T10:01:21.426 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 1 2024-04-06T10:01:21.426 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T10:01:21.426 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=5, tag_tid=2, entry_tid=1], [object_number=4, tag_tid=2, entry_tid=0], [object_number=6, tag_tid=1, entry_tid=5794], [object_number=7, tag_tid=1, entry_tid=5791]]], state=connected] 2024-04-06T10:01:21.427 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=6, tag_tid=1, entry_tid=5794], [object_number=5, tag_tid=1, entry_tid=5793], [object_number=4, tag_tid=1, entry_tid=5792], [object_number=7, tag_tid=1, entry_tid=5791]]], state=connected] 2024-04-06T10:01:21.430 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_14.status 2024-04-06T10:01:21.431 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=6, tag_tid=1, entry_tid=5794' 2024-04-06T10:01:21.432 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=5, tag_tid=2, entry_tid=1' -a 'object_number=5, tag_tid=2, entry_tid=1' = 'object_number=6, tag_tid=1, entry_tid=5794' 2024-04-06T10:01:21.432 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=6, tag_tid=1, entry_tid=5794' '!=' 'object_number=6, tag_tid=1, entry_tid=5794' 2024-04-06T10:01:21.432 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T10:01:21.432 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.4 2024-04-06T10:01:21.833 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror image_14 2024-04-06T10:01:21.833 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T10:01:21.833 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:01:21.833 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_14 2024-04-06T10:01:21.834 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T10:01:21.834 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n image_14 ']' 2024-04-06T10:01:21.834 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/image_14' 2024-04-06T10:01:21.834 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_14 2024-04-06T10:01:21.834 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T10:01:21.834 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T10:01:21.834 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T10:01:21.834 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T10:01:21.834 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T10:01:21.834 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T10:01:21.834 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T10:01:21.835 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T10:01:21.835 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T10:01:21.839 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T10:01:21.839 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_14 2024-04-06T10:01:21.839 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T10:01:21.839 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T10:01:21.839 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T10:01:21.840 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T10:01:21.840 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T10:01:21.840 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T10:01:21.840 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T10:01:21.840 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T10:01:21.840 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T10:01:21.840 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T10:01:21.840 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T10:01:21.840 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:01:21.840 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T10:01:21.840 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T10:01:21.840 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T10:01:21.840 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T10:01:21.840 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T10:01:21.840 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T10:01:21.841 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T10:01:21.841 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T10:01:21.841 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T10:01:21.841 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T10:01:21.841 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:01:21.841 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T10:01:21.841 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T10:01:21.841 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T10:01:21.841 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T10:01:21.841 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T10:01:21.842 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T10:01:21.842 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T10:01:21.865 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T10:01:21.865 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T10:01:21.866 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T10:01:21.866 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_14 2024-04-06T10:01:21.973 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:Invalid command: unused arguments: ['mirror/image_14'] 2024-04-06T10:01:21.973 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:rbd mirror flush : flush rbd mirror 2024-04-06T10:01:21.973 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:admin_socket: invalid command 2024-04-06T10:01:21.975 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T10:01:21.975 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/image_14 2024-04-06T10:01:21.975 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:1 2024-04-06T10:01:21.975 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T10:01:21.975 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T10:01:21.975 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-06T10:01:21.975 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:1 2024-04-06T10:01:21.975 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T10:01:21.975 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T10:01:21.975 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T10:01:21.975 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=1 2024-04-06T10:01:21.975 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-06T10:01:21.976 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T10:01:21.976 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:01:21.976 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=1 2024-04-06T10:01:21.976 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=1 2024-04-06T10:01:21.977 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-06T10:01:21.977 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:1 2024-04-06T10:01:21.977 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T10:01:21.977 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T10:01:21.977 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-06T10:01:21.977 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:1 2024-04-06T10:01:21.977 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T10:01:21.978 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T10:01:21.978 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:01:21.978 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=1 2024-04-06T10:01:21.978 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-06T10:01:21.978 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T10:01:21.978 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T10:01:21.978 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=1 2024-04-06T10:01:21.978 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=1 2024-04-06T10:01:21.978 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-06T10:01:22.002 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T10:01:22.002 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T10:01:22.002 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T10:01:22.002 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/image_14 2024-04-06T10:01:22.136 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T10:01:22.138 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror image_14 2024-04-06T10:01:22.138 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:01:22.138 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:01:22.138 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_14 2024-04-06T10:01:22.138 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror image_14 '' 2024-04-06T10:01:22.138 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:01:22.138 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:01:22.138 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_14 2024-04-06T10:01:22.138 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T10:01:22.138 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-image_14.status 2024-04-06T10:01:22.139 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-image_14.status 2024-04-06T10:01:22.139 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T10:01:22.140 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-image_14.status 2024-04-06T10:01:22.141 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_14.status 2024-04-06T10:01:22.141 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_14 2024-04-06T10:01:22.194 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 1 2024-04-06T10:01:22.194 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 1 2024-04-06T10:01:22.194 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T10:01:22.194 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=5, tag_tid=2, entry_tid=1], [object_number=4, tag_tid=2, entry_tid=0], [object_number=6, tag_tid=1, entry_tid=5794], [object_number=7, tag_tid=1, entry_tid=5791]]], state=connected] 2024-04-06T10:01:22.194 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=6, tag_tid=1, entry_tid=5794], [object_number=5, tag_tid=1, entry_tid=5793], [object_number=4, tag_tid=1, entry_tid=5792], [object_number=7, tag_tid=1, entry_tid=5791]]], state=connected] 2024-04-06T10:01:22.197 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_14.status 2024-04-06T10:01:22.199 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=5, tag_tid=2, entry_tid=1' 2024-04-06T10:01:22.200 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror image_14 2024-04-06T10:01:22.200 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:01:22.200 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:01:22.200 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_14 2024-04-06T10:01:22.200 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror image_14 '..*' 2024-04-06T10:01:22.200 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:01:22.200 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:01:22.200 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_14 2024-04-06T10:01:22.200 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T10:01:22.201 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-image_14.status 2024-04-06T10:01:22.201 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-image_14.status 2024-04-06T10:01:22.201 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T10:01:22.201 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-image_14.status 2024-04-06T10:01:22.201 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_14 2024-04-06T10:01:22.201 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_14.status 2024-04-06T10:01:22.246 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 1 2024-04-06T10:01:22.247 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 1 2024-04-06T10:01:22.247 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T10:01:22.247 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=5, tag_tid=2, entry_tid=1], [object_number=4, tag_tid=2, entry_tid=0], [object_number=6, tag_tid=1, entry_tid=5794], [object_number=7, tag_tid=1, entry_tid=5791]]], state=connected] 2024-04-06T10:01:22.247 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=6, tag_tid=1, entry_tid=5794], [object_number=5, tag_tid=1, entry_tid=5793], [object_number=4, tag_tid=1, entry_tid=5792], [object_number=7, tag_tid=1, entry_tid=5791]]], state=connected] 2024-04-06T10:01:22.250 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_14.status 2024-04-06T10:01:22.252 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=6, tag_tid=1, entry_tid=5794' 2024-04-06T10:01:22.252 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=5, tag_tid=2, entry_tid=1' -a 'object_number=5, tag_tid=2, entry_tid=1' = 'object_number=6, tag_tid=1, entry_tid=5794' 2024-04-06T10:01:22.252 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=6, tag_tid=1, entry_tid=5794' '!=' 'object_number=6, tag_tid=1, entry_tid=5794' 2024-04-06T10:01:22.252 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T10:01:22.252 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.8 2024-04-06T10:01:23.054 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror image_14 2024-04-06T10:01:23.054 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T10:01:23.054 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:01:23.054 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_14 2024-04-06T10:01:23.054 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T10:01:23.054 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n image_14 ']' 2024-04-06T10:01:23.054 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/image_14' 2024-04-06T10:01:23.055 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_14 2024-04-06T10:01:23.055 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T10:01:23.055 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T10:01:23.055 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T10:01:23.055 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T10:01:23.055 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T10:01:23.055 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T10:01:23.055 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T10:01:23.056 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T10:01:23.056 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T10:01:23.059 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T10:01:23.059 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_14 2024-04-06T10:01:23.059 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T10:01:23.059 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T10:01:23.059 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T10:01:23.059 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T10:01:23.059 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T10:01:23.059 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T10:01:23.059 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T10:01:23.059 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T10:01:23.059 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T10:01:23.059 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T10:01:23.060 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T10:01:23.060 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:01:23.060 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T10:01:23.060 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T10:01:23.060 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T10:01:23.060 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T10:01:23.060 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T10:01:23.060 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T10:01:23.060 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T10:01:23.060 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T10:01:23.060 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T10:01:23.060 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T10:01:23.060 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:01:23.060 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T10:01:23.060 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T10:01:23.061 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T10:01:23.061 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T10:01:23.061 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T10:01:23.061 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T10:01:23.061 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T10:01:23.087 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T10:01:23.087 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T10:01:23.087 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T10:01:23.087 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_14 2024-04-06T10:01:23.193 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:Invalid command: unused arguments: ['mirror/image_14'] 2024-04-06T10:01:23.193 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:rbd mirror flush : flush rbd mirror 2024-04-06T10:01:23.193 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:admin_socket: invalid command 2024-04-06T10:01:23.196 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T10:01:23.196 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/image_14 2024-04-06T10:01:23.196 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:1 2024-04-06T10:01:23.196 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T10:01:23.196 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T10:01:23.196 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-06T10:01:23.196 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:1 2024-04-06T10:01:23.197 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T10:01:23.197 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T10:01:23.197 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T10:01:23.197 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=1 2024-04-06T10:01:23.197 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-06T10:01:23.197 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T10:01:23.197 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:01:23.197 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=1 2024-04-06T10:01:23.197 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=1 2024-04-06T10:01:23.198 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-06T10:01:23.198 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:1 2024-04-06T10:01:23.198 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T10:01:23.198 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T10:01:23.198 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-06T10:01:23.198 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:1 2024-04-06T10:01:23.199 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T10:01:23.199 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T10:01:23.199 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:01:23.199 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=1 2024-04-06T10:01:23.199 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-06T10:01:23.199 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T10:01:23.199 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T10:01:23.199 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=1 2024-04-06T10:01:23.199 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=1 2024-04-06T10:01:23.199 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-06T10:01:23.217 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T10:01:23.218 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T10:01:23.218 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T10:01:23.218 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/image_14 2024-04-06T10:01:23.361 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T10:01:23.362 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror image_14 2024-04-06T10:01:23.362 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:01:23.362 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:01:23.362 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_14 2024-04-06T10:01:23.362 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror image_14 '' 2024-04-06T10:01:23.362 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:01:23.362 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:01:23.362 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_14 2024-04-06T10:01:23.362 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T10:01:23.363 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-image_14.status 2024-04-06T10:01:23.363 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-image_14.status 2024-04-06T10:01:23.363 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T10:01:23.365 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-image_14.status 2024-04-06T10:01:23.365 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_14 2024-04-06T10:01:23.365 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_14.status 2024-04-06T10:01:23.413 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 1 2024-04-06T10:01:23.413 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 1 2024-04-06T10:01:23.413 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T10:01:23.413 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=5, tag_tid=2, entry_tid=1], [object_number=4, tag_tid=2, entry_tid=0], [object_number=6, tag_tid=1, entry_tid=5794], [object_number=7, tag_tid=1, entry_tid=5791]]], state=connected] 2024-04-06T10:01:23.413 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=6, tag_tid=1, entry_tid=5794], [object_number=5, tag_tid=1, entry_tid=5793], [object_number=4, tag_tid=1, entry_tid=5792], [object_number=7, tag_tid=1, entry_tid=5791]]], state=connected] 2024-04-06T10:01:23.416 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_14.status 2024-04-06T10:01:23.418 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=5, tag_tid=2, entry_tid=1' 2024-04-06T10:01:23.419 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror image_14 2024-04-06T10:01:23.419 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:01:23.419 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:01:23.419 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_14 2024-04-06T10:01:23.419 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror image_14 '..*' 2024-04-06T10:01:23.419 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:01:23.419 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:01:23.419 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_14 2024-04-06T10:01:23.419 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T10:01:23.419 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-image_14.status 2024-04-06T10:01:23.420 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-image_14.status 2024-04-06T10:01:23.420 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T10:01:23.420 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-image_14.status 2024-04-06T10:01:23.420 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_14 2024-04-06T10:01:23.420 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_14.status 2024-04-06T10:01:23.465 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 1 2024-04-06T10:01:23.465 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 1 2024-04-06T10:01:23.465 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T10:01:23.465 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=5, tag_tid=2, entry_tid=1], [object_number=4, tag_tid=2, entry_tid=0], [object_number=6, tag_tid=1, entry_tid=5794], [object_number=7, tag_tid=1, entry_tid=5791]]], state=connected] 2024-04-06T10:01:23.465 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=6, tag_tid=1, entry_tid=5794], [object_number=5, tag_tid=1, entry_tid=5793], [object_number=4, tag_tid=1, entry_tid=5792], [object_number=7, tag_tid=1, entry_tid=5791]]], state=connected] 2024-04-06T10:01:23.468 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_14.status 2024-04-06T10:01:23.470 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=6, tag_tid=1, entry_tid=5794' 2024-04-06T10:01:23.471 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=5, tag_tid=2, entry_tid=1' -a 'object_number=5, tag_tid=2, entry_tid=1' = 'object_number=6, tag_tid=1, entry_tid=5794' 2024-04-06T10:01:23.471 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=6, tag_tid=1, entry_tid=5794' '!=' 'object_number=6, tag_tid=1, entry_tid=5794' 2024-04-06T10:01:23.471 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T10:01:23.471 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 1.6 2024-04-06T10:01:25.072 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror image_14 2024-04-06T10:01:25.072 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T10:01:25.073 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:01:25.073 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_14 2024-04-06T10:01:25.073 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T10:01:25.073 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n image_14 ']' 2024-04-06T10:01:25.073 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/image_14' 2024-04-06T10:01:25.073 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_14 2024-04-06T10:01:25.073 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T10:01:25.073 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T10:01:25.073 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T10:01:25.073 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T10:01:25.073 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T10:01:25.073 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T10:01:25.073 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T10:01:25.074 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T10:01:25.074 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T10:01:25.076 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T10:01:25.076 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_14 2024-04-06T10:01:25.076 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T10:01:25.077 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T10:01:25.077 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T10:01:25.077 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T10:01:25.077 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T10:01:25.077 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T10:01:25.077 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T10:01:25.077 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T10:01:25.077 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T10:01:25.077 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T10:01:25.077 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T10:01:25.077 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:01:25.077 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T10:01:25.077 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T10:01:25.079 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T10:01:25.079 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T10:01:25.079 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T10:01:25.079 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T10:01:25.079 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T10:01:25.079 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T10:01:25.079 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T10:01:25.079 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T10:01:25.079 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:01:25.079 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T10:01:25.079 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T10:01:25.080 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T10:01:25.080 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T10:01:25.080 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T10:01:25.080 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T10:01:25.080 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T10:01:25.103 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T10:01:25.104 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T10:01:25.104 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T10:01:25.104 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_14 2024-04-06T10:01:25.209 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:Invalid command: unused arguments: ['mirror/image_14'] 2024-04-06T10:01:25.209 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:rbd mirror flush : flush rbd mirror 2024-04-06T10:01:25.209 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:admin_socket: invalid command 2024-04-06T10:01:25.211 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T10:01:25.211 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/image_14 2024-04-06T10:01:25.211 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:1 2024-04-06T10:01:25.211 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T10:01:25.212 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T10:01:25.212 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-06T10:01:25.212 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:1 2024-04-06T10:01:25.212 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T10:01:25.212 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T10:01:25.212 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T10:01:25.212 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=1 2024-04-06T10:01:25.212 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-06T10:01:25.212 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T10:01:25.212 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:01:25.212 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=1 2024-04-06T10:01:25.212 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=1 2024-04-06T10:01:25.213 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-06T10:01:25.214 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:1 2024-04-06T10:01:25.214 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T10:01:25.214 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T10:01:25.214 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-06T10:01:25.214 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:1 2024-04-06T10:01:25.214 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T10:01:25.214 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T10:01:25.214 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:01:25.214 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=1 2024-04-06T10:01:25.214 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-06T10:01:25.214 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T10:01:25.214 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T10:01:25.214 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=1 2024-04-06T10:01:25.215 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=1 2024-04-06T10:01:25.215 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-06T10:01:25.234 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T10:01:25.234 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T10:01:25.234 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T10:01:25.234 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/image_14 2024-04-06T10:01:25.385 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T10:01:25.386 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror image_14 2024-04-06T10:01:25.386 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:01:25.387 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:01:25.387 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_14 2024-04-06T10:01:25.387 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror image_14 '' 2024-04-06T10:01:25.387 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:01:25.387 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:01:25.387 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_14 2024-04-06T10:01:25.387 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T10:01:25.388 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-image_14.status 2024-04-06T10:01:25.388 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-image_14.status 2024-04-06T10:01:25.388 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T10:01:25.389 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-image_14.status 2024-04-06T10:01:25.390 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_14 2024-04-06T10:01:25.390 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_14.status 2024-04-06T10:01:25.436 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 1 2024-04-06T10:01:25.436 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 1 2024-04-06T10:01:25.436 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T10:01:25.436 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=5, tag_tid=2, entry_tid=1], [object_number=4, tag_tid=2, entry_tid=0], [object_number=6, tag_tid=1, entry_tid=5794], [object_number=7, tag_tid=1, entry_tid=5791]]], state=connected] 2024-04-06T10:01:25.436 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=6, tag_tid=1, entry_tid=5794], [object_number=5, tag_tid=1, entry_tid=5793], [object_number=4, tag_tid=1, entry_tid=5792], [object_number=7, tag_tid=1, entry_tid=5791]]], state=connected] 2024-04-06T10:01:25.440 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_14.status 2024-04-06T10:01:25.441 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=5, tag_tid=2, entry_tid=1' 2024-04-06T10:01:25.442 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror image_14 2024-04-06T10:01:25.442 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:01:25.442 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:01:25.442 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_14 2024-04-06T10:01:25.442 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror image_14 '..*' 2024-04-06T10:01:25.442 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:01:25.442 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:01:25.443 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_14 2024-04-06T10:01:25.443 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T10:01:25.443 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-image_14.status 2024-04-06T10:01:25.443 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-image_14.status 2024-04-06T10:01:25.443 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T10:01:25.443 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-image_14.status 2024-04-06T10:01:25.443 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_14 2024-04-06T10:01:25.444 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_14.status 2024-04-06T10:01:25.487 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 1 2024-04-06T10:01:25.487 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 1 2024-04-06T10:01:25.487 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T10:01:25.487 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=5, tag_tid=2, entry_tid=1], [object_number=4, tag_tid=2, entry_tid=0], [object_number=6, tag_tid=1, entry_tid=5794], [object_number=7, tag_tid=1, entry_tid=5791]]], state=connected] 2024-04-06T10:01:25.488 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=6, tag_tid=1, entry_tid=5794], [object_number=5, tag_tid=1, entry_tid=5793], [object_number=4, tag_tid=1, entry_tid=5792], [object_number=7, tag_tid=1, entry_tid=5791]]], state=connected] 2024-04-06T10:01:25.491 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_14.status 2024-04-06T10:01:25.492 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=6, tag_tid=1, entry_tid=5794' 2024-04-06T10:01:25.492 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=5, tag_tid=2, entry_tid=1' -a 'object_number=5, tag_tid=2, entry_tid=1' = 'object_number=6, tag_tid=1, entry_tid=5794' 2024-04-06T10:01:25.493 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=6, tag_tid=1, entry_tid=5794' '!=' 'object_number=6, tag_tid=1, entry_tid=5794' 2024-04-06T10:01:25.493 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T10:01:25.493 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 2 2024-04-06T10:01:27.495 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror image_14 2024-04-06T10:01:27.495 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T10:01:27.495 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:01:27.495 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_14 2024-04-06T10:01:27.495 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T10:01:27.495 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n image_14 ']' 2024-04-06T10:01:27.495 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/image_14' 2024-04-06T10:01:27.495 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_14 2024-04-06T10:01:27.495 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T10:01:27.495 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T10:01:27.495 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T10:01:27.496 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T10:01:27.496 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T10:01:27.496 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T10:01:27.496 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T10:01:27.496 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T10:01:27.496 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T10:01:27.499 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T10:01:27.499 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_14 2024-04-06T10:01:27.499 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T10:01:27.499 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T10:01:27.499 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T10:01:27.499 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T10:01:27.499 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T10:01:27.499 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T10:01:27.499 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T10:01:27.499 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T10:01:27.499 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T10:01:27.499 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T10:01:27.499 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T10:01:27.500 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:01:27.500 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T10:01:27.500 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T10:01:27.500 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T10:01:27.500 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T10:01:27.500 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T10:01:27.500 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T10:01:27.500 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T10:01:27.500 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T10:01:27.500 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T10:01:27.500 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T10:01:27.500 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:01:27.500 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T10:01:27.500 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T10:01:27.501 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T10:01:27.501 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T10:01:27.501 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T10:01:27.501 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T10:01:27.501 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T10:01:27.525 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T10:01:27.525 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T10:01:27.525 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T10:01:27.525 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_14 2024-04-06T10:01:27.634 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:Invalid command: unused arguments: ['mirror/image_14'] 2024-04-06T10:01:27.634 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:rbd mirror flush : flush rbd mirror 2024-04-06T10:01:27.634 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:admin_socket: invalid command 2024-04-06T10:01:27.636 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T10:01:27.636 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/image_14 2024-04-06T10:01:27.636 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:1 2024-04-06T10:01:27.636 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T10:01:27.636 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T10:01:27.636 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-06T10:01:27.637 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:1 2024-04-06T10:01:27.637 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T10:01:27.637 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T10:01:27.637 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T10:01:27.637 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=1 2024-04-06T10:01:27.637 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-06T10:01:27.637 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T10:01:27.637 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:01:27.637 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=1 2024-04-06T10:01:27.637 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=1 2024-04-06T10:01:27.639 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-06T10:01:27.639 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:1 2024-04-06T10:01:27.639 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T10:01:27.639 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T10:01:27.639 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-06T10:01:27.639 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:1 2024-04-06T10:01:27.639 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T10:01:27.639 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T10:01:27.639 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:01:27.640 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=1 2024-04-06T10:01:27.640 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-06T10:01:27.640 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T10:01:27.640 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T10:01:27.640 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=1 2024-04-06T10:01:27.640 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=1 2024-04-06T10:01:27.640 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-06T10:01:27.658 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T10:01:27.658 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T10:01:27.658 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T10:01:27.658 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/image_14 2024-04-06T10:01:27.802 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T10:01:27.803 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror image_14 2024-04-06T10:01:27.803 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:01:27.803 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:01:27.803 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_14 2024-04-06T10:01:27.803 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror image_14 '' 2024-04-06T10:01:27.803 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:01:27.803 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:01:27.803 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_14 2024-04-06T10:01:27.804 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T10:01:27.804 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-image_14.status 2024-04-06T10:01:27.804 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-image_14.status 2024-04-06T10:01:27.804 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T10:01:27.806 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-image_14.status 2024-04-06T10:01:27.806 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_14.status 2024-04-06T10:01:27.806 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_14 2024-04-06T10:01:27.860 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 1 2024-04-06T10:01:27.860 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 1 2024-04-06T10:01:27.860 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T10:01:27.860 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=5, tag_tid=2, entry_tid=1], [object_number=4, tag_tid=2, entry_tid=0], [object_number=6, tag_tid=1, entry_tid=5794], [object_number=7, tag_tid=1, entry_tid=5791]]], state=connected] 2024-04-06T10:01:27.860 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=5, tag_tid=2, entry_tid=1], [object_number=4, tag_tid=2, entry_tid=0], [object_number=6, tag_tid=1, entry_tid=5794], [object_number=7, tag_tid=1, entry_tid=5791]]], state=connected] 2024-04-06T10:01:27.864 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_14.status 2024-04-06T10:01:27.865 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=5, tag_tid=2, entry_tid=1' 2024-04-06T10:01:27.866 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror image_14 2024-04-06T10:01:27.866 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:01:27.866 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:01:27.866 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_14 2024-04-06T10:01:27.866 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror image_14 '..*' 2024-04-06T10:01:27.866 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:01:27.866 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:01:27.866 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_14 2024-04-06T10:01:27.866 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T10:01:27.866 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-image_14.status 2024-04-06T10:01:27.867 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-image_14.status 2024-04-06T10:01:27.867 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T10:01:27.868 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-image_14.status 2024-04-06T10:01:27.868 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_14 2024-04-06T10:01:27.868 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_14.status 2024-04-06T10:01:27.912 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 1 2024-04-06T10:01:27.912 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 1 2024-04-06T10:01:27.913 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T10:01:27.913 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=5, tag_tid=2, entry_tid=1], [object_number=4, tag_tid=2, entry_tid=0], [object_number=6, tag_tid=1, entry_tid=5794], [object_number=7, tag_tid=1, entry_tid=5791]]], state=connected] 2024-04-06T10:01:27.913 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=5, tag_tid=2, entry_tid=1], [object_number=4, tag_tid=2, entry_tid=0], [object_number=6, tag_tid=1, entry_tid=5794], [object_number=7, tag_tid=1, entry_tid=5791]]], state=connected] 2024-04-06T10:01:27.916 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_14.status 2024-04-06T10:01:27.918 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=5, tag_tid=2, entry_tid=1' 2024-04-06T10:01:27.918 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=5, tag_tid=2, entry_tid=1' -a 'object_number=5, tag_tid=2, entry_tid=1' = 'object_number=5, tag_tid=2, entry_tid=1' 2024-04-06T10:01:27.918 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T10:01:27.918 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_snap_present cluster1 mirror image_14 snap 2024-04-06T10:01:27.918 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T10:01:27.918 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:01:27.918 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_14 2024-04-06T10:01:27.919 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local snap_name=snap 2024-04-06T10:01:27.919 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local s 2024-04-06T10:01:27.919 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-04-06T10:01:27.919 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 1 2024-04-06T10:01:28.919 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 info mirror/image_14@snap 2024-04-06T10:01:28.985 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:rbd image 'image_14': 2024-04-06T10:01:28.985 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: size 128 MiB in 32 objects 2024-04-06T10:01:28.986 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: order 22 (4 MiB objects) 2024-04-06T10:01:28.986 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: snapshot_count: 1 2024-04-06T10:01:28.986 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: id: 10ac8c2d3880 2024-04-06T10:01:28.986 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: block_name_prefix: rbd_data.10ac8c2d3880 2024-04-06T10:01:28.986 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: format: 2 2024-04-06T10:01:28.986 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: features: layering, exclusive-lock, journaling 2024-04-06T10:01:28.986 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: op_features: 2024-04-06T10:01:28.986 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: flags: 2024-04-06T10:01:28.986 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: create_timestamp: Sat Apr 6 09:52:53 2024 2024-04-06T10:01:28.986 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: access_timestamp: Sat Apr 6 09:52:53 2024 2024-04-06T10:01:28.986 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: modify_timestamp: Sat Apr 6 09:52:53 2024 2024-04-06T10:01:28.986 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: protected: False 2024-04-06T10:01:28.987 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: journal: 10ac8c2d3880 2024-04-06T10:01:28.987 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: mirroring state: enabled 2024-04-06T10:01:28.987 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: mirroring mode: journal 2024-04-06T10:01:28.987 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: mirroring global id: f4cd16f6-418c-4fd1-8de5-c6c4996e00d4 2024-04-06T10:01:28.987 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: mirroring primary: false 2024-04-06T10:01:28.990 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T10:01:28.990 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ compare_image_snaps mirror image_14 snap 2024-04-06T10:01:28.990 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:01:28.990 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_14 2024-04-06T10:01:28.991 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local snap_name=snap 2024-04-06T10:01:28.991 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local ret=0 2024-04-06T10:01:28.991 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local rmt_export=/tmp/tmp.BhW6xKmROU/cluster2-mirror-image_14.export 2024-04-06T10:01:28.991 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loc_export=/tmp/tmp.BhW6xKmROU/cluster1-mirror-image_14.export 2024-04-06T10:01:28.991 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rm -f /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_14.export /tmp/tmp.BhW6xKmROU/cluster1-mirror-image_14.export 2024-04-06T10:01:28.991 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster2 -p mirror export image_14@snap /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_14.export 2024-04-06T10:01:29.155 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-04-06T10:01:29.162 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 -p mirror export image_14@snap /tmp/tmp.BhW6xKmROU/cluster1-mirror-image_14.export 2024-04-06T10:01:29.508 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-04-06T10:01:29.520 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmp /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_14.export /tmp/tmp.BhW6xKmROU/cluster1-mirror-image_14.export 2024-04-06T10:01:29.599 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rm -f /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_14.export /tmp/tmp.BhW6xKmROU/cluster1-mirror-image_14.export 2024-04-06T10:01:29.640 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T10:01:29.640 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-06T10:01:29.640 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ image=image_15 2024-04-06T10:01:29.640 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ create_snap cluster2 mirror image_15 snap 2024-04-06T10:01:29.640 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-04-06T10:01:29.640 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:01:29.640 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_15 2024-04-06T10:01:29.641 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local snap_name=snap 2024-04-06T10:01:29.641 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster2 -p mirror snap create image_15@snap --debug-rbd=20 --debug-journaler=20 2024-04-06T10:01:30.852 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_image_replay_started cluster1 mirror image_15 2024-04-06T10:01:30.852 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T10:01:30.852 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:01:30.853 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_15 2024-04-06T10:01:30.853 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_image_replay_state cluster1 mirror image_15 started 2024-04-06T10:01:30.853 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T10:01:30.853 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:01:30.853 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_15 2024-04-06T10:01:30.853 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state=started 2024-04-06T10:01:30.853 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local s 2024-04-06T10:01:30.853 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T10:01:30.853 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 1 2024-04-06T10:01:31.854 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test_image_replay_state cluster1 mirror image_15 started 2024-04-06T10:01:31.854 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T10:01:31.854 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:01:31.855 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_15 2024-04-06T10:01:31.855 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local test_state=started 2024-04-06T10:01:31.855 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local status_result 2024-04-06T10:01:31.855 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local current_state=stopped 2024-04-06T10:01:31.855 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_15 2024-04-06T10:01:31.856 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -i state 2024-04-06T10:01:31.856 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_instance=cluster1 2024-04-06T10:01:31.856 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ shift 2024-04-06T10:01:31.856 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T10:01:31.856 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance=cluster1 2024-04-06T10:01:31.856 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local loop_instance 2024-04-06T10:01:31.856 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T10:01:31.856 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sleep 0 2024-04-06T10:01:31.857 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-04-06T10:01:31.857 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ seq 0 3 2024-04-06T10:01:31.859 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T10:01:31.859 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_15 2024-04-06T10:01:31.859 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1:0 2024-04-06T10:01:31.859 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ shift 2024-04-06T10:01:31.859 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T10:01:31.859 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-04-06T10:01:31.859 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T10:01:31.859 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=cluster 2024-04-06T10:01:31.859 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T10:01:31.859 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:01:31.860 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T10:01:31.860 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T10:01:31.860 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval cluster=cluster1 2024-04-06T10:01:31.860 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ cluster=cluster1 2024-04-06T10:01:31.860 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T10:01:31.860 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T10:01:31.861 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-04-06T10:01:31.862 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local local_cluster=cluster1:0 2024-04-06T10:01:31.862 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local cluster=cluster1 2024-04-06T10:01:31.862 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local instance 2024-04-06T10:01:31.862 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T10:01:31.862 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local val=cluster1:0 2024-04-06T10:01:31.862 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local cluster_var_name=local_cluster 2024-04-06T10:01:31.862 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local instance_var_name=instance 2024-04-06T10:01:31.862 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ cluster=cluster1 2024-04-06T10:01:31.862 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T10:01:31.862 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ '[' 0 = cluster1:0 ']' 2024-04-06T10:01:31.863 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ eval local_cluster=cluster1 2024-04-06T10:01:31.863 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++++ local_cluster=cluster1 2024-04-06T10:01:31.863 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ eval instance=0 2024-04-06T10:01:31.863 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++++ instance=0 2024-04-06T10:01:31.863 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T10:01:31.881 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T10:01:31.882 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T10:01:31.882 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T10:01:31.882 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_15 2024-04-06T10:01:31.990 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:Invalid command: unused arguments: ['mirror/image_15'] 2024-04-06T10:01:31.990 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:rbd mirror status : get status for rbd mirror 2024-04-06T10:01:31.990 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:admin_socket: invalid command 2024-04-06T10:01:31.994 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T10:01:31.994 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_15 2024-04-06T10:01:31.994 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1:1 2024-04-06T10:01:31.994 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ shift 2024-04-06T10:01:31.994 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T10:01:31.995 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-04-06T10:01:31.995 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:1 2024-04-06T10:01:31.995 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=cluster 2024-04-06T10:01:31.995 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T10:01:31.995 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:01:31.995 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=1 2024-04-06T10:01:31.995 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-06T10:01:31.996 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval cluster=cluster1 2024-04-06T10:01:31.996 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ cluster=cluster1 2024-04-06T10:01:31.996 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=1 2024-04-06T10:01:31.996 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=1 2024-04-06T10:01:31.996 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-04-06T10:01:31.996 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local local_cluster=cluster1:1 2024-04-06T10:01:31.996 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local cluster=cluster1 2024-04-06T10:01:31.996 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local instance 2024-04-06T10:01:31.996 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-06T10:01:31.996 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local val=cluster1:1 2024-04-06T10:01:31.996 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local cluster_var_name=local_cluster 2024-04-06T10:01:31.996 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local instance_var_name=instance 2024-04-06T10:01:31.997 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ cluster=cluster1 2024-04-06T10:01:31.997 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=1 2024-04-06T10:01:31.997 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ '[' 1 = cluster1:1 ']' 2024-04-06T10:01:31.997 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ eval local_cluster=cluster1 2024-04-06T10:01:31.997 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++++ local_cluster=cluster1 2024-04-06T10:01:31.997 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ eval instance=1 2024-04-06T10:01:31.997 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++++ instance=1 2024-04-06T10:01:31.997 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-06T10:01:32.012 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T10:01:32.012 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T10:01:32.013 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T10:01:32.013 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_15 2024-04-06T10:01:32.157 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ return 0 2024-04-06T10:01:32.158 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ status_result=' "state": "Replaying"' 2024-04-06T10:01:32.158 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ echo ' "state": "Replaying"' 2024-04-06T10:01:32.159 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep -i Replaying 2024-04-06T10:01:32.161 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: "state": "Replaying" 2024-04-06T10:01:32.162 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ current_state=started 2024-04-06T10:01:32.162 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test started = started 2024-04-06T10:01:32.162 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T10:01:32.162 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_15 2024-04-06T10:01:32.162 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local local_cluster=cluster1 2024-04-06T10:01:32.162 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-04-06T10:01:32.162 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:01:32.162 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_15 2024-04-06T10:01:32.162 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' journal = journal ']' 2024-04-06T10:01:32.163 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_journal_replay_complete cluster1 cluster2 mirror image_15 2024-04-06T10:01:32.163 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local local_cluster=cluster1 2024-04-06T10:01:32.163 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-04-06T10:01:32.163 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:01:32.163 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_15 2024-04-06T10:01:32.163 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local s master_pos mirror_pos last_mirror_pos 2024-04-06T10:01:32.163 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local master_tag master_entry mirror_tag mirror_entry 2024-04-06T10:01:32.163 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T10:01:32.163 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T10:01:32.163 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T10:01:32.363 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror image_15 2024-04-06T10:01:32.363 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T10:01:32.364 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:01:32.364 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_15 2024-04-06T10:01:32.364 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T10:01:32.364 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n image_15 ']' 2024-04-06T10:01:32.364 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/image_15' 2024-04-06T10:01:32.364 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_15 2024-04-06T10:01:32.364 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T10:01:32.364 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T10:01:32.364 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T10:01:32.364 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T10:01:32.364 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T10:01:32.364 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T10:01:32.364 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T10:01:32.365 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T10:01:32.365 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T10:01:32.368 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T10:01:32.369 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_15 2024-04-06T10:01:32.369 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T10:01:32.369 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T10:01:32.369 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T10:01:32.369 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T10:01:32.369 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T10:01:32.369 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T10:01:32.369 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T10:01:32.369 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T10:01:32.369 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T10:01:32.369 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T10:01:32.369 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T10:01:32.369 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:01:32.370 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T10:01:32.370 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T10:01:32.370 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T10:01:32.371 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T10:01:32.371 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T10:01:32.371 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T10:01:32.371 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T10:01:32.371 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T10:01:32.371 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T10:01:32.371 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T10:01:32.371 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:01:32.372 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T10:01:32.372 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T10:01:32.372 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T10:01:32.372 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T10:01:32.372 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T10:01:32.372 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T10:01:32.372 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T10:01:32.394 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T10:01:32.394 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T10:01:32.394 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T10:01:32.394 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_15 2024-04-06T10:01:32.497 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:Invalid command: unused arguments: ['mirror/image_15'] 2024-04-06T10:01:32.497 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:rbd mirror flush : flush rbd mirror 2024-04-06T10:01:32.497 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:admin_socket: invalid command 2024-04-06T10:01:32.500 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T10:01:32.500 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/image_15 2024-04-06T10:01:32.500 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:1 2024-04-06T10:01:32.500 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T10:01:32.500 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T10:01:32.500 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-06T10:01:32.500 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:1 2024-04-06T10:01:32.501 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T10:01:32.501 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T10:01:32.501 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T10:01:32.501 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=1 2024-04-06T10:01:32.501 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-06T10:01:32.501 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T10:01:32.501 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:01:32.501 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=1 2024-04-06T10:01:32.501 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=1 2024-04-06T10:01:32.502 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-06T10:01:32.502 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:1 2024-04-06T10:01:32.502 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T10:01:32.502 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T10:01:32.502 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-06T10:01:32.502 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:1 2024-04-06T10:01:32.502 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T10:01:32.503 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T10:01:32.503 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:01:32.503 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=1 2024-04-06T10:01:32.503 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-06T10:01:32.503 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T10:01:32.503 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T10:01:32.503 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=1 2024-04-06T10:01:32.503 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=1 2024-04-06T10:01:32.503 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-06T10:01:32.520 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T10:01:32.520 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T10:01:32.520 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T10:01:32.521 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/image_15 2024-04-06T10:01:32.683 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T10:01:32.683 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror image_15 2024-04-06T10:01:32.683 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:01:32.683 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:01:32.683 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_15 2024-04-06T10:01:32.683 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror image_15 '' 2024-04-06T10:01:32.683 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:01:32.683 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:01:32.683 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_15 2024-04-06T10:01:32.684 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T10:01:32.684 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-image_15.status 2024-04-06T10:01:32.684 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-image_15.status 2024-04-06T10:01:32.684 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T10:01:32.684 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-image_15.status 2024-04-06T10:01:32.685 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_15 2024-04-06T10:01:32.685 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_15.status 2024-04-06T10:01:32.734 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 1 2024-04-06T10:01:32.734 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 1 2024-04-06T10:01:32.734 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T10:01:32.734 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=5, tag_tid=2, entry_tid=1], [object_number=4, tag_tid=2, entry_tid=0], [object_number=6, tag_tid=1, entry_tid=4502], [object_number=7, tag_tid=1, entry_tid=4499]]], state=connected] 2024-04-06T10:01:32.735 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=6, tag_tid=1, entry_tid=4502], [object_number=5, tag_tid=1, entry_tid=4501], [object_number=4, tag_tid=1, entry_tid=4500], [object_number=7, tag_tid=1, entry_tid=4499]]], state=connected] 2024-04-06T10:01:32.738 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_15.status 2024-04-06T10:01:32.741 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=5, tag_tid=2, entry_tid=1' 2024-04-06T10:01:32.741 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror image_15 2024-04-06T10:01:32.741 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:01:32.741 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:01:32.741 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_15 2024-04-06T10:01:32.741 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror image_15 '..*' 2024-04-06T10:01:32.741 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:01:32.741 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:01:32.741 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_15 2024-04-06T10:01:32.741 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T10:01:32.741 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-image_15.status 2024-04-06T10:01:32.742 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-image_15.status 2024-04-06T10:01:32.742 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T10:01:32.742 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-image_15.status 2024-04-06T10:01:32.742 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_15 2024-04-06T10:01:32.742 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_15.status 2024-04-06T10:01:32.786 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 1 2024-04-06T10:01:32.786 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 1 2024-04-06T10:01:32.786 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T10:01:32.786 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=5, tag_tid=2, entry_tid=1], [object_number=4, tag_tid=2, entry_tid=0], [object_number=6, tag_tid=1, entry_tid=4502], [object_number=7, tag_tid=1, entry_tid=4499]]], state=connected] 2024-04-06T10:01:32.786 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=6, tag_tid=1, entry_tid=4502], [object_number=5, tag_tid=1, entry_tid=4501], [object_number=4, tag_tid=1, entry_tid=4500], [object_number=7, tag_tid=1, entry_tid=4499]]], state=connected] 2024-04-06T10:01:32.789 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_15.status 2024-04-06T10:01:32.791 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=6, tag_tid=1, entry_tid=4502' 2024-04-06T10:01:32.792 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=5, tag_tid=2, entry_tid=1' -a 'object_number=5, tag_tid=2, entry_tid=1' = 'object_number=6, tag_tid=1, entry_tid=4502' 2024-04-06T10:01:32.792 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=6, tag_tid=1, entry_tid=4502' '!=' '' 2024-04-06T10:01:32.792 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T10:01:32.792 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=6, tag_tid=1, entry_tid=4502' = '' 2024-04-06T10:01:32.792 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=6, tag_tid=1, entry_tid=4502' 2024-04-06T10:01:32.792 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=5, tag_tid=2, entry_tid=1' 2024-04-06T10:01:32.792 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T10:01:32.792 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T10:01:32.793 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=2 2024-04-06T10:01:32.793 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=6, tag_tid=1, entry_tid=4502' 2024-04-06T10:01:32.793 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T10:01:32.793 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T10:01:32.794 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=1 2024-04-06T10:01:32.795 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=5, tag_tid=2, entry_tid=1' 2024-04-06T10:01:32.795 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T10:01:32.796 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T10:01:32.796 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T10:01:32.797 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=6, tag_tid=1, entry_tid=4502' 2024-04-06T10:01:32.797 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T10:01:32.797 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T10:01:32.798 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=4502 2024-04-06T10:01:32.798 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 2 = 1 -a 1 -le 4502 2024-04-06T10:01:32.798 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T10:01:32.798 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T10:01:32.798 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T10:01:33.000 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror image_15 2024-04-06T10:01:33.000 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T10:01:33.001 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:01:33.001 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_15 2024-04-06T10:01:33.001 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T10:01:33.001 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n image_15 ']' 2024-04-06T10:01:33.001 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/image_15' 2024-04-06T10:01:33.001 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_15 2024-04-06T10:01:33.001 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T10:01:33.001 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T10:01:33.001 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T10:01:33.001 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T10:01:33.001 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T10:01:33.001 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T10:01:33.001 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T10:01:33.002 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T10:01:33.002 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T10:01:33.005 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T10:01:33.006 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_15 2024-04-06T10:01:33.006 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T10:01:33.006 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T10:01:33.006 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T10:01:33.006 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T10:01:33.006 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T10:01:33.006 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T10:01:33.006 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T10:01:33.006 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T10:01:33.006 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T10:01:33.006 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T10:01:33.006 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T10:01:33.007 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:01:33.007 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T10:01:33.007 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T10:01:33.007 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T10:01:33.007 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T10:01:33.007 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T10:01:33.007 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T10:01:33.007 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T10:01:33.007 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T10:01:33.007 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T10:01:33.007 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T10:01:33.007 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:01:33.007 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T10:01:33.007 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T10:01:33.007 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T10:01:33.007 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T10:01:33.008 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T10:01:33.008 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T10:01:33.008 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T10:01:33.030 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T10:01:33.030 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T10:01:33.031 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T10:01:33.031 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_15 2024-04-06T10:01:33.137 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:Invalid command: unused arguments: ['mirror/image_15'] 2024-04-06T10:01:33.137 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:rbd mirror flush : flush rbd mirror 2024-04-06T10:01:33.138 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:admin_socket: invalid command 2024-04-06T10:01:33.142 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T10:01:33.142 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/image_15 2024-04-06T10:01:33.142 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:1 2024-04-06T10:01:33.143 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T10:01:33.143 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T10:01:33.143 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-06T10:01:33.143 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:1 2024-04-06T10:01:33.143 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T10:01:33.143 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T10:01:33.143 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T10:01:33.143 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=1 2024-04-06T10:01:33.143 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-06T10:01:33.143 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T10:01:33.143 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:01:33.144 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=1 2024-04-06T10:01:33.144 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=1 2024-04-06T10:01:33.144 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-06T10:01:33.144 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:1 2024-04-06T10:01:33.144 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T10:01:33.144 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T10:01:33.144 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-06T10:01:33.144 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:1 2024-04-06T10:01:33.144 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T10:01:33.144 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T10:01:33.144 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:01:33.144 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=1 2024-04-06T10:01:33.145 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-06T10:01:33.145 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T10:01:33.145 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T10:01:33.145 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=1 2024-04-06T10:01:33.145 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=1 2024-04-06T10:01:33.145 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-06T10:01:33.166 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T10:01:33.166 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T10:01:33.166 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T10:01:33.166 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/image_15 2024-04-06T10:01:33.309 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T10:01:33.311 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror image_15 2024-04-06T10:01:33.311 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:01:33.311 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:01:33.311 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_15 2024-04-06T10:01:33.311 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror image_15 '' 2024-04-06T10:01:33.311 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:01:33.311 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:01:33.311 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_15 2024-04-06T10:01:33.311 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T10:01:33.311 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-image_15.status 2024-04-06T10:01:33.312 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-image_15.status 2024-04-06T10:01:33.312 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T10:01:33.313 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-image_15.status 2024-04-06T10:01:33.314 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_15 2024-04-06T10:01:33.314 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_15.status 2024-04-06T10:01:33.358 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 1 2024-04-06T10:01:33.358 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 1 2024-04-06T10:01:33.358 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T10:01:33.358 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=5, tag_tid=2, entry_tid=1], [object_number=4, tag_tid=2, entry_tid=0], [object_number=6, tag_tid=1, entry_tid=4502], [object_number=7, tag_tid=1, entry_tid=4499]]], state=connected] 2024-04-06T10:01:33.359 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=6, tag_tid=1, entry_tid=4502], [object_number=5, tag_tid=1, entry_tid=4501], [object_number=4, tag_tid=1, entry_tid=4500], [object_number=7, tag_tid=1, entry_tid=4499]]], state=connected] 2024-04-06T10:01:33.362 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_15.status 2024-04-06T10:01:33.363 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=5, tag_tid=2, entry_tid=1' 2024-04-06T10:01:33.364 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror image_15 2024-04-06T10:01:33.364 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:01:33.364 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:01:33.364 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_15 2024-04-06T10:01:33.364 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror image_15 '..*' 2024-04-06T10:01:33.364 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:01:33.364 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:01:33.365 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_15 2024-04-06T10:01:33.365 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T10:01:33.365 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-image_15.status 2024-04-06T10:01:33.365 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-image_15.status 2024-04-06T10:01:33.365 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T10:01:33.365 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-image_15.status 2024-04-06T10:01:33.366 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_15 2024-04-06T10:01:33.366 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_15.status 2024-04-06T10:01:33.411 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 1 2024-04-06T10:01:33.411 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 1 2024-04-06T10:01:33.411 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T10:01:33.411 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=5, tag_tid=2, entry_tid=1], [object_number=4, tag_tid=2, entry_tid=0], [object_number=6, tag_tid=1, entry_tid=4502], [object_number=7, tag_tid=1, entry_tid=4499]]], state=connected] 2024-04-06T10:01:33.411 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=6, tag_tid=1, entry_tid=4502], [object_number=5, tag_tid=1, entry_tid=4501], [object_number=4, tag_tid=1, entry_tid=4500], [object_number=7, tag_tid=1, entry_tid=4499]]], state=connected] 2024-04-06T10:01:33.414 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_15.status 2024-04-06T10:01:33.416 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=6, tag_tid=1, entry_tid=4502' 2024-04-06T10:01:33.417 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=5, tag_tid=2, entry_tid=1' -a 'object_number=5, tag_tid=2, entry_tid=1' = 'object_number=6, tag_tid=1, entry_tid=4502' 2024-04-06T10:01:33.417 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=6, tag_tid=1, entry_tid=4502' '!=' 'object_number=6, tag_tid=1, entry_tid=4502' 2024-04-06T10:01:33.417 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T10:01:33.417 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.4 2024-04-06T10:01:33.818 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror image_15 2024-04-06T10:01:33.818 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T10:01:33.818 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:01:33.818 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_15 2024-04-06T10:01:33.818 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T10:01:33.819 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n image_15 ']' 2024-04-06T10:01:33.819 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/image_15' 2024-04-06T10:01:33.819 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_15 2024-04-06T10:01:33.819 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T10:01:33.819 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T10:01:33.819 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T10:01:33.819 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T10:01:33.819 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T10:01:33.819 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T10:01:33.819 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T10:01:33.820 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T10:01:33.820 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T10:01:33.821 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T10:01:33.821 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_15 2024-04-06T10:01:33.822 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T10:01:33.822 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T10:01:33.822 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T10:01:33.822 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T10:01:33.822 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T10:01:33.822 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T10:01:33.822 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T10:01:33.822 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T10:01:33.822 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T10:01:33.822 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T10:01:33.822 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T10:01:33.822 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:01:33.822 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T10:01:33.823 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T10:01:33.824 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T10:01:33.824 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T10:01:33.824 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T10:01:33.824 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T10:01:33.824 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T10:01:33.824 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T10:01:33.824 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T10:01:33.824 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T10:01:33.824 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:01:33.824 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T10:01:33.824 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T10:01:33.824 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T10:01:33.825 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T10:01:33.825 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T10:01:33.825 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T10:01:33.825 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T10:01:33.851 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T10:01:33.851 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T10:01:33.851 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T10:01:33.851 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_15 2024-04-06T10:01:33.955 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:Invalid command: unused arguments: ['mirror/image_15'] 2024-04-06T10:01:33.955 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:rbd mirror flush : flush rbd mirror 2024-04-06T10:01:33.955 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:admin_socket: invalid command 2024-04-06T10:01:33.957 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T10:01:33.957 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/image_15 2024-04-06T10:01:33.957 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:1 2024-04-06T10:01:33.958 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T10:01:33.958 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T10:01:33.958 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-06T10:01:33.958 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:1 2024-04-06T10:01:33.958 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T10:01:33.958 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T10:01:33.958 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T10:01:33.958 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=1 2024-04-06T10:01:33.958 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-06T10:01:33.958 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T10:01:33.958 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:01:33.958 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=1 2024-04-06T10:01:33.958 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=1 2024-04-06T10:01:33.960 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-06T10:01:33.960 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:1 2024-04-06T10:01:33.960 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T10:01:33.960 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T10:01:33.960 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-06T10:01:33.960 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:1 2024-04-06T10:01:33.960 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T10:01:33.960 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T10:01:33.960 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:01:33.960 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=1 2024-04-06T10:01:33.961 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-06T10:01:33.961 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T10:01:33.961 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T10:01:33.961 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=1 2024-04-06T10:01:33.961 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=1 2024-04-06T10:01:33.961 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-06T10:01:33.983 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T10:01:33.983 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T10:01:33.983 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T10:01:33.983 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/image_15 2024-04-06T10:01:34.127 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T10:01:34.128 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror image_15 2024-04-06T10:01:34.128 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:01:34.128 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:01:34.128 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_15 2024-04-06T10:01:34.129 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror image_15 '' 2024-04-06T10:01:34.129 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:01:34.129 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:01:34.129 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_15 2024-04-06T10:01:34.129 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T10:01:34.129 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-image_15.status 2024-04-06T10:01:34.129 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T10:01:34.129 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-image_15.status 2024-04-06T10:01:34.131 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-image_15.status 2024-04-06T10:01:34.131 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_15 2024-04-06T10:01:34.131 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_15.status 2024-04-06T10:01:34.181 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 1 2024-04-06T10:01:34.181 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 1 2024-04-06T10:01:34.181 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T10:01:34.181 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=5, tag_tid=2, entry_tid=1], [object_number=4, tag_tid=2, entry_tid=0], [object_number=6, tag_tid=1, entry_tid=4502], [object_number=7, tag_tid=1, entry_tid=4499]]], state=connected] 2024-04-06T10:01:34.181 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=6, tag_tid=1, entry_tid=4502], [object_number=5, tag_tid=1, entry_tid=4501], [object_number=4, tag_tid=1, entry_tid=4500], [object_number=7, tag_tid=1, entry_tid=4499]]], state=connected] 2024-04-06T10:01:34.184 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_15.status 2024-04-06T10:01:34.186 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=5, tag_tid=2, entry_tid=1' 2024-04-06T10:01:34.187 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror image_15 2024-04-06T10:01:34.187 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:01:34.187 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:01:34.187 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_15 2024-04-06T10:01:34.187 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror image_15 '..*' 2024-04-06T10:01:34.187 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:01:34.187 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:01:34.187 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_15 2024-04-06T10:01:34.188 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T10:01:34.188 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-image_15.status 2024-04-06T10:01:34.188 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-image_15.status 2024-04-06T10:01:34.188 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T10:01:34.188 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-image_15.status 2024-04-06T10:01:34.189 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_15 2024-04-06T10:01:34.189 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_15.status 2024-04-06T10:01:34.234 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 1 2024-04-06T10:01:34.234 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 1 2024-04-06T10:01:34.234 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T10:01:34.234 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=5, tag_tid=2, entry_tid=1], [object_number=4, tag_tid=2, entry_tid=0], [object_number=6, tag_tid=1, entry_tid=4502], [object_number=7, tag_tid=1, entry_tid=4499]]], state=connected] 2024-04-06T10:01:34.235 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=6, tag_tid=1, entry_tid=4502], [object_number=5, tag_tid=1, entry_tid=4501], [object_number=4, tag_tid=1, entry_tid=4500], [object_number=7, tag_tid=1, entry_tid=4499]]], state=connected] 2024-04-06T10:01:34.238 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_15.status 2024-04-06T10:01:34.239 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=6, tag_tid=1, entry_tid=4502' 2024-04-06T10:01:34.240 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=5, tag_tid=2, entry_tid=1' -a 'object_number=5, tag_tid=2, entry_tid=1' = 'object_number=6, tag_tid=1, entry_tid=4502' 2024-04-06T10:01:34.240 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=6, tag_tid=1, entry_tid=4502' '!=' 'object_number=6, tag_tid=1, entry_tid=4502' 2024-04-06T10:01:34.240 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T10:01:34.240 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.8 2024-04-06T10:01:35.042 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror image_15 2024-04-06T10:01:35.042 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T10:01:35.042 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:01:35.042 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_15 2024-04-06T10:01:35.042 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T10:01:35.042 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n image_15 ']' 2024-04-06T10:01:35.042 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/image_15' 2024-04-06T10:01:35.042 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_15 2024-04-06T10:01:35.043 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T10:01:35.043 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T10:01:35.043 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T10:01:35.043 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T10:01:35.043 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T10:01:35.043 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T10:01:35.043 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T10:01:35.043 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T10:01:35.043 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T10:01:35.046 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T10:01:35.046 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_15 2024-04-06T10:01:35.047 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T10:01:35.047 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T10:01:35.047 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T10:01:35.047 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T10:01:35.047 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T10:01:35.047 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T10:01:35.047 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T10:01:35.047 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T10:01:35.047 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T10:01:35.047 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T10:01:35.047 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T10:01:35.047 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:01:35.047 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T10:01:35.047 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T10:01:35.047 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T10:01:35.048 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T10:01:35.049 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T10:01:35.049 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T10:01:35.049 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T10:01:35.049 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T10:01:35.049 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T10:01:35.049 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T10:01:35.049 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:01:35.049 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T10:01:35.049 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T10:01:35.049 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T10:01:35.049 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T10:01:35.050 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T10:01:35.050 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T10:01:35.050 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T10:01:35.072 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T10:01:35.072 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T10:01:35.072 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T10:01:35.073 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_15 2024-04-06T10:01:35.177 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:Invalid command: unused arguments: ['mirror/image_15'] 2024-04-06T10:01:35.177 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:rbd mirror flush : flush rbd mirror 2024-04-06T10:01:35.177 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:admin_socket: invalid command 2024-04-06T10:01:35.179 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T10:01:35.179 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/image_15 2024-04-06T10:01:35.179 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:1 2024-04-06T10:01:35.179 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T10:01:35.180 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T10:01:35.180 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-06T10:01:35.180 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:1 2024-04-06T10:01:35.180 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T10:01:35.180 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T10:01:35.180 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T10:01:35.180 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=1 2024-04-06T10:01:35.180 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-06T10:01:35.180 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T10:01:35.180 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:01:35.180 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=1 2024-04-06T10:01:35.180 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=1 2024-04-06T10:01:35.182 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-06T10:01:35.182 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:1 2024-04-06T10:01:35.182 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T10:01:35.182 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T10:01:35.182 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-06T10:01:35.182 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:1 2024-04-06T10:01:35.182 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T10:01:35.182 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T10:01:35.182 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:01:35.182 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=1 2024-04-06T10:01:35.182 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-06T10:01:35.182 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T10:01:35.182 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T10:01:35.183 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=1 2024-04-06T10:01:35.183 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=1 2024-04-06T10:01:35.183 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-06T10:01:35.206 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T10:01:35.206 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T10:01:35.206 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T10:01:35.206 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/image_15 2024-04-06T10:01:35.347 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T10:01:35.348 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror image_15 2024-04-06T10:01:35.348 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:01:35.349 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:01:35.349 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_15 2024-04-06T10:01:35.349 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror image_15 '' 2024-04-06T10:01:35.349 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:01:35.349 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:01:35.349 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_15 2024-04-06T10:01:35.349 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T10:01:35.349 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-image_15.status 2024-04-06T10:01:35.350 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-image_15.status 2024-04-06T10:01:35.350 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T10:01:35.354 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-image_15.status 2024-04-06T10:01:35.354 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_15 2024-04-06T10:01:35.354 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_15.status 2024-04-06T10:01:35.396 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 1 2024-04-06T10:01:35.396 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 1 2024-04-06T10:01:35.396 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T10:01:35.396 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=5, tag_tid=2, entry_tid=1], [object_number=4, tag_tid=2, entry_tid=0], [object_number=6, tag_tid=1, entry_tid=4502], [object_number=7, tag_tid=1, entry_tid=4499]]], state=connected] 2024-04-06T10:01:35.396 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=6, tag_tid=1, entry_tid=4502], [object_number=5, tag_tid=1, entry_tid=4501], [object_number=4, tag_tid=1, entry_tid=4500], [object_number=7, tag_tid=1, entry_tid=4499]]], state=connected] 2024-04-06T10:01:35.400 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_15.status 2024-04-06T10:01:35.401 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=5, tag_tid=2, entry_tid=1' 2024-04-06T10:01:35.402 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror image_15 2024-04-06T10:01:35.403 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:01:35.403 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:01:35.403 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_15 2024-04-06T10:01:35.403 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror image_15 '..*' 2024-04-06T10:01:35.403 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:01:35.403 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:01:35.403 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_15 2024-04-06T10:01:35.403 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T10:01:35.403 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-image_15.status 2024-04-06T10:01:35.403 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-image_15.status 2024-04-06T10:01:35.403 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T10:01:35.404 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-image_15.status 2024-04-06T10:01:35.404 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_15 2024-04-06T10:01:35.404 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_15.status 2024-04-06T10:01:35.448 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 1 2024-04-06T10:01:35.448 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 1 2024-04-06T10:01:35.448 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T10:01:35.448 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=5, tag_tid=2, entry_tid=1], [object_number=4, tag_tid=2, entry_tid=0], [object_number=6, tag_tid=1, entry_tid=4502], [object_number=7, tag_tid=1, entry_tid=4499]]], state=connected] 2024-04-06T10:01:35.448 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=6, tag_tid=1, entry_tid=4502], [object_number=5, tag_tid=1, entry_tid=4501], [object_number=4, tag_tid=1, entry_tid=4500], [object_number=7, tag_tid=1, entry_tid=4499]]], state=connected] 2024-04-06T10:01:35.452 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_15.status 2024-04-06T10:01:35.453 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=6, tag_tid=1, entry_tid=4502' 2024-04-06T10:01:35.453 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=5, tag_tid=2, entry_tid=1' -a 'object_number=5, tag_tid=2, entry_tid=1' = 'object_number=6, tag_tid=1, entry_tid=4502' 2024-04-06T10:01:35.454 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=6, tag_tid=1, entry_tid=4502' '!=' 'object_number=6, tag_tid=1, entry_tid=4502' 2024-04-06T10:01:35.454 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T10:01:35.454 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 1.6 2024-04-06T10:01:37.056 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror image_15 2024-04-06T10:01:37.056 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T10:01:37.056 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:01:37.056 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_15 2024-04-06T10:01:37.056 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T10:01:37.056 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n image_15 ']' 2024-04-06T10:01:37.056 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/image_15' 2024-04-06T10:01:37.056 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_15 2024-04-06T10:01:37.056 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T10:01:37.056 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T10:01:37.056 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T10:01:37.057 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T10:01:37.057 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T10:01:37.057 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T10:01:37.057 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T10:01:37.057 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T10:01:37.057 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T10:01:37.061 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T10:01:37.061 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_15 2024-04-06T10:01:37.061 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T10:01:37.061 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T10:01:37.061 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T10:01:37.061 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T10:01:37.061 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T10:01:37.061 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T10:01:37.061 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T10:01:37.061 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T10:01:37.061 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T10:01:37.062 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T10:01:37.062 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T10:01:37.062 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:01:37.062 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T10:01:37.062 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T10:01:37.062 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T10:01:37.062 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T10:01:37.062 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T10:01:37.062 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T10:01:37.062 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T10:01:37.062 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T10:01:37.062 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T10:01:37.062 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T10:01:37.062 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:01:37.063 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T10:01:37.063 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T10:01:37.063 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T10:01:37.063 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T10:01:37.063 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T10:01:37.063 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T10:01:37.063 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T10:01:37.088 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T10:01:37.088 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T10:01:37.088 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T10:01:37.088 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_15 2024-04-06T10:01:37.199 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:Invalid command: unused arguments: ['mirror/image_15'] 2024-04-06T10:01:37.199 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:rbd mirror flush : flush rbd mirror 2024-04-06T10:01:37.199 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:admin_socket: invalid command 2024-04-06T10:01:37.201 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T10:01:37.201 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/image_15 2024-04-06T10:01:37.201 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:1 2024-04-06T10:01:37.201 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T10:01:37.201 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T10:01:37.202 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-06T10:01:37.202 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:1 2024-04-06T10:01:37.202 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T10:01:37.202 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T10:01:37.202 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T10:01:37.202 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=1 2024-04-06T10:01:37.202 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-06T10:01:37.202 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T10:01:37.202 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:01:37.202 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=1 2024-04-06T10:01:37.202 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=1 2024-04-06T10:01:37.204 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-06T10:01:37.204 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:1 2024-04-06T10:01:37.204 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T10:01:37.204 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T10:01:37.204 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-06T10:01:37.204 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:1 2024-04-06T10:01:37.204 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T10:01:37.204 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T10:01:37.204 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:01:37.204 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=1 2024-04-06T10:01:37.204 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-06T10:01:37.204 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T10:01:37.204 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T10:01:37.205 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=1 2024-04-06T10:01:37.205 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=1 2024-04-06T10:01:37.205 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-06T10:01:37.222 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T10:01:37.222 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T10:01:37.222 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T10:01:37.222 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/image_15 2024-04-06T10:01:37.385 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T10:01:37.394 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror image_15 2024-04-06T10:01:37.394 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:01:37.394 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:01:37.394 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_15 2024-04-06T10:01:37.394 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror image_15 '' 2024-04-06T10:01:37.394 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:01:37.394 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:01:37.394 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_15 2024-04-06T10:01:37.394 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T10:01:37.395 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-image_15.status 2024-04-06T10:01:37.395 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-image_15.status 2024-04-06T10:01:37.395 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T10:01:37.397 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-image_15.status 2024-04-06T10:01:37.398 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_15 2024-04-06T10:01:37.398 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_15.status 2024-04-06T10:01:37.447 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 1 2024-04-06T10:01:37.448 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 1 2024-04-06T10:01:37.448 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T10:01:37.448 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=5, tag_tid=2, entry_tid=1], [object_number=4, tag_tid=2, entry_tid=0], [object_number=6, tag_tid=1, entry_tid=4502], [object_number=7, tag_tid=1, entry_tid=4499]]], state=connected] 2024-04-06T10:01:37.448 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=6, tag_tid=1, entry_tid=4502], [object_number=5, tag_tid=1, entry_tid=4501], [object_number=4, tag_tid=1, entry_tid=4500], [object_number=7, tag_tid=1, entry_tid=4499]]], state=connected] 2024-04-06T10:01:37.451 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_15.status 2024-04-06T10:01:37.452 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=5, tag_tid=2, entry_tid=1' 2024-04-06T10:01:37.453 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror image_15 2024-04-06T10:01:37.453 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:01:37.454 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:01:37.454 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_15 2024-04-06T10:01:37.454 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror image_15 '..*' 2024-04-06T10:01:37.454 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:01:37.454 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:01:37.454 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_15 2024-04-06T10:01:37.454 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T10:01:37.454 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-image_15.status 2024-04-06T10:01:37.454 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-image_15.status 2024-04-06T10:01:37.454 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T10:01:37.455 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-image_15.status 2024-04-06T10:01:37.455 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_15 2024-04-06T10:01:37.455 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_15.status 2024-04-06T10:01:37.499 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 1 2024-04-06T10:01:37.499 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 1 2024-04-06T10:01:37.499 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T10:01:37.500 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=5, tag_tid=2, entry_tid=1], [object_number=4, tag_tid=2, entry_tid=0], [object_number=6, tag_tid=1, entry_tid=4502], [object_number=7, tag_tid=1, entry_tid=4499]]], state=connected] 2024-04-06T10:01:37.500 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=6, tag_tid=1, entry_tid=4502], [object_number=5, tag_tid=1, entry_tid=4501], [object_number=4, tag_tid=1, entry_tid=4500], [object_number=7, tag_tid=1, entry_tid=4499]]], state=connected] 2024-04-06T10:01:37.503 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_15.status 2024-04-06T10:01:37.505 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=6, tag_tid=1, entry_tid=4502' 2024-04-06T10:01:37.505 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=5, tag_tid=2, entry_tid=1' -a 'object_number=5, tag_tid=2, entry_tid=1' = 'object_number=6, tag_tid=1, entry_tid=4502' 2024-04-06T10:01:37.505 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=6, tag_tid=1, entry_tid=4502' '!=' 'object_number=6, tag_tid=1, entry_tid=4502' 2024-04-06T10:01:37.505 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T10:01:37.505 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 2 2024-04-06T10:01:39.507 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror image_15 2024-04-06T10:01:39.507 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T10:01:39.507 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:01:39.507 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_15 2024-04-06T10:01:39.508 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T10:01:39.508 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n image_15 ']' 2024-04-06T10:01:39.508 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/image_15' 2024-04-06T10:01:39.508 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_15 2024-04-06T10:01:39.508 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T10:01:39.508 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T10:01:39.508 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T10:01:39.508 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T10:01:39.508 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T10:01:39.508 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T10:01:39.508 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T10:01:39.509 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T10:01:39.509 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T10:01:39.512 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T10:01:39.512 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_15 2024-04-06T10:01:39.512 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T10:01:39.512 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T10:01:39.512 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T10:01:39.512 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T10:01:39.512 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T10:01:39.512 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T10:01:39.512 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T10:01:39.512 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T10:01:39.513 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T10:01:39.513 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T10:01:39.513 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T10:01:39.513 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:01:39.513 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T10:01:39.513 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T10:01:39.513 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T10:01:39.513 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T10:01:39.513 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T10:01:39.513 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T10:01:39.513 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T10:01:39.513 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T10:01:39.514 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T10:01:39.514 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T10:01:39.514 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:01:39.514 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T10:01:39.514 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T10:01:39.514 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T10:01:39.514 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T10:01:39.514 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T10:01:39.514 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T10:01:39.514 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T10:01:39.537 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T10:01:39.538 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T10:01:39.538 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T10:01:39.538 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_15 2024-04-06T10:01:39.649 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:Invalid command: unused arguments: ['mirror/image_15'] 2024-04-06T10:01:39.649 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:rbd mirror flush : flush rbd mirror 2024-04-06T10:01:39.649 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:admin_socket: invalid command 2024-04-06T10:01:39.652 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T10:01:39.652 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/image_15 2024-04-06T10:01:39.652 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:1 2024-04-06T10:01:39.652 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T10:01:39.652 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T10:01:39.652 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-06T10:01:39.652 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:1 2024-04-06T10:01:39.652 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T10:01:39.652 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T10:01:39.653 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T10:01:39.653 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=1 2024-04-06T10:01:39.653 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-06T10:01:39.653 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T10:01:39.653 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:01:39.653 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=1 2024-04-06T10:01:39.653 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=1 2024-04-06T10:01:39.654 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-06T10:01:39.654 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:1 2024-04-06T10:01:39.654 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T10:01:39.654 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T10:01:39.654 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-06T10:01:39.654 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:1 2024-04-06T10:01:39.654 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T10:01:39.654 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T10:01:39.655 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:01:39.655 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=1 2024-04-06T10:01:39.655 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-06T10:01:39.655 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T10:01:39.655 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T10:01:39.655 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=1 2024-04-06T10:01:39.655 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=1 2024-04-06T10:01:39.655 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-06T10:01:39.673 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T10:01:39.673 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T10:01:39.673 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T10:01:39.673 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/image_15 2024-04-06T10:01:39.814 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T10:01:39.815 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror image_15 2024-04-06T10:01:39.815 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:01:39.815 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:01:39.815 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_15 2024-04-06T10:01:39.815 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror image_15 '' 2024-04-06T10:01:39.816 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:01:39.816 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:01:39.816 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_15 2024-04-06T10:01:39.816 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T10:01:39.816 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-image_15.status 2024-04-06T10:01:39.816 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-image_15.status 2024-04-06T10:01:39.817 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T10:01:39.819 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-image_15.status 2024-04-06T10:01:39.820 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_15 2024-04-06T10:01:39.820 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_15.status 2024-04-06T10:01:39.870 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 1 2024-04-06T10:01:39.870 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 1 2024-04-06T10:01:39.870 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T10:01:39.874 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=5, tag_tid=2, entry_tid=1], [object_number=4, tag_tid=2, entry_tid=0], [object_number=6, tag_tid=1, entry_tid=4502], [object_number=7, tag_tid=1, entry_tid=4499]]], state=connected] 2024-04-06T10:01:39.875 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=6, tag_tid=1, entry_tid=4502], [object_number=5, tag_tid=1, entry_tid=4501], [object_number=4, tag_tid=1, entry_tid=4500], [object_number=7, tag_tid=1, entry_tid=4499]]], state=connected] 2024-04-06T10:01:39.875 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_15.status 2024-04-06T10:01:39.875 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=5, tag_tid=2, entry_tid=1' 2024-04-06T10:01:39.876 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror image_15 2024-04-06T10:01:39.877 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:01:39.877 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:01:39.877 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_15 2024-04-06T10:01:39.877 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror image_15 '..*' 2024-04-06T10:01:39.877 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:01:39.877 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:01:39.877 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_15 2024-04-06T10:01:39.877 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T10:01:39.877 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-image_15.status 2024-04-06T10:01:39.877 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-image_15.status 2024-04-06T10:01:39.877 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T10:01:39.878 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-image_15.status 2024-04-06T10:01:39.878 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_15 2024-04-06T10:01:39.878 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_15.status 2024-04-06T10:01:39.924 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 1 2024-04-06T10:01:39.924 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 1 2024-04-06T10:01:39.924 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T10:01:39.924 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=5, tag_tid=2, entry_tid=1], [object_number=4, tag_tid=2, entry_tid=0], [object_number=6, tag_tid=1, entry_tid=4502], [object_number=7, tag_tid=1, entry_tid=4499]]], state=connected] 2024-04-06T10:01:39.924 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=6, tag_tid=1, entry_tid=4502], [object_number=5, tag_tid=1, entry_tid=4501], [object_number=4, tag_tid=1, entry_tid=4500], [object_number=7, tag_tid=1, entry_tid=4499]]], state=connected] 2024-04-06T10:01:39.927 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_15.status 2024-04-06T10:01:39.929 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=6, tag_tid=1, entry_tid=4502' 2024-04-06T10:01:39.929 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=5, tag_tid=2, entry_tid=1' -a 'object_number=5, tag_tid=2, entry_tid=1' = 'object_number=6, tag_tid=1, entry_tid=4502' 2024-04-06T10:01:39.929 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=6, tag_tid=1, entry_tid=4502' '!=' 'object_number=6, tag_tid=1, entry_tid=4502' 2024-04-06T10:01:39.929 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T10:01:39.929 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 2 2024-04-06T10:01:41.931 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror image_15 2024-04-06T10:01:41.931 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T10:01:41.931 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:01:41.931 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_15 2024-04-06T10:01:41.931 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T10:01:41.932 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n image_15 ']' 2024-04-06T10:01:41.932 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/image_15' 2024-04-06T10:01:41.932 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_15 2024-04-06T10:01:41.932 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T10:01:41.932 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T10:01:41.932 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T10:01:41.932 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T10:01:41.932 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T10:01:41.932 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T10:01:41.932 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T10:01:41.932 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T10:01:41.933 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T10:01:41.936 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T10:01:41.937 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_15 2024-04-06T10:01:41.937 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T10:01:41.937 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T10:01:41.937 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T10:01:41.937 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T10:01:41.937 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T10:01:41.937 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T10:01:41.937 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T10:01:41.937 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T10:01:41.937 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T10:01:41.937 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T10:01:41.937 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T10:01:41.937 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:01:41.937 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T10:01:41.938 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T10:01:41.938 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T10:01:41.938 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T10:01:41.938 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T10:01:41.938 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T10:01:41.938 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T10:01:41.938 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T10:01:41.938 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T10:01:41.938 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T10:01:41.939 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:01:41.939 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T10:01:41.939 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T10:01:41.939 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T10:01:41.939 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T10:01:41.939 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T10:01:41.939 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T10:01:41.939 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T10:01:41.962 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T10:01:41.963 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T10:01:41.963 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T10:01:41.963 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_15 2024-04-06T10:01:42.069 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:Invalid command: unused arguments: ['mirror/image_15'] 2024-04-06T10:01:42.069 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:rbd mirror flush : flush rbd mirror 2024-04-06T10:01:42.069 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:admin_socket: invalid command 2024-04-06T10:01:42.071 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T10:01:42.071 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/image_15 2024-04-06T10:01:42.071 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:1 2024-04-06T10:01:42.071 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T10:01:42.071 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T10:01:42.071 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-06T10:01:42.071 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:1 2024-04-06T10:01:42.071 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T10:01:42.071 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T10:01:42.071 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T10:01:42.071 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=1 2024-04-06T10:01:42.072 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-06T10:01:42.072 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T10:01:42.072 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:01:42.072 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=1 2024-04-06T10:01:42.072 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=1 2024-04-06T10:01:42.073 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-06T10:01:42.073 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:1 2024-04-06T10:01:42.074 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T10:01:42.074 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T10:01:42.074 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-06T10:01:42.074 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:1 2024-04-06T10:01:42.074 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T10:01:42.074 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T10:01:42.074 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:01:42.074 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=1 2024-04-06T10:01:42.074 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-06T10:01:42.074 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T10:01:42.074 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T10:01:42.074 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=1 2024-04-06T10:01:42.074 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=1 2024-04-06T10:01:42.074 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-06T10:01:42.095 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T10:01:42.096 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T10:01:42.096 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T10:01:42.096 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/image_15 2024-04-06T10:01:42.239 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T10:01:42.241 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror image_15 2024-04-06T10:01:42.241 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:01:42.241 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:01:42.241 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_15 2024-04-06T10:01:42.241 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror image_15 '' 2024-04-06T10:01:42.241 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:01:42.241 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:01:42.241 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_15 2024-04-06T10:01:42.241 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T10:01:42.241 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-image_15.status 2024-04-06T10:01:42.241 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-image_15.status 2024-04-06T10:01:42.242 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T10:01:42.244 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-image_15.status 2024-04-06T10:01:42.244 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_15 2024-04-06T10:01:42.244 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_15.status 2024-04-06T10:01:42.289 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 1 2024-04-06T10:01:42.289 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 1 2024-04-06T10:01:42.289 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T10:01:42.289 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=5, tag_tid=2, entry_tid=1], [object_number=4, tag_tid=2, entry_tid=0], [object_number=6, tag_tid=1, entry_tid=4502], [object_number=7, tag_tid=1, entry_tid=4499]]], state=connected] 2024-04-06T10:01:42.289 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=5, tag_tid=2, entry_tid=1], [object_number=4, tag_tid=2, entry_tid=0], [object_number=6, tag_tid=1, entry_tid=4502], [object_number=7, tag_tid=1, entry_tid=4499]]], state=connected] 2024-04-06T10:01:42.293 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_15.status 2024-04-06T10:01:42.294 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=5, tag_tid=2, entry_tid=1' 2024-04-06T10:01:42.295 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror image_15 2024-04-06T10:01:42.295 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:01:42.296 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:01:42.296 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_15 2024-04-06T10:01:42.296 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror image_15 '..*' 2024-04-06T10:01:42.296 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:01:42.296 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:01:42.296 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_15 2024-04-06T10:01:42.296 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T10:01:42.296 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-image_15.status 2024-04-06T10:01:42.296 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-image_15.status 2024-04-06T10:01:42.296 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T10:01:42.297 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-image_15.status 2024-04-06T10:01:42.297 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_15 2024-04-06T10:01:42.297 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_15.status 2024-04-06T10:01:42.341 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 1 2024-04-06T10:01:42.342 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 1 2024-04-06T10:01:42.342 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T10:01:42.342 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=5, tag_tid=2, entry_tid=1], [object_number=4, tag_tid=2, entry_tid=0], [object_number=6, tag_tid=1, entry_tid=4502], [object_number=7, tag_tid=1, entry_tid=4499]]], state=connected] 2024-04-06T10:01:42.342 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=5, tag_tid=2, entry_tid=1], [object_number=4, tag_tid=2, entry_tid=0], [object_number=6, tag_tid=1, entry_tid=4502], [object_number=7, tag_tid=1, entry_tid=4499]]], state=connected] 2024-04-06T10:01:42.345 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_15.status 2024-04-06T10:01:42.348 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=5, tag_tid=2, entry_tid=1' 2024-04-06T10:01:42.348 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=5, tag_tid=2, entry_tid=1' -a 'object_number=5, tag_tid=2, entry_tid=1' = 'object_number=5, tag_tid=2, entry_tid=1' 2024-04-06T10:01:42.348 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T10:01:42.348 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_snap_present cluster1 mirror image_15 snap 2024-04-06T10:01:42.348 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T10:01:42.348 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:01:42.348 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_15 2024-04-06T10:01:42.348 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local snap_name=snap 2024-04-06T10:01:42.348 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local s 2024-04-06T10:01:42.348 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-04-06T10:01:42.348 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 1 2024-04-06T10:01:43.348 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 info mirror/image_15@snap 2024-04-06T10:01:43.419 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:rbd image 'image_15': 2024-04-06T10:01:43.419 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: size 128 MiB in 32 objects 2024-04-06T10:01:43.420 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: order 22 (4 MiB objects) 2024-04-06T10:01:43.420 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: snapshot_count: 1 2024-04-06T10:01:43.420 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: id: 10acd9703140 2024-04-06T10:01:43.420 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: block_name_prefix: rbd_data.10acd9703140 2024-04-06T10:01:43.420 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: format: 2 2024-04-06T10:01:43.420 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: features: layering, exclusive-lock, journaling 2024-04-06T10:01:43.420 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: op_features: 2024-04-06T10:01:43.420 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: flags: 2024-04-06T10:01:43.420 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: create_timestamp: Sat Apr 6 09:53:01 2024 2024-04-06T10:01:43.421 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: access_timestamp: Sat Apr 6 09:53:01 2024 2024-04-06T10:01:43.421 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: modify_timestamp: Sat Apr 6 09:53:01 2024 2024-04-06T10:01:43.421 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: protected: False 2024-04-06T10:01:43.421 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: journal: 10acd9703140 2024-04-06T10:01:43.421 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: mirroring state: enabled 2024-04-06T10:01:43.421 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: mirroring mode: journal 2024-04-06T10:01:43.421 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: mirroring global id: 2ef564ed-0213-46fd-bbc0-783ade4ccf9c 2024-04-06T10:01:43.421 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: mirroring primary: false 2024-04-06T10:01:43.425 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T10:01:43.425 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ compare_image_snaps mirror image_15 snap 2024-04-06T10:01:43.425 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:01:43.425 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_15 2024-04-06T10:01:43.425 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local snap_name=snap 2024-04-06T10:01:43.426 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local ret=0 2024-04-06T10:01:43.426 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local rmt_export=/tmp/tmp.BhW6xKmROU/cluster2-mirror-image_15.export 2024-04-06T10:01:43.426 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loc_export=/tmp/tmp.BhW6xKmROU/cluster1-mirror-image_15.export 2024-04-06T10:01:43.426 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rm -f /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_15.export /tmp/tmp.BhW6xKmROU/cluster1-mirror-image_15.export 2024-04-06T10:01:43.426 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster2 -p mirror export image_15@snap /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_15.export 2024-04-06T10:01:43.580 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-04-06T10:01:43.589 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 -p mirror export image_15@snap /tmp/tmp.BhW6xKmROU/cluster1-mirror-image_15.export 2024-04-06T10:01:43.928 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-04-06T10:01:43.940 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmp /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_15.export /tmp/tmp.BhW6xKmROU/cluster1-mirror-image_15.export 2024-04-06T10:01:44.020 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rm -f /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_15.export /tmp/tmp.BhW6xKmROU/cluster1-mirror-image_15.export 2024-04-06T10:01:44.061 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T10:01:44.062 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-06T10:01:44.062 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ image=image_16 2024-04-06T10:01:44.062 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ create_snap cluster2 mirror image_16 snap 2024-04-06T10:01:44.062 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-04-06T10:01:44.062 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:01:44.062 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_16 2024-04-06T10:01:44.062 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local snap_name=snap 2024-04-06T10:01:44.062 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster2 -p mirror snap create image_16@snap --debug-rbd=20 --debug-journaler=20 2024-04-06T10:01:46.844 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_image_replay_started cluster1 mirror image_16 2024-04-06T10:01:46.844 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T10:01:46.844 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:01:46.844 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_16 2024-04-06T10:01:46.844 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_image_replay_state cluster1 mirror image_16 started 2024-04-06T10:01:46.844 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T10:01:46.844 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:01:46.844 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_16 2024-04-06T10:01:46.844 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state=started 2024-04-06T10:01:46.844 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local s 2024-04-06T10:01:46.844 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T10:01:46.844 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 1 2024-04-06T10:01:47.845 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test_image_replay_state cluster1 mirror image_16 started 2024-04-06T10:01:47.846 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T10:01:47.846 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:01:47.846 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_16 2024-04-06T10:01:47.846 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local test_state=started 2024-04-06T10:01:47.846 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local status_result 2024-04-06T10:01:47.846 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local current_state=stopped 2024-04-06T10:01:47.847 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_16 2024-04-06T10:01:47.847 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -i state 2024-04-06T10:01:47.847 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_instance=cluster1 2024-04-06T10:01:47.847 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ shift 2024-04-06T10:01:47.847 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T10:01:47.847 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance=cluster1 2024-04-06T10:01:47.847 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local loop_instance 2024-04-06T10:01:47.847 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T10:01:47.847 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sleep 0 2024-04-06T10:01:47.849 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-04-06T10:01:47.849 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ seq 0 3 2024-04-06T10:01:47.853 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T10:01:47.853 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_16 2024-04-06T10:01:47.853 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1:0 2024-04-06T10:01:47.853 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ shift 2024-04-06T10:01:47.853 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T10:01:47.853 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-04-06T10:01:47.853 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T10:01:47.853 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=cluster 2024-04-06T10:01:47.853 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T10:01:47.854 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:01:47.854 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T10:01:47.854 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T10:01:47.854 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval cluster=cluster1 2024-04-06T10:01:47.854 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ cluster=cluster1 2024-04-06T10:01:47.854 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T10:01:47.854 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T10:01:47.854 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-04-06T10:01:47.854 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local local_cluster=cluster1:0 2024-04-06T10:01:47.854 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local cluster=cluster1 2024-04-06T10:01:47.854 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local instance 2024-04-06T10:01:47.854 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T10:01:47.854 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local val=cluster1:0 2024-04-06T10:01:47.854 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local cluster_var_name=local_cluster 2024-04-06T10:01:47.855 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local instance_var_name=instance 2024-04-06T10:01:47.855 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ cluster=cluster1 2024-04-06T10:01:47.855 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T10:01:47.855 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ '[' 0 = cluster1:0 ']' 2024-04-06T10:01:47.855 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ eval local_cluster=cluster1 2024-04-06T10:01:47.855 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++++ local_cluster=cluster1 2024-04-06T10:01:47.855 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ eval instance=0 2024-04-06T10:01:47.855 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++++ instance=0 2024-04-06T10:01:47.855 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T10:01:47.876 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T10:01:47.877 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T10:01:47.877 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T10:01:47.877 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_16 2024-04-06T10:01:47.986 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:Invalid command: unused arguments: ['mirror/image_16'] 2024-04-06T10:01:47.987 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:rbd mirror status : get status for rbd mirror 2024-04-06T10:01:47.987 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:admin_socket: invalid command 2024-04-06T10:01:47.988 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T10:01:47.988 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_16 2024-04-06T10:01:47.988 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1:1 2024-04-06T10:01:47.988 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ shift 2024-04-06T10:01:47.989 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T10:01:47.989 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-04-06T10:01:47.989 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:1 2024-04-06T10:01:47.989 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=cluster 2024-04-06T10:01:47.989 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T10:01:47.989 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:01:47.989 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=1 2024-04-06T10:01:47.989 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-06T10:01:47.989 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval cluster=cluster1 2024-04-06T10:01:47.989 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ cluster=cluster1 2024-04-06T10:01:47.989 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=1 2024-04-06T10:01:47.989 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=1 2024-04-06T10:01:47.991 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-04-06T10:01:47.991 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local local_cluster=cluster1:1 2024-04-06T10:01:47.991 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local cluster=cluster1 2024-04-06T10:01:47.991 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local instance 2024-04-06T10:01:47.991 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-06T10:01:47.991 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local val=cluster1:1 2024-04-06T10:01:47.991 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local cluster_var_name=local_cluster 2024-04-06T10:01:47.991 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local instance_var_name=instance 2024-04-06T10:01:47.991 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ cluster=cluster1 2024-04-06T10:01:47.991 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=1 2024-04-06T10:01:47.991 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ '[' 1 = cluster1:1 ']' 2024-04-06T10:01:47.991 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ eval local_cluster=cluster1 2024-04-06T10:01:47.992 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++++ local_cluster=cluster1 2024-04-06T10:01:47.992 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ eval instance=1 2024-04-06T10:01:47.992 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++++ instance=1 2024-04-06T10:01:47.992 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-06T10:01:48.009 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T10:01:48.010 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T10:01:48.010 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T10:01:48.010 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_16 2024-04-06T10:01:48.164 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ return 0 2024-04-06T10:01:48.165 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ status_result=' "state": "Replaying"' 2024-04-06T10:01:48.165 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ echo ' "state": "Replaying"' 2024-04-06T10:01:48.166 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep -i Replaying 2024-04-06T10:01:48.167 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: "state": "Replaying" 2024-04-06T10:01:48.169 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ current_state=started 2024-04-06T10:01:48.169 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test started = started 2024-04-06T10:01:48.169 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T10:01:48.169 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_16 2024-04-06T10:01:48.169 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local local_cluster=cluster1 2024-04-06T10:01:48.169 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-04-06T10:01:48.169 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:01:48.169 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_16 2024-04-06T10:01:48.169 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' journal = journal ']' 2024-04-06T10:01:48.169 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_journal_replay_complete cluster1 cluster2 mirror image_16 2024-04-06T10:01:48.169 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local local_cluster=cluster1 2024-04-06T10:01:48.170 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-04-06T10:01:48.170 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:01:48.170 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_16 2024-04-06T10:01:48.170 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local s master_pos mirror_pos last_mirror_pos 2024-04-06T10:01:48.170 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local master_tag master_entry mirror_tag mirror_entry 2024-04-06T10:01:48.170 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T10:01:48.170 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T10:01:48.170 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T10:01:48.371 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror image_16 2024-04-06T10:01:48.371 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T10:01:48.372 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:01:48.372 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_16 2024-04-06T10:01:48.372 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T10:01:48.372 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n image_16 ']' 2024-04-06T10:01:48.372 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/image_16' 2024-04-06T10:01:48.372 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_16 2024-04-06T10:01:48.372 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T10:01:48.372 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T10:01:48.372 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T10:01:48.372 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T10:01:48.372 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T10:01:48.372 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T10:01:48.372 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T10:01:48.373 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T10:01:48.374 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T10:01:48.376 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T10:01:48.376 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_16 2024-04-06T10:01:48.376 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T10:01:48.376 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T10:01:48.376 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T10:01:48.376 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T10:01:48.376 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T10:01:48.376 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T10:01:48.376 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T10:01:48.376 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T10:01:48.376 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T10:01:48.377 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T10:01:48.377 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T10:01:48.377 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:01:48.377 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T10:01:48.377 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T10:01:48.378 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T10:01:48.378 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T10:01:48.378 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T10:01:48.378 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T10:01:48.378 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T10:01:48.378 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T10:01:48.378 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T10:01:48.378 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T10:01:48.379 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:01:48.379 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T10:01:48.379 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T10:01:48.379 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T10:01:48.379 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T10:01:48.379 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T10:01:48.379 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T10:01:48.379 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T10:01:48.404 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T10:01:48.404 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T10:01:48.404 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T10:01:48.405 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_16 2024-04-06T10:01:48.507 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:Invalid command: unused arguments: ['mirror/image_16'] 2024-04-06T10:01:48.507 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:rbd mirror flush : flush rbd mirror 2024-04-06T10:01:48.507 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:admin_socket: invalid command 2024-04-06T10:01:48.509 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T10:01:48.509 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/image_16 2024-04-06T10:01:48.510 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:1 2024-04-06T10:01:48.510 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T10:01:48.510 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T10:01:48.510 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-06T10:01:48.510 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:1 2024-04-06T10:01:48.510 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T10:01:48.510 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T10:01:48.510 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T10:01:48.510 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=1 2024-04-06T10:01:48.510 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-06T10:01:48.510 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T10:01:48.510 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:01:48.510 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=1 2024-04-06T10:01:48.510 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=1 2024-04-06T10:01:48.512 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-06T10:01:48.512 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:1 2024-04-06T10:01:48.512 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T10:01:48.512 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T10:01:48.512 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-06T10:01:48.512 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:1 2024-04-06T10:01:48.512 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T10:01:48.512 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T10:01:48.512 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:01:48.512 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=1 2024-04-06T10:01:48.513 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-06T10:01:48.513 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T10:01:48.513 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T10:01:48.513 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=1 2024-04-06T10:01:48.513 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=1 2024-04-06T10:01:48.513 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-06T10:01:48.536 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T10:01:48.537 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T10:01:48.537 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T10:01:48.537 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/image_16 2024-04-06T10:01:48.673 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T10:01:48.674 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror image_16 2024-04-06T10:01:48.675 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:01:48.675 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:01:48.675 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_16 2024-04-06T10:01:48.675 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror image_16 '' 2024-04-06T10:01:48.675 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:01:48.675 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:01:48.675 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_16 2024-04-06T10:01:48.675 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T10:01:48.676 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-image_16.status 2024-04-06T10:01:48.676 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-image_16.status 2024-04-06T10:01:48.676 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T10:01:48.677 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-image_16.status 2024-04-06T10:01:48.678 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_16 2024-04-06T10:01:48.678 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_16.status 2024-04-06T10:01:48.723 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 1 2024-04-06T10:01:48.724 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 1 2024-04-06T10:01:48.724 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T10:01:48.724 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=5, tag_tid=2, entry_tid=1], [object_number=4, tag_tid=2, entry_tid=0], [object_number=7, tag_tid=1, entry_tid=5759], [object_number=6, tag_tid=1, entry_tid=5758]]], state=connected] 2024-04-06T10:01:48.724 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=7, tag_tid=1, entry_tid=5759], [object_number=6, tag_tid=1, entry_tid=5758], [object_number=5, tag_tid=1, entry_tid=5757], [object_number=4, tag_tid=1, entry_tid=5756]]], state=connected] 2024-04-06T10:01:48.727 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_16.status 2024-04-06T10:01:48.728 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=5, tag_tid=2, entry_tid=1' 2024-04-06T10:01:48.730 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror image_16 2024-04-06T10:01:48.730 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:01:48.730 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:01:48.730 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_16 2024-04-06T10:01:48.730 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror image_16 '..*' 2024-04-06T10:01:48.730 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:01:48.730 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:01:48.730 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_16 2024-04-06T10:01:48.730 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T10:01:48.730 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-image_16.status 2024-04-06T10:01:48.730 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-image_16.status 2024-04-06T10:01:48.730 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T10:01:48.731 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-image_16.status 2024-04-06T10:01:48.731 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_16 2024-04-06T10:01:48.731 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_16.status 2024-04-06T10:01:48.776 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 1 2024-04-06T10:01:48.776 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 1 2024-04-06T10:01:48.776 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T10:01:48.776 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=5, tag_tid=2, entry_tid=1], [object_number=4, tag_tid=2, entry_tid=0], [object_number=7, tag_tid=1, entry_tid=5759], [object_number=6, tag_tid=1, entry_tid=5758]]], state=connected] 2024-04-06T10:01:48.776 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=7, tag_tid=1, entry_tid=5759], [object_number=6, tag_tid=1, entry_tid=5758], [object_number=5, tag_tid=1, entry_tid=5757], [object_number=4, tag_tid=1, entry_tid=5756]]], state=connected] 2024-04-06T10:01:48.779 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_16.status 2024-04-06T10:01:48.781 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=7, tag_tid=1, entry_tid=5759' 2024-04-06T10:01:48.781 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=5, tag_tid=2, entry_tid=1' -a 'object_number=5, tag_tid=2, entry_tid=1' = 'object_number=7, tag_tid=1, entry_tid=5759' 2024-04-06T10:01:48.781 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=7, tag_tid=1, entry_tid=5759' '!=' '' 2024-04-06T10:01:48.781 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T10:01:48.781 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=7, tag_tid=1, entry_tid=5759' = '' 2024-04-06T10:01:48.781 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=7, tag_tid=1, entry_tid=5759' 2024-04-06T10:01:48.782 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=5, tag_tid=2, entry_tid=1' 2024-04-06T10:01:48.782 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T10:01:48.782 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T10:01:48.783 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=2 2024-04-06T10:01:48.783 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=7, tag_tid=1, entry_tid=5759' 2024-04-06T10:01:48.783 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T10:01:48.783 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T10:01:48.784 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=1 2024-04-06T10:01:48.785 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=5, tag_tid=2, entry_tid=1' 2024-04-06T10:01:48.785 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T10:01:48.785 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T10:01:48.786 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T10:01:48.786 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=7, tag_tid=1, entry_tid=5759' 2024-04-06T10:01:48.787 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T10:01:48.787 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T10:01:48.788 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=5759 2024-04-06T10:01:48.788 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 2 = 1 -a 1 -le 5759 2024-04-06T10:01:48.788 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T10:01:48.788 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T10:01:48.788 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T10:01:48.990 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror image_16 2024-04-06T10:01:48.990 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T10:01:48.990 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:01:48.990 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_16 2024-04-06T10:01:48.990 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T10:01:48.990 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n image_16 ']' 2024-04-06T10:01:48.990 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/image_16' 2024-04-06T10:01:48.991 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_16 2024-04-06T10:01:48.991 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T10:01:48.991 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T10:01:48.991 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T10:01:48.991 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T10:01:48.991 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T10:01:48.991 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T10:01:48.991 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T10:01:48.991 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T10:01:48.992 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T10:01:48.995 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T10:01:48.995 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_16 2024-04-06T10:01:48.995 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T10:01:48.995 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T10:01:48.995 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T10:01:48.996 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T10:01:48.996 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T10:01:48.996 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T10:01:48.996 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T10:01:48.996 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T10:01:48.996 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T10:01:48.996 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T10:01:48.996 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T10:01:48.996 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:01:48.996 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T10:01:48.996 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T10:01:48.996 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T10:01:48.996 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T10:01:48.997 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T10:01:48.997 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T10:01:48.997 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T10:01:48.997 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T10:01:48.997 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T10:01:48.997 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T10:01:48.997 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:01:48.997 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T10:01:48.997 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T10:01:48.997 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T10:01:48.997 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T10:01:48.997 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T10:01:48.997 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T10:01:48.998 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T10:01:49.022 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T10:01:49.023 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T10:01:49.023 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T10:01:49.023 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_16 2024-04-06T10:01:49.128 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:Invalid command: unused arguments: ['mirror/image_16'] 2024-04-06T10:01:49.128 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:rbd mirror flush : flush rbd mirror 2024-04-06T10:01:49.128 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:admin_socket: invalid command 2024-04-06T10:01:49.131 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T10:01:49.132 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/image_16 2024-04-06T10:01:49.132 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:1 2024-04-06T10:01:49.132 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T10:01:49.132 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T10:01:49.132 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-06T10:01:49.132 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:1 2024-04-06T10:01:49.132 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T10:01:49.132 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T10:01:49.132 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T10:01:49.132 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=1 2024-04-06T10:01:49.132 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-06T10:01:49.132 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T10:01:49.132 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:01:49.132 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=1 2024-04-06T10:01:49.133 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=1 2024-04-06T10:01:49.133 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-06T10:01:49.133 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:1 2024-04-06T10:01:49.133 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T10:01:49.133 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T10:01:49.133 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-06T10:01:49.133 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:1 2024-04-06T10:01:49.133 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T10:01:49.133 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T10:01:49.133 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:01:49.133 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=1 2024-04-06T10:01:49.133 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-06T10:01:49.133 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T10:01:49.133 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T10:01:49.133 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=1 2024-04-06T10:01:49.134 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=1 2024-04-06T10:01:49.134 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-06T10:01:49.156 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T10:01:49.156 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T10:01:49.156 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T10:01:49.156 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/image_16 2024-04-06T10:01:49.305 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T10:01:49.307 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror image_16 2024-04-06T10:01:49.307 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:01:49.307 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:01:49.307 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_16 2024-04-06T10:01:49.307 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror image_16 '' 2024-04-06T10:01:49.307 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:01:49.307 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:01:49.307 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_16 2024-04-06T10:01:49.307 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T10:01:49.307 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-image_16.status 2024-04-06T10:01:49.308 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-image_16.status 2024-04-06T10:01:49.308 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T10:01:49.309 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-image_16.status 2024-04-06T10:01:49.309 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_16 2024-04-06T10:01:49.310 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_16.status 2024-04-06T10:01:49.353 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 1 2024-04-06T10:01:49.354 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 1 2024-04-06T10:01:49.354 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T10:01:49.354 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=5, tag_tid=2, entry_tid=1], [object_number=4, tag_tid=2, entry_tid=0], [object_number=7, tag_tid=1, entry_tid=5759], [object_number=6, tag_tid=1, entry_tid=5758]]], state=connected] 2024-04-06T10:01:49.354 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=7, tag_tid=1, entry_tid=5759], [object_number=6, tag_tid=1, entry_tid=5758], [object_number=5, tag_tid=1, entry_tid=5757], [object_number=4, tag_tid=1, entry_tid=5756]]], state=connected] 2024-04-06T10:01:49.357 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_16.status 2024-04-06T10:01:49.358 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=5, tag_tid=2, entry_tid=1' 2024-04-06T10:01:49.359 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror image_16 2024-04-06T10:01:49.359 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:01:49.360 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:01:49.360 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_16 2024-04-06T10:01:49.360 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror image_16 '..*' 2024-04-06T10:01:49.360 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:01:49.360 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:01:49.360 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_16 2024-04-06T10:01:49.360 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T10:01:49.360 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-image_16.status 2024-04-06T10:01:49.360 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-image_16.status 2024-04-06T10:01:49.360 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T10:01:49.361 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-image_16.status 2024-04-06T10:01:49.361 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_16 2024-04-06T10:01:49.361 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_16.status 2024-04-06T10:01:49.405 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 1 2024-04-06T10:01:49.405 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 1 2024-04-06T10:01:49.405 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T10:01:49.405 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=5, tag_tid=2, entry_tid=1], [object_number=4, tag_tid=2, entry_tid=0], [object_number=7, tag_tid=1, entry_tid=5759], [object_number=6, tag_tid=1, entry_tid=5758]]], state=connected] 2024-04-06T10:01:49.405 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=7, tag_tid=1, entry_tid=5759], [object_number=6, tag_tid=1, entry_tid=5758], [object_number=5, tag_tid=1, entry_tid=5757], [object_number=4, tag_tid=1, entry_tid=5756]]], state=connected] 2024-04-06T10:01:49.409 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_16.status 2024-04-06T10:01:49.410 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=7, tag_tid=1, entry_tid=5759' 2024-04-06T10:01:49.410 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=5, tag_tid=2, entry_tid=1' -a 'object_number=5, tag_tid=2, entry_tid=1' = 'object_number=7, tag_tid=1, entry_tid=5759' 2024-04-06T10:01:49.410 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=7, tag_tid=1, entry_tid=5759' '!=' 'object_number=7, tag_tid=1, entry_tid=5759' 2024-04-06T10:01:49.411 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T10:01:49.411 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.4 2024-04-06T10:01:49.813 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror image_16 2024-04-06T10:01:49.813 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T10:01:49.813 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:01:49.813 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_16 2024-04-06T10:01:49.813 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T10:01:49.813 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n image_16 ']' 2024-04-06T10:01:49.813 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/image_16' 2024-04-06T10:01:49.814 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_16 2024-04-06T10:01:49.814 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T10:01:49.814 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T10:01:49.814 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T10:01:49.814 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T10:01:49.814 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T10:01:49.814 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T10:01:49.814 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T10:01:49.815 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T10:01:49.816 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T10:01:49.816 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T10:01:49.816 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_16 2024-04-06T10:01:49.816 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T10:01:49.816 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T10:01:49.816 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T10:01:49.816 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T10:01:49.816 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T10:01:49.816 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T10:01:49.816 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T10:01:49.817 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T10:01:49.817 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T10:01:49.817 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T10:01:49.817 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T10:01:49.817 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:01:49.817 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T10:01:49.817 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T10:01:49.818 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T10:01:49.818 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T10:01:49.818 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T10:01:49.819 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T10:01:49.819 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T10:01:49.819 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T10:01:49.819 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T10:01:49.819 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T10:01:49.819 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:01:49.819 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T10:01:49.819 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T10:01:49.819 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T10:01:49.819 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T10:01:49.819 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T10:01:49.820 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T10:01:49.820 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T10:01:49.843 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T10:01:49.843 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T10:01:49.843 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T10:01:49.843 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_16 2024-04-06T10:01:49.948 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:Invalid command: unused arguments: ['mirror/image_16'] 2024-04-06T10:01:49.948 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:rbd mirror flush : flush rbd mirror 2024-04-06T10:01:49.948 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:admin_socket: invalid command 2024-04-06T10:01:49.952 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T10:01:49.952 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/image_16 2024-04-06T10:01:49.952 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:1 2024-04-06T10:01:49.952 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T10:01:49.952 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T10:01:49.952 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-06T10:01:49.952 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:1 2024-04-06T10:01:49.952 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T10:01:49.952 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T10:01:49.952 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T10:01:49.952 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=1 2024-04-06T10:01:49.953 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-06T10:01:49.953 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T10:01:49.953 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:01:49.953 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=1 2024-04-06T10:01:49.953 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=1 2024-04-06T10:01:49.953 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-06T10:01:49.953 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:1 2024-04-06T10:01:49.953 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T10:01:49.953 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T10:01:49.953 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-06T10:01:49.953 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:1 2024-04-06T10:01:49.953 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T10:01:49.953 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T10:01:49.953 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:01:49.954 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=1 2024-04-06T10:01:49.954 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-06T10:01:49.954 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T10:01:49.954 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T10:01:49.954 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=1 2024-04-06T10:01:49.954 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=1 2024-04-06T10:01:49.954 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-06T10:01:49.978 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T10:01:49.978 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T10:01:49.978 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T10:01:49.978 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/image_16 2024-04-06T10:01:50.122 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T10:01:50.124 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror image_16 2024-04-06T10:01:50.124 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:01:50.124 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:01:50.124 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_16 2024-04-06T10:01:50.124 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror image_16 '' 2024-04-06T10:01:50.124 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:01:50.124 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:01:50.124 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_16 2024-04-06T10:01:50.124 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T10:01:50.124 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-image_16.status 2024-04-06T10:01:50.125 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-image_16.status 2024-04-06T10:01:50.125 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T10:01:50.125 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-image_16.status 2024-04-06T10:01:50.126 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_16 2024-04-06T10:01:50.126 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_16.status 2024-04-06T10:01:50.171 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 1 2024-04-06T10:01:50.171 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 1 2024-04-06T10:01:50.172 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T10:01:50.172 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=5, tag_tid=2, entry_tid=1], [object_number=4, tag_tid=2, entry_tid=0], [object_number=7, tag_tid=1, entry_tid=5759], [object_number=6, tag_tid=1, entry_tid=5758]]], state=connected] 2024-04-06T10:01:50.172 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=7, tag_tid=1, entry_tid=5759], [object_number=6, tag_tid=1, entry_tid=5758], [object_number=5, tag_tid=1, entry_tid=5757], [object_number=4, tag_tid=1, entry_tid=5756]]], state=connected] 2024-04-06T10:01:50.175 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_16.status 2024-04-06T10:01:50.177 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=5, tag_tid=2, entry_tid=1' 2024-04-06T10:01:50.178 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror image_16 2024-04-06T10:01:50.178 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:01:50.178 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:01:50.178 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_16 2024-04-06T10:01:50.178 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror image_16 '..*' 2024-04-06T10:01:50.179 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:01:50.179 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:01:50.179 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_16 2024-04-06T10:01:50.179 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T10:01:50.179 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-image_16.status 2024-04-06T10:01:50.179 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-image_16.status 2024-04-06T10:01:50.179 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T10:01:50.180 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-image_16.status 2024-04-06T10:01:50.180 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_16 2024-04-06T10:01:50.180 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_16.status 2024-04-06T10:01:50.224 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 1 2024-04-06T10:01:50.224 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 1 2024-04-06T10:01:50.224 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T10:01:50.224 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=5, tag_tid=2, entry_tid=1], [object_number=4, tag_tid=2, entry_tid=0], [object_number=7, tag_tid=1, entry_tid=5759], [object_number=6, tag_tid=1, entry_tid=5758]]], state=connected] 2024-04-06T10:01:50.224 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=7, tag_tid=1, entry_tid=5759], [object_number=6, tag_tid=1, entry_tid=5758], [object_number=5, tag_tid=1, entry_tid=5757], [object_number=4, tag_tid=1, entry_tid=5756]]], state=connected] 2024-04-06T10:01:50.227 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_16.status 2024-04-06T10:01:50.229 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=7, tag_tid=1, entry_tid=5759' 2024-04-06T10:01:50.229 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=5, tag_tid=2, entry_tid=1' -a 'object_number=5, tag_tid=2, entry_tid=1' = 'object_number=7, tag_tid=1, entry_tid=5759' 2024-04-06T10:01:50.229 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=7, tag_tid=1, entry_tid=5759' '!=' 'object_number=7, tag_tid=1, entry_tid=5759' 2024-04-06T10:01:50.229 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T10:01:50.229 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.8 2024-04-06T10:01:51.031 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror image_16 2024-04-06T10:01:51.031 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T10:01:51.031 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:01:51.031 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_16 2024-04-06T10:01:51.031 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T10:01:51.031 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n image_16 ']' 2024-04-06T10:01:51.032 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/image_16' 2024-04-06T10:01:51.032 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_16 2024-04-06T10:01:51.032 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T10:01:51.032 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T10:01:51.032 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T10:01:51.032 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T10:01:51.032 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T10:01:51.032 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T10:01:51.032 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T10:01:51.032 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T10:01:51.033 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T10:01:51.035 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T10:01:51.035 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_16 2024-04-06T10:01:51.035 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T10:01:51.035 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T10:01:51.035 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T10:01:51.035 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T10:01:51.035 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T10:01:51.035 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T10:01:51.035 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T10:01:51.035 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T10:01:51.036 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T10:01:51.036 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T10:01:51.036 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T10:01:51.036 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:01:51.036 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T10:01:51.036 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T10:01:51.037 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T10:01:51.037 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T10:01:51.037 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T10:01:51.037 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T10:01:51.037 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T10:01:51.037 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T10:01:51.037 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T10:01:51.038 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T10:01:51.038 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:01:51.038 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T10:01:51.038 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T10:01:51.038 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T10:01:51.038 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T10:01:51.038 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T10:01:51.038 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T10:01:51.038 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T10:01:51.062 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T10:01:51.062 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T10:01:51.062 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T10:01:51.062 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_16 2024-04-06T10:01:51.173 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:Invalid command: unused arguments: ['mirror/image_16'] 2024-04-06T10:01:51.173 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:rbd mirror flush : flush rbd mirror 2024-04-06T10:01:51.173 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:admin_socket: invalid command 2024-04-06T10:01:51.175 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T10:01:51.175 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/image_16 2024-04-06T10:01:51.175 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:1 2024-04-06T10:01:51.175 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T10:01:51.175 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T10:01:51.175 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-06T10:01:51.175 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:1 2024-04-06T10:01:51.175 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T10:01:51.175 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T10:01:51.175 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T10:01:51.176 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=1 2024-04-06T10:01:51.176 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-06T10:01:51.176 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T10:01:51.176 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:01:51.176 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=1 2024-04-06T10:01:51.176 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=1 2024-04-06T10:01:51.177 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-06T10:01:51.177 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:1 2024-04-06T10:01:51.177 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T10:01:51.177 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T10:01:51.177 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-06T10:01:51.178 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:1 2024-04-06T10:01:51.178 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T10:01:51.178 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T10:01:51.178 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:01:51.178 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=1 2024-04-06T10:01:51.178 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-06T10:01:51.178 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T10:01:51.178 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T10:01:51.178 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=1 2024-04-06T10:01:51.178 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=1 2024-04-06T10:01:51.178 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-06T10:01:51.200 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T10:01:51.201 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T10:01:51.201 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T10:01:51.201 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/image_16 2024-04-06T10:01:51.354 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T10:01:51.354 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror image_16 2024-04-06T10:01:51.355 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:01:51.355 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:01:51.355 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_16 2024-04-06T10:01:51.355 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror image_16 '' 2024-04-06T10:01:51.355 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:01:51.355 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:01:51.355 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_16 2024-04-06T10:01:51.355 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T10:01:51.355 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-image_16.status 2024-04-06T10:01:51.355 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-image_16.status 2024-04-06T10:01:51.355 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T10:01:51.356 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-image_16.status 2024-04-06T10:01:51.357 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_16 2024-04-06T10:01:51.357 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_16.status 2024-04-06T10:01:51.408 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 1 2024-04-06T10:01:51.408 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 1 2024-04-06T10:01:51.409 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T10:01:51.409 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=5, tag_tid=2, entry_tid=1], [object_number=4, tag_tid=2, entry_tid=0], [object_number=7, tag_tid=1, entry_tid=5759], [object_number=6, tag_tid=1, entry_tid=5758]]], state=connected] 2024-04-06T10:01:51.409 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=5, tag_tid=2, entry_tid=1], [object_number=4, tag_tid=2, entry_tid=0], [object_number=7, tag_tid=1, entry_tid=5759], [object_number=6, tag_tid=1, entry_tid=5758]]], state=connected] 2024-04-06T10:01:51.412 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_16.status 2024-04-06T10:01:51.413 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=5, tag_tid=2, entry_tid=1' 2024-04-06T10:01:51.415 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror image_16 2024-04-06T10:01:51.415 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:01:51.415 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:01:51.415 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_16 2024-04-06T10:01:51.415 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror image_16 '..*' 2024-04-06T10:01:51.415 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:01:51.415 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:01:51.415 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_16 2024-04-06T10:01:51.415 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T10:01:51.415 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-image_16.status 2024-04-06T10:01:51.415 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-image_16.status 2024-04-06T10:01:51.415 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T10:01:51.416 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-image_16.status 2024-04-06T10:01:51.416 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_16 2024-04-06T10:01:51.416 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_16.status 2024-04-06T10:01:51.453 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.0 with skip (0.3158205131673032) > weight (0.25) 2024-04-06T10:01:51.453 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.1 with skip (0.546054841087737) > weight (0.25) 2024-04-06T10:01:51.453 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.2 with skip (0.5515905352217514) > weight (0.25) 2024-04-06T10:01:51.453 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.3 with skip (0.7462796070573641) > weight (0.25) 2024-04-06T10:01:51.453 INFO:tasks.rbd_mirror_thrash:waiting for 76 secs before thrashing 2024-04-06T10:01:51.600 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 1 2024-04-06T10:01:51.600 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 1 2024-04-06T10:01:51.601 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T10:01:51.601 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=5, tag_tid=2, entry_tid=1], [object_number=4, tag_tid=2, entry_tid=0], [object_number=7, tag_tid=1, entry_tid=5759], [object_number=6, tag_tid=1, entry_tid=5758]]], state=connected] 2024-04-06T10:01:51.601 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=5, tag_tid=2, entry_tid=1], [object_number=4, tag_tid=2, entry_tid=0], [object_number=7, tag_tid=1, entry_tid=5759], [object_number=6, tag_tid=1, entry_tid=5758]]], state=connected] 2024-04-06T10:01:51.601 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_16.status 2024-04-06T10:01:51.603 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=5, tag_tid=2, entry_tid=1' 2024-04-06T10:01:51.603 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=5, tag_tid=2, entry_tid=1' -a 'object_number=5, tag_tid=2, entry_tid=1' = 'object_number=5, tag_tid=2, entry_tid=1' 2024-04-06T10:01:51.604 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T10:01:51.604 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_snap_present cluster1 mirror image_16 snap 2024-04-06T10:01:51.604 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T10:01:51.604 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:01:51.604 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_16 2024-04-06T10:01:51.604 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local snap_name=snap 2024-04-06T10:01:51.604 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local s 2024-04-06T10:01:51.604 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-04-06T10:01:51.604 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 1 2024-04-06T10:01:52.606 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 info mirror/image_16@snap 2024-04-06T10:01:52.669 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:rbd image 'image_16': 2024-04-06T10:01:52.669 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: size 128 MiB in 32 objects 2024-04-06T10:01:52.669 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: order 22 (4 MiB objects) 2024-04-06T10:01:52.670 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: snapshot_count: 1 2024-04-06T10:01:52.670 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: id: 10ac9d0d9c76 2024-04-06T10:01:52.670 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: block_name_prefix: rbd_data.10ac9d0d9c76 2024-04-06T10:01:52.670 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: format: 2 2024-04-06T10:01:52.670 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: features: layering, exclusive-lock, journaling 2024-04-06T10:01:52.670 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: op_features: 2024-04-06T10:01:52.670 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: flags: 2024-04-06T10:01:52.670 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: create_timestamp: Sat Apr 6 09:53:06 2024 2024-04-06T10:01:52.670 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: access_timestamp: Sat Apr 6 09:53:06 2024 2024-04-06T10:01:52.670 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: modify_timestamp: Sat Apr 6 09:53:06 2024 2024-04-06T10:01:52.670 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: protected: False 2024-04-06T10:01:52.670 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: journal: 10ac9d0d9c76 2024-04-06T10:01:52.670 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: mirroring state: enabled 2024-04-06T10:01:52.670 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: mirroring mode: journal 2024-04-06T10:01:52.671 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: mirroring global id: 4a6be008-a799-43aa-83bb-16d78180d7e2 2024-04-06T10:01:52.671 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: mirroring primary: false 2024-04-06T10:01:52.674 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T10:01:52.675 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ compare_image_snaps mirror image_16 snap 2024-04-06T10:01:52.675 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:01:52.675 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_16 2024-04-06T10:01:52.675 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local snap_name=snap 2024-04-06T10:01:52.675 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local ret=0 2024-04-06T10:01:52.675 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local rmt_export=/tmp/tmp.BhW6xKmROU/cluster2-mirror-image_16.export 2024-04-06T10:01:52.675 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loc_export=/tmp/tmp.BhW6xKmROU/cluster1-mirror-image_16.export 2024-04-06T10:01:52.675 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rm -f /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_16.export /tmp/tmp.BhW6xKmROU/cluster1-mirror-image_16.export 2024-04-06T10:01:52.675 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster2 -p mirror export image_16@snap /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_16.export 2024-04-06T10:01:52.848 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-04-06T10:01:52.857 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 -p mirror export image_16@snap /tmp/tmp.BhW6xKmROU/cluster1-mirror-image_16.export 2024-04-06T10:01:53.266 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-04-06T10:01:53.281 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmp /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_16.export /tmp/tmp.BhW6xKmROU/cluster1-mirror-image_16.export 2024-04-06T10:01:53.363 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rm -f /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_16.export /tmp/tmp.BhW6xKmROU/cluster1-mirror-image_16.export 2024-04-06T10:01:53.404 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T10:01:53.404 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-06T10:01:53.404 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ image=image_17 2024-04-06T10:01:53.404 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ create_snap cluster2 mirror image_17 snap 2024-04-06T10:01:53.405 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-04-06T10:01:53.405 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:01:53.405 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_17 2024-04-06T10:01:53.405 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local snap_name=snap 2024-04-06T10:01:53.405 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster2 -p mirror snap create image_17@snap --debug-rbd=20 --debug-journaler=20 2024-04-06T10:01:53.889 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_image_replay_started cluster1 mirror image_17 2024-04-06T10:01:53.889 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T10:01:53.889 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:01:53.889 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_17 2024-04-06T10:01:53.889 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_image_replay_state cluster1 mirror image_17 started 2024-04-06T10:01:53.889 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T10:01:53.889 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:01:53.889 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_17 2024-04-06T10:01:53.889 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state=started 2024-04-06T10:01:53.889 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local s 2024-04-06T10:01:53.889 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T10:01:53.890 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 1 2024-04-06T10:01:54.890 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test_image_replay_state cluster1 mirror image_17 started 2024-04-06T10:01:54.890 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T10:01:54.891 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:01:54.891 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_17 2024-04-06T10:01:54.891 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local test_state=started 2024-04-06T10:01:54.891 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local status_result 2024-04-06T10:01:54.891 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local current_state=stopped 2024-04-06T10:01:54.892 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_17 2024-04-06T10:01:54.892 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -i state 2024-04-06T10:01:54.892 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_instance=cluster1 2024-04-06T10:01:54.892 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ shift 2024-04-06T10:01:54.893 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T10:01:54.893 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance=cluster1 2024-04-06T10:01:54.893 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local loop_instance 2024-04-06T10:01:54.893 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T10:01:54.893 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sleep 0 2024-04-06T10:01:54.893 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-04-06T10:01:54.893 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ seq 0 3 2024-04-06T10:01:54.895 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T10:01:54.895 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_17 2024-04-06T10:01:54.895 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1:0 2024-04-06T10:01:54.895 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ shift 2024-04-06T10:01:54.895 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T10:01:54.895 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-04-06T10:01:54.895 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T10:01:54.895 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=cluster 2024-04-06T10:01:54.895 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T10:01:54.895 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:01:54.896 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T10:01:54.896 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T10:01:54.896 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval cluster=cluster1 2024-04-06T10:01:54.896 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ cluster=cluster1 2024-04-06T10:01:54.896 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T10:01:54.896 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T10:01:54.897 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-04-06T10:01:54.897 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local local_cluster=cluster1:0 2024-04-06T10:01:54.897 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local cluster=cluster1 2024-04-06T10:01:54.897 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local instance 2024-04-06T10:01:54.897 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T10:01:54.898 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local val=cluster1:0 2024-04-06T10:01:54.898 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local cluster_var_name=local_cluster 2024-04-06T10:01:54.898 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local instance_var_name=instance 2024-04-06T10:01:54.898 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ cluster=cluster1 2024-04-06T10:01:54.898 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T10:01:54.898 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ '[' 0 = cluster1:0 ']' 2024-04-06T10:01:54.898 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ eval local_cluster=cluster1 2024-04-06T10:01:54.898 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++++ local_cluster=cluster1 2024-04-06T10:01:54.898 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ eval instance=0 2024-04-06T10:01:54.899 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++++ instance=0 2024-04-06T10:01:54.899 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T10:01:54.918 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T10:01:54.918 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T10:01:54.919 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T10:01:54.919 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_17 2024-04-06T10:01:55.022 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:Invalid command: unused arguments: ['mirror/image_17'] 2024-04-06T10:01:55.022 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:rbd mirror status : get status for rbd mirror 2024-04-06T10:01:55.022 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:admin_socket: invalid command 2024-04-06T10:01:55.024 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T10:01:55.024 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_17 2024-04-06T10:01:55.024 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1:1 2024-04-06T10:01:55.024 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ shift 2024-04-06T10:01:55.024 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T10:01:55.024 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-04-06T10:01:55.024 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:1 2024-04-06T10:01:55.024 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=cluster 2024-04-06T10:01:55.024 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T10:01:55.024 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:01:55.024 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=1 2024-04-06T10:01:55.025 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-06T10:01:55.025 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval cluster=cluster1 2024-04-06T10:01:55.025 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ cluster=cluster1 2024-04-06T10:01:55.025 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=1 2024-04-06T10:01:55.025 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=1 2024-04-06T10:01:55.026 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-04-06T10:01:55.026 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local local_cluster=cluster1:1 2024-04-06T10:01:55.026 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local cluster=cluster1 2024-04-06T10:01:55.026 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local instance 2024-04-06T10:01:55.026 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-06T10:01:55.026 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local val=cluster1:1 2024-04-06T10:01:55.027 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local cluster_var_name=local_cluster 2024-04-06T10:01:55.027 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local instance_var_name=instance 2024-04-06T10:01:55.027 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ cluster=cluster1 2024-04-06T10:01:55.027 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=1 2024-04-06T10:01:55.027 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ '[' 1 = cluster1:1 ']' 2024-04-06T10:01:55.027 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ eval local_cluster=cluster1 2024-04-06T10:01:55.027 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++++ local_cluster=cluster1 2024-04-06T10:01:55.027 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ eval instance=1 2024-04-06T10:01:55.027 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++++ instance=1 2024-04-06T10:01:55.027 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-06T10:01:55.051 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T10:01:55.051 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T10:01:55.051 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T10:01:55.051 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_17 2024-04-06T10:01:55.201 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ return 0 2024-04-06T10:01:55.202 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ status_result=' "state": "Replaying"' 2024-04-06T10:01:55.202 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ echo ' "state": "Replaying"' 2024-04-06T10:01:55.202 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep -i Replaying 2024-04-06T10:01:55.205 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: "state": "Replaying" 2024-04-06T10:01:55.207 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ current_state=started 2024-04-06T10:01:55.207 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test started = started 2024-04-06T10:01:55.207 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T10:01:55.207 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_17 2024-04-06T10:01:55.207 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local local_cluster=cluster1 2024-04-06T10:01:55.207 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-04-06T10:01:55.207 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:01:55.207 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_17 2024-04-06T10:01:55.207 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' journal = journal ']' 2024-04-06T10:01:55.207 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_journal_replay_complete cluster1 cluster2 mirror image_17 2024-04-06T10:01:55.207 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local local_cluster=cluster1 2024-04-06T10:01:55.207 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-04-06T10:01:55.207 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:01:55.208 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_17 2024-04-06T10:01:55.208 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local s master_pos mirror_pos last_mirror_pos 2024-04-06T10:01:55.208 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local master_tag master_entry mirror_tag mirror_entry 2024-04-06T10:01:55.208 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T10:01:55.208 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T10:01:55.208 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T10:01:55.408 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror image_17 2024-04-06T10:01:55.408 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T10:01:55.408 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:01:55.408 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_17 2024-04-06T10:01:55.408 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T10:01:55.408 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n image_17 ']' 2024-04-06T10:01:55.408 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/image_17' 2024-04-06T10:01:55.409 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_17 2024-04-06T10:01:55.409 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T10:01:55.409 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T10:01:55.409 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T10:01:55.409 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T10:01:55.409 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T10:01:55.409 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T10:01:55.409 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T10:01:55.409 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T10:01:55.413 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T10:01:55.413 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T10:01:55.413 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_17 2024-04-06T10:01:55.413 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T10:01:55.413 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T10:01:55.413 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T10:01:55.413 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T10:01:55.413 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T10:01:55.414 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T10:01:55.414 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T10:01:55.414 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T10:01:55.414 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T10:01:55.414 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T10:01:55.414 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T10:01:55.414 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:01:55.414 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T10:01:55.414 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T10:01:55.414 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T10:01:55.414 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T10:01:55.414 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T10:01:55.414 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T10:01:55.414 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T10:01:55.415 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T10:01:55.415 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T10:01:55.415 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T10:01:55.415 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:01:55.415 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T10:01:55.415 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T10:01:55.415 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T10:01:55.415 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T10:01:55.415 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T10:01:55.415 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T10:01:55.415 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T10:01:55.437 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T10:01:55.437 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T10:01:55.437 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T10:01:55.438 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_17 2024-04-06T10:01:55.542 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:Invalid command: unused arguments: ['mirror/image_17'] 2024-04-06T10:01:55.543 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:rbd mirror flush : flush rbd mirror 2024-04-06T10:01:55.543 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:admin_socket: invalid command 2024-04-06T10:01:55.545 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T10:01:55.545 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/image_17 2024-04-06T10:01:55.545 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:1 2024-04-06T10:01:55.545 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T10:01:55.545 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T10:01:55.545 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-06T10:01:55.545 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:1 2024-04-06T10:01:55.545 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T10:01:55.545 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T10:01:55.545 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T10:01:55.545 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=1 2024-04-06T10:01:55.546 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-06T10:01:55.546 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T10:01:55.546 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:01:55.546 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=1 2024-04-06T10:01:55.546 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=1 2024-04-06T10:01:55.547 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-06T10:01:55.547 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:1 2024-04-06T10:01:55.547 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T10:01:55.547 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T10:01:55.547 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-06T10:01:55.547 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:1 2024-04-06T10:01:55.548 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T10:01:55.548 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T10:01:55.548 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:01:55.548 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=1 2024-04-06T10:01:55.548 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-06T10:01:55.548 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T10:01:55.548 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T10:01:55.548 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=1 2024-04-06T10:01:55.548 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=1 2024-04-06T10:01:55.548 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-06T10:01:55.571 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T10:01:55.571 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T10:01:55.571 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T10:01:55.572 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/image_17 2024-04-06T10:01:55.714 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T10:01:55.715 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror image_17 2024-04-06T10:01:55.715 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:01:55.716 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:01:55.716 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_17 2024-04-06T10:01:55.716 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror image_17 '' 2024-04-06T10:01:55.716 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:01:55.716 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:01:55.716 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_17 2024-04-06T10:01:55.716 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T10:01:55.716 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-image_17.status 2024-04-06T10:01:55.716 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-image_17.status 2024-04-06T10:01:55.717 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T10:01:55.718 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-image_17.status 2024-04-06T10:01:55.718 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_17 2024-04-06T10:01:55.718 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_17.status 2024-04-06T10:01:55.765 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 1 2024-04-06T10:01:55.777 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 1 2024-04-06T10:01:55.777 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T10:01:55.777 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=5, tag_tid=3, entry_tid=1], [object_number=4, tag_tid=3, entry_tid=0], [object_number=7, tag_tid=2, entry_tid=3], [object_number=6, tag_tid=2, entry_tid=2]]], state=connected] 2024-04-06T10:01:55.777 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=7, tag_tid=2, entry_tid=3], [object_number=6, tag_tid=2, entry_tid=2], [object_number=5, tag_tid=2, entry_tid=1], [object_number=4, tag_tid=2, entry_tid=0]]], state=connected] 2024-04-06T10:01:55.779 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_17.status 2024-04-06T10:01:55.779 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=5, tag_tid=3, entry_tid=1' 2024-04-06T10:01:55.779 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror image_17 2024-04-06T10:01:55.779 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:01:55.779 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:01:55.779 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_17 2024-04-06T10:01:55.779 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror image_17 '..*' 2024-04-06T10:01:55.780 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:01:55.780 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:01:55.780 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_17 2024-04-06T10:01:55.780 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T10:01:55.780 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-image_17.status 2024-04-06T10:01:55.780 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-image_17.status 2024-04-06T10:01:55.780 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T10:01:55.780 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-image_17.status 2024-04-06T10:01:55.780 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_17 2024-04-06T10:01:55.780 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_17.status 2024-04-06T10:01:55.821 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 1 2024-04-06T10:01:55.821 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 1 2024-04-06T10:01:55.821 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T10:01:55.821 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=5, tag_tid=3, entry_tid=1], [object_number=4, tag_tid=3, entry_tid=0], [object_number=7, tag_tid=2, entry_tid=3], [object_number=6, tag_tid=2, entry_tid=2]]], state=connected] 2024-04-06T10:01:55.821 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=7, tag_tid=2, entry_tid=3], [object_number=6, tag_tid=2, entry_tid=2], [object_number=5, tag_tid=2, entry_tid=1], [object_number=4, tag_tid=2, entry_tid=0]]], state=connected] 2024-04-06T10:01:55.824 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_17.status 2024-04-06T10:01:55.827 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=7, tag_tid=2, entry_tid=3' 2024-04-06T10:01:55.827 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=5, tag_tid=3, entry_tid=1' -a 'object_number=5, tag_tid=3, entry_tid=1' = 'object_number=7, tag_tid=2, entry_tid=3' 2024-04-06T10:01:55.827 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=7, tag_tid=2, entry_tid=3' '!=' '' 2024-04-06T10:01:55.827 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T10:01:55.827 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=7, tag_tid=2, entry_tid=3' = '' 2024-04-06T10:01:55.827 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=7, tag_tid=2, entry_tid=3' 2024-04-06T10:01:55.827 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=5, tag_tid=3, entry_tid=1' 2024-04-06T10:01:55.827 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T10:01:55.827 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T10:01:55.828 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=3 2024-04-06T10:01:55.828 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=7, tag_tid=2, entry_tid=3' 2024-04-06T10:01:55.828 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T10:01:55.828 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T10:01:55.829 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=2 2024-04-06T10:01:55.830 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=5, tag_tid=3, entry_tid=1' 2024-04-06T10:01:55.830 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T10:01:55.830 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T10:01:55.831 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T10:01:55.832 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=7, tag_tid=2, entry_tid=3' 2024-04-06T10:01:55.832 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T10:01:55.832 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T10:01:55.833 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=3 2024-04-06T10:01:55.833 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 3 = 2 -a 1 -le 3 2024-04-06T10:01:55.833 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T10:01:55.833 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T10:01:55.833 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T10:01:56.035 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror image_17 2024-04-06T10:01:56.035 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T10:01:56.035 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:01:56.035 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_17 2024-04-06T10:01:56.035 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T10:01:56.035 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n image_17 ']' 2024-04-06T10:01:56.036 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/image_17' 2024-04-06T10:01:56.036 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_17 2024-04-06T10:01:56.036 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T10:01:56.036 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T10:01:56.036 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T10:01:56.036 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T10:01:56.036 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T10:01:56.036 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T10:01:56.036 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T10:01:56.037 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T10:01:56.037 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T10:01:56.040 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T10:01:56.040 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_17 2024-04-06T10:01:56.040 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T10:01:56.040 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T10:01:56.040 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T10:01:56.040 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T10:01:56.040 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T10:01:56.040 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T10:01:56.041 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T10:01:56.041 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T10:01:56.041 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T10:01:56.041 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T10:01:56.041 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T10:01:56.041 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:01:56.041 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T10:01:56.041 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T10:01:56.041 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T10:01:56.041 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T10:01:56.041 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T10:01:56.041 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T10:01:56.041 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T10:01:56.041 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T10:01:56.041 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T10:01:56.041 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T10:01:56.042 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:01:56.042 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T10:01:56.042 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T10:01:56.042 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T10:01:56.042 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T10:01:56.042 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T10:01:56.042 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T10:01:56.042 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T10:01:56.064 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T10:01:56.065 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T10:01:56.065 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T10:01:56.065 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_17 2024-04-06T10:01:56.171 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:Invalid command: unused arguments: ['mirror/image_17'] 2024-04-06T10:01:56.171 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:rbd mirror flush : flush rbd mirror 2024-04-06T10:01:56.171 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:admin_socket: invalid command 2024-04-06T10:01:56.173 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T10:01:56.173 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/image_17 2024-04-06T10:01:56.174 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:1 2024-04-06T10:01:56.174 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T10:01:56.174 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T10:01:56.174 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-06T10:01:56.174 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:1 2024-04-06T10:01:56.174 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T10:01:56.174 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T10:01:56.174 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T10:01:56.175 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=1 2024-04-06T10:01:56.175 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-06T10:01:56.175 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T10:01:56.175 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:01:56.175 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=1 2024-04-06T10:01:56.175 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=1 2024-04-06T10:01:56.177 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-06T10:01:56.177 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:1 2024-04-06T10:01:56.177 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T10:01:56.177 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T10:01:56.177 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-06T10:01:56.177 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:1 2024-04-06T10:01:56.178 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T10:01:56.178 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T10:01:56.178 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:01:56.178 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=1 2024-04-06T10:01:56.178 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-06T10:01:56.178 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T10:01:56.178 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T10:01:56.178 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=1 2024-04-06T10:01:56.179 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=1 2024-04-06T10:01:56.179 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-06T10:01:56.195 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T10:01:56.195 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T10:01:56.195 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T10:01:56.195 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/image_17 2024-04-06T10:01:56.337 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T10:01:56.338 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror image_17 2024-04-06T10:01:56.339 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:01:56.339 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:01:56.339 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_17 2024-04-06T10:01:56.339 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror image_17 '' 2024-04-06T10:01:56.339 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:01:56.339 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:01:56.339 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_17 2024-04-06T10:01:56.339 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T10:01:56.339 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-image_17.status 2024-04-06T10:01:56.340 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-image_17.status 2024-04-06T10:01:56.340 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T10:01:56.341 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-image_17.status 2024-04-06T10:01:56.342 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_17 2024-04-06T10:01:56.342 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_17.status 2024-04-06T10:01:56.393 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 1 2024-04-06T10:01:56.393 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 1 2024-04-06T10:01:56.393 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T10:01:56.393 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=5, tag_tid=3, entry_tid=1], [object_number=4, tag_tid=3, entry_tid=0], [object_number=7, tag_tid=2, entry_tid=3], [object_number=6, tag_tid=2, entry_tid=2]]], state=connected] 2024-04-06T10:01:56.393 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=7, tag_tid=2, entry_tid=3], [object_number=6, tag_tid=2, entry_tid=2], [object_number=5, tag_tid=2, entry_tid=1], [object_number=4, tag_tid=2, entry_tid=0]]], state=connected] 2024-04-06T10:01:56.397 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_17.status 2024-04-06T10:01:56.398 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=5, tag_tid=3, entry_tid=1' 2024-04-06T10:01:56.399 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror image_17 2024-04-06T10:01:56.400 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:01:56.400 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:01:56.400 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_17 2024-04-06T10:01:56.400 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror image_17 '..*' 2024-04-06T10:01:56.400 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:01:56.400 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:01:56.400 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_17 2024-04-06T10:01:56.400 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T10:01:56.400 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-image_17.status 2024-04-06T10:01:56.400 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-image_17.status 2024-04-06T10:01:56.400 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T10:01:56.401 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-image_17.status 2024-04-06T10:01:56.401 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_17 2024-04-06T10:01:56.401 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_17.status 2024-04-06T10:01:56.447 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 1 2024-04-06T10:01:56.447 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 1 2024-04-06T10:01:56.447 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T10:01:56.447 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=5, tag_tid=3, entry_tid=1], [object_number=4, tag_tid=3, entry_tid=0], [object_number=7, tag_tid=2, entry_tid=3], [object_number=6, tag_tid=2, entry_tid=2]]], state=connected] 2024-04-06T10:01:56.447 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=7, tag_tid=2, entry_tid=3], [object_number=6, tag_tid=2, entry_tid=2], [object_number=5, tag_tid=2, entry_tid=1], [object_number=4, tag_tid=2, entry_tid=0]]], state=connected] 2024-04-06T10:01:56.450 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_17.status 2024-04-06T10:01:56.452 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=7, tag_tid=2, entry_tid=3' 2024-04-06T10:01:56.452 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=5, tag_tid=3, entry_tid=1' -a 'object_number=5, tag_tid=3, entry_tid=1' = 'object_number=7, tag_tid=2, entry_tid=3' 2024-04-06T10:01:56.452 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=7, tag_tid=2, entry_tid=3' '!=' 'object_number=7, tag_tid=2, entry_tid=3' 2024-04-06T10:01:56.452 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T10:01:56.452 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.4 2024-04-06T10:01:56.854 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror image_17 2024-04-06T10:01:56.854 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T10:01:56.854 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:01:56.854 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_17 2024-04-06T10:01:56.854 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T10:01:56.854 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n image_17 ']' 2024-04-06T10:01:56.855 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/image_17' 2024-04-06T10:01:56.855 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_17 2024-04-06T10:01:56.855 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T10:01:56.855 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T10:01:56.855 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T10:01:56.855 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T10:01:56.855 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T10:01:56.855 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T10:01:56.855 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T10:01:56.856 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T10:01:56.856 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T10:01:56.861 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T10:01:56.861 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_17 2024-04-06T10:01:56.861 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T10:01:56.861 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T10:01:56.861 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T10:01:56.861 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T10:01:56.862 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T10:01:56.862 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T10:01:56.862 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T10:01:56.862 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T10:01:56.862 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T10:01:56.862 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T10:01:56.863 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T10:01:56.863 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:01:56.863 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T10:01:56.863 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T10:01:56.864 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T10:01:56.864 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T10:01:56.864 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T10:01:56.864 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T10:01:56.864 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T10:01:56.864 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T10:01:56.865 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T10:01:56.865 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T10:01:56.865 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:01:56.865 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T10:01:56.865 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T10:01:56.865 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T10:01:56.866 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T10:01:56.866 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T10:01:56.866 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T10:01:56.866 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T10:01:56.885 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T10:01:56.885 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T10:01:56.885 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T10:01:56.885 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_17 2024-04-06T10:01:56.992 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:Invalid command: unused arguments: ['mirror/image_17'] 2024-04-06T10:01:56.992 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:rbd mirror flush : flush rbd mirror 2024-04-06T10:01:56.992 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:admin_socket: invalid command 2024-04-06T10:01:56.995 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T10:01:56.996 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/image_17 2024-04-06T10:01:56.996 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:1 2024-04-06T10:01:56.996 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T10:01:56.996 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T10:01:56.996 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-06T10:01:56.996 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:1 2024-04-06T10:01:56.996 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T10:01:56.996 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T10:01:56.996 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T10:01:56.996 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=1 2024-04-06T10:01:56.996 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-06T10:01:56.996 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T10:01:56.996 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:01:56.997 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=1 2024-04-06T10:01:56.997 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=1 2024-04-06T10:01:56.997 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-06T10:01:56.997 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:1 2024-04-06T10:01:56.997 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T10:01:56.997 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T10:01:56.997 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-06T10:01:56.997 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:1 2024-04-06T10:01:56.997 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T10:01:56.997 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T10:01:56.997 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:01:56.997 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=1 2024-04-06T10:01:56.997 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-06T10:01:56.997 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T10:01:56.998 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T10:01:56.998 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=1 2024-04-06T10:01:56.998 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=1 2024-04-06T10:01:56.998 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-06T10:01:57.019 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T10:01:57.020 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T10:01:57.020 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T10:01:57.020 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/image_17 2024-04-06T10:01:57.154 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T10:01:57.155 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror image_17 2024-04-06T10:01:57.155 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:01:57.155 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:01:57.155 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_17 2024-04-06T10:01:57.155 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror image_17 '' 2024-04-06T10:01:57.155 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:01:57.155 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:01:57.156 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_17 2024-04-06T10:01:57.156 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T10:01:57.156 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-image_17.status 2024-04-06T10:01:57.156 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-image_17.status 2024-04-06T10:01:57.156 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T10:01:57.157 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-image_17.status 2024-04-06T10:01:57.158 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_17 2024-04-06T10:01:57.158 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_17.status 2024-04-06T10:01:57.204 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 1 2024-04-06T10:01:57.204 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 1 2024-04-06T10:01:57.204 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T10:01:57.204 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=5, tag_tid=3, entry_tid=1], [object_number=4, tag_tid=3, entry_tid=0], [object_number=7, tag_tid=2, entry_tid=3], [object_number=6, tag_tid=2, entry_tid=2]]], state=connected] 2024-04-06T10:01:57.204 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=7, tag_tid=2, entry_tid=3], [object_number=6, tag_tid=2, entry_tid=2], [object_number=5, tag_tid=2, entry_tid=1], [object_number=4, tag_tid=2, entry_tid=0]]], state=connected] 2024-04-06T10:01:57.207 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_17.status 2024-04-06T10:01:57.209 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=5, tag_tid=3, entry_tid=1' 2024-04-06T10:01:57.210 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror image_17 2024-04-06T10:01:57.210 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:01:57.210 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:01:57.211 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_17 2024-04-06T10:01:57.211 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror image_17 '..*' 2024-04-06T10:01:57.211 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:01:57.211 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:01:57.211 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_17 2024-04-06T10:01:57.211 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T10:01:57.211 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-image_17.status 2024-04-06T10:01:57.211 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-image_17.status 2024-04-06T10:01:57.211 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T10:01:57.212 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-image_17.status 2024-04-06T10:01:57.212 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_17 2024-04-06T10:01:57.212 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_17.status 2024-04-06T10:01:57.256 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 1 2024-04-06T10:01:57.257 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 1 2024-04-06T10:01:57.257 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T10:01:57.257 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=5, tag_tid=3, entry_tid=1], [object_number=4, tag_tid=3, entry_tid=0], [object_number=7, tag_tid=2, entry_tid=3], [object_number=6, tag_tid=2, entry_tid=2]]], state=connected] 2024-04-06T10:01:57.257 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=7, tag_tid=2, entry_tid=3], [object_number=6, tag_tid=2, entry_tid=2], [object_number=5, tag_tid=2, entry_tid=1], [object_number=4, tag_tid=2, entry_tid=0]]], state=connected] 2024-04-06T10:01:57.260 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_17.status 2024-04-06T10:01:57.262 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=7, tag_tid=2, entry_tid=3' 2024-04-06T10:01:57.262 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=5, tag_tid=3, entry_tid=1' -a 'object_number=5, tag_tid=3, entry_tid=1' = 'object_number=7, tag_tid=2, entry_tid=3' 2024-04-06T10:01:57.262 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=7, tag_tid=2, entry_tid=3' '!=' 'object_number=7, tag_tid=2, entry_tid=3' 2024-04-06T10:01:57.262 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T10:01:57.262 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.8 2024-04-06T10:01:58.064 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror image_17 2024-04-06T10:01:58.065 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T10:01:58.065 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:01:58.065 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_17 2024-04-06T10:01:58.065 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T10:01:58.065 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n image_17 ']' 2024-04-06T10:01:58.065 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/image_17' 2024-04-06T10:01:58.065 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_17 2024-04-06T10:01:58.065 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T10:01:58.065 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T10:01:58.065 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T10:01:58.065 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T10:01:58.065 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T10:01:58.065 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T10:01:58.065 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T10:01:58.066 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T10:01:58.066 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T10:01:58.069 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T10:01:58.069 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_17 2024-04-06T10:01:58.069 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T10:01:58.069 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T10:01:58.069 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T10:01:58.069 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T10:01:58.069 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T10:01:58.069 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T10:01:58.070 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T10:01:58.070 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T10:01:58.070 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T10:01:58.070 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T10:01:58.070 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T10:01:58.070 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:01:58.070 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T10:01:58.070 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T10:01:58.070 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T10:01:58.070 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T10:01:58.070 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T10:01:58.070 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T10:01:58.070 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T10:01:58.070 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T10:01:58.071 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T10:01:58.071 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T10:01:58.071 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:01:58.071 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T10:01:58.071 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T10:01:58.071 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T10:01:58.071 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T10:01:58.071 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T10:01:58.071 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T10:01:58.071 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T10:01:58.096 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T10:01:58.096 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T10:01:58.096 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T10:01:58.096 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_17 2024-04-06T10:01:58.203 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:Invalid command: unused arguments: ['mirror/image_17'] 2024-04-06T10:01:58.203 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:rbd mirror flush : flush rbd mirror 2024-04-06T10:01:58.203 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:admin_socket: invalid command 2024-04-06T10:01:58.207 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T10:01:58.207 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/image_17 2024-04-06T10:01:58.207 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:1 2024-04-06T10:01:58.207 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T10:01:58.207 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T10:01:58.207 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-06T10:01:58.207 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:1 2024-04-06T10:01:58.207 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T10:01:58.207 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T10:01:58.207 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T10:01:58.207 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=1 2024-04-06T10:01:58.207 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-06T10:01:58.208 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T10:01:58.208 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:01:58.208 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=1 2024-04-06T10:01:58.208 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=1 2024-04-06T10:01:58.208 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-06T10:01:58.208 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:1 2024-04-06T10:01:58.208 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T10:01:58.208 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T10:01:58.208 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-06T10:01:58.208 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:1 2024-04-06T10:01:58.208 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T10:01:58.208 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T10:01:58.208 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:01:58.208 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=1 2024-04-06T10:01:58.208 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-06T10:01:58.209 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T10:01:58.209 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T10:01:58.209 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=1 2024-04-06T10:01:58.209 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=1 2024-04-06T10:01:58.209 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-06T10:01:58.232 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T10:01:58.232 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T10:01:58.232 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T10:01:58.232 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/image_17 2024-04-06T10:01:58.374 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T10:01:58.375 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror image_17 2024-04-06T10:01:58.375 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:01:58.375 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:01:58.375 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_17 2024-04-06T10:01:58.375 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror image_17 '' 2024-04-06T10:01:58.375 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:01:58.376 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:01:58.376 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_17 2024-04-06T10:01:58.376 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T10:01:58.376 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-image_17.status 2024-04-06T10:01:58.376 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-image_17.status 2024-04-06T10:01:58.377 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T10:01:58.378 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-image_17.status 2024-04-06T10:01:58.378 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_17 2024-04-06T10:01:58.378 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_17.status 2024-04-06T10:01:58.423 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 1 2024-04-06T10:01:58.424 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 1 2024-04-06T10:01:58.424 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T10:01:58.424 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=5, tag_tid=3, entry_tid=1], [object_number=4, tag_tid=3, entry_tid=0], [object_number=7, tag_tid=2, entry_tid=3], [object_number=6, tag_tid=2, entry_tid=2]]], state=connected] 2024-04-06T10:01:58.424 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=7, tag_tid=2, entry_tid=3], [object_number=6, tag_tid=2, entry_tid=2], [object_number=5, tag_tid=2, entry_tid=1], [object_number=4, tag_tid=2, entry_tid=0]]], state=connected] 2024-04-06T10:01:58.426 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_17.status 2024-04-06T10:01:58.428 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=5, tag_tid=3, entry_tid=1' 2024-04-06T10:01:58.429 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror image_17 2024-04-06T10:01:58.429 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:01:58.429 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:01:58.429 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_17 2024-04-06T10:01:58.429 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror image_17 '..*' 2024-04-06T10:01:58.429 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:01:58.429 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:01:58.429 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_17 2024-04-06T10:01:58.429 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T10:01:58.430 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-image_17.status 2024-04-06T10:01:58.430 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-image_17.status 2024-04-06T10:01:58.430 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T10:01:58.430 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-image_17.status 2024-04-06T10:01:58.430 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_17 2024-04-06T10:01:58.430 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_17.status 2024-04-06T10:01:58.475 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 1 2024-04-06T10:01:58.476 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 1 2024-04-06T10:01:58.476 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T10:01:58.476 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=5, tag_tid=3, entry_tid=1], [object_number=4, tag_tid=3, entry_tid=0], [object_number=7, tag_tid=2, entry_tid=3], [object_number=6, tag_tid=2, entry_tid=2]]], state=connected] 2024-04-06T10:01:58.476 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=7, tag_tid=2, entry_tid=3], [object_number=6, tag_tid=2, entry_tid=2], [object_number=5, tag_tid=2, entry_tid=1], [object_number=4, tag_tid=2, entry_tid=0]]], state=connected] 2024-04-06T10:01:58.479 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_17.status 2024-04-06T10:01:58.481 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=7, tag_tid=2, entry_tid=3' 2024-04-06T10:01:58.481 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=5, tag_tid=3, entry_tid=1' -a 'object_number=5, tag_tid=3, entry_tid=1' = 'object_number=7, tag_tid=2, entry_tid=3' 2024-04-06T10:01:58.481 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=7, tag_tid=2, entry_tid=3' '!=' 'object_number=7, tag_tid=2, entry_tid=3' 2024-04-06T10:01:58.481 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T10:01:58.481 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 1.6 2024-04-06T10:02:00.084 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror image_17 2024-04-06T10:02:00.084 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T10:02:00.085 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:02:00.085 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_17 2024-04-06T10:02:00.085 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T10:02:00.085 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n image_17 ']' 2024-04-06T10:02:00.085 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/image_17' 2024-04-06T10:02:00.085 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_17 2024-04-06T10:02:00.085 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T10:02:00.085 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T10:02:00.085 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T10:02:00.085 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T10:02:00.085 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T10:02:00.085 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T10:02:00.085 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T10:02:00.085 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T10:02:00.089 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T10:02:00.089 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T10:02:00.089 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_17 2024-04-06T10:02:00.089 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T10:02:00.090 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T10:02:00.090 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T10:02:00.090 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T10:02:00.090 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T10:02:00.090 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T10:02:00.090 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T10:02:00.090 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T10:02:00.090 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T10:02:00.090 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T10:02:00.090 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T10:02:00.090 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:02:00.090 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T10:02:00.091 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T10:02:00.091 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T10:02:00.091 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T10:02:00.091 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T10:02:00.091 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T10:02:00.091 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T10:02:00.091 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T10:02:00.091 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T10:02:00.091 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T10:02:00.091 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:02:00.091 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T10:02:00.091 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T10:02:00.092 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T10:02:00.092 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T10:02:00.092 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T10:02:00.092 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T10:02:00.092 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T10:02:00.115 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T10:02:00.116 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T10:02:00.116 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T10:02:00.116 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_17 2024-04-06T10:02:00.230 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:Invalid command: unused arguments: ['mirror/image_17'] 2024-04-06T10:02:00.230 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:rbd mirror flush : flush rbd mirror 2024-04-06T10:02:00.230 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:admin_socket: invalid command 2024-04-06T10:02:00.233 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T10:02:00.234 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/image_17 2024-04-06T10:02:00.234 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:1 2024-04-06T10:02:00.234 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T10:02:00.234 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T10:02:00.234 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-06T10:02:00.234 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:1 2024-04-06T10:02:00.234 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T10:02:00.234 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T10:02:00.234 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T10:02:00.234 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=1 2024-04-06T10:02:00.234 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-06T10:02:00.234 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T10:02:00.234 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:02:00.234 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=1 2024-04-06T10:02:00.235 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=1 2024-04-06T10:02:00.235 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-06T10:02:00.235 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:1 2024-04-06T10:02:00.235 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T10:02:00.235 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T10:02:00.235 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-06T10:02:00.235 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:1 2024-04-06T10:02:00.235 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T10:02:00.235 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T10:02:00.235 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:02:00.235 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=1 2024-04-06T10:02:00.235 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-06T10:02:00.235 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T10:02:00.236 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T10:02:00.236 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=1 2024-04-06T10:02:00.236 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=1 2024-04-06T10:02:00.236 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-06T10:02:00.256 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T10:02:00.257 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T10:02:00.257 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T10:02:00.257 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/image_17 2024-04-06T10:02:00.402 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T10:02:00.404 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror image_17 2024-04-06T10:02:00.404 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:02:00.404 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:02:00.405 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_17 2024-04-06T10:02:00.405 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror image_17 '' 2024-04-06T10:02:00.405 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:02:00.405 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:02:00.405 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_17 2024-04-06T10:02:00.405 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T10:02:00.405 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-image_17.status 2024-04-06T10:02:00.405 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-image_17.status 2024-04-06T10:02:00.406 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T10:02:00.406 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-image_17.status 2024-04-06T10:02:00.407 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_17 2024-04-06T10:02:00.407 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_17.status 2024-04-06T10:02:00.456 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 1 2024-04-06T10:02:00.456 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 1 2024-04-06T10:02:00.456 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T10:02:00.456 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=5, tag_tid=3, entry_tid=1], [object_number=4, tag_tid=3, entry_tid=0], [object_number=7, tag_tid=2, entry_tid=3], [object_number=6, tag_tid=2, entry_tid=2]]], state=connected] 2024-04-06T10:02:00.456 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=5, tag_tid=3, entry_tid=1], [object_number=4, tag_tid=3, entry_tid=0], [object_number=7, tag_tid=2, entry_tid=3], [object_number=6, tag_tid=2, entry_tid=2]]], state=connected] 2024-04-06T10:02:00.459 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_17.status 2024-04-06T10:02:00.461 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=5, tag_tid=3, entry_tid=1' 2024-04-06T10:02:00.462 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror image_17 2024-04-06T10:02:00.462 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:02:00.462 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:02:00.462 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_17 2024-04-06T10:02:00.462 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror image_17 '..*' 2024-04-06T10:02:00.462 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:02:00.462 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:02:00.462 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_17 2024-04-06T10:02:00.463 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T10:02:00.463 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-image_17.status 2024-04-06T10:02:00.463 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-image_17.status 2024-04-06T10:02:00.463 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T10:02:00.463 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-image_17.status 2024-04-06T10:02:00.464 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_17 2024-04-06T10:02:00.464 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_17.status 2024-04-06T10:02:00.509 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 1 2024-04-06T10:02:00.509 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 1 2024-04-06T10:02:00.509 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T10:02:00.509 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=5, tag_tid=3, entry_tid=1], [object_number=4, tag_tid=3, entry_tid=0], [object_number=7, tag_tid=2, entry_tid=3], [object_number=6, tag_tid=2, entry_tid=2]]], state=connected] 2024-04-06T10:02:00.509 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=5, tag_tid=3, entry_tid=1], [object_number=4, tag_tid=3, entry_tid=0], [object_number=7, tag_tid=2, entry_tid=3], [object_number=6, tag_tid=2, entry_tid=2]]], state=connected] 2024-04-06T10:02:00.513 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_17.status 2024-04-06T10:02:00.515 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=5, tag_tid=3, entry_tid=1' 2024-04-06T10:02:00.515 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=5, tag_tid=3, entry_tid=1' -a 'object_number=5, tag_tid=3, entry_tid=1' = 'object_number=5, tag_tid=3, entry_tid=1' 2024-04-06T10:02:00.515 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T10:02:00.515 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_snap_present cluster1 mirror image_17 snap 2024-04-06T10:02:00.515 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T10:02:00.515 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:02:00.516 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_17 2024-04-06T10:02:00.516 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local snap_name=snap 2024-04-06T10:02:00.516 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local s 2024-04-06T10:02:00.516 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-04-06T10:02:00.516 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 1 2024-04-06T10:02:01.515 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 info mirror/image_17@snap 2024-04-06T10:02:01.583 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:rbd image 'image_17': 2024-04-06T10:02:01.583 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: size 128 MiB in 32 objects 2024-04-06T10:02:01.583 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: order 22 (4 MiB objects) 2024-04-06T10:02:01.583 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: snapshot_count: 1 2024-04-06T10:02:01.583 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: id: 10ac754bf89c 2024-04-06T10:02:01.583 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: block_name_prefix: rbd_data.10ac754bf89c 2024-04-06T10:02:01.583 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: format: 2 2024-04-06T10:02:01.583 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: features: layering, exclusive-lock, journaling 2024-04-06T10:02:01.583 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: op_features: 2024-04-06T10:02:01.583 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: flags: 2024-04-06T10:02:01.584 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: create_timestamp: Sat Apr 6 09:53:13 2024 2024-04-06T10:02:01.584 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: access_timestamp: Sat Apr 6 09:53:13 2024 2024-04-06T10:02:01.584 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: modify_timestamp: Sat Apr 6 09:54:14 2024 2024-04-06T10:02:01.584 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: protected: False 2024-04-06T10:02:01.584 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: journal: 10ac754bf89c 2024-04-06T10:02:01.584 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: mirroring state: enabled 2024-04-06T10:02:01.584 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: mirroring mode: journal 2024-04-06T10:02:01.585 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: mirroring global id: 2440ae50-6889-4ca9-8f78-a5f7b22dc7ac 2024-04-06T10:02:01.585 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: mirroring primary: false 2024-04-06T10:02:01.589 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T10:02:01.589 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ compare_image_snaps mirror image_17 snap 2024-04-06T10:02:01.589 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:02:01.590 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_17 2024-04-06T10:02:01.590 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local snap_name=snap 2024-04-06T10:02:01.590 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local ret=0 2024-04-06T10:02:01.590 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local rmt_export=/tmp/tmp.BhW6xKmROU/cluster2-mirror-image_17.export 2024-04-06T10:02:01.590 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loc_export=/tmp/tmp.BhW6xKmROU/cluster1-mirror-image_17.export 2024-04-06T10:02:01.590 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rm -f /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_17.export /tmp/tmp.BhW6xKmROU/cluster1-mirror-image_17.export 2024-04-06T10:02:01.590 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster2 -p mirror export image_17@snap /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_17.export 2024-04-06T10:02:01.760 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-04-06T10:02:01.768 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 -p mirror export image_17@snap /tmp/tmp.BhW6xKmROU/cluster1-mirror-image_17.export 2024-04-06T10:02:02.109 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-04-06T10:02:02.124 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmp /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_17.export /tmp/tmp.BhW6xKmROU/cluster1-mirror-image_17.export 2024-04-06T10:02:02.205 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rm -f /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_17.export /tmp/tmp.BhW6xKmROU/cluster1-mirror-image_17.export 2024-04-06T10:02:02.247 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T10:02:02.247 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-06T10:02:02.247 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ image=image_18 2024-04-06T10:02:02.247 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ create_snap cluster2 mirror image_18 snap 2024-04-06T10:02:02.247 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-04-06T10:02:02.247 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:02:02.247 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_18 2024-04-06T10:02:02.247 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local snap_name=snap 2024-04-06T10:02:02.247 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster2 -p mirror snap create image_18@snap --debug-rbd=20 --debug-journaler=20 2024-04-06T10:02:04.870 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_image_replay_started cluster1 mirror image_18 2024-04-06T10:02:04.870 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T10:02:04.870 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:02:04.870 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_18 2024-04-06T10:02:04.870 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_image_replay_state cluster1 mirror image_18 started 2024-04-06T10:02:04.870 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T10:02:04.870 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:02:04.870 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_18 2024-04-06T10:02:04.870 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state=started 2024-04-06T10:02:04.870 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local s 2024-04-06T10:02:04.871 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T10:02:04.871 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 1 2024-04-06T10:02:05.874 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test_image_replay_state cluster1 mirror image_18 started 2024-04-06T10:02:05.875 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T10:02:05.875 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:02:05.875 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_18 2024-04-06T10:02:05.875 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local test_state=started 2024-04-06T10:02:05.875 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local status_result 2024-04-06T10:02:05.875 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local current_state=stopped 2024-04-06T10:02:05.875 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_18 2024-04-06T10:02:05.875 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -i state 2024-04-06T10:02:05.875 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_instance=cluster1 2024-04-06T10:02:05.875 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ shift 2024-04-06T10:02:05.875 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T10:02:05.875 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance=cluster1 2024-04-06T10:02:05.876 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local loop_instance 2024-04-06T10:02:05.876 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T10:02:05.876 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sleep 0 2024-04-06T10:02:05.876 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-04-06T10:02:05.876 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ seq 0 3 2024-04-06T10:02:05.877 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T10:02:05.877 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_18 2024-04-06T10:02:05.877 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1:0 2024-04-06T10:02:05.877 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ shift 2024-04-06T10:02:05.878 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T10:02:05.878 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-04-06T10:02:05.878 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T10:02:05.878 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=cluster 2024-04-06T10:02:05.878 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T10:02:05.878 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:02:05.878 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T10:02:05.878 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T10:02:05.878 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval cluster=cluster1 2024-04-06T10:02:05.878 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ cluster=cluster1 2024-04-06T10:02:05.878 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T10:02:05.878 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T10:02:05.880 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-04-06T10:02:05.880 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local local_cluster=cluster1:0 2024-04-06T10:02:05.880 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local cluster=cluster1 2024-04-06T10:02:05.880 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local instance 2024-04-06T10:02:05.880 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T10:02:05.880 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local val=cluster1:0 2024-04-06T10:02:05.880 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local cluster_var_name=local_cluster 2024-04-06T10:02:05.880 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local instance_var_name=instance 2024-04-06T10:02:05.880 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ cluster=cluster1 2024-04-06T10:02:05.880 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T10:02:05.881 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ '[' 0 = cluster1:0 ']' 2024-04-06T10:02:05.881 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ eval local_cluster=cluster1 2024-04-06T10:02:05.881 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++++ local_cluster=cluster1 2024-04-06T10:02:05.881 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ eval instance=0 2024-04-06T10:02:05.881 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++++ instance=0 2024-04-06T10:02:05.881 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T10:02:05.905 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T10:02:05.905 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T10:02:05.905 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T10:02:05.905 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_18 2024-04-06T10:02:06.009 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:Invalid command: unused arguments: ['mirror/image_18'] 2024-04-06T10:02:06.009 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:rbd mirror status : get status for rbd mirror 2024-04-06T10:02:06.009 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:admin_socket: invalid command 2024-04-06T10:02:06.012 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T10:02:06.012 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_18 2024-04-06T10:02:06.012 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1:1 2024-04-06T10:02:06.012 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ shift 2024-04-06T10:02:06.012 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T10:02:06.012 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-04-06T10:02:06.012 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:1 2024-04-06T10:02:06.012 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=cluster 2024-04-06T10:02:06.012 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T10:02:06.012 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:02:06.012 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=1 2024-04-06T10:02:06.013 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-06T10:02:06.013 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval cluster=cluster1 2024-04-06T10:02:06.013 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ cluster=cluster1 2024-04-06T10:02:06.013 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=1 2024-04-06T10:02:06.013 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=1 2024-04-06T10:02:06.014 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-04-06T10:02:06.014 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local local_cluster=cluster1:1 2024-04-06T10:02:06.014 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local cluster=cluster1 2024-04-06T10:02:06.014 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local instance 2024-04-06T10:02:06.014 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-06T10:02:06.014 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local val=cluster1:1 2024-04-06T10:02:06.014 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local cluster_var_name=local_cluster 2024-04-06T10:02:06.015 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local instance_var_name=instance 2024-04-06T10:02:06.015 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ cluster=cluster1 2024-04-06T10:02:06.015 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=1 2024-04-06T10:02:06.015 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ '[' 1 = cluster1:1 ']' 2024-04-06T10:02:06.015 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ eval local_cluster=cluster1 2024-04-06T10:02:06.015 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++++ local_cluster=cluster1 2024-04-06T10:02:06.015 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ eval instance=1 2024-04-06T10:02:06.015 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++++ instance=1 2024-04-06T10:02:06.015 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-06T10:02:06.033 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T10:02:06.033 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T10:02:06.033 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T10:02:06.033 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_18 2024-04-06T10:02:06.186 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ return 0 2024-04-06T10:02:06.187 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ status_result=' "state": "Replaying"' 2024-04-06T10:02:06.187 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ echo ' "state": "Replaying"' 2024-04-06T10:02:06.187 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep -i Replaying 2024-04-06T10:02:06.189 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: "state": "Replaying" 2024-04-06T10:02:06.191 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ current_state=started 2024-04-06T10:02:06.191 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test started = started 2024-04-06T10:02:06.191 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T10:02:06.191 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_18 2024-04-06T10:02:06.191 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local local_cluster=cluster1 2024-04-06T10:02:06.191 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-04-06T10:02:06.191 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:02:06.191 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_18 2024-04-06T10:02:06.191 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' journal = journal ']' 2024-04-06T10:02:06.191 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_journal_replay_complete cluster1 cluster2 mirror image_18 2024-04-06T10:02:06.191 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local local_cluster=cluster1 2024-04-06T10:02:06.191 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-04-06T10:02:06.192 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:02:06.192 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_18 2024-04-06T10:02:06.192 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local s master_pos mirror_pos last_mirror_pos 2024-04-06T10:02:06.192 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local master_tag master_entry mirror_tag mirror_entry 2024-04-06T10:02:06.192 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T10:02:06.192 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T10:02:06.192 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T10:02:06.393 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror image_18 2024-04-06T10:02:06.393 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T10:02:06.393 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:02:06.393 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_18 2024-04-06T10:02:06.393 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T10:02:06.393 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n image_18 ']' 2024-04-06T10:02:06.394 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/image_18' 2024-04-06T10:02:06.394 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_18 2024-04-06T10:02:06.394 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T10:02:06.394 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T10:02:06.394 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T10:02:06.394 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T10:02:06.394 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T10:02:06.394 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T10:02:06.394 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T10:02:06.395 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T10:02:06.395 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T10:02:06.397 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T10:02:06.397 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_18 2024-04-06T10:02:06.397 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T10:02:06.397 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T10:02:06.397 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T10:02:06.397 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T10:02:06.397 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T10:02:06.397 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T10:02:06.397 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T10:02:06.398 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T10:02:06.398 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T10:02:06.398 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T10:02:06.398 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T10:02:06.398 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:02:06.398 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T10:02:06.398 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T10:02:06.400 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T10:02:06.400 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T10:02:06.400 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T10:02:06.400 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T10:02:06.400 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T10:02:06.400 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T10:02:06.400 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T10:02:06.400 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T10:02:06.400 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:02:06.400 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T10:02:06.400 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T10:02:06.401 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T10:02:06.401 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T10:02:06.401 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T10:02:06.401 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T10:02:06.401 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T10:02:06.425 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T10:02:06.425 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T10:02:06.425 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T10:02:06.425 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_18 2024-04-06T10:02:06.532 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:Invalid command: unused arguments: ['mirror/image_18'] 2024-04-06T10:02:06.533 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:rbd mirror flush : flush rbd mirror 2024-04-06T10:02:06.533 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:admin_socket: invalid command 2024-04-06T10:02:06.535 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T10:02:06.535 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/image_18 2024-04-06T10:02:06.535 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:1 2024-04-06T10:02:06.535 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T10:02:06.535 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T10:02:06.535 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-06T10:02:06.535 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:1 2024-04-06T10:02:06.535 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T10:02:06.535 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T10:02:06.535 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T10:02:06.535 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=1 2024-04-06T10:02:06.535 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-06T10:02:06.536 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T10:02:06.536 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:02:06.536 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=1 2024-04-06T10:02:06.536 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=1 2024-04-06T10:02:06.537 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-06T10:02:06.537 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:1 2024-04-06T10:02:06.537 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T10:02:06.537 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T10:02:06.537 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-06T10:02:06.537 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:1 2024-04-06T10:02:06.537 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T10:02:06.538 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T10:02:06.538 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:02:06.538 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=1 2024-04-06T10:02:06.538 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-06T10:02:06.538 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T10:02:06.538 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T10:02:06.538 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=1 2024-04-06T10:02:06.539 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=1 2024-04-06T10:02:06.539 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-06T10:02:06.560 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T10:02:06.561 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T10:02:06.561 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T10:02:06.561 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/image_18 2024-04-06T10:02:06.699 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T10:02:06.700 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror image_18 2024-04-06T10:02:06.700 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:02:06.700 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:02:06.701 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_18 2024-04-06T10:02:06.701 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror image_18 '' 2024-04-06T10:02:06.701 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:02:06.701 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:02:06.701 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_18 2024-04-06T10:02:06.701 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T10:02:06.701 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-image_18.status 2024-04-06T10:02:06.702 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-image_18.status 2024-04-06T10:02:06.702 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T10:02:06.703 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-image_18.status 2024-04-06T10:02:06.704 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_18 2024-04-06T10:02:06.704 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_18.status 2024-04-06T10:02:06.758 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 2 2024-04-06T10:02:06.758 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 2 2024-04-06T10:02:06.758 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T10:02:06.758 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=9, tag_tid=2, entry_tid=1], [object_number=8, tag_tid=2, entry_tid=0], [object_number=11, tag_tid=1, entry_tid=10035], [object_number=10, tag_tid=1, entry_tid=10034]]], state=connected] 2024-04-06T10:02:06.759 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=8, tag_tid=1, entry_tid=10036], [object_number=11, tag_tid=1, entry_tid=10035], [object_number=10, tag_tid=1, entry_tid=10034], [object_number=9, tag_tid=1, entry_tid=10033]]], state=connected] 2024-04-06T10:02:06.762 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_18.status 2024-04-06T10:02:06.763 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=9, tag_tid=2, entry_tid=1' 2024-04-06T10:02:06.764 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror image_18 2024-04-06T10:02:06.765 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:02:06.765 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:02:06.765 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_18 2024-04-06T10:02:06.765 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror image_18 '..*' 2024-04-06T10:02:06.765 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:02:06.765 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:02:06.765 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_18 2024-04-06T10:02:06.766 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T10:02:06.766 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-image_18.status 2024-04-06T10:02:06.766 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-image_18.status 2024-04-06T10:02:06.766 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T10:02:06.767 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-image_18.status 2024-04-06T10:02:06.768 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_18 2024-04-06T10:02:06.768 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_18.status 2024-04-06T10:02:06.812 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 2 2024-04-06T10:02:06.812 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 2 2024-04-06T10:02:06.812 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T10:02:06.812 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=9, tag_tid=2, entry_tid=1], [object_number=8, tag_tid=2, entry_tid=0], [object_number=11, tag_tid=1, entry_tid=10035], [object_number=10, tag_tid=1, entry_tid=10034]]], state=connected] 2024-04-06T10:02:06.812 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=8, tag_tid=1, entry_tid=10036], [object_number=11, tag_tid=1, entry_tid=10035], [object_number=10, tag_tid=1, entry_tid=10034], [object_number=9, tag_tid=1, entry_tid=10033]]], state=connected] 2024-04-06T10:02:06.816 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_18.status 2024-04-06T10:02:06.818 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=8, tag_tid=1, entry_tid=10036' 2024-04-06T10:02:06.818 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=9, tag_tid=2, entry_tid=1' -a 'object_number=9, tag_tid=2, entry_tid=1' = 'object_number=8, tag_tid=1, entry_tid=10036' 2024-04-06T10:02:06.818 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=8, tag_tid=1, entry_tid=10036' '!=' '' 2024-04-06T10:02:06.818 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T10:02:06.818 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=8, tag_tid=1, entry_tid=10036' = '' 2024-04-06T10:02:06.818 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=8, tag_tid=1, entry_tid=10036' 2024-04-06T10:02:06.819 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=9, tag_tid=2, entry_tid=1' 2024-04-06T10:02:06.819 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T10:02:06.819 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T10:02:06.819 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=2 2024-04-06T10:02:06.820 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=8, tag_tid=1, entry_tid=10036' 2024-04-06T10:02:06.820 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T10:02:06.820 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T10:02:06.821 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=1 2024-04-06T10:02:06.822 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=9, tag_tid=2, entry_tid=1' 2024-04-06T10:02:06.822 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T10:02:06.822 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T10:02:06.823 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T10:02:06.824 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=8, tag_tid=1, entry_tid=10036' 2024-04-06T10:02:06.824 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T10:02:06.824 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T10:02:06.825 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=10036 2024-04-06T10:02:06.825 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 2 = 1 -a 1 -le 10036 2024-04-06T10:02:06.825 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T10:02:06.825 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T10:02:06.826 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T10:02:07.027 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror image_18 2024-04-06T10:02:07.027 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T10:02:07.027 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:02:07.028 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_18 2024-04-06T10:02:07.028 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T10:02:07.028 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n image_18 ']' 2024-04-06T10:02:07.028 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/image_18' 2024-04-06T10:02:07.028 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_18 2024-04-06T10:02:07.028 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T10:02:07.028 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T10:02:07.028 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T10:02:07.028 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T10:02:07.028 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T10:02:07.028 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T10:02:07.028 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T10:02:07.029 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T10:02:07.029 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T10:02:07.031 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T10:02:07.031 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_18 2024-04-06T10:02:07.031 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T10:02:07.031 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T10:02:07.032 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T10:02:07.032 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T10:02:07.032 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T10:02:07.032 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T10:02:07.032 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T10:02:07.032 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T10:02:07.032 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T10:02:07.032 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T10:02:07.032 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T10:02:07.032 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:02:07.032 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T10:02:07.032 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T10:02:07.033 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T10:02:07.034 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T10:02:07.034 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T10:02:07.034 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T10:02:07.034 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T10:02:07.034 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T10:02:07.034 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T10:02:07.034 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T10:02:07.034 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:02:07.034 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T10:02:07.034 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T10:02:07.034 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T10:02:07.034 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T10:02:07.035 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T10:02:07.035 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T10:02:07.035 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T10:02:07.060 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T10:02:07.061 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T10:02:07.061 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T10:02:07.061 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_18 2024-04-06T10:02:07.163 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:Invalid command: unused arguments: ['mirror/image_18'] 2024-04-06T10:02:07.163 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:rbd mirror flush : flush rbd mirror 2024-04-06T10:02:07.163 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:admin_socket: invalid command 2024-04-06T10:02:07.166 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T10:02:07.166 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/image_18 2024-04-06T10:02:07.166 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:1 2024-04-06T10:02:07.166 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T10:02:07.166 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T10:02:07.166 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-06T10:02:07.166 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:1 2024-04-06T10:02:07.166 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T10:02:07.166 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T10:02:07.166 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T10:02:07.166 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=1 2024-04-06T10:02:07.166 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-06T10:02:07.166 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T10:02:07.167 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:02:07.167 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=1 2024-04-06T10:02:07.167 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=1 2024-04-06T10:02:07.168 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-06T10:02:07.168 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:1 2024-04-06T10:02:07.168 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T10:02:07.168 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T10:02:07.168 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-06T10:02:07.168 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:1 2024-04-06T10:02:07.169 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T10:02:07.169 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T10:02:07.169 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:02:07.169 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=1 2024-04-06T10:02:07.169 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-06T10:02:07.169 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T10:02:07.169 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T10:02:07.169 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=1 2024-04-06T10:02:07.169 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=1 2024-04-06T10:02:07.169 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-06T10:02:07.191 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T10:02:07.191 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T10:02:07.192 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T10:02:07.192 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/image_18 2024-04-06T10:02:07.343 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T10:02:07.344 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror image_18 2024-04-06T10:02:07.344 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:02:07.344 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:02:07.344 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_18 2024-04-06T10:02:07.345 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror image_18 '' 2024-04-06T10:02:07.345 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:02:07.345 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:02:07.345 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_18 2024-04-06T10:02:07.345 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T10:02:07.346 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-image_18.status 2024-04-06T10:02:07.346 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-image_18.status 2024-04-06T10:02:07.346 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T10:02:07.348 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-image_18.status 2024-04-06T10:02:07.349 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_18 2024-04-06T10:02:07.349 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_18.status 2024-04-06T10:02:07.395 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 2 2024-04-06T10:02:07.395 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 2 2024-04-06T10:02:07.395 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T10:02:07.395 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=9, tag_tid=2, entry_tid=1], [object_number=8, tag_tid=2, entry_tid=0], [object_number=11, tag_tid=1, entry_tid=10035], [object_number=10, tag_tid=1, entry_tid=10034]]], state=connected] 2024-04-06T10:02:07.395 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=8, tag_tid=1, entry_tid=10036], [object_number=11, tag_tid=1, entry_tid=10035], [object_number=10, tag_tid=1, entry_tid=10034], [object_number=9, tag_tid=1, entry_tid=10033]]], state=connected] 2024-04-06T10:02:07.399 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_18.status 2024-04-06T10:02:07.400 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=9, tag_tid=2, entry_tid=1' 2024-04-06T10:02:07.401 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror image_18 2024-04-06T10:02:07.401 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:02:07.401 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:02:07.401 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_18 2024-04-06T10:02:07.401 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror image_18 '..*' 2024-04-06T10:02:07.401 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:02:07.402 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:02:07.402 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_18 2024-04-06T10:02:07.402 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T10:02:07.402 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-image_18.status 2024-04-06T10:02:07.402 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-image_18.status 2024-04-06T10:02:07.402 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T10:02:07.403 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-image_18.status 2024-04-06T10:02:07.403 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_18 2024-04-06T10:02:07.403 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_18.status 2024-04-06T10:02:07.447 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 2 2024-04-06T10:02:07.447 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 2 2024-04-06T10:02:07.447 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T10:02:07.447 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=9, tag_tid=2, entry_tid=1], [object_number=8, tag_tid=2, entry_tid=0], [object_number=11, tag_tid=1, entry_tid=10035], [object_number=10, tag_tid=1, entry_tid=10034]]], state=connected] 2024-04-06T10:02:07.447 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=8, tag_tid=1, entry_tid=10036], [object_number=11, tag_tid=1, entry_tid=10035], [object_number=10, tag_tid=1, entry_tid=10034], [object_number=9, tag_tid=1, entry_tid=10033]]], state=connected] 2024-04-06T10:02:07.450 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_18.status 2024-04-06T10:02:07.452 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=8, tag_tid=1, entry_tid=10036' 2024-04-06T10:02:07.452 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=9, tag_tid=2, entry_tid=1' -a 'object_number=9, tag_tid=2, entry_tid=1' = 'object_number=8, tag_tid=1, entry_tid=10036' 2024-04-06T10:02:07.452 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=8, tag_tid=1, entry_tid=10036' '!=' 'object_number=8, tag_tid=1, entry_tid=10036' 2024-04-06T10:02:07.452 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T10:02:07.452 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.4 2024-04-06T10:02:07.854 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror image_18 2024-04-06T10:02:07.855 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T10:02:07.855 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:02:07.855 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_18 2024-04-06T10:02:07.855 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T10:02:07.855 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n image_18 ']' 2024-04-06T10:02:07.855 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/image_18' 2024-04-06T10:02:07.855 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_18 2024-04-06T10:02:07.855 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T10:02:07.855 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T10:02:07.855 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T10:02:07.855 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T10:02:07.856 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T10:02:07.856 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T10:02:07.856 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T10:02:07.856 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T10:02:07.856 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T10:02:07.858 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T10:02:07.858 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_18 2024-04-06T10:02:07.859 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T10:02:07.859 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T10:02:07.859 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T10:02:07.859 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T10:02:07.859 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T10:02:07.859 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T10:02:07.859 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T10:02:07.859 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T10:02:07.859 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T10:02:07.859 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T10:02:07.859 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T10:02:07.859 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:02:07.859 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T10:02:07.860 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T10:02:07.861 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T10:02:07.861 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T10:02:07.862 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T10:02:07.862 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T10:02:07.862 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T10:02:07.862 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T10:02:07.862 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T10:02:07.862 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T10:02:07.862 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:02:07.862 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T10:02:07.862 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T10:02:07.862 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T10:02:07.862 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T10:02:07.862 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T10:02:07.862 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T10:02:07.863 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T10:02:07.889 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T10:02:07.889 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T10:02:07.890 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T10:02:07.890 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_18 2024-04-06T10:02:07.994 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:Invalid command: unused arguments: ['mirror/image_18'] 2024-04-06T10:02:07.994 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:rbd mirror flush : flush rbd mirror 2024-04-06T10:02:07.995 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:admin_socket: invalid command 2024-04-06T10:02:07.996 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T10:02:07.996 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/image_18 2024-04-06T10:02:07.997 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:1 2024-04-06T10:02:07.997 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T10:02:07.997 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T10:02:07.997 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-06T10:02:07.997 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:1 2024-04-06T10:02:07.997 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T10:02:07.997 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T10:02:07.997 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T10:02:07.997 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=1 2024-04-06T10:02:07.997 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-06T10:02:07.997 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T10:02:07.997 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:02:07.997 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=1 2024-04-06T10:02:07.997 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=1 2024-04-06T10:02:07.999 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-06T10:02:07.999 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:1 2024-04-06T10:02:07.999 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T10:02:07.999 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T10:02:07.999 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-06T10:02:07.999 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:1 2024-04-06T10:02:08.000 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T10:02:08.000 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T10:02:08.000 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:02:08.000 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=1 2024-04-06T10:02:08.000 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-06T10:02:08.000 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T10:02:08.000 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T10:02:08.000 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=1 2024-04-06T10:02:08.000 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=1 2024-04-06T10:02:08.000 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-06T10:02:08.018 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T10:02:08.018 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T10:02:08.018 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T10:02:08.018 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/image_18 2024-04-06T10:02:08.164 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T10:02:08.165 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror image_18 2024-04-06T10:02:08.165 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:02:08.165 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:02:08.165 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_18 2024-04-06T10:02:08.165 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror image_18 '' 2024-04-06T10:02:08.165 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:02:08.165 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:02:08.166 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_18 2024-04-06T10:02:08.166 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T10:02:08.166 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-image_18.status 2024-04-06T10:02:08.166 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-image_18.status 2024-04-06T10:02:08.166 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T10:02:08.169 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-image_18.status 2024-04-06T10:02:08.169 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_18 2024-04-06T10:02:08.169 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_18.status 2024-04-06T10:02:08.219 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 2 2024-04-06T10:02:08.219 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 2 2024-04-06T10:02:08.219 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T10:02:08.219 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=9, tag_tid=2, entry_tid=1], [object_number=8, tag_tid=2, entry_tid=0], [object_number=11, tag_tid=1, entry_tid=10035], [object_number=10, tag_tid=1, entry_tid=10034]]], state=connected] 2024-04-06T10:02:08.219 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=8, tag_tid=1, entry_tid=10036], [object_number=11, tag_tid=1, entry_tid=10035], [object_number=10, tag_tid=1, entry_tid=10034], [object_number=9, tag_tid=1, entry_tid=10033]]], state=connected] 2024-04-06T10:02:08.222 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_18.status 2024-04-06T10:02:08.224 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=9, tag_tid=2, entry_tid=1' 2024-04-06T10:02:08.225 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror image_18 2024-04-06T10:02:08.225 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:02:08.225 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:02:08.225 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_18 2024-04-06T10:02:08.225 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror image_18 '..*' 2024-04-06T10:02:08.225 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:02:08.225 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:02:08.225 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_18 2024-04-06T10:02:08.225 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T10:02:08.225 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-image_18.status 2024-04-06T10:02:08.225 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-image_18.status 2024-04-06T10:02:08.226 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T10:02:08.227 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-image_18.status 2024-04-06T10:02:08.227 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_18 2024-04-06T10:02:08.227 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_18.status 2024-04-06T10:02:08.272 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 2 2024-04-06T10:02:08.273 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 2 2024-04-06T10:02:08.273 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T10:02:08.273 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=9, tag_tid=2, entry_tid=1], [object_number=8, tag_tid=2, entry_tid=0], [object_number=11, tag_tid=1, entry_tid=10035], [object_number=10, tag_tid=1, entry_tid=10034]]], state=connected] 2024-04-06T10:02:08.273 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=8, tag_tid=1, entry_tid=10036], [object_number=11, tag_tid=1, entry_tid=10035], [object_number=10, tag_tid=1, entry_tid=10034], [object_number=9, tag_tid=1, entry_tid=10033]]], state=connected] 2024-04-06T10:02:08.276 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_18.status 2024-04-06T10:02:08.278 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=8, tag_tid=1, entry_tid=10036' 2024-04-06T10:02:08.278 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=9, tag_tid=2, entry_tid=1' -a 'object_number=9, tag_tid=2, entry_tid=1' = 'object_number=8, tag_tid=1, entry_tid=10036' 2024-04-06T10:02:08.278 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=8, tag_tid=1, entry_tid=10036' '!=' 'object_number=8, tag_tid=1, entry_tid=10036' 2024-04-06T10:02:08.278 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T10:02:08.278 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.8 2024-04-06T10:02:09.080 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror image_18 2024-04-06T10:02:09.080 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T10:02:09.080 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:02:09.080 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_18 2024-04-06T10:02:09.080 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T10:02:09.080 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n image_18 ']' 2024-04-06T10:02:09.080 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/image_18' 2024-04-06T10:02:09.080 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_18 2024-04-06T10:02:09.080 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T10:02:09.081 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T10:02:09.081 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T10:02:09.081 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T10:02:09.081 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T10:02:09.081 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T10:02:09.081 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T10:02:09.081 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T10:02:09.082 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T10:02:09.084 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T10:02:09.084 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_18 2024-04-06T10:02:09.084 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T10:02:09.085 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T10:02:09.085 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T10:02:09.085 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T10:02:09.085 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T10:02:09.085 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T10:02:09.085 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T10:02:09.085 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T10:02:09.085 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T10:02:09.085 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T10:02:09.085 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T10:02:09.085 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:02:09.085 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T10:02:09.085 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T10:02:09.087 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T10:02:09.088 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T10:02:09.088 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T10:02:09.088 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T10:02:09.088 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T10:02:09.088 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T10:02:09.088 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T10:02:09.088 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T10:02:09.088 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:02:09.088 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T10:02:09.088 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T10:02:09.088 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T10:02:09.088 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T10:02:09.088 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T10:02:09.088 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T10:02:09.089 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T10:02:09.111 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T10:02:09.112 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T10:02:09.112 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T10:02:09.112 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_18 2024-04-06T10:02:09.216 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:Invalid command: unused arguments: ['mirror/image_18'] 2024-04-06T10:02:09.216 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:rbd mirror flush : flush rbd mirror 2024-04-06T10:02:09.216 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:admin_socket: invalid command 2024-04-06T10:02:09.218 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T10:02:09.219 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/image_18 2024-04-06T10:02:09.219 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:1 2024-04-06T10:02:09.219 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T10:02:09.219 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T10:02:09.219 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-06T10:02:09.219 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:1 2024-04-06T10:02:09.219 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T10:02:09.219 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T10:02:09.219 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T10:02:09.219 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=1 2024-04-06T10:02:09.219 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-06T10:02:09.219 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T10:02:09.219 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:02:09.219 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=1 2024-04-06T10:02:09.220 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=1 2024-04-06T10:02:09.221 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-06T10:02:09.221 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:1 2024-04-06T10:02:09.221 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T10:02:09.221 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T10:02:09.221 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-06T10:02:09.221 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:1 2024-04-06T10:02:09.221 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T10:02:09.221 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T10:02:09.221 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:02:09.222 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=1 2024-04-06T10:02:09.222 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-06T10:02:09.222 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T10:02:09.222 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T10:02:09.222 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=1 2024-04-06T10:02:09.222 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=1 2024-04-06T10:02:09.222 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-06T10:02:09.244 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T10:02:09.244 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T10:02:09.244 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T10:02:09.244 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/image_18 2024-04-06T10:02:09.392 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T10:02:09.393 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror image_18 2024-04-06T10:02:09.394 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:02:09.394 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:02:09.394 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_18 2024-04-06T10:02:09.394 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror image_18 '' 2024-04-06T10:02:09.394 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:02:09.394 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:02:09.394 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_18 2024-04-06T10:02:09.394 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T10:02:09.395 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-image_18.status 2024-04-06T10:02:09.395 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-image_18.status 2024-04-06T10:02:09.395 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T10:02:09.396 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-image_18.status 2024-04-06T10:02:09.396 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_18 2024-04-06T10:02:09.397 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_18.status 2024-04-06T10:02:09.441 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 2 2024-04-06T10:02:09.442 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 2 2024-04-06T10:02:09.442 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T10:02:09.442 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=9, tag_tid=2, entry_tid=1], [object_number=8, tag_tid=2, entry_tid=0], [object_number=11, tag_tid=1, entry_tid=10035], [object_number=10, tag_tid=1, entry_tid=10034]]], state=connected] 2024-04-06T10:02:09.442 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=8, tag_tid=1, entry_tid=10036], [object_number=11, tag_tid=1, entry_tid=10035], [object_number=10, tag_tid=1, entry_tid=10034], [object_number=9, tag_tid=1, entry_tid=10033]]], state=connected] 2024-04-06T10:02:09.445 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_18.status 2024-04-06T10:02:09.446 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=9, tag_tid=2, entry_tid=1' 2024-04-06T10:02:09.448 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror image_18 2024-04-06T10:02:09.448 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:02:09.448 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:02:09.448 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_18 2024-04-06T10:02:09.448 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror image_18 '..*' 2024-04-06T10:02:09.448 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:02:09.448 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:02:09.448 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_18 2024-04-06T10:02:09.448 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T10:02:09.448 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-image_18.status 2024-04-06T10:02:09.448 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-image_18.status 2024-04-06T10:02:09.448 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T10:02:09.449 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-image_18.status 2024-04-06T10:02:09.449 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_18 2024-04-06T10:02:09.449 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_18.status 2024-04-06T10:02:09.493 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 2 2024-04-06T10:02:09.493 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 2 2024-04-06T10:02:09.494 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T10:02:09.494 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=9, tag_tid=2, entry_tid=1], [object_number=8, tag_tid=2, entry_tid=0], [object_number=11, tag_tid=1, entry_tid=10035], [object_number=10, tag_tid=1, entry_tid=10034]]], state=connected] 2024-04-06T10:02:09.494 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=8, tag_tid=1, entry_tid=10036], [object_number=11, tag_tid=1, entry_tid=10035], [object_number=10, tag_tid=1, entry_tid=10034], [object_number=9, tag_tid=1, entry_tid=10033]]], state=connected] 2024-04-06T10:02:09.497 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_18.status 2024-04-06T10:02:09.498 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=8, tag_tid=1, entry_tid=10036' 2024-04-06T10:02:09.499 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=9, tag_tid=2, entry_tid=1' -a 'object_number=9, tag_tid=2, entry_tid=1' = 'object_number=8, tag_tid=1, entry_tid=10036' 2024-04-06T10:02:09.499 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=8, tag_tid=1, entry_tid=10036' '!=' 'object_number=8, tag_tid=1, entry_tid=10036' 2024-04-06T10:02:09.499 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T10:02:09.499 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 1.6 2024-04-06T10:02:11.101 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror image_18 2024-04-06T10:02:11.101 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T10:02:11.101 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:02:11.101 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_18 2024-04-06T10:02:11.101 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T10:02:11.101 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n image_18 ']' 2024-04-06T10:02:11.101 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/image_18' 2024-04-06T10:02:11.101 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_18 2024-04-06T10:02:11.101 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T10:02:11.101 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T10:02:11.102 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T10:02:11.102 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T10:02:11.102 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T10:02:11.102 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T10:02:11.102 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T10:02:11.102 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T10:02:11.103 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T10:02:11.105 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T10:02:11.105 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_18 2024-04-06T10:02:11.105 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T10:02:11.105 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T10:02:11.105 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T10:02:11.105 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T10:02:11.105 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T10:02:11.106 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T10:02:11.106 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T10:02:11.106 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T10:02:11.106 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T10:02:11.106 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T10:02:11.106 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T10:02:11.106 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:02:11.106 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T10:02:11.107 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T10:02:11.107 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T10:02:11.107 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T10:02:11.108 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T10:02:11.108 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T10:02:11.108 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T10:02:11.108 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T10:02:11.108 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T10:02:11.108 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T10:02:11.108 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:02:11.108 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T10:02:11.108 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T10:02:11.108 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T10:02:11.108 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T10:02:11.108 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T10:02:11.108 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T10:02:11.108 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T10:02:11.131 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T10:02:11.132 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T10:02:11.132 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T10:02:11.132 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_18 2024-04-06T10:02:11.236 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:Invalid command: unused arguments: ['mirror/image_18'] 2024-04-06T10:02:11.236 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:rbd mirror flush : flush rbd mirror 2024-04-06T10:02:11.237 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:admin_socket: invalid command 2024-04-06T10:02:11.241 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T10:02:11.241 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/image_18 2024-04-06T10:02:11.242 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:1 2024-04-06T10:02:11.242 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T10:02:11.242 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T10:02:11.242 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-06T10:02:11.242 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:1 2024-04-06T10:02:11.242 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T10:02:11.243 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T10:02:11.243 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T10:02:11.243 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=1 2024-04-06T10:02:11.243 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-06T10:02:11.243 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T10:02:11.243 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:02:11.243 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=1 2024-04-06T10:02:11.243 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=1 2024-04-06T10:02:11.243 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-06T10:02:11.243 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:1 2024-04-06T10:02:11.243 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T10:02:11.243 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T10:02:11.243 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-06T10:02:11.243 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:1 2024-04-06T10:02:11.244 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T10:02:11.244 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T10:02:11.244 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:02:11.244 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=1 2024-04-06T10:02:11.244 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-06T10:02:11.244 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T10:02:11.244 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T10:02:11.244 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=1 2024-04-06T10:02:11.244 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=1 2024-04-06T10:02:11.244 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-06T10:02:11.264 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T10:02:11.264 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T10:02:11.264 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T10:02:11.264 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/image_18 2024-04-06T10:02:11.419 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T10:02:11.420 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror image_18 2024-04-06T10:02:11.420 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:02:11.420 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:02:11.420 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_18 2024-04-06T10:02:11.420 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror image_18 '' 2024-04-06T10:02:11.420 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:02:11.420 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:02:11.420 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_18 2024-04-06T10:02:11.420 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T10:02:11.421 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-image_18.status 2024-04-06T10:02:11.421 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-image_18.status 2024-04-06T10:02:11.421 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T10:02:11.422 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-image_18.status 2024-04-06T10:02:11.423 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_18 2024-04-06T10:02:11.423 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_18.status 2024-04-06T10:02:11.475 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 2 2024-04-06T10:02:11.475 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 2 2024-04-06T10:02:11.475 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T10:02:11.475 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=9, tag_tid=2, entry_tid=1], [object_number=8, tag_tid=2, entry_tid=0], [object_number=11, tag_tid=1, entry_tid=10035], [object_number=10, tag_tid=1, entry_tid=10034]]], state=connected] 2024-04-06T10:02:11.475 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=9, tag_tid=2, entry_tid=1], [object_number=8, tag_tid=2, entry_tid=0], [object_number=11, tag_tid=1, entry_tid=10035], [object_number=10, tag_tid=1, entry_tid=10034]]], state=connected] 2024-04-06T10:02:11.478 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_18.status 2024-04-06T10:02:11.480 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=9, tag_tid=2, entry_tid=1' 2024-04-06T10:02:11.481 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror image_18 2024-04-06T10:02:11.481 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:02:11.482 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:02:11.482 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_18 2024-04-06T10:02:11.482 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror image_18 '..*' 2024-04-06T10:02:11.482 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:02:11.482 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:02:11.482 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_18 2024-04-06T10:02:11.482 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T10:02:11.482 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-image_18.status 2024-04-06T10:02:11.482 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-image_18.status 2024-04-06T10:02:11.483 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T10:02:11.483 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-image_18.status 2024-04-06T10:02:11.483 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_18 2024-04-06T10:02:11.484 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_18.status 2024-04-06T10:02:11.529 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 2 2024-04-06T10:02:11.529 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 2 2024-04-06T10:02:11.529 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T10:02:11.529 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=9, tag_tid=2, entry_tid=1], [object_number=8, tag_tid=2, entry_tid=0], [object_number=11, tag_tid=1, entry_tid=10035], [object_number=10, tag_tid=1, entry_tid=10034]]], state=connected] 2024-04-06T10:02:11.530 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=9, tag_tid=2, entry_tid=1], [object_number=8, tag_tid=2, entry_tid=0], [object_number=11, tag_tid=1, entry_tid=10035], [object_number=10, tag_tid=1, entry_tid=10034]]], state=connected] 2024-04-06T10:02:11.533 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_18.status 2024-04-06T10:02:11.536 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=9, tag_tid=2, entry_tid=1' 2024-04-06T10:02:11.536 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=9, tag_tid=2, entry_tid=1' -a 'object_number=9, tag_tid=2, entry_tid=1' = 'object_number=9, tag_tid=2, entry_tid=1' 2024-04-06T10:02:11.536 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T10:02:11.536 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_snap_present cluster1 mirror image_18 snap 2024-04-06T10:02:11.537 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T10:02:11.537 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:02:11.537 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_18 2024-04-06T10:02:11.537 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local snap_name=snap 2024-04-06T10:02:11.537 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local s 2024-04-06T10:02:11.537 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-04-06T10:02:11.537 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 1 2024-04-06T10:02:12.536 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 info mirror/image_18@snap 2024-04-06T10:02:12.600 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:rbd image 'image_18': 2024-04-06T10:02:12.600 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: size 128 MiB in 32 objects 2024-04-06T10:02:12.600 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: order 22 (4 MiB objects) 2024-04-06T10:02:12.600 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: snapshot_count: 1 2024-04-06T10:02:12.600 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: id: 10ac2c51cbb7 2024-04-06T10:02:12.600 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: block_name_prefix: rbd_data.10ac2c51cbb7 2024-04-06T10:02:12.600 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: format: 2 2024-04-06T10:02:12.600 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: features: layering, exclusive-lock, journaling 2024-04-06T10:02:12.600 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: op_features: 2024-04-06T10:02:12.600 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: flags: 2024-04-06T10:02:12.600 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: create_timestamp: Sat Apr 6 09:53:18 2024 2024-04-06T10:02:12.601 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: access_timestamp: Sat Apr 6 09:53:18 2024 2024-04-06T10:02:12.601 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: modify_timestamp: Sat Apr 6 09:54:19 2024 2024-04-06T10:02:12.601 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: protected: False 2024-04-06T10:02:12.601 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: journal: 10ac2c51cbb7 2024-04-06T10:02:12.601 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: mirroring state: enabled 2024-04-06T10:02:12.601 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: mirroring mode: journal 2024-04-06T10:02:12.601 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: mirroring global id: d6382df1-d576-4dd7-b937-9e187ca7ef11 2024-04-06T10:02:12.601 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: mirroring primary: false 2024-04-06T10:02:12.605 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T10:02:12.605 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ compare_image_snaps mirror image_18 snap 2024-04-06T10:02:12.605 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:02:12.605 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_18 2024-04-06T10:02:12.605 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local snap_name=snap 2024-04-06T10:02:12.605 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local ret=0 2024-04-06T10:02:12.605 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local rmt_export=/tmp/tmp.BhW6xKmROU/cluster2-mirror-image_18.export 2024-04-06T10:02:12.605 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loc_export=/tmp/tmp.BhW6xKmROU/cluster1-mirror-image_18.export 2024-04-06T10:02:12.605 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rm -f /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_18.export /tmp/tmp.BhW6xKmROU/cluster1-mirror-image_18.export 2024-04-06T10:02:12.606 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster2 -p mirror export image_18@snap /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_18.export 2024-04-06T10:02:12.775 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-04-06T10:02:12.783 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 -p mirror export image_18@snap /tmp/tmp.BhW6xKmROU/cluster1-mirror-image_18.export 2024-04-06T10:02:13.409 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-04-06T10:02:13.422 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmp /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_18.export /tmp/tmp.BhW6xKmROU/cluster1-mirror-image_18.export 2024-04-06T10:02:13.503 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rm -f /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_18.export /tmp/tmp.BhW6xKmROU/cluster1-mirror-image_18.export 2024-04-06T10:02:13.547 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T10:02:13.547 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-06T10:02:13.547 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ image=image_19 2024-04-06T10:02:13.547 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ create_snap cluster2 mirror image_19 snap 2024-04-06T10:02:13.547 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-04-06T10:02:13.547 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:02:13.547 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_19 2024-04-06T10:02:13.547 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local snap_name=snap 2024-04-06T10:02:13.548 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster2 -p mirror snap create image_19@snap --debug-rbd=20 --debug-journaler=20 2024-04-06T10:02:15.902 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_image_replay_started cluster1 mirror image_19 2024-04-06T10:02:15.902 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T10:02:15.902 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:02:15.902 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_19 2024-04-06T10:02:15.903 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_image_replay_state cluster1 mirror image_19 started 2024-04-06T10:02:15.903 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T10:02:15.903 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:02:15.903 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_19 2024-04-06T10:02:15.903 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state=started 2024-04-06T10:02:15.903 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local s 2024-04-06T10:02:15.903 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T10:02:15.903 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 1 2024-04-06T10:02:16.904 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test_image_replay_state cluster1 mirror image_19 started 2024-04-06T10:02:16.904 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T10:02:16.904 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:02:16.904 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_19 2024-04-06T10:02:16.904 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local test_state=started 2024-04-06T10:02:16.904 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local status_result 2024-04-06T10:02:16.905 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local current_state=stopped 2024-04-06T10:02:16.905 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_19 2024-04-06T10:02:16.906 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_instance=cluster1 2024-04-06T10:02:16.906 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ shift 2024-04-06T10:02:16.906 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T10:02:16.906 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance=cluster1 2024-04-06T10:02:16.906 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local loop_instance 2024-04-06T10:02:16.906 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T10:02:16.906 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sleep 0 2024-04-06T10:02:16.906 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -i state 2024-04-06T10:02:16.907 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-04-06T10:02:16.907 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ seq 0 3 2024-04-06T10:02:16.909 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T10:02:16.909 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_19 2024-04-06T10:02:16.909 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1:0 2024-04-06T10:02:16.910 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ shift 2024-04-06T10:02:16.910 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T10:02:16.910 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-04-06T10:02:16.910 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T10:02:16.910 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=cluster 2024-04-06T10:02:16.910 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T10:02:16.910 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:02:16.910 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T10:02:16.910 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T10:02:16.910 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval cluster=cluster1 2024-04-06T10:02:16.910 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ cluster=cluster1 2024-04-06T10:02:16.910 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T10:02:16.910 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T10:02:16.911 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-04-06T10:02:16.911 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local local_cluster=cluster1:0 2024-04-06T10:02:16.911 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local cluster=cluster1 2024-04-06T10:02:16.911 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local instance 2024-04-06T10:02:16.912 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T10:02:16.912 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local val=cluster1:0 2024-04-06T10:02:16.912 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local cluster_var_name=local_cluster 2024-04-06T10:02:16.912 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local instance_var_name=instance 2024-04-06T10:02:16.912 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ cluster=cluster1 2024-04-06T10:02:16.912 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T10:02:16.912 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ '[' 0 = cluster1:0 ']' 2024-04-06T10:02:16.913 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ eval local_cluster=cluster1 2024-04-06T10:02:16.913 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++++ local_cluster=cluster1 2024-04-06T10:02:16.913 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ eval instance=0 2024-04-06T10:02:16.913 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++++ instance=0 2024-04-06T10:02:16.913 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T10:02:16.932 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T10:02:16.932 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T10:02:16.932 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T10:02:16.932 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_19 2024-04-06T10:02:17.035 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:Invalid command: unused arguments: ['mirror/image_19'] 2024-04-06T10:02:17.035 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:rbd mirror status : get status for rbd mirror 2024-04-06T10:02:17.035 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:admin_socket: invalid command 2024-04-06T10:02:17.037 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T10:02:17.037 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_19 2024-04-06T10:02:17.038 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1:1 2024-04-06T10:02:17.038 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ shift 2024-04-06T10:02:17.038 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T10:02:17.038 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-04-06T10:02:17.038 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:1 2024-04-06T10:02:17.038 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=cluster 2024-04-06T10:02:17.038 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T10:02:17.038 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:02:17.038 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=1 2024-04-06T10:02:17.038 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-06T10:02:17.038 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval cluster=cluster1 2024-04-06T10:02:17.038 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ cluster=cluster1 2024-04-06T10:02:17.038 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=1 2024-04-06T10:02:17.038 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=1 2024-04-06T10:02:17.040 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-04-06T10:02:17.040 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local local_cluster=cluster1:1 2024-04-06T10:02:17.040 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local cluster=cluster1 2024-04-06T10:02:17.040 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local instance 2024-04-06T10:02:17.041 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-06T10:02:17.041 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local val=cluster1:1 2024-04-06T10:02:17.041 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local cluster_var_name=local_cluster 2024-04-06T10:02:17.041 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local instance_var_name=instance 2024-04-06T10:02:17.041 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ cluster=cluster1 2024-04-06T10:02:17.041 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=1 2024-04-06T10:02:17.041 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ '[' 1 = cluster1:1 ']' 2024-04-06T10:02:17.041 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ eval local_cluster=cluster1 2024-04-06T10:02:17.041 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++++ local_cluster=cluster1 2024-04-06T10:02:17.041 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ eval instance=1 2024-04-06T10:02:17.041 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++++ instance=1 2024-04-06T10:02:17.041 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-06T10:02:17.063 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T10:02:17.063 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T10:02:17.063 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T10:02:17.063 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_19 2024-04-06T10:02:17.218 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ return 0 2024-04-06T10:02:17.218 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ status_result=' "state": "Replaying"' 2024-04-06T10:02:17.219 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ echo ' "state": "Replaying"' 2024-04-06T10:02:17.219 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep -i Replaying 2024-04-06T10:02:17.221 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: "state": "Replaying" 2024-04-06T10:02:17.223 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ current_state=started 2024-04-06T10:02:17.223 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test started = started 2024-04-06T10:02:17.223 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T10:02:17.223 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_19 2024-04-06T10:02:17.224 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local local_cluster=cluster1 2024-04-06T10:02:17.224 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-04-06T10:02:17.224 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:02:17.224 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_19 2024-04-06T10:02:17.224 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' journal = journal ']' 2024-04-06T10:02:17.224 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_journal_replay_complete cluster1 cluster2 mirror image_19 2024-04-06T10:02:17.224 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local local_cluster=cluster1 2024-04-06T10:02:17.224 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-04-06T10:02:17.224 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:02:17.224 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_19 2024-04-06T10:02:17.224 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local s master_pos mirror_pos last_mirror_pos 2024-04-06T10:02:17.224 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local master_tag master_entry mirror_tag mirror_entry 2024-04-06T10:02:17.224 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T10:02:17.224 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T10:02:17.225 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T10:02:17.424 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror image_19 2024-04-06T10:02:17.424 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T10:02:17.424 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:02:17.424 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_19 2024-04-06T10:02:17.424 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T10:02:17.424 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n image_19 ']' 2024-04-06T10:02:17.424 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/image_19' 2024-04-06T10:02:17.424 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_19 2024-04-06T10:02:17.424 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T10:02:17.424 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T10:02:17.425 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T10:02:17.425 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T10:02:17.425 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T10:02:17.425 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T10:02:17.425 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T10:02:17.425 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T10:02:17.426 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T10:02:17.428 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T10:02:17.428 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_19 2024-04-06T10:02:17.428 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T10:02:17.429 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T10:02:17.429 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T10:02:17.429 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T10:02:17.429 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T10:02:17.429 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T10:02:17.429 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T10:02:17.429 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T10:02:17.429 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T10:02:17.429 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T10:02:17.429 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T10:02:17.429 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:02:17.429 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T10:02:17.429 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T10:02:17.429 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T10:02:17.430 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T10:02:17.430 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T10:02:17.430 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T10:02:17.430 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T10:02:17.430 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T10:02:17.430 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T10:02:17.430 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T10:02:17.430 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:02:17.430 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T10:02:17.430 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T10:02:17.430 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T10:02:17.430 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T10:02:17.430 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T10:02:17.430 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T10:02:17.431 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T10:02:17.454 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T10:02:17.455 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T10:02:17.455 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T10:02:17.455 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_19 2024-04-06T10:02:17.558 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:Invalid command: unused arguments: ['mirror/image_19'] 2024-04-06T10:02:17.559 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:rbd mirror flush : flush rbd mirror 2024-04-06T10:02:17.559 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:admin_socket: invalid command 2024-04-06T10:02:17.561 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T10:02:17.561 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/image_19 2024-04-06T10:02:17.561 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:1 2024-04-06T10:02:17.561 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T10:02:17.561 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T10:02:17.562 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-06T10:02:17.562 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:1 2024-04-06T10:02:17.562 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T10:02:17.562 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T10:02:17.562 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T10:02:17.562 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=1 2024-04-06T10:02:17.562 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-06T10:02:17.562 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T10:02:17.562 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:02:17.562 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=1 2024-04-06T10:02:17.565 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=1 2024-04-06T10:02:17.566 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-06T10:02:17.567 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:1 2024-04-06T10:02:17.567 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T10:02:17.567 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T10:02:17.567 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-06T10:02:17.567 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:1 2024-04-06T10:02:17.567 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T10:02:17.567 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T10:02:17.567 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:02:17.567 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=1 2024-04-06T10:02:17.567 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-06T10:02:17.567 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T10:02:17.567 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T10:02:17.568 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=1 2024-04-06T10:02:17.568 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=1 2024-04-06T10:02:17.568 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-06T10:02:17.588 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T10:02:17.588 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T10:02:17.588 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T10:02:17.588 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/image_19 2024-04-06T10:02:17.726 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T10:02:17.727 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror image_19 2024-04-06T10:02:17.727 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:02:17.727 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:02:17.727 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_19 2024-04-06T10:02:17.727 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror image_19 '' 2024-04-06T10:02:17.727 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:02:17.728 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:02:17.728 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_19 2024-04-06T10:02:17.728 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T10:02:17.728 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-image_19.status 2024-04-06T10:02:17.728 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-image_19.status 2024-04-06T10:02:17.729 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T10:02:17.730 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-image_19.status 2024-04-06T10:02:17.730 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_19 2024-04-06T10:02:17.730 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_19.status 2024-04-06T10:02:17.776 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 2 2024-04-06T10:02:17.776 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 2 2024-04-06T10:02:17.776 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T10:02:17.776 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=9, tag_tid=2, entry_tid=1], [object_number=8, tag_tid=2, entry_tid=0], [object_number=11, tag_tid=1, entry_tid=8887], [object_number=10, tag_tid=1, entry_tid=8886]]], state=connected] 2024-04-06T10:02:17.776 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=11, tag_tid=1, entry_tid=8887], [object_number=10, tag_tid=1, entry_tid=8886], [object_number=9, tag_tid=1, entry_tid=8885], [object_number=8, tag_tid=1, entry_tid=8884]]], state=connected] 2024-04-06T10:02:17.779 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_19.status 2024-04-06T10:02:17.781 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=9, tag_tid=2, entry_tid=1' 2024-04-06T10:02:17.782 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror image_19 2024-04-06T10:02:17.782 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:02:17.782 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:02:17.782 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_19 2024-04-06T10:02:17.783 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror image_19 '..*' 2024-04-06T10:02:17.783 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:02:17.783 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:02:17.783 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_19 2024-04-06T10:02:17.783 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T10:02:17.783 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-image_19.status 2024-04-06T10:02:17.783 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-image_19.status 2024-04-06T10:02:17.783 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T10:02:17.784 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-image_19.status 2024-04-06T10:02:17.784 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_19 2024-04-06T10:02:17.784 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_19.status 2024-04-06T10:02:17.828 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 2 2024-04-06T10:02:17.829 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 2 2024-04-06T10:02:17.829 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T10:02:17.829 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=9, tag_tid=2, entry_tid=1], [object_number=8, tag_tid=2, entry_tid=0], [object_number=11, tag_tid=1, entry_tid=8887], [object_number=10, tag_tid=1, entry_tid=8886]]], state=connected] 2024-04-06T10:02:17.829 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=11, tag_tid=1, entry_tid=8887], [object_number=10, tag_tid=1, entry_tid=8886], [object_number=9, tag_tid=1, entry_tid=8885], [object_number=8, tag_tid=1, entry_tid=8884]]], state=connected] 2024-04-06T10:02:17.832 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_19.status 2024-04-06T10:02:17.834 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=11, tag_tid=1, entry_tid=8887' 2024-04-06T10:02:17.835 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=9, tag_tid=2, entry_tid=1' -a 'object_number=9, tag_tid=2, entry_tid=1' = 'object_number=11, tag_tid=1, entry_tid=8887' 2024-04-06T10:02:17.835 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=11, tag_tid=1, entry_tid=8887' '!=' '' 2024-04-06T10:02:17.835 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T10:02:17.835 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=11, tag_tid=1, entry_tid=8887' = '' 2024-04-06T10:02:17.835 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=11, tag_tid=1, entry_tid=8887' 2024-04-06T10:02:17.835 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=9, tag_tid=2, entry_tid=1' 2024-04-06T10:02:17.835 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T10:02:17.835 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T10:02:17.835 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=2 2024-04-06T10:02:17.836 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=11, tag_tid=1, entry_tid=8887' 2024-04-06T10:02:17.836 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T10:02:17.836 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T10:02:17.837 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=1 2024-04-06T10:02:17.838 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=9, tag_tid=2, entry_tid=1' 2024-04-06T10:02:17.838 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T10:02:17.838 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T10:02:17.839 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T10:02:17.840 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=11, tag_tid=1, entry_tid=8887' 2024-04-06T10:02:17.840 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T10:02:17.840 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T10:02:17.841 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=8887 2024-04-06T10:02:17.841 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 2 = 1 -a 1 -le 8887 2024-04-06T10:02:17.841 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T10:02:17.842 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T10:02:17.842 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T10:02:18.043 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror image_19 2024-04-06T10:02:18.044 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T10:02:18.044 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:02:18.044 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_19 2024-04-06T10:02:18.044 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T10:02:18.044 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n image_19 ']' 2024-04-06T10:02:18.044 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/image_19' 2024-04-06T10:02:18.044 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_19 2024-04-06T10:02:18.044 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T10:02:18.044 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T10:02:18.044 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T10:02:18.044 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T10:02:18.044 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T10:02:18.044 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T10:02:18.044 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T10:02:18.045 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T10:02:18.045 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T10:02:18.049 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T10:02:18.049 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_19 2024-04-06T10:02:18.049 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T10:02:18.049 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T10:02:18.050 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T10:02:18.050 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T10:02:18.050 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T10:02:18.050 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T10:02:18.050 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T10:02:18.050 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T10:02:18.050 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T10:02:18.050 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T10:02:18.050 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T10:02:18.050 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:02:18.050 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T10:02:18.050 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T10:02:18.050 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T10:02:18.050 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T10:02:18.051 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T10:02:18.051 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T10:02:18.051 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T10:02:18.051 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T10:02:18.051 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T10:02:18.051 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T10:02:18.051 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:02:18.051 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T10:02:18.051 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T10:02:18.051 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T10:02:18.051 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T10:02:18.051 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T10:02:18.051 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T10:02:18.051 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T10:02:18.074 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T10:02:18.075 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T10:02:18.075 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T10:02:18.075 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_19 2024-04-06T10:02:18.181 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:Invalid command: unused arguments: ['mirror/image_19'] 2024-04-06T10:02:18.181 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:rbd mirror flush : flush rbd mirror 2024-04-06T10:02:18.182 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:admin_socket: invalid command 2024-04-06T10:02:18.184 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T10:02:18.184 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/image_19 2024-04-06T10:02:18.184 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:1 2024-04-06T10:02:18.184 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T10:02:18.184 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T10:02:18.184 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-06T10:02:18.184 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:1 2024-04-06T10:02:18.184 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T10:02:18.184 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T10:02:18.184 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T10:02:18.184 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=1 2024-04-06T10:02:18.185 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-06T10:02:18.185 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T10:02:18.185 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:02:18.185 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=1 2024-04-06T10:02:18.185 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=1 2024-04-06T10:02:18.186 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-06T10:02:18.186 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:1 2024-04-06T10:02:18.186 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T10:02:18.186 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T10:02:18.186 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-06T10:02:18.186 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:1 2024-04-06T10:02:18.186 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T10:02:18.187 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T10:02:18.187 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:02:18.187 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=1 2024-04-06T10:02:18.187 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-06T10:02:18.187 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T10:02:18.187 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T10:02:18.187 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=1 2024-04-06T10:02:18.187 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=1 2024-04-06T10:02:18.187 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-06T10:02:18.205 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T10:02:18.205 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T10:02:18.205 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T10:02:18.206 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/image_19 2024-04-06T10:02:18.346 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T10:02:18.347 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror image_19 2024-04-06T10:02:18.348 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:02:18.348 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:02:18.348 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_19 2024-04-06T10:02:18.348 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror image_19 '' 2024-04-06T10:02:18.348 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:02:18.348 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:02:18.348 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_19 2024-04-06T10:02:18.348 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T10:02:18.349 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-image_19.status 2024-04-06T10:02:18.349 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T10:02:18.349 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-image_19.status 2024-04-06T10:02:18.350 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-image_19.status 2024-04-06T10:02:18.350 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_19 2024-04-06T10:02:18.350 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_19.status 2024-04-06T10:02:18.396 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 2 2024-04-06T10:02:18.396 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 2 2024-04-06T10:02:18.396 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T10:02:18.397 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=9, tag_tid=2, entry_tid=1], [object_number=8, tag_tid=2, entry_tid=0], [object_number=11, tag_tid=1, entry_tid=8887], [object_number=10, tag_tid=1, entry_tid=8886]]], state=connected] 2024-04-06T10:02:18.397 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=11, tag_tid=1, entry_tid=8887], [object_number=10, tag_tid=1, entry_tid=8886], [object_number=9, tag_tid=1, entry_tid=8885], [object_number=8, tag_tid=1, entry_tid=8884]]], state=connected] 2024-04-06T10:02:18.400 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_19.status 2024-04-06T10:02:18.401 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=9, tag_tid=2, entry_tid=1' 2024-04-06T10:02:18.402 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror image_19 2024-04-06T10:02:18.402 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:02:18.402 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:02:18.403 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_19 2024-04-06T10:02:18.403 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror image_19 '..*' 2024-04-06T10:02:18.403 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:02:18.403 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:02:18.403 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_19 2024-04-06T10:02:18.403 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T10:02:18.403 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-image_19.status 2024-04-06T10:02:18.403 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-image_19.status 2024-04-06T10:02:18.403 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T10:02:18.404 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-image_19.status 2024-04-06T10:02:18.404 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_19 2024-04-06T10:02:18.404 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_19.status 2024-04-06T10:02:18.448 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 2 2024-04-06T10:02:18.449 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 2 2024-04-06T10:02:18.449 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T10:02:18.449 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=9, tag_tid=2, entry_tid=1], [object_number=8, tag_tid=2, entry_tid=0], [object_number=11, tag_tid=1, entry_tid=8887], [object_number=10, tag_tid=1, entry_tid=8886]]], state=connected] 2024-04-06T10:02:18.449 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=11, tag_tid=1, entry_tid=8887], [object_number=10, tag_tid=1, entry_tid=8886], [object_number=9, tag_tid=1, entry_tid=8885], [object_number=8, tag_tid=1, entry_tid=8884]]], state=connected] 2024-04-06T10:02:18.452 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_19.status 2024-04-06T10:02:18.454 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=11, tag_tid=1, entry_tid=8887' 2024-04-06T10:02:18.454 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=9, tag_tid=2, entry_tid=1' -a 'object_number=9, tag_tid=2, entry_tid=1' = 'object_number=11, tag_tid=1, entry_tid=8887' 2024-04-06T10:02:18.454 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=11, tag_tid=1, entry_tid=8887' '!=' 'object_number=11, tag_tid=1, entry_tid=8887' 2024-04-06T10:02:18.454 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T10:02:18.454 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.4 2024-04-06T10:02:18.857 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror image_19 2024-04-06T10:02:18.857 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T10:02:18.857 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:02:18.857 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_19 2024-04-06T10:02:18.857 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T10:02:18.857 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n image_19 ']' 2024-04-06T10:02:18.857 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/image_19' 2024-04-06T10:02:18.857 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_19 2024-04-06T10:02:18.857 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T10:02:18.857 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T10:02:18.858 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T10:02:18.858 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T10:02:18.858 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T10:02:18.858 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T10:02:18.858 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T10:02:18.859 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T10:02:18.859 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T10:02:18.860 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T10:02:18.860 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_19 2024-04-06T10:02:18.860 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T10:02:18.860 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T10:02:18.860 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T10:02:18.861 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T10:02:18.861 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T10:02:18.861 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T10:02:18.861 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T10:02:18.861 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T10:02:18.861 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T10:02:18.861 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T10:02:18.861 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T10:02:18.862 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:02:18.862 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T10:02:18.862 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T10:02:18.863 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T10:02:18.863 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T10:02:18.863 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T10:02:18.863 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T10:02:18.863 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T10:02:18.863 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T10:02:18.863 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T10:02:18.863 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T10:02:18.863 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:02:18.863 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T10:02:18.863 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T10:02:18.863 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T10:02:18.864 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T10:02:18.864 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T10:02:18.864 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T10:02:18.864 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T10:02:18.887 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T10:02:18.887 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T10:02:18.887 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T10:02:18.887 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_19 2024-04-06T10:02:18.999 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:Invalid command: unused arguments: ['mirror/image_19'] 2024-04-06T10:02:18.999 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:rbd mirror flush : flush rbd mirror 2024-04-06T10:02:18.999 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:admin_socket: invalid command 2024-04-06T10:02:19.001 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T10:02:19.001 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/image_19 2024-04-06T10:02:19.001 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:1 2024-04-06T10:02:19.001 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T10:02:19.001 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T10:02:19.002 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-06T10:02:19.002 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:1 2024-04-06T10:02:19.002 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T10:02:19.002 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T10:02:19.002 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T10:02:19.002 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=1 2024-04-06T10:02:19.002 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-06T10:02:19.002 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T10:02:19.002 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:02:19.002 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=1 2024-04-06T10:02:19.002 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=1 2024-04-06T10:02:19.003 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-06T10:02:19.004 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:1 2024-04-06T10:02:19.004 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T10:02:19.004 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T10:02:19.004 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-06T10:02:19.004 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:1 2024-04-06T10:02:19.004 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T10:02:19.004 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T10:02:19.004 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:02:19.004 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=1 2024-04-06T10:02:19.004 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-06T10:02:19.004 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T10:02:19.004 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T10:02:19.004 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=1 2024-04-06T10:02:19.004 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=1 2024-04-06T10:02:19.004 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-06T10:02:19.027 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T10:02:19.028 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T10:02:19.028 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T10:02:19.028 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/image_19 2024-04-06T10:02:19.164 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T10:02:19.165 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror image_19 2024-04-06T10:02:19.165 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:02:19.165 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:02:19.165 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_19 2024-04-06T10:02:19.165 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror image_19 '' 2024-04-06T10:02:19.165 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:02:19.165 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:02:19.165 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_19 2024-04-06T10:02:19.166 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T10:02:19.166 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-image_19.status 2024-04-06T10:02:19.166 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-image_19.status 2024-04-06T10:02:19.166 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T10:02:19.168 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-image_19.status 2024-04-06T10:02:19.169 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_19 2024-04-06T10:02:19.169 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_19.status 2024-04-06T10:02:19.220 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 2 2024-04-06T10:02:19.220 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 2 2024-04-06T10:02:19.220 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T10:02:19.220 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=9, tag_tid=2, entry_tid=1], [object_number=8, tag_tid=2, entry_tid=0], [object_number=11, tag_tid=1, entry_tid=8887], [object_number=10, tag_tid=1, entry_tid=8886]]], state=connected] 2024-04-06T10:02:19.220 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=11, tag_tid=1, entry_tid=8887], [object_number=10, tag_tid=1, entry_tid=8886], [object_number=9, tag_tid=1, entry_tid=8885], [object_number=8, tag_tid=1, entry_tid=8884]]], state=connected] 2024-04-06T10:02:19.223 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_19.status 2024-04-06T10:02:19.224 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=9, tag_tid=2, entry_tid=1' 2024-04-06T10:02:19.226 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror image_19 2024-04-06T10:02:19.226 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:02:19.226 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:02:19.226 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_19 2024-04-06T10:02:19.226 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror image_19 '..*' 2024-04-06T10:02:19.226 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:02:19.226 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:02:19.226 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_19 2024-04-06T10:02:19.226 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T10:02:19.226 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-image_19.status 2024-04-06T10:02:19.226 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-image_19.status 2024-04-06T10:02:19.226 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T10:02:19.227 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-image_19.status 2024-04-06T10:02:19.227 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_19 2024-04-06T10:02:19.227 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_19.status 2024-04-06T10:02:19.272 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 2 2024-04-06T10:02:19.272 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 2 2024-04-06T10:02:19.272 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T10:02:19.272 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=9, tag_tid=2, entry_tid=1], [object_number=8, tag_tid=2, entry_tid=0], [object_number=11, tag_tid=1, entry_tid=8887], [object_number=10, tag_tid=1, entry_tid=8886]]], state=connected] 2024-04-06T10:02:19.272 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=11, tag_tid=1, entry_tid=8887], [object_number=10, tag_tid=1, entry_tid=8886], [object_number=9, tag_tid=1, entry_tid=8885], [object_number=8, tag_tid=1, entry_tid=8884]]], state=connected] 2024-04-06T10:02:19.276 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_19.status 2024-04-06T10:02:19.277 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=11, tag_tid=1, entry_tid=8887' 2024-04-06T10:02:19.277 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=9, tag_tid=2, entry_tid=1' -a 'object_number=9, tag_tid=2, entry_tid=1' = 'object_number=11, tag_tid=1, entry_tid=8887' 2024-04-06T10:02:19.277 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=11, tag_tid=1, entry_tid=8887' '!=' 'object_number=11, tag_tid=1, entry_tid=8887' 2024-04-06T10:02:19.278 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T10:02:19.278 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.8 2024-04-06T10:02:20.079 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror image_19 2024-04-06T10:02:20.079 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T10:02:20.079 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:02:20.080 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_19 2024-04-06T10:02:20.080 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T10:02:20.080 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n image_19 ']' 2024-04-06T10:02:20.080 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/image_19' 2024-04-06T10:02:20.080 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_19 2024-04-06T10:02:20.080 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T10:02:20.080 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T10:02:20.080 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T10:02:20.080 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T10:02:20.080 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T10:02:20.080 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T10:02:20.080 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T10:02:20.081 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T10:02:20.082 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T10:02:20.084 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T10:02:20.084 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_19 2024-04-06T10:02:20.084 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T10:02:20.084 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T10:02:20.084 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T10:02:20.085 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T10:02:20.085 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T10:02:20.085 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T10:02:20.085 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T10:02:20.085 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T10:02:20.085 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T10:02:20.085 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T10:02:20.085 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T10:02:20.085 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:02:20.085 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T10:02:20.085 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T10:02:20.086 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T10:02:20.086 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T10:02:20.086 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T10:02:20.086 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T10:02:20.087 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T10:02:20.087 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T10:02:20.087 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T10:02:20.087 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T10:02:20.087 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:02:20.087 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T10:02:20.087 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T10:02:20.087 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T10:02:20.087 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T10:02:20.087 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T10:02:20.087 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T10:02:20.087 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T10:02:20.110 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T10:02:20.111 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T10:02:20.111 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T10:02:20.111 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_19 2024-04-06T10:02:20.224 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:Invalid command: unused arguments: ['mirror/image_19'] 2024-04-06T10:02:20.224 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:rbd mirror flush : flush rbd mirror 2024-04-06T10:02:20.225 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:admin_socket: invalid command 2024-04-06T10:02:20.227 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T10:02:20.227 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/image_19 2024-04-06T10:02:20.227 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:1 2024-04-06T10:02:20.227 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T10:02:20.227 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T10:02:20.227 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-06T10:02:20.227 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:1 2024-04-06T10:02:20.227 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T10:02:20.227 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T10:02:20.227 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T10:02:20.227 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=1 2024-04-06T10:02:20.228 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-06T10:02:20.228 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T10:02:20.228 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:02:20.228 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=1 2024-04-06T10:02:20.228 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=1 2024-04-06T10:02:20.229 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-06T10:02:20.229 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:1 2024-04-06T10:02:20.229 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T10:02:20.229 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T10:02:20.229 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-06T10:02:20.229 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:1 2024-04-06T10:02:20.229 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T10:02:20.230 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T10:02:20.230 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:02:20.230 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=1 2024-04-06T10:02:20.230 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-06T10:02:20.230 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T10:02:20.230 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T10:02:20.230 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=1 2024-04-06T10:02:20.230 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=1 2024-04-06T10:02:20.230 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-06T10:02:20.254 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T10:02:20.254 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T10:02:20.254 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T10:02:20.254 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/image_19 2024-04-06T10:02:20.398 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T10:02:20.399 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror image_19 2024-04-06T10:02:20.399 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:02:20.399 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:02:20.399 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_19 2024-04-06T10:02:20.399 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror image_19 '' 2024-04-06T10:02:20.399 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:02:20.399 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:02:20.399 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_19 2024-04-06T10:02:20.399 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T10:02:20.400 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-image_19.status 2024-04-06T10:02:20.400 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-image_19.status 2024-04-06T10:02:20.400 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T10:02:20.402 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-image_19.status 2024-04-06T10:02:20.402 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_19 2024-04-06T10:02:20.402 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_19.status 2024-04-06T10:02:20.447 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 2 2024-04-06T10:02:20.447 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 2 2024-04-06T10:02:20.447 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T10:02:20.447 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=9, tag_tid=2, entry_tid=1], [object_number=8, tag_tid=2, entry_tid=0], [object_number=11, tag_tid=1, entry_tid=8887], [object_number=10, tag_tid=1, entry_tid=8886]]], state=connected] 2024-04-06T10:02:20.447 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=9, tag_tid=2, entry_tid=1], [object_number=8, tag_tid=2, entry_tid=0], [object_number=11, tag_tid=1, entry_tid=8887], [object_number=10, tag_tid=1, entry_tid=8886]]], state=connected] 2024-04-06T10:02:20.450 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_19.status 2024-04-06T10:02:20.452 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=9, tag_tid=2, entry_tid=1' 2024-04-06T10:02:20.453 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror image_19 2024-04-06T10:02:20.453 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:02:20.453 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:02:20.453 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_19 2024-04-06T10:02:20.453 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror image_19 '..*' 2024-04-06T10:02:20.453 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:02:20.453 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:02:20.454 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_19 2024-04-06T10:02:20.454 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T10:02:20.454 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-image_19.status 2024-04-06T10:02:20.454 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-image_19.status 2024-04-06T10:02:20.454 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T10:02:20.454 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-image_19.status 2024-04-06T10:02:20.455 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_19 2024-04-06T10:02:20.455 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_19.status 2024-04-06T10:02:20.499 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 2 2024-04-06T10:02:20.499 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 2 2024-04-06T10:02:20.500 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T10:02:20.500 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=9, tag_tid=2, entry_tid=1], [object_number=8, tag_tid=2, entry_tid=0], [object_number=11, tag_tid=1, entry_tid=8887], [object_number=10, tag_tid=1, entry_tid=8886]]], state=connected] 2024-04-06T10:02:20.500 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=9, tag_tid=2, entry_tid=1], [object_number=8, tag_tid=2, entry_tid=0], [object_number=11, tag_tid=1, entry_tid=8887], [object_number=10, tag_tid=1, entry_tid=8886]]], state=connected] 2024-04-06T10:02:20.503 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_19.status 2024-04-06T10:02:20.505 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=9, tag_tid=2, entry_tid=1' 2024-04-06T10:02:20.505 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=9, tag_tid=2, entry_tid=1' -a 'object_number=9, tag_tid=2, entry_tid=1' = 'object_number=9, tag_tid=2, entry_tid=1' 2024-04-06T10:02:20.505 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T10:02:20.505 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_snap_present cluster1 mirror image_19 snap 2024-04-06T10:02:20.505 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T10:02:20.505 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:02:20.505 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_19 2024-04-06T10:02:20.505 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local snap_name=snap 2024-04-06T10:02:20.505 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local s 2024-04-06T10:02:20.505 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-04-06T10:02:20.505 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 1 2024-04-06T10:02:21.506 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 info mirror/image_19@snap 2024-04-06T10:02:21.574 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:rbd image 'image_19': 2024-04-06T10:02:21.574 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: size 128 MiB in 32 objects 2024-04-06T10:02:21.574 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: order 22 (4 MiB objects) 2024-04-06T10:02:21.574 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: snapshot_count: 1 2024-04-06T10:02:21.574 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: id: 10ac2917db2a 2024-04-06T10:02:21.574 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: block_name_prefix: rbd_data.10ac2917db2a 2024-04-06T10:02:21.575 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: format: 2 2024-04-06T10:02:21.575 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: features: layering, exclusive-lock, journaling 2024-04-06T10:02:21.575 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: op_features: 2024-04-06T10:02:21.575 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: flags: 2024-04-06T10:02:21.575 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: create_timestamp: Sat Apr 6 09:53:30 2024 2024-04-06T10:02:21.575 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: access_timestamp: Sat Apr 6 09:53:30 2024 2024-04-06T10:02:21.575 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: modify_timestamp: Sat Apr 6 09:54:31 2024 2024-04-06T10:02:21.575 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: protected: False 2024-04-06T10:02:21.575 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: journal: 10ac2917db2a 2024-04-06T10:02:21.576 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: mirroring state: enabled 2024-04-06T10:02:21.576 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: mirroring mode: journal 2024-04-06T10:02:21.576 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: mirroring global id: ba53e426-2f84-40c7-8148-fb24cc38cce2 2024-04-06T10:02:21.576 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: mirroring primary: false 2024-04-06T10:02:21.579 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T10:02:21.579 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ compare_image_snaps mirror image_19 snap 2024-04-06T10:02:21.580 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:02:21.580 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_19 2024-04-06T10:02:21.580 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local snap_name=snap 2024-04-06T10:02:21.580 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local ret=0 2024-04-06T10:02:21.580 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local rmt_export=/tmp/tmp.BhW6xKmROU/cluster2-mirror-image_19.export 2024-04-06T10:02:21.581 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loc_export=/tmp/tmp.BhW6xKmROU/cluster1-mirror-image_19.export 2024-04-06T10:02:21.581 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rm -f /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_19.export /tmp/tmp.BhW6xKmROU/cluster1-mirror-image_19.export 2024-04-06T10:02:21.581 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster2 -p mirror export image_19@snap /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_19.export 2024-04-06T10:02:21.745 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-04-06T10:02:21.753 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 -p mirror export image_19@snap /tmp/tmp.BhW6xKmROU/cluster1-mirror-image_19.export 2024-04-06T10:02:22.177 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-04-06T10:02:22.190 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmp /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_19.export /tmp/tmp.BhW6xKmROU/cluster1-mirror-image_19.export 2024-04-06T10:02:22.720 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rm -f /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_19.export /tmp/tmp.BhW6xKmROU/cluster1-mirror-image_19.export 2024-04-06T10:02:22.761 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T10:02:22.761 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-06T10:02:22.761 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ image=image_20 2024-04-06T10:02:22.761 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ create_snap cluster2 mirror image_20 snap 2024-04-06T10:02:22.761 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-04-06T10:02:22.761 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:02:22.761 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_20 2024-04-06T10:02:22.761 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local snap_name=snap 2024-04-06T10:02:22.761 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster2 -p mirror snap create image_20@snap --debug-rbd=20 --debug-journaler=20 2024-04-06T10:02:24.869 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_image_replay_started cluster1 mirror image_20 2024-04-06T10:02:24.869 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T10:02:24.869 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:02:24.870 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_20 2024-04-06T10:02:24.870 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_image_replay_state cluster1 mirror image_20 started 2024-04-06T10:02:24.870 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T10:02:24.870 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:02:24.870 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_20 2024-04-06T10:02:24.870 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state=started 2024-04-06T10:02:24.870 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local s 2024-04-06T10:02:24.870 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T10:02:24.870 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 1 2024-04-06T10:02:25.871 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test_image_replay_state cluster1 mirror image_20 started 2024-04-06T10:02:25.872 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T10:02:25.872 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:02:25.872 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_20 2024-04-06T10:02:25.872 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local test_state=started 2024-04-06T10:02:25.872 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local status_result 2024-04-06T10:02:25.872 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local current_state=stopped 2024-04-06T10:02:25.873 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_20 2024-04-06T10:02:25.873 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -i state 2024-04-06T10:02:25.873 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_instance=cluster1 2024-04-06T10:02:25.873 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ shift 2024-04-06T10:02:25.873 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T10:02:25.873 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance=cluster1 2024-04-06T10:02:25.873 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local loop_instance 2024-04-06T10:02:25.874 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T10:02:25.874 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sleep 0 2024-04-06T10:02:25.874 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-04-06T10:02:25.875 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ seq 0 3 2024-04-06T10:02:25.878 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T10:02:25.878 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_20 2024-04-06T10:02:25.878 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1:0 2024-04-06T10:02:25.878 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ shift 2024-04-06T10:02:25.878 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T10:02:25.878 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-04-06T10:02:25.878 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T10:02:25.878 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=cluster 2024-04-06T10:02:25.878 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T10:02:25.878 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:02:25.878 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T10:02:25.878 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T10:02:25.879 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval cluster=cluster1 2024-04-06T10:02:25.879 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ cluster=cluster1 2024-04-06T10:02:25.879 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T10:02:25.879 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T10:02:25.879 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-04-06T10:02:25.879 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local local_cluster=cluster1:0 2024-04-06T10:02:25.879 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local cluster=cluster1 2024-04-06T10:02:25.879 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local instance 2024-04-06T10:02:25.879 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T10:02:25.879 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local val=cluster1:0 2024-04-06T10:02:25.879 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local cluster_var_name=local_cluster 2024-04-06T10:02:25.879 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local instance_var_name=instance 2024-04-06T10:02:25.880 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ cluster=cluster1 2024-04-06T10:02:25.880 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T10:02:25.880 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ '[' 0 = cluster1:0 ']' 2024-04-06T10:02:25.880 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ eval local_cluster=cluster1 2024-04-06T10:02:25.880 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++++ local_cluster=cluster1 2024-04-06T10:02:25.880 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ eval instance=0 2024-04-06T10:02:25.880 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++++ instance=0 2024-04-06T10:02:25.880 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T10:02:25.904 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T10:02:25.904 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T10:02:25.904 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T10:02:25.905 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_20 2024-04-06T10:02:26.010 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:Invalid command: unused arguments: ['mirror/image_20'] 2024-04-06T10:02:26.011 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:rbd mirror status : get status for rbd mirror 2024-04-06T10:02:26.011 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:admin_socket: invalid command 2024-04-06T10:02:26.013 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T10:02:26.013 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_20 2024-04-06T10:02:26.013 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1:1 2024-04-06T10:02:26.014 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ shift 2024-04-06T10:02:26.014 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T10:02:26.014 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-04-06T10:02:26.014 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:1 2024-04-06T10:02:26.014 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=cluster 2024-04-06T10:02:26.014 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T10:02:26.014 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:02:26.014 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=1 2024-04-06T10:02:26.014 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-06T10:02:26.014 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval cluster=cluster1 2024-04-06T10:02:26.014 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ cluster=cluster1 2024-04-06T10:02:26.014 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=1 2024-04-06T10:02:26.014 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=1 2024-04-06T10:02:26.016 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-04-06T10:02:26.016 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local local_cluster=cluster1:1 2024-04-06T10:02:26.016 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local cluster=cluster1 2024-04-06T10:02:26.016 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local instance 2024-04-06T10:02:26.016 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-06T10:02:26.016 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local val=cluster1:1 2024-04-06T10:02:26.016 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local cluster_var_name=local_cluster 2024-04-06T10:02:26.016 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local instance_var_name=instance 2024-04-06T10:02:26.017 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ cluster=cluster1 2024-04-06T10:02:26.017 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=1 2024-04-06T10:02:26.017 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ '[' 1 = cluster1:1 ']' 2024-04-06T10:02:26.017 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ eval local_cluster=cluster1 2024-04-06T10:02:26.017 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++++ local_cluster=cluster1 2024-04-06T10:02:26.017 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ eval instance=1 2024-04-06T10:02:26.017 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++++ instance=1 2024-04-06T10:02:26.017 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-06T10:02:26.039 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T10:02:26.040 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T10:02:26.040 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T10:02:26.040 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_20 2024-04-06T10:02:26.198 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ return 0 2024-04-06T10:02:26.199 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ status_result=' "state": "Replaying"' 2024-04-06T10:02:26.200 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ echo ' "state": "Replaying"' 2024-04-06T10:02:26.200 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep -i Replaying 2024-04-06T10:02:26.201 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: "state": "Replaying" 2024-04-06T10:02:26.203 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ current_state=started 2024-04-06T10:02:26.203 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test started = started 2024-04-06T10:02:26.203 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T10:02:26.203 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_20 2024-04-06T10:02:26.203 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local local_cluster=cluster1 2024-04-06T10:02:26.203 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-04-06T10:02:26.203 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:02:26.203 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_20 2024-04-06T10:02:26.203 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' journal = journal ']' 2024-04-06T10:02:26.203 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_journal_replay_complete cluster1 cluster2 mirror image_20 2024-04-06T10:02:26.203 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local local_cluster=cluster1 2024-04-06T10:02:26.203 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-04-06T10:02:26.203 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:02:26.204 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_20 2024-04-06T10:02:26.204 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local s master_pos mirror_pos last_mirror_pos 2024-04-06T10:02:26.204 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local master_tag master_entry mirror_tag mirror_entry 2024-04-06T10:02:26.204 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T10:02:26.204 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T10:02:26.204 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T10:02:26.404 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror image_20 2024-04-06T10:02:26.404 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T10:02:26.404 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:02:26.404 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_20 2024-04-06T10:02:26.404 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T10:02:26.404 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n image_20 ']' 2024-04-06T10:02:26.404 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/image_20' 2024-04-06T10:02:26.404 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_20 2024-04-06T10:02:26.405 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T10:02:26.405 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T10:02:26.405 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T10:02:26.405 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T10:02:26.405 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T10:02:26.405 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T10:02:26.405 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T10:02:26.406 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T10:02:26.406 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T10:02:26.408 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T10:02:26.408 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_20 2024-04-06T10:02:26.408 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T10:02:26.408 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T10:02:26.408 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T10:02:26.408 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T10:02:26.408 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T10:02:26.408 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T10:02:26.408 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T10:02:26.409 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T10:02:26.409 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T10:02:26.409 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T10:02:26.409 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T10:02:26.409 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:02:26.409 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T10:02:26.409 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T10:02:26.409 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T10:02:26.409 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T10:02:26.409 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T10:02:26.409 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T10:02:26.409 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T10:02:26.409 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T10:02:26.409 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T10:02:26.409 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T10:02:26.410 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:02:26.410 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T10:02:26.410 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T10:02:26.410 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T10:02:26.410 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T10:02:26.410 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T10:02:26.410 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T10:02:26.410 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T10:02:26.434 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T10:02:26.434 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T10:02:26.434 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T10:02:26.435 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_20 2024-04-06T10:02:26.542 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:Invalid command: unused arguments: ['mirror/image_20'] 2024-04-06T10:02:26.542 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:rbd mirror flush : flush rbd mirror 2024-04-06T10:02:26.542 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:admin_socket: invalid command 2024-04-06T10:02:26.544 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T10:02:26.545 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/image_20 2024-04-06T10:02:26.545 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:1 2024-04-06T10:02:26.545 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T10:02:26.545 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T10:02:26.545 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-06T10:02:26.545 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:1 2024-04-06T10:02:26.545 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T10:02:26.545 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T10:02:26.545 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T10:02:26.545 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=1 2024-04-06T10:02:26.545 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-06T10:02:26.545 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T10:02:26.545 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:02:26.545 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=1 2024-04-06T10:02:26.545 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=1 2024-04-06T10:02:26.546 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-06T10:02:26.546 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:1 2024-04-06T10:02:26.546 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T10:02:26.547 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T10:02:26.547 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-06T10:02:26.547 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:1 2024-04-06T10:02:26.547 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T10:02:26.547 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T10:02:26.547 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:02:26.547 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=1 2024-04-06T10:02:26.547 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-06T10:02:26.547 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T10:02:26.547 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T10:02:26.547 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=1 2024-04-06T10:02:26.547 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=1 2024-04-06T10:02:26.547 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-06T10:02:26.571 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T10:02:26.571 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T10:02:26.572 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T10:02:26.572 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/image_20 2024-04-06T10:02:26.709 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T10:02:26.710 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror image_20 2024-04-06T10:02:26.710 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:02:26.710 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:02:26.710 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_20 2024-04-06T10:02:26.711 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror image_20 '' 2024-04-06T10:02:26.711 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:02:26.711 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:02:26.711 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_20 2024-04-06T10:02:26.711 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T10:02:26.711 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-image_20.status 2024-04-06T10:02:26.711 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-image_20.status 2024-04-06T10:02:26.712 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T10:02:26.714 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-image_20.status 2024-04-06T10:02:26.714 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_20 2024-04-06T10:02:26.714 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_20.status 2024-04-06T10:02:26.764 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 2 2024-04-06T10:02:26.764 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 2 2024-04-06T10:02:26.764 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T10:02:26.764 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=9, tag_tid=2, entry_tid=1], [object_number=8, tag_tid=2, entry_tid=0], [object_number=11, tag_tid=1, entry_tid=9615], [object_number=10, tag_tid=1, entry_tid=9614]]], state=connected] 2024-04-06T10:02:26.764 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=11, tag_tid=1, entry_tid=9615], [object_number=10, tag_tid=1, entry_tid=9614], [object_number=9, tag_tid=1, entry_tid=9613], [object_number=8, tag_tid=1, entry_tid=9612]]], state=connected] 2024-04-06T10:02:26.767 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_20.status 2024-04-06T10:02:26.769 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=9, tag_tid=2, entry_tid=1' 2024-04-06T10:02:26.770 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror image_20 2024-04-06T10:02:26.770 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:02:26.770 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:02:26.770 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_20 2024-04-06T10:02:26.770 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror image_20 '..*' 2024-04-06T10:02:26.770 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:02:26.770 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:02:26.770 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_20 2024-04-06T10:02:26.770 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T10:02:26.770 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-image_20.status 2024-04-06T10:02:26.771 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-image_20.status 2024-04-06T10:02:26.771 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T10:02:26.771 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-image_20.status 2024-04-06T10:02:26.771 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_20 2024-04-06T10:02:26.772 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_20.status 2024-04-06T10:02:26.817 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 2 2024-04-06T10:02:26.818 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 2 2024-04-06T10:02:26.818 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T10:02:26.818 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=9, tag_tid=2, entry_tid=1], [object_number=8, tag_tid=2, entry_tid=0], [object_number=11, tag_tid=1, entry_tid=9615], [object_number=10, tag_tid=1, entry_tid=9614]]], state=connected] 2024-04-06T10:02:26.818 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=11, tag_tid=1, entry_tid=9615], [object_number=10, tag_tid=1, entry_tid=9614], [object_number=9, tag_tid=1, entry_tid=9613], [object_number=8, tag_tid=1, entry_tid=9612]]], state=connected] 2024-04-06T10:02:26.821 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_20.status 2024-04-06T10:02:26.823 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=11, tag_tid=1, entry_tid=9615' 2024-04-06T10:02:26.823 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=9, tag_tid=2, entry_tid=1' -a 'object_number=9, tag_tid=2, entry_tid=1' = 'object_number=11, tag_tid=1, entry_tid=9615' 2024-04-06T10:02:26.823 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=11, tag_tid=1, entry_tid=9615' '!=' '' 2024-04-06T10:02:26.823 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T10:02:26.823 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=11, tag_tid=1, entry_tid=9615' = '' 2024-04-06T10:02:26.823 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=11, tag_tid=1, entry_tid=9615' 2024-04-06T10:02:26.824 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=9, tag_tid=2, entry_tid=1' 2024-04-06T10:02:26.824 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T10:02:26.824 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T10:02:26.825 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=2 2024-04-06T10:02:26.825 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=11, tag_tid=1, entry_tid=9615' 2024-04-06T10:02:26.826 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T10:02:26.826 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T10:02:26.826 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=1 2024-04-06T10:02:26.827 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=9, tag_tid=2, entry_tid=1' 2024-04-06T10:02:26.827 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T10:02:26.827 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T10:02:26.828 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T10:02:26.829 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=11, tag_tid=1, entry_tid=9615' 2024-04-06T10:02:26.829 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T10:02:26.829 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T10:02:26.830 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=9615 2024-04-06T10:02:26.830 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 2 = 1 -a 1 -le 9615 2024-04-06T10:02:26.830 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T10:02:26.830 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T10:02:26.831 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T10:02:27.032 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror image_20 2024-04-06T10:02:27.032 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T10:02:27.032 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:02:27.032 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_20 2024-04-06T10:02:27.033 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T10:02:27.033 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n image_20 ']' 2024-04-06T10:02:27.033 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/image_20' 2024-04-06T10:02:27.033 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_20 2024-04-06T10:02:27.033 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T10:02:27.033 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T10:02:27.033 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T10:02:27.033 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T10:02:27.033 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T10:02:27.033 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T10:02:27.033 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T10:02:27.034 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T10:02:27.034 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T10:02:27.036 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T10:02:27.036 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_20 2024-04-06T10:02:27.036 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T10:02:27.037 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T10:02:27.037 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T10:02:27.037 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T10:02:27.037 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T10:02:27.037 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T10:02:27.037 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T10:02:27.037 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T10:02:27.037 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T10:02:27.037 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T10:02:27.037 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T10:02:27.037 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:02:27.037 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T10:02:27.037 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T10:02:27.039 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T10:02:27.039 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T10:02:27.039 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T10:02:27.039 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T10:02:27.039 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T10:02:27.039 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T10:02:27.039 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T10:02:27.039 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T10:02:27.039 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:02:27.039 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T10:02:27.040 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T10:02:27.040 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T10:02:27.040 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T10:02:27.040 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T10:02:27.040 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T10:02:27.040 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T10:02:27.062 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T10:02:27.063 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T10:02:27.063 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T10:02:27.063 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_20 2024-04-06T10:02:27.172 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:Invalid command: unused arguments: ['mirror/image_20'] 2024-04-06T10:02:27.173 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:rbd mirror flush : flush rbd mirror 2024-04-06T10:02:27.173 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:admin_socket: invalid command 2024-04-06T10:02:27.175 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T10:02:27.175 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/image_20 2024-04-06T10:02:27.175 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:1 2024-04-06T10:02:27.176 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T10:02:27.176 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T10:02:27.176 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-06T10:02:27.176 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:1 2024-04-06T10:02:27.176 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T10:02:27.176 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T10:02:27.176 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T10:02:27.176 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=1 2024-04-06T10:02:27.176 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-06T10:02:27.176 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T10:02:27.176 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:02:27.176 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=1 2024-04-06T10:02:27.176 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=1 2024-04-06T10:02:27.176 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-06T10:02:27.176 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:1 2024-04-06T10:02:27.177 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T10:02:27.177 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T10:02:27.177 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-06T10:02:27.177 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:1 2024-04-06T10:02:27.177 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T10:02:27.177 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T10:02:27.177 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:02:27.177 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=1 2024-04-06T10:02:27.177 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-06T10:02:27.177 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T10:02:27.177 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T10:02:27.177 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=1 2024-04-06T10:02:27.177 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=1 2024-04-06T10:02:27.178 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-06T10:02:27.200 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T10:02:27.200 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T10:02:27.200 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T10:02:27.200 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/image_20 2024-04-06T10:02:27.337 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T10:02:27.338 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror image_20 2024-04-06T10:02:27.338 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:02:27.338 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:02:27.338 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_20 2024-04-06T10:02:27.338 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror image_20 '' 2024-04-06T10:02:27.339 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:02:27.339 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:02:27.339 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_20 2024-04-06T10:02:27.339 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T10:02:27.339 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-image_20.status 2024-04-06T10:02:27.339 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-image_20.status 2024-04-06T10:02:27.339 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T10:02:27.341 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-image_20.status 2024-04-06T10:02:27.342 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_20 2024-04-06T10:02:27.342 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_20.status 2024-04-06T10:02:27.387 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 2 2024-04-06T10:02:27.387 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 2 2024-04-06T10:02:27.387 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T10:02:27.387 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=9, tag_tid=2, entry_tid=1], [object_number=8, tag_tid=2, entry_tid=0], [object_number=11, tag_tid=1, entry_tid=9615], [object_number=10, tag_tid=1, entry_tid=9614]]], state=connected] 2024-04-06T10:02:27.388 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=11, tag_tid=1, entry_tid=9615], [object_number=10, tag_tid=1, entry_tid=9614], [object_number=9, tag_tid=1, entry_tid=9613], [object_number=8, tag_tid=1, entry_tid=9612]]], state=connected] 2024-04-06T10:02:27.391 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_20.status 2024-04-06T10:02:27.392 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=9, tag_tid=2, entry_tid=1' 2024-04-06T10:02:27.393 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror image_20 2024-04-06T10:02:27.393 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:02:27.393 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:02:27.393 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_20 2024-04-06T10:02:27.394 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror image_20 '..*' 2024-04-06T10:02:27.394 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:02:27.394 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:02:27.394 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_20 2024-04-06T10:02:27.394 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T10:02:27.394 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-image_20.status 2024-04-06T10:02:27.394 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-image_20.status 2024-04-06T10:02:27.394 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T10:02:27.395 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-image_20.status 2024-04-06T10:02:27.395 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_20 2024-04-06T10:02:27.395 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_20.status 2024-04-06T10:02:27.438 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 2 2024-04-06T10:02:27.438 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 2 2024-04-06T10:02:27.438 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T10:02:27.438 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=9, tag_tid=2, entry_tid=1], [object_number=8, tag_tid=2, entry_tid=0], [object_number=11, tag_tid=1, entry_tid=9615], [object_number=10, tag_tid=1, entry_tid=9614]]], state=connected] 2024-04-06T10:02:27.438 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=11, tag_tid=1, entry_tid=9615], [object_number=10, tag_tid=1, entry_tid=9614], [object_number=9, tag_tid=1, entry_tid=9613], [object_number=8, tag_tid=1, entry_tid=9612]]], state=connected] 2024-04-06T10:02:27.442 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_20.status 2024-04-06T10:02:27.443 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=11, tag_tid=1, entry_tid=9615' 2024-04-06T10:02:27.443 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=9, tag_tid=2, entry_tid=1' -a 'object_number=9, tag_tid=2, entry_tid=1' = 'object_number=11, tag_tid=1, entry_tid=9615' 2024-04-06T10:02:27.443 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=11, tag_tid=1, entry_tid=9615' '!=' 'object_number=11, tag_tid=1, entry_tid=9615' 2024-04-06T10:02:27.443 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T10:02:27.443 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.4 2024-04-06T10:02:27.845 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror image_20 2024-04-06T10:02:27.845 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T10:02:27.845 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:02:27.846 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_20 2024-04-06T10:02:27.846 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T10:02:27.846 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n image_20 ']' 2024-04-06T10:02:27.846 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/image_20' 2024-04-06T10:02:27.846 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_20 2024-04-06T10:02:27.846 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T10:02:27.846 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T10:02:27.846 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T10:02:27.846 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T10:02:27.846 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T10:02:27.846 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T10:02:27.846 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T10:02:27.847 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T10:02:27.847 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T10:02:27.851 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T10:02:27.852 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_20 2024-04-06T10:02:27.852 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T10:02:27.852 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T10:02:27.852 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T10:02:27.852 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T10:02:27.852 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T10:02:27.852 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T10:02:27.852 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T10:02:27.852 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T10:02:27.852 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T10:02:27.852 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T10:02:27.852 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T10:02:27.852 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:02:27.852 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T10:02:27.852 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T10:02:27.853 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T10:02:27.853 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T10:02:27.853 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T10:02:27.853 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T10:02:27.853 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T10:02:27.853 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T10:02:27.853 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T10:02:27.853 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T10:02:27.853 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:02:27.853 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T10:02:27.853 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T10:02:27.853 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T10:02:27.853 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T10:02:27.853 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T10:02:27.854 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T10:02:27.854 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T10:02:27.876 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T10:02:27.877 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T10:02:27.877 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T10:02:27.877 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_20 2024-04-06T10:02:27.981 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:Invalid command: unused arguments: ['mirror/image_20'] 2024-04-06T10:02:27.981 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:rbd mirror flush : flush rbd mirror 2024-04-06T10:02:27.981 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:admin_socket: invalid command 2024-04-06T10:02:27.983 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T10:02:27.983 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/image_20 2024-04-06T10:02:27.983 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:1 2024-04-06T10:02:27.983 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T10:02:27.984 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T10:02:27.984 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-06T10:02:27.984 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:1 2024-04-06T10:02:27.984 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T10:02:27.984 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T10:02:27.984 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T10:02:27.984 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=1 2024-04-06T10:02:27.984 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-06T10:02:27.984 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T10:02:27.984 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:02:27.984 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=1 2024-04-06T10:02:27.984 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=1 2024-04-06T10:02:27.986 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-06T10:02:27.986 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:1 2024-04-06T10:02:27.986 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T10:02:27.986 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T10:02:27.987 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-06T10:02:27.987 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:1 2024-04-06T10:02:27.987 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T10:02:27.987 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T10:02:27.987 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:02:27.987 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=1 2024-04-06T10:02:27.987 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-06T10:02:27.987 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T10:02:27.987 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T10:02:27.987 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=1 2024-04-06T10:02:27.987 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=1 2024-04-06T10:02:27.987 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-06T10:02:28.004 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T10:02:28.004 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T10:02:28.005 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T10:02:28.005 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/image_20 2024-04-06T10:02:28.140 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T10:02:28.141 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror image_20 2024-04-06T10:02:28.141 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:02:28.141 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:02:28.141 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_20 2024-04-06T10:02:28.141 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror image_20 '' 2024-04-06T10:02:28.142 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:02:28.142 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:02:28.142 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_20 2024-04-06T10:02:28.142 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T10:02:28.142 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-image_20.status 2024-04-06T10:02:28.142 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T10:02:28.142 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-image_20.status 2024-04-06T10:02:28.144 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-image_20.status 2024-04-06T10:02:28.144 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_20 2024-04-06T10:02:28.144 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_20.status 2024-04-06T10:02:28.193 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 2 2024-04-06T10:02:28.194 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 2 2024-04-06T10:02:28.194 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T10:02:28.194 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=9, tag_tid=2, entry_tid=1], [object_number=8, tag_tid=2, entry_tid=0], [object_number=11, tag_tid=1, entry_tid=9615], [object_number=10, tag_tid=1, entry_tid=9614]]], state=connected] 2024-04-06T10:02:28.194 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=11, tag_tid=1, entry_tid=9615], [object_number=10, tag_tid=1, entry_tid=9614], [object_number=9, tag_tid=1, entry_tid=9613], [object_number=8, tag_tid=1, entry_tid=9612]]], state=connected] 2024-04-06T10:02:28.197 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_20.status 2024-04-06T10:02:28.199 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=9, tag_tid=2, entry_tid=1' 2024-04-06T10:02:28.200 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror image_20 2024-04-06T10:02:28.200 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:02:28.200 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:02:28.200 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_20 2024-04-06T10:02:28.200 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror image_20 '..*' 2024-04-06T10:02:28.200 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:02:28.200 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:02:28.200 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_20 2024-04-06T10:02:28.200 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T10:02:28.201 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-image_20.status 2024-04-06T10:02:28.201 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-image_20.status 2024-04-06T10:02:28.201 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T10:02:28.201 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-image_20.status 2024-04-06T10:02:28.202 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_20 2024-04-06T10:02:28.202 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_20.status 2024-04-06T10:02:28.246 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 2 2024-04-06T10:02:28.247 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 2 2024-04-06T10:02:28.247 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T10:02:28.247 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=9, tag_tid=2, entry_tid=1], [object_number=8, tag_tid=2, entry_tid=0], [object_number=11, tag_tid=1, entry_tid=9615], [object_number=10, tag_tid=1, entry_tid=9614]]], state=connected] 2024-04-06T10:02:28.247 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=11, tag_tid=1, entry_tid=9615], [object_number=10, tag_tid=1, entry_tid=9614], [object_number=9, tag_tid=1, entry_tid=9613], [object_number=8, tag_tid=1, entry_tid=9612]]], state=connected] 2024-04-06T10:02:28.250 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_20.status 2024-04-06T10:02:28.252 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=11, tag_tid=1, entry_tid=9615' 2024-04-06T10:02:28.252 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=9, tag_tid=2, entry_tid=1' -a 'object_number=9, tag_tid=2, entry_tid=1' = 'object_number=11, tag_tid=1, entry_tid=9615' 2024-04-06T10:02:28.252 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=11, tag_tid=1, entry_tid=9615' '!=' 'object_number=11, tag_tid=1, entry_tid=9615' 2024-04-06T10:02:28.252 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T10:02:28.252 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.8 2024-04-06T10:02:29.054 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror image_20 2024-04-06T10:02:29.055 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T10:02:29.055 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:02:29.055 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_20 2024-04-06T10:02:29.055 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T10:02:29.055 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n image_20 ']' 2024-04-06T10:02:29.055 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/image_20' 2024-04-06T10:02:29.055 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_20 2024-04-06T10:02:29.055 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T10:02:29.055 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T10:02:29.055 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T10:02:29.055 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T10:02:29.055 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T10:02:29.056 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T10:02:29.056 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T10:02:29.056 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T10:02:29.056 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T10:02:29.058 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T10:02:29.058 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_20 2024-04-06T10:02:29.058 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T10:02:29.058 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T10:02:29.059 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T10:02:29.059 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T10:02:29.059 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T10:02:29.059 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T10:02:29.059 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T10:02:29.059 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T10:02:29.059 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T10:02:29.059 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T10:02:29.059 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T10:02:29.059 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:02:29.059 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T10:02:29.059 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T10:02:29.060 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T10:02:29.061 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T10:02:29.061 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T10:02:29.061 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T10:02:29.061 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T10:02:29.061 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T10:02:29.062 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T10:02:29.062 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T10:02:29.062 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:02:29.062 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T10:02:29.062 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T10:02:29.062 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T10:02:29.062 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T10:02:29.062 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T10:02:29.062 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T10:02:29.062 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T10:02:29.084 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T10:02:29.085 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T10:02:29.085 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T10:02:29.085 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_20 2024-04-06T10:02:29.191 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:Invalid command: unused arguments: ['mirror/image_20'] 2024-04-06T10:02:29.191 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:rbd mirror flush : flush rbd mirror 2024-04-06T10:02:29.191 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:admin_socket: invalid command 2024-04-06T10:02:29.194 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T10:02:29.194 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/image_20 2024-04-06T10:02:29.194 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:1 2024-04-06T10:02:29.195 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T10:02:29.195 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T10:02:29.195 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-06T10:02:29.195 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:1 2024-04-06T10:02:29.195 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T10:02:29.195 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T10:02:29.195 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T10:02:29.195 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=1 2024-04-06T10:02:29.195 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-06T10:02:29.195 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T10:02:29.195 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:02:29.195 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=1 2024-04-06T10:02:29.195 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=1 2024-04-06T10:02:29.196 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-06T10:02:29.196 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:1 2024-04-06T10:02:29.196 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T10:02:29.196 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T10:02:29.196 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-06T10:02:29.196 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:1 2024-04-06T10:02:29.196 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T10:02:29.196 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T10:02:29.196 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:02:29.196 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=1 2024-04-06T10:02:29.196 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-06T10:02:29.196 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T10:02:29.196 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T10:02:29.197 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=1 2024-04-06T10:02:29.197 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=1 2024-04-06T10:02:29.197 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-06T10:02:29.219 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T10:02:29.219 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T10:02:29.220 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T10:02:29.220 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/image_20 2024-04-06T10:02:29.362 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T10:02:29.363 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror image_20 2024-04-06T10:02:29.363 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:02:29.363 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:02:29.363 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_20 2024-04-06T10:02:29.363 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror image_20 '' 2024-04-06T10:02:29.363 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:02:29.364 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:02:29.364 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_20 2024-04-06T10:02:29.364 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T10:02:29.364 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-image_20.status 2024-04-06T10:02:29.364 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-image_20.status 2024-04-06T10:02:29.364 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T10:02:29.366 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-image_20.status 2024-04-06T10:02:29.366 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_20 2024-04-06T10:02:29.367 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_20.status 2024-04-06T10:02:29.418 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 2 2024-04-06T10:02:29.418 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 2 2024-04-06T10:02:29.418 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T10:02:29.418 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=9, tag_tid=2, entry_tid=1], [object_number=8, tag_tid=2, entry_tid=0], [object_number=11, tag_tid=1, entry_tid=9615], [object_number=10, tag_tid=1, entry_tid=9614]]], state=connected] 2024-04-06T10:02:29.419 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=11, tag_tid=1, entry_tid=9615], [object_number=10, tag_tid=1, entry_tid=9614], [object_number=9, tag_tid=1, entry_tid=9613], [object_number=8, tag_tid=1, entry_tid=9612]]], state=connected] 2024-04-06T10:02:29.422 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_20.status 2024-04-06T10:02:29.423 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=9, tag_tid=2, entry_tid=1' 2024-04-06T10:02:29.425 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror image_20 2024-04-06T10:02:29.425 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:02:29.425 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:02:29.425 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_20 2024-04-06T10:02:29.425 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror image_20 '..*' 2024-04-06T10:02:29.425 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:02:29.425 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:02:29.425 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_20 2024-04-06T10:02:29.425 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T10:02:29.425 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-image_20.status 2024-04-06T10:02:29.425 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-image_20.status 2024-04-06T10:02:29.425 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T10:02:29.426 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-image_20.status 2024-04-06T10:02:29.426 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_20 2024-04-06T10:02:29.426 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_20.status 2024-04-06T10:02:29.473 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 2 2024-04-06T10:02:29.473 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 2 2024-04-06T10:02:29.473 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T10:02:29.474 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=9, tag_tid=2, entry_tid=1], [object_number=8, tag_tid=2, entry_tid=0], [object_number=11, tag_tid=1, entry_tid=9615], [object_number=10, tag_tid=1, entry_tid=9614]]], state=connected] 2024-04-06T10:02:29.474 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=11, tag_tid=1, entry_tid=9615], [object_number=10, tag_tid=1, entry_tid=9614], [object_number=9, tag_tid=1, entry_tid=9613], [object_number=8, tag_tid=1, entry_tid=9612]]], state=connected] 2024-04-06T10:02:29.477 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_20.status 2024-04-06T10:02:29.479 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=11, tag_tid=1, entry_tid=9615' 2024-04-06T10:02:29.479 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=9, tag_tid=2, entry_tid=1' -a 'object_number=9, tag_tid=2, entry_tid=1' = 'object_number=11, tag_tid=1, entry_tid=9615' 2024-04-06T10:02:29.479 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=11, tag_tid=1, entry_tid=9615' '!=' 'object_number=11, tag_tid=1, entry_tid=9615' 2024-04-06T10:02:29.479 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T10:02:29.479 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 1.6 2024-04-06T10:02:31.081 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror image_20 2024-04-06T10:02:31.081 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T10:02:31.081 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:02:31.081 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_20 2024-04-06T10:02:31.081 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T10:02:31.081 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n image_20 ']' 2024-04-06T10:02:31.081 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/image_20' 2024-04-06T10:02:31.082 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_20 2024-04-06T10:02:31.082 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T10:02:31.082 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T10:02:31.082 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T10:02:31.082 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T10:02:31.082 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T10:02:31.082 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T10:02:31.082 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T10:02:31.082 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T10:02:31.083 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T10:02:31.085 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T10:02:31.085 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_20 2024-04-06T10:02:31.085 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T10:02:31.085 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T10:02:31.085 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T10:02:31.085 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T10:02:31.085 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T10:02:31.085 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T10:02:31.085 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T10:02:31.085 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T10:02:31.086 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T10:02:31.086 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T10:02:31.086 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T10:02:31.086 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:02:31.086 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T10:02:31.086 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T10:02:31.087 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T10:02:31.087 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T10:02:31.087 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T10:02:31.087 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T10:02:31.087 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T10:02:31.087 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T10:02:31.087 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T10:02:31.087 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T10:02:31.087 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:02:31.087 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T10:02:31.088 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T10:02:31.088 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T10:02:31.088 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T10:02:31.088 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T10:02:31.088 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T10:02:31.088 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T10:02:31.111 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T10:02:31.112 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T10:02:31.112 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T10:02:31.112 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_20 2024-04-06T10:02:31.222 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:Invalid command: unused arguments: ['mirror/image_20'] 2024-04-06T10:02:31.222 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:rbd mirror flush : flush rbd mirror 2024-04-06T10:02:31.222 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:admin_socket: invalid command 2024-04-06T10:02:31.226 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T10:02:31.226 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/image_20 2024-04-06T10:02:31.226 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:1 2024-04-06T10:02:31.227 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T10:02:31.227 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T10:02:31.227 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-06T10:02:31.227 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:1 2024-04-06T10:02:31.227 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T10:02:31.227 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T10:02:31.227 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T10:02:31.227 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=1 2024-04-06T10:02:31.227 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-06T10:02:31.228 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T10:02:31.228 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:02:31.228 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=1 2024-04-06T10:02:31.228 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=1 2024-04-06T10:02:31.228 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-06T10:02:31.228 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:1 2024-04-06T10:02:31.228 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T10:02:31.228 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T10:02:31.228 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-06T10:02:31.228 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:1 2024-04-06T10:02:31.228 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T10:02:31.228 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T10:02:31.229 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:02:31.229 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=1 2024-04-06T10:02:31.229 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-06T10:02:31.229 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T10:02:31.229 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T10:02:31.229 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=1 2024-04-06T10:02:31.229 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=1 2024-04-06T10:02:31.229 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-06T10:02:31.250 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T10:02:31.251 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T10:02:31.251 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T10:02:31.251 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/image_20 2024-04-06T10:02:31.394 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T10:02:31.396 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror image_20 2024-04-06T10:02:31.396 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:02:31.396 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:02:31.396 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_20 2024-04-06T10:02:31.396 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror image_20 '' 2024-04-06T10:02:31.396 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:02:31.396 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:02:31.397 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_20 2024-04-06T10:02:31.397 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T10:02:31.397 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-image_20.status 2024-04-06T10:02:31.397 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-image_20.status 2024-04-06T10:02:31.397 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T10:02:31.398 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-image_20.status 2024-04-06T10:02:31.398 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_20 2024-04-06T10:02:31.399 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_20.status 2024-04-06T10:02:31.448 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 2 2024-04-06T10:02:31.448 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 2 2024-04-06T10:02:31.448 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T10:02:31.448 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=9, tag_tid=2, entry_tid=1], [object_number=8, tag_tid=2, entry_tid=0], [object_number=11, tag_tid=1, entry_tid=9615], [object_number=10, tag_tid=1, entry_tid=9614]]], state=connected] 2024-04-06T10:02:31.448 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=9, tag_tid=2, entry_tid=1], [object_number=8, tag_tid=2, entry_tid=0], [object_number=11, tag_tid=1, entry_tid=9615], [object_number=10, tag_tid=1, entry_tid=9614]]], state=connected] 2024-04-06T10:02:31.452 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_20.status 2024-04-06T10:02:31.453 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=9, tag_tid=2, entry_tid=1' 2024-04-06T10:02:31.455 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror image_20 2024-04-06T10:02:31.455 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:02:31.455 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:02:31.455 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_20 2024-04-06T10:02:31.455 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror image_20 '..*' 2024-04-06T10:02:31.455 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:02:31.455 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:02:31.455 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_20 2024-04-06T10:02:31.455 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T10:02:31.455 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-image_20.status 2024-04-06T10:02:31.456 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-image_20.status 2024-04-06T10:02:31.456 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T10:02:31.456 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-image_20.status 2024-04-06T10:02:31.456 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_20 2024-04-06T10:02:31.456 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_20.status 2024-04-06T10:02:31.622 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 2 2024-04-06T10:02:31.622 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 2 2024-04-06T10:02:31.622 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T10:02:31.622 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=9, tag_tid=2, entry_tid=1], [object_number=8, tag_tid=2, entry_tid=0], [object_number=11, tag_tid=1, entry_tid=9615], [object_number=10, tag_tid=1, entry_tid=9614]]], state=connected] 2024-04-06T10:02:31.622 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=9, tag_tid=2, entry_tid=1], [object_number=8, tag_tid=2, entry_tid=0], [object_number=11, tag_tid=1, entry_tid=9615], [object_number=10, tag_tid=1, entry_tid=9614]]], state=connected] 2024-04-06T10:02:31.622 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_20.status 2024-04-06T10:02:31.624 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=9, tag_tid=2, entry_tid=1' 2024-04-06T10:02:31.624 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=9, tag_tid=2, entry_tid=1' -a 'object_number=9, tag_tid=2, entry_tid=1' = 'object_number=9, tag_tid=2, entry_tid=1' 2024-04-06T10:02:31.624 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T10:02:31.624 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_snap_present cluster1 mirror image_20 snap 2024-04-06T10:02:31.624 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T10:02:31.624 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:02:31.625 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_20 2024-04-06T10:02:31.625 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local snap_name=snap 2024-04-06T10:02:31.625 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local s 2024-04-06T10:02:31.625 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-04-06T10:02:31.625 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 1 2024-04-06T10:02:32.626 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 info mirror/image_20@snap 2024-04-06T10:02:32.736 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:rbd image 'image_20': 2024-04-06T10:02:32.737 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: size 128 MiB in 32 objects 2024-04-06T10:02:32.737 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: order 22 (4 MiB objects) 2024-04-06T10:02:32.737 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: snapshot_count: 1 2024-04-06T10:02:32.737 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: id: 10ac9f3ea6aa 2024-04-06T10:02:32.737 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: block_name_prefix: rbd_data.10ac9f3ea6aa 2024-04-06T10:02:32.737 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: format: 2 2024-04-06T10:02:32.737 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: features: layering, exclusive-lock, journaling 2024-04-06T10:02:32.737 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: op_features: 2024-04-06T10:02:32.737 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: flags: 2024-04-06T10:02:32.737 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: create_timestamp: Sat Apr 6 09:53:41 2024 2024-04-06T10:02:32.737 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: access_timestamp: Sat Apr 6 09:53:41 2024 2024-04-06T10:02:32.737 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: modify_timestamp: Sat Apr 6 09:54:41 2024 2024-04-06T10:02:32.737 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: protected: False 2024-04-06T10:02:32.737 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: journal: 10ac9f3ea6aa 2024-04-06T10:02:32.738 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: mirroring state: enabled 2024-04-06T10:02:32.738 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: mirroring mode: journal 2024-04-06T10:02:32.738 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: mirroring global id: 6d47c15f-669d-4891-b6d9-7978b4e1edf5 2024-04-06T10:02:32.738 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: mirroring primary: false 2024-04-06T10:02:32.741 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T10:02:32.741 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ compare_image_snaps mirror image_20 snap 2024-04-06T10:02:32.741 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:02:32.742 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_20 2024-04-06T10:02:32.742 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local snap_name=snap 2024-04-06T10:02:32.742 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local ret=0 2024-04-06T10:02:32.742 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local rmt_export=/tmp/tmp.BhW6xKmROU/cluster2-mirror-image_20.export 2024-04-06T10:02:32.742 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loc_export=/tmp/tmp.BhW6xKmROU/cluster1-mirror-image_20.export 2024-04-06T10:02:32.742 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rm -f /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_20.export /tmp/tmp.BhW6xKmROU/cluster1-mirror-image_20.export 2024-04-06T10:02:32.742 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster2 -p mirror export image_20@snap /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_20.export 2024-04-06T10:02:32.915 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-04-06T10:02:32.924 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 -p mirror export image_20@snap /tmp/tmp.BhW6xKmROU/cluster1-mirror-image_20.export 2024-04-06T10:02:33.285 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-04-06T10:02:33.301 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmp /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_20.export /tmp/tmp.BhW6xKmROU/cluster1-mirror-image_20.export 2024-04-06T10:02:33.379 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rm -f /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_20.export /tmp/tmp.BhW6xKmROU/cluster1-mirror-image_20.export 2024-04-06T10:02:33.421 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T10:02:33.421 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-06T10:02:33.421 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ image=image_21 2024-04-06T10:02:33.421 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ create_snap cluster2 mirror image_21 snap 2024-04-06T10:02:33.421 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-04-06T10:02:33.421 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:02:33.421 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_21 2024-04-06T10:02:33.421 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local snap_name=snap 2024-04-06T10:02:33.421 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster2 -p mirror snap create image_21@snap --debug-rbd=20 --debug-journaler=20 2024-04-06T10:02:34.882 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_image_replay_started cluster1 mirror image_21 2024-04-06T10:02:34.883 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T10:02:34.883 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:02:34.883 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_21 2024-04-06T10:02:34.883 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_image_replay_state cluster1 mirror image_21 started 2024-04-06T10:02:34.883 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T10:02:34.883 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:02:34.883 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_21 2024-04-06T10:02:34.883 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state=started 2024-04-06T10:02:34.883 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local s 2024-04-06T10:02:34.883 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T10:02:34.883 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 1 2024-04-06T10:02:35.881 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test_image_replay_state cluster1 mirror image_21 started 2024-04-06T10:02:35.881 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T10:02:35.881 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:02:35.881 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_21 2024-04-06T10:02:35.881 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local test_state=started 2024-04-06T10:02:35.881 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local status_result 2024-04-06T10:02:35.881 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local current_state=stopped 2024-04-06T10:02:35.882 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_21 2024-04-06T10:02:35.883 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -i state 2024-04-06T10:02:35.883 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_instance=cluster1 2024-04-06T10:02:35.883 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ shift 2024-04-06T10:02:35.883 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T10:02:35.883 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance=cluster1 2024-04-06T10:02:35.883 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local loop_instance 2024-04-06T10:02:35.883 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T10:02:35.883 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sleep 0 2024-04-06T10:02:35.884 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-04-06T10:02:35.884 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ seq 0 3 2024-04-06T10:02:35.887 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T10:02:35.888 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_21 2024-04-06T10:02:35.888 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1:0 2024-04-06T10:02:35.888 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ shift 2024-04-06T10:02:35.888 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T10:02:35.888 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-04-06T10:02:35.888 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T10:02:35.888 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=cluster 2024-04-06T10:02:35.888 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T10:02:35.888 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:02:35.888 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T10:02:35.888 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T10:02:35.888 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval cluster=cluster1 2024-04-06T10:02:35.888 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ cluster=cluster1 2024-04-06T10:02:35.888 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T10:02:35.889 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T10:02:35.889 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-04-06T10:02:35.889 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local local_cluster=cluster1:0 2024-04-06T10:02:35.889 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local cluster=cluster1 2024-04-06T10:02:35.889 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local instance 2024-04-06T10:02:35.889 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T10:02:35.889 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local val=cluster1:0 2024-04-06T10:02:35.889 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local cluster_var_name=local_cluster 2024-04-06T10:02:35.889 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local instance_var_name=instance 2024-04-06T10:02:35.889 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ cluster=cluster1 2024-04-06T10:02:35.889 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T10:02:35.889 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ '[' 0 = cluster1:0 ']' 2024-04-06T10:02:35.889 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ eval local_cluster=cluster1 2024-04-06T10:02:35.889 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++++ local_cluster=cluster1 2024-04-06T10:02:35.889 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ eval instance=0 2024-04-06T10:02:35.889 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++++ instance=0 2024-04-06T10:02:35.890 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T10:02:35.910 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T10:02:35.911 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T10:02:35.911 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T10:02:35.911 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_21 2024-04-06T10:02:36.014 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:Invalid command: unused arguments: ['mirror/image_21'] 2024-04-06T10:02:36.014 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:rbd mirror status : get status for rbd mirror 2024-04-06T10:02:36.015 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:admin_socket: invalid command 2024-04-06T10:02:36.018 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T10:02:36.018 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_21 2024-04-06T10:02:36.018 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1:1 2024-04-06T10:02:36.018 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ shift 2024-04-06T10:02:36.018 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T10:02:36.018 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-04-06T10:02:36.018 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:1 2024-04-06T10:02:36.018 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=cluster 2024-04-06T10:02:36.018 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T10:02:36.018 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:02:36.018 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=1 2024-04-06T10:02:36.018 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-06T10:02:36.019 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval cluster=cluster1 2024-04-06T10:02:36.019 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ cluster=cluster1 2024-04-06T10:02:36.019 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=1 2024-04-06T10:02:36.019 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=1 2024-04-06T10:02:36.019 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-04-06T10:02:36.019 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local local_cluster=cluster1:1 2024-04-06T10:02:36.019 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local cluster=cluster1 2024-04-06T10:02:36.019 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local instance 2024-04-06T10:02:36.019 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-06T10:02:36.019 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local val=cluster1:1 2024-04-06T10:02:36.019 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local cluster_var_name=local_cluster 2024-04-06T10:02:36.019 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local instance_var_name=instance 2024-04-06T10:02:36.019 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ cluster=cluster1 2024-04-06T10:02:36.019 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=1 2024-04-06T10:02:36.019 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ '[' 1 = cluster1:1 ']' 2024-04-06T10:02:36.020 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ eval local_cluster=cluster1 2024-04-06T10:02:36.020 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++++ local_cluster=cluster1 2024-04-06T10:02:36.020 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ eval instance=1 2024-04-06T10:02:36.020 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++++ instance=1 2024-04-06T10:02:36.020 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-06T10:02:36.042 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T10:02:36.042 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T10:02:36.042 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T10:02:36.043 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_21 2024-04-06T10:02:36.199 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ return 0 2024-04-06T10:02:36.199 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ status_result=' "state": "Replaying"' 2024-04-06T10:02:36.200 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ echo ' "state": "Replaying"' 2024-04-06T10:02:36.200 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep -i Replaying 2024-04-06T10:02:36.201 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: "state": "Replaying" 2024-04-06T10:02:36.203 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ current_state=started 2024-04-06T10:02:36.203 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test started = started 2024-04-06T10:02:36.203 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T10:02:36.203 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_21 2024-04-06T10:02:36.203 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local local_cluster=cluster1 2024-04-06T10:02:36.203 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-04-06T10:02:36.203 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:02:36.203 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_21 2024-04-06T10:02:36.203 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' journal = journal ']' 2024-04-06T10:02:36.203 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_journal_replay_complete cluster1 cluster2 mirror image_21 2024-04-06T10:02:36.203 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local local_cluster=cluster1 2024-04-06T10:02:36.203 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-04-06T10:02:36.204 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:02:36.204 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_21 2024-04-06T10:02:36.204 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local s master_pos mirror_pos last_mirror_pos 2024-04-06T10:02:36.204 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local master_tag master_entry mirror_tag mirror_entry 2024-04-06T10:02:36.204 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T10:02:36.204 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T10:02:36.204 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T10:02:36.404 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror image_21 2024-04-06T10:02:36.405 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T10:02:36.405 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:02:36.405 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_21 2024-04-06T10:02:36.405 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T10:02:36.405 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n image_21 ']' 2024-04-06T10:02:36.405 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/image_21' 2024-04-06T10:02:36.405 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_21 2024-04-06T10:02:36.405 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T10:02:36.405 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T10:02:36.405 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T10:02:36.405 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T10:02:36.405 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T10:02:36.405 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T10:02:36.406 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T10:02:36.406 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T10:02:36.406 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T10:02:36.408 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T10:02:36.408 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_21 2024-04-06T10:02:36.408 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T10:02:36.408 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T10:02:36.408 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T10:02:36.408 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T10:02:36.408 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T10:02:36.408 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T10:02:36.408 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T10:02:36.408 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T10:02:36.408 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T10:02:36.409 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T10:02:36.409 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T10:02:36.409 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:02:36.409 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T10:02:36.409 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T10:02:36.410 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T10:02:36.410 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T10:02:36.410 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T10:02:36.410 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T10:02:36.410 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T10:02:36.410 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T10:02:36.411 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T10:02:36.411 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T10:02:36.411 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:02:36.411 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T10:02:36.411 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T10:02:36.411 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T10:02:36.411 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T10:02:36.411 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T10:02:36.411 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T10:02:36.411 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T10:02:36.435 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T10:02:36.435 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T10:02:36.435 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T10:02:36.435 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_21 2024-04-06T10:02:36.546 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:Invalid command: unused arguments: ['mirror/image_21'] 2024-04-06T10:02:36.546 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:rbd mirror flush : flush rbd mirror 2024-04-06T10:02:36.546 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:admin_socket: invalid command 2024-04-06T10:02:36.548 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T10:02:36.548 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/image_21 2024-04-06T10:02:36.548 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:1 2024-04-06T10:02:36.549 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T10:02:36.549 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T10:02:36.549 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-06T10:02:36.549 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:1 2024-04-06T10:02:36.549 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T10:02:36.549 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T10:02:36.549 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T10:02:36.549 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=1 2024-04-06T10:02:36.549 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-06T10:02:36.549 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T10:02:36.549 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:02:36.549 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=1 2024-04-06T10:02:36.549 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=1 2024-04-06T10:02:36.550 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-06T10:02:36.550 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:1 2024-04-06T10:02:36.551 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T10:02:36.551 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T10:02:36.551 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-06T10:02:36.551 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:1 2024-04-06T10:02:36.551 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T10:02:36.551 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T10:02:36.551 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:02:36.551 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=1 2024-04-06T10:02:36.551 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-06T10:02:36.551 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T10:02:36.551 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T10:02:36.551 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=1 2024-04-06T10:02:36.551 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=1 2024-04-06T10:02:36.552 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-06T10:02:36.569 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T10:02:36.569 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T10:02:36.569 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T10:02:36.569 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/image_21 2024-04-06T10:02:36.704 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T10:02:36.706 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror image_21 2024-04-06T10:02:36.706 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:02:36.706 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:02:36.706 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_21 2024-04-06T10:02:36.706 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror image_21 '' 2024-04-06T10:02:36.706 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:02:36.706 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:02:36.706 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_21 2024-04-06T10:02:36.706 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T10:02:36.706 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-image_21.status 2024-04-06T10:02:36.707 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-image_21.status 2024-04-06T10:02:36.707 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T10:02:36.709 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-image_21.status 2024-04-06T10:02:36.709 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_21 2024-04-06T10:02:36.710 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_21.status 2024-04-06T10:02:36.762 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 2 2024-04-06T10:02:36.763 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 2 2024-04-06T10:02:36.763 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T10:02:36.763 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=9, tag_tid=2, entry_tid=1], [object_number=8, tag_tid=2, entry_tid=0], [object_number=11, tag_tid=1, entry_tid=9743], [object_number=10, tag_tid=1, entry_tid=9742]]], state=connected] 2024-04-06T10:02:36.763 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=9, tag_tid=1, entry_tid=9745], [object_number=8, tag_tid=1, entry_tid=9744], [object_number=11, tag_tid=1, entry_tid=9743], [object_number=10, tag_tid=1, entry_tid=9742]]], state=connected] 2024-04-06T10:02:36.765 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_21.status 2024-04-06T10:02:36.766 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=9, tag_tid=2, entry_tid=1' 2024-04-06T10:02:36.768 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror image_21 2024-04-06T10:02:36.768 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:02:36.768 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:02:36.768 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_21 2024-04-06T10:02:36.768 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror image_21 '..*' 2024-04-06T10:02:36.768 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:02:36.768 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:02:36.768 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_21 2024-04-06T10:02:36.769 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T10:02:36.769 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-image_21.status 2024-04-06T10:02:36.769 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-image_21.status 2024-04-06T10:02:36.769 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T10:02:36.770 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-image_21.status 2024-04-06T10:02:36.770 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_21 2024-04-06T10:02:36.770 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_21.status 2024-04-06T10:02:36.816 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 2 2024-04-06T10:02:36.816 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 2 2024-04-06T10:02:36.816 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T10:02:36.816 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=9, tag_tid=2, entry_tid=1], [object_number=8, tag_tid=2, entry_tid=0], [object_number=11, tag_tid=1, entry_tid=9743], [object_number=10, tag_tid=1, entry_tid=9742]]], state=connected] 2024-04-06T10:02:36.817 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=9, tag_tid=1, entry_tid=9745], [object_number=8, tag_tid=1, entry_tid=9744], [object_number=11, tag_tid=1, entry_tid=9743], [object_number=10, tag_tid=1, entry_tid=9742]]], state=connected] 2024-04-06T10:02:36.820 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_21.status 2024-04-06T10:02:36.823 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=9, tag_tid=1, entry_tid=9745' 2024-04-06T10:02:36.824 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=9, tag_tid=2, entry_tid=1' -a 'object_number=9, tag_tid=2, entry_tid=1' = 'object_number=9, tag_tid=1, entry_tid=9745' 2024-04-06T10:02:36.824 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=9, tag_tid=1, entry_tid=9745' '!=' '' 2024-04-06T10:02:36.824 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T10:02:36.824 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=9, tag_tid=1, entry_tid=9745' = '' 2024-04-06T10:02:36.824 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=9, tag_tid=1, entry_tid=9745' 2024-04-06T10:02:36.824 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=9, tag_tid=2, entry_tid=1' 2024-04-06T10:02:36.824 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T10:02:36.824 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T10:02:36.824 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=2 2024-04-06T10:02:36.826 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=9, tag_tid=1, entry_tid=9745' 2024-04-06T10:02:36.826 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T10:02:36.826 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T10:02:36.826 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=1 2024-04-06T10:02:36.826 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=9, tag_tid=2, entry_tid=1' 2024-04-06T10:02:36.826 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T10:02:36.826 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T10:02:36.828 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T10:02:36.828 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=9, tag_tid=1, entry_tid=9745' 2024-04-06T10:02:36.828 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T10:02:36.828 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T10:02:36.829 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=9745 2024-04-06T10:02:36.829 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 2 = 1 -a 1 -le 9745 2024-04-06T10:02:36.829 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T10:02:36.829 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T10:02:36.829 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T10:02:37.030 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror image_21 2024-04-06T10:02:37.031 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T10:02:37.031 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:02:37.031 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_21 2024-04-06T10:02:37.031 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T10:02:37.031 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n image_21 ']' 2024-04-06T10:02:37.031 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/image_21' 2024-04-06T10:02:37.031 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_21 2024-04-06T10:02:37.031 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T10:02:37.031 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T10:02:37.031 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T10:02:37.031 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T10:02:37.031 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T10:02:37.031 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T10:02:37.031 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T10:02:37.032 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T10:02:37.033 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T10:02:37.035 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T10:02:37.035 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_21 2024-04-06T10:02:37.035 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T10:02:37.035 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T10:02:37.035 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T10:02:37.035 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T10:02:37.035 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T10:02:37.035 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T10:02:37.035 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T10:02:37.035 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T10:02:37.035 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T10:02:37.035 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T10:02:37.036 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T10:02:37.036 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:02:37.036 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T10:02:37.036 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T10:02:37.037 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T10:02:37.037 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T10:02:37.037 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T10:02:37.037 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T10:02:37.037 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T10:02:37.037 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T10:02:37.037 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T10:02:37.037 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T10:02:37.037 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:02:37.037 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T10:02:37.037 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T10:02:37.037 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T10:02:37.037 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T10:02:37.038 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T10:02:37.038 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T10:02:37.038 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T10:02:37.063 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T10:02:37.063 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T10:02:37.063 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T10:02:37.063 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_21 2024-04-06T10:02:37.174 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:Invalid command: unused arguments: ['mirror/image_21'] 2024-04-06T10:02:37.174 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:rbd mirror flush : flush rbd mirror 2024-04-06T10:02:37.174 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:admin_socket: invalid command 2024-04-06T10:02:37.177 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T10:02:37.177 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/image_21 2024-04-06T10:02:37.177 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:1 2024-04-06T10:02:37.177 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T10:02:37.177 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T10:02:37.177 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-06T10:02:37.177 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:1 2024-04-06T10:02:37.177 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T10:02:37.177 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T10:02:37.177 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T10:02:37.177 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=1 2024-04-06T10:02:37.178 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-06T10:02:37.178 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T10:02:37.178 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:02:37.178 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=1 2024-04-06T10:02:37.178 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=1 2024-04-06T10:02:37.179 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-06T10:02:37.179 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:1 2024-04-06T10:02:37.179 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T10:02:37.179 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T10:02:37.179 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-06T10:02:37.180 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:1 2024-04-06T10:02:37.180 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T10:02:37.180 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T10:02:37.180 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:02:37.180 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=1 2024-04-06T10:02:37.180 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-06T10:02:37.180 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T10:02:37.180 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T10:02:37.180 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=1 2024-04-06T10:02:37.180 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=1 2024-04-06T10:02:37.180 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-06T10:02:37.203 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T10:02:37.203 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T10:02:37.203 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T10:02:37.204 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/image_21 2024-04-06T10:02:37.345 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T10:02:37.345 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror image_21 2024-04-06T10:02:37.346 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:02:37.346 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:02:37.346 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_21 2024-04-06T10:02:37.346 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror image_21 '' 2024-04-06T10:02:37.346 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:02:37.346 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:02:37.346 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_21 2024-04-06T10:02:37.346 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T10:02:37.347 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-image_21.status 2024-04-06T10:02:37.347 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-image_21.status 2024-04-06T10:02:37.347 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T10:02:37.349 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-image_21.status 2024-04-06T10:02:37.350 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_21 2024-04-06T10:02:37.350 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_21.status 2024-04-06T10:02:37.401 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 2 2024-04-06T10:02:37.401 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 2 2024-04-06T10:02:37.401 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T10:02:37.401 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=9, tag_tid=2, entry_tid=1], [object_number=8, tag_tid=2, entry_tid=0], [object_number=11, tag_tid=1, entry_tid=9743], [object_number=10, tag_tid=1, entry_tid=9742]]], state=connected] 2024-04-06T10:02:37.402 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=9, tag_tid=1, entry_tid=9745], [object_number=8, tag_tid=1, entry_tid=9744], [object_number=11, tag_tid=1, entry_tid=9743], [object_number=10, tag_tid=1, entry_tid=9742]]], state=connected] 2024-04-06T10:02:37.405 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_21.status 2024-04-06T10:02:37.406 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=9, tag_tid=2, entry_tid=1' 2024-04-06T10:02:37.407 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror image_21 2024-04-06T10:02:37.407 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:02:37.408 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:02:37.408 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_21 2024-04-06T10:02:37.408 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror image_21 '..*' 2024-04-06T10:02:37.408 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:02:37.408 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:02:37.408 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_21 2024-04-06T10:02:37.408 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T10:02:37.408 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-image_21.status 2024-04-06T10:02:37.408 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-image_21.status 2024-04-06T10:02:37.408 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T10:02:37.409 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-image_21.status 2024-04-06T10:02:37.409 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_21 2024-04-06T10:02:37.409 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_21.status 2024-04-06T10:02:37.455 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 2 2024-04-06T10:02:37.455 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 2 2024-04-06T10:02:37.455 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T10:02:37.455 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=9, tag_tid=2, entry_tid=1], [object_number=8, tag_tid=2, entry_tid=0], [object_number=11, tag_tid=1, entry_tid=9743], [object_number=10, tag_tid=1, entry_tid=9742]]], state=connected] 2024-04-06T10:02:37.455 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=9, tag_tid=1, entry_tid=9745], [object_number=8, tag_tid=1, entry_tid=9744], [object_number=11, tag_tid=1, entry_tid=9743], [object_number=10, tag_tid=1, entry_tid=9742]]], state=connected] 2024-04-06T10:02:37.459 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_21.status 2024-04-06T10:02:37.460 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=9, tag_tid=1, entry_tid=9745' 2024-04-06T10:02:37.460 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=9, tag_tid=2, entry_tid=1' -a 'object_number=9, tag_tid=2, entry_tid=1' = 'object_number=9, tag_tid=1, entry_tid=9745' 2024-04-06T10:02:37.461 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=9, tag_tid=1, entry_tid=9745' '!=' 'object_number=9, tag_tid=1, entry_tid=9745' 2024-04-06T10:02:37.461 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T10:02:37.461 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.4 2024-04-06T10:02:37.863 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror image_21 2024-04-06T10:02:37.863 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T10:02:37.863 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:02:37.863 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_21 2024-04-06T10:02:37.863 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T10:02:37.863 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n image_21 ']' 2024-04-06T10:02:37.863 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/image_21' 2024-04-06T10:02:37.863 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_21 2024-04-06T10:02:37.863 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T10:02:37.863 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T10:02:37.863 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T10:02:37.864 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T10:02:37.864 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T10:02:37.864 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T10:02:37.864 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T10:02:37.864 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T10:02:37.864 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T10:02:37.867 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T10:02:37.867 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_21 2024-04-06T10:02:37.867 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T10:02:37.867 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T10:02:37.867 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T10:02:37.867 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T10:02:37.867 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T10:02:37.867 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T10:02:37.867 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T10:02:37.867 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T10:02:37.867 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T10:02:37.867 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T10:02:37.868 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T10:02:37.868 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:02:37.868 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T10:02:37.868 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T10:02:37.868 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T10:02:37.868 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T10:02:37.869 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T10:02:37.869 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T10:02:37.869 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T10:02:37.869 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T10:02:37.869 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T10:02:37.869 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T10:02:37.869 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:02:37.869 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T10:02:37.869 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T10:02:37.869 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T10:02:37.869 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T10:02:37.870 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T10:02:37.870 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T10:02:37.870 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T10:02:37.893 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T10:02:37.894 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T10:02:37.894 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T10:02:37.894 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_21 2024-04-06T10:02:38.000 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:Invalid command: unused arguments: ['mirror/image_21'] 2024-04-06T10:02:38.000 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:rbd mirror flush : flush rbd mirror 2024-04-06T10:02:38.000 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:admin_socket: invalid command 2024-04-06T10:02:38.003 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T10:02:38.003 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/image_21 2024-04-06T10:02:38.003 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:1 2024-04-06T10:02:38.003 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T10:02:38.003 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T10:02:38.003 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-06T10:02:38.003 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:1 2024-04-06T10:02:38.003 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T10:02:38.003 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T10:02:38.003 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T10:02:38.004 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=1 2024-04-06T10:02:38.004 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-06T10:02:38.004 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T10:02:38.004 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:02:38.004 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=1 2024-04-06T10:02:38.004 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=1 2024-04-06T10:02:38.005 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-06T10:02:38.005 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:1 2024-04-06T10:02:38.005 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T10:02:38.005 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T10:02:38.005 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-06T10:02:38.005 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:1 2024-04-06T10:02:38.006 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T10:02:38.006 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T10:02:38.006 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:02:38.006 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=1 2024-04-06T10:02:38.006 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-06T10:02:38.006 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T10:02:38.006 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T10:02:38.006 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=1 2024-04-06T10:02:38.006 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=1 2024-04-06T10:02:38.006 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-06T10:02:38.028 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T10:02:38.028 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T10:02:38.028 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T10:02:38.028 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/image_21 2024-04-06T10:02:38.165 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T10:02:38.166 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror image_21 2024-04-06T10:02:38.166 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:02:38.166 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:02:38.166 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_21 2024-04-06T10:02:38.166 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror image_21 '' 2024-04-06T10:02:38.166 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:02:38.167 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:02:38.167 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_21 2024-04-06T10:02:38.167 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T10:02:38.167 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-image_21.status 2024-04-06T10:02:38.167 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-image_21.status 2024-04-06T10:02:38.167 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T10:02:38.169 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-image_21.status 2024-04-06T10:02:38.169 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_21 2024-04-06T10:02:38.169 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_21.status 2024-04-06T10:02:38.220 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 2 2024-04-06T10:02:38.220 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 2 2024-04-06T10:02:38.221 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T10:02:38.221 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=9, tag_tid=2, entry_tid=1], [object_number=8, tag_tid=2, entry_tid=0], [object_number=11, tag_tid=1, entry_tid=9743], [object_number=10, tag_tid=1, entry_tid=9742]]], state=connected] 2024-04-06T10:02:38.221 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=9, tag_tid=1, entry_tid=9745], [object_number=8, tag_tid=1, entry_tid=9744], [object_number=11, tag_tid=1, entry_tid=9743], [object_number=10, tag_tid=1, entry_tid=9742]]], state=connected] 2024-04-06T10:02:38.224 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_21.status 2024-04-06T10:02:38.225 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=9, tag_tid=2, entry_tid=1' 2024-04-06T10:02:38.226 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror image_21 2024-04-06T10:02:38.226 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:02:38.227 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:02:38.227 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_21 2024-04-06T10:02:38.227 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror image_21 '..*' 2024-04-06T10:02:38.227 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:02:38.227 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:02:38.227 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_21 2024-04-06T10:02:38.227 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T10:02:38.227 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-image_21.status 2024-04-06T10:02:38.227 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-image_21.status 2024-04-06T10:02:38.227 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T10:02:38.228 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-image_21.status 2024-04-06T10:02:38.228 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_21 2024-04-06T10:02:38.228 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_21.status 2024-04-06T10:02:38.273 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 2 2024-04-06T10:02:38.273 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 2 2024-04-06T10:02:38.274 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T10:02:38.274 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=9, tag_tid=2, entry_tid=1], [object_number=8, tag_tid=2, entry_tid=0], [object_number=11, tag_tid=1, entry_tid=9743], [object_number=10, tag_tid=1, entry_tid=9742]]], state=connected] 2024-04-06T10:02:38.274 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=9, tag_tid=1, entry_tid=9745], [object_number=8, tag_tid=1, entry_tid=9744], [object_number=11, tag_tid=1, entry_tid=9743], [object_number=10, tag_tid=1, entry_tid=9742]]], state=connected] 2024-04-06T10:02:38.277 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_21.status 2024-04-06T10:02:38.279 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=9, tag_tid=1, entry_tid=9745' 2024-04-06T10:02:38.279 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=9, tag_tid=2, entry_tid=1' -a 'object_number=9, tag_tid=2, entry_tid=1' = 'object_number=9, tag_tid=1, entry_tid=9745' 2024-04-06T10:02:38.279 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=9, tag_tid=1, entry_tid=9745' '!=' 'object_number=9, tag_tid=1, entry_tid=9745' 2024-04-06T10:02:38.279 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T10:02:38.279 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.8 2024-04-06T10:02:39.081 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror image_21 2024-04-06T10:02:39.081 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T10:02:39.081 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:02:39.081 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_21 2024-04-06T10:02:39.081 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T10:02:39.081 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n image_21 ']' 2024-04-06T10:02:39.081 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/image_21' 2024-04-06T10:02:39.081 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_21 2024-04-06T10:02:39.082 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T10:02:39.082 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T10:02:39.082 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T10:02:39.082 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T10:02:39.082 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T10:02:39.082 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T10:02:39.082 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T10:02:39.082 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T10:02:39.083 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T10:02:39.086 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T10:02:39.086 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_21 2024-04-06T10:02:39.086 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T10:02:39.086 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T10:02:39.086 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T10:02:39.087 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T10:02:39.087 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T10:02:39.087 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T10:02:39.087 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T10:02:39.087 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T10:02:39.087 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T10:02:39.087 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T10:02:39.087 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T10:02:39.087 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:02:39.087 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T10:02:39.087 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T10:02:39.087 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T10:02:39.087 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T10:02:39.089 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T10:02:39.089 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T10:02:39.089 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T10:02:39.089 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T10:02:39.089 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T10:02:39.089 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T10:02:39.089 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:02:39.089 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T10:02:39.089 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T10:02:39.089 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T10:02:39.089 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T10:02:39.089 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T10:02:39.089 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T10:02:39.089 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T10:02:39.113 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T10:02:39.113 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T10:02:39.113 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T10:02:39.113 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_21 2024-04-06T10:02:39.218 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:Invalid command: unused arguments: ['mirror/image_21'] 2024-04-06T10:02:39.219 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:rbd mirror flush : flush rbd mirror 2024-04-06T10:02:39.219 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:admin_socket: invalid command 2024-04-06T10:02:39.221 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T10:02:39.221 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/image_21 2024-04-06T10:02:39.221 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:1 2024-04-06T10:02:39.221 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T10:02:39.221 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T10:02:39.221 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-06T10:02:39.221 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:1 2024-04-06T10:02:39.221 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T10:02:39.222 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T10:02:39.222 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T10:02:39.222 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=1 2024-04-06T10:02:39.222 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-06T10:02:39.222 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T10:02:39.222 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:02:39.222 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=1 2024-04-06T10:02:39.222 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=1 2024-04-06T10:02:39.223 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-06T10:02:39.223 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:1 2024-04-06T10:02:39.223 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T10:02:39.224 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T10:02:39.224 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-06T10:02:39.224 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:1 2024-04-06T10:02:39.224 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T10:02:39.224 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T10:02:39.224 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:02:39.224 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=1 2024-04-06T10:02:39.224 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-06T10:02:39.224 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T10:02:39.224 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T10:02:39.224 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=1 2024-04-06T10:02:39.224 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=1 2024-04-06T10:02:39.224 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-06T10:02:39.247 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T10:02:39.248 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T10:02:39.248 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T10:02:39.248 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/image_21 2024-04-06T10:02:39.396 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T10:02:39.397 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror image_21 2024-04-06T10:02:39.397 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:02:39.397 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:02:39.397 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_21 2024-04-06T10:02:39.397 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror image_21 '' 2024-04-06T10:02:39.397 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:02:39.398 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:02:39.398 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_21 2024-04-06T10:02:39.398 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T10:02:39.398 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-image_21.status 2024-04-06T10:02:39.398 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-image_21.status 2024-04-06T10:02:39.398 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T10:02:39.400 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-image_21.status 2024-04-06T10:02:39.401 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_21 2024-04-06T10:02:39.401 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_21.status 2024-04-06T10:02:39.445 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 2 2024-04-06T10:02:39.445 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 2 2024-04-06T10:02:39.445 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T10:02:39.446 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=9, tag_tid=2, entry_tid=1], [object_number=8, tag_tid=2, entry_tid=0], [object_number=11, tag_tid=1, entry_tid=9743], [object_number=10, tag_tid=1, entry_tid=9742]]], state=connected] 2024-04-06T10:02:39.446 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=9, tag_tid=1, entry_tid=9745], [object_number=8, tag_tid=1, entry_tid=9744], [object_number=11, tag_tid=1, entry_tid=9743], [object_number=10, tag_tid=1, entry_tid=9742]]], state=connected] 2024-04-06T10:02:39.449 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_21.status 2024-04-06T10:02:39.450 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=9, tag_tid=2, entry_tid=1' 2024-04-06T10:02:39.451 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror image_21 2024-04-06T10:02:39.452 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:02:39.452 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:02:39.452 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_21 2024-04-06T10:02:39.452 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror image_21 '..*' 2024-04-06T10:02:39.452 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:02:39.452 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:02:39.452 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_21 2024-04-06T10:02:39.452 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T10:02:39.452 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-image_21.status 2024-04-06T10:02:39.452 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-image_21.status 2024-04-06T10:02:39.452 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T10:02:39.453 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-image_21.status 2024-04-06T10:02:39.453 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_21 2024-04-06T10:02:39.453 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_21.status 2024-04-06T10:02:39.497 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 2 2024-04-06T10:02:39.498 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 2 2024-04-06T10:02:39.498 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T10:02:39.498 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=9, tag_tid=2, entry_tid=1], [object_number=8, tag_tid=2, entry_tid=0], [object_number=11, tag_tid=1, entry_tid=9743], [object_number=10, tag_tid=1, entry_tid=9742]]], state=connected] 2024-04-06T10:02:39.498 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=9, tag_tid=1, entry_tid=9745], [object_number=8, tag_tid=1, entry_tid=9744], [object_number=11, tag_tid=1, entry_tid=9743], [object_number=10, tag_tid=1, entry_tid=9742]]], state=connected] 2024-04-06T10:02:39.501 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_21.status 2024-04-06T10:02:39.503 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=9, tag_tid=1, entry_tid=9745' 2024-04-06T10:02:39.503 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=9, tag_tid=2, entry_tid=1' -a 'object_number=9, tag_tid=2, entry_tid=1' = 'object_number=9, tag_tid=1, entry_tid=9745' 2024-04-06T10:02:39.503 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=9, tag_tid=1, entry_tid=9745' '!=' 'object_number=9, tag_tid=1, entry_tid=9745' 2024-04-06T10:02:39.503 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T10:02:39.503 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 1.6 2024-04-06T10:02:41.105 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror image_21 2024-04-06T10:02:41.105 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T10:02:41.105 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:02:41.105 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_21 2024-04-06T10:02:41.105 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T10:02:41.105 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n image_21 ']' 2024-04-06T10:02:41.105 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/image_21' 2024-04-06T10:02:41.105 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_21 2024-04-06T10:02:41.105 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T10:02:41.105 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T10:02:41.105 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T10:02:41.106 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T10:02:41.106 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T10:02:41.106 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T10:02:41.106 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T10:02:41.106 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T10:02:41.106 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T10:02:41.109 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T10:02:41.109 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_21 2024-04-06T10:02:41.109 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T10:02:41.109 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T10:02:41.109 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T10:02:41.109 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T10:02:41.109 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T10:02:41.109 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T10:02:41.109 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T10:02:41.109 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T10:02:41.109 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T10:02:41.109 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T10:02:41.109 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T10:02:41.109 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:02:41.110 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T10:02:41.110 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T10:02:41.111 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T10:02:41.111 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T10:02:41.111 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T10:02:41.111 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T10:02:41.111 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T10:02:41.111 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T10:02:41.111 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T10:02:41.112 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T10:02:41.112 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:02:41.112 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T10:02:41.112 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T10:02:41.112 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T10:02:41.112 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T10:02:41.112 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T10:02:41.113 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T10:02:41.113 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T10:02:41.137 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T10:02:41.137 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T10:02:41.137 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T10:02:41.137 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_21 2024-04-06T10:02:41.239 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:Invalid command: unused arguments: ['mirror/image_21'] 2024-04-06T10:02:41.239 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:rbd mirror flush : flush rbd mirror 2024-04-06T10:02:41.239 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:admin_socket: invalid command 2024-04-06T10:02:41.241 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T10:02:41.241 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/image_21 2024-04-06T10:02:41.241 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:1 2024-04-06T10:02:41.242 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T10:02:41.242 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T10:02:41.242 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-06T10:02:41.242 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:1 2024-04-06T10:02:41.242 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T10:02:41.242 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T10:02:41.242 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T10:02:41.242 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=1 2024-04-06T10:02:41.242 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-06T10:02:41.242 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T10:02:41.242 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:02:41.242 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=1 2024-04-06T10:02:41.242 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=1 2024-04-06T10:02:41.243 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-06T10:02:41.243 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:1 2024-04-06T10:02:41.243 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T10:02:41.243 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T10:02:41.243 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-06T10:02:41.243 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:1 2024-04-06T10:02:41.243 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T10:02:41.243 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T10:02:41.243 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:02:41.243 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=1 2024-04-06T10:02:41.243 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-06T10:02:41.243 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T10:02:41.243 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T10:02:41.243 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=1 2024-04-06T10:02:41.243 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=1 2024-04-06T10:02:41.244 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-06T10:02:41.267 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T10:02:41.267 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T10:02:41.267 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T10:02:41.267 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/image_21 2024-04-06T10:02:41.403 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T10:02:41.404 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror image_21 2024-04-06T10:02:41.404 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:02:41.404 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:02:41.404 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_21 2024-04-06T10:02:41.404 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror image_21 '' 2024-04-06T10:02:41.405 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:02:41.405 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:02:41.405 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_21 2024-04-06T10:02:41.405 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T10:02:41.405 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-image_21.status 2024-04-06T10:02:41.405 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-image_21.status 2024-04-06T10:02:41.406 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T10:02:41.407 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-image_21.status 2024-04-06T10:02:41.407 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_21 2024-04-06T10:02:41.408 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_21.status 2024-04-06T10:02:41.452 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 2 2024-04-06T10:02:41.453 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 2 2024-04-06T10:02:41.453 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T10:02:41.453 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=9, tag_tid=2, entry_tid=1], [object_number=8, tag_tid=2, entry_tid=0], [object_number=11, tag_tid=1, entry_tid=9743], [object_number=10, tag_tid=1, entry_tid=9742]]], state=connected] 2024-04-06T10:02:41.453 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=9, tag_tid=1, entry_tid=9745], [object_number=8, tag_tid=1, entry_tid=9744], [object_number=11, tag_tid=1, entry_tid=9743], [object_number=10, tag_tid=1, entry_tid=9742]]], state=connected] 2024-04-06T10:02:41.456 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_21.status 2024-04-06T10:02:41.458 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=9, tag_tid=2, entry_tid=1' 2024-04-06T10:02:41.459 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror image_21 2024-04-06T10:02:41.459 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:02:41.459 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:02:41.459 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_21 2024-04-06T10:02:41.459 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror image_21 '..*' 2024-04-06T10:02:41.459 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:02:41.459 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:02:41.459 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_21 2024-04-06T10:02:41.459 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T10:02:41.459 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-image_21.status 2024-04-06T10:02:41.459 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-image_21.status 2024-04-06T10:02:41.459 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T10:02:41.460 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-image_21.status 2024-04-06T10:02:41.460 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_21 2024-04-06T10:02:41.460 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_21.status 2024-04-06T10:02:41.505 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 2 2024-04-06T10:02:41.505 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 2 2024-04-06T10:02:41.505 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T10:02:41.505 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=9, tag_tid=2, entry_tid=1], [object_number=8, tag_tid=2, entry_tid=0], [object_number=11, tag_tid=1, entry_tid=9743], [object_number=10, tag_tid=1, entry_tid=9742]]], state=connected] 2024-04-06T10:02:41.505 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=9, tag_tid=1, entry_tid=9745], [object_number=8, tag_tid=1, entry_tid=9744], [object_number=11, tag_tid=1, entry_tid=9743], [object_number=10, tag_tid=1, entry_tid=9742]]], state=connected] 2024-04-06T10:02:41.509 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_21.status 2024-04-06T10:02:41.510 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=9, tag_tid=1, entry_tid=9745' 2024-04-06T10:02:41.510 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=9, tag_tid=2, entry_tid=1' -a 'object_number=9, tag_tid=2, entry_tid=1' = 'object_number=9, tag_tid=1, entry_tid=9745' 2024-04-06T10:02:41.511 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=9, tag_tid=1, entry_tid=9745' '!=' 'object_number=9, tag_tid=1, entry_tid=9745' 2024-04-06T10:02:41.511 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T10:02:41.511 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 2 2024-04-06T10:02:43.512 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror image_21 2024-04-06T10:02:43.512 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T10:02:43.512 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:02:43.512 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_21 2024-04-06T10:02:43.513 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T10:02:43.513 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n image_21 ']' 2024-04-06T10:02:43.513 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/image_21' 2024-04-06T10:02:43.513 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_21 2024-04-06T10:02:43.513 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T10:02:43.513 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T10:02:43.513 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T10:02:43.513 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T10:02:43.513 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T10:02:43.513 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T10:02:43.513 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T10:02:43.514 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T10:02:43.514 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T10:02:43.516 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T10:02:43.516 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_21 2024-04-06T10:02:43.516 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T10:02:43.516 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T10:02:43.516 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T10:02:43.516 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T10:02:43.516 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T10:02:43.516 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T10:02:43.517 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T10:02:43.517 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T10:02:43.517 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T10:02:43.517 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T10:02:43.517 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T10:02:43.517 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:02:43.517 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T10:02:43.517 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T10:02:43.518 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T10:02:43.518 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T10:02:43.518 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T10:02:43.518 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T10:02:43.519 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T10:02:43.519 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T10:02:43.519 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T10:02:43.519 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T10:02:43.519 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:02:43.519 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T10:02:43.519 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T10:02:43.519 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T10:02:43.519 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T10:02:43.520 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T10:02:43.520 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T10:02:43.520 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T10:02:43.543 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T10:02:43.543 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T10:02:43.543 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T10:02:43.543 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_21 2024-04-06T10:02:43.656 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:Invalid command: unused arguments: ['mirror/image_21'] 2024-04-06T10:02:43.656 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:rbd mirror flush : flush rbd mirror 2024-04-06T10:02:43.656 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:admin_socket: invalid command 2024-04-06T10:02:43.658 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T10:02:43.659 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/image_21 2024-04-06T10:02:43.659 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:1 2024-04-06T10:02:43.659 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T10:02:43.659 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T10:02:43.659 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-06T10:02:43.659 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:1 2024-04-06T10:02:43.659 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T10:02:43.659 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T10:02:43.659 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T10:02:43.659 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=1 2024-04-06T10:02:43.659 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-06T10:02:43.660 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T10:02:43.660 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:02:43.660 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=1 2024-04-06T10:02:43.660 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=1 2024-04-06T10:02:43.661 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-06T10:02:43.661 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:1 2024-04-06T10:02:43.661 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T10:02:43.661 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T10:02:43.661 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-06T10:02:43.661 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:1 2024-04-06T10:02:43.661 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T10:02:43.661 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T10:02:43.662 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:02:43.662 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=1 2024-04-06T10:02:43.662 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-06T10:02:43.662 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T10:02:43.662 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T10:02:43.662 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=1 2024-04-06T10:02:43.662 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=1 2024-04-06T10:02:43.662 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-06T10:02:43.684 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T10:02:43.684 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T10:02:43.684 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T10:02:43.685 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/image_21 2024-04-06T10:02:43.834 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T10:02:43.835 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror image_21 2024-04-06T10:02:43.835 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:02:43.835 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:02:43.835 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_21 2024-04-06T10:02:43.836 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror image_21 '' 2024-04-06T10:02:43.836 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:02:43.836 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:02:43.836 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_21 2024-04-06T10:02:43.836 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T10:02:43.836 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-image_21.status 2024-04-06T10:02:43.836 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-image_21.status 2024-04-06T10:02:43.837 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T10:02:43.838 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-image_21.status 2024-04-06T10:02:43.838 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_21 2024-04-06T10:02:43.838 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_21.status 2024-04-06T10:02:43.883 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 2 2024-04-06T10:02:43.883 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 2 2024-04-06T10:02:43.883 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T10:02:43.884 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=9, tag_tid=2, entry_tid=1], [object_number=8, tag_tid=2, entry_tid=0], [object_number=11, tag_tid=1, entry_tid=9743], [object_number=10, tag_tid=1, entry_tid=9742]]], state=connected] 2024-04-06T10:02:43.884 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=9, tag_tid=1, entry_tid=9745], [object_number=8, tag_tid=1, entry_tid=9744], [object_number=11, tag_tid=1, entry_tid=9743], [object_number=10, tag_tid=1, entry_tid=9742]]], state=connected] 2024-04-06T10:02:43.887 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_21.status 2024-04-06T10:02:43.888 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=9, tag_tid=2, entry_tid=1' 2024-04-06T10:02:43.889 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror image_21 2024-04-06T10:02:43.889 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:02:43.889 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:02:43.889 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_21 2024-04-06T10:02:43.889 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror image_21 '..*' 2024-04-06T10:02:43.890 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:02:43.890 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:02:43.890 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_21 2024-04-06T10:02:43.890 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T10:02:43.890 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-image_21.status 2024-04-06T10:02:43.890 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-image_21.status 2024-04-06T10:02:43.890 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T10:02:43.890 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-image_21.status 2024-04-06T10:02:43.891 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_21 2024-04-06T10:02:43.891 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_21.status 2024-04-06T10:02:43.936 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 2 2024-04-06T10:02:43.936 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 2 2024-04-06T10:02:43.936 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T10:02:43.936 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=9, tag_tid=2, entry_tid=1], [object_number=8, tag_tid=2, entry_tid=0], [object_number=11, tag_tid=1, entry_tid=9743], [object_number=10, tag_tid=1, entry_tid=9742]]], state=connected] 2024-04-06T10:02:43.936 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=9, tag_tid=1, entry_tid=9745], [object_number=8, tag_tid=1, entry_tid=9744], [object_number=11, tag_tid=1, entry_tid=9743], [object_number=10, tag_tid=1, entry_tid=9742]]], state=connected] 2024-04-06T10:02:43.940 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_21.status 2024-04-06T10:02:43.941 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=9, tag_tid=1, entry_tid=9745' 2024-04-06T10:02:43.942 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=9, tag_tid=2, entry_tid=1' -a 'object_number=9, tag_tid=2, entry_tid=1' = 'object_number=9, tag_tid=1, entry_tid=9745' 2024-04-06T10:02:43.942 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=9, tag_tid=1, entry_tid=9745' '!=' 'object_number=9, tag_tid=1, entry_tid=9745' 2024-04-06T10:02:43.942 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T10:02:43.942 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 2 2024-04-06T10:02:45.943 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror image_21 2024-04-06T10:02:45.944 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T10:02:45.944 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:02:45.944 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_21 2024-04-06T10:02:45.944 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T10:02:45.944 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n image_21 ']' 2024-04-06T10:02:45.944 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/image_21' 2024-04-06T10:02:45.944 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_21 2024-04-06T10:02:45.944 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T10:02:45.944 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T10:02:45.944 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T10:02:45.944 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T10:02:45.944 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T10:02:45.944 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T10:02:45.944 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T10:02:45.945 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T10:02:45.945 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T10:02:45.947 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T10:02:45.948 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_21 2024-04-06T10:02:45.948 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T10:02:45.948 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T10:02:45.948 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T10:02:45.948 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T10:02:45.948 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T10:02:45.948 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T10:02:45.948 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T10:02:45.948 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T10:02:45.948 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T10:02:45.948 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T10:02:45.948 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T10:02:45.948 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:02:45.949 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T10:02:45.949 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T10:02:45.949 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T10:02:45.950 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T10:02:45.950 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T10:02:45.950 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T10:02:45.950 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T10:02:45.950 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T10:02:45.950 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T10:02:45.950 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T10:02:45.950 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:02:45.950 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T10:02:45.950 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T10:02:45.950 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T10:02:45.950 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T10:02:45.950 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T10:02:45.951 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T10:02:45.951 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T10:02:45.976 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T10:02:45.976 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T10:02:45.976 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T10:02:45.977 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_21 2024-04-06T10:02:46.080 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:Invalid command: unused arguments: ['mirror/image_21'] 2024-04-06T10:02:46.080 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:rbd mirror flush : flush rbd mirror 2024-04-06T10:02:46.081 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:admin_socket: invalid command 2024-04-06T10:02:46.085 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T10:02:46.085 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/image_21 2024-04-06T10:02:46.085 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:1 2024-04-06T10:02:46.085 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T10:02:46.085 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T10:02:46.085 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-06T10:02:46.085 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:1 2024-04-06T10:02:46.085 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T10:02:46.085 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T10:02:46.085 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T10:02:46.085 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=1 2024-04-06T10:02:46.085 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-06T10:02:46.085 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T10:02:46.086 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:02:46.086 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=1 2024-04-06T10:02:46.086 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=1 2024-04-06T10:02:46.086 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-06T10:02:46.086 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:1 2024-04-06T10:02:46.086 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T10:02:46.086 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T10:02:46.086 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-06T10:02:46.086 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:1 2024-04-06T10:02:46.086 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T10:02:46.086 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T10:02:46.086 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:02:46.086 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=1 2024-04-06T10:02:46.086 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-06T10:02:46.087 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T10:02:46.087 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T10:02:46.087 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=1 2024-04-06T10:02:46.087 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=1 2024-04-06T10:02:46.087 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-06T10:02:46.108 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T10:02:46.108 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T10:02:46.108 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T10:02:46.108 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/image_21 2024-04-06T10:02:46.256 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T10:02:46.257 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror image_21 2024-04-06T10:02:46.257 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:02:46.257 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:02:46.257 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_21 2024-04-06T10:02:46.257 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror image_21 '' 2024-04-06T10:02:46.257 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:02:46.257 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:02:46.258 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_21 2024-04-06T10:02:46.258 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T10:02:46.258 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-image_21.status 2024-04-06T10:02:46.258 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-image_21.status 2024-04-06T10:02:46.258 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T10:02:46.261 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-image_21.status 2024-04-06T10:02:46.261 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_21 2024-04-06T10:02:46.262 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_21.status 2024-04-06T10:02:46.312 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 2 2024-04-06T10:02:46.312 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 2 2024-04-06T10:02:46.312 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T10:02:46.313 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=9, tag_tid=2, entry_tid=1], [object_number=8, tag_tid=2, entry_tid=0], [object_number=11, tag_tid=1, entry_tid=9743], [object_number=10, tag_tid=1, entry_tid=9742]]], state=connected] 2024-04-06T10:02:46.313 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=9, tag_tid=2, entry_tid=1], [object_number=8, tag_tid=2, entry_tid=0], [object_number=11, tag_tid=1, entry_tid=9743], [object_number=10, tag_tid=1, entry_tid=9742]]], state=connected] 2024-04-06T10:02:46.316 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_21.status 2024-04-06T10:02:46.317 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=9, tag_tid=2, entry_tid=1' 2024-04-06T10:02:46.318 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror image_21 2024-04-06T10:02:46.319 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:02:46.319 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:02:46.319 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_21 2024-04-06T10:02:46.319 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror image_21 '..*' 2024-04-06T10:02:46.319 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:02:46.319 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:02:46.319 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_21 2024-04-06T10:02:46.319 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T10:02:46.319 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-image_21.status 2024-04-06T10:02:46.319 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-image_21.status 2024-04-06T10:02:46.319 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T10:02:46.320 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-image_21.status 2024-04-06T10:02:46.320 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_21 2024-04-06T10:02:46.320 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_21.status 2024-04-06T10:02:46.365 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 2 2024-04-06T10:02:46.365 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 2 2024-04-06T10:02:46.365 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T10:02:46.365 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=9, tag_tid=2, entry_tid=1], [object_number=8, tag_tid=2, entry_tid=0], [object_number=11, tag_tid=1, entry_tid=9743], [object_number=10, tag_tid=1, entry_tid=9742]]], state=connected] 2024-04-06T10:02:46.365 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=9, tag_tid=2, entry_tid=1], [object_number=8, tag_tid=2, entry_tid=0], [object_number=11, tag_tid=1, entry_tid=9743], [object_number=10, tag_tid=1, entry_tid=9742]]], state=connected] 2024-04-06T10:02:46.369 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_21.status 2024-04-06T10:02:46.371 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=9, tag_tid=2, entry_tid=1' 2024-04-06T10:02:46.371 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=9, tag_tid=2, entry_tid=1' -a 'object_number=9, tag_tid=2, entry_tid=1' = 'object_number=9, tag_tid=2, entry_tid=1' 2024-04-06T10:02:46.371 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T10:02:46.371 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_snap_present cluster1 mirror image_21 snap 2024-04-06T10:02:46.371 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T10:02:46.371 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:02:46.371 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_21 2024-04-06T10:02:46.371 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local snap_name=snap 2024-04-06T10:02:46.371 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local s 2024-04-06T10:02:46.371 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-04-06T10:02:46.372 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 1 2024-04-06T10:02:47.372 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 info mirror/image_21@snap 2024-04-06T10:02:47.440 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:rbd image 'image_21': 2024-04-06T10:02:47.440 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: size 128 MiB in 32 objects 2024-04-06T10:02:47.440 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: order 22 (4 MiB objects) 2024-04-06T10:02:47.440 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: snapshot_count: 1 2024-04-06T10:02:47.440 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: id: 10ac80cb2567 2024-04-06T10:02:47.441 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: block_name_prefix: rbd_data.10ac80cb2567 2024-04-06T10:02:47.441 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: format: 2 2024-04-06T10:02:47.441 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: features: layering, exclusive-lock, journaling 2024-04-06T10:02:47.441 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: op_features: 2024-04-06T10:02:47.441 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: flags: 2024-04-06T10:02:47.441 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: create_timestamp: Sat Apr 6 09:53:54 2024 2024-04-06T10:02:47.441 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: access_timestamp: Sat Apr 6 09:53:54 2024 2024-04-06T10:02:47.441 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: modify_timestamp: Sat Apr 6 09:54:55 2024 2024-04-06T10:02:47.441 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: protected: False 2024-04-06T10:02:47.441 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: journal: 10ac80cb2567 2024-04-06T10:02:47.441 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: mirroring state: enabled 2024-04-06T10:02:47.441 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: mirroring mode: journal 2024-04-06T10:02:47.441 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: mirroring global id: 37a7e9c0-c21d-45e8-b609-9a32ca25ca42 2024-04-06T10:02:47.441 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: mirroring primary: false 2024-04-06T10:02:47.445 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T10:02:47.445 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ compare_image_snaps mirror image_21 snap 2024-04-06T10:02:47.445 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:02:47.445 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_21 2024-04-06T10:02:47.445 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local snap_name=snap 2024-04-06T10:02:47.445 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local ret=0 2024-04-06T10:02:47.446 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local rmt_export=/tmp/tmp.BhW6xKmROU/cluster2-mirror-image_21.export 2024-04-06T10:02:47.446 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loc_export=/tmp/tmp.BhW6xKmROU/cluster1-mirror-image_21.export 2024-04-06T10:02:47.446 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rm -f /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_21.export /tmp/tmp.BhW6xKmROU/cluster1-mirror-image_21.export 2024-04-06T10:02:47.446 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster2 -p mirror export image_21@snap /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_21.export 2024-04-06T10:02:47.618 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-04-06T10:02:47.626 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 -p mirror export image_21@snap /tmp/tmp.BhW6xKmROU/cluster1-mirror-image_21.export 2024-04-06T10:02:47.942 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-04-06T10:02:47.955 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmp /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_21.export /tmp/tmp.BhW6xKmROU/cluster1-mirror-image_21.export 2024-04-06T10:02:48.034 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rm -f /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_21.export /tmp/tmp.BhW6xKmROU/cluster1-mirror-image_21.export 2024-04-06T10:02:48.076 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T10:02:48.076 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-06T10:02:48.076 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ image=image_22 2024-04-06T10:02:48.076 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ create_snap cluster2 mirror image_22 snap 2024-04-06T10:02:48.076 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-04-06T10:02:48.076 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:02:48.076 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_22 2024-04-06T10:02:48.076 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local snap_name=snap 2024-04-06T10:02:48.076 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster2 -p mirror snap create image_22@snap --debug-rbd=20 --debug-journaler=20 2024-04-06T10:02:49.882 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_image_replay_started cluster1 mirror image_22 2024-04-06T10:02:49.882 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T10:02:49.882 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:02:49.882 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_22 2024-04-06T10:02:49.882 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_image_replay_state cluster1 mirror image_22 started 2024-04-06T10:02:49.883 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T10:02:49.883 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:02:49.883 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_22 2024-04-06T10:02:49.883 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state=started 2024-04-06T10:02:49.883 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local s 2024-04-06T10:02:49.883 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T10:02:49.883 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 1 2024-04-06T10:02:50.884 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test_image_replay_state cluster1 mirror image_22 started 2024-04-06T10:02:50.884 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T10:02:50.884 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:02:50.884 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_22 2024-04-06T10:02:50.884 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local test_state=started 2024-04-06T10:02:50.884 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local status_result 2024-04-06T10:02:50.884 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local current_state=stopped 2024-04-06T10:02:50.885 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_22 2024-04-06T10:02:50.886 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_instance=cluster1 2024-04-06T10:02:50.886 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ shift 2024-04-06T10:02:50.886 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T10:02:50.886 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -i state 2024-04-06T10:02:50.886 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance=cluster1 2024-04-06T10:02:50.886 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local loop_instance 2024-04-06T10:02:50.886 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T10:02:50.886 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sleep 0 2024-04-06T10:02:50.887 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-04-06T10:02:50.887 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ seq 0 3 2024-04-06T10:02:50.889 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T10:02:50.889 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_22 2024-04-06T10:02:50.890 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1:0 2024-04-06T10:02:50.890 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ shift 2024-04-06T10:02:50.890 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T10:02:50.890 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-04-06T10:02:50.890 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T10:02:50.890 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=cluster 2024-04-06T10:02:50.890 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T10:02:50.890 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:02:50.890 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T10:02:50.890 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T10:02:50.891 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval cluster=cluster1 2024-04-06T10:02:50.891 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ cluster=cluster1 2024-04-06T10:02:50.891 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T10:02:50.891 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T10:02:50.891 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-04-06T10:02:50.891 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local local_cluster=cluster1:0 2024-04-06T10:02:50.891 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local cluster=cluster1 2024-04-06T10:02:50.891 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local instance 2024-04-06T10:02:50.891 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T10:02:50.891 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local val=cluster1:0 2024-04-06T10:02:50.891 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local cluster_var_name=local_cluster 2024-04-06T10:02:50.891 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local instance_var_name=instance 2024-04-06T10:02:50.891 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ cluster=cluster1 2024-04-06T10:02:50.891 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T10:02:50.892 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ '[' 0 = cluster1:0 ']' 2024-04-06T10:02:50.892 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ eval local_cluster=cluster1 2024-04-06T10:02:50.892 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++++ local_cluster=cluster1 2024-04-06T10:02:50.892 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ eval instance=0 2024-04-06T10:02:50.892 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++++ instance=0 2024-04-06T10:02:50.892 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T10:02:50.915 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T10:02:50.915 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T10:02:50.915 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T10:02:50.915 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_22 2024-04-06T10:02:51.025 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:Invalid command: unused arguments: ['mirror/image_22'] 2024-04-06T10:02:51.025 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:rbd mirror status : get status for rbd mirror 2024-04-06T10:02:51.025 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:admin_socket: invalid command 2024-04-06T10:02:51.027 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T10:02:51.027 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_22 2024-04-06T10:02:51.027 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1:1 2024-04-06T10:02:51.027 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ shift 2024-04-06T10:02:51.027 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T10:02:51.027 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-04-06T10:02:51.027 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:1 2024-04-06T10:02:51.027 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=cluster 2024-04-06T10:02:51.028 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T10:02:51.028 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:02:51.028 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=1 2024-04-06T10:02:51.028 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-06T10:02:51.028 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval cluster=cluster1 2024-04-06T10:02:51.028 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ cluster=cluster1 2024-04-06T10:02:51.028 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=1 2024-04-06T10:02:51.028 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=1 2024-04-06T10:02:51.029 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-04-06T10:02:51.029 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local local_cluster=cluster1:1 2024-04-06T10:02:51.030 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local cluster=cluster1 2024-04-06T10:02:51.030 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local instance 2024-04-06T10:02:51.030 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-06T10:02:51.030 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local val=cluster1:1 2024-04-06T10:02:51.030 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local cluster_var_name=local_cluster 2024-04-06T10:02:51.030 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local instance_var_name=instance 2024-04-06T10:02:51.030 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ cluster=cluster1 2024-04-06T10:02:51.030 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=1 2024-04-06T10:02:51.030 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ '[' 1 = cluster1:1 ']' 2024-04-06T10:02:51.030 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ eval local_cluster=cluster1 2024-04-06T10:02:51.030 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++++ local_cluster=cluster1 2024-04-06T10:02:51.030 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ eval instance=1 2024-04-06T10:02:51.030 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++++ instance=1 2024-04-06T10:02:51.031 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-06T10:02:51.052 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T10:02:51.052 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T10:02:51.052 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T10:02:51.052 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_22 2024-04-06T10:02:51.205 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ return 0 2024-04-06T10:02:51.206 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ status_result=' "state": "Replaying"' 2024-04-06T10:02:51.206 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ echo ' "state": "Replaying"' 2024-04-06T10:02:51.207 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep -i Replaying 2024-04-06T10:02:51.208 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: "state": "Replaying" 2024-04-06T10:02:51.210 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ current_state=started 2024-04-06T10:02:51.210 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test started = started 2024-04-06T10:02:51.210 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T10:02:51.210 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_22 2024-04-06T10:02:51.210 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local local_cluster=cluster1 2024-04-06T10:02:51.210 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-04-06T10:02:51.210 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:02:51.210 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_22 2024-04-06T10:02:51.210 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' journal = journal ']' 2024-04-06T10:02:51.210 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_journal_replay_complete cluster1 cluster2 mirror image_22 2024-04-06T10:02:51.210 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local local_cluster=cluster1 2024-04-06T10:02:51.211 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-04-06T10:02:51.211 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:02:51.211 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_22 2024-04-06T10:02:51.211 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local s master_pos mirror_pos last_mirror_pos 2024-04-06T10:02:51.211 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local master_tag master_entry mirror_tag mirror_entry 2024-04-06T10:02:51.211 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T10:02:51.211 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T10:02:51.211 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T10:02:51.412 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror image_22 2024-04-06T10:02:51.412 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T10:02:51.412 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:02:51.412 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_22 2024-04-06T10:02:51.412 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T10:02:51.412 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n image_22 ']' 2024-04-06T10:02:51.412 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/image_22' 2024-04-06T10:02:51.412 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_22 2024-04-06T10:02:51.412 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T10:02:51.413 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T10:02:51.413 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T10:02:51.413 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T10:02:51.413 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T10:02:51.413 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T10:02:51.413 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T10:02:51.416 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T10:02:51.416 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T10:02:51.416 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T10:02:51.416 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_22 2024-04-06T10:02:51.416 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T10:02:51.416 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T10:02:51.416 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T10:02:51.416 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T10:02:51.416 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T10:02:51.416 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T10:02:51.416 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T10:02:51.417 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T10:02:51.417 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T10:02:51.417 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T10:02:51.417 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T10:02:51.417 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:02:51.417 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T10:02:51.417 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T10:02:51.417 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T10:02:51.417 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T10:02:51.417 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T10:02:51.417 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T10:02:51.417 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T10:02:51.417 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T10:02:51.417 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T10:02:51.417 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T10:02:51.418 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:02:51.418 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T10:02:51.418 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T10:02:51.418 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T10:02:51.418 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T10:02:51.418 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T10:02:51.418 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T10:02:51.418 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T10:02:51.442 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T10:02:51.443 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T10:02:51.443 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T10:02:51.443 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_22 2024-04-06T10:02:51.548 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:Invalid command: unused arguments: ['mirror/image_22'] 2024-04-06T10:02:51.548 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:rbd mirror flush : flush rbd mirror 2024-04-06T10:02:51.548 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:admin_socket: invalid command 2024-04-06T10:02:51.551 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T10:02:51.551 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/image_22 2024-04-06T10:02:51.551 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:1 2024-04-06T10:02:51.551 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T10:02:51.551 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T10:02:51.551 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-06T10:02:51.551 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:1 2024-04-06T10:02:51.551 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T10:02:51.551 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T10:02:51.551 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T10:02:51.551 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=1 2024-04-06T10:02:51.551 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-06T10:02:51.552 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T10:02:51.552 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:02:51.552 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=1 2024-04-06T10:02:51.552 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=1 2024-04-06T10:02:51.553 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-06T10:02:51.553 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:1 2024-04-06T10:02:51.553 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T10:02:51.553 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T10:02:51.553 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-06T10:02:51.553 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:1 2024-04-06T10:02:51.554 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T10:02:51.554 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T10:02:51.554 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:02:51.554 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=1 2024-04-06T10:02:51.554 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-06T10:02:51.554 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T10:02:51.554 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T10:02:51.554 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=1 2024-04-06T10:02:51.554 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=1 2024-04-06T10:02:51.554 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-06T10:02:51.578 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T10:02:51.579 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T10:02:51.579 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T10:02:51.579 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/image_22 2024-04-06T10:02:51.712 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T10:02:51.713 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror image_22 2024-04-06T10:02:51.713 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:02:51.713 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:02:51.713 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_22 2024-04-06T10:02:51.713 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror image_22 '' 2024-04-06T10:02:51.713 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:02:51.714 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:02:51.714 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_22 2024-04-06T10:02:51.714 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T10:02:51.715 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-image_22.status 2024-04-06T10:02:51.715 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-image_22.status 2024-04-06T10:02:51.715 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T10:02:51.716 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-image_22.status 2024-04-06T10:02:51.717 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_22.status 2024-04-06T10:02:51.717 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_22 2024-04-06T10:02:51.775 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 3 2024-04-06T10:02:51.775 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 3 2024-04-06T10:02:51.775 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T10:02:51.775 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=13, tag_tid=2, entry_tid=1], [object_number=12, tag_tid=2, entry_tid=0], [object_number=14, tag_tid=1, entry_tid=13174], [object_number=15, tag_tid=1, entry_tid=13171]]], state=connected] 2024-04-06T10:02:51.775 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=14, tag_tid=1, entry_tid=13174], [object_number=13, tag_tid=1, entry_tid=13173], [object_number=12, tag_tid=1, entry_tid=13172], [object_number=15, tag_tid=1, entry_tid=13171]]], state=connected] 2024-04-06T10:02:51.778 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_22.status 2024-04-06T10:02:51.780 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=13, tag_tid=2, entry_tid=1' 2024-04-06T10:02:51.781 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror image_22 2024-04-06T10:02:51.781 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:02:51.781 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:02:51.781 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_22 2024-04-06T10:02:51.781 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror image_22 '..*' 2024-04-06T10:02:51.782 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:02:51.782 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:02:51.782 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_22 2024-04-06T10:02:51.782 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T10:02:51.782 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-image_22.status 2024-04-06T10:02:51.782 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-image_22.status 2024-04-06T10:02:51.782 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T10:02:51.783 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-image_22.status 2024-04-06T10:02:51.783 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_22 2024-04-06T10:02:51.783 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_22.status 2024-04-06T10:02:51.831 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 3 2024-04-06T10:02:51.831 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 3 2024-04-06T10:02:51.831 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T10:02:51.831 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=13, tag_tid=2, entry_tid=1], [object_number=12, tag_tid=2, entry_tid=0], [object_number=14, tag_tid=1, entry_tid=13174], [object_number=15, tag_tid=1, entry_tid=13171]]], state=connected] 2024-04-06T10:02:51.831 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=14, tag_tid=1, entry_tid=13174], [object_number=13, tag_tid=1, entry_tid=13173], [object_number=12, tag_tid=1, entry_tid=13172], [object_number=15, tag_tid=1, entry_tid=13171]]], state=connected] 2024-04-06T10:02:51.835 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_22.status 2024-04-06T10:02:51.836 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=14, tag_tid=1, entry_tid=13174' 2024-04-06T10:02:51.836 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=13, tag_tid=2, entry_tid=1' -a 'object_number=13, tag_tid=2, entry_tid=1' = 'object_number=14, tag_tid=1, entry_tid=13174' 2024-04-06T10:02:51.836 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=14, tag_tid=1, entry_tid=13174' '!=' '' 2024-04-06T10:02:51.837 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T10:02:51.837 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=14, tag_tid=1, entry_tid=13174' = '' 2024-04-06T10:02:51.837 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=14, tag_tid=1, entry_tid=13174' 2024-04-06T10:02:51.837 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=13, tag_tid=2, entry_tid=1' 2024-04-06T10:02:51.838 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T10:02:51.838 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T10:02:51.839 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=2 2024-04-06T10:02:51.839 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=14, tag_tid=1, entry_tid=13174' 2024-04-06T10:02:51.839 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T10:02:51.839 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T10:02:51.840 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=1 2024-04-06T10:02:51.840 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=13, tag_tid=2, entry_tid=1' 2024-04-06T10:02:51.840 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T10:02:51.841 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T10:02:51.841 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T10:02:51.842 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=14, tag_tid=1, entry_tid=13174' 2024-04-06T10:02:51.842 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T10:02:51.842 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T10:02:51.844 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=13174 2024-04-06T10:02:51.844 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 2 = 1 -a 1 -le 13174 2024-04-06T10:02:51.844 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T10:02:51.844 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T10:02:51.844 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T10:02:52.045 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror image_22 2024-04-06T10:02:52.045 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T10:02:52.045 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:02:52.045 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_22 2024-04-06T10:02:52.045 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T10:02:52.046 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n image_22 ']' 2024-04-06T10:02:52.046 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/image_22' 2024-04-06T10:02:52.046 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_22 2024-04-06T10:02:52.046 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T10:02:52.046 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T10:02:52.046 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T10:02:52.046 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T10:02:52.046 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T10:02:52.046 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T10:02:52.046 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T10:02:52.047 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T10:02:52.047 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T10:02:52.051 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T10:02:52.051 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_22 2024-04-06T10:02:52.051 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T10:02:52.051 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T10:02:52.051 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T10:02:52.052 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T10:02:52.052 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T10:02:52.052 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T10:02:52.052 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T10:02:52.052 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T10:02:52.052 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T10:02:52.052 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T10:02:52.052 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T10:02:52.052 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:02:52.052 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T10:02:52.052 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T10:02:52.052 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T10:02:52.052 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T10:02:52.052 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T10:02:52.052 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T10:02:52.053 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T10:02:52.053 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T10:02:52.053 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T10:02:52.053 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T10:02:52.053 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:02:52.053 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T10:02:52.053 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T10:02:52.053 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T10:02:52.053 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T10:02:52.053 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T10:02:52.053 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T10:02:52.053 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T10:02:52.077 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T10:02:52.077 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T10:02:52.077 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T10:02:52.077 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_22 2024-04-06T10:02:52.179 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:Invalid command: unused arguments: ['mirror/image_22'] 2024-04-06T10:02:52.179 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:rbd mirror flush : flush rbd mirror 2024-04-06T10:02:52.179 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:admin_socket: invalid command 2024-04-06T10:02:52.182 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T10:02:52.182 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/image_22 2024-04-06T10:02:52.182 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:1 2024-04-06T10:02:52.182 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T10:02:52.182 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T10:02:52.182 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-06T10:02:52.182 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:1 2024-04-06T10:02:52.182 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T10:02:52.182 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T10:02:52.182 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T10:02:52.182 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=1 2024-04-06T10:02:52.182 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-06T10:02:52.183 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T10:02:52.183 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:02:52.183 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=1 2024-04-06T10:02:52.183 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=1 2024-04-06T10:02:52.183 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-06T10:02:52.183 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:1 2024-04-06T10:02:52.183 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T10:02:52.183 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T10:02:52.183 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-06T10:02:52.183 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:1 2024-04-06T10:02:52.183 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T10:02:52.183 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T10:02:52.183 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:02:52.183 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=1 2024-04-06T10:02:52.183 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-06T10:02:52.184 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T10:02:52.184 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T10:02:52.184 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=1 2024-04-06T10:02:52.184 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=1 2024-04-06T10:02:52.184 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-06T10:02:52.207 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T10:02:52.207 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T10:02:52.207 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T10:02:52.207 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/image_22 2024-04-06T10:02:52.350 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T10:02:52.351 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror image_22 2024-04-06T10:02:52.351 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:02:52.351 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:02:52.351 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_22 2024-04-06T10:02:52.351 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror image_22 '' 2024-04-06T10:02:52.351 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:02:52.352 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:02:52.352 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_22 2024-04-06T10:02:52.352 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T10:02:52.352 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-image_22.status 2024-04-06T10:02:52.352 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-image_22.status 2024-04-06T10:02:52.352 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T10:02:52.354 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-image_22.status 2024-04-06T10:02:52.355 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_22 2024-04-06T10:02:52.355 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_22.status 2024-04-06T10:02:52.400 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 3 2024-04-06T10:02:52.401 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 3 2024-04-06T10:02:52.401 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T10:02:52.401 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=13, tag_tid=2, entry_tid=1], [object_number=12, tag_tid=2, entry_tid=0], [object_number=14, tag_tid=1, entry_tid=13174], [object_number=15, tag_tid=1, entry_tid=13171]]], state=connected] 2024-04-06T10:02:52.401 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=14, tag_tid=1, entry_tid=13174], [object_number=13, tag_tid=1, entry_tid=13173], [object_number=12, tag_tid=1, entry_tid=13172], [object_number=15, tag_tid=1, entry_tid=13171]]], state=connected] 2024-04-06T10:02:52.404 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_22.status 2024-04-06T10:02:52.405 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=13, tag_tid=2, entry_tid=1' 2024-04-06T10:02:52.406 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror image_22 2024-04-06T10:02:52.406 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:02:52.406 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:02:52.407 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_22 2024-04-06T10:02:52.407 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror image_22 '..*' 2024-04-06T10:02:52.407 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:02:52.407 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:02:52.407 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_22 2024-04-06T10:02:52.407 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T10:02:52.407 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-image_22.status 2024-04-06T10:02:52.407 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-image_22.status 2024-04-06T10:02:52.407 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T10:02:52.408 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-image_22.status 2024-04-06T10:02:52.408 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_22 2024-04-06T10:02:52.408 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_22.status 2024-04-06T10:02:52.452 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 3 2024-04-06T10:02:52.453 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 3 2024-04-06T10:02:52.453 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T10:02:52.453 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=13, tag_tid=2, entry_tid=1], [object_number=12, tag_tid=2, entry_tid=0], [object_number=14, tag_tid=1, entry_tid=13174], [object_number=15, tag_tid=1, entry_tid=13171]]], state=connected] 2024-04-06T10:02:52.453 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=14, tag_tid=1, entry_tid=13174], [object_number=13, tag_tid=1, entry_tid=13173], [object_number=12, tag_tid=1, entry_tid=13172], [object_number=15, tag_tid=1, entry_tid=13171]]], state=connected] 2024-04-06T10:02:52.456 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_22.status 2024-04-06T10:02:52.458 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=14, tag_tid=1, entry_tid=13174' 2024-04-06T10:02:52.458 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=13, tag_tid=2, entry_tid=1' -a 'object_number=13, tag_tid=2, entry_tid=1' = 'object_number=14, tag_tid=1, entry_tid=13174' 2024-04-06T10:02:52.458 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=14, tag_tid=1, entry_tid=13174' '!=' 'object_number=14, tag_tid=1, entry_tid=13174' 2024-04-06T10:02:52.459 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T10:02:52.459 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.4 2024-04-06T10:02:52.860 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror image_22 2024-04-06T10:02:52.860 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T10:02:52.860 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:02:52.860 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_22 2024-04-06T10:02:52.860 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T10:02:52.860 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n image_22 ']' 2024-04-06T10:02:52.861 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/image_22' 2024-04-06T10:02:52.861 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_22 2024-04-06T10:02:52.861 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T10:02:52.861 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T10:02:52.861 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T10:02:52.861 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T10:02:52.861 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T10:02:52.861 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T10:02:52.861 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T10:02:52.861 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T10:02:52.862 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T10:02:52.865 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T10:02:52.865 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_22 2024-04-06T10:02:52.866 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T10:02:52.866 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T10:02:52.866 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T10:02:52.866 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T10:02:52.866 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T10:02:52.866 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T10:02:52.866 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T10:02:52.866 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T10:02:52.866 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T10:02:52.866 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T10:02:52.866 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T10:02:52.866 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:02:52.866 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T10:02:52.866 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T10:02:52.867 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T10:02:52.867 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T10:02:52.867 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T10:02:52.867 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T10:02:52.867 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T10:02:52.867 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T10:02:52.867 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T10:02:52.867 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T10:02:52.867 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:02:52.867 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T10:02:52.868 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T10:02:52.868 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T10:02:52.868 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T10:02:52.869 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T10:02:52.869 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T10:02:52.869 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T10:02:52.892 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T10:02:52.893 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T10:02:52.893 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T10:02:52.893 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_22 2024-04-06T10:02:52.998 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:Invalid command: unused arguments: ['mirror/image_22'] 2024-04-06T10:02:52.998 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:rbd mirror flush : flush rbd mirror 2024-04-06T10:02:52.998 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:admin_socket: invalid command 2024-04-06T10:02:53.001 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T10:02:53.001 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/image_22 2024-04-06T10:02:53.001 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:1 2024-04-06T10:02:53.001 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T10:02:53.001 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T10:02:53.001 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-06T10:02:53.001 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:1 2024-04-06T10:02:53.001 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T10:02:53.001 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T10:02:53.001 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T10:02:53.001 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=1 2024-04-06T10:02:53.001 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-06T10:02:53.002 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T10:02:53.002 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:02:53.002 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=1 2024-04-06T10:02:53.002 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=1 2024-04-06T10:02:53.003 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-06T10:02:53.003 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:1 2024-04-06T10:02:53.003 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T10:02:53.003 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T10:02:53.003 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-06T10:02:53.004 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:1 2024-04-06T10:02:53.004 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T10:02:53.004 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T10:02:53.004 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:02:53.004 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=1 2024-04-06T10:02:53.004 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-06T10:02:53.004 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T10:02:53.004 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T10:02:53.004 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=1 2024-04-06T10:02:53.004 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=1 2024-04-06T10:02:53.004 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-06T10:02:53.022 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T10:02:53.022 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T10:02:53.023 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T10:02:53.023 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/image_22 2024-04-06T10:02:53.167 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T10:02:53.168 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror image_22 2024-04-06T10:02:53.168 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:02:53.168 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:02:53.168 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_22 2024-04-06T10:02:53.169 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror image_22 '' 2024-04-06T10:02:53.169 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:02:53.169 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:02:53.169 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_22 2024-04-06T10:02:53.169 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T10:02:53.169 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-image_22.status 2024-04-06T10:02:53.169 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T10:02:53.169 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-image_22.status 2024-04-06T10:02:53.171 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-image_22.status 2024-04-06T10:02:53.171 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_22 2024-04-06T10:02:53.171 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_22.status 2024-04-06T10:02:53.423 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 3 2024-04-06T10:02:53.423 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 3 2024-04-06T10:02:53.423 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T10:02:53.423 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=13, tag_tid=2, entry_tid=1], [object_number=12, tag_tid=2, entry_tid=0], [object_number=14, tag_tid=1, entry_tid=13174], [object_number=15, tag_tid=1, entry_tid=13171]]], state=connected] 2024-04-06T10:02:53.424 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=14, tag_tid=1, entry_tid=13174], [object_number=13, tag_tid=1, entry_tid=13173], [object_number=12, tag_tid=1, entry_tid=13172], [object_number=15, tag_tid=1, entry_tid=13171]]], state=connected] 2024-04-06T10:02:53.430 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_22.status 2024-04-06T10:02:53.432 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=13, tag_tid=2, entry_tid=1' 2024-04-06T10:02:53.433 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror image_22 2024-04-06T10:02:53.433 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:02:53.433 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:02:53.433 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_22 2024-04-06T10:02:53.434 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror image_22 '..*' 2024-04-06T10:02:53.434 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:02:53.434 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:02:53.434 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_22 2024-04-06T10:02:53.434 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T10:02:53.434 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-image_22.status 2024-04-06T10:02:53.434 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-image_22.status 2024-04-06T10:02:53.434 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T10:02:53.436 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-image_22.status 2024-04-06T10:02:53.436 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_22 2024-04-06T10:02:53.436 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_22.status 2024-04-06T10:02:53.490 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 3 2024-04-06T10:02:53.491 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 3 2024-04-06T10:02:53.491 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T10:02:53.491 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=13, tag_tid=2, entry_tid=1], [object_number=12, tag_tid=2, entry_tid=0], [object_number=14, tag_tid=1, entry_tid=13174], [object_number=15, tag_tid=1, entry_tid=13171]]], state=connected] 2024-04-06T10:02:53.491 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=14, tag_tid=1, entry_tid=13174], [object_number=13, tag_tid=1, entry_tid=13173], [object_number=12, tag_tid=1, entry_tid=13172], [object_number=15, tag_tid=1, entry_tid=13171]]], state=connected] 2024-04-06T10:02:53.495 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_22.status 2024-04-06T10:02:53.496 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=14, tag_tid=1, entry_tid=13174' 2024-04-06T10:02:53.496 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=13, tag_tid=2, entry_tid=1' -a 'object_number=13, tag_tid=2, entry_tid=1' = 'object_number=14, tag_tid=1, entry_tid=13174' 2024-04-06T10:02:53.497 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=14, tag_tid=1, entry_tid=13174' '!=' 'object_number=14, tag_tid=1, entry_tid=13174' 2024-04-06T10:02:53.497 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T10:02:53.497 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.8 2024-04-06T10:02:54.299 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror image_22 2024-04-06T10:02:54.299 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T10:02:54.299 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:02:54.299 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_22 2024-04-06T10:02:54.299 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T10:02:54.299 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n image_22 ']' 2024-04-06T10:02:54.299 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/image_22' 2024-04-06T10:02:54.299 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_22 2024-04-06T10:02:54.299 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T10:02:54.300 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T10:02:54.300 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T10:02:54.300 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T10:02:54.300 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T10:02:54.300 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T10:02:54.300 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T10:02:54.300 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T10:02:54.300 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T10:02:54.303 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T10:02:54.303 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_22 2024-04-06T10:02:54.303 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T10:02:54.303 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T10:02:54.303 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T10:02:54.304 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T10:02:54.304 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T10:02:54.304 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T10:02:54.304 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T10:02:54.304 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T10:02:54.304 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T10:02:54.304 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T10:02:54.304 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T10:02:54.304 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:02:54.304 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T10:02:54.304 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T10:02:54.304 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T10:02:54.304 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T10:02:54.304 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T10:02:54.304 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T10:02:54.305 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T10:02:54.305 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T10:02:54.305 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T10:02:54.305 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T10:02:54.305 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:02:54.305 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T10:02:54.305 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T10:02:54.305 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T10:02:54.305 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T10:02:54.305 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T10:02:54.305 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T10:02:54.306 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T10:02:54.329 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T10:02:54.330 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T10:02:54.330 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T10:02:54.330 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_22 2024-04-06T10:02:54.434 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:Invalid command: unused arguments: ['mirror/image_22'] 2024-04-06T10:02:54.435 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:rbd mirror flush : flush rbd mirror 2024-04-06T10:02:54.435 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:admin_socket: invalid command 2024-04-06T10:02:54.437 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T10:02:54.437 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/image_22 2024-04-06T10:02:54.437 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:1 2024-04-06T10:02:54.437 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T10:02:54.437 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T10:02:54.437 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-06T10:02:54.437 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:1 2024-04-06T10:02:54.437 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T10:02:54.437 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T10:02:54.437 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T10:02:54.437 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=1 2024-04-06T10:02:54.437 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-06T10:02:54.437 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T10:02:54.438 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:02:54.438 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=1 2024-04-06T10:02:54.438 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=1 2024-04-06T10:02:54.439 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-06T10:02:54.439 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:1 2024-04-06T10:02:54.439 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T10:02:54.439 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T10:02:54.439 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-06T10:02:54.439 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:1 2024-04-06T10:02:54.439 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T10:02:54.440 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T10:02:54.440 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:02:54.440 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=1 2024-04-06T10:02:54.440 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-06T10:02:54.440 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T10:02:54.440 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T10:02:54.440 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=1 2024-04-06T10:02:54.440 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=1 2024-04-06T10:02:54.440 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-06T10:02:54.458 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T10:02:54.458 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T10:02:54.458 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T10:02:54.458 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/image_22 2024-04-06T10:02:54.602 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T10:02:54.603 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror image_22 2024-04-06T10:02:54.603 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:02:54.603 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:02:54.603 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_22 2024-04-06T10:02:54.603 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror image_22 '' 2024-04-06T10:02:54.603 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:02:54.603 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:02:54.603 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_22 2024-04-06T10:02:54.604 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T10:02:54.604 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-image_22.status 2024-04-06T10:02:54.604 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-image_22.status 2024-04-06T10:02:54.604 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T10:02:54.606 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-image_22.status 2024-04-06T10:02:54.607 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_22 2024-04-06T10:02:54.607 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_22.status 2024-04-06T10:02:54.658 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 3 2024-04-06T10:02:54.658 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 3 2024-04-06T10:02:54.658 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T10:02:54.658 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=13, tag_tid=2, entry_tid=1], [object_number=12, tag_tid=2, entry_tid=0], [object_number=14, tag_tid=1, entry_tid=13174], [object_number=15, tag_tid=1, entry_tid=13171]]], state=connected] 2024-04-06T10:02:54.658 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=14, tag_tid=1, entry_tid=13174], [object_number=13, tag_tid=1, entry_tid=13173], [object_number=12, tag_tid=1, entry_tid=13172], [object_number=15, tag_tid=1, entry_tid=13171]]], state=connected] 2024-04-06T10:02:54.660 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_22.status 2024-04-06T10:02:54.662 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=13, tag_tid=2, entry_tid=1' 2024-04-06T10:02:54.663 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror image_22 2024-04-06T10:02:54.663 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:02:54.663 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:02:54.663 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_22 2024-04-06T10:02:54.664 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror image_22 '..*' 2024-04-06T10:02:54.664 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:02:54.664 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:02:54.664 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_22 2024-04-06T10:02:54.664 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T10:02:54.664 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-image_22.status 2024-04-06T10:02:54.664 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-image_22.status 2024-04-06T10:02:54.664 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T10:02:54.665 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-image_22.status 2024-04-06T10:02:54.665 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_22 2024-04-06T10:02:54.665 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_22.status 2024-04-06T10:02:54.710 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 3 2024-04-06T10:02:54.710 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 3 2024-04-06T10:02:54.710 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T10:02:54.710 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=13, tag_tid=2, entry_tid=1], [object_number=12, tag_tid=2, entry_tid=0], [object_number=14, tag_tid=1, entry_tid=13174], [object_number=15, tag_tid=1, entry_tid=13171]]], state=connected] 2024-04-06T10:02:54.711 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=14, tag_tid=1, entry_tid=13174], [object_number=13, tag_tid=1, entry_tid=13173], [object_number=12, tag_tid=1, entry_tid=13172], [object_number=15, tag_tid=1, entry_tid=13171]]], state=connected] 2024-04-06T10:02:54.714 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_22.status 2024-04-06T10:02:54.716 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=14, tag_tid=1, entry_tid=13174' 2024-04-06T10:02:54.716 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=13, tag_tid=2, entry_tid=1' -a 'object_number=13, tag_tid=2, entry_tid=1' = 'object_number=14, tag_tid=1, entry_tid=13174' 2024-04-06T10:02:54.716 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=14, tag_tid=1, entry_tid=13174' '!=' 'object_number=14, tag_tid=1, entry_tid=13174' 2024-04-06T10:02:54.716 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T10:02:54.716 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 1.6 2024-04-06T10:02:56.318 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror image_22 2024-04-06T10:02:56.318 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T10:02:56.318 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:02:56.318 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_22 2024-04-06T10:02:56.318 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T10:02:56.318 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n image_22 ']' 2024-04-06T10:02:56.318 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/image_22' 2024-04-06T10:02:56.318 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_22 2024-04-06T10:02:56.318 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T10:02:56.318 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T10:02:56.318 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T10:02:56.319 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T10:02:56.319 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T10:02:56.319 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T10:02:56.319 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T10:02:56.319 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T10:02:56.319 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T10:02:56.323 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T10:02:56.323 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_22 2024-04-06T10:02:56.324 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T10:02:56.324 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T10:02:56.324 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T10:02:56.324 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T10:02:56.324 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T10:02:56.324 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T10:02:56.324 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T10:02:56.324 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T10:02:56.324 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T10:02:56.324 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T10:02:56.324 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T10:02:56.324 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:02:56.324 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T10:02:56.324 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T10:02:56.325 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T10:02:56.325 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T10:02:56.325 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T10:02:56.325 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T10:02:56.325 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T10:02:56.325 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T10:02:56.325 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T10:02:56.325 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T10:02:56.325 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:02:56.325 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T10:02:56.325 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T10:02:56.325 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T10:02:56.325 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T10:02:56.325 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T10:02:56.326 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T10:02:56.326 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T10:02:56.349 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T10:02:56.349 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T10:02:56.349 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T10:02:56.349 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_22 2024-04-06T10:02:56.454 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:Invalid command: unused arguments: ['mirror/image_22'] 2024-04-06T10:02:56.454 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:rbd mirror flush : flush rbd mirror 2024-04-06T10:02:56.454 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:admin_socket: invalid command 2024-04-06T10:02:56.456 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T10:02:56.456 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/image_22 2024-04-06T10:02:56.456 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:1 2024-04-06T10:02:56.456 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T10:02:56.456 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T10:02:56.456 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-06T10:02:56.457 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:1 2024-04-06T10:02:56.457 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T10:02:56.457 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T10:02:56.457 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T10:02:56.457 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=1 2024-04-06T10:02:56.457 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-06T10:02:56.457 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T10:02:56.457 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:02:56.457 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=1 2024-04-06T10:02:56.457 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=1 2024-04-06T10:02:56.458 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-06T10:02:56.458 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:1 2024-04-06T10:02:56.458 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T10:02:56.458 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T10:02:56.459 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-06T10:02:56.459 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:1 2024-04-06T10:02:56.459 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T10:02:56.459 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T10:02:56.459 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:02:56.459 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=1 2024-04-06T10:02:56.459 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-06T10:02:56.459 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T10:02:56.459 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T10:02:56.459 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=1 2024-04-06T10:02:56.459 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=1 2024-04-06T10:02:56.459 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-06T10:02:56.477 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T10:02:56.478 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T10:02:56.478 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T10:02:56.478 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/image_22 2024-04-06T10:02:56.628 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T10:02:56.629 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror image_22 2024-04-06T10:02:56.629 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:02:56.629 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:02:56.629 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_22 2024-04-06T10:02:56.629 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror image_22 '' 2024-04-06T10:02:56.629 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:02:56.629 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:02:56.630 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_22 2024-04-06T10:02:56.630 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T10:02:56.630 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-image_22.status 2024-04-06T10:02:56.630 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T10:02:56.630 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-image_22.status 2024-04-06T10:02:56.631 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-image_22.status 2024-04-06T10:02:56.632 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_22 2024-04-06T10:02:56.632 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_22.status 2024-04-06T10:02:56.686 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 3 2024-04-06T10:02:56.686 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 3 2024-04-06T10:02:56.686 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T10:02:56.686 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=13, tag_tid=2, entry_tid=1], [object_number=12, tag_tid=2, entry_tid=0], [object_number=14, tag_tid=1, entry_tid=13174], [object_number=15, tag_tid=1, entry_tid=13171]]], state=connected] 2024-04-06T10:02:56.686 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=13, tag_tid=2, entry_tid=1], [object_number=12, tag_tid=2, entry_tid=0], [object_number=14, tag_tid=1, entry_tid=13174], [object_number=15, tag_tid=1, entry_tid=13171]]], state=connected] 2024-04-06T10:02:56.689 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_22.status 2024-04-06T10:02:56.691 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=13, tag_tid=2, entry_tid=1' 2024-04-06T10:02:56.692 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror image_22 2024-04-06T10:02:56.692 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:02:56.692 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:02:56.692 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_22 2024-04-06T10:02:56.692 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror image_22 '..*' 2024-04-06T10:02:56.692 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:02:56.692 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:02:56.692 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_22 2024-04-06T10:02:56.692 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T10:02:56.692 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-image_22.status 2024-04-06T10:02:56.692 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-image_22.status 2024-04-06T10:02:56.693 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T10:02:56.693 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-image_22.status 2024-04-06T10:02:56.693 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_22 2024-04-06T10:02:56.694 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_22.status 2024-04-06T10:02:56.739 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 3 2024-04-06T10:02:56.739 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 3 2024-04-06T10:02:56.739 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T10:02:56.739 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=13, tag_tid=2, entry_tid=1], [object_number=12, tag_tid=2, entry_tid=0], [object_number=14, tag_tid=1, entry_tid=13174], [object_number=15, tag_tid=1, entry_tid=13171]]], state=connected] 2024-04-06T10:02:56.739 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=13, tag_tid=2, entry_tid=1], [object_number=12, tag_tid=2, entry_tid=0], [object_number=14, tag_tid=1, entry_tid=13174], [object_number=15, tag_tid=1, entry_tid=13171]]], state=connected] 2024-04-06T10:02:56.743 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_22.status 2024-04-06T10:02:56.745 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=13, tag_tid=2, entry_tid=1' 2024-04-06T10:02:56.745 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=13, tag_tid=2, entry_tid=1' -a 'object_number=13, tag_tid=2, entry_tid=1' = 'object_number=13, tag_tid=2, entry_tid=1' 2024-04-06T10:02:56.745 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T10:02:56.745 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_snap_present cluster1 mirror image_22 snap 2024-04-06T10:02:56.745 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T10:02:56.745 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:02:56.745 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_22 2024-04-06T10:02:56.745 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local snap_name=snap 2024-04-06T10:02:56.745 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local s 2024-04-06T10:02:56.745 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-04-06T10:02:56.745 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 1 2024-04-06T10:02:57.746 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 info mirror/image_22@snap 2024-04-06T10:02:57.809 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:rbd image 'image_22': 2024-04-06T10:02:57.809 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: size 128 MiB in 32 objects 2024-04-06T10:02:57.810 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: order 22 (4 MiB objects) 2024-04-06T10:02:57.810 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: snapshot_count: 1 2024-04-06T10:02:57.810 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: id: 10ace4cde79b 2024-04-06T10:02:57.810 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: block_name_prefix: rbd_data.10ace4cde79b 2024-04-06T10:02:57.810 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: format: 2 2024-04-06T10:02:57.810 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: features: layering, exclusive-lock, journaling 2024-04-06T10:02:57.810 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: op_features: 2024-04-06T10:02:57.810 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: flags: 2024-04-06T10:02:57.810 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: create_timestamp: Sat Apr 6 09:54:04 2024 2024-04-06T10:02:57.810 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: access_timestamp: Sat Apr 6 09:54:04 2024 2024-04-06T10:02:57.810 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: modify_timestamp: Sat Apr 6 09:55:05 2024 2024-04-06T10:02:57.810 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: protected: False 2024-04-06T10:02:57.810 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: journal: 10ace4cde79b 2024-04-06T10:02:57.811 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: mirroring state: enabled 2024-04-06T10:02:57.811 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: mirroring mode: journal 2024-04-06T10:02:57.811 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: mirroring global id: f32b6e84-8adb-4d59-95c7-516d7996015e 2024-04-06T10:02:57.811 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: mirroring primary: false 2024-04-06T10:02:57.815 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T10:02:57.815 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ compare_image_snaps mirror image_22 snap 2024-04-06T10:02:57.815 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:02:57.815 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_22 2024-04-06T10:02:57.815 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local snap_name=snap 2024-04-06T10:02:57.815 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local ret=0 2024-04-06T10:02:57.815 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local rmt_export=/tmp/tmp.BhW6xKmROU/cluster2-mirror-image_22.export 2024-04-06T10:02:57.815 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loc_export=/tmp/tmp.BhW6xKmROU/cluster1-mirror-image_22.export 2024-04-06T10:02:57.815 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rm -f /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_22.export /tmp/tmp.BhW6xKmROU/cluster1-mirror-image_22.export 2024-04-06T10:02:57.816 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster2 -p mirror export image_22@snap /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_22.export 2024-04-06T10:02:57.999 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-04-06T10:02:58.007 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 -p mirror export image_22@snap /tmp/tmp.BhW6xKmROU/cluster1-mirror-image_22.export 2024-04-06T10:02:58.421 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-04-06T10:02:58.436 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmp /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_22.export /tmp/tmp.BhW6xKmROU/cluster1-mirror-image_22.export 2024-04-06T10:02:58.515 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rm -f /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_22.export /tmp/tmp.BhW6xKmROU/cluster1-mirror-image_22.export 2024-04-06T10:02:58.556 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T10:02:58.556 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-06T10:02:58.556 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ image=image_23 2024-04-06T10:02:58.556 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ create_snap cluster2 mirror image_23 snap 2024-04-06T10:02:58.556 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-04-06T10:02:58.556 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:02:58.556 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_23 2024-04-06T10:02:58.556 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local snap_name=snap 2024-04-06T10:02:58.556 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster2 -p mirror snap create image_23@snap --debug-rbd=20 --debug-journaler=20 2024-04-06T10:02:59.856 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_image_replay_started cluster1 mirror image_23 2024-04-06T10:02:59.856 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T10:02:59.856 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:02:59.856 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_23 2024-04-06T10:02:59.856 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_image_replay_state cluster1 mirror image_23 started 2024-04-06T10:02:59.856 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T10:02:59.857 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:02:59.857 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_23 2024-04-06T10:02:59.857 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state=started 2024-04-06T10:02:59.857 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local s 2024-04-06T10:02:59.857 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T10:02:59.857 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 1 2024-04-06T10:03:00.857 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test_image_replay_state cluster1 mirror image_23 started 2024-04-06T10:03:00.857 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T10:03:00.857 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:03:00.857 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_23 2024-04-06T10:03:00.857 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local test_state=started 2024-04-06T10:03:00.857 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local status_result 2024-04-06T10:03:00.857 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local current_state=stopped 2024-04-06T10:03:00.858 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_23 2024-04-06T10:03:00.858 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_instance=cluster1 2024-04-06T10:03:00.858 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ shift 2024-04-06T10:03:00.858 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -i state 2024-04-06T10:03:00.859 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T10:03:00.859 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance=cluster1 2024-04-06T10:03:00.859 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local loop_instance 2024-04-06T10:03:00.859 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T10:03:00.859 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sleep 0 2024-04-06T10:03:00.859 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-04-06T10:03:00.860 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ seq 0 3 2024-04-06T10:03:00.863 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T10:03:00.863 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_23 2024-04-06T10:03:00.863 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1:0 2024-04-06T10:03:00.863 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ shift 2024-04-06T10:03:00.863 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T10:03:00.863 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-04-06T10:03:00.863 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T10:03:00.863 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=cluster 2024-04-06T10:03:00.863 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T10:03:00.863 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:03:00.864 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T10:03:00.864 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T10:03:00.864 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval cluster=cluster1 2024-04-06T10:03:00.864 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ cluster=cluster1 2024-04-06T10:03:00.864 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T10:03:00.864 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T10:03:00.864 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-04-06T10:03:00.864 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local local_cluster=cluster1:0 2024-04-06T10:03:00.864 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local cluster=cluster1 2024-04-06T10:03:00.864 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local instance 2024-04-06T10:03:00.864 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T10:03:00.864 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local val=cluster1:0 2024-04-06T10:03:00.864 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local cluster_var_name=local_cluster 2024-04-06T10:03:00.864 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local instance_var_name=instance 2024-04-06T10:03:00.865 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ cluster=cluster1 2024-04-06T10:03:00.865 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T10:03:00.865 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ '[' 0 = cluster1:0 ']' 2024-04-06T10:03:00.865 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ eval local_cluster=cluster1 2024-04-06T10:03:00.865 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++++ local_cluster=cluster1 2024-04-06T10:03:00.866 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ eval instance=0 2024-04-06T10:03:00.866 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++++ instance=0 2024-04-06T10:03:00.866 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T10:03:00.888 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T10:03:00.888 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T10:03:00.888 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T10:03:00.888 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_23 2024-04-06T10:03:01.001 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:Invalid command: unused arguments: ['mirror/image_23'] 2024-04-06T10:03:01.002 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:rbd mirror status : get status for rbd mirror 2024-04-06T10:03:01.002 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:admin_socket: invalid command 2024-04-06T10:03:01.005 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T10:03:01.005 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_23 2024-04-06T10:03:01.005 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1:1 2024-04-06T10:03:01.005 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ shift 2024-04-06T10:03:01.005 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T10:03:01.005 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-04-06T10:03:01.005 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:1 2024-04-06T10:03:01.006 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=cluster 2024-04-06T10:03:01.006 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T10:03:01.006 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:03:01.006 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=1 2024-04-06T10:03:01.006 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-06T10:03:01.006 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval cluster=cluster1 2024-04-06T10:03:01.006 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ cluster=cluster1 2024-04-06T10:03:01.006 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=1 2024-04-06T10:03:01.006 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=1 2024-04-06T10:03:01.006 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-04-06T10:03:01.006 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local local_cluster=cluster1:1 2024-04-06T10:03:01.006 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local cluster=cluster1 2024-04-06T10:03:01.006 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local instance 2024-04-06T10:03:01.006 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-06T10:03:01.007 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local val=cluster1:1 2024-04-06T10:03:01.007 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local cluster_var_name=local_cluster 2024-04-06T10:03:01.007 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local instance_var_name=instance 2024-04-06T10:03:01.007 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ cluster=cluster1 2024-04-06T10:03:01.007 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=1 2024-04-06T10:03:01.007 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ '[' 1 = cluster1:1 ']' 2024-04-06T10:03:01.007 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ eval local_cluster=cluster1 2024-04-06T10:03:01.007 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++++ local_cluster=cluster1 2024-04-06T10:03:01.007 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ eval instance=1 2024-04-06T10:03:01.007 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++++ instance=1 2024-04-06T10:03:01.007 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-06T10:03:01.025 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T10:03:01.025 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T10:03:01.025 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T10:03:01.025 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_23 2024-04-06T10:03:01.172 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ return 0 2024-04-06T10:03:01.172 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ status_result=' "state": "Replaying"' 2024-04-06T10:03:01.173 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ echo ' "state": "Replaying"' 2024-04-06T10:03:01.173 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep -i Replaying 2024-04-06T10:03:01.174 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: "state": "Replaying" 2024-04-06T10:03:01.176 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ current_state=started 2024-04-06T10:03:01.176 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test started = started 2024-04-06T10:03:01.176 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T10:03:01.176 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_23 2024-04-06T10:03:01.176 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local local_cluster=cluster1 2024-04-06T10:03:01.176 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-04-06T10:03:01.176 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:03:01.176 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_23 2024-04-06T10:03:01.177 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' journal = journal ']' 2024-04-06T10:03:01.177 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_journal_replay_complete cluster1 cluster2 mirror image_23 2024-04-06T10:03:01.177 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local local_cluster=cluster1 2024-04-06T10:03:01.177 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-04-06T10:03:01.177 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:03:01.177 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_23 2024-04-06T10:03:01.177 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local s master_pos mirror_pos last_mirror_pos 2024-04-06T10:03:01.177 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local master_tag master_entry mirror_tag mirror_entry 2024-04-06T10:03:01.177 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T10:03:01.177 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T10:03:01.178 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T10:03:01.378 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror image_23 2024-04-06T10:03:01.378 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T10:03:01.378 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:03:01.378 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_23 2024-04-06T10:03:01.378 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T10:03:01.378 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n image_23 ']' 2024-04-06T10:03:01.379 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/image_23' 2024-04-06T10:03:01.379 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_23 2024-04-06T10:03:01.379 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T10:03:01.379 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T10:03:01.379 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T10:03:01.379 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T10:03:01.379 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T10:03:01.379 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T10:03:01.379 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T10:03:01.379 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T10:03:01.380 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T10:03:01.382 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T10:03:01.382 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_23 2024-04-06T10:03:01.382 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T10:03:01.382 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T10:03:01.382 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T10:03:01.382 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T10:03:01.383 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T10:03:01.383 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T10:03:01.383 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T10:03:01.383 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T10:03:01.383 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T10:03:01.383 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T10:03:01.383 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T10:03:01.383 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:03:01.383 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T10:03:01.383 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T10:03:01.383 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T10:03:01.383 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T10:03:01.383 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T10:03:01.383 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T10:03:01.384 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T10:03:01.384 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T10:03:01.384 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T10:03:01.384 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T10:03:01.384 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:03:01.384 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T10:03:01.384 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T10:03:01.384 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T10:03:01.384 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T10:03:01.384 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T10:03:01.384 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T10:03:01.384 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T10:03:01.407 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T10:03:01.408 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T10:03:01.408 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T10:03:01.408 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_23 2024-04-06T10:03:01.510 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:Invalid command: unused arguments: ['mirror/image_23'] 2024-04-06T10:03:01.510 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:rbd mirror flush : flush rbd mirror 2024-04-06T10:03:01.510 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:admin_socket: invalid command 2024-04-06T10:03:01.512 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T10:03:01.512 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/image_23 2024-04-06T10:03:01.512 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:1 2024-04-06T10:03:01.512 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T10:03:01.512 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T10:03:01.512 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-06T10:03:01.512 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:1 2024-04-06T10:03:01.512 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T10:03:01.513 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T10:03:01.513 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T10:03:01.513 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=1 2024-04-06T10:03:01.513 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-06T10:03:01.513 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T10:03:01.513 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:03:01.513 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=1 2024-04-06T10:03:01.513 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=1 2024-04-06T10:03:01.515 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-06T10:03:01.515 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:1 2024-04-06T10:03:01.515 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T10:03:01.515 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T10:03:01.515 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-06T10:03:01.515 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:1 2024-04-06T10:03:01.515 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T10:03:01.515 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T10:03:01.515 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:03:01.515 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=1 2024-04-06T10:03:01.515 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-06T10:03:01.515 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T10:03:01.516 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T10:03:01.516 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=1 2024-04-06T10:03:01.516 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=1 2024-04-06T10:03:01.516 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-06T10:03:01.532 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T10:03:01.533 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T10:03:01.533 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T10:03:01.533 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/image_23 2024-04-06T10:03:01.682 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T10:03:01.683 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror image_23 2024-04-06T10:03:01.683 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:03:01.683 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:03:01.683 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_23 2024-04-06T10:03:01.683 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror image_23 '' 2024-04-06T10:03:01.683 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:03:01.683 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:03:01.683 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_23 2024-04-06T10:03:01.683 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T10:03:01.684 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-image_23.status 2024-04-06T10:03:01.684 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-image_23.status 2024-04-06T10:03:01.684 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T10:03:01.686 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-image_23.status 2024-04-06T10:03:01.686 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_23 2024-04-06T10:03:01.686 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_23.status 2024-04-06T10:03:01.736 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 1 2024-04-06T10:03:01.736 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 1 2024-04-06T10:03:01.736 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T10:03:01.736 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=5, tag_tid=2, entry_tid=1], [object_number=4, tag_tid=2, entry_tid=0], [object_number=7, tag_tid=1, entry_tid=5243], [object_number=6, tag_tid=1, entry_tid=5242]]], state=connected] 2024-04-06T10:03:01.736 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=5, tag_tid=2, entry_tid=1], [object_number=4, tag_tid=2, entry_tid=0], [object_number=7, tag_tid=1, entry_tid=5243], [object_number=6, tag_tid=1, entry_tid=5242]]], state=connected] 2024-04-06T10:03:01.739 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_23.status 2024-04-06T10:03:01.741 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=5, tag_tid=2, entry_tid=1' 2024-04-06T10:03:01.742 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror image_23 2024-04-06T10:03:01.742 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:03:01.742 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:03:01.742 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_23 2024-04-06T10:03:01.742 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror image_23 '..*' 2024-04-06T10:03:01.742 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:03:01.742 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:03:01.742 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_23 2024-04-06T10:03:01.742 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T10:03:01.742 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-image_23.status 2024-04-06T10:03:01.742 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-image_23.status 2024-04-06T10:03:01.743 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T10:03:01.743 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-image_23.status 2024-04-06T10:03:01.743 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_23 2024-04-06T10:03:01.743 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_23.status 2024-04-06T10:03:01.790 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 1 2024-04-06T10:03:01.790 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 1 2024-04-06T10:03:01.790 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T10:03:01.790 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=5, tag_tid=2, entry_tid=1], [object_number=4, tag_tid=2, entry_tid=0], [object_number=7, tag_tid=1, entry_tid=5243], [object_number=6, tag_tid=1, entry_tid=5242]]], state=connected] 2024-04-06T10:03:01.790 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=5, tag_tid=2, entry_tid=1], [object_number=4, tag_tid=2, entry_tid=0], [object_number=7, tag_tid=1, entry_tid=5243], [object_number=6, tag_tid=1, entry_tid=5242]]], state=connected] 2024-04-06T10:03:01.794 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_23.status 2024-04-06T10:03:01.796 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=5, tag_tid=2, entry_tid=1' 2024-04-06T10:03:01.796 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=5, tag_tid=2, entry_tid=1' -a 'object_number=5, tag_tid=2, entry_tid=1' = 'object_number=5, tag_tid=2, entry_tid=1' 2024-04-06T10:03:01.796 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T10:03:01.796 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_snap_present cluster1 mirror image_23 snap 2024-04-06T10:03:01.796 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T10:03:01.796 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:03:01.796 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_23 2024-04-06T10:03:01.796 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local snap_name=snap 2024-04-06T10:03:01.796 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local s 2024-04-06T10:03:01.796 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-04-06T10:03:01.796 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 1 2024-04-06T10:03:02.797 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 info mirror/image_23@snap 2024-04-06T10:03:02.861 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:rbd image 'image_23': 2024-04-06T10:03:02.861 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: size 128 MiB in 32 objects 2024-04-06T10:03:02.861 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: order 22 (4 MiB objects) 2024-04-06T10:03:02.861 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: snapshot_count: 1 2024-04-06T10:03:02.861 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: id: 10ac1cadbe5d 2024-04-06T10:03:02.861 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: block_name_prefix: rbd_data.10ac1cadbe5d 2024-04-06T10:03:02.861 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: format: 2 2024-04-06T10:03:02.861 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: features: layering, exclusive-lock, journaling 2024-04-06T10:03:02.861 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: op_features: 2024-04-06T10:03:02.861 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: flags: 2024-04-06T10:03:02.862 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: create_timestamp: Sat Apr 6 09:54:14 2024 2024-04-06T10:03:02.862 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: access_timestamp: Sat Apr 6 09:54:14 2024 2024-04-06T10:03:02.862 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: modify_timestamp: Sat Apr 6 09:54:14 2024 2024-04-06T10:03:02.862 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: protected: False 2024-04-06T10:03:02.862 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: journal: 10ac1cadbe5d 2024-04-06T10:03:02.862 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: mirroring state: enabled 2024-04-06T10:03:02.862 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: mirroring mode: journal 2024-04-06T10:03:02.862 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: mirroring global id: ce4fdea6-b325-481f-a8d4-31308aa3cff1 2024-04-06T10:03:02.862 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: mirroring primary: false 2024-04-06T10:03:02.866 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T10:03:02.866 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ compare_image_snaps mirror image_23 snap 2024-04-06T10:03:02.866 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:03:02.866 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_23 2024-04-06T10:03:02.866 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local snap_name=snap 2024-04-06T10:03:02.866 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local ret=0 2024-04-06T10:03:02.866 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local rmt_export=/tmp/tmp.BhW6xKmROU/cluster2-mirror-image_23.export 2024-04-06T10:03:02.866 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loc_export=/tmp/tmp.BhW6xKmROU/cluster1-mirror-image_23.export 2024-04-06T10:03:02.866 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rm -f /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_23.export /tmp/tmp.BhW6xKmROU/cluster1-mirror-image_23.export 2024-04-06T10:03:02.867 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster2 -p mirror export image_23@snap /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_23.export 2024-04-06T10:03:03.240 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-04-06T10:03:03.249 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 -p mirror export image_23@snap /tmp/tmp.BhW6xKmROU/cluster1-mirror-image_23.export 2024-04-06T10:03:03.577 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-04-06T10:03:03.590 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmp /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_23.export /tmp/tmp.BhW6xKmROU/cluster1-mirror-image_23.export 2024-04-06T10:03:03.668 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rm -f /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_23.export /tmp/tmp.BhW6xKmROU/cluster1-mirror-image_23.export 2024-04-06T10:03:03.709 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T10:03:03.709 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-06T10:03:03.709 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ image=image_24 2024-04-06T10:03:03.709 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ create_snap cluster2 mirror image_24 snap 2024-04-06T10:03:03.709 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-04-06T10:03:03.710 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:03:03.710 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_24 2024-04-06T10:03:03.710 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local snap_name=snap 2024-04-06T10:03:03.710 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster2 -p mirror snap create image_24@snap --debug-rbd=20 --debug-journaler=20 2024-04-06T10:03:05.887 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_image_replay_started cluster1 mirror image_24 2024-04-06T10:03:05.888 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T10:03:05.888 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:03:05.888 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_24 2024-04-06T10:03:05.888 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_image_replay_state cluster1 mirror image_24 started 2024-04-06T10:03:05.888 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T10:03:05.888 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:03:05.888 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_24 2024-04-06T10:03:05.888 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state=started 2024-04-06T10:03:05.888 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local s 2024-04-06T10:03:05.888 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T10:03:05.888 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 1 2024-04-06T10:03:06.889 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test_image_replay_state cluster1 mirror image_24 started 2024-04-06T10:03:06.889 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T10:03:06.889 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:03:06.889 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_24 2024-04-06T10:03:06.890 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local test_state=started 2024-04-06T10:03:06.890 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local status_result 2024-04-06T10:03:06.890 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local current_state=stopped 2024-04-06T10:03:06.890 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_24 2024-04-06T10:03:06.891 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_instance=cluster1 2024-04-06T10:03:06.891 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ shift 2024-04-06T10:03:06.891 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T10:03:06.891 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance=cluster1 2024-04-06T10:03:06.891 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local loop_instance 2024-04-06T10:03:06.891 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T10:03:06.891 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sleep 0 2024-04-06T10:03:06.891 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -i state 2024-04-06T10:03:06.892 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-04-06T10:03:06.892 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ seq 0 3 2024-04-06T10:03:06.895 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T10:03:06.895 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_24 2024-04-06T10:03:06.895 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1:0 2024-04-06T10:03:06.895 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ shift 2024-04-06T10:03:06.895 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T10:03:06.895 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-04-06T10:03:06.895 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T10:03:06.895 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=cluster 2024-04-06T10:03:06.895 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T10:03:06.895 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:03:06.895 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T10:03:06.896 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T10:03:06.896 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval cluster=cluster1 2024-04-06T10:03:06.896 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ cluster=cluster1 2024-04-06T10:03:06.896 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T10:03:06.896 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T10:03:06.896 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-04-06T10:03:06.896 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local local_cluster=cluster1:0 2024-04-06T10:03:06.896 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local cluster=cluster1 2024-04-06T10:03:06.896 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local instance 2024-04-06T10:03:06.896 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T10:03:06.896 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local val=cluster1:0 2024-04-06T10:03:06.896 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local cluster_var_name=local_cluster 2024-04-06T10:03:06.896 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local instance_var_name=instance 2024-04-06T10:03:06.896 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ cluster=cluster1 2024-04-06T10:03:06.897 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T10:03:06.897 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ '[' 0 = cluster1:0 ']' 2024-04-06T10:03:06.897 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ eval local_cluster=cluster1 2024-04-06T10:03:06.897 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++++ local_cluster=cluster1 2024-04-06T10:03:06.897 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ eval instance=0 2024-04-06T10:03:06.897 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++++ instance=0 2024-04-06T10:03:06.897 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T10:03:06.917 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T10:03:06.917 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T10:03:06.918 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T10:03:06.918 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_24 2024-04-06T10:03:07.027 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:Invalid command: unused arguments: ['mirror/image_24'] 2024-04-06T10:03:07.027 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:rbd mirror status : get status for rbd mirror 2024-04-06T10:03:07.028 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:admin_socket: invalid command 2024-04-06T10:03:07.030 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T10:03:07.030 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_24 2024-04-06T10:03:07.030 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1:1 2024-04-06T10:03:07.030 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ shift 2024-04-06T10:03:07.030 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T10:03:07.030 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-04-06T10:03:07.030 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:1 2024-04-06T10:03:07.030 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=cluster 2024-04-06T10:03:07.030 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T10:03:07.030 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:03:07.030 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=1 2024-04-06T10:03:07.030 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-06T10:03:07.031 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval cluster=cluster1 2024-04-06T10:03:07.031 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ cluster=cluster1 2024-04-06T10:03:07.031 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=1 2024-04-06T10:03:07.031 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=1 2024-04-06T10:03:07.032 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-04-06T10:03:07.032 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local local_cluster=cluster1:1 2024-04-06T10:03:07.032 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local cluster=cluster1 2024-04-06T10:03:07.032 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local instance 2024-04-06T10:03:07.032 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-06T10:03:07.032 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local val=cluster1:1 2024-04-06T10:03:07.032 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local cluster_var_name=local_cluster 2024-04-06T10:03:07.032 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local instance_var_name=instance 2024-04-06T10:03:07.033 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ cluster=cluster1 2024-04-06T10:03:07.033 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=1 2024-04-06T10:03:07.033 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ '[' 1 = cluster1:1 ']' 2024-04-06T10:03:07.033 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ eval local_cluster=cluster1 2024-04-06T10:03:07.033 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++++ local_cluster=cluster1 2024-04-06T10:03:07.033 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ eval instance=1 2024-04-06T10:03:07.033 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++++ instance=1 2024-04-06T10:03:07.033 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-06T10:03:07.056 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T10:03:07.056 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T10:03:07.056 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T10:03:07.056 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_24 2024-04-06T10:03:07.216 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ return 0 2024-04-06T10:03:07.217 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ status_result=' "state": "Replaying"' 2024-04-06T10:03:07.217 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ echo ' "state": "Replaying"' 2024-04-06T10:03:07.217 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep -i Replaying 2024-04-06T10:03:07.219 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: "state": "Replaying" 2024-04-06T10:03:07.220 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ current_state=started 2024-04-06T10:03:07.220 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test started = started 2024-04-06T10:03:07.220 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T10:03:07.220 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_24 2024-04-06T10:03:07.220 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local local_cluster=cluster1 2024-04-06T10:03:07.220 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-04-06T10:03:07.221 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:03:07.221 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_24 2024-04-06T10:03:07.221 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' journal = journal ']' 2024-04-06T10:03:07.221 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_journal_replay_complete cluster1 cluster2 mirror image_24 2024-04-06T10:03:07.221 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local local_cluster=cluster1 2024-04-06T10:03:07.221 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-04-06T10:03:07.221 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:03:07.221 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_24 2024-04-06T10:03:07.221 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local s master_pos mirror_pos last_mirror_pos 2024-04-06T10:03:07.221 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local master_tag master_entry mirror_tag mirror_entry 2024-04-06T10:03:07.221 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T10:03:07.221 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T10:03:07.221 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T10:03:07.422 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror image_24 2024-04-06T10:03:07.422 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T10:03:07.422 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:03:07.423 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_24 2024-04-06T10:03:07.423 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T10:03:07.423 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n image_24 ']' 2024-04-06T10:03:07.423 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/image_24' 2024-04-06T10:03:07.423 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_24 2024-04-06T10:03:07.423 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T10:03:07.423 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T10:03:07.423 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T10:03:07.423 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T10:03:07.423 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T10:03:07.423 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T10:03:07.423 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T10:03:07.424 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T10:03:07.424 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T10:03:07.428 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T10:03:07.428 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_24 2024-04-06T10:03:07.428 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T10:03:07.428 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T10:03:07.428 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T10:03:07.428 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T10:03:07.428 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T10:03:07.428 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T10:03:07.428 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T10:03:07.428 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T10:03:07.429 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T10:03:07.429 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T10:03:07.429 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T10:03:07.429 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:03:07.429 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T10:03:07.429 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T10:03:07.429 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T10:03:07.429 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T10:03:07.429 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T10:03:07.429 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T10:03:07.429 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T10:03:07.429 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T10:03:07.429 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T10:03:07.429 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T10:03:07.429 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:03:07.430 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T10:03:07.430 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T10:03:07.430 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T10:03:07.430 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T10:03:07.430 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T10:03:07.430 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T10:03:07.430 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T10:03:07.454 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.0 with skip (0.7949423096785914) > weight (0.25) 2024-04-06T10:03:07.454 INFO:tasks.rbd_mirror_thrash:kill cluster2.client.mirror.1 2024-04-06T10:03:07.454 INFO:tasks.rbd_mirror.cluster2.client.mirror.1:Sent signal 15 2024-04-06T10:03:07.455 INFO:tasks.rbd_mirror_thrash:waiting for 0 secs before reviving daemons 2024-04-06T10:03:07.455 INFO:tasks.rbd_mirror_thrash:waiting for cluster2.client.mirror.1 2024-04-06T10:03:07.455 INFO:teuthology.orchestra.run:waiting for 600 2024-04-06T10:03:07.455 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T10:03:07.456 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T10:03:07.456 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T10:03:07.456 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_24 2024-04-06T10:03:07.456 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi192.stderr:2024-04-06T10:03:07.455+0000 7f7c60131640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.1 (PID: 123570) UID: 1000 2024-04-06T10:03:07.574 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:Invalid command: unused arguments: ['mirror/image_24'] 2024-04-06T10:03:07.574 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:rbd mirror flush : flush rbd mirror 2024-04-06T10:03:07.574 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:admin_socket: invalid command 2024-04-06T10:03:07.577 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T10:03:07.578 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/image_24 2024-04-06T10:03:07.578 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:1 2024-04-06T10:03:07.578 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T10:03:07.578 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T10:03:07.578 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-06T10:03:07.578 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:1 2024-04-06T10:03:07.578 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T10:03:07.578 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T10:03:07.578 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T10:03:07.578 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=1 2024-04-06T10:03:07.578 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-06T10:03:07.578 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T10:03:07.578 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:03:07.579 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=1 2024-04-06T10:03:07.579 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=1 2024-04-06T10:03:07.579 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-06T10:03:07.579 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:1 2024-04-06T10:03:07.579 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T10:03:07.579 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T10:03:07.579 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-06T10:03:07.579 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:1 2024-04-06T10:03:07.579 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T10:03:07.579 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T10:03:07.579 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:03:07.579 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=1 2024-04-06T10:03:07.579 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-06T10:03:07.579 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T10:03:07.580 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T10:03:07.580 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=1 2024-04-06T10:03:07.580 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=1 2024-04-06T10:03:07.580 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-06T10:03:07.597 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T10:03:07.597 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T10:03:07.597 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T10:03:07.598 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/image_24 2024-04-06T10:03:07.738 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T10:03:07.739 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror image_24 2024-04-06T10:03:07.739 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:03:07.739 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:03:07.739 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_24 2024-04-06T10:03:07.739 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror image_24 '' 2024-04-06T10:03:07.739 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:03:07.740 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:03:07.740 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_24 2024-04-06T10:03:07.740 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T10:03:07.740 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-image_24.status 2024-04-06T10:03:07.740 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-image_24.status 2024-04-06T10:03:07.740 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T10:03:07.743 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-image_24.status 2024-04-06T10:03:07.743 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_24 2024-04-06T10:03:07.743 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_24.status 2024-04-06T10:03:07.796 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 1 2024-04-06T10:03:07.797 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 1 2024-04-06T10:03:07.797 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T10:03:07.797 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=5, tag_tid=2, entry_tid=1], [object_number=4, tag_tid=2, entry_tid=0], [object_number=7, tag_tid=1, entry_tid=6903], [object_number=6, tag_tid=1, entry_tid=6902]]], state=connected] 2024-04-06T10:03:07.797 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=4, tag_tid=1, entry_tid=6904], [object_number=7, tag_tid=1, entry_tid=6903], [object_number=6, tag_tid=1, entry_tid=6902], [object_number=5, tag_tid=1, entry_tid=6901]]], state=connected] 2024-04-06T10:03:07.800 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_24.status 2024-04-06T10:03:07.801 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=5, tag_tid=2, entry_tid=1' 2024-04-06T10:03:07.802 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror image_24 2024-04-06T10:03:07.803 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:03:07.803 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:03:07.803 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_24 2024-04-06T10:03:07.803 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror image_24 '..*' 2024-04-06T10:03:07.803 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:03:07.803 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:03:07.803 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_24 2024-04-06T10:03:07.803 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T10:03:07.803 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-image_24.status 2024-04-06T10:03:07.803 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-image_24.status 2024-04-06T10:03:07.803 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T10:03:07.804 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-image_24.status 2024-04-06T10:03:07.804 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_24 2024-04-06T10:03:07.804 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_24.status 2024-04-06T10:03:07.849 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 1 2024-04-06T10:03:07.849 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 1 2024-04-06T10:03:07.849 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T10:03:07.849 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=5, tag_tid=2, entry_tid=1], [object_number=4, tag_tid=2, entry_tid=0], [object_number=7, tag_tid=1, entry_tid=6903], [object_number=6, tag_tid=1, entry_tid=6902]]], state=connected] 2024-04-06T10:03:07.849 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=4, tag_tid=1, entry_tid=6904], [object_number=7, tag_tid=1, entry_tid=6903], [object_number=6, tag_tid=1, entry_tid=6902], [object_number=5, tag_tid=1, entry_tid=6901]]], state=connected] 2024-04-06T10:03:07.852 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_24.status 2024-04-06T10:03:07.854 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=4, tag_tid=1, entry_tid=6904' 2024-04-06T10:03:07.854 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=5, tag_tid=2, entry_tid=1' -a 'object_number=5, tag_tid=2, entry_tid=1' = 'object_number=4, tag_tid=1, entry_tid=6904' 2024-04-06T10:03:07.854 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=4, tag_tid=1, entry_tid=6904' '!=' '' 2024-04-06T10:03:07.854 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T10:03:07.854 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=4, tag_tid=1, entry_tid=6904' = '' 2024-04-06T10:03:07.854 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=4, tag_tid=1, entry_tid=6904' 2024-04-06T10:03:07.855 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=5, tag_tid=2, entry_tid=1' 2024-04-06T10:03:07.855 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T10:03:07.855 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T10:03:07.856 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=2 2024-04-06T10:03:07.856 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=4, tag_tid=1, entry_tid=6904' 2024-04-06T10:03:07.856 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T10:03:07.856 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T10:03:07.857 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=1 2024-04-06T10:03:07.858 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=5, tag_tid=2, entry_tid=1' 2024-04-06T10:03:07.858 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T10:03:07.858 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T10:03:07.859 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T10:03:07.860 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=4, tag_tid=1, entry_tid=6904' 2024-04-06T10:03:07.860 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T10:03:07.860 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T10:03:07.861 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=6904 2024-04-06T10:03:07.861 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 2 = 1 -a 1 -le 6904 2024-04-06T10:03:07.861 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T10:03:07.861 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T10:03:07.861 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T10:03:08.063 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror image_24 2024-04-06T10:03:08.063 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T10:03:08.063 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:03:08.063 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_24 2024-04-06T10:03:08.063 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T10:03:08.063 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n image_24 ']' 2024-04-06T10:03:08.064 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/image_24' 2024-04-06T10:03:08.064 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_24 2024-04-06T10:03:08.064 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T10:03:08.064 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T10:03:08.064 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T10:03:08.064 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T10:03:08.064 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T10:03:08.064 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T10:03:08.064 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T10:03:08.065 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T10:03:08.065 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T10:03:08.067 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T10:03:08.067 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_24 2024-04-06T10:03:08.067 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T10:03:08.067 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T10:03:08.067 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T10:03:08.067 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T10:03:08.067 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T10:03:08.067 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T10:03:08.067 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T10:03:08.068 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T10:03:08.068 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T10:03:08.068 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T10:03:08.068 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T10:03:08.068 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:03:08.068 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T10:03:08.068 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T10:03:08.068 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T10:03:08.068 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T10:03:08.068 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T10:03:08.068 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T10:03:08.068 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T10:03:08.068 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T10:03:08.069 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T10:03:08.069 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T10:03:08.069 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:03:08.069 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T10:03:08.069 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T10:03:08.069 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T10:03:08.069 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T10:03:08.069 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T10:03:08.069 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T10:03:08.069 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T10:03:08.093 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T10:03:08.093 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T10:03:08.093 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T10:03:08.093 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_24 2024-04-06T10:03:08.195 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:Invalid command: unused arguments: ['mirror/image_24'] 2024-04-06T10:03:08.195 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:rbd mirror flush : flush rbd mirror 2024-04-06T10:03:08.196 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:admin_socket: invalid command 2024-04-06T10:03:08.198 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T10:03:08.198 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/image_24 2024-04-06T10:03:08.198 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:1 2024-04-06T10:03:08.198 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T10:03:08.198 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T10:03:08.198 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-06T10:03:08.198 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:1 2024-04-06T10:03:08.198 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T10:03:08.198 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T10:03:08.198 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T10:03:08.198 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=1 2024-04-06T10:03:08.198 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-06T10:03:08.199 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T10:03:08.199 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:03:08.199 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=1 2024-04-06T10:03:08.199 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=1 2024-04-06T10:03:08.200 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-06T10:03:08.200 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:1 2024-04-06T10:03:08.200 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T10:03:08.200 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T10:03:08.200 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-06T10:03:08.200 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:1 2024-04-06T10:03:08.201 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T10:03:08.201 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T10:03:08.201 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:03:08.201 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=1 2024-04-06T10:03:08.201 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-06T10:03:08.201 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T10:03:08.201 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T10:03:08.201 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=1 2024-04-06T10:03:08.201 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=1 2024-04-06T10:03:08.201 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-06T10:03:08.224 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T10:03:08.225 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T10:03:08.225 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T10:03:08.225 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/image_24 2024-04-06T10:03:08.371 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T10:03:08.373 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror image_24 2024-04-06T10:03:08.373 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:03:08.373 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:03:08.373 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_24 2024-04-06T10:03:08.373 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror image_24 '' 2024-04-06T10:03:08.373 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:03:08.373 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:03:08.373 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_24 2024-04-06T10:03:08.373 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T10:03:08.373 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-image_24.status 2024-04-06T10:03:08.374 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T10:03:08.374 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-image_24.status 2024-04-06T10:03:08.375 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-image_24.status 2024-04-06T10:03:08.375 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_24 2024-04-06T10:03:08.375 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_24.status 2024-04-06T10:03:08.420 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 1 2024-04-06T10:03:08.420 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 1 2024-04-06T10:03:08.420 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T10:03:08.420 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=5, tag_tid=2, entry_tid=1], [object_number=4, tag_tid=2, entry_tid=0], [object_number=7, tag_tid=1, entry_tid=6903], [object_number=6, tag_tid=1, entry_tid=6902]]], state=connected] 2024-04-06T10:03:08.420 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=4, tag_tid=1, entry_tid=6904], [object_number=7, tag_tid=1, entry_tid=6903], [object_number=6, tag_tid=1, entry_tid=6902], [object_number=5, tag_tid=1, entry_tid=6901]]], state=connected] 2024-04-06T10:03:08.423 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_24.status 2024-04-06T10:03:08.425 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=5, tag_tid=2, entry_tid=1' 2024-04-06T10:03:08.426 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror image_24 2024-04-06T10:03:08.426 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:03:08.426 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:03:08.426 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_24 2024-04-06T10:03:08.426 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror image_24 '..*' 2024-04-06T10:03:08.426 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:03:08.427 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:03:08.427 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_24 2024-04-06T10:03:08.427 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T10:03:08.427 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-image_24.status 2024-04-06T10:03:08.427 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-image_24.status 2024-04-06T10:03:08.427 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T10:03:08.427 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-image_24.status 2024-04-06T10:03:08.428 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_24 2024-04-06T10:03:08.428 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_24.status 2024-04-06T10:03:08.472 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 1 2024-04-06T10:03:08.472 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 1 2024-04-06T10:03:08.473 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T10:03:08.473 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=5, tag_tid=2, entry_tid=1], [object_number=4, tag_tid=2, entry_tid=0], [object_number=7, tag_tid=1, entry_tid=6903], [object_number=6, tag_tid=1, entry_tid=6902]]], state=connected] 2024-04-06T10:03:08.473 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=4, tag_tid=1, entry_tid=6904], [object_number=7, tag_tid=1, entry_tid=6903], [object_number=6, tag_tid=1, entry_tid=6902], [object_number=5, tag_tid=1, entry_tid=6901]]], state=connected] 2024-04-06T10:03:08.480 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_24.status 2024-04-06T10:03:08.482 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=4, tag_tid=1, entry_tid=6904' 2024-04-06T10:03:08.482 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=5, tag_tid=2, entry_tid=1' -a 'object_number=5, tag_tid=2, entry_tid=1' = 'object_number=4, tag_tid=1, entry_tid=6904' 2024-04-06T10:03:08.482 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=4, tag_tid=1, entry_tid=6904' '!=' 'object_number=4, tag_tid=1, entry_tid=6904' 2024-04-06T10:03:08.482 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T10:03:08.483 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.4 2024-04-06T10:03:08.884 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror image_24 2024-04-06T10:03:08.884 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T10:03:08.885 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:03:08.885 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_24 2024-04-06T10:03:08.885 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T10:03:08.885 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n image_24 ']' 2024-04-06T10:03:08.885 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/image_24' 2024-04-06T10:03:08.885 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_24 2024-04-06T10:03:08.885 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T10:03:08.885 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T10:03:08.885 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T10:03:08.885 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T10:03:08.885 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T10:03:08.885 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T10:03:08.885 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T10:03:08.886 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T10:03:08.886 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T10:03:08.888 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T10:03:08.888 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_24 2024-04-06T10:03:08.888 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T10:03:08.888 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T10:03:08.888 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T10:03:08.888 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T10:03:08.888 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T10:03:08.888 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T10:03:08.888 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T10:03:08.888 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T10:03:08.888 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T10:03:08.889 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T10:03:08.889 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T10:03:08.889 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:03:08.889 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T10:03:08.889 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T10:03:08.889 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T10:03:08.889 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T10:03:08.891 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T10:03:08.891 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T10:03:08.891 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T10:03:08.891 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T10:03:08.891 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T10:03:08.891 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T10:03:08.891 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:03:08.891 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T10:03:08.891 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T10:03:08.891 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T10:03:08.891 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T10:03:08.891 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T10:03:08.892 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T10:03:08.892 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T10:03:08.914 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T10:03:08.914 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T10:03:08.915 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T10:03:08.915 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_24 2024-04-06T10:03:09.019 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:Invalid command: unused arguments: ['mirror/image_24'] 2024-04-06T10:03:09.019 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:rbd mirror flush : flush rbd mirror 2024-04-06T10:03:09.019 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:admin_socket: invalid command 2024-04-06T10:03:09.024 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T10:03:09.024 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/image_24 2024-04-06T10:03:09.024 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:1 2024-04-06T10:03:09.024 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T10:03:09.024 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T10:03:09.024 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-06T10:03:09.024 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:1 2024-04-06T10:03:09.024 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T10:03:09.024 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T10:03:09.024 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T10:03:09.024 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=1 2024-04-06T10:03:09.024 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-06T10:03:09.024 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T10:03:09.025 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:03:09.025 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=1 2024-04-06T10:03:09.025 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=1 2024-04-06T10:03:09.025 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-06T10:03:09.025 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:1 2024-04-06T10:03:09.025 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T10:03:09.025 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T10:03:09.025 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-06T10:03:09.025 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:1 2024-04-06T10:03:09.025 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T10:03:09.025 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T10:03:09.025 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:03:09.025 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=1 2024-04-06T10:03:09.026 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-06T10:03:09.026 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T10:03:09.026 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T10:03:09.026 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=1 2024-04-06T10:03:09.026 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=1 2024-04-06T10:03:09.026 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-06T10:03:09.049 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T10:03:09.049 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T10:03:09.049 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T10:03:09.049 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/image_24 2024-04-06T10:03:09.184 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T10:03:09.185 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror image_24 2024-04-06T10:03:09.185 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:03:09.185 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:03:09.185 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_24 2024-04-06T10:03:09.186 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror image_24 '' 2024-04-06T10:03:09.186 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:03:09.186 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:03:09.186 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_24 2024-04-06T10:03:09.186 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T10:03:09.186 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-image_24.status 2024-04-06T10:03:09.187 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-image_24.status 2024-04-06T10:03:09.187 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T10:03:09.188 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-image_24.status 2024-04-06T10:03:09.188 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_24 2024-04-06T10:03:09.188 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_24.status 2024-04-06T10:03:09.242 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 1 2024-04-06T10:03:09.242 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 1 2024-04-06T10:03:09.242 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T10:03:09.242 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=5, tag_tid=2, entry_tid=1], [object_number=4, tag_tid=2, entry_tid=0], [object_number=7, tag_tid=1, entry_tid=6903], [object_number=6, tag_tid=1, entry_tid=6902]]], state=connected] 2024-04-06T10:03:09.242 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=4, tag_tid=1, entry_tid=6904], [object_number=7, tag_tid=1, entry_tid=6903], [object_number=6, tag_tid=1, entry_tid=6902], [object_number=5, tag_tid=1, entry_tid=6901]]], state=connected] 2024-04-06T10:03:09.246 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_24.status 2024-04-06T10:03:09.247 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=5, tag_tid=2, entry_tid=1' 2024-04-06T10:03:09.248 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror image_24 2024-04-06T10:03:09.248 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:03:09.248 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:03:09.248 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_24 2024-04-06T10:03:09.248 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror image_24 '..*' 2024-04-06T10:03:09.249 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:03:09.249 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:03:09.249 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_24 2024-04-06T10:03:09.249 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T10:03:09.249 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-image_24.status 2024-04-06T10:03:09.249 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-image_24.status 2024-04-06T10:03:09.249 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T10:03:09.249 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-image_24.status 2024-04-06T10:03:09.250 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_24 2024-04-06T10:03:09.250 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_24.status 2024-04-06T10:03:09.295 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 1 2024-04-06T10:03:09.295 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 1 2024-04-06T10:03:09.295 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T10:03:09.295 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=5, tag_tid=2, entry_tid=1], [object_number=4, tag_tid=2, entry_tid=0], [object_number=7, tag_tid=1, entry_tid=6903], [object_number=6, tag_tid=1, entry_tid=6902]]], state=connected] 2024-04-06T10:03:09.295 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=4, tag_tid=1, entry_tid=6904], [object_number=7, tag_tid=1, entry_tid=6903], [object_number=6, tag_tid=1, entry_tid=6902], [object_number=5, tag_tid=1, entry_tid=6901]]], state=connected] 2024-04-06T10:03:09.299 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_24.status 2024-04-06T10:03:09.301 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=4, tag_tid=1, entry_tid=6904' 2024-04-06T10:03:09.301 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=5, tag_tid=2, entry_tid=1' -a 'object_number=5, tag_tid=2, entry_tid=1' = 'object_number=4, tag_tid=1, entry_tid=6904' 2024-04-06T10:03:09.301 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=4, tag_tid=1, entry_tid=6904' '!=' 'object_number=4, tag_tid=1, entry_tid=6904' 2024-04-06T10:03:09.301 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T10:03:09.301 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.8 2024-04-06T10:03:09.655 INFO:tasks.rbd_mirror_thrash:kill cluster1.client.mirror.0 2024-04-06T10:03:09.655 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Sent signal 15 2024-04-06T10:03:09.655 INFO:tasks.rbd_mirror_thrash:waiting for 5 secs before reviving daemons 2024-04-06T10:03:09.657 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi192.stderr:2024-04-06T10:03:09.659+0000 7f3852698640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.0 (PID: 142506) UID: 1000 2024-04-06T10:03:10.103 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror image_24 2024-04-06T10:03:10.103 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T10:03:10.103 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:03:10.103 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_24 2024-04-06T10:03:10.103 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T10:03:10.103 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n image_24 ']' 2024-04-06T10:03:10.103 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/image_24' 2024-04-06T10:03:10.103 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_24 2024-04-06T10:03:10.103 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T10:03:10.103 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T10:03:10.103 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T10:03:10.103 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T10:03:10.103 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T10:03:10.104 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T10:03:10.104 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T10:03:10.104 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T10:03:10.104 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T10:03:10.108 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T10:03:10.108 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_24 2024-04-06T10:03:10.108 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T10:03:10.108 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T10:03:10.108 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T10:03:10.108 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T10:03:10.108 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T10:03:10.108 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T10:03:10.109 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T10:03:10.109 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T10:03:10.109 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T10:03:10.109 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T10:03:10.109 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T10:03:10.109 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:03:10.109 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T10:03:10.109 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T10:03:10.110 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T10:03:10.110 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T10:03:10.110 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T10:03:10.110 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T10:03:10.110 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T10:03:10.110 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T10:03:10.110 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T10:03:10.110 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T10:03:10.110 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:03:10.110 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T10:03:10.110 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T10:03:10.110 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T10:03:10.111 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T10:03:10.111 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T10:03:10.111 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T10:03:10.111 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T10:03:10.135 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T10:03:10.135 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T10:03:10.135 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T10:03:10.135 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_24 2024-04-06T10:03:10.233 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:admin_socket: exception getting command descriptions: [Errno 2] No such file or directory 2024-04-06T10:03:10.236 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T10:03:10.236 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/image_24 2024-04-06T10:03:10.236 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:1 2024-04-06T10:03:10.236 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T10:03:10.236 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T10:03:10.236 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-06T10:03:10.236 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:1 2024-04-06T10:03:10.236 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T10:03:10.236 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T10:03:10.236 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T10:03:10.236 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=1 2024-04-06T10:03:10.236 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-06T10:03:10.237 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T10:03:10.237 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:03:10.237 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=1 2024-04-06T10:03:10.237 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=1 2024-04-06T10:03:10.238 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-06T10:03:10.239 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:1 2024-04-06T10:03:10.239 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T10:03:10.239 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T10:03:10.239 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-06T10:03:10.239 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:1 2024-04-06T10:03:10.239 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T10:03:10.239 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T10:03:10.239 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:03:10.239 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=1 2024-04-06T10:03:10.239 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-06T10:03:10.239 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T10:03:10.239 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T10:03:10.240 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=1 2024-04-06T10:03:10.240 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=1 2024-04-06T10:03:10.240 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-06T10:03:10.262 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T10:03:10.263 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T10:03:10.263 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T10:03:10.263 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/image_24 2024-04-06T10:03:10.398 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T10:03:10.399 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror image_24 2024-04-06T10:03:10.399 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:03:10.399 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:03:10.399 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_24 2024-04-06T10:03:10.399 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror image_24 '' 2024-04-06T10:03:10.399 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:03:10.400 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:03:10.400 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_24 2024-04-06T10:03:10.400 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T10:03:10.400 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-image_24.status 2024-04-06T10:03:10.400 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-image_24.status 2024-04-06T10:03:10.401 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T10:03:10.402 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-image_24.status 2024-04-06T10:03:10.403 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_24 2024-04-06T10:03:10.403 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_24.status 2024-04-06T10:03:10.454 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 1 2024-04-06T10:03:10.454 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 1 2024-04-06T10:03:10.454 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T10:03:10.454 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=5, tag_tid=2, entry_tid=1], [object_number=4, tag_tid=2, entry_tid=0], [object_number=7, tag_tid=1, entry_tid=6903], [object_number=6, tag_tid=1, entry_tid=6902]]], state=connected] 2024-04-06T10:03:10.455 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=4, tag_tid=1, entry_tid=6904], [object_number=7, tag_tid=1, entry_tid=6903], [object_number=6, tag_tid=1, entry_tid=6902], [object_number=5, tag_tid=1, entry_tid=6901]]], state=connected] 2024-04-06T10:03:10.458 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_24.status 2024-04-06T10:03:10.459 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=5, tag_tid=2, entry_tid=1' 2024-04-06T10:03:10.461 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror image_24 2024-04-06T10:03:10.461 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:03:10.461 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:03:10.461 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_24 2024-04-06T10:03:10.461 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror image_24 '..*' 2024-04-06T10:03:10.461 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:03:10.461 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:03:10.461 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_24 2024-04-06T10:03:10.461 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T10:03:10.461 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-image_24.status 2024-04-06T10:03:10.461 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-image_24.status 2024-04-06T10:03:10.461 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T10:03:10.462 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-image_24.status 2024-04-06T10:03:10.462 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_24 2024-04-06T10:03:10.462 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_24.status 2024-04-06T10:03:10.507 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 1 2024-04-06T10:03:10.507 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 1 2024-04-06T10:03:10.507 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T10:03:10.507 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=5, tag_tid=2, entry_tid=1], [object_number=4, tag_tid=2, entry_tid=0], [object_number=7, tag_tid=1, entry_tid=6903], [object_number=6, tag_tid=1, entry_tid=6902]]], state=connected] 2024-04-06T10:03:10.507 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=4, tag_tid=1, entry_tid=6904], [object_number=7, tag_tid=1, entry_tid=6903], [object_number=6, tag_tid=1, entry_tid=6902], [object_number=5, tag_tid=1, entry_tid=6901]]], state=connected] 2024-04-06T10:03:10.511 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_24.status 2024-04-06T10:03:10.512 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=4, tag_tid=1, entry_tid=6904' 2024-04-06T10:03:10.513 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=5, tag_tid=2, entry_tid=1' -a 'object_number=5, tag_tid=2, entry_tid=1' = 'object_number=4, tag_tid=1, entry_tid=6904' 2024-04-06T10:03:10.513 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=4, tag_tid=1, entry_tid=6904' '!=' 'object_number=4, tag_tid=1, entry_tid=6904' 2024-04-06T10:03:10.513 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T10:03:10.513 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 1.6 2024-04-06T10:03:12.114 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror image_24 2024-04-06T10:03:12.115 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T10:03:12.115 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:03:12.115 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_24 2024-04-06T10:03:12.115 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T10:03:12.115 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n image_24 ']' 2024-04-06T10:03:12.115 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/image_24' 2024-04-06T10:03:12.115 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_24 2024-04-06T10:03:12.115 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T10:03:12.115 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T10:03:12.115 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T10:03:12.115 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T10:03:12.115 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T10:03:12.115 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T10:03:12.116 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T10:03:12.116 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T10:03:12.116 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T10:03:12.118 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T10:03:12.118 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_24 2024-04-06T10:03:12.118 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T10:03:12.118 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T10:03:12.119 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T10:03:12.119 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T10:03:12.119 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T10:03:12.119 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T10:03:12.119 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T10:03:12.119 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T10:03:12.119 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T10:03:12.119 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T10:03:12.119 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T10:03:12.119 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:03:12.119 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T10:03:12.119 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T10:03:12.120 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T10:03:12.120 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T10:03:12.120 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T10:03:12.120 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T10:03:12.121 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T10:03:12.121 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T10:03:12.121 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T10:03:12.121 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T10:03:12.121 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:03:12.121 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T10:03:12.121 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T10:03:12.121 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T10:03:12.121 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T10:03:12.121 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T10:03:12.121 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T10:03:12.121 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T10:03:12.146 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T10:03:12.147 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T10:03:12.147 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T10:03:12.147 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_24 2024-04-06T10:03:12.251 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:admin_socket: exception getting command descriptions: [Errno 2] No such file or directory 2024-04-06T10:03:12.253 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T10:03:12.253 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/image_24 2024-04-06T10:03:12.253 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:1 2024-04-06T10:03:12.253 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T10:03:12.253 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T10:03:12.254 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-06T10:03:12.254 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:1 2024-04-06T10:03:12.254 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T10:03:12.254 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T10:03:12.254 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T10:03:12.254 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=1 2024-04-06T10:03:12.254 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-06T10:03:12.254 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T10:03:12.254 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:03:12.254 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=1 2024-04-06T10:03:12.254 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=1 2024-04-06T10:03:12.256 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-06T10:03:12.256 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:1 2024-04-06T10:03:12.256 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T10:03:12.256 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T10:03:12.256 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-06T10:03:12.256 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:1 2024-04-06T10:03:12.256 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T10:03:12.256 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T10:03:12.256 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:03:12.256 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=1 2024-04-06T10:03:12.256 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-06T10:03:12.256 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T10:03:12.256 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T10:03:12.256 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=1 2024-04-06T10:03:12.257 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=1 2024-04-06T10:03:12.257 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-06T10:03:12.275 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T10:03:12.275 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T10:03:12.275 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T10:03:12.275 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/image_24 2024-04-06T10:03:12.412 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T10:03:12.413 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror image_24 2024-04-06T10:03:12.413 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:03:12.413 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:03:12.413 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_24 2024-04-06T10:03:12.413 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror image_24 '' 2024-04-06T10:03:12.413 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:03:12.414 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:03:12.414 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_24 2024-04-06T10:03:12.414 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T10:03:12.414 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-image_24.status 2024-04-06T10:03:12.414 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-image_24.status 2024-04-06T10:03:12.414 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T10:03:12.416 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-image_24.status 2024-04-06T10:03:12.417 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_24 2024-04-06T10:03:12.417 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_24.status 2024-04-06T10:03:12.461 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 1 2024-04-06T10:03:12.461 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 1 2024-04-06T10:03:12.461 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T10:03:12.461 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=5, tag_tid=2, entry_tid=1], [object_number=4, tag_tid=2, entry_tid=0], [object_number=7, tag_tid=1, entry_tid=6903], [object_number=6, tag_tid=1, entry_tid=6902]]], state=connected] 2024-04-06T10:03:12.461 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=4, tag_tid=1, entry_tid=6904], [object_number=7, tag_tid=1, entry_tid=6903], [object_number=6, tag_tid=1, entry_tid=6902], [object_number=5, tag_tid=1, entry_tid=6901]]], state=connected] 2024-04-06T10:03:12.464 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_24.status 2024-04-06T10:03:12.466 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=5, tag_tid=2, entry_tid=1' 2024-04-06T10:03:12.467 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror image_24 2024-04-06T10:03:12.467 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:03:12.467 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:03:12.467 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_24 2024-04-06T10:03:12.467 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror image_24 '..*' 2024-04-06T10:03:12.467 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:03:12.467 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:03:12.467 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_24 2024-04-06T10:03:12.467 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T10:03:12.467 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-image_24.status 2024-04-06T10:03:12.467 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-image_24.status 2024-04-06T10:03:12.468 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T10:03:12.468 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-image_24.status 2024-04-06T10:03:12.468 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_24 2024-04-06T10:03:12.468 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_24.status 2024-04-06T10:03:12.514 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 1 2024-04-06T10:03:12.514 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 1 2024-04-06T10:03:12.514 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T10:03:12.514 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=5, tag_tid=2, entry_tid=1], [object_number=4, tag_tid=2, entry_tid=0], [object_number=7, tag_tid=1, entry_tid=6903], [object_number=6, tag_tid=1, entry_tid=6902]]], state=connected] 2024-04-06T10:03:12.514 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=4, tag_tid=1, entry_tid=6904], [object_number=7, tag_tid=1, entry_tid=6903], [object_number=6, tag_tid=1, entry_tid=6902], [object_number=5, tag_tid=1, entry_tid=6901]]], state=connected] 2024-04-06T10:03:12.516 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_24.status 2024-04-06T10:03:12.518 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=4, tag_tid=1, entry_tid=6904' 2024-04-06T10:03:12.518 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=5, tag_tid=2, entry_tid=1' -a 'object_number=5, tag_tid=2, entry_tid=1' = 'object_number=4, tag_tid=1, entry_tid=6904' 2024-04-06T10:03:12.518 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=4, tag_tid=1, entry_tid=6904' '!=' 'object_number=4, tag_tid=1, entry_tid=6904' 2024-04-06T10:03:12.518 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T10:03:12.519 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 2 2024-04-06T10:03:13.556 INFO:tasks.rbd_mirror_thrash:reviving cluster2.client.mirror.1 2024-04-06T10:03:13.556 INFO:tasks.rbd_mirror.cluster2.client.mirror.1:Restarting daemon 2024-04-06T10:03:13.557 DEBUG:teuthology.orchestra.run.smithi192:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.1 2024-04-06T10:03:13.560 INFO:tasks.rbd_mirror.cluster2.client.mirror.1:Started 2024-04-06T10:03:13.560 INFO:tasks.rbd_mirror_thrash:waiting for 62 secs before thrashing 2024-04-06T10:03:14.520 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror image_24 2024-04-06T10:03:14.520 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T10:03:14.520 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:03:14.520 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_24 2024-04-06T10:03:14.521 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T10:03:14.521 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n image_24 ']' 2024-04-06T10:03:14.521 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/image_24' 2024-04-06T10:03:14.521 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_24 2024-04-06T10:03:14.521 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T10:03:14.521 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T10:03:14.521 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T10:03:14.521 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T10:03:14.521 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T10:03:14.521 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T10:03:14.521 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T10:03:14.522 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T10:03:14.522 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T10:03:14.524 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T10:03:14.524 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_24 2024-04-06T10:03:14.525 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T10:03:14.525 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T10:03:14.525 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T10:03:14.525 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T10:03:14.526 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T10:03:14.526 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T10:03:14.526 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T10:03:14.526 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T10:03:14.526 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T10:03:14.526 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T10:03:14.526 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T10:03:14.526 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:03:14.526 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T10:03:14.526 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T10:03:14.527 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T10:03:14.527 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T10:03:14.528 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T10:03:14.528 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T10:03:14.528 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T10:03:14.528 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T10:03:14.528 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T10:03:14.528 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T10:03:14.528 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:03:14.528 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T10:03:14.528 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T10:03:14.528 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T10:03:14.528 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T10:03:14.528 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T10:03:14.528 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T10:03:14.528 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T10:03:14.551 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T10:03:14.551 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T10:03:14.552 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T10:03:14.552 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_24 2024-04-06T10:03:14.655 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:admin_socket: exception getting command descriptions: [Errno 2] No such file or directory 2024-04-06T10:03:14.656 INFO:tasks.rbd_mirror_thrash:waiting for cluster1.client.mirror.0 2024-04-06T10:03:14.656 INFO:teuthology.orchestra.run:waiting for 600 2024-04-06T10:03:14.656 INFO:tasks.rbd_mirror_thrash:reviving cluster1.client.mirror.0 2024-04-06T10:03:14.656 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Restarting daemon 2024-04-06T10:03:14.656 DEBUG:teuthology.orchestra.run.smithi192:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.0 2024-04-06T10:03:14.659 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T10:03:14.659 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/image_24 2024-04-06T10:03:14.659 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:1 2024-04-06T10:03:14.659 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T10:03:14.659 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T10:03:14.659 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-06T10:03:14.659 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:1 2024-04-06T10:03:14.659 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T10:03:14.660 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T10:03:14.660 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T10:03:14.660 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=1 2024-04-06T10:03:14.660 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-06T10:03:14.660 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T10:03:14.660 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:03:14.660 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=1 2024-04-06T10:03:14.660 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=1 2024-04-06T10:03:14.660 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-06T10:03:14.660 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:1 2024-04-06T10:03:14.660 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T10:03:14.660 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T10:03:14.660 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-06T10:03:14.660 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:1 2024-04-06T10:03:14.660 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T10:03:14.660 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T10:03:14.661 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:03:14.661 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=1 2024-04-06T10:03:14.661 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-06T10:03:14.661 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T10:03:14.661 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T10:03:14.661 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=1 2024-04-06T10:03:14.661 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=1 2024-04-06T10:03:14.661 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-06T10:03:14.662 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Started 2024-04-06T10:03:14.662 INFO:tasks.rbd_mirror_thrash:waiting for 72 secs before thrashing 2024-04-06T10:03:14.682 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T10:03:14.682 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T10:03:14.682 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T10:03:14.682 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/image_24 2024-04-06T10:03:14.838 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T10:03:14.838 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror image_24 2024-04-06T10:03:14.838 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:03:14.838 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:03:14.838 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_24 2024-04-06T10:03:14.838 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror image_24 '' 2024-04-06T10:03:14.838 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:03:14.838 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:03:14.838 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_24 2024-04-06T10:03:14.838 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T10:03:14.839 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-image_24.status 2024-04-06T10:03:14.839 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-image_24.status 2024-04-06T10:03:14.839 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T10:03:14.840 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-image_24.status 2024-04-06T10:03:14.840 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_24 2024-04-06T10:03:14.840 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_24.status 2024-04-06T10:03:14.886 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 1 2024-04-06T10:03:14.886 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 1 2024-04-06T10:03:14.886 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T10:03:14.886 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=5, tag_tid=2, entry_tid=1], [object_number=4, tag_tid=2, entry_tid=0], [object_number=7, tag_tid=1, entry_tid=6903], [object_number=6, tag_tid=1, entry_tid=6902]]], state=connected] 2024-04-06T10:03:14.886 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=4, tag_tid=1, entry_tid=6904], [object_number=7, tag_tid=1, entry_tid=6903], [object_number=6, tag_tid=1, entry_tid=6902], [object_number=5, tag_tid=1, entry_tid=6901]]], state=connected] 2024-04-06T10:03:14.890 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_24.status 2024-04-06T10:03:14.891 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=5, tag_tid=2, entry_tid=1' 2024-04-06T10:03:14.892 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror image_24 2024-04-06T10:03:14.892 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:03:14.892 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:03:14.892 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_24 2024-04-06T10:03:14.892 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror image_24 '..*' 2024-04-06T10:03:14.893 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:03:14.893 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:03:14.893 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_24 2024-04-06T10:03:14.893 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T10:03:14.893 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-image_24.status 2024-04-06T10:03:14.893 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-image_24.status 2024-04-06T10:03:14.893 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T10:03:14.893 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-image_24.status 2024-04-06T10:03:14.894 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_24 2024-04-06T10:03:14.894 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_24.status 2024-04-06T10:03:14.942 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 1 2024-04-06T10:03:14.942 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 1 2024-04-06T10:03:14.942 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T10:03:14.942 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=5, tag_tid=2, entry_tid=1], [object_number=4, tag_tid=2, entry_tid=0], [object_number=7, tag_tid=1, entry_tid=6903], [object_number=6, tag_tid=1, entry_tid=6902]]], state=connected] 2024-04-06T10:03:14.943 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=4, tag_tid=1, entry_tid=6904], [object_number=7, tag_tid=1, entry_tid=6903], [object_number=6, tag_tid=1, entry_tid=6902], [object_number=5, tag_tid=1, entry_tid=6901]]], state=connected] 2024-04-06T10:03:14.946 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_24.status 2024-04-06T10:03:14.947 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=4, tag_tid=1, entry_tid=6904' 2024-04-06T10:03:14.947 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=5, tag_tid=2, entry_tid=1' -a 'object_number=5, tag_tid=2, entry_tid=1' = 'object_number=4, tag_tid=1, entry_tid=6904' 2024-04-06T10:03:14.948 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=4, tag_tid=1, entry_tid=6904' '!=' 'object_number=4, tag_tid=1, entry_tid=6904' 2024-04-06T10:03:14.948 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T10:03:14.948 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 2 2024-04-06T10:03:16.949 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror image_24 2024-04-06T10:03:16.950 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T10:03:16.950 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:03:16.950 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_24 2024-04-06T10:03:16.950 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T10:03:16.950 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n image_24 ']' 2024-04-06T10:03:16.950 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/image_24' 2024-04-06T10:03:16.950 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_24 2024-04-06T10:03:16.950 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T10:03:16.950 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T10:03:16.950 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T10:03:16.950 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T10:03:16.951 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T10:03:16.951 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T10:03:16.951 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T10:03:16.951 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T10:03:16.951 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T10:03:16.956 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T10:03:16.956 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_24 2024-04-06T10:03:16.956 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T10:03:16.956 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T10:03:16.956 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T10:03:16.956 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T10:03:16.956 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T10:03:16.956 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T10:03:16.956 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T10:03:16.957 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T10:03:16.957 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T10:03:16.957 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T10:03:16.957 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T10:03:16.957 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:03:16.957 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T10:03:16.957 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T10:03:16.957 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T10:03:16.957 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T10:03:16.957 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T10:03:16.957 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T10:03:16.957 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T10:03:16.957 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T10:03:16.957 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T10:03:16.957 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T10:03:16.958 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:03:16.958 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T10:03:16.958 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T10:03:16.958 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T10:03:16.958 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T10:03:16.958 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T10:03:16.958 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T10:03:16.958 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T10:03:16.980 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T10:03:16.981 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T10:03:16.981 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T10:03:16.981 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_24 2024-04-06T10:03:17.088 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:Invalid command: unused arguments: ['mirror/image_24'] 2024-04-06T10:03:17.088 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:rbd mirror flush : flush rbd mirror 2024-04-06T10:03:17.088 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:admin_socket: invalid command 2024-04-06T10:03:17.092 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T10:03:17.092 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/image_24 2024-04-06T10:03:17.092 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:1 2024-04-06T10:03:17.092 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T10:03:17.092 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T10:03:17.092 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-06T10:03:17.092 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:1 2024-04-06T10:03:17.092 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T10:03:17.093 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T10:03:17.093 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T10:03:17.093 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=1 2024-04-06T10:03:17.093 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-06T10:03:17.093 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T10:03:17.093 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:03:17.093 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=1 2024-04-06T10:03:17.093 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=1 2024-04-06T10:03:17.093 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-06T10:03:17.093 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:1 2024-04-06T10:03:17.093 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T10:03:17.093 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T10:03:17.093 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-06T10:03:17.094 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:1 2024-04-06T10:03:17.094 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T10:03:17.094 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T10:03:17.094 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:03:17.094 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=1 2024-04-06T10:03:17.094 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-06T10:03:17.094 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T10:03:17.094 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T10:03:17.095 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=1 2024-04-06T10:03:17.095 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=1 2024-04-06T10:03:17.095 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-06T10:03:17.116 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T10:03:17.116 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T10:03:17.117 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T10:03:17.117 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/image_24 2024-04-06T10:03:17.260 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T10:03:17.261 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror image_24 2024-04-06T10:03:17.261 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:03:17.262 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:03:17.262 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_24 2024-04-06T10:03:17.262 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror image_24 '' 2024-04-06T10:03:17.262 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:03:17.262 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:03:17.262 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_24 2024-04-06T10:03:17.262 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T10:03:17.262 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-image_24.status 2024-04-06T10:03:17.263 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T10:03:17.263 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-image_24.status 2024-04-06T10:03:17.264 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-image_24.status 2024-04-06T10:03:17.264 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_24 2024-04-06T10:03:17.264 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_24.status 2024-04-06T10:03:17.311 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 1 2024-04-06T10:03:17.311 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 1 2024-04-06T10:03:17.311 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T10:03:17.311 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=5, tag_tid=2, entry_tid=1], [object_number=4, tag_tid=2, entry_tid=0], [object_number=7, tag_tid=1, entry_tid=6903], [object_number=6, tag_tid=1, entry_tid=6902]]], state=connected] 2024-04-06T10:03:17.311 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=5, tag_tid=2, entry_tid=1], [object_number=4, tag_tid=2, entry_tid=0], [object_number=7, tag_tid=1, entry_tid=6903], [object_number=6, tag_tid=1, entry_tid=6902]]], state=connected] 2024-04-06T10:03:17.315 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_24.status 2024-04-06T10:03:17.316 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=5, tag_tid=2, entry_tid=1' 2024-04-06T10:03:17.317 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror image_24 2024-04-06T10:03:17.317 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:03:17.317 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:03:17.318 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_24 2024-04-06T10:03:17.318 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror image_24 '..*' 2024-04-06T10:03:17.318 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:03:17.318 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:03:17.318 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_24 2024-04-06T10:03:17.318 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T10:03:17.318 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-image_24.status 2024-04-06T10:03:17.318 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-image_24.status 2024-04-06T10:03:17.318 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T10:03:17.319 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-image_24.status 2024-04-06T10:03:17.319 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_24 2024-04-06T10:03:17.320 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_24.status 2024-04-06T10:03:17.363 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 1 2024-04-06T10:03:17.364 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 1 2024-04-06T10:03:17.364 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T10:03:17.364 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=5, tag_tid=2, entry_tid=1], [object_number=4, tag_tid=2, entry_tid=0], [object_number=7, tag_tid=1, entry_tid=6903], [object_number=6, tag_tid=1, entry_tid=6902]]], state=connected] 2024-04-06T10:03:17.364 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=5, tag_tid=2, entry_tid=1], [object_number=4, tag_tid=2, entry_tid=0], [object_number=7, tag_tid=1, entry_tid=6903], [object_number=6, tag_tid=1, entry_tid=6902]]], state=connected] 2024-04-06T10:03:17.367 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_24.status 2024-04-06T10:03:17.369 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=5, tag_tid=2, entry_tid=1' 2024-04-06T10:03:17.369 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=5, tag_tid=2, entry_tid=1' -a 'object_number=5, tag_tid=2, entry_tid=1' = 'object_number=5, tag_tid=2, entry_tid=1' 2024-04-06T10:03:17.369 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T10:03:17.369 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_snap_present cluster1 mirror image_24 snap 2024-04-06T10:03:17.369 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T10:03:17.370 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:03:17.370 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_24 2024-04-06T10:03:17.370 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local snap_name=snap 2024-04-06T10:03:17.370 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local s 2024-04-06T10:03:17.370 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-04-06T10:03:17.370 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 1 2024-04-06T10:03:18.370 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 info mirror/image_24@snap 2024-04-06T10:03:18.436 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:rbd image 'image_24': 2024-04-06T10:03:18.436 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: size 128 MiB in 32 objects 2024-04-06T10:03:18.436 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: order 22 (4 MiB objects) 2024-04-06T10:03:18.437 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: snapshot_count: 1 2024-04-06T10:03:18.437 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: id: 10ac6dcdb469 2024-04-06T10:03:18.437 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: block_name_prefix: rbd_data.10ac6dcdb469 2024-04-06T10:03:18.437 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: format: 2 2024-04-06T10:03:18.437 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: features: layering, exclusive-lock, journaling 2024-04-06T10:03:18.437 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: op_features: 2024-04-06T10:03:18.437 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: flags: 2024-04-06T10:03:18.437 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: create_timestamp: Sat Apr 6 09:54:21 2024 2024-04-06T10:03:18.437 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: access_timestamp: Sat Apr 6 09:54:21 2024 2024-04-06T10:03:18.437 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: modify_timestamp: Sat Apr 6 09:54:21 2024 2024-04-06T10:03:18.437 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: protected: False 2024-04-06T10:03:18.437 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: journal: 10ac6dcdb469 2024-04-06T10:03:18.437 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: mirroring state: enabled 2024-04-06T10:03:18.437 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: mirroring mode: journal 2024-04-06T10:03:18.437 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: mirroring global id: 34872b5c-78d0-4a3a-8ef8-35ba2b33e53e 2024-04-06T10:03:18.438 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: mirroring primary: false 2024-04-06T10:03:18.441 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T10:03:18.441 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ compare_image_snaps mirror image_24 snap 2024-04-06T10:03:18.441 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:03:18.441 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_24 2024-04-06T10:03:18.441 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local snap_name=snap 2024-04-06T10:03:18.442 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local ret=0 2024-04-06T10:03:18.442 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local rmt_export=/tmp/tmp.BhW6xKmROU/cluster2-mirror-image_24.export 2024-04-06T10:03:18.442 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loc_export=/tmp/tmp.BhW6xKmROU/cluster1-mirror-image_24.export 2024-04-06T10:03:18.442 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rm -f /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_24.export /tmp/tmp.BhW6xKmROU/cluster1-mirror-image_24.export 2024-04-06T10:03:18.442 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster2 -p mirror export image_24@snap /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_24.export 2024-04-06T10:03:18.607 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-04-06T10:03:18.615 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 -p mirror export image_24@snap /tmp/tmp.BhW6xKmROU/cluster1-mirror-image_24.export 2024-04-06T10:03:18.912 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-04-06T10:03:18.925 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmp /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_24.export /tmp/tmp.BhW6xKmROU/cluster1-mirror-image_24.export 2024-04-06T10:03:19.005 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rm -f /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_24.export /tmp/tmp.BhW6xKmROU/cluster1-mirror-image_24.export 2024-04-06T10:03:19.047 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T10:03:19.047 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-06T10:03:19.047 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ image=image_25 2024-04-06T10:03:19.047 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ create_snap cluster2 mirror image_25 snap 2024-04-06T10:03:19.047 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-04-06T10:03:19.048 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:03:19.048 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_25 2024-04-06T10:03:19.048 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local snap_name=snap 2024-04-06T10:03:19.048 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster2 -p mirror snap create image_25@snap --debug-rbd=20 --debug-journaler=20 2024-04-06T10:03:19.880 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_image_replay_started cluster1 mirror image_25 2024-04-06T10:03:19.880 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T10:03:19.881 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:03:19.881 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_25 2024-04-06T10:03:19.881 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_image_replay_state cluster1 mirror image_25 started 2024-04-06T10:03:19.881 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T10:03:19.881 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:03:19.881 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_25 2024-04-06T10:03:19.881 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state=started 2024-04-06T10:03:19.881 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local s 2024-04-06T10:03:19.881 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T10:03:19.881 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 1 2024-04-06T10:03:20.882 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test_image_replay_state cluster1 mirror image_25 started 2024-04-06T10:03:20.882 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T10:03:20.882 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:03:20.882 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_25 2024-04-06T10:03:20.882 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local test_state=started 2024-04-06T10:03:20.882 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local status_result 2024-04-06T10:03:20.882 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local current_state=stopped 2024-04-06T10:03:20.883 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_25 2024-04-06T10:03:20.884 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_instance=cluster1 2024-04-06T10:03:20.884 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -i state 2024-04-06T10:03:20.884 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ shift 2024-04-06T10:03:20.884 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T10:03:20.884 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance=cluster1 2024-04-06T10:03:20.884 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local loop_instance 2024-04-06T10:03:20.884 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T10:03:20.884 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sleep 0 2024-04-06T10:03:20.885 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-04-06T10:03:20.885 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ seq 0 3 2024-04-06T10:03:20.887 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T10:03:20.887 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_25 2024-04-06T10:03:20.887 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1:0 2024-04-06T10:03:20.887 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ shift 2024-04-06T10:03:20.887 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T10:03:20.888 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-04-06T10:03:20.888 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T10:03:20.888 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=cluster 2024-04-06T10:03:20.888 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T10:03:20.888 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:03:20.888 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T10:03:20.888 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T10:03:20.888 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval cluster=cluster1 2024-04-06T10:03:20.888 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ cluster=cluster1 2024-04-06T10:03:20.888 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T10:03:20.888 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T10:03:20.888 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-04-06T10:03:20.890 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local local_cluster=cluster1:0 2024-04-06T10:03:20.890 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local cluster=cluster1 2024-04-06T10:03:20.890 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local instance 2024-04-06T10:03:20.890 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T10:03:20.890 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local val=cluster1:0 2024-04-06T10:03:20.890 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local cluster_var_name=local_cluster 2024-04-06T10:03:20.890 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local instance_var_name=instance 2024-04-06T10:03:20.890 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ cluster=cluster1 2024-04-06T10:03:20.890 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T10:03:20.890 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ '[' 0 = cluster1:0 ']' 2024-04-06T10:03:20.890 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ eval local_cluster=cluster1 2024-04-06T10:03:20.890 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++++ local_cluster=cluster1 2024-04-06T10:03:20.891 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ eval instance=0 2024-04-06T10:03:20.891 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++++ instance=0 2024-04-06T10:03:20.891 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T10:03:20.913 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T10:03:20.913 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T10:03:20.913 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T10:03:20.913 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_25 2024-04-06T10:03:21.020 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:Invalid command: unused arguments: ['mirror/image_25'] 2024-04-06T10:03:21.021 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:rbd mirror status : get status for rbd mirror 2024-04-06T10:03:21.021 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:admin_socket: invalid command 2024-04-06T10:03:21.022 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T10:03:21.022 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_25 2024-04-06T10:03:21.022 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1:1 2024-04-06T10:03:21.022 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ shift 2024-04-06T10:03:21.023 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T10:03:21.023 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-04-06T10:03:21.023 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:1 2024-04-06T10:03:21.023 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=cluster 2024-04-06T10:03:21.023 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T10:03:21.023 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:03:21.023 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=1 2024-04-06T10:03:21.023 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-06T10:03:21.023 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval cluster=cluster1 2024-04-06T10:03:21.023 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ cluster=cluster1 2024-04-06T10:03:21.023 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=1 2024-04-06T10:03:21.023 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=1 2024-04-06T10:03:21.024 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-04-06T10:03:21.024 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local local_cluster=cluster1:1 2024-04-06T10:03:21.024 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local cluster=cluster1 2024-04-06T10:03:21.024 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local instance 2024-04-06T10:03:21.024 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-06T10:03:21.024 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local val=cluster1:1 2024-04-06T10:03:21.024 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local cluster_var_name=local_cluster 2024-04-06T10:03:21.024 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local instance_var_name=instance 2024-04-06T10:03:21.024 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ cluster=cluster1 2024-04-06T10:03:21.025 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=1 2024-04-06T10:03:21.025 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ '[' 1 = cluster1:1 ']' 2024-04-06T10:03:21.025 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ eval local_cluster=cluster1 2024-04-06T10:03:21.025 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++++ local_cluster=cluster1 2024-04-06T10:03:21.025 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ eval instance=1 2024-04-06T10:03:21.025 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++++ instance=1 2024-04-06T10:03:21.025 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-06T10:03:21.048 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T10:03:21.048 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T10:03:21.048 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T10:03:21.048 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_25 2024-04-06T10:03:21.200 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ return 0 2024-04-06T10:03:21.201 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ status_result=' "state": "Replaying"' 2024-04-06T10:03:21.201 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep -i Replaying 2024-04-06T10:03:21.201 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ echo ' "state": "Replaying"' 2024-04-06T10:03:21.202 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: "state": "Replaying" 2024-04-06T10:03:21.203 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ current_state=started 2024-04-06T10:03:21.204 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test started = started 2024-04-06T10:03:21.204 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T10:03:21.204 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_25 2024-04-06T10:03:21.204 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local local_cluster=cluster1 2024-04-06T10:03:21.204 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-04-06T10:03:21.204 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:03:21.204 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_25 2024-04-06T10:03:21.204 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' journal = journal ']' 2024-04-06T10:03:21.204 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_journal_replay_complete cluster1 cluster2 mirror image_25 2024-04-06T10:03:21.204 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local local_cluster=cluster1 2024-04-06T10:03:21.204 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-04-06T10:03:21.204 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:03:21.204 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_25 2024-04-06T10:03:21.205 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local s master_pos mirror_pos last_mirror_pos 2024-04-06T10:03:21.205 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local master_tag master_entry mirror_tag mirror_entry 2024-04-06T10:03:21.205 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T10:03:21.205 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T10:03:21.205 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T10:03:21.406 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror image_25 2024-04-06T10:03:21.406 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T10:03:21.406 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:03:21.406 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_25 2024-04-06T10:03:21.406 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T10:03:21.406 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n image_25 ']' 2024-04-06T10:03:21.406 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/image_25' 2024-04-06T10:03:21.406 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_25 2024-04-06T10:03:21.406 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T10:03:21.407 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T10:03:21.407 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T10:03:21.407 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T10:03:21.407 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T10:03:21.407 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T10:03:21.407 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T10:03:21.407 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T10:03:21.407 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T10:03:21.411 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T10:03:21.411 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_25 2024-04-06T10:03:21.411 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T10:03:21.412 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T10:03:21.412 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T10:03:21.412 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T10:03:21.412 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T10:03:21.412 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T10:03:21.412 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T10:03:21.412 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T10:03:21.412 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T10:03:21.412 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T10:03:21.412 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T10:03:21.412 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:03:21.412 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T10:03:21.412 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T10:03:21.412 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T10:03:21.413 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T10:03:21.413 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T10:03:21.413 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T10:03:21.413 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T10:03:21.413 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T10:03:21.413 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T10:03:21.413 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T10:03:21.413 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:03:21.413 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T10:03:21.413 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T10:03:21.413 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T10:03:21.413 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T10:03:21.413 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T10:03:21.414 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T10:03:21.414 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T10:03:21.436 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T10:03:21.437 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T10:03:21.437 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T10:03:21.437 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_25 2024-04-06T10:03:21.539 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:Invalid command: unused arguments: ['mirror/image_25'] 2024-04-06T10:03:21.539 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:rbd mirror flush : flush rbd mirror 2024-04-06T10:03:21.539 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:admin_socket: invalid command 2024-04-06T10:03:21.541 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T10:03:21.541 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/image_25 2024-04-06T10:03:21.541 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:1 2024-04-06T10:03:21.541 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T10:03:21.542 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T10:03:21.542 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-06T10:03:21.542 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:1 2024-04-06T10:03:21.542 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T10:03:21.542 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T10:03:21.542 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T10:03:21.542 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=1 2024-04-06T10:03:21.542 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-06T10:03:21.542 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T10:03:21.542 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:03:21.542 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=1 2024-04-06T10:03:21.542 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=1 2024-04-06T10:03:21.544 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-06T10:03:21.544 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:1 2024-04-06T10:03:21.544 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T10:03:21.545 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T10:03:21.545 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-06T10:03:21.545 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:1 2024-04-06T10:03:21.545 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T10:03:21.545 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T10:03:21.545 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:03:21.545 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=1 2024-04-06T10:03:21.545 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-06T10:03:21.545 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T10:03:21.545 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T10:03:21.545 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=1 2024-04-06T10:03:21.545 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=1 2024-04-06T10:03:21.545 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-06T10:03:21.568 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T10:03:21.568 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T10:03:21.568 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T10:03:21.568 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/image_25 2024-04-06T10:03:21.704 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T10:03:21.705 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror image_25 2024-04-06T10:03:21.706 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:03:21.706 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:03:21.706 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_25 2024-04-06T10:03:21.706 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror image_25 '' 2024-04-06T10:03:21.706 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:03:21.706 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:03:21.706 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_25 2024-04-06T10:03:21.706 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T10:03:21.706 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-image_25.status 2024-04-06T10:03:21.707 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-image_25.status 2024-04-06T10:03:21.707 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T10:03:21.707 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-image_25.status 2024-04-06T10:03:21.708 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_25 2024-04-06T10:03:21.708 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_25.status 2024-04-06T10:03:21.764 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 1 2024-04-06T10:03:21.764 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 1 2024-04-06T10:03:21.764 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T10:03:21.764 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=5, tag_tid=3, entry_tid=1], [object_number=4, tag_tid=3, entry_tid=0], [object_number=6, tag_tid=1, entry_tid=7138], [object_number=7, tag_tid=1, entry_tid=7135]]], state=connected] 2024-04-06T10:03:21.764 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=5, tag_tid=2, entry_tid=1], [object_number=4, tag_tid=2, entry_tid=0], [object_number=6, tag_tid=1, entry_tid=7138], [object_number=7, tag_tid=1, entry_tid=7135]]], state=connected] 2024-04-06T10:03:21.767 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_25.status 2024-04-06T10:03:21.769 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=5, tag_tid=3, entry_tid=1' 2024-04-06T10:03:21.770 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror image_25 2024-04-06T10:03:21.770 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:03:21.770 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:03:21.770 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_25 2024-04-06T10:03:21.770 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror image_25 '..*' 2024-04-06T10:03:21.770 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:03:21.770 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:03:21.770 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_25 2024-04-06T10:03:21.771 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T10:03:21.771 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-image_25.status 2024-04-06T10:03:21.771 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-image_25.status 2024-04-06T10:03:21.771 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T10:03:21.771 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-image_25.status 2024-04-06T10:03:21.772 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_25 2024-04-06T10:03:21.772 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_25.status 2024-04-06T10:03:21.817 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 1 2024-04-06T10:03:21.817 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 1 2024-04-06T10:03:21.817 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T10:03:21.817 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=5, tag_tid=3, entry_tid=1], [object_number=4, tag_tid=3, entry_tid=0], [object_number=6, tag_tid=1, entry_tid=7138], [object_number=7, tag_tid=1, entry_tid=7135]]], state=connected] 2024-04-06T10:03:21.817 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=5, tag_tid=2, entry_tid=1], [object_number=4, tag_tid=2, entry_tid=0], [object_number=6, tag_tid=1, entry_tid=7138], [object_number=7, tag_tid=1, entry_tid=7135]]], state=connected] 2024-04-06T10:03:21.820 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_25.status 2024-04-06T10:03:21.823 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=5, tag_tid=2, entry_tid=1' 2024-04-06T10:03:21.823 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=5, tag_tid=3, entry_tid=1' -a 'object_number=5, tag_tid=3, entry_tid=1' = 'object_number=5, tag_tid=2, entry_tid=1' 2024-04-06T10:03:21.823 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=5, tag_tid=2, entry_tid=1' '!=' '' 2024-04-06T10:03:21.823 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T10:03:21.823 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=5, tag_tid=2, entry_tid=1' = '' 2024-04-06T10:03:21.823 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=5, tag_tid=2, entry_tid=1' 2024-04-06T10:03:21.823 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=5, tag_tid=3, entry_tid=1' 2024-04-06T10:03:21.823 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T10:03:21.823 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T10:03:21.824 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=3 2024-04-06T10:03:21.824 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=5, tag_tid=2, entry_tid=1' 2024-04-06T10:03:21.824 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T10:03:21.825 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T10:03:21.825 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=2 2024-04-06T10:03:21.826 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=5, tag_tid=3, entry_tid=1' 2024-04-06T10:03:21.826 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T10:03:21.826 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T10:03:21.827 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T10:03:21.828 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=5, tag_tid=2, entry_tid=1' 2024-04-06T10:03:21.828 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T10:03:21.828 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T10:03:21.829 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=1 2024-04-06T10:03:21.829 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 3 = 2 -a 1 -le 1 2024-04-06T10:03:21.829 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T10:03:21.829 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T10:03:21.830 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T10:03:22.031 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror image_25 2024-04-06T10:03:22.031 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T10:03:22.031 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:03:22.031 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_25 2024-04-06T10:03:22.032 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T10:03:22.032 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n image_25 ']' 2024-04-06T10:03:22.032 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/image_25' 2024-04-06T10:03:22.032 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_25 2024-04-06T10:03:22.032 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T10:03:22.032 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T10:03:22.032 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T10:03:22.032 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T10:03:22.032 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T10:03:22.032 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T10:03:22.032 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T10:03:22.033 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T10:03:22.033 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T10:03:22.036 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T10:03:22.036 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_25 2024-04-06T10:03:22.036 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T10:03:22.037 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T10:03:22.037 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T10:03:22.037 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T10:03:22.037 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T10:03:22.037 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T10:03:22.037 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T10:03:22.037 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T10:03:22.037 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T10:03:22.037 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T10:03:22.037 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T10:03:22.037 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:03:22.037 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T10:03:22.037 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T10:03:22.038 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T10:03:22.038 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T10:03:22.038 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T10:03:22.038 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T10:03:22.038 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T10:03:22.038 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T10:03:22.038 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T10:03:22.038 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T10:03:22.038 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:03:22.038 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T10:03:22.039 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T10:03:22.039 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T10:03:22.039 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T10:03:22.039 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T10:03:22.039 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T10:03:22.039 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T10:03:22.062 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T10:03:22.063 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T10:03:22.063 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T10:03:22.063 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_25 2024-04-06T10:03:22.169 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:Invalid command: unused arguments: ['mirror/image_25'] 2024-04-06T10:03:22.169 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:rbd mirror flush : flush rbd mirror 2024-04-06T10:03:22.169 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:admin_socket: invalid command 2024-04-06T10:03:22.176 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T10:03:22.177 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/image_25 2024-04-06T10:03:22.177 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:1 2024-04-06T10:03:22.177 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T10:03:22.177 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T10:03:22.177 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-06T10:03:22.177 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:1 2024-04-06T10:03:22.177 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T10:03:22.177 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T10:03:22.177 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T10:03:22.177 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=1 2024-04-06T10:03:22.177 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-06T10:03:22.177 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T10:03:22.178 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:03:22.178 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=1 2024-04-06T10:03:22.178 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=1 2024-04-06T10:03:22.178 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-06T10:03:22.178 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:1 2024-04-06T10:03:22.178 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T10:03:22.178 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T10:03:22.178 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-06T10:03:22.178 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:1 2024-04-06T10:03:22.178 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T10:03:22.178 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T10:03:22.178 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:03:22.178 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=1 2024-04-06T10:03:22.179 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-06T10:03:22.179 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T10:03:22.179 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T10:03:22.179 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=1 2024-04-06T10:03:22.179 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=1 2024-04-06T10:03:22.179 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-06T10:03:22.192 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T10:03:22.192 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T10:03:22.193 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T10:03:22.193 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/image_25 2024-04-06T10:03:22.335 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T10:03:22.337 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror image_25 2024-04-06T10:03:22.337 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:03:22.337 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:03:22.337 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_25 2024-04-06T10:03:22.337 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror image_25 '' 2024-04-06T10:03:22.337 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:03:22.337 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:03:22.337 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_25 2024-04-06T10:03:22.337 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T10:03:22.337 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-image_25.status 2024-04-06T10:03:22.338 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-image_25.status 2024-04-06T10:03:22.338 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T10:03:22.340 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-image_25.status 2024-04-06T10:03:22.340 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_25 2024-04-06T10:03:22.341 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_25.status 2024-04-06T10:03:22.387 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 1 2024-04-06T10:03:22.387 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 1 2024-04-06T10:03:22.387 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T10:03:22.387 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=5, tag_tid=3, entry_tid=1], [object_number=4, tag_tid=3, entry_tid=0], [object_number=6, tag_tid=1, entry_tid=7138], [object_number=7, tag_tid=1, entry_tid=7135]]], state=connected] 2024-04-06T10:03:22.387 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=5, tag_tid=2, entry_tid=1], [object_number=4, tag_tid=2, entry_tid=0], [object_number=6, tag_tid=1, entry_tid=7138], [object_number=7, tag_tid=1, entry_tid=7135]]], state=connected] 2024-04-06T10:03:22.390 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_25.status 2024-04-06T10:03:22.392 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=5, tag_tid=3, entry_tid=1' 2024-04-06T10:03:22.393 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror image_25 2024-04-06T10:03:22.393 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:03:22.393 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:03:22.393 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_25 2024-04-06T10:03:22.393 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror image_25 '..*' 2024-04-06T10:03:22.393 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:03:22.393 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:03:22.393 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_25 2024-04-06T10:03:22.393 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T10:03:22.394 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-image_25.status 2024-04-06T10:03:22.394 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-image_25.status 2024-04-06T10:03:22.394 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T10:03:22.394 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-image_25.status 2024-04-06T10:03:22.394 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_25 2024-04-06T10:03:22.394 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_25.status 2024-04-06T10:03:22.440 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 1 2024-04-06T10:03:22.440 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 1 2024-04-06T10:03:22.440 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T10:03:22.440 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=5, tag_tid=3, entry_tid=1], [object_number=4, tag_tid=3, entry_tid=0], [object_number=6, tag_tid=1, entry_tid=7138], [object_number=7, tag_tid=1, entry_tid=7135]]], state=connected] 2024-04-06T10:03:22.440 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=5, tag_tid=2, entry_tid=1], [object_number=4, tag_tid=2, entry_tid=0], [object_number=6, tag_tid=1, entry_tid=7138], [object_number=7, tag_tid=1, entry_tid=7135]]], state=connected] 2024-04-06T10:03:22.443 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_25.status 2024-04-06T10:03:22.445 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=5, tag_tid=2, entry_tid=1' 2024-04-06T10:03:22.445 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=5, tag_tid=3, entry_tid=1' -a 'object_number=5, tag_tid=3, entry_tid=1' = 'object_number=5, tag_tid=2, entry_tid=1' 2024-04-06T10:03:22.445 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=5, tag_tid=2, entry_tid=1' '!=' 'object_number=5, tag_tid=2, entry_tid=1' 2024-04-06T10:03:22.445 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T10:03:22.445 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.4 2024-04-06T10:03:22.847 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror image_25 2024-04-06T10:03:22.847 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T10:03:22.848 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:03:22.848 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_25 2024-04-06T10:03:22.848 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T10:03:22.848 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n image_25 ']' 2024-04-06T10:03:22.848 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/image_25' 2024-04-06T10:03:22.848 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_25 2024-04-06T10:03:22.848 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T10:03:22.848 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T10:03:22.848 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T10:03:22.848 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T10:03:22.848 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T10:03:22.848 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T10:03:22.848 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T10:03:22.849 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T10:03:22.849 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T10:03:22.852 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T10:03:22.852 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_25 2024-04-06T10:03:22.852 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T10:03:22.852 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T10:03:22.852 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T10:03:22.852 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T10:03:22.852 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T10:03:22.852 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T10:03:22.853 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T10:03:22.853 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T10:03:22.853 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T10:03:22.853 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T10:03:22.853 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T10:03:22.853 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:03:22.853 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T10:03:22.853 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T10:03:22.853 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T10:03:22.853 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T10:03:22.853 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T10:03:22.853 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T10:03:22.853 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T10:03:22.853 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T10:03:22.853 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T10:03:22.854 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T10:03:22.854 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:03:22.854 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T10:03:22.854 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T10:03:22.854 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T10:03:22.854 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T10:03:22.854 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T10:03:22.854 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T10:03:22.854 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T10:03:22.879 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T10:03:22.879 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T10:03:22.879 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T10:03:22.879 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_25 2024-04-06T10:03:22.998 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:Invalid command: unused arguments: ['mirror/image_25'] 2024-04-06T10:03:22.998 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:rbd mirror flush : flush rbd mirror 2024-04-06T10:03:22.998 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:admin_socket: invalid command 2024-04-06T10:03:22.998 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T10:03:22.998 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/image_25 2024-04-06T10:03:22.998 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:1 2024-04-06T10:03:22.998 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T10:03:22.998 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T10:03:22.998 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-06T10:03:22.999 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:1 2024-04-06T10:03:22.999 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T10:03:22.999 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T10:03:22.999 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T10:03:22.999 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=1 2024-04-06T10:03:22.999 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-06T10:03:22.999 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T10:03:22.999 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:03:22.999 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=1 2024-04-06T10:03:22.999 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=1 2024-04-06T10:03:22.999 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-06T10:03:22.999 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:1 2024-04-06T10:03:22.999 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T10:03:22.999 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T10:03:22.999 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-06T10:03:22.999 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:1 2024-04-06T10:03:23.000 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T10:03:23.000 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T10:03:23.000 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:03:23.000 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=1 2024-04-06T10:03:23.000 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-06T10:03:23.000 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T10:03:23.000 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T10:03:23.000 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=1 2024-04-06T10:03:23.000 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=1 2024-04-06T10:03:23.000 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-06T10:03:23.020 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T10:03:23.021 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T10:03:23.021 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T10:03:23.021 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/image_25 2024-04-06T10:03:23.158 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T10:03:23.160 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror image_25 2024-04-06T10:03:23.160 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:03:23.160 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:03:23.160 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_25 2024-04-06T10:03:23.160 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror image_25 '' 2024-04-06T10:03:23.160 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:03:23.160 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:03:23.160 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_25 2024-04-06T10:03:23.160 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T10:03:23.160 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-image_25.status 2024-04-06T10:03:23.161 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-image_25.status 2024-04-06T10:03:23.161 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T10:03:23.162 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-image_25.status 2024-04-06T10:03:23.162 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_25 2024-04-06T10:03:23.163 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_25.status 2024-04-06T10:03:23.217 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 1 2024-04-06T10:03:23.217 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 1 2024-04-06T10:03:23.217 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T10:03:23.217 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=5, tag_tid=3, entry_tid=1], [object_number=4, tag_tid=3, entry_tid=0], [object_number=6, tag_tid=1, entry_tid=7138], [object_number=7, tag_tid=1, entry_tid=7135]]], state=connected] 2024-04-06T10:03:23.217 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=5, tag_tid=2, entry_tid=1], [object_number=4, tag_tid=2, entry_tid=0], [object_number=6, tag_tid=1, entry_tid=7138], [object_number=7, tag_tid=1, entry_tid=7135]]], state=connected] 2024-04-06T10:03:23.221 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_25.status 2024-04-06T10:03:23.224 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=5, tag_tid=3, entry_tid=1' 2024-04-06T10:03:23.224 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror image_25 2024-04-06T10:03:23.224 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:03:23.224 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:03:23.224 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_25 2024-04-06T10:03:23.224 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror image_25 '..*' 2024-04-06T10:03:23.224 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:03:23.225 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:03:23.225 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_25 2024-04-06T10:03:23.225 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T10:03:23.225 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-image_25.status 2024-04-06T10:03:23.225 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-image_25.status 2024-04-06T10:03:23.225 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T10:03:23.225 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-image_25.status 2024-04-06T10:03:23.226 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_25 2024-04-06T10:03:23.226 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_25.status 2024-04-06T10:03:23.273 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 1 2024-04-06T10:03:23.273 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 1 2024-04-06T10:03:23.273 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T10:03:23.273 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=5, tag_tid=3, entry_tid=1], [object_number=4, tag_tid=3, entry_tid=0], [object_number=6, tag_tid=1, entry_tid=7138], [object_number=7, tag_tid=1, entry_tid=7135]]], state=connected] 2024-04-06T10:03:23.273 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=5, tag_tid=2, entry_tid=1], [object_number=4, tag_tid=2, entry_tid=0], [object_number=6, tag_tid=1, entry_tid=7138], [object_number=7, tag_tid=1, entry_tid=7135]]], state=connected] 2024-04-06T10:03:23.276 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_25.status 2024-04-06T10:03:23.278 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=5, tag_tid=2, entry_tid=1' 2024-04-06T10:03:23.278 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=5, tag_tid=3, entry_tid=1' -a 'object_number=5, tag_tid=3, entry_tid=1' = 'object_number=5, tag_tid=2, entry_tid=1' 2024-04-06T10:03:23.278 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=5, tag_tid=2, entry_tid=1' '!=' 'object_number=5, tag_tid=2, entry_tid=1' 2024-04-06T10:03:23.278 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T10:03:23.279 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.8 2024-04-06T10:03:24.080 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror image_25 2024-04-06T10:03:24.080 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T10:03:24.080 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:03:24.080 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_25 2024-04-06T10:03:24.080 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T10:03:24.081 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n image_25 ']' 2024-04-06T10:03:24.081 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/image_25' 2024-04-06T10:03:24.081 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_25 2024-04-06T10:03:24.081 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T10:03:24.081 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T10:03:24.081 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T10:03:24.081 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T10:03:24.081 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T10:03:24.081 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T10:03:24.081 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T10:03:24.082 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T10:03:24.082 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T10:03:24.084 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T10:03:24.084 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_25 2024-04-06T10:03:24.085 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T10:03:24.085 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T10:03:24.085 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T10:03:24.085 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T10:03:24.085 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T10:03:24.085 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T10:03:24.085 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T10:03:24.085 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T10:03:24.085 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T10:03:24.085 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T10:03:24.085 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T10:03:24.085 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:03:24.085 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T10:03:24.085 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T10:03:24.087 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T10:03:24.087 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T10:03:24.087 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T10:03:24.087 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T10:03:24.087 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T10:03:24.087 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T10:03:24.087 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T10:03:24.087 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T10:03:24.088 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:03:24.088 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T10:03:24.088 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T10:03:24.088 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T10:03:24.088 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T10:03:24.088 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T10:03:24.088 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T10:03:24.088 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T10:03:24.111 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T10:03:24.112 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T10:03:24.112 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T10:03:24.112 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_25 2024-04-06T10:03:24.223 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:Invalid command: unused arguments: ['mirror/image_25'] 2024-04-06T10:03:24.223 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:rbd mirror flush : flush rbd mirror 2024-04-06T10:03:24.223 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:admin_socket: invalid command 2024-04-06T10:03:24.225 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T10:03:24.225 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/image_25 2024-04-06T10:03:24.225 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:1 2024-04-06T10:03:24.225 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T10:03:24.225 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T10:03:24.226 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-06T10:03:24.226 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:1 2024-04-06T10:03:24.226 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T10:03:24.226 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T10:03:24.226 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T10:03:24.226 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=1 2024-04-06T10:03:24.226 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-06T10:03:24.226 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T10:03:24.226 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:03:24.226 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=1 2024-04-06T10:03:24.226 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=1 2024-04-06T10:03:24.226 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-06T10:03:24.226 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:1 2024-04-06T10:03:24.226 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T10:03:24.227 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T10:03:24.227 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-06T10:03:24.227 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:1 2024-04-06T10:03:24.227 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T10:03:24.227 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T10:03:24.227 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:03:24.227 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=1 2024-04-06T10:03:24.227 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-06T10:03:24.227 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T10:03:24.227 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T10:03:24.227 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=1 2024-04-06T10:03:24.227 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=1 2024-04-06T10:03:24.228 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-06T10:03:24.251 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T10:03:24.251 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T10:03:24.251 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T10:03:24.251 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/image_25 2024-04-06T10:03:24.387 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T10:03:24.389 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror image_25 2024-04-06T10:03:24.389 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:03:24.389 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:03:24.389 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_25 2024-04-06T10:03:24.389 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror image_25 '' 2024-04-06T10:03:24.389 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:03:24.390 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:03:24.390 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_25 2024-04-06T10:03:24.390 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T10:03:24.390 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-image_25.status 2024-04-06T10:03:24.390 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-image_25.status 2024-04-06T10:03:24.390 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T10:03:24.392 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-image_25.status 2024-04-06T10:03:24.393 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_25 2024-04-06T10:03:24.393 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_25.status 2024-04-06T10:03:24.448 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 1 2024-04-06T10:03:24.448 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 1 2024-04-06T10:03:24.448 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T10:03:24.448 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=5, tag_tid=3, entry_tid=1], [object_number=4, tag_tid=3, entry_tid=0], [object_number=6, tag_tid=1, entry_tid=7138], [object_number=7, tag_tid=1, entry_tid=7135]]], state=connected] 2024-04-06T10:03:24.448 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=5, tag_tid=2, entry_tid=1], [object_number=4, tag_tid=2, entry_tid=0], [object_number=6, tag_tid=1, entry_tid=7138], [object_number=7, tag_tid=1, entry_tid=7135]]], state=connected] 2024-04-06T10:03:24.452 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_25.status 2024-04-06T10:03:24.453 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=5, tag_tid=3, entry_tid=1' 2024-04-06T10:03:24.454 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror image_25 2024-04-06T10:03:24.454 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:03:24.454 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:03:24.454 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_25 2024-04-06T10:03:24.455 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror image_25 '..*' 2024-04-06T10:03:24.455 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:03:24.455 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:03:24.455 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_25 2024-04-06T10:03:24.455 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T10:03:24.455 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-image_25.status 2024-04-06T10:03:24.455 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-image_25.status 2024-04-06T10:03:24.455 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T10:03:24.455 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-image_25.status 2024-04-06T10:03:24.456 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_25 2024-04-06T10:03:24.456 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_25.status 2024-04-06T10:03:24.501 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 1 2024-04-06T10:03:24.501 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 1 2024-04-06T10:03:24.502 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T10:03:24.502 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=5, tag_tid=3, entry_tid=1], [object_number=4, tag_tid=3, entry_tid=0], [object_number=6, tag_tid=1, entry_tid=7138], [object_number=7, tag_tid=1, entry_tid=7135]]], state=connected] 2024-04-06T10:03:24.502 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=5, tag_tid=2, entry_tid=1], [object_number=4, tag_tid=2, entry_tid=0], [object_number=6, tag_tid=1, entry_tid=7138], [object_number=7, tag_tid=1, entry_tid=7135]]], state=connected] 2024-04-06T10:03:24.505 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_25.status 2024-04-06T10:03:24.507 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=5, tag_tid=2, entry_tid=1' 2024-04-06T10:03:24.507 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=5, tag_tid=3, entry_tid=1' -a 'object_number=5, tag_tid=3, entry_tid=1' = 'object_number=5, tag_tid=2, entry_tid=1' 2024-04-06T10:03:24.507 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=5, tag_tid=2, entry_tid=1' '!=' 'object_number=5, tag_tid=2, entry_tid=1' 2024-04-06T10:03:24.507 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T10:03:24.507 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 1.6 2024-04-06T10:03:26.109 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror image_25 2024-04-06T10:03:26.109 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T10:03:26.109 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:03:26.109 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_25 2024-04-06T10:03:26.110 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T10:03:26.110 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n image_25 ']' 2024-04-06T10:03:26.110 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/image_25' 2024-04-06T10:03:26.110 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_25 2024-04-06T10:03:26.110 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T10:03:26.110 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T10:03:26.110 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T10:03:26.110 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T10:03:26.110 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T10:03:26.110 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T10:03:26.110 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T10:03:26.110 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T10:03:26.112 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T10:03:26.112 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T10:03:26.112 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_25 2024-04-06T10:03:26.112 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T10:03:26.112 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T10:03:26.112 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T10:03:26.112 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T10:03:26.112 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T10:03:26.113 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T10:03:26.113 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T10:03:26.113 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T10:03:26.113 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T10:03:26.113 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T10:03:26.113 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T10:03:26.113 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:03:26.113 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T10:03:26.113 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T10:03:26.113 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T10:03:26.113 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T10:03:26.113 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T10:03:26.113 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T10:03:26.113 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T10:03:26.114 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T10:03:26.114 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T10:03:26.114 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T10:03:26.114 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:03:26.114 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T10:03:26.115 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T10:03:26.115 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T10:03:26.115 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T10:03:26.115 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T10:03:26.115 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T10:03:26.115 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T10:03:26.139 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T10:03:26.139 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T10:03:26.139 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T10:03:26.139 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_25 2024-04-06T10:03:26.243 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:Invalid command: unused arguments: ['mirror/image_25'] 2024-04-06T10:03:26.244 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:rbd mirror flush : flush rbd mirror 2024-04-06T10:03:26.244 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:admin_socket: invalid command 2024-04-06T10:03:26.246 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T10:03:26.246 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/image_25 2024-04-06T10:03:26.246 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:1 2024-04-06T10:03:26.246 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T10:03:26.246 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T10:03:26.246 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-06T10:03:26.246 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:1 2024-04-06T10:03:26.246 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T10:03:26.247 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T10:03:26.247 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T10:03:26.247 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=1 2024-04-06T10:03:26.247 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-06T10:03:26.247 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T10:03:26.247 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:03:26.247 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=1 2024-04-06T10:03:26.247 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=1 2024-04-06T10:03:26.248 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-06T10:03:26.248 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:1 2024-04-06T10:03:26.248 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T10:03:26.249 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T10:03:26.249 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-06T10:03:26.249 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:1 2024-04-06T10:03:26.249 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T10:03:26.249 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T10:03:26.249 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:03:26.249 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=1 2024-04-06T10:03:26.249 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-06T10:03:26.249 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T10:03:26.249 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T10:03:26.249 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=1 2024-04-06T10:03:26.249 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=1 2024-04-06T10:03:26.249 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-06T10:03:26.271 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T10:03:26.271 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T10:03:26.271 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T10:03:26.272 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/image_25 2024-04-06T10:03:26.407 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T10:03:26.408 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror image_25 2024-04-06T10:03:26.408 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:03:26.408 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:03:26.408 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_25 2024-04-06T10:03:26.408 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror image_25 '' 2024-04-06T10:03:26.408 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:03:26.408 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:03:26.408 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_25 2024-04-06T10:03:26.408 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T10:03:26.409 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-image_25.status 2024-04-06T10:03:26.409 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-image_25.status 2024-04-06T10:03:26.409 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T10:03:26.410 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-image_25.status 2024-04-06T10:03:26.411 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_25.status 2024-04-06T10:03:26.411 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_25 2024-04-06T10:03:26.457 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 1 2024-04-06T10:03:26.457 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 1 2024-04-06T10:03:26.457 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T10:03:26.457 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=5, tag_tid=3, entry_tid=1], [object_number=4, tag_tid=3, entry_tid=0], [object_number=6, tag_tid=1, entry_tid=7138], [object_number=7, tag_tid=1, entry_tid=7135]]], state=connected] 2024-04-06T10:03:26.457 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=5, tag_tid=2, entry_tid=1], [object_number=4, tag_tid=2, entry_tid=0], [object_number=6, tag_tid=1, entry_tid=7138], [object_number=7, tag_tid=1, entry_tid=7135]]], state=connected] 2024-04-06T10:03:26.460 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_25.status 2024-04-06T10:03:26.461 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=5, tag_tid=3, entry_tid=1' 2024-04-06T10:03:26.462 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror image_25 2024-04-06T10:03:26.462 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:03:26.463 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:03:26.463 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_25 2024-04-06T10:03:26.463 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror image_25 '..*' 2024-04-06T10:03:26.463 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:03:26.463 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:03:26.463 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_25 2024-04-06T10:03:26.463 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T10:03:26.463 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-image_25.status 2024-04-06T10:03:26.463 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-image_25.status 2024-04-06T10:03:26.463 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T10:03:26.464 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-image_25.status 2024-04-06T10:03:26.464 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_25 2024-04-06T10:03:26.464 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_25.status 2024-04-06T10:03:26.508 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 1 2024-04-06T10:03:26.508 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 1 2024-04-06T10:03:26.508 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T10:03:26.508 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=5, tag_tid=3, entry_tid=1], [object_number=4, tag_tid=3, entry_tid=0], [object_number=6, tag_tid=1, entry_tid=7138], [object_number=7, tag_tid=1, entry_tid=7135]]], state=connected] 2024-04-06T10:03:26.508 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=5, tag_tid=2, entry_tid=1], [object_number=4, tag_tid=2, entry_tid=0], [object_number=6, tag_tid=1, entry_tid=7138], [object_number=7, tag_tid=1, entry_tid=7135]]], state=connected] 2024-04-06T10:03:26.512 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_25.status 2024-04-06T10:03:26.514 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=5, tag_tid=2, entry_tid=1' 2024-04-06T10:03:26.514 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=5, tag_tid=3, entry_tid=1' -a 'object_number=5, tag_tid=3, entry_tid=1' = 'object_number=5, tag_tid=2, entry_tid=1' 2024-04-06T10:03:26.514 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=5, tag_tid=2, entry_tid=1' '!=' 'object_number=5, tag_tid=2, entry_tid=1' 2024-04-06T10:03:26.514 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T10:03:26.514 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 2 2024-04-06T10:03:28.516 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror image_25 2024-04-06T10:03:28.516 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T10:03:28.516 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:03:28.517 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_25 2024-04-06T10:03:28.517 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T10:03:28.517 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n image_25 ']' 2024-04-06T10:03:28.517 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/image_25' 2024-04-06T10:03:28.517 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_25 2024-04-06T10:03:28.517 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T10:03:28.517 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T10:03:28.517 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T10:03:28.517 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T10:03:28.517 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T10:03:28.517 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T10:03:28.517 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T10:03:28.517 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T10:03:28.519 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T10:03:28.519 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T10:03:28.520 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_25 2024-04-06T10:03:28.520 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T10:03:28.520 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T10:03:28.520 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T10:03:28.520 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T10:03:28.520 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T10:03:28.520 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T10:03:28.520 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T10:03:28.520 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T10:03:28.520 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T10:03:28.520 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T10:03:28.520 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T10:03:28.520 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:03:28.521 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T10:03:28.521 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T10:03:28.521 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T10:03:28.526 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T10:03:28.526 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T10:03:28.526 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T10:03:28.526 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T10:03:28.526 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T10:03:28.526 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T10:03:28.526 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T10:03:28.526 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:03:28.526 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T10:03:28.526 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T10:03:28.526 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T10:03:28.526 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T10:03:28.527 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T10:03:28.527 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T10:03:28.527 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T10:03:28.546 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T10:03:28.546 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T10:03:28.546 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T10:03:28.546 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_25 2024-04-06T10:03:28.651 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:Invalid command: unused arguments: ['mirror/image_25'] 2024-04-06T10:03:28.651 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:rbd mirror flush : flush rbd mirror 2024-04-06T10:03:28.651 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:admin_socket: invalid command 2024-04-06T10:03:28.653 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T10:03:28.653 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/image_25 2024-04-06T10:03:28.653 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:1 2024-04-06T10:03:28.653 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T10:03:28.653 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T10:03:28.653 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-06T10:03:28.653 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:1 2024-04-06T10:03:28.653 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T10:03:28.653 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T10:03:28.654 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T10:03:28.654 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=1 2024-04-06T10:03:28.654 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-06T10:03:28.654 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T10:03:28.654 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:03:28.654 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=1 2024-04-06T10:03:28.654 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=1 2024-04-06T10:03:28.655 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-06T10:03:28.655 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:1 2024-04-06T10:03:28.655 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T10:03:28.655 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T10:03:28.655 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-06T10:03:28.655 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:1 2024-04-06T10:03:28.655 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T10:03:28.655 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T10:03:28.655 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:03:28.655 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=1 2024-04-06T10:03:28.655 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-06T10:03:28.656 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T10:03:28.656 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T10:03:28.656 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=1 2024-04-06T10:03:28.656 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=1 2024-04-06T10:03:28.656 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-06T10:03:28.680 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T10:03:28.680 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T10:03:28.680 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T10:03:28.680 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/image_25 2024-04-06T10:03:28.830 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T10:03:28.832 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror image_25 2024-04-06T10:03:28.832 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:03:28.832 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:03:28.832 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_25 2024-04-06T10:03:28.832 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror image_25 '' 2024-04-06T10:03:28.832 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:03:28.832 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:03:28.832 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_25 2024-04-06T10:03:28.832 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T10:03:28.832 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-image_25.status 2024-04-06T10:03:28.833 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-image_25.status 2024-04-06T10:03:28.833 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T10:03:28.834 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-image_25.status 2024-04-06T10:03:28.834 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_25 2024-04-06T10:03:28.834 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_25.status 2024-04-06T10:03:28.879 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 1 2024-04-06T10:03:28.879 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 1 2024-04-06T10:03:28.879 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T10:03:28.880 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=5, tag_tid=3, entry_tid=1], [object_number=4, tag_tid=3, entry_tid=0], [object_number=6, tag_tid=1, entry_tid=7138], [object_number=7, tag_tid=1, entry_tid=7135]]], state=connected] 2024-04-06T10:03:28.880 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=5, tag_tid=3, entry_tid=1], [object_number=4, tag_tid=3, entry_tid=0], [object_number=6, tag_tid=1, entry_tid=7138], [object_number=7, tag_tid=1, entry_tid=7135]]], state=connected] 2024-04-06T10:03:28.883 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_25.status 2024-04-06T10:03:28.884 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=5, tag_tid=3, entry_tid=1' 2024-04-06T10:03:28.886 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror image_25 2024-04-06T10:03:28.886 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:03:28.886 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:03:28.886 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_25 2024-04-06T10:03:28.886 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror image_25 '..*' 2024-04-06T10:03:28.886 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:03:28.886 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:03:28.886 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_25 2024-04-06T10:03:28.886 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T10:03:28.886 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-image_25.status 2024-04-06T10:03:28.886 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-image_25.status 2024-04-06T10:03:28.887 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T10:03:28.887 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-image_25.status 2024-04-06T10:03:28.887 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_25 2024-04-06T10:03:28.887 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_25.status 2024-04-06T10:03:28.932 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 1 2024-04-06T10:03:28.932 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 1 2024-04-06T10:03:28.932 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T10:03:28.932 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=5, tag_tid=3, entry_tid=1], [object_number=4, tag_tid=3, entry_tid=0], [object_number=6, tag_tid=1, entry_tid=7138], [object_number=7, tag_tid=1, entry_tid=7135]]], state=connected] 2024-04-06T10:03:28.932 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=5, tag_tid=3, entry_tid=1], [object_number=4, tag_tid=3, entry_tid=0], [object_number=6, tag_tid=1, entry_tid=7138], [object_number=7, tag_tid=1, entry_tid=7135]]], state=connected] 2024-04-06T10:03:28.936 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_25.status 2024-04-06T10:03:28.938 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=5, tag_tid=3, entry_tid=1' 2024-04-06T10:03:28.938 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=5, tag_tid=3, entry_tid=1' -a 'object_number=5, tag_tid=3, entry_tid=1' = 'object_number=5, tag_tid=3, entry_tid=1' 2024-04-06T10:03:28.938 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T10:03:28.938 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_snap_present cluster1 mirror image_25 snap 2024-04-06T10:03:28.938 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T10:03:28.938 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:03:28.938 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_25 2024-04-06T10:03:28.938 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local snap_name=snap 2024-04-06T10:03:28.938 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local s 2024-04-06T10:03:28.938 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-04-06T10:03:28.938 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 1 2024-04-06T10:03:29.939 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 info mirror/image_25@snap 2024-04-06T10:03:30.001 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:rbd image 'image_25': 2024-04-06T10:03:30.001 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: size 128 MiB in 32 objects 2024-04-06T10:03:30.001 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: order 22 (4 MiB objects) 2024-04-06T10:03:30.002 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: snapshot_count: 1 2024-04-06T10:03:30.002 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: id: 10ac1c90ccd8 2024-04-06T10:03:30.002 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: block_name_prefix: rbd_data.10ac1c90ccd8 2024-04-06T10:03:30.002 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: format: 2 2024-04-06T10:03:30.002 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: features: layering, exclusive-lock, journaling 2024-04-06T10:03:30.002 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: op_features: 2024-04-06T10:03:30.002 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: flags: 2024-04-06T10:03:30.002 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: create_timestamp: Sat Apr 6 09:54:31 2024 2024-04-06T10:03:30.002 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: access_timestamp: Sat Apr 6 09:54:31 2024 2024-04-06T10:03:30.002 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: modify_timestamp: Sat Apr 6 09:55:31 2024 2024-04-06T10:03:30.002 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: protected: False 2024-04-06T10:03:30.002 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: journal: 10ac1c90ccd8 2024-04-06T10:03:30.003 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: mirroring state: enabled 2024-04-06T10:03:30.003 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: mirroring mode: journal 2024-04-06T10:03:30.003 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: mirroring global id: fa0d76c7-1a6b-4bb2-a9d2-ba2535b601e8 2024-04-06T10:03:30.003 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: mirroring primary: false 2024-04-06T10:03:30.006 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T10:03:30.006 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ compare_image_snaps mirror image_25 snap 2024-04-06T10:03:30.006 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:03:30.006 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_25 2024-04-06T10:03:30.006 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local snap_name=snap 2024-04-06T10:03:30.006 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local ret=0 2024-04-06T10:03:30.006 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local rmt_export=/tmp/tmp.BhW6xKmROU/cluster2-mirror-image_25.export 2024-04-06T10:03:30.006 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loc_export=/tmp/tmp.BhW6xKmROU/cluster1-mirror-image_25.export 2024-04-06T10:03:30.006 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rm -f /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_25.export /tmp/tmp.BhW6xKmROU/cluster1-mirror-image_25.export 2024-04-06T10:03:30.007 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster2 -p mirror export image_25@snap /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_25.export 2024-04-06T10:03:30.185 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-04-06T10:03:30.192 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 -p mirror export image_25@snap /tmp/tmp.BhW6xKmROU/cluster1-mirror-image_25.export 2024-04-06T10:03:30.577 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-04-06T10:03:30.588 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmp /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_25.export /tmp/tmp.BhW6xKmROU/cluster1-mirror-image_25.export 2024-04-06T10:03:30.668 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rm -f /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_25.export /tmp/tmp.BhW6xKmROU/cluster1-mirror-image_25.export 2024-04-06T10:03:30.709 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T10:03:30.709 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-06T10:03:30.709 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ image=image_26 2024-04-06T10:03:30.709 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ create_snap cluster2 mirror image_26 snap 2024-04-06T10:03:30.709 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-04-06T10:03:30.709 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:03:30.709 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_26 2024-04-06T10:03:30.710 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local snap_name=snap 2024-04-06T10:03:30.710 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster2 -p mirror snap create image_26@snap --debug-rbd=20 --debug-journaler=20 2024-04-06T10:03:31.903 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_image_replay_started cluster1 mirror image_26 2024-04-06T10:03:31.903 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T10:03:31.903 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:03:31.903 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_26 2024-04-06T10:03:31.903 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_image_replay_state cluster1 mirror image_26 started 2024-04-06T10:03:31.903 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T10:03:31.903 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:03:31.903 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_26 2024-04-06T10:03:31.903 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state=started 2024-04-06T10:03:31.903 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local s 2024-04-06T10:03:31.904 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T10:03:31.904 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 1 2024-04-06T10:03:32.904 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test_image_replay_state cluster1 mirror image_26 started 2024-04-06T10:03:32.904 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T10:03:32.905 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:03:32.905 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_26 2024-04-06T10:03:32.905 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local test_state=started 2024-04-06T10:03:32.905 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local status_result 2024-04-06T10:03:32.905 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local current_state=stopped 2024-04-06T10:03:32.906 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_26 2024-04-06T10:03:32.906 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_instance=cluster1 2024-04-06T10:03:32.906 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -i state 2024-04-06T10:03:32.906 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ shift 2024-04-06T10:03:32.906 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T10:03:32.906 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance=cluster1 2024-04-06T10:03:32.906 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local loop_instance 2024-04-06T10:03:32.906 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T10:03:32.906 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sleep 0 2024-04-06T10:03:32.907 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-04-06T10:03:32.908 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ seq 0 3 2024-04-06T10:03:32.911 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T10:03:32.911 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_26 2024-04-06T10:03:32.911 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1:0 2024-04-06T10:03:32.911 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ shift 2024-04-06T10:03:32.911 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T10:03:32.911 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-04-06T10:03:32.911 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T10:03:32.911 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=cluster 2024-04-06T10:03:32.911 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T10:03:32.911 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:03:32.911 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T10:03:32.911 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T10:03:32.911 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval cluster=cluster1 2024-04-06T10:03:32.912 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ cluster=cluster1 2024-04-06T10:03:32.912 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T10:03:32.912 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T10:03:32.912 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-04-06T10:03:32.912 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local local_cluster=cluster1:0 2024-04-06T10:03:32.912 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local cluster=cluster1 2024-04-06T10:03:32.912 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local instance 2024-04-06T10:03:32.912 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T10:03:32.912 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local val=cluster1:0 2024-04-06T10:03:32.913 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local cluster_var_name=local_cluster 2024-04-06T10:03:32.913 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local instance_var_name=instance 2024-04-06T10:03:32.913 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ cluster=cluster1 2024-04-06T10:03:32.913 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T10:03:32.913 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ '[' 0 = cluster1:0 ']' 2024-04-06T10:03:32.913 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ eval local_cluster=cluster1 2024-04-06T10:03:32.913 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++++ local_cluster=cluster1 2024-04-06T10:03:32.913 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ eval instance=0 2024-04-06T10:03:32.913 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++++ instance=0 2024-04-06T10:03:32.913 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T10:03:32.936 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T10:03:32.936 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T10:03:32.936 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T10:03:32.936 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_26 2024-04-06T10:03:33.041 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:Invalid command: unused arguments: ['mirror/image_26'] 2024-04-06T10:03:33.041 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:rbd mirror status : get status for rbd mirror 2024-04-06T10:03:33.041 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:admin_socket: invalid command 2024-04-06T10:03:33.044 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T10:03:33.044 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_26 2024-04-06T10:03:33.044 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1:1 2024-04-06T10:03:33.044 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ shift 2024-04-06T10:03:33.044 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T10:03:33.044 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-04-06T10:03:33.044 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:1 2024-04-06T10:03:33.044 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=cluster 2024-04-06T10:03:33.044 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T10:03:33.045 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:03:33.045 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=1 2024-04-06T10:03:33.045 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-06T10:03:33.045 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval cluster=cluster1 2024-04-06T10:03:33.045 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ cluster=cluster1 2024-04-06T10:03:33.045 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=1 2024-04-06T10:03:33.045 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=1 2024-04-06T10:03:33.045 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-04-06T10:03:33.045 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local local_cluster=cluster1:1 2024-04-06T10:03:33.045 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local cluster=cluster1 2024-04-06T10:03:33.045 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local instance 2024-04-06T10:03:33.045 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-06T10:03:33.045 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local val=cluster1:1 2024-04-06T10:03:33.045 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local cluster_var_name=local_cluster 2024-04-06T10:03:33.046 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local instance_var_name=instance 2024-04-06T10:03:33.046 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ cluster=cluster1 2024-04-06T10:03:33.046 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=1 2024-04-06T10:03:33.046 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ '[' 1 = cluster1:1 ']' 2024-04-06T10:03:33.046 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ eval local_cluster=cluster1 2024-04-06T10:03:33.046 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++++ local_cluster=cluster1 2024-04-06T10:03:33.046 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ eval instance=1 2024-04-06T10:03:33.046 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++++ instance=1 2024-04-06T10:03:33.046 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-06T10:03:33.067 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T10:03:33.068 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T10:03:33.068 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T10:03:33.068 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_26 2024-04-06T10:03:33.224 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ return 0 2024-04-06T10:03:33.225 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ status_result=' "state": "Replaying"' 2024-04-06T10:03:33.225 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ echo ' "state": "Replaying"' 2024-04-06T10:03:33.225 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep -i Replaying 2024-04-06T10:03:33.227 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: "state": "Replaying" 2024-04-06T10:03:33.228 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ current_state=started 2024-04-06T10:03:33.229 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test started = started 2024-04-06T10:03:33.229 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T10:03:33.229 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_26 2024-04-06T10:03:33.229 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local local_cluster=cluster1 2024-04-06T10:03:33.229 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-04-06T10:03:33.229 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:03:33.229 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_26 2024-04-06T10:03:33.229 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' journal = journal ']' 2024-04-06T10:03:33.229 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_journal_replay_complete cluster1 cluster2 mirror image_26 2024-04-06T10:03:33.229 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local local_cluster=cluster1 2024-04-06T10:03:33.229 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-04-06T10:03:33.229 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:03:33.229 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_26 2024-04-06T10:03:33.230 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local s master_pos mirror_pos last_mirror_pos 2024-04-06T10:03:33.230 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local master_tag master_entry mirror_tag mirror_entry 2024-04-06T10:03:33.230 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T10:03:33.230 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T10:03:33.230 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T10:03:33.430 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror image_26 2024-04-06T10:03:33.430 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T10:03:33.430 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:03:33.430 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_26 2024-04-06T10:03:33.430 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T10:03:33.430 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n image_26 ']' 2024-04-06T10:03:33.430 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/image_26' 2024-04-06T10:03:33.430 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_26 2024-04-06T10:03:33.430 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T10:03:33.431 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T10:03:33.431 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T10:03:33.431 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T10:03:33.431 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T10:03:33.431 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T10:03:33.431 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T10:03:33.431 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T10:03:33.431 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T10:03:33.435 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T10:03:33.436 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_26 2024-04-06T10:03:33.436 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T10:03:33.436 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T10:03:33.436 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T10:03:33.436 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T10:03:33.436 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T10:03:33.436 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T10:03:33.436 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T10:03:33.436 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T10:03:33.436 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T10:03:33.436 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T10:03:33.436 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T10:03:33.436 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:03:33.437 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T10:03:33.437 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T10:03:33.437 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T10:03:33.437 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T10:03:33.437 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T10:03:33.437 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T10:03:33.437 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T10:03:33.437 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T10:03:33.437 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T10:03:33.437 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T10:03:33.437 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:03:33.437 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T10:03:33.437 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T10:03:33.437 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T10:03:33.438 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T10:03:33.438 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T10:03:33.438 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T10:03:33.438 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T10:03:33.461 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T10:03:33.461 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T10:03:33.461 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T10:03:33.461 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_26 2024-04-06T10:03:33.563 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:Invalid command: unused arguments: ['mirror/image_26'] 2024-04-06T10:03:33.564 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:rbd mirror flush : flush rbd mirror 2024-04-06T10:03:33.564 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:admin_socket: invalid command 2024-04-06T10:03:33.567 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T10:03:33.567 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/image_26 2024-04-06T10:03:33.567 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:1 2024-04-06T10:03:33.567 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T10:03:33.568 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T10:03:33.568 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-06T10:03:33.568 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:1 2024-04-06T10:03:33.568 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T10:03:33.568 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T10:03:33.568 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T10:03:33.568 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=1 2024-04-06T10:03:33.568 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-06T10:03:33.568 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T10:03:33.568 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:03:33.568 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=1 2024-04-06T10:03:33.568 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=1 2024-04-06T10:03:33.568 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-06T10:03:33.568 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:1 2024-04-06T10:03:33.569 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T10:03:33.569 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T10:03:33.569 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-06T10:03:33.569 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:1 2024-04-06T10:03:33.569 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T10:03:33.569 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T10:03:33.569 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:03:33.569 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=1 2024-04-06T10:03:33.569 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-06T10:03:33.569 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T10:03:33.569 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T10:03:33.569 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=1 2024-04-06T10:03:33.569 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=1 2024-04-06T10:03:33.570 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-06T10:03:33.592 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T10:03:33.592 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T10:03:33.592 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T10:03:33.593 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/image_26 2024-04-06T10:03:33.753 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T10:03:33.754 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror image_26 2024-04-06T10:03:33.754 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:03:33.754 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:03:33.754 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_26 2024-04-06T10:03:33.754 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror image_26 '' 2024-04-06T10:03:33.755 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:03:33.755 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:03:33.755 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_26 2024-04-06T10:03:33.755 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T10:03:33.755 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-image_26.status 2024-04-06T10:03:33.755 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-image_26.status 2024-04-06T10:03:33.755 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T10:03:33.758 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-image_26.status 2024-04-06T10:03:33.758 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_26 2024-04-06T10:03:33.758 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_26.status 2024-04-06T10:03:33.808 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 1 2024-04-06T10:03:33.808 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 1 2024-04-06T10:03:33.808 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T10:03:33.808 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=5, tag_tid=3, entry_tid=1], [object_number=4, tag_tid=3, entry_tid=0], [object_number=7, tag_tid=1, entry_tid=6831], [object_number=6, tag_tid=1, entry_tid=6830]]], state=connected] 2024-04-06T10:03:33.808 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=5, tag_tid=1, entry_tid=6833], [object_number=4, tag_tid=1, entry_tid=6832], [object_number=7, tag_tid=1, entry_tid=6831], [object_number=6, tag_tid=1, entry_tid=6830]]], state=connected] 2024-04-06T10:03:33.812 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_26.status 2024-04-06T10:03:33.813 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=5, tag_tid=3, entry_tid=1' 2024-04-06T10:03:33.814 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror image_26 2024-04-06T10:03:33.815 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:03:33.815 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:03:33.815 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_26 2024-04-06T10:03:33.815 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror image_26 '..*' 2024-04-06T10:03:33.815 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:03:33.815 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:03:33.815 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_26 2024-04-06T10:03:33.815 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T10:03:33.815 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-image_26.status 2024-04-06T10:03:33.815 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-image_26.status 2024-04-06T10:03:33.815 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T10:03:33.816 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-image_26.status 2024-04-06T10:03:33.816 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_26 2024-04-06T10:03:33.816 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_26.status 2024-04-06T10:03:33.863 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 1 2024-04-06T10:03:33.863 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 1 2024-04-06T10:03:33.863 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T10:03:33.863 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=5, tag_tid=3, entry_tid=1], [object_number=4, tag_tid=3, entry_tid=0], [object_number=7, tag_tid=1, entry_tid=6831], [object_number=6, tag_tid=1, entry_tid=6830]]], state=connected] 2024-04-06T10:03:33.863 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=5, tag_tid=1, entry_tid=6833], [object_number=4, tag_tid=1, entry_tid=6832], [object_number=7, tag_tid=1, entry_tid=6831], [object_number=6, tag_tid=1, entry_tid=6830]]], state=connected] 2024-04-06T10:03:33.867 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_26.status 2024-04-06T10:03:33.868 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=5, tag_tid=1, entry_tid=6833' 2024-04-06T10:03:33.869 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=5, tag_tid=3, entry_tid=1' -a 'object_number=5, tag_tid=3, entry_tid=1' = 'object_number=5, tag_tid=1, entry_tid=6833' 2024-04-06T10:03:33.869 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=5, tag_tid=1, entry_tid=6833' '!=' '' 2024-04-06T10:03:33.869 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T10:03:33.869 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=5, tag_tid=1, entry_tid=6833' = '' 2024-04-06T10:03:33.869 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=5, tag_tid=1, entry_tid=6833' 2024-04-06T10:03:33.869 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=5, tag_tid=3, entry_tid=1' 2024-04-06T10:03:33.869 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T10:03:33.869 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T10:03:33.870 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=3 2024-04-06T10:03:33.870 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=5, tag_tid=1, entry_tid=6833' 2024-04-06T10:03:33.871 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T10:03:33.871 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T10:03:33.872 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=1 2024-04-06T10:03:33.872 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=5, tag_tid=3, entry_tid=1' 2024-04-06T10:03:33.872 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T10:03:33.872 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T10:03:33.873 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T10:03:33.874 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=5, tag_tid=1, entry_tid=6833' 2024-04-06T10:03:33.874 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T10:03:33.874 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T10:03:33.876 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=6833 2024-04-06T10:03:33.876 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 3 = 1 -a 1 -le 6833 2024-04-06T10:03:33.876 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T10:03:33.876 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T10:03:33.876 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T10:03:34.077 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror image_26 2024-04-06T10:03:34.077 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T10:03:34.077 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:03:34.078 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_26 2024-04-06T10:03:34.078 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T10:03:34.078 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n image_26 ']' 2024-04-06T10:03:34.078 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/image_26' 2024-04-06T10:03:34.078 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_26 2024-04-06T10:03:34.078 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T10:03:34.078 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T10:03:34.078 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T10:03:34.078 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T10:03:34.078 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T10:03:34.078 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T10:03:34.078 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T10:03:34.079 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T10:03:34.079 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T10:03:34.083 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T10:03:34.083 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_26 2024-04-06T10:03:34.083 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T10:03:34.083 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T10:03:34.083 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T10:03:34.083 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T10:03:34.083 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T10:03:34.083 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T10:03:34.083 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T10:03:34.083 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T10:03:34.083 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T10:03:34.083 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T10:03:34.084 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T10:03:34.084 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:03:34.084 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T10:03:34.084 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T10:03:34.084 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T10:03:34.084 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T10:03:34.085 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T10:03:34.085 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T10:03:34.085 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T10:03:34.085 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T10:03:34.085 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T10:03:34.085 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T10:03:34.085 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:03:34.085 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T10:03:34.085 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T10:03:34.085 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T10:03:34.085 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T10:03:34.085 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T10:03:34.085 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T10:03:34.086 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T10:03:34.107 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T10:03:34.108 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T10:03:34.108 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T10:03:34.108 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_26 2024-04-06T10:03:34.218 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:Invalid command: unused arguments: ['mirror/image_26'] 2024-04-06T10:03:34.219 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:rbd mirror flush : flush rbd mirror 2024-04-06T10:03:34.219 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:admin_socket: invalid command 2024-04-06T10:03:34.221 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T10:03:34.221 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/image_26 2024-04-06T10:03:34.221 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:1 2024-04-06T10:03:34.222 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T10:03:34.222 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T10:03:34.222 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-06T10:03:34.222 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:1 2024-04-06T10:03:34.222 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T10:03:34.222 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T10:03:34.222 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T10:03:34.222 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=1 2024-04-06T10:03:34.222 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-06T10:03:34.222 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T10:03:34.222 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:03:34.222 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=1 2024-04-06T10:03:34.222 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=1 2024-04-06T10:03:34.222 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-06T10:03:34.222 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:1 2024-04-06T10:03:34.223 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T10:03:34.223 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T10:03:34.223 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-06T10:03:34.223 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:1 2024-04-06T10:03:34.223 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T10:03:34.223 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T10:03:34.223 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:03:34.223 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=1 2024-04-06T10:03:34.223 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-06T10:03:34.223 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T10:03:34.223 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T10:03:34.223 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=1 2024-04-06T10:03:34.223 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=1 2024-04-06T10:03:34.224 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-06T10:03:34.247 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T10:03:34.247 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T10:03:34.247 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T10:03:34.247 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/image_26 2024-04-06T10:03:34.397 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T10:03:34.399 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror image_26 2024-04-06T10:03:34.399 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:03:34.399 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:03:34.399 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_26 2024-04-06T10:03:34.399 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror image_26 '' 2024-04-06T10:03:34.400 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:03:34.400 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:03:34.400 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_26 2024-04-06T10:03:34.400 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T10:03:34.400 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-image_26.status 2024-04-06T10:03:34.400 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T10:03:34.400 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-image_26.status 2024-04-06T10:03:34.401 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-image_26.status 2024-04-06T10:03:34.402 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_26.status 2024-04-06T10:03:34.402 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_26 2024-04-06T10:03:34.452 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 1 2024-04-06T10:03:34.453 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 1 2024-04-06T10:03:34.453 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T10:03:34.453 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=5, tag_tid=3, entry_tid=1], [object_number=4, tag_tid=3, entry_tid=0], [object_number=7, tag_tid=1, entry_tid=6831], [object_number=6, tag_tid=1, entry_tid=6830]]], state=connected] 2024-04-06T10:03:34.453 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=5, tag_tid=3, entry_tid=1], [object_number=4, tag_tid=3, entry_tid=0], [object_number=7, tag_tid=1, entry_tid=6831], [object_number=6, tag_tid=1, entry_tid=6830]]], state=connected] 2024-04-06T10:03:34.456 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_26.status 2024-04-06T10:03:34.457 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=5, tag_tid=3, entry_tid=1' 2024-04-06T10:03:34.459 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror image_26 2024-04-06T10:03:34.459 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:03:34.459 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:03:34.459 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_26 2024-04-06T10:03:34.459 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror image_26 '..*' 2024-04-06T10:03:34.459 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:03:34.459 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:03:34.459 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_26 2024-04-06T10:03:34.459 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T10:03:34.459 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-image_26.status 2024-04-06T10:03:34.459 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-image_26.status 2024-04-06T10:03:34.461 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T10:03:34.462 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-image_26.status 2024-04-06T10:03:34.462 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_26 2024-04-06T10:03:34.462 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_26.status 2024-04-06T10:03:34.505 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 1 2024-04-06T10:03:34.505 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 1 2024-04-06T10:03:34.505 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T10:03:34.506 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=5, tag_tid=3, entry_tid=1], [object_number=4, tag_tid=3, entry_tid=0], [object_number=7, tag_tid=1, entry_tid=6831], [object_number=6, tag_tid=1, entry_tid=6830]]], state=connected] 2024-04-06T10:03:34.506 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=5, tag_tid=3, entry_tid=1], [object_number=4, tag_tid=3, entry_tid=0], [object_number=7, tag_tid=1, entry_tid=6831], [object_number=6, tag_tid=1, entry_tid=6830]]], state=connected] 2024-04-06T10:03:34.509 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_26.status 2024-04-06T10:03:34.511 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=5, tag_tid=3, entry_tid=1' 2024-04-06T10:03:34.511 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=5, tag_tid=3, entry_tid=1' -a 'object_number=5, tag_tid=3, entry_tid=1' = 'object_number=5, tag_tid=3, entry_tid=1' 2024-04-06T10:03:34.511 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T10:03:34.511 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_snap_present cluster1 mirror image_26 snap 2024-04-06T10:03:34.511 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T10:03:34.511 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:03:34.511 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_26 2024-04-06T10:03:34.511 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local snap_name=snap 2024-04-06T10:03:34.511 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local s 2024-04-06T10:03:34.511 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-04-06T10:03:34.512 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 1 2024-04-06T10:03:35.512 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 info mirror/image_26@snap 2024-04-06T10:03:35.579 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:rbd image 'image_26': 2024-04-06T10:03:35.579 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: size 128 MiB in 32 objects 2024-04-06T10:03:35.579 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: order 22 (4 MiB objects) 2024-04-06T10:03:35.579 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: snapshot_count: 1 2024-04-06T10:03:35.579 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: id: 10acd35197a4 2024-04-06T10:03:35.579 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: block_name_prefix: rbd_data.10acd35197a4 2024-04-06T10:03:35.580 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: format: 2 2024-04-06T10:03:35.580 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: features: layering, exclusive-lock, journaling 2024-04-06T10:03:35.580 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: op_features: 2024-04-06T10:03:35.580 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: flags: 2024-04-06T10:03:35.580 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: create_timestamp: Sat Apr 6 09:54:40 2024 2024-04-06T10:03:35.580 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: access_timestamp: Sat Apr 6 09:54:40 2024 2024-04-06T10:03:35.580 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: modify_timestamp: Sat Apr 6 09:55:46 2024 2024-04-06T10:03:35.580 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: protected: False 2024-04-06T10:03:35.580 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: journal: 10acd35197a4 2024-04-06T10:03:35.580 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: mirroring state: enabled 2024-04-06T10:03:35.580 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: mirroring mode: journal 2024-04-06T10:03:35.580 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: mirroring global id: 9c2d155e-fde6-4a67-8628-5095a5923fed 2024-04-06T10:03:35.580 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: mirroring primary: false 2024-04-06T10:03:35.584 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T10:03:35.584 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ compare_image_snaps mirror image_26 snap 2024-04-06T10:03:35.584 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:03:35.585 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_26 2024-04-06T10:03:35.585 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local snap_name=snap 2024-04-06T10:03:35.585 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local ret=0 2024-04-06T10:03:35.585 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local rmt_export=/tmp/tmp.BhW6xKmROU/cluster2-mirror-image_26.export 2024-04-06T10:03:35.585 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loc_export=/tmp/tmp.BhW6xKmROU/cluster1-mirror-image_26.export 2024-04-06T10:03:35.585 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rm -f /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_26.export /tmp/tmp.BhW6xKmROU/cluster1-mirror-image_26.export 2024-04-06T10:03:35.585 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster2 -p mirror export image_26@snap /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_26.export 2024-04-06T10:03:35.981 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-04-06T10:03:35.989 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 -p mirror export image_26@snap /tmp/tmp.BhW6xKmROU/cluster1-mirror-image_26.export 2024-04-06T10:03:36.311 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-04-06T10:03:36.323 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmp /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_26.export /tmp/tmp.BhW6xKmROU/cluster1-mirror-image_26.export 2024-04-06T10:03:36.402 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rm -f /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_26.export /tmp/tmp.BhW6xKmROU/cluster1-mirror-image_26.export 2024-04-06T10:03:36.444 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T10:03:36.444 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-06T10:03:36.444 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ image=image_27 2024-04-06T10:03:36.444 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ create_snap cluster2 mirror image_27 snap 2024-04-06T10:03:36.444 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-04-06T10:03:36.444 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:03:36.444 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_27 2024-04-06T10:03:36.444 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local snap_name=snap 2024-04-06T10:03:36.444 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster2 -p mirror snap create image_27@snap --debug-rbd=20 --debug-journaler=20 2024-04-06T10:03:36.914 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_image_replay_started cluster1 mirror image_27 2024-04-06T10:03:36.914 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T10:03:36.914 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:03:36.914 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_27 2024-04-06T10:03:36.914 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_image_replay_state cluster1 mirror image_27 started 2024-04-06T10:03:36.914 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T10:03:36.914 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:03:36.914 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_27 2024-04-06T10:03:36.915 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state=started 2024-04-06T10:03:36.915 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local s 2024-04-06T10:03:36.915 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T10:03:36.915 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 1 2024-04-06T10:03:37.915 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test_image_replay_state cluster1 mirror image_27 started 2024-04-06T10:03:37.915 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T10:03:37.915 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:03:37.915 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_27 2024-04-06T10:03:37.916 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local test_state=started 2024-04-06T10:03:37.916 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local status_result 2024-04-06T10:03:37.916 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local current_state=stopped 2024-04-06T10:03:37.917 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_27 2024-04-06T10:03:37.917 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_instance=cluster1 2024-04-06T10:03:37.917 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ shift 2024-04-06T10:03:37.917 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T10:03:37.917 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance=cluster1 2024-04-06T10:03:37.917 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local loop_instance 2024-04-06T10:03:37.917 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T10:03:37.917 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sleep 0 2024-04-06T10:03:37.917 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -i state 2024-04-06T10:03:37.918 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-04-06T10:03:37.918 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ seq 0 3 2024-04-06T10:03:37.920 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T10:03:37.920 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_27 2024-04-06T10:03:37.920 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1:0 2024-04-06T10:03:37.920 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ shift 2024-04-06T10:03:37.920 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T10:03:37.920 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-04-06T10:03:37.920 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T10:03:37.920 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=cluster 2024-04-06T10:03:37.920 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T10:03:37.920 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:03:37.920 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T10:03:37.920 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T10:03:37.921 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval cluster=cluster1 2024-04-06T10:03:37.921 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ cluster=cluster1 2024-04-06T10:03:37.921 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T10:03:37.921 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T10:03:37.922 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-04-06T10:03:37.922 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local local_cluster=cluster1:0 2024-04-06T10:03:37.922 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local cluster=cluster1 2024-04-06T10:03:37.922 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local instance 2024-04-06T10:03:37.923 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T10:03:37.923 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local val=cluster1:0 2024-04-06T10:03:37.923 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local cluster_var_name=local_cluster 2024-04-06T10:03:37.923 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local instance_var_name=instance 2024-04-06T10:03:37.923 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ cluster=cluster1 2024-04-06T10:03:37.923 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T10:03:37.923 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ '[' 0 = cluster1:0 ']' 2024-04-06T10:03:37.923 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ eval local_cluster=cluster1 2024-04-06T10:03:37.923 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++++ local_cluster=cluster1 2024-04-06T10:03:37.923 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ eval instance=0 2024-04-06T10:03:37.923 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++++ instance=0 2024-04-06T10:03:37.923 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T10:03:37.947 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T10:03:37.947 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T10:03:37.947 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T10:03:37.947 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_27 2024-04-06T10:03:38.051 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:Invalid command: unused arguments: ['mirror/image_27'] 2024-04-06T10:03:38.051 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:rbd mirror status : get status for rbd mirror 2024-04-06T10:03:38.051 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:admin_socket: invalid command 2024-04-06T10:03:38.053 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T10:03:38.053 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_27 2024-04-06T10:03:38.053 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1:1 2024-04-06T10:03:38.054 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ shift 2024-04-06T10:03:38.054 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T10:03:38.054 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-04-06T10:03:38.054 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:1 2024-04-06T10:03:38.054 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=cluster 2024-04-06T10:03:38.054 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T10:03:38.054 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:03:38.054 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=1 2024-04-06T10:03:38.054 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-06T10:03:38.054 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval cluster=cluster1 2024-04-06T10:03:38.054 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ cluster=cluster1 2024-04-06T10:03:38.054 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=1 2024-04-06T10:03:38.054 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=1 2024-04-06T10:03:38.054 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-04-06T10:03:38.055 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local local_cluster=cluster1:1 2024-04-06T10:03:38.055 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local cluster=cluster1 2024-04-06T10:03:38.055 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local instance 2024-04-06T10:03:38.055 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-06T10:03:38.055 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local val=cluster1:1 2024-04-06T10:03:38.055 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local cluster_var_name=local_cluster 2024-04-06T10:03:38.055 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local instance_var_name=instance 2024-04-06T10:03:38.055 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ cluster=cluster1 2024-04-06T10:03:38.055 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=1 2024-04-06T10:03:38.055 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ '[' 1 = cluster1:1 ']' 2024-04-06T10:03:38.055 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ eval local_cluster=cluster1 2024-04-06T10:03:38.055 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++++ local_cluster=cluster1 2024-04-06T10:03:38.055 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ eval instance=1 2024-04-06T10:03:38.055 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++++ instance=1 2024-04-06T10:03:38.056 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-06T10:03:38.073 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T10:03:38.074 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T10:03:38.074 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T10:03:38.074 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_27 2024-04-06T10:03:38.225 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ return 0 2024-04-06T10:03:38.225 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ status_result=' "state": "Replaying"' 2024-04-06T10:03:38.226 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ echo ' "state": "Replaying"' 2024-04-06T10:03:38.226 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep -i Replaying 2024-04-06T10:03:38.227 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: "state": "Replaying" 2024-04-06T10:03:38.229 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ current_state=started 2024-04-06T10:03:38.229 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test started = started 2024-04-06T10:03:38.229 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T10:03:38.230 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_27 2024-04-06T10:03:38.230 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local local_cluster=cluster1 2024-04-06T10:03:38.230 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-04-06T10:03:38.230 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:03:38.230 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_27 2024-04-06T10:03:38.230 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' journal = journal ']' 2024-04-06T10:03:38.230 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_journal_replay_complete cluster1 cluster2 mirror image_27 2024-04-06T10:03:38.230 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local local_cluster=cluster1 2024-04-06T10:03:38.230 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-04-06T10:03:38.230 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:03:38.230 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_27 2024-04-06T10:03:38.230 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local s master_pos mirror_pos last_mirror_pos 2024-04-06T10:03:38.230 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local master_tag master_entry mirror_tag mirror_entry 2024-04-06T10:03:38.231 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T10:03:38.231 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T10:03:38.231 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T10:03:38.430 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror image_27 2024-04-06T10:03:38.430 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T10:03:38.430 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:03:38.430 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_27 2024-04-06T10:03:38.431 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T10:03:38.431 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n image_27 ']' 2024-04-06T10:03:38.431 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/image_27' 2024-04-06T10:03:38.431 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_27 2024-04-06T10:03:38.431 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T10:03:38.431 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T10:03:38.431 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T10:03:38.431 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T10:03:38.431 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T10:03:38.431 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T10:03:38.431 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T10:03:38.432 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T10:03:38.432 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T10:03:38.435 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T10:03:38.435 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_27 2024-04-06T10:03:38.435 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T10:03:38.435 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T10:03:38.435 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T10:03:38.435 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T10:03:38.435 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T10:03:38.436 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T10:03:38.436 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T10:03:38.436 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T10:03:38.436 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T10:03:38.436 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T10:03:38.436 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T10:03:38.436 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:03:38.436 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T10:03:38.436 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T10:03:38.436 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T10:03:38.436 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T10:03:38.436 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T10:03:38.436 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T10:03:38.436 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T10:03:38.436 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T10:03:38.436 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T10:03:38.437 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T10:03:38.437 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:03:38.437 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T10:03:38.437 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T10:03:38.437 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T10:03:38.437 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T10:03:38.437 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T10:03:38.437 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T10:03:38.437 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T10:03:38.462 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T10:03:38.462 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T10:03:38.462 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T10:03:38.462 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_27 2024-04-06T10:03:38.569 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:Invalid command: unused arguments: ['mirror/image_27'] 2024-04-06T10:03:38.569 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:rbd mirror flush : flush rbd mirror 2024-04-06T10:03:38.569 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:admin_socket: invalid command 2024-04-06T10:03:38.572 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T10:03:38.573 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/image_27 2024-04-06T10:03:38.573 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:1 2024-04-06T10:03:38.573 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T10:03:38.573 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T10:03:38.573 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-06T10:03:38.573 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:1 2024-04-06T10:03:38.573 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T10:03:38.573 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T10:03:38.573 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T10:03:38.573 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=1 2024-04-06T10:03:38.573 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-06T10:03:38.573 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T10:03:38.573 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:03:38.573 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=1 2024-04-06T10:03:38.574 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=1 2024-04-06T10:03:38.574 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-06T10:03:38.574 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:1 2024-04-06T10:03:38.574 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T10:03:38.574 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T10:03:38.574 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-06T10:03:38.574 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:1 2024-04-06T10:03:38.574 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T10:03:38.574 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T10:03:38.574 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:03:38.574 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=1 2024-04-06T10:03:38.574 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-06T10:03:38.574 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T10:03:38.575 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T10:03:38.575 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=1 2024-04-06T10:03:38.575 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=1 2024-04-06T10:03:38.575 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-06T10:03:38.595 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T10:03:38.596 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T10:03:38.596 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T10:03:38.596 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/image_27 2024-04-06T10:03:38.734 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T10:03:38.735 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror image_27 2024-04-06T10:03:38.735 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:03:38.736 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:03:38.736 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_27 2024-04-06T10:03:38.736 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror image_27 '' 2024-04-06T10:03:38.736 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:03:38.736 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:03:38.736 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_27 2024-04-06T10:03:38.736 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T10:03:38.736 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-image_27.status 2024-04-06T10:03:38.737 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-image_27.status 2024-04-06T10:03:38.737 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T10:03:38.738 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-image_27.status 2024-04-06T10:03:38.739 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_27 2024-04-06T10:03:38.739 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_27.status 2024-04-06T10:03:38.784 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 0 2024-04-06T10:03:38.784 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 0 2024-04-06T10:03:38.784 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T10:03:38.784 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=1, tag_tid=3, entry_tid=1], [object_number=0, tag_tid=3, entry_tid=0], [object_number=2, tag_tid=1, entry_tid=3702], [object_number=3, tag_tid=1, entry_tid=3699]]], state=connected] 2024-04-06T10:03:38.784 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=1, tag_tid=2, entry_tid=1], [object_number=0, tag_tid=2, entry_tid=0], [object_number=2, tag_tid=1, entry_tid=3702], [object_number=3, tag_tid=1, entry_tid=3699]]], state=connected] 2024-04-06T10:03:38.788 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_27.status 2024-04-06T10:03:38.789 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=1, tag_tid=3, entry_tid=1' 2024-04-06T10:03:38.790 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror image_27 2024-04-06T10:03:38.790 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:03:38.790 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:03:38.790 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_27 2024-04-06T10:03:38.791 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror image_27 '..*' 2024-04-06T10:03:38.791 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:03:38.791 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:03:38.791 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_27 2024-04-06T10:03:38.791 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T10:03:38.791 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-image_27.status 2024-04-06T10:03:38.791 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-image_27.status 2024-04-06T10:03:38.791 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T10:03:38.791 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-image_27.status 2024-04-06T10:03:38.792 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_27 2024-04-06T10:03:38.792 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_27.status 2024-04-06T10:03:38.837 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 0 2024-04-06T10:03:38.837 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 0 2024-04-06T10:03:38.837 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T10:03:38.837 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=1, tag_tid=3, entry_tid=1], [object_number=0, tag_tid=3, entry_tid=0], [object_number=2, tag_tid=1, entry_tid=3702], [object_number=3, tag_tid=1, entry_tid=3699]]], state=connected] 2024-04-06T10:03:38.837 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=1, tag_tid=2, entry_tid=1], [object_number=0, tag_tid=2, entry_tid=0], [object_number=2, tag_tid=1, entry_tid=3702], [object_number=3, tag_tid=1, entry_tid=3699]]], state=connected] 2024-04-06T10:03:38.841 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_27.status 2024-04-06T10:03:38.843 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=1, tag_tid=2, entry_tid=1' 2024-04-06T10:03:38.843 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=1, tag_tid=3, entry_tid=1' -a 'object_number=1, tag_tid=3, entry_tid=1' = 'object_number=1, tag_tid=2, entry_tid=1' 2024-04-06T10:03:38.843 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=1, tag_tid=2, entry_tid=1' '!=' '' 2024-04-06T10:03:38.843 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T10:03:38.843 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=1, tag_tid=2, entry_tid=1' = '' 2024-04-06T10:03:38.843 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=1, tag_tid=2, entry_tid=1' 2024-04-06T10:03:38.844 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=1, tag_tid=3, entry_tid=1' 2024-04-06T10:03:38.844 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T10:03:38.844 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T10:03:38.844 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=3 2024-04-06T10:03:38.845 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=1, tag_tid=2, entry_tid=1' 2024-04-06T10:03:38.845 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T10:03:38.845 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T10:03:38.846 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=2 2024-04-06T10:03:38.847 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=1, tag_tid=3, entry_tid=1' 2024-04-06T10:03:38.847 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T10:03:38.847 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T10:03:38.848 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T10:03:38.848 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=1, tag_tid=2, entry_tid=1' 2024-04-06T10:03:38.848 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T10:03:38.849 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T10:03:38.850 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=1 2024-04-06T10:03:38.850 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 3 = 2 -a 1 -le 1 2024-04-06T10:03:38.850 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T10:03:38.850 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T10:03:38.850 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T10:03:39.052 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror image_27 2024-04-06T10:03:39.052 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T10:03:39.052 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:03:39.052 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_27 2024-04-06T10:03:39.052 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T10:03:39.052 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n image_27 ']' 2024-04-06T10:03:39.052 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/image_27' 2024-04-06T10:03:39.052 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_27 2024-04-06T10:03:39.053 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T10:03:39.053 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T10:03:39.053 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T10:03:39.053 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T10:03:39.053 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T10:03:39.053 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T10:03:39.053 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T10:03:39.053 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T10:03:39.054 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T10:03:39.056 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T10:03:39.056 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_27 2024-04-06T10:03:39.056 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T10:03:39.056 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T10:03:39.056 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T10:03:39.056 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T10:03:39.056 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T10:03:39.056 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T10:03:39.056 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T10:03:39.057 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T10:03:39.057 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T10:03:39.057 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T10:03:39.057 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T10:03:39.057 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:03:39.057 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T10:03:39.057 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T10:03:39.057 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T10:03:39.057 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T10:03:39.057 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T10:03:39.057 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T10:03:39.057 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T10:03:39.057 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T10:03:39.057 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T10:03:39.057 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T10:03:39.058 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:03:39.058 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T10:03:39.058 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T10:03:39.058 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T10:03:39.058 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T10:03:39.058 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T10:03:39.058 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T10:03:39.058 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T10:03:39.081 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T10:03:39.082 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T10:03:39.082 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T10:03:39.082 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_27 2024-04-06T10:03:39.189 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:Invalid command: unused arguments: ['mirror/image_27'] 2024-04-06T10:03:39.189 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:rbd mirror flush : flush rbd mirror 2024-04-06T10:03:39.189 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:admin_socket: invalid command 2024-04-06T10:03:39.193 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T10:03:39.193 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/image_27 2024-04-06T10:03:39.193 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:1 2024-04-06T10:03:39.193 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T10:03:39.194 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T10:03:39.194 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-06T10:03:39.194 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:1 2024-04-06T10:03:39.194 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T10:03:39.194 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T10:03:39.194 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T10:03:39.194 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=1 2024-04-06T10:03:39.194 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-06T10:03:39.194 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T10:03:39.194 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:03:39.194 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=1 2024-04-06T10:03:39.194 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=1 2024-04-06T10:03:39.195 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-06T10:03:39.195 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:1 2024-04-06T10:03:39.195 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T10:03:39.195 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T10:03:39.195 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-06T10:03:39.195 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:1 2024-04-06T10:03:39.195 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T10:03:39.195 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T10:03:39.195 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:03:39.195 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=1 2024-04-06T10:03:39.195 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-06T10:03:39.195 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T10:03:39.195 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T10:03:39.196 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=1 2024-04-06T10:03:39.196 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=1 2024-04-06T10:03:39.196 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-06T10:03:39.216 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T10:03:39.216 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T10:03:39.216 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T10:03:39.216 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/image_27 2024-04-06T10:03:39.366 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T10:03:39.367 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror image_27 2024-04-06T10:03:39.367 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:03:39.367 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:03:39.367 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_27 2024-04-06T10:03:39.367 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror image_27 '' 2024-04-06T10:03:39.367 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:03:39.367 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:03:39.368 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_27 2024-04-06T10:03:39.368 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T10:03:39.368 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-image_27.status 2024-04-06T10:03:39.368 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-image_27.status 2024-04-06T10:03:39.369 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T10:03:39.370 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-image_27.status 2024-04-06T10:03:39.371 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_27 2024-04-06T10:03:39.371 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_27.status 2024-04-06T10:03:39.416 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 0 2024-04-06T10:03:39.416 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 0 2024-04-06T10:03:39.417 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T10:03:39.417 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=1, tag_tid=3, entry_tid=1], [object_number=0, tag_tid=3, entry_tid=0], [object_number=2, tag_tid=1, entry_tid=3702], [object_number=3, tag_tid=1, entry_tid=3699]]], state=connected] 2024-04-06T10:03:39.417 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=1, tag_tid=2, entry_tid=1], [object_number=0, tag_tid=2, entry_tid=0], [object_number=2, tag_tid=1, entry_tid=3702], [object_number=3, tag_tid=1, entry_tid=3699]]], state=connected] 2024-04-06T10:03:39.420 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_27.status 2024-04-06T10:03:39.421 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=1, tag_tid=3, entry_tid=1' 2024-04-06T10:03:39.422 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror image_27 2024-04-06T10:03:39.423 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:03:39.423 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:03:39.423 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_27 2024-04-06T10:03:39.423 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror image_27 '..*' 2024-04-06T10:03:39.423 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:03:39.423 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:03:39.423 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_27 2024-04-06T10:03:39.423 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T10:03:39.423 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-image_27.status 2024-04-06T10:03:39.423 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-image_27.status 2024-04-06T10:03:39.423 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T10:03:39.424 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-image_27.status 2024-04-06T10:03:39.424 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_27 2024-04-06T10:03:39.424 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_27.status 2024-04-06T10:03:39.469 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 0 2024-04-06T10:03:39.469 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 0 2024-04-06T10:03:39.469 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T10:03:39.470 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=1, tag_tid=3, entry_tid=1], [object_number=0, tag_tid=3, entry_tid=0], [object_number=2, tag_tid=1, entry_tid=3702], [object_number=3, tag_tid=1, entry_tid=3699]]], state=connected] 2024-04-06T10:03:39.470 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=1, tag_tid=2, entry_tid=1], [object_number=0, tag_tid=2, entry_tid=0], [object_number=2, tag_tid=1, entry_tid=3702], [object_number=3, tag_tid=1, entry_tid=3699]]], state=connected] 2024-04-06T10:03:39.472 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_27.status 2024-04-06T10:03:39.474 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=1, tag_tid=2, entry_tid=1' 2024-04-06T10:03:39.474 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=1, tag_tid=3, entry_tid=1' -a 'object_number=1, tag_tid=3, entry_tid=1' = 'object_number=1, tag_tid=2, entry_tid=1' 2024-04-06T10:03:39.474 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=1, tag_tid=2, entry_tid=1' '!=' 'object_number=1, tag_tid=2, entry_tid=1' 2024-04-06T10:03:39.474 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T10:03:39.475 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.4 2024-04-06T10:03:39.876 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror image_27 2024-04-06T10:03:39.876 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T10:03:39.876 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:03:39.876 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_27 2024-04-06T10:03:39.877 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T10:03:39.877 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n image_27 ']' 2024-04-06T10:03:39.877 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/image_27' 2024-04-06T10:03:39.877 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_27 2024-04-06T10:03:39.877 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T10:03:39.877 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T10:03:39.877 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T10:03:39.877 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T10:03:39.877 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T10:03:39.877 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T10:03:39.877 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T10:03:39.878 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T10:03:39.878 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T10:03:39.880 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T10:03:39.880 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_27 2024-04-06T10:03:39.880 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T10:03:39.880 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T10:03:39.880 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T10:03:39.880 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T10:03:39.881 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T10:03:39.881 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T10:03:39.881 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T10:03:39.881 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T10:03:39.881 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T10:03:39.881 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T10:03:39.881 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T10:03:39.881 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:03:39.881 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T10:03:39.881 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T10:03:39.883 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T10:03:39.883 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T10:03:39.883 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T10:03:39.883 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T10:03:39.883 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T10:03:39.883 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T10:03:39.883 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T10:03:39.883 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T10:03:39.883 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:03:39.883 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T10:03:39.883 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T10:03:39.883 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T10:03:39.883 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T10:03:39.883 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T10:03:39.884 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T10:03:39.884 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T10:03:39.907 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T10:03:39.907 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T10:03:39.908 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T10:03:39.908 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_27 2024-04-06T10:03:40.014 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:Invalid command: unused arguments: ['mirror/image_27'] 2024-04-06T10:03:40.015 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:rbd mirror flush : flush rbd mirror 2024-04-06T10:03:40.015 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:admin_socket: invalid command 2024-04-06T10:03:40.017 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T10:03:40.017 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/image_27 2024-04-06T10:03:40.017 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:1 2024-04-06T10:03:40.017 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T10:03:40.017 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T10:03:40.017 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-06T10:03:40.017 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:1 2024-04-06T10:03:40.017 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T10:03:40.017 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T10:03:40.018 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T10:03:40.018 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=1 2024-04-06T10:03:40.018 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-06T10:03:40.018 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T10:03:40.018 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:03:40.018 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=1 2024-04-06T10:03:40.018 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=1 2024-04-06T10:03:40.019 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-06T10:03:40.020 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:1 2024-04-06T10:03:40.020 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T10:03:40.020 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T10:03:40.020 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-06T10:03:40.020 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:1 2024-04-06T10:03:40.020 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T10:03:40.020 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T10:03:40.020 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:03:40.020 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=1 2024-04-06T10:03:40.020 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-06T10:03:40.020 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T10:03:40.020 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T10:03:40.021 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=1 2024-04-06T10:03:40.021 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=1 2024-04-06T10:03:40.021 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-06T10:03:40.044 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T10:03:40.045 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T10:03:40.045 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T10:03:40.045 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/image_27 2024-04-06T10:03:40.182 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T10:03:40.183 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror image_27 2024-04-06T10:03:40.183 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:03:40.184 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:03:40.184 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_27 2024-04-06T10:03:40.184 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror image_27 '' 2024-04-06T10:03:40.184 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:03:40.184 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:03:40.184 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_27 2024-04-06T10:03:40.184 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T10:03:40.185 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-image_27.status 2024-04-06T10:03:40.185 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-image_27.status 2024-04-06T10:03:40.185 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T10:03:40.186 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-image_27.status 2024-04-06T10:03:40.187 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_27 2024-04-06T10:03:40.187 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_27.status 2024-04-06T10:03:40.233 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 0 2024-04-06T10:03:40.233 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 0 2024-04-06T10:03:40.233 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T10:03:40.233 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=1, tag_tid=3, entry_tid=1], [object_number=0, tag_tid=3, entry_tid=0], [object_number=2, tag_tid=1, entry_tid=3702], [object_number=3, tag_tid=1, entry_tid=3699]]], state=connected] 2024-04-06T10:03:40.233 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=1, tag_tid=2, entry_tid=1], [object_number=0, tag_tid=2, entry_tid=0], [object_number=2, tag_tid=1, entry_tid=3702], [object_number=3, tag_tid=1, entry_tid=3699]]], state=connected] 2024-04-06T10:03:40.236 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_27.status 2024-04-06T10:03:40.238 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=1, tag_tid=3, entry_tid=1' 2024-04-06T10:03:40.239 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror image_27 2024-04-06T10:03:40.239 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:03:40.239 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:03:40.239 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_27 2024-04-06T10:03:40.239 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror image_27 '..*' 2024-04-06T10:03:40.240 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:03:40.240 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:03:40.240 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_27 2024-04-06T10:03:40.240 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T10:03:40.240 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-image_27.status 2024-04-06T10:03:40.240 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-image_27.status 2024-04-06T10:03:40.240 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T10:03:40.241 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-image_27.status 2024-04-06T10:03:40.241 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_27 2024-04-06T10:03:40.241 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_27.status 2024-04-06T10:03:40.285 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 0 2024-04-06T10:03:40.286 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 0 2024-04-06T10:03:40.286 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T10:03:40.286 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=1, tag_tid=3, entry_tid=1], [object_number=0, tag_tid=3, entry_tid=0], [object_number=2, tag_tid=1, entry_tid=3702], [object_number=3, tag_tid=1, entry_tid=3699]]], state=connected] 2024-04-06T10:03:40.286 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=1, tag_tid=2, entry_tid=1], [object_number=0, tag_tid=2, entry_tid=0], [object_number=2, tag_tid=1, entry_tid=3702], [object_number=3, tag_tid=1, entry_tid=3699]]], state=connected] 2024-04-06T10:03:40.289 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_27.status 2024-04-06T10:03:40.291 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=1, tag_tid=2, entry_tid=1' 2024-04-06T10:03:40.291 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=1, tag_tid=3, entry_tid=1' -a 'object_number=1, tag_tid=3, entry_tid=1' = 'object_number=1, tag_tid=2, entry_tid=1' 2024-04-06T10:03:40.291 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=1, tag_tid=2, entry_tid=1' '!=' 'object_number=1, tag_tid=2, entry_tid=1' 2024-04-06T10:03:40.291 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T10:03:40.292 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.8 2024-04-06T10:03:41.093 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror image_27 2024-04-06T10:03:41.093 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T10:03:41.093 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:03:41.093 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_27 2024-04-06T10:03:41.094 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T10:03:41.094 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n image_27 ']' 2024-04-06T10:03:41.094 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/image_27' 2024-04-06T10:03:41.094 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_27 2024-04-06T10:03:41.094 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T10:03:41.094 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T10:03:41.094 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T10:03:41.094 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T10:03:41.094 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T10:03:41.094 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T10:03:41.094 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T10:03:41.095 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T10:03:41.095 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T10:03:41.097 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T10:03:41.097 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_27 2024-04-06T10:03:41.097 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T10:03:41.097 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T10:03:41.097 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T10:03:41.097 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T10:03:41.097 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T10:03:41.097 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T10:03:41.097 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T10:03:41.097 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T10:03:41.097 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T10:03:41.098 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T10:03:41.098 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T10:03:41.098 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:03:41.098 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T10:03:41.099 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T10:03:41.100 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T10:03:41.100 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T10:03:41.100 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T10:03:41.100 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T10:03:41.100 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T10:03:41.100 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T10:03:41.100 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T10:03:41.100 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T10:03:41.100 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:03:41.100 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T10:03:41.100 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T10:03:41.100 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T10:03:41.100 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T10:03:41.100 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T10:03:41.101 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T10:03:41.101 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T10:03:41.124 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T10:03:41.125 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T10:03:41.125 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T10:03:41.125 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_27 2024-04-06T10:03:41.230 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:Invalid command: unused arguments: ['mirror/image_27'] 2024-04-06T10:03:41.231 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:rbd mirror flush : flush rbd mirror 2024-04-06T10:03:41.231 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:admin_socket: invalid command 2024-04-06T10:03:41.234 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T10:03:41.234 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/image_27 2024-04-06T10:03:41.234 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:1 2024-04-06T10:03:41.234 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T10:03:41.234 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T10:03:41.234 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-06T10:03:41.234 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:1 2024-04-06T10:03:41.234 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T10:03:41.234 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T10:03:41.234 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T10:03:41.235 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=1 2024-04-06T10:03:41.235 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-06T10:03:41.235 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T10:03:41.235 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:03:41.235 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=1 2024-04-06T10:03:41.235 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=1 2024-04-06T10:03:41.235 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-06T10:03:41.235 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:1 2024-04-06T10:03:41.235 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T10:03:41.235 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T10:03:41.235 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-06T10:03:41.235 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:1 2024-04-06T10:03:41.235 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T10:03:41.235 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T10:03:41.235 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:03:41.235 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=1 2024-04-06T10:03:41.236 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-06T10:03:41.236 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T10:03:41.236 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T10:03:41.236 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=1 2024-04-06T10:03:41.236 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=1 2024-04-06T10:03:41.236 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-06T10:03:41.259 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T10:03:41.259 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T10:03:41.259 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T10:03:41.259 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/image_27 2024-04-06T10:03:41.398 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T10:03:41.398 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror image_27 2024-04-06T10:03:41.398 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:03:41.398 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:03:41.398 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_27 2024-04-06T10:03:41.398 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror image_27 '' 2024-04-06T10:03:41.398 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:03:41.398 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:03:41.398 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_27 2024-04-06T10:03:41.398 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T10:03:41.399 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-image_27.status 2024-04-06T10:03:41.399 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-image_27.status 2024-04-06T10:03:41.399 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T10:03:41.400 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-image_27.status 2024-04-06T10:03:41.400 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_27 2024-04-06T10:03:41.401 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_27.status 2024-04-06T10:03:41.456 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 0 2024-04-06T10:03:41.456 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 0 2024-04-06T10:03:41.456 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T10:03:41.456 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=1, tag_tid=3, entry_tid=1], [object_number=0, tag_tid=3, entry_tid=0], [object_number=2, tag_tid=1, entry_tid=3702], [object_number=3, tag_tid=1, entry_tid=3699]]], state=connected] 2024-04-06T10:03:41.456 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=1, tag_tid=2, entry_tid=1], [object_number=0, tag_tid=2, entry_tid=0], [object_number=2, tag_tid=1, entry_tid=3702], [object_number=3, tag_tid=1, entry_tid=3699]]], state=connected] 2024-04-06T10:03:41.459 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_27.status 2024-04-06T10:03:41.461 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=1, tag_tid=3, entry_tid=1' 2024-04-06T10:03:41.462 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror image_27 2024-04-06T10:03:41.462 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:03:41.462 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:03:41.462 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_27 2024-04-06T10:03:41.462 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror image_27 '..*' 2024-04-06T10:03:41.462 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:03:41.463 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:03:41.463 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_27 2024-04-06T10:03:41.463 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T10:03:41.463 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-image_27.status 2024-04-06T10:03:41.463 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-image_27.status 2024-04-06T10:03:41.463 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T10:03:41.463 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-image_27.status 2024-04-06T10:03:41.464 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_27 2024-04-06T10:03:41.464 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_27.status 2024-04-06T10:03:41.509 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 0 2024-04-06T10:03:41.509 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 0 2024-04-06T10:03:41.509 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T10:03:41.509 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=1, tag_tid=3, entry_tid=1], [object_number=0, tag_tid=3, entry_tid=0], [object_number=2, tag_tid=1, entry_tid=3702], [object_number=3, tag_tid=1, entry_tid=3699]]], state=connected] 2024-04-06T10:03:41.509 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=1, tag_tid=2, entry_tid=1], [object_number=0, tag_tid=2, entry_tid=0], [object_number=2, tag_tid=1, entry_tid=3702], [object_number=3, tag_tid=1, entry_tid=3699]]], state=connected] 2024-04-06T10:03:41.513 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_27.status 2024-04-06T10:03:41.514 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=1, tag_tid=2, entry_tid=1' 2024-04-06T10:03:41.515 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=1, tag_tid=3, entry_tid=1' -a 'object_number=1, tag_tid=3, entry_tid=1' = 'object_number=1, tag_tid=2, entry_tid=1' 2024-04-06T10:03:41.515 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=1, tag_tid=2, entry_tid=1' '!=' 'object_number=1, tag_tid=2, entry_tid=1' 2024-04-06T10:03:41.515 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T10:03:41.515 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 1.6 2024-04-06T10:03:43.116 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror image_27 2024-04-06T10:03:43.116 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T10:03:43.117 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:03:43.117 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_27 2024-04-06T10:03:43.117 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T10:03:43.117 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n image_27 ']' 2024-04-06T10:03:43.117 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/image_27' 2024-04-06T10:03:43.117 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_27 2024-04-06T10:03:43.117 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T10:03:43.117 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T10:03:43.117 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T10:03:43.117 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T10:03:43.117 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T10:03:43.117 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T10:03:43.117 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T10:03:43.118 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T10:03:43.118 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T10:03:43.121 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T10:03:43.121 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_27 2024-04-06T10:03:43.121 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T10:03:43.122 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T10:03:43.122 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T10:03:43.122 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T10:03:43.122 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T10:03:43.122 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T10:03:43.122 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T10:03:43.122 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T10:03:43.122 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T10:03:43.122 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T10:03:43.122 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T10:03:43.122 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:03:43.122 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T10:03:43.122 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T10:03:43.123 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T10:03:43.123 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T10:03:43.123 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T10:03:43.123 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T10:03:43.123 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T10:03:43.123 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T10:03:43.123 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T10:03:43.123 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T10:03:43.123 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:03:43.123 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T10:03:43.123 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T10:03:43.123 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T10:03:43.123 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T10:03:43.124 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T10:03:43.124 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T10:03:43.124 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T10:03:43.148 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T10:03:43.148 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T10:03:43.148 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T10:03:43.148 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_27 2024-04-06T10:03:43.254 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:Invalid command: unused arguments: ['mirror/image_27'] 2024-04-06T10:03:43.254 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:rbd mirror flush : flush rbd mirror 2024-04-06T10:03:43.254 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:admin_socket: invalid command 2024-04-06T10:03:43.258 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T10:03:43.258 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/image_27 2024-04-06T10:03:43.258 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:1 2024-04-06T10:03:43.258 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T10:03:43.258 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T10:03:43.258 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-06T10:03:43.259 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:1 2024-04-06T10:03:43.259 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T10:03:43.259 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T10:03:43.259 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T10:03:43.259 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=1 2024-04-06T10:03:43.259 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-06T10:03:43.259 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T10:03:43.259 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:03:43.259 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=1 2024-04-06T10:03:43.259 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=1 2024-04-06T10:03:43.259 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-06T10:03:43.259 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:1 2024-04-06T10:03:43.259 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T10:03:43.259 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T10:03:43.259 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-06T10:03:43.260 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:1 2024-04-06T10:03:43.260 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T10:03:43.260 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T10:03:43.260 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:03:43.260 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=1 2024-04-06T10:03:43.260 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-06T10:03:43.260 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T10:03:43.260 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T10:03:43.260 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=1 2024-04-06T10:03:43.260 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=1 2024-04-06T10:03:43.260 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-06T10:03:43.283 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T10:03:43.283 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T10:03:43.283 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T10:03:43.284 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/image_27 2024-04-06T10:03:43.420 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T10:03:43.421 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror image_27 2024-04-06T10:03:43.422 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:03:43.422 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:03:43.422 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_27 2024-04-06T10:03:43.422 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror image_27 '' 2024-04-06T10:03:43.422 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:03:43.422 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:03:43.422 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_27 2024-04-06T10:03:43.422 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T10:03:43.423 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-image_27.status 2024-04-06T10:03:43.423 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-image_27.status 2024-04-06T10:03:43.423 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T10:03:43.424 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-image_27.status 2024-04-06T10:03:43.424 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_27.status 2024-04-06T10:03:43.425 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_27 2024-04-06T10:03:43.474 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 0 2024-04-06T10:03:43.475 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 0 2024-04-06T10:03:43.475 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T10:03:43.475 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=1, tag_tid=3, entry_tid=1], [object_number=0, tag_tid=3, entry_tid=0], [object_number=2, tag_tid=1, entry_tid=3702], [object_number=3, tag_tid=1, entry_tid=3699]]], state=connected] 2024-04-06T10:03:43.475 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=1, tag_tid=2, entry_tid=1], [object_number=0, tag_tid=2, entry_tid=0], [object_number=2, tag_tid=1, entry_tid=3702], [object_number=3, tag_tid=1, entry_tid=3699]]], state=connected] 2024-04-06T10:03:43.478 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_27.status 2024-04-06T10:03:43.480 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=1, tag_tid=3, entry_tid=1' 2024-04-06T10:03:43.481 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror image_27 2024-04-06T10:03:43.481 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:03:43.482 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:03:43.482 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_27 2024-04-06T10:03:43.482 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror image_27 '..*' 2024-04-06T10:03:43.482 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:03:43.482 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:03:43.482 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_27 2024-04-06T10:03:43.482 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T10:03:43.482 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-image_27.status 2024-04-06T10:03:43.482 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-image_27.status 2024-04-06T10:03:43.482 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T10:03:43.483 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-image_27.status 2024-04-06T10:03:43.483 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_27 2024-04-06T10:03:43.483 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_27.status 2024-04-06T10:03:43.528 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 0 2024-04-06T10:03:43.528 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 0 2024-04-06T10:03:43.528 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T10:03:43.528 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=1, tag_tid=3, entry_tid=1], [object_number=0, tag_tid=3, entry_tid=0], [object_number=2, tag_tid=1, entry_tid=3702], [object_number=3, tag_tid=1, entry_tid=3699]]], state=connected] 2024-04-06T10:03:43.529 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=1, tag_tid=2, entry_tid=1], [object_number=0, tag_tid=2, entry_tid=0], [object_number=2, tag_tid=1, entry_tid=3702], [object_number=3, tag_tid=1, entry_tid=3699]]], state=connected] 2024-04-06T10:03:43.532 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_27.status 2024-04-06T10:03:43.534 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=1, tag_tid=2, entry_tid=1' 2024-04-06T10:03:43.534 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=1, tag_tid=3, entry_tid=1' -a 'object_number=1, tag_tid=3, entry_tid=1' = 'object_number=1, tag_tid=2, entry_tid=1' 2024-04-06T10:03:43.534 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=1, tag_tid=2, entry_tid=1' '!=' 'object_number=1, tag_tid=2, entry_tid=1' 2024-04-06T10:03:43.534 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T10:03:43.534 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 2 2024-04-06T10:03:45.535 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror image_27 2024-04-06T10:03:45.536 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T10:03:45.536 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:03:45.536 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_27 2024-04-06T10:03:45.536 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T10:03:45.536 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n image_27 ']' 2024-04-06T10:03:45.536 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/image_27' 2024-04-06T10:03:45.536 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_27 2024-04-06T10:03:45.536 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T10:03:45.536 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T10:03:45.536 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T10:03:45.536 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T10:03:45.537 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T10:03:45.537 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T10:03:45.537 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T10:03:45.537 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T10:03:45.537 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T10:03:45.539 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T10:03:45.539 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_27 2024-04-06T10:03:45.539 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T10:03:45.539 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T10:03:45.539 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T10:03:45.539 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T10:03:45.539 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T10:03:45.539 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T10:03:45.539 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T10:03:45.540 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T10:03:45.540 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T10:03:45.540 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T10:03:45.540 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T10:03:45.540 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:03:45.540 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T10:03:45.540 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T10:03:45.541 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T10:03:45.541 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T10:03:45.541 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T10:03:45.541 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T10:03:45.542 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T10:03:45.542 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T10:03:45.542 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T10:03:45.542 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T10:03:45.542 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:03:45.542 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T10:03:45.542 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T10:03:45.542 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T10:03:45.542 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T10:03:45.542 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T10:03:45.542 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T10:03:45.543 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T10:03:45.566 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T10:03:45.566 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T10:03:45.567 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T10:03:45.567 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_27 2024-04-06T10:03:45.672 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:Invalid command: unused arguments: ['mirror/image_27'] 2024-04-06T10:03:45.672 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:rbd mirror flush : flush rbd mirror 2024-04-06T10:03:45.672 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:admin_socket: invalid command 2024-04-06T10:03:45.674 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T10:03:45.674 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/image_27 2024-04-06T10:03:45.674 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:1 2024-04-06T10:03:45.674 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T10:03:45.674 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T10:03:45.674 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-06T10:03:45.674 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:1 2024-04-06T10:03:45.674 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T10:03:45.675 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T10:03:45.675 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T10:03:45.675 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=1 2024-04-06T10:03:45.675 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-06T10:03:45.675 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T10:03:45.675 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:03:45.675 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=1 2024-04-06T10:03:45.675 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=1 2024-04-06T10:03:45.676 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-06T10:03:45.676 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:1 2024-04-06T10:03:45.677 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T10:03:45.677 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T10:03:45.677 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-06T10:03:45.677 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:1 2024-04-06T10:03:45.677 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T10:03:45.677 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T10:03:45.677 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:03:45.677 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=1 2024-04-06T10:03:45.677 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-06T10:03:45.677 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T10:03:45.677 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T10:03:45.677 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=1 2024-04-06T10:03:45.677 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=1 2024-04-06T10:03:45.677 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-06T10:03:45.695 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T10:03:45.695 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T10:03:45.695 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T10:03:45.695 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/image_27 2024-04-06T10:03:45.831 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T10:03:45.832 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror image_27 2024-04-06T10:03:45.832 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:03:45.832 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:03:45.832 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_27 2024-04-06T10:03:45.833 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror image_27 '' 2024-04-06T10:03:45.833 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:03:45.833 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:03:45.833 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_27 2024-04-06T10:03:45.833 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T10:03:45.833 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-image_27.status 2024-04-06T10:03:45.833 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-image_27.status 2024-04-06T10:03:45.834 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T10:03:45.836 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-image_27.status 2024-04-06T10:03:45.837 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_27 2024-04-06T10:03:45.837 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_27.status 2024-04-06T10:03:45.892 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 0 2024-04-06T10:03:45.892 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 0 2024-04-06T10:03:45.892 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T10:03:45.893 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=1, tag_tid=3, entry_tid=1], [object_number=0, tag_tid=3, entry_tid=0], [object_number=2, tag_tid=1, entry_tid=3702], [object_number=3, tag_tid=1, entry_tid=3699]]], state=connected] 2024-04-06T10:03:45.893 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=1, tag_tid=2, entry_tid=1], [object_number=0, tag_tid=2, entry_tid=0], [object_number=2, tag_tid=1, entry_tid=3702], [object_number=3, tag_tid=1, entry_tid=3699]]], state=connected] 2024-04-06T10:03:45.896 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_27.status 2024-04-06T10:03:45.897 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=1, tag_tid=3, entry_tid=1' 2024-04-06T10:03:45.898 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror image_27 2024-04-06T10:03:45.899 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:03:45.899 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:03:45.899 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_27 2024-04-06T10:03:45.899 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror image_27 '..*' 2024-04-06T10:03:45.899 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:03:45.899 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:03:45.899 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_27 2024-04-06T10:03:45.899 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T10:03:45.899 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-image_27.status 2024-04-06T10:03:45.899 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-image_27.status 2024-04-06T10:03:45.899 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T10:03:45.900 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-image_27.status 2024-04-06T10:03:45.900 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_27 2024-04-06T10:03:45.900 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_27.status 2024-04-06T10:03:45.946 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 0 2024-04-06T10:03:45.946 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 0 2024-04-06T10:03:45.946 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T10:03:45.946 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=1, tag_tid=3, entry_tid=1], [object_number=0, tag_tid=3, entry_tid=0], [object_number=2, tag_tid=1, entry_tid=3702], [object_number=3, tag_tid=1, entry_tid=3699]]], state=connected] 2024-04-06T10:03:45.946 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=1, tag_tid=2, entry_tid=1], [object_number=0, tag_tid=2, entry_tid=0], [object_number=2, tag_tid=1, entry_tid=3702], [object_number=3, tag_tid=1, entry_tid=3699]]], state=connected] 2024-04-06T10:03:45.950 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_27.status 2024-04-06T10:03:45.951 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=1, tag_tid=2, entry_tid=1' 2024-04-06T10:03:45.951 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=1, tag_tid=3, entry_tid=1' -a 'object_number=1, tag_tid=3, entry_tid=1' = 'object_number=1, tag_tid=2, entry_tid=1' 2024-04-06T10:03:45.952 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=1, tag_tid=2, entry_tid=1' '!=' 'object_number=1, tag_tid=2, entry_tid=1' 2024-04-06T10:03:45.952 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T10:03:45.952 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 2 2024-04-06T10:03:47.953 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror image_27 2024-04-06T10:03:47.954 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T10:03:47.954 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:03:47.954 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_27 2024-04-06T10:03:47.954 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T10:03:47.954 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n image_27 ']' 2024-04-06T10:03:47.954 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/image_27' 2024-04-06T10:03:47.954 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_27 2024-04-06T10:03:47.954 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T10:03:47.954 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T10:03:47.954 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T10:03:47.954 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T10:03:47.954 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T10:03:47.954 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T10:03:47.955 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T10:03:47.955 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T10:03:47.955 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T10:03:47.958 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T10:03:47.958 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_27 2024-04-06T10:03:47.958 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T10:03:47.958 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T10:03:47.958 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T10:03:47.958 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T10:03:47.958 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T10:03:47.958 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T10:03:47.958 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T10:03:47.958 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T10:03:47.958 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T10:03:47.959 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T10:03:47.959 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T10:03:47.959 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:03:47.959 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T10:03:47.959 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T10:03:47.959 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T10:03:47.959 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T10:03:47.959 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T10:03:47.959 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T10:03:47.959 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T10:03:47.960 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T10:03:47.960 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T10:03:47.960 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T10:03:47.961 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:03:47.961 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T10:03:47.961 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T10:03:47.961 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T10:03:47.961 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T10:03:47.961 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T10:03:47.961 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T10:03:47.961 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T10:03:47.987 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T10:03:47.987 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T10:03:47.987 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T10:03:47.987 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_27 2024-04-06T10:03:48.091 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:Invalid command: unused arguments: ['mirror/image_27'] 2024-04-06T10:03:48.092 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:rbd mirror flush : flush rbd mirror 2024-04-06T10:03:48.092 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:admin_socket: invalid command 2024-04-06T10:03:48.096 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T10:03:48.096 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/image_27 2024-04-06T10:03:48.096 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:1 2024-04-06T10:03:48.096 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T10:03:48.096 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T10:03:48.096 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-06T10:03:48.096 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:1 2024-04-06T10:03:48.097 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T10:03:48.097 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T10:03:48.097 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T10:03:48.097 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=1 2024-04-06T10:03:48.097 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-06T10:03:48.097 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T10:03:48.097 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:03:48.097 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=1 2024-04-06T10:03:48.097 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=1 2024-04-06T10:03:48.097 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-06T10:03:48.097 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:1 2024-04-06T10:03:48.097 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T10:03:48.097 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T10:03:48.097 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-06T10:03:48.098 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:1 2024-04-06T10:03:48.098 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T10:03:48.098 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T10:03:48.098 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:03:48.098 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=1 2024-04-06T10:03:48.098 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-06T10:03:48.098 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T10:03:48.098 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T10:03:48.098 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=1 2024-04-06T10:03:48.098 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=1 2024-04-06T10:03:48.098 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-06T10:03:48.120 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T10:03:48.120 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T10:03:48.120 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T10:03:48.120 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/image_27 2024-04-06T10:03:48.263 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T10:03:48.264 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror image_27 2024-04-06T10:03:48.265 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:03:48.265 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:03:48.265 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_27 2024-04-06T10:03:48.265 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror image_27 '' 2024-04-06T10:03:48.265 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:03:48.265 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:03:48.265 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_27 2024-04-06T10:03:48.265 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T10:03:48.266 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-image_27.status 2024-04-06T10:03:48.266 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-image_27.status 2024-04-06T10:03:48.266 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T10:03:48.268 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-image_27.status 2024-04-06T10:03:48.269 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_27 2024-04-06T10:03:48.269 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_27.status 2024-04-06T10:03:48.324 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 0 2024-04-06T10:03:48.324 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 0 2024-04-06T10:03:48.324 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T10:03:48.324 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=1, tag_tid=3, entry_tid=1], [object_number=0, tag_tid=3, entry_tid=0], [object_number=2, tag_tid=1, entry_tid=3702], [object_number=3, tag_tid=1, entry_tid=3699]]], state=connected] 2024-04-06T10:03:48.325 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=1, tag_tid=3, entry_tid=1], [object_number=0, tag_tid=3, entry_tid=0], [object_number=2, tag_tid=1, entry_tid=3702], [object_number=3, tag_tid=1, entry_tid=3699]]], state=connected] 2024-04-06T10:03:48.328 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_27.status 2024-04-06T10:03:48.329 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=1, tag_tid=3, entry_tid=1' 2024-04-06T10:03:48.330 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror image_27 2024-04-06T10:03:48.331 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:03:48.331 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:03:48.331 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_27 2024-04-06T10:03:48.331 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror image_27 '..*' 2024-04-06T10:03:48.331 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:03:48.331 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:03:48.331 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_27 2024-04-06T10:03:48.331 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T10:03:48.331 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-image_27.status 2024-04-06T10:03:48.331 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-image_27.status 2024-04-06T10:03:48.331 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T10:03:48.332 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-image_27.status 2024-04-06T10:03:48.332 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_27 2024-04-06T10:03:48.332 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_27.status 2024-04-06T10:03:48.379 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 0 2024-04-06T10:03:48.379 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 0 2024-04-06T10:03:48.379 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T10:03:48.379 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=1, tag_tid=3, entry_tid=1], [object_number=0, tag_tid=3, entry_tid=0], [object_number=2, tag_tid=1, entry_tid=3702], [object_number=3, tag_tid=1, entry_tid=3699]]], state=connected] 2024-04-06T10:03:48.380 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=1, tag_tid=3, entry_tid=1], [object_number=0, tag_tid=3, entry_tid=0], [object_number=2, tag_tid=1, entry_tid=3702], [object_number=3, tag_tid=1, entry_tid=3699]]], state=connected] 2024-04-06T10:03:48.383 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_27.status 2024-04-06T10:03:48.385 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=1, tag_tid=3, entry_tid=1' 2024-04-06T10:03:48.385 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=1, tag_tid=3, entry_tid=1' -a 'object_number=1, tag_tid=3, entry_tid=1' = 'object_number=1, tag_tid=3, entry_tid=1' 2024-04-06T10:03:48.385 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T10:03:48.385 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_snap_present cluster1 mirror image_27 snap 2024-04-06T10:03:48.385 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T10:03:48.385 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:03:48.385 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_27 2024-04-06T10:03:48.385 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local snap_name=snap 2024-04-06T10:03:48.385 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local s 2024-04-06T10:03:48.385 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-04-06T10:03:48.386 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 1 2024-04-06T10:03:49.386 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 info mirror/image_27@snap 2024-04-06T10:03:49.453 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:rbd image 'image_27': 2024-04-06T10:03:49.453 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: size 128 MiB in 32 objects 2024-04-06T10:03:49.453 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: order 22 (4 MiB objects) 2024-04-06T10:03:49.453 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: snapshot_count: 1 2024-04-06T10:03:49.453 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: id: 10ac1f8b33c4 2024-04-06T10:03:49.453 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: block_name_prefix: rbd_data.10ac1f8b33c4 2024-04-06T10:03:49.453 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: format: 2 2024-04-06T10:03:49.453 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: features: layering, exclusive-lock, journaling 2024-04-06T10:03:49.454 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: op_features: 2024-04-06T10:03:49.454 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: flags: 2024-04-06T10:03:49.454 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: create_timestamp: Sat Apr 6 09:54:48 2024 2024-04-06T10:03:49.454 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: access_timestamp: Sat Apr 6 09:54:48 2024 2024-04-06T10:03:49.454 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: modify_timestamp: Sat Apr 6 09:55:49 2024 2024-04-06T10:03:49.454 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: protected: False 2024-04-06T10:03:49.454 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: journal: 10ac1f8b33c4 2024-04-06T10:03:49.454 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: mirroring state: enabled 2024-04-06T10:03:49.454 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: mirroring mode: journal 2024-04-06T10:03:49.454 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: mirroring global id: 41f46006-65f8-4c48-a687-51177330576e 2024-04-06T10:03:49.454 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: mirroring primary: false 2024-04-06T10:03:49.459 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T10:03:49.459 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ compare_image_snaps mirror image_27 snap 2024-04-06T10:03:49.459 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:03:49.459 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_27 2024-04-06T10:03:49.459 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local snap_name=snap 2024-04-06T10:03:49.459 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local ret=0 2024-04-06T10:03:49.459 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local rmt_export=/tmp/tmp.BhW6xKmROU/cluster2-mirror-image_27.export 2024-04-06T10:03:49.459 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loc_export=/tmp/tmp.BhW6xKmROU/cluster1-mirror-image_27.export 2024-04-06T10:03:49.459 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rm -f /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_27.export /tmp/tmp.BhW6xKmROU/cluster1-mirror-image_27.export 2024-04-06T10:03:49.459 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster2 -p mirror export image_27@snap /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_27.export 2024-04-06T10:03:49.633 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-04-06T10:03:49.641 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 -p mirror export image_27@snap /tmp/tmp.BhW6xKmROU/cluster1-mirror-image_27.export 2024-04-06T10:03:49.976 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-04-06T10:03:49.988 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmp /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_27.export /tmp/tmp.BhW6xKmROU/cluster1-mirror-image_27.export 2024-04-06T10:03:50.089 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rm -f /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_27.export /tmp/tmp.BhW6xKmROU/cluster1-mirror-image_27.export 2024-04-06T10:03:50.133 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T10:03:50.133 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-06T10:03:50.133 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ image=image_28 2024-04-06T10:03:50.134 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ create_snap cluster2 mirror image_28 snap 2024-04-06T10:03:50.134 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-04-06T10:03:50.134 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:03:50.134 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_28 2024-04-06T10:03:50.134 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local snap_name=snap 2024-04-06T10:03:50.134 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster2 -p mirror snap create image_28@snap --debug-rbd=20 --debug-journaler=20 2024-04-06T10:03:51.899 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_image_replay_started cluster1 mirror image_28 2024-04-06T10:03:51.899 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T10:03:51.899 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:03:51.900 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_28 2024-04-06T10:03:51.900 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_image_replay_state cluster1 mirror image_28 started 2024-04-06T10:03:51.900 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T10:03:51.900 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:03:51.900 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_28 2024-04-06T10:03:51.900 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state=started 2024-04-06T10:03:51.900 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local s 2024-04-06T10:03:51.900 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T10:03:51.900 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 1 2024-04-06T10:03:52.901 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test_image_replay_state cluster1 mirror image_28 started 2024-04-06T10:03:52.901 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T10:03:52.901 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:03:52.901 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_28 2024-04-06T10:03:52.901 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local test_state=started 2024-04-06T10:03:52.902 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local status_result 2024-04-06T10:03:52.902 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local current_state=stopped 2024-04-06T10:03:52.903 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_28 2024-04-06T10:03:52.903 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -i state 2024-04-06T10:03:52.903 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_instance=cluster1 2024-04-06T10:03:52.903 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ shift 2024-04-06T10:03:52.903 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T10:03:52.903 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance=cluster1 2024-04-06T10:03:52.903 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local loop_instance 2024-04-06T10:03:52.903 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T10:03:52.904 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sleep 0 2024-04-06T10:03:52.904 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-04-06T10:03:52.905 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ seq 0 3 2024-04-06T10:03:52.907 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T10:03:52.907 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_28 2024-04-06T10:03:52.907 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1:0 2024-04-06T10:03:52.907 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ shift 2024-04-06T10:03:52.907 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T10:03:52.907 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-04-06T10:03:52.907 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T10:03:52.908 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=cluster 2024-04-06T10:03:52.908 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T10:03:52.908 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:03:52.908 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T10:03:52.908 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T10:03:52.908 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval cluster=cluster1 2024-04-06T10:03:52.908 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ cluster=cluster1 2024-04-06T10:03:52.908 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T10:03:52.908 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T10:03:52.908 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-04-06T10:03:52.908 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local local_cluster=cluster1:0 2024-04-06T10:03:52.908 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local cluster=cluster1 2024-04-06T10:03:52.908 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local instance 2024-04-06T10:03:52.909 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T10:03:52.909 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local val=cluster1:0 2024-04-06T10:03:52.909 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local cluster_var_name=local_cluster 2024-04-06T10:03:52.909 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local instance_var_name=instance 2024-04-06T10:03:52.910 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ cluster=cluster1 2024-04-06T10:03:52.910 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T10:03:52.910 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ '[' 0 = cluster1:0 ']' 2024-04-06T10:03:52.910 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ eval local_cluster=cluster1 2024-04-06T10:03:52.910 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++++ local_cluster=cluster1 2024-04-06T10:03:52.910 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ eval instance=0 2024-04-06T10:03:52.910 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++++ instance=0 2024-04-06T10:03:52.911 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T10:03:52.933 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T10:03:52.933 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T10:03:52.933 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T10:03:52.933 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_28 2024-04-06T10:03:53.046 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:Invalid command: unused arguments: ['mirror/image_28'] 2024-04-06T10:03:53.047 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:rbd mirror status : get status for rbd mirror 2024-04-06T10:03:53.047 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:admin_socket: invalid command 2024-04-06T10:03:53.048 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T10:03:53.048 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_28 2024-04-06T10:03:53.048 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1:1 2024-04-06T10:03:53.049 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ shift 2024-04-06T10:03:53.049 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T10:03:53.049 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-04-06T10:03:53.049 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:1 2024-04-06T10:03:53.049 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=cluster 2024-04-06T10:03:53.049 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T10:03:53.049 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:03:53.049 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=1 2024-04-06T10:03:53.049 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-06T10:03:53.049 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval cluster=cluster1 2024-04-06T10:03:53.049 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ cluster=cluster1 2024-04-06T10:03:53.049 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=1 2024-04-06T10:03:53.049 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=1 2024-04-06T10:03:53.051 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-04-06T10:03:53.051 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local local_cluster=cluster1:1 2024-04-06T10:03:53.051 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local cluster=cluster1 2024-04-06T10:03:53.051 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local instance 2024-04-06T10:03:53.051 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-06T10:03:53.051 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local val=cluster1:1 2024-04-06T10:03:53.051 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local cluster_var_name=local_cluster 2024-04-06T10:03:53.051 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local instance_var_name=instance 2024-04-06T10:03:53.051 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ cluster=cluster1 2024-04-06T10:03:53.052 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=1 2024-04-06T10:03:53.052 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ '[' 1 = cluster1:1 ']' 2024-04-06T10:03:53.052 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ eval local_cluster=cluster1 2024-04-06T10:03:53.052 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++++ local_cluster=cluster1 2024-04-06T10:03:53.052 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ eval instance=1 2024-04-06T10:03:53.052 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++++ instance=1 2024-04-06T10:03:53.052 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-06T10:03:53.068 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T10:03:53.069 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T10:03:53.069 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T10:03:53.069 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_28 2024-04-06T10:03:53.216 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ return 0 2024-04-06T10:03:53.217 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ status_result=' "state": "Replaying"' 2024-04-06T10:03:53.218 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ echo ' "state": "Replaying"' 2024-04-06T10:03:53.218 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep -i Replaying 2024-04-06T10:03:53.219 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: "state": "Replaying" 2024-04-06T10:03:53.220 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ current_state=started 2024-04-06T10:03:53.220 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test started = started 2024-04-06T10:03:53.220 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T10:03:53.220 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_28 2024-04-06T10:03:53.220 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local local_cluster=cluster1 2024-04-06T10:03:53.221 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-04-06T10:03:53.221 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:03:53.221 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_28 2024-04-06T10:03:53.221 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' journal = journal ']' 2024-04-06T10:03:53.221 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_journal_replay_complete cluster1 cluster2 mirror image_28 2024-04-06T10:03:53.221 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local local_cluster=cluster1 2024-04-06T10:03:53.221 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-04-06T10:03:53.221 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:03:53.221 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_28 2024-04-06T10:03:53.221 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local s master_pos mirror_pos last_mirror_pos 2024-04-06T10:03:53.221 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local master_tag master_entry mirror_tag mirror_entry 2024-04-06T10:03:53.221 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T10:03:53.221 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T10:03:53.221 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T10:03:53.422 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror image_28 2024-04-06T10:03:53.422 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T10:03:53.422 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:03:53.422 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_28 2024-04-06T10:03:53.422 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T10:03:53.422 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n image_28 ']' 2024-04-06T10:03:53.422 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/image_28' 2024-04-06T10:03:53.422 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_28 2024-04-06T10:03:53.422 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T10:03:53.423 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T10:03:53.423 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T10:03:53.423 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T10:03:53.423 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T10:03:53.423 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T10:03:53.423 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T10:03:53.423 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T10:03:53.423 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T10:03:53.426 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T10:03:53.426 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_28 2024-04-06T10:03:53.426 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T10:03:53.426 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T10:03:53.427 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T10:03:53.427 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T10:03:53.427 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T10:03:53.427 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T10:03:53.427 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T10:03:53.427 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T10:03:53.427 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T10:03:53.427 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T10:03:53.427 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T10:03:53.427 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:03:53.427 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T10:03:53.427 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T10:03:53.427 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T10:03:53.427 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T10:03:53.428 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T10:03:53.428 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T10:03:53.428 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T10:03:53.428 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T10:03:53.428 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T10:03:53.428 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T10:03:53.428 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:03:53.428 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T10:03:53.428 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T10:03:53.428 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T10:03:53.428 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T10:03:53.428 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T10:03:53.428 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T10:03:53.428 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T10:03:53.452 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T10:03:53.452 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T10:03:53.452 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T10:03:53.452 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_28 2024-04-06T10:03:53.557 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:Invalid command: unused arguments: ['mirror/image_28'] 2024-04-06T10:03:53.557 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:rbd mirror flush : flush rbd mirror 2024-04-06T10:03:53.557 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:admin_socket: invalid command 2024-04-06T10:03:53.559 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T10:03:53.559 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/image_28 2024-04-06T10:03:53.559 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:1 2024-04-06T10:03:53.560 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T10:03:53.560 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T10:03:53.560 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-06T10:03:53.560 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:1 2024-04-06T10:03:53.560 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T10:03:53.560 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T10:03:53.560 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T10:03:53.560 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=1 2024-04-06T10:03:53.560 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-06T10:03:53.560 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T10:03:53.560 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:03:53.560 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=1 2024-04-06T10:03:53.560 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=1 2024-04-06T10:03:53.562 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-06T10:03:53.562 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:1 2024-04-06T10:03:53.562 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T10:03:53.563 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T10:03:53.563 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-06T10:03:53.563 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:1 2024-04-06T10:03:53.563 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T10:03:53.563 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T10:03:53.563 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:03:53.563 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=1 2024-04-06T10:03:53.563 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-06T10:03:53.563 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T10:03:53.563 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T10:03:53.563 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=1 2024-04-06T10:03:53.563 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=1 2024-04-06T10:03:53.563 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-06T10:03:53.585 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T10:03:53.586 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T10:03:53.586 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T10:03:53.586 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/image_28 2024-04-06T10:03:53.729 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T10:03:53.730 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror image_28 2024-04-06T10:03:53.730 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:03:53.730 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:03:53.730 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_28 2024-04-06T10:03:53.730 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror image_28 '' 2024-04-06T10:03:53.730 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:03:53.730 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:03:53.730 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_28 2024-04-06T10:03:53.730 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T10:03:53.731 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-image_28.status 2024-04-06T10:03:53.731 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-image_28.status 2024-04-06T10:03:53.731 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T10:03:53.734 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-image_28.status 2024-04-06T10:03:53.734 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_28 2024-04-06T10:03:53.734 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_28.status 2024-04-06T10:03:53.786 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 2 2024-04-06T10:03:53.786 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 2 2024-04-06T10:03:53.786 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T10:03:53.786 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=9, tag_tid=2, entry_tid=1], [object_number=8, tag_tid=2, entry_tid=0], [object_number=11, tag_tid=1, entry_tid=10683], [object_number=10, tag_tid=1, entry_tid=10682]]], state=connected] 2024-04-06T10:03:53.786 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=9, tag_tid=1, entry_tid=10685], [object_number=8, tag_tid=1, entry_tid=10684], [object_number=11, tag_tid=1, entry_tid=10683], [object_number=10, tag_tid=1, entry_tid=10682]]], state=connected] 2024-04-06T10:03:53.789 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_28.status 2024-04-06T10:03:53.791 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=9, tag_tid=2, entry_tid=1' 2024-04-06T10:03:53.792 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror image_28 2024-04-06T10:03:53.792 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:03:53.792 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:03:53.792 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_28 2024-04-06T10:03:53.792 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror image_28 '..*' 2024-04-06T10:03:53.792 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:03:53.792 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:03:53.792 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_28 2024-04-06T10:03:53.792 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T10:03:53.793 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-image_28.status 2024-04-06T10:03:53.793 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-image_28.status 2024-04-06T10:03:53.793 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T10:03:53.793 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-image_28.status 2024-04-06T10:03:53.794 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_28 2024-04-06T10:03:53.794 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_28.status 2024-04-06T10:03:53.841 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 2 2024-04-06T10:03:53.841 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 2 2024-04-06T10:03:53.841 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T10:03:53.841 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=9, tag_tid=2, entry_tid=1], [object_number=8, tag_tid=2, entry_tid=0], [object_number=11, tag_tid=1, entry_tid=10683], [object_number=10, tag_tid=1, entry_tid=10682]]], state=connected] 2024-04-06T10:03:53.841 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=9, tag_tid=1, entry_tid=10685], [object_number=8, tag_tid=1, entry_tid=10684], [object_number=11, tag_tid=1, entry_tid=10683], [object_number=10, tag_tid=1, entry_tid=10682]]], state=connected] 2024-04-06T10:03:53.845 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_28.status 2024-04-06T10:03:53.846 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=9, tag_tid=1, entry_tid=10685' 2024-04-06T10:03:53.846 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=9, tag_tid=2, entry_tid=1' -a 'object_number=9, tag_tid=2, entry_tid=1' = 'object_number=9, tag_tid=1, entry_tid=10685' 2024-04-06T10:03:53.847 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=9, tag_tid=1, entry_tid=10685' '!=' '' 2024-04-06T10:03:53.847 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T10:03:53.847 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=9, tag_tid=1, entry_tid=10685' = '' 2024-04-06T10:03:53.847 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=9, tag_tid=1, entry_tid=10685' 2024-04-06T10:03:53.847 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=9, tag_tid=2, entry_tid=1' 2024-04-06T10:03:53.847 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T10:03:53.848 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T10:03:53.848 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=2 2024-04-06T10:03:53.849 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=9, tag_tid=1, entry_tid=10685' 2024-04-06T10:03:53.849 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T10:03:53.849 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T10:03:53.850 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=1 2024-04-06T10:03:53.850 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=9, tag_tid=2, entry_tid=1' 2024-04-06T10:03:53.851 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T10:03:53.851 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T10:03:53.851 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T10:03:53.852 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=9, tag_tid=1, entry_tid=10685' 2024-04-06T10:03:53.852 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T10:03:53.852 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T10:03:53.853 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=10685 2024-04-06T10:03:53.854 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 2 = 1 -a 1 -le 10685 2024-04-06T10:03:53.854 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T10:03:53.854 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T10:03:53.854 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T10:03:54.056 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror image_28 2024-04-06T10:03:54.056 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T10:03:54.056 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:03:54.056 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_28 2024-04-06T10:03:54.056 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T10:03:54.056 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n image_28 ']' 2024-04-06T10:03:54.056 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/image_28' 2024-04-06T10:03:54.057 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_28 2024-04-06T10:03:54.057 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T10:03:54.057 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T10:03:54.057 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T10:03:54.057 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T10:03:54.057 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T10:03:54.057 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T10:03:54.057 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T10:03:54.057 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T10:03:54.058 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T10:03:54.060 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T10:03:54.061 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_28 2024-04-06T10:03:54.061 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T10:03:54.061 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T10:03:54.061 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T10:03:54.061 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T10:03:54.061 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T10:03:54.061 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T10:03:54.061 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T10:03:54.061 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T10:03:54.061 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T10:03:54.061 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T10:03:54.061 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T10:03:54.061 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:03:54.062 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T10:03:54.062 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T10:03:54.062 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T10:03:54.062 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T10:03:54.062 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T10:03:54.062 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T10:03:54.062 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T10:03:54.062 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T10:03:54.062 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T10:03:54.062 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T10:03:54.062 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:03:54.062 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T10:03:54.062 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T10:03:54.062 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T10:03:54.063 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T10:03:54.063 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T10:03:54.063 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T10:03:54.063 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T10:03:54.087 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T10:03:54.087 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T10:03:54.087 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T10:03:54.087 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_28 2024-04-06T10:03:54.191 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:Invalid command: unused arguments: ['mirror/image_28'] 2024-04-06T10:03:54.191 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:rbd mirror flush : flush rbd mirror 2024-04-06T10:03:54.191 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:admin_socket: invalid command 2024-04-06T10:03:54.193 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T10:03:54.194 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/image_28 2024-04-06T10:03:54.194 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:1 2024-04-06T10:03:54.194 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T10:03:54.194 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T10:03:54.194 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-06T10:03:54.194 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:1 2024-04-06T10:03:54.194 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T10:03:54.194 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T10:03:54.194 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T10:03:54.194 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=1 2024-04-06T10:03:54.194 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-06T10:03:54.194 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T10:03:54.194 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:03:54.195 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=1 2024-04-06T10:03:54.195 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=1 2024-04-06T10:03:54.196 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-06T10:03:54.196 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:1 2024-04-06T10:03:54.196 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T10:03:54.196 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T10:03:54.196 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-06T10:03:54.197 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:1 2024-04-06T10:03:54.197 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T10:03:54.197 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T10:03:54.197 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:03:54.197 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=1 2024-04-06T10:03:54.197 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-06T10:03:54.197 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T10:03:54.197 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T10:03:54.197 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=1 2024-04-06T10:03:54.197 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=1 2024-04-06T10:03:54.197 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-06T10:03:54.219 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T10:03:54.220 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T10:03:54.220 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T10:03:54.220 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/image_28 2024-04-06T10:03:54.369 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T10:03:54.370 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror image_28 2024-04-06T10:03:54.371 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:03:54.371 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:03:54.371 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_28 2024-04-06T10:03:54.371 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror image_28 '' 2024-04-06T10:03:54.371 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:03:54.371 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:03:54.371 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_28 2024-04-06T10:03:54.371 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T10:03:54.372 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-image_28.status 2024-04-06T10:03:54.372 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-image_28.status 2024-04-06T10:03:54.372 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T10:03:54.374 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-image_28.status 2024-04-06T10:03:54.375 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_28 2024-04-06T10:03:54.375 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_28.status 2024-04-06T10:03:54.425 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 2 2024-04-06T10:03:54.425 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 2 2024-04-06T10:03:54.425 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T10:03:54.425 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=9, tag_tid=2, entry_tid=1], [object_number=8, tag_tid=2, entry_tid=0], [object_number=11, tag_tid=1, entry_tid=10683], [object_number=10, tag_tid=1, entry_tid=10682]]], state=connected] 2024-04-06T10:03:54.425 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=9, tag_tid=1, entry_tid=10685], [object_number=8, tag_tid=1, entry_tid=10684], [object_number=11, tag_tid=1, entry_tid=10683], [object_number=10, tag_tid=1, entry_tid=10682]]], state=connected] 2024-04-06T10:03:54.428 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_28.status 2024-04-06T10:03:54.430 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=9, tag_tid=2, entry_tid=1' 2024-04-06T10:03:54.431 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror image_28 2024-04-06T10:03:54.431 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:03:54.431 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:03:54.431 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_28 2024-04-06T10:03:54.431 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror image_28 '..*' 2024-04-06T10:03:54.431 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:03:54.431 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:03:54.431 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_28 2024-04-06T10:03:54.432 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T10:03:54.432 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-image_28.status 2024-04-06T10:03:54.432 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-image_28.status 2024-04-06T10:03:54.432 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T10:03:54.432 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-image_28.status 2024-04-06T10:03:54.432 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_28 2024-04-06T10:03:54.433 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_28.status 2024-04-06T10:03:54.479 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 2 2024-04-06T10:03:54.479 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 2 2024-04-06T10:03:54.479 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T10:03:54.479 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=9, tag_tid=2, entry_tid=1], [object_number=8, tag_tid=2, entry_tid=0], [object_number=11, tag_tid=1, entry_tid=10683], [object_number=10, tag_tid=1, entry_tid=10682]]], state=connected] 2024-04-06T10:03:54.479 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=9, tag_tid=1, entry_tid=10685], [object_number=8, tag_tid=1, entry_tid=10684], [object_number=11, tag_tid=1, entry_tid=10683], [object_number=10, tag_tid=1, entry_tid=10682]]], state=connected] 2024-04-06T10:03:54.482 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_28.status 2024-04-06T10:03:54.484 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=9, tag_tid=1, entry_tid=10685' 2024-04-06T10:03:54.484 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=9, tag_tid=2, entry_tid=1' -a 'object_number=9, tag_tid=2, entry_tid=1' = 'object_number=9, tag_tid=1, entry_tid=10685' 2024-04-06T10:03:54.484 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=9, tag_tid=1, entry_tid=10685' '!=' 'object_number=9, tag_tid=1, entry_tid=10685' 2024-04-06T10:03:54.484 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T10:03:54.485 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.4 2024-04-06T10:03:54.886 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror image_28 2024-04-06T10:03:54.886 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T10:03:54.886 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:03:54.886 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_28 2024-04-06T10:03:54.887 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T10:03:54.887 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n image_28 ']' 2024-04-06T10:03:54.887 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/image_28' 2024-04-06T10:03:54.887 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_28 2024-04-06T10:03:54.887 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T10:03:54.887 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T10:03:54.887 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T10:03:54.887 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T10:03:54.887 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T10:03:54.887 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T10:03:54.887 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T10:03:54.888 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T10:03:54.888 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T10:03:54.891 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T10:03:54.892 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_28 2024-04-06T10:03:54.892 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T10:03:54.892 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T10:03:54.892 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T10:03:54.892 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T10:03:54.892 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T10:03:54.892 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T10:03:54.892 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T10:03:54.892 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T10:03:54.892 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T10:03:54.892 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T10:03:54.892 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T10:03:54.892 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:03:54.892 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T10:03:54.893 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T10:03:54.893 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T10:03:54.893 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T10:03:54.893 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T10:03:54.893 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T10:03:54.893 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T10:03:54.893 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T10:03:54.893 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T10:03:54.893 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T10:03:54.893 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:03:54.893 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T10:03:54.893 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T10:03:54.893 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T10:03:54.893 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T10:03:54.893 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T10:03:54.894 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T10:03:54.894 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T10:03:54.917 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T10:03:54.917 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T10:03:54.917 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T10:03:54.917 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_28 2024-04-06T10:03:55.023 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:Invalid command: unused arguments: ['mirror/image_28'] 2024-04-06T10:03:55.023 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:rbd mirror flush : flush rbd mirror 2024-04-06T10:03:55.023 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:admin_socket: invalid command 2024-04-06T10:03:55.026 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T10:03:55.026 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/image_28 2024-04-06T10:03:55.026 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:1 2024-04-06T10:03:55.026 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T10:03:55.026 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T10:03:55.026 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-06T10:03:55.026 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:1 2024-04-06T10:03:55.027 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T10:03:55.027 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T10:03:55.027 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T10:03:55.027 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=1 2024-04-06T10:03:55.027 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-06T10:03:55.027 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T10:03:55.027 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:03:55.027 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=1 2024-04-06T10:03:55.027 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=1 2024-04-06T10:03:55.027 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-06T10:03:55.027 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:1 2024-04-06T10:03:55.027 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T10:03:55.027 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T10:03:55.027 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-06T10:03:55.027 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:1 2024-04-06T10:03:55.028 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T10:03:55.028 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T10:03:55.028 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:03:55.028 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=1 2024-04-06T10:03:55.028 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-06T10:03:55.028 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T10:03:55.028 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T10:03:55.028 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=1 2024-04-06T10:03:55.028 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=1 2024-04-06T10:03:55.028 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-06T10:03:55.052 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T10:03:55.052 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T10:03:55.052 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T10:03:55.052 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/image_28 2024-04-06T10:03:55.190 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T10:03:55.191 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror image_28 2024-04-06T10:03:55.191 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:03:55.191 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:03:55.192 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_28 2024-04-06T10:03:55.192 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror image_28 '' 2024-04-06T10:03:55.192 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:03:55.192 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:03:55.192 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_28 2024-04-06T10:03:55.192 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T10:03:55.192 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-image_28.status 2024-04-06T10:03:55.193 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-image_28.status 2024-04-06T10:03:55.193 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T10:03:55.194 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-image_28.status 2024-04-06T10:03:55.194 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_28 2024-04-06T10:03:55.194 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_28.status 2024-04-06T10:03:55.240 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 2 2024-04-06T10:03:55.240 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 2 2024-04-06T10:03:55.240 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T10:03:55.240 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=9, tag_tid=2, entry_tid=1], [object_number=8, tag_tid=2, entry_tid=0], [object_number=11, tag_tid=1, entry_tid=10683], [object_number=10, tag_tid=1, entry_tid=10682]]], state=connected] 2024-04-06T10:03:55.240 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=9, tag_tid=1, entry_tid=10685], [object_number=8, tag_tid=1, entry_tid=10684], [object_number=11, tag_tid=1, entry_tid=10683], [object_number=10, tag_tid=1, entry_tid=10682]]], state=connected] 2024-04-06T10:03:55.243 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_28.status 2024-04-06T10:03:55.244 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=9, tag_tid=2, entry_tid=1' 2024-04-06T10:03:55.245 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror image_28 2024-04-06T10:03:55.245 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:03:55.246 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:03:55.246 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_28 2024-04-06T10:03:55.246 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror image_28 '..*' 2024-04-06T10:03:55.246 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:03:55.246 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:03:55.246 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_28 2024-04-06T10:03:55.246 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T10:03:55.246 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-image_28.status 2024-04-06T10:03:55.246 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-image_28.status 2024-04-06T10:03:55.246 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T10:03:55.247 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-image_28.status 2024-04-06T10:03:55.247 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_28 2024-04-06T10:03:55.247 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_28.status 2024-04-06T10:03:55.292 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 2 2024-04-06T10:03:55.292 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 2 2024-04-06T10:03:55.293 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T10:03:55.293 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=9, tag_tid=2, entry_tid=1], [object_number=8, tag_tid=2, entry_tid=0], [object_number=11, tag_tid=1, entry_tid=10683], [object_number=10, tag_tid=1, entry_tid=10682]]], state=connected] 2024-04-06T10:03:55.293 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=9, tag_tid=1, entry_tid=10685], [object_number=8, tag_tid=1, entry_tid=10684], [object_number=11, tag_tid=1, entry_tid=10683], [object_number=10, tag_tid=1, entry_tid=10682]]], state=connected] 2024-04-06T10:03:55.296 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_28.status 2024-04-06T10:03:55.297 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=9, tag_tid=1, entry_tid=10685' 2024-04-06T10:03:55.298 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=9, tag_tid=2, entry_tid=1' -a 'object_number=9, tag_tid=2, entry_tid=1' = 'object_number=9, tag_tid=1, entry_tid=10685' 2024-04-06T10:03:55.298 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=9, tag_tid=1, entry_tid=10685' '!=' 'object_number=9, tag_tid=1, entry_tid=10685' 2024-04-06T10:03:55.298 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T10:03:55.298 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.8 2024-04-06T10:03:56.099 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror image_28 2024-04-06T10:03:56.100 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T10:03:56.100 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:03:56.100 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_28 2024-04-06T10:03:56.100 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T10:03:56.100 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n image_28 ']' 2024-04-06T10:03:56.100 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/image_28' 2024-04-06T10:03:56.100 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_28 2024-04-06T10:03:56.100 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T10:03:56.100 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T10:03:56.100 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T10:03:56.100 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T10:03:56.100 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T10:03:56.101 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T10:03:56.101 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T10:03:56.101 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T10:03:56.101 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T10:03:56.103 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T10:03:56.103 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_28 2024-04-06T10:03:56.103 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T10:03:56.103 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T10:03:56.103 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T10:03:56.103 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T10:03:56.104 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T10:03:56.104 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T10:03:56.104 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T10:03:56.104 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T10:03:56.104 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T10:03:56.104 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T10:03:56.104 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T10:03:56.104 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:03:56.104 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T10:03:56.104 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T10:03:56.105 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T10:03:56.105 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T10:03:56.105 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T10:03:56.106 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T10:03:56.106 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T10:03:56.106 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T10:03:56.106 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T10:03:56.106 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T10:03:56.106 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:03:56.106 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T10:03:56.107 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T10:03:56.107 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T10:03:56.107 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T10:03:56.107 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T10:03:56.107 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T10:03:56.107 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T10:03:56.129 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T10:03:56.130 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T10:03:56.130 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T10:03:56.130 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_28 2024-04-06T10:03:56.234 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:Invalid command: unused arguments: ['mirror/image_28'] 2024-04-06T10:03:56.234 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:rbd mirror flush : flush rbd mirror 2024-04-06T10:03:56.235 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:admin_socket: invalid command 2024-04-06T10:03:56.237 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T10:03:56.237 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/image_28 2024-04-06T10:03:56.237 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:1 2024-04-06T10:03:56.237 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T10:03:56.237 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T10:03:56.237 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-06T10:03:56.237 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:1 2024-04-06T10:03:56.237 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T10:03:56.237 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T10:03:56.237 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T10:03:56.237 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=1 2024-04-06T10:03:56.237 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-06T10:03:56.237 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T10:03:56.238 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:03:56.238 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=1 2024-04-06T10:03:56.238 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=1 2024-04-06T10:03:56.238 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-06T10:03:56.239 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:1 2024-04-06T10:03:56.239 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T10:03:56.239 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T10:03:56.239 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-06T10:03:56.239 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:1 2024-04-06T10:03:56.239 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T10:03:56.239 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T10:03:56.239 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:03:56.239 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=1 2024-04-06T10:03:56.239 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-06T10:03:56.239 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T10:03:56.239 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T10:03:56.239 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=1 2024-04-06T10:03:56.239 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=1 2024-04-06T10:03:56.240 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-06T10:03:56.258 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T10:03:56.258 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T10:03:56.258 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T10:03:56.258 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/image_28 2024-04-06T10:03:56.401 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T10:03:56.403 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror image_28 2024-04-06T10:03:56.403 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:03:56.403 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:03:56.403 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_28 2024-04-06T10:03:56.403 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror image_28 '' 2024-04-06T10:03:56.403 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:03:56.403 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:03:56.403 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_28 2024-04-06T10:03:56.403 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T10:03:56.403 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-image_28.status 2024-04-06T10:03:56.404 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T10:03:56.404 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-image_28.status 2024-04-06T10:03:56.405 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-image_28.status 2024-04-06T10:03:56.406 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_28 2024-04-06T10:03:56.406 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_28.status 2024-04-06T10:03:56.456 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 2 2024-04-06T10:03:56.456 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 2 2024-04-06T10:03:56.456 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T10:03:56.456 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=9, tag_tid=2, entry_tid=1], [object_number=8, tag_tid=2, entry_tid=0], [object_number=11, tag_tid=1, entry_tid=10683], [object_number=10, tag_tid=1, entry_tid=10682]]], state=connected] 2024-04-06T10:03:56.456 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=9, tag_tid=1, entry_tid=10685], [object_number=8, tag_tid=1, entry_tid=10684], [object_number=11, tag_tid=1, entry_tid=10683], [object_number=10, tag_tid=1, entry_tid=10682]]], state=connected] 2024-04-06T10:03:56.459 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_28.status 2024-04-06T10:03:56.461 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=9, tag_tid=2, entry_tid=1' 2024-04-06T10:03:56.462 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror image_28 2024-04-06T10:03:56.462 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:03:56.462 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:03:56.462 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_28 2024-04-06T10:03:56.462 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror image_28 '..*' 2024-04-06T10:03:56.462 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:03:56.462 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:03:56.462 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_28 2024-04-06T10:03:56.462 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T10:03:56.462 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-image_28.status 2024-04-06T10:03:56.462 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-image_28.status 2024-04-06T10:03:56.463 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T10:03:56.463 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-image_28.status 2024-04-06T10:03:56.463 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_28 2024-04-06T10:03:56.463 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_28.status 2024-04-06T10:03:56.712 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 2 2024-04-06T10:03:56.712 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 2 2024-04-06T10:03:56.712 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T10:03:56.712 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=9, tag_tid=2, entry_tid=1], [object_number=8, tag_tid=2, entry_tid=0], [object_number=11, tag_tid=1, entry_tid=10683], [object_number=10, tag_tid=1, entry_tid=10682]]], state=connected] 2024-04-06T10:03:56.713 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=9, tag_tid=1, entry_tid=10685], [object_number=8, tag_tid=1, entry_tid=10684], [object_number=11, tag_tid=1, entry_tid=10683], [object_number=10, tag_tid=1, entry_tid=10682]]], state=connected] 2024-04-06T10:03:56.719 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_28.status 2024-04-06T10:03:56.721 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=9, tag_tid=1, entry_tid=10685' 2024-04-06T10:03:56.722 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=9, tag_tid=2, entry_tid=1' -a 'object_number=9, tag_tid=2, entry_tid=1' = 'object_number=9, tag_tid=1, entry_tid=10685' 2024-04-06T10:03:56.722 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=9, tag_tid=1, entry_tid=10685' '!=' 'object_number=9, tag_tid=1, entry_tid=10685' 2024-04-06T10:03:56.722 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T10:03:56.722 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 1.6 2024-04-06T10:03:58.324 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror image_28 2024-04-06T10:03:58.324 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T10:03:58.324 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:03:58.324 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_28 2024-04-06T10:03:58.324 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T10:03:58.324 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n image_28 ']' 2024-04-06T10:03:58.324 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/image_28' 2024-04-06T10:03:58.324 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_28 2024-04-06T10:03:58.325 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T10:03:58.325 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T10:03:58.325 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T10:03:58.325 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T10:03:58.325 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T10:03:58.325 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T10:03:58.325 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T10:03:58.325 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T10:03:58.325 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T10:03:58.328 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T10:03:58.328 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_28 2024-04-06T10:03:58.328 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T10:03:58.328 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T10:03:58.328 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T10:03:58.328 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T10:03:58.328 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T10:03:58.328 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T10:03:58.328 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T10:03:58.329 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T10:03:58.329 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T10:03:58.329 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T10:03:58.329 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T10:03:58.329 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:03:58.329 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T10:03:58.329 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T10:03:58.329 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T10:03:58.330 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T10:03:58.331 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T10:03:58.331 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T10:03:58.331 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T10:03:58.331 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T10:03:58.331 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T10:03:58.331 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T10:03:58.331 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:03:58.331 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T10:03:58.331 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T10:03:58.332 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T10:03:58.332 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T10:03:58.332 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T10:03:58.332 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T10:03:58.332 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T10:03:58.354 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T10:03:58.355 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T10:03:58.355 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T10:03:58.355 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_28 2024-04-06T10:03:58.462 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:Invalid command: unused arguments: ['mirror/image_28'] 2024-04-06T10:03:58.462 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:rbd mirror flush : flush rbd mirror 2024-04-06T10:03:58.462 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:admin_socket: invalid command 2024-04-06T10:03:58.466 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T10:03:58.466 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/image_28 2024-04-06T10:03:58.466 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:1 2024-04-06T10:03:58.466 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T10:03:58.466 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T10:03:58.466 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-06T10:03:58.466 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:1 2024-04-06T10:03:58.466 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T10:03:58.466 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T10:03:58.466 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T10:03:58.466 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=1 2024-04-06T10:03:58.466 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-06T10:03:58.467 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T10:03:58.467 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:03:58.467 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=1 2024-04-06T10:03:58.467 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=1 2024-04-06T10:03:58.467 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-06T10:03:58.467 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:1 2024-04-06T10:03:58.467 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T10:03:58.467 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T10:03:58.467 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-06T10:03:58.467 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:1 2024-04-06T10:03:58.467 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T10:03:58.467 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T10:03:58.467 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:03:58.467 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=1 2024-04-06T10:03:58.467 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-06T10:03:58.468 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T10:03:58.468 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T10:03:58.468 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=1 2024-04-06T10:03:58.468 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=1 2024-04-06T10:03:58.468 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-06T10:03:58.491 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T10:03:58.491 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T10:03:58.491 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T10:03:58.491 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/image_28 2024-04-06T10:03:58.631 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T10:03:58.632 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror image_28 2024-04-06T10:03:58.632 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:03:58.632 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:03:58.633 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_28 2024-04-06T10:03:58.633 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror image_28 '' 2024-04-06T10:03:58.633 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:03:58.633 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:03:58.633 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_28 2024-04-06T10:03:58.633 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T10:03:58.633 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-image_28.status 2024-04-06T10:03:58.633 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-image_28.status 2024-04-06T10:03:58.634 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T10:03:58.634 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-image_28.status 2024-04-06T10:03:58.635 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_28 2024-04-06T10:03:58.635 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_28.status 2024-04-06T10:03:58.682 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 2 2024-04-06T10:03:58.682 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 2 2024-04-06T10:03:58.682 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T10:03:58.682 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=9, tag_tid=2, entry_tid=1], [object_number=8, tag_tid=2, entry_tid=0], [object_number=11, tag_tid=1, entry_tid=10683], [object_number=10, tag_tid=1, entry_tid=10682]]], state=connected] 2024-04-06T10:03:58.682 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=9, tag_tid=1, entry_tid=10685], [object_number=8, tag_tid=1, entry_tid=10684], [object_number=11, tag_tid=1, entry_tid=10683], [object_number=10, tag_tid=1, entry_tid=10682]]], state=connected] 2024-04-06T10:03:58.686 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_28.status 2024-04-06T10:03:58.687 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=9, tag_tid=2, entry_tid=1' 2024-04-06T10:03:58.688 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror image_28 2024-04-06T10:03:58.688 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:03:58.688 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:03:58.688 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_28 2024-04-06T10:03:58.688 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror image_28 '..*' 2024-04-06T10:03:58.689 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:03:58.689 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:03:58.689 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_28 2024-04-06T10:03:58.689 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T10:03:58.689 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-image_28.status 2024-04-06T10:03:58.689 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-image_28.status 2024-04-06T10:03:58.689 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T10:03:58.690 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-image_28.status 2024-04-06T10:03:58.690 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_28 2024-04-06T10:03:58.690 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_28.status 2024-04-06T10:03:58.734 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 2 2024-04-06T10:03:58.734 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 2 2024-04-06T10:03:58.734 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T10:03:58.734 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=9, tag_tid=2, entry_tid=1], [object_number=8, tag_tid=2, entry_tid=0], [object_number=11, tag_tid=1, entry_tid=10683], [object_number=10, tag_tid=1, entry_tid=10682]]], state=connected] 2024-04-06T10:03:58.735 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=9, tag_tid=1, entry_tid=10685], [object_number=8, tag_tid=1, entry_tid=10684], [object_number=11, tag_tid=1, entry_tid=10683], [object_number=10, tag_tid=1, entry_tid=10682]]], state=connected] 2024-04-06T10:03:58.738 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_28.status 2024-04-06T10:03:58.740 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=9, tag_tid=1, entry_tid=10685' 2024-04-06T10:03:58.740 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=9, tag_tid=2, entry_tid=1' -a 'object_number=9, tag_tid=2, entry_tid=1' = 'object_number=9, tag_tid=1, entry_tid=10685' 2024-04-06T10:03:58.740 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=9, tag_tid=1, entry_tid=10685' '!=' 'object_number=9, tag_tid=1, entry_tid=10685' 2024-04-06T10:03:58.740 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T10:03:58.740 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 2 2024-04-06T10:04:00.742 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror image_28 2024-04-06T10:04:00.742 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T10:04:00.742 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:04:00.742 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_28 2024-04-06T10:04:00.742 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T10:04:00.742 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n image_28 ']' 2024-04-06T10:04:00.743 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/image_28' 2024-04-06T10:04:00.743 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_28 2024-04-06T10:04:00.743 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T10:04:00.743 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T10:04:00.743 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T10:04:00.743 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T10:04:00.743 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T10:04:00.743 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T10:04:00.743 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T10:04:00.743 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T10:04:00.744 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T10:04:00.748 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T10:04:00.748 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_28 2024-04-06T10:04:00.748 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T10:04:00.749 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T10:04:00.749 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T10:04:00.749 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T10:04:00.749 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T10:04:00.749 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T10:04:00.749 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T10:04:00.749 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T10:04:00.750 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T10:04:00.750 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T10:04:00.750 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T10:04:00.750 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:04:00.750 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T10:04:00.750 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T10:04:00.750 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T10:04:00.751 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T10:04:00.751 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T10:04:00.751 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T10:04:00.751 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T10:04:00.751 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T10:04:00.751 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T10:04:00.751 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T10:04:00.751 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:04:00.752 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T10:04:00.752 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T10:04:00.752 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T10:04:00.752 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T10:04:00.752 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T10:04:00.752 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T10:04:00.752 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T10:04:00.774 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T10:04:00.774 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T10:04:00.774 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T10:04:00.775 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_28 2024-04-06T10:04:00.879 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:Invalid command: unused arguments: ['mirror/image_28'] 2024-04-06T10:04:00.879 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:rbd mirror flush : flush rbd mirror 2024-04-06T10:04:00.879 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:admin_socket: invalid command 2024-04-06T10:04:00.882 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T10:04:00.882 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/image_28 2024-04-06T10:04:00.882 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:1 2024-04-06T10:04:00.882 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T10:04:00.882 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T10:04:00.882 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-06T10:04:00.882 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:1 2024-04-06T10:04:00.882 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T10:04:00.882 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T10:04:00.882 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T10:04:00.882 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=1 2024-04-06T10:04:00.883 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-06T10:04:00.883 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T10:04:00.883 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:04:00.883 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=1 2024-04-06T10:04:00.883 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=1 2024-04-06T10:04:00.884 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-06T10:04:00.884 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:1 2024-04-06T10:04:00.884 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T10:04:00.884 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T10:04:00.884 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-06T10:04:00.884 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:1 2024-04-06T10:04:00.884 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T10:04:00.885 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T10:04:00.885 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:04:00.885 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=1 2024-04-06T10:04:00.885 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-06T10:04:00.885 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T10:04:00.885 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T10:04:00.885 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=1 2024-04-06T10:04:00.885 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=1 2024-04-06T10:04:00.885 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-06T10:04:00.903 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T10:04:00.904 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T10:04:00.904 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T10:04:00.904 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/image_28 2024-04-06T10:04:01.059 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T10:04:01.061 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror image_28 2024-04-06T10:04:01.061 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:04:01.061 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:04:01.061 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_28 2024-04-06T10:04:01.062 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror image_28 '' 2024-04-06T10:04:01.062 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:04:01.062 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:04:01.062 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_28 2024-04-06T10:04:01.062 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T10:04:01.062 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-image_28.status 2024-04-06T10:04:01.063 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T10:04:01.063 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-image_28.status 2024-04-06T10:04:01.064 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-image_28.status 2024-04-06T10:04:01.064 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_28 2024-04-06T10:04:01.065 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_28.status 2024-04-06T10:04:01.118 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 2 2024-04-06T10:04:01.118 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 2 2024-04-06T10:04:01.118 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T10:04:01.118 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=9, tag_tid=2, entry_tid=1], [object_number=8, tag_tid=2, entry_tid=0], [object_number=11, tag_tid=1, entry_tid=10683], [object_number=10, tag_tid=1, entry_tid=10682]]], state=connected] 2024-04-06T10:04:01.119 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=9, tag_tid=2, entry_tid=1], [object_number=8, tag_tid=2, entry_tid=0], [object_number=11, tag_tid=1, entry_tid=10683], [object_number=10, tag_tid=1, entry_tid=10682]]], state=connected] 2024-04-06T10:04:01.122 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_28.status 2024-04-06T10:04:01.123 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=9, tag_tid=2, entry_tid=1' 2024-04-06T10:04:01.124 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror image_28 2024-04-06T10:04:01.125 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:04:01.125 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:04:01.125 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_28 2024-04-06T10:04:01.125 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror image_28 '..*' 2024-04-06T10:04:01.125 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:04:01.125 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:04:01.125 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_28 2024-04-06T10:04:01.125 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T10:04:01.125 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-image_28.status 2024-04-06T10:04:01.125 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-image_28.status 2024-04-06T10:04:01.125 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T10:04:01.126 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-image_28.status 2024-04-06T10:04:01.126 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_28 2024-04-06T10:04:01.126 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_28.status 2024-04-06T10:04:01.377 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 2 2024-04-06T10:04:01.377 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 2 2024-04-06T10:04:01.377 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T10:04:01.377 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=9, tag_tid=2, entry_tid=1], [object_number=8, tag_tid=2, entry_tid=0], [object_number=11, tag_tid=1, entry_tid=10683], [object_number=10, tag_tid=1, entry_tid=10682]]], state=connected] 2024-04-06T10:04:01.377 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=9, tag_tid=2, entry_tid=1], [object_number=8, tag_tid=2, entry_tid=0], [object_number=11, tag_tid=1, entry_tid=10683], [object_number=10, tag_tid=1, entry_tid=10682]]], state=connected] 2024-04-06T10:04:01.381 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_28.status 2024-04-06T10:04:01.383 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=9, tag_tid=2, entry_tid=1' 2024-04-06T10:04:01.383 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=9, tag_tid=2, entry_tid=1' -a 'object_number=9, tag_tid=2, entry_tid=1' = 'object_number=9, tag_tid=2, entry_tid=1' 2024-04-06T10:04:01.383 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T10:04:01.383 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_snap_present cluster1 mirror image_28 snap 2024-04-06T10:04:01.383 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T10:04:01.384 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:04:01.384 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_28 2024-04-06T10:04:01.384 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local snap_name=snap 2024-04-06T10:04:01.384 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local s 2024-04-06T10:04:01.384 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-04-06T10:04:01.384 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 1 2024-04-06T10:04:02.384 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 info mirror/image_28@snap 2024-04-06T10:04:02.448 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:rbd image 'image_28': 2024-04-06T10:04:02.448 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: size 128 MiB in 32 objects 2024-04-06T10:04:02.449 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: order 22 (4 MiB objects) 2024-04-06T10:04:02.449 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: snapshot_count: 1 2024-04-06T10:04:02.449 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: id: 10ac897b159b 2024-04-06T10:04:02.449 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: block_name_prefix: rbd_data.10ac897b159b 2024-04-06T10:04:02.449 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: format: 2 2024-04-06T10:04:02.449 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: features: layering, exclusive-lock, journaling 2024-04-06T10:04:02.449 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: op_features: 2024-04-06T10:04:02.449 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: flags: 2024-04-06T10:04:02.449 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: create_timestamp: Sat Apr 6 09:54:53 2024 2024-04-06T10:04:02.449 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: access_timestamp: Sat Apr 6 09:54:53 2024 2024-04-06T10:04:02.449 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: modify_timestamp: Sat Apr 6 09:55:54 2024 2024-04-06T10:04:02.449 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: protected: False 2024-04-06T10:04:02.450 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: journal: 10ac897b159b 2024-04-06T10:04:02.450 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: mirroring state: enabled 2024-04-06T10:04:02.450 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: mirroring mode: journal 2024-04-06T10:04:02.450 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: mirroring global id: 70065c36-9d55-4028-aa8b-efafed9ec495 2024-04-06T10:04:02.450 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: mirroring primary: false 2024-04-06T10:04:02.453 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T10:04:02.453 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ compare_image_snaps mirror image_28 snap 2024-04-06T10:04:02.453 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:04:02.453 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_28 2024-04-06T10:04:02.453 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local snap_name=snap 2024-04-06T10:04:02.453 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local ret=0 2024-04-06T10:04:02.453 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local rmt_export=/tmp/tmp.BhW6xKmROU/cluster2-mirror-image_28.export 2024-04-06T10:04:02.453 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loc_export=/tmp/tmp.BhW6xKmROU/cluster1-mirror-image_28.export 2024-04-06T10:04:02.453 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rm -f /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_28.export /tmp/tmp.BhW6xKmROU/cluster1-mirror-image_28.export 2024-04-06T10:04:02.454 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster2 -p mirror export image_28@snap /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_28.export 2024-04-06T10:04:02.627 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-04-06T10:04:02.643 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 -p mirror export image_28@snap /tmp/tmp.BhW6xKmROU/cluster1-mirror-image_28.export 2024-04-06T10:04:02.951 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T10:04:02.948+0000 7f2e84d66640 -1 reset not still connected to 0x560d27567380 2024-04-06T10:04:02.954 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T10:04:02.952+0000 7f2e84d66640 -1 reset not still connected to 0x560d27c3edd0 2024-04-06T10:04:02.962 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T10:04:02.960+0000 7f2e84d66640 -1 reset not still connected to 0x560d27c3f2b0 2024-04-06T10:04:02.962 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T10:04:02.960+0000 7f2e84d66640 -1 reset not still connected to 0x560d29d5b520 2024-04-06T10:04:02.962 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T10:04:02.960+0000 7f2e84d66640 -1 reset not still connected to 0x560d2a17d450 2024-04-06T10:04:02.963 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T10:04:02.960+0000 7f2e84d66640 -1 reset not still connected to 0x560d2a17d6c0 2024-04-06T10:04:02.963 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T10:04:02.960+0000 7f2e84d66640 -1 reset not still connected to 0x560d2a2336c0 2024-04-06T10:04:02.963 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T10:04:02.960+0000 7f2e84d66640 -1 reset not still connected to 0x560d2a5705b0 2024-04-06T10:04:03.012 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-04-06T10:04:03.028 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmp /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_28.export /tmp/tmp.BhW6xKmROU/cluster1-mirror-image_28.export 2024-04-06T10:04:03.109 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rm -f /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_28.export /tmp/tmp.BhW6xKmROU/cluster1-mirror-image_28.export 2024-04-06T10:04:03.151 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T10:04:03.151 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-06T10:04:03.151 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ image=image_29 2024-04-06T10:04:03.151 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ create_snap cluster2 mirror image_29 snap 2024-04-06T10:04:03.151 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-04-06T10:04:03.151 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:04:03.151 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_29 2024-04-06T10:04:03.151 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local snap_name=snap 2024-04-06T10:04:03.151 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster2 -p mirror snap create image_29@snap --debug-rbd=20 --debug-journaler=20 2024-04-06T10:04:06.882 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_image_replay_started cluster1 mirror image_29 2024-04-06T10:04:06.882 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T10:04:06.882 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:04:06.882 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_29 2024-04-06T10:04:06.882 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_image_replay_state cluster1 mirror image_29 started 2024-04-06T10:04:06.882 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T10:04:06.882 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:04:06.882 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_29 2024-04-06T10:04:06.882 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state=started 2024-04-06T10:04:06.882 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local s 2024-04-06T10:04:06.882 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T10:04:06.883 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 1 2024-04-06T10:04:07.884 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test_image_replay_state cluster1 mirror image_29 started 2024-04-06T10:04:07.884 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T10:04:07.884 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:04:07.884 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_29 2024-04-06T10:04:07.884 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local test_state=started 2024-04-06T10:04:07.884 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local status_result 2024-04-06T10:04:07.884 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local current_state=stopped 2024-04-06T10:04:07.885 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_29 2024-04-06T10:04:07.885 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -i state 2024-04-06T10:04:07.885 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_instance=cluster1 2024-04-06T10:04:07.886 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ shift 2024-04-06T10:04:07.886 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T10:04:07.886 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance=cluster1 2024-04-06T10:04:07.886 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local loop_instance 2024-04-06T10:04:07.886 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T10:04:07.886 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sleep 0 2024-04-06T10:04:07.886 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-04-06T10:04:07.887 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ seq 0 3 2024-04-06T10:04:07.890 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T10:04:07.890 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_29 2024-04-06T10:04:07.890 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1:0 2024-04-06T10:04:07.890 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ shift 2024-04-06T10:04:07.890 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T10:04:07.890 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-04-06T10:04:07.890 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T10:04:07.890 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=cluster 2024-04-06T10:04:07.890 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T10:04:07.890 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:04:07.891 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T10:04:07.891 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T10:04:07.891 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval cluster=cluster1 2024-04-06T10:04:07.891 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ cluster=cluster1 2024-04-06T10:04:07.891 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T10:04:07.891 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T10:04:07.891 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-04-06T10:04:07.891 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local local_cluster=cluster1:0 2024-04-06T10:04:07.891 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local cluster=cluster1 2024-04-06T10:04:07.891 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local instance 2024-04-06T10:04:07.891 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T10:04:07.891 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local val=cluster1:0 2024-04-06T10:04:07.891 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local cluster_var_name=local_cluster 2024-04-06T10:04:07.891 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local instance_var_name=instance 2024-04-06T10:04:07.891 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ cluster=cluster1 2024-04-06T10:04:07.892 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T10:04:07.892 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ '[' 0 = cluster1:0 ']' 2024-04-06T10:04:07.892 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ eval local_cluster=cluster1 2024-04-06T10:04:07.892 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++++ local_cluster=cluster1 2024-04-06T10:04:07.892 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ eval instance=0 2024-04-06T10:04:07.892 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++++ instance=0 2024-04-06T10:04:07.892 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T10:04:07.914 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T10:04:07.915 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T10:04:07.915 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T10:04:07.915 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_29 2024-04-06T10:04:08.025 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:Invalid command: unused arguments: ['mirror/image_29'] 2024-04-06T10:04:08.025 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:rbd mirror status : get status for rbd mirror 2024-04-06T10:04:08.025 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:admin_socket: invalid command 2024-04-06T10:04:08.029 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T10:04:08.029 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_29 2024-04-06T10:04:08.029 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1:1 2024-04-06T10:04:08.029 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ shift 2024-04-06T10:04:08.029 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T10:04:08.029 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-04-06T10:04:08.030 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:1 2024-04-06T10:04:08.030 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=cluster 2024-04-06T10:04:08.030 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T10:04:08.030 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:04:08.030 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=1 2024-04-06T10:04:08.030 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-06T10:04:08.030 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval cluster=cluster1 2024-04-06T10:04:08.030 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ cluster=cluster1 2024-04-06T10:04:08.030 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=1 2024-04-06T10:04:08.031 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=1 2024-04-06T10:04:08.031 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-04-06T10:04:08.031 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local local_cluster=cluster1:1 2024-04-06T10:04:08.031 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local cluster=cluster1 2024-04-06T10:04:08.031 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local instance 2024-04-06T10:04:08.031 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-06T10:04:08.031 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local val=cluster1:1 2024-04-06T10:04:08.031 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local cluster_var_name=local_cluster 2024-04-06T10:04:08.031 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local instance_var_name=instance 2024-04-06T10:04:08.031 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ cluster=cluster1 2024-04-06T10:04:08.031 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=1 2024-04-06T10:04:08.031 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ '[' 1 = cluster1:1 ']' 2024-04-06T10:04:08.031 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ eval local_cluster=cluster1 2024-04-06T10:04:08.032 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++++ local_cluster=cluster1 2024-04-06T10:04:08.032 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ eval instance=1 2024-04-06T10:04:08.032 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++++ instance=1 2024-04-06T10:04:08.032 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-06T10:04:08.051 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T10:04:08.052 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T10:04:08.052 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T10:04:08.052 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_29 2024-04-06T10:04:08.202 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ return 0 2024-04-06T10:04:08.203 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ status_result=' "state": "Replaying"' 2024-04-06T10:04:08.203 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ echo ' "state": "Replaying"' 2024-04-06T10:04:08.204 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep -i Replaying 2024-04-06T10:04:08.205 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: "state": "Replaying" 2024-04-06T10:04:08.207 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ current_state=started 2024-04-06T10:04:08.207 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test started = started 2024-04-06T10:04:08.207 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T10:04:08.207 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_29 2024-04-06T10:04:08.207 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local local_cluster=cluster1 2024-04-06T10:04:08.207 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-04-06T10:04:08.207 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:04:08.207 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_29 2024-04-06T10:04:08.208 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' journal = journal ']' 2024-04-06T10:04:08.208 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_journal_replay_complete cluster1 cluster2 mirror image_29 2024-04-06T10:04:08.208 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local local_cluster=cluster1 2024-04-06T10:04:08.208 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-04-06T10:04:08.208 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:04:08.208 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_29 2024-04-06T10:04:08.208 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local s master_pos mirror_pos last_mirror_pos 2024-04-06T10:04:08.209 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local master_tag master_entry mirror_tag mirror_entry 2024-04-06T10:04:08.209 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T10:04:08.209 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T10:04:08.209 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T10:04:08.408 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror image_29 2024-04-06T10:04:08.408 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T10:04:08.408 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:04:08.408 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_29 2024-04-06T10:04:08.408 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T10:04:08.408 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n image_29 ']' 2024-04-06T10:04:08.408 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/image_29' 2024-04-06T10:04:08.408 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_29 2024-04-06T10:04:08.408 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T10:04:08.409 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T10:04:08.409 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T10:04:08.409 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T10:04:08.409 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T10:04:08.409 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T10:04:08.409 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T10:04:08.409 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T10:04:08.410 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T10:04:08.412 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T10:04:08.412 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_29 2024-04-06T10:04:08.413 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T10:04:08.413 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T10:04:08.413 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T10:04:08.413 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T10:04:08.413 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T10:04:08.413 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T10:04:08.413 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T10:04:08.413 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T10:04:08.413 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T10:04:08.413 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T10:04:08.413 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T10:04:08.413 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:04:08.413 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T10:04:08.414 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T10:04:08.414 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T10:04:08.414 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T10:04:08.414 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T10:04:08.414 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T10:04:08.414 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T10:04:08.414 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T10:04:08.414 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T10:04:08.414 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T10:04:08.414 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:04:08.414 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T10:04:08.414 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T10:04:08.414 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T10:04:08.415 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T10:04:08.415 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T10:04:08.415 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T10:04:08.415 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T10:04:08.438 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T10:04:08.438 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T10:04:08.438 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T10:04:08.438 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_29 2024-04-06T10:04:08.545 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:Invalid command: unused arguments: ['mirror/image_29'] 2024-04-06T10:04:08.545 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:rbd mirror flush : flush rbd mirror 2024-04-06T10:04:08.545 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:admin_socket: invalid command 2024-04-06T10:04:08.549 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T10:04:08.549 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/image_29 2024-04-06T10:04:08.549 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:1 2024-04-06T10:04:08.549 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T10:04:08.549 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T10:04:08.549 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-06T10:04:08.550 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:1 2024-04-06T10:04:08.550 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T10:04:08.550 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T10:04:08.550 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T10:04:08.550 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=1 2024-04-06T10:04:08.550 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-06T10:04:08.550 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T10:04:08.550 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:04:08.550 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=1 2024-04-06T10:04:08.550 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=1 2024-04-06T10:04:08.550 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-06T10:04:08.550 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:1 2024-04-06T10:04:08.550 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T10:04:08.550 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T10:04:08.551 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-06T10:04:08.551 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:1 2024-04-06T10:04:08.551 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T10:04:08.551 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T10:04:08.551 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:04:08.551 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=1 2024-04-06T10:04:08.551 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-06T10:04:08.551 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T10:04:08.551 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T10:04:08.551 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=1 2024-04-06T10:04:08.551 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=1 2024-04-06T10:04:08.551 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-06T10:04:08.573 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T10:04:08.573 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T10:04:08.573 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T10:04:08.573 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/image_29 2024-04-06T10:04:08.719 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T10:04:08.719 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror image_29 2024-04-06T10:04:08.719 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:04:08.719 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:04:08.720 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_29 2024-04-06T10:04:08.720 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror image_29 '' 2024-04-06T10:04:08.720 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:04:08.720 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:04:08.720 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_29 2024-04-06T10:04:08.720 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T10:04:08.720 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-image_29.status 2024-04-06T10:04:08.720 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-image_29.status 2024-04-06T10:04:08.721 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T10:04:08.722 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-image_29.status 2024-04-06T10:04:08.723 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_29 2024-04-06T10:04:08.723 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_29.status 2024-04-06T10:04:08.767 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 2 2024-04-06T10:04:08.768 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 2 2024-04-06T10:04:08.768 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T10:04:08.768 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=9, tag_tid=2, entry_tid=1], [object_number=8, tag_tid=2, entry_tid=0], [object_number=11, tag_tid=1, entry_tid=12599], [object_number=10, tag_tid=1, entry_tid=12598]]], state=connected] 2024-04-06T10:04:08.768 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=8, tag_tid=1, entry_tid=12600], [object_number=11, tag_tid=1, entry_tid=12599], [object_number=10, tag_tid=1, entry_tid=12598], [object_number=9, tag_tid=1, entry_tid=12597]]], state=connected] 2024-04-06T10:04:08.771 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_29.status 2024-04-06T10:04:08.773 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=9, tag_tid=2, entry_tid=1' 2024-04-06T10:04:08.774 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror image_29 2024-04-06T10:04:08.774 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:04:08.774 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:04:08.774 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_29 2024-04-06T10:04:08.774 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror image_29 '..*' 2024-04-06T10:04:08.774 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:04:08.774 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:04:08.774 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_29 2024-04-06T10:04:08.774 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T10:04:08.774 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-image_29.status 2024-04-06T10:04:08.775 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-image_29.status 2024-04-06T10:04:08.775 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T10:04:08.775 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-image_29.status 2024-04-06T10:04:08.775 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_29 2024-04-06T10:04:08.775 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_29.status 2024-04-06T10:04:08.820 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 2 2024-04-06T10:04:08.820 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 2 2024-04-06T10:04:08.820 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T10:04:08.821 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=9, tag_tid=2, entry_tid=1], [object_number=8, tag_tid=2, entry_tid=0], [object_number=11, tag_tid=1, entry_tid=12599], [object_number=10, tag_tid=1, entry_tid=12598]]], state=connected] 2024-04-06T10:04:08.821 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=8, tag_tid=1, entry_tid=12600], [object_number=11, tag_tid=1, entry_tid=12599], [object_number=10, tag_tid=1, entry_tid=12598], [object_number=9, tag_tid=1, entry_tid=12597]]], state=connected] 2024-04-06T10:04:08.823 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_29.status 2024-04-06T10:04:08.826 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=8, tag_tid=1, entry_tid=12600' 2024-04-06T10:04:08.826 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=9, tag_tid=2, entry_tid=1' -a 'object_number=9, tag_tid=2, entry_tid=1' = 'object_number=8, tag_tid=1, entry_tid=12600' 2024-04-06T10:04:08.826 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=8, tag_tid=1, entry_tid=12600' '!=' '' 2024-04-06T10:04:08.826 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T10:04:08.826 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=8, tag_tid=1, entry_tid=12600' = '' 2024-04-06T10:04:08.826 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=8, tag_tid=1, entry_tid=12600' 2024-04-06T10:04:08.826 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=9, tag_tid=2, entry_tid=1' 2024-04-06T10:04:08.826 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T10:04:08.826 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T10:04:08.827 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=2 2024-04-06T10:04:08.827 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=8, tag_tid=1, entry_tid=12600' 2024-04-06T10:04:08.827 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T10:04:08.828 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T10:04:08.828 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=1 2024-04-06T10:04:08.829 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=9, tag_tid=2, entry_tid=1' 2024-04-06T10:04:08.829 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T10:04:08.829 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T10:04:08.830 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T10:04:08.832 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=8, tag_tid=1, entry_tid=12600' 2024-04-06T10:04:08.832 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T10:04:08.832 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T10:04:08.832 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=12600 2024-04-06T10:04:08.832 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 2 = 1 -a 1 -le 12600 2024-04-06T10:04:08.832 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T10:04:08.832 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T10:04:08.833 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T10:04:09.034 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror image_29 2024-04-06T10:04:09.034 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T10:04:09.034 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:04:09.034 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_29 2024-04-06T10:04:09.034 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T10:04:09.034 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n image_29 ']' 2024-04-06T10:04:09.035 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/image_29' 2024-04-06T10:04:09.035 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_29 2024-04-06T10:04:09.035 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T10:04:09.035 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T10:04:09.035 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T10:04:09.035 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T10:04:09.035 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T10:04:09.035 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T10:04:09.035 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T10:04:09.035 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T10:04:09.036 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T10:04:09.038 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T10:04:09.038 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_29 2024-04-06T10:04:09.038 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T10:04:09.038 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T10:04:09.038 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T10:04:09.039 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T10:04:09.039 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T10:04:09.039 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T10:04:09.039 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T10:04:09.039 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T10:04:09.039 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T10:04:09.039 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T10:04:09.039 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T10:04:09.039 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:04:09.039 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T10:04:09.039 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T10:04:09.041 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T10:04:09.041 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T10:04:09.041 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T10:04:09.041 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T10:04:09.041 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T10:04:09.041 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T10:04:09.041 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T10:04:09.041 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T10:04:09.041 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:04:09.041 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T10:04:09.041 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T10:04:09.041 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T10:04:09.041 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T10:04:09.042 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T10:04:09.042 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T10:04:09.042 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T10:04:09.065 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T10:04:09.065 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T10:04:09.065 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T10:04:09.065 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_29 2024-04-06T10:04:09.168 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:Invalid command: unused arguments: ['mirror/image_29'] 2024-04-06T10:04:09.169 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:rbd mirror flush : flush rbd mirror 2024-04-06T10:04:09.169 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:admin_socket: invalid command 2024-04-06T10:04:09.171 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T10:04:09.171 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/image_29 2024-04-06T10:04:09.171 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:1 2024-04-06T10:04:09.171 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T10:04:09.171 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T10:04:09.171 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-06T10:04:09.171 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:1 2024-04-06T10:04:09.172 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T10:04:09.172 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T10:04:09.172 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T10:04:09.172 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=1 2024-04-06T10:04:09.172 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-06T10:04:09.172 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T10:04:09.172 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:04:09.172 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=1 2024-04-06T10:04:09.172 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=1 2024-04-06T10:04:09.173 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-06T10:04:09.174 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:1 2024-04-06T10:04:09.174 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T10:04:09.174 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T10:04:09.174 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-06T10:04:09.174 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:1 2024-04-06T10:04:09.174 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T10:04:09.174 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T10:04:09.174 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:04:09.174 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=1 2024-04-06T10:04:09.174 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-06T10:04:09.174 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T10:04:09.174 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T10:04:09.174 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=1 2024-04-06T10:04:09.174 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=1 2024-04-06T10:04:09.175 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-06T10:04:09.196 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T10:04:09.196 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T10:04:09.197 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T10:04:09.197 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/image_29 2024-04-06T10:04:09.336 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T10:04:09.337 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror image_29 2024-04-06T10:04:09.337 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:04:09.337 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:04:09.337 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_29 2024-04-06T10:04:09.337 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror image_29 '' 2024-04-06T10:04:09.337 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:04:09.337 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:04:09.337 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_29 2024-04-06T10:04:09.337 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T10:04:09.338 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-image_29.status 2024-04-06T10:04:09.338 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-image_29.status 2024-04-06T10:04:09.338 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T10:04:09.340 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-image_29.status 2024-04-06T10:04:09.341 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_29 2024-04-06T10:04:09.341 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_29.status 2024-04-06T10:04:09.391 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 2 2024-04-06T10:04:09.391 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 2 2024-04-06T10:04:09.391 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T10:04:09.391 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=9, tag_tid=2, entry_tid=1], [object_number=8, tag_tid=2, entry_tid=0], [object_number=11, tag_tid=1, entry_tid=12599], [object_number=10, tag_tid=1, entry_tid=12598]]], state=connected] 2024-04-06T10:04:09.391 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=8, tag_tid=1, entry_tid=12600], [object_number=11, tag_tid=1, entry_tid=12599], [object_number=10, tag_tid=1, entry_tid=12598], [object_number=9, tag_tid=1, entry_tid=12597]]], state=connected] 2024-04-06T10:04:09.395 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_29.status 2024-04-06T10:04:09.396 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=9, tag_tid=2, entry_tid=1' 2024-04-06T10:04:09.397 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror image_29 2024-04-06T10:04:09.397 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:04:09.397 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:04:09.397 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_29 2024-04-06T10:04:09.397 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror image_29 '..*' 2024-04-06T10:04:09.397 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:04:09.398 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:04:09.398 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_29 2024-04-06T10:04:09.398 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T10:04:09.398 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-image_29.status 2024-04-06T10:04:09.398 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-image_29.status 2024-04-06T10:04:09.398 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T10:04:09.398 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-image_29.status 2024-04-06T10:04:09.399 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_29 2024-04-06T10:04:09.399 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_29.status 2024-04-06T10:04:09.444 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 2 2024-04-06T10:04:09.444 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 2 2024-04-06T10:04:09.444 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T10:04:09.444 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=9, tag_tid=2, entry_tid=1], [object_number=8, tag_tid=2, entry_tid=0], [object_number=11, tag_tid=1, entry_tid=12599], [object_number=10, tag_tid=1, entry_tid=12598]]], state=connected] 2024-04-06T10:04:09.445 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=8, tag_tid=1, entry_tid=12600], [object_number=11, tag_tid=1, entry_tid=12599], [object_number=10, tag_tid=1, entry_tid=12598], [object_number=9, tag_tid=1, entry_tid=12597]]], state=connected] 2024-04-06T10:04:09.448 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_29.status 2024-04-06T10:04:09.450 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=8, tag_tid=1, entry_tid=12600' 2024-04-06T10:04:09.450 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=9, tag_tid=2, entry_tid=1' -a 'object_number=9, tag_tid=2, entry_tid=1' = 'object_number=8, tag_tid=1, entry_tid=12600' 2024-04-06T10:04:09.450 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=8, tag_tid=1, entry_tid=12600' '!=' 'object_number=8, tag_tid=1, entry_tid=12600' 2024-04-06T10:04:09.450 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T10:04:09.450 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.4 2024-04-06T10:04:09.852 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror image_29 2024-04-06T10:04:09.852 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T10:04:09.852 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:04:09.852 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_29 2024-04-06T10:04:09.852 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T10:04:09.852 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n image_29 ']' 2024-04-06T10:04:09.852 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/image_29' 2024-04-06T10:04:09.853 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_29 2024-04-06T10:04:09.853 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T10:04:09.853 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T10:04:09.853 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T10:04:09.853 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T10:04:09.853 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T10:04:09.853 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T10:04:09.853 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T10:04:09.853 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T10:04:09.854 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T10:04:09.856 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T10:04:09.857 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_29 2024-04-06T10:04:09.857 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T10:04:09.857 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T10:04:09.857 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T10:04:09.857 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T10:04:09.857 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T10:04:09.857 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T10:04:09.857 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T10:04:09.857 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T10:04:09.857 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T10:04:09.857 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T10:04:09.857 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T10:04:09.857 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:04:09.858 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T10:04:09.858 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T10:04:09.859 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T10:04:09.859 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T10:04:09.859 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T10:04:09.859 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T10:04:09.859 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T10:04:09.859 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T10:04:09.859 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T10:04:09.859 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T10:04:09.859 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:04:09.859 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T10:04:09.859 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T10:04:09.860 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T10:04:09.860 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T10:04:09.860 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T10:04:09.860 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T10:04:09.860 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T10:04:09.883 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T10:04:09.883 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T10:04:09.884 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T10:04:09.884 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_29 2024-04-06T10:04:09.998 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:Invalid command: unused arguments: ['mirror/image_29'] 2024-04-06T10:04:09.998 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:rbd mirror flush : flush rbd mirror 2024-04-06T10:04:09.998 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:admin_socket: invalid command 2024-04-06T10:04:10.001 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T10:04:10.001 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/image_29 2024-04-06T10:04:10.002 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:1 2024-04-06T10:04:10.002 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T10:04:10.002 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T10:04:10.002 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-06T10:04:10.002 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:1 2024-04-06T10:04:10.002 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T10:04:10.002 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T10:04:10.002 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T10:04:10.002 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=1 2024-04-06T10:04:10.002 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-06T10:04:10.002 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T10:04:10.002 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:04:10.002 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=1 2024-04-06T10:04:10.003 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=1 2024-04-06T10:04:10.003 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-06T10:04:10.003 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:1 2024-04-06T10:04:10.003 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T10:04:10.003 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T10:04:10.003 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-06T10:04:10.003 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:1 2024-04-06T10:04:10.003 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T10:04:10.003 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T10:04:10.003 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:04:10.003 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=1 2024-04-06T10:04:10.003 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-06T10:04:10.003 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T10:04:10.003 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T10:04:10.003 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=1 2024-04-06T10:04:10.003 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=1 2024-04-06T10:04:10.003 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-06T10:04:10.027 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T10:04:10.027 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T10:04:10.027 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T10:04:10.027 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/image_29 2024-04-06T10:04:10.177 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T10:04:10.179 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror image_29 2024-04-06T10:04:10.179 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:04:10.179 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:04:10.179 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_29 2024-04-06T10:04:10.179 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror image_29 '' 2024-04-06T10:04:10.179 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:04:10.179 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:04:10.179 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_29 2024-04-06T10:04:10.179 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T10:04:10.179 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-image_29.status 2024-04-06T10:04:10.180 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-image_29.status 2024-04-06T10:04:10.180 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T10:04:10.181 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-image_29.status 2024-04-06T10:04:10.182 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_29 2024-04-06T10:04:10.182 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_29.status 2024-04-06T10:04:10.232 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 2 2024-04-06T10:04:10.232 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 2 2024-04-06T10:04:10.232 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T10:04:10.232 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=9, tag_tid=2, entry_tid=1], [object_number=8, tag_tid=2, entry_tid=0], [object_number=11, tag_tid=1, entry_tid=12599], [object_number=10, tag_tid=1, entry_tid=12598]]], state=connected] 2024-04-06T10:04:10.232 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=8, tag_tid=1, entry_tid=12600], [object_number=11, tag_tid=1, entry_tid=12599], [object_number=10, tag_tid=1, entry_tid=12598], [object_number=9, tag_tid=1, entry_tid=12597]]], state=connected] 2024-04-06T10:04:10.236 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_29.status 2024-04-06T10:04:10.237 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=9, tag_tid=2, entry_tid=1' 2024-04-06T10:04:10.238 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror image_29 2024-04-06T10:04:10.238 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:04:10.238 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:04:10.239 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_29 2024-04-06T10:04:10.239 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror image_29 '..*' 2024-04-06T10:04:10.239 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:04:10.239 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:04:10.239 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_29 2024-04-06T10:04:10.239 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T10:04:10.239 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-image_29.status 2024-04-06T10:04:10.239 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-image_29.status 2024-04-06T10:04:10.239 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T10:04:10.240 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-image_29.status 2024-04-06T10:04:10.240 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_29 2024-04-06T10:04:10.240 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_29.status 2024-04-06T10:04:10.284 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 2 2024-04-06T10:04:10.284 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 2 2024-04-06T10:04:10.284 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T10:04:10.284 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=9, tag_tid=2, entry_tid=1], [object_number=8, tag_tid=2, entry_tid=0], [object_number=11, tag_tid=1, entry_tid=12599], [object_number=10, tag_tid=1, entry_tid=12598]]], state=connected] 2024-04-06T10:04:10.285 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=8, tag_tid=1, entry_tid=12600], [object_number=11, tag_tid=1, entry_tid=12599], [object_number=10, tag_tid=1, entry_tid=12598], [object_number=9, tag_tid=1, entry_tid=12597]]], state=connected] 2024-04-06T10:04:10.288 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_29.status 2024-04-06T10:04:10.290 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=8, tag_tid=1, entry_tid=12600' 2024-04-06T10:04:10.290 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=9, tag_tid=2, entry_tid=1' -a 'object_number=9, tag_tid=2, entry_tid=1' = 'object_number=8, tag_tid=1, entry_tid=12600' 2024-04-06T10:04:10.290 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=8, tag_tid=1, entry_tid=12600' '!=' 'object_number=8, tag_tid=1, entry_tid=12600' 2024-04-06T10:04:10.290 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T10:04:10.290 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.8 2024-04-06T10:04:11.093 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror image_29 2024-04-06T10:04:11.093 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T10:04:11.093 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:04:11.093 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_29 2024-04-06T10:04:11.093 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T10:04:11.094 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n image_29 ']' 2024-04-06T10:04:11.094 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/image_29' 2024-04-06T10:04:11.094 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_29 2024-04-06T10:04:11.094 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T10:04:11.094 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T10:04:11.094 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T10:04:11.094 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T10:04:11.094 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T10:04:11.094 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T10:04:11.095 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T10:04:11.095 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T10:04:11.099 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T10:04:11.099 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T10:04:11.099 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_29 2024-04-06T10:04:11.100 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T10:04:11.100 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T10:04:11.100 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T10:04:11.100 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T10:04:11.100 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T10:04:11.100 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T10:04:11.100 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T10:04:11.100 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T10:04:11.100 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T10:04:11.100 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T10:04:11.101 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T10:04:11.101 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:04:11.101 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T10:04:11.101 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T10:04:11.101 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T10:04:11.101 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T10:04:11.101 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T10:04:11.101 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T10:04:11.101 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T10:04:11.101 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T10:04:11.102 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T10:04:11.102 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T10:04:11.102 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:04:11.102 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T10:04:11.102 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T10:04:11.102 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T10:04:11.102 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T10:04:11.102 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T10:04:11.102 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T10:04:11.102 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T10:04:11.122 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T10:04:11.122 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T10:04:11.122 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T10:04:11.122 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_29 2024-04-06T10:04:11.227 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:Invalid command: unused arguments: ['mirror/image_29'] 2024-04-06T10:04:11.227 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:rbd mirror flush : flush rbd mirror 2024-04-06T10:04:11.227 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:admin_socket: invalid command 2024-04-06T10:04:11.229 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T10:04:11.229 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/image_29 2024-04-06T10:04:11.229 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:1 2024-04-06T10:04:11.230 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T10:04:11.230 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T10:04:11.230 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-06T10:04:11.230 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:1 2024-04-06T10:04:11.230 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T10:04:11.230 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T10:04:11.230 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T10:04:11.230 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=1 2024-04-06T10:04:11.230 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-06T10:04:11.230 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T10:04:11.230 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:04:11.230 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=1 2024-04-06T10:04:11.230 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=1 2024-04-06T10:04:11.230 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-06T10:04:11.231 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:1 2024-04-06T10:04:11.231 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T10:04:11.231 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T10:04:11.231 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-06T10:04:11.231 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:1 2024-04-06T10:04:11.231 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T10:04:11.231 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T10:04:11.231 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:04:11.231 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=1 2024-04-06T10:04:11.231 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-06T10:04:11.231 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T10:04:11.231 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T10:04:11.231 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=1 2024-04-06T10:04:11.231 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=1 2024-04-06T10:04:11.232 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-06T10:04:11.251 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T10:04:11.251 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T10:04:11.251 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T10:04:11.251 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/image_29 2024-04-06T10:04:11.394 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T10:04:11.395 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror image_29 2024-04-06T10:04:11.395 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:04:11.395 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:04:11.395 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_29 2024-04-06T10:04:11.395 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror image_29 '' 2024-04-06T10:04:11.395 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:04:11.395 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:04:11.395 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_29 2024-04-06T10:04:11.395 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T10:04:11.396 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-image_29.status 2024-04-06T10:04:11.396 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-image_29.status 2024-04-06T10:04:11.396 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T10:04:11.398 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-image_29.status 2024-04-06T10:04:11.398 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_29 2024-04-06T10:04:11.399 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_29.status 2024-04-06T10:04:11.448 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 2 2024-04-06T10:04:11.448 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 2 2024-04-06T10:04:11.448 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T10:04:11.449 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=9, tag_tid=2, entry_tid=1], [object_number=8, tag_tid=2, entry_tid=0], [object_number=11, tag_tid=1, entry_tid=12599], [object_number=10, tag_tid=1, entry_tid=12598]]], state=connected] 2024-04-06T10:04:11.449 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=8, tag_tid=1, entry_tid=12600], [object_number=11, tag_tid=1, entry_tid=12599], [object_number=10, tag_tid=1, entry_tid=12598], [object_number=9, tag_tid=1, entry_tid=12597]]], state=connected] 2024-04-06T10:04:11.452 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_29.status 2024-04-06T10:04:11.453 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=9, tag_tid=2, entry_tid=1' 2024-04-06T10:04:11.454 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror image_29 2024-04-06T10:04:11.454 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:04:11.454 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:04:11.454 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_29 2024-04-06T10:04:11.454 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror image_29 '..*' 2024-04-06T10:04:11.455 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:04:11.455 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:04:11.455 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_29 2024-04-06T10:04:11.455 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T10:04:11.455 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-image_29.status 2024-04-06T10:04:11.455 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-image_29.status 2024-04-06T10:04:11.455 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T10:04:11.456 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-image_29.status 2024-04-06T10:04:11.456 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_29 2024-04-06T10:04:11.456 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_29.status 2024-04-06T10:04:11.501 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 2 2024-04-06T10:04:11.501 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 2 2024-04-06T10:04:11.501 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T10:04:11.501 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=9, tag_tid=2, entry_tid=1], [object_number=8, tag_tid=2, entry_tid=0], [object_number=11, tag_tid=1, entry_tid=12599], [object_number=10, tag_tid=1, entry_tid=12598]]], state=connected] 2024-04-06T10:04:11.501 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=8, tag_tid=1, entry_tid=12600], [object_number=11, tag_tid=1, entry_tid=12599], [object_number=10, tag_tid=1, entry_tid=12598], [object_number=9, tag_tid=1, entry_tid=12597]]], state=connected] 2024-04-06T10:04:11.505 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_29.status 2024-04-06T10:04:11.507 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=8, tag_tid=1, entry_tid=12600' 2024-04-06T10:04:11.507 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=9, tag_tid=2, entry_tid=1' -a 'object_number=9, tag_tid=2, entry_tid=1' = 'object_number=8, tag_tid=1, entry_tid=12600' 2024-04-06T10:04:11.507 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=8, tag_tid=1, entry_tid=12600' '!=' 'object_number=8, tag_tid=1, entry_tid=12600' 2024-04-06T10:04:11.507 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T10:04:11.507 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 1.6 2024-04-06T10:04:13.109 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror image_29 2024-04-06T10:04:13.109 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T10:04:13.109 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:04:13.109 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_29 2024-04-06T10:04:13.109 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T10:04:13.109 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n image_29 ']' 2024-04-06T10:04:13.109 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/image_29' 2024-04-06T10:04:13.110 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_29 2024-04-06T10:04:13.110 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T10:04:13.110 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T10:04:13.110 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T10:04:13.110 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T10:04:13.110 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T10:04:13.110 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T10:04:13.110 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T10:04:13.110 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T10:04:13.111 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T10:04:13.113 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T10:04:13.113 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_29 2024-04-06T10:04:13.113 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T10:04:13.113 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T10:04:13.113 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T10:04:13.113 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T10:04:13.113 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T10:04:13.113 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T10:04:13.113 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T10:04:13.113 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T10:04:13.113 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T10:04:13.113 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T10:04:13.113 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T10:04:13.113 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:04:13.114 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T10:04:13.114 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T10:04:13.114 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T10:04:13.114 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T10:04:13.114 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T10:04:13.116 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T10:04:13.116 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T10:04:13.116 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T10:04:13.116 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T10:04:13.116 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T10:04:13.116 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:04:13.117 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T10:04:13.117 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T10:04:13.117 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T10:04:13.117 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T10:04:13.117 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T10:04:13.117 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T10:04:13.118 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T10:04:13.138 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T10:04:13.139 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T10:04:13.139 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T10:04:13.139 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_29 2024-04-06T10:04:13.244 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:Invalid command: unused arguments: ['mirror/image_29'] 2024-04-06T10:04:13.244 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:rbd mirror flush : flush rbd mirror 2024-04-06T10:04:13.244 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:admin_socket: invalid command 2024-04-06T10:04:13.245 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T10:04:13.246 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/image_29 2024-04-06T10:04:13.246 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:1 2024-04-06T10:04:13.246 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T10:04:13.246 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T10:04:13.246 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-06T10:04:13.246 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:1 2024-04-06T10:04:13.246 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T10:04:13.246 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T10:04:13.246 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T10:04:13.246 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=1 2024-04-06T10:04:13.246 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-06T10:04:13.246 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T10:04:13.246 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:04:13.247 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=1 2024-04-06T10:04:13.247 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=1 2024-04-06T10:04:13.248 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-06T10:04:13.248 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:1 2024-04-06T10:04:13.248 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T10:04:13.248 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T10:04:13.248 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-06T10:04:13.248 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:1 2024-04-06T10:04:13.249 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T10:04:13.249 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T10:04:13.249 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:04:13.249 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=1 2024-04-06T10:04:13.249 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-06T10:04:13.249 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T10:04:13.249 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T10:04:13.249 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=1 2024-04-06T10:04:13.249 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=1 2024-04-06T10:04:13.249 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-06T10:04:13.266 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T10:04:13.266 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T10:04:13.267 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T10:04:13.267 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/image_29 2024-04-06T10:04:13.450 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T10:04:13.452 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror image_29 2024-04-06T10:04:13.452 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:04:13.452 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:04:13.452 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_29 2024-04-06T10:04:13.453 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror image_29 '' 2024-04-06T10:04:13.453 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:04:13.453 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:04:13.453 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_29 2024-04-06T10:04:13.453 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T10:04:13.453 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-image_29.status 2024-04-06T10:04:13.453 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-image_29.status 2024-04-06T10:04:13.453 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T10:04:13.454 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-image_29.status 2024-04-06T10:04:13.454 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_29 2024-04-06T10:04:13.454 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_29.status 2024-04-06T10:04:13.499 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 2 2024-04-06T10:04:13.499 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 2 2024-04-06T10:04:13.499 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T10:04:13.499 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=9, tag_tid=2, entry_tid=1], [object_number=8, tag_tid=2, entry_tid=0], [object_number=11, tag_tid=1, entry_tid=12599], [object_number=10, tag_tid=1, entry_tid=12598]]], state=connected] 2024-04-06T10:04:13.499 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=9, tag_tid=2, entry_tid=1], [object_number=8, tag_tid=2, entry_tid=0], [object_number=11, tag_tid=1, entry_tid=12599], [object_number=10, tag_tid=1, entry_tid=12598]]], state=connected] 2024-04-06T10:04:13.503 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_29.status 2024-04-06T10:04:13.503 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=9, tag_tid=2, entry_tid=1' 2024-04-06T10:04:13.505 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror image_29 2024-04-06T10:04:13.505 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:04:13.505 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:04:13.505 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_29 2024-04-06T10:04:13.505 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror image_29 '..*' 2024-04-06T10:04:13.505 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:04:13.505 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:04:13.506 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_29 2024-04-06T10:04:13.506 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T10:04:13.506 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-image_29.status 2024-04-06T10:04:13.506 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-image_29.status 2024-04-06T10:04:13.506 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T10:04:13.507 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-image_29.status 2024-04-06T10:04:13.507 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_29 2024-04-06T10:04:13.507 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_29.status 2024-04-06T10:04:13.552 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 2 2024-04-06T10:04:13.552 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 2 2024-04-06T10:04:13.552 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T10:04:13.552 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=9, tag_tid=2, entry_tid=1], [object_number=8, tag_tid=2, entry_tid=0], [object_number=11, tag_tid=1, entry_tid=12599], [object_number=10, tag_tid=1, entry_tid=12598]]], state=connected] 2024-04-06T10:04:13.552 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=9, tag_tid=2, entry_tid=1], [object_number=8, tag_tid=2, entry_tid=0], [object_number=11, tag_tid=1, entry_tid=12599], [object_number=10, tag_tid=1, entry_tid=12598]]], state=connected] 2024-04-06T10:04:13.555 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_29.status 2024-04-06T10:04:13.557 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=9, tag_tid=2, entry_tid=1' 2024-04-06T10:04:13.557 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=9, tag_tid=2, entry_tid=1' -a 'object_number=9, tag_tid=2, entry_tid=1' = 'object_number=9, tag_tid=2, entry_tid=1' 2024-04-06T10:04:13.558 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T10:04:13.558 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_snap_present cluster1 mirror image_29 snap 2024-04-06T10:04:13.558 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T10:04:13.558 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:04:13.558 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_29 2024-04-06T10:04:13.559 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local snap_name=snap 2024-04-06T10:04:13.559 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local s 2024-04-06T10:04:13.559 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-04-06T10:04:13.559 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 1 2024-04-06T10:04:14.558 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 info mirror/image_29@snap 2024-04-06T10:04:14.623 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:rbd image 'image_29': 2024-04-06T10:04:14.623 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: size 128 MiB in 32 objects 2024-04-06T10:04:14.623 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: order 22 (4 MiB objects) 2024-04-06T10:04:14.623 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: snapshot_count: 1 2024-04-06T10:04:14.623 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: id: 10ac93c58bad 2024-04-06T10:04:14.623 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: block_name_prefix: rbd_data.10ac93c58bad 2024-04-06T10:04:14.624 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: format: 2 2024-04-06T10:04:14.624 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: features: layering, exclusive-lock, journaling 2024-04-06T10:04:14.624 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: op_features: 2024-04-06T10:04:14.624 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: flags: 2024-04-06T10:04:14.624 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: create_timestamp: Sat Apr 6 09:55:04 2024 2024-04-06T10:04:14.624 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: access_timestamp: Sat Apr 6 09:55:04 2024 2024-04-06T10:04:14.624 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: modify_timestamp: Sat Apr 6 09:56:05 2024 2024-04-06T10:04:14.624 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: protected: False 2024-04-06T10:04:14.624 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: journal: 10ac93c58bad 2024-04-06T10:04:14.624 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: mirroring state: enabled 2024-04-06T10:04:14.624 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: mirroring mode: journal 2024-04-06T10:04:14.624 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: mirroring global id: 1563d746-f159-4fc8-932b-3b122e0eb842 2024-04-06T10:04:14.624 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: mirroring primary: false 2024-04-06T10:04:14.627 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T10:04:14.627 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ compare_image_snaps mirror image_29 snap 2024-04-06T10:04:14.627 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:04:14.628 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_29 2024-04-06T10:04:14.628 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local snap_name=snap 2024-04-06T10:04:14.628 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local ret=0 2024-04-06T10:04:14.628 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local rmt_export=/tmp/tmp.BhW6xKmROU/cluster2-mirror-image_29.export 2024-04-06T10:04:14.628 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loc_export=/tmp/tmp.BhW6xKmROU/cluster1-mirror-image_29.export 2024-04-06T10:04:14.628 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rm -f /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_29.export /tmp/tmp.BhW6xKmROU/cluster1-mirror-image_29.export 2024-04-06T10:04:14.628 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster2 -p mirror export image_29@snap /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_29.export 2024-04-06T10:04:14.808 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-04-06T10:04:14.817 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 -p mirror export image_29@snap /tmp/tmp.BhW6xKmROU/cluster1-mirror-image_29.export 2024-04-06T10:04:15.290 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-04-06T10:04:15.302 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmp /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_29.export /tmp/tmp.BhW6xKmROU/cluster1-mirror-image_29.export 2024-04-06T10:04:15.383 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rm -f /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_29.export /tmp/tmp.BhW6xKmROU/cluster1-mirror-image_29.export 2024-04-06T10:04:15.427 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T10:04:15.427 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-06T10:04:15.427 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ image=image_30 2024-04-06T10:04:15.427 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ create_snap cluster2 mirror image_30 snap 2024-04-06T10:04:15.427 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-04-06T10:04:15.427 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:04:15.427 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_30 2024-04-06T10:04:15.427 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local snap_name=snap 2024-04-06T10:04:15.428 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster2 -p mirror snap create image_30@snap --debug-rbd=20 --debug-journaler=20 2024-04-06T10:04:15.561 INFO:tasks.rbd_mirror_thrash:kill cluster2.client.mirror.0 2024-04-06T10:04:15.562 INFO:tasks.rbd_mirror.cluster2.client.mirror.0:Sent signal 15 2024-04-06T10:04:15.562 INFO:tasks.rbd_mirror_thrash:waiting for 0 secs before reviving daemons 2024-04-06T10:04:15.562 INFO:tasks.rbd_mirror_thrash:waiting for cluster2.client.mirror.0 2024-04-06T10:04:15.562 INFO:teuthology.orchestra.run:waiting for 600 2024-04-06T10:04:15.563 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi192.stderr:2024-04-06T10:04:15.561+0000 7fa139812640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.0 (PID: 140869) UID: 1000 2024-04-06T10:04:17.898 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_image_replay_started cluster1 mirror image_30 2024-04-06T10:04:17.902 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T10:04:17.902 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:04:17.902 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_30 2024-04-06T10:04:17.902 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_image_replay_state cluster1 mirror image_30 started 2024-04-06T10:04:17.902 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T10:04:17.902 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:04:17.902 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_30 2024-04-06T10:04:17.902 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state=started 2024-04-06T10:04:17.903 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local s 2024-04-06T10:04:17.903 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T10:04:17.903 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 1 2024-04-06T10:04:18.900 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test_image_replay_state cluster1 mirror image_30 started 2024-04-06T10:04:18.900 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T10:04:18.900 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:04:18.900 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_30 2024-04-06T10:04:18.900 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local test_state=started 2024-04-06T10:04:18.900 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local status_result 2024-04-06T10:04:18.900 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local current_state=stopped 2024-04-06T10:04:18.901 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_30 2024-04-06T10:04:18.901 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -i state 2024-04-06T10:04:18.901 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_instance=cluster1 2024-04-06T10:04:18.902 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ shift 2024-04-06T10:04:18.902 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T10:04:18.902 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance=cluster1 2024-04-06T10:04:18.902 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local loop_instance 2024-04-06T10:04:18.902 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T10:04:18.902 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sleep 0 2024-04-06T10:04:18.902 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-04-06T10:04:18.903 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ seq 0 3 2024-04-06T10:04:18.905 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T10:04:18.905 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_30 2024-04-06T10:04:18.905 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1:0 2024-04-06T10:04:18.905 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ shift 2024-04-06T10:04:18.905 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T10:04:18.905 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-04-06T10:04:18.905 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T10:04:18.905 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=cluster 2024-04-06T10:04:18.905 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T10:04:18.905 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:04:18.905 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T10:04:18.906 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T10:04:18.906 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval cluster=cluster1 2024-04-06T10:04:18.906 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ cluster=cluster1 2024-04-06T10:04:18.907 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T10:04:18.907 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T10:04:18.908 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-04-06T10:04:18.908 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local local_cluster=cluster1:0 2024-04-06T10:04:18.908 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local cluster=cluster1 2024-04-06T10:04:18.908 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local instance 2024-04-06T10:04:18.908 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T10:04:18.908 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local val=cluster1:0 2024-04-06T10:04:18.908 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local cluster_var_name=local_cluster 2024-04-06T10:04:18.908 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local instance_var_name=instance 2024-04-06T10:04:18.908 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ cluster=cluster1 2024-04-06T10:04:18.908 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T10:04:18.908 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ '[' 0 = cluster1:0 ']' 2024-04-06T10:04:18.908 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ eval local_cluster=cluster1 2024-04-06T10:04:18.908 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++++ local_cluster=cluster1 2024-04-06T10:04:18.908 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ eval instance=0 2024-04-06T10:04:18.908 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++++ instance=0 2024-04-06T10:04:18.909 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T10:04:18.930 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T10:04:18.931 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T10:04:18.931 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T10:04:18.931 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_30 2024-04-06T10:04:19.032 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:Invalid command: unused arguments: ['mirror/image_30'] 2024-04-06T10:04:19.033 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:rbd mirror status : get status for rbd mirror 2024-04-06T10:04:19.033 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:admin_socket: invalid command 2024-04-06T10:04:19.035 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T10:04:19.035 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_30 2024-04-06T10:04:19.035 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1:1 2024-04-06T10:04:19.035 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ shift 2024-04-06T10:04:19.035 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T10:04:19.035 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-04-06T10:04:19.035 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:1 2024-04-06T10:04:19.035 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=cluster 2024-04-06T10:04:19.035 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T10:04:19.035 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:04:19.036 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=1 2024-04-06T10:04:19.036 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-06T10:04:19.036 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval cluster=cluster1 2024-04-06T10:04:19.036 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ cluster=cluster1 2024-04-06T10:04:19.036 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=1 2024-04-06T10:04:19.036 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=1 2024-04-06T10:04:19.039 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-04-06T10:04:19.039 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local local_cluster=cluster1:1 2024-04-06T10:04:19.039 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local cluster=cluster1 2024-04-06T10:04:19.039 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local instance 2024-04-06T10:04:19.039 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-06T10:04:19.039 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local val=cluster1:1 2024-04-06T10:04:19.039 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local cluster_var_name=local_cluster 2024-04-06T10:04:19.039 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local instance_var_name=instance 2024-04-06T10:04:19.039 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ cluster=cluster1 2024-04-06T10:04:19.039 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=1 2024-04-06T10:04:19.039 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ '[' 1 = cluster1:1 ']' 2024-04-06T10:04:19.039 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ eval local_cluster=cluster1 2024-04-06T10:04:19.039 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++++ local_cluster=cluster1 2024-04-06T10:04:19.040 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ eval instance=1 2024-04-06T10:04:19.040 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++++ instance=1 2024-04-06T10:04:19.040 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-06T10:04:19.060 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T10:04:19.060 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T10:04:19.060 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T10:04:19.060 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_30 2024-04-06T10:04:19.205 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ return 0 2024-04-06T10:04:19.205 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ status_result=' "state": "Replaying"' 2024-04-06T10:04:19.206 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ echo ' "state": "Replaying"' 2024-04-06T10:04:19.206 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep -i Replaying 2024-04-06T10:04:19.208 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: "state": "Replaying" 2024-04-06T10:04:19.210 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ current_state=started 2024-04-06T10:04:19.210 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test started = started 2024-04-06T10:04:19.210 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T10:04:19.210 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_30 2024-04-06T10:04:19.210 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local local_cluster=cluster1 2024-04-06T10:04:19.210 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-04-06T10:04:19.210 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:04:19.210 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_30 2024-04-06T10:04:19.210 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' journal = journal ']' 2024-04-06T10:04:19.210 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_journal_replay_complete cluster1 cluster2 mirror image_30 2024-04-06T10:04:19.210 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local local_cluster=cluster1 2024-04-06T10:04:19.210 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-04-06T10:04:19.211 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:04:19.211 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_30 2024-04-06T10:04:19.211 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local s master_pos mirror_pos last_mirror_pos 2024-04-06T10:04:19.211 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local master_tag master_entry mirror_tag mirror_entry 2024-04-06T10:04:19.211 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T10:04:19.211 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T10:04:19.211 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T10:04:19.411 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror image_30 2024-04-06T10:04:19.411 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T10:04:19.411 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:04:19.411 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_30 2024-04-06T10:04:19.411 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T10:04:19.411 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n image_30 ']' 2024-04-06T10:04:19.411 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/image_30' 2024-04-06T10:04:19.411 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_30 2024-04-06T10:04:19.411 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T10:04:19.412 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T10:04:19.412 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T10:04:19.412 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T10:04:19.412 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T10:04:19.412 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T10:04:19.412 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T10:04:19.412 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T10:04:19.413 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T10:04:19.416 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T10:04:19.417 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_30 2024-04-06T10:04:19.417 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T10:04:19.417 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T10:04:19.417 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T10:04:19.417 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T10:04:19.417 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T10:04:19.417 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T10:04:19.417 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T10:04:19.417 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T10:04:19.417 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T10:04:19.417 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T10:04:19.417 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T10:04:19.417 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:04:19.418 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T10:04:19.418 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T10:04:19.418 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T10:04:19.418 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T10:04:19.418 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T10:04:19.418 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T10:04:19.419 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T10:04:19.419 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T10:04:19.419 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T10:04:19.419 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T10:04:19.419 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:04:19.419 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T10:04:19.419 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T10:04:19.419 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T10:04:19.419 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T10:04:19.419 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T10:04:19.419 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T10:04:19.419 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T10:04:19.441 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T10:04:19.441 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T10:04:19.441 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T10:04:19.441 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_30 2024-04-06T10:04:19.543 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:Invalid command: unused arguments: ['mirror/image_30'] 2024-04-06T10:04:19.543 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:rbd mirror flush : flush rbd mirror 2024-04-06T10:04:19.543 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:admin_socket: invalid command 2024-04-06T10:04:19.546 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T10:04:19.546 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/image_30 2024-04-06T10:04:19.547 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:1 2024-04-06T10:04:19.547 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T10:04:19.547 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T10:04:19.547 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-06T10:04:19.547 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:1 2024-04-06T10:04:19.547 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T10:04:19.547 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T10:04:19.547 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T10:04:19.547 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=1 2024-04-06T10:04:19.547 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-06T10:04:19.547 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T10:04:19.547 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:04:19.547 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=1 2024-04-06T10:04:19.548 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=1 2024-04-06T10:04:19.548 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-06T10:04:19.548 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:1 2024-04-06T10:04:19.548 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T10:04:19.548 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T10:04:19.548 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-06T10:04:19.548 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:1 2024-04-06T10:04:19.548 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T10:04:19.548 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T10:04:19.548 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:04:19.548 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=1 2024-04-06T10:04:19.548 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-06T10:04:19.548 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T10:04:19.548 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T10:04:19.549 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=1 2024-04-06T10:04:19.549 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=1 2024-04-06T10:04:19.549 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-06T10:04:19.565 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T10:04:19.565 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T10:04:19.565 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T10:04:19.565 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/image_30 2024-04-06T10:04:19.688 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T10:04:19.689 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror image_30 2024-04-06T10:04:19.689 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:04:19.689 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:04:19.689 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_30 2024-04-06T10:04:19.689 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror image_30 '' 2024-04-06T10:04:19.689 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:04:19.689 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:04:19.689 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_30 2024-04-06T10:04:19.690 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T10:04:19.690 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-image_30.status 2024-04-06T10:04:19.690 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-image_30.status 2024-04-06T10:04:19.690 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T10:04:19.691 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-image_30.status 2024-04-06T10:04:19.692 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_30 2024-04-06T10:04:19.692 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_30.status 2024-04-06T10:04:19.744 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 1 2024-04-06T10:04:19.744 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 1 2024-04-06T10:04:19.745 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T10:04:19.745 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=5, tag_tid=2, entry_tid=1], [object_number=4, tag_tid=2, entry_tid=0], [object_number=6, tag_tid=1, entry_tid=5842], [object_number=7, tag_tid=1, entry_tid=5839]]], state=connected] 2024-04-06T10:04:19.745 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=6, tag_tid=1, entry_tid=5842], [object_number=5, tag_tid=1, entry_tid=5841], [object_number=4, tag_tid=1, entry_tid=5840], [object_number=7, tag_tid=1, entry_tid=5839]]], state=connected] 2024-04-06T10:04:19.748 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_30.status 2024-04-06T10:04:19.749 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=5, tag_tid=2, entry_tid=1' 2024-04-06T10:04:19.750 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror image_30 2024-04-06T10:04:19.750 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:04:19.750 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:04:19.750 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_30 2024-04-06T10:04:19.750 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror image_30 '..*' 2024-04-06T10:04:19.750 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:04:19.751 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:04:19.751 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_30 2024-04-06T10:04:19.751 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T10:04:19.751 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-image_30.status 2024-04-06T10:04:19.751 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-image_30.status 2024-04-06T10:04:19.751 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T10:04:19.752 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-image_30.status 2024-04-06T10:04:19.752 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_30 2024-04-06T10:04:19.752 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_30.status 2024-04-06T10:04:19.794 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 1 2024-04-06T10:04:19.794 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 1 2024-04-06T10:04:19.794 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T10:04:19.794 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=5, tag_tid=2, entry_tid=1], [object_number=4, tag_tid=2, entry_tid=0], [object_number=6, tag_tid=1, entry_tid=5842], [object_number=7, tag_tid=1, entry_tid=5839]]], state=connected] 2024-04-06T10:04:19.794 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=6, tag_tid=1, entry_tid=5842], [object_number=5, tag_tid=1, entry_tid=5841], [object_number=4, tag_tid=1, entry_tid=5840], [object_number=7, tag_tid=1, entry_tid=5839]]], state=connected] 2024-04-06T10:04:19.797 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_30.status 2024-04-06T10:04:19.799 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=6, tag_tid=1, entry_tid=5842' 2024-04-06T10:04:19.800 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=5, tag_tid=2, entry_tid=1' -a 'object_number=5, tag_tid=2, entry_tid=1' = 'object_number=6, tag_tid=1, entry_tid=5842' 2024-04-06T10:04:19.800 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=6, tag_tid=1, entry_tid=5842' '!=' '' 2024-04-06T10:04:19.800 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T10:04:19.800 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=6, tag_tid=1, entry_tid=5842' = '' 2024-04-06T10:04:19.800 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=6, tag_tid=1, entry_tid=5842' 2024-04-06T10:04:19.800 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=5, tag_tid=2, entry_tid=1' 2024-04-06T10:04:19.800 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T10:04:19.800 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T10:04:19.801 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=2 2024-04-06T10:04:19.801 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=6, tag_tid=1, entry_tid=5842' 2024-04-06T10:04:19.801 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T10:04:19.801 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T10:04:19.802 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=1 2024-04-06T10:04:19.803 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=5, tag_tid=2, entry_tid=1' 2024-04-06T10:04:19.803 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T10:04:19.803 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T10:04:19.803 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T10:04:19.804 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=6, tag_tid=1, entry_tid=5842' 2024-04-06T10:04:19.804 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T10:04:19.804 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T10:04:19.805 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=5842 2024-04-06T10:04:19.805 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 2 = 1 -a 1 -le 5842 2024-04-06T10:04:19.805 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T10:04:19.806 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T10:04:19.806 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T10:04:20.007 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror image_30 2024-04-06T10:04:20.008 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T10:04:20.008 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:04:20.008 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_30 2024-04-06T10:04:20.008 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T10:04:20.008 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n image_30 ']' 2024-04-06T10:04:20.008 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/image_30' 2024-04-06T10:04:20.008 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_30 2024-04-06T10:04:20.008 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T10:04:20.008 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T10:04:20.008 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T10:04:20.008 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T10:04:20.009 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T10:04:20.009 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T10:04:20.009 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T10:04:20.009 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T10:04:20.009 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T10:04:20.013 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T10:04:20.013 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_30 2024-04-06T10:04:20.013 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T10:04:20.013 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T10:04:20.013 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T10:04:20.013 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T10:04:20.013 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T10:04:20.013 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T10:04:20.013 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T10:04:20.013 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T10:04:20.014 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T10:04:20.014 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T10:04:20.014 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T10:04:20.014 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:04:20.014 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T10:04:20.014 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T10:04:20.014 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T10:04:20.014 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T10:04:20.014 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T10:04:20.014 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T10:04:20.014 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T10:04:20.014 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T10:04:20.014 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T10:04:20.014 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T10:04:20.015 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:04:20.015 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T10:04:20.015 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T10:04:20.015 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T10:04:20.015 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T10:04:20.015 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T10:04:20.015 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T10:04:20.015 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T10:04:20.037 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T10:04:20.037 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T10:04:20.037 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T10:04:20.037 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_30 2024-04-06T10:04:20.142 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:Invalid command: unused arguments: ['mirror/image_30'] 2024-04-06T10:04:20.143 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:rbd mirror flush : flush rbd mirror 2024-04-06T10:04:20.143 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:admin_socket: invalid command 2024-04-06T10:04:20.145 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T10:04:20.146 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/image_30 2024-04-06T10:04:20.146 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:1 2024-04-06T10:04:20.146 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T10:04:20.146 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T10:04:20.146 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-06T10:04:20.146 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:1 2024-04-06T10:04:20.146 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T10:04:20.146 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T10:04:20.146 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T10:04:20.146 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=1 2024-04-06T10:04:20.147 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-06T10:04:20.147 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T10:04:20.147 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:04:20.147 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=1 2024-04-06T10:04:20.147 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=1 2024-04-06T10:04:20.147 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-06T10:04:20.147 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:1 2024-04-06T10:04:20.147 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T10:04:20.147 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T10:04:20.147 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-06T10:04:20.147 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:1 2024-04-06T10:04:20.147 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T10:04:20.148 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T10:04:20.148 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:04:20.148 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=1 2024-04-06T10:04:20.148 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-06T10:04:20.148 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T10:04:20.148 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T10:04:20.148 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=1 2024-04-06T10:04:20.148 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=1 2024-04-06T10:04:20.149 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-06T10:04:20.167 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T10:04:20.167 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T10:04:20.167 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T10:04:20.167 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/image_30 2024-04-06T10:04:20.310 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T10:04:20.311 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror image_30 2024-04-06T10:04:20.312 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:04:20.312 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:04:20.312 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_30 2024-04-06T10:04:20.312 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror image_30 '' 2024-04-06T10:04:20.312 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:04:20.312 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:04:20.312 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_30 2024-04-06T10:04:20.312 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T10:04:20.312 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-image_30.status 2024-04-06T10:04:20.312 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-image_30.status 2024-04-06T10:04:20.313 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T10:04:20.314 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-image_30.status 2024-04-06T10:04:20.314 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_30 2024-04-06T10:04:20.314 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_30.status 2024-04-06T10:04:20.366 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 1 2024-04-06T10:04:20.366 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 1 2024-04-06T10:04:20.367 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T10:04:20.367 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=5, tag_tid=2, entry_tid=1], [object_number=4, tag_tid=2, entry_tid=0], [object_number=6, tag_tid=1, entry_tid=5842], [object_number=7, tag_tid=1, entry_tid=5839]]], state=connected] 2024-04-06T10:04:20.367 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=6, tag_tid=1, entry_tid=5842], [object_number=5, tag_tid=1, entry_tid=5841], [object_number=4, tag_tid=1, entry_tid=5840], [object_number=7, tag_tid=1, entry_tid=5839]]], state=connected] 2024-04-06T10:04:20.368 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_30.status 2024-04-06T10:04:20.371 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=5, tag_tid=2, entry_tid=1' 2024-04-06T10:04:20.371 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror image_30 2024-04-06T10:04:20.371 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:04:20.371 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:04:20.371 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_30 2024-04-06T10:04:20.371 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror image_30 '..*' 2024-04-06T10:04:20.371 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:04:20.372 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:04:20.372 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_30 2024-04-06T10:04:20.372 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T10:04:20.372 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-image_30.status 2024-04-06T10:04:20.372 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-image_30.status 2024-04-06T10:04:20.372 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T10:04:20.373 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-image_30.status 2024-04-06T10:04:20.373 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_30 2024-04-06T10:04:20.373 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_30.status 2024-04-06T10:04:20.414 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 1 2024-04-06T10:04:20.414 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 1 2024-04-06T10:04:20.414 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T10:04:20.415 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=5, tag_tid=2, entry_tid=1], [object_number=4, tag_tid=2, entry_tid=0], [object_number=6, tag_tid=1, entry_tid=5842], [object_number=7, tag_tid=1, entry_tid=5839]]], state=connected] 2024-04-06T10:04:20.415 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=6, tag_tid=1, entry_tid=5842], [object_number=5, tag_tid=1, entry_tid=5841], [object_number=4, tag_tid=1, entry_tid=5840], [object_number=7, tag_tid=1, entry_tid=5839]]], state=connected] 2024-04-06T10:04:20.418 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_30.status 2024-04-06T10:04:20.419 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=6, tag_tid=1, entry_tid=5842' 2024-04-06T10:04:20.420 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=5, tag_tid=2, entry_tid=1' -a 'object_number=5, tag_tid=2, entry_tid=1' = 'object_number=6, tag_tid=1, entry_tid=5842' 2024-04-06T10:04:20.420 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=6, tag_tid=1, entry_tid=5842' '!=' 'object_number=6, tag_tid=1, entry_tid=5842' 2024-04-06T10:04:20.420 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T10:04:20.420 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.4 2024-04-06T10:04:20.822 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror image_30 2024-04-06T10:04:20.822 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T10:04:20.822 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:04:20.822 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_30 2024-04-06T10:04:20.822 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T10:04:20.822 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n image_30 ']' 2024-04-06T10:04:20.822 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/image_30' 2024-04-06T10:04:20.822 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_30 2024-04-06T10:04:20.822 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T10:04:20.823 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T10:04:20.823 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T10:04:20.823 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T10:04:20.823 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T10:04:20.823 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T10:04:20.823 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T10:04:20.824 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T10:04:20.824 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T10:04:20.827 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T10:04:20.827 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_30 2024-04-06T10:04:20.827 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T10:04:20.827 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T10:04:20.827 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T10:04:20.828 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T10:04:20.828 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T10:04:20.828 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T10:04:20.828 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T10:04:20.828 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T10:04:20.828 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T10:04:20.828 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T10:04:20.829 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T10:04:20.829 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:04:20.829 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T10:04:20.829 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T10:04:20.829 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T10:04:20.829 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T10:04:20.829 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T10:04:20.829 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T10:04:20.829 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T10:04:20.829 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T10:04:20.829 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T10:04:20.829 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T10:04:20.829 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:04:20.830 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T10:04:20.830 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T10:04:20.830 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T10:04:20.830 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T10:04:20.830 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T10:04:20.830 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T10:04:20.830 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T10:04:20.853 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T10:04:20.853 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T10:04:20.853 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T10:04:20.853 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_30 2024-04-06T10:04:20.958 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:Invalid command: unused arguments: ['mirror/image_30'] 2024-04-06T10:04:20.958 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:rbd mirror flush : flush rbd mirror 2024-04-06T10:04:20.958 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:admin_socket: invalid command 2024-04-06T10:04:20.960 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T10:04:20.960 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/image_30 2024-04-06T10:04:20.960 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:1 2024-04-06T10:04:20.960 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T10:04:20.961 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T10:04:20.961 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-06T10:04:20.961 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:1 2024-04-06T10:04:20.961 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T10:04:20.961 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T10:04:20.961 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T10:04:20.961 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=1 2024-04-06T10:04:20.961 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-06T10:04:20.961 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T10:04:20.961 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:04:20.961 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=1 2024-04-06T10:04:20.961 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=1 2024-04-06T10:04:20.963 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-06T10:04:20.963 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:1 2024-04-06T10:04:20.963 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T10:04:20.963 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T10:04:20.963 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-06T10:04:20.964 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:1 2024-04-06T10:04:20.964 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T10:04:20.964 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T10:04:20.964 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:04:20.964 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=1 2024-04-06T10:04:20.964 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-06T10:04:20.964 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T10:04:20.964 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T10:04:20.964 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=1 2024-04-06T10:04:20.964 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=1 2024-04-06T10:04:20.964 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-06T10:04:20.989 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T10:04:20.989 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T10:04:20.989 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T10:04:20.989 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/image_30 2024-04-06T10:04:21.130 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T10:04:21.131 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror image_30 2024-04-06T10:04:21.131 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:04:21.132 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:04:21.132 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_30 2024-04-06T10:04:21.132 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror image_30 '' 2024-04-06T10:04:21.132 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:04:21.132 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:04:21.132 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_30 2024-04-06T10:04:21.132 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T10:04:21.132 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-image_30.status 2024-04-06T10:04:21.133 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-image_30.status 2024-04-06T10:04:21.133 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T10:04:21.134 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-image_30.status 2024-04-06T10:04:21.134 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_30 2024-04-06T10:04:21.135 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_30.status 2024-04-06T10:04:21.182 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 1 2024-04-06T10:04:21.183 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 1 2024-04-06T10:04:21.183 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T10:04:21.183 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=5, tag_tid=2, entry_tid=1], [object_number=4, tag_tid=2, entry_tid=0], [object_number=6, tag_tid=1, entry_tid=5842], [object_number=7, tag_tid=1, entry_tid=5839]]], state=connected] 2024-04-06T10:04:21.183 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=6, tag_tid=1, entry_tid=5842], [object_number=5, tag_tid=1, entry_tid=5841], [object_number=4, tag_tid=1, entry_tid=5840], [object_number=7, tag_tid=1, entry_tid=5839]]], state=connected] 2024-04-06T10:04:21.183 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_30.status 2024-04-06T10:04:21.184 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=5, tag_tid=2, entry_tid=1' 2024-04-06T10:04:21.184 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror image_30 2024-04-06T10:04:21.184 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:04:21.184 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:04:21.185 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_30 2024-04-06T10:04:21.185 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror image_30 '..*' 2024-04-06T10:04:21.185 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:04:21.185 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:04:21.185 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_30 2024-04-06T10:04:21.185 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T10:04:21.185 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-image_30.status 2024-04-06T10:04:21.185 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-image_30.status 2024-04-06T10:04:21.185 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T10:04:21.186 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-image_30.status 2024-04-06T10:04:21.186 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_30 2024-04-06T10:04:21.186 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_30.status 2024-04-06T10:04:21.230 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 1 2024-04-06T10:04:21.230 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 1 2024-04-06T10:04:21.231 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T10:04:21.231 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=5, tag_tid=2, entry_tid=1], [object_number=4, tag_tid=2, entry_tid=0], [object_number=6, tag_tid=1, entry_tid=5842], [object_number=7, tag_tid=1, entry_tid=5839]]], state=connected] 2024-04-06T10:04:21.231 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=6, tag_tid=1, entry_tid=5842], [object_number=5, tag_tid=1, entry_tid=5841], [object_number=4, tag_tid=1, entry_tid=5840], [object_number=7, tag_tid=1, entry_tid=5839]]], state=connected] 2024-04-06T10:04:21.233 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_30.status 2024-04-06T10:04:21.235 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=6, tag_tid=1, entry_tid=5842' 2024-04-06T10:04:21.235 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=5, tag_tid=2, entry_tid=1' -a 'object_number=5, tag_tid=2, entry_tid=1' = 'object_number=6, tag_tid=1, entry_tid=5842' 2024-04-06T10:04:21.235 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=6, tag_tid=1, entry_tid=5842' '!=' 'object_number=6, tag_tid=1, entry_tid=5842' 2024-04-06T10:04:21.235 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T10:04:21.235 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.8 2024-04-06T10:04:21.664 INFO:tasks.rbd_mirror_thrash:reviving cluster2.client.mirror.0 2024-04-06T10:04:21.664 INFO:tasks.rbd_mirror.cluster2.client.mirror.0:Restarting daemon 2024-04-06T10:04:21.664 DEBUG:teuthology.orchestra.run.smithi192:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.0 2024-04-06T10:04:21.667 INFO:tasks.rbd_mirror.cluster2.client.mirror.0:Started 2024-04-06T10:04:21.667 INFO:tasks.rbd_mirror_thrash:waiting for 67 secs before thrashing 2024-04-06T10:04:22.037 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror image_30 2024-04-06T10:04:22.037 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T10:04:22.037 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:04:22.037 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_30 2024-04-06T10:04:22.037 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T10:04:22.037 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n image_30 ']' 2024-04-06T10:04:22.037 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/image_30' 2024-04-06T10:04:22.037 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_30 2024-04-06T10:04:22.037 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T10:04:22.037 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T10:04:22.038 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T10:04:22.038 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T10:04:22.038 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T10:04:22.038 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T10:04:22.038 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T10:04:22.038 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T10:04:22.038 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T10:04:22.040 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T10:04:22.040 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_30 2024-04-06T10:04:22.040 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T10:04:22.040 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T10:04:22.040 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T10:04:22.040 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T10:04:22.040 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T10:04:22.040 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T10:04:22.040 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T10:04:22.040 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T10:04:22.041 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T10:04:22.041 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T10:04:22.041 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T10:04:22.041 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:04:22.041 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T10:04:22.041 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T10:04:22.042 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T10:04:22.042 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T10:04:22.042 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T10:04:22.042 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T10:04:22.042 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T10:04:22.042 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T10:04:22.043 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T10:04:22.043 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T10:04:22.043 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:04:22.043 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T10:04:22.043 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T10:04:22.043 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T10:04:22.043 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T10:04:22.043 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T10:04:22.043 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T10:04:22.044 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T10:04:22.069 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T10:04:22.069 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T10:04:22.069 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T10:04:22.069 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_30 2024-04-06T10:04:22.174 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:Invalid command: unused arguments: ['mirror/image_30'] 2024-04-06T10:04:22.174 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:rbd mirror flush : flush rbd mirror 2024-04-06T10:04:22.174 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:admin_socket: invalid command 2024-04-06T10:04:22.176 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T10:04:22.176 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/image_30 2024-04-06T10:04:22.176 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:1 2024-04-06T10:04:22.176 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T10:04:22.176 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T10:04:22.176 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-06T10:04:22.176 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:1 2024-04-06T10:04:22.177 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T10:04:22.177 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T10:04:22.177 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T10:04:22.177 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=1 2024-04-06T10:04:22.177 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-06T10:04:22.177 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T10:04:22.177 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:04:22.177 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=1 2024-04-06T10:04:22.177 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=1 2024-04-06T10:04:22.179 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-06T10:04:22.179 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:1 2024-04-06T10:04:22.179 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T10:04:22.179 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T10:04:22.179 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-06T10:04:22.179 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:1 2024-04-06T10:04:22.179 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T10:04:22.179 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T10:04:22.180 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:04:22.180 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=1 2024-04-06T10:04:22.180 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-06T10:04:22.180 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T10:04:22.180 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T10:04:22.180 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=1 2024-04-06T10:04:22.180 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=1 2024-04-06T10:04:22.180 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-06T10:04:22.197 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T10:04:22.197 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T10:04:22.197 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T10:04:22.197 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/image_30 2024-04-06T10:04:22.339 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T10:04:22.340 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror image_30 2024-04-06T10:04:22.340 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:04:22.340 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:04:22.340 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_30 2024-04-06T10:04:22.340 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror image_30 '' 2024-04-06T10:04:22.341 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:04:22.341 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:04:22.341 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_30 2024-04-06T10:04:22.341 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T10:04:22.341 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-image_30.status 2024-04-06T10:04:22.341 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-image_30.status 2024-04-06T10:04:22.341 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T10:04:22.343 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-image_30.status 2024-04-06T10:04:22.343 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_30 2024-04-06T10:04:22.343 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_30.status 2024-04-06T10:04:22.389 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 1 2024-04-06T10:04:22.389 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 1 2024-04-06T10:04:22.389 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T10:04:22.389 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=5, tag_tid=2, entry_tid=1], [object_number=4, tag_tid=2, entry_tid=0], [object_number=6, tag_tid=1, entry_tid=5842], [object_number=7, tag_tid=1, entry_tid=5839]]], state=connected] 2024-04-06T10:04:22.389 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=6, tag_tid=1, entry_tid=5842], [object_number=5, tag_tid=1, entry_tid=5841], [object_number=4, tag_tid=1, entry_tid=5840], [object_number=7, tag_tid=1, entry_tid=5839]]], state=connected] 2024-04-06T10:04:22.392 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_30.status 2024-04-06T10:04:22.395 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=5, tag_tid=2, entry_tid=1' 2024-04-06T10:04:22.395 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror image_30 2024-04-06T10:04:22.395 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:04:22.395 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:04:22.395 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_30 2024-04-06T10:04:22.395 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror image_30 '..*' 2024-04-06T10:04:22.395 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:04:22.395 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:04:22.395 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_30 2024-04-06T10:04:22.396 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T10:04:22.396 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-image_30.status 2024-04-06T10:04:22.396 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-image_30.status 2024-04-06T10:04:22.396 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T10:04:22.396 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-image_30.status 2024-04-06T10:04:22.396 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_30 2024-04-06T10:04:22.397 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_30.status 2024-04-06T10:04:22.441 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 1 2024-04-06T10:04:22.441 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 1 2024-04-06T10:04:22.441 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T10:04:22.441 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=5, tag_tid=2, entry_tid=1], [object_number=4, tag_tid=2, entry_tid=0], [object_number=6, tag_tid=1, entry_tid=5842], [object_number=7, tag_tid=1, entry_tid=5839]]], state=connected] 2024-04-06T10:04:22.441 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=6, tag_tid=1, entry_tid=5842], [object_number=5, tag_tid=1, entry_tid=5841], [object_number=4, tag_tid=1, entry_tid=5840], [object_number=7, tag_tid=1, entry_tid=5839]]], state=connected] 2024-04-06T10:04:22.444 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_30.status 2024-04-06T10:04:22.446 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=6, tag_tid=1, entry_tid=5842' 2024-04-06T10:04:22.446 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=5, tag_tid=2, entry_tid=1' -a 'object_number=5, tag_tid=2, entry_tid=1' = 'object_number=6, tag_tid=1, entry_tid=5842' 2024-04-06T10:04:22.446 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=6, tag_tid=1, entry_tid=5842' '!=' 'object_number=6, tag_tid=1, entry_tid=5842' 2024-04-06T10:04:22.446 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T10:04:22.446 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 1.6 2024-04-06T10:04:24.048 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror image_30 2024-04-06T10:04:24.048 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T10:04:24.049 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:04:24.049 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_30 2024-04-06T10:04:24.049 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T10:04:24.049 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n image_30 ']' 2024-04-06T10:04:24.049 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/image_30' 2024-04-06T10:04:24.049 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_30 2024-04-06T10:04:24.049 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T10:04:24.049 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T10:04:24.049 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T10:04:24.049 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T10:04:24.049 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T10:04:24.049 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T10:04:24.049 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T10:04:24.051 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T10:04:24.051 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T10:04:24.051 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T10:04:24.051 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_30 2024-04-06T10:04:24.051 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T10:04:24.051 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T10:04:24.051 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T10:04:24.051 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T10:04:24.051 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T10:04:24.052 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T10:04:24.052 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T10:04:24.052 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T10:04:24.052 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T10:04:24.052 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T10:04:24.052 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T10:04:24.052 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:04:24.052 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T10:04:24.052 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T10:04:24.053 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T10:04:24.053 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T10:04:24.053 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T10:04:24.054 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T10:04:24.054 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T10:04:24.054 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T10:04:24.054 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T10:04:24.054 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T10:04:24.054 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:04:24.054 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T10:04:24.054 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T10:04:24.054 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T10:04:24.054 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T10:04:24.054 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T10:04:24.054 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T10:04:24.054 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T10:04:24.079 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T10:04:24.079 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T10:04:24.079 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T10:04:24.079 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_30 2024-04-06T10:04:24.185 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:Invalid command: unused arguments: ['mirror/image_30'] 2024-04-06T10:04:24.185 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:rbd mirror flush : flush rbd mirror 2024-04-06T10:04:24.186 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:admin_socket: invalid command 2024-04-06T10:04:24.189 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T10:04:24.189 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/image_30 2024-04-06T10:04:24.190 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:1 2024-04-06T10:04:24.190 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T10:04:24.190 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T10:04:24.190 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-06T10:04:24.190 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:1 2024-04-06T10:04:24.190 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T10:04:24.190 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T10:04:24.190 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T10:04:24.190 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=1 2024-04-06T10:04:24.190 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-06T10:04:24.190 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T10:04:24.190 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:04:24.190 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=1 2024-04-06T10:04:24.190 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=1 2024-04-06T10:04:24.191 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-06T10:04:24.191 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:1 2024-04-06T10:04:24.191 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T10:04:24.191 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T10:04:24.191 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-06T10:04:24.191 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:1 2024-04-06T10:04:24.191 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T10:04:24.191 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T10:04:24.191 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:04:24.191 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=1 2024-04-06T10:04:24.191 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-06T10:04:24.191 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T10:04:24.191 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T10:04:24.191 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=1 2024-04-06T10:04:24.191 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=1 2024-04-06T10:04:24.192 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-06T10:04:24.213 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T10:04:24.213 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T10:04:24.213 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T10:04:24.213 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/image_30 2024-04-06T10:04:24.356 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T10:04:24.357 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror image_30 2024-04-06T10:04:24.357 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:04:24.358 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:04:24.358 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_30 2024-04-06T10:04:24.358 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror image_30 '' 2024-04-06T10:04:24.358 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:04:24.358 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:04:24.358 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_30 2024-04-06T10:04:24.358 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T10:04:24.358 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-image_30.status 2024-04-06T10:04:24.359 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-image_30.status 2024-04-06T10:04:24.359 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T10:04:24.361 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-image_30.status 2024-04-06T10:04:24.361 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_30 2024-04-06T10:04:24.361 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_30.status 2024-04-06T10:04:24.404 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 1 2024-04-06T10:04:24.404 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 1 2024-04-06T10:04:24.404 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T10:04:24.405 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=5, tag_tid=2, entry_tid=1], [object_number=4, tag_tid=2, entry_tid=0], [object_number=6, tag_tid=1, entry_tid=5842], [object_number=7, tag_tid=1, entry_tid=5839]]], state=connected] 2024-04-06T10:04:24.405 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=6, tag_tid=1, entry_tid=5842], [object_number=5, tag_tid=1, entry_tid=5841], [object_number=4, tag_tid=1, entry_tid=5840], [object_number=7, tag_tid=1, entry_tid=5839]]], state=connected] 2024-04-06T10:04:24.408 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_30.status 2024-04-06T10:04:24.409 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=5, tag_tid=2, entry_tid=1' 2024-04-06T10:04:24.410 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror image_30 2024-04-06T10:04:24.410 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:04:24.411 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:04:24.411 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_30 2024-04-06T10:04:24.411 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror image_30 '..*' 2024-04-06T10:04:24.411 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:04:24.411 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:04:24.411 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_30 2024-04-06T10:04:24.411 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T10:04:24.411 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-image_30.status 2024-04-06T10:04:24.411 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-image_30.status 2024-04-06T10:04:24.411 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T10:04:24.412 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-image_30.status 2024-04-06T10:04:24.412 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_30 2024-04-06T10:04:24.412 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_30.status 2024-04-06T10:04:24.456 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 1 2024-04-06T10:04:24.456 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 1 2024-04-06T10:04:24.456 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T10:04:24.456 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=5, tag_tid=2, entry_tid=1], [object_number=4, tag_tid=2, entry_tid=0], [object_number=6, tag_tid=1, entry_tid=5842], [object_number=7, tag_tid=1, entry_tid=5839]]], state=connected] 2024-04-06T10:04:24.456 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=6, tag_tid=1, entry_tid=5842], [object_number=5, tag_tid=1, entry_tid=5841], [object_number=4, tag_tid=1, entry_tid=5840], [object_number=7, tag_tid=1, entry_tid=5839]]], state=connected] 2024-04-06T10:04:24.459 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_30.status 2024-04-06T10:04:24.461 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=6, tag_tid=1, entry_tid=5842' 2024-04-06T10:04:24.461 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=5, tag_tid=2, entry_tid=1' -a 'object_number=5, tag_tid=2, entry_tid=1' = 'object_number=6, tag_tid=1, entry_tid=5842' 2024-04-06T10:04:24.461 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=6, tag_tid=1, entry_tid=5842' '!=' 'object_number=6, tag_tid=1, entry_tid=5842' 2024-04-06T10:04:24.461 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T10:04:24.461 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 2 2024-04-06T10:04:26.463 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror image_30 2024-04-06T10:04:26.463 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T10:04:26.463 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:04:26.463 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_30 2024-04-06T10:04:26.463 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T10:04:26.463 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n image_30 ']' 2024-04-06T10:04:26.463 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/image_30' 2024-04-06T10:04:26.463 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_30 2024-04-06T10:04:26.463 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T10:04:26.464 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T10:04:26.464 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T10:04:26.464 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T10:04:26.464 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T10:04:26.464 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T10:04:26.464 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T10:04:26.464 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T10:04:26.465 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T10:04:26.466 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T10:04:26.467 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_30 2024-04-06T10:04:26.467 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T10:04:26.467 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T10:04:26.467 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T10:04:26.467 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T10:04:26.467 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T10:04:26.467 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T10:04:26.467 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T10:04:26.467 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T10:04:26.467 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T10:04:26.467 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T10:04:26.467 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T10:04:26.467 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:04:26.467 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T10:04:26.467 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T10:04:26.469 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T10:04:26.469 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T10:04:26.469 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T10:04:26.469 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T10:04:26.469 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T10:04:26.469 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T10:04:26.469 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T10:04:26.469 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T10:04:26.469 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:04:26.469 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T10:04:26.470 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T10:04:26.470 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T10:04:26.470 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T10:04:26.470 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T10:04:26.471 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T10:04:26.471 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T10:04:26.494 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T10:04:26.495 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T10:04:26.495 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T10:04:26.495 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_30 2024-04-06T10:04:26.601 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:Invalid command: unused arguments: ['mirror/image_30'] 2024-04-06T10:04:26.602 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:rbd mirror flush : flush rbd mirror 2024-04-06T10:04:26.602 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:admin_socket: invalid command 2024-04-06T10:04:26.603 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T10:04:26.603 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/image_30 2024-04-06T10:04:26.603 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:1 2024-04-06T10:04:26.603 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T10:04:26.603 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T10:04:26.604 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-06T10:04:26.604 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:1 2024-04-06T10:04:26.604 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T10:04:26.604 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T10:04:26.604 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T10:04:26.604 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=1 2024-04-06T10:04:26.604 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-06T10:04:26.604 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T10:04:26.604 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:04:26.604 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=1 2024-04-06T10:04:26.604 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=1 2024-04-06T10:04:26.605 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-06T10:04:26.606 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:1 2024-04-06T10:04:26.606 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T10:04:26.606 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T10:04:26.606 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-06T10:04:26.606 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:1 2024-04-06T10:04:26.606 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T10:04:26.606 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T10:04:26.606 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:04:26.606 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=1 2024-04-06T10:04:26.606 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-06T10:04:26.606 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T10:04:26.606 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T10:04:26.606 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=1 2024-04-06T10:04:26.607 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=1 2024-04-06T10:04:26.607 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-06T10:04:26.630 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T10:04:26.630 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T10:04:26.630 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T10:04:26.630 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/image_30 2024-04-06T10:04:26.663 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.0 with skip (0.6173733797283996) > weight (0.25) 2024-04-06T10:04:26.663 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.1 with skip (0.7112196484790932) > weight (0.25) 2024-04-06T10:04:26.663 INFO:tasks.rbd_mirror_thrash:kill cluster1.client.mirror.2 2024-04-06T10:04:26.663 INFO:tasks.rbd_mirror.cluster1.client.mirror.2:Sent signal 15 2024-04-06T10:04:26.664 INFO:tasks.rbd_mirror_thrash:waiting for 5 secs before reviving daemons 2024-04-06T10:04:26.664 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi192.stderr:2024-04-06T10:04:26.661+0000 7f41c3c63640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.2 (PID: 130990) UID: 1000 2024-04-06T10:04:26.784 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T10:04:26.785 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror image_30 2024-04-06T10:04:26.785 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:04:26.785 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:04:26.785 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_30 2024-04-06T10:04:26.785 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror image_30 '' 2024-04-06T10:04:26.785 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:04:26.786 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:04:26.786 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_30 2024-04-06T10:04:26.786 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T10:04:26.786 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-image_30.status 2024-04-06T10:04:26.786 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-image_30.status 2024-04-06T10:04:26.786 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T10:04:26.787 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-image_30.status 2024-04-06T10:04:26.788 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_30 2024-04-06T10:04:26.788 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_30.status 2024-04-06T10:04:26.843 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 1 2024-04-06T10:04:26.843 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 1 2024-04-06T10:04:26.843 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T10:04:26.843 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=5, tag_tid=2, entry_tid=1], [object_number=4, tag_tid=2, entry_tid=0], [object_number=6, tag_tid=1, entry_tid=5842], [object_number=7, tag_tid=1, entry_tid=5839]]], state=connected] 2024-04-06T10:04:26.843 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=5, tag_tid=2, entry_tid=1], [object_number=4, tag_tid=2, entry_tid=0], [object_number=6, tag_tid=1, entry_tid=5842], [object_number=7, tag_tid=1, entry_tid=5839]]], state=connected] 2024-04-06T10:04:26.847 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_30.status 2024-04-06T10:04:26.848 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=5, tag_tid=2, entry_tid=1' 2024-04-06T10:04:26.850 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror image_30 2024-04-06T10:04:26.850 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:04:26.850 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:04:26.850 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_30 2024-04-06T10:04:26.850 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror image_30 '..*' 2024-04-06T10:04:26.850 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:04:26.850 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:04:26.850 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_30 2024-04-06T10:04:26.851 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T10:04:26.851 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-image_30.status 2024-04-06T10:04:26.851 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-image_30.status 2024-04-06T10:04:26.851 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T10:04:26.851 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-image_30.status 2024-04-06T10:04:26.852 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_30 2024-04-06T10:04:26.852 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_30.status 2024-04-06T10:04:26.900 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 1 2024-04-06T10:04:26.900 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 1 2024-04-06T10:04:26.900 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T10:04:26.900 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=5, tag_tid=2, entry_tid=1], [object_number=4, tag_tid=2, entry_tid=0], [object_number=6, tag_tid=1, entry_tid=5842], [object_number=7, tag_tid=1, entry_tid=5839]]], state=connected] 2024-04-06T10:04:26.900 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=5, tag_tid=2, entry_tid=1], [object_number=4, tag_tid=2, entry_tid=0], [object_number=6, tag_tid=1, entry_tid=5842], [object_number=7, tag_tid=1, entry_tid=5839]]], state=connected] 2024-04-06T10:04:26.904 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_30.status 2024-04-06T10:04:26.906 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=5, tag_tid=2, entry_tid=1' 2024-04-06T10:04:26.906 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=5, tag_tid=2, entry_tid=1' -a 'object_number=5, tag_tid=2, entry_tid=1' = 'object_number=5, tag_tid=2, entry_tid=1' 2024-04-06T10:04:26.906 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T10:04:26.906 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_snap_present cluster1 mirror image_30 snap 2024-04-06T10:04:26.906 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T10:04:26.906 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:04:26.907 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_30 2024-04-06T10:04:26.907 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local snap_name=snap 2024-04-06T10:04:26.907 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local s 2024-04-06T10:04:26.907 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-04-06T10:04:26.907 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 1 2024-04-06T10:04:27.906 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 info mirror/image_30@snap 2024-04-06T10:04:27.979 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:rbd image 'image_30': 2024-04-06T10:04:27.979 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: size 128 MiB in 32 objects 2024-04-06T10:04:27.979 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: order 22 (4 MiB objects) 2024-04-06T10:04:27.979 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: snapshot_count: 1 2024-04-06T10:04:27.979 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: id: 10ac60980fd9 2024-04-06T10:04:27.979 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: block_name_prefix: rbd_data.10ac60980fd9 2024-04-06T10:04:27.979 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: format: 2 2024-04-06T10:04:27.979 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: features: layering, exclusive-lock, journaling 2024-04-06T10:04:27.980 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: op_features: 2024-04-06T10:04:27.980 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: flags: 2024-04-06T10:04:27.980 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: create_timestamp: Sat Apr 6 09:55:18 2024 2024-04-06T10:04:27.980 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: access_timestamp: Sat Apr 6 09:55:18 2024 2024-04-06T10:04:27.980 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: modify_timestamp: Sat Apr 6 09:55:18 2024 2024-04-06T10:04:27.980 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: protected: False 2024-04-06T10:04:27.980 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: journal: 10ac60980fd9 2024-04-06T10:04:27.980 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: mirroring state: enabled 2024-04-06T10:04:27.980 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: mirroring mode: journal 2024-04-06T10:04:27.980 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: mirroring global id: b99cbdf9-e735-45a8-ab79-1c02c602482b 2024-04-06T10:04:27.980 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: mirroring primary: false 2024-04-06T10:04:27.984 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T10:04:27.984 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ compare_image_snaps mirror image_30 snap 2024-04-06T10:04:27.984 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:04:27.984 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_30 2024-04-06T10:04:27.984 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local snap_name=snap 2024-04-06T10:04:27.984 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local ret=0 2024-04-06T10:04:27.984 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local rmt_export=/tmp/tmp.BhW6xKmROU/cluster2-mirror-image_30.export 2024-04-06T10:04:27.984 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loc_export=/tmp/tmp.BhW6xKmROU/cluster1-mirror-image_30.export 2024-04-06T10:04:27.984 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rm -f /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_30.export /tmp/tmp.BhW6xKmROU/cluster1-mirror-image_30.export 2024-04-06T10:04:27.984 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster2 -p mirror export image_30@snap /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_30.export 2024-04-06T10:04:28.139 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-04-06T10:04:28.147 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 -p mirror export image_30@snap /tmp/tmp.BhW6xKmROU/cluster1-mirror-image_30.export 2024-04-06T10:04:28.474 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-04-06T10:04:28.488 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmp /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_30.export /tmp/tmp.BhW6xKmROU/cluster1-mirror-image_30.export 2024-04-06T10:04:28.566 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rm -f /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_30.export /tmp/tmp.BhW6xKmROU/cluster1-mirror-image_30.export 2024-04-06T10:04:28.608 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T10:04:28.608 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-06T10:04:28.608 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ image=image_31 2024-04-06T10:04:28.608 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ create_snap cluster2 mirror image_31 snap 2024-04-06T10:04:28.608 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-04-06T10:04:28.608 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:04:28.609 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_31 2024-04-06T10:04:28.609 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local snap_name=snap 2024-04-06T10:04:28.609 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster2 -p mirror snap create image_31@snap --debug-rbd=20 --debug-journaler=20 2024-04-06T10:04:28.903 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_image_replay_started cluster1 mirror image_31 2024-04-06T10:04:28.903 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T10:04:28.904 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:04:28.904 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_31 2024-04-06T10:04:28.904 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_image_replay_state cluster1 mirror image_31 started 2024-04-06T10:04:28.904 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T10:04:28.904 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:04:28.904 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_31 2024-04-06T10:04:28.904 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state=started 2024-04-06T10:04:28.904 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local s 2024-04-06T10:04:28.904 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T10:04:28.904 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 1 2024-04-06T10:04:29.905 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test_image_replay_state cluster1 mirror image_31 started 2024-04-06T10:04:29.905 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T10:04:29.905 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:04:29.905 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_31 2024-04-06T10:04:29.905 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local test_state=started 2024-04-06T10:04:29.906 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local status_result 2024-04-06T10:04:29.906 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local current_state=stopped 2024-04-06T10:04:29.907 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_31 2024-04-06T10:04:29.907 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -i state 2024-04-06T10:04:29.907 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_instance=cluster1 2024-04-06T10:04:29.907 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ shift 2024-04-06T10:04:29.907 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T10:04:29.907 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance=cluster1 2024-04-06T10:04:29.907 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local loop_instance 2024-04-06T10:04:29.907 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T10:04:29.907 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sleep 0 2024-04-06T10:04:29.908 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-04-06T10:04:29.909 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ seq 0 3 2024-04-06T10:04:29.912 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T10:04:29.912 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_31 2024-04-06T10:04:29.912 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1:0 2024-04-06T10:04:29.912 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ shift 2024-04-06T10:04:29.912 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T10:04:29.912 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-04-06T10:04:29.912 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T10:04:29.912 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=cluster 2024-04-06T10:04:29.912 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T10:04:29.912 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:04:29.912 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T10:04:29.912 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T10:04:29.912 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval cluster=cluster1 2024-04-06T10:04:29.913 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ cluster=cluster1 2024-04-06T10:04:29.913 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T10:04:29.913 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T10:04:29.913 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-04-06T10:04:29.913 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local local_cluster=cluster1:0 2024-04-06T10:04:29.913 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local cluster=cluster1 2024-04-06T10:04:29.913 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local instance 2024-04-06T10:04:29.913 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T10:04:29.913 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local val=cluster1:0 2024-04-06T10:04:29.913 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local cluster_var_name=local_cluster 2024-04-06T10:04:29.913 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local instance_var_name=instance 2024-04-06T10:04:29.913 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ cluster=cluster1 2024-04-06T10:04:29.913 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T10:04:29.913 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ '[' 0 = cluster1:0 ']' 2024-04-06T10:04:29.914 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ eval local_cluster=cluster1 2024-04-06T10:04:29.914 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++++ local_cluster=cluster1 2024-04-06T10:04:29.914 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ eval instance=0 2024-04-06T10:04:29.914 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++++ instance=0 2024-04-06T10:04:29.914 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T10:04:29.933 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T10:04:29.934 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T10:04:29.934 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T10:04:29.934 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_31 2024-04-06T10:04:30.043 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:Invalid command: unused arguments: ['mirror/image_31'] 2024-04-06T10:04:30.043 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:rbd mirror status : get status for rbd mirror 2024-04-06T10:04:30.044 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:admin_socket: invalid command 2024-04-06T10:04:30.045 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T10:04:30.046 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_31 2024-04-06T10:04:30.046 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1:1 2024-04-06T10:04:30.046 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ shift 2024-04-06T10:04:30.046 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T10:04:30.046 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-04-06T10:04:30.046 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:1 2024-04-06T10:04:30.046 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=cluster 2024-04-06T10:04:30.046 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T10:04:30.046 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:04:30.046 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=1 2024-04-06T10:04:30.046 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-06T10:04:30.047 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval cluster=cluster1 2024-04-06T10:04:30.047 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ cluster=cluster1 2024-04-06T10:04:30.047 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=1 2024-04-06T10:04:30.047 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=1 2024-04-06T10:04:30.048 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-04-06T10:04:30.048 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local local_cluster=cluster1:1 2024-04-06T10:04:30.048 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local cluster=cluster1 2024-04-06T10:04:30.048 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local instance 2024-04-06T10:04:30.049 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-06T10:04:30.049 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local val=cluster1:1 2024-04-06T10:04:30.049 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local cluster_var_name=local_cluster 2024-04-06T10:04:30.049 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local instance_var_name=instance 2024-04-06T10:04:30.049 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ cluster=cluster1 2024-04-06T10:04:30.049 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=1 2024-04-06T10:04:30.049 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ '[' 1 = cluster1:1 ']' 2024-04-06T10:04:30.049 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ eval local_cluster=cluster1 2024-04-06T10:04:30.049 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++++ local_cluster=cluster1 2024-04-06T10:04:30.049 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ eval instance=1 2024-04-06T10:04:30.049 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++++ instance=1 2024-04-06T10:04:30.049 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-06T10:04:30.067 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T10:04:30.068 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T10:04:30.068 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T10:04:30.068 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_31 2024-04-06T10:04:30.223 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ return 0 2024-04-06T10:04:30.224 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ status_result=' "state": "Replaying"' 2024-04-06T10:04:30.224 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ echo ' "state": "Replaying"' 2024-04-06T10:04:30.224 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep -i Replaying 2024-04-06T10:04:30.226 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: "state": "Replaying" 2024-04-06T10:04:30.229 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ current_state=started 2024-04-06T10:04:30.229 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test started = started 2024-04-06T10:04:30.229 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T10:04:30.229 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_31 2024-04-06T10:04:30.229 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local local_cluster=cluster1 2024-04-06T10:04:30.229 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-04-06T10:04:30.229 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:04:30.229 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_31 2024-04-06T10:04:30.229 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' journal = journal ']' 2024-04-06T10:04:30.229 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_journal_replay_complete cluster1 cluster2 mirror image_31 2024-04-06T10:04:30.230 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local local_cluster=cluster1 2024-04-06T10:04:30.230 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-04-06T10:04:30.230 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:04:30.230 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_31 2024-04-06T10:04:30.230 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local s master_pos mirror_pos last_mirror_pos 2024-04-06T10:04:30.230 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local master_tag master_entry mirror_tag mirror_entry 2024-04-06T10:04:30.230 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T10:04:30.230 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T10:04:30.230 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T10:04:30.429 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror image_31 2024-04-06T10:04:30.429 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T10:04:30.429 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:04:30.429 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_31 2024-04-06T10:04:30.429 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T10:04:30.430 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n image_31 ']' 2024-04-06T10:04:30.430 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/image_31' 2024-04-06T10:04:30.430 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_31 2024-04-06T10:04:30.430 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T10:04:30.430 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T10:04:30.430 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T10:04:30.430 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T10:04:30.430 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T10:04:30.430 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T10:04:30.430 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T10:04:30.431 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T10:04:30.431 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T10:04:30.434 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T10:04:30.434 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_31 2024-04-06T10:04:30.434 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T10:04:30.434 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T10:04:30.434 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T10:04:30.435 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T10:04:30.435 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T10:04:30.435 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T10:04:30.435 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T10:04:30.435 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T10:04:30.435 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T10:04:30.435 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T10:04:30.435 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T10:04:30.435 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:04:30.435 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T10:04:30.435 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T10:04:30.435 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T10:04:30.435 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T10:04:30.435 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T10:04:30.436 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T10:04:30.436 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T10:04:30.436 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T10:04:30.436 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T10:04:30.436 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T10:04:30.437 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:04:30.437 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T10:04:30.437 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T10:04:30.437 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T10:04:30.437 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T10:04:30.437 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T10:04:30.437 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T10:04:30.437 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T10:04:30.460 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T10:04:30.460 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T10:04:30.460 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T10:04:30.460 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_31 2024-04-06T10:04:30.566 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:Invalid command: unused arguments: ['mirror/image_31'] 2024-04-06T10:04:30.566 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:rbd mirror flush : flush rbd mirror 2024-04-06T10:04:30.566 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:admin_socket: invalid command 2024-04-06T10:04:30.568 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T10:04:30.569 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/image_31 2024-04-06T10:04:30.569 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:1 2024-04-06T10:04:30.569 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T10:04:30.569 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T10:04:30.569 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-06T10:04:30.569 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:1 2024-04-06T10:04:30.569 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T10:04:30.569 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T10:04:30.569 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T10:04:30.569 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=1 2024-04-06T10:04:30.569 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-06T10:04:30.569 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T10:04:30.569 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:04:30.569 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=1 2024-04-06T10:04:30.570 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=1 2024-04-06T10:04:30.571 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-06T10:04:30.571 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:1 2024-04-06T10:04:30.571 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T10:04:30.571 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T10:04:30.571 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-06T10:04:30.571 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:1 2024-04-06T10:04:30.571 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T10:04:30.572 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T10:04:30.572 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:04:30.572 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=1 2024-04-06T10:04:30.572 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-06T10:04:30.572 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T10:04:30.572 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T10:04:30.572 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=1 2024-04-06T10:04:30.572 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=1 2024-04-06T10:04:30.572 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-06T10:04:30.593 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T10:04:30.594 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T10:04:30.594 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T10:04:30.594 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/image_31 2024-04-06T10:04:30.720 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi192.stderr:2024-04-06T10:04:30.717+0000 7f3d223c1640 -1 rbd::mirror::image_replayer::journal::Replayer: 0x56067e094a80 handle_flush_local_replay: error flushing local replay: (108) Cannot send after transport endpoint shutdown 2024-04-06T10:04:30.734 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T10:04:30.735 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror image_31 2024-04-06T10:04:30.735 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:04:30.735 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:04:30.735 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_31 2024-04-06T10:04:30.735 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror image_31 '' 2024-04-06T10:04:30.735 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:04:30.735 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:04:30.735 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_31 2024-04-06T10:04:30.736 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T10:04:30.736 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-image_31.status 2024-04-06T10:04:30.736 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-image_31.status 2024-04-06T10:04:30.736 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T10:04:30.738 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-image_31.status 2024-04-06T10:04:30.739 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_31 2024-04-06T10:04:30.739 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_31.status 2024-04-06T10:04:30.790 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 0 2024-04-06T10:04:30.790 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 0 2024-04-06T10:04:30.790 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T10:04:30.790 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=1, tag_tid=3, entry_tid=1], [object_number=0, tag_tid=3, entry_tid=0], [object_number=3, tag_tid=1, entry_tid=2635], [object_number=2, tag_tid=1, entry_tid=2634]]], state=connected] 2024-04-06T10:04:30.790 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=3, tag_tid=1, entry_tid=2411], [object_number=2, tag_tid=1, entry_tid=2410], [object_number=1, tag_tid=1, entry_tid=2409], [object_number=0, tag_tid=1, entry_tid=2408]]], state=connected] 2024-04-06T10:04:30.793 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_31.status 2024-04-06T10:04:30.794 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=1, tag_tid=3, entry_tid=1' 2024-04-06T10:04:30.795 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror image_31 2024-04-06T10:04:30.795 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:04:30.795 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:04:30.795 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_31 2024-04-06T10:04:30.796 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror image_31 '..*' 2024-04-06T10:04:30.796 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:04:30.796 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:04:30.796 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_31 2024-04-06T10:04:30.796 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T10:04:30.796 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-image_31.status 2024-04-06T10:04:30.796 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-image_31.status 2024-04-06T10:04:30.796 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T10:04:30.797 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-image_31.status 2024-04-06T10:04:30.797 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_31 2024-04-06T10:04:30.797 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_31.status 2024-04-06T10:04:30.844 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 0 2024-04-06T10:04:30.844 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 0 2024-04-06T10:04:30.844 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T10:04:30.844 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=1, tag_tid=3, entry_tid=1], [object_number=0, tag_tid=3, entry_tid=0], [object_number=3, tag_tid=1, entry_tid=2635], [object_number=2, tag_tid=1, entry_tid=2634]]], state=connected] 2024-04-06T10:04:30.844 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=3, tag_tid=1, entry_tid=2411], [object_number=2, tag_tid=1, entry_tid=2410], [object_number=1, tag_tid=1, entry_tid=2409], [object_number=0, tag_tid=1, entry_tid=2408]]], state=connected] 2024-04-06T10:04:30.848 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_31.status 2024-04-06T10:04:30.850 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=3, tag_tid=1, entry_tid=2411' 2024-04-06T10:04:30.850 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=1, tag_tid=3, entry_tid=1' -a 'object_number=1, tag_tid=3, entry_tid=1' = 'object_number=3, tag_tid=1, entry_tid=2411' 2024-04-06T10:04:30.850 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=3, tag_tid=1, entry_tid=2411' '!=' '' 2024-04-06T10:04:30.850 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T10:04:30.850 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=3, tag_tid=1, entry_tid=2411' = '' 2024-04-06T10:04:30.850 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=3, tag_tid=1, entry_tid=2411' 2024-04-06T10:04:30.851 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=1, tag_tid=3, entry_tid=1' 2024-04-06T10:04:30.851 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T10:04:30.851 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T10:04:30.851 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=3 2024-04-06T10:04:30.852 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=3, tag_tid=1, entry_tid=2411' 2024-04-06T10:04:30.852 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T10:04:30.852 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T10:04:30.853 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=1 2024-04-06T10:04:30.854 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=1, tag_tid=3, entry_tid=1' 2024-04-06T10:04:30.854 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T10:04:30.854 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T10:04:30.854 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T10:04:30.855 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=3, tag_tid=1, entry_tid=2411' 2024-04-06T10:04:30.855 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T10:04:30.855 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T10:04:30.857 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=2411 2024-04-06T10:04:30.857 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 3 = 1 -a 1 -le 2411 2024-04-06T10:04:30.857 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T10:04:30.857 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T10:04:30.857 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T10:04:31.058 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror image_31 2024-04-06T10:04:31.058 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T10:04:31.058 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:04:31.059 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_31 2024-04-06T10:04:31.059 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T10:04:31.059 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n image_31 ']' 2024-04-06T10:04:31.059 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/image_31' 2024-04-06T10:04:31.059 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_31 2024-04-06T10:04:31.059 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T10:04:31.059 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T10:04:31.059 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T10:04:31.059 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T10:04:31.059 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T10:04:31.059 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T10:04:31.059 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T10:04:31.060 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T10:04:31.060 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T10:04:31.064 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T10:04:31.064 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_31 2024-04-06T10:04:31.064 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T10:04:31.064 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T10:04:31.064 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T10:04:31.064 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T10:04:31.064 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T10:04:31.064 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T10:04:31.064 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T10:04:31.064 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T10:04:31.064 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T10:04:31.065 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T10:04:31.065 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T10:04:31.065 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:04:31.065 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T10:04:31.065 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T10:04:31.065 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T10:04:31.065 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T10:04:31.065 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T10:04:31.065 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T10:04:31.065 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T10:04:31.065 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T10:04:31.065 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T10:04:31.065 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T10:04:31.065 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:04:31.065 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T10:04:31.066 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T10:04:31.066 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T10:04:31.066 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T10:04:31.066 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T10:04:31.066 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T10:04:31.066 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T10:04:31.089 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T10:04:31.089 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T10:04:31.089 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T10:04:31.089 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_31 2024-04-06T10:04:31.194 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:Invalid command: unused arguments: ['mirror/image_31'] 2024-04-06T10:04:31.194 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:rbd mirror flush : flush rbd mirror 2024-04-06T10:04:31.194 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:admin_socket: invalid command 2024-04-06T10:04:31.196 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T10:04:31.197 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/image_31 2024-04-06T10:04:31.197 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:1 2024-04-06T10:04:31.197 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T10:04:31.197 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T10:04:31.197 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-06T10:04:31.197 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:1 2024-04-06T10:04:31.197 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T10:04:31.197 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T10:04:31.197 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T10:04:31.197 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=1 2024-04-06T10:04:31.197 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-06T10:04:31.197 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T10:04:31.197 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:04:31.198 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=1 2024-04-06T10:04:31.198 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=1 2024-04-06T10:04:31.199 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-06T10:04:31.199 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:1 2024-04-06T10:04:31.199 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T10:04:31.199 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T10:04:31.199 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-06T10:04:31.199 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:1 2024-04-06T10:04:31.199 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T10:04:31.199 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T10:04:31.199 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:04:31.199 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=1 2024-04-06T10:04:31.199 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-06T10:04:31.199 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T10:04:31.199 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T10:04:31.200 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=1 2024-04-06T10:04:31.200 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=1 2024-04-06T10:04:31.200 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-06T10:04:31.218 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T10:04:31.218 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T10:04:31.218 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T10:04:31.218 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/image_31 2024-04-06T10:04:31.368 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T10:04:31.370 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror image_31 2024-04-06T10:04:31.370 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:04:31.370 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:04:31.370 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_31 2024-04-06T10:04:31.370 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror image_31 '' 2024-04-06T10:04:31.371 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:04:31.371 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:04:31.371 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_31 2024-04-06T10:04:31.371 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T10:04:31.371 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-image_31.status 2024-04-06T10:04:31.371 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-image_31.status 2024-04-06T10:04:31.371 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T10:04:31.373 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-image_31.status 2024-04-06T10:04:31.373 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_31 2024-04-06T10:04:31.373 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_31.status 2024-04-06T10:04:31.418 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 0 2024-04-06T10:04:31.418 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 0 2024-04-06T10:04:31.418 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T10:04:31.418 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=1, tag_tid=3, entry_tid=1], [object_number=0, tag_tid=3, entry_tid=0], [object_number=3, tag_tid=1, entry_tid=2635], [object_number=2, tag_tid=1, entry_tid=2634]]], state=connected] 2024-04-06T10:04:31.418 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=1, tag_tid=1, entry_tid=2637], [object_number=0, tag_tid=1, entry_tid=2636], [object_number=3, tag_tid=1, entry_tid=2635], [object_number=2, tag_tid=1, entry_tid=2634]]], state=connected] 2024-04-06T10:04:31.421 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_31.status 2024-04-06T10:04:31.423 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=1, tag_tid=3, entry_tid=1' 2024-04-06T10:04:31.424 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror image_31 2024-04-06T10:04:31.424 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:04:31.424 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:04:31.424 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_31 2024-04-06T10:04:31.424 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror image_31 '..*' 2024-04-06T10:04:31.424 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:04:31.424 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:04:31.424 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_31 2024-04-06T10:04:31.424 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T10:04:31.425 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-image_31.status 2024-04-06T10:04:31.425 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-image_31.status 2024-04-06T10:04:31.425 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T10:04:31.425 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-image_31.status 2024-04-06T10:04:31.426 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_31 2024-04-06T10:04:31.426 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_31.status 2024-04-06T10:04:31.471 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 0 2024-04-06T10:04:31.471 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 0 2024-04-06T10:04:31.471 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T10:04:31.471 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=1, tag_tid=3, entry_tid=1], [object_number=0, tag_tid=3, entry_tid=0], [object_number=3, tag_tid=1, entry_tid=2635], [object_number=2, tag_tid=1, entry_tid=2634]]], state=connected] 2024-04-06T10:04:31.471 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=1, tag_tid=1, entry_tid=2637], [object_number=0, tag_tid=1, entry_tid=2636], [object_number=3, tag_tid=1, entry_tid=2635], [object_number=2, tag_tid=1, entry_tid=2634]]], state=connected] 2024-04-06T10:04:31.475 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_31.status 2024-04-06T10:04:31.477 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=1, tag_tid=1, entry_tid=2637' 2024-04-06T10:04:31.477 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=1, tag_tid=3, entry_tid=1' -a 'object_number=1, tag_tid=3, entry_tid=1' = 'object_number=1, tag_tid=1, entry_tid=2637' 2024-04-06T10:04:31.477 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=1, tag_tid=1, entry_tid=2637' '!=' 'object_number=3, tag_tid=1, entry_tid=2411' 2024-04-06T10:04:31.477 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T10:04:31.477 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=1, tag_tid=1, entry_tid=2637' = 'object_number=3, tag_tid=1, entry_tid=2411' 2024-04-06T10:04:31.477 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=1, tag_tid=1, entry_tid=2637' 2024-04-06T10:04:31.477 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=1, tag_tid=3, entry_tid=1' 2024-04-06T10:04:31.478 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T10:04:31.478 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T10:04:31.478 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=3 2024-04-06T10:04:31.479 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=1, tag_tid=1, entry_tid=2637' 2024-04-06T10:04:31.479 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T10:04:31.479 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T10:04:31.480 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=1 2024-04-06T10:04:31.481 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=1, tag_tid=3, entry_tid=1' 2024-04-06T10:04:31.481 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T10:04:31.481 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T10:04:31.482 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T10:04:31.482 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=1, tag_tid=1, entry_tid=2637' 2024-04-06T10:04:31.483 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T10:04:31.483 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T10:04:31.483 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=2637 2024-04-06T10:04:31.484 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 3 = 1 -a 1 -le 2637 2024-04-06T10:04:31.484 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T10:04:31.484 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T10:04:31.484 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T10:04:31.664 INFO:tasks.rbd_mirror_thrash:waiting for cluster1.client.mirror.2 2024-04-06T10:04:31.665 INFO:teuthology.orchestra.run:waiting for 600 2024-04-06T10:04:31.665 INFO:tasks.rbd_mirror_thrash:reviving cluster1.client.mirror.2 2024-04-06T10:04:31.665 INFO:tasks.rbd_mirror.cluster1.client.mirror.2:Restarting daemon 2024-04-06T10:04:31.665 DEBUG:teuthology.orchestra.run.smithi192:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.2 2024-04-06T10:04:31.668 INFO:tasks.rbd_mirror.cluster1.client.mirror.2:Started 2024-04-06T10:04:31.668 INFO:tasks.rbd_mirror_thrash:waiting for 62 secs before thrashing 2024-04-06T10:04:31.685 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror image_31 2024-04-06T10:04:31.685 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T10:04:31.686 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:04:31.686 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_31 2024-04-06T10:04:31.686 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T10:04:31.686 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n image_31 ']' 2024-04-06T10:04:31.686 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/image_31' 2024-04-06T10:04:31.686 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_31 2024-04-06T10:04:31.686 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T10:04:31.686 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T10:04:31.686 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T10:04:31.686 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T10:04:31.686 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T10:04:31.686 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T10:04:31.686 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T10:04:31.687 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T10:04:31.687 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T10:04:31.689 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T10:04:31.689 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_31 2024-04-06T10:04:31.689 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T10:04:31.689 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T10:04:31.689 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T10:04:31.689 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T10:04:31.689 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T10:04:31.689 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T10:04:31.689 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T10:04:31.689 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T10:04:31.689 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T10:04:31.689 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T10:04:31.689 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T10:04:31.690 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:04:31.690 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T10:04:31.690 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T10:04:31.691 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T10:04:31.691 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T10:04:31.691 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T10:04:31.691 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T10:04:31.691 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T10:04:31.691 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T10:04:31.691 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T10:04:31.691 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T10:04:31.691 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:04:31.691 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T10:04:31.691 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T10:04:31.691 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T10:04:31.691 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T10:04:31.692 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T10:04:31.692 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T10:04:31.692 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T10:04:31.715 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T10:04:31.715 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T10:04:31.715 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T10:04:31.716 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_31 2024-04-06T10:04:31.819 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:Invalid command: unused arguments: ['mirror/image_31'] 2024-04-06T10:04:31.819 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:rbd mirror flush : flush rbd mirror 2024-04-06T10:04:31.819 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:admin_socket: invalid command 2024-04-06T10:04:31.822 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T10:04:31.822 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/image_31 2024-04-06T10:04:31.822 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:1 2024-04-06T10:04:31.822 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T10:04:31.822 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T10:04:31.823 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-06T10:04:31.823 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:1 2024-04-06T10:04:31.823 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T10:04:31.823 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T10:04:31.823 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T10:04:31.823 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=1 2024-04-06T10:04:31.823 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-06T10:04:31.823 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T10:04:31.823 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:04:31.823 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=1 2024-04-06T10:04:31.823 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=1 2024-04-06T10:04:31.823 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-06T10:04:31.823 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:1 2024-04-06T10:04:31.824 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T10:04:31.824 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T10:04:31.824 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-06T10:04:31.824 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:1 2024-04-06T10:04:31.824 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T10:04:31.824 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T10:04:31.824 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:04:31.824 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=1 2024-04-06T10:04:31.824 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-06T10:04:31.824 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T10:04:31.825 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T10:04:31.825 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=1 2024-04-06T10:04:31.825 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=1 2024-04-06T10:04:31.825 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-06T10:04:31.844 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T10:04:31.844 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T10:04:31.844 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T10:04:31.844 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/image_31 2024-04-06T10:04:31.982 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T10:04:31.983 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror image_31 2024-04-06T10:04:31.983 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:04:31.984 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:04:31.984 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_31 2024-04-06T10:04:31.984 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror image_31 '' 2024-04-06T10:04:31.984 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:04:31.984 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:04:31.984 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_31 2024-04-06T10:04:31.984 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T10:04:31.984 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-image_31.status 2024-04-06T10:04:31.985 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-image_31.status 2024-04-06T10:04:31.985 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T10:04:31.986 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-image_31.status 2024-04-06T10:04:31.987 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_31 2024-04-06T10:04:31.987 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_31.status 2024-04-06T10:04:32.031 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 0 2024-04-06T10:04:32.032 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 0 2024-04-06T10:04:32.032 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T10:04:32.032 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=1, tag_tid=3, entry_tid=1], [object_number=0, tag_tid=3, entry_tid=0], [object_number=3, tag_tid=1, entry_tid=2635], [object_number=2, tag_tid=1, entry_tid=2634]]], state=connected] 2024-04-06T10:04:32.032 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=1, tag_tid=1, entry_tid=2637], [object_number=0, tag_tid=1, entry_tid=2636], [object_number=3, tag_tid=1, entry_tid=2635], [object_number=2, tag_tid=1, entry_tid=2634]]], state=connected] 2024-04-06T10:04:32.035 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_31.status 2024-04-06T10:04:32.036 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=1, tag_tid=3, entry_tid=1' 2024-04-06T10:04:32.038 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror image_31 2024-04-06T10:04:32.038 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:04:32.038 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:04:32.038 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_31 2024-04-06T10:04:32.039 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror image_31 '..*' 2024-04-06T10:04:32.039 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:04:32.039 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:04:32.039 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_31 2024-04-06T10:04:32.039 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T10:04:32.039 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-image_31.status 2024-04-06T10:04:32.039 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-image_31.status 2024-04-06T10:04:32.039 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T10:04:32.040 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-image_31.status 2024-04-06T10:04:32.041 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_31 2024-04-06T10:04:32.041 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_31.status 2024-04-06T10:04:32.090 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 0 2024-04-06T10:04:32.090 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 0 2024-04-06T10:04:32.090 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T10:04:32.090 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=1, tag_tid=3, entry_tid=1], [object_number=0, tag_tid=3, entry_tid=0], [object_number=3, tag_tid=1, entry_tid=2635], [object_number=2, tag_tid=1, entry_tid=2634]]], state=connected] 2024-04-06T10:04:32.090 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=1, tag_tid=1, entry_tid=2637], [object_number=0, tag_tid=1, entry_tid=2636], [object_number=3, tag_tid=1, entry_tid=2635], [object_number=2, tag_tid=1, entry_tid=2634]]], state=connected] 2024-04-06T10:04:32.093 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_31.status 2024-04-06T10:04:32.095 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=1, tag_tid=1, entry_tid=2637' 2024-04-06T10:04:32.095 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=1, tag_tid=3, entry_tid=1' -a 'object_number=1, tag_tid=3, entry_tid=1' = 'object_number=1, tag_tid=1, entry_tid=2637' 2024-04-06T10:04:32.095 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=1, tag_tid=1, entry_tid=2637' '!=' 'object_number=1, tag_tid=1, entry_tid=2637' 2024-04-06T10:04:32.095 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T10:04:32.096 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.4 2024-04-06T10:04:32.497 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror image_31 2024-04-06T10:04:32.498 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T10:04:32.498 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:04:32.498 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_31 2024-04-06T10:04:32.498 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T10:04:32.498 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n image_31 ']' 2024-04-06T10:04:32.498 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/image_31' 2024-04-06T10:04:32.498 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_31 2024-04-06T10:04:32.498 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T10:04:32.498 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T10:04:32.498 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T10:04:32.498 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T10:04:32.498 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T10:04:32.499 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T10:04:32.499 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T10:04:32.499 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T10:04:32.499 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T10:04:32.502 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T10:04:32.502 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_31 2024-04-06T10:04:32.503 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T10:04:32.503 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T10:04:32.503 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T10:04:32.503 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T10:04:32.503 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T10:04:32.503 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T10:04:32.503 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T10:04:32.503 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T10:04:32.503 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T10:04:32.503 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T10:04:32.503 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T10:04:32.503 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:04:32.503 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T10:04:32.503 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T10:04:32.503 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T10:04:32.504 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T10:04:32.504 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T10:04:32.504 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T10:04:32.504 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T10:04:32.504 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T10:04:32.504 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T10:04:32.504 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T10:04:32.504 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:04:32.504 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T10:04:32.504 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T10:04:32.505 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T10:04:32.505 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T10:04:32.505 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T10:04:32.505 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T10:04:32.505 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T10:04:32.528 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T10:04:32.528 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T10:04:32.528 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T10:04:32.528 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_31 2024-04-06T10:04:32.644 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:Invalid command: unused arguments: ['mirror/image_31'] 2024-04-06T10:04:32.644 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:rbd mirror flush : flush rbd mirror 2024-04-06T10:04:32.644 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:admin_socket: invalid command 2024-04-06T10:04:32.650 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T10:04:32.651 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/image_31 2024-04-06T10:04:32.651 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:1 2024-04-06T10:04:32.651 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T10:04:32.651 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T10:04:32.651 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-06T10:04:32.651 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:1 2024-04-06T10:04:32.651 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T10:04:32.652 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T10:04:32.652 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T10:04:32.652 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=1 2024-04-06T10:04:32.652 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-06T10:04:32.652 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T10:04:32.652 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:04:32.652 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=1 2024-04-06T10:04:32.652 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=1 2024-04-06T10:04:32.653 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-06T10:04:32.653 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:1 2024-04-06T10:04:32.653 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T10:04:32.653 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T10:04:32.653 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-06T10:04:32.653 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:1 2024-04-06T10:04:32.654 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T10:04:32.654 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T10:04:32.654 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:04:32.654 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=1 2024-04-06T10:04:32.654 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-06T10:04:32.654 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T10:04:32.655 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T10:04:32.655 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=1 2024-04-06T10:04:32.655 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=1 2024-04-06T10:04:32.655 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-06T10:04:32.674 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T10:04:32.674 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T10:04:32.675 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T10:04:32.675 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/image_31 2024-04-06T10:04:32.829 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T10:04:32.830 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror image_31 2024-04-06T10:04:32.830 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:04:32.830 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:04:32.830 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_31 2024-04-06T10:04:32.830 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror image_31 '' 2024-04-06T10:04:32.831 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:04:32.831 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:04:32.831 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_31 2024-04-06T10:04:32.831 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T10:04:32.831 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-image_31.status 2024-04-06T10:04:32.832 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-image_31.status 2024-04-06T10:04:32.832 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T10:04:32.833 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-image_31.status 2024-04-06T10:04:32.834 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_31 2024-04-06T10:04:32.834 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_31.status 2024-04-06T10:04:32.879 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 0 2024-04-06T10:04:32.879 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 0 2024-04-06T10:04:32.880 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T10:04:32.880 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=1, tag_tid=3, entry_tid=1], [object_number=0, tag_tid=3, entry_tid=0], [object_number=3, tag_tid=1, entry_tid=2635], [object_number=2, tag_tid=1, entry_tid=2634]]], state=connected] 2024-04-06T10:04:32.880 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=1, tag_tid=3, entry_tid=1], [object_number=0, tag_tid=3, entry_tid=0], [object_number=3, tag_tid=1, entry_tid=2635], [object_number=2, tag_tid=1, entry_tid=2634]]], state=connected] 2024-04-06T10:04:32.883 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_31.status 2024-04-06T10:04:32.884 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=1, tag_tid=3, entry_tid=1' 2024-04-06T10:04:32.886 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror image_31 2024-04-06T10:04:32.886 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:04:32.886 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:04:32.886 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_31 2024-04-06T10:04:32.887 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror image_31 '..*' 2024-04-06T10:04:32.887 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:04:32.887 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:04:32.887 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_31 2024-04-06T10:04:32.887 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T10:04:32.887 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-image_31.status 2024-04-06T10:04:32.887 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-image_31.status 2024-04-06T10:04:32.887 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T10:04:32.888 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-image_31.status 2024-04-06T10:04:32.888 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_31 2024-04-06T10:04:32.888 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_31.status 2024-04-06T10:04:32.932 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 0 2024-04-06T10:04:32.932 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 0 2024-04-06T10:04:32.932 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T10:04:32.932 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=1, tag_tid=3, entry_tid=1], [object_number=0, tag_tid=3, entry_tid=0], [object_number=3, tag_tid=1, entry_tid=2635], [object_number=2, tag_tid=1, entry_tid=2634]]], state=connected] 2024-04-06T10:04:32.932 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=1, tag_tid=3, entry_tid=1], [object_number=0, tag_tid=3, entry_tid=0], [object_number=3, tag_tid=1, entry_tid=2635], [object_number=2, tag_tid=1, entry_tid=2634]]], state=connected] 2024-04-06T10:04:32.935 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_31.status 2024-04-06T10:04:32.938 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=1, tag_tid=3, entry_tid=1' 2024-04-06T10:04:32.938 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=1, tag_tid=3, entry_tid=1' -a 'object_number=1, tag_tid=3, entry_tid=1' = 'object_number=1, tag_tid=3, entry_tid=1' 2024-04-06T10:04:32.938 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T10:04:32.938 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_snap_present cluster1 mirror image_31 snap 2024-04-06T10:04:32.939 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T10:04:32.939 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:04:32.939 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_31 2024-04-06T10:04:32.939 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local snap_name=snap 2024-04-06T10:04:32.939 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local s 2024-04-06T10:04:32.939 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-04-06T10:04:32.939 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 1 2024-04-06T10:04:33.938 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 info mirror/image_31@snap 2024-04-06T10:04:34.005 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:rbd image 'image_31': 2024-04-06T10:04:34.005 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: size 128 MiB in 32 objects 2024-04-06T10:04:34.005 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: order 22 (4 MiB objects) 2024-04-06T10:04:34.005 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: snapshot_count: 1 2024-04-06T10:04:34.005 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: id: 10ace90d059a 2024-04-06T10:04:34.005 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: block_name_prefix: rbd_data.10ace90d059a 2024-04-06T10:04:34.005 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: format: 2 2024-04-06T10:04:34.005 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: features: layering, exclusive-lock, journaling 2024-04-06T10:04:34.005 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: op_features: 2024-04-06T10:04:34.005 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: flags: 2024-04-06T10:04:34.005 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: create_timestamp: Sat Apr 6 09:55:27 2024 2024-04-06T10:04:34.005 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: access_timestamp: Sat Apr 6 09:55:27 2024 2024-04-06T10:04:34.005 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: modify_timestamp: Sat Apr 6 09:56:28 2024 2024-04-06T10:04:34.005 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: protected: False 2024-04-06T10:04:34.006 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: journal: 10ace90d059a 2024-04-06T10:04:34.006 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: mirroring state: enabled 2024-04-06T10:04:34.006 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: mirroring mode: journal 2024-04-06T10:04:34.006 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: mirroring global id: 9b51b22a-d0e7-4ef4-9e0c-6f0008710f5c 2024-04-06T10:04:34.006 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: mirroring primary: false 2024-04-06T10:04:34.011 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T10:04:34.011 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ compare_image_snaps mirror image_31 snap 2024-04-06T10:04:34.012 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:04:34.012 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_31 2024-04-06T10:04:34.012 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local snap_name=snap 2024-04-06T10:04:34.012 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local ret=0 2024-04-06T10:04:34.012 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local rmt_export=/tmp/tmp.BhW6xKmROU/cluster2-mirror-image_31.export 2024-04-06T10:04:34.012 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loc_export=/tmp/tmp.BhW6xKmROU/cluster1-mirror-image_31.export 2024-04-06T10:04:34.012 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rm -f /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_31.export /tmp/tmp.BhW6xKmROU/cluster1-mirror-image_31.export 2024-04-06T10:04:34.012 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster2 -p mirror export image_31@snap /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_31.export 2024-04-06T10:04:34.160 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-04-06T10:04:34.169 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 -p mirror export image_31@snap /tmp/tmp.BhW6xKmROU/cluster1-mirror-image_31.export 2024-04-06T10:04:34.430 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-04-06T10:04:34.441 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmp /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_31.export /tmp/tmp.BhW6xKmROU/cluster1-mirror-image_31.export 2024-04-06T10:04:34.523 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rm -f /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_31.export /tmp/tmp.BhW6xKmROU/cluster1-mirror-image_31.export 2024-04-06T10:04:34.563 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T10:04:34.564 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-06T10:04:34.564 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ image=image_32 2024-04-06T10:04:34.564 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ create_snap cluster2 mirror image_32 snap 2024-04-06T10:04:34.564 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-04-06T10:04:34.564 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:04:34.564 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_32 2024-04-06T10:04:34.564 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local snap_name=snap 2024-04-06T10:04:34.564 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster2 -p mirror snap create image_32@snap --debug-rbd=20 --debug-journaler=20 2024-04-06T10:04:34.925 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_image_replay_started cluster1 mirror image_32 2024-04-06T10:04:34.925 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T10:04:34.926 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:04:34.926 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_32 2024-04-06T10:04:34.926 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_image_replay_state cluster1 mirror image_32 started 2024-04-06T10:04:34.926 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T10:04:34.926 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:04:34.926 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_32 2024-04-06T10:04:34.926 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state=started 2024-04-06T10:04:34.926 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local s 2024-04-06T10:04:34.926 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T10:04:34.926 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 1 2024-04-06T10:04:35.927 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test_image_replay_state cluster1 mirror image_32 started 2024-04-06T10:04:35.927 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T10:04:35.927 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:04:35.927 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_32 2024-04-06T10:04:35.927 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local test_state=started 2024-04-06T10:04:35.927 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local status_result 2024-04-06T10:04:35.927 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local current_state=stopped 2024-04-06T10:04:35.928 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_32 2024-04-06T10:04:35.929 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -i state 2024-04-06T10:04:35.929 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_instance=cluster1 2024-04-06T10:04:35.929 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ shift 2024-04-06T10:04:35.929 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T10:04:35.929 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance=cluster1 2024-04-06T10:04:35.929 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local loop_instance 2024-04-06T10:04:35.929 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T10:04:35.929 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sleep 0 2024-04-06T10:04:35.930 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-04-06T10:04:35.930 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ seq 0 3 2024-04-06T10:04:35.933 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T10:04:35.933 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_32 2024-04-06T10:04:35.933 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1:0 2024-04-06T10:04:35.933 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ shift 2024-04-06T10:04:35.933 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T10:04:35.934 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-04-06T10:04:35.934 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T10:04:35.934 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=cluster 2024-04-06T10:04:35.934 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T10:04:35.934 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:04:35.934 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T10:04:35.934 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T10:04:35.934 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval cluster=cluster1 2024-04-06T10:04:35.934 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ cluster=cluster1 2024-04-06T10:04:35.934 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T10:04:35.934 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T10:04:35.935 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-04-06T10:04:35.936 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local local_cluster=cluster1:0 2024-04-06T10:04:35.936 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local cluster=cluster1 2024-04-06T10:04:35.936 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local instance 2024-04-06T10:04:35.937 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T10:04:35.937 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local val=cluster1:0 2024-04-06T10:04:35.937 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local cluster_var_name=local_cluster 2024-04-06T10:04:35.937 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local instance_var_name=instance 2024-04-06T10:04:35.937 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ cluster=cluster1 2024-04-06T10:04:35.937 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T10:04:35.937 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ '[' 0 = cluster1:0 ']' 2024-04-06T10:04:35.937 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ eval local_cluster=cluster1 2024-04-06T10:04:35.937 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++++ local_cluster=cluster1 2024-04-06T10:04:35.937 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ eval instance=0 2024-04-06T10:04:35.937 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++++ instance=0 2024-04-06T10:04:35.937 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T10:04:35.955 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T10:04:35.956 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T10:04:35.956 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T10:04:35.956 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_32 2024-04-06T10:04:36.060 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:Invalid command: unused arguments: ['mirror/image_32'] 2024-04-06T10:04:36.060 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:rbd mirror status : get status for rbd mirror 2024-04-06T10:04:36.060 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:admin_socket: invalid command 2024-04-06T10:04:36.064 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T10:04:36.065 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_32 2024-04-06T10:04:36.065 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1:1 2024-04-06T10:04:36.065 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ shift 2024-04-06T10:04:36.065 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T10:04:36.065 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-04-06T10:04:36.065 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:1 2024-04-06T10:04:36.065 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=cluster 2024-04-06T10:04:36.065 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T10:04:36.065 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:04:36.065 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=1 2024-04-06T10:04:36.065 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-06T10:04:36.065 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval cluster=cluster1 2024-04-06T10:04:36.065 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ cluster=cluster1 2024-04-06T10:04:36.065 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=1 2024-04-06T10:04:36.065 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=1 2024-04-06T10:04:36.066 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-04-06T10:04:36.066 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local local_cluster=cluster1:1 2024-04-06T10:04:36.066 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local cluster=cluster1 2024-04-06T10:04:36.066 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local instance 2024-04-06T10:04:36.066 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-06T10:04:36.066 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local val=cluster1:1 2024-04-06T10:04:36.066 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local cluster_var_name=local_cluster 2024-04-06T10:04:36.066 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local instance_var_name=instance 2024-04-06T10:04:36.066 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ cluster=cluster1 2024-04-06T10:04:36.066 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=1 2024-04-06T10:04:36.066 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ '[' 1 = cluster1:1 ']' 2024-04-06T10:04:36.066 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ eval local_cluster=cluster1 2024-04-06T10:04:36.066 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++++ local_cluster=cluster1 2024-04-06T10:04:36.066 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ eval instance=1 2024-04-06T10:04:36.067 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++++ instance=1 2024-04-06T10:04:36.067 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-06T10:04:36.087 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T10:04:36.088 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T10:04:36.088 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T10:04:36.088 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_32 2024-04-06T10:04:36.239 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ return 0 2024-04-06T10:04:36.240 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ status_result=' "state": "Replaying"' 2024-04-06T10:04:36.240 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ echo ' "state": "Replaying"' 2024-04-06T10:04:36.241 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep -i Replaying 2024-04-06T10:04:36.243 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: "state": "Replaying" 2024-04-06T10:04:36.244 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ current_state=started 2024-04-06T10:04:36.245 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test started = started 2024-04-06T10:04:36.245 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T10:04:36.245 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_32 2024-04-06T10:04:36.245 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local local_cluster=cluster1 2024-04-06T10:04:36.245 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-04-06T10:04:36.245 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:04:36.245 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_32 2024-04-06T10:04:36.245 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' journal = journal ']' 2024-04-06T10:04:36.245 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_journal_replay_complete cluster1 cluster2 mirror image_32 2024-04-06T10:04:36.245 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local local_cluster=cluster1 2024-04-06T10:04:36.245 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-04-06T10:04:36.245 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:04:36.245 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_32 2024-04-06T10:04:36.245 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local s master_pos mirror_pos last_mirror_pos 2024-04-06T10:04:36.245 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local master_tag master_entry mirror_tag mirror_entry 2024-04-06T10:04:36.246 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T10:04:36.246 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T10:04:36.246 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T10:04:36.447 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror image_32 2024-04-06T10:04:36.447 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T10:04:36.447 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:04:36.447 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_32 2024-04-06T10:04:36.447 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T10:04:36.447 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n image_32 ']' 2024-04-06T10:04:36.447 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/image_32' 2024-04-06T10:04:36.447 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_32 2024-04-06T10:04:36.447 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T10:04:36.447 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T10:04:36.448 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T10:04:36.448 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T10:04:36.448 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T10:04:36.448 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T10:04:36.448 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T10:04:36.448 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T10:04:36.449 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T10:04:36.451 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T10:04:36.452 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_32 2024-04-06T10:04:36.452 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T10:04:36.452 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T10:04:36.452 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T10:04:36.452 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T10:04:36.452 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T10:04:36.452 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T10:04:36.452 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T10:04:36.452 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T10:04:36.452 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T10:04:36.453 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T10:04:36.453 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T10:04:36.453 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:04:36.453 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T10:04:36.453 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T10:04:36.453 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T10:04:36.453 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T10:04:36.453 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T10:04:36.453 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T10:04:36.453 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T10:04:36.453 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T10:04:36.454 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T10:04:36.454 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T10:04:36.454 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:04:36.454 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T10:04:36.454 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T10:04:36.454 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T10:04:36.454 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T10:04:36.454 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T10:04:36.454 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T10:04:36.454 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T10:04:36.478 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T10:04:36.478 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T10:04:36.478 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T10:04:36.479 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_32 2024-04-06T10:04:36.588 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:Invalid command: unused arguments: ['mirror/image_32'] 2024-04-06T10:04:36.588 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:rbd mirror flush : flush rbd mirror 2024-04-06T10:04:36.588 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:admin_socket: invalid command 2024-04-06T10:04:36.591 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T10:04:36.591 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/image_32 2024-04-06T10:04:36.591 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:1 2024-04-06T10:04:36.591 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T10:04:36.591 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T10:04:36.591 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-06T10:04:36.591 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:1 2024-04-06T10:04:36.592 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T10:04:36.592 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T10:04:36.592 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T10:04:36.592 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=1 2024-04-06T10:04:36.592 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-06T10:04:36.592 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T10:04:36.592 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:04:36.592 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=1 2024-04-06T10:04:36.592 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=1 2024-04-06T10:04:36.594 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-06T10:04:36.594 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:1 2024-04-06T10:04:36.594 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T10:04:36.594 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T10:04:36.594 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-06T10:04:36.594 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:1 2024-04-06T10:04:36.594 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T10:04:36.594 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T10:04:36.594 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:04:36.594 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=1 2024-04-06T10:04:36.594 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-06T10:04:36.595 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T10:04:36.595 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T10:04:36.595 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=1 2024-04-06T10:04:36.595 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=1 2024-04-06T10:04:36.595 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-06T10:04:36.617 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T10:04:36.617 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T10:04:36.617 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T10:04:36.617 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/image_32 2024-04-06T10:04:36.754 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T10:04:36.755 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror image_32 2024-04-06T10:04:36.755 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:04:36.755 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:04:36.755 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_32 2024-04-06T10:04:36.755 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror image_32 '' 2024-04-06T10:04:36.755 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:04:36.755 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:04:36.755 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_32 2024-04-06T10:04:36.755 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T10:04:36.756 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-image_32.status 2024-04-06T10:04:36.756 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-image_32.status 2024-04-06T10:04:36.756 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T10:04:36.758 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-image_32.status 2024-04-06T10:04:36.758 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_32 2024-04-06T10:04:36.759 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_32.status 2024-04-06T10:04:36.809 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 1 2024-04-06T10:04:36.809 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 1 2024-04-06T10:04:36.809 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T10:04:36.809 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=5, tag_tid=3, entry_tid=1], [object_number=4, tag_tid=3, entry_tid=0], [object_number=7, tag_tid=1, entry_tid=6511], [object_number=6, tag_tid=1, entry_tid=6510]]], state=connected] 2024-04-06T10:04:36.809 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=5, tag_tid=2, entry_tid=1], [object_number=4, tag_tid=2, entry_tid=0], [object_number=7, tag_tid=1, entry_tid=6511], [object_number=6, tag_tid=1, entry_tid=6510]]], state=connected] 2024-04-06T10:04:36.813 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_32.status 2024-04-06T10:04:36.814 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=5, tag_tid=3, entry_tid=1' 2024-04-06T10:04:36.815 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror image_32 2024-04-06T10:04:36.815 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:04:36.815 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:04:36.815 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_32 2024-04-06T10:04:36.816 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror image_32 '..*' 2024-04-06T10:04:36.816 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:04:36.816 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:04:36.816 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_32 2024-04-06T10:04:36.816 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T10:04:36.816 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-image_32.status 2024-04-06T10:04:36.816 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-image_32.status 2024-04-06T10:04:36.816 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T10:04:36.817 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-image_32.status 2024-04-06T10:04:36.817 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_32 2024-04-06T10:04:36.817 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_32.status 2024-04-06T10:04:36.861 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 1 2024-04-06T10:04:36.862 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 1 2024-04-06T10:04:36.862 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T10:04:36.862 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=5, tag_tid=3, entry_tid=1], [object_number=4, tag_tid=3, entry_tid=0], [object_number=7, tag_tid=1, entry_tid=6511], [object_number=6, tag_tid=1, entry_tid=6510]]], state=connected] 2024-04-06T10:04:36.862 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=5, tag_tid=2, entry_tid=1], [object_number=4, tag_tid=2, entry_tid=0], [object_number=7, tag_tid=1, entry_tid=6511], [object_number=6, tag_tid=1, entry_tid=6510]]], state=connected] 2024-04-06T10:04:36.865 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_32.status 2024-04-06T10:04:36.867 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=5, tag_tid=2, entry_tid=1' 2024-04-06T10:04:36.867 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=5, tag_tid=3, entry_tid=1' -a 'object_number=5, tag_tid=3, entry_tid=1' = 'object_number=5, tag_tid=2, entry_tid=1' 2024-04-06T10:04:36.867 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=5, tag_tid=2, entry_tid=1' '!=' '' 2024-04-06T10:04:36.867 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T10:04:36.867 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=5, tag_tid=2, entry_tid=1' = '' 2024-04-06T10:04:36.867 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=5, tag_tid=2, entry_tid=1' 2024-04-06T10:04:36.868 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=5, tag_tid=3, entry_tid=1' 2024-04-06T10:04:36.868 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T10:04:36.868 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T10:04:36.868 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=3 2024-04-06T10:04:36.869 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=5, tag_tid=2, entry_tid=1' 2024-04-06T10:04:36.869 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T10:04:36.869 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T10:04:36.870 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=2 2024-04-06T10:04:36.871 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=5, tag_tid=3, entry_tid=1' 2024-04-06T10:04:36.871 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T10:04:36.871 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T10:04:36.872 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T10:04:36.872 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=5, tag_tid=2, entry_tid=1' 2024-04-06T10:04:36.872 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T10:04:36.873 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T10:04:36.874 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=1 2024-04-06T10:04:36.874 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 3 = 2 -a 1 -le 1 2024-04-06T10:04:36.874 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T10:04:36.874 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T10:04:36.874 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T10:04:37.076 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror image_32 2024-04-06T10:04:37.076 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T10:04:37.076 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:04:37.076 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_32 2024-04-06T10:04:37.077 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T10:04:37.077 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n image_32 ']' 2024-04-06T10:04:37.077 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/image_32' 2024-04-06T10:04:37.077 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_32 2024-04-06T10:04:37.077 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T10:04:37.077 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T10:04:37.077 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T10:04:37.077 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T10:04:37.077 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T10:04:37.077 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T10:04:37.077 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T10:04:37.078 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T10:04:37.078 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T10:04:37.081 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T10:04:37.081 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_32 2024-04-06T10:04:37.081 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T10:04:37.082 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T10:04:37.082 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T10:04:37.082 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T10:04:37.082 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T10:04:37.082 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T10:04:37.082 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T10:04:37.082 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T10:04:37.082 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T10:04:37.082 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T10:04:37.082 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T10:04:37.082 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:04:37.082 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T10:04:37.082 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T10:04:37.082 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T10:04:37.083 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T10:04:37.083 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T10:04:37.083 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T10:04:37.083 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T10:04:37.083 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T10:04:37.083 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T10:04:37.083 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T10:04:37.083 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:04:37.083 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T10:04:37.083 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T10:04:37.083 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T10:04:37.083 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T10:04:37.083 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T10:04:37.083 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T10:04:37.083 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T10:04:37.106 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T10:04:37.106 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T10:04:37.107 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T10:04:37.107 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_32 2024-04-06T10:04:37.213 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:Invalid command: unused arguments: ['mirror/image_32'] 2024-04-06T10:04:37.213 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:rbd mirror flush : flush rbd mirror 2024-04-06T10:04:37.213 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:admin_socket: invalid command 2024-04-06T10:04:37.216 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T10:04:37.217 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/image_32 2024-04-06T10:04:37.217 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:1 2024-04-06T10:04:37.217 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T10:04:37.217 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T10:04:37.217 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-06T10:04:37.217 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:1 2024-04-06T10:04:37.217 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T10:04:37.217 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T10:04:37.217 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T10:04:37.217 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=1 2024-04-06T10:04:37.217 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-06T10:04:37.217 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T10:04:37.218 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:04:37.218 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=1 2024-04-06T10:04:37.218 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=1 2024-04-06T10:04:37.218 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-06T10:04:37.218 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:1 2024-04-06T10:04:37.218 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T10:04:37.218 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T10:04:37.218 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-06T10:04:37.218 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:1 2024-04-06T10:04:37.218 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T10:04:37.218 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T10:04:37.218 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:04:37.218 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=1 2024-04-06T10:04:37.219 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-06T10:04:37.219 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T10:04:37.219 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T10:04:37.219 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=1 2024-04-06T10:04:37.219 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=1 2024-04-06T10:04:37.219 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-06T10:04:37.241 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T10:04:37.241 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T10:04:37.241 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T10:04:37.241 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/image_32 2024-04-06T10:04:37.391 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T10:04:37.392 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror image_32 2024-04-06T10:04:37.392 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:04:37.392 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:04:37.393 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_32 2024-04-06T10:04:37.393 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror image_32 '' 2024-04-06T10:04:37.393 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:04:37.393 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:04:37.393 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_32 2024-04-06T10:04:37.393 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T10:04:37.393 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-image_32.status 2024-04-06T10:04:37.393 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T10:04:37.393 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-image_32.status 2024-04-06T10:04:37.394 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-image_32.status 2024-04-06T10:04:37.395 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_32 2024-04-06T10:04:37.395 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_32.status 2024-04-06T10:04:37.440 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 1 2024-04-06T10:04:37.440 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 1 2024-04-06T10:04:37.441 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T10:04:37.441 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=5, tag_tid=3, entry_tid=1], [object_number=4, tag_tid=3, entry_tid=0], [object_number=7, tag_tid=1, entry_tid=6511], [object_number=6, tag_tid=1, entry_tid=6510]]], state=connected] 2024-04-06T10:04:37.441 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=5, tag_tid=2, entry_tid=1], [object_number=4, tag_tid=2, entry_tid=0], [object_number=7, tag_tid=1, entry_tid=6511], [object_number=6, tag_tid=1, entry_tid=6510]]], state=connected] 2024-04-06T10:04:37.444 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_32.status 2024-04-06T10:04:37.445 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=5, tag_tid=3, entry_tid=1' 2024-04-06T10:04:37.446 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror image_32 2024-04-06T10:04:37.446 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:04:37.447 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:04:37.447 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_32 2024-04-06T10:04:37.447 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror image_32 '..*' 2024-04-06T10:04:37.447 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:04:37.447 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:04:37.447 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_32 2024-04-06T10:04:37.447 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T10:04:37.447 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-image_32.status 2024-04-06T10:04:37.447 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-image_32.status 2024-04-06T10:04:37.447 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T10:04:37.448 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-image_32.status 2024-04-06T10:04:37.448 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_32 2024-04-06T10:04:37.448 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_32.status 2024-04-06T10:04:37.492 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 1 2024-04-06T10:04:37.492 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 1 2024-04-06T10:04:37.492 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T10:04:37.492 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=5, tag_tid=3, entry_tid=1], [object_number=4, tag_tid=3, entry_tid=0], [object_number=7, tag_tid=1, entry_tid=6511], [object_number=6, tag_tid=1, entry_tid=6510]]], state=connected] 2024-04-06T10:04:37.492 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=5, tag_tid=2, entry_tid=1], [object_number=4, tag_tid=2, entry_tid=0], [object_number=7, tag_tid=1, entry_tid=6511], [object_number=6, tag_tid=1, entry_tid=6510]]], state=connected] 2024-04-06T10:04:37.495 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_32.status 2024-04-06T10:04:37.497 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=5, tag_tid=2, entry_tid=1' 2024-04-06T10:04:37.497 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=5, tag_tid=3, entry_tid=1' -a 'object_number=5, tag_tid=3, entry_tid=1' = 'object_number=5, tag_tid=2, entry_tid=1' 2024-04-06T10:04:37.497 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=5, tag_tid=2, entry_tid=1' '!=' 'object_number=5, tag_tid=2, entry_tid=1' 2024-04-06T10:04:37.497 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T10:04:37.498 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.4 2024-04-06T10:04:37.899 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror image_32 2024-04-06T10:04:37.899 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T10:04:37.899 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:04:37.899 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_32 2024-04-06T10:04:37.899 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T10:04:37.899 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n image_32 ']' 2024-04-06T10:04:37.900 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/image_32' 2024-04-06T10:04:37.900 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_32 2024-04-06T10:04:37.900 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T10:04:37.900 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T10:04:37.900 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T10:04:37.900 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T10:04:37.900 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T10:04:37.900 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T10:04:37.900 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T10:04:37.900 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T10:04:37.901 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T10:04:37.903 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T10:04:37.903 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_32 2024-04-06T10:04:37.904 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T10:04:37.904 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T10:04:37.904 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T10:04:37.904 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T10:04:37.904 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T10:04:37.904 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T10:04:37.904 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T10:04:37.904 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T10:04:37.904 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T10:04:37.904 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T10:04:37.904 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T10:04:37.904 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:04:37.904 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T10:04:37.904 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T10:04:37.905 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T10:04:37.905 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T10:04:37.905 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T10:04:37.905 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T10:04:37.905 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T10:04:37.905 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T10:04:37.905 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T10:04:37.905 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T10:04:37.905 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:04:37.905 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T10:04:37.905 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T10:04:37.905 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T10:04:37.905 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T10:04:37.905 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T10:04:37.906 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T10:04:37.906 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T10:04:37.930 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T10:04:37.930 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T10:04:37.930 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T10:04:37.931 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_32 2024-04-06T10:04:38.040 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:Invalid command: unused arguments: ['mirror/image_32'] 2024-04-06T10:04:38.040 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:rbd mirror flush : flush rbd mirror 2024-04-06T10:04:38.041 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:admin_socket: invalid command 2024-04-06T10:04:38.043 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T10:04:38.043 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/image_32 2024-04-06T10:04:38.043 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:1 2024-04-06T10:04:38.043 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T10:04:38.043 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T10:04:38.043 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-06T10:04:38.043 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:1 2024-04-06T10:04:38.043 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T10:04:38.043 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T10:04:38.043 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T10:04:38.044 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=1 2024-04-06T10:04:38.044 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-06T10:04:38.044 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T10:04:38.044 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:04:38.044 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=1 2024-04-06T10:04:38.044 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=1 2024-04-06T10:04:38.045 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-06T10:04:38.046 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:1 2024-04-06T10:04:38.046 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T10:04:38.046 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T10:04:38.046 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-06T10:04:38.046 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:1 2024-04-06T10:04:38.046 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T10:04:38.046 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T10:04:38.046 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:04:38.046 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=1 2024-04-06T10:04:38.046 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-06T10:04:38.046 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T10:04:38.046 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T10:04:38.046 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=1 2024-04-06T10:04:38.046 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=1 2024-04-06T10:04:38.047 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-06T10:04:38.069 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T10:04:38.069 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T10:04:38.069 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T10:04:38.069 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/image_32 2024-04-06T10:04:38.206 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T10:04:38.207 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror image_32 2024-04-06T10:04:38.207 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:04:38.208 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:04:38.208 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_32 2024-04-06T10:04:38.208 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror image_32 '' 2024-04-06T10:04:38.208 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:04:38.208 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:04:38.208 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_32 2024-04-06T10:04:38.209 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T10:04:38.209 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-image_32.status 2024-04-06T10:04:38.210 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-image_32.status 2024-04-06T10:04:38.210 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T10:04:38.211 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-image_32.status 2024-04-06T10:04:38.211 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_32.status 2024-04-06T10:04:38.211 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_32 2024-04-06T10:04:38.261 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 1 2024-04-06T10:04:38.261 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 1 2024-04-06T10:04:38.262 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T10:04:38.262 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=5, tag_tid=3, entry_tid=1], [object_number=4, tag_tid=3, entry_tid=0], [object_number=7, tag_tid=1, entry_tid=6511], [object_number=6, tag_tid=1, entry_tid=6510]]], state=connected] 2024-04-06T10:04:38.262 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=5, tag_tid=2, entry_tid=1], [object_number=4, tag_tid=2, entry_tid=0], [object_number=7, tag_tid=1, entry_tid=6511], [object_number=6, tag_tid=1, entry_tid=6510]]], state=connected] 2024-04-06T10:04:38.266 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_32.status 2024-04-06T10:04:38.267 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=5, tag_tid=3, entry_tid=1' 2024-04-06T10:04:38.268 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror image_32 2024-04-06T10:04:38.268 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:04:38.268 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:04:38.269 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_32 2024-04-06T10:04:38.269 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror image_32 '..*' 2024-04-06T10:04:38.269 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:04:38.269 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:04:38.269 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_32 2024-04-06T10:04:38.269 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T10:04:38.269 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-image_32.status 2024-04-06T10:04:38.269 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-image_32.status 2024-04-06T10:04:38.269 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T10:04:38.270 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-image_32.status 2024-04-06T10:04:38.270 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_32 2024-04-06T10:04:38.270 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_32.status 2024-04-06T10:04:38.316 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 1 2024-04-06T10:04:38.316 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 1 2024-04-06T10:04:38.317 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T10:04:38.317 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=5, tag_tid=3, entry_tid=1], [object_number=4, tag_tid=3, entry_tid=0], [object_number=7, tag_tid=1, entry_tid=6511], [object_number=6, tag_tid=1, entry_tid=6510]]], state=connected] 2024-04-06T10:04:38.317 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=5, tag_tid=2, entry_tid=1], [object_number=4, tag_tid=2, entry_tid=0], [object_number=7, tag_tid=1, entry_tid=6511], [object_number=6, tag_tid=1, entry_tid=6510]]], state=connected] 2024-04-06T10:04:38.320 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_32.status 2024-04-06T10:04:38.322 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=5, tag_tid=2, entry_tid=1' 2024-04-06T10:04:38.322 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=5, tag_tid=3, entry_tid=1' -a 'object_number=5, tag_tid=3, entry_tid=1' = 'object_number=5, tag_tid=2, entry_tid=1' 2024-04-06T10:04:38.322 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=5, tag_tid=2, entry_tid=1' '!=' 'object_number=5, tag_tid=2, entry_tid=1' 2024-04-06T10:04:38.322 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T10:04:38.322 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.8 2024-04-06T10:04:39.124 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror image_32 2024-04-06T10:04:39.124 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T10:04:39.124 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:04:39.124 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_32 2024-04-06T10:04:39.124 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T10:04:39.125 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n image_32 ']' 2024-04-06T10:04:39.125 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/image_32' 2024-04-06T10:04:39.125 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_32 2024-04-06T10:04:39.125 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T10:04:39.125 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T10:04:39.125 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T10:04:39.125 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T10:04:39.125 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T10:04:39.125 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T10:04:39.125 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T10:04:39.126 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T10:04:39.126 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T10:04:39.127 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T10:04:39.127 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_32 2024-04-06T10:04:39.127 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T10:04:39.127 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T10:04:39.128 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T10:04:39.128 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T10:04:39.128 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T10:04:39.128 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T10:04:39.128 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T10:04:39.128 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T10:04:39.128 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T10:04:39.128 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T10:04:39.128 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T10:04:39.130 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:04:39.130 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T10:04:39.130 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T10:04:39.130 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T10:04:39.130 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T10:04:39.130 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T10:04:39.130 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T10:04:39.130 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T10:04:39.130 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T10:04:39.131 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T10:04:39.131 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T10:04:39.131 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:04:39.131 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T10:04:39.131 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T10:04:39.131 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T10:04:39.131 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T10:04:39.131 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T10:04:39.131 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T10:04:39.131 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T10:04:39.156 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T10:04:39.156 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T10:04:39.157 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T10:04:39.157 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_32 2024-04-06T10:04:39.262 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:Invalid command: unused arguments: ['mirror/image_32'] 2024-04-06T10:04:39.262 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:rbd mirror flush : flush rbd mirror 2024-04-06T10:04:39.262 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:admin_socket: invalid command 2024-04-06T10:04:39.264 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T10:04:39.264 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/image_32 2024-04-06T10:04:39.265 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:1 2024-04-06T10:04:39.265 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T10:04:39.265 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T10:04:39.265 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-06T10:04:39.265 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:1 2024-04-06T10:04:39.265 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T10:04:39.265 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T10:04:39.265 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T10:04:39.265 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=1 2024-04-06T10:04:39.265 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-06T10:04:39.265 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T10:04:39.265 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:04:39.265 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=1 2024-04-06T10:04:39.265 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=1 2024-04-06T10:04:39.267 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-06T10:04:39.267 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:1 2024-04-06T10:04:39.267 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T10:04:39.267 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T10:04:39.267 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-06T10:04:39.267 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:1 2024-04-06T10:04:39.267 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T10:04:39.267 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T10:04:39.267 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:04:39.267 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=1 2024-04-06T10:04:39.267 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-06T10:04:39.267 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T10:04:39.267 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T10:04:39.268 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=1 2024-04-06T10:04:39.268 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=1 2024-04-06T10:04:39.268 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-06T10:04:39.285 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T10:04:39.286 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T10:04:39.286 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T10:04:39.286 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/image_32 2024-04-06T10:04:39.429 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T10:04:39.430 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror image_32 2024-04-06T10:04:39.430 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:04:39.430 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:04:39.430 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_32 2024-04-06T10:04:39.430 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror image_32 '' 2024-04-06T10:04:39.430 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:04:39.431 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:04:39.431 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_32 2024-04-06T10:04:39.431 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T10:04:39.431 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-image_32.status 2024-04-06T10:04:39.431 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-image_32.status 2024-04-06T10:04:39.431 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T10:04:39.432 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-image_32.status 2024-04-06T10:04:39.432 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_32.status 2024-04-06T10:04:39.432 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_32 2024-04-06T10:04:39.492 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 1 2024-04-06T10:04:39.492 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 1 2024-04-06T10:04:39.492 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T10:04:39.492 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=5, tag_tid=3, entry_tid=1], [object_number=4, tag_tid=3, entry_tid=0], [object_number=7, tag_tid=1, entry_tid=6511], [object_number=6, tag_tid=1, entry_tid=6510]]], state=connected] 2024-04-06T10:04:39.492 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=5, tag_tid=2, entry_tid=1], [object_number=4, tag_tid=2, entry_tid=0], [object_number=7, tag_tid=1, entry_tid=6511], [object_number=6, tag_tid=1, entry_tid=6510]]], state=connected] 2024-04-06T10:04:39.496 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_32.status 2024-04-06T10:04:39.497 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=5, tag_tid=3, entry_tid=1' 2024-04-06T10:04:39.498 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror image_32 2024-04-06T10:04:39.498 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:04:39.498 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:04:39.498 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_32 2024-04-06T10:04:39.499 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror image_32 '..*' 2024-04-06T10:04:39.499 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:04:39.499 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:04:39.499 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_32 2024-04-06T10:04:39.499 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T10:04:39.499 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-image_32.status 2024-04-06T10:04:39.499 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-image_32.status 2024-04-06T10:04:39.499 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T10:04:39.500 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-image_32.status 2024-04-06T10:04:39.500 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_32 2024-04-06T10:04:39.500 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_32.status 2024-04-06T10:04:39.546 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 1 2024-04-06T10:04:39.546 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 1 2024-04-06T10:04:39.547 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T10:04:39.547 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=5, tag_tid=3, entry_tid=1], [object_number=4, tag_tid=3, entry_tid=0], [object_number=7, tag_tid=1, entry_tid=6511], [object_number=6, tag_tid=1, entry_tid=6510]]], state=connected] 2024-04-06T10:04:39.547 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=5, tag_tid=2, entry_tid=1], [object_number=4, tag_tid=2, entry_tid=0], [object_number=7, tag_tid=1, entry_tid=6511], [object_number=6, tag_tid=1, entry_tid=6510]]], state=connected] 2024-04-06T10:04:39.550 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_32.status 2024-04-06T10:04:39.552 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=5, tag_tid=2, entry_tid=1' 2024-04-06T10:04:39.552 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=5, tag_tid=3, entry_tid=1' -a 'object_number=5, tag_tid=3, entry_tid=1' = 'object_number=5, tag_tid=2, entry_tid=1' 2024-04-06T10:04:39.552 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=5, tag_tid=2, entry_tid=1' '!=' 'object_number=5, tag_tid=2, entry_tid=1' 2024-04-06T10:04:39.552 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T10:04:39.552 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 1.6 2024-04-06T10:04:41.153 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror image_32 2024-04-06T10:04:41.154 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T10:04:41.154 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:04:41.154 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_32 2024-04-06T10:04:41.154 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T10:04:41.154 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n image_32 ']' 2024-04-06T10:04:41.154 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/image_32' 2024-04-06T10:04:41.154 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_32 2024-04-06T10:04:41.154 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T10:04:41.154 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T10:04:41.154 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T10:04:41.154 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T10:04:41.155 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T10:04:41.155 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T10:04:41.155 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T10:04:41.155 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T10:04:41.155 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T10:04:41.157 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T10:04:41.157 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_32 2024-04-06T10:04:41.157 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T10:04:41.157 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T10:04:41.158 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T10:04:41.158 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T10:04:41.158 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T10:04:41.158 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T10:04:41.158 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T10:04:41.158 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T10:04:41.158 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T10:04:41.158 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T10:04:41.158 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T10:04:41.158 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:04:41.158 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T10:04:41.158 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T10:04:41.159 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T10:04:41.160 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T10:04:41.160 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T10:04:41.160 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T10:04:41.160 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T10:04:41.160 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T10:04:41.160 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T10:04:41.160 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T10:04:41.160 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:04:41.160 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T10:04:41.160 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T10:04:41.160 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T10:04:41.160 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T10:04:41.160 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T10:04:41.160 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T10:04:41.161 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T10:04:41.184 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T10:04:41.185 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T10:04:41.185 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T10:04:41.185 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_32 2024-04-06T10:04:41.290 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:Invalid command: unused arguments: ['mirror/image_32'] 2024-04-06T10:04:41.290 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:rbd mirror flush : flush rbd mirror 2024-04-06T10:04:41.290 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:admin_socket: invalid command 2024-04-06T10:04:41.292 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T10:04:41.293 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/image_32 2024-04-06T10:04:41.293 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:1 2024-04-06T10:04:41.293 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T10:04:41.293 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T10:04:41.293 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-06T10:04:41.293 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:1 2024-04-06T10:04:41.293 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T10:04:41.293 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T10:04:41.293 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T10:04:41.293 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=1 2024-04-06T10:04:41.293 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-06T10:04:41.293 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T10:04:41.293 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:04:41.294 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=1 2024-04-06T10:04:41.294 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=1 2024-04-06T10:04:41.295 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-06T10:04:41.295 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:1 2024-04-06T10:04:41.295 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T10:04:41.295 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T10:04:41.295 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-06T10:04:41.295 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:1 2024-04-06T10:04:41.295 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T10:04:41.295 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T10:04:41.295 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:04:41.295 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=1 2024-04-06T10:04:41.295 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-06T10:04:41.295 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T10:04:41.296 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T10:04:41.296 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=1 2024-04-06T10:04:41.296 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=1 2024-04-06T10:04:41.296 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-06T10:04:41.314 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T10:04:41.314 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T10:04:41.314 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T10:04:41.314 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/image_32 2024-04-06T10:04:41.457 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T10:04:41.459 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror image_32 2024-04-06T10:04:41.459 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:04:41.459 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:04:41.459 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_32 2024-04-06T10:04:41.459 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror image_32 '' 2024-04-06T10:04:41.459 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:04:41.459 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:04:41.459 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_32 2024-04-06T10:04:41.459 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T10:04:41.459 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-image_32.status 2024-04-06T10:04:41.460 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-image_32.status 2024-04-06T10:04:41.460 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T10:04:41.462 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-image_32.status 2024-04-06T10:04:41.462 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_32 2024-04-06T10:04:41.462 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_32.status 2024-04-06T10:04:41.509 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 1 2024-04-06T10:04:41.509 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 1 2024-04-06T10:04:41.509 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T10:04:41.510 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=5, tag_tid=3, entry_tid=1], [object_number=4, tag_tid=3, entry_tid=0], [object_number=7, tag_tid=1, entry_tid=6511], [object_number=6, tag_tid=1, entry_tid=6510]]], state=connected] 2024-04-06T10:04:41.510 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=5, tag_tid=2, entry_tid=1], [object_number=4, tag_tid=2, entry_tid=0], [object_number=7, tag_tid=1, entry_tid=6511], [object_number=6, tag_tid=1, entry_tid=6510]]], state=connected] 2024-04-06T10:04:41.513 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_32.status 2024-04-06T10:04:41.514 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=5, tag_tid=3, entry_tid=1' 2024-04-06T10:04:41.515 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror image_32 2024-04-06T10:04:41.515 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:04:41.516 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:04:41.516 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_32 2024-04-06T10:04:41.516 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror image_32 '..*' 2024-04-06T10:04:41.516 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:04:41.516 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:04:41.516 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_32 2024-04-06T10:04:41.516 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T10:04:41.516 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-image_32.status 2024-04-06T10:04:41.516 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-image_32.status 2024-04-06T10:04:41.516 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T10:04:41.517 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-image_32.status 2024-04-06T10:04:41.517 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_32 2024-04-06T10:04:41.517 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_32.status 2024-04-06T10:04:41.563 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 1 2024-04-06T10:04:41.563 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 1 2024-04-06T10:04:41.564 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T10:04:41.564 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=5, tag_tid=3, entry_tid=1], [object_number=4, tag_tid=3, entry_tid=0], [object_number=7, tag_tid=1, entry_tid=6511], [object_number=6, tag_tid=1, entry_tid=6510]]], state=connected] 2024-04-06T10:04:41.564 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=5, tag_tid=2, entry_tid=1], [object_number=4, tag_tid=2, entry_tid=0], [object_number=7, tag_tid=1, entry_tid=6511], [object_number=6, tag_tid=1, entry_tid=6510]]], state=connected] 2024-04-06T10:04:41.567 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_32.status 2024-04-06T10:04:41.569 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=5, tag_tid=2, entry_tid=1' 2024-04-06T10:04:41.569 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=5, tag_tid=3, entry_tid=1' -a 'object_number=5, tag_tid=3, entry_tid=1' = 'object_number=5, tag_tid=2, entry_tid=1' 2024-04-06T10:04:41.569 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=5, tag_tid=2, entry_tid=1' '!=' 'object_number=5, tag_tid=2, entry_tid=1' 2024-04-06T10:04:41.569 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T10:04:41.569 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 2 2024-04-06T10:04:43.571 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror image_32 2024-04-06T10:04:43.571 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T10:04:43.571 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:04:43.571 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_32 2024-04-06T10:04:43.571 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T10:04:43.571 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n image_32 ']' 2024-04-06T10:04:43.571 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/image_32' 2024-04-06T10:04:43.571 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_32 2024-04-06T10:04:43.571 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T10:04:43.571 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T10:04:43.571 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T10:04:43.572 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T10:04:43.572 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T10:04:43.572 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T10:04:43.572 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T10:04:43.572 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T10:04:43.572 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T10:04:43.575 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T10:04:43.575 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_32 2024-04-06T10:04:43.575 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T10:04:43.575 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T10:04:43.575 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T10:04:43.575 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T10:04:43.576 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T10:04:43.576 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T10:04:43.576 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T10:04:43.576 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T10:04:43.576 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T10:04:43.576 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T10:04:43.576 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T10:04:43.576 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:04:43.576 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T10:04:43.576 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T10:04:43.576 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T10:04:43.576 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T10:04:43.576 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T10:04:43.576 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T10:04:43.577 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T10:04:43.577 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T10:04:43.577 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T10:04:43.577 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T10:04:43.577 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:04:43.577 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T10:04:43.577 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T10:04:43.577 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T10:04:43.577 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T10:04:43.577 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T10:04:43.577 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T10:04:43.578 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T10:04:43.601 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T10:04:43.602 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T10:04:43.602 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T10:04:43.602 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_32 2024-04-06T10:04:43.708 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:Invalid command: unused arguments: ['mirror/image_32'] 2024-04-06T10:04:43.708 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:rbd mirror flush : flush rbd mirror 2024-04-06T10:04:43.708 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:admin_socket: invalid command 2024-04-06T10:04:43.710 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T10:04:43.710 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/image_32 2024-04-06T10:04:43.710 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:1 2024-04-06T10:04:43.710 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T10:04:43.710 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T10:04:43.710 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-06T10:04:43.711 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:1 2024-04-06T10:04:43.711 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T10:04:43.711 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T10:04:43.711 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T10:04:43.711 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=1 2024-04-06T10:04:43.711 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-06T10:04:43.711 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T10:04:43.711 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:04:43.711 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=1 2024-04-06T10:04:43.711 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=1 2024-04-06T10:04:43.714 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-06T10:04:43.714 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:1 2024-04-06T10:04:43.715 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T10:04:43.715 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T10:04:43.715 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-06T10:04:43.715 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:1 2024-04-06T10:04:43.715 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T10:04:43.715 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T10:04:43.715 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:04:43.715 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=1 2024-04-06T10:04:43.715 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-06T10:04:43.715 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T10:04:43.715 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T10:04:43.715 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=1 2024-04-06T10:04:43.715 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=1 2024-04-06T10:04:43.715 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-06T10:04:43.737 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T10:04:43.738 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T10:04:43.738 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T10:04:43.738 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/image_32 2024-04-06T10:04:43.889 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T10:04:43.890 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror image_32 2024-04-06T10:04:43.890 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:04:43.890 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:04:43.890 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_32 2024-04-06T10:04:43.890 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror image_32 '' 2024-04-06T10:04:43.890 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:04:43.891 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:04:43.891 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_32 2024-04-06T10:04:43.891 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T10:04:43.891 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-image_32.status 2024-04-06T10:04:43.891 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-image_32.status 2024-04-06T10:04:43.891 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T10:04:43.892 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-image_32.status 2024-04-06T10:04:43.893 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_32 2024-04-06T10:04:43.893 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_32.status 2024-04-06T10:04:43.946 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 1 2024-04-06T10:04:43.946 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 1 2024-04-06T10:04:43.946 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T10:04:43.946 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=5, tag_tid=3, entry_tid=1], [object_number=4, tag_tid=3, entry_tid=0], [object_number=7, tag_tid=1, entry_tid=6511], [object_number=6, tag_tid=1, entry_tid=6510]]], state=connected] 2024-04-06T10:04:43.946 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=5, tag_tid=3, entry_tid=1], [object_number=4, tag_tid=3, entry_tid=0], [object_number=7, tag_tid=1, entry_tid=6511], [object_number=6, tag_tid=1, entry_tid=6510]]], state=connected] 2024-04-06T10:04:43.950 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_32.status 2024-04-06T10:04:43.952 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=5, tag_tid=3, entry_tid=1' 2024-04-06T10:04:43.952 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror image_32 2024-04-06T10:04:43.952 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:04:43.952 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:04:43.952 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_32 2024-04-06T10:04:43.952 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror image_32 '..*' 2024-04-06T10:04:43.952 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:04:43.953 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:04:43.953 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_32 2024-04-06T10:04:43.953 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T10:04:43.953 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-image_32.status 2024-04-06T10:04:43.953 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-image_32.status 2024-04-06T10:04:43.953 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T10:04:43.953 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-image_32.status 2024-04-06T10:04:43.954 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_32 2024-04-06T10:04:43.954 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_32.status 2024-04-06T10:04:44.000 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 1 2024-04-06T10:04:44.000 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 1 2024-04-06T10:04:44.000 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T10:04:44.000 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=5, tag_tid=3, entry_tid=1], [object_number=4, tag_tid=3, entry_tid=0], [object_number=7, tag_tid=1, entry_tid=6511], [object_number=6, tag_tid=1, entry_tid=6510]]], state=connected] 2024-04-06T10:04:44.000 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=5, tag_tid=3, entry_tid=1], [object_number=4, tag_tid=3, entry_tid=0], [object_number=7, tag_tid=1, entry_tid=6511], [object_number=6, tag_tid=1, entry_tid=6510]]], state=connected] 2024-04-06T10:04:44.004 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_32.status 2024-04-06T10:04:44.006 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=5, tag_tid=3, entry_tid=1' 2024-04-06T10:04:44.006 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=5, tag_tid=3, entry_tid=1' -a 'object_number=5, tag_tid=3, entry_tid=1' = 'object_number=5, tag_tid=3, entry_tid=1' 2024-04-06T10:04:44.006 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T10:04:44.006 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_snap_present cluster1 mirror image_32 snap 2024-04-06T10:04:44.006 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T10:04:44.006 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:04:44.006 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_32 2024-04-06T10:04:44.006 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local snap_name=snap 2024-04-06T10:04:44.007 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local s 2024-04-06T10:04:44.007 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-04-06T10:04:44.007 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 1 2024-04-06T10:04:45.007 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 info mirror/image_32@snap 2024-04-06T10:04:45.074 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:rbd image 'image_32': 2024-04-06T10:04:45.074 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: size 128 MiB in 32 objects 2024-04-06T10:04:45.074 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: order 22 (4 MiB objects) 2024-04-06T10:04:45.074 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: snapshot_count: 1 2024-04-06T10:04:45.074 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: id: 10ac42167b4b 2024-04-06T10:04:45.074 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: block_name_prefix: rbd_data.10ac42167b4b 2024-04-06T10:04:45.075 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: format: 2 2024-04-06T10:04:45.075 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: features: layering, exclusive-lock, journaling 2024-04-06T10:04:45.075 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: op_features: 2024-04-06T10:04:45.075 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: flags: 2024-04-06T10:04:45.075 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: create_timestamp: Sat Apr 6 09:55:35 2024 2024-04-06T10:04:45.075 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: access_timestamp: Sat Apr 6 09:55:35 2024 2024-04-06T10:04:45.075 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: modify_timestamp: Sat Apr 6 09:56:41 2024 2024-04-06T10:04:45.075 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: protected: False 2024-04-06T10:04:45.075 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: journal: 10ac42167b4b 2024-04-06T10:04:45.075 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: mirroring state: enabled 2024-04-06T10:04:45.075 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: mirroring mode: journal 2024-04-06T10:04:45.075 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: mirroring global id: 9d3f7406-7b0d-4008-8b07-5e200aecaae8 2024-04-06T10:04:45.075 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: mirroring primary: false 2024-04-06T10:04:45.079 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T10:04:45.080 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ compare_image_snaps mirror image_32 snap 2024-04-06T10:04:45.080 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:04:45.080 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_32 2024-04-06T10:04:45.080 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local snap_name=snap 2024-04-06T10:04:45.080 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local ret=0 2024-04-06T10:04:45.080 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local rmt_export=/tmp/tmp.BhW6xKmROU/cluster2-mirror-image_32.export 2024-04-06T10:04:45.080 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loc_export=/tmp/tmp.BhW6xKmROU/cluster1-mirror-image_32.export 2024-04-06T10:04:45.080 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rm -f /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_32.export /tmp/tmp.BhW6xKmROU/cluster1-mirror-image_32.export 2024-04-06T10:04:45.080 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster2 -p mirror export image_32@snap /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_32.export 2024-04-06T10:04:45.255 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-04-06T10:04:45.264 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 -p mirror export image_32@snap /tmp/tmp.BhW6xKmROU/cluster1-mirror-image_32.export 2024-04-06T10:04:45.621 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-04-06T10:04:45.633 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmp /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_32.export /tmp/tmp.BhW6xKmROU/cluster1-mirror-image_32.export 2024-04-06T10:04:45.717 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rm -f /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_32.export /tmp/tmp.BhW6xKmROU/cluster1-mirror-image_32.export 2024-04-06T10:04:45.758 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T10:04:45.759 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-06T10:04:45.759 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ image=image_33 2024-04-06T10:04:45.759 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ create_snap cluster2 mirror image_33 snap 2024-04-06T10:04:45.759 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-04-06T10:04:45.759 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:04:45.759 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_33 2024-04-06T10:04:45.759 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local snap_name=snap 2024-04-06T10:04:45.759 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster2 -p mirror snap create image_33@snap --debug-rbd=20 --debug-journaler=20 2024-04-06T10:04:47.926 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_image_replay_started cluster1 mirror image_33 2024-04-06T10:04:47.927 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T10:04:47.927 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:04:47.927 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_33 2024-04-06T10:04:47.927 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_image_replay_state cluster1 mirror image_33 started 2024-04-06T10:04:47.927 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T10:04:47.927 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:04:47.927 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_33 2024-04-06T10:04:47.927 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state=started 2024-04-06T10:04:47.927 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local s 2024-04-06T10:04:47.927 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T10:04:47.927 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 1 2024-04-06T10:04:48.928 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test_image_replay_state cluster1 mirror image_33 started 2024-04-06T10:04:48.929 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T10:04:48.929 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:04:48.929 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_33 2024-04-06T10:04:48.929 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local test_state=started 2024-04-06T10:04:48.929 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local status_result 2024-04-06T10:04:48.929 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local current_state=stopped 2024-04-06T10:04:48.930 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_33 2024-04-06T10:04:48.930 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_instance=cluster1 2024-04-06T10:04:48.930 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ shift 2024-04-06T10:04:48.930 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T10:04:48.930 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance=cluster1 2024-04-06T10:04:48.931 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local loop_instance 2024-04-06T10:04:48.931 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T10:04:48.931 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sleep 0 2024-04-06T10:04:48.931 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -i state 2024-04-06T10:04:48.931 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-04-06T10:04:48.931 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ seq 0 3 2024-04-06T10:04:48.932 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T10:04:48.932 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_33 2024-04-06T10:04:48.933 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1:0 2024-04-06T10:04:48.933 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ shift 2024-04-06T10:04:48.933 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T10:04:48.933 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-04-06T10:04:48.933 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T10:04:48.933 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=cluster 2024-04-06T10:04:48.933 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T10:04:48.933 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:04:48.933 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T10:04:48.933 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T10:04:48.933 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval cluster=cluster1 2024-04-06T10:04:48.933 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ cluster=cluster1 2024-04-06T10:04:48.933 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T10:04:48.933 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T10:04:48.935 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-04-06T10:04:48.935 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local local_cluster=cluster1:0 2024-04-06T10:04:48.935 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local cluster=cluster1 2024-04-06T10:04:48.935 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local instance 2024-04-06T10:04:48.935 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T10:04:48.935 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local val=cluster1:0 2024-04-06T10:04:48.935 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local cluster_var_name=local_cluster 2024-04-06T10:04:48.935 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local instance_var_name=instance 2024-04-06T10:04:48.935 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ cluster=cluster1 2024-04-06T10:04:48.936 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T10:04:48.936 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ '[' 0 = cluster1:0 ']' 2024-04-06T10:04:48.936 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ eval local_cluster=cluster1 2024-04-06T10:04:48.936 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++++ local_cluster=cluster1 2024-04-06T10:04:48.937 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ eval instance=0 2024-04-06T10:04:48.937 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++++ instance=0 2024-04-06T10:04:48.937 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T10:04:48.956 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T10:04:48.957 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T10:04:48.957 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T10:04:48.957 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_33 2024-04-06T10:04:49.061 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:Invalid command: unused arguments: ['mirror/image_33'] 2024-04-06T10:04:49.061 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:rbd mirror status : get status for rbd mirror 2024-04-06T10:04:49.061 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:admin_socket: invalid command 2024-04-06T10:04:49.064 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T10:04:49.064 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_33 2024-04-06T10:04:49.064 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1:1 2024-04-06T10:04:49.064 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ shift 2024-04-06T10:04:49.064 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T10:04:49.065 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-04-06T10:04:49.065 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:1 2024-04-06T10:04:49.065 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=cluster 2024-04-06T10:04:49.065 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T10:04:49.065 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:04:49.065 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=1 2024-04-06T10:04:49.065 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-06T10:04:49.065 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval cluster=cluster1 2024-04-06T10:04:49.065 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ cluster=cluster1 2024-04-06T10:04:49.065 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=1 2024-04-06T10:04:49.065 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=1 2024-04-06T10:04:49.065 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-04-06T10:04:49.065 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local local_cluster=cluster1:1 2024-04-06T10:04:49.065 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local cluster=cluster1 2024-04-06T10:04:49.066 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local instance 2024-04-06T10:04:49.066 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-06T10:04:49.066 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local val=cluster1:1 2024-04-06T10:04:49.066 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local cluster_var_name=local_cluster 2024-04-06T10:04:49.066 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local instance_var_name=instance 2024-04-06T10:04:49.066 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ cluster=cluster1 2024-04-06T10:04:49.066 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=1 2024-04-06T10:04:49.066 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ '[' 1 = cluster1:1 ']' 2024-04-06T10:04:49.066 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ eval local_cluster=cluster1 2024-04-06T10:04:49.066 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++++ local_cluster=cluster1 2024-04-06T10:04:49.066 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ eval instance=1 2024-04-06T10:04:49.066 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++++ instance=1 2024-04-06T10:04:49.066 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-06T10:04:49.083 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T10:04:49.083 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T10:04:49.083 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T10:04:49.083 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_33 2024-04-06T10:04:49.230 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ return 0 2024-04-06T10:04:49.230 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ status_result=' "state": "Replaying"' 2024-04-06T10:04:49.231 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep -i Replaying 2024-04-06T10:04:49.231 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ echo ' "state": "Replaying"' 2024-04-06T10:04:49.232 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: "state": "Replaying" 2024-04-06T10:04:49.233 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ current_state=started 2024-04-06T10:04:49.234 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test started = started 2024-04-06T10:04:49.234 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T10:04:49.234 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_33 2024-04-06T10:04:49.234 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local local_cluster=cluster1 2024-04-06T10:04:49.234 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-04-06T10:04:49.234 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:04:49.234 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_33 2024-04-06T10:04:49.234 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' journal = journal ']' 2024-04-06T10:04:49.234 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_journal_replay_complete cluster1 cluster2 mirror image_33 2024-04-06T10:04:49.234 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local local_cluster=cluster1 2024-04-06T10:04:49.234 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-04-06T10:04:49.235 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:04:49.235 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_33 2024-04-06T10:04:49.235 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local s master_pos mirror_pos last_mirror_pos 2024-04-06T10:04:49.235 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local master_tag master_entry mirror_tag mirror_entry 2024-04-06T10:04:49.235 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T10:04:49.235 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T10:04:49.235 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T10:04:49.435 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror image_33 2024-04-06T10:04:49.435 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T10:04:49.435 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:04:49.435 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_33 2024-04-06T10:04:49.436 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T10:04:49.436 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n image_33 ']' 2024-04-06T10:04:49.436 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/image_33' 2024-04-06T10:04:49.436 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_33 2024-04-06T10:04:49.436 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T10:04:49.436 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T10:04:49.436 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T10:04:49.436 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T10:04:49.436 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T10:04:49.436 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T10:04:49.436 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T10:04:49.437 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T10:04:49.437 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T10:04:49.443 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T10:04:49.444 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_33 2024-04-06T10:04:49.444 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T10:04:49.444 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T10:04:49.444 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T10:04:49.444 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T10:04:49.444 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T10:04:49.444 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T10:04:49.445 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T10:04:49.445 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T10:04:49.445 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T10:04:49.445 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T10:04:49.445 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T10:04:49.445 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:04:49.445 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T10:04:49.445 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T10:04:49.445 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T10:04:49.445 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T10:04:49.445 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T10:04:49.445 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T10:04:49.445 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T10:04:49.445 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T10:04:49.445 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T10:04:49.446 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T10:04:49.446 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:04:49.446 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T10:04:49.446 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T10:04:49.446 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T10:04:49.446 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T10:04:49.446 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T10:04:49.446 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T10:04:49.446 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T10:04:49.467 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T10:04:49.467 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T10:04:49.467 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T10:04:49.467 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_33 2024-04-06T10:04:49.585 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:Invalid command: unused arguments: ['mirror/image_33'] 2024-04-06T10:04:49.585 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:rbd mirror flush : flush rbd mirror 2024-04-06T10:04:49.585 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:admin_socket: invalid command 2024-04-06T10:04:49.589 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T10:04:49.589 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/image_33 2024-04-06T10:04:49.589 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:1 2024-04-06T10:04:49.589 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T10:04:49.589 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T10:04:49.589 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-06T10:04:49.589 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:1 2024-04-06T10:04:49.589 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T10:04:49.589 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T10:04:49.589 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T10:04:49.589 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=1 2024-04-06T10:04:49.590 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-06T10:04:49.590 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T10:04:49.590 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:04:49.590 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=1 2024-04-06T10:04:49.590 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=1 2024-04-06T10:04:49.590 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-06T10:04:49.590 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:1 2024-04-06T10:04:49.590 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T10:04:49.590 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T10:04:49.590 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-06T10:04:49.590 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:1 2024-04-06T10:04:49.590 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T10:04:49.591 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T10:04:49.591 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:04:49.591 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=1 2024-04-06T10:04:49.591 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-06T10:04:49.591 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T10:04:49.591 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T10:04:49.591 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=1 2024-04-06T10:04:49.591 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=1 2024-04-06T10:04:49.591 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-06T10:04:49.611 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T10:04:49.611 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T10:04:49.611 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T10:04:49.612 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/image_33 2024-04-06T10:04:49.749 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T10:04:49.749 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror image_33 2024-04-06T10:04:49.750 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:04:49.750 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:04:49.750 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_33 2024-04-06T10:04:49.750 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror image_33 '' 2024-04-06T10:04:49.750 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:04:49.750 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:04:49.750 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_33 2024-04-06T10:04:49.750 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T10:04:49.751 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-image_33.status 2024-04-06T10:04:49.751 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-image_33.status 2024-04-06T10:04:49.751 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T10:04:49.753 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-image_33.status 2024-04-06T10:04:49.754 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_33.status 2024-04-06T10:04:49.754 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_33 2024-04-06T10:04:49.800 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 1 2024-04-06T10:04:49.801 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 1 2024-04-06T10:04:49.801 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T10:04:49.801 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=5, tag_tid=2, entry_tid=1], [object_number=4, tag_tid=2, entry_tid=0], [object_number=7, tag_tid=1, entry_tid=6267], [object_number=6, tag_tid=1, entry_tid=6266]]], state=connected] 2024-04-06T10:04:49.801 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=5, tag_tid=1, entry_tid=6269], [object_number=4, tag_tid=1, entry_tid=6268], [object_number=7, tag_tid=1, entry_tid=6267], [object_number=6, tag_tid=1, entry_tid=6266]]], state=connected] 2024-04-06T10:04:49.804 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_33.status 2024-04-06T10:04:49.805 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=5, tag_tid=2, entry_tid=1' 2024-04-06T10:04:49.806 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror image_33 2024-04-06T10:04:49.807 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:04:49.807 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:04:49.807 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_33 2024-04-06T10:04:49.807 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror image_33 '..*' 2024-04-06T10:04:49.807 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:04:49.807 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:04:49.807 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_33 2024-04-06T10:04:49.807 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T10:04:49.807 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-image_33.status 2024-04-06T10:04:49.807 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-image_33.status 2024-04-06T10:04:49.807 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T10:04:49.808 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-image_33.status 2024-04-06T10:04:49.808 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_33 2024-04-06T10:04:49.808 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_33.status 2024-04-06T10:04:49.854 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 1 2024-04-06T10:04:49.854 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 1 2024-04-06T10:04:49.854 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T10:04:49.854 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=5, tag_tid=2, entry_tid=1], [object_number=4, tag_tid=2, entry_tid=0], [object_number=7, tag_tid=1, entry_tid=6267], [object_number=6, tag_tid=1, entry_tid=6266]]], state=connected] 2024-04-06T10:04:49.854 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=5, tag_tid=1, entry_tid=6269], [object_number=4, tag_tid=1, entry_tid=6268], [object_number=7, tag_tid=1, entry_tid=6267], [object_number=6, tag_tid=1, entry_tid=6266]]], state=connected] 2024-04-06T10:04:49.857 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_33.status 2024-04-06T10:04:49.859 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=5, tag_tid=1, entry_tid=6269' 2024-04-06T10:04:49.859 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=5, tag_tid=2, entry_tid=1' -a 'object_number=5, tag_tid=2, entry_tid=1' = 'object_number=5, tag_tid=1, entry_tid=6269' 2024-04-06T10:04:49.859 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=5, tag_tid=1, entry_tid=6269' '!=' '' 2024-04-06T10:04:49.859 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T10:04:49.860 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=5, tag_tid=1, entry_tid=6269' = '' 2024-04-06T10:04:49.860 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=5, tag_tid=1, entry_tid=6269' 2024-04-06T10:04:49.860 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=5, tag_tid=2, entry_tid=1' 2024-04-06T10:04:49.860 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T10:04:49.860 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T10:04:49.860 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=2 2024-04-06T10:04:49.861 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=5, tag_tid=1, entry_tid=6269' 2024-04-06T10:04:49.861 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T10:04:49.861 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T10:04:49.862 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=1 2024-04-06T10:04:49.863 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=5, tag_tid=2, entry_tid=1' 2024-04-06T10:04:49.863 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T10:04:49.863 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T10:04:49.864 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T10:04:49.864 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=5, tag_tid=1, entry_tid=6269' 2024-04-06T10:04:49.865 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T10:04:49.865 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T10:04:49.866 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=6269 2024-04-06T10:04:49.866 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 2 = 1 -a 1 -le 6269 2024-04-06T10:04:49.866 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T10:04:49.866 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T10:04:49.866 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T10:04:50.068 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror image_33 2024-04-06T10:04:50.068 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T10:04:50.068 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:04:50.068 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_33 2024-04-06T10:04:50.068 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T10:04:50.068 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n image_33 ']' 2024-04-06T10:04:50.068 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/image_33' 2024-04-06T10:04:50.069 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_33 2024-04-06T10:04:50.069 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T10:04:50.069 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T10:04:50.069 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T10:04:50.069 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T10:04:50.069 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T10:04:50.069 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T10:04:50.069 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T10:04:50.070 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T10:04:50.070 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T10:04:50.071 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T10:04:50.071 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_33 2024-04-06T10:04:50.071 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T10:04:50.071 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T10:04:50.071 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T10:04:50.072 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T10:04:50.072 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T10:04:50.072 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T10:04:50.072 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T10:04:50.072 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T10:04:50.072 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T10:04:50.072 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T10:04:50.072 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T10:04:50.072 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:04:50.072 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T10:04:50.072 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T10:04:50.074 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T10:04:50.074 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T10:04:50.074 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T10:04:50.074 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T10:04:50.074 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T10:04:50.074 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T10:04:50.074 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T10:04:50.074 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T10:04:50.074 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:04:50.075 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T10:04:50.075 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T10:04:50.075 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T10:04:50.075 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T10:04:50.075 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T10:04:50.075 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T10:04:50.075 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T10:04:50.098 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T10:04:50.099 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T10:04:50.099 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T10:04:50.099 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_33 2024-04-06T10:04:50.204 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:Invalid command: unused arguments: ['mirror/image_33'] 2024-04-06T10:04:50.204 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:rbd mirror flush : flush rbd mirror 2024-04-06T10:04:50.204 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:admin_socket: invalid command 2024-04-06T10:04:50.208 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T10:04:50.208 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/image_33 2024-04-06T10:04:50.208 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:1 2024-04-06T10:04:50.208 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T10:04:50.208 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T10:04:50.208 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-06T10:04:50.208 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:1 2024-04-06T10:04:50.209 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T10:04:50.209 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T10:04:50.209 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T10:04:50.209 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=1 2024-04-06T10:04:50.209 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-06T10:04:50.209 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T10:04:50.209 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:04:50.209 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=1 2024-04-06T10:04:50.209 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=1 2024-04-06T10:04:50.209 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-06T10:04:50.209 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:1 2024-04-06T10:04:50.209 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T10:04:50.209 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T10:04:50.209 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-06T10:04:50.209 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:1 2024-04-06T10:04:50.210 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T10:04:50.210 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T10:04:50.210 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:04:50.210 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=1 2024-04-06T10:04:50.210 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-06T10:04:50.210 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T10:04:50.210 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T10:04:50.210 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=1 2024-04-06T10:04:50.210 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=1 2024-04-06T10:04:50.210 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-06T10:04:50.232 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T10:04:50.232 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T10:04:50.232 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T10:04:50.232 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/image_33 2024-04-06T10:04:50.372 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T10:04:50.373 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror image_33 2024-04-06T10:04:50.373 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:04:50.373 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:04:50.373 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_33 2024-04-06T10:04:50.374 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror image_33 '' 2024-04-06T10:04:50.374 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:04:50.374 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:04:50.374 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_33 2024-04-06T10:04:50.374 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T10:04:50.374 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-image_33.status 2024-04-06T10:04:50.374 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-image_33.status 2024-04-06T10:04:50.374 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T10:04:50.376 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-image_33.status 2024-04-06T10:04:50.376 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_33 2024-04-06T10:04:50.376 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_33.status 2024-04-06T10:04:50.428 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 1 2024-04-06T10:04:50.428 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 1 2024-04-06T10:04:50.428 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T10:04:50.428 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=5, tag_tid=2, entry_tid=1], [object_number=4, tag_tid=2, entry_tid=0], [object_number=7, tag_tid=1, entry_tid=6267], [object_number=6, tag_tid=1, entry_tid=6266]]], state=connected] 2024-04-06T10:04:50.428 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=5, tag_tid=1, entry_tid=6269], [object_number=4, tag_tid=1, entry_tid=6268], [object_number=7, tag_tid=1, entry_tid=6267], [object_number=6, tag_tid=1, entry_tid=6266]]], state=connected] 2024-04-06T10:04:50.431 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_33.status 2024-04-06T10:04:50.432 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=5, tag_tid=2, entry_tid=1' 2024-04-06T10:04:50.434 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror image_33 2024-04-06T10:04:50.434 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:04:50.434 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:04:50.434 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_33 2024-04-06T10:04:50.434 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror image_33 '..*' 2024-04-06T10:04:50.434 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:04:50.435 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:04:50.435 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_33 2024-04-06T10:04:50.435 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T10:04:50.435 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-image_33.status 2024-04-06T10:04:50.435 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-image_33.status 2024-04-06T10:04:50.435 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T10:04:50.436 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-image_33.status 2024-04-06T10:04:50.436 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_33 2024-04-06T10:04:50.436 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_33.status 2024-04-06T10:04:50.480 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 1 2024-04-06T10:04:50.481 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 1 2024-04-06T10:04:50.481 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T10:04:50.481 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=5, tag_tid=2, entry_tid=1], [object_number=4, tag_tid=2, entry_tid=0], [object_number=7, tag_tid=1, entry_tid=6267], [object_number=6, tag_tid=1, entry_tid=6266]]], state=connected] 2024-04-06T10:04:50.481 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=5, tag_tid=1, entry_tid=6269], [object_number=4, tag_tid=1, entry_tid=6268], [object_number=7, tag_tid=1, entry_tid=6267], [object_number=6, tag_tid=1, entry_tid=6266]]], state=connected] 2024-04-06T10:04:50.484 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_33.status 2024-04-06T10:04:50.486 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=5, tag_tid=1, entry_tid=6269' 2024-04-06T10:04:50.486 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=5, tag_tid=2, entry_tid=1' -a 'object_number=5, tag_tid=2, entry_tid=1' = 'object_number=5, tag_tid=1, entry_tid=6269' 2024-04-06T10:04:50.486 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=5, tag_tid=1, entry_tid=6269' '!=' 'object_number=5, tag_tid=1, entry_tid=6269' 2024-04-06T10:04:50.486 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T10:04:50.486 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.4 2024-04-06T10:04:50.888 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror image_33 2024-04-06T10:04:50.888 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T10:04:50.888 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:04:50.888 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_33 2024-04-06T10:04:50.888 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T10:04:50.888 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n image_33 ']' 2024-04-06T10:04:50.889 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/image_33' 2024-04-06T10:04:50.889 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_33 2024-04-06T10:04:50.889 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T10:04:50.889 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T10:04:50.889 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T10:04:50.889 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T10:04:50.889 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T10:04:50.889 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T10:04:50.889 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T10:04:50.890 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T10:04:50.890 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T10:04:50.892 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T10:04:50.892 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_33 2024-04-06T10:04:50.892 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T10:04:50.893 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T10:04:50.893 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T10:04:50.893 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T10:04:50.893 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T10:04:50.893 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T10:04:50.893 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T10:04:50.893 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T10:04:50.893 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T10:04:50.893 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T10:04:50.893 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T10:04:50.893 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:04:50.893 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T10:04:50.893 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T10:04:50.893 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T10:04:50.895 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T10:04:50.895 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T10:04:50.895 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T10:04:50.895 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T10:04:50.895 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T10:04:50.895 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T10:04:50.895 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T10:04:50.895 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:04:50.895 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T10:04:50.895 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T10:04:50.896 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T10:04:50.896 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T10:04:50.896 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T10:04:50.896 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T10:04:50.896 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T10:04:50.919 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T10:04:50.919 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T10:04:50.920 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T10:04:50.920 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_33 2024-04-06T10:04:51.026 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:Invalid command: unused arguments: ['mirror/image_33'] 2024-04-06T10:04:51.026 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:rbd mirror flush : flush rbd mirror 2024-04-06T10:04:51.026 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:admin_socket: invalid command 2024-04-06T10:04:51.028 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T10:04:51.029 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/image_33 2024-04-06T10:04:51.029 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:1 2024-04-06T10:04:51.029 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T10:04:51.029 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T10:04:51.029 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-06T10:04:51.029 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:1 2024-04-06T10:04:51.029 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T10:04:51.029 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T10:04:51.029 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T10:04:51.029 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=1 2024-04-06T10:04:51.029 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-06T10:04:51.029 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T10:04:51.029 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:04:51.030 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=1 2024-04-06T10:04:51.030 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=1 2024-04-06T10:04:51.031 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-06T10:04:51.032 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:1 2024-04-06T10:04:51.032 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T10:04:51.032 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T10:04:51.032 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-06T10:04:51.032 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:1 2024-04-06T10:04:51.032 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T10:04:51.032 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T10:04:51.032 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:04:51.032 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=1 2024-04-06T10:04:51.032 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-06T10:04:51.032 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T10:04:51.032 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T10:04:51.032 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=1 2024-04-06T10:04:51.032 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=1 2024-04-06T10:04:51.032 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-06T10:04:51.055 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T10:04:51.056 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T10:04:51.056 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T10:04:51.056 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/image_33 2024-04-06T10:04:51.193 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T10:04:51.194 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror image_33 2024-04-06T10:04:51.194 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:04:51.194 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:04:51.194 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_33 2024-04-06T10:04:51.194 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror image_33 '' 2024-04-06T10:04:51.194 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:04:51.194 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:04:51.194 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_33 2024-04-06T10:04:51.195 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T10:04:51.195 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-image_33.status 2024-04-06T10:04:51.195 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-image_33.status 2024-04-06T10:04:51.195 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T10:04:51.198 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-image_33.status 2024-04-06T10:04:51.198 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_33 2024-04-06T10:04:51.198 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_33.status 2024-04-06T10:04:51.257 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 1 2024-04-06T10:04:51.257 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 1 2024-04-06T10:04:51.257 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T10:04:51.257 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=5, tag_tid=2, entry_tid=1], [object_number=4, tag_tid=2, entry_tid=0], [object_number=7, tag_tid=1, entry_tid=6267], [object_number=6, tag_tid=1, entry_tid=6266]]], state=connected] 2024-04-06T10:04:51.257 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=5, tag_tid=1, entry_tid=6269], [object_number=4, tag_tid=1, entry_tid=6268], [object_number=7, tag_tid=1, entry_tid=6267], [object_number=6, tag_tid=1, entry_tid=6266]]], state=connected] 2024-04-06T10:04:51.261 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_33.status 2024-04-06T10:04:51.262 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=5, tag_tid=2, entry_tid=1' 2024-04-06T10:04:51.264 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror image_33 2024-04-06T10:04:51.264 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:04:51.264 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:04:51.264 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_33 2024-04-06T10:04:51.264 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror image_33 '..*' 2024-04-06T10:04:51.264 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:04:51.265 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:04:51.265 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_33 2024-04-06T10:04:51.265 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T10:04:51.265 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-image_33.status 2024-04-06T10:04:51.265 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-image_33.status 2024-04-06T10:04:51.265 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T10:04:51.266 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-image_33.status 2024-04-06T10:04:51.267 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_33 2024-04-06T10:04:51.267 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_33.status 2024-04-06T10:04:51.311 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 1 2024-04-06T10:04:51.311 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 1 2024-04-06T10:04:51.311 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T10:04:51.311 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=5, tag_tid=2, entry_tid=1], [object_number=4, tag_tid=2, entry_tid=0], [object_number=7, tag_tid=1, entry_tid=6267], [object_number=6, tag_tid=1, entry_tid=6266]]], state=connected] 2024-04-06T10:04:51.311 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=5, tag_tid=1, entry_tid=6269], [object_number=4, tag_tid=1, entry_tid=6268], [object_number=7, tag_tid=1, entry_tid=6267], [object_number=6, tag_tid=1, entry_tid=6266]]], state=connected] 2024-04-06T10:04:51.314 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_33.status 2024-04-06T10:04:51.316 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=5, tag_tid=1, entry_tid=6269' 2024-04-06T10:04:51.316 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=5, tag_tid=2, entry_tid=1' -a 'object_number=5, tag_tid=2, entry_tid=1' = 'object_number=5, tag_tid=1, entry_tid=6269' 2024-04-06T10:04:51.316 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=5, tag_tid=1, entry_tid=6269' '!=' 'object_number=5, tag_tid=1, entry_tid=6269' 2024-04-06T10:04:51.316 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T10:04:51.316 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.8 2024-04-06T10:04:52.118 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror image_33 2024-04-06T10:04:52.118 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T10:04:52.118 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:04:52.118 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_33 2024-04-06T10:04:52.118 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T10:04:52.118 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n image_33 ']' 2024-04-06T10:04:52.119 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/image_33' 2024-04-06T10:04:52.119 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_33 2024-04-06T10:04:52.119 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T10:04:52.119 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T10:04:52.119 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T10:04:52.119 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T10:04:52.119 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T10:04:52.119 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T10:04:52.119 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T10:04:52.120 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T10:04:52.120 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T10:04:52.122 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T10:04:52.122 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_33 2024-04-06T10:04:52.122 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T10:04:52.122 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T10:04:52.123 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T10:04:52.123 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T10:04:52.123 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T10:04:52.123 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T10:04:52.123 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T10:04:52.123 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T10:04:52.123 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T10:04:52.123 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T10:04:52.123 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T10:04:52.123 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:04:52.123 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T10:04:52.123 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T10:04:52.123 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T10:04:52.124 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T10:04:52.124 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T10:04:52.124 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T10:04:52.124 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T10:04:52.124 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T10:04:52.124 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T10:04:52.124 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T10:04:52.124 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:04:52.124 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T10:04:52.124 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T10:04:52.124 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T10:04:52.124 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T10:04:52.124 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T10:04:52.124 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T10:04:52.125 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T10:04:52.147 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T10:04:52.148 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T10:04:52.148 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T10:04:52.148 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_33 2024-04-06T10:04:52.254 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:Invalid command: unused arguments: ['mirror/image_33'] 2024-04-06T10:04:52.254 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:rbd mirror flush : flush rbd mirror 2024-04-06T10:04:52.254 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:admin_socket: invalid command 2024-04-06T10:04:52.256 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T10:04:52.256 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/image_33 2024-04-06T10:04:52.256 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:1 2024-04-06T10:04:52.256 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T10:04:52.256 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T10:04:52.256 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-06T10:04:52.256 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:1 2024-04-06T10:04:52.256 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T10:04:52.256 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T10:04:52.256 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T10:04:52.256 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=1 2024-04-06T10:04:52.256 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-06T10:04:52.256 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T10:04:52.257 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:04:52.257 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=1 2024-04-06T10:04:52.257 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=1 2024-04-06T10:04:52.258 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-06T10:04:52.258 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:1 2024-04-06T10:04:52.259 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T10:04:52.259 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T10:04:52.259 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-06T10:04:52.259 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:1 2024-04-06T10:04:52.259 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T10:04:52.259 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T10:04:52.259 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:04:52.259 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=1 2024-04-06T10:04:52.259 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-06T10:04:52.259 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T10:04:52.259 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T10:04:52.259 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=1 2024-04-06T10:04:52.259 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=1 2024-04-06T10:04:52.259 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-06T10:04:52.281 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T10:04:52.281 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T10:04:52.281 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T10:04:52.281 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/image_33 2024-04-06T10:04:52.434 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T10:04:52.436 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror image_33 2024-04-06T10:04:52.436 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:04:52.436 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:04:52.436 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_33 2024-04-06T10:04:52.436 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror image_33 '' 2024-04-06T10:04:52.436 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:04:52.436 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:04:52.436 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_33 2024-04-06T10:04:52.436 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T10:04:52.436 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-image_33.status 2024-04-06T10:04:52.437 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-image_33.status 2024-04-06T10:04:52.437 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T10:04:52.438 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-image_33.status 2024-04-06T10:04:52.439 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_33 2024-04-06T10:04:52.439 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_33.status 2024-04-06T10:04:52.490 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 1 2024-04-06T10:04:52.491 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 1 2024-04-06T10:04:52.491 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T10:04:52.491 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=5, tag_tid=2, entry_tid=1], [object_number=4, tag_tid=2, entry_tid=0], [object_number=7, tag_tid=1, entry_tid=6267], [object_number=6, tag_tid=1, entry_tid=6266]]], state=connected] 2024-04-06T10:04:52.491 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=5, tag_tid=1, entry_tid=6269], [object_number=4, tag_tid=1, entry_tid=6268], [object_number=7, tag_tid=1, entry_tid=6267], [object_number=6, tag_tid=1, entry_tid=6266]]], state=connected] 2024-04-06T10:04:52.494 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_33.status 2024-04-06T10:04:52.495 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=5, tag_tid=2, entry_tid=1' 2024-04-06T10:04:52.496 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror image_33 2024-04-06T10:04:52.496 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:04:52.496 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:04:52.497 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_33 2024-04-06T10:04:52.497 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror image_33 '..*' 2024-04-06T10:04:52.497 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:04:52.497 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:04:52.497 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_33 2024-04-06T10:04:52.497 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T10:04:52.497 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-image_33.status 2024-04-06T10:04:52.497 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-image_33.status 2024-04-06T10:04:52.497 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T10:04:52.497 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-image_33.status 2024-04-06T10:04:52.498 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_33 2024-04-06T10:04:52.498 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_33.status 2024-04-06T10:04:52.544 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 1 2024-04-06T10:04:52.544 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 1 2024-04-06T10:04:52.544 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T10:04:52.544 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=5, tag_tid=2, entry_tid=1], [object_number=4, tag_tid=2, entry_tid=0], [object_number=7, tag_tid=1, entry_tid=6267], [object_number=6, tag_tid=1, entry_tid=6266]]], state=connected] 2024-04-06T10:04:52.544 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=5, tag_tid=1, entry_tid=6269], [object_number=4, tag_tid=1, entry_tid=6268], [object_number=7, tag_tid=1, entry_tid=6267], [object_number=6, tag_tid=1, entry_tid=6266]]], state=connected] 2024-04-06T10:04:52.547 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_33.status 2024-04-06T10:04:52.549 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=5, tag_tid=1, entry_tid=6269' 2024-04-06T10:04:52.549 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=5, tag_tid=2, entry_tid=1' -a 'object_number=5, tag_tid=2, entry_tid=1' = 'object_number=5, tag_tid=1, entry_tid=6269' 2024-04-06T10:04:52.549 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=5, tag_tid=1, entry_tid=6269' '!=' 'object_number=5, tag_tid=1, entry_tid=6269' 2024-04-06T10:04:52.549 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T10:04:52.549 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 1.6 2024-04-06T10:04:54.151 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror image_33 2024-04-06T10:04:54.151 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T10:04:54.151 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:04:54.151 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_33 2024-04-06T10:04:54.152 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T10:04:54.152 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n image_33 ']' 2024-04-06T10:04:54.152 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/image_33' 2024-04-06T10:04:54.152 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_33 2024-04-06T10:04:54.152 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T10:04:54.152 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T10:04:54.152 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T10:04:54.152 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T10:04:54.152 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T10:04:54.152 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T10:04:54.152 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T10:04:54.153 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T10:04:54.153 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T10:04:54.155 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T10:04:54.155 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_33 2024-04-06T10:04:54.155 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T10:04:54.155 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T10:04:54.155 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T10:04:54.155 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T10:04:54.155 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T10:04:54.156 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T10:04:54.156 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T10:04:54.156 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T10:04:54.156 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T10:04:54.156 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T10:04:54.156 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T10:04:54.156 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:04:54.156 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T10:04:54.156 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T10:04:54.156 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T10:04:54.156 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T10:04:54.158 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T10:04:54.158 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T10:04:54.158 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T10:04:54.158 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T10:04:54.158 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T10:04:54.158 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T10:04:54.158 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:04:54.158 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T10:04:54.158 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T10:04:54.159 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T10:04:54.159 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T10:04:54.159 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T10:04:54.159 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T10:04:54.159 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T10:04:54.182 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T10:04:54.182 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T10:04:54.182 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T10:04:54.182 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_33 2024-04-06T10:04:54.287 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:Invalid command: unused arguments: ['mirror/image_33'] 2024-04-06T10:04:54.287 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:rbd mirror flush : flush rbd mirror 2024-04-06T10:04:54.287 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:admin_socket: invalid command 2024-04-06T10:04:54.289 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T10:04:54.289 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/image_33 2024-04-06T10:04:54.290 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:1 2024-04-06T10:04:54.290 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T10:04:54.290 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T10:04:54.290 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-06T10:04:54.290 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:1 2024-04-06T10:04:54.290 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T10:04:54.290 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T10:04:54.290 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T10:04:54.290 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=1 2024-04-06T10:04:54.290 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-06T10:04:54.290 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T10:04:54.290 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:04:54.290 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=1 2024-04-06T10:04:54.291 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=1 2024-04-06T10:04:54.291 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-06T10:04:54.291 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:1 2024-04-06T10:04:54.291 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T10:04:54.291 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T10:04:54.291 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-06T10:04:54.291 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:1 2024-04-06T10:04:54.291 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T10:04:54.291 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T10:04:54.291 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:04:54.291 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=1 2024-04-06T10:04:54.291 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-06T10:04:54.291 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T10:04:54.291 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T10:04:54.291 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=1 2024-04-06T10:04:54.292 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=1 2024-04-06T10:04:54.292 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-06T10:04:54.310 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T10:04:54.311 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T10:04:54.311 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T10:04:54.311 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/image_33 2024-04-06T10:04:54.463 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T10:04:54.465 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror image_33 2024-04-06T10:04:54.465 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:04:54.465 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:04:54.465 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_33 2024-04-06T10:04:54.465 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror image_33 '' 2024-04-06T10:04:54.465 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:04:54.465 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:04:54.465 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_33 2024-04-06T10:04:54.465 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T10:04:54.466 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-image_33.status 2024-04-06T10:04:54.466 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-image_33.status 2024-04-06T10:04:54.466 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T10:04:54.468 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-image_33.status 2024-04-06T10:04:54.469 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_33 2024-04-06T10:04:54.469 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_33.status 2024-04-06T10:04:54.514 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 1 2024-04-06T10:04:54.514 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 1 2024-04-06T10:04:54.514 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T10:04:54.514 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=5, tag_tid=2, entry_tid=1], [object_number=4, tag_tid=2, entry_tid=0], [object_number=7, tag_tid=1, entry_tid=6267], [object_number=6, tag_tid=1, entry_tid=6266]]], state=connected] 2024-04-06T10:04:54.515 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=5, tag_tid=1, entry_tid=6269], [object_number=4, tag_tid=1, entry_tid=6268], [object_number=7, tag_tid=1, entry_tid=6267], [object_number=6, tag_tid=1, entry_tid=6266]]], state=connected] 2024-04-06T10:04:54.518 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_33.status 2024-04-06T10:04:54.519 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=5, tag_tid=2, entry_tid=1' 2024-04-06T10:04:54.520 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror image_33 2024-04-06T10:04:54.520 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:04:54.520 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:04:54.521 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_33 2024-04-06T10:04:54.521 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror image_33 '..*' 2024-04-06T10:04:54.521 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:04:54.521 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:04:54.521 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_33 2024-04-06T10:04:54.521 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T10:04:54.521 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-image_33.status 2024-04-06T10:04:54.521 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-image_33.status 2024-04-06T10:04:54.521 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T10:04:54.522 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-image_33.status 2024-04-06T10:04:54.522 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_33 2024-04-06T10:04:54.522 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_33.status 2024-04-06T10:04:54.567 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 1 2024-04-06T10:04:54.567 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 1 2024-04-06T10:04:54.567 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T10:04:54.567 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=5, tag_tid=2, entry_tid=1], [object_number=4, tag_tid=2, entry_tid=0], [object_number=7, tag_tid=1, entry_tid=6267], [object_number=6, tag_tid=1, entry_tid=6266]]], state=connected] 2024-04-06T10:04:54.567 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=5, tag_tid=1, entry_tid=6269], [object_number=4, tag_tid=1, entry_tid=6268], [object_number=7, tag_tid=1, entry_tid=6267], [object_number=6, tag_tid=1, entry_tid=6266]]], state=connected] 2024-04-06T10:04:54.571 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_33.status 2024-04-06T10:04:54.573 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=5, tag_tid=1, entry_tid=6269' 2024-04-06T10:04:54.573 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=5, tag_tid=2, entry_tid=1' -a 'object_number=5, tag_tid=2, entry_tid=1' = 'object_number=5, tag_tid=1, entry_tid=6269' 2024-04-06T10:04:54.573 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=5, tag_tid=1, entry_tid=6269' '!=' 'object_number=5, tag_tid=1, entry_tid=6269' 2024-04-06T10:04:54.573 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T10:04:54.573 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 2 2024-04-06T10:04:56.575 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror image_33 2024-04-06T10:04:56.575 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T10:04:56.575 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:04:56.575 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_33 2024-04-06T10:04:56.575 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T10:04:56.575 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n image_33 ']' 2024-04-06T10:04:56.575 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/image_33' 2024-04-06T10:04:56.575 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_33 2024-04-06T10:04:56.575 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T10:04:56.575 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T10:04:56.575 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T10:04:56.575 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T10:04:56.576 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T10:04:56.576 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T10:04:56.576 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T10:04:56.576 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T10:04:56.576 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T10:04:56.579 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T10:04:56.579 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_33 2024-04-06T10:04:56.579 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T10:04:56.579 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T10:04:56.579 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T10:04:56.580 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T10:04:56.580 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T10:04:56.580 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T10:04:56.580 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T10:04:56.580 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T10:04:56.580 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T10:04:56.580 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T10:04:56.580 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T10:04:56.580 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:04:56.580 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T10:04:56.580 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T10:04:56.580 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T10:04:56.580 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T10:04:56.580 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T10:04:56.580 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T10:04:56.581 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T10:04:56.581 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T10:04:56.581 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T10:04:56.581 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T10:04:56.581 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:04:56.581 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T10:04:56.581 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T10:04:56.581 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T10:04:56.581 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T10:04:56.581 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T10:04:56.581 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T10:04:56.582 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T10:04:56.606 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T10:04:56.607 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T10:04:56.607 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T10:04:56.607 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_33 2024-04-06T10:04:56.714 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:Invalid command: unused arguments: ['mirror/image_33'] 2024-04-06T10:04:56.714 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:rbd mirror flush : flush rbd mirror 2024-04-06T10:04:56.714 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:admin_socket: invalid command 2024-04-06T10:04:56.718 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T10:04:56.718 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/image_33 2024-04-06T10:04:56.718 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:1 2024-04-06T10:04:56.718 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T10:04:56.718 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T10:04:56.718 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-06T10:04:56.718 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:1 2024-04-06T10:04:56.718 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T10:04:56.718 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T10:04:56.718 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T10:04:56.718 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=1 2024-04-06T10:04:56.718 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-06T10:04:56.718 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T10:04:56.719 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:04:56.719 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=1 2024-04-06T10:04:56.719 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=1 2024-04-06T10:04:56.719 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-06T10:04:56.719 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:1 2024-04-06T10:04:56.719 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T10:04:56.719 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T10:04:56.719 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-06T10:04:56.719 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:1 2024-04-06T10:04:56.719 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T10:04:56.719 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T10:04:56.719 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:04:56.719 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=1 2024-04-06T10:04:56.719 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-06T10:04:56.719 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T10:04:56.719 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T10:04:56.719 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=1 2024-04-06T10:04:56.720 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=1 2024-04-06T10:04:56.720 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-06T10:04:56.736 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T10:04:56.737 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T10:04:56.737 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T10:04:56.737 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/image_33 2024-04-06T10:04:56.875 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T10:04:56.876 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror image_33 2024-04-06T10:04:56.877 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:04:56.877 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:04:56.877 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_33 2024-04-06T10:04:56.877 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror image_33 '' 2024-04-06T10:04:56.877 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:04:56.877 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:04:56.877 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_33 2024-04-06T10:04:56.877 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T10:04:56.878 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-image_33.status 2024-04-06T10:04:56.878 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-image_33.status 2024-04-06T10:04:56.878 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T10:04:56.879 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-image_33.status 2024-04-06T10:04:56.879 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_33 2024-04-06T10:04:56.879 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_33.status 2024-04-06T10:04:56.933 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 1 2024-04-06T10:04:56.933 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 1 2024-04-06T10:04:56.933 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T10:04:56.933 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=5, tag_tid=2, entry_tid=1], [object_number=4, tag_tid=2, entry_tid=0], [object_number=7, tag_tid=1, entry_tid=6267], [object_number=6, tag_tid=1, entry_tid=6266]]], state=connected] 2024-04-06T10:04:56.933 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=5, tag_tid=1, entry_tid=6269], [object_number=4, tag_tid=1, entry_tid=6268], [object_number=7, tag_tid=1, entry_tid=6267], [object_number=6, tag_tid=1, entry_tid=6266]]], state=connected] 2024-04-06T10:04:56.936 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_33.status 2024-04-06T10:04:56.937 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=5, tag_tid=2, entry_tid=1' 2024-04-06T10:04:56.939 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror image_33 2024-04-06T10:04:56.939 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:04:56.939 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:04:56.939 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_33 2024-04-06T10:04:56.939 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror image_33 '..*' 2024-04-06T10:04:56.939 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:04:56.939 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:04:56.939 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_33 2024-04-06T10:04:56.939 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T10:04:56.939 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-image_33.status 2024-04-06T10:04:56.939 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-image_33.status 2024-04-06T10:04:56.939 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T10:04:56.940 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-image_33.status 2024-04-06T10:04:56.940 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_33 2024-04-06T10:04:56.940 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_33.status 2024-04-06T10:04:56.982 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 1 2024-04-06T10:04:56.982 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 1 2024-04-06T10:04:56.983 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T10:04:56.983 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=5, tag_tid=2, entry_tid=1], [object_number=4, tag_tid=2, entry_tid=0], [object_number=7, tag_tid=1, entry_tid=6267], [object_number=6, tag_tid=1, entry_tid=6266]]], state=connected] 2024-04-06T10:04:56.983 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=5, tag_tid=1, entry_tid=6269], [object_number=4, tag_tid=1, entry_tid=6268], [object_number=7, tag_tid=1, entry_tid=6267], [object_number=6, tag_tid=1, entry_tid=6266]]], state=connected] 2024-04-06T10:04:56.986 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_33.status 2024-04-06T10:04:56.988 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=5, tag_tid=1, entry_tid=6269' 2024-04-06T10:04:56.988 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=5, tag_tid=2, entry_tid=1' -a 'object_number=5, tag_tid=2, entry_tid=1' = 'object_number=5, tag_tid=1, entry_tid=6269' 2024-04-06T10:04:56.988 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=5, tag_tid=1, entry_tid=6269' '!=' 'object_number=5, tag_tid=1, entry_tid=6269' 2024-04-06T10:04:56.988 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T10:04:56.988 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 2 2024-04-06T10:04:58.990 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror image_33 2024-04-06T10:04:58.990 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T10:04:58.990 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:04:58.990 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_33 2024-04-06T10:04:58.990 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T10:04:58.990 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n image_33 ']' 2024-04-06T10:04:58.990 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/image_33' 2024-04-06T10:04:58.990 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_33 2024-04-06T10:04:58.990 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T10:04:58.990 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T10:04:58.991 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T10:04:58.991 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T10:04:58.991 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T10:04:58.991 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T10:04:58.991 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T10:04:58.991 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T10:04:58.991 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T10:04:58.995 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T10:04:58.995 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_33 2024-04-06T10:04:58.995 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T10:04:58.995 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T10:04:58.995 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T10:04:58.995 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T10:04:58.995 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T10:04:58.995 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T10:04:58.995 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T10:04:58.995 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T10:04:58.995 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T10:04:58.995 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T10:04:58.995 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T10:04:58.995 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:04:58.996 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T10:04:58.996 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T10:04:58.996 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T10:04:58.996 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T10:04:58.996 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T10:04:58.996 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T10:04:58.996 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T10:04:58.996 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T10:04:58.996 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T10:04:58.996 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T10:04:58.996 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:04:58.996 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T10:04:58.996 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T10:04:58.996 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T10:04:58.996 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T10:04:58.996 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T10:04:58.997 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T10:04:58.997 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T10:04:59.019 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T10:04:59.020 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T10:04:59.020 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T10:04:59.020 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_33 2024-04-06T10:04:59.126 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:Invalid command: unused arguments: ['mirror/image_33'] 2024-04-06T10:04:59.126 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:rbd mirror flush : flush rbd mirror 2024-04-06T10:04:59.126 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:admin_socket: invalid command 2024-04-06T10:04:59.129 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T10:04:59.129 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/image_33 2024-04-06T10:04:59.129 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:1 2024-04-06T10:04:59.129 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T10:04:59.129 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T10:04:59.129 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-06T10:04:59.129 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:1 2024-04-06T10:04:59.129 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T10:04:59.129 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T10:04:59.129 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T10:04:59.129 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=1 2024-04-06T10:04:59.129 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-06T10:04:59.129 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T10:04:59.129 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:04:59.129 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=1 2024-04-06T10:04:59.130 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=1 2024-04-06T10:04:59.130 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-06T10:04:59.131 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:1 2024-04-06T10:04:59.131 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T10:04:59.131 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T10:04:59.131 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-06T10:04:59.131 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:1 2024-04-06T10:04:59.131 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T10:04:59.131 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T10:04:59.131 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:04:59.131 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=1 2024-04-06T10:04:59.131 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-06T10:04:59.131 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T10:04:59.131 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T10:04:59.132 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=1 2024-04-06T10:04:59.132 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=1 2024-04-06T10:04:59.132 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-06T10:04:59.149 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T10:04:59.149 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T10:04:59.149 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T10:04:59.149 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/image_33 2024-04-06T10:04:59.293 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T10:04:59.294 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror image_33 2024-04-06T10:04:59.294 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:04:59.294 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:04:59.294 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_33 2024-04-06T10:04:59.295 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror image_33 '' 2024-04-06T10:04:59.295 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:04:59.295 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:04:59.295 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_33 2024-04-06T10:04:59.295 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T10:04:59.295 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-image_33.status 2024-04-06T10:04:59.295 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-image_33.status 2024-04-06T10:04:59.296 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T10:04:59.297 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-image_33.status 2024-04-06T10:04:59.297 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_33 2024-04-06T10:04:59.297 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_33.status 2024-04-06T10:04:59.347 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 1 2024-04-06T10:04:59.347 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 1 2024-04-06T10:04:59.347 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T10:04:59.347 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=5, tag_tid=2, entry_tid=1], [object_number=4, tag_tid=2, entry_tid=0], [object_number=7, tag_tid=1, entry_tid=6267], [object_number=6, tag_tid=1, entry_tid=6266]]], state=connected] 2024-04-06T10:04:59.347 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=5, tag_tid=2, entry_tid=1], [object_number=4, tag_tid=2, entry_tid=0], [object_number=7, tag_tid=1, entry_tid=6267], [object_number=6, tag_tid=1, entry_tid=6266]]], state=connected] 2024-04-06T10:04:59.351 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_33.status 2024-04-06T10:04:59.352 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=5, tag_tid=2, entry_tid=1' 2024-04-06T10:04:59.353 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror image_33 2024-04-06T10:04:59.353 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:04:59.354 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:04:59.354 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_33 2024-04-06T10:04:59.354 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror image_33 '..*' 2024-04-06T10:04:59.354 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:04:59.354 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:04:59.354 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_33 2024-04-06T10:04:59.354 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T10:04:59.354 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-image_33.status 2024-04-06T10:04:59.354 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-image_33.status 2024-04-06T10:04:59.354 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T10:04:59.355 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-image_33.status 2024-04-06T10:04:59.355 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_33 2024-04-06T10:04:59.355 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_33.status 2024-04-06T10:04:59.400 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 1 2024-04-06T10:04:59.400 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 1 2024-04-06T10:04:59.401 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T10:04:59.401 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=5, tag_tid=2, entry_tid=1], [object_number=4, tag_tid=2, entry_tid=0], [object_number=7, tag_tid=1, entry_tid=6267], [object_number=6, tag_tid=1, entry_tid=6266]]], state=connected] 2024-04-06T10:04:59.401 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=5, tag_tid=2, entry_tid=1], [object_number=4, tag_tid=2, entry_tid=0], [object_number=7, tag_tid=1, entry_tid=6267], [object_number=6, tag_tid=1, entry_tid=6266]]], state=connected] 2024-04-06T10:04:59.404 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_33.status 2024-04-06T10:04:59.406 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=5, tag_tid=2, entry_tid=1' 2024-04-06T10:04:59.406 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=5, tag_tid=2, entry_tid=1' -a 'object_number=5, tag_tid=2, entry_tid=1' = 'object_number=5, tag_tid=2, entry_tid=1' 2024-04-06T10:04:59.406 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T10:04:59.406 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_snap_present cluster1 mirror image_33 snap 2024-04-06T10:04:59.406 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T10:04:59.407 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:04:59.407 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_33 2024-04-06T10:04:59.407 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local snap_name=snap 2024-04-06T10:04:59.407 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local s 2024-04-06T10:04:59.407 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-04-06T10:04:59.407 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 1 2024-04-06T10:05:00.407 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 info mirror/image_33@snap 2024-04-06T10:05:00.474 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:rbd image 'image_33': 2024-04-06T10:05:00.475 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: size 128 MiB in 32 objects 2024-04-06T10:05:00.475 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: order 22 (4 MiB objects) 2024-04-06T10:05:00.475 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: snapshot_count: 1 2024-04-06T10:05:00.475 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: id: 10ac1e6bfd3a 2024-04-06T10:05:00.475 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: block_name_prefix: rbd_data.10ac1e6bfd3a 2024-04-06T10:05:00.475 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: format: 2 2024-04-06T10:05:00.475 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: features: layering, exclusive-lock, journaling 2024-04-06T10:05:00.475 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: op_features: 2024-04-06T10:05:00.475 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: flags: 2024-04-06T10:05:00.475 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: create_timestamp: Sat Apr 6 09:55:41 2024 2024-04-06T10:05:00.475 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: access_timestamp: Sat Apr 6 09:55:41 2024 2024-04-06T10:05:00.475 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: modify_timestamp: Sat Apr 6 09:56:41 2024 2024-04-06T10:05:00.475 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: protected: False 2024-04-06T10:05:00.475 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: journal: 10ac1e6bfd3a 2024-04-06T10:05:00.476 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: mirroring state: enabled 2024-04-06T10:05:00.476 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: mirroring mode: journal 2024-04-06T10:05:00.476 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: mirroring global id: bb335656-a4de-409b-944a-ecc11253243c 2024-04-06T10:05:00.476 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: mirroring primary: false 2024-04-06T10:05:00.479 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T10:05:00.479 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ compare_image_snaps mirror image_33 snap 2024-04-06T10:05:00.480 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:05:00.480 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_33 2024-04-06T10:05:00.480 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local snap_name=snap 2024-04-06T10:05:00.480 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local ret=0 2024-04-06T10:05:00.480 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local rmt_export=/tmp/tmp.BhW6xKmROU/cluster2-mirror-image_33.export 2024-04-06T10:05:00.480 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loc_export=/tmp/tmp.BhW6xKmROU/cluster1-mirror-image_33.export 2024-04-06T10:05:00.480 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rm -f /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_33.export /tmp/tmp.BhW6xKmROU/cluster1-mirror-image_33.export 2024-04-06T10:05:00.480 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster2 -p mirror export image_33@snap /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_33.export 2024-04-06T10:05:00.654 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-04-06T10:05:00.662 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 -p mirror export image_33@snap /tmp/tmp.BhW6xKmROU/cluster1-mirror-image_33.export 2024-04-06T10:05:01.020 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-04-06T10:05:01.031 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmp /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_33.export /tmp/tmp.BhW6xKmROU/cluster1-mirror-image_33.export 2024-04-06T10:05:01.112 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rm -f /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_33.export /tmp/tmp.BhW6xKmROU/cluster1-mirror-image_33.export 2024-04-06T10:05:01.153 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T10:05:01.153 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-06T10:05:01.153 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ image=image_34 2024-04-06T10:05:01.153 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ create_snap cluster2 mirror image_34 snap 2024-04-06T10:05:01.153 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-04-06T10:05:01.153 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:05:01.153 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_34 2024-04-06T10:05:01.153 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local snap_name=snap 2024-04-06T10:05:01.153 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster2 -p mirror snap create image_34@snap --debug-rbd=20 --debug-journaler=20 2024-04-06T10:05:03.913 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_image_replay_started cluster1 mirror image_34 2024-04-06T10:05:03.914 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T10:05:03.914 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:05:03.914 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_34 2024-04-06T10:05:03.914 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_image_replay_state cluster1 mirror image_34 started 2024-04-06T10:05:03.914 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T10:05:03.914 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:05:03.914 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_34 2024-04-06T10:05:03.914 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state=started 2024-04-06T10:05:03.914 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local s 2024-04-06T10:05:03.914 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T10:05:03.914 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 1 2024-04-06T10:05:04.915 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test_image_replay_state cluster1 mirror image_34 started 2024-04-06T10:05:04.916 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T10:05:04.916 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:05:04.916 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_34 2024-04-06T10:05:04.916 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local test_state=started 2024-04-06T10:05:04.916 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local status_result 2024-04-06T10:05:04.916 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local current_state=stopped 2024-04-06T10:05:04.917 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_34 2024-04-06T10:05:04.917 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -i state 2024-04-06T10:05:04.917 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_instance=cluster1 2024-04-06T10:05:04.917 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ shift 2024-04-06T10:05:04.917 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T10:05:04.917 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance=cluster1 2024-04-06T10:05:04.917 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local loop_instance 2024-04-06T10:05:04.917 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T10:05:04.917 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sleep 0 2024-04-06T10:05:04.918 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-04-06T10:05:04.918 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ seq 0 3 2024-04-06T10:05:04.920 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T10:05:04.920 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_34 2024-04-06T10:05:04.921 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1:0 2024-04-06T10:05:04.921 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ shift 2024-04-06T10:05:04.921 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T10:05:04.921 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-04-06T10:05:04.921 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T10:05:04.921 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=cluster 2024-04-06T10:05:04.921 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T10:05:04.921 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:05:04.921 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T10:05:04.921 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T10:05:04.921 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval cluster=cluster1 2024-04-06T10:05:04.921 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ cluster=cluster1 2024-04-06T10:05:04.921 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T10:05:04.921 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T10:05:04.921 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-04-06T10:05:04.922 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local local_cluster=cluster1:0 2024-04-06T10:05:04.923 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local cluster=cluster1 2024-04-06T10:05:04.923 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local instance 2024-04-06T10:05:04.923 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T10:05:04.923 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local val=cluster1:0 2024-04-06T10:05:04.923 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local cluster_var_name=local_cluster 2024-04-06T10:05:04.923 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local instance_var_name=instance 2024-04-06T10:05:04.923 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ cluster=cluster1 2024-04-06T10:05:04.923 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T10:05:04.923 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ '[' 0 = cluster1:0 ']' 2024-04-06T10:05:04.923 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ eval local_cluster=cluster1 2024-04-06T10:05:04.924 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++++ local_cluster=cluster1 2024-04-06T10:05:04.924 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ eval instance=0 2024-04-06T10:05:04.924 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++++ instance=0 2024-04-06T10:05:04.924 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T10:05:04.943 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T10:05:04.943 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T10:05:04.943 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T10:05:04.943 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_34 2024-04-06T10:05:05.048 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:Invalid command: unused arguments: ['mirror/image_34'] 2024-04-06T10:05:05.048 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:rbd mirror status : get status for rbd mirror 2024-04-06T10:05:05.048 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:admin_socket: invalid command 2024-04-06T10:05:05.050 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T10:05:05.050 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_34 2024-04-06T10:05:05.050 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1:1 2024-04-06T10:05:05.051 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ shift 2024-04-06T10:05:05.051 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T10:05:05.051 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-04-06T10:05:05.051 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:1 2024-04-06T10:05:05.051 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=cluster 2024-04-06T10:05:05.051 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T10:05:05.051 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:05:05.051 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=1 2024-04-06T10:05:05.051 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-06T10:05:05.051 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval cluster=cluster1 2024-04-06T10:05:05.051 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ cluster=cluster1 2024-04-06T10:05:05.051 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=1 2024-04-06T10:05:05.051 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=1 2024-04-06T10:05:05.053 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-04-06T10:05:05.053 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local local_cluster=cluster1:1 2024-04-06T10:05:05.053 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local cluster=cluster1 2024-04-06T10:05:05.053 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local instance 2024-04-06T10:05:05.053 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-06T10:05:05.053 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local val=cluster1:1 2024-04-06T10:05:05.053 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local cluster_var_name=local_cluster 2024-04-06T10:05:05.053 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local instance_var_name=instance 2024-04-06T10:05:05.053 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ cluster=cluster1 2024-04-06T10:05:05.054 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=1 2024-04-06T10:05:05.054 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ '[' 1 = cluster1:1 ']' 2024-04-06T10:05:05.054 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ eval local_cluster=cluster1 2024-04-06T10:05:05.054 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++++ local_cluster=cluster1 2024-04-06T10:05:05.054 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ eval instance=1 2024-04-06T10:05:05.054 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++++ instance=1 2024-04-06T10:05:05.054 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-06T10:05:05.075 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T10:05:05.076 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T10:05:05.076 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T10:05:05.076 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_34 2024-04-06T10:05:05.228 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ return 0 2024-04-06T10:05:05.228 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ status_result=' "state": "Replaying"' 2024-04-06T10:05:05.228 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ echo ' "state": "Replaying"' 2024-04-06T10:05:05.228 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep -i Replaying 2024-04-06T10:05:05.229 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: "state": "Replaying" 2024-04-06T10:05:05.231 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ current_state=started 2024-04-06T10:05:05.231 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test started = started 2024-04-06T10:05:05.231 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T10:05:05.231 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_34 2024-04-06T10:05:05.231 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local local_cluster=cluster1 2024-04-06T10:05:05.231 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-04-06T10:05:05.231 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:05:05.231 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_34 2024-04-06T10:05:05.232 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' journal = journal ']' 2024-04-06T10:05:05.232 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_journal_replay_complete cluster1 cluster2 mirror image_34 2024-04-06T10:05:05.233 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local local_cluster=cluster1 2024-04-06T10:05:05.233 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-04-06T10:05:05.233 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:05:05.233 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_34 2024-04-06T10:05:05.233 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local s master_pos mirror_pos last_mirror_pos 2024-04-06T10:05:05.234 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local master_tag master_entry mirror_tag mirror_entry 2024-04-06T10:05:05.234 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T10:05:05.234 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T10:05:05.234 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T10:05:05.432 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror image_34 2024-04-06T10:05:05.432 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T10:05:05.432 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:05:05.432 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_34 2024-04-06T10:05:05.432 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T10:05:05.433 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n image_34 ']' 2024-04-06T10:05:05.433 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/image_34' 2024-04-06T10:05:05.433 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_34 2024-04-06T10:05:05.433 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T10:05:05.433 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T10:05:05.433 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T10:05:05.433 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T10:05:05.434 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T10:05:05.434 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T10:05:05.434 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T10:05:05.434 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T10:05:05.438 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T10:05:05.439 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T10:05:05.439 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_34 2024-04-06T10:05:05.439 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T10:05:05.439 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T10:05:05.439 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T10:05:05.439 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T10:05:05.439 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T10:05:05.439 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T10:05:05.439 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T10:05:05.440 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T10:05:05.440 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T10:05:05.440 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T10:05:05.440 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T10:05:05.440 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:05:05.440 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T10:05:05.440 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T10:05:05.440 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T10:05:05.440 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T10:05:05.440 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T10:05:05.440 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T10:05:05.440 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T10:05:05.441 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T10:05:05.441 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T10:05:05.441 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T10:05:05.441 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:05:05.441 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T10:05:05.441 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T10:05:05.441 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T10:05:05.441 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T10:05:05.442 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T10:05:05.442 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T10:05:05.442 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T10:05:05.461 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T10:05:05.461 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T10:05:05.461 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T10:05:05.461 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_34 2024-04-06T10:05:05.565 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:Invalid command: unused arguments: ['mirror/image_34'] 2024-04-06T10:05:05.565 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:rbd mirror flush : flush rbd mirror 2024-04-06T10:05:05.565 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:admin_socket: invalid command 2024-04-06T10:05:05.567 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T10:05:05.567 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/image_34 2024-04-06T10:05:05.567 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:1 2024-04-06T10:05:05.567 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T10:05:05.567 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T10:05:05.568 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-06T10:05:05.568 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:1 2024-04-06T10:05:05.568 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T10:05:05.568 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T10:05:05.568 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T10:05:05.568 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=1 2024-04-06T10:05:05.568 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-06T10:05:05.568 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T10:05:05.568 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:05:05.568 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=1 2024-04-06T10:05:05.569 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=1 2024-04-06T10:05:05.570 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-06T10:05:05.571 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:1 2024-04-06T10:05:05.571 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T10:05:05.571 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T10:05:05.571 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-06T10:05:05.571 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:1 2024-04-06T10:05:05.571 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T10:05:05.571 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T10:05:05.571 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:05:05.571 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=1 2024-04-06T10:05:05.572 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-06T10:05:05.572 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T10:05:05.572 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T10:05:05.572 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=1 2024-04-06T10:05:05.572 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=1 2024-04-06T10:05:05.572 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-06T10:05:05.594 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T10:05:05.594 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T10:05:05.594 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T10:05:05.594 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/image_34 2024-04-06T10:05:05.731 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T10:05:05.732 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror image_34 2024-04-06T10:05:05.732 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:05:05.733 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:05:05.733 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_34 2024-04-06T10:05:05.733 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror image_34 '' 2024-04-06T10:05:05.733 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:05:05.733 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:05:05.733 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_34 2024-04-06T10:05:05.733 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T10:05:05.734 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-image_34.status 2024-04-06T10:05:05.734 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-image_34.status 2024-04-06T10:05:05.734 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T10:05:05.736 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-image_34.status 2024-04-06T10:05:05.737 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_34 2024-04-06T10:05:05.737 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_34.status 2024-04-06T10:05:05.782 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 1 2024-04-06T10:05:05.783 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 1 2024-04-06T10:05:05.783 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T10:05:05.783 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=5, tag_tid=2, entry_tid=1], [object_number=4, tag_tid=2, entry_tid=0], [object_number=7, tag_tid=1, entry_tid=8019], [object_number=6, tag_tid=1, entry_tid=8018]]], state=connected] 2024-04-06T10:05:05.783 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=5, tag_tid=1, entry_tid=8021], [object_number=4, tag_tid=1, entry_tid=8020], [object_number=7, tag_tid=1, entry_tid=8019], [object_number=6, tag_tid=1, entry_tid=8018]]], state=connected] 2024-04-06T10:05:05.785 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_34.status 2024-04-06T10:05:05.787 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=5, tag_tid=2, entry_tid=1' 2024-04-06T10:05:05.788 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror image_34 2024-04-06T10:05:05.788 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:05:05.788 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:05:05.788 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_34 2024-04-06T10:05:05.788 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror image_34 '..*' 2024-04-06T10:05:05.788 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:05:05.788 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:05:05.788 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_34 2024-04-06T10:05:05.788 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T10:05:05.789 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-image_34.status 2024-04-06T10:05:05.789 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-image_34.status 2024-04-06T10:05:05.789 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T10:05:05.789 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-image_34.status 2024-04-06T10:05:05.790 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_34 2024-04-06T10:05:05.790 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_34.status 2024-04-06T10:05:05.836 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 1 2024-04-06T10:05:05.836 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 1 2024-04-06T10:05:05.837 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T10:05:05.837 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=5, tag_tid=2, entry_tid=1], [object_number=4, tag_tid=2, entry_tid=0], [object_number=7, tag_tid=1, entry_tid=8019], [object_number=6, tag_tid=1, entry_tid=8018]]], state=connected] 2024-04-06T10:05:05.837 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=5, tag_tid=1, entry_tid=8021], [object_number=4, tag_tid=1, entry_tid=8020], [object_number=7, tag_tid=1, entry_tid=8019], [object_number=6, tag_tid=1, entry_tid=8018]]], state=connected] 2024-04-06T10:05:05.840 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_34.status 2024-04-06T10:05:05.842 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=5, tag_tid=1, entry_tid=8021' 2024-04-06T10:05:05.843 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=5, tag_tid=2, entry_tid=1' -a 'object_number=5, tag_tid=2, entry_tid=1' = 'object_number=5, tag_tid=1, entry_tid=8021' 2024-04-06T10:05:05.843 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=5, tag_tid=1, entry_tid=8021' '!=' '' 2024-04-06T10:05:05.843 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T10:05:05.843 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=5, tag_tid=1, entry_tid=8021' = '' 2024-04-06T10:05:05.843 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=5, tag_tid=1, entry_tid=8021' 2024-04-06T10:05:05.843 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=5, tag_tid=2, entry_tid=1' 2024-04-06T10:05:05.843 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T10:05:05.843 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T10:05:05.844 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=2 2024-04-06T10:05:05.844 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=5, tag_tid=1, entry_tid=8021' 2024-04-06T10:05:05.844 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T10:05:05.845 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T10:05:05.845 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=1 2024-04-06T10:05:05.846 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=5, tag_tid=2, entry_tid=1' 2024-04-06T10:05:05.846 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T10:05:05.846 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T10:05:05.847 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T10:05:05.848 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=5, tag_tid=1, entry_tid=8021' 2024-04-06T10:05:05.848 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T10:05:05.848 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T10:05:05.849 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=8021 2024-04-06T10:05:05.849 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 2 = 1 -a 1 -le 8021 2024-04-06T10:05:05.850 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T10:05:05.850 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T10:05:05.850 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T10:05:06.051 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror image_34 2024-04-06T10:05:06.052 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T10:05:06.052 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:05:06.052 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_34 2024-04-06T10:05:06.052 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T10:05:06.052 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n image_34 ']' 2024-04-06T10:05:06.052 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/image_34' 2024-04-06T10:05:06.052 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_34 2024-04-06T10:05:06.052 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T10:05:06.052 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T10:05:06.052 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T10:05:06.052 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T10:05:06.052 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T10:05:06.053 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T10:05:06.053 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T10:05:06.053 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T10:05:06.053 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T10:05:06.055 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T10:05:06.055 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_34 2024-04-06T10:05:06.056 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T10:05:06.056 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T10:05:06.056 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T10:05:06.056 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T10:05:06.056 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T10:05:06.056 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T10:05:06.056 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T10:05:06.056 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T10:05:06.056 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T10:05:06.056 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T10:05:06.056 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T10:05:06.056 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:05:06.057 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T10:05:06.057 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T10:05:06.057 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T10:05:06.057 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T10:05:06.057 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T10:05:06.057 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T10:05:06.057 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T10:05:06.057 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T10:05:06.057 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T10:05:06.057 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T10:05:06.057 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:05:06.057 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T10:05:06.058 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T10:05:06.058 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T10:05:06.058 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T10:05:06.058 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T10:05:06.058 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T10:05:06.058 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T10:05:06.091 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T10:05:06.092 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T10:05:06.092 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T10:05:06.092 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_34 2024-04-06T10:05:06.192 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:Invalid command: unused arguments: ['mirror/image_34'] 2024-04-06T10:05:06.192 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:rbd mirror flush : flush rbd mirror 2024-04-06T10:05:06.192 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:admin_socket: invalid command 2024-04-06T10:05:06.196 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T10:05:06.196 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/image_34 2024-04-06T10:05:06.196 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:1 2024-04-06T10:05:06.197 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T10:05:06.197 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T10:05:06.197 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-06T10:05:06.197 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:1 2024-04-06T10:05:06.197 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T10:05:06.197 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T10:05:06.197 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T10:05:06.197 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=1 2024-04-06T10:05:06.197 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-06T10:05:06.197 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T10:05:06.197 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:05:06.197 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=1 2024-04-06T10:05:06.198 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=1 2024-04-06T10:05:06.198 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-06T10:05:06.198 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:1 2024-04-06T10:05:06.198 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T10:05:06.198 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T10:05:06.198 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-06T10:05:06.198 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:1 2024-04-06T10:05:06.198 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T10:05:06.198 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T10:05:06.198 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:05:06.198 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=1 2024-04-06T10:05:06.199 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-06T10:05:06.199 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T10:05:06.199 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T10:05:06.199 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=1 2024-04-06T10:05:06.199 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=1 2024-04-06T10:05:06.199 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-06T10:05:06.219 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T10:05:06.220 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T10:05:06.220 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T10:05:06.220 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/image_34 2024-04-06T10:05:06.364 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T10:05:06.365 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror image_34 2024-04-06T10:05:06.365 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:05:06.365 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:05:06.365 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_34 2024-04-06T10:05:06.366 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror image_34 '' 2024-04-06T10:05:06.366 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:05:06.366 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:05:06.366 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_34 2024-04-06T10:05:06.366 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T10:05:06.366 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-image_34.status 2024-04-06T10:05:06.367 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-image_34.status 2024-04-06T10:05:06.367 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T10:05:06.368 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-image_34.status 2024-04-06T10:05:06.368 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_34 2024-04-06T10:05:06.368 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_34.status 2024-04-06T10:05:06.419 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 1 2024-04-06T10:05:06.420 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 1 2024-04-06T10:05:06.420 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T10:05:06.420 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=5, tag_tid=2, entry_tid=1], [object_number=4, tag_tid=2, entry_tid=0], [object_number=7, tag_tid=1, entry_tid=8019], [object_number=6, tag_tid=1, entry_tid=8018]]], state=connected] 2024-04-06T10:05:06.420 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=5, tag_tid=1, entry_tid=8021], [object_number=4, tag_tid=1, entry_tid=8020], [object_number=7, tag_tid=1, entry_tid=8019], [object_number=6, tag_tid=1, entry_tid=8018]]], state=connected] 2024-04-06T10:05:06.423 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_34.status 2024-04-06T10:05:06.424 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=5, tag_tid=2, entry_tid=1' 2024-04-06T10:05:06.426 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror image_34 2024-04-06T10:05:06.426 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:05:06.426 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:05:06.426 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_34 2024-04-06T10:05:06.426 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror image_34 '..*' 2024-04-06T10:05:06.426 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:05:06.426 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:05:06.426 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_34 2024-04-06T10:05:06.427 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T10:05:06.427 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-image_34.status 2024-04-06T10:05:06.427 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-image_34.status 2024-04-06T10:05:06.427 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T10:05:06.427 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-image_34.status 2024-04-06T10:05:06.428 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_34 2024-04-06T10:05:06.428 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_34.status 2024-04-06T10:05:06.472 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 1 2024-04-06T10:05:06.473 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 1 2024-04-06T10:05:06.473 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T10:05:06.473 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=5, tag_tid=2, entry_tid=1], [object_number=4, tag_tid=2, entry_tid=0], [object_number=7, tag_tid=1, entry_tid=8019], [object_number=6, tag_tid=1, entry_tid=8018]]], state=connected] 2024-04-06T10:05:06.473 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=5, tag_tid=1, entry_tid=8021], [object_number=4, tag_tid=1, entry_tid=8020], [object_number=7, tag_tid=1, entry_tid=8019], [object_number=6, tag_tid=1, entry_tid=8018]]], state=connected] 2024-04-06T10:05:06.476 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_34.status 2024-04-06T10:05:06.478 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=5, tag_tid=1, entry_tid=8021' 2024-04-06T10:05:06.478 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=5, tag_tid=2, entry_tid=1' -a 'object_number=5, tag_tid=2, entry_tid=1' = 'object_number=5, tag_tid=1, entry_tid=8021' 2024-04-06T10:05:06.478 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=5, tag_tid=1, entry_tid=8021' '!=' 'object_number=5, tag_tid=1, entry_tid=8021' 2024-04-06T10:05:06.478 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T10:05:06.478 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.4 2024-04-06T10:05:06.880 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror image_34 2024-04-06T10:05:06.880 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T10:05:06.880 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:05:06.880 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_34 2024-04-06T10:05:06.880 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T10:05:06.880 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n image_34 ']' 2024-04-06T10:05:06.880 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/image_34' 2024-04-06T10:05:06.881 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_34 2024-04-06T10:05:06.881 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T10:05:06.881 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T10:05:06.881 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T10:05:06.881 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T10:05:06.881 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T10:05:06.881 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T10:05:06.881 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T10:05:06.883 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T10:05:06.883 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T10:05:06.884 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T10:05:06.884 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_34 2024-04-06T10:05:06.884 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T10:05:06.884 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T10:05:06.884 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T10:05:06.884 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T10:05:06.884 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T10:05:06.884 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T10:05:06.884 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T10:05:06.884 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T10:05:06.884 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T10:05:06.884 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T10:05:06.884 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T10:05:06.884 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:05:06.884 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T10:05:06.886 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T10:05:06.886 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T10:05:06.886 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T10:05:06.886 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T10:05:06.886 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T10:05:06.887 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T10:05:06.887 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T10:05:06.887 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T10:05:06.887 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T10:05:06.887 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:05:06.887 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T10:05:06.887 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T10:05:06.887 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T10:05:06.887 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T10:05:06.887 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T10:05:06.887 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T10:05:06.887 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T10:05:06.910 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T10:05:06.911 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T10:05:06.911 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T10:05:06.911 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_34 2024-04-06T10:05:07.016 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:Invalid command: unused arguments: ['mirror/image_34'] 2024-04-06T10:05:07.016 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:rbd mirror flush : flush rbd mirror 2024-04-06T10:05:07.017 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:admin_socket: invalid command 2024-04-06T10:05:07.019 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T10:05:07.019 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/image_34 2024-04-06T10:05:07.019 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:1 2024-04-06T10:05:07.019 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T10:05:07.019 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T10:05:07.019 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-06T10:05:07.019 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:1 2024-04-06T10:05:07.019 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T10:05:07.019 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T10:05:07.019 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T10:05:07.020 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=1 2024-04-06T10:05:07.020 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-06T10:05:07.020 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T10:05:07.020 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:05:07.020 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=1 2024-04-06T10:05:07.020 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=1 2024-04-06T10:05:07.021 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-06T10:05:07.021 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:1 2024-04-06T10:05:07.021 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T10:05:07.021 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T10:05:07.021 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-06T10:05:07.021 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:1 2024-04-06T10:05:07.022 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T10:05:07.022 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T10:05:07.022 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:05:07.022 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=1 2024-04-06T10:05:07.022 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-06T10:05:07.022 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T10:05:07.022 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T10:05:07.022 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=1 2024-04-06T10:05:07.022 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=1 2024-04-06T10:05:07.022 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-06T10:05:07.044 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T10:05:07.044 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T10:05:07.044 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T10:05:07.045 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/image_34 2024-04-06T10:05:07.181 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T10:05:07.183 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror image_34 2024-04-06T10:05:07.183 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:05:07.183 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:05:07.183 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_34 2024-04-06T10:05:07.183 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror image_34 '' 2024-04-06T10:05:07.183 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:05:07.183 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:05:07.183 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_34 2024-04-06T10:05:07.183 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T10:05:07.183 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-image_34.status 2024-04-06T10:05:07.184 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-image_34.status 2024-04-06T10:05:07.184 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T10:05:07.186 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-image_34.status 2024-04-06T10:05:07.186 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_34 2024-04-06T10:05:07.186 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_34.status 2024-04-06T10:05:07.238 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 1 2024-04-06T10:05:07.238 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 1 2024-04-06T10:05:07.239 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T10:05:07.239 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=5, tag_tid=2, entry_tid=1], [object_number=4, tag_tid=2, entry_tid=0], [object_number=7, tag_tid=1, entry_tid=8019], [object_number=6, tag_tid=1, entry_tid=8018]]], state=connected] 2024-04-06T10:05:07.239 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=5, tag_tid=1, entry_tid=8021], [object_number=4, tag_tid=1, entry_tid=8020], [object_number=7, tag_tid=1, entry_tid=8019], [object_number=6, tag_tid=1, entry_tid=8018]]], state=connected] 2024-04-06T10:05:07.242 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_34.status 2024-04-06T10:05:07.243 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=5, tag_tid=2, entry_tid=1' 2024-04-06T10:05:07.244 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror image_34 2024-04-06T10:05:07.244 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:05:07.244 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:05:07.244 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_34 2024-04-06T10:05:07.244 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror image_34 '..*' 2024-04-06T10:05:07.245 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:05:07.245 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:05:07.245 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_34 2024-04-06T10:05:07.245 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T10:05:07.245 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-image_34.status 2024-04-06T10:05:07.245 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-image_34.status 2024-04-06T10:05:07.245 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T10:05:07.246 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-image_34.status 2024-04-06T10:05:07.246 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_34 2024-04-06T10:05:07.246 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_34.status 2024-04-06T10:05:07.291 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 1 2024-04-06T10:05:07.291 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 1 2024-04-06T10:05:07.291 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T10:05:07.291 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=5, tag_tid=2, entry_tid=1], [object_number=4, tag_tid=2, entry_tid=0], [object_number=7, tag_tid=1, entry_tid=8019], [object_number=6, tag_tid=1, entry_tid=8018]]], state=connected] 2024-04-06T10:05:07.291 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=5, tag_tid=1, entry_tid=8021], [object_number=4, tag_tid=1, entry_tid=8020], [object_number=7, tag_tid=1, entry_tid=8019], [object_number=6, tag_tid=1, entry_tid=8018]]], state=connected] 2024-04-06T10:05:07.295 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_34.status 2024-04-06T10:05:07.296 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=5, tag_tid=1, entry_tid=8021' 2024-04-06T10:05:07.296 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=5, tag_tid=2, entry_tid=1' -a 'object_number=5, tag_tid=2, entry_tid=1' = 'object_number=5, tag_tid=1, entry_tid=8021' 2024-04-06T10:05:07.296 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=5, tag_tid=1, entry_tid=8021' '!=' 'object_number=5, tag_tid=1, entry_tid=8021' 2024-04-06T10:05:07.296 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T10:05:07.296 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.8 2024-04-06T10:05:08.100 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror image_34 2024-04-06T10:05:08.100 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T10:05:08.100 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:05:08.100 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_34 2024-04-06T10:05:08.100 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T10:05:08.100 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n image_34 ']' 2024-04-06T10:05:08.100 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/image_34' 2024-04-06T10:05:08.100 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_34 2024-04-06T10:05:08.100 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T10:05:08.101 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T10:05:08.101 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T10:05:08.101 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T10:05:08.101 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T10:05:08.101 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T10:05:08.101 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T10:05:08.101 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T10:05:08.104 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T10:05:08.104 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T10:05:08.104 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_34 2024-04-06T10:05:08.104 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T10:05:08.104 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T10:05:08.104 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T10:05:08.104 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T10:05:08.104 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T10:05:08.104 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T10:05:08.105 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T10:05:08.105 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T10:05:08.105 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T10:05:08.105 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T10:05:08.105 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T10:05:08.105 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:05:08.105 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T10:05:08.105 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T10:05:08.105 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T10:05:08.105 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T10:05:08.105 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T10:05:08.105 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T10:05:08.105 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T10:05:08.105 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T10:05:08.106 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T10:05:08.106 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T10:05:08.106 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:05:08.106 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T10:05:08.106 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T10:05:08.106 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T10:05:08.106 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T10:05:08.106 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T10:05:08.106 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T10:05:08.106 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T10:05:08.130 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T10:05:08.131 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T10:05:08.131 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T10:05:08.131 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_34 2024-04-06T10:05:08.236 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:Invalid command: unused arguments: ['mirror/image_34'] 2024-04-06T10:05:08.236 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:rbd mirror flush : flush rbd mirror 2024-04-06T10:05:08.236 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:admin_socket: invalid command 2024-04-06T10:05:08.239 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T10:05:08.239 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/image_34 2024-04-06T10:05:08.240 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:1 2024-04-06T10:05:08.240 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T10:05:08.240 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T10:05:08.240 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-06T10:05:08.240 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:1 2024-04-06T10:05:08.240 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T10:05:08.241 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T10:05:08.241 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T10:05:08.241 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=1 2024-04-06T10:05:08.241 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-06T10:05:08.241 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T10:05:08.241 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:05:08.241 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=1 2024-04-06T10:05:08.242 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=1 2024-04-06T10:05:08.242 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-06T10:05:08.242 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:1 2024-04-06T10:05:08.242 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T10:05:08.242 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T10:05:08.242 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-06T10:05:08.242 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:1 2024-04-06T10:05:08.242 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T10:05:08.242 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T10:05:08.242 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:05:08.243 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=1 2024-04-06T10:05:08.243 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-06T10:05:08.243 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T10:05:08.243 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T10:05:08.243 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=1 2024-04-06T10:05:08.243 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=1 2024-04-06T10:05:08.243 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-06T10:05:08.264 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T10:05:08.264 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T10:05:08.264 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T10:05:08.265 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/image_34 2024-04-06T10:05:08.407 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T10:05:08.409 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror image_34 2024-04-06T10:05:08.409 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:05:08.409 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:05:08.409 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_34 2024-04-06T10:05:08.409 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror image_34 '' 2024-04-06T10:05:08.409 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:05:08.409 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:05:08.409 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_34 2024-04-06T10:05:08.409 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T10:05:08.410 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-image_34.status 2024-04-06T10:05:08.410 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-image_34.status 2024-04-06T10:05:08.410 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T10:05:08.412 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-image_34.status 2024-04-06T10:05:08.413 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_34 2024-04-06T10:05:08.413 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_34.status 2024-04-06T10:05:08.463 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 1 2024-04-06T10:05:08.464 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 1 2024-04-06T10:05:08.464 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T10:05:08.464 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=5, tag_tid=2, entry_tid=1], [object_number=4, tag_tid=2, entry_tid=0], [object_number=7, tag_tid=1, entry_tid=8019], [object_number=6, tag_tid=1, entry_tid=8018]]], state=connected] 2024-04-06T10:05:08.464 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=5, tag_tid=1, entry_tid=8021], [object_number=4, tag_tid=1, entry_tid=8020], [object_number=7, tag_tid=1, entry_tid=8019], [object_number=6, tag_tid=1, entry_tid=8018]]], state=connected] 2024-04-06T10:05:08.467 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_34.status 2024-04-06T10:05:08.469 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=5, tag_tid=2, entry_tid=1' 2024-04-06T10:05:08.469 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror image_34 2024-04-06T10:05:08.469 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:05:08.470 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:05:08.470 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_34 2024-04-06T10:05:08.470 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror image_34 '..*' 2024-04-06T10:05:08.470 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:05:08.470 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:05:08.470 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_34 2024-04-06T10:05:08.470 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T10:05:08.470 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-image_34.status 2024-04-06T10:05:08.471 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-image_34.status 2024-04-06T10:05:08.471 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T10:05:08.471 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-image_34.status 2024-04-06T10:05:08.472 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_34 2024-04-06T10:05:08.472 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_34.status 2024-04-06T10:05:08.517 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 1 2024-04-06T10:05:08.517 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 1 2024-04-06T10:05:08.517 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T10:05:08.517 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=5, tag_tid=2, entry_tid=1], [object_number=4, tag_tid=2, entry_tid=0], [object_number=7, tag_tid=1, entry_tid=8019], [object_number=6, tag_tid=1, entry_tid=8018]]], state=connected] 2024-04-06T10:05:08.517 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=5, tag_tid=1, entry_tid=8021], [object_number=4, tag_tid=1, entry_tid=8020], [object_number=7, tag_tid=1, entry_tid=8019], [object_number=6, tag_tid=1, entry_tid=8018]]], state=connected] 2024-04-06T10:05:08.521 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_34.status 2024-04-06T10:05:08.522 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=5, tag_tid=1, entry_tid=8021' 2024-04-06T10:05:08.522 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=5, tag_tid=2, entry_tid=1' -a 'object_number=5, tag_tid=2, entry_tid=1' = 'object_number=5, tag_tid=1, entry_tid=8021' 2024-04-06T10:05:08.522 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=5, tag_tid=1, entry_tid=8021' '!=' 'object_number=5, tag_tid=1, entry_tid=8021' 2024-04-06T10:05:08.523 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T10:05:08.523 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 1.6 2024-04-06T10:05:10.125 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror image_34 2024-04-06T10:05:10.125 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T10:05:10.125 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:05:10.125 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_34 2024-04-06T10:05:10.126 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T10:05:10.126 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n image_34 ']' 2024-04-06T10:05:10.126 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/image_34' 2024-04-06T10:05:10.126 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_34 2024-04-06T10:05:10.126 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T10:05:10.126 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T10:05:10.126 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T10:05:10.126 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T10:05:10.126 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T10:05:10.126 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T10:05:10.126 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T10:05:10.129 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T10:05:10.129 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T10:05:10.129 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T10:05:10.129 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_34 2024-04-06T10:05:10.129 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T10:05:10.129 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T10:05:10.129 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T10:05:10.129 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T10:05:10.129 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T10:05:10.129 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T10:05:10.129 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T10:05:10.130 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T10:05:10.130 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T10:05:10.130 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T10:05:10.130 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T10:05:10.130 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:05:10.130 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T10:05:10.130 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T10:05:10.130 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T10:05:10.130 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T10:05:10.130 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T10:05:10.130 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T10:05:10.130 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T10:05:10.131 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T10:05:10.131 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T10:05:10.132 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T10:05:10.132 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:05:10.132 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T10:05:10.132 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T10:05:10.132 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T10:05:10.132 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T10:05:10.132 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T10:05:10.132 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T10:05:10.133 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T10:05:10.155 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T10:05:10.155 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T10:05:10.155 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T10:05:10.155 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_34 2024-04-06T10:05:10.260 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:Invalid command: unused arguments: ['mirror/image_34'] 2024-04-06T10:05:10.260 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:rbd mirror flush : flush rbd mirror 2024-04-06T10:05:10.260 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:admin_socket: invalid command 2024-04-06T10:05:10.264 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T10:05:10.264 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/image_34 2024-04-06T10:05:10.264 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:1 2024-04-06T10:05:10.264 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T10:05:10.264 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T10:05:10.265 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-06T10:05:10.265 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:1 2024-04-06T10:05:10.265 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T10:05:10.265 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T10:05:10.265 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T10:05:10.265 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=1 2024-04-06T10:05:10.265 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-06T10:05:10.265 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T10:05:10.265 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:05:10.265 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=1 2024-04-06T10:05:10.265 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=1 2024-04-06T10:05:10.265 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-06T10:05:10.265 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:1 2024-04-06T10:05:10.266 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T10:05:10.266 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T10:05:10.266 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-06T10:05:10.266 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:1 2024-04-06T10:05:10.266 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T10:05:10.266 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T10:05:10.266 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:05:10.266 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=1 2024-04-06T10:05:10.266 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-06T10:05:10.266 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T10:05:10.266 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T10:05:10.266 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=1 2024-04-06T10:05:10.266 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=1 2024-04-06T10:05:10.267 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-06T10:05:10.288 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T10:05:10.288 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T10:05:10.288 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T10:05:10.289 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/image_34 2024-04-06T10:05:10.432 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T10:05:10.433 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror image_34 2024-04-06T10:05:10.433 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:05:10.434 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:05:10.434 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_34 2024-04-06T10:05:10.434 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror image_34 '' 2024-04-06T10:05:10.434 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:05:10.434 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:05:10.434 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_34 2024-04-06T10:05:10.434 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T10:05:10.435 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-image_34.status 2024-04-06T10:05:10.435 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-image_34.status 2024-04-06T10:05:10.435 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T10:05:10.436 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-image_34.status 2024-04-06T10:05:10.436 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_34 2024-04-06T10:05:10.436 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_34.status 2024-04-06T10:05:10.482 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 1 2024-04-06T10:05:10.482 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 1 2024-04-06T10:05:10.482 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T10:05:10.483 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=5, tag_tid=2, entry_tid=1], [object_number=4, tag_tid=2, entry_tid=0], [object_number=7, tag_tid=1, entry_tid=8019], [object_number=6, tag_tid=1, entry_tid=8018]]], state=connected] 2024-04-06T10:05:10.483 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=5, tag_tid=2, entry_tid=1], [object_number=4, tag_tid=2, entry_tid=0], [object_number=7, tag_tid=1, entry_tid=8019], [object_number=6, tag_tid=1, entry_tid=8018]]], state=connected] 2024-04-06T10:05:10.486 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_34.status 2024-04-06T10:05:10.487 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=5, tag_tid=2, entry_tid=1' 2024-04-06T10:05:10.488 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror image_34 2024-04-06T10:05:10.488 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:05:10.488 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:05:10.488 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_34 2024-04-06T10:05:10.489 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror image_34 '..*' 2024-04-06T10:05:10.489 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:05:10.489 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:05:10.489 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_34 2024-04-06T10:05:10.489 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T10:05:10.489 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-image_34.status 2024-04-06T10:05:10.489 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-image_34.status 2024-04-06T10:05:10.489 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T10:05:10.490 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-image_34.status 2024-04-06T10:05:10.490 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_34 2024-04-06T10:05:10.490 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_34.status 2024-04-06T10:05:10.534 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 1 2024-04-06T10:05:10.535 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 1 2024-04-06T10:05:10.535 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T10:05:10.535 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=5, tag_tid=2, entry_tid=1], [object_number=4, tag_tid=2, entry_tid=0], [object_number=7, tag_tid=1, entry_tid=8019], [object_number=6, tag_tid=1, entry_tid=8018]]], state=connected] 2024-04-06T10:05:10.535 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=5, tag_tid=2, entry_tid=1], [object_number=4, tag_tid=2, entry_tid=0], [object_number=7, tag_tid=1, entry_tid=8019], [object_number=6, tag_tid=1, entry_tid=8018]]], state=connected] 2024-04-06T10:05:10.538 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_34.status 2024-04-06T10:05:10.540 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=5, tag_tid=2, entry_tid=1' 2024-04-06T10:05:10.541 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=5, tag_tid=2, entry_tid=1' -a 'object_number=5, tag_tid=2, entry_tid=1' = 'object_number=5, tag_tid=2, entry_tid=1' 2024-04-06T10:05:10.541 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T10:05:10.541 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_snap_present cluster1 mirror image_34 snap 2024-04-06T10:05:10.541 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T10:05:10.541 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:05:10.541 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_34 2024-04-06T10:05:10.541 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local snap_name=snap 2024-04-06T10:05:10.542 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local s 2024-04-06T10:05:10.542 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-04-06T10:05:10.542 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 1 2024-04-06T10:05:11.541 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 info mirror/image_34@snap 2024-04-06T10:05:11.605 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:rbd image 'image_34': 2024-04-06T10:05:11.605 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: size 128 MiB in 32 objects 2024-04-06T10:05:11.605 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: order 22 (4 MiB objects) 2024-04-06T10:05:11.606 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: snapshot_count: 1 2024-04-06T10:05:11.606 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: id: 10ac5dc10e61 2024-04-06T10:05:11.606 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: block_name_prefix: rbd_data.10ac5dc10e61 2024-04-06T10:05:11.606 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: format: 2 2024-04-06T10:05:11.606 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: features: layering, exclusive-lock, journaling 2024-04-06T10:05:11.606 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: op_features: 2024-04-06T10:05:11.606 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: flags: 2024-04-06T10:05:11.606 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: create_timestamp: Sat Apr 6 09:55:49 2024 2024-04-06T10:05:11.606 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: access_timestamp: Sat Apr 6 09:55:49 2024 2024-04-06T10:05:11.606 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: modify_timestamp: Sat Apr 6 09:56:49 2024 2024-04-06T10:05:11.606 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: protected: False 2024-04-06T10:05:11.606 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: journal: 10ac5dc10e61 2024-04-06T10:05:11.606 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: mirroring state: enabled 2024-04-06T10:05:11.607 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: mirroring mode: journal 2024-04-06T10:05:11.607 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: mirroring global id: 2aacc711-8a89-476a-9b90-b1c16525b95c 2024-04-06T10:05:11.607 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: mirroring primary: false 2024-04-06T10:05:11.610 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T10:05:11.610 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ compare_image_snaps mirror image_34 snap 2024-04-06T10:05:11.610 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:05:11.610 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_34 2024-04-06T10:05:11.611 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local snap_name=snap 2024-04-06T10:05:11.611 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local ret=0 2024-04-06T10:05:11.611 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local rmt_export=/tmp/tmp.BhW6xKmROU/cluster2-mirror-image_34.export 2024-04-06T10:05:11.611 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loc_export=/tmp/tmp.BhW6xKmROU/cluster1-mirror-image_34.export 2024-04-06T10:05:11.611 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rm -f /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_34.export /tmp/tmp.BhW6xKmROU/cluster1-mirror-image_34.export 2024-04-06T10:05:11.611 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster2 -p mirror export image_34@snap /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_34.export 2024-04-06T10:05:11.782 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-04-06T10:05:11.791 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 -p mirror export image_34@snap /tmp/tmp.BhW6xKmROU/cluster1-mirror-image_34.export 2024-04-06T10:05:12.084 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-04-06T10:05:12.097 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmp /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_34.export /tmp/tmp.BhW6xKmROU/cluster1-mirror-image_34.export 2024-04-06T10:05:12.182 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rm -f /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_34.export /tmp/tmp.BhW6xKmROU/cluster1-mirror-image_34.export 2024-04-06T10:05:12.218 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T10:05:12.218 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-06T10:05:12.218 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ image=image_35 2024-04-06T10:05:12.218 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ create_snap cluster2 mirror image_35 snap 2024-04-06T10:05:12.218 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-04-06T10:05:12.219 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:05:12.219 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_35 2024-04-06T10:05:12.219 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local snap_name=snap 2024-04-06T10:05:12.219 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster2 -p mirror snap create image_35@snap --debug-rbd=20 --debug-journaler=20 2024-04-06T10:05:12.926 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_image_replay_started cluster1 mirror image_35 2024-04-06T10:05:12.926 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T10:05:12.926 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:05:12.926 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_35 2024-04-06T10:05:12.927 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_image_replay_state cluster1 mirror image_35 started 2024-04-06T10:05:12.927 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T10:05:12.927 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:05:12.927 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_35 2024-04-06T10:05:12.927 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state=started 2024-04-06T10:05:12.927 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local s 2024-04-06T10:05:12.927 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T10:05:12.927 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 1 2024-04-06T10:05:13.929 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test_image_replay_state cluster1 mirror image_35 started 2024-04-06T10:05:13.929 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T10:05:13.929 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:05:13.929 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_35 2024-04-06T10:05:13.929 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local test_state=started 2024-04-06T10:05:13.929 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local status_result 2024-04-06T10:05:13.929 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local current_state=stopped 2024-04-06T10:05:13.930 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_35 2024-04-06T10:05:13.930 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_instance=cluster1 2024-04-06T10:05:13.930 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -i state 2024-04-06T10:05:13.931 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ shift 2024-04-06T10:05:13.931 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T10:05:13.931 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance=cluster1 2024-04-06T10:05:13.931 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local loop_instance 2024-04-06T10:05:13.931 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T10:05:13.931 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sleep 0 2024-04-06T10:05:13.931 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-04-06T10:05:13.931 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ seq 0 3 2024-04-06T10:05:13.934 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T10:05:13.934 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_35 2024-04-06T10:05:13.934 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1:0 2024-04-06T10:05:13.934 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ shift 2024-04-06T10:05:13.934 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T10:05:13.934 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-04-06T10:05:13.934 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T10:05:13.934 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=cluster 2024-04-06T10:05:13.934 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T10:05:13.934 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:05:13.935 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T10:05:13.935 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T10:05:13.935 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval cluster=cluster1 2024-04-06T10:05:13.935 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ cluster=cluster1 2024-04-06T10:05:13.935 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T10:05:13.935 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T10:05:13.935 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-04-06T10:05:13.935 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local local_cluster=cluster1:0 2024-04-06T10:05:13.935 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local cluster=cluster1 2024-04-06T10:05:13.935 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local instance 2024-04-06T10:05:13.935 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T10:05:13.935 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local val=cluster1:0 2024-04-06T10:05:13.935 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local cluster_var_name=local_cluster 2024-04-06T10:05:13.935 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local instance_var_name=instance 2024-04-06T10:05:13.936 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ cluster=cluster1 2024-04-06T10:05:13.936 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T10:05:13.936 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ '[' 0 = cluster1:0 ']' 2024-04-06T10:05:13.937 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ eval local_cluster=cluster1 2024-04-06T10:05:13.937 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++++ local_cluster=cluster1 2024-04-06T10:05:13.937 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ eval instance=0 2024-04-06T10:05:13.937 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++++ instance=0 2024-04-06T10:05:13.937 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T10:05:13.960 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T10:05:13.960 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T10:05:13.960 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T10:05:13.960 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_35 2024-04-06T10:05:14.062 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:Invalid command: unused arguments: ['mirror/image_35'] 2024-04-06T10:05:14.063 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:rbd mirror status : get status for rbd mirror 2024-04-06T10:05:14.063 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:admin_socket: invalid command 2024-04-06T10:05:14.065 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T10:05:14.065 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_35 2024-04-06T10:05:14.066 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1:1 2024-04-06T10:05:14.066 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ shift 2024-04-06T10:05:14.066 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T10:05:14.066 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-04-06T10:05:14.066 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:1 2024-04-06T10:05:14.066 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=cluster 2024-04-06T10:05:14.066 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T10:05:14.066 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:05:14.066 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=1 2024-04-06T10:05:14.066 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-06T10:05:14.066 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval cluster=cluster1 2024-04-06T10:05:14.066 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ cluster=cluster1 2024-04-06T10:05:14.066 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=1 2024-04-06T10:05:14.067 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=1 2024-04-06T10:05:14.067 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-04-06T10:05:14.067 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local local_cluster=cluster1:1 2024-04-06T10:05:14.067 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local cluster=cluster1 2024-04-06T10:05:14.067 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local instance 2024-04-06T10:05:14.067 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-06T10:05:14.067 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local val=cluster1:1 2024-04-06T10:05:14.067 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local cluster_var_name=local_cluster 2024-04-06T10:05:14.067 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local instance_var_name=instance 2024-04-06T10:05:14.067 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ cluster=cluster1 2024-04-06T10:05:14.067 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=1 2024-04-06T10:05:14.067 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ '[' 1 = cluster1:1 ']' 2024-04-06T10:05:14.067 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ eval local_cluster=cluster1 2024-04-06T10:05:14.067 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++++ local_cluster=cluster1 2024-04-06T10:05:14.067 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ eval instance=1 2024-04-06T10:05:14.067 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++++ instance=1 2024-04-06T10:05:14.068 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-06T10:05:14.091 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T10:05:14.091 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T10:05:14.092 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T10:05:14.092 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_35 2024-04-06T10:05:14.254 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ return 0 2024-04-06T10:05:14.255 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ status_result=' "state": "Replaying"' 2024-04-06T10:05:14.255 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ echo ' "state": "Replaying"' 2024-04-06T10:05:14.255 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep -i Replaying 2024-04-06T10:05:14.257 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: "state": "Replaying" 2024-04-06T10:05:14.258 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ current_state=started 2024-04-06T10:05:14.258 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test started = started 2024-04-06T10:05:14.258 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T10:05:14.259 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_35 2024-04-06T10:05:14.259 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local local_cluster=cluster1 2024-04-06T10:05:14.259 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-04-06T10:05:14.259 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:05:14.259 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_35 2024-04-06T10:05:14.259 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' journal = journal ']' 2024-04-06T10:05:14.259 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_journal_replay_complete cluster1 cluster2 mirror image_35 2024-04-06T10:05:14.259 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local local_cluster=cluster1 2024-04-06T10:05:14.259 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-04-06T10:05:14.259 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:05:14.259 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_35 2024-04-06T10:05:14.259 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local s master_pos mirror_pos last_mirror_pos 2024-04-06T10:05:14.259 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local master_tag master_entry mirror_tag mirror_entry 2024-04-06T10:05:14.259 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T10:05:14.259 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T10:05:14.260 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T10:05:14.460 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror image_35 2024-04-06T10:05:14.461 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T10:05:14.461 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:05:14.461 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_35 2024-04-06T10:05:14.461 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T10:05:14.461 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n image_35 ']' 2024-04-06T10:05:14.461 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/image_35' 2024-04-06T10:05:14.461 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_35 2024-04-06T10:05:14.461 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T10:05:14.461 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T10:05:14.461 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T10:05:14.461 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T10:05:14.461 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T10:05:14.461 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T10:05:14.461 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T10:05:14.462 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T10:05:14.462 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T10:05:14.464 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T10:05:14.464 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_35 2024-04-06T10:05:14.464 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T10:05:14.464 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T10:05:14.464 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T10:05:14.464 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T10:05:14.464 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T10:05:14.465 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T10:05:14.465 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T10:05:14.465 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T10:05:14.465 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T10:05:14.465 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T10:05:14.465 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T10:05:14.465 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:05:14.465 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T10:05:14.465 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T10:05:14.465 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T10:05:14.465 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T10:05:14.465 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T10:05:14.465 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T10:05:14.465 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T10:05:14.465 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T10:05:14.465 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T10:05:14.465 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T10:05:14.466 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:05:14.466 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T10:05:14.466 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T10:05:14.466 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T10:05:14.466 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T10:05:14.466 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T10:05:14.466 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T10:05:14.466 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T10:05:14.491 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T10:05:14.491 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T10:05:14.491 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T10:05:14.492 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_35 2024-04-06T10:05:14.603 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:Invalid command: unused arguments: ['mirror/image_35'] 2024-04-06T10:05:14.603 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:rbd mirror flush : flush rbd mirror 2024-04-06T10:05:14.603 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:admin_socket: invalid command 2024-04-06T10:05:14.606 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T10:05:14.606 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/image_35 2024-04-06T10:05:14.606 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:1 2024-04-06T10:05:14.606 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T10:05:14.606 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T10:05:14.606 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-06T10:05:14.606 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:1 2024-04-06T10:05:14.606 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T10:05:14.606 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T10:05:14.606 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T10:05:14.606 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=1 2024-04-06T10:05:14.606 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-06T10:05:14.606 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T10:05:14.607 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:05:14.607 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=1 2024-04-06T10:05:14.607 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=1 2024-04-06T10:05:14.607 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-06T10:05:14.607 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:1 2024-04-06T10:05:14.607 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T10:05:14.607 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T10:05:14.607 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-06T10:05:14.607 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:1 2024-04-06T10:05:14.607 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T10:05:14.607 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T10:05:14.607 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:05:14.607 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=1 2024-04-06T10:05:14.607 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-06T10:05:14.607 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T10:05:14.608 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T10:05:14.608 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=1 2024-04-06T10:05:14.608 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=1 2024-04-06T10:05:14.608 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-06T10:05:14.626 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T10:05:14.627 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T10:05:14.627 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T10:05:14.627 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/image_35 2024-04-06T10:05:14.776 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T10:05:14.777 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror image_35 2024-04-06T10:05:14.777 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:05:14.777 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:05:14.777 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_35 2024-04-06T10:05:14.777 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror image_35 '' 2024-04-06T10:05:14.778 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:05:14.778 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:05:14.778 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_35 2024-04-06T10:05:14.778 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T10:05:14.778 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-image_35.status 2024-04-06T10:05:14.778 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-image_35.status 2024-04-06T10:05:14.778 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T10:05:14.780 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-image_35.status 2024-04-06T10:05:14.781 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_35 2024-04-06T10:05:14.781 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_35.status 2024-04-06T10:05:14.831 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 0 2024-04-06T10:05:14.831 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 0 2024-04-06T10:05:14.832 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T10:05:14.832 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=1, tag_tid=3, entry_tid=1], [object_number=0, tag_tid=3, entry_tid=0], [object_number=3, tag_tid=1, entry_tid=3559], [object_number=2, tag_tid=1, entry_tid=3558]]], state=connected] 2024-04-06T10:05:14.832 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=1, tag_tid=2, entry_tid=1], [object_number=0, tag_tid=2, entry_tid=0], [object_number=3, tag_tid=1, entry_tid=3559], [object_number=2, tag_tid=1, entry_tid=3558]]], state=connected] 2024-04-06T10:05:14.835 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_35.status 2024-04-06T10:05:14.836 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=1, tag_tid=3, entry_tid=1' 2024-04-06T10:05:14.837 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror image_35 2024-04-06T10:05:14.838 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:05:14.838 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:05:14.838 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_35 2024-04-06T10:05:14.838 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror image_35 '..*' 2024-04-06T10:05:14.838 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:05:14.838 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:05:14.838 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_35 2024-04-06T10:05:14.838 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T10:05:14.838 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-image_35.status 2024-04-06T10:05:14.838 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-image_35.status 2024-04-06T10:05:14.838 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T10:05:14.839 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-image_35.status 2024-04-06T10:05:14.839 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_35 2024-04-06T10:05:14.839 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_35.status 2024-04-06T10:05:14.886 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 0 2024-04-06T10:05:14.886 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 0 2024-04-06T10:05:14.886 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T10:05:14.886 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=1, tag_tid=3, entry_tid=1], [object_number=0, tag_tid=3, entry_tid=0], [object_number=3, tag_tid=1, entry_tid=3559], [object_number=2, tag_tid=1, entry_tid=3558]]], state=connected] 2024-04-06T10:05:14.886 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=1, tag_tid=2, entry_tid=1], [object_number=0, tag_tid=2, entry_tid=0], [object_number=3, tag_tid=1, entry_tid=3559], [object_number=2, tag_tid=1, entry_tid=3558]]], state=connected] 2024-04-06T10:05:14.890 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_35.status 2024-04-06T10:05:14.891 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=1, tag_tid=2, entry_tid=1' 2024-04-06T10:05:14.891 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=1, tag_tid=3, entry_tid=1' -a 'object_number=1, tag_tid=3, entry_tid=1' = 'object_number=1, tag_tid=2, entry_tid=1' 2024-04-06T10:05:14.892 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=1, tag_tid=2, entry_tid=1' '!=' '' 2024-04-06T10:05:14.892 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T10:05:14.892 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=1, tag_tid=2, entry_tid=1' = '' 2024-04-06T10:05:14.892 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=1, tag_tid=2, entry_tid=1' 2024-04-06T10:05:14.892 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=1, tag_tid=3, entry_tid=1' 2024-04-06T10:05:14.892 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T10:05:14.892 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T10:05:14.893 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=3 2024-04-06T10:05:14.894 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=1, tag_tid=2, entry_tid=1' 2024-04-06T10:05:14.894 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T10:05:14.894 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T10:05:14.895 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=2 2024-04-06T10:05:14.895 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=1, tag_tid=3, entry_tid=1' 2024-04-06T10:05:14.895 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T10:05:14.896 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T10:05:14.896 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T10:05:14.897 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=1, tag_tid=2, entry_tid=1' 2024-04-06T10:05:14.897 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T10:05:14.897 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T10:05:14.898 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=1 2024-04-06T10:05:14.898 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 3 = 2 -a 1 -le 1 2024-04-06T10:05:14.898 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T10:05:14.899 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T10:05:14.899 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T10:05:15.100 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror image_35 2024-04-06T10:05:15.100 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T10:05:15.101 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:05:15.101 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_35 2024-04-06T10:05:15.101 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T10:05:15.101 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n image_35 ']' 2024-04-06T10:05:15.101 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/image_35' 2024-04-06T10:05:15.101 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_35 2024-04-06T10:05:15.101 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T10:05:15.101 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T10:05:15.101 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T10:05:15.101 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T10:05:15.101 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T10:05:15.101 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T10:05:15.101 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T10:05:15.102 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T10:05:15.102 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T10:05:15.104 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T10:05:15.104 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_35 2024-04-06T10:05:15.104 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T10:05:15.104 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T10:05:15.104 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T10:05:15.105 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T10:05:15.105 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T10:05:15.105 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T10:05:15.105 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T10:05:15.105 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T10:05:15.105 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T10:05:15.105 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T10:05:15.105 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T10:05:15.105 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:05:15.105 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T10:05:15.105 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T10:05:15.105 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T10:05:15.105 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T10:05:15.105 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T10:05:15.105 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T10:05:15.106 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T10:05:15.106 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T10:05:15.106 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T10:05:15.106 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T10:05:15.106 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:05:15.106 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T10:05:15.106 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T10:05:15.106 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T10:05:15.106 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T10:05:15.106 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T10:05:15.106 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T10:05:15.107 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T10:05:15.131 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T10:05:15.131 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T10:05:15.131 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T10:05:15.131 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_35 2024-04-06T10:05:15.238 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:Invalid command: unused arguments: ['mirror/image_35'] 2024-04-06T10:05:15.238 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:rbd mirror flush : flush rbd mirror 2024-04-06T10:05:15.238 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:admin_socket: invalid command 2024-04-06T10:05:15.241 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T10:05:15.241 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/image_35 2024-04-06T10:05:15.241 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:1 2024-04-06T10:05:15.241 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T10:05:15.241 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T10:05:15.241 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-06T10:05:15.241 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:1 2024-04-06T10:05:15.241 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T10:05:15.241 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T10:05:15.241 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T10:05:15.242 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=1 2024-04-06T10:05:15.242 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-06T10:05:15.242 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T10:05:15.242 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:05:15.242 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=1 2024-04-06T10:05:15.242 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=1 2024-04-06T10:05:15.246 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-06T10:05:15.246 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:1 2024-04-06T10:05:15.246 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T10:05:15.246 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T10:05:15.247 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-06T10:05:15.247 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:1 2024-04-06T10:05:15.247 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T10:05:15.247 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T10:05:15.247 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:05:15.247 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=1 2024-04-06T10:05:15.247 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-06T10:05:15.247 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T10:05:15.247 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T10:05:15.247 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=1 2024-04-06T10:05:15.247 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=1 2024-04-06T10:05:15.247 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-06T10:05:15.267 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T10:05:15.267 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T10:05:15.267 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T10:05:15.267 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/image_35 2024-04-06T10:05:15.418 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T10:05:15.419 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror image_35 2024-04-06T10:05:15.419 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:05:15.419 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:05:15.419 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_35 2024-04-06T10:05:15.419 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror image_35 '' 2024-04-06T10:05:15.419 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:05:15.419 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:05:15.420 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_35 2024-04-06T10:05:15.420 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T10:05:15.420 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-image_35.status 2024-04-06T10:05:15.421 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-image_35.status 2024-04-06T10:05:15.421 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T10:05:15.422 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-image_35.status 2024-04-06T10:05:15.540 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_35 2024-04-06T10:05:15.540 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_35.status 2024-04-06T10:05:15.544 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 0 2024-04-06T10:05:15.544 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 0 2024-04-06T10:05:15.545 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T10:05:15.545 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=1, tag_tid=3, entry_tid=1], [object_number=0, tag_tid=3, entry_tid=0], [object_number=3, tag_tid=1, entry_tid=3559], [object_number=2, tag_tid=1, entry_tid=3558]]], state=connected] 2024-04-06T10:05:15.545 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=1, tag_tid=2, entry_tid=1], [object_number=0, tag_tid=2, entry_tid=0], [object_number=3, tag_tid=1, entry_tid=3559], [object_number=2, tag_tid=1, entry_tid=3558]]], state=connected] 2024-04-06T10:05:15.545 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_35.status 2024-04-06T10:05:15.545 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=1, tag_tid=3, entry_tid=1' 2024-04-06T10:05:15.545 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror image_35 2024-04-06T10:05:15.545 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:05:15.545 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:05:15.545 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_35 2024-04-06T10:05:15.545 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror image_35 '..*' 2024-04-06T10:05:15.545 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:05:15.546 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:05:15.546 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_35 2024-04-06T10:05:15.546 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T10:05:15.546 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-image_35.status 2024-04-06T10:05:15.546 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-image_35.status 2024-04-06T10:05:15.546 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T10:05:15.546 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-image_35.status 2024-04-06T10:05:15.546 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_35 2024-04-06T10:05:15.546 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_35.status 2024-04-06T10:05:15.546 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 0 2024-04-06T10:05:15.546 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 0 2024-04-06T10:05:15.546 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T10:05:15.546 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=1, tag_tid=3, entry_tid=1], [object_number=0, tag_tid=3, entry_tid=0], [object_number=3, tag_tid=1, entry_tid=3559], [object_number=2, tag_tid=1, entry_tid=3558]]], state=connected] 2024-04-06T10:05:15.547 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=1, tag_tid=2, entry_tid=1], [object_number=0, tag_tid=2, entry_tid=0], [object_number=3, tag_tid=1, entry_tid=3559], [object_number=2, tag_tid=1, entry_tid=3558]]], state=connected] 2024-04-06T10:05:15.547 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_35.status 2024-04-06T10:05:15.547 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=1, tag_tid=2, entry_tid=1' 2024-04-06T10:05:15.547 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=1, tag_tid=3, entry_tid=1' -a 'object_number=1, tag_tid=3, entry_tid=1' = 'object_number=1, tag_tid=2, entry_tid=1' 2024-04-06T10:05:15.547 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=1, tag_tid=2, entry_tid=1' '!=' 'object_number=1, tag_tid=2, entry_tid=1' 2024-04-06T10:05:15.547 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T10:05:15.547 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.4 2024-04-06T10:05:15.927 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror image_35 2024-04-06T10:05:15.927 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T10:05:15.927 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:05:15.927 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_35 2024-04-06T10:05:15.927 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T10:05:15.927 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n image_35 ']' 2024-04-06T10:05:15.927 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/image_35' 2024-04-06T10:05:15.928 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_35 2024-04-06T10:05:15.928 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T10:05:15.928 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T10:05:15.928 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T10:05:15.928 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T10:05:15.928 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T10:05:15.928 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T10:05:15.928 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T10:05:15.928 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T10:05:15.930 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T10:05:15.932 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T10:05:15.933 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_35 2024-04-06T10:05:15.933 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T10:05:15.933 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T10:05:15.933 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T10:05:15.933 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T10:05:15.933 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T10:05:15.933 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T10:05:15.933 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T10:05:15.933 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T10:05:15.933 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T10:05:15.933 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T10:05:15.933 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T10:05:15.933 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:05:15.933 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T10:05:15.934 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T10:05:15.934 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T10:05:15.934 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T10:05:15.934 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T10:05:15.934 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T10:05:15.934 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T10:05:15.934 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T10:05:15.934 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T10:05:15.934 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T10:05:15.934 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:05:15.934 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T10:05:15.934 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T10:05:15.934 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T10:05:15.934 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T10:05:15.934 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T10:05:15.935 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T10:05:15.935 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T10:05:15.957 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T10:05:15.958 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T10:05:15.958 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T10:05:15.958 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_35 2024-04-06T10:05:16.068 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:Invalid command: unused arguments: ['mirror/image_35'] 2024-04-06T10:05:16.068 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:rbd mirror flush : flush rbd mirror 2024-04-06T10:05:16.068 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:admin_socket: invalid command 2024-04-06T10:05:16.070 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T10:05:16.070 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/image_35 2024-04-06T10:05:16.070 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:1 2024-04-06T10:05:16.070 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T10:05:16.070 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T10:05:16.070 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-06T10:05:16.070 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:1 2024-04-06T10:05:16.071 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T10:05:16.071 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T10:05:16.071 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T10:05:16.071 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=1 2024-04-06T10:05:16.071 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-06T10:05:16.071 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T10:05:16.071 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:05:16.071 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=1 2024-04-06T10:05:16.071 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=1 2024-04-06T10:05:16.073 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-06T10:05:16.073 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:1 2024-04-06T10:05:16.073 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T10:05:16.073 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T10:05:16.073 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-06T10:05:16.073 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:1 2024-04-06T10:05:16.073 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T10:05:16.073 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T10:05:16.073 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:05:16.073 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=1 2024-04-06T10:05:16.073 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-06T10:05:16.073 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T10:05:16.073 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T10:05:16.074 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=1 2024-04-06T10:05:16.074 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=1 2024-04-06T10:05:16.074 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-06T10:05:16.091 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T10:05:16.091 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T10:05:16.091 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T10:05:16.092 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/image_35 2024-04-06T10:05:16.235 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T10:05:16.235 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror image_35 2024-04-06T10:05:16.236 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:05:16.236 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:05:16.236 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_35 2024-04-06T10:05:16.236 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror image_35 '' 2024-04-06T10:05:16.236 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:05:16.236 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:05:16.236 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_35 2024-04-06T10:05:16.236 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T10:05:16.237 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-image_35.status 2024-04-06T10:05:16.237 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-image_35.status 2024-04-06T10:05:16.237 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T10:05:16.238 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-image_35.status 2024-04-06T10:05:16.239 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_35 2024-04-06T10:05:16.239 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_35.status 2024-04-06T10:05:16.285 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 0 2024-04-06T10:05:16.285 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 0 2024-04-06T10:05:16.285 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T10:05:16.285 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=1, tag_tid=3, entry_tid=1], [object_number=0, tag_tid=3, entry_tid=0], [object_number=3, tag_tid=1, entry_tid=3559], [object_number=2, tag_tid=1, entry_tid=3558]]], state=connected] 2024-04-06T10:05:16.285 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=1, tag_tid=2, entry_tid=1], [object_number=0, tag_tid=2, entry_tid=0], [object_number=3, tag_tid=1, entry_tid=3559], [object_number=2, tag_tid=1, entry_tid=3558]]], state=connected] 2024-04-06T10:05:16.288 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_35.status 2024-04-06T10:05:16.290 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=1, tag_tid=3, entry_tid=1' 2024-04-06T10:05:16.291 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror image_35 2024-04-06T10:05:16.291 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:05:16.291 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:05:16.291 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_35 2024-04-06T10:05:16.291 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror image_35 '..*' 2024-04-06T10:05:16.291 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:05:16.291 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:05:16.291 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_35 2024-04-06T10:05:16.292 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T10:05:16.292 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-image_35.status 2024-04-06T10:05:16.292 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-image_35.status 2024-04-06T10:05:16.292 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T10:05:16.292 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-image_35.status 2024-04-06T10:05:16.292 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_35 2024-04-06T10:05:16.293 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_35.status 2024-04-06T10:05:16.338 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 0 2024-04-06T10:05:16.338 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 0 2024-04-06T10:05:16.338 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T10:05:16.338 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=1, tag_tid=3, entry_tid=1], [object_number=0, tag_tid=3, entry_tid=0], [object_number=3, tag_tid=1, entry_tid=3559], [object_number=2, tag_tid=1, entry_tid=3558]]], state=connected] 2024-04-06T10:05:16.338 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=1, tag_tid=2, entry_tid=1], [object_number=0, tag_tid=2, entry_tid=0], [object_number=3, tag_tid=1, entry_tid=3559], [object_number=2, tag_tid=1, entry_tid=3558]]], state=connected] 2024-04-06T10:05:16.342 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_35.status 2024-04-06T10:05:16.343 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=1, tag_tid=2, entry_tid=1' 2024-04-06T10:05:16.343 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=1, tag_tid=3, entry_tid=1' -a 'object_number=1, tag_tid=3, entry_tid=1' = 'object_number=1, tag_tid=2, entry_tid=1' 2024-04-06T10:05:16.343 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=1, tag_tid=2, entry_tid=1' '!=' 'object_number=1, tag_tid=2, entry_tid=1' 2024-04-06T10:05:16.343 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T10:05:16.343 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.8 2024-04-06T10:05:17.145 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror image_35 2024-04-06T10:05:17.145 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T10:05:17.145 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:05:17.145 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_35 2024-04-06T10:05:17.145 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T10:05:17.145 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n image_35 ']' 2024-04-06T10:05:17.146 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/image_35' 2024-04-06T10:05:17.146 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_35 2024-04-06T10:05:17.146 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T10:05:17.146 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T10:05:17.146 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T10:05:17.146 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T10:05:17.146 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T10:05:17.146 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T10:05:17.146 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T10:05:17.147 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T10:05:17.147 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T10:05:17.149 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T10:05:17.149 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_35 2024-04-06T10:05:17.149 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T10:05:17.149 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T10:05:17.149 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T10:05:17.149 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T10:05:17.149 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T10:05:17.149 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T10:05:17.149 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T10:05:17.150 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T10:05:17.150 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T10:05:17.150 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T10:05:17.150 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T10:05:17.150 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:05:17.150 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T10:05:17.150 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T10:05:17.151 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T10:05:17.151 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T10:05:17.151 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T10:05:17.151 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T10:05:17.152 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T10:05:17.152 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T10:05:17.152 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T10:05:17.152 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T10:05:17.152 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:05:17.152 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T10:05:17.152 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T10:05:17.152 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T10:05:17.152 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T10:05:17.153 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T10:05:17.153 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T10:05:17.153 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T10:05:17.176 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T10:05:17.177 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T10:05:17.177 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T10:05:17.177 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_35 2024-04-06T10:05:17.281 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:Invalid command: unused arguments: ['mirror/image_35'] 2024-04-06T10:05:17.282 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:rbd mirror flush : flush rbd mirror 2024-04-06T10:05:17.282 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:admin_socket: invalid command 2024-04-06T10:05:17.285 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T10:05:17.286 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/image_35 2024-04-06T10:05:17.286 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:1 2024-04-06T10:05:17.286 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T10:05:17.286 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T10:05:17.286 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-06T10:05:17.286 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:1 2024-04-06T10:05:17.286 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T10:05:17.286 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T10:05:17.289 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T10:05:17.290 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=1 2024-04-06T10:05:17.290 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-06T10:05:17.290 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T10:05:17.290 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:05:17.290 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=1 2024-04-06T10:05:17.290 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=1 2024-04-06T10:05:17.290 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-06T10:05:17.290 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:1 2024-04-06T10:05:17.290 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T10:05:17.290 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T10:05:17.290 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-06T10:05:17.290 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:1 2024-04-06T10:05:17.291 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T10:05:17.291 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T10:05:17.291 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:05:17.291 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=1 2024-04-06T10:05:17.291 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-06T10:05:17.291 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T10:05:17.291 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T10:05:17.291 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=1 2024-04-06T10:05:17.291 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=1 2024-04-06T10:05:17.291 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-06T10:05:17.311 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T10:05:17.311 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T10:05:17.311 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T10:05:17.311 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/image_35 2024-04-06T10:05:17.460 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T10:05:17.461 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror image_35 2024-04-06T10:05:17.461 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:05:17.461 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:05:17.462 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_35 2024-04-06T10:05:17.462 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror image_35 '' 2024-04-06T10:05:17.462 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:05:17.462 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:05:17.462 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_35 2024-04-06T10:05:17.462 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T10:05:17.462 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-image_35.status 2024-04-06T10:05:17.463 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-image_35.status 2024-04-06T10:05:17.463 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T10:05:17.465 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-image_35.status 2024-04-06T10:05:17.465 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_35 2024-04-06T10:05:17.465 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_35.status 2024-04-06T10:05:17.515 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 0 2024-04-06T10:05:17.515 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 0 2024-04-06T10:05:17.515 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T10:05:17.515 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=1, tag_tid=3, entry_tid=1], [object_number=0, tag_tid=3, entry_tid=0], [object_number=3, tag_tid=1, entry_tid=3559], [object_number=2, tag_tid=1, entry_tid=3558]]], state=connected] 2024-04-06T10:05:17.515 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=1, tag_tid=2, entry_tid=1], [object_number=0, tag_tid=2, entry_tid=0], [object_number=3, tag_tid=1, entry_tid=3559], [object_number=2, tag_tid=1, entry_tid=3558]]], state=connected] 2024-04-06T10:05:17.519 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_35.status 2024-04-06T10:05:17.520 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=1, tag_tid=3, entry_tid=1' 2024-04-06T10:05:17.521 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror image_35 2024-04-06T10:05:17.521 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:05:17.521 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:05:17.521 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_35 2024-04-06T10:05:17.522 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror image_35 '..*' 2024-04-06T10:05:17.522 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:05:17.522 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:05:17.522 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_35 2024-04-06T10:05:17.522 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T10:05:17.522 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-image_35.status 2024-04-06T10:05:17.522 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-image_35.status 2024-04-06T10:05:17.522 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T10:05:17.522 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-image_35.status 2024-04-06T10:05:17.523 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_35 2024-04-06T10:05:17.523 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_35.status 2024-04-06T10:05:17.569 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 0 2024-04-06T10:05:17.569 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 0 2024-04-06T10:05:17.569 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T10:05:17.569 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=1, tag_tid=3, entry_tid=1], [object_number=0, tag_tid=3, entry_tid=0], [object_number=3, tag_tid=1, entry_tid=3559], [object_number=2, tag_tid=1, entry_tid=3558]]], state=connected] 2024-04-06T10:05:17.569 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=1, tag_tid=2, entry_tid=1], [object_number=0, tag_tid=2, entry_tid=0], [object_number=3, tag_tid=1, entry_tid=3559], [object_number=2, tag_tid=1, entry_tid=3558]]], state=connected] 2024-04-06T10:05:17.572 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_35.status 2024-04-06T10:05:17.574 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=1, tag_tid=2, entry_tid=1' 2024-04-06T10:05:17.574 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=1, tag_tid=3, entry_tid=1' -a 'object_number=1, tag_tid=3, entry_tid=1' = 'object_number=1, tag_tid=2, entry_tid=1' 2024-04-06T10:05:17.574 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=1, tag_tid=2, entry_tid=1' '!=' 'object_number=1, tag_tid=2, entry_tid=1' 2024-04-06T10:05:17.574 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T10:05:17.574 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 1.6 2024-04-06T10:05:19.176 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror image_35 2024-04-06T10:05:19.176 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T10:05:19.176 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:05:19.176 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_35 2024-04-06T10:05:19.176 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T10:05:19.176 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n image_35 ']' 2024-04-06T10:05:19.176 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/image_35' 2024-04-06T10:05:19.176 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_35 2024-04-06T10:05:19.176 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T10:05:19.176 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T10:05:19.177 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T10:05:19.177 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T10:05:19.177 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T10:05:19.177 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T10:05:19.177 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T10:05:19.177 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T10:05:19.177 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T10:05:19.180 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T10:05:19.180 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_35 2024-04-06T10:05:19.180 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T10:05:19.180 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T10:05:19.180 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T10:05:19.180 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T10:05:19.181 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T10:05:19.181 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T10:05:19.181 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T10:05:19.181 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T10:05:19.181 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T10:05:19.181 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T10:05:19.181 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T10:05:19.181 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:05:19.181 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T10:05:19.181 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T10:05:19.181 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T10:05:19.181 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T10:05:19.181 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T10:05:19.181 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T10:05:19.181 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T10:05:19.181 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T10:05:19.182 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T10:05:19.183 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T10:05:19.183 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:05:19.183 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T10:05:19.183 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T10:05:19.183 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T10:05:19.183 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T10:05:19.183 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T10:05:19.183 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T10:05:19.183 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T10:05:19.206 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T10:05:19.207 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T10:05:19.207 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T10:05:19.207 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_35 2024-04-06T10:05:19.313 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:Invalid command: unused arguments: ['mirror/image_35'] 2024-04-06T10:05:19.313 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:rbd mirror flush : flush rbd mirror 2024-04-06T10:05:19.313 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:admin_socket: invalid command 2024-04-06T10:05:19.315 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T10:05:19.315 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/image_35 2024-04-06T10:05:19.315 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:1 2024-04-06T10:05:19.315 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T10:05:19.315 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T10:05:19.315 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-06T10:05:19.315 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:1 2024-04-06T10:05:19.315 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T10:05:19.315 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T10:05:19.315 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T10:05:19.315 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=1 2024-04-06T10:05:19.315 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-06T10:05:19.316 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T10:05:19.316 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:05:19.316 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=1 2024-04-06T10:05:19.316 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=1 2024-04-06T10:05:19.317 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-06T10:05:19.317 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:1 2024-04-06T10:05:19.317 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T10:05:19.317 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T10:05:19.317 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-06T10:05:19.317 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:1 2024-04-06T10:05:19.317 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T10:05:19.317 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T10:05:19.317 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:05:19.317 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=1 2024-04-06T10:05:19.317 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-06T10:05:19.317 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T10:05:19.318 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T10:05:19.318 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=1 2024-04-06T10:05:19.318 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=1 2024-04-06T10:05:19.318 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-06T10:05:19.340 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T10:05:19.340 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T10:05:19.340 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T10:05:19.340 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/image_35 2024-04-06T10:05:19.484 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T10:05:19.485 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror image_35 2024-04-06T10:05:19.485 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:05:19.485 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:05:19.485 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_35 2024-04-06T10:05:19.486 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror image_35 '' 2024-04-06T10:05:19.486 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:05:19.486 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:05:19.486 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_35 2024-04-06T10:05:19.486 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T10:05:19.486 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-image_35.status 2024-04-06T10:05:19.487 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T10:05:19.487 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-image_35.status 2024-04-06T10:05:19.488 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-image_35.status 2024-04-06T10:05:19.488 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_35 2024-04-06T10:05:19.488 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_35.status 2024-04-06T10:05:19.546 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 0 2024-04-06T10:05:19.546 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 0 2024-04-06T10:05:19.546 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T10:05:19.546 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=1, tag_tid=3, entry_tid=1], [object_number=0, tag_tid=3, entry_tid=0], [object_number=3, tag_tid=1, entry_tid=3559], [object_number=2, tag_tid=1, entry_tid=3558]]], state=connected] 2024-04-06T10:05:19.546 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=1, tag_tid=2, entry_tid=1], [object_number=0, tag_tid=2, entry_tid=0], [object_number=3, tag_tid=1, entry_tid=3559], [object_number=2, tag_tid=1, entry_tid=3558]]], state=connected] 2024-04-06T10:05:19.550 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_35.status 2024-04-06T10:05:19.551 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=1, tag_tid=3, entry_tid=1' 2024-04-06T10:05:19.552 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror image_35 2024-04-06T10:05:19.552 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:05:19.552 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:05:19.552 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_35 2024-04-06T10:05:19.552 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror image_35 '..*' 2024-04-06T10:05:19.552 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:05:19.553 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:05:19.553 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_35 2024-04-06T10:05:19.553 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T10:05:19.553 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-image_35.status 2024-04-06T10:05:19.553 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-image_35.status 2024-04-06T10:05:19.553 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T10:05:19.553 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-image_35.status 2024-04-06T10:05:19.554 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_35 2024-04-06T10:05:19.554 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_35.status 2024-04-06T10:05:19.600 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 0 2024-04-06T10:05:19.601 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 0 2024-04-06T10:05:19.601 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T10:05:19.601 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=1, tag_tid=3, entry_tid=1], [object_number=0, tag_tid=3, entry_tid=0], [object_number=3, tag_tid=1, entry_tid=3559], [object_number=2, tag_tid=1, entry_tid=3558]]], state=connected] 2024-04-06T10:05:19.601 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=1, tag_tid=2, entry_tid=1], [object_number=0, tag_tid=2, entry_tid=0], [object_number=3, tag_tid=1, entry_tid=3559], [object_number=2, tag_tid=1, entry_tid=3558]]], state=connected] 2024-04-06T10:05:19.604 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_35.status 2024-04-06T10:05:19.606 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=1, tag_tid=2, entry_tid=1' 2024-04-06T10:05:19.606 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=1, tag_tid=3, entry_tid=1' -a 'object_number=1, tag_tid=3, entry_tid=1' = 'object_number=1, tag_tid=2, entry_tid=1' 2024-04-06T10:05:19.606 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=1, tag_tid=2, entry_tid=1' '!=' 'object_number=1, tag_tid=2, entry_tid=1' 2024-04-06T10:05:19.606 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T10:05:19.606 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 2 2024-04-06T10:05:21.608 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror image_35 2024-04-06T10:05:21.608 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T10:05:21.608 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:05:21.609 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_35 2024-04-06T10:05:21.609 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T10:05:21.609 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n image_35 ']' 2024-04-06T10:05:21.609 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/image_35' 2024-04-06T10:05:21.609 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_35 2024-04-06T10:05:21.609 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T10:05:21.609 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T10:05:21.609 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T10:05:21.609 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T10:05:21.609 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T10:05:21.609 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T10:05:21.609 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T10:05:21.610 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T10:05:21.610 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T10:05:21.612 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T10:05:21.612 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_35 2024-04-06T10:05:21.612 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T10:05:21.612 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T10:05:21.612 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T10:05:21.612 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T10:05:21.612 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T10:05:21.612 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T10:05:21.612 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T10:05:21.612 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T10:05:21.613 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T10:05:21.613 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T10:05:21.613 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T10:05:21.613 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:05:21.613 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T10:05:21.613 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T10:05:21.613 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T10:05:21.614 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T10:05:21.615 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T10:05:21.615 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T10:05:21.615 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T10:05:21.615 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T10:05:21.615 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T10:05:21.615 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T10:05:21.615 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:05:21.615 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T10:05:21.615 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T10:05:21.615 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T10:05:21.615 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T10:05:21.615 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T10:05:21.615 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T10:05:21.615 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T10:05:21.640 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T10:05:21.640 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T10:05:21.640 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T10:05:21.640 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_35 2024-04-06T10:05:21.751 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:Invalid command: unused arguments: ['mirror/image_35'] 2024-04-06T10:05:21.751 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:rbd mirror flush : flush rbd mirror 2024-04-06T10:05:21.751 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:admin_socket: invalid command 2024-04-06T10:05:21.755 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T10:05:21.756 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/image_35 2024-04-06T10:05:21.756 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:1 2024-04-06T10:05:21.756 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T10:05:21.756 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T10:05:21.756 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-06T10:05:21.756 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:1 2024-04-06T10:05:21.756 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T10:05:21.756 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T10:05:21.756 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T10:05:21.756 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=1 2024-04-06T10:05:21.756 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-06T10:05:21.756 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T10:05:21.756 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:05:21.756 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=1 2024-04-06T10:05:21.757 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=1 2024-04-06T10:05:21.757 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-06T10:05:21.757 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:1 2024-04-06T10:05:21.757 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T10:05:21.757 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T10:05:21.757 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-06T10:05:21.757 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:1 2024-04-06T10:05:21.757 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T10:05:21.757 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T10:05:21.757 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:05:21.757 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=1 2024-04-06T10:05:21.757 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-06T10:05:21.757 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T10:05:21.757 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T10:05:21.757 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=1 2024-04-06T10:05:21.758 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=1 2024-04-06T10:05:21.758 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-06T10:05:21.779 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T10:05:21.780 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T10:05:21.780 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T10:05:21.781 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/image_35 2024-04-06T10:05:21.929 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T10:05:21.930 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror image_35 2024-04-06T10:05:21.930 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:05:21.930 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:05:21.930 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_35 2024-04-06T10:05:21.931 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror image_35 '' 2024-04-06T10:05:21.931 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:05:21.931 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:05:21.931 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_35 2024-04-06T10:05:21.931 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T10:05:21.931 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-image_35.status 2024-04-06T10:05:21.931 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-image_35.status 2024-04-06T10:05:21.931 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T10:05:21.933 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-image_35.status 2024-04-06T10:05:21.934 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_35.status 2024-04-06T10:05:21.934 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_35 2024-04-06T10:05:21.986 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 0 2024-04-06T10:05:21.986 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 0 2024-04-06T10:05:21.987 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T10:05:21.987 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=1, tag_tid=3, entry_tid=1], [object_number=0, tag_tid=3, entry_tid=0], [object_number=3, tag_tid=1, entry_tid=3559], [object_number=2, tag_tid=1, entry_tid=3558]]], state=connected] 2024-04-06T10:05:21.987 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=1, tag_tid=2, entry_tid=1], [object_number=0, tag_tid=2, entry_tid=0], [object_number=3, tag_tid=1, entry_tid=3559], [object_number=2, tag_tid=1, entry_tid=3558]]], state=connected] 2024-04-06T10:05:21.990 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_35.status 2024-04-06T10:05:21.991 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=1, tag_tid=3, entry_tid=1' 2024-04-06T10:05:21.992 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror image_35 2024-04-06T10:05:21.993 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:05:21.993 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:05:21.993 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_35 2024-04-06T10:05:21.993 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror image_35 '..*' 2024-04-06T10:05:21.993 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:05:21.993 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:05:21.993 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_35 2024-04-06T10:05:21.993 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T10:05:21.993 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-image_35.status 2024-04-06T10:05:21.993 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-image_35.status 2024-04-06T10:05:21.993 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T10:05:21.994 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-image_35.status 2024-04-06T10:05:21.994 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_35 2024-04-06T10:05:21.994 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_35.status 2024-04-06T10:05:22.040 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 0 2024-04-06T10:05:22.040 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 0 2024-04-06T10:05:22.040 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T10:05:22.040 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=1, tag_tid=3, entry_tid=1], [object_number=0, tag_tid=3, entry_tid=0], [object_number=3, tag_tid=1, entry_tid=3559], [object_number=2, tag_tid=1, entry_tid=3558]]], state=connected] 2024-04-06T10:05:22.040 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=1, tag_tid=2, entry_tid=1], [object_number=0, tag_tid=2, entry_tid=0], [object_number=3, tag_tid=1, entry_tid=3559], [object_number=2, tag_tid=1, entry_tid=3558]]], state=connected] 2024-04-06T10:05:22.044 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_35.status 2024-04-06T10:05:22.046 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=1, tag_tid=2, entry_tid=1' 2024-04-06T10:05:22.046 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=1, tag_tid=3, entry_tid=1' -a 'object_number=1, tag_tid=3, entry_tid=1' = 'object_number=1, tag_tid=2, entry_tid=1' 2024-04-06T10:05:22.046 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=1, tag_tid=2, entry_tid=1' '!=' 'object_number=1, tag_tid=2, entry_tid=1' 2024-04-06T10:05:22.046 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T10:05:22.046 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 2 2024-04-06T10:05:24.048 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror image_35 2024-04-06T10:05:24.048 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T10:05:24.048 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:05:24.048 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_35 2024-04-06T10:05:24.048 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T10:05:24.048 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n image_35 ']' 2024-04-06T10:05:24.048 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/image_35' 2024-04-06T10:05:24.048 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_35 2024-04-06T10:05:24.048 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T10:05:24.048 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T10:05:24.048 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T10:05:24.048 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T10:05:24.049 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T10:05:24.049 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T10:05:24.049 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T10:05:24.049 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T10:05:24.049 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T10:05:24.051 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T10:05:24.051 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_35 2024-04-06T10:05:24.051 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T10:05:24.051 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T10:05:24.052 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T10:05:24.052 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T10:05:24.052 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T10:05:24.052 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T10:05:24.052 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T10:05:24.052 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T10:05:24.052 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T10:05:24.052 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T10:05:24.052 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T10:05:24.052 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:05:24.052 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T10:05:24.052 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T10:05:24.053 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T10:05:24.053 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T10:05:24.053 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T10:05:24.053 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T10:05:24.053 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T10:05:24.053 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T10:05:24.053 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T10:05:24.054 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T10:05:24.054 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:05:24.054 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T10:05:24.054 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T10:05:24.054 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T10:05:24.054 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T10:05:24.054 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T10:05:24.054 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T10:05:24.054 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T10:05:24.079 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T10:05:24.080 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T10:05:24.080 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T10:05:24.080 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_35 2024-04-06T10:05:24.187 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:Invalid command: unused arguments: ['mirror/image_35'] 2024-04-06T10:05:24.187 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:rbd mirror flush : flush rbd mirror 2024-04-06T10:05:24.187 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:admin_socket: invalid command 2024-04-06T10:05:24.190 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T10:05:24.190 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/image_35 2024-04-06T10:05:24.190 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:1 2024-04-06T10:05:24.190 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T10:05:24.190 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T10:05:24.190 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-06T10:05:24.190 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:1 2024-04-06T10:05:24.190 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T10:05:24.190 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T10:05:24.190 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T10:05:24.190 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=1 2024-04-06T10:05:24.191 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-06T10:05:24.191 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T10:05:24.191 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:05:24.192 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=1 2024-04-06T10:05:24.192 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=1 2024-04-06T10:05:24.192 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-06T10:05:24.192 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:1 2024-04-06T10:05:24.192 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T10:05:24.193 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T10:05:24.193 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-06T10:05:24.193 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:1 2024-04-06T10:05:24.193 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T10:05:24.193 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T10:05:24.193 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:05:24.193 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=1 2024-04-06T10:05:24.193 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-06T10:05:24.193 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T10:05:24.193 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T10:05:24.193 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=1 2024-04-06T10:05:24.193 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=1 2024-04-06T10:05:24.193 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-06T10:05:24.217 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T10:05:24.217 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T10:05:24.217 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T10:05:24.217 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/image_35 2024-04-06T10:05:24.365 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T10:05:24.365 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror image_35 2024-04-06T10:05:24.365 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:05:24.365 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:05:24.366 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_35 2024-04-06T10:05:24.366 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror image_35 '' 2024-04-06T10:05:24.366 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:05:24.366 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:05:24.366 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_35 2024-04-06T10:05:24.366 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T10:05:24.367 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-image_35.status 2024-04-06T10:05:24.367 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-image_35.status 2024-04-06T10:05:24.367 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T10:05:24.368 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-image_35.status 2024-04-06T10:05:24.368 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_35 2024-04-06T10:05:24.368 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_35.status 2024-04-06T10:05:24.419 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 0 2024-04-06T10:05:24.420 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 0 2024-04-06T10:05:24.420 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T10:05:24.420 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=1, tag_tid=3, entry_tid=1], [object_number=0, tag_tid=3, entry_tid=0], [object_number=3, tag_tid=1, entry_tid=3559], [object_number=2, tag_tid=1, entry_tid=3558]]], state=connected] 2024-04-06T10:05:24.420 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=1, tag_tid=3, entry_tid=1], [object_number=0, tag_tid=3, entry_tid=0], [object_number=3, tag_tid=1, entry_tid=3559], [object_number=2, tag_tid=1, entry_tid=3558]]], state=connected] 2024-04-06T10:05:24.423 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_35.status 2024-04-06T10:05:24.424 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=1, tag_tid=3, entry_tid=1' 2024-04-06T10:05:24.425 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror image_35 2024-04-06T10:05:24.425 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:05:24.426 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:05:24.426 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_35 2024-04-06T10:05:24.426 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror image_35 '..*' 2024-04-06T10:05:24.426 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:05:24.426 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:05:24.426 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_35 2024-04-06T10:05:24.426 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T10:05:24.426 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-image_35.status 2024-04-06T10:05:24.426 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-image_35.status 2024-04-06T10:05:24.426 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T10:05:24.427 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-image_35.status 2024-04-06T10:05:24.427 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_35 2024-04-06T10:05:24.427 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_35.status 2024-04-06T10:05:24.473 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 0 2024-04-06T10:05:24.473 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 0 2024-04-06T10:05:24.473 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T10:05:24.473 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=1, tag_tid=3, entry_tid=1], [object_number=0, tag_tid=3, entry_tid=0], [object_number=3, tag_tid=1, entry_tid=3559], [object_number=2, tag_tid=1, entry_tid=3558]]], state=connected] 2024-04-06T10:05:24.473 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=1, tag_tid=3, entry_tid=1], [object_number=0, tag_tid=3, entry_tid=0], [object_number=3, tag_tid=1, entry_tid=3559], [object_number=2, tag_tid=1, entry_tid=3558]]], state=connected] 2024-04-06T10:05:24.476 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_35.status 2024-04-06T10:05:24.478 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=1, tag_tid=3, entry_tid=1' 2024-04-06T10:05:24.478 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=1, tag_tid=3, entry_tid=1' -a 'object_number=1, tag_tid=3, entry_tid=1' = 'object_number=1, tag_tid=3, entry_tid=1' 2024-04-06T10:05:24.479 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T10:05:24.479 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_snap_present cluster1 mirror image_35 snap 2024-04-06T10:05:24.479 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T10:05:24.479 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:05:24.479 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_35 2024-04-06T10:05:24.479 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local snap_name=snap 2024-04-06T10:05:24.479 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local s 2024-04-06T10:05:24.479 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-04-06T10:05:24.479 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 1 2024-04-06T10:05:25.479 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 info mirror/image_35@snap 2024-04-06T10:05:25.547 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:rbd image 'image_35': 2024-04-06T10:05:25.547 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: size 128 MiB in 32 objects 2024-04-06T10:05:25.547 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: order 22 (4 MiB objects) 2024-04-06T10:05:25.547 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: snapshot_count: 1 2024-04-06T10:05:25.547 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: id: 10ac6b04570c 2024-04-06T10:05:25.547 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: block_name_prefix: rbd_data.10ac6b04570c 2024-04-06T10:05:25.547 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: format: 2 2024-04-06T10:05:25.548 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: features: layering, exclusive-lock, journaling 2024-04-06T10:05:25.548 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: op_features: 2024-04-06T10:05:25.548 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: flags: 2024-04-06T10:05:25.548 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: create_timestamp: Sat Apr 6 09:55:59 2024 2024-04-06T10:05:25.548 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: access_timestamp: Sat Apr 6 09:55:59 2024 2024-04-06T10:05:25.548 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: modify_timestamp: Sat Apr 6 09:57:00 2024 2024-04-06T10:05:25.548 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: protected: False 2024-04-06T10:05:25.548 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: journal: 10ac6b04570c 2024-04-06T10:05:25.548 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: mirroring state: enabled 2024-04-06T10:05:25.548 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: mirroring mode: journal 2024-04-06T10:05:25.548 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: mirroring global id: b6c2b350-558f-4363-bbfa-39ce9ca3e774 2024-04-06T10:05:25.548 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: mirroring primary: false 2024-04-06T10:05:25.552 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T10:05:25.552 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ compare_image_snaps mirror image_35 snap 2024-04-06T10:05:25.552 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:05:25.552 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_35 2024-04-06T10:05:25.552 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local snap_name=snap 2024-04-06T10:05:25.552 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local ret=0 2024-04-06T10:05:25.552 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local rmt_export=/tmp/tmp.BhW6xKmROU/cluster2-mirror-image_35.export 2024-04-06T10:05:25.552 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loc_export=/tmp/tmp.BhW6xKmROU/cluster1-mirror-image_35.export 2024-04-06T10:05:25.552 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rm -f /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_35.export /tmp/tmp.BhW6xKmROU/cluster1-mirror-image_35.export 2024-04-06T10:05:25.553 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster2 -p mirror export image_35@snap /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_35.export 2024-04-06T10:05:25.710 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-04-06T10:05:25.717 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 -p mirror export image_35@snap /tmp/tmp.BhW6xKmROU/cluster1-mirror-image_35.export 2024-04-06T10:05:26.057 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-04-06T10:05:26.070 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmp /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_35.export /tmp/tmp.BhW6xKmROU/cluster1-mirror-image_35.export 2024-04-06T10:05:26.152 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rm -f /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_35.export /tmp/tmp.BhW6xKmROU/cluster1-mirror-image_35.export 2024-04-06T10:05:26.202 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T10:05:26.202 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-06T10:05:26.202 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ image=image_36 2024-04-06T10:05:26.202 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ create_snap cluster2 mirror image_36 snap 2024-04-06T10:05:26.203 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-04-06T10:05:26.203 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:05:26.203 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_36 2024-04-06T10:05:26.203 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local snap_name=snap 2024-04-06T10:05:26.203 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster2 -p mirror snap create image_36@snap --debug-rbd=20 --debug-journaler=20 2024-04-06T10:05:28.668 INFO:tasks.rbd_mirror_thrash:kill cluster2.client.mirror.0 2024-04-06T10:05:28.669 INFO:tasks.rbd_mirror.cluster2.client.mirror.0:Sent signal 15 2024-04-06T10:05:28.669 INFO:tasks.rbd_mirror_thrash:waiting for 1 secs before reviving daemons 2024-04-06T10:05:28.670 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi192.stderr:2024-04-06T10:05:28.672+0000 7f46df5c8640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.0 (PID: 152790) UID: 1000 2024-04-06T10:05:28.913 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_image_replay_started cluster1 mirror image_36 2024-04-06T10:05:28.913 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T10:05:28.913 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:05:28.913 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_36 2024-04-06T10:05:28.913 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_image_replay_state cluster1 mirror image_36 started 2024-04-06T10:05:28.913 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T10:05:28.914 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:05:28.914 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_36 2024-04-06T10:05:28.914 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state=started 2024-04-06T10:05:28.914 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local s 2024-04-06T10:05:28.914 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T10:05:28.914 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 1 2024-04-06T10:05:29.669 INFO:tasks.rbd_mirror_thrash:waiting for cluster2.client.mirror.0 2024-04-06T10:05:29.670 INFO:teuthology.orchestra.run:waiting for 600 2024-04-06T10:05:29.670 INFO:tasks.rbd_mirror_thrash:reviving cluster2.client.mirror.0 2024-04-06T10:05:29.670 INFO:tasks.rbd_mirror.cluster2.client.mirror.0:Restarting daemon 2024-04-06T10:05:29.670 DEBUG:teuthology.orchestra.run.smithi192:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.0 2024-04-06T10:05:29.674 INFO:tasks.rbd_mirror.cluster2.client.mirror.0:Started 2024-04-06T10:05:29.674 INFO:tasks.rbd_mirror_thrash:waiting for 93 secs before thrashing 2024-04-06T10:05:29.915 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test_image_replay_state cluster1 mirror image_36 started 2024-04-06T10:05:29.915 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T10:05:29.915 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:05:29.915 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_36 2024-04-06T10:05:29.915 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local test_state=started 2024-04-06T10:05:29.915 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local status_result 2024-04-06T10:05:29.915 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local current_state=stopped 2024-04-06T10:05:29.916 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_36 2024-04-06T10:05:29.916 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -i state 2024-04-06T10:05:29.916 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_instance=cluster1 2024-04-06T10:05:29.916 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ shift 2024-04-06T10:05:29.916 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T10:05:29.916 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance=cluster1 2024-04-06T10:05:29.916 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local loop_instance 2024-04-06T10:05:29.916 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T10:05:29.916 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sleep 0 2024-04-06T10:05:29.918 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-04-06T10:05:29.918 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ seq 0 3 2024-04-06T10:05:29.918 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T10:05:29.918 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_36 2024-04-06T10:05:29.918 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1:0 2024-04-06T10:05:29.918 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ shift 2024-04-06T10:05:29.918 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T10:05:29.918 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-04-06T10:05:29.919 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T10:05:29.919 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=cluster 2024-04-06T10:05:29.919 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T10:05:29.919 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:05:29.919 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T10:05:29.919 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T10:05:29.919 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval cluster=cluster1 2024-04-06T10:05:29.919 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ cluster=cluster1 2024-04-06T10:05:29.919 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T10:05:29.919 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T10:05:29.919 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-04-06T10:05:29.919 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local local_cluster=cluster1:0 2024-04-06T10:05:29.921 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local cluster=cluster1 2024-04-06T10:05:29.921 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local instance 2024-04-06T10:05:29.921 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T10:05:29.921 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local val=cluster1:0 2024-04-06T10:05:29.921 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local cluster_var_name=local_cluster 2024-04-06T10:05:29.921 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local instance_var_name=instance 2024-04-06T10:05:29.921 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ cluster=cluster1 2024-04-06T10:05:29.921 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T10:05:29.921 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ '[' 0 = cluster1:0 ']' 2024-04-06T10:05:29.921 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ eval local_cluster=cluster1 2024-04-06T10:05:29.921 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++++ local_cluster=cluster1 2024-04-06T10:05:29.921 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ eval instance=0 2024-04-06T10:05:29.921 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++++ instance=0 2024-04-06T10:05:29.922 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T10:05:29.939 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T10:05:29.939 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T10:05:29.939 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T10:05:29.939 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_36 2024-04-06T10:05:30.039 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:Invalid command: unused arguments: ['mirror/image_36'] 2024-04-06T10:05:30.039 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:rbd mirror status : get status for rbd mirror 2024-04-06T10:05:30.039 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:admin_socket: invalid command 2024-04-06T10:05:30.042 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T10:05:30.043 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_36 2024-04-06T10:05:30.043 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1:1 2024-04-06T10:05:30.043 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ shift 2024-04-06T10:05:30.043 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T10:05:30.043 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-04-06T10:05:30.043 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:1 2024-04-06T10:05:30.043 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=cluster 2024-04-06T10:05:30.043 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T10:05:30.043 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:05:30.043 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=1 2024-04-06T10:05:30.043 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-06T10:05:30.043 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval cluster=cluster1 2024-04-06T10:05:30.043 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ cluster=cluster1 2024-04-06T10:05:30.044 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=1 2024-04-06T10:05:30.044 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=1 2024-04-06T10:05:30.044 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-04-06T10:05:30.044 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local local_cluster=cluster1:1 2024-04-06T10:05:30.044 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local cluster=cluster1 2024-04-06T10:05:30.044 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local instance 2024-04-06T10:05:30.044 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-06T10:05:30.044 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local val=cluster1:1 2024-04-06T10:05:30.044 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local cluster_var_name=local_cluster 2024-04-06T10:05:30.044 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local instance_var_name=instance 2024-04-06T10:05:30.044 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ cluster=cluster1 2024-04-06T10:05:30.044 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=1 2024-04-06T10:05:30.044 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ '[' 1 = cluster1:1 ']' 2024-04-06T10:05:30.045 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ eval local_cluster=cluster1 2024-04-06T10:05:30.045 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++++ local_cluster=cluster1 2024-04-06T10:05:30.045 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ eval instance=1 2024-04-06T10:05:30.045 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++++ instance=1 2024-04-06T10:05:30.045 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-06T10:05:30.061 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T10:05:30.062 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T10:05:30.062 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T10:05:30.062 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_36 2024-04-06T10:05:30.217 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ return 0 2024-04-06T10:05:30.217 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ status_result=' "state": "Replaying"' 2024-04-06T10:05:30.218 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ echo ' "state": "Replaying"' 2024-04-06T10:05:30.218 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep -i Replaying 2024-04-06T10:05:30.220 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: "state": "Replaying" 2024-04-06T10:05:30.222 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ current_state=started 2024-04-06T10:05:30.222 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test started = started 2024-04-06T10:05:30.222 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T10:05:30.222 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_36 2024-04-06T10:05:30.222 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local local_cluster=cluster1 2024-04-06T10:05:30.222 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-04-06T10:05:30.222 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:05:30.222 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_36 2024-04-06T10:05:30.222 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' journal = journal ']' 2024-04-06T10:05:30.222 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_journal_replay_complete cluster1 cluster2 mirror image_36 2024-04-06T10:05:30.222 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local local_cluster=cluster1 2024-04-06T10:05:30.222 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-04-06T10:05:30.222 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:05:30.223 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_36 2024-04-06T10:05:30.223 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local s master_pos mirror_pos last_mirror_pos 2024-04-06T10:05:30.223 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local master_tag master_entry mirror_tag mirror_entry 2024-04-06T10:05:30.223 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T10:05:30.223 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T10:05:30.223 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T10:05:30.423 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror image_36 2024-04-06T10:05:30.423 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T10:05:30.424 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:05:30.424 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_36 2024-04-06T10:05:30.424 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T10:05:30.424 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n image_36 ']' 2024-04-06T10:05:30.424 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/image_36' 2024-04-06T10:05:30.424 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_36 2024-04-06T10:05:30.424 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T10:05:30.424 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T10:05:30.424 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T10:05:30.424 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T10:05:30.424 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T10:05:30.424 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T10:05:30.425 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T10:05:30.425 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T10:05:30.425 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T10:05:30.430 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T10:05:30.430 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_36 2024-04-06T10:05:30.430 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T10:05:30.430 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T10:05:30.430 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T10:05:30.430 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T10:05:30.430 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T10:05:30.430 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T10:05:30.430 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T10:05:30.430 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T10:05:30.430 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T10:05:30.431 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T10:05:30.431 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T10:05:30.431 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:05:30.431 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T10:05:30.431 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T10:05:30.431 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T10:05:30.431 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T10:05:30.431 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T10:05:30.431 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T10:05:30.431 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T10:05:30.431 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T10:05:30.431 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T10:05:30.432 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T10:05:30.432 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:05:30.432 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T10:05:30.432 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T10:05:30.432 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T10:05:30.432 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T10:05:30.432 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T10:05:30.432 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T10:05:30.432 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T10:05:30.454 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T10:05:30.454 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T10:05:30.454 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T10:05:30.454 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_36 2024-04-06T10:05:30.560 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:Invalid command: unused arguments: ['mirror/image_36'] 2024-04-06T10:05:30.560 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:rbd mirror flush : flush rbd mirror 2024-04-06T10:05:30.560 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:admin_socket: invalid command 2024-04-06T10:05:30.562 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T10:05:30.562 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/image_36 2024-04-06T10:05:30.562 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:1 2024-04-06T10:05:30.562 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T10:05:30.562 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T10:05:30.562 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-06T10:05:30.562 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:1 2024-04-06T10:05:30.563 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T10:05:30.563 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T10:05:30.563 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T10:05:30.563 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=1 2024-04-06T10:05:30.563 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-06T10:05:30.563 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T10:05:30.563 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:05:30.563 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=1 2024-04-06T10:05:30.563 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=1 2024-04-06T10:05:30.565 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-06T10:05:30.565 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:1 2024-04-06T10:05:30.565 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T10:05:30.565 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T10:05:30.565 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-06T10:05:30.565 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:1 2024-04-06T10:05:30.566 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T10:05:30.566 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T10:05:30.566 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:05:30.566 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=1 2024-04-06T10:05:30.566 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-06T10:05:30.566 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T10:05:30.566 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T10:05:30.566 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=1 2024-04-06T10:05:30.566 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=1 2024-04-06T10:05:30.566 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-06T10:05:30.588 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T10:05:30.588 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T10:05:30.588 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T10:05:30.588 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/image_36 2024-04-06T10:05:30.741 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T10:05:30.742 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror image_36 2024-04-06T10:05:30.742 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:05:30.743 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:05:30.743 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_36 2024-04-06T10:05:30.743 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror image_36 '' 2024-04-06T10:05:30.743 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:05:30.743 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:05:30.743 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_36 2024-04-06T10:05:30.743 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T10:05:30.743 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-image_36.status 2024-04-06T10:05:30.743 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-image_36.status 2024-04-06T10:05:30.744 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T10:05:30.746 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-image_36.status 2024-04-06T10:05:30.746 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_36 2024-04-06T10:05:30.747 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_36.status 2024-04-06T10:05:30.797 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 1 2024-04-06T10:05:30.798 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 1 2024-04-06T10:05:30.798 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T10:05:30.798 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=5, tag_tid=2, entry_tid=1], [object_number=4, tag_tid=2, entry_tid=0], [object_number=7, tag_tid=1, entry_tid=7807], [object_number=6, tag_tid=1, entry_tid=7806]]], state=connected] 2024-04-06T10:05:30.798 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=5, tag_tid=1, entry_tid=7809], [object_number=4, tag_tid=1, entry_tid=7808], [object_number=7, tag_tid=1, entry_tid=7807], [object_number=6, tag_tid=1, entry_tid=7806]]], state=connected] 2024-04-06T10:05:30.801 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_36.status 2024-04-06T10:05:30.803 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=5, tag_tid=2, entry_tid=1' 2024-04-06T10:05:30.804 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror image_36 2024-04-06T10:05:30.804 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:05:30.804 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:05:30.804 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_36 2024-04-06T10:05:30.804 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror image_36 '..*' 2024-04-06T10:05:30.804 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:05:30.804 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:05:30.804 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_36 2024-04-06T10:05:30.804 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T10:05:30.804 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-image_36.status 2024-04-06T10:05:30.804 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-image_36.status 2024-04-06T10:05:30.804 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T10:05:30.805 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-image_36.status 2024-04-06T10:05:30.805 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_36 2024-04-06T10:05:30.805 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_36.status 2024-04-06T10:05:30.850 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 1 2024-04-06T10:05:30.850 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 1 2024-04-06T10:05:30.850 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T10:05:30.850 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=5, tag_tid=2, entry_tid=1], [object_number=4, tag_tid=2, entry_tid=0], [object_number=7, tag_tid=1, entry_tid=7807], [object_number=6, tag_tid=1, entry_tid=7806]]], state=connected] 2024-04-06T10:05:30.850 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=5, tag_tid=1, entry_tid=7809], [object_number=4, tag_tid=1, entry_tid=7808], [object_number=7, tag_tid=1, entry_tid=7807], [object_number=6, tag_tid=1, entry_tid=7806]]], state=connected] 2024-04-06T10:05:30.853 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_36.status 2024-04-06T10:05:30.855 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=5, tag_tid=1, entry_tid=7809' 2024-04-06T10:05:30.855 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=5, tag_tid=2, entry_tid=1' -a 'object_number=5, tag_tid=2, entry_tid=1' = 'object_number=5, tag_tid=1, entry_tid=7809' 2024-04-06T10:05:30.855 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=5, tag_tid=1, entry_tid=7809' '!=' '' 2024-04-06T10:05:30.855 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T10:05:30.855 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=5, tag_tid=1, entry_tid=7809' = '' 2024-04-06T10:05:30.855 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=5, tag_tid=1, entry_tid=7809' 2024-04-06T10:05:30.856 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=5, tag_tid=2, entry_tid=1' 2024-04-06T10:05:30.856 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T10:05:30.856 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T10:05:30.857 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=2 2024-04-06T10:05:30.857 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=5, tag_tid=1, entry_tid=7809' 2024-04-06T10:05:30.857 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T10:05:30.857 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T10:05:30.858 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=1 2024-04-06T10:05:30.859 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=5, tag_tid=2, entry_tid=1' 2024-04-06T10:05:30.859 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T10:05:30.859 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T10:05:30.860 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T10:05:30.861 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=5, tag_tid=1, entry_tid=7809' 2024-04-06T10:05:30.861 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T10:05:30.861 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T10:05:30.862 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=7809 2024-04-06T10:05:30.862 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 2 = 1 -a 1 -le 7809 2024-04-06T10:05:30.862 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T10:05:30.862 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T10:05:30.862 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T10:05:31.064 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror image_36 2024-04-06T10:05:31.065 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T10:05:31.065 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:05:31.065 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_36 2024-04-06T10:05:31.065 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T10:05:31.065 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n image_36 ']' 2024-04-06T10:05:31.065 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/image_36' 2024-04-06T10:05:31.065 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_36 2024-04-06T10:05:31.065 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T10:05:31.065 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T10:05:31.065 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T10:05:31.065 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T10:05:31.065 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T10:05:31.066 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T10:05:31.066 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T10:05:31.066 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T10:05:31.069 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T10:05:31.069 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T10:05:31.069 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_36 2024-04-06T10:05:31.069 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T10:05:31.069 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T10:05:31.070 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T10:05:31.070 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T10:05:31.070 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T10:05:31.070 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T10:05:31.070 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T10:05:31.070 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T10:05:31.070 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T10:05:31.070 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T10:05:31.070 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T10:05:31.070 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:05:31.070 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T10:05:31.070 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T10:05:31.070 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T10:05:31.070 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T10:05:31.071 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T10:05:31.071 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T10:05:31.071 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T10:05:31.071 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T10:05:31.071 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T10:05:31.071 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T10:05:31.071 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:05:31.071 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T10:05:31.071 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T10:05:31.071 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T10:05:31.071 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T10:05:31.071 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T10:05:31.071 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T10:05:31.071 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T10:05:31.094 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T10:05:31.094 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T10:05:31.094 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T10:05:31.095 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_36 2024-04-06T10:05:31.200 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:Invalid command: unused arguments: ['mirror/image_36'] 2024-04-06T10:05:31.200 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:rbd mirror flush : flush rbd mirror 2024-04-06T10:05:31.200 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:admin_socket: invalid command 2024-04-06T10:05:31.204 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T10:05:31.204 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/image_36 2024-04-06T10:05:31.204 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:1 2024-04-06T10:05:31.205 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T10:05:31.205 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T10:05:31.205 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-06T10:05:31.205 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:1 2024-04-06T10:05:31.205 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T10:05:31.205 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T10:05:31.205 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T10:05:31.205 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=1 2024-04-06T10:05:31.205 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-06T10:05:31.205 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T10:05:31.205 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:05:31.205 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=1 2024-04-06T10:05:31.205 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=1 2024-04-06T10:05:31.205 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-06T10:05:31.205 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:1 2024-04-06T10:05:31.206 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T10:05:31.206 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T10:05:31.206 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-06T10:05:31.206 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:1 2024-04-06T10:05:31.206 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T10:05:31.206 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T10:05:31.206 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:05:31.206 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=1 2024-04-06T10:05:31.206 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-06T10:05:31.206 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T10:05:31.206 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T10:05:31.206 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=1 2024-04-06T10:05:31.206 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=1 2024-04-06T10:05:31.206 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-06T10:05:31.228 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T10:05:31.228 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T10:05:31.228 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T10:05:31.228 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/image_36 2024-04-06T10:05:31.365 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T10:05:31.366 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror image_36 2024-04-06T10:05:31.367 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:05:31.367 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:05:31.367 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_36 2024-04-06T10:05:31.367 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror image_36 '' 2024-04-06T10:05:31.367 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:05:31.367 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:05:31.367 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_36 2024-04-06T10:05:31.367 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T10:05:31.367 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-image_36.status 2024-04-06T10:05:31.368 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-image_36.status 2024-04-06T10:05:31.368 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T10:05:31.370 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-image_36.status 2024-04-06T10:05:31.370 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_36 2024-04-06T10:05:31.370 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_36.status 2024-04-06T10:05:31.415 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 1 2024-04-06T10:05:31.415 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 1 2024-04-06T10:05:31.415 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T10:05:31.415 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=5, tag_tid=2, entry_tid=1], [object_number=4, tag_tid=2, entry_tid=0], [object_number=7, tag_tid=1, entry_tid=7807], [object_number=6, tag_tid=1, entry_tid=7806]]], state=connected] 2024-04-06T10:05:31.415 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=5, tag_tid=1, entry_tid=7809], [object_number=4, tag_tid=1, entry_tid=7808], [object_number=7, tag_tid=1, entry_tid=7807], [object_number=6, tag_tid=1, entry_tid=7806]]], state=connected] 2024-04-06T10:05:31.419 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_36.status 2024-04-06T10:05:31.420 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=5, tag_tid=2, entry_tid=1' 2024-04-06T10:05:31.421 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror image_36 2024-04-06T10:05:31.421 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:05:31.421 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:05:31.421 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_36 2024-04-06T10:05:31.421 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror image_36 '..*' 2024-04-06T10:05:31.421 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:05:31.421 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:05:31.421 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_36 2024-04-06T10:05:31.422 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T10:05:31.422 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-image_36.status 2024-04-06T10:05:31.422 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-image_36.status 2024-04-06T10:05:31.422 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T10:05:31.422 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-image_36.status 2024-04-06T10:05:31.422 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_36 2024-04-06T10:05:31.423 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_36.status 2024-04-06T10:05:31.467 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 1 2024-04-06T10:05:31.467 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 1 2024-04-06T10:05:31.467 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T10:05:31.467 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=5, tag_tid=2, entry_tid=1], [object_number=4, tag_tid=2, entry_tid=0], [object_number=7, tag_tid=1, entry_tid=7807], [object_number=6, tag_tid=1, entry_tid=7806]]], state=connected] 2024-04-06T10:05:31.467 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=5, tag_tid=1, entry_tid=7809], [object_number=4, tag_tid=1, entry_tid=7808], [object_number=7, tag_tid=1, entry_tid=7807], [object_number=6, tag_tid=1, entry_tid=7806]]], state=connected] 2024-04-06T10:05:31.471 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_36.status 2024-04-06T10:05:31.472 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=5, tag_tid=1, entry_tid=7809' 2024-04-06T10:05:31.473 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=5, tag_tid=2, entry_tid=1' -a 'object_number=5, tag_tid=2, entry_tid=1' = 'object_number=5, tag_tid=1, entry_tid=7809' 2024-04-06T10:05:31.473 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=5, tag_tid=1, entry_tid=7809' '!=' 'object_number=5, tag_tid=1, entry_tid=7809' 2024-04-06T10:05:31.473 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T10:05:31.473 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.4 2024-04-06T10:05:31.875 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror image_36 2024-04-06T10:05:31.875 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T10:05:31.875 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:05:31.876 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_36 2024-04-06T10:05:31.876 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T10:05:31.876 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n image_36 ']' 2024-04-06T10:05:31.876 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/image_36' 2024-04-06T10:05:31.876 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_36 2024-04-06T10:05:31.876 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T10:05:31.876 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T10:05:31.876 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T10:05:31.876 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T10:05:31.876 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T10:05:31.876 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T10:05:31.876 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T10:05:31.877 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T10:05:31.877 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T10:05:31.879 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T10:05:31.879 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_36 2024-04-06T10:05:31.879 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T10:05:31.879 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T10:05:31.879 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T10:05:31.879 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T10:05:31.879 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T10:05:31.879 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T10:05:31.880 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T10:05:31.880 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T10:05:31.880 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T10:05:31.880 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T10:05:31.880 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T10:05:31.880 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:05:31.880 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T10:05:31.880 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T10:05:31.880 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T10:05:31.881 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T10:05:31.881 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T10:05:31.882 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T10:05:31.882 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T10:05:31.882 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T10:05:31.882 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T10:05:31.882 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T10:05:31.882 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:05:31.882 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T10:05:31.882 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T10:05:31.882 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T10:05:31.882 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T10:05:31.882 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T10:05:31.882 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T10:05:31.882 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T10:05:31.904 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T10:05:31.905 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T10:05:31.905 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T10:05:31.905 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_36 2024-04-06T10:05:32.011 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:Invalid command: unused arguments: ['mirror/image_36'] 2024-04-06T10:05:32.011 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:rbd mirror flush : flush rbd mirror 2024-04-06T10:05:32.011 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:admin_socket: invalid command 2024-04-06T10:05:32.013 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T10:05:32.014 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/image_36 2024-04-06T10:05:32.014 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:1 2024-04-06T10:05:32.014 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T10:05:32.014 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T10:05:32.014 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-06T10:05:32.014 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:1 2024-04-06T10:05:32.014 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T10:05:32.014 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T10:05:32.014 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T10:05:32.014 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=1 2024-04-06T10:05:32.014 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-06T10:05:32.014 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T10:05:32.014 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:05:32.014 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=1 2024-04-06T10:05:32.015 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=1 2024-04-06T10:05:32.016 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-06T10:05:32.016 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:1 2024-04-06T10:05:32.016 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T10:05:32.016 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T10:05:32.016 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-06T10:05:32.016 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:1 2024-04-06T10:05:32.016 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T10:05:32.016 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T10:05:32.016 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:05:32.016 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=1 2024-04-06T10:05:32.017 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-06T10:05:32.017 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T10:05:32.017 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T10:05:32.017 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=1 2024-04-06T10:05:32.017 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=1 2024-04-06T10:05:32.017 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-06T10:05:32.035 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T10:05:32.035 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T10:05:32.035 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T10:05:32.035 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/image_36 2024-04-06T10:05:32.180 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T10:05:32.182 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror image_36 2024-04-06T10:05:32.182 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:05:32.182 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:05:32.182 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_36 2024-04-06T10:05:32.182 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror image_36 '' 2024-04-06T10:05:32.182 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:05:32.182 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:05:32.182 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_36 2024-04-06T10:05:32.183 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T10:05:32.183 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-image_36.status 2024-04-06T10:05:32.183 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-image_36.status 2024-04-06T10:05:32.183 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T10:05:32.184 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-image_36.status 2024-04-06T10:05:32.184 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_36 2024-04-06T10:05:32.185 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_36.status 2024-04-06T10:05:32.233 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 1 2024-04-06T10:05:32.234 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 1 2024-04-06T10:05:32.234 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T10:05:32.234 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=5, tag_tid=2, entry_tid=1], [object_number=4, tag_tid=2, entry_tid=0], [object_number=7, tag_tid=1, entry_tid=7807], [object_number=6, tag_tid=1, entry_tid=7806]]], state=connected] 2024-04-06T10:05:32.234 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=5, tag_tid=1, entry_tid=7809], [object_number=4, tag_tid=1, entry_tid=7808], [object_number=7, tag_tid=1, entry_tid=7807], [object_number=6, tag_tid=1, entry_tid=7806]]], state=connected] 2024-04-06T10:05:32.237 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_36.status 2024-04-06T10:05:32.238 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=5, tag_tid=2, entry_tid=1' 2024-04-06T10:05:32.240 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror image_36 2024-04-06T10:05:32.240 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:05:32.240 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:05:32.240 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_36 2024-04-06T10:05:32.240 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror image_36 '..*' 2024-04-06T10:05:32.240 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:05:32.240 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:05:32.240 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_36 2024-04-06T10:05:32.240 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T10:05:32.240 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-image_36.status 2024-04-06T10:05:32.240 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-image_36.status 2024-04-06T10:05:32.240 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T10:05:32.241 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-image_36.status 2024-04-06T10:05:32.241 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_36 2024-04-06T10:05:32.241 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_36.status 2024-04-06T10:05:32.285 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 1 2024-04-06T10:05:32.286 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 1 2024-04-06T10:05:32.286 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T10:05:32.286 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=5, tag_tid=2, entry_tid=1], [object_number=4, tag_tid=2, entry_tid=0], [object_number=7, tag_tid=1, entry_tid=7807], [object_number=6, tag_tid=1, entry_tid=7806]]], state=connected] 2024-04-06T10:05:32.286 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=5, tag_tid=1, entry_tid=7809], [object_number=4, tag_tid=1, entry_tid=7808], [object_number=7, tag_tid=1, entry_tid=7807], [object_number=6, tag_tid=1, entry_tid=7806]]], state=connected] 2024-04-06T10:05:32.289 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_36.status 2024-04-06T10:05:32.291 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=5, tag_tid=1, entry_tid=7809' 2024-04-06T10:05:32.291 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=5, tag_tid=2, entry_tid=1' -a 'object_number=5, tag_tid=2, entry_tid=1' = 'object_number=5, tag_tid=1, entry_tid=7809' 2024-04-06T10:05:32.291 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=5, tag_tid=1, entry_tid=7809' '!=' 'object_number=5, tag_tid=1, entry_tid=7809' 2024-04-06T10:05:32.291 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T10:05:32.291 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.8 2024-04-06T10:05:33.093 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror image_36 2024-04-06T10:05:33.093 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T10:05:33.093 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:05:33.094 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_36 2024-04-06T10:05:33.094 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T10:05:33.094 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n image_36 ']' 2024-04-06T10:05:33.094 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/image_36' 2024-04-06T10:05:33.094 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_36 2024-04-06T10:05:33.094 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T10:05:33.094 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T10:05:33.094 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T10:05:33.094 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T10:05:33.094 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T10:05:33.094 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T10:05:33.094 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T10:05:33.097 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T10:05:33.097 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T10:05:33.097 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T10:05:33.097 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_36 2024-04-06T10:05:33.097 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T10:05:33.097 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T10:05:33.097 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T10:05:33.097 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T10:05:33.097 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T10:05:33.097 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T10:05:33.097 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T10:05:33.097 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T10:05:33.097 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T10:05:33.098 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T10:05:33.098 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T10:05:33.098 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:05:33.098 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T10:05:33.098 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T10:05:33.098 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T10:05:33.098 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T10:05:33.098 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T10:05:33.098 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T10:05:33.098 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T10:05:33.098 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T10:05:33.098 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T10:05:33.099 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T10:05:33.099 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:05:33.099 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T10:05:33.099 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T10:05:33.099 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T10:05:33.100 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T10:05:33.100 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T10:05:33.100 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T10:05:33.100 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T10:05:33.123 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T10:05:33.123 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T10:05:33.123 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T10:05:33.123 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_36 2024-04-06T10:05:33.227 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:Invalid command: unused arguments: ['mirror/image_36'] 2024-04-06T10:05:33.228 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:rbd mirror flush : flush rbd mirror 2024-04-06T10:05:33.228 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:admin_socket: invalid command 2024-04-06T10:05:33.230 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T10:05:33.230 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/image_36 2024-04-06T10:05:33.230 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:1 2024-04-06T10:05:33.230 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T10:05:33.230 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T10:05:33.230 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-06T10:05:33.230 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:1 2024-04-06T10:05:33.230 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T10:05:33.231 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T10:05:33.231 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T10:05:33.231 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=1 2024-04-06T10:05:33.231 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-06T10:05:33.231 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T10:05:33.231 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:05:33.231 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=1 2024-04-06T10:05:33.231 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=1 2024-04-06T10:05:33.232 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-06T10:05:33.233 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:1 2024-04-06T10:05:33.233 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T10:05:33.233 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T10:05:33.233 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-06T10:05:33.233 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:1 2024-04-06T10:05:33.233 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T10:05:33.233 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T10:05:33.233 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:05:33.233 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=1 2024-04-06T10:05:33.233 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-06T10:05:33.233 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T10:05:33.233 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T10:05:33.233 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=1 2024-04-06T10:05:33.233 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=1 2024-04-06T10:05:33.233 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-06T10:05:33.256 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T10:05:33.257 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T10:05:33.257 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T10:05:33.257 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/image_36 2024-04-06T10:05:33.397 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T10:05:33.397 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror image_36 2024-04-06T10:05:33.398 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:05:33.398 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:05:33.398 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_36 2024-04-06T10:05:33.398 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror image_36 '' 2024-04-06T10:05:33.398 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:05:33.398 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:05:33.398 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_36 2024-04-06T10:05:33.398 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T10:05:33.399 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-image_36.status 2024-04-06T10:05:33.399 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-image_36.status 2024-04-06T10:05:33.399 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T10:05:33.402 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-image_36.status 2024-04-06T10:05:33.402 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_36 2024-04-06T10:05:33.402 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_36.status 2024-04-06T10:05:33.456 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 1 2024-04-06T10:05:33.456 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 1 2024-04-06T10:05:33.456 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T10:05:33.457 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=5, tag_tid=2, entry_tid=1], [object_number=4, tag_tid=2, entry_tid=0], [object_number=7, tag_tid=1, entry_tid=7807], [object_number=6, tag_tid=1, entry_tid=7806]]], state=connected] 2024-04-06T10:05:33.457 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=5, tag_tid=1, entry_tid=7809], [object_number=4, tag_tid=1, entry_tid=7808], [object_number=7, tag_tid=1, entry_tid=7807], [object_number=6, tag_tid=1, entry_tid=7806]]], state=connected] 2024-04-06T10:05:33.460 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_36.status 2024-04-06T10:05:33.461 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=5, tag_tid=2, entry_tid=1' 2024-04-06T10:05:33.462 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror image_36 2024-04-06T10:05:33.463 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:05:33.463 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:05:33.463 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_36 2024-04-06T10:05:33.463 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror image_36 '..*' 2024-04-06T10:05:33.463 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:05:33.463 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:05:33.463 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_36 2024-04-06T10:05:33.463 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T10:05:33.463 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-image_36.status 2024-04-06T10:05:33.463 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-image_36.status 2024-04-06T10:05:33.463 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T10:05:33.464 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-image_36.status 2024-04-06T10:05:33.464 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_36 2024-04-06T10:05:33.464 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_36.status 2024-04-06T10:05:33.509 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 1 2024-04-06T10:05:33.509 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 1 2024-04-06T10:05:33.510 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T10:05:33.510 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=5, tag_tid=2, entry_tid=1], [object_number=4, tag_tid=2, entry_tid=0], [object_number=7, tag_tid=1, entry_tid=7807], [object_number=6, tag_tid=1, entry_tid=7806]]], state=connected] 2024-04-06T10:05:33.510 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=5, tag_tid=1, entry_tid=7809], [object_number=4, tag_tid=1, entry_tid=7808], [object_number=7, tag_tid=1, entry_tid=7807], [object_number=6, tag_tid=1, entry_tid=7806]]], state=connected] 2024-04-06T10:05:33.513 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_36.status 2024-04-06T10:05:33.515 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=5, tag_tid=1, entry_tid=7809' 2024-04-06T10:05:33.515 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=5, tag_tid=2, entry_tid=1' -a 'object_number=5, tag_tid=2, entry_tid=1' = 'object_number=5, tag_tid=1, entry_tid=7809' 2024-04-06T10:05:33.515 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=5, tag_tid=1, entry_tid=7809' '!=' 'object_number=5, tag_tid=1, entry_tid=7809' 2024-04-06T10:05:33.515 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T10:05:33.516 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 1.6 2024-04-06T10:05:33.669 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.0 with skip (0.6208898599677426) > weight (0.25) 2024-04-06T10:05:33.669 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.1 with skip (0.38491353824056485) > weight (0.25) 2024-04-06T10:05:33.669 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.2 with skip (0.8768364858042811) > weight (0.25) 2024-04-06T10:05:33.669 INFO:tasks.rbd_mirror_thrash:kill cluster1.client.mirror.3 2024-04-06T10:05:33.669 INFO:tasks.rbd_mirror.cluster1.client.mirror.3:Sent signal 15 2024-04-06T10:05:33.670 INFO:tasks.rbd_mirror_thrash:waiting for 6 secs before reviving daemons 2024-04-06T10:05:33.671 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi192.stderr:2024-04-06T10:05:33.672+0000 7f302a547640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.3 (PID: 132024) UID: 1000 2024-04-06T10:05:35.117 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror image_36 2024-04-06T10:05:35.117 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T10:05:35.117 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:05:35.117 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_36 2024-04-06T10:05:35.118 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T10:05:35.118 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n image_36 ']' 2024-04-06T10:05:35.118 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/image_36' 2024-04-06T10:05:35.118 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_36 2024-04-06T10:05:35.118 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T10:05:35.118 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T10:05:35.118 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T10:05:35.118 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T10:05:35.118 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T10:05:35.118 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T10:05:35.118 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T10:05:35.119 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T10:05:35.119 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T10:05:35.123 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T10:05:35.123 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_36 2024-04-06T10:05:35.123 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T10:05:35.123 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T10:05:35.123 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T10:05:35.123 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T10:05:35.123 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T10:05:35.124 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T10:05:35.124 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T10:05:35.124 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T10:05:35.124 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T10:05:35.124 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T10:05:35.124 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T10:05:35.124 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:05:35.124 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T10:05:35.124 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T10:05:35.124 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T10:05:35.124 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T10:05:35.124 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T10:05:35.124 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T10:05:35.124 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T10:05:35.124 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T10:05:35.125 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T10:05:35.125 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T10:05:35.125 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:05:35.125 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T10:05:35.125 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T10:05:35.125 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T10:05:35.125 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T10:05:35.125 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T10:05:35.125 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T10:05:35.125 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T10:05:35.148 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T10:05:35.148 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T10:05:35.148 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T10:05:35.148 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_36 2024-04-06T10:05:35.253 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:Invalid command: unused arguments: ['mirror/image_36'] 2024-04-06T10:05:35.254 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:rbd mirror flush : flush rbd mirror 2024-04-06T10:05:35.254 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:admin_socket: invalid command 2024-04-06T10:05:35.256 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T10:05:35.256 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/image_36 2024-04-06T10:05:35.256 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:1 2024-04-06T10:05:35.256 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T10:05:35.256 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T10:05:35.256 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-06T10:05:35.256 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:1 2024-04-06T10:05:35.256 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T10:05:35.257 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T10:05:35.257 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T10:05:35.257 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=1 2024-04-06T10:05:35.257 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-06T10:05:35.257 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T10:05:35.257 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:05:35.257 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=1 2024-04-06T10:05:35.257 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=1 2024-04-06T10:05:35.258 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-06T10:05:35.259 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:1 2024-04-06T10:05:35.259 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T10:05:35.259 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T10:05:35.259 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-06T10:05:35.259 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:1 2024-04-06T10:05:35.259 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T10:05:35.259 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T10:05:35.259 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:05:35.259 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=1 2024-04-06T10:05:35.259 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-06T10:05:35.259 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T10:05:35.259 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T10:05:35.259 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=1 2024-04-06T10:05:35.260 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=1 2024-04-06T10:05:35.260 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-06T10:05:35.277 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T10:05:35.277 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T10:05:35.277 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T10:05:35.277 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/image_36 2024-04-06T10:05:35.419 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T10:05:35.420 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror image_36 2024-04-06T10:05:35.420 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:05:35.421 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:05:35.421 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_36 2024-04-06T10:05:35.421 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror image_36 '' 2024-04-06T10:05:35.421 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:05:35.421 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:05:35.421 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_36 2024-04-06T10:05:35.421 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T10:05:35.421 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-image_36.status 2024-04-06T10:05:35.422 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-image_36.status 2024-04-06T10:05:35.422 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T10:05:35.424 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-image_36.status 2024-04-06T10:05:35.425 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_36 2024-04-06T10:05:35.425 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_36.status 2024-04-06T10:05:35.480 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 1 2024-04-06T10:05:35.480 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 1 2024-04-06T10:05:35.480 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T10:05:35.480 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=5, tag_tid=2, entry_tid=1], [object_number=4, tag_tid=2, entry_tid=0], [object_number=7, tag_tid=1, entry_tid=7807], [object_number=6, tag_tid=1, entry_tid=7806]]], state=connected] 2024-04-06T10:05:35.480 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=5, tag_tid=1, entry_tid=7809], [object_number=4, tag_tid=1, entry_tid=7808], [object_number=7, tag_tid=1, entry_tid=7807], [object_number=6, tag_tid=1, entry_tid=7806]]], state=connected] 2024-04-06T10:05:35.483 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_36.status 2024-04-06T10:05:35.486 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=5, tag_tid=2, entry_tid=1' 2024-04-06T10:05:35.486 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror image_36 2024-04-06T10:05:35.486 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:05:35.486 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:05:35.486 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_36 2024-04-06T10:05:35.486 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror image_36 '..*' 2024-04-06T10:05:35.486 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:05:35.486 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:05:35.486 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_36 2024-04-06T10:05:35.487 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T10:05:35.487 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-image_36.status 2024-04-06T10:05:35.487 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-image_36.status 2024-04-06T10:05:35.487 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T10:05:35.487 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-image_36.status 2024-04-06T10:05:35.487 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_36 2024-04-06T10:05:35.488 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_36.status 2024-04-06T10:05:35.534 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 1 2024-04-06T10:05:35.534 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 1 2024-04-06T10:05:35.534 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T10:05:35.534 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=5, tag_tid=2, entry_tid=1], [object_number=4, tag_tid=2, entry_tid=0], [object_number=7, tag_tid=1, entry_tid=7807], [object_number=6, tag_tid=1, entry_tid=7806]]], state=connected] 2024-04-06T10:05:35.534 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=5, tag_tid=1, entry_tid=7809], [object_number=4, tag_tid=1, entry_tid=7808], [object_number=7, tag_tid=1, entry_tid=7807], [object_number=6, tag_tid=1, entry_tid=7806]]], state=connected] 2024-04-06T10:05:35.536 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_36.status 2024-04-06T10:05:35.538 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=5, tag_tid=1, entry_tid=7809' 2024-04-06T10:05:35.538 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=5, tag_tid=2, entry_tid=1' -a 'object_number=5, tag_tid=2, entry_tid=1' = 'object_number=5, tag_tid=1, entry_tid=7809' 2024-04-06T10:05:35.538 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=5, tag_tid=1, entry_tid=7809' '!=' 'object_number=5, tag_tid=1, entry_tid=7809' 2024-04-06T10:05:35.538 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T10:05:35.539 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 2 2024-04-06T10:05:37.540 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror image_36 2024-04-06T10:05:37.540 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T10:05:37.540 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:05:37.540 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_36 2024-04-06T10:05:37.540 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T10:05:37.541 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n image_36 ']' 2024-04-06T10:05:37.541 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/image_36' 2024-04-06T10:05:37.541 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_36 2024-04-06T10:05:37.541 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T10:05:37.541 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T10:05:37.541 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T10:05:37.541 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T10:05:37.541 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T10:05:37.541 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T10:05:37.541 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T10:05:37.542 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T10:05:37.542 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T10:05:37.545 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T10:05:37.546 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_36 2024-04-06T10:05:37.546 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T10:05:37.546 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T10:05:37.546 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T10:05:37.546 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T10:05:37.546 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T10:05:37.546 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T10:05:37.546 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T10:05:37.546 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T10:05:37.546 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T10:05:37.547 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T10:05:37.547 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T10:05:37.547 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:05:37.547 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T10:05:37.547 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T10:05:37.547 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T10:05:37.547 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T10:05:37.547 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T10:05:37.547 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T10:05:37.547 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T10:05:37.547 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T10:05:37.547 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T10:05:37.547 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T10:05:37.547 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:05:37.548 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T10:05:37.548 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T10:05:37.548 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T10:05:37.548 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T10:05:37.548 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T10:05:37.548 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T10:05:37.548 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T10:05:37.570 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T10:05:37.571 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T10:05:37.571 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T10:05:37.571 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_36 2024-04-06T10:05:37.677 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:Invalid command: unused arguments: ['mirror/image_36'] 2024-04-06T10:05:37.677 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:rbd mirror flush : flush rbd mirror 2024-04-06T10:05:37.677 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:admin_socket: invalid command 2024-04-06T10:05:37.680 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T10:05:37.680 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/image_36 2024-04-06T10:05:37.680 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:1 2024-04-06T10:05:37.680 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T10:05:37.680 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T10:05:37.680 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-06T10:05:37.680 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:1 2024-04-06T10:05:37.680 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T10:05:37.680 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T10:05:37.680 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T10:05:37.680 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=1 2024-04-06T10:05:37.681 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-06T10:05:37.681 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T10:05:37.681 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:05:37.681 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=1 2024-04-06T10:05:37.681 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=1 2024-04-06T10:05:37.682 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-06T10:05:37.682 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:1 2024-04-06T10:05:37.682 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T10:05:37.682 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T10:05:37.682 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-06T10:05:37.682 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:1 2024-04-06T10:05:37.683 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T10:05:37.683 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T10:05:37.683 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:05:37.683 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=1 2024-04-06T10:05:37.683 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-06T10:05:37.683 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T10:05:37.683 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T10:05:37.683 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=1 2024-04-06T10:05:37.683 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=1 2024-04-06T10:05:37.683 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-06T10:05:37.700 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T10:05:37.701 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T10:05:37.701 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T10:05:37.701 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/image_36 2024-04-06T10:05:37.839 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T10:05:37.840 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror image_36 2024-04-06T10:05:37.841 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:05:37.841 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:05:37.841 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_36 2024-04-06T10:05:37.841 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror image_36 '' 2024-04-06T10:05:37.841 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:05:37.841 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:05:37.841 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_36 2024-04-06T10:05:37.841 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T10:05:37.841 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-image_36.status 2024-04-06T10:05:37.842 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T10:05:37.842 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-image_36.status 2024-04-06T10:05:37.843 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-image_36.status 2024-04-06T10:05:37.844 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_36 2024-04-06T10:05:37.844 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_36.status 2024-04-06T10:05:37.893 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 1 2024-04-06T10:05:37.893 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 1 2024-04-06T10:05:37.893 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T10:05:37.893 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=5, tag_tid=2, entry_tid=1], [object_number=4, tag_tid=2, entry_tid=0], [object_number=7, tag_tid=1, entry_tid=7807], [object_number=6, tag_tid=1, entry_tid=7806]]], state=connected] 2024-04-06T10:05:37.894 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=5, tag_tid=1, entry_tid=7809], [object_number=4, tag_tid=1, entry_tid=7808], [object_number=7, tag_tid=1, entry_tid=7807], [object_number=6, tag_tid=1, entry_tid=7806]]], state=connected] 2024-04-06T10:05:37.897 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_36.status 2024-04-06T10:05:37.898 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=5, tag_tid=2, entry_tid=1' 2024-04-06T10:05:37.899 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror image_36 2024-04-06T10:05:37.899 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:05:37.899 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:05:37.900 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_36 2024-04-06T10:05:37.900 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror image_36 '..*' 2024-04-06T10:05:37.900 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:05:37.900 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:05:37.900 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_36 2024-04-06T10:05:37.900 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T10:05:37.900 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-image_36.status 2024-04-06T10:05:37.900 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-image_36.status 2024-04-06T10:05:37.900 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T10:05:37.901 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-image_36.status 2024-04-06T10:05:37.901 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_36 2024-04-06T10:05:37.901 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_36.status 2024-04-06T10:05:37.946 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 1 2024-04-06T10:05:37.946 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 1 2024-04-06T10:05:37.946 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T10:05:37.946 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=5, tag_tid=2, entry_tid=1], [object_number=4, tag_tid=2, entry_tid=0], [object_number=7, tag_tid=1, entry_tid=7807], [object_number=6, tag_tid=1, entry_tid=7806]]], state=connected] 2024-04-06T10:05:37.946 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=5, tag_tid=1, entry_tid=7809], [object_number=4, tag_tid=1, entry_tid=7808], [object_number=7, tag_tid=1, entry_tid=7807], [object_number=6, tag_tid=1, entry_tid=7806]]], state=connected] 2024-04-06T10:05:37.949 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_36.status 2024-04-06T10:05:37.951 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=5, tag_tid=1, entry_tid=7809' 2024-04-06T10:05:37.951 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=5, tag_tid=2, entry_tid=1' -a 'object_number=5, tag_tid=2, entry_tid=1' = 'object_number=5, tag_tid=1, entry_tid=7809' 2024-04-06T10:05:37.951 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=5, tag_tid=1, entry_tid=7809' '!=' 'object_number=5, tag_tid=1, entry_tid=7809' 2024-04-06T10:05:37.951 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T10:05:37.951 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 2 2024-04-06T10:05:39.670 INFO:tasks.rbd_mirror_thrash:waiting for cluster1.client.mirror.3 2024-04-06T10:05:39.670 INFO:teuthology.orchestra.run:waiting for 600 2024-04-06T10:05:39.671 INFO:tasks.rbd_mirror_thrash:reviving cluster1.client.mirror.3 2024-04-06T10:05:39.671 INFO:tasks.rbd_mirror.cluster1.client.mirror.3:Restarting daemon 2024-04-06T10:05:39.671 DEBUG:teuthology.orchestra.run.smithi192:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.3 2024-04-06T10:05:39.674 INFO:tasks.rbd_mirror.cluster1.client.mirror.3:Started 2024-04-06T10:05:39.674 INFO:tasks.rbd_mirror_thrash:waiting for 68 secs before thrashing 2024-04-06T10:05:39.952 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror image_36 2024-04-06T10:05:39.952 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T10:05:39.952 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:05:39.953 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_36 2024-04-06T10:05:39.953 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T10:05:39.953 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n image_36 ']' 2024-04-06T10:05:39.953 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/image_36' 2024-04-06T10:05:39.953 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_36 2024-04-06T10:05:39.953 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T10:05:39.953 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T10:05:39.953 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T10:05:39.953 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T10:05:39.953 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T10:05:39.953 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T10:05:39.953 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T10:05:39.956 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T10:05:39.956 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T10:05:39.956 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T10:05:39.956 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_36 2024-04-06T10:05:39.956 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T10:05:39.956 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T10:05:39.956 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T10:05:39.956 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T10:05:39.956 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T10:05:39.956 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T10:05:39.956 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T10:05:39.957 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T10:05:39.957 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T10:05:39.957 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T10:05:39.957 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T10:05:39.957 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:05:39.957 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T10:05:39.957 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T10:05:39.957 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T10:05:39.957 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T10:05:39.957 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T10:05:39.957 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T10:05:39.957 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T10:05:39.957 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T10:05:39.957 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T10:05:39.958 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T10:05:39.958 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:05:39.958 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T10:05:39.958 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T10:05:39.958 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T10:05:39.958 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T10:05:39.958 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T10:05:39.958 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T10:05:39.958 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T10:05:39.976 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T10:05:39.976 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T10:05:39.976 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T10:05:39.976 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_36 2024-04-06T10:05:40.080 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:Invalid command: unused arguments: ['mirror/image_36'] 2024-04-06T10:05:40.080 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:rbd mirror flush : flush rbd mirror 2024-04-06T10:05:40.080 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:admin_socket: invalid command 2024-04-06T10:05:40.084 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T10:05:40.084 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/image_36 2024-04-06T10:05:40.084 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:1 2024-04-06T10:05:40.084 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T10:05:40.084 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T10:05:40.084 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-06T10:05:40.085 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:1 2024-04-06T10:05:40.085 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T10:05:40.085 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T10:05:40.085 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T10:05:40.085 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=1 2024-04-06T10:05:40.085 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-06T10:05:40.085 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T10:05:40.085 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:05:40.085 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=1 2024-04-06T10:05:40.085 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=1 2024-04-06T10:05:40.085 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-06T10:05:40.085 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:1 2024-04-06T10:05:40.085 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T10:05:40.085 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T10:05:40.086 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-06T10:05:40.086 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:1 2024-04-06T10:05:40.086 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T10:05:40.086 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T10:05:40.086 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:05:40.086 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=1 2024-04-06T10:05:40.086 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-06T10:05:40.086 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T10:05:40.086 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T10:05:40.086 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=1 2024-04-06T10:05:40.086 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=1 2024-04-06T10:05:40.086 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-06T10:05:40.102 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T10:05:40.103 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T10:05:40.103 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T10:05:40.103 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/image_36 2024-04-06T10:05:40.248 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T10:05:40.249 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror image_36 2024-04-06T10:05:40.249 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:05:40.249 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:05:40.249 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_36 2024-04-06T10:05:40.249 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror image_36 '' 2024-04-06T10:05:40.250 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:05:40.250 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:05:40.250 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_36 2024-04-06T10:05:40.250 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T10:05:40.250 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-image_36.status 2024-04-06T10:05:40.250 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T10:05:40.250 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-image_36.status 2024-04-06T10:05:40.251 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-image_36.status 2024-04-06T10:05:40.252 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_36 2024-04-06T10:05:40.252 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_36.status 2024-04-06T10:05:40.304 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 1 2024-04-06T10:05:40.305 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 1 2024-04-06T10:05:40.305 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T10:05:40.305 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=5, tag_tid=2, entry_tid=1], [object_number=4, tag_tid=2, entry_tid=0], [object_number=7, tag_tid=1, entry_tid=7807], [object_number=6, tag_tid=1, entry_tid=7806]]], state=connected] 2024-04-06T10:05:40.305 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=5, tag_tid=2, entry_tid=1], [object_number=4, tag_tid=2, entry_tid=0], [object_number=7, tag_tid=1, entry_tid=7807], [object_number=6, tag_tid=1, entry_tid=7806]]], state=connected] 2024-04-06T10:05:40.308 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_36.status 2024-04-06T10:05:40.309 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=5, tag_tid=2, entry_tid=1' 2024-04-06T10:05:40.311 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror image_36 2024-04-06T10:05:40.311 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:05:40.311 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:05:40.311 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_36 2024-04-06T10:05:40.311 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror image_36 '..*' 2024-04-06T10:05:40.311 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:05:40.311 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:05:40.311 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_36 2024-04-06T10:05:40.311 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T10:05:40.311 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-image_36.status 2024-04-06T10:05:40.312 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-image_36.status 2024-04-06T10:05:40.312 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T10:05:40.312 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-image_36.status 2024-04-06T10:05:40.312 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_36 2024-04-06T10:05:40.312 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_36.status 2024-04-06T10:05:40.357 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 1 2024-04-06T10:05:40.357 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 1 2024-04-06T10:05:40.357 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T10:05:40.357 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=5, tag_tid=2, entry_tid=1], [object_number=4, tag_tid=2, entry_tid=0], [object_number=7, tag_tid=1, entry_tid=7807], [object_number=6, tag_tid=1, entry_tid=7806]]], state=connected] 2024-04-06T10:05:40.357 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=5, tag_tid=2, entry_tid=1], [object_number=4, tag_tid=2, entry_tid=0], [object_number=7, tag_tid=1, entry_tid=7807], [object_number=6, tag_tid=1, entry_tid=7806]]], state=connected] 2024-04-06T10:05:40.360 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_36.status 2024-04-06T10:05:40.362 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=5, tag_tid=2, entry_tid=1' 2024-04-06T10:05:40.363 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=5, tag_tid=2, entry_tid=1' -a 'object_number=5, tag_tid=2, entry_tid=1' = 'object_number=5, tag_tid=2, entry_tid=1' 2024-04-06T10:05:40.363 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T10:05:40.363 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_snap_present cluster1 mirror image_36 snap 2024-04-06T10:05:40.363 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T10:05:40.363 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:05:40.363 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_36 2024-04-06T10:05:40.363 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local snap_name=snap 2024-04-06T10:05:40.363 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local s 2024-04-06T10:05:40.363 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-04-06T10:05:40.363 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 1 2024-04-06T10:05:41.363 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 info mirror/image_36@snap 2024-04-06T10:05:41.432 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:rbd image 'image_36': 2024-04-06T10:05:41.432 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: size 128 MiB in 32 objects 2024-04-06T10:05:41.432 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: order 22 (4 MiB objects) 2024-04-06T10:05:41.432 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: snapshot_count: 1 2024-04-06T10:05:41.432 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: id: 10acbc0ebab 2024-04-06T10:05:41.432 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: block_name_prefix: rbd_data.10acbc0ebab 2024-04-06T10:05:41.432 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: format: 2 2024-04-06T10:05:41.432 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: features: layering, exclusive-lock, journaling 2024-04-06T10:05:41.433 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: op_features: 2024-04-06T10:05:41.433 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: flags: 2024-04-06T10:05:41.433 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: create_timestamp: Sat Apr 6 09:56:04 2024 2024-04-06T10:05:41.433 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: access_timestamp: Sat Apr 6 09:56:04 2024 2024-04-06T10:05:41.433 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: modify_timestamp: Sat Apr 6 09:57:05 2024 2024-04-06T10:05:41.433 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: protected: False 2024-04-06T10:05:41.433 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: journal: 10acbc0ebab 2024-04-06T10:05:41.433 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: mirroring state: enabled 2024-04-06T10:05:41.433 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: mirroring mode: journal 2024-04-06T10:05:41.433 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: mirroring global id: db57671b-f795-4e2d-adf8-4fec11e77eb0 2024-04-06T10:05:41.433 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: mirroring primary: false 2024-04-06T10:05:41.437 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T10:05:41.437 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ compare_image_snaps mirror image_36 snap 2024-04-06T10:05:41.437 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:05:41.437 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_36 2024-04-06T10:05:41.437 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local snap_name=snap 2024-04-06T10:05:41.437 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local ret=0 2024-04-06T10:05:41.437 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local rmt_export=/tmp/tmp.BhW6xKmROU/cluster2-mirror-image_36.export 2024-04-06T10:05:41.437 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loc_export=/tmp/tmp.BhW6xKmROU/cluster1-mirror-image_36.export 2024-04-06T10:05:41.437 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rm -f /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_36.export /tmp/tmp.BhW6xKmROU/cluster1-mirror-image_36.export 2024-04-06T10:05:41.438 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster2 -p mirror export image_36@snap /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_36.export 2024-04-06T10:05:41.606 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-04-06T10:05:41.614 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 -p mirror export image_36@snap /tmp/tmp.BhW6xKmROU/cluster1-mirror-image_36.export 2024-04-06T10:05:41.968 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-04-06T10:05:41.981 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmp /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_36.export /tmp/tmp.BhW6xKmROU/cluster1-mirror-image_36.export 2024-04-06T10:05:42.060 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rm -f /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_36.export /tmp/tmp.BhW6xKmROU/cluster1-mirror-image_36.export 2024-04-06T10:05:42.101 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T10:05:42.101 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-06T10:05:42.101 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ image=image_37 2024-04-06T10:05:42.101 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ create_snap cluster2 mirror image_37 snap 2024-04-06T10:05:42.101 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-04-06T10:05:42.101 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:05:42.101 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_37 2024-04-06T10:05:42.101 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local snap_name=snap 2024-04-06T10:05:42.101 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster2 -p mirror snap create image_37@snap --debug-rbd=20 --debug-journaler=20 2024-04-06T10:05:42.922 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_image_replay_started cluster1 mirror image_37 2024-04-06T10:05:42.922 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T10:05:42.922 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:05:42.922 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_37 2024-04-06T10:05:42.922 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_image_replay_state cluster1 mirror image_37 started 2024-04-06T10:05:42.922 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T10:05:42.922 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:05:42.923 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_37 2024-04-06T10:05:42.923 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state=started 2024-04-06T10:05:42.923 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local s 2024-04-06T10:05:42.923 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T10:05:42.923 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 1 2024-04-06T10:05:43.925 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test_image_replay_state cluster1 mirror image_37 started 2024-04-06T10:05:43.927 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T10:05:43.927 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:05:43.927 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_37 2024-04-06T10:05:43.927 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local test_state=started 2024-04-06T10:05:43.927 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local status_result 2024-04-06T10:05:43.927 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local current_state=stopped 2024-04-06T10:05:43.927 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_37 2024-04-06T10:05:43.927 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -i state 2024-04-06T10:05:43.927 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_instance=cluster1 2024-04-06T10:05:43.927 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ shift 2024-04-06T10:05:43.927 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T10:05:43.927 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance=cluster1 2024-04-06T10:05:43.927 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local loop_instance 2024-04-06T10:05:43.928 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T10:05:43.928 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sleep 0 2024-04-06T10:05:43.928 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-04-06T10:05:43.930 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ seq 0 3 2024-04-06T10:05:43.930 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T10:05:43.930 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_37 2024-04-06T10:05:43.930 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1:0 2024-04-06T10:05:43.930 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ shift 2024-04-06T10:05:43.930 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T10:05:43.931 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-04-06T10:05:43.931 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T10:05:43.931 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=cluster 2024-04-06T10:05:43.931 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T10:05:43.931 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:05:43.931 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T10:05:43.931 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T10:05:43.931 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval cluster=cluster1 2024-04-06T10:05:43.931 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ cluster=cluster1 2024-04-06T10:05:43.931 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T10:05:43.931 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T10:05:43.931 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-04-06T10:05:43.931 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local local_cluster=cluster1:0 2024-04-06T10:05:43.931 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local cluster=cluster1 2024-04-06T10:05:43.932 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local instance 2024-04-06T10:05:43.932 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T10:05:43.932 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local val=cluster1:0 2024-04-06T10:05:43.932 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local cluster_var_name=local_cluster 2024-04-06T10:05:43.932 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local instance_var_name=instance 2024-04-06T10:05:43.932 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ cluster=cluster1 2024-04-06T10:05:43.932 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T10:05:43.932 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ '[' 0 = cluster1:0 ']' 2024-04-06T10:05:43.932 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ eval local_cluster=cluster1 2024-04-06T10:05:43.932 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++++ local_cluster=cluster1 2024-04-06T10:05:43.932 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ eval instance=0 2024-04-06T10:05:43.932 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++++ instance=0 2024-04-06T10:05:43.932 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T10:05:43.955 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T10:05:43.955 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T10:05:43.955 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T10:05:43.955 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_37 2024-04-06T10:05:44.059 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:Invalid command: unused arguments: ['mirror/image_37'] 2024-04-06T10:05:44.059 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:rbd mirror status : get status for rbd mirror 2024-04-06T10:05:44.059 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:admin_socket: invalid command 2024-04-06T10:05:44.062 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T10:05:44.062 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_37 2024-04-06T10:05:44.062 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1:1 2024-04-06T10:05:44.062 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ shift 2024-04-06T10:05:44.062 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T10:05:44.062 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-04-06T10:05:44.062 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:1 2024-04-06T10:05:44.062 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=cluster 2024-04-06T10:05:44.062 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T10:05:44.062 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:05:44.062 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=1 2024-04-06T10:05:44.063 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-06T10:05:44.063 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval cluster=cluster1 2024-04-06T10:05:44.063 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ cluster=cluster1 2024-04-06T10:05:44.063 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=1 2024-04-06T10:05:44.063 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=1 2024-04-06T10:05:44.064 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-04-06T10:05:44.064 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local local_cluster=cluster1:1 2024-04-06T10:05:44.064 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local cluster=cluster1 2024-04-06T10:05:44.064 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local instance 2024-04-06T10:05:44.064 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-06T10:05:44.065 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local val=cluster1:1 2024-04-06T10:05:44.065 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local cluster_var_name=local_cluster 2024-04-06T10:05:44.065 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local instance_var_name=instance 2024-04-06T10:05:44.065 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ cluster=cluster1 2024-04-06T10:05:44.065 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=1 2024-04-06T10:05:44.065 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ '[' 1 = cluster1:1 ']' 2024-04-06T10:05:44.065 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ eval local_cluster=cluster1 2024-04-06T10:05:44.065 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++++ local_cluster=cluster1 2024-04-06T10:05:44.065 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ eval instance=1 2024-04-06T10:05:44.065 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++++ instance=1 2024-04-06T10:05:44.065 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-06T10:05:44.088 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T10:05:44.088 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T10:05:44.088 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T10:05:44.088 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_37 2024-04-06T10:05:44.242 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ return 0 2024-04-06T10:05:44.243 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ status_result=' "state": "Replaying"' 2024-04-06T10:05:44.243 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ echo ' "state": "Replaying"' 2024-04-06T10:05:44.243 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep -i Replaying 2024-04-06T10:05:44.244 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: "state": "Replaying" 2024-04-06T10:05:44.246 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ current_state=started 2024-04-06T10:05:44.246 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test started = started 2024-04-06T10:05:44.246 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T10:05:44.246 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_37 2024-04-06T10:05:44.247 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local local_cluster=cluster1 2024-04-06T10:05:44.247 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-04-06T10:05:44.247 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:05:44.247 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_37 2024-04-06T10:05:44.247 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' journal = journal ']' 2024-04-06T10:05:44.247 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_journal_replay_complete cluster1 cluster2 mirror image_37 2024-04-06T10:05:44.247 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local local_cluster=cluster1 2024-04-06T10:05:44.247 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-04-06T10:05:44.247 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:05:44.247 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_37 2024-04-06T10:05:44.247 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local s master_pos mirror_pos last_mirror_pos 2024-04-06T10:05:44.247 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local master_tag master_entry mirror_tag mirror_entry 2024-04-06T10:05:44.247 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T10:05:44.247 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T10:05:44.247 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T10:05:44.448 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror image_37 2024-04-06T10:05:44.448 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T10:05:44.448 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:05:44.448 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_37 2024-04-06T10:05:44.448 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T10:05:44.448 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n image_37 ']' 2024-04-06T10:05:44.448 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/image_37' 2024-04-06T10:05:44.449 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_37 2024-04-06T10:05:44.449 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T10:05:44.449 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T10:05:44.449 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T10:05:44.449 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T10:05:44.449 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T10:05:44.449 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T10:05:44.449 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T10:05:44.450 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T10:05:44.450 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T10:05:44.453 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T10:05:44.453 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_37 2024-04-06T10:05:44.453 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T10:05:44.453 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T10:05:44.453 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T10:05:44.453 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T10:05:44.453 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T10:05:44.453 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T10:05:44.453 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T10:05:44.453 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T10:05:44.453 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T10:05:44.453 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T10:05:44.453 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T10:05:44.454 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:05:44.454 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T10:05:44.454 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T10:05:44.454 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T10:05:44.454 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T10:05:44.454 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T10:05:44.454 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T10:05:44.454 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T10:05:44.454 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T10:05:44.454 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T10:05:44.454 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T10:05:44.454 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:05:44.455 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T10:05:44.455 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T10:05:44.455 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T10:05:44.455 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T10:05:44.455 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T10:05:44.455 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T10:05:44.455 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T10:05:44.479 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T10:05:44.479 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T10:05:44.479 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T10:05:44.479 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_37 2024-04-06T10:05:44.584 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:Invalid command: unused arguments: ['mirror/image_37'] 2024-04-06T10:05:44.584 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:rbd mirror flush : flush rbd mirror 2024-04-06T10:05:44.584 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:admin_socket: invalid command 2024-04-06T10:05:44.586 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T10:05:44.586 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/image_37 2024-04-06T10:05:44.586 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:1 2024-04-06T10:05:44.587 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T10:05:44.587 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T10:05:44.587 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-06T10:05:44.587 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:1 2024-04-06T10:05:44.587 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T10:05:44.587 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T10:05:44.587 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T10:05:44.587 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=1 2024-04-06T10:05:44.587 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-06T10:05:44.587 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T10:05:44.587 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:05:44.587 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=1 2024-04-06T10:05:44.587 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=1 2024-04-06T10:05:44.589 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-06T10:05:44.589 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:1 2024-04-06T10:05:44.589 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T10:05:44.589 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T10:05:44.589 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-06T10:05:44.589 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:1 2024-04-06T10:05:44.589 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T10:05:44.589 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T10:05:44.589 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:05:44.589 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=1 2024-04-06T10:05:44.589 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-06T10:05:44.590 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T10:05:44.590 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T10:05:44.590 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=1 2024-04-06T10:05:44.590 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=1 2024-04-06T10:05:44.590 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-06T10:05:44.612 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T10:05:44.612 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T10:05:44.613 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T10:05:44.613 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/image_37 2024-04-06T10:05:44.748 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi192.stderr:2024-04-06T10:05:44.748+0000 7f3d223c1640 -1 rbd::mirror::image_replayer::journal::Replayer: 0x56067b1fe000 handle_flush_local_replay: error flushing local replay: (108) Cannot send after transport endpoint shutdown 2024-04-06T10:05:44.761 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T10:05:44.762 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror image_37 2024-04-06T10:05:44.762 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:05:44.762 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:05:44.762 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_37 2024-04-06T10:05:44.762 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror image_37 '' 2024-04-06T10:05:44.762 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:05:44.762 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:05:44.762 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_37 2024-04-06T10:05:44.762 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T10:05:44.763 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-image_37.status 2024-04-06T10:05:44.763 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-image_37.status 2024-04-06T10:05:44.763 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T10:05:44.765 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-image_37.status 2024-04-06T10:05:44.766 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_37 2024-04-06T10:05:44.766 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_37.status 2024-04-06T10:05:44.816 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 0 2024-04-06T10:05:44.816 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 0 2024-04-06T10:05:44.816 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T10:05:44.816 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=1, tag_tid=3, entry_tid=1], [object_number=0, tag_tid=3, entry_tid=0], [object_number=2, tag_tid=1, entry_tid=3230], [object_number=3, tag_tid=1, entry_tid=3227]]], state=connected] 2024-04-06T10:05:44.816 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=1, tag_tid=1, entry_tid=2269], [object_number=0, tag_tid=1, entry_tid=2268], [object_number=3, tag_tid=1, entry_tid=2267], [object_number=2, tag_tid=1, entry_tid=2266]]], state=connected] 2024-04-06T10:05:44.820 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_37.status 2024-04-06T10:05:44.821 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=1, tag_tid=3, entry_tid=1' 2024-04-06T10:05:44.823 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror image_37 2024-04-06T10:05:44.823 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:05:44.823 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:05:44.823 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_37 2024-04-06T10:05:44.823 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror image_37 '..*' 2024-04-06T10:05:44.823 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:05:44.823 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:05:44.823 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_37 2024-04-06T10:05:44.823 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T10:05:44.823 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-image_37.status 2024-04-06T10:05:44.823 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-image_37.status 2024-04-06T10:05:44.823 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T10:05:44.824 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-image_37.status 2024-04-06T10:05:44.824 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_37 2024-04-06T10:05:44.824 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_37.status 2024-04-06T10:05:44.870 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 0 2024-04-06T10:05:44.871 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 0 2024-04-06T10:05:44.871 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T10:05:44.871 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=1, tag_tid=3, entry_tid=1], [object_number=0, tag_tid=3, entry_tid=0], [object_number=2, tag_tid=1, entry_tid=3230], [object_number=3, tag_tid=1, entry_tid=3227]]], state=connected] 2024-04-06T10:05:44.871 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=1, tag_tid=1, entry_tid=2269], [object_number=0, tag_tid=1, entry_tid=2268], [object_number=3, tag_tid=1, entry_tid=2267], [object_number=2, tag_tid=1, entry_tid=2266]]], state=connected] 2024-04-06T10:05:44.874 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_37.status 2024-04-06T10:05:44.876 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=1, tag_tid=1, entry_tid=2269' 2024-04-06T10:05:44.876 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=1, tag_tid=3, entry_tid=1' -a 'object_number=1, tag_tid=3, entry_tid=1' = 'object_number=1, tag_tid=1, entry_tid=2269' 2024-04-06T10:05:44.876 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=1, tag_tid=1, entry_tid=2269' '!=' '' 2024-04-06T10:05:44.877 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T10:05:44.877 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=1, tag_tid=1, entry_tid=2269' = '' 2024-04-06T10:05:44.877 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=1, tag_tid=1, entry_tid=2269' 2024-04-06T10:05:44.877 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=1, tag_tid=3, entry_tid=1' 2024-04-06T10:05:44.877 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T10:05:44.877 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T10:05:44.877 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=3 2024-04-06T10:05:44.878 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=1, tag_tid=1, entry_tid=2269' 2024-04-06T10:05:44.878 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T10:05:44.878 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T10:05:44.879 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=1 2024-04-06T10:05:44.880 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=1, tag_tid=3, entry_tid=1' 2024-04-06T10:05:44.881 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T10:05:44.881 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T10:05:44.882 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T10:05:44.882 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=1, tag_tid=1, entry_tid=2269' 2024-04-06T10:05:44.882 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T10:05:44.882 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T10:05:44.883 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=2269 2024-04-06T10:05:44.883 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 3 = 1 -a 1 -le 2269 2024-04-06T10:05:44.883 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T10:05:44.883 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T10:05:44.883 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T10:05:45.084 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror image_37 2024-04-06T10:05:45.085 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T10:05:45.085 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:05:45.085 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_37 2024-04-06T10:05:45.085 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T10:05:45.085 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n image_37 ']' 2024-04-06T10:05:45.085 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/image_37' 2024-04-06T10:05:45.085 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_37 2024-04-06T10:05:45.086 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T10:05:45.086 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T10:05:45.086 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T10:05:45.086 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T10:05:45.086 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T10:05:45.086 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T10:05:45.086 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T10:05:45.087 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T10:05:45.087 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T10:05:45.089 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T10:05:45.089 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_37 2024-04-06T10:05:45.090 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T10:05:45.090 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T10:05:45.090 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T10:05:45.090 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T10:05:45.090 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T10:05:45.090 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T10:05:45.090 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T10:05:45.090 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T10:05:45.090 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T10:05:45.090 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T10:05:45.090 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T10:05:45.090 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:05:45.090 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T10:05:45.091 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T10:05:45.091 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T10:05:45.091 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T10:05:45.091 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T10:05:45.091 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T10:05:45.091 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T10:05:45.091 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T10:05:45.091 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T10:05:45.091 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T10:05:45.091 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:05:45.091 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T10:05:45.091 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T10:05:45.091 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T10:05:45.091 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T10:05:45.092 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T10:05:45.092 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T10:05:45.092 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T10:05:45.117 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T10:05:45.117 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T10:05:45.117 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T10:05:45.117 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_37 2024-04-06T10:05:45.235 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:Invalid command: unused arguments: ['mirror/image_37'] 2024-04-06T10:05:45.235 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:rbd mirror flush : flush rbd mirror 2024-04-06T10:05:45.235 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:admin_socket: invalid command 2024-04-06T10:05:45.238 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T10:05:45.238 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/image_37 2024-04-06T10:05:45.238 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:1 2024-04-06T10:05:45.238 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T10:05:45.238 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T10:05:45.238 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-06T10:05:45.238 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:1 2024-04-06T10:05:45.238 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T10:05:45.239 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T10:05:45.239 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T10:05:45.239 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=1 2024-04-06T10:05:45.239 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-06T10:05:45.239 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T10:05:45.239 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:05:45.239 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=1 2024-04-06T10:05:45.239 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=1 2024-04-06T10:05:45.239 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-06T10:05:45.239 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:1 2024-04-06T10:05:45.239 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T10:05:45.239 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T10:05:45.239 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-06T10:05:45.240 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:1 2024-04-06T10:05:45.240 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T10:05:45.240 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T10:05:45.240 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:05:45.240 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=1 2024-04-06T10:05:45.240 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-06T10:05:45.240 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T10:05:45.240 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T10:05:45.240 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=1 2024-04-06T10:05:45.240 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=1 2024-04-06T10:05:45.241 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-06T10:05:45.263 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T10:05:45.263 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T10:05:45.263 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T10:05:45.263 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/image_37 2024-04-06T10:05:45.412 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T10:05:45.413 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror image_37 2024-04-06T10:05:45.414 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:05:45.414 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:05:45.414 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_37 2024-04-06T10:05:45.414 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror image_37 '' 2024-04-06T10:05:45.414 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:05:45.414 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:05:45.414 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_37 2024-04-06T10:05:45.414 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T10:05:45.415 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-image_37.status 2024-04-06T10:05:45.415 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-image_37.status 2024-04-06T10:05:45.415 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T10:05:45.418 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-image_37.status 2024-04-06T10:05:45.418 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_37 2024-04-06T10:05:45.418 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_37.status 2024-04-06T10:05:45.463 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 0 2024-04-06T10:05:45.463 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 0 2024-04-06T10:05:45.463 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T10:05:45.463 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=1, tag_tid=3, entry_tid=1], [object_number=0, tag_tid=3, entry_tid=0], [object_number=2, tag_tid=1, entry_tid=3230], [object_number=3, tag_tid=1, entry_tid=3227]]], state=connected] 2024-04-06T10:05:45.463 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=2, tag_tid=1, entry_tid=3230], [object_number=1, tag_tid=1, entry_tid=3229], [object_number=0, tag_tid=1, entry_tid=3228], [object_number=3, tag_tid=1, entry_tid=3227]]], state=connected] 2024-04-06T10:05:45.467 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_37.status 2024-04-06T10:05:45.468 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=1, tag_tid=3, entry_tid=1' 2024-04-06T10:05:45.469 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror image_37 2024-04-06T10:05:45.469 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:05:45.469 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:05:45.469 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_37 2024-04-06T10:05:45.469 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror image_37 '..*' 2024-04-06T10:05:45.470 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:05:45.470 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:05:45.470 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_37 2024-04-06T10:05:45.470 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T10:05:45.470 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-image_37.status 2024-04-06T10:05:45.470 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-image_37.status 2024-04-06T10:05:45.470 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T10:05:45.470 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-image_37.status 2024-04-06T10:05:45.471 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_37 2024-04-06T10:05:45.471 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_37.status 2024-04-06T10:05:45.516 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 0 2024-04-06T10:05:45.516 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 0 2024-04-06T10:05:45.516 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T10:05:45.516 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=1, tag_tid=3, entry_tid=1], [object_number=0, tag_tid=3, entry_tid=0], [object_number=2, tag_tid=1, entry_tid=3230], [object_number=3, tag_tid=1, entry_tid=3227]]], state=connected] 2024-04-06T10:05:45.517 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=2, tag_tid=1, entry_tid=3230], [object_number=1, tag_tid=1, entry_tid=3229], [object_number=0, tag_tid=1, entry_tid=3228], [object_number=3, tag_tid=1, entry_tid=3227]]], state=connected] 2024-04-06T10:05:45.520 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_37.status 2024-04-06T10:05:45.522 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=2, tag_tid=1, entry_tid=3230' 2024-04-06T10:05:45.522 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=1, tag_tid=3, entry_tid=1' -a 'object_number=1, tag_tid=3, entry_tid=1' = 'object_number=2, tag_tid=1, entry_tid=3230' 2024-04-06T10:05:45.522 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=2, tag_tid=1, entry_tid=3230' '!=' 'object_number=1, tag_tid=1, entry_tid=2269' 2024-04-06T10:05:45.523 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T10:05:45.523 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=2, tag_tid=1, entry_tid=3230' = 'object_number=1, tag_tid=1, entry_tid=2269' 2024-04-06T10:05:45.523 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=2, tag_tid=1, entry_tid=3230' 2024-04-06T10:05:45.523 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=1, tag_tid=3, entry_tid=1' 2024-04-06T10:05:45.523 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T10:05:45.523 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T10:05:45.523 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=3 2024-04-06T10:05:45.524 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=2, tag_tid=1, entry_tid=3230' 2024-04-06T10:05:45.524 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T10:05:45.524 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T10:05:45.525 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=1 2024-04-06T10:05:45.526 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=1, tag_tid=3, entry_tid=1' 2024-04-06T10:05:45.526 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T10:05:45.526 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T10:05:45.528 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T10:05:45.528 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=2, tag_tid=1, entry_tid=3230' 2024-04-06T10:05:45.528 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T10:05:45.528 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T10:05:45.529 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=3230 2024-04-06T10:05:45.529 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 3 = 1 -a 1 -le 3230 2024-04-06T10:05:45.529 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T10:05:45.529 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T10:05:45.529 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T10:05:45.731 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror image_37 2024-04-06T10:05:45.731 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T10:05:45.731 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:05:45.731 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_37 2024-04-06T10:05:45.731 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T10:05:45.731 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n image_37 ']' 2024-04-06T10:05:45.731 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/image_37' 2024-04-06T10:05:45.731 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_37 2024-04-06T10:05:45.731 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T10:05:45.732 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T10:05:45.732 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T10:05:45.732 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T10:05:45.732 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T10:05:45.732 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T10:05:45.732 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T10:05:45.732 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T10:05:45.733 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T10:05:45.736 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T10:05:45.736 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_37 2024-04-06T10:05:45.736 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T10:05:45.736 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T10:05:45.736 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T10:05:45.736 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T10:05:45.737 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T10:05:45.737 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T10:05:45.737 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T10:05:45.737 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T10:05:45.737 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T10:05:45.737 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T10:05:45.737 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T10:05:45.737 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:05:45.737 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T10:05:45.737 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T10:05:45.737 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T10:05:45.737 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T10:05:45.737 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T10:05:45.738 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T10:05:45.738 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T10:05:45.738 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T10:05:45.738 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T10:05:45.738 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T10:05:45.738 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:05:45.738 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T10:05:45.738 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T10:05:45.738 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T10:05:45.738 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T10:05:45.738 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T10:05:45.738 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T10:05:45.738 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T10:05:45.762 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T10:05:45.762 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T10:05:45.762 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T10:05:45.762 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_37 2024-04-06T10:05:45.868 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:Invalid command: unused arguments: ['mirror/image_37'] 2024-04-06T10:05:45.868 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:rbd mirror flush : flush rbd mirror 2024-04-06T10:05:45.868 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:admin_socket: invalid command 2024-04-06T10:05:45.870 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T10:05:45.870 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/image_37 2024-04-06T10:05:45.870 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:1 2024-04-06T10:05:45.871 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T10:05:45.871 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T10:05:45.871 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-06T10:05:45.871 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:1 2024-04-06T10:05:45.871 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T10:05:45.871 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T10:05:45.871 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T10:05:45.871 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=1 2024-04-06T10:05:45.871 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-06T10:05:45.871 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T10:05:45.871 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:05:45.871 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=1 2024-04-06T10:05:45.871 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=1 2024-04-06T10:05:45.872 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-06T10:05:45.873 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:1 2024-04-06T10:05:45.873 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T10:05:45.873 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T10:05:45.873 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-06T10:05:45.873 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:1 2024-04-06T10:05:45.873 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T10:05:45.873 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T10:05:45.873 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:05:45.873 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=1 2024-04-06T10:05:45.873 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-06T10:05:45.873 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T10:05:45.873 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T10:05:45.873 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=1 2024-04-06T10:05:45.873 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=1 2024-04-06T10:05:45.873 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-06T10:05:45.896 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T10:05:45.896 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T10:05:45.896 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T10:05:45.896 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/image_37 2024-04-06T10:05:46.041 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T10:05:46.042 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror image_37 2024-04-06T10:05:46.042 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:05:46.042 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:05:46.042 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_37 2024-04-06T10:05:46.042 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror image_37 '' 2024-04-06T10:05:46.042 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:05:46.042 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:05:46.042 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_37 2024-04-06T10:05:46.043 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T10:05:46.043 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-image_37.status 2024-04-06T10:05:46.043 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-image_37.status 2024-04-06T10:05:46.043 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T10:05:46.046 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-image_37.status 2024-04-06T10:05:46.046 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_37 2024-04-06T10:05:46.046 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_37.status 2024-04-06T10:05:46.098 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 0 2024-04-06T10:05:46.098 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 0 2024-04-06T10:05:46.099 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T10:05:46.099 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=1, tag_tid=3, entry_tid=1], [object_number=0, tag_tid=3, entry_tid=0], [object_number=2, tag_tid=1, entry_tid=3230], [object_number=3, tag_tid=1, entry_tid=3227]]], state=connected] 2024-04-06T10:05:46.099 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=2, tag_tid=1, entry_tid=3230], [object_number=1, tag_tid=1, entry_tid=3229], [object_number=0, tag_tid=1, entry_tid=3228], [object_number=3, tag_tid=1, entry_tid=3227]]], state=connected] 2024-04-06T10:05:46.102 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_37.status 2024-04-06T10:05:46.103 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=1, tag_tid=3, entry_tid=1' 2024-04-06T10:05:46.105 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror image_37 2024-04-06T10:05:46.105 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:05:46.105 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:05:46.105 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_37 2024-04-06T10:05:46.105 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror image_37 '..*' 2024-04-06T10:05:46.105 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:05:46.105 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:05:46.105 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_37 2024-04-06T10:05:46.105 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T10:05:46.105 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-image_37.status 2024-04-06T10:05:46.106 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-image_37.status 2024-04-06T10:05:46.106 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T10:05:46.107 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-image_37.status 2024-04-06T10:05:46.107 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_37 2024-04-06T10:05:46.107 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_37.status 2024-04-06T10:05:46.152 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 0 2024-04-06T10:05:46.198 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 0 2024-04-06T10:05:46.198 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T10:05:46.198 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=1, tag_tid=3, entry_tid=1], [object_number=0, tag_tid=3, entry_tid=0], [object_number=2, tag_tid=1, entry_tid=3230], [object_number=3, tag_tid=1, entry_tid=3227]]], state=connected] 2024-04-06T10:05:46.198 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=2, tag_tid=1, entry_tid=3230], [object_number=1, tag_tid=1, entry_tid=3229], [object_number=0, tag_tid=1, entry_tid=3228], [object_number=3, tag_tid=1, entry_tid=3227]]], state=connected] 2024-04-06T10:05:46.199 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_37.status 2024-04-06T10:05:46.199 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=2, tag_tid=1, entry_tid=3230' 2024-04-06T10:05:46.199 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=1, tag_tid=3, entry_tid=1' -a 'object_number=1, tag_tid=3, entry_tid=1' = 'object_number=2, tag_tid=1, entry_tid=3230' 2024-04-06T10:05:46.200 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=2, tag_tid=1, entry_tid=3230' '!=' 'object_number=2, tag_tid=1, entry_tid=3230' 2024-04-06T10:05:46.200 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T10:05:46.200 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.4 2024-04-06T10:05:46.560 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror image_37 2024-04-06T10:05:46.560 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T10:05:46.560 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:05:46.560 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_37 2024-04-06T10:05:46.560 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T10:05:46.560 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n image_37 ']' 2024-04-06T10:05:46.560 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/image_37' 2024-04-06T10:05:46.560 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_37 2024-04-06T10:05:46.560 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T10:05:46.560 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T10:05:46.560 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T10:05:46.561 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T10:05:46.561 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T10:05:46.561 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T10:05:46.561 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T10:05:46.561 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T10:05:46.561 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T10:05:46.563 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T10:05:46.563 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_37 2024-04-06T10:05:46.564 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T10:05:46.564 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T10:05:46.564 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T10:05:46.564 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T10:05:46.564 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T10:05:46.564 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T10:05:46.564 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T10:05:46.564 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T10:05:46.564 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T10:05:46.564 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T10:05:46.564 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T10:05:46.564 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:05:46.564 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T10:05:46.564 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T10:05:46.565 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T10:05:46.566 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T10:05:46.566 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T10:05:46.566 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T10:05:46.566 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T10:05:46.566 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T10:05:46.566 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T10:05:46.566 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T10:05:46.566 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:05:46.566 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T10:05:46.566 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T10:05:46.566 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T10:05:46.566 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T10:05:46.567 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T10:05:46.567 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T10:05:46.567 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T10:05:46.594 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T10:05:46.594 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T10:05:46.594 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T10:05:46.594 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_37 2024-04-06T10:05:46.696 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:Invalid command: unused arguments: ['mirror/image_37'] 2024-04-06T10:05:46.697 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:rbd mirror flush : flush rbd mirror 2024-04-06T10:05:46.697 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:admin_socket: invalid command 2024-04-06T10:05:46.699 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T10:05:46.699 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/image_37 2024-04-06T10:05:46.699 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:1 2024-04-06T10:05:46.699 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T10:05:46.699 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T10:05:46.699 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-06T10:05:46.699 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:1 2024-04-06T10:05:46.699 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T10:05:46.700 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T10:05:46.700 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T10:05:46.700 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=1 2024-04-06T10:05:46.700 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-06T10:05:46.700 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T10:05:46.700 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:05:46.700 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=1 2024-04-06T10:05:46.700 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=1 2024-04-06T10:05:46.701 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-06T10:05:46.701 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:1 2024-04-06T10:05:46.702 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T10:05:46.702 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T10:05:46.702 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-06T10:05:46.702 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:1 2024-04-06T10:05:46.702 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T10:05:46.702 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T10:05:46.702 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:05:46.702 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=1 2024-04-06T10:05:46.702 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-06T10:05:46.702 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T10:05:46.702 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T10:05:46.702 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=1 2024-04-06T10:05:46.703 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=1 2024-04-06T10:05:46.703 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-06T10:05:46.720 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T10:05:46.720 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T10:05:46.720 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T10:05:46.720 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/image_37 2024-04-06T10:05:46.879 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T10:05:46.880 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror image_37 2024-04-06T10:05:46.880 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:05:46.880 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:05:46.880 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_37 2024-04-06T10:05:46.880 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror image_37 '' 2024-04-06T10:05:46.880 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:05:46.880 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:05:46.880 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_37 2024-04-06T10:05:46.880 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T10:05:46.881 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-image_37.status 2024-04-06T10:05:46.881 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-image_37.status 2024-04-06T10:05:46.881 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T10:05:46.883 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-image_37.status 2024-04-06T10:05:46.883 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_37 2024-04-06T10:05:46.883 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_37.status 2024-04-06T10:05:46.934 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 0 2024-04-06T10:05:46.934 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 0 2024-04-06T10:05:46.934 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T10:05:46.934 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=1, tag_tid=3, entry_tid=1], [object_number=0, tag_tid=3, entry_tid=0], [object_number=2, tag_tid=1, entry_tid=3230], [object_number=3, tag_tid=1, entry_tid=3227]]], state=connected] 2024-04-06T10:05:46.935 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=1, tag_tid=3, entry_tid=1], [object_number=0, tag_tid=3, entry_tid=0], [object_number=2, tag_tid=1, entry_tid=3230], [object_number=3, tag_tid=1, entry_tid=3227]]], state=connected] 2024-04-06T10:05:46.938 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_37.status 2024-04-06T10:05:46.939 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=1, tag_tid=3, entry_tid=1' 2024-04-06T10:05:46.940 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror image_37 2024-04-06T10:05:46.940 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:05:46.940 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:05:46.941 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_37 2024-04-06T10:05:46.941 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror image_37 '..*' 2024-04-06T10:05:46.941 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:05:46.941 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:05:46.941 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_37 2024-04-06T10:05:46.941 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T10:05:46.941 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-image_37.status 2024-04-06T10:05:46.941 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-image_37.status 2024-04-06T10:05:46.941 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T10:05:46.942 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-image_37.status 2024-04-06T10:05:46.942 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_37 2024-04-06T10:05:46.942 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_37.status 2024-04-06T10:05:46.987 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 0 2024-04-06T10:05:46.987 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 0 2024-04-06T10:05:46.987 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T10:05:46.988 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=1, tag_tid=3, entry_tid=1], [object_number=0, tag_tid=3, entry_tid=0], [object_number=2, tag_tid=1, entry_tid=3230], [object_number=3, tag_tid=1, entry_tid=3227]]], state=connected] 2024-04-06T10:05:46.988 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=1, tag_tid=3, entry_tid=1], [object_number=0, tag_tid=3, entry_tid=0], [object_number=2, tag_tid=1, entry_tid=3230], [object_number=3, tag_tid=1, entry_tid=3227]]], state=connected] 2024-04-06T10:05:46.991 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_37.status 2024-04-06T10:05:46.993 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=1, tag_tid=3, entry_tid=1' 2024-04-06T10:05:46.993 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=1, tag_tid=3, entry_tid=1' -a 'object_number=1, tag_tid=3, entry_tid=1' = 'object_number=1, tag_tid=3, entry_tid=1' 2024-04-06T10:05:46.993 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T10:05:46.993 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_snap_present cluster1 mirror image_37 snap 2024-04-06T10:05:46.993 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T10:05:46.993 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:05:46.993 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_37 2024-04-06T10:05:46.993 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local snap_name=snap 2024-04-06T10:05:46.993 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local s 2024-04-06T10:05:46.993 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-04-06T10:05:46.993 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 1 2024-04-06T10:05:47.994 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 info mirror/image_37@snap 2024-04-06T10:05:48.086 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:rbd image 'image_37': 2024-04-06T10:05:48.086 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: size 128 MiB in 32 objects 2024-04-06T10:05:48.086 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: order 22 (4 MiB objects) 2024-04-06T10:05:48.086 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: snapshot_count: 1 2024-04-06T10:05:48.086 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: id: 10acc05f4294 2024-04-06T10:05:48.086 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: block_name_prefix: rbd_data.10acc05f4294 2024-04-06T10:05:48.087 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: format: 2 2024-04-06T10:05:48.087 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: features: layering, exclusive-lock, journaling 2024-04-06T10:05:48.087 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: op_features: 2024-04-06T10:05:48.087 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: flags: 2024-04-06T10:05:48.087 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: create_timestamp: Sat Apr 6 09:56:14 2024 2024-04-06T10:05:48.087 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: access_timestamp: Sat Apr 6 09:56:14 2024 2024-04-06T10:05:48.087 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: modify_timestamp: Sat Apr 6 09:57:15 2024 2024-04-06T10:05:48.087 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: protected: False 2024-04-06T10:05:48.087 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: journal: 10acc05f4294 2024-04-06T10:05:48.087 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: mirroring state: enabled 2024-04-06T10:05:48.087 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: mirroring mode: journal 2024-04-06T10:05:48.087 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: mirroring global id: 67e4f2dc-2644-4a66-9eab-4189397f7ef4 2024-04-06T10:05:48.087 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: mirroring primary: false 2024-04-06T10:05:48.092 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T10:05:48.092 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ compare_image_snaps mirror image_37 snap 2024-04-06T10:05:48.092 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:05:48.092 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_37 2024-04-06T10:05:48.092 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local snap_name=snap 2024-04-06T10:05:48.092 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local ret=0 2024-04-06T10:05:48.092 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local rmt_export=/tmp/tmp.BhW6xKmROU/cluster2-mirror-image_37.export 2024-04-06T10:05:48.092 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loc_export=/tmp/tmp.BhW6xKmROU/cluster1-mirror-image_37.export 2024-04-06T10:05:48.092 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rm -f /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_37.export /tmp/tmp.BhW6xKmROU/cluster1-mirror-image_37.export 2024-04-06T10:05:48.093 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster2 -p mirror export image_37@snap /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_37.export 2024-04-06T10:05:48.251 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-04-06T10:05:48.260 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 -p mirror export image_37@snap /tmp/tmp.BhW6xKmROU/cluster1-mirror-image_37.export 2024-04-06T10:05:48.541 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-04-06T10:05:48.551 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmp /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_37.export /tmp/tmp.BhW6xKmROU/cluster1-mirror-image_37.export 2024-04-06T10:05:48.634 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rm -f /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_37.export /tmp/tmp.BhW6xKmROU/cluster1-mirror-image_37.export 2024-04-06T10:05:48.676 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T10:05:48.677 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-06T10:05:48.677 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ image=image_38 2024-04-06T10:05:48.677 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ create_snap cluster2 mirror image_38 snap 2024-04-06T10:05:48.677 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-04-06T10:05:48.677 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:05:48.677 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_38 2024-04-06T10:05:48.677 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local snap_name=snap 2024-04-06T10:05:48.677 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster2 -p mirror snap create image_38@snap --debug-rbd=20 --debug-journaler=20 2024-04-06T10:05:50.931 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_image_replay_started cluster1 mirror image_38 2024-04-06T10:05:50.931 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T10:05:50.931 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:05:50.931 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_38 2024-04-06T10:05:50.931 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_image_replay_state cluster1 mirror image_38 started 2024-04-06T10:05:50.931 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T10:05:50.931 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:05:50.932 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_38 2024-04-06T10:05:50.932 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state=started 2024-04-06T10:05:50.932 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local s 2024-04-06T10:05:50.932 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T10:05:50.932 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 1 2024-04-06T10:05:51.933 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test_image_replay_state cluster1 mirror image_38 started 2024-04-06T10:05:51.933 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T10:05:51.933 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:05:51.933 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_38 2024-04-06T10:05:51.933 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local test_state=started 2024-04-06T10:05:51.933 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local status_result 2024-04-06T10:05:51.933 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local current_state=stopped 2024-04-06T10:05:51.934 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_38 2024-04-06T10:05:51.934 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -i state 2024-04-06T10:05:51.935 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_instance=cluster1 2024-04-06T10:05:51.935 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ shift 2024-04-06T10:05:51.935 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T10:05:51.935 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance=cluster1 2024-04-06T10:05:51.935 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local loop_instance 2024-04-06T10:05:51.935 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T10:05:51.935 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sleep 0 2024-04-06T10:05:51.936 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-04-06T10:05:51.936 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ seq 0 3 2024-04-06T10:05:51.939 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T10:05:51.939 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_38 2024-04-06T10:05:51.939 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1:0 2024-04-06T10:05:51.939 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ shift 2024-04-06T10:05:51.940 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T10:05:51.940 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-04-06T10:05:51.940 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T10:05:51.940 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=cluster 2024-04-06T10:05:51.940 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T10:05:51.940 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:05:51.940 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T10:05:51.940 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T10:05:51.940 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval cluster=cluster1 2024-04-06T10:05:51.940 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ cluster=cluster1 2024-04-06T10:05:51.940 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T10:05:51.940 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T10:05:51.940 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-04-06T10:05:51.940 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local local_cluster=cluster1:0 2024-04-06T10:05:51.940 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local cluster=cluster1 2024-04-06T10:05:51.941 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local instance 2024-04-06T10:05:51.941 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T10:05:51.941 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local val=cluster1:0 2024-04-06T10:05:51.941 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local cluster_var_name=local_cluster 2024-04-06T10:05:51.941 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local instance_var_name=instance 2024-04-06T10:05:51.941 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ cluster=cluster1 2024-04-06T10:05:51.941 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T10:05:51.941 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ '[' 0 = cluster1:0 ']' 2024-04-06T10:05:51.941 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ eval local_cluster=cluster1 2024-04-06T10:05:51.941 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++++ local_cluster=cluster1 2024-04-06T10:05:51.941 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ eval instance=0 2024-04-06T10:05:51.941 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++++ instance=0 2024-04-06T10:05:51.941 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T10:05:51.966 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T10:05:51.967 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T10:05:51.967 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T10:05:51.967 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_38 2024-04-06T10:05:52.074 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:Invalid command: unused arguments: ['mirror/image_38'] 2024-04-06T10:05:52.074 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:rbd mirror status : get status for rbd mirror 2024-04-06T10:05:52.074 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:admin_socket: invalid command 2024-04-06T10:05:52.076 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T10:05:52.076 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_38 2024-04-06T10:05:52.076 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1:1 2024-04-06T10:05:52.077 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ shift 2024-04-06T10:05:52.077 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T10:05:52.077 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-04-06T10:05:52.077 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:1 2024-04-06T10:05:52.077 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=cluster 2024-04-06T10:05:52.077 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T10:05:52.077 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:05:52.077 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=1 2024-04-06T10:05:52.077 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-06T10:05:52.077 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval cluster=cluster1 2024-04-06T10:05:52.077 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ cluster=cluster1 2024-04-06T10:05:52.077 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=1 2024-04-06T10:05:52.077 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=1 2024-04-06T10:05:52.079 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-04-06T10:05:52.079 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local local_cluster=cluster1:1 2024-04-06T10:05:52.079 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local cluster=cluster1 2024-04-06T10:05:52.079 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local instance 2024-04-06T10:05:52.079 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-06T10:05:52.079 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local val=cluster1:1 2024-04-06T10:05:52.079 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local cluster_var_name=local_cluster 2024-04-06T10:05:52.079 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local instance_var_name=instance 2024-04-06T10:05:52.080 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ cluster=cluster1 2024-04-06T10:05:52.080 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=1 2024-04-06T10:05:52.080 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ '[' 1 = cluster1:1 ']' 2024-04-06T10:05:52.080 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ eval local_cluster=cluster1 2024-04-06T10:05:52.080 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++++ local_cluster=cluster1 2024-04-06T10:05:52.080 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ eval instance=1 2024-04-06T10:05:52.080 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++++ instance=1 2024-04-06T10:05:52.080 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-06T10:05:52.097 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T10:05:52.097 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T10:05:52.097 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T10:05:52.097 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_38 2024-04-06T10:05:52.246 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ return 0 2024-04-06T10:05:52.246 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ status_result=' "state": "Replaying"' 2024-04-06T10:05:52.247 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ echo ' "state": "Replaying"' 2024-04-06T10:05:52.247 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep -i Replaying 2024-04-06T10:05:52.248 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: "state": "Replaying" 2024-04-06T10:05:52.249 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ current_state=started 2024-04-06T10:05:52.249 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test started = started 2024-04-06T10:05:52.249 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T10:05:52.249 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_38 2024-04-06T10:05:52.249 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local local_cluster=cluster1 2024-04-06T10:05:52.249 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-04-06T10:05:52.249 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:05:52.249 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_38 2024-04-06T10:05:52.249 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' journal = journal ']' 2024-04-06T10:05:52.250 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_journal_replay_complete cluster1 cluster2 mirror image_38 2024-04-06T10:05:52.250 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local local_cluster=cluster1 2024-04-06T10:05:52.250 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-04-06T10:05:52.250 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:05:52.250 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_38 2024-04-06T10:05:52.250 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local s master_pos mirror_pos last_mirror_pos 2024-04-06T10:05:52.250 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local master_tag master_entry mirror_tag mirror_entry 2024-04-06T10:05:52.250 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T10:05:52.250 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T10:05:52.250 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T10:05:52.451 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror image_38 2024-04-06T10:05:52.451 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T10:05:52.451 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:05:52.451 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_38 2024-04-06T10:05:52.451 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T10:05:52.451 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n image_38 ']' 2024-04-06T10:05:52.451 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/image_38' 2024-04-06T10:05:52.451 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_38 2024-04-06T10:05:52.451 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T10:05:52.451 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T10:05:52.452 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T10:05:52.452 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T10:05:52.452 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T10:05:52.452 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T10:05:52.452 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T10:05:52.452 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T10:05:52.453 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T10:05:52.456 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T10:05:52.456 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_38 2024-04-06T10:05:52.456 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T10:05:52.456 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T10:05:52.456 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T10:05:52.456 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T10:05:52.456 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T10:05:52.456 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T10:05:52.456 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T10:05:52.456 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T10:05:52.456 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T10:05:52.457 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T10:05:52.457 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T10:05:52.457 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:05:52.457 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T10:05:52.457 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T10:05:52.457 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T10:05:52.457 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T10:05:52.457 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T10:05:52.457 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T10:05:52.457 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T10:05:52.457 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T10:05:52.457 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T10:05:52.457 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T10:05:52.457 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:05:52.457 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T10:05:52.458 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T10:05:52.458 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T10:05:52.458 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T10:05:52.458 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T10:05:52.458 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T10:05:52.458 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T10:05:52.482 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T10:05:52.482 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T10:05:52.482 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T10:05:52.482 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_38 2024-04-06T10:05:52.588 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:Invalid command: unused arguments: ['mirror/image_38'] 2024-04-06T10:05:52.588 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:rbd mirror flush : flush rbd mirror 2024-04-06T10:05:52.588 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:admin_socket: invalid command 2024-04-06T10:05:52.590 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T10:05:52.590 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/image_38 2024-04-06T10:05:52.590 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:1 2024-04-06T10:05:52.590 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T10:05:52.590 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T10:05:52.590 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-06T10:05:52.590 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:1 2024-04-06T10:05:52.590 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T10:05:52.591 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T10:05:52.591 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T10:05:52.591 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=1 2024-04-06T10:05:52.591 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-06T10:05:52.591 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T10:05:52.591 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:05:52.591 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=1 2024-04-06T10:05:52.591 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=1 2024-04-06T10:05:52.593 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-06T10:05:52.593 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:1 2024-04-06T10:05:52.593 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T10:05:52.593 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T10:05:52.593 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-06T10:05:52.593 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:1 2024-04-06T10:05:52.593 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T10:05:52.593 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T10:05:52.593 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:05:52.593 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=1 2024-04-06T10:05:52.593 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-06T10:05:52.594 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T10:05:52.594 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T10:05:52.594 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=1 2024-04-06T10:05:52.594 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=1 2024-04-06T10:05:52.594 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-06T10:05:52.616 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T10:05:52.616 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T10:05:52.616 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T10:05:52.617 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/image_38 2024-04-06T10:05:52.752 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T10:05:52.753 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror image_38 2024-04-06T10:05:52.753 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:05:52.753 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:05:52.753 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_38 2024-04-06T10:05:52.753 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror image_38 '' 2024-04-06T10:05:52.753 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:05:52.753 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:05:52.753 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_38 2024-04-06T10:05:52.753 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T10:05:52.754 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-image_38.status 2024-04-06T10:05:52.754 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-image_38.status 2024-04-06T10:05:52.754 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T10:05:52.757 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-image_38.status 2024-04-06T10:05:52.758 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_38 2024-04-06T10:05:52.758 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_38.status 2024-04-06T10:05:52.803 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 2 2024-04-06T10:05:52.803 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 2 2024-04-06T10:05:52.804 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T10:05:52.804 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=9, tag_tid=2, entry_tid=1], [object_number=8, tag_tid=2, entry_tid=0], [object_number=11, tag_tid=1, entry_tid=10135], [object_number=10, tag_tid=1, entry_tid=10134]]], state=connected] 2024-04-06T10:05:52.804 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=8, tag_tid=1, entry_tid=10136], [object_number=11, tag_tid=1, entry_tid=10135], [object_number=10, tag_tid=1, entry_tid=10134], [object_number=9, tag_tid=1, entry_tid=10133]]], state=connected] 2024-04-06T10:05:52.807 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_38.status 2024-04-06T10:05:52.808 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=9, tag_tid=2, entry_tid=1' 2024-04-06T10:05:52.810 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror image_38 2024-04-06T10:05:52.810 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:05:52.810 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:05:52.810 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_38 2024-04-06T10:05:52.810 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror image_38 '..*' 2024-04-06T10:05:52.810 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:05:52.810 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:05:52.810 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_38 2024-04-06T10:05:52.810 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T10:05:52.810 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-image_38.status 2024-04-06T10:05:52.811 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-image_38.status 2024-04-06T10:05:52.811 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T10:05:52.811 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-image_38.status 2024-04-06T10:05:52.811 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_38 2024-04-06T10:05:52.811 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_38.status 2024-04-06T10:05:52.857 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 2 2024-04-06T10:05:52.857 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 2 2024-04-06T10:05:52.857 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T10:05:52.857 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=9, tag_tid=2, entry_tid=1], [object_number=8, tag_tid=2, entry_tid=0], [object_number=11, tag_tid=1, entry_tid=10135], [object_number=10, tag_tid=1, entry_tid=10134]]], state=connected] 2024-04-06T10:05:52.857 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=8, tag_tid=1, entry_tid=10136], [object_number=11, tag_tid=1, entry_tid=10135], [object_number=10, tag_tid=1, entry_tid=10134], [object_number=9, tag_tid=1, entry_tid=10133]]], state=connected] 2024-04-06T10:05:52.860 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_38.status 2024-04-06T10:05:52.863 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=8, tag_tid=1, entry_tid=10136' 2024-04-06T10:05:52.863 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=9, tag_tid=2, entry_tid=1' -a 'object_number=9, tag_tid=2, entry_tid=1' = 'object_number=8, tag_tid=1, entry_tid=10136' 2024-04-06T10:05:52.863 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=8, tag_tid=1, entry_tid=10136' '!=' '' 2024-04-06T10:05:52.863 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T10:05:52.863 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=8, tag_tid=1, entry_tid=10136' = '' 2024-04-06T10:05:52.863 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=8, tag_tid=1, entry_tid=10136' 2024-04-06T10:05:52.863 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=9, tag_tid=2, entry_tid=1' 2024-04-06T10:05:52.863 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T10:05:52.863 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T10:05:52.864 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=2 2024-04-06T10:05:52.864 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=8, tag_tid=1, entry_tid=10136' 2024-04-06T10:05:52.864 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T10:05:52.864 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T10:05:52.865 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=1 2024-04-06T10:05:52.867 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=9, tag_tid=2, entry_tid=1' 2024-04-06T10:05:52.867 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T10:05:52.867 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T10:05:52.868 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T10:05:52.868 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=8, tag_tid=1, entry_tid=10136' 2024-04-06T10:05:52.869 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T10:05:52.869 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T10:05:52.870 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=10136 2024-04-06T10:05:52.870 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 2 = 1 -a 1 -le 10136 2024-04-06T10:05:52.870 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T10:05:52.870 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T10:05:52.871 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T10:05:53.071 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror image_38 2024-04-06T10:05:53.072 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T10:05:53.072 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:05:53.072 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_38 2024-04-06T10:05:53.072 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T10:05:53.072 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n image_38 ']' 2024-04-06T10:05:53.072 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/image_38' 2024-04-06T10:05:53.072 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_38 2024-04-06T10:05:53.072 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T10:05:53.072 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T10:05:53.072 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T10:05:53.073 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T10:05:53.073 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T10:05:53.073 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T10:05:53.073 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T10:05:53.073 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T10:05:53.073 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T10:05:53.077 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T10:05:53.077 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_38 2024-04-06T10:05:53.077 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T10:05:53.077 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T10:05:53.077 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T10:05:53.077 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T10:05:53.077 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T10:05:53.077 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T10:05:53.077 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T10:05:53.077 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T10:05:53.077 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T10:05:53.077 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T10:05:53.077 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T10:05:53.078 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:05:53.078 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T10:05:53.078 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T10:05:53.078 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T10:05:53.078 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T10:05:53.078 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T10:05:53.078 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T10:05:53.078 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T10:05:53.078 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T10:05:53.078 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T10:05:53.078 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T10:05:53.078 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:05:53.079 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T10:05:53.079 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T10:05:53.079 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T10:05:53.079 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T10:05:53.079 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T10:05:53.079 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T10:05:53.079 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T10:05:53.102 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T10:05:53.102 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T10:05:53.102 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T10:05:53.102 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_38 2024-04-06T10:05:53.208 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:Invalid command: unused arguments: ['mirror/image_38'] 2024-04-06T10:05:53.208 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:rbd mirror flush : flush rbd mirror 2024-04-06T10:05:53.208 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:admin_socket: invalid command 2024-04-06T10:05:53.212 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T10:05:53.212 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/image_38 2024-04-06T10:05:53.212 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:1 2024-04-06T10:05:53.212 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T10:05:53.212 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T10:05:53.212 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-06T10:05:53.212 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:1 2024-04-06T10:05:53.212 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T10:05:53.213 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T10:05:53.213 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T10:05:53.213 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=1 2024-04-06T10:05:53.213 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-06T10:05:53.213 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T10:05:53.213 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:05:53.214 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=1 2024-04-06T10:05:53.214 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=1 2024-04-06T10:05:53.214 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-06T10:05:53.214 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:1 2024-04-06T10:05:53.214 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T10:05:53.214 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T10:05:53.214 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-06T10:05:53.214 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:1 2024-04-06T10:05:53.214 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T10:05:53.214 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T10:05:53.215 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:05:53.215 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=1 2024-04-06T10:05:53.215 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-06T10:05:53.215 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T10:05:53.215 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T10:05:53.215 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=1 2024-04-06T10:05:53.215 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=1 2024-04-06T10:05:53.215 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-06T10:05:53.231 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T10:05:53.231 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T10:05:53.231 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T10:05:53.231 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/image_38 2024-04-06T10:05:53.374 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T10:05:53.375 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror image_38 2024-04-06T10:05:53.376 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:05:53.376 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:05:53.376 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_38 2024-04-06T10:05:53.376 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror image_38 '' 2024-04-06T10:05:53.376 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:05:53.376 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:05:53.376 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_38 2024-04-06T10:05:53.376 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T10:05:53.377 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-image_38.status 2024-04-06T10:05:53.377 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-image_38.status 2024-04-06T10:05:53.377 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T10:05:53.378 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-image_38.status 2024-04-06T10:05:53.379 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_38 2024-04-06T10:05:53.379 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_38.status 2024-04-06T10:05:53.424 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 2 2024-04-06T10:05:53.424 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 2 2024-04-06T10:05:53.424 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T10:05:53.424 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=9, tag_tid=2, entry_tid=1], [object_number=8, tag_tid=2, entry_tid=0], [object_number=11, tag_tid=1, entry_tid=10135], [object_number=10, tag_tid=1, entry_tid=10134]]], state=connected] 2024-04-06T10:05:53.424 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=8, tag_tid=1, entry_tid=10136], [object_number=11, tag_tid=1, entry_tid=10135], [object_number=10, tag_tid=1, entry_tid=10134], [object_number=9, tag_tid=1, entry_tid=10133]]], state=connected] 2024-04-06T10:05:53.427 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_38.status 2024-04-06T10:05:53.429 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=9, tag_tid=2, entry_tid=1' 2024-04-06T10:05:53.430 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror image_38 2024-04-06T10:05:53.430 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:05:53.430 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:05:53.430 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_38 2024-04-06T10:05:53.430 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror image_38 '..*' 2024-04-06T10:05:53.430 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:05:53.430 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:05:53.431 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_38 2024-04-06T10:05:53.431 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T10:05:53.431 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-image_38.status 2024-04-06T10:05:53.431 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-image_38.status 2024-04-06T10:05:53.431 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T10:05:53.432 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-image_38.status 2024-04-06T10:05:53.432 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_38 2024-04-06T10:05:53.432 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_38.status 2024-04-06T10:05:53.478 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 2 2024-04-06T10:05:53.478 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 2 2024-04-06T10:05:53.479 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T10:05:53.479 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=9, tag_tid=2, entry_tid=1], [object_number=8, tag_tid=2, entry_tid=0], [object_number=11, tag_tid=1, entry_tid=10135], [object_number=10, tag_tid=1, entry_tid=10134]]], state=connected] 2024-04-06T10:05:53.479 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=8, tag_tid=1, entry_tid=10136], [object_number=11, tag_tid=1, entry_tid=10135], [object_number=10, tag_tid=1, entry_tid=10134], [object_number=9, tag_tid=1, entry_tid=10133]]], state=connected] 2024-04-06T10:05:53.482 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_38.status 2024-04-06T10:05:53.483 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=8, tag_tid=1, entry_tid=10136' 2024-04-06T10:05:53.484 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=9, tag_tid=2, entry_tid=1' -a 'object_number=9, tag_tid=2, entry_tid=1' = 'object_number=8, tag_tid=1, entry_tid=10136' 2024-04-06T10:05:53.484 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=8, tag_tid=1, entry_tid=10136' '!=' 'object_number=8, tag_tid=1, entry_tid=10136' 2024-04-06T10:05:53.484 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T10:05:53.484 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.4 2024-04-06T10:05:53.886 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror image_38 2024-04-06T10:05:53.886 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T10:05:53.886 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:05:53.886 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_38 2024-04-06T10:05:53.886 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T10:05:53.886 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n image_38 ']' 2024-04-06T10:05:53.886 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/image_38' 2024-04-06T10:05:53.886 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_38 2024-04-06T10:05:53.887 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T10:05:53.887 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T10:05:53.887 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T10:05:53.887 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T10:05:53.887 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T10:05:53.887 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T10:05:53.887 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T10:05:53.887 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T10:05:53.888 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T10:05:53.890 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T10:05:53.890 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_38 2024-04-06T10:05:53.891 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T10:05:53.891 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T10:05:53.891 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T10:05:53.891 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T10:05:53.891 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T10:05:53.891 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T10:05:53.891 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T10:05:53.891 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T10:05:53.891 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T10:05:53.891 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T10:05:53.891 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T10:05:53.891 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:05:53.891 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T10:05:53.892 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T10:05:53.892 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T10:05:53.893 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T10:05:53.893 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T10:05:53.893 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T10:05:53.893 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T10:05:53.893 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T10:05:53.893 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T10:05:53.893 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T10:05:53.893 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:05:53.893 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T10:05:53.893 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T10:05:53.893 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T10:05:53.893 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T10:05:53.894 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T10:05:53.894 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T10:05:53.894 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T10:05:53.917 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T10:05:53.918 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T10:05:53.918 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T10:05:53.918 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_38 2024-04-06T10:05:54.023 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:Invalid command: unused arguments: ['mirror/image_38'] 2024-04-06T10:05:54.023 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:rbd mirror flush : flush rbd mirror 2024-04-06T10:05:54.023 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:admin_socket: invalid command 2024-04-06T10:05:54.025 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T10:05:54.025 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/image_38 2024-04-06T10:05:54.026 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:1 2024-04-06T10:05:54.026 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T10:05:54.026 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T10:05:54.026 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-06T10:05:54.026 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:1 2024-04-06T10:05:54.026 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T10:05:54.026 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T10:05:54.026 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T10:05:54.026 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=1 2024-04-06T10:05:54.026 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-06T10:05:54.026 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T10:05:54.026 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:05:54.027 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=1 2024-04-06T10:05:54.027 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=1 2024-04-06T10:05:54.028 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-06T10:05:54.028 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:1 2024-04-06T10:05:54.028 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T10:05:54.028 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T10:05:54.028 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-06T10:05:54.028 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:1 2024-04-06T10:05:54.028 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T10:05:54.029 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T10:05:54.029 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:05:54.029 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=1 2024-04-06T10:05:54.029 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-06T10:05:54.029 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T10:05:54.029 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T10:05:54.029 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=1 2024-04-06T10:05:54.029 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=1 2024-04-06T10:05:54.029 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-06T10:05:54.051 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T10:05:54.052 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T10:05:54.052 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T10:05:54.052 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/image_38 2024-04-06T10:05:54.208 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T10:05:54.208 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror image_38 2024-04-06T10:05:54.209 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:05:54.209 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:05:54.209 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_38 2024-04-06T10:05:54.209 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror image_38 '' 2024-04-06T10:05:54.209 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:05:54.209 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:05:54.209 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_38 2024-04-06T10:05:54.209 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T10:05:54.210 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-image_38.status 2024-04-06T10:05:54.210 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T10:05:54.211 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-image_38.status 2024-04-06T10:05:54.212 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-image_38.status 2024-04-06T10:05:54.212 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_38.status 2024-04-06T10:05:54.212 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_38 2024-04-06T10:05:54.265 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 2 2024-04-06T10:05:54.265 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 2 2024-04-06T10:05:54.265 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T10:05:54.265 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=9, tag_tid=2, entry_tid=1], [object_number=8, tag_tid=2, entry_tid=0], [object_number=11, tag_tid=1, entry_tid=10135], [object_number=10, tag_tid=1, entry_tid=10134]]], state=connected] 2024-04-06T10:05:54.265 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=8, tag_tid=1, entry_tid=10136], [object_number=11, tag_tid=1, entry_tid=10135], [object_number=10, tag_tid=1, entry_tid=10134], [object_number=9, tag_tid=1, entry_tid=10133]]], state=connected] 2024-04-06T10:05:54.268 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_38.status 2024-04-06T10:05:54.270 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=9, tag_tid=2, entry_tid=1' 2024-04-06T10:05:54.271 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror image_38 2024-04-06T10:05:54.271 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:05:54.271 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:05:54.271 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_38 2024-04-06T10:05:54.271 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror image_38 '..*' 2024-04-06T10:05:54.272 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:05:54.272 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:05:54.272 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_38 2024-04-06T10:05:54.272 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T10:05:54.272 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-image_38.status 2024-04-06T10:05:54.272 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-image_38.status 2024-04-06T10:05:54.272 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T10:05:54.273 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-image_38.status 2024-04-06T10:05:54.273 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_38 2024-04-06T10:05:54.273 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_38.status 2024-04-06T10:05:54.319 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 2 2024-04-06T10:05:54.319 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 2 2024-04-06T10:05:54.319 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T10:05:54.319 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=9, tag_tid=2, entry_tid=1], [object_number=8, tag_tid=2, entry_tid=0], [object_number=11, tag_tid=1, entry_tid=10135], [object_number=10, tag_tid=1, entry_tid=10134]]], state=connected] 2024-04-06T10:05:54.319 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=8, tag_tid=1, entry_tid=10136], [object_number=11, tag_tid=1, entry_tid=10135], [object_number=10, tag_tid=1, entry_tid=10134], [object_number=9, tag_tid=1, entry_tid=10133]]], state=connected] 2024-04-06T10:05:54.323 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_38.status 2024-04-06T10:05:54.324 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=8, tag_tid=1, entry_tid=10136' 2024-04-06T10:05:54.324 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=9, tag_tid=2, entry_tid=1' -a 'object_number=9, tag_tid=2, entry_tid=1' = 'object_number=8, tag_tid=1, entry_tid=10136' 2024-04-06T10:05:54.325 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=8, tag_tid=1, entry_tid=10136' '!=' 'object_number=8, tag_tid=1, entry_tid=10136' 2024-04-06T10:05:54.325 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T10:05:54.325 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.8 2024-04-06T10:05:55.126 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror image_38 2024-04-06T10:05:55.126 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T10:05:55.126 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:05:55.126 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_38 2024-04-06T10:05:55.127 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T10:05:55.127 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n image_38 ']' 2024-04-06T10:05:55.127 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/image_38' 2024-04-06T10:05:55.127 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_38 2024-04-06T10:05:55.127 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T10:05:55.127 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T10:05:55.127 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T10:05:55.127 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T10:05:55.127 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T10:05:55.127 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T10:05:55.127 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T10:05:55.128 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T10:05:55.128 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T10:05:55.130 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T10:05:55.130 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_38 2024-04-06T10:05:55.130 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T10:05:55.130 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T10:05:55.130 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T10:05:55.130 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T10:05:55.130 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T10:05:55.130 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T10:05:55.130 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T10:05:55.131 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T10:05:55.131 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T10:05:55.131 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T10:05:55.131 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T10:05:55.131 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:05:55.131 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T10:05:55.131 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T10:05:55.132 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T10:05:55.132 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T10:05:55.132 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T10:05:55.132 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T10:05:55.132 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T10:05:55.133 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T10:05:55.133 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T10:05:55.133 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T10:05:55.133 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:05:55.133 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T10:05:55.133 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T10:05:55.134 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T10:05:55.134 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T10:05:55.134 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T10:05:55.134 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T10:05:55.134 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T10:05:55.157 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T10:05:55.158 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T10:05:55.158 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T10:05:55.158 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_38 2024-04-06T10:05:55.261 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:Invalid command: unused arguments: ['mirror/image_38'] 2024-04-06T10:05:55.261 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:rbd mirror flush : flush rbd mirror 2024-04-06T10:05:55.261 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:admin_socket: invalid command 2024-04-06T10:05:55.263 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T10:05:55.263 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/image_38 2024-04-06T10:05:55.263 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:1 2024-04-06T10:05:55.263 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T10:05:55.263 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T10:05:55.263 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-06T10:05:55.263 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:1 2024-04-06T10:05:55.263 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T10:05:55.264 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T10:05:55.264 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T10:05:55.264 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=1 2024-04-06T10:05:55.264 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-06T10:05:55.264 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T10:05:55.264 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:05:55.264 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=1 2024-04-06T10:05:55.264 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=1 2024-04-06T10:05:55.266 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-06T10:05:55.266 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:1 2024-04-06T10:05:55.266 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T10:05:55.266 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T10:05:55.266 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-06T10:05:55.266 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:1 2024-04-06T10:05:55.266 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T10:05:55.267 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T10:05:55.267 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:05:55.267 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=1 2024-04-06T10:05:55.267 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-06T10:05:55.267 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T10:05:55.267 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T10:05:55.267 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=1 2024-04-06T10:05:55.267 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=1 2024-04-06T10:05:55.267 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-06T10:05:55.288 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T10:05:55.288 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T10:05:55.288 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T10:05:55.288 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/image_38 2024-04-06T10:05:55.437 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T10:05:55.438 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror image_38 2024-04-06T10:05:55.438 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:05:55.438 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:05:55.438 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_38 2024-04-06T10:05:55.438 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror image_38 '' 2024-04-06T10:05:55.438 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:05:55.438 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:05:55.439 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_38 2024-04-06T10:05:55.439 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T10:05:55.439 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-image_38.status 2024-04-06T10:05:55.439 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-image_38.status 2024-04-06T10:05:55.439 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T10:05:55.441 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-image_38.status 2024-04-06T10:05:55.442 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_38 2024-04-06T10:05:55.442 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_38.status 2024-04-06T10:05:55.494 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 2 2024-04-06T10:05:55.494 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 2 2024-04-06T10:05:55.494 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T10:05:55.494 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=9, tag_tid=2, entry_tid=1], [object_number=8, tag_tid=2, entry_tid=0], [object_number=11, tag_tid=1, entry_tid=10135], [object_number=10, tag_tid=1, entry_tid=10134]]], state=connected] 2024-04-06T10:05:55.494 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=8, tag_tid=1, entry_tid=10136], [object_number=11, tag_tid=1, entry_tid=10135], [object_number=10, tag_tid=1, entry_tid=10134], [object_number=9, tag_tid=1, entry_tid=10133]]], state=connected] 2024-04-06T10:05:55.498 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_38.status 2024-04-06T10:05:55.499 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=9, tag_tid=2, entry_tid=1' 2024-04-06T10:05:55.500 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror image_38 2024-04-06T10:05:55.500 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:05:55.500 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:05:55.500 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_38 2024-04-06T10:05:55.501 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror image_38 '..*' 2024-04-06T10:05:55.501 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:05:55.501 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:05:55.501 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_38 2024-04-06T10:05:55.501 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T10:05:55.501 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-image_38.status 2024-04-06T10:05:55.501 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-image_38.status 2024-04-06T10:05:55.501 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T10:05:55.502 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-image_38.status 2024-04-06T10:05:55.502 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_38 2024-04-06T10:05:55.502 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_38.status 2024-04-06T10:05:55.547 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 2 2024-04-06T10:05:55.547 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 2 2024-04-06T10:05:55.547 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T10:05:55.547 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=9, tag_tid=2, entry_tid=1], [object_number=8, tag_tid=2, entry_tid=0], [object_number=11, tag_tid=1, entry_tid=10135], [object_number=10, tag_tid=1, entry_tid=10134]]], state=connected] 2024-04-06T10:05:55.547 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=8, tag_tid=1, entry_tid=10136], [object_number=11, tag_tid=1, entry_tid=10135], [object_number=10, tag_tid=1, entry_tid=10134], [object_number=9, tag_tid=1, entry_tid=10133]]], state=connected] 2024-04-06T10:05:55.550 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_38.status 2024-04-06T10:05:55.552 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=8, tag_tid=1, entry_tid=10136' 2024-04-06T10:05:55.552 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=9, tag_tid=2, entry_tid=1' -a 'object_number=9, tag_tid=2, entry_tid=1' = 'object_number=8, tag_tid=1, entry_tid=10136' 2024-04-06T10:05:55.552 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=8, tag_tid=1, entry_tid=10136' '!=' 'object_number=8, tag_tid=1, entry_tid=10136' 2024-04-06T10:05:55.552 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T10:05:55.553 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 1.6 2024-04-06T10:05:57.155 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror image_38 2024-04-06T10:05:57.155 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T10:05:57.155 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:05:57.155 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_38 2024-04-06T10:05:57.155 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T10:05:57.155 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n image_38 ']' 2024-04-06T10:05:57.156 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/image_38' 2024-04-06T10:05:57.156 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_38 2024-04-06T10:05:57.156 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T10:05:57.156 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T10:05:57.156 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T10:05:57.156 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T10:05:57.156 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T10:05:57.156 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T10:05:57.156 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T10:05:57.156 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T10:05:57.158 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T10:05:57.158 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T10:05:57.158 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_38 2024-04-06T10:05:57.158 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T10:05:57.158 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T10:05:57.158 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T10:05:57.158 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T10:05:57.158 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T10:05:57.158 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T10:05:57.159 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T10:05:57.159 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T10:05:57.159 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T10:05:57.159 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T10:05:57.159 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T10:05:57.159 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:05:57.161 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T10:05:57.161 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T10:05:57.161 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T10:05:57.161 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T10:05:57.161 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T10:05:57.161 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T10:05:57.161 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T10:05:57.161 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T10:05:57.161 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T10:05:57.161 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T10:05:57.162 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:05:57.162 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T10:05:57.162 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T10:05:57.162 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T10:05:57.162 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T10:05:57.162 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T10:05:57.162 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T10:05:57.162 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T10:05:57.184 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T10:05:57.184 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T10:05:57.185 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T10:05:57.185 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_38 2024-04-06T10:05:57.290 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:Invalid command: unused arguments: ['mirror/image_38'] 2024-04-06T10:05:57.290 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:rbd mirror flush : flush rbd mirror 2024-04-06T10:05:57.290 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:admin_socket: invalid command 2024-04-06T10:05:57.297 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T10:05:57.298 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/image_38 2024-04-06T10:05:57.298 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:1 2024-04-06T10:05:57.298 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T10:05:57.298 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T10:05:57.298 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-06T10:05:57.298 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:1 2024-04-06T10:05:57.299 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T10:05:57.299 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T10:05:57.299 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T10:05:57.299 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=1 2024-04-06T10:05:57.299 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-06T10:05:57.299 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T10:05:57.299 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:05:57.300 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=1 2024-04-06T10:05:57.300 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=1 2024-04-06T10:05:57.300 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-06T10:05:57.300 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:1 2024-04-06T10:05:57.300 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T10:05:57.300 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T10:05:57.301 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-06T10:05:57.301 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:1 2024-04-06T10:05:57.301 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T10:05:57.301 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T10:05:57.301 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:05:57.301 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=1 2024-04-06T10:05:57.302 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-06T10:05:57.302 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T10:05:57.302 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T10:05:57.302 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=1 2024-04-06T10:05:57.302 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=1 2024-04-06T10:05:57.302 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-06T10:05:57.319 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T10:05:57.319 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T10:05:57.319 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T10:05:57.319 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/image_38 2024-04-06T10:05:57.463 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T10:05:57.465 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror image_38 2024-04-06T10:05:57.465 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:05:57.465 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:05:57.465 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_38 2024-04-06T10:05:57.465 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror image_38 '' 2024-04-06T10:05:57.465 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:05:57.465 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:05:57.465 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_38 2024-04-06T10:05:57.465 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T10:05:57.465 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-image_38.status 2024-04-06T10:05:57.466 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T10:05:57.466 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-image_38.status 2024-04-06T10:05:57.467 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-image_38.status 2024-04-06T10:05:57.467 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_38 2024-04-06T10:05:57.467 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_38.status 2024-04-06T10:05:57.519 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 2 2024-04-06T10:05:57.519 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 2 2024-04-06T10:05:57.519 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T10:05:57.520 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=9, tag_tid=2, entry_tid=1], [object_number=8, tag_tid=2, entry_tid=0], [object_number=11, tag_tid=1, entry_tid=10135], [object_number=10, tag_tid=1, entry_tid=10134]]], state=connected] 2024-04-06T10:05:57.520 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=9, tag_tid=2, entry_tid=1], [object_number=8, tag_tid=2, entry_tid=0], [object_number=11, tag_tid=1, entry_tid=10135], [object_number=10, tag_tid=1, entry_tid=10134]]], state=connected] 2024-04-06T10:05:57.523 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_38.status 2024-04-06T10:05:57.524 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=9, tag_tid=2, entry_tid=1' 2024-04-06T10:05:57.526 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror image_38 2024-04-06T10:05:57.526 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:05:57.526 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:05:57.526 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_38 2024-04-06T10:05:57.526 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror image_38 '..*' 2024-04-06T10:05:57.526 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:05:57.526 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:05:57.527 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_38 2024-04-06T10:05:57.527 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T10:05:57.527 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-image_38.status 2024-04-06T10:05:57.527 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-image_38.status 2024-04-06T10:05:57.527 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T10:05:57.528 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-image_38.status 2024-04-06T10:05:57.528 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_38 2024-04-06T10:05:57.528 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_38.status 2024-04-06T10:05:57.572 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 2 2024-04-06T10:05:57.573 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 2 2024-04-06T10:05:57.573 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T10:05:57.573 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=9, tag_tid=2, entry_tid=1], [object_number=8, tag_tid=2, entry_tid=0], [object_number=11, tag_tid=1, entry_tid=10135], [object_number=10, tag_tid=1, entry_tid=10134]]], state=connected] 2024-04-06T10:05:57.573 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=9, tag_tid=2, entry_tid=1], [object_number=8, tag_tid=2, entry_tid=0], [object_number=11, tag_tid=1, entry_tid=10135], [object_number=10, tag_tid=1, entry_tid=10134]]], state=connected] 2024-04-06T10:05:57.576 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_38.status 2024-04-06T10:05:57.578 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=9, tag_tid=2, entry_tid=1' 2024-04-06T10:05:57.578 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=9, tag_tid=2, entry_tid=1' -a 'object_number=9, tag_tid=2, entry_tid=1' = 'object_number=9, tag_tid=2, entry_tid=1' 2024-04-06T10:05:57.578 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T10:05:57.578 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_snap_present cluster1 mirror image_38 snap 2024-04-06T10:05:57.578 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T10:05:57.578 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:05:57.579 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_38 2024-04-06T10:05:57.579 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local snap_name=snap 2024-04-06T10:05:57.579 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local s 2024-04-06T10:05:57.579 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-04-06T10:05:57.579 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 1 2024-04-06T10:05:58.579 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 info mirror/image_38@snap 2024-04-06T10:05:58.645 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:rbd image 'image_38': 2024-04-06T10:05:58.645 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: size 128 MiB in 32 objects 2024-04-06T10:05:58.645 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: order 22 (4 MiB objects) 2024-04-06T10:05:58.645 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: snapshot_count: 1 2024-04-06T10:05:58.645 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: id: 10acf1892b15 2024-04-06T10:05:58.645 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: block_name_prefix: rbd_data.10acf1892b15 2024-04-06T10:05:58.645 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: format: 2 2024-04-06T10:05:58.645 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: features: layering, exclusive-lock, journaling 2024-04-06T10:05:58.646 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: op_features: 2024-04-06T10:05:58.646 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: flags: 2024-04-06T10:05:58.646 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: create_timestamp: Sat Apr 6 09:56:19 2024 2024-04-06T10:05:58.646 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: access_timestamp: Sat Apr 6 09:56:19 2024 2024-04-06T10:05:58.646 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: modify_timestamp: Sat Apr 6 09:57:20 2024 2024-04-06T10:05:58.646 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: protected: False 2024-04-06T10:05:58.646 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: journal: 10acf1892b15 2024-04-06T10:05:58.646 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: mirroring state: enabled 2024-04-06T10:05:58.646 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: mirroring mode: journal 2024-04-06T10:05:58.646 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: mirroring global id: 016a21f1-458d-4a7f-9377-a9b80e8c9abe 2024-04-06T10:05:58.646 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: mirroring primary: false 2024-04-06T10:05:58.650 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T10:05:58.650 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ compare_image_snaps mirror image_38 snap 2024-04-06T10:05:58.651 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:05:58.651 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_38 2024-04-06T10:05:58.651 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local snap_name=snap 2024-04-06T10:05:58.651 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local ret=0 2024-04-06T10:05:58.651 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local rmt_export=/tmp/tmp.BhW6xKmROU/cluster2-mirror-image_38.export 2024-04-06T10:05:58.651 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loc_export=/tmp/tmp.BhW6xKmROU/cluster1-mirror-image_38.export 2024-04-06T10:05:58.651 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rm -f /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_38.export /tmp/tmp.BhW6xKmROU/cluster1-mirror-image_38.export 2024-04-06T10:05:58.651 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster2 -p mirror export image_38@snap /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_38.export 2024-04-06T10:05:58.831 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-04-06T10:05:58.840 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 -p mirror export image_38@snap /tmp/tmp.BhW6xKmROU/cluster1-mirror-image_38.export 2024-04-06T10:05:59.285 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-04-06T10:05:59.300 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmp /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_38.export /tmp/tmp.BhW6xKmROU/cluster1-mirror-image_38.export 2024-04-06T10:05:59.378 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rm -f /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_38.export /tmp/tmp.BhW6xKmROU/cluster1-mirror-image_38.export 2024-04-06T10:05:59.420 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T10:05:59.421 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-06T10:05:59.421 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ image=image_39 2024-04-06T10:05:59.421 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ create_snap cluster2 mirror image_39 snap 2024-04-06T10:05:59.421 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-04-06T10:05:59.421 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:05:59.421 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_39 2024-04-06T10:05:59.421 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local snap_name=snap 2024-04-06T10:05:59.421 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster2 -p mirror snap create image_39@snap --debug-rbd=20 --debug-journaler=20 2024-04-06T10:06:02.927 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_image_replay_started cluster1 mirror image_39 2024-04-06T10:06:02.928 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T10:06:02.928 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:06:02.928 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_39 2024-04-06T10:06:02.928 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_image_replay_state cluster1 mirror image_39 started 2024-04-06T10:06:02.928 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T10:06:02.928 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:06:02.928 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_39 2024-04-06T10:06:02.928 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state=started 2024-04-06T10:06:02.928 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local s 2024-04-06T10:06:02.928 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T10:06:02.928 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 1 2024-04-06T10:06:03.929 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test_image_replay_state cluster1 mirror image_39 started 2024-04-06T10:06:03.929 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T10:06:03.929 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:06:03.930 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_39 2024-04-06T10:06:03.930 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local test_state=started 2024-04-06T10:06:03.930 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local status_result 2024-04-06T10:06:03.930 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local current_state=stopped 2024-04-06T10:06:03.931 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_39 2024-04-06T10:06:03.931 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_instance=cluster1 2024-04-06T10:06:03.931 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -i state 2024-04-06T10:06:03.931 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ shift 2024-04-06T10:06:03.931 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T10:06:03.931 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance=cluster1 2024-04-06T10:06:03.931 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local loop_instance 2024-04-06T10:06:03.931 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T10:06:03.932 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sleep 0 2024-04-06T10:06:03.932 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-04-06T10:06:03.932 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ seq 0 3 2024-04-06T10:06:03.936 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T10:06:03.936 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_39 2024-04-06T10:06:03.936 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1:0 2024-04-06T10:06:03.936 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ shift 2024-04-06T10:06:03.936 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T10:06:03.937 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-04-06T10:06:03.937 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T10:06:03.937 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=cluster 2024-04-06T10:06:03.937 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T10:06:03.937 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:06:03.937 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T10:06:03.937 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T10:06:03.937 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval cluster=cluster1 2024-04-06T10:06:03.937 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ cluster=cluster1 2024-04-06T10:06:03.937 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T10:06:03.937 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T10:06:03.937 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-04-06T10:06:03.937 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local local_cluster=cluster1:0 2024-04-06T10:06:03.937 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local cluster=cluster1 2024-04-06T10:06:03.937 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local instance 2024-04-06T10:06:03.937 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T10:06:03.938 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local val=cluster1:0 2024-04-06T10:06:03.938 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local cluster_var_name=local_cluster 2024-04-06T10:06:03.938 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local instance_var_name=instance 2024-04-06T10:06:03.938 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ cluster=cluster1 2024-04-06T10:06:03.938 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T10:06:03.938 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ '[' 0 = cluster1:0 ']' 2024-04-06T10:06:03.938 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ eval local_cluster=cluster1 2024-04-06T10:06:03.938 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++++ local_cluster=cluster1 2024-04-06T10:06:03.938 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ eval instance=0 2024-04-06T10:06:03.938 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++++ instance=0 2024-04-06T10:06:03.938 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T10:06:03.958 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T10:06:03.959 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T10:06:03.959 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T10:06:03.959 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_39 2024-04-06T10:06:04.062 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:Invalid command: unused arguments: ['mirror/image_39'] 2024-04-06T10:06:04.062 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:rbd mirror status : get status for rbd mirror 2024-04-06T10:06:04.062 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:admin_socket: invalid command 2024-04-06T10:06:04.066 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T10:06:04.066 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_39 2024-04-06T10:06:04.066 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1:1 2024-04-06T10:06:04.066 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ shift 2024-04-06T10:06:04.066 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T10:06:04.066 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-04-06T10:06:04.066 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:1 2024-04-06T10:06:04.066 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=cluster 2024-04-06T10:06:04.066 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T10:06:04.066 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:06:04.066 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=1 2024-04-06T10:06:04.067 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-06T10:06:04.067 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval cluster=cluster1 2024-04-06T10:06:04.067 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ cluster=cluster1 2024-04-06T10:06:04.067 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=1 2024-04-06T10:06:04.067 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=1 2024-04-06T10:06:04.067 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-04-06T10:06:04.067 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local local_cluster=cluster1:1 2024-04-06T10:06:04.067 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local cluster=cluster1 2024-04-06T10:06:04.067 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local instance 2024-04-06T10:06:04.067 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-06T10:06:04.067 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local val=cluster1:1 2024-04-06T10:06:04.067 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local cluster_var_name=local_cluster 2024-04-06T10:06:04.067 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local instance_var_name=instance 2024-04-06T10:06:04.067 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ cluster=cluster1 2024-04-06T10:06:04.067 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=1 2024-04-06T10:06:04.067 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ '[' 1 = cluster1:1 ']' 2024-04-06T10:06:04.068 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ eval local_cluster=cluster1 2024-04-06T10:06:04.068 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++++ local_cluster=cluster1 2024-04-06T10:06:04.068 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ eval instance=1 2024-04-06T10:06:04.068 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++++ instance=1 2024-04-06T10:06:04.068 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-06T10:06:04.090 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T10:06:04.091 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T10:06:04.091 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T10:06:04.091 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_39 2024-04-06T10:06:04.255 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ return 0 2024-04-06T10:06:04.255 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ status_result=' "state": "Replaying"' 2024-04-06T10:06:04.256 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ echo ' "state": "Replaying"' 2024-04-06T10:06:04.256 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep -i Replaying 2024-04-06T10:06:04.257 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: "state": "Replaying" 2024-04-06T10:06:04.259 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ current_state=started 2024-04-06T10:06:04.259 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test started = started 2024-04-06T10:06:04.259 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T10:06:04.260 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_39 2024-04-06T10:06:04.260 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local local_cluster=cluster1 2024-04-06T10:06:04.260 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-04-06T10:06:04.260 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:06:04.260 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_39 2024-04-06T10:06:04.260 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' journal = journal ']' 2024-04-06T10:06:04.260 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_journal_replay_complete cluster1 cluster2 mirror image_39 2024-04-06T10:06:04.260 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local local_cluster=cluster1 2024-04-06T10:06:04.260 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-04-06T10:06:04.260 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:06:04.260 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_39 2024-04-06T10:06:04.260 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local s master_pos mirror_pos last_mirror_pos 2024-04-06T10:06:04.260 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local master_tag master_entry mirror_tag mirror_entry 2024-04-06T10:06:04.261 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T10:06:04.261 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T10:06:04.261 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T10:06:04.460 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror image_39 2024-04-06T10:06:04.461 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T10:06:04.461 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:06:04.461 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_39 2024-04-06T10:06:04.461 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T10:06:04.461 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n image_39 ']' 2024-04-06T10:06:04.461 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/image_39' 2024-04-06T10:06:04.461 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_39 2024-04-06T10:06:04.461 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T10:06:04.461 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T10:06:04.461 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T10:06:04.461 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T10:06:04.461 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T10:06:04.461 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T10:06:04.461 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T10:06:04.462 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T10:06:04.462 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T10:06:04.465 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T10:06:04.465 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_39 2024-04-06T10:06:04.465 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T10:06:04.465 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T10:06:04.465 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T10:06:04.465 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T10:06:04.465 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T10:06:04.465 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T10:06:04.465 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T10:06:04.466 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T10:06:04.466 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T10:06:04.466 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T10:06:04.466 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T10:06:04.466 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:06:04.466 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T10:06:04.466 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T10:06:04.467 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T10:06:04.467 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T10:06:04.467 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T10:06:04.467 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T10:06:04.467 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T10:06:04.467 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T10:06:04.467 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T10:06:04.468 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T10:06:04.468 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:06:04.468 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T10:06:04.468 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T10:06:04.468 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T10:06:04.468 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T10:06:04.468 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T10:06:04.468 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T10:06:04.468 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T10:06:04.491 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T10:06:04.492 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T10:06:04.492 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T10:06:04.492 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_39 2024-04-06T10:06:04.605 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:Invalid command: unused arguments: ['mirror/image_39'] 2024-04-06T10:06:04.605 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:rbd mirror flush : flush rbd mirror 2024-04-06T10:06:04.605 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:admin_socket: invalid command 2024-04-06T10:06:04.607 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T10:06:04.607 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/image_39 2024-04-06T10:06:04.607 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:1 2024-04-06T10:06:04.607 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T10:06:04.607 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T10:06:04.607 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-06T10:06:04.607 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:1 2024-04-06T10:06:04.607 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T10:06:04.608 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T10:06:04.608 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T10:06:04.608 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=1 2024-04-06T10:06:04.608 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-06T10:06:04.608 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T10:06:04.608 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:06:04.608 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=1 2024-04-06T10:06:04.608 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=1 2024-04-06T10:06:04.610 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-06T10:06:04.610 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:1 2024-04-06T10:06:04.610 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T10:06:04.610 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T10:06:04.610 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-06T10:06:04.610 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:1 2024-04-06T10:06:04.610 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T10:06:04.610 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T10:06:04.611 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:06:04.611 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=1 2024-04-06T10:06:04.611 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-06T10:06:04.611 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T10:06:04.611 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T10:06:04.611 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=1 2024-04-06T10:06:04.611 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=1 2024-04-06T10:06:04.611 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-06T10:06:04.633 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T10:06:04.633 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T10:06:04.634 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T10:06:04.634 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/image_39 2024-04-06T10:06:04.777 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T10:06:04.778 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror image_39 2024-04-06T10:06:04.778 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:06:04.778 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:06:04.778 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_39 2024-04-06T10:06:04.778 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror image_39 '' 2024-04-06T10:06:04.778 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:06:04.778 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:06:04.778 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_39 2024-04-06T10:06:04.778 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T10:06:04.779 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-image_39.status 2024-04-06T10:06:04.779 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-image_39.status 2024-04-06T10:06:04.779 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T10:06:04.780 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-image_39.status 2024-04-06T10:06:04.780 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_39 2024-04-06T10:06:04.781 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_39.status 2024-04-06T10:06:04.825 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 1 2024-04-06T10:06:04.826 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 1 2024-04-06T10:06:04.826 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T10:06:04.826 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=5, tag_tid=2, entry_tid=1], [object_number=4, tag_tid=2, entry_tid=0], [object_number=7, tag_tid=1, entry_tid=7719], [object_number=6, tag_tid=1, entry_tid=7718]]], state=connected] 2024-04-06T10:06:04.826 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=4, tag_tid=1, entry_tid=7720], [object_number=7, tag_tid=1, entry_tid=7719], [object_number=6, tag_tid=1, entry_tid=7718], [object_number=5, tag_tid=1, entry_tid=7717]]], state=connected] 2024-04-06T10:06:04.829 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_39.status 2024-04-06T10:06:04.830 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=5, tag_tid=2, entry_tid=1' 2024-04-06T10:06:04.833 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror image_39 2024-04-06T10:06:04.833 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:06:04.834 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:06:04.834 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_39 2024-04-06T10:06:04.834 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror image_39 '..*' 2024-04-06T10:06:04.834 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:06:04.834 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:06:04.835 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_39 2024-04-06T10:06:04.835 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T10:06:04.835 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-image_39.status 2024-04-06T10:06:04.835 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-image_39.status 2024-04-06T10:06:04.835 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T10:06:04.835 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-image_39.status 2024-04-06T10:06:04.836 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_39 2024-04-06T10:06:04.836 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_39.status 2024-04-06T10:06:04.877 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 1 2024-04-06T10:06:04.877 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 1 2024-04-06T10:06:04.877 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T10:06:04.877 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=5, tag_tid=2, entry_tid=1], [object_number=4, tag_tid=2, entry_tid=0], [object_number=7, tag_tid=1, entry_tid=7719], [object_number=6, tag_tid=1, entry_tid=7718]]], state=connected] 2024-04-06T10:06:04.877 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=4, tag_tid=1, entry_tid=7720], [object_number=7, tag_tid=1, entry_tid=7719], [object_number=6, tag_tid=1, entry_tid=7718], [object_number=5, tag_tid=1, entry_tid=7717]]], state=connected] 2024-04-06T10:06:04.881 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_39.status 2024-04-06T10:06:04.883 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=4, tag_tid=1, entry_tid=7720' 2024-04-06T10:06:04.883 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=5, tag_tid=2, entry_tid=1' -a 'object_number=5, tag_tid=2, entry_tid=1' = 'object_number=4, tag_tid=1, entry_tid=7720' 2024-04-06T10:06:04.883 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=4, tag_tid=1, entry_tid=7720' '!=' '' 2024-04-06T10:06:04.883 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T10:06:04.883 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=4, tag_tid=1, entry_tid=7720' = '' 2024-04-06T10:06:04.883 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=4, tag_tid=1, entry_tid=7720' 2024-04-06T10:06:04.883 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=5, tag_tid=2, entry_tid=1' 2024-04-06T10:06:04.883 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T10:06:04.884 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T10:06:04.884 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=2 2024-04-06T10:06:04.885 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=4, tag_tid=1, entry_tid=7720' 2024-04-06T10:06:04.885 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T10:06:04.885 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T10:06:04.886 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=1 2024-04-06T10:06:04.886 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=5, tag_tid=2, entry_tid=1' 2024-04-06T10:06:04.886 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T10:06:04.887 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T10:06:04.887 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T10:06:04.888 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=4, tag_tid=1, entry_tid=7720' 2024-04-06T10:06:04.888 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T10:06:04.888 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T10:06:04.890 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=7720 2024-04-06T10:06:04.890 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 2 = 1 -a 1 -le 7720 2024-04-06T10:06:04.890 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T10:06:04.890 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T10:06:04.890 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T10:06:05.092 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror image_39 2024-04-06T10:06:05.092 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T10:06:05.092 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:06:05.092 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_39 2024-04-06T10:06:05.092 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T10:06:05.092 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n image_39 ']' 2024-04-06T10:06:05.092 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/image_39' 2024-04-06T10:06:05.092 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_39 2024-04-06T10:06:05.092 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T10:06:05.093 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T10:06:05.093 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T10:06:05.093 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T10:06:05.093 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T10:06:05.093 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T10:06:05.093 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T10:06:05.093 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T10:06:05.093 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T10:06:05.095 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T10:06:05.096 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_39 2024-04-06T10:06:05.096 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T10:06:05.096 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T10:06:05.096 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T10:06:05.096 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T10:06:05.096 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T10:06:05.096 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T10:06:05.096 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T10:06:05.096 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T10:06:05.096 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T10:06:05.096 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T10:06:05.096 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T10:06:05.096 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:06:05.097 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T10:06:05.097 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T10:06:05.097 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T10:06:05.097 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T10:06:05.097 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T10:06:05.097 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T10:06:05.097 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T10:06:05.097 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T10:06:05.097 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T10:06:05.097 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T10:06:05.097 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:06:05.097 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T10:06:05.097 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T10:06:05.097 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T10:06:05.098 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T10:06:05.098 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T10:06:05.098 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T10:06:05.098 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T10:06:05.122 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T10:06:05.122 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T10:06:05.122 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T10:06:05.122 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_39 2024-04-06T10:06:05.229 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:Invalid command: unused arguments: ['mirror/image_39'] 2024-04-06T10:06:05.229 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:rbd mirror flush : flush rbd mirror 2024-04-06T10:06:05.229 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:admin_socket: invalid command 2024-04-06T10:06:05.232 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T10:06:05.232 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/image_39 2024-04-06T10:06:05.233 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:1 2024-04-06T10:06:05.233 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T10:06:05.233 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T10:06:05.233 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-06T10:06:05.233 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:1 2024-04-06T10:06:05.233 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T10:06:05.233 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T10:06:05.233 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T10:06:05.233 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=1 2024-04-06T10:06:05.233 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-06T10:06:05.233 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T10:06:05.233 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:06:05.233 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=1 2024-04-06T10:06:05.233 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=1 2024-04-06T10:06:05.233 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-06T10:06:05.234 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:1 2024-04-06T10:06:05.234 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T10:06:05.234 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T10:06:05.234 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-06T10:06:05.234 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:1 2024-04-06T10:06:05.234 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T10:06:05.234 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T10:06:05.234 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:06:05.234 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=1 2024-04-06T10:06:05.234 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-06T10:06:05.234 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T10:06:05.234 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T10:06:05.234 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=1 2024-04-06T10:06:05.234 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=1 2024-04-06T10:06:05.235 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-06T10:06:05.256 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T10:06:05.257 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T10:06:05.257 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T10:06:05.257 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/image_39 2024-04-06T10:06:05.401 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T10:06:05.402 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror image_39 2024-04-06T10:06:05.402 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:06:05.402 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:06:05.402 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_39 2024-04-06T10:06:05.402 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror image_39 '' 2024-04-06T10:06:05.402 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:06:05.402 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:06:05.402 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_39 2024-04-06T10:06:05.402 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T10:06:05.403 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-image_39.status 2024-04-06T10:06:05.403 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-image_39.status 2024-04-06T10:06:05.403 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T10:06:05.405 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-image_39.status 2024-04-06T10:06:05.406 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_39 2024-04-06T10:06:05.406 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_39.status 2024-04-06T10:06:05.463 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 1 2024-04-06T10:06:05.463 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 1 2024-04-06T10:06:05.463 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T10:06:05.463 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=5, tag_tid=2, entry_tid=1], [object_number=4, tag_tid=2, entry_tid=0], [object_number=7, tag_tid=1, entry_tid=7719], [object_number=6, tag_tid=1, entry_tid=7718]]], state=connected] 2024-04-06T10:06:05.463 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=4, tag_tid=1, entry_tid=7720], [object_number=7, tag_tid=1, entry_tid=7719], [object_number=6, tag_tid=1, entry_tid=7718], [object_number=5, tag_tid=1, entry_tid=7717]]], state=connected] 2024-04-06T10:06:05.466 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_39.status 2024-04-06T10:06:05.468 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=5, tag_tid=2, entry_tid=1' 2024-04-06T10:06:05.470 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror image_39 2024-04-06T10:06:05.470 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:06:05.470 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:06:05.470 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_39 2024-04-06T10:06:05.470 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror image_39 '..*' 2024-04-06T10:06:05.470 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:06:05.470 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:06:05.471 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_39 2024-04-06T10:06:05.471 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T10:06:05.471 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-image_39.status 2024-04-06T10:06:05.471 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-image_39.status 2024-04-06T10:06:05.471 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T10:06:05.472 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-image_39.status 2024-04-06T10:06:05.472 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_39 2024-04-06T10:06:05.472 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_39.status 2024-04-06T10:06:05.517 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 1 2024-04-06T10:06:05.517 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 1 2024-04-06T10:06:05.517 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T10:06:05.517 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=5, tag_tid=2, entry_tid=1], [object_number=4, tag_tid=2, entry_tid=0], [object_number=7, tag_tid=1, entry_tid=7719], [object_number=6, tag_tid=1, entry_tid=7718]]], state=connected] 2024-04-06T10:06:05.517 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=4, tag_tid=1, entry_tid=7720], [object_number=7, tag_tid=1, entry_tid=7719], [object_number=6, tag_tid=1, entry_tid=7718], [object_number=5, tag_tid=1, entry_tid=7717]]], state=connected] 2024-04-06T10:06:05.521 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_39.status 2024-04-06T10:06:05.523 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=4, tag_tid=1, entry_tid=7720' 2024-04-06T10:06:05.523 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=5, tag_tid=2, entry_tid=1' -a 'object_number=5, tag_tid=2, entry_tid=1' = 'object_number=4, tag_tid=1, entry_tid=7720' 2024-04-06T10:06:05.523 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=4, tag_tid=1, entry_tid=7720' '!=' 'object_number=4, tag_tid=1, entry_tid=7720' 2024-04-06T10:06:05.523 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T10:06:05.523 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.4 2024-04-06T10:06:05.925 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror image_39 2024-04-06T10:06:05.926 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T10:06:05.926 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:06:05.926 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_39 2024-04-06T10:06:05.926 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T10:06:05.926 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n image_39 ']' 2024-04-06T10:06:05.926 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/image_39' 2024-04-06T10:06:05.926 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_39 2024-04-06T10:06:05.926 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T10:06:05.926 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T10:06:05.926 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T10:06:05.926 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T10:06:05.927 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T10:06:05.927 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T10:06:05.927 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T10:06:05.928 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T10:06:05.928 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T10:06:05.930 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T10:06:05.930 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_39 2024-04-06T10:06:05.930 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T10:06:05.930 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T10:06:05.930 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T10:06:05.930 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T10:06:05.930 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T10:06:05.930 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T10:06:05.930 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T10:06:05.930 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T10:06:05.930 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T10:06:05.931 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T10:06:05.931 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T10:06:05.933 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:06:05.933 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T10:06:05.933 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T10:06:05.933 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T10:06:05.933 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T10:06:05.933 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T10:06:05.933 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T10:06:05.933 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T10:06:05.933 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T10:06:05.933 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T10:06:05.933 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T10:06:05.934 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:06:05.934 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T10:06:05.934 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T10:06:05.934 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T10:06:05.934 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T10:06:05.934 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T10:06:05.934 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T10:06:05.934 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T10:06:05.956 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T10:06:05.956 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T10:06:05.956 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T10:06:05.956 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_39 2024-04-06T10:06:06.064 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:Invalid command: unused arguments: ['mirror/image_39'] 2024-04-06T10:06:06.064 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:rbd mirror flush : flush rbd mirror 2024-04-06T10:06:06.064 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:admin_socket: invalid command 2024-04-06T10:06:06.066 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T10:06:06.066 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/image_39 2024-04-06T10:06:06.066 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:1 2024-04-06T10:06:06.066 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T10:06:06.067 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T10:06:06.067 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-06T10:06:06.067 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:1 2024-04-06T10:06:06.067 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T10:06:06.067 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T10:06:06.067 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T10:06:06.067 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=1 2024-04-06T10:06:06.067 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-06T10:06:06.067 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T10:06:06.067 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:06:06.067 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=1 2024-04-06T10:06:06.068 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=1 2024-04-06T10:06:06.069 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-06T10:06:06.070 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:1 2024-04-06T10:06:06.070 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T10:06:06.070 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T10:06:06.070 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-06T10:06:06.070 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:1 2024-04-06T10:06:06.070 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T10:06:06.070 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T10:06:06.070 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:06:06.070 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=1 2024-04-06T10:06:06.070 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-06T10:06:06.070 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T10:06:06.070 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T10:06:06.070 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=1 2024-04-06T10:06:06.071 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=1 2024-04-06T10:06:06.071 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-06T10:06:06.094 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T10:06:06.094 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T10:06:06.094 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T10:06:06.094 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/image_39 2024-04-06T10:06:06.232 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T10:06:06.233 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror image_39 2024-04-06T10:06:06.233 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:06:06.233 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:06:06.233 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_39 2024-04-06T10:06:06.233 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror image_39 '' 2024-04-06T10:06:06.234 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:06:06.234 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:06:06.234 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_39 2024-04-06T10:06:06.234 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T10:06:06.235 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-image_39.status 2024-04-06T10:06:06.235 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T10:06:06.235 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-image_39.status 2024-04-06T10:06:06.236 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-image_39.status 2024-04-06T10:06:06.236 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_39 2024-04-06T10:06:06.236 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_39.status 2024-04-06T10:06:06.292 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 1 2024-04-06T10:06:06.292 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 1 2024-04-06T10:06:06.292 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T10:06:06.292 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=5, tag_tid=2, entry_tid=1], [object_number=4, tag_tid=2, entry_tid=0], [object_number=7, tag_tid=1, entry_tid=7719], [object_number=6, tag_tid=1, entry_tid=7718]]], state=connected] 2024-04-06T10:06:06.292 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=4, tag_tid=1, entry_tid=7720], [object_number=7, tag_tid=1, entry_tid=7719], [object_number=6, tag_tid=1, entry_tid=7718], [object_number=5, tag_tid=1, entry_tid=7717]]], state=connected] 2024-04-06T10:06:06.295 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_39.status 2024-04-06T10:06:06.297 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=5, tag_tid=2, entry_tid=1' 2024-04-06T10:06:06.298 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror image_39 2024-04-06T10:06:06.298 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:06:06.298 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:06:06.298 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_39 2024-04-06T10:06:06.299 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror image_39 '..*' 2024-04-06T10:06:06.299 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:06:06.299 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:06:06.299 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_39 2024-04-06T10:06:06.299 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T10:06:06.299 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-image_39.status 2024-04-06T10:06:06.299 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-image_39.status 2024-04-06T10:06:06.299 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T10:06:06.300 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-image_39.status 2024-04-06T10:06:06.300 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_39 2024-04-06T10:06:06.300 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_39.status 2024-04-06T10:06:06.344 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 1 2024-04-06T10:06:06.344 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 1 2024-04-06T10:06:06.344 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T10:06:06.344 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=5, tag_tid=2, entry_tid=1], [object_number=4, tag_tid=2, entry_tid=0], [object_number=7, tag_tid=1, entry_tid=7719], [object_number=6, tag_tid=1, entry_tid=7718]]], state=connected] 2024-04-06T10:06:06.344 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=4, tag_tid=1, entry_tid=7720], [object_number=7, tag_tid=1, entry_tid=7719], [object_number=6, tag_tid=1, entry_tid=7718], [object_number=5, tag_tid=1, entry_tid=7717]]], state=connected] 2024-04-06T10:06:06.347 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_39.status 2024-04-06T10:06:06.349 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=4, tag_tid=1, entry_tid=7720' 2024-04-06T10:06:06.350 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=5, tag_tid=2, entry_tid=1' -a 'object_number=5, tag_tid=2, entry_tid=1' = 'object_number=4, tag_tid=1, entry_tid=7720' 2024-04-06T10:06:06.350 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=4, tag_tid=1, entry_tid=7720' '!=' 'object_number=4, tag_tid=1, entry_tid=7720' 2024-04-06T10:06:06.350 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T10:06:06.350 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.8 2024-04-06T10:06:07.151 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror image_39 2024-04-06T10:06:07.151 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T10:06:07.151 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:06:07.151 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_39 2024-04-06T10:06:07.151 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T10:06:07.151 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n image_39 ']' 2024-04-06T10:06:07.152 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/image_39' 2024-04-06T10:06:07.152 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_39 2024-04-06T10:06:07.152 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T10:06:07.152 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T10:06:07.152 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T10:06:07.152 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T10:06:07.152 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T10:06:07.152 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T10:06:07.152 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T10:06:07.152 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T10:06:07.153 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T10:06:07.155 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T10:06:07.155 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_39 2024-04-06T10:06:07.155 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T10:06:07.155 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T10:06:07.155 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T10:06:07.155 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T10:06:07.155 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T10:06:07.156 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T10:06:07.156 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T10:06:07.156 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T10:06:07.156 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T10:06:07.156 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T10:06:07.156 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T10:06:07.156 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:06:07.156 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T10:06:07.156 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T10:06:07.156 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T10:06:07.156 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T10:06:07.156 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T10:06:07.156 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T10:06:07.156 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T10:06:07.156 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T10:06:07.156 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T10:06:07.157 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T10:06:07.157 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:06:07.157 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T10:06:07.157 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T10:06:07.157 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T10:06:07.157 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T10:06:07.157 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T10:06:07.157 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T10:06:07.157 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T10:06:07.182 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T10:06:07.183 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T10:06:07.183 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T10:06:07.183 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_39 2024-04-06T10:06:07.287 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:Invalid command: unused arguments: ['mirror/image_39'] 2024-04-06T10:06:07.287 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:rbd mirror flush : flush rbd mirror 2024-04-06T10:06:07.287 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:admin_socket: invalid command 2024-04-06T10:06:07.290 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T10:06:07.290 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/image_39 2024-04-06T10:06:07.290 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:1 2024-04-06T10:06:07.290 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T10:06:07.290 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T10:06:07.290 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-06T10:06:07.290 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:1 2024-04-06T10:06:07.290 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T10:06:07.290 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T10:06:07.290 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T10:06:07.290 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=1 2024-04-06T10:06:07.290 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-06T10:06:07.291 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T10:06:07.291 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:06:07.291 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=1 2024-04-06T10:06:07.291 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=1 2024-04-06T10:06:07.291 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-06T10:06:07.291 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:1 2024-04-06T10:06:07.291 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T10:06:07.291 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T10:06:07.291 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-06T10:06:07.291 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:1 2024-04-06T10:06:07.291 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T10:06:07.291 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T10:06:07.291 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:06:07.291 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=1 2024-04-06T10:06:07.292 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-06T10:06:07.292 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T10:06:07.292 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T10:06:07.292 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=1 2024-04-06T10:06:07.292 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=1 2024-04-06T10:06:07.292 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-06T10:06:07.315 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T10:06:07.315 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T10:06:07.315 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T10:06:07.315 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/image_39 2024-04-06T10:06:07.462 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T10:06:07.463 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror image_39 2024-04-06T10:06:07.463 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:06:07.463 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:06:07.463 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_39 2024-04-06T10:06:07.463 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror image_39 '' 2024-04-06T10:06:07.463 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:06:07.463 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:06:07.463 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_39 2024-04-06T10:06:07.463 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T10:06:07.464 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-image_39.status 2024-04-06T10:06:07.464 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-image_39.status 2024-04-06T10:06:07.464 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T10:06:07.466 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-image_39.status 2024-04-06T10:06:07.467 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_39 2024-04-06T10:06:07.467 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_39.status 2024-04-06T10:06:07.723 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 1 2024-04-06T10:06:07.723 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 1 2024-04-06T10:06:07.723 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T10:06:07.723 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=5, tag_tid=2, entry_tid=1], [object_number=4, tag_tid=2, entry_tid=0], [object_number=7, tag_tid=1, entry_tid=7719], [object_number=6, tag_tid=1, entry_tid=7718]]], state=connected] 2024-04-06T10:06:07.723 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=5, tag_tid=2, entry_tid=1], [object_number=4, tag_tid=2, entry_tid=0], [object_number=7, tag_tid=1, entry_tid=7719], [object_number=6, tag_tid=1, entry_tid=7718]]], state=connected] 2024-04-06T10:06:07.728 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_39.status 2024-04-06T10:06:07.730 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=5, tag_tid=2, entry_tid=1' 2024-04-06T10:06:07.731 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror image_39 2024-04-06T10:06:07.731 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:06:07.731 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:06:07.732 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_39 2024-04-06T10:06:07.732 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror image_39 '..*' 2024-04-06T10:06:07.732 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:06:07.732 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:06:07.732 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_39 2024-04-06T10:06:07.732 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T10:06:07.732 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-image_39.status 2024-04-06T10:06:07.733 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-image_39.status 2024-04-06T10:06:07.733 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T10:06:07.734 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-image_39.status 2024-04-06T10:06:07.734 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_39 2024-04-06T10:06:07.734 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_39.status 2024-04-06T10:06:07.787 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 1 2024-04-06T10:06:07.787 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 1 2024-04-06T10:06:07.788 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T10:06:07.788 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=5, tag_tid=2, entry_tid=1], [object_number=4, tag_tid=2, entry_tid=0], [object_number=7, tag_tid=1, entry_tid=7719], [object_number=6, tag_tid=1, entry_tid=7718]]], state=connected] 2024-04-06T10:06:07.788 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=5, tag_tid=2, entry_tid=1], [object_number=4, tag_tid=2, entry_tid=0], [object_number=7, tag_tid=1, entry_tid=7719], [object_number=6, tag_tid=1, entry_tid=7718]]], state=connected] 2024-04-06T10:06:07.791 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_39.status 2024-04-06T10:06:07.794 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=5, tag_tid=2, entry_tid=1' 2024-04-06T10:06:07.794 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=5, tag_tid=2, entry_tid=1' -a 'object_number=5, tag_tid=2, entry_tid=1' = 'object_number=5, tag_tid=2, entry_tid=1' 2024-04-06T10:06:07.794 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T10:06:07.794 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_snap_present cluster1 mirror image_39 snap 2024-04-06T10:06:07.794 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T10:06:07.794 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:06:07.794 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_39 2024-04-06T10:06:07.794 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local snap_name=snap 2024-04-06T10:06:07.794 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local s 2024-04-06T10:06:07.794 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-04-06T10:06:07.795 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 1 2024-04-06T10:06:08.795 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 info mirror/image_39@snap 2024-04-06T10:06:08.857 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:rbd image 'image_39': 2024-04-06T10:06:08.858 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: size 128 MiB in 32 objects 2024-04-06T10:06:08.858 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: order 22 (4 MiB objects) 2024-04-06T10:06:08.858 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: snapshot_count: 1 2024-04-06T10:06:08.858 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: id: 10ac6c1eaa48 2024-04-06T10:06:08.858 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: block_name_prefix: rbd_data.10ac6c1eaa48 2024-04-06T10:06:08.858 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: format: 2 2024-04-06T10:06:08.858 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: features: layering, exclusive-lock, journaling 2024-04-06T10:06:08.858 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: op_features: 2024-04-06T10:06:08.858 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: flags: 2024-04-06T10:06:08.858 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: create_timestamp: Sat Apr 6 09:56:32 2024 2024-04-06T10:06:08.858 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: access_timestamp: Sat Apr 6 09:56:32 2024 2024-04-06T10:06:08.858 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: modify_timestamp: Sat Apr 6 09:57:33 2024 2024-04-06T10:06:08.859 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: protected: False 2024-04-06T10:06:08.859 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: journal: 10ac6c1eaa48 2024-04-06T10:06:08.859 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: mirroring state: enabled 2024-04-06T10:06:08.859 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: mirroring mode: journal 2024-04-06T10:06:08.859 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: mirroring global id: d3d1e3a2-4afe-4a34-9227-1a450ffbace6 2024-04-06T10:06:08.859 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: mirroring primary: false 2024-04-06T10:06:08.862 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T10:06:08.863 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ compare_image_snaps mirror image_39 snap 2024-04-06T10:06:08.863 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:06:08.863 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_39 2024-04-06T10:06:08.863 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local snap_name=snap 2024-04-06T10:06:08.863 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local ret=0 2024-04-06T10:06:08.863 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local rmt_export=/tmp/tmp.BhW6xKmROU/cluster2-mirror-image_39.export 2024-04-06T10:06:08.863 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loc_export=/tmp/tmp.BhW6xKmROU/cluster1-mirror-image_39.export 2024-04-06T10:06:08.863 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rm -f /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_39.export /tmp/tmp.BhW6xKmROU/cluster1-mirror-image_39.export 2024-04-06T10:06:08.863 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster2 -p mirror export image_39@snap /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_39.export 2024-04-06T10:06:09.029 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-04-06T10:06:09.038 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 -p mirror export image_39@snap /tmp/tmp.BhW6xKmROU/cluster1-mirror-image_39.export 2024-04-06T10:06:09.394 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-04-06T10:06:09.409 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmp /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_39.export /tmp/tmp.BhW6xKmROU/cluster1-mirror-image_39.export 2024-04-06T10:06:09.493 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rm -f /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_39.export /tmp/tmp.BhW6xKmROU/cluster1-mirror-image_39.export 2024-04-06T10:06:09.536 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T10:06:09.536 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-06T10:06:09.536 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ image=image_40 2024-04-06T10:06:09.536 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ create_snap cluster2 mirror image_40 snap 2024-04-06T10:06:09.537 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-04-06T10:06:09.537 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:06:09.537 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_40 2024-04-06T10:06:09.537 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local snap_name=snap 2024-04-06T10:06:09.537 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster2 -p mirror snap create image_40@snap --debug-rbd=20 --debug-journaler=20 2024-04-06T10:06:09.913 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_image_replay_started cluster1 mirror image_40 2024-04-06T10:06:09.914 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T10:06:09.914 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:06:09.914 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_40 2024-04-06T10:06:09.914 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_image_replay_state cluster1 mirror image_40 started 2024-04-06T10:06:09.914 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T10:06:09.914 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:06:09.914 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_40 2024-04-06T10:06:09.914 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state=started 2024-04-06T10:06:09.914 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local s 2024-04-06T10:06:09.914 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T10:06:09.915 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 1 2024-04-06T10:06:10.915 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test_image_replay_state cluster1 mirror image_40 started 2024-04-06T10:06:10.915 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T10:06:10.916 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:06:10.916 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_40 2024-04-06T10:06:10.916 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local test_state=started 2024-04-06T10:06:10.916 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local status_result 2024-04-06T10:06:10.916 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local current_state=stopped 2024-04-06T10:06:10.917 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_40 2024-04-06T10:06:10.917 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -i state 2024-04-06T10:06:10.917 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_instance=cluster1 2024-04-06T10:06:10.917 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ shift 2024-04-06T10:06:10.917 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T10:06:10.917 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance=cluster1 2024-04-06T10:06:10.917 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local loop_instance 2024-04-06T10:06:10.917 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T10:06:10.917 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sleep 0 2024-04-06T10:06:10.918 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-04-06T10:06:10.918 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ seq 0 3 2024-04-06T10:06:10.921 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T10:06:10.921 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_40 2024-04-06T10:06:10.922 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1:0 2024-04-06T10:06:10.922 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ shift 2024-04-06T10:06:10.922 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T10:06:10.922 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-04-06T10:06:10.922 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T10:06:10.923 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=cluster 2024-04-06T10:06:10.923 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T10:06:10.923 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:06:10.923 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T10:06:10.923 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T10:06:10.923 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval cluster=cluster1 2024-04-06T10:06:10.924 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ cluster=cluster1 2024-04-06T10:06:10.924 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T10:06:10.924 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T10:06:10.924 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-04-06T10:06:10.924 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local local_cluster=cluster1:0 2024-04-06T10:06:10.924 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local cluster=cluster1 2024-04-06T10:06:10.925 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local instance 2024-04-06T10:06:10.925 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T10:06:10.925 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local val=cluster1:0 2024-04-06T10:06:10.925 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local cluster_var_name=local_cluster 2024-04-06T10:06:10.925 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local instance_var_name=instance 2024-04-06T10:06:10.926 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ cluster=cluster1 2024-04-06T10:06:10.926 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T10:06:10.926 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ '[' 0 = cluster1:0 ']' 2024-04-06T10:06:10.926 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ eval local_cluster=cluster1 2024-04-06T10:06:10.926 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++++ local_cluster=cluster1 2024-04-06T10:06:10.926 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ eval instance=0 2024-04-06T10:06:10.927 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++++ instance=0 2024-04-06T10:06:10.928 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T10:06:10.944 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T10:06:10.944 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T10:06:10.944 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T10:06:10.944 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_40 2024-04-06T10:06:11.058 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:Invalid command: unused arguments: ['mirror/image_40'] 2024-04-06T10:06:11.058 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:rbd mirror status : get status for rbd mirror 2024-04-06T10:06:11.058 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:admin_socket: invalid command 2024-04-06T10:06:11.064 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T10:06:11.064 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_40 2024-04-06T10:06:11.064 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1:1 2024-04-06T10:06:11.064 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ shift 2024-04-06T10:06:11.064 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T10:06:11.065 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-04-06T10:06:11.065 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:1 2024-04-06T10:06:11.065 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=cluster 2024-04-06T10:06:11.065 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T10:06:11.065 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:06:11.065 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=1 2024-04-06T10:06:11.066 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-06T10:06:11.066 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval cluster=cluster1 2024-04-06T10:06:11.066 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ cluster=cluster1 2024-04-06T10:06:11.067 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=1 2024-04-06T10:06:11.067 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=1 2024-04-06T10:06:11.067 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-04-06T10:06:11.067 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local local_cluster=cluster1:1 2024-04-06T10:06:11.067 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local cluster=cluster1 2024-04-06T10:06:11.067 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local instance 2024-04-06T10:06:11.068 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-06T10:06:11.068 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local val=cluster1:1 2024-04-06T10:06:11.068 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local cluster_var_name=local_cluster 2024-04-06T10:06:11.068 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local instance_var_name=instance 2024-04-06T10:06:11.069 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ cluster=cluster1 2024-04-06T10:06:11.069 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=1 2024-04-06T10:06:11.069 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ '[' 1 = cluster1:1 ']' 2024-04-06T10:06:11.069 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ eval local_cluster=cluster1 2024-04-06T10:06:11.069 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++++ local_cluster=cluster1 2024-04-06T10:06:11.069 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ eval instance=1 2024-04-06T10:06:11.070 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++++ instance=1 2024-04-06T10:06:11.070 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-06T10:06:11.080 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T10:06:11.081 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T10:06:11.081 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T10:06:11.081 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_40 2024-04-06T10:06:11.237 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ return 0 2024-04-06T10:06:11.238 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ status_result=' "state": "Replaying"' 2024-04-06T10:06:11.239 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ echo ' "state": "Replaying"' 2024-04-06T10:06:11.239 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep -i Replaying 2024-04-06T10:06:11.240 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: "state": "Replaying" 2024-04-06T10:06:11.241 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ current_state=started 2024-04-06T10:06:11.241 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test started = started 2024-04-06T10:06:11.241 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T10:06:11.241 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_40 2024-04-06T10:06:11.242 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local local_cluster=cluster1 2024-04-06T10:06:11.242 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-04-06T10:06:11.242 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:06:11.242 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_40 2024-04-06T10:06:11.242 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' journal = journal ']' 2024-04-06T10:06:11.242 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_journal_replay_complete cluster1 cluster2 mirror image_40 2024-04-06T10:06:11.242 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local local_cluster=cluster1 2024-04-06T10:06:11.242 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-04-06T10:06:11.242 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:06:11.242 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_40 2024-04-06T10:06:11.242 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local s master_pos mirror_pos last_mirror_pos 2024-04-06T10:06:11.242 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local master_tag master_entry mirror_tag mirror_entry 2024-04-06T10:06:11.242 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T10:06:11.243 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T10:06:11.243 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T10:06:11.443 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror image_40 2024-04-06T10:06:11.444 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T10:06:11.444 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:06:11.444 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_40 2024-04-06T10:06:11.444 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T10:06:11.444 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n image_40 ']' 2024-04-06T10:06:11.444 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/image_40' 2024-04-06T10:06:11.444 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_40 2024-04-06T10:06:11.444 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T10:06:11.444 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T10:06:11.444 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T10:06:11.444 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T10:06:11.444 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T10:06:11.445 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T10:06:11.445 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T10:06:11.445 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T10:06:11.445 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T10:06:11.447 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T10:06:11.447 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_40 2024-04-06T10:06:11.447 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T10:06:11.447 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T10:06:11.447 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T10:06:11.447 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T10:06:11.447 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T10:06:11.448 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T10:06:11.448 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T10:06:11.448 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T10:06:11.448 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T10:06:11.448 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T10:06:11.448 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T10:06:11.448 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:06:11.448 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T10:06:11.448 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T10:06:11.448 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T10:06:11.448 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T10:06:11.449 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T10:06:11.449 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T10:06:11.449 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T10:06:11.449 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T10:06:11.449 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T10:06:11.449 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T10:06:11.449 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:06:11.449 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T10:06:11.449 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T10:06:11.449 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T10:06:11.449 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T10:06:11.449 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T10:06:11.449 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T10:06:11.450 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T10:06:11.474 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T10:06:11.474 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T10:06:11.474 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T10:06:11.474 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_40 2024-04-06T10:06:11.581 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:Invalid command: unused arguments: ['mirror/image_40'] 2024-04-06T10:06:11.581 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:rbd mirror flush : flush rbd mirror 2024-04-06T10:06:11.581 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:admin_socket: invalid command 2024-04-06T10:06:11.583 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T10:06:11.583 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/image_40 2024-04-06T10:06:11.583 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:1 2024-04-06T10:06:11.583 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T10:06:11.583 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T10:06:11.583 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-06T10:06:11.583 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:1 2024-04-06T10:06:11.583 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T10:06:11.584 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T10:06:11.584 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T10:06:11.584 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=1 2024-04-06T10:06:11.584 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-06T10:06:11.584 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T10:06:11.584 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:06:11.584 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=1 2024-04-06T10:06:11.584 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=1 2024-04-06T10:06:11.585 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-06T10:06:11.585 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:1 2024-04-06T10:06:11.586 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T10:06:11.586 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T10:06:11.586 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-06T10:06:11.586 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:1 2024-04-06T10:06:11.586 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T10:06:11.586 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T10:06:11.586 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:06:11.586 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=1 2024-04-06T10:06:11.586 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-06T10:06:11.586 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T10:06:11.586 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T10:06:11.587 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=1 2024-04-06T10:06:11.587 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=1 2024-04-06T10:06:11.587 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-06T10:06:11.609 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T10:06:11.609 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T10:06:11.609 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T10:06:11.610 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/image_40 2024-04-06T10:06:11.744 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T10:06:11.745 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror image_40 2024-04-06T10:06:11.745 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:06:11.746 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:06:11.746 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_40 2024-04-06T10:06:11.746 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror image_40 '' 2024-04-06T10:06:11.746 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:06:11.746 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:06:11.746 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_40 2024-04-06T10:06:11.746 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T10:06:11.746 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-image_40.status 2024-04-06T10:06:11.747 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-image_40.status 2024-04-06T10:06:11.747 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T10:06:11.749 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-image_40.status 2024-04-06T10:06:11.749 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_40 2024-04-06T10:06:11.749 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_40.status 2024-04-06T10:06:11.795 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 0 2024-04-06T10:06:11.795 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 0 2024-04-06T10:06:11.795 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T10:06:11.796 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=1, tag_tid=3, entry_tid=1], [object_number=0, tag_tid=3, entry_tid=0], [object_number=3, tag_tid=1, entry_tid=3211], [object_number=2, tag_tid=1, entry_tid=3210]]], state=connected] 2024-04-06T10:06:11.796 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=1, tag_tid=2, entry_tid=1], [object_number=0, tag_tid=2, entry_tid=0], [object_number=3, tag_tid=1, entry_tid=3211], [object_number=2, tag_tid=1, entry_tid=3210]]], state=connected] 2024-04-06T10:06:11.799 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_40.status 2024-04-06T10:06:11.800 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=1, tag_tid=3, entry_tid=1' 2024-04-06T10:06:11.801 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror image_40 2024-04-06T10:06:11.801 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:06:11.801 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:06:11.801 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_40 2024-04-06T10:06:11.802 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror image_40 '..*' 2024-04-06T10:06:11.802 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:06:11.802 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:06:11.802 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_40 2024-04-06T10:06:11.802 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T10:06:11.802 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-image_40.status 2024-04-06T10:06:11.802 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-image_40.status 2024-04-06T10:06:11.802 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T10:06:11.803 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-image_40.status 2024-04-06T10:06:11.803 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_40 2024-04-06T10:06:11.803 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_40.status 2024-04-06T10:06:11.847 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 0 2024-04-06T10:06:11.847 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 0 2024-04-06T10:06:11.847 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T10:06:11.847 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=1, tag_tid=3, entry_tid=1], [object_number=0, tag_tid=3, entry_tid=0], [object_number=3, tag_tid=1, entry_tid=3211], [object_number=2, tag_tid=1, entry_tid=3210]]], state=connected] 2024-04-06T10:06:11.847 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=1, tag_tid=2, entry_tid=1], [object_number=0, tag_tid=2, entry_tid=0], [object_number=3, tag_tid=1, entry_tid=3211], [object_number=2, tag_tid=1, entry_tid=3210]]], state=connected] 2024-04-06T10:06:11.850 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_40.status 2024-04-06T10:06:11.852 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=1, tag_tid=2, entry_tid=1' 2024-04-06T10:06:11.853 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=1, tag_tid=3, entry_tid=1' -a 'object_number=1, tag_tid=3, entry_tid=1' = 'object_number=1, tag_tid=2, entry_tid=1' 2024-04-06T10:06:11.853 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=1, tag_tid=2, entry_tid=1' '!=' '' 2024-04-06T10:06:11.853 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T10:06:11.853 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=1, tag_tid=2, entry_tid=1' = '' 2024-04-06T10:06:11.853 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=1, tag_tid=2, entry_tid=1' 2024-04-06T10:06:11.853 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=1, tag_tid=3, entry_tid=1' 2024-04-06T10:06:11.853 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T10:06:11.853 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T10:06:11.854 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=3 2024-04-06T10:06:11.854 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=1, tag_tid=2, entry_tid=1' 2024-04-06T10:06:11.854 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T10:06:11.855 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T10:06:11.856 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=2 2024-04-06T10:06:11.856 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=1, tag_tid=3, entry_tid=1' 2024-04-06T10:06:11.856 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T10:06:11.856 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T10:06:11.857 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T10:06:11.858 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=1, tag_tid=2, entry_tid=1' 2024-04-06T10:06:11.858 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T10:06:11.858 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T10:06:11.859 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=1 2024-04-06T10:06:11.859 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 3 = 2 -a 1 -le 1 2024-04-06T10:06:11.859 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T10:06:11.859 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T10:06:11.859 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T10:06:12.061 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror image_40 2024-04-06T10:06:12.061 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T10:06:12.061 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:06:12.062 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_40 2024-04-06T10:06:12.062 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T10:06:12.062 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n image_40 ']' 2024-04-06T10:06:12.062 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/image_40' 2024-04-06T10:06:12.062 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_40 2024-04-06T10:06:12.062 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T10:06:12.062 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T10:06:12.062 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T10:06:12.062 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T10:06:12.062 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T10:06:12.062 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T10:06:12.063 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T10:06:12.063 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T10:06:12.063 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T10:06:12.065 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T10:06:12.065 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_40 2024-04-06T10:06:12.065 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T10:06:12.065 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T10:06:12.066 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T10:06:12.066 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T10:06:12.066 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T10:06:12.066 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T10:06:12.066 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T10:06:12.066 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T10:06:12.066 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T10:06:12.066 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T10:06:12.066 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T10:06:12.066 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:06:12.066 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T10:06:12.067 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T10:06:12.067 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T10:06:12.067 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T10:06:12.067 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T10:06:12.067 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T10:06:12.067 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T10:06:12.067 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T10:06:12.067 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T10:06:12.067 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T10:06:12.067 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:06:12.067 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T10:06:12.067 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T10:06:12.067 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T10:06:12.068 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T10:06:12.068 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T10:06:12.068 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T10:06:12.068 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T10:06:12.093 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T10:06:12.093 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T10:06:12.093 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T10:06:12.094 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_40 2024-04-06T10:06:12.204 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:Invalid command: unused arguments: ['mirror/image_40'] 2024-04-06T10:06:12.205 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:rbd mirror flush : flush rbd mirror 2024-04-06T10:06:12.205 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:admin_socket: invalid command 2024-04-06T10:06:12.212 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T10:06:12.212 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/image_40 2024-04-06T10:06:12.212 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:1 2024-04-06T10:06:12.212 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T10:06:12.212 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T10:06:12.212 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-06T10:06:12.212 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:1 2024-04-06T10:06:12.212 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T10:06:12.212 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T10:06:12.212 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T10:06:12.212 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=1 2024-04-06T10:06:12.212 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-06T10:06:12.213 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T10:06:12.213 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:06:12.213 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=1 2024-04-06T10:06:12.213 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=1 2024-04-06T10:06:12.213 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-06T10:06:12.213 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:1 2024-04-06T10:06:12.213 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T10:06:12.213 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T10:06:12.213 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-06T10:06:12.213 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:1 2024-04-06T10:06:12.213 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T10:06:12.213 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T10:06:12.213 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:06:12.213 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=1 2024-04-06T10:06:12.213 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-06T10:06:12.214 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T10:06:12.214 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T10:06:12.214 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=1 2024-04-06T10:06:12.214 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=1 2024-04-06T10:06:12.214 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-06T10:06:12.234 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T10:06:12.235 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T10:06:12.235 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T10:06:12.235 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/image_40 2024-04-06T10:06:12.374 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T10:06:12.375 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror image_40 2024-04-06T10:06:12.376 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:06:12.376 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:06:12.376 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_40 2024-04-06T10:06:12.376 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror image_40 '' 2024-04-06T10:06:12.376 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:06:12.376 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:06:12.376 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_40 2024-04-06T10:06:12.376 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T10:06:12.376 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-image_40.status 2024-04-06T10:06:12.377 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-image_40.status 2024-04-06T10:06:12.377 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T10:06:12.378 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-image_40.status 2024-04-06T10:06:12.379 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_40 2024-04-06T10:06:12.379 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_40.status 2024-04-06T10:06:12.428 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 0 2024-04-06T10:06:12.428 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 0 2024-04-06T10:06:12.428 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T10:06:12.428 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=1, tag_tid=3, entry_tid=1], [object_number=0, tag_tid=3, entry_tid=0], [object_number=3, tag_tid=1, entry_tid=3211], [object_number=2, tag_tid=1, entry_tid=3210]]], state=connected] 2024-04-06T10:06:12.428 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=1, tag_tid=2, entry_tid=1], [object_number=0, tag_tid=2, entry_tid=0], [object_number=3, tag_tid=1, entry_tid=3211], [object_number=2, tag_tid=1, entry_tid=3210]]], state=connected] 2024-04-06T10:06:12.432 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_40.status 2024-04-06T10:06:12.433 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=1, tag_tid=3, entry_tid=1' 2024-04-06T10:06:12.434 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror image_40 2024-04-06T10:06:12.434 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:06:12.434 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:06:12.434 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_40 2024-04-06T10:06:12.434 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror image_40 '..*' 2024-04-06T10:06:12.434 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:06:12.434 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:06:12.434 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_40 2024-04-06T10:06:12.434 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T10:06:12.434 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-image_40.status 2024-04-06T10:06:12.435 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-image_40.status 2024-04-06T10:06:12.435 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T10:06:12.435 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-image_40.status 2024-04-06T10:06:12.436 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_40 2024-04-06T10:06:12.436 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_40.status 2024-04-06T10:06:12.480 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 0 2024-04-06T10:06:12.480 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 0 2024-04-06T10:06:12.480 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T10:06:12.481 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=1, tag_tid=3, entry_tid=1], [object_number=0, tag_tid=3, entry_tid=0], [object_number=3, tag_tid=1, entry_tid=3211], [object_number=2, tag_tid=1, entry_tid=3210]]], state=connected] 2024-04-06T10:06:12.481 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=1, tag_tid=2, entry_tid=1], [object_number=0, tag_tid=2, entry_tid=0], [object_number=3, tag_tid=1, entry_tid=3211], [object_number=2, tag_tid=1, entry_tid=3210]]], state=connected] 2024-04-06T10:06:12.484 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_40.status 2024-04-06T10:06:12.485 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=1, tag_tid=2, entry_tid=1' 2024-04-06T10:06:12.486 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=1, tag_tid=3, entry_tid=1' -a 'object_number=1, tag_tid=3, entry_tid=1' = 'object_number=1, tag_tid=2, entry_tid=1' 2024-04-06T10:06:12.486 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=1, tag_tid=2, entry_tid=1' '!=' 'object_number=1, tag_tid=2, entry_tid=1' 2024-04-06T10:06:12.486 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T10:06:12.486 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.4 2024-04-06T10:06:12.887 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror image_40 2024-04-06T10:06:12.888 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T10:06:12.888 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:06:12.888 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_40 2024-04-06T10:06:12.888 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T10:06:12.888 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n image_40 ']' 2024-04-06T10:06:12.888 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/image_40' 2024-04-06T10:06:12.888 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_40 2024-04-06T10:06:12.888 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T10:06:12.888 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T10:06:12.888 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T10:06:12.888 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T10:06:12.888 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T10:06:12.888 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T10:06:12.889 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T10:06:12.889 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T10:06:12.889 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T10:06:12.891 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T10:06:12.892 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_40 2024-04-06T10:06:12.892 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T10:06:12.892 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T10:06:12.892 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T10:06:12.892 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T10:06:12.892 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T10:06:12.892 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T10:06:12.892 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T10:06:12.892 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T10:06:12.892 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T10:06:12.892 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T10:06:12.892 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T10:06:12.892 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:06:12.892 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T10:06:12.893 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T10:06:12.893 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T10:06:12.893 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T10:06:12.893 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T10:06:12.893 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T10:06:12.893 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T10:06:12.893 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T10:06:12.893 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T10:06:12.893 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T10:06:12.893 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:06:12.893 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T10:06:12.893 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T10:06:12.893 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T10:06:12.894 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T10:06:12.894 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T10:06:12.894 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T10:06:12.894 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T10:06:12.918 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T10:06:12.918 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T10:06:12.918 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T10:06:12.918 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_40 2024-04-06T10:06:13.022 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:Invalid command: unused arguments: ['mirror/image_40'] 2024-04-06T10:06:13.023 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:rbd mirror flush : flush rbd mirror 2024-04-06T10:06:13.023 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:admin_socket: invalid command 2024-04-06T10:06:13.027 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T10:06:13.027 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/image_40 2024-04-06T10:06:13.027 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:1 2024-04-06T10:06:13.027 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T10:06:13.027 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T10:06:13.028 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-06T10:06:13.028 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:1 2024-04-06T10:06:13.028 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T10:06:13.028 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T10:06:13.028 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T10:06:13.028 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=1 2024-04-06T10:06:13.028 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-06T10:06:13.028 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T10:06:13.028 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:06:13.028 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=1 2024-04-06T10:06:13.028 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=1 2024-04-06T10:06:13.028 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-06T10:06:13.028 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:1 2024-04-06T10:06:13.030 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T10:06:13.030 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T10:06:13.030 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-06T10:06:13.030 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:1 2024-04-06T10:06:13.030 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T10:06:13.030 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T10:06:13.030 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:06:13.030 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=1 2024-04-06T10:06:13.030 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-06T10:06:13.030 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T10:06:13.031 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T10:06:13.031 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=1 2024-04-06T10:06:13.031 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=1 2024-04-06T10:06:13.031 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-06T10:06:13.045 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T10:06:13.045 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T10:06:13.046 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T10:06:13.046 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/image_40 2024-04-06T10:06:13.186 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T10:06:13.187 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror image_40 2024-04-06T10:06:13.187 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:06:13.188 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:06:13.188 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_40 2024-04-06T10:06:13.188 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror image_40 '' 2024-04-06T10:06:13.188 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:06:13.188 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:06:13.188 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_40 2024-04-06T10:06:13.188 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T10:06:13.189 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-image_40.status 2024-04-06T10:06:13.189 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-image_40.status 2024-04-06T10:06:13.189 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T10:06:13.190 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-image_40.status 2024-04-06T10:06:13.191 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_40 2024-04-06T10:06:13.191 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_40.status 2024-04-06T10:06:13.236 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 0 2024-04-06T10:06:13.236 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 0 2024-04-06T10:06:13.237 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T10:06:13.237 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=1, tag_tid=3, entry_tid=1], [object_number=0, tag_tid=3, entry_tid=0], [object_number=3, tag_tid=1, entry_tid=3211], [object_number=2, tag_tid=1, entry_tid=3210]]], state=connected] 2024-04-06T10:06:13.237 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=1, tag_tid=2, entry_tid=1], [object_number=0, tag_tid=2, entry_tid=0], [object_number=3, tag_tid=1, entry_tid=3211], [object_number=2, tag_tid=1, entry_tid=3210]]], state=connected] 2024-04-06T10:06:13.240 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_40.status 2024-04-06T10:06:13.241 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=1, tag_tid=3, entry_tid=1' 2024-04-06T10:06:13.242 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror image_40 2024-04-06T10:06:13.242 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:06:13.242 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:06:13.242 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_40 2024-04-06T10:06:13.242 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror image_40 '..*' 2024-04-06T10:06:13.242 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:06:13.242 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:06:13.243 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_40 2024-04-06T10:06:13.243 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T10:06:13.243 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-image_40.status 2024-04-06T10:06:13.243 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-image_40.status 2024-04-06T10:06:13.243 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T10:06:13.243 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-image_40.status 2024-04-06T10:06:13.244 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_40 2024-04-06T10:06:13.244 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_40.status 2024-04-06T10:06:13.287 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 0 2024-04-06T10:06:13.288 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 0 2024-04-06T10:06:13.288 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T10:06:13.288 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=1, tag_tid=3, entry_tid=1], [object_number=0, tag_tid=3, entry_tid=0], [object_number=3, tag_tid=1, entry_tid=3211], [object_number=2, tag_tid=1, entry_tid=3210]]], state=connected] 2024-04-06T10:06:13.288 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=1, tag_tid=2, entry_tid=1], [object_number=0, tag_tid=2, entry_tid=0], [object_number=3, tag_tid=1, entry_tid=3211], [object_number=2, tag_tid=1, entry_tid=3210]]], state=connected] 2024-04-06T10:06:13.291 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_40.status 2024-04-06T10:06:13.293 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=1, tag_tid=2, entry_tid=1' 2024-04-06T10:06:13.293 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=1, tag_tid=3, entry_tid=1' -a 'object_number=1, tag_tid=3, entry_tid=1' = 'object_number=1, tag_tid=2, entry_tid=1' 2024-04-06T10:06:13.293 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=1, tag_tid=2, entry_tid=1' '!=' 'object_number=1, tag_tid=2, entry_tid=1' 2024-04-06T10:06:13.293 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T10:06:13.293 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.8 2024-04-06T10:06:14.095 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror image_40 2024-04-06T10:06:14.095 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T10:06:14.095 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:06:14.095 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_40 2024-04-06T10:06:14.095 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T10:06:14.095 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n image_40 ']' 2024-04-06T10:06:14.095 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/image_40' 2024-04-06T10:06:14.095 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_40 2024-04-06T10:06:14.095 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T10:06:14.095 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T10:06:14.095 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T10:06:14.095 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T10:06:14.096 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T10:06:14.096 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T10:06:14.096 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T10:06:14.096 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T10:06:14.096 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T10:06:14.100 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T10:06:14.101 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_40 2024-04-06T10:06:14.101 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T10:06:14.101 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T10:06:14.101 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T10:06:14.101 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T10:06:14.101 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T10:06:14.101 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T10:06:14.101 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T10:06:14.102 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T10:06:14.102 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T10:06:14.102 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T10:06:14.102 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T10:06:14.102 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:06:14.102 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T10:06:14.102 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T10:06:14.103 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T10:06:14.103 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T10:06:14.103 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T10:06:14.103 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T10:06:14.103 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T10:06:14.103 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T10:06:14.103 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T10:06:14.103 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T10:06:14.103 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:06:14.104 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T10:06:14.104 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T10:06:14.104 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T10:06:14.104 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T10:06:14.104 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T10:06:14.104 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T10:06:14.104 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T10:06:14.126 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T10:06:14.126 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T10:06:14.126 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T10:06:14.126 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_40 2024-04-06T10:06:14.232 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:Invalid command: unused arguments: ['mirror/image_40'] 2024-04-06T10:06:14.232 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:rbd mirror flush : flush rbd mirror 2024-04-06T10:06:14.232 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:admin_socket: invalid command 2024-04-06T10:06:14.235 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T10:06:14.235 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/image_40 2024-04-06T10:06:14.235 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:1 2024-04-06T10:06:14.235 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T10:06:14.235 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T10:06:14.235 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-06T10:06:14.235 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:1 2024-04-06T10:06:14.235 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T10:06:14.235 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T10:06:14.236 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T10:06:14.236 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=1 2024-04-06T10:06:14.236 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-06T10:06:14.236 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T10:06:14.236 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:06:14.236 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=1 2024-04-06T10:06:14.236 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=1 2024-04-06T10:06:14.237 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-06T10:06:14.237 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:1 2024-04-06T10:06:14.237 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T10:06:14.237 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T10:06:14.237 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-06T10:06:14.238 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:1 2024-04-06T10:06:14.238 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T10:06:14.238 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T10:06:14.238 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:06:14.238 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=1 2024-04-06T10:06:14.238 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-06T10:06:14.238 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T10:06:14.238 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T10:06:14.238 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=1 2024-04-06T10:06:14.238 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=1 2024-04-06T10:06:14.238 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-06T10:06:14.260 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T10:06:14.261 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T10:06:14.261 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T10:06:14.261 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/image_40 2024-04-06T10:06:14.410 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T10:06:14.411 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror image_40 2024-04-06T10:06:14.411 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:06:14.411 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:06:14.411 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_40 2024-04-06T10:06:14.411 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror image_40 '' 2024-04-06T10:06:14.411 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:06:14.411 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:06:14.411 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_40 2024-04-06T10:06:14.412 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T10:06:14.412 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-image_40.status 2024-04-06T10:06:14.412 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-image_40.status 2024-04-06T10:06:14.412 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T10:06:14.414 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-image_40.status 2024-04-06T10:06:14.414 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_40 2024-04-06T10:06:14.414 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_40.status 2024-04-06T10:06:14.670 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 0 2024-04-06T10:06:14.670 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 0 2024-04-06T10:06:14.670 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T10:06:14.670 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=1, tag_tid=3, entry_tid=1], [object_number=0, tag_tid=3, entry_tid=0], [object_number=3, tag_tid=1, entry_tid=3211], [object_number=2, tag_tid=1, entry_tid=3210]]], state=connected] 2024-04-06T10:06:14.670 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=1, tag_tid=2, entry_tid=1], [object_number=0, tag_tid=2, entry_tid=0], [object_number=3, tag_tid=1, entry_tid=3211], [object_number=2, tag_tid=1, entry_tid=3210]]], state=connected] 2024-04-06T10:06:14.674 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_40.status 2024-04-06T10:06:14.676 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=1, tag_tid=3, entry_tid=1' 2024-04-06T10:06:14.677 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror image_40 2024-04-06T10:06:14.677 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:06:14.677 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:06:14.677 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_40 2024-04-06T10:06:14.677 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror image_40 '..*' 2024-04-06T10:06:14.677 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:06:14.677 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:06:14.677 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_40 2024-04-06T10:06:14.677 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T10:06:14.677 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-image_40.status 2024-04-06T10:06:14.678 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-image_40.status 2024-04-06T10:06:14.678 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T10:06:14.678 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-image_40.status 2024-04-06T10:06:14.679 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_40 2024-04-06T10:06:14.679 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_40.status 2024-04-06T10:06:14.728 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 0 2024-04-06T10:06:14.728 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 0 2024-04-06T10:06:14.728 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T10:06:14.728 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=1, tag_tid=3, entry_tid=1], [object_number=0, tag_tid=3, entry_tid=0], [object_number=3, tag_tid=1, entry_tid=3211], [object_number=2, tag_tid=1, entry_tid=3210]]], state=connected] 2024-04-06T10:06:14.728 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=1, tag_tid=2, entry_tid=1], [object_number=0, tag_tid=2, entry_tid=0], [object_number=3, tag_tid=1, entry_tid=3211], [object_number=2, tag_tid=1, entry_tid=3210]]], state=connected] 2024-04-06T10:06:14.732 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_40.status 2024-04-06T10:06:14.734 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=1, tag_tid=2, entry_tid=1' 2024-04-06T10:06:14.734 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=1, tag_tid=3, entry_tid=1' -a 'object_number=1, tag_tid=3, entry_tid=1' = 'object_number=1, tag_tid=2, entry_tid=1' 2024-04-06T10:06:14.734 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=1, tag_tid=2, entry_tid=1' '!=' 'object_number=1, tag_tid=2, entry_tid=1' 2024-04-06T10:06:14.734 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T10:06:14.734 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 1.6 2024-04-06T10:06:16.336 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror image_40 2024-04-06T10:06:16.336 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T10:06:16.336 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:06:16.337 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_40 2024-04-06T10:06:16.337 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T10:06:16.337 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n image_40 ']' 2024-04-06T10:06:16.337 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/image_40' 2024-04-06T10:06:16.337 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_40 2024-04-06T10:06:16.337 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T10:06:16.337 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T10:06:16.337 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T10:06:16.337 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T10:06:16.337 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T10:06:16.337 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T10:06:16.337 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T10:06:16.338 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T10:06:16.338 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T10:06:16.340 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T10:06:16.340 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_40 2024-04-06T10:06:16.340 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T10:06:16.340 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T10:06:16.340 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T10:06:16.340 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T10:06:16.340 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T10:06:16.340 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T10:06:16.340 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T10:06:16.340 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T10:06:16.340 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T10:06:16.340 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T10:06:16.341 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T10:06:16.341 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:06:16.341 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T10:06:16.341 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T10:06:16.342 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T10:06:16.342 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T10:06:16.342 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T10:06:16.342 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T10:06:16.342 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T10:06:16.342 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T10:06:16.343 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T10:06:16.343 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T10:06:16.343 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:06:16.343 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T10:06:16.343 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T10:06:16.343 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T10:06:16.343 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T10:06:16.343 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T10:06:16.343 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T10:06:16.343 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T10:06:16.367 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T10:06:16.367 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T10:06:16.367 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T10:06:16.367 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_40 2024-04-06T10:06:16.474 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:Invalid command: unused arguments: ['mirror/image_40'] 2024-04-06T10:06:16.474 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:rbd mirror flush : flush rbd mirror 2024-04-06T10:06:16.474 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:admin_socket: invalid command 2024-04-06T10:06:16.476 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T10:06:16.476 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/image_40 2024-04-06T10:06:16.476 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:1 2024-04-06T10:06:16.476 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T10:06:16.477 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T10:06:16.477 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-06T10:06:16.477 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:1 2024-04-06T10:06:16.477 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T10:06:16.477 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T10:06:16.477 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T10:06:16.477 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=1 2024-04-06T10:06:16.477 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-06T10:06:16.477 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T10:06:16.477 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:06:16.477 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=1 2024-04-06T10:06:16.477 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=1 2024-04-06T10:06:16.479 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-06T10:06:16.479 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:1 2024-04-06T10:06:16.479 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T10:06:16.479 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T10:06:16.479 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-06T10:06:16.479 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:1 2024-04-06T10:06:16.479 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T10:06:16.479 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T10:06:16.479 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:06:16.479 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=1 2024-04-06T10:06:16.479 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-06T10:06:16.479 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T10:06:16.479 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T10:06:16.480 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=1 2024-04-06T10:06:16.480 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=1 2024-04-06T10:06:16.480 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-06T10:06:16.503 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T10:06:16.503 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T10:06:16.503 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T10:06:16.503 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/image_40 2024-04-06T10:06:16.653 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T10:06:16.655 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror image_40 2024-04-06T10:06:16.655 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:06:16.655 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:06:16.655 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_40 2024-04-06T10:06:16.655 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror image_40 '' 2024-04-06T10:06:16.655 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:06:16.655 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:06:16.655 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_40 2024-04-06T10:06:16.656 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T10:06:16.656 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-image_40.status 2024-04-06T10:06:16.656 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-image_40.status 2024-04-06T10:06:16.656 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T10:06:16.657 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-image_40.status 2024-04-06T10:06:16.658 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_40 2024-04-06T10:06:16.658 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_40.status 2024-04-06T10:06:16.702 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 0 2024-04-06T10:06:16.702 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 0 2024-04-06T10:06:16.702 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T10:06:16.702 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=1, tag_tid=3, entry_tid=1], [object_number=0, tag_tid=3, entry_tid=0], [object_number=3, tag_tid=1, entry_tid=3211], [object_number=2, tag_tid=1, entry_tid=3210]]], state=connected] 2024-04-06T10:06:16.703 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=1, tag_tid=2, entry_tid=1], [object_number=0, tag_tid=2, entry_tid=0], [object_number=3, tag_tid=1, entry_tid=3211], [object_number=2, tag_tid=1, entry_tid=3210]]], state=connected] 2024-04-06T10:06:16.706 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_40.status 2024-04-06T10:06:16.708 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=1, tag_tid=3, entry_tid=1' 2024-04-06T10:06:16.708 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror image_40 2024-04-06T10:06:16.708 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:06:16.708 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:06:16.708 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_40 2024-04-06T10:06:16.709 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror image_40 '..*' 2024-04-06T10:06:16.709 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:06:16.709 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:06:16.709 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_40 2024-04-06T10:06:16.709 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T10:06:16.709 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-image_40.status 2024-04-06T10:06:16.709 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-image_40.status 2024-04-06T10:06:16.709 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T10:06:16.710 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-image_40.status 2024-04-06T10:06:16.710 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_40 2024-04-06T10:06:16.710 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_40.status 2024-04-06T10:06:16.754 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 0 2024-04-06T10:06:16.754 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 0 2024-04-06T10:06:16.755 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T10:06:16.755 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=1, tag_tid=3, entry_tid=1], [object_number=0, tag_tid=3, entry_tid=0], [object_number=3, tag_tid=1, entry_tid=3211], [object_number=2, tag_tid=1, entry_tid=3210]]], state=connected] 2024-04-06T10:06:16.755 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=1, tag_tid=2, entry_tid=1], [object_number=0, tag_tid=2, entry_tid=0], [object_number=3, tag_tid=1, entry_tid=3211], [object_number=2, tag_tid=1, entry_tid=3210]]], state=connected] 2024-04-06T10:06:16.758 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_40.status 2024-04-06T10:06:16.759 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=1, tag_tid=2, entry_tid=1' 2024-04-06T10:06:16.759 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=1, tag_tid=3, entry_tid=1' -a 'object_number=1, tag_tid=3, entry_tid=1' = 'object_number=1, tag_tid=2, entry_tid=1' 2024-04-06T10:06:16.759 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=1, tag_tid=2, entry_tid=1' '!=' 'object_number=1, tag_tid=2, entry_tid=1' 2024-04-06T10:06:16.759 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T10:06:16.760 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 2 2024-04-06T10:06:18.761 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror image_40 2024-04-06T10:06:18.761 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T10:06:18.761 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:06:18.761 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_40 2024-04-06T10:06:18.762 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T10:06:18.762 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n image_40 ']' 2024-04-06T10:06:18.762 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/image_40' 2024-04-06T10:06:18.762 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_40 2024-04-06T10:06:18.762 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T10:06:18.762 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T10:06:18.762 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T10:06:18.762 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T10:06:18.762 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T10:06:18.762 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T10:06:18.762 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T10:06:18.763 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T10:06:18.763 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T10:06:18.766 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T10:06:18.766 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_40 2024-04-06T10:06:18.766 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T10:06:18.766 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T10:06:18.766 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T10:06:18.766 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T10:06:18.766 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T10:06:18.766 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T10:06:18.766 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T10:06:18.766 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T10:06:18.767 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T10:06:18.767 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T10:06:18.767 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T10:06:18.767 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:06:18.767 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T10:06:18.767 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T10:06:18.767 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T10:06:18.767 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T10:06:18.767 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T10:06:18.767 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T10:06:18.767 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T10:06:18.767 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T10:06:18.767 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T10:06:18.767 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T10:06:18.767 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:06:18.768 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T10:06:18.768 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T10:06:18.768 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T10:06:18.768 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T10:06:18.768 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T10:06:18.768 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T10:06:18.768 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T10:06:18.793 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T10:06:18.793 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T10:06:18.793 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T10:06:18.793 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_40 2024-04-06T10:06:18.897 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:Invalid command: unused arguments: ['mirror/image_40'] 2024-04-06T10:06:18.897 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:rbd mirror flush : flush rbd mirror 2024-04-06T10:06:18.897 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:admin_socket: invalid command 2024-04-06T10:06:18.899 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T10:06:18.900 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/image_40 2024-04-06T10:06:18.900 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:1 2024-04-06T10:06:18.900 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T10:06:18.900 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T10:06:18.900 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-06T10:06:18.900 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:1 2024-04-06T10:06:18.900 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T10:06:18.900 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T10:06:18.900 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T10:06:18.900 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=1 2024-04-06T10:06:18.900 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-06T10:06:18.900 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T10:06:18.900 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:06:18.900 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=1 2024-04-06T10:06:18.901 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=1 2024-04-06T10:06:18.902 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-06T10:06:18.902 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:1 2024-04-06T10:06:18.902 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T10:06:18.902 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T10:06:18.902 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-06T10:06:18.902 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:1 2024-04-06T10:06:18.902 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T10:06:18.902 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T10:06:18.902 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:06:18.902 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=1 2024-04-06T10:06:18.903 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-06T10:06:18.903 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T10:06:18.903 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T10:06:18.903 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=1 2024-04-06T10:06:18.903 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=1 2024-04-06T10:06:18.903 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-06T10:06:18.924 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T10:06:18.925 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T10:06:18.925 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T10:06:18.925 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/image_40 2024-04-06T10:06:19.072 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T10:06:19.073 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror image_40 2024-04-06T10:06:19.073 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:06:19.073 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:06:19.073 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_40 2024-04-06T10:06:19.073 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror image_40 '' 2024-04-06T10:06:19.074 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:06:19.074 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:06:19.074 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_40 2024-04-06T10:06:19.074 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T10:06:19.074 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-image_40.status 2024-04-06T10:06:19.074 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-image_40.status 2024-04-06T10:06:19.075 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T10:06:19.077 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-image_40.status 2024-04-06T10:06:19.078 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_40 2024-04-06T10:06:19.078 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_40.status 2024-04-06T10:06:19.123 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 0 2024-04-06T10:06:19.123 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 0 2024-04-06T10:06:19.124 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T10:06:19.124 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=1, tag_tid=3, entry_tid=1], [object_number=0, tag_tid=3, entry_tid=0], [object_number=3, tag_tid=1, entry_tid=3211], [object_number=2, tag_tid=1, entry_tid=3210]]], state=connected] 2024-04-06T10:06:19.124 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=1, tag_tid=3, entry_tid=1], [object_number=0, tag_tid=3, entry_tid=0], [object_number=3, tag_tid=1, entry_tid=3211], [object_number=2, tag_tid=1, entry_tid=3210]]], state=connected] 2024-04-06T10:06:19.127 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_40.status 2024-04-06T10:06:19.128 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=1, tag_tid=3, entry_tid=1' 2024-04-06T10:06:19.129 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror image_40 2024-04-06T10:06:19.129 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:06:19.129 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:06:19.129 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_40 2024-04-06T10:06:19.129 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror image_40 '..*' 2024-04-06T10:06:19.130 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:06:19.130 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:06:19.130 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_40 2024-04-06T10:06:19.130 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T10:06:19.130 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-image_40.status 2024-04-06T10:06:19.130 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-image_40.status 2024-04-06T10:06:19.130 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T10:06:19.131 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-image_40.status 2024-04-06T10:06:19.131 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_40 2024-04-06T10:06:19.131 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_40.status 2024-04-06T10:06:19.175 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 0 2024-04-06T10:06:19.176 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 0 2024-04-06T10:06:19.176 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T10:06:19.176 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=1, tag_tid=3, entry_tid=1], [object_number=0, tag_tid=3, entry_tid=0], [object_number=3, tag_tid=1, entry_tid=3211], [object_number=2, tag_tid=1, entry_tid=3210]]], state=connected] 2024-04-06T10:06:19.176 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=1, tag_tid=3, entry_tid=1], [object_number=0, tag_tid=3, entry_tid=0], [object_number=3, tag_tid=1, entry_tid=3211], [object_number=2, tag_tid=1, entry_tid=3210]]], state=connected] 2024-04-06T10:06:19.179 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_40.status 2024-04-06T10:06:19.181 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=1, tag_tid=3, entry_tid=1' 2024-04-06T10:06:19.181 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=1, tag_tid=3, entry_tid=1' -a 'object_number=1, tag_tid=3, entry_tid=1' = 'object_number=1, tag_tid=3, entry_tid=1' 2024-04-06T10:06:19.181 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T10:06:19.181 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_snap_present cluster1 mirror image_40 snap 2024-04-06T10:06:19.181 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T10:06:19.181 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:06:19.181 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_40 2024-04-06T10:06:19.182 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local snap_name=snap 2024-04-06T10:06:19.182 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local s 2024-04-06T10:06:19.182 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-04-06T10:06:19.182 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 1 2024-04-06T10:06:20.182 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 info mirror/image_40@snap 2024-04-06T10:06:20.249 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:rbd image 'image_40': 2024-04-06T10:06:20.249 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: size 128 MiB in 32 objects 2024-04-06T10:06:20.249 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: order 22 (4 MiB objects) 2024-04-06T10:06:20.249 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: snapshot_count: 1 2024-04-06T10:06:20.249 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: id: 10acc552dcff 2024-04-06T10:06:20.250 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: block_name_prefix: rbd_data.10acc552dcff 2024-04-06T10:06:20.250 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: format: 2 2024-04-06T10:06:20.250 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: features: layering, exclusive-lock, journaling 2024-04-06T10:06:20.250 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: op_features: 2024-04-06T10:06:20.250 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: flags: 2024-04-06T10:06:20.250 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: create_timestamp: Sat Apr 6 09:56:44 2024 2024-04-06T10:06:20.250 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: access_timestamp: Sat Apr 6 09:56:44 2024 2024-04-06T10:06:20.250 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: modify_timestamp: Sat Apr 6 09:57:45 2024 2024-04-06T10:06:20.250 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: protected: False 2024-04-06T10:06:20.250 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: journal: 10acc552dcff 2024-04-06T10:06:20.250 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: mirroring state: enabled 2024-04-06T10:06:20.250 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: mirroring mode: journal 2024-04-06T10:06:20.250 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: mirroring global id: c71392be-ba76-48f6-a351-3d323d4c9b02 2024-04-06T10:06:20.250 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: mirroring primary: false 2024-04-06T10:06:20.254 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T10:06:20.254 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ compare_image_snaps mirror image_40 snap 2024-04-06T10:06:20.255 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:06:20.255 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_40 2024-04-06T10:06:20.255 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local snap_name=snap 2024-04-06T10:06:20.255 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local ret=0 2024-04-06T10:06:20.255 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local rmt_export=/tmp/tmp.BhW6xKmROU/cluster2-mirror-image_40.export 2024-04-06T10:06:20.255 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loc_export=/tmp/tmp.BhW6xKmROU/cluster1-mirror-image_40.export 2024-04-06T10:06:20.255 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rm -f /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_40.export /tmp/tmp.BhW6xKmROU/cluster1-mirror-image_40.export 2024-04-06T10:06:20.255 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster2 -p mirror export image_40@snap /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_40.export 2024-04-06T10:06:20.423 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-04-06T10:06:20.432 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 -p mirror export image_40@snap /tmp/tmp.BhW6xKmROU/cluster1-mirror-image_40.export 2024-04-06T10:06:20.742 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-04-06T10:06:20.752 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmp /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_40.export /tmp/tmp.BhW6xKmROU/cluster1-mirror-image_40.export 2024-04-06T10:06:20.830 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rm -f /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_40.export /tmp/tmp.BhW6xKmROU/cluster1-mirror-image_40.export 2024-04-06T10:06:20.874 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T10:06:20.874 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-06T10:06:20.874 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ image=image_41 2024-04-06T10:06:20.874 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ create_snap cluster2 mirror image_41 snap 2024-04-06T10:06:20.874 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-04-06T10:06:20.874 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:06:20.874 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_41 2024-04-06T10:06:20.874 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local snap_name=snap 2024-04-06T10:06:20.874 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster2 -p mirror snap create image_41@snap --debug-rbd=20 --debug-journaler=20 2024-04-06T10:06:22.946 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_image_replay_started cluster1 mirror image_41 2024-04-06T10:06:22.946 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T10:06:22.946 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:06:22.946 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_41 2024-04-06T10:06:22.946 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_image_replay_state cluster1 mirror image_41 started 2024-04-06T10:06:22.946 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T10:06:22.946 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:06:22.946 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_41 2024-04-06T10:06:22.946 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state=started 2024-04-06T10:06:22.946 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local s 2024-04-06T10:06:22.946 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T10:06:22.947 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 1 2024-04-06T10:06:23.947 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test_image_replay_state cluster1 mirror image_41 started 2024-04-06T10:06:23.947 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T10:06:23.948 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:06:23.948 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_41 2024-04-06T10:06:23.948 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local test_state=started 2024-04-06T10:06:23.948 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local status_result 2024-04-06T10:06:23.948 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local current_state=stopped 2024-04-06T10:06:23.949 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_41 2024-04-06T10:06:23.949 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -i state 2024-04-06T10:06:23.949 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_instance=cluster1 2024-04-06T10:06:23.949 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ shift 2024-04-06T10:06:23.949 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T10:06:23.949 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance=cluster1 2024-04-06T10:06:23.950 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local loop_instance 2024-04-06T10:06:23.950 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T10:06:23.950 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sleep 0 2024-04-06T10:06:23.950 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-04-06T10:06:23.951 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ seq 0 3 2024-04-06T10:06:23.954 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T10:06:23.954 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_41 2024-04-06T10:06:23.954 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1:0 2024-04-06T10:06:23.954 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ shift 2024-04-06T10:06:23.954 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T10:06:23.954 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-04-06T10:06:23.954 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T10:06:23.954 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=cluster 2024-04-06T10:06:23.954 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T10:06:23.954 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:06:23.954 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T10:06:23.954 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T10:06:23.955 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval cluster=cluster1 2024-04-06T10:06:23.955 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ cluster=cluster1 2024-04-06T10:06:23.955 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T10:06:23.955 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T10:06:23.955 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-04-06T10:06:23.955 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local local_cluster=cluster1:0 2024-04-06T10:06:23.955 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local cluster=cluster1 2024-04-06T10:06:23.955 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local instance 2024-04-06T10:06:23.955 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T10:06:23.955 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local val=cluster1:0 2024-04-06T10:06:23.955 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local cluster_var_name=local_cluster 2024-04-06T10:06:23.955 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local instance_var_name=instance 2024-04-06T10:06:23.955 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ cluster=cluster1 2024-04-06T10:06:23.955 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T10:06:23.955 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ '[' 0 = cluster1:0 ']' 2024-04-06T10:06:23.956 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ eval local_cluster=cluster1 2024-04-06T10:06:23.956 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++++ local_cluster=cluster1 2024-04-06T10:06:23.956 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ eval instance=0 2024-04-06T10:06:23.956 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++++ instance=0 2024-04-06T10:06:23.956 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T10:06:23.979 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T10:06:23.979 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T10:06:23.979 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T10:06:23.979 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_41 2024-04-06T10:06:24.081 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:Invalid command: unused arguments: ['mirror/image_41'] 2024-04-06T10:06:24.081 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:rbd mirror status : get status for rbd mirror 2024-04-06T10:06:24.081 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:admin_socket: invalid command 2024-04-06T10:06:24.084 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T10:06:24.084 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_41 2024-04-06T10:06:24.084 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1:1 2024-04-06T10:06:24.084 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ shift 2024-04-06T10:06:24.084 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T10:06:24.084 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-04-06T10:06:24.084 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:1 2024-04-06T10:06:24.084 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=cluster 2024-04-06T10:06:24.084 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T10:06:24.084 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:06:24.084 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=1 2024-04-06T10:06:24.085 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-06T10:06:24.085 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval cluster=cluster1 2024-04-06T10:06:24.085 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ cluster=cluster1 2024-04-06T10:06:24.085 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=1 2024-04-06T10:06:24.085 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=1 2024-04-06T10:06:24.086 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-04-06T10:06:24.086 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local local_cluster=cluster1:1 2024-04-06T10:06:24.086 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local cluster=cluster1 2024-04-06T10:06:24.086 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local instance 2024-04-06T10:06:24.086 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-06T10:06:24.087 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local val=cluster1:1 2024-04-06T10:06:24.087 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local cluster_var_name=local_cluster 2024-04-06T10:06:24.087 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local instance_var_name=instance 2024-04-06T10:06:24.087 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ cluster=cluster1 2024-04-06T10:06:24.087 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=1 2024-04-06T10:06:24.087 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ '[' 1 = cluster1:1 ']' 2024-04-06T10:06:24.087 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ eval local_cluster=cluster1 2024-04-06T10:06:24.087 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++++ local_cluster=cluster1 2024-04-06T10:06:24.087 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ eval instance=1 2024-04-06T10:06:24.087 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++++ instance=1 2024-04-06T10:06:24.087 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-06T10:06:24.108 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T10:06:24.109 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T10:06:24.109 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T10:06:24.109 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_41 2024-04-06T10:06:24.266 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ return 0 2024-04-06T10:06:24.267 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ status_result=' "state": "Replaying"' 2024-04-06T10:06:24.268 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ echo ' "state": "Replaying"' 2024-04-06T10:06:24.268 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep -i Replaying 2024-04-06T10:06:24.269 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: "state": "Replaying" 2024-04-06T10:06:24.270 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ current_state=started 2024-04-06T10:06:24.271 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test started = started 2024-04-06T10:06:24.271 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T10:06:24.271 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_41 2024-04-06T10:06:24.271 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local local_cluster=cluster1 2024-04-06T10:06:24.271 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-04-06T10:06:24.271 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:06:24.271 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_41 2024-04-06T10:06:24.271 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' journal = journal ']' 2024-04-06T10:06:24.271 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_journal_replay_complete cluster1 cluster2 mirror image_41 2024-04-06T10:06:24.271 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local local_cluster=cluster1 2024-04-06T10:06:24.271 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-04-06T10:06:24.271 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:06:24.272 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_41 2024-04-06T10:06:24.272 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local s master_pos mirror_pos last_mirror_pos 2024-04-06T10:06:24.272 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local master_tag master_entry mirror_tag mirror_entry 2024-04-06T10:06:24.272 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T10:06:24.272 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T10:06:24.272 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T10:06:24.472 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror image_41 2024-04-06T10:06:24.472 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T10:06:24.473 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:06:24.473 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_41 2024-04-06T10:06:24.473 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T10:06:24.473 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n image_41 ']' 2024-04-06T10:06:24.473 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/image_41' 2024-04-06T10:06:24.473 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_41 2024-04-06T10:06:24.473 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T10:06:24.473 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T10:06:24.473 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T10:06:24.473 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T10:06:24.473 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T10:06:24.473 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T10:06:24.473 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T10:06:24.474 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T10:06:24.474 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T10:06:24.477 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T10:06:24.477 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_41 2024-04-06T10:06:24.477 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T10:06:24.477 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T10:06:24.477 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T10:06:24.477 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T10:06:24.477 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T10:06:24.478 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T10:06:24.478 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T10:06:24.478 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T10:06:24.478 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T10:06:24.478 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T10:06:24.478 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T10:06:24.478 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:06:24.478 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T10:06:24.478 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T10:06:24.479 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T10:06:24.479 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T10:06:24.479 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T10:06:24.479 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T10:06:24.479 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T10:06:24.479 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T10:06:24.479 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T10:06:24.479 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T10:06:24.480 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:06:24.480 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T10:06:24.480 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T10:06:24.480 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T10:06:24.480 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T10:06:24.480 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T10:06:24.480 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T10:06:24.480 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T10:06:24.503 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T10:06:24.503 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T10:06:24.503 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T10:06:24.503 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_41 2024-04-06T10:06:24.615 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:Invalid command: unused arguments: ['mirror/image_41'] 2024-04-06T10:06:24.615 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:rbd mirror flush : flush rbd mirror 2024-04-06T10:06:24.615 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:admin_socket: invalid command 2024-04-06T10:06:24.618 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T10:06:24.618 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/image_41 2024-04-06T10:06:24.618 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:1 2024-04-06T10:06:24.618 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T10:06:24.618 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T10:06:24.618 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-06T10:06:24.618 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:1 2024-04-06T10:06:24.618 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T10:06:24.618 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T10:06:24.618 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T10:06:24.618 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=1 2024-04-06T10:06:24.619 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-06T10:06:24.619 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T10:06:24.619 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:06:24.619 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=1 2024-04-06T10:06:24.619 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=1 2024-04-06T10:06:24.620 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-06T10:06:24.620 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:1 2024-04-06T10:06:24.620 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T10:06:24.620 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T10:06:24.620 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-06T10:06:24.620 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:1 2024-04-06T10:06:24.620 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T10:06:24.620 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T10:06:24.620 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:06:24.620 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=1 2024-04-06T10:06:24.620 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-06T10:06:24.620 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T10:06:24.621 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T10:06:24.621 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=1 2024-04-06T10:06:24.621 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=1 2024-04-06T10:06:24.621 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-06T10:06:24.643 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T10:06:24.643 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T10:06:24.644 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T10:06:24.644 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/image_41 2024-04-06T10:06:24.798 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T10:06:24.800 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror image_41 2024-04-06T10:06:24.800 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:06:24.800 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:06:24.800 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_41 2024-04-06T10:06:24.800 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror image_41 '' 2024-04-06T10:06:24.800 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:06:24.800 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:06:24.800 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_41 2024-04-06T10:06:24.800 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T10:06:24.801 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-image_41.status 2024-04-06T10:06:24.801 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-image_41.status 2024-04-06T10:06:24.801 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T10:06:24.802 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-image_41.status 2024-04-06T10:06:24.803 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_41 2024-04-06T10:06:24.803 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_41.status 2024-04-06T10:06:24.852 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 2 2024-04-06T10:06:24.852 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 2 2024-04-06T10:06:24.853 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T10:06:24.853 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=9, tag_tid=2, entry_tid=1], [object_number=8, tag_tid=2, entry_tid=0], [object_number=11, tag_tid=1, entry_tid=11723], [object_number=10, tag_tid=1, entry_tid=11722]]], state=connected] 2024-04-06T10:06:24.853 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=9, tag_tid=2, entry_tid=1], [object_number=8, tag_tid=2, entry_tid=0], [object_number=11, tag_tid=1, entry_tid=11723], [object_number=10, tag_tid=1, entry_tid=11722]]], state=connected] 2024-04-06T10:06:24.856 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_41.status 2024-04-06T10:06:24.857 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=9, tag_tid=2, entry_tid=1' 2024-04-06T10:06:24.858 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror image_41 2024-04-06T10:06:24.858 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:06:24.858 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:06:24.858 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_41 2024-04-06T10:06:24.859 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror image_41 '..*' 2024-04-06T10:06:24.859 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:06:24.859 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:06:24.859 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_41 2024-04-06T10:06:24.859 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T10:06:24.859 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-image_41.status 2024-04-06T10:06:24.859 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-image_41.status 2024-04-06T10:06:24.859 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T10:06:24.860 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-image_41.status 2024-04-06T10:06:24.860 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_41 2024-04-06T10:06:24.860 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_41.status 2024-04-06T10:06:24.905 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 2 2024-04-06T10:06:24.905 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 2 2024-04-06T10:06:24.905 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T10:06:24.905 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=9, tag_tid=2, entry_tid=1], [object_number=8, tag_tid=2, entry_tid=0], [object_number=11, tag_tid=1, entry_tid=11723], [object_number=10, tag_tid=1, entry_tid=11722]]], state=connected] 2024-04-06T10:06:24.905 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=9, tag_tid=2, entry_tid=1], [object_number=8, tag_tid=2, entry_tid=0], [object_number=11, tag_tid=1, entry_tid=11723], [object_number=10, tag_tid=1, entry_tid=11722]]], state=connected] 2024-04-06T10:06:24.909 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_41.status 2024-04-06T10:06:24.911 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=9, tag_tid=2, entry_tid=1' 2024-04-06T10:06:24.911 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=9, tag_tid=2, entry_tid=1' -a 'object_number=9, tag_tid=2, entry_tid=1' = 'object_number=9, tag_tid=2, entry_tid=1' 2024-04-06T10:06:24.911 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T10:06:24.911 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_snap_present cluster1 mirror image_41 snap 2024-04-06T10:06:24.911 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T10:06:24.911 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:06:24.911 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_41 2024-04-06T10:06:24.911 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local snap_name=snap 2024-04-06T10:06:24.911 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local s 2024-04-06T10:06:24.912 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-04-06T10:06:24.912 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 1 2024-04-06T10:06:25.912 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 info mirror/image_41@snap 2024-04-06T10:06:25.979 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:rbd image 'image_41': 2024-04-06T10:06:25.980 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: size 128 MiB in 32 objects 2024-04-06T10:06:25.980 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: order 22 (4 MiB objects) 2024-04-06T10:06:25.980 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: snapshot_count: 1 2024-04-06T10:06:25.980 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: id: 10accb0d03d5 2024-04-06T10:06:25.980 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: block_name_prefix: rbd_data.10accb0d03d5 2024-04-06T10:06:25.980 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: format: 2 2024-04-06T10:06:25.980 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: features: layering, exclusive-lock, journaling 2024-04-06T10:06:25.980 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: op_features: 2024-04-06T10:06:25.980 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: flags: 2024-04-06T10:06:25.980 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: create_timestamp: Sat Apr 6 09:56:49 2024 2024-04-06T10:06:25.980 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: access_timestamp: Sat Apr 6 09:56:49 2024 2024-04-06T10:06:25.980 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: modify_timestamp: Sat Apr 6 09:57:50 2024 2024-04-06T10:06:25.980 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: protected: False 2024-04-06T10:06:25.980 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: journal: 10accb0d03d5 2024-04-06T10:06:25.980 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: mirroring state: enabled 2024-04-06T10:06:25.981 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: mirroring mode: journal 2024-04-06T10:06:25.981 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: mirroring global id: 53a28273-6a0a-4ec0-beb5-485a2da99128 2024-04-06T10:06:25.981 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: mirroring primary: false 2024-04-06T10:06:25.985 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T10:06:25.985 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ compare_image_snaps mirror image_41 snap 2024-04-06T10:06:25.985 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:06:25.985 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_41 2024-04-06T10:06:25.985 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local snap_name=snap 2024-04-06T10:06:25.985 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local ret=0 2024-04-06T10:06:25.985 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local rmt_export=/tmp/tmp.BhW6xKmROU/cluster2-mirror-image_41.export 2024-04-06T10:06:25.985 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loc_export=/tmp/tmp.BhW6xKmROU/cluster1-mirror-image_41.export 2024-04-06T10:06:25.985 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rm -f /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_41.export /tmp/tmp.BhW6xKmROU/cluster1-mirror-image_41.export 2024-04-06T10:06:25.986 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster2 -p mirror export image_41@snap /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_41.export 2024-04-06T10:06:26.156 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-04-06T10:06:26.163 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 -p mirror export image_41@snap /tmp/tmp.BhW6xKmROU/cluster1-mirror-image_41.export 2024-04-06T10:06:26.614 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-04-06T10:06:26.630 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmp /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_41.export /tmp/tmp.BhW6xKmROU/cluster1-mirror-image_41.export 2024-04-06T10:06:26.709 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rm -f /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_41.export /tmp/tmp.BhW6xKmROU/cluster1-mirror-image_41.export 2024-04-06T10:06:26.752 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T10:06:26.753 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-06T10:06:26.753 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ image=image_42 2024-04-06T10:06:26.753 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ create_snap cluster2 mirror image_42 snap 2024-04-06T10:06:26.753 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-04-06T10:06:26.753 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:06:26.753 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_42 2024-04-06T10:06:26.753 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local snap_name=snap 2024-04-06T10:06:26.753 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster2 -p mirror snap create image_42@snap --debug-rbd=20 --debug-journaler=20 2024-04-06T10:06:27.925 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_image_replay_started cluster1 mirror image_42 2024-04-06T10:06:27.925 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T10:06:27.925 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:06:27.925 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_42 2024-04-06T10:06:27.925 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_image_replay_state cluster1 mirror image_42 started 2024-04-06T10:06:27.926 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T10:06:27.926 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:06:27.926 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_42 2024-04-06T10:06:27.926 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state=started 2024-04-06T10:06:27.926 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local s 2024-04-06T10:06:27.926 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T10:06:27.926 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 1 2024-04-06T10:06:28.927 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test_image_replay_state cluster1 mirror image_42 started 2024-04-06T10:06:28.927 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T10:06:28.927 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:06:28.927 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_42 2024-04-06T10:06:28.927 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local test_state=started 2024-04-06T10:06:28.927 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local status_result 2024-04-06T10:06:28.927 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local current_state=stopped 2024-04-06T10:06:28.928 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_42 2024-04-06T10:06:28.928 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_instance=cluster1 2024-04-06T10:06:28.929 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ shift 2024-04-06T10:06:28.929 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T10:06:28.929 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance=cluster1 2024-04-06T10:06:28.929 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local loop_instance 2024-04-06T10:06:28.929 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T10:06:28.929 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sleep 0 2024-04-06T10:06:28.929 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -i state 2024-04-06T10:06:28.929 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-04-06T10:06:28.930 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ seq 0 3 2024-04-06T10:06:28.932 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T10:06:28.932 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_42 2024-04-06T10:06:28.932 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1:0 2024-04-06T10:06:28.933 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ shift 2024-04-06T10:06:28.933 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T10:06:28.933 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-04-06T10:06:28.933 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T10:06:28.933 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=cluster 2024-04-06T10:06:28.933 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T10:06:28.933 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:06:28.933 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T10:06:28.933 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T10:06:28.933 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval cluster=cluster1 2024-04-06T10:06:28.933 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ cluster=cluster1 2024-04-06T10:06:28.933 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T10:06:28.933 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T10:06:28.933 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-04-06T10:06:28.933 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local local_cluster=cluster1:0 2024-04-06T10:06:28.933 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local cluster=cluster1 2024-04-06T10:06:28.933 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local instance 2024-04-06T10:06:28.934 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T10:06:28.934 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local val=cluster1:0 2024-04-06T10:06:28.934 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local cluster_var_name=local_cluster 2024-04-06T10:06:28.934 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local instance_var_name=instance 2024-04-06T10:06:28.934 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ cluster=cluster1 2024-04-06T10:06:28.934 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T10:06:28.934 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ '[' 0 = cluster1:0 ']' 2024-04-06T10:06:28.934 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ eval local_cluster=cluster1 2024-04-06T10:06:28.934 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++++ local_cluster=cluster1 2024-04-06T10:06:28.934 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ eval instance=0 2024-04-06T10:06:28.934 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++++ instance=0 2024-04-06T10:06:28.935 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T10:06:28.958 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T10:06:28.958 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T10:06:28.958 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T10:06:28.958 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_42 2024-04-06T10:06:29.065 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:Invalid command: unused arguments: ['mirror/image_42'] 2024-04-06T10:06:29.065 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:rbd mirror status : get status for rbd mirror 2024-04-06T10:06:29.065 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:admin_socket: invalid command 2024-04-06T10:06:29.067 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T10:06:29.067 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_42 2024-04-06T10:06:29.067 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1:1 2024-04-06T10:06:29.067 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ shift 2024-04-06T10:06:29.067 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T10:06:29.067 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-04-06T10:06:29.067 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:1 2024-04-06T10:06:29.067 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=cluster 2024-04-06T10:06:29.067 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T10:06:29.067 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:06:29.067 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=1 2024-04-06T10:06:29.068 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-06T10:06:29.068 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval cluster=cluster1 2024-04-06T10:06:29.068 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ cluster=cluster1 2024-04-06T10:06:29.068 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=1 2024-04-06T10:06:29.068 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=1 2024-04-06T10:06:29.069 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-04-06T10:06:29.069 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local local_cluster=cluster1:1 2024-04-06T10:06:29.069 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local cluster=cluster1 2024-04-06T10:06:29.069 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local instance 2024-04-06T10:06:29.069 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-06T10:06:29.070 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local val=cluster1:1 2024-04-06T10:06:29.070 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local cluster_var_name=local_cluster 2024-04-06T10:06:29.070 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local instance_var_name=instance 2024-04-06T10:06:29.070 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ cluster=cluster1 2024-04-06T10:06:29.070 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=1 2024-04-06T10:06:29.070 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ '[' 1 = cluster1:1 ']' 2024-04-06T10:06:29.070 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ eval local_cluster=cluster1 2024-04-06T10:06:29.070 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++++ local_cluster=cluster1 2024-04-06T10:06:29.070 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ eval instance=1 2024-04-06T10:06:29.070 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++++ instance=1 2024-04-06T10:06:29.070 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-06T10:06:29.092 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T10:06:29.093 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T10:06:29.093 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T10:06:29.093 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_42 2024-04-06T10:06:29.251 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ return 0 2024-04-06T10:06:29.251 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ status_result=' "state": "Replaying"' 2024-04-06T10:06:29.252 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ echo ' "state": "Replaying"' 2024-04-06T10:06:29.252 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep -i Replaying 2024-04-06T10:06:29.254 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: "state": "Replaying" 2024-04-06T10:06:29.255 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ current_state=started 2024-04-06T10:06:29.255 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test started = started 2024-04-06T10:06:29.256 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T10:06:29.256 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_42 2024-04-06T10:06:29.256 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local local_cluster=cluster1 2024-04-06T10:06:29.256 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-04-06T10:06:29.256 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:06:29.256 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_42 2024-04-06T10:06:29.256 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' journal = journal ']' 2024-04-06T10:06:29.256 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_journal_replay_complete cluster1 cluster2 mirror image_42 2024-04-06T10:06:29.256 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local local_cluster=cluster1 2024-04-06T10:06:29.256 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-04-06T10:06:29.256 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:06:29.256 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_42 2024-04-06T10:06:29.257 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local s master_pos mirror_pos last_mirror_pos 2024-04-06T10:06:29.257 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local master_tag master_entry mirror_tag mirror_entry 2024-04-06T10:06:29.257 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T10:06:29.257 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T10:06:29.257 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T10:06:29.458 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror image_42 2024-04-06T10:06:29.458 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T10:06:29.458 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:06:29.458 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_42 2024-04-06T10:06:29.458 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T10:06:29.458 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n image_42 ']' 2024-04-06T10:06:29.458 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/image_42' 2024-04-06T10:06:29.458 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_42 2024-04-06T10:06:29.458 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T10:06:29.458 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T10:06:29.458 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T10:06:29.459 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T10:06:29.459 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T10:06:29.459 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T10:06:29.459 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T10:06:29.459 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T10:06:29.459 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T10:06:29.462 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T10:06:29.462 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_42 2024-04-06T10:06:29.462 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T10:06:29.462 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T10:06:29.462 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T10:06:29.462 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T10:06:29.462 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T10:06:29.462 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T10:06:29.462 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T10:06:29.463 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T10:06:29.463 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T10:06:29.463 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T10:06:29.463 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T10:06:29.463 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:06:29.463 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T10:06:29.463 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T10:06:29.464 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T10:06:29.464 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T10:06:29.464 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T10:06:29.464 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T10:06:29.464 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T10:06:29.464 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T10:06:29.464 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T10:06:29.464 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T10:06:29.464 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:06:29.465 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T10:06:29.465 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T10:06:29.465 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T10:06:29.465 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T10:06:29.465 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T10:06:29.465 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T10:06:29.465 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T10:06:29.489 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T10:06:29.490 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T10:06:29.490 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T10:06:29.490 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_42 2024-04-06T10:06:29.595 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:Invalid command: unused arguments: ['mirror/image_42'] 2024-04-06T10:06:29.595 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:rbd mirror flush : flush rbd mirror 2024-04-06T10:06:29.595 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:admin_socket: invalid command 2024-04-06T10:06:29.598 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T10:06:29.599 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/image_42 2024-04-06T10:06:29.599 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:1 2024-04-06T10:06:29.599 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T10:06:29.599 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T10:06:29.599 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-06T10:06:29.599 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:1 2024-04-06T10:06:29.599 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T10:06:29.599 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T10:06:29.599 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T10:06:29.599 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=1 2024-04-06T10:06:29.599 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-06T10:06:29.599 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T10:06:29.599 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:06:29.600 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=1 2024-04-06T10:06:29.600 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=1 2024-04-06T10:06:29.600 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-06T10:06:29.600 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:1 2024-04-06T10:06:29.600 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T10:06:29.600 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T10:06:29.600 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-06T10:06:29.600 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:1 2024-04-06T10:06:29.600 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T10:06:29.600 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T10:06:29.600 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:06:29.600 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=1 2024-04-06T10:06:29.600 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-06T10:06:29.600 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T10:06:29.600 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T10:06:29.601 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=1 2024-04-06T10:06:29.601 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=1 2024-04-06T10:06:29.601 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-06T10:06:29.623 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T10:06:29.623 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T10:06:29.624 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T10:06:29.624 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/image_42 2024-04-06T10:06:29.767 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T10:06:29.768 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror image_42 2024-04-06T10:06:29.768 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:06:29.769 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:06:29.769 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_42 2024-04-06T10:06:29.769 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror image_42 '' 2024-04-06T10:06:29.769 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:06:29.769 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:06:29.769 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_42 2024-04-06T10:06:29.769 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T10:06:29.770 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-image_42.status 2024-04-06T10:06:29.770 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-image_42.status 2024-04-06T10:06:29.770 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T10:06:29.772 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-image_42.status 2024-04-06T10:06:29.772 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_42.status 2024-04-06T10:06:29.772 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_42 2024-04-06T10:06:29.823 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 1 2024-04-06T10:06:29.823 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 1 2024-04-06T10:06:29.824 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T10:06:29.824 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=5, tag_tid=2, entry_tid=1], [object_number=4, tag_tid=2, entry_tid=0], [object_number=6, tag_tid=1, entry_tid=7518], [object_number=7, tag_tid=1, entry_tid=7515]]], state=connected] 2024-04-06T10:06:29.824 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=6, tag_tid=1, entry_tid=7518], [object_number=5, tag_tid=1, entry_tid=7517], [object_number=4, tag_tid=1, entry_tid=7516], [object_number=7, tag_tid=1, entry_tid=7515]]], state=connected] 2024-04-06T10:06:29.827 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_42.status 2024-04-06T10:06:29.829 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=5, tag_tid=2, entry_tid=1' 2024-04-06T10:06:29.829 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror image_42 2024-04-06T10:06:29.829 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:06:29.829 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:06:29.829 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_42 2024-04-06T10:06:29.830 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror image_42 '..*' 2024-04-06T10:06:29.830 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:06:29.830 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:06:29.831 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_42 2024-04-06T10:06:29.831 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T10:06:29.831 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-image_42.status 2024-04-06T10:06:29.831 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-image_42.status 2024-04-06T10:06:29.831 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T10:06:29.831 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-image_42.status 2024-04-06T10:06:29.831 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_42 2024-04-06T10:06:29.832 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_42.status 2024-04-06T10:06:29.878 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 1 2024-04-06T10:06:29.878 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 1 2024-04-06T10:06:29.878 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T10:06:29.878 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=5, tag_tid=2, entry_tid=1], [object_number=4, tag_tid=2, entry_tid=0], [object_number=6, tag_tid=1, entry_tid=7518], [object_number=7, tag_tid=1, entry_tid=7515]]], state=connected] 2024-04-06T10:06:29.878 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=6, tag_tid=1, entry_tid=7518], [object_number=5, tag_tid=1, entry_tid=7517], [object_number=4, tag_tid=1, entry_tid=7516], [object_number=7, tag_tid=1, entry_tid=7515]]], state=connected] 2024-04-06T10:06:29.881 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_42.status 2024-04-06T10:06:29.884 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=6, tag_tid=1, entry_tid=7518' 2024-04-06T10:06:29.884 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=5, tag_tid=2, entry_tid=1' -a 'object_number=5, tag_tid=2, entry_tid=1' = 'object_number=6, tag_tid=1, entry_tid=7518' 2024-04-06T10:06:29.884 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=6, tag_tid=1, entry_tid=7518' '!=' '' 2024-04-06T10:06:29.884 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T10:06:29.884 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=6, tag_tid=1, entry_tid=7518' = '' 2024-04-06T10:06:29.884 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=6, tag_tid=1, entry_tid=7518' 2024-04-06T10:06:29.884 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=5, tag_tid=2, entry_tid=1' 2024-04-06T10:06:29.884 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T10:06:29.884 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T10:06:29.885 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=2 2024-04-06T10:06:29.886 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=6, tag_tid=1, entry_tid=7518' 2024-04-06T10:06:29.886 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T10:06:29.886 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T10:06:29.887 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=1 2024-04-06T10:06:29.887 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=5, tag_tid=2, entry_tid=1' 2024-04-06T10:06:29.887 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T10:06:29.887 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T10:06:29.888 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T10:06:29.889 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=6, tag_tid=1, entry_tid=7518' 2024-04-06T10:06:29.889 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T10:06:29.889 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T10:06:29.891 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=7518 2024-04-06T10:06:29.891 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 2 = 1 -a 1 -le 7518 2024-04-06T10:06:29.891 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T10:06:29.891 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T10:06:29.891 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T10:06:30.092 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror image_42 2024-04-06T10:06:30.093 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T10:06:30.093 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:06:30.093 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_42 2024-04-06T10:06:30.093 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T10:06:30.093 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n image_42 ']' 2024-04-06T10:06:30.093 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/image_42' 2024-04-06T10:06:30.093 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_42 2024-04-06T10:06:30.093 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T10:06:30.093 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T10:06:30.093 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T10:06:30.093 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T10:06:30.093 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T10:06:30.093 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T10:06:30.094 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T10:06:30.094 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T10:06:30.094 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T10:06:30.098 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T10:06:30.098 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_42 2024-04-06T10:06:30.098 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T10:06:30.098 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T10:06:30.098 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T10:06:30.098 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T10:06:30.098 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T10:06:30.098 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T10:06:30.098 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T10:06:30.098 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T10:06:30.098 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T10:06:30.098 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T10:06:30.099 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T10:06:30.099 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:06:30.099 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T10:06:30.099 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T10:06:30.099 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T10:06:30.099 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T10:06:30.099 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T10:06:30.099 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T10:06:30.099 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T10:06:30.099 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T10:06:30.099 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T10:06:30.099 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T10:06:30.099 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:06:30.099 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T10:06:30.099 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T10:06:30.100 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T10:06:30.100 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T10:06:30.100 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T10:06:30.100 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T10:06:30.100 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T10:06:30.124 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T10:06:30.124 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T10:06:30.124 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T10:06:30.124 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_42 2024-04-06T10:06:30.233 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:Invalid command: unused arguments: ['mirror/image_42'] 2024-04-06T10:06:30.233 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:rbd mirror flush : flush rbd mirror 2024-04-06T10:06:30.233 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:admin_socket: invalid command 2024-04-06T10:06:30.235 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T10:06:30.235 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/image_42 2024-04-06T10:06:30.235 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:1 2024-04-06T10:06:30.235 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T10:06:30.236 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T10:06:30.236 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-06T10:06:30.236 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:1 2024-04-06T10:06:30.236 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T10:06:30.236 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T10:06:30.236 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T10:06:30.236 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=1 2024-04-06T10:06:30.236 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-06T10:06:30.236 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T10:06:30.236 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:06:30.236 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=1 2024-04-06T10:06:30.236 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=1 2024-04-06T10:06:30.236 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-06T10:06:30.236 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:1 2024-04-06T10:06:30.236 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T10:06:30.237 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T10:06:30.237 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-06T10:06:30.237 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:1 2024-04-06T10:06:30.237 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T10:06:30.237 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T10:06:30.237 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:06:30.237 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=1 2024-04-06T10:06:30.237 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-06T10:06:30.237 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T10:06:30.237 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T10:06:30.237 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=1 2024-04-06T10:06:30.237 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=1 2024-04-06T10:06:30.238 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-06T10:06:30.260 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T10:06:30.260 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T10:06:30.260 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T10:06:30.261 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/image_42 2024-04-06T10:06:30.412 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T10:06:30.413 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror image_42 2024-04-06T10:06:30.413 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:06:30.413 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:06:30.413 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_42 2024-04-06T10:06:30.413 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror image_42 '' 2024-04-06T10:06:30.413 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:06:30.413 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:06:30.413 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_42 2024-04-06T10:06:30.413 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T10:06:30.414 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-image_42.status 2024-04-06T10:06:30.414 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-image_42.status 2024-04-06T10:06:30.414 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T10:06:30.415 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-image_42.status 2024-04-06T10:06:30.416 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_42.status 2024-04-06T10:06:30.416 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_42 2024-04-06T10:06:30.466 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 1 2024-04-06T10:06:30.466 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 1 2024-04-06T10:06:30.467 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T10:06:30.467 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=5, tag_tid=2, entry_tid=1], [object_number=4, tag_tid=2, entry_tid=0], [object_number=6, tag_tid=1, entry_tid=7518], [object_number=7, tag_tid=1, entry_tid=7515]]], state=connected] 2024-04-06T10:06:30.467 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=6, tag_tid=1, entry_tid=7518], [object_number=5, tag_tid=1, entry_tid=7517], [object_number=4, tag_tid=1, entry_tid=7516], [object_number=7, tag_tid=1, entry_tid=7515]]], state=connected] 2024-04-06T10:06:30.501 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_42.status 2024-04-06T10:06:30.502 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=5, tag_tid=2, entry_tid=1' 2024-04-06T10:06:30.504 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror image_42 2024-04-06T10:06:30.504 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:06:30.504 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:06:30.504 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_42 2024-04-06T10:06:30.504 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror image_42 '..*' 2024-04-06T10:06:30.504 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:06:30.505 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:06:30.505 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_42 2024-04-06T10:06:30.505 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T10:06:30.505 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-image_42.status 2024-04-06T10:06:30.505 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T10:06:30.505 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-image_42.status 2024-04-06T10:06:30.505 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-image_42.status 2024-04-06T10:06:30.506 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_42 2024-04-06T10:06:30.506 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_42.status 2024-04-06T10:06:30.553 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 1 2024-04-06T10:06:30.553 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 1 2024-04-06T10:06:30.553 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T10:06:30.553 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=5, tag_tid=2, entry_tid=1], [object_number=4, tag_tid=2, entry_tid=0], [object_number=6, tag_tid=1, entry_tid=7518], [object_number=7, tag_tid=1, entry_tid=7515]]], state=connected] 2024-04-06T10:06:30.553 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=6, tag_tid=1, entry_tid=7518], [object_number=5, tag_tid=1, entry_tid=7517], [object_number=4, tag_tid=1, entry_tid=7516], [object_number=7, tag_tid=1, entry_tid=7515]]], state=connected] 2024-04-06T10:06:30.556 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_42.status 2024-04-06T10:06:30.558 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=6, tag_tid=1, entry_tid=7518' 2024-04-06T10:06:30.558 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=5, tag_tid=2, entry_tid=1' -a 'object_number=5, tag_tid=2, entry_tid=1' = 'object_number=6, tag_tid=1, entry_tid=7518' 2024-04-06T10:06:30.559 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=6, tag_tid=1, entry_tid=7518' '!=' 'object_number=6, tag_tid=1, entry_tid=7518' 2024-04-06T10:06:30.559 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T10:06:30.559 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.4 2024-04-06T10:06:30.960 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror image_42 2024-04-06T10:06:30.961 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T10:06:30.961 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:06:30.961 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_42 2024-04-06T10:06:30.961 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T10:06:30.961 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n image_42 ']' 2024-04-06T10:06:30.961 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/image_42' 2024-04-06T10:06:30.961 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_42 2024-04-06T10:06:30.961 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T10:06:30.961 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T10:06:30.961 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T10:06:30.961 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T10:06:30.961 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T10:06:30.961 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T10:06:30.962 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T10:06:30.963 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T10:06:30.963 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T10:06:30.963 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T10:06:30.963 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_42 2024-04-06T10:06:30.964 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T10:06:30.964 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T10:06:30.964 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T10:06:30.964 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T10:06:30.964 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T10:06:30.964 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T10:06:30.964 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T10:06:30.964 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T10:06:30.964 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T10:06:30.964 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T10:06:30.964 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T10:06:30.964 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:06:30.964 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T10:06:30.965 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T10:06:30.967 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T10:06:30.967 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T10:06:30.967 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T10:06:30.967 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T10:06:30.967 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T10:06:30.967 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T10:06:30.967 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T10:06:30.967 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T10:06:30.967 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:06:30.967 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T10:06:30.967 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T10:06:30.968 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T10:06:30.968 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T10:06:30.968 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T10:06:30.968 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T10:06:30.968 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T10:06:30.991 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T10:06:30.991 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T10:06:30.991 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T10:06:30.991 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_42 2024-04-06T10:06:31.095 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:Invalid command: unused arguments: ['mirror/image_42'] 2024-04-06T10:06:31.096 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:rbd mirror flush : flush rbd mirror 2024-04-06T10:06:31.096 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:admin_socket: invalid command 2024-04-06T10:06:31.100 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T10:06:31.100 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/image_42 2024-04-06T10:06:31.100 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:1 2024-04-06T10:06:31.100 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T10:06:31.100 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T10:06:31.100 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-06T10:06:31.100 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:1 2024-04-06T10:06:31.100 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T10:06:31.100 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T10:06:31.100 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T10:06:31.100 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=1 2024-04-06T10:06:31.100 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-06T10:06:31.100 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T10:06:31.101 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:06:31.101 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=1 2024-04-06T10:06:31.101 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=1 2024-04-06T10:06:31.101 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-06T10:06:31.101 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:1 2024-04-06T10:06:31.101 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T10:06:31.101 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T10:06:31.101 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-06T10:06:31.101 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:1 2024-04-06T10:06:31.101 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T10:06:31.101 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T10:06:31.101 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:06:31.101 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=1 2024-04-06T10:06:31.101 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-06T10:06:31.101 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T10:06:31.102 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T10:06:31.102 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=1 2024-04-06T10:06:31.102 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=1 2024-04-06T10:06:31.102 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-06T10:06:31.119 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T10:06:31.119 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T10:06:31.119 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T10:06:31.119 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/image_42 2024-04-06T10:06:31.259 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T10:06:31.260 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror image_42 2024-04-06T10:06:31.260 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:06:31.260 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:06:31.260 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_42 2024-04-06T10:06:31.260 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror image_42 '' 2024-04-06T10:06:31.260 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:06:31.260 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:06:31.260 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_42 2024-04-06T10:06:31.260 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T10:06:31.261 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-image_42.status 2024-04-06T10:06:31.261 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-image_42.status 2024-04-06T10:06:31.261 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T10:06:31.263 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-image_42.status 2024-04-06T10:06:31.263 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_42 2024-04-06T10:06:31.263 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_42.status 2024-04-06T10:06:31.320 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 1 2024-04-06T10:06:31.320 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 1 2024-04-06T10:06:31.321 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T10:06:31.321 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=5, tag_tid=2, entry_tid=1], [object_number=4, tag_tid=2, entry_tid=0], [object_number=6, tag_tid=1, entry_tid=7518], [object_number=7, tag_tid=1, entry_tid=7515]]], state=connected] 2024-04-06T10:06:31.321 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=6, tag_tid=1, entry_tid=7518], [object_number=5, tag_tid=1, entry_tid=7517], [object_number=4, tag_tid=1, entry_tid=7516], [object_number=7, tag_tid=1, entry_tid=7515]]], state=connected] 2024-04-06T10:06:31.324 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_42.status 2024-04-06T10:06:31.325 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=5, tag_tid=2, entry_tid=1' 2024-04-06T10:06:31.326 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror image_42 2024-04-06T10:06:31.327 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:06:31.327 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:06:31.327 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_42 2024-04-06T10:06:31.327 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror image_42 '..*' 2024-04-06T10:06:31.327 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:06:31.327 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:06:31.327 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_42 2024-04-06T10:06:31.327 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T10:06:31.327 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-image_42.status 2024-04-06T10:06:31.327 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-image_42.status 2024-04-06T10:06:31.327 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T10:06:31.328 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-image_42.status 2024-04-06T10:06:31.328 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_42 2024-04-06T10:06:31.328 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_42.status 2024-04-06T10:06:31.425 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 1 2024-04-06T10:06:31.425 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 1 2024-04-06T10:06:31.425 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T10:06:31.425 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=5, tag_tid=2, entry_tid=1], [object_number=4, tag_tid=2, entry_tid=0], [object_number=6, tag_tid=1, entry_tid=7518], [object_number=7, tag_tid=1, entry_tid=7515]]], state=connected] 2024-04-06T10:06:31.426 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=6, tag_tid=1, entry_tid=7518], [object_number=5, tag_tid=1, entry_tid=7517], [object_number=4, tag_tid=1, entry_tid=7516], [object_number=7, tag_tid=1, entry_tid=7515]]], state=connected] 2024-04-06T10:06:31.426 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_42.status 2024-04-06T10:06:31.428 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=6, tag_tid=1, entry_tid=7518' 2024-04-06T10:06:31.428 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=5, tag_tid=2, entry_tid=1' -a 'object_number=5, tag_tid=2, entry_tid=1' = 'object_number=6, tag_tid=1, entry_tid=7518' 2024-04-06T10:06:31.428 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=6, tag_tid=1, entry_tid=7518' '!=' 'object_number=6, tag_tid=1, entry_tid=7518' 2024-04-06T10:06:31.428 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T10:06:31.428 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.8 2024-04-06T10:06:32.230 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror image_42 2024-04-06T10:06:32.231 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T10:06:32.231 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:06:32.231 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_42 2024-04-06T10:06:32.231 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T10:06:32.231 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n image_42 ']' 2024-04-06T10:06:32.231 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/image_42' 2024-04-06T10:06:32.231 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_42 2024-04-06T10:06:32.231 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T10:06:32.231 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T10:06:32.231 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T10:06:32.231 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T10:06:32.231 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T10:06:32.232 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T10:06:32.232 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T10:06:32.232 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T10:06:32.232 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T10:06:32.234 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T10:06:32.234 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_42 2024-04-06T10:06:32.234 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T10:06:32.234 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T10:06:32.234 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T10:06:32.234 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T10:06:32.234 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T10:06:32.234 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T10:06:32.234 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T10:06:32.234 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T10:06:32.234 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T10:06:32.234 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T10:06:32.235 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T10:06:32.235 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:06:32.235 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T10:06:32.235 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T10:06:32.236 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T10:06:32.236 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T10:06:32.236 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T10:06:32.236 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T10:06:32.236 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T10:06:32.237 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T10:06:32.237 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T10:06:32.237 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T10:06:32.237 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:06:32.237 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T10:06:32.237 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T10:06:32.237 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T10:06:32.237 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T10:06:32.237 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T10:06:32.237 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T10:06:32.237 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T10:06:32.260 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T10:06:32.261 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T10:06:32.261 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T10:06:32.261 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_42 2024-04-06T10:06:32.366 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:Invalid command: unused arguments: ['mirror/image_42'] 2024-04-06T10:06:32.366 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:rbd mirror flush : flush rbd mirror 2024-04-06T10:06:32.366 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:admin_socket: invalid command 2024-04-06T10:06:32.370 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T10:06:32.370 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/image_42 2024-04-06T10:06:32.370 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:1 2024-04-06T10:06:32.370 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T10:06:32.371 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T10:06:32.371 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-06T10:06:32.371 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:1 2024-04-06T10:06:32.371 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T10:06:32.371 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T10:06:32.371 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T10:06:32.371 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=1 2024-04-06T10:06:32.371 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-06T10:06:32.371 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T10:06:32.371 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:06:32.371 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=1 2024-04-06T10:06:32.371 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=1 2024-04-06T10:06:32.371 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-06T10:06:32.371 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:1 2024-04-06T10:06:32.372 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T10:06:32.372 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T10:06:32.372 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-06T10:06:32.372 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:1 2024-04-06T10:06:32.372 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T10:06:32.372 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T10:06:32.372 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:06:32.372 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=1 2024-04-06T10:06:32.372 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-06T10:06:32.372 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T10:06:32.372 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T10:06:32.372 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=1 2024-04-06T10:06:32.372 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=1 2024-04-06T10:06:32.372 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-06T10:06:32.395 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T10:06:32.395 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T10:06:32.395 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T10:06:32.395 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/image_42 2024-04-06T10:06:32.534 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T10:06:32.536 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror image_42 2024-04-06T10:06:32.536 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:06:32.536 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:06:32.536 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_42 2024-04-06T10:06:32.536 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror image_42 '' 2024-04-06T10:06:32.536 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:06:32.536 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:06:32.536 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_42 2024-04-06T10:06:32.536 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T10:06:32.536 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-image_42.status 2024-04-06T10:06:32.537 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T10:06:32.537 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-image_42.status 2024-04-06T10:06:32.537 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-image_42.status 2024-04-06T10:06:32.538 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_42 2024-04-06T10:06:32.538 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_42.status 2024-04-06T10:06:32.792 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 1 2024-04-06T10:06:32.792 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 1 2024-04-06T10:06:32.792 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T10:06:32.792 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=5, tag_tid=2, entry_tid=1], [object_number=4, tag_tid=2, entry_tid=0], [object_number=6, tag_tid=1, entry_tid=7518], [object_number=7, tag_tid=1, entry_tid=7515]]], state=connected] 2024-04-06T10:06:32.792 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=6, tag_tid=1, entry_tid=7518], [object_number=5, tag_tid=1, entry_tid=7517], [object_number=4, tag_tid=1, entry_tid=7516], [object_number=7, tag_tid=1, entry_tid=7515]]], state=connected] 2024-04-06T10:06:32.798 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_42.status 2024-04-06T10:06:32.800 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=5, tag_tid=2, entry_tid=1' 2024-04-06T10:06:32.801 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror image_42 2024-04-06T10:06:32.801 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:06:32.802 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:06:32.802 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_42 2024-04-06T10:06:32.802 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror image_42 '..*' 2024-04-06T10:06:32.802 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:06:32.802 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:06:32.802 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_42 2024-04-06T10:06:32.802 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T10:06:32.802 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-image_42.status 2024-04-06T10:06:32.803 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-image_42.status 2024-04-06T10:06:32.803 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T10:06:32.804 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-image_42.status 2024-04-06T10:06:32.805 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_42 2024-04-06T10:06:32.805 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_42.status 2024-04-06T10:06:32.858 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 1 2024-04-06T10:06:32.858 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 1 2024-04-06T10:06:32.858 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T10:06:32.858 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=5, tag_tid=2, entry_tid=1], [object_number=4, tag_tid=2, entry_tid=0], [object_number=6, tag_tid=1, entry_tid=7518], [object_number=7, tag_tid=1, entry_tid=7515]]], state=connected] 2024-04-06T10:06:32.858 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=6, tag_tid=1, entry_tid=7518], [object_number=5, tag_tid=1, entry_tid=7517], [object_number=4, tag_tid=1, entry_tid=7516], [object_number=7, tag_tid=1, entry_tid=7515]]], state=connected] 2024-04-06T10:06:32.862 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_42.status 2024-04-06T10:06:32.863 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=6, tag_tid=1, entry_tid=7518' 2024-04-06T10:06:32.864 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=5, tag_tid=2, entry_tid=1' -a 'object_number=5, tag_tid=2, entry_tid=1' = 'object_number=6, tag_tid=1, entry_tid=7518' 2024-04-06T10:06:32.864 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=6, tag_tid=1, entry_tid=7518' '!=' 'object_number=6, tag_tid=1, entry_tid=7518' 2024-04-06T10:06:32.864 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T10:06:32.864 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 1.6 2024-04-06T10:06:34.465 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror image_42 2024-04-06T10:06:34.466 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T10:06:34.466 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:06:34.466 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_42 2024-04-06T10:06:34.466 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T10:06:34.466 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n image_42 ']' 2024-04-06T10:06:34.466 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/image_42' 2024-04-06T10:06:34.466 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_42 2024-04-06T10:06:34.466 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T10:06:34.466 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T10:06:34.467 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T10:06:34.467 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T10:06:34.467 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T10:06:34.467 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T10:06:34.467 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T10:06:34.467 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T10:06:34.468 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T10:06:34.471 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T10:06:34.471 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_42 2024-04-06T10:06:34.472 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T10:06:34.472 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T10:06:34.472 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T10:06:34.472 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T10:06:34.472 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T10:06:34.472 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T10:06:34.472 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T10:06:34.472 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T10:06:34.472 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T10:06:34.472 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T10:06:34.472 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T10:06:34.472 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:06:34.472 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T10:06:34.473 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T10:06:34.473 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T10:06:34.473 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T10:06:34.473 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T10:06:34.473 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T10:06:34.473 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T10:06:34.473 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T10:06:34.473 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T10:06:34.473 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T10:06:34.474 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:06:34.475 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T10:06:34.475 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T10:06:34.475 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T10:06:34.475 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T10:06:34.475 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T10:06:34.475 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T10:06:34.475 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T10:06:34.497 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T10:06:34.498 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T10:06:34.498 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T10:06:34.498 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_42 2024-04-06T10:06:34.604 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:Invalid command: unused arguments: ['mirror/image_42'] 2024-04-06T10:06:34.604 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:rbd mirror flush : flush rbd mirror 2024-04-06T10:06:34.604 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:admin_socket: invalid command 2024-04-06T10:06:34.607 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T10:06:34.607 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/image_42 2024-04-06T10:06:34.607 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:1 2024-04-06T10:06:34.607 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T10:06:34.607 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T10:06:34.608 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-06T10:06:34.608 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:1 2024-04-06T10:06:34.608 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T10:06:34.608 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T10:06:34.608 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T10:06:34.608 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=1 2024-04-06T10:06:34.608 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-06T10:06:34.608 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T10:06:34.608 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:06:34.608 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=1 2024-04-06T10:06:34.608 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=1 2024-04-06T10:06:34.608 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-06T10:06:34.609 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:1 2024-04-06T10:06:34.609 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T10:06:34.609 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T10:06:34.609 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-06T10:06:34.609 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:1 2024-04-06T10:06:34.609 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T10:06:34.609 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T10:06:34.609 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:06:34.609 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=1 2024-04-06T10:06:34.609 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-06T10:06:34.609 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T10:06:34.609 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T10:06:34.609 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=1 2024-04-06T10:06:34.609 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=1 2024-04-06T10:06:34.610 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-06T10:06:34.633 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T10:06:34.634 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T10:06:34.634 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T10:06:34.634 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/image_42 2024-04-06T10:06:34.776 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T10:06:34.777 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror image_42 2024-04-06T10:06:34.778 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:06:34.778 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:06:34.778 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_42 2024-04-06T10:06:34.778 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror image_42 '' 2024-04-06T10:06:34.778 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:06:34.778 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:06:34.778 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_42 2024-04-06T10:06:34.778 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T10:06:34.778 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-image_42.status 2024-04-06T10:06:34.778 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-image_42.status 2024-04-06T10:06:34.779 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T10:06:34.781 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-image_42.status 2024-04-06T10:06:34.781 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_42 2024-04-06T10:06:34.781 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_42.status 2024-04-06T10:06:34.825 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 1 2024-04-06T10:06:34.825 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 1 2024-04-06T10:06:34.825 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T10:06:34.825 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=5, tag_tid=2, entry_tid=1], [object_number=4, tag_tid=2, entry_tid=0], [object_number=6, tag_tid=1, entry_tid=7518], [object_number=7, tag_tid=1, entry_tid=7515]]], state=connected] 2024-04-06T10:06:34.825 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=6, tag_tid=1, entry_tid=7518], [object_number=5, tag_tid=1, entry_tid=7517], [object_number=4, tag_tid=1, entry_tid=7516], [object_number=7, tag_tid=1, entry_tid=7515]]], state=connected] 2024-04-06T10:06:34.829 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_42.status 2024-04-06T10:06:34.830 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=5, tag_tid=2, entry_tid=1' 2024-04-06T10:06:34.831 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror image_42 2024-04-06T10:06:34.832 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:06:34.832 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:06:34.832 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_42 2024-04-06T10:06:34.832 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror image_42 '..*' 2024-04-06T10:06:34.832 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:06:34.832 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:06:34.832 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_42 2024-04-06T10:06:34.832 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T10:06:34.832 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-image_42.status 2024-04-06T10:06:34.832 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-image_42.status 2024-04-06T10:06:34.832 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T10:06:34.833 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-image_42.status 2024-04-06T10:06:34.833 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_42 2024-04-06T10:06:34.833 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_42.status 2024-04-06T10:06:34.879 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 1 2024-04-06T10:06:34.879 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 1 2024-04-06T10:06:34.880 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T10:06:34.880 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=5, tag_tid=2, entry_tid=1], [object_number=4, tag_tid=2, entry_tid=0], [object_number=6, tag_tid=1, entry_tid=7518], [object_number=7, tag_tid=1, entry_tid=7515]]], state=connected] 2024-04-06T10:06:34.880 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=6, tag_tid=1, entry_tid=7518], [object_number=5, tag_tid=1, entry_tid=7517], [object_number=4, tag_tid=1, entry_tid=7516], [object_number=7, tag_tid=1, entry_tid=7515]]], state=connected] 2024-04-06T10:06:34.883 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_42.status 2024-04-06T10:06:34.885 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=6, tag_tid=1, entry_tid=7518' 2024-04-06T10:06:34.885 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=5, tag_tid=2, entry_tid=1' -a 'object_number=5, tag_tid=2, entry_tid=1' = 'object_number=6, tag_tid=1, entry_tid=7518' 2024-04-06T10:06:34.885 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=6, tag_tid=1, entry_tid=7518' '!=' 'object_number=6, tag_tid=1, entry_tid=7518' 2024-04-06T10:06:34.885 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T10:06:34.885 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 2 2024-04-06T10:06:36.887 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror image_42 2024-04-06T10:06:36.887 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T10:06:36.887 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:06:36.887 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_42 2024-04-06T10:06:36.887 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T10:06:36.888 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n image_42 ']' 2024-04-06T10:06:36.888 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/image_42' 2024-04-06T10:06:36.888 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_42 2024-04-06T10:06:36.888 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T10:06:36.888 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T10:06:36.888 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T10:06:36.888 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T10:06:36.888 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T10:06:36.888 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T10:06:36.888 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T10:06:36.889 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T10:06:36.889 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T10:06:36.891 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T10:06:36.891 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_42 2024-04-06T10:06:36.892 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T10:06:36.892 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T10:06:36.892 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T10:06:36.892 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T10:06:36.892 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T10:06:36.892 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T10:06:36.892 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T10:06:36.892 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T10:06:36.892 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T10:06:36.892 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T10:06:36.892 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T10:06:36.892 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:06:36.892 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T10:06:36.892 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T10:06:36.893 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T10:06:36.893 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T10:06:36.893 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T10:06:36.894 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T10:06:36.894 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T10:06:36.894 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T10:06:36.894 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T10:06:36.894 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T10:06:36.894 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:06:36.894 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T10:06:36.894 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T10:06:36.894 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T10:06:36.894 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T10:06:36.894 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T10:06:36.894 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T10:06:36.895 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T10:06:36.918 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T10:06:36.918 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T10:06:36.918 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T10:06:36.918 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_42 2024-04-06T10:06:37.023 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:Invalid command: unused arguments: ['mirror/image_42'] 2024-04-06T10:06:37.023 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:rbd mirror flush : flush rbd mirror 2024-04-06T10:06:37.024 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:admin_socket: invalid command 2024-04-06T10:06:37.026 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T10:06:37.026 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/image_42 2024-04-06T10:06:37.026 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:1 2024-04-06T10:06:37.026 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T10:06:37.026 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T10:06:37.026 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-06T10:06:37.026 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:1 2024-04-06T10:06:37.026 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T10:06:37.026 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T10:06:37.026 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T10:06:37.026 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=1 2024-04-06T10:06:37.026 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-06T10:06:37.027 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T10:06:37.027 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:06:37.027 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=1 2024-04-06T10:06:37.027 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=1 2024-04-06T10:06:37.028 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-06T10:06:37.028 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:1 2024-04-06T10:06:37.028 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T10:06:37.028 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T10:06:37.028 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-06T10:06:37.028 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:1 2024-04-06T10:06:37.029 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T10:06:37.029 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T10:06:37.029 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:06:37.029 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=1 2024-04-06T10:06:37.029 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-06T10:06:37.029 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T10:06:37.029 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T10:06:37.029 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=1 2024-04-06T10:06:37.029 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=1 2024-04-06T10:06:37.029 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-06T10:06:37.047 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T10:06:37.047 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T10:06:37.048 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T10:06:37.048 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/image_42 2024-04-06T10:06:37.199 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T10:06:37.200 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror image_42 2024-04-06T10:06:37.200 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:06:37.200 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:06:37.201 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_42 2024-04-06T10:06:37.201 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror image_42 '' 2024-04-06T10:06:37.201 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:06:37.201 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:06:37.201 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_42 2024-04-06T10:06:37.201 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T10:06:37.201 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-image_42.status 2024-04-06T10:06:37.202 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-image_42.status 2024-04-06T10:06:37.202 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T10:06:37.204 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-image_42.status 2024-04-06T10:06:37.205 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_42 2024-04-06T10:06:37.205 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_42.status 2024-04-06T10:06:37.250 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 1 2024-04-06T10:06:37.250 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 1 2024-04-06T10:06:37.250 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T10:06:37.250 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=5, tag_tid=2, entry_tid=1], [object_number=4, tag_tid=2, entry_tid=0], [object_number=6, tag_tid=1, entry_tid=7518], [object_number=7, tag_tid=1, entry_tid=7515]]], state=connected] 2024-04-06T10:06:37.250 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=5, tag_tid=2, entry_tid=1], [object_number=4, tag_tid=2, entry_tid=0], [object_number=6, tag_tid=1, entry_tid=7518], [object_number=7, tag_tid=1, entry_tid=7515]]], state=connected] 2024-04-06T10:06:37.253 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_42.status 2024-04-06T10:06:37.255 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=5, tag_tid=2, entry_tid=1' 2024-04-06T10:06:37.256 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror image_42 2024-04-06T10:06:37.256 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:06:37.256 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:06:37.256 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_42 2024-04-06T10:06:37.256 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror image_42 '..*' 2024-04-06T10:06:37.256 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:06:37.256 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:06:37.256 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_42 2024-04-06T10:06:37.257 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T10:06:37.257 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-image_42.status 2024-04-06T10:06:37.257 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-image_42.status 2024-04-06T10:06:37.257 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T10:06:37.257 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-image_42.status 2024-04-06T10:06:37.257 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_42 2024-04-06T10:06:37.258 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_42.status 2024-04-06T10:06:37.512 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 1 2024-04-06T10:06:37.512 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 1 2024-04-06T10:06:37.512 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T10:06:37.512 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=5, tag_tid=2, entry_tid=1], [object_number=4, tag_tid=2, entry_tid=0], [object_number=6, tag_tid=1, entry_tid=7518], [object_number=7, tag_tid=1, entry_tid=7515]]], state=connected] 2024-04-06T10:06:37.513 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=5, tag_tid=2, entry_tid=1], [object_number=4, tag_tid=2, entry_tid=0], [object_number=6, tag_tid=1, entry_tid=7518], [object_number=7, tag_tid=1, entry_tid=7515]]], state=connected] 2024-04-06T10:06:37.517 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_42.status 2024-04-06T10:06:37.519 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=5, tag_tid=2, entry_tid=1' 2024-04-06T10:06:37.519 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=5, tag_tid=2, entry_tid=1' -a 'object_number=5, tag_tid=2, entry_tid=1' = 'object_number=5, tag_tid=2, entry_tid=1' 2024-04-06T10:06:37.519 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T10:06:37.519 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_snap_present cluster1 mirror image_42 snap 2024-04-06T10:06:37.519 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T10:06:37.519 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:06:37.519 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_42 2024-04-06T10:06:37.519 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local snap_name=snap 2024-04-06T10:06:37.519 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local s 2024-04-06T10:06:37.519 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-04-06T10:06:37.519 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 1 2024-04-06T10:06:38.520 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 info mirror/image_42@snap 2024-04-06T10:06:38.582 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:rbd image 'image_42': 2024-04-06T10:06:38.583 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: size 128 MiB in 32 objects 2024-04-06T10:06:38.583 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: order 22 (4 MiB objects) 2024-04-06T10:06:38.583 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: snapshot_count: 1 2024-04-06T10:06:38.583 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: id: 10ac8332b075 2024-04-06T10:06:38.583 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: block_name_prefix: rbd_data.10ac8332b075 2024-04-06T10:06:38.583 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: format: 2 2024-04-06T10:06:38.583 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: features: layering, exclusive-lock, journaling 2024-04-06T10:06:38.583 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: op_features: 2024-04-06T10:06:38.583 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: flags: 2024-04-06T10:06:38.583 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: create_timestamp: Sat Apr 6 09:57:04 2024 2024-04-06T10:06:38.583 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: access_timestamp: Sat Apr 6 09:57:04 2024 2024-04-06T10:06:38.583 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: modify_timestamp: Sat Apr 6 09:58:05 2024 2024-04-06T10:06:38.584 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: protected: False 2024-04-06T10:06:38.584 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: journal: 10ac8332b075 2024-04-06T10:06:38.584 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: mirroring state: enabled 2024-04-06T10:06:38.584 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: mirroring mode: journal 2024-04-06T10:06:38.584 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: mirroring global id: 8aac32aa-6459-4c5f-a099-a2cb9aff6b69 2024-04-06T10:06:38.584 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: mirroring primary: false 2024-04-06T10:06:38.589 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T10:06:38.589 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ compare_image_snaps mirror image_42 snap 2024-04-06T10:06:38.589 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:06:38.589 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_42 2024-04-06T10:06:38.589 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local snap_name=snap 2024-04-06T10:06:38.590 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local ret=0 2024-04-06T10:06:38.590 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local rmt_export=/tmp/tmp.BhW6xKmROU/cluster2-mirror-image_42.export 2024-04-06T10:06:38.590 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loc_export=/tmp/tmp.BhW6xKmROU/cluster1-mirror-image_42.export 2024-04-06T10:06:38.590 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rm -f /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_42.export /tmp/tmp.BhW6xKmROU/cluster1-mirror-image_42.export 2024-04-06T10:06:38.590 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster2 -p mirror export image_42@snap /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_42.export 2024-04-06T10:06:38.749 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-04-06T10:06:38.757 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 -p mirror export image_42@snap /tmp/tmp.BhW6xKmROU/cluster1-mirror-image_42.export 2024-04-06T10:06:39.085 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-04-06T10:06:39.097 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmp /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_42.export /tmp/tmp.BhW6xKmROU/cluster1-mirror-image_42.export 2024-04-06T10:06:39.178 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rm -f /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_42.export /tmp/tmp.BhW6xKmROU/cluster1-mirror-image_42.export 2024-04-06T10:06:39.220 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T10:06:39.220 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-06T10:06:39.220 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ image=image_43 2024-04-06T10:06:39.220 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ create_snap cluster2 mirror image_43 snap 2024-04-06T10:06:39.220 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-04-06T10:06:39.220 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:06:39.220 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_43 2024-04-06T10:06:39.220 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local snap_name=snap 2024-04-06T10:06:39.220 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster2 -p mirror snap create image_43@snap --debug-rbd=20 --debug-journaler=20 2024-04-06T10:06:40.897 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_image_replay_started cluster1 mirror image_43 2024-04-06T10:06:40.897 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T10:06:40.897 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:06:40.897 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_43 2024-04-06T10:06:40.897 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_image_replay_state cluster1 mirror image_43 started 2024-04-06T10:06:40.898 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T10:06:40.898 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:06:40.898 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_43 2024-04-06T10:06:40.898 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state=started 2024-04-06T10:06:40.898 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local s 2024-04-06T10:06:40.898 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T10:06:40.898 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 1 2024-04-06T10:06:41.898 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test_image_replay_state cluster1 mirror image_43 started 2024-04-06T10:06:41.899 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T10:06:41.899 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:06:41.899 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_43 2024-04-06T10:06:41.899 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local test_state=started 2024-04-06T10:06:41.899 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local status_result 2024-04-06T10:06:41.899 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local current_state=stopped 2024-04-06T10:06:41.900 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_43 2024-04-06T10:06:41.900 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_instance=cluster1 2024-04-06T10:06:41.900 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ shift 2024-04-06T10:06:41.900 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T10:06:41.900 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -i state 2024-04-06T10:06:41.900 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance=cluster1 2024-04-06T10:06:41.900 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local loop_instance 2024-04-06T10:06:41.900 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T10:06:41.900 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sleep 0 2024-04-06T10:06:41.901 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-04-06T10:06:41.902 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ seq 0 3 2024-04-06T10:06:41.904 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T10:06:41.904 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_43 2024-04-06T10:06:41.904 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1:0 2024-04-06T10:06:41.904 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ shift 2024-04-06T10:06:41.904 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T10:06:41.904 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-04-06T10:06:41.904 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T10:06:41.904 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=cluster 2024-04-06T10:06:41.904 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T10:06:41.905 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:06:41.905 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T10:06:41.905 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T10:06:41.905 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval cluster=cluster1 2024-04-06T10:06:41.905 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ cluster=cluster1 2024-04-06T10:06:41.905 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T10:06:41.905 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T10:06:41.905 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-04-06T10:06:41.905 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local local_cluster=cluster1:0 2024-04-06T10:06:41.905 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local cluster=cluster1 2024-04-06T10:06:41.905 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local instance 2024-04-06T10:06:41.906 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T10:06:41.906 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local val=cluster1:0 2024-04-06T10:06:41.907 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local cluster_var_name=local_cluster 2024-04-06T10:06:41.907 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local instance_var_name=instance 2024-04-06T10:06:41.907 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ cluster=cluster1 2024-04-06T10:06:41.907 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T10:06:41.907 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ '[' 0 = cluster1:0 ']' 2024-04-06T10:06:41.907 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ eval local_cluster=cluster1 2024-04-06T10:06:41.908 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++++ local_cluster=cluster1 2024-04-06T10:06:41.908 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ eval instance=0 2024-04-06T10:06:41.908 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++++ instance=0 2024-04-06T10:06:41.908 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T10:06:41.926 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T10:06:41.926 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T10:06:41.927 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T10:06:41.927 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_43 2024-04-06T10:06:42.032 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:Invalid command: unused arguments: ['mirror/image_43'] 2024-04-06T10:06:42.032 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:rbd mirror status : get status for rbd mirror 2024-04-06T10:06:42.032 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:admin_socket: invalid command 2024-04-06T10:06:42.034 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T10:06:42.034 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_43 2024-04-06T10:06:42.034 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1:1 2024-04-06T10:06:42.034 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ shift 2024-04-06T10:06:42.034 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T10:06:42.034 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-04-06T10:06:42.034 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:1 2024-04-06T10:06:42.034 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=cluster 2024-04-06T10:06:42.034 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T10:06:42.034 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:06:42.035 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=1 2024-04-06T10:06:42.035 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-06T10:06:42.035 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval cluster=cluster1 2024-04-06T10:06:42.035 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ cluster=cluster1 2024-04-06T10:06:42.035 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=1 2024-04-06T10:06:42.035 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=1 2024-04-06T10:06:42.037 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-04-06T10:06:42.037 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local local_cluster=cluster1:1 2024-04-06T10:06:42.037 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local cluster=cluster1 2024-04-06T10:06:42.037 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local instance 2024-04-06T10:06:42.037 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-06T10:06:42.037 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local val=cluster1:1 2024-04-06T10:06:42.037 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local cluster_var_name=local_cluster 2024-04-06T10:06:42.037 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local instance_var_name=instance 2024-04-06T10:06:42.037 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ cluster=cluster1 2024-04-06T10:06:42.037 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=1 2024-04-06T10:06:42.037 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ '[' 1 = cluster1:1 ']' 2024-04-06T10:06:42.037 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ eval local_cluster=cluster1 2024-04-06T10:06:42.038 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++++ local_cluster=cluster1 2024-04-06T10:06:42.038 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ eval instance=1 2024-04-06T10:06:42.038 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++++ instance=1 2024-04-06T10:06:42.038 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-06T10:06:42.054 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T10:06:42.055 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T10:06:42.055 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T10:06:42.055 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_43 2024-04-06T10:06:42.213 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ return 0 2024-04-06T10:06:42.214 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ status_result=' "state": "Replaying"' 2024-04-06T10:06:42.214 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ echo ' "state": "Replaying"' 2024-04-06T10:06:42.214 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep -i Replaying 2024-04-06T10:06:42.216 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: "state": "Replaying" 2024-04-06T10:06:42.218 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ current_state=started 2024-04-06T10:06:42.218 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test started = started 2024-04-06T10:06:42.218 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T10:06:42.218 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_43 2024-04-06T10:06:42.218 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local local_cluster=cluster1 2024-04-06T10:06:42.218 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-04-06T10:06:42.219 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:06:42.219 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_43 2024-04-06T10:06:42.219 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' journal = journal ']' 2024-04-06T10:06:42.219 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_journal_replay_complete cluster1 cluster2 mirror image_43 2024-04-06T10:06:42.219 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local local_cluster=cluster1 2024-04-06T10:06:42.219 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-04-06T10:06:42.219 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:06:42.219 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_43 2024-04-06T10:06:42.219 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local s master_pos mirror_pos last_mirror_pos 2024-04-06T10:06:42.219 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local master_tag master_entry mirror_tag mirror_entry 2024-04-06T10:06:42.219 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T10:06:42.219 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T10:06:42.219 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T10:06:42.419 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror image_43 2024-04-06T10:06:42.420 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T10:06:42.420 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:06:42.420 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_43 2024-04-06T10:06:42.420 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T10:06:42.420 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n image_43 ']' 2024-04-06T10:06:42.420 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/image_43' 2024-04-06T10:06:42.420 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_43 2024-04-06T10:06:42.420 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T10:06:42.420 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T10:06:42.420 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T10:06:42.420 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T10:06:42.420 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T10:06:42.421 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T10:06:42.421 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T10:06:42.421 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T10:06:42.422 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T10:06:42.423 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T10:06:42.423 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_43 2024-04-06T10:06:42.424 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T10:06:42.424 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T10:06:42.424 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T10:06:42.424 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T10:06:42.424 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T10:06:42.424 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T10:06:42.424 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T10:06:42.424 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T10:06:42.424 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T10:06:42.424 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T10:06:42.424 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T10:06:42.424 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:06:42.424 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T10:06:42.424 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T10:06:42.426 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T10:06:42.426 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T10:06:42.426 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T10:06:42.426 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T10:06:42.426 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T10:06:42.426 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T10:06:42.426 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T10:06:42.427 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T10:06:42.427 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:06:42.427 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T10:06:42.427 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T10:06:42.427 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T10:06:42.427 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T10:06:42.427 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T10:06:42.427 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T10:06:42.427 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T10:06:42.452 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T10:06:42.452 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T10:06:42.452 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T10:06:42.452 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_43 2024-04-06T10:06:42.569 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:Invalid command: unused arguments: ['mirror/image_43'] 2024-04-06T10:06:42.569 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:rbd mirror flush : flush rbd mirror 2024-04-06T10:06:42.569 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:admin_socket: invalid command 2024-04-06T10:06:42.571 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T10:06:42.571 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/image_43 2024-04-06T10:06:42.572 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:1 2024-04-06T10:06:42.572 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T10:06:42.572 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T10:06:42.572 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-06T10:06:42.572 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:1 2024-04-06T10:06:42.572 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T10:06:42.572 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T10:06:42.572 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T10:06:42.572 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=1 2024-04-06T10:06:42.572 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-06T10:06:42.572 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T10:06:42.572 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:06:42.572 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=1 2024-04-06T10:06:42.572 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=1 2024-04-06T10:06:42.574 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-06T10:06:42.574 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:1 2024-04-06T10:06:42.574 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T10:06:42.574 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T10:06:42.574 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-06T10:06:42.574 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:1 2024-04-06T10:06:42.574 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T10:06:42.574 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T10:06:42.574 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:06:42.574 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=1 2024-04-06T10:06:42.575 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-06T10:06:42.575 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T10:06:42.575 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T10:06:42.575 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=1 2024-04-06T10:06:42.575 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=1 2024-04-06T10:06:42.575 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-06T10:06:42.596 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T10:06:42.597 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T10:06:42.597 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T10:06:42.597 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/image_43 2024-04-06T10:06:42.755 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T10:06:42.756 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror image_43 2024-04-06T10:06:42.756 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:06:42.756 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:06:42.756 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_43 2024-04-06T10:06:42.757 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror image_43 '' 2024-04-06T10:06:42.757 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:06:42.757 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:06:42.757 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_43 2024-04-06T10:06:42.757 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T10:06:42.757 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-image_43.status 2024-04-06T10:06:42.758 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-image_43.status 2024-04-06T10:06:42.758 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T10:06:42.759 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-image_43.status 2024-04-06T10:06:42.760 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_43.status 2024-04-06T10:06:42.760 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_43 2024-04-06T10:06:42.815 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 2 2024-04-06T10:06:42.815 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 2 2024-04-06T10:06:42.816 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T10:06:42.816 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=9, tag_tid=2, entry_tid=1], [object_number=8, tag_tid=2, entry_tid=0], [object_number=11, tag_tid=1, entry_tid=9751], [object_number=10, tag_tid=1, entry_tid=9750]]], state=connected] 2024-04-06T10:06:42.816 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=9, tag_tid=2, entry_tid=1], [object_number=8, tag_tid=2, entry_tid=0], [object_number=11, tag_tid=1, entry_tid=9751], [object_number=10, tag_tid=1, entry_tid=9750]]], state=connected] 2024-04-06T10:06:42.819 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_43.status 2024-04-06T10:06:42.821 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=9, tag_tid=2, entry_tid=1' 2024-04-06T10:06:42.822 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror image_43 2024-04-06T10:06:42.822 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:06:42.822 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:06:42.822 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_43 2024-04-06T10:06:42.822 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror image_43 '..*' 2024-04-06T10:06:42.822 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:06:42.822 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:06:42.822 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_43 2024-04-06T10:06:42.822 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T10:06:42.823 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-image_43.status 2024-04-06T10:06:42.823 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-image_43.status 2024-04-06T10:06:42.823 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T10:06:42.823 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-image_43.status 2024-04-06T10:06:42.823 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_43 2024-04-06T10:06:42.823 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_43.status 2024-04-06T10:06:42.868 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 2 2024-04-06T10:06:42.868 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 2 2024-04-06T10:06:42.869 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T10:06:42.869 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=9, tag_tid=2, entry_tid=1], [object_number=8, tag_tid=2, entry_tid=0], [object_number=11, tag_tid=1, entry_tid=9751], [object_number=10, tag_tid=1, entry_tid=9750]]], state=connected] 2024-04-06T10:06:42.869 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=9, tag_tid=2, entry_tid=1], [object_number=8, tag_tid=2, entry_tid=0], [object_number=11, tag_tid=1, entry_tid=9751], [object_number=10, tag_tid=1, entry_tid=9750]]], state=connected] 2024-04-06T10:06:42.872 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_43.status 2024-04-06T10:06:42.874 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=9, tag_tid=2, entry_tid=1' 2024-04-06T10:06:42.874 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=9, tag_tid=2, entry_tid=1' -a 'object_number=9, tag_tid=2, entry_tid=1' = 'object_number=9, tag_tid=2, entry_tid=1' 2024-04-06T10:06:42.874 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T10:06:42.874 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_snap_present cluster1 mirror image_43 snap 2024-04-06T10:06:42.874 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T10:06:42.874 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:06:42.874 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_43 2024-04-06T10:06:42.875 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local snap_name=snap 2024-04-06T10:06:42.875 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local s 2024-04-06T10:06:42.875 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-04-06T10:06:42.875 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 1 2024-04-06T10:06:43.875 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 info mirror/image_43@snap 2024-04-06T10:06:43.942 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:rbd image 'image_43': 2024-04-06T10:06:43.942 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: size 128 MiB in 32 objects 2024-04-06T10:06:43.942 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: order 22 (4 MiB objects) 2024-04-06T10:06:43.942 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: snapshot_count: 1 2024-04-06T10:06:43.942 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: id: 10acbb4c7791 2024-04-06T10:06:43.942 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: block_name_prefix: rbd_data.10acbb4c7791 2024-04-06T10:06:43.942 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: format: 2 2024-04-06T10:06:43.942 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: features: layering, exclusive-lock, journaling 2024-04-06T10:06:43.942 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: op_features: 2024-04-06T10:06:43.942 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: flags: 2024-04-06T10:06:43.943 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: create_timestamp: Sat Apr 6 09:57:13 2024 2024-04-06T10:06:43.943 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: access_timestamp: Sat Apr 6 09:57:13 2024 2024-04-06T10:06:43.943 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: modify_timestamp: Sat Apr 6 09:58:14 2024 2024-04-06T10:06:43.943 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: protected: False 2024-04-06T10:06:43.943 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: journal: 10acbb4c7791 2024-04-06T10:06:43.943 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: mirroring state: enabled 2024-04-06T10:06:43.943 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: mirroring mode: journal 2024-04-06T10:06:43.943 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: mirroring global id: a724e9e9-b8cb-4cc9-826a-522071a5ebd9 2024-04-06T10:06:43.943 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: mirroring primary: false 2024-04-06T10:06:43.947 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T10:06:43.947 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ compare_image_snaps mirror image_43 snap 2024-04-06T10:06:43.948 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:06:43.948 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_43 2024-04-06T10:06:43.948 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local snap_name=snap 2024-04-06T10:06:43.948 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local ret=0 2024-04-06T10:06:43.948 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local rmt_export=/tmp/tmp.BhW6xKmROU/cluster2-mirror-image_43.export 2024-04-06T10:06:43.948 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loc_export=/tmp/tmp.BhW6xKmROU/cluster1-mirror-image_43.export 2024-04-06T10:06:43.948 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rm -f /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_43.export /tmp/tmp.BhW6xKmROU/cluster1-mirror-image_43.export 2024-04-06T10:06:43.949 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster2 -p mirror export image_43@snap /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_43.export 2024-04-06T10:06:44.132 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-04-06T10:06:44.140 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 -p mirror export image_43@snap /tmp/tmp.BhW6xKmROU/cluster1-mirror-image_43.export 2024-04-06T10:06:44.559 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-04-06T10:06:44.576 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmp /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_43.export /tmp/tmp.BhW6xKmROU/cluster1-mirror-image_43.export 2024-04-06T10:06:44.658 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rm -f /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_43.export /tmp/tmp.BhW6xKmROU/cluster1-mirror-image_43.export 2024-04-06T10:06:44.700 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T10:06:44.700 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-06T10:06:44.700 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ image=image_44 2024-04-06T10:06:44.700 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ create_snap cluster2 mirror image_44 snap 2024-04-06T10:06:44.700 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-04-06T10:06:44.700 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:06:44.701 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_44 2024-04-06T10:06:44.701 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local snap_name=snap 2024-04-06T10:06:44.701 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster2 -p mirror snap create image_44@snap --debug-rbd=20 --debug-journaler=20 2024-04-06T10:06:44.921 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_image_replay_started cluster1 mirror image_44 2024-04-06T10:06:44.921 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T10:06:44.922 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:06:44.922 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_44 2024-04-06T10:06:44.922 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_image_replay_state cluster1 mirror image_44 started 2024-04-06T10:06:44.922 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T10:06:44.922 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:06:44.922 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_44 2024-04-06T10:06:44.922 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state=started 2024-04-06T10:06:44.922 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local s 2024-04-06T10:06:44.922 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T10:06:44.922 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 1 2024-04-06T10:06:45.923 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test_image_replay_state cluster1 mirror image_44 started 2024-04-06T10:06:45.923 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T10:06:45.923 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:06:45.923 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_44 2024-04-06T10:06:45.923 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local test_state=started 2024-04-06T10:06:45.923 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local status_result 2024-04-06T10:06:45.923 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local current_state=stopped 2024-04-06T10:06:45.924 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_44 2024-04-06T10:06:45.925 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -i state 2024-04-06T10:06:45.925 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_instance=cluster1 2024-04-06T10:06:45.925 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ shift 2024-04-06T10:06:45.925 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T10:06:45.925 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance=cluster1 2024-04-06T10:06:45.925 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local loop_instance 2024-04-06T10:06:45.925 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T10:06:45.925 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sleep 0 2024-04-06T10:06:45.926 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-04-06T10:06:45.926 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ seq 0 3 2024-04-06T10:06:45.928 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T10:06:45.929 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_44 2024-04-06T10:06:45.929 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1:0 2024-04-06T10:06:45.929 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ shift 2024-04-06T10:06:45.929 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T10:06:45.929 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-04-06T10:06:45.929 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T10:06:45.929 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=cluster 2024-04-06T10:06:45.929 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T10:06:45.929 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:06:45.929 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T10:06:45.929 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T10:06:45.929 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval cluster=cluster1 2024-04-06T10:06:45.929 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ cluster=cluster1 2024-04-06T10:06:45.930 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T10:06:45.930 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T10:06:45.930 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-04-06T10:06:45.930 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local local_cluster=cluster1:0 2024-04-06T10:06:45.930 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local cluster=cluster1 2024-04-06T10:06:45.931 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local instance 2024-04-06T10:06:45.931 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T10:06:45.931 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local val=cluster1:0 2024-04-06T10:06:45.931 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local cluster_var_name=local_cluster 2024-04-06T10:06:45.931 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local instance_var_name=instance 2024-04-06T10:06:45.931 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ cluster=cluster1 2024-04-06T10:06:45.932 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T10:06:45.932 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ '[' 0 = cluster1:0 ']' 2024-04-06T10:06:45.932 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ eval local_cluster=cluster1 2024-04-06T10:06:45.932 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++++ local_cluster=cluster1 2024-04-06T10:06:45.932 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ eval instance=0 2024-04-06T10:06:45.932 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++++ instance=0 2024-04-06T10:06:45.932 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T10:06:45.951 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T10:06:45.951 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T10:06:45.951 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T10:06:45.951 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_44 2024-04-06T10:06:46.057 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:Invalid command: unused arguments: ['mirror/image_44'] 2024-04-06T10:06:46.057 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:rbd mirror status : get status for rbd mirror 2024-04-06T10:06:46.057 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:admin_socket: invalid command 2024-04-06T10:06:46.059 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T10:06:46.059 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_44 2024-04-06T10:06:46.059 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1:1 2024-04-06T10:06:46.059 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ shift 2024-04-06T10:06:46.060 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T10:06:46.060 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-04-06T10:06:46.060 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:1 2024-04-06T10:06:46.060 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=cluster 2024-04-06T10:06:46.060 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T10:06:46.060 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:06:46.060 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=1 2024-04-06T10:06:46.060 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-06T10:06:46.060 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval cluster=cluster1 2024-04-06T10:06:46.060 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ cluster=cluster1 2024-04-06T10:06:46.060 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=1 2024-04-06T10:06:46.060 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=1 2024-04-06T10:06:46.062 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-04-06T10:06:46.062 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local local_cluster=cluster1:1 2024-04-06T10:06:46.062 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local cluster=cluster1 2024-04-06T10:06:46.062 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local instance 2024-04-06T10:06:46.062 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-06T10:06:46.062 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local val=cluster1:1 2024-04-06T10:06:46.062 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local cluster_var_name=local_cluster 2024-04-06T10:06:46.062 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local instance_var_name=instance 2024-04-06T10:06:46.062 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ cluster=cluster1 2024-04-06T10:06:46.062 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=1 2024-04-06T10:06:46.062 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ '[' 1 = cluster1:1 ']' 2024-04-06T10:06:46.062 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ eval local_cluster=cluster1 2024-04-06T10:06:46.062 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++++ local_cluster=cluster1 2024-04-06T10:06:46.062 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ eval instance=1 2024-04-06T10:06:46.063 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++++ instance=1 2024-04-06T10:06:46.063 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-06T10:06:46.085 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T10:06:46.085 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T10:06:46.085 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T10:06:46.085 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_44 2024-04-06T10:06:46.243 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ return 0 2024-04-06T10:06:46.243 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ status_result=' "state": "Replaying"' 2024-04-06T10:06:46.244 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ echo ' "state": "Replaying"' 2024-04-06T10:06:46.244 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep -i Replaying 2024-04-06T10:06:46.245 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: "state": "Replaying" 2024-04-06T10:06:46.246 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ current_state=started 2024-04-06T10:06:46.246 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test started = started 2024-04-06T10:06:46.246 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T10:06:46.247 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_44 2024-04-06T10:06:46.247 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local local_cluster=cluster1 2024-04-06T10:06:46.247 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-04-06T10:06:46.247 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:06:46.247 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_44 2024-04-06T10:06:46.247 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' journal = journal ']' 2024-04-06T10:06:46.247 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_journal_replay_complete cluster1 cluster2 mirror image_44 2024-04-06T10:06:46.247 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local local_cluster=cluster1 2024-04-06T10:06:46.247 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-04-06T10:06:46.247 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:06:46.247 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_44 2024-04-06T10:06:46.247 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local s master_pos mirror_pos last_mirror_pos 2024-04-06T10:06:46.247 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local master_tag master_entry mirror_tag mirror_entry 2024-04-06T10:06:46.247 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T10:06:46.247 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T10:06:46.247 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T10:06:46.448 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror image_44 2024-04-06T10:06:46.448 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T10:06:46.448 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:06:46.449 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_44 2024-04-06T10:06:46.449 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T10:06:46.449 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n image_44 ']' 2024-04-06T10:06:46.449 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/image_44' 2024-04-06T10:06:46.449 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_44 2024-04-06T10:06:46.449 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T10:06:46.449 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T10:06:46.449 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T10:06:46.449 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T10:06:46.449 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T10:06:46.449 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T10:06:46.449 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T10:06:46.450 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T10:06:46.450 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T10:06:46.452 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T10:06:46.452 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_44 2024-04-06T10:06:46.452 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T10:06:46.452 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T10:06:46.452 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T10:06:46.452 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T10:06:46.452 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T10:06:46.452 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T10:06:46.452 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T10:06:46.452 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T10:06:46.452 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T10:06:46.452 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T10:06:46.452 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T10:06:46.453 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:06:46.453 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T10:06:46.453 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T10:06:46.454 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T10:06:46.454 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T10:06:46.454 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T10:06:46.454 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T10:06:46.454 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T10:06:46.454 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T10:06:46.455 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T10:06:46.455 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T10:06:46.455 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:06:46.455 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T10:06:46.455 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T10:06:46.455 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T10:06:46.455 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T10:06:46.455 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T10:06:46.455 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T10:06:46.455 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T10:06:46.481 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T10:06:46.481 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T10:06:46.481 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T10:06:46.481 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_44 2024-04-06T10:06:46.600 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:Invalid command: unused arguments: ['mirror/image_44'] 2024-04-06T10:06:46.600 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:rbd mirror flush : flush rbd mirror 2024-04-06T10:06:46.601 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:admin_socket: invalid command 2024-04-06T10:06:46.603 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T10:06:46.603 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/image_44 2024-04-06T10:06:46.603 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:1 2024-04-06T10:06:46.603 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T10:06:46.603 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T10:06:46.603 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-06T10:06:46.603 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:1 2024-04-06T10:06:46.603 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T10:06:46.603 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T10:06:46.603 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T10:06:46.603 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=1 2024-04-06T10:06:46.603 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-06T10:06:46.603 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T10:06:46.604 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:06:46.604 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=1 2024-04-06T10:06:46.604 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=1 2024-04-06T10:06:46.605 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-06T10:06:46.606 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:1 2024-04-06T10:06:46.606 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T10:06:46.606 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T10:06:46.606 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-06T10:06:46.606 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:1 2024-04-06T10:06:46.606 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T10:06:46.606 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T10:06:46.606 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:06:46.606 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=1 2024-04-06T10:06:46.606 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-06T10:06:46.606 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T10:06:46.606 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T10:06:46.606 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=1 2024-04-06T10:06:46.607 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=1 2024-04-06T10:06:46.607 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-06T10:06:46.626 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T10:06:46.626 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T10:06:46.627 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T10:06:46.627 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/image_44 2024-04-06T10:06:46.763 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T10:06:46.765 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror image_44 2024-04-06T10:06:46.765 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:06:46.765 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:06:46.765 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_44 2024-04-06T10:06:46.765 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror image_44 '' 2024-04-06T10:06:46.765 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:06:46.765 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:06:46.765 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_44 2024-04-06T10:06:46.765 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T10:06:46.766 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-image_44.status 2024-04-06T10:06:46.766 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T10:06:46.766 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-image_44.status 2024-04-06T10:06:46.767 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-image_44.status 2024-04-06T10:06:46.767 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_44 2024-04-06T10:06:46.767 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_44.status 2024-04-06T10:06:46.817 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 0 2024-04-06T10:06:46.817 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 0 2024-04-06T10:06:46.818 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T10:06:46.818 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=1, tag_tid=3, entry_tid=1], [object_number=0, tag_tid=3, entry_tid=0], [object_number=3, tag_tid=1, entry_tid=2567], [object_number=2, tag_tid=1, entry_tid=2566]]], state=connected] 2024-04-06T10:06:46.818 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=1, tag_tid=2, entry_tid=1], [object_number=0, tag_tid=2, entry_tid=0], [object_number=3, tag_tid=1, entry_tid=2567], [object_number=2, tag_tid=1, entry_tid=2566]]], state=connected] 2024-04-06T10:06:46.821 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_44.status 2024-04-06T10:06:46.822 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=1, tag_tid=3, entry_tid=1' 2024-04-06T10:06:46.823 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror image_44 2024-04-06T10:06:46.823 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:06:46.823 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:06:46.824 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_44 2024-04-06T10:06:46.824 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror image_44 '..*' 2024-04-06T10:06:46.824 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:06:46.824 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:06:46.824 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_44 2024-04-06T10:06:46.824 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T10:06:46.824 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-image_44.status 2024-04-06T10:06:46.824 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-image_44.status 2024-04-06T10:06:46.824 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T10:06:46.825 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-image_44.status 2024-04-06T10:06:46.825 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_44 2024-04-06T10:06:46.825 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_44.status 2024-04-06T10:06:46.870 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 0 2024-04-06T10:06:46.870 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 0 2024-04-06T10:06:46.870 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T10:06:46.870 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=1, tag_tid=3, entry_tid=1], [object_number=0, tag_tid=3, entry_tid=0], [object_number=3, tag_tid=1, entry_tid=2567], [object_number=2, tag_tid=1, entry_tid=2566]]], state=connected] 2024-04-06T10:06:46.870 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=1, tag_tid=2, entry_tid=1], [object_number=0, tag_tid=2, entry_tid=0], [object_number=3, tag_tid=1, entry_tid=2567], [object_number=2, tag_tid=1, entry_tid=2566]]], state=connected] 2024-04-06T10:06:46.874 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_44.status 2024-04-06T10:06:46.875 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=1, tag_tid=2, entry_tid=1' 2024-04-06T10:06:46.876 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=1, tag_tid=3, entry_tid=1' -a 'object_number=1, tag_tid=3, entry_tid=1' = 'object_number=1, tag_tid=2, entry_tid=1' 2024-04-06T10:06:46.876 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=1, tag_tid=2, entry_tid=1' '!=' '' 2024-04-06T10:06:46.876 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T10:06:46.876 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=1, tag_tid=2, entry_tid=1' = '' 2024-04-06T10:06:46.876 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=1, tag_tid=2, entry_tid=1' 2024-04-06T10:06:46.876 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=1, tag_tid=3, entry_tid=1' 2024-04-06T10:06:46.877 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T10:06:46.877 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T10:06:46.877 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=3 2024-04-06T10:06:46.878 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=1, tag_tid=2, entry_tid=1' 2024-04-06T10:06:46.878 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T10:06:46.878 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T10:06:46.879 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=2 2024-04-06T10:06:46.879 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=1, tag_tid=3, entry_tid=1' 2024-04-06T10:06:46.880 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T10:06:46.880 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T10:06:46.881 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T10:06:46.881 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=1, tag_tid=2, entry_tid=1' 2024-04-06T10:06:46.881 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T10:06:46.881 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T10:06:46.883 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=1 2024-04-06T10:06:46.883 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 3 = 2 -a 1 -le 1 2024-04-06T10:06:46.883 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T10:06:46.883 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T10:06:46.883 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T10:06:47.085 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror image_44 2024-04-06T10:06:47.085 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T10:06:47.085 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:06:47.085 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_44 2024-04-06T10:06:47.085 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T10:06:47.085 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n image_44 ']' 2024-04-06T10:06:47.085 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/image_44' 2024-04-06T10:06:47.085 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_44 2024-04-06T10:06:47.085 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T10:06:47.085 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T10:06:47.085 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T10:06:47.085 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T10:06:47.085 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T10:06:47.086 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T10:06:47.086 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T10:06:47.086 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T10:06:47.086 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T10:06:47.088 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T10:06:47.088 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_44 2024-04-06T10:06:47.088 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T10:06:47.088 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T10:06:47.088 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T10:06:47.088 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T10:06:47.088 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T10:06:47.088 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T10:06:47.089 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T10:06:47.089 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T10:06:47.089 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T10:06:47.089 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T10:06:47.089 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T10:06:47.089 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:06:47.089 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T10:06:47.089 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T10:06:47.090 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T10:06:47.090 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T10:06:47.090 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T10:06:47.091 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T10:06:47.091 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T10:06:47.091 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T10:06:47.091 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T10:06:47.091 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T10:06:47.091 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:06:47.091 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T10:06:47.091 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T10:06:47.091 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T10:06:47.091 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T10:06:47.091 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T10:06:47.091 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T10:06:47.091 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T10:06:47.116 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T10:06:47.117 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T10:06:47.117 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T10:06:47.117 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_44 2024-04-06T10:06:47.224 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:Invalid command: unused arguments: ['mirror/image_44'] 2024-04-06T10:06:47.225 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:rbd mirror flush : flush rbd mirror 2024-04-06T10:06:47.225 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:admin_socket: invalid command 2024-04-06T10:06:47.227 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T10:06:47.227 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/image_44 2024-04-06T10:06:47.227 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:1 2024-04-06T10:06:47.227 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T10:06:47.227 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T10:06:47.227 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-06T10:06:47.227 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:1 2024-04-06T10:06:47.227 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T10:06:47.227 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T10:06:47.228 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T10:06:47.228 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=1 2024-04-06T10:06:47.228 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-06T10:06:47.228 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T10:06:47.228 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:06:47.228 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=1 2024-04-06T10:06:47.228 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=1 2024-04-06T10:06:47.229 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-06T10:06:47.229 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:1 2024-04-06T10:06:47.229 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T10:06:47.230 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T10:06:47.230 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-06T10:06:47.230 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:1 2024-04-06T10:06:47.230 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T10:06:47.230 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T10:06:47.230 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:06:47.230 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=1 2024-04-06T10:06:47.230 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-06T10:06:47.230 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T10:06:47.230 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T10:06:47.230 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=1 2024-04-06T10:06:47.230 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=1 2024-04-06T10:06:47.230 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-06T10:06:47.254 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T10:06:47.255 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T10:06:47.255 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T10:06:47.255 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/image_44 2024-04-06T10:06:47.393 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T10:06:47.394 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror image_44 2024-04-06T10:06:47.394 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:06:47.394 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:06:47.395 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_44 2024-04-06T10:06:47.395 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror image_44 '' 2024-04-06T10:06:47.395 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:06:47.395 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:06:47.395 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_44 2024-04-06T10:06:47.395 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T10:06:47.395 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-image_44.status 2024-04-06T10:06:47.395 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-image_44.status 2024-04-06T10:06:47.396 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T10:06:47.398 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-image_44.status 2024-04-06T10:06:47.398 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_44.status 2024-04-06T10:06:47.398 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_44 2024-04-06T10:06:47.450 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 0 2024-04-06T10:06:47.450 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 0 2024-04-06T10:06:47.450 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T10:06:47.450 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=1, tag_tid=3, entry_tid=1], [object_number=0, tag_tid=3, entry_tid=0], [object_number=3, tag_tid=1, entry_tid=2567], [object_number=2, tag_tid=1, entry_tid=2566]]], state=connected] 2024-04-06T10:06:47.450 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=1, tag_tid=2, entry_tid=1], [object_number=0, tag_tid=2, entry_tid=0], [object_number=3, tag_tid=1, entry_tid=2567], [object_number=2, tag_tid=1, entry_tid=2566]]], state=connected] 2024-04-06T10:06:47.453 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_44.status 2024-04-06T10:06:47.455 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=1, tag_tid=3, entry_tid=1' 2024-04-06T10:06:47.456 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror image_44 2024-04-06T10:06:47.456 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:06:47.456 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:06:47.456 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_44 2024-04-06T10:06:47.456 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror image_44 '..*' 2024-04-06T10:06:47.456 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:06:47.456 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:06:47.456 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_44 2024-04-06T10:06:47.456 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T10:06:47.456 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-image_44.status 2024-04-06T10:06:47.457 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-image_44.status 2024-04-06T10:06:47.457 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T10:06:47.457 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-image_44.status 2024-04-06T10:06:47.457 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_44 2024-04-06T10:06:47.458 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_44.status 2024-04-06T10:06:47.504 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 0 2024-04-06T10:06:47.504 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 0 2024-04-06T10:06:47.504 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T10:06:47.504 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=1, tag_tid=3, entry_tid=1], [object_number=0, tag_tid=3, entry_tid=0], [object_number=3, tag_tid=1, entry_tid=2567], [object_number=2, tag_tid=1, entry_tid=2566]]], state=connected] 2024-04-06T10:06:47.504 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=1, tag_tid=2, entry_tid=1], [object_number=0, tag_tid=2, entry_tid=0], [object_number=3, tag_tid=1, entry_tid=2567], [object_number=2, tag_tid=1, entry_tid=2566]]], state=connected] 2024-04-06T10:06:47.507 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_44.status 2024-04-06T10:06:47.509 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=1, tag_tid=2, entry_tid=1' 2024-04-06T10:06:47.509 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=1, tag_tid=3, entry_tid=1' -a 'object_number=1, tag_tid=3, entry_tid=1' = 'object_number=1, tag_tid=2, entry_tid=1' 2024-04-06T10:06:47.509 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=1, tag_tid=2, entry_tid=1' '!=' 'object_number=1, tag_tid=2, entry_tid=1' 2024-04-06T10:06:47.509 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T10:06:47.510 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.4 2024-04-06T10:06:47.675 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.0 with skip (0.2711821520412999) > weight (0.25) 2024-04-06T10:06:47.675 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.1 with skip (0.6424431895787304) > weight (0.25) 2024-04-06T10:06:47.675 INFO:tasks.rbd_mirror_thrash:kill cluster1.client.mirror.2 2024-04-06T10:06:47.675 INFO:tasks.rbd_mirror.cluster1.client.mirror.2:Sent signal 15 2024-04-06T10:06:47.675 INFO:tasks.rbd_mirror_thrash:waiting for 4 secs before reviving daemons 2024-04-06T10:06:47.677 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi192.stderr:2024-04-06T10:06:47.675+0000 7fcea3cb2640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.2 (PID: 153423) UID: 1000 2024-04-06T10:06:47.911 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror image_44 2024-04-06T10:06:47.911 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T10:06:47.911 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:06:47.911 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_44 2024-04-06T10:06:47.911 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T10:06:47.911 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n image_44 ']' 2024-04-06T10:06:47.911 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/image_44' 2024-04-06T10:06:47.911 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_44 2024-04-06T10:06:47.912 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T10:06:47.912 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T10:06:47.912 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T10:06:47.912 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T10:06:47.912 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T10:06:47.912 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T10:06:47.912 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T10:06:47.912 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T10:06:47.913 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T10:06:47.914 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T10:06:47.914 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_44 2024-04-06T10:06:47.914 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T10:06:47.914 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T10:06:47.914 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T10:06:47.914 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T10:06:47.914 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T10:06:47.914 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T10:06:47.914 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T10:06:47.914 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T10:06:47.914 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T10:06:47.915 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T10:06:47.915 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T10:06:47.915 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:06:47.915 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T10:06:47.915 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T10:06:47.916 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T10:06:47.916 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T10:06:47.916 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T10:06:47.916 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T10:06:47.917 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T10:06:47.917 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T10:06:47.917 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T10:06:47.917 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T10:06:47.917 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:06:47.917 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T10:06:47.917 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T10:06:47.917 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T10:06:47.917 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T10:06:47.917 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T10:06:47.917 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T10:06:47.917 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T10:06:47.941 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T10:06:47.942 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T10:06:47.942 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T10:06:47.942 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_44 2024-04-06T10:06:48.047 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:Invalid command: unused arguments: ['mirror/image_44'] 2024-04-06T10:06:48.048 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:rbd mirror flush : flush rbd mirror 2024-04-06T10:06:48.048 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:admin_socket: invalid command 2024-04-06T10:06:48.050 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T10:06:48.050 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/image_44 2024-04-06T10:06:48.050 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:1 2024-04-06T10:06:48.050 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T10:06:48.050 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T10:06:48.050 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-06T10:06:48.050 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:1 2024-04-06T10:06:48.050 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T10:06:48.050 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T10:06:48.050 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T10:06:48.051 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=1 2024-04-06T10:06:48.051 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-06T10:06:48.051 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T10:06:48.051 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:06:48.051 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=1 2024-04-06T10:06:48.051 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=1 2024-04-06T10:06:48.053 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-06T10:06:48.053 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:1 2024-04-06T10:06:48.053 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T10:06:48.053 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T10:06:48.053 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-06T10:06:48.053 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:1 2024-04-06T10:06:48.053 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T10:06:48.053 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T10:06:48.053 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:06:48.053 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=1 2024-04-06T10:06:48.053 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-06T10:06:48.053 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T10:06:48.053 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T10:06:48.053 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=1 2024-04-06T10:06:48.054 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=1 2024-04-06T10:06:48.054 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-06T10:06:48.076 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T10:06:48.076 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T10:06:48.076 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T10:06:48.076 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/image_44 2024-04-06T10:06:48.226 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T10:06:48.227 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror image_44 2024-04-06T10:06:48.227 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:06:48.227 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:06:48.227 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_44 2024-04-06T10:06:48.227 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror image_44 '' 2024-04-06T10:06:48.227 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:06:48.227 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:06:48.227 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_44 2024-04-06T10:06:48.228 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T10:06:48.228 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-image_44.status 2024-04-06T10:06:48.228 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-image_44.status 2024-04-06T10:06:48.228 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T10:06:48.230 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-image_44.status 2024-04-06T10:06:48.230 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_44.status 2024-04-06T10:06:48.230 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_44 2024-04-06T10:06:48.276 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 0 2024-04-06T10:06:48.276 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 0 2024-04-06T10:06:48.276 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T10:06:48.276 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=1, tag_tid=3, entry_tid=1], [object_number=0, tag_tid=3, entry_tid=0], [object_number=3, tag_tid=1, entry_tid=2567], [object_number=2, tag_tid=1, entry_tid=2566]]], state=connected] 2024-04-06T10:06:48.276 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=1, tag_tid=2, entry_tid=1], [object_number=0, tag_tid=2, entry_tid=0], [object_number=3, tag_tid=1, entry_tid=2567], [object_number=2, tag_tid=1, entry_tid=2566]]], state=connected] 2024-04-06T10:06:48.280 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_44.status 2024-04-06T10:06:48.281 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=1, tag_tid=3, entry_tid=1' 2024-04-06T10:06:48.282 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror image_44 2024-04-06T10:06:48.282 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:06:48.282 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:06:48.282 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_44 2024-04-06T10:06:48.282 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror image_44 '..*' 2024-04-06T10:06:48.283 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:06:48.283 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:06:48.283 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_44 2024-04-06T10:06:48.283 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T10:06:48.283 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-image_44.status 2024-04-06T10:06:48.283 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-image_44.status 2024-04-06T10:06:48.283 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T10:06:48.283 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-image_44.status 2024-04-06T10:06:48.284 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_44 2024-04-06T10:06:48.284 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_44.status 2024-04-06T10:06:48.328 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 0 2024-04-06T10:06:48.328 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 0 2024-04-06T10:06:48.328 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T10:06:48.328 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=1, tag_tid=3, entry_tid=1], [object_number=0, tag_tid=3, entry_tid=0], [object_number=3, tag_tid=1, entry_tid=2567], [object_number=2, tag_tid=1, entry_tid=2566]]], state=connected] 2024-04-06T10:06:48.328 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=1, tag_tid=2, entry_tid=1], [object_number=0, tag_tid=2, entry_tid=0], [object_number=3, tag_tid=1, entry_tid=2567], [object_number=2, tag_tid=1, entry_tid=2566]]], state=connected] 2024-04-06T10:06:48.331 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_44.status 2024-04-06T10:06:48.333 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=1, tag_tid=2, entry_tid=1' 2024-04-06T10:06:48.333 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=1, tag_tid=3, entry_tid=1' -a 'object_number=1, tag_tid=3, entry_tid=1' = 'object_number=1, tag_tid=2, entry_tid=1' 2024-04-06T10:06:48.333 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=1, tag_tid=2, entry_tid=1' '!=' 'object_number=1, tag_tid=2, entry_tid=1' 2024-04-06T10:06:48.333 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T10:06:48.333 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.8 2024-04-06T10:06:49.135 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror image_44 2024-04-06T10:06:49.135 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T10:06:49.136 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:06:49.136 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_44 2024-04-06T10:06:49.136 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T10:06:49.136 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n image_44 ']' 2024-04-06T10:06:49.136 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/image_44' 2024-04-06T10:06:49.136 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_44 2024-04-06T10:06:49.136 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T10:06:49.136 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T10:06:49.136 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T10:06:49.136 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T10:06:49.136 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T10:06:49.136 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T10:06:49.136 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T10:06:49.137 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T10:06:49.137 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T10:06:49.139 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T10:06:49.139 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_44 2024-04-06T10:06:49.139 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T10:06:49.139 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T10:06:49.139 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T10:06:49.140 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T10:06:49.140 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T10:06:49.140 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T10:06:49.140 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T10:06:49.140 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T10:06:49.140 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T10:06:49.140 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T10:06:49.140 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T10:06:49.140 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:06:49.140 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T10:06:49.141 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T10:06:49.141 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T10:06:49.141 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T10:06:49.141 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T10:06:49.141 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T10:06:49.141 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T10:06:49.141 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T10:06:49.142 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T10:06:49.142 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T10:06:49.142 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:06:49.142 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T10:06:49.142 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T10:06:49.142 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T10:06:49.142 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T10:06:49.142 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T10:06:49.142 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T10:06:49.142 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T10:06:49.167 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T10:06:49.168 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T10:06:49.168 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T10:06:49.168 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_44 2024-04-06T10:06:49.273 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:Invalid command: unused arguments: ['mirror/image_44'] 2024-04-06T10:06:49.273 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:rbd mirror flush : flush rbd mirror 2024-04-06T10:06:49.273 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:admin_socket: invalid command 2024-04-06T10:06:49.276 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T10:06:49.276 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/image_44 2024-04-06T10:06:49.276 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:1 2024-04-06T10:06:49.276 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T10:06:49.276 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T10:06:49.276 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-06T10:06:49.276 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:1 2024-04-06T10:06:49.276 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T10:06:49.276 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T10:06:49.276 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T10:06:49.276 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=1 2024-04-06T10:06:49.277 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-06T10:06:49.277 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T10:06:49.277 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:06:49.277 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=1 2024-04-06T10:06:49.277 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=1 2024-04-06T10:06:49.278 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-06T10:06:49.279 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:1 2024-04-06T10:06:49.279 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T10:06:49.279 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T10:06:49.279 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-06T10:06:49.279 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:1 2024-04-06T10:06:49.279 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T10:06:49.279 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T10:06:49.279 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:06:49.279 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=1 2024-04-06T10:06:49.279 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-06T10:06:49.279 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T10:06:49.279 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T10:06:49.279 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=1 2024-04-06T10:06:49.279 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=1 2024-04-06T10:06:49.280 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-06T10:06:49.301 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T10:06:49.302 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T10:06:49.302 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T10:06:49.302 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/image_44 2024-04-06T10:06:49.440 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T10:06:49.441 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror image_44 2024-04-06T10:06:49.441 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:06:49.441 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:06:49.441 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_44 2024-04-06T10:06:49.441 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror image_44 '' 2024-04-06T10:06:49.441 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:06:49.441 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:06:49.441 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_44 2024-04-06T10:06:49.441 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T10:06:49.441 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-image_44.status 2024-04-06T10:06:49.442 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-image_44.status 2024-04-06T10:06:49.442 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T10:06:49.445 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-image_44.status 2024-04-06T10:06:49.445 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_44 2024-04-06T10:06:49.445 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_44.status 2024-04-06T10:06:49.497 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 0 2024-04-06T10:06:49.498 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 0 2024-04-06T10:06:49.498 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T10:06:49.498 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=1, tag_tid=3, entry_tid=1], [object_number=0, tag_tid=3, entry_tid=0], [object_number=3, tag_tid=1, entry_tid=2567], [object_number=2, tag_tid=1, entry_tid=2566]]], state=connected] 2024-04-06T10:06:49.498 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=1, tag_tid=2, entry_tid=1], [object_number=0, tag_tid=2, entry_tid=0], [object_number=3, tag_tid=1, entry_tid=2567], [object_number=2, tag_tid=1, entry_tid=2566]]], state=connected] 2024-04-06T10:06:49.500 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_44.status 2024-04-06T10:06:49.501 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=1, tag_tid=3, entry_tid=1' 2024-04-06T10:06:49.502 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror image_44 2024-04-06T10:06:49.502 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:06:49.502 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:06:49.502 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_44 2024-04-06T10:06:49.503 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror image_44 '..*' 2024-04-06T10:06:49.503 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:06:49.503 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:06:49.503 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_44 2024-04-06T10:06:49.503 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T10:06:49.503 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-image_44.status 2024-04-06T10:06:49.503 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-image_44.status 2024-04-06T10:06:49.503 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T10:06:49.504 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-image_44.status 2024-04-06T10:06:49.504 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_44 2024-04-06T10:06:49.504 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_44.status 2024-04-06T10:06:49.964 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 0 2024-04-06T10:06:49.964 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 0 2024-04-06T10:06:49.964 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T10:06:49.964 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=1, tag_tid=3, entry_tid=1], [object_number=0, tag_tid=3, entry_tid=0], [object_number=3, tag_tid=1, entry_tid=2567], [object_number=2, tag_tid=1, entry_tid=2566]]], state=connected] 2024-04-06T10:06:49.965 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=1, tag_tid=2, entry_tid=1], [object_number=0, tag_tid=2, entry_tid=0], [object_number=3, tag_tid=1, entry_tid=2567], [object_number=2, tag_tid=1, entry_tid=2566]]], state=connected] 2024-04-06T10:06:49.970 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_44.status 2024-04-06T10:06:49.973 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=1, tag_tid=2, entry_tid=1' 2024-04-06T10:06:49.973 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=1, tag_tid=3, entry_tid=1' -a 'object_number=1, tag_tid=3, entry_tid=1' = 'object_number=1, tag_tid=2, entry_tid=1' 2024-04-06T10:06:49.973 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=1, tag_tid=2, entry_tid=1' '!=' 'object_number=1, tag_tid=2, entry_tid=1' 2024-04-06T10:06:49.973 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T10:06:49.973 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 1.6 2024-04-06T10:06:51.575 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror image_44 2024-04-06T10:06:51.575 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T10:06:51.576 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:06:51.576 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_44 2024-04-06T10:06:51.576 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T10:06:51.576 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n image_44 ']' 2024-04-06T10:06:51.576 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/image_44' 2024-04-06T10:06:51.576 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_44 2024-04-06T10:06:51.576 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T10:06:51.576 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T10:06:51.576 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T10:06:51.576 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T10:06:51.576 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T10:06:51.576 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T10:06:51.576 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T10:06:51.577 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T10:06:51.577 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T10:06:51.580 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T10:06:51.580 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_44 2024-04-06T10:06:51.580 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T10:06:51.580 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T10:06:51.580 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T10:06:51.580 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T10:06:51.580 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T10:06:51.580 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T10:06:51.580 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T10:06:51.581 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T10:06:51.581 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T10:06:51.581 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T10:06:51.581 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T10:06:51.581 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:06:51.581 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T10:06:51.581 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T10:06:51.581 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T10:06:51.581 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T10:06:51.581 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T10:06:51.581 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T10:06:51.581 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T10:06:51.581 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T10:06:51.581 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T10:06:51.581 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T10:06:51.582 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:06:51.582 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T10:06:51.582 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T10:06:51.582 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T10:06:51.582 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T10:06:51.582 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T10:06:51.582 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T10:06:51.582 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T10:06:51.609 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T10:06:51.609 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T10:06:51.609 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T10:06:51.609 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_44 2024-04-06T10:06:51.676 INFO:tasks.rbd_mirror_thrash:waiting for cluster1.client.mirror.2 2024-04-06T10:06:51.677 INFO:teuthology.orchestra.run:waiting for 600 2024-04-06T10:06:51.677 INFO:tasks.rbd_mirror_thrash:reviving cluster1.client.mirror.2 2024-04-06T10:06:51.677 INFO:tasks.rbd_mirror.cluster1.client.mirror.2:Restarting daemon 2024-04-06T10:06:51.678 DEBUG:teuthology.orchestra.run.smithi192:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.2 2024-04-06T10:06:51.680 INFO:tasks.rbd_mirror.cluster1.client.mirror.2:Started 2024-04-06T10:06:51.680 INFO:tasks.rbd_mirror_thrash:waiting for 84 secs before thrashing 2024-04-06T10:06:51.715 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:Invalid command: unused arguments: ['mirror/image_44'] 2024-04-06T10:06:51.715 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:rbd mirror flush : flush rbd mirror 2024-04-06T10:06:51.716 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:admin_socket: invalid command 2024-04-06T10:06:51.719 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T10:06:51.719 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/image_44 2024-04-06T10:06:51.719 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:1 2024-04-06T10:06:51.719 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T10:06:51.719 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T10:06:51.720 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-06T10:06:51.720 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:1 2024-04-06T10:06:51.720 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T10:06:51.720 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T10:06:51.720 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T10:06:51.720 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=1 2024-04-06T10:06:51.720 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-06T10:06:51.720 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T10:06:51.720 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:06:51.720 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=1 2024-04-06T10:06:51.720 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=1 2024-04-06T10:06:51.720 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-06T10:06:51.720 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:1 2024-04-06T10:06:51.720 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T10:06:51.721 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T10:06:51.721 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-06T10:06:51.721 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:1 2024-04-06T10:06:51.721 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T10:06:51.721 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T10:06:51.721 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:06:51.721 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=1 2024-04-06T10:06:51.721 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-06T10:06:51.721 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T10:06:51.721 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T10:06:51.721 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=1 2024-04-06T10:06:51.721 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=1 2024-04-06T10:06:51.721 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-06T10:06:51.745 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T10:06:51.746 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T10:06:51.746 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T10:06:51.746 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/image_44 2024-04-06T10:06:51.891 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T10:06:51.892 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror image_44 2024-04-06T10:06:51.892 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:06:51.893 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:06:51.893 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_44 2024-04-06T10:06:51.893 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror image_44 '' 2024-04-06T10:06:51.893 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:06:51.893 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:06:51.893 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_44 2024-04-06T10:06:51.893 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T10:06:51.893 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-image_44.status 2024-04-06T10:06:51.893 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T10:06:51.893 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-image_44.status 2024-04-06T10:06:51.894 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-image_44.status 2024-04-06T10:06:51.894 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_44 2024-04-06T10:06:51.894 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_44.status 2024-04-06T10:06:51.942 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 0 2024-04-06T10:06:51.942 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 0 2024-04-06T10:06:51.942 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T10:06:51.942 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=1, tag_tid=3, entry_tid=1], [object_number=0, tag_tid=3, entry_tid=0], [object_number=3, tag_tid=1, entry_tid=2567], [object_number=2, tag_tid=1, entry_tid=2566]]], state=connected] 2024-04-06T10:06:51.942 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=1, tag_tid=3, entry_tid=1], [object_number=0, tag_tid=3, entry_tid=0], [object_number=3, tag_tid=1, entry_tid=2567], [object_number=2, tag_tid=1, entry_tid=2566]]], state=connected] 2024-04-06T10:06:51.946 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_44.status 2024-04-06T10:06:51.947 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=1, tag_tid=3, entry_tid=1' 2024-04-06T10:06:51.948 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror image_44 2024-04-06T10:06:51.948 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:06:51.948 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:06:51.949 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_44 2024-04-06T10:06:51.949 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror image_44 '..*' 2024-04-06T10:06:51.949 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:06:51.949 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:06:51.949 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_44 2024-04-06T10:06:51.949 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T10:06:51.949 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-image_44.status 2024-04-06T10:06:51.949 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-image_44.status 2024-04-06T10:06:51.949 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T10:06:51.950 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-image_44.status 2024-04-06T10:06:51.950 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_44 2024-04-06T10:06:51.950 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_44.status 2024-04-06T10:06:51.996 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 0 2024-04-06T10:06:51.997 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 0 2024-04-06T10:06:51.997 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T10:06:51.997 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=1, tag_tid=3, entry_tid=1], [object_number=0, tag_tid=3, entry_tid=0], [object_number=3, tag_tid=1, entry_tid=2567], [object_number=2, tag_tid=1, entry_tid=2566]]], state=connected] 2024-04-06T10:06:51.997 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=1, tag_tid=3, entry_tid=1], [object_number=0, tag_tid=3, entry_tid=0], [object_number=3, tag_tid=1, entry_tid=2567], [object_number=2, tag_tid=1, entry_tid=2566]]], state=connected] 2024-04-06T10:06:52.000 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_44.status 2024-04-06T10:06:52.002 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=1, tag_tid=3, entry_tid=1' 2024-04-06T10:06:52.002 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=1, tag_tid=3, entry_tid=1' -a 'object_number=1, tag_tid=3, entry_tid=1' = 'object_number=1, tag_tid=3, entry_tid=1' 2024-04-06T10:06:52.002 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T10:06:52.002 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_snap_present cluster1 mirror image_44 snap 2024-04-06T10:06:52.003 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T10:06:52.003 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:06:52.003 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_44 2024-04-06T10:06:52.003 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local snap_name=snap 2024-04-06T10:06:52.003 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local s 2024-04-06T10:06:52.003 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-04-06T10:06:52.003 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 1 2024-04-06T10:06:53.003 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 info mirror/image_44@snap 2024-04-06T10:06:53.073 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:rbd image 'image_44': 2024-04-06T10:06:53.073 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: size 128 MiB in 32 objects 2024-04-06T10:06:53.073 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: order 22 (4 MiB objects) 2024-04-06T10:06:53.073 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: snapshot_count: 1 2024-04-06T10:06:53.073 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: id: 10ac3fb29d1d 2024-04-06T10:06:53.073 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: block_name_prefix: rbd_data.10ac3fb29d1d 2024-04-06T10:06:53.073 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: format: 2 2024-04-06T10:06:53.073 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: features: layering, exclusive-lock, journaling 2024-04-06T10:06:53.074 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: op_features: 2024-04-06T10:06:53.074 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: flags: 2024-04-06T10:06:53.074 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: create_timestamp: Sat Apr 6 09:57:22 2024 2024-04-06T10:06:53.074 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: access_timestamp: Sat Apr 6 09:57:22 2024 2024-04-06T10:06:53.074 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: modify_timestamp: Sat Apr 6 09:58:23 2024 2024-04-06T10:06:53.074 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: protected: False 2024-04-06T10:06:53.074 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: journal: 10ac3fb29d1d 2024-04-06T10:06:53.074 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: mirroring state: enabled 2024-04-06T10:06:53.074 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: mirroring mode: journal 2024-04-06T10:06:53.074 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: mirroring global id: 36afb126-a79a-44a9-b89a-b434df912082 2024-04-06T10:06:53.074 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: mirroring primary: false 2024-04-06T10:06:53.078 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T10:06:53.078 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ compare_image_snaps mirror image_44 snap 2024-04-06T10:06:53.078 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:06:53.078 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_44 2024-04-06T10:06:53.079 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local snap_name=snap 2024-04-06T10:06:53.079 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local ret=0 2024-04-06T10:06:53.079 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local rmt_export=/tmp/tmp.BhW6xKmROU/cluster2-mirror-image_44.export 2024-04-06T10:06:53.079 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loc_export=/tmp/tmp.BhW6xKmROU/cluster1-mirror-image_44.export 2024-04-06T10:06:53.079 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rm -f /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_44.export /tmp/tmp.BhW6xKmROU/cluster1-mirror-image_44.export 2024-04-06T10:06:53.079 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster2 -p mirror export image_44@snap /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_44.export 2024-04-06T10:06:53.237 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-04-06T10:06:53.245 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 -p mirror export image_44@snap /tmp/tmp.BhW6xKmROU/cluster1-mirror-image_44.export 2024-04-06T10:06:53.432 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-04-06T10:06:53.440 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmp /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_44.export /tmp/tmp.BhW6xKmROU/cluster1-mirror-image_44.export 2024-04-06T10:06:53.517 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rm -f /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_44.export /tmp/tmp.BhW6xKmROU/cluster1-mirror-image_44.export 2024-04-06T10:06:53.553 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T10:06:53.553 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-06T10:06:53.554 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ image=image_45 2024-04-06T10:06:53.554 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ create_snap cluster2 mirror image_45 snap 2024-04-06T10:06:53.554 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-04-06T10:06:53.554 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:06:53.554 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_45 2024-04-06T10:06:53.555 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local snap_name=snap 2024-04-06T10:06:53.555 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster2 -p mirror snap create image_45@snap --debug-rbd=20 --debug-journaler=20 2024-04-06T10:06:53.934 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_image_replay_started cluster1 mirror image_45 2024-04-06T10:06:53.934 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T10:06:53.934 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:06:53.934 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_45 2024-04-06T10:06:53.934 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_image_replay_state cluster1 mirror image_45 started 2024-04-06T10:06:53.934 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T10:06:53.934 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:06:53.934 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_45 2024-04-06T10:06:53.935 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state=started 2024-04-06T10:06:53.935 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local s 2024-04-06T10:06:53.935 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T10:06:53.935 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 1 2024-04-06T10:06:54.935 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test_image_replay_state cluster1 mirror image_45 started 2024-04-06T10:06:54.936 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T10:06:54.936 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:06:54.936 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_45 2024-04-06T10:06:54.936 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local test_state=started 2024-04-06T10:06:54.936 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local status_result 2024-04-06T10:06:54.936 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local current_state=stopped 2024-04-06T10:06:54.937 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_45 2024-04-06T10:06:54.937 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_instance=cluster1 2024-04-06T10:06:54.937 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -i state 2024-04-06T10:06:54.937 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ shift 2024-04-06T10:06:54.937 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T10:06:54.937 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance=cluster1 2024-04-06T10:06:54.937 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local loop_instance 2024-04-06T10:06:54.937 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T10:06:54.938 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sleep 0 2024-04-06T10:06:54.939 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-04-06T10:06:54.939 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ seq 0 3 2024-04-06T10:06:54.942 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T10:06:54.942 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_45 2024-04-06T10:06:54.942 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1:0 2024-04-06T10:06:54.942 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ shift 2024-04-06T10:06:54.942 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T10:06:54.942 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-04-06T10:06:54.942 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T10:06:54.942 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=cluster 2024-04-06T10:06:54.942 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T10:06:54.942 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:06:54.942 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T10:06:54.942 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T10:06:54.943 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval cluster=cluster1 2024-04-06T10:06:54.943 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ cluster=cluster1 2024-04-06T10:06:54.943 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T10:06:54.943 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T10:06:54.943 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-04-06T10:06:54.943 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local local_cluster=cluster1:0 2024-04-06T10:06:54.943 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local cluster=cluster1 2024-04-06T10:06:54.943 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local instance 2024-04-06T10:06:54.943 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T10:06:54.943 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local val=cluster1:0 2024-04-06T10:06:54.943 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local cluster_var_name=local_cluster 2024-04-06T10:06:54.943 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local instance_var_name=instance 2024-04-06T10:06:54.943 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ cluster=cluster1 2024-04-06T10:06:54.943 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T10:06:54.943 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ '[' 0 = cluster1:0 ']' 2024-04-06T10:06:54.944 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ eval local_cluster=cluster1 2024-04-06T10:06:54.946 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++++ local_cluster=cluster1 2024-04-06T10:06:54.946 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ eval instance=0 2024-04-06T10:06:54.946 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++++ instance=0 2024-04-06T10:06:54.946 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T10:06:54.965 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T10:06:54.965 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T10:06:54.966 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T10:06:54.966 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_45 2024-04-06T10:06:55.074 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:Invalid command: unused arguments: ['mirror/image_45'] 2024-04-06T10:06:55.074 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:rbd mirror status : get status for rbd mirror 2024-04-06T10:06:55.074 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:admin_socket: invalid command 2024-04-06T10:06:55.077 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T10:06:55.077 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_45 2024-04-06T10:06:55.077 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1:1 2024-04-06T10:06:55.077 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ shift 2024-04-06T10:06:55.077 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T10:06:55.077 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-04-06T10:06:55.077 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:1 2024-04-06T10:06:55.077 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=cluster 2024-04-06T10:06:55.077 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T10:06:55.077 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:06:55.077 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=1 2024-04-06T10:06:55.078 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-06T10:06:55.078 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval cluster=cluster1 2024-04-06T10:06:55.078 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ cluster=cluster1 2024-04-06T10:06:55.078 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=1 2024-04-06T10:06:55.078 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=1 2024-04-06T10:06:55.079 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-04-06T10:06:55.080 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local local_cluster=cluster1:1 2024-04-06T10:06:55.080 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local cluster=cluster1 2024-04-06T10:06:55.080 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local instance 2024-04-06T10:06:55.080 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-06T10:06:55.080 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local val=cluster1:1 2024-04-06T10:06:55.080 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local cluster_var_name=local_cluster 2024-04-06T10:06:55.080 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local instance_var_name=instance 2024-04-06T10:06:55.080 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ cluster=cluster1 2024-04-06T10:06:55.080 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=1 2024-04-06T10:06:55.080 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ '[' 1 = cluster1:1 ']' 2024-04-06T10:06:55.080 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ eval local_cluster=cluster1 2024-04-06T10:06:55.080 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++++ local_cluster=cluster1 2024-04-06T10:06:55.080 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ eval instance=1 2024-04-06T10:06:55.080 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++++ instance=1 2024-04-06T10:06:55.081 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-06T10:06:55.099 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T10:06:55.099 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T10:06:55.100 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T10:06:55.100 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_45 2024-04-06T10:06:55.259 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ return 0 2024-04-06T10:06:55.260 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ status_result=' "state": "Replaying"' 2024-04-06T10:06:55.260 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ echo ' "state": "Replaying"' 2024-04-06T10:06:55.260 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep -i Replaying 2024-04-06T10:06:55.262 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: "state": "Replaying" 2024-04-06T10:06:55.263 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ current_state=started 2024-04-06T10:06:55.263 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test started = started 2024-04-06T10:06:55.263 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T10:06:55.263 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_45 2024-04-06T10:06:55.263 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local local_cluster=cluster1 2024-04-06T10:06:55.263 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-04-06T10:06:55.263 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:06:55.264 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_45 2024-04-06T10:06:55.264 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' journal = journal ']' 2024-04-06T10:06:55.264 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_journal_replay_complete cluster1 cluster2 mirror image_45 2024-04-06T10:06:55.264 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local local_cluster=cluster1 2024-04-06T10:06:55.264 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-04-06T10:06:55.264 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:06:55.264 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_45 2024-04-06T10:06:55.264 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local s master_pos mirror_pos last_mirror_pos 2024-04-06T10:06:55.264 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local master_tag master_entry mirror_tag mirror_entry 2024-04-06T10:06:55.264 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T10:06:55.264 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T10:06:55.264 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T10:06:55.465 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror image_45 2024-04-06T10:06:55.465 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T10:06:55.465 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:06:55.465 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_45 2024-04-06T10:06:55.465 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T10:06:55.466 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n image_45 ']' 2024-04-06T10:06:55.466 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/image_45' 2024-04-06T10:06:55.466 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_45 2024-04-06T10:06:55.466 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T10:06:55.466 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T10:06:55.466 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T10:06:55.466 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T10:06:55.466 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T10:06:55.466 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T10:06:55.466 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T10:06:55.467 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T10:06:55.467 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T10:06:55.470 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T10:06:55.470 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_45 2024-04-06T10:06:55.470 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T10:06:55.470 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T10:06:55.470 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T10:06:55.471 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T10:06:55.471 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T10:06:55.471 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T10:06:55.471 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T10:06:55.471 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T10:06:55.471 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T10:06:55.471 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T10:06:55.471 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T10:06:55.471 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:06:55.471 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T10:06:55.471 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T10:06:55.471 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T10:06:55.471 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T10:06:55.472 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T10:06:55.472 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T10:06:55.472 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T10:06:55.472 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T10:06:55.472 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T10:06:55.472 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T10:06:55.472 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:06:55.472 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T10:06:55.472 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T10:06:55.472 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T10:06:55.472 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T10:06:55.472 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T10:06:55.472 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T10:06:55.473 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T10:06:55.496 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T10:06:55.496 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T10:06:55.497 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T10:06:55.497 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_45 2024-04-06T10:06:55.604 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:Invalid command: unused arguments: ['mirror/image_45'] 2024-04-06T10:06:55.604 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:rbd mirror flush : flush rbd mirror 2024-04-06T10:06:55.604 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:admin_socket: invalid command 2024-04-06T10:06:55.606 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T10:06:55.606 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/image_45 2024-04-06T10:06:55.607 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:1 2024-04-06T10:06:55.607 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T10:06:55.607 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T10:06:55.607 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-06T10:06:55.607 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:1 2024-04-06T10:06:55.607 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T10:06:55.607 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T10:06:55.607 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T10:06:55.607 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=1 2024-04-06T10:06:55.607 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-06T10:06:55.607 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T10:06:55.607 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:06:55.607 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=1 2024-04-06T10:06:55.607 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=1 2024-04-06T10:06:55.609 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-06T10:06:55.609 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:1 2024-04-06T10:06:55.609 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T10:06:55.609 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T10:06:55.609 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-06T10:06:55.609 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:1 2024-04-06T10:06:55.609 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T10:06:55.609 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T10:06:55.609 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:06:55.609 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=1 2024-04-06T10:06:55.609 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-06T10:06:55.609 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T10:06:55.609 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T10:06:55.610 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=1 2024-04-06T10:06:55.610 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=1 2024-04-06T10:06:55.610 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-06T10:06:55.632 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T10:06:55.633 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T10:06:55.633 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T10:06:55.633 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/image_45 2024-04-06T10:06:55.777 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T10:06:55.778 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror image_45 2024-04-06T10:06:55.778 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:06:55.778 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:06:55.778 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_45 2024-04-06T10:06:55.778 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror image_45 '' 2024-04-06T10:06:55.778 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:06:55.778 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:06:55.778 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_45 2024-04-06T10:06:55.778 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T10:06:55.779 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-image_45.status 2024-04-06T10:06:55.779 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-image_45.status 2024-04-06T10:06:55.779 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T10:06:55.782 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-image_45.status 2024-04-06T10:06:55.782 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_45 2024-04-06T10:06:55.782 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_45.status 2024-04-06T10:06:55.839 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 1 2024-04-06T10:06:55.840 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 1 2024-04-06T10:06:55.840 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T10:06:55.840 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=5, tag_tid=3, entry_tid=1], [object_number=4, tag_tid=3, entry_tid=0], [object_number=6, tag_tid=1, entry_tid=5450], [object_number=7, tag_tid=1, entry_tid=5447]]], state=connected] 2024-04-06T10:06:55.840 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=5, tag_tid=2, entry_tid=1], [object_number=4, tag_tid=2, entry_tid=0], [object_number=6, tag_tid=1, entry_tid=5450], [object_number=7, tag_tid=1, entry_tid=5447]]], state=connected] 2024-04-06T10:06:55.843 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_45.status 2024-04-06T10:06:55.845 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=5, tag_tid=3, entry_tid=1' 2024-04-06T10:06:55.846 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror image_45 2024-04-06T10:06:55.846 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:06:55.846 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:06:55.846 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_45 2024-04-06T10:06:55.846 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror image_45 '..*' 2024-04-06T10:06:55.846 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:06:55.846 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:06:55.847 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_45 2024-04-06T10:06:55.847 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T10:06:55.847 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-image_45.status 2024-04-06T10:06:55.847 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-image_45.status 2024-04-06T10:06:55.847 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T10:06:55.847 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-image_45.status 2024-04-06T10:06:55.847 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_45 2024-04-06T10:06:55.848 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_45.status 2024-04-06T10:06:55.893 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 1 2024-04-06T10:06:55.893 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 1 2024-04-06T10:06:55.893 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T10:06:55.893 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=5, tag_tid=3, entry_tid=1], [object_number=4, tag_tid=3, entry_tid=0], [object_number=6, tag_tid=1, entry_tid=5450], [object_number=7, tag_tid=1, entry_tid=5447]]], state=connected] 2024-04-06T10:06:55.894 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=5, tag_tid=2, entry_tid=1], [object_number=4, tag_tid=2, entry_tid=0], [object_number=6, tag_tid=1, entry_tid=5450], [object_number=7, tag_tid=1, entry_tid=5447]]], state=connected] 2024-04-06T10:06:55.897 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_45.status 2024-04-06T10:06:55.898 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=5, tag_tid=2, entry_tid=1' 2024-04-06T10:06:55.899 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=5, tag_tid=3, entry_tid=1' -a 'object_number=5, tag_tid=3, entry_tid=1' = 'object_number=5, tag_tid=2, entry_tid=1' 2024-04-06T10:06:55.899 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=5, tag_tid=2, entry_tid=1' '!=' '' 2024-04-06T10:06:55.899 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T10:06:55.899 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=5, tag_tid=2, entry_tid=1' = '' 2024-04-06T10:06:55.899 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=5, tag_tid=2, entry_tid=1' 2024-04-06T10:06:55.899 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=5, tag_tid=3, entry_tid=1' 2024-04-06T10:06:55.900 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T10:06:55.900 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T10:06:55.900 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=3 2024-04-06T10:06:55.901 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=5, tag_tid=2, entry_tid=1' 2024-04-06T10:06:55.901 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T10:06:55.901 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T10:06:55.902 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=2 2024-04-06T10:06:55.903 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=5, tag_tid=3, entry_tid=1' 2024-04-06T10:06:55.903 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T10:06:55.903 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T10:06:55.904 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T10:06:55.904 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=5, tag_tid=2, entry_tid=1' 2024-04-06T10:06:55.905 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T10:06:55.905 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T10:06:55.906 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=1 2024-04-06T10:06:55.906 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 3 = 2 -a 1 -le 1 2024-04-06T10:06:55.906 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T10:06:55.906 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T10:06:55.907 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T10:06:56.108 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror image_45 2024-04-06T10:06:56.108 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T10:06:56.108 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:06:56.108 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_45 2024-04-06T10:06:56.108 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T10:06:56.108 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n image_45 ']' 2024-04-06T10:06:56.109 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/image_45' 2024-04-06T10:06:56.109 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_45 2024-04-06T10:06:56.109 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T10:06:56.109 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T10:06:56.109 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T10:06:56.109 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T10:06:56.109 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T10:06:56.109 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T10:06:56.109 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T10:06:56.112 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T10:06:56.112 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T10:06:56.112 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T10:06:56.112 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_45 2024-04-06T10:06:56.112 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T10:06:56.112 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T10:06:56.112 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T10:06:56.112 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T10:06:56.112 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T10:06:56.112 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T10:06:56.112 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T10:06:56.112 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T10:06:56.113 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T10:06:56.113 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T10:06:56.113 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T10:06:56.113 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:06:56.113 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T10:06:56.113 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T10:06:56.113 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T10:06:56.113 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T10:06:56.113 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T10:06:56.113 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T10:06:56.113 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T10:06:56.113 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T10:06:56.113 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T10:06:56.113 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T10:06:56.113 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:06:56.113 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T10:06:56.114 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T10:06:56.114 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T10:06:56.114 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T10:06:56.114 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T10:06:56.114 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T10:06:56.114 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T10:06:56.138 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T10:06:56.138 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T10:06:56.138 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T10:06:56.138 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_45 2024-04-06T10:06:56.246 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:Invalid command: unused arguments: ['mirror/image_45'] 2024-04-06T10:06:56.247 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:rbd mirror flush : flush rbd mirror 2024-04-06T10:06:56.247 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:admin_socket: invalid command 2024-04-06T10:06:56.250 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T10:06:56.251 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/image_45 2024-04-06T10:06:56.251 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:1 2024-04-06T10:06:56.251 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T10:06:56.251 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T10:06:56.251 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-06T10:06:56.251 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:1 2024-04-06T10:06:56.251 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T10:06:56.251 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T10:06:56.251 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T10:06:56.251 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=1 2024-04-06T10:06:56.251 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-06T10:06:56.251 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T10:06:56.252 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:06:56.252 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=1 2024-04-06T10:06:56.252 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=1 2024-04-06T10:06:56.252 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-06T10:06:56.252 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:1 2024-04-06T10:06:56.252 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T10:06:56.252 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T10:06:56.252 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-06T10:06:56.252 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:1 2024-04-06T10:06:56.252 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T10:06:56.252 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T10:06:56.252 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:06:56.252 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=1 2024-04-06T10:06:56.252 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-06T10:06:56.252 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T10:06:56.253 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T10:06:56.253 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=1 2024-04-06T10:06:56.253 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=1 2024-04-06T10:06:56.253 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-06T10:06:56.275 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T10:06:56.275 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T10:06:56.275 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T10:06:56.275 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/image_45 2024-04-06T10:06:56.410 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T10:06:56.411 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror image_45 2024-04-06T10:06:56.411 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:06:56.411 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:06:56.411 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_45 2024-04-06T10:06:56.411 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror image_45 '' 2024-04-06T10:06:56.412 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:06:56.412 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:06:56.412 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_45 2024-04-06T10:06:56.412 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T10:06:56.412 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-image_45.status 2024-04-06T10:06:56.412 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-image_45.status 2024-04-06T10:06:56.412 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T10:06:56.413 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-image_45.status 2024-04-06T10:06:56.414 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_45 2024-04-06T10:06:56.414 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_45.status 2024-04-06T10:06:56.464 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 1 2024-04-06T10:06:56.464 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 1 2024-04-06T10:06:56.464 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T10:06:56.464 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=5, tag_tid=3, entry_tid=1], [object_number=4, tag_tid=3, entry_tid=0], [object_number=6, tag_tid=1, entry_tid=5450], [object_number=7, tag_tid=1, entry_tid=5447]]], state=connected] 2024-04-06T10:06:56.464 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=5, tag_tid=2, entry_tid=1], [object_number=4, tag_tid=2, entry_tid=0], [object_number=6, tag_tid=1, entry_tid=5450], [object_number=7, tag_tid=1, entry_tid=5447]]], state=connected] 2024-04-06T10:06:56.467 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_45.status 2024-04-06T10:06:56.468 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=5, tag_tid=3, entry_tid=1' 2024-04-06T10:06:56.470 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror image_45 2024-04-06T10:06:56.470 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:06:56.470 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:06:56.470 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_45 2024-04-06T10:06:56.470 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror image_45 '..*' 2024-04-06T10:06:56.470 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:06:56.470 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:06:56.470 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_45 2024-04-06T10:06:56.470 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T10:06:56.470 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-image_45.status 2024-04-06T10:06:56.471 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-image_45.status 2024-04-06T10:06:56.471 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T10:06:56.471 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-image_45.status 2024-04-06T10:06:56.471 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_45 2024-04-06T10:06:56.471 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_45.status 2024-04-06T10:06:56.720 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 1 2024-04-06T10:06:56.720 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 1 2024-04-06T10:06:56.720 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T10:06:56.720 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=5, tag_tid=3, entry_tid=1], [object_number=4, tag_tid=3, entry_tid=0], [object_number=6, tag_tid=1, entry_tid=5450], [object_number=7, tag_tid=1, entry_tid=5447]]], state=connected] 2024-04-06T10:06:56.720 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=5, tag_tid=2, entry_tid=1], [object_number=4, tag_tid=2, entry_tid=0], [object_number=6, tag_tid=1, entry_tid=5450], [object_number=7, tag_tid=1, entry_tid=5447]]], state=connected] 2024-04-06T10:06:56.725 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_45.status 2024-04-06T10:06:56.728 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=5, tag_tid=2, entry_tid=1' 2024-04-06T10:06:56.728 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=5, tag_tid=3, entry_tid=1' -a 'object_number=5, tag_tid=3, entry_tid=1' = 'object_number=5, tag_tid=2, entry_tid=1' 2024-04-06T10:06:56.728 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=5, tag_tid=2, entry_tid=1' '!=' 'object_number=5, tag_tid=2, entry_tid=1' 2024-04-06T10:06:56.728 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T10:06:56.728 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.4 2024-04-06T10:06:57.130 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror image_45 2024-04-06T10:06:57.130 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T10:06:57.130 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:06:57.131 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_45 2024-04-06T10:06:57.131 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T10:06:57.131 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n image_45 ']' 2024-04-06T10:06:57.131 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/image_45' 2024-04-06T10:06:57.131 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_45 2024-04-06T10:06:57.131 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T10:06:57.131 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T10:06:57.131 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T10:06:57.131 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T10:06:57.131 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T10:06:57.131 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T10:06:57.131 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T10:06:57.132 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T10:06:57.132 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T10:06:57.136 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T10:06:57.136 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_45 2024-04-06T10:06:57.136 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T10:06:57.136 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T10:06:57.136 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T10:06:57.136 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T10:06:57.136 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T10:06:57.136 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T10:06:57.136 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T10:06:57.136 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T10:06:57.136 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T10:06:57.136 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T10:06:57.136 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T10:06:57.137 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:06:57.137 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T10:06:57.137 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T10:06:57.137 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T10:06:57.137 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T10:06:57.137 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T10:06:57.137 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T10:06:57.137 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T10:06:57.137 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T10:06:57.137 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T10:06:57.137 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T10:06:57.137 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:06:57.137 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T10:06:57.137 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T10:06:57.137 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T10:06:57.138 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T10:06:57.138 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T10:06:57.138 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T10:06:57.138 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T10:06:57.162 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T10:06:57.163 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T10:06:57.163 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T10:06:57.163 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_45 2024-04-06T10:06:57.266 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:Invalid command: unused arguments: ['mirror/image_45'] 2024-04-06T10:06:57.266 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:rbd mirror flush : flush rbd mirror 2024-04-06T10:06:57.266 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:admin_socket: invalid command 2024-04-06T10:06:57.267 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T10:06:57.267 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/image_45 2024-04-06T10:06:57.268 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:1 2024-04-06T10:06:57.268 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T10:06:57.268 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T10:06:57.268 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-06T10:06:57.268 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:1 2024-04-06T10:06:57.268 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T10:06:57.268 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T10:06:57.268 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T10:06:57.268 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=1 2024-04-06T10:06:57.268 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-06T10:06:57.268 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T10:06:57.268 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:06:57.268 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=1 2024-04-06T10:06:57.269 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=1 2024-04-06T10:06:57.270 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-06T10:06:57.270 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:1 2024-04-06T10:06:57.270 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T10:06:57.271 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T10:06:57.271 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-06T10:06:57.271 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:1 2024-04-06T10:06:57.271 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T10:06:57.271 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T10:06:57.271 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:06:57.271 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=1 2024-04-06T10:06:57.271 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-06T10:06:57.271 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T10:06:57.271 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T10:06:57.271 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=1 2024-04-06T10:06:57.271 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=1 2024-04-06T10:06:57.271 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-06T10:06:57.292 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T10:06:57.293 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T10:06:57.293 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T10:06:57.293 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/image_45 2024-04-06T10:06:57.445 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T10:06:57.446 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror image_45 2024-04-06T10:06:57.446 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:06:57.447 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:06:57.447 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_45 2024-04-06T10:06:57.447 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror image_45 '' 2024-04-06T10:06:57.447 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:06:57.447 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:06:57.447 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_45 2024-04-06T10:06:57.447 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T10:06:57.448 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-image_45.status 2024-04-06T10:06:57.448 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-image_45.status 2024-04-06T10:06:57.448 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T10:06:57.450 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-image_45.status 2024-04-06T10:06:57.450 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_45 2024-04-06T10:06:57.450 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_45.status 2024-04-06T10:06:57.502 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 1 2024-04-06T10:06:57.502 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 1 2024-04-06T10:06:57.502 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T10:06:57.502 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=5, tag_tid=3, entry_tid=1], [object_number=4, tag_tid=3, entry_tid=0], [object_number=6, tag_tid=1, entry_tid=5450], [object_number=7, tag_tid=1, entry_tid=5447]]], state=connected] 2024-04-06T10:06:57.502 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=5, tag_tid=2, entry_tid=1], [object_number=4, tag_tid=2, entry_tid=0], [object_number=6, tag_tid=1, entry_tid=5450], [object_number=7, tag_tid=1, entry_tid=5447]]], state=connected] 2024-04-06T10:06:57.505 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_45.status 2024-04-06T10:06:57.507 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=5, tag_tid=3, entry_tid=1' 2024-04-06T10:06:57.508 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror image_45 2024-04-06T10:06:57.508 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:06:57.508 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:06:57.508 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_45 2024-04-06T10:06:57.508 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror image_45 '..*' 2024-04-06T10:06:57.508 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:06:57.508 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:06:57.508 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_45 2024-04-06T10:06:57.509 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T10:06:57.509 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-image_45.status 2024-04-06T10:06:57.509 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-image_45.status 2024-04-06T10:06:57.509 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T10:06:57.509 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-image_45.status 2024-04-06T10:06:57.510 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_45 2024-04-06T10:06:57.510 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_45.status 2024-04-06T10:06:57.555 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 1 2024-04-06T10:06:57.555 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 1 2024-04-06T10:06:57.555 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T10:06:57.555 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=5, tag_tid=3, entry_tid=1], [object_number=4, tag_tid=3, entry_tid=0], [object_number=6, tag_tid=1, entry_tid=5450], [object_number=7, tag_tid=1, entry_tid=5447]]], state=connected] 2024-04-06T10:06:57.555 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=5, tag_tid=2, entry_tid=1], [object_number=4, tag_tid=2, entry_tid=0], [object_number=6, tag_tid=1, entry_tid=5450], [object_number=7, tag_tid=1, entry_tid=5447]]], state=connected] 2024-04-06T10:06:57.558 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_45.status 2024-04-06T10:06:57.560 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=5, tag_tid=2, entry_tid=1' 2024-04-06T10:06:57.560 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=5, tag_tid=3, entry_tid=1' -a 'object_number=5, tag_tid=3, entry_tid=1' = 'object_number=5, tag_tid=2, entry_tid=1' 2024-04-06T10:06:57.560 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=5, tag_tid=2, entry_tid=1' '!=' 'object_number=5, tag_tid=2, entry_tid=1' 2024-04-06T10:06:57.560 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T10:06:57.560 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.8 2024-04-06T10:06:58.362 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror image_45 2024-04-06T10:06:58.362 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T10:06:58.362 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:06:58.363 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_45 2024-04-06T10:06:58.363 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T10:06:58.363 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n image_45 ']' 2024-04-06T10:06:58.363 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/image_45' 2024-04-06T10:06:58.363 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_45 2024-04-06T10:06:58.363 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T10:06:58.363 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T10:06:58.363 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T10:06:58.363 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T10:06:58.363 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T10:06:58.363 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T10:06:58.363 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T10:06:58.364 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T10:06:58.364 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T10:06:58.367 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T10:06:58.367 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_45 2024-04-06T10:06:58.367 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T10:06:58.367 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T10:06:58.367 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T10:06:58.367 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T10:06:58.367 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T10:06:58.367 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T10:06:58.367 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T10:06:58.368 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T10:06:58.368 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T10:06:58.368 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T10:06:58.368 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T10:06:58.368 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:06:58.368 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T10:06:58.368 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T10:06:58.369 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T10:06:58.369 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T10:06:58.369 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T10:06:58.369 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T10:06:58.369 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T10:06:58.369 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T10:06:58.369 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T10:06:58.369 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T10:06:58.369 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:06:58.369 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T10:06:58.369 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T10:06:58.369 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T10:06:58.369 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T10:06:58.370 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T10:06:58.370 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T10:06:58.370 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T10:06:58.393 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T10:06:58.394 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T10:06:58.394 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T10:06:58.394 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_45 2024-04-06T10:06:58.500 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:Invalid command: unused arguments: ['mirror/image_45'] 2024-04-06T10:06:58.500 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:rbd mirror flush : flush rbd mirror 2024-04-06T10:06:58.501 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:admin_socket: invalid command 2024-04-06T10:06:58.503 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T10:06:58.503 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/image_45 2024-04-06T10:06:58.503 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:1 2024-04-06T10:06:58.503 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T10:06:58.503 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T10:06:58.503 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-06T10:06:58.503 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:1 2024-04-06T10:06:58.503 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T10:06:58.503 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T10:06:58.504 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T10:06:58.504 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=1 2024-04-06T10:06:58.504 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-06T10:06:58.504 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T10:06:58.504 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:06:58.504 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=1 2024-04-06T10:06:58.504 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=1 2024-04-06T10:06:58.505 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-06T10:06:58.505 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:1 2024-04-06T10:06:58.505 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T10:06:58.505 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T10:06:58.505 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-06T10:06:58.505 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:1 2024-04-06T10:06:58.506 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T10:06:58.506 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T10:06:58.506 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:06:58.506 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=1 2024-04-06T10:06:58.506 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-06T10:06:58.506 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T10:06:58.506 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T10:06:58.506 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=1 2024-04-06T10:06:58.506 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=1 2024-04-06T10:06:58.506 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-06T10:06:58.525 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T10:06:58.526 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T10:06:58.526 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T10:06:58.526 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/image_45 2024-04-06T10:06:58.664 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T10:06:58.665 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror image_45 2024-04-06T10:06:58.665 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:06:58.665 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:06:58.666 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_45 2024-04-06T10:06:58.666 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror image_45 '' 2024-04-06T10:06:58.666 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:06:58.666 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:06:58.666 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_45 2024-04-06T10:06:58.666 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T10:06:58.666 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-image_45.status 2024-04-06T10:06:58.667 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-image_45.status 2024-04-06T10:06:58.667 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T10:06:58.669 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-image_45.status 2024-04-06T10:06:58.669 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_45 2024-04-06T10:06:58.669 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_45.status 2024-04-06T10:06:58.714 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 1 2024-04-06T10:06:58.714 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 1 2024-04-06T10:06:58.714 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T10:06:58.714 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=5, tag_tid=3, entry_tid=1], [object_number=4, tag_tid=3, entry_tid=0], [object_number=6, tag_tid=1, entry_tid=5450], [object_number=7, tag_tid=1, entry_tid=5447]]], state=connected] 2024-04-06T10:06:58.714 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=5, tag_tid=2, entry_tid=1], [object_number=4, tag_tid=2, entry_tid=0], [object_number=6, tag_tid=1, entry_tid=5450], [object_number=7, tag_tid=1, entry_tid=5447]]], state=connected] 2024-04-06T10:06:58.718 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_45.status 2024-04-06T10:06:58.719 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=5, tag_tid=3, entry_tid=1' 2024-04-06T10:06:58.720 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror image_45 2024-04-06T10:06:58.720 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:06:58.720 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:06:58.720 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_45 2024-04-06T10:06:58.720 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror image_45 '..*' 2024-04-06T10:06:58.721 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:06:58.721 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:06:58.721 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_45 2024-04-06T10:06:58.721 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T10:06:58.721 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-image_45.status 2024-04-06T10:06:58.721 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-image_45.status 2024-04-06T10:06:58.721 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T10:06:58.721 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-image_45.status 2024-04-06T10:06:58.722 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_45 2024-04-06T10:06:58.722 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_45.status 2024-04-06T10:06:58.766 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 1 2024-04-06T10:06:58.766 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 1 2024-04-06T10:06:58.767 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T10:06:58.767 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=5, tag_tid=3, entry_tid=1], [object_number=4, tag_tid=3, entry_tid=0], [object_number=6, tag_tid=1, entry_tid=5450], [object_number=7, tag_tid=1, entry_tid=5447]]], state=connected] 2024-04-06T10:06:58.767 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=5, tag_tid=2, entry_tid=1], [object_number=4, tag_tid=2, entry_tid=0], [object_number=6, tag_tid=1, entry_tid=5450], [object_number=7, tag_tid=1, entry_tid=5447]]], state=connected] 2024-04-06T10:06:58.770 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_45.status 2024-04-06T10:06:58.772 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=5, tag_tid=2, entry_tid=1' 2024-04-06T10:06:58.772 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=5, tag_tid=3, entry_tid=1' -a 'object_number=5, tag_tid=3, entry_tid=1' = 'object_number=5, tag_tid=2, entry_tid=1' 2024-04-06T10:06:58.772 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=5, tag_tid=2, entry_tid=1' '!=' 'object_number=5, tag_tid=2, entry_tid=1' 2024-04-06T10:06:58.772 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T10:06:58.772 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 1.6 2024-04-06T10:07:00.374 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror image_45 2024-04-06T10:07:00.375 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T10:07:00.375 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:07:00.375 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_45 2024-04-06T10:07:00.375 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T10:07:00.375 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n image_45 ']' 2024-04-06T10:07:00.375 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/image_45' 2024-04-06T10:07:00.375 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_45 2024-04-06T10:07:00.375 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T10:07:00.375 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T10:07:00.375 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T10:07:00.375 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T10:07:00.375 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T10:07:00.375 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T10:07:00.376 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T10:07:00.376 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T10:07:00.376 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T10:07:00.379 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T10:07:00.380 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_45 2024-04-06T10:07:00.380 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T10:07:00.380 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T10:07:00.380 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T10:07:00.380 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T10:07:00.380 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T10:07:00.380 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T10:07:00.380 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T10:07:00.380 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T10:07:00.380 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T10:07:00.380 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T10:07:00.381 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T10:07:00.381 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:07:00.381 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T10:07:00.381 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T10:07:00.381 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T10:07:00.381 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T10:07:00.381 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T10:07:00.381 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T10:07:00.381 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T10:07:00.381 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T10:07:00.382 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T10:07:00.382 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T10:07:00.382 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:07:00.382 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T10:07:00.382 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T10:07:00.382 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T10:07:00.383 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T10:07:00.383 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T10:07:00.383 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T10:07:00.383 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T10:07:00.405 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T10:07:00.406 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T10:07:00.406 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T10:07:00.406 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_45 2024-04-06T10:07:00.511 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:Invalid command: unused arguments: ['mirror/image_45'] 2024-04-06T10:07:00.511 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:rbd mirror flush : flush rbd mirror 2024-04-06T10:07:00.511 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:admin_socket: invalid command 2024-04-06T10:07:00.513 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T10:07:00.513 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/image_45 2024-04-06T10:07:00.514 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:1 2024-04-06T10:07:00.514 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T10:07:00.514 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T10:07:00.514 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-06T10:07:00.514 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:1 2024-04-06T10:07:00.514 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T10:07:00.514 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T10:07:00.514 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T10:07:00.514 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=1 2024-04-06T10:07:00.514 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-06T10:07:00.514 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T10:07:00.514 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:07:00.515 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=1 2024-04-06T10:07:00.515 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=1 2024-04-06T10:07:00.515 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-06T10:07:00.515 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:1 2024-04-06T10:07:00.515 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T10:07:00.515 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T10:07:00.515 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-06T10:07:00.515 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:1 2024-04-06T10:07:00.515 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T10:07:00.516 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T10:07:00.516 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:07:00.516 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=1 2024-04-06T10:07:00.516 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-06T10:07:00.516 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T10:07:00.516 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T10:07:00.516 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=1 2024-04-06T10:07:00.516 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=1 2024-04-06T10:07:00.517 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-06T10:07:00.540 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T10:07:00.540 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T10:07:00.540 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T10:07:00.540 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/image_45 2024-04-06T10:07:00.684 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T10:07:00.686 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror image_45 2024-04-06T10:07:00.686 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:07:00.686 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:07:00.686 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_45 2024-04-06T10:07:00.686 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror image_45 '' 2024-04-06T10:07:00.686 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:07:00.687 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:07:00.687 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_45 2024-04-06T10:07:00.687 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T10:07:00.687 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-image_45.status 2024-04-06T10:07:00.687 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-image_45.status 2024-04-06T10:07:00.687 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T10:07:00.689 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-image_45.status 2024-04-06T10:07:00.689 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_45 2024-04-06T10:07:00.689 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_45.status 2024-04-06T10:07:00.739 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 1 2024-04-06T10:07:00.739 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 1 2024-04-06T10:07:00.739 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T10:07:00.739 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=5, tag_tid=3, entry_tid=1], [object_number=4, tag_tid=3, entry_tid=0], [object_number=6, tag_tid=1, entry_tid=5450], [object_number=7, tag_tid=1, entry_tid=5447]]], state=connected] 2024-04-06T10:07:00.739 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=5, tag_tid=2, entry_tid=1], [object_number=4, tag_tid=2, entry_tid=0], [object_number=6, tag_tid=1, entry_tid=5450], [object_number=7, tag_tid=1, entry_tid=5447]]], state=connected] 2024-04-06T10:07:00.742 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_45.status 2024-04-06T10:07:00.744 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=5, tag_tid=3, entry_tid=1' 2024-04-06T10:07:00.745 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror image_45 2024-04-06T10:07:00.745 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:07:00.745 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:07:00.745 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_45 2024-04-06T10:07:00.746 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror image_45 '..*' 2024-04-06T10:07:00.746 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:07:00.746 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:07:00.746 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_45 2024-04-06T10:07:00.746 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T10:07:00.746 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-image_45.status 2024-04-06T10:07:00.746 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-image_45.status 2024-04-06T10:07:00.746 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T10:07:00.747 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-image_45.status 2024-04-06T10:07:00.747 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_45 2024-04-06T10:07:00.747 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_45.status 2024-04-06T10:07:00.791 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 1 2024-04-06T10:07:00.791 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 1 2024-04-06T10:07:00.791 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T10:07:00.791 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=5, tag_tid=3, entry_tid=1], [object_number=4, tag_tid=3, entry_tid=0], [object_number=6, tag_tid=1, entry_tid=5450], [object_number=7, tag_tid=1, entry_tid=5447]]], state=connected] 2024-04-06T10:07:00.791 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=5, tag_tid=2, entry_tid=1], [object_number=4, tag_tid=2, entry_tid=0], [object_number=6, tag_tid=1, entry_tid=5450], [object_number=7, tag_tid=1, entry_tid=5447]]], state=connected] 2024-04-06T10:07:00.795 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_45.status 2024-04-06T10:07:00.796 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=5, tag_tid=2, entry_tid=1' 2024-04-06T10:07:00.796 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=5, tag_tid=3, entry_tid=1' -a 'object_number=5, tag_tid=3, entry_tid=1' = 'object_number=5, tag_tid=2, entry_tid=1' 2024-04-06T10:07:00.796 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=5, tag_tid=2, entry_tid=1' '!=' 'object_number=5, tag_tid=2, entry_tid=1' 2024-04-06T10:07:00.797 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T10:07:00.797 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 2 2024-04-06T10:07:02.675 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.0 with skip (0.4787021307530991) > weight (0.25) 2024-04-06T10:07:02.675 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.1 with skip (0.45384181841503624) > weight (0.25) 2024-04-06T10:07:02.676 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.2 with skip (0.5366668552189043) > weight (0.25) 2024-04-06T10:07:02.676 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.3 with skip (0.5769926593318144) > weight (0.25) 2024-04-06T10:07:02.676 INFO:tasks.rbd_mirror_thrash:waiting for 83 secs before thrashing 2024-04-06T10:07:02.798 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror image_45 2024-04-06T10:07:02.798 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T10:07:02.798 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:07:02.798 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_45 2024-04-06T10:07:02.798 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T10:07:02.798 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n image_45 ']' 2024-04-06T10:07:02.798 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/image_45' 2024-04-06T10:07:02.799 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_45 2024-04-06T10:07:02.799 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T10:07:02.799 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T10:07:02.799 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T10:07:02.799 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T10:07:02.799 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T10:07:02.799 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T10:07:02.799 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T10:07:02.803 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T10:07:02.804 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T10:07:02.804 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T10:07:02.804 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_45 2024-04-06T10:07:02.804 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T10:07:02.804 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T10:07:02.804 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T10:07:02.804 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T10:07:02.804 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T10:07:02.804 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T10:07:02.804 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T10:07:02.804 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T10:07:02.804 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T10:07:02.804 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T10:07:02.804 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T10:07:02.804 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:07:02.805 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T10:07:02.805 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T10:07:02.805 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T10:07:02.805 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T10:07:02.805 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T10:07:02.805 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T10:07:02.805 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T10:07:02.805 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T10:07:02.805 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T10:07:02.805 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T10:07:02.805 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:07:02.805 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T10:07:02.805 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T10:07:02.805 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T10:07:02.806 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T10:07:02.806 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T10:07:02.806 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T10:07:02.806 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T10:07:02.828 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T10:07:02.828 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T10:07:02.828 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T10:07:02.828 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_45 2024-04-06T10:07:02.937 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:Invalid command: unused arguments: ['mirror/image_45'] 2024-04-06T10:07:02.937 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:rbd mirror flush : flush rbd mirror 2024-04-06T10:07:02.937 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:admin_socket: invalid command 2024-04-06T10:07:02.939 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T10:07:02.939 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/image_45 2024-04-06T10:07:02.939 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:1 2024-04-06T10:07:02.939 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T10:07:02.939 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T10:07:02.939 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-06T10:07:02.940 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:1 2024-04-06T10:07:02.940 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T10:07:02.940 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T10:07:02.940 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T10:07:02.940 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=1 2024-04-06T10:07:02.940 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-06T10:07:02.940 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T10:07:02.940 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:07:02.940 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=1 2024-04-06T10:07:02.940 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=1 2024-04-06T10:07:02.942 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-06T10:07:02.942 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:1 2024-04-06T10:07:02.942 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T10:07:02.942 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T10:07:02.942 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-06T10:07:02.942 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:1 2024-04-06T10:07:02.942 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T10:07:02.942 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T10:07:02.942 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:07:02.942 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=1 2024-04-06T10:07:02.942 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-06T10:07:02.942 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T10:07:02.943 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T10:07:02.943 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=1 2024-04-06T10:07:02.943 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=1 2024-04-06T10:07:02.943 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-06T10:07:02.966 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T10:07:02.966 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T10:07:02.967 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T10:07:02.967 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/image_45 2024-04-06T10:07:03.118 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T10:07:03.119 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror image_45 2024-04-06T10:07:03.119 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:07:03.119 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:07:03.119 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_45 2024-04-06T10:07:03.119 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror image_45 '' 2024-04-06T10:07:03.120 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:07:03.120 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:07:03.120 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_45 2024-04-06T10:07:03.120 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T10:07:03.120 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-image_45.status 2024-04-06T10:07:03.120 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-image_45.status 2024-04-06T10:07:03.121 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T10:07:03.123 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-image_45.status 2024-04-06T10:07:03.123 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_45 2024-04-06T10:07:03.123 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_45.status 2024-04-06T10:07:03.167 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 1 2024-04-06T10:07:03.167 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 1 2024-04-06T10:07:03.167 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T10:07:03.167 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=5, tag_tid=3, entry_tid=1], [object_number=4, tag_tid=3, entry_tid=0], [object_number=6, tag_tid=1, entry_tid=5450], [object_number=7, tag_tid=1, entry_tid=5447]]], state=connected] 2024-04-06T10:07:03.167 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=5, tag_tid=3, entry_tid=1], [object_number=4, tag_tid=3, entry_tid=0], [object_number=6, tag_tid=1, entry_tid=5450], [object_number=7, tag_tid=1, entry_tid=5447]]], state=connected] 2024-04-06T10:07:03.171 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_45.status 2024-04-06T10:07:03.172 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=5, tag_tid=3, entry_tid=1' 2024-04-06T10:07:03.173 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror image_45 2024-04-06T10:07:03.173 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:07:03.173 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:07:03.173 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_45 2024-04-06T10:07:03.173 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror image_45 '..*' 2024-04-06T10:07:03.173 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:07:03.174 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:07:03.174 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_45 2024-04-06T10:07:03.174 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T10:07:03.174 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-image_45.status 2024-04-06T10:07:03.174 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-image_45.status 2024-04-06T10:07:03.174 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T10:07:03.175 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-image_45.status 2024-04-06T10:07:03.175 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_45 2024-04-06T10:07:03.175 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_45.status 2024-04-06T10:07:03.219 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 1 2024-04-06T10:07:03.219 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 1 2024-04-06T10:07:03.219 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T10:07:03.219 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=5, tag_tid=3, entry_tid=1], [object_number=4, tag_tid=3, entry_tid=0], [object_number=6, tag_tid=1, entry_tid=5450], [object_number=7, tag_tid=1, entry_tid=5447]]], state=connected] 2024-04-06T10:07:03.219 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=5, tag_tid=3, entry_tid=1], [object_number=4, tag_tid=3, entry_tid=0], [object_number=6, tag_tid=1, entry_tid=5450], [object_number=7, tag_tid=1, entry_tid=5447]]], state=connected] 2024-04-06T10:07:03.223 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_45.status 2024-04-06T10:07:03.224 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=5, tag_tid=3, entry_tid=1' 2024-04-06T10:07:03.225 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=5, tag_tid=3, entry_tid=1' -a 'object_number=5, tag_tid=3, entry_tid=1' = 'object_number=5, tag_tid=3, entry_tid=1' 2024-04-06T10:07:03.225 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T10:07:03.225 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_snap_present cluster1 mirror image_45 snap 2024-04-06T10:07:03.225 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T10:07:03.225 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:07:03.225 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_45 2024-04-06T10:07:03.225 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local snap_name=snap 2024-04-06T10:07:03.225 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local s 2024-04-06T10:07:03.225 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-04-06T10:07:03.225 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 1 2024-04-06T10:07:04.226 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 info mirror/image_45@snap 2024-04-06T10:07:04.293 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:rbd image 'image_45': 2024-04-06T10:07:04.293 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: size 128 MiB in 32 objects 2024-04-06T10:07:04.293 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: order 22 (4 MiB objects) 2024-04-06T10:07:04.293 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: snapshot_count: 1 2024-04-06T10:07:04.293 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: id: 10acbb7e5d3c 2024-04-06T10:07:04.293 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: block_name_prefix: rbd_data.10acbb7e5d3c 2024-04-06T10:07:04.293 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: format: 2 2024-04-06T10:07:04.293 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: features: layering, exclusive-lock, journaling 2024-04-06T10:07:04.293 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: op_features: 2024-04-06T10:07:04.294 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: flags: 2024-04-06T10:07:04.294 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: create_timestamp: Sat Apr 6 09:57:29 2024 2024-04-06T10:07:04.294 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: access_timestamp: Sat Apr 6 09:57:29 2024 2024-04-06T10:07:04.294 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: modify_timestamp: Sat Apr 6 09:58:30 2024 2024-04-06T10:07:04.294 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: protected: False 2024-04-06T10:07:04.294 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: journal: 10acbb7e5d3c 2024-04-06T10:07:04.294 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: mirroring state: enabled 2024-04-06T10:07:04.294 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: mirroring mode: journal 2024-04-06T10:07:04.294 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: mirroring global id: 9cd08207-7987-4da9-8523-c1f42913c70b 2024-04-06T10:07:04.294 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: mirroring primary: false 2024-04-06T10:07:04.298 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T10:07:04.298 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ compare_image_snaps mirror image_45 snap 2024-04-06T10:07:04.298 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:07:04.299 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_45 2024-04-06T10:07:04.299 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local snap_name=snap 2024-04-06T10:07:04.299 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local ret=0 2024-04-06T10:07:04.299 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local rmt_export=/tmp/tmp.BhW6xKmROU/cluster2-mirror-image_45.export 2024-04-06T10:07:04.299 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loc_export=/tmp/tmp.BhW6xKmROU/cluster1-mirror-image_45.export 2024-04-06T10:07:04.299 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rm -f /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_45.export /tmp/tmp.BhW6xKmROU/cluster1-mirror-image_45.export 2024-04-06T10:07:04.299 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster2 -p mirror export image_45@snap /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_45.export 2024-04-06T10:07:04.472 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-04-06T10:07:04.480 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 -p mirror export image_45@snap /tmp/tmp.BhW6xKmROU/cluster1-mirror-image_45.export 2024-04-06T10:07:04.804 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-04-06T10:07:04.818 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmp /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_45.export /tmp/tmp.BhW6xKmROU/cluster1-mirror-image_45.export 2024-04-06T10:07:04.898 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rm -f /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_45.export /tmp/tmp.BhW6xKmROU/cluster1-mirror-image_45.export 2024-04-06T10:07:04.940 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T10:07:04.941 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-06T10:07:04.941 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ image=image_46 2024-04-06T10:07:04.941 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ create_snap cluster2 mirror image_46 snap 2024-04-06T10:07:04.941 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-04-06T10:07:04.941 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:07:04.941 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_46 2024-04-06T10:07:04.941 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local snap_name=snap 2024-04-06T10:07:04.941 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster2 -p mirror snap create image_46@snap --debug-rbd=20 --debug-journaler=20 2024-04-06T10:07:05.936 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_image_replay_started cluster1 mirror image_46 2024-04-06T10:07:05.936 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T10:07:05.936 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:07:05.936 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_46 2024-04-06T10:07:05.936 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_image_replay_state cluster1 mirror image_46 started 2024-04-06T10:07:05.936 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T10:07:05.936 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:07:05.936 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_46 2024-04-06T10:07:05.936 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state=started 2024-04-06T10:07:05.936 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local s 2024-04-06T10:07:05.936 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T10:07:05.937 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 1 2024-04-06T10:07:06.938 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test_image_replay_state cluster1 mirror image_46 started 2024-04-06T10:07:06.938 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T10:07:06.938 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:07:06.938 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_46 2024-04-06T10:07:06.938 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local test_state=started 2024-04-06T10:07:06.938 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local status_result 2024-04-06T10:07:06.938 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local current_state=stopped 2024-04-06T10:07:06.939 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_46 2024-04-06T10:07:06.939 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -i state 2024-04-06T10:07:06.939 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_instance=cluster1 2024-04-06T10:07:06.940 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ shift 2024-04-06T10:07:06.940 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T10:07:06.940 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance=cluster1 2024-04-06T10:07:06.940 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local loop_instance 2024-04-06T10:07:06.940 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T10:07:06.940 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sleep 0 2024-04-06T10:07:06.940 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-04-06T10:07:06.941 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ seq 0 3 2024-04-06T10:07:06.944 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T10:07:06.944 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_46 2024-04-06T10:07:06.944 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1:0 2024-04-06T10:07:06.944 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ shift 2024-04-06T10:07:06.944 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T10:07:06.944 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-04-06T10:07:06.944 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T10:07:06.944 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=cluster 2024-04-06T10:07:06.944 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T10:07:06.944 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:07:06.945 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T10:07:06.945 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T10:07:06.945 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval cluster=cluster1 2024-04-06T10:07:06.945 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ cluster=cluster1 2024-04-06T10:07:06.945 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T10:07:06.945 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T10:07:06.945 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-04-06T10:07:06.945 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local local_cluster=cluster1:0 2024-04-06T10:07:06.945 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local cluster=cluster1 2024-04-06T10:07:06.945 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local instance 2024-04-06T10:07:06.945 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T10:07:06.945 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local val=cluster1:0 2024-04-06T10:07:06.945 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local cluster_var_name=local_cluster 2024-04-06T10:07:06.945 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local instance_var_name=instance 2024-04-06T10:07:06.945 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ cluster=cluster1 2024-04-06T10:07:06.945 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T10:07:06.946 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ '[' 0 = cluster1:0 ']' 2024-04-06T10:07:06.946 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ eval local_cluster=cluster1 2024-04-06T10:07:06.946 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++++ local_cluster=cluster1 2024-04-06T10:07:06.946 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ eval instance=0 2024-04-06T10:07:06.946 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++++ instance=0 2024-04-06T10:07:06.946 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T10:07:06.968 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T10:07:06.968 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T10:07:06.969 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T10:07:06.969 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_46 2024-04-06T10:07:07.073 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:Invalid command: unused arguments: ['mirror/image_46'] 2024-04-06T10:07:07.073 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:rbd mirror status : get status for rbd mirror 2024-04-06T10:07:07.073 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:admin_socket: invalid command 2024-04-06T10:07:07.075 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T10:07:07.076 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_46 2024-04-06T10:07:07.076 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1:1 2024-04-06T10:07:07.076 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ shift 2024-04-06T10:07:07.076 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T10:07:07.076 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-04-06T10:07:07.076 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:1 2024-04-06T10:07:07.076 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=cluster 2024-04-06T10:07:07.076 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T10:07:07.076 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:07:07.076 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=1 2024-04-06T10:07:07.076 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-06T10:07:07.076 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval cluster=cluster1 2024-04-06T10:07:07.076 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ cluster=cluster1 2024-04-06T10:07:07.077 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=1 2024-04-06T10:07:07.077 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=1 2024-04-06T10:07:07.078 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-04-06T10:07:07.079 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local local_cluster=cluster1:1 2024-04-06T10:07:07.079 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local cluster=cluster1 2024-04-06T10:07:07.079 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local instance 2024-04-06T10:07:07.079 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-06T10:07:07.079 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local val=cluster1:1 2024-04-06T10:07:07.079 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local cluster_var_name=local_cluster 2024-04-06T10:07:07.079 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local instance_var_name=instance 2024-04-06T10:07:07.079 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ cluster=cluster1 2024-04-06T10:07:07.079 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=1 2024-04-06T10:07:07.079 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ '[' 1 = cluster1:1 ']' 2024-04-06T10:07:07.079 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ eval local_cluster=cluster1 2024-04-06T10:07:07.079 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++++ local_cluster=cluster1 2024-04-06T10:07:07.079 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ eval instance=1 2024-04-06T10:07:07.080 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++++ instance=1 2024-04-06T10:07:07.080 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-06T10:07:07.101 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T10:07:07.101 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T10:07:07.101 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T10:07:07.101 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_46 2024-04-06T10:07:07.251 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ return 0 2024-04-06T10:07:07.251 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ status_result=' "state": "Replaying"' 2024-04-06T10:07:07.252 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ echo ' "state": "Replaying"' 2024-04-06T10:07:07.252 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep -i Replaying 2024-04-06T10:07:07.253 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: "state": "Replaying" 2024-04-06T10:07:07.255 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ current_state=started 2024-04-06T10:07:07.255 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test started = started 2024-04-06T10:07:07.255 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T10:07:07.255 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_46 2024-04-06T10:07:07.255 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local local_cluster=cluster1 2024-04-06T10:07:07.255 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-04-06T10:07:07.255 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:07:07.255 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_46 2024-04-06T10:07:07.255 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' journal = journal ']' 2024-04-06T10:07:07.255 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_journal_replay_complete cluster1 cluster2 mirror image_46 2024-04-06T10:07:07.255 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local local_cluster=cluster1 2024-04-06T10:07:07.255 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-04-06T10:07:07.256 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:07:07.256 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_46 2024-04-06T10:07:07.256 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local s master_pos mirror_pos last_mirror_pos 2024-04-06T10:07:07.256 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local master_tag master_entry mirror_tag mirror_entry 2024-04-06T10:07:07.256 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T10:07:07.256 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T10:07:07.256 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T10:07:07.456 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror image_46 2024-04-06T10:07:07.456 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T10:07:07.456 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:07:07.456 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_46 2024-04-06T10:07:07.456 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T10:07:07.456 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n image_46 ']' 2024-04-06T10:07:07.456 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/image_46' 2024-04-06T10:07:07.457 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_46 2024-04-06T10:07:07.457 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T10:07:07.457 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T10:07:07.457 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T10:07:07.457 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T10:07:07.457 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T10:07:07.457 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T10:07:07.457 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T10:07:07.457 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T10:07:07.458 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T10:07:07.459 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T10:07:07.460 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_46 2024-04-06T10:07:07.460 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T10:07:07.460 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T10:07:07.460 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T10:07:07.460 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T10:07:07.460 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T10:07:07.460 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T10:07:07.460 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T10:07:07.460 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T10:07:07.460 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T10:07:07.460 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T10:07:07.460 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T10:07:07.460 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:07:07.461 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T10:07:07.461 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T10:07:07.463 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T10:07:07.463 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T10:07:07.463 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T10:07:07.463 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T10:07:07.463 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T10:07:07.463 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T10:07:07.463 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T10:07:07.463 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T10:07:07.463 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:07:07.463 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T10:07:07.463 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T10:07:07.463 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T10:07:07.464 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T10:07:07.464 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T10:07:07.464 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T10:07:07.464 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T10:07:07.486 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T10:07:07.487 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T10:07:07.487 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T10:07:07.487 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_46 2024-04-06T10:07:07.593 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:Invalid command: unused arguments: ['mirror/image_46'] 2024-04-06T10:07:07.593 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:rbd mirror flush : flush rbd mirror 2024-04-06T10:07:07.593 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:admin_socket: invalid command 2024-04-06T10:07:07.595 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T10:07:07.595 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/image_46 2024-04-06T10:07:07.595 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:1 2024-04-06T10:07:07.595 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T10:07:07.596 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T10:07:07.596 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-06T10:07:07.596 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:1 2024-04-06T10:07:07.596 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T10:07:07.596 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T10:07:07.596 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T10:07:07.596 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=1 2024-04-06T10:07:07.596 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-06T10:07:07.596 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T10:07:07.596 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:07:07.596 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=1 2024-04-06T10:07:07.596 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=1 2024-04-06T10:07:07.598 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-06T10:07:07.598 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:1 2024-04-06T10:07:07.598 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T10:07:07.598 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T10:07:07.598 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-06T10:07:07.598 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:1 2024-04-06T10:07:07.598 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T10:07:07.598 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T10:07:07.598 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:07:07.598 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=1 2024-04-06T10:07:07.598 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-06T10:07:07.598 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T10:07:07.598 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T10:07:07.598 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=1 2024-04-06T10:07:07.599 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=1 2024-04-06T10:07:07.599 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-06T10:07:07.621 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T10:07:07.622 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T10:07:07.622 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T10:07:07.622 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/image_46 2024-04-06T10:07:07.761 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T10:07:07.762 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror image_46 2024-04-06T10:07:07.762 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:07:07.762 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:07:07.762 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_46 2024-04-06T10:07:07.762 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror image_46 '' 2024-04-06T10:07:07.762 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:07:07.762 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:07:07.763 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_46 2024-04-06T10:07:07.763 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T10:07:07.763 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-image_46.status 2024-04-06T10:07:07.763 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-image_46.status 2024-04-06T10:07:07.763 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T10:07:07.766 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-image_46.status 2024-04-06T10:07:07.766 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_46 2024-04-06T10:07:07.766 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_46.status 2024-04-06T10:07:07.816 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 0 2024-04-06T10:07:07.816 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 0 2024-04-06T10:07:07.816 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T10:07:07.816 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=1, tag_tid=3, entry_tid=1], [object_number=0, tag_tid=3, entry_tid=0], [object_number=3, tag_tid=1, entry_tid=4091], [object_number=2, tag_tid=1, entry_tid=4090]]], state=connected] 2024-04-06T10:07:07.816 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=1, tag_tid=1, entry_tid=3901], [object_number=0, tag_tid=1, entry_tid=3900], [object_number=3, tag_tid=1, entry_tid=3899], [object_number=2, tag_tid=1, entry_tid=3898]]], state=connected] 2024-04-06T10:07:07.820 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_46.status 2024-04-06T10:07:07.821 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=1, tag_tid=3, entry_tid=1' 2024-04-06T10:07:07.822 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror image_46 2024-04-06T10:07:07.822 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:07:07.822 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:07:07.822 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_46 2024-04-06T10:07:07.823 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror image_46 '..*' 2024-04-06T10:07:07.823 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:07:07.823 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:07:07.823 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_46 2024-04-06T10:07:07.823 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T10:07:07.823 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-image_46.status 2024-04-06T10:07:07.823 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-image_46.status 2024-04-06T10:07:07.823 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T10:07:07.824 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-image_46.status 2024-04-06T10:07:07.824 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_46 2024-04-06T10:07:07.824 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_46.status 2024-04-06T10:07:07.868 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 0 2024-04-06T10:07:07.868 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 0 2024-04-06T10:07:07.868 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T10:07:07.868 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=1, tag_tid=3, entry_tid=1], [object_number=0, tag_tid=3, entry_tid=0], [object_number=3, tag_tid=1, entry_tid=4091], [object_number=2, tag_tid=1, entry_tid=4090]]], state=connected] 2024-04-06T10:07:07.868 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=1, tag_tid=1, entry_tid=3901], [object_number=0, tag_tid=1, entry_tid=3900], [object_number=3, tag_tid=1, entry_tid=3899], [object_number=2, tag_tid=1, entry_tid=3898]]], state=connected] 2024-04-06T10:07:07.872 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_46.status 2024-04-06T10:07:07.873 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=1, tag_tid=1, entry_tid=3901' 2024-04-06T10:07:07.874 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=1, tag_tid=3, entry_tid=1' -a 'object_number=1, tag_tid=3, entry_tid=1' = 'object_number=1, tag_tid=1, entry_tid=3901' 2024-04-06T10:07:07.874 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=1, tag_tid=1, entry_tid=3901' '!=' '' 2024-04-06T10:07:07.874 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T10:07:07.874 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=1, tag_tid=1, entry_tid=3901' = '' 2024-04-06T10:07:07.874 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=1, tag_tid=1, entry_tid=3901' 2024-04-06T10:07:07.874 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=1, tag_tid=3, entry_tid=1' 2024-04-06T10:07:07.875 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T10:07:07.875 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T10:07:07.875 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=3 2024-04-06T10:07:07.876 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=1, tag_tid=1, entry_tid=3901' 2024-04-06T10:07:07.876 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T10:07:07.876 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T10:07:07.877 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=1 2024-04-06T10:07:07.878 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=1, tag_tid=3, entry_tid=1' 2024-04-06T10:07:07.878 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T10:07:07.878 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T10:07:07.879 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T10:07:07.879 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=1, tag_tid=1, entry_tid=3901' 2024-04-06T10:07:07.879 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T10:07:07.880 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T10:07:07.881 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=3901 2024-04-06T10:07:07.881 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 3 = 1 -a 1 -le 3901 2024-04-06T10:07:07.881 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T10:07:07.881 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T10:07:07.881 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T10:07:08.083 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror image_46 2024-04-06T10:07:08.083 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T10:07:08.083 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:07:08.083 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_46 2024-04-06T10:07:08.083 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T10:07:08.083 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n image_46 ']' 2024-04-06T10:07:08.083 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/image_46' 2024-04-06T10:07:08.083 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_46 2024-04-06T10:07:08.083 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T10:07:08.083 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T10:07:08.083 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T10:07:08.083 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T10:07:08.084 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T10:07:08.084 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T10:07:08.084 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T10:07:08.084 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T10:07:08.085 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T10:07:08.088 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T10:07:08.088 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_46 2024-04-06T10:07:08.088 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T10:07:08.088 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T10:07:08.088 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T10:07:08.088 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T10:07:08.088 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T10:07:08.088 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T10:07:08.089 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T10:07:08.089 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T10:07:08.089 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T10:07:08.089 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T10:07:08.089 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T10:07:08.089 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:07:08.089 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T10:07:08.089 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T10:07:08.089 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T10:07:08.089 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T10:07:08.089 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T10:07:08.089 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T10:07:08.089 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T10:07:08.089 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T10:07:08.090 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T10:07:08.090 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T10:07:08.090 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:07:08.090 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T10:07:08.090 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T10:07:08.090 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T10:07:08.090 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T10:07:08.090 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T10:07:08.090 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T10:07:08.090 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T10:07:08.113 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T10:07:08.114 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T10:07:08.114 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T10:07:08.114 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_46 2024-04-06T10:07:08.225 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:Invalid command: unused arguments: ['mirror/image_46'] 2024-04-06T10:07:08.225 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:rbd mirror flush : flush rbd mirror 2024-04-06T10:07:08.225 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:admin_socket: invalid command 2024-04-06T10:07:08.229 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T10:07:08.229 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/image_46 2024-04-06T10:07:08.229 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:1 2024-04-06T10:07:08.229 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T10:07:08.229 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T10:07:08.229 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-06T10:07:08.229 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:1 2024-04-06T10:07:08.229 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T10:07:08.229 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T10:07:08.229 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T10:07:08.230 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=1 2024-04-06T10:07:08.230 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-06T10:07:08.230 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T10:07:08.230 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:07:08.230 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=1 2024-04-06T10:07:08.230 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=1 2024-04-06T10:07:08.230 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-06T10:07:08.230 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:1 2024-04-06T10:07:08.230 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T10:07:08.230 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T10:07:08.230 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-06T10:07:08.230 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:1 2024-04-06T10:07:08.230 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T10:07:08.230 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T10:07:08.230 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:07:08.231 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=1 2024-04-06T10:07:08.231 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-06T10:07:08.231 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T10:07:08.231 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T10:07:08.231 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=1 2024-04-06T10:07:08.231 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=1 2024-04-06T10:07:08.231 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-06T10:07:08.248 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T10:07:08.249 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T10:07:08.249 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T10:07:08.249 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/image_46 2024-04-06T10:07:08.386 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T10:07:08.388 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror image_46 2024-04-06T10:07:08.388 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:07:08.388 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:07:08.388 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_46 2024-04-06T10:07:08.388 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror image_46 '' 2024-04-06T10:07:08.388 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:07:08.388 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:07:08.388 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_46 2024-04-06T10:07:08.388 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T10:07:08.388 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-image_46.status 2024-04-06T10:07:08.389 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-image_46.status 2024-04-06T10:07:08.389 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T10:07:08.391 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-image_46.status 2024-04-06T10:07:08.391 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_46 2024-04-06T10:07:08.391 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_46.status 2024-04-06T10:07:08.434 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 0 2024-04-06T10:07:08.434 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 0 2024-04-06T10:07:08.434 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T10:07:08.434 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=1, tag_tid=3, entry_tid=1], [object_number=0, tag_tid=3, entry_tid=0], [object_number=3, tag_tid=1, entry_tid=4091], [object_number=2, tag_tid=1, entry_tid=4090]]], state=connected] 2024-04-06T10:07:08.434 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=1, tag_tid=1, entry_tid=3901], [object_number=0, tag_tid=1, entry_tid=3900], [object_number=3, tag_tid=1, entry_tid=3899], [object_number=2, tag_tid=1, entry_tid=3898]]], state=connected] 2024-04-06T10:07:08.437 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_46.status 2024-04-06T10:07:08.439 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=1, tag_tid=3, entry_tid=1' 2024-04-06T10:07:08.440 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror image_46 2024-04-06T10:07:08.440 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:07:08.440 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:07:08.440 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_46 2024-04-06T10:07:08.440 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror image_46 '..*' 2024-04-06T10:07:08.440 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:07:08.440 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:07:08.440 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_46 2024-04-06T10:07:08.440 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T10:07:08.440 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-image_46.status 2024-04-06T10:07:08.441 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-image_46.status 2024-04-06T10:07:08.441 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T10:07:08.441 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-image_46.status 2024-04-06T10:07:08.441 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_46 2024-04-06T10:07:08.441 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_46.status 2024-04-06T10:07:08.486 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 0 2024-04-06T10:07:08.486 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 0 2024-04-06T10:07:08.486 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T10:07:08.486 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=1, tag_tid=3, entry_tid=1], [object_number=0, tag_tid=3, entry_tid=0], [object_number=3, tag_tid=1, entry_tid=4091], [object_number=2, tag_tid=1, entry_tid=4090]]], state=connected] 2024-04-06T10:07:08.486 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=1, tag_tid=1, entry_tid=3901], [object_number=0, tag_tid=1, entry_tid=3900], [object_number=3, tag_tid=1, entry_tid=3899], [object_number=2, tag_tid=1, entry_tid=3898]]], state=connected] 2024-04-06T10:07:08.489 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_46.status 2024-04-06T10:07:08.491 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=1, tag_tid=1, entry_tid=3901' 2024-04-06T10:07:08.491 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=1, tag_tid=3, entry_tid=1' -a 'object_number=1, tag_tid=3, entry_tid=1' = 'object_number=1, tag_tid=1, entry_tid=3901' 2024-04-06T10:07:08.491 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=1, tag_tid=1, entry_tid=3901' '!=' 'object_number=1, tag_tid=1, entry_tid=3901' 2024-04-06T10:07:08.491 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T10:07:08.491 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.4 2024-04-06T10:07:08.893 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror image_46 2024-04-06T10:07:08.893 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T10:07:08.893 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:07:08.893 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_46 2024-04-06T10:07:08.894 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T10:07:08.894 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n image_46 ']' 2024-04-06T10:07:08.894 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/image_46' 2024-04-06T10:07:08.894 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_46 2024-04-06T10:07:08.894 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T10:07:08.894 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T10:07:08.894 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T10:07:08.894 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T10:07:08.894 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T10:07:08.894 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T10:07:08.894 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T10:07:08.895 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T10:07:08.895 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T10:07:08.898 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T10:07:08.898 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_46 2024-04-06T10:07:08.898 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T10:07:08.899 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T10:07:08.899 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T10:07:08.899 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T10:07:08.899 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T10:07:08.899 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T10:07:08.899 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T10:07:08.899 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T10:07:08.899 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T10:07:08.899 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T10:07:08.899 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T10:07:08.899 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:07:08.899 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T10:07:08.899 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T10:07:08.899 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T10:07:08.899 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T10:07:08.900 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T10:07:08.900 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T10:07:08.900 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T10:07:08.900 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T10:07:08.900 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T10:07:08.900 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T10:07:08.900 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:07:08.900 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T10:07:08.900 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T10:07:08.900 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T10:07:08.900 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T10:07:08.900 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T10:07:08.900 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T10:07:08.900 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T10:07:08.923 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T10:07:08.924 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T10:07:08.924 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T10:07:08.924 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_46 2024-04-06T10:07:09.029 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:Invalid command: unused arguments: ['mirror/image_46'] 2024-04-06T10:07:09.030 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:rbd mirror flush : flush rbd mirror 2024-04-06T10:07:09.030 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:admin_socket: invalid command 2024-04-06T10:07:09.032 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T10:07:09.032 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/image_46 2024-04-06T10:07:09.032 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:1 2024-04-06T10:07:09.032 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T10:07:09.032 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T10:07:09.032 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-06T10:07:09.032 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:1 2024-04-06T10:07:09.032 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T10:07:09.033 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T10:07:09.033 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T10:07:09.033 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=1 2024-04-06T10:07:09.033 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-06T10:07:09.033 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T10:07:09.033 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:07:09.033 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=1 2024-04-06T10:07:09.033 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=1 2024-04-06T10:07:09.035 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-06T10:07:09.035 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:1 2024-04-06T10:07:09.035 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T10:07:09.035 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T10:07:09.035 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-06T10:07:09.035 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:1 2024-04-06T10:07:09.035 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T10:07:09.035 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T10:07:09.035 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:07:09.035 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=1 2024-04-06T10:07:09.035 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-06T10:07:09.036 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T10:07:09.036 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T10:07:09.036 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=1 2024-04-06T10:07:09.036 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=1 2024-04-06T10:07:09.036 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-06T10:07:09.058 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T10:07:09.059 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T10:07:09.059 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T10:07:09.059 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/image_46 2024-04-06T10:07:09.195 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T10:07:09.197 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror image_46 2024-04-06T10:07:09.197 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:07:09.198 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:07:09.198 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_46 2024-04-06T10:07:09.198 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror image_46 '' 2024-04-06T10:07:09.198 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:07:09.198 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:07:09.198 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_46 2024-04-06T10:07:09.199 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T10:07:09.199 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-image_46.status 2024-04-06T10:07:09.199 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-image_46.status 2024-04-06T10:07:09.199 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T10:07:09.200 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-image_46.status 2024-04-06T10:07:09.200 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_46 2024-04-06T10:07:09.201 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_46.status 2024-04-06T10:07:09.255 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 0 2024-04-06T10:07:09.255 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 0 2024-04-06T10:07:09.255 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T10:07:09.255 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=1, tag_tid=3, entry_tid=1], [object_number=0, tag_tid=3, entry_tid=0], [object_number=3, tag_tid=1, entry_tid=4091], [object_number=2, tag_tid=1, entry_tid=4090]]], state=connected] 2024-04-06T10:07:09.255 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=1, tag_tid=1, entry_tid=3901], [object_number=0, tag_tid=1, entry_tid=3900], [object_number=3, tag_tid=1, entry_tid=3899], [object_number=2, tag_tid=1, entry_tid=3898]]], state=connected] 2024-04-06T10:07:09.259 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_46.status 2024-04-06T10:07:09.260 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=1, tag_tid=3, entry_tid=1' 2024-04-06T10:07:09.261 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror image_46 2024-04-06T10:07:09.261 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:07:09.261 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:07:09.261 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_46 2024-04-06T10:07:09.261 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror image_46 '..*' 2024-04-06T10:07:09.261 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:07:09.261 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:07:09.262 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_46 2024-04-06T10:07:09.262 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T10:07:09.262 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-image_46.status 2024-04-06T10:07:09.262 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-image_46.status 2024-04-06T10:07:09.262 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T10:07:09.262 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-image_46.status 2024-04-06T10:07:09.263 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_46 2024-04-06T10:07:09.263 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_46.status 2024-04-06T10:07:09.307 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 0 2024-04-06T10:07:09.307 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 0 2024-04-06T10:07:09.307 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T10:07:09.307 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=1, tag_tid=3, entry_tid=1], [object_number=0, tag_tid=3, entry_tid=0], [object_number=3, tag_tid=1, entry_tid=4091], [object_number=2, tag_tid=1, entry_tid=4090]]], state=connected] 2024-04-06T10:07:09.307 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=1, tag_tid=1, entry_tid=3901], [object_number=0, tag_tid=1, entry_tid=3900], [object_number=3, tag_tid=1, entry_tid=3899], [object_number=2, tag_tid=1, entry_tid=3898]]], state=connected] 2024-04-06T10:07:09.311 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_46.status 2024-04-06T10:07:09.313 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=1, tag_tid=1, entry_tid=3901' 2024-04-06T10:07:09.313 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=1, tag_tid=3, entry_tid=1' -a 'object_number=1, tag_tid=3, entry_tid=1' = 'object_number=1, tag_tid=1, entry_tid=3901' 2024-04-06T10:07:09.313 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=1, tag_tid=1, entry_tid=3901' '!=' 'object_number=1, tag_tid=1, entry_tid=3901' 2024-04-06T10:07:09.313 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T10:07:09.313 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.8 2024-04-06T10:07:10.115 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror image_46 2024-04-06T10:07:10.115 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T10:07:10.115 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:07:10.115 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_46 2024-04-06T10:07:10.115 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T10:07:10.115 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n image_46 ']' 2024-04-06T10:07:10.115 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/image_46' 2024-04-06T10:07:10.115 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_46 2024-04-06T10:07:10.115 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T10:07:10.116 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T10:07:10.116 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T10:07:10.116 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T10:07:10.116 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T10:07:10.116 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T10:07:10.116 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T10:07:10.116 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T10:07:10.117 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T10:07:10.119 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T10:07:10.119 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_46 2024-04-06T10:07:10.119 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T10:07:10.119 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T10:07:10.119 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T10:07:10.119 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T10:07:10.119 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T10:07:10.120 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T10:07:10.120 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T10:07:10.120 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T10:07:10.120 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T10:07:10.120 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T10:07:10.120 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T10:07:10.120 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:07:10.120 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T10:07:10.120 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T10:07:10.121 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T10:07:10.121 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T10:07:10.121 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T10:07:10.122 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T10:07:10.122 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T10:07:10.122 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T10:07:10.122 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T10:07:10.122 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T10:07:10.122 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:07:10.122 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T10:07:10.122 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T10:07:10.122 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T10:07:10.122 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T10:07:10.122 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T10:07:10.122 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T10:07:10.122 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T10:07:10.153 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T10:07:10.154 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T10:07:10.154 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T10:07:10.154 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_46 2024-04-06T10:07:10.263 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:Invalid command: unused arguments: ['mirror/image_46'] 2024-04-06T10:07:10.264 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:rbd mirror flush : flush rbd mirror 2024-04-06T10:07:10.264 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:admin_socket: invalid command 2024-04-06T10:07:10.266 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T10:07:10.266 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/image_46 2024-04-06T10:07:10.266 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:1 2024-04-06T10:07:10.266 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T10:07:10.266 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T10:07:10.266 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-06T10:07:10.266 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:1 2024-04-06T10:07:10.266 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T10:07:10.267 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T10:07:10.267 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T10:07:10.267 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=1 2024-04-06T10:07:10.267 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-06T10:07:10.267 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T10:07:10.267 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:07:10.267 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=1 2024-04-06T10:07:10.267 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=1 2024-04-06T10:07:10.269 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-06T10:07:10.269 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:1 2024-04-06T10:07:10.269 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T10:07:10.269 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T10:07:10.269 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-06T10:07:10.269 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:1 2024-04-06T10:07:10.269 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T10:07:10.269 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T10:07:10.269 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:07:10.269 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=1 2024-04-06T10:07:10.269 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-06T10:07:10.269 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T10:07:10.269 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T10:07:10.270 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=1 2024-04-06T10:07:10.270 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=1 2024-04-06T10:07:10.270 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-06T10:07:10.291 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T10:07:10.291 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T10:07:10.291 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T10:07:10.292 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/image_46 2024-04-06T10:07:10.434 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T10:07:10.436 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror image_46 2024-04-06T10:07:10.436 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:07:10.436 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:07:10.436 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_46 2024-04-06T10:07:10.436 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror image_46 '' 2024-04-06T10:07:10.436 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:07:10.436 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:07:10.436 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_46 2024-04-06T10:07:10.436 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T10:07:10.437 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-image_46.status 2024-04-06T10:07:10.437 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-image_46.status 2024-04-06T10:07:10.437 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T10:07:10.438 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-image_46.status 2024-04-06T10:07:10.439 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_46 2024-04-06T10:07:10.439 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_46.status 2024-04-06T10:07:10.483 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 0 2024-04-06T10:07:10.483 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 0 2024-04-06T10:07:10.483 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T10:07:10.483 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=1, tag_tid=3, entry_tid=1], [object_number=0, tag_tid=3, entry_tid=0], [object_number=3, tag_tid=1, entry_tid=4091], [object_number=2, tag_tid=1, entry_tid=4090]]], state=connected] 2024-04-06T10:07:10.483 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=0, tag_tid=1, entry_tid=4092], [object_number=3, tag_tid=1, entry_tid=4091], [object_number=2, tag_tid=1, entry_tid=4090], [object_number=1, tag_tid=1, entry_tid=4089]]], state=connected] 2024-04-06T10:07:10.486 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_46.status 2024-04-06T10:07:10.488 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=1, tag_tid=3, entry_tid=1' 2024-04-06T10:07:10.489 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror image_46 2024-04-06T10:07:10.489 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:07:10.489 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:07:10.489 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_46 2024-04-06T10:07:10.489 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror image_46 '..*' 2024-04-06T10:07:10.489 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:07:10.489 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:07:10.489 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_46 2024-04-06T10:07:10.490 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T10:07:10.490 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-image_46.status 2024-04-06T10:07:10.490 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-image_46.status 2024-04-06T10:07:10.490 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T10:07:10.490 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-image_46.status 2024-04-06T10:07:10.491 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_46 2024-04-06T10:07:10.491 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_46.status 2024-04-06T10:07:10.534 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 0 2024-04-06T10:07:10.535 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 0 2024-04-06T10:07:10.535 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T10:07:10.535 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=1, tag_tid=3, entry_tid=1], [object_number=0, tag_tid=3, entry_tid=0], [object_number=3, tag_tid=1, entry_tid=4091], [object_number=2, tag_tid=1, entry_tid=4090]]], state=connected] 2024-04-06T10:07:10.535 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=0, tag_tid=1, entry_tid=4092], [object_number=3, tag_tid=1, entry_tid=4091], [object_number=2, tag_tid=1, entry_tid=4090], [object_number=1, tag_tid=1, entry_tid=4089]]], state=connected] 2024-04-06T10:07:10.538 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_46.status 2024-04-06T10:07:10.540 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=0, tag_tid=1, entry_tid=4092' 2024-04-06T10:07:10.540 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=1, tag_tid=3, entry_tid=1' -a 'object_number=1, tag_tid=3, entry_tid=1' = 'object_number=0, tag_tid=1, entry_tid=4092' 2024-04-06T10:07:10.540 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=0, tag_tid=1, entry_tid=4092' '!=' 'object_number=1, tag_tid=1, entry_tid=3901' 2024-04-06T10:07:10.540 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T10:07:10.541 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=0, tag_tid=1, entry_tid=4092' = 'object_number=1, tag_tid=1, entry_tid=3901' 2024-04-06T10:07:10.541 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=0, tag_tid=1, entry_tid=4092' 2024-04-06T10:07:10.541 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=1, tag_tid=3, entry_tid=1' 2024-04-06T10:07:10.541 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T10:07:10.541 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T10:07:10.542 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=3 2024-04-06T10:07:10.542 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=0, tag_tid=1, entry_tid=4092' 2024-04-06T10:07:10.542 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T10:07:10.543 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T10:07:10.543 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=1 2024-04-06T10:07:10.544 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=1, tag_tid=3, entry_tid=1' 2024-04-06T10:07:10.544 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T10:07:10.544 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T10:07:10.545 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T10:07:10.546 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=0, tag_tid=1, entry_tid=4092' 2024-04-06T10:07:10.546 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T10:07:10.546 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T10:07:10.547 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=4092 2024-04-06T10:07:10.547 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 3 = 1 -a 1 -le 4092 2024-04-06T10:07:10.547 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T10:07:10.547 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T10:07:10.547 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T10:07:10.749 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror image_46 2024-04-06T10:07:10.750 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T10:07:10.750 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:07:10.750 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_46 2024-04-06T10:07:10.750 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T10:07:10.750 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n image_46 ']' 2024-04-06T10:07:10.750 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/image_46' 2024-04-06T10:07:10.750 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_46 2024-04-06T10:07:10.750 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T10:07:10.750 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T10:07:10.750 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T10:07:10.750 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T10:07:10.750 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T10:07:10.751 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T10:07:10.751 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T10:07:10.751 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T10:07:10.751 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T10:07:10.754 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T10:07:10.754 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_46 2024-04-06T10:07:10.754 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T10:07:10.754 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T10:07:10.754 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T10:07:10.754 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T10:07:10.754 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T10:07:10.754 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T10:07:10.755 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T10:07:10.755 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T10:07:10.755 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T10:07:10.755 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T10:07:10.755 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T10:07:10.755 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:07:10.755 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T10:07:10.755 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T10:07:10.755 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T10:07:10.755 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T10:07:10.755 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T10:07:10.755 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T10:07:10.755 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T10:07:10.755 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T10:07:10.756 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T10:07:10.756 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T10:07:10.756 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:07:10.756 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T10:07:10.756 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T10:07:10.756 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T10:07:10.756 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T10:07:10.756 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T10:07:10.756 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T10:07:10.756 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T10:07:10.779 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T10:07:10.779 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T10:07:10.779 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T10:07:10.779 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_46 2024-04-06T10:07:10.883 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:Invalid command: unused arguments: ['mirror/image_46'] 2024-04-06T10:07:10.883 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:rbd mirror flush : flush rbd mirror 2024-04-06T10:07:10.883 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:admin_socket: invalid command 2024-04-06T10:07:10.886 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T10:07:10.886 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/image_46 2024-04-06T10:07:10.886 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:1 2024-04-06T10:07:10.886 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T10:07:10.886 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T10:07:10.886 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-06T10:07:10.886 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:1 2024-04-06T10:07:10.886 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T10:07:10.886 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T10:07:10.886 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T10:07:10.886 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=1 2024-04-06T10:07:10.887 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-06T10:07:10.887 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T10:07:10.887 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:07:10.887 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=1 2024-04-06T10:07:10.887 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=1 2024-04-06T10:07:10.888 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-06T10:07:10.888 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:1 2024-04-06T10:07:10.888 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T10:07:10.888 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T10:07:10.888 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-06T10:07:10.889 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:1 2024-04-06T10:07:10.889 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T10:07:10.889 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T10:07:10.889 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:07:10.889 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=1 2024-04-06T10:07:10.889 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-06T10:07:10.889 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T10:07:10.889 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T10:07:10.889 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=1 2024-04-06T10:07:10.889 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=1 2024-04-06T10:07:10.889 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-06T10:07:10.907 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T10:07:10.907 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T10:07:10.907 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T10:07:10.907 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/image_46 2024-04-06T10:07:11.052 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T10:07:11.052 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror image_46 2024-04-06T10:07:11.052 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:07:11.053 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:07:11.053 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_46 2024-04-06T10:07:11.053 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror image_46 '' 2024-04-06T10:07:11.053 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:07:11.053 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:07:11.053 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_46 2024-04-06T10:07:11.053 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T10:07:11.054 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-image_46.status 2024-04-06T10:07:11.054 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-image_46.status 2024-04-06T10:07:11.054 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T10:07:11.056 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-image_46.status 2024-04-06T10:07:11.056 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_46 2024-04-06T10:07:11.056 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_46.status 2024-04-06T10:07:11.113 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 0 2024-04-06T10:07:11.113 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 0 2024-04-06T10:07:11.113 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T10:07:11.113 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=1, tag_tid=3, entry_tid=1], [object_number=0, tag_tid=3, entry_tid=0], [object_number=3, tag_tid=1, entry_tid=4091], [object_number=2, tag_tid=1, entry_tid=4090]]], state=connected] 2024-04-06T10:07:11.113 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=0, tag_tid=1, entry_tid=4092], [object_number=3, tag_tid=1, entry_tid=4091], [object_number=2, tag_tid=1, entry_tid=4090], [object_number=1, tag_tid=1, entry_tid=4089]]], state=connected] 2024-04-06T10:07:11.117 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_46.status 2024-04-06T10:07:11.118 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=1, tag_tid=3, entry_tid=1' 2024-04-06T10:07:11.119 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror image_46 2024-04-06T10:07:11.119 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:07:11.119 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:07:11.119 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_46 2024-04-06T10:07:11.119 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror image_46 '..*' 2024-04-06T10:07:11.119 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:07:11.119 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:07:11.119 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_46 2024-04-06T10:07:11.119 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T10:07:11.119 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-image_46.status 2024-04-06T10:07:11.120 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-image_46.status 2024-04-06T10:07:11.120 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T10:07:11.121 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-image_46.status 2024-04-06T10:07:11.121 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_46 2024-04-06T10:07:11.121 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_46.status 2024-04-06T10:07:11.166 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 0 2024-04-06T10:07:11.166 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 0 2024-04-06T10:07:11.166 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T10:07:11.166 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=1, tag_tid=3, entry_tid=1], [object_number=0, tag_tid=3, entry_tid=0], [object_number=3, tag_tid=1, entry_tid=4091], [object_number=2, tag_tid=1, entry_tid=4090]]], state=connected] 2024-04-06T10:07:11.166 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=0, tag_tid=1, entry_tid=4092], [object_number=3, tag_tid=1, entry_tid=4091], [object_number=2, tag_tid=1, entry_tid=4090], [object_number=1, tag_tid=1, entry_tid=4089]]], state=connected] 2024-04-06T10:07:11.170 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_46.status 2024-04-06T10:07:11.171 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=0, tag_tid=1, entry_tid=4092' 2024-04-06T10:07:11.172 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=1, tag_tid=3, entry_tid=1' -a 'object_number=1, tag_tid=3, entry_tid=1' = 'object_number=0, tag_tid=1, entry_tid=4092' 2024-04-06T10:07:11.172 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=0, tag_tid=1, entry_tid=4092' '!=' 'object_number=0, tag_tid=1, entry_tid=4092' 2024-04-06T10:07:11.172 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T10:07:11.172 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.4 2024-04-06T10:07:11.574 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror image_46 2024-04-06T10:07:11.574 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T10:07:11.574 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:07:11.574 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_46 2024-04-06T10:07:11.574 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T10:07:11.574 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n image_46 ']' 2024-04-06T10:07:11.574 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/image_46' 2024-04-06T10:07:11.574 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_46 2024-04-06T10:07:11.574 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T10:07:11.574 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T10:07:11.575 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T10:07:11.575 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T10:07:11.575 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T10:07:11.575 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T10:07:11.575 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T10:07:11.575 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T10:07:11.575 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T10:07:11.579 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T10:07:11.579 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_46 2024-04-06T10:07:11.579 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T10:07:11.579 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T10:07:11.579 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T10:07:11.579 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T10:07:11.579 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T10:07:11.580 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T10:07:11.580 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T10:07:11.580 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T10:07:11.580 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T10:07:11.580 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T10:07:11.580 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T10:07:11.581 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:07:11.581 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T10:07:11.581 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T10:07:11.581 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T10:07:11.581 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T10:07:11.581 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T10:07:11.582 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T10:07:11.582 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T10:07:11.582 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T10:07:11.582 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T10:07:11.582 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T10:07:11.582 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:07:11.582 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T10:07:11.582 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T10:07:11.582 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T10:07:11.582 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T10:07:11.583 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T10:07:11.583 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T10:07:11.583 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T10:07:11.606 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T10:07:11.606 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T10:07:11.606 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T10:07:11.606 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_46 2024-04-06T10:07:11.712 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:Invalid command: unused arguments: ['mirror/image_46'] 2024-04-06T10:07:11.713 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:rbd mirror flush : flush rbd mirror 2024-04-06T10:07:11.713 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:admin_socket: invalid command 2024-04-06T10:07:11.715 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T10:07:11.715 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/image_46 2024-04-06T10:07:11.716 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:1 2024-04-06T10:07:11.716 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T10:07:11.716 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T10:07:11.716 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-06T10:07:11.716 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:1 2024-04-06T10:07:11.716 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T10:07:11.716 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T10:07:11.716 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T10:07:11.716 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=1 2024-04-06T10:07:11.716 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-06T10:07:11.716 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T10:07:11.716 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:07:11.716 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=1 2024-04-06T10:07:11.716 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=1 2024-04-06T10:07:11.716 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-06T10:07:11.716 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:1 2024-04-06T10:07:11.716 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T10:07:11.717 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T10:07:11.717 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-06T10:07:11.717 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:1 2024-04-06T10:07:11.717 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T10:07:11.717 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T10:07:11.717 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:07:11.717 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=1 2024-04-06T10:07:11.717 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-06T10:07:11.717 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T10:07:11.717 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T10:07:11.718 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=1 2024-04-06T10:07:11.718 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=1 2024-04-06T10:07:11.718 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-06T10:07:11.741 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T10:07:11.741 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T10:07:11.741 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T10:07:11.741 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/image_46 2024-04-06T10:07:11.888 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T10:07:11.889 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror image_46 2024-04-06T10:07:11.889 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:07:11.889 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:07:11.889 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_46 2024-04-06T10:07:11.889 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror image_46 '' 2024-04-06T10:07:11.889 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:07:11.889 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:07:11.889 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_46 2024-04-06T10:07:11.889 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T10:07:11.890 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-image_46.status 2024-04-06T10:07:11.890 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-image_46.status 2024-04-06T10:07:11.890 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T10:07:11.892 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-image_46.status 2024-04-06T10:07:11.893 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_46 2024-04-06T10:07:11.893 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_46.status 2024-04-06T10:07:11.942 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 0 2024-04-06T10:07:11.942 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 0 2024-04-06T10:07:11.943 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T10:07:11.943 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=1, tag_tid=3, entry_tid=1], [object_number=0, tag_tid=3, entry_tid=0], [object_number=3, tag_tid=1, entry_tid=4091], [object_number=2, tag_tid=1, entry_tid=4090]]], state=connected] 2024-04-06T10:07:11.943 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=1, tag_tid=3, entry_tid=1], [object_number=0, tag_tid=3, entry_tid=0], [object_number=3, tag_tid=1, entry_tid=4091], [object_number=2, tag_tid=1, entry_tid=4090]]], state=connected] 2024-04-06T10:07:11.946 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_46.status 2024-04-06T10:07:11.949 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=1, tag_tid=3, entry_tid=1' 2024-04-06T10:07:11.949 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror image_46 2024-04-06T10:07:11.949 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:07:11.949 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:07:11.949 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_46 2024-04-06T10:07:11.949 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror image_46 '..*' 2024-04-06T10:07:11.949 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:07:11.949 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:07:11.949 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_46 2024-04-06T10:07:11.949 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T10:07:11.950 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-image_46.status 2024-04-06T10:07:11.950 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-image_46.status 2024-04-06T10:07:11.950 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T10:07:11.950 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-image_46.status 2024-04-06T10:07:11.950 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_46 2024-04-06T10:07:11.950 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_46.status 2024-04-06T10:07:11.995 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 0 2024-04-06T10:07:11.995 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 0 2024-04-06T10:07:11.995 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T10:07:11.995 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=1, tag_tid=3, entry_tid=1], [object_number=0, tag_tid=3, entry_tid=0], [object_number=3, tag_tid=1, entry_tid=4091], [object_number=2, tag_tid=1, entry_tid=4090]]], state=connected] 2024-04-06T10:07:11.995 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=1, tag_tid=3, entry_tid=1], [object_number=0, tag_tid=3, entry_tid=0], [object_number=3, tag_tid=1, entry_tid=4091], [object_number=2, tag_tid=1, entry_tid=4090]]], state=connected] 2024-04-06T10:07:11.999 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_46.status 2024-04-06T10:07:12.001 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=1, tag_tid=3, entry_tid=1' 2024-04-06T10:07:12.001 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=1, tag_tid=3, entry_tid=1' -a 'object_number=1, tag_tid=3, entry_tid=1' = 'object_number=1, tag_tid=3, entry_tid=1' 2024-04-06T10:07:12.001 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T10:07:12.001 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_snap_present cluster1 mirror image_46 snap 2024-04-06T10:07:12.001 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T10:07:12.001 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:07:12.001 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_46 2024-04-06T10:07:12.001 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local snap_name=snap 2024-04-06T10:07:12.001 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local s 2024-04-06T10:07:12.001 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-04-06T10:07:12.002 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 1 2024-04-06T10:07:13.002 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 info mirror/image_46@snap 2024-04-06T10:07:13.067 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:rbd image 'image_46': 2024-04-06T10:07:13.067 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: size 128 MiB in 32 objects 2024-04-06T10:07:13.067 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: order 22 (4 MiB objects) 2024-04-06T10:07:13.067 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: snapshot_count: 1 2024-04-06T10:07:13.067 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: id: 10ace1cfee90 2024-04-06T10:07:13.067 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: block_name_prefix: rbd_data.10ace1cfee90 2024-04-06T10:07:13.068 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: format: 2 2024-04-06T10:07:13.068 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: features: layering, exclusive-lock, journaling 2024-04-06T10:07:13.068 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: op_features: 2024-04-06T10:07:13.068 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: flags: 2024-04-06T10:07:13.068 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: create_timestamp: Sat Apr 6 09:57:33 2024 2024-04-06T10:07:13.068 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: access_timestamp: Sat Apr 6 09:57:33 2024 2024-04-06T10:07:13.068 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: modify_timestamp: Sat Apr 6 09:58:34 2024 2024-04-06T10:07:13.068 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: protected: False 2024-04-06T10:07:13.068 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: journal: 10ace1cfee90 2024-04-06T10:07:13.068 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: mirroring state: enabled 2024-04-06T10:07:13.068 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: mirroring mode: journal 2024-04-06T10:07:13.068 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: mirroring global id: 19262d81-00d0-41da-954e-0f9f943b4130 2024-04-06T10:07:13.068 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: mirroring primary: false 2024-04-06T10:07:13.072 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T10:07:13.072 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ compare_image_snaps mirror image_46 snap 2024-04-06T10:07:13.072 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:07:13.072 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_46 2024-04-06T10:07:13.072 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local snap_name=snap 2024-04-06T10:07:13.072 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local ret=0 2024-04-06T10:07:13.072 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local rmt_export=/tmp/tmp.BhW6xKmROU/cluster2-mirror-image_46.export 2024-04-06T10:07:13.072 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loc_export=/tmp/tmp.BhW6xKmROU/cluster1-mirror-image_46.export 2024-04-06T10:07:13.072 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rm -f /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_46.export /tmp/tmp.BhW6xKmROU/cluster1-mirror-image_46.export 2024-04-06T10:07:13.073 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster2 -p mirror export image_46@snap /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_46.export 2024-04-06T10:07:13.236 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-04-06T10:07:13.244 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 -p mirror export image_46@snap /tmp/tmp.BhW6xKmROU/cluster1-mirror-image_46.export 2024-04-06T10:07:13.598 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-04-06T10:07:13.612 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmp /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_46.export /tmp/tmp.BhW6xKmROU/cluster1-mirror-image_46.export 2024-04-06T10:07:13.691 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rm -f /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_46.export /tmp/tmp.BhW6xKmROU/cluster1-mirror-image_46.export 2024-04-06T10:07:13.733 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T10:07:13.733 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-06T10:07:13.734 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ image=image_47 2024-04-06T10:07:13.734 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ create_snap cluster2 mirror image_47 snap 2024-04-06T10:07:13.734 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-04-06T10:07:13.734 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:07:13.734 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_47 2024-04-06T10:07:13.734 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local snap_name=snap 2024-04-06T10:07:13.734 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster2 -p mirror snap create image_47@snap --debug-rbd=20 --debug-journaler=20 2024-04-06T10:07:15.944 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_image_replay_started cluster1 mirror image_47 2024-04-06T10:07:15.944 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T10:07:15.944 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:07:15.944 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_47 2024-04-06T10:07:15.944 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_image_replay_state cluster1 mirror image_47 started 2024-04-06T10:07:15.944 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T10:07:15.945 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:07:15.945 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_47 2024-04-06T10:07:15.945 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state=started 2024-04-06T10:07:15.945 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local s 2024-04-06T10:07:15.945 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T10:07:15.945 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 1 2024-04-06T10:07:16.946 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test_image_replay_state cluster1 mirror image_47 started 2024-04-06T10:07:16.946 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T10:07:16.946 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:07:16.946 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_47 2024-04-06T10:07:16.946 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local test_state=started 2024-04-06T10:07:16.946 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local status_result 2024-04-06T10:07:16.946 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local current_state=stopped 2024-04-06T10:07:16.947 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_47 2024-04-06T10:07:16.947 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_instance=cluster1 2024-04-06T10:07:16.947 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ shift 2024-04-06T10:07:16.947 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -i state 2024-04-06T10:07:16.947 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T10:07:16.947 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance=cluster1 2024-04-06T10:07:16.947 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local loop_instance 2024-04-06T10:07:16.947 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T10:07:16.948 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sleep 0 2024-04-06T10:07:16.948 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-04-06T10:07:16.949 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ seq 0 3 2024-04-06T10:07:16.951 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T10:07:16.952 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_47 2024-04-06T10:07:16.952 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1:0 2024-04-06T10:07:16.952 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ shift 2024-04-06T10:07:16.952 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T10:07:16.952 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-04-06T10:07:16.952 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T10:07:16.952 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=cluster 2024-04-06T10:07:16.952 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T10:07:16.952 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:07:16.952 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T10:07:16.952 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T10:07:16.952 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval cluster=cluster1 2024-04-06T10:07:16.952 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ cluster=cluster1 2024-04-06T10:07:16.953 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T10:07:16.953 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T10:07:16.953 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-04-06T10:07:16.953 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local local_cluster=cluster1:0 2024-04-06T10:07:16.953 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local cluster=cluster1 2024-04-06T10:07:16.953 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local instance 2024-04-06T10:07:16.953 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T10:07:16.953 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local val=cluster1:0 2024-04-06T10:07:16.953 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local cluster_var_name=local_cluster 2024-04-06T10:07:16.953 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local instance_var_name=instance 2024-04-06T10:07:16.953 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ cluster=cluster1 2024-04-06T10:07:16.953 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T10:07:16.953 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ '[' 0 = cluster1:0 ']' 2024-04-06T10:07:16.953 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ eval local_cluster=cluster1 2024-04-06T10:07:16.954 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++++ local_cluster=cluster1 2024-04-06T10:07:16.954 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ eval instance=0 2024-04-06T10:07:16.954 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++++ instance=0 2024-04-06T10:07:16.954 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T10:07:16.974 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T10:07:16.975 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T10:07:16.975 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T10:07:16.975 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_47 2024-04-06T10:07:17.088 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:Invalid command: unused arguments: ['mirror/image_47'] 2024-04-06T10:07:17.088 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:rbd mirror status : get status for rbd mirror 2024-04-06T10:07:17.088 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:admin_socket: invalid command 2024-04-06T10:07:17.090 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T10:07:17.090 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_47 2024-04-06T10:07:17.090 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1:1 2024-04-06T10:07:17.090 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ shift 2024-04-06T10:07:17.090 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T10:07:17.090 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-04-06T10:07:17.090 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:1 2024-04-06T10:07:17.090 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=cluster 2024-04-06T10:07:17.090 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T10:07:17.090 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:07:17.091 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=1 2024-04-06T10:07:17.091 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-06T10:07:17.091 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval cluster=cluster1 2024-04-06T10:07:17.091 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ cluster=cluster1 2024-04-06T10:07:17.091 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=1 2024-04-06T10:07:17.091 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=1 2024-04-06T10:07:17.092 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-04-06T10:07:17.092 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local local_cluster=cluster1:1 2024-04-06T10:07:17.092 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local cluster=cluster1 2024-04-06T10:07:17.093 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local instance 2024-04-06T10:07:17.093 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-06T10:07:17.093 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local val=cluster1:1 2024-04-06T10:07:17.093 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local cluster_var_name=local_cluster 2024-04-06T10:07:17.093 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local instance_var_name=instance 2024-04-06T10:07:17.093 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ cluster=cluster1 2024-04-06T10:07:17.093 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=1 2024-04-06T10:07:17.093 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ '[' 1 = cluster1:1 ']' 2024-04-06T10:07:17.093 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ eval local_cluster=cluster1 2024-04-06T10:07:17.093 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++++ local_cluster=cluster1 2024-04-06T10:07:17.093 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ eval instance=1 2024-04-06T10:07:17.093 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++++ instance=1 2024-04-06T10:07:17.093 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-06T10:07:17.114 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T10:07:17.115 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T10:07:17.115 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T10:07:17.115 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_47 2024-04-06T10:07:17.280 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ return 0 2024-04-06T10:07:17.281 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ status_result=' "state": "Replaying"' 2024-04-06T10:07:17.281 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ echo ' "state": "Replaying"' 2024-04-06T10:07:17.282 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep -i Replaying 2024-04-06T10:07:17.284 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: "state": "Replaying" 2024-04-06T10:07:17.285 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ current_state=started 2024-04-06T10:07:17.285 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test started = started 2024-04-06T10:07:17.285 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T10:07:17.285 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_47 2024-04-06T10:07:17.285 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local local_cluster=cluster1 2024-04-06T10:07:17.285 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-04-06T10:07:17.286 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:07:17.286 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_47 2024-04-06T10:07:17.286 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' journal = journal ']' 2024-04-06T10:07:17.286 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_journal_replay_complete cluster1 cluster2 mirror image_47 2024-04-06T10:07:17.286 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local local_cluster=cluster1 2024-04-06T10:07:17.286 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-04-06T10:07:17.286 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:07:17.286 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_47 2024-04-06T10:07:17.286 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local s master_pos mirror_pos last_mirror_pos 2024-04-06T10:07:17.286 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local master_tag master_entry mirror_tag mirror_entry 2024-04-06T10:07:17.286 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T10:07:17.286 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T10:07:17.286 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T10:07:17.487 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror image_47 2024-04-06T10:07:17.487 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T10:07:17.487 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:07:17.487 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_47 2024-04-06T10:07:17.487 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T10:07:17.487 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n image_47 ']' 2024-04-06T10:07:17.487 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/image_47' 2024-04-06T10:07:17.487 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_47 2024-04-06T10:07:17.487 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T10:07:17.487 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T10:07:17.487 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T10:07:17.488 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T10:07:17.488 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T10:07:17.488 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T10:07:17.488 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T10:07:17.488 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T10:07:17.488 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T10:07:17.491 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T10:07:17.491 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_47 2024-04-06T10:07:17.491 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T10:07:17.491 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T10:07:17.491 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T10:07:17.491 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T10:07:17.491 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T10:07:17.491 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T10:07:17.491 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T10:07:17.491 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T10:07:17.491 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T10:07:17.492 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T10:07:17.492 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T10:07:17.492 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:07:17.492 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T10:07:17.492 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T10:07:17.492 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T10:07:17.492 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T10:07:17.492 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T10:07:17.492 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T10:07:17.492 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T10:07:17.492 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T10:07:17.492 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T10:07:17.492 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T10:07:17.492 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:07:17.492 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T10:07:17.493 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T10:07:17.493 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T10:07:17.493 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T10:07:17.493 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T10:07:17.493 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T10:07:17.493 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T10:07:17.518 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T10:07:17.518 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T10:07:17.518 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T10:07:17.518 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_47 2024-04-06T10:07:17.623 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:Invalid command: unused arguments: ['mirror/image_47'] 2024-04-06T10:07:17.624 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:rbd mirror flush : flush rbd mirror 2024-04-06T10:07:17.624 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:admin_socket: invalid command 2024-04-06T10:07:17.626 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T10:07:17.626 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/image_47 2024-04-06T10:07:17.626 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:1 2024-04-06T10:07:17.626 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T10:07:17.626 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T10:07:17.626 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-06T10:07:17.627 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:1 2024-04-06T10:07:17.627 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T10:07:17.627 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T10:07:17.627 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T10:07:17.627 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=1 2024-04-06T10:07:17.627 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-06T10:07:17.627 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T10:07:17.627 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:07:17.627 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=1 2024-04-06T10:07:17.627 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=1 2024-04-06T10:07:17.627 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-06T10:07:17.627 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:1 2024-04-06T10:07:17.627 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T10:07:17.627 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T10:07:17.627 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-06T10:07:17.628 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:1 2024-04-06T10:07:17.628 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T10:07:17.628 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T10:07:17.628 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:07:17.628 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=1 2024-04-06T10:07:17.628 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-06T10:07:17.628 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T10:07:17.628 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T10:07:17.628 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=1 2024-04-06T10:07:17.628 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=1 2024-04-06T10:07:17.628 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-06T10:07:17.652 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T10:07:17.652 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T10:07:17.652 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T10:07:17.652 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/image_47 2024-04-06T10:07:17.792 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T10:07:17.793 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror image_47 2024-04-06T10:07:17.793 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:07:17.793 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:07:17.793 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_47 2024-04-06T10:07:17.793 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror image_47 '' 2024-04-06T10:07:17.793 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:07:17.793 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:07:17.793 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_47 2024-04-06T10:07:17.793 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T10:07:17.794 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-image_47.status 2024-04-06T10:07:17.794 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-image_47.status 2024-04-06T10:07:17.794 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T10:07:17.796 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-image_47.status 2024-04-06T10:07:17.797 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_47 2024-04-06T10:07:17.797 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_47.status 2024-04-06T10:07:17.847 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 2 2024-04-06T10:07:17.847 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 2 2024-04-06T10:07:17.847 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T10:07:17.847 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=9, tag_tid=2, entry_tid=1], [object_number=8, tag_tid=2, entry_tid=0], [object_number=11, tag_tid=1, entry_tid=8711], [object_number=10, tag_tid=1, entry_tid=8710]]], state=connected] 2024-04-06T10:07:17.847 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=11, tag_tid=1, entry_tid=8711], [object_number=10, tag_tid=1, entry_tid=8710], [object_number=9, tag_tid=1, entry_tid=8709], [object_number=8, tag_tid=1, entry_tid=8708]]], state=connected] 2024-04-06T10:07:17.850 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_47.status 2024-04-06T10:07:17.852 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=9, tag_tid=2, entry_tid=1' 2024-04-06T10:07:17.853 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror image_47 2024-04-06T10:07:17.853 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:07:17.853 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:07:17.853 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_47 2024-04-06T10:07:17.853 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror image_47 '..*' 2024-04-06T10:07:17.853 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:07:17.853 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:07:17.853 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_47 2024-04-06T10:07:17.853 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T10:07:17.853 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-image_47.status 2024-04-06T10:07:17.853 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-image_47.status 2024-04-06T10:07:17.853 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T10:07:17.854 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-image_47.status 2024-04-06T10:07:17.854 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_47 2024-04-06T10:07:17.854 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_47.status 2024-04-06T10:07:17.899 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 2 2024-04-06T10:07:17.899 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 2 2024-04-06T10:07:17.899 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T10:07:17.899 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=9, tag_tid=2, entry_tid=1], [object_number=8, tag_tid=2, entry_tid=0], [object_number=11, tag_tid=1, entry_tid=8711], [object_number=10, tag_tid=1, entry_tid=8710]]], state=connected] 2024-04-06T10:07:17.900 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=11, tag_tid=1, entry_tid=8711], [object_number=10, tag_tid=1, entry_tid=8710], [object_number=9, tag_tid=1, entry_tid=8709], [object_number=8, tag_tid=1, entry_tid=8708]]], state=connected] 2024-04-06T10:07:17.903 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_47.status 2024-04-06T10:07:17.905 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=11, tag_tid=1, entry_tid=8711' 2024-04-06T10:07:17.905 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=9, tag_tid=2, entry_tid=1' -a 'object_number=9, tag_tid=2, entry_tid=1' = 'object_number=11, tag_tid=1, entry_tid=8711' 2024-04-06T10:07:17.905 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=11, tag_tid=1, entry_tid=8711' '!=' '' 2024-04-06T10:07:17.905 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T10:07:17.905 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=11, tag_tid=1, entry_tid=8711' = '' 2024-04-06T10:07:17.905 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=11, tag_tid=1, entry_tid=8711' 2024-04-06T10:07:17.906 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=9, tag_tid=2, entry_tid=1' 2024-04-06T10:07:17.906 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T10:07:17.906 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T10:07:17.906 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=2 2024-04-06T10:07:17.907 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=11, tag_tid=1, entry_tid=8711' 2024-04-06T10:07:17.907 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T10:07:17.907 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T10:07:17.908 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=1 2024-04-06T10:07:17.909 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=9, tag_tid=2, entry_tid=1' 2024-04-06T10:07:17.909 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T10:07:17.909 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T10:07:17.910 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T10:07:17.910 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=11, tag_tid=1, entry_tid=8711' 2024-04-06T10:07:17.911 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T10:07:17.911 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T10:07:17.912 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=8711 2024-04-06T10:07:17.912 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 2 = 1 -a 1 -le 8711 2024-04-06T10:07:17.912 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T10:07:17.912 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T10:07:17.912 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T10:07:18.114 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror image_47 2024-04-06T10:07:18.114 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T10:07:18.114 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:07:18.114 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_47 2024-04-06T10:07:18.115 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T10:07:18.115 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n image_47 ']' 2024-04-06T10:07:18.115 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/image_47' 2024-04-06T10:07:18.115 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_47 2024-04-06T10:07:18.115 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T10:07:18.115 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T10:07:18.115 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T10:07:18.115 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T10:07:18.115 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T10:07:18.115 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T10:07:18.115 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T10:07:18.115 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T10:07:18.116 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T10:07:18.119 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T10:07:18.120 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_47 2024-04-06T10:07:18.120 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T10:07:18.120 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T10:07:18.120 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T10:07:18.120 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T10:07:18.120 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T10:07:18.120 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T10:07:18.120 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T10:07:18.120 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T10:07:18.120 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T10:07:18.120 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T10:07:18.120 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T10:07:18.121 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:07:18.121 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T10:07:18.121 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T10:07:18.121 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T10:07:18.121 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T10:07:18.121 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T10:07:18.121 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T10:07:18.121 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T10:07:18.121 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T10:07:18.121 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T10:07:18.121 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T10:07:18.121 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:07:18.121 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T10:07:18.121 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T10:07:18.121 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T10:07:18.122 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T10:07:18.122 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T10:07:18.122 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T10:07:18.122 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T10:07:18.149 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T10:07:18.149 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T10:07:18.149 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T10:07:18.150 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_47 2024-04-06T10:07:18.255 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:Invalid command: unused arguments: ['mirror/image_47'] 2024-04-06T10:07:18.255 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:rbd mirror flush : flush rbd mirror 2024-04-06T10:07:18.256 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:admin_socket: invalid command 2024-04-06T10:07:18.258 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T10:07:18.258 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/image_47 2024-04-06T10:07:18.258 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:1 2024-04-06T10:07:18.258 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T10:07:18.258 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T10:07:18.258 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-06T10:07:18.258 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:1 2024-04-06T10:07:18.258 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T10:07:18.258 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T10:07:18.259 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T10:07:18.259 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=1 2024-04-06T10:07:18.259 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-06T10:07:18.259 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T10:07:18.259 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:07:18.259 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=1 2024-04-06T10:07:18.259 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=1 2024-04-06T10:07:18.260 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-06T10:07:18.260 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:1 2024-04-06T10:07:18.260 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T10:07:18.260 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T10:07:18.260 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-06T10:07:18.260 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:1 2024-04-06T10:07:18.260 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T10:07:18.260 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T10:07:18.260 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:07:18.260 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=1 2024-04-06T10:07:18.260 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-06T10:07:18.260 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T10:07:18.260 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T10:07:18.260 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=1 2024-04-06T10:07:18.261 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=1 2024-04-06T10:07:18.261 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-06T10:07:18.283 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T10:07:18.284 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T10:07:18.284 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T10:07:18.284 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/image_47 2024-04-06T10:07:18.437 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T10:07:18.438 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror image_47 2024-04-06T10:07:18.438 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:07:18.439 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:07:18.439 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_47 2024-04-06T10:07:18.439 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror image_47 '' 2024-04-06T10:07:18.439 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:07:18.439 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:07:18.439 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_47 2024-04-06T10:07:18.439 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T10:07:18.440 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-image_47.status 2024-04-06T10:07:18.440 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-image_47.status 2024-04-06T10:07:18.440 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T10:07:18.442 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-image_47.status 2024-04-06T10:07:18.442 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_47 2024-04-06T10:07:18.442 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_47.status 2024-04-06T10:07:18.489 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 2 2024-04-06T10:07:18.489 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 2 2024-04-06T10:07:18.489 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T10:07:18.489 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=9, tag_tid=2, entry_tid=1], [object_number=8, tag_tid=2, entry_tid=0], [object_number=11, tag_tid=1, entry_tid=8711], [object_number=10, tag_tid=1, entry_tid=8710]]], state=connected] 2024-04-06T10:07:18.489 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=9, tag_tid=2, entry_tid=1], [object_number=8, tag_tid=2, entry_tid=0], [object_number=11, tag_tid=1, entry_tid=8711], [object_number=10, tag_tid=1, entry_tid=8710]]], state=connected] 2024-04-06T10:07:18.492 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_47.status 2024-04-06T10:07:18.494 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=9, tag_tid=2, entry_tid=1' 2024-04-06T10:07:18.495 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror image_47 2024-04-06T10:07:18.495 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:07:18.495 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:07:18.495 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_47 2024-04-06T10:07:18.495 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror image_47 '..*' 2024-04-06T10:07:18.495 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:07:18.495 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:07:18.495 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_47 2024-04-06T10:07:18.496 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T10:07:18.496 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-image_47.status 2024-04-06T10:07:18.496 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-image_47.status 2024-04-06T10:07:18.496 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T10:07:18.496 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-image_47.status 2024-04-06T10:07:18.496 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_47 2024-04-06T10:07:18.497 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_47.status 2024-04-06T10:07:18.541 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 2 2024-04-06T10:07:18.541 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 2 2024-04-06T10:07:18.541 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T10:07:18.541 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=9, tag_tid=2, entry_tid=1], [object_number=8, tag_tid=2, entry_tid=0], [object_number=11, tag_tid=1, entry_tid=8711], [object_number=10, tag_tid=1, entry_tid=8710]]], state=connected] 2024-04-06T10:07:18.541 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=9, tag_tid=2, entry_tid=1], [object_number=8, tag_tid=2, entry_tid=0], [object_number=11, tag_tid=1, entry_tid=8711], [object_number=10, tag_tid=1, entry_tid=8710]]], state=connected] 2024-04-06T10:07:18.544 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_47.status 2024-04-06T10:07:18.546 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=9, tag_tid=2, entry_tid=1' 2024-04-06T10:07:18.546 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=9, tag_tid=2, entry_tid=1' -a 'object_number=9, tag_tid=2, entry_tid=1' = 'object_number=9, tag_tid=2, entry_tid=1' 2024-04-06T10:07:18.547 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T10:07:18.547 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_snap_present cluster1 mirror image_47 snap 2024-04-06T10:07:18.547 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T10:07:18.547 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:07:18.547 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_47 2024-04-06T10:07:18.547 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local snap_name=snap 2024-04-06T10:07:18.547 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local s 2024-04-06T10:07:18.547 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-04-06T10:07:18.547 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 1 2024-04-06T10:07:19.547 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 info mirror/image_47@snap 2024-04-06T10:07:19.610 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:rbd image 'image_47': 2024-04-06T10:07:19.610 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: size 128 MiB in 32 objects 2024-04-06T10:07:19.610 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: order 22 (4 MiB objects) 2024-04-06T10:07:19.610 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: snapshot_count: 1 2024-04-06T10:07:19.610 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: id: 10acb60d08 2024-04-06T10:07:19.610 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: block_name_prefix: rbd_data.10acb60d08 2024-04-06T10:07:19.610 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: format: 2 2024-04-06T10:07:19.610 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: features: layering, exclusive-lock, journaling 2024-04-06T10:07:19.611 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: op_features: 2024-04-06T10:07:19.611 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: flags: 2024-04-06T10:07:19.611 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: create_timestamp: Sat Apr 6 09:57:40 2024 2024-04-06T10:07:19.611 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: access_timestamp: Sat Apr 6 09:57:40 2024 2024-04-06T10:07:19.611 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: modify_timestamp: Sat Apr 6 09:58:41 2024 2024-04-06T10:07:19.611 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: protected: False 2024-04-06T10:07:19.611 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: journal: 10acb60d08 2024-04-06T10:07:19.611 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: mirroring state: enabled 2024-04-06T10:07:19.611 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: mirroring mode: journal 2024-04-06T10:07:19.611 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: mirroring global id: 1e325db5-afb7-48d0-99a5-9b2d3cbd51dd 2024-04-06T10:07:19.612 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: mirroring primary: false 2024-04-06T10:07:19.615 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T10:07:19.615 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ compare_image_snaps mirror image_47 snap 2024-04-06T10:07:19.615 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:07:19.615 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_47 2024-04-06T10:07:19.615 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local snap_name=snap 2024-04-06T10:07:19.615 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local ret=0 2024-04-06T10:07:19.616 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local rmt_export=/tmp/tmp.BhW6xKmROU/cluster2-mirror-image_47.export 2024-04-06T10:07:19.616 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loc_export=/tmp/tmp.BhW6xKmROU/cluster1-mirror-image_47.export 2024-04-06T10:07:19.616 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rm -f /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_47.export /tmp/tmp.BhW6xKmROU/cluster1-mirror-image_47.export 2024-04-06T10:07:19.616 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster2 -p mirror export image_47@snap /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_47.export 2024-04-06T10:07:19.785 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-04-06T10:07:19.794 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 -p mirror export image_47@snap /tmp/tmp.BhW6xKmROU/cluster1-mirror-image_47.export 2024-04-06T10:07:20.197 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-04-06T10:07:20.212 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmp /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_47.export /tmp/tmp.BhW6xKmROU/cluster1-mirror-image_47.export 2024-04-06T10:07:20.292 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rm -f /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_47.export /tmp/tmp.BhW6xKmROU/cluster1-mirror-image_47.export 2024-04-06T10:07:20.334 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T10:07:20.334 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-06T10:07:20.334 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ image=image_48 2024-04-06T10:07:20.334 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ create_snap cluster2 mirror image_48 snap 2024-04-06T10:07:20.334 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-04-06T10:07:20.334 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:07:20.334 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_48 2024-04-06T10:07:20.335 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local snap_name=snap 2024-04-06T10:07:20.335 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster2 -p mirror snap create image_48@snap --debug-rbd=20 --debug-journaler=20 2024-04-06T10:07:20.946 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_image_replay_started cluster1 mirror image_48 2024-04-06T10:07:20.946 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T10:07:20.946 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:07:20.946 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_48 2024-04-06T10:07:20.946 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_image_replay_state cluster1 mirror image_48 started 2024-04-06T10:07:20.946 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T10:07:20.946 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:07:20.946 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_48 2024-04-06T10:07:20.947 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state=started 2024-04-06T10:07:20.947 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local s 2024-04-06T10:07:20.947 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T10:07:20.947 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 1 2024-04-06T10:07:21.947 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test_image_replay_state cluster1 mirror image_48 started 2024-04-06T10:07:21.948 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T10:07:21.948 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:07:21.948 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_48 2024-04-06T10:07:21.948 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local test_state=started 2024-04-06T10:07:21.948 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local status_result 2024-04-06T10:07:21.948 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local current_state=stopped 2024-04-06T10:07:21.949 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_48 2024-04-06T10:07:21.949 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -i state 2024-04-06T10:07:21.949 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_instance=cluster1 2024-04-06T10:07:21.949 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ shift 2024-04-06T10:07:21.949 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T10:07:21.949 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance=cluster1 2024-04-06T10:07:21.949 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local loop_instance 2024-04-06T10:07:21.949 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T10:07:21.950 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sleep 0 2024-04-06T10:07:21.950 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-04-06T10:07:21.951 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ seq 0 3 2024-04-06T10:07:21.953 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T10:07:21.953 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_48 2024-04-06T10:07:21.953 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1:0 2024-04-06T10:07:21.953 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ shift 2024-04-06T10:07:21.953 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T10:07:21.953 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-04-06T10:07:21.953 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T10:07:21.953 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=cluster 2024-04-06T10:07:21.954 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T10:07:21.954 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:07:21.954 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T10:07:21.954 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T10:07:21.954 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval cluster=cluster1 2024-04-06T10:07:21.954 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ cluster=cluster1 2024-04-06T10:07:21.954 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T10:07:21.954 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T10:07:21.954 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-04-06T10:07:21.954 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local local_cluster=cluster1:0 2024-04-06T10:07:21.954 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local cluster=cluster1 2024-04-06T10:07:21.954 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local instance 2024-04-06T10:07:21.954 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T10:07:21.955 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local val=cluster1:0 2024-04-06T10:07:21.955 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local cluster_var_name=local_cluster 2024-04-06T10:07:21.955 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local instance_var_name=instance 2024-04-06T10:07:21.955 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ cluster=cluster1 2024-04-06T10:07:21.955 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T10:07:21.955 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ '[' 0 = cluster1:0 ']' 2024-04-06T10:07:21.955 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ eval local_cluster=cluster1 2024-04-06T10:07:21.955 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++++ local_cluster=cluster1 2024-04-06T10:07:21.956 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ eval instance=0 2024-04-06T10:07:21.956 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++++ instance=0 2024-04-06T10:07:21.956 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T10:07:21.978 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T10:07:21.979 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T10:07:21.979 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T10:07:21.979 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_48 2024-04-06T10:07:22.084 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:Invalid command: unused arguments: ['mirror/image_48'] 2024-04-06T10:07:22.084 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:rbd mirror status : get status for rbd mirror 2024-04-06T10:07:22.084 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:admin_socket: invalid command 2024-04-06T10:07:22.087 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T10:07:22.088 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_48 2024-04-06T10:07:22.088 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1:1 2024-04-06T10:07:22.088 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ shift 2024-04-06T10:07:22.088 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T10:07:22.088 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-04-06T10:07:22.088 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:1 2024-04-06T10:07:22.088 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=cluster 2024-04-06T10:07:22.088 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T10:07:22.088 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:07:22.088 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=1 2024-04-06T10:07:22.088 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-06T10:07:22.088 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval cluster=cluster1 2024-04-06T10:07:22.088 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ cluster=cluster1 2024-04-06T10:07:22.088 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=1 2024-04-06T10:07:22.089 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=1 2024-04-06T10:07:22.089 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-04-06T10:07:22.089 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local local_cluster=cluster1:1 2024-04-06T10:07:22.089 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local cluster=cluster1 2024-04-06T10:07:22.089 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local instance 2024-04-06T10:07:22.089 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-06T10:07:22.089 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local val=cluster1:1 2024-04-06T10:07:22.089 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local cluster_var_name=local_cluster 2024-04-06T10:07:22.089 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local instance_var_name=instance 2024-04-06T10:07:22.089 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ cluster=cluster1 2024-04-06T10:07:22.089 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=1 2024-04-06T10:07:22.089 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ '[' 1 = cluster1:1 ']' 2024-04-06T10:07:22.089 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ eval local_cluster=cluster1 2024-04-06T10:07:22.089 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++++ local_cluster=cluster1 2024-04-06T10:07:22.089 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ eval instance=1 2024-04-06T10:07:22.089 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++++ instance=1 2024-04-06T10:07:22.090 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-06T10:07:22.107 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T10:07:22.107 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T10:07:22.107 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T10:07:22.107 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_48 2024-04-06T10:07:22.263 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ return 0 2024-04-06T10:07:22.263 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ status_result=' "state": "Replaying"' 2024-04-06T10:07:22.265 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ echo ' "state": "Replaying"' 2024-04-06T10:07:22.266 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep -i Replaying 2024-04-06T10:07:22.266 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: "state": "Replaying" 2024-04-06T10:07:22.267 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ current_state=started 2024-04-06T10:07:22.267 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test started = started 2024-04-06T10:07:22.267 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T10:07:22.267 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_48 2024-04-06T10:07:22.268 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local local_cluster=cluster1 2024-04-06T10:07:22.268 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-04-06T10:07:22.268 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:07:22.268 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_48 2024-04-06T10:07:22.268 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' journal = journal ']' 2024-04-06T10:07:22.268 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_journal_replay_complete cluster1 cluster2 mirror image_48 2024-04-06T10:07:22.268 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local local_cluster=cluster1 2024-04-06T10:07:22.268 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-04-06T10:07:22.268 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:07:22.268 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_48 2024-04-06T10:07:22.268 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local s master_pos mirror_pos last_mirror_pos 2024-04-06T10:07:22.268 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local master_tag master_entry mirror_tag mirror_entry 2024-04-06T10:07:22.268 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T10:07:22.268 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T10:07:22.268 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T10:07:22.468 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror image_48 2024-04-06T10:07:22.468 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T10:07:22.468 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:07:22.468 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_48 2024-04-06T10:07:22.468 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T10:07:22.469 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n image_48 ']' 2024-04-06T10:07:22.469 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/image_48' 2024-04-06T10:07:22.469 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_48 2024-04-06T10:07:22.469 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T10:07:22.469 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T10:07:22.469 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T10:07:22.469 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T10:07:22.469 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T10:07:22.469 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T10:07:22.469 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T10:07:22.470 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T10:07:22.470 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T10:07:22.476 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T10:07:22.476 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_48 2024-04-06T10:07:22.476 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T10:07:22.476 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T10:07:22.476 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T10:07:22.476 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T10:07:22.477 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T10:07:22.477 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T10:07:22.477 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T10:07:22.477 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T10:07:22.477 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T10:07:22.477 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T10:07:22.477 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T10:07:22.477 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:07:22.477 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T10:07:22.477 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T10:07:22.477 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T10:07:22.477 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T10:07:22.477 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T10:07:22.477 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T10:07:22.477 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T10:07:22.478 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T10:07:22.478 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T10:07:22.478 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T10:07:22.478 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:07:22.478 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T10:07:22.478 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T10:07:22.478 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T10:07:22.478 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T10:07:22.478 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T10:07:22.478 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T10:07:22.478 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T10:07:22.498 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T10:07:22.499 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T10:07:22.499 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T10:07:22.499 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_48 2024-04-06T10:07:22.606 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:Invalid command: unused arguments: ['mirror/image_48'] 2024-04-06T10:07:22.606 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:rbd mirror flush : flush rbd mirror 2024-04-06T10:07:22.606 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:admin_socket: invalid command 2024-04-06T10:07:22.608 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T10:07:22.608 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/image_48 2024-04-06T10:07:22.608 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:1 2024-04-06T10:07:22.608 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T10:07:22.608 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T10:07:22.609 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-06T10:07:22.609 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:1 2024-04-06T10:07:22.609 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T10:07:22.609 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T10:07:22.609 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T10:07:22.609 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=1 2024-04-06T10:07:22.609 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-06T10:07:22.609 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T10:07:22.609 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:07:22.609 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=1 2024-04-06T10:07:22.609 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=1 2024-04-06T10:07:22.611 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-06T10:07:22.611 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:1 2024-04-06T10:07:22.611 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T10:07:22.611 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T10:07:22.611 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-06T10:07:22.611 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:1 2024-04-06T10:07:22.611 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T10:07:22.611 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T10:07:22.612 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:07:22.612 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=1 2024-04-06T10:07:22.612 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-06T10:07:22.612 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T10:07:22.612 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T10:07:22.612 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=1 2024-04-06T10:07:22.612 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=1 2024-04-06T10:07:22.612 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-06T10:07:22.628 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T10:07:22.629 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T10:07:22.629 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T10:07:22.629 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/image_48 2024-04-06T10:07:22.774 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T10:07:22.775 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror image_48 2024-04-06T10:07:22.775 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:07:22.775 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:07:22.775 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_48 2024-04-06T10:07:22.775 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror image_48 '' 2024-04-06T10:07:22.776 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:07:22.776 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:07:22.776 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_48 2024-04-06T10:07:22.776 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T10:07:22.776 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-image_48.status 2024-04-06T10:07:22.776 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-image_48.status 2024-04-06T10:07:22.777 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T10:07:22.779 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-image_48.status 2024-04-06T10:07:22.779 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_48 2024-04-06T10:07:22.779 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_48.status 2024-04-06T10:07:22.825 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 1 2024-04-06T10:07:22.825 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 1 2024-04-06T10:07:22.825 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T10:07:22.825 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=5, tag_tid=3, entry_tid=1], [object_number=4, tag_tid=3, entry_tid=0], [object_number=7, tag_tid=1, entry_tid=5427], [object_number=6, tag_tid=1, entry_tid=5426]]], state=connected] 2024-04-06T10:07:22.825 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=5, tag_tid=2, entry_tid=1], [object_number=4, tag_tid=2, entry_tid=0], [object_number=7, tag_tid=1, entry_tid=5427], [object_number=6, tag_tid=1, entry_tid=5426]]], state=connected] 2024-04-06T10:07:22.829 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_48.status 2024-04-06T10:07:22.830 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=5, tag_tid=3, entry_tid=1' 2024-04-06T10:07:22.831 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror image_48 2024-04-06T10:07:22.832 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:07:22.832 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:07:22.832 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_48 2024-04-06T10:07:22.832 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror image_48 '..*' 2024-04-06T10:07:22.832 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:07:22.832 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:07:22.832 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_48 2024-04-06T10:07:22.832 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T10:07:22.832 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-image_48.status 2024-04-06T10:07:22.832 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-image_48.status 2024-04-06T10:07:22.832 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T10:07:22.833 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-image_48.status 2024-04-06T10:07:22.833 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_48 2024-04-06T10:07:22.833 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_48.status 2024-04-06T10:07:22.878 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 1 2024-04-06T10:07:22.878 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 1 2024-04-06T10:07:22.878 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T10:07:22.878 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=5, tag_tid=3, entry_tid=1], [object_number=4, tag_tid=3, entry_tid=0], [object_number=7, tag_tid=1, entry_tid=5427], [object_number=6, tag_tid=1, entry_tid=5426]]], state=connected] 2024-04-06T10:07:22.878 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=5, tag_tid=2, entry_tid=1], [object_number=4, tag_tid=2, entry_tid=0], [object_number=7, tag_tid=1, entry_tid=5427], [object_number=6, tag_tid=1, entry_tid=5426]]], state=connected] 2024-04-06T10:07:22.881 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_48.status 2024-04-06T10:07:22.883 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=5, tag_tid=2, entry_tid=1' 2024-04-06T10:07:22.883 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=5, tag_tid=3, entry_tid=1' -a 'object_number=5, tag_tid=3, entry_tid=1' = 'object_number=5, tag_tid=2, entry_tid=1' 2024-04-06T10:07:22.883 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=5, tag_tid=2, entry_tid=1' '!=' '' 2024-04-06T10:07:22.883 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T10:07:22.883 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=5, tag_tid=2, entry_tid=1' = '' 2024-04-06T10:07:22.883 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=5, tag_tid=2, entry_tid=1' 2024-04-06T10:07:22.884 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=5, tag_tid=3, entry_tid=1' 2024-04-06T10:07:22.884 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T10:07:22.884 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T10:07:22.886 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=3 2024-04-06T10:07:22.886 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=5, tag_tid=2, entry_tid=1' 2024-04-06T10:07:22.886 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T10:07:22.886 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T10:07:22.887 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=2 2024-04-06T10:07:22.887 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=5, tag_tid=3, entry_tid=1' 2024-04-06T10:07:22.887 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T10:07:22.887 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T10:07:22.888 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T10:07:22.888 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=5, tag_tid=2, entry_tid=1' 2024-04-06T10:07:22.889 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T10:07:22.889 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T10:07:22.890 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=1 2024-04-06T10:07:22.890 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 3 = 2 -a 1 -le 1 2024-04-06T10:07:22.890 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T10:07:22.890 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T10:07:22.890 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T10:07:23.092 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror image_48 2024-04-06T10:07:23.092 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T10:07:23.092 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:07:23.092 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_48 2024-04-06T10:07:23.092 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T10:07:23.093 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n image_48 ']' 2024-04-06T10:07:23.093 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/image_48' 2024-04-06T10:07:23.093 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_48 2024-04-06T10:07:23.093 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T10:07:23.093 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T10:07:23.093 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T10:07:23.093 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T10:07:23.093 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T10:07:23.093 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T10:07:23.093 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T10:07:23.094 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T10:07:23.094 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T10:07:23.096 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T10:07:23.097 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_48 2024-04-06T10:07:23.097 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T10:07:23.097 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T10:07:23.097 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T10:07:23.097 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T10:07:23.097 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T10:07:23.097 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T10:07:23.097 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T10:07:23.097 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T10:07:23.097 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T10:07:23.097 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T10:07:23.097 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T10:07:23.097 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:07:23.097 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T10:07:23.098 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T10:07:23.099 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T10:07:23.099 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T10:07:23.099 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T10:07:23.099 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T10:07:23.099 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T10:07:23.099 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T10:07:23.099 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T10:07:23.099 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T10:07:23.099 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:07:23.099 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T10:07:23.099 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T10:07:23.099 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T10:07:23.099 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T10:07:23.100 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T10:07:23.100 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T10:07:23.100 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T10:07:23.125 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T10:07:23.125 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T10:07:23.125 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T10:07:23.125 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_48 2024-04-06T10:07:23.230 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:Invalid command: unused arguments: ['mirror/image_48'] 2024-04-06T10:07:23.230 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:rbd mirror flush : flush rbd mirror 2024-04-06T10:07:23.231 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:admin_socket: invalid command 2024-04-06T10:07:23.233 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T10:07:23.233 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/image_48 2024-04-06T10:07:23.233 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:1 2024-04-06T10:07:23.233 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T10:07:23.233 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T10:07:23.233 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-06T10:07:23.233 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:1 2024-04-06T10:07:23.233 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T10:07:23.233 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T10:07:23.233 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T10:07:23.233 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=1 2024-04-06T10:07:23.233 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-06T10:07:23.234 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T10:07:23.234 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:07:23.234 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=1 2024-04-06T10:07:23.234 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=1 2024-04-06T10:07:23.236 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-06T10:07:23.236 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:1 2024-04-06T10:07:23.236 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T10:07:23.236 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T10:07:23.236 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-06T10:07:23.236 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:1 2024-04-06T10:07:23.237 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T10:07:23.237 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T10:07:23.237 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:07:23.237 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=1 2024-04-06T10:07:23.237 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-06T10:07:23.237 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T10:07:23.237 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T10:07:23.237 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=1 2024-04-06T10:07:23.237 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=1 2024-04-06T10:07:23.237 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-06T10:07:23.254 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T10:07:23.254 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T10:07:23.254 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T10:07:23.254 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/image_48 2024-04-06T10:07:23.398 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T10:07:23.400 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror image_48 2024-04-06T10:07:23.400 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:07:23.400 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:07:23.400 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_48 2024-04-06T10:07:23.400 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror image_48 '' 2024-04-06T10:07:23.400 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:07:23.400 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:07:23.400 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_48 2024-04-06T10:07:23.400 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T10:07:23.401 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-image_48.status 2024-04-06T10:07:23.401 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-image_48.status 2024-04-06T10:07:23.401 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T10:07:23.402 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-image_48.status 2024-04-06T10:07:23.403 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_48.status 2024-04-06T10:07:23.403 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_48 2024-04-06T10:07:23.449 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 1 2024-04-06T10:07:23.454 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 1 2024-04-06T10:07:23.454 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T10:07:23.454 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=5, tag_tid=3, entry_tid=1], [object_number=4, tag_tid=3, entry_tid=0], [object_number=7, tag_tid=1, entry_tid=5427], [object_number=6, tag_tid=1, entry_tid=5426]]], state=connected] 2024-04-06T10:07:23.454 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=5, tag_tid=2, entry_tid=1], [object_number=4, tag_tid=2, entry_tid=0], [object_number=7, tag_tid=1, entry_tid=5427], [object_number=6, tag_tid=1, entry_tid=5426]]], state=connected] 2024-04-06T10:07:23.455 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_48.status 2024-04-06T10:07:23.455 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=5, tag_tid=3, entry_tid=1' 2024-04-06T10:07:23.456 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror image_48 2024-04-06T10:07:23.456 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:07:23.456 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:07:23.456 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_48 2024-04-06T10:07:23.456 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror image_48 '..*' 2024-04-06T10:07:23.456 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:07:23.456 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:07:23.456 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_48 2024-04-06T10:07:23.456 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T10:07:23.456 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-image_48.status 2024-04-06T10:07:23.456 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-image_48.status 2024-04-06T10:07:23.456 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T10:07:23.457 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-image_48.status 2024-04-06T10:07:23.457 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_48 2024-04-06T10:07:23.457 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_48.status 2024-04-06T10:07:23.502 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 1 2024-04-06T10:07:23.503 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 1 2024-04-06T10:07:23.503 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T10:07:23.503 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=5, tag_tid=3, entry_tid=1], [object_number=4, tag_tid=3, entry_tid=0], [object_number=7, tag_tid=1, entry_tid=5427], [object_number=6, tag_tid=1, entry_tid=5426]]], state=connected] 2024-04-06T10:07:23.503 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=5, tag_tid=2, entry_tid=1], [object_number=4, tag_tid=2, entry_tid=0], [object_number=7, tag_tid=1, entry_tid=5427], [object_number=6, tag_tid=1, entry_tid=5426]]], state=connected] 2024-04-06T10:07:23.506 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_48.status 2024-04-06T10:07:23.508 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=5, tag_tid=2, entry_tid=1' 2024-04-06T10:07:23.508 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=5, tag_tid=3, entry_tid=1' -a 'object_number=5, tag_tid=3, entry_tid=1' = 'object_number=5, tag_tid=2, entry_tid=1' 2024-04-06T10:07:23.508 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=5, tag_tid=2, entry_tid=1' '!=' 'object_number=5, tag_tid=2, entry_tid=1' 2024-04-06T10:07:23.508 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T10:07:23.508 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.4 2024-04-06T10:07:23.911 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror image_48 2024-04-06T10:07:23.911 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T10:07:23.911 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:07:23.911 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_48 2024-04-06T10:07:23.911 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T10:07:23.911 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n image_48 ']' 2024-04-06T10:07:23.911 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/image_48' 2024-04-06T10:07:23.911 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_48 2024-04-06T10:07:23.911 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T10:07:23.911 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T10:07:23.911 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T10:07:23.911 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T10:07:23.912 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T10:07:23.912 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T10:07:23.912 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T10:07:23.912 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T10:07:23.912 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T10:07:23.914 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T10:07:23.914 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_48 2024-04-06T10:07:23.914 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T10:07:23.915 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T10:07:23.915 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T10:07:23.915 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T10:07:23.915 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T10:07:23.915 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T10:07:23.915 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T10:07:23.916 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T10:07:23.916 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T10:07:23.916 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T10:07:23.916 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T10:07:23.916 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:07:23.916 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T10:07:23.917 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T10:07:23.917 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T10:07:23.918 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T10:07:23.918 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T10:07:23.918 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T10:07:23.918 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T10:07:23.918 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T10:07:23.918 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T10:07:23.918 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T10:07:23.918 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:07:23.918 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T10:07:23.918 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T10:07:23.918 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T10:07:23.919 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T10:07:23.919 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T10:07:23.919 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T10:07:23.919 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T10:07:23.941 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T10:07:23.942 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T10:07:23.942 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T10:07:23.942 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_48 2024-04-06T10:07:24.044 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:Invalid command: unused arguments: ['mirror/image_48'] 2024-04-06T10:07:24.044 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:rbd mirror flush : flush rbd mirror 2024-04-06T10:07:24.044 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:admin_socket: invalid command 2024-04-06T10:07:24.047 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T10:07:24.047 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/image_48 2024-04-06T10:07:24.047 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:1 2024-04-06T10:07:24.047 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T10:07:24.047 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T10:07:24.047 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-06T10:07:24.047 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:1 2024-04-06T10:07:24.047 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T10:07:24.047 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T10:07:24.047 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T10:07:24.047 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=1 2024-04-06T10:07:24.047 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-06T10:07:24.048 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T10:07:24.048 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:07:24.048 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=1 2024-04-06T10:07:24.048 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=1 2024-04-06T10:07:24.048 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-06T10:07:24.048 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:1 2024-04-06T10:07:24.048 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T10:07:24.048 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T10:07:24.048 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-06T10:07:24.048 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:1 2024-04-06T10:07:24.048 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T10:07:24.048 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T10:07:24.048 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:07:24.048 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=1 2024-04-06T10:07:24.048 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-06T10:07:24.049 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T10:07:24.049 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T10:07:24.049 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=1 2024-04-06T10:07:24.049 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=1 2024-04-06T10:07:24.049 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-06T10:07:24.073 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T10:07:24.073 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T10:07:24.073 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T10:07:24.073 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/image_48 2024-04-06T10:07:24.215 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T10:07:24.216 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror image_48 2024-04-06T10:07:24.216 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:07:24.216 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:07:24.216 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_48 2024-04-06T10:07:24.216 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror image_48 '' 2024-04-06T10:07:24.216 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:07:24.216 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:07:24.216 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_48 2024-04-06T10:07:24.216 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T10:07:24.217 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-image_48.status 2024-04-06T10:07:24.217 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-image_48.status 2024-04-06T10:07:24.217 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T10:07:24.218 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-image_48.status 2024-04-06T10:07:24.219 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_48 2024-04-06T10:07:24.219 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_48.status 2024-04-06T10:07:24.264 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 1 2024-04-06T10:07:24.264 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 1 2024-04-06T10:07:24.264 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T10:07:24.264 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=5, tag_tid=3, entry_tid=1], [object_number=4, tag_tid=3, entry_tid=0], [object_number=7, tag_tid=1, entry_tid=5427], [object_number=6, tag_tid=1, entry_tid=5426]]], state=connected] 2024-04-06T10:07:24.264 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=5, tag_tid=2, entry_tid=1], [object_number=4, tag_tid=2, entry_tid=0], [object_number=7, tag_tid=1, entry_tid=5427], [object_number=6, tag_tid=1, entry_tid=5426]]], state=connected] 2024-04-06T10:07:24.268 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_48.status 2024-04-06T10:07:24.269 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=5, tag_tid=3, entry_tid=1' 2024-04-06T10:07:24.270 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror image_48 2024-04-06T10:07:24.270 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:07:24.270 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:07:24.270 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_48 2024-04-06T10:07:24.270 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror image_48 '..*' 2024-04-06T10:07:24.271 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:07:24.271 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:07:24.271 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_48 2024-04-06T10:07:24.271 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T10:07:24.271 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-image_48.status 2024-04-06T10:07:24.271 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-image_48.status 2024-04-06T10:07:24.271 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T10:07:24.272 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-image_48.status 2024-04-06T10:07:24.272 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_48 2024-04-06T10:07:24.272 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_48.status 2024-04-06T10:07:24.317 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 1 2024-04-06T10:07:24.317 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 1 2024-04-06T10:07:24.317 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T10:07:24.317 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=5, tag_tid=3, entry_tid=1], [object_number=4, tag_tid=3, entry_tid=0], [object_number=7, tag_tid=1, entry_tid=5427], [object_number=6, tag_tid=1, entry_tid=5426]]], state=connected] 2024-04-06T10:07:24.317 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=5, tag_tid=2, entry_tid=1], [object_number=4, tag_tid=2, entry_tid=0], [object_number=7, tag_tid=1, entry_tid=5427], [object_number=6, tag_tid=1, entry_tid=5426]]], state=connected] 2024-04-06T10:07:24.320 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_48.status 2024-04-06T10:07:24.322 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=5, tag_tid=2, entry_tid=1' 2024-04-06T10:07:24.322 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=5, tag_tid=3, entry_tid=1' -a 'object_number=5, tag_tid=3, entry_tid=1' = 'object_number=5, tag_tid=2, entry_tid=1' 2024-04-06T10:07:24.322 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=5, tag_tid=2, entry_tid=1' '!=' 'object_number=5, tag_tid=2, entry_tid=1' 2024-04-06T10:07:24.322 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T10:07:24.322 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.8 2024-04-06T10:07:25.124 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror image_48 2024-04-06T10:07:25.125 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T10:07:25.125 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:07:25.125 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_48 2024-04-06T10:07:25.125 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T10:07:25.125 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n image_48 ']' 2024-04-06T10:07:25.125 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/image_48' 2024-04-06T10:07:25.125 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_48 2024-04-06T10:07:25.125 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T10:07:25.125 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T10:07:25.125 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T10:07:25.125 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T10:07:25.125 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T10:07:25.125 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T10:07:25.126 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T10:07:25.126 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T10:07:25.126 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T10:07:25.130 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T10:07:25.130 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_48 2024-04-06T10:07:25.130 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T10:07:25.130 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T10:07:25.130 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T10:07:25.130 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T10:07:25.130 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T10:07:25.131 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T10:07:25.131 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T10:07:25.131 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T10:07:25.131 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T10:07:25.131 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T10:07:25.131 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T10:07:25.131 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:07:25.131 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T10:07:25.131 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T10:07:25.131 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T10:07:25.131 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T10:07:25.131 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T10:07:25.131 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T10:07:25.131 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T10:07:25.131 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T10:07:25.132 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T10:07:25.132 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T10:07:25.132 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:07:25.132 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T10:07:25.132 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T10:07:25.132 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T10:07:25.132 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T10:07:25.132 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T10:07:25.132 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T10:07:25.132 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T10:07:25.156 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T10:07:25.156 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T10:07:25.157 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T10:07:25.157 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_48 2024-04-06T10:07:25.268 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:Invalid command: unused arguments: ['mirror/image_48'] 2024-04-06T10:07:25.268 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:rbd mirror flush : flush rbd mirror 2024-04-06T10:07:25.268 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:admin_socket: invalid command 2024-04-06T10:07:25.271 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T10:07:25.271 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/image_48 2024-04-06T10:07:25.271 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:1 2024-04-06T10:07:25.271 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T10:07:25.271 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T10:07:25.271 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-06T10:07:25.271 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:1 2024-04-06T10:07:25.271 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T10:07:25.271 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T10:07:25.271 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T10:07:25.271 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=1 2024-04-06T10:07:25.271 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-06T10:07:25.272 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T10:07:25.272 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:07:25.272 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=1 2024-04-06T10:07:25.272 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=1 2024-04-06T10:07:25.273 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-06T10:07:25.273 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:1 2024-04-06T10:07:25.273 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T10:07:25.273 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T10:07:25.273 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-06T10:07:25.273 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:1 2024-04-06T10:07:25.273 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T10:07:25.273 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T10:07:25.274 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:07:25.274 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=1 2024-04-06T10:07:25.274 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-06T10:07:25.274 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T10:07:25.274 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T10:07:25.274 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=1 2024-04-06T10:07:25.274 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=1 2024-04-06T10:07:25.274 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-06T10:07:25.297 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T10:07:25.297 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T10:07:25.297 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T10:07:25.297 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/image_48 2024-04-06T10:07:25.434 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T10:07:25.435 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror image_48 2024-04-06T10:07:25.435 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:07:25.435 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:07:25.436 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_48 2024-04-06T10:07:25.436 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror image_48 '' 2024-04-06T10:07:25.436 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:07:25.436 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:07:25.436 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_48 2024-04-06T10:07:25.436 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T10:07:25.437 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-image_48.status 2024-04-06T10:07:25.437 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-image_48.status 2024-04-06T10:07:25.437 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T10:07:25.438 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-image_48.status 2024-04-06T10:07:25.439 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_48 2024-04-06T10:07:25.439 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_48.status 2024-04-06T10:07:25.485 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 1 2024-04-06T10:07:25.486 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 1 2024-04-06T10:07:25.486 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T10:07:25.486 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=5, tag_tid=3, entry_tid=1], [object_number=4, tag_tid=3, entry_tid=0], [object_number=7, tag_tid=1, entry_tid=5427], [object_number=6, tag_tid=1, entry_tid=5426]]], state=connected] 2024-04-06T10:07:25.486 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=5, tag_tid=2, entry_tid=1], [object_number=4, tag_tid=2, entry_tid=0], [object_number=7, tag_tid=1, entry_tid=5427], [object_number=6, tag_tid=1, entry_tid=5426]]], state=connected] 2024-04-06T10:07:25.489 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_48.status 2024-04-06T10:07:25.491 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=5, tag_tid=3, entry_tid=1' 2024-04-06T10:07:25.492 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror image_48 2024-04-06T10:07:25.492 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:07:25.492 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:07:25.492 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_48 2024-04-06T10:07:25.492 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror image_48 '..*' 2024-04-06T10:07:25.492 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:07:25.492 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:07:25.492 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_48 2024-04-06T10:07:25.492 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T10:07:25.493 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-image_48.status 2024-04-06T10:07:25.493 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-image_48.status 2024-04-06T10:07:25.493 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T10:07:25.493 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-image_48.status 2024-04-06T10:07:25.493 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_48 2024-04-06T10:07:25.494 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_48.status 2024-04-06T10:07:25.538 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 1 2024-04-06T10:07:25.538 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 1 2024-04-06T10:07:25.538 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T10:07:25.539 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=5, tag_tid=3, entry_tid=1], [object_number=4, tag_tid=3, entry_tid=0], [object_number=7, tag_tid=1, entry_tid=5427], [object_number=6, tag_tid=1, entry_tid=5426]]], state=connected] 2024-04-06T10:07:25.539 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=5, tag_tid=2, entry_tid=1], [object_number=4, tag_tid=2, entry_tid=0], [object_number=7, tag_tid=1, entry_tid=5427], [object_number=6, tag_tid=1, entry_tid=5426]]], state=connected] 2024-04-06T10:07:25.542 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_48.status 2024-04-06T10:07:25.544 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=5, tag_tid=2, entry_tid=1' 2024-04-06T10:07:25.544 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=5, tag_tid=3, entry_tid=1' -a 'object_number=5, tag_tid=3, entry_tid=1' = 'object_number=5, tag_tid=2, entry_tid=1' 2024-04-06T10:07:25.544 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=5, tag_tid=2, entry_tid=1' '!=' 'object_number=5, tag_tid=2, entry_tid=1' 2024-04-06T10:07:25.544 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T10:07:25.544 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 1.6 2024-04-06T10:07:27.146 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror image_48 2024-04-06T10:07:27.146 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T10:07:27.146 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:07:27.146 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_48 2024-04-06T10:07:27.146 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T10:07:27.147 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n image_48 ']' 2024-04-06T10:07:27.147 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/image_48' 2024-04-06T10:07:27.147 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_48 2024-04-06T10:07:27.147 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T10:07:27.147 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T10:07:27.147 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T10:07:27.147 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T10:07:27.147 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T10:07:27.147 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T10:07:27.147 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T10:07:27.148 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T10:07:27.148 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T10:07:27.151 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T10:07:27.151 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_48 2024-04-06T10:07:27.151 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T10:07:27.151 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T10:07:27.151 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T10:07:27.151 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T10:07:27.151 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T10:07:27.151 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T10:07:27.152 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T10:07:27.152 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T10:07:27.152 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T10:07:27.152 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T10:07:27.152 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T10:07:27.152 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:07:27.152 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T10:07:27.152 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T10:07:27.152 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T10:07:27.152 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T10:07:27.153 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T10:07:27.153 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T10:07:27.154 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T10:07:27.154 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T10:07:27.154 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T10:07:27.154 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T10:07:27.154 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:07:27.154 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T10:07:27.154 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T10:07:27.154 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T10:07:27.154 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T10:07:27.154 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T10:07:27.154 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T10:07:27.154 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T10:07:27.177 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T10:07:27.178 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T10:07:27.178 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T10:07:27.178 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_48 2024-04-06T10:07:27.283 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:Invalid command: unused arguments: ['mirror/image_48'] 2024-04-06T10:07:27.283 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:rbd mirror flush : flush rbd mirror 2024-04-06T10:07:27.283 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:admin_socket: invalid command 2024-04-06T10:07:27.286 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T10:07:27.286 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/image_48 2024-04-06T10:07:27.287 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:1 2024-04-06T10:07:27.287 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T10:07:27.287 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T10:07:27.287 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-06T10:07:27.287 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:1 2024-04-06T10:07:27.287 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T10:07:27.287 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T10:07:27.287 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T10:07:27.287 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=1 2024-04-06T10:07:27.287 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-06T10:07:27.287 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T10:07:27.287 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:07:27.287 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=1 2024-04-06T10:07:27.288 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=1 2024-04-06T10:07:27.288 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-06T10:07:27.288 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:1 2024-04-06T10:07:27.288 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T10:07:27.288 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T10:07:27.288 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-06T10:07:27.288 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:1 2024-04-06T10:07:27.288 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T10:07:27.288 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T10:07:27.288 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:07:27.288 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=1 2024-04-06T10:07:27.288 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-06T10:07:27.288 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T10:07:27.288 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T10:07:27.288 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=1 2024-04-06T10:07:27.289 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=1 2024-04-06T10:07:27.289 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-06T10:07:27.311 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T10:07:27.311 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T10:07:27.312 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T10:07:27.312 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/image_48 2024-04-06T10:07:27.449 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T10:07:27.450 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror image_48 2024-04-06T10:07:27.450 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:07:27.450 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:07:27.450 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_48 2024-04-06T10:07:27.450 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror image_48 '' 2024-04-06T10:07:27.450 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:07:27.450 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:07:27.450 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_48 2024-04-06T10:07:27.450 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T10:07:27.450 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-image_48.status 2024-04-06T10:07:27.451 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-image_48.status 2024-04-06T10:07:27.451 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T10:07:27.453 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-image_48.status 2024-04-06T10:07:27.453 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_48 2024-04-06T10:07:27.453 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_48.status 2024-04-06T10:07:27.503 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 1 2024-04-06T10:07:27.504 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 1 2024-04-06T10:07:27.504 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T10:07:27.504 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=5, tag_tid=3, entry_tid=1], [object_number=4, tag_tid=3, entry_tid=0], [object_number=7, tag_tid=1, entry_tid=5427], [object_number=6, tag_tid=1, entry_tid=5426]]], state=connected] 2024-04-06T10:07:27.504 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=5, tag_tid=2, entry_tid=1], [object_number=4, tag_tid=2, entry_tid=0], [object_number=7, tag_tid=1, entry_tid=5427], [object_number=6, tag_tid=1, entry_tid=5426]]], state=connected] 2024-04-06T10:07:27.508 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_48.status 2024-04-06T10:07:27.509 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=5, tag_tid=3, entry_tid=1' 2024-04-06T10:07:27.510 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror image_48 2024-04-06T10:07:27.511 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:07:27.511 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:07:27.511 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_48 2024-04-06T10:07:27.511 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror image_48 '..*' 2024-04-06T10:07:27.511 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:07:27.511 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:07:27.511 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_48 2024-04-06T10:07:27.511 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T10:07:27.511 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-image_48.status 2024-04-06T10:07:27.511 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-image_48.status 2024-04-06T10:07:27.511 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T10:07:27.512 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-image_48.status 2024-04-06T10:07:27.512 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_48 2024-04-06T10:07:27.512 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_48.status 2024-04-06T10:07:27.557 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 1 2024-04-06T10:07:27.557 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 1 2024-04-06T10:07:27.557 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T10:07:27.558 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=5, tag_tid=3, entry_tid=1], [object_number=4, tag_tid=3, entry_tid=0], [object_number=7, tag_tid=1, entry_tid=5427], [object_number=6, tag_tid=1, entry_tid=5426]]], state=connected] 2024-04-06T10:07:27.558 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=5, tag_tid=2, entry_tid=1], [object_number=4, tag_tid=2, entry_tid=0], [object_number=7, tag_tid=1, entry_tid=5427], [object_number=6, tag_tid=1, entry_tid=5426]]], state=connected] 2024-04-06T10:07:27.561 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_48.status 2024-04-06T10:07:27.562 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=5, tag_tid=2, entry_tid=1' 2024-04-06T10:07:27.563 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=5, tag_tid=3, entry_tid=1' -a 'object_number=5, tag_tid=3, entry_tid=1' = 'object_number=5, tag_tid=2, entry_tid=1' 2024-04-06T10:07:27.563 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=5, tag_tid=2, entry_tid=1' '!=' 'object_number=5, tag_tid=2, entry_tid=1' 2024-04-06T10:07:27.563 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T10:07:27.563 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 2 2024-04-06T10:07:29.565 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror image_48 2024-04-06T10:07:29.565 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T10:07:29.565 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:07:29.565 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_48 2024-04-06T10:07:29.565 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T10:07:29.565 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n image_48 ']' 2024-04-06T10:07:29.565 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/image_48' 2024-04-06T10:07:29.565 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_48 2024-04-06T10:07:29.565 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T10:07:29.566 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T10:07:29.566 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T10:07:29.566 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T10:07:29.566 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T10:07:29.566 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T10:07:29.566 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T10:07:29.566 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T10:07:29.567 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T10:07:29.570 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T10:07:29.570 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_48 2024-04-06T10:07:29.570 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T10:07:29.570 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T10:07:29.570 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T10:07:29.570 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T10:07:29.570 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T10:07:29.570 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T10:07:29.570 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T10:07:29.571 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T10:07:29.571 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T10:07:29.571 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T10:07:29.571 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T10:07:29.571 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:07:29.571 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T10:07:29.571 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T10:07:29.571 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T10:07:29.571 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T10:07:29.571 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T10:07:29.571 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T10:07:29.571 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T10:07:29.571 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T10:07:29.571 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T10:07:29.571 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T10:07:29.572 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:07:29.572 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T10:07:29.572 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T10:07:29.572 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T10:07:29.572 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T10:07:29.572 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T10:07:29.572 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T10:07:29.572 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T10:07:29.596 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T10:07:29.596 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T10:07:29.596 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T10:07:29.597 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_48 2024-04-06T10:07:29.701 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:Invalid command: unused arguments: ['mirror/image_48'] 2024-04-06T10:07:29.701 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:rbd mirror flush : flush rbd mirror 2024-04-06T10:07:29.701 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:admin_socket: invalid command 2024-04-06T10:07:29.707 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T10:07:29.707 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/image_48 2024-04-06T10:07:29.707 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:1 2024-04-06T10:07:29.708 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T10:07:29.708 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T10:07:29.708 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-06T10:07:29.708 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:1 2024-04-06T10:07:29.708 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T10:07:29.708 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T10:07:29.708 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T10:07:29.708 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=1 2024-04-06T10:07:29.708 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-06T10:07:29.708 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T10:07:29.708 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:07:29.709 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=1 2024-04-06T10:07:29.709 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=1 2024-04-06T10:07:29.709 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-06T10:07:29.709 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:1 2024-04-06T10:07:29.709 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T10:07:29.709 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T10:07:29.709 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-06T10:07:29.709 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:1 2024-04-06T10:07:29.709 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T10:07:29.709 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T10:07:29.709 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:07:29.709 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=1 2024-04-06T10:07:29.709 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-06T10:07:29.709 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T10:07:29.709 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T10:07:29.710 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=1 2024-04-06T10:07:29.710 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=1 2024-04-06T10:07:29.710 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-06T10:07:29.729 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T10:07:29.729 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T10:07:29.729 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T10:07:29.729 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/image_48 2024-04-06T10:07:29.866 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T10:07:29.867 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror image_48 2024-04-06T10:07:29.867 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:07:29.867 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:07:29.867 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_48 2024-04-06T10:07:29.867 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror image_48 '' 2024-04-06T10:07:29.867 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:07:29.867 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:07:29.867 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_48 2024-04-06T10:07:29.867 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T10:07:29.868 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-image_48.status 2024-04-06T10:07:29.868 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-image_48.status 2024-04-06T10:07:29.868 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T10:07:29.869 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-image_48.status 2024-04-06T10:07:29.870 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_48 2024-04-06T10:07:29.870 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_48.status 2024-04-06T10:07:29.922 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 1 2024-04-06T10:07:29.922 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 1 2024-04-06T10:07:29.922 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T10:07:29.923 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=5, tag_tid=3, entry_tid=1], [object_number=4, tag_tid=3, entry_tid=0], [object_number=7, tag_tid=1, entry_tid=5427], [object_number=6, tag_tid=1, entry_tid=5426]]], state=connected] 2024-04-06T10:07:29.923 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=5, tag_tid=2, entry_tid=1], [object_number=4, tag_tid=2, entry_tid=0], [object_number=7, tag_tid=1, entry_tid=5427], [object_number=6, tag_tid=1, entry_tid=5426]]], state=connected] 2024-04-06T10:07:29.926 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_48.status 2024-04-06T10:07:29.927 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=5, tag_tid=3, entry_tid=1' 2024-04-06T10:07:29.929 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror image_48 2024-04-06T10:07:29.929 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:07:29.929 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:07:29.929 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_48 2024-04-06T10:07:29.929 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror image_48 '..*' 2024-04-06T10:07:29.929 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:07:29.929 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:07:29.929 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_48 2024-04-06T10:07:29.929 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T10:07:29.930 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-image_48.status 2024-04-06T10:07:29.930 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-image_48.status 2024-04-06T10:07:29.930 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T10:07:29.930 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-image_48.status 2024-04-06T10:07:29.930 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_48 2024-04-06T10:07:29.931 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_48.status 2024-04-06T10:07:29.976 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 1 2024-04-06T10:07:29.976 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 1 2024-04-06T10:07:29.976 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T10:07:29.976 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=5, tag_tid=3, entry_tid=1], [object_number=4, tag_tid=3, entry_tid=0], [object_number=7, tag_tid=1, entry_tid=5427], [object_number=6, tag_tid=1, entry_tid=5426]]], state=connected] 2024-04-06T10:07:29.976 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=5, tag_tid=2, entry_tid=1], [object_number=4, tag_tid=2, entry_tid=0], [object_number=7, tag_tid=1, entry_tid=5427], [object_number=6, tag_tid=1, entry_tid=5426]]], state=connected] 2024-04-06T10:07:29.979 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_48.status 2024-04-06T10:07:29.981 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=5, tag_tid=2, entry_tid=1' 2024-04-06T10:07:29.981 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=5, tag_tid=3, entry_tid=1' -a 'object_number=5, tag_tid=3, entry_tid=1' = 'object_number=5, tag_tid=2, entry_tid=1' 2024-04-06T10:07:29.981 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=5, tag_tid=2, entry_tid=1' '!=' 'object_number=5, tag_tid=2, entry_tid=1' 2024-04-06T10:07:29.982 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T10:07:29.982 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 2 2024-04-06T10:07:31.984 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror image_48 2024-04-06T10:07:31.984 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T10:07:31.984 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:07:31.984 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_48 2024-04-06T10:07:31.984 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T10:07:31.984 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n image_48 ']' 2024-04-06T10:07:31.984 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/image_48' 2024-04-06T10:07:31.984 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_48 2024-04-06T10:07:31.984 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T10:07:31.985 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T10:07:31.985 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T10:07:31.985 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T10:07:31.985 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T10:07:31.985 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T10:07:31.985 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T10:07:31.985 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T10:07:31.986 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T10:07:31.988 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T10:07:31.988 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_48 2024-04-06T10:07:31.989 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T10:07:31.989 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T10:07:31.989 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T10:07:31.989 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T10:07:31.989 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T10:07:31.989 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T10:07:31.989 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T10:07:31.989 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T10:07:31.989 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T10:07:31.989 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T10:07:31.989 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T10:07:31.989 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:07:31.990 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T10:07:31.990 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T10:07:31.990 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T10:07:31.990 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T10:07:31.990 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T10:07:31.990 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T10:07:31.990 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T10:07:31.990 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T10:07:31.990 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T10:07:31.990 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T10:07:31.990 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:07:31.990 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T10:07:31.991 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T10:07:31.991 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T10:07:31.991 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T10:07:31.991 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T10:07:31.991 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T10:07:31.991 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T10:07:32.014 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T10:07:32.014 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T10:07:32.014 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T10:07:32.014 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_48 2024-04-06T10:07:32.122 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:Invalid command: unused arguments: ['mirror/image_48'] 2024-04-06T10:07:32.122 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:rbd mirror flush : flush rbd mirror 2024-04-06T10:07:32.122 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:admin_socket: invalid command 2024-04-06T10:07:32.125 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T10:07:32.125 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/image_48 2024-04-06T10:07:32.125 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:1 2024-04-06T10:07:32.125 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T10:07:32.125 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T10:07:32.125 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-06T10:07:32.125 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:1 2024-04-06T10:07:32.125 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T10:07:32.125 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T10:07:32.125 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T10:07:32.125 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=1 2024-04-06T10:07:32.125 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-06T10:07:32.125 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T10:07:32.126 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:07:32.126 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=1 2024-04-06T10:07:32.126 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=1 2024-04-06T10:07:32.127 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-06T10:07:32.127 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:1 2024-04-06T10:07:32.127 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T10:07:32.127 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T10:07:32.127 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-06T10:07:32.127 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:1 2024-04-06T10:07:32.127 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T10:07:32.127 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T10:07:32.127 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:07:32.127 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=1 2024-04-06T10:07:32.127 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-06T10:07:32.127 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T10:07:32.128 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T10:07:32.128 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=1 2024-04-06T10:07:32.128 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=1 2024-04-06T10:07:32.128 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-06T10:07:32.150 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T10:07:32.151 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T10:07:32.151 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T10:07:32.151 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/image_48 2024-04-06T10:07:32.304 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T10:07:32.306 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror image_48 2024-04-06T10:07:32.306 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:07:32.306 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:07:32.306 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_48 2024-04-06T10:07:32.307 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror image_48 '' 2024-04-06T10:07:32.307 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:07:32.307 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:07:32.307 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_48 2024-04-06T10:07:32.307 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T10:07:32.307 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-image_48.status 2024-04-06T10:07:32.307 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-image_48.status 2024-04-06T10:07:32.307 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T10:07:32.308 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-image_48.status 2024-04-06T10:07:32.308 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_48 2024-04-06T10:07:32.308 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_48.status 2024-04-06T10:07:32.359 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 1 2024-04-06T10:07:32.359 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 1 2024-04-06T10:07:32.359 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T10:07:32.359 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=5, tag_tid=3, entry_tid=1], [object_number=4, tag_tid=3, entry_tid=0], [object_number=7, tag_tid=1, entry_tid=5427], [object_number=6, tag_tid=1, entry_tid=5426]]], state=connected] 2024-04-06T10:07:32.359 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=5, tag_tid=3, entry_tid=1], [object_number=4, tag_tid=3, entry_tid=0], [object_number=7, tag_tid=1, entry_tid=5427], [object_number=6, tag_tid=1, entry_tid=5426]]], state=connected] 2024-04-06T10:07:32.362 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_48.status 2024-04-06T10:07:32.364 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=5, tag_tid=3, entry_tid=1' 2024-04-06T10:07:32.365 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror image_48 2024-04-06T10:07:32.365 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:07:32.365 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:07:32.365 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_48 2024-04-06T10:07:32.365 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror image_48 '..*' 2024-04-06T10:07:32.365 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:07:32.365 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:07:32.365 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_48 2024-04-06T10:07:32.366 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T10:07:32.366 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-image_48.status 2024-04-06T10:07:32.366 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-image_48.status 2024-04-06T10:07:32.366 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T10:07:32.366 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-image_48.status 2024-04-06T10:07:32.367 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_48 2024-04-06T10:07:32.367 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_48.status 2024-04-06T10:07:32.413 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 1 2024-04-06T10:07:32.413 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 1 2024-04-06T10:07:32.413 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T10:07:32.413 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=5, tag_tid=3, entry_tid=1], [object_number=4, tag_tid=3, entry_tid=0], [object_number=7, tag_tid=1, entry_tid=5427], [object_number=6, tag_tid=1, entry_tid=5426]]], state=connected] 2024-04-06T10:07:32.413 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=5, tag_tid=3, entry_tid=1], [object_number=4, tag_tid=3, entry_tid=0], [object_number=7, tag_tid=1, entry_tid=5427], [object_number=6, tag_tid=1, entry_tid=5426]]], state=connected] 2024-04-06T10:07:32.417 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_48.status 2024-04-06T10:07:32.419 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=5, tag_tid=3, entry_tid=1' 2024-04-06T10:07:32.419 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=5, tag_tid=3, entry_tid=1' -a 'object_number=5, tag_tid=3, entry_tid=1' = 'object_number=5, tag_tid=3, entry_tid=1' 2024-04-06T10:07:32.419 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T10:07:32.419 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_snap_present cluster1 mirror image_48 snap 2024-04-06T10:07:32.419 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T10:07:32.419 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:07:32.419 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_48 2024-04-06T10:07:32.419 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local snap_name=snap 2024-04-06T10:07:32.419 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local s 2024-04-06T10:07:32.419 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-04-06T10:07:32.419 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 1 2024-04-06T10:07:33.419 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 info mirror/image_48@snap 2024-04-06T10:07:33.489 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:rbd image 'image_48': 2024-04-06T10:07:33.489 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: size 128 MiB in 32 objects 2024-04-06T10:07:33.489 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: order 22 (4 MiB objects) 2024-04-06T10:07:33.489 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: snapshot_count: 1 2024-04-06T10:07:33.489 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: id: 10ac8e99ac0d 2024-04-06T10:07:33.489 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: block_name_prefix: rbd_data.10ac8e99ac0d 2024-04-06T10:07:33.489 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: format: 2 2024-04-06T10:07:33.489 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: features: layering, exclusive-lock, journaling 2024-04-06T10:07:33.489 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: op_features: 2024-04-06T10:07:33.489 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: flags: 2024-04-06T10:07:33.489 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: create_timestamp: Sat Apr 6 09:57:51 2024 2024-04-06T10:07:33.489 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: access_timestamp: Sat Apr 6 09:57:51 2024 2024-04-06T10:07:33.490 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: modify_timestamp: Sat Apr 6 09:58:52 2024 2024-04-06T10:07:33.490 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: protected: False 2024-04-06T10:07:33.490 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: journal: 10ac8e99ac0d 2024-04-06T10:07:33.490 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: mirroring state: enabled 2024-04-06T10:07:33.490 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: mirroring mode: journal 2024-04-06T10:07:33.490 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: mirroring global id: 6e6d2b94-ef30-4a86-94d9-88a1ebc2039b 2024-04-06T10:07:33.490 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: mirroring primary: false 2024-04-06T10:07:33.494 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T10:07:33.494 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ compare_image_snaps mirror image_48 snap 2024-04-06T10:07:33.494 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:07:33.494 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_48 2024-04-06T10:07:33.494 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local snap_name=snap 2024-04-06T10:07:33.494 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local ret=0 2024-04-06T10:07:33.494 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local rmt_export=/tmp/tmp.BhW6xKmROU/cluster2-mirror-image_48.export 2024-04-06T10:07:33.494 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loc_export=/tmp/tmp.BhW6xKmROU/cluster1-mirror-image_48.export 2024-04-06T10:07:33.494 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rm -f /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_48.export /tmp/tmp.BhW6xKmROU/cluster1-mirror-image_48.export 2024-04-06T10:07:33.495 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster2 -p mirror export image_48@snap /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_48.export 2024-04-06T10:07:33.662 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-04-06T10:07:33.671 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 -p mirror export image_48@snap /tmp/tmp.BhW6xKmROU/cluster1-mirror-image_48.export 2024-04-06T10:07:34.003 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-04-06T10:07:34.016 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmp /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_48.export /tmp/tmp.BhW6xKmROU/cluster1-mirror-image_48.export 2024-04-06T10:07:34.097 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rm -f /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_48.export /tmp/tmp.BhW6xKmROU/cluster1-mirror-image_48.export 2024-04-06T10:07:34.139 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T10:07:34.139 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-06T10:07:34.139 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ image=image_49 2024-04-06T10:07:34.139 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ create_snap cluster2 mirror image_49 snap 2024-04-06T10:07:34.139 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-04-06T10:07:34.141 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:07:34.141 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_49 2024-04-06T10:07:34.141 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local snap_name=snap 2024-04-06T10:07:34.141 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster2 -p mirror snap create image_49@snap --debug-rbd=20 --debug-journaler=20 2024-04-06T10:07:34.944 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_image_replay_started cluster1 mirror image_49 2024-04-06T10:07:34.944 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T10:07:34.944 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:07:34.944 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_49 2024-04-06T10:07:34.944 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_image_replay_state cluster1 mirror image_49 started 2024-04-06T10:07:34.944 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T10:07:34.944 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:07:34.945 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_49 2024-04-06T10:07:34.945 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state=started 2024-04-06T10:07:34.945 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local s 2024-04-06T10:07:34.945 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T10:07:34.945 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 1 2024-04-06T10:07:35.946 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test_image_replay_state cluster1 mirror image_49 started 2024-04-06T10:07:35.946 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T10:07:35.946 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:07:35.946 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_49 2024-04-06T10:07:35.946 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local test_state=started 2024-04-06T10:07:35.946 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local status_result 2024-04-06T10:07:35.947 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local current_state=stopped 2024-04-06T10:07:35.947 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_49 2024-04-06T10:07:35.948 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_instance=cluster1 2024-04-06T10:07:35.948 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ shift 2024-04-06T10:07:35.948 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -i state 2024-04-06T10:07:35.948 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T10:07:35.948 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance=cluster1 2024-04-06T10:07:35.948 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local loop_instance 2024-04-06T10:07:35.948 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T10:07:35.948 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sleep 0 2024-04-06T10:07:35.949 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-04-06T10:07:35.950 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ seq 0 3 2024-04-06T10:07:35.952 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T10:07:35.952 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_49 2024-04-06T10:07:35.952 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1:0 2024-04-06T10:07:35.952 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ shift 2024-04-06T10:07:35.953 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T10:07:35.953 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-04-06T10:07:35.953 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T10:07:35.953 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=cluster 2024-04-06T10:07:35.953 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T10:07:35.953 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:07:35.953 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T10:07:35.953 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T10:07:35.953 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval cluster=cluster1 2024-04-06T10:07:35.953 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ cluster=cluster1 2024-04-06T10:07:35.953 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T10:07:35.953 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T10:07:35.953 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-04-06T10:07:35.954 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local local_cluster=cluster1:0 2024-04-06T10:07:35.954 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local cluster=cluster1 2024-04-06T10:07:35.954 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local instance 2024-04-06T10:07:35.954 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T10:07:35.954 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local val=cluster1:0 2024-04-06T10:07:35.954 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local cluster_var_name=local_cluster 2024-04-06T10:07:35.954 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local instance_var_name=instance 2024-04-06T10:07:35.954 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ cluster=cluster1 2024-04-06T10:07:35.954 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T10:07:35.954 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ '[' 0 = cluster1:0 ']' 2024-04-06T10:07:35.954 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ eval local_cluster=cluster1 2024-04-06T10:07:35.955 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++++ local_cluster=cluster1 2024-04-06T10:07:35.955 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ eval instance=0 2024-04-06T10:07:35.955 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++++ instance=0 2024-04-06T10:07:35.955 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T10:07:35.974 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T10:07:35.974 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T10:07:35.974 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T10:07:35.974 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_49 2024-04-06T10:07:36.080 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:Invalid command: unused arguments: ['mirror/image_49'] 2024-04-06T10:07:36.081 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:rbd mirror status : get status for rbd mirror 2024-04-06T10:07:36.081 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:admin_socket: invalid command 2024-04-06T10:07:36.083 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T10:07:36.083 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_49 2024-04-06T10:07:36.083 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1:1 2024-04-06T10:07:36.083 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ shift 2024-04-06T10:07:36.083 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T10:07:36.083 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-04-06T10:07:36.083 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:1 2024-04-06T10:07:36.083 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=cluster 2024-04-06T10:07:36.083 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T10:07:36.083 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:07:36.083 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=1 2024-04-06T10:07:36.083 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-06T10:07:36.083 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval cluster=cluster1 2024-04-06T10:07:36.083 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ cluster=cluster1 2024-04-06T10:07:36.084 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=1 2024-04-06T10:07:36.084 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=1 2024-04-06T10:07:36.085 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-04-06T10:07:36.085 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local local_cluster=cluster1:1 2024-04-06T10:07:36.085 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local cluster=cluster1 2024-04-06T10:07:36.085 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local instance 2024-04-06T10:07:36.085 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-06T10:07:36.085 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local val=cluster1:1 2024-04-06T10:07:36.085 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local cluster_var_name=local_cluster 2024-04-06T10:07:36.085 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local instance_var_name=instance 2024-04-06T10:07:36.086 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ cluster=cluster1 2024-04-06T10:07:36.086 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=1 2024-04-06T10:07:36.086 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ '[' 1 = cluster1:1 ']' 2024-04-06T10:07:36.086 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ eval local_cluster=cluster1 2024-04-06T10:07:36.086 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++++ local_cluster=cluster1 2024-04-06T10:07:36.086 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ eval instance=1 2024-04-06T10:07:36.086 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++++ instance=1 2024-04-06T10:07:36.086 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-06T10:07:36.109 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T10:07:36.109 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T10:07:36.109 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T10:07:36.109 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_49 2024-04-06T10:07:36.262 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ return 0 2024-04-06T10:07:36.262 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ status_result=' "state": "Replaying"' 2024-04-06T10:07:36.263 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ echo ' "state": "Replaying"' 2024-04-06T10:07:36.263 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep -i Replaying 2024-04-06T10:07:36.265 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: "state": "Replaying" 2024-04-06T10:07:36.266 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ current_state=started 2024-04-06T10:07:36.266 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test started = started 2024-04-06T10:07:36.266 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T10:07:36.266 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_49 2024-04-06T10:07:36.266 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local local_cluster=cluster1 2024-04-06T10:07:36.266 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-04-06T10:07:36.266 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:07:36.266 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_49 2024-04-06T10:07:36.266 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' journal = journal ']' 2024-04-06T10:07:36.266 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_journal_replay_complete cluster1 cluster2 mirror image_49 2024-04-06T10:07:36.267 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local local_cluster=cluster1 2024-04-06T10:07:36.267 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-04-06T10:07:36.267 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:07:36.267 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_49 2024-04-06T10:07:36.267 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local s master_pos mirror_pos last_mirror_pos 2024-04-06T10:07:36.267 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local master_tag master_entry mirror_tag mirror_entry 2024-04-06T10:07:36.267 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T10:07:36.267 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T10:07:36.267 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T10:07:36.467 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror image_49 2024-04-06T10:07:36.468 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T10:07:36.468 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:07:36.468 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_49 2024-04-06T10:07:36.468 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T10:07:36.468 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n image_49 ']' 2024-04-06T10:07:36.468 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/image_49' 2024-04-06T10:07:36.468 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_49 2024-04-06T10:07:36.468 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T10:07:36.468 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T10:07:36.468 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T10:07:36.468 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T10:07:36.468 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T10:07:36.468 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T10:07:36.468 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T10:07:36.469 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T10:07:36.469 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T10:07:36.473 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T10:07:36.473 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_49 2024-04-06T10:07:36.473 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T10:07:36.473 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T10:07:36.473 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T10:07:36.474 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T10:07:36.474 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T10:07:36.474 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T10:07:36.474 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T10:07:36.474 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T10:07:36.474 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T10:07:36.474 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T10:07:36.474 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T10:07:36.474 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:07:36.474 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T10:07:36.474 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T10:07:36.474 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T10:07:36.474 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T10:07:36.474 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T10:07:36.475 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T10:07:36.475 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T10:07:36.475 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T10:07:36.475 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T10:07:36.475 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T10:07:36.475 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:07:36.475 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T10:07:36.475 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T10:07:36.475 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T10:07:36.475 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T10:07:36.475 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T10:07:36.475 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T10:07:36.475 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T10:07:36.500 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T10:07:36.500 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T10:07:36.500 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T10:07:36.500 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_49 2024-04-06T10:07:36.606 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:Invalid command: unused arguments: ['mirror/image_49'] 2024-04-06T10:07:36.606 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:rbd mirror flush : flush rbd mirror 2024-04-06T10:07:36.606 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:admin_socket: invalid command 2024-04-06T10:07:36.609 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T10:07:36.609 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/image_49 2024-04-06T10:07:36.609 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:1 2024-04-06T10:07:36.609 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T10:07:36.609 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T10:07:36.609 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-06T10:07:36.609 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:1 2024-04-06T10:07:36.609 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T10:07:36.609 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T10:07:36.609 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T10:07:36.609 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=1 2024-04-06T10:07:36.609 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-06T10:07:36.610 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T10:07:36.610 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:07:36.610 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=1 2024-04-06T10:07:36.610 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=1 2024-04-06T10:07:36.611 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-06T10:07:36.611 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:1 2024-04-06T10:07:36.611 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T10:07:36.611 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T10:07:36.611 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-06T10:07:36.611 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:1 2024-04-06T10:07:36.611 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T10:07:36.611 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T10:07:36.611 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:07:36.611 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=1 2024-04-06T10:07:36.611 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-06T10:07:36.612 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T10:07:36.612 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T10:07:36.612 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=1 2024-04-06T10:07:36.612 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=1 2024-04-06T10:07:36.612 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-06T10:07:36.628 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T10:07:36.629 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T10:07:36.629 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T10:07:36.629 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/image_49 2024-04-06T10:07:36.767 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T10:07:36.767 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror image_49 2024-04-06T10:07:36.768 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:07:36.768 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:07:36.768 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_49 2024-04-06T10:07:36.768 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror image_49 '' 2024-04-06T10:07:36.768 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:07:36.768 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:07:36.768 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_49 2024-04-06T10:07:36.768 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T10:07:36.769 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-image_49.status 2024-04-06T10:07:36.769 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-image_49.status 2024-04-06T10:07:36.769 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T10:07:36.770 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-image_49.status 2024-04-06T10:07:36.771 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_49 2024-04-06T10:07:36.771 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_49.status 2024-04-06T10:07:36.817 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 1 2024-04-06T10:07:36.817 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 1 2024-04-06T10:07:36.817 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T10:07:36.817 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=5, tag_tid=3, entry_tid=1], [object_number=4, tag_tid=3, entry_tid=0], [object_number=7, tag_tid=1, entry_tid=8215], [object_number=6, tag_tid=1, entry_tid=8214]]], state=connected] 2024-04-06T10:07:36.817 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=5, tag_tid=2, entry_tid=1], [object_number=4, tag_tid=2, entry_tid=0], [object_number=7, tag_tid=1, entry_tid=8215], [object_number=6, tag_tid=1, entry_tid=8214]]], state=connected] 2024-04-06T10:07:36.820 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_49.status 2024-04-06T10:07:36.821 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=5, tag_tid=3, entry_tid=1' 2024-04-06T10:07:36.823 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror image_49 2024-04-06T10:07:36.823 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:07:36.823 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:07:36.823 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_49 2024-04-06T10:07:36.823 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror image_49 '..*' 2024-04-06T10:07:36.823 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:07:36.823 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:07:36.823 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_49 2024-04-06T10:07:36.823 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T10:07:36.823 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-image_49.status 2024-04-06T10:07:36.823 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-image_49.status 2024-04-06T10:07:36.823 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T10:07:36.824 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-image_49.status 2024-04-06T10:07:36.824 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_49 2024-04-06T10:07:36.824 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_49.status 2024-04-06T10:07:36.869 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 1 2024-04-06T10:07:36.869 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 1 2024-04-06T10:07:36.869 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T10:07:36.869 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=5, tag_tid=3, entry_tid=1], [object_number=4, tag_tid=3, entry_tid=0], [object_number=7, tag_tid=1, entry_tid=8215], [object_number=6, tag_tid=1, entry_tid=8214]]], state=connected] 2024-04-06T10:07:36.869 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=5, tag_tid=2, entry_tid=1], [object_number=4, tag_tid=2, entry_tid=0], [object_number=7, tag_tid=1, entry_tid=8215], [object_number=6, tag_tid=1, entry_tid=8214]]], state=connected] 2024-04-06T10:07:36.872 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_49.status 2024-04-06T10:07:36.875 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=5, tag_tid=2, entry_tid=1' 2024-04-06T10:07:36.875 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=5, tag_tid=3, entry_tid=1' -a 'object_number=5, tag_tid=3, entry_tid=1' = 'object_number=5, tag_tid=2, entry_tid=1' 2024-04-06T10:07:36.875 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=5, tag_tid=2, entry_tid=1' '!=' '' 2024-04-06T10:07:36.875 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-04-06T10:07:36.875 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=5, tag_tid=2, entry_tid=1' = '' 2024-04-06T10:07:36.875 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_mirror_pos='object_number=5, tag_tid=2, entry_tid=1' 2024-04-06T10:07:36.875 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=5, tag_tid=3, entry_tid=1' 2024-04-06T10:07:36.875 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T10:07:36.875 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T10:07:36.876 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_tag=3 2024-04-06T10:07:36.876 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=5, tag_tid=2, entry_tid=1' 2024-04-06T10:07:36.877 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-04-06T10:07:36.877 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T10:07:36.877 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_tag=2 2024-04-06T10:07:36.878 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=5, tag_tid=3, entry_tid=1' 2024-04-06T10:07:36.878 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T10:07:36.878 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T10:07:36.879 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_entry=1 2024-04-06T10:07:36.880 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo 'object_number=5, tag_tid=2, entry_tid=1' 2024-04-06T10:07:36.880 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-04-06T10:07:36.880 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut -d= -f 2 2024-04-06T10:07:36.881 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_entry=1 2024-04-06T10:07:36.881 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 3 = 2 -a 1 -le 1 2024-04-06T10:07:36.881 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T10:07:36.882 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T10:07:36.882 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T10:07:37.083 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror image_49 2024-04-06T10:07:37.083 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T10:07:37.083 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:07:37.083 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_49 2024-04-06T10:07:37.083 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T10:07:37.083 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n image_49 ']' 2024-04-06T10:07:37.083 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/image_49' 2024-04-06T10:07:37.083 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_49 2024-04-06T10:07:37.084 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T10:07:37.084 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T10:07:37.084 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T10:07:37.084 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T10:07:37.084 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T10:07:37.084 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T10:07:37.084 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T10:07:37.084 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T10:07:37.085 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T10:07:37.087 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T10:07:37.087 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_49 2024-04-06T10:07:37.087 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T10:07:37.087 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T10:07:37.087 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T10:07:37.087 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T10:07:37.087 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T10:07:37.087 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T10:07:37.088 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T10:07:37.088 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T10:07:37.088 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T10:07:37.088 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T10:07:37.088 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T10:07:37.088 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:07:37.088 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T10:07:37.088 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T10:07:37.088 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T10:07:37.088 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T10:07:37.088 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T10:07:37.088 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T10:07:37.088 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T10:07:37.088 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T10:07:37.088 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T10:07:37.089 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T10:07:37.089 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:07:37.089 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T10:07:37.089 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T10:07:37.089 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T10:07:37.089 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T10:07:37.089 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T10:07:37.089 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T10:07:37.089 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T10:07:37.116 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T10:07:37.116 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T10:07:37.117 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T10:07:37.117 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_49 2024-04-06T10:07:37.225 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:Invalid command: unused arguments: ['mirror/image_49'] 2024-04-06T10:07:37.225 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:rbd mirror flush : flush rbd mirror 2024-04-06T10:07:37.225 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:admin_socket: invalid command 2024-04-06T10:07:37.230 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T10:07:37.230 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/image_49 2024-04-06T10:07:37.230 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:1 2024-04-06T10:07:37.230 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T10:07:37.230 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T10:07:37.231 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-06T10:07:37.231 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:1 2024-04-06T10:07:37.231 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T10:07:37.231 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T10:07:37.231 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T10:07:37.231 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=1 2024-04-06T10:07:37.231 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-06T10:07:37.231 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T10:07:37.231 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:07:37.231 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=1 2024-04-06T10:07:37.231 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=1 2024-04-06T10:07:37.231 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-06T10:07:37.231 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:1 2024-04-06T10:07:37.232 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T10:07:37.232 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T10:07:37.232 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-06T10:07:37.232 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:1 2024-04-06T10:07:37.232 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T10:07:37.232 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T10:07:37.232 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:07:37.232 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=1 2024-04-06T10:07:37.232 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-06T10:07:37.232 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T10:07:37.233 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T10:07:37.233 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=1 2024-04-06T10:07:37.233 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=1 2024-04-06T10:07:37.233 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-06T10:07:37.248 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T10:07:37.249 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T10:07:37.249 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T10:07:37.249 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/image_49 2024-04-06T10:07:37.391 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T10:07:37.392 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror image_49 2024-04-06T10:07:37.392 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:07:37.392 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:07:37.392 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_49 2024-04-06T10:07:37.392 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror image_49 '' 2024-04-06T10:07:37.393 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:07:37.393 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:07:37.393 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_49 2024-04-06T10:07:37.393 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T10:07:37.393 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-image_49.status 2024-04-06T10:07:37.393 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-image_49.status 2024-04-06T10:07:37.393 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T10:07:37.394 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-image_49.status 2024-04-06T10:07:37.394 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_49 2024-04-06T10:07:37.395 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_49.status 2024-04-06T10:07:37.440 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 1 2024-04-06T10:07:37.440 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 1 2024-04-06T10:07:37.440 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T10:07:37.440 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=5, tag_tid=3, entry_tid=1], [object_number=4, tag_tid=3, entry_tid=0], [object_number=7, tag_tid=1, entry_tid=8215], [object_number=6, tag_tid=1, entry_tid=8214]]], state=connected] 2024-04-06T10:07:37.440 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=5, tag_tid=2, entry_tid=1], [object_number=4, tag_tid=2, entry_tid=0], [object_number=7, tag_tid=1, entry_tid=8215], [object_number=6, tag_tid=1, entry_tid=8214]]], state=connected] 2024-04-06T10:07:37.443 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_49.status 2024-04-06T10:07:37.445 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=5, tag_tid=3, entry_tid=1' 2024-04-06T10:07:37.446 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror image_49 2024-04-06T10:07:37.446 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:07:37.446 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:07:37.446 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_49 2024-04-06T10:07:37.446 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror image_49 '..*' 2024-04-06T10:07:37.446 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:07:37.447 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:07:37.447 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_49 2024-04-06T10:07:37.447 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T10:07:37.447 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-image_49.status 2024-04-06T10:07:37.447 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-image_49.status 2024-04-06T10:07:37.447 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T10:07:37.447 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-image_49.status 2024-04-06T10:07:37.448 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_49 2024-04-06T10:07:37.448 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_49.status 2024-04-06T10:07:37.493 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 1 2024-04-06T10:07:37.493 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 1 2024-04-06T10:07:37.493 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T10:07:37.493 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=5, tag_tid=3, entry_tid=1], [object_number=4, tag_tid=3, entry_tid=0], [object_number=7, tag_tid=1, entry_tid=8215], [object_number=6, tag_tid=1, entry_tid=8214]]], state=connected] 2024-04-06T10:07:37.493 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=5, tag_tid=2, entry_tid=1], [object_number=4, tag_tid=2, entry_tid=0], [object_number=7, tag_tid=1, entry_tid=8215], [object_number=6, tag_tid=1, entry_tid=8214]]], state=connected] 2024-04-06T10:07:37.497 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_49.status 2024-04-06T10:07:37.498 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=5, tag_tid=2, entry_tid=1' 2024-04-06T10:07:37.498 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=5, tag_tid=3, entry_tid=1' -a 'object_number=5, tag_tid=3, entry_tid=1' = 'object_number=5, tag_tid=2, entry_tid=1' 2024-04-06T10:07:37.498 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=5, tag_tid=2, entry_tid=1' '!=' 'object_number=5, tag_tid=2, entry_tid=1' 2024-04-06T10:07:37.498 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T10:07:37.499 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.4 2024-04-06T10:07:37.900 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror image_49 2024-04-06T10:07:37.900 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T10:07:37.900 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:07:37.900 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_49 2024-04-06T10:07:37.900 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T10:07:37.901 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n image_49 ']' 2024-04-06T10:07:37.901 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/image_49' 2024-04-06T10:07:37.901 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_49 2024-04-06T10:07:37.901 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T10:07:37.901 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T10:07:37.901 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T10:07:37.901 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T10:07:37.901 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T10:07:37.901 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T10:07:37.901 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T10:07:37.903 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T10:07:37.903 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T10:07:37.903 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T10:07:37.903 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_49 2024-04-06T10:07:37.903 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T10:07:37.903 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T10:07:37.903 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T10:07:37.903 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T10:07:37.903 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T10:07:37.903 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T10:07:37.903 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T10:07:37.904 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T10:07:37.904 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T10:07:37.904 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T10:07:37.904 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T10:07:37.904 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:07:37.904 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T10:07:37.904 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T10:07:37.906 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T10:07:37.906 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T10:07:37.906 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T10:07:37.906 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T10:07:37.906 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T10:07:37.906 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T10:07:37.906 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T10:07:37.906 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T10:07:37.906 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:07:37.906 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T10:07:37.906 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T10:07:37.907 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T10:07:37.907 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T10:07:37.907 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T10:07:37.907 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T10:07:37.907 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T10:07:37.929 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T10:07:37.930 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T10:07:37.930 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T10:07:37.930 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_49 2024-04-06T10:07:38.034 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:Invalid command: unused arguments: ['mirror/image_49'] 2024-04-06T10:07:38.034 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:rbd mirror flush : flush rbd mirror 2024-04-06T10:07:38.034 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:admin_socket: invalid command 2024-04-06T10:07:38.036 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T10:07:38.036 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/image_49 2024-04-06T10:07:38.036 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:1 2024-04-06T10:07:38.036 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T10:07:38.037 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T10:07:38.037 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-06T10:07:38.037 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:1 2024-04-06T10:07:38.037 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T10:07:38.037 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T10:07:38.037 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T10:07:38.037 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=1 2024-04-06T10:07:38.037 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-06T10:07:38.037 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T10:07:38.037 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:07:38.037 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=1 2024-04-06T10:07:38.037 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=1 2024-04-06T10:07:38.038 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-06T10:07:38.039 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:1 2024-04-06T10:07:38.039 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T10:07:38.039 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T10:07:38.039 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-06T10:07:38.039 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:1 2024-04-06T10:07:38.039 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T10:07:38.039 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T10:07:38.039 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:07:38.039 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=1 2024-04-06T10:07:38.039 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-06T10:07:38.039 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T10:07:38.039 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T10:07:38.040 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=1 2024-04-06T10:07:38.040 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=1 2024-04-06T10:07:38.040 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-06T10:07:38.063 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T10:07:38.063 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T10:07:38.063 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T10:07:38.063 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/image_49 2024-04-06T10:07:38.213 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T10:07:38.214 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror image_49 2024-04-06T10:07:38.214 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:07:38.214 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:07:38.214 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_49 2024-04-06T10:07:38.214 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror image_49 '' 2024-04-06T10:07:38.214 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:07:38.214 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:07:38.214 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_49 2024-04-06T10:07:38.214 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T10:07:38.215 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-image_49.status 2024-04-06T10:07:38.215 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-image_49.status 2024-04-06T10:07:38.215 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T10:07:38.218 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-image_49.status 2024-04-06T10:07:38.218 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_49 2024-04-06T10:07:38.218 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_49.status 2024-04-06T10:07:38.263 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 1 2024-04-06T10:07:38.264 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 1 2024-04-06T10:07:38.264 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T10:07:38.264 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=5, tag_tid=3, entry_tid=1], [object_number=4, tag_tid=3, entry_tid=0], [object_number=7, tag_tid=1, entry_tid=8215], [object_number=6, tag_tid=1, entry_tid=8214]]], state=connected] 2024-04-06T10:07:38.264 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=5, tag_tid=2, entry_tid=1], [object_number=4, tag_tid=2, entry_tid=0], [object_number=7, tag_tid=1, entry_tid=8215], [object_number=6, tag_tid=1, entry_tid=8214]]], state=connected] 2024-04-06T10:07:38.267 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_49.status 2024-04-06T10:07:38.268 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=5, tag_tid=3, entry_tid=1' 2024-04-06T10:07:38.270 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror image_49 2024-04-06T10:07:38.270 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:07:38.270 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:07:38.270 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_49 2024-04-06T10:07:38.270 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror image_49 '..*' 2024-04-06T10:07:38.270 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:07:38.270 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:07:38.270 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_49 2024-04-06T10:07:38.270 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T10:07:38.270 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-image_49.status 2024-04-06T10:07:38.271 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-image_49.status 2024-04-06T10:07:38.271 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T10:07:38.271 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-image_49.status 2024-04-06T10:07:38.271 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_49.status 2024-04-06T10:07:38.271 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_49 2024-04-06T10:07:38.316 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 1 2024-04-06T10:07:38.316 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 1 2024-04-06T10:07:38.316 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T10:07:38.316 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=5, tag_tid=3, entry_tid=1], [object_number=4, tag_tid=3, entry_tid=0], [object_number=7, tag_tid=1, entry_tid=8215], [object_number=6, tag_tid=1, entry_tid=8214]]], state=connected] 2024-04-06T10:07:38.316 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=5, tag_tid=2, entry_tid=1], [object_number=4, tag_tid=2, entry_tid=0], [object_number=7, tag_tid=1, entry_tid=8215], [object_number=6, tag_tid=1, entry_tid=8214]]], state=connected] 2024-04-06T10:07:38.320 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_49.status 2024-04-06T10:07:38.321 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=5, tag_tid=2, entry_tid=1' 2024-04-06T10:07:38.321 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=5, tag_tid=3, entry_tid=1' -a 'object_number=5, tag_tid=3, entry_tid=1' = 'object_number=5, tag_tid=2, entry_tid=1' 2024-04-06T10:07:38.321 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=5, tag_tid=2, entry_tid=1' '!=' 'object_number=5, tag_tid=2, entry_tid=1' 2024-04-06T10:07:38.322 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T10:07:38.322 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.8 2024-04-06T10:07:39.124 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror image_49 2024-04-06T10:07:39.124 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T10:07:39.124 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:07:39.124 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_49 2024-04-06T10:07:39.124 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T10:07:39.124 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n image_49 ']' 2024-04-06T10:07:39.124 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/image_49' 2024-04-06T10:07:39.124 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_49 2024-04-06T10:07:39.125 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T10:07:39.125 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T10:07:39.125 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T10:07:39.125 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T10:07:39.125 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T10:07:39.125 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T10:07:39.125 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T10:07:39.125 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T10:07:39.125 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T10:07:39.127 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T10:07:39.127 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_49 2024-04-06T10:07:39.127 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T10:07:39.127 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T10:07:39.127 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T10:07:39.127 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T10:07:39.128 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T10:07:39.128 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T10:07:39.128 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T10:07:39.128 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T10:07:39.128 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T10:07:39.128 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T10:07:39.128 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T10:07:39.130 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:07:39.130 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T10:07:39.130 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T10:07:39.130 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T10:07:39.130 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T10:07:39.130 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T10:07:39.130 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T10:07:39.130 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T10:07:39.130 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T10:07:39.130 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T10:07:39.131 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T10:07:39.131 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:07:39.131 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T10:07:39.131 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T10:07:39.131 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T10:07:39.131 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T10:07:39.131 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T10:07:39.131 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T10:07:39.131 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T10:07:39.154 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T10:07:39.154 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T10:07:39.154 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T10:07:39.155 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_49 2024-04-06T10:07:39.259 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:Invalid command: unused arguments: ['mirror/image_49'] 2024-04-06T10:07:39.260 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:rbd mirror flush : flush rbd mirror 2024-04-06T10:07:39.260 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:admin_socket: invalid command 2024-04-06T10:07:39.262 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T10:07:39.262 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/image_49 2024-04-06T10:07:39.262 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:1 2024-04-06T10:07:39.262 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T10:07:39.262 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T10:07:39.262 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-06T10:07:39.262 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:1 2024-04-06T10:07:39.262 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T10:07:39.262 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T10:07:39.262 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T10:07:39.263 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=1 2024-04-06T10:07:39.263 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-06T10:07:39.263 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T10:07:39.263 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:07:39.263 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=1 2024-04-06T10:07:39.263 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=1 2024-04-06T10:07:39.264 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-06T10:07:39.264 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:1 2024-04-06T10:07:39.264 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T10:07:39.264 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T10:07:39.264 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-06T10:07:39.265 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:1 2024-04-06T10:07:39.265 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T10:07:39.265 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T10:07:39.265 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:07:39.265 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=1 2024-04-06T10:07:39.265 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-06T10:07:39.265 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T10:07:39.265 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T10:07:39.265 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=1 2024-04-06T10:07:39.265 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=1 2024-04-06T10:07:39.265 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-06T10:07:39.289 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T10:07:39.289 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T10:07:39.289 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T10:07:39.289 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/image_49 2024-04-06T10:07:39.432 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T10:07:39.433 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror image_49 2024-04-06T10:07:39.434 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:07:39.434 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:07:39.434 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_49 2024-04-06T10:07:39.434 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror image_49 '' 2024-04-06T10:07:39.434 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:07:39.434 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:07:39.434 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_49 2024-04-06T10:07:39.434 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T10:07:39.434 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-image_49.status 2024-04-06T10:07:39.435 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-image_49.status 2024-04-06T10:07:39.435 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T10:07:39.437 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-image_49.status 2024-04-06T10:07:39.437 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_49 2024-04-06T10:07:39.437 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_49.status 2024-04-06T10:07:39.489 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 1 2024-04-06T10:07:39.489 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 1 2024-04-06T10:07:39.489 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T10:07:39.489 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=5, tag_tid=3, entry_tid=1], [object_number=4, tag_tid=3, entry_tid=0], [object_number=7, tag_tid=1, entry_tid=8215], [object_number=6, tag_tid=1, entry_tid=8214]]], state=connected] 2024-04-06T10:07:39.489 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=5, tag_tid=2, entry_tid=1], [object_number=4, tag_tid=2, entry_tid=0], [object_number=7, tag_tid=1, entry_tid=8215], [object_number=6, tag_tid=1, entry_tid=8214]]], state=connected] 2024-04-06T10:07:39.493 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_49.status 2024-04-06T10:07:39.494 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=5, tag_tid=3, entry_tid=1' 2024-04-06T10:07:39.495 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror image_49 2024-04-06T10:07:39.495 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:07:39.496 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:07:39.496 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_49 2024-04-06T10:07:39.496 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror image_49 '..*' 2024-04-06T10:07:39.496 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:07:39.496 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:07:39.496 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_49 2024-04-06T10:07:39.496 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T10:07:39.496 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-image_49.status 2024-04-06T10:07:39.496 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-image_49.status 2024-04-06T10:07:39.496 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T10:07:39.497 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-image_49.status 2024-04-06T10:07:39.497 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_49 2024-04-06T10:07:39.497 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_49.status 2024-04-06T10:07:39.542 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 1 2024-04-06T10:07:39.543 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 1 2024-04-06T10:07:39.543 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T10:07:39.543 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=5, tag_tid=3, entry_tid=1], [object_number=4, tag_tid=3, entry_tid=0], [object_number=7, tag_tid=1, entry_tid=8215], [object_number=6, tag_tid=1, entry_tid=8214]]], state=connected] 2024-04-06T10:07:39.543 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=5, tag_tid=2, entry_tid=1], [object_number=4, tag_tid=2, entry_tid=0], [object_number=7, tag_tid=1, entry_tid=8215], [object_number=6, tag_tid=1, entry_tid=8214]]], state=connected] 2024-04-06T10:07:39.546 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_49.status 2024-04-06T10:07:39.548 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=5, tag_tid=2, entry_tid=1' 2024-04-06T10:07:39.548 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=5, tag_tid=3, entry_tid=1' -a 'object_number=5, tag_tid=3, entry_tid=1' = 'object_number=5, tag_tid=2, entry_tid=1' 2024-04-06T10:07:39.548 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=5, tag_tid=2, entry_tid=1' '!=' 'object_number=5, tag_tid=2, entry_tid=1' 2024-04-06T10:07:39.548 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T10:07:39.548 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 1.6 2024-04-06T10:07:41.150 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror image_49 2024-04-06T10:07:41.150 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T10:07:41.150 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:07:41.150 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_49 2024-04-06T10:07:41.150 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T10:07:41.150 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n image_49 ']' 2024-04-06T10:07:41.150 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/image_49' 2024-04-06T10:07:41.150 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_49 2024-04-06T10:07:41.150 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T10:07:41.151 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T10:07:41.151 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T10:07:41.151 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T10:07:41.151 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T10:07:41.151 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T10:07:41.151 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T10:07:41.151 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T10:07:41.152 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T10:07:41.154 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T10:07:41.154 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_49 2024-04-06T10:07:41.154 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T10:07:41.154 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T10:07:41.154 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T10:07:41.154 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T10:07:41.155 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T10:07:41.155 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T10:07:41.155 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T10:07:41.155 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T10:07:41.155 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T10:07:41.155 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T10:07:41.155 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T10:07:41.155 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:07:41.155 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T10:07:41.155 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T10:07:41.157 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T10:07:41.157 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T10:07:41.157 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T10:07:41.157 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T10:07:41.157 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T10:07:41.157 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T10:07:41.157 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T10:07:41.157 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T10:07:41.157 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:07:41.157 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T10:07:41.158 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T10:07:41.158 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T10:07:41.158 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T10:07:41.158 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T10:07:41.159 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T10:07:41.159 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T10:07:41.182 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T10:07:41.182 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T10:07:41.183 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T10:07:41.183 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_49 2024-04-06T10:07:41.289 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:Invalid command: unused arguments: ['mirror/image_49'] 2024-04-06T10:07:41.289 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:rbd mirror flush : flush rbd mirror 2024-04-06T10:07:41.289 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:admin_socket: invalid command 2024-04-06T10:07:41.291 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T10:07:41.291 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/image_49 2024-04-06T10:07:41.291 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:1 2024-04-06T10:07:41.291 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T10:07:41.291 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T10:07:41.291 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-06T10:07:41.292 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:1 2024-04-06T10:07:41.292 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T10:07:41.292 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T10:07:41.292 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T10:07:41.292 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=1 2024-04-06T10:07:41.292 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-06T10:07:41.292 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T10:07:41.292 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:07:41.292 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=1 2024-04-06T10:07:41.292 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=1 2024-04-06T10:07:41.293 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-06T10:07:41.294 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:1 2024-04-06T10:07:41.294 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T10:07:41.294 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T10:07:41.294 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-06T10:07:41.294 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:1 2024-04-06T10:07:41.294 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T10:07:41.294 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T10:07:41.294 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:07:41.294 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=1 2024-04-06T10:07:41.294 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-06T10:07:41.294 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T10:07:41.294 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T10:07:41.295 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=1 2024-04-06T10:07:41.295 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=1 2024-04-06T10:07:41.295 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-06T10:07:41.316 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T10:07:41.316 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T10:07:41.316 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T10:07:41.317 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/image_49 2024-04-06T10:07:41.460 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T10:07:41.460 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror image_49 2024-04-06T10:07:41.460 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:07:41.461 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:07:41.461 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_49 2024-04-06T10:07:41.461 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror image_49 '' 2024-04-06T10:07:41.461 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:07:41.461 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:07:41.461 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_49 2024-04-06T10:07:41.461 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T10:07:41.462 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-image_49.status 2024-04-06T10:07:41.462 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-image_49.status 2024-04-06T10:07:41.462 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T10:07:41.464 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-image_49.status 2024-04-06T10:07:41.465 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_49 2024-04-06T10:07:41.465 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_49.status 2024-04-06T10:07:41.521 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 1 2024-04-06T10:07:41.521 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 1 2024-04-06T10:07:41.521 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T10:07:41.521 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=5, tag_tid=3, entry_tid=1], [object_number=4, tag_tid=3, entry_tid=0], [object_number=7, tag_tid=1, entry_tid=8215], [object_number=6, tag_tid=1, entry_tid=8214]]], state=connected] 2024-04-06T10:07:41.521 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=5, tag_tid=2, entry_tid=1], [object_number=4, tag_tid=2, entry_tid=0], [object_number=7, tag_tid=1, entry_tid=8215], [object_number=6, tag_tid=1, entry_tid=8214]]], state=connected] 2024-04-06T10:07:41.525 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_49.status 2024-04-06T10:07:41.526 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=5, tag_tid=3, entry_tid=1' 2024-04-06T10:07:41.527 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror image_49 2024-04-06T10:07:41.527 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:07:41.527 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:07:41.527 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_49 2024-04-06T10:07:41.528 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror image_49 '..*' 2024-04-06T10:07:41.528 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:07:41.528 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:07:41.528 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_49 2024-04-06T10:07:41.528 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T10:07:41.528 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-image_49.status 2024-04-06T10:07:41.528 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-image_49.status 2024-04-06T10:07:41.528 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T10:07:41.528 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-image_49.status 2024-04-06T10:07:41.529 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_49 2024-04-06T10:07:41.529 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_49.status 2024-04-06T10:07:41.575 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 1 2024-04-06T10:07:41.575 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 1 2024-04-06T10:07:41.575 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T10:07:41.575 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=5, tag_tid=3, entry_tid=1], [object_number=4, tag_tid=3, entry_tid=0], [object_number=7, tag_tid=1, entry_tid=8215], [object_number=6, tag_tid=1, entry_tid=8214]]], state=connected] 2024-04-06T10:07:41.575 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=5, tag_tid=2, entry_tid=1], [object_number=4, tag_tid=2, entry_tid=0], [object_number=7, tag_tid=1, entry_tid=8215], [object_number=6, tag_tid=1, entry_tid=8214]]], state=connected] 2024-04-06T10:07:41.579 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_49.status 2024-04-06T10:07:41.580 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=5, tag_tid=2, entry_tid=1' 2024-04-06T10:07:41.581 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=5, tag_tid=3, entry_tid=1' -a 'object_number=5, tag_tid=3, entry_tid=1' = 'object_number=5, tag_tid=2, entry_tid=1' 2024-04-06T10:07:41.581 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=5, tag_tid=2, entry_tid=1' '!=' 'object_number=5, tag_tid=2, entry_tid=1' 2024-04-06T10:07:41.581 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T10:07:41.581 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 2 2024-04-06T10:07:43.583 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror image_49 2024-04-06T10:07:43.583 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T10:07:43.583 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:07:43.583 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_49 2024-04-06T10:07:43.583 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T10:07:43.583 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n image_49 ']' 2024-04-06T10:07:43.583 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/image_49' 2024-04-06T10:07:43.583 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_49 2024-04-06T10:07:43.583 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T10:07:43.583 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T10:07:43.583 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T10:07:43.583 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T10:07:43.584 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T10:07:43.584 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T10:07:43.584 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T10:07:43.584 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T10:07:43.584 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T10:07:43.587 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T10:07:43.587 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_49 2024-04-06T10:07:43.587 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T10:07:43.587 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T10:07:43.587 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T10:07:43.587 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T10:07:43.587 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T10:07:43.587 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T10:07:43.587 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T10:07:43.587 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T10:07:43.587 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T10:07:43.588 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T10:07:43.588 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T10:07:43.588 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:07:43.588 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T10:07:43.588 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T10:07:43.588 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T10:07:43.588 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T10:07:43.588 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T10:07:43.588 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T10:07:43.588 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T10:07:43.588 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T10:07:43.588 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T10:07:43.588 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T10:07:43.588 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:07:43.588 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T10:07:43.589 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T10:07:43.589 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T10:07:43.589 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T10:07:43.589 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T10:07:43.589 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T10:07:43.589 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T10:07:43.610 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T10:07:43.610 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T10:07:43.610 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T10:07:43.610 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_49 2024-04-06T10:07:43.718 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:Invalid command: unused arguments: ['mirror/image_49'] 2024-04-06T10:07:43.718 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:rbd mirror flush : flush rbd mirror 2024-04-06T10:07:43.718 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:admin_socket: invalid command 2024-04-06T10:07:43.720 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T10:07:43.720 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/image_49 2024-04-06T10:07:43.720 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:1 2024-04-06T10:07:43.720 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T10:07:43.720 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T10:07:43.720 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-06T10:07:43.720 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:1 2024-04-06T10:07:43.720 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T10:07:43.721 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T10:07:43.721 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T10:07:43.721 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=1 2024-04-06T10:07:43.721 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-06T10:07:43.721 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T10:07:43.721 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:07:43.721 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=1 2024-04-06T10:07:43.721 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=1 2024-04-06T10:07:43.722 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-06T10:07:43.723 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:1 2024-04-06T10:07:43.723 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T10:07:43.723 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T10:07:43.723 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-06T10:07:43.723 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:1 2024-04-06T10:07:43.723 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T10:07:43.723 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T10:07:43.723 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:07:43.723 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=1 2024-04-06T10:07:43.723 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-06T10:07:43.723 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T10:07:43.723 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T10:07:43.723 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=1 2024-04-06T10:07:43.723 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=1 2024-04-06T10:07:43.724 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-06T10:07:43.740 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T10:07:43.740 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T10:07:43.741 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T10:07:43.741 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/image_49 2024-04-06T10:07:43.887 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T10:07:43.888 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror image_49 2024-04-06T10:07:43.888 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:07:43.888 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:07:43.888 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_49 2024-04-06T10:07:43.888 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror image_49 '' 2024-04-06T10:07:43.888 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:07:43.888 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:07:43.888 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_49 2024-04-06T10:07:43.888 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T10:07:43.889 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-image_49.status 2024-04-06T10:07:43.889 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-image_49.status 2024-04-06T10:07:43.889 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T10:07:43.891 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-image_49.status 2024-04-06T10:07:43.891 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_49 2024-04-06T10:07:43.891 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_49.status 2024-04-06T10:07:43.946 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 1 2024-04-06T10:07:43.946 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 1 2024-04-06T10:07:43.946 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T10:07:43.947 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=5, tag_tid=3, entry_tid=1], [object_number=4, tag_tid=3, entry_tid=0], [object_number=7, tag_tid=1, entry_tid=8215], [object_number=6, tag_tid=1, entry_tid=8214]]], state=connected] 2024-04-06T10:07:43.947 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=5, tag_tid=2, entry_tid=1], [object_number=4, tag_tid=2, entry_tid=0], [object_number=7, tag_tid=1, entry_tid=8215], [object_number=6, tag_tid=1, entry_tid=8214]]], state=connected] 2024-04-06T10:07:43.950 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_49.status 2024-04-06T10:07:43.952 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=5, tag_tid=3, entry_tid=1' 2024-04-06T10:07:43.953 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror image_49 2024-04-06T10:07:43.953 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:07:43.953 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:07:43.953 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_49 2024-04-06T10:07:43.953 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror image_49 '..*' 2024-04-06T10:07:43.953 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:07:43.953 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:07:43.953 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_49 2024-04-06T10:07:43.953 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T10:07:43.954 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-image_49.status 2024-04-06T10:07:43.954 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-image_49.status 2024-04-06T10:07:43.954 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T10:07:43.954 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-image_49.status 2024-04-06T10:07:43.954 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_49 2024-04-06T10:07:43.955 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_49.status 2024-04-06T10:07:44.000 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 1 2024-04-06T10:07:44.001 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 1 2024-04-06T10:07:44.001 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T10:07:44.001 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=5, tag_tid=3, entry_tid=1], [object_number=4, tag_tid=3, entry_tid=0], [object_number=7, tag_tid=1, entry_tid=8215], [object_number=6, tag_tid=1, entry_tid=8214]]], state=connected] 2024-04-06T10:07:44.001 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=5, tag_tid=2, entry_tid=1], [object_number=4, tag_tid=2, entry_tid=0], [object_number=7, tag_tid=1, entry_tid=8215], [object_number=6, tag_tid=1, entry_tid=8214]]], state=connected] 2024-04-06T10:07:44.004 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_49.status 2024-04-06T10:07:44.006 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=5, tag_tid=2, entry_tid=1' 2024-04-06T10:07:44.006 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=5, tag_tid=3, entry_tid=1' -a 'object_number=5, tag_tid=3, entry_tid=1' = 'object_number=5, tag_tid=2, entry_tid=1' 2024-04-06T10:07:44.006 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 'object_number=5, tag_tid=2, entry_tid=1' '!=' 'object_number=5, tag_tid=2, entry_tid=1' 2024-04-06T10:07:44.006 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T10:07:44.006 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 2 2024-04-06T10:07:46.008 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror image_49 2024-04-06T10:07:46.008 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T10:07:46.009 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:07:46.009 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_49 2024-04-06T10:07:46.009 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T10:07:46.009 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n image_49 ']' 2024-04-06T10:07:46.009 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/image_49' 2024-04-06T10:07:46.009 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_49 2024-04-06T10:07:46.009 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T10:07:46.009 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T10:07:46.009 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T10:07:46.009 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T10:07:46.009 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T10:07:46.009 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T10:07:46.009 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T10:07:46.010 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T10:07:46.010 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T10:07:46.013 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T10:07:46.013 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_49 2024-04-06T10:07:46.013 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T10:07:46.013 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T10:07:46.013 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T10:07:46.013 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T10:07:46.013 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T10:07:46.013 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T10:07:46.013 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T10:07:46.013 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T10:07:46.014 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T10:07:46.014 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T10:07:46.014 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T10:07:46.014 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:07:46.014 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T10:07:46.015 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T10:07:46.015 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T10:07:46.015 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T10:07:46.016 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T10:07:46.016 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T10:07:46.016 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T10:07:46.016 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T10:07:46.016 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T10:07:46.016 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T10:07:46.016 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:07:46.016 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T10:07:46.016 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T10:07:46.016 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T10:07:46.016 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T10:07:46.016 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T10:07:46.016 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T10:07:46.016 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T10:07:46.038 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T10:07:46.039 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T10:07:46.039 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T10:07:46.039 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_49 2024-04-06T10:07:46.146 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:Invalid command: unused arguments: ['mirror/image_49'] 2024-04-06T10:07:46.147 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:rbd mirror flush : flush rbd mirror 2024-04-06T10:07:46.147 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:admin_socket: invalid command 2024-04-06T10:07:46.149 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T10:07:46.149 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/image_49 2024-04-06T10:07:46.149 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:1 2024-04-06T10:07:46.149 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T10:07:46.149 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T10:07:46.149 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-06T10:07:46.149 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:1 2024-04-06T10:07:46.149 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T10:07:46.149 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T10:07:46.149 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T10:07:46.149 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=1 2024-04-06T10:07:46.150 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-06T10:07:46.150 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T10:07:46.150 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:07:46.150 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=1 2024-04-06T10:07:46.150 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=1 2024-04-06T10:07:46.151 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-06T10:07:46.151 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:1 2024-04-06T10:07:46.152 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T10:07:46.152 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T10:07:46.152 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-06T10:07:46.152 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:1 2024-04-06T10:07:46.152 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T10:07:46.152 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T10:07:46.152 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:07:46.152 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=1 2024-04-06T10:07:46.152 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-06T10:07:46.152 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T10:07:46.152 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T10:07:46.152 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=1 2024-04-06T10:07:46.152 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=1 2024-04-06T10:07:46.152 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-06T10:07:46.169 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T10:07:46.169 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T10:07:46.170 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T10:07:46.170 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/image_49 2024-04-06T10:07:46.319 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T10:07:46.320 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror image_49 2024-04-06T10:07:46.320 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:07:46.320 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:07:46.320 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_49 2024-04-06T10:07:46.320 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror image_49 '' 2024-04-06T10:07:46.320 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:07:46.320 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:07:46.320 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_49 2024-04-06T10:07:46.320 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T10:07:46.321 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-image_49.status 2024-04-06T10:07:46.321 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-image_49.status 2024-04-06T10:07:46.321 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T10:07:46.323 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-image_49.status 2024-04-06T10:07:46.323 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_49 2024-04-06T10:07:46.323 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_49.status 2024-04-06T10:07:46.378 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 1 2024-04-06T10:07:46.378 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 1 2024-04-06T10:07:46.378 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T10:07:46.378 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=5, tag_tid=3, entry_tid=1], [object_number=4, tag_tid=3, entry_tid=0], [object_number=7, tag_tid=1, entry_tid=8215], [object_number=6, tag_tid=1, entry_tid=8214]]], state=connected] 2024-04-06T10:07:46.378 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=5, tag_tid=3, entry_tid=1], [object_number=4, tag_tid=3, entry_tid=0], [object_number=7, tag_tid=1, entry_tid=8215], [object_number=6, tag_tid=1, entry_tid=8214]]], state=connected] 2024-04-06T10:07:46.382 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_49.status 2024-04-06T10:07:46.383 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=5, tag_tid=3, entry_tid=1' 2024-04-06T10:07:46.384 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror image_49 2024-04-06T10:07:46.384 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:07:46.384 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:07:46.385 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_49 2024-04-06T10:07:46.385 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror image_49 '..*' 2024-04-06T10:07:46.385 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:07:46.385 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:07:46.385 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_49 2024-04-06T10:07:46.385 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T10:07:46.385 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-image_49.status 2024-04-06T10:07:46.385 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-image_49.status 2024-04-06T10:07:46.385 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T10:07:46.385 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-image_49.status 2024-04-06T10:07:46.386 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_49 2024-04-06T10:07:46.386 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_49.status 2024-04-06T10:07:46.507 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 1 2024-04-06T10:07:46.507 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 1 2024-04-06T10:07:46.507 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T10:07:46.507 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=5, tag_tid=3, entry_tid=1], [object_number=4, tag_tid=3, entry_tid=0], [object_number=7, tag_tid=1, entry_tid=8215], [object_number=6, tag_tid=1, entry_tid=8214]]], state=connected] 2024-04-06T10:07:46.508 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=5, tag_tid=3, entry_tid=1], [object_number=4, tag_tid=3, entry_tid=0], [object_number=7, tag_tid=1, entry_tid=8215], [object_number=6, tag_tid=1, entry_tid=8214]]], state=connected] 2024-04-06T10:07:46.508 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_49.status 2024-04-06T10:07:46.510 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=5, tag_tid=3, entry_tid=1' 2024-04-06T10:07:46.510 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=5, tag_tid=3, entry_tid=1' -a 'object_number=5, tag_tid=3, entry_tid=1' = 'object_number=5, tag_tid=3, entry_tid=1' 2024-04-06T10:07:46.510 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T10:07:46.511 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_snap_present cluster1 mirror image_49 snap 2024-04-06T10:07:46.511 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T10:07:46.511 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:07:46.511 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_49 2024-04-06T10:07:46.511 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local snap_name=snap 2024-04-06T10:07:46.511 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local s 2024-04-06T10:07:46.511 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-04-06T10:07:46.511 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 1 2024-04-06T10:07:47.512 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 info mirror/image_49@snap 2024-04-06T10:07:47.584 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:rbd image 'image_49': 2024-04-06T10:07:47.584 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: size 128 MiB in 32 objects 2024-04-06T10:07:47.584 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: order 22 (4 MiB objects) 2024-04-06T10:07:47.584 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: snapshot_count: 1 2024-04-06T10:07:47.584 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: id: 10aca9f9d48a 2024-04-06T10:07:47.584 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: block_name_prefix: rbd_data.10aca9f9d48a 2024-04-06T10:07:47.584 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: format: 2 2024-04-06T10:07:47.584 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: features: layering, exclusive-lock, journaling 2024-04-06T10:07:47.584 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: op_features: 2024-04-06T10:07:47.584 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: flags: 2024-04-06T10:07:47.584 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: create_timestamp: Sat Apr 6 09:57:58 2024 2024-04-06T10:07:47.584 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: access_timestamp: Sat Apr 6 09:57:58 2024 2024-04-06T10:07:47.584 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: modify_timestamp: Sat Apr 6 09:58:59 2024 2024-04-06T10:07:47.585 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: protected: False 2024-04-06T10:07:47.585 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: journal: 10aca9f9d48a 2024-04-06T10:07:47.585 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: mirroring state: enabled 2024-04-06T10:07:47.585 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: mirroring mode: journal 2024-04-06T10:07:47.585 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: mirroring global id: 4669581c-8df4-4230-9014-f507eded1d43 2024-04-06T10:07:47.585 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: mirroring primary: false 2024-04-06T10:07:47.589 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T10:07:47.589 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ compare_image_snaps mirror image_49 snap 2024-04-06T10:07:47.589 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:07:47.589 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_49 2024-04-06T10:07:47.589 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local snap_name=snap 2024-04-06T10:07:47.589 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local ret=0 2024-04-06T10:07:47.589 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local rmt_export=/tmp/tmp.BhW6xKmROU/cluster2-mirror-image_49.export 2024-04-06T10:07:47.589 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loc_export=/tmp/tmp.BhW6xKmROU/cluster1-mirror-image_49.export 2024-04-06T10:07:47.589 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rm -f /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_49.export /tmp/tmp.BhW6xKmROU/cluster1-mirror-image_49.export 2024-04-06T10:07:47.590 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster2 -p mirror export image_49@snap /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_49.export 2024-04-06T10:07:47.794 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-04-06T10:07:47.803 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 -p mirror export image_49@snap /tmp/tmp.BhW6xKmROU/cluster1-mirror-image_49.export 2024-04-06T10:07:48.250 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-04-06T10:07:48.268 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmp /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_49.export /tmp/tmp.BhW6xKmROU/cluster1-mirror-image_49.export 2024-04-06T10:07:48.348 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rm -f /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_49.export /tmp/tmp.BhW6xKmROU/cluster1-mirror-image_49.export 2024-04-06T10:07:48.389 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T10:07:48.389 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-06T10:07:48.389 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ image=image_50 2024-04-06T10:07:48.389 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ create_snap cluster2 mirror image_50 snap 2024-04-06T10:07:48.389 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-04-06T10:07:48.389 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:07:48.389 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_50 2024-04-06T10:07:48.390 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local snap_name=snap 2024-04-06T10:07:48.390 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster2 -p mirror snap create image_50@snap --debug-rbd=20 --debug-journaler=20 2024-04-06T10:07:48.946 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_image_replay_started cluster1 mirror image_50 2024-04-06T10:07:48.947 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T10:07:48.947 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:07:48.947 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_50 2024-04-06T10:07:48.947 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_image_replay_state cluster1 mirror image_50 started 2024-04-06T10:07:48.947 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T10:07:48.947 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:07:48.947 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_50 2024-04-06T10:07:48.947 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state=started 2024-04-06T10:07:48.947 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local s 2024-04-06T10:07:48.947 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T10:07:48.947 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 1 2024-04-06T10:07:49.948 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test_image_replay_state cluster1 mirror image_50 started 2024-04-06T10:07:49.948 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T10:07:49.949 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:07:49.949 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_50 2024-04-06T10:07:49.949 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local test_state=started 2024-04-06T10:07:49.949 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local status_result 2024-04-06T10:07:49.949 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local current_state=stopped 2024-04-06T10:07:49.950 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_50 2024-04-06T10:07:49.950 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_instance=cluster1 2024-04-06T10:07:49.950 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -i state 2024-04-06T10:07:49.950 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ shift 2024-04-06T10:07:49.950 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T10:07:49.950 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance=cluster1 2024-04-06T10:07:49.951 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local loop_instance 2024-04-06T10:07:49.951 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T10:07:49.951 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sleep 0 2024-04-06T10:07:49.951 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-04-06T10:07:49.951 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ seq 0 3 2024-04-06T10:07:49.954 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T10:07:49.954 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_50 2024-04-06T10:07:49.954 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1:0 2024-04-06T10:07:49.954 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ shift 2024-04-06T10:07:49.955 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T10:07:49.955 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-04-06T10:07:49.955 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T10:07:49.955 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=cluster 2024-04-06T10:07:49.955 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T10:07:49.955 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:07:49.955 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T10:07:49.955 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T10:07:49.955 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval cluster=cluster1 2024-04-06T10:07:49.955 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ cluster=cluster1 2024-04-06T10:07:49.955 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T10:07:49.955 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T10:07:49.955 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-04-06T10:07:49.955 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local local_cluster=cluster1:0 2024-04-06T10:07:49.956 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local cluster=cluster1 2024-04-06T10:07:49.956 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local instance 2024-04-06T10:07:49.956 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T10:07:49.956 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local val=cluster1:0 2024-04-06T10:07:49.956 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local cluster_var_name=local_cluster 2024-04-06T10:07:49.956 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local instance_var_name=instance 2024-04-06T10:07:49.956 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ cluster=cluster1 2024-04-06T10:07:49.956 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T10:07:49.956 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ '[' 0 = cluster1:0 ']' 2024-04-06T10:07:49.956 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ eval local_cluster=cluster1 2024-04-06T10:07:49.956 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++++ local_cluster=cluster1 2024-04-06T10:07:49.956 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ eval instance=0 2024-04-06T10:07:49.956 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++++ instance=0 2024-04-06T10:07:49.956 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T10:07:49.975 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T10:07:49.976 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T10:07:49.976 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T10:07:49.976 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_50 2024-04-06T10:07:50.089 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:Invalid command: unused arguments: ['mirror/image_50'] 2024-04-06T10:07:50.089 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:rbd mirror status : get status for rbd mirror 2024-04-06T10:07:50.089 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:admin_socket: invalid command 2024-04-06T10:07:50.091 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T10:07:50.092 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_50 2024-04-06T10:07:50.092 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1:1 2024-04-06T10:07:50.092 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ shift 2024-04-06T10:07:50.092 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T10:07:50.092 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-04-06T10:07:50.092 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:1 2024-04-06T10:07:50.092 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=cluster 2024-04-06T10:07:50.092 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T10:07:50.092 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:07:50.092 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=1 2024-04-06T10:07:50.092 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-06T10:07:50.092 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval cluster=cluster1 2024-04-06T10:07:50.092 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ cluster=cluster1 2024-04-06T10:07:50.093 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=1 2024-04-06T10:07:50.093 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=1 2024-04-06T10:07:50.094 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-04-06T10:07:50.094 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local local_cluster=cluster1:1 2024-04-06T10:07:50.095 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local cluster=cluster1 2024-04-06T10:07:50.095 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local instance 2024-04-06T10:07:50.095 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-06T10:07:50.095 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local val=cluster1:1 2024-04-06T10:07:50.095 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local cluster_var_name=local_cluster 2024-04-06T10:07:50.095 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local instance_var_name=instance 2024-04-06T10:07:50.095 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ cluster=cluster1 2024-04-06T10:07:50.095 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=1 2024-04-06T10:07:50.095 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ '[' 1 = cluster1:1 ']' 2024-04-06T10:07:50.095 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ eval local_cluster=cluster1 2024-04-06T10:07:50.095 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++++ local_cluster=cluster1 2024-04-06T10:07:50.095 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ eval instance=1 2024-04-06T10:07:50.095 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++++ instance=1 2024-04-06T10:07:50.095 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-06T10:07:50.113 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T10:07:50.113 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T10:07:50.113 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T10:07:50.113 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_50 2024-04-06T10:07:50.267 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ return 0 2024-04-06T10:07:50.268 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ status_result=' "state": "Replaying"' 2024-04-06T10:07:50.268 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep -i Replaying 2024-04-06T10:07:50.268 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ echo ' "state": "Replaying"' 2024-04-06T10:07:50.269 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: "state": "Replaying" 2024-04-06T10:07:50.270 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ current_state=started 2024-04-06T10:07:50.271 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test started = started 2024-04-06T10:07:50.271 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T10:07:50.271 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_50 2024-04-06T10:07:50.271 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local local_cluster=cluster1 2024-04-06T10:07:50.271 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-04-06T10:07:50.271 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:07:50.271 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_50 2024-04-06T10:07:50.271 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' journal = journal ']' 2024-04-06T10:07:50.271 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_journal_replay_complete cluster1 cluster2 mirror image_50 2024-04-06T10:07:50.271 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local local_cluster=cluster1 2024-04-06T10:07:50.271 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-04-06T10:07:50.271 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:07:50.271 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_50 2024-04-06T10:07:50.271 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local s master_pos mirror_pos last_mirror_pos 2024-04-06T10:07:50.271 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local master_tag master_entry mirror_tag mirror_entry 2024-04-06T10:07:50.272 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T10:07:50.272 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T10:07:50.272 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.2 2024-04-06T10:07:50.472 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ flush cluster1 mirror image_50 2024-04-06T10:07:50.472 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T10:07:50.472 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:07:50.473 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_50 2024-04-06T10:07:50.473 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'cmd=rbd mirror flush' 2024-04-06T10:07:50.473 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n image_50 ']' 2024-04-06T10:07:50.473 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmd='rbd mirror flush mirror/image_50' 2024-04-06T10:07:50.473 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_50 2024-04-06T10:07:50.473 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_instance=cluster1 2024-04-06T10:07:50.473 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T10:07:50.473 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T10:07:50.473 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance=cluster1 2024-04-06T10:07:50.473 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loop_instance 2024-04-06T10:07:50.473 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T10:07:50.474 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T10:07:50.474 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-04-06T10:07:50.474 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T10:07:50.477 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T10:07:50.477 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_50 2024-04-06T10:07:50.477 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T10:07:50.477 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T10:07:50.477 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T10:07:50.477 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T10:07:50.477 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-04-06T10:07:50.477 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T10:07:50.477 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T10:07:50.477 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T10:07:50.477 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T10:07:50.477 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T10:07:50.477 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T10:07:50.478 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:07:50.478 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T10:07:50.478 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T10:07:50.479 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-06T10:07:50.479 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-04-06T10:07:50.479 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T10:07:50.479 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T10:07:50.479 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T10:07:50.479 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-04-06T10:07:50.479 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T10:07:50.479 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T10:07:50.479 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:07:50.479 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T10:07:50.479 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T10:07:50.480 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T10:07:50.480 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T10:07:50.480 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-04-06T10:07:50.480 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-04-06T10:07:50.480 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T10:07:50.504 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T10:07:50.505 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T10:07:50.505 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T10:07:50.505 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_50 2024-04-06T10:07:50.617 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:Invalid command: unused arguments: ['mirror/image_50'] 2024-04-06T10:07:50.618 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:rbd mirror flush : flush rbd mirror 2024-04-06T10:07:50.618 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:admin_socket: invalid command 2024-04-06T10:07:50.620 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T10:07:50.620 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/image_50 2024-04-06T10:07:50.620 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:1 2024-04-06T10:07:50.620 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T10:07:50.620 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T10:07:50.620 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-06T10:07:50.620 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:1 2024-04-06T10:07:50.620 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T10:07:50.620 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T10:07:50.620 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T10:07:50.621 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=1 2024-04-06T10:07:50.621 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-06T10:07:50.621 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T10:07:50.621 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:07:50.621 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=1 2024-04-06T10:07:50.621 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=1 2024-04-06T10:07:50.622 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-04-06T10:07:50.622 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:1 2024-04-06T10:07:50.623 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T10:07:50.623 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-04-06T10:07:50.623 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-06T10:07:50.623 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:1 2024-04-06T10:07:50.623 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-04-06T10:07:50.623 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-04-06T10:07:50.623 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:07:50.623 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=1 2024-04-06T10:07:50.623 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-06T10:07:50.623 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-04-06T10:07:50.623 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-04-06T10:07:50.623 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=1 2024-04-06T10:07:50.623 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=1 2024-04-06T10:07:50.623 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-06T10:07:50.641 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T10:07:50.641 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T10:07:50.642 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T10:07:50.642 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/image_50 2024-04-06T10:07:50.788 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T10:07:50.790 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_master_journal_position cluster2 mirror image_50 2024-04-06T10:07:50.790 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:07:50.790 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:07:50.790 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_50 2024-04-06T10:07:50.790 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror image_50 '' 2024-04-06T10:07:50.790 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:07:50.790 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:07:50.790 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_50 2024-04-06T10:07:50.790 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local id_regexp= 2024-04-06T10:07:50.790 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-image_50.status 2024-04-06T10:07:50.790 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-image_50.status 2024-04-06T10:07:50.791 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T10:07:50.792 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-image_50.status 2024-04-06T10:07:50.793 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_50 2024-04-06T10:07:50.793 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_50.status 2024-04-06T10:07:50.839 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 1 2024-04-06T10:07:50.839 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 1 2024-04-06T10:07:50.839 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T10:07:50.839 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=5, tag_tid=3, entry_tid=1], [object_number=4, tag_tid=3, entry_tid=0], [object_number=6, tag_tid=1, entry_tid=4482], [object_number=7, tag_tid=1, entry_tid=4479]]], state=connected] 2024-04-06T10:07:50.839 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=5, tag_tid=3, entry_tid=1], [object_number=4, tag_tid=3, entry_tid=0], [object_number=6, tag_tid=1, entry_tid=4482], [object_number=7, tag_tid=1, entry_tid=4479]]], state=connected] 2024-04-06T10:07:50.842 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_50.status 2024-04-06T10:07:50.844 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ master_pos='object_number=5, tag_tid=3, entry_tid=1' 2024-04-06T10:07:50.845 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_mirror_journal_position cluster2 mirror image_50 2024-04-06T10:07:50.845 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:07:50.845 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:07:50.845 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_50 2024-04-06T10:07:50.845 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_journal_position cluster2 mirror image_50 '..*' 2024-04-06T10:07:50.845 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:07:50.845 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:07:50.845 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_50 2024-04-06T10:07:50.845 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local 'id_regexp=..*' 2024-04-06T10:07:50.845 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ mkfname cluster2-mirror-image_50.status 2024-04-06T10:07:50.846 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo cluster2-mirror-image_50.status 2024-04-06T10:07:50.846 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ sed -e 's|[/ ]|_|g' 2024-04-06T10:07:50.846 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local status_log=/tmp/tmp.BhW6xKmROU/cluster2-mirror-image_50.status 2024-04-06T10:07:50.846 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_50 2024-04-06T10:07:50.846 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ tee /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_50.status 2024-04-06T10:07:50.895 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:minimum_set: 1 2024-04-06T10:07:50.895 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:active_set: 1 2024-04-06T10:07:50.895 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:registered clients: 2024-04-06T10:07:50.895 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=, commit_position=[positions=[[object_number=5, tag_tid=3, entry_tid=1], [object_number=4, tag_tid=3, entry_tid=0], [object_number=6, tag_tid=1, entry_tid=4482], [object_number=7, tag_tid=1, entry_tid=4479]]], state=connected] 2024-04-06T10:07:50.895 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: [id=24230d5c-19e3-4860-88b4-64913f21bf9f, commit_position=[positions=[[object_number=5, tag_tid=3, entry_tid=1], [object_number=4, tag_tid=3, entry_tid=0], [object_number=6, tag_tid=1, entry_tid=4482], [object_number=7, tag_tid=1, entry_tid=4479]]], state=connected] 2024-04-06T10:07:50.900 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_50.status 2024-04-06T10:07:50.902 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_pos='object_number=5, tag_tid=3, entry_tid=1' 2024-04-06T10:07:50.902 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 'object_number=5, tag_tid=3, entry_tid=1' -a 'object_number=5, tag_tid=3, entry_tid=1' = 'object_number=5, tag_tid=3, entry_tid=1' 2024-04-06T10:07:50.902 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T10:07:50.902 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_snap_present cluster1 mirror image_50 snap 2024-04-06T10:07:50.902 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T10:07:50.902 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:07:50.903 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_50 2024-04-06T10:07:50.903 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local snap_name=snap 2024-04-06T10:07:50.903 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local s 2024-04-06T10:07:50.903 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-04-06T10:07:50.903 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 1 2024-04-06T10:07:51.903 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 info mirror/image_50@snap 2024-04-06T10:07:51.967 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:rbd image 'image_50': 2024-04-06T10:07:51.968 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: size 128 MiB in 32 objects 2024-04-06T10:07:51.968 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: order 22 (4 MiB objects) 2024-04-06T10:07:51.968 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: snapshot_count: 1 2024-04-06T10:07:51.968 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: id: 10ac1a6c34d1 2024-04-06T10:07:51.968 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: block_name_prefix: rbd_data.10ac1a6c34d1 2024-04-06T10:07:51.968 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: format: 2 2024-04-06T10:07:51.968 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: features: layering, exclusive-lock, journaling 2024-04-06T10:07:51.968 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: op_features: 2024-04-06T10:07:51.968 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: flags: 2024-04-06T10:07:51.968 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: create_timestamp: Sat Apr 6 09:58:09 2024 2024-04-06T10:07:51.968 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: access_timestamp: Sat Apr 6 09:58:09 2024 2024-04-06T10:07:51.968 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: modify_timestamp: Sat Apr 6 09:59:10 2024 2024-04-06T10:07:51.968 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: protected: False 2024-04-06T10:07:51.968 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: journal: 10ac1a6c34d1 2024-04-06T10:07:51.968 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: mirroring state: enabled 2024-04-06T10:07:51.969 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: mirroring mode: journal 2024-04-06T10:07:51.969 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: mirroring global id: 9b282988-f3bb-4704-afa8-7aecc6d432ee 2024-04-06T10:07:51.969 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: mirroring primary: false 2024-04-06T10:07:51.973 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T10:07:51.973 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ compare_image_snaps mirror image_50 snap 2024-04-06T10:07:51.973 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:07:51.973 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_50 2024-04-06T10:07:51.973 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local snap_name=snap 2024-04-06T10:07:51.973 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local ret=0 2024-04-06T10:07:51.973 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local rmt_export=/tmp/tmp.BhW6xKmROU/cluster2-mirror-image_50.export 2024-04-06T10:07:51.973 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local loc_export=/tmp/tmp.BhW6xKmROU/cluster1-mirror-image_50.export 2024-04-06T10:07:51.973 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rm -f /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_50.export /tmp/tmp.BhW6xKmROU/cluster1-mirror-image_50.export 2024-04-06T10:07:51.974 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster2 -p mirror export image_50@snap /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_50.export 2024-04-06T10:07:52.153 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-04-06T10:07:52.161 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 -p mirror export image_50@snap /tmp/tmp.BhW6xKmROU/cluster1-mirror-image_50.export 2024-04-06T10:07:52.545 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-04-06T10:07:52.559 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cmp /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_50.export /tmp/tmp.BhW6xKmROU/cluster1-mirror-image_50.export 2024-04-06T10:07:53.006 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rm -f /tmp/tmp.BhW6xKmROU/cluster2-mirror-image_50.export /tmp/tmp.BhW6xKmROU/cluster1-mirror-image_50.export 2024-04-06T10:07:53.047 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T10:07:53.048 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ testlog 'TEST: delete many images' 2024-04-06T10:07:53.048 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ date '+%F %T' 2024-04-06T10:07:53.048 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ tee -a /tmp/tmp.BhW6xKmROU/rbd-mirror.test.log 2024-04-06T10:07:53.049 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ echo 2024-04-06 10:07:53 TEST: delete many images 2024-04-06T10:07:53.049 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:2024-04-06 10:07:53 TEST: delete many images 2024-04-06T10:07:53.050 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 1 50 2024-04-06T10:07:53.051 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-06T10:07:53.051 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ image=image_1 2024-04-06T10:07:53.051 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ remove_snapshot cluster2 mirror image_1 snap 2024-04-06T10:07:53.051 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-04-06T10:07:53.051 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:07:53.051 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_1 2024-04-06T10:07:53.051 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local snap=snap 2024-04-06T10:07:53.051 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster2 snap rm mirror/image_1@snap 2024-04-06T10:07:53.333 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: Removing snap: 100% complete...done. 2024-04-06T10:07:53.353 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ remove_image_retry cluster2 mirror image_1 2024-04-06T10:07:53.353 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-04-06T10:07:53.353 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:07:53.353 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_1 2024-04-06T10:07:53.353 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 16 32 2024-04-06T10:07:53.353 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T10:07:53.354 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ remove_image cluster2 mirror image_1 2024-04-06T10:07:53.354 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-04-06T10:07:53.354 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:07:53.354 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_1 2024-04-06T10:07:53.354 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_1 2024-04-06T10:07:53.404 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster=cluster2 rm mirror/image_1 2024-04-06T10:07:53.637 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-04-06T10:07:53.640 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T10:07:53.640 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-06T10:07:53.641 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ image=image_2 2024-04-06T10:07:53.641 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ remove_snapshot cluster2 mirror image_2 snap 2024-04-06T10:07:53.641 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-04-06T10:07:53.641 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:07:53.641 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_2 2024-04-06T10:07:53.641 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local snap=snap 2024-04-06T10:07:53.641 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster2 snap rm mirror/image_2@snap 2024-04-06T10:07:54.332 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: Removing snap: 100% complete...done. 2024-04-06T10:07:54.356 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ remove_image_retry cluster2 mirror image_2 2024-04-06T10:07:54.356 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-04-06T10:07:54.356 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:07:54.356 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_2 2024-04-06T10:07:54.356 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 16 32 2024-04-06T10:07:54.357 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T10:07:54.358 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ remove_image cluster2 mirror image_2 2024-04-06T10:07:54.358 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-04-06T10:07:54.358 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:07:54.358 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_2 2024-04-06T10:07:54.358 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_2 2024-04-06T10:07:54.411 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster=cluster2 rm mirror/image_2 2024-04-06T10:07:54.489 INFO:tasks.ceph.cluster2.osd.2.smithi192.stderr:2024-04-06T10:07:54.485+0000 7fa534aba640 -1 reset not still connected to 0x55933fc2c680 2024-04-06T10:07:54.489 INFO:tasks.ceph.cluster2.osd.2.smithi192.stderr:2024-04-06T10:07:54.485+0000 7fa534aba640 -1 reset not still connected to 0x559348ab0680 2024-04-06T10:07:54.527 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T10:07:54.521+0000 7f2e84d66640 -1 reset not still connected to 0x560d27567380 2024-04-06T10:07:54.527 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T10:07:54.521+0000 7f2e84d66640 -1 reset not still connected to 0x560d27839380 2024-04-06T10:07:54.527 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T10:07:54.529+0000 7f2e84d66640 -1 reset not still connected to 0x560d27839930 2024-04-06T10:07:54.527 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T10:07:54.529+0000 7f2e84d66640 -1 reset not still connected to 0x560d27c3edd0 2024-04-06T10:07:54.528 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T10:07:54.529+0000 7f2e84d66640 -1 reset not still connected to 0x560d27c3f2b0 2024-04-06T10:07:54.532 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T10:07:54.533+0000 7f2e84d66640 -1 reset not still connected to 0x560d281875f0 2024-04-06T10:07:54.536 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T10:07:54.537+0000 7f2e84d66640 -1 reset not still connected to 0x560d294908f0 2024-04-06T10:07:54.540 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T10:07:54.537+0000 7f2e84d66640 -1 reset not still connected to 0x560d294916c0 2024-04-06T10:07:54.540 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T10:07:54.537+0000 7f2e84d66640 -1 reset not still connected to 0x560d29cd8f70 2024-04-06T10:07:54.541 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T10:07:54.537+0000 7f2e84d66640 -1 reset not still connected to 0x560d29cd9ba0 2024-04-06T10:07:54.541 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T10:07:54.537+0000 7f2e84d66640 -1 reset not still connected to 0x560d29d5a340 2024-04-06T10:07:54.541 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T10:07:54.537+0000 7f2e84d66640 -1 reset not still connected to 0x560d29d5b520 2024-04-06T10:07:54.541 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T10:07:54.537+0000 7f2e84d66640 -1 reset not still connected to 0x560d2a17d450 2024-04-06T10:07:54.541 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T10:07:54.537+0000 7f2e84d66640 -1 reset not still connected to 0x560d2a204b60 2024-04-06T10:07:54.541 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T10:07:54.537+0000 7f2e84d66640 -1 reset not still connected to 0x560d2a205ba0 2024-04-06T10:07:54.541 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T10:07:54.537+0000 7f2e84d66640 -1 reset not still connected to 0x560d2a2336c0 2024-04-06T10:07:54.541 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T10:07:54.537+0000 7f2e84d66640 -1 reset not still connected to 0x560d2a570000 2024-04-06T10:07:54.541 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T10:07:54.537+0000 7f2e84d66640 -1 reset not still connected to 0x560d2a5705b0 2024-04-06T10:07:54.541 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T10:07:54.537+0000 7f2e84d66640 -1 reset not still connected to 0x560d35349380 2024-04-06T10:07:54.541 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T10:07:54.537+0000 7f2e84d66640 -1 reset not still connected to 0x560d3620b520 2024-04-06T10:07:54.541 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T10:07:54.537+0000 7f2e84d66640 -1 reset not still connected to 0x560d3620bee0 2024-04-06T10:07:54.541 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T10:07:54.537+0000 7f2e84d66640 -1 reset not still connected to 0x560d36666d00 2024-04-06T10:07:54.542 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T10:07:54.537+0000 7f2e84d66640 -1 reset not still connected to 0x560d36912ea0 2024-04-06T10:07:54.543 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T10:07:54.545+0000 7f2e84d66640 -1 reset not still connected to 0x560d381dcc30 2024-04-06T10:07:54.544 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T10:07:54.545+0000 7f2e84d66640 -1 reset not still connected to 0x560d381dd5f0 2024-04-06T10:07:54.544 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T10:07:54.545+0000 7f2e84d66640 -1 reset not still connected to 0x560d390dc8f0 2024-04-06T10:07:54.544 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T10:07:54.545+0000 7f2e84d66640 -1 reset not still connected to 0x560d390dca90 2024-04-06T10:07:54.544 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T10:07:54.545+0000 7f2e84d66640 -1 reset not still connected to 0x560d47e9aa90 2024-04-06T10:07:54.544 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T10:07:54.545+0000 7f2e84d66640 -1 reset not still connected to 0x560d4bc7e8f0 2024-04-06T10:07:54.544 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T10:07:54.545+0000 7f2e84d66640 -1 reset not still connected to 0x560d4bc7ec30 2024-04-06T10:07:54.544 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T10:07:54.545+0000 7f2e84d66640 -1 reset not still connected to 0x560d50adba00 2024-04-06T10:07:54.679 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-04-06T10:07:54.684 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T10:07:54.684 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-06T10:07:54.684 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ image=image_3 2024-04-06T10:07:54.684 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ remove_snapshot cluster2 mirror image_3 snap 2024-04-06T10:07:54.684 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-04-06T10:07:54.684 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:07:54.684 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_3 2024-04-06T10:07:54.684 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local snap=snap 2024-04-06T10:07:54.685 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster2 snap rm mirror/image_3@snap 2024-04-06T10:07:55.339 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: Removing snap: 100% complete...done. 2024-04-06T10:07:55.360 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ remove_image_retry cluster2 mirror image_3 2024-04-06T10:07:55.360 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-04-06T10:07:55.361 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:07:55.361 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_3 2024-04-06T10:07:55.361 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 16 32 2024-04-06T10:07:55.361 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T10:07:55.362 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ remove_image cluster2 mirror image_3 2024-04-06T10:07:55.362 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-04-06T10:07:55.362 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:07:55.363 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_3 2024-04-06T10:07:55.363 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_3 2024-04-06T10:07:55.415 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster=cluster2 rm mirror/image_3 2024-04-06T10:07:55.880 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-04-06T10:07:55.887 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T10:07:55.887 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-06T10:07:55.887 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ image=image_4 2024-04-06T10:07:55.887 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ remove_snapshot cluster2 mirror image_4 snap 2024-04-06T10:07:55.887 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-04-06T10:07:55.887 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:07:55.887 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_4 2024-04-06T10:07:55.887 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local snap=snap 2024-04-06T10:07:55.887 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster2 snap rm mirror/image_4@snap 2024-04-06T10:07:56.933 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: Removing snap: 100% complete...done. 2024-04-06T10:07:56.954 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ remove_image_retry cluster2 mirror image_4 2024-04-06T10:07:56.954 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-04-06T10:07:56.954 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:07:56.954 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_4 2024-04-06T10:07:56.954 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 16 32 2024-04-06T10:07:56.954 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T10:07:56.955 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ remove_image cluster2 mirror image_4 2024-04-06T10:07:56.956 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-04-06T10:07:56.956 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:07:56.956 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_4 2024-04-06T10:07:56.956 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_4 2024-04-06T10:07:57.010 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster=cluster2 rm mirror/image_4 2024-04-06T10:07:57.269 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-04-06T10:07:57.273 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T10:07:57.273 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-06T10:07:57.273 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ image=image_5 2024-04-06T10:07:57.274 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ remove_snapshot cluster2 mirror image_5 snap 2024-04-06T10:07:57.274 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-04-06T10:07:57.274 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:07:57.274 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_5 2024-04-06T10:07:57.274 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local snap=snap 2024-04-06T10:07:57.274 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster2 snap rm mirror/image_5@snap 2024-04-06T10:07:57.930 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: Removing snap: 100% complete...done. 2024-04-06T10:07:57.948 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ remove_image_retry cluster2 mirror image_5 2024-04-06T10:07:57.948 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-04-06T10:07:57.948 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:07:57.948 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_5 2024-04-06T10:07:57.948 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 16 32 2024-04-06T10:07:57.948 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T10:07:57.949 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ remove_image cluster2 mirror image_5 2024-04-06T10:07:57.949 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-04-06T10:07:57.949 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:07:57.949 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_5 2024-04-06T10:07:57.949 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_5 2024-04-06T10:07:58.004 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster=cluster2 rm mirror/image_5 2024-04-06T10:07:58.502 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-04-06T10:07:58.504 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T10:07:58.505 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-06T10:07:58.505 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ image=image_6 2024-04-06T10:07:58.505 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ remove_snapshot cluster2 mirror image_6 snap 2024-04-06T10:07:58.505 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-04-06T10:07:58.505 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:07:58.505 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_6 2024-04-06T10:07:58.505 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local snap=snap 2024-04-06T10:07:58.505 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster2 snap rm mirror/image_6@snap 2024-04-06T10:07:58.953 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: Removing snap: 100% complete...done. 2024-04-06T10:07:58.975 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ remove_image_retry cluster2 mirror image_6 2024-04-06T10:07:58.975 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-04-06T10:07:58.975 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:07:58.975 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_6 2024-04-06T10:07:58.975 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 16 32 2024-04-06T10:07:58.975 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T10:07:58.976 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ remove_image cluster2 mirror image_6 2024-04-06T10:07:58.976 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-04-06T10:07:58.976 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:07:58.976 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_6 2024-04-06T10:07:58.976 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_6 2024-04-06T10:07:59.031 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster=cluster2 rm mirror/image_6 2024-04-06T10:07:59.262 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-04-06T10:07:59.267 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T10:07:59.267 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-06T10:07:59.267 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ image=image_7 2024-04-06T10:07:59.267 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ remove_snapshot cluster2 mirror image_7 snap 2024-04-06T10:07:59.267 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-04-06T10:07:59.267 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:07:59.267 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_7 2024-04-06T10:07:59.267 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local snap=snap 2024-04-06T10:07:59.267 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster2 snap rm mirror/image_7@snap 2024-04-06T10:07:59.944 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: Removing snap: 100% complete...done. 2024-04-06T10:07:59.967 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ remove_image_retry cluster2 mirror image_7 2024-04-06T10:07:59.967 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-04-06T10:07:59.967 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:07:59.967 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_7 2024-04-06T10:07:59.968 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 16 32 2024-04-06T10:07:59.968 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T10:07:59.969 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ remove_image cluster2 mirror image_7 2024-04-06T10:07:59.969 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-04-06T10:07:59.969 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:07:59.969 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_7 2024-04-06T10:07:59.969 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_7 2024-04-06T10:08:00.026 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster=cluster2 rm mirror/image_7 2024-04-06T10:08:00.263 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-04-06T10:08:00.267 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T10:08:00.268 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-06T10:08:00.268 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ image=image_8 2024-04-06T10:08:00.268 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ remove_snapshot cluster2 mirror image_8 snap 2024-04-06T10:08:00.268 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-04-06T10:08:00.268 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:08:00.268 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_8 2024-04-06T10:08:00.268 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local snap=snap 2024-04-06T10:08:00.268 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster2 snap rm mirror/image_8@snap 2024-04-06T10:08:00.920 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: Removing snap: 100% complete...done. 2024-04-06T10:08:00.941 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ remove_image_retry cluster2 mirror image_8 2024-04-06T10:08:00.941 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-04-06T10:08:00.941 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:08:00.941 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_8 2024-04-06T10:08:00.941 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 16 32 2024-04-06T10:08:00.941 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T10:08:00.942 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ remove_image cluster2 mirror image_8 2024-04-06T10:08:00.942 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-04-06T10:08:00.942 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:08:00.942 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_8 2024-04-06T10:08:00.942 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_8 2024-04-06T10:08:00.991 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster=cluster2 rm mirror/image_8 2024-04-06T10:08:01.262 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-04-06T10:08:01.267 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T10:08:01.267 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-06T10:08:01.267 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ image=image_9 2024-04-06T10:08:01.267 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ remove_snapshot cluster2 mirror image_9 snap 2024-04-06T10:08:01.267 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-04-06T10:08:01.267 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:08:01.267 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_9 2024-04-06T10:08:01.267 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local snap=snap 2024-04-06T10:08:01.267 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster2 snap rm mirror/image_9@snap 2024-04-06T10:08:01.952 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: Removing snap: 100% complete...done. 2024-04-06T10:08:01.972 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ remove_image_retry cluster2 mirror image_9 2024-04-06T10:08:01.972 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-04-06T10:08:01.972 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:08:01.972 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_9 2024-04-06T10:08:01.972 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 16 32 2024-04-06T10:08:01.972 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T10:08:01.973 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ remove_image cluster2 mirror image_9 2024-04-06T10:08:01.973 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-04-06T10:08:01.973 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:08:01.973 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_9 2024-04-06T10:08:01.973 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_9 2024-04-06T10:08:02.025 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster=cluster2 rm mirror/image_9 2024-04-06T10:08:02.290 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-04-06T10:08:02.294 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T10:08:02.294 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-06T10:08:02.294 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ image=image_10 2024-04-06T10:08:02.294 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ remove_snapshot cluster2 mirror image_10 snap 2024-04-06T10:08:02.294 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-04-06T10:08:02.294 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:08:02.294 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_10 2024-04-06T10:08:02.294 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local snap=snap 2024-04-06T10:08:02.295 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster2 snap rm mirror/image_10@snap 2024-04-06T10:08:02.358 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T10:08:02.353+0000 7f2e84d66640 -1 reset not still connected to 0x560d381dd5f0 2024-04-06T10:08:02.974 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: Removing snap: 100% complete...done. 2024-04-06T10:08:02.997 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ remove_image_retry cluster2 mirror image_10 2024-04-06T10:08:02.997 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-04-06T10:08:02.997 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:08:02.997 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_10 2024-04-06T10:08:02.998 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 16 32 2024-04-06T10:08:02.998 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T10:08:02.999 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ remove_image cluster2 mirror image_10 2024-04-06T10:08:02.999 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-04-06T10:08:02.999 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:08:02.999 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_10 2024-04-06T10:08:02.999 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_10 2024-04-06T10:08:03.055 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster=cluster2 rm mirror/image_10 2024-04-06T10:08:03.304 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-04-06T10:08:03.309 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T10:08:03.309 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-06T10:08:03.309 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ image=image_11 2024-04-06T10:08:03.309 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ remove_snapshot cluster2 mirror image_11 snap 2024-04-06T10:08:03.309 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-04-06T10:08:03.309 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:08:03.309 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_11 2024-04-06T10:08:03.309 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local snap=snap 2024-04-06T10:08:03.309 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster2 snap rm mirror/image_11@snap 2024-04-06T10:08:03.977 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: Removing snap: 100% complete...done. 2024-04-06T10:08:03.993 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ remove_image_retry cluster2 mirror image_11 2024-04-06T10:08:03.993 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-04-06T10:08:03.993 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:08:03.993 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_11 2024-04-06T10:08:03.994 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 16 32 2024-04-06T10:08:03.994 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T10:08:03.995 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ remove_image cluster2 mirror image_11 2024-04-06T10:08:03.995 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-04-06T10:08:03.995 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:08:03.995 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_11 2024-04-06T10:08:03.995 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_11 2024-04-06T10:08:04.047 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster=cluster2 rm mirror/image_11 2024-04-06T10:08:04.288 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-04-06T10:08:04.292 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T10:08:04.292 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-06T10:08:04.292 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ image=image_12 2024-04-06T10:08:04.292 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ remove_snapshot cluster2 mirror image_12 snap 2024-04-06T10:08:04.292 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-04-06T10:08:04.293 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:08:04.293 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_12 2024-04-06T10:08:04.293 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local snap=snap 2024-04-06T10:08:04.293 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster2 snap rm mirror/image_12@snap 2024-04-06T10:08:04.975 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: Removing snap: 100% complete...done. 2024-04-06T10:08:04.995 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ remove_image_retry cluster2 mirror image_12 2024-04-06T10:08:04.995 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-04-06T10:08:04.996 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:08:04.996 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_12 2024-04-06T10:08:04.996 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 16 32 2024-04-06T10:08:04.996 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T10:08:04.996 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ remove_image cluster2 mirror image_12 2024-04-06T10:08:04.997 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-04-06T10:08:04.997 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:08:04.997 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_12 2024-04-06T10:08:04.997 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_12 2024-04-06T10:08:05.050 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster=cluster2 rm mirror/image_12 2024-04-06T10:08:05.264 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-04-06T10:08:05.270 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T10:08:05.270 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-06T10:08:05.270 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ image=image_13 2024-04-06T10:08:05.270 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ remove_snapshot cluster2 mirror image_13 snap 2024-04-06T10:08:05.270 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-04-06T10:08:05.270 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:08:05.270 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_13 2024-04-06T10:08:05.270 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local snap=snap 2024-04-06T10:08:05.270 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster2 snap rm mirror/image_13@snap 2024-04-06T10:08:05.585 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T10:08:05.581+0000 7f2e84d66640 -1 reset not still connected to 0x560d50adba00 2024-04-06T10:08:05.938 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: Removing snap: 100% complete...done. 2024-04-06T10:08:05.958 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ remove_image_retry cluster2 mirror image_13 2024-04-06T10:08:05.958 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-04-06T10:08:05.958 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:08:05.958 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_13 2024-04-06T10:08:05.958 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 16 32 2024-04-06T10:08:05.959 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T10:08:05.960 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ remove_image cluster2 mirror image_13 2024-04-06T10:08:05.960 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-04-06T10:08:05.960 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:08:05.960 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_13 2024-04-06T10:08:05.960 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_13 2024-04-06T10:08:06.012 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster=cluster2 rm mirror/image_13 2024-04-06T10:08:06.247 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-04-06T10:08:06.252 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T10:08:06.252 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-06T10:08:06.252 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ image=image_14 2024-04-06T10:08:06.252 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ remove_snapshot cluster2 mirror image_14 snap 2024-04-06T10:08:06.252 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-04-06T10:08:06.252 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:08:06.252 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_14 2024-04-06T10:08:06.252 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local snap=snap 2024-04-06T10:08:06.253 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster2 snap rm mirror/image_14@snap 2024-04-06T10:08:07.004 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: Removing snap: 100% complete...done. 2024-04-06T10:08:07.028 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ remove_image_retry cluster2 mirror image_14 2024-04-06T10:08:07.028 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-04-06T10:08:07.028 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:08:07.028 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_14 2024-04-06T10:08:07.028 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 16 32 2024-04-06T10:08:07.028 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T10:08:07.029 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ remove_image cluster2 mirror image_14 2024-04-06T10:08:07.029 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-04-06T10:08:07.029 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:08:07.030 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_14 2024-04-06T10:08:07.030 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_14 2024-04-06T10:08:07.084 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster=cluster2 rm mirror/image_14 2024-04-06T10:08:07.298 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-04-06T10:08:07.302 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T10:08:07.302 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-06T10:08:07.303 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ image=image_15 2024-04-06T10:08:07.303 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ remove_snapshot cluster2 mirror image_15 snap 2024-04-06T10:08:07.303 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-04-06T10:08:07.303 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:08:07.303 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_15 2024-04-06T10:08:07.303 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local snap=snap 2024-04-06T10:08:07.303 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster2 snap rm mirror/image_15@snap 2024-04-06T10:08:08.002 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: Removing snap: 100% complete...done. 2024-04-06T10:08:08.024 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ remove_image_retry cluster2 mirror image_15 2024-04-06T10:08:08.024 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-04-06T10:08:08.024 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:08:08.024 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_15 2024-04-06T10:08:08.024 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 16 32 2024-04-06T10:08:08.024 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T10:08:08.025 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ remove_image cluster2 mirror image_15 2024-04-06T10:08:08.025 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-04-06T10:08:08.025 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:08:08.025 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_15 2024-04-06T10:08:08.025 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_15 2024-04-06T10:08:08.080 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster=cluster2 rm mirror/image_15 2024-04-06T10:08:08.282 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-04-06T10:08:08.286 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T10:08:08.287 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-06T10:08:08.287 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ image=image_16 2024-04-06T10:08:08.287 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ remove_snapshot cluster2 mirror image_16 snap 2024-04-06T10:08:08.287 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-04-06T10:08:08.287 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:08:08.287 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_16 2024-04-06T10:08:08.287 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local snap=snap 2024-04-06T10:08:08.287 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster2 snap rm mirror/image_16@snap 2024-04-06T10:08:09.004 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: Removing snap: 100% complete...done. 2024-04-06T10:08:09.027 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ remove_image_retry cluster2 mirror image_16 2024-04-06T10:08:09.027 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-04-06T10:08:09.027 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:08:09.027 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_16 2024-04-06T10:08:09.028 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 16 32 2024-04-06T10:08:09.028 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T10:08:09.029 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ remove_image cluster2 mirror image_16 2024-04-06T10:08:09.029 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-04-06T10:08:09.029 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:08:09.029 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_16 2024-04-06T10:08:09.029 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_16 2024-04-06T10:08:09.083 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster=cluster2 rm mirror/image_16 2024-04-06T10:08:09.292 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-04-06T10:08:09.296 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T10:08:09.296 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-06T10:08:09.296 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ image=image_17 2024-04-06T10:08:09.296 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ remove_snapshot cluster2 mirror image_17 snap 2024-04-06T10:08:09.296 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-04-06T10:08:09.297 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:08:09.297 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_17 2024-04-06T10:08:09.297 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local snap=snap 2024-04-06T10:08:09.297 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster2 snap rm mirror/image_17@snap 2024-04-06T10:08:10.017 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: Removing snap: 100% complete...done. 2024-04-06T10:08:10.037 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ remove_image_retry cluster2 mirror image_17 2024-04-06T10:08:10.037 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-04-06T10:08:10.037 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:08:10.037 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_17 2024-04-06T10:08:10.037 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 16 32 2024-04-06T10:08:10.038 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T10:08:10.038 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ remove_image cluster2 mirror image_17 2024-04-06T10:08:10.038 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-04-06T10:08:10.039 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:08:10.039 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_17 2024-04-06T10:08:10.039 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_17 2024-04-06T10:08:10.091 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster=cluster2 rm mirror/image_17 2024-04-06T10:08:10.298 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-04-06T10:08:10.303 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T10:08:10.303 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-06T10:08:10.303 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ image=image_18 2024-04-06T10:08:10.303 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ remove_snapshot cluster2 mirror image_18 snap 2024-04-06T10:08:10.303 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-04-06T10:08:10.303 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:08:10.303 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_18 2024-04-06T10:08:10.303 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local snap=snap 2024-04-06T10:08:10.304 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster2 snap rm mirror/image_18@snap 2024-04-06T10:08:10.939 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: Removing snap: 100% complete...done. 2024-04-06T10:08:10.956 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ remove_image_retry cluster2 mirror image_18 2024-04-06T10:08:10.956 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-04-06T10:08:10.956 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:08:10.956 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_18 2024-04-06T10:08:10.956 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 16 32 2024-04-06T10:08:10.956 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T10:08:10.957 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ remove_image cluster2 mirror image_18 2024-04-06T10:08:10.957 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-04-06T10:08:10.957 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:08:10.957 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_18 2024-04-06T10:08:10.957 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_18 2024-04-06T10:08:11.011 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster=cluster2 rm mirror/image_18 2024-04-06T10:08:11.225 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-04-06T10:08:11.229 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T10:08:11.230 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-06T10:08:11.230 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ image=image_19 2024-04-06T10:08:11.230 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ remove_snapshot cluster2 mirror image_19 snap 2024-04-06T10:08:11.230 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-04-06T10:08:11.230 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:08:11.230 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_19 2024-04-06T10:08:11.230 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local snap=snap 2024-04-06T10:08:11.230 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster2 snap rm mirror/image_19@snap 2024-04-06T10:08:12.034 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: Removing snap: 100% complete...done. 2024-04-06T10:08:12.054 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ remove_image_retry cluster2 mirror image_19 2024-04-06T10:08:12.054 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-04-06T10:08:12.055 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:08:12.055 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_19 2024-04-06T10:08:12.055 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 16 32 2024-04-06T10:08:12.055 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T10:08:12.056 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ remove_image cluster2 mirror image_19 2024-04-06T10:08:12.056 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-04-06T10:08:12.056 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:08:12.056 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_19 2024-04-06T10:08:12.056 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_19 2024-04-06T10:08:12.112 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster=cluster2 rm mirror/image_19 2024-04-06T10:08:12.331 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-04-06T10:08:12.335 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T10:08:12.335 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-06T10:08:12.335 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ image=image_20 2024-04-06T10:08:12.335 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ remove_snapshot cluster2 mirror image_20 snap 2024-04-06T10:08:12.335 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-04-06T10:08:12.336 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:08:12.336 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_20 2024-04-06T10:08:12.336 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local snap=snap 2024-04-06T10:08:12.336 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster2 snap rm mirror/image_20@snap 2024-04-06T10:08:13.031 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: Removing snap: 100% complete...done. 2024-04-06T10:08:13.051 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ remove_image_retry cluster2 mirror image_20 2024-04-06T10:08:13.051 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-04-06T10:08:13.051 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:08:13.051 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_20 2024-04-06T10:08:13.051 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 16 32 2024-04-06T10:08:13.051 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T10:08:13.052 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ remove_image cluster2 mirror image_20 2024-04-06T10:08:13.052 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-04-06T10:08:13.052 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:08:13.052 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_20 2024-04-06T10:08:13.052 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_20 2024-04-06T10:08:13.109 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster=cluster2 rm mirror/image_20 2024-04-06T10:08:13.334 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-04-06T10:08:13.338 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T10:08:13.339 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-06T10:08:13.339 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ image=image_21 2024-04-06T10:08:13.339 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ remove_snapshot cluster2 mirror image_21 snap 2024-04-06T10:08:13.339 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-04-06T10:08:13.339 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:08:13.339 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_21 2024-04-06T10:08:13.339 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local snap=snap 2024-04-06T10:08:13.339 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster2 snap rm mirror/image_21@snap 2024-04-06T10:08:14.044 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: Removing snap: 100% complete...done. 2024-04-06T10:08:14.066 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ remove_image_retry cluster2 mirror image_21 2024-04-06T10:08:14.066 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-04-06T10:08:14.066 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:08:14.066 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_21 2024-04-06T10:08:14.066 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 16 32 2024-04-06T10:08:14.066 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T10:08:14.067 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ remove_image cluster2 mirror image_21 2024-04-06T10:08:14.067 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-04-06T10:08:14.067 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:08:14.067 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_21 2024-04-06T10:08:14.068 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_21 2024-04-06T10:08:14.120 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster=cluster2 rm mirror/image_21 2024-04-06T10:08:14.324 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-04-06T10:08:14.328 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T10:08:14.328 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-06T10:08:14.328 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ image=image_22 2024-04-06T10:08:14.328 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ remove_snapshot cluster2 mirror image_22 snap 2024-04-06T10:08:14.328 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-04-06T10:08:14.328 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:08:14.328 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_22 2024-04-06T10:08:14.328 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local snap=snap 2024-04-06T10:08:14.328 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster2 snap rm mirror/image_22@snap 2024-04-06T10:08:15.054 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: Removing snap: 100% complete...done. 2024-04-06T10:08:15.077 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ remove_image_retry cluster2 mirror image_22 2024-04-06T10:08:15.077 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-04-06T10:08:15.077 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:08:15.077 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_22 2024-04-06T10:08:15.077 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 16 32 2024-04-06T10:08:15.078 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T10:08:15.078 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ remove_image cluster2 mirror image_22 2024-04-06T10:08:15.078 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-04-06T10:08:15.078 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:08:15.079 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_22 2024-04-06T10:08:15.079 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_22 2024-04-06T10:08:15.135 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster=cluster2 rm mirror/image_22 2024-04-06T10:08:15.345 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-04-06T10:08:15.349 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T10:08:15.349 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-06T10:08:15.350 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ image=image_23 2024-04-06T10:08:15.350 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ remove_snapshot cluster2 mirror image_23 snap 2024-04-06T10:08:15.350 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-04-06T10:08:15.350 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:08:15.350 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_23 2024-04-06T10:08:15.350 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local snap=snap 2024-04-06T10:08:15.350 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster2 snap rm mirror/image_23@snap 2024-04-06T10:08:15.681 INFO:tasks.rbd_mirror_thrash:kill cluster1.client.mirror.0 2024-04-06T10:08:15.681 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Sent signal 15 2024-04-06T10:08:15.681 INFO:tasks.rbd_mirror_thrash:waiting for 8 secs before reviving daemons 2024-04-06T10:08:15.682 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi192.stderr:2024-04-06T10:08:15.685+0000 7fcae9c00640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.0 (PID: 149070) UID: 1000 2024-04-06T10:08:15.951 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: Removing snap: 100% complete...done. 2024-04-06T10:08:15.974 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ remove_image_retry cluster2 mirror image_23 2024-04-06T10:08:15.974 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-04-06T10:08:15.974 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:08:15.974 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_23 2024-04-06T10:08:15.974 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 16 32 2024-04-06T10:08:15.974 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T10:08:15.975 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ remove_image cluster2 mirror image_23 2024-04-06T10:08:15.975 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-04-06T10:08:15.975 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:08:15.975 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_23 2024-04-06T10:08:15.975 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_23 2024-04-06T10:08:16.031 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster=cluster2 rm mirror/image_23 2024-04-06T10:08:16.236 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-04-06T10:08:16.240 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T10:08:16.241 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-06T10:08:16.241 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ image=image_24 2024-04-06T10:08:16.241 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ remove_snapshot cluster2 mirror image_24 snap 2024-04-06T10:08:16.241 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-04-06T10:08:16.241 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:08:16.241 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_24 2024-04-06T10:08:16.241 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local snap=snap 2024-04-06T10:08:16.241 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster2 snap rm mirror/image_24@snap 2024-04-06T10:08:17.059 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: Removing snap: 100% complete...done. 2024-04-06T10:08:17.081 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ remove_image_retry cluster2 mirror image_24 2024-04-06T10:08:17.081 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-04-06T10:08:17.081 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:08:17.081 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_24 2024-04-06T10:08:17.081 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 16 32 2024-04-06T10:08:17.082 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T10:08:17.083 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ remove_image cluster2 mirror image_24 2024-04-06T10:08:17.083 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-04-06T10:08:17.083 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:08:17.083 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_24 2024-04-06T10:08:17.083 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_24 2024-04-06T10:08:17.137 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster=cluster2 rm mirror/image_24 2024-04-06T10:08:17.369 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-04-06T10:08:17.374 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T10:08:17.374 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-06T10:08:17.374 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ image=image_25 2024-04-06T10:08:17.374 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ remove_snapshot cluster2 mirror image_25 snap 2024-04-06T10:08:17.374 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-04-06T10:08:17.374 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:08:17.374 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_25 2024-04-06T10:08:17.374 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local snap=snap 2024-04-06T10:08:17.374 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster2 snap rm mirror/image_25@snap 2024-04-06T10:08:18.054 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: Removing snap: 100% complete...done. 2024-04-06T10:08:18.073 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ remove_image_retry cluster2 mirror image_25 2024-04-06T10:08:18.073 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-04-06T10:08:18.074 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:08:18.074 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_25 2024-04-06T10:08:18.074 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 16 32 2024-04-06T10:08:18.074 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T10:08:18.074 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ remove_image cluster2 mirror image_25 2024-04-06T10:08:18.075 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-04-06T10:08:18.075 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:08:18.075 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_25 2024-04-06T10:08:18.075 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_25 2024-04-06T10:08:18.127 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster=cluster2 rm mirror/image_25 2024-04-06T10:08:18.394 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-04-06T10:08:18.398 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T10:08:18.398 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-06T10:08:18.398 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ image=image_26 2024-04-06T10:08:18.398 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ remove_snapshot cluster2 mirror image_26 snap 2024-04-06T10:08:18.399 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-04-06T10:08:18.399 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:08:18.399 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_26 2024-04-06T10:08:18.399 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local snap=snap 2024-04-06T10:08:18.399 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster2 snap rm mirror/image_26@snap 2024-04-06T10:08:19.060 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: Removing snap: 100% complete...done. 2024-04-06T10:08:19.077 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ remove_image_retry cluster2 mirror image_26 2024-04-06T10:08:19.077 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-04-06T10:08:19.077 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:08:19.077 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_26 2024-04-06T10:08:19.077 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 16 32 2024-04-06T10:08:19.077 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T10:08:19.078 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ remove_image cluster2 mirror image_26 2024-04-06T10:08:19.078 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-04-06T10:08:19.078 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:08:19.078 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_26 2024-04-06T10:08:19.078 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_26 2024-04-06T10:08:19.128 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster=cluster2 rm mirror/image_26 2024-04-06T10:08:19.386 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-04-06T10:08:19.391 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T10:08:19.391 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-06T10:08:19.391 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ image=image_27 2024-04-06T10:08:19.391 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ remove_snapshot cluster2 mirror image_27 snap 2024-04-06T10:08:19.391 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-04-06T10:08:19.391 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:08:19.391 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_27 2024-04-06T10:08:19.391 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local snap=snap 2024-04-06T10:08:19.391 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster2 snap rm mirror/image_27@snap 2024-04-06T10:08:20.071 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: Removing snap: 100% complete...done. 2024-04-06T10:08:20.090 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ remove_image_retry cluster2 mirror image_27 2024-04-06T10:08:20.090 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-04-06T10:08:20.090 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:08:20.090 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_27 2024-04-06T10:08:20.090 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 16 32 2024-04-06T10:08:20.090 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T10:08:20.091 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ remove_image cluster2 mirror image_27 2024-04-06T10:08:20.091 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-04-06T10:08:20.091 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:08:20.091 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_27 2024-04-06T10:08:20.091 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_27 2024-04-06T10:08:20.144 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster=cluster2 rm mirror/image_27 2024-04-06T10:08:20.406 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-04-06T10:08:20.411 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T10:08:20.411 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-06T10:08:20.411 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ image=image_28 2024-04-06T10:08:20.411 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ remove_snapshot cluster2 mirror image_28 snap 2024-04-06T10:08:20.411 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-04-06T10:08:20.411 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:08:20.411 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_28 2024-04-06T10:08:20.411 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local snap=snap 2024-04-06T10:08:20.411 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster2 snap rm mirror/image_28@snap 2024-04-06T10:08:20.928 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: Removing snap: 100% complete...done. 2024-04-06T10:08:20.945 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ remove_image_retry cluster2 mirror image_28 2024-04-06T10:08:20.945 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-04-06T10:08:20.945 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:08:20.945 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_28 2024-04-06T10:08:20.945 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 16 32 2024-04-06T10:08:20.945 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T10:08:20.946 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ remove_image cluster2 mirror image_28 2024-04-06T10:08:20.946 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-04-06T10:08:20.947 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:08:20.947 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_28 2024-04-06T10:08:20.947 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_28 2024-04-06T10:08:20.999 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster=cluster2 rm mirror/image_28 2024-04-06T10:08:21.232 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-04-06T10:08:21.237 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T10:08:21.237 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-06T10:08:21.237 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ image=image_29 2024-04-06T10:08:21.237 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ remove_snapshot cluster2 mirror image_29 snap 2024-04-06T10:08:21.237 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-04-06T10:08:21.237 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:08:21.238 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_29 2024-04-06T10:08:21.238 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local snap=snap 2024-04-06T10:08:21.238 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster2 snap rm mirror/image_29@snap 2024-04-06T10:08:22.081 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: Removing snap: 100% complete...done. 2024-04-06T10:08:22.100 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ remove_image_retry cluster2 mirror image_29 2024-04-06T10:08:22.100 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-04-06T10:08:22.100 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:08:22.101 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_29 2024-04-06T10:08:22.101 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 16 32 2024-04-06T10:08:22.101 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T10:08:22.101 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ remove_image cluster2 mirror image_29 2024-04-06T10:08:22.101 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-04-06T10:08:22.101 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:08:22.101 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_29 2024-04-06T10:08:22.101 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_29 2024-04-06T10:08:22.154 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster=cluster2 rm mirror/image_29 2024-04-06T10:08:22.413 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-04-06T10:08:22.417 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T10:08:22.417 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-06T10:08:22.417 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ image=image_30 2024-04-06T10:08:22.417 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ remove_snapshot cluster2 mirror image_30 snap 2024-04-06T10:08:22.417 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-04-06T10:08:22.417 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:08:22.417 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_30 2024-04-06T10:08:22.417 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local snap=snap 2024-04-06T10:08:22.417 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster2 snap rm mirror/image_30@snap 2024-04-06T10:08:23.097 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: Removing snap: 100% complete...done. 2024-04-06T10:08:23.118 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ remove_image_retry cluster2 mirror image_30 2024-04-06T10:08:23.119 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-04-06T10:08:23.119 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:08:23.119 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_30 2024-04-06T10:08:23.119 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 16 32 2024-04-06T10:08:23.119 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T10:08:23.120 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ remove_image cluster2 mirror image_30 2024-04-06T10:08:23.120 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-04-06T10:08:23.120 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:08:23.120 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_30 2024-04-06T10:08:23.120 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_30 2024-04-06T10:08:23.174 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster=cluster2 rm mirror/image_30 2024-04-06T10:08:23.393 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-04-06T10:08:23.397 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T10:08:23.397 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-06T10:08:23.398 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ image=image_31 2024-04-06T10:08:23.398 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ remove_snapshot cluster2 mirror image_31 snap 2024-04-06T10:08:23.398 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-04-06T10:08:23.398 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:08:23.398 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_31 2024-04-06T10:08:23.398 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local snap=snap 2024-04-06T10:08:23.398 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster2 snap rm mirror/image_31@snap 2024-04-06T10:08:23.682 INFO:tasks.rbd_mirror_thrash:waiting for cluster1.client.mirror.0 2024-04-06T10:08:23.682 INFO:teuthology.orchestra.run:waiting for 600 2024-04-06T10:08:23.683 INFO:tasks.rbd_mirror_thrash:reviving cluster1.client.mirror.0 2024-04-06T10:08:23.683 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Restarting daemon 2024-04-06T10:08:23.683 DEBUG:teuthology.orchestra.run.smithi192:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.0 2024-04-06T10:08:23.686 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Started 2024-04-06T10:08:23.686 INFO:tasks.rbd_mirror_thrash:waiting for 95 secs before thrashing 2024-04-06T10:08:24.106 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: Removing snap: 100% complete...done. 2024-04-06T10:08:24.129 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ remove_image_retry cluster2 mirror image_31 2024-04-06T10:08:24.129 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-04-06T10:08:24.129 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:08:24.129 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_31 2024-04-06T10:08:24.129 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 16 32 2024-04-06T10:08:24.129 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T10:08:24.130 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ remove_image cluster2 mirror image_31 2024-04-06T10:08:24.130 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-04-06T10:08:24.130 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:08:24.130 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_31 2024-04-06T10:08:24.131 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_31 2024-04-06T10:08:24.185 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster=cluster2 rm mirror/image_31 2024-04-06T10:08:24.434 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-04-06T10:08:24.439 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T10:08:24.439 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-06T10:08:24.439 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ image=image_32 2024-04-06T10:08:24.439 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ remove_snapshot cluster2 mirror image_32 snap 2024-04-06T10:08:24.439 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-04-06T10:08:24.439 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:08:24.439 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_32 2024-04-06T10:08:24.439 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local snap=snap 2024-04-06T10:08:24.439 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster2 snap rm mirror/image_32@snap 2024-04-06T10:08:25.102 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: Removing snap: 100% complete...done. 2024-04-06T10:08:25.125 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ remove_image_retry cluster2 mirror image_32 2024-04-06T10:08:25.125 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-04-06T10:08:25.125 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:08:25.125 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_32 2024-04-06T10:08:25.125 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 16 32 2024-04-06T10:08:25.125 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T10:08:25.126 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ remove_image cluster2 mirror image_32 2024-04-06T10:08:25.126 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-04-06T10:08:25.126 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:08:25.126 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_32 2024-04-06T10:08:25.126 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_32 2024-04-06T10:08:25.181 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster=cluster2 rm mirror/image_32 2024-04-06T10:08:25.416 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-04-06T10:08:25.420 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T10:08:25.420 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-06T10:08:25.420 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ image=image_33 2024-04-06T10:08:25.420 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ remove_snapshot cluster2 mirror image_33 snap 2024-04-06T10:08:25.421 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-04-06T10:08:25.421 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:08:25.421 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_33 2024-04-06T10:08:25.421 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local snap=snap 2024-04-06T10:08:25.421 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster2 snap rm mirror/image_33@snap 2024-04-06T10:08:25.677 INFO:tasks.rbd_mirror_thrash:kill cluster2.client.mirror.0 2024-04-06T10:08:25.678 INFO:tasks.rbd_mirror.cluster2.client.mirror.0:Sent signal 15 2024-04-06T10:08:25.678 INFO:tasks.rbd_mirror_thrash:waiting for 9 secs before reviving daemons 2024-04-06T10:08:25.679 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi192.stderr:2024-04-06T10:08:25.681+0000 7ff021034640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.0 (PID: 156680) UID: 1000 2024-04-06T10:08:25.953 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: Removing snap: 100% complete...done. 2024-04-06T10:08:25.976 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ remove_image_retry cluster2 mirror image_33 2024-04-06T10:08:25.976 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-04-06T10:08:25.976 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:08:25.976 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_33 2024-04-06T10:08:25.976 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 16 32 2024-04-06T10:08:25.976 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T10:08:25.977 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ remove_image cluster2 mirror image_33 2024-04-06T10:08:25.978 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-04-06T10:08:25.978 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:08:25.978 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_33 2024-04-06T10:08:25.978 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_33 2024-04-06T10:08:26.032 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster=cluster2 rm mirror/image_33 2024-04-06T10:08:26.261 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-04-06T10:08:26.266 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T10:08:26.266 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-06T10:08:26.266 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ image=image_34 2024-04-06T10:08:26.266 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ remove_snapshot cluster2 mirror image_34 snap 2024-04-06T10:08:26.266 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-04-06T10:08:26.266 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:08:26.266 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_34 2024-04-06T10:08:26.266 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local snap=snap 2024-04-06T10:08:26.266 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster2 snap rm mirror/image_34@snap 2024-04-06T10:08:27.116 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: Removing snap: 100% complete...done. 2024-04-06T10:08:27.134 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ remove_image_retry cluster2 mirror image_34 2024-04-06T10:08:27.135 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-04-06T10:08:27.135 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:08:27.135 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_34 2024-04-06T10:08:27.135 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 16 32 2024-04-06T10:08:27.135 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T10:08:27.136 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ remove_image cluster2 mirror image_34 2024-04-06T10:08:27.136 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-04-06T10:08:27.136 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:08:27.136 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_34 2024-04-06T10:08:27.136 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_34 2024-04-06T10:08:27.190 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster=cluster2 rm mirror/image_34 2024-04-06T10:08:27.449 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-04-06T10:08:27.453 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T10:08:27.454 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-06T10:08:27.454 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ image=image_35 2024-04-06T10:08:27.454 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ remove_snapshot cluster2 mirror image_35 snap 2024-04-06T10:08:27.454 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-04-06T10:08:27.454 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:08:27.454 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_35 2024-04-06T10:08:27.454 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local snap=snap 2024-04-06T10:08:27.454 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster2 snap rm mirror/image_35@snap 2024-04-06T10:08:28.098 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: Removing snap: 100% complete...done. 2024-04-06T10:08:28.115 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ remove_image_retry cluster2 mirror image_35 2024-04-06T10:08:28.115 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-04-06T10:08:28.115 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:08:28.115 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_35 2024-04-06T10:08:28.115 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 16 32 2024-04-06T10:08:28.115 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T10:08:28.116 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ remove_image cluster2 mirror image_35 2024-04-06T10:08:28.116 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-04-06T10:08:28.116 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:08:28.116 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_35 2024-04-06T10:08:28.116 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_35 2024-04-06T10:08:28.179 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster=cluster2 rm mirror/image_35 2024-04-06T10:08:28.435 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-04-06T10:08:28.440 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T10:08:28.440 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-06T10:08:28.440 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ image=image_36 2024-04-06T10:08:28.440 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ remove_snapshot cluster2 mirror image_36 snap 2024-04-06T10:08:28.440 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-04-06T10:08:28.440 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:08:28.440 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_36 2024-04-06T10:08:28.440 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local snap=snap 2024-04-06T10:08:28.441 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster2 snap rm mirror/image_36@snap 2024-04-06T10:08:29.130 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: Removing snap: 100% complete...done. 2024-04-06T10:08:29.148 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ remove_image_retry cluster2 mirror image_36 2024-04-06T10:08:29.148 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-04-06T10:08:29.148 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:08:29.148 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_36 2024-04-06T10:08:29.148 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 16 32 2024-04-06T10:08:29.148 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T10:08:29.149 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ remove_image cluster2 mirror image_36 2024-04-06T10:08:29.149 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-04-06T10:08:29.149 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:08:29.149 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_36 2024-04-06T10:08:29.150 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_36 2024-04-06T10:08:29.203 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster=cluster2 rm mirror/image_36 2024-04-06T10:08:29.449 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-04-06T10:08:29.454 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T10:08:29.454 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-06T10:08:29.454 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ image=image_37 2024-04-06T10:08:29.454 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ remove_snapshot cluster2 mirror image_37 snap 2024-04-06T10:08:29.454 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-04-06T10:08:29.454 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:08:29.454 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_37 2024-04-06T10:08:29.454 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local snap=snap 2024-04-06T10:08:29.455 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster2 snap rm mirror/image_37@snap 2024-04-06T10:08:30.125 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: Removing snap: 100% complete...done. 2024-04-06T10:08:30.146 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ remove_image_retry cluster2 mirror image_37 2024-04-06T10:08:30.146 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-04-06T10:08:30.146 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:08:30.146 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_37 2024-04-06T10:08:30.146 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 16 32 2024-04-06T10:08:30.146 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T10:08:30.147 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ remove_image cluster2 mirror image_37 2024-04-06T10:08:30.147 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-04-06T10:08:30.147 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:08:30.147 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_37 2024-04-06T10:08:30.147 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_37 2024-04-06T10:08:30.204 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster=cluster2 rm mirror/image_37 2024-04-06T10:08:30.463 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-04-06T10:08:30.468 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T10:08:30.468 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-06T10:08:30.468 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ image=image_38 2024-04-06T10:08:30.468 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ remove_snapshot cluster2 mirror image_38 snap 2024-04-06T10:08:30.468 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-04-06T10:08:30.468 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:08:30.468 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_38 2024-04-06T10:08:30.468 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local snap=snap 2024-04-06T10:08:30.468 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster2 snap rm mirror/image_38@snap 2024-04-06T10:08:30.934 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: Removing snap: 100% complete...done. 2024-04-06T10:08:30.955 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ remove_image_retry cluster2 mirror image_38 2024-04-06T10:08:30.955 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-04-06T10:08:30.956 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:08:30.956 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_38 2024-04-06T10:08:30.956 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 16 32 2024-04-06T10:08:30.956 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T10:08:30.957 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ remove_image cluster2 mirror image_38 2024-04-06T10:08:30.957 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-04-06T10:08:30.957 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:08:30.957 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_38 2024-04-06T10:08:30.957 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_38 2024-04-06T10:08:31.022 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster=cluster2 rm mirror/image_38 2024-04-06T10:08:31.239 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-04-06T10:08:31.243 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T10:08:31.243 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-06T10:08:31.244 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ image=image_39 2024-04-06T10:08:31.244 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ remove_snapshot cluster2 mirror image_39 snap 2024-04-06T10:08:31.244 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-04-06T10:08:31.244 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:08:31.244 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_39 2024-04-06T10:08:31.244 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local snap=snap 2024-04-06T10:08:31.244 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster2 snap rm mirror/image_39@snap 2024-04-06T10:08:32.146 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: Removing snap: 100% complete...done. 2024-04-06T10:08:32.167 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ remove_image_retry cluster2 mirror image_39 2024-04-06T10:08:32.167 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-04-06T10:08:32.167 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:08:32.167 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_39 2024-04-06T10:08:32.167 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 16 32 2024-04-06T10:08:32.167 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T10:08:32.168 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ remove_image cluster2 mirror image_39 2024-04-06T10:08:32.168 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-04-06T10:08:32.168 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:08:32.168 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_39 2024-04-06T10:08:32.168 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_39 2024-04-06T10:08:32.223 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster=cluster2 rm mirror/image_39 2024-04-06T10:08:32.488 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-04-06T10:08:32.492 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T10:08:32.492 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-06T10:08:32.493 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ image=image_40 2024-04-06T10:08:32.493 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ remove_snapshot cluster2 mirror image_40 snap 2024-04-06T10:08:32.493 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-04-06T10:08:32.493 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:08:32.493 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_40 2024-04-06T10:08:32.493 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local snap=snap 2024-04-06T10:08:32.493 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster2 snap rm mirror/image_40@snap 2024-04-06T10:08:33.144 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: Removing snap: 100% complete...done. 2024-04-06T10:08:33.162 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ remove_image_retry cluster2 mirror image_40 2024-04-06T10:08:33.162 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-04-06T10:08:33.162 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:08:33.162 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_40 2024-04-06T10:08:33.162 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 16 32 2024-04-06T10:08:33.163 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T10:08:33.164 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ remove_image cluster2 mirror image_40 2024-04-06T10:08:33.164 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-04-06T10:08:33.164 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:08:33.164 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_40 2024-04-06T10:08:33.164 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_40 2024-04-06T10:08:33.218 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster=cluster2 rm mirror/image_40 2024-04-06T10:08:33.473 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-04-06T10:08:33.478 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T10:08:33.478 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-06T10:08:33.478 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ image=image_41 2024-04-06T10:08:33.478 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ remove_snapshot cluster2 mirror image_41 snap 2024-04-06T10:08:33.478 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-04-06T10:08:33.478 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:08:33.478 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_41 2024-04-06T10:08:33.478 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local snap=snap 2024-04-06T10:08:33.479 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster2 snap rm mirror/image_41@snap 2024-04-06T10:08:34.146 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: Removing snap: 100% complete...done. 2024-04-06T10:08:34.163 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ remove_image_retry cluster2 mirror image_41 2024-04-06T10:08:34.164 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-04-06T10:08:34.164 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:08:34.164 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_41 2024-04-06T10:08:34.164 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 16 32 2024-04-06T10:08:34.164 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T10:08:34.165 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ remove_image cluster2 mirror image_41 2024-04-06T10:08:34.165 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-04-06T10:08:34.165 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:08:34.165 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_41 2024-04-06T10:08:34.165 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_41 2024-04-06T10:08:34.217 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster=cluster2 rm mirror/image_41 2024-04-06T10:08:34.461 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-04-06T10:08:34.466 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T10:08:34.466 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-06T10:08:34.466 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ image=image_42 2024-04-06T10:08:34.466 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ remove_snapshot cluster2 mirror image_42 snap 2024-04-06T10:08:34.467 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-04-06T10:08:34.467 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:08:34.467 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_42 2024-04-06T10:08:34.467 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local snap=snap 2024-04-06T10:08:34.467 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster2 snap rm mirror/image_42@snap 2024-04-06T10:08:34.678 INFO:tasks.rbd_mirror_thrash:waiting for cluster2.client.mirror.0 2024-04-06T10:08:34.679 INFO:teuthology.orchestra.run:waiting for 600 2024-04-06T10:08:34.679 INFO:tasks.rbd_mirror_thrash:reviving cluster2.client.mirror.0 2024-04-06T10:08:34.679 INFO:tasks.rbd_mirror.cluster2.client.mirror.0:Restarting daemon 2024-04-06T10:08:34.679 DEBUG:teuthology.orchestra.run.smithi192:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.0 2024-04-06T10:08:34.682 INFO:tasks.rbd_mirror.cluster2.client.mirror.0:Started 2024-04-06T10:08:34.682 INFO:tasks.rbd_mirror_thrash:waiting for 83 secs before thrashing 2024-04-06T10:08:35.146 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: Removing snap: 100% complete...done. 2024-04-06T10:08:35.163 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ remove_image_retry cluster2 mirror image_42 2024-04-06T10:08:35.163 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-04-06T10:08:35.163 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:08:35.163 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_42 2024-04-06T10:08:35.163 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 16 32 2024-04-06T10:08:35.163 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T10:08:35.164 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ remove_image cluster2 mirror image_42 2024-04-06T10:08:35.164 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-04-06T10:08:35.164 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:08:35.164 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_42 2024-04-06T10:08:35.164 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_42 2024-04-06T10:08:35.213 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster=cluster2 rm mirror/image_42 2024-04-06T10:08:35.282 INFO:tasks.ceph.cluster2.osd.2.smithi192.stderr:2024-04-06T10:08:35.285+0000 7fa534aba640 -1 reset not still connected to 0x559372a44ea0 2024-04-06T10:08:35.459 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-04-06T10:08:35.464 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T10:08:35.464 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-06T10:08:35.464 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ image=image_43 2024-04-06T10:08:35.464 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ remove_snapshot cluster2 mirror image_43 snap 2024-04-06T10:08:35.464 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-04-06T10:08:35.464 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:08:35.464 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_43 2024-04-06T10:08:35.465 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local snap=snap 2024-04-06T10:08:35.465 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster2 snap rm mirror/image_43@snap 2024-04-06T10:08:35.957 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: Removing snap: 100% complete...done. 2024-04-06T10:08:35.981 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ remove_image_retry cluster2 mirror image_43 2024-04-06T10:08:35.981 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-04-06T10:08:35.981 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:08:35.981 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_43 2024-04-06T10:08:35.981 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 16 32 2024-04-06T10:08:35.981 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T10:08:35.982 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ remove_image cluster2 mirror image_43 2024-04-06T10:08:35.982 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-04-06T10:08:35.983 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:08:35.983 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_43 2024-04-06T10:08:35.983 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_43 2024-04-06T10:08:36.036 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster=cluster2 rm mirror/image_43 2024-04-06T10:08:36.248 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-04-06T10:08:36.308 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T10:08:36.308 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-06T10:08:36.308 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ image=image_44 2024-04-06T10:08:36.308 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ remove_snapshot cluster2 mirror image_44 snap 2024-04-06T10:08:36.309 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-04-06T10:08:36.309 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:08:36.309 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_44 2024-04-06T10:08:36.309 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local snap=snap 2024-04-06T10:08:36.309 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster2 snap rm mirror/image_44@snap 2024-04-06T10:08:37.162 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: Removing snap: 100% complete...done. 2024-04-06T10:08:37.183 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ remove_image_retry cluster2 mirror image_44 2024-04-06T10:08:37.183 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-04-06T10:08:37.183 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:08:37.183 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_44 2024-04-06T10:08:37.183 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 16 32 2024-04-06T10:08:37.183 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T10:08:37.185 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ remove_image cluster2 mirror image_44 2024-04-06T10:08:37.185 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-04-06T10:08:37.185 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:08:37.185 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_44 2024-04-06T10:08:37.185 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_44 2024-04-06T10:08:37.238 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster=cluster2 rm mirror/image_44 2024-04-06T10:08:37.491 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-04-06T10:08:37.495 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T10:08:37.495 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-06T10:08:37.496 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ image=image_45 2024-04-06T10:08:37.496 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ remove_snapshot cluster2 mirror image_45 snap 2024-04-06T10:08:37.496 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-04-06T10:08:37.496 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:08:37.496 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_45 2024-04-06T10:08:37.496 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local snap=snap 2024-04-06T10:08:37.496 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster2 snap rm mirror/image_45@snap 2024-04-06T10:08:38.161 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: Removing snap: 100% complete...done. 2024-04-06T10:08:38.179 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ remove_image_retry cluster2 mirror image_45 2024-04-06T10:08:38.179 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-04-06T10:08:38.180 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:08:38.180 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_45 2024-04-06T10:08:38.180 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 16 32 2024-04-06T10:08:38.180 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T10:08:38.181 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ remove_image cluster2 mirror image_45 2024-04-06T10:08:38.181 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-04-06T10:08:38.181 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:08:38.181 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_45 2024-04-06T10:08:38.181 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_45 2024-04-06T10:08:38.238 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster=cluster2 rm mirror/image_45 2024-04-06T10:08:38.453 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-04-06T10:08:38.458 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T10:08:38.458 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-06T10:08:38.458 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ image=image_46 2024-04-06T10:08:38.458 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ remove_snapshot cluster2 mirror image_46 snap 2024-04-06T10:08:38.458 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-04-06T10:08:38.458 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:08:38.458 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_46 2024-04-06T10:08:38.458 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local snap=snap 2024-04-06T10:08:38.458 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster2 snap rm mirror/image_46@snap 2024-04-06T10:08:39.176 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: Removing snap: 100% complete...done. 2024-04-06T10:08:39.196 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ remove_image_retry cluster2 mirror image_46 2024-04-06T10:08:39.196 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-04-06T10:08:39.196 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:08:39.196 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_46 2024-04-06T10:08:39.196 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 16 32 2024-04-06T10:08:39.197 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T10:08:39.197 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ remove_image cluster2 mirror image_46 2024-04-06T10:08:39.197 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-04-06T10:08:39.197 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:08:39.197 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_46 2024-04-06T10:08:39.197 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_46 2024-04-06T10:08:39.250 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster=cluster2 rm mirror/image_46 2024-04-06T10:08:39.518 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-04-06T10:08:39.522 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T10:08:39.522 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-06T10:08:39.522 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ image=image_47 2024-04-06T10:08:39.522 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ remove_snapshot cluster2 mirror image_47 snap 2024-04-06T10:08:39.522 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-04-06T10:08:39.522 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:08:39.522 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_47 2024-04-06T10:08:39.523 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local snap=snap 2024-04-06T10:08:39.523 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster2 snap rm mirror/image_47@snap 2024-04-06T10:08:40.201 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: Removing snap: 100% complete...done. 2024-04-06T10:08:40.225 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ remove_image_retry cluster2 mirror image_47 2024-04-06T10:08:40.225 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-04-06T10:08:40.225 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:08:40.225 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_47 2024-04-06T10:08:40.225 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 16 32 2024-04-06T10:08:40.225 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T10:08:40.227 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ remove_image cluster2 mirror image_47 2024-04-06T10:08:40.227 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-04-06T10:08:40.227 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:08:40.227 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_47 2024-04-06T10:08:40.227 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_47 2024-04-06T10:08:40.288 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster=cluster2 rm mirror/image_47 2024-04-06T10:08:40.496 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-04-06T10:08:40.500 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T10:08:40.500 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-06T10:08:40.500 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ image=image_48 2024-04-06T10:08:40.500 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ remove_snapshot cluster2 mirror image_48 snap 2024-04-06T10:08:40.501 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-04-06T10:08:40.501 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:08:40.501 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_48 2024-04-06T10:08:40.501 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local snap=snap 2024-04-06T10:08:40.501 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster2 snap rm mirror/image_48@snap 2024-04-06T10:08:40.955 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: Removing snap: 100% complete...done. 2024-04-06T10:08:40.978 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ remove_image_retry cluster2 mirror image_48 2024-04-06T10:08:40.978 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-04-06T10:08:40.979 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:08:40.979 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_48 2024-04-06T10:08:40.979 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 16 32 2024-04-06T10:08:40.979 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T10:08:40.979 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ remove_image cluster2 mirror image_48 2024-04-06T10:08:40.980 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-04-06T10:08:40.980 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:08:40.980 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_48 2024-04-06T10:08:40.980 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_48 2024-04-06T10:08:41.036 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster=cluster2 rm mirror/image_48 2024-04-06T10:08:41.250 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-04-06T10:08:41.255 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T10:08:41.255 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-06T10:08:41.255 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ image=image_49 2024-04-06T10:08:41.255 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ remove_snapshot cluster2 mirror image_49 snap 2024-04-06T10:08:41.255 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-04-06T10:08:41.255 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:08:41.255 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_49 2024-04-06T10:08:41.255 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local snap=snap 2024-04-06T10:08:41.256 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster2 snap rm mirror/image_49@snap 2024-04-06T10:08:42.194 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: Removing snap: 100% complete...done. 2024-04-06T10:08:42.215 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ remove_image_retry cluster2 mirror image_49 2024-04-06T10:08:42.215 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-04-06T10:08:42.215 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:08:42.215 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_49 2024-04-06T10:08:42.215 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 16 32 2024-04-06T10:08:42.215 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T10:08:42.216 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ remove_image cluster2 mirror image_49 2024-04-06T10:08:42.216 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-04-06T10:08:42.216 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:08:42.217 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_49 2024-04-06T10:08:42.217 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_49 2024-04-06T10:08:42.270 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster=cluster2 rm mirror/image_49 2024-04-06T10:08:42.565 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-04-06T10:08:42.570 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T10:08:42.570 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-06T10:08:42.570 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ image=image_50 2024-04-06T10:08:42.570 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ remove_snapshot cluster2 mirror image_50 snap 2024-04-06T10:08:42.570 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-04-06T10:08:42.570 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:08:42.570 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_50 2024-04-06T10:08:42.570 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local snap=snap 2024-04-06T10:08:42.570 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster2 snap rm mirror/image_50@snap 2024-04-06T10:08:43.203 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: Removing snap: 100% complete...done. 2024-04-06T10:08:43.225 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ remove_image_retry cluster2 mirror image_50 2024-04-06T10:08:43.225 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-04-06T10:08:43.225 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:08:43.225 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_50 2024-04-06T10:08:43.225 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 16 32 2024-04-06T10:08:43.225 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T10:08:43.226 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ remove_image cluster2 mirror image_50 2024-04-06T10:08:43.226 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-04-06T10:08:43.226 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:08:43.226 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_50 2024-04-06T10:08:43.226 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_50 2024-04-06T10:08:43.283 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster=cluster2 rm mirror/image_50 2024-04-06T10:08:43.503 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-04-06T10:08:43.508 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T10:08:43.508 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ testlog 'TEST: image deletions should propagate' 2024-04-06T10:08:43.508 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ tee -a /tmp/tmp.BhW6xKmROU/rbd-mirror.test.log 2024-04-06T10:08:43.508 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ date '+%F %T' 2024-04-06T10:08:43.510 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ echo 2024-04-06 10:08:43 TEST: image deletions should propagate 2024-04-06T10:08:43.510 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:2024-04-06 10:08:43 TEST: image deletions should propagate 2024-04-06T10:08:43.510 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_pool_images cluster1 mirror 0 2024-04-06T10:08:43.510 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T10:08:43.510 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:08:43.510 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image_count=0 2024-04-06T10:08:43.510 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local s 2024-04-06T10:08:43.510 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local count 2024-04-06T10:08:43.510 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local last_count=0 2024-04-06T10:08:43.510 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ true 2024-04-06T10:08:43.510 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 1 40 2024-04-06T10:08:43.511 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in `seq 1 40` 2024-04-06T10:08:43.511 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 1 -ne 1 2024-04-06T10:08:43.511 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster1 -p mirror mirror pool status 2024-04-06T10:08:43.511 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep 'images: ' 2024-04-06T10:08:43.511 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut '-d ' -f 2 2024-04-06T10:08:43.553 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:rbd: failed to query services: (13) Permission denied 2024-04-06T10:08:43.557 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ count=0 2024-04-06T10:08:43.558 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 0 = 0 2024-04-06T10:08:43.558 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T10:08:43.558 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_pool_healthy cluster1 mirror 0 2024-04-06T10:08:43.558 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T10:08:43.558 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:08:43.558 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local s 2024-04-06T10:08:43.558 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state 2024-04-06T10:08:43.558 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 1 40 2024-04-06T10:08:43.559 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in `seq 1 40` 2024-04-06T10:08:43.559 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 1 -ne 1 2024-04-06T10:08:43.559 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster1 -p mirror mirror pool status 2024-04-06T10:08:43.559 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep 'image health:' 2024-04-06T10:08:43.559 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cut '-d ' -f 3 2024-04-06T10:08:43.595 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:rbd: failed to query services: (13) Permission denied 2024-04-06T10:08:43.598 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ state=OK 2024-04-06T10:08:43.599 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test OK = ERROR 2024-04-06T10:08:43.599 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test OK = OK 2024-04-06T10:08:43.599 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T10:08:43.599 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 1 50 2024-04-06T10:08:43.600 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-06T10:08:43.601 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ image=image_1 2024-04-06T10:08:43.601 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_image_present cluster1 mirror image_1 deleted 2024-04-06T10:08:43.601 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T10:08:43.601 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:08:43.601 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_1 2024-04-06T10:08:43.601 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state=deleted 2024-04-06T10:08:43.601 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image_id= 2024-04-06T10:08:43.601 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local s 2024-04-06T10:08:43.601 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n '' 2024-04-06T10:08:43.601 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_image_id cluster1 mirror image_1 2024-04-06T10:08:43.601 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T10:08:43.601 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:08:43.601 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_1 2024-04-06T10:08:43.601 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster=cluster1 info mirror/image_1 2024-04-06T10:08:43.602 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-04-06T10:08:43.636 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:rbd: error opening image image_1: (2) No such file or directory 2024-04-06T10:08:43.639 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ image_id= 2024-04-06T10:08:43.639 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-04-06T10:08:43.639 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.1 2024-04-06T10:08:43.741 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test_image_present cluster1 mirror image_1 deleted '' 2024-04-06T10:08:43.742 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T10:08:43.742 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:08:43.742 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_1 2024-04-06T10:08:43.742 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local test_state=deleted 2024-04-06T10:08:43.742 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image_id= 2024-04-06T10:08:43.742 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local current_state=deleted 2024-04-06T10:08:43.742 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local current_image_id 2024-04-06T10:08:43.742 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_image_id cluster1 mirror image_1 2024-04-06T10:08:43.742 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T10:08:43.742 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:08:43.742 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_1 2024-04-06T10:08:43.743 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster=cluster1 info mirror/image_1 2024-04-06T10:08:43.743 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-04-06T10:08:43.787 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:rbd: error opening image image_1: (2) No such file or directory 2024-04-06T10:08:43.792 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ current_image_id= 2024-04-06T10:08:43.792 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n '' 2024-04-06T10:08:43.792 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test deleted = deleted 2024-04-06T10:08:43.793 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T10:08:43.793 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-06T10:08:43.793 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ image=image_2 2024-04-06T10:08:43.793 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_image_present cluster1 mirror image_2 deleted 2024-04-06T10:08:43.793 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T10:08:43.793 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:08:43.793 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_2 2024-04-06T10:08:43.793 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state=deleted 2024-04-06T10:08:43.793 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image_id= 2024-04-06T10:08:43.793 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local s 2024-04-06T10:08:43.793 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n '' 2024-04-06T10:08:43.793 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_image_id cluster1 mirror image_2 2024-04-06T10:08:43.793 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T10:08:43.793 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:08:43.793 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_2 2024-04-06T10:08:43.794 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster=cluster1 info mirror/image_2 2024-04-06T10:08:43.794 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-04-06T10:08:43.832 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:rbd: error opening image image_2: (2) No such file or directory 2024-04-06T10:08:43.836 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ image_id= 2024-04-06T10:08:43.836 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-04-06T10:08:43.836 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.1 2024-04-06T10:08:43.937 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test_image_present cluster1 mirror image_2 deleted '' 2024-04-06T10:08:43.938 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T10:08:43.938 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:08:43.938 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_2 2024-04-06T10:08:43.938 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local test_state=deleted 2024-04-06T10:08:43.938 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image_id= 2024-04-06T10:08:43.938 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local current_state=deleted 2024-04-06T10:08:43.938 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local current_image_id 2024-04-06T10:08:43.939 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_image_id cluster1 mirror image_2 2024-04-06T10:08:43.939 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T10:08:43.939 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:08:43.939 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_2 2024-04-06T10:08:43.939 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster=cluster1 info mirror/image_2 2024-04-06T10:08:43.939 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-04-06T10:08:43.989 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:rbd: error opening image image_2: (2) No such file or directory 2024-04-06T10:08:43.994 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ current_image_id= 2024-04-06T10:08:43.994 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n '' 2024-04-06T10:08:43.994 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test deleted = deleted 2024-04-06T10:08:43.994 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T10:08:43.995 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-06T10:08:43.995 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ image=image_3 2024-04-06T10:08:43.995 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_image_present cluster1 mirror image_3 deleted 2024-04-06T10:08:43.995 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T10:08:43.995 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:08:43.995 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_3 2024-04-06T10:08:43.995 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state=deleted 2024-04-06T10:08:43.995 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image_id= 2024-04-06T10:08:43.995 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local s 2024-04-06T10:08:43.995 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n '' 2024-04-06T10:08:43.995 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_image_id cluster1 mirror image_3 2024-04-06T10:08:43.995 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T10:08:43.995 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:08:43.995 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_3 2024-04-06T10:08:43.995 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster=cluster1 info mirror/image_3 2024-04-06T10:08:43.995 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-04-06T10:08:44.033 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:rbd: error opening image image_3: (2) No such file or directory 2024-04-06T10:08:44.038 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ image_id= 2024-04-06T10:08:44.038 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-04-06T10:08:44.038 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.1 2024-04-06T10:08:44.139 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test_image_present cluster1 mirror image_3 deleted '' 2024-04-06T10:08:44.139 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T10:08:44.139 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:08:44.140 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_3 2024-04-06T10:08:44.140 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local test_state=deleted 2024-04-06T10:08:44.140 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image_id= 2024-04-06T10:08:44.140 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local current_state=deleted 2024-04-06T10:08:44.140 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local current_image_id 2024-04-06T10:08:44.141 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_image_id cluster1 mirror image_3 2024-04-06T10:08:44.141 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T10:08:44.141 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:08:44.141 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_3 2024-04-06T10:08:44.141 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster=cluster1 info mirror/image_3 2024-04-06T10:08:44.141 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-04-06T10:08:44.187 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:rbd: error opening image image_3: (2) No such file or directory 2024-04-06T10:08:44.192 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ current_image_id= 2024-04-06T10:08:44.192 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n '' 2024-04-06T10:08:44.192 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test deleted = deleted 2024-04-06T10:08:44.192 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T10:08:44.193 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-06T10:08:44.193 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ image=image_4 2024-04-06T10:08:44.193 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_image_present cluster1 mirror image_4 deleted 2024-04-06T10:08:44.193 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T10:08:44.193 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:08:44.193 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_4 2024-04-06T10:08:44.193 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state=deleted 2024-04-06T10:08:44.193 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image_id= 2024-04-06T10:08:44.193 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local s 2024-04-06T10:08:44.193 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n '' 2024-04-06T10:08:44.193 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_image_id cluster1 mirror image_4 2024-04-06T10:08:44.193 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T10:08:44.193 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:08:44.193 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_4 2024-04-06T10:08:44.193 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster=cluster1 info mirror/image_4 2024-04-06T10:08:44.194 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-04-06T10:08:44.256 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:rbd: error opening image image_4: (2) No such file or directory 2024-04-06T10:08:44.260 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ image_id= 2024-04-06T10:08:44.260 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-04-06T10:08:44.260 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.1 2024-04-06T10:08:44.362 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test_image_present cluster1 mirror image_4 deleted '' 2024-04-06T10:08:44.362 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T10:08:44.362 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:08:44.362 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_4 2024-04-06T10:08:44.362 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local test_state=deleted 2024-04-06T10:08:44.362 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image_id= 2024-04-06T10:08:44.362 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local current_state=deleted 2024-04-06T10:08:44.362 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local current_image_id 2024-04-06T10:08:44.363 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_image_id cluster1 mirror image_4 2024-04-06T10:08:44.363 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T10:08:44.363 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:08:44.363 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_4 2024-04-06T10:08:44.363 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster=cluster1 info mirror/image_4 2024-04-06T10:08:44.363 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-04-06T10:08:44.407 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:rbd: error opening image image_4: (2) No such file or directory 2024-04-06T10:08:44.413 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ current_image_id= 2024-04-06T10:08:44.413 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n '' 2024-04-06T10:08:44.413 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test deleted = deleted 2024-04-06T10:08:44.413 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T10:08:44.413 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-06T10:08:44.413 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ image=image_5 2024-04-06T10:08:44.413 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_image_present cluster1 mirror image_5 deleted 2024-04-06T10:08:44.413 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T10:08:44.413 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:08:44.413 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_5 2024-04-06T10:08:44.414 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state=deleted 2024-04-06T10:08:44.414 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image_id= 2024-04-06T10:08:44.414 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local s 2024-04-06T10:08:44.414 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n '' 2024-04-06T10:08:44.414 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_image_id cluster1 mirror image_5 2024-04-06T10:08:44.414 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T10:08:44.414 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:08:44.414 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_5 2024-04-06T10:08:44.414 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster=cluster1 info mirror/image_5 2024-04-06T10:08:44.414 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-04-06T10:08:44.448 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:rbd: error opening image image_5: (2) No such file or directory 2024-04-06T10:08:44.452 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ image_id= 2024-04-06T10:08:44.452 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-04-06T10:08:44.452 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.1 2024-04-06T10:08:44.554 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test_image_present cluster1 mirror image_5 deleted '' 2024-04-06T10:08:44.554 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T10:08:44.554 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:08:44.554 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_5 2024-04-06T10:08:44.554 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local test_state=deleted 2024-04-06T10:08:44.554 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image_id= 2024-04-06T10:08:44.554 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local current_state=deleted 2024-04-06T10:08:44.554 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local current_image_id 2024-04-06T10:08:44.555 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_image_id cluster1 mirror image_5 2024-04-06T10:08:44.555 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T10:08:44.555 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:08:44.555 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_5 2024-04-06T10:08:44.555 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster=cluster1 info mirror/image_5 2024-04-06T10:08:44.555 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-04-06T10:08:44.599 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:rbd: error opening image image_5: (2) No such file or directory 2024-04-06T10:08:44.604 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ current_image_id= 2024-04-06T10:08:44.604 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n '' 2024-04-06T10:08:44.604 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test deleted = deleted 2024-04-06T10:08:44.604 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T10:08:44.605 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-06T10:08:44.605 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ image=image_6 2024-04-06T10:08:44.605 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_image_present cluster1 mirror image_6 deleted 2024-04-06T10:08:44.605 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T10:08:44.605 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:08:44.605 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_6 2024-04-06T10:08:44.605 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state=deleted 2024-04-06T10:08:44.605 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image_id= 2024-04-06T10:08:44.605 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local s 2024-04-06T10:08:44.605 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n '' 2024-04-06T10:08:44.605 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_image_id cluster1 mirror image_6 2024-04-06T10:08:44.605 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T10:08:44.605 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:08:44.605 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_6 2024-04-06T10:08:44.605 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster=cluster1 info mirror/image_6 2024-04-06T10:08:44.606 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-04-06T10:08:44.645 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:rbd: error opening image image_6: (2) No such file or directory 2024-04-06T10:08:44.649 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ image_id= 2024-04-06T10:08:44.649 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-04-06T10:08:44.649 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.1 2024-04-06T10:08:44.751 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test_image_present cluster1 mirror image_6 deleted '' 2024-04-06T10:08:44.751 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T10:08:44.751 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:08:44.751 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_6 2024-04-06T10:08:44.751 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local test_state=deleted 2024-04-06T10:08:44.751 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image_id= 2024-04-06T10:08:44.751 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local current_state=deleted 2024-04-06T10:08:44.751 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local current_image_id 2024-04-06T10:08:44.751 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_image_id cluster1 mirror image_6 2024-04-06T10:08:44.751 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T10:08:44.751 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:08:44.751 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_6 2024-04-06T10:08:44.752 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster=cluster1 info mirror/image_6 2024-04-06T10:08:44.752 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-04-06T10:08:44.799 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:rbd: error opening image image_6: (2) No such file or directory 2024-04-06T10:08:44.804 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ current_image_id= 2024-04-06T10:08:44.804 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n '' 2024-04-06T10:08:44.805 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test deleted = deleted 2024-04-06T10:08:44.805 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T10:08:44.805 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-06T10:08:44.805 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ image=image_7 2024-04-06T10:08:44.805 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_image_present cluster1 mirror image_7 deleted 2024-04-06T10:08:44.805 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T10:08:44.805 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:08:44.805 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_7 2024-04-06T10:08:44.805 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state=deleted 2024-04-06T10:08:44.805 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image_id= 2024-04-06T10:08:44.805 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local s 2024-04-06T10:08:44.805 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n '' 2024-04-06T10:08:44.805 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_image_id cluster1 mirror image_7 2024-04-06T10:08:44.805 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T10:08:44.806 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:08:44.806 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_7 2024-04-06T10:08:44.806 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster=cluster1 info mirror/image_7 2024-04-06T10:08:44.806 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-04-06T10:08:44.843 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:rbd: error opening image image_7: (2) No such file or directory 2024-04-06T10:08:44.847 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ image_id= 2024-04-06T10:08:44.847 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-04-06T10:08:44.847 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.1 2024-04-06T10:08:44.949 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test_image_present cluster1 mirror image_7 deleted '' 2024-04-06T10:08:44.949 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T10:08:44.949 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:08:44.949 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_7 2024-04-06T10:08:44.949 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local test_state=deleted 2024-04-06T10:08:44.949 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image_id= 2024-04-06T10:08:44.949 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local current_state=deleted 2024-04-06T10:08:44.949 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local current_image_id 2024-04-06T10:08:44.950 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_image_id cluster1 mirror image_7 2024-04-06T10:08:44.950 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T10:08:44.950 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:08:44.950 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_7 2024-04-06T10:08:44.950 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster=cluster1 info mirror/image_7 2024-04-06T10:08:44.951 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-04-06T10:08:45.001 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:rbd: error opening image image_7: (2) No such file or directory 2024-04-06T10:08:45.007 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ current_image_id= 2024-04-06T10:08:45.007 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n '' 2024-04-06T10:08:45.007 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test deleted = deleted 2024-04-06T10:08:45.007 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T10:08:45.007 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-06T10:08:45.007 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ image=image_8 2024-04-06T10:08:45.007 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_image_present cluster1 mirror image_8 deleted 2024-04-06T10:08:45.007 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T10:08:45.007 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:08:45.007 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_8 2024-04-06T10:08:45.007 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state=deleted 2024-04-06T10:08:45.007 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image_id= 2024-04-06T10:08:45.008 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local s 2024-04-06T10:08:45.008 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n '' 2024-04-06T10:08:45.008 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_image_id cluster1 mirror image_8 2024-04-06T10:08:45.008 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T10:08:45.008 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:08:45.008 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_8 2024-04-06T10:08:45.008 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster=cluster1 info mirror/image_8 2024-04-06T10:08:45.008 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-04-06T10:08:45.047 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:rbd: error opening image image_8: (2) No such file or directory 2024-04-06T10:08:45.051 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ image_id= 2024-04-06T10:08:45.051 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-04-06T10:08:45.051 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.1 2024-04-06T10:08:45.152 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test_image_present cluster1 mirror image_8 deleted '' 2024-04-06T10:08:45.153 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T10:08:45.153 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:08:45.153 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_8 2024-04-06T10:08:45.153 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local test_state=deleted 2024-04-06T10:08:45.153 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image_id= 2024-04-06T10:08:45.153 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local current_state=deleted 2024-04-06T10:08:45.153 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local current_image_id 2024-04-06T10:08:45.154 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_image_id cluster1 mirror image_8 2024-04-06T10:08:45.154 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T10:08:45.154 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:08:45.154 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_8 2024-04-06T10:08:45.154 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster=cluster1 info mirror/image_8 2024-04-06T10:08:45.154 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-04-06T10:08:45.202 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:rbd: error opening image image_8: (2) No such file or directory 2024-04-06T10:08:45.208 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ current_image_id= 2024-04-06T10:08:45.208 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n '' 2024-04-06T10:08:45.208 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test deleted = deleted 2024-04-06T10:08:45.208 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T10:08:45.208 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-06T10:08:45.208 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ image=image_9 2024-04-06T10:08:45.208 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_image_present cluster1 mirror image_9 deleted 2024-04-06T10:08:45.208 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T10:08:45.208 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:08:45.208 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_9 2024-04-06T10:08:45.209 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state=deleted 2024-04-06T10:08:45.209 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image_id= 2024-04-06T10:08:45.209 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local s 2024-04-06T10:08:45.209 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n '' 2024-04-06T10:08:45.209 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_image_id cluster1 mirror image_9 2024-04-06T10:08:45.209 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T10:08:45.209 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:08:45.209 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_9 2024-04-06T10:08:45.209 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster=cluster1 info mirror/image_9 2024-04-06T10:08:45.209 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-04-06T10:08:45.247 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:rbd: error opening image image_9: (2) No such file or directory 2024-04-06T10:08:45.251 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ image_id= 2024-04-06T10:08:45.251 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-04-06T10:08:45.251 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.1 2024-04-06T10:08:45.353 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test_image_present cluster1 mirror image_9 deleted '' 2024-04-06T10:08:45.353 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T10:08:45.353 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:08:45.353 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_9 2024-04-06T10:08:45.353 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local test_state=deleted 2024-04-06T10:08:45.353 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image_id= 2024-04-06T10:08:45.353 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local current_state=deleted 2024-04-06T10:08:45.353 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local current_image_id 2024-04-06T10:08:45.353 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_image_id cluster1 mirror image_9 2024-04-06T10:08:45.353 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T10:08:45.354 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:08:45.354 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_9 2024-04-06T10:08:45.354 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster=cluster1 info mirror/image_9 2024-04-06T10:08:45.354 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-04-06T10:08:45.399 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:rbd: error opening image image_9: (2) No such file or directory 2024-04-06T10:08:45.404 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ current_image_id= 2024-04-06T10:08:45.404 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n '' 2024-04-06T10:08:45.404 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test deleted = deleted 2024-04-06T10:08:45.404 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T10:08:45.404 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-06T10:08:45.404 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ image=image_10 2024-04-06T10:08:45.404 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_image_present cluster1 mirror image_10 deleted 2024-04-06T10:08:45.404 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T10:08:45.404 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:08:45.405 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_10 2024-04-06T10:08:45.405 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state=deleted 2024-04-06T10:08:45.405 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image_id= 2024-04-06T10:08:45.405 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local s 2024-04-06T10:08:45.405 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n '' 2024-04-06T10:08:45.405 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_image_id cluster1 mirror image_10 2024-04-06T10:08:45.405 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T10:08:45.405 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:08:45.405 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_10 2024-04-06T10:08:45.405 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster=cluster1 info mirror/image_10 2024-04-06T10:08:45.405 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-04-06T10:08:45.443 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:rbd: error opening image image_10: (2) No such file or directory 2024-04-06T10:08:45.447 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ image_id= 2024-04-06T10:08:45.447 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-04-06T10:08:45.447 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.1 2024-04-06T10:08:45.548 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test_image_present cluster1 mirror image_10 deleted '' 2024-04-06T10:08:45.548 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T10:08:45.548 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:08:45.548 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_10 2024-04-06T10:08:45.548 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local test_state=deleted 2024-04-06T10:08:45.549 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image_id= 2024-04-06T10:08:45.549 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local current_state=deleted 2024-04-06T10:08:45.549 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local current_image_id 2024-04-06T10:08:45.550 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_image_id cluster1 mirror image_10 2024-04-06T10:08:45.550 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T10:08:45.550 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:08:45.550 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_10 2024-04-06T10:08:45.550 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster=cluster1 info mirror/image_10 2024-04-06T10:08:45.550 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-04-06T10:08:45.597 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:rbd: error opening image image_10: (2) No such file or directory 2024-04-06T10:08:45.602 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ current_image_id= 2024-04-06T10:08:45.602 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n '' 2024-04-06T10:08:45.602 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test deleted = deleted 2024-04-06T10:08:45.602 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T10:08:45.602 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-06T10:08:45.602 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ image=image_11 2024-04-06T10:08:45.602 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_image_present cluster1 mirror image_11 deleted 2024-04-06T10:08:45.602 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T10:08:45.602 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:08:45.602 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_11 2024-04-06T10:08:45.603 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state=deleted 2024-04-06T10:08:45.603 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image_id= 2024-04-06T10:08:45.603 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local s 2024-04-06T10:08:45.603 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n '' 2024-04-06T10:08:45.603 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_image_id cluster1 mirror image_11 2024-04-06T10:08:45.603 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T10:08:45.603 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:08:45.603 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_11 2024-04-06T10:08:45.603 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster=cluster1 info mirror/image_11 2024-04-06T10:08:45.603 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-04-06T10:08:45.641 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:rbd: error opening image image_11: (2) No such file or directory 2024-04-06T10:08:45.645 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ image_id= 2024-04-06T10:08:45.646 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-04-06T10:08:45.646 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.1 2024-04-06T10:08:45.747 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test_image_present cluster1 mirror image_11 deleted '' 2024-04-06T10:08:45.747 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T10:08:45.748 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:08:45.748 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_11 2024-04-06T10:08:45.748 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local test_state=deleted 2024-04-06T10:08:45.748 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image_id= 2024-04-06T10:08:45.748 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local current_state=deleted 2024-04-06T10:08:45.748 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local current_image_id 2024-04-06T10:08:45.749 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_image_id cluster1 mirror image_11 2024-04-06T10:08:45.749 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T10:08:45.749 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:08:45.749 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_11 2024-04-06T10:08:45.749 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster=cluster1 info mirror/image_11 2024-04-06T10:08:45.749 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-04-06T10:08:45.793 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:rbd: error opening image image_11: (2) No such file or directory 2024-04-06T10:08:45.799 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ current_image_id= 2024-04-06T10:08:45.799 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n '' 2024-04-06T10:08:45.799 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test deleted = deleted 2024-04-06T10:08:45.799 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T10:08:45.799 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-06T10:08:45.799 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ image=image_12 2024-04-06T10:08:45.799 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_image_present cluster1 mirror image_12 deleted 2024-04-06T10:08:45.800 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T10:08:45.800 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:08:45.800 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_12 2024-04-06T10:08:45.800 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state=deleted 2024-04-06T10:08:45.800 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image_id= 2024-04-06T10:08:45.800 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local s 2024-04-06T10:08:45.800 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n '' 2024-04-06T10:08:45.800 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_image_id cluster1 mirror image_12 2024-04-06T10:08:45.800 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T10:08:45.800 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:08:45.800 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_12 2024-04-06T10:08:45.800 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster=cluster1 info mirror/image_12 2024-04-06T10:08:45.800 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-04-06T10:08:45.838 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:rbd: error opening image image_12: (2) No such file or directory 2024-04-06T10:08:45.842 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ image_id= 2024-04-06T10:08:45.842 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-04-06T10:08:45.843 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.1 2024-04-06T10:08:45.944 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test_image_present cluster1 mirror image_12 deleted '' 2024-04-06T10:08:45.944 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T10:08:45.944 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:08:45.944 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_12 2024-04-06T10:08:45.945 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local test_state=deleted 2024-04-06T10:08:45.945 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image_id= 2024-04-06T10:08:45.945 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local current_state=deleted 2024-04-06T10:08:45.945 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local current_image_id 2024-04-06T10:08:45.945 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_image_id cluster1 mirror image_12 2024-04-06T10:08:45.945 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T10:08:45.945 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:08:45.945 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_12 2024-04-06T10:08:45.945 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster=cluster1 info mirror/image_12 2024-04-06T10:08:45.946 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-04-06T10:08:45.992 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:rbd: error opening image image_12: (2) No such file or directory 2024-04-06T10:08:45.997 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ current_image_id= 2024-04-06T10:08:45.997 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n '' 2024-04-06T10:08:45.997 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test deleted = deleted 2024-04-06T10:08:45.997 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T10:08:45.997 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-06T10:08:45.998 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ image=image_13 2024-04-06T10:08:45.998 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_image_present cluster1 mirror image_13 deleted 2024-04-06T10:08:45.998 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T10:08:45.998 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:08:45.998 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_13 2024-04-06T10:08:45.998 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state=deleted 2024-04-06T10:08:45.998 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image_id= 2024-04-06T10:08:45.998 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local s 2024-04-06T10:08:45.998 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n '' 2024-04-06T10:08:45.998 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_image_id cluster1 mirror image_13 2024-04-06T10:08:45.998 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T10:08:45.998 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:08:45.998 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_13 2024-04-06T10:08:45.999 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster=cluster1 info mirror/image_13 2024-04-06T10:08:45.999 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-04-06T10:08:46.035 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:rbd: error opening image image_13: (2) No such file or directory 2024-04-06T10:08:46.039 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ image_id= 2024-04-06T10:08:46.039 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-04-06T10:08:46.039 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.1 2024-04-06T10:08:46.141 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test_image_present cluster1 mirror image_13 deleted '' 2024-04-06T10:08:46.141 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T10:08:46.141 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:08:46.141 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_13 2024-04-06T10:08:46.141 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local test_state=deleted 2024-04-06T10:08:46.141 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image_id= 2024-04-06T10:08:46.141 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local current_state=deleted 2024-04-06T10:08:46.141 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local current_image_id 2024-04-06T10:08:46.142 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_image_id cluster1 mirror image_13 2024-04-06T10:08:46.142 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T10:08:46.142 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:08:46.142 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_13 2024-04-06T10:08:46.142 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster=cluster1 info mirror/image_13 2024-04-06T10:08:46.142 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-04-06T10:08:46.189 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:rbd: error opening image image_13: (2) No such file or directory 2024-04-06T10:08:46.199 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ current_image_id= 2024-04-06T10:08:46.199 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n '' 2024-04-06T10:08:46.199 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test deleted = deleted 2024-04-06T10:08:46.199 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T10:08:46.199 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-06T10:08:46.199 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ image=image_14 2024-04-06T10:08:46.199 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_image_present cluster1 mirror image_14 deleted 2024-04-06T10:08:46.199 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T10:08:46.199 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:08:46.199 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_14 2024-04-06T10:08:46.199 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state=deleted 2024-04-06T10:08:46.199 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image_id= 2024-04-06T10:08:46.199 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local s 2024-04-06T10:08:46.199 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n '' 2024-04-06T10:08:46.200 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_image_id cluster1 mirror image_14 2024-04-06T10:08:46.200 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T10:08:46.200 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:08:46.200 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_14 2024-04-06T10:08:46.200 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster=cluster1 info mirror/image_14 2024-04-06T10:08:46.200 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-04-06T10:08:46.237 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:rbd: error opening image image_14: (2) No such file or directory 2024-04-06T10:08:46.241 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ image_id= 2024-04-06T10:08:46.241 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-04-06T10:08:46.241 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.1 2024-04-06T10:08:46.343 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test_image_present cluster1 mirror image_14 deleted '' 2024-04-06T10:08:46.343 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T10:08:46.343 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:08:46.343 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_14 2024-04-06T10:08:46.343 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local test_state=deleted 2024-04-06T10:08:46.344 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image_id= 2024-04-06T10:08:46.344 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local current_state=deleted 2024-04-06T10:08:46.344 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local current_image_id 2024-04-06T10:08:46.344 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_image_id cluster1 mirror image_14 2024-04-06T10:08:46.344 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T10:08:46.344 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:08:46.344 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_14 2024-04-06T10:08:46.345 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster=cluster1 info mirror/image_14 2024-04-06T10:08:46.345 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-04-06T10:08:46.389 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:rbd: error opening image image_14: (2) No such file or directory 2024-04-06T10:08:46.395 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ current_image_id= 2024-04-06T10:08:46.395 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n '' 2024-04-06T10:08:46.395 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test deleted = deleted 2024-04-06T10:08:46.395 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T10:08:46.395 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-06T10:08:46.395 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ image=image_15 2024-04-06T10:08:46.395 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_image_present cluster1 mirror image_15 deleted 2024-04-06T10:08:46.395 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T10:08:46.395 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:08:46.396 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_15 2024-04-06T10:08:46.396 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state=deleted 2024-04-06T10:08:46.396 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image_id= 2024-04-06T10:08:46.396 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local s 2024-04-06T10:08:46.396 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n '' 2024-04-06T10:08:46.396 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_image_id cluster1 mirror image_15 2024-04-06T10:08:46.396 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T10:08:46.396 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:08:46.396 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_15 2024-04-06T10:08:46.396 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster=cluster1 info mirror/image_15 2024-04-06T10:08:46.396 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-04-06T10:08:46.434 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:rbd: error opening image image_15: (2) No such file or directory 2024-04-06T10:08:46.438 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ image_id= 2024-04-06T10:08:46.438 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-04-06T10:08:46.438 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.1 2024-04-06T10:08:46.539 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test_image_present cluster1 mirror image_15 deleted '' 2024-04-06T10:08:46.540 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T10:08:46.540 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:08:46.540 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_15 2024-04-06T10:08:46.540 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local test_state=deleted 2024-04-06T10:08:46.540 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image_id= 2024-04-06T10:08:46.540 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local current_state=deleted 2024-04-06T10:08:46.540 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local current_image_id 2024-04-06T10:08:46.540 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_image_id cluster1 mirror image_15 2024-04-06T10:08:46.540 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T10:08:46.540 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:08:46.540 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_15 2024-04-06T10:08:46.541 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster=cluster1 info mirror/image_15 2024-04-06T10:08:46.541 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-04-06T10:08:46.587 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:rbd: error opening image image_15: (2) No such file or directory 2024-04-06T10:08:46.592 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ current_image_id= 2024-04-06T10:08:46.592 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n '' 2024-04-06T10:08:46.592 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test deleted = deleted 2024-04-06T10:08:46.592 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T10:08:46.592 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-06T10:08:46.592 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ image=image_16 2024-04-06T10:08:46.593 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_image_present cluster1 mirror image_16 deleted 2024-04-06T10:08:46.593 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T10:08:46.593 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:08:46.593 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_16 2024-04-06T10:08:46.593 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state=deleted 2024-04-06T10:08:46.593 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image_id= 2024-04-06T10:08:46.593 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local s 2024-04-06T10:08:46.593 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n '' 2024-04-06T10:08:46.593 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_image_id cluster1 mirror image_16 2024-04-06T10:08:46.593 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T10:08:46.593 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:08:46.593 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_16 2024-04-06T10:08:46.593 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster=cluster1 info mirror/image_16 2024-04-06T10:08:46.593 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-04-06T10:08:46.631 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:rbd: error opening image image_16: (2) No such file or directory 2024-04-06T10:08:46.635 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ image_id= 2024-04-06T10:08:46.635 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-04-06T10:08:46.635 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.1 2024-04-06T10:08:46.737 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test_image_present cluster1 mirror image_16 deleted '' 2024-04-06T10:08:46.737 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T10:08:46.737 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:08:46.737 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_16 2024-04-06T10:08:46.737 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local test_state=deleted 2024-04-06T10:08:46.737 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image_id= 2024-04-06T10:08:46.737 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local current_state=deleted 2024-04-06T10:08:46.737 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local current_image_id 2024-04-06T10:08:46.738 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_image_id cluster1 mirror image_16 2024-04-06T10:08:46.738 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T10:08:46.739 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:08:46.739 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_16 2024-04-06T10:08:46.739 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster=cluster1 info mirror/image_16 2024-04-06T10:08:46.739 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-04-06T10:08:46.784 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:rbd: error opening image image_16: (2) No such file or directory 2024-04-06T10:08:46.789 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ current_image_id= 2024-04-06T10:08:46.789 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n '' 2024-04-06T10:08:46.790 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test deleted = deleted 2024-04-06T10:08:46.790 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T10:08:46.790 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-06T10:08:46.790 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ image=image_17 2024-04-06T10:08:46.790 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_image_present cluster1 mirror image_17 deleted 2024-04-06T10:08:46.790 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T10:08:46.790 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:08:46.790 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_17 2024-04-06T10:08:46.790 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state=deleted 2024-04-06T10:08:46.790 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image_id= 2024-04-06T10:08:46.790 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local s 2024-04-06T10:08:46.790 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n '' 2024-04-06T10:08:46.790 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_image_id cluster1 mirror image_17 2024-04-06T10:08:46.790 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T10:08:46.791 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:08:46.791 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_17 2024-04-06T10:08:46.791 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster=cluster1 info mirror/image_17 2024-04-06T10:08:46.791 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-04-06T10:08:46.827 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:rbd: error opening image image_17: (2) No such file or directory 2024-04-06T10:08:46.832 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ image_id= 2024-04-06T10:08:46.832 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-04-06T10:08:46.832 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.1 2024-04-06T10:08:46.933 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test_image_present cluster1 mirror image_17 deleted '' 2024-04-06T10:08:46.934 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T10:08:46.934 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:08:46.934 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_17 2024-04-06T10:08:46.934 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local test_state=deleted 2024-04-06T10:08:46.934 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image_id= 2024-04-06T10:08:46.934 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local current_state=deleted 2024-04-06T10:08:46.934 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local current_image_id 2024-04-06T10:08:46.935 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_image_id cluster1 mirror image_17 2024-04-06T10:08:46.935 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T10:08:46.935 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:08:46.935 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_17 2024-04-06T10:08:46.935 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster=cluster1 info mirror/image_17 2024-04-06T10:08:46.935 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-04-06T10:08:46.979 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:rbd: error opening image image_17: (2) No such file or directory 2024-04-06T10:08:46.985 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ current_image_id= 2024-04-06T10:08:46.985 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n '' 2024-04-06T10:08:46.985 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test deleted = deleted 2024-04-06T10:08:46.985 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T10:08:46.985 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-06T10:08:46.985 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ image=image_18 2024-04-06T10:08:46.985 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_image_present cluster1 mirror image_18 deleted 2024-04-06T10:08:46.985 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T10:08:46.985 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:08:46.985 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_18 2024-04-06T10:08:46.985 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state=deleted 2024-04-06T10:08:46.985 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image_id= 2024-04-06T10:08:46.985 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local s 2024-04-06T10:08:46.986 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n '' 2024-04-06T10:08:46.986 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_image_id cluster1 mirror image_18 2024-04-06T10:08:46.986 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T10:08:46.986 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:08:46.986 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_18 2024-04-06T10:08:46.986 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster=cluster1 info mirror/image_18 2024-04-06T10:08:46.986 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-04-06T10:08:47.025 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:rbd: error opening image image_18: (2) No such file or directory 2024-04-06T10:08:47.029 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ image_id= 2024-04-06T10:08:47.029 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-04-06T10:08:47.029 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.1 2024-04-06T10:08:47.131 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test_image_present cluster1 mirror image_18 deleted '' 2024-04-06T10:08:47.132 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T10:08:47.132 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:08:47.132 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_18 2024-04-06T10:08:47.132 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local test_state=deleted 2024-04-06T10:08:47.132 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image_id= 2024-04-06T10:08:47.132 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local current_state=deleted 2024-04-06T10:08:47.132 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local current_image_id 2024-04-06T10:08:47.132 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_image_id cluster1 mirror image_18 2024-04-06T10:08:47.132 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T10:08:47.132 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:08:47.132 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_18 2024-04-06T10:08:47.133 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster=cluster1 info mirror/image_18 2024-04-06T10:08:47.133 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-04-06T10:08:47.178 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:rbd: error opening image image_18: (2) No such file or directory 2024-04-06T10:08:47.183 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ current_image_id= 2024-04-06T10:08:47.183 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n '' 2024-04-06T10:08:47.183 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test deleted = deleted 2024-04-06T10:08:47.184 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T10:08:47.184 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-06T10:08:47.184 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ image=image_19 2024-04-06T10:08:47.184 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_image_present cluster1 mirror image_19 deleted 2024-04-06T10:08:47.184 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T10:08:47.184 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:08:47.184 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_19 2024-04-06T10:08:47.184 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state=deleted 2024-04-06T10:08:47.184 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image_id= 2024-04-06T10:08:47.184 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local s 2024-04-06T10:08:47.184 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n '' 2024-04-06T10:08:47.184 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_image_id cluster1 mirror image_19 2024-04-06T10:08:47.184 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T10:08:47.184 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:08:47.185 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_19 2024-04-06T10:08:47.185 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster=cluster1 info mirror/image_19 2024-04-06T10:08:47.185 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-04-06T10:08:47.223 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:rbd: error opening image image_19: (2) No such file or directory 2024-04-06T10:08:47.227 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ image_id= 2024-04-06T10:08:47.227 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-04-06T10:08:47.227 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.1 2024-04-06T10:08:47.328 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test_image_present cluster1 mirror image_19 deleted '' 2024-04-06T10:08:47.329 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T10:08:47.329 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:08:47.329 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_19 2024-04-06T10:08:47.329 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local test_state=deleted 2024-04-06T10:08:47.329 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image_id= 2024-04-06T10:08:47.329 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local current_state=deleted 2024-04-06T10:08:47.329 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local current_image_id 2024-04-06T10:08:47.330 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_image_id cluster1 mirror image_19 2024-04-06T10:08:47.330 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T10:08:47.330 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:08:47.331 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_19 2024-04-06T10:08:47.331 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster=cluster1 info mirror/image_19 2024-04-06T10:08:47.331 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-04-06T10:08:47.377 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:rbd: error opening image image_19: (2) No such file or directory 2024-04-06T10:08:47.383 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ current_image_id= 2024-04-06T10:08:47.383 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n '' 2024-04-06T10:08:47.383 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test deleted = deleted 2024-04-06T10:08:47.383 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T10:08:47.383 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-06T10:08:47.383 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ image=image_20 2024-04-06T10:08:47.383 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_image_present cluster1 mirror image_20 deleted 2024-04-06T10:08:47.383 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T10:08:47.383 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:08:47.383 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_20 2024-04-06T10:08:47.383 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state=deleted 2024-04-06T10:08:47.383 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image_id= 2024-04-06T10:08:47.383 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local s 2024-04-06T10:08:47.383 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n '' 2024-04-06T10:08:47.384 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_image_id cluster1 mirror image_20 2024-04-06T10:08:47.384 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T10:08:47.384 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:08:47.384 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_20 2024-04-06T10:08:47.384 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster=cluster1 info mirror/image_20 2024-04-06T10:08:47.384 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-04-06T10:08:47.420 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:rbd: error opening image image_20: (2) No such file or directory 2024-04-06T10:08:47.424 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ image_id= 2024-04-06T10:08:47.424 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-04-06T10:08:47.424 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.1 2024-04-06T10:08:47.526 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test_image_present cluster1 mirror image_20 deleted '' 2024-04-06T10:08:47.526 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T10:08:47.526 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:08:47.526 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_20 2024-04-06T10:08:47.526 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local test_state=deleted 2024-04-06T10:08:47.526 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image_id= 2024-04-06T10:08:47.526 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local current_state=deleted 2024-04-06T10:08:47.526 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local current_image_id 2024-04-06T10:08:47.527 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_image_id cluster1 mirror image_20 2024-04-06T10:08:47.527 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T10:08:47.527 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:08:47.527 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_20 2024-04-06T10:08:47.527 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster=cluster1 info mirror/image_20 2024-04-06T10:08:47.527 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-04-06T10:08:47.571 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:rbd: error opening image image_20: (2) No such file or directory 2024-04-06T10:08:47.576 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ current_image_id= 2024-04-06T10:08:47.576 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n '' 2024-04-06T10:08:47.576 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test deleted = deleted 2024-04-06T10:08:47.576 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T10:08:47.577 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-06T10:08:47.577 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ image=image_21 2024-04-06T10:08:47.577 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_image_present cluster1 mirror image_21 deleted 2024-04-06T10:08:47.577 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T10:08:47.577 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:08:47.577 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_21 2024-04-06T10:08:47.577 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state=deleted 2024-04-06T10:08:47.577 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image_id= 2024-04-06T10:08:47.577 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local s 2024-04-06T10:08:47.577 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n '' 2024-04-06T10:08:47.577 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_image_id cluster1 mirror image_21 2024-04-06T10:08:47.577 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T10:08:47.577 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:08:47.578 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_21 2024-04-06T10:08:47.578 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster=cluster1 info mirror/image_21 2024-04-06T10:08:47.578 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-04-06T10:08:47.615 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:rbd: error opening image image_21: (2) No such file or directory 2024-04-06T10:08:47.619 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ image_id= 2024-04-06T10:08:47.620 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-04-06T10:08:47.620 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.1 2024-04-06T10:08:47.721 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test_image_present cluster1 mirror image_21 deleted '' 2024-04-06T10:08:47.721 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T10:08:47.721 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:08:47.721 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_21 2024-04-06T10:08:47.721 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local test_state=deleted 2024-04-06T10:08:47.721 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image_id= 2024-04-06T10:08:47.721 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local current_state=deleted 2024-04-06T10:08:47.722 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local current_image_id 2024-04-06T10:08:47.722 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_image_id cluster1 mirror image_21 2024-04-06T10:08:47.722 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T10:08:47.723 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:08:47.723 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_21 2024-04-06T10:08:47.723 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster=cluster1 info mirror/image_21 2024-04-06T10:08:47.723 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-04-06T10:08:47.766 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:rbd: error opening image image_21: (2) No such file or directory 2024-04-06T10:08:47.772 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ current_image_id= 2024-04-06T10:08:47.772 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n '' 2024-04-06T10:08:47.773 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test deleted = deleted 2024-04-06T10:08:47.773 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T10:08:47.773 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-06T10:08:47.773 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ image=image_22 2024-04-06T10:08:47.773 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_image_present cluster1 mirror image_22 deleted 2024-04-06T10:08:47.773 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T10:08:47.773 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:08:47.773 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_22 2024-04-06T10:08:47.773 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state=deleted 2024-04-06T10:08:47.773 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image_id= 2024-04-06T10:08:47.773 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local s 2024-04-06T10:08:47.773 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n '' 2024-04-06T10:08:47.774 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_image_id cluster1 mirror image_22 2024-04-06T10:08:47.774 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T10:08:47.774 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:08:47.774 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_22 2024-04-06T10:08:47.774 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster=cluster1 info mirror/image_22 2024-04-06T10:08:47.774 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-04-06T10:08:47.811 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:rbd: error opening image image_22: (2) No such file or directory 2024-04-06T10:08:47.816 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ image_id= 2024-04-06T10:08:47.816 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-04-06T10:08:47.816 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.1 2024-04-06T10:08:47.917 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test_image_present cluster1 mirror image_22 deleted '' 2024-04-06T10:08:47.917 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T10:08:47.917 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:08:47.918 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_22 2024-04-06T10:08:47.918 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local test_state=deleted 2024-04-06T10:08:47.918 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image_id= 2024-04-06T10:08:47.918 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local current_state=deleted 2024-04-06T10:08:47.918 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local current_image_id 2024-04-06T10:08:47.919 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_image_id cluster1 mirror image_22 2024-04-06T10:08:47.919 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T10:08:47.919 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:08:47.919 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_22 2024-04-06T10:08:47.919 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster=cluster1 info mirror/image_22 2024-04-06T10:08:47.919 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-04-06T10:08:47.966 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:rbd: error opening image image_22: (2) No such file or directory 2024-04-06T10:08:47.971 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ current_image_id= 2024-04-06T10:08:47.972 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n '' 2024-04-06T10:08:47.972 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test deleted = deleted 2024-04-06T10:08:47.972 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T10:08:47.972 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-06T10:08:47.972 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ image=image_23 2024-04-06T10:08:47.972 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_image_present cluster1 mirror image_23 deleted 2024-04-06T10:08:47.972 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T10:08:47.972 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:08:47.972 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_23 2024-04-06T10:08:47.972 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state=deleted 2024-04-06T10:08:47.972 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image_id= 2024-04-06T10:08:47.972 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local s 2024-04-06T10:08:47.972 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n '' 2024-04-06T10:08:47.972 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_image_id cluster1 mirror image_23 2024-04-06T10:08:47.972 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T10:08:47.973 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:08:47.973 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_23 2024-04-06T10:08:47.973 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster=cluster1 info mirror/image_23 2024-04-06T10:08:47.973 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-04-06T10:08:48.012 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:rbd: error opening image image_23: (2) No such file or directory 2024-04-06T10:08:48.016 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ image_id= 2024-04-06T10:08:48.017 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-04-06T10:08:48.017 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.1 2024-04-06T10:08:48.118 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test_image_present cluster1 mirror image_23 deleted '' 2024-04-06T10:08:48.118 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T10:08:48.118 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:08:48.118 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_23 2024-04-06T10:08:48.118 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local test_state=deleted 2024-04-06T10:08:48.119 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image_id= 2024-04-06T10:08:48.119 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local current_state=deleted 2024-04-06T10:08:48.119 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local current_image_id 2024-04-06T10:08:48.119 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_image_id cluster1 mirror image_23 2024-04-06T10:08:48.120 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T10:08:48.120 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:08:48.120 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_23 2024-04-06T10:08:48.120 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster=cluster1 info mirror/image_23 2024-04-06T10:08:48.120 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-04-06T10:08:48.165 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:rbd: error opening image image_23: (2) No such file or directory 2024-04-06T10:08:48.170 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ current_image_id= 2024-04-06T10:08:48.170 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n '' 2024-04-06T10:08:48.170 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test deleted = deleted 2024-04-06T10:08:48.170 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T10:08:48.170 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-06T10:08:48.171 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ image=image_24 2024-04-06T10:08:48.171 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_image_present cluster1 mirror image_24 deleted 2024-04-06T10:08:48.171 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T10:08:48.171 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:08:48.171 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_24 2024-04-06T10:08:48.171 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state=deleted 2024-04-06T10:08:48.171 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image_id= 2024-04-06T10:08:48.171 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local s 2024-04-06T10:08:48.171 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n '' 2024-04-06T10:08:48.171 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_image_id cluster1 mirror image_24 2024-04-06T10:08:48.171 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T10:08:48.171 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:08:48.171 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_24 2024-04-06T10:08:48.171 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster=cluster1 info mirror/image_24 2024-04-06T10:08:48.172 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-04-06T10:08:48.208 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:rbd: error opening image image_24: (2) No such file or directory 2024-04-06T10:08:48.213 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ image_id= 2024-04-06T10:08:48.213 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-04-06T10:08:48.213 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.1 2024-04-06T10:08:48.315 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test_image_present cluster1 mirror image_24 deleted '' 2024-04-06T10:08:48.315 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T10:08:48.315 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:08:48.315 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_24 2024-04-06T10:08:48.315 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local test_state=deleted 2024-04-06T10:08:48.315 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image_id= 2024-04-06T10:08:48.315 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local current_state=deleted 2024-04-06T10:08:48.315 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local current_image_id 2024-04-06T10:08:48.315 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_image_id cluster1 mirror image_24 2024-04-06T10:08:48.315 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T10:08:48.315 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:08:48.315 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_24 2024-04-06T10:08:48.316 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster=cluster1 info mirror/image_24 2024-04-06T10:08:48.316 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-04-06T10:08:48.359 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:rbd: error opening image image_24: (2) No such file or directory 2024-04-06T10:08:48.365 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ current_image_id= 2024-04-06T10:08:48.365 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n '' 2024-04-06T10:08:48.365 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test deleted = deleted 2024-04-06T10:08:48.365 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T10:08:48.365 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-06T10:08:48.365 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ image=image_25 2024-04-06T10:08:48.365 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_image_present cluster1 mirror image_25 deleted 2024-04-06T10:08:48.365 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T10:08:48.365 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:08:48.365 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_25 2024-04-06T10:08:48.365 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state=deleted 2024-04-06T10:08:48.365 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image_id= 2024-04-06T10:08:48.365 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local s 2024-04-06T10:08:48.366 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n '' 2024-04-06T10:08:48.366 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_image_id cluster1 mirror image_25 2024-04-06T10:08:48.366 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T10:08:48.366 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:08:48.366 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_25 2024-04-06T10:08:48.366 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster=cluster1 info mirror/image_25 2024-04-06T10:08:48.366 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-04-06T10:08:48.403 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:rbd: error opening image image_25: (2) No such file or directory 2024-04-06T10:08:48.408 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ image_id= 2024-04-06T10:08:48.408 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-04-06T10:08:48.408 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.1 2024-04-06T10:08:48.510 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test_image_present cluster1 mirror image_25 deleted '' 2024-04-06T10:08:48.510 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T10:08:48.510 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:08:48.510 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_25 2024-04-06T10:08:48.510 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local test_state=deleted 2024-04-06T10:08:48.510 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image_id= 2024-04-06T10:08:48.511 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local current_state=deleted 2024-04-06T10:08:48.511 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local current_image_id 2024-04-06T10:08:48.511 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_image_id cluster1 mirror image_25 2024-04-06T10:08:48.511 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T10:08:48.511 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:08:48.511 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_25 2024-04-06T10:08:48.511 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster=cluster1 info mirror/image_25 2024-04-06T10:08:48.512 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-04-06T10:08:48.559 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:rbd: error opening image image_25: (2) No such file or directory 2024-04-06T10:08:48.564 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ current_image_id= 2024-04-06T10:08:48.564 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n '' 2024-04-06T10:08:48.564 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test deleted = deleted 2024-04-06T10:08:48.564 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T10:08:48.564 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-06T10:08:48.564 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ image=image_26 2024-04-06T10:08:48.564 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_image_present cluster1 mirror image_26 deleted 2024-04-06T10:08:48.564 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T10:08:48.564 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:08:48.564 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_26 2024-04-06T10:08:48.564 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state=deleted 2024-04-06T10:08:48.564 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image_id= 2024-04-06T10:08:48.564 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local s 2024-04-06T10:08:48.565 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n '' 2024-04-06T10:08:48.565 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_image_id cluster1 mirror image_26 2024-04-06T10:08:48.565 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T10:08:48.565 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:08:48.565 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_26 2024-04-06T10:08:48.565 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster=cluster1 info mirror/image_26 2024-04-06T10:08:48.565 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-04-06T10:08:48.601 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:rbd: error opening image image_26: (2) No such file or directory 2024-04-06T10:08:48.605 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ image_id= 2024-04-06T10:08:48.605 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-04-06T10:08:48.605 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.1 2024-04-06T10:08:48.707 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test_image_present cluster1 mirror image_26 deleted '' 2024-04-06T10:08:48.707 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T10:08:48.707 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:08:48.707 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_26 2024-04-06T10:08:48.707 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local test_state=deleted 2024-04-06T10:08:48.707 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image_id= 2024-04-06T10:08:48.707 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local current_state=deleted 2024-04-06T10:08:48.707 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local current_image_id 2024-04-06T10:08:48.708 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_image_id cluster1 mirror image_26 2024-04-06T10:08:48.708 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T10:08:48.708 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:08:48.708 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_26 2024-04-06T10:08:48.708 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster=cluster1 info mirror/image_26 2024-04-06T10:08:48.709 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-04-06T10:08:48.753 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:rbd: error opening image image_26: (2) No such file or directory 2024-04-06T10:08:48.759 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ current_image_id= 2024-04-06T10:08:48.759 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n '' 2024-04-06T10:08:48.759 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test deleted = deleted 2024-04-06T10:08:48.759 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T10:08:48.759 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-06T10:08:48.759 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ image=image_27 2024-04-06T10:08:48.759 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_image_present cluster1 mirror image_27 deleted 2024-04-06T10:08:48.759 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T10:08:48.759 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:08:48.759 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_27 2024-04-06T10:08:48.760 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state=deleted 2024-04-06T10:08:48.760 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image_id= 2024-04-06T10:08:48.760 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local s 2024-04-06T10:08:48.760 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n '' 2024-04-06T10:08:48.760 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_image_id cluster1 mirror image_27 2024-04-06T10:08:48.760 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T10:08:48.760 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:08:48.760 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_27 2024-04-06T10:08:48.760 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster=cluster1 info mirror/image_27 2024-04-06T10:08:48.760 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-04-06T10:08:48.798 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:rbd: error opening image image_27: (2) No such file or directory 2024-04-06T10:08:48.802 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ image_id= 2024-04-06T10:08:48.802 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-04-06T10:08:48.802 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.1 2024-04-06T10:08:48.904 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test_image_present cluster1 mirror image_27 deleted '' 2024-04-06T10:08:48.904 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T10:08:48.904 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:08:48.904 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_27 2024-04-06T10:08:48.904 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local test_state=deleted 2024-04-06T10:08:48.904 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image_id= 2024-04-06T10:08:48.904 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local current_state=deleted 2024-04-06T10:08:48.904 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local current_image_id 2024-04-06T10:08:48.905 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_image_id cluster1 mirror image_27 2024-04-06T10:08:48.905 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T10:08:48.906 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:08:48.906 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_27 2024-04-06T10:08:48.906 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster=cluster1 info mirror/image_27 2024-04-06T10:08:48.906 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-04-06T10:08:48.953 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:rbd: error opening image image_27: (2) No such file or directory 2024-04-06T10:08:48.959 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ current_image_id= 2024-04-06T10:08:48.959 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n '' 2024-04-06T10:08:48.959 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test deleted = deleted 2024-04-06T10:08:48.959 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T10:08:48.959 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-06T10:08:48.959 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ image=image_28 2024-04-06T10:08:48.959 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_image_present cluster1 mirror image_28 deleted 2024-04-06T10:08:48.959 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T10:08:48.959 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:08:48.959 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_28 2024-04-06T10:08:48.959 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state=deleted 2024-04-06T10:08:48.959 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image_id= 2024-04-06T10:08:48.960 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local s 2024-04-06T10:08:48.960 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n '' 2024-04-06T10:08:48.960 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_image_id cluster1 mirror image_28 2024-04-06T10:08:48.960 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T10:08:48.960 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:08:48.960 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_28 2024-04-06T10:08:48.960 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster=cluster1 info mirror/image_28 2024-04-06T10:08:48.960 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-04-06T10:08:48.998 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:rbd: error opening image image_28: (2) No such file or directory 2024-04-06T10:08:49.002 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ image_id= 2024-04-06T10:08:49.002 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-04-06T10:08:49.002 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.1 2024-04-06T10:08:49.104 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test_image_present cluster1 mirror image_28 deleted '' 2024-04-06T10:08:49.105 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T10:08:49.105 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:08:49.105 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_28 2024-04-06T10:08:49.105 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local test_state=deleted 2024-04-06T10:08:49.105 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image_id= 2024-04-06T10:08:49.105 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local current_state=deleted 2024-04-06T10:08:49.105 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local current_image_id 2024-04-06T10:08:49.107 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_image_id cluster1 mirror image_28 2024-04-06T10:08:49.107 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T10:08:49.107 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:08:49.107 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_28 2024-04-06T10:08:49.107 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster=cluster1 info mirror/image_28 2024-04-06T10:08:49.107 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-04-06T10:08:49.166 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:rbd: error opening image image_28: (2) No such file or directory 2024-04-06T10:08:49.171 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ current_image_id= 2024-04-06T10:08:49.171 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n '' 2024-04-06T10:08:49.171 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test deleted = deleted 2024-04-06T10:08:49.172 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T10:08:49.172 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-06T10:08:49.172 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ image=image_29 2024-04-06T10:08:49.172 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_image_present cluster1 mirror image_29 deleted 2024-04-06T10:08:49.172 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T10:08:49.172 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:08:49.172 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_29 2024-04-06T10:08:49.172 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state=deleted 2024-04-06T10:08:49.172 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image_id= 2024-04-06T10:08:49.172 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local s 2024-04-06T10:08:49.172 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n '' 2024-04-06T10:08:49.172 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_image_id cluster1 mirror image_29 2024-04-06T10:08:49.172 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T10:08:49.173 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:08:49.173 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_29 2024-04-06T10:08:49.173 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster=cluster1 info mirror/image_29 2024-04-06T10:08:49.173 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-04-06T10:08:49.227 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:rbd: error opening image image_29: (2) No such file or directory 2024-04-06T10:08:49.232 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ image_id= 2024-04-06T10:08:49.232 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-04-06T10:08:49.232 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.1 2024-04-06T10:08:49.334 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test_image_present cluster1 mirror image_29 deleted '' 2024-04-06T10:08:49.334 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T10:08:49.334 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:08:49.334 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_29 2024-04-06T10:08:49.334 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local test_state=deleted 2024-04-06T10:08:49.334 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image_id= 2024-04-06T10:08:49.334 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local current_state=deleted 2024-04-06T10:08:49.335 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local current_image_id 2024-04-06T10:08:49.335 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_image_id cluster1 mirror image_29 2024-04-06T10:08:49.335 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T10:08:49.335 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:08:49.335 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_29 2024-04-06T10:08:49.335 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster=cluster1 info mirror/image_29 2024-04-06T10:08:49.336 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-04-06T10:08:49.379 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:rbd: error opening image image_29: (2) No such file or directory 2024-04-06T10:08:49.384 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ current_image_id= 2024-04-06T10:08:49.384 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n '' 2024-04-06T10:08:49.385 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test deleted = deleted 2024-04-06T10:08:49.385 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T10:08:49.385 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-06T10:08:49.385 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ image=image_30 2024-04-06T10:08:49.385 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_image_present cluster1 mirror image_30 deleted 2024-04-06T10:08:49.385 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T10:08:49.385 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:08:49.385 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_30 2024-04-06T10:08:49.385 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state=deleted 2024-04-06T10:08:49.385 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image_id= 2024-04-06T10:08:49.385 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local s 2024-04-06T10:08:49.385 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n '' 2024-04-06T10:08:49.385 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_image_id cluster1 mirror image_30 2024-04-06T10:08:49.385 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T10:08:49.385 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:08:49.386 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_30 2024-04-06T10:08:49.386 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster=cluster1 info mirror/image_30 2024-04-06T10:08:49.386 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-04-06T10:08:49.423 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:rbd: error opening image image_30: (2) No such file or directory 2024-04-06T10:08:49.427 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ image_id= 2024-04-06T10:08:49.427 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-04-06T10:08:49.427 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.1 2024-04-06T10:08:49.529 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test_image_present cluster1 mirror image_30 deleted '' 2024-04-06T10:08:49.529 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T10:08:49.529 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:08:49.529 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_30 2024-04-06T10:08:49.529 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local test_state=deleted 2024-04-06T10:08:49.529 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image_id= 2024-04-06T10:08:49.529 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local current_state=deleted 2024-04-06T10:08:49.529 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local current_image_id 2024-04-06T10:08:49.530 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_image_id cluster1 mirror image_30 2024-04-06T10:08:49.530 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T10:08:49.530 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:08:49.530 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_30 2024-04-06T10:08:49.530 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster=cluster1 info mirror/image_30 2024-04-06T10:08:49.531 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-04-06T10:08:49.578 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:rbd: error opening image image_30: (2) No such file or directory 2024-04-06T10:08:49.583 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ current_image_id= 2024-04-06T10:08:49.583 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n '' 2024-04-06T10:08:49.584 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test deleted = deleted 2024-04-06T10:08:49.584 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T10:08:49.584 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-06T10:08:49.584 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ image=image_31 2024-04-06T10:08:49.584 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_image_present cluster1 mirror image_31 deleted 2024-04-06T10:08:49.584 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T10:08:49.584 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:08:49.584 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_31 2024-04-06T10:08:49.584 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state=deleted 2024-04-06T10:08:49.584 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image_id= 2024-04-06T10:08:49.584 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local s 2024-04-06T10:08:49.584 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n '' 2024-04-06T10:08:49.584 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_image_id cluster1 mirror image_31 2024-04-06T10:08:49.584 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T10:08:49.584 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:08:49.585 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_31 2024-04-06T10:08:49.585 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster=cluster1 info mirror/image_31 2024-04-06T10:08:49.585 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-04-06T10:08:49.623 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:rbd: error opening image image_31: (2) No such file or directory 2024-04-06T10:08:49.627 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ image_id= 2024-04-06T10:08:49.627 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-04-06T10:08:49.627 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.1 2024-04-06T10:08:49.729 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test_image_present cluster1 mirror image_31 deleted '' 2024-04-06T10:08:49.729 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T10:08:49.729 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:08:49.729 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_31 2024-04-06T10:08:49.729 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local test_state=deleted 2024-04-06T10:08:49.729 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image_id= 2024-04-06T10:08:49.729 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local current_state=deleted 2024-04-06T10:08:49.729 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local current_image_id 2024-04-06T10:08:49.730 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_image_id cluster1 mirror image_31 2024-04-06T10:08:49.730 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T10:08:49.730 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:08:49.730 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_31 2024-04-06T10:08:49.730 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster=cluster1 info mirror/image_31 2024-04-06T10:08:49.730 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-04-06T10:08:49.778 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:rbd: error opening image image_31: (2) No such file or directory 2024-04-06T10:08:49.783 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ current_image_id= 2024-04-06T10:08:49.783 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n '' 2024-04-06T10:08:49.783 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test deleted = deleted 2024-04-06T10:08:49.783 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T10:08:49.783 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-06T10:08:49.783 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ image=image_32 2024-04-06T10:08:49.783 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_image_present cluster1 mirror image_32 deleted 2024-04-06T10:08:49.784 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T10:08:49.784 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:08:49.784 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_32 2024-04-06T10:08:49.784 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state=deleted 2024-04-06T10:08:49.784 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image_id= 2024-04-06T10:08:49.784 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local s 2024-04-06T10:08:49.784 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n '' 2024-04-06T10:08:49.784 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_image_id cluster1 mirror image_32 2024-04-06T10:08:49.784 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T10:08:49.784 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:08:49.784 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_32 2024-04-06T10:08:49.784 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster=cluster1 info mirror/image_32 2024-04-06T10:08:49.784 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-04-06T10:08:49.823 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:rbd: error opening image image_32: (2) No such file or directory 2024-04-06T10:08:49.828 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ image_id= 2024-04-06T10:08:49.828 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-04-06T10:08:49.828 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.1 2024-04-06T10:08:49.929 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test_image_present cluster1 mirror image_32 deleted '' 2024-04-06T10:08:49.929 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T10:08:49.930 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:08:49.930 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_32 2024-04-06T10:08:49.930 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local test_state=deleted 2024-04-06T10:08:49.930 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image_id= 2024-04-06T10:08:49.930 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local current_state=deleted 2024-04-06T10:08:49.930 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local current_image_id 2024-04-06T10:08:49.931 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_image_id cluster1 mirror image_32 2024-04-06T10:08:49.931 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T10:08:49.931 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:08:49.931 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_32 2024-04-06T10:08:49.931 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster=cluster1 info mirror/image_32 2024-04-06T10:08:49.931 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-04-06T10:08:49.977 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:rbd: error opening image image_32: (2) No such file or directory 2024-04-06T10:08:49.982 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ current_image_id= 2024-04-06T10:08:49.982 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n '' 2024-04-06T10:08:49.983 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test deleted = deleted 2024-04-06T10:08:49.983 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T10:08:49.983 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-06T10:08:49.983 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ image=image_33 2024-04-06T10:08:49.983 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_image_present cluster1 mirror image_33 deleted 2024-04-06T10:08:49.983 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T10:08:49.983 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:08:49.983 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_33 2024-04-06T10:08:49.983 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state=deleted 2024-04-06T10:08:49.983 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image_id= 2024-04-06T10:08:49.983 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local s 2024-04-06T10:08:49.983 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n '' 2024-04-06T10:08:49.983 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_image_id cluster1 mirror image_33 2024-04-06T10:08:49.983 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T10:08:49.984 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:08:49.984 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_33 2024-04-06T10:08:49.984 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster=cluster1 info mirror/image_33 2024-04-06T10:08:49.984 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-04-06T10:08:50.030 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:rbd: error opening image image_33: (2) No such file or directory 2024-04-06T10:08:50.035 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ image_id= 2024-04-06T10:08:50.036 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-04-06T10:08:50.036 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.1 2024-04-06T10:08:50.138 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test_image_present cluster1 mirror image_33 deleted '' 2024-04-06T10:08:50.138 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T10:08:50.138 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:08:50.138 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_33 2024-04-06T10:08:50.138 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local test_state=deleted 2024-04-06T10:08:50.138 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image_id= 2024-04-06T10:08:50.138 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local current_state=deleted 2024-04-06T10:08:50.138 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local current_image_id 2024-04-06T10:08:50.138 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_image_id cluster1 mirror image_33 2024-04-06T10:08:50.138 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T10:08:50.139 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:08:50.139 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_33 2024-04-06T10:08:50.139 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster=cluster1 info mirror/image_33 2024-04-06T10:08:50.139 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-04-06T10:08:50.181 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:rbd: error opening image image_33: (2) No such file or directory 2024-04-06T10:08:50.187 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ current_image_id= 2024-04-06T10:08:50.187 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n '' 2024-04-06T10:08:50.187 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test deleted = deleted 2024-04-06T10:08:50.187 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T10:08:50.187 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-06T10:08:50.187 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ image=image_34 2024-04-06T10:08:50.187 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_image_present cluster1 mirror image_34 deleted 2024-04-06T10:08:50.187 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T10:08:50.187 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:08:50.187 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_34 2024-04-06T10:08:50.187 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state=deleted 2024-04-06T10:08:50.187 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image_id= 2024-04-06T10:08:50.188 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local s 2024-04-06T10:08:50.188 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n '' 2024-04-06T10:08:50.188 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_image_id cluster1 mirror image_34 2024-04-06T10:08:50.188 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T10:08:50.188 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:08:50.188 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_34 2024-04-06T10:08:50.188 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster=cluster1 info mirror/image_34 2024-04-06T10:08:50.188 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-04-06T10:08:50.225 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:rbd: error opening image image_34: (2) No such file or directory 2024-04-06T10:08:50.229 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ image_id= 2024-04-06T10:08:50.229 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-04-06T10:08:50.229 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.1 2024-04-06T10:08:50.331 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test_image_present cluster1 mirror image_34 deleted '' 2024-04-06T10:08:50.331 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T10:08:50.331 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:08:50.331 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_34 2024-04-06T10:08:50.332 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local test_state=deleted 2024-04-06T10:08:50.332 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image_id= 2024-04-06T10:08:50.332 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local current_state=deleted 2024-04-06T10:08:50.332 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local current_image_id 2024-04-06T10:08:50.332 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_image_id cluster1 mirror image_34 2024-04-06T10:08:50.332 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T10:08:50.332 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:08:50.332 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_34 2024-04-06T10:08:50.333 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster=cluster1 info mirror/image_34 2024-04-06T10:08:50.333 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-04-06T10:08:50.376 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:rbd: error opening image image_34: (2) No such file or directory 2024-04-06T10:08:50.381 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ current_image_id= 2024-04-06T10:08:50.382 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n '' 2024-04-06T10:08:50.382 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test deleted = deleted 2024-04-06T10:08:50.382 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T10:08:50.382 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-06T10:08:50.382 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ image=image_35 2024-04-06T10:08:50.382 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_image_present cluster1 mirror image_35 deleted 2024-04-06T10:08:50.382 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T10:08:50.382 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:08:50.382 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_35 2024-04-06T10:08:50.382 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state=deleted 2024-04-06T10:08:50.382 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image_id= 2024-04-06T10:08:50.382 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local s 2024-04-06T10:08:50.382 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n '' 2024-04-06T10:08:50.382 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_image_id cluster1 mirror image_35 2024-04-06T10:08:50.382 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T10:08:50.383 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:08:50.383 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_35 2024-04-06T10:08:50.383 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster=cluster1 info mirror/image_35 2024-04-06T10:08:50.383 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-04-06T10:08:50.421 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:rbd: error opening image image_35: (2) No such file or directory 2024-04-06T10:08:50.425 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ image_id= 2024-04-06T10:08:50.425 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-04-06T10:08:50.425 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.1 2024-04-06T10:08:50.527 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test_image_present cluster1 mirror image_35 deleted '' 2024-04-06T10:08:50.527 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T10:08:50.527 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:08:50.527 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_35 2024-04-06T10:08:50.527 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local test_state=deleted 2024-04-06T10:08:50.527 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image_id= 2024-04-06T10:08:50.527 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local current_state=deleted 2024-04-06T10:08:50.527 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local current_image_id 2024-04-06T10:08:50.528 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_image_id cluster1 mirror image_35 2024-04-06T10:08:50.528 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T10:08:50.528 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:08:50.528 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_35 2024-04-06T10:08:50.528 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster=cluster1 info mirror/image_35 2024-04-06T10:08:50.529 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-04-06T10:08:50.575 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:rbd: error opening image image_35: (2) No such file or directory 2024-04-06T10:08:50.581 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ current_image_id= 2024-04-06T10:08:50.581 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n '' 2024-04-06T10:08:50.581 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test deleted = deleted 2024-04-06T10:08:50.581 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T10:08:50.581 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-06T10:08:50.581 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ image=image_36 2024-04-06T10:08:50.581 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_image_present cluster1 mirror image_36 deleted 2024-04-06T10:08:50.581 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T10:08:50.582 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:08:50.582 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_36 2024-04-06T10:08:50.582 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state=deleted 2024-04-06T10:08:50.582 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image_id= 2024-04-06T10:08:50.582 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local s 2024-04-06T10:08:50.582 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n '' 2024-04-06T10:08:50.582 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_image_id cluster1 mirror image_36 2024-04-06T10:08:50.582 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T10:08:50.582 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:08:50.582 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_36 2024-04-06T10:08:50.582 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster=cluster1 info mirror/image_36 2024-04-06T10:08:50.582 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-04-06T10:08:50.621 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:rbd: error opening image image_36: (2) No such file or directory 2024-04-06T10:08:50.625 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ image_id= 2024-04-06T10:08:50.625 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-04-06T10:08:50.625 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.1 2024-04-06T10:08:50.727 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test_image_present cluster1 mirror image_36 deleted '' 2024-04-06T10:08:50.727 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T10:08:50.727 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:08:50.727 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_36 2024-04-06T10:08:50.727 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local test_state=deleted 2024-04-06T10:08:50.727 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image_id= 2024-04-06T10:08:50.727 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local current_state=deleted 2024-04-06T10:08:50.727 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local current_image_id 2024-04-06T10:08:50.728 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_image_id cluster1 mirror image_36 2024-04-06T10:08:50.728 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T10:08:50.728 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:08:50.728 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_36 2024-04-06T10:08:50.729 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster=cluster1 info mirror/image_36 2024-04-06T10:08:50.729 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-04-06T10:08:50.773 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:rbd: error opening image image_36: (2) No such file or directory 2024-04-06T10:08:50.779 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ current_image_id= 2024-04-06T10:08:50.779 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n '' 2024-04-06T10:08:50.780 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test deleted = deleted 2024-04-06T10:08:50.780 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T10:08:50.780 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-06T10:08:50.780 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ image=image_37 2024-04-06T10:08:50.780 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_image_present cluster1 mirror image_37 deleted 2024-04-06T10:08:50.780 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T10:08:50.780 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:08:50.780 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_37 2024-04-06T10:08:50.780 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state=deleted 2024-04-06T10:08:50.780 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image_id= 2024-04-06T10:08:50.780 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local s 2024-04-06T10:08:50.780 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n '' 2024-04-06T10:08:50.780 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_image_id cluster1 mirror image_37 2024-04-06T10:08:50.781 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T10:08:50.781 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:08:50.781 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_37 2024-04-06T10:08:50.781 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster=cluster1 info mirror/image_37 2024-04-06T10:08:50.781 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-04-06T10:08:50.821 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:rbd: error opening image image_37: (2) No such file or directory 2024-04-06T10:08:50.825 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ image_id= 2024-04-06T10:08:50.825 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-04-06T10:08:50.825 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.1 2024-04-06T10:08:50.926 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test_image_present cluster1 mirror image_37 deleted '' 2024-04-06T10:08:50.927 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T10:08:50.927 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:08:50.927 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_37 2024-04-06T10:08:50.927 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local test_state=deleted 2024-04-06T10:08:50.927 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image_id= 2024-04-06T10:08:50.927 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local current_state=deleted 2024-04-06T10:08:50.927 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local current_image_id 2024-04-06T10:08:50.928 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_image_id cluster1 mirror image_37 2024-04-06T10:08:50.928 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T10:08:50.928 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:08:50.928 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_37 2024-04-06T10:08:50.928 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster=cluster1 info mirror/image_37 2024-04-06T10:08:50.928 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-04-06T10:08:50.975 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:rbd: error opening image image_37: (2) No such file or directory 2024-04-06T10:08:50.981 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ current_image_id= 2024-04-06T10:08:50.981 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n '' 2024-04-06T10:08:50.981 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test deleted = deleted 2024-04-06T10:08:50.981 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T10:08:50.981 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-06T10:08:50.981 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ image=image_38 2024-04-06T10:08:50.982 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_image_present cluster1 mirror image_38 deleted 2024-04-06T10:08:50.982 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T10:08:50.982 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:08:50.982 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_38 2024-04-06T10:08:50.982 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state=deleted 2024-04-06T10:08:50.982 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image_id= 2024-04-06T10:08:50.982 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local s 2024-04-06T10:08:50.982 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n '' 2024-04-06T10:08:50.982 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_image_id cluster1 mirror image_38 2024-04-06T10:08:50.982 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T10:08:50.982 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:08:50.982 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_38 2024-04-06T10:08:50.982 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster=cluster1 info mirror/image_38 2024-04-06T10:08:50.982 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-04-06T10:08:51.019 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:rbd: error opening image image_38: (2) No such file or directory 2024-04-06T10:08:51.023 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ image_id= 2024-04-06T10:08:51.023 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-04-06T10:08:51.024 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.1 2024-04-06T10:08:51.125 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test_image_present cluster1 mirror image_38 deleted '' 2024-04-06T10:08:51.125 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T10:08:51.125 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:08:51.125 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_38 2024-04-06T10:08:51.125 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local test_state=deleted 2024-04-06T10:08:51.125 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image_id= 2024-04-06T10:08:51.125 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local current_state=deleted 2024-04-06T10:08:51.126 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local current_image_id 2024-04-06T10:08:51.126 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_image_id cluster1 mirror image_38 2024-04-06T10:08:51.127 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T10:08:51.127 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:08:51.127 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_38 2024-04-06T10:08:51.127 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster=cluster1 info mirror/image_38 2024-04-06T10:08:51.127 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-04-06T10:08:51.171 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:rbd: error opening image image_38: (2) No such file or directory 2024-04-06T10:08:51.176 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ current_image_id= 2024-04-06T10:08:51.176 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n '' 2024-04-06T10:08:51.176 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test deleted = deleted 2024-04-06T10:08:51.176 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T10:08:51.176 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-06T10:08:51.176 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ image=image_39 2024-04-06T10:08:51.176 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_image_present cluster1 mirror image_39 deleted 2024-04-06T10:08:51.177 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T10:08:51.177 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:08:51.177 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_39 2024-04-06T10:08:51.177 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state=deleted 2024-04-06T10:08:51.177 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image_id= 2024-04-06T10:08:51.177 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local s 2024-04-06T10:08:51.177 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n '' 2024-04-06T10:08:51.177 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_image_id cluster1 mirror image_39 2024-04-06T10:08:51.177 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T10:08:51.177 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:08:51.177 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_39 2024-04-06T10:08:51.177 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster=cluster1 info mirror/image_39 2024-04-06T10:08:51.177 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-04-06T10:08:51.216 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:rbd: error opening image image_39: (2) No such file or directory 2024-04-06T10:08:51.220 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ image_id= 2024-04-06T10:08:51.221 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-04-06T10:08:51.221 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.1 2024-04-06T10:08:51.322 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test_image_present cluster1 mirror image_39 deleted '' 2024-04-06T10:08:51.323 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T10:08:51.323 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:08:51.323 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_39 2024-04-06T10:08:51.323 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local test_state=deleted 2024-04-06T10:08:51.323 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image_id= 2024-04-06T10:08:51.323 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local current_state=deleted 2024-04-06T10:08:51.323 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local current_image_id 2024-04-06T10:08:51.323 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_image_id cluster1 mirror image_39 2024-04-06T10:08:51.323 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T10:08:51.323 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:08:51.323 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_39 2024-04-06T10:08:51.324 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster=cluster1 info mirror/image_39 2024-04-06T10:08:51.324 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-04-06T10:08:51.369 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:rbd: error opening image image_39: (2) No such file or directory 2024-04-06T10:08:51.374 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ current_image_id= 2024-04-06T10:08:51.374 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n '' 2024-04-06T10:08:51.374 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test deleted = deleted 2024-04-06T10:08:51.374 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T10:08:51.375 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-06T10:08:51.375 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ image=image_40 2024-04-06T10:08:51.375 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_image_present cluster1 mirror image_40 deleted 2024-04-06T10:08:51.375 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T10:08:51.375 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:08:51.375 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_40 2024-04-06T10:08:51.375 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state=deleted 2024-04-06T10:08:51.375 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image_id= 2024-04-06T10:08:51.375 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local s 2024-04-06T10:08:51.375 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n '' 2024-04-06T10:08:51.375 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_image_id cluster1 mirror image_40 2024-04-06T10:08:51.375 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T10:08:51.375 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:08:51.375 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_40 2024-04-06T10:08:51.376 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster=cluster1 info mirror/image_40 2024-04-06T10:08:51.376 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-04-06T10:08:51.414 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:rbd: error opening image image_40: (2) No such file or directory 2024-04-06T10:08:51.418 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ image_id= 2024-04-06T10:08:51.418 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-04-06T10:08:51.419 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.1 2024-04-06T10:08:51.520 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test_image_present cluster1 mirror image_40 deleted '' 2024-04-06T10:08:51.520 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T10:08:51.520 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:08:51.520 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_40 2024-04-06T10:08:51.520 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local test_state=deleted 2024-04-06T10:08:51.520 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image_id= 2024-04-06T10:08:51.520 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local current_state=deleted 2024-04-06T10:08:51.520 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local current_image_id 2024-04-06T10:08:51.521 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_image_id cluster1 mirror image_40 2024-04-06T10:08:51.521 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T10:08:51.521 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:08:51.521 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_40 2024-04-06T10:08:51.521 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster=cluster1 info mirror/image_40 2024-04-06T10:08:51.521 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-04-06T10:08:51.568 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:rbd: error opening image image_40: (2) No such file or directory 2024-04-06T10:08:51.574 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ current_image_id= 2024-04-06T10:08:51.574 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n '' 2024-04-06T10:08:51.574 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test deleted = deleted 2024-04-06T10:08:51.574 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T10:08:51.574 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-06T10:08:51.574 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ image=image_41 2024-04-06T10:08:51.574 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_image_present cluster1 mirror image_41 deleted 2024-04-06T10:08:51.574 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T10:08:51.574 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:08:51.574 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_41 2024-04-06T10:08:51.574 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state=deleted 2024-04-06T10:08:51.575 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image_id= 2024-04-06T10:08:51.575 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local s 2024-04-06T10:08:51.575 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n '' 2024-04-06T10:08:51.575 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_image_id cluster1 mirror image_41 2024-04-06T10:08:51.575 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T10:08:51.575 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:08:51.575 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_41 2024-04-06T10:08:51.575 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster=cluster1 info mirror/image_41 2024-04-06T10:08:51.575 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-04-06T10:08:51.615 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:rbd: error opening image image_41: (2) No such file or directory 2024-04-06T10:08:51.619 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ image_id= 2024-04-06T10:08:51.619 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-04-06T10:08:51.619 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.1 2024-04-06T10:08:51.721 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test_image_present cluster1 mirror image_41 deleted '' 2024-04-06T10:08:51.721 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T10:08:51.721 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:08:51.721 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_41 2024-04-06T10:08:51.721 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local test_state=deleted 2024-04-06T10:08:51.721 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image_id= 2024-04-06T10:08:51.722 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local current_state=deleted 2024-04-06T10:08:51.722 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local current_image_id 2024-04-06T10:08:51.722 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_image_id cluster1 mirror image_41 2024-04-06T10:08:51.723 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T10:08:51.723 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:08:51.723 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_41 2024-04-06T10:08:51.723 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster=cluster1 info mirror/image_41 2024-04-06T10:08:51.723 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-04-06T10:08:51.767 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:rbd: error opening image image_41: (2) No such file or directory 2024-04-06T10:08:51.773 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ current_image_id= 2024-04-06T10:08:51.773 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n '' 2024-04-06T10:08:51.773 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test deleted = deleted 2024-04-06T10:08:51.773 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T10:08:51.773 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-06T10:08:51.773 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ image=image_42 2024-04-06T10:08:51.773 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_image_present cluster1 mirror image_42 deleted 2024-04-06T10:08:51.773 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T10:08:51.773 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:08:51.773 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_42 2024-04-06T10:08:51.774 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state=deleted 2024-04-06T10:08:51.774 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image_id= 2024-04-06T10:08:51.774 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local s 2024-04-06T10:08:51.774 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n '' 2024-04-06T10:08:51.774 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_image_id cluster1 mirror image_42 2024-04-06T10:08:51.774 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T10:08:51.774 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:08:51.774 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_42 2024-04-06T10:08:51.774 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster=cluster1 info mirror/image_42 2024-04-06T10:08:51.774 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-04-06T10:08:51.813 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:rbd: error opening image image_42: (2) No such file or directory 2024-04-06T10:08:51.818 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ image_id= 2024-04-06T10:08:51.818 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-04-06T10:08:51.818 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.1 2024-04-06T10:08:51.921 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test_image_present cluster1 mirror image_42 deleted '' 2024-04-06T10:08:51.922 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T10:08:51.922 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:08:51.922 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_42 2024-04-06T10:08:51.922 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local test_state=deleted 2024-04-06T10:08:51.922 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image_id= 2024-04-06T10:08:51.922 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local current_state=deleted 2024-04-06T10:08:51.922 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local current_image_id 2024-04-06T10:08:51.922 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_image_id cluster1 mirror image_42 2024-04-06T10:08:51.922 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T10:08:51.922 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:08:51.922 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_42 2024-04-06T10:08:51.922 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster=cluster1 info mirror/image_42 2024-04-06T10:08:51.923 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-04-06T10:08:51.966 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:rbd: error opening image image_42: (2) No such file or directory 2024-04-06T10:08:51.972 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ current_image_id= 2024-04-06T10:08:51.972 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n '' 2024-04-06T10:08:51.972 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test deleted = deleted 2024-04-06T10:08:51.972 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T10:08:51.972 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-06T10:08:51.972 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ image=image_43 2024-04-06T10:08:51.972 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_image_present cluster1 mirror image_43 deleted 2024-04-06T10:08:51.972 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T10:08:51.973 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:08:51.973 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_43 2024-04-06T10:08:51.973 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state=deleted 2024-04-06T10:08:51.973 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image_id= 2024-04-06T10:08:51.973 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local s 2024-04-06T10:08:51.973 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n '' 2024-04-06T10:08:51.973 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_image_id cluster1 mirror image_43 2024-04-06T10:08:51.973 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T10:08:51.973 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:08:51.973 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_43 2024-04-06T10:08:51.973 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster=cluster1 info mirror/image_43 2024-04-06T10:08:51.973 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-04-06T10:08:52.012 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:rbd: error opening image image_43: (2) No such file or directory 2024-04-06T10:08:52.016 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ image_id= 2024-04-06T10:08:52.016 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-04-06T10:08:52.016 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.1 2024-04-06T10:08:52.118 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test_image_present cluster1 mirror image_43 deleted '' 2024-04-06T10:08:52.118 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T10:08:52.118 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:08:52.118 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_43 2024-04-06T10:08:52.118 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local test_state=deleted 2024-04-06T10:08:52.118 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image_id= 2024-04-06T10:08:52.118 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local current_state=deleted 2024-04-06T10:08:52.118 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local current_image_id 2024-04-06T10:08:52.119 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_image_id cluster1 mirror image_43 2024-04-06T10:08:52.119 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T10:08:52.119 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:08:52.119 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_43 2024-04-06T10:08:52.119 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster=cluster1 info mirror/image_43 2024-04-06T10:08:52.120 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-04-06T10:08:52.167 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:rbd: error opening image image_43: (2) No such file or directory 2024-04-06T10:08:52.172 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ current_image_id= 2024-04-06T10:08:52.172 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n '' 2024-04-06T10:08:52.173 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test deleted = deleted 2024-04-06T10:08:52.173 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T10:08:52.173 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-06T10:08:52.173 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ image=image_44 2024-04-06T10:08:52.173 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_image_present cluster1 mirror image_44 deleted 2024-04-06T10:08:52.173 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T10:08:52.173 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:08:52.173 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_44 2024-04-06T10:08:52.173 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state=deleted 2024-04-06T10:08:52.173 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image_id= 2024-04-06T10:08:52.173 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local s 2024-04-06T10:08:52.173 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n '' 2024-04-06T10:08:52.173 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_image_id cluster1 mirror image_44 2024-04-06T10:08:52.174 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T10:08:52.174 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:08:52.174 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_44 2024-04-06T10:08:52.174 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster=cluster1 info mirror/image_44 2024-04-06T10:08:52.174 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-04-06T10:08:52.212 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:rbd: error opening image image_44: (2) No such file or directory 2024-04-06T10:08:52.216 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ image_id= 2024-04-06T10:08:52.216 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-04-06T10:08:52.216 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.1 2024-04-06T10:08:52.318 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test_image_present cluster1 mirror image_44 deleted '' 2024-04-06T10:08:52.318 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T10:08:52.318 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:08:52.319 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_44 2024-04-06T10:08:52.319 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local test_state=deleted 2024-04-06T10:08:52.319 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image_id= 2024-04-06T10:08:52.319 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local current_state=deleted 2024-04-06T10:08:52.319 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local current_image_id 2024-04-06T10:08:52.319 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_image_id cluster1 mirror image_44 2024-04-06T10:08:52.319 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T10:08:52.319 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:08:52.319 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_44 2024-04-06T10:08:52.320 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster=cluster1 info mirror/image_44 2024-04-06T10:08:52.320 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-04-06T10:08:52.364 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:rbd: error opening image image_44: (2) No such file or directory 2024-04-06T10:08:52.369 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ current_image_id= 2024-04-06T10:08:52.370 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n '' 2024-04-06T10:08:52.370 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test deleted = deleted 2024-04-06T10:08:52.370 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T10:08:52.370 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-06T10:08:52.370 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ image=image_45 2024-04-06T10:08:52.370 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_image_present cluster1 mirror image_45 deleted 2024-04-06T10:08:52.370 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T10:08:52.370 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:08:52.370 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_45 2024-04-06T10:08:52.370 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state=deleted 2024-04-06T10:08:52.370 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image_id= 2024-04-06T10:08:52.370 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local s 2024-04-06T10:08:52.371 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n '' 2024-04-06T10:08:52.371 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_image_id cluster1 mirror image_45 2024-04-06T10:08:52.371 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T10:08:52.371 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:08:52.371 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_45 2024-04-06T10:08:52.371 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster=cluster1 info mirror/image_45 2024-04-06T10:08:52.371 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-04-06T10:08:52.408 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:rbd: error opening image image_45: (2) No such file or directory 2024-04-06T10:08:52.413 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ image_id= 2024-04-06T10:08:52.413 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-04-06T10:08:52.413 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.1 2024-04-06T10:08:52.514 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test_image_present cluster1 mirror image_45 deleted '' 2024-04-06T10:08:52.514 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T10:08:52.514 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:08:52.514 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_45 2024-04-06T10:08:52.514 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local test_state=deleted 2024-04-06T10:08:52.514 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image_id= 2024-04-06T10:08:52.514 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local current_state=deleted 2024-04-06T10:08:52.514 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local current_image_id 2024-04-06T10:08:52.515 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_image_id cluster1 mirror image_45 2024-04-06T10:08:52.515 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T10:08:52.515 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:08:52.515 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_45 2024-04-06T10:08:52.515 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster=cluster1 info mirror/image_45 2024-04-06T10:08:52.516 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-04-06T10:08:52.562 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:rbd: error opening image image_45: (2) No such file or directory 2024-04-06T10:08:52.569 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ current_image_id= 2024-04-06T10:08:52.570 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n '' 2024-04-06T10:08:52.570 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test deleted = deleted 2024-04-06T10:08:52.570 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T10:08:52.570 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-06T10:08:52.570 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ image=image_46 2024-04-06T10:08:52.570 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_image_present cluster1 mirror image_46 deleted 2024-04-06T10:08:52.571 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T10:08:52.571 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:08:52.571 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_46 2024-04-06T10:08:52.571 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state=deleted 2024-04-06T10:08:52.571 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image_id= 2024-04-06T10:08:52.572 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local s 2024-04-06T10:08:52.572 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n '' 2024-04-06T10:08:52.572 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_image_id cluster1 mirror image_46 2024-04-06T10:08:52.572 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T10:08:52.572 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:08:52.572 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_46 2024-04-06T10:08:52.572 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster=cluster1 info mirror/image_46 2024-04-06T10:08:52.573 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-04-06T10:08:52.608 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:rbd: error opening image image_46: (2) No such file or directory 2024-04-06T10:08:52.612 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ image_id= 2024-04-06T10:08:52.613 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-04-06T10:08:52.613 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.1 2024-04-06T10:08:52.714 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test_image_present cluster1 mirror image_46 deleted '' 2024-04-06T10:08:52.714 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T10:08:52.714 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:08:52.714 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_46 2024-04-06T10:08:52.714 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local test_state=deleted 2024-04-06T10:08:52.714 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image_id= 2024-04-06T10:08:52.714 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local current_state=deleted 2024-04-06T10:08:52.714 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local current_image_id 2024-04-06T10:08:52.715 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_image_id cluster1 mirror image_46 2024-04-06T10:08:52.715 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T10:08:52.715 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:08:52.715 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_46 2024-04-06T10:08:52.715 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster=cluster1 info mirror/image_46 2024-04-06T10:08:52.715 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-04-06T10:08:52.761 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:rbd: error opening image image_46: (2) No such file or directory 2024-04-06T10:08:52.766 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ current_image_id= 2024-04-06T10:08:52.767 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n '' 2024-04-06T10:08:52.767 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test deleted = deleted 2024-04-06T10:08:52.767 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T10:08:52.767 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-06T10:08:52.767 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ image=image_47 2024-04-06T10:08:52.767 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_image_present cluster1 mirror image_47 deleted 2024-04-06T10:08:52.767 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T10:08:52.767 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:08:52.767 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_47 2024-04-06T10:08:52.767 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state=deleted 2024-04-06T10:08:52.767 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image_id= 2024-04-06T10:08:52.767 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local s 2024-04-06T10:08:52.767 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n '' 2024-04-06T10:08:52.767 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_image_id cluster1 mirror image_47 2024-04-06T10:08:52.767 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T10:08:52.768 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:08:52.768 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_47 2024-04-06T10:08:52.768 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster=cluster1 info mirror/image_47 2024-04-06T10:08:52.768 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-04-06T10:08:52.807 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:rbd: error opening image image_47: (2) No such file or directory 2024-04-06T10:08:52.812 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ image_id= 2024-04-06T10:08:52.812 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-04-06T10:08:52.812 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.1 2024-04-06T10:08:52.913 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test_image_present cluster1 mirror image_47 deleted '' 2024-04-06T10:08:52.914 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T10:08:52.914 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:08:52.914 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_47 2024-04-06T10:08:52.914 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local test_state=deleted 2024-04-06T10:08:52.914 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image_id= 2024-04-06T10:08:52.914 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local current_state=deleted 2024-04-06T10:08:52.914 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local current_image_id 2024-04-06T10:08:52.915 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_image_id cluster1 mirror image_47 2024-04-06T10:08:52.915 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T10:08:52.915 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:08:52.915 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_47 2024-04-06T10:08:52.915 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster=cluster1 info mirror/image_47 2024-04-06T10:08:52.915 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-04-06T10:08:52.961 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:rbd: error opening image image_47: (2) No such file or directory 2024-04-06T10:08:52.966 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ current_image_id= 2024-04-06T10:08:52.966 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n '' 2024-04-06T10:08:52.966 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test deleted = deleted 2024-04-06T10:08:52.966 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T10:08:52.966 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-06T10:08:52.967 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ image=image_48 2024-04-06T10:08:52.967 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_image_present cluster1 mirror image_48 deleted 2024-04-06T10:08:52.967 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T10:08:52.967 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:08:52.967 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_48 2024-04-06T10:08:52.967 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state=deleted 2024-04-06T10:08:52.967 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image_id= 2024-04-06T10:08:52.967 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local s 2024-04-06T10:08:52.967 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n '' 2024-04-06T10:08:52.967 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_image_id cluster1 mirror image_48 2024-04-06T10:08:52.967 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T10:08:52.967 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:08:52.967 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_48 2024-04-06T10:08:52.967 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster=cluster1 info mirror/image_48 2024-04-06T10:08:52.967 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-04-06T10:08:53.005 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:rbd: error opening image image_48: (2) No such file or directory 2024-04-06T10:08:53.010 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ image_id= 2024-04-06T10:08:53.010 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-04-06T10:08:53.010 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.1 2024-04-06T10:08:53.112 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test_image_present cluster1 mirror image_48 deleted '' 2024-04-06T10:08:53.112 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T10:08:53.112 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:08:53.113 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_48 2024-04-06T10:08:53.113 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local test_state=deleted 2024-04-06T10:08:53.113 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image_id= 2024-04-06T10:08:53.113 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local current_state=deleted 2024-04-06T10:08:53.113 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local current_image_id 2024-04-06T10:08:53.113 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_image_id cluster1 mirror image_48 2024-04-06T10:08:53.113 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T10:08:53.113 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:08:53.113 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_48 2024-04-06T10:08:53.114 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster=cluster1 info mirror/image_48 2024-04-06T10:08:53.114 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-04-06T10:08:53.160 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:rbd: error opening image image_48: (2) No such file or directory 2024-04-06T10:08:53.166 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ current_image_id= 2024-04-06T10:08:53.166 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n '' 2024-04-06T10:08:53.166 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test deleted = deleted 2024-04-06T10:08:53.166 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T10:08:53.166 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-06T10:08:53.166 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ image=image_49 2024-04-06T10:08:53.166 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_image_present cluster1 mirror image_49 deleted 2024-04-06T10:08:53.167 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T10:08:53.167 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:08:53.167 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_49 2024-04-06T10:08:53.167 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state=deleted 2024-04-06T10:08:53.167 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image_id= 2024-04-06T10:08:53.167 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local s 2024-04-06T10:08:53.167 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n '' 2024-04-06T10:08:53.167 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_image_id cluster1 mirror image_49 2024-04-06T10:08:53.167 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T10:08:53.167 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:08:53.167 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_49 2024-04-06T10:08:53.167 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster=cluster1 info mirror/image_49 2024-04-06T10:08:53.167 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-04-06T10:08:53.204 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:rbd: error opening image image_49: (2) No such file or directory 2024-04-06T10:08:53.208 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ image_id= 2024-04-06T10:08:53.208 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-04-06T10:08:53.208 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.1 2024-04-06T10:08:53.309 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test_image_present cluster1 mirror image_49 deleted '' 2024-04-06T10:08:53.309 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T10:08:53.309 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:08:53.310 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_49 2024-04-06T10:08:53.310 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local test_state=deleted 2024-04-06T10:08:53.310 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image_id= 2024-04-06T10:08:53.310 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local current_state=deleted 2024-04-06T10:08:53.310 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local current_image_id 2024-04-06T10:08:53.310 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_image_id cluster1 mirror image_49 2024-04-06T10:08:53.311 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T10:08:53.311 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:08:53.311 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_49 2024-04-06T10:08:53.311 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster=cluster1 info mirror/image_49 2024-04-06T10:08:53.311 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-04-06T10:08:53.359 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:rbd: error opening image image_49: (2) No such file or directory 2024-04-06T10:08:53.364 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ current_image_id= 2024-04-06T10:08:53.365 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n '' 2024-04-06T10:08:53.365 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test deleted = deleted 2024-04-06T10:08:53.365 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T10:08:53.365 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-06T10:08:53.365 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ image=image_50 2024-04-06T10:08:53.365 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_image_present cluster1 mirror image_50 deleted 2024-04-06T10:08:53.365 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T10:08:53.365 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:08:53.365 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_50 2024-04-06T10:08:53.365 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state=deleted 2024-04-06T10:08:53.365 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image_id= 2024-04-06T10:08:53.365 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local s 2024-04-06T10:08:53.365 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n '' 2024-04-06T10:08:53.365 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_image_id cluster1 mirror image_50 2024-04-06T10:08:53.365 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T10:08:53.366 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:08:53.366 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_50 2024-04-06T10:08:53.366 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster=cluster1 info mirror/image_50 2024-04-06T10:08:53.366 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-04-06T10:08:53.402 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:rbd: error opening image image_50: (2) No such file or directory 2024-04-06T10:08:53.407 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ image_id= 2024-04-06T10:08:53.407 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-04-06T10:08:53.407 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.1 2024-04-06T10:08:53.508 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test_image_present cluster1 mirror image_50 deleted '' 2024-04-06T10:08:53.508 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T10:08:53.508 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:08:53.509 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_50 2024-04-06T10:08:53.509 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local test_state=deleted 2024-04-06T10:08:53.509 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image_id= 2024-04-06T10:08:53.509 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local current_state=deleted 2024-04-06T10:08:53.509 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local current_image_id 2024-04-06T10:08:53.511 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_image_id cluster1 mirror image_50 2024-04-06T10:08:53.511 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T10:08:53.511 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:08:53.511 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_50 2024-04-06T10:08:53.512 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster=cluster1 info mirror/image_50 2024-04-06T10:08:53.512 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-04-06T10:08:53.556 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:rbd: error opening image image_50: (2) No such file or directory 2024-04-06T10:08:53.562 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ current_image_id= 2024-04-06T10:08:53.563 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n '' 2024-04-06T10:08:53.563 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test deleted = deleted 2024-04-06T10:08:53.563 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T10:08:53.563 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ testlog 'TEST: delete images during bootstrap' 2024-04-06T10:08:53.563 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ tee -a /tmp/tmp.BhW6xKmROU/rbd-mirror.test.log 2024-04-06T10:08:53.563 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ date '+%F %T' 2024-04-06T10:08:53.563 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ echo 2024-04-06 10:08:53 TEST: delete images during bootstrap 2024-04-06T10:08:53.563 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:2024-04-06 10:08:53 TEST: delete images during bootstrap 2024-04-06T10:08:53.563 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_pool_mirror_mode cluster1 mirror image 2024-04-06T10:08:53.563 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T10:08:53.563 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:08:53.563 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local mode=image 2024-04-06T10:08:53.563 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster=cluster1 mirror pool enable mirror image 2024-04-06T10:08:53.600 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:note: changing mirroring mode from pool to image 2024-04-06T10:08:53.616 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_pool_mirror_mode cluster2 mirror image 2024-04-06T10:08:53.617 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-04-06T10:08:53.617 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:08:53.617 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local mode=image 2024-04-06T10:08:53.617 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster=cluster2 mirror pool enable mirror image 2024-04-06T10:08:53.650 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:note: changing mirroring mode from pool to image 2024-04-06T10:08:53.660 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ start_mirror cluster1 2024-04-06T10:08:53.661 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T10:08:53.661 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-04-06T10:08:53.661 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1 cluster instance 2024-04-06T10:08:53.661 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1 2024-04-06T10:08:53.661 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-04-06T10:08:53.661 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-04-06T10:08:53.661 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-04-06T10:08:53.661 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=cluster1 2024-04-06T10:08:53.661 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' cluster1 = cluster1 ']' 2024-04-06T10:08:53.661 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-04-06T10:08:53.661 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-04-06T10:08:53.661 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-04-06T10:08:53.661 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-04-06T10:08:53.661 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-04-06T10:08:53.662 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 1 2024-04-06T10:08:53.662 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 2024-04-06T10:08:53.662 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ image=test 2024-04-06T10:08:53.662 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 1 10 2024-04-06T10:08:53.663 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in `seq 1 10` 2024-04-06T10:08:53.663 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ image=image_1 2024-04-06T10:08:53.663 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ create_image cluster2 mirror image_1 512M 2024-04-06T10:08:53.663 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-04-06T10:08:53.663 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T10:08:53.663 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:08:53.663 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T10:08:53.663 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_1 2024-04-06T10:08:53.663 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T10:08:53.663 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local size=128 2024-04-06T10:08:53.663 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n 512M ']' 2024-04-06T10:08:53.663 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ size=512M 2024-04-06T10:08:53.663 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T10:08:53.664 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster2 create --size 512M --image-feature layering,exclusive-lock,journaling mirror/image_1 2024-04-06T10:08:53.721 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ enable_mirror cluster2 mirror image_1 2024-04-06T10:08:53.721 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-04-06T10:08:53.721 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:08:53.721 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_1 2024-04-06T10:08:53.721 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local mode=journal 2024-04-06T10:08:53.721 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_1 journal 2024-04-06T10:08:53.770 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Mirroring enabled 2024-04-06T10:08:53.776 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster=cluster2 info mirror/image_1 2024-04-06T10:08:53.814 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:rbd image 'image_1': 2024-04-06T10:08:53.815 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: size 512 MiB in 128 objects 2024-04-06T10:08:53.815 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: order 22 (4 MiB objects) 2024-04-06T10:08:53.815 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: snapshot_count: 0 2024-04-06T10:08:53.815 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: id: 46fdcc8688b9 2024-04-06T10:08:53.815 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: block_name_prefix: rbd_data.46fdcc8688b9 2024-04-06T10:08:53.815 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: format: 2 2024-04-06T10:08:53.815 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: features: layering, exclusive-lock, journaling 2024-04-06T10:08:53.815 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: op_features: 2024-04-06T10:08:53.815 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: flags: 2024-04-06T10:08:53.815 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: create_timestamp: Sat Apr 6 10:08:53 2024 2024-04-06T10:08:53.815 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: access_timestamp: Sat Apr 6 10:08:53 2024 2024-04-06T10:08:53.815 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: modify_timestamp: Sat Apr 6 10:08:53 2024 2024-04-06T10:08:53.816 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: journal: 46fdcc8688b9 2024-04-06T10:08:53.816 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: mirroring state: enabled 2024-04-06T10:08:53.816 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: mirroring mode: journal 2024-04-06T10:08:53.816 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: mirroring global id: 7ce9e214-e534-4489-8a7c-ff96b51d92c7 2024-04-06T10:08:53.816 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: mirroring primary: true 2024-04-06T10:08:53.819 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ stress_write_image cluster2 mirror image_1 2024-04-06T10:08:53.819 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-04-06T10:08:53.819 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:08:53.819 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_1 2024-04-06T10:08:53.819 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-04-06T10:08:53.820 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local duration=11 2024-04-06T10:08:53.821 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set +e 2024-04-06T10:08:53.821 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ timeout 11s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_1 --debug-rbd=20 --debug-journaler=20 2024-04-06T10:08:57.051 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:100 IOs, 44767232 bytes 2024-04-06T10:08:58.777 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:200 IOs, 87846912 bytes 2024-04-06T10:09:01.231 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:300 IOs, 124791808 bytes 2024-04-06T10:09:04.341 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:400 IOs, 204827648 bytes 2024-04-06T10:09:04.829 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ error_code=124 2024-04-06T10:09:04.829 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set -e 2024-04-06T10:09:04.829 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 124 -eq 124 ']' 2024-04-06T10:09:04.829 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T10:09:04.829 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_image_present cluster1 mirror image_1 present 2024-04-06T10:09:04.829 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T10:09:04.829 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:09:04.829 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_1 2024-04-06T10:09:04.829 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state=present 2024-04-06T10:09:04.830 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image_id= 2024-04-06T10:09:04.830 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local s 2024-04-06T10:09:04.830 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n '' 2024-04-06T10:09:04.831 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_image_id cluster1 mirror image_1 2024-04-06T10:09:04.831 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T10:09:04.831 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:09:04.831 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_1 2024-04-06T10:09:04.831 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster=cluster1 info mirror/image_1 2024-04-06T10:09:04.831 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-04-06T10:09:04.891 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ image_id=10ac58cf4061 2024-04-06T10:09:04.892 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-04-06T10:09:04.892 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.1 2024-04-06T10:09:04.992 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test_image_present cluster1 mirror image_1 present 10ac58cf4061 2024-04-06T10:09:04.993 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T10:09:04.993 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:09:04.993 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_1 2024-04-06T10:09:04.993 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local test_state=present 2024-04-06T10:09:04.993 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image_id=10ac58cf4061 2024-04-06T10:09:04.993 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local current_state=deleted 2024-04-06T10:09:04.993 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local current_image_id 2024-04-06T10:09:04.993 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_image_id cluster1 mirror image_1 2024-04-06T10:09:04.993 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T10:09:04.993 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:09:04.993 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_1 2024-04-06T10:09:04.994 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster=cluster1 info mirror/image_1 2024-04-06T10:09:04.994 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-04-06T10:09:05.052 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ current_image_id=10ac58cf4061 2024-04-06T10:09:05.052 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 10ac58cf4061 2024-04-06T10:09:05.052 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -z 10ac58cf4061 -o 10ac58cf4061 = 10ac58cf4061 2024-04-06T10:09:05.052 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ current_state=present 2024-04-06T10:09:05.052 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test present = present 2024-04-06T10:09:05.053 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T10:09:05.053 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ disable_mirror cluster2 mirror image_1 2024-04-06T10:09:05.053 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-04-06T10:09:05.053 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:09:05.053 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_1 2024-04-06T10:09:05.053 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster=cluster2 mirror image disable mirror/image_1 2024-04-06T10:09:05.992 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi192.stderr:2024-04-06T10:09:05.992+0000 7f3d28bce640 -1 rbd::mirror::InstanceWatcher: C_NotifyInstanceRequest: 0x56067b2cd200 finish: resending after timeout 2024-04-06T10:09:10.993 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi192.stderr:2024-04-06T10:09:10.996+0000 7f3d293cf640 -1 rbd::mirror::InstanceWatcher: C_NotifyInstanceRequest: 0x56067b2cd200 finish: resending after timeout 2024-04-06T10:09:15.995 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi192.stderr:2024-04-06T10:09:15.996+0000 7f3d28bce640 -1 rbd::mirror::InstanceWatcher: C_NotifyInstanceRequest: 0x56067b2cd200 finish: resending after timeout 2024-04-06T10:09:20.996 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi192.stderr:2024-04-06T10:09:20.996+0000 7f3d293cf640 -1 rbd::mirror::InstanceWatcher: C_NotifyInstanceRequest: 0x56067b2cd200 finish: resending after timeout 2024-04-06T10:09:25.997 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi192.stderr:2024-04-06T10:09:26.000+0000 7f3d28bce640 -1 rbd::mirror::InstanceWatcher: C_NotifyInstanceRequest: 0x56067b2cd200 finish: resending after timeout 2024-04-06T10:09:30.999 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi192.stderr:2024-04-06T10:09:31.000+0000 7f3d293cf640 -1 rbd::mirror::InstanceWatcher: C_NotifyInstanceRequest: 0x56067b2cd200 finish: resending after timeout 2024-04-06T10:09:36.000 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi192.stderr:2024-04-06T10:09:36.000+0000 7f3d28bce640 -1 rbd::mirror::InstanceWatcher: C_NotifyInstanceRequest: 0x56067b2cd200 finish: resending after timeout 2024-04-06T10:09:40.153 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi192.stderr:2024-04-06T10:09:40.152+0000 7f3d1bbb4640 -1 rbd::mirror::image_replayer::journal::SyncPointHandler: 0x560665aefb90 handle_update_sync_points: failed to update remote journal client meta for image 7ce9e214-e534-4489-8a7c-ff96b51d92c7: (2) No such file or directory 2024-04-06T10:09:40.153 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi192.stderr:2024-04-06T10:09:40.152+0000 7f3d1bbb4640 -1 rbd::mirror::image_sync::SyncPointPruneRequest: 0x56067a3a6780 handle_update_sync_points: failed to update client data: (2) No such file or directory 2024-04-06T10:09:40.153 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi192.stderr:2024-04-06T10:09:40.152+0000 7f3d1bbb4640 -1 rbd::mirror::ImageSync: 0x560665170c60 handle_prune_sync_points: failed to prune sync point: (2) No such file or directory 2024-04-06T10:09:40.153 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi192.stderr:2024-04-06T10:09:40.152+0000 7f3d1bbb4640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x560664edab60 handle_image_sync: failed to sync remote image: (2) No such file or directory 2024-04-06T10:09:40.231 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Mirroring disabled 2024-04-06T10:09:40.240 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_image_present cluster1 mirror image_1 deleted 2024-04-06T10:09:40.240 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T10:09:40.240 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:09:40.240 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_1 2024-04-06T10:09:40.240 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state=deleted 2024-04-06T10:09:40.240 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image_id= 2024-04-06T10:09:40.240 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local s 2024-04-06T10:09:40.240 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n '' 2024-04-06T10:09:40.240 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_image_id cluster1 mirror image_1 2024-04-06T10:09:40.241 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T10:09:40.241 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:09:40.241 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_1 2024-04-06T10:09:40.241 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster=cluster1 info mirror/image_1 2024-04-06T10:09:40.241 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-04-06T10:09:40.301 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ image_id=10ac58cf4061 2024-04-06T10:09:40.301 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-04-06T10:09:40.301 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.1 2024-04-06T10:09:40.402 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test_image_present cluster1 mirror image_1 deleted 10ac58cf4061 2024-04-06T10:09:40.403 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T10:09:40.403 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:09:40.403 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_1 2024-04-06T10:09:40.403 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local test_state=deleted 2024-04-06T10:09:40.403 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image_id=10ac58cf4061 2024-04-06T10:09:40.403 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local current_state=deleted 2024-04-06T10:09:40.403 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local current_image_id 2024-04-06T10:09:40.404 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_image_id cluster1 mirror image_1 2024-04-06T10:09:40.404 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T10:09:40.404 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:09:40.404 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_1 2024-04-06T10:09:40.404 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster=cluster1 info mirror/image_1 2024-04-06T10:09:40.404 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-04-06T10:09:40.450 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:rbd: error opening image image_1: (2) No such file or directory 2024-04-06T10:09:40.455 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ current_image_id= 2024-04-06T10:09:40.455 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n '' 2024-04-06T10:09:40.455 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test deleted = deleted 2024-04-06T10:09:40.455 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T10:09:40.455 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ purge_snapshots cluster2 mirror image_1 2024-04-06T10:09:40.455 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-04-06T10:09:40.455 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:09:40.455 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_1 2024-04-06T10:09:40.455 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster2 snap purge mirror/image_1 2024-04-06T10:09:40.504 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ remove_image_retry cluster2 mirror image_1 2024-04-06T10:09:40.505 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-04-06T10:09:40.505 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:09:40.505 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_1 2024-04-06T10:09:40.505 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 16 32 2024-04-06T10:09:40.505 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T10:09:40.505 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ remove_image cluster2 mirror image_1 2024-04-06T10:09:40.505 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-04-06T10:09:40.506 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:09:40.506 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_1 2024-04-06T10:09:40.506 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_1 2024-04-06T10:09:40.552 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster=cluster2 rm mirror/image_1 2024-04-06T10:09:40.808 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2024-04-06T10:09:40.813 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T10:09:40.813 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in `seq 1 10` 2024-04-06T10:09:40.813 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ image=image_2 2024-04-06T10:09:40.813 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ create_image cluster2 mirror image_2 512M 2024-04-06T10:09:40.813 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-04-06T10:09:40.813 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T10:09:40.813 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:09:40.813 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T10:09:40.813 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_2 2024-04-06T10:09:40.813 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T10:09:40.814 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local size=128 2024-04-06T10:09:40.814 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n 512M ']' 2024-04-06T10:09:40.814 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ size=512M 2024-04-06T10:09:40.814 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T10:09:40.814 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster2 create --size 512M --image-feature layering,exclusive-lock,journaling mirror/image_2 2024-04-06T10:09:40.872 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ enable_mirror cluster2 mirror image_2 2024-04-06T10:09:40.873 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-04-06T10:09:40.873 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:09:40.873 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_2 2024-04-06T10:09:40.873 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local mode=journal 2024-04-06T10:09:40.873 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_2 journal 2024-04-06T10:09:40.921 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Mirroring enabled 2024-04-06T10:09:40.930 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster=cluster2 info mirror/image_2 2024-04-06T10:09:40.969 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:rbd image 'image_2': 2024-04-06T10:09:40.970 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: size 512 MiB in 128 objects 2024-04-06T10:09:40.970 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: order 22 (4 MiB objects) 2024-04-06T10:09:40.970 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: snapshot_count: 0 2024-04-06T10:09:40.970 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: id: 470d70e25ee9 2024-04-06T10:09:40.970 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: block_name_prefix: rbd_data.470d70e25ee9 2024-04-06T10:09:40.970 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: format: 2 2024-04-06T10:09:40.970 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: features: layering, exclusive-lock, journaling 2024-04-06T10:09:40.970 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: op_features: 2024-04-06T10:09:40.970 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: flags: 2024-04-06T10:09:40.971 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: create_timestamp: Sat Apr 6 10:09:40 2024 2024-04-06T10:09:40.971 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: access_timestamp: Sat Apr 6 10:09:40 2024 2024-04-06T10:09:40.971 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: modify_timestamp: Sat Apr 6 10:09:40 2024 2024-04-06T10:09:40.971 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: journal: 470d70e25ee9 2024-04-06T10:09:40.971 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: mirroring state: enabled 2024-04-06T10:09:40.971 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: mirroring mode: journal 2024-04-06T10:09:40.971 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: mirroring global id: dece73a2-ad07-4af9-ab10-59076f29f9fb 2024-04-06T10:09:40.971 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: mirroring primary: true 2024-04-06T10:09:40.974 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ stress_write_image cluster2 mirror image_2 2024-04-06T10:09:40.974 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-04-06T10:09:40.974 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:09:40.974 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_2 2024-04-06T10:09:40.974 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-04-06T10:09:40.975 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local duration=6 2024-04-06T10:09:40.976 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set +e 2024-04-06T10:09:40.976 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ timeout 6s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_2 --debug-rbd=20 --debug-journaler=20 2024-04-06T10:09:42.769 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:100 IOs, 25404416 bytes 2024-04-06T10:09:43.088 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:200 IOs, 26017792 bytes 2024-04-06T10:09:46.984 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ error_code=124 2024-04-06T10:09:46.985 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set -e 2024-04-06T10:09:46.985 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 124 -eq 124 ']' 2024-04-06T10:09:46.985 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T10:09:46.985 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_image_present cluster1 mirror image_2 present 2024-04-06T10:09:46.985 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T10:09:46.985 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:09:46.985 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_2 2024-04-06T10:09:46.985 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state=present 2024-04-06T10:09:46.985 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image_id= 2024-04-06T10:09:46.985 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local s 2024-04-06T10:09:46.985 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n '' 2024-04-06T10:09:46.985 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_image_id cluster1 mirror image_2 2024-04-06T10:09:46.985 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T10:09:46.986 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:09:46.986 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_2 2024-04-06T10:09:46.986 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-04-06T10:09:46.986 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster=cluster1 info mirror/image_2 2024-04-06T10:09:47.051 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ image_id=10aca6b717e2 2024-04-06T10:09:47.051 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-04-06T10:09:47.051 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.1 2024-04-06T10:09:47.152 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test_image_present cluster1 mirror image_2 present 10aca6b717e2 2024-04-06T10:09:47.152 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T10:09:47.152 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:09:47.153 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_2 2024-04-06T10:09:47.153 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local test_state=present 2024-04-06T10:09:47.153 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image_id=10aca6b717e2 2024-04-06T10:09:47.153 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local current_state=deleted 2024-04-06T10:09:47.153 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local current_image_id 2024-04-06T10:09:47.153 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_image_id cluster1 mirror image_2 2024-04-06T10:09:47.153 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T10:09:47.154 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:09:47.154 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_2 2024-04-06T10:09:47.154 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster=cluster1 info mirror/image_2 2024-04-06T10:09:47.154 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-04-06T10:09:47.226 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ current_image_id=10aca6b717e2 2024-04-06T10:09:47.226 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 10aca6b717e2 2024-04-06T10:09:47.226 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -z 10aca6b717e2 -o 10aca6b717e2 = 10aca6b717e2 2024-04-06T10:09:47.226 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ current_state=present 2024-04-06T10:09:47.226 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test present = present 2024-04-06T10:09:47.226 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T10:09:47.226 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ disable_mirror cluster2 mirror image_2 2024-04-06T10:09:47.226 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-04-06T10:09:47.227 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:09:47.227 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_2 2024-04-06T10:09:47.227 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster=cluster2 mirror image disable mirror/image_2 2024-04-06T10:09:57.684 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.0 with skip (0.6905244405054551) > weight (0.25) 2024-04-06T10:09:57.684 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.1 with skip (0.4002098211289431) > weight (0.25) 2024-04-06T10:09:57.684 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.2 with skip (0.508304988640164) > weight (0.25) 2024-04-06T10:09:57.684 INFO:tasks.rbd_mirror_thrash:kill cluster2.client.mirror.3 2024-04-06T10:09:57.684 INFO:tasks.rbd_mirror.cluster2.client.mirror.3:Sent signal 15 2024-04-06T10:09:57.685 INFO:tasks.rbd_mirror_thrash:waiting for 2 secs before reviving daemons 2024-04-06T10:09:57.686 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi192.stderr:2024-04-06T10:09:57.687+0000 7f053fc2a640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.3 (PID: 131883) UID: 1000 2024-04-06T10:09:58.687 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.0 with skip (0.6085290913950405) > weight (0.25) 2024-04-06T10:09:58.687 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.1 with skip (0.5857215941858737) > weight (0.25) 2024-04-06T10:09:58.687 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.2 with skip (0.6006270153504641) > weight (0.25) 2024-04-06T10:09:58.687 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.3 with skip (0.3912869836262861) > weight (0.25) 2024-04-06T10:09:58.687 INFO:tasks.rbd_mirror_thrash:waiting for 75 secs before thrashing 2024-04-06T10:09:59.686 INFO:tasks.rbd_mirror_thrash:waiting for cluster2.client.mirror.3 2024-04-06T10:09:59.686 INFO:teuthology.orchestra.run:waiting for 600 2024-04-06T10:09:59.686 INFO:tasks.rbd_mirror_thrash:reviving cluster2.client.mirror.3 2024-04-06T10:09:59.686 INFO:tasks.rbd_mirror.cluster2.client.mirror.3:Restarting daemon 2024-04-06T10:09:59.687 DEBUG:teuthology.orchestra.run.smithi192:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.3 2024-04-06T10:09:59.689 INFO:tasks.rbd_mirror.cluster2.client.mirror.3:Started 2024-04-06T10:09:59.690 INFO:tasks.rbd_mirror_thrash:waiting for 69 secs before thrashing 2024-04-06T10:10:17.340 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Mirroring disabled 2024-04-06T10:10:17.355 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_image_present cluster1 mirror image_2 deleted 2024-04-06T10:10:17.355 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T10:10:17.355 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:10:17.355 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_2 2024-04-06T10:10:17.355 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state=deleted 2024-04-06T10:10:17.355 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image_id= 2024-04-06T10:10:17.355 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local s 2024-04-06T10:10:17.355 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n '' 2024-04-06T10:10:17.356 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_image_id cluster1 mirror image_2 2024-04-06T10:10:17.356 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T10:10:17.356 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:10:17.356 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_2 2024-04-06T10:10:17.356 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster=cluster1 info mirror/image_2 2024-04-06T10:10:17.356 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-04-06T10:10:17.397 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:rbd: error opening image image_2: (2) No such file or directory 2024-04-06T10:10:17.402 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ image_id= 2024-04-06T10:10:17.402 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-04-06T10:10:17.402 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.1 2024-04-06T10:10:17.503 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test_image_present cluster1 mirror image_2 deleted '' 2024-04-06T10:10:17.503 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T10:10:17.504 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:10:17.504 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_2 2024-04-06T10:10:17.504 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local test_state=deleted 2024-04-06T10:10:17.504 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image_id= 2024-04-06T10:10:17.504 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local current_state=deleted 2024-04-06T10:10:17.504 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local current_image_id 2024-04-06T10:10:17.505 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_image_id cluster1 mirror image_2 2024-04-06T10:10:17.505 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T10:10:17.505 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:10:17.505 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_2 2024-04-06T10:10:17.505 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster=cluster1 info mirror/image_2 2024-04-06T10:10:17.505 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-04-06T10:10:17.549 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:rbd: error opening image image_2: (2) No such file or directory 2024-04-06T10:10:17.554 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ current_image_id= 2024-04-06T10:10:17.554 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n '' 2024-04-06T10:10:17.554 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test deleted = deleted 2024-04-06T10:10:17.554 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T10:10:17.554 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ purge_snapshots cluster2 mirror image_2 2024-04-06T10:10:17.554 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-04-06T10:10:17.554 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:10:17.555 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_2 2024-04-06T10:10:17.555 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster2 snap purge mirror/image_2 2024-04-06T10:10:17.610 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ remove_image_retry cluster2 mirror image_2 2024-04-06T10:10:17.610 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-04-06T10:10:17.610 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:10:17.610 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_2 2024-04-06T10:10:17.611 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 16 32 2024-04-06T10:10:17.611 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T10:10:17.611 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ remove_image cluster2 mirror image_2 2024-04-06T10:10:17.611 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-04-06T10:10:17.611 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:10:17.611 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_2 2024-04-06T10:10:17.611 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_2 2024-04-06T10:10:17.659 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster=cluster2 rm mirror/image_2 2024-04-06T10:10:17.907 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2024-04-06T10:10:17.911 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T10:10:17.912 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in `seq 1 10` 2024-04-06T10:10:17.912 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ image=image_3 2024-04-06T10:10:17.912 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ create_image cluster2 mirror image_3 512M 2024-04-06T10:10:17.912 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-04-06T10:10:17.912 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T10:10:17.912 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:10:17.912 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T10:10:17.912 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_3 2024-04-06T10:10:17.912 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T10:10:17.912 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local size=128 2024-04-06T10:10:17.912 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n 512M ']' 2024-04-06T10:10:17.912 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ size=512M 2024-04-06T10:10:17.912 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T10:10:17.912 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster2 create --size 512M --image-feature layering,exclusive-lock,journaling mirror/image_3 2024-04-06T10:10:17.970 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ enable_mirror cluster2 mirror image_3 2024-04-06T10:10:17.970 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-04-06T10:10:17.971 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:10:17.971 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_3 2024-04-06T10:10:17.971 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local mode=journal 2024-04-06T10:10:17.971 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_3 journal 2024-04-06T10:10:18.018 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Mirroring enabled 2024-04-06T10:10:18.025 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster=cluster2 info mirror/image_3 2024-04-06T10:10:18.064 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:rbd image 'image_3': 2024-04-06T10:10:18.064 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: size 512 MiB in 128 objects 2024-04-06T10:10:18.064 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: order 22 (4 MiB objects) 2024-04-06T10:10:18.064 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: snapshot_count: 0 2024-04-06T10:10:18.064 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: id: 4728e3b43162 2024-04-06T10:10:18.064 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: block_name_prefix: rbd_data.4728e3b43162 2024-04-06T10:10:18.064 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: format: 2 2024-04-06T10:10:18.064 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: features: layering, exclusive-lock, journaling 2024-04-06T10:10:18.064 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: op_features: 2024-04-06T10:10:18.064 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: flags: 2024-04-06T10:10:18.065 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: create_timestamp: Sat Apr 6 10:10:17 2024 2024-04-06T10:10:18.065 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: access_timestamp: Sat Apr 6 10:10:17 2024 2024-04-06T10:10:18.065 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: modify_timestamp: Sat Apr 6 10:10:17 2024 2024-04-06T10:10:18.065 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: journal: 4728e3b43162 2024-04-06T10:10:18.065 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: mirroring state: enabled 2024-04-06T10:10:18.065 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: mirroring mode: journal 2024-04-06T10:10:18.065 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: mirroring global id: 0f3cf0d0-bef2-4be3-989e-1bef92f7054a 2024-04-06T10:10:18.065 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: mirroring primary: true 2024-04-06T10:10:18.068 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ stress_write_image cluster2 mirror image_3 2024-04-06T10:10:18.068 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-04-06T10:10:18.068 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:10:18.068 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_3 2024-04-06T10:10:18.068 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-04-06T10:10:18.070 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local duration=8 2024-04-06T10:10:18.070 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set +e 2024-04-06T10:10:18.070 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ timeout 8s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_3 --debug-rbd=20 --debug-journaler=20 2024-04-06T10:10:19.860 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:100 IOs, 23319552 bytes 2024-04-06T10:10:24.096 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:200 IOs, 75927552 bytes 2024-04-06T10:10:26.079 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ error_code=124 2024-04-06T10:10:26.079 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set -e 2024-04-06T10:10:26.079 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 124 -eq 124 ']' 2024-04-06T10:10:26.079 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T10:10:26.079 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_image_present cluster1 mirror image_3 present 2024-04-06T10:10:26.079 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T10:10:26.080 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:10:26.080 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_3 2024-04-06T10:10:26.080 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state=present 2024-04-06T10:10:26.080 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image_id= 2024-04-06T10:10:26.080 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local s 2024-04-06T10:10:26.080 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n '' 2024-04-06T10:10:26.080 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_image_id cluster1 mirror image_3 2024-04-06T10:10:26.080 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T10:10:26.080 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:10:26.080 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_3 2024-04-06T10:10:26.081 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-04-06T10:10:26.081 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster=cluster1 info mirror/image_3 2024-04-06T10:10:26.354 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ image_id=10ac417a0c3c 2024-04-06T10:10:26.354 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-04-06T10:10:26.354 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.1 2024-04-06T10:10:26.455 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test_image_present cluster1 mirror image_3 present 10ac417a0c3c 2024-04-06T10:10:26.456 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T10:10:26.456 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:10:26.456 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_3 2024-04-06T10:10:26.456 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local test_state=present 2024-04-06T10:10:26.456 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image_id=10ac417a0c3c 2024-04-06T10:10:26.456 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local current_state=deleted 2024-04-06T10:10:26.456 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local current_image_id 2024-04-06T10:10:26.457 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_image_id cluster1 mirror image_3 2024-04-06T10:10:26.457 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T10:10:26.457 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:10:26.457 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_3 2024-04-06T10:10:26.457 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster=cluster1 info mirror/image_3 2024-04-06T10:10:26.457 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-04-06T10:10:26.520 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ current_image_id=10ac417a0c3c 2024-04-06T10:10:26.520 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 10ac417a0c3c 2024-04-06T10:10:26.520 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -z 10ac417a0c3c -o 10ac417a0c3c = 10ac417a0c3c 2024-04-06T10:10:26.521 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ current_state=present 2024-04-06T10:10:26.521 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test present = present 2024-04-06T10:10:26.521 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T10:10:26.521 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ disable_mirror cluster2 mirror image_3 2024-04-06T10:10:26.521 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-04-06T10:10:26.521 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:10:26.521 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_3 2024-04-06T10:10:26.521 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster=cluster2 mirror image disable mirror/image_3 2024-04-06T10:10:26.691 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi192.stderr:2024-04-06T10:10:26.695+0000 7f3d293cf640 -1 rbd::mirror::InstanceWatcher: C_NotifyInstanceRequest: 0x56066544d440 finish: resending after timeout 2024-04-06T10:10:31.693 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi192.stderr:2024-04-06T10:10:31.691+0000 7f3d293cf640 -1 rbd::mirror::InstanceWatcher: C_NotifyInstanceRequest: 0x56066544d440 finish: resending after timeout 2024-04-06T10:10:32.619 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi192.stderr:2024-04-06T10:10:32.619+0000 7f4415487640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x561be61ce680 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-04-06T10:10:32.619 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi192.stderr:2024-04-06T10:10:32.619+0000 7f4415487640 -1 rbd::mirror::ImageReplayer: 0x561be6e5d900 [3/0f3cf0d0-bef2-4be3-989e-1bef92f7054a] operator(): start failed: (85) Interrupted system call should be restarted 2024-04-06T10:10:36.694 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi192.stderr:2024-04-06T10:10:36.695+0000 7f3d293cf640 -1 rbd::mirror::InstanceWatcher: C_NotifyInstanceRequest: 0x56066544d440 finish: resending after timeout 2024-04-06T10:10:41.695 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi192.stderr:2024-04-06T10:10:41.694+0000 7f3d28bce640 -1 rbd::mirror::InstanceWatcher: C_NotifyInstanceRequest: 0x56066544d440 finish: resending after timeout 2024-04-06T10:10:46.696 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi192.stderr:2024-04-06T10:10:46.694+0000 7f3d293cf640 -1 rbd::mirror::InstanceWatcher: C_NotifyInstanceRequest: 0x56066544d440 finish: resending after timeout 2024-04-06T10:10:51.697 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi192.stderr:2024-04-06T10:10:51.698+0000 7f3d28bce640 -1 rbd::mirror::InstanceWatcher: C_NotifyInstanceRequest: 0x56066544d440 finish: resending after timeout 2024-04-06T10:10:56.699 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi192.stderr:2024-04-06T10:10:56.698+0000 7f3d293cf640 -1 rbd::mirror::InstanceWatcher: C_NotifyInstanceRequest: 0x56066544d440 finish: resending after timeout 2024-04-06T10:11:02.616 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi192.stderr:2024-04-06T10:11:02.614+0000 7f4415c88640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x561be6226000 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-04-06T10:11:02.617 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi192.stderr:2024-04-06T10:11:02.614+0000 7f4415c88640 -1 rbd::mirror::ImageReplayer: 0x561be6e5d900 [3/0f3cf0d0-bef2-4be3-989e-1bef92f7054a] operator(): start failed: (85) Interrupted system call should be restarted 2024-04-06T10:11:03.256 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi192.stderr:2024-04-06T10:11:03.254+0000 7f3d293cf640 -1 rbd::mirror::InstanceWatcher: C_NotifyInstanceRequest: 0x56066544d440 finish: resending after timeout 2024-04-06T10:11:06.967 INFO:tasks.ceph.cluster2.osd.2.smithi192.stderr:2024-04-06T10:11:06.966+0000 7fa534aba640 -1 reset not still connected to 0x55933640fba0 2024-04-06T10:11:06.971 INFO:tasks.ceph.cluster2.osd.2.smithi192.stderr:2024-04-06T10:11:06.970+0000 7fa534aba640 -1 reset not still connected to 0x55933bc92d00 2024-04-06T10:11:06.971 INFO:tasks.ceph.cluster2.osd.2.smithi192.stderr:2024-04-06T10:11:06.970+0000 7fa534aba640 -1 reset not still connected to 0x55933ce91110 2024-04-06T10:11:06.972 INFO:tasks.ceph.cluster2.osd.2.smithi192.stderr:2024-04-06T10:11:06.970+0000 7fa534aba640 -1 reset not still connected to 0x55934847eb60 2024-04-06T10:11:06.972 INFO:tasks.ceph.cluster2.osd.2.smithi192.stderr:2024-04-06T10:11:06.970+0000 7fa534aba640 -1 reset not still connected to 0x559365721e10 2024-04-06T10:11:06.972 INFO:tasks.ceph.cluster2.osd.2.smithi192.stderr:2024-04-06T10:11:06.970+0000 7fa534aba640 -1 reset not still connected to 0x55933e3c0ea0 2024-04-06T10:11:08.025 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi192.stderr:2024-04-06T10:11:08.026+0000 7f3d1c3b5640 -1 rbd::mirror::image_replayer::journal::SyncPointHandler: 0x560668b79440 handle_update_sync_points: failed to update remote journal client meta for image 0f3cf0d0-bef2-4be3-989e-1bef92f7054a: (2) No such file or directory 2024-04-06T10:11:08.026 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi192.stderr:2024-04-06T10:11:08.026+0000 7f3d1c3b5640 -1 rbd::mirror::image_sync::SyncPointPruneRequest: 0x56068d24b440 handle_update_sync_points: failed to update client data: (2) No such file or directory 2024-04-06T10:11:08.026 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi192.stderr:2024-04-06T10:11:08.026+0000 7f3d1c3b5640 -1 rbd::mirror::ImageSync: 0x56067b2909a0 handle_prune_sync_points: failed to prune sync point: (2) No such file or directory 2024-04-06T10:11:08.026 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi192.stderr:2024-04-06T10:11:08.026+0000 7f3d1c3b5640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x560664c0e680 handle_image_sync: failed to sync remote image: (2) No such file or directory 2024-04-06T10:11:08.181 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Mirroring disabled 2024-04-06T10:11:08.192 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_image_present cluster1 mirror image_3 deleted 2024-04-06T10:11:08.192 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T10:11:08.192 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:11:08.192 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_3 2024-04-06T10:11:08.192 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state=deleted 2024-04-06T10:11:08.192 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image_id= 2024-04-06T10:11:08.192 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local s 2024-04-06T10:11:08.192 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n '' 2024-04-06T10:11:08.192 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_image_id cluster1 mirror image_3 2024-04-06T10:11:08.192 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T10:11:08.192 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:11:08.193 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_3 2024-04-06T10:11:08.193 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster=cluster1 info mirror/image_3 2024-04-06T10:11:08.193 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-04-06T10:11:08.254 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ image_id=10ac417a0c3c 2024-04-06T10:11:08.254 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-04-06T10:11:08.254 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.1 2024-04-06T10:11:08.356 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test_image_present cluster1 mirror image_3 deleted 10ac417a0c3c 2024-04-06T10:11:08.356 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T10:11:08.356 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:11:08.356 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_3 2024-04-06T10:11:08.356 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local test_state=deleted 2024-04-06T10:11:08.356 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image_id=10ac417a0c3c 2024-04-06T10:11:08.356 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local current_state=deleted 2024-04-06T10:11:08.356 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local current_image_id 2024-04-06T10:11:08.357 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_image_id cluster1 mirror image_3 2024-04-06T10:11:08.357 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T10:11:08.357 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:11:08.357 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_3 2024-04-06T10:11:08.357 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster=cluster1 info mirror/image_3 2024-04-06T10:11:08.358 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-04-06T10:11:08.404 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:rbd: error opening image image_3: (2) No such file or directory 2024-04-06T10:11:08.409 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ current_image_id= 2024-04-06T10:11:08.410 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n '' 2024-04-06T10:11:08.410 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test deleted = deleted 2024-04-06T10:11:08.410 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T10:11:08.410 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ purge_snapshots cluster2 mirror image_3 2024-04-06T10:11:08.410 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-04-06T10:11:08.410 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:11:08.410 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_3 2024-04-06T10:11:08.410 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster2 snap purge mirror/image_3 2024-04-06T10:11:08.460 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ remove_image_retry cluster2 mirror image_3 2024-04-06T10:11:08.460 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-04-06T10:11:08.460 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:11:08.460 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_3 2024-04-06T10:11:08.460 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 16 32 2024-04-06T10:11:08.460 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T10:11:08.461 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ remove_image cluster2 mirror image_3 2024-04-06T10:11:08.461 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-04-06T10:11:08.461 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:11:08.461 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_3 2024-04-06T10:11:08.461 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_3 2024-04-06T10:11:08.509 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster=cluster2 rm mirror/image_3 2024-04-06T10:11:08.690 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.0 with skip (0.6771656657700643) > weight (0.25) 2024-04-06T10:11:08.691 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.1 with skip (0.628748820165363) > weight (0.25) 2024-04-06T10:11:08.691 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.2 with skip (0.9419525934084587) > weight (0.25) 2024-04-06T10:11:08.691 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.3 with skip (0.9554615595738253) > weight (0.25) 2024-04-06T10:11:08.691 INFO:tasks.rbd_mirror_thrash:waiting for 69 secs before thrashing 2024-04-06T10:11:08.799 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2024-04-06T10:11:08.803 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T10:11:08.804 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in `seq 1 10` 2024-04-06T10:11:08.804 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ image=image_4 2024-04-06T10:11:08.804 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ create_image cluster2 mirror image_4 512M 2024-04-06T10:11:08.804 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-04-06T10:11:08.804 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T10:11:08.804 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:11:08.804 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T10:11:08.804 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_4 2024-04-06T10:11:08.804 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T10:11:08.804 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local size=128 2024-04-06T10:11:08.804 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n 512M ']' 2024-04-06T10:11:08.804 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ size=512M 2024-04-06T10:11:08.804 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T10:11:08.804 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster2 create --size 512M --image-feature layering,exclusive-lock,journaling mirror/image_4 2024-04-06T10:11:08.864 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ enable_mirror cluster2 mirror image_4 2024-04-06T10:11:08.864 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-04-06T10:11:08.865 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:11:08.865 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_4 2024-04-06T10:11:08.865 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local mode=journal 2024-04-06T10:11:08.865 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_4 journal 2024-04-06T10:11:08.912 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Mirroring enabled 2024-04-06T10:11:08.919 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster=cluster2 info mirror/image_4 2024-04-06T10:11:08.959 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:rbd image 'image_4': 2024-04-06T10:11:08.959 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: size 512 MiB in 128 objects 2024-04-06T10:11:08.959 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: order 22 (4 MiB objects) 2024-04-06T10:11:08.959 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: snapshot_count: 0 2024-04-06T10:11:08.959 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: id: 473d5936afad 2024-04-06T10:11:08.959 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: block_name_prefix: rbd_data.473d5936afad 2024-04-06T10:11:08.959 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: format: 2 2024-04-06T10:11:08.959 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: features: layering, exclusive-lock, journaling 2024-04-06T10:11:08.959 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: op_features: 2024-04-06T10:11:08.959 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: flags: 2024-04-06T10:11:08.960 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: create_timestamp: Sat Apr 6 10:11:08 2024 2024-04-06T10:11:08.960 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: access_timestamp: Sat Apr 6 10:11:08 2024 2024-04-06T10:11:08.960 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: modify_timestamp: Sat Apr 6 10:11:08 2024 2024-04-06T10:11:08.960 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: journal: 473d5936afad 2024-04-06T10:11:08.960 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: mirroring state: enabled 2024-04-06T10:11:08.960 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: mirroring mode: journal 2024-04-06T10:11:08.960 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: mirroring global id: 7b7fc9cb-3bf5-4a87-9481-5a9521e44142 2024-04-06T10:11:08.960 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: mirroring primary: true 2024-04-06T10:11:08.963 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ stress_write_image cluster2 mirror image_4 2024-04-06T10:11:08.963 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-04-06T10:11:08.963 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:11:08.964 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_4 2024-04-06T10:11:08.964 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-04-06T10:11:08.965 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local duration=8 2024-04-06T10:11:08.965 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set +e 2024-04-06T10:11:08.965 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ timeout 8s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_4 --debug-rbd=20 --debug-journaler=20 2024-04-06T10:11:13.688 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.0 with skip (0.39042890941002517) > weight (0.25) 2024-04-06T10:11:13.688 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.1 with skip (0.7447472242472744) > weight (0.25) 2024-04-06T10:11:13.688 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.2 with skip (0.8577674494107171) > weight (0.25) 2024-04-06T10:11:13.688 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.3 with skip (0.71266969905125) > weight (0.25) 2024-04-06T10:11:13.688 INFO:tasks.rbd_mirror_thrash:waiting for 111 secs before thrashing 2024-04-06T10:11:14.208 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:100 IOs, 69189632 bytes 2024-04-06T10:11:16.974 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ error_code=124 2024-04-06T10:11:16.974 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set -e 2024-04-06T10:11:16.974 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 124 -eq 124 ']' 2024-04-06T10:11:16.974 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T10:11:16.974 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_image_present cluster1 mirror image_4 present 2024-04-06T10:11:16.974 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T10:11:16.974 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:11:16.974 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_4 2024-04-06T10:11:16.974 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state=present 2024-04-06T10:11:16.974 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image_id= 2024-04-06T10:11:16.975 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local s 2024-04-06T10:11:16.975 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n '' 2024-04-06T10:11:16.975 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_image_id cluster1 mirror image_4 2024-04-06T10:11:16.975 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T10:11:16.975 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:11:16.975 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_4 2024-04-06T10:11:16.975 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster=cluster1 info mirror/image_4 2024-04-06T10:11:16.975 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-04-06T10:11:17.036 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ image_id=10acbcfbccd0 2024-04-06T10:11:17.036 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-04-06T10:11:17.036 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.1 2024-04-06T10:11:17.137 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test_image_present cluster1 mirror image_4 present 10acbcfbccd0 2024-04-06T10:11:17.137 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T10:11:17.137 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:11:17.137 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_4 2024-04-06T10:11:17.137 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local test_state=present 2024-04-06T10:11:17.137 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image_id=10acbcfbccd0 2024-04-06T10:11:17.137 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local current_state=deleted 2024-04-06T10:11:17.137 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local current_image_id 2024-04-06T10:11:17.138 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_image_id cluster1 mirror image_4 2024-04-06T10:11:17.138 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T10:11:17.139 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:11:17.139 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_4 2024-04-06T10:11:17.139 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster=cluster1 info mirror/image_4 2024-04-06T10:11:17.139 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-04-06T10:11:17.208 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ current_image_id=10acbcfbccd0 2024-04-06T10:11:17.208 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 10acbcfbccd0 2024-04-06T10:11:17.208 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -z 10acbcfbccd0 -o 10acbcfbccd0 = 10acbcfbccd0 2024-04-06T10:11:17.208 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ current_state=present 2024-04-06T10:11:17.208 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test present = present 2024-04-06T10:11:17.208 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T10:11:17.209 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ disable_mirror cluster2 mirror image_4 2024-04-06T10:11:17.209 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-04-06T10:11:17.209 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:11:17.209 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_4 2024-04-06T10:11:17.209 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster=cluster2 mirror image disable mirror/image_4 2024-04-06T10:11:45.124 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Mirroring disabled 2024-04-06T10:11:45.134 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_image_present cluster1 mirror image_4 deleted 2024-04-06T10:11:45.135 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T10:11:45.135 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:11:45.135 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_4 2024-04-06T10:11:45.135 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state=deleted 2024-04-06T10:11:45.135 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image_id= 2024-04-06T10:11:45.135 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local s 2024-04-06T10:11:45.135 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n '' 2024-04-06T10:11:45.135 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_image_id cluster1 mirror image_4 2024-04-06T10:11:45.135 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T10:11:45.136 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:11:45.136 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_4 2024-04-06T10:11:45.136 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster=cluster1 info mirror/image_4 2024-04-06T10:11:45.136 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-04-06T10:11:45.171 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:rbd: error opening image image_4: (2) No such file or directory 2024-04-06T10:11:45.175 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ image_id= 2024-04-06T10:11:45.175 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-04-06T10:11:45.175 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.1 2024-04-06T10:11:45.277 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test_image_present cluster1 mirror image_4 deleted '' 2024-04-06T10:11:45.277 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T10:11:45.277 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:11:45.277 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_4 2024-04-06T10:11:45.277 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local test_state=deleted 2024-04-06T10:11:45.277 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image_id= 2024-04-06T10:11:45.277 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local current_state=deleted 2024-04-06T10:11:45.277 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local current_image_id 2024-04-06T10:11:45.278 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_image_id cluster1 mirror image_4 2024-04-06T10:11:45.278 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T10:11:45.278 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:11:45.278 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_4 2024-04-06T10:11:45.279 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster=cluster1 info mirror/image_4 2024-04-06T10:11:45.279 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-04-06T10:11:45.323 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:rbd: error opening image image_4: (2) No such file or directory 2024-04-06T10:11:45.328 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ current_image_id= 2024-04-06T10:11:45.328 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n '' 2024-04-06T10:11:45.328 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test deleted = deleted 2024-04-06T10:11:45.328 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T10:11:45.328 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ purge_snapshots cluster2 mirror image_4 2024-04-06T10:11:45.328 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-04-06T10:11:45.328 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:11:45.328 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_4 2024-04-06T10:11:45.329 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster2 snap purge mirror/image_4 2024-04-06T10:11:45.378 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ remove_image_retry cluster2 mirror image_4 2024-04-06T10:11:45.378 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-04-06T10:11:45.379 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:11:45.379 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_4 2024-04-06T10:11:45.379 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 16 32 2024-04-06T10:11:45.379 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T10:11:45.379 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ remove_image cluster2 mirror image_4 2024-04-06T10:11:45.379 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-04-06T10:11:45.380 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:11:45.380 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_4 2024-04-06T10:11:45.380 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_4 2024-04-06T10:11:45.428 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster=cluster2 rm mirror/image_4 2024-04-06T10:11:45.682 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2024-04-06T10:11:45.687 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T10:11:45.687 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in `seq 1 10` 2024-04-06T10:11:45.687 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ image=image_5 2024-04-06T10:11:45.687 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ create_image cluster2 mirror image_5 512M 2024-04-06T10:11:45.688 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-04-06T10:11:45.688 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T10:11:45.688 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:11:45.688 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T10:11:45.688 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_5 2024-04-06T10:11:45.688 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T10:11:45.688 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local size=128 2024-04-06T10:11:45.688 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n 512M ']' 2024-04-06T10:11:45.688 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ size=512M 2024-04-06T10:11:45.688 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T10:11:45.688 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster2 create --size 512M --image-feature layering,exclusive-lock,journaling mirror/image_5 2024-04-06T10:11:45.749 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ enable_mirror cluster2 mirror image_5 2024-04-06T10:11:45.749 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-04-06T10:11:45.749 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:11:45.749 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_5 2024-04-06T10:11:45.749 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local mode=journal 2024-04-06T10:11:45.749 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_5 journal 2024-04-06T10:11:45.802 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Mirroring enabled 2024-04-06T10:11:45.809 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster=cluster2 info mirror/image_5 2024-04-06T10:11:45.849 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:rbd image 'image_5': 2024-04-06T10:11:45.849 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: size 512 MiB in 128 objects 2024-04-06T10:11:45.849 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: order 22 (4 MiB objects) 2024-04-06T10:11:45.849 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: snapshot_count: 0 2024-04-06T10:11:45.849 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: id: 474f46c360e6 2024-04-06T10:11:45.849 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: block_name_prefix: rbd_data.474f46c360e6 2024-04-06T10:11:45.849 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: format: 2 2024-04-06T10:11:45.849 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: features: layering, exclusive-lock, journaling 2024-04-06T10:11:45.849 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: op_features: 2024-04-06T10:11:45.849 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: flags: 2024-04-06T10:11:45.849 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: create_timestamp: Sat Apr 6 10:11:45 2024 2024-04-06T10:11:45.849 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: access_timestamp: Sat Apr 6 10:11:45 2024 2024-04-06T10:11:45.850 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: modify_timestamp: Sat Apr 6 10:11:45 2024 2024-04-06T10:11:45.850 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: journal: 474f46c360e6 2024-04-06T10:11:45.850 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: mirroring state: enabled 2024-04-06T10:11:45.850 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: mirroring mode: journal 2024-04-06T10:11:45.850 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: mirroring global id: 34c92cd3-1c76-4cfa-88e2-8b0618ccc4ac 2024-04-06T10:11:45.850 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: mirroring primary: true 2024-04-06T10:11:45.853 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ stress_write_image cluster2 mirror image_5 2024-04-06T10:11:45.853 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-04-06T10:11:45.853 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:11:45.853 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_5 2024-04-06T10:11:45.853 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-04-06T10:11:45.855 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local duration=6 2024-04-06T10:11:45.855 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set +e 2024-04-06T10:11:45.855 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ timeout 6s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_5 --debug-rbd=20 --debug-journaler=20 2024-04-06T10:11:47.737 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:100 IOs, 24243200 bytes 2024-04-06T10:11:47.922 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-04-06T10:11:47.922+0000 7f2e84d66640 -1 reset not still connected to 0x560d29d5b040 2024-04-06T10:11:50.942 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:200 IOs, 57806848 bytes 2024-04-06T10:11:51.507 INFO:tasks.ceph.cluster2.osd.0.smithi192.stderr:2024-04-06T10:11:51.505+0000 7f789d271640 -1 bdev(0x56041f2dce00 /var/lib/ceph/osd/cluster2-0/block) aio_submit retries 1 2024-04-06T10:11:51.508 INFO:tasks.ceph.cluster2.osd.0.smithi192.stderr:2024-04-06T10:11:51.505+0000 7f789f275640 -1 bdev(0x56041f2dce00 /var/lib/ceph/osd/cluster2-0/block) aio_submit retries 1 2024-04-06T10:11:51.865 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ error_code=124 2024-04-06T10:11:51.865 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set -e 2024-04-06T10:11:51.865 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 124 -eq 124 ']' 2024-04-06T10:11:51.866 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T10:11:51.866 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_image_present cluster1 mirror image_5 present 2024-04-06T10:11:51.866 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T10:11:51.866 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:11:51.866 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_5 2024-04-06T10:11:51.866 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state=present 2024-04-06T10:11:51.866 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image_id= 2024-04-06T10:11:51.866 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local s 2024-04-06T10:11:51.866 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n '' 2024-04-06T10:11:51.866 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_image_id cluster1 mirror image_5 2024-04-06T10:11:51.866 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T10:11:51.866 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:11:51.866 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_5 2024-04-06T10:11:51.867 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-04-06T10:11:51.867 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster=cluster1 info mirror/image_5 2024-04-06T10:11:51.928 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ image_id=10ac2af9bf9b 2024-04-06T10:11:51.928 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-04-06T10:11:51.928 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.1 2024-04-06T10:11:52.029 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test_image_present cluster1 mirror image_5 present 10ac2af9bf9b 2024-04-06T10:11:52.030 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T10:11:52.030 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:11:52.030 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_5 2024-04-06T10:11:52.030 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local test_state=present 2024-04-06T10:11:52.030 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image_id=10ac2af9bf9b 2024-04-06T10:11:52.030 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local current_state=deleted 2024-04-06T10:11:52.030 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local current_image_id 2024-04-06T10:11:52.031 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_image_id cluster1 mirror image_5 2024-04-06T10:11:52.031 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T10:11:52.031 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:11:52.031 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_5 2024-04-06T10:11:52.031 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster=cluster1 info mirror/image_5 2024-04-06T10:11:52.031 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-04-06T10:11:52.097 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ current_image_id=10ac2af9bf9b 2024-04-06T10:11:52.097 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 10ac2af9bf9b 2024-04-06T10:11:52.097 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -z 10ac2af9bf9b -o 10ac2af9bf9b = 10ac2af9bf9b 2024-04-06T10:11:52.097 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ current_state=present 2024-04-06T10:11:52.097 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test present = present 2024-04-06T10:11:52.097 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T10:11:52.097 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ disable_mirror cluster2 mirror image_5 2024-04-06T10:11:52.097 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-04-06T10:11:52.097 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:11:52.097 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_5 2024-04-06T10:11:52.097 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster=cluster2 mirror image disable mirror/image_5 2024-04-06T10:11:52.349 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-04-06T10:11:52.350+0000 7fd1b6210640 -1 reset not still connected to 0x55ef997761a0 2024-04-06T10:11:52.350 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-04-06T10:11:52.350+0000 7fd1b6210640 -1 reset not still connected to 0x55ef99776680 2024-04-06T10:11:52.350 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-04-06T10:11:52.350+0000 7fd1b6210640 -1 reset not still connected to 0x55ef99b88d00 2024-04-06T10:11:52.350 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-04-06T10:11:52.350+0000 7fd1b6210640 -1 reset not still connected to 0x55ef9ac60270 2024-04-06T10:12:17.692 INFO:tasks.rbd_mirror_thrash:kill cluster2.client.mirror.0 2024-04-06T10:12:17.693 INFO:tasks.rbd_mirror.cluster2.client.mirror.0:Sent signal 15 2024-04-06T10:12:17.693 INFO:tasks.rbd_mirror_thrash:waiting for 8 secs before reviving daemons 2024-04-06T10:12:17.694 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi192.stderr:2024-04-06T10:12:17.697+0000 7fa345920640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.0 (PID: 168477) UID: 1000 2024-04-06T10:12:22.124 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Mirroring disabled 2024-04-06T10:12:22.134 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_image_present cluster1 mirror image_5 deleted 2024-04-06T10:12:22.134 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T10:12:22.135 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:12:22.135 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_5 2024-04-06T10:12:22.135 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state=deleted 2024-04-06T10:12:22.135 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image_id= 2024-04-06T10:12:22.135 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local s 2024-04-06T10:12:22.135 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n '' 2024-04-06T10:12:22.136 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_image_id cluster1 mirror image_5 2024-04-06T10:12:22.136 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T10:12:22.136 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:12:22.136 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_5 2024-04-06T10:12:22.136 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster=cluster1 info mirror/image_5 2024-04-06T10:12:22.136 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-04-06T10:12:22.173 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:rbd: error opening image image_5: (2) No such file or directory 2024-04-06T10:12:22.176 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ image_id= 2024-04-06T10:12:22.177 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-04-06T10:12:22.177 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.1 2024-04-06T10:12:22.278 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test_image_present cluster1 mirror image_5 deleted '' 2024-04-06T10:12:22.278 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T10:12:22.278 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:12:22.279 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_5 2024-04-06T10:12:22.279 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local test_state=deleted 2024-04-06T10:12:22.279 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image_id= 2024-04-06T10:12:22.279 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local current_state=deleted 2024-04-06T10:12:22.279 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local current_image_id 2024-04-06T10:12:22.280 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_image_id cluster1 mirror image_5 2024-04-06T10:12:22.280 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T10:12:22.280 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:12:22.280 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_5 2024-04-06T10:12:22.280 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster=cluster1 info mirror/image_5 2024-04-06T10:12:22.280 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-04-06T10:12:22.323 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:rbd: error opening image image_5: (2) No such file or directory 2024-04-06T10:12:22.327 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ current_image_id= 2024-04-06T10:12:22.327 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n '' 2024-04-06T10:12:22.327 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test deleted = deleted 2024-04-06T10:12:22.328 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T10:12:22.328 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ purge_snapshots cluster2 mirror image_5 2024-04-06T10:12:22.328 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-04-06T10:12:22.328 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:12:22.328 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_5 2024-04-06T10:12:22.328 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster2 snap purge mirror/image_5 2024-04-06T10:12:22.377 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ remove_image_retry cluster2 mirror image_5 2024-04-06T10:12:22.377 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-04-06T10:12:22.377 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:12:22.378 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_5 2024-04-06T10:12:22.378 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 16 32 2024-04-06T10:12:22.378 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T10:12:22.378 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ remove_image cluster2 mirror image_5 2024-04-06T10:12:22.378 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-04-06T10:12:22.378 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:12:22.378 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_5 2024-04-06T10:12:22.379 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_5 2024-04-06T10:12:22.425 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster=cluster2 rm mirror/image_5 2024-04-06T10:12:22.676 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2024-04-06T10:12:22.680 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T10:12:22.680 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in `seq 1 10` 2024-04-06T10:12:22.680 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ image=image_6 2024-04-06T10:12:22.680 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ create_image cluster2 mirror image_6 512M 2024-04-06T10:12:22.681 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-04-06T10:12:22.681 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T10:12:22.681 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:12:22.681 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T10:12:22.681 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_6 2024-04-06T10:12:22.681 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T10:12:22.681 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local size=128 2024-04-06T10:12:22.681 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n 512M ']' 2024-04-06T10:12:22.681 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ size=512M 2024-04-06T10:12:22.681 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T10:12:22.681 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster2 create --size 512M --image-feature layering,exclusive-lock,journaling mirror/image_6 2024-04-06T10:12:22.741 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ enable_mirror cluster2 mirror image_6 2024-04-06T10:12:22.741 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-04-06T10:12:22.742 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:12:22.742 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_6 2024-04-06T10:12:22.742 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local mode=journal 2024-04-06T10:12:22.742 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_6 journal 2024-04-06T10:12:22.790 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Mirroring enabled 2024-04-06T10:12:22.797 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster=cluster2 info mirror/image_6 2024-04-06T10:12:22.835 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:rbd image 'image_6': 2024-04-06T10:12:22.835 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: size 512 MiB in 128 objects 2024-04-06T10:12:22.835 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: order 22 (4 MiB objects) 2024-04-06T10:12:22.836 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: snapshot_count: 0 2024-04-06T10:12:22.836 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: id: 475f27393720 2024-04-06T10:12:22.836 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: block_name_prefix: rbd_data.475f27393720 2024-04-06T10:12:22.836 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: format: 2 2024-04-06T10:12:22.836 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: features: layering, exclusive-lock, journaling 2024-04-06T10:12:22.836 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: op_features: 2024-04-06T10:12:22.836 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: flags: 2024-04-06T10:12:22.836 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: create_timestamp: Sat Apr 6 10:12:22 2024 2024-04-06T10:12:22.836 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: access_timestamp: Sat Apr 6 10:12:22 2024 2024-04-06T10:12:22.836 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: modify_timestamp: Sat Apr 6 10:12:22 2024 2024-04-06T10:12:22.837 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: journal: 475f27393720 2024-04-06T10:12:22.837 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: mirroring state: enabled 2024-04-06T10:12:22.837 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: mirroring mode: journal 2024-04-06T10:12:22.837 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: mirroring global id: 9edbb6fe-8d39-4dda-89f8-b51dae699f0e 2024-04-06T10:12:22.837 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: mirroring primary: true 2024-04-06T10:12:22.840 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ stress_write_image cluster2 mirror image_6 2024-04-06T10:12:22.840 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-04-06T10:12:22.840 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:12:22.840 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_6 2024-04-06T10:12:22.841 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-04-06T10:12:22.842 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local duration=10 2024-04-06T10:12:22.842 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set +e 2024-04-06T10:12:22.842 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ timeout 10s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_6 --debug-rbd=20 --debug-journaler=20 2024-04-06T10:12:25.694 INFO:tasks.rbd_mirror_thrash:waiting for cluster2.client.mirror.0 2024-04-06T10:12:25.694 INFO:teuthology.orchestra.run:waiting for 600 2024-04-06T10:12:25.694 INFO:tasks.rbd_mirror_thrash:reviving cluster2.client.mirror.0 2024-04-06T10:12:25.695 INFO:tasks.rbd_mirror.cluster2.client.mirror.0:Restarting daemon 2024-04-06T10:12:25.695 DEBUG:teuthology.orchestra.run.smithi192:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.0 2024-04-06T10:12:25.697 INFO:tasks.rbd_mirror.cluster2.client.mirror.0:Started 2024-04-06T10:12:25.698 INFO:tasks.rbd_mirror_thrash:waiting for 62 secs before thrashing 2024-04-06T10:12:30.088 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:100 IOs, 109028352 bytes 2024-04-06T10:12:31.017 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:200 IOs, 118320128 bytes 2024-04-06T10:12:31.943 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:300 IOs, 137344000 bytes 2024-04-06T10:12:32.850 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ error_code=124 2024-04-06T10:12:32.850 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set -e 2024-04-06T10:12:32.850 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 124 -eq 124 ']' 2024-04-06T10:12:32.850 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T10:12:32.850 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_image_present cluster1 mirror image_6 present 2024-04-06T10:12:32.851 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T10:12:32.851 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:12:32.851 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_6 2024-04-06T10:12:32.851 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state=present 2024-04-06T10:12:32.851 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image_id= 2024-04-06T10:12:32.851 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local s 2024-04-06T10:12:32.851 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n '' 2024-04-06T10:12:32.851 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_image_id cluster1 mirror image_6 2024-04-06T10:12:32.851 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T10:12:32.851 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:12:32.851 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_6 2024-04-06T10:12:32.851 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster=cluster1 info mirror/image_6 2024-04-06T10:12:32.851 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-04-06T10:12:32.912 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ image_id=10ac82eecb47 2024-04-06T10:12:32.912 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-04-06T10:12:32.912 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.1 2024-04-06T10:12:33.013 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test_image_present cluster1 mirror image_6 present 10ac82eecb47 2024-04-06T10:12:33.014 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T10:12:33.014 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:12:33.014 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_6 2024-04-06T10:12:33.014 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local test_state=present 2024-04-06T10:12:33.014 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image_id=10ac82eecb47 2024-04-06T10:12:33.014 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local current_state=deleted 2024-04-06T10:12:33.014 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local current_image_id 2024-04-06T10:12:33.015 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_image_id cluster1 mirror image_6 2024-04-06T10:12:33.015 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T10:12:33.015 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:12:33.015 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_6 2024-04-06T10:12:33.015 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster=cluster1 info mirror/image_6 2024-04-06T10:12:33.015 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-04-06T10:12:33.080 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ current_image_id=10ac82eecb47 2024-04-06T10:12:33.080 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 10ac82eecb47 2024-04-06T10:12:33.080 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -z 10ac82eecb47 -o 10ac82eecb47 = 10ac82eecb47 2024-04-06T10:12:33.080 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ current_state=present 2024-04-06T10:12:33.080 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test present = present 2024-04-06T10:12:33.080 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T10:12:33.081 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ disable_mirror cluster2 mirror image_6 2024-04-06T10:12:33.081 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-04-06T10:12:33.081 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:12:33.081 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_6 2024-04-06T10:12:33.081 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster=cluster2 mirror image disable mirror/image_6 2024-04-06T10:13:03.498 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Mirroring disabled 2024-04-06T10:13:03.508 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_image_present cluster1 mirror image_6 deleted 2024-04-06T10:13:03.508 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T10:13:03.508 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:13:03.508 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_6 2024-04-06T10:13:03.508 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state=deleted 2024-04-06T10:13:03.508 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image_id= 2024-04-06T10:13:03.508 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local s 2024-04-06T10:13:03.508 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n '' 2024-04-06T10:13:03.508 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_image_id cluster1 mirror image_6 2024-04-06T10:13:03.508 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T10:13:03.508 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:13:03.509 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_6 2024-04-06T10:13:03.509 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster=cluster1 info mirror/image_6 2024-04-06T10:13:03.509 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-04-06T10:13:03.545 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:rbd: error opening image image_6: (2) No such file or directory 2024-04-06T10:13:03.549 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ image_id= 2024-04-06T10:13:03.549 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-04-06T10:13:03.549 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.1 2024-04-06T10:13:03.650 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test_image_present cluster1 mirror image_6 deleted '' 2024-04-06T10:13:03.651 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T10:13:03.651 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:13:03.651 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_6 2024-04-06T10:13:03.651 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local test_state=deleted 2024-04-06T10:13:03.651 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image_id= 2024-04-06T10:13:03.651 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local current_state=deleted 2024-04-06T10:13:03.651 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local current_image_id 2024-04-06T10:13:03.652 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_image_id cluster1 mirror image_6 2024-04-06T10:13:03.652 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T10:13:03.652 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:13:03.652 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_6 2024-04-06T10:13:03.652 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster=cluster1 info mirror/image_6 2024-04-06T10:13:03.652 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-04-06T10:13:03.698 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:rbd: error opening image image_6: (2) No such file or directory 2024-04-06T10:13:03.703 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ current_image_id= 2024-04-06T10:13:03.703 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n '' 2024-04-06T10:13:03.703 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test deleted = deleted 2024-04-06T10:13:03.703 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T10:13:03.703 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ purge_snapshots cluster2 mirror image_6 2024-04-06T10:13:03.703 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-04-06T10:13:03.703 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:13:03.703 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_6 2024-04-06T10:13:03.703 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster2 snap purge mirror/image_6 2024-04-06T10:13:03.754 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ remove_image_retry cluster2 mirror image_6 2024-04-06T10:13:03.754 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-04-06T10:13:03.754 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:13:03.754 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_6 2024-04-06T10:13:03.754 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 16 32 2024-04-06T10:13:03.754 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T10:13:03.755 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ remove_image cluster2 mirror image_6 2024-04-06T10:13:03.755 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-04-06T10:13:03.755 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:13:03.755 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_6 2024-04-06T10:13:03.755 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_6 2024-04-06T10:13:03.803 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster=cluster2 rm mirror/image_6 2024-04-06T10:13:04.056 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2024-04-06T10:13:04.060 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T10:13:04.061 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in `seq 1 10` 2024-04-06T10:13:04.061 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ image=image_7 2024-04-06T10:13:04.061 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ create_image cluster2 mirror image_7 512M 2024-04-06T10:13:04.061 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-04-06T10:13:04.061 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T10:13:04.061 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:13:04.061 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T10:13:04.061 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_7 2024-04-06T10:13:04.061 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T10:13:04.061 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local size=128 2024-04-06T10:13:04.061 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n 512M ']' 2024-04-06T10:13:04.061 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ size=512M 2024-04-06T10:13:04.061 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T10:13:04.062 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster2 create --size 512M --image-feature layering,exclusive-lock,journaling mirror/image_7 2024-04-06T10:13:04.120 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ enable_mirror cluster2 mirror image_7 2024-04-06T10:13:04.120 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-04-06T10:13:04.120 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:13:04.120 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_7 2024-04-06T10:13:04.120 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local mode=journal 2024-04-06T10:13:04.120 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_7 journal 2024-04-06T10:13:04.169 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Mirroring enabled 2024-04-06T10:13:04.176 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster=cluster2 info mirror/image_7 2024-04-06T10:13:04.214 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:rbd image 'image_7': 2024-04-06T10:13:04.215 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: size 512 MiB in 128 objects 2024-04-06T10:13:04.215 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: order 22 (4 MiB objects) 2024-04-06T10:13:04.215 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: snapshot_count: 0 2024-04-06T10:13:04.215 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: id: 47799cca4b7d 2024-04-06T10:13:04.215 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: block_name_prefix: rbd_data.47799cca4b7d 2024-04-06T10:13:04.215 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: format: 2 2024-04-06T10:13:04.215 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: features: layering, exclusive-lock, journaling 2024-04-06T10:13:04.215 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: op_features: 2024-04-06T10:13:04.215 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: flags: 2024-04-06T10:13:04.215 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: create_timestamp: Sat Apr 6 10:13:04 2024 2024-04-06T10:13:04.215 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: access_timestamp: Sat Apr 6 10:13:04 2024 2024-04-06T10:13:04.215 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: modify_timestamp: Sat Apr 6 10:13:04 2024 2024-04-06T10:13:04.216 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: journal: 47799cca4b7d 2024-04-06T10:13:04.216 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: mirroring state: enabled 2024-04-06T10:13:04.216 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: mirroring mode: journal 2024-04-06T10:13:04.216 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: mirroring global id: 46da49c6-1935-41b7-bf7d-c38720e47964 2024-04-06T10:13:04.216 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: mirroring primary: true 2024-04-06T10:13:04.219 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ stress_write_image cluster2 mirror image_7 2024-04-06T10:13:04.219 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-04-06T10:13:04.219 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:13:04.219 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_7 2024-04-06T10:13:04.219 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-04-06T10:13:04.221 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local duration=9 2024-04-06T10:13:04.221 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set +e 2024-04-06T10:13:04.221 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ timeout 9s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_7 --debug-rbd=20 --debug-journaler=20 2024-04-06T10:13:04.689 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.0 with skip (0.9788553186989161) > weight (0.25) 2024-04-06T10:13:04.689 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.1 with skip (0.7447012370207144) > weight (0.25) 2024-04-06T10:13:04.690 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.2 with skip (0.7968322606234072) > weight (0.25) 2024-04-06T10:13:04.690 INFO:tasks.rbd_mirror_thrash:kill cluster1.client.mirror.3 2024-04-06T10:13:04.690 INFO:tasks.rbd_mirror.cluster1.client.mirror.3:Sent signal 15 2024-04-06T10:13:04.690 INFO:tasks.rbd_mirror_thrash:waiting for 6 secs before reviving daemons 2024-04-06T10:13:04.691 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi192.stderr:2024-04-06T10:13:04.692+0000 7f9757f9d640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.3 (PID: 157489) UID: 1000 2024-04-06T10:13:06.118 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:100 IOs, 41722880 bytes 2024-04-06T10:13:10.691 INFO:tasks.rbd_mirror_thrash:waiting for cluster1.client.mirror.3 2024-04-06T10:13:10.692 INFO:teuthology.orchestra.run:waiting for 600 2024-04-06T10:13:10.692 INFO:tasks.rbd_mirror_thrash:reviving cluster1.client.mirror.3 2024-04-06T10:13:10.692 INFO:tasks.rbd_mirror.cluster1.client.mirror.3:Restarting daemon 2024-04-06T10:13:10.692 DEBUG:teuthology.orchestra.run.smithi192:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.3 2024-04-06T10:13:10.695 INFO:tasks.rbd_mirror.cluster1.client.mirror.3:Started 2024-04-06T10:13:10.695 INFO:tasks.rbd_mirror_thrash:waiting for 107 secs before thrashing 2024-04-06T10:13:10.891 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:200 IOs, 124625920 bytes 2024-04-06T10:13:13.111 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:300 IOs, 167943168 bytes 2024-04-06T10:13:13.234 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ error_code=124 2024-04-06T10:13:13.234 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set -e 2024-04-06T10:13:13.234 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 124 -eq 124 ']' 2024-04-06T10:13:13.234 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T10:13:13.234 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_image_present cluster1 mirror image_7 present 2024-04-06T10:13:13.235 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T10:13:13.235 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:13:13.235 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_7 2024-04-06T10:13:13.235 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state=present 2024-04-06T10:13:13.235 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image_id= 2024-04-06T10:13:13.235 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local s 2024-04-06T10:13:13.235 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n '' 2024-04-06T10:13:13.235 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_image_id cluster1 mirror image_7 2024-04-06T10:13:13.235 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T10:13:13.235 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:13:13.235 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_7 2024-04-06T10:13:13.235 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster=cluster1 info mirror/image_7 2024-04-06T10:13:13.235 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-04-06T10:13:13.304 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ image_id=10ac42cec0f4 2024-04-06T10:13:13.304 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-04-06T10:13:13.304 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.1 2024-04-06T10:13:13.406 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test_image_present cluster1 mirror image_7 present 10ac42cec0f4 2024-04-06T10:13:13.406 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T10:13:13.406 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:13:13.406 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_7 2024-04-06T10:13:13.406 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local test_state=present 2024-04-06T10:13:13.406 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image_id=10ac42cec0f4 2024-04-06T10:13:13.406 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local current_state=deleted 2024-04-06T10:13:13.406 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local current_image_id 2024-04-06T10:13:13.407 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_image_id cluster1 mirror image_7 2024-04-06T10:13:13.407 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T10:13:13.407 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:13:13.407 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_7 2024-04-06T10:13:13.407 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster=cluster1 info mirror/image_7 2024-04-06T10:13:13.407 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-04-06T10:13:13.476 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ current_image_id=10ac42cec0f4 2024-04-06T10:13:13.476 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 10ac42cec0f4 2024-04-06T10:13:13.476 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -z 10ac42cec0f4 -o 10ac42cec0f4 = 10ac42cec0f4 2024-04-06T10:13:13.476 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ current_state=present 2024-04-06T10:13:13.476 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test present = present 2024-04-06T10:13:13.477 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T10:13:13.477 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ disable_mirror cluster2 mirror image_7 2024-04-06T10:13:13.477 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-04-06T10:13:13.477 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:13:13.477 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_7 2024-04-06T10:13:13.477 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster=cluster2 mirror image disable mirror/image_7 2024-04-06T10:13:27.698 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.0 with skip (0.67952442180197) > weight (0.25) 2024-04-06T10:13:27.699 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.1 with skip (0.30737729531569713) > weight (0.25) 2024-04-06T10:13:27.699 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.2 with skip (0.29796753770478757) > weight (0.25) 2024-04-06T10:13:27.699 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.3 with skip (0.2808303279219) > weight (0.25) 2024-04-06T10:13:27.699 INFO:tasks.rbd_mirror_thrash:waiting for 115 secs before thrashing 2024-04-06T10:13:42.193 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Mirroring disabled 2024-04-06T10:13:42.205 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_image_present cluster1 mirror image_7 deleted 2024-04-06T10:13:42.205 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T10:13:42.205 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:13:42.205 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_7 2024-04-06T10:13:42.205 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state=deleted 2024-04-06T10:13:42.205 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image_id= 2024-04-06T10:13:42.205 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local s 2024-04-06T10:13:42.205 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n '' 2024-04-06T10:13:42.205 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_image_id cluster1 mirror image_7 2024-04-06T10:13:42.205 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T10:13:42.205 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:13:42.206 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_7 2024-04-06T10:13:42.206 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster=cluster1 info mirror/image_7 2024-04-06T10:13:42.206 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-04-06T10:13:42.241 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:rbd: error opening image image_7: (2) No such file or directory 2024-04-06T10:13:42.245 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ image_id= 2024-04-06T10:13:42.245 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-04-06T10:13:42.245 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.1 2024-04-06T10:13:42.346 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test_image_present cluster1 mirror image_7 deleted '' 2024-04-06T10:13:42.347 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T10:13:42.347 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:13:42.347 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_7 2024-04-06T10:13:42.347 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local test_state=deleted 2024-04-06T10:13:42.347 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image_id= 2024-04-06T10:13:42.347 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local current_state=deleted 2024-04-06T10:13:42.347 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local current_image_id 2024-04-06T10:13:42.348 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_image_id cluster1 mirror image_7 2024-04-06T10:13:42.348 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T10:13:42.348 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:13:42.348 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_7 2024-04-06T10:13:42.348 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster=cluster1 info mirror/image_7 2024-04-06T10:13:42.348 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-04-06T10:13:42.390 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:rbd: error opening image image_7: (2) No such file or directory 2024-04-06T10:13:42.395 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ current_image_id= 2024-04-06T10:13:42.395 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n '' 2024-04-06T10:13:42.395 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test deleted = deleted 2024-04-06T10:13:42.395 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T10:13:42.395 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ purge_snapshots cluster2 mirror image_7 2024-04-06T10:13:42.395 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-04-06T10:13:42.395 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:13:42.395 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_7 2024-04-06T10:13:42.395 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster2 snap purge mirror/image_7 2024-04-06T10:13:42.442 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ remove_image_retry cluster2 mirror image_7 2024-04-06T10:13:42.442 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-04-06T10:13:42.442 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:13:42.442 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_7 2024-04-06T10:13:42.443 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 16 32 2024-04-06T10:13:42.443 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T10:13:42.443 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ remove_image cluster2 mirror image_7 2024-04-06T10:13:42.444 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-04-06T10:13:42.444 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:13:42.444 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_7 2024-04-06T10:13:42.444 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_7 2024-04-06T10:13:42.489 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster=cluster2 rm mirror/image_7 2024-04-06T10:13:42.733 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2024-04-06T10:13:42.737 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T10:13:42.737 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in `seq 1 10` 2024-04-06T10:13:42.737 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ image=image_8 2024-04-06T10:13:42.738 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ create_image cluster2 mirror image_8 512M 2024-04-06T10:13:42.738 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-04-06T10:13:42.738 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T10:13:42.738 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:13:42.738 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T10:13:42.738 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_8 2024-04-06T10:13:42.738 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T10:13:42.738 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local size=128 2024-04-06T10:13:42.738 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n 512M ']' 2024-04-06T10:13:42.738 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ size=512M 2024-04-06T10:13:42.738 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T10:13:42.738 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster2 create --size 512M --image-feature layering,exclusive-lock,journaling mirror/image_8 2024-04-06T10:13:42.796 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ enable_mirror cluster2 mirror image_8 2024-04-06T10:13:42.796 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-04-06T10:13:42.796 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:13:42.796 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_8 2024-04-06T10:13:42.796 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local mode=journal 2024-04-06T10:13:42.796 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_8 journal 2024-04-06T10:13:42.845 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Mirroring enabled 2024-04-06T10:13:42.852 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster=cluster2 info mirror/image_8 2024-04-06T10:13:42.890 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:rbd image 'image_8': 2024-04-06T10:13:42.891 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: size 512 MiB in 128 objects 2024-04-06T10:13:42.891 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: order 22 (4 MiB objects) 2024-04-06T10:13:42.891 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: snapshot_count: 0 2024-04-06T10:13:42.891 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: id: 478f1bd5e4a 2024-04-06T10:13:42.891 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: block_name_prefix: rbd_data.478f1bd5e4a 2024-04-06T10:13:42.891 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: format: 2 2024-04-06T10:13:42.891 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: features: layering, exclusive-lock, journaling 2024-04-06T10:13:42.891 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: op_features: 2024-04-06T10:13:42.891 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: flags: 2024-04-06T10:13:42.891 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: create_timestamp: Sat Apr 6 10:13:42 2024 2024-04-06T10:13:42.891 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: access_timestamp: Sat Apr 6 10:13:42 2024 2024-04-06T10:13:42.891 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: modify_timestamp: Sat Apr 6 10:13:42 2024 2024-04-06T10:13:42.891 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: journal: 478f1bd5e4a 2024-04-06T10:13:42.891 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: mirroring state: enabled 2024-04-06T10:13:42.891 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: mirroring mode: journal 2024-04-06T10:13:42.892 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: mirroring global id: e770d3ff-f98b-4c5d-9d35-a6b80835182e 2024-04-06T10:13:42.892 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: mirroring primary: true 2024-04-06T10:13:42.894 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ stress_write_image cluster2 mirror image_8 2024-04-06T10:13:42.895 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-04-06T10:13:42.895 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:13:42.895 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_8 2024-04-06T10:13:42.895 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-04-06T10:13:42.897 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local duration=14 2024-04-06T10:13:42.897 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set +e 2024-04-06T10:13:42.897 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ timeout 14s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_8 --debug-rbd=20 --debug-journaler=20 2024-04-06T10:13:45.379 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:100 IOs, 37652480 bytes 2024-04-06T10:13:47.578 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:200 IOs, 67738624 bytes 2024-04-06T10:13:48.176 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:300 IOs, 68356096 bytes 2024-04-06T10:13:48.382 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:400 IOs, 68981760 bytes 2024-04-06T10:13:50.339 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:500 IOs, 99014656 bytes 2024-04-06T10:13:52.660 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:600 IOs, 135721984 bytes 2024-04-06T10:13:53.675 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:700 IOs, 148725760 bytes 2024-04-06T10:13:55.975 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:800 IOs, 181920768 bytes 2024-04-06T10:13:56.906 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ error_code=124 2024-04-06T10:13:56.906 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set -e 2024-04-06T10:13:56.906 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 124 -eq 124 ']' 2024-04-06T10:13:56.906 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T10:13:56.906 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_image_present cluster1 mirror image_8 present 2024-04-06T10:13:56.906 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T10:13:56.906 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:13:56.906 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_8 2024-04-06T10:13:56.906 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state=present 2024-04-06T10:13:56.906 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image_id= 2024-04-06T10:13:56.906 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local s 2024-04-06T10:13:56.907 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n '' 2024-04-06T10:13:56.907 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_image_id cluster1 mirror image_8 2024-04-06T10:13:56.907 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T10:13:56.907 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:13:56.907 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_8 2024-04-06T10:13:56.907 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster=cluster1 info mirror/image_8 2024-04-06T10:13:56.907 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-04-06T10:13:56.970 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ image_id=10aced8471f3 2024-04-06T10:13:56.970 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-04-06T10:13:56.970 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.1 2024-04-06T10:13:57.069 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test_image_present cluster1 mirror image_8 present 10aced8471f3 2024-04-06T10:13:57.069 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T10:13:57.070 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:13:57.070 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_8 2024-04-06T10:13:57.070 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local test_state=present 2024-04-06T10:13:57.070 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image_id=10aced8471f3 2024-04-06T10:13:57.070 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local current_state=deleted 2024-04-06T10:13:57.070 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local current_image_id 2024-04-06T10:13:57.071 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_image_id cluster1 mirror image_8 2024-04-06T10:13:57.071 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T10:13:57.071 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:13:57.071 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_8 2024-04-06T10:13:57.071 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster=cluster1 info mirror/image_8 2024-04-06T10:13:57.072 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-04-06T10:13:57.138 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ current_image_id=10aced8471f3 2024-04-06T10:13:57.139 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 10aced8471f3 2024-04-06T10:13:57.139 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -z 10aced8471f3 -o 10aced8471f3 = 10aced8471f3 2024-04-06T10:13:57.139 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ current_state=present 2024-04-06T10:13:57.139 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test present = present 2024-04-06T10:13:57.139 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T10:13:57.139 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ disable_mirror cluster2 mirror image_8 2024-04-06T10:13:57.139 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-04-06T10:13:57.139 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:13:57.139 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_8 2024-04-06T10:13:57.139 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster=cluster2 mirror image disable mirror/image_8 2024-04-06T10:14:25.333 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Mirroring disabled 2024-04-06T10:14:25.344 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_image_present cluster1 mirror image_8 deleted 2024-04-06T10:14:25.344 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T10:14:25.344 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:14:25.345 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_8 2024-04-06T10:14:25.345 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state=deleted 2024-04-06T10:14:25.345 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image_id= 2024-04-06T10:14:25.345 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local s 2024-04-06T10:14:25.345 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n '' 2024-04-06T10:14:25.346 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_image_id cluster1 mirror image_8 2024-04-06T10:14:25.346 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T10:14:25.346 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:14:25.346 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_8 2024-04-06T10:14:25.346 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster=cluster1 info mirror/image_8 2024-04-06T10:14:25.346 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-04-06T10:14:25.383 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:rbd: error opening image image_8: (2) No such file or directory 2024-04-06T10:14:25.388 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ image_id= 2024-04-06T10:14:25.388 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-04-06T10:14:25.388 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.1 2024-04-06T10:14:25.489 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test_image_present cluster1 mirror image_8 deleted '' 2024-04-06T10:14:25.489 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T10:14:25.489 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:14:25.489 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_8 2024-04-06T10:14:25.490 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local test_state=deleted 2024-04-06T10:14:25.490 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image_id= 2024-04-06T10:14:25.490 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local current_state=deleted 2024-04-06T10:14:25.490 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local current_image_id 2024-04-06T10:14:25.490 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_image_id cluster1 mirror image_8 2024-04-06T10:14:25.490 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T10:14:25.490 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:14:25.490 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_8 2024-04-06T10:14:25.490 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-04-06T10:14:25.490 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster=cluster1 info mirror/image_8 2024-04-06T10:14:25.534 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:rbd: error opening image image_8: (2) No such file or directory 2024-04-06T10:14:25.539 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ current_image_id= 2024-04-06T10:14:25.539 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n '' 2024-04-06T10:14:25.539 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test deleted = deleted 2024-04-06T10:14:25.539 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T10:14:25.539 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ purge_snapshots cluster2 mirror image_8 2024-04-06T10:14:25.539 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-04-06T10:14:25.539 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:14:25.539 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_8 2024-04-06T10:14:25.539 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster2 snap purge mirror/image_8 2024-04-06T10:14:25.586 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ remove_image_retry cluster2 mirror image_8 2024-04-06T10:14:25.586 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-04-06T10:14:25.586 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:14:25.586 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_8 2024-04-06T10:14:25.586 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 16 32 2024-04-06T10:14:25.587 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T10:14:25.587 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ remove_image cluster2 mirror image_8 2024-04-06T10:14:25.587 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-04-06T10:14:25.587 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:14:25.587 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_8 2024-04-06T10:14:25.588 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_8 2024-04-06T10:14:25.635 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster=cluster2 rm mirror/image_8 2024-04-06T10:14:25.892 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2024-04-06T10:14:25.896 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T10:14:25.896 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in `seq 1 10` 2024-04-06T10:14:25.896 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ image=image_9 2024-04-06T10:14:25.896 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ create_image cluster2 mirror image_9 512M 2024-04-06T10:14:25.897 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-04-06T10:14:25.897 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T10:14:25.897 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:14:25.897 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T10:14:25.897 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_9 2024-04-06T10:14:25.897 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T10:14:25.897 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local size=128 2024-04-06T10:14:25.897 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n 512M ']' 2024-04-06T10:14:25.897 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ size=512M 2024-04-06T10:14:25.897 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T10:14:25.897 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster2 create --size 512M --image-feature layering,exclusive-lock,journaling mirror/image_9 2024-04-06T10:14:25.956 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ enable_mirror cluster2 mirror image_9 2024-04-06T10:14:25.956 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-04-06T10:14:25.957 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:14:25.957 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_9 2024-04-06T10:14:25.957 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local mode=journal 2024-04-06T10:14:25.957 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_9 journal 2024-04-06T10:14:26.006 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Mirroring enabled 2024-04-06T10:14:26.013 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster=cluster2 info mirror/image_9 2024-04-06T10:14:26.258 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:rbd image 'image_9': 2024-04-06T10:14:26.258 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: size 512 MiB in 128 objects 2024-04-06T10:14:26.258 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: order 22 (4 MiB objects) 2024-04-06T10:14:26.259 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: snapshot_count: 0 2024-04-06T10:14:26.259 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: id: 479f841ec97f 2024-04-06T10:14:26.259 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: block_name_prefix: rbd_data.479f841ec97f 2024-04-06T10:14:26.259 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: format: 2 2024-04-06T10:14:26.259 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: features: layering, exclusive-lock, journaling 2024-04-06T10:14:26.259 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: op_features: 2024-04-06T10:14:26.259 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: flags: 2024-04-06T10:14:26.259 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: create_timestamp: Sat Apr 6 10:14:25 2024 2024-04-06T10:14:26.259 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: access_timestamp: Sat Apr 6 10:14:25 2024 2024-04-06T10:14:26.259 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: modify_timestamp: Sat Apr 6 10:14:25 2024 2024-04-06T10:14:26.259 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: journal: 479f841ec97f 2024-04-06T10:14:26.260 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: mirroring state: enabled 2024-04-06T10:14:26.260 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: mirroring mode: journal 2024-04-06T10:14:26.260 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: mirroring global id: 8f59b629-f2bd-4087-8198-05fe72891548 2024-04-06T10:14:26.260 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: mirroring primary: true 2024-04-06T10:14:26.263 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ stress_write_image cluster2 mirror image_9 2024-04-06T10:14:26.263 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-04-06T10:14:26.263 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:14:26.263 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_9 2024-04-06T10:14:26.264 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-04-06T10:14:26.265 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local duration=9 2024-04-06T10:14:26.265 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set +e 2024-04-06T10:14:26.266 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ timeout 9s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_9 --debug-rbd=20 --debug-journaler=20 2024-04-06T10:14:26.369 INFO:tasks.ceph.cluster2.osd.2.smithi192.stderr:2024-04-06T10:14:26.371+0000 7fa534aba640 -1 reset not still connected to 0x559365721790 2024-04-06T10:14:28.247 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:100 IOs, 27482112 bytes 2024-04-06T10:14:31.347 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:200 IOs, 89996288 bytes 2024-04-06T10:14:33.305 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi192.stderr:2024-04-06T10:14:33.307+0000 7f3d293cf640 -1 rbd::mirror::InstanceWatcher: C_NotifyInstanceRequest: 0x56066bd9acc0 finish: resending after timeout 2024-04-06T10:14:35.276 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ error_code=124 2024-04-06T10:14:35.276 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set -e 2024-04-06T10:14:35.277 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 124 -eq 124 ']' 2024-04-06T10:14:35.277 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T10:14:35.277 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_image_present cluster1 mirror image_9 present 2024-04-06T10:14:35.277 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T10:14:35.277 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:14:35.277 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_9 2024-04-06T10:14:35.277 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state=present 2024-04-06T10:14:35.277 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image_id= 2024-04-06T10:14:35.277 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local s 2024-04-06T10:14:35.277 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n '' 2024-04-06T10:14:35.277 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_image_id cluster1 mirror image_9 2024-04-06T10:14:35.277 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T10:14:35.277 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:14:35.277 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_9 2024-04-06T10:14:35.277 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster=cluster1 info mirror/image_9 2024-04-06T10:14:35.277 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-04-06T10:14:35.336 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ image_id=10ac6e658b63 2024-04-06T10:14:35.336 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-04-06T10:14:35.336 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.1 2024-04-06T10:14:35.438 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test_image_present cluster1 mirror image_9 present 10ac6e658b63 2024-04-06T10:14:35.438 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T10:14:35.438 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:14:35.438 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_9 2024-04-06T10:14:35.438 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local test_state=present 2024-04-06T10:14:35.438 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image_id=10ac6e658b63 2024-04-06T10:14:35.438 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local current_state=deleted 2024-04-06T10:14:35.438 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local current_image_id 2024-04-06T10:14:35.438 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_image_id cluster1 mirror image_9 2024-04-06T10:14:35.439 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T10:14:35.439 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:14:35.439 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_9 2024-04-06T10:14:35.439 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster=cluster1 info mirror/image_9 2024-04-06T10:14:35.439 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-04-06T10:14:35.505 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ current_image_id=10ac6e658b63 2024-04-06T10:14:35.505 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 10ac6e658b63 2024-04-06T10:14:35.505 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -z 10ac6e658b63 -o 10ac6e658b63 = 10ac6e658b63 2024-04-06T10:14:35.505 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ current_state=present 2024-04-06T10:14:35.505 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test present = present 2024-04-06T10:14:35.506 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T10:14:35.506 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ disable_mirror cluster2 mirror image_9 2024-04-06T10:14:35.506 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-04-06T10:14:35.506 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:14:35.506 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_9 2024-04-06T10:14:35.506 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster=cluster2 mirror image disable mirror/image_9 2024-04-06T10:14:57.696 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.0 with skip (0.4083714007856112) > weight (0.25) 2024-04-06T10:14:57.696 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.1 with skip (0.6814886501852916) > weight (0.25) 2024-04-06T10:14:57.696 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.2 with skip (0.8371075532297172) > weight (0.25) 2024-04-06T10:14:57.696 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.3 with skip (0.7807528050028433) > weight (0.25) 2024-04-06T10:14:57.696 INFO:tasks.rbd_mirror_thrash:waiting for 76 secs before thrashing 2024-04-06T10:15:05.232 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Mirroring disabled 2024-04-06T10:15:05.242 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_image_present cluster1 mirror image_9 deleted 2024-04-06T10:15:05.242 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T10:15:05.242 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:15:05.242 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_9 2024-04-06T10:15:05.242 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state=deleted 2024-04-06T10:15:05.242 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image_id= 2024-04-06T10:15:05.242 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local s 2024-04-06T10:15:05.242 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n '' 2024-04-06T10:15:05.242 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_image_id cluster1 mirror image_9 2024-04-06T10:15:05.242 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T10:15:05.242 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:15:05.243 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_9 2024-04-06T10:15:05.243 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster=cluster1 info mirror/image_9 2024-04-06T10:15:05.243 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-04-06T10:15:05.279 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:rbd: error opening image image_9: (2) No such file or directory 2024-04-06T10:15:05.282 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ image_id= 2024-04-06T10:15:05.282 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-04-06T10:15:05.282 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.1 2024-04-06T10:15:05.384 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test_image_present cluster1 mirror image_9 deleted '' 2024-04-06T10:15:05.384 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T10:15:05.384 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:15:05.384 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_9 2024-04-06T10:15:05.384 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local test_state=deleted 2024-04-06T10:15:05.384 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image_id= 2024-04-06T10:15:05.384 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local current_state=deleted 2024-04-06T10:15:05.384 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local current_image_id 2024-04-06T10:15:05.385 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_image_id cluster1 mirror image_9 2024-04-06T10:15:05.385 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T10:15:05.385 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:15:05.385 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_9 2024-04-06T10:15:05.385 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster=cluster1 info mirror/image_9 2024-04-06T10:15:05.385 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-04-06T10:15:05.430 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:rbd: error opening image image_9: (2) No such file or directory 2024-04-06T10:15:05.435 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ current_image_id= 2024-04-06T10:15:05.435 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n '' 2024-04-06T10:15:05.435 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test deleted = deleted 2024-04-06T10:15:05.435 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T10:15:05.435 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ purge_snapshots cluster2 mirror image_9 2024-04-06T10:15:05.435 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-04-06T10:15:05.436 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:15:05.436 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_9 2024-04-06T10:15:05.436 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster2 snap purge mirror/image_9 2024-04-06T10:15:05.482 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ remove_image_retry cluster2 mirror image_9 2024-04-06T10:15:05.483 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-04-06T10:15:05.483 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:15:05.483 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_9 2024-04-06T10:15:05.483 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 16 32 2024-04-06T10:15:05.483 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T10:15:05.484 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ remove_image cluster2 mirror image_9 2024-04-06T10:15:05.484 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-04-06T10:15:05.484 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:15:05.484 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_9 2024-04-06T10:15:05.484 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_9 2024-04-06T10:15:05.533 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster=cluster2 rm mirror/image_9 2024-04-06T10:15:05.776 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2024-04-06T10:15:05.781 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T10:15:05.781 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in `seq 1 10` 2024-04-06T10:15:05.781 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ image=image_10 2024-04-06T10:15:05.781 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ create_image cluster2 mirror image_10 512M 2024-04-06T10:15:05.781 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-04-06T10:15:05.781 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T10:15:05.781 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:15:05.782 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T10:15:05.782 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_10 2024-04-06T10:15:05.782 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T10:15:05.782 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local size=128 2024-04-06T10:15:05.782 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n 512M ']' 2024-04-06T10:15:05.782 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ size=512M 2024-04-06T10:15:05.782 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-04-06T10:15:05.782 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster2 create --size 512M --image-feature layering,exclusive-lock,journaling mirror/image_10 2024-04-06T10:15:05.842 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ enable_mirror cluster2 mirror image_10 2024-04-06T10:15:05.842 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-04-06T10:15:05.842 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:15:05.842 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_10 2024-04-06T10:15:05.843 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local mode=journal 2024-04-06T10:15:05.843 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_10 journal 2024-04-06T10:15:05.891 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Mirroring enabled 2024-04-06T10:15:05.898 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster=cluster2 info mirror/image_10 2024-04-06T10:15:06.143 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:rbd image 'image_10': 2024-04-06T10:15:06.143 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: size 512 MiB in 128 objects 2024-04-06T10:15:06.143 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: order 22 (4 MiB objects) 2024-04-06T10:15:06.143 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: snapshot_count: 0 2024-04-06T10:15:06.143 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: id: 47b08812588d 2024-04-06T10:15:06.143 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: block_name_prefix: rbd_data.47b08812588d 2024-04-06T10:15:06.143 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: format: 2 2024-04-06T10:15:06.143 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: features: layering, exclusive-lock, journaling 2024-04-06T10:15:06.143 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: op_features: 2024-04-06T10:15:06.143 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: flags: 2024-04-06T10:15:06.143 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: create_timestamp: Sat Apr 6 10:15:05 2024 2024-04-06T10:15:06.144 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: access_timestamp: Sat Apr 6 10:15:05 2024 2024-04-06T10:15:06.144 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: modify_timestamp: Sat Apr 6 10:15:05 2024 2024-04-06T10:15:06.144 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: journal: 47b08812588d 2024-04-06T10:15:06.144 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: mirroring state: enabled 2024-04-06T10:15:06.144 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: mirroring mode: journal 2024-04-06T10:15:06.144 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: mirroring global id: 0c6a663f-2c11-4f91-9c80-41c26240c1e5 2024-04-06T10:15:06.144 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: mirroring primary: true 2024-04-06T10:15:06.149 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ stress_write_image cluster2 mirror image_10 2024-04-06T10:15:06.149 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-04-06T10:15:06.149 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:15:06.149 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_10 2024-04-06T10:15:06.149 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-04-06T10:15:06.152 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local duration=12 2024-04-06T10:15:06.152 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set +e 2024-04-06T10:15:06.152 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ timeout 12s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_10 --debug-rbd=20 --debug-journaler=20 2024-04-06T10:15:10.658 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:100 IOs, 104873984 bytes 2024-04-06T10:15:13.012 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:200 IOs, 136347648 bytes 2024-04-06T10:15:17.350 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:300 IOs, 211935232 bytes 2024-04-06T10:15:18.163 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ error_code=124 2024-04-06T10:15:18.163 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set -e 2024-04-06T10:15:18.163 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 124 -eq 124 ']' 2024-04-06T10:15:18.164 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T10:15:18.164 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_image_present cluster1 mirror image_10 present 2024-04-06T10:15:18.164 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T10:15:18.164 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:15:18.164 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_10 2024-04-06T10:15:18.164 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state=present 2024-04-06T10:15:18.164 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image_id= 2024-04-06T10:15:18.164 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local s 2024-04-06T10:15:18.164 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n '' 2024-04-06T10:15:18.164 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_image_id cluster1 mirror image_10 2024-04-06T10:15:18.164 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T10:15:18.164 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:15:18.164 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_10 2024-04-06T10:15:18.165 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster=cluster1 info mirror/image_10 2024-04-06T10:15:18.165 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-04-06T10:15:18.226 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ image_id=10ac9fb8db07 2024-04-06T10:15:18.227 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-04-06T10:15:18.227 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.1 2024-04-06T10:15:18.328 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test_image_present cluster1 mirror image_10 present 10ac9fb8db07 2024-04-06T10:15:18.328 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T10:15:18.328 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:15:18.328 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_10 2024-04-06T10:15:18.328 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local test_state=present 2024-04-06T10:15:18.328 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image_id=10ac9fb8db07 2024-04-06T10:15:18.328 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local current_state=deleted 2024-04-06T10:15:18.329 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local current_image_id 2024-04-06T10:15:18.329 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_image_id cluster1 mirror image_10 2024-04-06T10:15:18.329 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T10:15:18.329 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:15:18.329 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_10 2024-04-06T10:15:18.329 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster=cluster1 info mirror/image_10 2024-04-06T10:15:18.329 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-04-06T10:15:18.400 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ current_image_id=10ac9fb8db07 2024-04-06T10:15:18.401 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 10ac9fb8db07 2024-04-06T10:15:18.401 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -z 10ac9fb8db07 -o 10ac9fb8db07 = 10ac9fb8db07 2024-04-06T10:15:18.401 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ current_state=present 2024-04-06T10:15:18.401 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test present = present 2024-04-06T10:15:18.401 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T10:15:18.401 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ disable_mirror cluster2 mirror image_10 2024-04-06T10:15:18.401 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-04-06T10:15:18.401 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:15:18.401 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_10 2024-04-06T10:15:18.401 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster=cluster2 mirror image disable mirror/image_10 2024-04-06T10:15:22.700 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.0 with skip (0.770199735846351) > weight (0.25) 2024-04-06T10:15:22.700 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.1 with skip (0.945274362950379) > weight (0.25) 2024-04-06T10:15:22.700 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.2 with skip (0.37464363599296024) > weight (0.25) 2024-04-06T10:15:22.700 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.3 with skip (0.2678803467546311) > weight (0.25) 2024-04-06T10:15:22.700 INFO:tasks.rbd_mirror_thrash:waiting for 108 secs before thrashing 2024-04-06T10:15:47.269 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Mirroring disabled 2024-04-06T10:15:47.277 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_image_present cluster1 mirror image_10 deleted 2024-04-06T10:15:47.278 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T10:15:47.278 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:15:47.278 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_10 2024-04-06T10:15:47.278 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state=deleted 2024-04-06T10:15:47.278 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image_id= 2024-04-06T10:15:47.278 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local s 2024-04-06T10:15:47.278 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n '' 2024-04-06T10:15:47.278 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_image_id cluster1 mirror image_10 2024-04-06T10:15:47.278 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T10:15:47.278 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:15:47.278 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_10 2024-04-06T10:15:47.279 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster=cluster1 info mirror/image_10 2024-04-06T10:15:47.279 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-04-06T10:15:47.315 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:rbd: error opening image image_10: (2) No such file or directory 2024-04-06T10:15:47.319 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ image_id= 2024-04-06T10:15:47.319 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-04-06T10:15:47.319 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0.1 2024-04-06T10:15:47.420 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test_image_present cluster1 mirror image_10 deleted '' 2024-04-06T10:15:47.420 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T10:15:47.420 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:15:47.420 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_10 2024-04-06T10:15:47.420 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local test_state=deleted 2024-04-06T10:15:47.420 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image_id= 2024-04-06T10:15:47.420 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local current_state=deleted 2024-04-06T10:15:47.421 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local current_image_id 2024-04-06T10:15:47.421 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ get_image_id cluster1 mirror image_10 2024-04-06T10:15:47.421 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T10:15:47.421 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:15:47.421 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local image=image_10 2024-04-06T10:15:47.421 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-04-06T10:15:47.422 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster=cluster1 info mirror/image_10 2024-04-06T10:15:47.469 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:rbd: error opening image image_10: (2) No such file or directory 2024-04-06T10:15:47.473 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ current_image_id= 2024-04-06T10:15:47.473 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n '' 2024-04-06T10:15:47.473 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test deleted = deleted 2024-04-06T10:15:47.473 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T10:15:47.474 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ purge_snapshots cluster2 mirror image_10 2024-04-06T10:15:47.474 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-04-06T10:15:47.474 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:15:47.474 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_10 2024-04-06T10:15:47.474 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster2 snap purge mirror/image_10 2024-04-06T10:15:47.522 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ remove_image_retry cluster2 mirror image_10 2024-04-06T10:15:47.523 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-04-06T10:15:47.523 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:15:47.523 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_10 2024-04-06T10:15:47.523 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 4 8 16 32 2024-04-06T10:15:47.523 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T10:15:47.523 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ remove_image cluster2 mirror image_10 2024-04-06T10:15:47.524 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-04-06T10:15:47.524 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:15:47.524 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image_10 2024-04-06T10:15:47.524 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_10 2024-04-06T10:15:47.570 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster=cluster2 rm mirror/image_10 2024-04-06T10:15:47.825 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2024-04-06T10:15:47.829 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T10:15:47.829 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ testlog 'TEST: check if removed images'\'' OMAP are removed' 2024-04-06T10:15:47.830 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ tee -a /tmp/tmp.BhW6xKmROU/rbd-mirror.test.log 2024-04-06T10:15:47.830 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ date '+%F %T' 2024-04-06T10:15:47.833 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ echo 2024-04-06 10:15:47 TEST: check if removed 'images'\''' OMAP are removed 2024-04-06T10:15:47.833 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:2024-04-06 10:15:47 TEST: check if removed images' OMAP are removed 2024-04-06T10:15:47.833 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_image_in_omap cluster1 mirror 2024-04-06T10:15:47.833 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T10:15:47.833 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:15:47.833 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_omap_keys cluster1 mirror rbd_mirroring status_global 2024-04-06T10:15:47.833 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T10:15:47.833 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:15:47.833 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local obj_name=rbd_mirroring 2024-04-06T10:15:47.834 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local filter=status_global 2024-04-06T10:15:47.834 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 2 4 4 8 8 8 16 16 32 2024-04-06T10:15:47.834 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T10:15:47.834 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set +e 2024-04-06T10:15:47.834 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ count_omap_keys_with_filter cluster1 mirror rbd_mirroring status_global 2024-04-06T10:15:47.834 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T10:15:47.834 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:15:47.834 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local obj_name=rbd_mirroring 2024-04-06T10:15:47.834 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local filter=status_global 2024-04-06T10:15:47.834 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ list_omap_keys cluster1 mirror rbd_mirroring 2024-04-06T10:15:47.834 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T10:15:47.834 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:15:47.834 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local obj_name=rbd_mirroring 2024-04-06T10:15:47.835 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rados --cluster cluster1 -p mirror listomapkeys rbd_mirroring 2024-04-06T10:15:47.835 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -c status_global 2024-04-06T10:15:48.019 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 0 = 0 2024-04-06T10:15:48.019 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ error_code=0 2024-04-06T10:15:48.019 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set -e 2024-04-06T10:15:48.020 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 -eq 0 ']' 2024-04-06T10:15:48.020 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T10:15:48.020 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_omap_keys cluster1 mirror rbd_mirroring image_ 2024-04-06T10:15:48.020 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T10:15:48.020 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:15:48.020 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local obj_name=rbd_mirroring 2024-04-06T10:15:48.020 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local filter=image_ 2024-04-06T10:15:48.020 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 2 4 4 8 8 8 16 16 32 2024-04-06T10:15:48.020 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T10:15:48.021 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set +e 2024-04-06T10:15:48.021 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ count_omap_keys_with_filter cluster1 mirror rbd_mirroring image_ 2024-04-06T10:15:48.021 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T10:15:48.021 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:15:48.021 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local obj_name=rbd_mirroring 2024-04-06T10:15:48.021 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local filter=image_ 2024-04-06T10:15:48.021 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ list_omap_keys cluster1 mirror rbd_mirroring 2024-04-06T10:15:48.022 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T10:15:48.022 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:15:48.022 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local obj_name=rbd_mirroring 2024-04-06T10:15:48.022 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rados --cluster cluster1 -p mirror listomapkeys rbd_mirroring 2024-04-06T10:15:48.022 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -c image_ 2024-04-06T10:15:48.065 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 0 = 0 2024-04-06T10:15:48.065 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ error_code=0 2024-04-06T10:15:48.065 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set -e 2024-04-06T10:15:48.065 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 -eq 0 ']' 2024-04-06T10:15:48.065 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T10:15:48.065 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_omap_keys cluster1 mirror rbd_mirror_leader image_map 2024-04-06T10:15:48.065 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T10:15:48.066 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:15:48.066 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local obj_name=rbd_mirror_leader 2024-04-06T10:15:48.066 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local filter=image_map 2024-04-06T10:15:48.066 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 2 4 4 8 8 8 16 16 32 2024-04-06T10:15:48.066 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T10:15:48.067 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set +e 2024-04-06T10:15:48.067 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ count_omap_keys_with_filter cluster1 mirror rbd_mirror_leader image_map 2024-04-06T10:15:48.067 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T10:15:48.067 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:15:48.067 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local obj_name=rbd_mirror_leader 2024-04-06T10:15:48.067 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local filter=image_map 2024-04-06T10:15:48.067 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ list_omap_keys cluster1 mirror rbd_mirror_leader 2024-04-06T10:15:48.067 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-04-06T10:15:48.067 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:15:48.068 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local obj_name=rbd_mirror_leader 2024-04-06T10:15:48.068 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rados --cluster cluster1 -p mirror listomapkeys rbd_mirror_leader 2024-04-06T10:15:48.068 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -c image_map 2024-04-06T10:15:48.108 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 0 = 0 2024-04-06T10:15:48.109 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ error_code=0 2024-04-06T10:15:48.109 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set -e 2024-04-06T10:15:48.109 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 -eq 0 ']' 2024-04-06T10:15:48.109 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T10:15:48.109 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_image_in_omap cluster2 mirror 2024-04-06T10:15:48.109 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-04-06T10:15:48.109 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:15:48.109 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_omap_keys cluster2 mirror rbd_mirroring status_global 2024-04-06T10:15:48.109 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-04-06T10:15:48.109 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:15:48.109 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local obj_name=rbd_mirroring 2024-04-06T10:15:48.109 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local filter=status_global 2024-04-06T10:15:48.109 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 2 4 4 8 8 8 16 16 32 2024-04-06T10:15:48.109 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T10:15:48.110 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set +e 2024-04-06T10:15:48.111 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ count_omap_keys_with_filter cluster2 mirror rbd_mirroring status_global 2024-04-06T10:15:48.111 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:15:48.111 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:15:48.111 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local obj_name=rbd_mirroring 2024-04-06T10:15:48.111 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local filter=status_global 2024-04-06T10:15:48.111 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ list_omap_keys cluster2 mirror rbd_mirroring 2024-04-06T10:15:48.111 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:15:48.111 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:15:48.111 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local obj_name=rbd_mirroring 2024-04-06T10:15:48.111 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rados --cluster cluster2 -p mirror listomapkeys rbd_mirroring 2024-04-06T10:15:48.111 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -c status_global 2024-04-06T10:15:48.147 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 0 = 0 2024-04-06T10:15:48.147 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ error_code=0 2024-04-06T10:15:48.147 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set -e 2024-04-06T10:15:48.147 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 -eq 0 ']' 2024-04-06T10:15:48.147 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T10:15:48.147 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_omap_keys cluster2 mirror rbd_mirroring image_ 2024-04-06T10:15:48.147 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-04-06T10:15:48.147 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:15:48.147 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local obj_name=rbd_mirroring 2024-04-06T10:15:48.147 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local filter=image_ 2024-04-06T10:15:48.148 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 2 4 4 8 8 8 16 16 32 2024-04-06T10:15:48.148 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T10:15:48.148 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set +e 2024-04-06T10:15:48.149 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ count_omap_keys_with_filter cluster2 mirror rbd_mirroring image_ 2024-04-06T10:15:48.149 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:15:48.149 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:15:48.149 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local obj_name=rbd_mirroring 2024-04-06T10:15:48.149 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local filter=image_ 2024-04-06T10:15:48.149 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ list_omap_keys cluster2 mirror rbd_mirroring 2024-04-06T10:15:48.149 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:15:48.149 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:15:48.149 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local obj_name=rbd_mirroring 2024-04-06T10:15:48.149 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rados --cluster cluster2 -p mirror listomapkeys rbd_mirroring 2024-04-06T10:15:48.149 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -c image_ 2024-04-06T10:15:48.184 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 0 = 0 2024-04-06T10:15:48.184 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ error_code=0 2024-04-06T10:15:48.184 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set -e 2024-04-06T10:15:48.185 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 -eq 0 ']' 2024-04-06T10:15:48.185 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T10:15:48.185 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_omap_keys cluster2 mirror rbd_mirror_leader image_map 2024-04-06T10:15:48.185 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-04-06T10:15:48.185 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-04-06T10:15:48.185 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local obj_name=rbd_mirror_leader 2024-04-06T10:15:48.185 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local filter=image_map 2024-04-06T10:15:48.185 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 0 1 2 2 4 4 8 8 8 16 16 32 2024-04-06T10:15:48.185 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 0 2024-04-06T10:15:48.186 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set +e 2024-04-06T10:15:48.186 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ count_omap_keys_with_filter cluster2 mirror rbd_mirror_leader image_map 2024-04-06T10:15:48.186 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:15:48.187 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:15:48.187 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local obj_name=rbd_mirror_leader 2024-04-06T10:15:48.187 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local filter=image_map 2024-04-06T10:15:48.187 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ list_omap_keys cluster2 mirror rbd_mirror_leader 2024-04-06T10:15:48.187 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-04-06T10:15:48.187 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local pool=mirror 2024-04-06T10:15:48.187 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local obj_name=rbd_mirror_leader 2024-04-06T10:15:48.187 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rados --cluster cluster2 -p mirror listomapkeys rbd_mirror_leader 2024-04-06T10:15:48.187 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ grep -c image_map 2024-04-06T10:15:48.221 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 0 = 0 2024-04-06T10:15:48.222 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ error_code=0 2024-04-06T10:15:48.222 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set -e 2024-04-06T10:15:48.222 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 -eq 0 ']' 2024-04-06T10:15:48.222 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-04-06T10:15:48.222 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cleanup 0 2024-04-06T10:15:48.222 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local error_code=0 2024-04-06T10:15:48.222 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set +e 2024-04-06T10:15:48.222 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 -ne 0 ']' 2024-04-06T10:15:48.222 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -z '' ']' 2024-04-06T10:15:48.222 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster instance 2024-04-06T10:15:48.222 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ CEPH_ARGS= 2024-04-06T10:15:48.222 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --cluster cluster1 osd pool rm mirror mirror --yes-i-really-really-mean-it 2024-04-06T10:15:49.479 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:pool 'mirror' does not exist 2024-04-06T10:15:49.497 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ CEPH_ARGS= 2024-04-06T10:15:49.497 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --cluster cluster2 osd pool rm mirror mirror --yes-i-really-really-mean-it 2024-04-06T10:15:50.013 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:pool 'mirror' does not exist 2024-04-06T10:15:50.032 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ CEPH_ARGS= 2024-04-06T10:15:50.032 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --cluster cluster1 osd pool rm mirror_parent mirror_parent --yes-i-really-really-mean-it 2024-04-06T10:15:50.479 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:pool 'mirror_parent' does not exist 2024-04-06T10:15:50.496 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ CEPH_ARGS= 2024-04-06T10:15:50.496 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --cluster cluster2 osd pool rm mirror_parent mirror_parent --yes-i-really-really-mean-it 2024-04-06T10:15:51.002 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:pool 'mirror_parent' does not exist 2024-04-06T10:15:51.025 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for cluster in "${CLUSTER1}" "${CLUSTER2}" 2024-04-06T10:15:51.025 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ stop_mirrors cluster1 2024-04-06T10:15:51.025 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-04-06T10:15:51.025 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local sig= 2024-04-06T10:15:51.025 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T10:15:51.026 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T10:15:51.027 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ stop_mirror cluster1:0 '' 2024-04-06T10:15:51.027 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-04-06T10:15:51.027 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local sig= 2024-04-06T10:15:51.027 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 1 2024-04-06T10:15:51.027 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 2024-04-06T10:15:51.027 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T10:15:51.027 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ stop_mirror cluster1:1 '' 2024-04-06T10:15:51.027 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:1 2024-04-06T10:15:51.027 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local sig= 2024-04-06T10:15:51.028 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 1 2024-04-06T10:15:51.028 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 2024-04-06T10:15:51.028 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T10:15:51.028 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ stop_mirror cluster1:2 '' 2024-04-06T10:15:51.028 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:2 2024-04-06T10:15:51.028 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local sig= 2024-04-06T10:15:51.028 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 1 2024-04-06T10:15:51.028 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 2024-04-06T10:15:51.028 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T10:15:51.028 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ stop_mirror cluster1:3 '' 2024-04-06T10:15:51.028 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:3 2024-04-06T10:15:51.028 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local sig= 2024-04-06T10:15:51.028 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 1 2024-04-06T10:15:51.028 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 2024-04-06T10:15:51.028 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for cluster in "${CLUSTER1}" "${CLUSTER2}" 2024-04-06T10:15:51.029 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ stop_mirrors cluster2 2024-04-06T10:15:51.029 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-04-06T10:15:51.029 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local sig= 2024-04-06T10:15:51.029 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 3 2024-04-06T10:15:51.029 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T10:15:51.029 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ stop_mirror cluster2:0 '' 2024-04-06T10:15:51.029 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2:0 2024-04-06T10:15:51.029 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local sig= 2024-04-06T10:15:51.029 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 1 2024-04-06T10:15:51.029 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 2024-04-06T10:15:51.030 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T10:15:51.030 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ stop_mirror cluster2:1 '' 2024-04-06T10:15:51.030 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2:1 2024-04-06T10:15:51.030 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local sig= 2024-04-06T10:15:51.030 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 1 2024-04-06T10:15:51.030 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 2024-04-06T10:15:51.030 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T10:15:51.030 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ stop_mirror cluster2:2 '' 2024-04-06T10:15:51.030 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2:2 2024-04-06T10:15:51.030 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local sig= 2024-04-06T10:15:51.030 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 1 2024-04-06T10:15:51.030 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 2024-04-06T10:15:51.030 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T10:15:51.030 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ stop_mirror cluster2:3 '' 2024-04-06T10:15:51.030 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2:3 2024-04-06T10:15:51.031 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local sig= 2024-04-06T10:15:51.031 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n 1 2024-04-06T10:15:51.031 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 2024-04-06T10:15:51.031 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -z 1 ']' 2024-04-06T10:15:51.031 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test '' = /tmp/tmp.BhW6xKmROU 2024-04-06T10:15:51.031 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rm -Rf /tmp/tmp.BhW6xKmROU 2024-04-06T10:15:51.088 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi192.stderr:2024-04-06T10:15:51.090+0000 7f3d293cf640 -1 rbd::mirror::LeaderWatcher: 0x560664c4f500 handle_notify_heartbeat: error notifying heartbeat: (2) No such file or directory, releasing leader 2024-04-06T10:15:51.088 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi192.stderr:2024-04-06T10:15:51.090+0000 7f3d14ba6640 -1 rbd::mirror::image_deleter::TrashWatcher: 0x560664eabb00 handle_unregister_watcher: error unregistering watcher for trash directory: (2) No such file or directory 2024-04-06T10:15:51.089 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi192.stderr:2024-04-06T10:15:51.090+0000 7f3d14ba6640 -1 rbd::mirror::PoolWatcher: 0x56066aaeb2c0 operator(): error unregistering watcher for rbd_mirroring object: (2) No such file or directory 2024-04-06T10:15:51.115 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 -eq 0 ']' 2024-04-06T10:15:51.115 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ echo OK 2024-04-06T10:15:51.115 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ exit 0 2024-04-06T10:15:51.116 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:OK 2024-04-06T10:15:51.116 INFO:teuthology.orchestra.run:Running command with timeout 3600 2024-04-06T10:15:51.116 DEBUG:teuthology.orchestra.run.smithi192:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp 2024-04-06T10:15:51.304 INFO:tasks.workunit:Stopping ['rbd/rbd_mirror_stress.sh'] on cluster1.client.mirror... 2024-04-06T10:15:51.304 DEBUG:teuthology.orchestra.run.smithi192:> sudo rm -rf -- /home/ubuntu/cephtest/workunits.list.cluster1.client.mirror /home/ubuntu/cephtest/clone.cluster1.client.mirror 2024-04-06T10:15:51.585 DEBUG:teuthology.parallel:result is None 2024-04-06T10:15:51.586 DEBUG:teuthology.orchestra.run.smithi192:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror 2024-04-06T10:15:51.597 INFO:tasks.workunit:Deleted dir /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror 2024-04-06T10:15:51.597 DEBUG:teuthology.orchestra.run.smithi192:> rmdir -- /home/ubuntu/cephtest/mnt.cluster1.mirror 2024-04-06T10:15:51.658 INFO:tasks.workunit:Deleted artificial mount point /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror 2024-04-06T10:15:51.661 DEBUG:teuthology.run_tasks:Unwinding manager rbd-mirror-thrash 2024-04-06T10:15:51.674 INFO:tasks.rbd_mirror_thrash:joining rbd_mirror_thrash 2024-04-06T10:15:51.674 INFO:tasks.rbd_mirror_thrash:stat['kill'] = 19 2024-04-06T10:15:51.674 INFO:tasks.rbd_mirror_thrash:done joining 2024-04-06T10:15:51.674 DEBUG:teuthology.run_tasks:Unwinding manager rbd-mirror-thrash 2024-04-06T10:15:51.684 INFO:tasks.rbd_mirror_thrash:joining rbd_mirror_thrash 2024-04-06T10:15:51.684 INFO:tasks.rbd_mirror_thrash:stat['kill'] = 18 2024-04-06T10:15:51.684 INFO:tasks.rbd_mirror_thrash:done joining 2024-04-06T10:15:51.684 DEBUG:teuthology.run_tasks:Unwinding manager rbd-mirror 2024-04-06T10:15:51.694 DEBUG:tasks.rbd_mirror.cluster2.client.mirror.3:waiting for process to exit 2024-04-06T10:15:51.694 INFO:teuthology.orchestra.run:waiting for 300 2024-04-06T10:15:51.700 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi192.stderr:2024-04-06T10:15:51.702+0000 7fa0973b3640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.3 (PID: 172048) UID: 1000 2024-04-06T10:15:51.701 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi192.stderr:2024-04-06T10:15:51.702+0000 7fa096bb2640 -1 rbd::mirror::LeaderWatcher: 0x5588f5c1ac00 handle_unregister_watch: error unregistering leader watcher for rbd_mirror_leader object: (2) No such file or directory 2024-04-06T10:15:51.701 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi192.stderr:2024-04-06T10:15:51.702+0000 7fa0963b1640 -1 rbd::mirror::InstanceWatcher: 0x5588f5a3c380 handle_unregister_watch: error unregistering instance watcher for rbd_mirror_instance.18203 object: (2) No such file or directory 2024-04-06T10:15:51.702 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi192.stderr:2024-04-06T10:15:51.702+0000 7fa0903a5640 -1 rbd::mirror::InstanceWatcher: 0x5588f5a3c700 handle_unregister_instance: error unregistering instance: (2) No such file or directory 2024-04-06T10:15:51.702 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi192.stderr:2024-04-06T10:15:51.702+0000 7fa0963b1640 -1 rbd::mirror::InstanceWatcher: 0x5588f5a3ca80 handle_unregister_watch: error unregistering instance watcher for rbd_mirror_instance.18205 object: (2) No such file or directory 2024-04-06T10:15:51.702 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi192.stderr:2024-04-06T10:15:51.702+0000 7fa08fba4640 -1 rbd::mirror::InstanceWatcher: 0x5588f5a3c380 handle_unregister_instance: error unregistering instance: (2) No such file or directory 2024-04-06T10:15:51.704 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi192.stderr:2024-04-06T10:15:51.702+0000 7fa089397640 -1 rbd::mirror::MirrorStatusUpdater 0x5588f5a925a0 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-04-06T10:15:51.704 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi192.stderr:2024-04-06T10:15:51.702+0000 7fa0963b1640 -1 rbd::mirror::NamespaceReplayer: 0x5588f5a329c0 handle_shut_down_remote_status_updater: error shutting remote mirror status updater down: (2) No such file or directory 2024-04-06T10:15:51.704 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi192.stderr:2024-04-06T10:15:51.702+0000 7fa089397640 -1 rbd::mirror::MirrorStatusUpdater 0x5588f5a92360 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-04-06T10:15:51.704 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi192.stderr:2024-04-06T10:15:51.702+0000 7fa096bb2640 -1 rbd::mirror::NamespaceReplayer: 0x5588f5a32820 handle_shut_down_remote_status_updater: error shutting remote mirror status updater down: (2) No such file or directory 2024-04-06T10:15:51.704 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi192.stderr:2024-04-06T10:15:51.702+0000 7fa082b8a640 -1 rbd::mirror::InstanceWatcher: 0x5588f5a3ca80 handle_unregister_instance: error unregistering instance: (2) No such file or directory 2024-04-06T10:15:51.704 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi192.stderr:2024-04-06T10:15:51.702+0000 7fa0903a5640 -1 rbd::mirror::MirrorStatusUpdater 0x5588f5a92120 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-04-06T10:15:51.704 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi192.stderr:2024-04-06T10:15:51.702+0000 7fa096bb2640 -1 rbd::mirror::NamespaceReplayer: 0x5588f5a329c0 handle_shut_down_local_status_updater: error shutting local mirror status updater down: (2) No such file or directory 2024-04-06T10:15:51.704 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi192.stderr:2024-04-06T10:15:51.702+0000 7fa0903a5640 -1 rbd::mirror::MirrorStatusUpdater 0x5588f5a92000 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-04-06T10:15:51.704 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi192.stderr:2024-04-06T10:15:51.702+0000 7fa0963b1640 -1 rbd::mirror::NamespaceReplayer: 0x5588f5a32820 handle_shut_down_local_status_updater: error shutting local mirror status updater down: (2) No such file or directory 2024-04-06T10:15:51.704 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi192.stderr:2024-04-06T10:15:51.706+0000 7fa07c37d640 -1 rbd::mirror::MirrorStatusUpdater 0x5588f5c967e0 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-04-06T10:15:51.705 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi192.stderr:2024-04-06T10:15:51.706+0000 7fa096bb2640 -1 rbd::mirror::NamespaceReplayer: 0x5588f5e6c4e0 handle_shut_down_remote_status_updater: error shutting remote mirror status updater down: (2) No such file or directory 2024-04-06T10:15:51.705 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi192.stderr:2024-04-06T10:15:51.706+0000 7fa082b8a640 -1 rbd::mirror::MirrorStatusUpdater 0x5588f5c8c240 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-04-06T10:15:51.705 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi192.stderr:2024-04-06T10:15:51.706+0000 7fa0963b1640 -1 rbd::mirror::NamespaceReplayer: 0x5588f5e6c4e0 handle_shut_down_local_status_updater: error shutting local mirror status updater down: (2) No such file or directory 2024-04-06T10:15:51.707 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi192.stderr:2024-04-06T10:15:51.710+0000 7fa096bb2640 -1 rbd::mirror::LeaderWatcher: 0x5588f5978300 handle_unregister_watch: error unregistering leader watcher for rbd_mirror_leader object: (2) No such file or directory 2024-04-06T10:15:51.708 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi192.stderr:2024-04-06T10:15:51.710+0000 7fa096bb2640 -1 rbd::mirror::InstanceWatcher: 0x5588f5a3c000 handle_unregister_watch: error unregistering instance watcher for rbd_mirror_instance.18203 object: (2) No such file or directory 2024-04-06T10:15:51.709 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi192.stderr:2024-04-06T10:15:51.710+0000 7fa0903a5640 -1 rbd::mirror::InstanceWatcher: 0x5588f5a3c000 handle_unregister_instance: error unregistering instance: (2) No such file or directory 2024-04-06T10:15:51.709 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi192.stderr:2024-04-06T10:15:51.710+0000 7fa089397640 -1 rbd::mirror::MirrorStatusUpdater 0x5588f64906c0 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-04-06T10:15:51.709 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi192.stderr:2024-04-06T10:15:51.710+0000 7fa096bb2640 -1 rbd::mirror::NamespaceReplayer: 0x5588f5a32000 handle_shut_down_remote_status_updater: error shutting remote mirror status updater down: (2) No such file or directory 2024-04-06T10:15:51.710 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi192.stderr:2024-04-06T10:15:51.710+0000 7fa0903a5640 -1 rbd::mirror::MirrorStatusUpdater 0x5588f64307e0 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-04-06T10:15:51.710 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi192.stderr:2024-04-06T10:15:51.710+0000 7fa0963b1640 -1 rbd::mirror::NamespaceReplayer: 0x5588f5a32000 handle_shut_down_local_status_updater: error shutting local mirror status updater down: (2) No such file or directory 2024-04-06T10:15:51.725 INFO:tasks.rbd_mirror.cluster2.client.mirror.3:Stopped 2024-04-06T10:15:51.726 DEBUG:teuthology.run_tasks:Unwinding manager rbd-mirror 2024-04-06T10:15:51.736 DEBUG:tasks.rbd_mirror.cluster2.client.mirror.2:waiting for process to exit 2024-04-06T10:15:51.736 INFO:teuthology.orchestra.run:waiting for 300 2024-04-06T10:15:51.768 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi192.stderr:2024-04-06T10:15:51.770+0000 7f4416489640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.2 (PID: 136350) UID: 1000 2024-04-06T10:15:51.770 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi192.stderr:2024-04-06T10:15:51.770+0000 7f440f47b640 -1 rbd::mirror::image_deleter::TrashWatcher: 0x561be61c2480 handle_unregister_watcher: error unregistering watcher for trash directory: (2) No such file or directory 2024-04-06T10:15:51.770 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi192.stderr:2024-04-06T10:15:51.770+0000 7f440f47b640 -1 rbd::mirror::image_deleter::TrashWatcher: 0x561be6af7200 handle_unregister_watcher: error unregistering watcher for trash directory: (2) No such file or directory 2024-04-06T10:15:51.770 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi192.stderr:2024-04-06T10:15:51.770+0000 7f4401c60640 -1 rbd::mirror::image_deleter::TrashWatcher: 0x561be6f16240 handle_unregister_watcher: error unregistering watcher for trash directory: (2) No such file or directory 2024-04-06T10:15:51.770 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi192.stderr:2024-04-06T10:15:51.770+0000 7f440f47b640 -1 rbd::mirror::PoolWatcher: 0x561be4d8fa40 operator(): error unregistering watcher for rbd_mirroring object: (2) No such file or directory 2024-04-06T10:15:51.770 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi192.stderr:2024-04-06T10:15:51.770+0000 7f4408c6e640 -1 rbd::mirror::PoolWatcher: 0x561be6274000 operator(): error unregistering watcher for rbd_mirroring object: (2) No such file or directory 2024-04-06T10:15:51.770 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi192.stderr:2024-04-06T10:15:51.770+0000 7f4408c6e640 -1 rbd::mirror::PoolWatcher: 0x561be4d8fc20 operator(): error unregistering watcher for rbd_mirroring object: (2) No such file or directory 2024-04-06T10:15:51.770 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi192.stderr:2024-04-06T10:15:51.770+0000 7f440f47b640 -1 rbd::mirror::PoolWatcher: 0x561be4d8f860 operator(): error unregistering watcher for rbd_mirroring object: (2) No such file or directory 2024-04-06T10:15:51.770 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi192.stderr:2024-04-06T10:15:51.770+0000 7f43fb453640 -1 rbd::mirror::PoolWatcher: 0x561be6758d20 operator(): error unregistering watcher for rbd_mirroring object: (2) No such file or directory 2024-04-06T10:15:51.770 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi192.stderr:2024-04-06T10:15:51.770+0000 7f4401c60640 -1 rbd::mirror::PoolWatcher: 0x561be6759e00 operator(): error unregistering watcher for rbd_mirroring object: (2) No such file or directory 2024-04-06T10:15:51.773 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi192.stderr:2024-04-06T10:15:51.770+0000 7f4415487640 -1 rbd::mirror::InstanceWatcher: 0x561be5cf6380 handle_unregister_watch: error unregistering instance watcher for rbd_mirror_instance.15115 object: (2) No such file or directory 2024-04-06T10:15:51.773 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi192.stderr:2024-04-06T10:15:51.770+0000 7f4415487640 -1 rbd::mirror::LeaderWatcher: 0x561be6062300 handle_notify_lock_released: error notifying leader lock released: (2) No such file or directory 2024-04-06T10:15:51.773 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi192.stderr:2024-04-06T10:15:51.770+0000 7f4415487640 -1 rbd::mirror::InstanceWatcher: 0x561be5cf6700 handle_unregister_watch: error unregistering instance watcher for rbd_mirror_instance.15115 object: (2) No such file or directory 2024-04-06T10:15:51.773 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi192.stderr:2024-04-06T10:15:51.770+0000 7f4415487640 -1 rbd::mirror::LeaderWatcher: 0x561be6062300 handle_shut_down_leader_lock: error shutting down leader lock: (2) No such file or directory 2024-04-06T10:15:51.773 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi192.stderr:2024-04-06T10:15:51.770+0000 7f440ec7a640 -1 rbd::mirror::InstanceWatcher: 0x561be5cf6380 handle_unregister_instance: error unregistering instance: (2) No such file or directory 2024-04-06T10:15:51.773 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi192.stderr:2024-04-06T10:15:51.770+0000 7f4415c88640 -1 rbd::mirror::LeaderWatcher: 0x561be6062300 handle_unregister_watch: error unregistering leader watcher for rbd_mirror_leader object: (2) No such file or directory 2024-04-06T10:15:51.773 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi192.stderr:2024-04-06T10:15:51.770+0000 7f440ec7a640 -1 rbd::mirror::InstanceWatcher: 0x561be5cf6700 handle_unregister_instance: error unregistering instance: (2) No such file or directory 2024-04-06T10:15:51.773 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi192.stderr:2024-04-06T10:15:51.770+0000 7f440846d640 -1 rbd::mirror::MirrorStatusUpdater 0x561be5d425a0 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-04-06T10:15:51.773 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi192.stderr:2024-04-06T10:15:51.770+0000 7f440846d640 -1 rbd::mirror::MirrorStatusUpdater 0x561be5d42360 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-04-06T10:15:51.773 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi192.stderr:2024-04-06T10:15:51.770+0000 7f4415c88640 -1 rbd::mirror::NamespaceReplayer: 0x561be5cf4820 handle_shut_down_remote_status_updater: error shutting remote mirror status updater down: (2) No such file or directory 2024-04-06T10:15:51.773 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi192.stderr:2024-04-06T10:15:51.770+0000 7f4415c88640 -1 rbd::mirror::NamespaceReplayer: 0x561be5cf49c0 handle_shut_down_remote_status_updater: error shutting remote mirror status updater down: (2) No such file or directory 2024-04-06T10:15:51.774 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi192.stderr:2024-04-06T10:15:51.774+0000 7f440ec7a640 -1 rbd::mirror::MirrorStatusUpdater 0x561be5d42000 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-04-06T10:15:51.774 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi192.stderr:2024-04-06T10:15:51.774+0000 7f440ec7a640 -1 rbd::mirror::MirrorStatusUpdater 0x561be5d42120 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-04-06T10:15:51.774 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi192.stderr:2024-04-06T10:15:51.774+0000 7f4415487640 -1 rbd::mirror::InstanceWatcher: 0x561be619e000 handle_unregister_watch: error unregistering instance watcher for rbd_mirror_instance.15118 object: (2) No such file or directory 2024-04-06T10:15:51.774 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi192.stderr:2024-04-06T10:15:51.774+0000 7f4415487640 -1 rbd::mirror::NamespaceReplayer: 0x561be5cf4820 handle_shut_down_local_status_updater: error shutting local mirror status updater down: (2) No such file or directory 2024-04-06T10:15:51.774 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi192.stderr:2024-04-06T10:15:51.774+0000 7f4415487640 -1 rbd::mirror::NamespaceReplayer: 0x561be5cf49c0 handle_shut_down_local_status_updater: error shutting local mirror status updater down: (2) No such file or directory 2024-04-06T10:15:51.774 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi192.stderr:2024-04-06T10:15:51.774+0000 7f440145f640 -1 rbd::mirror::InstanceWatcher: 0x561be619e000 handle_unregister_instance: error unregistering instance: (2) No such file or directory 2024-04-06T10:15:51.774 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi192.stderr:2024-04-06T10:15:51.774+0000 7f43fb453640 -1 rbd::mirror::MirrorStatusUpdater 0x561be5d42b40 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-04-06T10:15:51.774 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi192.stderr:2024-04-06T10:15:51.774+0000 7f4415487640 -1 rbd::mirror::NamespaceReplayer: 0x561be60fa4e0 handle_shut_down_remote_status_updater: error shutting remote mirror status updater down: (2) No such file or directory 2024-04-06T10:15:51.774 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi192.stderr:2024-04-06T10:15:51.774+0000 7f440145f640 -1 rbd::mirror::MirrorStatusUpdater 0x561be5f1e480 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-04-06T10:15:51.774 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi192.stderr:2024-04-06T10:15:51.774+0000 7f4415c88640 -1 rbd::mirror::NamespaceReplayer: 0x561be60fa4e0 handle_shut_down_local_status_updater: error shutting local mirror status updater down: (2) No such file or directory 2024-04-06T10:15:51.777 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi192.stderr:2024-04-06T10:15:51.778+0000 7f440ec7a640 -1 rbd::mirror::image_deleter::TrashWatcher: 0x561be6af6fc0 handle_unregister_watcher: error unregistering watcher for trash directory: (2) No such file or directory 2024-04-06T10:15:51.777 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi192.stderr:2024-04-06T10:15:51.778+0000 7f440f47b640 -1 rbd::mirror::PoolWatcher: 0x561be4d8fe00 operator(): error unregistering watcher for rbd_mirroring object: (2) No such file or directory 2024-04-06T10:15:51.777 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi192.stderr:2024-04-06T10:15:51.778+0000 7f4408c6e640 -1 rbd::mirror::PoolWatcher: 0x561be62741e0 operator(): error unregistering watcher for rbd_mirroring object: (2) No such file or directory 2024-04-06T10:15:51.778 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi192.stderr:2024-04-06T10:15:51.778+0000 7f4415c88640 -1 rbd::mirror::LeaderWatcher: 0x561be6969b00 handle_notify_lock_released: error notifying leader lock released: (2) No such file or directory 2024-04-06T10:15:51.778 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi192.stderr:2024-04-06T10:15:51.778+0000 7f4415c88640 -1 rbd::mirror::LeaderWatcher: 0x561be6969b00 handle_shut_down_leader_lock: error shutting down leader lock: (2) No such file or directory 2024-04-06T10:15:51.778 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi192.stderr:2024-04-06T10:15:51.778+0000 7f4415487640 -1 rbd::mirror::LeaderWatcher: 0x561be6969b00 handle_unregister_watch: error unregistering leader watcher for rbd_mirror_leader object: (2) No such file or directory 2024-04-06T10:15:51.778 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi192.stderr:2024-04-06T10:15:51.778+0000 7f4415487640 -1 rbd::mirror::InstanceWatcher: 0x561be5cf6000 handle_unregister_watch: error unregistering instance watcher for rbd_mirror_instance.15115 object: (2) No such file or directory 2024-04-06T10:15:51.779 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi192.stderr:2024-04-06T10:15:51.778+0000 7f440ec7a640 -1 rbd::mirror::InstanceWatcher: 0x561be5cf6000 handle_unregister_instance: error unregistering instance: (2) No such file or directory 2024-04-06T10:15:51.779 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi192.stderr:2024-04-06T10:15:51.778+0000 7f4408c6e640 -1 rbd::mirror::MirrorStatusUpdater 0x561be694e6c0 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-04-06T10:15:51.780 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi192.stderr:2024-04-06T10:15:51.778+0000 7f4415487640 -1 rbd::mirror::NamespaceReplayer: 0x561be5cf4000 handle_shut_down_remote_status_updater: error shutting remote mirror status updater down: (2) No such file or directory 2024-04-06T10:15:51.780 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi192.stderr:2024-04-06T10:15:51.782+0000 7f440f47b640 -1 rbd::mirror::MirrorStatusUpdater 0x561be69006c0 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-04-06T10:15:51.780 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi192.stderr:2024-04-06T10:15:51.782+0000 7f4415c88640 -1 rbd::mirror::NamespaceReplayer: 0x561be5cf4000 handle_shut_down_local_status_updater: error shutting local mirror status updater down: (2) No such file or directory 2024-04-06T10:15:51.798 INFO:tasks.rbd_mirror.cluster2.client.mirror.2:Stopped 2024-04-06T10:15:51.798 DEBUG:teuthology.run_tasks:Unwinding manager rbd-mirror 2024-04-06T10:15:51.809 DEBUG:tasks.rbd_mirror.cluster2.client.mirror.1:waiting for process to exit 2024-04-06T10:15:51.809 INFO:teuthology.orchestra.run:waiting for 300 2024-04-06T10:15:51.840 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi192.stderr:2024-04-06T10:15:51.842+0000 7fd94022c640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.1 (PID: 148971) UID: 1000 2024-04-06T10:15:51.841 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi192.stderr:2024-04-06T10:15:51.842+0000 7fd93fa2b640 -1 rbd::mirror::LeaderWatcher: 0x56159c00c300 handle_unregister_watch: error unregistering leader watcher for rbd_mirror_leader object: (2) No such file or directory 2024-04-06T10:15:51.842 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi192.stderr:2024-04-06T10:15:51.842+0000 7fd93f22a640 -1 rbd::mirror::InstanceWatcher: 0x56159bc06380 handle_unregister_watch: error unregistering instance watcher for rbd_mirror_instance.16232 object: (2) No such file or directory 2024-04-06T10:15:51.842 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi192.stderr:2024-04-06T10:15:51.842+0000 7fd93f22a640 -1 rbd::mirror::InstanceWatcher: 0x56159bc06700 handle_unregister_watch: error unregistering instance watcher for rbd_mirror_instance.16232 object: (2) No such file or directory 2024-04-06T10:15:51.843 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi192.stderr:2024-04-06T10:15:51.842+0000 7fd93921e640 -1 rbd::mirror::InstanceWatcher: 0x56159bc06380 handle_unregister_instance: error unregistering instance: (2) No such file or directory 2024-04-06T10:15:51.843 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi192.stderr:2024-04-06T10:15:51.842+0000 7fd93f22a640 -1 rbd::mirror::InstanceWatcher: 0x56159c0ba000 handle_unregister_watch: error unregistering instance watcher for rbd_mirror_instance.16235 object: (2) No such file or directory 2024-04-06T10:15:51.843 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi192.stderr:2024-04-06T10:15:51.842+0000 7fd938a1d640 -1 rbd::mirror::InstanceWatcher: 0x56159bc06700 handle_unregister_instance: error unregistering instance: (2) No such file or directory 2024-04-06T10:15:51.843 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi192.stderr:2024-04-06T10:15:51.842+0000 7fd92b202640 -1 rbd::mirror::InstanceWatcher: 0x56159c0ba000 handle_unregister_instance: error unregistering instance: (2) No such file or directory 2024-04-06T10:15:51.843 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi192.stderr:2024-04-06T10:15:51.842+0000 7fd932210640 -1 rbd::mirror::MirrorStatusUpdater 0x56159bc525a0 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-04-06T10:15:51.844 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi192.stderr:2024-04-06T10:15:51.842+0000 7fd932210640 -1 rbd::mirror::MirrorStatusUpdater 0x56159bc52360 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-04-06T10:15:51.844 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi192.stderr:2024-04-06T10:15:51.842+0000 7fd93fa2b640 -1 rbd::mirror::NamespaceReplayer: 0x56159bbf8820 handle_shut_down_remote_status_updater: error shutting remote mirror status updater down: (2) No such file or directory 2024-04-06T10:15:51.844 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi192.stderr:2024-04-06T10:15:51.842+0000 7fd93fa2b640 -1 rbd::mirror::NamespaceReplayer: 0x56159bbf89c0 handle_shut_down_remote_status_updater: error shutting remote mirror status updater down: (2) No such file or directory 2024-04-06T10:15:51.845 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi192.stderr:2024-04-06T10:15:51.846+0000 7fd9249f5640 -1 rbd::mirror::MirrorStatusUpdater 0x56159bc52b40 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-04-06T10:15:51.845 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi192.stderr:2024-04-06T10:15:51.846+0000 7fd93f22a640 -1 rbd::mirror::NamespaceReplayer: 0x56159bf6a680 handle_shut_down_remote_status_updater: error shutting remote mirror status updater down: (2) No such file or directory 2024-04-06T10:15:51.845 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi192.stderr:2024-04-06T10:15:51.846+0000 7fd93921e640 -1 rbd::mirror::MirrorStatusUpdater 0x56159bc52000 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-04-06T10:15:51.845 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi192.stderr:2024-04-06T10:15:51.846+0000 7fd93921e640 -1 rbd::mirror::MirrorStatusUpdater 0x56159bc52120 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-04-06T10:15:51.845 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi192.stderr:2024-04-06T10:15:51.846+0000 7fd93f22a640 -1 rbd::mirror::NamespaceReplayer: 0x56159bbf8820 handle_shut_down_local_status_updater: error shutting local mirror status updater down: (2) No such file or directory 2024-04-06T10:15:51.845 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi192.stderr:2024-04-06T10:15:51.846+0000 7fd93f22a640 -1 rbd::mirror::NamespaceReplayer: 0x56159bbf89c0 handle_shut_down_local_status_updater: error shutting local mirror status updater down: (2) No such file or directory 2024-04-06T10:15:51.845 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi192.stderr:2024-04-06T10:15:51.846+0000 7fd92ba03640 -1 rbd::mirror::MirrorStatusUpdater 0x56159be526c0 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-04-06T10:15:51.845 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi192.stderr:2024-04-06T10:15:51.846+0000 7fd93fa2b640 -1 rbd::mirror::NamespaceReplayer: 0x56159bf6a680 handle_shut_down_local_status_updater: error shutting local mirror status updater down: (2) No such file or directory 2024-04-06T10:15:51.852 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi192.stderr:2024-04-06T10:15:51.854+0000 7fd93f22a640 -1 rbd::mirror::LeaderWatcher: 0x56159b9cfb00 handle_unregister_watch: error unregistering leader watcher for rbd_mirror_leader object: (2) No such file or directory 2024-04-06T10:15:51.852 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi192.stderr:2024-04-06T10:15:51.854+0000 7fd93f22a640 -1 rbd::mirror::InstanceWatcher: 0x56159bc06000 handle_unregister_watch: error unregistering instance watcher for rbd_mirror_instance.16232 object: (2) No such file or directory 2024-04-06T10:15:51.853 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi192.stderr:2024-04-06T10:15:51.854+0000 7fd93921e640 -1 rbd::mirror::InstanceWatcher: 0x56159bc06000 handle_unregister_instance: error unregistering instance: (2) No such file or directory 2024-04-06T10:15:51.854 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi192.stderr:2024-04-06T10:15:51.854+0000 7fd932a11640 -1 rbd::mirror::MirrorStatusUpdater 0x56159c6566c0 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-04-06T10:15:51.854 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi192.stderr:2024-04-06T10:15:51.854+0000 7fd93f22a640 -1 rbd::mirror::NamespaceReplayer: 0x56159bbf8000 handle_shut_down_remote_status_updater: error shutting remote mirror status updater down: (2) No such file or directory 2024-04-06T10:15:51.854 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi192.stderr:2024-04-06T10:15:51.854+0000 7fd93921e640 -1 rbd::mirror::MirrorStatusUpdater 0x56159c6047e0 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-04-06T10:15:51.854 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi192.stderr:2024-04-06T10:15:51.854+0000 7fd93fa2b640 -1 rbd::mirror::NamespaceReplayer: 0x56159bbf8000 handle_shut_down_local_status_updater: error shutting local mirror status updater down: (2) No such file or directory 2024-04-06T10:15:51.880 INFO:tasks.rbd_mirror.cluster2.client.mirror.1:Stopped 2024-04-06T10:15:51.880 DEBUG:teuthology.run_tasks:Unwinding manager rbd-mirror 2024-04-06T10:15:51.892 DEBUG:tasks.rbd_mirror.cluster2.client.mirror.0:waiting for process to exit 2024-04-06T10:15:51.892 INFO:teuthology.orchestra.run:waiting for 300 2024-04-06T10:15:51.924 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi192.stderr:2024-04-06T10:15:51.926+0000 7fc23d128640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.0 (PID: 173211) UID: 1000 2024-04-06T10:15:51.926 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi192.stderr:2024-04-06T10:15:51.926+0000 7fc2288ff640 -1 rbd::mirror::InstanceWatcher: 0x564264647c00 handle_unregister_instance: error unregistering instance: (2) No such file or directory 2024-04-06T10:15:51.926 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi192.stderr:2024-04-06T10:15:51.926+0000 7fc235919640 -1 rbd::mirror::InstanceWatcher: 0x564264647880 handle_unregister_instance: error unregistering instance: (2) No such file or directory 2024-04-06T10:15:51.926 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi192.stderr:2024-04-06T10:15:51.926+0000 7fc235919640 -1 rbd::mirror::InstanceWatcher: 0x564264647500 handle_unregister_instance: error unregistering instance: (2) No such file or directory 2024-04-06T10:15:51.927 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi192.stderr:2024-04-06T10:15:51.926+0000 7fc2218f1640 -1 rbd::mirror::MirrorStatusUpdater 0x5642645e5200 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-04-06T10:15:51.927 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi192.stderr:2024-04-06T10:15:51.926+0000 7fc23c927640 -1 rbd::mirror::NamespaceReplayer: 0x564264798680 handle_shut_down_remote_status_updater: error shutting remote mirror status updater down: (2) No such file or directory 2024-04-06T10:15:51.927 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi192.stderr:2024-04-06T10:15:51.926+0000 7fc22f90d640 -1 rbd::mirror::MirrorStatusUpdater 0x5642645e4b40 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-04-06T10:15:51.927 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi192.stderr:2024-04-06T10:15:51.926+0000 7fc22f90d640 -1 rbd::mirror::MirrorStatusUpdater 0x5642645e46c0 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-04-06T10:15:51.927 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi192.stderr:2024-04-06T10:15:51.926+0000 7fc23c927640 -1 rbd::mirror::NamespaceReplayer: 0x564264798000 handle_shut_down_remote_status_updater: error shutting remote mirror status updater down: (2) No such file or directory 2024-04-06T10:15:51.927 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi192.stderr:2024-04-06T10:15:51.926+0000 7fc23c927640 -1 rbd::mirror::NamespaceReplayer: 0x5642647981a0 handle_shut_down_remote_status_updater: error shutting remote mirror status updater down: (2) No such file or directory 2024-04-06T10:15:51.931 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi192.stderr:2024-04-06T10:15:51.930+0000 7fc23611a640 -1 rbd::mirror::InstanceWatcher: 0x564264e87500 handle_unregister_instance: error unregistering instance: (2) No such file or directory 2024-04-06T10:15:51.932 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi192.stderr:2024-04-06T10:15:51.934+0000 7fc22f10c640 -1 rbd::mirror::MirrorStatusUpdater 0x564264ff26c0 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-04-06T10:15:51.932 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi192.stderr:2024-04-06T10:15:51.934+0000 7fc23c927640 -1 rbd::mirror::NamespaceReplayer: 0x564264e66680 handle_shut_down_remote_status_updater: error shutting remote mirror status updater down: (2) No such file or directory 2024-04-06T10:15:51.953 INFO:tasks.rbd_mirror.cluster2.client.mirror.0:Stopped 2024-04-06T10:15:51.953 DEBUG:teuthology.run_tasks:Unwinding manager rbd-mirror 2024-04-06T10:15:51.966 DEBUG:tasks.rbd_mirror.cluster1.client.mirror.3:waiting for process to exit 2024-04-06T10:15:51.966 INFO:teuthology.orchestra.run:waiting for 300 2024-04-06T10:15:52.001 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi192.stderr:2024-04-06T10:15:51.998+0000 7fd6012a8640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.3 (PID: 173633) UID: 1000 2024-04-06T10:15:52.004 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi192.stderr:2024-04-06T10:15:52.002+0000 7fd5eca7f640 -1 rbd::mirror::InstanceWatcher: 0x55dd0051ca80 handle_unregister_instance: error unregistering instance: (2) No such file or directory 2024-04-06T10:15:52.004 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi192.stderr:2024-04-06T10:15:52.006+0000 7fd6002a6640 -1 rbd::mirror::InstanceWatcher: 0x55dd0051c700 handle_unregister_watch: error unregistering instance watcher for rbd_mirror_instance.5450 object: (2) No such file or directory 2024-04-06T10:15:52.005 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi192.stderr:2024-04-06T10:15:52.006+0000 7fd6002a6640 -1 rbd::mirror::InstanceWatcher: 0x55dd0051c380 handle_unregister_watch: error unregistering instance watcher for rbd_mirror_instance.5450 object: (2) No such file or directory 2024-04-06T10:15:52.005 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi192.stderr:2024-04-06T10:15:52.006+0000 7fd5f9a99640 -1 rbd::mirror::InstanceWatcher: 0x55dd0051c700 handle_unregister_instance: error unregistering instance: (2) No such file or directory 2024-04-06T10:15:52.005 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi192.stderr:2024-04-06T10:15:52.006+0000 7fd5f9a99640 -1 rbd::mirror::InstanceWatcher: 0x55dd0051c380 handle_unregister_instance: error unregistering instance: (2) No such file or directory 2024-04-06T10:15:52.006 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi192.stderr:2024-04-06T10:15:52.006+0000 7fd5fa29a640 -1 rbd::mirror::MirrorStatusUpdater 0x55dd0058a120 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-04-06T10:15:52.006 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi192.stderr:2024-04-06T10:15:52.006+0000 7fd5fa29a640 -1 rbd::mirror::MirrorStatusUpdater 0x55dd0058a000 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-04-06T10:15:52.006 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi192.stderr:2024-04-06T10:15:52.006+0000 7fd6002a6640 -1 rbd::mirror::NamespaceReplayer: 0x55dd005881a0 handle_shut_down_local_status_updater: error shutting local mirror status updater down: (2) No such file or directory 2024-04-06T10:15:52.006 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi192.stderr:2024-04-06T10:15:52.006+0000 7fd6002a6640 -1 rbd::mirror::NamespaceReplayer: 0x55dd00588000 handle_shut_down_local_status_updater: error shutting local mirror status updater down: (2) No such file or directory 2024-04-06T10:15:52.013 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi192.stderr:2024-04-06T10:15:52.014+0000 7fd6002a6640 -1 rbd::mirror::LeaderWatcher: 0x55dd002e1b00 handle_unregister_watch: error unregistering leader watcher for rbd_mirror_leader object: (2) No such file or directory 2024-04-06T10:15:52.027 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi192.stderr:2024-04-06T10:15:52.022+0000 7f7e295b6640 -1 rbd::mirror::InstanceWatcher: 0x55baa3308380 handle_unregister_watch: error unregistering instance watcher for rbd_mirror_instance.4998 object: (2) No such file or directory 2024-04-06T10:15:52.027 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi192.stderr:2024-04-06T10:15:52.022+0000 7f7e295b6640 -1 rbd::mirror::InstanceWatcher: 0x55baa3308700 handle_unregister_watch: error unregistering instance watcher for rbd_mirror_instance.4998 object: (2) No such file or directory 2024-04-06T10:15:52.027 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi192.stderr:2024-04-06T10:15:52.022+0000 7f7e225a8640 -1 rbd::mirror::InstanceWatcher: 0x55baa3308380 handle_unregister_instance: error unregistering instance: (2) No such file or directory 2024-04-06T10:15:52.027 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi192.stderr:2024-04-06T10:15:52.022+0000 7f7e22da9640 -1 rbd::mirror::InstanceWatcher: 0x55baa3308700 handle_unregister_instance: error unregistering instance: (2) No such file or directory 2024-04-06T10:15:52.027 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi192.stderr:2024-04-06T10:15:52.022+0000 7f7e1c59c640 -1 rbd::mirror::MirrorStatusUpdater 0x55baa3378360 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-04-06T10:15:52.027 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi192.stderr:2024-04-06T10:15:52.026+0000 7f7e1bd9b640 -1 rbd::mirror::MirrorStatusUpdater 0x55baa33785a0 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-04-06T10:15:52.027 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi192.stderr:2024-04-06T10:15:52.026+0000 7f7e295b6640 -1 rbd::mirror::NamespaceReplayer: 0x55baa33761a0 handle_shut_down_remote_status_updater: error shutting remote mirror status updater down: (2) No such file or directory 2024-04-06T10:15:52.027 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi192.stderr:2024-04-06T10:15:52.026+0000 7f7e295b6640 -1 rbd::mirror::NamespaceReplayer: 0x55baa3376000 handle_shut_down_remote_status_updater: error shutting remote mirror status updater down: (2) No such file or directory 2024-04-06T10:15:52.027 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi192.stderr:2024-04-06T10:15:52.026+0000 7f7e225a8640 -1 rbd::mirror::MirrorStatusUpdater 0x55baa3378120 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-04-06T10:15:52.027 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi192.stderr:2024-04-06T10:15:52.026+0000 7f7e225a8640 -1 rbd::mirror::MirrorStatusUpdater 0x55baa3378000 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-04-06T10:15:52.028 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi192.stderr:2024-04-06T10:15:52.026+0000 7f7e28db5640 -1 rbd::mirror::NamespaceReplayer: 0x55baa33761a0 handle_shut_down_local_status_updater: error shutting local mirror status updater down: (2) No such file or directory 2024-04-06T10:15:52.028 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi192.stderr:2024-04-06T10:15:52.026+0000 7f7e28db5640 -1 rbd::mirror::NamespaceReplayer: 0x55baa3376000 handle_shut_down_local_status_updater: error shutting local mirror status updater down: (2) No such file or directory 2024-04-06T10:15:52.028 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi192.stderr:2024-04-06T10:15:52.022+0000 7fd6002a6640 -1 rbd::mirror::InstanceWatcher: 0x55dd0051c000 handle_unregister_watch: error unregistering instance watcher for rbd_mirror_instance.5450 object: (2) No such file or directory 2024-04-06T10:15:52.028 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi192.stderr:2024-04-06T10:15:52.022+0000 7fd5f9a99640 -1 rbd::mirror::InstanceWatcher: 0x55dd0051c000 handle_unregister_instance: error unregistering instance: (2) No such file or directory 2024-04-06T10:15:52.028 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi192.stderr:2024-04-06T10:15:52.026+0000 7fd5f9a99640 -1 rbd::mirror::MirrorStatusUpdater 0x55dd00ee86c0 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-04-06T10:15:52.028 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi192.stderr:2024-04-06T10:15:52.026+0000 7fd6002a6640 -1 rbd::mirror::NamespaceReplayer: 0x55dd00460340 handle_shut_down_local_status_updater: error shutting local mirror status updater down: (2) No such file or directory 2024-04-06T10:15:52.062 INFO:tasks.rbd_mirror.cluster1.client.mirror.3:Stopped 2024-04-06T10:15:52.062 DEBUG:teuthology.run_tasks:Unwinding manager rbd-mirror 2024-04-06T10:15:52.075 DEBUG:tasks.rbd_mirror.cluster1.client.mirror.2:waiting for process to exit 2024-04-06T10:15:52.075 INFO:teuthology.orchestra.run:waiting for 300 2024-04-06T10:15:52.075 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi192.stderr:2024-04-06T10:15:52.074+0000 7f7e0ed81640 -1 rbd::mirror::RemotePoolPoller: 0x55baa23cf860 handle_mirror_peer_ping: failed to ping remote mirror peer: (2) No such file or directory 2024-04-06T10:15:52.076 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi192.stderr:2024-04-06T10:15:52.074+0000 7f7e0ed81640 -1 rbd::mirror::RemotePoolPoller: 0x55baa23cf860 handle_mirror_peer_list: failed to retrieve mirror peers: (2) No such file or directory 2024-04-06T10:15:52.076 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi192.stderr:2024-04-06T10:15:52.074+0000 7f7e0ed81640 -1 rbd::mirror::PoolReplayer: 0x55baa3318000 handle_remote_pool_meta_updated: remote pool metadata updated unexpectedly 2024-04-06T10:15:52.076 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi192.stderr:2024-04-06T10:15:52.078+0000 7f7e29db7640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.2 (PID: 161532) UID: 1000 2024-04-06T10:15:52.076 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi192.stderr:2024-04-06T10:15:52.078+0000 7f7e295b6640 -1 rbd::mirror::LeaderWatcher: 0x55baa368a300 handle_unregister_watch: error unregistering leader watcher for rbd_mirror_leader object: (2) No such file or directory 2024-04-06T10:15:52.077 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi192.stderr:2024-04-06T10:15:52.078+0000 7f7e295b6640 -1 rbd::mirror::InstanceWatcher: 0x55baa37cc000 handle_unregister_watch: error unregistering instance watcher for rbd_mirror_instance.5001 object: (2) No such file or directory 2024-04-06T10:15:52.078 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi192.stderr:2024-04-06T10:15:52.078+0000 7f7e14d8d640 -1 rbd::mirror::InstanceWatcher: 0x55baa37cc000 handle_unregister_instance: error unregistering instance: (2) No such file or directory 2024-04-06T10:15:52.080 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi192.stderr:2024-04-06T10:15:52.078+0000 7f7e0e580640 -1 rbd::mirror::MirrorStatusUpdater 0x55baa3378b40 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-04-06T10:15:52.080 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi192.stderr:2024-04-06T10:15:52.078+0000 7f7e295b6640 -1 rbd::mirror::NamespaceReplayer: 0x55baa3376ea0 handle_shut_down_remote_status_updater: error shutting remote mirror status updater down: (2) No such file or directory 2024-04-06T10:15:52.082 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi192.stderr:2024-04-06T10:15:52.082+0000 7f7e1558e640 -1 rbd::mirror::MirrorStatusUpdater 0x55baa3526480 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-04-06T10:15:52.082 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi192.stderr:2024-04-06T10:15:52.082+0000 7f7e28db5640 -1 rbd::mirror::NamespaceReplayer: 0x55baa3376ea0 handle_shut_down_local_status_updater: error shutting local mirror status updater down: (2) No such file or directory 2024-04-06T10:15:52.084 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi192.stderr:2024-04-06T10:15:52.086+0000 7f7e295b6640 -1 rbd::mirror::LeaderWatcher: 0x55baa3f79b00 handle_unregister_watch: error unregistering leader watcher for rbd_mirror_leader object: (2) No such file or directory 2024-04-06T10:15:52.085 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi192.stderr:2024-04-06T10:15:52.086+0000 7f7e295b6640 -1 rbd::mirror::InstanceWatcher: 0x55baa3308000 handle_unregister_watch: error unregistering instance watcher for rbd_mirror_instance.4998 object: (2) No such file or directory 2024-04-06T10:15:52.085 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi192.stderr:2024-04-06T10:15:52.086+0000 7f7e22da9640 -1 rbd::mirror::InstanceWatcher: 0x55baa3308000 handle_unregister_instance: error unregistering instance: (2) No such file or directory 2024-04-06T10:15:52.086 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi192.stderr:2024-04-06T10:15:52.086+0000 7f7e1bd9b640 -1 rbd::mirror::MirrorStatusUpdater 0x55baa3f606c0 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-04-06T10:15:52.086 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi192.stderr:2024-04-06T10:15:52.086+0000 7f7e295b6640 -1 rbd::mirror::NamespaceReplayer: 0x55baa40f24e0 handle_shut_down_remote_status_updater: error shutting remote mirror status updater down: (2) No such file or directory 2024-04-06T10:15:52.086 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi192.stderr:2024-04-06T10:15:52.086+0000 7f7e225a8640 -1 rbd::mirror::MirrorStatusUpdater 0x55baa3ee86c0 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-04-06T10:15:52.086 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi192.stderr:2024-04-06T10:15:52.086+0000 7f7e28db5640 -1 rbd::mirror::NamespaceReplayer: 0x55baa40f24e0 handle_shut_down_local_status_updater: error shutting local mirror status updater down: (2) No such file or directory 2024-04-06T10:15:52.105 INFO:tasks.rbd_mirror.cluster1.client.mirror.2:Stopped 2024-04-06T10:15:52.105 DEBUG:teuthology.run_tasks:Unwinding manager rbd-mirror 2024-04-06T10:15:52.160 DEBUG:tasks.rbd_mirror.cluster1.client.mirror.1:waiting for process to exit 2024-04-06T10:15:52.160 INFO:teuthology.orchestra.run:waiting for 300 2024-04-06T10:15:52.160 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi192.stderr:2024-04-06T10:15:52.162+0000 7f3d29bd0640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.1 (PID: 27474) UID: 1000 2024-04-06T10:15:52.162 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi192.stderr:2024-04-06T10:15:52.162+0000 7f3d22bc2640 -1 rbd::mirror::image_deleter::TrashWatcher: 0x560664922fc0 handle_unregister_watcher: error unregistering watcher for trash directory: (2) No such file or directory 2024-04-06T10:15:52.162 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi192.stderr:2024-04-06T10:15:52.162+0000 7f3d22bc2640 -1 rbd::mirror::image_deleter::TrashWatcher: 0x560664922d80 handle_unregister_watcher: error unregistering watcher for trash directory: (2) No such file or directory 2024-04-06T10:15:52.162 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi192.stderr:2024-04-06T10:15:52.162+0000 7f3d22bc2640 -1 rbd::mirror::PoolWatcher: 0x5606637683c0 operator(): error unregistering watcher for rbd_mirroring object: (2) No such file or directory 2024-04-06T10:15:52.162 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi192.stderr:2024-04-06T10:15:52.162+0000 7f3d22bc2640 -1 rbd::mirror::PoolWatcher: 0x5606637690e0 operator(): error unregistering watcher for rbd_mirroring object: (2) No such file or directory 2024-04-06T10:15:52.162 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi192.stderr:2024-04-06T10:15:52.162+0000 7f3d1bbb4640 -1 rbd::mirror::PoolWatcher: 0x560663769a40 operator(): error unregistering watcher for rbd_mirroring object: (2) No such file or directory 2024-04-06T10:15:52.162 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi192.stderr:2024-04-06T10:15:52.162+0000 7f3d1bbb4640 -1 rbd::mirror::PoolWatcher: 0x560663769860 operator(): error unregistering watcher for rbd_mirroring object: (2) No such file or directory 2024-04-06T10:15:52.163 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi192.stderr:2024-04-06T10:15:52.162+0000 7f3d28bce640 -1 rbd::mirror::InstanceWatcher: 0x5606649c0380 handle_unregister_watch: error unregistering instance watcher for rbd_mirror_instance.4268 object: (2) No such file or directory 2024-04-06T10:15:52.163 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi192.stderr:2024-04-06T10:15:52.162+0000 7f3d28bce640 -1 rbd::mirror::InstanceWatcher: 0x5606649c0700 handle_unregister_watch: error unregistering instance watcher for rbd_mirror_instance.4268 object: (2) No such file or directory 2024-04-06T10:15:52.163 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi192.stderr:2024-04-06T10:15:52.162+0000 7f3d223c1640 -1 rbd::mirror::InstanceWatcher: 0x5606649c0380 handle_unregister_instance: error unregistering instance: (2) No such file or directory 2024-04-06T10:15:52.163 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi192.stderr:2024-04-06T10:15:52.162+0000 7f3d22bc2640 -1 rbd::mirror::InstanceWatcher: 0x5606649c0700 handle_unregister_instance: error unregistering instance: (2) No such file or directory 2024-04-06T10:15:52.163 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi192.stderr:2024-04-06T10:15:52.162+0000 7f3d1bbb4640 -1 rbd::mirror::MirrorStatusUpdater 0x560664a365a0 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-04-06T10:15:52.163 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi192.stderr:2024-04-06T10:15:52.162+0000 7f3d1bbb4640 -1 rbd::mirror::MirrorStatusUpdater 0x560664a36360 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-04-06T10:15:52.164 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi192.stderr:2024-04-06T10:15:52.162+0000 7f3d293cf640 -1 rbd::mirror::NamespaceReplayer: 0x5606649be820 handle_shut_down_remote_status_updater: error shutting remote mirror status updater down: (2) No such file or directory 2024-04-06T10:15:52.164 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi192.stderr:2024-04-06T10:15:52.162+0000 7f3d293cf640 -1 rbd::mirror::NamespaceReplayer: 0x5606649be9c0 handle_shut_down_remote_status_updater: error shutting remote mirror status updater down: (2) No such file or directory 2024-04-06T10:15:52.164 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi192.stderr:2024-04-06T10:15:52.166+0000 7f3d22bc2640 -1 rbd::mirror::MirrorStatusUpdater 0x560664a36000 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-04-06T10:15:52.164 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi192.stderr:2024-04-06T10:15:52.166+0000 7f3d28bce640 -1 rbd::mirror::NamespaceReplayer: 0x5606649be820 handle_shut_down_local_status_updater: error shutting local mirror status updater down: (2) No such file or directory 2024-04-06T10:15:52.164 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi192.stderr:2024-04-06T10:15:52.166+0000 7f3d223c1640 -1 rbd::mirror::MirrorStatusUpdater 0x560664a36120 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-04-06T10:15:52.164 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi192.stderr:2024-04-06T10:15:52.166+0000 7f3d293cf640 -1 rbd::mirror::NamespaceReplayer: 0x5606649be9c0 handle_shut_down_local_status_updater: error shutting local mirror status updater down: (2) No such file or directory 2024-04-06T10:15:52.583 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi192.stderr:2024-04-06T10:15:52.582+0000 7f3d28bce640 -1 rbd::mirror::LeaderWatcher: 0x5606647d1500 handle_notify_heartbeat: error notifying heartbeat: (2) No such file or directory, releasing leader 2024-04-06T10:15:52.583 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi192.stderr:2024-04-06T10:15:52.582+0000 7f3d223c1640 -1 rbd::mirror::image_deleter::TrashWatcher: 0x560664a2e240 handle_unregister_watcher: error unregistering watcher for trash directory: (2) No such file or directory 2024-04-06T10:15:52.584 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi192.stderr:2024-04-06T10:15:52.586+0000 7f3d1c3b5640 -1 rbd::mirror::PoolWatcher: 0x5606650e8000 operator(): error unregistering watcher for rbd_mirroring object: (2) No such file or directory 2024-04-06T10:15:52.584 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi192.stderr:2024-04-06T10:15:52.586+0000 7f3d223c1640 -1 rbd::mirror::PoolWatcher: 0x5606637681e0 operator(): error unregistering watcher for rbd_mirroring object: (2) No such file or directory 2024-04-06T10:15:52.585 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi192.stderr:2024-04-06T10:15:52.586+0000 7f3d28bce640 -1 rbd::mirror::LeaderWatcher: 0x5606647d1500 handle_notify_lock_released: error notifying leader lock released: (2) No such file or directory 2024-04-06T10:15:52.585 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi192.stderr:2024-04-06T10:15:52.586+0000 7f3d28bce640 -1 rbd::mirror::LeaderWatcher: 0x5606647d1500 handle_release_leader_lock: error releasing lock: (2) No such file or directory 2024-04-06T10:15:53.688 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi192.stderr:2024-04-06T10:15:53.690+0000 7f3d0eb9a640 -1 rbd::mirror::PoolWatcher: 0x56066aaeba40 operator(): error unregistering watcher for rbd_mirroring object: (2) No such file or directory 2024-04-06T10:15:53.690 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi192.stderr:2024-04-06T10:15:53.690+0000 7f3d293cf640 -1 rbd::mirror::LeaderWatcher: 0x560664c4f500 handle_notify_lock_released: error notifying leader lock released: (2) No such file or directory 2024-04-06T10:15:53.690 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi192.stderr:2024-04-06T10:15:53.690+0000 7f3d293cf640 -1 rbd::mirror::LeaderWatcher: 0x560664c4f500 handle_release_leader_lock: error releasing lock: (2) No such file or directory 2024-04-06T10:15:53.691 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi192.stderr:2024-04-06T10:15:53.690+0000 7f3d28bce640 -1 rbd::mirror::LeaderWatcher: 0x560664c4f500 handle_unregister_watch: error unregistering leader watcher for rbd_mirror_leader object: (2) No such file or directory 2024-04-06T10:15:53.693 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi192.stderr:2024-04-06T10:15:53.694+0000 7f3d28bce640 -1 rbd::mirror::InstanceWatcher: 0x5606649c0a80 handle_unregister_watch: error unregistering instance watcher for rbd_mirror_instance.4287 object: (2) No such file or directory 2024-04-06T10:15:53.694 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi192.stderr:2024-04-06T10:15:53.694+0000 7f3d14ba6640 -1 rbd::mirror::InstanceWatcher: 0x5606649c0a80 handle_unregister_instance: error unregistering instance: (2) No such file or directory 2024-04-06T10:15:53.695 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi192.stderr:2024-04-06T10:15:53.694+0000 7f3d0eb9a640 -1 rbd::mirror::MirrorStatusUpdater 0x560664c765a0 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-04-06T10:15:53.695 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi192.stderr:2024-04-06T10:15:53.694+0000 7f3d28bce640 -1 rbd::mirror::NamespaceReplayer: 0x560664df24e0 handle_shut_down_remote_status_updater: error shutting remote mirror status updater down: (2) No such file or directory 2024-04-06T10:15:53.695 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi192.stderr:2024-04-06T10:15:53.698+0000 7f3d153a7640 -1 rbd::mirror::MirrorStatusUpdater 0x560664c5e360 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-04-06T10:15:53.696 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi192.stderr:2024-04-06T10:15:53.698+0000 7f3d293cf640 -1 rbd::mirror::NamespaceReplayer: 0x560664df24e0 handle_shut_down_local_status_updater: error shutting local mirror status updater down: (2) No such file or directory 2024-04-06T10:15:53.702 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi192.stderr:2024-04-06T10:15:53.702+0000 7f3d28bce640 -1 rbd::mirror::LeaderWatcher: 0x5606647d1500 handle_unregister_watch: error unregistering leader watcher for rbd_mirror_leader object: (2) No such file or directory 2024-04-06T10:15:53.703 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi192.stderr:2024-04-06T10:15:53.702+0000 7f3d28bce640 -1 rbd::mirror::InstanceWatcher: 0x5606649c0000 handle_unregister_watch: error unregistering instance watcher for rbd_mirror_instance.4268 object: (2) No such file or directory 2024-04-06T10:15:53.704 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi192.stderr:2024-04-06T10:15:53.706+0000 7f3d223c1640 -1 rbd::mirror::InstanceWatcher: 0x5606649c0000 handle_unregister_instance: error unregistering instance: (2) No such file or directory 2024-04-06T10:15:53.704 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi192.stderr:2024-04-06T10:15:53.706+0000 7f3d1c3b5640 -1 rbd::mirror::MirrorStatusUpdater 0x5606647ba6c0 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-04-06T10:15:53.705 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi192.stderr:2024-04-06T10:15:53.706+0000 7f3d28bce640 -1 rbd::mirror::NamespaceReplayer: 0x5606649be000 handle_shut_down_remote_status_updater: error shutting remote mirror status updater down: (2) No such file or directory 2024-04-06T10:15:53.705 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi192.stderr:2024-04-06T10:15:53.706+0000 7f3d223c1640 -1 rbd::mirror::MirrorStatusUpdater 0x56066534a6c0 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-04-06T10:15:53.706 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi192.stderr:2024-04-06T10:15:53.706+0000 7f3d293cf640 -1 rbd::mirror::NamespaceReplayer: 0x5606649be000 handle_shut_down_local_status_updater: error shutting local mirror status updater down: (2) No such file or directory 2024-04-06T10:15:54.007 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi192.stderr:2024-04-06T10:15:54.010+0000 7feb0bc47640 -1 rbd::mirror::InstanceWatcher: 0x55b7f203c700 handle_unregister_watch: error unregistering instance watcher for rbd_mirror_instance.5050 object: (2) No such file or directory 2024-04-06T10:15:54.007 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi192.stderr:2024-04-06T10:15:54.010+0000 7feb0bc47640 -1 rbd::mirror::InstanceWatcher: 0x55b7f203c380 handle_unregister_watch: error unregistering instance watcher for rbd_mirror_instance.5050 object: (2) No such file or directory 2024-04-06T10:15:54.007 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi192.stderr:2024-04-06T10:15:54.010+0000 7feb0543a640 -1 rbd::mirror::InstanceWatcher: 0x55b7f203c700 handle_unregister_instance: error unregistering instance: (2) No such file or directory 2024-04-06T10:15:54.007 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi192.stderr:2024-04-06T10:15:54.010+0000 7feb0543a640 -1 rbd::mirror::InstanceWatcher: 0x55b7f203c380 handle_unregister_instance: error unregistering instance: (2) No such file or directory 2024-04-06T10:15:54.008 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi192.stderr:2024-04-06T10:15:54.010+0000 7feafec2d640 -1 rbd::mirror::MirrorStatusUpdater 0x55b7f20ac7e0 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-04-06T10:15:54.008 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi192.stderr:2024-04-06T10:15:54.010+0000 7feafec2d640 -1 rbd::mirror::MirrorStatusUpdater 0x55b7f20ac360 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-04-06T10:15:54.008 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi192.stderr:2024-04-06T10:15:54.010+0000 7feb0bc47640 -1 rbd::mirror::NamespaceReplayer: 0x55b7f202e820 handle_shut_down_remote_status_updater: error shutting remote mirror status updater down: (2) No such file or directory 2024-04-06T10:15:54.008 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi192.stderr:2024-04-06T10:15:54.010+0000 7feb0bc47640 -1 rbd::mirror::NamespaceReplayer: 0x55b7f202e9c0 handle_shut_down_remote_status_updater: error shutting remote mirror status updater down: (2) No such file or directory 2024-04-06T10:15:54.009 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi192.stderr:2024-04-06T10:15:54.010+0000 7feb04c39640 -1 rbd::mirror::MirrorStatusUpdater 0x55b7f20ac000 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-04-06T10:15:54.009 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi192.stderr:2024-04-06T10:15:54.010+0000 7feb04c39640 -1 rbd::mirror::MirrorStatusUpdater 0x55b7f20ac120 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-04-06T10:15:54.009 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi192.stderr:2024-04-06T10:15:54.010+0000 7feb0b446640 -1 rbd::mirror::NamespaceReplayer: 0x55b7f202e820 handle_shut_down_local_status_updater: error shutting local mirror status updater down: (2) No such file or directory 2024-04-06T10:15:54.009 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi192.stderr:2024-04-06T10:15:54.010+0000 7feb0b446640 -1 rbd::mirror::NamespaceReplayer: 0x55b7f202e9c0 handle_shut_down_local_status_updater: error shutting local mirror status updater down: (2) No such file or directory 2024-04-06T10:15:54.217 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi192.stderr:2024-04-06T10:15:54.218+0000 7feb0bc47640 -1 rbd::mirror::LeaderWatcher: 0x55b7f2ca7b00 handle_unregister_watch: error unregistering leader watcher for rbd_mirror_leader object: (2) No such file or directory 2024-04-06T10:15:54.219 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi192.stderr:2024-04-06T10:15:54.218+0000 7feb0bc47640 -1 rbd::mirror::InstanceWatcher: 0x55b7f203c000 handle_unregister_watch: error unregistering instance watcher for rbd_mirror_instance.5050 object: (2) No such file or directory 2024-04-06T10:15:54.220 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi192.stderr:2024-04-06T10:15:54.222+0000 7feb04c39640 -1 rbd::mirror::InstanceWatcher: 0x55b7f203c000 handle_unregister_instance: error unregistering instance: (2) No such file or directory 2024-04-06T10:15:54.221 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi192.stderr:2024-04-06T10:15:54.222+0000 7feafec2d640 -1 rbd::mirror::MirrorStatusUpdater 0x55b7f2c8e6c0 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-04-06T10:15:54.221 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi192.stderr:2024-04-06T10:15:54.222+0000 7feb0bc47640 -1 rbd::mirror::NamespaceReplayer: 0x55b7f202e000 handle_shut_down_remote_status_updater: error shutting remote mirror status updater down: (2) No such file or directory 2024-04-06T10:15:54.222 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi192.stderr:2024-04-06T10:15:54.222+0000 7feb0543a640 -1 rbd::mirror::MirrorStatusUpdater 0x55b7f2c127e0 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-04-06T10:15:54.222 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi192.stderr:2024-04-06T10:15:54.222+0000 7feb0b446640 -1 rbd::mirror::NamespaceReplayer: 0x55b7f202e000 handle_shut_down_local_status_updater: error shutting local mirror status updater down: (2) No such file or directory 2024-04-06T10:15:54.227 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi192.stderr:2024-04-06T10:15:54.226+0000 7feb0bc47640 -1 rbd::mirror::LeaderWatcher: 0x55b7f2458300 handle_unregister_watch: error unregistering leader watcher for rbd_mirror_leader object: (2) No such file or directory 2024-04-06T10:15:54.229 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi192.stderr:2024-04-06T10:15:54.230+0000 7feb0bc47640 -1 rbd::mirror::InstanceWatcher: 0x55b7f203ca80 handle_unregister_watch: error unregistering instance watcher for rbd_mirror_instance.5053 object: (2) No such file or directory 2024-04-06T10:15:54.230 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi192.stderr:2024-04-06T10:15:54.230+0000 7feaf741e640 -1 rbd::mirror::InstanceWatcher: 0x55b7f203ca80 handle_unregister_instance: error unregistering instance: (2) No such file or directory 2024-04-06T10:15:54.230 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi192.stderr:2024-04-06T10:15:54.230+0000 7feaf1412640 -1 rbd::mirror::MirrorStatusUpdater 0x55b7f22aa7e0 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-04-06T10:15:54.230 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi192.stderr:2024-04-06T10:15:54.230+0000 7feb0bc47640 -1 rbd::mirror::NamespaceReplayer: 0x55b7f245e4e0 handle_shut_down_remote_status_updater: error shutting remote mirror status updater down: (2) No such file or directory 2024-04-06T10:15:54.231 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi192.stderr:2024-04-06T10:15:54.230+0000 7feaf7c1f640 -1 rbd::mirror::MirrorStatusUpdater 0x55b7f226a240 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-04-06T10:15:54.231 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi192.stderr:2024-04-06T10:15:54.230+0000 7feb0b446640 -1 rbd::mirror::NamespaceReplayer: 0x55b7f245e4e0 handle_shut_down_local_status_updater: error shutting local mirror status updater down: (2) No such file or directory 2024-04-06T10:15:58.262 INFO:tasks.rbd_mirror.cluster1.client.mirror.1:Stopped 2024-04-06T10:15:58.262 DEBUG:teuthology.run_tasks:Unwinding manager rbd-mirror 2024-04-06T10:15:58.273 DEBUG:tasks.rbd_mirror.cluster1.client.mirror.0:waiting for process to exit 2024-04-06T10:15:58.273 INFO:teuthology.orchestra.run:waiting for 300 2024-04-06T10:15:58.274 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi192.stderr:2024-04-06T10:15:58.273+0000 7feb0c448640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.0 (PID: 167561) UID: 1000 2024-04-06T10:15:58.298 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Stopped 2024-04-06T10:15:58.298 DEBUG:teuthology.run_tasks:Unwinding manager ceph 2024-04-06T10:15:58.308 INFO:tasks.ceph.ceph_manager.cluster2:waiting for clean 2024-04-06T10:15:58.309 DEBUG:teuthology.orchestra.run.smithi192:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2024-04-06T10:15:58.635 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-04-06T10:15:58.635 INFO:teuthology.orchestra.run.smithi192.stderr:dumped all 2024-04-06T10:15:58.652 INFO:teuthology.orchestra.run.smithi192.stdout:{"pg_ready":true,"pg_map":{"version":1784,"stamp":"2024-04-06T10:15:57.975766+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":4241876,"kb_used_data":14196,"kb_used_omap":455,"kb_used_meta":4227192,"kb_avail":278873644,"statfs":{"total":289910292480,"available":285566611456,"internally_reserved":0,"allocated":14536704,"data_stored":8030891,"data_compressed":60694,"data_compressed_allocated":81920,"data_compressed_original":1146880,"omap_allocated":466269,"internal_metadata":4328645283},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":44,"apply_latency_ms":44,"commit_latency_ns":44000000,"apply_latency_ns":44000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-19,"num_objects":-8,"num_object_clones":0,"num_object_copies":-16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-8,"num_whiteouts":0,"num_read":-2605,"num_read_kb":-2479,"num_write":-498,"num_write_kb":-384,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":-2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.133018"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":422,"reported_epoch":404,"state":"active+clean","last_fresh":"2024-04-06T10:15:49.859638+0000","last_change":"2024-04-06T09:30:02.384212+0000","last_active":"2024-04-06T10:15:49.859638+0000","last_peered":"2024-04-06T10:15:49.859638+0000","last_clean":"2024-04-06T10:15:49.859638+0000","last_became_active":"2024-04-06T09:30:02.383710+0000","last_became_peered":"2024-04-06T09:30:02.383710+0000","last_unstale":"2024-04-06T10:15:49.859638+0000","last_undegraded":"2024-04-06T10:15:49.859638+0000","last_fullsized":"2024-04-06T10:15:49.859638+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T09:30:01.359377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T09:30:01.359377+0000","last_clean_scrub_stamp":"2024-04-06T09:30:01.359377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T18:51:44.859927+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":422,"reported_epoch":404,"state":"active+clean","last_fresh":"2024-04-06T10:15:49.854844+0000","last_change":"2024-04-06T09:30:02.385797+0000","last_active":"2024-04-06T10:15:49.854844+0000","last_peered":"2024-04-06T10:15:49.854844+0000","last_clean":"2024-04-06T10:15:49.854844+0000","last_became_active":"2024-04-06T09:30:02.385501+0000","last_became_peered":"2024-04-06T09:30:02.385501+0000","last_unstale":"2024-04-06T10:15:49.854844+0000","last_undegraded":"2024-04-06T10:15:49.854844+0000","last_fullsized":"2024-04-06T10:15:49.854844+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T09:30:01.359377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T09:30:01.359377+0000","last_clean_scrub_stamp":"2024-04-06T09:30:01.359377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T15:08:40.865789+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":422,"reported_epoch":404,"state":"active+clean","last_fresh":"2024-04-06T10:15:49.859497+0000","last_change":"2024-04-06T09:30:02.383199+0000","last_active":"2024-04-06T10:15:49.859497+0000","last_peered":"2024-04-06T10:15:49.859497+0000","last_clean":"2024-04-06T10:15:49.859497+0000","last_became_active":"2024-04-06T09:30:02.382798+0000","last_became_peered":"2024-04-06T09:30:02.382798+0000","last_unstale":"2024-04-06T10:15:49.859497+0000","last_undegraded":"2024-04-06T10:15:49.859497+0000","last_fullsized":"2024-04-06T10:15:49.859497+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T09:30:01.359377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T09:30:01.359377+0000","last_clean_scrub_stamp":"2024-04-06T09:30:01.359377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T18:11:34.478290+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":423,"reported_epoch":405,"state":"active+clean","last_fresh":"2024-04-06T10:15:51.159679+0000","last_change":"2024-04-06T09:30:02.383483+0000","last_active":"2024-04-06T10:15:51.159679+0000","last_peered":"2024-04-06T10:15:51.159679+0000","last_clean":"2024-04-06T10:15:51.159679+0000","last_became_active":"2024-04-06T09:30:02.382963+0000","last_became_peered":"2024-04-06T09:30:02.382963+0000","last_unstale":"2024-04-06T10:15:51.159679+0000","last_undegraded":"2024-04-06T10:15:51.159679+0000","last_fullsized":"2024-04-06T10:15:51.159679+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T09:30:01.359377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T09:30:01.359377+0000","last_clean_scrub_stamp":"2024-04-06T09:30:01.359377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T10:19:04.998532+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":422,"reported_epoch":405,"state":"active+clean","last_fresh":"2024-04-06T10:15:52.035138+0000","last_change":"2024-04-06T09:30:02.383104+0000","last_active":"2024-04-06T10:15:52.035138+0000","last_peered":"2024-04-06T10:15:52.035138+0000","last_clean":"2024-04-06T10:15:52.035138+0000","last_became_active":"2024-04-06T09:30:02.382554+0000","last_became_peered":"2024-04-06T09:30:02.382554+0000","last_unstale":"2024-04-06T10:15:52.035138+0000","last_undegraded":"2024-04-06T10:15:52.035138+0000","last_fullsized":"2024-04-06T10:15:52.035138+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T09:30:01.359377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T09:30:01.359377+0000","last_clean_scrub_stamp":"2024-04-06T09:30:01.359377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T18:47:16.568825+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":421,"reported_epoch":405,"state":"active+clean","last_fresh":"2024-04-06T10:15:50.877447+0000","last_change":"2024-04-06T09:30:02.386812+0000","last_active":"2024-04-06T10:15:50.877447+0000","last_peered":"2024-04-06T10:15:50.877447+0000","last_clean":"2024-04-06T10:15:50.877447+0000","last_became_active":"2024-04-06T09:30:02.386256+0000","last_became_peered":"2024-04-06T09:30:02.386256+0000","last_unstale":"2024-04-06T10:15:50.877447+0000","last_undegraded":"2024-04-06T10:15:50.877447+0000","last_fullsized":"2024-04-06T10:15:50.877447+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T09:30:01.359377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T09:30:01.359377+0000","last_clean_scrub_stamp":"2024-04-06T09:30:01.359377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T21:22:07.757399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":421,"reported_epoch":403,"state":"active+clean","last_fresh":"2024-04-06T10:15:47.030627+0000","last_change":"2024-04-06T09:30:02.386748+0000","last_active":"2024-04-06T10:15:47.030627+0000","last_peered":"2024-04-06T10:15:47.030627+0000","last_clean":"2024-04-06T10:15:47.030627+0000","last_became_active":"2024-04-06T09:30:02.386318+0000","last_became_peered":"2024-04-06T09:30:02.386318+0000","last_unstale":"2024-04-06T10:15:47.030627+0000","last_undegraded":"2024-04-06T10:15:47.030627+0000","last_fullsized":"2024-04-06T10:15:47.030627+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T09:30:01.359377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T09:30:01.359377+0000","last_clean_scrub_stamp":"2024-04-06T09:30:01.359377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T10:50:42.918574+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":423,"reported_epoch":405,"state":"active+clean","last_fresh":"2024-04-06T10:15:51.206729+0000","last_change":"2024-04-06T09:30:02.382580+0000","last_active":"2024-04-06T10:15:51.206729+0000","last_peered":"2024-04-06T10:15:51.206729+0000","last_clean":"2024-04-06T10:15:51.206729+0000","last_became_active":"2024-04-06T09:30:02.382103+0000","last_became_peered":"2024-04-06T09:30:02.382103+0000","last_unstale":"2024-04-06T10:15:51.206729+0000","last_undegraded":"2024-04-06T10:15:51.206729+0000","last_fullsized":"2024-04-06T10:15:51.206729+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T09:30:01.359377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T09:30:01.359377+0000","last_clean_scrub_stamp":"2024-04-06T09:30:01.359377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T18:50:24.144774+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"10'97","reported_seq":553,"reported_epoch":404,"state":"active+clean","last_fresh":"2024-04-06T10:15:49.853583+0000","last_change":"2024-04-06T09:29:56.421931+0000","last_active":"2024-04-06T10:15:49.853583+0000","last_peered":"2024-04-06T10:15:49.853583+0000","last_clean":"2024-04-06T10:15:49.853583+0000","last_became_active":"2024-04-06T09:29:56.419754+0000","last_became_peered":"2024-04-06T09:29:56.419754+0000","last_unstale":"2024-04-06T10:15:49.853583+0000","last_undegraded":"2024-04-06T10:15:49.853583+0000","last_fullsized":"2024-04-06T10:15:49.853583+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T09:29:55.213061+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T09:29:55.213061+0000","last_clean_scrub_stamp":"2024-04-06T09:29:55.213061+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T19:28:59.898297+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":122880,"data_stored":1180736,"data_compressed":60694,"data_compressed_allocated":81920,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738926,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1740896,"kb_used_data":4692,"kb_used_omap":113,"kb_used_meta":1736078,"kb_avail":92630944,"statfs":{"total":96636764160,"available":94854086656,"internally_reserved":0,"allocated":4804608,"data_stored":2283385,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":116632,"internal_metadata":1777743976},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 6 10:15:16 2024","interfaces":[{"interface":"back","average":{"1min":1.048,"5min":0.60499999999999998,"15min":0.50600000000000001},"min":{"1min":0.32900000000000001,"5min":0.156,"15min":0.156},"max":{"1min":9.9320000000000004,"5min":9.9320000000000004,"15min":9.9320000000000004},"last":0.38600000000000001},{"interface":"front","average":{"1min":0.52700000000000002,"5min":0.55100000000000005,"15min":0.53800000000000003},"min":{"1min":0.22600000000000001,"5min":0.18099999999999999,"15min":0.18099999999999999},"max":{"1min":0.80400000000000005,"5min":3.484,"15min":3.484},"last":0.46800000000000003}]},{"osd":1,"last update":"Sat Apr 6 10:15:16 2024","interfaces":[{"interface":"back","average":{"1min":1.079,"5min":0.70499999999999996,"15min":0.60199999999999998},"min":{"1min":0.30599999999999999,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":9.8179999999999996,"5min":9.8179999999999996,"15min":9.8179999999999996},"last":0.52200000000000002},{"interface":"front","average":{"1min":1.169,"5min":0.79000000000000004,"15min":0.68500000000000005},"min":{"1min":0.35799999999999998,"5min":0.23999999999999999,"15min":0.23999999999999999},"max":{"1min":9.8469999999999995,"5min":9.8469999999999995,"15min":9.8469999999999995},"last":0.56299999999999994}]}]},{"osd":1,"up_from":8,"seq":34359738923,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1296216,"kb_used_data":4752,"kb_used_omap":162,"kb_used_meta":1291293,"kb_avail":93075624,"statfs":{"total":96636764160,"available":95309438976,"internally_reserved":0,"allocated":4866048,"data_stored":2873753,"data_compressed":30347,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":166081,"internal_metadata":1322284863},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 6 10:15:08 2024","interfaces":[{"interface":"back","average":{"1min":0.41999999999999998,"5min":0.496,"15min":0.50700000000000001},"min":{"1min":0.19800000000000001,"5min":0.17999999999999999,"15min":0.16},"max":{"1min":0.60699999999999998,"5min":3.976,"15min":9.5169999999999995},"last":0.50700000000000001},{"interface":"front","average":{"1min":0.43099999999999999,"5min":0.47599999999999998,"15min":0.50900000000000001},"min":{"1min":0.193,"5min":0.193,"15min":0.19},"max":{"1min":0.63700000000000001,"5min":1.0349999999999999,"15min":3.7709999999999999},"last":0.57999999999999996}]},{"osd":2,"last update":"Sat Apr 6 10:15:08 2024","interfaces":[{"interface":"back","average":{"1min":0.42799999999999999,"5min":0.53100000000000003,"15min":0.52100000000000002},"min":{"1min":0.20000000000000001,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":0.61499999999999999,"5min":3.6360000000000001,"15min":3.6360000000000001},"last":0.54200000000000004},{"interface":"front","average":{"1min":0.503,"5min":0.56200000000000006,"15min":0.59099999999999997},"min":{"1min":0.23200000000000001,"5min":0.23200000000000001,"15min":0.23200000000000001},"max":{"1min":0.67700000000000005,"5min":1.452,"15min":2.5099999999999998},"last":0.627}]}]},{"osd":0,"up_from":8,"seq":34359738925,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1204764,"kb_used_data":4752,"kb_used_omap":179,"kb_used_meta":1199820,"kb_avail":93167076,"statfs":{"total":96636764160,"available":95403085824,"internally_reserved":0,"allocated":4866048,"data_stored":2873753,"data_compressed":30347,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":183556,"internal_metadata":1228616444},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":41,"apply_latency_ms":41,"commit_latency_ns":41000000,"apply_latency_ns":41000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Apr 6 10:15:17 2024","interfaces":[{"interface":"back","average":{"1min":0.42899999999999999,"5min":0.45000000000000001,"15min":0.45900000000000002},"min":{"1min":0.189,"5min":0.159,"15min":0.152},"max":{"1min":0.72499999999999998,"5min":0.73099999999999998,"15min":6.5620000000000003},"last":0.65800000000000003},{"interface":"front","average":{"1min":0.49299999999999999,"5min":0.53600000000000003,"15min":0.56699999999999995},"min":{"1min":0.16300000000000001,"5min":0.16300000000000001,"15min":0.16300000000000001},"max":{"1min":0.755,"5min":0.79400000000000004,"15min":3.4820000000000002},"last":0.68999999999999995}]},{"osd":2,"last update":"Sat Apr 6 10:15:23 2024","interfaces":[{"interface":"back","average":{"1min":0.432,"5min":0.47499999999999998,"15min":0.54200000000000004},"min":{"1min":0.189,"5min":0.17199999999999999,"15min":0.17199999999999999},"max":{"1min":0.66300000000000003,"5min":0.70399999999999996,"15min":6.5640000000000001},"last":0.51800000000000002},{"interface":"front","average":{"1min":0.55200000000000005,"5min":0.57399999999999995,"15min":0.56200000000000006},"min":{"1min":0.252,"5min":0.191,"15min":0.184},"max":{"1min":0.79000000000000004,"5min":0.89600000000000002,"15min":2.431},"last":0.61599999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":590368,"data_compressed":30347,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":590368,"data_compressed":30347,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-06T10:15:58.653 DEBUG:teuthology.orchestra.run.smithi192:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2024-04-06T10:15:58.930 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-04-06T10:15:58.930 INFO:teuthology.orchestra.run.smithi192.stderr:dumped all 2024-04-06T10:15:58.947 INFO:teuthology.orchestra.run.smithi192.stdout:{"pg_ready":true,"pg_map":{"version":1784,"stamp":"2024-04-06T10:15:57.975766+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":4241876,"kb_used_data":14196,"kb_used_omap":455,"kb_used_meta":4227192,"kb_avail":278873644,"statfs":{"total":289910292480,"available":285566611456,"internally_reserved":0,"allocated":14536704,"data_stored":8030891,"data_compressed":60694,"data_compressed_allocated":81920,"data_compressed_original":1146880,"omap_allocated":466269,"internal_metadata":4328645283},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":44,"apply_latency_ms":44,"commit_latency_ns":44000000,"apply_latency_ns":44000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-19,"num_objects":-8,"num_object_clones":0,"num_object_copies":-16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-8,"num_whiteouts":0,"num_read":-2605,"num_read_kb":-2479,"num_write":-498,"num_write_kb":-384,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":-2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.133018"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":422,"reported_epoch":404,"state":"active+clean","last_fresh":"2024-04-06T10:15:49.859638+0000","last_change":"2024-04-06T09:30:02.384212+0000","last_active":"2024-04-06T10:15:49.859638+0000","last_peered":"2024-04-06T10:15:49.859638+0000","last_clean":"2024-04-06T10:15:49.859638+0000","last_became_active":"2024-04-06T09:30:02.383710+0000","last_became_peered":"2024-04-06T09:30:02.383710+0000","last_unstale":"2024-04-06T10:15:49.859638+0000","last_undegraded":"2024-04-06T10:15:49.859638+0000","last_fullsized":"2024-04-06T10:15:49.859638+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T09:30:01.359377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T09:30:01.359377+0000","last_clean_scrub_stamp":"2024-04-06T09:30:01.359377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T18:51:44.859927+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":422,"reported_epoch":404,"state":"active+clean","last_fresh":"2024-04-06T10:15:49.854844+0000","last_change":"2024-04-06T09:30:02.385797+0000","last_active":"2024-04-06T10:15:49.854844+0000","last_peered":"2024-04-06T10:15:49.854844+0000","last_clean":"2024-04-06T10:15:49.854844+0000","last_became_active":"2024-04-06T09:30:02.385501+0000","last_became_peered":"2024-04-06T09:30:02.385501+0000","last_unstale":"2024-04-06T10:15:49.854844+0000","last_undegraded":"2024-04-06T10:15:49.854844+0000","last_fullsized":"2024-04-06T10:15:49.854844+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T09:30:01.359377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T09:30:01.359377+0000","last_clean_scrub_stamp":"2024-04-06T09:30:01.359377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T15:08:40.865789+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":422,"reported_epoch":404,"state":"active+clean","last_fresh":"2024-04-06T10:15:49.859497+0000","last_change":"2024-04-06T09:30:02.383199+0000","last_active":"2024-04-06T10:15:49.859497+0000","last_peered":"2024-04-06T10:15:49.859497+0000","last_clean":"2024-04-06T10:15:49.859497+0000","last_became_active":"2024-04-06T09:30:02.382798+0000","last_became_peered":"2024-04-06T09:30:02.382798+0000","last_unstale":"2024-04-06T10:15:49.859497+0000","last_undegraded":"2024-04-06T10:15:49.859497+0000","last_fullsized":"2024-04-06T10:15:49.859497+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T09:30:01.359377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T09:30:01.359377+0000","last_clean_scrub_stamp":"2024-04-06T09:30:01.359377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T18:11:34.478290+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":423,"reported_epoch":405,"state":"active+clean","last_fresh":"2024-04-06T10:15:51.159679+0000","last_change":"2024-04-06T09:30:02.383483+0000","last_active":"2024-04-06T10:15:51.159679+0000","last_peered":"2024-04-06T10:15:51.159679+0000","last_clean":"2024-04-06T10:15:51.159679+0000","last_became_active":"2024-04-06T09:30:02.382963+0000","last_became_peered":"2024-04-06T09:30:02.382963+0000","last_unstale":"2024-04-06T10:15:51.159679+0000","last_undegraded":"2024-04-06T10:15:51.159679+0000","last_fullsized":"2024-04-06T10:15:51.159679+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T09:30:01.359377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T09:30:01.359377+0000","last_clean_scrub_stamp":"2024-04-06T09:30:01.359377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T10:19:04.998532+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":422,"reported_epoch":405,"state":"active+clean","last_fresh":"2024-04-06T10:15:52.035138+0000","last_change":"2024-04-06T09:30:02.383104+0000","last_active":"2024-04-06T10:15:52.035138+0000","last_peered":"2024-04-06T10:15:52.035138+0000","last_clean":"2024-04-06T10:15:52.035138+0000","last_became_active":"2024-04-06T09:30:02.382554+0000","last_became_peered":"2024-04-06T09:30:02.382554+0000","last_unstale":"2024-04-06T10:15:52.035138+0000","last_undegraded":"2024-04-06T10:15:52.035138+0000","last_fullsized":"2024-04-06T10:15:52.035138+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T09:30:01.359377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T09:30:01.359377+0000","last_clean_scrub_stamp":"2024-04-06T09:30:01.359377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T18:47:16.568825+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":421,"reported_epoch":405,"state":"active+clean","last_fresh":"2024-04-06T10:15:50.877447+0000","last_change":"2024-04-06T09:30:02.386812+0000","last_active":"2024-04-06T10:15:50.877447+0000","last_peered":"2024-04-06T10:15:50.877447+0000","last_clean":"2024-04-06T10:15:50.877447+0000","last_became_active":"2024-04-06T09:30:02.386256+0000","last_became_peered":"2024-04-06T09:30:02.386256+0000","last_unstale":"2024-04-06T10:15:50.877447+0000","last_undegraded":"2024-04-06T10:15:50.877447+0000","last_fullsized":"2024-04-06T10:15:50.877447+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T09:30:01.359377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T09:30:01.359377+0000","last_clean_scrub_stamp":"2024-04-06T09:30:01.359377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T21:22:07.757399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":421,"reported_epoch":403,"state":"active+clean","last_fresh":"2024-04-06T10:15:47.030627+0000","last_change":"2024-04-06T09:30:02.386748+0000","last_active":"2024-04-06T10:15:47.030627+0000","last_peered":"2024-04-06T10:15:47.030627+0000","last_clean":"2024-04-06T10:15:47.030627+0000","last_became_active":"2024-04-06T09:30:02.386318+0000","last_became_peered":"2024-04-06T09:30:02.386318+0000","last_unstale":"2024-04-06T10:15:47.030627+0000","last_undegraded":"2024-04-06T10:15:47.030627+0000","last_fullsized":"2024-04-06T10:15:47.030627+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T09:30:01.359377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T09:30:01.359377+0000","last_clean_scrub_stamp":"2024-04-06T09:30:01.359377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T10:50:42.918574+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":423,"reported_epoch":405,"state":"active+clean","last_fresh":"2024-04-06T10:15:51.206729+0000","last_change":"2024-04-06T09:30:02.382580+0000","last_active":"2024-04-06T10:15:51.206729+0000","last_peered":"2024-04-06T10:15:51.206729+0000","last_clean":"2024-04-06T10:15:51.206729+0000","last_became_active":"2024-04-06T09:30:02.382103+0000","last_became_peered":"2024-04-06T09:30:02.382103+0000","last_unstale":"2024-04-06T10:15:51.206729+0000","last_undegraded":"2024-04-06T10:15:51.206729+0000","last_fullsized":"2024-04-06T10:15:51.206729+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T09:30:01.359377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T09:30:01.359377+0000","last_clean_scrub_stamp":"2024-04-06T09:30:01.359377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T18:50:24.144774+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"10'97","reported_seq":553,"reported_epoch":404,"state":"active+clean","last_fresh":"2024-04-06T10:15:49.853583+0000","last_change":"2024-04-06T09:29:56.421931+0000","last_active":"2024-04-06T10:15:49.853583+0000","last_peered":"2024-04-06T10:15:49.853583+0000","last_clean":"2024-04-06T10:15:49.853583+0000","last_became_active":"2024-04-06T09:29:56.419754+0000","last_became_peered":"2024-04-06T09:29:56.419754+0000","last_unstale":"2024-04-06T10:15:49.853583+0000","last_undegraded":"2024-04-06T10:15:49.853583+0000","last_fullsized":"2024-04-06T10:15:49.853583+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T09:29:55.213061+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T09:29:55.213061+0000","last_clean_scrub_stamp":"2024-04-06T09:29:55.213061+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T19:28:59.898297+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":122880,"data_stored":1180736,"data_compressed":60694,"data_compressed_allocated":81920,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738926,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1740896,"kb_used_data":4692,"kb_used_omap":113,"kb_used_meta":1736078,"kb_avail":92630944,"statfs":{"total":96636764160,"available":94854086656,"internally_reserved":0,"allocated":4804608,"data_stored":2283385,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":116632,"internal_metadata":1777743976},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 6 10:15:16 2024","interfaces":[{"interface":"back","average":{"1min":1.048,"5min":0.60499999999999998,"15min":0.50600000000000001},"min":{"1min":0.32900000000000001,"5min":0.156,"15min":0.156},"max":{"1min":9.9320000000000004,"5min":9.9320000000000004,"15min":9.9320000000000004},"last":0.38600000000000001},{"interface":"front","average":{"1min":0.52700000000000002,"5min":0.55100000000000005,"15min":0.53800000000000003},"min":{"1min":0.22600000000000001,"5min":0.18099999999999999,"15min":0.18099999999999999},"max":{"1min":0.80400000000000005,"5min":3.484,"15min":3.484},"last":0.46800000000000003}]},{"osd":1,"last update":"Sat Apr 6 10:15:16 2024","interfaces":[{"interface":"back","average":{"1min":1.079,"5min":0.70499999999999996,"15min":0.60199999999999998},"min":{"1min":0.30599999999999999,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":9.8179999999999996,"5min":9.8179999999999996,"15min":9.8179999999999996},"last":0.52200000000000002},{"interface":"front","average":{"1min":1.169,"5min":0.79000000000000004,"15min":0.68500000000000005},"min":{"1min":0.35799999999999998,"5min":0.23999999999999999,"15min":0.23999999999999999},"max":{"1min":9.8469999999999995,"5min":9.8469999999999995,"15min":9.8469999999999995},"last":0.56299999999999994}]}]},{"osd":1,"up_from":8,"seq":34359738923,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1296216,"kb_used_data":4752,"kb_used_omap":162,"kb_used_meta":1291293,"kb_avail":93075624,"statfs":{"total":96636764160,"available":95309438976,"internally_reserved":0,"allocated":4866048,"data_stored":2873753,"data_compressed":30347,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":166081,"internal_metadata":1322284863},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 6 10:15:08 2024","interfaces":[{"interface":"back","average":{"1min":0.41999999999999998,"5min":0.496,"15min":0.50700000000000001},"min":{"1min":0.19800000000000001,"5min":0.17999999999999999,"15min":0.16},"max":{"1min":0.60699999999999998,"5min":3.976,"15min":9.5169999999999995},"last":0.50700000000000001},{"interface":"front","average":{"1min":0.43099999999999999,"5min":0.47599999999999998,"15min":0.50900000000000001},"min":{"1min":0.193,"5min":0.193,"15min":0.19},"max":{"1min":0.63700000000000001,"5min":1.0349999999999999,"15min":3.7709999999999999},"last":0.57999999999999996}]},{"osd":2,"last update":"Sat Apr 6 10:15:08 2024","interfaces":[{"interface":"back","average":{"1min":0.42799999999999999,"5min":0.53100000000000003,"15min":0.52100000000000002},"min":{"1min":0.20000000000000001,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":0.61499999999999999,"5min":3.6360000000000001,"15min":3.6360000000000001},"last":0.54200000000000004},{"interface":"front","average":{"1min":0.503,"5min":0.56200000000000006,"15min":0.59099999999999997},"min":{"1min":0.23200000000000001,"5min":0.23200000000000001,"15min":0.23200000000000001},"max":{"1min":0.67700000000000005,"5min":1.452,"15min":2.5099999999999998},"last":0.627}]}]},{"osd":0,"up_from":8,"seq":34359738925,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1204764,"kb_used_data":4752,"kb_used_omap":179,"kb_used_meta":1199820,"kb_avail":93167076,"statfs":{"total":96636764160,"available":95403085824,"internally_reserved":0,"allocated":4866048,"data_stored":2873753,"data_compressed":30347,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":183556,"internal_metadata":1228616444},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":41,"apply_latency_ms":41,"commit_latency_ns":41000000,"apply_latency_ns":41000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Apr 6 10:15:17 2024","interfaces":[{"interface":"back","average":{"1min":0.42899999999999999,"5min":0.45000000000000001,"15min":0.45900000000000002},"min":{"1min":0.189,"5min":0.159,"15min":0.152},"max":{"1min":0.72499999999999998,"5min":0.73099999999999998,"15min":6.5620000000000003},"last":0.65800000000000003},{"interface":"front","average":{"1min":0.49299999999999999,"5min":0.53600000000000003,"15min":0.56699999999999995},"min":{"1min":0.16300000000000001,"5min":0.16300000000000001,"15min":0.16300000000000001},"max":{"1min":0.755,"5min":0.79400000000000004,"15min":3.4820000000000002},"last":0.68999999999999995}]},{"osd":2,"last update":"Sat Apr 6 10:15:23 2024","interfaces":[{"interface":"back","average":{"1min":0.432,"5min":0.47499999999999998,"15min":0.54200000000000004},"min":{"1min":0.189,"5min":0.17199999999999999,"15min":0.17199999999999999},"max":{"1min":0.66300000000000003,"5min":0.70399999999999996,"15min":6.5640000000000001},"last":0.51800000000000002},{"interface":"front","average":{"1min":0.55200000000000005,"5min":0.57399999999999995,"15min":0.56200000000000006},"min":{"1min":0.252,"5min":0.191,"15min":0.184},"max":{"1min":0.79000000000000004,"5min":0.89600000000000002,"15min":2.431},"last":0.61599999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":590368,"data_compressed":30347,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":590368,"data_compressed":30347,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-06T10:15:58.948 INFO:tasks.ceph.ceph_manager.cluster2:clean! 2024-04-06T10:15:58.948 DEBUG:teuthology.orchestra.run.smithi192:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2024-04-06T10:15:59.228 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-04-06T10:15:59.228 INFO:teuthology.orchestra.run.smithi192.stderr:dumped all 2024-04-06T10:15:59.245 INFO:teuthology.orchestra.run.smithi192.stdout:{"pg_ready":true,"pg_map":{"version":1784,"stamp":"2024-04-06T10:15:57.975766+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":4241876,"kb_used_data":14196,"kb_used_omap":455,"kb_used_meta":4227192,"kb_avail":278873644,"statfs":{"total":289910292480,"available":285566611456,"internally_reserved":0,"allocated":14536704,"data_stored":8030891,"data_compressed":60694,"data_compressed_allocated":81920,"data_compressed_original":1146880,"omap_allocated":466269,"internal_metadata":4328645283},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":44,"apply_latency_ms":44,"commit_latency_ns":44000000,"apply_latency_ns":44000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-19,"num_objects":-8,"num_object_clones":0,"num_object_copies":-16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-8,"num_whiteouts":0,"num_read":-2605,"num_read_kb":-2479,"num_write":-498,"num_write_kb":-384,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":-2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.133018"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":422,"reported_epoch":404,"state":"active+clean","last_fresh":"2024-04-06T10:15:49.859638+0000","last_change":"2024-04-06T09:30:02.384212+0000","last_active":"2024-04-06T10:15:49.859638+0000","last_peered":"2024-04-06T10:15:49.859638+0000","last_clean":"2024-04-06T10:15:49.859638+0000","last_became_active":"2024-04-06T09:30:02.383710+0000","last_became_peered":"2024-04-06T09:30:02.383710+0000","last_unstale":"2024-04-06T10:15:49.859638+0000","last_undegraded":"2024-04-06T10:15:49.859638+0000","last_fullsized":"2024-04-06T10:15:49.859638+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T09:30:01.359377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T09:30:01.359377+0000","last_clean_scrub_stamp":"2024-04-06T09:30:01.359377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T18:51:44.859927+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":422,"reported_epoch":404,"state":"active+clean","last_fresh":"2024-04-06T10:15:49.854844+0000","last_change":"2024-04-06T09:30:02.385797+0000","last_active":"2024-04-06T10:15:49.854844+0000","last_peered":"2024-04-06T10:15:49.854844+0000","last_clean":"2024-04-06T10:15:49.854844+0000","last_became_active":"2024-04-06T09:30:02.385501+0000","last_became_peered":"2024-04-06T09:30:02.385501+0000","last_unstale":"2024-04-06T10:15:49.854844+0000","last_undegraded":"2024-04-06T10:15:49.854844+0000","last_fullsized":"2024-04-06T10:15:49.854844+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T09:30:01.359377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T09:30:01.359377+0000","last_clean_scrub_stamp":"2024-04-06T09:30:01.359377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T15:08:40.865789+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":422,"reported_epoch":404,"state":"active+clean","last_fresh":"2024-04-06T10:15:49.859497+0000","last_change":"2024-04-06T09:30:02.383199+0000","last_active":"2024-04-06T10:15:49.859497+0000","last_peered":"2024-04-06T10:15:49.859497+0000","last_clean":"2024-04-06T10:15:49.859497+0000","last_became_active":"2024-04-06T09:30:02.382798+0000","last_became_peered":"2024-04-06T09:30:02.382798+0000","last_unstale":"2024-04-06T10:15:49.859497+0000","last_undegraded":"2024-04-06T10:15:49.859497+0000","last_fullsized":"2024-04-06T10:15:49.859497+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T09:30:01.359377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T09:30:01.359377+0000","last_clean_scrub_stamp":"2024-04-06T09:30:01.359377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T18:11:34.478290+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":423,"reported_epoch":405,"state":"active+clean","last_fresh":"2024-04-06T10:15:51.159679+0000","last_change":"2024-04-06T09:30:02.383483+0000","last_active":"2024-04-06T10:15:51.159679+0000","last_peered":"2024-04-06T10:15:51.159679+0000","last_clean":"2024-04-06T10:15:51.159679+0000","last_became_active":"2024-04-06T09:30:02.382963+0000","last_became_peered":"2024-04-06T09:30:02.382963+0000","last_unstale":"2024-04-06T10:15:51.159679+0000","last_undegraded":"2024-04-06T10:15:51.159679+0000","last_fullsized":"2024-04-06T10:15:51.159679+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T09:30:01.359377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T09:30:01.359377+0000","last_clean_scrub_stamp":"2024-04-06T09:30:01.359377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T10:19:04.998532+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":422,"reported_epoch":405,"state":"active+clean","last_fresh":"2024-04-06T10:15:52.035138+0000","last_change":"2024-04-06T09:30:02.383104+0000","last_active":"2024-04-06T10:15:52.035138+0000","last_peered":"2024-04-06T10:15:52.035138+0000","last_clean":"2024-04-06T10:15:52.035138+0000","last_became_active":"2024-04-06T09:30:02.382554+0000","last_became_peered":"2024-04-06T09:30:02.382554+0000","last_unstale":"2024-04-06T10:15:52.035138+0000","last_undegraded":"2024-04-06T10:15:52.035138+0000","last_fullsized":"2024-04-06T10:15:52.035138+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T09:30:01.359377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T09:30:01.359377+0000","last_clean_scrub_stamp":"2024-04-06T09:30:01.359377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T18:47:16.568825+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":421,"reported_epoch":405,"state":"active+clean","last_fresh":"2024-04-06T10:15:50.877447+0000","last_change":"2024-04-06T09:30:02.386812+0000","last_active":"2024-04-06T10:15:50.877447+0000","last_peered":"2024-04-06T10:15:50.877447+0000","last_clean":"2024-04-06T10:15:50.877447+0000","last_became_active":"2024-04-06T09:30:02.386256+0000","last_became_peered":"2024-04-06T09:30:02.386256+0000","last_unstale":"2024-04-06T10:15:50.877447+0000","last_undegraded":"2024-04-06T10:15:50.877447+0000","last_fullsized":"2024-04-06T10:15:50.877447+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T09:30:01.359377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T09:30:01.359377+0000","last_clean_scrub_stamp":"2024-04-06T09:30:01.359377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T21:22:07.757399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":421,"reported_epoch":403,"state":"active+clean","last_fresh":"2024-04-06T10:15:47.030627+0000","last_change":"2024-04-06T09:30:02.386748+0000","last_active":"2024-04-06T10:15:47.030627+0000","last_peered":"2024-04-06T10:15:47.030627+0000","last_clean":"2024-04-06T10:15:47.030627+0000","last_became_active":"2024-04-06T09:30:02.386318+0000","last_became_peered":"2024-04-06T09:30:02.386318+0000","last_unstale":"2024-04-06T10:15:47.030627+0000","last_undegraded":"2024-04-06T10:15:47.030627+0000","last_fullsized":"2024-04-06T10:15:47.030627+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T09:30:01.359377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T09:30:01.359377+0000","last_clean_scrub_stamp":"2024-04-06T09:30:01.359377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T10:50:42.918574+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":423,"reported_epoch":405,"state":"active+clean","last_fresh":"2024-04-06T10:15:51.206729+0000","last_change":"2024-04-06T09:30:02.382580+0000","last_active":"2024-04-06T10:15:51.206729+0000","last_peered":"2024-04-06T10:15:51.206729+0000","last_clean":"2024-04-06T10:15:51.206729+0000","last_became_active":"2024-04-06T09:30:02.382103+0000","last_became_peered":"2024-04-06T09:30:02.382103+0000","last_unstale":"2024-04-06T10:15:51.206729+0000","last_undegraded":"2024-04-06T10:15:51.206729+0000","last_fullsized":"2024-04-06T10:15:51.206729+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T09:30:01.359377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T09:30:01.359377+0000","last_clean_scrub_stamp":"2024-04-06T09:30:01.359377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T18:50:24.144774+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"10'97","reported_seq":553,"reported_epoch":404,"state":"active+clean","last_fresh":"2024-04-06T10:15:49.853583+0000","last_change":"2024-04-06T09:29:56.421931+0000","last_active":"2024-04-06T10:15:49.853583+0000","last_peered":"2024-04-06T10:15:49.853583+0000","last_clean":"2024-04-06T10:15:49.853583+0000","last_became_active":"2024-04-06T09:29:56.419754+0000","last_became_peered":"2024-04-06T09:29:56.419754+0000","last_unstale":"2024-04-06T10:15:49.853583+0000","last_undegraded":"2024-04-06T10:15:49.853583+0000","last_fullsized":"2024-04-06T10:15:49.853583+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T09:29:55.213061+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T09:29:55.213061+0000","last_clean_scrub_stamp":"2024-04-06T09:29:55.213061+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T19:28:59.898297+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":122880,"data_stored":1180736,"data_compressed":60694,"data_compressed_allocated":81920,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738926,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1740896,"kb_used_data":4692,"kb_used_omap":113,"kb_used_meta":1736078,"kb_avail":92630944,"statfs":{"total":96636764160,"available":94854086656,"internally_reserved":0,"allocated":4804608,"data_stored":2283385,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":116632,"internal_metadata":1777743976},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 6 10:15:16 2024","interfaces":[{"interface":"back","average":{"1min":1.048,"5min":0.60499999999999998,"15min":0.50600000000000001},"min":{"1min":0.32900000000000001,"5min":0.156,"15min":0.156},"max":{"1min":9.9320000000000004,"5min":9.9320000000000004,"15min":9.9320000000000004},"last":0.38600000000000001},{"interface":"front","average":{"1min":0.52700000000000002,"5min":0.55100000000000005,"15min":0.53800000000000003},"min":{"1min":0.22600000000000001,"5min":0.18099999999999999,"15min":0.18099999999999999},"max":{"1min":0.80400000000000005,"5min":3.484,"15min":3.484},"last":0.46800000000000003}]},{"osd":1,"last update":"Sat Apr 6 10:15:16 2024","interfaces":[{"interface":"back","average":{"1min":1.079,"5min":0.70499999999999996,"15min":0.60199999999999998},"min":{"1min":0.30599999999999999,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":9.8179999999999996,"5min":9.8179999999999996,"15min":9.8179999999999996},"last":0.52200000000000002},{"interface":"front","average":{"1min":1.169,"5min":0.79000000000000004,"15min":0.68500000000000005},"min":{"1min":0.35799999999999998,"5min":0.23999999999999999,"15min":0.23999999999999999},"max":{"1min":9.8469999999999995,"5min":9.8469999999999995,"15min":9.8469999999999995},"last":0.56299999999999994}]}]},{"osd":1,"up_from":8,"seq":34359738923,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1296216,"kb_used_data":4752,"kb_used_omap":162,"kb_used_meta":1291293,"kb_avail":93075624,"statfs":{"total":96636764160,"available":95309438976,"internally_reserved":0,"allocated":4866048,"data_stored":2873753,"data_compressed":30347,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":166081,"internal_metadata":1322284863},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 6 10:15:08 2024","interfaces":[{"interface":"back","average":{"1min":0.41999999999999998,"5min":0.496,"15min":0.50700000000000001},"min":{"1min":0.19800000000000001,"5min":0.17999999999999999,"15min":0.16},"max":{"1min":0.60699999999999998,"5min":3.976,"15min":9.5169999999999995},"last":0.50700000000000001},{"interface":"front","average":{"1min":0.43099999999999999,"5min":0.47599999999999998,"15min":0.50900000000000001},"min":{"1min":0.193,"5min":0.193,"15min":0.19},"max":{"1min":0.63700000000000001,"5min":1.0349999999999999,"15min":3.7709999999999999},"last":0.57999999999999996}]},{"osd":2,"last update":"Sat Apr 6 10:15:08 2024","interfaces":[{"interface":"back","average":{"1min":0.42799999999999999,"5min":0.53100000000000003,"15min":0.52100000000000002},"min":{"1min":0.20000000000000001,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":0.61499999999999999,"5min":3.6360000000000001,"15min":3.6360000000000001},"last":0.54200000000000004},{"interface":"front","average":{"1min":0.503,"5min":0.56200000000000006,"15min":0.59099999999999997},"min":{"1min":0.23200000000000001,"5min":0.23200000000000001,"15min":0.23200000000000001},"max":{"1min":0.67700000000000005,"5min":1.452,"15min":2.5099999999999998},"last":0.627}]}]},{"osd":0,"up_from":8,"seq":34359738925,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1204764,"kb_used_data":4752,"kb_used_omap":179,"kb_used_meta":1199820,"kb_avail":93167076,"statfs":{"total":96636764160,"available":95403085824,"internally_reserved":0,"allocated":4866048,"data_stored":2873753,"data_compressed":30347,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":183556,"internal_metadata":1228616444},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":41,"apply_latency_ms":41,"commit_latency_ns":41000000,"apply_latency_ns":41000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Apr 6 10:15:17 2024","interfaces":[{"interface":"back","average":{"1min":0.42899999999999999,"5min":0.45000000000000001,"15min":0.45900000000000002},"min":{"1min":0.189,"5min":0.159,"15min":0.152},"max":{"1min":0.72499999999999998,"5min":0.73099999999999998,"15min":6.5620000000000003},"last":0.65800000000000003},{"interface":"front","average":{"1min":0.49299999999999999,"5min":0.53600000000000003,"15min":0.56699999999999995},"min":{"1min":0.16300000000000001,"5min":0.16300000000000001,"15min":0.16300000000000001},"max":{"1min":0.755,"5min":0.79400000000000004,"15min":3.4820000000000002},"last":0.68999999999999995}]},{"osd":2,"last update":"Sat Apr 6 10:15:23 2024","interfaces":[{"interface":"back","average":{"1min":0.432,"5min":0.47499999999999998,"15min":0.54200000000000004},"min":{"1min":0.189,"5min":0.17199999999999999,"15min":0.17199999999999999},"max":{"1min":0.66300000000000003,"5min":0.70399999999999996,"15min":6.5640000000000001},"last":0.51800000000000002},{"interface":"front","average":{"1min":0.55200000000000005,"5min":0.57399999999999995,"15min":0.56200000000000006},"min":{"1min":0.252,"5min":0.191,"15min":0.184},"max":{"1min":0.79000000000000004,"5min":0.89600000000000002,"15min":2.431},"last":0.61599999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":590368,"data_compressed":30347,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":590368,"data_compressed":30347,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-06T10:15:59.245 DEBUG:teuthology.orchestra.run.smithi192:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 osd dump --format=json 2024-04-06T10:15:59.535 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-04-06T10:15:59.535 INFO:teuthology.orchestra.run.smithi192.stdout:{"epoch":405,"fsid":"cb2d68e3-446e-4a71-a214-bfd667a21bca","created":"2024-04-06T09:29:45.413204+0000","modified":"2024-04-06T10:15:50.849308+0000","last_up_change":"2024-04-06T09:29:54.209374+0000","last_in_change":"2024-04-06T09:29:46.341898+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-06T09:29:55.025846+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"11","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":2.9900000095367432,"score_stable":2.9900000095367432,"optimal_score":0.67000001668930054,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-04-06T09:30:00.619189+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"14","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"92ff8878-e8ea-4f7a-80e8-a98fd9a387c5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":29,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6816","nonce":23962},{"type":"v1","addr":"172.21.15.192:6817","nonce":23962}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6818","nonce":23962},{"type":"v1","addr":"172.21.15.192:6819","nonce":23962}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6822","nonce":23962},{"type":"v1","addr":"172.21.15.192:6823","nonce":23962}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6820","nonce":23962},{"type":"v1","addr":"172.21.15.192:6821","nonce":23962}]},"public_addr":"172.21.15.192:6817/23962","cluster_addr":"172.21.15.192:6819/23962","heartbeat_back_addr":"172.21.15.192:6823/23962","heartbeat_front_addr":"172.21.15.192:6821/23962","state":["exists","up"]},{"osd":1,"uuid":"7beb4d05-1fb2-4116-b33e-559d2734ebe7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6802","nonce":23961},{"type":"v1","addr":"172.21.15.192:6804","nonce":23961}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6806","nonce":23961},{"type":"v1","addr":"172.21.15.192:6808","nonce":23961}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6814","nonce":23961},{"type":"v1","addr":"172.21.15.192:6815","nonce":23961}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6810","nonce":23961},{"type":"v1","addr":"172.21.15.192:6812","nonce":23961}]},"public_addr":"172.21.15.192:6804/23961","cluster_addr":"172.21.15.192:6808/23961","heartbeat_back_addr":"172.21.15.192:6815/23961","heartbeat_front_addr":"172.21.15.192:6812/23961","state":["exists","up"]},{"osd":2,"uuid":"c99a1679-ff03-419c-a26b-797120489d61","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":29,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6800","nonce":23963},{"type":"v1","addr":"172.21.15.192:6801","nonce":23963}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6803","nonce":23963},{"type":"v1","addr":"172.21.15.192:6805","nonce":23963}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6811","nonce":23963},{"type":"v1","addr":"172.21.15.192:6813","nonce":23963}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6807","nonce":23963},{"type":"v1","addr":"172.21.15.192:6809","nonce":23963}]},"public_addr":"172.21.15.192:6801/23963","cluster_addr":"172.21.15.192:6805/23963","heartbeat_back_addr":"172.21.15.192:6813/23963","heartbeat_front_addr":"172.21.15.192:6809/23963","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T09:29:51.734405+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T09:29:51.033517+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T09:29:51.288986+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.192:0/3652264520":"2024-04-06T11:15:46.216521+0000","172.21.15.192:0/3040078596":"2024-04-06T11:15:04.785566+0000","172.21.15.192:0/1692416518":"2024-04-06T11:13:41.776752+0000","172.21.15.192:0/2518790572":"2024-04-06T11:13:02.450462+0000","172.21.15.192:0/1067669479":"2024-04-06T11:12:21.035985+0000","172.21.15.192:0/868813305":"2024-04-06T11:10:54.863758+0000","172.21.15.192:0/3828341330":"2024-04-06T11:10:16.026144+0000","172.21.15.192:0/452492275":"2024-04-06T11:07:13.785108+0000","172.21.15.192:0/2545583640":"2024-04-06T10:58:36.066198+0000","172.21.15.192:0/642779399":"2024-04-06T10:57:54.927164+0000","172.21.15.192:0/1526839294":"2024-04-06T10:40:09.167972+0000","172.21.15.192:0/3094062405":"2024-04-06T10:56:32.697841+0000","172.21.15.192:0/1354876251":"2024-04-06T10:47:31.780778+0000","172.21.15.192:0/3404237165":"2024-04-06T11:01:15.813392+0000","172.21.15.192:0/1773042783":"2024-04-06T10:56:07.846992+0000","172.21.15.192:0/2379383444":"2024-04-06T10:59:14.800802+0000","172.21.15.192:0/750604504":"2024-04-06T10:59:32.250497+0000","172.21.15.192:0/3356453430":"2024-04-06T10:42:12.436920+0000","172.21.15.192:0/3052314336":"2024-04-06T10:52:39.600222+0000","172.21.15.192:0/2415241439":"2024-04-06T11:02:33.475531+0000","172.21.15.192:0/1541366141":"2024-04-06T10:55:20.664903+0000","172.21.15.192:0/688051477":"2024-04-06T10:58:43.295330+0000","172.21.15.192:0/1602527816":"2024-04-06T10:59:23.336652+0000","172.21.15.192:0/3746614709":"2024-04-06T11:02:22.813998+0000","172.21.15.192:0/2455922384":"2024-04-06T10:55:06.192490+0000","172.21.15.192:0/320699586":"2024-04-06T10:58:24.568492+0000","172.21.15.192:0/2171307582":"2024-04-06T10:53:44.999976+0000","172.21.15.192:0/4282894604":"2024-04-06T11:05:01.211735+0000","172.21.15.192:0/3669487567":"2024-04-06T11:11:44.014557+0000","172.21.15.192:0/706228779":"2024-04-06T10:36:59.545284+0000","172.21.15.192:0/2838507813":"2024-04-06T10:38:20.588324+0000","172.21.15.192:0/16910622":"2024-04-06T10:44:28.771172+0000","172.21.15.192:0/87252160":"2024-04-06T11:00:33.616829+0000","172.21.15.192:0/2958832804":"2024-04-06T11:01:29.693128+0000","172.21.15.192:0/2033560996":"2024-04-06T10:34:56.576611+0000","172.21.15.192:0/4125972497":"2024-04-06T11:04:45.809085+0000","172.21.15.192:0/3147888916":"2024-04-06T10:46:25.769458+0000","172.21.15.192:0/910710289":"2024-04-06T10:58:07.011011+0000","172.21.15.192:0/327352287":"2024-04-06T11:02:48.129149+0000","172.21.15.192:0/3422849476":"2024-04-06T11:00:49.314209+0000","172.21.15.192:0/3440101357":"2024-04-06T10:56:46.209442+0000","172.21.15.192:0/3466184018":"2024-04-06T10:32:43.939362+0000","172.21.15.192:0/3261365263":"2024-04-06T11:05:26.256304+0000","172.21.15.192:0/3919643389":"2024-04-06T10:55:14.732442+0000","172.21.15.192:0/3433985788":"2024-04-06T11:03:50.189412+0000","172.21.15.192:0/2192230004":"2024-04-06T10:57:16.825313+0000","172.21.15.192:0/1230946922":"2024-04-06T10:59:43.362811+0000","172.21.15.192:0/490749782":"2024-04-06T10:59:53.347868+0000","172.21.15.192:0/3287549810":"2024-04-06T11:00:04.349289+0000","172.21.15.192:0/3425728423":"2024-04-06T10:52:33.030365+0000","172.21.15.192:0/681946882":"2024-04-06T11:00:15.825546+0000","172.21.15.192:0/3456222804":"2024-04-06T11:14:24.522364+0000","172.21.15.192:0/583183931":"2024-04-06T11:00:54.582399+0000","172.21.15.192:0/1783822727":"2024-04-06T11:02:02.306247+0000","172.21.15.192:0/598640706":"2024-04-06T10:58:00.768850+0000","172.21.15.192:0/3502578500":"2024-04-06T11:04:03.203229+0000","172.21.15.192:0/4167774556":"2024-04-06T11:05:59.472695+0000","172.21.15.192:0/653749661":"2024-04-06T10:56:00.256165+0000","172.21.15.192:0/3654746646":"2024-04-06T11:06:20.933526+0000","172.21.15.192:0/670409394":"2024-04-06T11:01:44.122230+0000","172.21.15.192:0/1786340182":"2024-04-06T11:02:13.601698+0000","172.21.15.192:0/2110460031":"2024-04-06T11:02:58.607165+0000","172.21.15.192:0/1078057278":"2024-04-06T11:09:33.996623+0000","172.21.15.192:0/4125849053":"2024-04-06T11:03:03.760795+0000","172.21.15.192:0/2936214683":"2024-04-06T11:05:48.730372+0000","172.21.15.192:0/1059968663":"2024-04-06T11:06:26.804556+0000","172.21.15.192:0/1983306768":"2024-04-06T10:48:41.728704+0000","172.21.15.192:0/1833862412":"2024-04-06T10:53:21.215426+0000","172.21.15.192:0/3272682859":"2024-04-06T11:04:15.486265+0000","172.21.15.192:0/66504004":"2024-04-06T11:06:39.277033+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-06T10:16:00.554 INFO:tasks.ceph:Scrubbing cluster2.osd.0 2024-04-06T10:16:00.555 DEBUG:teuthology.orchestra.run.smithi192:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 tell osd.0 config set osd_debug_deep_scrub_sleep 0 2024-04-06T10:16:00.712 INFO:teuthology.orchestra.run.smithi192.stdout:{ 2024-04-06T10:16:00.712 INFO:teuthology.orchestra.run.smithi192.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-04-06T10:16:00.712 INFO:teuthology.orchestra.run.smithi192.stdout:} 2024-04-06T10:16:00.728 DEBUG:teuthology.orchestra.run.smithi192:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 osd deep-scrub 0 2024-04-06T10:16:01.000 INFO:teuthology.orchestra.run.smithi192.stderr:instructed osd(s) 0 to deep-scrub 2024-04-06T10:16:01.017 INFO:tasks.ceph:Scrubbing cluster2.osd.1 2024-04-06T10:16:01.017 DEBUG:teuthology.orchestra.run.smithi192:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 tell osd.1 config set osd_debug_deep_scrub_sleep 0 2024-04-06T10:16:01.163 INFO:teuthology.orchestra.run.smithi192.stdout:{ 2024-04-06T10:16:01.163 INFO:teuthology.orchestra.run.smithi192.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-04-06T10:16:01.163 INFO:teuthology.orchestra.run.smithi192.stdout:} 2024-04-06T10:16:01.178 DEBUG:teuthology.orchestra.run.smithi192:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 osd deep-scrub 1 2024-04-06T10:16:01.448 INFO:teuthology.orchestra.run.smithi192.stderr:instructed osd(s) 1 to deep-scrub 2024-04-06T10:16:01.466 INFO:tasks.ceph:Scrubbing cluster2.osd.2 2024-04-06T10:16:01.466 DEBUG:teuthology.orchestra.run.smithi192:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 tell osd.2 config set osd_debug_deep_scrub_sleep 0 2024-04-06T10:16:01.604 INFO:teuthology.orchestra.run.smithi192.stdout:{ 2024-04-06T10:16:01.604 INFO:teuthology.orchestra.run.smithi192.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-04-06T10:16:01.604 INFO:teuthology.orchestra.run.smithi192.stdout:} 2024-04-06T10:16:01.619 DEBUG:teuthology.orchestra.run.smithi192:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 osd deep-scrub 2 2024-04-06T10:16:01.884 INFO:teuthology.orchestra.run.smithi192.stderr:instructed osd(s) 2 to deep-scrub 2024-04-06T10:16:01.902 DEBUG:teuthology.orchestra.run.smithi192:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2024-04-06T10:16:02.172 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-04-06T10:16:02.172 INFO:teuthology.orchestra.run.smithi192.stderr:dumped all 2024-04-06T10:16:02.192 INFO:teuthology.orchestra.run.smithi192.stdout:{"pg_ready":true,"pg_map":{"version":1786,"stamp":"2024-04-06T10:16:01.976770+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":4241876,"kb_used_data":14196,"kb_used_omap":455,"kb_used_meta":4227192,"kb_avail":278873644,"statfs":{"total":289910292480,"available":285566611456,"internally_reserved":0,"allocated":14536704,"data_stored":8030891,"data_compressed":60694,"data_compressed_allocated":81920,"data_compressed_original":1146880,"omap_allocated":466269,"internal_metadata":4328645283},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"11.125310"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":422,"reported_epoch":404,"state":"active+clean","last_fresh":"2024-04-06T10:15:49.859638+0000","last_change":"2024-04-06T09:30:02.384212+0000","last_active":"2024-04-06T10:15:49.859638+0000","last_peered":"2024-04-06T10:15:49.859638+0000","last_clean":"2024-04-06T10:15:49.859638+0000","last_became_active":"2024-04-06T09:30:02.383710+0000","last_became_peered":"2024-04-06T09:30:02.383710+0000","last_unstale":"2024-04-06T10:15:49.859638+0000","last_undegraded":"2024-04-06T10:15:49.859638+0000","last_fullsized":"2024-04-06T10:15:49.859638+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T09:30:01.359377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T09:30:01.359377+0000","last_clean_scrub_stamp":"2024-04-06T09:30:01.359377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T18:51:44.859927+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":422,"reported_epoch":404,"state":"active+clean","last_fresh":"2024-04-06T10:15:49.854844+0000","last_change":"2024-04-06T09:30:02.385797+0000","last_active":"2024-04-06T10:15:49.854844+0000","last_peered":"2024-04-06T10:15:49.854844+0000","last_clean":"2024-04-06T10:15:49.854844+0000","last_became_active":"2024-04-06T09:30:02.385501+0000","last_became_peered":"2024-04-06T09:30:02.385501+0000","last_unstale":"2024-04-06T10:15:49.854844+0000","last_undegraded":"2024-04-06T10:15:49.854844+0000","last_fullsized":"2024-04-06T10:15:49.854844+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T09:30:01.359377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T09:30:01.359377+0000","last_clean_scrub_stamp":"2024-04-06T09:30:01.359377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T15:08:40.865789+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":422,"reported_epoch":404,"state":"active+clean","last_fresh":"2024-04-06T10:15:49.859497+0000","last_change":"2024-04-06T09:30:02.383199+0000","last_active":"2024-04-06T10:15:49.859497+0000","last_peered":"2024-04-06T10:15:49.859497+0000","last_clean":"2024-04-06T10:15:49.859497+0000","last_became_active":"2024-04-06T09:30:02.382798+0000","last_became_peered":"2024-04-06T09:30:02.382798+0000","last_unstale":"2024-04-06T10:15:49.859497+0000","last_undegraded":"2024-04-06T10:15:49.859497+0000","last_fullsized":"2024-04-06T10:15:49.859497+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T09:30:01.359377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T09:30:01.359377+0000","last_clean_scrub_stamp":"2024-04-06T09:30:01.359377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T18:11:34.478290+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":423,"reported_epoch":405,"state":"active+clean","last_fresh":"2024-04-06T10:15:51.159679+0000","last_change":"2024-04-06T09:30:02.383483+0000","last_active":"2024-04-06T10:15:51.159679+0000","last_peered":"2024-04-06T10:15:51.159679+0000","last_clean":"2024-04-06T10:15:51.159679+0000","last_became_active":"2024-04-06T09:30:02.382963+0000","last_became_peered":"2024-04-06T09:30:02.382963+0000","last_unstale":"2024-04-06T10:15:51.159679+0000","last_undegraded":"2024-04-06T10:15:51.159679+0000","last_fullsized":"2024-04-06T10:15:51.159679+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T09:30:01.359377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T09:30:01.359377+0000","last_clean_scrub_stamp":"2024-04-06T09:30:01.359377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T10:19:04.998532+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":422,"reported_epoch":405,"state":"active+clean","last_fresh":"2024-04-06T10:15:52.035138+0000","last_change":"2024-04-06T09:30:02.383104+0000","last_active":"2024-04-06T10:15:52.035138+0000","last_peered":"2024-04-06T10:15:52.035138+0000","last_clean":"2024-04-06T10:15:52.035138+0000","last_became_active":"2024-04-06T09:30:02.382554+0000","last_became_peered":"2024-04-06T09:30:02.382554+0000","last_unstale":"2024-04-06T10:15:52.035138+0000","last_undegraded":"2024-04-06T10:15:52.035138+0000","last_fullsized":"2024-04-06T10:15:52.035138+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T09:30:01.359377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T09:30:01.359377+0000","last_clean_scrub_stamp":"2024-04-06T09:30:01.359377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T18:47:16.568825+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":421,"reported_epoch":405,"state":"active+clean","last_fresh":"2024-04-06T10:15:50.877447+0000","last_change":"2024-04-06T09:30:02.386812+0000","last_active":"2024-04-06T10:15:50.877447+0000","last_peered":"2024-04-06T10:15:50.877447+0000","last_clean":"2024-04-06T10:15:50.877447+0000","last_became_active":"2024-04-06T09:30:02.386256+0000","last_became_peered":"2024-04-06T09:30:02.386256+0000","last_unstale":"2024-04-06T10:15:50.877447+0000","last_undegraded":"2024-04-06T10:15:50.877447+0000","last_fullsized":"2024-04-06T10:15:50.877447+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T09:30:01.359377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T09:30:01.359377+0000","last_clean_scrub_stamp":"2024-04-06T09:30:01.359377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T21:22:07.757399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":421,"reported_epoch":403,"state":"active+clean","last_fresh":"2024-04-06T10:15:47.030627+0000","last_change":"2024-04-06T09:30:02.386748+0000","last_active":"2024-04-06T10:15:47.030627+0000","last_peered":"2024-04-06T10:15:47.030627+0000","last_clean":"2024-04-06T10:15:47.030627+0000","last_became_active":"2024-04-06T09:30:02.386318+0000","last_became_peered":"2024-04-06T09:30:02.386318+0000","last_unstale":"2024-04-06T10:15:47.030627+0000","last_undegraded":"2024-04-06T10:15:47.030627+0000","last_fullsized":"2024-04-06T10:15:47.030627+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T09:30:01.359377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T09:30:01.359377+0000","last_clean_scrub_stamp":"2024-04-06T09:30:01.359377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T10:50:42.918574+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":423,"reported_epoch":405,"state":"active+clean","last_fresh":"2024-04-06T10:15:51.206729+0000","last_change":"2024-04-06T09:30:02.382580+0000","last_active":"2024-04-06T10:15:51.206729+0000","last_peered":"2024-04-06T10:15:51.206729+0000","last_clean":"2024-04-06T10:15:51.206729+0000","last_became_active":"2024-04-06T09:30:02.382103+0000","last_became_peered":"2024-04-06T09:30:02.382103+0000","last_unstale":"2024-04-06T10:15:51.206729+0000","last_undegraded":"2024-04-06T10:15:51.206729+0000","last_fullsized":"2024-04-06T10:15:51.206729+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T09:30:01.359377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T09:30:01.359377+0000","last_clean_scrub_stamp":"2024-04-06T09:30:01.359377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T18:50:24.144774+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"10'97","reported_seq":553,"reported_epoch":404,"state":"active+clean","last_fresh":"2024-04-06T10:15:49.853583+0000","last_change":"2024-04-06T09:29:56.421931+0000","last_active":"2024-04-06T10:15:49.853583+0000","last_peered":"2024-04-06T10:15:49.853583+0000","last_clean":"2024-04-06T10:15:49.853583+0000","last_became_active":"2024-04-06T09:29:56.419754+0000","last_became_peered":"2024-04-06T09:29:56.419754+0000","last_unstale":"2024-04-06T10:15:49.853583+0000","last_undegraded":"2024-04-06T10:15:49.853583+0000","last_fullsized":"2024-04-06T10:15:49.853583+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T09:29:55.213061+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T09:29:55.213061+0000","last_clean_scrub_stamp":"2024-04-06T09:29:55.213061+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T19:28:59.898297+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":122880,"data_stored":1180736,"data_compressed":60694,"data_compressed_allocated":81920,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738927,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1740896,"kb_used_data":4692,"kb_used_omap":113,"kb_used_meta":1736078,"kb_avail":92630944,"statfs":{"total":96636764160,"available":94854086656,"internally_reserved":0,"allocated":4804608,"data_stored":2283385,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":116632,"internal_metadata":1777743976},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 6 10:15:16 2024","interfaces":[{"interface":"back","average":{"1min":1.048,"5min":0.60499999999999998,"15min":0.50600000000000001},"min":{"1min":0.32900000000000001,"5min":0.156,"15min":0.156},"max":{"1min":9.9320000000000004,"5min":9.9320000000000004,"15min":9.9320000000000004},"last":0.29799999999999999},{"interface":"front","average":{"1min":0.52700000000000002,"5min":0.55100000000000005,"15min":0.53800000000000003},"min":{"1min":0.22600000000000001,"5min":0.18099999999999999,"15min":0.18099999999999999},"max":{"1min":0.80400000000000005,"5min":3.484,"15min":3.484},"last":0.33800000000000002}]},{"osd":1,"last update":"Sat Apr 6 10:15:16 2024","interfaces":[{"interface":"back","average":{"1min":1.079,"5min":0.70499999999999996,"15min":0.60199999999999998},"min":{"1min":0.30599999999999999,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":9.8179999999999996,"5min":9.8179999999999996,"15min":9.8179999999999996},"last":0.36799999999999999},{"interface":"front","average":{"1min":1.169,"5min":0.79000000000000004,"15min":0.68500000000000005},"min":{"1min":0.35799999999999998,"5min":0.23999999999999999,"15min":0.23999999999999999},"max":{"1min":9.8469999999999995,"5min":9.8469999999999995,"15min":9.8469999999999995},"last":0.39200000000000002}]}]},{"osd":1,"up_from":8,"seq":34359738924,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1296216,"kb_used_data":4752,"kb_used_omap":162,"kb_used_meta":1291293,"kb_avail":93075624,"statfs":{"total":96636764160,"available":95309438976,"internally_reserved":0,"allocated":4866048,"data_stored":2873753,"data_compressed":30347,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":166081,"internal_metadata":1322284863},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 6 10:15:08 2024","interfaces":[{"interface":"back","average":{"1min":0.41999999999999998,"5min":0.496,"15min":0.50700000000000001},"min":{"1min":0.19800000000000001,"5min":0.17999999999999999,"15min":0.16},"max":{"1min":0.60699999999999998,"5min":3.976,"15min":9.5169999999999995},"last":0.52100000000000002},{"interface":"front","average":{"1min":0.43099999999999999,"5min":0.47599999999999998,"15min":0.50900000000000001},"min":{"1min":0.193,"5min":0.193,"15min":0.19},"max":{"1min":0.63700000000000001,"5min":1.0349999999999999,"15min":3.7709999999999999},"last":0.56899999999999995}]},{"osd":2,"last update":"Sat Apr 6 10:15:08 2024","interfaces":[{"interface":"back","average":{"1min":0.42799999999999999,"5min":0.53100000000000003,"15min":0.52100000000000002},"min":{"1min":0.20000000000000001,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":0.61499999999999999,"5min":3.6360000000000001,"15min":3.6360000000000001},"last":0.61799999999999999},{"interface":"front","average":{"1min":0.503,"5min":0.56200000000000006,"15min":0.59099999999999997},"min":{"1min":0.23200000000000001,"5min":0.23200000000000001,"15min":0.23200000000000001},"max":{"1min":0.67700000000000005,"5min":1.452,"15min":2.5099999999999998},"last":0.68000000000000005}]}]},{"osd":0,"up_from":8,"seq":34359738926,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1204764,"kb_used_data":4752,"kb_used_omap":179,"kb_used_meta":1199820,"kb_avail":93167076,"statfs":{"total":96636764160,"available":95403085824,"internally_reserved":0,"allocated":4866048,"data_stored":2873753,"data_compressed":30347,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":183556,"internal_metadata":1228616444},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Apr 6 10:15:17 2024","interfaces":[{"interface":"back","average":{"1min":0.42899999999999999,"5min":0.45000000000000001,"15min":0.45900000000000002},"min":{"1min":0.189,"5min":0.159,"15min":0.152},"max":{"1min":0.72499999999999998,"5min":0.73099999999999998,"15min":6.5620000000000003},"last":0.27800000000000002},{"interface":"front","average":{"1min":0.49299999999999999,"5min":0.53600000000000003,"15min":0.56699999999999995},"min":{"1min":0.16300000000000001,"5min":0.16300000000000001,"15min":0.16300000000000001},"max":{"1min":0.755,"5min":0.79400000000000004,"15min":3.4820000000000002},"last":0.28899999999999998}]},{"osd":2,"last update":"Sat Apr 6 10:15:23 2024","interfaces":[{"interface":"back","average":{"1min":0.432,"5min":0.47499999999999998,"15min":0.54200000000000004},"min":{"1min":0.189,"5min":0.17199999999999999,"15min":0.17199999999999999},"max":{"1min":0.66300000000000003,"5min":0.70399999999999996,"15min":6.5640000000000001},"last":0.33000000000000002},{"interface":"front","average":{"1min":0.55200000000000005,"5min":0.57399999999999995,"15min":0.56200000000000006},"min":{"1min":0.252,"5min":0.191,"15min":0.184},"max":{"1min":0.79000000000000004,"5min":0.89600000000000002,"15min":2.431},"last":0.40100000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":590368,"data_compressed":30347,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":590368,"data_compressed":30347,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-06T10:16:02.196 INFO:tasks.ceph:pgid 2.7 last_scrub_stamp 2024-04-06T09:30:01.359377+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=6, tm_hour=9, tm_min=30, tm_sec=1, tm_wday=5, tm_yday=97, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=6, tm_hour=10, tm_min=15, tm_sec=59, tm_wday=5, tm_yday=97, tm_isdst=0) 2024-04-06T10:16:02.196 INFO:tasks.ceph:pgid 2.6 last_scrub_stamp 2024-04-06T09:30:01.359377+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=6, tm_hour=9, tm_min=30, tm_sec=1, tm_wday=5, tm_yday=97, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=6, tm_hour=10, tm_min=15, tm_sec=59, tm_wday=5, tm_yday=97, tm_isdst=0) 2024-04-06T10:16:02.196 INFO:tasks.ceph:pgid 2.5 last_scrub_stamp 2024-04-06T09:30:01.359377+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=6, tm_hour=9, tm_min=30, tm_sec=1, tm_wday=5, tm_yday=97, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=6, tm_hour=10, tm_min=15, tm_sec=59, tm_wday=5, tm_yday=97, tm_isdst=0) 2024-04-06T10:16:02.196 INFO:tasks.ceph:pgid 2.4 last_scrub_stamp 2024-04-06T09:30:01.359377+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=6, tm_hour=9, tm_min=30, tm_sec=1, tm_wday=5, tm_yday=97, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=6, tm_hour=10, tm_min=15, tm_sec=59, tm_wday=5, tm_yday=97, tm_isdst=0) 2024-04-06T10:16:02.196 INFO:tasks.ceph:pgid 2.2 last_scrub_stamp 2024-04-06T09:30:01.359377+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=6, tm_hour=9, tm_min=30, tm_sec=1, tm_wday=5, tm_yday=97, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=6, tm_hour=10, tm_min=15, tm_sec=59, tm_wday=5, tm_yday=97, tm_isdst=0) 2024-04-06T10:16:02.197 INFO:tasks.ceph:pgid 2.1 last_scrub_stamp 2024-04-06T09:30:01.359377+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=6, tm_hour=9, tm_min=30, tm_sec=1, tm_wday=5, tm_yday=97, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=6, tm_hour=10, tm_min=15, tm_sec=59, tm_wday=5, tm_yday=97, tm_isdst=0) 2024-04-06T10:16:02.197 INFO:tasks.ceph:pgid 2.0 last_scrub_stamp 2024-04-06T09:30:01.359377+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=6, tm_hour=9, tm_min=30, tm_sec=1, tm_wday=5, tm_yday=97, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=6, tm_hour=10, tm_min=15, tm_sec=59, tm_wday=5, tm_yday=97, tm_isdst=0) 2024-04-06T10:16:02.197 INFO:tasks.ceph:pgid 2.3 last_scrub_stamp 2024-04-06T09:30:01.359377+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=6, tm_hour=9, tm_min=30, tm_sec=1, tm_wday=5, tm_yday=97, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=6, tm_hour=10, tm_min=15, tm_sec=59, tm_wday=5, tm_yday=97, tm_isdst=0) 2024-04-06T10:16:02.197 INFO:tasks.ceph:pgid 1.0 last_scrub_stamp 2024-04-06T09:29:55.213061+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=6, tm_hour=9, tm_min=29, tm_sec=55, tm_wday=5, tm_yday=97, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=6, tm_hour=10, tm_min=15, tm_sec=59, tm_wday=5, tm_yday=97, tm_isdst=0) 2024-04-06T10:16:02.197 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2024-04-06T10:16:22.198 DEBUG:teuthology.orchestra.run.smithi192:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2024-04-06T10:16:22.504 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-04-06T10:16:22.505 INFO:teuthology.orchestra.run.smithi192.stderr:dumped all 2024-04-06T10:16:22.523 INFO:teuthology.orchestra.run.smithi192.stdout:{"pg_ready":true,"pg_map":{"version":1796,"stamp":"2024-04-06T10:16:21.981756+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":4241868,"kb_used_data":14196,"kb_used_omap":455,"kb_used_meta":4227192,"kb_avail":278873652,"statfs":{"total":289910292480,"available":285566619648,"internally_reserved":0,"allocated":14536704,"data_stored":8030891,"data_compressed":60694,"data_compressed_allocated":81920,"data_compressed_original":1146880,"omap_allocated":466269,"internal_metadata":4328645283},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002987"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":434,"reported_epoch":405,"state":"active+clean","last_fresh":"2024-04-06T10:16:07.944678+0000","last_change":"2024-04-06T10:16:07.944806+0000","last_active":"2024-04-06T10:16:07.944678+0000","last_peered":"2024-04-06T10:16:07.944678+0000","last_clean":"2024-04-06T10:16:07.944678+0000","last_became_active":"2024-04-06T09:30:02.383710+0000","last_became_peered":"2024-04-06T09:30:02.383710+0000","last_unstale":"2024-04-06T10:16:07.944678+0000","last_undegraded":"2024-04-06T10:16:07.944678+0000","last_fullsized":"2024-04-06T10:16:07.944678+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T10:16:07.944659+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T10:16:07.944659+0000","last_clean_scrub_stamp":"2024-04-06T10:16:07.944659+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T16:45:05.784872+0000","scrub_duration":0.31300551199999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":434,"reported_epoch":405,"state":"active+clean","last_fresh":"2024-04-06T10:16:05.720901+0000","last_change":"2024-04-06T10:16:05.721173+0000","last_active":"2024-04-06T10:16:05.720901+0000","last_peered":"2024-04-06T10:16:05.720901+0000","last_clean":"2024-04-06T10:16:05.720901+0000","last_became_active":"2024-04-06T09:30:02.385501+0000","last_became_peered":"2024-04-06T09:30:02.385501+0000","last_unstale":"2024-04-06T10:16:05.720901+0000","last_undegraded":"2024-04-06T10:16:05.720901+0000","last_fullsized":"2024-04-06T10:16:05.720901+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T10:16:05.720853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T10:16:05.720853+0000","last_clean_scrub_stamp":"2024-04-06T10:16:05.720853+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T19:36:11.807909+0000","scrub_duration":0.120018562,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":435,"reported_epoch":405,"state":"active+clean","last_fresh":"2024-04-06T10:16:04.901731+0000","last_change":"2024-04-06T10:16:04.901731+0000","last_active":"2024-04-06T10:16:04.901731+0000","last_peered":"2024-04-06T10:16:04.901731+0000","last_clean":"2024-04-06T10:16:04.901731+0000","last_became_active":"2024-04-06T09:30:02.382798+0000","last_became_peered":"2024-04-06T09:30:02.382798+0000","last_unstale":"2024-04-06T10:16:04.901731+0000","last_undegraded":"2024-04-06T10:16:04.901731+0000","last_fullsized":"2024-04-06T10:16:04.901731+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T10:16:04.901456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T10:16:04.901456+0000","last_clean_scrub_stamp":"2024-04-06T10:16:04.901456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T20:14:24.369154+0000","scrub_duration":0.32018545500000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":435,"reported_epoch":405,"state":"active+clean","last_fresh":"2024-04-06T10:16:03.536982+0000","last_change":"2024-04-06T10:16:03.537199+0000","last_active":"2024-04-06T10:16:03.536982+0000","last_peered":"2024-04-06T10:16:03.536982+0000","last_clean":"2024-04-06T10:16:03.536982+0000","last_became_active":"2024-04-06T09:30:02.382963+0000","last_became_peered":"2024-04-06T09:30:02.382963+0000","last_unstale":"2024-04-06T10:16:03.536982+0000","last_undegraded":"2024-04-06T10:16:03.536982+0000","last_fullsized":"2024-04-06T10:16:03.536982+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T10:16:03.536925+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T10:16:03.536925+0000","last_clean_scrub_stamp":"2024-04-06T10:16:03.536925+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T18:48:22.065645+0000","scrub_duration":0.0033898740000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":434,"reported_epoch":405,"state":"active+clean","last_fresh":"2024-04-06T10:16:05.639157+0000","last_change":"2024-04-06T10:16:05.639309+0000","last_active":"2024-04-06T10:16:05.639157+0000","last_peered":"2024-04-06T10:16:05.639157+0000","last_clean":"2024-04-06T10:16:05.639157+0000","last_became_active":"2024-04-06T09:30:02.382554+0000","last_became_peered":"2024-04-06T09:30:02.382554+0000","last_unstale":"2024-04-06T10:16:05.639157+0000","last_undegraded":"2024-04-06T10:16:05.639157+0000","last_fullsized":"2024-04-06T10:16:05.639157+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T10:16:05.639133+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T10:16:05.639133+0000","last_clean_scrub_stamp":"2024-04-06T10:16:05.639133+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T17:09:39.061646+0000","scrub_duration":0.091216057000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":433,"reported_epoch":405,"state":"active+clean","last_fresh":"2024-04-06T10:16:02.302291+0000","last_change":"2024-04-06T10:16:02.302428+0000","last_active":"2024-04-06T10:16:02.302291+0000","last_peered":"2024-04-06T10:16:02.302291+0000","last_clean":"2024-04-06T10:16:02.302291+0000","last_became_active":"2024-04-06T09:30:02.386256+0000","last_became_peered":"2024-04-06T09:30:02.386256+0000","last_unstale":"2024-04-06T10:16:02.302291+0000","last_undegraded":"2024-04-06T10:16:02.302291+0000","last_fullsized":"2024-04-06T10:16:02.302291+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T10:16:02.302272+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T10:16:02.302272+0000","last_clean_scrub_stamp":"2024-04-06T10:16:02.302272+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T18:02:29.263900+0000","scrub_duration":0.13637849099999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":434,"reported_epoch":405,"state":"active+clean","last_fresh":"2024-04-06T10:16:08.154265+0000","last_change":"2024-04-06T10:16:08.154265+0000","last_active":"2024-04-06T10:16:08.154265+0000","last_peered":"2024-04-06T10:16:08.154265+0000","last_clean":"2024-04-06T10:16:08.154265+0000","last_became_active":"2024-04-06T09:30:02.386318+0000","last_became_peered":"2024-04-06T09:30:02.386318+0000","last_unstale":"2024-04-06T10:16:08.154265+0000","last_undegraded":"2024-04-06T10:16:08.154265+0000","last_fullsized":"2024-04-06T10:16:08.154265+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T10:16:08.154111+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T10:16:08.154111+0000","last_clean_scrub_stamp":"2024-04-06T10:16:08.154111+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T13:00:19.988046+0000","scrub_duration":0.11972606,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":435,"reported_epoch":405,"state":"active+clean","last_fresh":"2024-04-06T10:16:09.860397+0000","last_change":"2024-04-06T10:16:09.860519+0000","last_active":"2024-04-06T10:16:09.860397+0000","last_peered":"2024-04-06T10:16:09.860397+0000","last_clean":"2024-04-06T10:16:09.860397+0000","last_became_active":"2024-04-06T09:30:02.382103+0000","last_became_peered":"2024-04-06T09:30:02.382103+0000","last_unstale":"2024-04-06T10:16:09.860397+0000","last_undegraded":"2024-04-06T10:16:09.860397+0000","last_fullsized":"2024-04-06T10:16:09.860397+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T10:16:09.860376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T10:16:09.860376+0000","last_clean_scrub_stamp":"2024-04-06T10:16:09.860376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T19:19:55.062328+0000","scrub_duration":0.29329823900000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"405'99","reported_seq":568,"reported_epoch":405,"state":"active+clean","last_fresh":"2024-04-06T10:16:02.561999+0000","last_change":"2024-04-06T10:16:02.561999+0000","last_active":"2024-04-06T10:16:02.561999+0000","last_peered":"2024-04-06T10:16:02.561999+0000","last_clean":"2024-04-06T10:16:02.561999+0000","last_became_active":"2024-04-06T09:29:56.419754+0000","last_became_peered":"2024-04-06T09:29:56.419754+0000","last_unstale":"2024-04-06T10:16:02.561999+0000","last_undegraded":"2024-04-06T10:16:02.561999+0000","last_fullsized":"2024-04-06T10:16:02.561999+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"405'99","last_scrub_stamp":"2024-04-06T10:16:02.561728+0000","last_deep_scrub":"405'99","last_deep_scrub_stamp":"2024-04-06T10:16:02.561728+0000","last_clean_scrub_stamp":"2024-04-06T10:16:02.561728+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T13:30:19.690116+0000","scrub_duration":0.0092750699999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":122880,"data_stored":1180736,"data_compressed":60694,"data_compressed_allocated":81920,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738931,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1740892,"kb_used_data":4692,"kb_used_omap":113,"kb_used_meta":1736078,"kb_avail":92630948,"statfs":{"total":96636764160,"available":94854090752,"internally_reserved":0,"allocated":4804608,"data_stored":2283385,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":116632,"internal_metadata":1777743976},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 6 10:16:16 2024","interfaces":[{"interface":"back","average":{"1min":0.496,"5min":0.61399999999999999,"15min":0.50900000000000001},"min":{"1min":0.29799999999999999,"5min":0.156,"15min":0.156},"max":{"1min":0.71399999999999997,"5min":9.9320000000000004,"15min":9.9320000000000004},"last":0.45000000000000001},{"interface":"front","average":{"1min":0.54500000000000004,"5min":0.55500000000000005,"15min":0.53900000000000003},"min":{"1min":0.33800000000000002,"5min":0.18099999999999999,"15min":0.18099999999999999},"max":{"1min":0.68700000000000006,"5min":3.484,"15min":3.484},"last":0.53500000000000003}]},{"osd":1,"last update":"Sat Apr 6 10:16:16 2024","interfaces":[{"interface":"back","average":{"1min":0.59699999999999998,"5min":0.65300000000000002,"15min":0.60499999999999998},"min":{"1min":0.36799999999999999,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":0.96899999999999997,"5min":9.8179999999999996,"15min":9.8179999999999996},"last":0.623},{"interface":"front","average":{"1min":0.68300000000000005,"5min":0.73899999999999999,"15min":0.69499999999999995},"min":{"1min":0.39200000000000002,"5min":0.23999999999999999,"15min":0.23999999999999999},"max":{"1min":1.129,"5min":9.8469999999999995,"15min":9.8469999999999995},"last":0.67400000000000004}]}]},{"osd":1,"up_from":8,"seq":34359738928,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1296216,"kb_used_data":4752,"kb_used_omap":162,"kb_used_meta":1291293,"kb_avail":93075624,"statfs":{"total":96636764160,"available":95309438976,"internally_reserved":0,"allocated":4866048,"data_stored":2873753,"data_compressed":30347,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":166081,"internal_metadata":1322284863},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 6 10:16:11 2024","interfaces":[{"interface":"back","average":{"1min":0.49399999999999999,"5min":0.498,"15min":0.47599999999999998},"min":{"1min":0.192,"5min":0.17999999999999999,"15min":0.16},"max":{"1min":0.64800000000000002,"5min":3.976,"15min":3.976},"last":0.66200000000000003},{"interface":"front","average":{"1min":0.53500000000000003,"5min":0.48199999999999998,"15min":0.503},"min":{"1min":0.20300000000000001,"5min":0.193,"15min":0.19},"max":{"1min":0.748,"5min":1.0349999999999999,"15min":2.895},"last":0.55500000000000005}]},{"osd":2,"last update":"Sat Apr 6 10:16:11 2024","interfaces":[{"interface":"back","average":{"1min":0.54800000000000004,"5min":0.54100000000000004,"15min":0.52400000000000002},"min":{"1min":0.27000000000000002,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":0.70699999999999996,"5min":3.6360000000000001,"15min":3.6360000000000001},"last":0.52100000000000002},{"interface":"front","average":{"1min":0.61799999999999999,"5min":0.57299999999999995,"15min":0.59099999999999997},"min":{"1min":0.23200000000000001,"5min":0.23200000000000001,"15min":0.23200000000000001},"max":{"1min":0.82099999999999995,"5min":1.452,"15min":2.5099999999999998},"last":0.627}]}]},{"osd":0,"up_from":8,"seq":34359738930,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1204760,"kb_used_data":4752,"kb_used_omap":179,"kb_used_meta":1199820,"kb_avail":93167080,"statfs":{"total":96636764160,"available":95403089920,"internally_reserved":0,"allocated":4866048,"data_stored":2873753,"data_compressed":30347,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":183556,"internal_metadata":1228616444},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Apr 6 10:16:19 2024","interfaces":[{"interface":"back","average":{"1min":0.495,"5min":0.46600000000000003,"15min":0.46200000000000002},"min":{"1min":0.27800000000000002,"5min":0.182,"15min":0.152},"max":{"1min":0.80900000000000005,"5min":0.80900000000000005,"15min":6.5620000000000003},"last":0.46300000000000002},{"interface":"front","average":{"1min":0.60699999999999998,"5min":0.55500000000000005,"15min":0.57499999999999996},"min":{"1min":0.28899999999999998,"5min":0.16300000000000001,"15min":0.16300000000000001},"max":{"1min":0.68999999999999995,"5min":0.79400000000000004,"15min":3.4820000000000002},"last":0.63900000000000001}]},{"osd":2,"last update":"Sat Apr 6 10:15:23 2024","interfaces":[{"interface":"back","average":{"1min":0.432,"5min":0.47499999999999998,"15min":0.54200000000000004},"min":{"1min":0.189,"5min":0.17199999999999999,"15min":0.17199999999999999},"max":{"1min":0.66300000000000003,"5min":0.70399999999999996,"15min":6.5640000000000001},"last":0.50600000000000001},{"interface":"front","average":{"1min":0.55200000000000005,"5min":0.57399999999999995,"15min":0.56200000000000006},"min":{"1min":0.252,"5min":0.191,"15min":0.184},"max":{"1min":0.79000000000000004,"5min":0.89600000000000002,"15min":2.431},"last":0.60699999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":590368,"data_compressed":30347,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":590368,"data_compressed":30347,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-06T10:16:22.524 DEBUG:teuthology.orchestra.run.smithi192:> sudo ceph --cluster cluster2 config set global mon_health_to_clog false 2024-04-06T10:16:22.839 INFO:teuthology.misc:Shutting down mds daemons... 2024-04-06T10:16:22.839 INFO:teuthology.misc:Shutting down osd daemons... 2024-04-06T10:16:22.839 DEBUG:tasks.ceph.cluster2.osd.0:waiting for process to exit 2024-04-06T10:16:22.839 INFO:teuthology.orchestra.run:waiting for 300 2024-04-06T10:16:27.977 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~0s 2024-04-06T10:16:28.941 INFO:tasks.ceph.cluster2.osd.0:Stopped 2024-04-06T10:16:28.942 DEBUG:tasks.ceph.cluster2.osd.1:waiting for process to exit 2024-04-06T10:16:28.942 INFO:teuthology.orchestra.run:waiting for 300 2024-04-06T10:16:33.281 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.1 is failed for ~0s 2024-04-06T10:16:33.281 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 has been restored 2024-04-06T10:16:35.044 INFO:tasks.ceph.cluster2.osd.1:Stopped 2024-04-06T10:16:35.044 DEBUG:tasks.ceph.cluster2.osd.2:waiting for process to exit 2024-04-06T10:16:35.045 INFO:teuthology.orchestra.run:waiting for 300 2024-04-06T10:16:35.131 INFO:tasks.ceph.cluster2.osd.2:Stopped 2024-04-06T10:16:35.132 INFO:teuthology.misc:Shutting down mgr daemons... 2024-04-06T10:16:35.132 DEBUG:tasks.ceph.cluster2.mgr.x:waiting for process to exit 2024-04-06T10:16:35.132 INFO:teuthology.orchestra.run:waiting for 300 2024-04-06T10:16:35.180 INFO:tasks.ceph.cluster2.mgr.x:Stopped 2024-04-06T10:16:35.181 INFO:teuthology.misc:Shutting down mon daemons... 2024-04-06T10:16:35.181 DEBUG:tasks.ceph.cluster2.mon.a:waiting for process to exit 2024-04-06T10:16:35.181 INFO:teuthology.orchestra.run:waiting for 300 2024-04-06T10:16:35.200 INFO:tasks.ceph.cluster2.mon.a:Stopped 2024-04-06T10:16:35.200 INFO:tasks.ceph:Checking cluster log for badness... 2024-04-06T10:16:35.200 DEBUG:teuthology.orchestra.run.smithi192:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/cluster2.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(OSD_SLOW_PING_TIME' | head -n 1 2024-04-06T10:16:35.407 INFO:teuthology.orchestra.run.smithi192.stdout:2024-04-06T09:32:51.464342+0000 mon.a (mon.0) 214 : cluster [WRN] Health check failed: Reduced data availability: 1 pg inactive, 1 pg peering (PG_AVAILABILITY) 2024-04-06T10:16:35.407 WARNING:tasks.ceph:Found errors (ERR|WRN|SEC) in cluster log 2024-04-06T10:16:35.407 DEBUG:teuthology.orchestra.run.smithi192:> sudo egrep '\[SEC\]' /var/log/ceph/cluster2.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(OSD_SLOW_PING_TIME' | head -n 1 2024-04-06T10:16:35.422 DEBUG:teuthology.orchestra.run.smithi192:> sudo egrep '\[ERR\]' /var/log/ceph/cluster2.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(OSD_SLOW_PING_TIME' | head -n 1 2024-04-06T10:16:35.476 DEBUG:teuthology.orchestra.run.smithi192:> sudo egrep '\[WRN\]' /var/log/ceph/cluster2.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(OSD_SLOW_PING_TIME' | head -n 1 2024-04-06T10:16:35.532 INFO:teuthology.orchestra.run.smithi192.stdout:2024-04-06T09:32:51.464342+0000 mon.a (mon.0) 214 : cluster [WRN] Health check failed: Reduced data availability: 1 pg inactive, 1 pg peering (PG_AVAILABILITY) 2024-04-06T10:16:35.533 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/cluster2-0 on ubuntu@smithi192.front.sepia.ceph.com 2024-04-06T10:16:35.533 DEBUG:teuthology.orchestra.run.smithi192:> sync && sudo umount -f /var/lib/ceph/osd/cluster2-0 2024-04-06T10:16:36.042 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/cluster2-1 on ubuntu@smithi192.front.sepia.ceph.com 2024-04-06T10:16:36.042 DEBUG:teuthology.orchestra.run.smithi192:> sync && sudo umount -f /var/lib/ceph/osd/cluster2-1 2024-04-06T10:16:36.406 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/cluster2-2 on ubuntu@smithi192.front.sepia.ceph.com 2024-04-06T10:16:36.406 DEBUG:teuthology.orchestra.run.smithi192:> sync && sudo umount -f /var/lib/ceph/osd/cluster2-2 2024-04-06T10:16:36.826 INFO:tasks.ceph:Archiving mon data... 2024-04-06T10:16:36.827 DEBUG:teuthology.misc:Transferring archived files from smithi192:/var/lib/ceph/mon/cluster2-a to /home/teuthworker/archive/teuthology-2024-04-02_22:16:03-rbd-reef-distro-default-smithi/7637502/data/cluster2.mon.a.tgz 2024-04-06T10:16:36.828 DEBUG:teuthology.orchestra.run.smithi192:> mktemp 2024-04-06T10:16:36.833 INFO:teuthology.orchestra.run.smithi192.stdout:/tmp/tmp.6zEVueBrcA 2024-04-06T10:16:36.833 DEBUG:teuthology.orchestra.run.smithi192:> sudo tar cz -f - -C /var/lib/ceph/mon/cluster2-a -- . > /tmp/tmp.6zEVueBrcA 2024-04-06T10:16:37.003 DEBUG:teuthology.orchestra.run.smithi192:> sudo chmod 0666 /tmp/tmp.6zEVueBrcA 2024-04-06T10:16:37.088 DEBUG:teuthology.orchestra.remote:smithi192:/tmp/tmp.6zEVueBrcA is 424KB 2024-04-06T10:16:37.149 DEBUG:teuthology.orchestra.run.smithi192:> rm -fr /tmp/tmp.6zEVueBrcA 2024-04-06T10:16:37.155 INFO:tasks.ceph:Cleaning ceph cluster... 2024-04-06T10:16:37.155 DEBUG:teuthology.orchestra.run.smithi006:> sudo rm -rf -- /etc/ceph/cluster2.conf /etc/ceph/cluster2.keyring /home/ubuntu/cephtest/cluster2.data /home/ubuntu/cephtest/cluster2.monmap /home/ubuntu/cephtest/../*.pid 2024-04-06T10:16:37.158 DEBUG:teuthology.orchestra.run.smithi192:> sudo rm -rf -- /etc/ceph/cluster2.conf /etc/ceph/cluster2.keyring /home/ubuntu/cephtest/cluster2.data /home/ubuntu/cephtest/cluster2.monmap /home/ubuntu/cephtest/../*.pid 2024-04-06T10:16:37.361 DEBUG:teuthology.run_tasks:Unwinding manager ceph 2024-04-06T10:16:37.373 INFO:tasks.ceph.ceph_manager.cluster1:waiting for clean 2024-04-06T10:16:37.373 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 pg dump --format=json 2024-04-06T10:16:37.921 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-04-06T10:16:37.922 INFO:teuthology.orchestra.run.smithi006.stderr:dumped all 2024-04-06T10:16:37.959 INFO:teuthology.orchestra.run.smithi006.stdout:{"pg_ready":true,"pg_map":{"version":1761,"stamp":"2024-04-06T10:16:36.324307+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":4862736,"kb_used_data":8424,"kb_used_omap":638,"kb_used_meta":4853633,"kb_avail":278252784,"statfs":{"total":289910292480,"available":284930850816,"internally_reserved":0,"allocated":8626176,"data_stored":5667623,"data_compressed":60676,"data_compressed_allocated":81920,"data_compressed_original":1146880,"omap_allocated":653651,"internal_metadata":4970120877},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003158"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":360,"reported_epoch":343,"state":"active+clean","last_fresh":"2024-04-06T10:15:49.390117+0000","last_change":"2024-04-06T09:29:25.738609+0000","last_active":"2024-04-06T10:15:49.390117+0000","last_peered":"2024-04-06T10:15:49.390117+0000","last_clean":"2024-04-06T10:15:49.390117+0000","last_became_active":"2024-04-06T09:29:25.737557+0000","last_became_peered":"2024-04-06T09:29:25.737557+0000","last_unstale":"2024-04-06T10:15:49.390117+0000","last_undegraded":"2024-04-06T10:15:49.390117+0000","last_fullsized":"2024-04-06T10:15:49.390117+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T09:29:24.717518+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T09:29:24.717518+0000","last_clean_scrub_stamp":"2024-04-06T09:29:24.717518+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T15:44:35.416009+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":360,"reported_epoch":343,"state":"active+clean","last_fresh":"2024-04-06T10:15:49.403104+0000","last_change":"2024-04-06T09:29:25.738328+0000","last_active":"2024-04-06T10:15:49.403104+0000","last_peered":"2024-04-06T10:15:49.403104+0000","last_clean":"2024-04-06T10:15:49.403104+0000","last_became_active":"2024-04-06T09:29:25.737348+0000","last_became_peered":"2024-04-06T09:29:25.737348+0000","last_unstale":"2024-04-06T10:15:49.403104+0000","last_undegraded":"2024-04-06T10:15:49.403104+0000","last_fullsized":"2024-04-06T10:15:49.403104+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T09:29:24.717518+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T09:29:24.717518+0000","last_clean_scrub_stamp":"2024-04-06T09:29:24.717518+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T10:00:30.760505+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":361,"reported_epoch":343,"state":"active+clean","last_fresh":"2024-04-06T10:15:49.395195+0000","last_change":"2024-04-06T09:29:25.738960+0000","last_active":"2024-04-06T10:15:49.395195+0000","last_peered":"2024-04-06T10:15:49.395195+0000","last_clean":"2024-04-06T10:15:49.395195+0000","last_became_active":"2024-04-06T09:29:25.738591+0000","last_became_peered":"2024-04-06T09:29:25.738591+0000","last_unstale":"2024-04-06T10:15:49.395195+0000","last_undegraded":"2024-04-06T10:15:49.395195+0000","last_fullsized":"2024-04-06T10:15:49.395195+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T09:29:24.717518+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T09:29:24.717518+0000","last_clean_scrub_stamp":"2024-04-06T09:29:24.717518+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T14:16:54.527503+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":361,"reported_epoch":343,"state":"active+clean","last_fresh":"2024-04-06T10:15:49.387668+0000","last_change":"2024-04-06T09:29:25.738138+0000","last_active":"2024-04-06T10:15:49.387668+0000","last_peered":"2024-04-06T10:15:49.387668+0000","last_clean":"2024-04-06T10:15:49.387668+0000","last_became_active":"2024-04-06T09:29:25.737263+0000","last_became_peered":"2024-04-06T09:29:25.737263+0000","last_unstale":"2024-04-06T10:15:49.387668+0000","last_undegraded":"2024-04-06T10:15:49.387668+0000","last_fullsized":"2024-04-06T10:15:49.387668+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T09:29:24.717518+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T09:29:24.717518+0000","last_clean_scrub_stamp":"2024-04-06T09:29:24.717518+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T14:30:58.846520+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":360,"reported_epoch":343,"state":"active+clean","last_fresh":"2024-04-06T10:15:49.819953+0000","last_change":"2024-04-06T09:29:25.739673+0000","last_active":"2024-04-06T10:15:49.819953+0000","last_peered":"2024-04-06T10:15:49.819953+0000","last_clean":"2024-04-06T10:15:49.819953+0000","last_became_active":"2024-04-06T09:29:25.738678+0000","last_became_peered":"2024-04-06T09:29:25.738678+0000","last_unstale":"2024-04-06T10:15:49.819953+0000","last_undegraded":"2024-04-06T10:15:49.819953+0000","last_fullsized":"2024-04-06T10:15:49.819953+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T09:29:24.717518+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T09:29:24.717518+0000","last_clean_scrub_stamp":"2024-04-06T09:29:24.717518+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T12:22:31.093977+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":360,"reported_epoch":342,"state":"active+clean","last_fresh":"2024-04-06T10:15:29.161731+0000","last_change":"2024-04-06T09:29:25.739006+0000","last_active":"2024-04-06T10:15:29.161731+0000","last_peered":"2024-04-06T10:15:29.161731+0000","last_clean":"2024-04-06T10:15:29.161731+0000","last_became_active":"2024-04-06T09:29:25.738605+0000","last_became_peered":"2024-04-06T09:29:25.738605+0000","last_unstale":"2024-04-06T10:15:29.161731+0000","last_undegraded":"2024-04-06T10:15:29.161731+0000","last_fullsized":"2024-04-06T10:15:29.161731+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T09:29:24.717518+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T09:29:24.717518+0000","last_clean_scrub_stamp":"2024-04-06T09:29:24.717518+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T20:26:25.794984+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":361,"reported_epoch":342,"state":"active+clean","last_fresh":"2024-04-06T10:15:38.753445+0000","last_change":"2024-04-06T09:29:25.739486+0000","last_active":"2024-04-06T10:15:38.753445+0000","last_peered":"2024-04-06T10:15:38.753445+0000","last_clean":"2024-04-06T10:15:38.753445+0000","last_became_active":"2024-04-06T09:29:25.738108+0000","last_became_peered":"2024-04-06T09:29:25.738108+0000","last_unstale":"2024-04-06T10:15:38.753445+0000","last_undegraded":"2024-04-06T10:15:38.753445+0000","last_fullsized":"2024-04-06T10:15:38.753445+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T09:29:24.717518+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T09:29:24.717518+0000","last_clean_scrub_stamp":"2024-04-06T09:29:24.717518+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T18:32:48.633225+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":361,"reported_epoch":343,"state":"active+clean","last_fresh":"2024-04-06T10:15:49.401068+0000","last_change":"2024-04-06T09:29:25.737897+0000","last_active":"2024-04-06T10:15:49.401068+0000","last_peered":"2024-04-06T10:15:49.401068+0000","last_clean":"2024-04-06T10:15:49.401068+0000","last_became_active":"2024-04-06T09:29:25.737369+0000","last_became_peered":"2024-04-06T09:29:25.737369+0000","last_unstale":"2024-04-06T10:15:49.401068+0000","last_undegraded":"2024-04-06T10:15:49.401068+0000","last_fullsized":"2024-04-06T10:15:49.401068+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T09:29:24.717518+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T09:29:24.717518+0000","last_clean_scrub_stamp":"2024-04-06T09:29:24.717518+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T16:57:27.614162+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"12'97","reported_seq":492,"reported_epoch":343,"state":"active+clean","last_fresh":"2024-04-06T10:15:49.389204+0000","last_change":"2024-04-06T09:29:21.046364+0000","last_active":"2024-04-06T10:15:49.389204+0000","last_peered":"2024-04-06T10:15:49.389204+0000","last_clean":"2024-04-06T10:15:49.389204+0000","last_became_active":"2024-04-06T09:29:21.043913+0000","last_became_peered":"2024-04-06T09:29:21.043913+0000","last_unstale":"2024-04-06T10:15:49.389204+0000","last_undegraded":"2024-04-06T10:15:49.389204+0000","last_fullsized":"2024-04-06T10:15:49.389204+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T09:29:19.704883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T09:29:19.704883+0000","last_clean_scrub_stamp":"2024-04-06T09:29:19.704883+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T14:04:37.575084+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":122880,"data_stored":1180736,"data_compressed":60676,"data_compressed_allocated":81920,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738939,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1745176,"kb_used_data":2768,"kb_used_omap":120,"kb_used_meta":1742279,"kb_avail":92626664,"statfs":{"total":96636764160,"available":94849703936,"internally_reserved":0,"allocated":2834432,"data_stored":1495629,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":122911,"internal_metadata":1784094689},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 6 10:15:47 2024","interfaces":[{"interface":"back","average":{"1min":0.503,"5min":0.53000000000000003,"15min":0.60999999999999999},"min":{"1min":0.42699999999999999,"5min":0.40200000000000002,"15min":0.26400000000000001},"max":{"1min":0.66300000000000003,"5min":0.90300000000000002,"15min":15.619},"last":0.42199999999999999},{"interface":"front","average":{"1min":0.67600000000000005,"5min":0.68100000000000005,"15min":0.76300000000000001},"min":{"1min":0.57099999999999995,"5min":0.55300000000000005,"15min":0.36399999999999999},"max":{"1min":0.76200000000000001,"5min":0.998,"15min":15.901},"last":0.54100000000000004}]},{"osd":1,"last update":"Sat Apr 6 10:15:47 2024","interfaces":[{"interface":"back","average":{"1min":0.58999999999999997,"5min":0.60199999999999998,"15min":0.61499999999999999},"min":{"1min":0.51900000000000002,"5min":0.47099999999999997,"15min":0.35399999999999998},"max":{"1min":0.65700000000000003,"5min":1.0629999999999999,"15min":4.3579999999999997},"last":0.48799999999999999},{"interface":"front","average":{"1min":0.56399999999999995,"5min":0.58599999999999997,"15min":0.61499999999999999},"min":{"1min":0.46999999999999997,"5min":0.45300000000000001,"15min":0.32500000000000001},"max":{"1min":0.67300000000000004,"5min":0.94799999999999995,"15min":4.2119999999999997},"last":0.64000000000000001}]}]},{"osd":1,"up_from":8,"seq":34359738939,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1310172,"kb_used_data":2828,"kb_used_omap":236,"kb_used_meta":1307091,"kb_avail":93061668,"statfs":{"total":96636764160,"available":95295148032,"internally_reserved":0,"allocated":2895872,"data_stored":2085997,"data_compressed":30338,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":241997,"internal_metadata":1338461875},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 6 10:16:15 2024","interfaces":[{"interface":"back","average":{"1min":0.51500000000000001,"5min":0.51400000000000001,"15min":0.53800000000000003},"min":{"1min":0.38200000000000001,"5min":0.38200000000000001,"15min":0.214},"max":{"1min":0.68600000000000005,"5min":0.73799999999999999,"15min":4.3319999999999999},"last":0.44800000000000001},{"interface":"front","average":{"1min":0.65300000000000002,"5min":0.65100000000000002,"15min":0.64200000000000002},"min":{"1min":0.497,"5min":0.497,"15min":0.255},"max":{"1min":0.75600000000000001,"5min":0.86099999999999999,"15min":4.2960000000000003},"last":0.59399999999999997}]},{"osd":2,"last update":"Sat Apr 6 10:15:56 2024","interfaces":[{"interface":"back","average":{"1min":0.64400000000000002,"5min":0.64500000000000002,"15min":0.67500000000000004},"min":{"1min":0.47899999999999998,"5min":0.47899999999999998,"15min":0.316},"max":{"1min":0.80100000000000005,"5min":0.96599999999999997,"15min":4.2229999999999999},"last":0.50900000000000001},{"interface":"front","average":{"1min":0.63400000000000001,"5min":0.63300000000000001,"15min":0.65100000000000002},"min":{"1min":0.46999999999999997,"5min":0.44900000000000001,"15min":0.29299999999999998},"max":{"1min":0.82199999999999995,"5min":1.26,"15min":3.1579999999999999},"last":0.65500000000000003}]}]},{"osd":0,"up_from":8,"seq":34359738939,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1807388,"kb_used_data":2828,"kb_used_omap":281,"kb_used_meta":1804262,"kb_avail":92564452,"statfs":{"total":96636764160,"available":94785998848,"internally_reserved":0,"allocated":2895872,"data_stored":2085997,"data_compressed":30338,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":288743,"internal_metadata":1847564313},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Apr 6 10:15:49 2024","interfaces":[{"interface":"back","average":{"1min":0.53200000000000003,"5min":0.54900000000000004,"15min":0.55300000000000005},"min":{"1min":0.44800000000000001,"5min":0.24199999999999999,"15min":0.17999999999999999},"max":{"1min":0.70699999999999996,"5min":2.173,"15min":3.7679999999999998},"last":0.61399999999999999},{"interface":"front","average":{"1min":0.63,"5min":0.60899999999999999,"15min":0.61499999999999999},"min":{"1min":0.5,"5min":0.42399999999999999,"15min":0.216},"max":{"1min":0.81399999999999995,"5min":1.105,"15min":3.9969999999999999},"last":0.68999999999999995}]},{"osd":2,"last update":"Sat Apr 6 10:15:49 2024","interfaces":[{"interface":"back","average":{"1min":0.66700000000000004,"5min":0.65700000000000003,"15min":0.65400000000000003},"min":{"1min":0.52100000000000002,"5min":0.47099999999999997,"15min":0.26100000000000001},"max":{"1min":0.79600000000000004,"5min":1.3580000000000001,"15min":4.1609999999999996},"last":0.80700000000000005},{"interface":"front","average":{"1min":0.60099999999999998,"5min":0.61499999999999999,"15min":0.626},"min":{"1min":0.48799999999999999,"5min":0.28100000000000003,"15min":0.23200000000000001},"max":{"1min":0.69399999999999995,"5min":0.95499999999999996,"15min":2.4729999999999999},"last":0.65300000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":590368,"data_compressed":30338,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":590368,"data_compressed":30338,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-06T10:16:37.960 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 pg dump --format=json 2024-04-06T10:16:38.283 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.1 has been restored 2024-04-06T10:16:38.288 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-06T10:16:38.291+0000 7fec31c44640 0 --2- 172.21.15.6:0/4258216481 >> v2:172.21.15.6:3300/0 conn(0x7fec2c0b1030 0x7fec2c06fa60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request get_initial_auth_request returned -2 2024-04-06T10:16:38.464 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-04-06T10:16:38.465 INFO:teuthology.orchestra.run.smithi006.stderr:dumped all 2024-04-06T10:16:38.482 INFO:teuthology.orchestra.run.smithi006.stdout:{"pg_ready":true,"pg_map":{"version":1762,"stamp":"2024-04-06T10:16:38.324751+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":4862736,"kb_used_data":8424,"kb_used_omap":638,"kb_used_meta":4853633,"kb_avail":278252784,"statfs":{"total":289910292480,"available":284930850816,"internally_reserved":0,"allocated":8626176,"data_stored":5667623,"data_compressed":60676,"data_compressed_allocated":81920,"data_compressed_original":1146880,"omap_allocated":653651,"internal_metadata":4970120877},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003037"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":360,"reported_epoch":343,"state":"active+clean","last_fresh":"2024-04-06T10:15:49.390117+0000","last_change":"2024-04-06T09:29:25.738609+0000","last_active":"2024-04-06T10:15:49.390117+0000","last_peered":"2024-04-06T10:15:49.390117+0000","last_clean":"2024-04-06T10:15:49.390117+0000","last_became_active":"2024-04-06T09:29:25.737557+0000","last_became_peered":"2024-04-06T09:29:25.737557+0000","last_unstale":"2024-04-06T10:15:49.390117+0000","last_undegraded":"2024-04-06T10:15:49.390117+0000","last_fullsized":"2024-04-06T10:15:49.390117+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T09:29:24.717518+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T09:29:24.717518+0000","last_clean_scrub_stamp":"2024-04-06T09:29:24.717518+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T15:44:35.416009+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":360,"reported_epoch":343,"state":"active+clean","last_fresh":"2024-04-06T10:15:49.403104+0000","last_change":"2024-04-06T09:29:25.738328+0000","last_active":"2024-04-06T10:15:49.403104+0000","last_peered":"2024-04-06T10:15:49.403104+0000","last_clean":"2024-04-06T10:15:49.403104+0000","last_became_active":"2024-04-06T09:29:25.737348+0000","last_became_peered":"2024-04-06T09:29:25.737348+0000","last_unstale":"2024-04-06T10:15:49.403104+0000","last_undegraded":"2024-04-06T10:15:49.403104+0000","last_fullsized":"2024-04-06T10:15:49.403104+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T09:29:24.717518+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T09:29:24.717518+0000","last_clean_scrub_stamp":"2024-04-06T09:29:24.717518+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T10:00:30.760505+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":361,"reported_epoch":343,"state":"active+clean","last_fresh":"2024-04-06T10:15:49.395195+0000","last_change":"2024-04-06T09:29:25.738960+0000","last_active":"2024-04-06T10:15:49.395195+0000","last_peered":"2024-04-06T10:15:49.395195+0000","last_clean":"2024-04-06T10:15:49.395195+0000","last_became_active":"2024-04-06T09:29:25.738591+0000","last_became_peered":"2024-04-06T09:29:25.738591+0000","last_unstale":"2024-04-06T10:15:49.395195+0000","last_undegraded":"2024-04-06T10:15:49.395195+0000","last_fullsized":"2024-04-06T10:15:49.395195+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T09:29:24.717518+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T09:29:24.717518+0000","last_clean_scrub_stamp":"2024-04-06T09:29:24.717518+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T14:16:54.527503+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":361,"reported_epoch":343,"state":"active+clean","last_fresh":"2024-04-06T10:15:49.387668+0000","last_change":"2024-04-06T09:29:25.738138+0000","last_active":"2024-04-06T10:15:49.387668+0000","last_peered":"2024-04-06T10:15:49.387668+0000","last_clean":"2024-04-06T10:15:49.387668+0000","last_became_active":"2024-04-06T09:29:25.737263+0000","last_became_peered":"2024-04-06T09:29:25.737263+0000","last_unstale":"2024-04-06T10:15:49.387668+0000","last_undegraded":"2024-04-06T10:15:49.387668+0000","last_fullsized":"2024-04-06T10:15:49.387668+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T09:29:24.717518+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T09:29:24.717518+0000","last_clean_scrub_stamp":"2024-04-06T09:29:24.717518+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T14:30:58.846520+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":360,"reported_epoch":343,"state":"active+clean","last_fresh":"2024-04-06T10:15:49.819953+0000","last_change":"2024-04-06T09:29:25.739673+0000","last_active":"2024-04-06T10:15:49.819953+0000","last_peered":"2024-04-06T10:15:49.819953+0000","last_clean":"2024-04-06T10:15:49.819953+0000","last_became_active":"2024-04-06T09:29:25.738678+0000","last_became_peered":"2024-04-06T09:29:25.738678+0000","last_unstale":"2024-04-06T10:15:49.819953+0000","last_undegraded":"2024-04-06T10:15:49.819953+0000","last_fullsized":"2024-04-06T10:15:49.819953+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T09:29:24.717518+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T09:29:24.717518+0000","last_clean_scrub_stamp":"2024-04-06T09:29:24.717518+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T12:22:31.093977+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":360,"reported_epoch":342,"state":"active+clean","last_fresh":"2024-04-06T10:15:29.161731+0000","last_change":"2024-04-06T09:29:25.739006+0000","last_active":"2024-04-06T10:15:29.161731+0000","last_peered":"2024-04-06T10:15:29.161731+0000","last_clean":"2024-04-06T10:15:29.161731+0000","last_became_active":"2024-04-06T09:29:25.738605+0000","last_became_peered":"2024-04-06T09:29:25.738605+0000","last_unstale":"2024-04-06T10:15:29.161731+0000","last_undegraded":"2024-04-06T10:15:29.161731+0000","last_fullsized":"2024-04-06T10:15:29.161731+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T09:29:24.717518+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T09:29:24.717518+0000","last_clean_scrub_stamp":"2024-04-06T09:29:24.717518+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T20:26:25.794984+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":361,"reported_epoch":342,"state":"active+clean","last_fresh":"2024-04-06T10:15:38.753445+0000","last_change":"2024-04-06T09:29:25.739486+0000","last_active":"2024-04-06T10:15:38.753445+0000","last_peered":"2024-04-06T10:15:38.753445+0000","last_clean":"2024-04-06T10:15:38.753445+0000","last_became_active":"2024-04-06T09:29:25.738108+0000","last_became_peered":"2024-04-06T09:29:25.738108+0000","last_unstale":"2024-04-06T10:15:38.753445+0000","last_undegraded":"2024-04-06T10:15:38.753445+0000","last_fullsized":"2024-04-06T10:15:38.753445+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T09:29:24.717518+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T09:29:24.717518+0000","last_clean_scrub_stamp":"2024-04-06T09:29:24.717518+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T18:32:48.633225+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":361,"reported_epoch":343,"state":"active+clean","last_fresh":"2024-04-06T10:15:49.401068+0000","last_change":"2024-04-06T09:29:25.737897+0000","last_active":"2024-04-06T10:15:49.401068+0000","last_peered":"2024-04-06T10:15:49.401068+0000","last_clean":"2024-04-06T10:15:49.401068+0000","last_became_active":"2024-04-06T09:29:25.737369+0000","last_became_peered":"2024-04-06T09:29:25.737369+0000","last_unstale":"2024-04-06T10:15:49.401068+0000","last_undegraded":"2024-04-06T10:15:49.401068+0000","last_fullsized":"2024-04-06T10:15:49.401068+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T09:29:24.717518+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T09:29:24.717518+0000","last_clean_scrub_stamp":"2024-04-06T09:29:24.717518+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T16:57:27.614162+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"12'97","reported_seq":492,"reported_epoch":343,"state":"active+clean","last_fresh":"2024-04-06T10:15:49.389204+0000","last_change":"2024-04-06T09:29:21.046364+0000","last_active":"2024-04-06T10:15:49.389204+0000","last_peered":"2024-04-06T10:15:49.389204+0000","last_clean":"2024-04-06T10:15:49.389204+0000","last_became_active":"2024-04-06T09:29:21.043913+0000","last_became_peered":"2024-04-06T09:29:21.043913+0000","last_unstale":"2024-04-06T10:15:49.389204+0000","last_undegraded":"2024-04-06T10:15:49.389204+0000","last_fullsized":"2024-04-06T10:15:49.389204+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T09:29:19.704883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T09:29:19.704883+0000","last_clean_scrub_stamp":"2024-04-06T09:29:19.704883+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T14:04:37.575084+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":122880,"data_stored":1180736,"data_compressed":60676,"data_compressed_allocated":81920,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738939,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1745176,"kb_used_data":2768,"kb_used_omap":120,"kb_used_meta":1742279,"kb_avail":92626664,"statfs":{"total":96636764160,"available":94849703936,"internally_reserved":0,"allocated":2834432,"data_stored":1495629,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":122911,"internal_metadata":1784094689},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 6 10:15:47 2024","interfaces":[{"interface":"back","average":{"1min":0.503,"5min":0.53000000000000003,"15min":0.60999999999999999},"min":{"1min":0.42699999999999999,"5min":0.40200000000000002,"15min":0.26400000000000001},"max":{"1min":0.66300000000000003,"5min":0.90300000000000002,"15min":15.619},"last":0.42199999999999999},{"interface":"front","average":{"1min":0.67600000000000005,"5min":0.68100000000000005,"15min":0.76300000000000001},"min":{"1min":0.57099999999999995,"5min":0.55300000000000005,"15min":0.36399999999999999},"max":{"1min":0.76200000000000001,"5min":0.998,"15min":15.901},"last":0.54100000000000004}]},{"osd":1,"last update":"Sat Apr 6 10:15:47 2024","interfaces":[{"interface":"back","average":{"1min":0.58999999999999997,"5min":0.60199999999999998,"15min":0.61499999999999999},"min":{"1min":0.51900000000000002,"5min":0.47099999999999997,"15min":0.35399999999999998},"max":{"1min":0.65700000000000003,"5min":1.0629999999999999,"15min":4.3579999999999997},"last":0.48799999999999999},{"interface":"front","average":{"1min":0.56399999999999995,"5min":0.58599999999999997,"15min":0.61499999999999999},"min":{"1min":0.46999999999999997,"5min":0.45300000000000001,"15min":0.32500000000000001},"max":{"1min":0.67300000000000004,"5min":0.94799999999999995,"15min":4.2119999999999997},"last":0.64000000000000001}]}]},{"osd":1,"up_from":8,"seq":34359738939,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1310172,"kb_used_data":2828,"kb_used_omap":236,"kb_used_meta":1307091,"kb_avail":93061668,"statfs":{"total":96636764160,"available":95295148032,"internally_reserved":0,"allocated":2895872,"data_stored":2085997,"data_compressed":30338,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":241997,"internal_metadata":1338461875},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 6 10:16:15 2024","interfaces":[{"interface":"back","average":{"1min":0.51500000000000001,"5min":0.51400000000000001,"15min":0.53800000000000003},"min":{"1min":0.38200000000000001,"5min":0.38200000000000001,"15min":0.214},"max":{"1min":0.68600000000000005,"5min":0.73799999999999999,"15min":4.3319999999999999},"last":0.44800000000000001},{"interface":"front","average":{"1min":0.65300000000000002,"5min":0.65100000000000002,"15min":0.64200000000000002},"min":{"1min":0.497,"5min":0.497,"15min":0.255},"max":{"1min":0.75600000000000001,"5min":0.86099999999999999,"15min":4.2960000000000003},"last":0.59399999999999997}]},{"osd":2,"last update":"Sat Apr 6 10:15:56 2024","interfaces":[{"interface":"back","average":{"1min":0.64400000000000002,"5min":0.64500000000000002,"15min":0.67500000000000004},"min":{"1min":0.47899999999999998,"5min":0.47899999999999998,"15min":0.316},"max":{"1min":0.80100000000000005,"5min":0.96599999999999997,"15min":4.2229999999999999},"last":0.50900000000000001},{"interface":"front","average":{"1min":0.63400000000000001,"5min":0.63300000000000001,"15min":0.65100000000000002},"min":{"1min":0.46999999999999997,"5min":0.44900000000000001,"15min":0.29299999999999998},"max":{"1min":0.82199999999999995,"5min":1.26,"15min":3.1579999999999999},"last":0.65500000000000003}]}]},{"osd":0,"up_from":8,"seq":34359738939,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1807388,"kb_used_data":2828,"kb_used_omap":281,"kb_used_meta":1804262,"kb_avail":92564452,"statfs":{"total":96636764160,"available":94785998848,"internally_reserved":0,"allocated":2895872,"data_stored":2085997,"data_compressed":30338,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":288743,"internal_metadata":1847564313},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Apr 6 10:15:49 2024","interfaces":[{"interface":"back","average":{"1min":0.53200000000000003,"5min":0.54900000000000004,"15min":0.55300000000000005},"min":{"1min":0.44800000000000001,"5min":0.24199999999999999,"15min":0.17999999999999999},"max":{"1min":0.70699999999999996,"5min":2.173,"15min":3.7679999999999998},"last":0.61399999999999999},{"interface":"front","average":{"1min":0.63,"5min":0.60899999999999999,"15min":0.61499999999999999},"min":{"1min":0.5,"5min":0.42399999999999999,"15min":0.216},"max":{"1min":0.81399999999999995,"5min":1.105,"15min":3.9969999999999999},"last":0.68999999999999995}]},{"osd":2,"last update":"Sat Apr 6 10:15:49 2024","interfaces":[{"interface":"back","average":{"1min":0.66700000000000004,"5min":0.65700000000000003,"15min":0.65400000000000003},"min":{"1min":0.52100000000000002,"5min":0.47099999999999997,"15min":0.26100000000000001},"max":{"1min":0.79600000000000004,"5min":1.3580000000000001,"15min":4.1609999999999996},"last":0.80700000000000005},{"interface":"front","average":{"1min":0.60099999999999998,"5min":0.61499999999999999,"15min":0.626},"min":{"1min":0.48799999999999999,"5min":0.28100000000000003,"15min":0.23200000000000001},"max":{"1min":0.69399999999999995,"5min":0.95499999999999996,"15min":2.4729999999999999},"last":0.65300000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":590368,"data_compressed":30338,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":590368,"data_compressed":30338,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-06T10:16:38.483 INFO:tasks.ceph.ceph_manager.cluster1:clean! 2024-04-06T10:16:38.483 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 pg dump --format=json 2024-04-06T10:16:38.776 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-04-06T10:16:38.776 INFO:teuthology.orchestra.run.smithi006.stderr:dumped all 2024-04-06T10:16:38.795 INFO:teuthology.orchestra.run.smithi006.stdout:{"pg_ready":true,"pg_map":{"version":1762,"stamp":"2024-04-06T10:16:38.324751+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":4862736,"kb_used_data":8424,"kb_used_omap":638,"kb_used_meta":4853633,"kb_avail":278252784,"statfs":{"total":289910292480,"available":284930850816,"internally_reserved":0,"allocated":8626176,"data_stored":5667623,"data_compressed":60676,"data_compressed_allocated":81920,"data_compressed_original":1146880,"omap_allocated":653651,"internal_metadata":4970120877},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003037"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":360,"reported_epoch":343,"state":"active+clean","last_fresh":"2024-04-06T10:15:49.390117+0000","last_change":"2024-04-06T09:29:25.738609+0000","last_active":"2024-04-06T10:15:49.390117+0000","last_peered":"2024-04-06T10:15:49.390117+0000","last_clean":"2024-04-06T10:15:49.390117+0000","last_became_active":"2024-04-06T09:29:25.737557+0000","last_became_peered":"2024-04-06T09:29:25.737557+0000","last_unstale":"2024-04-06T10:15:49.390117+0000","last_undegraded":"2024-04-06T10:15:49.390117+0000","last_fullsized":"2024-04-06T10:15:49.390117+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T09:29:24.717518+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T09:29:24.717518+0000","last_clean_scrub_stamp":"2024-04-06T09:29:24.717518+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T15:44:35.416009+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":360,"reported_epoch":343,"state":"active+clean","last_fresh":"2024-04-06T10:15:49.403104+0000","last_change":"2024-04-06T09:29:25.738328+0000","last_active":"2024-04-06T10:15:49.403104+0000","last_peered":"2024-04-06T10:15:49.403104+0000","last_clean":"2024-04-06T10:15:49.403104+0000","last_became_active":"2024-04-06T09:29:25.737348+0000","last_became_peered":"2024-04-06T09:29:25.737348+0000","last_unstale":"2024-04-06T10:15:49.403104+0000","last_undegraded":"2024-04-06T10:15:49.403104+0000","last_fullsized":"2024-04-06T10:15:49.403104+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T09:29:24.717518+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T09:29:24.717518+0000","last_clean_scrub_stamp":"2024-04-06T09:29:24.717518+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T10:00:30.760505+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":361,"reported_epoch":343,"state":"active+clean","last_fresh":"2024-04-06T10:15:49.395195+0000","last_change":"2024-04-06T09:29:25.738960+0000","last_active":"2024-04-06T10:15:49.395195+0000","last_peered":"2024-04-06T10:15:49.395195+0000","last_clean":"2024-04-06T10:15:49.395195+0000","last_became_active":"2024-04-06T09:29:25.738591+0000","last_became_peered":"2024-04-06T09:29:25.738591+0000","last_unstale":"2024-04-06T10:15:49.395195+0000","last_undegraded":"2024-04-06T10:15:49.395195+0000","last_fullsized":"2024-04-06T10:15:49.395195+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T09:29:24.717518+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T09:29:24.717518+0000","last_clean_scrub_stamp":"2024-04-06T09:29:24.717518+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T14:16:54.527503+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":361,"reported_epoch":343,"state":"active+clean","last_fresh":"2024-04-06T10:15:49.387668+0000","last_change":"2024-04-06T09:29:25.738138+0000","last_active":"2024-04-06T10:15:49.387668+0000","last_peered":"2024-04-06T10:15:49.387668+0000","last_clean":"2024-04-06T10:15:49.387668+0000","last_became_active":"2024-04-06T09:29:25.737263+0000","last_became_peered":"2024-04-06T09:29:25.737263+0000","last_unstale":"2024-04-06T10:15:49.387668+0000","last_undegraded":"2024-04-06T10:15:49.387668+0000","last_fullsized":"2024-04-06T10:15:49.387668+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T09:29:24.717518+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T09:29:24.717518+0000","last_clean_scrub_stamp":"2024-04-06T09:29:24.717518+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T14:30:58.846520+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":360,"reported_epoch":343,"state":"active+clean","last_fresh":"2024-04-06T10:15:49.819953+0000","last_change":"2024-04-06T09:29:25.739673+0000","last_active":"2024-04-06T10:15:49.819953+0000","last_peered":"2024-04-06T10:15:49.819953+0000","last_clean":"2024-04-06T10:15:49.819953+0000","last_became_active":"2024-04-06T09:29:25.738678+0000","last_became_peered":"2024-04-06T09:29:25.738678+0000","last_unstale":"2024-04-06T10:15:49.819953+0000","last_undegraded":"2024-04-06T10:15:49.819953+0000","last_fullsized":"2024-04-06T10:15:49.819953+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T09:29:24.717518+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T09:29:24.717518+0000","last_clean_scrub_stamp":"2024-04-06T09:29:24.717518+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T12:22:31.093977+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":360,"reported_epoch":342,"state":"active+clean","last_fresh":"2024-04-06T10:15:29.161731+0000","last_change":"2024-04-06T09:29:25.739006+0000","last_active":"2024-04-06T10:15:29.161731+0000","last_peered":"2024-04-06T10:15:29.161731+0000","last_clean":"2024-04-06T10:15:29.161731+0000","last_became_active":"2024-04-06T09:29:25.738605+0000","last_became_peered":"2024-04-06T09:29:25.738605+0000","last_unstale":"2024-04-06T10:15:29.161731+0000","last_undegraded":"2024-04-06T10:15:29.161731+0000","last_fullsized":"2024-04-06T10:15:29.161731+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T09:29:24.717518+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T09:29:24.717518+0000","last_clean_scrub_stamp":"2024-04-06T09:29:24.717518+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T20:26:25.794984+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":361,"reported_epoch":342,"state":"active+clean","last_fresh":"2024-04-06T10:15:38.753445+0000","last_change":"2024-04-06T09:29:25.739486+0000","last_active":"2024-04-06T10:15:38.753445+0000","last_peered":"2024-04-06T10:15:38.753445+0000","last_clean":"2024-04-06T10:15:38.753445+0000","last_became_active":"2024-04-06T09:29:25.738108+0000","last_became_peered":"2024-04-06T09:29:25.738108+0000","last_unstale":"2024-04-06T10:15:38.753445+0000","last_undegraded":"2024-04-06T10:15:38.753445+0000","last_fullsized":"2024-04-06T10:15:38.753445+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T09:29:24.717518+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T09:29:24.717518+0000","last_clean_scrub_stamp":"2024-04-06T09:29:24.717518+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T18:32:48.633225+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":361,"reported_epoch":343,"state":"active+clean","last_fresh":"2024-04-06T10:15:49.401068+0000","last_change":"2024-04-06T09:29:25.737897+0000","last_active":"2024-04-06T10:15:49.401068+0000","last_peered":"2024-04-06T10:15:49.401068+0000","last_clean":"2024-04-06T10:15:49.401068+0000","last_became_active":"2024-04-06T09:29:25.737369+0000","last_became_peered":"2024-04-06T09:29:25.737369+0000","last_unstale":"2024-04-06T10:15:49.401068+0000","last_undegraded":"2024-04-06T10:15:49.401068+0000","last_fullsized":"2024-04-06T10:15:49.401068+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T09:29:24.717518+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T09:29:24.717518+0000","last_clean_scrub_stamp":"2024-04-06T09:29:24.717518+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T16:57:27.614162+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"12'97","reported_seq":492,"reported_epoch":343,"state":"active+clean","last_fresh":"2024-04-06T10:15:49.389204+0000","last_change":"2024-04-06T09:29:21.046364+0000","last_active":"2024-04-06T10:15:49.389204+0000","last_peered":"2024-04-06T10:15:49.389204+0000","last_clean":"2024-04-06T10:15:49.389204+0000","last_became_active":"2024-04-06T09:29:21.043913+0000","last_became_peered":"2024-04-06T09:29:21.043913+0000","last_unstale":"2024-04-06T10:15:49.389204+0000","last_undegraded":"2024-04-06T10:15:49.389204+0000","last_fullsized":"2024-04-06T10:15:49.389204+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T09:29:19.704883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T09:29:19.704883+0000","last_clean_scrub_stamp":"2024-04-06T09:29:19.704883+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T14:04:37.575084+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":122880,"data_stored":1180736,"data_compressed":60676,"data_compressed_allocated":81920,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738939,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1745176,"kb_used_data":2768,"kb_used_omap":120,"kb_used_meta":1742279,"kb_avail":92626664,"statfs":{"total":96636764160,"available":94849703936,"internally_reserved":0,"allocated":2834432,"data_stored":1495629,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":122911,"internal_metadata":1784094689},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 6 10:15:47 2024","interfaces":[{"interface":"back","average":{"1min":0.503,"5min":0.53000000000000003,"15min":0.60999999999999999},"min":{"1min":0.42699999999999999,"5min":0.40200000000000002,"15min":0.26400000000000001},"max":{"1min":0.66300000000000003,"5min":0.90300000000000002,"15min":15.619},"last":0.42199999999999999},{"interface":"front","average":{"1min":0.67600000000000005,"5min":0.68100000000000005,"15min":0.76300000000000001},"min":{"1min":0.57099999999999995,"5min":0.55300000000000005,"15min":0.36399999999999999},"max":{"1min":0.76200000000000001,"5min":0.998,"15min":15.901},"last":0.54100000000000004}]},{"osd":1,"last update":"Sat Apr 6 10:15:47 2024","interfaces":[{"interface":"back","average":{"1min":0.58999999999999997,"5min":0.60199999999999998,"15min":0.61499999999999999},"min":{"1min":0.51900000000000002,"5min":0.47099999999999997,"15min":0.35399999999999998},"max":{"1min":0.65700000000000003,"5min":1.0629999999999999,"15min":4.3579999999999997},"last":0.48799999999999999},{"interface":"front","average":{"1min":0.56399999999999995,"5min":0.58599999999999997,"15min":0.61499999999999999},"min":{"1min":0.46999999999999997,"5min":0.45300000000000001,"15min":0.32500000000000001},"max":{"1min":0.67300000000000004,"5min":0.94799999999999995,"15min":4.2119999999999997},"last":0.64000000000000001}]}]},{"osd":1,"up_from":8,"seq":34359738939,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1310172,"kb_used_data":2828,"kb_used_omap":236,"kb_used_meta":1307091,"kb_avail":93061668,"statfs":{"total":96636764160,"available":95295148032,"internally_reserved":0,"allocated":2895872,"data_stored":2085997,"data_compressed":30338,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":241997,"internal_metadata":1338461875},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 6 10:16:15 2024","interfaces":[{"interface":"back","average":{"1min":0.51500000000000001,"5min":0.51400000000000001,"15min":0.53800000000000003},"min":{"1min":0.38200000000000001,"5min":0.38200000000000001,"15min":0.214},"max":{"1min":0.68600000000000005,"5min":0.73799999999999999,"15min":4.3319999999999999},"last":0.44800000000000001},{"interface":"front","average":{"1min":0.65300000000000002,"5min":0.65100000000000002,"15min":0.64200000000000002},"min":{"1min":0.497,"5min":0.497,"15min":0.255},"max":{"1min":0.75600000000000001,"5min":0.86099999999999999,"15min":4.2960000000000003},"last":0.59399999999999997}]},{"osd":2,"last update":"Sat Apr 6 10:15:56 2024","interfaces":[{"interface":"back","average":{"1min":0.64400000000000002,"5min":0.64500000000000002,"15min":0.67500000000000004},"min":{"1min":0.47899999999999998,"5min":0.47899999999999998,"15min":0.316},"max":{"1min":0.80100000000000005,"5min":0.96599999999999997,"15min":4.2229999999999999},"last":0.50900000000000001},{"interface":"front","average":{"1min":0.63400000000000001,"5min":0.63300000000000001,"15min":0.65100000000000002},"min":{"1min":0.46999999999999997,"5min":0.44900000000000001,"15min":0.29299999999999998},"max":{"1min":0.82199999999999995,"5min":1.26,"15min":3.1579999999999999},"last":0.65500000000000003}]}]},{"osd":0,"up_from":8,"seq":34359738939,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1807388,"kb_used_data":2828,"kb_used_omap":281,"kb_used_meta":1804262,"kb_avail":92564452,"statfs":{"total":96636764160,"available":94785998848,"internally_reserved":0,"allocated":2895872,"data_stored":2085997,"data_compressed":30338,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":288743,"internal_metadata":1847564313},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Apr 6 10:15:49 2024","interfaces":[{"interface":"back","average":{"1min":0.53200000000000003,"5min":0.54900000000000004,"15min":0.55300000000000005},"min":{"1min":0.44800000000000001,"5min":0.24199999999999999,"15min":0.17999999999999999},"max":{"1min":0.70699999999999996,"5min":2.173,"15min":3.7679999999999998},"last":0.61399999999999999},{"interface":"front","average":{"1min":0.63,"5min":0.60899999999999999,"15min":0.61499999999999999},"min":{"1min":0.5,"5min":0.42399999999999999,"15min":0.216},"max":{"1min":0.81399999999999995,"5min":1.105,"15min":3.9969999999999999},"last":0.68999999999999995}]},{"osd":2,"last update":"Sat Apr 6 10:15:49 2024","interfaces":[{"interface":"back","average":{"1min":0.66700000000000004,"5min":0.65700000000000003,"15min":0.65400000000000003},"min":{"1min":0.52100000000000002,"5min":0.47099999999999997,"15min":0.26100000000000001},"max":{"1min":0.79600000000000004,"5min":1.3580000000000001,"15min":4.1609999999999996},"last":0.80700000000000005},{"interface":"front","average":{"1min":0.60099999999999998,"5min":0.61499999999999999,"15min":0.626},"min":{"1min":0.48799999999999999,"5min":0.28100000000000003,"15min":0.23200000000000001},"max":{"1min":0.69399999999999995,"5min":0.95499999999999996,"15min":2.4729999999999999},"last":0.65300000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":590368,"data_compressed":30338,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":590368,"data_compressed":30338,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-06T10:16:38.796 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 osd dump --format=json 2024-04-06T10:16:39.092 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-04-06T10:16:39.093 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":344,"fsid":"06a46034-4ef3-4e36-9fc5-05c8000433c6","created":"2024-04-06T09:29:08.480618+0000","modified":"2024-04-06T10:15:50.379053+0000","last_up_change":"2024-04-06T09:29:17.701553+0000","last_in_change":"2024-04-06T09:29:09.417013+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-06T09:29:19.376917+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"12","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":2.9900000095367432,"score_stable":2.9900000095367432,"optimal_score":0.67000001668930054,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-04-06T09:29:23.842851+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"15","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"cb53ae9a-8742-4bb3-8546-e051ff9595dc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6808","nonce":23799},{"type":"v1","addr":"172.21.15.6:6809","nonce":23799}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6810","nonce":23799},{"type":"v1","addr":"172.21.15.6:6811","nonce":23799}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6814","nonce":23799},{"type":"v1","addr":"172.21.15.6:6815","nonce":23799}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6812","nonce":23799},{"type":"v1","addr":"172.21.15.6:6813","nonce":23799}]},"public_addr":"172.21.15.6:6809/23799","cluster_addr":"172.21.15.6:6811/23799","heartbeat_back_addr":"172.21.15.6:6815/23799","heartbeat_front_addr":"172.21.15.6:6813/23799","state":["exists","up"]},{"osd":1,"uuid":"9da3b4ba-409a-4e1c-9589-1d70a6c058f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":29,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6800","nonce":23798},{"type":"v1","addr":"172.21.15.6:6801","nonce":23798}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6802","nonce":23798},{"type":"v1","addr":"172.21.15.6:6803","nonce":23798}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6806","nonce":23798},{"type":"v1","addr":"172.21.15.6:6807","nonce":23798}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6804","nonce":23798},{"type":"v1","addr":"172.21.15.6:6805","nonce":23798}]},"public_addr":"172.21.15.6:6801/23798","cluster_addr":"172.21.15.6:6803/23798","heartbeat_back_addr":"172.21.15.6:6807/23798","heartbeat_front_addr":"172.21.15.6:6805/23798","state":["exists","up"]},{"osd":2,"uuid":"2d732d23-05b4-4b5a-bbb2-e022b113a84c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6816","nonce":23800},{"type":"v1","addr":"172.21.15.6:6817","nonce":23800}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6818","nonce":23800},{"type":"v1","addr":"172.21.15.6:6819","nonce":23800}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6822","nonce":23800},{"type":"v1","addr":"172.21.15.6:6823","nonce":23800}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6820","nonce":23800},{"type":"v1","addr":"172.21.15.6:6821","nonce":23800}]},"public_addr":"172.21.15.6:6817/23800","cluster_addr":"172.21.15.6:6819/23800","heartbeat_back_addr":"172.21.15.6:6823/23800","heartbeat_front_addr":"172.21.15.6:6821/23800","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T09:29:14.935316+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T09:29:14.876558+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T09:29:14.907628+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-06T10:16:40.113 INFO:tasks.ceph:Scrubbing cluster1.osd.0 2024-04-06T10:16:40.113 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 tell osd.0 config set osd_debug_deep_scrub_sleep 0 2024-04-06T10:16:40.276 INFO:teuthology.orchestra.run.smithi006.stdout:{ 2024-04-06T10:16:40.276 INFO:teuthology.orchestra.run.smithi006.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-04-06T10:16:40.276 INFO:teuthology.orchestra.run.smithi006.stdout:} 2024-04-06T10:16:40.293 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 osd deep-scrub 0 2024-04-06T10:16:40.579 INFO:teuthology.orchestra.run.smithi006.stderr:instructed osd(s) 0 to deep-scrub 2024-04-06T10:16:40.598 INFO:tasks.ceph:Scrubbing cluster1.osd.1 2024-04-06T10:16:40.598 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 tell osd.1 config set osd_debug_deep_scrub_sleep 0 2024-04-06T10:16:40.756 INFO:teuthology.orchestra.run.smithi006.stdout:{ 2024-04-06T10:16:40.756 INFO:teuthology.orchestra.run.smithi006.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-04-06T10:16:40.756 INFO:teuthology.orchestra.run.smithi006.stdout:} 2024-04-06T10:16:40.774 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 osd deep-scrub 1 2024-04-06T10:16:41.057 INFO:teuthology.orchestra.run.smithi006.stderr:instructed osd(s) 1 to deep-scrub 2024-04-06T10:16:41.076 INFO:tasks.ceph:Scrubbing cluster1.osd.2 2024-04-06T10:16:41.076 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 tell osd.2 config set osd_debug_deep_scrub_sleep 0 2024-04-06T10:16:41.219 INFO:teuthology.orchestra.run.smithi006.stdout:{ 2024-04-06T10:16:41.219 INFO:teuthology.orchestra.run.smithi006.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-04-06T10:16:41.219 INFO:teuthology.orchestra.run.smithi006.stdout:} 2024-04-06T10:16:41.233 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 osd deep-scrub 2 2024-04-06T10:16:41.508 INFO:teuthology.orchestra.run.smithi006.stderr:instructed osd(s) 2 to deep-scrub 2024-04-06T10:16:41.526 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 pg dump --format=json 2024-04-06T10:16:41.804 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-04-06T10:16:41.805 INFO:teuthology.orchestra.run.smithi006.stderr:dumped all 2024-04-06T10:16:41.822 INFO:teuthology.orchestra.run.smithi006.stdout:{"pg_ready":true,"pg_map":{"version":1763,"stamp":"2024-04-06T10:16:40.325106+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":4862736,"kb_used_data":8424,"kb_used_omap":638,"kb_used_meta":4853633,"kb_avail":278252784,"statfs":{"total":289910292480,"available":284930850816,"internally_reserved":0,"allocated":8626176,"data_stored":5667623,"data_compressed":60676,"data_compressed_allocated":81920,"data_compressed_original":1146880,"omap_allocated":653651,"internal_metadata":4970120877},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002920"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":360,"reported_epoch":343,"state":"active+clean","last_fresh":"2024-04-06T10:15:49.390117+0000","last_change":"2024-04-06T09:29:25.738609+0000","last_active":"2024-04-06T10:15:49.390117+0000","last_peered":"2024-04-06T10:15:49.390117+0000","last_clean":"2024-04-06T10:15:49.390117+0000","last_became_active":"2024-04-06T09:29:25.737557+0000","last_became_peered":"2024-04-06T09:29:25.737557+0000","last_unstale":"2024-04-06T10:15:49.390117+0000","last_undegraded":"2024-04-06T10:15:49.390117+0000","last_fullsized":"2024-04-06T10:15:49.390117+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T09:29:24.717518+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T09:29:24.717518+0000","last_clean_scrub_stamp":"2024-04-06T09:29:24.717518+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T15:44:35.416009+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":360,"reported_epoch":343,"state":"active+clean","last_fresh":"2024-04-06T10:15:49.403104+0000","last_change":"2024-04-06T09:29:25.738328+0000","last_active":"2024-04-06T10:15:49.403104+0000","last_peered":"2024-04-06T10:15:49.403104+0000","last_clean":"2024-04-06T10:15:49.403104+0000","last_became_active":"2024-04-06T09:29:25.737348+0000","last_became_peered":"2024-04-06T09:29:25.737348+0000","last_unstale":"2024-04-06T10:15:49.403104+0000","last_undegraded":"2024-04-06T10:15:49.403104+0000","last_fullsized":"2024-04-06T10:15:49.403104+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T09:29:24.717518+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T09:29:24.717518+0000","last_clean_scrub_stamp":"2024-04-06T09:29:24.717518+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T10:00:30.760505+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":361,"reported_epoch":343,"state":"active+clean","last_fresh":"2024-04-06T10:15:49.395195+0000","last_change":"2024-04-06T09:29:25.738960+0000","last_active":"2024-04-06T10:15:49.395195+0000","last_peered":"2024-04-06T10:15:49.395195+0000","last_clean":"2024-04-06T10:15:49.395195+0000","last_became_active":"2024-04-06T09:29:25.738591+0000","last_became_peered":"2024-04-06T09:29:25.738591+0000","last_unstale":"2024-04-06T10:15:49.395195+0000","last_undegraded":"2024-04-06T10:15:49.395195+0000","last_fullsized":"2024-04-06T10:15:49.395195+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T09:29:24.717518+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T09:29:24.717518+0000","last_clean_scrub_stamp":"2024-04-06T09:29:24.717518+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T14:16:54.527503+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":361,"reported_epoch":343,"state":"active+clean","last_fresh":"2024-04-06T10:15:49.387668+0000","last_change":"2024-04-06T09:29:25.738138+0000","last_active":"2024-04-06T10:15:49.387668+0000","last_peered":"2024-04-06T10:15:49.387668+0000","last_clean":"2024-04-06T10:15:49.387668+0000","last_became_active":"2024-04-06T09:29:25.737263+0000","last_became_peered":"2024-04-06T09:29:25.737263+0000","last_unstale":"2024-04-06T10:15:49.387668+0000","last_undegraded":"2024-04-06T10:15:49.387668+0000","last_fullsized":"2024-04-06T10:15:49.387668+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T09:29:24.717518+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T09:29:24.717518+0000","last_clean_scrub_stamp":"2024-04-06T09:29:24.717518+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T14:30:58.846520+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":360,"reported_epoch":343,"state":"active+clean","last_fresh":"2024-04-06T10:15:49.819953+0000","last_change":"2024-04-06T09:29:25.739673+0000","last_active":"2024-04-06T10:15:49.819953+0000","last_peered":"2024-04-06T10:15:49.819953+0000","last_clean":"2024-04-06T10:15:49.819953+0000","last_became_active":"2024-04-06T09:29:25.738678+0000","last_became_peered":"2024-04-06T09:29:25.738678+0000","last_unstale":"2024-04-06T10:15:49.819953+0000","last_undegraded":"2024-04-06T10:15:49.819953+0000","last_fullsized":"2024-04-06T10:15:49.819953+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T09:29:24.717518+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T09:29:24.717518+0000","last_clean_scrub_stamp":"2024-04-06T09:29:24.717518+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T12:22:31.093977+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":360,"reported_epoch":342,"state":"active+clean","last_fresh":"2024-04-06T10:15:29.161731+0000","last_change":"2024-04-06T09:29:25.739006+0000","last_active":"2024-04-06T10:15:29.161731+0000","last_peered":"2024-04-06T10:15:29.161731+0000","last_clean":"2024-04-06T10:15:29.161731+0000","last_became_active":"2024-04-06T09:29:25.738605+0000","last_became_peered":"2024-04-06T09:29:25.738605+0000","last_unstale":"2024-04-06T10:15:29.161731+0000","last_undegraded":"2024-04-06T10:15:29.161731+0000","last_fullsized":"2024-04-06T10:15:29.161731+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T09:29:24.717518+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T09:29:24.717518+0000","last_clean_scrub_stamp":"2024-04-06T09:29:24.717518+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T20:26:25.794984+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":361,"reported_epoch":342,"state":"active+clean","last_fresh":"2024-04-06T10:15:38.753445+0000","last_change":"2024-04-06T09:29:25.739486+0000","last_active":"2024-04-06T10:15:38.753445+0000","last_peered":"2024-04-06T10:15:38.753445+0000","last_clean":"2024-04-06T10:15:38.753445+0000","last_became_active":"2024-04-06T09:29:25.738108+0000","last_became_peered":"2024-04-06T09:29:25.738108+0000","last_unstale":"2024-04-06T10:15:38.753445+0000","last_undegraded":"2024-04-06T10:15:38.753445+0000","last_fullsized":"2024-04-06T10:15:38.753445+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T09:29:24.717518+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T09:29:24.717518+0000","last_clean_scrub_stamp":"2024-04-06T09:29:24.717518+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T18:32:48.633225+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":361,"reported_epoch":343,"state":"active+clean","last_fresh":"2024-04-06T10:15:49.401068+0000","last_change":"2024-04-06T09:29:25.737897+0000","last_active":"2024-04-06T10:15:49.401068+0000","last_peered":"2024-04-06T10:15:49.401068+0000","last_clean":"2024-04-06T10:15:49.401068+0000","last_became_active":"2024-04-06T09:29:25.737369+0000","last_became_peered":"2024-04-06T09:29:25.737369+0000","last_unstale":"2024-04-06T10:15:49.401068+0000","last_undegraded":"2024-04-06T10:15:49.401068+0000","last_fullsized":"2024-04-06T10:15:49.401068+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T09:29:24.717518+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T09:29:24.717518+0000","last_clean_scrub_stamp":"2024-04-06T09:29:24.717518+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T16:57:27.614162+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"12'97","reported_seq":492,"reported_epoch":343,"state":"active+clean","last_fresh":"2024-04-06T10:15:49.389204+0000","last_change":"2024-04-06T09:29:21.046364+0000","last_active":"2024-04-06T10:15:49.389204+0000","last_peered":"2024-04-06T10:15:49.389204+0000","last_clean":"2024-04-06T10:15:49.389204+0000","last_became_active":"2024-04-06T09:29:21.043913+0000","last_became_peered":"2024-04-06T09:29:21.043913+0000","last_unstale":"2024-04-06T10:15:49.389204+0000","last_undegraded":"2024-04-06T10:15:49.389204+0000","last_fullsized":"2024-04-06T10:15:49.389204+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T09:29:19.704883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T09:29:19.704883+0000","last_clean_scrub_stamp":"2024-04-06T09:29:19.704883+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T14:04:37.575084+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":122880,"data_stored":1180736,"data_compressed":60676,"data_compressed_allocated":81920,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738940,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1745176,"kb_used_data":2768,"kb_used_omap":120,"kb_used_meta":1742279,"kb_avail":92626664,"statfs":{"total":96636764160,"available":94849703936,"internally_reserved":0,"allocated":2834432,"data_stored":1495629,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":122911,"internal_metadata":1784094689},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 6 10:15:47 2024","interfaces":[{"interface":"back","average":{"1min":0.503,"5min":0.53000000000000003,"15min":0.60999999999999999},"min":{"1min":0.42699999999999999,"5min":0.40200000000000002,"15min":0.26400000000000001},"max":{"1min":0.66300000000000003,"5min":0.90300000000000002,"15min":15.619},"last":0.55200000000000005},{"interface":"front","average":{"1min":0.67600000000000005,"5min":0.68100000000000005,"15min":0.76300000000000001},"min":{"1min":0.57099999999999995,"5min":0.55300000000000005,"15min":0.36399999999999999},"max":{"1min":0.76200000000000001,"5min":0.998,"15min":15.901},"last":0.72299999999999998}]},{"osd":1,"last update":"Sat Apr 6 10:15:47 2024","interfaces":[{"interface":"back","average":{"1min":0.58999999999999997,"5min":0.60199999999999998,"15min":0.61499999999999999},"min":{"1min":0.51900000000000002,"5min":0.47099999999999997,"15min":0.35399999999999998},"max":{"1min":0.65700000000000003,"5min":1.0629999999999999,"15min":4.3579999999999997},"last":0.65900000000000003},{"interface":"front","average":{"1min":0.56399999999999995,"5min":0.58599999999999997,"15min":0.61499999999999999},"min":{"1min":0.46999999999999997,"5min":0.45300000000000001,"15min":0.32500000000000001},"max":{"1min":0.67300000000000004,"5min":0.94799999999999995,"15min":4.2119999999999997},"last":0.59599999999999997}]}]},{"osd":1,"up_from":8,"seq":34359738940,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1310172,"kb_used_data":2828,"kb_used_omap":236,"kb_used_meta":1307091,"kb_avail":93061668,"statfs":{"total":96636764160,"available":95295148032,"internally_reserved":0,"allocated":2895872,"data_stored":2085997,"data_compressed":30338,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":241997,"internal_metadata":1338461875},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 6 10:16:15 2024","interfaces":[{"interface":"back","average":{"1min":0.51500000000000001,"5min":0.51400000000000001,"15min":0.53800000000000003},"min":{"1min":0.38200000000000001,"5min":0.38200000000000001,"15min":0.214},"max":{"1min":0.68600000000000005,"5min":0.73799999999999999,"15min":4.3319999999999999},"last":0.44800000000000001},{"interface":"front","average":{"1min":0.65300000000000002,"5min":0.65100000000000002,"15min":0.64200000000000002},"min":{"1min":0.497,"5min":0.497,"15min":0.255},"max":{"1min":0.75600000000000001,"5min":0.86099999999999999,"15min":4.2960000000000003},"last":0.59399999999999997}]},{"osd":2,"last update":"Sat Apr 6 10:15:56 2024","interfaces":[{"interface":"back","average":{"1min":0.64400000000000002,"5min":0.64500000000000002,"15min":0.67500000000000004},"min":{"1min":0.47899999999999998,"5min":0.47899999999999998,"15min":0.316},"max":{"1min":0.80100000000000005,"5min":0.96599999999999997,"15min":4.2229999999999999},"last":0.50900000000000001},{"interface":"front","average":{"1min":0.63400000000000001,"5min":0.63300000000000001,"15min":0.65100000000000002},"min":{"1min":0.46999999999999997,"5min":0.44900000000000001,"15min":0.29299999999999998},"max":{"1min":0.82199999999999995,"5min":1.26,"15min":3.1579999999999999},"last":0.65500000000000003}]}]},{"osd":0,"up_from":8,"seq":34359738940,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1807388,"kb_used_data":2828,"kb_used_omap":281,"kb_used_meta":1804262,"kb_avail":92564452,"statfs":{"total":96636764160,"available":94785998848,"internally_reserved":0,"allocated":2895872,"data_stored":2085997,"data_compressed":30338,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":288743,"internal_metadata":1847564313},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Apr 6 10:15:49 2024","interfaces":[{"interface":"back","average":{"1min":0.53200000000000003,"5min":0.54900000000000004,"15min":0.55300000000000005},"min":{"1min":0.44800000000000001,"5min":0.24199999999999999,"15min":0.17999999999999999},"max":{"1min":0.70699999999999996,"5min":2.173,"15min":3.7679999999999998},"last":0.63600000000000001},{"interface":"front","average":{"1min":0.63,"5min":0.60899999999999999,"15min":0.61499999999999999},"min":{"1min":0.5,"5min":0.42399999999999999,"15min":0.216},"max":{"1min":0.81399999999999995,"5min":1.105,"15min":3.9969999999999999},"last":0.55900000000000005}]},{"osd":2,"last update":"Sat Apr 6 10:15:49 2024","interfaces":[{"interface":"back","average":{"1min":0.66700000000000004,"5min":0.65700000000000003,"15min":0.65400000000000003},"min":{"1min":0.52100000000000002,"5min":0.47099999999999997,"15min":0.26100000000000001},"max":{"1min":0.79600000000000004,"5min":1.3580000000000001,"15min":4.1609999999999996},"last":0.68000000000000005},{"interface":"front","average":{"1min":0.60099999999999998,"5min":0.61499999999999999,"15min":0.626},"min":{"1min":0.48799999999999999,"5min":0.28100000000000003,"15min":0.23200000000000001},"max":{"1min":0.69399999999999995,"5min":0.95499999999999996,"15min":2.4729999999999999},"last":0.71899999999999997}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":590368,"data_compressed":30338,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":590368,"data_compressed":30338,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-06T10:16:41.823 INFO:tasks.ceph:pgid 2.7 last_scrub_stamp 2024-04-06T09:29:24.717518+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=6, tm_hour=9, tm_min=29, tm_sec=24, tm_wday=5, tm_yday=97, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=6, tm_hour=10, tm_min=16, tm_sec=39, tm_wday=5, tm_yday=97, tm_isdst=0) 2024-04-06T10:16:41.823 INFO:tasks.ceph:pgid 2.6 last_scrub_stamp 2024-04-06T09:29:24.717518+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=6, tm_hour=9, tm_min=29, tm_sec=24, tm_wday=5, tm_yday=97, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=6, tm_hour=10, tm_min=16, tm_sec=39, tm_wday=5, tm_yday=97, tm_isdst=0) 2024-04-06T10:16:41.823 INFO:tasks.ceph:pgid 2.5 last_scrub_stamp 2024-04-06T09:29:24.717518+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=6, tm_hour=9, tm_min=29, tm_sec=24, tm_wday=5, tm_yday=97, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=6, tm_hour=10, tm_min=16, tm_sec=39, tm_wday=5, tm_yday=97, tm_isdst=0) 2024-04-06T10:16:41.823 INFO:tasks.ceph:pgid 2.4 last_scrub_stamp 2024-04-06T09:29:24.717518+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=6, tm_hour=9, tm_min=29, tm_sec=24, tm_wday=5, tm_yday=97, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=6, tm_hour=10, tm_min=16, tm_sec=39, tm_wday=5, tm_yday=97, tm_isdst=0) 2024-04-06T10:16:41.823 INFO:tasks.ceph:pgid 2.2 last_scrub_stamp 2024-04-06T09:29:24.717518+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=6, tm_hour=9, tm_min=29, tm_sec=24, tm_wday=5, tm_yday=97, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=6, tm_hour=10, tm_min=16, tm_sec=39, tm_wday=5, tm_yday=97, tm_isdst=0) 2024-04-06T10:16:41.824 INFO:tasks.ceph:pgid 2.1 last_scrub_stamp 2024-04-06T09:29:24.717518+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=6, tm_hour=9, tm_min=29, tm_sec=24, tm_wday=5, tm_yday=97, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=6, tm_hour=10, tm_min=16, tm_sec=39, tm_wday=5, tm_yday=97, tm_isdst=0) 2024-04-06T10:16:41.824 INFO:tasks.ceph:pgid 2.0 last_scrub_stamp 2024-04-06T09:29:24.717518+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=6, tm_hour=9, tm_min=29, tm_sec=24, tm_wday=5, tm_yday=97, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=6, tm_hour=10, tm_min=16, tm_sec=39, tm_wday=5, tm_yday=97, tm_isdst=0) 2024-04-06T10:16:41.824 INFO:tasks.ceph:pgid 2.3 last_scrub_stamp 2024-04-06T09:29:24.717518+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=6, tm_hour=9, tm_min=29, tm_sec=24, tm_wday=5, tm_yday=97, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=6, tm_hour=10, tm_min=16, tm_sec=39, tm_wday=5, tm_yday=97, tm_isdst=0) 2024-04-06T10:16:41.824 INFO:tasks.ceph:pgid 1.0 last_scrub_stamp 2024-04-06T09:29:19.704883+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=6, tm_hour=9, tm_min=29, tm_sec=19, tm_wday=5, tm_yday=97, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=6, tm_hour=10, tm_min=16, tm_sec=39, tm_wday=5, tm_yday=97, tm_isdst=0) 2024-04-06T10:16:41.824 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2024-04-06T10:17:01.825 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 pg dump --format=json 2024-04-06T10:17:02.133 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-04-06T10:17:02.133 INFO:teuthology.orchestra.run.smithi006.stderr:dumped all 2024-04-06T10:17:02.151 INFO:teuthology.orchestra.run.smithi006.stdout:{"pg_ready":true,"pg_map":{"version":1773,"stamp":"2024-04-06T10:17:00.330169+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":4862720,"kb_used_data":8424,"kb_used_omap":638,"kb_used_meta":4853633,"kb_avail":278252800,"statfs":{"total":289910292480,"available":284930867200,"internally_reserved":0,"allocated":8626176,"data_stored":5667623,"data_compressed":60676,"data_compressed_allocated":81920,"data_compressed_original":1146880,"omap_allocated":653651,"internal_metadata":4970120877},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003040"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":372,"reported_epoch":344,"state":"active+clean","last_fresh":"2024-04-06T10:16:55.179337+0000","last_change":"2024-04-06T10:16:55.179473+0000","last_active":"2024-04-06T10:16:55.179337+0000","last_peered":"2024-04-06T10:16:55.179337+0000","last_clean":"2024-04-06T10:16:55.179337+0000","last_became_active":"2024-04-06T09:29:25.737557+0000","last_became_peered":"2024-04-06T09:29:25.737557+0000","last_unstale":"2024-04-06T10:16:55.179337+0000","last_undegraded":"2024-04-06T10:16:55.179337+0000","last_fullsized":"2024-04-06T10:16:55.179337+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T10:16:55.179319+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T10:16:55.179319+0000","last_clean_scrub_stamp":"2024-04-06T10:16:55.179319+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T12:05:48.954082+0000","scrub_duration":0.35756300899999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":372,"reported_epoch":344,"state":"active+clean","last_fresh":"2024-04-06T10:16:55.980983+0000","last_change":"2024-04-06T10:16:55.981228+0000","last_active":"2024-04-06T10:16:55.980983+0000","last_peered":"2024-04-06T10:16:55.980983+0000","last_clean":"2024-04-06T10:16:55.980983+0000","last_became_active":"2024-04-06T09:29:25.737348+0000","last_became_peered":"2024-04-06T09:29:25.737348+0000","last_unstale":"2024-04-06T10:16:55.980983+0000","last_undegraded":"2024-04-06T10:16:55.980983+0000","last_fullsized":"2024-04-06T10:16:55.980983+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T10:16:55.980933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T10:16:55.980933+0000","last_clean_scrub_stamp":"2024-04-06T10:16:55.980933+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T21:33:26.449904+0000","scrub_duration":0.161957358,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":373,"reported_epoch":344,"state":"active+clean","last_fresh":"2024-04-06T10:16:45.133269+0000","last_change":"2024-04-06T10:16:45.133434+0000","last_active":"2024-04-06T10:16:45.133269+0000","last_peered":"2024-04-06T10:16:45.133269+0000","last_clean":"2024-04-06T10:16:45.133269+0000","last_became_active":"2024-04-06T09:29:25.738591+0000","last_became_peered":"2024-04-06T09:29:25.738591+0000","last_unstale":"2024-04-06T10:16:45.133269+0000","last_undegraded":"2024-04-06T10:16:45.133269+0000","last_fullsized":"2024-04-06T10:16:45.133269+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T10:16:45.133251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T10:16:45.133251+0000","last_clean_scrub_stamp":"2024-04-06T10:16:45.133251+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T14:45:09.612935+0000","scrub_duration":0.34021375199999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":373,"reported_epoch":344,"state":"active+clean","last_fresh":"2024-04-06T10:16:47.833423+0000","last_change":"2024-04-06T10:16:47.833700+0000","last_active":"2024-04-06T10:16:47.833423+0000","last_peered":"2024-04-06T10:16:47.833423+0000","last_clean":"2024-04-06T10:16:47.833423+0000","last_became_active":"2024-04-06T09:29:25.737263+0000","last_became_peered":"2024-04-06T09:29:25.737263+0000","last_unstale":"2024-04-06T10:16:47.833423+0000","last_undegraded":"2024-04-06T10:16:47.833423+0000","last_fullsized":"2024-04-06T10:16:47.833423+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T10:16:47.833373+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T10:16:47.833373+0000","last_clean_scrub_stamp":"2024-04-06T10:16:47.833373+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T13:54:02.468653+0000","scrub_duration":0.003202253,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":372,"reported_epoch":344,"state":"active+clean","last_fresh":"2024-04-06T10:16:42.253492+0000","last_change":"2024-04-06T10:16:42.253645+0000","last_active":"2024-04-06T10:16:42.253492+0000","last_peered":"2024-04-06T10:16:42.253492+0000","last_clean":"2024-04-06T10:16:42.253492+0000","last_became_active":"2024-04-06T09:29:25.738678+0000","last_became_peered":"2024-04-06T09:29:25.738678+0000","last_unstale":"2024-04-06T10:16:42.253492+0000","last_undegraded":"2024-04-06T10:16:42.253492+0000","last_fullsized":"2024-04-06T10:16:42.253492+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T10:16:42.253458+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T10:16:42.253458+0000","last_clean_scrub_stamp":"2024-04-06T10:16:42.253458+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T13:43:13.000659+0000","scrub_duration":0.18550340500000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":372,"reported_epoch":344,"state":"active+clean","last_fresh":"2024-04-06T10:16:50.238473+0000","last_change":"2024-04-06T10:16:50.238604+0000","last_active":"2024-04-06T10:16:50.238473+0000","last_peered":"2024-04-06T10:16:50.238473+0000","last_clean":"2024-04-06T10:16:50.238473+0000","last_became_active":"2024-04-06T09:29:25.738605+0000","last_became_peered":"2024-04-06T09:29:25.738605+0000","last_unstale":"2024-04-06T10:16:50.238473+0000","last_undegraded":"2024-04-06T10:16:50.238473+0000","last_fullsized":"2024-04-06T10:16:50.238473+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T10:16:50.238455+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T10:16:50.238455+0000","last_clean_scrub_stamp":"2024-04-06T10:16:50.238455+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T13:17:28.383133+0000","scrub_duration":0.34182422099999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":373,"reported_epoch":344,"state":"active+clean","last_fresh":"2024-04-06T10:16:42.152522+0000","last_change":"2024-04-06T10:16:42.152646+0000","last_active":"2024-04-06T10:16:42.152522+0000","last_peered":"2024-04-06T10:16:42.152522+0000","last_clean":"2024-04-06T10:16:42.152522+0000","last_became_active":"2024-04-06T09:29:25.738108+0000","last_became_peered":"2024-04-06T09:29:25.738108+0000","last_unstale":"2024-04-06T10:16:42.152522+0000","last_undegraded":"2024-04-06T10:16:42.152522+0000","last_fullsized":"2024-04-06T10:16:42.152522+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T10:16:42.152504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T10:16:42.152504+0000","last_clean_scrub_stamp":"2024-04-06T10:16:42.152504+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T21:52:00.340203+0000","scrub_duration":0.26372385300000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":373,"reported_epoch":344,"state":"active+clean","last_fresh":"2024-04-06T10:16:49.263221+0000","last_change":"2024-04-06T10:16:49.263340+0000","last_active":"2024-04-06T10:16:49.263221+0000","last_peered":"2024-04-06T10:16:49.263221+0000","last_clean":"2024-04-06T10:16:49.263221+0000","last_became_active":"2024-04-06T09:29:25.737369+0000","last_became_peered":"2024-04-06T09:29:25.737369+0000","last_unstale":"2024-04-06T10:16:49.263221+0000","last_undegraded":"2024-04-06T10:16:49.263221+0000","last_fullsized":"2024-04-06T10:16:49.263221+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T10:16:49.263202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T10:16:49.263202+0000","last_clean_scrub_stamp":"2024-04-06T10:16:49.263202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T20:58:52.806771+0000","scrub_duration":0.39977122999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"344'99","reported_seq":506,"reported_epoch":344,"state":"active+clean","last_fresh":"2024-04-06T10:16:43.775794+0000","last_change":"2024-04-06T10:16:43.776082+0000","last_active":"2024-04-06T10:16:43.775794+0000","last_peered":"2024-04-06T10:16:43.775794+0000","last_clean":"2024-04-06T10:16:43.775794+0000","last_became_active":"2024-04-06T09:29:21.043913+0000","last_became_peered":"2024-04-06T09:29:21.043913+0000","last_unstale":"2024-04-06T10:16:43.775794+0000","last_undegraded":"2024-04-06T10:16:43.775794+0000","last_fullsized":"2024-04-06T10:16:43.775794+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"344'99","last_scrub_stamp":"2024-04-06T10:16:43.775740+0000","last_deep_scrub":"344'99","last_deep_scrub_stamp":"2024-04-06T10:16:43.775740+0000","last_clean_scrub_stamp":"2024-04-06T10:16:43.775740+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T15:17:05.403687+0000","scrub_duration":0.010422407999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":122880,"data_stored":1180736,"data_compressed":60676,"data_compressed_allocated":81920,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738944,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1745176,"kb_used_data":2768,"kb_used_omap":120,"kb_used_meta":1742279,"kb_avail":92626664,"statfs":{"total":96636764160,"available":94849703936,"internally_reserved":0,"allocated":2834432,"data_stored":1495629,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":122911,"internal_metadata":1784094689},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 6 10:16:49 2024","interfaces":[{"interface":"back","average":{"1min":0.48599999999999999,"5min":0.52300000000000002,"15min":0.60699999999999998},"min":{"1min":0.308,"5min":0.308,"15min":0.26400000000000001},"max":{"1min":0.58799999999999997,"5min":0.90300000000000002,"15min":15.619},"last":0.442},{"interface":"front","average":{"1min":0.66600000000000004,"5min":0.67800000000000005,"15min":0.76100000000000001},"min":{"1min":0.42199999999999999,"5min":0.42199999999999999,"15min":0.36399999999999999},"max":{"1min":0.78400000000000003,"5min":0.998,"15min":15.901},"last":0.67900000000000005}]},{"osd":1,"last update":"Sat Apr 6 10:16:49 2024","interfaces":[{"interface":"back","average":{"1min":0.59399999999999997,"5min":0.59799999999999998,"15min":0.61399999999999999},"min":{"1min":0.26900000000000002,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":0.71599999999999997,"5min":1.0629999999999999,"15min":4.3579999999999997},"last":0.60999999999999999},{"interface":"front","average":{"1min":0.57899999999999996,"5min":0.58799999999999997,"15min":0.60699999999999998},"min":{"1min":0.36899999999999999,"5min":0.36899999999999999,"15min":0.32500000000000001},"max":{"1min":0.74299999999999999,"5min":0.94799999999999995,"15min":4.2119999999999997},"last":0.56399999999999995}]}]},{"osd":1,"up_from":8,"seq":34359738944,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1310164,"kb_used_data":2828,"kb_used_omap":236,"kb_used_meta":1307091,"kb_avail":93061676,"statfs":{"total":96636764160,"available":95295156224,"internally_reserved":0,"allocated":2895872,"data_stored":2085997,"data_compressed":30338,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":241997,"internal_metadata":1338461875},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 6 10:16:15 2024","interfaces":[{"interface":"back","average":{"1min":0.51500000000000001,"5min":0.51400000000000001,"15min":0.53800000000000003},"min":{"1min":0.38200000000000001,"5min":0.38200000000000001,"15min":0.214},"max":{"1min":0.68600000000000005,"5min":0.73799999999999999,"15min":4.3319999999999999},"last":0.44800000000000001},{"interface":"front","average":{"1min":0.65300000000000002,"5min":0.65100000000000002,"15min":0.64200000000000002},"min":{"1min":0.497,"5min":0.497,"15min":0.255},"max":{"1min":0.75600000000000001,"5min":0.86099999999999999,"15min":4.2960000000000003},"last":0.66800000000000004}]},{"osd":2,"last update":"Sat Apr 6 10:15:56 2024","interfaces":[{"interface":"back","average":{"1min":0.64400000000000002,"5min":0.64500000000000002,"15min":0.67500000000000004},"min":{"1min":0.47899999999999998,"5min":0.47899999999999998,"15min":0.316},"max":{"1min":0.80100000000000005,"5min":0.96599999999999997,"15min":4.2229999999999999},"last":0.60599999999999998},{"interface":"front","average":{"1min":0.63400000000000001,"5min":0.63300000000000001,"15min":0.65100000000000002},"min":{"1min":0.46999999999999997,"5min":0.44900000000000001,"15min":0.29299999999999998},"max":{"1min":0.82199999999999995,"5min":1.26,"15min":3.1579999999999999},"last":0.48999999999999999}]}]},{"osd":0,"up_from":8,"seq":34359738944,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1807380,"kb_used_data":2828,"kb_used_omap":281,"kb_used_meta":1804262,"kb_avail":92564460,"statfs":{"total":96636764160,"available":94786007040,"internally_reserved":0,"allocated":2895872,"data_stored":2085997,"data_compressed":30338,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":288743,"internal_metadata":1847564313},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Apr 6 10:16:50 2024","interfaces":[{"interface":"back","average":{"1min":0.56100000000000005,"5min":0.55700000000000005,"15min":0.55200000000000005},"min":{"1min":0.46500000000000002,"5min":0.24199999999999999,"15min":0.17999999999999999},"max":{"1min":0.67800000000000005,"5min":2.173,"15min":3.7679999999999998},"last":0.45300000000000001},{"interface":"front","average":{"1min":0.58299999999999996,"5min":0.60899999999999999,"15min":0.61599999999999999},"min":{"1min":0.47099999999999997,"5min":0.42399999999999999,"15min":0.216},"max":{"1min":0.68999999999999995,"5min":1.105,"15min":3.9969999999999999},"last":0.54900000000000004}]},{"osd":2,"last update":"Sat Apr 6 10:16:50 2024","interfaces":[{"interface":"back","average":{"1min":0.65600000000000003,"5min":0.66100000000000003,"15min":0.65600000000000003},"min":{"1min":0.55500000000000005,"5min":0.52100000000000002,"15min":0.26100000000000001},"max":{"1min":0.80700000000000005,"5min":1.3580000000000001,"15min":4.1609999999999996},"last":0.66000000000000003},{"interface":"front","average":{"1min":0.64300000000000002,"5min":0.624,"15min":0.625},"min":{"1min":0.53600000000000003,"5min":0.28100000000000003,"15min":0.23200000000000001},"max":{"1min":0.75600000000000001,"5min":0.95499999999999996,"15min":2.4729999999999999},"last":0.59499999999999997}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":590368,"data_compressed":30338,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":590368,"data_compressed":30338,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-06T10:17:02.153 DEBUG:teuthology.orchestra.run.smithi006:> sudo ceph --cluster cluster1 config set global mon_health_to_clog false 2024-04-06T10:17:02.474 INFO:teuthology.misc:Shutting down mds daemons... 2024-04-06T10:17:02.475 INFO:teuthology.misc:Shutting down osd daemons... 2024-04-06T10:17:02.475 DEBUG:tasks.ceph.cluster1.osd.0:waiting for process to exit 2024-04-06T10:17:02.475 INFO:teuthology.orchestra.run:waiting for 300 2024-04-06T10:17:08.357 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster1.osd.0 is failed for ~0s 2024-04-06T10:17:08.576 INFO:tasks.ceph.cluster1.osd.0:Stopped 2024-04-06T10:17:08.577 DEBUG:tasks.ceph.cluster1.osd.1:waiting for process to exit 2024-04-06T10:17:08.577 INFO:teuthology.orchestra.run:waiting for 300 2024-04-06T10:17:08.669 INFO:tasks.ceph.cluster1.osd.1:Stopped 2024-04-06T10:17:08.669 DEBUG:tasks.ceph.cluster1.osd.2:waiting for process to exit 2024-04-06T10:17:08.669 INFO:teuthology.orchestra.run:waiting for 300 2024-04-06T10:17:13.560 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster1.osd.2 is failed for ~0s 2024-04-06T10:17:13.560 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster1.osd.0 has been restored 2024-04-06T10:17:14.771 INFO:tasks.ceph.cluster1.osd.2:Stopped 2024-04-06T10:17:14.771 INFO:teuthology.misc:Shutting down mgr daemons... 2024-04-06T10:17:14.772 DEBUG:tasks.ceph.cluster1.mgr.x:waiting for process to exit 2024-04-06T10:17:14.772 INFO:teuthology.orchestra.run:waiting for 300 2024-04-06T10:17:14.803 INFO:tasks.ceph.cluster1.mgr.x:Stopped 2024-04-06T10:17:14.803 INFO:teuthology.misc:Shutting down mon daemons... 2024-04-06T10:17:14.803 DEBUG:tasks.ceph.cluster1.mon.a:waiting for process to exit 2024-04-06T10:17:14.803 INFO:teuthology.orchestra.run:waiting for 300 2024-04-06T10:17:14.864 INFO:tasks.ceph.cluster1.mon.a:Stopped 2024-04-06T10:17:14.864 INFO:tasks.ceph:Checking cluster log for badness... 2024-04-06T10:17:14.864 DEBUG:teuthology.orchestra.run.smithi006:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/cluster1.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(OSD_SLOW_PING_TIME' | head -n 1 2024-04-06T10:17:14.964 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/cluster1-0 on ubuntu@smithi006.front.sepia.ceph.com 2024-04-06T10:17:14.964 DEBUG:teuthology.orchestra.run.smithi006:> sync && sudo umount -f /var/lib/ceph/osd/cluster1-0 2024-04-06T10:17:15.446 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/cluster1-1 on ubuntu@smithi006.front.sepia.ceph.com 2024-04-06T10:17:15.446 DEBUG:teuthology.orchestra.run.smithi006:> sync && sudo umount -f /var/lib/ceph/osd/cluster1-1 2024-04-06T10:17:15.830 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/cluster1-2 on ubuntu@smithi006.front.sepia.ceph.com 2024-04-06T10:17:15.830 DEBUG:teuthology.orchestra.run.smithi006:> sync && sudo umount -f /var/lib/ceph/osd/cluster1-2 2024-04-06T10:17:16.300 INFO:tasks.ceph:Archiving mon data... 2024-04-06T10:17:16.301 DEBUG:teuthology.misc:Transferring archived files from smithi006:/var/lib/ceph/mon/cluster1-a to /home/teuthworker/archive/teuthology-2024-04-02_22:16:03-rbd-reef-distro-default-smithi/7637502/data/cluster1.mon.a.tgz 2024-04-06T10:17:16.301 DEBUG:teuthology.orchestra.run.smithi006:> mktemp 2024-04-06T10:17:16.320 INFO:teuthology.orchestra.run.smithi006.stdout:/tmp/tmp.vgoBB23E2g 2024-04-06T10:17:16.320 DEBUG:teuthology.orchestra.run.smithi006:> sudo tar cz -f - -C /var/lib/ceph/mon/cluster1-a -- . > /tmp/tmp.vgoBB23E2g 2024-04-06T10:17:16.485 DEBUG:teuthology.orchestra.run.smithi006:> sudo chmod 0666 /tmp/tmp.vgoBB23E2g 2024-04-06T10:17:16.582 DEBUG:teuthology.orchestra.remote:smithi006:/tmp/tmp.vgoBB23E2g is 364KB 2024-04-06T10:17:16.646 DEBUG:teuthology.orchestra.run.smithi006:> rm -fr /tmp/tmp.vgoBB23E2g 2024-04-06T10:17:16.651 INFO:tasks.ceph:Cleaning ceph cluster... 2024-04-06T10:17:16.651 DEBUG:teuthology.orchestra.run.smithi006:> sudo rm -rf -- /etc/ceph/cluster1.conf /etc/ceph/cluster1.keyring /home/ubuntu/cephtest/cluster1.data /home/ubuntu/cephtest/cluster1.monmap /home/ubuntu/cephtest/../*.pid 2024-04-06T10:17:16.697 DEBUG:teuthology.orchestra.run.smithi192:> sudo rm -rf -- /etc/ceph/cluster1.conf /etc/ceph/cluster1.keyring /home/ubuntu/cephtest/cluster1.data /home/ubuntu/cephtest/cluster1.monmap /home/ubuntu/cephtest/../*.pid 2024-04-06T10:17:16.711 INFO:tasks.ceph:Checking for errors in any valgrind logs... 2024-04-06T10:17:16.711 DEBUG:teuthology.orchestra.run.smithi006:> sudo zgrep '' /var/log/ceph/valgrind/* /dev/null | sort | uniq 2024-04-06T10:17:16.754 DEBUG:teuthology.orchestra.run.smithi192:> sudo zgrep '' /var/log/ceph/valgrind/* /dev/null | sort | uniq 2024-04-06T10:17:16.775 INFO:teuthology.orchestra.run.smithi192.stderr:gzip: /var/log/ceph/valgrind/*.gz: No such file or directory 2024-04-06T10:17:16.779 INFO:teuthology.orchestra.run.smithi006.stderr:gzip: /var/log/ceph/valgrind/*.gz: No such file or directory 2024-04-06T10:17:16.783 INFO:tasks.ceph:Archiving crash dumps... 2024-04-06T10:17:16.785 DEBUG:teuthology.misc:Transferring archived files from smithi006:/var/lib/ceph/crash to /home/teuthworker/archive/teuthology-2024-04-02_22:16:03-rbd-reef-distro-default-smithi/7637502/remote/smithi006/crash 2024-04-06T10:17:16.786 DEBUG:teuthology.orchestra.run.smithi006:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-04-06T10:17:16.834 DEBUG:teuthology.misc:Transferring archived files from smithi192:/var/lib/ceph/crash to /home/teuthworker/archive/teuthology-2024-04-02_22:16:03-rbd-reef-distro-default-smithi/7637502/remote/smithi192/crash 2024-04-06T10:17:16.835 DEBUG:teuthology.orchestra.run.smithi192:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-04-06T10:17:16.846 INFO:tasks.ceph:Compressing logs... 2024-04-06T10:17:16.847 DEBUG:teuthology.orchestra.run.smithi006:> sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-04-06T10:17:16.881 DEBUG:teuthology.orchestra.run.smithi192:> sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-04-06T10:17:18.561 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster1.osd.2 has been restored 2024-04-06T10:22:35.025 INFO:tasks.ceph:Archiving logs... 2024-04-06T10:22:35.026 DEBUG:teuthology.misc:Transferring archived files from smithi006:/var/log/ceph to /home/teuthworker/archive/teuthology-2024-04-02_22:16:03-rbd-reef-distro-default-smithi/7637502/remote/smithi006/log 2024-04-06T10:22:35.027 DEBUG:teuthology.orchestra.run.smithi006:> sudo tar c -f - -C /var/log/ceph -- . 2024-04-06T10:22:54.678 DEBUG:teuthology.misc:Transferring archived files from smithi192:/var/log/ceph to /home/teuthworker/archive/teuthology-2024-04-02_22:16:03-rbd-reef-distro-default-smithi/7637502/remote/smithi192/log 2024-04-06T10:22:54.679 DEBUG:teuthology.orchestra.run.smithi192:> sudo tar c -f - -C /var/log/ceph -- . 2024-04-06T10:23:25.871 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-04-06T10:23:25.905 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-04-06T10:23:25.905 DEBUG:teuthology.orchestra.run.smithi006:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-04-06T10:23:25.908 DEBUG:teuthology.orchestra.run.smithi192:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-04-06T10:23:25.973 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-04-06T10:23:25.989 INFO:teuthology.task.clock:Checking final clock skew... 2024-04-06T10:23:25.989 DEBUG:teuthology.orchestra.run.smithi006:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-06T10:23:25.992 DEBUG:teuthology.orchestra.run.smithi192:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-06T10:23:26.038 INFO:teuthology.orchestra.run.smithi006.stdout: remote refid st t when poll reach delay offset jitter 2024-04-06T10:23:26.038 INFO:teuthology.orchestra.run.smithi006.stdout:============================================================================== 2024-04-06T10:23:26.039 INFO:teuthology.orchestra.run.smithi006.stdout:*hv01.front.sepi 44.4.53.2 2 u 99 128 377 0.125 -2.825 0.143 2024-04-06T10:23:26.039 INFO:teuthology.orchestra.run.smithi006.stdout:+hv02.front.sepi 63.231.80.2 3 u 106 128 377 0.200 -1.172 0.210 2024-04-06T10:23:26.039 INFO:teuthology.orchestra.run.smithi006.stdout:+hv03.front.sepi 74.6.168.72 3 u 79 128 377 0.201 -3.014 0.199 2024-04-06T10:23:26.039 INFO:teuthology.orchestra.run.smithi006.stdout: hv04.front.sepi .INIT. 16 u - 1024 0 0.000 +0.000 0.000 2024-04-06T10:23:26.043 INFO:teuthology.orchestra.run.smithi192.stdout: remote refid st t when poll reach delay offset jitter 2024-04-06T10:23:26.043 INFO:teuthology.orchestra.run.smithi192.stdout:============================================================================== 2024-04-06T10:23:26.043 INFO:teuthology.orchestra.run.smithi192.stdout:*hv01.front.sepi 44.4.53.2 2 u 115 128 377 0.182 -4.247 0.896 2024-04-06T10:23:26.043 INFO:teuthology.orchestra.run.smithi192.stdout:+hv02.front.sepi 63.231.80.2 3 u 98 128 377 0.162 -2.797 0.853 2024-04-06T10:23:26.043 INFO:teuthology.orchestra.run.smithi192.stdout:+hv03.front.sepi 74.6.168.72 3 u 30 128 377 0.191 -4.625 0.770 2024-04-06T10:23:26.043 INFO:teuthology.orchestra.run.smithi192.stdout: hv04.front.sepi .INIT. 16 u - 1024 0 0.000 +0.000 0.000 2024-04-06T10:23:26.044 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-04-06T10:23:26.077 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-04-06T10:23:26.078 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-04-06T10:23:26.159 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-04-06T10:23:26.174 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-04-06T10:23:26.192 INFO:teuthology.task.internal:Duration was 3807.618365 seconds 2024-04-06T10:23:26.193 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-04-06T10:23:26.208 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-04-06T10:23:26.208 DEBUG:teuthology.orchestra.run.smithi006:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-04-06T10:23:26.210 DEBUG:teuthology.orchestra.run.smithi192:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-04-06T10:23:26.380 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-04-06T10:23:26.380 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi006.front.sepia.ceph.com 2024-04-06T10:23:26.380 DEBUG:teuthology.orchestra.run.smithi006:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-04-06T10:23:26.392 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi192.front.sepia.ceph.com 2024-04-06T10:23:26.392 DEBUG:teuthology.orchestra.run.smithi192:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-04-06T10:23:26.404 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-04-06T10:23:26.404 DEBUG:teuthology.orchestra.run.smithi006:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-04-06T10:23:26.437 DEBUG:teuthology.orchestra.run.smithi192:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-04-06T10:23:26.489 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-04-06T10:23:26.489 DEBUG:teuthology.orchestra.run.smithi006:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-04-06T10:23:26.513 DEBUG:teuthology.orchestra.run.smithi192:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-04-06T10:23:26.694 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-04-06T10:23:26.708 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-04-06T10:23:26.708 DEBUG:teuthology.orchestra.run.smithi006:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-04-06T10:23:26.733 DEBUG:teuthology.orchestra.run.smithi192:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-04-06T10:23:26.757 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-04-06T10:23:26.773 DEBUG:teuthology.orchestra.run.smithi006:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-04-06T10:23:26.778 DEBUG:teuthology.orchestra.run.smithi192:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-04-06T10:23:26.797 INFO:teuthology.orchestra.run.smithi006.stdout:kernel.core_pattern = core 2024-04-06T10:23:26.818 INFO:teuthology.orchestra.run.smithi192.stdout:kernel.core_pattern = core 2024-04-06T10:23:26.847 DEBUG:teuthology.orchestra.run.smithi006:> test -e /home/ubuntu/cephtest/archive/coredump 2024-04-06T10:23:26.887 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T10:23:26.888 DEBUG:teuthology.orchestra.run.smithi192:> test -e /home/ubuntu/cephtest/archive/coredump 2024-04-06T10:23:26.893 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T10:23:26.893 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-04-06T10:23:26.906 INFO:teuthology.task.internal:Transferring archived files... 2024-04-06T10:23:26.907 DEBUG:teuthology.misc:Transferring archived files from smithi006:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/teuthology-2024-04-02_22:16:03-rbd-reef-distro-default-smithi/7637502/remote/smithi006 2024-04-06T10:23:26.907 DEBUG:teuthology.orchestra.run.smithi006:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-04-06T10:23:26.953 DEBUG:teuthology.misc:Transferring archived files from smithi192:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/teuthology-2024-04-02_22:16:03-rbd-reef-distro-default-smithi/7637502/remote/smithi192 2024-04-06T10:23:26.953 DEBUG:teuthology.orchestra.run.smithi192:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-04-06T10:23:26.975 INFO:teuthology.task.internal:Removing archive directory... 2024-04-06T10:23:26.975 DEBUG:teuthology.orchestra.run.smithi006:> rm -rf -- /home/ubuntu/cephtest/archive 2024-04-06T10:23:26.993 DEBUG:teuthology.orchestra.run.smithi192:> rm -rf -- /home/ubuntu/cephtest/archive 2024-04-06T10:23:27.016 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-04-06T10:23:27.033 INFO:teuthology.task.internal:Not uploading archives. 2024-04-06T10:23:27.033 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-04-06T10:23:27.050 INFO:teuthology.task.internal:Tidying up after the test... 2024-04-06T10:23:27.050 DEBUG:teuthology.orchestra.run.smithi006:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-04-06T10:23:27.053 DEBUG:teuthology.orchestra.run.smithi192:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-04-06T10:23:27.056 INFO:teuthology.orchestra.run.smithi006.stdout: 658640 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Apr 6 10:23 /home/ubuntu/cephtest 2024-04-06T10:23:27.064 INFO:teuthology.orchestra.run.smithi192.stdout: 658625 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Apr 6 10:23 /home/ubuntu/cephtest 2024-04-06T10:23:27.066 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-04-06T10:23:27.081 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-04-06T10:23:27.118 INFO:teuthology.nuke:Checking targets against current locks 2024-04-06T10:23:27.146 DEBUG:teuthology.nuke:shortname: smithi006 2024-04-06T10:23:27.146 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-04-06T10:23:27.171 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi006.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-04-02_22:16:03-rbd-reef-distro-default-smithi/7637502', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'ubuntu', 'os_version': '22.04', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-06 09:11:24.751410', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBPRsjjIxbbsh/IkYvxkow7Ob8LWlTfeg943sr/m1r913UrvId+M7tk4dUHrqVGT1fibXJxU9TjyR75QZn7tVsG0='} 2024-04-06T10:23:27.175 DEBUG:teuthology.nuke:shortname: smithi192 2024-04-06T10:23:27.175 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-04-06T10:23:27.199 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi192.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-04-02_22:16:03-rbd-reef-distro-default-smithi/7637502', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'ubuntu', 'os_version': '22.04', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-06 09:11:24.749428', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBFwM+L3rHFCaXoKHuY/3HZtsZsr61XOLlFJZtgkQKNRHUQSe/PZxCqw1jcy0QQ7XEb3wk4LbI1nEnCQwkpC2S8M='} 2024-04-06T10:23:27.229 INFO:teuthology.orchestra.console.smithi006:Power off 2024-04-06T10:23:27.229 DEBUG:teuthology.orchestra.console.smithi006:pexpect command: ipmitool -H smithi006.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-04-06T10:23:27.254 DEBUG:teuthology.orchestra.console.smithi006:power off output: Chassis Power Control: Down/Off 2024-04-06T10:23:27.255 DEBUG:teuthology.orchestra.console.smithi006:pexpect command: ipmitool -H smithi006.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-06T10:23:27.272 INFO:teuthology.orchestra.console.smithi192:Power off 2024-04-06T10:23:27.272 DEBUG:teuthology.orchestra.console.smithi192:pexpect command: ipmitool -H smithi192.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-04-06T10:23:27.283 DEBUG:teuthology.orchestra.console.smithi006:check power output: Chassis Power is on 2024-04-06T10:23:27.295 DEBUG:teuthology.orchestra.console.smithi192:power off output: Chassis Power Control: Down/Off 2024-04-06T10:23:27.295 DEBUG:teuthology.orchestra.console.smithi192:pexpect command: ipmitool -H smithi192.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-06T10:23:27.318 DEBUG:teuthology.orchestra.console.smithi192:check power output: Chassis Power is on 2024-04-06T10:23:31.284 DEBUG:teuthology.orchestra.console.smithi006:pexpect command: ipmitool -H smithi006.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-06T10:23:31.320 DEBUG:teuthology.orchestra.console.smithi192:pexpect command: ipmitool -H smithi192.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-06T10:23:31.398 DEBUG:teuthology.orchestra.console.smithi006:check power output: Chassis Power is on 2024-04-06T10:23:31.430 DEBUG:teuthology.orchestra.console.smithi192:check power output: Chassis Power is on 2024-04-06T10:23:35.401 DEBUG:teuthology.orchestra.console.smithi006:pexpect command: ipmitool -H smithi006.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-06T10:23:35.431 DEBUG:teuthology.orchestra.console.smithi192:pexpect command: ipmitool -H smithi192.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-06T10:23:35.512 DEBUG:teuthology.orchestra.console.smithi006:check power output: Chassis Power is off 2024-04-06T10:23:35.540 DEBUG:teuthology.orchestra.console.smithi192:check power output: Chassis Power is off 2024-04-06T10:23:35.613 INFO:teuthology.orchestra.console.smithi006:Power off completed 2024-04-06T10:23:35.641 INFO:teuthology.orchestra.console.smithi192:Power off completed 2024-04-06T10:23:35.746 INFO:teuthology.run:Summary data: description: rbd/mirror-thrash/{base/install clients/mirror cluster/{2-node openstack} conf/{disable-pool-app} msgr-failures/few objectstore/bluestore-comp-snappy policy/none rbd-mirror/four-per-cluster supported-random-distro$/{ubuntu_latest} workloads/rbd-mirror-journal-stress-workunit} duration: 3807.6183648109436 failure_reason: '"2024-04-06T09:32:51.464342+0000 mon.a (mon.0) 214 : cluster [WRN] Health check failed: Reduced data availability: 1 pg inactive, 1 pg peering (PG_AVAILABILITY)" in cluster log' flavor: default owner: scheduled_teuthology@teuthology success: false 2024-04-06T10:23:35.746 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-04-06T10:23:35.836 INFO:teuthology.run:FAIL